CS294-32: Dynamic Data

Race Detection
Koushik Sen
UC Berkeley
 Race Conditions

class Ref {
int i;
void inc() {
int t = i + 1;
i = t;
}
}

Courtesy Cormac Flanagan

 Race Conditions

class Ref {
int i; A race condition occurs if
void inc() {
int t = i + 1; • two threads access a
i = t; shared variable at the
} same time without
}
synchronization
Ref x = new Ref(0); • at least one of those
parallel {
accesses is a write
x.inc(); // two calls happen
x.inc(); // in parallel
}
assert x.i == 2;
 Race Conditions

t1 t2
class Ref {
int i;
RD(i)
void inc() {
int t = i + 1; RD(i)
i = t;
} WR(i)
}

WR(i)
Ref x = new Ref(0);
parallel {
x.inc(); // two calls happen
x.inc(); // in parallel
}
assert x.i == 2;
 Lock-Based Synchronization
class Ref {
int i; // guarded by this
void inc() {
synchronized (this) { • Field guarded by a
int t = i + 1; lock
• Lock acquired
i = t;
}
} before accessing
}
field
• Ensures race
Ref x = new Ref(0);
parallel {
x.inc(); // two calls happen freedom
x.inc(); // in parallel
}
assert x.i == 2;
 Dynamic Race Detection
• Happens Before [Dinning and Schonberg
1991]
• Lockset:
– Eraser [Savage et al. 1997]
– Precise Lockset [Choi et al. 2002]
• Hybrid [O'Callahan and Choi 2003]
 Dynamic Race Detection
• Advantages
– Precise knowledge of the execution
• No False positive [Happens Before]
• Unless you try to predict data races [Lockset]
• Disadvantages
– Produce false negatives
• because they only consider a subset of possible
program executions.
 What we are going to analyze?
• A trace representing an actual execution of a
program
• Trace is sequence of events:
– MEM(m,a,t): thread t accessed memory local m,
where the access a 2 {RD,WR}
• m can be o.f, C.f, a[i]
– ACQ(l,t): thread t acquires lock l
• Ignore re-acquire of locks. l can be o.
– REL(l,t): thread t releases lock l
– SND(g,t): thread t sends message g
– RCV(g,t): thread t receive message g
• If t1 calls t2.start(), then generate SND(g,t1) and
RCV(g,t2)
• If t1 calls t2.join(), then generate SND(g,t2) and RCV(g,t1)
 How to generate a trace?
class Ref {
Instrument a Program int i; // guarded by this
void inc() {
class Ref { print(“ACQ(“+id(this)+”,”+thisThread+”)”);
int i; // guarded by this synchronized (this) {
void inc() { print(“MEM(“+id(this)+”.i,RD,”+thisThread+”)”);
synchronized (this) {
int t = i + 1;
int t = i + 1;
print(“MEM(“+id(this)+”.i,WR,”+thisThread+”)”);
i = t;
i = t;
}
}
}
print(“REL(“+id(this)+”,”+thisThread+”)”);
}
}
}
Ref x = new Ref(0);
Ref x = new Ref(0);
parallel {
parallel {
x.inc(); // two calls happen
x.inc(); // two calls happen
x.inc(); // in parallel
x.inc(); // in parallel
}
}
assert x.i == 2;
assert x.i == 2;
 Sample Trace
class Ref {
int i; // guarded by this
Sample Trace void inc() {
print(“ACQ(“+id(this)+”,”+thisThread+”)”);
ACQ(4365,t1); synchronized (this) {
MEM(4365.i,RD,t1) print(“MEM(“+id(this)+”.i,RD,”+thisThread+”)”);
print(“MEM(“+id(this)+”.i,WR,”+thisThread+”)”);
MEM(4365.i,WR,t1)
i = i + 1;
REL(4365,t1); }
ACQ(4365,t2); print(“REL(“+id(this)+”,”+thisThread+”)”);
}
MEM(4365.i,RD,t2) }
MEM(4365.i,WR,t2)
Ref x = new Ref(0);
REL(4365,t2); parallel {
x.inc(); // two calls happen
x.inc(); // in parallel
}
assert x.i == 2;
Compute Locks Held by a Thread
L(t) = locks held by thread t. How do we compute L(t)?

Locks Held
Sample Trace
L(t1)={}, L(t2)={}
ACQ(4365,t1); L(t1)={4365}, L(t2)={}
MEM(4365.i,RD,t1) L(t1)={4365}, L(t2)={}
MEM(4365.i,WR,t1) L(t1)={4365}, L(t2)={}
REL(4365,t1); L(t1)={}, L(t2)={}
ACQ(4365,t2); L(t1)={}, L(t2)={4365}
MEM(4365.i,RD,t2) L(t1)={}, L(t2)={4365}
MEM(4365.i,WR,t2) L(t1)={}, L(t2)={4365}
REL(4365,t2); L(t1)={}, L(t2)={}
 Let us now analyze a trace
• Instrument Program
• Run Program => A Trace File
• Analyze Trace File
 Happens-before relation
• [Dinning and Schonberg 1991]
• Idea: Infer a happens-before relation Á between events
in a trace
• We say e1 Á e2
– If e1 and e2 are events from the same thread and e1 appears
before e2 in the trace
– If e1 = SND(g,t) and e2 = RCV(g,t’)
– If there is a e’ such that e1 Á e’ and e’ Á e2
• REL(l,t) and ACQ(g,t’) generates SND(g,t) and RCV(g,t’)
• We say e1 and e2 are in race, if
– e1 and e2 are not related by Á,
– e1 and e2 are from different threads
– e1 and e2 access the same memory location and one of the
accesses is a write
 Happens-before: example 1
Thread 1 Thread 2
x := x + 1

ACQ(mutex)

v := v + 1
REL(mutex)

Any two accesses

of shared variables ACQ(mutex)

are in the relation v := v + 1

happens-before REL(mutex)

x := x + 1
Happens-before: example 2
Thread 1 Thread 2
ACQ(mutex)

v := v + 1

REL(mutex)

x := x + 1

Therefore, only
x := x + 1 this second
ACQ(mutex) execution reveals
the existing
v := v + 1
datarace!!
REL(mutex)
 Eraser Lockset
• Savage,Burrows,Nelson,Sobalvarro,Anderson
• Assume a database D storing tuples
(m,L)
where:
– m is a memory location
– L is a set of locks that protect m
• Initially D contains a tuple (m,U) for
each memory location m, where U is the
universal set
 How it works?
• For an event MEM(m,a,t) generate the
tuple (m,L(t))
• Let (m, L’) be the tuple present in D
– Report race over memory location m
if L(t) Å L’ = empty set

• Replace (m, L’) by (m, L(t) Å L’) in D

 Eraser: Example 1
Thread 1 Thread 2

ACQ(mutex)

v := v + 1 L(t2) = {mutex}
(v,{mutex}) 2 D
REL(mutex)

L(t1)={mutex} ACQ(mutex)
(v,{mutex}) 2 D
v := v + 1

REL(mutex)
 Eraser: Example 2
Thread 1 Thread 2

ACQ(mutex1)

v := v + 1 L(t2) = {mutex2}
(v,{}) 2 D
REL(mutex1)

L(t1) = {mutex1} ACQ(mutex2)

(v,{mutex1}) 2 D
v := v + 1

REL(mutex2)

Warning!!
 Lockset

any thread
r/w

Shared-read/write
Shared-exclusive
Track lockset

race condition!
 Extending Lockset (Thread
first thread Local Data)
r/w

Thread
Local

second
thread
r/w
any thread
r/w

Shared-read/write
Shared-exclusive
Track lockset

race condition!
 Extending Lockset (Read
first thread Shared Data)
r/w

Thread
Local
second
thread
second read
thread
any thread write
r/w

Shared-read/write
Shared-exclusive Read
any thread
Track lockset any thread Shared read
write

race condition!
 Eraser: Problem

Thread 1 Thread 2 Thread 3

ACQ(L1,L2) ACQ(L2,L3) ACQ(L3,L1)

v := v + 1 v := v + 1 v := v + 1

REL(L2,L1) REL(L3,L2) REL(L1,L3)

T1(L1,L2)
 false alarm
T2(L2,L3) v

T3(L1,L3)
 Precise Lockset
• Choi, Lee, Loginov, O'Callahan, Sarkar, Sridharan

• Assume a database D storing tuples

(m,t,L,a)
where:
– m is a memory location
– t is a thread accessing m
– L is a set of locks held by t while accessing m
– a is the type of access (read or write)

• Initially D is empty
 How it works?
• For an event MEM(m,a,t) generate the tuple
(m,a,L(t),t)

• If there is a tuple (m’,a’,L’,t’) in D such that

– m = m’,
– (a = WR) Ç (a’=WR)
– L(t) Å L’ = empty set
– t  t’
– Report race over memory location m

• Add (m,a,L(t),t) in D
 Optimizations
• Stop adding tuples on m once a race on
m is detected
• Do not add (m,a,L,t) to D if (m,a,L’,t) is
already in D and L’ µ L
• Many more …
 Precise Lockset
Thread 1 Thread 2
x := x + 1 ACQ(mutex)

ACQ(mutex) v := v + 1

v := v + 1 REL(mutex)
REL(mutex) x := x + 1

D
(x, RD,{},t1) (v,RD,{mutex},t2)

(x,WR,{},t1) (v,WR,{mutex},t2)

(v,RD,{mutex},t1)
Conflict (x,RD,{},t2)

(v,WR,{mutex},t1) detected! (x,WR,{},t2)

 Precise Lockset
Thread 1 Thread 2 Thread 3
ACQ(m1,m2) ACQ(m2,m3) ACQ(m3,m1)

v := v + 1 v := v + 1 v := v + 1

REL(m2,m1) REL(m3,m2) REL(m1,m3)

D
(v,RD,{m1,m2},t1) (v,RD,{m2,m3},t2) (v,RD,{m1,m3},t3)

(v,WR,{m1,m2},t1) (v,WR,{m2,m3},t2) (v,WR,{m1,m3},t3)

No conflicts detected!
Precise Lockset: Not so Precise
Thread 1 Thread 2
x := x + 1
t2.start()

Precise Lockset
x := x + 1
gives Warning: But
no warning with
Happens-Before
Hybrid Dynamic Data Race Detection
• Relax Happens Before
– No happens before relation between
REL(l,t) and a subsequent ACQ(l,t)
• Maintain Precise Lockset along with
Relaxed Happens-Before
 Hybrid
• O'Callahan and Choi

• Assume a database D storing tuples

(m,t,L,a,e)
where:
– m is a memory location
– t is a thread accessing m
– L is a set of locks held by t while accessing m
– a is the type of access (read or write)
– e is the event associated with the access

• Initially D is empty
 How it works?
• For an event e = MEM(m,a,t) generate the
tuple (m,a,L(t),t,e)
• If there is a tuple (m’,a’,L’,t’,e’) in D such that
– m = m’,
– (a = WR) Ç (a’=WR)
– L(t) Å L’ = empty set
– t  t’
– e and e’ are not related by the happens before
relation, i.e., :(e Á e’) Æ :(e’ Á e)
– Report race over memory location m

• Add (m,a,L(t),t,e) in D
Hybrid Dynamic Data Race Detection
Thread 1 Thread 2
x := x + 1
t2.start() Precise Lockset
detects a data race,
but e2 Á e3.
Therefore, hybrid
technique gives no
warning
x := x + 1

D
(x,RD,{},t1,e1) (x,RD,{},t2,e3)

(x,WR,{},t1,e2) (x,WR,{},t2,e4)
 Distributed Computation
• A set of processes: {p1,p2,…,pn}
• No shared memory
– Communication through messages
• Each process executes a sequence of events
– send(m) : sends a message with content m
– receive(m): receives a message with content m
– Internal event: changes local state of a process
• ith event from process pj is denoted by eij
p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2

m3
p1
e11 e21 e31
Physical Time
 Distributed Computation as a
Partial Order
• Distributed Computation defines a partial
order on the events
– e ! e’
• e and e’ are events from the same process and e executes
before e’
• e is the send of a message and e’ is the receive of the
same message
• there is a e’’ such that e ! e’’ and e’’ ! e’
p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2

m3
p1
e11 e21 e31
Physical Time
 Distributed Computation as a
Partial Order
• Problem: An external process or observer wants to infer the
partial order or the computation for debugging
– No global clock
– At each event a process can send a message to the observer to
inform about the event
– Message delay is unbounded

Observer

p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2

m3
p1
e11 e21 e31
Physical Time
 Can we infer the partial order?
• From the observation:

e12 e13 e11 e21 e23 e43 e33 e31 e32 e22

• Can we associate a suitable value with

every event such that
– V(e) < V(e’) , e ! e’
• We need the notion of clock (logical)
 Lamport’s Logical Time
• All processes use a counter (clock) with initial
value of zero
• The counter is incremented by and assigned
to each event, as its timestamp
• A send (message) event carries its timestamp
• For a receive (message) event the counter is
updated by
– Max(receiver-counter, message-timestamp) + 1
• Send the counter value along with an event to
the observer
 Example

1 2 3 4
p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2
1 5 6
2 m3
1
p1
e11 e21 e31 3
Physical Time
 Example
• Problem with Lamport’s logical clock:
– e ! e’ ) C(e) < C(e’)
– C(e) < C(e’) )
X e ! e’
1 2 3 4
p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2
1 5 6
2 m3
1
p1
e11 e21 e31 3
Physical Time
 Example
• Problem with Lamport’s logical clock:
– e ! e’ ) C(e) < C(e’)
– C(e) < C(e’) )
X e ! e’
1 2 3 4
p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2
1 5 6
2 m3
1
p1
e11 e21 e31 3
Physical Time
 Vector Clock
• Vector Clock: Process ! Nat
• V: P ! N
• Associate a vector clock Vp with every process p
• Update vector clock with every event as follows:
– Internal event at p_i:
• Vp(p) := Vp(p) + 1
– Send Message from p:
• Vp(p) := Vp(p) + 1
• Send Vp with message
– Receive message m at p:
• Vp(i) := max(Vp(i),Vm(i)) for all i 2 P, where Vm is the vector clock
sent with the message m
• Vp(p) := Vp(p) + 1
 Example
V = (a,b,c) means V(p1)=a, V(p2)=b, and V(p3)=c

(0,0,1) (0,1,2) (2,1,3) (2,1,4)

p3
e13 e23 e33 e43
m1 m4
m2 e22 e32
e12
p2
(0,1,0) (2,2,4) (2,3,4)

(1,0,0) m3
(2,0,0)
p1
e11 e21 e31(3,0,0)
Physical Time
 Intuitive Meaning of a Vector Clock
• If Vp = (a,b,c) after some event then
– p is affected by the ath event from p1
– p is affected by the bth event from p2
– p is affected by the cth event from p3
 Comparing Vector Clocks
• V · V’ iff for all p 2 P, V(p) · V’(p)
• V = V’ iff for all p 2 P, V(p) = V’(p)
• V < V’ iff V · V’ and V  V’

• Theorem: Ve < Ve’ iff e ! e’

• Send an event along with its vector
clock to the observer
 Definition of Data Race
• Traditional Definition (Netzer and Miller 1992)

x=1

if (x==1) …

46
 Definition of Data Race
• Traditional Definition (Netzer and Miller 1992)

x=1

receive(m)
X send(m)

if (x==1) …

47
Operational Definition of Data
Race
• We say that the execution of two
statements are in race if they could be
executed by different threads
temporally next to each other and both
access the same memory location and at
least one of the accesses is a write
x=1

receive(m)
X send(m)
if (x==1) … x=1

if (x==1) …

Temporally next
to each48other
 Race Directed Random Testing: RACEFUZZER

• RaceFuzzer: Race directed random

testing
• STEP1: Use an existing technique to
find set of pairs of state transitions
that could potentially race
– We use hybrid dynamic race detection
– Static race detection can also be used
– Transitions are approximated using
program statements

49
 Race Directed Random Testing: RACEFUZZER

• RaceFuzzer: Race directed random testing

• STEP1: Use an existing technique to find set
of pairs of state transitions that could
potentially race
– We use hybrid dynamic race detection
– Static race detection can also be used
– Transitions are approximated using program
statements
• STEP2: Bias a random scheduler so that two
transitions under race can be executed
temporally next to each other
50
 RACEFUZZER using an
example
Thread1 Thread2 Thread3
foo(o1); bar(o1); foo(o2);

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Run ERASER: Statement pair (s5,s6) are in race

51
 RACEFUZZER using an
example
Thread1 Thread2 Thread3
foo(o1); bar(o1); foo(o2);

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Run ERASER: Statement pair (s5,s6) are in race

52
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3
foo(o1); bar(o1); foo(o2);

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Goal: Create a trace exhibiting the race

53
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Example Trace:
foo(o1); bar(o1); foo(o2); s1: g1();
s2: g2();
sync foo(C x) { bar(C y) { s3: g3();
s1: g1(); s6: if (y.f==1) s1: g1();
s2: g2(); s7: ERROR; s2: g2();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o1.f = 1;
Racing Statements
Temporally Adjacent
} s6: if (o1.f==1)
s7: ERROR;
s4: g4();
s5: o2.f = 1;
Goal: Create a trace exhibiting the race
54
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2);

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

55
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

56
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();

sync foo(C x) { bar(C y) {

s1: g1(); s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

57
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) {
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

58
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) {
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

59
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s6: if (o1.f==1)
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

60
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s6: if (o1.f==1)
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

61
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s6: if (o1.f==1)
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Postponed = { }
62
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s6: if (o1.f==1)
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}
Do not postpone
if there is a deadlock

Postponed = { }
s6: if (o1.f==1) 63
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) {
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

64
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

65
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1)
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

66
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR;
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

67
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); }
s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

68
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

69
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1) }

70
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o2.f = 1;
}

Postponed = {s6: if (o1.f==1) }

71
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o2.f = 1;
}

Postponed = {s6: if (o1.f==1) }

72
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o2.f = 1;
}

Race?
Postponed = {s6: if (o1.f==1) }
73
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o2.f = 1;
}

Race?
NO
Postponed = {s6: if (o1.f==1) }
o1.f ≠ o2.f
74
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s5: o2.f = 1;
}

Postponed = {s6: if (o1.f==1), s5: o2.f }= 1;

75
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1;
}

Postponed = {s6: if (o1.f==1), s5: o2.f = 1; }

76
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
}

Postponed = {s6: if (o1.f==1), s5: o2.f = 1; }

77
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} s5: o1.f = 1;

Postponed = {s6: if (o1.f==1), s5: o2.f = 1; }

78
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} s5: o1.f = 1;

Postponed = {s6: if (o1.f==1), s5: o2.f = 1; }

79
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} s5: o1.f = 1;

Race?
YES
Postponed = {s6: if (o1.f==1), s5: o2.f = 1; }
o1.f = o1.f
80
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
}
s6: if (o1.f==1) s5: o1.f = 1;

Postponed = {s5: o2.f = 1; }

81
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} s5: o1.f = 1;
s6: if (o1.f==1)

Postponed = {s5: o2.f = 1; }

82
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} Racing Statements
Temporally Adjacent
s5: o1.f = 1;
s6: if (o1.f==1)

Postponed = {s5: o2.f = 1; }

83
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} Racing Statements
Temporally Adjacent
s5: o1.f = 1;
s6: if (o1.f==1)
s7: ERROR;

Postponed = {s5: o2.f = 1; }

84
 RACEFUZZER using (s5,s6)
an in race
example
Thread1 Thread2 Thread3 Execution:
foo(o1); bar(o1); foo(o2); s1: g1();
s1: g1();
sync foo(C x) { bar(C y) { s2: g2();
s1: g1() s6: if (y.f==1) s2: g2();
s2: g2(); s7: ERROR; s3: g3();
s3: g3(); } s3: g3();
s4: g4(); s4: g4();
s5: x.f = 1; s4: g4();
} Racing Statements
Temporally Adjacent
s5: o1.f = 1;
s6: if (o1.f==1)
s7: ERROR;
s5: o2.f = 1;
Postponed = { }
85
 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5();
7: unlock(L);
8: if (x==0)
9: ERROR;
}

86
 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5(); Race
7: unlock(L);
8: if (x==0)
9: ERROR;
}

Racing Pair: (8,10) 87

 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5();
7: unlock(L);
8: if (x==0)
9: ERROR;
}

Racing Pair: (8,10) Postponed Set

88 = {Thread2}
 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5();
7: unlock(L);
8: if (x==0)
9: ERROR;
}

89
 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5();
7: unlock(L);
8: if (x==0)
9: ERROR;
}

90
 Another Example
Thread1{ Thread2{
1: lock(L); 10: x = 1;
2: f1(); 11: lock(L);
3: f2(); 12: f6();
4: f3(); 13: unlock(L);
5: f4(); }
6: f5();
7: unlock(L);
8: if (x==0)
Hit error with 0.5 probability
9: ERROR;
}

91
 Implementation
• RaceFuzzer: Part of
CalFuzzer tool suite
• Instrument using SOOT
compiler framework lock(L1);
• Instrumentations are used to
“hijack” the scheduler X=1;
– Implement a custom unlock(L1);
scheduler
– Run one thread at a time
– Use semaphores to control lock(L2);
threads Y=2;
• Deadlock detector unlock(L2);
– Because we cannot
instrument native method
calls
92
 Implementation
• RaceFuzzer: Part of
CalFuzzer tool suite
• Instrument using SOOT ins_lock(L1);
compiler framework lock(L1);
• Instrumentations are used to ins_write(&X);
“hijack” the scheduler X=1;
– Implement a custom unlock(L1);
scheduler ins_unlock(L1); Custom
Scheduler
– Run one thread at a time
ins_lock(L1);
– Use semaphores to control lock(L2);
threads Y=2;
• Deadlock detector unlock(L2);
ins_unlock(L1);
– Because we cannot
instrument native method
calls
93
Experimental Results

94
 RACEFUZZER: Useful
•
Features
Classify real races from false alarms
• Inexpensive replay of a concurrent execution
exhibiting a real race
• Separate some harmful races from benign races
• No false warning
• Very efficient
– We instrument at most two memory access statements and
all synchronization statements
• Embarrassingly parallel

95
 RACEFUZZER: Limitations
• Not complete: can miss a real race
– Can only detect races that happen on the given test suite on
some schedule
• May not be able to separate all real races from false
warnings
– Being random in nature
• May not be able to separate harmful races from
benign races
– If a harmful race does not cause in a program crash
• Each test run is sequential

96
 Summary
• Claim: testing (a.k.a verification in industry) is the
most practical way to find software bugs
– We need to make software testing systematic and rigorous
• Random testing works amazingly well in practice
– Randomizing a scheduler is more effective than randomizing
inputs
– We need to make random testing smarter and closer to
verification
• Bias random testing
– Prioritize random testing
• Find interesting preemption points in the programs
• Randomly preempt threads at these interesting points

97
 java.lang.StringBuffer
/**
... used by the compiler to implement the binary
string concatenation operator ...

String buffers are safe for use by multiple threads.

The methods are synchronized so that all the
operations on any particular instance behave as if
they occur in some serial order that is consistent
with the order of the method calls made by each of
the individual threads involved.
*/
/*# atomic */ public class StringBuffer { ... }
 java.lang.StringBuffer
public class StringBuffer {
private int count;
public synchronized int length() { return count; }
public synchronized void getChars(...) { ... }
/*# atomic */
public synchronized void append(StringBuffer sb){
sb.length() acquires lock on sb,
int len = sb.length(); gets length, and releases lock
...
...
other threads can change sb
...
sb.getChars(...,len,...);
... use of stale len may yield
} StringIndexOutOfBoundsException
} inside getChars(...)
 java.lang.StringBuffer
public class StringBuffer {
private int count;
public synchronized int length() { return count; }
public synchronized void getChars(...) { ... }
/*# atomic */
public synchronized void append(StringBuffer sb){
StringBuffer.append is not atomic:
int len = sb.length(); Start:
... at StringBuffer.append(StringBuffer.jav
at Thread1.run(Example.java:17)
...
... Commit: Lock Release
sb.getChars(...,len,...); at StringBuffer.length(StringBuffer.java
... at StringBuffer.append(StringBuffer.jav
at Thread1.run(Example.java:17)
}
} Error: Lock Acquire
at StringBuffer.getChars(StringBuffer.ja
 Related work
• Stoller et al. and Edelstein et al. [ConTest]
– Inserts yield() and sleep() randomly in Java code
• Parallel randomized depth-first search by Dwyer et
al.
– Modifies search strategy in Java Pathfinder by Visser et al.
• Iterative context bounding (Musuvathi and Qadeer)
– Systematic testing with bounded context switches
• Satish Narayanasamy, Zhenghao Wang, Jordan Tigani,
Andrew Edwards and Brad Calder “Automatically
Classifying Benign and Harmful Data Races Using
Replay Analysis”, PLDI 07

101