Barrier Synchronization

Barrier Synchronization
Companion slides for

The Art of Multiprocessor Programming
by Maurice Herlihy & Nir Shavit
Simple Video Game
• Prepare frame for display
– By graphics coprocessor
• “soft real-time” application
– Need at least 35 frames/second
– OK to mess up rarely
Art of Multiprocessor 2
Programming
Simple Video Game
while (true) {
frame.prepare();
frame.display();
}
Programming
Simple Video Game
while (true) {
frame.prepare();
frame.display();
}
• What about overlapping work?

– 1st thread displays frame
– 2nd prepares next frame
Programming
Two-Phase Rendering
while (true) { while (true) {
if (phase) { if (phase) {
frame[0].display(); frame[1].prepare();
} else { } else {
} }
phase = !phase; phase = !phase;
} }
Programming
Two-Phase Rendering
} else { } else {
} }
} }
even phases
Programming
Two-Phase Rendering
} else { } else {
} }
} }
odd phases
Programming
Synchronization Problems
• How do threads stay in phase?
• Too early?
– “we render no frame before its time”
• Too late?
– Recycle memory before frame is displayed
Programming
Ideal Parallel Computation
0 0
0 1 1
Programming
Ideal Parallel Computation
2 2
2 1 1
Programming
Real-Life Parallel Computation
0 0
0 zzz… 1 1
Programming
Real-Life Parallel Computation
2
1zzz… 1
0
Uh, oh
Programming
barrier
0 0
Programming
barrier
1 1
Programming
barrier
Until every
thread has left
here
No thread
Programming
enters here
Why Do We Care?
• Mostly of interest to
– Scientific & numeric computation
• Elsewhere
– Garbage collection
– Less common in systems programming
– Still important topic
Programming
Duality
• Dual to mutual exclusion
– Include others, not exclude them
• Same implementation issues
– Interaction with caches …
• Invalidation?
• Local spinning?
Programming
Example: Parallel Prefix
a b c d before
a+b+c
after a a+b a+b+c
+d
Programming
Parallel Prefix
One thread
Per entry a b c d
Programming
Parallel Prefix: Phase 1
a b c d
a a+b b+c c+d
Programming
Parallel Prefix: Phase 2
a b c d
a+b+c
a a+b a+b+c
+d
Programming
Parallel Prefix
• N threads can compute
– Parallel prefix
– Of N entries
– In log2 N rounds
• What if system is asynchronous?
– Why we need barriers
Programming
Prefix
class Prefix extends Thread {
int[] a;
int i;
Barrier b;
void Prefix(int[] a,
Barrier b, int i) {
a = a;
b = b;
i = i;
}
Programming
Prefix
int[] a;
int i;
Barrier b;
Barrier b, int i) {
a = a;
Array of input
b = b; values
i = i;
}
Programming
Prefix
int[] a;
int i;
Barrier b;
Barrier b, int i) {
a = a;
b = b; Thread index
i = i;
}
Programming
Prefix
int[] a;
int i;
Barrier b;
Barrier b, int i) {
a = a;
b = b; Shared barrier
i = i;
}
Programming
Prefix
int[] a;
int i;
Barrier b; Initialize fields
Barrier b, int i) {
a = a;
b = b;
i = i;
}
Programming
Where Do the Barriers Go?
public void run() {
int d = 1, sum = 0;
while (d < N) {
if (i >= d)
sum = a[i-d];
if (i >= d)
a[i] += sum;
d = d * 2;
}}}
Programming
public void run() {
int d = 1, sum = 0;
while (d < N) {
if (i >= d)
sum = a[i-d];
b.await(); Make sure everyone reads
if (i >= d) before anyone writes
a[i] += sum;
d = d * 2;
}}}
Programming
public void run() {
int d = 1, sum = 0;
while (d < N) {
if (i >= d)
sum = a[i-d];
b.await(); Make sure everyone reads
if (i >= d) before anyone writes
a[i] += sum;
b.await(); Make sure everyone writes
d = d * 2; before anyone reads
}}}
Programming
Barrier Implementations
• Cache coherence
– Spin on locally-cached locations?
– Spin on statically-defined locations?
• Latency
– How many steps?
• Symmetry
– Do all threads do the same thing?
Programming
Barriers
public class Barrier {
AtomicInteger count;
int size;
public Barrier(int n){
count = AtomicInteger(n);
size = n;
}
public void await() {
if (count.getAndDecrement()==1) {
count.set(size);
} else {
while (count.get() != 0);
}}}} Art of Multiprocessor 32
Programming
Barriers
int size;
size = n; Number of threads
} not yet arrived
count.set(size);
} else {
Programming
Barriers
int size; Number of threads
public Barrier(int n){ participating
size = n;
}
count.set(size);
} else {
Programming
Barriers
public class Barrier { Initialization
int size;
size = n;
}
count.set(size);
} else {
Programming
Barriers
AtomicInteger count; Principal method
int size;
size = n;
}
count.set(size);
} else {
Programming
Barriers
AtomicInteger count; If I’m last, reset fields
int size; for next time
size = n;
}
count.set(size);
} else {
Programming
Barriers
int size; Otherwise, wait for
public Barrier(int n){ everyone else
size = n;
}
count.set(size);
} else {
Programming
Barriers
int size;
size = n;
} What’s wrong with this protocol?
count.set(size);
} else {
Programming
Reuse
Barrier b = new Barrier(n);
while ( mumble() ) {
work(); do work
repeat
b.await() synchronize
}
Programming
Barriers
int size;
size = n;
}
count.set(size);
} else {
Programming
Barriers
AtomicInteger count; Waiting for
int size; Phase 1 to finish
size = n;
}
count.set(size);
} else {
Programming
Barriers
public classPhase
Barrier1 {
AtomicInteger count; Waiting for
is so over
Phase 1 to finish
int size;
size = n;
}
count.set(size);
} else {
}}}} Programming
Barriers
public class Barrier
Prepare for {
phase
AtomicInteger 2
count; ZZZZZ….
int size;
size = n;
}
count.set(size);
} else {
}}}} Programming
Uh-Oh
AtomicInteger count;Waiting for
Phase 2 to finish
int size;
count = AtomicInteger(n); Waiting for
size = n; Phase 1 to finish
}
count.set(size);
} else {
Art of Multiprocessor
}}}} Programming
Basic Problem
• One thread “wraps around” to start
phase 2
• While another thread is still waiting for
phase 1
• One solution:
– Always use two barriers
Programming
Sense-Reversing Barriers
int size;
volatile boolean sense = false;
threadSense = new ThreadLocal<boolean>…
public void await {

boolean mySense = threadSense.get();
count.set(size); sense = mySense
} else {
while (sense != mySense) {}
}
threadSense.set(!mySense)}}}
Programming
Completed odd or
public class Barrier { even-numbered
int size;
phase?
public void await {

} else {
}
Programming
public class Barrier { Store sense for
int size; next phase
public void await {

} else {
}
Programming
int size; Get new sense determined
by last phase
public void await {

} else {
}
Programming
int size; If I’m last, reverse sense
for next time
public void await {

} else {
}
Programming
Otherwise, wait for
int size;
sense to flip
public void await {

} else {
}
Programming
AtomicInteger count; Prepare sense for
int size; next phase
public void await {

} else {
}
Programming
Combining Tree Barriers
2-barrier
2-barrier 2-barrier
Programming
Combining Tree Barriers
2-barrier
2-barrier 2-barrier
Programming
Combining Tree Barrier
public class Node{
AtomicInteger count; int size;
Node parent; volatile boolean sense;
public void await() {…

if (parent != null)
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{ Parent barrier in tree

if (parent != null)
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{
AtomicInteger count; int size; Am I last?

if (parent != null)
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{ Proceed to parent barrier

if (parent != null)
parent.await();
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{
Prepare for next phase

if (parent != null)
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{ Notify others at this node

if (parent != null)
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
public class Node{
I’m not last, so wait for
Node parent; volatile booleannotification
sense;

if (parent != null) {
parent.await()
count.set(size);
sense = mySense
} else {
}…}}}
Programming
• No sequential bottleneck
– Parallel getAndDecrement() calls
• Low memory contention
– Same reason
• Cache behavior
– Local spinning on bus-based architecture
– Not so good for NUMA
Programming
Remarks
• Everyone spins on sense field
– Local spinning on bus-based (good)
– Network hot-spot on distributed
architecture (bad)
• Not really scalable
Programming
Tournament Tree Barrier
• If tree nodes have fan-in 2
– Don’t need to call getAndDecrement()
– Winner chosen statically
• At level i
– If i-th bit of id is 0, move up
– Otherwise keep back
Programming
Tournament Tree Barriers
root
winner loser
winner loser winner loser

Programming
All flags blue
Programming
Loser thread sets

winner’s flag
Programming
Loser spins on
own flag
Programming
Winner
spins on
own flag
Programming
Winner sees own

flag, moves up,
spins
Programming
Bingo!
Programming
Sense-reversing:
next time use
blue flags
Programming
Tournament Barrier
class TBarrier {
volatile boolean flag;
TBarrier partner;
TBarrier parent;
boolean top;
…
}
Programming
Tournament Barrier
class TBarrier {
TBarrier partner; Notifications
TBarrier parent; delivered here
boolean top;
…
}
Programming
Tournament Barrier
class TBarrier {
TBarrier partner;
Other thead at
TBarrier parent;
same level
boolean top;
…
}
Programming
Tournament Barrier
class TBarrier {
TBarrier partner;
Parent (winner) or
TBarrier parent;
null (loser)
boolean top;
…
}
Programming
Tournament Barrier
class TBarrier {
TBarrier partner; Am I the root?
TBarrier parent;
boolean top;
…
}
Programming
Tournament Barrier
void await(boolean mySense) {
if (top) {
return;
} else if (parent != null) {
while (flag != mySense) {};
parent.await(mySense);
partner.flag = mySense;
} else {
}}}
Programming
Tournament Barrier
void await(boolean mySense) { Current sense
if (top) {
return; Le root, c’est moi
} else {
}}}
Programming
Tournament Barrier
I am already a
if (top) {
winner
return;
} else {
}}}
Programming
Tournament Barrier
Wait for partner
if (top) {
return;
} else {
}}}
Programming
Tournament Barrier
if (top) {
Synchronize upstairs
return;
} else {
}}}
Programming
Tournament Barrier
if (top) {
Inform partner
return;
} else {
}}}
Programming
Tournament Barrier
if (top) {
Inform partner
return;
} else {
}}} Order is important (why?)
Programming
Tournament Barrier
if (top) {
Natural-born loser
return;
} else {
}}}
Programming
Tournament Barrier
if (top) {
Tell partner I’m here
return;
} else {
}}}
Programming
Tournament Barrier
if (top) {
Wait for notification
return;
from partner
} else {
}}}
Programming
Remarks
• No need for read-modify-write calls
• Each thread spins on fixed location
– Good for bus-based architectures
– Good for NUMA architectures
Programming
Dissemination Barrier
• At round i
– Thread A notifies thread A+2i (mod n)
• Requires log n rounds
Programming
Dissemination Barrier
+1 +2 +4
Programming
Remarks
• Elegant
• Good source of homework problems
• Not cache-friendly
Programming
Ideas So Far
• Sense-reversing
– Reuse without reinitializing
• Combining tree
– Like counters, locks …
• Tournament tree
– Optimized combining tree
• Dissemination barrier
– Intellectually Pleasing (matter of taste)
Programming
Which is best for Multicore?
• On a cache coherent multicore chip:
perhaps none of the above…
• Here is another (arguably) better
algorithm …
Programming
Static Tree Barrier
One node per

thread, statically
assigned
Programming
Static Tree Barrier
Sense-reversing
flag
Programming
Static Tree Barrier
2 0
0 0
Node has count of

missing children
Programming
Static Tree Barrier
2
Spin until zero …
2 0
0 0
Programming
Static Tree Barrier
2
1 0
0 My
0 counter is zero,
decrement parent
Programming
Static Tree Barrier
2
1 0
Spin on done
0 0 flag

Programming
Static Tree Barrier
1
2
2
1 0
0 0

Programming
Static Tree Barrier
1
2
2
1 0
0 0

Programming
Static Tree Barrier
1
2
2
1 0
0 0

Programming
Static Tree Barrier
1
2
0
1 0
0 0

Programming
Static Tree Barrier
1
2
0
1 0
0 0

Programming
Static Tree Barrier
yowzah! 1
0
0
1 0
0 0

Programming
yowzah!
Static Tree Barrier
1
0
0
1 0
0 0

Programming
yowzah!
Static Tree Barrier
1
0
0
1 0
0 0

Programming
Static Tree Barrier
yes!
yes! 1
0
0
1 yes!
0
yes!
0 0 yes!

Programming
Static Tree Barrier
1
2
2
1 0
0 0

Programming
Remarks
• Very little cache traffic
• Minimal space overhead
• On message-passing architecture
– Send notification & sense down tree

Programming
The Nature of Progress*
Companion slides for

The Art of Multiprocessor Programming
by Maurice Herlihy & Nir Shavit
Concurrent Programming
• Many real-word data structures
– blocking (lock-based) implementations &
– non-blocking (no locks) implementations
• For example:
– linked lists, queues, stacks, hash maps,…
113
Concurrent Programming
• Many data structures combine blocking
& non-blocking methods
• Java™ concurrency package
– skiplists, hash tables, exchangers
– on 10 million desktops.
114
Progress Conditions
• Deadlock-free:
– Some thread eventually acquires lock.
• Starvation-free:
– Every thread eventually acquires lock.
• Lock-free:
– Some method call returns.
• Wait-free:
– Every method call returns.
• Obstruction-free:
– Every method call returns if it executes in isolation
We will show an
example shortly 115
List-Based Sets
• Unordered collection of elements
• No duplicates
• Methods
– Add() a new element
– Remove() an element
– Contains() if element is present
116
Coarse Grained Locking
a b d
Lock is starvation-free: every attempt to

acquire the lock eventually succeeds.
117
Fine Grained (Lock Coupling)
a b d
Overlapping locks detect c

overlapping operations
Deadlock-free: some thread eventually acquires
lock.
118
Optimistic Fine Grained
a b c e
add(), remove(), contains() lock

destination nodes in order
Deadlock-free: some thread trying to acquire

the locks eventually succeeds.
119
Obstruction-free contains()
a b c d
Snapshot: if all nodes traversed

twice are the same
Obstruction-free: the method returns if it

executes in isolation for long enough.
120
The Simple Snapshot is
Obstruction-Free
• Put increasing labels on each entry
• Collect twice
• If both agree,
Collect1 Collect2
– We’re done
1
• Otherwise,
1
22 22
1 1
– Try again 7 = 7
13 13
18 18
12 12
121
Obstruction-freedom
• In the simple snapshot alg:
– The update method is wait-free
– But scan is obstruction-free
• Completes if it executes in isolation
• (no concurrent updates).
122
Wait-free contains()
a 0 b 0 dc 1
0 e 0
Use mark bit + list ordering

1. Not marked → in the set
2. Marked or missing → not in the set
123
Lazy List-based Set Alg
a 0 b 0 dc 1
0 e 0
Combine blocking and non-blocking: deadlock-free

add() and remove() and wait-free
contains()
124
Lock-free List-Based Set
Logical Removal =
Set Mark Bit
a 0 b 0 cc 1
0 e 0
mark and reference

d 0
CASed together
CAS will fail
Lock-free add() and remove() and
wait-free contains()
125
So how can this make sense?
• Why have methods with different
progress conditions?
• Let us try to understand this…

Programming© Copyright Herlihy-
Progress Conditions
• Deadlock-free:
– Some thread eventually acquires lock.
• Starvation-free:
– Every thread eventually acquires lock.
• Lock-free:
– Some method call returns.
• Wait-free:
– Every method call returns.
• Obstruction-free:
– Every method call returns if it executes in isolation
127
A “Periodic Table” of
Progress Conditions
Non-Blocking Blocking
All make
Wait- Obstruction- Starvation-
progress
free free free
Some
Lock- Deadlock-
make
free free
progress
128
More Formally
• Standard notion of abstract object
• Progress conditions relate to method
calls of an object
– A thread is active if it takes an infinite
number of concrete (machine level) steps
– And is suspended if not.
129
Maximal vs. Minimal
• Minimal progress
– some call eventually completes
– System matters, not individuals
• Maximal progress
– every call eventually completes.
– Individuals matter
Flags courtesy of
www.theodora.com/flags used
with permission
130
The “Periodic Table” of
Progress Conditions
Maximal
progress
free free free
Minimal
Lock- Deadlock-
progress
free free
131
The Scheduler’s Role
Multiprocessor progress properties:
• Are not about the guarantees a
method's implementation provides.
• Are about scheduling needed to provide
minimal or maximal progress.
132
Fair Scheduling
• A history is fair if each thread takes an
infinite number of steps
• A method implementation is deadlock-

free if it guarantees minimal progress
in every fair history.
133
Starvation Freedom
• A method implementation is starvation-
free if it guarantees maximal progress
in every fair history.
134
Dependent Progress
• Dependent progress conditions
– Do not guarantee minimal progress in
every history
• Independent ones do.
• Blocking progress conditions
– deadlock-freedom, Starvation-freedom
– are dependent.
135
Non-blocking Independent
Conditions
• A lock-free method guarantees
– minimal progress
– in every history.
• A wait-free method guarantees
– maximal progress
– in every history.
136
Progress Conditions
Maximal
progress
free free free
Minimal
Lock- Deadlock-
progress
free free
Independent Dependent
137
Uniformly Isolating Schedules
• A history is uniformly isolating if any
thread eventually runs by itself for “long
enough”
• Modern systems do this with backoff,
yield, etc.
138
A Non-blocking Dependent
Condition
• A method implementation is obstruction-
free if it guarantees
– maximal progress
– in every uniformly isolating history.
139
Progress Conditions
Maximal Uniform iso

Fair scheduler
scheduler
progress free free free
Minimal Lock- Fair schedulerDeadlock-

progress free free
140
Progress Conditions
Maximal Wait- Obstruction- Starvation-

Minimal Lock- Clash- Deadlock-

progress free
?
free free
141
Clash-Freedom: the
“Einsteinium” of Progress
• A method implementation is clash-free if
it guarantees
– minimal progress
– in every uniformly isolating history.
• Thm: clash-freedom strictly weaker than
obstruction-freedom
142
Getting from Minimal to
Maximal
Maximal Wait- Obstruction- Starvation-

?
Minimal Lock- Clash- Deadlock- Helping
progress free
?
free free
143
Maximal Progress Postulate
•Programmers want maximal progress.
•Methods’ progress conditions define
–What we expect from the scheduler
–For example
•Don’t halt in critical section
•Let me run in isolation long enough …
s147
Why Lock-Free is OK
• We all want maximal progress
– Wait-free
• Yet we often write lock-free or deadlock-
free lock-based algorithms
• OK if we expect the scheduler to be
benevolent
– Often true (not always!)

Programming
Shared-Memory Computability
Shared Memory
10011
• What is (and is not) concurrently computable

• Wait-free Atomic Registers
• Lock-free/Wait-free Hierarchy
and Universal Constructions
149
Troubling Intellectual
I think I think,
Question… therefore I think

I am (Ambrose Bierce)
• Why use non-blocking lock-free and

wait-free conditions when most code
uses locks?
150
The Answer
• Not about being non-blocking…

• About being independent!
• Do not rely on the good behavior of the
scheduler.
151
Shared-Memory Computability
Shared Memory
10011
• Independent progress: use Lock-free and Wait-

free Memory Hierarchy and Universal
Constructions
153
Programmers Expect the Best
• Programmers expect maximal progress.

• Progress conditions define scheduler
requirements necessary to achieve it.
154
This work is licensed under a Creative Commons Attribution-
ShareAlike 2.5 License.
• You are free:
– to Share — to copy, distribute and transmit the work
– to Remix — to adapt the work
• Under the following conditions:
– Attribution. You must attribute the work to “The Art of
Multiprocessor Programming” (but not in any way that
suggests that the authors endorse you or your use of the
work).
– Share Alike. If you alter, transform, or build upon this work,
you may distribute the resulting work only under the same,
similar or a compatible license.
• For any reuse or distribution, you must make clear to others the
license terms of this work. The best way to do this is with a link
to
– http://creativecommons.org/licenses/by-sa/3.0/.
• Any of the above conditions can be waived if you get permission
from the copyright holder.
• Nothing in this license impairs or restricts the author's moral
rights.

Programming

Barrier Synchronization

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Barrier Synchronization

Uploaded by

Copyright:

Available Formats

Barrier Synchronization

Companion slides for

• What about overlapping work?

a a+b b+c c+d

public void await {

public void await {

public void await {

public void await {

public void await {

public void await {

public void await {

public void await() {…

public void await() {…

public void await() {…

public void await() {…

public void await() {…

public void await() {…

public void await() {…

winner loser winner loser

All flags blue

Loser thread sets

Winner sees own

One node per

Node has count of

Art of Multiprocessor 100

Art of Multiprocessor 101

Art of Multiprocessor 102

Art of Multiprocessor 103

Art of Multiprocessor 104

Art of Multiprocessor 105

Art of Multiprocessor 106

Art of Multiprocessor 107

Art of Multiprocessor 108

Art of Multiprocessor 109

Art of Multiprocessor 110

Art of Multiprocessor 111

Companion slides for

Lock is starvation-free: every attempt to

Overlapping locks detect c

add(), remove(), contains() lock

Deadlock-free: some thread trying to acquire

Snapshot: if all nodes traversed

Obstruction-free: the method returns if it

Use mark bit + list ordering

Combine blocking and non-blocking: deadlock-free

mark and reference

Art of Multiprocessor 126

• A method implementation is deadlock-

Maximal Uniform iso

Minimal Lock- Fair schedulerDeadlock-

Maximal Wait- Obstruction- Starvation-

Minimal Lock- Clash- Deadlock-

Maximal Wait- Obstruction- Starvation-

Art of Multiprocessor 148

• What is (and is not) concurrently computable

Question… therefore I think

• Why use non-blocking lock-free and

• Not about being non-blocking…

• Independent progress: use Lock-free and Wait-

• Programmers expect maximal progress.

Art of Multiprocessor 156

You might also like