Parallel Program Design

Tasks, Critical Path

Ned Nedialkov

McMaster University
Canada

SE 3F03
January 2015
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Outline

Foster’s Methodology

Tasks

Degrees of concurrency

Critical path

Examples

c 2013–15 Ned Nedialkov 2/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Foster’s Methodology

1. Partitioning. Divide the computation into small tasks that

can be done in parallel
2. Communication. Determine the communications between
the tasks
3. Aggregation (or agglomeration). Combine tasks and
communications into larger tasks
4. Mapping. Assign the combined tasks to processes (or
threads)

From Ian Foster, Designing and Building Parallel Programs,

http://www.mcs.anl.gov/~itf/dbpp/

c 2013–15 Ned Nedialkov 3/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Tasks

I We can decompose a computation into smaller parts or

tasks
I The goal is to determine which can be executed in parallel
I Fine-grained decomposition: many small tasks
I Coarse-grained decomposition: small number of large
tasks
I Task-dependency graph
I directed acyclic graph
I nodes are tasks
I there is an edge between task A and task B if B must be
executed after A

c 2013–15 Ned Nedialkov 4/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Degrees of concurrency

I Maximum degree of concurrency: the maximum number of

tasks that can be executed in parallel
I Average degree of concurrency: the average number of
tasks that can be executed in parallel
I The average degree of concurrency is a more useful
measure

c 2013–15 Ned Nedialkov 5/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Critical path

I Start nodes: nodes with no incoming edges

I Finish nodes: nodes with no outgoing edges
I Critical path: the longest directed path between any pair of
start and finish nodes
I Critical path length: sum of the weights of the nodes on a
critical path
I

total amount of work

ave degree of concurrency =
critical path length

c 2013–15 Ned Nedialkov 6/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Example
Task-dependency graph

I maximum degree of concurrency is 6

I critical path length is 5
I total amount of work is 14 (assuming each task takes one
unit of time)
I average degree of concurrency is 14/5 = 2.8

c 2013–15 Ned Nedialkov 7/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 2 processes and the order in which the

tasks are executed
I First number is the process number
I The speedup is 14/7 = 2

0,1 1,1 0,2 1,2 0,3 1,5

1,3 0,4 1,4

1,7 0,5 1,6

0,6

0,7

c 2013–15 Ned Nedialkov 8/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 3 processes
I The speedup is 14/5 = 2.8

1,3 2,3 0,1 1,1 2,1 2,2

0,2 1,2
1,4

1,5 0,3 2,4

0,4

0,5

c 2013–15 Ned Nedialkov 9/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 4 processes
I The speedup is 14/5 = 2.8
1,2 2,2 2,1 3,1 0,1 1,1

1,3 3,2 0,2

1,4 0,3 2,3

0,4

0,5

I What is the speedup on 8 processes?

c 2013–15 Ned Nedialkov 10/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Example: LU decomposition

I We want to compute the LU decomposition of a matrix A

I Assume A consists of 3 × 3 blocks
I We want
    
A11 A12 A13 L11 0 0 U11 U12 U13
A21 A22 A23  → L21 L22 0   0 U22 U23 
A31 A32 A33 L31 L32 L33 0 0 U33
| {z }| {z }
L U

c 2013–15 Ned Nedialkov 11/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I We need to determine the Lij and Uij

I Consider L times the first column of U
I We have

A11 = L11 U11 compute L11 and U11 (1)

−1
A21 = L21 U11 L21 = A21 U11 (2)
−1
A31 = L31 U11 L31 = A31 U11 (3)

I From multiplying the first row of L times U, we obtain

A12 = L11 U12 U12 = L−1

11 A21 (4)
A13 = L11 U13 U13 = L−1
11 A13 (5)

c 2013–15 Ned Nedialkov 12/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I Then we write

A22 = L21 U12 + L22 U22 A022 = A22 − L21 U12 = L22 U22 (6)
compute L22 and U22 (7)
A23 = L21 U13 + L22 U23 A023 = A23 − L21 U13 = L22 U23 (8)
U23 = L−1 0
22 A23 (9)
A32 = L31 U12 + L32 U22 A032 = A32 − L31 U12 = L32 U22 (10)
L32 = A032 U22
−1
(11)
A33 = L31 U13 + L32 U23 + L33 U33 A033 = A33 − L31 U13 (12)
A0033 = A033 − L32 U23 = L33 U33 (13)
compute L33 and U22 (14)

c 2013–15 Ned Nedialkov 13/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I Using the right column in (1–14), we can decompose the

computation in the following tasks
1. A11 = L11 U11 (compute LU factorization)
−1
2. L21 = A21 U11
−1
3. L31 = A31 U11
4. U12 = L−111 A21
5. U13 = L−111 A13
6. A022 = A22 − L21 U12
7. A032 = A32 − L31 U12
8. A023 = A23 − L21 U13
9. A033 = A33 − L31 U13
10. A022 = L22 U22 (compute LU factorization)
−1
11. L32 = A032 U22
−1 0
12. U23 = L22 A23
13. A0033 = A033 − L32 U23
14. A0033 = L33 U33 (compute LU factorization)

c 2013–15 Ned Nedialkov 14/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

2 5 4 3

8 6 9 7

10

12 11

13

14

Figure: Task-dependency graph

c 2013–15 Ned Nedialkov 15/16
Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

1 P0

P0 P1 P2 P3

2 5 4 3

8 6 9 7
P0 P1 P2 P3

10 P0

P0 P1
12 11

13 P0

P0
14

Figure: Task distribution on 4 processes. Maximum speedup is 2

c 2013–15 Ned Nedialkov 16/16