2010 - 02 -17

CS 515: Parallel Algorithms

Chandrima Sarkar Atanu Roy

Agenda
Architecture Parallel Programming Languages Precedence Graph Elementary Parallel Algorithms Sorting Matrix Multiplication Download :http://www.cs.montana.edu/~atanu.roy/Classes/ CS515.html

Architecture
Flynns Classification S = single , M = multiple , I = instruction (stream), D = data (stream)

SISD

SIMD

Architecture
Flynns Classification S = single , M = multiple , I = instruction (stream), D = data (stream)

MISD

MIMD

Static Inter-connection Network

Linear Array Ring

Ring arranged to use short wires

Fully Connected Topology

Chordal ring

Multidimensional Meshes and Torus

Tree

Tree Cont.

FAT TREE

STAR

Hypercube
100 000 110

010
101 111

001 0-D 1-D 2-D

011 3-D 4-D

5-D

Parallel Programming Languages

Control Mechanism Communication Mechanism

Shared Memory Control driven

Message-passing

Fortran 90/HPF , C++ , HEP CSP , Ada , OCCAM (Von PL/I , Ada , Concurrent Neumann Language Pascal Modula-2 , MultiLisp Extension ) (MIMD), Lisp Connection Machine (SIMD) VAL , ID LAU , SISAL ( dataflow languages )
Concurrent Prolog ( Shapiro ) Actors FP

Data driven
Pattern driven Demand driven ( reduction language )

Dijkstras High Level language construct

Degree of Parallelism is static Algol-68,CSP Precendence Graph

A
parbegin C begin B parbegin D E parend G end parend H

Elementary Parallel Algorithms

Finding sum using a 2D mesh architecture

Finding sum of 16 values in a Shuffle Exchange SIMD Model

Parallel summation in a Hypercube SIMD Model

Broadcast in a Hypercube
Algorithm 1

Algorithm 2

Odd Even Transposition Sort

(1) p = n

14 5 15 8 4 11 13 12 odd-even 14 5 15 8 4 11 13 12 even-odd 14 5 15 4 8 11 13 12 odd-even 5 14 4 15 8 11 12 13 even-odd 5 4 14 8 15 11 12 13 odd-even 4 5 8 14 11 15 12 13 even-odd 4 5 8 11 14 12 15 13 odd-even 4 5 8 11 12 14 13 15 even-odd 4 5 8 11 12 13 14 15

Odd Even Transposition Sort (contd)

(2) p << n
S= {12, 7, 2, 4, 1, 11, 9, 5, 6, 3, 10, 8}, p = 4
P1 {12, 7, 2} {2, 7, 12} {1, 2, 4} {1, 2, 4} {1, 2, 3} {1, 2, 3} P2 {4, 1, 11} {1, 4, 11} {7, 11, 12} {3, 5, 6} {4, 5, 6} {4, 5, 6} P3 {9, 5, 6} {5, 6, 9} {3, 5, 6} {7, 11, 12} {7, 8, 9} {7, 8, 9} P4 {3, 10, 8} {3, 8, 10} {8, 9, 10} {8, 9, 10} {10, 11, 12} {10, 11, 12}

Pseudocode
Proc MERGE-SPLIT(S)
for i:= 1 to p do in parallel
QUICKSORT(Si)

end for for (i := 1 to ceil(p/2)) for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for end for

2 D mesh with Snake Order

Thompson and Kung (1977)

Input : {23, 6, 1, 5, 11, 13, 55, 19, -3, 12, -5, -7, 9, 55, 28, -2}

Snake Order (contd.)

Bitonic Merge Sort

Bitonic Sequence :- 1, 3, 7, 8 Comparator Note :- Batchers Bitonic Merge Sort compares elements whose indices differ by a single bit. 6, 5, 4, 2

Bitonic Merge Sort

Shuffle-Exchange Network
Bitonic Mergesort on Shuffle-Exchange Network A list of n = 2k unsorted elements can be sorted in time (lg2 n) with a network 2k-1[k (k-1) + 1] comparators using the shuffle-exchange network.

Sorting Network

Odd Even Merging Network

Systolic Matrix Multiplication

1. 2. 3. 4. Multiply ai,k by ak,j Add the result to ri,j Send ai,k to cell ci+1,j Send bk,j to cell ci,j+1

Home Work
Show how the following 16 values would be sorted by Batchers Bitonic sort. 16, 7, 4, 12, 2, 10, 13, 9, 1, 8, 11, 3, 15, 6, 5, 14