Divide-and-Conquer

Divide-and-Conquer
"Divide et impera" "Veni, vidi, vici" - Julius Caesar 100BC - 44BC

Most widespread application of divide-and-conquer.

s

Break up problem into two pieces of equal size. Solve two sub-problems independently by recursion. Combine two results in overall solution in linear time.

Consequence.
s

Brute force / nave solution: N2. Divide-and-conquer: N log N.

Familiar example.
s

Mergesort.

This course.
s

Counting inversions, closest pair of points, order statistics, fast matrix multiplication, fast integer multiplication, FFT.

Why Does It Matter?

Run time (nanoseconds) 1000 Time to solve a problem of size 10,000 100,000 million Seconds Equivalent 1.3 N3 10 N2 47 N log2N 0.4 msec 6 msec 78 msec 0.94 seconds 11 seconds 1 million 49 million 2.4 trillion 50 trillion 10+ 48 N 0.048 msec 0.48 msec 4.8 msec 48 msec 0.48 seconds 21 million 1.3 billion 76 trillion 1,800 trillion 10 1 10 102 103 104 105 106 107 108 109 1010 ... 1021
3

Orders of Magnitude
1 second 10 seconds 1.7 minutes 17 minutes 2.8 hours 1.1 days 1.6 weeks 3.8 months 3.1 years 3.1 decades 3.1 centuries forever age of universe Powers of 2 210 220 230 thousand million billion
4

Meters Per Second 10-10 10-8 10-6 10-4 10-2 1 102 104 106 108

Imperial Units 1.2 in / decade 1 ft / year 3.4 in / day 1.2 ft / hour 2 ft / minute 2.2 mi / hour 220 mi / hour 370 mi / min 620 mi / sec 62,000 mi / sec

Example Continental drift Hair growing Glacier Gastro-intestinal tract Ant Human walk Propeller airplane Space shuttle Earth in galactic orbit 1/3 speed of light

1.3 seconds 22 minutes 15 days 41 years

10 msec 1 second 1.7 minutes 2.8 hours 1.7 weeks 10,000 77,000 600,000 2.9 million 100

10 million 41 millennia Max size problem solved in one second minute hour day 920 3,600 14,000 41,000 1,000

N multiplied by 10, time multiplied by

A Useful Recurrence Relation

T(N) = worst case running time on input of size N.
s

Analysis of Recurrence: Recursion Tree

Assuming N is a power of 2.
if N = 1 0 T(N ) = 2T (N / 2) + cN otherwise

Note: O(1) is "standard" abuse of notation.

if N n 0 O( 1 ) T( N ) T ( N / 2 ) + T ( N / 2 ) + O ( N ) otherwise 2 14 4 2 3 13 14243 solve right half combine solve left half

T(N/2) Alternate informal form: T(N) T(N/2) + O(N).
s

T(N)

cN 2(cN/2)

T(N/2)

Implicitly assumes N is a power of 2. Implicitly assume T(N) O(1) for sufficiently small N.

T(N/4)

T(N/4)

T(N/4)

T(N/4) log2N

4(cN/4) ...

Solution.
s

T(N / 2k)

2k (cN / 2k) ...

Any function satisfying above recurrence is O(N log2 N). Equivalently, O(logb N) for any b > 1. T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2)

N/2 (2c) cN log2N

Analysis of Recurrence
if N = 1 0 T( N ) T ( N / 2 ) + T ( N / 2 ) + cN otherwise 14 4 2 3 { 14243 solve right half combine solve left half T ( N ) cN log 2 N
Proof by induction on N.
s s s

Counting Inversions
Web site tries to match your preferences with others on Internet. You rank N songs. Web site consults database to find people with similar rankings.

Closeness metric.
s

My rank = { 1, 2, . . . , N }. Your rank = { a1, a2, . . . , aN }. Number of inversions between two preference lists. Songs i and j inverted if i < j, but ai > aj Songs

Base case: N = 1. Define n1 = n / 2 , n2 = n / 2. Induction step: assume true for 1, 2, . . . , N 1.

s s

T ( N ) T ( n1 ) + T ( n2 ) + cn cn1 log 2 n1 + cn2 log 2 n2 + cn cn1 log 2 n2 + cn2 log 2 n2 + cn = cn log 2 n2 + cn cn( log 2 n 1 ) + cn = cn log 2 n

n2

= n / 2 log n 2 2 /2

A Me You 1 1

B 2 3

C 3 4 Inversion

D 4 2

E 5 5

Inversions {3, 2}, {4, 2}

log 2 n2 log 2 n 1

Counting Inversions
Brute-force solution.
s

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.

Check all pairs i and j such that i < j. (N2) comparisons.

Note: there can be a quadratic number of inversions.

s

Asymptotically faster algorithm must compute total number without even looking at each inversion individually. 1 5 4 8 10 2 6 9 12 11 3 7

10

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.
s

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.
s

Divide: separate list into two pieces.

Divide: separate list into two pieces. Conquer: recursively count inversions in each half separately.

1 1

5 5

4 4

8 8

10 10

2 2

6 6

9 9

12 11 12 11

3 3

7 7

O(1)

1 1

5 5

4 4

8 8

10 10

2 2

6 6

9 9

12 11 12 11

3 3

7 7

O(1)

2T(N / 2)

5 blue-blue inversions

8 green-green inversions

11

12

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.
s

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.
s

Divide: separate list into two pieces. Conquer: recursively count inversions in each half. Combine: count inversions where ai and aj are in different halves.

Divide: separate list into two pieces. Conquer: recursively count inversions in each half. Combine: count inversions where ai and aj are in different halves. Return sum of three quantities.

1 1

5 5

4 4

8 8

10 10

2 2

6 6

9 9

12 11 12 11

3 3

7 7

O(1)

1 1

5 5

4 4

8 8

10 10

2 2

6 6

9 9

12 11 12 11

3 3

7 7

O(1)

2T(N / 2)

2T(N / 2)

5 blue-blue inversions

8 green-green inversions O(N2)

5 blue-blue inversions

8 green-green inversions O(N2)

9 blue-green inversions: {5-3, 4-3, 8-6, 8-3, 8-7, 10-6, 10-9, 10-3, 10-7}

9 blue-green inversions: {5-3, 4-3, 8-6, 8-3, 8-7, 10-6, 10-9, 10-3, 10-7} Total = 5 + 8 + 9 = 22.
13

O(1)
14

Counting Inversions: Divide-and-Conquer

Divide-and-conquer.
s

Counting Inversions: Good Combine

Combine: count inversions where ai and aj are in different halves.
s

Divide: separate list into two pieces. Conquer: recursively count inversions in each half. Combine: count inversions where ai and aj are in different halves. Return sum of three quantities.

Key idea: easy if each half is sorted. Sort each half. Count inversions.

1 1

5 5

4 4

8 8

10 10

2 2

6 6

9 9

12 11 12 11

3 3

7 7

O(1)

10

12 11

2T(N / 2) 1 2 4 5 8 10 3 6 7 9 11 12 O(N log N)

5 blue-blue inversions

8 green-green inversions Can we do this step in sub-quadratic time? O(1)

15

9 blue-green inversions: {5-3, 4-3, 8-6, 8-3, 8-7, 10-6, 10-9, 10-3, 10-7} Total = 5 + 8 + 9 = 22.

9 blue-green inversions: 4 + 2 + 2 + 1 + 0 + 0

O(N)

T(N ) = T ( N / 2 ) + T ( N / 2 ) + O (N log N )

T(N ) = O (N log2 N )
16

Counting Inversions: Better Combine

Combine: count inversions where ai and aj are in different halves.
s

Closest Pair
Given N points in the plane, find a pair that is closest together.
s

Assume each half is pre-sorted. Count inversions. Merge two sorted halves into sorted whole.

For concreteness, we assume Euclidean distances. Foundation of then-fledgling field of computational geometry. Graphics, computer vision, geographic information systems, molecular modeling, air traffic control.

Brute force solution. 3 7 10 14 18 19 2 11 16 17 23 25 O(N)

s

Check all pairs of points p and q. (N2) comparisons.

13 blue-green inversions: 6 + 3 + 2 + 2 + 0 + 0 2 3 7 10 11 14 16 17 18 19 23 25

One dimensional version (points on a line). O(N)

s

O(N log N) easy.

Assumption to make presentation cleaner.

T (N ) = T ( N / 2 ) + T ( N / 2 ) + O (N ) T(N ) = O (N log N )
17

No two points have same x coordinate.

18

Closest Pair
Algorithm.
s

Closest Pair
Algorithm.
s

Divide: draw vertical line so that roughly N / 2 points on each side.

Divide: draw vertical line so that roughly N / 2 points on each side. Conquer: find closest pair in each side recursively.

21

12

19

20

Closest Pair
Algorithm.
s

Closest Pair
Algorithm.
s

Divide: draw vertical line so that roughly N / 2 points on each side. Conquer: find closest pair in each side recursively. Combine: find closest pair with one point in each side.

Divide: draw vertical line so that roughly N / 2 points on each side. Conquer: find closest pair in each side recursively. Combine: find closest pair with one point in each side. Return best of 3 solutions.

21

21

12

12

21

22

Closest Pair
Algorithm.
s

Closest Pair
Key step: find closest pair with one point in each side.
s

Divide: draw vertical line so that roughly N / 2 points on each side. Conquer: find closest pair in each side recursively. Combine: find closest pair with one point in each side. Return best of 3 solutions.

Extra information: closest pair entirely in one side had distance .

21 = min(12, 21)

12

23

24

Closest Pair
Key step: find closest pair with one point in each side.
s

Closest Pair
Key step: find closest pair with one point in each side.
s

Extra information: closest pair entirely in one side had distance . Observation: only need to consider points S within of line.

Extra information: closest pair entirely in one side had distance . Observation: only need to consider points S within of line. Sort points in strip S by their y coordinate. suffices to compute distances for pairs within constant number of positions of each other in sorted list!

7 6

21 = min(12, 21)

5 4

21 = min(12, 21)

12

12

2 1

25

26

Closest Pair
S = list of points in the strip sorted by their y coordinate. Crucial fact: if p and q are in S, and if d(p, q) < , then they are within 11 positions of each other in S.
s

Closest Pair
= ClosestPair (p1, p2 , . . . , pN ) O(N log N)
39 31

Compute separation line x = xmed such that half the points have x coordinate less than xmed, and half are greater. 1 = ClosestPair(left half) 2 = ClosestPair(right half) = min (1 , 2 ) Delete all points further than from separation line. Sort remaining points in strip by y coordinate. Scan in y order, and compute distance between each point and next 11 neighbors. If any of these distances is less than , update .

2T(N / 2)

No two points lie in same box. Two points at least 2 rows apart have distance 2 / 2.
2 rows

/2
30 29 28

O(N) O(N log N) O(N) O(N)

/2 /2

27

26 25

T(N ) = T ( N / 2 ) + T ( N / 2 ) + O (N log N )

T(N ) = O (N log2 N )
28

27

Closest Pair
Can we achieve O(N log N)?
s

Integer Arithmetic
Given two N-digit integers a and b, compute a + b.
s

Yes. Dont sort points in strip from scratch each time. Each recursive call should return two lists: all points sorted by y coordinate, and all points sorted by x coordinate. Sorting is accomplished by merging two already sorted lists.

O(N) bit operations.

Multiplication: given two N-digit integers a and b, compute ab.

s

Brute force solution: (N2) bit operations.

1 1 0 1 0 1 0 1 * 0 1 1 1 1 1 0 1 1 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 1 0 1 1 0 1 0 1 0 1 0

T (N ) = T ( N / 2 ) + T ( N / 2 ) + O (N )

T(N ) = O (N log N )

Application.
s

Cryptography.

1 + 1

1 1 0 0

1 1 1 1

1 0 1 0

1 1 1 1

1 0 1 0

0 1 1 0

1 0 0 1 1 1 0

1 1 0 1 0 1 0 1 0 1 1 0 1 0 1 0 1 0 1 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0

29

30

Divide-and-Conquer Multiplication: First Attempt

To multiply two N-digit integers:
s

Karatsuba Multiplication
To multiply two N-digit integers:
s

Multiply four N/2-digit integers. Add two N/2-digit integers, and shift to obtain result.
3 3

Add two N/2 digit integers. Multiply three N/2-digit integers. Subtract two N/2-digit integers, and shift to obtain result.

123,456 987,654 = (10 w + x ) (10 y + z ) = 106 (wy ) + 103 (wz + xy ) + 100 ( xz ) = 106 (121,401) + 103 (80,442 + 450,072 ) + 100 (298,224 )

123,456 987,654 = (103 w + x ) (103 y + z ) = 106 (wy ) + 103 (wz + xy ) + 100 ( xz )

w x y z

= 123 = 456 = 987 = 654

= 121,401,29 9,224

w x y z

= 123 = 456 = 987 = 654

= 106 ( p ) + 103 (r p q ) + 100 (q ) = 106 (121,401) + 103 (950,139 - 121,401 - 298,224 ) + 100 (298,224 ) 121,401,29 9,224

ab = (10N / 2 w + x )(10N / 2 y + z )
N is a power of 2
T(N ) = 4 ) 1T (N43) + 142N43 42 / 2 add,(shift recursive calls T(N ) = (N 2 )

p = wy (wz + xy ) = r p q q = xz r = (w + x )( y + z )

31

32

Karatsuba Multiplication: Analysis

To multiply two N-digit integers:
s

Matrix Multiplication
Given two N x N matrices A and B, compute C = AB.

Add two N/2 digit integers. Multiply three N/2-digit integers. Subtract two N/2-digit integers, and shift to obtain result.
s

cij = aik bkj

k =1

Brute force: (N3) time.

26 62 98 80 224 368 134 386 638 0 2 4 6 8 10 12 14 16 1 7 13 3 9 15 5 11 17
b11 b12 b21 b22 b b32 31 b N 1 bN 2

Karatsuba-Ofman (1962).
s

O(N1.585) bit operations.

p = wy (wz + xy ) = r p q q = xz r = (w + x )( y + z )

ab = (10N / 2 w + x )(10N / 2 y + z )
T(N ) T ( N / 2 T(N ) = O (N

N 14444)4+4T ( 4 2244+ 4(444424 ) 4 4 / ) 4 T 1 + N / 3

recursive calls log 2 3

add, subtract, shift

( ) 142N43

c11 c12 c21 c22 c c32 31 c N 1 cN 2

L c L c L c M M O M L c c
c13 c23 c33
N3

3N NN
1N 2N

a11 a12 a21 a22 a a32 31 a N 1 aN 2

L a L a L a M M O M a L a
a13 a23 a33
N3

3N NN
1N 2N

L b L b L b M M O M b L b
b13 b23 b33
N3

3N NN
1N 2N

Hard to imagine nave algorithm can be improved upon.

33 34

Matrix Multiplication: Warmup

Warmup: divide-and-conquer.
s

Matrix Multiplication: Idea

Idea: multiply 2 x 2 matrices with only 7 scalar multiplications.
r t

Divide: partition A and B into N/2 x N/2 blocks. Conquer: multiply 8 N/2 x N/2 recursively. Combine: add appropriate products using 4 matrix additions.

s a b e g = u c d f h

C11 C12 A11 A12 B11 B12 = C 21 C22 A21 A22 B21 B22

C11 C12 C21 C22

= = = =

(A11 B11 ) + (A12 B21 ) (A11 B12 ) + (A12 B22 ) (A21 B11 ) + (A22 B21 ) (A21 B12 ) + (A22 B22 )
T(N ) = (N 3 )
s

P1 P2 P3 P4 P5 P6 P7

= a (g h ) = (a + b ) h = (c + d ) e = d (f e ) = (a + d ) (e + h ) = ( b d ) (f + h ) = (a c ) (e + g )

r s t u

= P5 + P4 P2 + P6 = P1 + P2 = P3 + P4 = P5 + P1 P3 P7

T(N ) =

8 1T (N 43) + add,44 24)44 s 42 / 2 1 form submatrice 4(N 2 3 recursive calls

7 multiplications. 18 = 10 + 8 additions and subtractions.

Note: did not rely on commutativity of scalar multiplication.

35 36

Matrix Multiplication: Strassen

Generalize to matrices.
s

Beyond Strassen
Can you multiply two 2 x 2 matrices with only 7 scalar multiplications? ! Yes! Strassen (1969). (N log2 7) = O (N 2.81) Can you multiply two 2 x 2 matrix with only 6 scalar multiplications? ! Impossible (Hopcroft and Kerr, 1971). (N log2 6) = O (N 2.59 ) Two 3 x 3 matrices with only 21 scalar multiplications? ! Also impossible. (N log3 21) = O (N 2.77 ) Two 70 x 70 matrices with only 143,640 scalar multiplications? ! Yes! (Pan, 1980). (N log70 143640 ) = O (N 2.80 ) Decimal wars. ! December, 1979: O(N2.521813). ! January, 1980: O(N2.521801). Coppersmith-Winograd (1987): O(N2.376).
37 38

Divide: partition A and B into N/2 x N/2 blocks. Compute: 14 N/2 x N/2 matrices via 10 matrix add/subtract. Conquer: multiply 7 N/2 x N/2 recursively. Combine: 7 products into 4 terms using 8 matrix add/subtract.
C11 C12 A11 A12 B11 B12 = C 21 C22 A21 A22 B21 B22

Analysis.
s

Assume N is a power of 2. T(N) = # arithmetic operations.

T(N ) = 7 1T (N 43) + 14243 42 / 2 add,4(N 2 )4 recursive calls subtract T(N ) = (N log2 7 ) = O (N 2.81)

Strassen in Practice?
Practical considerations.
s

Order Statistics
Given N linearly ordered elements, find ith smallest element.
s

Stop recursion around N = 100. Numerical stability. Harder to parallelize. Caching effects.

Minimum if i = 1. Maximum if i = N. Median: i = (N+1) / 2 if N is odd i = N/2 or i = N/2 + 1 Easy to do with O(N) comparisons if i or N i is a constant. Easy to do in general with O(N log2N) comparisons by sorting.

Can we do in worst-case O(N) comparisons?

s

Yes. (Blum, Floyd, Pratt, Rivest, Tarjan, 1973) Cool and simple idea. Ahead of its time.

Assumption to make presentation cleaner.

s

All items have distinct values.

39

40

Fast Select
Similar to quicksort, but throw away useless "half" at each iteration.
s

Fast Partition
FastPartition().
s

Select

ith

smallest element from a1, a2, . . . , aN. FastSelect (ith, N, a1, a2, . . . , aN)

Divide N elements into N/5 groups of 5 elements each, plus extra.

x FastPartition(N, a1, a2, ..., aN) k rank(x) if (i == k) return x

x = partition element is kth smallest

29 22 28 14 45

10 44 23 09 39

38 52 06 05 50

37 11 26 03 15

02 53 40 54 25

55 12 19 30 16

18 13 01 48 41

24 43 46 47 17

34 20 31 32 33

35 04 49 51 07

36 27 08 21

else if (i < k) b[] all items of a[] less than x return FastSelect(ith, k-1, b1, b2, ..., bk-1) else if (i > k) c[] all items of a[] greater than x return FastSelect((i-k)th, N-k, c1, c2, ..., cN-k)

N = 54
41 42

Fast Partition
FastPartition().
s

Fast Partition
FastPartition().
s

Divide N elements into N/5 groups of 5 elements each, plus extra. Brute force sort each of the 5-element groups.

Divide N elements into N/5 groups of 5 elements each, plus extra. Brute force sort each of the 5-element groups. Find x = "median of medians" using FastSelect() recursively.

14 29 22 28 29 14 45

10 9 10 44 23 39 09 44 39

05 38 06 52 38 06 50 05 52 50

03 37 11 15 26 26 03 37 15

02 25 53 40 53 54 54 25

12 55 16 12 19 30 53 16

01 18 13 18 01 41 48 48 41

17 24 24 43 43 46 46 47 47 17

20 34 31 20 32 31 33 32 34 33

04 35 07 04 35 49 49 51 51 07

36 27 08 21

14 22 28 29 45

9 10 23 39 44

05 06 38 50 52

03 11 15 26 37

02 25 40 53 54

12 16 19 30 53

01 13 18 41 48

17 24 43 46 47

20 31 32 33 34

04 07 35 49 51

36 27 08 21

43

44

Fast Selection and Fast Partition

FastPartition().
s

Fast Selection Analysis

Crux of proof: at least 25% of elements thrown away at each step.
s

Divide N elements into N/5 groups of 5 elements each, plus extra. Brute force sort each of the 5-element groups. Find x = "median of medians" using FastSelect() recursively.

At least 1/2 of 5 element medians x at least N / 5 / 2 = N / 10 medians x

FastSelect().
s

Call FastPartition(). Let x be partition element used, and let k be its rank. Call FastSelect() recursively to find

14 median of medians 22 28 29 45

9 10 23 39 44

05 06 38 50 52

03 11 15 26 37

02 25 40 53 54

12 16 19 30 53

01 13 18 41 48

17 24 43 46 47

20 31 32 33 34

04 07 35 49 51

36 27 08 21

ith smallest

element.

return x if i = k return ith smallest on left side if i < k return (i-k)th smallest on right side if i > k

45

46

Fast Selection Analysis

Crux of proof: at least 25% of elements thrown away at each step.
s

Fast Selection Analysis

Crux of proof: at least 25% of elements thrown away at each step.
s

At least 1/2 of 5 element medians x at least N / 5 / 2 = N / 10 medians x At least 3 N / 10 elements x.

At least 1/2 of 5 element medians x at least N / 5 / 2 = N / 10 medians x At least 3 N / 10 elements x. At least 3 N / 10 elements x.

14 median of medians 22 28 29 45

9 10 23 39 44

05 06 38 50 52

03 11 15 26 37

02 25 40 53 54

12 16 19 30 53

01 13 18 41 48

17 24 43 46 47

20 31 32 33 34

04 07 35 49 51

36 27 08 21 median of medians

14 22 28 29 45

9 10 23 39 44

05 06 38 50 52

03 11 15 26 37

02 25 40 53 54

12 16 19 30 53

01 13 18 41 48

17 24 43 46 47

20 31 32 33 34

04 07 35 49 51

36 27 08 21

47

48

Fast Selection Analysis

Crux of proof: at least 25% of elements thrown away at each step.
s

Fast Selection Analysis

Analysis of recurrence.
if N < 50 c T(N ) T ( N / 5 ) + T (N 3 N / 10 ) + cN otherwise

At least 1/2 of 5 element medians x at least N / 5 / 2 = N / 10 medians x At least 3 N / 10 elements x. At least 3 N / 10 elements x. FastSelect() called recursively with at most N - 3 N / 10 elements in last step
T(N )

Claim: T(N) 20cN.

s

median of medians

T N / 5 T( 3 10 ) O ) 42 3 1(4243) + 14N442N /443 + 1 (N4 4

recursive select

Base case: N < 50. Inductive step: assume true for 1, 2, . . . , N-1.

insertion sort

T (N ) = O (N ).

T (N ) T ( N / 5 ) + T (N 3 N / 10 ) + cN

20c N / 5 + 20c ( N 3 N / 10 ) + cN 20c (N / 5 ) + 20c (N ) 20c (N / 4 ) + cN = 20cN

For n 50, 3 N / 10 N / 4.

49

50

Linear Time Median Finding Postmortem

Practical considerations.
s

Constant (currently) too large to be useful. Practical variant: choose random partition element. O(N) expected running time ala quicksort. Open problem: guaranteed O(N) with better constant.

Quicksort.
s

Worst case O(N log N) if always partition on median. Justifies practical variants: median-of-3, median-of-5.

51