Divide-and-Conquer

5. Divide-and-Conquer Divide-and-conquer.
! Break up problem into several parts.
! Solve each part recursively.
! Combine solutions to sub-problems into overall solution.

Most common usage.

Divide et impera. !Break up problem of size n into two equal parts of size !n.
Veni, vidi, vici. !Solve two parts recursively.
- Julius Caesar !Combine two solutions into overall solution in linear time.

Consequence.
! Brute force: n2.
! Divide-and-conquer: n log n.

Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley 2

Sorting

5.1 Mergesort Sorting. Given n elements, rearrange in ascending order.

Obvious sorting applications. Non-obvious sorting applications.

! List files in a directory.
! Organize an MP3 library.
! List names in a phone book.
! Display Google PageRank results.
Problems become easier once sorted.
! Find the median.
! Find the closest pair.
! Binary search in a database.
! Identify statistical outliers.
! Find duplicates in a mailing list.
! Data compression.
! Computer graphics.
! Interval scheduling.
! Computational biology.
! Minimum spanning tree.
! Supply chain management.
! Simulate a system of particles.
! Book recommendations on Amazon.
! Load balancing on a parallel computer.
...

Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley 4
 Mergesort Merging

Mergesort. Merging. Combine two pre-sorted lists into a sorted whole.

!Divide array into two halves.
!Recursively sort each half. How to merge efficiently?
!Merge two halves to make sorted whole. !Linear number of comparisons.
!Use temporary array.
Jon von Neumann (1945)

A G L O R H I M S T
A L G O R I T H M S

A G H I
A L G O R I T H M S divide O(1)

A G L O R H I M S T sort 2T(n/2)

A G H I L M O R S T merge O(n) Challenge for the bored. In-place merge. [Kronrud, 1969]

using only a constant amount of extra storage

5 6

A Useful Recurrence Relation Proof by Recursion Tree

Def. T(n) = number of comparisons to mergesort an input of size n. " 0 if n = 1

$
T(n) = # n
2T(n /2) + { otherwise
14243
Mergesort recurrence. $% sorting both halves merging

' 0 if n = 1
) T(n)
T(n) " ( T ( #n /2$ ) + T ( %n /2& ) + { n otherwise ! n
) 14243 14243
merging
* solve left half solve right half

T(n/2) T(n/2) 2(n/2)

!
Solution. T(n) = O(n log2 n). T(n/4) T(n/4) T(n/4) T(n/4) 4(n/4)
log2n
...

Assorted proofs. We describe several ways to prove this recurrence. T(n / 2k) 2k (n / 2k)

Initially we assume n is a power of 2 and replace ! with =. ...

T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) n/2 (2)

n log2n
7 8
 Proof by Telescoping Proof by Induction

Claim. If T(n) satisfies this recurrence, then T(n) = n log2 n. Claim. If T(n) satisfies this recurrence, then T(n) = n log2 n.

assumes n is a power of 2 assumes n is a power of 2

" 0 if n = 1 " 0 if n = 1
$ $
T(n) = # n
2T(n /2) + { otherwise T(n) = # n
2T(n /2) + { otherwise
$% 14243 $% 14243
sorting both halves merging sorting both halves merging

Pf. For n!> 1: T(n) 2T(n /2) Pf. (by induction

! on n)
= +1
n n ! Base case: n = 1.
=
T(n /2)
+1
! Inductive hypothesis: T(n) = n log2 n.
n /2 ! Goal: show that T(2n) = 2n log2 (2n).
T(n / 4)
= +1 +1
n/4
L
T(n /n) T(2n) = 2T(n) + 2n
= + 1 +L+ 1 = 2n log2 n + 2n
n /n 1424 3
log 2 n
= 2n(log2 (2n) "1) + 2n
= log2 n
= 2n log2 (2n)

9 10

!
!

Analysis of Mergesort Recurrence

Claim. If T(n) satisfies the following recurrence, then T(n) ! n "lg n#.
5.3 Counting Inversions
' 0 if n = 1 log2n
)
T(n) " ( T ( #n /2$ ) + T ( %n /2& ) + { n otherwise
) 14243 14243 merging
* solve left half solve right half

Pf. (by induction on n)

!
! Base case: n = 1.
! Define n1 = $n / 2% , n2 = "n / 2#.
! Induction step: assume true for 1, 2, ... , n–1.

T(n) " T(n1 ) + T(n2 ) + n n2 = "n /2#

" n1# lg n1 $ + n2 # lg n2 $ + n lg n #

" n1# lg n2 $ + n2 # lg n2 $ + n
$ " 2" /2 #
" lg n #
= 2 /2
= n # lg n2 $ + n
" n( # lg n$ %1 ) + n % lg n2 $ " lg n# &1

= n # lg n$
11 Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley
!

!
 Counting Inversions Applications

Music site tries to match your song preferences with others.
!You rank n songs.
!Music site consults database to find people with similar tastes.
Similarity metric: number of inversions between two rankings.
! My rank: 1, 2, …, n.
! Your rank: a1, a2, …, an.
! Songs i and j inverted if i < j, but ai > aj.
Applications.
!Voting theory.
!Collaborative filtering.
!Measuring the "sortedness" of an array.
!Sensitivity analysis of Google's ranking function.
!Rank aggregation for meta-searching on the Web.
!Nonparametric statistics (e.g., Kendall's Tau distance).

Songs

A B C D E
Inversions
Me 1 2 3 4 5
3-2, 4-2
You 1 3 4 2 5

Brute force: check all &(n2) pairs i and j.

13 14

Counting Inversions: Divide-and-Conquer Counting Inversions: Divide-and-Conquer

Divide-and-conquer. Divide-and-conquer.
! Divide: separate list into two pieces.

Divide: O(1).
1 5 4 8 10 2 6 9 12 11 3 7 1 5 4 8 10 2 6 9 12 11 3 7

1 5 4 8 10 2 6 9 12 11 3 7

15 16
 Counting Inversions: Divide-and-Conquer Counting Inversions: Divide-and-Conquer

Divide-and-conquer. Divide-and-conquer.
! Divide: separate list into two pieces. ! Divide: separate list into two pieces.
! Conquer: recursively count inversions in each half. ! Conquer: recursively count inversions in each half.
! Combine: count inversions where ai and aj are in different halves,
and return sum of three quantities.

Divide: O(1). Divide: O(1).

1 5 4 8 10 2 6 9 12 11 3 7 1 5 4 8 10 2 6 9 12 11 3 7

1 5 4 8 10 2 6 9 12 11 3 7 Conquer: 2T(n / 2) 1 5 4 8 10 2 6 9 12 11 3 7 Conquer: 2T(n / 2)

5 blue-blue inversions 8 green-green inversions 5 blue-blue inversions 8 green-green inversions

5-4, 5-2, 4-2, 8-2, 10-2 6-3, 9-3, 9-7, 12-3, 12-7, 12-11, 11-3, 11-7
9 blue-green inversions
Combine: ???
5-3, 4-3, 8-6, 8-3, 8-7, 10-6, 10-9, 10-3, 10-7

Total = 5 + 8 + 9 = 22.

17 18

Counting Inversions: Combine Counting Inversions: Implementation

Combine: count blue-green inversions Pre-condition. [Merge-and-Count] A and B are sorted.

! Assume each half is sorted. Post-condition. [Sort-and-Count] L is sorted.
! Count inversions where ai and aj are in different halves.
! Merge two sorted halves into sorted whole.
to maintain sorted invariant Sort-and-Count(L) {
if list L has one element
return 0 and the list L
3 7 10 14 18 19 2 11 16 17 23 25
Divide the list into two halves A and B
6 3 2 2 0 0 (rA, A) ' Sort-and-Count(A)
(rB, B) ' Sort-and-Count(B)
13 blue-green inversions: 6 + 3 + 2 + 2 + 0 + 0 Count: O(n)
(rB, L) ' Merge-and-Count(A, B)

return r = rA + rB + r and the sorted list L

2 3 7 10 11 14 16 17 18 19 23 25 Merge: O(n)
}

T(n) " T ( #n /2$ ) + T ( %n /2& ) + O(n) ' T(n) = O(n log n)

19 20

!
 Closest Pair of Points

5.4 Closest Pair of Points Closest pair. Given n points in the plane, find a pair with smallest
Euclidean distance between them.

Fundamental geometric primitive.

! Graphics, computer vision, geographic information systems,
molecular modeling, air traffic control.
! Special case of nearest neighbor, Euclidean MST, Voronoi.
fast closest pair inspired fast algorithms for these problems

Brute force. Check all pairs of points p and q with &(n2) comparisons.

1-D version. O(n log n) easy if points are on a line.

Assumption. No two points have same x coordinate.

to make presentation cleaner

Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley 22

Closest Pair of Points: First Attempt Closest Pair of Points: First Attempt

Divide. Sub-divide region into 4 quadrants. Divide. Sub-divide region into 4 quadrants.
Obstacle. Impossible to ensure n/4 points in each piece.

L L

23 24
 Closest Pair of Points Closest Pair of Points

Algorithm. Algorithm.
! Divide: draw vertical line L so that roughly !n points on each side. ! Divide: draw vertical line L so that roughly !n points on each side.
! Conquer: find closest pair in each side recursively.

L L

21

12

25 26

Closest Pair of Points Closest Pair of Points

Algorithm. Find closest pair with one point in each side, assuming that distance < (.
! Divide: draw vertical line L so that roughly !n points on each side.
! Conquer: find closest pair in each side recursively.
! Combine: find closest pair with one point in each side. seems like &(n2)

! Return best of 3 solutions.

L L

8
21 21

( = min(12, 21)
12 12

27 28
 Closest Pair of Points Closest Pair of Points

Find closest pair with one point in each side, assuming that distance < (. Find closest pair with one point in each side, assuming that distance < (.
! Observation: only need to consider points within ( of line L. ! Observation: only need to consider points within ( of line L.
! Sort points in 2(-strip by their y coordinate.

L L
7

5
21 4 21

( = min(12, 21) ( = min(12, 21)

3
12 12

2
1

( 29 ( 30

Closest Pair of Points Closest Pair of Points

Find closest pair with one point in each side, assuming that distance < (. Def. Let si be the point in the 2(-strip, with
! Observation: only need to consider points within ( of line L. the ith smallest y-coordinate.
! Sort points in 2(-strip by their y coordinate.
Only check distances of those within 11 positions in sorted list! Claim. If |i – j| ) 12, then the distance between
j
!
39
si and sj is at least (.
31
Pf.
! No two points lie in same !(-by-!( box.
L ! Two points at least 2 rows apart !(
7 2 rows
have distance ) 2(!(). ! 30
6 29 !(
5
4 21 i 27
28 !(
Fact. Still true if we replace 12 with 7.
( = min(12, 21) 26
3
12 25

( 31
( ( 32
 Closest Pair Algorithm Closest Pair of Points: Analysis

Running time.
Closest-Pair(p1, …, pn) {
Compute separation line L such that half the points
O(n log n)
are on one side and half on the other side. T(n) " 2T ( n /2) + O(n log n) # T(n) = O(n log2 n)
(1 = Closest-Pair(left half)
2T(n / 2)
(2 = Closest-Pair(right half)
( = min((1, (2)
!
Q. Can we achieve O(n log n)?
Delete all points further than ( from separation line L O(n)

Sort remaining points by y-coordinate. O(n log n) A. Yes. Don't sort points in strip from scratch each time.
! Each recursive returns two lists: all points sorted by y coordinate,
Scan points in y-order and compare distance between and all points sorted by x coordinate.
each point and next 11 neighbors. If any of these O(n)
distances is less than (, update (. ! Sort by merging two pre-sorted lists.

return (.
} T(n) " 2T ( n /2) + O(n) # T(n) = O(n log n)

!
33 34

Integer Arithmetic

5.5 Integer Multiplication Add. Given two n-digit integers a and b, compute a + b.
!O(n) bit operations.

Multiply. Given two n-digit integers a and b, compute a " b.

!Brute force solution: &(n2) bit operations.
1 1 0 1 0 1 0 1
* 0 1 1 1 1 1 0 1
1 1 0 1 0 1 0 1 0
0 0 0 0 0 0 0 0 0
Multiply
1 1 0 1 0 1 0 1 0
1 1 0 1 0 1 0 1 0
1 1 1 1 1 1 0 1 1 1 0 1 0 1 0 1 0
1 1 0 1 0 1 0 1 1 1 0 1 0 1 0 1 0
+ 0 1 1 1 1 1 0 1 1 1 0 1 0 1 0 1 0
1 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0

Add 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0
Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley 36
 Divide-and-Conquer Multiplication: Warmup Karatsuba Multiplication

To multiply two n-digit integers: To multiply two n-digit integers:

! Multiply four !n-digit integers. ! Add two !n digit integers.
! Add two !n-digit integers, and shift to obtain result. ! Multiply three !n-digit integers.
! Add, subtract, and shift !n-digit integers to obtain result.
n/2
x = 2 " x1 + x0
x = 2 n / 2 " x1 + x0
y = 2 n / 2 " y1 + y0
y = 2 n / 2 " y1 + y0
xy = 2 n / 2 " x1 + x0 2 n / 2 " y1 + y0 = 2 n " x1 y1 + 2 n / 2 " ( x1 y0 + x0 y1 ) + x0 y0
( )( ) xy = 2 n " x1 y1 + 2 n / 2 " ( x1 y0 + x0 y1 ) + x0 y0
= 2 n " x1 y1 + 2 n / 2 " ( (x1 + x0 ) (y1 + y0 ) # x1 y1 # x0 y0 ) + x0 y0
A B A C C
!
Theorem.
!
[Karatsuba-Ofman, 1962] Can multiply two n-digit integers
in O(n1.585) bit operations.
T(n) = 4T (n /2 ) + "(n) # T(n) = "(n 2 )
1424 3 123
recursive calls add, shift
T(n) " T ( #n /2$ ) + T ( %n /2& ) + T ( 1+ %n /2& ) + '(n)
14444444244444443 14243
recursive calls add, subtract, shift

log 2 3
assumes n is a power of 2 ( T(n) = O(n ) = O(n 1.585
)
!

37 38

Karatsuba: Recursion Tree

Matrix Multiplication
1+ log 2 n
" 0 if n = 1 log 2 n
k ( 23) #1
T(n) = # T(n) = " n ( 23) = = 3nlog2 3 # 2
$ 3T(n /2) + n otherwise k=0
3 #1
2

! T(n)! n

T(n/2) T(n/2) T(n/2) 3(n/2)

T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) 9(n/4)

... ...

T(n / 2k) 3k (n / 2k)

... ...

T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) 3 lg n (2)

39 Algorithm Design by Éva Tardos and Jon Kleinberg • Slides by Kevin Wayne • Copyright © 2004 Addison Wesley
 Matrix Multiplication Matrix Multiplication: Warmup

Matrix multiplication. Given two n-by-n matrices A and B, compute C = AB. Divide-and-conquer.
! Divide: partition A and B into !n-by-!n blocks.
! Conquer: multiply 8 !n-by-!n recursively.
"c11 c12 L c1n % "a11 a12 L a1n % "b11 b12 L b1n %
n $ ' $ ' $ ' ! Combine: add appropriate products using 4 matrix additions.
cij = " a ik bkj $c21 c22 L c2n ' a
= $ 21
a 22 L a 2n ' b b
( $ 21 22
L b2n '
k =1 $M M O M' $M M O M ' $M M O M'
$c cn2 L cnn '& $a a n2 L a nn '& $b b L bnn '&
# n1 # n1 # n1 n2

! "C11 C12 % " A11 A12 % " B11 B12 % C11 = ( A11 " B11 ) + ( A12 " B21 )
$ ' = $ '( $ ' C12 ( A11 " B12 ) + ( A12 " B22 )
! #C21 C22 & # A21 A22 & # B21 B22 & =
C21 = ( A21 " B11 ) + ( A22 " B21 )
Brute force. &(n3) arithmetic operations. C22 = ( A21 " B12 ) + ( A22 " B22 )
!
Fundamental question. Can we improve upon brute force?
!
T(n) = 8T ( n /2 ) + "(n2 ) # T(n) = "(n 3 )
1424 3 1442443
recursive calls add, form submatrices

!
41 42

Matrix Multiplication: Key Idea Fast Matrix Multiplication

Key idea. multiply 2-by-2 block matrices with only 7 multiplications. Fast matrix multiplication. (Strassen, 1969)
! Divide: partition A and B into !n-by-!n blocks.
"C11 C12 % " A11 A12 % " B11 B12 %
! Compute: 14 !n-by-!n matrices via 10 matrix additions.
$ ' = $ ' ( $ ' P1 = A11 " ( B12 # B22 ) Conquer: multiply 7 !n-by-!n matrices recursively.
#C21 C22 & # A21 A22 & # B21 B22 & !

P2 = ( A11 + A12 ) " B22 ! Combine: 7 products into 4 terms using 8 matrix additions.
P3 = ( A21 + A22 ) " B11
! C11 = P5 + P4 " P2 + P6 P4 = A22 " ( B21 # B11 ) Analysis.
C12 = P1 + P2 P5 = ( A11 + A22 ) " ( B11 + B22 ) ! Assume n is a power of 2.
C21 = P3 + P4 P6 = ( A12 # A22 ) " ( B21 + B22 ) ! T(n) = # arithmetic operations.
C22 = P5 + P1 " P3 " P7 P7 = ( A11 # A21 ) " ( B11 + B12 )

T(n) = 7T (n /2 ) + "(n 2 ) # T(n) = "(n log2 7 ) = O(n 2.81 )

1424 3 14243
recursive calls add, subtract
! !7 multiplications. !
! 18 = 10 + 8 additions (or subtractions).
!

43 44
 Fast Matrix Multiplication in Practice Fast Matrix Multiplication in Theory

Implementation issues. Q. Multiply two 2-by-2 matrices with only 7 scalar multiplications?
!Sparsity. A. Yes! [Strassen, 1969] "(n log2 7 ) = O(n 2.81 )
!Caching effects.
!Numerical stability. Q. Multiply two 2-by-2 matrices with only 6 scalar multiplications?
!Odd matrix dimensions. A. Impossible. [Hopcroft and Kerr, 1971] !
log 2 6 2.59
"(n ) = O(n )
!Crossover to classical algorithm around n = 128.
Q. Two 3-by-3 matrices with only 21 scalar multiplications?
Common misperception: "Strassen is only a theoretical curiosity." A. Also impossible. ! " (n log3 21 ) = O(n 2.77 )
! Advanced Computation Group at Apple Computer reports 8x
speedup on G4 Velocity Engine when n ~ 2,500. Q. Two 70-by-70 matrices with only 143,640 scalar multiplications?
! Range of instances where it's useful is a subject of controversy. A. Yes! [Pan, 1980] !
log 70 143640 2.80
" (n ) = O(n )

Decimal wars.
Remark. Can "Strassenize" Ax=b, determinant, eigenvalues, and other !December, 1979: O(n2.521813). !
matrix ops. !January, 1980: O(n2.521801).

45 46

Fast Matrix Multiplication in Theory

Best known. O(n2.376) [Coppersmith-Winograd, 1987.]

Conjecture. O(n2+*) for any * > 0.

Caveat. Theoretical improvements to Strassen are progressively less

practical.

47