size n

Problem P
Algorithms
Professor John Reif

fi divide into

subproblems
size n
solve these and "glue" together
solutions
k
T(n) = T(ni ) + g(n)
i=1

ALG 1.3
Deterministic Selection
and Sorting:

,...,n

time to combine solutions

(a) Selection Algorithms and Lower

Bounds
(b) Sorting Algorithms and Lower
Bounds

Examples:
1s t lecture' s mult

fas t fourier trans form

Main Reading Selections:

CLR, Chapters 7, 9, 10
Auxillary Reading Selections:
AHU-Design, Chapters 2 and 3
AHU-Data, Chapter 8
BB, Sections 4.4, 4.6 and 10.1
1

M(n) = 3 M

F(n) = 2 F

+ q(n)

( )

+ q(n)

B(n) = B

merge s orting

S (n) = 2 S

n
2

n
2

( )

binary search

n
2

n
2

+ q(1)

( )

+ q(n)
2

Examples

Selection, and Sorting

Decision Tree Model

Time =

on

( #oncomparisons
longest path )

input a,b,c
Binary tree
a<b?
root

Y
b<c?

Y
a,b,c

facts:

a<c?

b,a,c

a<c?
Y

b<c?

N
Y

a,c,b

with L Leaves
(1) has = L-1 internal nodes
(2) max height
log L

c<a<b

b,c,a

N
c,b,a

Merging

Algorithm Insert
input
( X1 < X 2

2 lists with total of n keys

X1 < X2 < ... < X k
input

< ... < X ) , (Y 1 )

k

and

Case k=n-1

Y1 < Y2 < ... < Y n-k

output
ordered merge of two key lists

Algorithm : Binary Search

by Divide-and-Conquer

[1] Compare Y1 w ith X k

goal
provably asymptotically optimal
algorithm in

[2] i f Y1 > X

Decision Tree Model

use this Model because it

k + 1

(
5

< . . . < Xk

else Y1 X

of lower bounds
allows simple proofs
so easy
time = # comparisons
to bound time costs

insert Y 1 into

k
2

k
2

and insert Y1 into

X1 < . . . < X k
2

Case: Merging equal length lists

(X < X < ... < X )

Input

(Y < Y < ... < Y

1

Total Comparison Cost:

log (k+1)

log (n)

w h e re k =

n- k

n
2

Since a binary tree with n=k+1 leaves

has depth > log(n) , this is optimal!

Algorithm
[1] i 1, j 1
[2] while

i k and j

k do

i f Xi < Yj t h e n ou t p u t ( Xi ) an d s e t i i +1
e l s e ou t p u t ( Yj ) an d s e t jj+1

[3] output

remaining elements

Algorithm clearly uses 2k-1 =

7

n-1 comparisons
8

Sorting by Divide-and-Conquer
Algorithm

Lower bound:

input

consider case X
1 < Y1 < X 2 < Y2 <...< X k < Yk
any merge algorithm must compare

Merge Sort
set S of n keys

claim:
n
[1] partition S into s et X of
2
n
keys
and s et Y of
2

(1) X i with Y i for i=1 ,..., k

(2) Y i with X i+1 for j=1 ,..., k-1
(otherwise we could
fi so requires

flip

Y i < Xi with no change)

keys

[2] Recursively compute

Merge Sort (X) = (X 1 , X 2 , . . . , X

2k-1 = n-1 comparisons!

n
2

Merge Sort (Y) = (Y 1 , Y 2 , . . . , Y

n
2

[3] merge above sequences

using n-1 comparisons
9

[4] output

merged sequence

10

Time Analysis

( ) ( )

T(n) = T

n
2

+ T

T(l) = 0
fi T(n) = n

log n

n
2

-2

log n

Lower Bounds on Sorting

(on decision tree model)

+ n-1

+ 1
d epth log (n !)

= q (n log n)
.
...

n! distinct leaves

11

12

Easy Approximation

(via Integration)

log(n!) = log(n) + log(n-1) + . . . + log(2) + log(1)

n -1

log x dx + ... + log x dx

1

log x dx ( Since log k

1

k -1

Selection Problems

log x dx )
input

n log n - n log e + log e

output
Better bound
using

pn
2p

() (
n
e

e {1,...,n}

x (k) = the k'th best

Sterling Approximation
n

n!

X1 , X 2 ,..., X n
and index k

1+

1
n
12

fi log (n!) n log n - n log e +

1
pn)
log (2p
2

13

14

History:

Selection of the champion

Rev C.L. Dodge (Lewis Carol)

- X (1) is easily determined in

wrote article on lawn tennis

tournament
in James Gazett, 1883

felt prizes unjust

because:
..

(1)
- although winner X
always gets lst prize
- second X (2) may not get 2nd prize

X (1)

X(1)

X(2)

n-1

- X (1) requires

X (2)

n-1 comparisons

proof
everyone except the champion X(1)
must lose at least once!

note: X (2) not

declared 2nd best
if it is
left branch

Carol proposed his own (nonoptimal) tournament....

n-1 comparison

...

X(1)

X (1)

15

16

Selection of the second best

using

n-2 +

logn

X(2)

[2] Let Y be the set of players

comparisons

knocked out by champion

|Y| logn

X(1)

Algorithm
[1] form a balanced binary tree
for tournament to find X
using n-1 comparisons

[3] Play a tournament among the Y's

(1)

[4] output

x
x
x

using

(1)

X (2) = champion of the Y's

logn

-1 more comparisons

Y
(1)

log n

height

(1)

x (1)
17

18

Lower Bounds on finding

requires

n-2 +

X (2)

logn comparisons

Claim m 1 n-1, since at end we

must know X (1) as well as X

(2)

proof
#comparison
where m

Claim m 2 (#who lost to X

)-1
(1)
since everyone (except X
(2) ) who
lost to X (1) must also have
lost one more time.

m 1 + m 2 + ...

= #players who lost i or

more matches

19

20

declare

lemma
(#who lost to X

(1)

Xi > Xj if

) logn

in worst case
(a) X previously undefeated and
i
X j lost at least once
(b) both undefeated but Xi played
more matches

proof
Use oracle who "fixes" results
of games so that champion X
plays logn matches

(c) otherwise, decide consistently

with previous decisions
(1)

fi forces path from X

to have

21

(1)

to root

length logn

22

[1] if n < c o then output X

and halt

Selection by Divide-and-Conquer

(k)

by sorting X

[ 2] divide X into

Algorithm

Select

n sequences
d
of d elements each (with < d leftover),
and sort each sequence

(X)

[3] let M be the

medians
these sequences
[4] m Select

input

set X of n keys and index k

[5] let X
let X

|M|
2

of each of

(M)

= {x e X | x < m}
= {x e X | x > m}

| k then output Select

+
else if n - |X | = k then output

[6] if |X

else
23

output

Select

k-(n - |X |)

(X )

m
+
(X )
24

Columns

smallest

sequences

n
d

n ot X

M in sorted order
d

n ot X

(assuming d is constant)

If say d=5,
largest

d+1
Propos ition |X |, |X | each n - 2

T(n)

c1

for a sufficiently large constant

c1
T

T(n)

20c n = O(n)
1

n
2d 3 n

if n < co

( )
n
d

+ T

( )
3
n
4

+ c1n
25

26

proof
Fix a path p from root to leaf
The comparisons done on p define a
relation R p
+

Let R p = trans itiv e clos ure of Rp

Lower Bounds for Selecting X

input
Theorem

X = {x

,..., x

n },

Lemma
(k)

index k

proof

m either x i Rp x m or x m Rp x i
then for all i

Every leaf of Decision Tree has

depth

If path p determines X m = X (k)

Suppose x i is un related to x m by Rp

Then can replace x i in linear order either

before or after x m to violate x m = x (k)

n-1

Let the "key" comparison

x i is compared with x

for x i be when
j where either

(1) j=m
+

(2) x i Rp x j and x j Rp x m , or
+

(3) x j Rp x i and x m Rp x j
Fact

27

x i has unique "key" comparison determining

+
+
either x i Rp x m or x m Rp x i

fi So there are n-1 "key" comparisons, each distinct!

28

n
(1) D 2

(2) w hile D > 0 do

for i = D +1 to n do

A hard to analyze sort:

SHELLSORT
input keys

X1 ,..., X n

increment
sort

begin
j i- D
while j>0 do
if x j > x

j+ D

then

begin
SWAP (x j , x
j j- D

j+ D)

end
else j 0
end

D D/2

AHU Data Structures & Alg., pp. 290-291

29

30

procedure
increment sort (Y

passes of SHELLSORT:
1

increment s ort

increment s ort

(
(

Xk , X

n
+ k
2

Xk , X n
4

+k

for k=1, . . . ,

, Xn
2

+k

, X3
4

n
2

n+ k

facts

,..., Y

for i = z by 1 until i>n or X i-1 < X i

do for j=1 by -1 until 1 do

if X j-1 > X j then s w ap X j-1 , X j

(1) if X i , X n
p

n
for k=1, . . . ,
4

sorted in pass p
2

+ 1

fi they remain sorted in later passes

(2) dis tance betw een comparis ons diminis h
n n
n
, ....
as , , . . . ,
2 4
p2
(3) The bes t know n time bound is 0

31

(n )
1. 5

32

procedure RADIXSORT

input
[1]

-- Costs O(n) time

X1 , ... , X n e {1 ,..., n}

-- avoids

for j=1 ,..., n

do
initialize B[j] to be the empty list

[2]

for i=1 ,..., n

do
add i to B[X i ]

[3]

let L = (i 1 , i 2 ,..., i n ) be the

concatenation of B[1] ,..., B[n]

[4]

output

X i 1 X i 2 ...

on unit cost RAM

W(nlogn) lower bound on SORT

by avoiding comparisons
instead uses indexing of RAM
-- generalizes (in c passes) to key
domains {1 ,..., n c }

X in

33

34

open problems in sorting

(1)

Complexity of SHELLSORT
-- very good in practice
claims Sedgewick
-- Is it

(2)

q(n 1.5 )?

Complexity of variable length

-- sort on multitape TM or RAM
-- Is it W(n log n)?
35