Module 3: Sorting and Randomized Algorithms

CS 240 - Data Structures and Data Management

Arne Storjohann
Based on lecture notes by R. Dorrigiv and D. Roche
David R. Cheriton School of Computer Science, University of Waterloo

Fall 2015

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

1 / 24

Selection vs. Sorting

The selection problem is: Given an array A of n numbers, and 0 k < n,
find the element in position k of the sorted array.
Observation: the kth largest element is the element at position n k.
Best heap-based algorithmhad
 time cost (n + k log n).
For median selection, k = n2 , giving cost (n log n).
This is the same cost as our best sorting algorithms.
Question: Can we do selection in linear time?
The quick-select algorithm answers this question in the affirmative.
Observation: Finding the element at a given position is tough, but
finding the position of a given element is simple.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

2 / 24

Crucial Subroutines

quick-select and the related algorithm quick-sort rely on two subroutines:

choose-pivot(A): Choose an index i such that A[i] will make a
good pivot (hopefully near the middle of the order).
partition(A, p): Using pivot A[p], rearrange A so that
all items the pivot come first,
followed by the pivot,
followed by all items greater than the pivot.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

3 / 24

Selecting a pivot

Ideally, we would select a median as the pivot.

But this is the problem were trying to solve!
First idea: Always select first element in array
choose-pivot1(A)
1.
return 0

We will consider more sophisticated ideas later on.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

4 / 24

Partition Algorithm
partition(A, p)
A: array of size n, p: integer s.t. 0 p < n
1.
swap(A[0], A[p])
2.
i 1, j n 1
3.
loop
4.
while i < n and A[i] A[0] do
5.
i i +1
6.
while j 1 and A[j] > A[0] do
7.
j j 1
8.
if j < i then break
9.
else swap(A[i], A[j])
10. end loop
11. swap(A[0], A[j])
12. return j

Idea: Keep swapping the outer-most wrongly-positioned pairs.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

5 / 24

QuickSelect Algorithm

quick-select1(A, k)
A: array of size n, k: integer s.t. 0 k < n
1.
p choose-pivot1(A)
2.
i partition(A, p)
3.
if i = k then
4.
return A[i]
5.
else if i > k then
6.
return quick-select1(A[0, 1, . . . , i 1], k)
7.
else if i < k then
8.
return quick-select1(A[i + 1, i + 2, . . . , n 1], k i 1)

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

6 / 24

Analysis of quick-select1

Worst-case analysis: Recursive

 call could always have size n 1.
T (n 1) + cn, n 2
Recurrence given by T (n) =
d,
n=1
Solution: T (n) = cn + c(n 1) + c(n 2) + + c 2 + d (n2 )
Best-case analysis: First chosen pivot could be the kth element
No recursive calls; total cost is (n).

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

7 / 24

Average-case analysis of quick-select1

Assume all n! permutations are equally likely.
Average cost is sum of costs for all permutations, divided by n!.
Define T (n, k) as average cost for selecting kth item from size-n array:
!
k1
n1
X
1 X
T (n i 1, k i 1) +
T (i, k)
T (n, k) = cn +
n
i=0

i=k+1

We could analyze this recurrence directly,

or be a little lazier and still get the same asymptotic result.
For simplicity, define T (n) = max T (n, k).
0k<n

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

8 / 24

Average-case analysis of quick-select1

The cost is determined by i, the position of the pivot A[0].
For more than half of the n! permutations, n4 i < 3n
4 .
 
In this case, the recursive call will have length at most 3n
4 , for any k.
The average cost is then given by:

(


cn + 12 T (n) + T b3n/4c , n 2
T (n)
d,
n=1
Rearranging gives:


T (n) 2cn + T b3n/4c 2cn + 2c(3n/4) + 2c(9n/16) + + d
 i
X
3
d + 2cn
O(n)
4
i=0

Since T (n) must be (n) (why?), T (n) (n).

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

9 / 24

Randomized algorithms
A randomized algorithm is one which relies on some random numbers
in addition to the input.
The cost will depend on the input and the random numbers used.
Generating random numbers: Computers cant generate randomness.
Instead, some external source is used (e.g. clock, mouse, gamma rays,. . . )
This is expensive, so we use a pseudo-random number generator (PRNG),
a deterministic program that uses a true-random initial value or seed.
This is much faster and often indistinguishable from truly random.
Goal: To shift the probability distribution from what we cant control
(the input), to what we can control (the random numbers).
There should be no more bad instances, just unlucky numbers.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

10 / 24

Expected running time

Define T (I , R) as the running time of the randomized algorithm for a
particular input I and the sequence of random numbers R.
The expected running time T (exp) (I ) of a randomized algorithm for a
particular input I is the expected value for T (I , R):
X
T (exp) (I ) = E[T (I , R)] =
T (I , R) Pr [R]
R

The worst-case expected running time is then

T (exp) (n) = max T (exp) (I ).
size(I )=n

For many randomized algorithms, worst-, best-, and average-case expected

times are the same (why?).
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

11 / 24

Randomized QuickSelect

random(n) returns an integer uniformly from {0, 1, 2, . . . , n 1}.

First idea: Randomly permute the input first using shuffle:
shuffle(A)
A: array of size n
1.
for i 0 to n 2 do
2.
swap ( A[i], A[i + random(n i)] )

Expected cost becomes the same as the average cost, which is (n).

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

12 / 24

Randomized QuickSelect
Second idea: Change the pivot selection.
choose-pivot2(A)
1.
return random(n)

quick-select2(A, k)
1.
p choose-pivot2(A)
2.
...

With probability at least 21 , the random pivot has position

so the analysis is just like that for the average-case.
The expected cost is again (n).

n
4

i <

3n
4 ,

This is generally the fastest quick-select implementation.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

13 / 24

Worst-case linear time

Blum, Floyd, Pratt, Rivest, and Tarjan invented the medians-of-five
algorithm in 1973 for pivot selection:
choose-pivot3(A)
A: array of size n
1.
m bn/5c 1
2.
for i 0 to m do
3.
j index of median of A[5i, . . . , 5i + 4]
4.
swap(A[i], A[j])
5.
return quick-select3(A[0, . . . , m], bm/2c)

quick-select3(A, k)
1.
p choose-pivot3(A)
2.
...

This mutually recursive algorithm can be shown to be (n) in the worst

case, but its a little beyond the scope of this course.
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

14 / 24

QuickSort
QuickSelect is based on a sorting method developed by Hoare in 1960:
quick-sort1(A)
A: array of size n
1.
if n 1 then return
2.
p choose-pivot1(A)
3.
i partition(A, p)
4.
quick-sort1(A[0, 1, . . . , i 1])
5.
quick-sort1(A[i + 1, . . . , size(A) 1])

Worst case: T (worst) (n) = T (worst) (n 1) + (n)

Same as quick-select1; T (worst) (n) (n2 )




(best) ( n1 ) + (n)
Best case: T (best) (n) = T (best) ( n1
2 )+T
2
Similar to merge-sort; T (best) (n) (n log n)
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

15 / 24

Average-case analysis of quick-sort1

Of all n! permutations, (n 1)! have pivot A[0] at a given position i.

Average cost over all permutations is given by:
T (n) =

n1


1X
T (i) + T (n i 1) + (n),
n

n2

i=0

It is possible to solve this recursion directly.

Instead, notice that the cost at each level of the recursion tree is O(n).
So lets consider the height (or depth) of the recursion, on average.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

16 / 24

Average depth of recursion for quick-sort1

Define H(n) as the average recursion depth for size-n inputs. So



Pn1
1 + n1 i=0
max H(i), H(n i 1) , n 2
H(n) =
0,
n1
Let i be the position of the pivot A[0].
Again, n4 i < 3n
4 for more than half of all permutations.
 
Then larger recursive call has length at most 3n
4 .
This will determine the recursion depth for at least half of all inputs.
 3n 

Therefore H(n) 1 + 12 H(n) + H(
 3n 4 ) for n 2,
which simplifies to H(n) 2 + H( 4 ).
So H(n) O(log n).
Average cost is O(nH(n)) O(n log n).
Since best-case is (n log n), average must be (n log n).
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

17 / 24

More notes on QuickSort

We can randomize by using choose-pivot2, giving

(n log n) expected time for quick-sort2.
We can use choose-pivot3 (along with quick-select3)
to get quick-sort3 with (n log n) worst-case time.
We can use tail recursion to save space on one of the recursive calls.
By making sure the other one is always smaller, the auxiliary space is
(log n) in the worst case, even for quick-sort1.
QuickSort is often the most efficient algorithm in practice.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

18 / 24

Lower bounds for sorting

We have seen many sorting algorithms:
Sort
Selection Sort
Insertion Sort
Merge Sort
Heap Sort
quick-sort1
quick-sort2
quick-sort3

Running time
(n2 )
(n2 )
(n log n)
(n log n)
(n log n)
(n log n)
(n log n)

Analysis
worst-case
worst-case
worst-case
worst-case
average-case
expected
worst-case

Question: Can one do better than (n log n)?

Answer: Yes and no! It depends on what we allow .
No: Comparison-based sorting lower bound is (n log n).
Yes: Non-comparison-based sorting can achieve O(n).
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

19 / 24

The Comparison Model

In the comparison model data can only be accessed in two ways:

comparing two elements
moving elements around (e.g. copying, swapping)
This makes very few assumptions on the kind of things we are sorting.
We count the number of above operations.
All sorting algorithms seen so far are in the comparison model.

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

20 / 24

Lower bound for sorting in the comparison model

Theorem. Any correct comparison-based sorting algorithm requires at
least (n log n) comparison operations.
Proof.
A correct algorithm takes different actions (moves, swaps, etc.) for
each of the n! possible permutations.
The choice of actions is determined only by comparisons.
The algorithm can be viewed as a decision tree.
Each internal node is a comparison, each leaf is a set of actions.
Each permutation must correspond to a leaf.
The worst-case number of comparisons is the longest path to a leaf.
Since the tree has at least n! leaves, the height is at least lg n!.
Therefore worst-case number of comparisons is (n log n).

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

21 / 24

Counting Sort
Requirement: Each A[i] satisfies 0 A[i] < k; k is given.
counting-sort(A, k)
A: array of size n, k: positive integer
1.
C array of size k, filled with zeros
2.
for i 0 to n 1 do
3.
increment C [A[i]]
4.
for i 1 to k 1 do
5.
C [i] C [i] + C [i 1]
6.
B copy (A)
7.
for i n 1 down to 0 do
8.
decrement C [B[i]]
9.
A[C [B[i]]] B[i]

Time cost: (n + k), which is (n) if k O(n).

Auxiliary space: (n + k), can be made (k)
counting-sort is stable: equal items stay in original order.
Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

22 / 24

Radix Sort
Requirement: Each A[i] is a string of d digits xd1 xd2 x0 ,
and each xi satisfies 0 xi < k.
Example: integers between 0 and k d 1
radix-sort(A, d, k)
A: array of size n, d: positive integer, k: positive integer
1.
for i 0 to d 1 do
2.
Call counting-sort(A, k) with each xi as the key

Time cost: (d(n + k))

Auxiliary space: (n + k)

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

23 / 24

Summary of sorting

Randomized algorithms can eliminate bad cases

Best-case, worst-case, average-case, expected-case can all differ
Sorting is an important and very well-studied algorithm
Can be done in (n log n) time; faster is not possible for general input
HeapSort is the only fast algorithm we have seen with
O(1) auxiliary space.
QuickSort is often the fastest in practice
MergeSort is also (n log n), selection & insertion sorts are (n2 ).
CountingSort, RadixSort can achieve o(n log n) if the input is special

Storjohann (SCS, UW)

CS240 - Module 3

Fall 2015

24 / 24