Monte Carlo method

generate M random permutations of {1,,20}

let H be the number of those in which
exactly one person gets his/her card back

output H/M
Lower bounds
number from {1,2,3,,9}
3 yes/no questions
Can you always figure out the number?
number from {1,2,3,,8}
3 yes/no questions
Lower bounds
Can you always figure out the number?
number from {1,2,3,,8}
3 yes/no questions
Lower bounds
in {1,2,3,4} ?
in {1,2} ? in {5,6} ?
=1 ? =3 ? =5 ?
=7 ?
1 2
3 4 5 6
7 8
Y N
Y
Y
Y
Y
Y
Y
N
N
N
N
N
N
number from {1,2,3,,n}
k yes/no questions
Lower bounds
number from {1,2,3,,n}
k yes/no questions
Lower bounds
k = log
2
n (
number from {1,2,3,,n}
k yes/no questions
Lower bounds
k = log
2
n (
INFORMATION THEORETIC LOWER BOUND:
k> log
2
n
n animals = {dog,cat,fish,eagle,snake, }
yes/no questions
Lower bounds
INFORMATION THEORETIC LOWER BOUND:
k> log
2
n
sorting by comparisons
yes/no questions: is A[i]>A[j] ?
Lower bounds for sorting
A[1..n]
data are not used to control
computation in any other way
1 2 3
1 3 2
2 1 3
3 1
3 1 2
3 2 1
sorting by comparisons
yes/no questions: is A[i]>A[j] ?
Lower bounds for sorting
A[1..n]
k > log
2
n! >
log
2
n + log
2
(n-1) + log
2
1 >
(n/2) log
2
(n/2) = O(n log n)
log a*b = log a + log b
Theorem:
Any comparison based sorting algorithm
requires O(n ln n) comparisons in the
worst-case.
Lower bounds for sorting
Lower bounds for search in sorted array
INPUT: array A[1..n], element x
OUTPUT: a position of x in A if x is in A
otherwise
Lower bounds for search in sorted array
INPUT: array A[1..n], element x
OUTPUT: a position of x in A if x is in A
otherwise
Theorem:
Any comparison based algorithm for
searching an element in a sorted array
requires O(ln n) comparisons in the
worst-case.
Lower bounds for minimum
INPUT: array A[1..n]
OUTPUT: the smallest element of A
Lower bounds for minimum
INPUT: array A[1..n]
OUTPUT: the smallest element of A
INFORMATION THEORETIC LOWER BOUND:
at least O(ln n) comparisons
ADVERSARY LOWER BOUND:
at least O(n) comparisons
Counting sort
array A[1..n] containing numbers from {1,,k}
1
st
pass:
count how many times ie {1,,k} occurs
2
nd
pass:
put the elements in B
Counting sort
array A[1..n] containing numbers from {1,,k}
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=0
for I 1 to k+1 do D[i+1] D[i]+C[i]

for j 1 to n do
D[A[j]]++
B[ D[A[j]] ] A[j]
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 1 5 12 15
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 2 5 12 15
1
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 2 6 12 15
1 2
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 2 6 13 15
1 2 3
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 2 7 13 15
1 2 3 2
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

1 2 3 2 2 4 2 1 1 2 3 3 2 1 2 4
C 4 7 3 2
D 5 12 15 17
1 2 3 2 1 1 1 2 2 2 2 2 3 3 4 4
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

stable sort = after sorting the items with the
same key dont switch order
running time = O(n+k)
Counting sort
for i 1 to k do C[i] 0
for j 1 to n do C[A[j]]++
D[1]=1
for I 1 to k-1 do D[i+1] D[i]+C[i]

for j 1 to n do
B[ D[A[j]] ] A[j]
D[A[j]]++

stable sort = after sorting the items with the
same key dont switch order
running time = O(n+k)
What if e.g., k = n
2
?
Radix sort
array A[1..n] containing numbers from {0,,k
2
- 1}
1) sort using counting sort with
key = x mod k
2) sort using counting sort with
key = x/k

Running time = ?
Radix sort
array A[1..n] containing numbers from {0,,k
2
- 1}
1) sort using counting sort with
key = x mod k
2) sort using counting sort with
key = x/k

Running time = O(n + k)
Radix sort
array A[1..n] containing numbers from {0,,k
2
- 1}
example k=10
28
21
42
43
23
32
70
18
29
20
70
20
21
42
32
43
23
28
18
29
Radix sort
array A[1..n] containing numbers from {0,,k
2
- 1}
example k=10
28
21
42
43
23
32
70
18
29
20
70
20
21
42
32
43
23
28
18
29
Radix sort
array A[1..n] containing numbers from {0,,k
2
- 1}
example k=10
28
21
42
43
23
32
70
18
29
20
70
20
21
42
32
43
23
28
18
29
18
20
21
23
28
29
32
42
43
70
Radix sort
array A[1..n] containing numbers from {0,,k
d
- 1}
1) sort using counting sort with
key = x mod k
2) sort using counting sort with
key = x/k mod k
3) sort using counting sort with
key = x/k
2
mod k

d) sort using counting sort with
key = x/k
d-1
mod k
Radix sort
array A[1..n] containing numbers from {0,,k
d
- 1}
Correctness: after s-th step the numbers
are sorted according to x mod k
s
Proof: By induction. Base case s=1 is trivial.
1) sort using counting sort with
key = x mod k
Radix sort
array A[1..n] containing numbers from {0,,k
d
- 1}
Correctness: after s-th step the numbers
are sorted according to x mod k
s
Proof: Now assume IH and execute s+1
st
step. Let x,y
be such that x mod k
s+1
< y mod k
s+1
. Then either
x/k
s
mod k < y/k
s
mod k or
x/k
s
mod k = y/k
s
mod k and x mod k
s
< y mod k
s
Bucket sort
linear time sorting algorithm on average
Assume some distribution on input.
INPUT: n independently random numbers from the
uniform distribution on the interval [0,1].
Bucket sort
INPUT: n independently random numbers from the
uniform distribution on the interval [0,1].
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
Bucket sort
INPUT: n independently random numbers from the
uniform distribution on the interval [0,1].
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
0.13, 0.23, 0.56, 0.74, 0.18, 0.34, 0.12, 0.82, 0.14, 0.19
Bucket sort
INPUT: n independently random numbers from the
uniform distribution on the interval [0,1].
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
0.13, 0.23, 0.56, 0.74, 0.18, 0.34, 0.12, 0.52, 0.14, 0.19
1 2 5 7 1 3 1 5 1 1
B[1]: 0.13, 0.18, 0.12, 0.14, 0.19
B[2]: 0.23
B[3]: 0.34
B[5]: 0.56, 0.52
B[7]: 0.74
Bucket sort
INPUT: n independently random numbers from the
uniform distribution on the interval [0,1].
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
0.13, 0.23, 0.56, 0.74, 0.18, 0.34, 0.12, 0.52, 0.14, 0.19
1 2 5 7 1 3 1 5 1 1
B[1]: 0.12, 0.13, 0.14, 0.18, 0.19
B[2]: 0.23
B[3]: 0.34
B[5]: 0.52, 0.56
B[7]: 0.74
Bucket sort
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
assume we use insert-sort

worst-case running time?
Bucket sort
for i 1 to n do
insert A[i] into list B[ A[i]*n ]

for i 0 to n-1 do
sort list B[i]

output lists B[0],,B[n-1]
assume we use insert-sort

average-case running time?
X
0
, X
1
, , X
n-1
where X
i
is the number of
items that fall inside the i-th bucket
Bucket sort
X
0
, X
1
, , X
n-1
where X
i
is the number of
items that fall inside the i-th bucket
X
0
2
+ X
1
2
+ + X
n-1
2
What is E[X
0
2
+ X
1
2
+ + X
n-1
2
] ?
E[X
0
2
+ + X
n-1
2
] =
E[X
0
2
] + + E[X
n-1
2
] =
n E[X
0
2
]
linearity of expectation
symmetry of the problem
Bucket sort
E[X
0
2
] What is E[X
0
] ?
0 (1-p)
n

1 n (1-p)
n-1


k binomial(n,k) p
k
(1-p)
n-k


n p
n
value of X
0
p=1/n
Bucket sort
E[X
0
2
] E[X
0
] = 1
0 (1-p)
n

1 n (1-p)
n-1


k binomial(n,k) p
k
(1-p)
n-k


n p
n
p=1/n
E[X
0
] = k * binomial(n,k) p
k
(1-p)
n-k

k=0
n
Bucket sort
E[X
0
2
] E[X
0
] = 1
p=1/n
E[X
0
] = k * binomial(n,k) p
k
(1-p)
n-k

k=1
n
binomial (n,k) = (n/k) * binomial (n-1,k-1)

binomial(n,k) p
k
(1-p)
n-k
= 1
k=0
n
Bucket sort
E[X
0
2
] E[X
0
] = 1
p=1/n
E[X
0
] = n* binomial(n-1,k-1) p
k
(1-p)
n-k

k=1
n


binomial(n,k) p
k
(1-p)
n-k
= 1
k=0
n
= n*p binomial(n-1,k-1) p
k-1
(1-p)
n-k


= n*p
k=1
n
Bucket sort
E[X
0
2
]
0 (1-p)
n

1 n (1-p)
n-1


k binomial(n,k) p
k
(1-p)
n-k


n p
n
p=1/n
E[X
0
2
]= k
2
* binomial(n,k) p
k
(1-p)
n-k

k=0
n
Bucket sort
E[X
0
2
]
p=1/n
E[X
0
2
]= k
2
* binomial(n,k) p
k
(1-p)
n-k


k=0
n
= k * binomial(n,k) p
k
(1-p)
n-k


k=0
n
+
k(k-1) * binomial(n,k) p
k
(1-p)
n-k


k=0
n
Bucket sort
E[X
0
2
]
p=1/n
E[X
0
2
]= k
2
* binomial(n,k) p
k
(1-p)
n-k


k=0
n
= k * binomial(n,k) p
k
(1-p)
n-k


k=0
n
+
k(k-1) * binomial(n,k) p
k
(1-p)
n-k


k=0
n
n*p
Bucket sort
E[X
0
2
]
p=1/n
E[X
0
2
]= k(k-1) * binomial(n,k) p
k
(1-p)
n-k


+n*p



k=2
n
binomial (n,k) = (n/k) * binomial (n-1,k-1) =
(n/k) * ((n-1)/(k-1)) * binomial (n-2,k-2)
E[X
0
2
] = n* (n-1) * p
2
+ n*p