Analysis of Algorithms

Input Algorithm Output

© 2010 Goodrich, Tamassia Analysis of Algorithms 1

Running Time
 Most algorithms transform best case

input objects into output average case

worst case
objects. 120

 The running time of an 100

algorithm typically grows

Running Time
80
with the input size.
60
 Average case time is often
difficult to determine. 40

 We focus on the worst case 20

running time. 0
1000 2000 3000 4000
 Easier to analyze Input Size
 Crucial to applications such as
games, finance and robotics
© 2010 Goodrich, Tamassia Analysis of Algorithms 2
Experimental Studies
 Write a program 9000

implementing the 8000

algorithm 7000
 Run the program with 6000

Time (ms)
inputs of varying size and 5000
composition 4000
 Use a method like 3000
System.currentTimeMillis() to 2000
get an accurate measure
1000
of the actual running time
0
 Plot the results 0 50 100
Input Size

© 2010 Goodrich, Tamassia Analysis of Algorithms 3

Limitations of Experiments
 It is necessary to implement the
algorithm, which may be difficult
 Results may not be indicative of the
running time on other inputs not included
in the experiment.
 In order to compare two algorithms, the
same hardware and software
environments must be used

© 2010 Goodrich, Tamassia Analysis of Algorithms 4

Theoretical Analysis
 Uses a high-level description of the
algorithm instead of an implementation
 Characterizes running time as a function
of the input size, n.
 Takes into account all possible inputs
 Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment
© 2010 Goodrich, Tamassia Analysis of Algorithms 5
Pseudocode
 High-level description Example: find max
of an algorithm element of an array
 More structured than Algorithm arrayMax(A, n)
English prose Input array A of n integers
 Less detailed than a Output maximum element of A
program
 Preferred notation for currentMax  A[0]
describing algorithms for i  1 to n  1 do
 Hides program design if A[i]  currentMax then
issues currentMax  A[i]
return currentMax

© 2010 Goodrich, Tamassia Analysis of Algorithms 6

Pseudocode Details
 Control flow  Method call
 if … then … [else …] var.method (arg [, arg…])
 while … do …  Return value
 repeat … until … return expression
 for … do …  Expressions
 Indentation replaces braces  Assignment
(like  in Java)
 Method declaration  Equality testing
Algorithm method (arg [, arg…]) (like  in Java)
Input … n2 Superscripts and other
Output … mathematical
formatting allowed

© 2010 Goodrich, Tamassia Analysis of Algorithms 7

The Random Access Machine
(RAM) Model
 A CPU

 An potentially unbounded
bank of memory cells, 2
1
each of which can hold an 0
arbitrary number or
character
Memory cells are numbered and accessing
any cell in memory takes unit time.
© 2010 Goodrich, Tamassia Analysis of Algorithms 8
Seven Important Functions
 Seven functions that
often appear in algorithm 1E+30
1E+28
analysis: 1E+26
Cubic
 Constant  1 1E+24 Quadratic
 Logarithmic  log n 1E+22
Linear
1E+20
 Linear  n 1E+18
N-Log-N  n log n 1E+16
T (n )

1E+14
 Quadratic  n2 1E+12
 Cubic  n3 1E+10
 Exponential  2n 1E+8
1E+6
1E+4
 In a log-log chart, the 1E+2
slope of the line 1E+0
corresponds to the 1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
growth rate
© 2010 Goodrich, Tamassia Analysis of Algorithms 9
Functions Graphed
Using “Normal” Scale
g(n) = n lg n
g(n) = 1 g(n) = 2n

g(n) = n2
g(n) = lg n

g(n) = n
g(n) = n3

© 2010 Stallmann Analysis of Algorithms 10

Primitive Operations
 Basic computations  Examples:
performed by an algorithm  Evaluating an
 Identifiable in pseudocode expression
 Largely independent from the  Assigning a value
to a variable
programming language  Indexing into an
 Exact definition not important array
(we will see why later)  Calling a method
Returning from a
Assumed to take a constant


method
amount of time in the RAM
model

© 2010 Goodrich, Tamassia Analysis of Algorithms 11

Counting Primitive Operations
 By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by an
algorithm, as a function of the input size

Algorithm arrayMax(A, n)
currentMax  A[0] 2
for i  1 to n  1 do 2n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax 1
Total 8n  2

© 2010 Goodrich, Tamassia Analysis of Algorithms 12

Estimating Running Time
 Algorithm arrayMax executes 8n  2 primitive
operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
 Let T(n) be worst-case time of arrayMax. Then
a (8n  2)  T(n)  b(8n  2)
 Hence, the running time T(n) is bounded by two
linear functions

© 2010 Goodrich, Tamassia Analysis of Algorithms 13

Growth Rate of Running Time
 Changing the hardware/ software
environment
 Affects T(n) by a constant factor, but
 Does not alter the growth rate of T(n)
 The linear growth rate of the running
time T(n) is an intrinsic property of
algorithm arrayMax

© 2010 Goodrich, Tamassia Analysis of Algorithms 14

Why Growth Rate Matters
if runtime time for n time for 2 time for 4
is... +1 n n
c (lg n +
c lg n c lg (n + 1) c(lg n + 2)
1)
cn c (n + 1) 2c n 4c n
~ c n lg n 2c n lg n 4c n lg n + runtime
c n lg n quadruples
+ cn + 2cn 4cn
when
~ c n2 + 2c problem
cn 2
4c n2 16c n2
n size doubles
~ c n3 + 3c
cn 3
8c n3 64c n3
n2
c 2n c2 n+1
c2 2n
c2 4n

© 2010 Stallmann Analysis of Algorithms 15

Comparison of Two Algorithms
insertion sort is
n2 / 4
merge sort is
2 n lg n
sort a million items?
insertion sort takes
roughly 70 hours
while
merge sort takes
roughly 40 seconds

This is a slow machine, but if

100 x as fast then it’s 40 minutes
versus less than 0.5 seconds
© 2010 Stallmann Analysis of Algorithms 16
Constant Factors
1E+26
 The growth rate is 1E+24 Quadratic
Quadratic
not affected by 1E+22
1E+20 Linear
 constant factors or 1E+18 Linear
 lower-order terms 1E+16
1E+14
 Examples T (n ) 1E+12
1E+10
 102n  105 is a linear 1E+8
function 1E+6
 105n2  108n is a 1E+4
quadratic function 1E+2
1E+0
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n

© 2010 Goodrich, Tamassia Analysis of Algorithms 17

Big-Oh Notation
10,000
 Given functions f(n) and 3n
g(n), we say that f(n) is 2n+10
1,000
O(g(n)) if there are
n
positive constants
c and n0 such that 100

f(n)  cg(n) for n  n0

10
 Example: 2n  10 is O(n)
 2n  10  cn 1
 (c  2) n 10 1 10 100 1,000
 n 10(c  2) n
 Pick c 3 and n0 10

© 2010 Goodrich, Tamassia Analysis of Algorithms 18

© 2010 Goodrich, Tamassia Analysis of Algorithms 19
Big-Oh Example
1,000,000
n^2
 Example: the function 100n
100,000
n2 is not O(n) 10n
 n2  cn 10,000 n
 nc
 The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10

1
1 10 100 1,000
n

© 2010 Goodrich, Tamassia Analysis of Algorithms 20

More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1
 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21
 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c•log n for n  n0
this is true for c = 8 and n0 = 2
© 2010 Goodrich, Tamassia Analysis of Algorithms 21
Big-Oh and Growth Rate
 The big-Oh notation gives an upper bound on the
growth rate of a function
 The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is no more than the growth rate of g(n)
 We can use the big-Oh notation to rank functions
according to their growth rate

f(n) is O(g(n)) g(n) is O(f(n))

g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
© 2010 Goodrich, Tamassia Analysis of Algorithms 22
 Big-Oh Rules

 If is f(n) a polynomial of degree d, then f(n) is

O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
 Use the smallest possible class of functions
 Say “2n is O(n)” instead of “2n is O(n2)”
 Use the simplest expression of the class
 Say “3n  5 is O(n)” instead of “3n  5 is O(3n)”

© 2010 Goodrich, Tamassia Analysis of Algorithms 23

Asymptotic Algorithm Analysis
 The asymptotic analysis of an algorithm determines
the running time in big-Oh notation
 To perform the asymptotic analysis
 We find the worst-case number of primitive operations
executed as a function of the input size
 We express this function with big-Oh notation
 Example:
 We determine that algorithm arrayMax executes at most
8n  2 primitive operations
 We say that algorithm arrayMax “runs in O(n) time”
 Since constant factors and lower-order terms are
eventually dropped anyhow, we can disregard them
when counting primitive operations

© 2010 Goodrich, Tamassia Analysis of Algorithms 24

© 2010 Goodrich, Tamassia Analysis of Algorithms 25
Computing Prefix Averages
 We further illustrate 35
asymptotic analysis with X
two algorithms for prefix 30 A
averages 25
 The i-th prefix average of
20
an array X is average of the
first (i  1) elements of X: 15
A[i] X[0]  X[1]  …  X[i])/(i+1) 10

 Computing the array A of 5

prefix averages of another 0
array X has applications to 1 2 3 4 5 6 7
financial analysis

© 2010 Goodrich, Tamassia Analysis of Algorithms 26

Prefix Averages (Quadratic)
The following algorithm computes prefix averages in
quadratic time by applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do 1 2 … (n  1)
s  s  X[j] 1 2 … (n  1)
A[i]  s  (i  1) n
return A 1
© 2010 Goodrich, Tamassia Analysis of Algorithms 27
Arithmetic Progression
7
 The running time of
6
prefixAverages1 is
O(1 2 …n) 5
 The sum of the first n 4
integers is n(n 1) 2 3
 There is a simple visual
proof of this fact 2
 Thus, algorithm 1
prefixAverages1 runs in 0
O(n2) time
1 2 3 4 5 6

© 2010 Goodrich, Tamassia Analysis of Algorithms 28

Prefix Averages (Linear)
The following algorithm computes prefix averages in
linear time by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X
#operations
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s  X[i] n
A[i]  s  (i  1) n
return A 1
Algorithm prefixAverages2 runs in O(n) time
© 2010 Goodrich, Tamassia Analysis of Algorithms 29
Math you need to Review
Summations
Logarithms and Exponents
 properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb
 properties of exponentials:
a(b+c) = aba c
Proof techniques abc = (ab)c
Basic probability ab /ac = a(b-c)
b = a logab
bc = a c*logab

© 2010 Goodrich, Tamassia Analysis of Algorithms 30

Relatives of Big-Oh
big-Omega
 f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0  1 such that

f(n)  c•g(n) for n  n0

big-Theta
 f(n) is (g(n)) if there are constants c’ > 0 and c’’

> 0 and an integer constant n0  1 such that

c’•g(n)  f(n)  c’’•g(n) for n  n0

© 2010 Goodrich, Tamassia Analysis of Algorithms 31

Intuition for Asymptotic
Notation
Big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically

less than or equal to g(n)

big-Omega
 f(n) is (g(n)) if f(n) is asymptotically

greater than or equal to g(n)

big-Theta
 f(n) is (g(n)) if f(n) is asymptotically

equal to g(n)

© 2010 Goodrich, Tamassia Analysis of Algorithms 32

Example Uses of the
Relatives of Big-Oh
 5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
 5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
 5n2 is (n2)
f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and
an integer constant n0  1 such that f(n) < c•g(n) for n  n0
Let c = 5 and n0 = 1
© 2010 Goodrich, Tamassia Analysis of Algorithms 33
 More Examples
• For each of the following pairs of functions, either f(n) is
O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)). Determine
which relationship is correct.
– f(n) = log n2; g(n) = log n + 5 f(n) =  (g(n))
– f(n) = n; g(n) = log n2 f(n) = (g(n))
– f(n) = log log n; g(n) = log n f(n) = O(g(n))
– f(n) = n; g(n) = log2 n f(n) = (g(n))
– f(n) = n log n + n; g(n) = log n f(n) = (g(n))
– f(n) = 10; g(n) = log 10 f(n) = (g(n))
– f(n) = 2n; g(n) = 10n2 f(n) = (g(n))
– f(n) = 2n; g(n) = 3n f(n) = O(g(n))
34
 Properties
• Theorem:
f(n) = (g(n))  f = O(g(n)) and f = (g(n))
• Transitivity:
– f(n) = (g(n)) and g(n) = (h(n))  f(n) = (h(n))
– Same for O and 
• Reflexivity:
– f(n) = (f(n))
– Same for O and 
• Symmetry:
– f(n) = (g(n)) if and only if g(n) = (f(n))
• Transpose symmetry:
– f(n) = O(g(n)) if and only if g(n) = (f(n))
35
 Example Problems

1. What does it mean if:

f(n)  O(g(n)) and g(n)  O(f(n)) ?

2. Is 2n+1 = O(2n) ?
Is 22n = O(2n) ?

3. Does f(n) = O(f(n)) ?

4. If f(n) = O(g(n)) and g(n) = O(h(n)),

can we say f(n) = O(h(n)) ?