Analysis of Algorithms

9-Feb-23
 Time and space

• To analyze an algorithm means:

– developing a formula for predicting how fast an
algorithm is, based on the size of the input (time
complexity), and/or
– developing a formula for predicting how much
memory an algorithm requires, based on the
size of the input (space complexity)
• Usually time is our biggest concern
– Most algorithms require a fixed amount of
space
2
 Average, best, and worst cases
• Usually we would like to find the average time to perform an
algorithm
• However,
– Sometimes the “average” isn’t well defined
• Example: Sorting an “average” array
– Time typically depends on how out of order the array is
– How out of order is the “average” unsorted array?
– Sometimes finding the average is too difficult
• Often we have to be satisfied with finding the worst (longest)
time required
• The best (fastest) case is seldom of interest

3
 Constant time
• Constant time means there is some constant
k such that this operation always takes k
nanoseconds
• If a statement involves a choice (if or
switch) among operations, each of which
takes constant time, we consider the
statement to take constant time
– This is consistent with worst-case analysis

4
 Linear time
for (i = 0, j = 1; i < n; i++)
{
j = j * i;
}
– This loop takes time k*n + c, for some constants k and c
k : How long it takes to go through the loop once
(the time for j = j * i)
n : The number of times through the loop
(we can use this as the “size” of the problem)
c : The time it takes to initialize the loop
– The total time k*n + c is linear in n

5
 Constant time is (usually) better than linear time

• Suppose we have two algorithms to solve a task:

– Algorithm A takes 5000 time units
– Algorithm B takes 100*n time units
• Which is better?
– Clearly, algorithm B is better if our problem size is small,
that is, if n < 50
– Algorithm A is better for larger problems, with n > 50
– So B is better on small problems that are quick anyway
– But A is better for large problems, where it matters more
• We usually care most about very large problems
– But not always!
6
 The array subset problem
• Suppose you have two sets, represented as unsorted arrays:
int[] sub = { 7, 1, 3, 2, 5 };
int[] super = { 8, 4, 7, 1, 2, 3, 9 };
and you want to test whether every element of the first set
(sub) also occurs in the second set (super):
• (The answer in this case should be false, because sub
contains the integer 5, and super doesn’t)
• We are going to write method subset and compute its time
complexity (how fast it is)
• Let’s start with a helper function, member, to test whether
one number is in an array

7
 member
static boolean member(int x, int[] a)
{
int n = a.length;
for (int i = 0; i < n; i++) {
if (x == a[i]) return true;
}
return false;
}
• If x is not in a, the loop executes n times, where
n = a.length
– This is the worst case
• If x is in a, the loop executes n/2 times on
average
• Either way, linear time is required: k*n+c
8
 subset
static boolean subset(int[] sub, int[] super)
{
int m = sub.length;
for (int i = 0; i < m; i++)
if (!member(sub[i], super) return false;
return true;
}
• The loop (and the call to member) will execute:
– m = sub.length times, if sub is a subset of super
• This is the worst case, and therefore the one we are most
interested in
– Fewer than sub.length times (but we don’t know how few)
• We would need to figure this out in order to compute average time complexity
• The worst case is a linear number of times through the loop
• But the loop body doesn’t take constant time, since it calls member,
which takes linear time
9
 Analysis of array subset algorithm
• We’ve seen that the loop in subset executes m =
sub.length times (in the worst case)
• Also, the loop in subset calls member, which
executes in time linear in n = super.length
• Hence, the execution time of the array subset
method is m*n, along with assorted constants
– We go through the loop in subset m times, calling
member each time
– We go through the loop in member n times
– If m and n are similar, this is roughly quadratic

10
 What about the constants?
• Forget the constants!
• An added constant, f(n)+c, becomes less and
less important as n gets larger
• A constant multiplier, k*f(n), does not get less
important, but...
– Improving k gives a linear speedup (cutting k in half
cuts the time required in half)
– Improving k is usually accomplished by careful code
optimization, not by better algorithms

11
 Simplifying the formulae
• Throwing out the constants is one of two
things we do in analysis of algorithms
– By throwing out constants, we simplify 12n2 +
35 to just n2
• Our timing formula is a polynomial, and may
have terms of various orders (constant, linear,
quadratic, cubic, etc.)
– We usually discard all but the highest-order term
• We simplify n2 + 3n + 5 to just n2

12
 Big O notation
• When we have a polynomial that describes
the time requirements of an algorithm, we
simplify it by:
– Throwing out all but the highest-order term
– Throwing out all the constants
• If an algorithm takes 12n3+4n2+8n+35
time, we simplify this formula to just n3
• We say the algorithm requires O(n3) time
– We call this Big O notation

13
 Big O for subset algorithm
• Recall that, if n is the size of the set, and m is
the size of the (possible) subset:
– We go through the loop in subset m times,
calling member each time
– We go through the loop in member n times
• Hence, the actual running time should be
k*(m*n) + c, for some constants k and c
• We say that subset takes O(m*n) time

14
 Can we justify Big O notation?
• Big O notation is a huge simplification; can we
justify it?
– It only makes sense for large problem sizes
– For sufficiently large problem sizes, the
highest-order term swamps all the rest!
• Consider R = x2 + 3x + 5 as x varies:
x= 0 x2 = 0 3x = 0 5= 5 R=5
x= 10 x2 = 100 3x = 30 5= 5 R = 135
x= 100 x2 = 10000 3x = 300 5= 5 R = 10,305
x= 1000 x2 = 1000000 3x = 3000 5= 5 R = 1,003,005
x= 10,000 x2 = 108 3x = 3*104 5= 5 R = 100,030,005
x= 100,000 x2 = 1010 3x = 3*105 5= 5 R = 10,000,300,005

15
 y = x2 + 3x + 5, for x=1..10

16
 y = x2 + 3x + 5, for x=1..20

17
 Common time complexities
BETTER
• O(1) constant time
• O(log n) log time
• O(n) linear time
• O(n log n) log linear time
• O(n2) quadratic time
• O(n3) cubic time
WORSE • O(2n) exponential time

18
 Examples
1)
int x = 0;
for (int i = 0; i < 100; i++)
x += i;

2)
int x = 0;
for (int i = 0; i < n2; i++)
x += i;
 Examples
3)
int x = 0;
for (int i = 1; i < n; i *= 2)
x += i;

4)
int x = 0;
for (int i = 1; i < n; i++)
for (int j = 1; j < i; j++)
x += i + j;
Example 4:
int x = 0;
Count =count+1;
for (int i = 1; i <=n; i++)
Count =count+1; //for the outer for loop
for (int j = 1; j < =i; j++)
Count =count+1; //for the inner for loop
x += i + j;
Count =count+1; //assignment
Count =count+1; //for the last time of inner
for loop
Count =count+1; //for the last time of outer
for loop 1 + n (x) + 1 where x represents inner loop
X is equal to
When i =1 , step count =2 + 1
When i=2, step count = 4 + 1
When i=3, step count = 6 + 1
When i=I step count = 2i +1
So
1 + n [2(i) +1]
1+ 2in +n
 Examples
5)
int x = 0;
for (int i = 1; i < n; i++)
for (int j = i; j < 100; j++)
x += i + j;

6)
int x = 0;
for (int i = 1; i < n; i++)
for (int j = n; j > i; j /= 3)
x += i + j;
Example 6
int x = 0;
Count =count+1;
for (int i = 1; i < n; i++)
Count =count+1; //for the outer for loop
for (int j = n; j > i; j /= 3)
Count =count+1; //for the inner for loop
x += i + j;
Count =count+1; // for the assignment
Count =count+1; //for the last time of inner
for loop
Count =count+1; //for the last time of outer
for loop 1 + n (x) + 1 where x represents inner loop
Suppose max vakue of n=30;
X is equal to
When j=30 and i=1, step count =3(2) + 1
When j=30, and i=2 step count = 3(2) + 1
When j=30 and i=3, step count = 6 + 1
When i=I step count = 2i +1
So
1 + n [2(log3n) +1]
1+ 2nlogn +n
Example 6
int x = 0;
Count =count+1;
for (int i = 1; i < n; i++)
Count =count+1; //for the outer
for loop
for (int j = n; j > i; j /=
3)
Count =count+1; //for the inner
for loop
x += i + j;
Count =count+1; // for the
assignment
Count =count+1; //for the last
time of inner for loop
Count =count+1; //for the last
time of outer for loop
1 + n (x) + 1 where x represents inner loop
Suppose max vakue of n=30;
X is equal to
When j=30 and i=1, step count =3(2) + 1
When j=30, and i=2 step count = 3(2) + 1
When j=30 and i=3, step count = 2(2) + 1
..
.
When j=30 and i= 10 step count =1(2) + 1
..
..
When j=30 and i= 20 step count =1(2) + 1
When i=I step count = 2i +1
So
1 + n [2log10n +1]
1+ 2n. log10n +n
 Examples

7)

int x = 0;
for (int i = 1; i < n * n; i++)
for (int j = 1; j < i; j++)
x += i + j;
 Analyzing Recursive Algorithms
• Often a recurrence equation is used as the starting point
to analyze a recursive algorithm
– In the recurrence equation, T(n) denotes the running time of the
recursive algorithm for an input of size n
• We will try to convert the recurrence equation into a
closed form equation to have a better understanding of
the time complexity
– Closed Form: No reference to T(n) on the right side of the
equation
– Conversions to the closed form solution can be very challenging
• Example: Factorial

int factorial (int n)

/* Pre: n is an integer no less than 0
Post: The factorial of n (n!) is returned
Uses: The function factorial recursively */
{
if (n == 0)
return 1; 1
else
return n * factorial (n - 1);
} T ( n  1)  3
}

factorial(0) is 1 (comparison)
factorial(n) is (1 comparison, 1 multiplication, 1 subtraction and
time for factorial(n-1)
Factorial
Fibonacii Series (Recursive)
int fib(int n) {
if (n == 1)
return 0; //First digit in the series is 0
else if (n == 2)
return 1; //Second digit in the series is 1
else
return (fib(n - 1) + fib(n - 2));
 Fibonacii Series (Recursive)
• Let’s use T(n) to denote the time complexity of F(n).
• The number of additions required to compute F(n-1)
and F(n-2) will then be T(n-1) and T(n-2), respectively.
We have one more addition to sum our results.
• for n > 1:
• T(n) = T(n-1) + T(n-2) + 1
• When n = 0 and n = 1, no additions occur. This implies
that:
• T(0) = T(1) = 0
• Now that we have our equation, we need to solve
for T(n).
 Simplifying T(n)

• Substituting the value of T(n-2) = T(n-1) into

our relation T(n), we get:
• T(n) = T(n-1) + T(n-1) + 1 = 2*T(n-1) + 1
• By doing this, we have reduced T(n) into a
much simpler recurrence. As a result, we can
now solve for T(n) using backward
substitution.
 Solving T(n) Using Backward
Substitution
So, T(n) = 2*T(n-1) + 1
We first substitute T(n-1) into the right hand side of our
recurrence. Since T(n-1) = 2T(n-2) + 1 we get
T(n) = 2*[ 2*T(n-2)+1] +1 = 4T(n-2) + 3
Next, we can substitute in T(n-2) = 2*T(n-3) +1
T(n) = 2*[2*[2*T(n-3)+1]+1]+1 = 8*T(n-3) + 7
And once more for T(n-3 ) = 2*(T(n-4) +1
T(n) = 2*[2*[2*[2*T(n-4)+1]+1]+1]+1 = 16*T(n-4) + 15
We can see a pattern starting to emerge here.
So, lets attempt to form a general solution for T(n).
It appears to stand that
T(n) = 2K * T(n-k) + (2k -1)
 Fibonacii Series (Recursive)
T(n) = 2K * T(n-k) + (2k -1)

• Finally, we can find k and, thereby, solve T(n),

by substituting in T(0) = 1.
• For T(0), we can see that n – k = 0.
Rearranging, we get k = n. Now, substituting in
our values for T(0) and k, we get:
Fibonacii Series (Iterative)
 Fibonacii Series (Iterative)
• Firstly, our assignments of F[0] and F[1] cost
O(1) each.
• Secondly, our loop performs one assignment
per iteration and executes (n-1)-2 times,
costing a total of O(n-3) = O(n).
• Therefore, our iterative algorithm has a time
complexity of O(n) + O(1) + O(1) = O(n).
• This is a marked improvement from our
recursive algorithm!
 Asymptotic Notations
• Describing complexity
 O-Notation (Upper Bound)

f(n) = O(g(n)) if there are positive constants n0

and c such that to the right of n0, the value of
f(n) always lies on or below cg(n) or
f(n)<=c*g(n) for all n> n0.

n
0
O-Notation Contd..

f(n) <= c * g(n) for all n> n0

f(n) = 3n +2
3n+2 <= 4n for all n>=2
 O-Notation Examples
• n2+10n = O(n3)
• 5n3+6 =O(n3)
• 3logn+8 = O(logn)
• n3+10n = O(n3)
 O-Notation Examples
• n2+10n = O(n2)
• 5n3+6 =O(n3)
• 3logn+8 = O(logn)
• n3+10n = O(n3)
 O-Notation
• The statement f(n) = O (g(n)) states that g( n) is an
upper bound on the value of f(n), for all n > n0.
• It does not say anything about how good this
bound is.
• For the statement f(n) = O (g(n)) to be
informative, g(n) should be as small a function of
n
• Hence 3n+3 = O(n) and
• we never say that 3n+3 = O(n2)
 General Rules for analysis
• In expressing running times, each
elementary step such as an assignment, an
addition or an initialization is counted as
one unit of time
• Leading constants and lower order terms are
ignored
 Properties of O notation.

• Constant factors can be omitted

– 2n3+6 ε O(n3) , 2 and 6 are omitted
• Growth rate of a sum is given by the rate of its fastest
growing term.
– 5n3+3n+8 ε O(n3)
• if f(n)>g(n), g(n)>b(n) then f(n)>b(n)
– O(n2)>O(n), O(n)>O(logn)so O(n2)>O(logn)
• Higher powers of n grows faster than lower powers
– O(n3)> O(n2)
 Example Problem 1

Read(A)
X:= A*A
-------
Write(X)
any segment for which each statement is
executed only once will have a total number of
statements executed that is independent of n.
Time complexity will be O(1)
 Example Problem2

for i :=1 to n do
for j:=1 to n do
{
---
}
• complexity n*n =O(n2)
 Example Problem3
for i:=1 to n do
for j:=1 to i do
{
---
}
complexity is 1+2+3+4+ -- --+n=n*(n+1)/2=O(n2)
 Example Problem2
Algorithm power(b,n)
{p:=1; q:=b;
While (n>0) do
{If n is odd, then p:=p*q;
n:=n /2;
q:=q*q;}
Return (p); }

• Complexity = O(logn)
 n

3 5 10 16 20 … 22 28 36 60 arr: sorted
 Exercise:
Write down the recursive procedure for tower of
Hanoii and Calculate its time complexity