CSE 202 : DATA STRUCTURES

Lecture # 03:

Complexity Analysis

Dr. Ahmed Abba Haruna

Assistant Professor,
University of Hafr Al-Batin (UoHB)
College of Computer Science and Engineering (CCSE)
Hafr Al-Batin 31991, Saudi Arabia.
aaharuna@uhb.edu.sa 1
© Slides adapted from the textbook, ICS 202 slides from KFUPM and/or CS 211 slides from Taibah University (Ms. Rana & Mr. Modhi) and/or many other online resources.
 Outline

• Motivation

• Average, Best, and Worst Case Complexity Analysis

• Asymptotic Analysis

2
Motivations for Complexity Analysis
• There are often many different algorithms which can be used to
solve the same problem.
– For example, assume that we want to search for a key in a sorted array.
• Thus, it makes sense to develop techniques that allow us to:
o compare different algorithms with respect to their “efficiency”
o choose the most efficient algorithm for the problem

• The efficiency of any algorithmic solution to a problem can be

measured according to the:
o Time efficiency: the time it takes to execute.
o Space efficiency: the space (primary or secondary memory) it uses.

• We will focus on an algorithm’s efficiency with respect to time.

3
 How do we Measure Efficiency?
• Running time in [micro/milli] seconds
– Advantages ?
– Disadvantages ?

• Instead,
o we count the number of basic operations the algorithm performs.
o we calculate how this number depends on the size of the input.

• A basic operation is an operation which takes a

constant amount of time to execute.
– E.g., additions, subtractions, multiplications, comparisons, …etc.
• Hence, the efficiency of an algorithm is determined in
terms of the number of basic operations it performs.
This number is most useful when expressed as a
function of the input size n.

4
 Example of Basic Operations:
• Arithmetic operations: *, /, %, +, -
• Assignment statements of simple data types.
• Reading of primitive types
• Writing of a primitive types
• Simple conditional tests: if (x < 12) ...
• Method call (Note: the execution time of the method itself may
depend on the value of parameter and it may not be constant)
• A method's return statement
• Memory access

• Note:
– We consider an operation such as ++ , += , and *= as consisting
of two basic operations.
– To simplify complexity analysis we will not consider memory
access (fetch or store) operations. 5
 Simple Complexity Analysis
public class TestAbstractClass
{
public static void main(String[] args)
{
Employee[] list = new Employee[3];
list[0] = new Executive(“Mike", 50000);
list[1] = new HourlyEmployee(“Tyson", 120);
list[2] = new MonthlyEmployee(“Abraham", 9000);
((Executive)list[0]).awardBonus(11000);
for(int i = 0; i < list.length; i++)
if(list[i] instanceof HourlyEmployee)
((HourlyEmployee)list[i]).addHours(60);
for(int i = 0; i < list.length; i++) {
list[i].print();
System.out.println("Paid: " + list[i].pay());
System.out.println("*************************");
} } }

How many number of basic operations are there ?

• Assignment statements:
• Additions:
• Print statements:
• Method calls: 6
 Simple Complexity Analysis
• Counting the number of basic operations
is cumbersome
– It is not important to count the number of all
basic operations
• Instead, we count / find the “number of
times” of the statement that gets
executed the most
– E.g. find the number of element comparisons

7
Simple Complexity Analysis: Simple Loops
• How to find the cost of an unnested loop?
– Find (a) the cost of the body of the loop
– In the case below, consider the number of multiplications
– Find (b) the number of iterations in loop
– Multiply the two numbers (i.e., (a) * (b)) to get
the total
– E.g. double x, y;
x = 2.5 ; y = 3.0;
for(int i = 0; i < n; i++){
a[i] = x * y;
x = 2.5 * x;
y = y + a[i];
}
8
Simple Complexity Analysis: Complex Loops
• Represent the cost of for loop in summation form.
– The main idea is to make sure that we find an
iterator that increase/decrease its values by 1.
– For example, consider finding the number of
times statements 1, 2 and 3 get executed below:
n −1
for (int i = 1; i < n; i++)
statement1;
1 = n − 1
i =1
for (int i = 1; i <= n; i++) { n n n n
for (int j = 1; j <= n; j++)
statement2;
 
1 = n =
i =1 j =1
n  1
i =1
= n 2

i =1
}

for (int i = 1; i <= n; i++) { n i n n ( n + 1)

for (int j = 1; j <= i; j++)
statement3;
1 = i =
i =1 j =1 i =1 2
} 9
 Useful Summation Formulas
n
 n 

i =m
c = c   1 = c . ( n − m + 1)
 i =m 

n n ( n + 1)( 2n + 1) Sum of square of

i
i =1
2
=
6
natural numbers
from 1 to n

 n ( n + 1) 
2
n

 i =
3 Sum of cube of

n ( n + 1)
natural numbers
n
 2 
i =
i =1 from 1 to n

i =1 2
n +1
n
a −1

i =0
a i
=
a −1
,a  1
Sum of natural
numbers from 1 to n 10
Simple Complexity Analysis: Complex Loops
• Represent the cost of the for loop in
summation form.
– The problem in the example below is that
the value of i does not increase by 1
for (int i = k; i <= n; i = i + m)
statement4;

• i: k , k + m , k + 2m , …, k + rm
– Here, we can assume without loss of generality that
k + rm = n, i.e. r = (n – k)/m
– i.e., an iterator s from 0, 1, 2, …,r can be used
n −k
r m
n −k n −k
1 = 
s =0 s =0
1=
m
− 0 +1 =
m
+1
11
Simple Complexity Analysis : Loops (with <=)
• In the following for-loop:
for (int i = k; i <= n; i = i + m){
statement1;
statement2;
}

• The number of iterations is: (n – k) / m +1

• The initialization statement, i = k, is executed one time.

• The condition, i <= n, is executed (n – k) / m +1 + 1 times.

• The update statement, i = i + m, is executed (n – k) / m +1 times.

• Each of statement1 and statement2 is executed (n – k) / m +1

times.

12
 Simple Complexity Analysis: Loops (with <)
• In the following for-loop:

for (int i = k; i < n; i = i + m){

statement1;
statement2;
}

The number of iterations is: (n – k ) / m

• The initialization statement, i = k, is executed one time.

• The condition, i < n, is executed (n – k ) / m + 1 times.

• The update statement, i = i + m, is executed (n – k ) / m times.

• Each of statement1 and statement2 is executed (n – k ) / m times.

13
Useful Logarithmic Formulas

14
Best, Average, Worst case complexities
• What is the best case complexity analysis?
• The smallest number of basic operations carried out by the
algorithm for a given input.
• What is the worst case complexity analysis?
• The largest number of basic operations carried out by the algorithm
for a given input.
• What is the average case complexity analysis?
• The number of basic operations carried out by the algorithm on average.


for each input i
(Probability of input i * Cost of input i)

• We are usually interested in the worst case complexity

• Easier to compute
• Represents an upper bound on the actual running time for
all inputs
• Crucial to real-time systems (e.g. air-traffic control)
15
How to analyze time complexity? (1/4)
• Running time depends on:
1. Single/Multiple processor machine
2. Read/Write speed to RAM
3. 32 bit/ 64 bit architecture of machine (configuration)
4. Input

• We focus on rate of growth of time taken w.r.t the input

• Consider the following hypothetical machine model.

1) Single processor
2) Sequential execution of instruction
3) 32 bit architecture
4) Takes 1 unit of time to execute simple arithmetic & logic operations.
5) Takes 1 unit of time for assignment and return to a method.
16
How to analyze time complexity? (2/4)

Example 1: consider the following pseudocode.

sum(a , b)
{ 1unit
return a + b
} 1unit

Time taken for executing above code,

Tsum = 1 + 1 = 2 units (always constant for all sizes of inputs)

The above example is a Constant Time algorithm.

17
How to analyze time complexity? (3/4)
Example 2:
COST REPEATITION

sumOfArray(A[ ] , size)
{
total=0 1 1
for i= 0 to n-1 and i=i+1 2 n+1
{
total = total + A[i] 2 n
}
return total 1 1
}

TsumOfArray = 1+ 2(n+1) + 2n +1
= 4n + 4
(OR) T(n) = Cn + C’ Where C and C’ are constants

18
How to analyze time complexity? (4/4)
So,
Tsum = k (a constant)

TsumOfArray = Cn + C’ Where C and C’ are constants

TsumOfMatrix = a n2 + b n + c Where a, b and c are constants

Plotting a graph

19
Asymptotic Notation- Big oh ‘O’
• Since counting the exact number of operations is
cumbersome / impossible, we can always focus
our attention on asymptotic analysis, where
constants and lower-order terms are ignored.
– E.g. n3, 1000n3, and 10n3+10000n2+5n-1 are all “the same”
– The reason we can do this is that we are always interested in
comparing different algorithms for arbitrarily large number of
inputs.
Asymptote in math is a line that a curve
approaches, as it heads towards infinity:
• So,
Tsum = k (a constant) → O(1)
TsumOfArray = Cn + C’ → O(n)
TsumOfMatrix = a n2 + b n + c → O(n2)
20
Asymptotic Growth of different functions

21
Running Times for Different Sizes
of Inputs of Different Functions

22
centuries
Different Asymptotic Notations (1/3)
‘O’ Big-oh notation:
O(g(n))={ f(n): if there exist constants C and n0 such that
f(n) ≤ Cg(n) , for n ≥ n0 }
For example:
f(n) = 5 n2 + 2 n + 1 O(n2)
g(n) = n2
C = 8 (5+2+1) f(n) ≤ 8g(n) , for n ≥ 1(n≠0)
n0 = 1

UPPER
BOUND

23
Different Asymptotic Notations (2/3)
‘Ω’ Omega notation:
Ω(g(n))={ f(n): there exist constants C and n0 such that
Cg(n) ≤ f(n) , for n ≥ n0 }
For example:
f(n) = 5 n2 + 2 n + 1 Ω(n2)
g(n) = n2
C=5 5n2 ≤ f(n) , for n ≥ 0
n0 = 0

LOWER
BOUND

24
Different Asymptotic Notations (3/3)
‘Ө’ Theta notation:
Ө(g(n))={ f(n): there exist constants C1 , C2 and n0 such that
C1g(n) ≤ f(n) ≤ C2g(n) , for n ≥ n0 }
For example:
f(n) = 5 n2 + 2 n + 1 Ө(n2)
g(n) = n2
C1 = 5 5n2 ≤ f(n) ≤ 8g(n) , for n ≥ 1
C2 = 8
n0 = 1

TIGHT
BOUND
25
 Asymptotic Complexity
• Finding the exact complexity, f(n) = number of basic
operations, of an algorithm is difficult.
• We approximate f(n) by a function g(n) in a way that
does not substantially change the magnitude of f(n). --
the function g(n) is sufficiently close to f(n) for large
values of the input size n.
• This "approximate" measure of efficiency is called
asymptotic complexity.

• Thus the asymptotic complexity measure does not

give the exact number of operations of an algorithm, but
it shows how that number grows with the size of input.
• This gives us a measure that will work for different
operating systems, compilers and CPUs.

• The most commonly used notation is Big-oh. 26

Big-O (asymptotic) Notation (1/3)

• The most commonly used notation for specifying asymptotic

complexity is the big-O notation.
• The Big-O notation, O(g(n)), is used to give an upper bound
on a positive runtime function f(n) where n is the input size.

Definition of Big-O:
• Consider a function f(n) that is non-negative  n  0. We
say that “f(n) is Big-O of g(n)” i.e., f(n) = O(g(n)), if  n0 
0 and a constant c > 0 such that f(n)  cg(n),  n  n0

The symbol ∀
means “for all”
or “for any”.

The symbol ∃
means “there
exists”.
27
Big-O (asymptotic) Notation (2/3)

Implication of the definition:

• For all sufficiently large n, c *g(n) is an upper bound of f(n)

Note: By the definition of Big-O…

f(n) = 3n + 4 is O(n)
it is also O(n2),
it is also O(n3),
...
it is also O(nn)

• However when Big-O notation is used, the function g in

the relationship f(n) is O(g(n)) is CHOSEN TO BE AS
SMALL AS POSSIBLE.
– We call such a function g a tight asymptotic bound of f(n)

28
Big-O (asymptotic) Notation (3/3)

Big-O complexity classes

( in order of magnitude from smallest to highest )
O(1) Constant
O(log(n)) Logarithmic
O(n) Linear
O(n log(n)) n log n
O(nx) {e.g., O(n2), O(n3)} Polynomial
O(an) {e.g., O(1.6n), O(2n)} Exponential
O(n!) Factorial
O(nn)

Examples of Algorithms and their big-O complexity

Method Best Case Worst Case Average Case
Selection sort O(n2) O(n2) O(n2)
Insertion sort O(n) O(n2) O(n2)
Merge sort O(n log n) O(n log n) O(n log n)
Quick sort O(n log n) O(n2) O(n log n)
29
 Warnings about O-Notation
• Big-O notation cannot compare algorithms in
the same complexity class.

• Big-O notation only gives sensible

comparisons of algorithms in different
complexity classes when n is large .

• Consider two algorithms for same task:

Linear: f(n) = 1000 n
Quadratic: f'(n) = n2/1000
The quadratic one is faster for n < 1000000.
30
 Rules for using big-O
• For large values of input n, the constants and terms
with lower degree of n are ignored.

1. Multiplicative Constants Rule: Ignoring constant factors.

O(c f(n)) = O(f(n)), where c is a constant;
Example: O(20 n3) = O(n3)

2. Addition Rule: Ignoring smaller terms.

If O(f(n)) < O(h(n)) then O(f(n) + h(n)) = O(h(n)).
Example: O(n2 log n + n3) = O(n3)
O(2000 n3 + 2n ! + n800 + 10n + 27n log n + 5) = O(n !)

3. Multiplication Rule: O(f(n) * h(n)) = O(f(n)) * O(h(n))

Example: O((n3 + 2n 2 + 3n log n + 7)(8n 2 + 5n + 2)) = O(n5)

31
How to determine complexity of code structures (1/6)

Loops: for, while, and do-while:

Complexity is determined by the number of iterations in
the loop times the complexity of the body of the loop.
Examples:
for (int i = 0; i < n; i++)
sum = sum - i; O(n)

for (int i = 0; i < n * n; i++)

O(n2)
sum = sum + i;

i=1;
while (i < n) {
sum = sum + i; O(log n)
i = i*2
}
32
How to determine complexity of code structures (2/6)

Nested Loops:
Examples:
sum = 0
for(int i = 0; i < n; i++)
for(int j = 0; j < n; j++) O(n2)
sum += i * j ;

i = 1;
while(i <= n) {
j = 1;
while(j <= n){
statements of constant complexity O(n log n)
j = j*2;
}
i = i+1;
}

33
How to determine complexity of code structures (3/6)

Sequence of statements: Use Addition rule

O(s1; s2; s3; … sk) = O(s1) + O(s2) + O(s3) + … + O(sk)
= O(max(s1, s2, s3, . . . , sk))

Example:
for (int j = 0; j < n * n; j++)
sum = sum + j;
for (int k = 0; k < n; k++)
sum = sum - l;
System.out.print("sum is now ” + sum);

Complexity is O(n2) + O(n) +O(1) = O(n2)

34
How to determine complexity of code structures (4/6)

Switch: Take the complexity of the most expensive case

char key;
int[] X = new int[n];
int[][] Y = new int[n][n];
........
switch(key) {
case 'a':
for(int i = 0; i < X.length; i++) o(n)
sum += X[i];
break;
case 'b':
for(int i = 0; i < Y.length; j++)
for(int j = 0; j < Y[0].length; j++) o(n2)
sum += Y[i][j];
break;
} // End of switch block
Overall Complexity: O(n2)
35
 How to determine complexity of code structures (5/6)
If Statement: Take the complexity of the most expensive case :
char key;
int[][] A = new int[n][n];
int[][] B = new int[n][n];
int[][] C = new int[n][n];
........
if(key == '+') {
for(int i = 0; i < n; i++)
for(int j = 0; j < n; j++)
C[i][j] = A[i][j] + B[i][j];
} // End of if block
O(n2) Overall
complexity
else if(key == 'x')
C = matrixMult(A, B); O(n3)
O(n3)
else
System.out.println("Error! Enter '+' or 'x'!");
O(1)

36
How to determine complexity of code structures (6/6)

• Sometimes if-else statements must be carefully checked:

O(if-else) = O(Condition)+ Max[O(if), O(else)]

int[] integers = new int[n];

........
if(hasPrimes(integers) == true)
integers[0] = 20; O(1)
else
integers[0] = -20; O(1)

public boolean hasPrimes(int[] arr) {

for(int i = 0; i < arr.length; i++)
..........
.......... O(n)
} // End of hasPrimes()

O(if-else) = O(Condition) = O(n) 37