University of Science, VNU-HCM

Faculty of Information Technology

Data Structure and Algorithm

Analysis of Algorithms

Lecturer: Lê Ngọc Thành

Email: lnthanh@fit.hcmus.edu.vn

HCM City
Contents

• Introduction to algorithms and algorithm

analysis
• Criteria and ways of evaluating algorithms
– Basic criteria
– 𝐵𝑖𝑔 − 𝑂, 𝐵𝑖𝑔 − Ω, 𝐵𝑖𝑔 − Θ
• Applying the assessment method

2
What is an algorithm?
• What is an algorithm?
– Algorithms are steps to take to solve a problem.
– For example, the problem of "fried chicken
eggs“
▪ Step 1: take the saucepan.
▪ Step 2: take the bottle of cooking oil
– Is there a bottle of cooking oil?
» If so, pour it into the pan
» If not, go buy it?

▪ Step 3: turn on the gas stove

▪…

3
Necessary to analyze the algorithm?
• How many ways to get from Ho Chi Minh
City to Hanoi?
– Aircraft
– Train
– Boat
– Buses
– Bus
– Ho Chi Minh Way...
• Why not optimize the only one way to go?
– Depends on circumstances, conditions,
availability, convenience, comfort, …
4
Necessary to analyze the algorithm?
• In computer science, there are also many
algorithms to solve the same problem.
– Example?
▪ Sorting problem: insertion sort, selection sort, quick
sort, ...
▪ Compression algorithm: Huffman, RLE compression,
...
• Algorithm analysis helps determine which
algorithms are effective in terms of space
and time, …
– How long does the program run?
– Why is the program out of memory?
–… 5
Object of algorithm analysis
• Aspect of consideration:
– Running time (*)
– Memory consumption (*)
– Ease of understanding
– Stability
– Fault tolerance
–…
• Now, we focus on running time.

6
Time-out analysis
• Time-out analysis:
– Consider how the processing time increases as
the size of the problem increases.
– The size of the problem usually depends on the
size of the input:
▪ Array size
▪ Number of matrix elements
▪ Number of bits expressed by input
▪ Number of vertexes and edges of a graph
▪…

7
How to compare algorithms?
• Measurement:
– Run time?
▪ Not good because it depends on the configuration of each
computer.
– Number of statements are performed?
▪ Not good because it depends on the programming
language as well as the style of each programmer.
• So is there any way to compare independently
of the machine, programming style, ...?
– Viewing the duration of a program's implementation
is a function of n.
𝑓 𝑛 =⋯
8
Growth rate
• Growth rate is the rate at which the cost of the
algorithm increases as the size of input
increases.
• The rate of increase usually depends on the
fastest element as 𝑛 ⟶ ∞:
– Example 1: when the monthly income is several
billion, a few thousand becomes odd, insignificant.
– Example 2: an algorithm with an running time
function
𝑓 𝑛 = 𝑛100 + 𝑛2 + 10000000 ≈ 𝑛100
• Thus, when comparing algorithms, it is usually
based on the growth rate.
9
Growth rate

10
Compare functions on growth rate

Growth
Rate
increases

11
Type of analysis
• Depends not only on input size, but the
layout, structure, type of input data
→ the growth rate is also different.
• Example: number of comparisons to find value 1 in
the following array?

12
Type of analysis
• There are three types of analysis:
– Best case (Big-Ω):
▪ The input data layout makes the algorithm run the
fastest.
▪ Rarely happens in practice.
– Average case:
▪ Random data layout
▪ Difficult to predict to distribution.
– Worst case, Big-O:
▪ Input data layout causes the algorithm to run at the
slowest.
▪ Make sure to upper bound.
𝑳𝒐𝒘𝒆𝒓 𝑩𝒐𝒖𝒏𝒅 ≤ 𝑨𝒗𝒆𝒓𝒂𝒈𝒆 𝑻𝒊𝒎𝒆 ≤ 𝑼𝒑𝒑𝒆𝒓 𝑩𝒐𝒖𝒏𝒅
13
Big-O
• In order to assess the worst case or upper
bound, one gives the concept of Big-O.

O
𝑳𝒐𝒘𝒆𝒓 𝑩𝒐𝒖𝒏𝒅 ≤ 𝑨𝒗𝒆𝒓𝒂𝒈𝒆 𝑻𝒊𝒎𝒆 ≤ 𝑼𝒑𝒑𝒆𝒓 𝑩𝒐𝒖𝒏𝒅

14
History of Big-O
• The notation Big-O was introduced in 1894 by
Paul Bachmann (Germany) in the book
Analytische Zahlentheorie (“Analytic Number
Theory") (2nd edition)
• This notation (later) was popularized by
mathematics Edmund Landau, so it's called the
Landau notation, or Bachmann-Landau notation
• Donald Knuth was the one who introduced the
notation into Computer Science in 1976 – “Big
Omicron and big Omega and big Theta” - ACM
SIGACT News, Volume 8, Issue 2

15
Big-O (tt)
• Given an algorithm’s time function to be 𝑓 𝑛
– Example: 𝑓 𝑛 = 𝑛4 + 100𝑛2 + 10𝑛 + 50
• Calling the upper bound function of f(n) is
𝑔(𝑛) when n is large enough. That is, there
is no large enough n value to make 𝑓 𝑛
beyond c𝑔(𝑛) (c: constant)
– Ví dụ: 𝑔 𝑛 = 𝑛4
• 𝑓(𝑛) is called the Big-O of g(n).
Or 𝑔 𝑛 𝑖𝑠 𝑡ℎ𝑒 ℎ𝑖𝑔ℎ𝑒𝑠𝑡 𝑔𝑟𝑜𝑤𝑡ℎ 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑓(𝑛)

16
Big-O (tt)
• Definition:
– Given f(n) and g(n) are two functions
– 𝑓 𝑛 = 𝑂(𝑔 𝑛 ), if there exists positive numbers
𝑐 and 𝑛0 so that:
0 ≤ 𝑓(𝑛) ≤ 𝑐𝑔(𝑛)
where 𝑛 ≥ 𝑛0
– 𝑓 is the Big-O of 𝑔 if a positive number exists so
that 𝑓 cannot be greater than 𝑐 ∗ 𝑔 when n is
large enough
→ 𝑔(𝑛) is called the upper bound of 𝑓(𝑛)

17
Big-O

For small n (𝑛 < 𝑛0 ) we often do not care because it is often in critical

and does not show the complexity of the algorithm.
18
Algorithm analysis with Big-O
• Comments:
– Each algorithm has many running time functions
corresponding to each input's layout resulting in
the best, average, and worst cases.
– 𝑔(𝑛) is the upper bound of all those functions.
• So comparing algorithms, we just need to
compare its 𝑔(𝑛) or Big-O.
– For example: Instead of using the complexity of the
algorithm is 2𝑛2 + 6𝑛 + 1, we will use the upper bound
of the complexity of the algorithm as 𝑛2

19
Is there one or more functions g(n)?
• If exists 𝑔′(𝑛) > 𝑔(𝑛), does 𝑔′(𝑛) satisfy Big-
O conditions?
• Notes:
– The best 𝑔(𝑛) is that the smallest function still
satisfies Big-O.
– As simple as possible (?)

20
Big-O exercises
• Find the upper bound function with the 𝑐 and 𝑛0
values of the following functions:
a) 𝑓 𝑛 = 3𝑛 + 8
b) 𝑓 𝑛 = 𝑛2 + 1
c) 𝑓 𝑛 = 𝑛4 + 100𝑛2 + 50
d) 𝑓 𝑛 = 2𝑛3 − 2𝑛2
e) 𝑓 𝑛 =𝑛
f) 𝑓 𝑛 = 410
• Solutions: (𝑔(𝑛), 𝑐, 𝑛0 )
a. (𝑛, 4, 8) b. (𝑛2 , 2, 1) c. (𝑛4 , 2, 100)
d. (𝑛3 , 2,1) e. (𝑛2 , 1,1) f. (1, 1, 1)
Is there another answer??
21
Big-O exercises
• Uniqueness?
– 𝑓 𝑛 = 100n + 5

22
Big-O exercises
• Identify the O(g(n)) of the following functions:
– f(n) = 10
– f(n) = 5n + 3
– f(n) = 10n2 – 3n +20
– f(n) = logn + 100
– f(n) = nlogn + logn + 5
• Which statement is correct?
– 2n+1 = O(2n) ?
– 22n = O(2n) ?

23
Big-Ω
• Lower bound or best case expressed by Big-
Ω.

Ω
𝑳𝒐𝒘𝒆𝒓 𝑩𝒐𝒖𝒏𝒅 ≤ 𝑨𝒗𝒆𝒓𝒂𝒈𝒆 𝑻𝒊𝒎𝒆 ≤ 𝑼𝒑𝒑𝒆𝒓 𝑩𝒐𝒖𝒏𝒅

24
Big-Ω
• Given an algorithm’s time function to be 𝑓 𝑛
– Example: 𝑓 𝑛 = 100𝑛2 + 10𝑛 + 50
• Call the lower bound function of 𝑓(𝑛) as
𝑔(𝑛) when n is large enough. This means
that no value of n is large enough to make
𝑐𝑔(𝑛) exceed 𝑓(𝑛) (𝑐: hằng số)
– Example: 𝑔 𝑛 = 𝑛2
• 𝑓(𝑛) called Big-Ω of 𝑔(𝑛).
or 𝑔(𝑛) is the minimum growth rate of 𝑓(𝑛)

25
Big-Ω
• Definition:
– Given f(n) and g(n) are two functions
– 𝑓 𝑛 = Ω(𝑔 𝑛 ), if there exists positive numbers
𝑐 and 𝑛0 so that:
0 ≤ 𝑐𝑔(𝑛) ≤ 𝑓(𝑛)
where 𝑛 ≥ 𝑛0
– 𝑓 is Big-Ω of 𝑔 if a positive number exists so that
f cannot be smaller c*g when n is large enough
→ 𝑔(𝑛) called the lower bound of 𝑓(𝑛)

26
Big-Ω

27
Big-Ω exercises
• Find the lower bound function with the 𝑐 and
𝑛0 values of the following functions :
a) 𝑓 𝑛 = 5𝑛2
b) 𝑓 𝑛 = 𝑛
c) 𝑓 𝑛 = 𝑛3
• Solutions:
a. Ω(𝑛) b. Ω(𝑙𝑜𝑔𝑛) c. Ω(𝑛2 )

28
Big-

• Big- used to determine if the lower and

upper bound are the same.

𝑳𝒐𝒘𝒆𝒓 𝑩𝒐𝒖𝒏𝒅 (𝛀) ≤ 𝑨𝒗𝒆𝒓𝒂𝒈𝒆 𝑻𝒊𝒎𝒆 ≤ 𝑼𝒑𝒑𝒆𝒓 𝑩𝒐𝒖𝒏𝒅(𝚶)


• If Ω and  are the same, the growth rate of
the average case is similar.

29
Big-
• Definition:
– Given f(n) and g(n) are two functions
– 𝑓 𝑛 = (𝑔 𝑛 ), if there exists positive numbers
𝑐 and 𝑛0 so that:
0 ≤ 𝑐1 𝑔(𝑛) ≤ 𝑓(𝑛) ≤ 𝑐2 𝑔(𝑛)
where 𝑛 ≥ 𝑛0
→ 𝑔(𝑛) is called the tight bound of 𝑓(𝑛)

30
Big-

31
Big-  Exercises
1) Find the tight bound function of the following
function:
𝑛2 𝑛
𝑓 𝑛 = −
2 2
2) Prove:
a. 𝑛 ≠ 𝜃(𝑛2 )
b. 6𝑛3 ≠ 𝜃(𝑛2 )
c. 𝑛 ≠ 𝜃(log 𝑛)
• Solution:
2 1
1) 𝜃 𝑛 , 𝑐1 = , 𝑐2 = 1, 𝑛0 = 1
5
2)
1
a. n ≤ (can't) …
𝑐1
32
Summary about Big-O, Big-, Big-

In general, the fact that people are only interested in the upper
bound (Big-O), while the lower limit (Big-Ω) does not matter, the tight
limit (Big- ) only considers when the upper and lower limits are the
same. 33
Compare some functions

34
Compare some functions

35
Run time

36
Complexity calculation method

37
Asymptotic analysis
• With an 𝑓(𝑛) algorithm, we always want to
find another function 𝑔(𝑛) that approximately
𝑓(𝑛) has a higher value.
• The 𝑔(𝑛) curve is called the asymptotic
curve.
• The problem of finding 𝑔(𝑛) is called
asymptotic analysis

38
Asymptotic analysis – Loop
Loop runtime is usually the execution time of
the instructions within the loop multiplied by
the number of iterations.
For example:
for (i=1; i<=n; i++)
{
m = m+2; // constant time c
}
Run time = 𝑐 × 𝑛 = 𝑐𝑛 = 𝑂(𝑛)

39
Nested loops
For a nested loop, we analyze the inside out.
The total time is the product of the size of all
loops.
For example:
for (i=1; i<=n; i++)
{
for (j=1; j<=n; j++)
{
k = k+1; // constant time c
}
}
Thời gian chạy = 𝑐 × 𝑛 × 𝑛= 𝑐𝑛2 = 𝑂(𝑛2 )
40
Sequential instructions
For sequential instructions, to calculate run
time, we add up the time complexity of each
instruction
Example:
x = x + 1: //constant time c0
for (i=1; i<=n; i++)
{
m = m + 2; //constant time c1
}
for (i=1; i<=n; i++)
{
for (j=1; j<=n; j++)
{
k = k+1; //constant time c2
}
}
Run time = 𝑐0 + 𝑐1𝑛 + 𝑐2𝑛2 = 𝑂(𝑛2 ) 41
Conditional statement
Usually we consider the worst case scenario,
the run time is equal to the longest part of the
conditional statement.

42
Conditional statement
Example:
//test: constant c0
if (length() != otherStack. length())
{
return false; //then part: constant c1
}
else
{
// else part: (constant + constant) * n
for (int n = 0; n < length(); n++)
{
if (!list[n].equals(otherStack.list[n]))
//constant c2
return false; // c3
}
}
Runtime = 𝑐0 + 𝑐1𝑛 + 𝑐2 + 𝑐3 𝑛 = 𝑂(𝑛) 43
Log complexity
• An algorithm has complexity 𝑂 (𝑙𝑜𝑔𝑛) if it
takes a constant time to reduce the problem
size by a fraction (usually ½).
• Example:
for (i=1; i<=n;)

{
i=i*2;
}

44
The rule of addition and multiplication

• The rule of addition:

If f1 = O(g1) and f2 = O(g2) then f1+f2 = O(max{g1,g2})
If we perform several operations in order, the execution
time is dominated by the operation that takes the most
time.
• The rule of multiplication:
If f1 = O(g1) and f2 = O(g2) then f1*f2 = O(g1*g2)
If we repeat an operation a number of times, the total
execution time is the execution time of the operation
multiplied by the number of iterations.

45
Some properties
• Transitive property:
𝑓 𝑛 = 𝑂 𝑔 𝑛 and 𝑔 𝑛 = 𝑂 ℎ 𝑛
→ 𝑓 𝑛 = 𝑂(ℎ 𝑛 )
• Reflectivity property:
𝑓 𝑛 = 𝑂(𝑓 𝑛 )

46
Summary of basic complexity calculations

• Simple statement (read, write, assign)

– O(1)
• Simple calculations (+ - * / == > >= < <=)
– O(1)
• Sequence of simple statements / calculations
– Rule of addition
• Loop for, do, while
– Rule of multiplication

47
Calls a function at the same level?

Function A calls Function B

…
• The complexity of the sequence of
operations calling the function?
𝑂 𝑛 = max{𝑂𝐴 𝑛 , 𝑂𝐵 𝑛 , 𝑂𝑐 𝑛 … }

48
Commonly used Logarithms and Summations

49
Big-O Exercises
• Calculate the upper bound (big-O) for the
following algorithms:
Ex1: for (i = 0; i < n; i++)
for (j = 0; j < n; j++)
b[i][j] += c;

Ex2: for (i = 0; i < n; i++)

if (a[i] == k) return 1;
return 0;

Ex3: for (i = 0; i < n; i++)

for (j = i+1; j < n; j++)
b[i][j] -= c;
50
Big-O Exercises
▪ Ex 4,5,6: matrix addition, multiplication and
transposition

51
Some more exercises
s = 0;
for (i=0; i<=n;i++){
p =1;
for (j=1;j<=i;j++)
p=p * x / j;
s = s+p;
}

52
Some more exercises
s = 1; p = 1;
for (i=1;i<=n;i++) {
p = p * x / i;
s = s + p;
}

53
Some more exercises
• s=n*(n-1) /2;

54
Some more exercises
for (i= 1;i<=n;i++)
for (j= 1;j<=m;j++)
x += 2;

55
Some more exercises

56
Some more exercises

57
Some more exercises

58
Some more exercises

59
Some more exercises

60
Some more exercises
for (i= 1;i<=n;i++)
u = u + 2;
for (j= 1;j<=m;j++)
u = u * 2;

61
Some more exercises
for (i= 1;i<=n;i++) {
for (u= 1;u<=m;u++)
for (v= 1;v<=n;v++)
h = x*100;
for j:= 1 to x do
for k:= 1 to z do
x = h/100;
}

62
Some more exercises
for (i= 1;i<=n;i++)
for (j= 1;j<=m;j++) {
for (k= 1;k<=x;k++)
//statement
for (h= 1;h<=y;h++)
//statement
}

63
Some more exercises
for (i= 1;i<=n;i++)
for (u= 1;u<= m;u++)
for (v= 1;v<=n;v++)
//statement
for (j= 1;j<=x;j++)
for (k= 1;k<=z;k++)
// statement

64
Some more exercises
P(x) = xmxm+am-1xm-1+ …+a1x+a0
Q(x) = bnxn+bn-1xn-1+…+b1x+b0
if (m<n) p = m; else p =n;
for (i=0;i<=p;i++)
c[i]=a[i] + b[i];
if (p<m)
for (i=p+1;i<=m;i++) c[i] = a[i];
else
for (i=p+1;i<=n;i++) c[i] = b[i];
while (p>0 && c[p] = 0) p = p-1;

65
Some more exercises
P(x) = xmxm+am-1xm-1+ …+a1x+a0
Q(x) = bnxn+bn-1xn-1+…+b1x+b0
p = m+n;
for (i=0;i<=p;i++) c[i] = 0;
for (i=0;i<=m;i++)
for (j=0;j<=n;j++)
c[i+j] = c[i+j] + a[i] + b[j];

66
Conclusion
• Although analyzing algorithm complexity
gives the most accurate comparison
between algorithms, it can in fact make it
difficult or impossible for programmers to
find Big-O for some programs has high
complexity.
• Therefore, people often return to the method
of running an experiment (measuring the
runtime) with the same hardware platform,
language, data set. Also try changing the
different data sets size and type.
67
The End.