BITS F232

Foundations of
Data Structures & Algorithms
Lect 4
Sameera Muhamed Salam
Dept of CSIS, BITS Pilani, Hyderabad Campus
BITS F232 Sameera M S 2
Introduction

Algorithm
A step by step procedure for solvig a problem in a finite amount of time

set of instructions
to
INPUT convert input OUTPUT
to
expected output

BITS F232 Sameera M S 3

Why do we need to analyze alorithms
I Different methods to solve same problem

BITS F232 Sameera M S 4

Why do we need to analyze alorithms
I Different methods to solve same problem
I To select one amoung them

BITS F232 Sameera M S 5

Why do we need to analyze alorithms
I Different methods to solve same problem
I To select one amoung them
I Need to compare them
I Example: Sorting algorithms

BITS F232 Sameera M S 6

Algorithm Complexity

I Calculated in terms of
I Time Complexity
I Time needed to execute a program that implements the algorithm
I Space Complexity
I Computer memory needed to execute a program that implements the algorithm

BITS F232 Sameera M S 7

Two Methods to Analyze Algorithms

Experimental method:
I Write a program that implements the
algorithm
I Run the program with data sets of
varying size and composition.
I Use a system call to get an accurate
measure of the actual running time.

BITS F232 Sameera M S 8

Two Methods to Analyze Algorithms

Disadvantages:

Experimental method:
I Write a program that implements the
algorithm
I Run the program with data sets of
varying size and composition.
I Use a system call to get an accurate
measure of the actual running time.

BITS F232 Sameera M S 9

Two Methods to Analyze Algorithms
Experimental method:
I Write a program that implements the
algorithm
I Run the program with data sets of
varying size and composition.
I Use a system call to get an accurate
measure of the actual running time.
BITS F232
Disadvantages:
I It is necessary to implement and test
the algorithm in order to determine
its running time.
I Experiments
I is done only on a limited set of
inputs,
I May not be indicative of the
running time on other inputs not
included in the experiment.
I In order to compare two algorithms,
the same hardware and software
environments are needed
Sameera M S 10
Two Methods to Analyze Algorithms

Theoretical method:
I To find Time Complexity
I Need to identify key operations performed in the algorithm.
I Examples
I algorithm to find average of 2 numbers
I algorithm to find roots of a quadratic equation
I algorithm to print values present in all nodes of a graph
I algorithm for inorder traversal of a tree
I algorithm to search a number in an array
I algorithm to sort n elements in an array in descending order
I algorithm to insert an element in agiven position of an array
I Count the number of times key operations are performed.

BITS F232 Sameera M S 11

Time Complexity: Example Linear Search

Algorithm LinearSearch(a,n,x):
INPUT: Array a storing n ≥ 1
integers, Example:
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array a
No otherwise.
I for(i=0 to n)
I if(a[i]==x)
Print Yes;
Break;
I Print No

BITS F232 Sameera M S 12

Time Complexity: Example Linear Search

Algorithm LinearSearch(a,n,x):
INPUT: Array a storing n ≥ 1
integers, Example:
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array a
No otherwise.
I for(i=0 to n)
I if(a[i]==x)
Print Yes;
Break;
I Print No

BITS F232 Sameera M S 13

Time Complexity: Example Linear Search

Algorithm LinearSearch(a,n,x):
INPUT: Array a storing n ≥ 1
integers, Example:
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array a
No otherwise.
I for(i=0 to n)
I if(a[i]==x)
Print Yes;
Break;
I Print No

BITS F232 Sameera M S 14

Time Complexity: Example Linear Search

Algorithm LinearSearch(a,n,x):
INPUT: Array a storing n ≥ 1
integers, Example:
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array a
No otherwise.
I for(i=0 to n)
I if(a[i]==x)
Print Yes;
Break;
I Print No

BITS F232 Sameera M S 15

Linear Search: Algorithm Analysis

I Best Case Complexity

I If x is located in a[0].
I Average Case Complexity
I x= a[i] with probability n1
I Average complexity will be
Σni=1 i × n1
1 n+1
n (1 + 2 + ...n) = 2
I Worst Case Complexity
I If x not present in the array a[] or
I x is present in a[n − 1].

BITS F232 Sameera M S 16

What is the running time of the following code snippet
for(n=1;n < 64;n*2)
printf(” Inside Loop”);

See how the value of n is changing

BITS F232 Sameera M S 17

What is the running time of the following code snippet
for(n=1;n < 64;n*2)
printf(” Inside Loop”);

See how the value of n is changing

20 → 21 → 22 → . . . → 25

BITS F232 Sameera M S 18

What is the running time of the following code snippet
for(n=1;n < 64;n*2)
printf(” Inside Loop”);

See how the value of n is changing

20 → 21 → 22 → . . . → 25

BITS F232 Sameera M S 19

What is the running time of the following code snippet
for(n=1;n < 64;n*2)
printf(” Inside Loop”);

See how the value of n is changing

20 → 21 → 22 → . . . → 25

BITS F232 Sameera M S 20

Binary Search
Algorithm BinarySearch(A,n,x): INPUT:
A Sorted array A storing n ≥ 1
integers,
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array A Suppose array contains
No otherwise. 20,27,30,45,52,64, 70
you want to search for 35.
I low = 0
I high = n
I repeat untill high < low
I mid = d(low + high)/2e
I if A[mid] == x then do
I Print Yes
I break;
I else if A[mid] < x then
low = mid + 1
I else high = mid − 1
BITS F232 Sameera M S 21
Binary Search
Algorithm BinarySearch(A,n,x): INPUT:
A Sorted array A storing n ≥ 1
integers,
x is the number to be
searched in the array a
OUTPUT: Yes if x is present in array A Suppose array contains
No otherwise. 20,27,30,45,52,64, 70
you want to search for 35.
I low = 0
I high = n
I repeat untill high < low
I mid = d(low + high)/2e
I if A[mid] == x then do
I Print Yes
I break;
I else if A[mid] < x then
low = mid + 1
I else high = mid − 1
BITS F232 Sameera M S 22
Binary Search
Algorithm BinarySearch(A,n,x): INPUT:
A Sorted array A storing n ≥ 1
integers, Suppose array contains
x is the number to be 20,27,30,45,52,64, 70
searched in the array a you want to search for 35.
OUTPUT: Yes if x is present in array A
No otherwise.
I low = 0
I high = n
I repeat untill high < low
I mid = d(low + high)/2e
I if A[mid] == x then do
I Print Yes
I break;
I else if A[mid] < x then
low = mid + 1
I else high = mid − 1
BITS F232 Sameera M S 23
Why do We Need Asymptotic Notations
I Function representing time complexity may not be a smooth function
(eg:1.6n2 + 3n + 2).
I Our goal is to identify broad classes of algorithms that have similar behavior
I we want to express the growth rate of running times and other functions in a way
that is insensitive to
I constant factors
I low order terms.

BITS F232 Sameera M S 24

Why do We Need Asymptotic Notations
I Function representing time complexity may not be a smooth function
(eg:1.6n2 + 3n + 2).
I Our goal is to identify broad classes of algorithms that have similar behavior
I we want to express the growth rate of running times and other functions in a way
that is insensitive to
I constant factors
I low order terms.

Capturing the essence:

How the running time of an algorithm increases with the size of the input in the limit.
I We use asymptotic notations to represent these function as a smooth upper
bound or lower bound or tight bound function.
I Big Oh (O) Notation: To represent asymptotic upper bound
I Big Omega (Ω) Notation: To represent asymptotic lower bound
I Big Theta (Θ) Notation: To represent asymptotic tight bound

BITS F232 Sameera M S 25

Asymptotic upper Bound

10N 2

100N

10 N

I For all values of N ≥ 10, 10N 2 ≥ 100N

BITS F232 Sameera M S 26
Big Oh Notation
I Big Oh Notation Definition: Let g and f be functions from the set of natural
numbers to itself. The function f is said to be O(g ), if there is a constant c > 0
and a natural number n0 such that f (n) ≤ cg (n) for all n ≥ n0 .

BITS F232 Sameera M S 27

 10N 2

100N

10 N

Figure: f (N) = 100N, cg (N) = 10N 2 , n0 = 10

BITS F232 Sameera M S 28

 10N 2

100N

10 N

Figure: f (N) = 100N, cg (N) = 10N 2 , n0 = 10

BITS F232 Sameera M S 29

Example: Big Oh Notation
1. 6n-1

BITS F232 Sameera M S 30

Example: Big Oh Notation
1. 6n-1
I 6n − 1 < 6n for every n ≥ 1
I c =?, n0 =?

BITS F232 Sameera M S 31

Example: Big Oh Notation
1. 6n-1
I 6n − 1 < 6n for every n ≥ 1
I c =?, n0 =?
I c = 6, n0 = 1
2. 3n2 + 5n
I let n = 5
I 3n2 + 5n = 3n2 + n2 = 4n2

For every n ≥ 5, 3n2 + 5n ≤ 4n2

BITS F232 Sameera M S 32

Example: Big Oh Notation
1. 6n-1
I 6n − 1 < 6n for every n ≥ 1
I c =?, n0 =?
I c = 6, n0 = 1
2. 3n2 + 5n
I let n = 5
I 3n2 + 5n = 3n2 + n2 = 4n2

For every n ≥ 5, 3n2 + 5n ≤ 4n2

3n2 + 5n = O(n2 ) for n0 = 5 and c = 4

BITS F232 Sameera M S 33

Exercise: Big Oh Notation
1. f (n) = 2n2 + 3n + 1
I 2n2 + 3n + 1 < 2n2 + 3n + n = 2n2 + 4n < 2n2 + n2 < 3n2
I c=?

BITS F232 Sameera M S 34

Exercise: Big Oh Notation
1. f (n) = 2n2 + 3n + 1
I 2n2 + 3n + 1 < 2n2 + 3n + n = 2n2 + 4n < 2n2 + n2 < 3n2
I c=?
I c=3
I n0 = ?

BITS F232 Sameera M S 35

Exercise: Big Oh Notation
1. f (n) = 2n2 + 3n + 1
I 2n2 + 3n + 1 < 2n2 + 3n + n = 2n2 + 4n < 2n2 + n2 < 3n2
I c=?
I c=3
I n0 = ?
I n0 = 4

BITS F232 Sameera M S 36

Exercise: Big Oh Notation
1. f (n) = 2n2 + 3n + 1
I 2n2 + 3n + 1 < 2n2 + 3n + n = 2n2 + 4n < 2n2 + n2 < 3n2
I c=?
I c=3
I n0 = ?
I n0 = 4
I f (n) = 2n2 + 3n + 1 = O(?)

BITS F232 Sameera M S 37

Exercise: Big Oh Notation
1. f (n) = 2n2 + 3n + 1
I 2n2 + 3n + 1 < 2n2 + 3n + n = 2n2 + 4n < 2n2 + n2 < 3n2
I c=?
I c=3
I n0 = ?
I n0 = 4
I f (n) = 2n2 + 3n + 1 = O(?)
I f (n) = 2n2 + 3n + 1 = O(n2 )

I A constant-time algorithm: O(1)

I A logarithmic algorithm: O(logn)
I A linear algorithm: O(n)
I A superlinear algorithm: O(nlogn)
I A polynomial algorithm: O(nc )
I A exponential algorithm: O(c n )
I A factorial algorithm : O(n!)
BITS F232 Sameera M S 38
Asymptotic Lower Bound

10N 2

100N

10 N

I For all values of N ≥ 10, 100N ≤ 10N 2

BITS F232 Sameera M S 39
Big Omega Notation

Big Omega Notation Definition:

Let g and f be functions from the set of natural numbers to itself. The function f is
said to be Ω(g ), if there is a constant c > 0 and a natural number n0 such that
f (n) ≥ cg (n) for all n ≥ n0 .

BITS F232 Sameera M S 40

Big Theta Notation
Big Theta Notation Definition:
Let g and f be functions from the set of natural numbers to itself. The function f is
said to be Θ(g ), if there exists 2 constants c1 > 0 and c2 > 0 and a natural number n0
such that c1 g (n) ≤ f (n) ≤ c2 g (n) for all n ≥ n0 .

BITS F232 Sameera M S 41

Example
I 10 log2 n + 4
I 10 log2 n + 4 ≤ 10 log2 n + log2 n
I 10 log2 n + 4 ≤ 11 log2 n =⇒ c2 = 11
I n0 =?

BITS F232 Sameera M S 42

Example
I 10 log2 n + 4
I 10 log2 n + 4 ≤ 10 log2 n + log2 n
I 10 log2 n + 4 ≤ 11 log2 n =⇒ c2 = 11
I n0 =?
log2 n = 4 =⇒ n = 24 = 16
I How to find c1 ?

BITS F232 Sameera M S 43

Example
I 10 log2 n + 4
I 10 log2 n + 4 ≤ 10 log2 n + log2 n
I 10 log2 n + 4 ≤ 11 log2 n =⇒ c2 = 11
I n0 =?
log2 n = 4 =⇒ n = 24 = 16
I How to find c1 ?
I 10 log2 n + 4 ≥ log2 n is true for all n ≥ 1
I c1 =?

BITS F232 Sameera M S 44

Example
I 10 log2 n + 4
I 10 log2 n + 4 ≤ 10 log2 n + log2 n
I 10 log2 n + 4 ≤ 11 log2 n =⇒ c2 = 11
I n0 =?
log2 n = 4 =⇒ n = 24 = 16
I How to find c1 ?
I 10 log2 n + 4 ≥ log2 n is true for all n ≥ 1
I c1 =?
I c1 = 1
I Therefore, log2 n ≤ 10 log2 n + 4 ≤ 11 log2 n
I What is n0 ?
11 log2 n
10 log2 n + 4

log2 n

16 n

BITS F232 Sameera M S 45

Example
I 10 log2 n + 4
I 10 log2 n + 4 ≤ 10 log2 n + log2 n
I 10 log2 n + 4 ≤ 11 log2 n =⇒ c2 = 11
I n0 =?
log2 n = 4 =⇒ n = 24 = 16
I How to find c1 ?
I 10 log2 n + 4 ≥ log2 n is true for all n ≥ 1
I c1 =?
I c1 = 1
I Therefore, log2 n ≤ 10 log2 n + 4 ≤ 11 log2 n
I What is n0 ?
11 log2 n
10 log2 n + 4

log2 n

16 n

I n0 = 16
BITS F232 Sameera M S 46
Properties of Asymptotic Notations

I Reflexive: f (n) = O(f (n)) True for Ω and Θ

I Symmetric: f (n) = Θ(g (n)) if and only if g (n) = Θ(f (n))
I Transitive: f (n) = O(g (n)) and g (n) = O(h(n)) =⇒ f (n) = O(h(n))
True for Ω and Θ
I Transpose Symmetry: f (n) = O(g (n)) if and only if g (n) = Ω(f (n))
I max(f (n), g (n)) = Θ(f (n) + g (n))
I O(f (n)) + O(g (n)) = O(max(f (n), g (n)))

BITS F232 Sameera M S 47

More Properties

BITS F232 Sameera M S 48

Examples:

BITS F232 Sameera M S 49

Examples:

BITS F232 Sameera M S 50

Exercise:

Which of the following are O(n2 ), Ω(n2 ) and Θ(n2 )

1. n2
2. n2 + 1000n
3. 1000n2 + 1000n + 32700
4. 1000n2 − 1000n − 1048315
5. n3
6. n2.00001
7. n1.99999
8. n log n
9. n2 log n

BITS F232 Sameera M S 51

Small Oh and Small Omega notations

Small Oh Notation (o)

Let g and f be functions from the set of natural numbers to itself. The function f is
said to be o(g ), if there is a constant c > 0 and a natural number n0 such that
f (n) < cg (n) for all n ≥ n0 .

Small Omegha Notation (ω)

Let g and f be functions from the set of natural numbers to itself. The function f is
said to be ω(g ), if there is a constant c > 0 and a natural number n0 such that
f (n) > cg (n) for all n ≥ n0 .

BITS F232 Sameera M S 52

Discussion Time!

BITS F232 Sameera M S 53

BITS F232 Sameera M S 54