Complexity of Algorithm:

But now a day's problem of space rarely occurs because space on

the computer (internal or external) is enough.

Broadly, we achieve the following types of analysis –

•Worst-case: f (n) defined by the maximum number of steps taken

on any instance of size n.
•Best-case: f (n) defined by the minimum number of steps taken on
any instance of size n.
•Average case: f (n) defined by the average number of steps taken
on any instance of size n.

Topic: Introduction 28
 Complexity of Algorithm:

It is very convenient to classify algorithm based on the relative

amount of time or relative amount of space they required and
specify the growth of time/space requirement as a function of input
size.

Time Complexity: Running time of a program as a function of the

size of the input.

Space Complexity: Some forms of analysis could be done based on

how much space an algorithm needs to complete its task. This
space complexity analysis was critical in the early days of
computing when storage space on the computer was limited. When
considering this algorithm are divided into those that need extra
space to do their work and those that work in place.
Topic: Introduction 29
 Big-O notation:

Def f(n) = O(g(n)) if and only if  2 positive constants c and n0,

such that
|f(n)|  c  |g(n)|  n  n0.
So, g(n) actually is the upper bound of f(n).
c | g (n) |

| f ( n) |

n0
Figure 1. Illustrating “big O”
Topic: Introduction 30
Examples:
• Is 17 n2 – 5 = O(n2)?
 17n 2 5  17n 2 n  1
(c 17, n0  1)
 17n 2 5  O(n 2 )
• Is 35 n3 + 100 = O(n3)?
 35n 3  100  36n 3 n  5
(c  36, n0  5)
 35n 3  100  O(n 3 )
• Is 6  2n + n2 = O(2n)?
 6  2n  n2  7  2n n  5
(c  7, n0  5)
 6  2 n  n 2  O( 2 n )
Topic: Introduction 31
Complexity classes

f(n) n! 2n n3
n2

n log n

n (linear time)

log n

n
Figure 2. Growth rates of some important complexity classes
Topic: Introduction 32
 Assume that we have a machine that can execute
1,000,000 required operations per sec

Algorithm 1 Algorithm 2 Algorithm 3 Algorithm 4 Algorithm 5

Frequency 33n 6nlogn 13n2 3.4n3 2n

count
n=10 < 1 sec < 1 sec < 1 sec < 1 sec < 1 sec
n=10,000 < 1 sec < 1 sec 22 min 39 days many many
centuries

Table 1. Execution time for algorithms with the

given time complexities

Topic: Introduction 33
Note:
log n
n(linear)
n log n polynomial time (easy or tractable)
n2
n3

2n
exponential time (hard or intractable)
n!

Topic: Introduction 34
How do we know a given problem is easy?

Give a polynomial time algorithm for the problem

 Need to be good in
design of algorithms (CS331)
 Need to be good in
analysis of algorithms (CS331)

How do we know a given problem is hard?

Show that it is as hard as those well-known “hard”

problems.
 Need help from the
theory of NP Completeness (CS331)

Topic: Introduction 35
Whether your program is the best or not?
(Comparing algorithms)

The problem: Counting the number of 1’s in a n-bit string

algorithms:

1. c ← 0;
for i ← 1 to n do
if bi = 1 then c ← c+1;

Complexity: T(n) = O(n)

Topic: Introduction 36
2. Assume B = bnbn-1…b2b1
c ← 0;
while B ≠ 0 do
c ← c+1;
B ← B  (B –1 ); ( Note that  is “logical and” )

Complexity: T(n) = O( # of 1’s)

example: B = 100011
c←1 100011 B
) 100010 B-1
c←2 100010 B’
) 100001 B’-1
c←3 100000 B’’
) 011111 B’’-1
000000 B’’’
Topic: Introduction 37
3. Can we do it in log n time?
B = 11010001
b8 b7 b6 b5 b4 b3 b2 b1

B 1 1 0 1 0 0 0 1

Odd 1 1 0 1
Shift even to right 1 0 0 0
B1 = Bodd + Beven 10 01 00 01
Odd 01 01
Shift even to right 10 00
B2 = Bodd + Beven 0011 0001

Odd 0001
Shift even to right 0011
B3 = Bodd + Beven 0100

Topic: Introduction 38
Idea:
4 11010001

3 1101 0001 1

==> T(n) = O(logn)

2 11 1 01 00 010 1

1 1 0 1 0 0 0 1
Note:
- The familiar machine that we used can only execute one
operation in each step  sequential machine.
-This stated algorithm requires the machine to have more than
one operation done in each step  parallel machine
Topic: Introduction 39
Ω notation
Def f (n)  ( g (n)) iff  two positive constants c and n0,
such that
| f (n) |  c | g (n) | n  n0
So, g(n) is the lower bound of f(n).
| f (n) |

c | g (n) |

n0
Figure 3. Illustrating Ω
Topic: Introduction 40
Examples:
•Is 3n  2  (n) ?
 3n  2  3  n n  1
(c  3, n0  1)
 3n  2  (n)
•Is 3n  2  (1) ?
 3n  2  1  1 n  1
(c  1, n0  1)
 3n  2  (1)
value of n0
•Is 6  2 n  n 2  (n100 )?
 6  2 n  n 2  ? n100 n  ?
When n is bigger, 2n will grow faster than n100. ( Yes, you can find n0 )
 6  2  n  (n
n 2 100
)
Topic: Introduction 41
Θ notation
Def f (n)  ( g (n)) iff  three positive constants, c1, c2, n0 ,

such that c1  | g (n) |  | f (n) |  c2  | g (n) | n  n0 .

c2  | g (n) |

| f ( n) |

c1  | g (n) |

n0 Figure 4. Illustrating Θ
Topic: Introduction 42
Examples:

• Is 3n + 2 = (n)?
 3n  3n  2  4n n  2
(c1  3, c 2  4, n0  2)
 3n  2  (n)

• Is 3n + 2 = (n2)?

 3n  2  (n ) 2

 3n  2  (n ) 2

Topic: Introduction 43
• Is 6  2 n
 n 2
 ( 2 n
)?

 6  2n  6  2n  n2  7  2n n  4
(c1  6, c 2  7, n0  4)
 6  2 n  n 2  ( 2 n )

• Is 4n 3  3n 2  (n 2 ) ?
 4n 3  3n 2  O(n 2 )
 4n 3  3n 2  (n 2 )

Topic: Introduction 44
 Property of Order

1) Transitive:
If f (n)  O( g (n)) and g (n)  O(h(n)) then f (n)  O(h(n))

If f (n)  ( g (n)) and g (n)  (h(n)) then f (n)  (h(n))

If f (n)  ( g (n)) and g (n)  (h(n)) then f (n)  (h(n))

2) Reflexive:
f (n)  O( f (n))
f (n)  ( f (n))
f (n)  ( f (n))

Topic: Introduction 45
3) Symmetric:

f (n)  ( g (n)) iff g (n)  ( f (n))

f (n)  O( g (n)) iff g (n)  ( f (n))

Topic: Introduction 46
 Analysis of Algorithms

a) Best Case, Worse Case Analysis

The problem:
Sort n keys in nondecreasing sequence.

Solving strategy:
Insertion sort – insert the ith item into a sorted list of
length (i – 1) by looking at numbers one at a time,  i >1.
Example: sort 3,5,4,1
step 1 step 2 step 3 step 4
3 3 3 1
5 5 4 3
4 4 5 4
1 1 1 5
Topic: Introduction 47
Algorithm:

A(0) ← -maxint; // for efficient to stop the loop

for i ← 2 to n do
target ← A(i);
j ← i –1;
while (target < A(j))
A( j + 1) ← A(j);
j ← j –1;
A( j + 1) ← target;

Topic: Introduction 48
 Analysis:

Best Case performance is Θ(n) or Ω(n), but not

Ω(n2) since insertion sort runs in Θ(n) when
the input is sorted.

Worst Case performance is Θ(n2) or O(n2)

The running time of insertion sort is between Ω(n)

to O(n2)

The running time is bound to O(n2)

Topic: Introduction 49
b) Average Case Analysis

The problem:

Is the key x in the array S of n keys?

Solving strategy:

Sequential search

Topic: Introduction 50
Algorithm:

Procedure search ( n, S, x, location) // n is total # of keys,

S is an array,
x is the key,
location is output
parameter
begin
location ← 1;
while (location ≤ n) and (S[location] ≠ x)
location ++;
if location > n location ← 0;
end;

Topic: Introduction 51
Average Case Analysis:

When ‘x is in the array’, we assume that

1
Probability prob ( x  k )  1  k  n
th

n
1) When x = kth, # of key comparisons = k

1 1 1 1 n
 A(n)   1   2     n   ( k )
n n n n k 1
1 n(n  1) n  1
  
n 2 2

about half array is searched.

Topic: Introduction 52
2) When ‘x may not be in the array’
1
Assume prob (x in array) = p,  prob( x  k )  p 
th

n
prob (x not in array) = 1 – p, and it takes
n comparisons to know that ‘is not in the array’
p n
 A(n)    k  (1  p )  n
n k 1
p n(n  1)
   (1  p)  n
n 2
p p
 n  (1  ) 
2 2
Topic: Introduction 53
 n 1
If p = 1, A(n)  (as calculated in case (1))
2

1 3 1
If p  , A(n)  n  (about ¾ of the array is
2 4 4
searched)

Topic: Introduction 54
 Design of Algorithms

a) Divide-and-Conquer

b) Greedy

c) Dynamic Programming

d) Backtracking & Branch-and-Bound

Topic: Introduction 55
 Some mathematical background

Definition. Logb x = y if b y = x b>1

1) Logb is a strictly increasing function,

if x1 < x2 then Logb x1 < Logb x2 .

2) Logb is a one-to-one function,

if Logb x1 = Logb x2 then x1 = x2 .

3) Logb 1 = 0 b

4) Logb xa = a  Logb x

5) Logb(x1  x2) = Logb x1 + Logb x2

Topic: Introduction 56
6) x1Logb x2 = x2Logb x1
Log b2 x
7) to convert from one base to another, Log b1 x 
Log b2 b1
n
n(n  1)
8) sum of consecutive integers i 
i 1 2
k 1
k
a 1
9) geometric sums

i 0
a i

a 1

10) Suppose f is a continuous, decreasing function, a, b are integers,

b 1 b b

then  f ( x)dx   f (i)   f ( x)dx

a i a a 1

Topic: Introduction 57
11) Suppose f is a continuous, increasing function, a, b are integers,

b b b 1
then
 f ( x)dx   f (i)   f ( x)dx
a 1 i a a

b
1
12)
a xdx  Log e b  Log e a

Note:
Log2 = Lg
Loge = Ln

Topic: Introduction 58