Time and Space Complexity Analysis

Lecture 2
Measuring Efficiency of an
Algorithm
Time Complexity Analysis using
Random Access Machine
(RAM)
By
Dr. Bonface Ngari
By the end of this lesson the learners should
be able to:
1. Describe the Tools for Comparing Algorithms
Objectively
2. Measure the efficiency of an algorithm using RAM
Method
3. Measure the efficiency of an algorithm using
Exact computational Method
4. Differentiate computational analysis and
asymptotic analysis of algorithms
Introduction
How do we measure the Efficiency of an
Algorithm?
1. There are often many approaches (algorithms)

to solve a problem. How do we choose
between them?
2. At the heart of computer program design are
two (sometimes conflicting) goals:
a) To design an algorithm that is easy to
understand, code, debug.
b) To design an algorithm that makes efficient
use of the computer’s resources.
Introduction(cont)
• Goal (1)  To design an algorithm that is easy to

understand, code, debug, is the concern of Software
Engineering.
• Goal (2)  To design an algorithm that makes efficient

use of the computer’s resources, is the concern of
data structures and algorithm analysis.
• When goal (2) is important,

how do we measure an algorithm’s cost?
How to Measure Efficiency?
1. Empirical comparison (run programs)

2. Asymptotic Algorithm Analysis
Critical resources 
Factors affecting running time:
• For most algorithms, running time depends on “size” of the

input.
• Running time is expressed as T(n) for some function T on input
size n.
• Tools for Comparing Algorithms Objectively:
• Exact Computational Complexity Analysis
• Worse-case, Best-case and Average-Case Behaviors
• The Big-Oh Notation
• Worse-case Analysis of Algorithms
• Space Complexity
• Growth Rates of Order Classes
• Why Logarithms are Important in Algorithm Design.
Example: Bubble sort running time
• The bubble step take (n2-n)/2 “steps”
• Assume the bubble sort takes the following number

of steps on specific CPUs:
• Intel Pentium IV CPU: 58*(n2-n)/2
• Motorola CPU: 84.4*(n2-2n)/2
• Intel Pentium V CPU: 44*(n2-n)/2
• Notice that each has an n2 term

• As n increases, the other terms will drop out
7
Bubble sort running time (cont.)
• This leaves us with:

• Intel Pentium IV CPU: 29n2
• Motorola CPU: 42.2n2
• Intel Pentium V CPU: 22n2
• As processors change, the constants will always change

• The exponent on n will not
• Thus, we can’t care about the constants
8
Comparing Algorithms: an Analytical Approach
(Asymptotic Algorithm Analysis)
• How to solve “which algorithm” problems without machines or test data
Primitives
• Primitive operations
• x=4 assignment
• ... x + 5 ... arithmetic
• if (x < y) ... comparison
• x[4] index an array
• *x dereference (C)
• x.foo( ) calling a method
• Others
• new/malloc memory usage
How many Bestoos?
for (j = 1; j <= N; ++j) {
for (k = 1; k <= M; ++k) {
besto( );
}
}
N M
ΣΣ 1 = NM
j = 1k =
1
How many Bestoos?
for (j = 1; j <= N; ++j) {
for (k = 1; k <= j; ++k) {
besto( );
}
}
N j N N (N + 1)
ΣΣ 1 = Σ j =
2
j = 1k = j=1
1
How many bestoos?
for (j = 0; j < N; ++j) {
for (k = 0; k < j; ++k) {
besto( ); N(N + 1)/2
}
}
for (j = 0; j < N; ++j) {
for (k = 0; k < M; ++k) {
besto( );
} NM
}
How many Bestoos?
void foo(int N) { T(0) = T(1) = T(2) = 1
if(N <= 2) T(n) = 1 + T(n/2) if n > 2
return;
besto(N / 2); T(n) = 1 + (1 + T(n/4))
} = 2 + T(n/4)
= 2 + (1 + T(n/8))
= 3 + T(n/8)
= 3 + (1 + T(n/16))
= 4 + T(n/16)
…
≈ log2 n
The trick
ank + bnk-1 + … + yn + z
ank
nk
Complexity Analysis
Tools for Comparing Algorithms Objectively
• Computational and Asymptotic Complexity

• The “Big-Oh” Notation
Complexity Analysis

Complexity Analysis:
Computational and Asymptotic Complexity
Algorithm #1
Problem solve Algorithm #2  Efficiency
…
Algorithm #n
Grow
proportionally with

 
the amount of data

Algorithm #1
Algorithm #2  Efficiency
…
Algorithm #n
Compare?
Computational Complexity :
Measure the degree of difficulty of an algorithm

How much effort ?

Computational
Complexity : How costly ?
Various ways of measuring
Our concern  efficiency criteria of time an
Focus
How can we measure time and space?
Exact Computational Complexity
Analysis
• Exact Time Complexity (unit: seconds)
• measure of an algorithm’s total time taken.
• Example: Two algorithms A and B, solving a problem instance of P, can take 3 and
100 millisecs.
• Useful
• as an initial gauge of performance.
• for profiling and finding bottlenecks.
• Therefore, A and B would have Time Complexities of:
• 100 * 4 (ns) and
• 1000 * 4 (ns)
• We had assumed
• that each work step consumed the same number of cycles.
• input data model is the same for both algorithms.
• features of the execution platform were the same.
Exact Computational Complexity is the total discrete

amount of work done by the algorithm.
Exact Time Complexity is some constant multiple of

the Exact Computational Complexity.
Exact Space Complexity
(unit: bytes)
•Measure of an algorithm’s total memory
requirements during runtime.
• Data structures,
• Temporary variables.
Exact Time Complexity

? NO!
Real-time units (sec, msec,…)
How to evaluate
an algorithm’s
efficiency? Logical units (work) ? YES!
Remember:
Exact Computational Complexity is the total discrete
amount of work done by the algorithm.
Exact Time Complexity is some constant multiple of

the Exact Computational Complexity.
Exact Time Complexity

NO!
Real-time units (sec, msec,…)
How to evaluate
an algorithm’s
efficiency? Logical units (work) YES!
Array or file of size n X time t to process the data
t1  cn1 t1  log 2 n1
n2  5n1  t 2  5t1 n2  2n1  t 2  log 2 2n1 
Function expressing relationship between n and t
• Complex
• Important (large bodies of data)
• Basic recipe  Eliminate non-contributing terms
Resulting function  APPROXIMATE MEASURE
Asymptotic Complexity
Complexity Analysis: Example
f n   n 2  100n  log 10 n  1000

Growth Rates of the Time Complexities of Algorithms
with respect to increasing problem sizes.
On a machine running at 1 GHz (assuming 1

cycle per work unit).
For small values of n:
What conclusions can we draw?

Which algorithms remain useful as problem size increases?
Growth Rates of the Time Complexities of Algorithms
with respect to increasing problem sizes.
For larger values of n:
What conclusions can we draw?

Which algorithms remain useful as problem size increases?
Tractable  log n  n  n log n  n 2
n
Exponential  2
Factorial  n!
Intractable!
Complexity Analysis

Formal Definition:
Big-O Notation
Let f and g be two functions such that
f n : N  R and g n : N  R
f  the numerical function that represents the

best, average, and worst behavior (it can be
any of the three)
g  the function for you to discover

Formal Definition:
Big-O Notation
Let f and g be two functions such that
f n : N  R and g n : N  R

if there exists positive constants c and n0 such th
f n   cg n , for all n  n0
or if
f n 
0  lim c
n  g n 
then we write
f n  Og n
So g(n) is an asymptotic upper-bound for f(n) as n increases
(g(n) bounds f(n) from above)
Informal Definition:
Big-O Notation
f(n) is O(g(n))
if f grows at most as fast as g.
Grow refers to the value of f(n) as n increases
cg(n) is an approximation to f(n), bounding from above

“Big-Oh” Examples
7n  2 O(n)
By definition, we need to find:
• a real constant c > 0

• an integer constant n0 >= 1
Such that 7n - 2 <= cn,
for every integer n >= n0

“Big-Oh” Examples
7n  2 O(n)
Possible choice is:
• c=7
• n0 = 1
7n - 2 <= 7n, for n >= 1

20n  10n log n  5
3
3
O(n )
20n 10n log n  5  35n , for n  1
3 3
3 log n  log log n
O(log n)
3 log n  log log n  4 log n, for n  2
100
2
O(1)
100
2  2 .1, for n  1
100
5n
O(1 n)
5 n  5(1 n), for n  1
Mergesort and Quicksort
Sorting algorithms
• Insertion, selection and bubble sort have quadratic worst-case

performance
• The faster comparison based algorithm ?
O(nlogn)
• Mergesort and Quicksort

Merge Sort
• Apply divide-and-conquer to sorting problem

• Problem: Given n elements, sort elements into non-
decreasing order
• Divide-and-Conquer:
• If n=1 terminate (every one-element list is already sorted)
• If n>1, partition elements into two or more sub-
collections; sort each; combine into a single sorted list
• How do we partition?
Partitioning - Choice 1
• First n-1 elements into set A, last element set B
• Sort A using this partitioning scheme recursively
• B already sorted
• Combine A and B using method Insert() (= insertion
into sorted array)
• Leads to recursive version of InsertionSort()
• Number of comparisons: O(n2)
• Best case = n-1
• Worst case = n
n(n  1)
c i 
i 2 2
• Put element with largest key in B, remaining elements
in A
• Sort A recursively
• To combine sorted A and B, append B to sorted A
• Use Max() to find largest element  recursive
SelectionSort()
• Use bubbling process to find and move largest element to
right-most position  recursive BubbleSort()
• All O(n2)
• Let’s try to achieve balanced partitioning

• A gets n/2 elements, B gets rest half
• Sort A and B recursively
• Combine sorted A and B using a process called merge, which
combines two sorted lists into one
• How? We will see soon
Example
• Partition into lists of size n/2
[10, 4, 6, 3, 8, 2, 5, 7]
[10, 4, 6, 3] [8, 2, 5, 7]
[10, 4] [6, 3] [8, 2] [5, 7]
[4] [10] [3][6] [2][8] [5][7]

Example Cont’d
• Merge
[2, 3, 4, 5, 6, 7, 8, 10 ]
[3, 4, 6, 10] [2, 5, 7, 8]
[4, 10] [3, 6] [2, 8] [5, 7]
[4] [10] [3][6] [2][8] [5][7]

Static Method mergeSort()
Public static void mergeSort(Comparable []a, int left,
int right)
{
// sort a[left:right]
if (left < right)
{// at least two elements
int mid = (left+right)/2; //midpoint
mergeSort(a, left, mid);
mergeSort(a, mid + 1, right);
merge(a, b, left, mid, right); //merge from a to b
copy(b, a, left, right); //copy result back to a
}
}
Merge Function
Evaluation
• Recurrence equation:
• Assume n is a power of 2
c1 if n=1
T(n) =
2T(n/2) + c2n if n>1, n=2k
Solution
By Substitution:
T(n) = 2T(n/2) + c2n
T(n/2) = 2T(n/4) + c2n/2
T(n) = 4T(n/4) + 2 c2n

T(n) = 8T(n/8) + 3 c2n
T(n) = 2iT(n/2i) + ic2n
Assuming n = 2k, expansion halts when we get T(1) on right side;

this happens when i=k T(n) = 2kT(1) + kc2n
Since 2k=n, we know k=logn; since T(1) = c1, we get
T(n) = c1n + c2nlogn;
thus an upper bound for TmergeSort(n) is O(nlogn)
Quicksort Algorithm
Given an array of n elements (e.g., integers):

• If array only contains one element, return
• Else
• pick one element to use as pivot.
• Partition elements into two sub-arrays:
• Elements less than or equal to pivot
• Elements greater than pivot
• Quicksort two sub-arrays
• Return results
Example
We are given array of n integers to sort:
40 20 10 80 60 50 7 30 100
Pick Pivot Element
There are a number of ways to pick the pivot element. In this
example, we will use the first element in the array:
40 20 10 80 60 50 7 30 100
Partitioning Array
Given a pivot, partition the elements of the array

such that the resulting array consists of:
1. One sub-array that contains elements >= pivot
2. Another sub-array that contains elements < pivot
The sub-arrays are stored in the original data array.
Partitioning loops through, swapping elements

below/above pivot.
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
too_big_index too_small_index
1. While data[too_big_index] <= data[pivot]
++too_big_index
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
4. While too_small_index > too_big_index, go to 1.
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
5. Swap data[too_small_index] and data[pivot_index]
pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 4 7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
Partition Result
7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
<= data[pivot] > data[pivot]

Recursion: Quicksort Sub-arrays
7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

Quicksort Analysis
• Assume that keys are random, uniformly distributed.

• What is best case running time?
Quicksort Analysis

• Recursion:
1. Partition splits array in two sub-arrays of size n/2
2. Quicksort each sub-array
Quicksort Analysis

• Recursion:
• Depth of recursion tree?
Quicksort Analysis

• Recursion:
• Depth of recursion tree? O(log2n)
Quicksort Analysis

• Recursion:
• Number of accesses in partition?
Quicksort Analysis

• Recursion:
• Number of accesses in partition? O(n)
Quicksort Analysis

• Best case running time: O(n log2n)
Quicksort Analysis

• Worst case running time?
Quicksort: Worst Case
• Assume first element is chosen as pivot.

• Assume we get array that is already in order:
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]
++too_big_index
--too_small_index
pivot_index = 0 2 4 10 12 13 50 57 63 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

Quicksort Analysis
• Assume that keys are random, uniformly

distributed.
• Recursion:
1. Partition splits array in two sub-arrays:
• one sub-array of size 0
• the other sub-array of size n-1
• Depth of recursion tree?
Quicksort Analysis

distributed.
• Recursion:
• Depth of recursion tree? O(n)
Quicksort Analysis

distributed.
• Recursion:
• Number of accesses per partition?
Quicksort Analysis

distributed.
• Recursion:
• Number of accesses per partition? O(n)
Quicksort Analysis

distributed.
• Worst case running time: O(n2)!!!
Quicksort Analysis

distributed.
• Worst case running time: O(n2)!!!
• What can we do to avoid worst case?
Improved Pivot Selection
Pick median value of three elements from data array:

data[0], data[n/2], and data[n-1].
Use this median value as pivot.

Improving Performance of Quicksort
• Improved selection of pivot.

• For sub-arrays of size 3 or less, apply brute force search:
• Sub-array of size 1: trivial
• Sub-array of size 2:
• if(data[first] > data[second]) swap them
• Sub-array of size 3: left as an exercise.

Time and Space Complexity Analysis

Uploaded by

Copyright:

Available Formats

You might also like

Time and Space Complexity Analysis

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Time and Space Complexity Analysis

Uploaded by

Copyright:

Available Formats

Lecture 2

1. There are often many approaches (algorithms)

• Goal (1)  To design an algorithm that is easy to

• Goal (2)  To design an algorithm that makes efficient

• When goal (2) is important,

1. Empirical comparison (run programs)

• For most algorithms, running time depends on “size” of the

• Assume the bubble sort takes the following number

• Notice that each has an n2 term

• This leaves us with:

• As processors change, the constants will always change

• Thus, we can’t care about the constants

Tools for Comparing Algorithms Objectively

• Computational and Asymptotic Complexity

Tools for Comparing Algorithms Objectively

• Computational and Asymptotic Complexity

Measure the degree of difficulty of an algorithm

How much effort ?

Various ways of measuring

Our concern  efficiency criteria of time an

Exact Computational Complexity is the total discrete

Exact Time Complexity is some constant multiple of

Exact Time Complexity

Exact Time Complexity is some constant multiple of

Exact Time Complexity

Array or file of size n X time t to process the data

Function expressing relationship between n and t

Resulting function  APPROXIMATE MEASURE

f n   n 2  100n  log 10 n  1000

On a machine running at 1 GHz (assuming 1

What conclusions can we draw?

For larger values of n:

What conclusions can we draw?

Tools for Comparing Algorithms Objectively

• Computational and Asymptotic Complexity

f n : N  R and g n : N  R

f  the numerical function that represents the

g  the function for you to discover

f n : N  R and g n : N  R

Grow refers to the value of f(n) as n increases

cg(n) is an approximation to f(n), bounding from above

• a real constant c > 0

Such that 7n - 2 <= cn,

for every integer n >= n0

7n - 2 <= 7n, for n >= 1

• Insertion, selection and bubble sort have quadratic worst-case

• Mergesort and Quicksort

• Apply divide-and-conquer to sorting problem

• Let’s try to achieve balanced partitioning

[10, 4] [6, 3] [8, 2] [5, 7]

[4] [10] [3][6] [2][8] [5][7]

[3, 4, 6, 10] [2, 5, 7, 8]

[4, 10] [3, 6] [2, 8] [5, 7]

[4] [10] [3][6] [2][8] [5][7]

T(n) = 4T(n/4) + 2 c2n

T(n) = 2iT(n/2i) + ic2n

Assuming n = 2k, expansion halts when we get T(1) on right side;