Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Fundamentals of the Analysis of
Algorithm Efficiency
Keun-Liang Sue
Information Management
NCU
2023/3/9 1
Analysis Framework
Time efficiency
How fast an algorithm in question runs.
Primarily concentrated.
Space efficiency
The extra space the algorithm required.
2023/3/9 Algorithms by KLSue/NCU 2

Measuring an Input’s Size
Investigate an algorithm’s efficiency

A function of the algorithm’s input size.
Measuring size of inputs
Not always equal to the number of input data.
e.g., checking whether an integer n is prime.
the number b of bits in n’s binary representation
b  log 2 n   1
Units for Measuring Running Time
Count the number of times

Each of the algorithm’s operation is executed.
Difficult and unnecessary.
Identify the most important operation
Basic operation
Operation contributing most of total running time.
Compute number of times basic operation is
executed.

 Define :
cop：Execution time of an algorithm’s basic
operation.
C(n)：the number of times this operation needs.
 Estimate running time T(n) by formula
T (n)  cop C (n)

What if double input size

Assume C ( n) 
1
2
n(n  1) ,how much longer if T(2n)
For all but very small values of n:
1 1 1 1
C ( n)  n(n  1)  n 2  n  n 2
2 2 2 2
Therefore:
1 2
T (2n) cop C (2n) 2 (2n)
  1 2 4
T ( n) cop C (n) 2
n
Note
Answer it without knowing the value of cop
Multiplicative constant, ½, for C(n)

It was also cancelled out.
So, concentrates on order of growth.
Orders of Growth

Orders of Growth
logarithmic function(log2n).
Function growing the slowest among these.
Algorithm with logarithmic basic-operation count

run instantaneously on inputs of all realistic sizes.
Orders of Growth
Why log2n? Can be log n?
Sure!!
Can omit a logarithm’s base and write simply logn.
See the formula
log a n  log a b log b n

logab is multiplicative constant, can be ignored.
• Q: Why constant can be ignored?

Orders of Growth
Bad functions
Exponential function 2n and factorial function n!
Grow so fast
The value become large even for small value
Both referred to as “exponential-growth functions”
Remember :
• “Algorithms that require an exponential number of
operations are practical for solving only problems of
very small sizes”
Worst-Case, Best-Case, and Average-

Case Efficiencies

Case Efficiencies
Worst-Case Efficiencies
Definition
Efficiency of the worst-case input of size n
An input the algorithm runs the longest
• among all possible inputs
The way to determine it
Analyze the algorithm
• see what kind of inputs yield the largest value of C(n)
Compute this worst-case value Cworst(n)

Case Efficiencies
Benefits from the analysis
Provide the upper bound of running time
Guarantee that running time will not exceed Cworst(n)

• For any instance of size n
Sequential search, Cworst(n)=n

Case Efficiencies
Best-Case Efficiencies
Definition
Efficiency for the best-case input of size n
Runs the fastest among all possible input of that size
Analyze the best-case efficiency
Determine the kind of inputs
• for which C(n) will be the smallest
Ascertain the value of C(n)
• on the most convenient inputs
• In our example, Cbest(n)=1

Case Efficiencies
Average-Case Efficiencies
Assumptions for Analyze our example
prob. of a successful search is p
(0  p  1)
prob. of first match occurring in ith position of the list
is the same for every i.

Case Efficiencies
In case of successful search

prob. of first match occurring in ith position is p/n
The number of comparison is i
In case of unsuccessful search

the number of comparisons is n
prob. of unsuccessful search is (1-p).

Case Efficiencies
Therefore,
p p p p
Cavg (n)  [1  2     i     n  ]  n  (1  p )
n n n n
p
 [12i n]n(1 p)
n
p n(n1) p(n1)
 n(1 p)  n(1 p)
n 2 2
If p=1, Cavg(n)=(n+1)/2

If p=0, Cavg(n)=n

Case Efficiencies
Note
Average-case is more difficult than two others
Q: Need average-case information?
• Yes
• Many algorithm, average is much better than worst
• Without average-case analysis,
 Computer scientists miss important algorithms
Average-case is not equal to
• Average of best-case and worst case efficiencies

Case Efficiencies
Amortized efficiency (分期攤還)
Definition
It applies to a sequence of operations
• Not to a single run of an algorithm
• performed on the same data structure.
Phenomenon
A single operation can be expensive
• In some situations
Total time for entire n such operations
• always better than worst-case efficiency * n

Asymptotic Notations and Basic Efficiency
Classes
Introduction
To compare and rank such orders of growth
Three Notations:
Ο(big oh),Ω(big omega),Θ(big theta).
Terms:
t(n) and g(n):
• Nonnegative functions defined on natural numbers
t(n): an algorithm’s running time.
g(n): simple function to compare the count with.

Classes
Informal Introduction
Ο(g(n)):
Set of all functions with a smaller or same order of
growth as g(n)
Within a constant multiple, as n goes to infinity
For examples:
1
n  O(n 2 ), 100n  5  O(n 2 ), n(n  1)  O(n 2 )
2
n 3  O(n 2 ), 0.00001n 3  O(n 2 ), n 4  n  1  O(n 2 )
Classes
Ω(g(n)):
set of all functions with a larger or same order of
growth as g(n)
Within a constant multiple, as n goes to infinity.
For examples:
1
n 3  (n 2 ), n(n  1)  (n 2 ), but 100n  5  (n 2 )
2

Classes
Θ(g(n)):
set of all functions have the same order of growth as
g(n)
Within a constant multiple, as n goes to infinity.
Every an 2  bn  c with a  0 is in Θ(n2).

Classes
 Ο-notation
Definition
A function t(n) is said to be in Ο(g(n)), denoted
t (n)  O( g (n))
• if t(n) is bounded above by some constant multiple of g(n)
for all large n.
i.e. exist positive constant c and nonnegative integer
n0 such that
t (n)  cg (n) for all n  n0


Classes
 Ο-notation
For example:
100n  5  O(n 2 )
Indeed,
100n  5  100n  n (for all n  5)  101n  101n 2
Take c =101 and n0=5

Classes

Classes
Ω-notation
Definition
A function t(n) is said to be in Ω(g(n)), denoted
t (n)  ( g (n))
• if t(n) is bounded below by some positive constant
multiple of g(n) for all large n.
i.e. exist positive constant c and nonnegative integer
n0 such that
t (n)  cg (n) for all n  n0
Classes
Ω-notation
example:
Proof that n 3  (n 2 ) :
n 3  n 2 for all n  0
i,e., select c =1 and n0=0

Classes

Classes
Θ-notation
Definition
A function t(n) is said to be in Θ(g(n)), denoted
t (n)  ( g (n))
• if t(n) is bounded both above and below by some positive
constant multiples of g(n) for all large n.
i,e., if there exist positive constant c1 ,c2 and nonnegative
integer n0 such that
c2 g (n)  t (n)  c1 g (n) for all n  n0

Classes
Θ-notation
example:
1
n(n  1)  (n 2 )
2
First, prove the upper bound:
1 1 1 1
n(n  1)  n 2  n  n 2 for all n  0
2 2 2 2

Classes
Θ-notation
Second, prove the lower bound:
1 1 1 1 1 1 1
n(n  1)  n 2  n  n 2  n n (for all n  2)  n 2
2 2 2 2 2 2 4
Select 1 1
c2  , c1  and n0  2
4 2

Classes
Θ-notation

Useful Property Involving the Asymptotic
Notations
Theorem:
If t1 (n)  O( g1 (n)) and t 2 (n)  O( g 2 (n)), then
t1 (n)  t 2 (n)  Omaxg1 (n), g 2 (n)
Proof:
Since t1 (n)  O ( g1 (n)),

t1 (n)  c1 g1 (n) for all n  n1
Since t 2 (n)  O( g 2 (n)),
t 2 ( n )  c2 g 2 ( n ) for all n  n2

Notations
Denote c3=max{c1,c2} and consider n >= max{n1,n2}

So,
t1 (n)  t 2 (n)  c1 g1 (n)  c2 g 2 (n)

 c3 g1 (n)  c3 g 2 (n)  c3 g1 (n)  g 2 (n)
 c3 2 maxg1 (n), g 2 (n)

Notations
Hence,
t1 (n)  t 2 (n)  Omaxg1 (n), g 2 (n)
With constant c=2c3=2max{c1,c2}
and n0=max{n1,n2}
The theorem implies
algorithm’s overall efficiency will be determined
by the part with a larger order of growth.
i.e., by its least efficiency part.
Using Limits for Computing Orders of

Growth
Compare orders of growth of two functions
Compute the limit of ratio of two functions
Three principal cases:
0 implies that t (n) has a smaller order of growth than g (n)
t ( n) 
lim  c implies that t (n) has the same order of growth as g (n)
n  g ( n)

 implies that t (n) has a larger order of growth than g (n).
Question:
First two case means ?
Last two case means ?
The second case means ?
Classes
Useful technique
L’Hôpital’s rule:
t ( n) t ' ( n)
lim  lim
n  g ( n) n  g ' ( n)
Stirling’s formula:
n
n
n! 2n   for large values of n.
e

Classes
Example: compare n(n-1)/2 and n2
1
n ( n 1) 1 n2  n 1 1 1
lim 2
 lim 2  lim(1  ) 
n  n2 2 n  n 2 n  n 2
The limit is equal to a positive constant

The functions have the same order of growth.
Symbolically,
1
n(n  1)  (n 2 )
2

Classes
Example: compare log 2 n and n
1
(log 2 e)
log 2 n (log 2 n) ' n  2 log e lim n  0
lim  lim  lim 2
n  n n  ( n)' n  1 n  n
2 n

Classes
Example: compare n! and 2n
n
2 n ( ) n
n! e nn n n
lim n  lim n
 lim 2 n n n
 lim 2 n ( ) 
n  2 n  2 n  2 e n  2e

Mathematical Analysis of Nonrecursive

Algorithms
Q: Finding value of largest element in a list

Algorithms
Who is basic operation
Two operations
Comparison A[i]>maxval
Assignment maxval←A[i]
Which is the basic operation?
Answer: the comparison A[i]>maxval
Define C(n):
the number of times this comparison is executed.

Algorithms
The sum for C(n):

n 1
C ( n)   1
i 1
Thus,
n 1
C ( n )   1  n  1  ( n )
i 1

General Plan for Analyzing Efficiency of
Nonrecursive Algorithms
Decide a parameter indicating an input’s size.
Identify the algorithm’s basic operation
As a rule, it is in the innermost loop
Check whether NBO depends on size of input.
If also depends on other property,
• The worst-case, average-case and best-case must
be investigated
Note:
• NBO means the number of times the basic
operation is executed
General Plan for Analyzing Efficiency of

Nonrecursive Algorithms
Set up a sum expressing C(n)
Using standard formulas and rules of sum

manipulation
either Find the closed-form formula
or, at least, Establish order of growth

Element uniqueness problem
Whether all elements in array are distinct?

Basic operation:
the comparison of two elements.
Define Cworst(n):
the largest number of comparison among all arrays of
size n
Two kind of worst-case
Array with no equal elements
Last two elements are the only pair of equal elements

Accordingly,
n  2 n 1 n2 n2
Cworst(n)   1  [(n  1)  (i  1)  1]   (n  1  i)
i 0 j i 1 i 0 i 0
n2 n2 n2
(n  2)(n  1)
  (n  1)   i  (n  1)1 
i 0 i 0 i 0 2
(n  2)(n  1) (n  1)n 1 2
 (n  1) 2    n  (n 2 )
2 2 2
Multiple square matrices of order n

Algorithm Efficiency
 Find the time efficiency of the algorithm

Basic operation:
the algorithm’s innermost loop.
M(n):
sum for the total number of multiplication executed.

Only one multiplication on each basic operation.

Governed by k ranging from 0 to n-1.
The number of multiplications is :

for each pair of specific i and j
n 1
1
k 0
Thus, the total number of multiplications M(n):
n 1 n 1 n 1 n 1 n 1 n 1
M (n)  1   n   n 2  n 3
i 0 j 0 k 0 i 0 j 0 i 0

Finds number of binary digits in binary
representation of a decimal integer

The most frequently executed operation:
The comparison n > 1
The comparison will be executed

Repetitions of the loop’s body + 1 times.
Thus, the comparison n > 1 will be executed
log 2 n  1 times
Mathematical Analysis of Recursive
Algorithms
Compute factorial function

Algorithms
Computing F(n)=n!
for an arbitrary non-negative integer n.
n! 1  (n  1)  n  (n  1)!n for n  1

Basic operation:
multiplication
the number of executions is denoted M(n)

Algorithms
F(n)=F(n-1)*n for n>0
M(n) must satisfy the equality:
M (n)  M (n  1)  1 for n  0
to compute F(n -1) to multiply F(n -1) by n
M ( 0)  0

Algorithms
Method of backward substitutions:
M (n)  M (n  1)  1 substitute M (n  1)  M (n  2)  1
 [ M (n  2)  1]  1  M (n  2)  2 substitute M (n  2)  M (n  3)  1
 [ M (n  3)  1]  2  M (n  3)  3
Thus,
M (n)  M (n  1)  1    M (n  i )  i    M (n  n)  n  n.

A General Plan for Analyzing Efficiency of
Recursive Algorithms
Decide a parameter indicating input’s size
Identify the algorithm’s basic operation
Check whether NBO vary on different inputs
of the same size
If also depends on other property
• The worst-case, average-case and best-case
must be investigated
Note:
• NBO means the number of times the basic
operation is executed
A General Plan for Analyzing Efficiency of

Recursive Algorithms
Set up a recurrence relation
for number of times basic operation is executed
with appropriate initial condition
Solve the recurrence

or at least ascertain order of growth of its solution

Tower of Hanoi
Definition
What is Tower of Hanoi ?
Analysis
Input size: the number of disks n
Basic operation: moving one disk
The number of moves: M(n)
Tower of Hanoi

Tower of Hanoi
Generalize the problem
General Case
Base Case
General Case
Tower of Hanoi
Thus,
M (n)  M (n  1)  1  M (n  1) for n  1
Recurrence relation:
M (n)  2 M (n  1)  1 for n  1
M (1)  1

Tower of Hanoi
Solve this recurrence by backward substitutions:
M (n)  2M (n  1)  1 sub. M (n  1)  2M (n  2)  1
 2[2M (n  2)  1]  1  2 2 M (n  2)  2  1 sub. M (n  2)  2 M (n  3)  1
 2 2 [2M (n  3)  1]  2  1  23 M (n  3)  2 2  2  1
After i substitutions:
M (n)  2i M (n  i )  2i 1  2i  2    2  1  2i M (n  i )  2i  1
Tower of Hanoi
Because initial condition is n=1
We let i = n-1
So,
M (n)  2 n 1 M (n  (n  1))  2 n 1  1
 2 n 1 M (1)  2 n 1  1  2 n 1  2 n 1  1  2 n  1

Tower of Hanoi
Tree of tower of Hanoi puzzle

Useful for analysis purpose.
Nodes correspond to recursive calls.
Label them with the value of the parameter
Counting the number of nodes in the tree:
n 1
C (n)   2l (where l is the level in the tree above)  2 n  1
i 0

Recursive version of previous discussion

Algorithms
Analysis
Recurrence:
A(n)  An / 2  1 for n  1
A(1)  0
By smoothness rule, n=2k, so:
A(2 k )  A(2 k 1 )  1 for k  0

A(2 0 )  0

Algorithms
Using backward substitution:
A(2 k )  A(2 k 1 )  1 substitute A(2 k 1 )  A(2 k  2 )  1

 [ A(2 k  2 )  1]  1  A(2 k  2 )  2 substitute A(2 k  2 )  A(2 k 3 )  1
 [ A(2 k 3 )  1]  2  A(2 k 3 )  3

 A(2 k i )  i

 A(2 k  k )  k

Algorithms
Thus,
A(2 k )  A(1)  k  k , and n  2 k  k  log 2 n

A(n)  log 2 n  (log n)
In fact, the exact solution is
A(n)  log 2 n 

Fibonacci Numbers
Fibonacci numbers:
0,1,1,2,3,5,8,13,21,34,…
Can be defined by simple recurrence:
F (n)  F (n  1)  F (n  2) for n  1
F (0)  0, F (1)  1
Explicit Formula for the nth Fibonacci

Number
Theorem
Homogeneous second-order linear recurrence
with constant coefficients:
aX (n)  bX (n  1)  cX (n  2)  0
a, b and c: coefficients of the recurrence
X(n): unknown sequence to be found

Number
Characteristic equation for recurrence:
ar 2  br  c  0
Apply this theorem to the case of Fibonacci
numbers:
F (n)  F (n  1)  F (n  2)  0

Number
Its characteristic equation:
r 2  r 1  0
With the roots:
1  1  4(1) 1  5
r1, 2  
2 2

Number
Use the formula of Theorem1 in Appendix B:
n n
1 5  1 5 
F (n)      
 


 2   2 
Thus,
0 0
1 5  1 5 
F (0)      
 
 0
   0
 2   2 
1 1 1 5  1 5 
1 5  1 5        1
F (1)      
  2 
 1 
2   2 
 2     

Number
We get
  1 / 5 and   -1/ 5
 thus,
n n
1 1 5  1 1 5  1  n n 
F ( n)            
5  2  5  2  5  
1 n
F ( n)   rounded to the nearest integer.
5
“The impact of second term can be obtained by

rounding off the value of first term to nearest integer”

Algorithms for Computing Fibonacci
Numbers

Numbers

Numbers
Analysis
Basic operation: addition
A(n): the number of additions

performed by the algorithm in computing F(n)
A(n)  A(n  1)  A(n  2)  1 for n  1,

A(0)  0, A(1)  0.

Numbers
Inhomogeneous recurrences
Similar to homogeneous recurrence
• But its right-hand side ≠0.
Reduce our inhomogeneous recurrence

• to a homogeneous one by rewriting it :
A(n)  1  A(n  1)  1  A(n  2)  1  0

Numbers
Substituting B(n)=A(n)+1:
B (n)  B (n  1)  B (n  2)  0
B (0)  1, B (1)  1.
The recurrence can be solved:
B(n)  F (n  1)
1  n 1
A(n)  B(n)  1  F (n  1)  1  (   )  1
n 1
5
Hence,
A(n)  ( n )

Numbers
Non-recursive version
Only n-1 additions
Extra array for storing elements can be avoided

Empirical Analysis of Algorithms
Some algorithm is difficult to analyze

The alternative is empirical analysis
Steps for the analysis
Understand the experiment’s purpose.
Decide on efficiency metric M and measure unit
Decide on characteristics of the input sample.

Prepare a program implementing the algorithm.
Generate a sample of inputs.
Run the algorithm on the sample’s inputs
Record the data observed.
Analyze the data obtained.

Random Number Generator
Random Number Generator
recommended parameters
seed: set to the current date and time
m: 2w
w: the computer’s word size
a: integer between 0.01m and 0.99m
b: can be chosen as 1

Scatterplot
Empirical data from experiment must be recorded
Then, presented for an analysis
Data Can be presented in a graph
Called scatterplot
Observation of scatterplot
Help to ascertain the efficiency class of a algorithm
Typical scatterplots
Concave shape: for a logarithmic algorithm.
Straight line: for a linear algorithm.
Convex shape: for
(n log n) or (n 2 ) or (n 3 )

Concave shape
Straight line

Convex shape
Algorithm Visualization
Another way to study algorithm

Use graph elements
Point, line segment, 3-dimension bar
Represent interesting events
principal variations of algorithm visualization
Static algorithm visualization.
Shows an algorithm’s progress.
Through a series of still images.

Algorithm Visualization
Dynamic algorithm visualization
Also called algorithm animation.
A movie-like presentation of an algorithm’s operation.




Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Uploaded by

Copyright:

Available Formats

You might also like

Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Uploaded by

Copyright:

Available Formats

Fundamentals of the Analysis of

2023/3/9 Algorithms by KLSue/NCU 2

Investigate an algorithm’s efficiency

Units for Measuring Running Time

Count the number of times

2023/3/9 Algorithms by KLSue/NCU 4

C(n)：the number of times this operation needs.

 Estimate running time T(n) by formula

T (n)  cop C (n)

Units for Measuring Running Time

What if double input size

Multiplicative constant, ½, for C(n)

So, concentrates on order of growth.

2023/3/9 Algorithms by KLSue/NCU 7

2023/3/9 Algorithms by KLSue/NCU 8

Algorithm with logarithmic basic-operation count

2023/3/9 Algorithms by KLSue/NCU 9

log a n  log a b log b n

2023/3/9 Algorithms by KLSue/NCU 10

2023/3/9 Algorithms by KLSue/NCU 11

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 12

2023/3/9 Algorithms by KLSue/NCU 13

Worst-Case, Best-Case, and Average-

Provide the upper bound of running time

Guarantee that running time will not exceed Cworst(n)

Sequential search, Cworst(n)=n

2023/3/9 Algorithms by KLSue/NCU 14

2023/3/9 Algorithms by KLSue/NCU 15

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 16

In case of successful search

In case of unsuccessful search

2023/3/9 Algorithms by KLSue/NCU 17

Worst-Case, Best-Case, and Average-

If p=1, Cavg(n)=(n+1)/2

2023/3/9 Algorithms by KLSue/NCU 18

2023/3/9 Algorithms by KLSue/NCU 19

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 20

2023/3/9 Algorithms by KLSue/NCU 21

Asymptotic Notations and Basic Efficiency

Within a constant multiple, as n goes to infinity

Within a constant multiple, as n goes to infinity.

Asymptotic Notations and Basic Efficiency

Within a constant multiple, as n goes to infinity.

Every an 2  bn  c with a  0 is in Θ(n2).

2023/3/9 Algorithms by KLSue/NCU 24

t (n)  cg (n) for all n  n0

Asymptotic Notations and Basic Efficiency

100n  5  100n  n (for all n  5)  101n  101n 2

Take c =101 and n0=5

2023/3/9 Algorithms by KLSue/NCU 26

2023/3/9 Algorithms by KLSue/NCU 27

Asymptotic Notations and Basic Efficiency

2023/3/9 Algorithms by KLSue/NCU 29

Asymptotic Notations and Basic Efficiency