Asymptotic

Asymptotic Notation
Notation

ymp - 1
 Asymptotic Complexity
 Running time of an algorithm as a function of
input size n for large n.
 Expressed using only the highest-order term in
the expression for the exact running time.
 Instead of exact running time, say (n2).
 Describes behavior of function in the limit.
 Written using Asymptotic Notation.

ymp - 2
 Asymptotic Notation
 , O, , o, 
 Defined for functions over the natural numbers.
 Ex: f(n) = (n2).
 Describes how f(n) grows in comparison to n2.
 Define a set of functions; in practice used to compare
two function sizes.
 The notations describe different rate-of-growth
relations between the defining function and the
defined set of functions.

ymp - 3
 -notation
For function g(n), we define (g(n)),
big-Theta of n, as the set:
(g(n)) = {f(n) :
 positive constants c1, c2, and n0,
such that n  n0,
we have 0  c1g(n)  f(n)  c2g(n)

}
Intuitively: Set of all functions that
have the same rate of growth as g(n).

g(n) is an asymptotically tight bound for f(n).

ymp - 4
 -notation
For function g(n), we define (g(n)),
big-Theta of n, as the set:
(g(n)) = {f(n) :
 positive constants c1, c2, and n0,
such that n  n0,
we have 0  c1g(n)  f(n)  c2g(n)

}
Technically, f(n)  (g(n)).
Older usage, f(n) = (g(n)).
I’ll accept either…

f(n) and g(n) are nonnegative, for large n.

ymp - 5
 Example
(g(n)) = {f(n) :  positive constants c1, c2, and n0,
such that n  n0, 0  c1g(n)  f(n)  c2g(n)}

 10n2 - 3n = (n2)
 What constants for n0, c1, and c2 will work?
 Make c1 a little smaller than the leading
coefficient, and c2 a little bigger.
 To compare orders of growth, look at the
leading term.
 Exercise: Prove that n2/2-3n= (n2)
ymp - 6
 Example
(g(n)) = {f(n) :  positive constants c1, c2, and n0,
such that n  n0, 0  c1g(n)  f(n)  c2g(n)}

 Is 3n3  (n4) ??
 How about 22n (2n)??

ymp - 7
 O-notation
For function g(n), we define O(g(n)),
big-O of n, as the set:
O(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
we have 0  f(n)  cg(n) }
Intuitively: Set of all functions
whose rate of growth is the same as
or lower than that of g(n).
g(n) is an asymptotic upper bound for f(n).
f(n) = (g(n))  f(n) = O(g(n)).
(g(n))  O(g(n)).
ymp - 8
  -notation
For function g(n), we define (g(n)),
big-Omega of n, as the set:
(g(n)) = {f(n) :
 positive constants c and n0,
such that n  n0,
we have 0  cg(n)  f(n)}
Intuitively: Set of all functions
whose rate of growth is the same
as or higher than that of g(n).
g(n) is an asymptotic lower bound for f(n).
f(n) = (g(n))  f(n) = (g(n)).
(g(n))  (g(n)).
ymp - 9
 Example
(g(n)) = {f(n) :  positive constants c and n0, such
that n  n0, we have 0  cg(n)  f(n)}

 n = (lg n). Choose c and n0.

ymp - 10
 Relations Between , O, 

ymp - 11
 Relations Between , , O
Theorem
Theorem :: ForFor any
any two
two functions
functions g(n)
g(n) and
and f(n),
f(n),
f(n) == (g(n))
f(n) (g(n)) iff
iff
f(n)
f(n) == O(g(n))
O(g(n)) and f(n) == (g(n)).
and f(n) (g(n)).
 I.e., (g(n)) = O(g(n))  (g(n))

 In practice, asymptotically tight bounds are

obtained from asymptotic upper and lower bounds.

ymp - 12
 Example
 Insertion sort takes (n2) in the worst case, so
sorting (as a problem) is O(n2). Why?

 Any sort algorithm must look at each item, so

sorting is (n).

 In fact, using (e.g.) merge sort, sorting is (n lg n)

in the worst case.

 Later, we will prove that we cannot hope that any

comparison sort to do better in the worst case.

ymp - 13
 Asymptotic Notation in Equations
 Can use asymptotic notation in equations to
replace expressions containing lower-order terms.
 For example,
4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + (n)
= 4n3 + (n2) = (n3). How to interpret?
 In equations, (f(n)) always stands for an
anonymous function g(n)  (f(n))
 In the example above, (n2) stands for
3n2 + 2n + 1.

ymp - 14
 o-notation
For a given function g(n), the set little-o:
o(g(n)) = {f(n):  c > 0,  n0 > 0 such that
 n  n0, we have 0  f(n) < cg(n)}.
f(n) becomes insignificant relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = 0
n

g(n) is an upper bound for f(n) that is not

asymptotically tight.
Observe the difference in this definition from previous
ones. Why?
ymp - 15
  -notation
For a given function g(n), the set little-omega:

(g(n)) = {f(n):  c > 0,  n 0 > 0 such that

 n  n0, we have 0  cg(n) < f(n)}.
f(n) becomes arbitrarily large relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = .
n

g(n) is a lower bound for f(n) that is not

asymptotically tight.

ymp - 16
 Comparison of Functions
fg  ab

f (n) = O(g(n))  a  b
f (n) = (g(n))  a  b
f (n) = (g(n))  a = b
f (n) = o(g(n))  a < b
f (n) = (g(n))  a > b

ymp - 17
 Algorithm Definition

A finite set of statements that guarantees an

optimal solution in finite interval of time

18

ymp - 18
 Good Algorithms?

 Run in less time

 Consume less memory

But computational resources (time

complexity) is usually more important

19

ymp - 19
 Measuring Efficiency

 The efficiency of an algorithm is a measure of the

amount of resources consumed in solving a
problem of size n.
 The resource we are most interested in is time
 We can use the same techniques to analyze the
consumption of other resources, such as memory space.
 It would seem that the most obvious way to
measure the efficiency of an algorithm is to run it
and measure how much processor time is needed

20
 Is it correct ?
ymp - 20
 Factors

Hardware
Operating System
Compiler
Size of input
Nature of Input
Algorithm

21

ymp - 21
 RUNNING TIME OF AN ALGORITHM

 Depends upon
• Input Size
• Nature of Input
 Generally time grows with size of input, so
running time of an algorithm is usually measured
as function of input size.
 Running time is measured in terms of number of
steps/primitive operations performed
 Independent from machine, OS
22

ymp - 22
 Finding running time of an Algorithm / Analyzing an
Algorithm

 Running time is measured by number of

steps/primitive operations performed

 Steps means elementary operation like

 ,+, *,<, =, A[i] etc

 We will measure number of steps taken in

term of size of input
23

ymp - 23
 Simple Example

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}

How should we analyse this?

24

ymp - 24
 Simple Example

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3
25

ymp - 25
 What Dominates in Previous Example?

What about the +3 and 5 in 5N+3?

 As N gets large, the +3 becomes insignificant
 5 is inaccurate, as different operations require varying amounts of time and
also does not have any significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large,
concentrate on the
highest order term:
 Drop lower order terms such as +3
 Drop the constant coefficient of the highest order
term i.e. N 26

ymp - 26
 Asymptotic Complexity

 The 5N+3 time bound is said to "grow

asymptotically" like N

 This gives us an approximation of the

complexity of the algorithm

 Ignores lots of (machine dependent)

details, concentrate on the bigger picture 27

ymp - 27
 COMPARING FUNCTIONS: ASYMPTOTIC
NOTATION

 Big Oh Notation: Upper bound

 Omega Notation: Lower bound

 Theta Notation: Tighter bound

28

ymp - 28
 Big Oh Notation

If f(N) and g(N) are two complexity

functions, we say
f(N) = O(g(N))
(read "f(N) is order g(N)", or "f(N) is big-O
of g(N)")
if there are constants c and N0 such that for N
> N0,
f(N) ≤ c * g(N)
for all sufficiently large N. 29

ymp - 29
 Big Oh Notation [2]

30

ymp - 30
 Big-Oh Notation

 Even though it is correct to say “7n - 3 is

O(n3)”, a better statement is “7n - 3 is O(n)”,
that is, one should make the approximation as
tight as possible

 Simple Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
31
8n2log n + 5n2 + n is O(n2log n)
ymp - 31
 Big Omega Notation
 If we wanted to say “running time is at least…” we
use Ω
 Big Omega notation, Ω, is used to express the
lower bounds on a function.
 If f(n) and g(n) are two complexity functions then
we can say:
f(n) is Ω(g(n)) if there exist positive numbers c and n0 such that 0<=f(n)>=cΩ (n) for all n>=n0

32

ymp - 32
 Big Theta Notation

 If we wish to express tight bounds we use the theta notation, Θ

 f(n) = Θ(g(n)) means that f(n) = O(g(n)) and f(n) = Ω(g(n))

33

ymp - 33
 What does this all mean?

 If f(n) = Θ(g(n)) we say that f(n) and g(n)

grow at the same rate, asymptotically
 If f(n) = O(g(n)) and f(n) ≠ Ω(g(n)), then we
say that f(n) is asymptotically slower
growing than g(n).
 If f(n) = Ω(g(n)) and f(n) ≠ O(g(n)), then we
say that f(n) is asymptotically faster
growing than g(n).
34

ymp - 34
 Which Notation do we use?

 As computer scientist we generally like to

express our algorithms as big O since we
would like to know the upper bounds of
our algorithms.

 If we know the worse case then we can

aim to improve it and/or avoid it.

35

ymp - 35
 Performance Classification
f(n) Classification
1 Constant: run time is fixed, and does not depend upon n. Most instructions are executed once, or
only a few times, regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which solve
large problems by transforming them into smaller problems. Exp : binary Search

n Linear: run time varies directly with n. Typically, a small amount of processing is done on each
element. Exp: Linear Search

n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem
down into smaller sub-problems, solves them independently, then combines solutions. Exp: Merge

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically
the program processes all pairs of input (e.g. in a double nested loop). Exp: Insertion Search

n3 Cubic: when n doubles, runtime increases eightfold. Exp: Matrix

2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force”
solution. Exp: Brute Force.
Note: logn, n, nlogn, n2>> less Input>>Polynomial
n3, 2n>>high input>> non polynomial 36
ymp - 36
 Size does matter[1]

What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076

37

ymp - 37
 Complexity Classes

Time (steps)

38

ymp - 38
 Analyzing Loops[1]
 Any loop has two parts:
 How many iterations are performed?
 How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;

 Loop executes N times (0..N-1)

 4 = O(1) steps per iteration

 Total time is N * O(1) = O(N*1) = O(N) 39

ymp - 39
 Analyzing Nested Loops[1]
 Treat just like a single loop and evaluate each
level of nesting as needed:
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

 Start with outer loop:

 How many iterations? N
 How much time per iteration? Need to evaluate inner loop

 Inner loop uses O(N) time

40
 Total time is N * O(N) = O(N*N) = O(N2)
ymp - 40
 Analyzing Nested Loops[2]

 What if the number of iterations of one loop

depends on the counter of the other?
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

 Analyze inner and outer loop together:

 Number of iterations of the inner loop is: 41

 0 + 1 + 2 + ... + (N-1) = O(N2)

ymp - 41
 How Did We Get This Answer?

 When doing Big-O analysis, we sometimes have

to compute a series like: 1 + 2 + 3 + ... + (n-1) + n

 i.e. Sum of first n numbers. What is the

complexity of this?

 Gauss figured out that the sum of the first n

numbers is always:

42

ymp - 42
 Sequence of Statements

 For a sequence of statements, compute their

complexity functions individually and add them
up

 Total cost is O(n2) + O(n) +O(1) = O(n2)

43

ymp - 43
 Details

 Algorithms can be classified according to

their complexity => O-Notation
 only relevant for large input sizes

 "Measurements" are machine independent

 worst-, average-, best-case analysis

44

ymp - 44
 A Big Warning
Do NOT ignore constants that are not
multipliers:
n3 is O(n2) is FALSE
3n is O(2n) is FALSE

When in doubt, refer to the rigorous

definition of Big-Oh
45

ymp - 45 June 20, 2012

 Examples
 True or false?
1.4+3n is O(n) True
False
2.n+2 logn is O(log n)
False
3.logn+2 is O(1)
True
4.n50 is O(1.1n)

46

ymp - 46 June 20, 2012

 Big Oh: Common Categories
From fastest to slowest
O(1) constant (or O(k) for constant k)
O(log n) logarithmic
O(n) linear
O(n log n) "n log n”
O(n2) quadratic
O(n3) cubic
O(nk) polynomial (where is k is constant)
O(kn) exponential (where constant k > 1)

47

ymp - 47 June 20, 2012

 Comment on Notation

 We say (3n2+17) is in O(n2)

 We may also say/write is as
 (3n2+17) is O(n2)
 (3n2+17) = O(n2)
 (3n2+17) ∈ O(n2)

 But it’s not ‘=‘ as in ‘equality’:

 We would never say O(n2) = (3n2+17)
48

ymp - 48 June 20, 2012

 Putting them in order

(…) < (…) ≤ f(n) ≤ O(…) < o(...)

49

ymp - 49 June 20, 2012

 Properties HomeWork
 Transitivity
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))
f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n))
f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n))

 Reflexivity
f(n) = (f(n))
f(n) = O(f(n))
f(n) = (f(n))

ymp - 50
 Properties HomeWork
 Symmetry
f(n) = (g(n)) iff g(n) = (f(n))

 Complementarity
f(n) = O(g(n)) iff g(n) = (f(n))
f(n) = o(g(n)) iff g(n) = ((f(n))

ymp - 51
 Common Functions

July 14, 2024

ymp - 52
 Monotonicity HomeWork
 f(n) is
 monotonically increasing if m  n  f(m)  f(n).
 monotonically decreasing if m  n  f(m)  f(n).
 strictly increasing if m < n  f(m) < f(n).
 strictly decreasing if m > n  f(m) > f(n).

ymp - 53
 Reading Assignment
 Chapter 1-2 of recommended text book

ymp - 54 Comp 122

 Reference
T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein,
Introduction to Algorithms, 3/e, MIT Press.

ymp - 55 Comp 122