The Growth of Functions

Rosen 2.2
Basic Rules of Logarithms
log
z
(xy)
log
z
(x/y)
log
z
(x
y
)
If x = y
If x < y
log
z
(-|x|) is undefined
= log
z
(x) + log
z
(y)
= log
z
(x) - log
z
(y)
= ylog
z
(x)
then log
z
(x) = log
z
(y)
then log
z
(x) < log
z
(y)

If log
z
(x) = a, then x = z
a

Growth
If f is a function from Z or R to R, how can
we quantify the rate of growth and compare
rates of growth of different functions?
Possible problem: Whether f(n) or g(n) is
larger at any point may depend on value of
n.
For example: n
2
> 100n if n > 100
How to quantify growth as n gets bigger?
-200
0
200
400
600
800
1000
1200
0 2 4 6 8 10 12
Big-O Notation
Let f and g be functions from the set of integers or
the set of real numbers to the set of real numbers.
We say that f(x) is O(g(x)) if there are constants
CeN and keR such that
|f(x)| s C|g(x)| whenever x > k.
We say f(x) is big-oh of g(x).
The intuitive meaning is that as x gets large, the
values of f(x) are no larger than a constant time the
values of g(x), or f(x) is growing no faster than g(x).
The supposition is that x gets large, it will approach
a simplified limit.
Show that 3x
3
+2x
2
+7x+9 is O(x
3
)
Proof: We must show that - constants CeN
and keR such that |3x
3
+2x
2
+7x+9| s C|x
3
|
whenever x > k.
Choose k = 1 then
3x
3
+2x
2
+7x+9 s 3x
3
+2x
3
+7x
3
+9x
3
= 21x
3
So let C = 21.
Then 3x
3
+2x
2
+7x+9 s 21 x
3
when x > 1.
Show that n! is O(n
n
)
Proof: We must show that - constants CeN and
keR such that |n!| s C|n
n
| whenever n > k.
n! = n(n-1)(n-2)(n-3)(3)(2)(1)
s n(n)(n)(n)(n)(n)(n) n times
=n
n
So choose k = 0 and C = 1


General Rules
Multiplication by a constant does not change the
rate of growth. If f(n) = kg(n) where k is a
constant, then f is O(g) and g is O(f).
The above means that there are an infinite number
of pairs C,k that satisfy the Big-O definition.
Addition of smaller terms does not change the rate
of growth. If f(n) = g(n) + smaller order terms,
then f is O(g) and g is O(f).

Ex.: f(n) = 4n
6
+ 3n
5
+ 100n
2
+ 2 is O(n
6
).

General Rules (cont.)
If f
1
(x) is O(g
1
(x)) and f
2
(x) is O(g
2
(x)),
then f
1
(x)f
2
(x) is O(g
1
(x)g
2
(x)).
Examples:
10xlog
2
x is O(xlog
2
x)
n!6n
3
is O(n!n
3
)
=O(n
n+3
)

Examples
f(x) = 10 is O(1)
f(x) = x
2
+ x + 1 is O(x
2
)
f(x) = 2x
5
+ 100 x
3
+ xlogx is O(x
5
)
f(x) = 2
n
+ n
10
is O(2
n
)
How would you prove this?
Prove that n
10
is O(2
n
)
Proof: We must show that - constants CeN and keR
such that |n
10
| s C|2
n
| whenever n > k.
Take log
2
of both expressions.
log
2
2
n
= nlog
2
2 =n, log
2
n
10
= 10log
2
n
When is 10log
2
n < n? or n/log
2
n > 10?
2/1 = 2, 4/2 = 2, 8/3 ~ 2.67, 16/4 = 4
32/5 = 6.2, 64/6 ~ 10.67
For n = 64, 2
64
> 64
10.
So, if we choose then k = 64, C =
1, then |n
10
| s 1*|2
n
| whenever n > 64.

Example: Big-Oh Not Symmetric
Order matters in big-oh. Sometimes f is
O(g) and g is O(f), but in general big-oh is
not symmetric.
Consider f(n) = 4n and g(n) = n
2
. f is O(g).
Can we prove that g is O(f)? Formally, -
constants CeN and keR such that |n
2
| s
C|4n| whenever n > k?
No. To show this, we must prove that
negation is true for all C and k. CeN,
keR, -n>k such that n
2
> C|4n|.
CeN, keR, -n>k such that n
2
> 4nC.

To prove that negation is true, start with
arbitrary C and k. Must show/construct an
n>k such that n
2
> 4nC
Easy to satisfy n > k, then
To satisfy n
2
>4nC, divide both sides by n to
get n>4C. Pick n = max(4C+1,k+1), which
proves the negation.

0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 20 40 60 80 100
n*n
4n
5*4n
10*4*n
20*4*n
Is 2
n
O(n!)?
We must show that - constants CeN and keZ
such that |2
n
| s C|n!| whenever n > k.
2
n
= 2(2)(2)(2)(2)(2) n times
s n(n-1)(n-2)(3)(2)(1) =n! if n = 4

So let C = 1 and k = 3.



Is 2
n
O(n!)?
Note that we could also choose k = 1 and C =
2
Since
|2
0
| s 2*|0!| = 2
|2
1
| s 2*|1!| = 2
|2
2
| s 2*|2!| = 4
|2
3
| s 2*|3!|= 12
Is f(x)=(x
2
+1)/(x+1) O(x)?
We must show that - constants CeN and
keR such that |f(x)| s C|x| whenever x > k.


x
x
x
x
x x
x
x
x
x
<
+
+
=
+
+ +
=
+
+
=
+
+
1
2
1
1
2 ) 1 )( 1 (
1
2 1
1
1
2 2
When x > 1 (Why?)
Therefore let k=1, C = 1
|(x
2
+1)/(x+1)| s |x| when x > 1
Hierarchy of functions
1
log
2
n

3
\n

\n

n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1
log
2
n

3
\n

\n

n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n


3
\n

\n

n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n

\n

n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n

n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n

nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n

n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n, n\n

n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n, n\n, n
2
n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n, n\n, n
2
, n
3
2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n, n\n, n
2
, n
3
, 2
n
n!

n
n
Hierarchy of functions
1, log
2
n,
3
\n, \n, n, nlog
2
n, n\n, n
2
, n
3
, 2
n
, n!, n
n
Each one is Big-Oh of any function to its right
Prove that log
10
x is O(log
2
x)
First we will prove the following lemma:
Lemma: log
10
x = clog
2
x where c is a
constant.
Proof:
Let y = log
2
x.
Then 2
y
= x and log
10
2
y
= log
10
x.
log
10
2
y
= ylog
10
2 = log
10
x. But since y =
log
2
x, this means that
log
2
xlog
10
2 = log
10
x. Therefore c = log
10
2
To Prove that log
10
x is O(log
2
x)
We must show that - constants CeN and
keR such that |log
10
x| s C|log
2
x| whenever
x > k.
From the lemma log
10
2log
2
x = log
10
x; so
choose C = log
10
2, k=0
Prove log(n!) is O(nlogn)
We must show that - constants CeN and
keR such that |logn!| s C|nlogn| whenever
x > k.
We know that n! s n
n
so
log(n!) s log(n
n
)= nlogn
So choose k = 1, C = 1
Time Complexity
We can use Big-O to find the time complexity of
algorithms (i.e., how long they take in terms of the
number of operations performed).
There are two types of complexity normally considered.
Worst-case complexity. The largest number of
operations needed for a problem of a given size in the
worst case. The number of operations that will
guarantee a solution.
Average-case complexity. The average number of
operations used to solve a problem over all inputs of a
given size.
Complexity of the Linear Search
Algorithm
Worst-case complexity
The algorithm loops over all the values in a list of n
values.
At each step, two comparisons are made. One to see whether
the end of the loop is reached, and one to compare the search
element x with the element in the list.
If x is equal to list element a
i
, then 2i comparisons are made.
If x is not in the list, then 2n comparisons are made.
The worst-case complexity is thus 2n and is O(n).
Complexity of the Linear Search
Algorithm
Average-case complexity
The algorithm loops over all the values in a list of n
values.
At each step, two comparisons are again made.
On average, the number of comparisons is
2 + 4 + 6 +.+ (2n)
n
Whats the numerator?
Average case is O(n)
k =
n(n +1)
2
k=1
n

2( )

Complexity of Pair-wise
Correlation
Assume that there are n elements to correlate