Intro To Algorithms 2

11-711 Algorithms for NLP
Introduction to Analysis of Algorithms
Reading:
Cormen, Leiserson, and Rivest,
Introduction to Algorithms
Chapters 1, 2, 3.1., 3.2.
Analysis of Algorithms
Analyzing an algorithm: predicting the computational resources that

the algorithm requires
– Most often interested in runtime of the algorithm
– Also memory, communication bandwidth, etc.
Computational model assumed: RAM

– Instructions are executed sequentially
– Each instruction takes constant time (different constants)
The behavior of an algorithm may be different for different inputs.

We want to summarize the behavior in general, easy-to-understand
formulas, such as worst-case, average-case, etc.
1 11-711 Algorithms for NLP

Generally, we are interested in computation time as a function of
input size.
– Different measures of input size depending on the problem
– Examples:
Problem Input Size
Sorting List of numbers to sort # of numbers to sort
Multiplication Numbers to multiply # of bits
Graph Set of nodes and edges # of nodes + # of edges
“Running time” of an algorithm: number of primitive “steps”
executed
– Think of each “step” as a constant number of instructions, each of
which takes constant time.
Example: Insertion-Sort (CLR, p. 8)

Insertion Sort (A)
cost times
1. for j 2 to length[A] n

2. do key A[j] n-1

3.
4. i j-1 n-1

5. while i 0 and A[i] key

6. do A[i+1] A[i]

7. i i-1

8. A[i+1] key n-1


Insertion-Sort
: # of times the while loop in line 5 is executed for the value

+ *
+ 0
+ 1
'
')
' / ,
' / ,
'2
"!

(
# $
# %&
#
%#
%#
%# ,
%#
.-,
.-,
.-,
&
&
&
In the best case, when the array is already sorted, the running time is
3
a linear function of .
4
!"

5
6%
In the worst case, when the array is opposite of sorted order, the
3
running time is a quadratic function of .
4
& 6%
!"

5

#
%

We are mostly interested in worst-case performance because

7
1. It is an upper bound on the running time for any input.

2. For some algorithms it occurs fairly often.
3. The average case is often roughly as bad as the worst case.
We are interested in the rate of growth of the runtime as the input size
7
increases. Thus, we only consider the leading term.

– Example: Insertion-Sort has worst case
<;:9
8
=
We will normally look for algorithms with the lowest order of growth
7
But not always:

7
for instance, a less efficient algorithm may be preferable if it’s an

>
anytime algorithm.

Analyzing Average Case of Expected Runtime
Non-experimental method:
?
1. Assume all inputs of length are equally likely
@
2. Analyze runtime for each particular input and calculate the
average over all the inputs
Experimental method:
?
1. Select a sample of test cases (large enough!)

2. Analyze runtime for each test case and calculate the average and
standard deviation over all test cases

Recursive Algorithms (Divide and Conquer)
General structure of recursive algorithms:

A
1. Divide the problem into smaller subproblems.

2. Solve (Conquer) each subproblem recursively.
3. Merge the solutions of the subproblems into a solution for the full
problem.
Analyzing a recursive algorithm usually involves solving a

A
recurrence equation.
Example: Merge-Sort (CLR, p. 13)

A

Recursive Algorithms
time to divide a problem of size

<GF
B
C
ED
E
time to combine solutions of size
GF
B
ED
E
H
Assume we divide a problem into subproblems, each the size of
J K
B
I
the original problem.
The resulting recurrence equation is:

B
OND VUD
P Q YZ[
E \]^
TSR
M LI
F XWF K
G<F
L
ED
C
ED
ED
P_
\
H
XWF

ij ij
ij ij m i m i
times
1
1
1
1
1
1
1
pv
i oq
: Initialization
up t
9
w
and
pv pv
mx zv
h i oj o lt
p pu o nt
fgc
ce st ij y iq y { {
on l
cd xk v x k ztv pk v pk v
3. create arrays
x
to
to
ba` <ml rm st w p p
xk k z i oj i oq t kx
ik j ik q st kz
do
do
Merge
w
4. for
6. for
10.
11.
1.
2.
5.
7.
8.
9.
(cont.): Actual Merge

10

times
n
n

g~

~
~ to

then

else
b}|
do if
12. for
Merge
13.
14.
15.
16.
17.
Merge Sort (A,p,r)
times
1. if p r

2. then q (p+r)/2

3. Merge-Sort(A,p,q)

¡

¡ ¡
4. Merge-Sort(A,q+1,r)

5. Merge(A,p,q,r)
¢

A note on Implementation
Note that in the pseudocode of the Merge operation, the for-loop is

£
always executed times, no matter how many items are in each of
¤
the subarrays.
This can be avoided in practical implementation.

£
Take-home message: don’t just blindly follow an algorithm.

£

Merge-Sort
<ª© ª ©
¥
¦
¬O§ ¨§
¨§ ¨§
« «
© ©
¥
According to the Merge-Sort algorithm, we divide the problem into 2

¥
subproblems of size .
® ¯
The resulting recurrence equation is:
¥
ª ¨ ¨µ
O¬§ §
¬ ¬
±² ±²
« ³°
© ® X´© ¯
ª©
°
¨§
¨§
«
©
By solving the recurrence equation we get ,
ª©
¥
¶¨§
¨§
·¸
¨¯
«
©
much better than for Insertion-Sort.
¯ ©
¨§
«
There are several ways of solving this recurrence equation. The

¥
simplest one uses a recurrence tree, where each node in the tree
indicates what the cost of solving it is. Then inspecting the tree can
give us the solution.
In this case, we get , because we divide the problem in half in
¹
º»
¾ ½¼
each recursive step.

Growth of Functions
We need to define a precise notation and meaning for order of growth

¿
of functions.
We usually don’t look at exact running times because they are too
¿
complex to calculate and not interesting enough.
Instead, we look at inputs of large enough size to make only the order
¿
of growth relevant - asymptotic behavior.
Generally, algorithms that behave better asymptotically will be the

¿
best choice (except possibly for very small inputs).

-notation (“Theta” notation)
À
Asymptotically tight bound
Á
Idea: if both functions grow “equally” fast

<ÆÅ
Á
ÄÂÃ
ÄÃ
Ç
ÈÃ
ÅÅ
Formally:
Á
there exist positive constants , , and

Æ ÅÅ
ÄÃ
ËÌ
ËÍ
Ä
Ç
Î
ÈÃ
ÂÄÃÉ
ÅÊ
such that for
Ë ÌÈ
ÄÃ
ÄÂÃ
Ë ÍÈ
ÄÃ
Ï
Ð
ÐÅ
ÐÅ
Å
all
Ä
ÑÄ
ÎÒ
means a constant function or a constant
Á
OÓÃ
Ç
Å

-notation (“Big-O” notation)
Ô
Asymptotic upper bound
Õ
Idea: if grows faster than , possibly much

Ú<Ù
Õ
ØÖ×
Ø×
Ø×
ØÖ×
Û
Ü×
Ü
ÙÙ
Ù
faster
Formally:
Õ
there exist positive constants and

Ú ÙÙ
Ø×
ß
Ø
Û
à
Ü×
ÖØ×Ý
ÙÞ
such that for all
ØÖ× Û Ü×
ß Ü ÙÙ
Ø×
Ø
á
â ÚÙ
â Ù Ø×
ãØ
àä
Ù
Note:
ÚÙ
Ú ÙÙ
Õ
ØÖ×
Ø×
ØÖ×
å
æ
Ü×
“Big-O” notation often allows us to describe runtime by just

Õ
inspecting the general structure of the algorithm.

– Example: The double loop in Insertion-Sort leads us to conclude
that the runtime of the algorithm is .
èçØ×
Û
Ù
We use “Big-O” analysis for worst-case complexity since that
Õ
guarantees an upper bound on any case.

-notation (“Big-Omega” notation)
é
Asymptotic lower bound

ê
Idea: if grows slower than , possibly

ï<î
ê
ð
íëì
íì
íì
íëì
ñì
ñ
îî
î
much slower
Formally:
ê
there exist positive constants and

ï îî
ð
íì
ô
í
õ
ñì
ëíìò
îó
such that for all
ôñ
íì
íëì
í
ö
÷
øí
õù
î÷
î
Theorem:
ê
For any two functions and ,

íëì ï î
íì î
ñ íñìì
î
iff î î and
ïî
ïî
ð
íëì
íì

í
íëì
íì
ú
û
ñì
ëì
ñì
îî
îî
-notation (“Little-o” notation)
ü
Upper bound that is not asymptotically tight
ý
Idea: if grows much faster than

ý

þÿ
þÿ
ÿ
ÿ

ÿ

Formally:
ý
for any positive constant , there exists

þÿ
ÿ
ÿ

a constant such that

þÿ
ÿ

for all

We will normally use “Big-O” notation since we won’t want to make
ý
hard claims about tightness.

-notation (“Little-omega” notation)
Lower bound that is not asymptotically tight

Idea: if grows much slower than

Note: if and only if

Formally:

for any positive constant , there exists

!

a constant such that

!
$
#
"

for all

%
"
&

Comparison of Functions
The , , , , and relations are transitive.

'
,
(
– Example: )

1 0

1 0

1 0

1 0
/.
/.
/.
5.
/.
/.
5.
/.
)
430
)
-
2.
-
0
0
0
The , , and relations are reflexive.
'
*
(
– Example:
1 0

/ .

/ .

)
-
-
.
0
0
Note the analogy to the comparison of two real numbers.
'
e.g.
10
8
7 0
/.
/ .

9
<;:
)
-
2.
e.g.
1 0

8
7 0
/.
/ .

9
;=
-
2.

B
B
A@
Comparison of Growth of Functions
?
@
B B F
B B B
nor
B
A @
A @

A @

22
? ? B
@ @ ? B
@
D F
I A@
?
@
C B C B

C B
E
A @
A @

are neither
A @

Transpose symmetry:
G G
Lack of Trichotomy:
G
iff
iff
iff
B B B B B
B B B B B B
Transitivity: all
B
A@ A@ A @
A @
A @

A @

? ? ?
Reflexivity:
@ @ @ G@ G@ B
Symmetry:
G@
D E F D E H A@
?
Some
CB CB CB C B
C B
C B

A@ A@ A@ A@ A@ A@
? ? ? ? ? ?
> > > > >

Intro To Algorithms 2

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Intro To Algorithms 2

Uploaded by

Copyright:

Available Formats

11-711 Algorithms for NLP

Introduction to Analysis of Algorithms

Analyzing an algorithm: predicting the computational resources that

Computational model assumed: RAM

The behavior of an algorithm may be different for different inputs.

1 11-711 Algorithms for NLP

2 11-711 Algorithms for NLP

3 11-711 Algorithms for NLP

running time is a quadratic function of .

4 11-711 Algorithms for NLP

We are mostly interested in worst-case performance because

1. It is an upper bound on the running time for any input.

increases. Thus, we only consider the leading term.

But not always:

for instance, a less efficient algorithm may be preferable if it’s an

5 11-711 Algorithms for NLP

1. Assume all inputs of length are equally likely

1. Select a sample of test cases (large enough!)

6 11-711 Algorithms for NLP

General structure of recursive algorithms:

1. Divide the problem into smaller subproblems.

Analyzing a recursive algorithm usually involves solving a

Example: Merge-Sort (CLR, p. 13)

7 11-711 Algorithms for NLP

time to divide a problem of size

Assume we divide a problem into subproblems, each the size of

The resulting recurrence equation is:

8 11-711 Algorithms for NLP

11 11-711 Algorithms for NLP

Note that in the pseudocode of the Merge operation, the for-loop is

always executed times, no matter how many items are in each of

This can be avoided in practical implementation.

Take-home message: don’t just blindly follow an algorithm.

12 11-711 Algorithms for NLP

According to the Merge-Sort algorithm, we divide the problem into 2

There are several ways of solving this recurrence equation. The

14 11-711 Algorithms for NLP

We need to define a precise notation and meaning for order of growth

complex to calculate and not interesting enough.

of growth relevant - asymptotic behavior.

Generally, algorithms that behave better asymptotically will be the

best choice (except possibly for very small inputs).

15 11-711 Algorithms for NLP

Idea: if both functions grow “equally” fast

there exist positive constants , , and

16 11-711 Algorithms for NLP

Idea: if grows faster than , possibly much

there exist positive constants and

“Big-O” notation often allows us to describe runtime by just

inspecting the general structure of the algorithm.

guarantees an upper bound on any case.

Asymptotic lower bound

Idea: if grows slower than , possibly

there exist positive constants and

For any two functions and ,

Idea: if grows much faster than

for any positive constant , there exists

hard claims about tightness.

19 11-711 Algorithms for NLP

Lower bound that is not asymptotically tight

Idea: if grows much slower than

for any positive constant , there exists