Lecture # 14_new

Design and Analysis of Algorithm
Tanveer Ahmed Siddiqui
Department of Computer Science

COMSATS University, Islamabad
Recap and Today Covered
Algorithm Design and Analysis Process

Understand the problem
Decide on : algorithm
design techniques etc.
Design an algorithm
Prove correctness
Analyze efficiency etc
Code the algorithm

Reading Material
Read Chapter 8
Dynamic Programming

Objectives
 How to Design Algorithm using Dynamic

Programming

New designing technique build from the idea of divide and conquer
 Perspective #1: Divide & Conquer with Memory

Table
 Solve the sub-problems one by one, smallest first,
possibly by recursion.
 Store the solutions to sub-problems in a table and
 Reuse the solutions of subproblem from table to
solve larger sub-problems
 Repeat above steps until the top (original)
problem instance is solved.

Procedure
 Characterize the Optimal Substructure Property for the

given optimization problem.
 Objective Function: What should be optimize
 Devise a recursive formula for the objective function.
 Compute the value of an optimal solution in a bottom-up
fashion.
 replace its recursive calls with table lookups. That is
instead of returning a value, record it in a table entry
 We devise “look-up template”
 Use base case of divide-and-conquer to fill in

start of table
 devise for-loops that fill the table using “look-
up template”
Chain Matrix Multiplication: Motivation
Problem: Given a sequence

of matrices, find the most
efficient way to multiply
these matrices together.
A1 = Translation along x axis

A1 · (A2 . (A3 . A4)))
A2 = Translation along y axis
(A1 · ((A2 . A3). A4))
A3 = Rotation
((A1 · A2). (A3 . A4))
A4 = Scaling
((A1 · (A2 . A3)). A4)
(((A1 · A2). A3). A4)
Statement: The chain-matrix multiplication problem can be stated as
below:
Given a chain of [A1, A2, . . . , An] of n matrices where for i = 1,
2, . . . , n, matrix Ai has dimension pi-1 x pi, find the order of
multiplication which
Department of Computer Scienceminimizes the number of scalar multiplications.
 Problem: Given a chain A1, A2, …, An of matrices. The dimension of
Ai is Pi-1 X Pi. How do we fully add parenthesis to the chain so that
the total number of scalar multiplication is minimized?
 Brute force: Try all possible combinations

A 1 A2 A 3 … … An
=> ( (A1 … Ak) (Ak+1… An) ) (Split at k)
Number of possibilities
to parenthesize n 1 if n  1
matrices 
P (=>
n)   n 1
 P ( k ) P ( n  k ) if n  2
 k 1
=> P( n )  C ( n  1)  1  2( n  1)   ( 4 )
n
n  n  1  n 3/ 2
C(n) - Catalan number
Department of Computer Science 8
Problem: Given a sequence

of matrices, find the most
efficient way to multiply
these matrices together.
A1 = Translation along x axis

A1 · (A2 . (A3 . A4)))
A2 = Translation along y axis
(A1 · ((A2 . A3). A4))
A3 = Rotation
((A1 · A2). (A3 . A4))
A4 = Scaling
((A1 · (A2 . A3)). A4)
(((A1 · A2). A3). A4) The problem is not actually to perform
the multiplications, but merely to
In order to find best decide in which order to perform the
multiplication order, what multiplications.
should we minimize?
Divide and Conquer: Recursive Algorithm
Recurrence Relation: After kth matrix, create two

sub-lists, one with k and other with n - k
matrices i.e.
(A1 A2A3A4A5 . . . Ak) (Ak+1Ak+2…An)

Divide and Conquer
 Let cost(i, j) be the minimum cost of computing

 A1 A2A3A4A5 . . . Ak.Ak+1Ak+2…An
 What is the cost of breaking the product at Ak?

(A1 A2A3A4A5 . . . Ak) (Ak+1Ak+2…An)
 It is cost(i, k) plus cost(k + 1, j) plus the cost of
multiplying an ri−1×rk matrix by an rk×rj matrix.
 Therefore, the cost of breaking the product at Ak
is cost(i, k) + cost(k + 1, j) + ri−1rkrj
 What is the base case?
 Observe that if i = j then the sequence contains only
one matrix, and so the cost is 0. (There is nothing to
Procedure
 Characterize the Optimal Substructure Property for the

given optimization problem.
 Devise a recursive formula for the objective
function.
Cost (i, j) = 0 i=j
Cost (i, j) = cost(i, k) + cost(k + 1, j) + ri−1rkrj otherwise

A simple recursive (top-down) solution
using the formula for m[i,j] we could solve the problem:
RECURSIVE-MATRIX-CHAIN(p, i, j)
1. if i=j then return 0
2. m[i, j] = 
3. for k ← 1 to j − 1
4. q ← RECURSIVE-MATRIX-CHAIN (p, i , k)
5. + RECURSIVE-MATRIX-CHAIN (p, k+1 ,
j)
6. + p[i-1] p[k] p[j]
7. if q < m[i, j] then m[i, j] ← q
T (1)  1
8. return m[i, j]
Complexity: n 1
T (n)  1   (T (k )  T (n  k )  1) for n > 1
k 1
Running Time of The Algorithm
n 1
T(n) has lower bound T ( n)  1   (T (k )  T (n  k )  1)

k 1
of 2n, too slow
n 1
 2 T (i )  n
i 1
n 1
 2 2i 1  n
i 1
n2
 2 2 i  n
i 0
 2( 2 n 1  1)  n
 2n  2  n
 2n

Divide and Conquer
The problem is, the algorithm solves

the same sub problems over and over
again!
Complexity of recursive solution
n 1 n 1
T ( n )  1   (T ( k )  T ( n  k )  1)  n  2  T (i )
k 1 i 1
 We will use the substitution method – we guess a solution and
then prove by using mathematical induction that it is correct.
 We will prove that T(n) = (2n). Specifically, that T(n) ≥ 2n-1 for
all n ≥1.
 Induction Base: T(1) ≥1=20
 Induction Assumption: assume T(k) ≥ 2k-1 for all 1 ≤ k < n
 Induction Step:
n 1 n2
T (n)  n  2 2i 1  n  2 2i 
i 1 i 0
n 1 n 1
 n  2(2  1)  n  2  2  2
n

Memoization
 A top-down variation of dynamic programming
 Idea: remember the solution to subproblems as they are
solved in the simple recursive algorithm
Initialize all m elements to  ; call Lookup-Chain(p, i, j)
LOOKUP-CHAIN(p,i,j) 3
Takes O(n ) time
if m[i,j] <  then return m[i,j]
if i =j then m[i,j]  0 Requires (n2) space
else
for k ← 1 to j − 1
q ← LOOKUP-CHAIN (p, i , k)
+ LOOKUP-CHAIN (p, k+1 , j) + p[i-1] p[k] p[j]
if q < m[i, j] then m[i, j] ← q
return m[i, j]
 The memoized algorithm has the advantage of solving
only those problems that are needed
 Dynamic programming is considered better when all
subproblems must be calculated, because there is no
overhead
Department for recursion
of Computer Science 17
Procedure
 Compute the value of an optimal solution in a bottom-

up fashion.
 replace its recursive calls with table lookups. That is
instead of returning a value, record it in a table
entry.

0 if i  j
minm[i, k ]  m[k  1, j ]  pi 1 pk p j  if i  j
m[i, j ]  

 ik  j

Procedure
 Compute the value of an optimal solution in a bottom-

up fashion.
m[1,1] m[1,2] m[1,3] m[1,4]

m[2,2] m[2,3] m[2,4]
m[3,3] m[3,4]
m[4,4]

Procedure

 Use base case of divide-and-conquer to fill
in start of table.
1
2
3
4
 devise for-loops that fill the table using

“look-up template”
Procedure

 Use base case of divide-and-conquer to fill
in start of table.

Procedure

 devise for-loops that fill the table using
“look-up template”.
 Compute entries of m[] in increasing order
of difference between parameters(j-i).

1 5 8 10
2 6 9
3 7
4

Filling in the First Super diagonal

Filling in the Second Super diagonal

Filling in the dth Super diagonal

Filling in the dth Super diagonal
Analysis:

Step 3: Compute the value of an optimal
solution buttom-up
Input: n ; an array p[0…n] containing matrix dimensions
State: m[1..n, 1..n] for storing m[i, j]
s[1..n, 1..n] for storing the optimal k that was used to calculate m[i, j]
Result: Minimum-cost table m and split table s
Cost of chains of length 1
MATRIX-CHAIN-ORDER(p, n)
for i ← 1 to n
m[i, i] ← 0 Use recursive formula to
for l ← 2 to n compute the cost of chains of
for i ← 1 to n-l+1 length l ≥2
j ← i+l-1
m[i, j] ← 
for k ← i to j-1
q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]
if q < m[i, j]
m[i, j] ← q Takes O(n 3
) time
s[i, j] ← k Requires (n2) space
return m and s
Computational Cost
n n n n i
T ( n)  n    ( j  i)   k
i 1 j i 1 i 1 k 1
(n  i )(n  i  1)
n
T ( n)  n  
i 1 2
1 n 2
T (n)  n   (n  2ni  i 2  n  i )
2 i 1
1 n 2 n n n n
T (n)  n  ( n   2ni   i 2   n   i )
2 i 1 i 1 i 1 i 1 i 1

Computational Cost
1 n 2 n n n n
T (n)  n  ( n   2ni   i 2   n   i )
2 i 1 i 1 i 1 i 1 i 1
1 2 n n n n n
T ( n)  n  ( n  1  2n  i   i 2  n  1   i )
2 i 1 i 1 i 1 i 1 i 1
1 2 n(n  1) n(n  1)(2n  1) n(n  1)

T (n)  n  (n .n  2n.   n.n  )
2 2 6 2
Computational Cost
1 2 n(n  1) n(n  1)(2n  1) n(n  1)

T (n)  n  (n .n  2n.   n.n  )
2 2 6 2
1 3 n(n  1)(2n  1) n(n  1)

T (n)  n  (n  n (n  1) 
2
n 
2
)
2 6 2
1
T (n)  n  (6n 3  6n 3  6n 2  2n 3  3n 2  n  6n 2  3n 2  3n)
12
1 1 1
T (n)  (12n  2n  2n)  (10n  2n )  (5n  n 3 )
3 3
12 12 6
Obtaining the optimal multiplication order
j
i 2 3 4 A1 30×1
1 1 1 1
A2 1×40
2 2 3 A3 40×10
3 3 A4 10×25
PRINT(s, 1, 4)
PRINT (s, 1, 1) PRINT (s, 2, 4)
PRINT (s, 2, 3) PRINT (s, 4, 4)

Output: (A1((A2A3)A4))
PRINT (s, 2, 2) PRINT (s, 3, 3)

Constructing an optimal solution
 Each entry s[i, j ]=k shows where to split the product Ai

Ai+1 … Aj for the minimum cost:
A1 … An = ( (A1 … As[i, n]) (As[i, n]+1… An) )
 To print the solution invoke the following function with ( s,
1, n) as the parameter:
PRINT-OPTIMAL-PARENS(s, i, j)
1. if i=j then print “A”i
2. else print “(”
3. PRINT-OPTIMAL-PARENS(s, i, s[i, j])
4. PRINT-OPTIMAL-PARENS(s, s[i, j]+1, j)
5. print “)”

Example: Dynamic Programming
 Problem: Compute optimal multiplication order
for a series of matrices given below
A1 A2 A3 A4
. . .
10  100 100  5 5  50 50  20
P0 = 10 m[1,1] m[1,2] m[1,3] m[1,4]

P1 = 100 m[2,2] m[2,3] m[2,4]
P2 = 5 m[3,3] m[3,4]
P3 = 50 m[4,4]
P4 = 20
Main Diagonal
m[i, i ]  0, i  1,...,4
m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )
ik  j
Main Diagonal 0
 m[1, 1] = 0 0
 m[2, 2] = 0 0
 m[3, 3] = 0 0
 m[4, 4] = 0

Computing m[1, 2], m[2, 3], m[3, 4]

ik  j
m[1,2]  min (m[1, k ]  m[k  1,2]  p0 . pk . p2 )

1 k  2
m[1,2]  min (m[1,1]  m[2,2]  p0 . p1. p2 )

0 5000
m[1, 2] = 0 + 0 + 10 . 100 . 5
0
= 5000
0
s[1, 2] = k = 1
0
Computing m[2, 3]

ik  j
m[2,3]  min (m[2, k ]  m[k  1,3]  p1. pk . p3 )

2 k 3
m[2,3]  min (m[2,2]  m[3,3]  p1. p2 . p3)

0 5000
m[2, 3] = 0 + 0 + 100 . 5 . 50
0 25000
= 25000
0
s[2, 3] = k = 2
0
Computing m[3, 4]

ik  j
m[3,4]  min (m[3, k ]  m[k  1,4]  p2 . pk . p4 )

3 k  4
m[3,4]  min (m[3,3]  m[4,4]  p2 . p3 . p4 )

0 5000
m[3, 4] = 0 + 0 + 5 . 50 . 20
0 25000
= 5000
0 5000
s[3, 4] = k = 3
0
Computing m[1, 3], m[2, 4]

ik  j
m[1,3]  min (m[1, k ]  m[k  1,3]  p0 . pk . p3 )

1 k 3
m[1,3]  min (m[1,1]  m[2,3]  p0 . p1. p3 ,
m[1,2]  m[3,3]  p0 . p2 . p3 ))
m[1, 3] = min(0+25000+10.100.50, 5000+0+10.5.50)
= min(75000, 2500) = 2500
s[1, 3] = k = 2
Computing m[2, 4]

ik  j
m[2,4]  min (m[2, k ]  m[k  1,4]  p1. pk . p4 )

2 k  4
m[2,4]  min (m[2,2]  m[3,4]  p1. p2 . p4 ,
m[2,3]  m[4,4]  p1. p3 . p4 ))

m[2, 4] = min(0+5000+100.5.20, 25000+0+100.50.20)
= min(15000, 35000) = 15000
s[2, 4] = k = 2
Computing m[1, 4]
ik  j
m[1,4]  min (m[1, k ]  m[k  1,4]  p0 . pk . p4 )

1 k  4
m[1,4]  min (m[1,1]  m[2,4]  p0 . p1. p4 ,
m[1,2]  m[3,4]  p0 . p2 . p4 , m[1,3]  m[4,4]  p0 . p3 . p4 )

m[1, 4] = min(0+15000+10.100.20, 5000+5000+
10.5.20, 2500+0+10.50.20)
= min(35000, 11000, 35000) = 11000
s[1, 4] = k = 2
Final Cost Matrix and Its Order of Computation
 Final Cost Matrix 0 5000 2500 11000

0 25000 15000
0 5000
0
1 5 8 10
 Order of Computation 2 6 9
3 7
4

K,s Values Leading Minimum m[i, j]
0 1 2 2
0 2 2
0 3
0
For, m(1, 4) 2
k=2
10x20
((A1 . A2) . (A3 . A4)) 1
3
10 x 5 5 x 20
A1 A2 A3 A4
10 x 100 100 x 5 5 x 50 50 x 20
Representing Order using Binary Tree
 The above computation shows that the minimum
cost for multiplying those four matrices is 11000.
 The optimal order for multiplication is
((A1 . A2) . (A3 . A4))
2
For, m(1, 4) 10x20
k=2 1
3
10 x 5 5 x 20
A1 A2 A3 A4
10 x 100 100 x 5 5 x 50 50 x 20

Similar
Problems
Meaning full learning
 Some typical optimization problems like:
 Matrix Chain Multiplication
 Optimal Binary Search Tree
 Optimal polygon triangulation
 Even though, every problem has its own nature and
description; however after changing certain constraints,
they become equivalent to each other.

Similarity between MCM and OPT
 Optimal polygon triangulation

 A polygon can be divided intro triangles in a number
of ways
 If there is a weight
associated with
each triangle,
 Find the triangulation of minimum (maximum)
weight
 Any weight, e.g. length of perimeter, will do!
 This problem maps to matrix chain
multiplication

Mapping Problems to related ones
 Polygon triangulation / matrix chain
multiplication

Similarity between OBST and MCM
MCM OBST
Same optimal substructure.

Recurrence relation of OBST
00 ifif jjii11

ee[[ii, ,jj]]
min {e[i, r  1]  e[r  1, j ]  w(i, j )} ififii jj
r  j {e[i , r  1]  e[ r  1, j ]  w(i , j )}
i min
 ir  j
Recurrence relation of OBST

0 if i  j
minm[i, k ]  m[k  1, j ]  pi 1 pk p j  if i  j
m[i, j ]  

 ik  j

Filling Pattern of OBST Filling Pattern of MCM

OBST
MCM
CONCLUSION

What we have learnt?
 Define a set of sub problems that can lead to the

solution of the original problem.
 Think: what information would make solving this
problem easier?
 Try to find a relation between the sub problems
you found above, and the current problem you're
trying to solve.
 Write the recursion solution of the original problem
from the solutions of sub problems.
 Build the solution lookup table in a bottom-up
fashion, until reaching the original problem.
 Initialize the lookup table.
 Fill the other terms through iteration by using the recurrence
relation.
What we have learnt?
 In a nutshell, dynamic programming is recursion

without repetition.
 Dynamic programming algorithms store the
solutions of intermediate subproblems, often but
not always in some kind of array or table.
 Many algorithms students make the mistake of
focusing on the table (because tables are easy and
familiar) instead of the much more important (and
difficult) task of finding a correct recurrence.
 As long as we memoize the correct recurrence, an explicit table
isn’t really necessary, but if the recursion is incorrect, nothing
works

Lecture # 14_new

Uploaded by

Copyright:

Available Formats

You might also like

Lecture # 14_new

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture # 14_new

Uploaded by

Copyright:

Available Formats

Design and Analysis of Algorithm

Tanveer Ahmed Siddiqui

Department of Computer Science

Algorithm Design and Analysis Process

Analyze efficiency etc

Code the algorithm

Department of Computer Science

 How to Design Algorithm using Dynamic

Department of Computer Science

 Perspective #1: Divide & Conquer with Memory

Department of Computer Science

 Characterize the Optimal Substructure Property for the

 Use base case of divide-and-conquer to fill in

Problem: Given a sequence

A1 = Translation along x axis

 Brute force: Try all possible combinations

Problem: Given a sequence

A1 = Translation along x axis

Recurrence Relation: After kth matrix, create two

Department of Computer Science

 Let cost(i, j) be the minimum cost of computing

 What is the cost of breaking the product at Ak?

 Characterize the Optimal Substructure Property for the

Department of Computer Science

T(n) has lower bound T ( n)  1   (T (k )  T (n  k )  1)

Department of Computer Science

The problem is, the algorithm solves

Department of Computer Science 16

 Compute the value of an optimal solution in a bottom-

 We devise “look-up template”

Department of Computer Science

 Compute the value of an optimal solution in a bottom-

m[1,1] m[1,2] m[1,3] m[1,4]

Department of Computer Science

 We devise “look-up template”

 devise for-loops that fill the table using

 We devise “look-up template”

Department of Computer Science

 We devise “look-up template”

of difference between parameters(j-i).

Department of Computer Science

Department of Computer Science

Department of Computer Science

Department of Computer Science

Department of Computer Science

Department of Computer Science

1 2 n(n  1) n(n  1)(2n  1) n(n  1)

1 2 n(n  1) n(n  1)(2n  1) n(n  1)

1 3 n(n  1)(2n  1) n(n  1)

PRINT (s, 2, 3) PRINT (s, 4, 4)

PRINT (s, 2, 2) PRINT (s, 3, 3)

Department of Computer Science 31

 Each entry s[i, j ]=k shows where to split the product Ai

Department of Computer Science 32

P0 = 10 m[1,1] m[1,2] m[1,3] m[1,4]

Department of Computer Science

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

m[1,2]  min (m[1, k ]  m[k  1,2]  p0 . pk . p2 )

m[1,2]  min (m[1,1]  m[2,2]  p0 . p1. p2 )