04.

Dynamic Programming
 Learning Objectives
When you complete this chapter you should be able to
understand:
• Understand the concept of dynamic programming
• Know characteristics of dynamic programming
• Know applications of dynamic programming

2
Dynamic programming – The concept
• Complex problems are sometimes solved quickly if approached in a
sequential manner.

• Dynamic Programming (DP) is a sequential or multistage decision making

process. i.e. in DP decisions are made at a number of stages.
• DP transforms a complex problem into a sequence of simpler problems.

• The concept of DP is:

– Problem is divided into smaller sub-problems

– Optimize these sub-problems without losing the integrity of the original

problem.

– Decisions made in a stage will affect the decisions at the subsequent stages
but are independent to each other.

Example: An N variable problem is represented by N single variable problems.

These problems are solved successively to get the optimal value of the original
Problem.
Dynamic programming – The concept
• Consider a single stage decision process
– Here,
S1 is the input state variable
S2 is the output state variable
X1 is the decision variables, and
NB1 is the net benefit

The transformation function for the input and output is

S2 = g(X1, S1)

Net benefits are expressed as a function of decision variables and

input variable
NB1 = h(X1, S1)
Dynamic programming – The concept
• Now, consider a multistage decision process consisting of T stages

• For the tth stage the stage transformation and the benefit functions are
– St+1 = g(Xt, St)
– NBt = h(Xt, St)
• Objective of this multistage problem is to find the optimum values of all
decision variables X1, X2,…, XT such that the individual net benefits of each
stage that is expressed by some objective function, f(NBt) and the total net
benefit which is expressed by f(NB1, NB2,…, NBT) should be maximized.
Dynamic programming – The concept
• Therefore, dynamic programming can be applied to this multistage
problem if the objective function is separable and monotonic.

• An objective function is separable, if it can be decomposed and expressed

as a sum or product of individual net benefits of each stage.

Satisfying the equations,

 St+1 = g(Xt, St)
 NBt = h(Xt, St) for t = 1,2,…,T
 Application areas of DPP
• Allocating scarce resources. E.g. water allocation to users

• Shortest route problem. E.g Water supply distribution

• Production scheduling problem E.g. Critical path in CPM and PERT

• Capacity expansion problem. E.g. Road expansion, Water supply

• Reservoir operation problem

• Equipment replacement problem

Approach to DPP
• The process of multi-stage optimization is based on the Bellman’s principle
of optimality which states that

“An optimal policy (or a set of decisions) has the property that
whatever the initial state and initial decision are, the remaining
decisions must constitute an optimal policy with regard to the
state resulting from the first decision.”
 How to solve a DPP?
• Recursive equations are used to solve a DPP in sequence.
– Recursive equations are used to structure a multistage decision
problem as a sequential process.
– Each recursive equation represents a stage at which a decision is
required.
– A series of equations are successively solved, each equation
depending on the output values of the previous equations.
• Recursive equations for a multistage decision process can be formulated in
• a backward manner, and
• a forward manner
• Backward recursion - A problem is solved by writing equation first for the
final stage and then proceeding backwards to the first stage.
• Forward recursion - The problem is solved by starting from stage 1 and
proceeding towards the last stage.
 How to solve a DPP? – Backward Recursion

S3 = S2-X2

S2 = S1-X1
 How to solve a DPP? – Forward Recursion

S1 = S2-X2

S2 = S3-X3
 How to solve a DPP?
To explain the concept of backward and forward recursion
approach, take the following Water Allocation Problem
• Consider a canal supplying water for three different Fields
for crop production.
• Maximum capacity of the canal is Q units of water
• Amount of water allocated to each field as Xi
How to solve a DPP?

• Let net benefits (returns) from producing the crops are

expressed as a function of the water allocated.

NB1 x1   5 x1  0.5 x12

NB2 x2   8 x2  1.5 x22

NB3 x3   7 x3  x32

• Optimization Problem: Determine the optimal allocations

xi to each crop that maximizes the total net benefits from
all the three crops.
How to solve a DPP?

• Objective function: To maximize the net benefits

3
Max NBi  xi 
i 1

• Subjected to the constraints

x1  x2  x3  Q
0  xi  Q for i  1, 2, 3

• Let f1(Q) be the maximum net benefits that can be

obtained from allocating water to crops 1, 2 and 3

f1 (Q)  Max  NBi xi 

 3

x1  x2  x3 Q 
 i 1 
x , x , x 0
1 2 3
 How to solve a DPP?
A. Backward Recursion:
 How to solve a DPP?
A. Backward Recursion:
  
f1 (Q)  Max  NB1 x1  
0 x1 Q  S1 
Max  NB2 x2   Max NB3 x3 
S1 Q 
0 x2 Q  x1  S 2
0 S 2 Q  0 x3  S 2  x2  S3
0 S 3 Q 

Transforming this into three problems each having only one

decision variable:
• Now starting from the last stage, let f3(S3) be the maximum
net benefits from crop 3
• State variable S3 for this stage can vary from 0 to Q
How to solve a DPP?

• Thus,

f 3* ( S3 )  Max NB3 x3 

0 x3  S3
0 S 3 Q

• But S3=S2-x2. Therefore, f3(S3) = f3(S2-x2)

• Hence, the maximum benefits derived from crops 2 and 3
for a given quantity S2 which can vary between 0 and Q

f 2* ( S 2 )  Max NB x   f
0  x2  S 2
2 2
*
3

( S 2  x2 )
0 S 2 Q
How to solve a DPP?

• Again,S2=S1-x1= Q-x1. Therefore, f2(S2)=f2(S1-x1). And, S1 =Q.

• Hence, the maximum benefits derived from crops 1, 2 and
3 for a given quantity S1 which is equal to Q


f 1* ( S1 )  f 1* (Q)  Max NB1 x1   f 2* ( S1  x1 )
0 x1 Q

S1 Q

Note:
Once the value of f3(S3) is calculated the value of f2(S2) can be
determined from which f1(Q) can be determined.
 How to solve a DPP?
A. Forward Recursion:
 How to solve a DPP?
A. Forward Recursion:
• Let the function fi(Si)be the total benefit from crops 1 to i
for a given input of Si which is allocated to those crops.
• Considering the first stage,
f 1* ( S1 )  Max NB1 x1 
0 x1  S1
0 S1 Q

• But S1=S2-x2
• Considering the first two crops, for an available quantity of
S2 , f2(S2) can be written as:
f 2* ( S 2 )  Max
0  x2  S 2
NB x   f
2 2 1
*

( S 2  x2 )
0 S 2 Q
How to solve a DPP?

• But S2=S3-x3
• Considering the whole system, f3(S3) can be expressed as:

 
f 3* ( S3 )  Max NB3 x3   f 2* ( S3  x3 )
0 x3  S3
S 3 Q
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem

• Net benefits from producing the crops can be expressed

as a function of the water allotted. They are calculated as
below.
NB1 ( x1 )  5 x1  0.5 x12
NB2 ( x2 )  8 x2  1.5 x22
NB3 ( x3 )  7 x3  x32
xi NB1(x1) NB2(x2) NB3(x3)
0 0.0 0.0 0.0
1 4.5 6.5 6.0
2 8.0 10.0 10.0
3 10.5 10.5 12.0
4 12.0 8.0 12.0
Numerical Example 1 . Resource Allocation Problem
Representation of the problem as a set of nodes and links
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem

Case A. Backward Recursion

• Sub-optimization function for the 3rd crop:
f 3 ( S3 )  Max NB3 x3 
0 x3  S 3
0 S 3 Q
Numerical Example 1 . Resource Allocation Problem
• Considering the last two stages together, the sub-
optimization function is

f 2* ( S 2 )  Max NB x   f
0  x2  S 2
2 2
*
3

( S 2  x2 )
0 S 2 Q
Numerical Example 1 . Resource Allocation Problem
Numerical Example 1 . Resource Allocation Problem

• Considering all the three stages together,


f 1* (Q)  Max NB1 x1   f 2* ( S1  x1 )
0 x1 Q

S1 Q
Numerical Example 1 . Resource Allocation Problem

Check :
NB1 ( x1 )  5 x1  0.5 x12  5 1  0.5 12  4.5
NB2 ( x2 )  8 x2  1.5 x22  8 1  1.5 12  6.5
NB3 ( x3 )  7 x3  x32  7  2  2 2  10
3
Max NBi  4.5  6.5  10  21
i 1
Numerical Example 1 . Resource Allocation Problem

Case B. Forward Recursion

• Start from the first stage and proceed towards the final
stage
• Sub-optimization function for the first stage

f 1* ( S1 )  Max NB1 x1 

0 x1  S1
0 S1 Q

• Range of values for S1 is from 0 to 4.

• The calculations are shown in the table
Numerical Example 1 . Resource Allocation Problem
 Numerical Example 1 . Resource Allocation Problem

 NB2 x2  
f ( S 2 )  Max  *
*

0  x2  S 2  f ( S  x )
2
0 S 2 Q  1 2 2 
Numerical Example 1 . Resource Allocation Problem

 
f 3* ( S3 )  Max NB3 x3   f 2* ( S3  x3 )
0 x3  S3
with S3  4
S 3 Q
Numerical Example 1 . Resource Allocation Problem
 Example 2. Short Route Problem
Consider A traveler wishes to minimize the length of a journey
from town A to J.

Can you find shorter route?

The total number of routes to be tested: 3321=18

 Example 2. Short Route Problem
Stage 1:
S1 : Node in stage 1
x1 : node from which S1 is reached
d(x1,S1) : distance between nodes x1 and S1
f*1(S1) : Minimum distance from source node A to node S1
: min[d(x1,S1)]

S1 x1 d(x1,S1) f1* (S1) x1*

B A 2 2 A
C A 4 4 A
D A 3 3 A
 Example 2. Short Route Problem
Stage 2:
S2 : Node in stage 2
x2 : node from which S2 is reached
d(x2,S2) : distance between nodes x2 and S2
f *2(S2) : Minimum distance from source node A to node S2
: min[d(x2,S2)+ f *1(x2)]
 Example 2. Short Route Problem

S2 x2 d(x2,S2) f1* (x2) d(x2,S2)+f1* (x2) f2* (S2) x2*

E B 7 2 9
C 3 4 7 7 C, D

D 4 3 7
F B 4 2 6
C 2 4 6 4 D

D 1 3 4
G B 6 2 8
C 4 4 8 8 B,C,D

D 5 3 8
 Example 2. Short Route Problem
Stage 3:
S3 : Node in stage 3
x3 : node from which S3 is reached
d(x3,S3) : distance between nodes x3 and S3
f3(S3) : Minimum distance from source node A to node S3
: min[d(x3,S3)+ f *2(x3)]
 Example 2. Short Route Problem

S3 x3 d(x3,S3) f2* (x3) d(x3,S3)+f2* (x3) f3* (S3) x3*

H E 1 7 8
F 6 4 10 8 E

G 3 8 11
I E 4 7 11
F 3 4 7 7 F

G 3 8 11
 Example 2. Short Route Problem
Stage 4:
S4 : Node in stage 4
x4 : node from which S4 is reached
d(x4,S4) : distance between nodes x4 and S4
f4(S4) : Minimum distance from source node A to node S4
: min[d(x4,S4)+ f *3(x4)]

S4 x4 d(x4,S4) f3* (x4) d(x4,S4)+f3* (x4) f4* (S4) x4*

J H 3 8 11
I 4 7 11 11 H,I
 Example 2. Short Route Problem
The optimal route is traced back:

 From the last table, to reach the destination J, we must come

from the node x4*= H, I take, H
 From stage 3 with S3=H, we reach x3*= E
 From stage 2 with S2=E, we reach x2*= C, D Take C
 From stage 1 with S1= C, we reach x1*= A

 The shortest path is thus, A – C – E – H – J and the shortest distance

is 11.

 But alternatively, we have A – D – F – I – J (if we take I at stage 4)

 Further, we have A – D – E – H – J (if we take D at stage 2)

 Example 2. Short Route Problem

Backtrack the shortest path

B E

A C F J

D G

Route A-C-E-H-J: 4+3+1+3=11

Route A-D-F-I-J: 4+3+1+3=11
Route A-D-E-H-J: 4+3+1+3=11
Thank You!
Questions?

45
 Home Take Assignment 6
1. A total of 6units of water is to be allocated optimally to three users. The
allocation is made in discrete steps of one unit ranging from 0 to 6. With the
three users denoted as user 1, user 2 and user 3 respectively. The returns
obtained from the users for a given allocation are given in the following table.
Find allocations to the three users so that the total return is maximized.
Amount of water allocated Return from

User 1 User 2 User 3

B1 (x) B2 (x) B3 (x)
0 0 0 0
1 5 5 7
2 8 6 12
3 9 3 15
4 8 -4 16
5 5 -15 15
6 0 -30 12

Answer: x1 = 2; x2 = 1; x3 = 3 and Maximum return = 28

 Home Take Assignment 6
2. Find the minimum cost of travel from start to End. The Petrol
and the hotel cost is given.
 Home Take Assignment 6
3. The government is planning to implement a mix of four
prioritized developmental projects with 3 million euros
obtained from Africa Development Bank. The profits that can
be gained by implementing the projects with 1 to 3 million
euros allocated are shown below. What is the mix of projects
to develop so that the return will be maximized?

x Proj #1 Proj #2 Proj #3 Proj #4

0 0 0 0 0
1 2 5 4 2
2 8 6 4 4
3 9 7 4 5