Notes on Dynamic Programming

Tristany Armangue-Jubert

1 The Neoclassical Growth Model

Recall that the problem of the planner in the NGM was:
∞
X
max β t u (F (kt , 1) + kt (1 − δ) − kt+1 )
{kt+1 }∞
t=0 t=0
s.t. k0 given

Now, let us name this maximized value function, which tells us the “value” of having
k0 :
X∞
V (k0 ) = max∞ β t u (F (kt , 1) + kt (1 − δ) − kt+1 )
{kt+1 }t=0
t=0

This should satisfy the following functional equation:

u (F (kt , 1) + kt (1 − δ) − kt+1 ) + βV (k ′ )

V (k) = max ′k

The reason this holds is because of consistency: if V (k) is the maximized value function
when the state is k, it must also be the maximized value function when the state is k ′ .
More formally, let us assume u is bounded. Then a problem of the form:
∞
X
v(x̃) = β t u(xt , xt+1 ),
t=0

where x̃ = (x0 , x1 , ...). Let x̃′ = (x1 , x2 , ...). Can be written as

∞
X
v(x̃) = u(x0 , x1 ) + β β t u(xt+1 , xt+2 )
t=0
′
= u(x0 , x1 ) + βv(x̃ )

1
In the context of the NGM, this is just saying that:
∞
X
β t u (F (kt , 1) + kt (1 − δ) − kt+1 ) = u(F (k0 , 1) + k0 (1 − δ) − k1 )
t=0
∞
X
+β β t u (F (kt+1 , 1) + kt+1 (1 − δ) − kt+2 )
|t=0 {z }
PV of sequence of k starting from k1

2 Introduction to Dynamic Programming

The problems that we are interested in are of the following form:
∞
X
max β t R(xt , xt+1 )
{xt+1 }∞
t=0 t=0
s.t. xt+1 ∈ Γ(xt )

Where:
1. xt ∈ X for all t
2. Γ : X → X is a map from xt to a subset of X (feasible actions).
3. R : X × X → R is a return function that maps any two elements from X to the
real line (generally a utility function in our case)
The above problem can be restated recursively as:

V (x) = max R(x, x′ ) + βV (x′ )


(1)
x′ ∈Γ(x)

Where x is the state variable and x′ is the choice variable. From the above, it follows
that the object of interest is now V . That is, we are interested in solving the above
functional equation (FE) to find the V .
Note that in the above representation, V (x) represents the value of entering a given
period with state x. Thus, solving for V is solving for the value of entering a period
with any state x ∈ X.

2.1 Solving the FE

Notice that the problem represented by equation 1 defines an operator on functions,
which may be written as:

(T V )(x) = max R(x, x′ ) + βV (x′ )


x′ ∈Γ(x)

2
Where T V is a new function given V .
The method that we will use to solve the functional equation is to iterate on the operator.
As we will show in what follows:
1. {V n (x)} → V
2. T V = V
That is, iterating the operator will lead to convergence to a limit function V and that
V is a fixed point of operator T . Finally, we will also show that V is a solution of the
problem specified in equation 1.

2.2 Mathematical Background

Recall that we are interested in showing that in our problems of interest, we can solve
the FE by iterating the value functions. To that end, we will exploit:

Theorem 2.1: Banach Fixed Point Theorem (Contraction Mapping Theorem)

If (S, ρ) is a complete metric space and T : S → S is a contraction mapping with

modulus β, then:
ˆ T has exactly one fixed point V in S.
ˆ For any V0 ∈ S: ρ(T n V0 , V ) ≤ β n ρ(V0 , V ).

The above, as related to our problem of interest, says that T has exactly one fixed point
V and that we can converge to V by iterating the operator regardless of our initial guess.
However, we require that:
ˆ (S, ρ) be a complete metric space. In our case S is a set of functions and ρ a metric
such as the supremum metric.
ˆ T must map a complete metric space to itself.
ˆ T must be a contraction mapping.

2.2.1 Complete Metric Spaces

Definition 2.1: Metric space

A metric space is a set S together with a metric ρ : S × S → R such that for all
x, y, z ∈ S:
ˆ ρ(x, y) ≥ 0 with equality only if x = y.
ˆ ρ(x, y) = ρ(y, x).

3
 ˆ ρ(x, z) ≤ ρ(x, y) + ρ(y, z)

In what follows, we will just accept that that (S, ρ), where S is the set of continuous
and bounded functions from A to R and ρ(x, y) = supt∈A |x(t) − y(t)| (the sup-metric),
is a metric space.
Now, a complete metric space is a metric space (S, ρ) where all Cauchy sequences have
a limit in (S, ρ).
Theorem 2.2: C(X) with the sup-metric is a complete metric space

Let X ⊆ Rn and let C(X) be the set of bounded continuous functions f : X → R

with the sup-metric ρ(x, y) = supt∈X |f (t)−g(t)|. Then, C(X) is a complete metric
space.

2.2.2 Contraction Mappings

Let us now turn to operators, such as the T we defined as:

(T V )(x) = max R(x, x′ ) + βV (x′ )


x′ ∈Γ(x)

Definition 2.2: Contraction Mapping

Let (S, ρ) be a metric space and T : S → S be a function mapping S into itself. T

is a contraction mapping with modulus β if for some β ∈ (0, 1):

ρ(T x, T y) ≤ βρ(x, y) ∀x, y ∈ S

Thus, a contraction maps points closer together:

4
 Figure 1: T is a contraction

Let us turn to showing that a function T is a contraction.

Theorem 2.3: Blackwell’s sufficiency conditions

Let X ⊆ Rn and let B(X) be the set of bounded functions f : X → R with the
metric ρ(x, y) = maxt∈X |f (t) − g(t)|.
Let T : B(X) → B(X) satisfy:
ˆ ∀f, g ∈ B(X) such that f (x) ≤ g(x) for all x ∈ X implies:

(T f )(x) ≤ (T g)(x) ∀x ∈ X

ˆ ∃β ∈ (0, 1) such that:

(T (f + a))(x) ≤ (T f )(x) + βa ∀f ∈ B(X)

Then T is a contraction mapping.

2.2.3 Summarizing
Consider once again the DP problem:

V (x, x′ ) = max {R(x, x′ ) + βV (x′ )}

x′ ∈Γ(x)

Let us introduce some requirements that guarantee that the solution to the FE exists,

5
is unique and has some desirable properties.
R1: X is a convex subset of Rn and Γ : X → X is a non-empty, compact-valued and
continuous correspondence.
R2: R is bounded and continuous and β ∈ (0, 1). This in turn ensures the value function
V will also be bounded.
Under requirements R1 and R2:
ˆ T : C(X) → C(X)
ˆ T has exactly one fixed point V in C(X).
ˆ For any V0 ∈ C(X), ρ(T n V0 , V ) ≤ β n ρ(V0 , V ) for n = 0, 1, 2, ...
ˆ G(x) = {x′ ∈ Γ(x) : V (x) = R(x, x′ ) + βV (x′ )} is compact valued and upper
hemicontinuous.
R3: ∀x′ ∈ X, R(,̇x′ ) is strictly increasing. (R is strictly increasing in the current
state).
R4: Γ is monotone: x ≤ x′ ⇒ Γ(x) ⊆ Γ(x′ ). (Feasibility set is expanding in the current
state).
Under R1-R4 V is strictly increasing.
R5: R is strictly concave.
R6: Γ is convex.
Under R1-R2 and R5-R6, V is strictly concave and G is continuous and single-valued.
R7: R is continuously differentiable in the interior of A = {(x, x′ ) ∈ X × X : x′ ∈
Γ(x)}.
Under R1-R2 and R5-R7, for any x0 ∈ X o such that g(x0 ) ∈ Γ(x0 )o , V is continuously
differentiable at x0 .

3 Computation
Recall that we are interested in solving problems of the form:

V (x) = max R(x, x′ ) + βV (x′ )


x′ ∈Γ(x)

We know that under some requirements on X, R, Γ and β, we can exploit the contraction
mapping theorem and iterate on the operator implied by the above representation to find
V ∗ , the function that satisfies the functional equation.
Before starting, let us look at a simpler application of the CMT. Suppose we are inter-
ested in finding the fixed point of f (x) = x2 on R with the Euclidean distance. Since we

6
have a contraction map on a complete metric space, we know there will be exactly one
fixed point and that we can get to it by starting from any initial guess x0 .
Computationally, this is simple:
# Dependencies
using CSV, DataFrames

# Define the function

f(x) = x/2

# Empty matrix for results

r = zeros(100,5)

# Pick starting guesses

r[1,2:5] = [-14, 124, 69, 2]

# For each guess iterate

for iter = 2:100
r[iter,2:5] = map(f, r[iter-1,2:5])
r[iter,1] = iter-1
end

# Save for plots

CSV.write("ex1.csv", DataFrame(r,:auto))

Plotting the results:

-14
124
100 69
2

50

0 1 2 3 4 5 6 7 8 9 10

Figure 2: Convergence to fixed point from different starting guesses

In the previous example, I show 4 starting guesses only to highlight the fact that the
only effect a starting guess has is on speed of convergence. Generally, since we know
that any starting guess will suffice, we will only make one.

7
3.1 Simple VFI
Let us now return to our object of interest, the functional equation. Now we have a
map from a set of functions onto itself. Note that we cannot apply our operator on a
continuous function on the computer to obtain a new function.
Consider the neoclassical growth model:

V (k) = max {ln(f (k) − k ′ ) + βV (k ′ )}

k′ ∈[0,f (k)]

where f (k) = k α + (1 − δ)k. To solve the above, we must discretize the state space such
that we can operate on our function V .
To that end, we typically select N points around k ∗ . Naturally, the larger the N the
higher the computational burden.
Given this discrete set K, we can define the following iteration rule:

V n (ki ) max ln(kiα + (1 − δ)ki − k ′ ) + βV n−1 (k ′ )


′
k ∈K

Thus, for every possible k, we check every possible k ′ and record which one attains
the maximum. After that, we check the distance between T V and V to check for
convergence.
Computationally, this may be done as:
# Dependencies
using CSV, DataFrames

# Settings
max_iter = 2000
tolerance = 0.00001

# Parameters
const beta = 0.9
const alpha = 0.75
const delta = 0.3

# Discretize the space

grid_K = LinRange(9, 12, 50)

# Initiate the value function guess

global V = zeros(length(grid_K),1)
global g = zeros(length(grid_K),1)

# Outer loop - iterations

for iter = 1:max_iter
# Placeholder for updated value function
global TV = zeros(length(grid_K),1)

8
 # Inner loop - state space
for (idx_k, k) in enumerate(grid_K)
# Empty array for all guesses for this k
U = zeros(length(grid_K))

# Loop over all possible choices

for (idx_kp, kp) in enumerate(grid_K)
# Consumption at this k, k' pair
c = k^(alpha) + (1-delta)*k - kp

# If consumption is positive -> valid choice

if c > 0
# Compute value at this pair (utility plus V)
U[idx_kp] = log(c) + V[idx_kp]
else
# Consumption non positive -> cannot be optimal
U[idx_kp] = -9e8
end
end

# Select the choice that attains the maximum

ax, bx = findmax(U)
TV[idx_k] = ax
g[idx_k] = bx
end

# Compute deviation
dev = maximum(abs.(TV-V))

# Assign for next period

global V = deepcopy(TV)

# Check convergence
if dev <= tolerance
break
end
end

# Save policy function for plotting

df = DataFrame(
x=grid_K,y=collect(grid_K[Int(g[s_i])] for s_i in 1:length(grid_K)),z=grid_K
)
CSV.write("ex2.csv", df)

And the resulting policy function will look as follows:

9
 13
Policy Function
45-degree line
12

11
k′

10

9
9 9.5 10 10.5 11 11.5 12 12.5 13
k

Figure 3: g(k)

10