Professional Documents
Culture Documents
Dynamic Optimization in Continuous
Dynamic Optimization in Continuous
dth
(19) E0 s
U[c(t),k(t)]h,
te
------
t=0
(20)
k(t+h)
------
k(t) = G[c(t),k(t),q(t+h),h],
k(0) given,
------------------------------------------------------------------------------------------------------------------------------------------------------------
that
the
transition
factor.
for
this
time
interval
function,
but
not
appears
as
an
argument
necessarily
as
arises
because
q(t+h)
is
the
multiplicative
of
The need
meant
to
be
Unlike in the
Rather, it is a sequence
The
To
enters the value function for the same reason that q(t)
influences c*(t).
J[k(t);q(t)] = max
(21)
(
)
{U[c(t),k(t)]h +e dhEtJ[k(t+h);q(t+h)]},
9
0
------
c(t)
The rationale
s=t
23
s=t+1
The equation is
But our
We would like to
J[k(t+h);q(t+h)]
h
------
J[k(t);q(t)]
.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0) to a well-
between the present setup and the usual setup of the calculus is
as follows.
Thus,
0.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Stochastic Calculus
Let X(t) be a random variable whose change between periods t
1 and t, DX(t) = X(t)
------
To
simplify
------
X(t
------
matters
Ill
assume
that
distributed, although
this
is
at
not
DX(t)
all
s .
is normally
necessary
for
the argument. 13
We are interested in the case where DX(t), the change in
random variable X over the period of length 1 between t
------
1 and
------
But we need to
------
1, t] into n disjoint
------------------------------------------------------------------------------------------------------------------------------------------------------------
25
these
matters,
(22) DX(t) = s
t
mh + sh
Then
1/2
n(i)
i=1
since
nh
1,
(22)
is
consistent
VDX(t) = s
s
t
with
our
initial
For example,
s En(i)n(j)/n = s En(i)2/n =
t
t
i=1 j=1
s .
i=1
sdz,
where
for
any
instant
t,
dz(t)
0 by
limh 0h1/2n(t).
L
(23)
------
------
So we use
(24)
dX = m(X,t)dt + s(X,t)dz.
------------------------------------------------------------------------------------------------------------------------------------------------------------
------
1) as
DX(t)
------
s n(i)/en ,
= t
q------
i=1
where n = 1/h is the number of increments in [t
1, t]. We know
from the central-limit theorem that as n
8, the right-hand side
above is likely to approach a limiting normal distribution even
if the n(i)s arent normal (so my assumptions above were
stronger than necessary). Obviously, also, X(t) X(t
h) will
2
be normally distributed with variance hs no matter how small h
is. But X(t)
X(t
h) divided by h therefore explodes as h
0
2
(its variance is s /h). This is why the sample paths of
diffusion processes are not differentiable in the usual sense.
27
------
------
------
------
------
differentials.
For
------
y2 = 2yh + h2 as h
0.
discarded.
Since this is
(25)
dzdt = 0.
------
0, and so we write
dz2 = dt.
(26)
^
Lets turn now to Itos
famous lemma.
in
f(Q) is
nonlinear, however,
there is
often
^
Itos
Lemma.
be
twice
differentiable.
differential of f(X) is
(27)
df(X) = f(X)dX +
1
f"(X)dX2.
2
------
------------------------------------------------------------------------------------------------------------------------------------------------------------
h)2 = E(h2n4
------
------
2h2 + h2 = 2h2.
29
The
stochastic
Comment.
(28)
df(X) = [m(x,t)f(X) +
s(X,t)2
-----------------------------------
f"(X)]dt + s(X,t)f(X)dz.
Youll notice that (28) differs from the "naive" chain rule only
in modifying the expected drift in f(X) by a term that depends on
the curvature of f(Q).
------
f[X(t)].
that Etf[X(t+dt)] > f[EtX(t+dt)] for convex f(Q), and that the
^
opposite inequality holds for concave f(Q). So Itos
Lemma
should not come as a surprise.16
------------------------------------------------------------------------------------------------------------------------------------------------------------
30
^
Motivation for Itos
Lemma.
^
The proof of Itos
Lemma is quite
f[X(t+h)]
+
implies that
f[X(t)] = f[X(t)][X(t+h)
------
1
f"{X(t) + x(h)[X(t+h)
2
------
For an
------
X(t)]
------
X(t)]}[X(t+h)
------
X(t)]2,
converges to (27) as h
0.
Beware.
------
X(t)]} on h.
------------------------------------------------------------------------------------------------------------------------------------------------------------
------
sp f(X ) >
t i i
i
f(s
tpiXi) for (p1,...,pn) in the unit simplex.
(Try it.)
So for
17For Taylors theorem with remainder, see any good calculus text.
31
^
motivate a multivariate version of Itos
Lemma using the
multivariate Taylor expansion.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
What is the
Ill
= X(t+h)
the
------
assume
simple
that
q(t+h)
diffusion
Under this
(29)
J[k(t)] = max
(
)
{U[c(t),k(t)]h + e dhEtJ[k(t+h)]}.
9
0
------
c(t)
------
dh.
(
0 = max {U[c(t),k(t)]h + EtJ[k(t+h)]
c ( t )9
Now let h
0.
------
J[k(t)]
------
dE J[k(t+h)]h}.
t
^
for k. Itos
Lemma then tells us that
(30)
dJ(k) = J(k)dk +
Thus as h
1
J"(k)dk2,
2
------
0, EtJ[k(t+h)]
1
J"[k(t)]Etdk(t)2.
2
------
------
J[k(t)]
Furthermore, as h
J[k(t)]Etdk(t) +
0, EtJ[k(t+h)]
J[k(t)].
PROPOSITION III.1.
At each
(31)
0 = U(c*,k)dt + J(k)EtG(c*,k,dX,dt)
1
J"(k)EtG(c*,k,dX,dt)2
dJ(k)dt
2
(
)
1
= max {U(c,k)dt + J(k)Etdk + J"(k)Etdk2
dJ(k)dt}.
2
c ( t )9
0
+
------
------
------
------
(32)
quite
to
that
of
Proposition
II.1.
H(c,k)
_ U(c,k) + J(k)
Etdk
----------------------
dt
Etdk2
1
+ J"(k)
.
2
dt
33
------
--------------------------
Thus the
as 18
H(c,k)
and
if
= U(c,k) + EtdJ(k) ,
dt
you
----------------------------------------
use
the
ordinary
chain
rule
to write the
Q = U(c,k) +
deterministic Hamiltonian (10) as U(c,k) + J(k)k
dJ(k)/dt.
The stochastic Bellman equation therefore implies the same
rule as in the deterministic case, but in an expected-value
sense. Once again, optimal consumption c* satisfies (11),
H(c*,k)
(33)
dk = kdX
------
cdt
s k dt.
(34)
To see
c)dt.
------
(
{U(c,k) + J(k)(rk
c ( t )9
0 = max
------
c) +
1
J"(k)k2s2
2
------
------
dJ(k)}.
we
proceed
as
before,
starting
off
------
by
maximizing
the
Hamiltonian.
Since k
is
(35)
Uc(c*,k) = J(k),
just as before.
35
0 = U[c(k),k] + J(k)[rk
(36)
------
c(k)] +
1
J"(k)k2s2
2
------
------
dJ(k).
The
result is
(37)
Uk(c*,k) + J(k)(r
+
d) + J"(k)ks
------
+ J"(k)(rk
------
c*)
1
J(k)k2s2 = 0,
2
------
l _ J(k).
This shadow price is known at time t, but its change over the
interval from t to t + dt is stochastic. Equation (37) differs
from (13) only by taking this randomness into account; and by
writing (37) in terms of l, we can see precisely how this is
done.
To do so we need two observations.
^
First, Itos
Lemma
36
(38)
dl = dJ(k) = J"(k)(kdX
------
cdt) +
1
J(k)k2s2dt
2
------
(39)
Etdl = J"(k)(rk
------
1
J(k)k2s2.
2
c) +
------
---------------------
dt
Second, the term J"(k)ks2 in (37) can be expressed as
(40)
J"(k)ks2 =
where R(k) _
------
------
J(k)R(k)s2,
Uk(c*,k) + l[r
------
R(k)s
------
d] +
Etdl
---------------------
dt
(41)
Uk + l[r
------
R(k)s2/2] + [(Etdl)/dt
------
lR(k)s2/2]
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
= d,
An
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Example
An individual maximizes the expected discounted utility of
consumption, E i8e dtU(c)dt, subject to a stochastic capital
------
0 0
dk = rkdt + skdz
------
cdt,
k(0) given.
Capital has a
c1 (1/e)
1
U(c) =
,
1
(1/e)
------
------
----------------------------------------------------------------------------
38
The
by
thinking
about
the
implications
of
the
If
dk = rkdt + skdz
------
c(k)dt = (r
------
h)kdt + skdz.
dc = (r
------
h)cdt + scdz.
( )
dl =
2
------2
1 c-1 (1/e)dc +
2
------
------
9 0
2
( )( )(
1 1
1 c-2-(1/e)dc2.
1 +
e
9209 e 09
0
2
22
22
2
22
22
----------2
22
22
------
2
2
------
h)cdt
-----------------------
--------------------------------
------
r +
&1*&
1 * 2$
72871 + e 8s 4
------
------
39
and
dt
= l[d
------
(r
Rs2)] = c-1/e[d
------
(r
------
------
s /e)].
( )
( )
1* 2
r + 1 &
1
+
s
e8
2 7
[d
(r
------
------
s /e)] =
22
22
------
9 0
2
22
2
------
------
------
e
2
2
2
2
c
h = r
e(r
------
------
d) +
(e
------
1)
-----------------------------------------
2e
s .
(Notice
(42)
In
h = (1
(42),
------
e)(r
------
appears
1 2
Rs ) + ed.
2
------
as
the
weighted
40
average
of
the
time-
preference
rate
and
risk-adjusted
expected
return
on
investment.
Problems still arise if h < 0.
In the
^
process well learn some more about the importance of Itos
Lemma. One way to approach this task is to calculate the (random)
path for k under an optimal consumption plan, observe that the
optimal contingency rule for consumption is c = hk, and then use
this formula to compute the optimal (random) consumption path and
lifetime expected utility.
level of k.
------
[Throughout the
first
that
the
optimal
capital-stock
dk/k = (r
------
h)dt + sdz.
A crucial warning.
log[k(t)] = log[k(0)] + (r
------
i
h)t + s dz(s),
j
2
------
------
1 2
s dt = (r
2
------
------
1 2
s )dt + sdz.
2
------
At
2
k(t) = k(0)e(r h s /2)t + s[z(t) z(0)].
------
an
optimum,
------
k(t)
------
will
be
conditionally
------
lognormally
h: E k(t)/k(0) =
0
------
J[k(0)] =
-1
------
------
2
z
2
2
c
(
E0{ i
e dt[hk(t)]1 (1/e)dt
j
9
------
------
------
------
)
}
0
-1
(i dt #
{je E03hk(0)e(r
9
e
2
2
------
------
2
z
2
c
------
1 (1/e)
dt
4
h s /2)t+s[z(t) z(0)]$
2
------
------
------
)
}
0
------
------
------
e
2
------
2
z
------
(
1 (1/e) i dt [1 (1/e)](r
{ hk(0)
je e
9
-1
2
2
c
------
------
------
1)
h s /2e)t
------
------
dt
------
------
d0
------
------2
2
z
2
c
[hk(0)]1 (1/e)
------
-1
------------------------------------------------------------------------------------------------------------------------------------------------------
9
2
------
(e
------
1)(r
------
Rs /2
------
d)
------
------
}.
2
------
Rs2/2
------
(e
------
1)(r
------
Rs2/2
------
Because
------------------------------------------------------------------------------------------------------------------------------------------------------------
21To move from the second to the third equality above, I used the
fact that the normal random variable [1
(1/e)]s[z(t)
z(0] has
2 2
mean zero and variance [1
(1/e)] s t conditional on t = 0
information.
43
------
------
------
[Had
c(k) = J(k) e.
------
Thus by (34),
[J(k)]1 e
0 =
+ J(k)[rk
1
(1/e)
------
---------------------------------------------------------------
1
J(k) e] + J"(k)k2s2
2
------
------
------
------
dJ(k).
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
modest one of illustrating the basic idea, I will spare you this.
------------------------------------------------------------------------------------------------------------------------------------------------------------
22As
does
Jean-Michel
Bismut,
"Growth
and
the
Optimal
Intertemporal Allocation of Risks," Journal of Economic Theory 10
(April 1975): 239-257.
44
PROPOSITION
III.2.
(Stochastic
Maximum
Principle)
Let
d(s t)
E0i
U[c(s),k(s)]ds
je
------
------
k(0) given,
------
c] + z(t)sk(t)
(44)
~
dH
(c*,k,l,z) = Uc(c*,k)
dc
------------
------
l = 0
Furthermore,
the
differential
costate
variable
obeys
the
stochastic
equation
(45)
dl = lddt
= lddt
------
------
~
dH
(c*,k,l,z)dt + zdz
dk
[Uk(c*,k) + lr + zs]dt + zdz
------------
45
for dk = rkdt
To
------
understand
how
this
proposition
follows
from
our
(46)
z = (E dldz)/dt = J"(k)sk.
t
H(c,k)
= U(c,k) + J(k)(rk
------
c) +
1
J"(k)s2k2,
2
------
we have
H~(c,k,l,z)
H(c,k)
1
J"(k)s2k2 =
2
------
H(c,k)
------
lR(k)s k/2,
Clearly (44)
dt and combine the result with (38), you will retrieve (45).
IV.
Conclusion
These notes have offered intuitive motivation for the basic
The Maximum
As I noted
47
you