Professional Documents
Culture Documents
Markov Chains
Markov Chains
Markov Chains
Markov Chains
A discrete time process {Xn , n = 0, 1, 2, . . .}
with discrete state space Xn {0, 1, 2, . . .} is a
Markov chain if it has the Markov property:
P[Xn+1 = j|Xn = i, Xn1 = in1 , . . . , X0 = i0 ]
= P[Xn+1 = j|Xn = i]
In words, the past is conditionally independent
of the future given the present state of the
process or given the present state, the past
contains no additional information on the future
evolution of the system.
The Markov property is common in probability
models because, by assumption, one supposes
that the important variables for the system being
modeled are all included in the state space.
We consider homogeneous Markov chains for
which P[Xn+1 = j | Xn = i] = P[X1 = j | X0 = i].
2
Pij =
P Xn+1 = k | Xn = i P Xn+2 = j | Xn+1 = k
k
=
P Xn+2 = j, Xn+1 = k | Xn = i
k
Xn+2 = j, Xn+1 = k Xn = i
= P
k
= P Xn+2 = j | Xn = i
Generalizing this calculation:
The matrix power Pijn gives the n-step transition probabilities.
(n)
probabilities at time n, so i = P Xn = i .
Then (n) = (0) P n , using standard
multiplication of a vector and a matrix. Why?
Example. Set X0 = 0,
and let Xn evolves as a
Markov chain with transition matrix
1/2 1/2 0
P = 0
0 1
.
1
0 0
(n)
Find 0
1/2
= P[Xn = 0] by
1/2
H
Y
HH
1 HH
1
1
?
2
Classification of States
State j is accessible from i if Pijk > 0 for some
k 0.
The transition matrix can be represented as a
directed graph with arrows corresponding to
positive one-step transition probabilities j is
accessible from i if there is a path from i to j.
For example,
- ...
ous by considering a
generic directed graph
for a periodic state:
- i+1
- i+2
6
i+d-1
...
?
...
State i is recurrent if P re-enter i | X0 = i] = 1,
where {re-enter i} is the event
S
n=1 {Xn = i, Xk 6= i for k = 1, . . . , n 1}.
If i is not recurrent, it is transient.
Let S0 = 0 and S1 , S2 , . . . be times of successive
returns to i, with Ni (t) = max {n : Sn t} being
the corresponding counting process.
If i is recurrent, then Ni (t) is a renewal process,
since the Markov property gives independence of
interarrival times XnA = Sn Sn1 .
Letting ii = E[X1A ], the expected return
time for i, we then have the following from
renewal theory:
10
Piin = .
11
12
13
Random Walks
The simple random walk is a Markov chain
on the integers, Z = {. . . , 1, 0, 1, 2, . . .} with
X0 = 0 and P Xn+1 = Xn + 1 = p,
P Xn+1 = Xn 1 = 1 p.
(d)
15
Stationary Distributions
= {i , i = 0, 1, . . .} is a stationary
P
distribution for P = [Pij ] if j = i=0 i Pij
P
with i 0 and i=0 i = 1.
P
In matrix notation, j = i=0 i Pij is = P
where is a row vector.
Proof
16
Proof continued
17
18
Pn
1
E h(X) n i=1 h(Xi )
If it is hard to generate an iid sample from ,
we may look to generate a sequence from a
Markov chain with limiting distribution .
This idea, called Monte Carlo Markov
Chain (MCMC), was introduced by
Metropolis and Hastings (1953). It has become a
fundamental computational method for the
physical and biological sciences. It is also
commonly used for Bayesian statistical inference.
19
Metropolis-Hastings Algorithm
(i) Choose a transition matrix Q = [qij ]
(ii) Set X0 = 0
(iii) for n = 1, 2, . . .
20
Proposition: Set
j 6= i
qij min(1, j qji /i qij )
Pij = q + Xq {1 min(1, q / q )} j = i
ik
k ki
i ik
ii
k6=i
Proof.
21
1
= 2 +
(p,q)N (i,j) I{Ypq = R} 2 .
The full distribution of Y is, in principle,
specified by the conditional distributions. In
practice, one simulates from using MCMC,
where the proposal distribution is to pick a
random (i, j) and swap it from R to D or vice
versa.
22
posterior mean E | X= x ,
such that P A | X= x = 1 .
23
Zn+1 Z.
(k)
24
25
d2
ds2
26
d
ds
> 0. Why?
> 0. Why?
27
0 w.p.
Z=
1 w.p.
2 w.p.
Find the chance
1/4
1/4 .
1/2
of extinction.
28
Time Reversal
Thinking backwards in time can be a powerful
strategy. For any Markov chain
{Xk , k = 0, 1, . . . , n}, we can set Yk = Xnk .
Then Yk is a Markov chain. Suppose Xk has
transition matrix Pij , then Yk has
[k]
inhomogeneous transition matrix Pij , where
[k]
Pij = P Yk+1 = j | Yk = i
= P Xnk1 = j | Xnk = i
=
P Xnk = i
Proof
Proof
31
Let P Xn = Xn +1 | Xn = i = pi and
P Xn = Xn 1 | Xn = i = qi = 1 pi .
If {Xn } is positive recurrent, it must
be reversible. Heuristic reason:
..
.
4
p3
q4
3
p2
q3
p1
q2
p0 =1
q1
j 6= i
qij min(1, j /i )
X
Pij =
qik {1 min(1, k /i )} j = i
qii +
k6=i
Solution
33
2
Z
Z 1
Z
Z 2
Z
Z
Z
2 Z
Z
5
34
35
Ergodicity
A stochastic process {Xn } is ergodic if limiting
time averages equal limiting probabilities, i.e.,
time in state i up to time n
lim
= lim P[Xn = i].
n
n
n
Show that an irreducible, aperiodic, positive
recurrent Markov chain is ergodic (i.e., this
general idea of ergodicity matches our definition
for Markov chains).
36
Semi-Markov Processes
A Semi-Markov process is a discrete state,
continuous time process {Z(t), t 0} for which
the sequence of states visited is a Markov chain
{Xn , n = 0, 1, . . .} and, conditional on {Xn }, the
times between transitions are independent.
Specifically, define transition times S0 , S1 , . . . by
S0 = 0 and Sn Sn1 Fij conditional on
Xn1 = i and Xn = j. Then, Z(t) = XN (t) where
N (t) = sup {n : Sn t}.
{Xn } is the embedded Markov chain.
The special case where Fij Exponential (i ) is
called a continuous time Markov chain
The special
case when
0, t < 1
Fij (t) =
1, t 1
results in Z(n) = Xn , so we retrieve the discrete
time Markov chain.
37
38
i
lim P Z(t)= i | Z(0)= j =
t
ii
time in i before t
= lim
w.p. 1
t
t
39
Proof
40
41
Proof (continued)
42