Stat 150 Stochastic Processes Spring 2009

Lecture 22: Continuous Time Markov Chains

Lecturer: Jim Pitman
Continuous Time Markov Chain
Idea: Want a model for a process (X
t
, t 0) with continuous time parameter t.
State space a nite/countable set S
Markov property (time homogenous transition mechanism)
Strong parallels with discrete time
Expression of Markov: conditionally given that X
t
= j, the past process
(X
s
, 0 s t) and the future (X
t+v
, v 0) are conditionally independent,
and the future is a distinct copy of (X
v
, v 0) given X
0
= j.
Notations
Recall in discrete P
n
= n step transition matrix, we need now P
t
= time t
transition matrix.
Conceptual challenge: raising a matrix to a real t power such as t = 1/2, 1/3.
Use notation P(t) instead of P
t
for the transition matrix over time t. Idea: P(t)
is something like a tth power:
P(s)P(t) = P(s + t).
This semigroup property of transition probability matrices follows easily from
the Markov property:
P
ij
(s + t) :=P
i
(X
s+t
= j)
=

k
P
i
(X
s
= k, X
s+t
= j)
=

k
P
i
(X
s
= k)P
k
(X
t
= j)
Basic examples: The Poisson process as a Markov chain.
Let (N
t
, t 0) be a Posson process with N
0
= 0 and rate .
1
Lecture 22: Continuous Time Markov Chains 2
t
N
t





S
1
S
2
S
3
S
4
S
5
W
1
W
2
W
3
W
4
W
5
S
1
, S
2
, . . . independent exp().
Obserbe (N
t
, t 0) is a MC with transition matrices
P
ij
(t) = P
0
(N
s+t
= j|N
s
= i)
= P
0
(N
s+t
N
s
= j i|N
s
= i)
= P
0
(N
s+t
N
s
= j i) since N
s+t
N
s
N
s
= P
0
(N
t
= j i)
= e
t
(t)
ji
(j i)!
1(j i)
Independence property of PP = (N
t
, t 0) is a MC in state space {0, 1, . . . }
with transition matrix P
ij
(t) = e
t
(t)
ji
(j i)!
1(j i).
Distinguish carefully two features of this process:
(1) The semi-group (P(s+t) = P(s)P(t)) of transition P(t), t 0: the entries
t P
ij
(t) are very smooth(dierentiable) functions of t.
t
1
0
P
ii
(t)=e
t
P
i,i+1
(t)
P
i,i+2
(t)
(2) The path of the process t N
t
is a step function.
Lecture 22: Continuous Time Markov Chains 3
t



Recall:
Discrete time :P
n+1
= P
n
P = PP
n
Continuous time :P(n + 1) = P(n)P(1) = P(1)P(n)
P(t + h) = P(t)P(h) = P(h)P(t) with h small
We should evaluate
d
dt
P(t) := lim
t0
P(t + h) P(t)
h
. We write this as a conve-
nient abbreviation for
d
dt
P
ij
(t) = lim
t0
P
ij
(t + h) P
ij
(t)
h
, i, j S.
d
dt
P(t) = lim
t0
P(t + h) P(t)
h
= lim
h0
P(t)
P(h) I
h
= P(t) lim
h0
P(h) I
h
= P(t)A, where A := lim
h0
P(h) I
h
=
d
dt
P(t)

t=0
Note that
d
dt
P(t) = P(t)A = AP(t). These are Kolmogorov forward and
backward dierence equations for a Markov semi-group. The matrix
A is called the innitesimal generator of the semi-group. It should be clear
that A is rather important. If you know A, you can gure out P(t) by solving
Kolmogorovs equations.
Compute A for the PP:
A
i,i
=
A
i,i+1
=
A
i,i+2
= 0
A
i,i+3
= 0
.
.
.
Lecture 22: Continuous Time Markov Chains 4
A
i,j
=
_

_
if j = i
if j = i + 1
0 else
Note that

j
P
ij
(t) = 1 =
d
dt

j
P
ij
(t) = 0 =

j
d
dt
P
ij
(t) = 0
Take t = 0 in this,
d
dt
P
ij
(t)

t=0
= A
ij
=

j
A
ij
= 0.
Notice A
ii
0 for all i, all transition matrix P(t).
P
ii
(0) = 1
P
ii
(t) 1
_
=
d
dt
P
ii
(t)

t=0
0
Customary notations: (see Pg. 396 of text)
q
i
:=A
ii
q
ij
:=A
ij
0 for all j = i
q
i
=

j=i
q
ij
Key idea: Start a continuous time chain in state i,
it will hold there for an exponentially distributed sojourn time with exp(q
i
),
mean 1/q
i
.
independent of length of this sojourn time, when it leaves for the rst time,
it jumps to j with probability q
ij
/q
i
.
A =
_

_
q
1
q
12
q
13

q
21
q
2
q
23

.
.
. q
3
.
.
.
.
.
.
_

_
Now sum = 0, diagonal D
ij
= q
i
0, o diagonal d
ij
= q
ij
0,

j=i
q
ij
= q
i
,

j=i
q
ij
/q
i
= 1.
Example: A Poisson process that starts in state i
hold in state i for exp(), = q
i
= A
ii
only way to leave state i is by jumping to i + 1, q
i,i+1
/q
i
= / = 1
Lecture 22: Continuous Time Markov Chains 5
for other states j, q
ij
/q
i
= 0/ = 0
Transition rate diagram: Mark o diagonal entries of A.
Two state MC: A =
1
2
_


_

#
"

1
2
Three state MC on a circle: N
(3)
t
= N
t
mod 3, N
t
= PP().
2
0
1

General 3 state chain:

2
0
1
q
20
q
02
q
01
q
10
q
12
q
21