Lecture 17 : Long run behaviour of Markov chains

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Vincent Gee <>

Basic Case: S is nite Markov matrix is P Assume that for some power of P has all entries > 0: k such that P k (i, j ) > 0i, j S Such P is called regular Then (Theorem): a unique stationary probability distribution such that: P = , meaning
i

i P (i, j ) j S

This is then the limit distribution of Xn as n , no matter what the initial distribution of X0 i.e. lim P n (i, j ) = lim Pi (Xn = j ) = j for all j S .
n n

Idea of most proofs: Show that exists Show lim P n (i, j ) = j

n

Uniqueness of is easy. Explicit representations of My favorite (not in text): let Ti = inf {n : n > 1, Xn = i} where inf = . Then Ei (# of visits to j before Ti ) = j i where is the unique invariant probability distribution Notice: by
j 1 we get Ei (Ti ) = , hence i

i =

1 Ei (Ti )

17-1

Lecture 17: Long run behaviour of Markov chains

17-2

This brings us to discussion of how to solve P = . In general, for complicated P , this can be a pain. But a simplifying method is often available: look for a reversible equilibrium. This may not exist, but if it does, its easy to compute. Reversible Equilibrium Recall: = P means for the Markov chain (X0 , X1 , ...) with transition matrix P that X0 = X1 That is X0 = X1 (equality in distribution) Ordinary Equilibrium: = Xn = X0 But sometimes, we also have Reversible Equilibrium (x0 , x1 ) = (x1 , x0 ) (equality of joint distributions) Look at the probability that either side has value (i, j ) to derive the equations i P (i, j ) = j P (j, i) for all pairs of states i, j Note: if |S | = N , we used to have N equations and N unknowns. Now N 2 =
N (N 1) N
d d d

equations, N unknowns
d

Note that reversibility of 2 steps implies reversibility of n steps: (x0 , x1 , ..., xn ) = (xn , ..., x1 , x0 ) Example 1: Any random walk on an interval of integers which has only transitions of +1, 1, 0 can be solved with reversible equilibrium. e.g. on states {0, 1, 2} P= Look for a reversible equilibrium: Start with generic 0
1 = 1 1 = 1 = 0 3 0 2 3 2 2 1 3 = 2 1 = 2 = 1 2 = 0 3

0 2 3 0 1 0

1 2 1 3

1 2

0=0

Lecture 17: Long run behaviour of Markov chains

17-3

Normalize: 0 + 1 + 2 = 1
3 + 0 = 1 0 + 0 2 3 0 (1 + 2 + 1) = 1

0 = 1 = 2 =

2 7 3 7 2 7

Example 2: Random walk on states in a circle. Take a circle with 5 points, with the probability of moving clockwise, p, and the probability of moving counter clockwise is q Easy to check that if is uniform then P = Reversible only if p = q = 1/2. Example 3: Random walk on {0, 1, 2, ...} reection at 0: P (0, 1) = 1 P (i, i + 1) = p for i 1 P (i, i 1) = q for i 1 Cases: Transient if p > q Null-recurrent if p = q Positive Recurrent (similar behavior to regular) if p < q i =
1 Ei (i ) j

For the case: p < q , nd there is a unique solution of P = , Find it as a reversible equilibrium: 0 1 = 1 q = 1 = 1 p = 2 q = 2 = n = 0 1 q
p q n1 2 0 q 0 p q q

j = 1.

1=
n=0

n = 0 +

0 q

0 q

p q

0 q

p q

+ ... = 0 (1 + 1 q

1 ) 1p/q