Department of Economics

December 2005
Arne Strøm

Solution of a problem in Taylor & Karlin:

An Introduction to Stochastic Modeling, 3rd ed
Exercise III.4.5, p. 128
The problem describes a Markov chain with 6 states, and the following transition
probability matrix.

H T HH HT TH TT
1 1
0 0 0 0
 
H 2 2
1 1 
T  0 0 0 0 2 2 
 
HH  0 0 1 0 0 0 
P = 
1

1 
HT 
 0 0 0 0 2 2 
 1 1 
TH  0 0 2 2 0 0 
TT 0 0 0 0 0 1

The states H and T can only occur at the very beginning. They represent the
possible outcomes of the first toss, Heads or Tails. The other states represent the
outcomes of the two most recent tosses, e.g., HT indicates heads followed by tails.
HH and TT are absorbing states, signaling the end of the game.
In order to find the probability uH,TT that a game starting with heads will
end with two tails, we do a first step analysis. We find

uH,TT = 21 uHH,TT + 12 uHT,TT = 0 + 21 uHT,TT

uHT,TT = 12 uTH,TT + 21 uTT,TT = 12 uTH,TT + 1
2
uTH,TT = 21 uHH,TT + 12 uHT,TT = 0 + 21 uHT,TT

These equations yield uHT,TT = 32 , uTH,TT = 31 , and uH,TT = 13 .

1
solutions 13.12.2005 1332