Final practice, Part II

PSTAT 160A FALL 2014

D EC 2014

(1) Capa plays either one or two chess games every day, with the number of games that she plays on
successive days being a Markov chain with transition probabilities
P1,1 = .2,

P1,2 = .8,

P2,1 = .4,

P2,2 = .6.

Capa wins each game with probability p. Suppose she plays two games on Monday.
(a) What is the probability that she wins all the games she plays on Tuesday?
(b) What is the expected number of games that she plays on Wednesday?
(c) In the long run, on what proportion of days does Capa win all her games.
Solution.
(a) Let time 0 be Monday, Then our Markov chain (Xn )n0 with transition probability P is supposed
to start from state 2. Let A be the event she wins all games on Tuesday, that is at time 1. So we can
write by conditioning first on X1 and then further on X0 that
P [A] = P [She wins all games on Tuesday]
= P [A | X2 = 1] P [X1 = 1] + P [A | X2 = 2] P [X1 = 2]
3
2
= pP2,1 + p2 P2,2 = p + p2 .
5
5
(b) Wednesday is time 2. So we need the distribution of X2 . In general
X
n
P [Xn = k ] =
P [X0 = j ] Pj,k
.
j

In our case X0 = 2 and n = 2 so

P [X2 = 1] = (P 2 )2,1 ,

P [X2 = 1] = (P 2 )2,2 .

The square of P is


2 
0.36 0.64
0.2 0.8
=
.
P =
0.32 0.68
0.4 0.6
2

(c) The Markov chain of this problem is irreducible, and has finite state space, so we can use the
ergodic theorem. By the ergodic theorem, in the long run, the proportion of days when Capa plays
1 games is 1 , where is the invariant distribution of the Markov chain. Simlarly, in the long run,
the proportion of days she plays 2 games is 2 .
Consider first those days when she plays 1 game. Within these days the proportion of time she wins
all games (1 game) converges to p. This is the law of large numbers. (The number of experiments
(days she plays one game) goes to infinity, and each experiment (game) independently of the other
experiments, yields a success (win) with probability p. Then the relative frequency of successes goes
to p).
The other case is similar. That is, in the long run, the proportion of time she wins all games within
those days, when she plays two games, is p2 .
So finally we get
1
# {days before N when she wins all games} 1 p + 2 p2
N
It remains to compute the invariant distribution.
P =

1 = 0.21 + 0.42
1

2 = 21

as N .

1 + 2 = 1 yields 1 = 1/3 and 2 = 2/3. So the long run proportion of days she wins all games is
2
1
p + p2 .
3
3
(2) Consider the following Markov chain
1/4

2
3/4

1/2

3
1/2

3/4

1/4

(a) Give the transition matrix for this chain.

(b) What are the classes for this chain? Which are transient and which are recurrent?
(c) Set
pi = P [1 is reached before 5 | X0 = i] , for i = 1, . . . , 5.
Derive a system of linear equations for p1 , . . . , p5 obtained from conditioning on the first step.
(d) Compute p1 , . . . , p5 by solving the system of linear equations of (c).
(e) Suppose that P [X0 = 2] = P [X0 = 3] = 1/2. Compute the probability that the chain visits the
state 1 before it visits the state 5.
Solution.
(a) The transition matrix is

1 0 0 0 0
3/4 0 1/4 0 0

1
1
P =
0 /2 0 /2 0
0 0 1/4 0 3/4
0 0 0 0 1
(b) From the graph it can be seen that 1 communicates only with 1 and similarly 5 communicates
only with itself. The states 2,3,4 are communicating, so the classes are {1}, {2, 3, 4} and {5}. 1 and 5
are absorbing states, these are recurrent. {2, 3, 4} is transient, because, e.g., P [n 1, Xn = 2 | X0 =
2] 1/4 < 1.
(c)
p1 = 1
3
p2 = p 1 +
4
1
p3 = p 2 +
2
1
p4 = p 3 +
4
p5 = 0.

1
p3
4
1
p4
2
3
p5
4

(d) Using the equtions in the reversed order we get

1
p4 = p3
4
1
p3 = p2 +
2
3
p2 = p1 +
4

1
1
p4 = p2 +
2
2
1
3
p3 = p1 +
4
4
2

1
4
p3 = p 2
8
7
1
7
7
p2 = p 1 = .
7
8
8

That is p2 = 78 , p3 = 12 , p4 = 18 .
Since there were only 3 unknowns, it is not so important, but we could reduce the number of unknowns by observing that pi = 1 p5i by the symmetry of the chain. This gives immediately that
p3 = 1/2 and the only unknown is p2 can be computed easily from
1
3
p2 = p1 + p3 .
4
4
(e) It is
1
1
(p2 + p3 ) =
2
2

7 1
+
8 2


=

11
.
16

(3) Let (Xn )n0 be a Markov chain with state space D = {a, b, c} and transition matrix

0.4 0.6 0
P = 0.5 0 0.5
1
0
0
(a) Find limn P [Xn = j | X0 = a], for j D.
(b) Let T be the first time that the particle hits the state c, that is, T = min {n 1 : Xn = c}.
Compute E [T | X0 = a], E [T | X0 = b], E [T | X0 = c].
Solution.
(a) The Markov chain is irreducible, this can be seen by observing that the path 1 2 3 1
has positive probability. It is also aperiodic as there is a loop at 1, the move 1 1 has positive
probability ({n : (P n )1,1 > 0} = {1, 2, . . .}). For irreducible, aperiodic Markov chains the transition
probabilities have a limit, without taking the time average. We called this in class, the limit theorem
for Markov chains.
lim P [Xn = j | Xn = i] = j ,
n

where is the stationary distribution. To get we have to solve

X
P =
i = 1.
In a detailed form
a = 0.4a + 0.5b + c
b = 0.6a
c = 0.5b = 0.3a
1 = a + b + c = a (1 + 0.6 + 0.3) = 1.9a
That is

6
3
10
, b =
c = .
19
19
19
Note, that without aperiodicity the transition probabilities have no limit!
(b) T is the return time to c. So E [T | X0 = c] = 1/c = 19
. For the other two values of mi =
3
E [T | X0 = i] we write the equations obtained from conditioning on the first step:
pa =

mc = 1 + ma ,
ma = 1 + .4ma + .6mb .
This gives ma = mc 1 =

16
3

and mb = ma 1/0.6 =
3

16
3

5
3

11
.
3

(4) An individual possesses r umbrellas that he employs in going from his home to office, and vice
versa. If he is at home (the office) at the beginning (end) of a day and it is raining, then he will take
an umbrella with him to the office (home), provided there is one to be taken. If it is not raining, then
he never takes an umbrella.
Assume that, independent of the past, it rains at the beginning (end) of a day with probability p.
Then we can define a Markov chain with r + 1 states ({0, 1 . . . , r}, state i meaning that there are i
umbrellas at the place where the individual is), which will help us to determine the proportion of
time that our man gets wet.
(a) Compute the transition probabilities.
(b) Compute the invariant distribution.
Hint. You can use the weighted graph method, consider the weights w(i, r i) = q for i =
0, 1 . . . , r and w(i, r i + 1) = p for i = 1, . . . , r and all other weights are zero. Note that if you use
this method, then you have to check that the transition probabilities obtained from the weights
are the same as the one in (a).
(c) What fraction of time does our man get wet? (Note: He gets wet if it is raining, and all umbrellas
are at his other location.)
(d) When r = 3, what value of p maximizes the fraction of time he gets wet?
Solution.
(a) The possible transitions are i r i when our man does (or can) not take any umbrella with
him, or i r + 1 i when our man takes an umbrella, because it is raining and there was at least
one umbrella at the place he just leaves. So

1
if i = 0 and j = r,

p
if i > 0 and j = r + 1 i,
Pi,j =

1 p if j = r i,

0
otherwise.
(b) Here is the proposed weighted graph for r = 3.
1p
1p
0

p
p
The graph is similar for other values of r. Each node except 0 has two joining edges, one with weight
p connecting i with r + 1 i and another with weight (1 p) connecting i to r i. So the random
walk on this weigthed graph has the following transition matrix

1
if i = 0 and j = r,

p
if i > 0 and j = r + 1 i,
1 p if j = r i,

0
otherwise,
which is P . So the invariant distribution has the following form
! (
X
c(1 p)
i=0
i = c
w(i, j) =
c(p + (1 p)) = c i = 1, . . . , r.
j
4

The normalizing factor c is obtained from 1 =

0 =

1p
,
r+1p

i = c(r + (1 p)), that is, c = 1/(r + 1 p) and

1 = = r =

1
.
r+1p

(c) To answer this question we use the ergodic theorem for Markov chains. To apply this theorem
we first have to check that our Markov chain is irreducible. It is true since it has positive probability
(in our problem) that in the next r days it rains in the morning and it is not raining in the afternoon.
The effect is that each day one umbrella is taken to the office and at the and of this r days period all
umbrellas are taken from home to the office. During this all states are visited and in the morning of
the r + 1th day the chain returns the starting state r.
So on the long run, the proportion of time our that man has no umbrella when he walkes between
his places is 0 . Within these walks the proportion of time when it is raining is p on the long run. So
we get that
number of walks when our man gets wet
during his first N walks between his home and office
p(1 p)
p0 =
.
N
r+1p
. The function is positive
(d) We want to find the maximum point in [0, 1] of the function f (p) = p(1p)
r+1p
in the interior of the interval [0, 1] and zero at the endpoints. So if it has one local extremum point,
it is a maximum point and can be find by solving f 0 (p) = 0. It is easier to deal with logarithm of f ,
just like when you want to find ML estimate in statistics.
ln f (p) = ln p + ln(1 p) ln(r + 1 p)
1
1
1
(1 2p)(r + 1 p) + p(1 p)
(ln f )0 (p) =
+
=
p 1p r+1p
p(1 p)(r + 1 p)
That is
(ln f )0 (p) = 0

p2 2(r + 1)p + r + 1 = 0
=

Here r + 1 +

p1,2 =

2r + 1

p
4(r + 1)2 4(r + 1)
= (r + 1) r(r + 1)
2

p
r(r + 1) is bigger then 1, while
r+1

r(r + 1) =

(r + 1)2 r(r + 1)
r+1
p
p
=
(0, 1).
r + 1 + r(r + 1)
r + 1 + r(r + 1)

So the maximum point and value for r = 3 is

p(1 p)
(4 2 3)(2 3 3)

p = 4 2 3, p1 =
= (2 3)2 = 7 4 3 0.0718.
=
r+1p
2 3
(5) You process auto insurance claims. You begin your job with a single insurance claim in your inbox
and it takes you one hour to process a single claim. Further suppose that, every hour, there is either
a single automobile accident or there are no automobile accidents, indepdently from the past.
When an accident occurs, two new claims are added to your inbox (one for each driver).
Assume that you never stop working in this problem and consider a claim that you are currently
processing to still be in your inbox.
5

(a) Suppose that the probability of an accident in a given hour is 60%. What is the probability that
your inbox eventually becomes empty?
(b) Suppose that the probability of an accident in a given hour is 20%. What is the average amount
of time that you will need to work to empty your inbox?
Hint. Relate the problem to the random walk on integer lattice, and either recall and use the result
obtained for asymmetric random walk, or recall the method used in these computation (conditioning on the first step plus symmetries of the walk) and derive the answer.
Solution.
(a) Let Sn be the size of your inbox at the end of the nth hour. In this problem S0 = 1 and Sn+1 =
max(Sn 1, 0) + 2Yn+1 , where Yn+1 is the indicator of the event that in the n + 1th hour an accident
accurs. We are interested in the sequence (Sn ) until the inbox empties, that is, up to the time T =
min {n 0 : Sn = 0}. The sequence Sn0 = Smin(T,n) is random walk on the integer lattice stopped
when it reaches the level 0. So what we need to compute in part (a) is the probability that an
asymmetric random walk started at 1 reaches the level 0. We computed in class that this probability
is 1 if p, the probability of up move, is less than or equal to 1/2. When p > 1/2 then we found that
this probability is (1 p)/p. Since p = .6 > 1/2 we get .4/.6 = 2/3 as the probability that the inbox
will eventually become empty.
(b) When p 1/2 then T the time needed to reach level 0 is finite given that the stating state is
positive, when p < 1/2 then even the expectation is finite. Let mi = E [T | S0 = i]. Then mi = im1 in
particular m2 = 2m1 and conditioning on the first step gives the equation
m1 = (1 p) 1 + p(1 + m2 ) = 1 + p2m1 =
With p = .4 we get E [T | S0 = 1] = 1/(1 2 .2) = 5/4.

Each problem is worth 10 points.

6

1
.
1 2p