IE 325: Stochastic Models

Homework Assignment 4
Solutions
Fall 2020

Question 1. A professor continually gives exams to her students. She can give three possi-
ble types of exams, and her class is graded as either having done well or badly. Let pi denote
the probability that the class does well on type i exam, and suppose that p1 = 0.3, p2 = 0.6,
and p3 = 0.9. If the class does well on an exam, then the next exam is equally likely to be
any of the three types. If the class does badly, then the next exam is always type 1. What
proportion of the exams are of type i, i = 1, 2, 3?

Solution of Question 1. Let Xn denote the type of exam in period n. Then (Xn )n≥0 is a
Markov Chain on state space E = {1, 2, 3}. The one-step transition probability matrix is:

1 2 3
 
1 0.8 0.1 0.1
P = 2 0.6 0.2 0.2 
3 0.4 0.3 0.3

Note that the Markov chain is ergodic. The limiting probability distribution π = (π1 , π2 , π3 )
is the solution of:

π1 = 0.8π1 + 0.6π2 + 0.4π3

π2 = 0.1π1 + 0.2π2 + 0.3π3
π3 = 0.1π1 + 0.2π2 + 0.3π3
π 1 + π2 + π3 = 1

Solving this we obtain π = ( 75 , 17 , 17 )

Question 2. Two urns A and B contain a total number of k balls. In each step, urn B is
chosen with probability p (0 < p < 12 ), and urn A is chosen with probability 1 − p. Then a
ball is selected from the chosen urn and placed in the other urn. If urn A becomes empty
we transfer 0 balls from urn A to urn B with probability 1 − p, and 1 ball from urn B to A
with probability p.
(a) Model this process as a Markov chain.

(b) Determine the one-step transition probability matrix.

1
 (c) Does the Markov chain have a limiting distribution? If so, find the limiting distribution.

Solution of Question 2. (a) Let Xn be the number of balls in urn A at time n, the number
of balls in urn A is only dependent to the previous step, therefore it is Markov Chain process.
And the state transition diagram for this MC will be:

(b)
0 1 2 3 ... k − 1 k
4
0 1−p p 0 0 0
... 0 0
 
1 1 − p 0 p 0 0
... 0 0
1−p
 
2 0 0 p 0
... 0 0
P = 3 0 0 1−p 0 p
... 0 0

 . .. .. .. ..
.. .. .. 
 .. . . . .. . .
k 0 0 0 0 0 ... p − 1 p
(c) This chain is irreducible since all states communicate with each other. It is also aperiodic
since, for instance, p00 > 0. Hence, it is ergodic and a unique limiting distribution can be
found by solving π = πP . Writing the equations we have the following:
p
For state 0, we can write π0 = (1 − p)π0 + (1 − p)π1 , which results in π1 = 1−p π0 .
For state 1, we can write π1 = pπ0 + (1 − p)π2 = (1 − p)π1 + (1 − p)π2 , which results in
p
π2 = 1−p π1 .
Similarly, for any j ∈ 1, 2, . . . , k, we obtain
p
πj = απj−1 , where α = .
1−p

Note that since 0 < p < 1/2, we conclude that 0 < α < 1. We obtain

πj = αj π0 , for j = 1, 2, . . . , k.
k+1
Finally, we have 1 = kj=0 πj = kj=0 αj π0 = 1−α
P P
1−α
π0 ( as 0 < α < 1).
1−α
Thus, π0 = 1−αk+1 and π = [π0 , π1 , . . . , πk ] is the limiting distribution.

Question 3. A machine, at any given day, can be in one of four different conditions: perfect,
good, average and critical.

ˆ If the machine is in perfect condition, it stays in perfect condition the next day with
probability 0.7 and deteriorates into good condition with probability 0.3.

ˆ If the machine is in good condition, it stays in good condition the next day with
probability 0.7 and deteriorates into average condition with probability 0.2. It breaks
down with probability 0.1.

2
 ˆ If the machine is in average condition, it stays in average condition the next day with
probability 0.7 and deteriorates into critical condition with probability 0.1. It breaks
down with probability 0.2.

ˆ If the machine is in critical condition, it stays in critical condition the next day with
probability 0.6 and breaks down with probability 0.4.

Condition Cost
Perfect $10
Good $15
Average $20
Critical $50

If the machine is not in perfect condition then some defective items are produced. The
associated costs per day for each condition is given as follows: When the machine breaks
down, it is immediately replaced. Replacement of the machine costs $200.

(a) Model this process as a Markov chain.

(b) Show that the limiting probabilities exist. Find the limiting probabilities.

(c) Find the expected cost per day.

(d) Now consider a replacement policy where we replace the machine once it reaches critical
condition. Assume that in this case, if the machine breaks down, we pay a penalty
of $50 in addition to the replacement cost. Calculate the expected cost of this policy.
Compare it with your result in (c).

Solution of Question 3. (a) Let the Markov chain be defined as follows:



 0, the machine is in perfect condition.

1, the machine is in good condition.
(Xn )n≥0 =


 2, the machine is in average condition.

3, the machine is in critical condition.

The set of states is E = {0, 1, 2, 3}. With the information given, following state
transition diagram is obtained: Transition probability matrix:

0 1 2 3
 
0 0.7 0.3 0 0
1 0.1 0.7 0.2 0 
P =  
2 0.2 0 0.7 0.1 
3 0.4 0 0 0.6

(b) As observed from the state transition diagram, the Markov chain consists a single
communicating class of states. Thus it is irreducible.

3
 For each state i, once the system leaves the state i, it is certain that the system will
visit state i after a finite units of time. Hence we have positive recurrence.
For each state i, once the system leaves the state i, it is probable to return to that
state in one unit of time (pii > 0). This information is sufficient to conclude that the
Markov chain is aperiodic.
A Markov chain which is irreducible, positive recurrent and aperiodic is known to
have a unique limiting probability distribution. Limiting probability distribution π =
[π0 , π1 , π2 , π3 ] is found by solving the following system of equations:

πP = π
X
πi = 1
i∈E

which can be explicitly written as follows:

0.7π0 + 0.1π1 + 0.2π2 + 0.4π3 = π0
0.3π0 + 0.7π1 + 0π2 + 0π3 = π1
0π0 + 0.2π1 + 0.7π2 + 0π3 = π2
0π0 + 0π1 + 0.1π2 + 0.6π3 = π3
π 0 + π1 + π2 + π 3 = 1

Solving the system of equation gives us π = [0.3529, 0.3529, 0.2353, 0.0588].

(c) The system, in the long run, is at state i with probability πi . Every time the system
hits state 0 we incur a replacement cost in addition to maintenance cost. Otherwise
only maintenance cost is incurred. Let M be the r.v. indicating daily cost. Then:
E[M ] = 10π0 + 15π1 + 20π2 + 50π3 + 200(0.1π1 + 0.2π2 + 0.4π3 )
| {z } | {z }
expected maintenance cost per day expected replacement cost per day
= $37.65

4
(d) The Markov chain corresponding to this system is defined as follows:


 0, the machine is in perfect condition, replaced without failure

0f, the machine is in perfect condition, replaced after failure
(X̃)n≥0 =


 1, the machine is in good condition.

2, the machine is in average condition.

The set of states is Ẽ = {0, 0f, 1, 2}. State transition diagram:

Transition probability matrix:

0 0f 1 2
 
0 0.7 0 0.3 0
0f
 0 0.7 0.3 0 

P =
1  0 0.1 0.7 0.2 
2 0.1 0.2 0 0.7

This Markov chain is also irreducible, aperiodic and positiveP

recurrent, therefore it has
a limiting probability distribution. Solving πP = π and i∈Ẽ πi = 1 together, we
obtain π = [0.0833, 0.2917, 0.3750, 0.2500].
Let M̃ be the r.v. indicating the daily cost incurred in this system. Then:

E[M̃ ] = 10π0 + 10π0f + 15π1 + 20π2 + 250(0.1π1 + 0.2π2 )

| {z } | {z }
expected maintenance cost per day expected replacement and penalty cost per day
+ 200(0.1π )
| {z 2}
expected replacement cost per day (without failure)

= $41.25

5
 Since E[M̃ ] > E[M ], the previous policy incurs a lower average daily cost.
Question 4. Consider the Markov chains whose transition probability matrices are given
below. For each Markov chain, classify its states and determine if the Markov chain is
ergodic. What can you say about the limit behavior of each Markov chain?
(a)
0 1 2
 
0 0 0.5 0.5
Pa = 1 0.5 0 0.5 
2 0.5 0.5 0
Solution:

• C = {0, 1, 2} (recurrent)
– Pa is irreducible since all the states communicate, it has a single communicating
class.
– All of the states are recurrent and since Pa is a finite state Markov Chain, all
recurrent states are positive recurrent.
– Pa is aperiodic.
• Therefore Pa is ergodic.
• To find the limiting distribution, one needs to solve π = π · Pa and π · 1 = 1.
 
0 0.5 0.5
    
π1 π2 π3 = π1 π2 π3 · 0.5 0 0.5

0.5 0.5 0
1 1
π 1 = π 2 + π3
2 2
1 1
π 2 = π 1 + π2
2 2
1 1
π 3 = π 1 + π2
2 2
1 = π 1 + π2 + π3
1 1 1
π1 = , π2 = , π3 = .
3 3 3
Note that it is also possible to conclude this result without these computations by real-
izing that the matrix is doubly stochastic and this implies that the limiting distribution
is the uniform distribution among the states.

6
(b)
0 1 2 3
 
0 0 0 0 1
1 0
 0 0 1
Pb =  
2 0.5 0.5 0 0
3 0 0 1 0
Solution:

• C = {0, 1, 2, 3} (recurrent)

– Pb is irreducible since all the states communicate, it has a single communicating

class.
– All of the states are recurrent and since Pb is a finite state Markov Chain, all
recurrent states are positive recurrent.
– Pb is periodic with period 3.

• Therefore Pb is not ergodic.

• Limiting distribution does not exist.

(c)
0 1 2 3 4
 
0 0.5 0 0.5 0 0
1 0.25 0.5 0.25 0 0 
Pc = 2 
 0.5 0 0.5 0 0 

3 0 0 0 0.5 0.5 
4 0 0 0 0.5 0.5
Solution:

7
Pc is reducible since all the states do not communicate. Therefore Pc is not ergodic and
it is not possible to find the limiting distribution by standard methods as it may not
exist or may depend on initial state. However, we can search for the limiting behaviour
depending on the initial state.
Note that the Markov chain has 3 communicating classes. Since Pc is a finite state
Markov Chain, all recurrent states are positive recurrent:
• C1 = {0, 2} (positive recurrent, aperiodic)
• C2 = {1} (transient)
• C3 = {3, 4} (positive recurrent, aperiodic)

– Let πij = lim P{Xn = j|X0 = i}

n→∞
– Since state 1 is transient, then πi1 = 0 ∀i = 0, · · · , 4.
– Since C1 and C3 are recurrent, then π03 = π04 = π23 = π24 = π30 = π32 = π40 =
π42 = 0.
– By handling C1 and C3 as two separate ergodic Markov chains, compute the
limiting probabilities of each class. The results are:
π00 = π02 = π20 = π22 = 21 .
π33 = π34 = π43 = π44 = 12 .
– In this part, we find the probability of absorption into C1 or C3 from the transient
state 1.
Let T = min{n ≥ 0|Xn ∈ C1 or Xn ∈ C3 }.
Note that P{XT ∈ C1 |X0 = 1} + P{XT ∈ C3 |X0 = 1} = 1.
Let νi = P{XT ∈ C1 |X0 = i}, then
νC1 = 1, νC3 = 0, ν1 = 0.5ν1 + 0.5νC1
ν1 = 1.
Starting from state 1, the Markov chain will end up in C1 and C3 with probabilities
1 and 0 respectively.
Hence π10 = 1 · π00 = 21
π12 = 1 · π00 = 21
π13 = 0
π14 = 0
Hence we have,

0 1 2 3 4
1 1

0 2 0 2
0 0
1 1
1
 21 0 2
0 0
1
π = 2
2 0 2
0 0
1 1
3 0 0 0 2 2
1 1
4 0 0 0 2 2

8
(d)
0 1 2 3 4
 
0 0.25 0.75 0 0 0
1 0.5 0.5 0 0 0
Pd = 2 
 0 0 1 0 0
1 2
3  0 0 3 3
0
4 1 0 0 0 0
Solution:

• C1 = {0, 1} (positive recurrent, aperiodic)

• C2 = {2} (positive recurrent, aperiodic)
• C3 = {3} (transient)
• C4 = {4} (transient)

– Pd is reducible since all the states do not communicate, it has 4 communicating

classes.
– Since Pc is a finite state Markov Chain, all recurrent states are positive recurrent.

• Therefore Pd is not ergodic.

• Limiting distribution does not exist. However, we can find the limiting behaviour
depending on the initial state.

– Let πij = lim P{Xn = j|X0 = i}

n→∞
– Since states 3 and 4 is transient, then πi3 = πi4 = 0 ∀i = 0, · · · , 4.
– Since C1 and C2 are recurrent, then π02 = π12 = π20 = π21 = 0.
– Since C2 is absorbing, then π22 = 1.
– By handling C1 as a single ergodic Markov chain, compute its limiting probabili-
ties. The results are:
π00 = π = 10 = 52 .
π01 = π11 = 35 .
– In this part, we find the probability of absorption into C1 or C2 from the transient
states 3 and 4.
Let T = min{n ≥ 0|Xn ∈ C1 or Xn ∈ C2 }.

9
Note that P{XT ∈ C1 |X0 = 3} + P{XT ∈ C2 |X0 = 3} = 1.
P{XT ∈ C1 |X0 = 4} + P{XT ∈ C2 |X0 = 4} = 1.
Let νi = P{XT ∈ C2 |X0 = i}, then
νC1 = 0, νC2 = 1, ν4 = 0, ν3 = 23 ν3 + 31 νC2
ν3 = 1.
Starting from state 3, the Markov chain will end up in C1 and C2 with probabilities
0 and 1 respectively.
Starting from state 4, the Markov chain will end up in C1 and C2 with probabilities
1 and 0 respectively.
Hence π30 = π31 = 0
π40 = π00 = 25
π41 = π11 = 53
Hence we have,

0 1 2 3 4
2 3

0 5 5
0 0 0
2 3
1
5 5
0 0 0
π = 2
0 0 1 0 0
3 0 0 1 0 0
4 52 3
5
0 0 0

10