UNIVERSITI TUNKU ABDUL RAHMAN (UTAR)

UDPS 2133Mathematical Programming

Tutorial 3: Markov Chain

1. Consider a stock model. At the end of a given day, the price is recorded. Whether the stock goes
up tomorrow depends upon whether it increased today and yesterday. If the stock increased today
and yesterday, it will increase tomorrow with probability α1. if the stock increased today and
decreased yesterday, it will increase tomorrow with probability α2. If the stock decreased today
and increased yesterday, it will increase tomorrow with probability α3. Finally, if the stock
decreased today and yesterday, it will increase tomorrow with probability α4.
Construct the (one-step) transition matrix of the Markov chain.

2. A particle moves on a circle through points that have been marked 0, 1, 2, 3, 4 (in a clockwise
order). The particle starts at point 0. At each step it has probability 0.5 of moving one point
clockwise (0 follows 4) and 0.5 of moving one point counterclockwise. Let Xn (n ≥ 0) denote its
location on the circle after step n. {Xn} is a Markov chain.

a) Construct the (one-step) transition matrix.

b) Determine the n-step transition matrix P(n) for n = 2 and 4.

3. Given the following (one-step) transition matrices of a Markov chain, determine the classes of the
Markov chain and whether they are recurrent:

State 0 1 2 State 0 1 2 3
0  0 0 1 0  0 1 3 1 3 1 3
a) P =    
1  0 .5 0 . 5 0 b) P = 1 1 3 0 1 3 1 3
2  0 1 0  2 1 3 1 3 0 1 3
  
3 1 3 1 3 1 3 0 


State 0 1 2 3 State 0 1 2 3
0 0 0 1 3 2 3 0  1 0 0 0 
   
c) P = 1 1 0 0 0  d) P = 1  0 0 .5 0 .5 0 
2 0 1 0 0  2  0 0 .5 0 .5 0 
   
3 0 1 0 0  3  0 .5 0 0 0 . 5 
  
4. Determine the period of each of the states in the Markov chain that has the following (one-step)
transition matrix:

State 0 1 2 3 4 5
0 0 0 0 23 0 1 3
 
1 0 0 1 0 0 0 
1 0 0 
P= 2 
0 0 0

3 0 1 4 0 0 34 0 
4 0 0 1 0 0 0 

5 0 1 2 0 0 1 2 0 


5. Consider the Markov chain that has the following (one-step) transition matrix:

State 0 1 2 3 4
0  0 45 0 15 0 
 
1 1 4 0 1 2 1 4 0 
P=
2  0 1 2 0 1 10 2 5 
 
3  0 0 0 1 0 
4 1 3 0 1 3 1 3 0 


a) Determine the classes of this Markov chain and, for each class, determine whether it is
recurrent or transient.

b) For each of the classes identified in part (a), determine the period of the states in that class.

6. Determine whether the following matrices are transition matrix. Explain why if it is not a
transition matrix. If it is a transition matrix, find the absorbing state, if any.
 1 0 0 0  1/ 2 1/ 4 1/ 4 
   
i. 1 / 2 1 / 3 0 1 / 6  ii.  0 2 / 3 1/ 3 
 0 0 1 0   3 / 4 0 1/ 4 
  
 0 2 − 1
  1 / 2 2 / 3 
iii. 1 / 2 1 / 2 0  iv.  
1 / 3 2 / 3 0   1 / 2 1 / 3 
 
1 / 5 1 / 5 3 / 5 
 
iv.  0 2 / 7 5 / 7
 0 0 1 

7. Customers in a certain city are continually switching the brand of soap they buy. If a customer is
now using brand A, the probability he will use brand A next week is 0.5, that he switches to brand
B is 0.2 and that he switches to brand C is 0.3. If he now uses brand B, the probability he uses B
next week is 0.6 and the probability that he switches to C is 0.4. If he now uses brand C, the
probability he uses C next week is 0.4, that he switches to A is 0.2 and to B is 0.4. Assume the
process is a Markov Chain.

(i) Find the probability a customer now using brand A will be using brand B in two weeks.

(ii) If the percentage of customers now using brand A is 30%, the percentage using brand B is
20% and the percentage using brand C is 50%, find the percentage of customers using brand
C in three weeks.

(iii) Find the probability a customer now using brand A will be using brand B in 6 weeks.
Answer:
1. Hint:.
Let 1 = increased today and yesterday
2 = increase today and decreased yesterday
3 = decreased today and increased yesterday
4 = decreased today and yesterday

2.

3. a) P has one recurrent communicating class

b) P has one recurrent communicating class
c) P has one recurrent communicating class
d) P has three communicating classes: absorbing, so recurrent; recurrent and transient.

4. P has one communicating class, so each state has the same period 4

5. a) P has two classes: transient and recurrent.

b) The period of {0, 1, 2, 4} is 2 and the period of {3} is 1.

6.

7. (i) 0.34
(ii) 38%
(iii) 0.4563