Markov Chain

Feri Afrinaldi, Ph.D

Department of Industrial Engineering
Andalas University
 Contents
• Markov Chain-Cost or Profit Function

2
Markov Chain-Cost or Profit Function
• We use Example 2, given that:
– Every week spent in town a resulted in a profit of
$1,000
– Every week spent in town b resulted in a profit of
$1,200
– Every week spent in town c resulted in a profit of
$1,250

3
Markov Chain-Cost or Profit Function
• If the original town is a, then the expected
profit of the nth week is:

E  Profit for week n | original town = a 

 Paan (1, 000)  Pabn (1, 200)  Pacn (1, 250)

4
Markov Chain-Cost or Profit Function
• Property 2
Let X = { Xn: n = 0, 1, …} be a Markov chain
with state space E, Markov matrix P, and profit
function f (a column vector), initial
probabilities a(0) (a row vector). The expected
profit of the nth step is given by:

E  f  X n  | X 0  i   a P f
(0) n

5
 Example 6
• Use example 2
What is the expected profit after 2 weeks if
initially he is in town a?

6
 Example 7
• Use example 2
Suppose that 50% chance that he is in town a,
a 30% chance is in town b, and 20% chance in
town c. What is the probability that the
salesman will be in town a next week?

7
 Example 8
• Use example 2
Using the initial probabilities and profit
function given above, what is the expected
profit in second week?

8
 Example 9
• A market analysis concerning consumer behavior in
auto purchases has been conducted. Body styles
traded in and purchased have been recorded by
particular dealer with the following results.
Number of customers Trade
275 Sedan for sedan
180 Sedan for station wagon
45 Sedan for convertible
80 Station wagon for sedan
120 Station wagon for station
wagon
150 Convertible for sedan
50 Convertible for convertible 9
 Example 9 (cont.)
i. Let assume that we have a customer whose behavior
is described by this model. The customer always buys
a new car in January of every year. It is now January
1997 and the customer enters the dealership with
sedan. What is the probability that this customer will
have 1997 with a convertible and then leave next
year (1998) with sedan?
ii. What is the probability that the customer who enters
the dealership now (1997) with a sedan will leave
with a sedan and also leave with sedan in the year
2000?

10
 Example 9 (cont.)
iii. Assume that a sedan yields a profit of
$1,200, a station wagon yields a profit of
$1,500, and a convertible yields a profit of
$2,500.
a. Determine profit we get from the customer
entering with sedan this year?
b. What is expected profit in year 1999 from
customer who enters with sedan in 1997?

11
 References
The materials on this presentation are based on:
• Resnick, S. (2002). Adventures in Stochastic
Processes. Boston: Birkhäuser.
• Taha, H.A. (2007). Operations Research: An
Introduction. Upper Saddle River, NJ: Pearson
Education.
• Feldman, R. M., Valdez-Flores, C. (2010).
Applied Probability and Stochastic Processes.
Berlin: Springer.

12
THANK YOU

13