Introduction to Markov Analysis

Important Concepts, Properties and Applications

Armin Paul D. Allado
Quantitative Methods and Information Technology
Ateneo de Manila University

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022
1 | Motivation (Finance)

Useful in answering:
1. What is the probability that an AAA-rated bond will
become AA in the next period?
2. In the long-term, what % of loans will end up being bad
loans?

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 2
1 | Motivation (Epidemiology)

Useful in answering:
1. What is the probability that
someone susceptible from a virus
will be infected?
2. Over the next week, what is the
expected % of people in a city that
will be infected?

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 3
1 | Motivation (Marketing)

Useful in answering:
1. What is the probability that brand
awareness will lead to consumer purchase?
2. On average, what is the expected % of
target market who will actually purchase the
product after awareness phase?

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 4
1 | Markov Chain

Markov Chain
 Models a stochastic (random) process {Xt} values in a countable set {0, 1, 2, …}
 Elements in the countable set are called states (can be finite or infinite)
 E.g. Model the behavior of a dynamic inventory system with possible ending
inventory levels (states) contained in the set {0, 1, 2, 3, 4}

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 5
1 | Markov Chain

Markov Chain
 Markov chain “jumps” from state to state to model the behavior of a dynamic
system over time t = 0, 1, 2,…
 E.g. Ending inventory level can jump from 2 to 4 in the next time t.

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 6
1 | Markov Chain

Finite State Illustration: Machine Maintenance

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 7
1 | Markov Chain

Infinite State Illustration: Job Shop

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 8
2 | Transition Matrix

Transition Matrix
 A matrix that shows the probability of transitioning from state i to state j in next
time t + 1
 E.g. Transition matrix of ending inventory level
State: Possible Ending Inventory Level {0, 1, 2, 3, 4}
0 1 2 3 4
0 0.25 0.10 0.25 0.15 0.25
1 0.40 0.30 0.15 0.10 0.05
2 0.05 0.10 0.35 0.25 0.25
3 0.05 0.05 0.20 0.45 0.25
4 0.15 0.15 0.20 0.20 0.30

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 9
3 | Illustration # 1

Consider the following model for the value of a stock. At the end of a given day,
the price is recorded. If the stock has gone up, the probability that it will go up
tomorrow is 0.7. If the stock has gone down, the probability that it will go up
tomorrow is only 0.5.

Construct a transition matrix.

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 10
3 | Illustration # 1

Define the states as follows:

Let state 0 represent the stock’s going up
Let state 1 represent the stock’s going down

Transition matrix:

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 11
3 | Illustration # 2

Suppose now that the stock market model is changed so that the stock’s going up
tomorrow depends upon whether it increased today and yesterday.

If the stock has increased for the past two days, it will increase tomorrow with
probability 0.9. If the stock increased today but decreased yesterday, then it will
increase tomorrow with probability 0.6. If the stock decreased today but increased
yesterday, then it will increase tomorrow with probability 0.5. Finally, if the stock
decreased for the past two days, then it will increase tomorrow with probability
0.3.

Construct a transition matrix.

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 12
3 | Illustration # 2

Transform to a Markov Chain by saying that the state at any time n is determined
by the behavior of the stock during both that day and the previous day.

Define the states as follows:

State 0: The stock increased both today and yesterday.
State 1: The stock increased today and decreased yesterday.
State 2: The stock decreased today and increased yesterday.
State 3: The stock decreased both today and yesterday.

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 13
3 | Illustration # 2

State 0: The stock increased both today and yesterday.

State 1: The stock increased today and decreased yesterday.
State 2: The stock decreased today and increased yesterday.
State 3: The stock decreased both today and yesterday.

Transition matrix:

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 14
3 | Illustration # 3

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 15
3 | Illustration # 4

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 16
4 | Properties of Markov Chain

Markovian (Memoryless) Property

 Pij = P{Xn+1 = j | Xn = i, Xn-1 = in-1, … , X0 = i0}
= P{Xn+1 = j | Xn = i}
 Pij = P{Xn+1 = j | Xn = i}
 These are called transition probabilities or one-step probabilities
 Future states are dependent only on the present state, and are independent of
past history

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 17
4 | Properties of Markov Chain

One-step Transition Probabilities

 Pij = P{Xn+1 = j | Xn = i, Xn-1 = in-1, … , X0 = i0}
= P{Xn+1 = j | Xn = i}
 Pij = P{Xn+1 = j | Xn = i}

n-step Transition Probabilities

 Pijn = P{Xn+m = j | Xm = i}, n,m ≥ 0, i,j ≥ 0

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 18
4 | Properties of Markov Chain

Transition Probability Matrix

 P = [Pij]

Property of Transition Probabilities

  Pij ≥ 0, = 1

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 19
5 | n-Step Transition Probabilities

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 20
5 | n-Step Transition Probabilities

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 21
5 | n-Step Transition Probabilities

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 22
5 | n-Step Transition Probabilities

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 23
5 | n-Step Transition Probabilities

QUANT 164: OR Modelling Applications| Armin Paul D. Allado | 1st Semester SY 2021-2022 24