Markov

Processes
Sharmaine, Miranda
Rachel, Orias
Markov process,
named after the
Russian mathematician
Andrey Andreyevich
Markov.
(1856 – 1922) was a
Russian mathematician
best known for his work
on stochastic
processes.
Stochastic
Processes
INTRODUCTION TO STOCHASTIC PROCESS

◦ Any variable whose value changes over time in an

uncertain way
◦ Discrete time SP- value of the variable can change
only at certain fixed points in time
◦ Continuous time SP- changes can take place at any
time
◦ Continuous variable/state SP- underlying variable
can take any value within a certain range.
◦ Discrete variable process- only certain discrete
values are possible
◦ Continuous variable, continuous-time stochastic
process for stock prices
 INTRODUCTION TO STOCHASTIC PROCESS

◦ Types of stochastic process where only

current value of a variable is relevant for
predicting the future
◦ Past history of the variable and the way that
the present has emerged from the past are
irrelevant
◦ Stock prices are usually follow a Markov
Process
◦
 Markov Chains
vs.
Markov Processes
◦ Important classes of stochastic
processes are Markov chains and
Markov processes. A Markov chain
is a discrete-time process for which
the future behavior, given the past
and the present, only depends on
the present and not on the past. A
Markov process is the continuous-
time version of a Markov chain.
Many queueing models are in fact
Markov processes.
Markov Chains
◦ Markov chain model of a baby's
behavior, you might include
"playing," "eating", "sleeping," and
"crying" as states, which together
with other behaviors could form a
'state space': a list of all possible
states. In addition, on top of the
state space, a Markov chain tells
you the probabilitiy of hopping, or
"transitioning," from one state to any
other state-e.g., the chance that a
baby currently playing will fall asleep
in the next five minutes without
Markov Processes
Markov Processes
In a Markov process we also have a discrete
set. However, the transition behaviour is
different from that in a Markov chain.
In each state there are a number
of possible events that can cause
a transition. The event that causes
a transition from state i to j, where
j ≠ i, takes place after an
exponential amount of time, say
with parameter 𝑞𝑖𝑗 .
 As a result, in this model transitions take place
at random points in time. According to the properties
of exponential random variables (cf. section 1.2.3)
we have:
• In state i a transition takes place after an
exponential amount of time with parameter

• The system makes a transition to state j with

probability

Define
 The matrix Q with elements 𝑞𝑖𝑗 is called
the generator of the Markov process. Note that
the definition of the 𝑞𝑖𝑗 implies that the row sums
of Q are 0. Under the conditions that

• all states of the Markov process communicate

with each other,

• the Markov process does not drift away to

infinity,
 the probability 𝑝𝑖 (t) that the system is in state i at
time t converges to a limit 𝑝𝑖 as t tends to infinity. Note that,
different from the case of a discrete time Markov chain, we
do not have to worry about periodicity. The randomness of
the time the system spends in each state guarantees that
the probability 𝑝𝑖 (t) converges to the limit 𝑝𝑖 . The limiting
probabilities, or equilibrium probabilities, can again be
computed from the balance equations. The balance
equations now balance the flow out of a state and the flow
into that state. The flow is the mean number of transitions
per time unit. If the system is in state i, then events that
cause the system to make a transition to state j occur with
a frequency or rate 𝑞𝑖𝑗 . So the mean number of transitions
per time unit from i to j is equal to 𝑝𝑖 𝑞𝑖𝑗 . This leads to the
balance equations.
Or

In vector-matrix notation this becomes, with p

the row vector with elements pi,
Together with the normalization equation

The solution of the set of equations is

unique.
Example1
In analysing switching by Business Class customers between airlines the following
data has been obtained by British Airways (BA):

Next flight by

BA Competition
Last flight by BA 0.85 0.15
Competition 0.10 0.90

For example if the last flight by a Business Class customer was by BA the
probability that their next flight is by BA is 0.85. Business Class customers make 2
flights a year on average.

Currently BA have 30% of the Business Class market. What would you forecast
BA's share of the Business Class market to be after two years?
Solution

We have the initial system state s1 given by s1 = [0.30, 0.70] and the transition matrix P is
given by

P = | 0.85 0.15 |2 = | 0.7375 0.2625 |

| 0.10 0.90 | | 0.1750 0.8250 |

where the square term arises as Business Class customers make 2 flights a year on average.
Hence after one year has elapsed the state of the system s 2 = s1P = [0.34375, 0.65625]
After two years have elapsed the state of the system = s 3 = s2P = [0.368, 0.632]

and note here that the elements of s2 and s3 add to one (as required).

So after two years have elapsed BA's share of the Business Class market is 36.8%
Example2
 An operational researcher is analysing switching between
two different products. She knows that in period 1 the market shares
for the two products were 55% and 45% but that in period 2 the
corresponding market shares were 67% and 33% and in period 3
70% and 30%.

The researcher believes that an accurate representation of

the market share in any period can be obtained using Markov
processes. Assuming her belief is correct:

• Estimate the transition matrix.

• Calculate the market shares in period 4 using the estimated
transition matrix.
• If the actual market shares for period 4 were 71% and 29% would
you revise your estimate of the transition matrix or not? Give
reasons for your decision.
Solution

We have that s1, the state of the system in period 1, is given by s1=[0.55,
0.45] with s2=[0.67, 0.33] and s3=[0.70, 0.30]
Assuming the researcher is correct then s1 and s2 are linked by s2 = s1(P)
where P is the transition matrix.

We also have that s3 and s2 are linked by s3 = s2(P).

Now we have that P will be a 2 by 2 matrix, and that the elements of
each row of P add to one, so that we can write
P = |x1 1-x1|

|x2 1-x2|

where x1 and x2 are unknown.

Using s2 = s1(P) we have
[0.67, 0.33] = [0.55, 0.45] |x1 1- x1|

|x2 1- x2|

and using s3 = s2(P) we have

[0.70, 0.30] = [0.67, 0.33] |x1 1- x1|

|x2 1- x2|
Hence, expanding, we have
0.67 = 0.55x1 + 0.45x2 (1)

0.33 = 0.55(1 - x1) + 0.45(1 - x2) (2)

0.70 = 0.67x1 + 0.33x2 (3)

0.30 = 0.67(1 - x1) + 0.33(1 - x2) (4)

Equation (2), when rearranged, becomes equation (1) and similarly equation (4), when
rearranged, becomes equation (3).

Hence we have two simultaneous equations (equations (1) and (3)) in two unknowns.
From equation (1)

x1 = (0.67 - 0.45x2)/0.55
so substituting for x1 in equation (3) we get

0.70 = 0.67[(0.67 - 0.45x2)/0.55] + 0.33x2

i.e. (0.70)(0.55) = (0.67)(0.67) - (0.67)(0.45)x2 + (0.33)(0.55)x2

i.e. x2 = [(0.67)(0.67)-(0.70)(0.55)]/[(0.67)(0.45)-(0.33)(0.55)]

i.e. x2 = 0.5325 and

x1 = (0.67 - 0.45x2)/0.55 = 0.7825

Hence our estimate of the transition matrix P
= |x1 1-x1| |x2 1-x2| = |0.7825 0.2175| |0.5325 0.4675|

Note as a check we can verify that, with this estimated

transition matrix, we have

s2 = s1(P) and s3 = s2(P).

The market shares for period 4 are given by
s4 = s3(P)
= [0.70, 0.30] |0.7825 0.2175| |0.5325 0.4675|

i.e. s4 = [0.7075, 0.2925]

and note here that the elements of s4 add to one (as required).
Hence the estimated market shares for period 4 are 70.75% and 29.25%.

If the actual market shares are 71% and 29% then this compares well with
the shares estimated above and so there would seem no reason to revise the
estimate of the transition matrix.
http://people.brunel.ac.uk/~mastjjb/jeb/or/moremk.html
https://www.win.tue.nl/~iadan/que/h3.pdf