Markov Chain

• By

• Dr.B.V.Dhandra
• Professor
• Christ (Deemed to be) University
• Bengaluru-560029
Stochastic Process
•
Revision
•
Classification of Stochastic Process

•
Classification of Stochastic Process
•
Markov Chains
 Markov Property
•
Definition:
•
 Transition Matrix
• The matrix describing the Markov chain is called the transition matrix. It is the most
important tool for analyzing Markov chains.

• The transition matrix is usually given the symbol P = (pij) .

In the transition matrix P:
•
Dynamics of Markov Chain:
•
Dynamics of Markov Chain:

•
 Theorem: If P(i0) is the initial probability that the system starts from state i0 and
P(i0, i1), P(i1, i2), …..are one step transition probabilities, then joint distribution of
X0, X1,…..,Xn can be expressed as product of initial probabilities and one step
transition probabilities.
•
contd

•
Example
•
 Imagine also that the following probabilities have been observed:
• Imagine also that the following probabilities have been observed:
• when the reader doesn’t visit TDS a day, he has 25% chance of still not visiting the next day,
50% chance to only visit and 25% to visit and read
• when the reader visits TDS without reading a day, he has 50% chance to visit again without
reading the next day and 50% to visit and read
• when the reader visits and read a day, he has 33% chance of not visiting the next day (hope that
this post won’t have this kind of effect!), 33% chance to only visit and 34% to visit and read
again.
• Then we have the following transition probability matrix
• P= 1—0.25 1, 1--0.5- 2,&
1---0.25 3
• 2----0.0- 1, 2—0.5 2,& 2—0.5- 3
• 3---0.33 1, 3---0.33 2& 3---0.34 3
 Solution:
•
 Two-Step Transition Probability:
•
 Three-Step Transition Probabilities:
•
 Chapman-Kolmogorov Equation:
•
 Proof:

•
Proof

•
 Computation of Higher step transition probabilities
•
 Solution:
•
 Contd…….
•
•
 Example:
• Purpose-flea zooms around the vertices of the transition diagram below. Let Xn be Purpose-flea’s state at time n. If
• It is modelld as a M.C with the following transition diagram:

• 1. Find the transition matrix, P

• 2. Find P(X2= 3 | X0 = 1).
• 3. Find P(X2= 2 | X0 = 1) and P(X2= 1 | X0 = 1).
• 4. Find P(X2= j | X0 = i), i= 1,2,3 and j= 1,2,3.
Solution:

•
Contd…

•
Solution:
•
 Special cases of Markov chains:
•
Examples:

Regular Not regular

Examples : Absorbing

State 2 is absorbing
Pii = 1 P22 = 1
Some applications:
• Physics
• Chemistry
• Testing: Markov chain statistical test (MCST), producing more efficient test samples as
replacement for exhaustive testing
• Speech Recognition
• Information sciences
• Queueing theory: Markov chains are the basis for the analytical treatment of queues.
Example of this is optimizing telecommunications performance.
• Internet applications: The PageRank of a webpage as used by google is defined by a Markov
chain , states are pages, and the transitions, which are all equally probable, are the links
between pages.
• Genetics
• Markov text generators: generate superficially real-looking text given a sample document,,
example: In bioinformatics, they can be used to simulate DNA sequences
Classification of states (1)
•
State Classification Example 1

• Consider the Markov chain

• Which states are accessible from state 0? Ans: {0,1}

• Which states are accessible from state 3? Ans: {2,3,4}
• Is state 0 accessible from state 4? Ans : {No state}
Draw its state transition diagram

• State transition diagram of the tpm:

• From this, we can observe that there are two classes {0,1} and {2, 3, 4}
State Classification Example 2
• Now consider a Markov chain with the following state transition
diagram

• Is state 2 accessible from state 0? Yes

• Is state 0 accessible from state 2? No
• Is state 1 accessible from state 0? Yes
• Is state 0 accessible from state 1? No
 State Classification -Communicability
• Definition: States i and j communicate to each other, if state j is accessible
from state i, and state i is accessible from state j (denote j ↔ i).
• Communicability is
• – Reflexive: Any state communicates with itself
• – Symmetric: If state i communicates with state j, then state j
communicates with state i
• – Transitive: If state i communicates with state j, and state j
communicates with state k, then state i communicates with state k
 State Classification -Communicability

• For the examples, which states communicate with each other?

State Classification -Communicability
State Classes
• Two states are said to be in the same class if the two states communicate
with each other, that is i ↔ j, then i and j are in same class.
• Thus, all states in a Markov chain can be partitioned into disjoint classes
• – If states i and j are in the same class, then i ↔ j.
• – If a state i is in one class and state j is in another class, then i and j
do not communicate.
• How many classes exist in the examples?
• • Which states belong to each class?
 State Classes

•
Gamblerʼs Ruin Example
• • Consider the gambling game with probability p=0.4 of winning on
any turn.
• State transition diagram and one-step transition probability matrix:

• How many classes are there?

• Ans: Three: {0} {1,2} {3}
Irreducibility

• A Markov chain is irreducible if all states belong to one class (all states
communicate with each other).
• If there exists some n for which pij (n) >0 for all i and j, then all states
communicate and the Markov chain is irreducible.
• If a Markov chain is not irreducible, it is called reducible.
• If a Markov chain has more than one class, it is reducible.
• Are the examples reducible or irreducible?
• Ex 1: Reducible {0,1} {2,3,4}
• Ex 2: Reducible {0} {1} {2}
• Gambler’s Ruin Ex: Reducible {0} {1,2} {3}
 Examples of Irreducible Chains

Weather example
Periodicity of the Gamblerʼs Ruin
Fig-1

• Observe: if you start from State 1 at time 0, you can only come back to it in
times 2, 4, 6, 8, …
• In othernwords, 2 is the greatest common divisor of all integers n > 0, for
which p ii > 0
• We say, the period of State 1 is 2. The period of State 2 is also 2. And
observe, they are in the same class.
• State 0 has a period of 1, called aperiodic
Periodicity
• The period of a state i is the greatest common divisor (gcd) of all integers
n > 0, for which pii (n) > 0
• Periodicity is a class property – If states i and j are in the same class, then
their periods are the same
• State i is called aperiodic if there are two consecutive numbers n and
(n+1) such that the process can be in state i at these times, i.e., the
period is 1.
• In other words, gcd{1,2,3,……..)} = 1
• gcd{3, 6, 9, 12, ….} = 3
Recurrent States:
Stationary distribution
• Definition: A (discrete-time) stochastic process {Xn : n ≥ 0} is
stationary if for any time points 1, . . . , n and any m ≥ 0, the joint
distribution of (X1 , . . . , Xn ) is the same as the joint distribution of
(X1+m, . . . , Xn+m).
• So “stationary” refers to “stationary in time”. In particular, for a
stationary process, the distribution of Xn is the same for all n.
• So why do we care if our Markov chain is stationary? Well, if it were
stationary and we knew what the distribution of each Xn was then we
would know a lot because we would know the long run proportion of
time that the Markov chain was in any state.
For example
• For example, suppose that the process was stationary and we knew that
P(Xn = 2) = 1/10 for every n. Then over 1000 time periods we should
expect that roughly 100 of those time periods was spent in state 2, and
over N time periods roughly N/10 = N*0.10 of those time periods was
spent in state 2. As N went to infinity, the proportion of time spent in
state 2 will converge to 1/10 (this can be proved rigorously by some
form of the Strong Law of Large Numbers). One of the attractive
features of Markov chains is that we can often make them stationary
and there is a nice and neat characterization of the distribution of Xn
when it is stationary. We discuss this next in stationary distributions of
a M.C
Stationary Distributions
• So how do we make a Markov chain stationary?
• If it can be made stationary (and not all of them can; for example, the simple
random walk cannot be made stationary and, more generally, a Markov chain
where all states were transient or null recurrent cannot be made stationary), then
making it stationary is simply a matter of choosing the right initial distribution
for X0. If the Markov chain is stationary, then we call the common distribution
of all the Xn the stationary distribution of the Markov chain.
 Proposition
•
Proof
•
Discussion:
• we have shown that by setting the distribution of X0 to be π, the
distribution of Xn is also π for all n ≥ 0, and this is enough to say that
πj can be interpreted as the long run proportion of time the Markov
chain spends in state j (if such a π exists).
• We also haven’t answered any questions about the existence
• or uniqueness of a stationary distribution.
Example:

•
Ex: . In Melbourne, during the first 3 months of 1983
• Consider the following data on weather observed:

• To calculate the probability that it will be dry two days after a wet day:
contd

•
contd

•
Example:

• A student has two ways of getting to college say A and B.

• Each day he picks one, and his choice is influenced only by his
previous days choice:
• if A, then P(A) next day = 1/ 2
• if B, then P(A) next day = 3/ 4
• When he first arrived at college (day 0), he had no preferences.
• 1. The transition matrix is given by:
Contd..

• 2. Probabilities of choosing A and B on day 1:

• 3. Probabilities of choosing A and B on day 2:

Contd….

•
 Remark
•
Two interpretations of π:

• 1. The probability that the process will be in state j after running for a
long time.
• 2. The proportion of time it spends in state j after running for a long
time.
Finding the limiting distribution
• We have

and

So, letting n → ∞,

πj ≥ 0 ∀ j ∈ S

•
•
Contd…

•
 Example:
•
Example:
• Three out of every four trucks on the road are followed by a car, while
only one out of every five cars is followed by a truck. What fraction of
vehicles on the road are trucks?
• Solution: Imagine sitting on the side of the road watching vehicles go
by. If a truck goes by the next vehicle will be a car with probability 3/4
and will be a truck with probability 1/4. If a car goes by the next
vehicle will be a car with probability 4/5 and will be a truck with
probability 1/5. We may set this up as a Markov chain with two states
0=truck and 1=car, and transition probability matrix
Contd..

•
Theorem:
• For an irreducible Markov chain, a stationary distribution π exists if and
only if all states are positive recurrent. In this case, the stationary
distribution is unique and πi = 1/µi , where µi is the mean recurrence time
to state i.
• In the previous example:
• If I see a truck pass by then the average number of vehicles that pass by
before I see another truck corresponds to the mean recurrence time to state
0, given that I am currently in state 0. By our theorem, the mean recurrence
time to state 0 is µ0 = 1/π0 = 19/4, which is roughly 5 vehicles
Example:
•
Contd….

•
Classification of States:

•
Contd….

•
•
•
summary

•
Example:
• Consider the Markov chain shown in Fig-1 below. Assume that 0<p<1/2. Does this MC
has limiting distribution

• Fig-1.

0 1 2 3……
0 1-p p 0 0……
1 1-p 0 p 0…..
2 0 1-p 0 p …..
………………………………………..
………………………………………..

The chain is irreducible, aperiodic and recurrent, since all states are communicating
forming a single closed class and P00 >0. Therefore, unique stationary distribution exists.