Markov Process

Definition :
P(2) = P2 = P * P Any ij th entry in P(n) gives the probability
P(3)= P3 = P*P*P that the chain changes it state from ith to
.
jth in n steps.
.

P(n)=Pn = P * P* n times. This is called Chapman’s theorem.

In the previous problem,

For eg 3rd row 2nd column entry denotes :

the probability that the ball moves from state 3(c) to state 2(B)
In two steps
Is
 EX:

Hence this markov chain is regular.

 Xn = m, means here n is step number
and m is state number
Here matrix is 4 X 4, hence we
Have 4 states 1, 2, 3, 4

Step n
States

1
2
3
4
1
2
States P = P(1) =
3
Step n-1 4

State
changes
from 1
to 2.
a)

Step number changes from 0 to 1. hence

Step length is 1. so we should look into matrix P or P(1).
In P, from Row of 1 and column of 2, we get 1. Therefore
So given that the chain is in state 1 in 0th step, the probability Answer is
that it is in state 2 in 1st step is
 State In P, from Row of 1 and column of 2, we get 1. Therefore
changes Answer is
from 1
b) to 2.

So given that the chain is in state 1 in 1st step, the

Step number changes from 1 to 2. hence probability that it is in state 2 in 2nd step is 1
Step length is 1. so we should look into matrix P or P(1).

Same answer we got in previous problem. Becoz in both step size is 1, even though step change was 0 to 1 in the first and
1 to 2 in the second. That is why the chain is called homogeneous. Step numbers don’t matter, only step size matters.

c) State In P, from Row of 2 and column of 1, we get 0.3. Therefore

changes Answer is 0.3.
from 2
So given that the chain is in state 2 in 1st step, the
to 1.
probability that it is in state 1 in 2nd step is 0.3
Step number changes from 1 to 2. hence
Step length is 1. so we should look into matrix P or P (1).
 State
changes
from 2 ii)
d)
to 2,
itself. LHS = 0.51, as shown by the matrix P(2)
Step number changes from 0 to 2. hence
Step length is 2. so we should look into matrix P(2).
By Chapmans theorem matrix P(2) = P2 RHS =

P2 =

This is obvious becoz :

If the chain has to move from state 2 to 2, in two steps,
there is one intermediate step say x.
In P2, from Row of 2 and column of 2, we get 0.51. Therefore 2 x 2, where x = 1, 2, 3, 4. These are
Answer is 0.51, that is probability of going from state 2 to state 2, Four mutually exclusive cases n
In two steps is 0.51. hence the probabilities have to get
Added up. This is exactly whats
Done in the RHS
A B
A B

Neglect becoz of
Markovian property.
3rd step depends only on 2nd.
Not on 1st or 0th step.
 = = (0.385, 0.336, 0.279)

Hence, = 0.279

A B
A B Becoz of Markovian property.
3rd step depends only on 2nd.
Not on 1st or 0th step and so on.

Similarly 2nd step

Depends on 1st only not on 0th step

using And
 Rain No Rain Tomorrow
(1) (2) Rain No Rain
(1) (2)

Rain (1)
Today

One step TPM, P = P(1) = No Rain

(2)
(ii)

= (0.3376, 0.6624 )

= 0.3376

(iii) P(X3 = 2) = 0.6624, complement of previous question

(iii) Third day means 2 steps. First day to second, second to third.
Hence we need to find

So we need P(2)= P2, by chapmans theorem

Is the probability that he goes by train on 3rd day