Markov Chains

Consider a sequence of random variable Xo, X1, X2, … .., and suppose that the set of possible
values of these random variables is (0,1,… .M). It will be helpful to interpret Xn as being the state
of some system at time n, and in accordance with this interpretation, we say that the system is in
state i at time n if Xn = i. The sequence of random variables is said to form a Markov Chain if
each time the system is in state i there is some fixed probability -- call it Pij -- that it will next be
in state j. That is, for all io … ,in-1, i, j,
P { X n + 1 = j| X n = i , X n− 1 = in− 1 , X n− 2 = in − 1 ,...., X 1 = i1 , X 0 = io } = Pij
The values Pij, 0 ≤i ≤M, 0 ≤j ≤ N, are called transition probabilities of the Markov chain and
they satisfy
M
Pij ≥ 0 ∑j =1
Pij = 1 i = 0,1,2,.... M

It is convient to arrange the transition probabilities Pij in a square array, as follows:

P00 P01 L P0 M
P10 P11 L P1M
M
PM0 PM1 L PMM
Such an array is called a matrix.
Knowledge of the transition probability matrix and the distribution of X0 enables us, in theory, to
compute all probabilities of interest . For instance, The joint probability mass function of
X0,… ,Xn is given by

P{Xn = in, Xn-1 = in-1,… ,X1=i1,X0=io}

= P{Xn = in|Xn-1=in-1,… ,X0=i0}P{Xn-1 = i n-1,… ,X0=i0}
= Pi n-1 ,i n P{X n − 1 = i n − 1 ,..., X 0 = i o}

And continual repetition of this argument yields that the above is equal to
= Pin− 1 ,in Pin− 2 ,in− 1 L Pi1 ,i2 Pio ,i1 P{ X 0 = i0 }

Example:
Suppose that whether or not it rains tomorrow depends on previous weather
conditions only through whether or not it is raining today. Suppose further tha if it is
raining today, then it will rain tomorrow with probability α, and if it is not rasining
today, then it will rain tomorrow with probability β.
If we say that the system is in state ) when it rains and state 1 when it does not, then
the above is a two-state Markov chain having transition probability matrix
α 1− α
β 1− β

That is, P00= α =1-P01, P10= β =1-P11

Example:
Consider a gambler who, at each play of the game, either wins 1 unit with probability
p or loses 1 unit with probability 1-p. If we suppose that the gambler will quit
playing when his fortune hits either 0 or M, then the gambler's sequence of fortunes is
a Markov chain having transition probabilities

Example:
The physicists P. and T. Ehrenfest considered a conceptual model for the movement of
molecules in which M molecules are distributed among 2 urns. At each time point one
of the molecules is chosen at random, and is removed from its urn and placed in the
other one. If we let Xn denote the number of molecules in the first urn immediately
after the nth exchange, then {X0, X1,… } is a Markov chain with transition
probabilities
M− i
Pi ,i + 1 = 0 ≤i ≤ M
M
i
Pi ,i − 1 = 0 ≤i ≤ M
M
Pij = 0 if j - i > 1

Thus, for a Markov chain, Pij represents the probability that a system in state I will
enter state j at the next transition. We can also define the two-stage transition
probability, Pij( 2 ) , that a system, presently in state I, will be in state j after two
additional transition. That is,
Pij( 2 ) = P { X m+ 2 = j| X m = i}

The Pij( 2 ) can be computed from the Pij as follows:

Pij( 2 ) = P { X 2 = j| X 0 = i}
M
= ∑ P{ X 2 = j , X 1 = k | X 0 = i}
k =0
M
= ∑ P { X 2 = j|, X 1 = kX 0 = i}P { X 1 = k | X 0 = i}
k =0
M
= ∑ Pkj Pik
k =0

In general, we define the n-stage transition probabilities, denoted as Pij( n ) , by

Pij( n ) = P { X n + m = j| X m = i}

This leads us to the Chapman-Kolmogorov equations, which shows how the Pij( n ) can
be computed
 Chapman-Kolmogorov Equations.

M
Pij( n ) = ∑ Pik( r ) Pkj( n − r ) For all 0 < r < n
k =0

Proof:
Pij( n ) = P { X n = j| X 0 = i}

= ∑ P{ X n = j , X r = k | X 0 = i}
k

= ∑ P { X n = j|, X r = kX 0 = i}P{ X r = k | X 0 = i}
k
M
= ∑ Pkj( n − r ) Pik( r )
k =0
Example
A Random Walk. An example of a Markov chain having a countable infinite state
space s the so-called random walk, which tracks a particle as it moves along a one
dimensional axis. Suppose that at each point in time the particle will either move one
step to the right or one step to the left with respective probabilities p and 1-p. That is,
suppose the particle's path follows a Markov chain with transition probabilities
Pi ,i + 1 = p = 1 − Pi ,i − 1 i = 0,±1,...
If the particle is at state i, then the probability it will be at state j after n transitions is the
probability that (n-i+j)/2 of these steps are to the right and n-[(n-i+j/2]=(n+i-j)/2 are to the
left. As each step will be to the right, independently of the other steps, with probability, it
follows that the above is just the binomial probability.
F
n I
GH
Pijn = n − i + j p ( n− i + JK j )/ 2
(1 − p) ( n+ i − j )/ 2

2
FGnIJ is taken to equal 0 when x is not a nonnegative integer less than or equal to n.
where
HxK
the above can be written as

=G
F 2n IJp (1 − p) ( n+ k ) n− − k
k = 0, ±1,... ±n
Hn + k K
2n
P i ,i + 2 k

=G
F 2n + 1 IJp (1 − p) k = 0,±1, ..... , ± n,-(n +1)
n+ k + 1 n− k

Hn + k + 1K
n +1
P i ,i + 2 k -1

Although the Pij( n ) denote conditional probabilities, we can, by conditioning on the initial state, use
them to derive expressions for unconditional probabilities. For instance,

P { X n = j} = ∑ P{ X n = j| X o = i} P{ X 0 = i}
i

= ∑ Pij( n ) P { X 0 = i}
i
For a large number of Markov chains it turns out that Pij( n ) converges as n→ ∞ , to a value j that
depends only on j. That is, for large values of n, the probability of being on state j after n transitions is
approximately equal to Π no matter what the initial state was. It can be shown that a sufficient
condition for a Markov chain to possess this property is that for some n>0
Pij( n ) >0 for all I,j,= 0,1,… .,M
Markov chains that satisfy this equation are said to be ergodic. Since the Chapman-Kolmogorov
equations yield
M
P ij
( n + 1)
= ∑ Pik( n ) Pkj
k =0
it follows, by letting n→ ∞ , that for egodic chains
M
Π j = ∑ Π j Pkj
k =0
M
Furthermore, simce 1 = ∑j =0
Pij( n ) , we also obtain, by letting n→ ∞ ,,
M

∑Π
j =0
j =1

In fact, it can be shown that the Π j, 0 ≤ j≤ M, are the unique nonnegative solutions of the above last
two equations. All this is summarized in the following theorem, which is stated without proof.

Theorem
For an ergodic Markov chain

Π j = lim Pij( n )
n→ ∞
Exists, and the Π j , 0 ≤ j ≤ M are the unique solutions of

M
Π j = ∑ Π k Pkj
k =0
M

∑Π
j =0
j =1

Example:
Consider the example from above, in which we assume that if it rains today,
then it will rain tomorrow with probability α, and if it does not rain today, then
it will rain tomorrow with probability β. From the above theorem it follow
that the limiting probabilities of rain and of no rain, Π o and Π 1 are given by
 Π o = α Π 0 + βΠ 1
Π 1 = (1 − α ) Π o + (1 − β ) Π 1
Π 0 + Π1 = 1
which yields
β 1− α
Π0 = Π1 =
1+ β − α 1+ β − α
For instance, if α=.6,β=.3, then the limiting probability of rain on the nth day is
Π 0 = 3/7.
The quantity Π j is also equal to the long run proportion of time that the Markov chain is
in state j, j=0,… .,M. To intuitively see why this might be so, let Pj denote the long run
proportion of time the chain is in state j. (It can be proven, using the string law of large
numbers, that for an ergodic chain such long run proportions exist and are constant).
Now, since the proportion of time the chain is in state k is Pk and since, when in state k,
the chain goes to state j with probability Pkj , it follows that the proportion of time the
Markov chain is entering state j from state k is equal to Pk Pkj . Summing over all k
shows that Pj, the proportion of time the Markov chain is entering state j, satisfies
Pj = ∑ kP k Pkj
Since clearly it is also true that
∑j
Pj = 1

it thus follows, since by our theorem the Π j j=0,… ,M are the unique solution of the
above, that pj = Π j j=0,… ,M .

Example
Suppose, in the example above that we are interested in the proportion of time
there are j molecules in urn 1, j-0,… ,M. By the Chaoman-Kolmolorov theorem,
these quantities will be the unique solution of
1
Π 0 = Π1 ×
M
M − j+ 1 j+ 1
Π j = Π j− 1 × + Π j+ 1 × j = 1,..., M .
M M
1
Π M = Π M− 1 ×
M
M

∑Π
j =0
j =1

However, as it it easily checked that

Πj =
FG M IJ( )
HjK
1 M
2 , j = 0,..., M.
 Satisfy the above equations it follows that these are the long run proportions of
time that the Markov chain is in each of the states.

Review of Markov Chains

This topic is important enough for us to spend a little more time reviewing and, in some
cases, repeating some of the preceding development..
Let us consider a random or stochastic process, :Xn, n=0,1,2,..}. The random variable
takes on a finite number of possible values. If Xn = i, the process is said to be in state i at
time n.
Suppose we assume that:

When the process is in state i ⇒ there is a fixed probabilty, Pij, that it will
next be in state j.
i.e.
Pij = P{ X n+ 1 = j| X n = i , X n − 1 = in − 1 ,..., X 0 = to }
for all states i0 , i1 ,....in − 1 .i , j and alln > 0
This process is called a Markovian process, or chain. We can interpret this as follows:
" A Markov chain can be described by the conditional
distribution of any future state, Xn+1, given the past states X0,
X1, … ,Xn-1, and the present state Xn. This conditional
probability is independent of the past states and depends only on
the present state".
Pij ≥ 0 i, j ≥ 0
∞

∑j =1
Pij = 1 i = 1,2,.....

Let us look at this from anpother angle. Assume a sequence of experiments

with the following properties:
1 Outcome of each experiment is one of a finite number of possible outcomes a1,
a2, … an.
2 Probability of outcome aj of any given experiment is not necessarily
independent of previous outcomes of experiments, but depends at most on
outcome of immediately preceding experiment.
3 Assume we can assign number, Pij, which represents probability of outcome aj,
given ai occurred. Note: outcome a1, a2, … ,ai, ar ⇒ states, and pij ⇒ transition
probabilities.
4 If the process is assumed to begin in some particular state, we have enough
information to determine the tree measure (flow of events) of the process and
calculate the probability of statements relating to experiments.
This is called a Markovian Chain Process, or simply a Markov process. The
transition probability can be exhibited in two ways:

1) Square array (transition matrix)

 Fp p12 p12 I
P = Gp JJ
11

GHp 21 p22
p32
p23
p33 K
31
or
2) Transition diagram. i.e. given 3 possible state:

½
1
a2
a1 ½
a3
1/3
2/3
The matrix associated with diagram is;

a1 a2 a3
F a1 0 1 0 I
P = a G0 JJ
G
1 1
2

a H K
2 2
1 2
30 3 3

Note: The sum of the ith row is 1, since the elements of each
row represents the probabilities for all possibilities
when in state ai
Example
Suppose that the chance of rain tomorrow depends only if it is raining today (not
on past weather. And ,if it is raining today, then it will rain tomorrow with
probability α. If it does not rain today, it will rain tomorrow with probability β.
We say that the process is in state 0 when it rains, and in state 1 when it does not
rain. This is a two-state Markov chain whose transition probability matrix is

P=
LM OP
α 1 − α ← Probabilty vector
N Q
β 1 − β ← Probabilty vector

Example
Gary is either cheerful, C, so-so, S or glum, G.
1) If he is cheerful today then he will be C, S, or
G tomorrow with probability .5, .4, or.1,
respectively.
2) If he is so-so today then he will be C, S, or G
tomorrow with probability .3, .4. or .3 ,
respectively.
3) If he is glum today then he will be C, S, or G
tomorrow with probability .2, .3. or .5 ,
respectively.
 If Xn → Gary's mood on the nth day, {Xn, n ≤0}
is a 3stage Markov chain. If we designate C=0,
S=1, and G=2 the probability matrix is:

Fp p01 p02 I F.5 .4 .1 I

P = Gp JJ= P = GG.3 .3J
00

GHp 10 p11
p21
p12
p22 K H.2
.4
.3
J
.5K
20

A kind of problem that we are most interested in is, suppose the process starts in
state i, what is the probability that after n steps it will be in state j? This is
denoted by the n transitional probability matrix Pij( n ) . We are actually interested
in this probability for all possible starting positions I, and all possible terminal
positions j.

Fp (n)
p12( n) p13( n ) I
= Gp JJ
11

GG p
(n) (n) ( n) (n)
i.e. P p p
ij

H
21
(n)
31 p
22
( n)
32 p
23
(n)
33
JK
As an example, let us find, for a Markov chain with transition probability
indicated bekow, the probability of being at various possible stages after three
steps. Assume the process starts at state a1;

Given

a1 a2 a3
F
a1 0 1 0 I
P = a G0 JJ
G
1 1
2

a H K
2 2
1 2
3 3 0 3

Construct a tree
 ½ a2

½
a2 ½

a3
1
a1 a2
½ 2/3 a3

a3
1/3

a1
So, P13( 3) = probability of going to state 3 from state 1 after 3 steps
= sum of the probabilities assigned to all paths thru tree which end at state a3
i.e
1 1 1 2 7
P13( 3) = 1 × × + 1 × × =
2 2 2 3 12
Similarly,
1 1 1
P12( 3) = 1 × × =
2 2 4
1 1 1
P11( 3) = 1 × × =
2 3 6
1 1 7 FG IJ
∴ P1(i 3) =
6 4 12 H K
We also can represent this as follows,
½ a2
a2
½
½ a3
a2 2/3 a3
½ ½
a3 1/3 a1
a2 2/3 a
½ 2/3
a3 3

1/3 a1
a3
1
1/3 a2
a1
P21( 3) = 21 × 23 × 13 + 1
2
× 21 × 13 = 7
36

P22( 3) = 21 × 13 ×1 + 1
2
× 21 × 21 = 7
24

P23( 3) = 21 × 12 × 21 + 1
2 × 21 × 23 + 1
2 × 23 × 23 = 37
72

7 7 37
P2(i3) = ( , , )
36 24 72
 a1 2/3 a2
1/3 1
a2 1/3 a3
a3
a1 1 a2
2/3 1/3

a3 1/3
2/3 a1
a3
2/3 a3

P31( 3) = 23 × 23 × 13 = 4
27

P32( 3) = 13 ×1 × 12 + 2
3
× 13 ×1 = 18
7

P33( 3) = 13 ×1 × 12 + 2
3 × 23 × 23 = 25
54

P3(i3) = ( 274 , 187 , 54

25
)
Hence.

a1 a2 a3
a1 F 1 1 7
I
P ( 3)
= a2 GG6
7
4
7
12
37 JJ
a3 GH
36
4
27
24
7
18
72
25
54
JK
Again, we see that the rows add up to 1, corresponding to the fact that if we start at a
given state, we must reach some state after 3 steps.

Example:
Suppose that we are interested in studying the way in which a given state votes
in a series of national elections. We shall base our prediction only on past
history of the outcomes of the elections (Republican or Democrat). As a first
approximation we assume that knowledge of the past, beyond the last election,
would not cause us to change the probability for the outcome on the next
election.
So we have a markov process , or chain with 2 states, Republican ( R ) and
Democrtat ( D ).
R D
R 1− a FG
a IJ
D b H
1− b K
where a can be estimated from past results and represents the fraction of years
in which there was a change from R to D., and b represents the fraction of
years in which there was a change from D to R.
We can obtain a better approximation from past results of two previous
elections.
 Our states now are RR, RD,DR,DD, and if next election is a Democrat, RR ->
RD, etc. The first letter of the new state must be the second letter of the old
state. So, if election outcomes for a series of elections is DDDRDRR our
process moves DD→ DD → DR → RD → DR → RR etc
Hence

RR DR RD DD
F
RR 1 − a 0 a 0 I
DR G b 1− b 0 J
G
RD G 0
0
1− c
J
c J
G
DD H 0 d
0
0
J
1− dK

where a,b,c,d, must be estimated.

Noting from Matrix theory that a regular matrix is one in which the power of
the matrix has only positive components. Then, if regular, the transition matrix
has following properties:
i) Pn → W
ii) Each row of W is same probability vector w.
iii) Components of w are positive
iv) If p is any probaboility vector, p Pn → w
v) w is the unique fixed point probability vector of P. (i.e. a
probability vector w, is a fixed point of the matrix P if w=wP)

The fixed probabilty vestor is

FG bd ad ad ca IJ
H
bd + 2ad + ca bd + 2ad + ca bd + 2ad + ca bd + 2ad + ca K
bd + ad
∴ probability of being in state RR and DR =
bd + 2ad + ca
Example:
Consider

P=G
F I and if w = (.6 .4) we see that
2 1

H JK
3 3
1 1
2 2

F I= (.6,.4) = w
(.6,.4) G
2 1

H JK
3 3
1 1
2 2
Suppose we compute the power of the matrix P
 F I×F I= FG.611 .389IJ
2 1 2 1
P2 = GH JK GH JK H.583 .417K
3
1
2
3
1
2
3
1
2
3
1
2

P3
F.611 .389IJFG IJ= FG.602 .398IJ
=G
2 1

H.583 .417KH K H.597 .403K

3 3
1 1
2 2
and so on until we could show

Pn →
FG
.6 .4 IJ
H
.6 .4 K
Each row is a fixed point w of matrix P
i.e.
w=wP
FG IJ
w1 2 1
= ( w1 , w2 ) 13 13
LM OP
H K
w2 2 2 N Q
w1 = 23 w1 + 1
2 w2
w2 = 21 w1 + 1
2 w2
w1 = 23 w2
We can state this more rigorously using the Chapman-Kolmogorov equations.
We define the n step transition probability, Pijn as the probability that a process
is in state I will be in state j after n additional y-transitions
Pijn = P{ X n + m = j | X m = i} n ≥ 0, i, j ≥ 0
The Chapman-Kolmogorov equations provide a method for computing the n
stage probability, and are

∞
Pijm+ n = ∑ Pikn Pkjm
k =0

Note: Pikn Pkjm → probability that starting in i the process will go to state j
in n + m steps through a path that takes it into state k at the nth transition.

Formally,

Pijn+ m = P{ X n+ m = j| X 0 = i} = ∑ 0 P{ X n+ m = j , X n = k |X o = i}
∞

∞
= ∑ P { X n + m = j| X n = k , X 0 = i} P { X n = k | X 0 = i}
k =0
∞
= ∑ Pkjm Pikn
k =0

If P(n) denotes the matrix of n-step transition probabilolity, Pijn , then

 P ( n + m) = P ( n) ⋅P ( m)
i.e.
P ( 2 ) = P (1) ⋅P (1) = P 2
In general,
P ( n ) = P ( n − 1+ 1) = P ( n − 1) ⋅P (1) = P n

Example:
Consider th e example in which th e weather is a two stage Markov Chain, with
transition probability matrix as α =.7, and β=.4,

P=
.7 .3 LM OP
.4 .6 N Q
Calculate the probability that it will rain four days from today, given that it
rained today.

P (2) = P 2 =
FG
.7 .3
×
IJ FG
.7 .3
=
IJ FG
.61 .39 IJ
H
.4 .6 K H
.4 .6 .52 .48K H K
2
P ( 4 ) = P ( 2) = P 4 =G
F.61 .39IF.61
JG .39I F.5749
J= G .425IJ
H.52 .48KH.52 .48K H.5668 .433K
Therefore, fir example, P004 = probability of going from state 0 (rain) to state
0 (rain) in 4 steps = 5749
The probabilities that we have been considering have all been conditional probabilities.

Pijn = probability that state at time n is j, given that the initial state at time 0 is i.
If we want the unconditional distribution of the state at time n, we need to specify the
probability distribution of the initial state. α i = P { X 0 = i} where i ≥ 0 and ∑ α i = 1
i
Example:
Let X = { X n ; n ∈ N } be a Markov Chain with states a,b,c. (state space,
E={a,b,c}) and transition matrix
LM
1/ 2 1/ 4 1/ 4 OP
MM
P 1/ 3 0 1/ 3 PP
N
3/ 5 2 / 5 0 Q
Then
P[ X 1 = b, X 2 = c, X 3 = a , X 4 = c, X 5 = a , X 6 = c, X 7 = b| X 0 = c]
= P ( c, b) P (b, c) P ( c, a ) P (a , c) P ( c, a ) P ( a , c) P ( c, b)
= 25 13 53 14 35 41 25 = 2500
3

The two step transition probabilities are

 LM1 / 2 OP LM
1/ 4 1/ 4 1/ 2 1/ 4 1/ 4 17 / 30 9 / 40 5 / 24 OP LM OP
P 2
= M1 / 3 PP MM
0 1 / 3 1 / 3 0 1 / 3 = 8 / 15 3 / 10 1 / 6 PP MM PP
MN3 / 5 2 / 5 0 3/ 5 2 / 5 0 QN 12 / 30 3 / 20 17 / 60 Q N Q
So that, for example
P { X n + 2 = c| X n = b} = P 2 (b, c) = 1 / 6

We can write
∞
P { X n = 5} = ∑ P{ X n = j| X 0 = c} P {X i = c}
i =0
∞
= ∑ Pijnα i
i =0
For example, considering the Markov Chain from the previous problem, given
α 0 =.4; α 1 =.6 , then the unconditional probability that it will rain 4 days from now is
P[ X 4 = 0} =.4 P004 + .6 P104
=.4(.5749) + .6(.5668)
=.5700
Example:
Let X be a Markov Chain with state space E=[1,2], initial distribution =
(1/3,2/3) and transition matrix

P=
.5 .5 LM OP
.3 .7 N Q
LM .4 .6 OP LM .38 .62 O LM .376 .624 OP
.628PQ
P2 = P3 = P4 =
N.36 .64 Q N.372 N.3744 .6256Q
In general, it can be shown that

m
L+
=M
3
8
5
8
(.2) m 5
8
− 5
8
(.2) m OP
P
MN − 3
8
3
8
(.2) m 5
8
+ 3
8
(.2) m PQ
We have
 P[ X 1 = 2, X 4 = 1, X 6 = 1, X 18 = 1| X 0 = 1]
= P (1,2) P 3 ( 2,1) P 2 (11 , ) = (.5)(.372)(.4)( 83 +
, ) P 12 (11 5
8
(.2)12 )
= 0.26100
And
P[ X 2 = 1, X 7 = 2, X 9 = 2| X 0 = 1]
= P 2 (11
, ) P 5 (1,2) P 2 ( 2,2) = (.4)( 58 − 5
8
(.2) 5 )(.64)
= 015995
.
P[ X 2 = 1, X 7 = 2] = ∑ P{ X 0 = i} P{ X 2 = 1, X 7 = 2| X 0 = i}
i

= π (1) P (11
2
, ) P 5 (1,2) + π ( 2) P 2 ( 2,1) P 5 (1,2) = 1 / 3(.4)( 58 − 5
8
(.2) 5 + 2 / 3(.36)( 58 − 5
8
(.2) 5
= 0.23326

State j is accessible from state I if Pijn > 0 for some n ≥ 0.

Two states that are accessible to each other are said to be communicate, i ↔ j.
Any state communicates with itself since Pii0 = P[ X 0 = i| X 0 = i ] = 1
Communication satisfies the following three properties:
i. State I communicates with state I, for all i ≥0.
ii. If state I communicates with state j, then state j communicates with
state i.
iii. If state I communicates with state j, and state j communicates with
state k, then state I communicates with state k.
Two states that communicate are of the same class. A chain is irreducible if there
is only one class (i.e. all states communicate with each other).

Example: Gambler's Ruin

Markov chains has implications in mant areas. For simplicity we use gamblind
as an example. Suppose you are gambling against a professional, or gambling
house. We select a particular game to play, where you have a chance , p[, of
winning. The gambler has made sure that he has the edge, so that p > 1/2.
However, in most situations, p will be close to 1/2.
At the start you have A dollars, and the gambler has B dollars. You bet $1 on
each game and play until one of you is ruined.
Find the probability that you will be ruined.
The answer depends on exact values of A and B. Let N= A+B - total amount
of dollars in the game. As states for the chain we choose 1,2,… .,N. At any
one moment the position of the chain is the amount of money you have.
The initial positions is:
Your money. His money.
O A B

If you win, your money increases by 1$ and his decreases by 1$. Theus, the
new position is one state to the right. If you lose, the chain moves 1 step to the
left. Thus, at any step there is a probability p of moving one step to right and
probability (1-p)=q of moving 1 step to the left. We have a: Markov Chain!
If the chain reaches 0 or N we stop. These are called ABSORBING
STATES, since once they are entered they are never left. We are interested in
the probability of your ruin (reaching 0).

Suppose p and N are fixed. We actually want he probability of ruin when we

start at A. But we will solve a more complicated problem. Find the ruin-
probability for every possible starting position. So, let's introduce the notation
Xi to stand for the probability of your ruin when you start at position i (i.e. if
you have i dollars). We solve for N=5, so we have unknown X1, X2, X3, X4,
X5, Suppose we start at position 2, the chain moves to 3 with probability p, or
to 1 with probability q.
Thus

P{ruin/start at 2}=P{ruin/start at 3}*p + P{ruin/start at 1}*q

But once it reaches stage 3, a Markov Chain behaves like it had been started
there. So,
P{ruin/start at 3} = X3
P{ruin/start at 1} = X1
Therefore we can write
X2 = pX3 + qX1
Substituting p+q=1
X2 = (p+q)X2 = pX3 + qX1
P(X2 -X3) = q(X1 -X2)

Hence,
X1 - X2 = r(X2 - X3)
Where r = p/q, and hence, r<1.
We can writ this for each of the four ordinal positions

X0 − X1 = r( X1 − X2 )
X0 − X1 = r( X1 − X2 )
X0 − X1 = r( X1 − X2 )
X1 − X 2 = r( X 2 − X 3)
X2 − X 3 = r( X 3 − X4 )
X3 − X 4 = r( X 4 − X5)
 We have our absorbing states X0 = 1 and X5 = 0. If we substitute X5 into the
above we get X3 - X4 = rX4, which is substituted into the previous equation.

X 4 = 1⋅ X 4 1
X 3 − X 4 = r ⋅X 4 2
X 2 − X 3 = r ⋅X 4
2
3
X 1 − X 2 = r ⋅X 4
3
4
X 0 − X 1 = r ⋅X 4
4
5

If we could add all these

X 0 = (1 + r + r 2 + r 3 + r 4 ) X 4
But X 0 = 1, and
(1 − r 5 )
(1 + r + r 2 + r 3 + r 4 ) =
(1 − r )
we get
1− r
X4 =
1− r5

Also, adding equations 1 and 2 gives;

1− r2
X 3 = (1 + r ) X 4 =
1− r5

Adding 1,2 and 3 gives,

1− r3
X2 =
1− r5

Etc.
1− r4
1 =
The same procedure will work for any value ofXN. In the5 denominator the expponent of r
1
is always N, In the numerator the exponenet is always N-A,− r which equals B. Thus, the
ruin probability is 1− r
X4 =
We recall that A is the amount of money you have, B1 −isrthe
5
gambler's stake, N=A+B, p
your probability of winning a game, and r = p/(1-p). In the tables below, are shown some
typical values of the ruin-probability. Some of these are quite startling. If the probability
of p is as low as .45 (odds against you on each game 11:9) and the gambler has $20 to put
up, you are almost sure to be ruined. Even in aB nearly fair game, say p - .495, with each
of you having $50 to start with, there − r chance for your ruin.
a 1.731
X is
A =
1− r N
It is worth examining the ruin-probability formula above. More closely. Since the
denominator is always less than 1, your probability of ruin is at least 1-rB. This estimate
does not depend on how much money you have, only on p and B. Since r is less than 1,
by making B large enough we can make rB practically 0, and hence, it is almost certain
that you will be ruined.
Suppose, for example, that a gambler wants to have probability .999 of ruining
you. (You can hardly call him a gambler under those circumstances!) He must make
sure that rB < .001. For example, if p = .495, the gambler needs $346 to have a
probability .999 of ruining you, even if you are a millionaire. If p = .48, he needs only
$87. And even for the almost fair game with p = .499, $1727 will suffice..
There are two ways that gamblers achieve this goal. Small gambling houses will
fix the odds quite a bit in their favor, making r much less than 1. Then even a relatively
small bank of B dollars suffices to assure them of winning. Larger houses, with B quite
sizable, can afford to let you play nearly fair games.

SUMMARIZING

Consider a random process or stochastic process, { Xn, n = 0,1,2,… } that takes on a finite
number of possible values. If Xn = i, then the process is said to be in state I at time n.
We suppose that whenever the process is in state i there is a fixed probability, Pij, that it
will bnext in state j'
P { X n + 1 = j| X n = i , X n− 1 = in− 1.... X 0 − i0} = Pij
i.e.
for all states i0 , i1 ,...in − 1 , i , j , and all n > 0
This process is called a Markov Chain. We can interpret this as follows; For a Markov
chain, the conditional distribution of any future state Xn+1, given the past states X0, X1,
… ..Xn-1, and the present state Xn, the conditional probability is independent of the past
states and depends only on the present state
Pij ≥ 0 i, j ≥ 0
∞
∑ Pij = 1
j =1
An interesting example is an absorbing state Matkov chain.

A state, Si, is absorbing if Pii = 1 and Pij =0 for j≠i. Once the system is in Si, it cannot
leave. Always arrange the matrix in a standard form (absorbing states listed first).
Consider 5 states, 2 absorbing
 LM 1 0 0 0 0OP
M
P = MP
0 1 0 0 0
PP LI 0 OP
MM P
31
41
P32
P42
P33
P43 P44 P45PP MN
P34 P35 =
R Q Q
MN P
51 P52 P53 P54 P55 PQ
Consider the case where R and Q are such that it is possible to eventually get into an
absorbing state from every other state. (There are many other variations)
Let mij = the mean number of times the system is in state Sj, given it started from SI,
where Si and Sj are non absorbing states. From Si, the system could go to an absorbing
state in one step, or could go to an non-absorbing state, sat Sk, and eventually be absorbed
from there. So sum over all non-absorbing states;

mij = δij + ∑ Pik mkj ; δij =

1 i= jRS
k 0 i≠ j T
δij accounts for the fact that if chains goes to absorbing state in one step, it was in state
Si one time. We can write this for all states;

M == I + QM
−1
M= I− Q
mi = ∑ k mik - over all non - absorbing states

Consider now the problem of absorption into various absorbing states. Let
aij = Probability of system absorbed in state Sj, given it started in state Si
(Si non-absorbing state and Sj absorbing state)
The system could move to Sj in one step, or it could move to a non-absorbing state, Sk,
and be absorbed from there
∴ aij = pij + ∑ pik a kj
k
non - absorbing states
∴ A = R + QA
or
A = ( I - Q) -1 R = MR
Example:
The manager of a plant has employees working at level I and level II. New
employees mat enter either level.. At the end of each year the performance of
employees are evaluated. They can be either reassigned to level I or II jobs,
terminated, or promoted to level III, in which case they may never come back
to levels I or II.
We can treat this as a Markovian process, with two absorbing states
Employment at level I - S3
 Employment at level II - S4
Records over long periods of time indiocate the following probabilites as valid
A1 A 2 A 3 A4
LM 1 0 | 0 0 OP
M
P = M− −
0 1 | 0
− − − − − − =
0
PP LM
I 0 OP
MM .2 .1 | .2 .5 PP N
R Q Q
N .1 .3 | .1 .5 Q
Thus, if employee enters levek I, the probabiltiy is .5 that she will jump to level
II and .2 that she is terminated.
Find: The expected number of evaluations an employee must go through in this
plant i.e. we want to find the expected number of steps to absorbtion from non-
absorbing state i, then sum over all i.

M = ( I − Q) − 1 and from P , we have R =

.2 .1 LM OP
and Q =
LM
.2 .5 OP
.1 .3 N Q N
.1 .5 Q
L.8 .1OP = LM
=M
−1 10
7
10
7
OP
N .1 .3Q N 2
7
16
7 Q
Hence,
m1 = m11 + m12 = 20/7
m2 = m21 + m22 = 18/7
Which concludes that new employees can expect to remain there through 20/7
evaluations while in level I.
Also
LM
10 10
.2 .1OP L OP
Q MN
A = MR = 72 167
7 N 7
.1 .3 Q
∴ A=
LM OP
3
7
4
7

N Q
2
7
5
7
That is, an employee entering at level I has a probability 4/7 of reaching level
II, whereas, at level II raises the probability to 5/7.

Consider a system that can be in any of a finite number of states and assume it moves
from state to state according to a prescribed probability law.
Let Xi be the state of the system at time point I, and that there are S1 and S2 possible
states. We are interested in probabilities of going from one state to another.
P { X i = Sk | X i − 1 = S j } = Pjk
Where Pjk is the transition probability from Sj to Sk. This probability is independent of i,
that is, the transition depends only on present state. The even {Xi = Sk | Xi-1 = Sj} is
independent of past history.
The probabilities can be conveniently displayed in a matrix
 LM P 11 P12 L P1mOP
P=M PP
M M
MM P
NP m1L mm P Q
Let X0 be starting stat of the system with probability given by
Pk( 0) = P ( X 0 = S k )
Let P{being in Sk after n steps} = Pk( n ) . These are conviently displayed as;
P ( 0) = [ P1( 0) , P2( 0) .... Pm( 0) ]
M
P ( n ) = [ P1( n) , P2( n ) .... Pm( n) ]
The relationship between these probabilities can be ascertain by considering the
following:

P(1) = probability of being in various states after 1 step = P ( 0) = [ P1( 0) , P2( 0) .... Pm( 0) ] . If we
multiply this vector by the transition probability matrix,
P(2) =[probability of being in state k after 1 step][probability of going to a next state]

Hence, we get P ( n ) = P ( n− 1) P .
Definition: If some power of P (Pn) has all positive entries, P is said to be regular. So,
one can get from state Sj to state Sk eventually, for any (j,k)
Definition: If P is regular, the Markov chain has a stationary distribution that gives the
probability of being in the respective states after many transitions. i.e. Pj(n)
must have a limit, Π j as n → ∝, which satisfies Π = Π P

Example:
Market stocks 3 brands of coffee, A, B, C, and it has been noticed that
customers switch from brand to brand according to the following transition
matrix:
LM3 / 4 1/ 4 0 OP
MM
P= 0 2/3 1/ 3
P
N1 / 4 1/ 4 1 / 2 PQ
where S1 → purchase of A
S2 → purchase of B
S3 → purchase of C
So, 3/4 of A today switches to A tomorrow (stays with A), 1/4 of A today
switches to B tomorrow, and 0 of A toad switches to C tomorrow (no one
switches).

(a) Let's find the probability a customer switches to A today will swithc to A
two weeks from now, assuming she buys coffee once/week
Customer starts with purchase of A:. then
 P (1) = P ( 0) P ; (1,0,0)[ D]
3 1
=( 0)
4 4

3 1
3/ 4 1/ 4 0LM OP
P =(
(2)
MM
0) 0 2 / 3 1 / 3 PP
4 4
N
1/ 4 1/ 4 1/ 2 Q
FG 9 17 1 IJ
=
H16 48 12 K
(b)In the long run, what fraction of customers will buy the respective brands?
We need the staionary distribution, Π = Π P

FΠ I LM 3 / 4 1/ 4 0 OP
GGΠ 1
JJ= b Π Π Π g 0
MM 2/3 1/ 3
P
GH Π
2

3
JK 1 2 3

N1 / 4 1/ 4 1 / 2QP
LM 3
4
Π + Π
1
1
4
OP
3
=M Π
1
+ Π + Π P
2 1

MN PQ
4 1 3 2 4 3
1
3
Π + Π
2
1
2 3

Solve this , along with the constraint that Π 1 + Π 2 + Π 3 = 1 (must be

somewhere).
From (1) equation (1 − 3 / 4) Π 1 = 1 / 4Π 3 ∴ Π 1 = Π 3
Using (2) or (3) 1 / 2Π 3 = 1 / 3Π 2 ∴ Π 2 = 3 / 2Π 3
Substituting this into Π 1 + Π 2 + Π 3 = 1 , we have
Π 3 + 3 / 2Π 3 + Π 3 = 1
∴ 7 / 2Π 3 = 1
And
Π 3 = 2 / 7, Π 1 = 2 / 7, Π 2 = 3 / 7

We therefor conclude that the store should stock more B than A or C coffee
 Discrete-Time Markov Chains

The Markov chain assumptions and the transition matrix

A store stocks floppy disks. The disks are sold one by one and an inventory is taken at the
end of each month. Let Xt denote the number in stock at the end of the tth month. (X0 is the
number in stock when the store first opened; X1 is the number in the end of the first month;… ) If
Xt falls below a fixed critical level, more disks are ordered. Xt+1 depends on two quantities: Xt and
the number sold between time t and time t+1. Since the number sold is random, Xt does not
completely determine Xt+1. But the basic assumption is that given the value of Xt, in trying to
predict what Xt+1 will be, all the past records Xt-1,Xt-2,… ,X0 are irrelevant; Xt+1 depends only on the
number Xt now in stock at time t and the number sold during this month. This is the Markov
assumption that the future is independent of the past given the state of affairs in the present.

Here is a more abstract formulation keeping all the essentials intact: Consider a "system"
(the shelf and the floppy disks in stock) that can be in any of a number of "states" (the actual
number of items in stock). The set of states is called the state space S and we will assume in
general that S={0,1,2,… }. Now suppose that a "particle" is free to jump among the states of the
state space S; its location at time t is Xt. In this way we have a stochastic process {X t }
∞
t =0
. The
location Xt is measured only at the discrete times t=0,1,2,… . X0 is the starting location at time 0.

The following Markov chain assumptions will be assumed to hold throughout the lecture

1. Suppose that the particle is in state (or location) i at time t. Then regardless of its history
prior to time t, the probability that it will jump next to another state j depends only on i. In
other words: At time t we see that the particle is occupying state i (Xt=i). Then whether it
was in that state for 500 years or whether it just jumped to i does not matter; in trying to
predict the particle's future state after time t, only the present state i is what counts.
Mathematically, this is expressed as follows: Let i, j, it − 1 , K , i0 ∈ S .Then for any time t,

P ( X t + 1 = j | X t = i, X t − 1 = it − 1 , K , X 0 = i0 )= P ( X t + 1 = j | X t = i )

That is, the future (time t+1), given the present (time t), is independent of the past (times t-
1,… ,0). The probability above is the transition or jump probability from state i to state j.

2. Not only are the transition probabilities independent of

the past states of the particle once it is known where
the particle is now, but the transition probabilities
are independent of t:

P ( X t + 1 = j | X t = i )= Π i , j
This assumption is called homogeneity in time.

A (homogeneous) Markov chain in discrete time consists of a particle that jumps at each
unit of time between states in a state space S. Xt denotes the state occupied at time t for
t=0,1,2,… . If the particle is in state i at time t, it will be in state j at time t+1 regardless of
the states occupied before time t with probability
 Π i, j = P ( X t+ 1 = j | X t = i )

Andrei Andreevich Markov (1856-1922) was Chebyshev's most outstanding student. Markov was
most involved in proving limit theorem and the law of large numbers of Chapter 10. Although
Markov originally used Markov chains in abstract settings to prove various theorems, they were
quickly applied. Markov himself illustrated the ideas by examining vowel-consonant interchanges
in 20,000 letters of Pushkin's poetry.

Example
Let S = (0,1) and the transition probabilities be
1 2 1 3
Π 0 ,0 = Π 0,1 = Π 1,0 = Π 1,1 =
3 3 4 4

Figure 1, below, depicts this.

1/3 3/4

2/3

. .
0 1
1/4

Figure 1

A table or matrix that displays the jump probabilities is

1 / 3 2 / 3
Π= 
1 / 4 3 / 4

There is a standard way of writing the jump probabilities Π i, j as a table. This is called
the transition matrix Π . The element in the ith row of and jth column is Π i, j - the probability
to that the particle jumps from i to j.

Π 0,0 Π 0,1 Π 0,2 L Π 0, N 

Π Π 1,1 Π 1,2 L Π 1,N 
Π= 
1, 0

L L L L L 
 
Π N ,0 Π N ,1 Π N ,2 L Π N ,N 
Using matrix terminology, the ij entry of Π , is Π i, j . Note that the ith row of Π displays the
jump probabilities from state i; the jth column displays the jump probabilities to state j. For
example, if the third column consisted of all 0's, that is, if

Π i,3 = 0
for all states i in the state space S, the particle could never enter state 3, then Π i 0, 3 > 0 .

Let Π be a Markov chain transition matrix.

Then

1. 0 ≤ Π i , j ≤1 for all i,j in the state

space S.
2. Π has row sums 1:
N N

∑ Π i , j = ∑ P ( X t + 1 = j | X t = i )= 1
j =0 j =0

Property 2 is true since at time t+1 the particle must be located somewhere. (If it could disappear,
the state of disappearance would have to be included as a state in the state space.)

Example
In this chain there are three states; S={0,1,2}. From state 0 the particle jumps to states
1 or 2 with equal probability 1/2. From state 2 the particle must next jump to state 1.
State 1 is absorbing; that is, once the particle enters state 1, it cannot leave. Draw the
diagram and write down the transition matrix.

. 1
1

1/2
1

. .
0 1/2 2

Figure 2

Solution:
 Fig. 2 depicts the transition probabilities. The zeroth row of Π consists of the jump
probabilities from state 0 and similarly for the other two rows. Check that

 1 1
0 2 2
Π = 0 1 0
 
0 1 0
 

State i is absorbing if Π i ,i = 1.

Example: A Random Walk on S ={0,1,2,… ,N}

From any of the interior states 1,2,… ,N-1 the particle jumps to the right to state i+1
with probability p and to the left to state i-1 with probability q=1-p. That is, for
1 ≤i ≤ N − 1 .

Π i ,i + 1 = p Π i ,i − 1 = q Π i, j = 0 for j ≠ i ±1

This corresponds in the language of gambling to the following game: Flip a coin; if
heads, then I win $1; if tails, then I lose $1. Consequently, at each flip I jump to state
i+1 with probability p or to state i-1 with probability q, assuming that I presently
have $i. Various assumptions can be made about the behavior of the particle at the
boundary states 0 and N.

Case 1. They could both be absorbing, in which case

Π 0 ,0 = 1 Π N ,N = 1

This corresponds to the fact that the game is over once I have $0 or if I win all the
opponent's money.

1 0 0 0 L L 0
q 0 p 0 L L 0
 
0 q 0 pL L 0 
Π= 
L L L L L L L 
0 0 0 0L q 0 p 
 
0 0 0 0L L L 1 

r
s
1-r q p 1-s
 . . . . . . .
0 1 i N-1 N

Figure 3

Case 2. They could be reflecting, in which case

Π 0 ,0 = 1 Π N ,N − 1 = 1
This corresponds to having my opponent give me one of his/her dollars when I run
out and conversely.

Case 3. They could be partially reflecting, as depicted in Fig. 3 in which case

Π 0 ,0 = r Π 0,1 = 1 − r Π N ,N = s Π N ,N − 1 = 1 − s

r 1-r 0 0 0 K . 0 
q 0 p 0 0 K . 0 
 
0 q 0 p 0 K . 0 
Π= 
...................................... 
0 0 0 0 0 .. q 0 p 
 
0 0 0 0 0 .....1-s s 

Note that case 3 includes cases 1 and 2 for particular choices of r and s.

Example A Renewal Process

Consider a component whose age can be 0 or 1 or 2 or … . Age 0 means "just

installed". Assume that no matter how old the component is, it will burn out during
the next interval of time with probability q or continue operating with probability
p=1-q. Thus the component obeys the lack of aging property. The state space is
S={0,1,2,… } and the state of the system is the age of the component presently
installed. Assume that as soon as the component burns out, it is instantly replaced and
then the state of the system becomes 0. Notice that the transition from state 0 to state
0 occurs if the component just installed immediately burns out.

q p 0 0 .....
q 0 p 0 0 ...
Π= 
q 0 0 p 0 ...
 
.................... 
 Note that the state space S, while discrete, has infinitely many states. (This model is
also called a birth or disaster model; see Problem 12.26.) Certain applications of this
model are clear from the terminology - component lifetimes. But here is one that is
not so obvious: Notices pile up on a bulletin board at a more-or-less constant rate
until someone decides to throw them all away. The state of the system is the number
of days since the bulletin board was last cleared. If the clearing of the board is done
at random independent of how many notices or the time since the last clearing, the
bulletin board will be cleared at the next unit of time with a constant probability q.

THE TTH-ORDER TRANSITION MATRIX

The transition matrix Π displays at a glance the transition probabilities Π i, j . Suppose that we
need to find probabilities such as

P(X t+ 3 = j | X t = i )

that the particle will be in state j three jumps from now. The one-step probabilities Π i, j are the
entries of the matrix Π . From these, how can one find the three-step probabilities, and more
generally the t-step probabilities?

Definition

The tth-order transition matrix is Π t , whose

ijth entry is

Π ti , j = P ( X t = j | X 0 = i )

which is the probability of jumping from i

to j in t steps.

Note that Π 1 = Π . Why? It should be clear that homogeneity in time (that the transition
probabilities do not depend on t) implies that regardless of the time u ≥ 0 ,

P ( X t + u = j | X u = i )= Π it , j

That is, the t-step transition probabilities depend only on the time difference.

A general algorithm is needed to find the tth-order transition matrix Π t for any given Markov
chain matrix Π .
To find the (t+1)st-order transition matrix from the tth, use the basic Markov assumptions:
Suppose that the particle starts in state i at time 0. For the particle to be in state j at time t+1, it
must have traveled through some state k at the intermediate time t. Consequently, where the
particle was at time t partitions the event "in state j at time t+1 given state i at time 0."

Π ti ,+ j1 = P ( X t + 1 = j | X 0 = i )
N
= ∑ P ( X t + 1 = j and X t = k | X 0 = i )
k =0
N
= ∑ P ( X t + 1 = j | X t = k and X 0 = i )P ( X t = k | X 0 = i )
k =0
N
= ∑ P ( X t + 1 = j | X t = k )P ( X t = k | X 0 = i )
k =0
N
= ∑ Π k , j Π ti ,k
k =0
N
= ∑ Π ti ,k Π k , j
k =0

The second equality follows by partitioning on where the particle was at time t. The third equality
follows from this identity

P ( A ∩ B | C )= P ( A | B ∩ C )P (B | C )

which follows from the definition of conditional probability (see Problem 3.8). The fourth
equality uses the Markov chain assumption that the chance that the particle is at j at time t+1
given that it was at k at time t is independent of the fact that it was at i at time 0.

In terms of matrix multiplication we have shown that

Π t+ 1 = Π t Π

The result can be generalized:

Π t + 2 = Π t + 1Π = (Π t ⋅Π )Π = Π t Π 2

using what we have just proved together with the fact that matrix multiplication is associative. In
general, we have proved the

Chapman-Kolmogorov Equation

Let times t, s ≥ 0 . Then for all states i,j

N
Π ti ,+ js = ∑ Π it ,k Π ks , j
k =0
 In terms of matrix multiplication, the (t+s)th-
order transition matrix is the product of the tth
and the sth:

Π t+ s = Π t Π s

In words: For the particle to start at i at time 0 and be in j at time t+s, it must be in some state k at
the intermediate time t.

Example
Convert the jump probability diagram of Fig. 4 into the corresponding Markov chain
and find the probability that the particle will be state 1 after three jumps given it
started in state 1.

0.9

0
. . 2

0.7
0.8 0.2
0.1 0.3

.
1

Figure 4
Solution
 0 .1 .9
Π = .8 0 .2
 

. 7 .3 0 

.71 .27 .02
Π = Π Π = .14 .14 .72
2 1 1
 

 .24 .07 . 69 

.230 .077 .693
Π = Π Π = .616 .230 .154
3 2
 

.539 . 231 . 230


Consequently,

P ( X 3 = 1 | X 0 = 1)= Π 13,1 = .230

The Probability Vector ρ t

So far we have seen how to compute conditional probabilities of the form P ( X t = j | X 0 = i ).

But now suppose that it is known for a fact that the particle started in state i0 at time 0. Then what
is P ( X t = j )? More generally, assume that the particle starts in state i with probability ρi at
time t=0. We want to answer the question: With the starting probabilities ρ0 , ρ1 , K , ρ N , what is
P ( X t = j ) for any state j?

Let the initial probability vector be defined

ρ = (ρ 0 , ρ1 , K , ρ N )
Note that 0 ≤ ρi ≤1 for all states i in the state space S and
ρ0 + ρ1 + K + ρ N = 1
since the particle must start somewhere at time 0. The probability vector at time t is defined as
ρ t = (ρ t 0 , ρ t 1 , K , ρ t N )
where
ρ tj = P ( X t = j | initial probabilit y vector ρ )
That is, ρ tj is the chance that the particle will be found in state j given that at time 0 it started in
the various states with probability ρi for i = 0,1,… ,N. Note that
ρ0 = ρ
and
N N

∑ ρ tj = ∑ P ( X t = j )= 1
j =0 j =0

That is, for each t, ρ t is a probability vector.

 Definition

A probability vector ρ = (ρ 0 , ρ1 , K , ρ N ) satisfies

1. 0 ≤ ρi ≤1 for each i = 0,1,… ,N

2. ρ0 + ρ1 + K + ρ N = 1

There is a straightforward method for obtaining the probability vector ρ t at time t given the
initial probability vector ρ 0 at time 0 and the tth-order transition matrix Π t :

N
ρ tj = ∑ ρi Π it , j
i =0
Expressed as multiplication between a vector and a
matrix, the probability vector at time t is

ρ t = ρΠ t = ρ 0 Π t

To see why this result is true, we compute

ρ tj = P ( X t = j )
N
= ∑ P ( X t = j | X 0 = i )P ( X 0 = i )
i =0
N
= ∑ Π it , j ρi
i =0

where the first and second equalities are definitions of ρ t , Π t and ρ . The second equality
follows from the law of total probability.

Example

For the Markov chain in Fig. 5 find the chance that the particle will be in state 0 at
time 3 if it started in state 0 with probability 1/3 and in state 1 with probability 2/3 at
time 0.

Solution
 1 3
 4
Π = 4 
 

1 0


and
ρ = (1 / 3, 2 / 3)

Consequently

13 3
 16 
Π 2 = Π ⋅Π = 16
1 3
 
4 4
25 39 
 64 
Π 3 = Π ⋅Π 2 = 64
13 3
 
16 16 

ρ 3 = ρΠ 3
1 2 
=  , Π 3
3 3 
129 63 
= , 
192 192 

from which we conclude that

P ( X 3 = 0)= ρ 03 =
129
192

1/4

3/4
. .
0 1 1

Figure 5

Suppose that the particle starts at time t=0 in state i. In the terminology of probability
vectors, this means that
 ρ 0 = (0, K ,0,1,0, K ,0)

ith entry

Therefore, the probability vector at time t is

ith entry

ρ t = (0, K ,0,1,0, K ,0 )Π t
= (Π it ,0 , Π ti ,1 , Π it ,2 , K , Π ti ,N ,)

which implies that given that the particle started in state i,

P ( X t = j )= ρ tj = Π ti , j

which confirms what we already know: The ij entry of the matrix Π t is the chance of
being in state j at time t given in state i at time 0.

The Steady State Probability Vector

Suppose that there is a large number n of particles each jumping from state to state among the
states of S guided by the transition matrix Π of jump probabilities. If all n particles start in state
0 at time t=0, then after one jump some will remain in state 0 (if Π 0, 0 > 0 ) and others will jump
to other states. In fact, we can expect nΠ 0, j particles to be in state j after one jump. (Why?) On
the other hand, suppose that we distribute the n so that nj start in state j at time 0 for j=0,1,2,… ,N.
Since n j Π j ,i of those particles starting in state j can be expected to be in state i after one jump is

∑nΠ
j =0
j j ,i

It may just happen that this number is the same as the number ni of particles that started in state i
at time 0. Each of the particles might change states, but the overall number in state i would
remain constant. If this were true for each state i ∈ S , the entire system of n particles would be in
a steady state; for each particle that leaves a state, one would replace it from another state.
N
ni = ∑ n j Π j ,i
j =0
Rather than the absolute number ni of particles in state i, let us phrase the equation in terms of the
relative number ni/n of particles in state i; this is the probability that any one particle occupies
state i.
N n
ni
=∑ Π j ,i
j

n j =0 n
If this were the case, the entire system of n particles would be in the steady state.
A probability vector φ represents the steady state if
N
φi = ∑ φj Π j ,i
j =0

that is, if φ = φ⋅Π = φ. Thus the probability that a particle is in state i is the same at time 1 as
1

at time 0. Note that if φ has this property of "reproducing itself" after one jump, this will also be
true for all times t:
φ1 = φΠ = φ
implies
φ2 = φ1Π = φΠ = φ
φ3 = φ2 Π = φΠ = φ
and in general,
φt = φt − 1Π = φΠ = φ
We call any probability vector with the property that φ = φΠ a steady-state probability vector. If
the particle starts in state i with probability φi for each state i, then at every time t it will be in
state i with probability φi .

Procedure for Finding the Steady-State

Probability Vector

1. Set up and solve these equations:

N
φj = ∑ φi Π i , j
i =0

for j=0,1,2,… ,N or alternatively, in matrix

notation

φ = φΠ

2. Normalize by insisting that

∑ φ =1
i =0
i

Step 1 involve solving N+1 equations in the N+1 unknowns φ0 , φ1 , K , φN . As we will see in the
examples, there will always be redundancy; one of the equations will be a combination of the
others. The equation of step 2 is actually the (N+1)st.

Example
Find the steady-state probability vector of the Markov chain in Fig. 6

Solution
 φ = φΠ

(φ0 , φ1 )= (φ0 , φ1 )Π = (φ0 , φ1 )

1 / 2 1 / 2
1 0 
1/2

1/2

. .
0 11

Figure 6
which yields two equations in the two unknowns φ0 and φ1 :
1
φ0 = φ0 + φ1
2
1
φ1 = φ0
2
Both these equations are really the same:
φ0
φ1 =
2
Using the normalization condition,
φ0 φ
1 = φ0 + φ1 = φ0 + =3 0
2 2
Thus
2 1
φ0 = φ1 =
3 3
Consequently, two-thirds of the time the particle will be found in state 1; one-third of
the time will it be found in state 2 in the steady state. A simulation was run in which
66 particles were started in state 0 and 33 in state 1. Each jumped according to the
transition probabilities. Here are the results: The first column n is the number of
transition performed by each of the 99 particles; the second column is the fraction in
state 0.

n Fraction in state 0

1 .667
2 .667
3 .717
4 .636
 5 .677
6 .667
7 .647
8 .707

There are random fluctuations, but about 2/3 of the particles are found in state 0 at
any time t if 2/3 start in state 0.

Example
Consider the Markov chain in Fig. 7 Find the steady-state probability vector φ.

2/3
1 1/32
. .

2/3
1/3
1/2 1/2

.0
Figure 7
Solution

 0 1 / 2 1 / 2
Π =  0 2 / 3 1 / 3
 

1 / 3 2 / 3 0 

 0 1 / 2 1 / 2
(φ0 , φ1 , φ2 )= (φ0 , φ1 , φ2 ) 0 2 / 3 1 / 3

1 / 3 2 / 3 0 
results in these equations:
1
φ0 = φ2
3
1 2 2
φ1 = φ0 + φ1 + φ2
2 3 3
1 1
φ2 = φ0 + φ1
2 3
Clearing the fractions and combining factors of each φi :
 − 3φ0 + φ2 = 0
3φ0 − 2φ1 + 4φ2 = 0
3φ0 + 2φ1 − 6φ2 = 0

Adding the second and third equations results in

6φ0 − 2φ2 = 0
which is essentially the same as the first equation. Hence the third equation is
redundant. From the first two equations
φ2 = 3φ0

φ1 =
1
(3φ0 + 4φ2 )= 15 φ0
2 2
Now use the normalization condition
 15  23
1 = φ0 + φ1 + φ2 = 1 + + 3 φ0 = φ0
 2  2
Consequently,
2
φ0 =
23
15 15
φ1 = φ0 =
2 23
6
φ2 = 3φ0 =
23

Example
Consider the cyclic process depicted in Fig. 8. There are N+1 states 0,1,2,… ,N. For
each state i, 0<qi<1. The particle remains in state i with probability qi. Otherwise, it
jumps to the next state i+1 with probability pi=1-qi. If i=N, then i+1 is taken to be
state 0. That is, there is a "wraparound" from state N to state 0.
q0 q1 q2
qN

p0 p1 p2

. . . . . ... .
0 1 2 3 N

Figure 8

Solve for the steady-state probability vector φ.

Solution
 q0 p0 0 K 0
0 q p 0 K 0 
 1 1 
Π = 0 0 q2 p2 K 0 
 
KKKKKKKK 
p N 0 KKK 0 q N 
 

The equation φ = φΠ implies that

(φ0 , φ1 , K , φN )= (φ0 , φ1 , K , φN )Π
which "uncouples" to these N+1 equations:

φ0 = q0φ0 + p N φN
φ1 = p0φ0 + q1φ1
φ2 = p1φ1 + q2φ2
KKKK
or

p0φ0 = p N φN
p1φ1 = p0φ0
p 2φ2 = p1φ1
L L

Solve successively for each φi in terms of φ0 starting with the second equation:

p0
φ1 = φ0
p1
p1 p
φ2 = φ1 = 0 φ0
p2 p2
p2 p
φ3 = φ2 = 0 φ0
p3 p3
L
p0
φN = φ0
pN

Using the normalization condition:

 1 = φ0 + φ1 + L + φN
 p0 p0 p 
=
1 + + + L + 0 φ0
 p1 p 2 pN 

 1 1 1 
=
1 + + + L + p0φ0
 p1 p2 pN 

Let
1 1 1 1
C= + + + L +
p 0 p1 p 2 pN
Then 1 = Cp0φ0 , which determines φ0 . From the equations (*) we have:

For the cyclic process,

1
φi =
Cpi

where

N
1
C=∑
j =0 pj

The Markov chains in the last three examples have each had unique steady-state
probability vectors. This need not always be the case, as the next two examples show.

Example 10
Let
1 / 2 1 / 2 0 0 
1 / 2 1 / 2 0 0 
Π=  
 0 1 / 3 1 / 3 1 / 3
 
0 0 0 1 
Here are two distinct steady-state probability vectors:
1 1 
φ =  , ,0,0,0 
2 2 
ψ = (0,0,0,0,1)
These can be verified simply by checking that φΠ = φ and
ψ Π =ψ .
Example
Consider a random walk on the integers S={0,1,2,… } in which transitions only to the
right can occur as in Fig. 9. Assume 0<p<1 and q=1-p.

q q q
 p p p

. . . . ...

0 1 2 3

Figure 9

This chain is similar to the cyclic process of Example 9. Here pi=p is constant and
N =∞.

q p 0 0 0 L 
0 q p 0 0 L 
Π= 
0 0 q p 0 L 
 
L L L L L L 

φ = φΠ implies that

qφ0 = φ0
pφ0 + qφ1 = φ1
pφ1 + qφ2 = φ2
L L L L L L
Since 0<p, q<1, the first equation, and then the second, and then the third, … imply

φ0 = 0
φ1 = 0
φ2 = 0

Consequently, there is no steady-state:

Why didn't the Markov chain in Example 10 have a unique steady-state? Why did it
have two? And why didn't the Markov chain in Example 11 have any steady state at
all?

Looking at what happens in the chain in Example 10 reveals the reason. If the
particle starts in states 0 or 1, it stays in these forever. If it starts in state 3, it stays
there forever. If, finally, it starts in state 2, sooner or later it will jump to either state 3
or to the combination of states 0 and 1. Consequently, the entire chain "splits" into
two separate pieces, each with its own steady state. The chain is said to be
decomposable.
 How about this example? The particle has a "drift" to the right. Consequently, there is
no steady state. Start with a million particles and eventually they will all be very far
to the right of state 0.

There is also another complication in finding the steady state:

Example

With
0 1 / 2 1 / 2
Π = 1 0 0 
 

1 0 0 


it is straightforward verification to check that φ = (1 / 2, 1 / 4, 1 / 4 ) is a steady-state

probability vector, and in fact it is the only one. For a large number of particles, there
is a steady state: If n/2 start in state 0 and n/4 in each of states 1 and 2, there will be
approximately this many in each of the states at all subsequent times. But for one
particle, there is no steady state. If it starts in state 0, it must jump to {1,2}, from
which … . At even times it will be in state 0; at odd times in either 1 or 2. There is a
periodicity problem.

Birth and Death Processes in Discrete Time: Random Walks

If the state space S is {0,1,2,… ,N}, the vector equation φ = φΠ yields N+1 equations in the N+1
unknowns φ0 , K , φN . For N larger than four or five, solutions can be time consuming.
Fortunately, there is a fast method for finding φ in one of the most important special cases.

Definition

A birth and death process in discrete time is a

Markov chain with this property:

From any state i ∈ S transitions in one jump to i-1,

i, or i+1 only can occur.

There may be infinitely many states in S or S may be finite. For some states a transition to the
right only or to the left only may be possible. But for all states, the particle can jump at most one
unit to the right or to the left.

All random walk models of Example 3 are birth and death chains.

To simplify notation, set

 bi = Π i ,i + 1 ri = Π i ,i d i = Π i ,i − 1

(for birth, remain, and death) as in Fig 10

For each i
bi + ri + d i = 1

The transition matrix is

r0 b0 0 0 0L L L L 
d r b1 0 0L L L L 
1 1 
Π = 0 d 2 r2 b2 0 L L L L 
 
L L L L L L L L L L 

L L L L L L L 0 d N rN 


Note that d0=0 and bN=0 (for N finite). Why?

r0 ri rN

. b0 di bi dN
..… .. ..… ..
i

. 0. 1 i- i i+ N- N
1 Figure 10 1 1

Now suppose that there is a large number n of particles each jumping according to the birth and
death chain. Then assuming that the entire system of n particles is in the steady state, there will be
nφi particles in state i at any instant of time. The number of these that will jump next to state i+1
is therefore (nφi )bi . But in the steady state this must be compensated for an equal number of
particles moving down from i+1. This is (nφi + )d i + 1 . Otherwise, particles would "bunch up" to the
left or to the right of state i and the system would not be in the steady state. Consequently (after
canceling the factor n),

φi + 1d i + 1 = φi bi

Procedure for finding the Steady-State

Probability Vector φ
 Use the equation

bi
φi + 1 = φi
di+ 1

to solve for each φi in terms of φ0 . Then use

normalization

1 = φ0 + φ1 + L + φN

Example

Solve for φ for the Markov chain in Fig. 11

1/2 1/10 3/4

1/ 5/10 1/
3 4
1/2 2/3 4/10

0 1 2 3

Figure 11

Solution

Using the procedure

 b0 1/ 2 2
φ1 = φ0 = φ0 = φ0
d1 1/ 3 3
b1 2/3 4 4 2
φ2 = φ1 = φ1 = φ1 = ⋅ φ0 = 2φ0
d2 5 / 10 3 3 3
b2 4 / 10 8 8 16
φ3 = φ2 = φ2 = φ2 = ⋅2φ0 = φ0
d3 1/ 4 5 5 5

Use normalization

 3 16  77
1 = φ0 + φ1 + φ2 + φ3 = 1 + + 2 + φ0 = φ0
 2 5 10
Thus
10 3φ0 15
φ0 = φ1 = =
77 2 77
20 16φ0 32
φ2 = 2φ0 = φ3 = =
77 5 77
Example: The Random Walk on {0,… ,N} with Partially Reflecting Boundaries
For i=1,… ,N-1, bi=b, ri=r, and di=d are constant as in Fig. 12. b0=b and dN=d.

1-b r 1-d
b d b d

0 ..… i ... N

Figure 12

Note that bi=b for i=0,1,… ,N-1 and di=d for i=1,2,… ,N.
Thus

bi b
φi + 1 = φi = φi
d i+ 1 d

for i=0,1,… ,N-1. Using this relation recursively,

2 i
b b  b 
φi = φi − 1 =   φi − 2 = L =   φ0
d d  d 

Use normalization
 1 = φ0 + φ1 + L + φN
 b  b 2  b 
N

= 1 + +   + L +   φ0
 d d  d  
 

There are two cases:

Case 1: The asymmetric random walks, b ≠ d . Then this

normalization sum is a geometric series whose sum we know from Section 4.3.

1 − (b / d )
N+ 1
1= φ0
1− b/d

With constant

1− b/d
C=
1 − (b / d )
N+1

i
b 
φi =   C
d 
for i = 0,1,2, K , N

Case 2: The symmetric random walk, b=d. Then the

normalization sum is

1 = (1 + 1 + 12 + L + 1N )φ0 = (N + 1)φ0

Consequently,

1
φ0 =
N+ 1
i
b  1
φi =   φ0 = 1i φ0 = φ0 =
d  N+ 1

for each i. In the symmetric random walk the chance of finding the particle at any
state i is the same 1/(N+1) regardless of i.

In the asymmetric random walk the steady-state probabilities φi appear to be fairly

complicated. But they make sense - at least qualitatively. Suppose, for example, that
b>d. Then the ratio b/d>1. Thus

i
b 
φi =   C
d 
 shows that φi is increasing with i. That is, the particle will be found with higher
probability closer to the right boundary than to the left. This is certainly what we
would expect since the right boundary "attracts" the particle (b>d). Here are the
actual probabilities for N=10 for several values of b/d:

STEADY-STATE PROBABILITY φi

State i b/d=1 b/d=1.2 b/d=1.5 b/d=2 b/d=5

0 .0909 .0311 .0058 .0005 .0000

1 .0909 .0373 .0088 .0010 .0000
2 .0909 .0448 .0132 .0020 .0000
3 .0909 .0537 .0197 .0039 .0000
4 .0909 .0645 .0296 .0078 .0001
5 .0909 .0774 .0444 .0156 .0003
6 .0909 .0929 .0666 .0313 .0013
7 .0909 .1115 .0999 .0625 .0064
8 .0909 .1337 .1499 .1251 .0320
9 .0909 .1605 .2248 .2501 .1600
10 .0909 .1926 .3372 .5002 .8000

Aperiodicity, Indecomposability, and the Approach to the Steady State

The Markov chain in above eample is periodic; from state 0 the particle must jump to either state
1 or 2; from state 1 and 2 it must next jump to state 0. Thus

{0}→ {1,2}→ {0}→ {1,2}→ L

The reason why there is no steady state in the intuitive sense, although a vector φ with φΠ = φ
exists is that the chain has a periodicity.

Definition

Suppose that the sets A1,A2,… ,Ar partition the

state space S. That is, suppose that

r
S = U Ai
i =1

disjointly. Assume that a transition from Ai to

Ai+1 must occur in one jump (where r+1 is taken to
be 1: The particle "wraps around" to A1 from Ar)-
 pictorially

A1 → A2 → L → Ar

That is, if at time t the particle is in one of

the states of Ai, then at time t+1 it must be in
one of the states of Ai+1. Then the Markov chain
is said to be periodic with period r. If the chain
is not periodic, it is said to be aperiodic.

Example
(a) The random walk infinite in both directions is periodic with period 2: Let A1=set
of even integers and A2=set of odd integers. Then
A1 → A2 → A1 → A2 → L
(b) The random walk with a boundary at 0 is periodic with period 2 if the reflecting
probability at 0 is r=1. If r<1, then the chain is aperiodic. This is the case since state
0 can be in one and only one partition set.
(c) The random walk with completely reflecting boundaries at 0 and N is periodic
with period 2 just as in part (a).
In looking for conditions implying existence of a steady state, we ought to exclude
periodic chains. Fortunately, there's an easy criterion to use to determine if a Markov
chain is aperiodic:

Suppose that

Π k ,k > 0

for any state k. Then the chain

is aperiodic.

This may not always apply: that is, there are chains with Π k ,k = 0 for all states k
which are periodic. But it is easy to spot entries Π k ,k along the main diagonal of Π
which are nonzero; if there are any, the chain must be aperiodic.

To see why the boxed result is true, suppose that A1,… ,Ar is a periodic decomposition
of the state space S; thus the Ai's are disjoint. But k ∈ Ai for some i: if Π k ,k > 0 ,
then k ∈ Ai + 1 , which contradicts the fact that Ai I Ai + 1 = 0/ .

Definition

A Markov chain is indecomposable if for

every pair i,j of states, either i → j or
j → i or both.

Example
 The random walk on S={0,1,2,… ,N} with absorbing barriers 0 and N is not
indecomposable since neither 0 → N nor N → 0 . If N is partially reflecting, the
cha
in is indecomposable since N → 0 , although 0 →/ N still.

Example

Is the chain in Fig. 13, (a) aperiodic? (b) indecomposable? (The arrows indicate
positive transition probabilities.)

Solution

(a) Note that i →/ 2 for all states i. Let us start at state 2 and see where it leads:

{2}→ {}
1 → {0,3}→ {0,3,4}→ {0,1,3,4}→ {0,1,3,4}→ L

where the next set is obtained by placing into it all the states that the states in the
previous set can jump to in one transition. Thus the chain is aperiodic.

0 . . 1

. . 2

3 4

Figure 13

(b) The chain is also indecomposable since for all pairs i,j either i → j or j → i
even though i →/ 2 for every i.

Suppose that ρ is an initial probability vector specifying the probabilities with

which the particle occupies the various states at time t=0. Then the probability vector
at time t is ρ t = ρΠ t , which quite clearly depends on the initial probability vector
ρ . On the other hand, the Markov assumption that the future is independent of the
past given the present suggests that after many jumps, the particle will "forget" where
it started at time 0. If a particle starts at one of the interior states in a random walk,
then after one jump, it is absolutely certain to have been at one of its neighboring
states at the previous time. After several jumps, however, we begin to lose track of
where it was originally. In the inventory model with which Section 12.1 began, the
 number of disks on the shelf at time 0 is crucial in knowing how many will be on the
shelf at the end of the first month. But after a few years go by, the number of disks
currently in stock depends less and less on the number originally stocked because of
the unpredictability in the number sold each month.

Given a transition matrix Π and an initial probability vector ρ , what will be the
steady state? What will be the limit of ρ t = ρΠ t as t → ∞ regardless of ρ ? Since
we have dwelt at length on the steady-state probability vector φ, it should come as
no surprise that the limit is φ. But we must exclude the types of complications that
can arise due to periodicity and decomposability. With these chains excluded, the
following theorem is extremely useful:

Limit Theorem

Suppose that Π is the transition matrix for an aperiodic and indecomposable Markov
chain. Assume that the vector equation
φΠ = φ
has a solution φ that is a probability vector with
0 < φi < 1 for all i ∈ S
N

∑ φ =1
i =0
i

Then the following hold:

1. φ is unique: it is the only probability vector satisfying φΠ = φ .
2. Regardless of the initial probability vector ρ ,
ρ t = ρΠ t → φ
as t → ∞ .
3. If the initial probability vector is φ, then for all t,
φ = φΠ = φ
t t

The theorem applies whether S is finite or not. It is very useful because solving for the steady-
state probability vector φ is often relatively straightforward (especially if one is willing to use a
computer), although perhaps tedious.

Note that property 2 can be phrased as

P ( X t = j | initial vector ρ )→ φj
as t → ∞ for all j ∈ S . And property 3 states P ( X t = j | initial vector φ)= φj
for all t and all j ∈ S .

Example
The general Two-State Process of Fig. 14
1 − α α 
Π=
 β 1− β

Case 1: α = 0 and β = 0 . The particle will not move; the chain is decomposable.
Case 2: α = 1 and β = 1 . The particle jumps back and forth; the chain is periodic
with period 2.
Case 3: Either 0 < α < 1 or 0 < β < 1 or both.
1− α α 
(φ0 , φ1 ) = (φ0 , φ1 )
 β 1− β

implies that
φ0 (1 − α )+ φ1 β = φ0
φ0α + φ1 (1 − β )= φ1
Both of these are the same equation:
αφ0 = βφ1
Using normalization,
 α
1 = φ0 + φ1 =  φ0
1 + β 
 
Therefore,
β α
φ0 = φ1 =
α+ β α+ β

1− α 1− β

. . α
β
0 1
Figure 14

The conclusion of the limit theorem is that regardless of the starting probabilities,
after many jumps the particle will be in states 0 and 1 with respective probabilities
φ0 and φ1 .
As a special case of the limit theorem, let us see the implications for the initial
probability vector
ρ = (0, K ,0,1,0, K ,0)
ith entry
That is, the particle starts in state i with probability 1. We have already seen in a
previous section that

ρ tj = Π it , j

for all states j. Therefore, if the particle starts in state i, the probability vector at time t
is the ith row of Π t . Hence the limit theorem states that the ith row of Π t
approaches the steady-state probability vector φ as t → ∞ .
 Corollary to the Limit Theorem

Under the same assumptions, the tth-order transition

matrix Π t has a limit matrix

lim Π t = M
t→ ∞

where M has constant rows each equal to the

steady-state probability vector φ.

Example

For the Markov chain of the example earlier, successive powers of the transition
matrix Π yield

 0 1 / 2 1 / 2 .1667 .6667 .1667

Π =  0 2 / 3 1 / 3 Π = .1111 .6667 .2222
2
   

1 / 3 2 / 3 0 
 
 0 .6111 .3889 

.0876 .6524 .2600 .0870 .6522 .2609
Π = .0872 .6523 .2605
4
Π = .0870 .6522 .2609
8
   

 .0861 .6519 .2620 
 
.0870 .6522 .2609


which agrees exactly to four significant digits with the steady-state probability vector
φ found in another example (that is, each row of Π t approaches φ).

Example 20
Consider this cyclic process - a special case , see Fig. 15
1 / 10 9 / 10 0 

Π= 0 3 / 4 1/ 4
 

5 / 7 0 2 / 7 

Using the notation of Example 9 yields
 9 1 5
p0 = p1 = p2 =
10 4 7
C = ( p0 + p1 + p2 )= 6.511
−1 −1 −1

1
φ0 = = .1706
Cp0
1
φ1 = = .6143
Cp1
1
φ2 = = .2150
Cp2

Computing powers successively implies

 .01 .765 .225 

Π = .1786 .5625 .2589
2 
 

. 2755 .6429 .0816 

.1712 .6133 .2154
Π = .1710 .6142 .2148
4
 

. 1692 . 6156 .2152 

.1706 .6143 .2150
Π 8 = .1706 .6143 .2150
 

.1706 .6143 . 2150 


The proof of the limit theorem would take us too far afield of our primary goal of
developing techniques for dealing with random models. Proofs can be found in the
books listed in the Bibliography.

A final note about the steady-state probability vector φ: In the limit theorem we
assumed that φ exists with φΠ = φ. Recall that in Example 11 no such φ exists
because the particle "dribbles" off to infinity without ever returning. For finitely
many states this cannot happen.

1/10 3/4

9/10
0 . . 1

5/7 1/4

. 2/7

2
 Figure 15

Existence Theorem

An aperiodic and indecomposable Markov chain with

finite state space has one and only one steady-state
probability vector.

If there is a steady-state probability vector, the limit theorem implies that there is only one. The
existence theorem guarantees one. The proof can be found in books listed in the Bibliography.