L18

Recap
9 / 19
Recap
▶ Stochastic process {X (t), t ∈ T } is a collection of random

variables defined such that for every t ∈ T we have a random
variable X (t) taking values in state space S.
9 / 19
Recap

▶ A stochastic process that satisfies the Markov property is a

Markov chain.
9 / 19
Recap


Markov chain.
▶ P(Xn = j|Xn1 = x1 , .., Xnk = i) = P(Xn = j|Xnk = i)
9 / 19
Recap


Markov chain.
▶ P(Xn = j|Xn1 = x1 , .., Xnk = i) = P(Xn = j|Xnk = i)
▶ Markov chain is a stochastic process where the next state of

the process depends on the present state butnot on previous
states.
9 / 19
Running example: Coin with memory!
10 / 19
▶ In a Markovian coin with memory, the outcome of the next
toss depends on the current toss.
10 / 19
▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.
10 / 19
▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

P(Xn+1 = −1|Xn = −1) = 0.8 for all n.
10 / 19

P(Xn+1 = −1|Xn = −1) = 0.8 for all n.
▶ Flippy Coin: P(Xn+1 = 1|Xn = 1) = 0.1 while

P(Xn+1 = −1|Xn = −1) = 0.3 for all n.
10 / 19

P(Xn+1 = −1|Xn = −1) = 0.8 for all n.

P(Xn+1 = −1|Xn = −1) = 0.3 for all n.
▶ This can be represented by a transition diagram (see board)
10 / 19

P(Xn+1 = −1|Xn = −1) = 0.8 for all n.

P(Xn+1 = −1|Xn = −1) = 0.3 for all n.
▶ The transition
" probability
# matrix
" P for
# the two cases is
0.9 .1 0.1 0.9
Ps = and Pf =
0.2 0.8 0.7 0.3
10 / 19

P(Xn+1 = −1|Xn = −1) = 0.8 for all n.

P(Xn+1 = −1|Xn = −1) = 0.3 for all n.
▶ The transition
" probability
# matrix
" P for
# the two cases is
0.9 .1 0.1 0.9
Ps = and Pf =
0.2 0.8 0.7 0.3
▶ The row corresponds to present state and the column

corresponds to next state. 10 / 19
Running example: Dice with memory!
11 / 19
▶ In a markovian dice with memory, the outcome of the next roll
depends on the current roll.
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example
 transition probability matrix

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9
▶ State transition diagram on board
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9

Pn Sn
▶ Consider Sn = i=1 Xi and µ̂n = n .
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9

Pn Sn
▶ Consider Sn = i=1 Xi and µ̂n = n . What is limn→∞ µ̂n ?
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9

Pn Sn
▶ Cannot invoke SLLN as {Xi } are not i.i.d.
11 / 19
▶ Xn = i for i ∈ S where S = {1, . . . , 6}.
▶ Example

0.9 .1 0 0 0 0
0 .9 .1 0 0 0 
 
0 0 0.9 0.1 0 0 
P= 
0 0 0 0.9 0.1 0 
0 0 0 0 0.9 0.1
0.1 0 0 0 0 0.9

Pn Sn
▶ Cannot invoke SLLN as {Xi } are not i.i.d.
▶ We will see later SLLN for Markov chains!
11 / 19
Finite dimensional distributions
12 / 19
▶ Consider a Markov dice with transition probability P.
12 / 19

▶ What is P(X0 = 4, X1 = 5, X2 = 6)?
12 / 19

▶ What is P(X0 = 4, X1 = 5, X2 = 6)?
▶ = P(X2 = 6|X1 = 5, X0 = 4)P(X1 = 5|X0 = 4)P(X0 = 4)
12 / 19

▶ What is P(X0 = 4, X1 = 5, X2 = 6)?
▶ = P(X2 = 6|X1 = 5, X0 = 4)P(X1 = 5|X0 = 4)P(X0 = 4)
▶ = p65 p54 P(X0 = 4).
12 / 19

▶ What is P(X0 = 4, X1 = 5, X2 = 6)?
▶ = P(X2 = 6|X1 = 5, X0 = 4)P(X1 = 5|X0 = 4)P(X0 = 4)
▶ = p65 p54 P(X0 = 4).
▶ What is P(X0 = 4)?
12 / 19

▶ What is P(X0 = 4, X1 = 5, X2 = 6)?
▶ = P(X2 = 6|X1 = 5, X0 = 4)P(X1 = 5|X0 = 4)P(X0 = 4)
▶ = p65 p54 P(X0 = 4).
▶ What is P(X0 = 4)?
▶ This probability of starting in a particular state is called initial
distribution of the markov chain.
12 / 19
13 / 19
▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.
13 / 19

▶ We assume M states and X0 denotes the initial state.
13 / 19

▶ You can start in any starting state or may pick your starting
state randomly.
13 / 19

state randomly.
▶ Let µ̄ = (µ1 , . . . , µM ) denote the initial distribution, i.e.,
P(X0 = x0 ) = µx0 .
13 / 19

state randomly.
P(X0 = x0 ) = µx0 .
▶ How does one obtain the finite dimensional distribution
P(X0 = x0 , X1 = x1 , , X2 = x2 ) ?
13 / 19

state randomly.
P(X0 = x0 ) = µx0 .
P(X0 = x0 , X1 = x1 , , X2 = x2 ) ?
▶ P(X0 = x0 , X1 = x1 , , X2 = x2 ) = px1 ,x2 px0 ,x1 µx0 .
13 / 19

state randomly.
P(X0 = x0 ) = µx0 .
P(X0 = x0 , X1 = x1 , , X2 = x2 ) ?
▶ P(X0 = x0 , X1 = x1 , , X2 = x2 ) = px1 ,x2 px0 ,x1 µx0 .
▶ In general,
P(X0 = x0 , X1 = x1 , . . . Xk = xk ) = pxk−1 ,xk × . . . × px0 ,x1 µx0
13 / 19
Chapman Kolmogorov Equations for DTMC
14 / 19
▶ Consider
" a Markov coin# and its transition probability matrix
p1,1 p1,−1
P= .
p−1,1 p−1,−1
14 / 19
▶ Consider
p1,1 p1,−1
P= .
p−1,1 p−1,−1
▶ Given X0 = 1, what is P(X2 = 1)?

P(X2 = 1|X0 = 1) = P(X2 = 1|X1 = 1, X0 = 1)P(X1 = 1|X0 = 1)
+ P(X2 = 1|X1 = −1, X0 = 1)P(X1 = −1|X0 = 1)
2
= p1,1 + p−1,1 p1,−1
14 / 19
▶ Consider
p1,1 p1,−1
P= .
p−1,1 p−1,−1

P(X2 = 1|X0 = 1) = P(X2 = 1|X1 = 1, X0 = 1)P(X1 = 1|X0 = 1)
+ P(X2 = 1|X1 = −1, X0 = 1)P(X1 = −1|X0 = 1)
2
= p1,1 + p−1,1 p1,−1
▶ Here the first inequality follow from the fact that
P(C |A) = P(C |BA)P(B|A) + P(C |B c A)P(B c |A) HW: Verify
14 / 19
▶ Consider
p1,1 p1,−1
P= .
p−1,1 p−1,−1

P(X2 = 1|X0 = 1) = P(X2 = 1|X1 = 1, X0 = 1)P(X1 = 1|X0 = 1)
+ P(X2 = 1|X1 = −1, X0 = 1)P(X1 = −1|X0 = 1)
2
= p1,1 + p−1,1 p1,−1

▶ Similarly, P(X2 = −1|X0 = 1), P(X2 = 1|X0 = −1), P(X2 =
−1|X0 = −1) can be obtained
14 / 19
▶ Consider
p1,1 p1,−1
P= .
p−1,1 p−1,−1

P(X2 = 1|X0 = 1) = P(X2 = 1|X1 = 1, X0 = 1)P(X1 = 1|X0 = 1)
+ P(X2 = 1|X1 = −1, X0 = 1)P(X1 = −1|X0 = 1)
2
= p1,1 + p−1,1 p1,−1

▶ Similarly, P(X2 = −1|X0 = 1), P(X2 = 1|X0 = −1), P(X2 =
−1|X0 = −1) can be obtained and these are elements of a
two-step transition matrix P (2) .
14 / 19
15 / 19
▶ The two" step transition probability matrix P (2) is given by#

p 2 +p
(2) 1,1 1,−1 p−1,1 p1,1 p1,−1 + p1,−1 p−1,−1
P = 2 .
p−1,1 p1,1 + p−1,−1 p−1,1 p−1,1 p1,−1 + p−1,−1
15 / 19

p 2 +p
(2) 1,1 1,−1 p−1,1 p1,1 p1,−1 + p1,−1 p−1,−1
P = 2 .
p−1,1 p1,1 + p−1,−1 p−1,1 p−1,1 p1,−1 + p−1,−1
▶ This implies that P (2) = P × P = P 2 .
15 / 19

p 2 +p
(2) 1,1 1,−1 p−1,1 p1,1 p1,−1 + p1,−1 p−1,−1
P = 2 .
p−1,1 p1,1 + p−1,−1 p−1,1 p−1,1 p1,−1 + p−1,−1
▶ In general, P (n) = P n .
15 / 19

p 2 +p
(2) 1,1 1,−1 p−1,1 p1,1 p1,−1 + p1,−1 p−1,−1
P = 2 .
p−1,1 p1,1 + p−1,−1 p−1,1 p−1,1 p1,−1 + p−1,−1
▶ Chapman-Kolmogorov equations are a further generalization

of this.
P (n+l) = P (n) P (l)
15 / 19

p 2 +p
(2) 1,1 1,−1 p−1,1 p1,1 p1,−1 + p1,−1 p−1,−1
P = 2 .
p−1,1 p1,1 + p−1,−1 p−1,1 p−1,1 p1,−1 + p−1,−1
▶ Chapman-Kolmogorov equations are a further generalization

of this.
P (n+l) = P (n) P (l)
▶ We wont see the proof of this.
15 / 19
Classification of states
16 / 19
▶ Consider a Markov process with state space S
16 / 19
▶ We say that j is accessible from i if pijn > 0 for some n.
16 / 19
▶ This is denoted by i → j.
16 / 19
▶ if i → j and j → i then we say that i and j communicate.

This is denoted by i ↔ j.
16 / 19

A chain is said to be irreducible if i ↔ j for all i, j ∈ S.
16 / 19

A chain is said to be irreducible if i ↔ j for all i, j ∈ S.
▶ Are the examples of Markovian coin and dice we have

considered till now irreducible? check!
16 / 19
Recurrent and Transient states
17 / 19
▶ We say that a state i is recurrent if
Fii = P( ever returning to i having started in i ) = 1.
17 / 19
▶ Fii is not easy to calculate. (Not part of this course)
17 / 19
▶ If a state is not recurrent, it is transient.
17 / 19
▶ For a transient state i, Fii < 1.
17 / 19
▶ For a transient state i, Fii < 1.
▶ If i ↔ j and i is recurrent, then j is recurrent.
17 / 19
Limiting probabilities
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
b a

▶ P = 1−a a
limn→∞ P n = a+b
b
a+b
a
b 1−b a+b a+b
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
b a

▶ P = 1−a a
b
a+b
a
b 1−b a+b a+b
▶ What is the interpretation of limn→∞ pij(n) = [limn→∞ P n ]ij ?
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
b a

▶ P = 1−a a
b
a+b
a
b 1−b a+b a+b
▶ πj = limn→∞ pij(n) denotes the probability of being in state j at

time n when starting in state i.
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
b a

▶ P = 1−a a
b
a+b
a
b 1−b a+b a+b

▶ For an M state DTMC, π = (π1 , . . . , πM ) denotes the limiting

or stationary distribution.
18 / 19
" # " # " #
0 0 1 .06 .3 .64 .23 .385 .385
▶ P= 0 0.6 0.4 P 5 = .18 .38 .44 P 30 = .23 .385 .385
0.6 0.4 0 .38 .44 .18 .23 .385 .385
b a

▶ P = 1−a a
b
a+b
a
b 1−b a+b a+b

▶ For an M state DTMC, π = (π1 , . . . , πM ) denotes the limiting

or stationary distribution.
▶ How do we obtain the stationary distribution π?
18 / 19
Stationary distribution
19 / 19
The stationary distribution can be obtained as a solution

to the equation π = πP.
19 / 19

▶ Obtain stationary distribution

 for the Markov
 Chain with
0 0 1
 
transition probability P =  0 0.6 0.4
0.6 0.4 0
19 / 19


 for the Markov
 Chain with
0 0 1
 
0.6 0.4 0
▶ The limiting distribution need not exist for some Markov

chains, but the stationary distribution exists.
19 / 19


 for the Markov
 Chain with
0 0 1
 
0.6 0.4 0

chains,
" but#the stationary distribution exists. For example for
0 1
P= .
1 0
19 / 19


 for the Markov
 Chain with
0 0 1
 
0.6 0.4 0

chains,
0 1
P= .
1 0
▶ How to tackle such cases?
19 / 19


 for the Markov
 Chain with
0 0 1
 
0.6 0.4 0

chains,
0 1
P= .
1 0
▶ How to tackle such cases? We will see it (among other thing)

in CS3.307.
19 / 19

L18

Uploaded by

Copyright:

Available Formats

You might also like

L18

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

L18

Uploaded by

Copyright:

Available Formats

Recap

▶ Stochastic process {X (t), t ∈ T } is a collection of random

▶ Stochastic process {X (t), t ∈ T } is a collection of random

▶ A stochastic process that satisfies the Markov property is a

▶ Stochastic process {X (t), t ∈ T } is a collection of random

▶ A stochastic process that satisfies the Markov property is a

▶ P(Xn = j|Xn1 = x1 , .., Xnk = i) = P(Xn = j|Xnk = i)

▶ Stochastic process {X (t), t ∈ T } is a collection of random

▶ A stochastic process that satisfies the Markov property is a

▶ P(Xn = j|Xn1 = x1 , .., Xnk = i) = P(Xn = j|Xnk = i)

▶ Markov chain is a stochastic process where the next state of

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

▶ Flippy Coin: P(Xn+1 = 1|Xn = 1) = 0.1 while

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

▶ Flippy Coin: P(Xn+1 = 1|Xn = 1) = 0.1 while

▶ This can be represented by a transition diagram (see board)

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

▶ Flippy Coin: P(Xn+1 = 1|Xn = 1) = 0.1 while

▶ This can be represented by a transition diagram (see board)

▶ Xn = 1 for heads and Xn = −1 otherwise. S = {+1, −1}.

▶ Sticky coin : P(Xn+1 = 1|Xn = 1) = 0.9 and

▶ Flippy Coin: P(Xn+1 = 1|Xn = 1) = 0.1 while

▶ This can be represented by a transition diagram (see board)

▶ The row corresponds to present state and the column

▶ State transition diagram on board

▶ State transition diagram on board

▶ State transition diagram on board

▶ State transition diagram on board

▶ State transition diagram on board

▶ Consider a Markov dice with transition probability P.

▶ Consider a Markov dice with transition probability P.

▶ Consider a Markov dice with transition probability P.

▶ Consider a Markov dice with transition probability P.

▶ Consider a Markov dice with transition probability P.

▶ Consider a Markov dice with transition probability P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Consider a DTMC {Xn , n ≥ 0} with transition matrix P.

▶ Given X0 = 1, what is P(X2 = 1)?

▶ Given X0 = 1, what is P(X2 = 1)?

▶ Here the first inequality follow from the fact that

P(C |A) = P(C |BA)P(B|A) + P(C |B c A)P(B c |A) HW: Verify

▶ Given X0 = 1, what is P(X2 = 1)?

▶ Here the first inequality follow from the fact that

P(C |A) = P(C |BA)P(B|A) + P(C |B c A)P(B c |A) HW: Verify

▶ Given X0 = 1, what is P(X2 = 1)?

▶ Here the first inequality follow from the fact that

P(C |A) = P(C |BA)P(B|A) + P(C |B c A)P(B c |A) HW: Verify