Professional Documents
Culture Documents
Markov Chains: Scott Sheffield
Markov Chains: Scott Sheffield
440: Lecture 33
Markov Chains
Scott Sheffield
MIT
1
18.440 Lecture 33
Outline
Markov chains
Examples
2
18.440 Lecture 33
Outline
Markov chains
Examples
3
18.440 Lecture 33
Markov chains
4
18.440 Lecture 33
Simple example
5
18.440 Lecture 33
Matrix representation
6
18.440 Lecture 33
Transitions via matrices
I Suppose that pi is the probability that system is in state i at
time zero.
I What does the following product represent?
P00 P01 . . . P0M
P10 P11 . . . P1M
·
p0 p1 . . . pM
·
·
PM0 PM1 . . . PMM
I Answer: the probability distribution at time one.
I How about the following product?
p0 p1 . . . pM An
(n)
I We write Pij for the probability to go from state i to state j
over n steps.
I From the matrix point of view
(n) (n) (n)
n
P00 P01 . . . P0M P00 P01 . . . P0M
(n) (n) (n)
P P11 . . . P1M
P10 P11 . . . P1M 10
· ·
=
· ·
·
·
(n) (n) (n) PM0 PM1 . . . PMM
P M0P M1... P MM
8
18.440 Lecture 33
Questions
9
18.440 Lecture 33
Outline
Markov chains
Examples
10
18.440 Lecture 33
Outline
Markov chains
Examples
11
18.440 Lecture 33
Simple example
I Note that
2 .64 .35
A =
.26 .74
.285719 .714281
I Can compute A10 =
.285713 .714287
12
18.440 Lecture 33
Does relationship status have the Markov property?
In a relationship
Married Engaged
Markov chains
Examples
14
18.440 Lecture 33
Outline
Markov chains
Examples
15
18.440 Lecture 33
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transition
matrix has all non-zero entries.
I Turns out that if chain has this property, then
(n)
πj := limn→∞ Pij exists and the πj are the unique
PM
non-negative solutions of πj = k=0 πk Pkj that sum to one.
I This means that the row vector
π = π0 π1 . . . πM
I Recall
that
10 .285719 .714281 2/7 5/7 π
A = ≈ =
.285713 .714287 2/7 5/7 π
17
18.440 Lecture 33
MIT OpenCourseWare
http://ocw.mit.edu
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.