Solutions to Durrett’s Probability: Theory and Examples

Lumiao

1 Martingales
1.1 Martingales, Almost Sure Convergence
Problem 1.1.1 Xn is a martingale w.r.t. Gn and let Fn = σ(X1 , · · · , Xn ). Then Gn ⊇ Fn and Xn is a
martingale w.r.t. Fn .

Proof Note that σ(Xm ) ⊆ Gm ⊆ Gn if m ≤ n. Thus:

n
[
Fn = σ(X1 , · · · , Xn ) = σ( σ(Xk )) ⊆ Gn
k=1

Obviously Xn ∈ Fn and is integrable. Also, using the property of conditional expectation:

E[Xn+1 |Fn ] = E[E[Xn+1 |Gn ]|Fn ] = E[Xn |Fn ] = Xn , ∀n ≥ 1

Thus (Xn )n≥1 is a martingale w.r.t Fn . 

Problem 1.1.2 Give an example of a submartingale Xn so that Xn2 is a supermartingale. Hint: Xn does not
have to be random.

Proof Let Xn = − n1 . 

Problem 1.1.3 Generalize (i) of Theorem 4.2.7 by showing that if Xn and Yn are submartingales w.r.t. Fn
then Xn ∨ Yn is also.

Proof Obviously Xn ∨ Yn ∈ Fn and is integrable. By the definition of submartingale:

E[Xn+1 ∨ Yn+1 |Fn ] ≥ E[Xn+1 |Fn ] ≥ Xn

E[Xn+1 ∨ Yn+1 |Fn ] ≥ E[Yn+1 |Fn ] ≥ Yn

Thus E[Xn+1 ∨ Yn+1 |Fn ] ≥ Xn ∨ Yn , which proves the result. 

Problem 1.1.4 Let Xn , n ≥ 0, be a submartingale with sup Xn < ∞. Let ξn = Xn − Xn−1 and suppose
E[sup ξn+ ] < +∞. Show that Xn converges a.s.

Proof Consider the stopping time NM = inf{n ≥ 0 : Xn ≥ M } for M ∈ N. XNM ∧n is a submartingale. Notice
+
that XNM ∧n
≤ M + sup ξn+ , we have:
+
E[XN M ∧n
] ≤ M + E[sup ξn+ ] < +∞

By Theorem 4.2.11, XNM ∧n converges

S∞to a limit a.s. Thus for each M , Xn converges a.s. to a limit on
{NM = ∞}. Since sup Xn < +∞, Ω = M =1 {NM = ∞}, which means that Xn converges almost surely on the
whole Ω. 

Problem 1.1.5 Give an example of a martingale Xn with Xn → −∞ a.s. Hint: Let Xn = ξ1 + · · · + ξn , where
the ξi are independent (but not identically distributed) with E[ξn ] = 0.

1
 1
Proof Following the hint, let Xn = ξ1 + · · · + ξn , where the ξi are independent. Each ξk satisfies P(ξk = −1) = 2k
1 1
P+∞ P+∞ 1
and P(ξk = 2k−1 ) = 1 − 2k . Then E[ξn ] = 0. Since ξk are independent and k=1 P(ξk = −1) = k=1 2k = +∞,
we have P(ξn = −1, i.o.) = 1, which means that Xn → −∞ a.s. 

Problem 1.1.6 Let YQ 1 , Y2 , · · · be nonnegative i.i.d. random variables with E[Ym ] = 1 and P(Ym = 1) < 1. By
Example 4.2.3 Xn = m≤n Ym defines a martingale.
(i) Use Theorem 4.2.12 and an argument by contradiction to show Xn → 0 a.s.
(ii) Use the strong law of large numbers to conclude n1 logXn → c < 0.
P
Proof (i) The convergence is obviously since Xn ≥ 0. To prove the limit is 0, we only need to prove that Xn −
→ 0.
We have the following observations:
1. Since we have proved Xn converges almost everywhere, Xn also converges in probability, i.e. P(|Xn −Xm | >
) → 0 as m, n → 0;
2. Since P(Ym = 1) ≤ 1, there exists 1 > 0 and δ > 0 such that P(|Ym − 1| > 1 ) > δ;
3. For any 2 > 0, take  = 1 · 2 and m = n + 1 in the first observation. Since |Xn+1 − Xn | = |Xn | · |Yn+1 − 1|,
we have:

P(|Xn − Xm | > 1 · 2 ) ≥ P(|Xn | > 2 , |Yn+1 − 1| > 1 ) = P(|Xn | > 2 ) · P(|Yn+1 − 1| > 1 ) ≥ δP(|Xn | > 2 )

This observations gives that XPnnconverges to 0 in probability

(ii) Notice that n1 logXn = n1 k=1 logYm . We need to discuss the existence of the expectation of log(Ym ).
Case 1 If E[|logYm |] < +∞, then we can use the Jensen’s Inequality:

0 = −logE[Ym ] < E[−logYm ]

The equality could not hold since P(Ym = 1) < 1. Now by the strong law of large number, we have:
n
1X a.s. 1 a.s.
−logYm −−→ E[−logYm ] > 0 ⇒ logXn −−→ c < 0
n n
k=1

Case 2 If E[|logYm |] = +∞. We need to check out only one of the expectation of (−logYm )+ and (−logYm )− is
infinity. In fact:

E[(−logYm )− ] = E[logYm 1Ym ≥1 ] ≤ E[(Ym − 1)1{Ym ≥1} ] ≤ 1 − P(Ym ≥ 1) < +∞

Thus only E[(−logYm )+ ] = +∞. By Theorem 2.4.5, we have:

n
1X a.s. 1 a.s.
−logYm −−→ +∞ ⇒ logXn −−→ −∞ < 0
n n
k=1

P+∞ 1.1.7 Let Xn and Yn be positive integrable and adapted to Fn . Suppose E[Xn+1 |Fn ] ≤ (1 + Yn )Xn
Problem
with n=1 Yn < +∞ a.s. Prove that Xn converges a.s. to a nite limit by nding a closely related supermartingale
to which Theorem 4.2.12 can be applied.

Proof Following the hint, define:

Xn
Zn = Qn−1 , n≥1
m=1 (1 + Ym )
Obviously Zn ∈ Fn and is integrable. Also:
n
!−1 n−1
!−1
Y Y
E[Zn+1 |Fn ] = (1 + Ym ) · E[Xn+1 |Fn ] ≤ (1 + Ym ) · Xn = Zn , ∀n ≥ 1
m=1 m=1

2
Thus Zn is a supermartingale which is positive.PBy Theorem 4.2.12, Zn converges
Q∞ to a limit which is finite
+∞
almost everywhere. Denote it by Z∞ . Also, by n=1 Yn < +∞ a.s., we know m=1 (1 + Ym ) converges almost
everywhere. As a result:
n−1 ∞
a.s.
Y Y
Xn = Zn · (1 + Ym ) −−→ Z∞ (1 + Ym )
m=1 m=1

where the limit is finite almost everywhere. 

Problem 1.1.8 The switching principle Suppose Xn1 and Xn2 are supermartingales with respect to Fn , and
1 2
N is a stopping time so that XN ≥ XN . Then:

Yn = Xn1 1{N >n} + Xn2 1{N ≤n} is a supermartingale

Zn = Xn1 1{N ≥n} + Xn2 1{N <n} is a supermartingale.

Proof Obviously Yn and Zn are in Fn and is integrable. We have two opinions to make inequalities, one is the
1 2
property of supermartingale and the other is the condition XN ≥ XN . Combine them properly. For Yn , we have:
1 2
E[Yn+1 |Fn ] = E[Xn+1 1{N >n+1} + Xn+1 1{N ≤n+1} |Fn ]
1 2
≤ E[Xn+1 1{N >n} + Xn+1 1{N ≤n} |Fn ]
1 2
({N > n}, {N ≤ n} ∈ Fn ) ≤ 1{N >n} E[Xn+1 |Fn ] + 1{N ≤n} E[Xn+1 |Fn ]
≤ 1{N >n} Xn1 + 1{N ≤n} Xn2
= Yn

Similarly, for Zn , we have:

1 2
E[Zn+1 |Fn ] = E[Xn+1 1{N ≥n+1} + Xn+1 1{N <n+1} |Fn ]
1 2
({N ≥ n + 1}, {N < n + 1} ∈ Fn ) ≤ 1{N ≥n+1} E[Xn+1 |Fn ] + 1{N <n+1} E[Xn+1 |Fn ]
≤ 1{N ≥n+1} Xn1 + 1{N <n+1} Xn2
≤ 1{N ≥n} Xn1 + 1{N <n} Xn2
= Zn

These inequalities show that both Yn and Zn are supermartingales! 

1.2 Examples
Problem 1.2.1 Let Xn and Yn be positive integrable and adapted to Fn . Suppose E[Xn+1 |Fn ] ≤ Xn + Yn , with
P∞ Pk
k=1 Yn < ∞ a.s. Prove that Xn converges a.s. to a finite limit. Hint: Let N = inf k { m=1 Ym > M }, and stop
your supermartingale at time N .

Proof The way to prove isPsimilar to that of Problem 1.1.4, but here it is a little subtle in the index of
n−1
summation. Let Zn = Xn − k=1 , n ≥ 1. Obviously Zn ∈ Fn and Zn is integrable. Also, by the assumption, we
have:
n
X n−1
X
E[Zn+1 |Fn ] = E[Xn+1 |Fn ] − Yk ≤ Xn − Yk = Zn , ∀n ≥ 1
k=1 k=1
Pk
Thus Zn is a supermartingale. To bound the supermartingale, let NM = inf k { m=1 Ym > M } and consider the
supermartingale Zn∧NM . We will find that:
n∧N
X M −1

Zn∧NM = Xn∧NM − Yk ≥ −M
k=1

3
Here the index of summation is very important! Thus we can apply Theorem S+∞4.2.12 to Zn∧NM + M , which
yields that Zn converges to a finite limit a.s. on {NM = ∞}. Notice that M =1 {NM = ∞} = Ω a.s. since
P∞ Pn−1
k=1 Yn < ∞ a.s. So Zn converges to a finite limit a.s. on Ω. Finally, Xn = Zn + k=1 Ym also converges to a
finite limit a.s. 
P∞ n−1
Problem 1.2.2 Show n=2 P(An | ∩m=1 Acm ) = ∞ implies P(∩∞ c
m=1 Am ) = 0

Proof 1 You may use Exercise 4.3.4 if you notice that:

n
Y
1 − P(An | ∩m−1 c n c


k=1 A ) = P(∩m=1 Am )
m=1

Proof 2 You can also use Borel-Cantelli II. Let Fn = σ({A1 , · · · , An }). Notice that:
n
\
P(An+1 | ∩nm=1 Acm ) = P(An+1 |Fn ) on Acm
m=1
T∞ c
Tn c
Since for each n ≥ 1, m=1 Am ⊆ m=1 Am , we have:
∞
\
P(An+1 | ∩nm=1 Acm ) = P(An+1 |Fn ) on Acm ∀n ≥ 1
m=1

This and the assumption of the problem give:

+∞
X ∞
\
P(An |Fn−1 ) = +∞ on Acm
n=2 m=1

By using Borel-Cantelli II, we have:

∞
\
Acm ⊆ {An i.o.} P − a.s.
m=1

This happens only when P ∩∞

m=1 Acm ) =0 
Zn
Problem 1.2.3 Show that if P(lim µn = 0) < 1, then it is = ρ and hence:
Zn
{lim > 0} = {Zn > 0 f or all n} a.s.
µn
Proof Generally, Zn > 0 does not indicate that the limit of Zn /µn is not zero. This is why this problem
meaningful! Notice that if µ ≤ 1, P(Zn = 0, ∃n ≥ 1) = 1 and then P(lim Z
µn = 0) = 1 which is contradictory to
n

the assumption. So µ > 1 and P(Zn > 0, ∀n ≥ 1) = 1 − ρ. Since we have the relationship:
Zn
{lim > 0} ⊆ {Zn > 0 f or all n}
µn
it is sufficient to prove P(lim Z
µn > 0) = 1 − ρ. Given Z1 = k, denote the number of child corresponding to each
n

subtree by Zni , 1 ≤ i ≤ k. Then Zn1 , · · · , Znk are independent and have the same distribution as Zn . So:
+∞
Zn X Zn
P(lim n
= 0) = P(lim n = 0|Z1 = k)P(Z1 = k)
µ µ
k=1
+∞ k i
X X Zn−1
= P(lim = 0) · pk
i=1
µn
k=0
+∞ k i
X \ Zn−1
= P( {lim = 0}) · pk
i=1
µn
k=0
+∞
X Zn
i.i.d = P(lim = 0)pk
µn
k=0

4
This means that P(lim Z Zn
µn = 0) is the solution of x = ϕ(x) in [0, 1), which is just ρ. Thus P(lim µn > 0) = 1 − ρ
n

and the problem is solved. 

Problem 1.2.4PLet Zn be a branching process with offspring distribution pk , defined in part (d) of Section 4.3,
∞
and let ϕ(θ) = k=1 pk θk . Suppose ρ < 1 has ϕ(ρ) = ρ. Show that ρZn is a martingale and use this to conclude
P(Zn = 0 f or some n ≥ 1|Z0 = x) = ρx .

Proof Obviously ρZn ∈ Fn . |ρZn | < 1 shows its integrability. Consider on the set {Zn = k}, we have:
n+1 n+1
E[ρZn+1 |Fn ] =E[ρξ1 +···+ξk
|Fn ]
ξ1n+1 +···+ξk
n+1
=E[ρ ]
 n+1
k
= E[ρξ1 ]
=ϕ(ρ)k = ρZn

Here we have used the definition of ϕ(θ) and ρ. Thus E[ρZn+1 |Fn ] = ρZn and ρZn is a martingale. For the second
part of the problem, we will defer it until the proof of Problem 1.4.2. 

1.3 Doob’s Inequality, Convergence in Lp , p > 1

Problem 1.3.1 Show that if j ≤ k then E[Xj ; N = j] ≤ E[Xk ; N = j] and sum over j to get a second proof of
EXN ≤ EXk .

Proof Since {N = j} ∈ Fj , by the definition of submartingale we have:

E[Xk ; N = j] = E[E[Xk |Fj ]; N = j] ≥ E[Xj ; N = j]

On {N = j}, Xj = XN , so sum over j we get EXN ≤ EXk . 

Problem 1.3.2 Generalize the proof of Theorem 4.4.1 to show that if Xn is a submartingale and M ≤ N are
stopping times with P(N ≤ k) = 1, then EXM ≤ EXN .

Proof Imitate the proof in the textbook, let Kn = 1{M <n≤N } . Since {M < n ≤ N } = {M ≤ n − 1} ∩ {N ≤
n − 1}c ∈ Fn−1 , Kn is predictable w.r.t Fn . Consider the stochastic integration:

(K · X)n = XN ∧n − XM ∧n , n ≥ 0

It is still a submartingale and therefore E[(K · X)0 ] ≤ E[(K · X)k ]. This is just E[XM ] ≤ E[XN ] 

Problem 1.3.3 Suppose M ≤ N are stopping times. If A ∈ FM then L = M · 1A + N · 1Ac is a stopping time.

Proof It is sufficient to show that {L ≤ n} ∈ Fn for all n ≥ 0. Notice that we have the decomposition:

{L ≤ n} = ({L ≤ n} ∩ A) ∪ ({L ≤ n} ∩ Ac ) = ({M ≤ n} ∩ A) ∪ ({N ≤ n} ∩ {M ≤ n} ∩ Ac ) ∈ Fn

Here we have used the fact that both {M ≤ n} ∩ A and {M ≤ n} ∩ Ac are in Fn . 

Problem 1.3.4 Use the stopping times from the previous exercise to strengthen the conclusion of Problem 1.3.2
to XM ≤ E[XN |FM ].

Proof By the definition of conditional distribution, we have to show that E[XM 1A ] ≤ E[XN 1A ] for ay A ∈ FM .
Now for each A ∈ FM , define the stopping time L as in the previous problem. Obviously L ≤ N ≤ k and we can
use the result of Problem 1.3.2:

E[XM 1A ] + E[XN 1Ac ] = E[XL ] ≤ E[XN ] = E[XN 1A ] + E[XN 1Ac ]

This indicates E[XM 1A ] ≤ E[XN 1A ] 

5
Problem 1.3.5 Prove the following variant of the conditional variance formula. If F ⊆ G, then:
2 2 2
E (E[Y |G] − E[Y |F]) = E (E[Y |G]) − E (E[Y |F])

Proof The formula proves useful if you take G and F from a filtration Fn which gives an estimation of kYm −Yn kL2 .
Here Yn denote the martingale E[Y |Fn ]. The formula itself is easy to prove once you notice that:
2
E[E[Y |G]E[Y |F]] = E[E[E[Y |G]E[Y |F]|F]] = E[E[Y |F]E[E[Y |G]|F]] = E (E[Y |F])

Take this back to the left hand side and the result follows. 

Problem 1.3.6 If we let Sn = ξ1 + P · · · + ξn where the ξm are independent and have Eξm = 0, |ξm | ≤ K,
2 2 n
σm = Eξm < ∞, then Sn2 − s2n = Sn2 − m=1 σm 2
is a martingale. Use this and Theorem 4.4.1 to conclude:

(x + K)2
 
P max |Sm | ≤ x ≤
1≤m≤n V arSn

Proof The way of proving is similar to that used in the section of optional stopping time. To get an accurate
estimation of probability, you may set a stopping time to get another martingale (sub\supermartingale). Use
the equality (inequality) of expectation and write it out at the stoping time. You might need the property of
uniform integrability or the theorems of convergence to remove the changing index n. Here we do not. Let
N = inf{n ≥ 0 : |Sn | > x}, A be the event that {max1≤m≤ |Sm | ≤ x}. Sn2 − s2n is a martingale and so is
2
Sn∧N − s2n∧N . Use this we have:
2
0 = E[S0∧N − s20∧N ] = E[Sn∧N
2
− s2n∧N ] = E[Sn∧N
2
] − E[s2n∧N ]

To split thing out, we have several observations. Ac implies {N ≤ n}. On A, we have |Sn∧N | = |Sn | ≤ x and on
Ac we have |Sn∧N | = |SN | ≤ K + x. Combine these together, we have:
2 2 2
E[Sn∧N ] = E[Sn∧N 1A ] + E[Sn∧N 1Ac ] ≤ x2 P(A) + (K + x)2 (1 − P(A))

For the other term, we have:

E[s2n∧N ] ≥ E[s2n∧N 1A ] = V ar[Sn2 ]P(A)
These two gives:
(K + x)2 ≥ K 2 + 2Kx + V ar[Sn2 ] P(A) ≥ V ar[Sn2 ]P(A)

This proves the result! 

Problem 1.3.7 The next result gives an extension of Theorem 4.4.2 to p = 1. Let Xn be a martingale with
X0 = 0 and EXn2 < ∞. Show that:
EXn2
 
P max Xm > λ ≤
1≤m≤n EXn2 + λ2

Proof Following the hint, for any c > −λ, consider the submartingale (Xm + c)2 . Using Doob’s Inequality,
we get:

E[(Xn + c)2 ]
     
P max Xm > λ = P max Xm + c > λ + c ≤ P max (Xm + c)2 > (λ + c)2 ≤
1≤m≤n 1≤m≤n 1≤m≤n (λ + c)2

Since c > −λ is arbitrary, calculate the minimum of the right hand side and the result will follow! 

Problem 1.3.8 Let Xn and Yn be martingales with EXn2 < ∞ and EYn2 < ∞. Then:
n
X
EXn Yn − EX0 Y0 = E(Xm − Xm−1 )(Ym − Ym−1 )
m=1

6
Proof Note that for each m ≥ 1:
E(Xm − Xm−1 )(Ym − Ym−1 ) =E[Xm Ym ] + E[Xm−1 Ym−1 ] − E[Xm Ym−1 ] − E[Xm−1 Ym ]
=E[Xm Ym ] + E[Xm−1 Ym−1 ] − E[E[Xm Ym−1 |Fm−1 ]] − E[E[Xm−1 Ym |Fm−1 ]]
=E[Xm Ym ] + E[Xm−1 Ym−1 ] − E[Ym−1 E[Xm |Fm−1 ]] − E[Xm−1 E[Ym |Fm−1 ]]
=E[Xm Ym ] − E[Xm−1 Ym−1 ]
Sum them up and the result follows. 
P∞
Problem 1.3.9 Let Xn , n ≥ 0, be a martingale and ξn = Xn − Xn−1 for n ≥ 1. If EX02 < ∞, m=1
2
Eξm <∞
then Xn → X∞ a.s. and in L2 .
Proof Use the result from the previous problem, setting Xn = Yn and we will get:
n
X
E[Xn2 ] = E[X02 ] + 2
E[ξm ]
m=1

This gives:
sup E[Xn2 ] < +∞
n≥0

By the Lp Convergence Theorem, we have Xn → X∞ a.s. and in L2 . 

1.4 Uniform Integrability, Convergence in L1

Problem 1.4.1 Let Xn be r.v.’s taking values in [0, ∞). Let D = {Xn = 0 f or some n ≥ 1} and assume:
P(D|X1 , · · · , Xn ) ≥ δ(x) > 0 a.s. on {Xn ≤ x}
Use Theorem 4.6.9 to conclude that P(D ∪ {limn Xn = ∞}) = 1.
Proof We only need to prove that {limn Xn = ∞})c ⊆ D. Note that:
+∞
[
{lim Xn = ∞}c = {lim inf Xn < ∞} = {lim inf Xn < M }
n n n
M =1

It is true that:
+∞
[ +∞
[ +∞
[
{lim inf Xn < M } = {Xn ≤ N i.o.} ⊆ {P(D|X1 , · · · , Xn ) ≥ δ(N ) i.o.}
n
M =1 N =1 N =1

By Theorem 4.6.9, we know that P(D|X1 , · · · , Xn ) converges to 1D almost surely. However, 1D is either 0 or
1, which indicates that on each {P(D|X1 , · · · , Xn ) ≥ δ(N ) i.o.}, 1D must be one. We can then conclude that:
{limn Xn = ∞})c ⊆ D a.s. This gives the desired result!. 
Problem 1.4.2 Let Zn be a branching process with offspring distribution pk (see the end of Section 5.3 for
denitions). Use the last result to show that if p0 > 0 then P(limn Zn = 0 or ∞) = 1.
Proof It is a direct corollary of the previous result. Note that:
P(D|Z1 , · · · , Zn ) ≥ P(Zn+1 = 0|Z1 , · · · , Zn ) ≥ pk0 on {Zn ≤ k}
We can then use the previous result. Here D = {limn Zn = 0} 
Problem 1.2.4 Revisited Now we can finish the proof of Problem 1.2.4! We have proved that ρZn is actually a
martingale. Since Zn converges to either 0 or ∞, the limit of ρZn is 0 or 1. So using the Bounded convergence
Theorem:
ρx = Ex [ρZn ] → 1 · Px (lim Zn = 0) + 0 · Px (lim Zn = ∞)
n n

(Here the subscript x denotes the initial state). Thus P(Zn = 0 f or some n ≥ 1|Z0 = x) = ρx . 

7
Problem 1.4.3 Show that if Fn % F∞ and Yn → Y in L1 then E[Yn |Fn ] → E[Y |F∞ ] in L1 .

Proof By triangular inequality:

E[|E[Yn |Fn ] − E[Y |F∞ ]|] ≤ E[|E[Yn |Fn ] − E[Y |Fn ]|] + E[|E[Y |Fn ] − E[Y |F∞ ]|]

For the first term:

E[|E[Yn |Fn ] − E[Y |Fn ]|] ≤ E[E[|Yn − Y ||Fn ]] = E[|Yn − Y |] → 0
For the second term, by the result of Theorem 4.6.7, E[Y |Fn ] → E[Y |F∞ ] in L1 , which indicates that the
second term tends to 0. Thus we get E[Yn |Fn ] → E[Y |F∞ ] in L1 . 

1.5 Backwards Martingales

1.6 Optional Stopping Theorems
Problem 1.6.1 If L ≤ M are stopping times and YM ∧n is a uniformly integrable submartingale, then EYL ≤ EYM
and:
YL ≤ E[YM |FL ]

Proof Before proving, we first need to justify the notation. On {M = ∞}, YM ∧n = Yn and it converges to some
limit, which we will denote by Y∞ . After noticing this, there will be no problem in the case when the stopping
W+∞
time index is infinity. Besides, there’s always no problem in the definition of F∞ since it equals n=1 Fn .
Now use Theorem 4.8.3 and let Xn = YM ∧n , N = L. We immediately get EYL ≤ EYM . To prove the last
result, considering for any A ∈ FL , use the stopping time in Problem 1.3.3 and let N = L · 1A + M · 1Ac . The
first result now shows EYN ≤ EYM . Since N = M on Ac , we have:

E[YL 1A ] ≤ E[YM 1A ]

Since A is arbitrary, use the definition of conditional expectation and we will get YL ≤ E[YM |FL ]. 

Problem 1.6.2 If Xn ≥ 0 is a supermartingale then P(sup Xn > λ) ≤ EX0 /λ.

Proof Let N = inf{n : Xn > λ}, A = {supn≥1 Xn > λ}. Using Theorem 4.8.4 we get:

E[X0 ] ≥ E[XN ] ≤ E[XN 1A ] ≥ λP(A)

This gives P(A) ≤ EX0 /λ. 

Problem 1.6.3 Let Sn = ξ1 + · · · + ξn where the ξi are independent with Eξi = 0 and V ar(ξi ) = σ 2 . Sn2 − nσ 2
is a martingale. Let T = min{n : |Sn | > a}. Use Theorem 4.8.2 to show that ET ≥ a2 /σ 2 .

Proof We might assume that ET is finite since otherwise the result is trivial. UsingTheorem 4.4.1, we have:

0 = E[S02 ] − 0 · σ 2 = E[Sn∧T
2
] − E[n ∧ T ]σ 2

This gives E[n ∧ T ]σ 2 = E[Sn∧T

2
]. Letting n → ∞ and use Fatou’s Lemma and Monotone Convergence
Theorem, we get:

σ 2 E[T ] = σ 2 lim E[n ∧ T ] = lim inf E[Sn∧T

2 2
] ≥ E[ lim Sn∧T ] = E[ST2 ] > a2
n→∞ n→∞ n→∞

(Here ST2 is well defined since we have assumed that ET is finite.) Thus ET ≥ a2 /σ 2 

Problem 1.6.4 (Wald’s second equation) Let Sn = ξ1 + · · · ξn , where the ξi are independent with Eξi = 0 and
V ar(ξi ) = σ 2 . Use the martingale from the previous problem to show that if T is a stopping time with ET < ∞
then EST2 = σ 2 ET .

8
Proof By the same method in the previous problem, we have E[n ∧ T ]σ 2 = E[Sn∧T
2
]. This gives for any n ≥ 1:
2
E[Sn∧T ] = E[n ∧ T ] ≤ E[T ] < ∞
2
Thus supn≥1 E[Sn∧T ] < ∞. Lp Convergence Theorem gives that the martingale Sn∧T converges to ST in L2 .
2
As a result, limn→∞ E[Sn∧T ] = E[ST2 ]. The result follows by letting n → ∞ in the first equation.
Remark We can also prove directly that {Sn∧T }∞ 2
n=1 is a Cauchy sequence in L (Ω). By the property of martin-
gale, if n > m:
E[(Sn∧T − Sm∧T )2 ] = ESn∧T
2 2
− ESm∧T = σ 2 (E[n ∧ T ] − E[m ∧ T ])
It is easy to check that the righthand side is cauchy sequence in R since we have known that E[n ∧ T ] converges
to E[T ] which is finite. This proves our claim! 

Problem 1.6.5 Variance of the time of gambler’s ruin Let ξ1 , ξ2 , · · · be independent with P(ξi = 1) = p
and P(ξi = −1) = q = 1 − p where p < 1/2. Let Sn = S0 + ξ1 + · · · + ξn and let V0 = min{n ≥ 0 : Sn = 0}.
Theorem 4.8.9 tells us that Ex V0 = x/(1 − 2p). The aim of this problem is to compute the variance of V0 . If
we let Yi = ξi − (p − q) and note that EYi = 0 and:

Var(Yi ) = Var(ξi ) = Eξi2 − (Eξi )2

then it follows that (Sn − (p − q)n)2 − n(1 − (p − q)2 ) is a martingale.

(a) Use this to conclude that when S0 = x the variance of V0 is:
1 − (p − q)2
x·
(q − p)3
(b) Why must the answer in (a) be in the form cx?

Proof Use the same method as in the previous problem. The martingale in the problem gives:
2
E[(Sn∧V0 − (p − q)(n ∧ V0 )) − (n ∧ V0 )σ 2 ] = x2

where σ 2 = 1 − (p − q)2 . This gives that:

2
E[(Sn∧V0 − (p − q)(n ∧ V0 )) ] = x2 + σ 2 E[n ∧ V0 ] ≤ x2 + σ 2 E[V0 ] < +∞
2
Thus we get supn≥1 E[(Sn∧V0 − (p − q)(n ∧ V0 )) ] < +∞. Lp Convergence Theorem gives that the martingale
Sn∧V0 − (p − q)(n ∧ V0 ) converges to SV0 − (p − q)V0 a.s. and in L2 . As a result:
2
lim E[(Sn∧V0 − (p − q)(n ∧ V0 )) ] = E[(SV0 − (p − q)V0 )2 ] = (p − q)2 E[V02 ]
n→∞


So letting n → ∞ in the first equation we can get that E[V02 ] = (p − q)x2 − x + (p − q)2 x /(p − q)3 . Notice that
E[V0 ] = x/(q − p), we come to:
1 − (p − q)2
Var(V0 ) = x ·
(q − p)3
Remark You can also exchange the limit and expectation by using Dominated Convergence Theorem.
Consider:
2 2
(Sn∧V0 − (p − q)(n ∧ V0 )) = Sn∧V0
− 2(p − q)(n ∧ V0 )Sn∧V0 + (p − q)2 (n ∧ V0 )2
2
For the first term, 0 ≤ Sn∧V 0
≤ (supn≥1 Sn )2 . Using Theorem 4.8.9 we can know that P(supn≥1 Sn ≥ m) =
(p/q) , which implies that E[(supn≥1 Sn )2 ] < +∞. For the last term, apply Fatou’s Lemma to the very first
m

equation in the first proof and we can get:

2
E[(0 − (p − q)V0 ) ] ≤ x2 + lim σ 2 E[n ∧ V0 ] = x2 + σ 2 E[V0 ] < +∞
n→∞

This shows that the third term (n ∧ V0 )2 could be dominated by V02 which is integrable. For the second term,
just use Cauchy Schwarz’s Inequality and we can find a dominator. Combining these observations, we can
exchange the expectation and the limit. 

9
Problem 1.6.6 Let Sn = ξ1 + · · · + ξn be a random walk. Suppose ϕ(θo ) = Eexp(θo ξ1 ) = 1 for some θo < 0
and ξi is not constant. In this special case of the exponential martingale Xn = exp(θo Sn ) is a martingale. Let
τ = inf{n : Sn ∈/ (a, b)} and Yn = Xn∧τ . Use Theorem 4.8.2 to conclude that EXτ = 1 and P(Sτ ≤ a) ≤
exp(−θo a).

Proof Following the hint, we need to check the condition in Theorem4.8.2. Obviously Xn 1{n<τ } is bounded
and thus uniformly integrable. Also, by using Fatou’s Lemma, we get that:

E[|Xτ |] = E[ lim |Xn∧τ |] ≤ lim inf E[|Xn∧τ |] < +∞

n→∞ n→∞

Thus by the theorem, we conclude that Xn∧τ is uniformly integrable. By letting n → ∞ in the equation:

1 = E[X0∧τ ] = E[Xn∧τ ]

We can get E[Xτ ] = 1. Finally, using Chebyshev’s Inequality we can get P(Sτ ≤ a) ≤ exp(−θo a). 

Problem 1.6.7 Continuing with the set-up of the previous problem, suppose that the ξi are integer valued with
P(ξi < −1) = 0, P(ξ1 = −1) > 0 and Eξi > 0. Let T = inf{n : Sn = a} with a < 0. Use the martingale
Xn = exp(θo Sn ) to conclude that P(T < ∞) = exp(−θo a).

10
2 Markov Chain
2.1 Examples
Problem 2.1.1 Let ξ1 , ξ2 , · · · be i.i.d. ∈ {1, 2, · · · , N } and taking each value with probability 1/N . Show that
Xn = |{ξ1 , · · · , ξn }| is a Markov chain and compute its transition probability.

Proof Obviously it is a Markov Chain, since the value of Xn+1 depends only on Xn and ξn+1 . It’s transition
probability could be represented as:
k k
p(k, k + 1) = 1 − , p(k, k) = , p(i, j) = 0 otherwise
N N


Problem 2.1.2 Let ξ1 , ξ2 , · · · be i.i.d. ∈ {−1, 1}, taking each value with probability 1/2. Let S0 = 0, Sn =
ξ1 + · · · ξn and Xn = max{Sm : 0 ≤ m ≤ n}. Show that Xn is not a Markov chain.

Proof Since Xn might equal to Sn and might not, we find that:

P(Xn+1 = m + 1|Xn = m) 6= P(Xn+1 = m + 1|Xn = m, Xn−1 = m)

Thus Xn is not a Markov chain. 

Problem 2.1.3 Let ξ0 , ξ1 , · · · be i.i.d. ∈ {H, T }, taking each value with probability 1/2. Show that Xn =
(ξn , ξn+1 ) is a Markov chain and compute its transition probability p. What is p2 ?

Proof Since Xn = (ξn , ξn+1 ), we know that Xn−1 = (ξn−2 , ξn−1 ), · · · , X0 = (ξ0 , ξ1 ) are independent with Xn
and thus Xn is a Markov chain. It’s transition probability is:
1 1 
2 2 0 0
0 0 1 1 
P= 2 2
 1 1 0 0
2 2
0 0 12 12 4×4

For p2 , since it represents the two step transition probability and by our previous observation, Xn+1 is independent
with Xn−1 , we immediately know that p2 (i, j) = 1/4 for any state i, j. 

2.2 Construction, Markov Properties

Problem 2.2.1 Let A ∈ σ(X0 , · · · , Xn ) and B ∈ σ(Xn , Xn+1 , · · · ). Use the Markov property to show that for
any initial distribution :
Pµ (A ∩ B|Xn ) = Pµ (A|Xn )Pµ (B|Xn )

Proof The left-hand side could be represented as:

Eµ [1A 1B |Xn ] =Eµ [Eµ [1A 1B |Fn ]|Xn ]

=Eµ [1A Eµ [1B |Fn ]|Xn ]
(B ∈ σ(Xn , Xn+1 , · · · )) =Eµ [1A Eµ [1B |Xn ]|Xn ]
=Eµ [1A |Xn ]Eµ [1B |Xn ]

This proves the equation. 

Problem 2.2.2 Let Xn be a Markov chain and suppose

P(∪∞
m=n+1 {Xm ∈ Bm }|Xn ) ≥ δ > 0 on {Xn ∈ An }

Then P({Xn ∈ An i.o.} − {Xn ∈ Bn i.o.}) = 0.

11
 . ∞
Proof Let En = ∪∞ m=n+1 {Xm ∈ Bm }. Then E = ∩n=1 En = {Xm ∈ Bm i.o.}. Also let Fn = σ(X0 , · · · , Xn )
and F∞ . Obviously 1En ∈ σ(Xn , Xn+1 , · · · ) and 1En convergence almost surely to 1E ∈ F∞ . So by the markov
property and Dominated Convergence Theorem for Conditional Expectation, we get:

P(En |Xn ) = P(En |Fn ) → P(E|F∞ ) = 1E , n → ∞

Since on {Xn ∈ An , i.o.}, P(En |Xn ) ≥ δ > 0 infinitely often, 1E must equal to one on {Xn ∈ An , i.o.}. This
means that {Xn ∈ An i.o.} ⊆ {Xn ∈ Bn i.o.}. 

Problem 2.2.3 A state a is called absorbing if Pa (X1 = a) = 1. Let D = {Xn = a f or some n ≥ 1} and let
h(x) = Px (D). Use the result of the previous exercise to conclude that h(Xn ) → 0 a.s. on Dc . Here a.s. means
Pµ a.s. for any initial distribution µ.

Proof For any  > 0, let A = {x : Px (D) ≥ }. By calculation:

+∞
[
PXn (D) = EXn [1D ] = Eµ [1D ◦ θn |Fn ] = Pµ ( {Xm = a}|Xn )
m=n+1

S+∞
Thus on {Xn ∈ A }, we have Pµ ( m=n+1 {Xm = a}|Xn ) ≥ . So by using the previous problem, we get that:

{Xn ∈ A i.o.} ⊆ {Xn = a i.o, } ⊆ D

As a result, on DC , for any  > 0, Xn ∈ A for only finite many times. This means that on Dc , h(Xn ) → 0. 

Problem 2.2.4 First Entrance Decomposition Let Ty = inf{n ≥ 1 : Xn = y}. Show that:
n
X
pn (x, y) = Px (Ty = m)pn−m (y, y)
m=1

Proof Notice that {Xn = y} ⊆ {Ty ≤ n}. Thus:

Px (Xn = y) =Ex [1{Xn =y} 1{Ty ≤n} ]

=Ex [1{Ty ≤n} Ex [1{Xn−Ty =y} ◦ θTy |FTy ]]
=Ex [1{Ty ≤n} EXTy [1{Xn−Ty =y} ]]
=Ex [1{Ty ≤n} pn−Ty (y, y)]
n
X
= Px (Ty = m)pn−m (y, y)
m=1

This completes the result. 

Problem 2.2.5 Show that:

n
X n+k
X
Px (Xm = x) = Px (Xm = x)
m=0 m=k

Proof Consider from the right hand side. Let Tx = inf{n ≥ k : Xn = x} be the first time Xn = x after time k.
By the same method as in the previous problem, we can prove that:
m
X
Px (Xm = x) = Px (Tx = l)pm−l (x, x), ∀m ≥ k
l=k

12
Put the equation into the right hand side of our goal:
n+k
X n+k
XX m
pm (x, x) ≤ Px (Tx = l)pm−l (x, x)
m=k m=k l=k
n+k
X n+k
X
= Px (Tx = l) pm−l (x, x)
l=k m=l
n+k
X n+k−l
X
= Px (Tx = l) pm (x, x)
l=k m=0
n
X n+k−m
X
= pm (x, x) Px (Tx = l)
m=0 l=k
Xn
≤ pm (x, x)
m=0

This proves the inequality. 

Problem 2.2.6 Let TC = inf{n ≥ 1 : Xn ∈ C}. Suppose that |S \ C| < ∞ and for ∀x ∈ S \ C, we have
Px (Tc < ∞) > 0. Prove that there is an N < ∞ and  > 0 such that ∀y ∈ S \ C, Py (TC > kN ) < (1 − )k .

Proof By assumption, for any x ∈ S \ C, there exists n(x) ∈ N and x > 0 such that Px (TC ≤ n(x)) > x . Let
N = maxx∈S\C {n(x)} and  = minx∈S\ {x }. Then Px (TC ≤ N ) >  holds for all x ∈ S \ C. Now consider:

Py (TC > kN ) =Ey [1{TC >kN } 1{TC >(k−1)N } ]

=Ey [1{TC >(k−1)N } Ey [1{TC >kN } |F(k−1)N ]]
=Ey [1{TC >(k−1)N } Ey [1{TC >N } ◦ θ(k−1)N |F(k−1)N ]]
=Ey [1{TC >(k−1)N } EX(k−1)N [1{TC >N } ]]
≤(1 − )Ey [1{TC >(k−1)N } ]
≤ · · · ≤ (1 − )k

This gives the result. 

Problem 2.2.7 Exit distributions Let VC = inf{n ≥ 0 : Xn ∈ C} and let h(x) = Px (VA < VB ). Suppose
A ∩ B 6= ∅, S − (A ∪ B) is finite, and Px (VA∪B < ∞) > 0 for all x ∈ S − (A ∪ B).
(i) Show that: X
h(x) = p(x, y)h(y) f or x ∈
/ A∩B
y

(ii) Show that if h satises the previous equation, then h(X(n ∧ VA∪B )) is a martingale.
(iii) Use this and Problem 2.2.6 to conclude that h(x) = Px (VA < VB ) is the only solution of the previous
equation that is 1 on A and 0 on B.

Proof (i) Define Y (ω) = 1 if VA (ω) < VB (ω) and Y (ω) = 0 otherwise. In other words, Y = 1{VA (ω)<VB (ω)} . By
the Markov Property, we have:
Ex [Y ◦ θ1 |F1 ] = EX1 [Y ]
Notice that when x ∈/ A ∪ B, Y ◦ θ1 = Y . So by integration, we get that:
X X
h(x) = Px (VA < VB ) = Ex [Y ] = Ex [EX1 [Y ]] = Px (X1 = y)Py (VA < VB ) = p(x, y)h(y)
y∈S y∈S

13
(ii) We have to check that Eµ [h(X(n+1)∧VA∪B )|Fn ] = h((Xn∧VA∪B ) for all n ≥ 0. For simplicity, we will omit
the subscript µ and denote the stopping time VA∪B by V . To prove this, it is sufficient to show it holds on both
{V ≤ n} and {V > n}. On {V > n}, we have:

E[h(X(n+1)∧V )|Fn ] = E[h(Xn+1 )|Fn ] = EXn [X1 ] = h(Xn ) = h(Xn∧V )

Here we have used the fact that Xn ∈

/ A ∪ B and the equation. On {V ≤ n}, h(X(n+1)∧V ) = h(Xn∧V ) ∈ Fn .
This gives:
E[h(X(n+1)∧V )|Fn ] = E[h(Xn∧V )|Fn ] = h(Xn∧V )
Thus h(Xn∧V ) is a martingale.
(iii) Use the result of Problem 2.2.6, P(VA∪B < ∞)=1. So h(Xn∧VA∪B ) → h(XVA∪B ) a.s. Also, notice that h
must be a bounded function since there are only finite many variables in S − A ∪ B and outside it h is either 0
or 1. Thus by the Bounded Convergence Theorem and the property of martingale, we have:

h(x) = Ex [h(X0∧VA∪B )] = Ex [h(Xn∧VA∪B )] → Ex [h(XVA∪B )] = 1 · Px (VA < VB ) + 0 · Px (VB < VA )

This proves the result! 

Problem 2.2.8 Let Xn be a Markov chain with S = {0, 1, · · · , N } and suppose that Xn is a martingale. Let
Vx = min{n ≥ 0 : Xn = x} and suppose Px (V0 ∧ VN < ∞) > 0 for all x. Show Px (VN < V0 ) = x/N .
a.s.
Proof By Problem 2.2.6 we know that P(V0 ∧ VN < ∞) = 1. Let T = V0 ∧ VN , then Xn∧T −−→ XT . Since
Xn∧T is also a martingale and is bounded, by using Dominated Convergence Theorem we know that:

x = Ex [X0∧T ] = Ex [Xn∧T ] → E[XT ] = N P(V0 > VN ) + 0 · P(V0 < VN ), n → ∞

Thus we get P(VN < V0 ) = x/N . 

2.3 Recurrence and Transience

Problem 2.3.1 Suppose y is recurrent and for k ≥ 0, let Rk = Tyk be the time of the kth return to y, and for
k ≥ 1 let rk = Rk − Rk1 be the kth interarrival time. Use the strong Markov property to conclude that under Py ,
the vectors vk = (rk , XRk−1 , · · · , XRk −1 ), k ≥ 1 are i.i.d.

Proof To prove that {vn }n≥1 are independent, we need to show that for each n ≥ 2, vn is independent of
v1 , · · · , vn−1 . Using the shift operator, we know that:

vn = v1 ◦ θRn−1 , n ≥ 2

Since every vi takes only countably possible values, we may just consider each v that vi might take. By the strong
Markov property, we know that:

Py (vn = v|FRn−1 ) = Py (v1 ◦ θRn−1 = v|FRn−1 ) = PXRn−1 (v1 = v) = Py (v1 = v)

This actually implies that vn is independent of FRn−1 because you can consider for any A ∈ FRn−1 , we have:

Py (vn = v, A) = Ey [Ey [1{vn =v} 1A |FRn−1 ]] = Ey [1A P(vn = v|FRn−1 )] = P(vn = v)P(A)

Thus vn is independent of FRn−1 and consequently independent of v1 , · · · , vn−1 . So we have proved that {vn }n≥1
are independent. Finally by taking expectation over the first equation, we get that Py (vn = v) = Py (v1 = v).
This shows that {vn }n≥1 are i.i.d. 

14
Problem 2.3.2 Use the strong Markov property to show that ρxz ≥ ρxy ρyz .

By the strong Markov property, we have:

Px (Tz ◦ θTy < ∞|FTy ) = Py (Tz < ∞), on {Ty < ∞}

Notice that:
{Ty < ∞} ∩ {Tz ◦ θTy < ∞} ⊆ {Tz < ∞}
Thus we have:
Px (Tz < ∞) ≥P(Tz ◦ θTy < ∞, Ty < ∞)
=Ex [Px (Tz ◦ θTy < ∞|FTy )1{Ty <∞} ]
=Ex [Py (Tz < ∞)1{Ty <∞} ]
=Py (Tz < ∞)Px (Ty < ∞)
This proves the inequality. 

Problem 2.3.3 Show that in the Ehrenfest chain (Example 5.1.3), all states are recurrent.

Proof Since for any x, y ∈ S = {0, 1, · · · , r}, ρxy > 0, we know that S is irreducible. Thus all states are recurrent
since |S| < +∞. 
P
Problem 2.3.4 f is said to be superharmonic if f (x) ≥ y p(x, y)f (y), or equivalently f (Xn ) is a supermartin-
gale. Suppose p is irreducible. Show that p is recurrent if and only if every nonnegative superharmonic function
is constant.

Proof (⇒) Since f (Xn ) is a nonnegative supermartingale, it converges almost surely. However, Xn is a irreducible
recurrent Markov chain, for each state i, Xn a.s. visits i for infinitely many times. This means that f could only
be a constant.
(⇐) Suppose any nonnegative superharmonic function is constant. By using contradiction, if p is not recurrent,
then all the states are transient. Fixing a state y, define:

f (x) = Px (Ty < ∞)

We first show that f is a nonnegative superharmonic function. By the Markov property:

X
Px (Ty < ∞) ≥ Px (Ty ◦ θ1 < ∞) = Ex [Px (Ty ◦ θ1 < ∞|F1 )] = Ex [PX1 (Ty < ∞)] = p(x, z)Pz (Ty < ∞)
z

This gives the result. Next we will show that it is impossible. Consider using the same method:

f (x) =Px (X1 = y) + Px (Ty < ∞, X1 6= y)

=p(x, y) + Px (Ty ◦ θ1 < ∞, X1 6= y)
X
=p(x, y) + p(x, z)Pz (Ty < ∞)
z6=y
X
(f is constant) =p(x, y) + p(x, z)f (x)
z6=y

=p(x, y) + f (x)(1 − p(x, y))

This means p(x, y)(f (x) − 1) = 0. Since we assume that the chain is irreducible, we know that f (x) = f (y) =
Py (Ty < ∞) < 1. Thus p(x, y) = 0 for any state x. This is impossible since the chain is irreducible! As a result,
p must be recurrent. 

15
2.4 Stationary Measure
Problem 2.4.1 Let wxy = Px (Ty < Tx ). Show that µx (y) = wxy /wyx .
Proof Recall that µx (y) = Ex [Ny ] denotes the expected times of visits to y before returning x. Here x is
recurrent. Consider for any k ≥ 1:
{Ny = k} = {0 < Ty1 < · · · < Tyk < Tx1 < Tyk+1 }
notice that since Tx1 is finite, all the Tyi are finite for i = 1, · · · , k. So by using the strong Markov property, we
get:
Px (Ny = k) =Ex [1{0<T 1 <···<tk <T 1 <Tyk+1 } ]
y y x

=Ex [1{0<Ty1 <···<Tyk <Tx1 } Ex [1{T 1 <Tyk+1 } |FTyk ]]

x

=Ex [1{0<Ty1 <···<Tyk <Tx1 } Ex [1{Tx1 <Ty1 }◦θT k |FTyk ]]

y

=Ex [1{0<Ty1 <···<Tyk <Tx1 } Ey [1{Tx1 <Ty1 } ]]

=Ex [1{0<Ty1 <···<Tyk <Tx1 } ]Py (Tx < Ty )
=···
k−1
=Px (Ty < Tx ) (Py (Ty < Tx )) Py (Tx < Ty )
k−1
=wxy (1 − wyx ) wyx
The second equation above follows from the fact that {0 < Ty1 < · · · < Tyk < Tx1 } ∈ FTyk . Thus we can get:
+∞
X wxy
µx (y) = Ex [Ny ] = k · wxy (1 − wyx )k−1 wyx =
wyx
k=1

This completes the proof! 

Remark Here is something subtle, here we could check directly that Px (Ny = ∞) = 0. If you use other methods,
you should first check it out. An easy way (maybe) use the remarks and technique notes after Theorem 5.5.7
Problem 2.4.2 Show that if p is irreducible and recurrent then:
µx (y)µy (z) = µx (z)
Proof From Theorem 5.5.9, we know that any two stationary measures differ only by a constant multiplier.
Thus if µx (z) and µy (z) are simultaneously zero, the equation obviously holds. Otherwise, by the theorem, we
have:
µx (y) µx (z)
=
µy (y) µy (z)
Notice that µy (y) = 1 and the desired results follows. 
Problem 2.4.3 Suppose p is irreducible and positive recurrent. Then Ex Ty < ∞ for all x, y.
Solution Since p is irreducible, any x, y ∈ S satisfy ρxy > 0. Thus we can take m = inf{n ∈ N : pn (x, y) > 0}
which is finite. By using C-K Equation, we can get z1 , · · · , zm−1 such that p(x, z1 )p(z1 , z2 ) · · · p(zm−1 , y) =
Px (X1 = z1 , · · · , Xm−1 = zm−1 , Xm = y) > 0. Here zi 6= x for all 1 ≤ i ≤ m − 1 by the definition of m. Now
consider:
Ex [Tx ] ≥Ex [Tx 1{X1 =z1 ,··· ,Xm−1 =zm−1 ,Xm =y} ]
=Ex [Ex [Tx |Fm ]1{X1 =z1 ,··· ,Xm−1 =zm−1 ,Xm =y} ]
(zi 6= x) ≥Ex [Ex [Tx ◦ θm |Fm ]1{X1 =z1 ,··· ,Xm−1 =zm−1 ,Xm =y} ]
=Ex [Ey [Tx ]1{X1 =z1 ,··· ,Xm−1 =zm−1 ,Xm =y} ]
=Ey [Tx ]Px (X1 = z1 , · · · , Xm−1 = zm−1 , Xm = y) > 0
Since Ex [Tx ] < ∞ and Px (X1 = z1 , · · · , Xm−1 = zm−1 , Xm = y) > 0, we get that Ey [Tx ] < ∞. This proves the
result. 

16
 P
Problem 2.4.4 Suppose p is irreducible and has a stationary measure with x µ(x) = ∞. Then p is not positive
recurrent.

Proof If the chain is irreducible and positive recurrent, then there exists a stationary distribution by Theorem
5.5.12. However Theorem 5.5.9 tells us any two stationary measures differ only by a constant multiplier. But
this could not happen since the stationary distribution has total mass 1 while the given stationary measure has
total mass ∞. Thus p is not positive recurrent. 

Problem 2.4.5 Use Theorems 5.5.7 and 5.5.9 to show that for simple random walk:
(a) The expected number of visits to k between successive visits to 0 is 1 for all k.
(b) If we start from k the expected number of visits to k before hitting 0 is 2k.

Proof (a) By the strong markov property, the desired expected number is equal to µ0 (k). Since we have known
that µ(x) ≡ 1 is a stationary measure, Theorem 5.5.9 gives that µ0 (k) = µ0 (0) = 1.
Proof 1 of (b) Let Nk denote the times of visits before hitting 0. Without the loss of generality, we might
assume that k > 0. Using the same method as in Problem 2.4.1, we can get for any j ≥ 1:
j−1
Pk (Nk = j) = Pk (Tk < T0 )j−1 Pk (T0 < Tk ) = (1 − Pk (T0 < Tk )) Pk (T0 < Tk )

We only need to calculate Pk (T0 < Tk ). If k ≥ 2, conditioning on the first step and use the markov property, we
can get:
Pk (T0 < Tk ) =Ek [Ek [1{T0 <Tk } |F1 ]]
=Ek [1{X1 =k−1} Ek [1{T0 <Tk } |F1 ]] + Ek [1{X1 =k+1} [Ek [1{T0 <Tk } |F1 ]]
=Ek [1{X1 =k−1} Pk−1 (T0 < Tk )] + 0
1
= Pk−1 (T0 < Tk )
2
The second is zero since if X1 = k + 1, it will always first reach k before 0. Now use Theorem 4.8.7 in the
textbook, we know that Pk−1 (T0 < Tk ) = 1/k and thus the desired probability is 1/2k. If k = 1, this obviously
holds. Finally we get:
+∞  j−1
X 1 1
Ek [Nk ] = j· · 1− = 2k
j=1
2k 2k

This completes the proof! 

Proof 2 of (b) For any k ≥ 1, we have:
0 −1
TX 0 −1
TX T
X −1 T
X −1
µ0 (k) = E0 [ 1{Xn =k} ] = E0 [ 1{Xn =k} ] = E0 [ 1{Xn =k,X1 =1} ] + E0 [ 1{Xn =k,X1 =−1} ]
n=0 n=1 n=1 n=1
PT −1
Since E0 [ n=1 1{Xn =k,X1 =−1} ] = 0, using markov property (the same as in the previous method):
T −1 T −1 T −1
X X 1 X
µ0 (k) = E0 [ 1{Xn =k,X1 =1} ] = P0 (X1 = 1)E1 [ 1{Xn =k} ] = E1 [ 1{Xn =k} ] = 1
n=1 n=0
2 n=0
PT −1
Thus E1 [ n=0 1Xn =k ] = 2. Now consider starting from k, before hitting k − 1, we visit k twice on average. Then
starting from k − 1, before hitting k − 2, we visit k twice on average. Repeat the argument, we would get that
starting from k, before hitting 0, we will visit k on average 2k times. 

2.5 Asymptotic Behaviour

Problem 2.5.1 Suppose S = {0, 1} and:
 
1−α α
p=
β 1−β

17
Use induction to show that:
 
β n β
Pµ (Xn = 0) = + (1 − α − β) µ(0) −
α+β α+β
Proof It’s easy to check that the equation holds when n = 0. Suppose it is right when n = k − 1. For n = k, use
the markov property, we get:

Pµ (Xk = 0) = Eµ [p(Xk−1 , 0)] = Pµ (Xk−1 = 0) · p(0, 0) + Pµ (Xk−1 = 1) · p(1, 0)

     
β k−1 β β k−1 β
= + (1 − α − β) µ(0) − (1 − α) + 1 − − (1 − α − β) µ(0) − β
α+β α+β α+β α+β
 
β β
= (1 − α − β) + β + (1 − α − β)k µ(0) −
α+β α+β
 
β β
= + (1 − α − β)k µ(0) −
α+β α+β
This shows that the equation holds when n = k and the proof is then complete. 

Problem 2.5.2 Show that if S is finite and p is irreducible and aperiodic, then there is an m so that pm (x, y) > 0
for all x, y.

Proof By Lemma 5.6.5 in the textbook, we know that for any x ∈ S, there is Nx ∈ N such that pn (x, x) > 0
whenever n ≥ Nx . Moreover, by the condition of irreducible, for any pair x, y ∈ S, there is n(x, y) ∈ N such that
pn(x,y) (x, y) > 0. Now let:  
N = max max{Nx }, max {n(x, y)}
x∈S x,y∈S

Notice that N is finite since |S| < ∞. Consider if n = 2N , we have:

p2N (x, y) ≥ pn(x,y) (x, y)p2N −n(x,y) (y, y) > 0

The first inequality follows from C-K equation. This gives the desired m = 2N . 

Problem 2.5.3 Show that if S is nite, p is irreducible and aperiodic, and T is the coupling time dened in the
proof of Theorem 5.6.6 then P(T > n) ≤ Crn for some r < 1 and C < ∞. So the convergence to equilibrium
occurs exponentially rapidly in this case. Hint: First consider the case in which p(x, y) > 0 for all x and y and
reduce the general case to this one by looking at a power of p.

Proof Case 1 If p(x, y) > 0 for all x and y, set  = minx,y {p(x, y)} > 0. Also, let N = |S|. Consider for any
x 6= y, we have:
X
P(T = n + 1|Xn = x, Yn = y, T > n) = P(Xn + 1 = Yn+1 |Xn = x, Yn = y) = p(x, z)p(y, z) ≥ 2 N
z∈S

Notice that: G
{T > n} = {T > n} ∩ {Xn = x, Yn = y}
x6=y

From this we can deduce that for all n ∈ N (just basic calculations of probability):

P(T = n + 1|T > n) ≥ 2 N ⇒ P(T > n + 1|T > n) < 1 − 2 N

Thus by induction, we get:

P(T > n) = P(T > n|T > n − 1)P(T > n − 1) < (1 − 2 N )P(T > n − 1) < · · · < (1 − 2 N )n P(T > 0)

Take r = 1 − 2 N and C = P(T > 0), the desired result follows.

Case 2 In the general case, we can use the result of Problem 2.5.2. It says there exists m ∈ N such that

18
pm (x, y) > 0 for all x and y. Now take  = minx,y {pm (x, y)} > 0. Use the same method as in case1, we can prove
that for all n ∈ N:
P(T > n + m|T > n) < 1 − 2 N
Thus by induction, we come to:
n n
P(T > n) < (1 − 2 N )b m c P(T > n(mod m)) ≤ (1 − 2 N ) m −1 P(T > n(mod m))

Take r = (1 − 2 N )1/m and C = 1r P(T > n(mod m)). The desired result follows. 

19