Probability Models

IE 502

Exercise 1. Ehrenfest chain: The following is a simple model of the exchange of heat or of
gas molecules between two isolated bodies. Suppose we have two boxes, labelled 1 and 2,
and d balls labelled 1, 2, . . . , d. Initially, some of these balls are in box 1 and the remainder
are in box 2. An integer is selected at random from 1, 2, . . . , d and the ball labelled by that
integer is removed from its box and placed in the opposite box. This procedure is repeated
indefinitely with the selections being independent from trial to trial. Model this as Markov
Chain. Classify the states.

Exercise 2. A machine can be either working or broken down on a given day. If it is work-
ing, it will break down in the next day with probability b, and will continue working with
probability 1 - b. If it breaks down on a given day, it will be repaired and be working in
the next day with probability r, and will continue to be broken down with probability 1-r.
Suppose that whenever the machine remains broken for a given number of l days, despite the
repair efforts, it is replaced by a new working machine. Model this machine as a Markov
chain.

Exercise 3. Packet arrive at a node of a communication network, where they are stored
in a buffer and then transmitted. The storage capacity of the buffer is m. Assume that
arrival and service does not occur simultaneously. Let arrival happens with probability b
and transmission is completed with probability d. Model this a Markov chain.

Exercise 4. A spider and a fly move along a straight line in unit increments. The spider
always moves towards the fly by one unit. The fly moves towards the spider by one unit
with probability 0.3, moves away from the spider by one unit with probability 0.3, and stays
in place with probability 0.4. The initial distance between spider and the fly is an integer.
When the spider and the fly land in the same position, the spider captures the fly. Construct a
Markov chain that describes the relative location of the spider and fly. Identify the transient
and recurrent states.

Exercise 5. Identify the communicating class from the following transition probability ma-
trix of a Markov chain.  
1/2 1/2 0 0 0
1/3 2/3 0 0 0 
 
 
 0 1/4 1/4 1/2 0 
 
 0 0 1/2 1/2 0 
 
0 0 0 1/2 1/2
Exercise 6. Two players play a very simple game of tennis. They compete for points, each
of which player 1 wins with probability p and player 2 with probability q. As soon as one
player achieves a lead of two points, the game is over.

(a) Model the evolution of this game as a Markov Chain with finite state space. Write down
the transition matrix.

(b) Identify the transient and recurrent states. Justify your answer.

Exercise 7. Gambler’s ruin chain: Suppose a gambler starts out with a certain initial
capital in dollars and makes a series of one dollar bets against the house. Assume that he
has respective probabilities p and q = 1 − p of winning and losing each bet, and that if his
capital ever reaches zero, he is ruined and his capital remains zero thereafter. Model this as
Markov chain. Classify the states.

Exercise 8. Specify the classes of the following Markov chains, and determine whether they
are transient or recurrent:

Exercise 9. {Xi : i ≥ 1} is a sequence of i.i.d. random variables with P (Xi = 1) = 1/2,

and P (Xi = −1) = 1/2. Let {Sn : n ≥ 0} is a process defined as follows. For some given
integer k, S0 = k and for n ≥ 1,
n
X
Sn = S0 + Xi .
i=1

Show that {Sn : n ≥ 1} is a Markov chain. Display its transition probability diagram. Clas-
sify the Markov chain {Sn } as follows,

• reducible or irreducible,

• transient or recurrent.

Exercise 10. Give an example of a Irreducible Markov chain, that is transient.

P∞
Exercise 11. Suppose that 0 ≤ pi ≤ 1, ∀i ∈ N, and that i=1 pi = ∞. Then, prove that
∞
Y
(1 − pi ) = 0
i=1

Exercise 12. Give an example of a sequence Xn of random variables for which E[Xn2 ] → 0,
but Xn does not converge almost surely to 0.

Exercise 13. Compute the limit of

1 1
x21 + · · · + x2n
Z Z
··· dx1 . . . dxn
0 0 x1 + · · · + xn

Exercise 14. X is uniformly distributed between -1 and 1. Let X1 , X2 , . . . , be i.i.d. random

variables with same distribution as X. Determine which of the following sequences (all with
i = 1,2,. . . ) are convergent in probability.

X1 +X2 +···+Xi
(a) Ui = i

2
(b) Wi = max(X1 , . . . , Xi )

(c) Vi = X1 · X2 · · · · · Xi

Exercise 15. In a state A, the voters vote for two candidates with equal probability, indepen-
dently of each other. If there are 5 million voters, what is the probability that the difference
of the votes given for the two candidates is less than 300 in absolute value.

Exercise 16. Use the Weak Law of Large Numbers to prove that for the true probability p of
an event and for its relative frequency X n based on an n-element i.i.d. Bernoulli sample:

p(1 − p) 1
P(| X n − p |≥ ) ≤ 2
≤ ∀  > 0.
n 4n2
Exercise 17. On the basis of the previous exercise find the number n of people to be inter-
viewed so that the true (but unknown) population support p of a candidate and its relative
frequency based on this poll differ at most 0.01 with probability at least 90 percent.