BSCS 32063 - Stochastic Processes

Assignment 1, due Monday, May. 7 at 5:30pm

1. A coin having probability p of resulting in heads is successively flipped until the rth
head appears. Let X be the total number of flips required. What is the distribution of
X? (Give the name and explain why you picked that distribution)
Argue that the probability mass function of X is given by
 
n−1 r
p(k) = p (1 − p)n−r for k=r, r+1, . . . .
r−1

Hint: How many successes must there be in the first n − 1 trials?

2. Consider a Markov chain on S = {1, 2, 3, 4, 5, 6} specified by the transition probability

matrix
1 3

4 4 0 0 0 0
1 1
0 0 0 0
2 2 
0 0 1 0 0 0
 
P = 1 2 .
0 0 3 3 0 0
 
1 0 0 0 0 0
0 0 0 0 0 1

(a) What are the (communicating) classes of this Markov chain? Is the Markov chain
irreducible ?

(b) Which states are transient and which are recurrent? Justify your answers.

(c) What is the period of each state of this Markov chain? Is the Markov chain ape-
riodic ?

(d) Let X0 be the initial state with distribution p(0) = (0, 41 , 34 , 0, 0, 0)> corresponding
to the probability of being in states 1, 2, 3, 4, 5, 6 respectively.
Let X0 , X1 , X2 , . . . be the Markov chain constructed using P above. What is
E(X1 )?
(e) What is Var(X1 )?
(f) What is E(X3 )?

3. The times that Tom wakes up on any given morning are 5:30 am, 6:00 am or 7:00 am
Assume that if Tom wakes up at 5:30 am on a given morning, then the probability that
he wakes up at 5:30 am the next morning is 80%, at 6:00 am is 10%, and at 7:00 am is
10%.
If Tom wakes up at at 6:00 am on a given morning, then the probability that he wakes
 up at 5:30 am the next morning is 20%, at 6:00 am is 60%, and at 7:00 am is 20%.
If Tom wakes up at at 7:00 am on a given morning, then the probability that he wakes
up at 5:30 am the next morning is 25%, at 6:00 am is 25%, and at 7:00 am is 50%.

(a) Form a Markov chain by following Tom’s daily waking up time over a few days.
Define the corresponding random process and the state space clearly.

(b) Set up the matrix of transition probabilities.

(c) Find the probability of Tom waking up tomorrow morning at 5:30 am if he woke
up at 7:00 am yesterday morning.

4. (a) Explain what is meant by a ”recurrent state”.

(b) Consider the Markov chain given below.

There are two recurrent classes, R1 and R2 . Given that R1 = {1, 2},
i. Find R2 .

ii. Assuming X0 = 3, find the probability that the chain gets absorbed in R1 .