SIE 321 Probabilistic Models in OR Homework 7

Problem 1. A manufacturer has a machine that, when operational at the beginning of a day, has a
probability of 0.1 of breaking down sometime during the day. When this happens, the
repair is done the next day and completed at the end of that day.

(a) Formulate the evolution of the status of the machine as a Markov chain by identifying
three possible states at the end of each day, and then constructing the (one-step)
transition matrix.
(b) Find the probability that the machine will be operational for 5 straight days before
breaking down. Compare this with the probability that a currently operational ma-
chine will be broken down on day 6.
(c) Find the expected first passage time µi j from state i to state j, for all i and j. Use
these results to identify the expected number of full days that the machine will remain
operational before the next breakdown after a repair is completed.

Problem 2. Consider the following gamblers ruin problem. A gambler bets $1 on each play of a game.
Each time, he has a probability p of winning and probability q = 1 − p of losing the dollar
bet. He will continue to play until he goes broke or nets a fortune of T dollars. Let X n
denote the number of dollars possessed by the gambler after the n-th play of the game.
Then
¨
X n + 1 with probability p
X n+1 = for 0 < X n < T
X n − 1 with probability 1 − p
X n+1 = X n for X n = 0 or X n = T

Defined in such a way X n is a Markov chain. The gambler starts with X 0 dollars, where
0 < X0 < T .

(a) Construct the (one-step) transition matrix

(b) Let T = 3 and p = 0.55. Find the probabilities of winning T dollars in three or four
rounds of play when the initial capital of the gambler is 1, . . . , T − 1 dollars.

1
Problem 3. A mouse maze consists of the paths shown below. At any intersection, the mouse is equally
likely to select any of the available paths.

(a) Formulate the (one-step) transition matrix

(b) Determine the probability that, starting at intersection 1, the mouse will reach inter-
section 5 for the first time in 4 runs (1 run = a run between two intersections)?
(c) What is the expected number of runs needed for the mouse to reach intersection 5
from intersection 1 for the first time?