Probability for Data Science – Master’s Degree in Data Science – A.Y.

2023/2024
Academic Staff: Francesca Collet, Paolo Dai Pra

PROBLEM SET 16
Continuous-time Markov chain I: construction and stationary distribution

P16.1 (v From past exam). Cars of type i = 1, 2 arrive at a one-way street at random time
instants with expected interarrival times of `i minutes (`1 = 3, `2 = 10). The street has one parking
space. If an arriving car finds the space vacant, the car parks there immediately. Otherwise, the
car drives away. A car of type i remains parked for an expected duration of mi minutes (m1 = 5,
m2 = 20). The interarrival times and parking times of type-i cars are mutually independent and
exponentially distributed. In addition, type-1 cars behave independently of type-2 cars. Currently,
the parking space is vacant.
(a) Model the state of the parking space as a three-state continuous-time Markov chain. Give the
holding time parameters and the transition matrix of the embedded jump chain.
(b) What is the probability that at least 2 cars will arrive on the street during the 5-minute time
interval starting one hour from now?
Solution.
(a) Let X(t) =“state of the parking space at time t” be the state of the system at time t. We
consider the following states for the system: 0 ←→ the parking space is vacant; 1 ←→ the
parking space is occupied by a type-1 car and 2 ←→ the parking space is occupied by a type-
2 car. We can characterize the continuous-time Markov chain (X(t))t≥0 on the state space
S = {0, 1, 2} as follows.
Let Ti be the waiting time for an arrival

of a type-i car
and let Ri be1the

parking time of1a
type-
i car (i = 1, 2). We have T1 ∼ Exp 31 , T2 ∼ Exp 10 1
, R1 ∼ Exp 5 and R2 ∼ Exp 20 . We
determine the holding time parameters and the transition matrix of the embedded jump chain.

Holding time parameters:

13 13


v0 = (the waiting time for a car arrival is min{T1 , T2 } ∼ Exp 30
)
30
1 (the waiting time for
a type-1 parked car to leave the parking
v1 = slot is R1 ∼ Exp 15 )
5
1 (the waiting time for
a type-2 parked car to leave the parking
v2 = slot is R2 ∼ Exp 201
)
20

Jump chain transition matrix (non-zero entries):

10 3
P01 = P (T1 < T2 ) = P02 = P (T2 < T1 ) = P10 = 1 P20 = 1,
13 13
that is
10 3
 
0 13 13
P = 1 0 0 .
 

1 0 0

(b) Let N (t) be the number of cars arriving to the street by time t. As the waiting times between
13
successive car arrivals are i.i.d. exponential random variables with parameter 30 cars per
13
minute, the counting process (N (t))t≥0 is a Poisson with rate 30 cars per minute. We want
to compute the probability that at least 2 cars
arrive to the street from the 60-th to the 65-th
minute. As N (65) − N (60) ∼ N (5) ∼ Po 13 6 (stationary increments and Poissonianity), we
obtain
19 − 13
P (N (65) − N (60) ≥ 2) = P (N (5) ≥ 2) = 1 − P (N (5) = 0) − P (N (5) = 1) = 1 − e 6.
6
P16.2. Suppose that a one-celled organism can be in one of two states–either A or B. An individual
in state A will change to state B at an exponential rate α; an individual in state B divides into two
new individuals of type A at an exponential rate β. Let XA (t) (resp. XB (t)) denote the number of
organisms in state A (resp. B) at time t. Define an appropriate continuous-time Markov chain for
the time-evolution of (XA (t), XB (t)).

Solution. We denote by X(t) = (XA (t), XB (t)) the state of the system at time t. The stochastic
process (X(t))t≥0 is a continuous-time Markov chain on S = N0 × N0 . We have to determine the
holding time parameters and the transition matrix of the embedded jump chain.
(A) (B)
Let Ti and Ti be respectively the time that the i-th organism in state A wait for changing to
state B and the time that the i-th organism in state B wait for dividing into two cells of type A.
(A) (B)
We have Ti ∼ Exp(α) and Ti ∼ Exp(β).
Observe that the only allowed transitions for the continuous-time Markov chain are the following:
• if the current state of the chain is (m, n), with m, n ≥ 1, then
– (m, n) ; (m − 1, n + 1) (transition À), whenever one of the organisms in state A changes
(A) (A)
to state B. The waiting time for this to happen is TA = min{T1 , . . . , Tm } ∼ Exp(mα).
– (m, n) ; (m + 2, n − 1) (transition Á), when one of the organisms in state B divides into
(B) (B)
two cells of type A. The waiting time for this to happen is TB = min{T1 , . . . , Tm } ∼
Exp(nβ).
• if the current state of the chain is (0, n), with n ≥ 1, then (0, n) ; (2, n − 1) (transition Á),
when one of the organisms in state B divides into two cells of type A. The waiting time for
(B) (B)
this to happen is TB = min{T1 , . . . , Tm } ∼ Exp(nβ).
• if the current state of the chain is (m, 0), with m ≥ 1, then (m, 0) ; (m − 1, 1) (transition À),
whenever one of the organisms in state A changes to state B. The waiting time for this to
(A) (A)
happen is TA = min{T1 , . . . , Tm } ∼ Exp(mα).
Therefore we can characterize the continuous-time Markov chain (X(t))t≥0 as follows.
Holding time parameters: if m, n ≥ 1, we obtain
(the waiting time for a transition of type Á to occur
v(0,n) = nβ is TB ∼ Exp(nβ))

(the waiting time for a transition (either of type À or

v(m,n) = mα + nβ of type Á) to occur is min{TA , TB } ∼ Exp(mα + nβ))

(the waiting time for a transition of type À to occur

v(m,0) = mα is TA ∼ Exp(mα))

Jump chain transition matrix (non-zero entries): if m, n ≥ 1, we obtain

P(0,n),(2,n−1) = 1
mα
P(m,n),(m−1,n+1) = P (TA < TB ) =
mα + nβ
nβ
P(m,n),(m+2,n−1) = P (TB < TA ) =
mα + nβ

P(m,0),(m−1,1) = 1.

P16.3 (A single-server queuing system – M/M/1). Suppose that customers arrive at a

single-server service station in accordance with a Poisson process having rate λ. Upon arrival, each
customer goes directly into service if the server is free; if not, then the customer joins the queue
(that is, he waits in line). When the server finishes serving a customer, the customer leaves the
system and the next customer in line, if there are any waiting, enters the service. The successive
service times are assumed to be independent exponential random variables with parameter µ. De-
fine an appropriate birth and death process for this model.
Solution. We denote by

X(t) = number of persons in the system at time t

the state of the system at time t. The state space is S = N0 . We have to determine the arrival and
departure rates relative to each state. Arrivals are described by a Poisson process with intensity
λ; thus, λ is the arrival rate. Since there is only one server, the departure rate coincides with the
parameter of the exponential distribution modeling the service time, that is µ. Therefore, (X(t))t≥0
is a birth and death process with rates

λn = λ, n≥0
µn = µ, n ≥ 1.

P16.4 (A multi-server queuing system – M/M/k). Consider a queuing system in which there
are k servers available, each serving at rate µ. Suppose that customers arrive in accordance with
a Poisson process having rate λ and that an entering customer first waits in line and then goes to
the first free server. Define an appropriate birth and death process for this model.

Solution. We denote by

X(t) = number of persons in the system at time t

the state of the system at time t. The state space is S = N0 . We have to determine the arrival and
departure rates relative to each state. Arrivals are described by a Poisson process with intensity λ;
thus, λ is the arrival rate. Since there are k servers, the departure rate will be the parameter of
an exponential random variable obtained as the minimum of the service times of the busy servers.
Now, if there are n customers in the system, with n ≤ k, then n servers will be busy. Since each of
these servers works at rate µ, the total departure rate will be nµ. On the other hand, if there are
n customers in the system, with n > k, then all k of the servers will be busy, and thus the total
departure rate will be kµ. Therefore, (X(t))t≥0 is a birth and death process with rates

λn = λ, n≥0

nµ, 1≤n≤k
µn =
kµ, n > k.

—•—
The preceding are known as the M/M/? queuing systems. The first M refers to the fact that the
arrival process is Markovian (since it is a Poisson process) and the second to the fact that the
service distribution is exponential (and, hence, Markovian). The ? refers to the number of available
servers.
—•—

P16.5 (á á very difficult). The birth and death process with parameters λn ≡ 0, for all n ≥ 0,
and µn = µ, for all n > 0, is called a pure death process. Find the transition probability func-
tions Pij (t).

Solution. Since the death rate is constant it follows that, as long as the system is non-empty,
departures occur according to a Poisson process (N (t))t≥0 with rate µ. Hence, we obtain

(µt)j−i
Pij (t) = P (X(t) = j|X(0) = i) = P (N (t) = j − i) = e−µt , if j ≤ i,
(j − i)!
X (µt)k
Pi0 (t) = P (X(t) = 0|X(0) = i) = P (N (t) ≥ i) = e−µt , if i ≥ 0.
k!
k≥i
P16.6. Consider two machines that are maintained by a single repairman. Machine i functions
for an exponential time with rate µi before breaking down (with i = 1, 2). The repair times (for
either machine) are exponential with rate µ. We are interested in time-evolution of the number of
working machines in the system. Can we analyze this as a birth and death process? If so, what are
the parameters? If not, how can we analyze it?

Solution. We cannot model the system as a birth and death process, since the information about
the number of working machines in general does not suffice to determine the birth and death rates
of the process. We need to know also which machine is working.
We define an appropriate continuous-time Markov chain for this model. We consider the following
states for the system

b ←→ both machine are working

1 ←→ machine 1 is working and machine 2 is under repair
2 ←→ machine 2 is working and machine 1 is under repair
01 ←→ both machines are down and machine 1 is under repair
02 ←→ both machines are down and machine 2 is under repair.

Let Ti be the time machine i is functioning (with i = 1, 2) and let T be the repair time. We have
Ti ∼ Exp(µi ) and T ∼ Exp(µ).
We characterize the continuous-time Markov chain (X(t))t≥0 on the state space S = {b, 1, 2, 01 , 02 }
as follows.
Holding time parameters:

(the time until one of the two machines breaks down is

vb = µ1 + µ2 min{T1 , T2 } ∼ Exp(µ1 + µ2 ))

(the time spent by the chain in state 1 is the waiting time for
v1 = µ1 + µ either machine 1 breaking down or machine 2 being repaired
and it is distributed as min{T1 , T } ∼ Exp(µ1 + µ))

(the time spent by the chain in state 1 is the waiting time for
v2 = µ2 + µ either machine 2 breaking down or machine 1 being repaired
and it is distributed as min{T2 , T } ∼ Exp(µ2 + µ))

v01 = v02 = µ (the time needed for repairing a machine is T ∼ Exp(µ))

Jump chain transition matrix (non-zero entries):

µ2 µ1
Pb1 = P (T2 < T1 ) = Pb2 = P (T1 < T2 ) =
µ1 + µ2 µ1 + µ2
µ µ1
P1b = P (T < T1 ) = P102 = P (T1 < T ) =
µ1 + µ µ1 + µ
µ µ2
P2b = P (T < T2 ) = P201 = P (T2 < T ) =
µ2 + µ µ2 + µ

P01 1 = 1 P02 2 = 1.

P16.7. Consider two machines, both of which have an exponential lifetime with mean λ1 . There is
a single repairman that can service machines at an exponential rate µ. We are interested in time-
evolution of the number of working machines in the system. Define an appropriate continuous-time
Markov chain for this model and determine the limiting probabilities.

Solution. We denote by

X(t) = number of working machines at time t

the state of the system at time t. The stochastic process (X(t))t≥0 is a continuous-time Markov
chain on S = {0, 1, 2}. We have to determine the holding time parameters and the transition matrix
of the embedded jump chain. We obtain the following characterization.
Holding time parameters:

v0 = µ (the time needed to repair a machine is Exp(µ))

(the chain stays in state 1 until either the broken machine is repaired or the working
v1 = λ + µ machine breaks; the waiting time for this to happen is the minimum between two
independent exponential random variables with parameters λ and µ)

(the chain stays in state 2 until one of the two machine breaks; the waiting time
v2 = 2λ for this to happen is the minimum between two independent exponential random
variables both with parameter λ)

Jump chain transition matrix (non-zero entries):

λ
P01 = 1 P10 = P (Exp(λ) < Exp(µ)) =
λ+µ
µ
P12 = P (Exp(µ) < Exp(λ)) = P21 = 1,
λ+µ
that is  
0 1 0
 λ µ 
P =  λ+µ 0 λ+µ 
.
0 1 0
To determine the limiting probabilities for the continuous-time Markov chain (X(t))t≥0 , we find
first the stationary distribution(s) of the embedded chain and then we apply the theorem seen in
lecture 39 to get the stationary distribution for the continuous-time process.
Observe that the jump chain is irreducible (0 → 1 → 2 → 1 → 0) and, moreover, being the state
space a finite set, it is positive recurrent. Therefore, there exists a unique stationary distribution π̃
for the embedded chain. To obtain it, we solve the system of linear equations

λ

 π̃0 = π̃1
λ+µ


( 

π̃ = π̃P

 π̃ = π̃ + π̃
1 0 2
⇐⇒
π̃0 + π̃1 + π̃2 = 1 µ
π̃2 = π̃1






 λ+µ
π̃0 + π̃1 + π̃2 = 1,


 
λ 1 µ P2 π̃i 2λ2 +2λµ+µ2
which gives π̃ = 2(λ+µ) 2 2(λ+µ) . As a consequence, since we calculate i=0 vi = 4λµ(λ+µ) ,
we have
2λ2 µ2
 
2λµ
π= .
2λ2 + 2λµ + µ2 2λ2 + 2λµ + µ2 2λ2 + 2λµ + µ2

P16.8. A small barbershop, operated by a single barber, has room for at most two customers.
Potential customers arrive at a Poisson rate of 3 per hour, and the successive service times are
independent exponential random variables with mean 41 hour. In the long-run, what is the average
number of customers in the shop?

Solution. We denote by

X(t) = number of customers in the shop at time t

the state of the system at time t. The stochastic process (X(t))t≥0 is a continuous-time Markov
chain on S = {0, 1, 2} and, in the long-run, the average number of clients in the barbershop is
π1 + 2π2 (expected value of the state variable, when distributed according to π), where π is a
stationary distribution for the chain. Hence, we need to determine the limiting probabilities for the
Markov chain (X(t))t≥0 .
Method 1. We can proceed in the very same way as we did in problem 7. We characterize the
continuous-time Markov chain (holding time parameters: v0 = 3, v1 = 7 and v2 = 4; jump chain
non-zero transition probabilities: P01 = 1, P10 = 47 , P12 = 73 and P21 = 1), we determine the
 π̃ = 27 21 14 3


(unique) stationary distribution of the (irreducible) embedded chain and finally
16 12 9


we get π = 37 37 37 .
Method 2. We analyze the process for what it is, a birth and death process, and we use directly
the expression for the stationary distribution we derived in class (see lecture 40). In particular,
(X(t))t≥0 is a birth and death process on S with birth rates λ0 = λ1 = 3, as clients arrive according
to a Poisson process with intensity 3, and with death rates µ1 = µ2 = 4, as the service times are
Exp(4). Therefore, we obtain
1 1 16
π0 = λ0 λ0 λ1
= 3 32
=
1+ µ1 + µ1 µ2 1+ 4 + 42
37

λ0 3 12
π1 =  = 3 32

=
µ1 1 + λ0
+ λ0 λ1 4 1+ 4 + 42
37
µ1 µ1 µ2

λ0 λ1 32 9
π2 =  = 32

= .
µ1 µ2 1 + λ0
+ λ0 λ1 42 1 + 34 + 42
37
µ1 µ1 µ2

In conclusion, in the long-run, the average number of customers in the shop is

12 9 30
π1 + 2π2 = +2· = ≈ 0.81.
37 37 37

P16.9 (v From past exam). Each time my printer is repaired, it functions for an exponential
amount of time with parameter λ and then it breaks down. Its failure is either of two types. If it
is a type-1 failure, then the time to repair the printer is exponential with parameter µ1 ; if it is a
type-2 failure, then the time to repair the printer is exponential with parameter µ2 . Each failure is,
independently, a type-1 failure with probability p and a type-2 failure with probability 1 − p, where
p ∈ (0, 1).
(a) Model the state of my printer as a three-state continuous-time Markov chain. Give the holding
time parameters and the transition matrix of the embedded jump chain.
(b) If it exists and it is unique (explain why or why not!), determine the stationary distribution
of the embedded chain. Do limiting probabilities for the jump chain exist?
(c) Determine the stationary distribution of the continuous-time Markov chain. In the long-run,
what is the proportion of time that my printer is functioning?
Solution.
(a) Let X(t) =“state of my printer at time t” be the state of the system at time t. We consider
the following states for the system: 0 ←→ the printer is functioning; 1 ←→ the printer is
under repair due to a type-1 failure and 2 ←→ the printer is under repair due to a type-2
failure. We can characterize the continuous-time Markov chain (X(t))t≥0 on the state space
S = {0, 1, 2} as follows.

Holding time parameters:

(the functioning time is an exponential random variable with

v0 = λ parameter λ)

(the repair time for a type-1 failure is an exponential random

v1 = µ1 variable with parameter µ1 )

(the repair time for a type-2 failure is an exponential random

v2 = µ2 variable with parameter µ2 )

Jump chain transition matrix (non-zero entries):

P01 = p P02 = 1 − p P10 = 1 P20 = 1,

 that is  
0 p 1−p
P = 1 0 0 .
 

1 0 0

(b) From the transition matrix P we deduce 1−p

the transition graph shown in the figure p
aside. All states communicate with each ( $
other (0 → 1 → 0 → 2 → 0), therefore •0 dh •1 •2
the jump chain has only one communication 1
class.
1

The embedded chain is irreducible. Moreover, as the state space is a finite set, it is positive
recurrent. As a consequence, we have existence and uniqueness of the stationary distribution
for this chain. To determine the stationary distribution π̃, we solve the following system of
linear equations (
π̃ = π̃ P
P2
i=0 π̃i = 1,

which is equivalent to

 π̃0 = π̃1 + π̃2 
1
 π̃0 =

 

 π̃ = p π̃  2
1 0 p
⇐⇒ π̃1 = 2

 π̃ 2 = (1 − p) π̃0 

 π̃ = 1−p
2 .

 2
π̃0 + π̃1 + π̃2 = 1


The limiting probabilities for the embedded jump chain do not exist, as it is a periodic chain
with period d = 2 (an even number of steps is needed to reenter any given state).
(c) We get the stationary distribution π of the continuous-time process via the stationary distri-
bution π̃ of the discrete-time embedded Markov chain (see lecture 39). Since we calculate
2
X π̃i 1 p 1−p µ1 µ2 + λµ2 p + λµ1 (1 − p)
= + + = ,
i=0
vi 2λ 2µ1 2µ2 2λµ1 µ2

we obtain
1
2λ µ1 µ2
π0 = =
µ1 µ2 + λµ2 p + λµ1 (1 − p) µ1 µ2 + λµ2 p + λµ1 (1 − p)
2λµ1 µ2
p
2µ1 λµ2 p
π1 = =
µ1 µ2 + λµ2 p + λµ1 (1 − p) µ1 µ2 + λµ2 p + λµ1 (1 − p)
2λµ1 µ2
1−p
2µ2 λµ1 (1 − p)
π2 = = .
µ1 µ2 + λµ2 p + λµ1 (1 − p) µ1 µ2 + λµ2 p + λµ1 (1 − p)
2λµ1 µ2
As a consequence, in the long-run, the proportion of time that my printer is functioning is
π0 = µ1 µ2 +λµµ21p+λµ
µ2
1 (1−p)
.

P16.10 (v From past exam). A virus can be in one of three different states: it can be active
and exist in strain A or in strain B, or it can be quiescent. The virus mutates its strain from A
to B, or vice versa from B to A, at an exponential rate 4. If the virus is in strain B, it becomes
quiescent at an exponential rate 2. If the virus is quiescent, it exits from quiescence in strain A at
an exponential rate 1 and in strain B at an exponential rate 2.
 (a) Model the state of the virus as a three-state continuous-time Markov chain. Give the holding
time parameters and the transition matrix of the embedded jump chain.
(b) If it exists and it is unique (explain why or why not!), determine the stationary distribution
of the embedded chain. Do limiting probabilities for the jump chain exist?
(c) Determine the stationary distribution of the continuous-time Markov chain. In the long-run,
what is the proportion of time that the virus is active?
Solution.
(a) Let X(t) =“state of the virus at time t” be the state of the system at time t. We consider the
following states for the system: A ←→ the virus is active and exists in strain A; B ←→ the
virus is active and exists in strain B and Q ←→ the virus is quiescent. We can characterize
the continuous-time Markov chain (X(t))t≥0 on the state space S = {A, B, Q} as follows.

Holding time parameters:

(the waiting time for the virus to mutate from strain A to
vA = 4 strain B is an exponential random variable with parameter 4)

(the waiting time for either the virus to mutate from strain B
to strain A or to become quiescent is the minimum between
vB = 6 two independent exponential random variables with respective
parameters 4 and 2)

(the waiting time the virus to exit quiescence is the minimum

vQ = 3 between two independent exponential random variables with
respective parameters 1 and 2)

Jump chain transition matrix (non-zero entries):

2 1
PAB = 1 PBA = P (Exp(4) < Exp(2)) = PBQ = P (Exp(2) < Exp(4)) =
3 3
1 2
PQA = P (Exp(1) < Exp(2)) = PQB = P (Exp(2) < Exp(1)) = ,
3 3
that is  
0 1 0
P =  23 0 1
3 .

1 2
3 3 0

1
(b) From the transition matrix P we deduce *
•A] j •K B
the transition graph shown in the figure
aside. All states communicate with each 2
3 1
3
other (A → B → Q → A), therefore the
1
jump chain has only one communication 3
2
3
class.
•Q
The embedded chain is irreducible. Moreover, as the state space is a finite set, it is positive
recurrent. As a consequence, we have existence and uniqueness of the stationary distribution
for this chain. To determine the stationary distribution π̃, we solve the following system of
linear equations (
π̃ = π̃ P
P
i∈S π̃i = 1,
which is equivalent to

 π̃A = 32 π̃B + 13 π̃Q 
7
 π̃A =

 
 π̃ = π̃ + 2 π̃
  19
B A 3 Q 9
1
⇐⇒ π̃B = 19
 π̃
 Q = π̃
3 B


 π̃ = 3
19 .

 Q
π̃A + π̃B + π̃Q = 1

 We check whether or not the embedded chain is aperiodic. Being periodicity a class prop-
2
erty, it suffices to analyze one single state. Let us focus on state A. Observe that PAA >0
3
(A → B → A) and PAA > 0 (A → B → Q → A). Therefore, since gcd(2, 3) = 1, we obtain
dA = 1 and the chain is aperiodic.
Since the chain is positive recurrent, irreducible and aperiodic, the convergence theorem
holds. Therefore,
the limiting probabilities are the entries of the stationary distribution
π̃ = π̃A π̃B π̃Q , that is

lim Pijn = π̃j for all i, j ∈ S.

n→+∞

(c) We get the stationary distribution π of the continuous-time process via the stationary dis-
tribution π̃ of the discrete-time embedded Markov chain (see lecture 39). Since we calculate
P π̃i 17
i∈S vi = 76 , we obtain

7 3 1
76 7 38 6 19 4
πA = 17 = , πB = 17 = and πQ = 17 = .
76
17 76
17 76
17

As a consequence, in the long-run, the proportion of time that the virus is active is
7 6 13
πA + πB = + = .
17 17 17