Markov Process: Properties, Analysis and

Applications
AJAY KUMAR
Assistant Professor
Atal Bihari VajpayeeIndian Institute of Information Technology & Management, Gwalior
Morena Link Road, Gwalior-474010
MP, India

March 10, 2010

1 / 111

Stochastic Process: A stochastic process is a collection of

random variables, {X (t) : t T }, where X (t) is a random variable
for each t T .

The set T is called the index set of the process and t T is

called and index or parameter.

The values taken by X (t) are called as states of the

stochastic process, corresponding to the index t.

If T is countable, the stochastic process is said to be discrete

time process

If T in an interval, the stochastic process is called a

continuous time process

2 / 111

Chain: If the state space S of a stochastic process

{X (t) : t T } is countable, the process is said to be a chain.
Based on the countable or continuous nature of the index set
and the space state, we have the following types of stochastic
processes:
I
I
I
I

Discrete time, discrete state space processes

Discrete time, continuous state space processes
Continuous time, discrete state space processes
Continuous time, continuous state space processes

3 / 111

Example: Consider a single machine prone to failures and let us

examine its states at discrete points of time, t0 , t1 , t2 , . Let us
define its state as 0, if the machine is up and 1, if machine is
down. The machine will keep changing its state with time. Let, Xi
denote the state of the machine at time ti , i = 0, 1, 2, , then
Xi : i N is a stochastic process.
I

Here Xi is a discrete random variable with range {0, 1}, so the

state space is S = {0, 1}.

Index set T = {t0 , t1 , t2 , }

The stochastic process, defined above is a discrete time chain

If the state of the machine is examined at any instant of time,

then the index set will be an interval [0, ] and the stochastic
process X (t) : t 0 will be a continuous time chain.

4 / 111

Markov Process: Markov process (or chain) is a stochastic

process which has the property that the probability of transition
from a given state to any future state depends only on the present
state and not on the manner in which it was reached. OR
Mathematically, a stochastic process is called a Markov process, if
the occurrence of the future state depends on the immediately
preceding state and only on it. Thus, if t0 , t1 , t2 , , represents the
points in time scale, then the family of random variables {X (tn )}
is said to be a Markov process if it holds the following property:
P {X (tn ) = xn |X (tn1 ) = xn1 , X (tn2 ) = xn2 , , X (t0 ) = x0 }
= P {X (tn ) = xn |X (tn1 ) = xn1 } , X (t0 ), X (t1 ), X (t2 ),

5 / 111

Transition Probability: The probability of moving from one state

to another is said to be transition probability. Mathematically,
pij = P{X (tn ) = j|X (tn1 = i)}
is called transition probability.
Transition Probability Matrix (TPM): The transition
probabilities can be arranged in a matrix form. Such a matrix is
said to be one step-transition probability matrix. It is denoted by P
and is given by:

p11 p12 p1m

p21 p22 p2m

P= .
..
..
..
..
.
.
.
pm1 pm2

pmm

6 / 111

Properties of TPM:
I

P is a square matrix

0 pij1 , i, j

Sum along each row is unity, i.e.,

m
P
j=1

pij = 1

7 / 111

Discrete Time Markov Chain (DTMC): A discrete time Markov

chain is a stochastic process, {Xn : n N}, with countable state
space S, such that for all n N, and for all
xi S, i = 0, 1, 2, , n:
P {X (tn ) = xn |X (tn1 ) = xn1 , X (tn2 ) = xn2 , , X (t0 ) = x0 }
= P {X (tn ) = xn |X (tn1 ) = xn1 }
Which implies, that given the current state of the system, the
future is independent of the past.

8 / 111

Let, P[sj (0)] be the probability that the system be in state sj at

time t = t0 , and
pij = P{X (tn ) = j|X (tn1 ) = i}
is a transition probability from state si to sj .
matrix,

p01 p02 p0m

p11 p12 p1m
P=
..
..
..
..
.
.
.
.

Then, the stochastic

..
.

called as TPM, along with P[sj (0)], completely defines the DTMC.

9 / 111

Question: What is the probability that the system will occupy

state sj after n transitions, if its state at n = 0, is known.
Let P[sj (n + 1)] = the probability that the system is in state sj at
time t = tn+1 , Then:
P[sj (n + 1)] = P[s1 (n)]p1j + P[s2 (n)]p2j + + P[sm (n)]pmj
Now, put j = 1, 2, , m, we get:
[P[s1 (n + 1)], P[s2 (n + 1)], , P[sm (n + 1)]]

p11 p12
p21 p22

= [[P[s1 (n)], P[s2 (n)], , P[sm (n)]] ..

..
.
.

..
.

p1m
p2m
..
.

pm1 pm2

pmm

10 / 111

The above system can be written as:

P(n + 1) = P(n) P
where, P(n + 1) is a row vector. Now,
P(1) = P(0) P
P(2) = P(1) P = P(0) P 2
P(3) = P(2) P = P(0) P 3

P(k) = P(k 1) P = P(0) P k

11 / 111

Question: Gwalior city is divided into three zones A, B and C.

From the records of drivers, the following information is available.
Out of passengers picked up in zone A (B) (C) 40% (30%) (30%)
go to a destination in zone A, 40% (40%) (50%) are taken to a
destination in zone B, 20% (30%) (20%) go to a destination in
zone C. Suppose in the beginning of the day 30% of the taxis are
in zone A, 30% are in zone B and 40% are in zone C. What is the
distribution of taxis after all have go one ride and two rides. Also
determine the long run distribution of taxis.
Answer:
P(1) = (0.33, 0.44, 0.23)
P(2) = (0.333, 0.423, 0.244)
= (11/33, 14/33, 8/33)

12 / 111

Limiting State Probabilities: As, if n , P(n) will be

constant. In this situation limiting state probabilities exist and are
independent of the initial conditions. The process in this case is
said to be ergodic process.
Let lim P(n) = , then:
n

lim P(n + 1) =

lim P(n) P

= P
Where, is a 1 m vector.

13 / 111

Transient State: If the system can leave the state but never
return to it, e.g., s1

Trapping or Absorbing State: If the system can enter the

state and cannot leave it, e.g., s2

Recurrent Chain: A recurrent chain is a collection of all

trapping states.
0.25

s1

s2

0.75
1

14 / 111

Continuous Time Markov Chain (CTMC): A discrete state

space, continuous time process {X (t) : t 0} with state space S
is called CTMC, if the following Markov or memoryless property is
satisfied, for all s 0, u 0, t s and i, j, x(u) S,
P {X (t) = j|X (s) = i, X (u) = x(u), for 0 u < s}
= P {X (t) = j|X (s) = i}
Transition Rate: aij is the transition rate of a process from state i
to state j, i 6= j.
Transition Rate Matrix: A = [aij ] is the transition rate matrix.
Transition Probability: Given that the present state is si , the
conditional probability that si sj , during time interval 4t is
aij 4t. The unit of aij is sec1 . So aij1 = expected time between
transitions
15 / 111

Question: Let P[si (t)]=probability that the system occupies state

i at time t. Find out the probability of being in state j at time
t + 4t.
Note that:
P[sj (t + 4t)] = P[sj (t)]P(no transition from j)
+ P[s1 (t)]a1j 4t + P[s2 (t)]a2j 4t
+ + P[sj1 (t)]aj1,j 4t + P[sj+1 (t)]aj+1,j 4t
+ + P[sm (t)]amj 4t
Next, the probability of marking no transition from j=1-probability
of making transition from j
=1

m
X

aji 4t

i6=j

16 / 111

Substituting,

P[sj (t + 4t)] = P[sj (t)] 1

m
X

aji 4t +

m
X

i6=j

P[si (t)]aij 4t

i6=j

j = 1, 2, 3, , m
Now define the diagonal elements of the transition rate matrix A
by:
m
X
ajj =
aji
i6=j

P[sj (t + 4t)] = P[sj (t)](1 + ajj 4t) +

m
X

P[si (t)]aij 4t

i6=j

or
P[sj (t + 4t)] P[sj (t)] =

m
X

P[si (t)]aij 4t

i=1
17 / 111

Dividing by 4t and taking limit 4t 0, leads to:

m

X
d
P[sj (t)] =
P[si (t)]aij ; j = 1, 2, , m
dt
i=1

Next, define:
P(t) = [P[s1 (t)], P[s2 (t)], , P[sm (t)]]
Which results in:

d
P(t) = P(t) A
dt

Which has the solution:

P(t) = P(0)e At

18 / 111

Steady State Solution: As time t , P(t) will be constant

and so from:
d
P(t) = P(t) A
dt
we have:
0 = P() A = A
= rate of leaving the state=rate of entering in it

19 / 111

Example: Consider a 2-component system, which has following

four states:
State
1
2
3
4

Component-1
operating
failed
operating
failed

Component-2
operating
operating
failed
failed

Assumptions:
I

Each component will be in one of the two states, operating or

failed

The failures among the components are independent

Initially the system is in working state

The failure behavior is exponential

20 / 111

If the two components are in parallel configuration, only state

4 results in system failure, so reliability is given by:
Rp (t) = R1 (t) + R2 (t) + R3 (t)

If the two components are in series configuration, states 2, 3,

and 4, result is system failure, so reliability is given by:
Rs (t) = R1 (t)

The system must be in one of the given four states, so

P1 (t) + P2 (t) + P3 (t) + P4 (t) = 1

The failures rates are 1 and 2 , respectively.

21 / 111

The transition diagram is given below:

dP1 (t)
+ (1 + 2 )P1 (t)
dt
dP2 (t)
+ 2 P2 (t)
dt
dP3 (t)
+ 1 P3 (t)
dt
dP4 (t)
dt

= 0
= 1 P1 (t)
= 2 P1 (t)
= 1 P3 (t) + 2 P2 (t)
22 / 111

The probabilities are given by:

P1 (t) = e (1 +2 )t
P2 (t) = e 2 t e (1 +2 )t
P3 (t) = e 1 t e (1 +2 )t
P4 (t) = 1 P1 (t) P2 (t) P3 (t)

Reliability for series system:

Rs (t) = P1 (t) = e (1 +2 )t

Reliability for parallel system:

Rp (t) = R1 (t) + R2 (t) + R3 (t)
= e 1 t + e 2 t e (1 +2 )t
23 / 111

Load-Sharing System: A system has two components, working in

parallel configuration. There is a dependency between the
components. If one component fails, the failure rate of the other
component increases as a result of additional load, placed on it.
+
Assume that, +
1 and 2 are the increased failure rates of
component-1 and component-2, respectively.

24 / 111

The transition diagram is shown below:

dP1 (t)
+ (1 + 2 )P1 (t) = 0
dt
dP2 (t)
+ +
2 P2 (t) = 1 P1 (t)
dt
dP3 (t)
+ +
1 P3 (t) = 2 P1 (t)
dt

25 / 111

The solution of the system is given below:

P1 (t) = e (1 +2 )t
1
P2 (t) =
1 + 2 +
2
2
P3 (t) =
1 + 2 +
1

h
i
+
( + )t
e 2 te 1 2
h
i
+
(1 +2 )t
e 1 te

The reliability is given by:

R(t) = P1 (t) + P2 (t) + P3 (t)
MTTF can be calculated as:
Z
MTTF =

R(t)dt
0

26 / 111

Standby Systems: Two components, A & B, are working in

standby redundancy mode, i.e., if A fails, B starts functioning.
Such systems are said to be standby systems.
The failure rate of standby unit will depend on the state of primary
(on-line) unit. We may consider:
I

The standby unit will have no failure or reduced failure rate,

while in its standby mode
While, the standby unit active, two cases arise:
I

it may experience the same failure rate as the online system, if

they are identical
it may experience different failure rate, if they are not identical

The problem is to model the system and to find out its:

(a) reliability, and (b) MTTF

27 / 111

The transition diagram is shown below:

dP1 (t)
+ (1 +
2 )P1 (t) = 0
dt
dP2 (t)
+ 2 P2 (t) = 1 P1 (t)
dt
dP3 (t)
+ 1 P3 (t) =
2 P1 (t)
dt

28 / 111

The solution of the system is given below:

P1 (t) = e (1 +2 )t

1
2 te (1 +2 )t
P2 (t) =
e
1 +
2 2

P3 (t) = e 1 t e (1 +2 )t
The reliability is given by:
R(t) = P1 (t) + P2 (t) + P3 (t)

1
1 t
2 te (1 +2 )t
= e
+
e
1 +
2 2
and
MTTF =

1
1
+
1 2 (1 +
2)

29 / 111

Degraded Systems: Some systems may continue to operate in a

degraded state.
Define the states as fully operational (state 1), degraded (state 2)
and failed (state 3). The transition diagram in this case is shown
below:

30 / 111

The system of ODEs can be generated as:

dP1 (t)
+ (1 + 2 )P1 (t) = 0
dt
dP2 (t)
+ 3 P2 (t) = 2 P1 (t)
dt
Solution:
P1 (t) = e (1 +2 )
h
i
2
P2 (t) =
e 3 t e (1 +2 )t
1 + 2 3
and
MTTF =

1
2
1
1
+

1 + 2 1 + 2 3 3 1 + 2

31 / 111

Availability: Availability is the probability that a system or

component is performing its required function at a given point in
time when operated and maintained in a prescribed manner.
Mathematically,
Availability =
=

uptime
uptime+downtime
MTBF
MTBF+MTTR

Note: Repairs are not considered while observing reliability, but

they are taken into account while calculating availability.

32 / 111

Calculate point availability and steady state availability for:

I

A system having one component

A system has two components, working in parallel

33 / 111

Example: Consider the CTMC, which has three states

0: set up, 1: Processing, and 2: Down
The transition diagram is given below:
r

0
s

2
f

Find out the ODEs for the above transition diagram and find out:
I

Transient/Instantaneous state probabilities

Steady state probabilities

34 / 111

For steady state, the values of 0 , 1 and 2 are given by:

0 =

rs
fs
pr
, 1 =
, 2 =
pr + rs + fs
pr + rs + fs
pr + rs + fs

Moreover, if s = 20 per hour, p = 4 per hour, f = 0.05 per hour,

and r = 1 per hour, then:
0 = 0.16, 1 = 0.80, and 2 = 0.04

35 / 111

Example: Consider a system with two machines, M1 and M2 ,

working in parallel configuration. Machine M1 is fast and M2 is a
slow machine. Raw parts are always available and have to undergo
one operation each, which may be carried out either on M1 or on
M2 . Each machine will immediately start processing, after finishing
the processing of one part. Each machine can fail randomly, and
after possible repair, it will resume processing.
The objective is to find CTMC for the above system, and to
analyze the system performance, with:
I

no repair facility

single repair facility

two identical repair facilities

36 / 111

The system can be modeled using CTMC with four states given
by:
I

0: M1 and M2 both working

1: M1 working and M2 down

2: M1 down and M2 working

3: M1 and M2 both down

Solution:
I

If no repair facility, state 3 will be absorbing state and 0, 1,

and 2 are transient states.
1
f

f
3

0
f

f
2

37 / 111

For one repair facility, assume that in state 3, where both the
machines are down, the repair facility will spend its time
equally between M1 and M2 .
r
1
f

r/2
f
3

0
f

f
2
r

r/2

The steady-state probabilities are given by:

0 =
1 = 2 =
3 =

r2
+ 2fr + 2f 2
fr
2
r + 2fr + 2f 2
2f 2
r 2 + 2fr + 2f 2
r2

38 / 111

When there are two identical repair facilities, the CTMC

model will be similar to that in last figure, except the labels
on the directed arcs from state 3 to state 2, and from state 3
to state 1.

The steady-state probabilities are given by:

0 =
1 = 2 =
3 =

r2
r 2 + 2fr + f 2
fr
2
r + 2fr + f 2
2f 2
r 2 + 2fr + f 2

39 / 111

Computational Techniques for CTMC

I

Transient or Instantaneous Analysis

I

Solution of system of linear ODEs

Steady State Analysis

I

Solution of A = 0

40 / 111

Methods for Steady-State Analysis

I
I
I
I

Power method
SOR
Gaussian elimination Method
Gauss-Seidel Method

Methods for Transient Analysis

I
I
I

Fully Symbolic Method (Laplas Transformation)

Eulers Method
Runge-Kutta forth order Method

41 / 111

Methods for Steady State

Analysis

42 / 111

Power Method: The problem is to solve A = 0

I

Step-1: choose, a max |aii |

Step-2: The new system will be:

A
= I+
a

Note A = I + Aa is a transition probability matrix of a DTMC.

I

Do iteration: (i) = (i1) A

Initial value: (0) = (0)

43 / 111

Example: Consider the following transition rate matrix to solve

A = 0:

2 2c 0 2(1 c)
0
c (1 c)

A=
0

0
0
0
0
0
0
0
Choose a = 2, we get:

0 c

0 21
A =
0 0
0 0

0 1c
c
2

1
0

1c
2

0
1

Now, take (0) = [1, 0, 0, 0] and find the rest.

44 / 111

SOR (Successive Over Relaxation) Method: Let A be a

symmetric matrix then the linear system Ax = b can be solved
using the following steps:
I

Start with an initial guess

Iterate until some criteria for convergence are satisfied:

x (k+1) = (D + L)1 [(1 )D U]x (k) + (D + L)1 b

where, x (k+1) is the solution vector at the k th iteration, L is a

lower triangular matrix, U is an upper triangular matrix, and
D is a diagonal matrix

Gauss-Seidel is a special case of SOR with = 1, i.e.,

x (k+1) = (D + L)1 Ux (k) + (D + L)1 b

45 / 111

Gauss Elimination Method: To solve the system Ax = b

I

find the augmented metrix, [A|b]

choose, ajk = max |aik |, k i n, and interchange rows k

and j

The elements aij

(k)

(k)

(k)

(k+1)

aij

with i, j k are given by:

(k)

= aij

(k)

aik

(k)
a
(k) kj
akk

i = k + 1, k + 2, , n
j = k + 1, k + 2, , n, n + 1
(1)

where, aij = aij

I

this method will always give the exact solution

46 / 111

Example: Solve the

2 1
4 0

3 2
1 3

system of equations:

x1
10
1 2

2 1
x2 = 8

x3
7
2 0
x4
5
2 1

using Gauss elimination method.

2
4
[A|b] =
3
1

4
2
=
3
1

1
0
2
3
0
1
2
3

1 2 10
2 1
8
R R1
2 0
7 2
2 1 5

2 1
8
1 2 10
R 1R , R 3R , R 1R
2 0
7 2 2 1 3 4 1 4 4 1
2 1 5
47 / 111

4
0
=
0
0

4
0
=
0
0

4
0
=
0
0

0 2
1
8
1 0 5/2 14
R R2
2 1/2 3/4
1 4
3 3/2 5/4 7

0 2
1
8
3 3/2 5/4 7
R3 2 R2 , R4 1 R2
2 1/2 3/4
1
3
3
1 0 5/2 14

0
2
1
8
3 3/2
5/4
7
R R3
0 1/2
1/12
17/3 4
0 1/2 25/12 35/3

48 / 111

4
0
=
0
0

0
2
1
8
3 3/2
5/4
7
R R3
0 1/2 1/12
17/3 4
0
0
13/6 52/3

Finally, we get:

4 0
2
1
0 3 3/2
5/4

0 0 1/2 1/12
0 0
0
13/6

x1
8
x2 7

x3 = 17/3
52/3
x4

and so, x4 = 8, x3 = 10, x2 = 6, x1 = 5

49 / 111

Methods for Transient Analysis

50 / 111

Fully Symbolic Method

I

Taking the Laplace transform on both sides of the

Kolmogorov differential equation and rearranging the terms:

P(s)
= P(0)(sI A)1

The transient state probability is obtained by computing the

inverse Laplace transform of P(s)

Solutions obtained will be closed-form and full symbolic in

both the system parameters and time t

51 / 111

Using Matrix Theory: The solution of Kolmogorov differential

equations is:
P(t) = P(0)e At
Now, the problem is to compute e At :
I

Input matrix A

Find out eigenvalues and eigenvectors of matrix A

A = V D V 1 and e A = V e D V 1 , where V is collection

of eigenvectors and D is diagonal matrix whose diagonal
elements are eigenvalues of A
in MATLAB, e A can be directly obtained using:

I
I

Y = expm(A), or
[V,D] = EIG(A);
EXPM(A) = V*diag(exp(diag(D)))/V

52 / 111

Euler Method: Consider the following differential equation:

u 0 = f (t, u), u(0) = u0
then:
uj+1 = uj + hfj , j = 0, 1, 2, , N 1

53 / 111

Example: Solve the ode:

u 0 = 2tu 2 , u(0) = 1
We have:
uj+1 = uj 2htj uj2
with h = 0.2 and u0 = 1
u(0.2) = u1 = 1.0
u(0.4) = u2 = 0.92
u(0.6) = u3 = 0.78458
u(0.8) = u4 = 0.63684
u(1) = u5 = 0.50706

54 / 111

Runge-Kutta Method of Fourth Order (RK4): Consider the

following differential equation:
u 0 = f (t, u), u(0) = u0
Then:

1
ui+1 = ui + [k1 + 2k2 + 2k3 + k4 ]
6

Where:
k1 = hf (ti , ui )

h
k1
k2 = hf ti + , ui +
2
2

h
k2
k3 = hf ti + , ui +
2
2
k4 = hf (ti + h, ui + k3 )

55 / 111

RK4 Method for System of ODEs: Consider the system of n

equations:
dU
= F (t, u1 , u2 , , un )
dt
U(t0 ) = U0
Where, U = [u1 , u2 , , un ]T ,
F = [f1 , f2 , , fn ]T ,
U0 = [u1 (0), u2 (0), , un (0)]

56 / 111

Then the solution is given by:

1
Uj+1 = Uj + (K1 + 2K2 + 2K3 + K4 )
6
Where:

Uj+1 =

u1,j+1
u2,j+1
..
.
un,j+1

Ki =

k1i
k2i
..
.

kni

and:
ki1 = hfi (tj , u1,j , u2,j , , un,j )
1
1
1
h
ki2 = hfi (tj + , u1,j + k11 , u2,j + k21 , , un,j + kn1 )
2
2
2
2
h
1
1
1
ki3 = hfi (tj + , u1,j + k12 , u2,j + k22 , , un,j + kn2 )
2
2
2
2
ki4 = hfi (tj , u1,j + k13 , u2,j + k23 , , un,j + kn3 )
57 / 111

Solve the system of equations:

u 0 = 3u + 2v , u(0) = 0
v 0 = 3u 4v , v (0) = 0.5
with h = 0.2 on the interval [0, 0.4]. Use RK4 method.

58 / 111

Eulers Method for System of ODEs: Consider the system of n

equations:
dU
= F (t, u1 , u2 , , un )
dt
U(t0 ) = U0
Where,
U = [u1 , u2 , , un ]T ,
F = [f1 , f2 , , fn ]T ,
U0 = [u1 (0), u2 (0), , un (0)]
The solution is given by:
Uj+1 = Uj + hUj0 ; j = 0, 1, 2, , N 1

59 / 111

Computational Techniques for DTMC

1: Given a two state Markov Chain, with the transition probability
matrix:

1a
a
; 0 a, b 1, |1 a b| < 1
P=
b
1b
then matrix P n is given by:

Pn =

b+a(1ab)n
a+b
bb(1ab)n
a+b

aa(1ab)n
a+b
a+b(1ab)n
a+b

60 / 111

Assignments
1. Make a computer program to solve a given DTMC having
2-states.
Inputs: TPM, P and initial probability, P(0)
Output: nstep transition probabilities, P(n)
2. Extend the program to m m TPM, using
P(n + 1) = P(0) P n or P(n + 1) = P(n) P

61 / 111

Pulping System: The four major actions carried out in the system
are, (i) cooking of chips, (ii) separation of knots, (iii) washing of
pulp, and (iv) opening of fibers. The pulping system consists of for
subsystems, namely:
I

Digester (A): One unit, used for cooking the chips whose
failure causes the complete failure of the cooking system.

Knotter (B): Two units, one working and other standby,

used to remove the knots from the cooked chips. Whose
complete failure occurs only if both the units fail.

62 / 111

Decker (C): Three units, arranged in series configuration,

used to remove liquor for the cooked chips. Failure of any one
causes the complete failure of the pulping system.

Opener (D): Two units, one working and other standby, used
to separate the fibers. Whose complete failure occurs only if
both the units fail.

The interactions among the various components of pulping system

are shown by RBD in Fig. 1.
B1
C1

A
Digester

D1

B2
Knotter

C2

C3

Deckers
D2
Opener

Figure: Reliability Block Diagram of Pulping System

63 / 111

Figure: CTMC for Pulping System

, , , : Failure rates of digester, knotter, decker and opener,

respectively.
, , , : Repair rates of digester, knotter, decker and opener,
respectively.
64 / 111

Assignments (Contd...)
Make computer programmes to find out steady state probabilities
using:
3. Power Method
4. SOR/Gauss-Seidel Method
5. Gauss Elimination Method

65 / 111

Assignments (Contd...)
Make computer programmes to find out transient probabilities
using:
6. Matrix Method
7. Eulers Method
8. RK4 Method
Also, simulate the system behavior for t = 100 hours.

66 / 111

Birth and Death Process (BD Process): A homogeneous

CTMC {X (t) : t 0}, with state space {0, 1, 2, } is called a
BD process, if there exist constants,
i ; i = 0, 1, 2, , and
i ; i = 1, 2, 3, such that the transition rates are given by:
pi,i+1 = i , i = 0, 1, 2,
pi,i1 = i , i = 1, 2, 3,
pij

= 0, for |i j| > 1

67 / 111

In steady-state,we get:
0 0 = 1 1
(k + k )k

= k1 k1 + k+1 k+1 , k 1

Now,
k k k+1 k+1 = k1 k1 k k
= k2 k2 k1 k1
= k3 k3 k2 k2
=

= 0 0 1 1
= 0

68 / 111

Thus, we have:
k k = k+1 k+1
or
k+1 =

k
k1
k = k =
k1 , for k 1
k+1
k

or

k1

k =

Y i
0 1 k1
0 =
1 2 k
i+1
i=0

69 / 111

Using the fact that

P
j

j = 1, we get:

0 +

k1

k1
Y
i=0

i.e.,

i
=1
i+1

0 =
1+

P k1
Q
k1 i=0

i
i+1

The limiting probabilities will exist provided the series

X k1
Y i
i+1
k1 i=0

converges.
70 / 111

Queues: A queue is a system into which customers arrive to

receive service. When the servers in the system are busy, incoming
customers wait for their turn. Upon completion of a service, the
customer to be served next is selected according to some queuing
discipline.
Queuing discipline: The queuing discipline is a rule for selecting
the customer for the next service among the set waiting customer.
This could be either:
I

First in first out (FIFO),

Last in first out (LIFO),

Service in random order (SIRO),

Shortest processing time (SPT), etc.

71 / 111

A queuing system can be completely described by:

(a). The input or arrival pattern
(b). The service mechanism or service pattern
(c). The number of service channels
(d). Customers behavior
(e). Maximum length of a queue
(f). The size of population

72 / 111

Notation for queues: Generally queuing model may be completely

specified in the following symbolic form: (A/B/C):(D/E/F), where
I

A: probability law for the arrival (or inter-arrival) time

B: probability law according to which the customers are being

served

C: number of service channels (service stations)

D: queue/service discipline

E: capacity of the system, i.e., the maximum number allowed

in the system (in service or waiting)

F: Population size

D. Kendall (1953) introduced the three characteristics (A/B/C).

Later, A. Lee (1966) added fourth, fifth and sixth characteristics to
the notations.

73 / 111

Let C1 , C2 , . . . be any arbitrary arrival stream of customers with:

I

tj = arrival epoch of Cj , (0 t1 t2 . . . < ), and

Tj = tj+1 tj is inter-arrival time between Cj and Cj+1

A customer who has arrived but not yet departed is said to be in

the system. While in the system, a customer may be either in
service or in the queue. Let,
I

Sj = service time of Cj

Dj = delay in queue of Cj , (waiting time in queue)

Wj = Dj + Sj = waiting time in system of Cj

74 / 111

The number of customers in the system at epoch t is given by

N(t), which can be written as
N(t) = Nq (t) + Ns (t)
where Nq (t) = number of customers in the queue at time t, and
Ns (t) = number of customers in the service at time t.
The number of arrivals by time t, nA (t) and number of departures
by time t, nD (t), are given by:
nA (t) = max{j : tj t}
nD (t) = nA (t) N(t)
The process {nA (t) : t 0} is called the arrival process and the
process {nD (t) : t 0} is called the departure process.

75 / 111

Performance Measures: The performance of a queue will be

defined in terms of properties of one or more of the following
stochastic processes: {N(t) : t 0}, {Nq (t) : t 0},
{Ns (t) : t 0}, {Wj : j N}, and {Dj : j N}.
Let us define the following parameters first:
= Arrival rate = limt
1

nA (t)
t

= Average time in service = limn

1
n

n
P
j=1

Sj , where is called

the service rate. We now define generic measures of performance

for a queue. Let, m be the number of identical servers in the
queuing system.

76 / 111

Average number of customers in the system:

1
L = lim
t t

Zt
N(u)du

(1)

Average number of customers in the queue (mean queue

length):
Zt
1
Nq (u)du
(2)
Q = lim
t t
0

Average number of customers in the service:

1
B = lim
t t

Zt
Ns (u)du

(3)

77 / 111

Average server utilization:

U=

B
m

(4)

Average waiting time in the system:

n

1X
Wj
n n

W = lim

(5)

j=1

Average waiting time in the queue:

n

1X
D = lim
Dj
n n

(6)

j=1

The measures L, Q, B and U are averages over time, whereas the

measures W and D are averages over customers. The values on
the right hand side are random variables.
78 / 111

Littles Result: The result states that, given a queue, if the limits:
nA (t)
t t

= lim

n
1 P
Wj
n n j=1

and W = lim

exist, with < and

W < , then the following limit,

1
L = lim
t t

Zt
N(u)du
0

exists, where L < and

L = W

(7)

A consequence of Littles result is that if the limits defining and

D exist, with < , D < , then the limit defining Q exists,
with Q < and Q = D. Note that this enables the result to be
applied to parameters connected with the queue alone.
79 / 111

The M/M/1 Queue: In the M/M/1 queue, the arrivals are

following Poisson distribution with rate , the departures are
following Poisson distribution with rate , and there is only single
server.
Let N(t) be the number of customers in the M/M/1 queue at time
t, then obviously {N(t) : t 0} is a CTMC. Also it is a special
case of Birth and Death Process with:
k = k 0
k = k 1
The ratio =

is called as traffic intensity of the system.

80 / 111

From the last equation of BD process, we have:

1
0 = P

= (1 )

(8)

k0

That is, the probability of an empty system is given by (1 ).

Thus represents the proportion of the time, the server is busy.
This is called the utilization U of the server or traffic intensity of
the queue.
k = k (1 ), k 0
(9)
The probability k is to be described, in the following ways:
I

as the proportion of time in the steady state the system has

exactly k customers,

in the steady state, a randomly arriving customer finds exactly

k customers in the system,

a departing customer leaves behind exactly k customers in the

system.
81 / 111

The mean number of customers in the system, L is found as the

expected value of random variable N, where,
P{N = k} = lim P{N(t) = k} = k
t

Thus we have,
L = E [N] =

kk = (1 )

k=0

But, we have,

kk

(10)

k=0

(1 )2

(11)

=
1

(12)

kk =

k=0

So by (10) and (11), we have,

L=

82 / 111

The mean number of customers, Q in the queue is similarly given

as
Q =

(k 1)k =

k=1

kk

k=1

= L (1 0 ) = L
2
2
=
=
1
( )

k=1

(13)

We can also rewrite as,

L=Q +
Now, W : the mean waiting time in the system or mean response
time, is given by,
1
W =D+
(14)

where,

is the service time.

83 / 111

To find out the explicit formula for W , we use Littles formula,

L = W

(15)

Using (15), we obtain the expressions for W and D for the M/M/1
queue. From (12) and (15), we have,
W =

L
1
1
=
=

(1 )

(16)

D=

1
1

=

( )

(17)

and, from (14)

84 / 111

Example: Consider an NC machine center processing raw parts

one at a time in M/M/1 fashion. Let = 8 parts/h and = 10
parts/h. Then find out:
a. Machine utilization,
b. Mean number of customers in the system,
c. Mean number of customers in the queue,
d. Mean waiting time in the system, and
e. Mean waiting time in queue.
The answers are 0.8, 4.0, 3.2, 0.5 h, and 0.4 h.

85 / 111

Stability of M/M/1 Queue: From (8), we see that:

1
0 = P

k0

We can solve this equation for 0 if and only if the series in the
numerator is convergent. The series will be convergent if and only
if < 1. When 1, the series diverges, which means that the
queue grows without bound and the values of L, Q, W and D
would go to infinity. We say that the queue is stable if < 1 and it
is unstable if 1. The stability condition physically means that
the arrival rate must be strictly less than the service rate.
When is close to 1, queues become very long and as 1, the
number in the system grows without bound. Transient analysis
reveals system behavior as 1 or short term behavior when
> 1.
86 / 111

Waiting Time Distributions: We have seen that W and D are

the mean values of the random variables w and d, whose
distributions can be defined by:
n

Fw (t) =

1X
P{Wj t}
n n
lim

j=1

Fd (t) =

lim

n
1X

P{Dj t}

j=1

Now we will calculate Fw (t) and Fd (t), assuming that the

customers are served in FIFO (or FCFS) order.

87 / 111

First note that d is a continuous random variable except that there

is a nonzero probability, 0 that d = 0 and the customers enters
service immediately on arrival. We have:
Fd (0) = P{d 0} = P{d = 0}
= P{System is empty on arrival}
So,
Fd (0) = 0 = 1

(18)

88 / 111

Now we will find Fd (t) for t > 0. If there are k customers in the
system upon arrival, for the customer to get into the system
between 0 and t, all k units must be served by the time t. Since
the service time distribution is memoryless, the distribution of the
time required for k service times is independent of the time of the
current arrival, and is the convolution of k exponential random
variables, which is an Erlang-k distribution. The probability density
function of the Erlang distribution is:
f (x; k, ) =

(x)k1 e x
(k 1)!

for x, 0

The parameter k is called the shape parameter and the parameter

is called the rate parameter.

89 / 111

The probability that an arrival finds k units in the system in the

steady state is simply the probability k . Thus we have:
Fd (t) = P(d t)

X
= Fd (0) +
P(0 < d t|k units in the system)k
k=1

P(0 < d t|k units in the system) = P(sum of k service times t)

Zt
(x)k1 e x
=
dx
(k 1)!
0

90 / 111

Therefore, we get for t > 0,

Fd (t) = (1 ) + (1 )

X
k=1

Zt
k

(x)k1 e x
dx
(k 1)!

Zt
e x

= (1 ) + (1 )
0

k=1

(x)k1
dx
(k 1)!

Zt
e x(1) dx

= (1 ) + (1 )

(19)

91 / 111

Simplifying (19) we get:

Fd (t) = 1 e (1)t for t > 0
The distribution for waiting time in queue is then:
(
1 ,
for t = 0
Fd (t) =
1 e (1)t for t > 0

(20)

From (20), we can compute the mean waiting time in queue as:
D = E [D] =

( )

(21)

92 / 111

Let us now compute the distribution of w , following the similar

arrangements. We have for t 0,
Fw (t) =

X
P(w t|k units in the system)k
k=0

P(sum of (k + 1) service times t)k

k=0

Z
X
(x)k e x
=
dx k (1 )
k!
t

k=0

93 / 111

Zt
e x

Fw (t) = (1 )

X
(x)k
k=0

k!

dx

Zt
= (1 )
= 1e

e x(1) dx

0
(1)t

= 1 e ()t

94 / 111

Thus we have:
(
0,
for t < 0
Fw (t) =
()t
1e
, for t 0.

(22)

Therefore, w is exponentially distributed with mean given by:

W =

It is noted that the mean values D and W are independent of the

queuing discipline, whereas the distributions of d and w depend on
the queuing discipline.

95 / 111

The M/M/1/N Queue: In an M/M/1/N queueing system, the

maximum number of jobs in the system is N, which implies a
maximum queue length of N 1. An arriving job enters the queue
if it finds fewer than N jobs in the system and is lost otherwise.
This behavior can be modeled by a birth-death process with:
(
for k = 0, 1, 2, . . . , N 1
k =
0 for k N
The service rate k = for all k 1. The CTMC here is a finite
birth and death process, with N + 1 states.

96 / 111

The steady state probabilities are given by:

k

(1 )k
, k = 0, 1, 2, . . . , N
1 N+1
= 0, k > N
=

(23)

Note that a system with finite population such as above will always
be stable for all values of .
The performance measures can be verified to be given by:
U = 1 0 =

(1 N )
1 N+1

(24)

97 / 111

Q=

2 [1 N NN1 (1 )]
(1 )(1 N+1 )

(25)

[1 N NN (1 )]
(1 )(1 N+1 )

(26)

1 N NN (1 )
(1 )(1 N )

(27)

[1 N NN1 (1 )]
(1 )(1 N )

(28)

L=

W =

D=

98 / 111

Example: It would be interesting to study the performance

measures of a single-server queuing system for various queue
capacities. First, consider the M/M/1 queue of Example 1, where
we had = 8 parts/h and = 10 parts/h. We have seen that
U = 0.8, L = 4, Q = 3.2, W = 0.5 h and D = 0.4 h.
At the other extreme, we have M/M/1/1 system in which no
queuing is allowed. In this case, there are only two states, 0 and 1,
and
0 =

= 0.556; and 1 = 0.444

Verify that U = 0.444, L = Q = 0.444, and W = D = 0.1h.

Can you calculate the performance measures for M/M/1/10
queue?

99 / 111

Example: Consider a machine center modeled as an M/M/1/N

queue with parameters and . Let C be the cost of operating
the machine at a rate of and let a profit of q accrue for every
finished part produced. Find out the service rate that maximizes
the total profit.
Solution: Production rate of the machine center=U
n)
N N )
= (
= (1
1N+1
N+1 N+1
The above is the number of parts produced in one time unit (for
example one hour, if is expressed in terms of parts per hour).
The corresponding total profit is given by:
P=

(N N )
q C
N+1 N+1

The service rate that maximizes the profit can now be obtained
in the usual way.
100 / 111

The M/M/m Queue: Consider a queuing system with Poisson

arrival rate as before, but where m( 1) exponential servers,
with rate each, share a common queue. This is referred to as the
M/M/m queue and is illustrated in Fig. 3. If the number of
customers in the system is greater than the number of servers,
then a queue forms.

Figure: M/M/m queue and its state diagram

101 / 111

Steady State Analysis: The M/M/m queue is also a special case

of the birth and death model with rates:
k = , k = 0, 1, 2, . . .

(29)

= k, k = 0, 1, . . . , m
= m, k > m

(30)

102 / 111

The state transition diagram is shown in Fig. 3. The steady state

probabilities could be obtained by using (29) and (30) in the
expressions of 0 and k of BD process. We find that:
k

=
=

0 (m)k
; k = 0, 1, 2, . . . , m
k!
0 mm k
; k>m
m!

where is defined as:

=

(31)

(32)

Equation (31) is valid for determining the steady state probabilities

whenever < 1, i.e., the stability condition holds.

103 / 111

The expression for 0 can now be obtained using (31) and the fact
that the steady state probabilities sum up to 1. Thus:

m1
X (m)k 1
(m)m
0 =
+
m!(1 )
k!

(33)

k=0

To compute the different performance measures, we first consider

Ns , the number of busy servers in the steady state. Ns is a discrete
random variable with range {0, 1, 2, . . . , m} and with pmf given by:
P(Ns = k) = k

X
j
=

k = 0, 1, . . . , m 1
k=m

j=m

104 / 111

The average number of busy servers in the steady state, B can be

immediately obtained as:
B = E [Ns ] =

m
X

kP(Ns = k) =

k=0

(34)

The mean utilization U of each server is simply the ration of B to

the total number of servers and hence:
U=

=
m

(35)

105 / 111

We now compute Q, the average number of customers waiting in

queue:

X
(m)m 0
Q=
(k m)k =
(36)
m!(1 )2
k=m

The average number L in the system is given by:

L=Q +B =

(m)m 0

+
m!(1 )2

(37)

106 / 111

We now use Littles law to compute the average waiting time in

the system. We have:
W =

L
(m)m 0
1
=
+
2

m!(1 )

(38)

The average waiting time in queue can now be obtained as:

D=W

1
(m)m 0
=

m!(1 )2

(39)

107 / 111

The M/M/ Queue: In an M/M/ queueing system we have a

Poisson arrival process with arrival rate and an infinite number of
servers with service rate each.
If there are k jobs in the system, then the overall service rate is k
because each arriving job immediately gets a server and does not
have to wait. Once again, the underlying CTMC is a birth-death
process.
From the expressions of 0 and k of BD process, we obtain the
steady-state probability of k jobs in the system:
k = 0

k1
Y
i=0

= 0
(i + 1)

k!

(40)

108 / 111

The steady state probability of no jobs in the system:

0 =
1+

1
k
P

k=1

and finally:

k
k =

=e

(41)

1
k!

(42)
k!
This is the Poisson pmf, and the expected number of jobs in the
system is:

L=
(43)

With Littles theorem, the mean waiting time in the system is

given by:
1
W =
(44)

109 / 111

Queuing models for self study:

A. M/M/s queuing model with finite storage and infinite
population, and
B. M/M/s queuing model with finite storage and finite population.

110 / 111

Thank You

111 / 111