Koorosh Aslansefat - University of Hull

Markov Modelling and Basic Concepts

Koorosh Aslansefat - University of Hull

Focus of the Session
“To learn the basic concepts of Markov Modelling.

 Understanding Symbols, basic modelling rules and mathematical bases for Reliable Analysis.

 Being familiar with different types of Markov Models

Koorosh Aslansefat - University of Hull 3

Introduction
Why Markov Modelling?

Ability to Model a System’s Dynamic

Behaviours such as:

• Repair
• Sequence
• Priority
• Reconfiguration

Koorosh Aslansefat - University of Hull 4

Definitions
What is Markov Chain Model?

 A stochastic model that describes the probabilities of transition among the

states of a system.
 It is a random process that undergoes transitions from one state to another in
a state space.
 Change of states depends probabilistically only on the current state of the
system.

Koorosh Aslansefat - University of Hull 5

Symbols
Markov Model is a mathematical modelling approach used to represent systems and processes
that exhibit stochastic behaviour characterized by the Markov property of memorylessness.

State Transition

Koorosh Aslansefat - University of Hull 6

Markov Model
Markov Model is a mathematical modelling approach used to represent systems and processes
that exhibit stochastic behaviour characterized by the Markov property of memorylessness.

𝜆 ∆𝑡

Op F

1 − 𝜆 ∆𝑡 1

Notes:
Op D F

1 − 𝜆 ∆𝑡 0 1
Normal State Shadow State Absorbing State

Koorosh Aslansefat - University of Hull 7

Markov Model

Common Model Name Time Space State Space

Discrete Time
Discrete Discrete
Markov Chain
Continuous Time
Continuous Discrete
Markov Chain
Continuous State, Discrete Time
Discrete Continuous
Markov Process

Continuous State, Continuous

Continuous Continuous
Time Markov Process

Koorosh Aslansefat - University of Hull 8

Markov Model

Aslansefat, K., Kabir, S., Gheraibia, Y., &

Papadopoulos, Y. (2020). Dynamic fault
tree analysis: state-of-the-art in modelling,
analysis, and tools. Reliability Management
and Engineering, 73-112.
Koorosh Aslansefat - University of Hull 9
Markov Model
𝐹 4 (𝑡)
𝐹 3 (𝑡 ) 𝐹 3 (𝑡 )
Hidden States 𝐹 2 (𝑡 ) 𝐹 2 (𝑡 ) 𝐹 2 (𝑡 )

of System’s 𝐹 1 (𝑡 ) 𝐹 1 (𝑡 ) 𝐹 1 (𝑡 ) 𝐹 1 (𝑡 )

Operation OP D1 D2 D3 F

𝑏1 𝑛 𝑏51

𝑏11
𝑏5 𝑛

System’s
Observations
O1 O2 O3 O4 O5 On

Koorosh Aslansefat - University of Hull 10

Markov Model

Cai, B., Liu, Y., Liu, Z., Tian, X., Li, H., &
Ren, C. (2012). Reliability analysis of
subsea blowout preventer control systems
subjected to multiple error shocks. Journal
of Loss Prevention in the Process
Industries, 25(6), 1044-1054.

Koorosh Aslansefat - University of Hull 11

Markov Calculation
Failure Probability
− 𝜆 .∆ 𝑡
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑖𝑡𝑦 =1− 𝑒
Using Taylor Series
2
( 𝜆 ∆ 𝑡 )
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑖𝑡𝑦 =1− 𝑒− 𝜆 .∆ 𝑡 ≈ 1− ( 1+ ( − 𝜆 ∆ 𝑡 ) ) + +…
2!

Estimation
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑖𝑡𝑦 ≈ 𝜆 ∆ 𝑡

Koorosh Aslansefat - University of Hull 12

Markov Calculation

Initial Probability

( )
¿ 𝑃 𝑂𝑝 ( 0 ) = 𝑃 ( 0 ) = 1
¿ 𝑃𝐹 ( 0) 0 ( )
Transition Matrix

[ ][ ][
¿ 𝑃 𝑂𝑝 ( 𝑡 + ∆ 𝑡 ) = ¿ 1− 𝜆 ∆ 𝑡 0 . ¿ 𝑃 𝑂𝑃 ( 𝑡 )
¿ 𝑃 𝐹 (𝑡+∆ 𝑡) ¿ 𝜆∆𝑡 1 ¿ 𝑃 𝐹 (𝑡 ) ]
𝑃 ( ∆ 𝑡 ) = 𝐴 . 𝑃 ( 0)
Koorosh Aslansefat - University of Hull 13
Markov Calculation

Initial Probability

( )
¿ 𝑃 𝑂𝑝 ( 0 ) = 𝑃 ( 0 ) = 1
¿ 𝑃𝐹 ( 0) 0 ( )
Transition Matrix

[ ][ ][
¿ 𝑃 𝑂𝑝 ( 𝑡 + ∆ 𝑡 ) = ¿ 1− 𝜆 ∆ 𝑡 0 . ¿ 𝑃 𝑂𝑃 ( 𝑡 )
¿ 𝑃 𝐹 (𝑡+∆ 𝑡) ¿ 𝜆∆𝑡 1 ¿ 𝑃 𝐹 (𝑡 ) ]
𝑃 ( ∆ 𝑡 ) = 𝐴 . 𝑃 ( 0)
Koorosh Aslansefat - University of Hull 14
Markov Calculation

𝑃 ( ∆ 𝑡 ) = 𝐴 . 𝑃 ( 0)

𝑃 ( 2 ∆ 𝑡 ) = 𝐴. 𝑃 ( ∆ 𝑡 ) = 𝐴. ( 𝐴 . 𝑃 ( 0 ) )= 𝐴 𝑃 ( 0 )
2

3
𝑃 ( 3 ∆ 𝑡 )= 𝐴 . 𝑃 ( 0 )
¿⋮
𝑛
¿ 𝑃 (𝑛 ∆ 𝑡)= 𝐴 . 𝑃 (0 )
Koorosh Aslansefat - University of Hull 15
Markov Calculation

Transition Matrix

{
𝑃 𝑂𝑝 ( 𝑡 + ∆ 𝑡 ) − 𝑃 𝑂𝑝 ( 𝑡 )
¿ =− 𝜆 𝑃 𝑂𝑝 ( 𝑡 )
∆𝑡
¿
𝑃 𝐹 (𝑡+∆ 𝑡) − 𝑃 𝐹 (𝑡)
¿ = 𝜆 𝑃 𝑂𝑝 ( 𝑡 )
∆𝑡

[ ˙ 𝑂𝑝 ( 𝑡 )
¿𝑃
¿𝑃˙ 𝐹 (𝑡) ] [
= ¿− 𝜆0 .
¿ 𝜆0 ][
¿ 𝑃 𝑂𝑝 ( 𝑡 )
¿ 𝑃𝐹 (𝑡 ) ] 𝑀=
𝝏𝑨
𝝏∆𝒕

−𝑀 ∗𝑡
𝑃 ( 𝑡 )= 𝑒 . 𝑃 ( 0)
Koorosh Aslansefat - University of Hull 16
Markov Calculation

import numpy as np # linear algebra

import sympy as sym # Symbolic Calculation

t = sym.Symbol('t')

− 0 .0001 ∗𝑡
L = 0.0001
𝐹𝑎𝑖𝑙𝑢𝑟𝑒 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑖𝑡𝑦 =1− 𝑒
P0 = sym.Matrix([[1],[0]])

M = sym.Matrix([[-L, 0],
[ L, 0]])

𝐹𝑎𝑖𝑙𝑢𝑟𝑒𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑖𝑡𝑦 (1000)=0.0951625819640404
P = sym.exp(M*t)*P0

P_Fail = P[-1]

time = 1000
P_Fail_Numerical = P_Fail.evalf(subs={t: time})

print(P_Fail)
print(P_Fail_Numerical)
print(1 - P_Fail_Numerical)
𝑅𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 (1000 )=0.904837418035960
https://colab.research.google.com/drive/13XlRX4jlDRG8aepu61TmE_NcyJ1iKbVO?usp=sharing

Koorosh Aslansefat - University of Hull 17

Markov Calculation
import numpy as np # linear algebra
import sympy as sym # Symbolic Calculation
import matplotlib.pyplot as plt # Plots
from tqdm import tqdm # Loop Loading

def Simple_Markov(Lambda=0.0001, time=1000):

t = sym.Symbol('t')
L = Lambda
P0 = sym.Matrix([[1],[0]])
M = sym.Matrix([[-L, 0],
[ L, 0]])
P = sym.exp(M*t)*P0
P_Fail = P[-1]
P_Fail_Numerical = P_Fail.evalf(subs={t: time})
return 1 - P_Fail_Numerical

Reliability = []

for time in tqdm(range(0, 20000, 100)):

Reliability.append(Simple_Markov(Lambda=0.0001, time=time))

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

axs.plot(range(0, 20000, 100), Reliability, linestyle='-')

axs.set_title('Reliability of a Single Component')
axs.set_xlabel('Time')
axs.set_ylabel('Reliability')
plt.show()

https://colab.research.google.com/drive/13XlRX4jlDRG8aepu61TmE_NcyJ1iKbVO?usp=sharing

Koorosh Aslansefat - University of Hull 18

Markov Calculation
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt
from tqdm import tqdm

def Simple_Markov(Lambda=0.0001, time=1000):

t = sym.Symbol('t')
L = Lambda
P0 = sym.Matrix([[1], [0]])
M = sym.Matrix([[-L, 0],
[L, 0]])
P = sym.exp(M*t)*P0
P_Fail = P[-1]
P_Fail_Numerical = P_Fail.evalf(subs={t: time})
return 1 - P_Fail_Numerical

Lambda_values = [0.001, 0.0001, 0.00001, 0.000001]

colors = ['blue', 'green', 'red', 'purple']
labels = ['Lambda 0.001', 'Lambda 0.0001', 'Lambda 0.00001',
'Lambda 0.000001']
times = range(0, 20000, 100)

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

for Lambda, color, label in zip(Lambda_values, colors, labels):

Reliability = [Simple_Markov(Lambda=Lambda, time=time) for
time in tqdm(times)]
axs.plot(times, Reliability, linestyle='-', color=color,
label=label)

axs.set_title('Reliability of a Single Component')

axs.set_xlabel('Time')
axs.set_ylabel('Reliability')
axs.legend()
plt.show()
Koorosh Aslansefat - University of Hull 19
Markov Calculation for Availability
𝜆 ∆𝑡

import numpy as np # linear algebra

Op F
import sympy as sym # Symbolic Calculation
import matplotlib.pyplot as plt # Plots
from tqdm import tqdm # Loop Loading
1 − 𝜆 ∆𝑡 𝜇∆𝑡
def Simple_Markov(Lambda=0.0001, Mu=0.001, time=1000):
t = sym.Symbol('t')
L = Lambda
mu = Mu
P0 = sym.Matrix([[1],[0]])
M = sym.Matrix([[-L, mu],
[ L, -mu]])
P = sym.exp(M*t)*P0
P_Fail = P[-1]
P_Fail_Numerical = P_Fail.evalf(subs={t: time})
return 1 - P_Fail_Numerical

Availability = []

for time in tqdm(range(0, 20000, 100)):

Availability.append(Simple_Markov(Lambda=0.0001, Mu=0.001,
time=time))

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

axs.plot(range(0, 20000, 100), Availability, linestyle='-')

axs.set_title('Availability of a Single Component')
axs.set_xlabel('Time')
axs.set_ylabel('Availability')
plt.show()

https://colab.research.google.com/drive/1l036JE2tNx3z89iDCLlnlqCDRsP2mfTo?usp=sharing

Koorosh Aslansefat - University of Hull 20

Markov Calculation for Availability
𝜆 ∆𝑡

import numpy as np # linear algebra Op F

import sympy as sym # Symbolic Calculation
import matplotlib.pyplot as plt # Plots
from tqdm import tqdm # Loop Loading
1 − 𝜆 ∆𝑡 𝜇∆𝑡
def Simple_Markov(Lambda=0.0001, Mu=0.01, time=1000):
t = sym.Symbol('t')
L = Lambda
mu = Mu
P0 = sym.Matrix([[1], [0]])
M = sym.Matrix([[-L, mu],
[L, -mu]])
P = sym.exp(M*t)*P0
P_Fail = P[-1]
P_Fail_Numerical = P_Fail.evalf(subs={t: time})
return 1 - P_Fail_Numerical

Mu_values = [0.1, 0.01, 0.001, 0.0001]

times = range(0, 20000, 100)
colors = ['blue', 'green', 'red', 'purple']
labels = ['Mu = 0.1', 'Mu = 0.01', 'Mu = 0.001', 'Mu = 0.0001']

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

for Mu, color, label in zip(Mu_values, colors, labels):
Availability = [Simple_Markov(Lambda=0.0001, Mu=Mu,
time=time) for time in tqdm(times)]
axs.plot(times, Availability, linestyle='-', color=color,
label=label)

axs.set_title('Availability of a Single Component')

axs.set_xlabel('Time')
axs.set_ylabel('Availability')
axs.legend()
Koorosh Aslansefat - University of Hull
plt.show()
21
 Markov Calculation for Availability (Imperfect
Maintenance)

𝜆 ∆𝑡 𝜇∆𝑡 ´ ∆𝑡
𝜆 𝜇∆𝑡 ¨ ∆𝑡
𝜆

Op F Op F Op F

1 − 𝜆 ∆𝑡 ´ ∆𝑡
1−𝜆 1 − 𝜆¨ ∆𝑡 1

Koorosh Aslansefat - University of Hull 22

Markov Calculation for Safety
import numpy as np # linear algebra
import sympy as sym # Symbolic Calculation
import matplotlib.pyplot as plt # Plots
from tqdm import tqdm # Loop Loading

def Simple_Markov(Lambda=0.0001, Coverage=0.8, time=1000):

t = sym.Symbol('t')
L = Lambda
c = Coverage
P0 = sym.Matrix([[1],[0],[0]])
M = sym.Matrix([[ -L, 0, 0],
[ c*L, 0, 0],
[(1-c)*L, 0, 0]])
P = sym.exp(M*t)*P0
P_Fail_Unsafe = P[-1]
P_Fail_Unsafe_Numerical = P_Fail_Unsafe.evalf(subs={t: time})
return 1- P_Fail_Unsafe_Numerical

Safety = []

for time in tqdm(range(0, 20000, 100)):

Safety.append(Simple_Markov(Lambda=0.0001, Coverage=0.2,
time=time))

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

axs.plot(range(0, 20000, 100), Safety, linestyle='-')

axs.set_title('Safety of a Single Component')
axs.set_xlabel('Time')
axs.set_ylabel('Safety')
plt.show()

https://colab.research.google.com/drive/1zmpgqKYylWoAJADIUwQHfYXDNs0S0ga5?usp=sharing

Koorosh Aslansefat - University of Hull 23

Markov Calculation for Safety
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt
from tqdm import tqdm

def Simple_Safety_Markov(Lambda=0.0001, Coverage=1, time=1000):

t = sym.Symbol('t')
L = Lambda
c = Coverage
P0 = sym.Matrix([[1], [0], [0]])
M = sym.Matrix([[-L, 0, 0],
[c*L, -L, 0],
[(1-c)*L, 0, -L]])
P = sym.exp(M*t)*P0
P_Fail_Unsafe = P[-1]
P_Fail_Unsafe_Numerical = P_Fail_Unsafe.evalf(subs={t: time})
return 1 - P_Fail_Unsafe_Numerical

Coverage_values = [0.9, 0.8, 0.7, 0.6]

times = range(0, 20000, 1000)
colors = ['blue', 'green', 'red', 'purple']
labels = ['Coverage = 0.9', 'Coverage = 0.8', 'Coverage = 0.7',
'Coverage = 0.6']

fig, axs = plt.subplots(1, 1, figsize=(10, 8))

for Coverage, color, label in zip(Coverage_values, colors,
labels):
Safety = [Simple_Safety_Markov(Lambda=0.00001,
Coverage=Coverage, time=time) for time in times]
axs.plot(times, Safety, linestyle='-', color=color,
label=label)

axs.set_title('Safety of a Single Component')

axs.set_xlabel('Time')
axs.set_ylabel('Safety')
Koorosh Aslansefat - University of Hull
axs.legend()
plt.show() 24
Markov for Reliability, Availability and Safety

Notes:

• In more complex systems we might have online repair in both reliability and safety models.
However, offline repair is considered only for availability models.
• For Safety models we could have more than two types of failure.

Koorosh Aslansefat - University of Hull 25

Towards the Future

Koorosh Aslansefat - University of Hull 26

Towards the Future

Koorosh Aslansefat - University of Hull 27

References

Trivedi, K. S., & Bobbio, A. (2017).

Reliability and availability engineering:
modelling, analysis, and applications.
Cambridge University Press

Koorosh Aslansefat - University of Hull 28

Summary

 Discussed why Markov is our choice in comparison to other existing models

like FTA, Petri Nets, etc.

 Discussed different variations of Markov: It detailed various forms of Markov

models, such as discrete-time and continuous-time Markov chains.

 Calculation and codes for reliability, availability, and safety calculation of a

single component.

Koorosh Aslansefat - University of Hull 29

 Thanks for Your Attention
If you have any question, please feel free to ask

Koorosh Aslansefat - University of Hull 30