IE3354

Stochastic OR
Week 5 Part I – DTMC: First Step Analysis and Absorbing Chains

Stochastic Processes
Discrete Time Markov Chains

Özgür TOY, Önder BULUT

Industrial Engineering, Yaşar University

1/22
First step analysis

Absorbing MC

- In which state, 0 or 2, is the process ultimately trapped?

- How long, on the average, does it take to reach one of these
states?

2/22
 𝑇 = min 𝑛 ≥ 0; 𝑋𝑛 = 0 or 𝑋𝑛 = 2

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 0 0

𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽 𝛽 𝛽 𝛽
1 1 1 1 1 1 1
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 2 2

If
𝑋1 = 0
or
𝑋1 = 2
then
𝑇=1 3/22
 𝑇 = min 𝑛 ≥ 0; 𝑋𝑛 = 0 or 𝑋𝑛 = 2

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 0 0

𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽 𝛽 𝛽 𝛽
1 1 1 1 1 1 1
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 2 2

If
If 𝑋2 = 0
𝑋1 = 1 or
𝑋2 = 2
then
𝑇=2 4/22
 𝑇 = min 𝑛 ≥ 0; 𝑋𝑛 = 0 or 𝑋𝑛 = 2

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 0 0

𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽 𝛽 𝛽 𝛽
1 1 1 1 1 1 1
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 2 2

If
If If 𝑋3 = 0
𝑋1 = 1 𝑋2 = 1 or
𝑋3 = 2
then
𝑇=3 5/22
 𝑇 = min 𝑛 ≥ 0; 𝑋𝑛 = 0 or 𝑋𝑛 = 2

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 0 0

𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽 𝛽 𝛽 𝛽
1 1 1 1 1 1 1
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 2 2

If
If If If 𝑋4 = 0
𝑋1 = 1 𝑋2 = 1 𝑋3 = 1 or
𝑋4 = 2
then
𝑇=4 6/22
 𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 ... 0 0 ...
𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽
1
𝛽
1 1
𝛽
1 1
𝛽 ... 1 1 ...
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 ... 2 2 ...

7/22
 𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 ... 0 0 ...
𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽
1
𝛽
1 1
𝛽
1 1
𝛽 ... 1 1 ...
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 ... 2 2 ...

8/22
 𝑢10 : the probability that process is trapped absorbed in state 0 given that the process starts in state 1

𝑢10 = 𝑃𝑟 𝑋𝑇 = 0|𝑋0 = 1

𝑢00 = 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 0 = 1

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4
...
0 0 ...
0 0 0 0
𝛼

𝑢20 = 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 2 = 0
𝑋0 𝑋1 𝑋2 𝑋3 𝑋4
𝛾
1

...
2 2 2 2 ... 2 2
9/22
 𝑢10 = 𝑃𝑟 𝑋𝑇 = 0|𝑋0 = 1

𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 1 = 𝑢10

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 ... 0 0 ...
𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽
1
𝛽
1 1
𝛽
1 1
𝛽 ... 1 1 ...
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 ... 2 2 ...

10/22
 𝑢10 = 𝑃𝑟 𝑋𝑇 = 0|𝑋0 = 1

𝑢10 = 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 0 𝑃𝑟 𝑋1 = 0|𝑋0 = 1 + 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 1 𝑃𝑟 𝑋1 = 1|𝑋0 = 1 + 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 2 𝑃𝑟 𝑋1 = 2|𝑋0 = 1

𝑢10 = 𝑢00 . 𝛼 + 𝑢10 . 𝛽 + 𝑢20 . 𝛾

𝑢10 = 1 . 𝛼 + 𝑢10 . 𝛽 + 0 . 𝛾
𝑢10 = 𝛼 + 𝑢10 . 𝛽
𝛼 𝛼
𝑢10 = =
1−𝛽 𝛼+𝛾

11/22
 𝑤1 : expected number of transitions until absorption given that the process starts in state 1

𝑤1 = 𝐸 𝑇|𝑋0 = 1

𝑤0 = 𝐸 𝑇|𝑋1 = 0 = 0

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4
...
0 0 ...
0 0 0 0
𝛼

𝑤2 = 𝐸 𝑇|𝑋1 = 2 = 0
𝑋0 𝑋1 𝑋2 𝑋3 𝑋4
𝛾
1

...
2 2 2 2 ... 2 2
12/22
 𝑤1 = 𝐸 𝑇|𝑋0 = 1

𝐸 𝑇|𝑋1 = 1 = 𝑤1

𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 ... 0 0 ...
𝛼 𝛼 𝛼 𝛼 𝛼 𝛼 𝛼

𝛽 𝛽 𝛽 𝛽
1
𝛽
1 1
𝛽
1 1
𝛽 ... 1 1 ...
𝛾 𝛾 𝛾 𝛾 𝛾 𝛾 𝛾

2 2 2 2 ... 2 2 ...

13/22
 𝑤1 = 𝐸 𝑇|𝑋0 = 1

𝑤1 = 1 + 𝑃𝑟 𝑋1 = 0|𝑋0 = 1 𝐸 𝑇|𝑋1 = 0 + 𝑃𝑟 𝑋1 = 1|𝑋0 = 1 𝐸 𝑇|𝑋1 = 1 + 𝑃𝑟 𝑋1 = 2|𝑋0 = 1 𝐸 𝑇|𝑋1 = 2

𝑤1 = 1 + 𝛼. 0 + 𝛽. 𝑤1 + 𝛾. 0

𝑤1 = 1 + 𝛽𝑤1

1 1
𝑤1 = =
1−𝛽 𝛼+𝛾

14/22
𝑋0 𝑋1 𝑋2 𝑋3 𝑋4

0 0 0 0 ... 0 0 ...

1 1 1 1 1 ... 1 1 ...

2 2 2 2 2 ... 2 2 ...

3 3 3 3 ... 3 33 ...
15/22
Consider possible starting states:

𝑋0 = 1
𝑢10 = 𝑃10 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 0 + 𝑃11 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 1 + 𝑃12 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 2 + 𝑃13 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 3

𝑢10 = 𝑃10 𝑢00 + 𝑃11 𝑢10 + 𝑃12 𝑢20 + 𝑃13 𝑢30 = 𝑃10 1 + 𝑃11 𝑢10 + 𝑃12 𝑢20 + 𝑃13 0 = 𝑃10 + 𝑃11 𝑢10 + 𝑃12 𝑢20

𝑋0 = 2
𝑢20 = 𝑃20 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 0 + 𝑃21 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 1 + 𝑃22 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 2 + 𝑃23 𝑃𝑟 𝑋𝑇 = 0|𝑋1 = 3

𝑢20 = 𝑃20 1 + 𝑃21 𝑢10 + 𝑃22 𝑢20 + 𝑃23 0 = 𝑃20 + 𝑃21 𝑢10 + 𝑃22 𝑢20

Two equations with two unknowns, solve for 𝑢10 and 𝑢20
16/22
 Example

𝑢10 = 0.4𝑢00 + 0.3𝑢10 +0.2𝑢20

𝑢10 = 0.4 + 0.3𝑢10 +0.2𝑢20
The probability that process will be absorbed in state 0 given that
it starts in state 1 is:
𝑢20 = 0.1𝑢00 + 0.3𝑢10 +0.3𝑢20
𝑢20 = 0.1 + 0.3𝑢10 +0.3𝑢20 30
𝑢10 =
43
or
The probability that process will be absorbed in state 0 given that
0.7𝑢10 −0.2𝑢20 = 0.4 it starts in state 2 is:
-0.3𝑢10 + 0.7𝑢20 = 0.1 19
𝑢20 =
43
13
The probability that process will be absorbed in state 3 given that it starts in state 1 is: 𝑢13 = 1 − 𝑢10 =
43
24
The probability that process will be absorbed in state 3 given that it starts in state 2 is: 𝑢23 = 1 − 𝑢20 = 17/22
43
Mean time until absorption
𝑤0 = 0, 𝑤3 = 0

𝑤1 = 1 + 𝑃10 𝑤0 + 𝑃11 𝑤1 + 𝑃12 𝑤2 + 𝑃13 𝑤3

𝑤1 = 1 + 0.3𝑤1 + 0.2𝑤2

𝑤2 = 1 + 𝑃20 𝑤0 + 𝑃21 𝑤1 + 𝑃22 𝑤2 + 𝑃23 𝑤3

𝑤2 = 1 + 0.3𝑤1 + 0.3𝑤2

90 100
𝑤1 = ,𝑤 =
43 2 43

18/22
19/22
20/22
What is the probability of absorption in the food compartment, given that the rat is initially dropped in compartment i ?

1 1
𝑢07 = 𝑢 + 𝑢27
2 17 2
1 1 1
𝑢17 = + 𝑢 + 3 𝑢37
3 3 07
1 1
𝑢27 = 𝑢 + 3 𝑢37
3 07
1 1 1 1
𝑢37 = 4
𝑢17 + 4
𝑢27 + 4 𝑢47 + 4
𝑢57
1 1 1
𝑢47 = 3 + 3 𝑢37 + 3 𝑢67

1 1
𝑢57 = 𝑢 + 3 𝑢67
3 37
1 1
𝑢67 = 𝑢47 + 𝑢57
2 2

21/22