Markov processes - III

• review of steady-state behavior

• probability of blocked phone calls

• calculating absorption probabilities

• calculating expected time to absorption

1
review of steady state behavior

• Markov chain with a single class of recurrent

states. aperiodic; and some transient states; then.

lim " i)'(n) =

n-+oo
lim P ( X " = j
11 -+00
I Xo = i) = " J' 'V i
• - - ',
.; ;'-,
(P(l(... :.j .; ~
... r1J ... ,. I )(0=' ..
1.,,­__ 0
; " \..­
_

/"1fj" 4-. 1P~O-)1

.... ­
• can be found as the unique solution to the
balance equations
j = 1 , .. . ,

• together with L "j = 1

J

2
on the use of steady state probabilities, example

III"O.S~1I'"\.J'O.'l. ~ 1
,
o.qqB
!).,8'"
iTi -to TTl. =d Od~ I
____M ,
O.S
• •
'2./r
. ' F

_ _ _ _ _ _ _ _3>
assume process starts in state 1
P (X l = 1 ilnd XIOO = 1 1Xo = 1 ) =
'"
",:,.5 ..e 5 " "
~ct.(, ',,1
"
IrCx,=.,/'ICo,.l).'p(X,oo='/ )(,:~~) t. ....0 ~..., 10 5 cl."> J
f" J' r Cq'l) 'R:i f" ~ 1Tj Co OS~~ t) O/d..t" .:> ~ ......,..~ ..
"
P (XIOO = 1 and XIOI = 2 1Xo = 1) = .N )
jP(X.o--I/ '1.0.')'" W(X.o. ~2...J XOOO= ~ ~I :£> PI' -Wc.o/
F ...

f,,(14)O) )I f't. W1f,"'l'l... ~ 2....,.00$ (c:. \ O<"C < I

P (XlOO = 1 and X f OO = I J XO = 1) = :}~./ ) _<::0.3

,i>OC.,._. Xo•• ). 0'( X?Q)~II X,eo ..'... ~, t.
_ C ::: o.~'}
= (,,{.oo) .... r.,(IOO) ~-,y;"',,. :'TI;'-: (~)
3
design of _a~hone system
(8 r '"",) v;lI. j~ _ t,.~

( ,,,,,I<) - - ­
• calls originate as a Poisson process. rate oX
• each call duration is exponential (parameter f.J.)
• need to decide on how many lines, B~
t
, ,• •• _ _ .1111 -< 7 ~1111

tIme "

.n
"

."

• for time slots of small duration at! ".l "

••
•
• • • • • • ~ II ,. M II ~

I I
l..,.S
P(a new call arrives) :::::: Ad

if you have i active calls. then P{a departure) ::::::: i J.L§

4
design of a phone system, a discrete time approximation

~---..,
,
time
• approximation: discrete time slots of (small) duration 8

!P(i. • ...,"'")~U ; TI(1."'II"'~ll":b.sy)~<I"S;

1
1 ). ;;::.]0 c.cJ/, I..:... ~
o #=->3
J. ""',....It.t
• balance e uations 'fo calls
A7fi_ l = iJ1-7ri
"! Ai
110 , 1r. -) -,rEo - --~- - ­ '13", '0 ?
1 - 11"0
11" " -
i "1
H

• P(arriving customer finds busy system) is@

5
calculating absorption probabilities

• absorbing state: recurrent state k with Pkk =

1
b..:
• what is the probability @ that the chain
eventually settles inA'given it started in i ?
S
i = 4, ai = I
i =5 , ai = O
I
3
.,
OJ
otherwise, a i = !
a.: ,BAd
"
06 - elL'" tons
1 r-_ _..:-.<lI::l:,..::;.1>/t.8
08 - ...,
• unique solution from i =
.'Ib..... .l
L. : Pij aj

I
().s =- -:L. / • -d '--­ 6
expected time to absorption

"
• find expected number of transitions I-'i "
until reaching 4, given that the initial state is i "
"
o.t­ J.Li=o for i= 4
for all others, j..1.'i = ?
0.4
, Ih:.IIO/B'
_ /-<'1.:: qt;~ :. ~

~1= 1Iy&

• unique solution from i = 1 +L .

PijI.Lj
J

7
mean first passage and recurrence times
"
• chain with one recurrent class; fix a recurrent state s

8
gambler's example

• a gambler starts with i dollars; each time. she bets $1 in a fair game, until she either has 0 or n dollars.
• what is the probability ai that she ends up with having n '&lIars?
, $.v •• r..,. t
•
= n, Ui =
•••
~~ >'-­
,~~
a...:
• expected wealth at the end!
,I
'-' ••
• how long does the gambler expect to stay in the game!
/.Li= expected number of plays. starting from i

a.;. =- % .""A-llcO

for i = 0 , n: J-li = 0 \ <.:

c .. ~ :. i + I ,/".1+1. -+

- in general

J-Li = 1 +L Pijl.Lj

•
J
in case of unfavorable odds? f ;: O·s •
....
r:: ,-t ~...: =­
1._1
f I- r"" 9
MIT OpenCourseWare
https://ocw.mit.edu

Resource: Introduction to Probability

John Tsitsiklis and Patrick Jaillet

The following may not correspond to a p articular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource.

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.