Unit - V

A random Vaviable (Rv) is a nule Cor func ti

HRata asig
ranlem
real numbey to eve outcane
experiment, wkile a random
bces is a
rale (or unctie)that asigs a
ime ne ton to erem eutcome t a random
expeiment.
For eample, Consider the randor gom
toing a dice at to and cbsei tte
numbey on the top face.
The Sample Space of this eapeiment Cersiata
tae eutcones S,2,>,63. For each outlame
the eperient, let
|unctian manne

atcoe 2

Functisn of time

The set functions

repre senta random po cess

Dekn A randon s a

Hx(s} at ae unctiohs of a red Vaiask

 namely time where ses (Sampe space) and
Aer Cpanameter set y index set).
Ihe ihdin
memer ef Ioe random broce s called `at Space
Any indiidual member tself is called a
the po ces.
Markov Poocess A random oces in ushics he
tue vadue depends ony enen the toe.sent
Nale nd not on the
the past Values, ts Called a
Marka þoocess.
Example: Comsider the Yandom experimentt
to ksing ain Coin mang times.
There are too possible outlones fer each
heat Cdensted by d) and
ail ldenoted by )

each wit probabity a

let x be
be the
the Y alenting the outcome
Then P(xn)e and px,eo)= fr
hl,2, 3,4
 are independent vs.
New let Sn di ba a rs deiad as folles
S,oa numlae ef hands 'n the tst htrials
4 Xn
S Can take þossibe vaues o, , 2.,.
Jet S k
Then

and PiSnnhS,k)
anà these fooabilities depend ny on tha vale
S
Sn and hot
not cn the
the vales
values ef s,, Sa, . Sa.
Simple exam ple ta Maskav chain

Note Random booceses xc3 (ito Mastkov þ-peaty)

br lentinus ane Called Matoy chain

Deinitin et a Makay chain

 s Called a
Ren the þoces {xn5,no,l, ... Manko
chain. CM:c)
* a , ^2, an are called tie states h the M.c,
The Cantemd
Coniien þoobabkilt P{xaj/xai3
DS Called the bne -step trastien poobak:iy toom
otate a to state a; at the nth sap(t ial) and
s denoted by k..Cn-l, h).
stap tamitien pockaliy does
not epend lhe step, then the Maoto chain s
Glled aa aomosenees Matkov chain.
the
|one-step transition pocbability is deoted by i;
jand the matix Pe (t.,) is called the ohe-.-step
transiEion foobab:it matix tpm),

NoteThe tpn t a Mastov chain is a tochastie

mati
(ie. Sum the elemn
 *The cantiona poebability thet the pooteys
in stae y at sep n, gven tbat it woasin stte a;
at stap o, e., pPXnax,-a;3 is called the

beginiki Rat the procass

n state a s t. (iel,2, k) at any abitrag
s e , tRen thoe
pootabity cst bution te poo ce at that
time
te) (6)
In þantias
initial
lo)

chapmankelrogec Theosem'
I3 P s the tpm of a homogeneus Mahav
 Resulat Matix: Asachatie mtbix Ps saidt

Some fosi tive nge

Maskor chain' A homogenemus Markov chain

|the poocess at an
ariray timeeen that the
bob dakibution ton
s pm) and Atbat
then
hemogenes nes Mc is
a

evey seuence o state poob detobutions ppoach

a uniae fixed ocb dit Calleal the stationany
cstate) dstsibution o steady sate distibutin
Maskav chain,
le, im{ 3 = , wkere the
ie state porb dist
at
step
) and te statim
Vectas
 chain,ten
Cn sa row vector .

Limating diatibutin
Suppose a þerson chooses between ou
blaces or dinnex , Home, Restaurants A, BB4 c.
14 he eas at one
place on a

hext
to ang o the tour place, it,
Vaias poobabities.
en belo
A B
H o25 20 25
17
A O30 30

O25
C

Ij he es at Rhome day 1, tRen the

atial Aistibuton s

The istbution
27'p: [o.2s
(2)
25
o262s o2325 o25]
 P,eceeAing
l6)
we
t
o2634

the state vecb

te, the ys ge on dstibti
peaches a
a icedl Vec to.Ths
steady state dat. o
ef te Martov chain.

Remazk Eveoy Markov chain need not have a

stetionag distbtion,
for example, Cosider aa Mc, ttt i m
and initil istbut on
)
Then P Lo1],F= P to1,
(2)

and þ
te, he ossilaes anddoes not thave a

saady ste dittoutich

 ()

classicatim
, n)
ef states a a Mastoy chain
Some n an or all i and i,
Ahen eveg state Can be reac hed fem every otRar
stele, and the Mc is Saia to be rreducibk
theuige the chain ssaid to be reducibe or
hoh ireclucible.
state i ot a Mic s calleÀ aretu state
Sone n>,

The peAed
beiod di i aa retun sae ii s the
nealest Comman dinsor all r such that
myo le, dy Gco(mi m>).sate i is þaiiodic
ith peried d Tt d l and aperiodic Th dil.
The poob. tbat ie chain retum to state i, having
started rom slate i, for tae first time at the n
s denoted by S.and s called te first relun
time poobti lty
3 F,= l, the retusn to state i is
Ceain and he sta te i s Sal t to be þersistent
Y
recuent The tae i is Said to be trasient
the retun to stae i s untestain le,
 nf K called the tmean Tecence
state i. the state i s Said tL
time
bonhull to tnean recurence
þes'stent i
inite and hull pesistent,
tire
;,
A nennull þersistant and apeidie sata
Calleal

Theorem!
(LH a M.e. s imeducile, all ct satos
lare o the Same
ype. They are eithe all
tyasi ent oy all null pesistent eY al boh-hull
persistent
An the staes are
ether aperiodic
periodie ito ame peiod.
a
Mic tinite and ieduible
then all b states are e non-null pesistent
 matriz ef a Man
The kranstien pacbab:ly
chain §x.3 hl, 2, 3,haing 3slates 1,a
{x3, b,
O4
Plo6 O2 and the niti0
distbution
Sel:
o5 O4
Given o2
and iniial
O4 03
Astaibion
to)

(6)
o2

(o)
and

) To qind PiX23
3

PB)

c2) (o)
e)+ (2)
 (2

(D)
Now P
O41 b4
2

026
24 O42 624
Lo o29

P{X,- 3 =Cobxo1)+ (o:34 xo2) +(a.21xo)

P{xa/x,+3, X4-a} Pix3,x+ 2

3 Pix/x,2ix,23
(6)
',3
3 2

(o4) o3) (62) (02)bo04g

 (13

) A man eithen doives a Cat ér catches atrain to

to akie ench day. He never
bw by trin but f he doives one day then the
next
jut ikaly to die
he ts to travel by dron. Now Sppose tat on Ihe
st da f the weck, the troat tossed a fain
Aice and dove to werk Tf and
appene Find i) tae bbi tbat he tates a ain
lon the thir day and i) the prob tbat he doives
bworkn the long
sol:
The ravel pate is a Matev chain
uita stae space (trais, Caa) or (T c)
The tpo te chain

The initial state boobdistbution s

Pltraveling by Can)
the dice)

and pltravelling by tGain)

 stak
find (at stte T)

(2)

p- (1 ) ( ) )
PCtae man aels by 24

Jet T=(T,,) be the

state pootb. dut. slationay dist. of the atky

chain
the
le, (Th,T)(),,Ta)
2
and

dst.
(2) and ( , we
and g7a

man avels by cat in the long un)

 in the ist
tosses, nd the tansiti pacbakly malyiz P R
|Maskev chain S3. End ab p and P{x,.
sl:
Stete spae: f, 2, 3, 4 5, 63
Jet Xn the toazinum f the numbers occwng

For Rxi to3. Consider tst ree t'rial reslt

S2,23, then as per the pooLlem statenet Xg is the
mazim un irst 3 tbases, X= Maz,2, 2'} 2
tb
Ln Cae
4 (o1) ) tial outeone is 4
then Xzmar{2,2,1 =2
2

So x, 2 then Xnt2 the cnt toiales ih

y2
Xnt3 f the Cht)t Lial resulk io
4
 L3, A5,

The isnsiion forbabiit matix of the clain

1 3

4 5 11
P 36

Tnitial state þ»ob distsibution

(o)
since all the Values
,2, .b are

(2)

*t4l44ll4 34):91
A+hree bys A, e nd c a ball t each
hen. A alany

Makovan. Find thet t o n maix and co 4The

staes
Sol: The tm, the footes {x34
A B
A
P B )

on stlas of Xn- bat

MatoN chain
O

Now P

(2) (2)
(3) (2)
. and al othen
>o. 1herekore, the chain s tneducible A

Va
Va 3/
VA
Va 9/s
And
 (-)

Thenere, the states a and 3 (ley B od cy

ete, are yo and GD

|35,b, l,

Thoto,the stak iCie state A) u poiodie ait

hee apenlodic.
Since the chain s inite and ieducise, al ct
ales are -hull persictant.
hon-ull

Al tbe stalas ae

bet Xoi hl, 2,3, be a Mhov chain wt slate

Space S-So,,23 and step taangition foebabi
mariz
P: s t) Is the chain
Flain (3 Find the invaiant þocbabilities.
Sol:
 A

stata o (3) (4)

(2)

Petod Geol2, 3,4,)2

sate aperiodic
shk ' (2) l3)
.o, 22 o, 22 >o..
Perod GeD(, 2, 3, )=l
sate 4 apeioic.
state'a
(2)

Peid GcD(2, 3, .. )+)

..gtate a is aperioic.
(2) (2) "o, Paa°,
(2) (2)
70

eucibe
The chein s s
states, so the chain
Alse, there are ony 3 imeducibl,
le
tthe choun is
states are hon-bull peristent.
Hene, all the
 Sinte all the statls are apeuodic and non-hult
persistent, all the states are

OTe nd the invaient oabiities.

let the stendy state foeb dut. be -[T.,15
ThenTP=T

We ks now tbat

4 4

tHente and

eA Yaining þrocoss s Considered as a two-shte

Mako chain. 3 t Yains, t s Comsidered t be
n sate o and T4 t doey not rain, the chais
tinad
 o 4
P

(o Eind also the uncmdtina byo tsat ll ain

stte o and sate i as o4 and oG

Sols
O4 ro6 o 4
P= -o2o.g o
2 .28
2 o3b 624
P= P P= o28 3)2

Focb dstibuti ater days

Loob: tat tuill rain o376

i) Given initial poob

re376 b24
lo:4 o ] 212 ob&so3874 o-Gb24]
Co33
 states of M.C wtb tpm
) Find Natue of

sol: 2
P -P. P
pPP:

2
2n+
= p P

Now ) (2)
2)
() (2)

C2)
t

M.e s ireducibk.
Also
Thenete (4
(2) -
the
- o ter ali all he (6)

|sates ot the chain are peridic ui to beniod

since te chain is inite and ireducibe, al
o staes are oull
honnu pesistent
A1 states
Ndte Trasiti am above bocblem)
8A gonbles has Rs 2l He beto RA. Iat a time
and
eses Rs. a o witns Rs 4. la what the tpm t
Hhe relted mc ? b what nthe prct tooat he bas
lost his oney at the end5 plays ? (c) what
Jaine last
Sol:
es Xn Yepresent tae amcunt utt, the
Hfe end ot the n t the
state space ot n3 (o,l, 2,3,4, 5,6) a the
\oses al the tooney
has Rs. b (Xn=6 (2t+))
The tpm the Mc
2 4 5

2
P: 3
4

The init
O,0,,o)
 to
(o)

(o)

: -('%, so,%°, )
,

P (

man Ras hast his money at the end { 5pay

-pfks C: the cnty coeapn dig tot
state o in (5)
5)

(1) 6)
128

Pthe game lasta more than#roanda3

-p\the
at te end ofthe seventb ound3
=PX,2, 3,4 er s3-L+o+2T4ot1327
128 |29

Note: "Thin is called a random Jalk wits absorbing

baies at o and b, &ince the ehain Cahnot Cone rat
et the sales o and b, on e t Ras ateed tham.
 3 kincs Coreals A,B and c.
she hever bgs the Same Cerea in succeysive
weeks. 1} she cereal A, the nent weat she
B or C,the nezt weok

she is 3 timnes as A s the otRo

Cereal Irte lona tun, how -ftan does she beg

each of the tree Ceyeas?
A
A

Let (T,.T,) be tae standy state dititi

the Markav hain Then TP1.

We kno tbat

35 35
& Assuvne that tRe weathat in a cetain lbea Can
bc moddled as the homegeneeus Ac ushose tp,
ven bal.
Todas wealher far
Far o15

cleudy
1 the initia sts tins
and linn)
h

O2

To&
O3 02

o72 o15 o.o85)

() 'o:&
o2
eP(o72
Db

\o.124S o92
 62

O2

Also
we
ll4 U,3e
S7

(5)
te (l4
51

3} let xn; n 2, 3,-3 be M:c en te stala space

a

S-,2, 33 wit, one step trasition poability mi

P= ti) Sketch The ansition diag 1am
b) Ts the chain ieducible ? xplain
 2
P Va
t1) (2)
23

the chain 'n imeducile

Va 4
(2)

period ot state 1.
d: Ge2, 3, +,4 54!
2GD2, 3, 4, 5.3 =)
3

.'. Al stales ae

All the shas aye non -null pesistant

Sine al tee statas
persistant, tey
A) Csnider (o,1,2) haing Erangi tion matix

ca) show tat te chain' irredulibe.

b Find the e
ey Find 4Re stinay dubibdion
 Tit+2= Also
2
Tt
Then
dstihuti stationay the Lat (es
.jl aeoie,, peied
2.
ste
5,3!peiodGeof3
S) 22
,*heI periotGcl
2,42,
stato
(3)
inedueia chain
y The
t2) L2) a)
) (2)
 Selving these, o

5) The tpm a a Mic wth three stales o, I, 2

P Va
and the initil stata iatrbin
VA
a a the chain is Pix.:ij4i o,, 2

Given P 4

2.

(2) (2)

4 4 4 3
 3 48

c A students skudy fats aare as

Studies higt, he n o'; Sure not to

next On the othendand,the poob that ta
Aoes not tuo

the long un
n, Rhas tan does le study ?
Sol,

stab spaca studing, Nt

s , NT
NT
SS
Nt O4

The steady stata prob it s

UP=TT

Also ,+,
We

the student studies A

time
T) Asalecman taietony Cns sts of tfree cties A, b and
He hever Sell 'n he Same city on Suc cessive dau
T he sells in city A, 4Ren the ezt Aay
B. lHowever, T he sells ib eitke
he stoice as ikay to sl 'n city Aay in t
sten
in
cy Yuh hos fan does e sell
ach f the cites ?
Sel: A
A

2l2
tet = (Wir T ) be the
staady stale foob diat
Then TP=T an

|22
3
3

these 2o
20 2

Re sells 4o the ime in

(
sTuo bs and B, nd too Gai and G are
a bal om to each other Each boy
ne to

unt oob y and to

each
taas the ball to each boy it boob %and have
un, hos oftan daes eack
Yeceive the bll

stalte Space ,B,,G, Ga3

The th

be the staady sl
prebabilit dist Then
2T 2

4these we

cach e ceiey the Lall time

1)|Suppose tat the foob o a dy day cstale o)
olleuing a rainy day (state 1) ab and tfat 4
oob ainy Holluing a

Given that tay a bady day, find the pors

May
Sot
sale spae Dyg day, Rany day

D R

'ritial bob duhibutim i Fcl,)

(2)

Por tbat May a

(3)
P
(S) 255
P= 432
l73
432