Scribd Logo
Upload

Welcome to Scribd!

  • 23 views

    Lecture 25 - Steady-State Behavior of Markov Chains

    Uploaded by

    K Virtuoso
    1) Markov processes can have either discrete or continuous time and discrete state spaces. The Markov property means the probability of moving to the next state depends only on the current state. 2) States can be recurrent, meaning the process will return to that state infinitely often, or transient, meaning it will eventually move away and not return. Recurrent states form recurrent classes. 3) Within a recurrent class, states can be periodic if the process cycles through groups of states. 4) For a single recurrent class that is not periodic, the transition probabilities converge to a unique stationary distribution as time increases without limit. This stationary distribution satisfies the balance equations.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    Lecture 25 - Steady-State Behavior of Markov Chains

    Uploaded by

    K Virtuoso
    23 views10 pages
    1) Markov processes can have either discrete or continuous time and discrete state spaces. The Markov property means the probability of moving to the next state depends only on the current state. 2) States can be recurrent, meaning the process will return to that state infinitely often, or transient, meaning it will eventually move away and not return. Recurrent states form recurrent classes. 3) Within a recurrent class, states can be periodic if the process cycles through groups of states. 4) For a single recurrent class that is not periodic, the transition probabilities converge to a unique stationary distribution as time increases without limit. This stationary distribution satisfies the balance equations.

    Original Description:

    Original Title

    Lecture 25_ Steady-State Behavior of Markov Chains

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    1) Markov processes can have either discrete or continuous time and discrete state spaces. The Markov property means the probability of moving to the next state depends only on the current state. 2) States can be recurrent, meaning the process will return to that state infinitely often, or transient, meaning it will eventually move away and not return. Recurrent states form recurrent classes. 3) Within a recurrent class, states can be periodic if the process cycles through groups of states. 4) For a single recurrent class that is not periodic, the transition probabilities converge to a unique stationary distribution as time increases without limit. This stationary distribution satisfies the balance equations.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    23 views10 pages

    Lecture 25 - Steady-State Behavior of Markov Chains

    Uploaded by

    K Virtuoso
    1) Markov processes can have either discrete or continuous time and discrete state spaces. The Markov property means the probability of moving to the next state depends only on the current state. 2) States can be recurrent, meaning the process will return to that state infinitely often, or transient, meaning it will eventually move away and not return. Recurrent states form recurrent classes. 3) Within a recurrent class, states can be periodic if the process cycles through groups of states. 4) For a single recurrent class that is not periodic, the transition probabilities converge to a unique stationary distribution as time increases without limit. This stationary distribution satisfies the balance equations.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 10

    Markov processes - II

    • review and some warm-up

    - definitions, Markov property


    - calculating the probabilities of trajectories
    • steady-state behavior
    -_.
    recurrent states, transient states,
    .
    recurrent classes
    .

    periodic states )

    convergence theorem )

    - balance equations )

    • birth-death processes V­
    1
    •
    review

    • discrete time, discrete state space,


    - transition probabilities f"':j ~
    - Markov property we l(S"" JI
    =
    = f~
    • rij (n) = P (X n "\" j I Xo ~ i) .
    = P (Xnt s = J I X s = , )
    'k -I r~j I) =- f~ j I "" ~ t.

    • key recursion:
    ..5+1
    I I
    , / ",+5
    ",,+$-1
    2
    warmup ,
    ...,....ll1p '-..Ii CIt n..lc.
    jr( B 1\ c. /I PI\-) .::.
    i( (t~ I~)" I (C:, All 8 )
    x ,?(.P IAllg/lt')
    ,- - ­
    -

    3
    review: recurrent and transient states

    • state i is recurrent if "starting from i, and from wherever you can go, there is a
    way of returning to i" Pe.<: •
    • if not rec
    ~
    .6 - N I
    0QU
    I
    3

    • recurrent class: a lIection of recurrent states communicating only between


    each other
    4
    periodic states in a recurrent class

    The states in a recurrent class are periodic if they can be grouped into d > I

    groups so that all transitions from one group lead to the next group

    .,~ ~1° 3l.. 1

    "'~1..

    5
    •

    _ .. s J..,-...Ja.J ~ ..: •
    L .. '
    • does Tij(n) - P (Xn ... = j I
    .
    Xo
    ... =
    ~
    i) converge to

    • theorem: yes, if:


    - recurrent states are all in a single class, and
    - single recurrent class is not periodic

    -
    ....
    - need also: L 1fj = 1 .
    j =1
    6
    example - .s;l.J(~ ('<""".,; c.t.-.rr
    0.5" '" 0 . 8
    - 'IoQ ftfl'oc/,"c,.
    0 .5
    2

    e "'
    '...."., '=' rr;" oS -4- Tr~. O.l
    r,,~
    '-
    "'- 71i" oS ~ Trz....

    f=~_ _-::::-

    TTI.l. ...
    L -

    "'if......~ = I

    7
    visit frequency interpretation

    • balance equations

    v ' L~ 7rkPkj
    ~ k "I

    I
    • (long run) frequency of being in j : 7rj
    I I,
    11'111'11'" ..... 11
    , YjC... 1 1\/ T'

    J
    I 1. l .. f • ,. , ,.
    '7
    -ff-«-
    ,.
    .,. "'--«t1­
    I'\A

    • frequency of transitions 1 --t j : 7r l P l j


    111, 111111,., . .... ,11 =;> Z'j("') 7r;'f'~
    I LJ ""' ''' .. -fi-c... 11 U
    • frequency of transitions into j: kPkj
    k
    8
    1
    birth-death processes I ,100
    .
    -4'~ ~
    I - PO I - PI - q l
    ,
    I

    PO
    ,
    I

    o 3 • -,. on
    ,
    '1m

    i '
    7i"::. + I = 7r-<")<-t.i
    ~ ..:.. '
    /fo '"7t,- / :> TTL , -, 11i
    :;:. / 110?
    4- '/I.; - I ~ 110 .' 11';- .. ~ ... IT;. ... ,...:..Ii +- .... - ::./

    U ,., ", ,z.


    9
    I · ~ ..

    birth-death processes II Pi ;

    _ >-.:/
    1 ••• 0 [8 1f , p, = 'II", +lQ, +l
    L, lI'J =
    "'IC" == It. ,. £ '{Ie[' +f+ -. J;/ 1

    ~'"
    special case: Pi = P and qi = q for all i
    P • "'TT: lr. - 2­
    p = pl q 1I"i+1 = 1I"i­ = 1I"iP IT,:. " o/ ' , / ' t . ' " / <:! /1;;// -'--­
    .
    7fi = 7ropt 'l... = a, 1 , ... ,m
    •
    I
    • assume P < q and m "" 00 1-/
    (in steady-s
    , .... • •

    10

    You might also like