Markov process

Page 1

Markov Process

Neelkant Newra(18111040)

A stochastic process whose evolution after a given time t does not depend on the evolution
before t , given that the value of the process at t is fixed. A Markov chain is a discrete-time
process for which the future behaviour, given the past and the present, only depends on the
present and not on the past. A Markov process is the continuous-time version of a Markov chain.

In a Markov process we have a discrete set of state S . However, the transition behaviour is
different from that in a Markov chain. In each state there are a number of possible events that can
cause a transition. The event that causes a transition state i to j , where j is not equ al to i ,
takes place after an exponential amount of time, say with parameter qij . As a result, in this model
transitions takes place at random points in time. According to the properties of exponential
random variables

In state i a transition takes place, after an exponential amount of time, with parameter

∑
qi j

j≠i
The system makes a transition to state j with probability

∑
pij := qij / qik

k≠i
And

∑
qii := − qij , i ϵ S

j≠i

The matrix Q with element qij is called the generator of the Markov process. The definition of qij
is in state i at time t converses to a limit π as t tends to infinity. The randomness of the time
system spends in each state guarantees that the probability Pi(t) converse to the limit π . The
limiting probabilities can again be computed from the balance equations. If the system is in state
i, then events that cause the system to make transition to state j occurs with a frequency or rate
qij . So the mean number of transition per time unit from i to j is equ al to pi qij . This leads to
the balance equations

∑ ∑
Pi qij = pj qji, i ϵS

j≠i j≠i

∑
or 0= pj qji

jϵS

In vector-matrix notation this becomes, with p the row vector with element π

PQ = 0

The solution of the set of this equation is unique

Together with the normalised equation is

∑
Pi = 1
iϵS