UNIT III MARKOV PROCESS AND MARKOV CHAINS

PART A (QUESTION AND ANSWERS)

1. Consider the random process
( ) + t t X cos ) (
where

is a random variable
with density function ( )
2 2
,
1

< <

f . Check whether or not the process is

wide sense stationary.
Soln:
Given ( ) + t t X cos ) ( and ( )
2 2
,
1

< <

f
{ }
{ }
{ }
t cons t
t t
t
d t
d f t X t X E
tan cos 2
1
) cos ( cos
1
sin(
1
) cos(
1
) ( ) ( ) (
2
2
2
2
2
2


+
+

So, X(t) is not WSS.

2. Consider a random process ( ) + wt t X cos ) ( , where w is a real constant
and is a uniform variable in ( )
2
, 0

. Show that X(t) is not wide sense
stationary.
Soln:
Given ( ) + wt t X cos ) ( .
Since is uniformly distributed in ( )
2
, 0

we have
2
0 ,
2
) (

< < f
{ }
{ }
{ }
t cons
wt wt
wt
d wt
d f t X t X E
tan
) sin cos
1
sin(
2
) cos(
2
) ( ) ( ) (
2
0
2
0
2
0

+
+

So, X(t) is not WSS.

3. Define a Random Process.
Soln:
A random process or Stochastic process X(s,t) is a function that maps each element of a
sample space into a time function called sample function.
4. Define First order stationary and Second order stationary process.
Soln:
A random process is said to be stationary to order one if the first order density functions
defined for all the random variables of the process are same.
A random process is said to be stationary to order two if for all
and t t
2 1
,
its second
order density functions satisfy the condition
( ) ). , ; , ( , ; ,
2 1 2 1 2 1 2 1
+ + t t x x f t t x x f
x x
5. Define Markov Process
Soln:
A random process or Stochastic process X(t) is said to be a Markov process if given the
value of X(t), the value of X(v) for v > t does not depend on values of X(u) for u < t. In
other words, the future behavior of the process depends only on the present value and not on
the past value.
6. Define Markov Chain
Soln:
A Markov process is called Markov chain if the states
{ }
i
X
is discrete no matter whether
t is discret or continuous.
7. Define Chapman-Kolmogrov Equation
Soln:
The Chapman-Kolmogrov equation provides a method to compute the n-step transition
probabilities. The equation can be represented as

+

0
0 ,
k
m
kj
n
ik
m n
ij
m n p p P .
8. When do you say that a Markov chain is irreducible?
Soln:
The Markov chain is irreducible if all states communicate with each other at some time.
9. When do you say the Markov chain is regular?
Soln:
A regular Markov chain is defined as a chain having a transition matrix P such that for
some power of P, it has only non-zero positive probability values.
10. When do you say that state i is periodic and aperiodic?
Soln:
Let A be the set of all positive integers such that 0
) (
>
n
ii
p and d be the Greatest Common
Divisor(G.C.D.) of the set A. We say state i is periodic if d>1 and aperiodic if d = 1.
11. Define Poisson process.
Soln:
Poisson process is a counting point process representing the number of occurrences of
certain event in a finite collection of non overlapping statistically independent time arrivals.
12. What are the properties of Poisson process.
Soln:
(a) The poisson process is not a stationary process. It is vivid from the expressions of
moments of poisson process that they are time dependent.
(b) The poisson process is a Markov process.
13. Determine whether the given matrix is irreducible or not.
1
1
1
]
1

8 . 0 2 . 0 0
5 . 0 4 . 0 1 . 0
0 7 . 0 3 . 0
P
Soln:
1
1
1
]
1

8 . 0 2 . 0 0
5 . 0 4 . 0 1 . 0
0 7 . 0 3 . 0
P
and
1
1
1
]
1

74 . 0 24 . 0 02 . 0
60 . 0 33 . 0 07 . 0
35 . 0 49 . 0 16 . 0
2
P
Here
j i P
n
ij
, , 0
) (
>
. So, P is irreducible.
14. When do you say the Markov chain is homogeneous?
Soln:
If the one-step transition probability does not depend on the step i.e.
) , 1 ( ) , 1 ( m m p n n p
ij ij

then the Markov chain is called a homogeneous Markov chain.
15. What are the different types of Random process.
Soln:
Continuous Random process, Discrete Random process, Continuous Random
sequence and Discrete Random sequence.
16. Define Birth and Death process.
Soln:
If X(t) represents the number of individuals present at time t in a population in
which two types of events occur one representing birth which contributes to
its increase and the other representing death which contributes to its decrease,
then the discrete random process {X(t)} is called the birth and death process.
17. A housewife buys 3 kinds of cereals A, B and C. She never buy the same
cereal in successive weeks. If she buys cereal A, the next week she buys B.
However if she buys B or C the next week she is 3 times as likely to buy A as
the other cereal. Construct the Transition Probability Matrix.
Soln:
1
1
1
1
]
1

4
1
0
4
3
4
1
0
4
3
0 1 0
C
B
A
P
C B A
18. Let
1
]
1

2 / 1 2 / 1
1 0
A
be a Stochastic matrix. Check whether it is regular.
Soln:
1
]
1

2 / 1 2 / 1
1 0
A
and
1
]
1

4 / 3 4 / 1
2 / 1 2 / 1
2
A
.
Since all entries in
2
A
are positive, A is regular.
19.The number of particles emitted by a radioactive source is Poisson distributed.
The source emits particles at the rate of 6 per minute. Each emitted particle has a
probability of 0.7 of being counted. Find the probability that 11 particles are
counted in 4 minutes.
Soln:
The number of particles N(t) emitted is poisson with parameter
p
= 6(0.7) =
4.2
( )
!
2 . 4
) ) ( (
2 . 4
m
t e
m t N P
m t

( )
. 038 . 0
! 11
) 4 ( 2 . 4
) 11 ) 4 ( (
11 ) 4 ( 2 . 4

e
N P
20.Prove that the sum of two independent Poisson process is a Poisson process.
Soln:
Let ) (
1
t X and ) (
2
t X be the Poisson process with mean
2 1
and
respectively.
) 1 (
) (
1
1
) (
iw
e t
t X
e w



and
) 1 (
) (
2
2
) (
iw
e t
t X
e w

Since
1
X and
2
X are independent,

+
) (
) ( ) (
2 1
w
t X t X

) (
) (
1
w
t X

.
) (
) (
2
w
t X

=
) 1 (
1
iw
e t
e


) 1 (
2
iw
e t
e

=
) 1 ( ) (
2 1
iw
e t
e
+
Since
1
X +
2
X follows Poisson Distribution with mean .
2 1
+
PART B
01. Define Random process. Specify the four different types of Random Process and
give an example to each type.
02. Prove that the difference of two independent Poisson Process is not a Poisson
Process.
03. The transition probability matrix of a Markov chain{ }
n
X
, n = 0,1,2,3
L
having 3
states 1, 2 and 3 is
0.1 0.5 0.4
p 0.6 0.2 0.2
0.3 0.4 0.3
1
1

1
1
]
and the initial distribution is ( ) p(0) 0.7, 0.2, 0.1
.
Find { }
2
P X 3
and { }
3 2 1 0
P X 2, X 3, X 3, X 2
.
04. A random process { } X(t), t T
has the probability distribution
{ }
( )
( )
n 1
n 1
at
, n 1, 2, 3,
1 at
P X(t) n
at
, n 0
1 at

+

'

+
L
. Show that the process is not stationary.
05. A man either drives a car or catches a train to go to office each day. He never goes two
days in a row by train but if he drives one day, then the next day he is just as likely to
drive again or he is travel by train. Now suppose that on the first day of the week, the
man tossed a fair die and drove to work if and only if 6 appeared. Find the
probability that he takes a train on the third day. Also find the probability
that h drives to work in the long run.