3.

Some Markov Chain Models

3.1 An Inventory Model
the quantity on hand at the end of period n .
the quantity demanded in the nth period.
stock size
A replenishment policy is prescribed by specifying two
nonnegative critical numbers s and whose interpretation is, if the end
of period stock quantity is not greater than s, then an amount sufficient
to increase the quantity of stock on hand up to the level S is
immediately procured. If, however, the available stock is in excess of s,
then no replenishment of stock is undertaken.
Example.

Suppose
Possible values for
To find the transition probability matrix.
Exercises
3.1. Consider a spare parts inventory model in which either 0,1, or 2
repair parts are demanded in any period, with
and suppose and Determine the transition probability matrix for the
Markov chain where is defined to be the quantity on hand at the end of
period n.
Soln:
and

Let
3.2 The Ehrenfest Urn Model
Let the number of balls in urn A at the nth stage
is a Markov chain on the states
with transition probabilities.

Eg. Total
Box A at n time, Box B

𝑖 𝑑−𝑖
𝑑 𝑑
Ex- 3.2 Consider two urns A and B containing a total of balls. An
experiment is performed in which a ball is selected at random (all
selections equally likely) at time from among the totality of balls.
Then an urn is selected at random (A is chosen with probability and
B is chosen with probability ) and the ball previously drawn is
placed in this urn. The state of the system at each trial is represented
by the number of balls in A. Determine the transition matrix for this
Markov chain.
Soln.
Total balls
Probability that A is chosen
Probability that B is chosen
Box A , Box B
3.3 Markov Chains in Genetics
• genes
• a-genes
• A-genes

The state space contains the values The transition probability matrix
is computed according to the binomial distribution.
3.4 A Discrete Queueing Markov Chain
Suppose that the actual number of customers that arrive during the nth
period

Pr{k customers arrive in a service period} for

Where and

Assume that are independent random

Where is the number of new customers having arrived in this period

Where
Ex- 3.5
Soln:
Let the number of red balls in the urn after n drawn, with
4. First Step Analysis
4.1 Simple First Step Analyses
Consider the Markov chain whose transition probability matrix is

, where and .
Markov chain begins in state 1 (transient), and
state 0 and 2 are absorption states.

(T is the time of absorption of process)

In terms of this random absorption time the two questions ask us to find
Considering separately the three contingencies and .

mean time to absorption

- absorption time T is always at least 1.
Eg . Four-State Markov Chain
Eg . Four-State Markov Chain
(1)

(2)
By solving eq(1) & eq(2)

(3)

(4)
By solving eq(3) & eq(4)
General,

Example A Maze
A white rat is put into the maze shown:

,
4.2 The General Absorbing Markov Chain

( random absorption time)

Transient state i
Rate
Mean total amount
Initial rate
Mean time rate

Mean time duration

Example A Model of Fecundity
Prepuberty Divorced
Single Widowed
Married
Interested in mean duration spent in married, to illustrate the
computations
Given: transition probability matrix

Soln.
So (1)

(2)
 (3)

(4)

(5)
Eq(4) & Eq(5) in Eq(3)

Ex: 4.1- 4.9 , Problem 4.5