MODUL PERKULIAHAN

Model Stokastik

Markov Chain (1/2)

Fakultas Program Studi Tatap Muka Kode MK Disusun Oleh

Fakultas Teknik Teknik Industri 05510004 Dr. Nova Indah Saragih, S.T., M.T.
09
Abstrak Kompetensi
Modul ini membahas proses Mahasiswa mampu menerapkan
stokastik, klasifikasi state dalam konsep Markov Chain.
Markove Chain, kondisi steady state
dari Markov Chain.
The preceding chapter focused on decision making in the face of uncertainty about one
future event (learning the true state of nature). However, some decisions need to take into
account uncertainty about many future events. We now begin laying the groundwork for
decision making in this broader context.

In particular, this chapter presents probability models for processes that evolve over time in
a probabilistic manner. Such processes are called stochastic processes. After briefly
introducing general stochastic processes in the first section, the remainder of the chapter
focuses on a special kind called a Markov chain. Markov chains have the special property
that probabilities involving how the process will evolve in the future depend only on the
present state of the process, and so are independent of events in the past. Many processes
fit this description, so Markov chains provide an especially important kind of probability
model.

Stochastic Processes

Let Xt be a random variable that characterizes the state of the system at discrete points in
time t = 1,2, ... The family of random variables {Xt} forms a stochastic process. The number
of states in a stochastic process may be finite or infinite, as the following two examples
demonstrate:

Example 1 (Machine Maintenance)

The condition of a machine at the time of the monthly preventive maintenance is
characterized as fair, good, or excellent. For month t, the stochastic process for this situation
can be represented as:

The random variable Xt is finite because it represents three states: poor (0), fair (1), and
good (2).

Example 2 (Job Shop)

Jobs arrive randomly at a job-shop at the average rate of 5 jobs per hour. The arrival
process follows a Poisson distribution which, theoretically, allows any number of jobs
between zero and infinity to arrive at the shop during the time interval (0, t). The inifinte-state
process describing the number of arriving jobs is

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

2 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
A stochastic process is defined to be an indexed collection of random variables {Xt}, where
the index t runs through a given set T. Often T is taken to be the set of nonnegative integers,
and Xt represents a measurable characteristic of interest at time t. For example, Xt might
represent the inventory level of a particular product at the end of week t.

Stochastic processes are of interest for describing the behavior of a system operating over
some period of time. A stochastic process often has the following structure.

The current status of the system can fall into any one of M + 1 mutually exclusive categories
called states. For notational convenience, these states are labeled 0, 1, . . . , M. The random
variable Xt represents the state of the system at time t, so its only possible values are 0, 1, .
. . , M. The system is observed at particular points of time, labelled t = 0, 1, 2, . . . . Thus, the
stochastic process {Xt} = {X0, X1, X2, . . .} provides a mathematical representation of how
the status of the physical system evolves over time.

This kind of process is referred to as being a discrete time stochastic process with a finite
state space.

Markov Chains

A stochastic process is a Markov process if the occurrence of a future state depends only on
the immediately preceding state. This means that given the chronological times t0, t1 ... , tn
the family of random variables {Xt} = {X0, X1, X2, . . .} is said to be a Markov process if it
possesses the following property:

In a Markovian process with n exhaustive and mutually exclusive states (outcomes), the
probabilities at a specific point in time t = 0, 1,2, ... is usually written as

This is known as the one-step transition probability of moving from state i at t - 1 to state j at
t. By definition, we have

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

3 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
A convenient way for summarizing the one-step transition probabilities is to use the following
matrix notation:

TIle matrix P defines the so-called Markov chain. It has the property that all its transition
probabilities Pij are fixed (stationary) and independent over time.

Example 3 (A Stock Example)

Consider the following model for the value of a stock. At the end of a given day, the price is
recorded. If the stock has gone up, the probability that it will go up tomorrow is 0.7. If the
stock has gone down, the probability that it will go up tomorrow is only 0.5. This is a Markov
chain, where state 0 represents the stock’s going up and state 1 represents the stock’s going
down. The transition matrix is given by

Example 4 (The Gardener Problem)

Every year, at the beginning of the gardening season (March through September), a
gardener uses a chemical test to check soil condition. Depending on the outcome of the test,
productivity for the new season falls in one of three states: (1) good, (2) fair, and (3) poor.
Over the years, the gardener has observed that last year's soil condition impacts current
year's productivity and that the situation can be described by the following Markov chain:

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

4 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
The transition probabilities show that the soil condition can either deteriotate or stay the
same but never improve. If this year's soil is good (state 1), there is a 20% chance it will not
change next year, a 50% chance it will become fair (state 2), and a 30% chance it will
deteriorate to a poor condition (state 3). If this year's soil condition is fair (state 2), next
year's productivity may remain fair with probability .5 or become poor (state 3), also with
probability .5. Finally, a poor condition this year (state 3) can only lead to an equal condition
next year (with probability 1). The gardener can alter the transition probabilities P by using
fertilizer to boost soil condition. In this case, the transition matrix becomes:

The use of fertilizer now allows improvements in the deteriorating condition. There is a 10%
chance that the soil condition will change from fair to good (state 2 to state 1), a 5% chance
it will change from poor to good (state 3 to state 1), and a 40% chance that a poor condition
will become fair (state 3 to state 2).

Example 5 (A Second Stock Example)

Suppose now that the stock market model is changed so that the stock’s going up tomorrow
depends upon whether it increased today and yesterday. In particular, if the stock has
increased for the past two days, it will increase tomorrow with probability 0.9. If the stock
increased today but decreased yesterday, then it will increase tomorrow with probability 0.6.
If the stock decreased today but increased yesterday, then it will increase tomorrow with
probability 0.5. Finally, if the stock decreased for the past two days, then it will increase
tomorrow with probability 0.3. If we define the state as representing whether the stock goes
up or down today, the system is no longer a Markov chain. However, we can transform the
system to a Markov chain by defining the states as follows:

State 0: The stock increased both today and yesterday.

State 1: The stock increased today and decreased yesterday.
State 2: The stock decreased today and increased yesterday.
State 3: The stock decreased both today and yesterday.

This leads to a four-state Markov chain with the following transition matrix:

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

5 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
Example 6 (A Gambling Example)
Another example involves gambling. Suppose that a player has $1 and with each play of the
game wins $1 with probability p > 0 or loses $1 with probability 1 - p. The game ends when
the player either accumulates $3 or goes broke. This game is a Markov chain with the states
representing the player’s current holding of money, that is, 0, $1, $2, or $3, and with the
transition matrix given by

Absolute And n-Step Transition Probabilities

Given the initial probabilities a(0) = {aj(o)} of starting in state j and the transition matrix P of a
Markov Chain, the absolute probabilities a(n) = {aj(n)} of being in state j after n transitions (n >
0) are computed as follows:

Continuing in the same manner, we get

The matrix Pn is known as the n-step transition matrix. From these calculations we can see
that

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

6 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
or

These are known as Chapman-Kolomogorov equations.

Example 7
The following transition matrix applies to the gardener problem with fertilizer:

The initial condition of the soil is good-that is a(0) = (1,0,0). Determine the absolute
probabilities of the three states of the system after 1,8, and 16 gardening seasons.

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

7 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id
Thus,

The rows of P8 and the vector of absolute probabilities a(8) are almost identical. The result is
more pronounced for P16. It demonstrates that, as the number of transitions increases, the
absolute probabilities are independent of the initial a(0). In this case the resulting probabilities
are known as the steady-state probabilities.

Daftar Pustaka

1. Hillier, F. S. and Lieberman, G. J. (2002): Introduction to Operations Research, Seventh

Edition, McGraw-Hill Science.
2. Taha, H. A. (2007): Operations Research: An Introduction, Eighth Edition, Pearson
Prentice Hall.

‘20 Model Stokastik Biro A kademik da n Pe mbe lajaran

8 Dr. Nova Indah Saragih, S.T., M.T. http://www.widyatama.ac.id