Modelling using Markov Chains
S.R.Rimitha, Vedasamhitha Abburu, Saumya Gupta
November 11, 2016
Abstract
In this document, we aim to study and ex-
plain Markov Chain modelling as a useful
tool in analysing computer networks for re-
liability, availability, and various other per-
formance metrics. Modelling of computer
systems and networks is becoming very nec-
essary, as a way to quantitatively analyse
the system, without relying only on intu-
itive arguments. Markov chains offer fairly
simple yet effective and practical modelling
solutions. The aim of this document is to
study these topics in detail and also learn
about their practical applications. These ap-
plications are very varied, and include using
Markov chains in the web for link prediction
in adaptive websites, page ranking, which is
a popular algorithm used by search engines
to list relevant web pages, modelling of queu-
ing networks, as well as performance analysis
of the MAC layer of WLANs.
Keywords: Markov chains, IEEE 802.11,
IEEE 802.15.4, Backoff algorithm, perfor-
mance analysis, throughput, access delay,
CSMA/CA, link prediction, queueing net-
works, PageRank
1
1 Learning Objective
Through our study, we hope to fully under-
stand the working of Markov Chain Models,
as well as to learn the various practical appli-
cations of Markov Chain Modelling in com-
puter networks. We aim at understanding
how the various computer network applica-
tions can be modelled using Markov mod-
els, as well as how these models can be used
to analyze and evaluate the network perfor-
mance. Markov models have been widely
used in real time applications and we hope
to have a fair understanding of the working
and implementation of the Markov models in
all such real time applications, as well as the
advantages of using such models over other
analytical methods and simulations.
2 Literature Survey
This section discusses the work done by var-
ious researchers in regard to Markov chain
modelling in wireless computer networks.
2.1 IEEE 802.11
In IEEE 802.11, Distributed Coordination
Function (DCF) is defined as the fundamen-
tal MAC layer standard for WLAN. DCF
implements the CSMA/CA collision avoid-
ance method of network access, and also uses
an exponential backoff (usually binary) algo-
rithm as the standard.
The method in which DCF works can be de-
scribed in the following way: when a sta-
tion wants to start transmitting, it senses
the medium, and checks whether or not it is
busy. If the link is sensed to be busy, the sta-
tion waits, and executes a backoff procedure.
Once the procedure has been completed, the
station can again try to transmit. This back-
off procedure has been specified in order to
reduce the chances of packets colliding in the
medium when multiple stations attempt to
transmit at the same time.
The standard backoff algorithm specified in
the IEEE 802.11 standard is the binary ex-
ponential backoff (BEB) algorithm.
Giuseppe Bianchi was among the first few re-
searchers to begin analysing the performance
of the BEB algorithm. His work ([2] and
[3]) has paved the way for not only devel-
opment in new models for analysing the ex-
isting algorithms, but also proposal of new
algorithms to improve the current ones. He
is truly a pioneer in this field.
In his papers [2], [3], Bianchi proposed to
derive ‘saturation throughput’ using a two-
dimensional Markov model. His work stems
slightly away from real-world implementa-
tion because of his assumption of saturation
2
condition and an error-free channel. Under
the assumption of saturation condition, ev-
ery station always has a packet to trans-
mit. That is, after a successful transmis-
sion, the station again has a packet ready
to be sent. This is not the case in the real
world, but this assumption is useful as it
gives the network more traffic than the usual;
thereby setting an upper bound for this pa-
rameter. This paper includes derivations for
both variations of the CSMA/CA protocol
– one which uses ACK, and another which
uses RTS/CTS handshaking.
Papers [21] and [9] fall back extensively on
Bianchi’s work. The environment in both the
papers operates under saturation conditions,
and assumes a noiseless channel. These pa-
pers have investigated the medium access de-
lay as well.
Paper [9] has deployed the use of a one-
dimensional Markov model to capture the
algorithm, and does so only for the ACK
mechanism. The analysis has gone beyond
BEB to consider the general case of Expo-
nential Backoff (EB) with a backoff factor r.
BEB algorithm is but a special case with r
= 2. The authors have further compared the
performance parameters in the presence and
absence of a maximum retry limit M. This
limit forces the dropping of a packet after it
has been involved in M collisions. This as-
sumption is significant as real-world imple-
mentations usually set M as 16. Paper [5]
also takes into account a retry limit; how-
ever, it deploys a two-dimensional model to
do so.
In paper [17], [1], two new algorithms that
can execute the backoff procedure have been
suggested. These are the Exponential In-
crease Exponential Decrease (EIED), and
the Exponential Increase Linear Decrease
(EILD) algorithms, and have been modelled
using a suitable Markov model. In [20], a
new analytical Markov model of the BEB,
as well as the EIED and the EILD has been
proposed. Both have been analysed to find
the average delay.
In paper [6], the authors propose a dynam-
ically “tuned” backoff algorithm, which es-
timates the network parameters to dynam-
ically calculate and set the optimal backoff
window size. A suitable Markov model has
been used to analyse the performance of this
algorithm.
The above papers have all considered only
saturated conditions. New Markov models
have been proposed in [12] and [18] that aim
to simulate a more realistic network load,
i.e., non-saturated network conditions.
2.2 IEEE 802.15.4
In this section we are going to look at a new
IEEE standard i.e. IEEE 802.15.4 which is
used to specify protocols for Wireless Per-
sonal Area Networks especially low rate ones.
It is used to specify the PL (Physical Layer)
and MAC(Media Access Control) for the
above specified network. It was a relatively
new standard being released in 2003. Its ma-
jor attention is to lower the cost and present
a low speed ubiquitous communication be-
tween nearby devices. It mostly is used for
embedded devices with a 10 meter range and
3
transfer rate as 250 kbit/second.
The focus of this architecture is to establish
a simple wireless network conceptually. It is
noteworthy to mention here that this stan-
dard has focused a lot on IEEE 802.11 while
developing certain implementations. The
network layers were mostly based on the OSI
model. There are various kinds of topolo-
gies possible using this standard to create
a PAN. However the most user is a peer to
peer topology or star topology. The most
important aspect with respect to this IEEE
standard is its reliability which is promised
through the CSMA/CA protocol which uses
random exponential backoff algorithm.
This report focuses on the role of Markov
models that has been used while propos-
ing various analytical models for CSMA/CA
(Carrier Sense multiple access/Collision
Avoidance) protocol for the above mentioned
IEEE 802.15.4 standard by various research-
ing teams. Though the CSMA/CA proto-
col used in this standard is extremely similar
to the one used in wireless LANs i.e. IEEE
802.11, there is one major difference. This is
the backoff algorithm used in 802.11 where
when the channel is idle, the count is reduced
by 1 whereas in 802.15.4 the count is lessened
by one regardless of the channel state.
This technical difference has been addressed
by a number of papers. [12] has addressed
this issue by proposing an analytical Markov
model for CSMA/CA in saturated condi-
tions. This was followed by [18] where they
have proposed a Markov model which modi-
fies the Double Carrier Sensing protocol used
in the MAC Layer to yield better perfor-
mance results. Paper [14] have also proposed
a model for the same aspect with different
aspects into consideration and various dif-
ferent assumptions. It must be noted here
that papers [12],[18],[14],[10] and [15] sug-
gest analytical models for CSMA/CA with
different constants taken into consideration.
For instance, the papers mentioned above
mostly went along with the assumption that
the Contention Access Period(CAP) is infi-
nite which is not the condition always. [15]
has its own assumptions while proceeding
with the model which might not hold true
at every situation. Paper [7], which we have
focused upon , has tried to specify a Markov
model for the same solution trying to over-
come all the drawbacks of its predecessors.
3 Outcome of Literature
Survey
From the literature survey, we deduce that
extensive work using Markov modelling has
been carried out in 3 major areas:
• Analyzing performance metrics
(throughput and delay) in the ex-
isting backoff algorithms of IEEE
802.11.
• Improving the existing algorithms, and
analyzing their metrics as well.
• Analyzing performance metrics in the
algorithms of IEEE 802.15.4. Conse-
quently, we present a gap analysis of the
work done in each of these areas.
4
3.1 Existing Backoff Algo-
rithms
The work carried out by Bianchi in his pa-
pers [2] and [3] has been used as the standard
by a lot of other researchers. Paper [3] analy-
ses the binary backoff algorithm and derives
an expression for throughput using a two-
dimensional Markov model. The paper at-
tempts to cap the value of throughput by as-
suming constant traffic in the network. The
following are the assumptions in this paper:
• Saturation conditions: Under this as-
sumption, every station always has a
packet to transmit. That is, after a suc-
cessful transmission, the station again
has a packet ready to be sent. This is
not the case in the real world, but this
assumption is useful as it gives the net-
work more traffic than the usual.
• Finite number of stations: This assump-
tion sticks correctly with current imple-
mentations.
• Noise-free channel: The paper assumes
an ideal link with no errors. As we
know, no media is completely free from
errors.
• Infinite attempts: There is no limit on
the maximum number of times a sta-
tion can resend a packet. Hence, a
packet will never be dropped. This
stems away from current implementa-
tions which usually set a maximum retry
limit of 16.
 Through mathematical manipulation,
Bianchi derived the following dependencies:
• For the protocol making use of the ACK
mechanism: The backoff window size
is usually the propagation delay. In
this mechanism, throughput increases
with increase in the minimum window
size. In simple terms, the probability
for number of collisions decreases when
the waiting period at each station in-
creases. Hence, throughput increases.
As far as the number of stations is con-
cerned, throughput is indirectly propor-
tional to it. That is, as the network
expands and grows, the throughput re-
duces.
• For the protocol making use of the
RTS/CTS mechanism: The paper
shows that throughput reduces with in-
crease in minimum contention window
size. Furthermore, in this mechanism,
the throughput remains largely unaf-
fected by the number of stations in the
wireless network.
The paper finally settles upon the result
that the handshaking mechanism performs
better than the basic method of acknowl-
edgement in large networks.
Paper [9] paper analyses the performance of
the Exponential Backoff (EB) algorithm, op-
posed to Binary Exponential Backoff (BEB)
analysed in paper [3]. In addition to satura-
tion throughput, the paper also investigates
the medium access delay of this algorithm.
The delay is the time between its arrival at
the MAC layer to the time it is successfully
5
transmitted out onto the network.
The paper derives formulae for the two per-
formance parameters under the following as-
sumptions:
• Saturation condition is in place.
• There are no errors in the channel.
• Number of nodes in the network is a fi-
nite constant.
• The ACKs are part of the data packets,
and are not explicitly sent. No hand-
shaking mechanism.
While Bianchi’s paper makes use of two-
dimensional Markov chain models to ana-
lyze the throughput, this paper uses a one-
dimensional Markov chain. The paper anal-
yses the algorithm in the presence as well as
absence of a retry limit M, as opposed to pa-
per [3] which ignores this limit. The limit
dictates that a packet should be dropped if
it is involved in more than M collisions.
The findings from this paper are:
• For EB without a maximum retry limit:
It has been shown that throughput re-
mains largely independent of the num-
ber of nodes in the network. The
medium access delay, however, is di-
rectly proportional to the number of
nodes.
• For EB with a maximum retry limit M:
As the number of stations in the net-
work increases, the throughput begins
to drop. The medium access delay in-
creases (not linearly) with increase in
number of nodes.
The algorithm setting a limit of M is usu-
ally deployed in real-world implementations.
The perk of this is that the other packets
awaiting transmission will not be kept under
indefinite hold.
Paper [21] implements a two-dimensional
Markov process to capture the Binary Ex-
ponential Backoff algorithm. In addition to
deriving an expression for throughput, this
paper accounts for the variation of access de-
lay with respect to the system variables.
The paper applies the Markov model to two
variations of the CSMA/CA protocol – one
which uses an acknowledgement mechanism
[ACK] for verifying successful transmissions,
and another which uses a handshaking mech-
anism [RTS (Request to send)/CTS (Clear
to send)]. The performance parameters have
been derived for both mechanisms. This pa-
per investigates the response of throughput
and delay with respect to changes in the net-
work parameters such as number of nodes,
channel bit rates, and contention window
size. The following are the outcomes of this
paper:
1. For the protocol making use of the ACK
mechanism:
• As the number of nodes in the net-
work increases, the throughput de-
creases.This is because, greater the
number of nodes, greater is the
probability for collision. As the
minimum contention window size
increases, so does the throughput.
Throughput decreases as the chan-
nel bit rate increases.
6
• The delay increases as the mini-
mum contention window increases
because retransmission will take
longer now. This delay is more
as compared to RTS/CTS. The de-
lay increases as the minimum con-
tention window increases. As the
channel bit rate increases, delay re-
duces. It also increases as the num-
ber of stations increases, and this
increase is more drastic as com-
pared to the below protocol.
2. For the protocol making use of the hand-
shake mechanism:
• The number of nodes in the net-
work does not have significant im-
pact on the throughput. The
throughput begins to dip as the
minimum contention window size
begins to increase. Throughput de-
creases as the channel bit rate in-
creases.
• The delay increases as the mini-
mum contention window increases.
As the channel bit rate increases,
delay reduces. It further increases
as number of stations in the net-
work increases.
We see that the results of this paper agree
with those obtained in paper [3] with regard
to throughput. Comparison for delay cannot
be made as paper [3] does not delve into its
study.
The paper concludes its findings by stating
that RTS/CTS has better throughput and
smaller delays as compared to the basic ac-
cess mechanism when the network load is
high. This agrees excellently with the out-
come of paper [3].
3.2 Variations in backoff algo-
rithms
The standard defined by IEEE 802.11 for
media access control, is DCF, which uses
BEB algorithm to execute the necessary
backoff procedure. However, the binary
backoff algorithm causes a station to reset
its contention window size after a successful
transmission. This results in a largely varied
contention window size. It also results
in a decrease in network performance,
as stations would have to start from the
minimum contention window size every time
a successful transmission occurs. When
there a lot of contention in the network
(many stations want to transmit over the
medium at the same time) this may result
in more collisions, as each new packet at
a station would wait a very small amount
of time before being sent, and would most
likely collide with another packet already
being transmitted. Therefore, in heavy
traffic conditions, the resetting used in the
BEB would not be beneficial to network
performance.
In order to counter this, a new backoff
algorithm (EILD) is suggested in [1], where
the window size is increased exponentially
(multiplied by a specified constant) follow-
ing a collision, but is decreased linearly
(subtracted by one) following a successful
7
transmission. When the network is under
heavy load, this algorithm allows the net-
work to decrease the number of collisions,
and thus improve performance. However,
when the network is under a light load, oc-
casional and even infrequent collisions may
cause the station to increase the window size
exponentially, and require a larger period
of time for the window size to decrease
again. During this time the window size
is decreasing, the link may be idle. This
can reduce the performance of the network
under lightly loaded conditions.
Paper [17] proposes an algorithm (EIED)
where the window size decreases expo-
nentially as well following a successful
transmission. Compared to the linear
decreasing mechanism described by [1],
the EILD algorithm in [17] will be able to
quickly recover from infrequent collisions,
as the backoff window size decreases much
faster in EIED than in EILD. However, a
problem of EIED algorithm is that it may
be unfair to stations in the network with
small number of competing stations.
3.3 Analysis in IEEE 802.15.4
The most important predecessor for this
standard has been the IEEE 802.11. It must
be noted here that due to the difference
in the nature of situations where the stan-
dard is applied, the backoff algorithm has
slight changed. For instance, the back off
algorithm is IEEE 802.15.4 decrements the
counter regardless of the state of the channel
while IEEE 802.11 does not perform that.
Another important aspect of this report is
the modelling of the backoff algorithm using
Markov chains in IEEE 802.15.4. Here it is
important to know that the parameters used
while designing the Markov model plays a
major role in differentiating them. For in-
stance paper [7] and [13] have only one dif-
ferentiating parameter i.e. paper [7] includes
the number of remaining Clear channel As-
sessments (CCA) which is an important pa-
rameter which paper [13] does not take it in
account. And the change can be simultane-
ously seen in the performance metrics when
simulated.
Also as it has been mentioned before, the
report focuses on all papers that suggest
a model for the backoff algorithm used in
the specified IEEE standard. The authors
have then followed their model by their sim-
ulation while recording various performance
metrics such as performance or time delay.
It also must be noted here that the models
are prescribed for slotted CSMA/CA. Papers
[12], [18], [14] and [10] have assumed satu-
rated traffic conditions. However the above
standard is prescribed for low power con-
sumption and low cost networks and hence
the traffic is less or in other words un-
saturated. Also these papers have consid-
ered Contention Access Period to be infi-
nite which is an ideal more than practical
case. An improvement is seen in the pa-
per [15] where they do consider the above
mentioned period to be finite. Also this pa-
per does taken into account one of the most
important aspects of PAN communication
8
which the use of superframes. However it
has not taken into consideration the inac-
tive time period of this superframe which is
the major advantage and the reason to use
them in IEEE 802.15.4 as they provide for
lower power consumption by giving the op-
tion of switching off the power when not in
use which is essential for all devices used in
a personal network. We personally believe
the highest improvement was done by paper
[7] which takes into account the superframes
structure and the retransmission schemes in
an unsaturated traffic condition. But this
paper does contain a few assumptions like
assuming no transmission errors etc., How-
ever it must be mentioned here that , with
the understanding of the paper we achieved
, it can be stated that the assumption on pa-
per [7] do not affect the performance metrics
and much as the previous papers and tries to
give an accurate reading during it’s simula-
tions.
4 Analytical model
As seen in the literature survey, the Markov
models have been applied in the field of
wireless connectivity. Furthermore, each
field applies a variation of the model –
one-dimensional, two-dimensional, or multi-
dimensional. In this section, we explain how
Markov models have been used to model the
backoff algorithms.
4.1 Bianchi’s model as given in • If the station transmits its frame without en-
tering the backoff procedure, it means that
paper [3]
The aim of the paper is to apply a two-
dimensional Markov model to the Binary
Exponential Backoff algorithm, in order to
determine the values of throughput and
delay under saturation conditions.
Assume that the network has a finite
number of stations n. Since the operation
is carried out under saturation conditions,
each station, after the completion of a
successful transmission, immediately has a
packet available for transmission.
Let b(t) denote the duration of the backoff
window for a particular node at time t.
Thus, b(t) is stochastic, but it is not Marko-
vian because the future is not completely
independent of the past, that is, the number
of collisions occurred previously influence
the backoff window size.
Say the maximum random number that
can be chosen is m. Then, there are m+1
values to choose from. Let the random
number chosen be i, and let s(t) be the
stochastic process when the backoff stage is
i (i ∈ (0, m)).
The value of the backoff counter is now
chosen in an unbiased manner from the
range (0,Wi−1 ), where Wi =2i Wmin .
Let pb be the probability that the channel
is busy, and let p be the probability that
a collision occurs. It is imperative to note
that both of the above probabilities are
independent of the state s(t) of the station.
We can now express our two-dimensional
9
the frame it has just sent out was success-
ful, and now the channel is free. Since p is
the probability that a collision occurred, 1- p
is the probability of successful transmission.
Similarly 1- pb is the probability of an idle
channel. Hence,
Markov model as the process {s(t), b(t)}.
This model is discrete-time. {i, k} excel-
lently captures the state of each node in
the network, where i indicates the chosen
random number and takes the values 0
through m, while k denotes the backoff
delay and takes the values 0 to Wi−1 in slot
times.
The following figure depicts the Markov
model used in the paper. An unusual
{−1, 0} state has been defined in the figure.
A station invokes the backoff procedure if it
has recently encountered a collision, or, if it
senses the channel as busy. There is a case
when the backoff algorithm is not invoked.
This is when the backoff counter equals zero,
and the node detects an idle channel after
waiting for a random period of time called
the DIFS (distributed interframe space).
In this case, the packet is sent without the
node having to invoke the backoff procedure.
Thus, state {1, 0} models the root state
where no procedure has been summoned.
• The station transmits a new frame and re-
sides in stage 0 of the backoff procedure if
it observes a busy channel after a success-
ful transmission (= (1- p)pb ), or, if it detects
that its transmitted frame has encountered
a collision (= p). Hence,
Through a series of mathematical manip-
ulation, the paper successfully derives a for-
mula throughput of the system.
4.2 Model as used in paper [9]
The aim of the paper is to apply a one-
dimensional Markov model to the general
Exponential Backoff algorithm, in order to
determine the values of throughput and de-
lay under saturation conditions.
The figure below depicts the one-dimensional
Markov model used to represent the backoff
procedure.
A station can be in any one of the infinite
number of backoff states in steady state. In-
finite because, in this case, we do not as-
sume any limitation on the number of re-
transmissions.
Let Pc denote the probability that the packet
is involved in a collision. In backoff state i
(i=0, 1, 2. . . ) the size of the contention win-
dow for a node is:
Wi = ri W0
where W0 is the size of the minimum con-
tention window.
10
After a successful transmission which oc-
curs with probability 1-pc , a node enters
backoff state i=0 with contention window
size W0 . On the other hand, if a node is
in the backoff state i=k, and it experiences
a failed transmission, then the node enters
backoff state i=k+1. The probability of this
happening is Pc .
Let Bk denote the k t h state that a node
resides in, in steady state. Then, Bk is a
Markov chain with the transition probabili-
ties pi ,j given as follows:
Explanation:
• A node can only enter the original
state 0 when there is a successful trans-
mission. The probability of successful
transmission is 1- pc , and so the first
probability follows.
• The backoff stage for a node increases
by 1 (from i to i+1) only when it is in-
volved in a collision. The probability of
a collision is pc , and so the second prob-
ability follows.
• There are no other transitions possible,
and so the probability is 0 for the re-
maining cases.
4.3 Models of variations of
backoff algorithm given in
paper [20]
The paper presents a new Markov chain
model to compute average delay in three dif-
ferent exponential backoff algorithms: the
BEB (Binary Exponential Backoff) algo-
rithm, EILD (Exponential Increase Linear
 Figure 1: Two-dimensional Markov Model for the backoff algorithm

• The station chooses a backoff delay of next

• A stop is put to the counter when it is sensed stage i after an unsuccessful transmission at
that the channel is busy. present stage i-1.

• The station has reached the last stage of the

• The counter is reduced by one when an idle backoff procedure. It continues to stay there
channel is detected. after unsuccessful transmissions.

11
• If the current frame has been sent success-
fully (= 1-p) and it is sensed that the channel
is occupied (= pb ), then when a station tries
to transmit a new frame, it has to summon
a backoff delay of stage 0.
• The station enters into the original (1,0)
state if it sent out it’s packet successfully and
if it senses that the channel is free again
Decrease) and EIED (Exponential Increase
Exponential Decrease). Whereas in the BEB
algorithm, the size of the contention window
in increased by a factor of two for each col-
lision until it reaches a maximum size, the
window size is reset following a successful
transmission.
The rapid decrease in the backoff window
size may lead to a decrease in performance,
as it may increase the probability of collisions
in the period following a successful transmis-
sion. In order to counter this problem when
there is a large number of stations that want
to send, a slower reduction in the size of the
backoff window is suggested. This led to
the development of algorithms that increase
the size of the backoff window after a col-
lision by doubling it, but decrease the size
of the window following a successful trans-
mission either exponentially (EIED) or lin-
early (EILD). EIED was originally proposed
in [17], and EILD was originally proposed in
[1].
In paper [20], the above mentioned algo-
rithms are explained and analysed. They
12
Figure 2: One-dimensional Markov chain
model applied for the backoff algorithm
are modelled using a new analytical Markov
model that is one-dimensional. By using this
model, the average delay under overloaded
conditions is calculated for each of the above
algorithms, and they are thus compared.
In the Markov chain models used below, a
common notation is used. Wi is the stage of
backoff, and acts as a counter that is decre-
mented for every slot that the link is idle
and paused when the link is busy. When the
counter attains a value of zero, the station
can begin transmitting.
In paper [20], the probability p that a col-
lision will occur is taken to be a constant
value that is independent of the stage of
backoff (Wi ). By making this assumption,
the Markov chain models proposed by pre-
vious authors can be reduced from two vari-
ables to one. This allows the use of a one-
dimensional Markov model, where a state i
of the model corresponds to the backoff stage
Wi . Also, the limit on the number of retries
is not considered. This means that the sys-
tem will stay in the final stage (Wm ) till the
packet has been transmitted without colli-
sion, where m is maximum backoff stage, or
the maximum retry count.
The following figure 3 is the Markov chain
model for the EIED algorithm.
The W0 stage is the stage where the sta-
tion begins transmitting. The final stage is
the Wm . The transition probability is given
by p, (packet collision probability at the par-
ticular time of transmission). If the medium
is sensed as idle (which has a p-1 probability)
the counter is decremented, and the system
moves from state Wi to Wi − 1.
The main difference between this algorithm
and the BEB, is that in every state, once the
packet has been successfully sent, the system
does not immediately move into the initial
state. In order to take this into considera-
tion, the model is modified such that pack-
ets can be sent not just from the initial state
W0 , but also from any state in the system.
This is done so that the history of conges-
tion is not erased, and the station can still
send packets. Note that the rate of increase
of the window size in this algorithm need not
be the same as the rate of decrease.
The following figure 4 is the model for the
EILD algorithm:
This algorithm linearly decreases the back-
off stage instead of exponentially decreasing
it, as is done in EIED. The states are rep-
resented the same way as they are in the
EIED model, although the difference being
that this model has a larger number of states
owing to the linear decrease in state. In this
model also, packets can be sent from any
13
state.
The models do not take the finite retry limit
into account (the limit before the packet is
dropped). However, by modifying the model
such that once the system stays in the fi-
nal state Wm once the retry limit has been
reached, it can be ensured that the system
cannot retry sending the packet any longer,
thus modelling finite retry limits as well.
4.4 Model as used in paper [7]
In paper [7], the authors try to establish
an enhanced Markov chain model for slotted
CSMA/CA for WPAN’s which is mentioned
in the IEEE standard explained above. They
have then given a simulation results for the
model prescribed by them in order to reveal
the contrast between their model and mod-
els prescribed in previous papers. The an-
alytical Markov chain model for the above
specified mechanism considers a superframe
structures as well as retransmission schemes
which is extremely important. It also works
on the assumption that the environment is
unsaturated which is more practical as it is
applied to personal area networks which in
general don’t contain many devices.They pa-
per starts off by briefly explaining to us the
IEEE 802.15.4 MAC protocol and then go on
to propose the analytical model of the slot-
ted CSMA/CA for the IEEE 902.15.4 MAC
protocol which is mainly implemented by the
Markov models. The major parameters of
the proposed Markov model are the stochas-
tic processes mentioned below :
• S(t) representing the backoff stage
 Figure 3: EIED
Figure 4: EILD
• C(t) representing the count of backoffs
• R(t) representing retransmission
counter state
• W(t) representing the number of re-
maining clear channel assessments
They then go on to examine the perfor-
mance through metrics like throughput ob-
tained while using the model explained and
conclude it by verifying their analytical re-
sults through simulations.
This paper considers a network consisting of
a PAN coordinator. The number of sensor
nodes is fixed to be ‘n’. It is also taken
that the system is single hop system. The
assumptions taken are:
1. Since the model is prescribed for per-
sonal uses, it is assumed that the de-
vices are within the range of successful
tranmission.
14
2. Assumption made by the authors is that
existence of errors by transmission or
channel sensing is null.
3. The topology prescribed is star. Also
transmitted frames can be lost only due
to collision as it is uplink data transmis-
sion.
4. In a situation that a device is transmit-
ting a frame, it is assumed that it does
not take in new frames from the above
layers.
5. Data frames arrival is according to the
Poisson process. Also device is assumed
to be capable of storing only one frame.
6. All nodes are homogeneous.
For the analysis, a Markov model taking into
consideration the key characteristics of the
IEEE 802.15.4 MAC protocols such as struc-
ture of the superframe, acknowledgment and
retransmission schemes is constructed.
Here the Markov chain states can be clas-
sifies into three different varieties. The first
variety is of the form {s(t), b(t), w(t), r(t)}
where s(t) establishes the number of back-
offs at time t.
Also b(t) represents the backoff counter’s
value at the time t, w(t) represents the num-
ber of clear channel assessments that are left
to be done for the transmission at that point
and r(t) represents the value of retransmis-
sion plane at time t.
In this state the parameters required by the
Markov models are P1 and P11 which are the
probabilities of the channel being idle while
performing the first clear channel assessment
and the second clear channel assessment re-
spectively.
The second type is to established the nonde-
ferred and the postponed tranmissions in the
s(t)th backoff stage on the r(t) transmission
plane at time t. Here the important param-
eters for the Markov model are Pcnd and Pcd
which denote the probabilities of the occur-
rence of a collision for the tranmissions men-
tioned above.
The final type is of the form {Ds(t), r(t)}
which represents the waiting state and helps
in computing the Pd which is the probabil-
ity that a postponed transmission can occur.
An other important parameter is α is the
probability of a situation where the device
is in transitioning states and a new frame
is entering. As we can see from the brief
, two parameters i.e Pd and α are easy to
15
compute. The paper then goes on to the
derivations of the other four parameters us-
ing these known parameters that shall be ex-
plained later. This can then be extended to
every level in order to get the entire model.
5 Applications
Markov chains have been extensively used in
the field of networks. It is predominantly
applied in the World Wide Web, Queueing
Network theory, and in the area of collision
avoidance algorithms – details of which have
been already explored in the document.
In the World Wide Web, it has two ap-
plications, namely, HTTP pre-fetching, and
the PageRank algorithm used by the Google
search engine. In Queueing Networks,
Markov chains are used to determine perfor-
mance parameters of switching techniques,
such as Packet switching and Circuit switch-
ing.
5.1 Queueing Networks
In networking, delay in the network is caused
due to a number of factors, and the aim is
to minimize it. We should first understand
what it is. Delay consists of the following
components:
1. Transmission delay
2. Propagation delay
3. Queueing delay
4. Processing delay
 Figure 5: Markov model for the prescribed mechanism
Using a queuing model, we can accurately
make quantitative predictions about the av-
erage packet delay in the network, as well as
qualitatively understand the performance of
the network, without having to simulate all
the individual components of the network.
16
5.1.1 Queueing theory
Networks of queues are systems with a num-
ber of queues are inter-connected queues.
This means that entities can enter one queue,
receive a particular service, and then join
other queues, or leave the network. This can
be described using the example of customers
coming to a service center. Customers ar-
rive at the service center, wait in the queue
if necessary, receive service from the server,
and depart (if they have no other service to
receive). This presents a very basic queuing
model. Queueing systems are described by a
combination of letters (in the Kendall nota-
tion): Figure 6: Queue
A/B/C/D/E
• A = Describes the arrival process
• B = Service time distribution
• C = Number of servers
Figure 7: Markov Chain Model for the
• D = Maximum number of entities in the M/M/1 Queue
queue (capacity of the queue/system)
• E = Queue discipline (order in which In order to solve the continuous time mod-
entities are serviced in the queue) els, the balance equations associated with
each state must be solved.
Now, we consider how Markov Chains can
be applied to a queuing model in order to By doing so, we can determine various ex-
obtain more information about it. pected values for the network, including:
Consider a simple queuing system – M/M/1. 1. The probability of being in state k of
In this system, the arrival process, as well the Markov chain, ie, the probability of
as the service time is Markovian or mem- having k customers in the system:
oryless. This means that it is a Poisson
process with exponential distribution (effec-
tively random).
where ρ = the expected number of cus-
The M/M/1 queueing system consists of cus-
tomers being served.
tomers (following the example above) arriv-
ing randomly according to a Poisson process
2. Expected delay per customer:
with rate λ, and one server with a service
time per customer that is exponentially dis-
tributed with rate µ. We assume other values
(such as number of entities) to be infinite.
As the buffer may contain any number of
3. Expected waiting time in the queue
customers, the number of states is not fi-
nite. Also, the transitions occur in contin-
uous time.

17
 4. Expected number of customers in the
queue
5. Expected throughput of the system
Application in network: The model de-
scribed above can be directly applied to net-
work processes. For example, packet switch-
ing and circuit switching can both be mod-
elled as an M/M/1 queue system in order to
calculate various expected network parame-
ters.
Consider the arrival of packets in a network.
• The number of times a packet can ar-
rive in a given time interval can be any
natural number, including 0, 1, 2 ...
• Arrival of one packet doesn’t affect the
arrival of another. Therefore, the arrival
of packets is an independent process.
If we assume that multiple packets don’t
arrive at the exact same time, and that the
rate of arrival of packets in a network is a
constant rate λ, then the necessary condi-
tions are satisfied. We can consider the ar-
rival of packets in a network to have Pois-
son distribution, and we can model it using
a Markov Chain Model.
Consider the arrival rate of m sessions to be
λ/m. Packet sizes are also effectively ran-
dom, and have an exponential distribution.
18
Consider the average packet size to be L
bits. The bandwidth of the link is µ bits
per second (bps). The average transit time
is equal to the packet size/bandwidth of the
link. Thus, average transit time is L/µ .
Using the formulae described above we can
now determine various expected values in the
network.
1. Packet Switching
In packet switching network, each ses-
sion operates independently of the other
sessions, and there is no prior allocation
of link bandwidth. Therefore, the entire
bandwidth is shared by all the m ses-
sions.
Expected delay per session:
Expected number of packets per session:
2. Circuit Switching
In circuit switching network, a link is
allocated to a session when it is set
up, and is used by the session until it
terminates. This means that each of
the m sessions are given only 1/m of
the link bandwidth.
Expected delay per session:
Expected number of packets (per ses-
sion):

From the above, we can see that in cir-
cuit switching, the expected number of pack-
ets (N) is m times more than in packet
switching. The expected delay (T) is also
m times more in circuit switching than in
packet switching.
This is a useful result that helps network
designers understand how the choice of a
switching method affects network perfor-
mance.
5.2 Applications in the World
Wide Web
Markov chains have been used in the World
Wide Web in two major fields: link predic-
tion, and search engine algorithms.
5.2.1 PageRank algorithm
History The PageRank algorithm was de-
vised by Larry Page and Sergey Brin,
while they pursuing their Masters Degree
at Stanford University. They are the ones
who founded Google, and consequently, the
search engine implementes the PageRank al-
gorithm. To this date, this algorithm plays
an important role in the Google search en-
gine, setting it apart from the rest of its com-
petitors [namely Bing and Yahoo].
Importance With so many pages on the
Internet, ordering of the search results is a
must. Search results yield millions of hits, all
19
of which are impossible to go through. Thus,
a search engine will be of help only if it dis-
plays the results in order of relevance. Statis-
tics show that 91% of the users only browse
the links on the first page of the search re-
sults. Around 5% take a look at Page 2, and
the rest 4% go through the remaining pages.
It is thus imperative for any search engine to
order/rank their pages in a relevant manner.
About PageRank algorithm designates
importance to each page by cumulating the
number of links to it. It not only totals the
number of links, it also checks how credible
these links are. The credibility of a link is
determined by the PageRank of the source
page. PageRank makes this assumption
because it is intuitive to see that the value
of a page is high if a lot of other pages cite
it. Both number and quality matter. That
is, a page having, say, 20 links to it from
less credible sources is certain to rank lower
as compared to a page that is cited by 5
credible sources.
To begin explaining the PageRank algo-
rithm, we start with the random surfer
model. It states that any visitor is likely to
start their quest from any random page on
the Web. Their journey on the Web how-
ever, continues mostly through hyperlinks
embedded in the pages. Selection of such
links is random, and thus is chosen with
uniform probability.
Hence, if a page i has ni links embedded in
it, then the probability that one of them is
clicked becomes 1/ni .
We define the quantity ‘PageRank’ for a
page i as the probability that the random
surfer will be at page i at any point of time.
Thus, PageRank is obtained by capturing
the random surfer model with a Markov
Chain. Iterating through the mathematical
expression several times leads to its station-
ary distribution, which is nothing but the
PageRank.
To implement the PageRank algorithm, we
represent the World Wide Web in terms of a
directed graph, G=(V,E), where V is the set
of vertices representing the webpages of the
Web, and E is the set of edges representing
links between the pages. The node i is
connected to the node j by an edge, i.e., (i,
j)  E, if page i has a link going to page j.
Let xi (k) be the probability that the random
surfer is visiting page i at time k, and let
x(k) be the vector containing all xi (k). That
is, for every k,
x(k) = [x1 (k), x2 (k), x3 (k). . . xi (k). . . xn (k)]
and, ni=1 x(k) = 1
P
(summation of probabilities is 1)
(n is the overall number of webpages on the
web)
The initial vector x(0) is a probability vector
in which xi (0) denotes the probability that
the random surfer starts his quest with page
i.
After 1 step, the distribution becomes:
x(1) = Ax(0)
After 2 steps, the distribution becomes:
x(2) = Ax(1)
20
Thus, the growth of x(k) can be expressed as
x(k + 1) = Ax(k)
The link matrix A = (aij )  Rnn is given
by aij = 1/nj if (j, i)  E and 0 otherwise,
where, nj is the number of outgoing links of
page j. Clearly, matrix A is the transition
probability matrix of the random surfer. If
we keep substituting x(k) with Ax(k-1), then
we finally get,
x(k+1) = Ak+1 x(0)
As stated earlier, PageRank is the direct
result of the stationary distribution of this
Markov process. Let the PageRank vector
be denoted by x*. Then,
x* = limk→∞ x(k)
The value in this vector denotes the prob-
ability that the surfer would be at page i
at any random time. Thus, a page having
high PageRank shall have higher probability
in this vector. It is the solution of the linear
equation:
x* = Ax*
This is because of the definition of a station-
ary distribution: it does not change with the
application of the transition matrix [that is,
x(k+1) = x(k)].
In mathematics, this is equivalent to: 1x* =
Ax* Where, 1 denotes the eigenvalue, and
x* denotes the eigenvector.
Definition: Strongly connected directed
graph: A directed graph is strongly con-
nected if there is a route between every pair
of vertices.
Definition: A matrix is said to be irre-
ducible if and only if its associated directed
graph is strongly connected. An irreducible
matrix has a unique eigenvector for a given
eigenvalue.
Considering the directed graph that repre-
sents the World Wide Web, we see that it
is not a strongly connected graph. This is
because, the Web has billions of pages, with
millions of them unrelated, and so it is im-
possible to have a path between every pair of
nodes/pages. Thus, the link matrix A is not
reducible and thus the eigenvector x* is not
unique. If x* is not unique, then each page
can have multiple ranks depending on which
eigenvector we use. This proves to be of no
help in giving accurate ranks to the pages.
To resolve this issue, we need to modify the
random surfer model. This brings us to the
Teleportation Model. The idea of the tele-
portation model is that the random surfer, at
any time, may become disinterested and stop
hopping along hyperlinks. In such a case,
the surfer skips to another page not directly
connected to the one currently visiting. This
page can in fact be completely unrelated in
the domains and/or the contents. If there
are N pages in the Web, then the probabil-
ity to reach any page via this unpredictable
jump is 1/N.
To capture the two ways to reach a page –
one via hyperlink (matrix A), while the other
as a random jump (matrix Q), we define a
new matrix M as follows:
M = (1-β)A + βQ
where Q is an N xN matrix, with Q = (qij )
21
and qij = 1/N for every i,j.
β is known as the damping factor. In the
original PageRank algorithm,
β = 0.15. That is, there is only a 15% chance
that a webpage has been reached via a ran-
dom jump.
Since matrix M includes the term βQ, all
the entries in M are positive. By Perron’s
theorem, the eigenvector is unique for every
eigenvalue in such matrices. We were only
concerned by the uniqueness of the eigenvec-
tor for eigenvalue 1, and now this holds for
the matrix M. Hence, we redefine the vector
x* by using M instead of A as follows:
x* = Mx*
The matrix M is an N xN matrix [N = num-
ber of pages on the web], and hence its large
dimension makes x* difficult to compute.
There are several ways to compute x*, how-
ever, those methods are beyond the scope of
this report.
5.2.2 Link Prediction
Before getting into the concept of link predic-
tion, the concept of adaptive website needs
to be reported. Adaptive websites are web-
sites that change themselves according to
the user patterns. One of the major actual
use of adaptive websites are in e-commerce
websites where other products are suggested
based on the present selected product. There
are many ways to achieve this in a website.
Another use of the link prediction is enhanc-
ing navigation. One of the most used effi-
cient methods used for navigating is search
engines as of now. However these search en-
gines don’t help in giving us a set of related
pages once a page is selected. There are cer-
tainly some alternate options available like
agent assisted navigation or tour generation.
Agent assisted navigation is where the sys-
tem suggests what the user can do while tour
generation is where the user is given a tour
of related pages. However the key here is
personalising the efforts done by the above
techniques and this is what is achieved by
link prediction. Link prediction is basically
trying to pre-fetch the next links that are
probable when the user is in the current link.
Ways to achieve link prediction In this
topic we will focus on how to achieve link
prediction especially using Markov chains as
that is our main topic. As we know every
markov model is characterised by transition
matrix and initial probability distribution of
states. And this information can be used to
predict the next state in the link given a par-
ticular state and the probabilities of the dif-
ferent transitions. And this is the major aim
of link prediction. A discrete Markov chain
model can be defined by the tuple (S,A, λ)
where S corresponds to the state space, A is
the matrix representing the transition prob-
abilities from one state to another for all pos-
sible combinations and λ is the initial prob-
ability distribution of every state present in
S. As we have seen above the fundamental
property of the Markov model is the depen-
dency on the previous state. So therefore if
the vector s(t) denotes the probability vector
for all the states at time t, then:
ŝ(t) = ŝ=(t-1)A
22
In this formulation, a Markov state can cor-
respond to an URL, HTTP request or any
action such as sending email. The matrix A
is being defined here as the following :
A(s,s’) = C(s,s’)/Σs”C(s,s”)
λ(s) = C(s) / Σs”C(s”)
where C(s,s’) is the count of the number of
times s’ follows s in the data. An element in
the matrix A, say A(s,s’) can be interpreted
as the probability of transitioning from state
s to s’ in one step. Similarly an element
of A*A will denote the probability of transi-
tioning from one state to another in two steps
and so on. Given the link history of the user,
we can represent each link as a vector with
the probability at that state for that time
which is denoted by i(t-k), i(t-k+1 ). . . i(t-
1). The Markov chain models estimation of
the probability of being in a state at a time
t is shown as:
ŝ(t) = î(t-1)A
This equation is then used to develop equa-
tions for transitions which are achieved
through more than one steps. However that
shall be discussed in the final report.
Applications In this section we shall see
four different application of link prediction
using Markov chains in brief.
Web Server HTTP Request Predic-
tion The main aim for this application is to
enhance the server performance.In the past
the analysis of HTTP requests have always
included request patterns or statistical anal-
ysis of file sizes. However here link modelling
can be applied and made user specific. Here
every client request can be processed by the
prediction system to list the probabilities of
the next requests based on the history of re-
quests.
Adaptive Web navigation In this
particular application, link prediction can be
modelled for each user. Here once a user
clicks on the link, the link predictor works
on the server side and suggests a list of links
that the user can go next possibly and these
links are ordered according to the probabil-
ity. This can be used to build a Navigation
system that leads the user to different pages
based on the statistical data of the history.
Tour Generation For this particular
application, the Tour Generation Using
Markov Models algorithm has been imple-
mented. Here the Markov model is used to
predict a list of states that can be visited
next. Every state should be marked visited
or unvisited in order to make sure there is no
cyclic actions. Also in case of same probabil-
ity, a method to choose the next best state
needs to be formulated. Here the first link
with the longest matching prefix URL is cho-
sen. And this process is implemented to get
the sequence of states ie URLs.
Personalised Hub Hub refers to any
link that can often be taken as good start-
ing states to find information. However these
hubs need to be personalised in order to be of
any use. Hence Markov models can be used
to generate an algorithm that is similar to
23
the one above on path traversal patterns in
order to list out a series of hubs that would
work for that particular user.
6 Conclusion
We hope to have established a fair account of
our understanding of the various aspects and
applications of Markov model. We aimed at
giving a brief of how Markov models is used
to analyse performance aspects and also its
practical application is areas such as adap-
tive websites and queueing networks. We
also want to conclude by stating that this
draft does not include the major derivations
which we hope to give a clear view in the
final document.
References
[1] V Bharghavan, A Demers, and S
Shenker. “MACAW: a media access
protocol for wireless LAN’s”. In: ACM
SIGCOMM Computer Communication
Review 24.4 (1994), pp. 212–225.
[2] Giuseppe Bianchi. “IEEE 802.11-
saturation throughput analysis”. In:
IEEE communications letters 2.12
(1998).
[3] Giuseppe Bianchi. “Performance anal-
ysis of the IEEE 802.11 distributed co-
ordination function”. In: IEEE Jour-
nal on selected areas in communica-
tions 18.3 (2000).
 [4] G. Bolch et al. Queueing Networks and
Markov Chains: Modeling and Perfor-
mance Evaluation with Computer Sci-
ence Applications. John Wiley Sons,
2006. isbn: 9780471193661.
[5] Cali et al. “IEEE 802.11 protocol: de-
sign and performance evaluation of
an adaptive backoff mechanism”. In:
IEEE journal on selected areas in com-
munications (2000), pp. 1774–1786.
[6] Nghia T. Dao and Robert A. Malaney.
“A new Markov model for non-
saturated 802.11 networks”. In: IEEE
Consumer Communications and Net-
working Conference (2008).
[7] Jung and Chang Yong. “Enhanced
Markov chain model and throughput
analysis of the slotted CSMA/CA for
IEEE 802.15. 4 under unsaturated traf-
fic conditions”. In: IEEE Transactions
on Vehicular Technology 58.1 (2009),
pp. 473–478.
[8] Ching Wai Ki, Huang Ximin, and
Ng Michaelk. “Markov Chains: Mod-
els, Algorithms and Applications”. In:
(2013).
[9] Byung-Jae Kwak, Nah-Oak Song,
and Leonard E. Miller. “Performance
analysis of exponential backoff”. In:
IEEE/ACM transactions on network-
ing 13.2 (2005).
[10] Tae-Jin Lee, Hae Rim Lee, and Min
Young Chung. “MAC throughput limit
analysis of slotted CSMA/CA in IEEE
802.15. 4 WPAN”. In: IEEE Commu-
nications Letters (2006), pp. 561–563.
24
[11] David Malone, Ken Duffy, and Doug
Leith. “Modeling the 802.11 dis-
tributed coordination function in non-
saturated heterogeneous conditions”.
In: IEEE/ACM Transactions on net-
working 15.1 (2007), pp. 159–172.
[12] TR Park et al. “Throughput and en-
ergy consumption analysis of IEEE
802.15. 4 slotted CSMA/CA”. In: Elec-
tronics Letters 41.18 (2005), pp. 1017–
1019.
[13] Park and Pangun. “A generalized
Markov chain model for effective anal-
ysis of slotted IEEE 802.15. 4”. In:
2009 IEEE 6th International Confer-
ence on Mobile Adhoc and Sensor Sys-
tems (2009).
[14] S. Pollin et al. “Performance analysis
of slotted carrier sense IEEE 802.15.4
medium access layer”. In: Proc. IEEE
GLOBECOM (2006), pp. 1–6.
[15] Ramachandran et al. “Analysis of
the contention access period of IEEE
802.15. 4 MAC”. In: ACM Transac-
tions on Sensor Networks (TOSN) 3.1
(2007).
[16] Ramesh R Sarukkai. “Link predic-
tion and path analysis using Markov
chains”. In: Computer Networks 33.1
(2000).
[17] N. Song et al. “Enhancement of IEEE
802.11 distributed coordination func-
tion with exponential increase expo-
nential decrease backoff algorithm”.
In: The 57th IEEE Semiannual Spring
 Vehicular Technology Conference Vol.4
(Apr. 2003), pp. 2775–2778.
[18] Zhifeng Tao et al. “Performance anal-
ysis and a proposed improvement for
the IEEE 802.15. 4 contention access
period”. In: IEEE Wireless Commu-
nications and Networking Conference,
2006 4 (2006).
[19] K. S Trivedi. Probability Statistics
with Reliability, Queuing and Com-
puter Science Applications. John Wi-
ley Sons, 2008. isbn: 9780471460817.
[20] Vukovic, Ivan N., and Natt Smavatkul.
“Delay analysis of different backoff al-
gorithms in IEEE 802.11”. In: Vehic-
ular technology conference, VTC2004-
Fall 2004 IEEE 60th Vol.6 (2004).
[21] Ziouva, Eustathia, and Theodore
Antonakopoulos. “CSMA/CA perfor-
mance under high traffic conditions:
throughput and delay analysis”.
In: Computer communications 25.3
(2002).

25
Members Contribution