COMM 552

Elgazeera High Institute for

Engineering and Technology

Information Theory and Coding

Lecture – 2

Mark off Model for Information Sources

Dr. Ahmed Abouelmagd

 Mark off Model for Information Sources
Mark off source:
A source puts out symbols belonging to a finite alphabet
according to certain probabilities depending on preceding
symbols as well as the particular symbol in question.

- Nodes in the graph represent states and transition between states

represented by a directed line from the initial to the final state.
- Transition probs. and the symbols emitted corresponding to the
transition will be shown marked along the lines of the graph.
Random process:
Is the statistical model of the system that produces a sequence of
symbols stated above which is governed by a set of probs.

Discrete source:
Is the random process that produces a discrete sequence of
symbols chosen from a finite set.
Discrete stationary Mark off process:

1. At the beginning of each symbol interval, the source will be in

the one of ‘n’ possible states 1, 2, ….. n. Where ‘n’ is defined
as: n ≤ (M)m
M = number of symbol / letters in the alphabet of a discrete
stationery source,
m = represents the order of the source.
 - The source changes state once during each symbol interval
from say i to j. The probability of this transition is Pij.
- Pij depends only on the initial state i and the final state j
- Pij does not depend on the states during any of the
preceding symbol intervals.
2. When the source changes state from i to j it emits a symbol.
Symbol emitted depends on the initial state i and the
transition i j.
3. Let s1, s2, ….. sM be the symbols of the alphabet, and let x1, x2,
x3, …… xk, be a sequence of random variables, where xk
represents the kth symbol in a sequence emitted by the source.
Then, the probability that the kth symbol emitted is sq will
depend on the previous symbols x1, x2, x3, …, xk–1 emitted by
the source. i.e., P (xk = sq / x1, x2, ……, xk–1)
4. The residual influence of x1, x2, ……, xk–1 on xk is represented
by the state of the system at the beginning of the kth symbol
interval. i.e. P (xk = sq / x1, x2, ……, xk–1) = P (xk = sq / sk) when
sk in a discrete random variable representing the state of the
system at the beginning of the kth interval.
Representation of Discrete Stationary Mark off sources:

a. Graph form:
- Nodes in the graph represent states and transition between states
represented by a directed line from the initial to the final state.
- Transition probs. and the symbols emitted corresponding to the
transition will be shown marked along the lines of the graph.
Illustrative example 1:

o A source emitting one of three symbols A, B, and C

o The probability of occurrence of a symbol depends on the
particular symbol in question and the symbol immediately
proceeding it.
o Residual or past influence lasts only for a duration of one
symbol.
o The last symbol emitted by the source can be A or B or C.
Hence past history can be represented by three states- one for
each of the three symbols of the alphabet.
o Suppose that the system is in state (1) and the last symbol
emitted by the source was A.
o The source now emits symbol (A) with probability ½ and
returns to state (1). OR
o The source emits symbol (B) with probability ¼ and goes to
state (3). OR
o The source emits symbol (C) with probability ¼ and goes to
state (2).
b. Tree diagram:
Tree diagram is a planar graph where the nodes correspond to
states and branches correspond to transitions. Transitions between
states occur once every Ts seconds.
Use of the tree diagram
Tree diagram can be used to obtain the
probabilities of generating various
symbol sequences
Generation a symbol sequence say AB
This can be generated by any one
of the following transitions:
P ( S1 = 1, S2 = 1, S3 = 3 ), or
P ( S1 = 2, S2 = 1, S3 = 3 ), or
P ( S1 = 3, S2 = 1, S3 = 3 )
P (AB) = P ( S1 = 1, S2 = 1, S3 = 3 )
+ P ( S1 = 2, S2 = 1, S3 = 3 )
+ P ( S1 = 3, S2 = 1, S3 = 3 )
P(A) = (1/3) (1/2) + (1/3) (1/4) + (1/3) (1/4) = (1/3) (4/4) = 1/3
P(C) = (1/3) (1/4) + (1/3) (1/2) + (1/3) (1/4) = (1/3) (4/4) = 1/3
P(B) = (1/3) (1/4) + (1/3) (1/4) + (1/3) (1/2) = (1/3) (4/4) = 1/3
P(AA) = (1/3) (1/4) + (1/3) (1/8) + (1/3) (1/8) = (1/3) (3/8) = 1/8
P(AC) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(AB) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(CA) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(CC) = (1/3) (1/8) + (1/3) (1/4) + (1/3) (1/8) = (1/3) (3/8) = 1/8
P(CB) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(BA) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
P(BC) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
P(BB) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
AAA
AAC
AAB
ACA
ACC
ACB
ABA
ABC
ABB
CAA
CAC
CAB
CCA
CCC
CCB
CBA
CBC
CBB
BAA
BAC
BAB
BCA
BCC
BCB
BBA
BBC
BBB
AAA P(AAA) = (1/3) (1/2) (1/2) (1/2) = (1/3) (1/8)
AAC P(AAC) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
AAB P(AAB) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
ACA P(ACA) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
ACC P(ACC) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
ACB P(ACB) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
ABA P(ABA) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
ABC P(ABC) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
ABB P(ABB) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
CAA P(CAA) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/64)
CAC P(CAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CAB P(CAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CCA P(CCA) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
CCC P(CCC) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
CCB P(CCB) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
CBA P(CBA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CBC P(CBC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CBB P(CBB) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BAA P(BAA) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BAC P(BAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BAB P(BAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BCA P(BCA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BCC P(BCC) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BCB P(BCB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BBA P(BBA) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
BBC P(BBC) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
BBB P(BBB) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
AAA P(AAA) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
AAC P(AAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
AAB P(AAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ACA P(ACA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ACC P(ACC) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
ACB P(ACB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABA P(ABA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABC P(ABC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABB P(ABB) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
CAA P(CAA) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
CAC P(CAC) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
CAB P(CAB) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
CCA P(CCA) = (1/3) (1/2) (1/2) (1/4) = (1/3) (1/16)
CCC P(CCC) = (1/3) (1/2) (1/2) (1/2) = (1/3) (1/8)
CCB P(CCB) = (1/3) (1/2) (1/2) (1/4) = (1/3) (1/16)
CBA P(CBA) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
CBC P(CBC) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
CBB P(CBB) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
BAA P(BAA) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BAC P(BAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BAB P(BAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BCA P(BCA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BCC P(BCC) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BCB P(BCB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
BBA P(BBA) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
BBC P(BBC) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
BBB P(BBB) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
AAA P(AAA) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
AAC P(AAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
AAB P(AAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ACA P(ACA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ACC P(ACC) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
ACB P(ACB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABA P(ABA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABC P(ABC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
ABB P(ABB) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
CAA P(CAA) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
CAC P(CAC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CAB P(CAB) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CCA P(CCA) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
CCC P(CCC) = (1/3) (1/4) (1/2) (1/2) = (1/3) (1/16)
CCB P(CCB) = (1/3) (1/4) (1/2) (1/4) = (1/3) (1/32)
CBA P(CBA) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CBC P(CBC) = (1/3) (1/4) (1/4) (1/4) = (1/3) (1/64)
CBB P(CBB) = (1/3) (1/4) (1/4) (1/2) = (1/3) (1/32)
BAA P(BAA) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
BAC P(BAC) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
BAB P(BAB) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
BCA P(BCA) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
BCC P(BCC) = (1/3) (1/2) (1/4) (1/2) = (1/3) (1/16)
BCB P(BCB) = (1/3) (1/2) (1/4) (1/4) = (1/3) (1/32)
BBA P(BBA) = (1/3) (1/2) (1/2) (1/4) = (1/3) (1/16)
BBC P(BBC) = (1/3) (1/2) (1/2) (1/4) = (1/3) (1/16)
BBB P(BBB) = (1/3) (1/2) (1/2) (1/2) = (1/3) (1/8)
P(AAA) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (4/16)
P(AAC) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(AAB) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(ACA) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(ACC) = (1/3) (1/16) + (1/3) (1/32) + (1/3) (1/32) = (1/3) (1/16) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (6/32)
P(ACB) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(ABA) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(ABC) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (1/32) + (1/3) (1/64) + (1/3) (1/64) = (1/3) (4/64)
P(ABB) = (1/3) (1/16) + (1/3) (1/32) + (1/3) (1/32) = (1/3) (1/16) + (1/3) (1/32) + (1/3) (1/32) = (1/3) (4/32)
P(CAA) = (1/3) (1/64) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (1/64) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (7/64)
P(CAC) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (4/64)
P(CAB) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (4/64)
P(CCA) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (4/32)
P(CCC) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (4/16)
P(CCB) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (4/32)
P(CBA) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (4/64)
P(CBC) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (1/64) + (1/3) (1/32) + (1/3) (1/64) = (1/3) (4/64)
P(CBB) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (4/32)
P(BAA) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (1/32) + (1/3) (1/16) + (1/3) (1/32) = (1/3) (4/32)
P(BAC) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (4/64)
P(BAB) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (4/64)
P(BCA) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (4/64)
P(BCC) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (4/32)
P(BCB) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (1/64) + (1/3) (1/64) + (1/3) (1/32) = (1/3) (4/64)
P(BBA) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (4/32)
P(BBC) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (1/32) + (1/3) (1/32) + (1/3) (1/16) = (1/3) (4/32)
P(BBB) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16)
P(AA) = (1/3) (1/4) + (1/3) (1/8) + (1/3) (1/8) = (1/3) (3/8) = 1/8
P(AC) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(AB) = (1/3) (1/8) + (1/3) (1/16) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(CA) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(CC) = (1/3) (1/8) + (1/3) (1/4) + (1/3) (1/8) = (1/3) (3/8) = 1/8
P(CB) = (1/3) (1/16) + (1/3) (1/8) + (1/3) (1/16) = (1/3) (4/16) = 1/12
P(BA) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
P(BC) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
P(BB) = (1/3) (1/16) + (1/3) (1/16) + (1/3) (1/8) = (1/3) (4/16) = 1/12
Example 2:

For the information source given draw the tree diagram and find the
probs. of messages of lengths 1, 2 and 3.
Solution 2:
Messages of length (1)
and their probs
A ½ x ¾ = 3/8
B ½ x ¾ = 3/8
C ½x¼+½x¼=¼

Message of length (2)

AA, AC, CB, CC, BB,
BC & CA
and their probs
AA : ½ x ¾ x ¾ = 9/32
AC: ½ x ¾ x ¼ = 3/32
 Entropy and Information Rate of Mark off Sources
Entropy of state ‘i’ is defined as:

Entropy of the source is defined as:

Pi = the proby that the source is in state ‘i

Average information rate for the source is defined as:
R = rs . H bits/sec
Where, ‘rs’ is the number of state transitions per second or the
symbol rate of the source.
Example 3:
Consider the information source in example 7. Find:
a. The source entropy H and
b. The average information content per symbol in messages
containing one, two and three symbols.
Messages of length (1)
and their probs
A ½ x ¾ = 3/8
B ½ x ¾ = 3/8
C ½x¼+½x¼=¼

Message of length (2)

AA, AC, CB, CC, BB,
BC & CA
and their probs
AA : ½ x ¾ x ¾ = 9/32
AC: ½ x ¾ x ¼ = 3/32
Solution 8.a: The source entropy H:
Solution 2.b:
The information content in the 7 messages (AA, AC, CB, CC,
BB, BC & CA) containing two symbols:

I(BC)
The average information content in the messages containing
two symbols: (the 7 messages (AA, AC, CB, CC, BB, BC & CA)
The average information content per symbol in messages
The average information content per symbol in messages

bits / symbol

Following the same steps for N=1 and N=3, we get:

Therefore, it can be stated that the average information per symbol

in the message reduces as the length of the message increases.
 The entropy of i state:

The entropy the source:

Then GN is a monotonic decreasing function of N and in the

limiting case it becomes: