INFORMATION THEORY AND CODING

Session 2
Entropy and rate of Information of an Information Source /
Model of a Markoff Source
1. Average Information Content of Symbols in Long Independence Sequences
Suppose that a source is emitting one of M possible symbols s
0
, s
1
.. s
M
in a
statically independent sequence
Let p
1
, p
2
, .. p
M
be the problems of occurrence of the M-symbols resply.
suppose further that during a long period of transmission a sequence of N symbols have
been generated.
On an average s
1
will occur NP
1
times
S
2
will occur NP
2
times
:
:
s
i
will occur NP
i
times
The information content of the i
th
symbol is I (s
i
) = log
|
|
.
|

\
|
i
p
1
bits
P
i
N occurrences of s
i
contributes an information content of
P
i
N . I (s
i
) = P
i
N . log
|
|
.
|

\
|
i
p
1
bits
Total information content of the message is = Sum of the contribution due to each of
M symbols of the source alphabet
i.e., I
total
=

=
|
|
.
|

\
|
M
1 i i
1
p
1
log NP bits
by given in symbol er p
content n inforamtio Average
H =

=
|
|
.
|

\
|
=
M
1 i i
1
total
p
1
log NP
N
I
symbol
per bits
---- IV
This is equation used by Shannon
Average information content per symbol is also called the source entropy.
- What is the average information associated with an extremely unlikely message?
- What is the average information associated with an extremely likely message?
- What is the dependence of H on the probabilities of messages?
To answer this, consider the situation where you have just two messages of probs.
p and (1-p).
Average information per message is H =
p 1
1
log ) p 1 (
p
1
log p

+
At p = O, H = O and at p = 1, H = O again,
The maximum value of H can be easily obtained as,
H
max
= log
2
2 + log
2
2 = log
2
2 = 1
H
max
= 1 bit / message
Plot and H can be shown below
The above observation can be generalized for a source with an alphabet of M
symbols.
Entropy will attain its maximum value, when the symbol probabilities are equal,
i.e., when p
1
= p
2
= p
3
= . = p
M
=
M
1
H
max
= log
2
M bits / symbol
H
max
=

M
M
p
1
log p
H
max
=

M
M
1
1
log p
H
1
O

P
H
max
=

= M log M log
M
1
2 2
- What do you mean by information rate?
If the source is emitting symbols at a fixed rate of r
s
symbols / sec, the average
source information rate R is defined as
R = r
s
. H bits / sec
- Illustrative Examples
1. Consider a discrete memoryless source with a source alphabet A = { s
o
, s
1
, s
2
}
with respective probs. p
0
= , p
1
= , p
2
= . Find the entropy of the source.
Solution: By definition, the entropy of a source is given by
H =

=
M
i i
i
p
p
1
1
log bits/ symbol
H for this example is
H (A) =

=
2
0
1
log
i i
i
p
p
Substituting the values given, we get
H (A) =
o
p log
o
P
1
+ P
1
log
2
2
1
1
log
1
p
p
p
+
=
2
log 4 +
2
log 4 +
2
log 2
=
|
.
|

\
|
2
3
= 1.5 bits
if
s
r = 1 per sec, then
H (A) =
s
r H (A) = 1.5 bits/sec
2. An analog signal is band limited to B Hz, sampled at the Nyquist rate, and the
samples are quantized into 4-levels. The quantization levels Q
1,
Q
2,
Q
3,
and Q
4
(messages) are assumed independent and occur with probs.
P
1
= P
2
=
8
1
and P
2
= P
3
=
8
3
. Find the information rate of the source.
Solution: By definition, the average information H is given by
H =
1
p log
1
1
p
+
2
p log
2
1
p
+
3
p log
3
1
p
+
4
p log
4
1
p
Substituting the values given, we get
H =
8
1
log 8 +
8
3
log
3
8
+
8
3
log
3
8
+
8
1
log 8
= 1.8 bits/ message.
Information rate of the source by definition is
R =
s
r H
R = 2B, (1.8) = (3.6 B) bits/sec
3. Compute the values of H and R, if in the above example, the quantities levels are
so chosen that they are equally likely to occur,
Solution:
Average information per message is
H = 4 ( log
2
4) = 2 bits/message
and R =
s
r H = 2B (2) = (4B) bits/sec
Markoff Model for Information Sources
Assumption
A source puts out symbols belonging to a finite alphabet according to certain
probabilities depending on preceding symbols as well as the particular symbol in
question.
- Define a random process
A statistical model of a system that produces a sequence of symbols stated above
is and which is governed by a set of probs. is known as a random process.
Therefore, we may consider a discrete source as a random process
and
the converse is also true.
i.e. A random process that produces a discrete sequence of symbols chosen from a
finite set may be considered as a discrete source.
- Can you give an example of such a source?
- What is a discrete stationary Markoff process?
Provides a statistical model for the symbol sequences emitted by a discrete
source.
General description of the model can be given as below:
1. At the beginning of each symbol interval, the source will be in the one of n possible
states 1, 2, .. n
Where n is defined as
n (M)
m
M = no of symbol / letters in the alphabet of a discrete stationery source,
m = source is emitting a symbol sequence with a residual influence lasting
m symbols.
i.e. m: represents the order of the source.
m = 2 means a 2
nd
order source
m = 1 means a first order source.
The source changes state once during each symbol interval from say i to j. The
proby of this transition is P
ij
. P
ij
depends only on the initial state i and the final state j but
does not depend on the states during any of the preceeding symbol intervals.
2. When the source changes state from i to j it emits a symbol.
Symbol emitted depends on the initial state i and the transition i j.
3. Let s
1
, s
2
, .. s
M
be the symbols of the alphabet, and let x
1
, x
2
, x
3
, x
k
, be a
sequence of random variables, where x
k
represents the k
th
symbol in a sequence
emitted by the source.
Then, the probability that the k
th
symbol emitted is s
q
will depend on the previous
symbols x
1
, x
2
, x
3
, , x
k1
emitted by the source.
i.e., P (X
k
= s
q
/ x
1
, x
2
, , x
k1
)
4. The residual influence of
x
1
, x
2
, , x
k1
on x
k
is represented by the state of the system at the beginning of the
k
th
symbol interval.
i.e. P (x
k
= s
q
/ x
1
, x
2
, , x
k1
) = P (x
k
= s
q
/ S
k
)
When S
k
in a discrete random variable representing the state of the system at the
beginning of the k
th
interval.
Term states is used to remember past history or residual influence in the same
context as the use of state variables in system theory / states in sequential logic circuits.