Two alternative approaches to prove

Shannon’s Source Coding Theorem I

Dushyant Kamlesh Pathaka,1
a
DA-IICT,Gandhinagar

Abstract
The text is an effort to present two alternate proofs to the Source Coding Theorem,an
integral part of C.E.Shannon’s classic paper[1] published in 1945.One of them focuses on the
Mathematical and Manipulative aspect of the proof,whereas the other involves an in-depth
understanding of what is actually happening.
Keywords: Source Coding,Entropy,Surviving Subsets,Code Rate

1. The Statement
As stated by Shannon in his classic 1948 paper [1]
It is impossible to compress the data such that the code rate (average number of bits per
symbol) is less than the Shannon entropy of the source, without it being virtually certain
that information will be lost. However it is possible to get the code rate arbitrarily close to
the Shannon entropy, with negligible probability of loss.
Hence
R ≥ H(X)

2. Proof 1:
Consider M to be the total number of transmitted bits.
The total number of subsets is given by M + 1
Total number of sequences = 2M
As
lim
M →∞

Only one subset will survive.This surviving subset is called as the typical set.
As M increases to a very large value,it is only the typical set which will survive.Hence the
probability of the typical set
lim ptyp = 1
M →∞

I
A different interpretation of the proof presented by the legendary C.E.Shannon in 1948
Email address: dushyantpathak6789@gmail.com (Dushyant Kamlesh Pathak)
Preprint submitted to Elsevier May 5, 2018
and the other subsets have vanishing probability of occurence.
The question that arises is:
what is the defining property of the typical set?
The answer to this lies in the definition of probability,given by[2]
Mtyp
p = lim
M →∞ M

[3]
Therefore,
Mtyp = M p
where Mtyp stands for the number of ones in any element of the typical set.
It follows that,number of zeroes=M (1−p) Probability of each sequence=p.p.p....M p ones∗(1−
p).(1 − p)...(1 − p)M zeroes
pi = ppM (1 − p)(1−p)M
If there are k sequences in the surviving set

ptyp = kpi

Therefore,
k = p−pM (1 − p)−(1−p)M
This is the size of the typical set,wit each j member with a probability of pi
Looking analogically
2M sequences require M bits
Hence k sequences will require log2 k bits
log2 k =
log2 (p−pM (1 − p)(1−p)M )
=
M (−p log2 p − (1 − p) log2 (1 − p))
The term in multiplication with M is H(X) for a binary channel, where H(X),the Entropy
or the degree or randomness is defined as being equal to
K
X K
X
pk nk = − pk log2 pk
k=1 k=1

Hence we get

k = 2M H(X)
Hence instead of M bits we need only MH(X)bits. Rate R is defined as
R=(Required Number of bits to represent information)/(Total number of bits)
M H(X)
=
M
2
 = H(X)

We know that the least number of bits required to size the entire message signal is MH(X).
Hence the least value of R is that given by equation 1. Hence we have

R ≥ H(X)

This proves the Source Coding theorem.

3. Proof:2
[4]
Entropy H(X) is defined as
L
X
H(X) = − pxi log2 pxi
i=1

Now entropy or the degree of randomness will be maximum when all symbols have equal
probability of occurrence.Hence the receiver wouldn’t be able to decide which symbols ac-
tually occurred.Hence for L symbols,each having equal probability,
1
p=
L
Therefore
L
X
H(X) = − pi log2 pi
i=1

This will be maximum when

1
pi =
L
H(X)max = log2 L
Hence

H(X) ≤ log2 L
Code Rate=Average number of bits per symbol Since there are L possible symbols. There-
fore,when L is a power of 2
R = log2 L
(from the analogy used in 1)
And when L is not a power of 2

3
 R = dlog2 Le
For example if L=5 we require
R = d5e = 3
bits to represent the symbols. Hence

R ≥ log2 L ≥ H(X)

Hence
R ≥ H(X)

This proves the Source Coding Theorem.

4. Acknowledgements
I am sincerely grateful and indebted to my CT111 course instructor Prof.Yash M Vasavada,who
introduced me to this wonderful topic of source coding theorem and also helped in guiding
my efforts in the writing of this description.I also am grateful and acknowledge the help I
have received from his thoughtfully planned Lecture slides and notes which helped me clear
the topic in my mind. I would like to offer my sincere gratitude to my present institute DA-
IICT,Gandhinagar for providing me with this opportunity to fuel my thought process.Also
am I thankful to my parents for helping me throughout my dedication to this work.

Dushyant Kamlesh Pathak

4
References
[1] Claude E Shannon. A mathematical theory of communication. 1948.
[2] Simon Haykin. Communication Systems.
[3] Yash M Vasavada. Lecture 12:source coding.
[4] John G Proakis. Digital Communications.