Chapter 3

Source Coding
Motto: To defeat yourself is the first
and the most beautiful of
all victories.
Democritus

3.1 What Is Coding and Why Is It Necessary?

Information is rarely transmitted directly to the receiver, without any processing,
due to the followings:
• source alphabet is different from channel one, therefore some adaptation of the
source to the channel is needed (information representation codes).
• channel must be very efficiently used (close as possible to its full capacity),
meaning that the source must be converted into an optimal one (pio) (statistical
adaptation of the source to the channel from information theory point of view);
keeping in mind the coding theory, we may say that for an efficient use of the
channel - in order to minimize transmission time and/or storage space - source
compression is needed (compression codes).
• it is also necessary to adapt the source to the channel in order to match the
spectrum to channels characteristics and also to ensure synchronization be-
tween transmitter and receiver (base band codes / line codes).
• information transmitted over a noisy channel is distorted by channel noise; this
is why error detecting and correcting codes are used in error control procedures
(error control codes).
• information confidentiality from unauthorized persons must be provided in
some applications (encryption).
The need for source processing prior to transmission (or storage) is leading to
the processing known as coding.
For a better understanding of coding, we will divide it into three main parts:
1. source coding (Cs) or coding for noiseless channels.
2. encryption (E), i.e. coding for keeping information confidentiality.
3. channel coding (Cc) – protection to channel noise (error control) and channel
adaptation (base band codes).
In real transmission, one, two or all these aspects arise, the necessary process-
ing degree being imposed by the application itself.
54 3 Source Coding

Coding is the process of finding a bijective correspondence between source (S)

messages and codewords set (C) obtained using the alphabet X.
Be a discreet memoryless source S and the corresponding codewords set C:

⎛ si ⎞ M
S : ⎜⎜ ⎟⎟, i = 1, M, ∑ p i = 1 (3.1)
⎝ p i = p(s i ) ⎠ i =1

⎛ ci ⎞
C : ⎜⎜ ⎟⎟, i = 1, M, p(c i ) = p(s i ) (3.2)
⎝ i
p = p(c i ⎠
)

where ci is a codeword and represents a finite row of symbols xj belonging to code

alphabet X, assumed memoryless too:
⎛ xj ⎞ m
X:⎜ ⎟, j = 1, m, ∑ p j = 1 (3.3)
⎜ p j = p(x j ) ⎟ j=1
⎝ ⎠
By encoding, the initial source S is transformed into a secondary one X, enter-
ing the channel (Fig. 3.1)

Fig. 3.1 Illustration of the transformation SÆX realized through encoding

For a source S and a given alphabet X, we may create a multitude of codes.

Table 3.1 shows some binary codes put into correspondence with a quaternary
(M = 4) source S.

Table 3.1 Binary codes associated to a quaternary source (M=4)

S A B C D E
S1 00 0 0 0 0
S2 01 01 10 10 100
S3 10 011 110 110 11
S4 11 0111 1110 111 110

Next, we will define some codes frequently used in applications:

• Uniform code is a code with same lengths of codewords (e.g.: code A).
• Non-uniform code is a code with different lengths of codewords (e.g.: codes B,
C, D, and E).