IT Lec2

LÝ THUYẾT THÔNG TIN
(INFORMATION THEORY)
DƯƠNG HIỂN THUẬN
0918-125-540
thuan.duong@sgu.edu.vn
dhthuan@gmail.com
Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

www.sgu.edu.vn thuan.dh@cb.sgu.edu.vn
• What is the minimum number of bits needed to represent a source?
• What is the maximum rate of communication?
Mutual
Entropy
Information

• Theory of error correcting codes (Channel coding) - is one of the most interesting
and applied part of mathematics and informatics.
• All real systems that work with digitally represented data, as CD players, TV, fax
machines, internet, satelites, mobiles, require to use error correcting codes
because all real channels are, to some extent, noisy.
▪ Channel coding theory problems are therefore among the very basic and most
frequent problems of storage and transmission of information.
▪ Channel coding theory results allow to create reliable systems out of unreliable
systems to store and/or to transmit information.
▪ Coding theory methods are often elegant applications of very basic concepts and
methods of (abstract) algebra.
• Chapter presents and illustrates the very basic problems, concepts, methods and
results of channel coding theory.

• Without error-correcting codes there would be no deep-space
travel and pictures, no satelite TV, no compact disc, no … no … no
….
• Error-correcting codes are used to correct messages when they
are transmitted through noisy channels.
Error correcting framework

Example
A code C over an alphabet S is a subset of S* - (C  S*).

A q -nary code is a code over an alphabet of q -symbols.
A binary code is a code over the alphabet {0,1}.
Examples of codes C1 = {00, 01, 10, 11} C2 = {000, 010, 101, 100}
C3 = {00000, 01101, 10111, 11011}

NOISE
may be caused by sunpots, lighting, meteor showers, random radio
disturbance, poor typing, poor hearing, ….
TRANSMISSION GOALS
1. Fast encoding of information.
2. Easy transmission of encoded messages.
3. Fast decoding of received messages.
4. Reliable correction of errors introduced in the channel.
5. Maximum transfer of information per unit time.
METHOD OF FIGHTING ERRORS: REDUNDANCY!!!

0 is encoded as 00000 and 1 is encoded as 11111.

BASIC IDEA
• The details of techniques used to protect information against
noise in practice are sometimes rather complicated, but basic
principles are easily understood.
• The key idea is that in order to protect a message against a

noise, we should encode the message by adding some
redundant information to the message.
• In such a case, even if the message is corrupted by a noise,

there will be enough redundancy in the encoded message to
recover, or to decode the message completely.

Channel Coding
• class of signal transformations designed to
improve communication performance by
enabling the transmitted signals to better
withstand channel distortions such as noise,
interference, and fading.
• Channel coding can be divided into two

major classes:
1. Waveform coding by signal design
2. Structured sequences by adding redundancy

Waveform Coding
– deals with transforming waveform into “better
waveform” robust to channel distortion hence
improving detector performance.
• Examples:
– Antipodal signaling
– Orthogonal signaling
– Bi-orthogonal signaling
– M-ary signaling
– Trellis-coded modulation

Structured Sequences
– deals with transforming sequences into “better sequences” by adding
structured redundancy (or redundant bits). The redundant bits are used to
detect and correct errors hence improves overall performance of the
communication system.
• Examples:
➢ Error detection: Parity check, CRC, Check Sum
➢ FEC (forward error coding)
– Linear codes
• Hamming codes
• BCH codes
• Cyclic codes
• Reed-Solomon codes
– Non-Linear codes
• Convolution codes
• Turbo codes
➢ ARQ (automatic repeat request):
• stop-and-wait/ go-back-N/ selective repeat … etc.

- Mã khối (Block codes: a code with all
words of the same length) – Linear Codes
+ Tuyến tính, đối xứng, đơn giản, sửa
ít bit lỗi
- Mã tích chập (Convolutional codes) –

Nonlinear Codes
+ Phức tạp hơn, hiệu quả sửa lỗi tốt
hơn và có thể giải mã mềm

• Mã khối: mã có các từ mã có cùng chiều dài, k bits thông
tin được mã hoá thành n bits từ mã (codework), n-k bits được
thêm vào để phát hiện sai và sữa sai, tỷ lệ mã k/n
- Trọng số Hamming (the number of nonzero components
it contains)
- Khoảng cách Hamming (The Hamming distance
between two words is the number of differences between
corresponding bits)
1. The Hamming distance d(000, 011) is 2 because
2. The Hamming distance d(10101, 11110) is 3 because

Properties of Hamming distance
(1) d(x, y) = 0  x = y
(2) d(x, y) = d(y, x)
(3) d(x, z) ≤ d(x, y) + d(y, z) triangle inequality
- Từ sai (error codework)

- Mã phát hiện sai
An important parameter of codes C is their minimal distance

(khoảng cách của bộ mã).
d(C) = min {d(x, y) | x,y  C, x  y},
because it gives the smallest number of errors needed to
change one codeword into other.

Khoảng cách bộ mã d là khoảng cách Hamming bé nhất
trong bộ mã (The minimum Hamming distance is the mallest
Hamming distance between all possible pairs in a set of
words)→ phát hiện các từ sai có trọng số Hamming ≤ (d-1)
và có ít nhất một từ sai có trọng số Hamming là d bộ mã
không phát hiện được. Bô mã này sửa được các từ sai có
trọng số Hamming ≤ phần nguyên của [(d-1)/2].
Theorem Basic error correcting theorem

(1) A code C can detect up to s errors if d(C) ≥ s + 1.
(2) A code C can correct up to t errors if d(C) ≥ 2t + 1.

Geometric concept for finding dmin in error detection

Geometric concept for finding dmin in error correction

Find the minimum Hamming distance of the coding
scheme C = {000, 011, 101, 110}
Solution
We first find all Hamming distances.
The dmin in this case is 2.

Find the minimum Hamming distance of the coding
scheme C={00000, 01011, 10101,11110}
Solution
We first find all the Hamming distances.
The dmin in this case is 3.

Basic assumptions about
channels
1. Code length preservation Each
output codeword of a channel has the
same length as the input codeword.
2. Independence of errors The
probability of any one symbol being
affected in transmissions is the same.

- Nguyên lý sửa sai
Basic strategy for decoding
For decoding we use the so-called maximal
likehood principle, or nearest neighbor decoding
strategy, which says that the receiver should
decode a word w' as that codeword w that is the
closest one to w'.
The intuitive concept of “closeness'' of two words
is well formalized through Hamming distance d(x,
y) of words x, y.

- Mã tuyến tính là bộ mã sao cho tổng (exclusive OR
(XOR)) của 2 từ mã thuộc bộ mã là một từ mã thuộc
bộ mã.
- Mã khối tuyến tính C(n,k), k là chiều dài bản tin
đựơc mã hoá, n là chiều dài từ mã, số từ mã trong bộ
mã |C|=2k, và được tạo từ ma trân sinh Gkxn ma trận
có k dòng và n cột. Các từ mã trong bộ mã C là các từ
mã {v} sao cho v = u × G với u là bản tin bất kỳ.
- Gk×n là ma trân sinh của bộ mã C thì ma trận tương
đương dòng của ma trận G’ cũng là ma trận sinh của
bộ mã C

Matrices A and B are said to be row equivalent if
B can be obtained from a by a finite sequence of
three types of elementary row operations:
➢ multiply all elements of a row by a scalar
➢ interchange a pair of rows
➢ add a scalar multiple of one row to another
row.
If A and B are row equivalent we write A ≅ B.
- Ma trận thử của bộ mã C(n,k) được tạo từ ma
trận sinh G được định nghĩa là là ma trận Hn×(n-k) có
n dòng và n-k cột sao cho:
G  H = 0; H T  G T = 0
Hay: vH = 0

- Nếu ma trận G có dạng G = [I X], Ik là ma trận đơn vị, Xk×(n-k) là
ma trận bất kỳ có k dòng và n-k cột thì ma trận H sẽ là
 X k ( n − k ) 
H = 
 I n−k 
Notation: An (n,k,d) - code C is a code such that

• n - is the length of codewords.
• k - is the length of messages.
• d - is the minimum distance in C.
Comment: A good (n,k,d) code

has small n and large k and d.

➢ Ví dụ: tìm ma trận thử của bộ mã tuyến tính có
ma trận sinh như sau:

Channel Decoding
1. Brut force Approach: list all possible messages
2. Syndrome Decoding
Brute Force Approach:
List all messages
This is done by generating a list of all the possible messages. For
something small like the Hamming [7,4] codes the task is feasible, but
for codes of greater length it is not. An example of a list is as follows:
Code words 1000011 0100101 0010110 …
0000011 0000101 0000110
Other 1000001 0100111 0010100
Received 0010011 0001101 1010110
Words 1100011 1100101 0110110
… … …
Brute Force Approach:
List all messages
For example, if the received code was 0001101 then it would
be decoded to 0100101 from the list.
Code words 1000011 0100101 0010110 …

0000011 0000101 0000110
Other 1000001 0100111 0010100
Received 0010011 0001101 1010110
Words 1100011 1100101 0110110
… … …

- Giải mã bằng ma trận thử: (v phát và
thu được w) nếu thu được từ mã w
không thuộc bộ mã C → bị sai. Tính S =
wH được gọi là syndrome của từ mã w.
Ứng với mỗi syndrome có một từ sai
tương ứng (ký hiệu e) và lấy từ sai này
cộng với từ mã thu được bị sai sẽ cho ra
từ mã đúng ➔ e + w = v

Syndrome Decoding
Consider a transmitted code cm and w is the received sequence,
w can be expressed as,
w = cm + e
where e denotes binary error vector.
The decoder calculate product
s = wH = ( cm + e ) H
= cm H + eH = eH
(n – k)–dimensional vector s is called the syndrome of the error
pattern. Or in other words, s contains the pattern of failure in
parity checks.

Standard Array
• Let us arrange the 2n n-tuples that represent all possible received
sequences in an array, such that first row contains all the
codewords (2k) and first column contains all correctable error
patterns. The standard array format of (k,n) code is as follows:
c0 c1 ci c2k −1
e1 c1 + e1 ci + e1 c2k −1 + e1
e2 c1 + e2 ci + e2 c2k −1 + e2
e2n−k −1 c1 + e2n−k −1 ci + e2n−k −1 c2k −1 + e2n−k −1

e2n−k c1 + e2n−k ci + e2n−k c2k −1 + e2n−k
Standard Array
• Note that codeword c0 is trivial codeword, that is, all-
zeros codeword. In addition, it is also error pattern of
all-zeros.
• Each row in the standard array is called a coset.
• An error pattern in the first column, called coset leader.
• A coset leader is the vector with the minimum weight in
the coset.
• Each coset consists of 2k n-tuples ➔
there are 2n –k cosets.

Standard Array
A standard array can be created as follows:
• List the codewords of C, starting with 0, as the first row
• Choose any vector of minimum weight not already in the
array. Write this as the first entry of the next row. This
vector is denoted the 'coset leader'.
• Fill out the row by adding the coset leader to the
codeword at the top of each column. The sum of the i-th
coset leader and the j-th codeword becomes the entry in
row i, column j.
• Repeat steps 2 and 3 until all rows/cosets are listed and
each vector appears exactly once.
Syndrome Decoding
The first step is to create a list of syndromes corresponding the
coset leaders. The syndrome of each vector w is found by
S = syn( w) = wH
When a code is received, the syndrome is computed and

compared to the list of syndromes. Let the coset leader to
the syndrome by e. Finally the code is decoded to c = w + e.

Types of Errors

Single-bit error

Single bit errors are the least likely type of
errors in serial data transmission because the
noise must have a very short duration which
is very rare. However this kind of errors can
happen in parallel transmission.
Example:
If data is sent at 1Mbps then each bit lasts
only 1/1,000,000 sec. or 1 μs.
For a single-bit error to occur, the noise must
have a duration of only 1 μs, which is very
rare.

Burst error

The term burst error means that two or
more bits in the data unit have changed from
1 to 0 or from 0 to 1.
Burst errors does not necessarily mean that

the errors occur in consecutive bits, the
length of the burst is measured from the first
corrupted bit to the last corrupted bit. Some
bits in between may not have been
corrupted.

Burst error is most likely to happen in serial
transmission since the duration of noise is
normally longer than the duration of a bit.
The number of bits affected depends on the data
rate and duration of noise.
Example:
If data is sent at rate = 1Kbps then a noise of 1/100 sec
can affect 10 bits.(1/100*1000)
If same data is sent at rate = 1Mbps then a noise of

1/100 sec can affect 10,000 bits.(1/100*106)
Error detection
Error detection means to decide whether the

received data is correct or not without having a
copy of the original message.
Error detection uses the concept of redundancy,

which means adding extra bits for detecting errors
at the destination.

Redundancy

Four types of redundancy checks are used
in data communications

Vertical Redundancy Check
VRC (Parity Check)

A simple parity-check code is a single bit
error detecting code in which n = k + 1 with
dmin = 2.
+ Even parity (ensures that a codeword has
an even number of 1’s) and
+ Odd parity (ensures that there are an odd
number of 1’s in the codeword)
Note: The parity bit is only suitable for
detecting errors; it cannot correct any errors,
as there is no way to determine which
particular bit is corrupted

Simple parity-check code C(5, 4)

Encoder and decoder for simple parity-check code

Let us look at some transmission scenarios. Assume the sender sends the
dataword 1011. The codeword created from this dataword is 10111, which
is sent to the receiver. We examine five cases:
1. No error occurs; the received codeword is 10111. The syndrome is 0.

The dataword 1011 is created.
2. One single-bit error changes a1 . The received codeword is 10011. The
syndrome is 1. No dataword is created.
3. One single-bit error changes r0 . The received codeword is 10110. The
syndrome is 1. No dataword is created.
4. An error changes r0 and a second error changes a3 . The received
codeword is 00110. The syndrome is 0. The dataword 0011 is created at the
receiver. Note that here the dataword is wrongly created due to the
syndrome value.
5. Three bits—a3, a2, and a1—are changed by errors. The received
codeword is 01011. The syndrome is 1.The dataword is not created. This
shows that the simple parity check, guaranteed to detect one single error,
can also find any odd number of errors.
Longitudinal Redundancy Check
LRC
(horizontal redundancy check): applied
independently to each of a parallel group of
bit streams. The data must be divided
into transmission blocks, to which the
additional check data is added

• Longitudinal Redundancy Check (LRC)
– Organize data into a table and create a parity for
each column
11100111 11011101 00111001 10101001
11100111
11011101
00111001
10101001
10101010
11100111 11011101 00111001 10101001 10101010

Original Data LRC

Performance
LCR increases the likelihood of detecting
burst errors.
If two bits in one data units are damaged
and two bits in exactly the same positions in
another data unit are also damaged, the LRC
checker will not detect an error.

VRC and LRC

Two-dimensional parity-check code: can detect up to three errors
that occur anywhere in the table (arrows point to the locations of
the created nonzero syndromes). However, errors affecting 4 bits
may not be detected

Cyclic Redundancy Check
CRC

- Mã vòng (cyclic code). Nếu dịch trái
hay phải một từ mã bất kỳ thuộc bộ
mã thì cho ra một từ mã thuộc bộ mã.
- Mã vòng có thể biểu diễn bằng ma
trận sinh và thử, tuy nhiện để đơn
giản thì biểu diễn bằng đa thức tạo
mã.
- Dự trên phép toán Modulo 2

Example: (6,2) code
C =  000000 , 010101 , 101010 , 111111  

is a cyclic code ?.

Example2: (5,2) linear block code
1 0 1 1 1
G= 
0 1 1 0 1
is a single error correcting code, the set of codeword are:
Thus, it is not a cyclic code since, for example,

0 0 0 0 0 the cyclic shift of [10111] is [11011]
1 0 1 1 1
C=
0 1 1 0 1
  C
1 1 0 1 0

• Powerful error detection scheme
• Rather than addition, binary division is
used → Finite Algebra Theory (Galois
Fields)
• Can be easily implemented with small
amount of hardware
– Shift registers
– XOR (for addition and subtraction)

VÍ DỤVỀ PHÉP TOÁN MODULO 2 (mod 2)
• Trong trường GF(2), các hệ số của đa thức là các số 1 và 0.
• Ví dụ cộng hai đa thức:
( x 3 + x) + ( x + 1) = x 3 + 2 x + 1 = x 3 + 1 (mod 2)
Vì 2x = x + x = x(1 + 1) = x.0 = 0 (mod 2)

• Phép nhân cũng vậy:
( x 2 + x).( x + 1) = x 3 + 2 x 2 + x = x 3 + x (mod 2)
• Chúng ta cũng có thể chia đa thức theo mod 2 để tìm thương
(quotient) và số dư (remainder):
x3 + x 2 + x 1 1
= ( x + 1) −
2
= ( x + 1) +
2
(mod 2)
x +1 x +1 x +1
Cyclic Redundancy Check
• Given a k-bit frame or message, the
transmitter generates an n-bit sequence,
known as a frame check sequence (FCS),
so that the resulting frame, consisting of
(k+n) bits, is exactly divisible by some
predetermined number.
• The receiver then divides the incoming
frame by the same number and, if there is
no remainder, assumes that there was no
error.
• Let us assume k message bits and
n bits of redundancy
xxxxxxxxxx yyyy Block of length k+n
k bits n bits
• Associate bits with coefficients of a

polynomial
1 0 1 1 0 1 1
1x6+0x5+1x4+1x3+0x2+1x+1
= x6+x4+x3+x+1

CÁCH BIỂU DIỄN MỘT CHUỖI SỐ NHỊ
PHÂN THÀNH ĐA THỨC NHỊ PHÂN
Chuỗi bit b(m-1) b(m-2) … b2 b1 b0

Số bit: m
Đa thức nhị phân biểu diễn chuỗi bit trên là:
b(m-1).x(m-1) + b(m-2).x(m-2) + … + b2.x2 + b1.x1 + b0.x0
• Vídụ: Chuỗi bit 101 có thể được biểu diễn dưới dạng đa thức
nhị phân như sau:
1.x 2 + 0.x1 + 1.x 0 = x 2 + 1

• Let M(x) be the message polynomial
• Let P(x) be the generator polynomial (hay G(x))
– P(x) is fixed for a given CRC scheme
– P(x) is known both by sender and receiver
• Create a block polynomial F(x) based on M(x)
and P(x) such that F(x) is divisible by P(x)
F ( x) 0
= Q( x) +
P( x) P( x)

• Sending
1. Multiply M(x) by xn
2. Divide xnM(x) by P(x)
3. Ignore the quotient and keep the reminder R(x)
4. Form and send F(x) = xnM(x)+R(x)
• Receiving
1. Receive F’(x)
2. Divide F’(x) by P(x)
3. Accept if remainder is 0, reject otherwise

Prove that x n M ( x) + R( x) is divisible by P( x)
Q( x)
P ( x) x n M ( x) , remainder R( x)
 x n M ( x) = P ( x)Q( x) + R( x)
x n M ( x) + R ( x) P ( x)Q ( x) R ( x) + R ( x)
= +
P( x) P( x) P( x)
Remainder 0 Remainder 0
Note: Binary modular addition is equivalent to
binary modular subtraction → R(x) + R(x) = 0

• Send • Receive
– M(x) = 110011 → x5+x4+x+1 (6 bits)
– P(x) = 11001 → x4+x3+1 (5 bits, n = 4) 11001 1100111001
→ 4 bits of redundancy
11001
– Form xnM(x) → 110011 0000
→ x9+x8+x5+x4 11001
– Divide xnM(x) by P(x) to find R(x)
11001
100001 00000
11001 1100110000
11001 No remainder
10000 → Accept
11001
1001 = R(x)
Send the block 110011 1001

Hardware Implementation
Circuit: 1100101
• Message = 1011011 k=7 1101 0000000000
P(x) = 1101 = x3+x2+x0 n=3 1000
1101
Conventional 1100101 1 1010
Method: 1010
1101 1011011000 1101
1101 0 1110
1100 0110
1101 0000
1 1100
Message
0011
0100
0000 0000
0111 1
1000
0000 1000
1110 0 1101
1101 1010
0110 0010
1 0000
0000 0100
1100 1100
1101 1
1101
001 001

Transmit:
Data SQ Bit 2 Bit 1 Bit 0
MSB
1 1 0 0 0
Serial Quotient (SQ) 0 1 1 0 1
x3 1 0 1 1 1
Message
x2
x0 1 0 1 1 0
+ 2 + 1 0 0 1 1 0 0
1 0 1 0 1
Data
1 1 0 1 0
LSB
Input 0 0 1
CRC
k shifts later, CRC is in register Send MSB first
Shift out (without any XOR) in n shifts

Receive:
Data SQ Bit 2 Bit 1 Bit 0
MSB
1 1 0 0 0
Serial Quotient (SQ) 0 1 1 0 1
x3 1 0 1 1 1
Message
x2
x0 1 0 1 1 0
+ 2 + 1 0 0 1 1 0 0
1 0 1 0 1
Data
1 1 0 1 0
LSB
Input
MSB
0 0 0 0 1
CRC 0 0 0 1 0
n+k shifts later, remainder is 0 1 0 1 0 0
Data accepted 0 0 0
LSB

x2
A x0
+
+ 2
1 0
Data In
A
Control Line A:
A 1: Make/Test CRC
0: Shift Out CRC
Data OK
For Transmitting:
Data Out Assert A true while feeding k bits of message
Assert A false for n clock cycles to output CRC
For Receiving:
A Assert A true while feeding k+n bits of message and CRC
Ignore Data Out, check Data OK for correctness

MỘT SỐ ĐA THỨC SINH P(x) or G(x)
CRC-n P(x) or G(x) USE
CRC-1 x+1 Hardware (parity bit)
CRC-4 x4 + x + 1 PCM-30
CRC-5 - CCITT x5 + x3+ x+1 ITU-G.704
CRC-5 - USB x5 + x2+ 1 USB token packets
CRC-7 x7 + x3+ 1 Telecom systems, MMC
CRC-8
x8 + x 7 + x 6 + x 4 + x2 + 1
CRC-12 x12 + x11 + x3 + x2 + x + 1 use: telecom systems
CRC-32 - MPEG2 x32 + x26 + x23 + x22 + x16 + x12 + x11
+ x10 + x8 + x7 + x5 + x4 + x2 + x + 1
…

Division in CRC encoder

Division in the CRC decoder for two cases

• Theorem Suppose C is a cyclic code of codewords of length
n+k with the generator polynomial
g(x) = g0 + g1x + … + gnxn.
• Then dim (C) = k and a generator matrix G for C is
 g0 g1 g2 ... gn 0 0 0 ... 0
 
0 g0 g1 g2 ... gn 0 0 ... 0 
G = 0 0 g0 g1 g2 ... gn 0 ... 0 
 
 .. .. .. 
0 0 ... 0 0 ... 0 g0 ... g n 


• Example: k = 4, n = 3
g(x) = 1 + x + x3
• Then a generator matrix G have dimension 4 x 7 as the
following

Checksum
The checksum is used in the Internet by several protocols although not at the data
link layer. Used by upper layer protocols

At the sender
The unit is divided into k sections, each of n
bits.
All sections are added together using one’s
complement to get the sum.
The sum is complemented and becomes the
checksum.
The checksum is sent with the data

At the receiver
The unit is divided into k sections, each of n
bits.
All sections are added together using one’s
complement to get the sum.
The sum is complemented.
If the result is zero, the data are accepted:
otherwise, they are rejected.

Performance
The checksum detects all errors involving an odd
number of bits.
It detects most errors involving an even number
of bits.
If one or more bits of a segment are damaged
and the corresponding bit or bits of opposite
value in a second segment are also damaged, the
sums of those columns will not change and the
receiver will not detect a problem.

Suppose our data is a list of five 4-bit numbers
that we want to send to a destination. In addition
to sending these numbers, we send the sum of the
numbers. For example, if the set of numbers is (7,
11, 12, 0, 6), we send (7, 11, 12, 0, 6, 36), where 36
is the sum of the original numbers.
The receiver adds the five numbers and compares
the result with the sum. If the two are the same,
the receiver assumes no error, accepts the five
numbers, and discards the sum. Otherwise, there
is an error somewhere and the data are not
accepted.
We can make the job of the receiver
easier if we send the negative
(complement) of the sum, called the
checksum. In this case, we send (7, 11,
12, 0, 6, −36). The receiver can add all
the numbers received (including the
checksum). If the result is 0, it assumes
no error; otherwise, there is an error.
How can we represent the number 21 in one’s
complement arithmetic using only four bits?
Solution
The number 21 in binary is 10101 (it needs five bits). We
can wrap the leftmost bit and add it to the four rightmost
bits. We have (0101 + 1) = 0110 or 6.

How can we represent the number −6 in
one’s complement arithmetic using only
four bits?
Solution
In one’s complement arithmetic, the negative
or complement of a number is found by
inverting all bits. Positive 6 is 0110; negative
6 is 1001. If we consider only unsigned
numbers, this is 9. In other words, the
complement of 6 is 9.
Hamming Codes
• Detect and correct 1-bit errors.
• Codes are of form: (2r-1, 2r-1 – r, 3) for any r > 1
• e.g. (3,1,3), (7,4,3), (15,11,3), (31, 26, 3), …
• which correspond to 2, 3, 4, 5, … “parity bits”
(i.e. n-k)
• The high-level idea is to “localize” the error.

General algorithm
▪ Number the bits starting from 1: bit 1, 2, 3, 4, 5, etc.
▪ Write the bit numbers in binary: 1, 10, 11, 100, 101, etc.
▪ All bit positions that are powers of two (have only one 1 bit in the binary
form of their position) are parity bits: 1, 2, 4, 8, etc. (1, 10, 100, 1000)
▪ All other bit positions, with two or more 1 bits in the binary form of their
position, are data bits.
▪ Each data bit is included in a unique set of 2 or more parity bits, as
determined by the binary form of its bit position.

➢ Parity bit 1 covers all bit positions which have the least significant bit set: bit 1
(the parity bit itself), 3, 5, 7, 9, etc.
➢ Parity bit 2 covers all bit positions which have the second least significant bit set:
bit 2 (the parity bit itself), 3, 6, 7, 10, 11, etc.
➢ Parity bit 4 covers all bit positions which have the third least significant bit set:
bits 4–7, 12–15, 20–23, etc.
➢ Parity bit 8 covers all bit positions which have the fourth least significant bit set:
bits 8–15, 24–31, 40–47, etc.
✓ In general each parity bit covers all bits where the bitwise AND of the parity
position and the bit position is non-zero.

• With code words of length 12, we observe that each
of the bits, numbered 1 though 12, can be expressed
in powers of 2. Thus:
1 = 20 5 = 22 + 2 0 9 = 23 + 2 0
2 = 21 6 = 22 + 2 1 10 = 2 3 + 2 1
3 = 21+ 2 0 7 = 22 + 21 + 2 0 11 = 2 3 + 2 1 + 2 0
4 = 22 8 = 23 12 = 2 3 + 2 2
– 1 (= 20) contributes to all of the odd-numbered
digits.
– 2 (= 21) contributes to the digits, 2, 3, 6, 7, 10, and
11.
– . . . And so forth . . .
• We can use this idea in the creation of our check bits.

• Using our code words of length 12, number
each bit position starting with 1 in the low-
order bit.
• Each bit position corresponding to a power of
2 will be occupied by a check bit (P).
• These check bits contain the parity of each bit
position for which it participates in the sum.

• Since 1 (=20) contributes to position 3 , 5, 7, 9, and
11, bit 1 will check parity over bits in these positions.
• Since 2 (= 21) contributes to position 3, 6, 7, 10, and
11, bit 2 will check parity over these bits.
• For the word 11010110 (base10=214), assuming
even parity (values: even 0 and odd 1), we have a
value of 1 for check bit 1, and a value of 0 for check
bit 2.
What are the values for the other parity bits?

• The completed code word is shown above.
– Bit 1 checks the bits 3, 5, 7, 9, and 11, so its value is 1
to ensure even parity within this group.
– Bit 2 checks the bits 3, 6, 7, 10, and 11, so its
value is 0
– Bit 4 checks the bits 5, 6, 7, and 12, so its value is 1.
– Bit 8 checks the bits 9, 10, 11, and 12, so its value is
also 1.
• Using the Hamming algorithm, we can not only detect
single bit errors in this code word, but also correct them!

• Suppose an error occurs in bit 5, as
shown above. Our parity bit values are:
– Bit 1 checks 3, 5, 7, 9, and 11. This is incorrect as
we have a total of 2 ones (even would be 0).
– Bit 2 checks bits 3, 6, 7, 10, and 11. The parity is
correct.
– Bit 4 checks bits 5, 6, 7, and 12. This parity is
incorrect, as we 2 ones.
– Bit 8 checks bit 9, 10, 11, and 12. This parity is
correct.
• We have erroneous parity for check bits 1 and 4.
• With two parity bits that don’t check, we know that
the error is in the data, and not in a parity bit.
• Which data bits are in error? We find out by
adding the bit positions of the erroneous bits.
• Simply, 1 + 4 = 5. This tells us that the error is in
bit 5. If we change bit 5 to a 1, all parity bits check
and our data is restored.
• If only one parity bit indicates an error, the parity bit
itself is in error
Bose-Chaudhuri-Hocquengham (BCH) Codes (59-60)

the polynomial of lowest degree having coefficients of a specified type, such
that α is a solution of the equation in which the polynomial is equated to zero

1 ( x ) =  2 ( x ) =  4 ( x ) =  8 ( x )
 3 ( x ) =  6 ( x ) =  9 ( x ) = 12 ( x )
 5 ( x ) = 10 ( x )
 7 ( x ) = 14 ( x )
The code has minimal Hamming distance at least 5 and corrects up to two errors (d = 5)

The code has minimal Hamming distance at least 7 and corrects up to three errors (d = 7)

Reed-Solomon
codes
(Tự đọc)
Cyclic Codes
BCH Codes
Linear
Codes
Hamming Codes

• Mã tích chập (convolutional code) còn gọi là mã xoắn. Không
giống với mã khối là nhóm các bit thành một khối rồi mã mà chuổi
bit thông tin liên tục được biến đổi thành chuổi bit liên tục ở ngõ ra
của bộ mã hoá. Quá trình biến đổi này có một cấu trúc nhất định
(dùng các thanh ghi dịch) để bên thu có thể giải mã lại chuổi bit ban
đầu
N stages
k bits
1 k 1 k 1 k
1 2 n
k bit thông tin sẽ được mã hoá (mapping) thành n bit → tỷ lệ mã k/n

- Ví dụ cấu trúc mã xoắn ½ và 2/3
, y2(0) , y1(0) , y0(0)
+ chuổi bit vào:
, x2 , x1 , x0 x = ( x0 , x1 , x2 )
+ chuổi bit sau mã hoá:
y = ( y0(0) , y0(1) , y1(0) , y1(1) , y2(0) , y2(1) , )

, y2(1) , y1(1) , y0(1)
, x2(0) , x1(0) , x0(0) , y2(0) , y1(0) , y0(0)
, y2(1) , y1(1) , y0(1)
, x2(1) , x1(1) , x0(1) , y2(2) , y1(2) , y0(2)

+ chuổi bit vào:
x = ( x0(0) , x0(1) , x1(0) , x1(1) , x2(0) , x2(1) )
+ chuổi bit sau mã hoá:
y = ( y0(0) , y0(1) , y0(2) , y1(0) , y1(1) , y1(2) , y2(0) , y2(1) , y2(2) , )
Mỗi phần tử trong trong chuổi ra y đã được ghép xen là sự kết hợp tuyến tính của của các
phần tử của chuổi ngõ vào x(0),x(1),…,x(k-1).
Ví dụ trong bộ mã ½
y0(1) = x0(0) + 0 + 0
y (1)
i =x (0)
i +x (0)
i −1 +x
(0)
i −3
y1(1) = x1(0) + x0(0) + 0
y2(1) = x2(0) + x1(0) + 0
y3(1) = x3(0) + x2(0) + x0(0) y (1) = 1 + D + D3
y4(1) = x4(0) + x3(0) + x1(0)
D thể hiện toán tử trễ

Tương tự cho nhánh thứ nhất
y (0)
= 1+ D + D2 3
Mỗi đa thức thể hiện cho mối quan hệ giữa 1 input và 1 output. Ngoài ra người ta
còn ký hiệu đa thức tạo mã như sau
g (0) = (1011) = 1 + D 2 + D3
g (1) = (1101) = 1 + D + D3
Một cách tương tự ta có đa thức tạo mã cho bộ mã 2/3
g0(0) = (1001) g1(0) = (0110)

g0(1) = ( 0110 ) g1(1) = (1010 )
g0(2) = (1100) g1(2) = (0100)
Nhánh vào (0) Nhánh vào (1) Nhánh ra (1)

Constrain length K được định nghĩa là chiều dài lớn nhất của chuổi
bit của một ngõ ra còn bị ảnh hưởng bởi bất kỳ bit nào ở ngõ vào
K = 1 + max mi mi là số phần tử nhớ của nhánh ith

i
- Phân tích mã xoắn người ta dựa trên giản đồ trạng thái (state
diagram) và giản đồ trellis (giản đồ hình cây)
- Giản đồ trạng thái chứa các phần tử nhớ và trạng thái của các
phần tử nhớ (phụ thuộc số phần tử nhớ lớn nhất của một nhánh
bất kỳ) với các bit ngõ ra khi biết một tập các bit ngõ vào.
- Ví dụ bộ mã hoá ½ ta có nhánh dài nhất có 3 phần tử nhớ → có 8
trạng thái của 3 phần tử nhớ này, giả thiết ta đặt là S0 = 000,
S1=100, S2=010, S3=110, …, S7=111 theo nhị phân. Trên mỗi nhánh
nối 2 trạng thái thể hiện X/YY với X là bit vào và YY là bit ngõ ra

- Giản đồ trạng thái
1/10
S1 S3
1/11 0/01 1/01 1/00
1/01 0/11
0/00 1/00 1/11
S0 S2 S5 S7
0/10
0/11 0/10 0/00

1/10
0/01
S4 S6

- Giản đồ Trellis (hình cây)
S7
S6
S5
S4
S3
S2
S1
S0

Decoding
• A message m is encoded into the code
sequence c.
• Each code sequence represents a path in
the trellis diagram.
• Minimum Distance Decoding
– Upon receiving the received sequence r,
search for the path that is closest ( in
Hamming distance) to r .
- Giải mã theo thuật giải Viterbi – Maximun Likelihood (soft decoder and hard
decoder) → chọn đường giống nhất
S7
- Pano’s sequential decoding
- Stack algorithm
S6 - Feedback decoding
S5
S4
S3
S2
S1
S0

The Viterbi Algorithm
• Walk through the trellis and compute the Hamming
distance between that branch of r and those in the trellis.
• At each level, consider the two paths entering the same
node and are identical from this node onwards. From
these two paths, the one that is closer to r at this stage
will still be so at any time in the future. This path is
retained, and the other path is discarded.
• Proceeding this way, at each stage one path will be saved
for each node. These paths are called the survivors. The
decoded sequence is guaranteed to be one of these
survivors.

• Each survivor is associated with a
metric of the accumulated Hamming
distance (the Hamming distance up to
this stage).
• Carry out this process until the
received sequence is considered
completely. Choose the survivor with
the smallest metric.
• The viterbi algorithm is used to decode
convolutional codes and any structure or
system that can be described by a trellis.
• It is a maximum likelihood decoding
algorithm that selects the most probable
path that maximizes the likelihood
function.
• The algorithm is based on add-compare-
select t h e b e s t p a t h e a c h t im e a t e a c h
sta te .
Example: For the convolutional code example in the previous lecture, starting from state zero,
Decode the following received sequence.
At the end of the trellis,

select the path with the
minimum cumulative
Hamming weight
This is the
survival path
in this
example
Decoded sequence is
m=[10 1110]
Compute the two possible paths at each

Add the weight of the path at
each state
state and select the one with less
cumulative Hamming weight  This is called the
survival path

Distance Properties of Conv. Codes
• Def: The free distance, dfree, is the minimum Hamming
distance between any two code sequences.
• Criteria for good convolutional codes:
– Large free distance, dfree.
– Small Hamming distance (i.e. as few differences as possible )
between the input information sequences that produce the
minimally separated code sequences. dinf
• There is no known constructive way of designing a conv.
code of given distance properties. However, a given
code can be analyzed to find its distance properties.

• Convolutional codes are linear. Therefore, the
Hamming distance between any pair of code
sequences corresponds to the Hamming distance
between the all-zero code sequence and some
nonzero code sequence. Thus for a study of the
distance properties it is possible to focus on the
Hamming distance between the all-zero code
sequence and all nonzero code sequences.
• The nonzero sequence of minimum Hamming
weight diverges from the all-zero path at some
point and remerges with the all-zero path at
some later point.
• sequence 2: Hamming weight = 5, dinf = 1
• sequence 3: Hamming weight = 7, dinf = 3.

Turbo Codes
Data X X
• Proposed by Source
Convolutional
Encoder 1 Y1
Berrou & Glavieux Interleaving Y
in 1993 (Y1,
Convolutional Y2
Encoder 2 Y2)
• Advantages
X: Information
– Use very large block lengths Yi: Redundancy Information
– Have feasible decoding complexity
– Perform very close to capacity
• Limitation – delay, complexity
Turbo Codes: Decoder
De-interleaving
Convolutional
Y1 Interleaver
Decoder 1
X Interleaving Convolutional
Decoder 2 De-interleaving X’
Y2
X’: Decoded Information

Main progress in turbo
coding/decoding since 1993
• Max-Log-MAP and Max*-Log-MAP algorithms • Simplicity

• Sliding window • Simplicity
• Duo-binary turbo codes • Performance and simplicity
• Circular (tail-biting) encoding • Performance
• Permutations • Performance
• Parallelism • Throughput
• Computation or estimation of Minimum Hamming • Maturity
distances (MHDs)
• Stopping criterion • Power consumption
• Bit-interleaved turbo coded modulation • Performance and simplicity

• Properties of Turbo Codes (TCs)
– Rely on soft iterative decoding to achieve

high coding gains
– Good performance, near channel capacity
for long blocks
– Easy adaptation in the standard frame
• (easy block size adaptation to the MAC
layer)
– Well controlled hardware development
and complexity
– TC advantages led to recent adoption in
standards
Rate 1/2 Turbo Coding Performance

Interleaving
Input Data a1, a2, a3, a4, a5, a6, a7, a8, a9, …
a1, a2, a3, a4 Write

a5, a6, a7, a8
a9, a10, a11, a12
Read
Interleaving
a13, a14, a15, a16
Transmitting Data a1, a5, a9, a13, a2, a6, a10, a14, a3, …
Through Air
Received Data a1, a5, a9, a13, a2, a6, a10, a14, a3, …
a1, a2, a3, a4 Read

Write
De-Interleaving a5, a6, a7, a8

a9, a10, a11, a12
a13, a14, a15, a16
Output Data a1, a2, a3, a4, a5, a6, a7, a8, a9, …

Interleaving (Example)
Burst
error
Transmitting 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0,…
Data
Read
0, 1, 0, 0
Write 0, 1, 0, 0
De-Interleaving
0, 1, 0, 0
1, 0, 0, 0
Output Data 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, …
Discrete errors

Forward Error Correction vs ARQ
• FEC requires a higher level of redundancy than ARQ
• FEC cannot recover correctly from multiple burst
errors and hence there is a much higher residual error
rate
• ARQ can make use of very powerful (CRC) error
detecting codes and can detect virtually all errors
• The delay caused by retransmitting is preferable to
residual errors in most applications
• FEC tends to be used mainly when there is no return
channel (E.g. Teletext) or in real-time applications (E.g.
Video on Demand) or over unreliable media


IT Lec2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IT Lec2

Uploaded by

Copyright:

Available Formats

LÝ THUYẾT THÔNG TIN

DƯƠNG HIỂN THUẬN

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Error correcting framework

A code C over an alphabet S is a subset of S* - (C  S*).

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

METHOD OF FIGHTING ERRORS: REDUNDANCY!!!

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

• The key idea is that in order to protect a message against a

• In such a case, even if the message is corrupted by a noise,

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

• Channel coding can be divided into two

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

- Mã tích chập (Convolutional codes) –

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

2. The Hamming distance d(10101, 11110) is 3 because

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

- Từ sai (error codework)

An important parameter of codes C is their minimal distance

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Theorem Basic error correcting theorem

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

The dmin in this case is 2.

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

The dmin in this case is 3.

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Notation: An (n,k,d) - code C is a code such that

Comment: A good (n,k,d) code

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Code words 1000011 0100101 0010110 …

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

e2n−k −1 c1 + e2n−k −1 ci + e2n−k −1 c2k −1 + e2n−k −1

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

When a code is received, the syndrome is computed and

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Burst errors does not necessarily mean that

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

If same data is sent at rate = 1Mbps then a noise of

Error detection means to decide whether the

Error detection uses the concept of redundancy,

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông

Trường Đại Học Sài Gòn Khoa Điện Tử Viễn Thông