Error Control Coding

Lecture Notes

Prof. Dr.-Ing. Volker Kühn

University of Rostock
Institute of Communications Engineering
Error Control Coding
Table of Contents

Basics of Digital Communications – A Summary

Two Lessons of Information Theory

Forward Error Correcting Codes – The Classics

Modern Error Correcting Codes

Concatenated Codes
Turbo Decoding Principle
EXIT Charts
LDPC codes

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 243 / 385
Low-Density Parity Check (LDPC) Codes
• Gallager invented LDPC codes in 1963
• LDPC codes are simple linear block codes with certain properties
• Specific iterative soft-in decoding makes them very powerful
• Comprehensive overviews can be found in
• R.M. Tanner: „A Recursive Approach to Low Complexity Codes“, IEEE Transactions on
Information Theory, vol. 27, pp. 533-547, September 1981

• S.Y. Chung, G.D. Forney, T.J. Richardson, R. Urbanke: „On the Design of Low Density
Parity Check Codes within 0.0045 dB of the Shannon Limit“, IEEE Communication
Letters,vol. 5, pp. 58-60, February 2001

• D.J.C. MacKay: „Good Error Correcting Codes Based on Very Sparse Matrices“, IEEE
Transactions on Information Theory, vol. 45, pp. 399-432, March 1999

• S. Lin, D. Costello: Error Control Coding, Pearson Prentice-Hall, 2004

• T. Richardson, A. Shokrollahi, R. Urbanke: „Design of Capacity-Approaching Irregular

Codes“, IEEE Transactions on Information Theory, vol. 47, pp. 619-637, February 2001

• G.D. Forney: „Codes on Graphs: Normal Realizations“, IEEE Transactions on

Information Theory, vol. 47, pp. 520-548, 2001
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 244 / 385
Regular LDPC Codes
• LDPC codes defined by their N × n parity check matrix H

• c · HT = 0 holds for all code words → code Γ is null space of H

• The number of rows N may differ from n − k representing number

of independent rows in H
Definition (regular LDPC code: Lin, Costello 2004)
An LDPC code is defined as the null space of a parity check matrix H
with properties:
1. each row consists of Nr ones
2. each column consists of Nc ones
3. number of 1’s in common between any two columns not larger than 1
4. Nc and Nr are small compared to N and n, i.e. H is sparse,
number of ones is small compared to size of H (low density)
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 245 / 385
Example: Cyclic (15,7) LDPC Code, Nr = Nc = 4
• rank(H) = 8 → null space of H has dimension k = 15 − 8 = 7
4
• Density of code: ρ = Nnr = NJc = 15 = 0.267
 0 0 0 0 0 0 0 1 1 0 1 0 0 0 1

 1 0 0 0 0 0 0 0 1 1 0 1 0 0 0
 0 1 0 0 0 0 0 0 0 1 1 0 1 0 0


 0 0 1 0 0 0 0 0 0 0 1 1 0 1 0 
 0 0 0 1 0 0 0 0 0 0 0 1 1 0 1
 

 1 0 0 0 1 0 0 0 0 0 0 0 1 1 0 
 0 1 0 0 0 1 0 0 0 0 0 0 0 1 1
 

H= 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1
 
 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0


 0 1 1 0 1 0 0 0 1 0 0 0 0 0 0
 

 0 0 1 1 0 1 0 0 0 1 0 0 0 0 0 
 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0
 

 0 0 0 0 1 1 0 1 0 0 0 1 0 0 0

0 0 0 0 0 1 1 0 1 0 0 0 1 0 0
 
0 0 0 0 0 0 1 1 0 1 0 0 0 1 0

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 246 / 385
Construction of LDPC Codes
• Various approaches for constructing powerful LDPC codes exist

• Basically, parity check matrix H has to be designed

• Random code construction
1. Randomly select new column with Nc 1’s

2. If new column has more than single 1 in common with old columns or
violates row norm Nr , it is rejected

3. Otherwise keep new column and continue with step 1. until n

columns are obtained

• Algebraic construction by finite geometry

• Euclidean Geometry and Projective Geometry LDPC codes

• Techniques are quite sophisticated and beyond scope of this lecture

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 247 / 385
Construction of Regular LDPC Codes
Construction of kNr × kNc matrix H by Gallager

• Parity check matrix comprises k × kNr sub-matrices Hj

iT
H = HT HT · · · HT
h
1 2 J

• Each sub-matrix Hj has Nr 1’s per row, but a single 1 in each

column only

• ith row of sub-matrix H1 has its 1’s in columns (i − 1)Nr + 1 to iNr

• Further sub-matrices Hj>1 consists of column permutations of H1

• No constructive way known to determine suited permutations

(computer searches needed)

• For very long codes, powerful permutations have been found leading
to performance very close to Shannon limits
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 248 / 385
Construction of Regular LDPC Codes
Example for code construction by Gallager with n = 20, Nr = 4, J = 3
• Each row has Nr = 4 ones, each column only a single 1
• J = 3 sub-matrices each of size k × kNr = 5 × 20
• First 5 rows represent H1 with consecutive 1’s in each row
(non-overlapping)
• Sub-matrices H2 and H3 obtained by permuting columns of H1
 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 
 − − − − − − − − − − − − − − − − − − − − 
 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 
 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 
0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0
0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0
 
0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1
 
 − − − − − − − − − − − − − − − − − − − − 
 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 
 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 
0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0
0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0
0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 249 / 385
LDPC Encoding
• Encoding requires generator matrix G

• G can be obtained from H by Gaussian elimination

• Example:
1 1 1 1 0 0 1 1 1 1 0 0
   

H = 0 0 1 1 0 1 → 0 1 1 0 1 0
e =
H
  
1 0 0 1 1 0 1 1 0 0 0 1
1 1 0 0 1 0 1 0 0 1 0 1
   

G = 0 1 0 1 1 1 ← G = 0 1 0 1 1 1
  e  
0 1 1 0 0 1 0 0 1 1 1 0

• Resulting G is generally not sparse → high encoding complexity

• Low complexity encoding by transforming H into partly triangular H̃

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 250 / 385
Turbo Decoding of LDPC Codes
Message passing based on graphical representation of code

• Hard decision decoding algorithms are very efficient but have quite
poor performance
• Majority logic decoding

• Bit-flipping decoding algorithm

• Soft decision decoding algorithms are more complex with very high
performance
• MAP decoding (generally extremely complex)

• Iterative message passing based on belief propagation (sum product

algorithm) exhibits high performance and moderate complexity

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 251 / 385
Example and Representation by Bipartite Factor Graph
1 1 1 1 0 0 n = 6 columns (code word length)
 

H= 0 0 1 1 0 1  n − k = 3 parity check equations

 
1 0 0 1 1 0 see chapter 3

y0 y1 y2 y3 y4 y5

variable nodes

check nodes

s1 s2 s3
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 252 / 385
Message Passing Algorithm
Operation at check nodes

• Each check node represents one row of parity check matrix H

1 1 1 1 0 0
 

H= 0 0 1 1 0 1 
 
1 0 0 1 1 0

• First check equation: c0 ⊕ c1 ⊕ c2 ⊕ c3 = 0 ⇒ c0 = c1 ⊕ c2 ⊕ c3

• Extrinsic information for bit c0 from check node s1

1+tanh(Lch y1 /2)·tanh(Lch y2 /2)·tanh(Lch y3 /2)
Le (1) (c0 ) = log 1−tanh(Lch y1 /2)·tanh(Lch y2 /2)·tanh(Lch y3 /2)

= 2 artanh tanh(Lch y21 ) tanh(Lch y22 ) tanh(Lch y23 )

 

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 253 / 385
Example: Operation at Check Node s1
y0 y1 y2 y3 y4 y5

variable nodes

check nodes

s1 s2 s3

(1)
Le (c0 ) = 2 · artanh[tanh(Lch y1 /2) · tanh(Lch y2 /2) · tanh(Lch y3 /2)]
(1)
Le (c1 ) = 2 · artanh[tanh(Lch y0 /2) · tanh(Lch y2 /2) · tanh(Lch y3 /2)]
(1)
Le (c2 ) = 2 · artanh[tanh(Lch y0 /2) · tanh(Lch y1 /2) · tanh(Lch y3 /2)]
(1)
Le (c3 ) = 2 · artanh[tanh(Lch y0 /2) · tanh(Lch y1 /2) · tanh(Lch y2 /2)]
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 254 / 385
Example: Operation at Check Node s2
y0 y1 y2 y3 y4 y5

variable nodes

check nodes

s1 s2 s3

Le (2) (c2 ) = 2 · artanh[tanh(Lch y3 /2) · tanh(Lch y5 /2)]

Le (2) (c3 ) = 2 · artanh[tanh(Lch y2 /2) · tanh(Lch y5 /2)]
Le (2) (c5 ) = 2 · artanh[tanh(Lch y2 /2) · tanh(Lch y3 /2)]

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 255 / 385
Example: Operation at Check Node s3
y0 y1 y2 y3 y4 y5

variable nodes

check nodes

s1 s2 s3

(3)
Le (c0 ) = 2 · artanh[tanh(Lch y3 /2) · tanh(Lch y4 /2)]
(3)
Le (c3 ) = 2 · artanh[tanh(Lch y0 /2) · tanh(Lch y4 /2)]
(3)
Le (c4 ) = 2 · artanh[tanh(Lch y0 /2) · tanh(Lch y3 /2)]

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 256 / 385
Passing Information back to Variable Nodes
• Extrinsic LLRs are passed from check nodes back to variable nodes
• Each variable node receives number of messages according to its
degree
• Messages contribute to new estimate of bit ci
L(ci ) = Lch · yi + L(1) (2) (3)
e (ci ) + Le (ci ) + Le (ci )
y0 y1 y2 y3 y4 y5

variable nodes

(3)
(1)
Le (c0 ) Le (c0 )

check nodes

s1 s2 s3
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 257 / 385
Passing Improved Information to Check Nodes
• Improved estimates at variable nodes are passed again to check
nodes which calculate extrinsic LLRs
• Check nodes must not receive their own extrinsic LLRs
• Check node receive sum of extrinsic LLRs excluding their own LLRs
• Example for check node 3
y0 y1 y2 y3 y4 y5

variable nodes

Lch y4
(1)
Lch y0 + Le (c0 ) (1) (2)
Lch y3 + Le (c3 ) + Le (c3 )

check nodes

s1 s2 s3
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 258 / 385
Message Passing Algorithm
Also known as Sum-Product or Belief Propagation Algorithm
• Iterative exchange of messages between variable and check nodes
• Similar to turbo decoding, extrinsic LLRs are exchanged for binary
LDPC codes
1. First iteration: Variable nodes pass LLRs of channel observations yi
to check nodes
2. Check nodes compute extrinsic LLR for each bit connected to it
3. Extrinsic LLRs are fed back to variable nodes
4. Each variable node computes sums of received extrinsic LLRs
(assuming their statistical independence), each sum excludes LLR of
one particular check node
5. Sums are passed back to check nodes (check node sj only gets
sums not containing its own LLRs)
V. Kühn | 6. Continue with stepError
| Modern
Error Control Coding 2. Correcting
until convergence
Codes → LDPCcriterion
codes is met 259 / 385
Convergence of Sum Product Algorithm
• After each iteration, syndrome can be checked to verify if valid code
word has been obtained

• Syndrome equal to zero stops decoding process

• Performance highly depends on cycles in graph
• Short cycles lead to high correlations among associated signals
reducing the gain per iteration significantly

• Iterative sum product algorithm might get stuck far away from ML
solution

• Code construction has to avoid short cycles (e.g. length 4) in graph

• EXIT chart analysis can be applied to analyze iterative message

passing and to design LDPC codes
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 260 / 385
BER Performance of LDPC Code with SP Algorithm
100
kk =
= 9507
9507
nn =
= 29507
29507
10−1 RRcc =
= 0.322
0.322
#1
10−2
BER →

#10
10−3
#15
#5
10−4 #20

10−5
0 1 2 3 4 5 6
10 log10 (Eb /N0 ) in dB →
V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 261 / 385
Improving Convergence of Sum Product Algorithm
Column and Row Splitting of Parity Check Matrix

• Splitting columns of a given parity check matrix into multiple columns

helps to break short cycles
• Each column of H might be split into q columns

• Ones in original columns must be appropriately distributed onto new

columns

• Extended parity check matrix Hext has qn columns

• Density of extended code is reduced from ρ = Nr

n to ρext = ρ
q = Nr
qn

• If Nc is no integer multiple of q , new columns have different weights

and irregular LDPC code is obtained

V. Kühn | Error Control Coding | Modern Error Correcting Codes → LDPC codes 262 / 385