INFORMATION THEORY AND CODING

I(X; Y ) = h(Y ) h(Y |X) = h(X) + h(Y ) h(X, Y )

Entropy and Mutual Information:

Discrete Random Variable

Chain Rule

Definitions:

h(X1 , X2 , , Xn ) =

H(X) =

h(Xi |Xi1 , , X1 )

p(x) log p(x)

P
H(X, Y ) = p(x, y) log p(x, y)
P
H(Y |X) = p(x)H(Y |X = x)

Important Distribution
Uniform: h(X) = log a, x [0, a]
Single Variable Normal: x N (0, 2 )

I(X; Y ) = H(X) H(X|Y )

h(x) =

I(X; Y |Z) = H(X|Z) H(X|Y, Z)

1
2

ln 2e 2

Multivariable Normal: (X1 , , Xn ) Nn (, K)

Basic Properties:

h(X1 , , Xn ) =

H(X) 0; I(X; Y ) 0
f (x) =

H(X) H(X|Y )

1
( 2)n |K| 2

1
2

ln(2e)n |K|
T

e 2 (x)

K1 (x)

Entropy Bound:

Basic Relations:

I(X; Y ) = H(Y ) H(Y |X) = H(X) + H(Y ) H(X, Y )

h(X1 , , Xn )

I(X; Y ) = I(Y ; X); I(X; X) = H(X)

X 3 E(X) = 0, E(XXT ) = K
maxE(XXT )=K h(X) =

Chain Rule:
H(X1 , X2 , , Xn ) =

H(Xi |Xi1 , , X1 )
P
I(X1 , X2 , , Xn ; Y ) = I(Xi ; Y |Xi1 , , X1 )

Independent Bound: H(X1 , , Xn )

AEP and Typicality:

AEP

n1 log p(X1 , , Xn ) H(X), in probability

H(Xi )

If X1 , X2 , , Xn are i.i.d. f (x), then

Important Inequality:
Markov Chain: p(x, y, z) = p(x)p(y|x)p(z|y)
Conditional Independence: p(x, z|y) = p(x|y)p(z|y)
X Y Z = X Y Z

n1 log f (X1 , , Xn ) h(X), in probability

Discrete Case
Definition:
(n)

I(X; Y ) I(X; Z), I(X; Y )

Pe = Pr{X 6= X}

X Y X;
H(X|Y )
H(Pe ) + Pe log |X | H(X|X)
H(X|Y )1
log |X |

Continuous Random Variable

Definitions
h(X) =

f (x) log f (x)dx

R
h(X1 , X2 , , Xn ) = f (xn ) log f (xn )dxn
S

A

(n)

X (n) A
(n)

Pr{A

h(AX) = h(X) + log | det(A)|

>1
(n)

Jointly Typical
Definition
A(n)
={x(n) , y (n) : |


1
log p(x(n) ) H(X)| < 
n

1
log p(y (n) ) H(y)| < 
n
1
| log p(x(n) , y (n) ) H(X, Y )| < }
n

Basic Properties:

h(X + c) = h(X); h(aX) = h(X) + log |a|

: H(X)  n1 log p(X (n) ) H(X) + 

(1 )2n(H(X)) |A | 2n(H(X)+)

I(X; Y ) = h(X) h(X|Y )

I(X; Y ) 0; h(X|Y ) h(X)

= {x(n) |2n(H(X)+) p(x(n) ) 2n(H(X)) }

Properties:

h(X|Y ) = h(X, Y ) h(Y )

Basic Relation

ln(2e)n |K|

If X1 , X2 , , Xn are i.i.d. p(x), then

Uniform Bound: H(X) log |X |

X Y Z
I(X; Y |Z)

1
2

Theorem:

Entropy Bound:

Pe

h(Xi )

where p(x(n) , y (n) ) = p(xi , yi ).

Properties:
(n)

|A | 2(H(X,Y )+)

(n)

(1 )2n(I(X;Y )+3) Pr{A } n(I(X;Y )3) 2

Continuous Case
Definition:
A(n) = {x(n) S n : |

1
log f (x(n) ) h(X)| }
n

where f (x(n) ) = f (xi )

Channel Capacity:

Memoryless Discrete Channel

Definition:
C = maxp(x) I(X; Y )
Properties:
C 0; C min(log |X |, log |Y|
Symmetric Channel:
p(x)p(y|x) = p(y)
weakly symmetric if every row of p(y|x) is permutation of every other rows of p(y|x).
C = log |Y| H(r)
Feedback does not increase capacity for discrete memoryless channel.
Gaussian channel: C = maxE(XT X)P I(X; Y )
Gaussian Channel:

Single Channel
Yi = Xi + Zi ; Zi N (0, N );
C=

1
2

log(1 +

1
n

x2i P

P
N)

Parallel Channel
The parallel Gaussian channels is defined as
Yj = Xj + Zj ;

1 j k;

Zj N (0, Nj )

The mutual information of the parallel channels are

I(X (k) ; Y (k) ) = h(Y (k) ) h(Y (k) |X (k) )
= h(Y (k) ) h(Z (k) |X (k) )
= h(Y (k) ) h(Z (k) )
X
= h(Y (k) )
h(Zi )
X

(h(Yi ) h(Zi ))
X1
Pi

log(1 +
)
2
Ni
The Lagrangian of the system is
L=

X1
2

log(1 +

X
X
Pi
)
i Pi + (
Pi P )
Ni

For optimality
i 0

i Pi = 0