EE5581 - Information Theory and Coding UMN, Fall 2018, Nov.

Homework 4
Instructor: Soheil Mohajer Due on: Nov. 12, 2:30 PM

Problem 1

Consider a Binary Symmetric Channel with cross probability p, given by

 
1−p p
p(y|x) =
p 1−p

(a) Compute C(p), the capacity of this channel.

Now, consider a communication channel with X = Y = {1, 2, 3, 4}, with transition probability matrix given
by
 
p 1−p 0 0
1 − p p 0 0 
p(y|x) =  
 0 0 q 1 − q
0 0 1−q q

where 0 < p < q < 1/2.

(b) Compute capacity of this channel, and write it in terms of C(p) and C(q).
Hint: Assume PX (1, 2, 3, 4) = (αr, (1 − α)r, β(1 − r), (1 − β)(1 − r)).

Next, consider a more general setting: there are two channels Ch1 : X1 → Y1 and Ch2 : X2 → Y2 , where
X1 ∩ X2 = ∅ and Y1 ∩ Y2 = ∅. Let P1 and P2 denote the transition probability matrices of channels, and the
capacity of two channels are given by C1 and C2 , respectively. A combined channel Ch : X1 ∪ X2 → Y1 ∪ Y2 is
defined by
 
P1 0
P = .
0 P2

(c) Compute the capacity of the channel P in terms of C1 and C2 .

Problem 2

Consider the communication channel with transition probabilities given in Fig. 1.

(a) Is this channel strongly symmetric? why?

1
 2/3
1 1
1/3
1/3
1/3
X 2 2 Y
1/3
1/3
3 2/3
3

Figure 1: Channel

(b) It it weekly symmetric? why?

(c) Compute the channel capacity. (simplify the result to a real number)
Hint: You may assume p(X = 1) = p(X = 3) in the optimum input probability distribution.

(d) What does the channel with its optimum distribution remind you? Make a connection between this channel
and one of the standard channels have seen in the class, and compare their capacities.

Problem 3

(a) Consider the channel with X = Y = {0, 1}, and Y = XZ, where X and Z are independent binary random
variables. Let Z ∼ Ber(α), i.e., P (Z = 1) = α. Find the capacity of this channel and the maximizing
distribution on X.

(b) Now suppose that the receiver can observe Z as well as Y . What is the capacity?

Problem 4

Consider a channel with input alphabet X = {0, 1, 2, 3} where the output is given by Y = X + Z, and Z is
independent from X and uniformly distributed Z = {a, b, c} (here + denotes usual summation over real numbers,
not over a finite field).

(a) What is H(Z)? Write the capacity expression for this channel.

(b) If you want to maximize the capacity, what alphabet should you choose for Z? Determine a, b, c.

(c) Give the maximizing distribution for p(x) to achieve the capacity.

(d) What is the minimum capacity over all choices for the Z alphabet? Give distinct integer values a, b, c and a
distribution p(x) on X to achieve this minimum.