EE5143 Problem Set 2

Solutions for graded questions

Instructor: Dr. Andrew Thangaraj
Jan - May 2023 Semester

1. (a) It is easy to see that repeats like ’000 . . .’ end with 1 and ’111 . . .’ end with 0. There-
fore, the repeat length Rk , k ∈ {1, 2, 3, 4, 5} is a geometric random variable with
parameter 0.5. The entropy is calculated as follows:
∞
X
H(R1 ) = − (0.5)k log2 ((0.5)k )
k=1
0.5 × log2 (0.5)
= − log2 (0.5) −
0.5
= 2 bits

(b) The process (R1 , R2 , . . . R5 ) is memoryless. This means that the repeat lengths Rk
are independent and identically distributed. The first symbol X1 is also independent
of the repeat lengths. Therefore, the joint entropy H(X1 , R1 , R2 , . . . R5 ) is calculated
as follows:

H(X1 , R1 , R2 , . . . R5 ) = H(X1 ) + H(R1 , R2 , . . . R5 )

= −2 × 0.5 × log2 (0.5) + 5 × H(R1 )
= 11 bits

(c) An optimal prefix code for R1 is,

R1 Codeword
1 1
2 01
3 001
.. ..
. .
P∞ 1
The expected code length is i=1 2i i =2
8 6 4 2 2 1
2. (a) We are given that (s1 , s2 , . . . s6 ) = ( 23 , , , , , ) and pi = αsi . From the
P6 P6 23 23 23 23 23
axioms of probability, i=1 pi = α i=1 si = 1. From this, we have that α = 1.
(b) If the search party members asked one question at a time, the corresponding number
of questions are {1, 2, 3, 4, 5, 5} in some order. Let qi be the number of questions
P6 asked
corresponding to room i. To minimize the expected number of questions i=1 pi qi ,
the order of asking questions should be based on the chances of finding Ella in the
largest room to the smallest room, i.e, if pi > pj , then qi ≤ qj . The expected number
of questions required is,
6
X 8 6 4 2 2 1
pi qi = 1 × +2× +3× +4× +5× +5×
23 23 23 23 23 23
i=1
= 2.39 bits

However, the minimum possible number of questions that need to be asked can
be obtained by using the idea of Huffman coding. With Huffman coding, we get

1
 8 8 8 9 14 23
(00) 23 23 23 23 23 23
6 6 6 8 9
(01) 23 23 23 23 23
4 4 5 8
(11) 23 23 23 23
2 3 4
(101) 23 23 23
2 2
(1000) 23 23
1
(1001) 23

codeword lengths as (2, 2, 2, 3, 4, 4). Thus, the expected number of questions to be

asked is,
6
X 8 6 4 2 2 1
pi qi = 2 × +2× +2× +3× +4× +4×
23 23 23 23 23 23
i=1
= 2.35 bits

3. (a) We know that the optimal code length is upper bounded by H(X) + 1.

H(X) ≤ L∗ < H(X) + 1.

Given that Aryan required 9 questions (optimal) to identify the tool Arjun has picked.
Hence, we have
9 < H(X) + 1 < log |χ| + 1.
Solving, we get the rough lower bound on number of tools in the tool-kit to be
28 = 256.
(b) Similar to problem-5.16 EIT.