Problem Sheet

1. How much information is gained when a die is thrown and (i) a four occurs (ii) a four or a
six occurs (iii) any number occurs? [2.58 bits, 1.58 bits, 0 bits]

2. A newspaper picture consists of 50 rows and 40 columns of black and white dots. Calculate
the information in the picture if black and white dots are (a) equally probable (b) the
probability of white dots is three times that of black dots. [2000 bits, 1622 bits]

3. The Chinese alphabet consists of more than 10,000 characters, although only 6000 are in
general use, so how much information is gained when a single Chinese character appears?
[12.55 bits]

4. How much information is gained by the appearance of a single decimal number in English
text? [3.32 bits]
 Problem Sheet 2

1. A message source produces two independent symbols A and B with probabilities p(A)=1/5
and p(B)=4/5. Calculate the entropy of the source. [0.722 bits/symbol]. What would the
probabilities have to be to achieve the maximum entropy?
[p(A)=p(B)]

2. A transmitter generates two symbols A and B with probabilities and respectively. Find
the average information received every second if (a) A and B take 1s each to transmit, (b) A
takes 1s and B takes 2s.
[0.81 bits, 0.47 bits]

3. A transmitter produces four source symbols (A,B,C,D) with corresponding probabilities 0.5,
0.2, 0.2 and 0.1 at a rate of 1024 symbols/s. Find the entropy, information rate and
redundancy if the source produces 1024 symbols/s and the symbols are statistically
independent.
[1.761 bits/symbol, 1803 bits/s, 12%]
 Problem Sheets 3

1. A deadly disease affects 1 in every 1000 people. A test is available which is positive 95% of
the time and negative 5% of the time if you have the disease. If you do not have the disease
the test is positive only 5% of the time. Suppose you test positive for the disease, what is the
likelihood that you actually have it?
[0.019]

2. A source generates a stream of binary digits as shown below:

1 1 0 0 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 1

Assuming the 21st binary digit to be a 1, find

(i) the source probabilities p(1) and p(0)
(ii) the joint probabilities p(1,1), p(0,0), p(1,0) and p(0,1)
(iii) the joint probabilities p(10), p(11), p(01) and p(00). Hence show that Bayess rule holds.
(iv) Finally calculate the conditional entropy (equation (1.5)) for this sequence and the
redundancy of the language.

3. An information source produces three symbols A, B and C with probabilities , and

respectively. Errors occur in transmission as shown in the table below. Find the probabilities of A, B
and C at the receiver and deduce the information transfer.

[You can use the expressions for the various probabilities or if you prefer, write out a table like that
shown in the example given in section 3.2 which reflects the transmission/reception (x,y) of A, B and
C].

ARX BRX CRX

ATX 2/3 1/3 0
BTX 0 1 0
CTX 0 1/3 2/3
 Problem sheet 4

1. A photodiode detector with a bandwidth of 6MHz is attached to a 50 load resistor and

operates at 35C.

(a) What is the rms thermal noise current?

[45.2nA]

(b) Calculate the rms thermal noise voltage that appears across the resistor and the thermal
noise generated.
[2.26V, 0.1pW]

(c) If, during operation, the average photocurrent is 1 nA and the dark current is zero what is
the rms shot-noise current?
[43.8pA]

(d) Calculate the rms signal voltage and the rms shot-noise voltage across the resistor.
[50nV, 2.19nV]

(e) Finally, calculate the signal-to-noise ratio with and without thermal noise.
[520 or 27dB, 510-4 or -33 dB]

2. A low power optical link is known to be thermal-noise-limited. The bit rate is 1 Mb/s
using NRZ coding and the system wavelength is 0.82 m. The receiver consists of a high quality
PD with unity quantum efficiency and a 100 load resistor. Calculate the optical power incident
on the PD, the photocurrent generated by the PD and the incident number of photons per bit if the
system requires a minimum error rate of 10-4. [You may assume that the PD bandwidth is just
sufficient for the operating speed, i.e. 106 Hz.]
[146nW, 96.4 nA, 6105 photons/bit]
 Problem Sheet 5
1. A scanner converts a black and white document, line by line, into binary data for
transmission. The scanner produces source data consisting of symbols which represent
runs of pixels of different lengths as indicated in the table below.
No. of pixels 1 2 3 4 5 6
Probability 0.2 0.4 0.15 0.1 0.06 0.09
Determine the entropy, the average length of a run (in pixels) and the corresponding
effective information rate when the scanner traverses at 1000 pixels/s.
[2.29 bits/symbol, 2.69 bits/symbol, 852 bits/s]

2. Find a compact instantaneous code for an alphabet consisting of five symbols S 1-S5
with probabilities 1/2, 1/4, 1/8, 1/16 and 1/5. Show that the average code length is equal
to the source entropy.

3. An information source contains 100 different, statistically independent and

equiprobable symbols. Find the maximum code efficiency if the transmitted symbols
are binary codewords of equal length. [95%]

1
 Problem Sheet 6

1. The painting Bathers at Asnires in the National Gallery by Seurat measures nearly 200300
cm and has a typical dot density of 25 dots per cm 2. If each dot is assigned 3 bytes (one for red,
blue and green) to identify its colour how much information is in the picture? Comment on
how this compares with the 8 megapixel camera on the iPhone 6.
http://www.nationalgallery.org.uk/paintings/georges-seurat-bathers-at-asnieres

2. Design a Huffman code for the expression BE_A_BEE (assume that this short phrase
represents the entire alphabet and also the probabilities of the letters) and compare the required
number of digits with the arithmetic code and ASCII.

3. Use arithmetic coding to compress the word LOSSLESS

 Problem Sheet 7

1. Find the Fourier series (sine and cosine form) of a periodic function (with period T =2 )
defined as f (x)=x 2 between < x .

2 n
(1)
[x = + 4 2 cos(nx)]
3 n=1 n

2. Find the Fourier series (sine and cosine form) of a periodic function (with period T =2 )
defined as f (x)=x between < x .
[see next exercise]

3. Show that the Fourier series of exercise 2 can be obtained differentiating term by term the
Fourier series of exercise 1.
 Problem Sheet 8

When we modulate the amplitude of a signal (see section 8.2), we define the modulation index m as:

As
m= .
Ac

1. How does the average power in an AM wave depend on the modulation index m?

2. How does the average power in an FM wave depend on the modulation index m?
 Problem Sheet 9

1. A binary symmetric channel has a binary error probability of 1/3. Two equiprobable
input symbols A and B are to be transmitted, coded as 11 and 00 respectively.
Determine:
(a) The channel capacity,
(b) The information transfer through the channel, assuming that the receiver interprets
11 as A, 00 as B and makes no decision for 01 and 10
(c) The information transfer if the receiver interprets 11, 01 or 10 as A and only 00 as
B
(d) The information transfer if the receiver interprets 01 and 10 randomly as A or B
The probabilities of A being received as 11, 01, 10 or 00 are 4/9, 2/9, 2/9 and 1/9. You
may find it easier to write out a sequence of As and Bs for parts (b) and (c).

2. The diagram below shows an (n,k) block codeword consisting of four information
digits and three parity check digits. The latter are given by the following equations:
P1=1I10I21I31I4
P2=1I11I20I31I4
P3=1I11I21I30I4

I1 I2 I3 I4 P1 P2 P3
Draw a Venn diagram that represents the relationship between the parity and check bits.
Calculate the output of the encoder output if the input is 1011.

1
 Problem Sheet 10

1. Using the same letters from section 10.5 and the values of D and e from table 10.1
(a) decipher 5729
(b) cipher the word RODENT.

2. Show that 2664 mod 115 = 41

3. A laser emits 800nm photons at a repetition rate of 4MHz. The laser is attenuated so that
the average power is 0.1pW. Calculate:
(i) The average number of photons per pulse
[4105 s-1]
(ii) The percentage of pulses that contain no photons, a single photon and more
than one photon
[90.5%, 9.05%, 0.47%]
Some extra solution.

1. How much information is gained when a die is thrown and (i) a four occurs I=log2(6)=2.58
bits, (ii) a four or a six occurs I=-log2 (1/6+1/6)=1.58 bits, (iii) a one, two, three, four, five
or six occurs? I=-log2(61/6) = 0 bits
2. A newspaper picture consists of 50 rows and 40 columns of black and white dots. Calculate
the information in the picture if black and white dots are (a) equally probable (b) the
probability of white dots is three times that of black dots. (a) I = -2000log2(1/2) = 2500 bits
(b) information gained by presence of a white dot is -log 2(3/4) = 0.415 bits and a black dot
is -log21/4 = 2 bits so 150 white + 500 black gives 1622 bits in total.
3. The Chinese alphabet consists of more than 10,000 characters, although only 6000 are in
general use, so how much information is gained when a single Chinese character appears? I
= -log2(1/6000) =12.55 bits assuming the characters are equally likely.
4. How much information is gained by the appearance of a single decimal number? I=-
log2(1/10) = 3.32 bits
5. A message source produces two independent symbols A and B with probabilities p(A)=1/5
and p(B)=4/5. Calculate the entropy of the source. [0.881 bits/symbol]. What would the
probabilities have to be to achieve the maximum entropy? [-0.2log0.2-
0.8log0.8=0.464+0.258=0.722 bits/symbol, p(A)=p(B)]
6. A transmitter generates two symbols A and B with probabilities and respectively. Find
the average information received every second if (a) A and B take 1s each to transmit H=-
1/4log2(1/4)-3/4log2(3/4) = 0.81 bits (b) A takes 1s and B takes 2s. In a typical sample of
four symbols there will be one A and three Bs which providing 2 bits and 30.42 bits
respectively giving a total of 3.26 bits in 7s information rate = 0.47 bits/s
7. A transmitter produces four source symbols (A,B,C,D) with corresponding probabilities 0.5,
0.2, 0.2 and 0.1 at a rate of 1024 symbols/s. Find the entropy, information rate and
redundancy if the source produces 1024 symbols/s and the symbols are statistically
independent.
H=-[0.5log20.5+20.2log20.2+0.1log20.1]=1.761 bits/symbol, information rate
=1.7611024= 1803 bits/s. Maximum entropy (equal probabilities) is H max=2 bits/symbol
giving a redundancy of R=1-H/Hmax ~ 12%.
8. A deadly disease affects 1 in every 1000 people. A test to determine if you have the disease
is positive 95% of the time and negative 5% of the time. If you do not have the disease the
test is positive 5% of the time. Suppose you test positive for the disease, what is the
likelihood that you actually have it? [0.019]

Use the following: D = data: the test is positive, H = hypothesis: you have the disease
H= the other hypothesis: you do not have the disease.
We know only that the a priori probability of having the disease is .001, which sets p(H).
We also need to know p(D). From the Sum Rule, we can calculate that the a priori
probability p(D) of testing positive is:
p(D) = p(DH)p(H) + p(D )p( ) =(.95)(.001)+(.05)(.999) = .051
and from Bayes' Rule, we can conclude that the probability that you actually have the
disease given that you tested positive for it, is much smaller than you may have thought:
p(HD) = p(DH)p(H)/p(D) =(0.95)(0.001)/(0.051) = 0.019 (less than 2%).
This quantity is the a posteriori probability because it is computed after the observation of
data; it tells us how likely the hypothesis is, given what we have observed.
(Note: it is an extremely common human fallacy to confound p(HD) with p(DH): In the
example given, most people would react to the positive test result by concluding that the
likelihood that they have the disease is 0.95, since that is the hit rate of the test. They
confound p(DH) = 0.95 with p(HD) = 0.019, which is what actually matters.)

9. A source generates a stream of binary digits as shown below:

1 1 0 0 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 1

Assuming the 21st binary digit to be a 1, find

the source probabilities p(1) and p(0) [p(1)=12/20, p(0)=8/20]
the joint probabilities p(1,1), p(0,0), p(1,0) and p(0,1) [p(1,1)=9/20, p(0,0)=5/20,
p(1,0)=3/20, p(0,1)=3/20]
the joint probabilities p(10), p(11), p(01) and p(00). Hence show that Bayess rule holds.

p(1) p(0|1) 12/203/12 3

p(1|0)= = =
[p(10)=3/8, p(11)=9/12, p(01)=3/12 and p(00)=5/8 p(0 ) 8/20 8 ]
10. Finally calculate the conditional entropy (equation (1.5)) for this sequence and the
redundancy of the language.
 [ Entropy=[ p(1,1)log2 p(1|1)+ p(0,0)log2 p(0|0)+ p(1,0)log 2 p(0|1)+ p(0,1)log 2 p(1|0)]
0.868 bits/symbol
If there is no conditional probability:
Entropy=[ p (1 )log 2 p (1)+ p(0 )log 2 p (0 )=0 . 971 bits/symbol
And redundancy R~10.6% using equation (2.4). In principle the maximum entropy for a
binary system is 1bit/symbol which would increase R to ~13%. The distinction between the
two is small.

11. An information source produces three symbols A, B and C with probabilities , and
respectively. Errors occur in transmission as shown in the table below. Find the probabilities
of A, B and C at the receiver and deduce the information transfer. [You can use the
expressions for the various probabilities or if you prefer, write out a table like that shown in
the example given in section 15.2 which reflects the transmission/reception (X,Y) of A, B
and C]. [p(A)=1/6, p(B)=1/2, p(C)=1/3, I (X,Y)= 0.53 bits].
ARX BRX CRX
ATX 2/3 1/3 0
BTX 0 1 0
CTX 0 1/3 2/3

This question was taken from Usher & Guy and their method is reproduced here:
The probability of symbol A at the receiver is given by the sum of all the conditional
probabilities p(A) =p(A,A) + p(B,A) + p(C,A). Since the table gives the conditional
probabilities we re-write this equation as: p(A) =p(A)p(A|A) + p(B)p(A|B) + p(C)p(A|
C)=1/42/3 +0 +0 = 1/6. Similar expressions leads to p(B)=1/2 and p(C)=1/3.
To find I(X,Y) we must use equation 15.2 for all possible combinations but first we must
calculate the joint probabilities p(A,A) = p(A)p(A|A), p(A,B), p(A,C) etc. to give

1 1 1
1 61 1 12 1 1/6 4 1 1/3
I ( X , Y )= log + log + log + log + l og
6 1 1 12 1 1 4 1 1 6 1/2 1/2 3 1 /2 1/3

4 6 4 2 4 2
0.53 bits
 It is also possible to answer this question by writing out a table of transmitted symbols (X)
which reflects the source probabilities given in the question and the received symbols (Y)
which reflects the errors according to the table given in the question.
X A A A B B B C C C C C C
Y A A B B B B B B C C C C
Now compile a table showing the probabilities of each combination of symbols using the
table just compiled:
X A A A B B B C C C
Y A B C A B C A B C
P(X) 1/4 1/4 1/4 1/4 1/4 1/4 1/2 1/2 1/2
P(Y) 2/12 6/12 4/12 2/12 6/12 4/12 2/12 6/12 4/12
P(X|Y) 1 1/6 0 0 3/6 0 0 2/6 1
P(Y|X) 2/3 1/3 0 0 1 0 0 1/3 2/3
P(X,Y) 2/12 1/12 0 0 3/12 0 0 2/12 4/12

Now you have all the necessary conditional and joint probabilities to complete the question.

12. (a) i2rms =4 k B 308 6 106 50=2.04 1015irms=45.2nA (b)

v rms =i 50=2.26 V and thermal noise

v 2rms =0.1 pW (c)
50

i2sh =2 ie f =1.92 1021ish=43.8 pA (d) 9

v s=i s R=1 10 50=50 nV ,

S 109
v rms , shot =i sh 50=2.19 nV (e) Only shot noise = =520 (or
N 2 1.6 1019 6 10 6
10log520 = 27dB). Including thermal noise

9 2
S ( 1 10 ) 50
= =5 104 ( 33 dB)
N 2 e R L f 10 +4 k 300 6 10
9 6

13. BER of 10-4 means a S/N (reading from figure 4.7) of 17.5 dB or 56.2. Hence

eP 2
56.2=
RL
h ( ) gives P=146nW. The current is given by i=
eP
h
which gives 96.4
4 kT f
nA justifying the claim that the system is thermal noise limited. The number of photons
incident in the bit interval is P/h which gives 610 5 photons/bit showing that a large
number of photons are required to achieve a BER of 10-4 in a thermal noise limited system.
14. A scanner converts a black and white document, line by line, into binary data for
transmission. The scanner produces source data consisting of symbols which represent runs
of pixels of different lengths as indicated in the table below.
No. of pixels 1 2 3 4 5 6
Probability 0.2 0.4 0.15 0.1 0.06 0.09
Determine the entropy, the average length of a run (in pixels) and the corresponding
effective information rate when the scanner traverses at 1000 pixels/s.
[2.29 bits/symbol, 2.69 pixels, 852 bits/s]
H= -(0.2log0.2+0.4log0.4+)=0.4644+0.5287+0.4105+0.0.3322+0.2435+0.3126) = 2.29
bits/symbol. To find the average run-length multiply no. of pixels probability of a run
and add to obtain 2.69 pixels. H2.691000 = 852 bits/s
15. Find a compact instantaneous code for an alphabet consisting of five symbols S 1-S5 with
probabilities 1/2, 1/4, 1/8, 1/16 and 1/16. Show that the average code length is equal to the
source entropy.
Construct a Huffman tree following the instructions given in the lecture. You should find
something like: S1 1, S2 01, S3 001, S4 0001, S5 0000 (it doesnt matter if your tree inverts 1s
and 0s). Then average code length is L = 10.5+20.25+30.125+240.0625 = 1.875
binary digits/symbol. H = 0.5log 0.5+20.0625log 0.0625=1.875 bits/symbol.
16. An information source contains 100 different, statistically independent and equiprobable
symbols. Find the maximum code efficiency if the transmitted symbols are binary
codewords of equal length. [95%]
Symbols are equiprobable so H=1000.01log0.01 = 6.644 bits/symbol. To accommodate
100 symbols need L=7 binary digits so code efficiency = H/L = 95%

17. The painting Bathers at Asnires in the National Gallery by Seurat measures nearly
200300 cm and has a typical dot density of 25 dots per cm 2. If each dot is assigned 3 bytes
(one for red, blue and green) to identify its colour how much information is in the picture?
Comment on how this compares with the 8 megapixel camera on the iPhone 6.
Painting covers 60000 cm2 and will contain 1.5 Mpots (pots = painted dots I made this
up). Both pots (painting) and pixels (i-Phone) will require 3 bytes to describe the colour so
an iPhone picture contains 5.33 as much information as Seurats picture. What if you took
 a photograph using the camera of the picture where would the extra information have
come from?

18. Design a Huffman code for the expression BE_A_BEE (assume that this short phrase
represents the entire alphabet and also the probabilities of the letters) and compare the
required number of digits with the arithmetic code and ASCII.
E: 3/8, B: 2/8, _: 2/8, A: 1/8. A Huffman code might be E: 1, B: 01, _: 001 and A: 000. The
phrase will then require 16 binary digits. ASCII would require 78=56 b.d.s. Arithmetic
coding just wins with 15 b.d.s.

19. Use arithmetic coding to compress the word LOSSLESS

The frequency of the letters is E:1, L:2, O:1 and S:4. Following the scheme shown in 6.2.3
subdivide the [0,1] range according to the frequency i.e. 1/8 for E, 2/8 for L etc. Starting
with L subdivide its range (2/8) into 64ths, continue to the next letter O etc. to obtain
eventually [4079616/16777216, 4083712/16777216]

20. A binary symmetric channel has a binary error probability of 1/3. Two equiprobable input
symbols A and B are to be transmitted, coded as 11 and 00 respectively. Determine:
The channel capacity,
Channel capacity C=1-H(p) where H(p)=p log p + plog p = 1/3 log 1/3 + 2/3 log 2/3 =
0.918 bits so C = 0.082 bits.
The information transfer through the channel, assuming that the receiver interprets 11 as A,
00 as B and makes no decision for 01 and 10

Writing out a sequence for the transmitter (X) and receiver (Y) which reflects the
probabilities of A being received as 11, 01, 10 or 00 are 4/9, 2/9, 2/9 and 1/9
X AAAAAAAAA B B B B B B B B B
Y AAAA B N N N N B B B B A N N N N

4 4 / 18 1 1/18 4 4/ 18
I ( X , Y )=2
log + 2 log +2. log =0.154 bits
18 1/2 5/18 18 1/2 5/ 18 18 1/2 8/18
Where AN and BN convey no information.

The information transfer if the receiver interprets 11, 01 or 10 as A and only 00 as B

X AAAAAAAAA B B B B B B B B B
 Y AAAA B N N N N B B B B A N N N N
Obs AAAA B AAAA B B B B AAAAA
8 8 /18 4 4/18 1 1/ 18 5 5/18
I ( X , Y )= log + log + log + log =0.106 bits
18 1/2 13/18 18 1/2 5/18 18 1/2 5 /18 18 1 /2 13 /18
The information transfer if the receiver interprets 01 and 10 randomly as A or B
The probabilities of A being received as 11, 01, 10 or 00 are 4/9, 2/9, 2/9 and 1/9.
X AAAAAAAAA B B B B B B B B B
Y AAAA B N N N N B B B B A N N N N
Obs AAAA B AA B B B B B B AAA B B
6 6 /18 3 3/18
I ( X , Y )=2 log +2 log =0.082bits
18 1/2 1/2 18 1/2 1/2

21. The diagram below shows an (n,k) block codeword consisting of four information digits and
three parity check digits. The latter are given by the following equations:
P1=1I10I21I31I4
P2=1I11I20I31I4
P3=1I11I21I30I4

I1 I2 I3 I4 P1 P2 P3
Draw a Venn diagram that represents the relationship between the parity and check bits.
Calculate the output of the encoder output if the input is 1011.

P1 does not include I2, P2 does not include I3 and P3 does not
P1 I4 include I4 (each of these is multiplied by 0). I 1 is included in
P2 the XOR operation for all three parity bits. This is indicated
I1 in the Venn diagram (left).
I3 I2 For the data sequence 1011 we have I1=I3=I4=1 while I2=0.
Then P1=1I10I21I31I4 = 1011 = 1 (odd
P3
number of 1s, or think of the clock analogy). Similarly
P2=1I11I20I31I4= 1001 = 0 and finally P3=0
so this (7,4) Hamming code is 1011100.

22. Using the same letters from section 10.5 and the values of D and e from table 10.1 (a)
decipher 5729 (b) cipher the word RODENT
(a) Raise each to the power of 7: 78125 823543 128 4782969, modulo 10 gives 5389
which is NEST (b) Using the code for the letters we obtain 762359. Raise these to the power
 e to get 343 216 8 27 125 729. Modulo 10 gives 3 6 8 7 5 9 which is the code that
would be sent. [check by deciphering following method in (a)].

Show that 2664 mod 115 = 41 (not easy is it?)

23. A laser emits 800 nm photons at a repetition rate of 4MHz with equal on-off periods. The
laser is attenuated so that the average power is 0.1pW. Calculate:
The average number of photons per pulse [4105 s-1]
Each photon has an energy of E=hc/ = 6.6210-34310880010-9= 2.4810-19 Joules so
the laser emits photons at a rate 0.110-122.4810-19=4105 s-1. At a repetition rate of 4MHz
(assuming equal periods of laser on/laser off) each pulse will contain 4105/4106 or 0.1
photons.
The percentage of the pulses that contain no photons, a single photon and more than one
photon [90.5%, 9.05%, 0.47%]
Since lasers are coherent light sources the photons obey Poisson statistics where the

n
n
probability of finding n photons is given by P ( n )= en so for an average of n=0.1
n!
photons per pulse the probability of obtaining no photons (n!=1) is 90.5%.