Lahore University of Management Sciences

School of Science and Engineering, Department of Electrical Engineering

EE564 - Stochastic Systems
Fall 2019
Homework 4

Date Assigned: November 21st , 2019 Date Due: December 4th , 2019 (In class)

Instructions

• In order to gain the maximum benefit, try to perform solo brainstorming before any
consultations. The thought process behind searching for the correct approach all on
your own will be invaluable. Only after having performed extensive brainstorming,
you should carry out discussion with your peers.

• While discussions with your peers is strongly encouraged, please keep in mind that
the homework is to be attempted on an individual level. Plagiarism can have dire
consequences.

• No extension of the submission deadline will be made. No late submissions

allowed either.
1. When grading the final exams of EE564, Prof. Uppal finds that there are two exams
without student IDs. All student IDs are accounted for except those of Khurram and
Talha. Prof. Uppal decides to use probability theory to estimate which exam belongs
to which student (he has no other way of correct identification).
For identification, Prof. Uppal only looks at the scores on Problem 1 which was
a 10-part objective-type question. Based on prior information (e.g. from the mid-
term), Prof. Uppal knows that the probability with which Khurram answers any
given part correctly is 0.75, while the probability with which Talha answers any
given part correctly is 0.9. Prof. Uppal further assumes that each part is answered
independently of all others, and that the two student have prepared and attempted
the exam independently of each other. After grading the exams, Exam-1 turns out
to have all 10 parts of Problem 1 answered correctly, while Exam-2 has only 6 parts
answered correctly.

(a) Given the information above, what should be Prof. Uppal ’s estimate: Does
Exam-1 belong to Khurram or Talha? Justify your answer.
(b) Given the information above, and the decision in part (a), what is the probability
of Prof. Uppal ’s decision being wrong?

2. Let H0 , W1 , . . . , WM be i.i.d. standard normal random variables. In addition, for

n = 1, . . . , M , let √ √
Hn = αHn−1 + 1 − αWn ,
where 0 ≤ α ≤ 1 is a constant. Let H = [H0 , . . . , HM ]T .
h i
(a) Determine µH = E H .
h i
(b) Determine the covariance matrix C = E (H − µH ) (H − µH )T . Do this be
computing each element C(i, j) 0 ≤ i, j ≤ M of the matrix C.

3. Let X1 , X2 , . . . , Xn and Y be jointly Gaussian random variables with zero means.

Given the observations on X1 , . . . , Xn , the LMMSE estimate of Y is given by
n
X
ŶL = ai Xi
i=1

where a1 , . . . , an satisfy the orthogonality conditions given by

n
" #
X
E (Y − ai Xi )Xk = 0 k = 1, . . . , n
i=1

Prove that for this case, the LMMSE estimate is also the MMSE estimate, i.e., show
that
n
X
E[Y |X1 , . . . , Xn ] = ai Xi
i=1

Hint: A straight-forward way to prove this is by computing the conditional pdf

f (y|x1 , . . . , xn ). While this is a valid method, there is another solution that is a lot
less tedious. Let the vector a be chosen so as to satisfy the orthogonality condition.
Then, use the following as your starting point.

E [Y |X] = E Y − aT X + aT X|X
 

2
4. Consider a system that not only scales the input by an unknown factor, but also adds
Gaussian noise. In particular, the input-output relationship at time n, n = 1, . . . , N ,
is given as
Yn = Hxn + Zn ,
where Yn and xn is the input and output at time n, respectively. The unknown scaling
factor is denoted by H, whereas Z1 , Z2 , . . . , ZN are i.i.d. zero-mean Gaussian random
variables with variance σ 2 . The model described above applies to many practical
applications (e.g. flat fading wireless channels), in which it becomes necessary to
estimate the value of H. A method that is often utilized to achieve this objective
is to provide a known input (referred to as pilot inputs), and use the observations
on the system output Y1 , . . . , YN to form an estimate. In this problem, you will be
required to apply various estimation methods to this problem.
For all parts below, assume that H is a standard normal random variable, and
x1 , . . . , xN are deterministic and known.

(a) Given the observations Yn = yn , n = 1, . . . , N , determine the ML estimate of H.

(b) Given the observations Yn = yn , n = 1, . . . , N , determine the LMMSE estimate
of H.
(c) Given the observations Yn = yn , n = 1, . . . , N , determine the MMSE estimate
Ĥ = E[H|Y1 = y1 , . . . , Yn = yn ] of H.

5. Now consider a generalization of Problem 4. In particular, consider a system that

convolves an input with an (unknown) impulse response in addition to introducing
noise. In particular, the output of the system at time n (n = 0, . . . , N ) is given as
M
X
Yn = Hk xn−k + Zn , (1)
k=0

where H0 , . . . , HM (M < N ) are the coefficients of the system’s unknown impulse

response that needs to be estimated, and x0 , . . . , xN are the known system input
(assume xn = 0 for n < 0). In addition, Z0 , . . . , ZN are i.i.d. zero-mean Gaussian
with variance σ 2 . Given the observations Y0 = y0 , Y1 = y1 , . . . , YN = yN , you are
required to determine the LMMSE estimates of the impulse response coefficients. In
particular, the linear estimates are given as
N
X
Ĥn = yi a(n,i) , 0 ≤ n ≤ M,
i=0

or in matrix form,
Ĥ = Ay,
where Ĥ = [Ĥ0 , . . . , ĤM ]T , y = [y0 , . . . , yN ]T , and A is a matrix whose element at
the nth row and the ith column is a(n,i) . Extend the analysis discussed in class to this
problem by determining the matrix A (as a function of RHH and the input symbols
x0 , . . . , xN ) that minimizes the mean-squared error defined as
M  M

N
!2 
X 2  X X
ε= E Hn − Ĥn = E  Hn − yi a(n,i) 
n=1 n=1 i=1

Hint: For ease of analysis, try to work with vectors/matrices. In particular, try to
determine a matrix X such that Y = XH + Z (meaning that convolution operation
can be represented as a matrix multiplication).

3
6. Consider the temperature sensing problem in Example 6.10 of the textbook. Repeat
the MAP estimation problem of this example when the prior distribution of Y is
given by
(y−µt )2

− 2
1
C √ 2 e 2σt

fy (y) = 10 ≤ y ≤ 20
2πσt
0 otherwise


where C is a constant, and 10 ≤ µt ≤ 20.

(Try but do not submit). How will the solution to the LMMSE estimation problem
of Example 6.13 change with the modified prior given above.

7. Consider once again the temperature estimation problem of Example 6.13. Let the
modified LMMSE estimate of the temperature be given by
n
X
Ŷ = a0 + ai Xi
i=1
h i
where a0 , a1 , . . . , an are chosen so as to minimize the MSE E (Y − Ŷ )2 .

(a) Provide a clearly stated argument to show that

n n
" # " #
X X
min E (Y − a0 − ai Xi )2 ≤ min E (Y − ai Xi )2
a0 ,...,an a1 ,...,an
i=1 i=1

(b) For the temperature estimation problem h of Examplei 6.13, determine the coeffi-
2
cients a0 , a1 , . . . , an that minimize E (Y − Ŷ ) . Your final answer will reveal
that this LMMSE estimate turns out to be exactly the same as the MAP esti-
mate of Example 6.10.

8. Consider i.i.d. random variables X1 , . . . , Xn that are characterized by a mean µ and

variance σ 2 . Suppose you estimate the mean µ through the sample mean
n
1X
Sn = Xi .
n
i=1

(a) Can you determine an n0 such that for all n ≥ n0 , the absolute error between
the true and the estimated mean is less than 1% of the standard deviation 99%
of the times. In other words, determine an n0 such that for all n > n0 ,

Pr (|Sn − µ| ≤ 0.01σ) ≥ 0.99

(b) Now suppose that you are given additional information that the random vari-
ables X1 , . . . , Xn follow a Gaussian distribution (with mean µ and variance σ 2 ).
Determine an n1 < n0 such that for all n > n1 ,

Pr (|Sn − µ| ≤ 0.01σ) ≥ 0.99.

(Food for thought). Why is n1 < n0 ?

4
 9. Consider i.i.d. random variables X1 , . . . , Xn that are characterized by a mean µ and
variance σ 2 . Let µ̂ = n1 ni=1 Xi be the sample mean. An intuitive estimate of the
P
variance is given by
n
1X
σ̂ 2 = (Xi − µ̂)2 .
n
i=1

(a) Show that the variance estimate given above is biased, i.e., show that

E σ̂ 2 6= σ 2 .
 

(b) Provide a simple modification to the estimate so that it becomes unbiased.

10. Let X[n] be a discrete-time WSS process defined by

X[n] = αX[n − 1] + W [n]

where |α| < 1 and W [n] is an i.i.d. zero-mean process with variance σ 2 . Moreover,
W [n] is assumed to be independent of X[n − 1].
h i
(a) Determine the mean function µX [n] , E X[n] .
h i
(b) Determine the autocorrelation function RXX [n, n − k] , E X[n]X[n − k] .

11. Recall the random telegraph signal considered in class in which the signal switched
between two states X(t) = 1 and X(t) = 0, with the initial state being X(0) = 0.
In addition, the time between switches were exponentially distributed with mean λ1 .
The first-order pmf of the random process was derived as
1 1 −2λt
P (X(t) = 0) = + e ,
2 2
1 1 −2λt
P (X(t) = 1) = − e .
2 2
Now consider a slight variation of this telegraph signal. In particular, consider a
situation where the initial state was also random, with P (X(0) = 1) = p and
P (X(0) = 0) = 1 − p. For this modified telegraph signal, determine (you may
directly use results derived/discussed in class)

(a) The probabilities P (X(t) = 1) and P (X(t) = 0).

(b) The joint probability P (X(t2 ) = 1, X(t1 ) = 1).

5
 Matlab Assignment

12. Acoustic Echo Cancellation: Consider an audio conference system in which the
far-end audio signal samples are denoted as X1 , . . . , XN . The audio signal, played
through the speaker, reverberates through the room and goes into the microphone
causing the far-end user to hear the echo of his/her own voice. The room response
is typically modeled as equation (1) mentioned in Problem 5 with Y0 , . . . , YN repre-
senting the audio signal measured by the microphone. Acoustic echo cancellers (e.g.
those employed in Skype) use the observations on Y0 , . . . , YN as well as X0 , . . . , XN
to determine an LMMSE estimate of the room response, before using it for echo
cancellation. Students are encouraged to find more details in relevant textbooks, in
addition to having discussions with the instructor on relevance to other important
practical applications.
On LMS, you will find a file named audio.mat, in which you will find data for two
audio samples corresponding to X and Y. The audio signal Y has been obtained
by convolving X with a length-32 filter (M = 31) that follows the auto-regressive
model specified in Problem 2 with α = 0.5, and adding Gaussian noise of variance
σ 2 = 5 × 10−4 . Using the results derived in Problem 5, as well as the actual audio
data in X and Y, write a Matlab script that computes the LMMSE estimate of the
room response.
You are required to submit your Matlab code (upload on your LMS DropBox), as well
as a hard copy of your estimate of the room response.
Hint: For verification, you are encouraged to generate test responses h using the
model in Problem 2 and then check their closeness to the estimates obtained by your
computer program.
Note: You may very well be called in for an interview by the instructor to explain
your Matlab code.