E9 261 – Speech Information Processing

Mid-term Exam 2
Date: May 23, 2020

Time: 12:00 noon, Duration: 3 days

Instructions

1. This exam is open book. Use of calculators is allowed. However, the use of computers,
mobile phones and other handheld devices are not allowed. Only the materials used in
the class as well as reference materials are allowed to be used.

2. Academic integrity is expected and all problems are required to be solved individually
without any consultation. This includes prohibition on consulation online or through any
other communication devices.

3. Any clarifications should be directly performed with the instructor.

4. Write the answers to each question with the question number clearly on white paper. Use
a separate sheet for each question. Scan all the sheets into one single pdf file. Name
the pdf file midTerm2-answers-{your-full-name}.pdf and the email the answers with the
subject line MidTerm2-Answers before the deadline.

5. Notation - bold symbols are vectors, capital bold symbols are matrices and regular symbols
are scalars.

6. Answer all questions [1-5].

1. Maximum Mutual Information (MMI) - HMM - In the conventional HMM, the
association between the state sequence and observation sequence is not taken into account.
In a modified formulation, we would like to add this component in the modeling. Let
H(O|q) denote the conditional entropy of the observation sequence O = {ot }Tt=1 given
the state sequence q = {qt }Tt=1 . The conditional entropy is defined as,
XXZ

H(O|q) = − P (ot , qt ) log P (ot |qt ) ∂ot
t qt ot

In the HMM formulation, we would like to minimize the conditional entropy (maximizing
the mutual information). Thus, we modify the ML method in the following manner,

J = −(1 − ǫ) H(O|q) + ǫ log P (O)

where ǫ is a positive constant whose maximum value is 1 and J is the function to be

maximized to determine the parameters. We assume continuous density HMM with state
emission probabilities as single Gaussian distribution (bj (ot ) ∼ N (ot , µj , Σj )). Derive
the update equations for mean and covariance of the emission probabilties for the MMI-
HMM.
(Points 25)
2. Sequence Distance
Veena is trying to find distance between two text sequences (denoted as a and b), where
the patterns are sequences of ASCII characters. When computing the text distance, three
elementary operations are possible,
• Insertion - Inserting a character in a. For example, if a = uv, then inserting character
x gives uxv. This can be denoted as x → ǫ, where ǫ is the blank character.
• Deletion - Deleting a character in b. For example, if b = pyq, then deleting character
y gives pq. This can be denoted as ǫ → y.
• Substitution - Substituting a character in a. For example, if a = uxv, then substi-
tuting character x with y gives uyv. This can be denoted as x → y.
Each of the above operations has an associated cost denoted as cins (x) = 1, cdel (y) = 1
and csub (x, y) = 2. Note that csub (x, x) = 0.
(a) Check whether the algorithm associative d(a, b) + d(b, c) = d(a, c) ?
(b) Using this case find the distance between two text sequences a = SOM ET HIN G
and b = N OT HIN G. Show all the steps involved.
(Points 25)
3. Consider a two-bit register. The register has four possible outputs: 00, 01, 10 and 11.
Initially, at time t = 0, the contents of the register is chosen at random to be one of these
four outputs, each with equal probability. At each time step, beginning at time t = 1,
the register is randomly manipulated as follows: with probability 12 , the register is left
unchanged; with probability 14 , the two bits of the register are exchanged (e.g., 01 becomes
10); and with probability 14 , the right bit is flipped (e.g., 01 becomes 00). Suppose that on
the first four time steps (t = 1, 2, 3, 4), we observe 01,10,11 and 11, what is the probabity
of observing this sequence. Use the most computationally efficient algorithm. (Points
15)
4. Weighted linear prediction In standard linear prediction (LP), the N th order predic-
tor defined by {aN,1 , aN,2 , ...aN,N } is obtained by minimizing the energy of the residual
signal E[|e[n]|2 ], where e[n] = x[n] − x̂[n] is the difference between original and predicted
signal.

(a) Formulate a solution to the LP problem when the objective is to minimize E[|w[n]e[n]|2 ]
where w[n] is a pre-defined weighting function.
(b) How is the orthogonality principle applied in this case.

(Points 15)

5. Poisson HMM A Poisson process HMM is one in which the observations are postive
integers and the emission probabilities are given by the Poisson density,

λkj e−λj
bj (ot = k) =
k!
where j = 1, ..., N denotes the HMM states and ot , t = 1, ..., T denotes the observation
sequence.

• Derive the update formula for the Poisson density parameter λj using the EM algo-
rithm. Assume N states for the HMM.
• How will the update equations if density parameter λj is shared among all states,
i.e., λj = λ for j = 1, ..., N .

(Points 20)