Optimum 

Linear Filter (Part II)
 Linear Prediction
• given a set of samples 𝑥 𝑛 , 𝑥 𝑛 1 , … , 𝑥 𝑛 𝑀 of a
stochastic process
• Use LMMSE to estimate the value of the same process, 𝑥 𝑛
𝑖 (where 𝑖 ∉ 0. . 𝑀 ), at a different time I
𝑥 𝑛 𝑖 𝑐 𝑛 𝑥 𝑛 𝑘

• the usual convention of negated predictor coefficients (i.e.

𝑐 𝑛 )
 Linear Prediction
• the error signal
𝑒 𝑛 𝑥 𝑛 𝑖 𝑥 𝑛 𝑖

𝑥 𝑛 𝑖 𝑐 𝑛 𝑥 𝑛 𝑘

• To determine the MMSE signal estimator we partition as follows:

𝑒 𝑛 𝑐 𝑛 𝑥 𝑛 𝑘 𝑥 𝑛 𝑖 𝑐 𝑛 𝑥 𝑛 𝑘

𝐜 𝑛 𝐱 𝑛 𝑥 𝑛 𝑖 𝐱 𝑛 𝐜 𝑛 𝑥 𝑛 𝑖

• where:
𝐱 𝑛 𝑥 𝑛 𝑥 𝑛 1 … 𝑥 𝑛 𝑖 1 𝑥 𝑛 𝑖 1 … 𝑥 𝑛 𝑀
𝐜 𝑛 𝑐 𝑛 𝑐 𝑛 … 𝑐 𝑛 𝑐 𝑛 … 𝑐 𝑛
 Linear Prediction
• Using the Orthogonality Principle:
𝐸 𝐱 𝑛 𝑒 𝑛
𝐸 𝐱 𝑛 𝐱 𝑛 𝐜 𝑛 𝑥 𝑛 𝑖
𝐸𝐱 𝑛 𝐱 𝑛 𝐜 𝑛 𝐸𝐱 𝑛 𝑥 𝑛 𝑖
0
𝐑 𝑛 𝐜 𝑛 𝐫 𝑛
• MMSE
𝑃 𝑛 𝐸 𝑥 𝑛 𝑖 𝑒 𝑛
𝐸 𝑥 𝑛 𝑖 𝐱 𝑛 𝐜 𝑛 𝑥 𝑛 𝑖

where 𝑃 𝑛 𝑖 𝐸 |𝑥 𝑛 𝑖 | 𝐸 𝑥 𝑛 𝑖 𝑥 𝑛 𝑖
 Linear Prediction
• Symmetric Linear Smoother
 Linear Prediction
• Symmetric Linear Smoother
 Linear Prediction
• Forward Linear Prediction
 Linear Prediction
• Forward Linear Prediction
 Linear Prediction
• Forward Linear Prediction
 Linear Prediction
• Backward Linear Prediction
 Linear Prediction
• Backward Linear Prediction
 Linear Prediction
• Backward Linear Prediction
 Linear Prediction
• Stationary Processes: If the process 𝑥 𝑛 is WSS, then the
elements of the correlation matrix 𝐑 𝑛 and correlation vector
𝐫 𝑛 no longer depend explicitly on the time-index n but only
on the difference.
𝐫 𝐸 𝐱 𝑛 1 𝑥 𝑛 𝐫
𝑻
𝐫 𝐸 𝐱 𝑛 𝑥 𝑛 𝑀 𝐉𝐫 𝑟 𝑀 𝑟 𝑀 1 … 𝑟 1

where J is the exchange matrix:

0 0 ⋯ 1
⋮ ⋮ ⋱ ⋮
𝐉
0 1 ⋯ 0
1 0 ⋯ 0
Linear Prediction
 Linear Prediction
• Properties
 Optimum Infinite Impulse Response 
Filters
• IIR filter stands for Optimum Infinite Impulse 
Response Filters;
• IIR filters use infinite observation data to produce 
estimate
• We discuss stationary signals only in IIR filters;
• There are two types of IIR filters: noncausal IIR 
filters & causal IIR filters;
• Both signals and systems in IIR systems are 
discrete, thus z‐transform (and inverse z‐
transform) will be used.
 Optimum Infinite Impulse Response 
Filters
• Noncausal IIR filter
 Optimum Infinite Impulse Response 
Filters
• Noncausal IIR filter

Z‐transform
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter
Noncausal IIR filter

The noncausal method is not 
applicable for casual system since 
the k ranges from only [0, +∞] for 
casual IIR filters, and such z‐
transform cannot be applied in 
this case. 
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter

1st sub‐filter: 2nd sub‐filter:

Converting an  Converting a white 
arbitrary x(n) input  noise signal w(n) to 
signal to a white  an estimate signal 
noise signal w(n) 𝑦 𝑛
 Optimum Infinite Impulse Response 
Filters
• 1st sub‐filter:Converting an arbitrary x(n) input 
signal to a white noise signal w(n).
X(z) W(z)
G(z)=?

• Let’s consider its inverse system first:
X(z) W(z)
‐1
H(z)=G (z)

• Then for the original “1st sub‐filter” 𝐺 𝑧 𝐻 𝑧
X(z) W(z)
G(z)=? Note: both H(z) and G(z) 
must be stable – zeros 
must be within the unit 
circle in z‐domain
 Optimum Infinite Impulse Response 
Filters
• 2nd sub‐filter:Converting a white noise signal 
w(n) to an estimate signal 
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter

1st sub‐filter: 2nd sub‐filter:

𝐺 𝑧 𝐻 𝑧

For the complete casual IIR filter:
Optimum Infinite Impulse Response 
Filters
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter

1st sub‐filter: 2nd sub‐filter:

𝐺 𝑧 𝐻 𝑧

For the complete casual IIR filter:
 Optimum Infinite Impulse Response 
Filters
• Causal IIR filter