Lecture 5

Spectrum Estimation
 Family of Non Parametric Methods

Classical Methods
(Fourier Transform Nonclassical Methods
Based) (Non Fourier Based)

Filterbank Approach
.Minimum Variance
Periodogram Based
.Periodogram ACF Based
.Modified Per. .Blackmen Tukey
.Bartlett
.Welch
 Periodogram Definition (recalled)
Based on the definition of PSD:
2
 1 M 
S x ( w)  lim E   x(n)e  jwn 
 2 M  1 n  M
M 


In practice we have one set of finite duration data.

This approach has 2 practical problems
1. Can’t do the expected value
2. Can’t do the limit

The periodogram is a method that ignores them both

2
ˆ 1 N 1
S PER ( w) 
N
 x
n 0
( n ) e  jwn

In practice we compute it using the discrete Fourier transform (DFT)

(possibly with zero padding) which computes the discrete time
Fourier transform (DTFT) at discrete frequency points
Periodogram-Viewed as a filterbank

Although the Periodogram is Always implemented using

the DFT, a filterbank interpretation is also very helpful.

Defining the impulse response of an FIR filter as

 1 jnwi
 e , 0  n <N
hi (n)   N
0, ow
The frequency response of this filter then becomes
N 1
H ( w)   hi (n)e  jwn
n 0

 jn ( w  wi )( N 1) / 2 sin( N ( w  wi ) / 2)
e
N sin(( w  wi ) / 2)
Periodogram-Viewed as a filterbank (cont)
With this definition of hi(n), the output of the ith filter becomes
n
yi (n)  x(n) * hi (n)  
k  n  N 1
x(k )hi (n  k )

1 n

N

k  n  N 1
x(k )e j ( n  k ) wi

2
Note that if one chooses n=N-1 the term yi ( N  1) gives the
periodogram
2 2
1 N 1
j ( N 1) wi 2 1 N 1
 
2 j ( N 1 k ) wi  jkwi
yi ( N  1)  x ( k ) e e x ( k ) e
N k 0 N k 0
2
1 N 1
  x ( k ) e  jkwi
 NSˆPER ( w)
N k 0
If a WSS random process x(n) is filtered with hi[n],
then the output process is
n
1 n
yi [ n]  x[ n]* hi [ n]   x[ k ]hi [ n  k ]   x[ k ]e j ( n  k ) wi
k  n  N 1 N k  n  N 1

Since H i (e jw )
w  wi
1

The power spectrum of x[n] and y[n] are equal at frequency wi

If the bandwidth of the filter is small enough then the power spectrum of
x[n] may be assumed to be approximately constant over the passband
of the filter, then the power of in yi[n] will be approximately

 2
1
E  yi [n ]2  
2  S x (e jw ) H i (e jw ) dw


w 1
 S x (e jwi )  S x (e jwi )
2 N
 Therefore

S x (e jwi )  NE yi [n]2 
Thus if we are able to estimate the power in yi[n], then the power
spectrum at frequency wi may be estimated as follows
Sˆ x (e jwi )  NE
ˆ  y [ n] 
i
2

One simple (crude) way to estimate the power is to use a one

point sample average

 
Eˆ yi [n]2  yi [ N  1]
2

This is equivalent to
2 2
1 N 1 1
Sˆx (e jwi )  N yi [ N  1]   x[k ]e
2
 Thus
2  jkwi  jkwi
yi [ N  1]  2 x[ k ]e
N k 0
N

This is equivalent to the periodogram

 Basic design steps of MVSE
• Design a bank of band pass filters hi[n] with center
frequency wi so that each filter rejects the maxiimum amount
of out of band power while passing the component at
frequency with no distortion

• Filter x[n] with each filter in the filterbank and estimate the
power in each output process yi[n]

• Set Sx(wi) equal to the power estimated in step 2 divided by

the filter bandwidth

The idea is to steer our filter to a particular frequency

and reject the signal at all other frequencies
 Goal for the Minimum Variance
Spectral Estimation Method
Figure out a way to design each filter in the filterbank to
minimize the leakage. This will lead to a data dependent design.
Collect Data - then - design Filters

Design Filters that minimize the sidelobes while

keeping the height of the mainlobe at 1.

Design Goals:
1. Want Hi(wi)=1
this will let through the desired Sx(wi).
2. Minimize the total output power in the filter

1
i  E  y [n] 
i
2

2 
2
S x ( w) H i ( w) dw


w 1
 S x ( wi )  S x ( wi )
2 N

This is equivalent to minimizing the sidelobe contribution

even though the integral includes the desired frequency wi.
 Work on the  term
The frequency response of the filter hi[n] is
p 1
H i ( w)   hi [n]e  jwn where p (<N) is the order of the filter
n 0

1  p 1   p 1
jlw 
i      i
 jkw *
h i [ k ]e h [l ] e  S x ( w)dw
2 k 0   l 0 
p 1 p 1
1 
  hi [k ]hi [l ] *
 S x ( w) e jw ( l  k )
dw
k 0 l 0 2  

1 
Noting that
2 
S x ( w)e jw( l  k ) dw  rx (l  k )
p 1 p 1
i    hi [ k ]hi*[l ]rx (l  k )
k 0 l 0

 hiH R xhi
Work on minimizing the Matrix Form
For each i, minimize
i  hiH R xhi
Subject to

H i ( wi )  1
hi ei = 1
where ei  [1 e jwi e j 2 wi . . . e j ( p 1) wi ]
Solve the above constrained optimization
problem using Lagrange multipliers
H H
J  h i R xhi   (h i R x  1)

i.e., choose hi and λ to minimize J

= cT
 MV Estimate of an AR(1) process
• Let x[n] be an AR(1) process k
rx [ k ]  2
,  <1
1
with autocorrelation
1 1
S x ( w)  
• The power spectrum of x[n] is ; 1   e  jw
2
1   2  2 cos( w)

ˆ p 1
• Given rx[k] for k<p the minimum S MV ( w)  H 1
e Rx e
variance spectral estimate is:
1
Rx  Toep (1,  ,  2 ,...,  p )
1 2

 1  0 ... 0 0 
 
  1  2  ... 0 0 
1
 0  1  2 ... 0 0 
Rx  
    ...   
 0 0 0 ... 1 2  

 0 
 0 0 ...  1 

 MV Estimate of an AR(1) process
p 1
SˆMV ( w) 
2  ( p  1)(1   2 )  2 p cos( w)

Note that minimum variance estimate converges

to the true estimate as p tends to infinity
MW Estimate of a Complex exponential in noise
Let x(n) be a random phase complex exponential in white noise
x(n)  A1e jnw1  w(n)
where
A1  A1 e j
and  is a rv uniformly distributed
2
over [-, ].
If the variance of w(n) is  w then the autocorrelation of x(n) is

rx (k )  A1 e jkw1   w2 (k )

Thus the autocorrelation matrix is

Rx  A1 e1e1H   w2 I

where
jw1 j 2 w1 jpw1 T
e1  1, e , e ,...e 
Woodbury matrix identity is

                                                                       

where A, U, C and V all denote matrices of the correct size.

Specifically, A is n-by-n, U is n-by-k, C is k-by-k and V is k-
by-n.
MW Estimate of a Complex exponential in noise

Using Woodbury’s identity

1 H
A e e
1  
1 1 1
1  w4 A1
Rx1  2 I  2 I  2 H
e1e1 
w A w   w  ( p  1) A1 
1  12 e1e1H
w
Thus,

p 1 p 1
SˆMV ( w)  H 1 
e Rx e 1 H  A1 H

e I  2 e1e1  e
 w2   w  ( p  1) A1 
 w2

A1 /( p  1) H 2
1 2 e e1
 w  ( p  1) A1
 MW Estimate of a Complex exponential in noise

Since
p p
e e1   e
H  jkw
e jkw1
  e  jk ( w w1 )  WR ( w  w1 )
k 0 k 0

We can express the minimum variance estimate as

 w2
SˆMV ( w) 
A /( p  1) 2
1 2 1 WR ( w  w1 )
 w  ( p  1) A1

From this expression we see that the MV estimate

attains its maximum at w=w1 with

 w2
SˆMV ( w)    w2  ( p  1) A1
w  w1 A1 ( p  1)
1 2
 w  ( p  1) A1
MV Estimate of a Complex exponential in noise

Therefore the estimate of the power in x(n) at frequency w=w1 is

1 ˆ  w2
ˆ ( w1 ) 
2
x S x ( w1 )   A1
p 1 p 1

If the signal to noise ratio is large then

ˆ x2 ( w1 )  A1

Furthermore if p>>1 and w≠w1, then WR(w-w1)0

SˆMV ( w)   w2
w  w1
 The Minimum Variance Spectral Estimation:
MATLAB code
function Px = minvar(x,p)
%MINVAR Spectrum estimation using the minimum variance method.
% The spectrum of a process x is estimated using the minimum
% variance method (sometimes called the maximum likelihood method).
% x : Input sequence
% p : Order of the minimum variance estimate - for short
% sequences, p is typically about length(x)/3
% The spectrum estimate is returned in Px using a dB scale.
%
x = x(:);
R = covar(x,p);
[v,d]=eig(R);
U = diag(inv(abs(d)+eps));
V = abs(fft(v,1024)).^2;
Px = 10*log10(p)-10*log10(V*U);
end;
function R=covar(x,p)
%COVAR Generates a covariance matrix/
%USAGE R=covar(x,p)
%
% Generates a p x p covariance matrix for the sequence x.

%---------------------------------------------------------------

x = x(:);
m = length(x);
x = x - ones(m,1)*(sum(x)/m);
R = convm(x,p)'*convm(x,p)/(m-1);
end;
Maximum Entropy Method
For a data record of length N the autocorrelation sequence
can only be estimated for |k|<N.

rˆx (k ) Is set to zero for |k|<N.

In reality this is not the case

How should this extrapolation be done so as to

improve the performance.

Maximum Entropy Method is one possible way.

 Maximum Entropy Method

Constraints: PSD should be real valued and nonnegative for all w.

+ Burg’s maximum entropy
Entropy – Uncertainty or randomness
Thus a maximum entropy extrapolation is equivalent
to finding the autocorrelation sequences re(k) that make
the process x(n) as white as possible. This constraint
places the least amount of structure on x(n)
Maximum Entropy Method
Maximum Entropy Method

From
Maximum Entropy Method
Maximum Entropy Method
Having determined the form of the MEM spectrum
one needs to calculate the coefficients ap(k) and b(0).
These coefficients should be chosen to satisfy

The coefficients ap(k) are the solution of autocorrelation

normal equations

Furthermore if

the above constraint is satisfied and the MEM spectrum becomes

Maximum Entropy Method
Summary:
Given a sequence of autocorrelations rx(k) for k=0,1,…,p
the mem spectrum is computed as
First autocorrelation normal equations are solved
for the all pole coefficients ap(k) and ep then MEM
spectrum is formed by

Since MEM is an all pole power spectrum the autocorrelation

sequence satisfies

These are the Yule Walker equations. It can easily be derived

by considering the difference equation of a AR(p) model.
Matlab Implementation

function Px = mem(x,p)
%MEM Spectrum estimation using the Maximum Entropy Method (MEM).
%--- %USAGE Px = mem(x,p) %
% The spectrum of a process x is estimated using the maximum
% entropy method, which uses the autocorrelation method to
% find a pth-order all-pole model for x(n), and then forms
% the estimate of the spectrum as follows: % Px = b^2(0)/|A(omega)|^2
% The spectrum estimate is returned in Px using a dB scale.
% x : Input sequence % p : Order of the all-pole model %
%---------------------------------------------------------------
% copyright 1996, by M.H. Hayes. For use with the book
% "Statistical Digital Signal Processing and Modeling"
% (John Wiley & Sons, 1996).
%---------------------------------------------------------------
[a,e] = acm(x,p);
Px = 10*log10(e/length(x))-20*log10(abs(fft(a,1024)));
end;
Matlab Implementation

function [a,err] = acm(x,p)

%ACM Find an all-pole model using the autocorrelation method %----
% The input sequence x is modeled as the unit sample response of
% a filter having a system function of the form
% H(z) = b(0)/A(z), where the coefficients of A(z) are contained in the vector
% a=[1, a(1), ... a(p)]
% The input p defines the number of poles in the model.
% The modeling error is returned in err.
% The numerator b(0) is typically set equal to the square
% root of err.

x = x(:); N = length(x);
if p>=length(x), error('Model order too large'),
end
X = convm(x,p+1);
Xq = X(1:N+p-1,1:p);
a = [1;-Xq\X(2:N+p,1)];
err = abs(X(1:N+p,1)'*X*a);
end;
Fs = 1000;
p=100;
t = 0:1/Fs:1;
N=length(t);
Nz=N;
x = cos(2*pi*t*200) + cos(2*pi*t*240) + randn(size(t));
Px = mem(x,p,Nz);
figure(1)
f=[0:Nz-1]*Fs/Nz; 25
plot(f,abs(Px))

20

15

10

0
0 100 200 300 400 500 600 700 800 900 1000
 8

Fs=1000; 6
p=10;
t = 0:1/Fs:10; 4
N=length(t);
Nz=N;
x = randn(size(t)); 2

aa=[1 -.5];
y=filter(1,aa,x); %This is an FIR filter 0

Py = mem(y,p,Nz);
-2
f=[0:Nz-1]*Fs/Nz;
H = FREQZ(1,aa,f,Fs);
-4
0 100 200 300 400 500 600 700 800 900 1000
figure(1)
4.5
plot(f,((Py)),'r')
hold on 4
plot(f,10*log10(abs(H).^2),'g')
3.5

figure(2) 3
plot(f,10.^(Py/10),'r')
hold on 2.5

plot(f,(abs(H).^2),'g')
2

1.5

0.5

0
0 100 200 300 400 500 600 700 800 900 1000
 MEM PSD estimate of an AR(1) process

p=2

4 8

3.5
6

4
2.5

2 2

1.5
0

-2
0.5

0 -4
0 100 200 300 400 500 600 700 800 900 1000 0 100 200 300 400 500 600 700 800 900 1000
MEM PSD estimate of an AR(2) process

p=5
18

16

14

12

10

0
0 100 200 300 400 500 600 700 800 900 1000
MEM PSD estimate of an MA(2) process

p=10; (h[1.0000 -1.0000 0.2500])

1.4

1.2

0.8

0.6

0.4

0.2

0
0 100 200 300 400 500 600 700 800 900 1000
MEM PSD estimate of an MA(2) process

p=50
1.4

1.2

0.8

0.6

0.4

0.2

0
0 100 200 300 400 500 600 700 800 900 1000
SOME FACTS about HARMONIC PROCESSES
And
Frequency Estimation
 Frequency Estimation

Eigen Decomposition of the autocorrelation Matrix:

Consider the following first order process: single exponential in white noise

The autocorrelation of this process is given by

The MxM autocorrelation matrix for the process x(n)

is thus the sum of the autocorrelation matrix due to
the signal Rs and an autocorrelation matrix due to
the noise Rn
 Eigen Decomposition of the autocorrelation Matrix

Rs has rank 1. Rn is of full rank

Note that the signal autocorrelation matrix can be written as

where
Since the rank of Rs is one, then Rs has only one nonzero eigenvalue

Thus the nonzero eigenvalue is MP1 and the corresponding eigenvector is e1.
Eigen-Decomposition of the autocorrelation Matrix
Since Rs is Hermitian the remaining eigenvectors v2, v3, … vM are
orthogonal to the eigenvector e1 which is due to the signal

If we let is be the eigenvalues of Rs

Therefore the eigenvectors of Rx are the same as those of Rs

and the eigenvalues of Rx are

As a result the largest eigenvalue of Rx is

The remaining M-1 eigenvalues are  w2

Eigen-Decomposition of the autocorrelation Matrix

Thus it is possible to extract all the parameters of interest

from the eigenvalues and eigenvectors of of Rx as follows
Eigen-Decomposition of the autocorrelation Matrix
The autocorrelation sequence is normally estimated. Small errors in the
estimated autocorrelation sequence may lead to large errors in the
eigenvalues and eigenvectors. Thus instead of estimating the frequency of
the complex exponential from a single eigenvector one may consider a
weighted average.
2
Let vi be a noise eigenvector of Rx (with an eigenvalue  w )
And let vi(k) be the kth component of vi. The DTFT of vi(k)

The orthogonality condition

implies that Vi(ejw)=0 at w=w1 i.e., at the frequency of the complex exponential.

Thus a frequency estimation of the form

will be large at w=w1.

Eigen-Decomposition of the autocorrelation Matrix

will be large at w=w1.

Thus the location of the peak of this frequency estimation function

may be used to estimate the frequency of the complex exponential

The above frequency estimation function uses a single eigenvector

therefore it may be sensitive to the errors in estimation of Rx.

Using a weighted average of all NOISE eigenvectors as

where αi are appropriately chosen constants

gives a more robust estimation.
Eigen-Decomposition of the autocorrelation Matrix

Consider now TWO complex exponentials in white noise

The autocorrelation matrix can be written as

(Due to noise)
Eigen-Decomposition of the autocorrelation Matrix
One can more concisely express Rx as

Mx2 matrix containing the

signal vectors e1 and e2.

Diagonal matrix
containing powers
Let us perform an eigendecomposition on Rx:

Let vi and λi be the eigenvectors and eigenvalues of Rx

with eigenvalues arranged in decreasing order
Eigen-Decomposition of the autocorrelation Matrix
Since the rank of Rs is equal to two, then Rs has only two nonzero eigenvalues.
Both of these are greater than zero since Rs is nonnegative definite.
2
Therefore the first two eigenvalues of Rx are greater than  w
2
and the remaining eigenvalues are equal to  w.

Thus the eigenvalues and eigenvectors of Rx can be divided into two groups:
The first group consisting of two eigenvectors that have eigenvalues
2
greater than  w are referred to as signal eigenvectors and
span a two dimensional subspace called the signal subspace.

The second group consisting of those eigenvectors that have eigenvalues

equal to . These are referred to as noise eigenvectors and they span an
M-2 dimensional subspace called the noise subspace.

Since Rx is Hermitian the eigenvectors form an orthonormal set.

Therefore the signal and noise subspaces are orthogonal.
That is to say for any vector u in the signal subspace and
any vector v in the noise subspace uHv=0
Eigen-Decomposition of the autocorrelation Matrix
Eigen-Decomposition of the autocorrelation Matrix
Generalize to the case of WSS process with p distinct harmonics
in white noise
The autocorrelation sequence is

The autocorrelation matrix can be written as

More concisely
Eigen-Decomposition of the autocorrelation Matrix
s
Since the eigenvalues of Rx are i  i   w where i are the eigenvalues of Rs
s 2

And since Rs is a matrix of

2
rank p, then the first p eigenvalues of
Rx will be greater than  w and the last M-p eigenvalues will be equal to  w
2

Again the eigenvalues aand eigenvectors of Rx can be divided into two groups:

The signal eigenvectors v1, v2, …., vp that have eigenvalues greater than  w2
2
The noise eigenvectors vp+1, vp+2, … , vM that have eigenvalues equal to  w

Assuming that the eigenvectors are normalized to unit norm,

Rx can be decomposed as

Or in matrix form
Eigen-Decomposition of the autocorrelation Matrix

Vss and Vnn are diagonal matrices that contain the eigenvalues
i  is   w2 and i   w2
Eigen-Decomposition of the autocorrelation Matrix

As in the case of one and two complex exponentials in white noise,

The orthogonality of the signal and noise subspaces may be used
to estimate the frequencies of the complex exponentials.

Since each signal vector e1, e2, … , ep lies in the signal subspace,
The orthogonality implies that ei will be orthogonal to each of the noise
Eigenvectors.

Therefore the frquencies may be estimated using a frequency

estimation function such as
 Pisarenko Harmonic Decomposition
• P complex exponentials in white noise, ( p known)
• (p+1) values of the autocorrelation sequence are known or estimated.
• With a (p+1)x(p+1) autocorrelation matrix, the dimension
of the noise subspace is one and it is spanned by the eigenvector
corresponding to the minimum eigenvalue

Denoting this NOISE eigenvector by vmin, it follows that vmin will

be orthogonal to each of the signal eigenvectors ei.

Therefore

Is equal to zero at each of the complex

exponential frequencies wi for i=1, 2, … , p.
Pisarenko Harmonic Decomposition
Consequently the z transform of the noise eigenvector known as
the eigenfilter has p zeros on the unit circle

Therefore the frequencies of the complex exponentials may be

extracted from the roots of the eigenfilter.

As an alternative to rooting Vmin(z), one may also form the

frequency estimation function

Note that this eigenspectrum (pseudospectrum) does not

contain any information about power in the complex expoonentials
 Pisarenko Harmonic Decomposition
Once the frequencies of the complex exponentials are determined,
The powers Pi may be found from the eigenvalues of Rx.
Let us assume that the signal subspace eigenvectors v1, v2, … , vp
H
Are normalized so that vi vi  1 . With

H
Multiplying both sides on the left by vi

Substituting for One gets

Which can be simplified to

Pisarenko Harmonic Decomposition

Thus the equation for the powers simplifies to

Which is p linear equations in p unknowns. In matrix fom:

Example …
 MUSIC (MUltiple SIgnal Classificaition)
• p complex exponentials in white noise, ( p known)
• (p+1) values of the autocorrelation sequence are known or estimated.
• With a (M)x(M) autocorrelation matrix,
• p signal eigenvetors corresponding to the largest p eigenvalues
• M-p noise eigenvectors with corresponding eigenvalues ideally equal to  w2

Since the eigenvectors of Rx are of length M each of the

noise subspace eigenfilters

Associated with the noise eigenvector vi will exhibit sharp peaks at the
frequencies of the complex exponentials. The remaining (M-p-1) zeros
May lie anywhere giving rise to spurious peaks if they are close to the unit circle.
Therefore when only one noise eigenvector is used to estimate
the complex exponential frequencies, there may be some ambiguity
In distinguishing the desired peaks from the spurious ones.

In the MUSIC algorithm the effects of these spurious peaks are

reduced by averaging using a frequency estimation function
Noise Subspace methods for Frequency Estimation