4.

CRAMER-RAO LOWER BOUND

4.1 Estimator Accuracy
We first determine how well an minimum variance unbiased (MVU) estimator can estimate a
parameter.
For example, a single sample is observed as
𝑥[0] = 𝐴 + 𝑤[0]

where 𝑤[0]~𝒩(0, 𝜎 2 ), and we desire to estimate 𝐴, an estimator is 𝐴̂ = 𝑥[0].

• The smaller the 𝜎 2 the better the estimate 𝐴̂.

• The larger the 𝜎 2 , the worse the estimate.

This can be seen through the pdf of 𝑥[0]

1 1
𝑝(𝑥[0], 𝐴) = exp [− (𝑥[0] − 𝐴)2 ]
√2𝜋𝜎 2 2𝜎 2

• As shown in the figure, if the variance is low, estimation accuracy increases.

• When the pdf is viewed as a function of the unknown parameter, it is termed the
likelihood function.
• Sharpness of the likelihood function determines the accuracy of the estimation.
• We sometimes consider the natural logarithm of the likelihood function, which we call
log-likelihood function
1
ln 𝑝(𝑥[0], 𝐴) = − ln √2𝜋𝜎 2 − 2 (𝑥[0] − 𝐴)2
2𝜎

Figure 4.1. A DC signal embedded in noise.

• Sharpness is effectively measured by the second-derivative of the log-likelihood

function
𝜕 ln 𝑝(𝑥[0], 𝐴) 1
= 2 (𝑥[0] − 𝐴)
𝜕𝐴 𝜎
 𝜕 2 ln 𝑝(𝑥[0], 𝐴) 1
− 2
= 2
𝜕𝐴 𝜎

• As the curvature increases, 𝜎 2 decreases. Since we already know that the estimator
𝐴̂ has variance 𝜎 2 , then, for this example,
1
var(𝐴̂) = − 2
𝜕 ln 𝑝(𝑥[0], 𝐴)
𝜕𝐴2
• In general the second derivative depends on the 𝑥[𝑛] itself. Thus, a more appropriate
way of measuring the curvature, which measures the average curvature of log-
likelihood is
𝜕 2 ln 𝑝(𝑥[𝑛]; 𝜃)
Measure of Curvature ~ − 𝐸 [ ]
𝜕𝜃 2

4.2 Cramer-Rao Lower Bound

THEOREM 1: CRAMER-RAO LOWER BOUND – SCALER PARAMETER

Assumption:

• It is assumed that the pdf 𝑝(𝐱; 𝜃) satisfies the regularity condition

𝜕 ln 𝑝(𝐱; 𝜃)
𝐸[ ]=0 ∀𝜃
𝜕𝜃

where the expectation is taken with respect to 𝑝(𝐱, 𝜃).

Then, the variance of any unbiased estimator 𝜃̂ must satisfy

1
var(𝜃̂) ≥
𝜕 2 ln 𝑝(𝐱; 𝜃)
−𝐸 [ ]
⏟ 𝜕𝜃 2
𝐼(𝜃)

where the derivative is evaluated at the true value of 𝜃 and the expectation is taken with
respect to 𝑝(𝐱; 𝜃). Furthermore, an unbiased estimator may be found that attains the bound
for all 𝜃 if and only if

𝜕 ln 𝑝(𝐱; 𝜃)
= 𝐼(𝜃)(𝑔(𝐱) − 𝜃)
𝜕𝜃

For some functions 𝐼 and 𝑔. That estimator, which is the MVU estimator, is 𝜃̂ = 𝑔(𝐱), and the
minimum variance is 1/𝐼(𝜃), where 𝐼(𝜃) is the Fisher information,

𝜕 2 ln 𝑝(𝐱; 𝜃)
𝐼(𝜃) = −𝐸 [ ].
𝜕𝜃 2
Example: Consider the previous example, where 𝑥[0] = 𝐴 + 𝑤[0]

var(𝐴̂) ≥ 𝜎 2 .

We can easily see that there cannot exists any unbiased estimator whose variance is lower
than 𝜎 2 .
𝜃=𝐴

𝜕 ln 𝑝(𝑥[0], 𝐴) 1
= 2 (𝑥[0] − 𝐴)
𝜕𝐴 𝜎
1
𝐼(𝜃) =
𝜎2
𝑔(𝑥[0]) = 𝑥[0]

so that all the equations in the theorem are satisfied.

Example: DC level in WGN

Generalizing the Example above, we consider multiple observations

𝑥[𝑛] = 𝐴 + 𝑤[𝑛], 𝑁 = 0, ⋯ 𝑁 − 1

where 𝑤[𝑛] is WGN with variance 𝜎 2 . To determine the CRLB for 𝐴

𝑁−1
1 1
𝑝(𝐱; 𝐴) = ∏ exp [− (𝑥[𝑛] − 𝐴)2 ]
√2𝜋𝜎 2 2𝜎 2
𝑛=0

𝑁−1
1 1 2
= 𝑁 exp [− 2𝜎 2 ∑(𝑥[𝑛] − 𝐴) ]
(2𝜋𝜎 2 ) 2 𝑛=0

Therefore,
𝑁−1
𝑁 1
ln 𝑝(𝐱; 𝐴) = − ln(2𝜋𝜎 2 ) − 2 ∑(𝑥[𝑛] − 𝐴)2
2 2𝜎
𝑛=0

Let us take the first derivative w.r.t. 𝐴

𝜕 ln 𝑝(𝐱; 𝜃)
= 𝐼(𝜃)(𝑔(𝐱) − 𝜃)
𝜕𝜃

𝑁−1 𝑁−1
𝜕 ln 𝑝(𝐱; 𝐴) 1 𝑁 1 𝑁
= 2 ∑(𝑥[𝑛] − 𝐴) = 2 ( ∑ 𝑥[𝑛]) − 𝐴 = 2 (𝑥̅ − 𝐴)
𝜕𝐴 𝜎 𝜎 𝑁 𝜎
𝑛=0 ⏟ 𝑛=0
( 𝑥̅ )
where 𝑥̅ is the sample mean. Differentiating again yields,

𝜕 2 ln 𝑝(𝐱; 𝐴) 𝑁
2
=− 2
𝜕𝐴 𝜎
the derivative is a constant. Therefore the expected value gives itself and

1 𝜎2
var(𝐴̂) ≥ =
𝜕 2 ln 𝑝(𝐱; 𝐴) 𝑁
−𝐸 [ ]
𝜕𝐴2

Example: Phase Estimation

Assume that we wish to estimate the phase of a sinusoid 𝜙 embedded in WGN

𝑥[𝑛] = 𝐴 cos(2𝜋𝑓0 𝑛 + 𝜙) + 𝑤[𝑛] , 𝑁 = 0, ⋯ , 𝑁 − 1

The amplitude and the frequency 𝑓0 are assumed to known. The pdf is
𝑁−1
1 1
𝑝(𝐱; 𝜙) = 2 𝑁/2
exp {− 2 ∑[𝑥[𝑛] − 𝐴 cos(2𝜋𝑓0 𝑛 + 𝜙)]2 }
(2𝜋𝜎 ) 2𝜎
𝑛=0

Log-likelihood function is
𝑁−1
𝑁 1
log 𝑝(𝐱; 𝜙) = − log(2𝜋𝜎 2 ) − 2 ∑[𝑥[𝑛] − 𝐴 cos(2𝜋𝑓0 𝑛 + 𝜙)]2
2 2𝜎
𝑛=0

Differentiating the log-likelihood function yields,

𝑁−1
𝜕 ln 𝑝(𝐱; 𝜙) 1
= − 2 ∑[𝑥[𝑛] − 𝐴 cos(2𝜋𝑓0 𝑛 + 𝜙)] 𝐴 sin(2𝜋𝑓0 𝑛 + 𝜙)
𝜕𝜙 𝜎
𝑛=0

𝑁−1
𝐴 𝐴
= − 2 ∑ [𝑥[𝑛] sin(2𝜋𝑓0 𝑛 + 𝜙) − sin(4𝜋𝑓0 𝑛 + 2𝜙)]
𝜎 2
𝑛=0

differentiating one more times gives

𝑁−1
𝜕 2 ln 𝑝(𝐱; 𝜙) 𝐴
2
= − 2 ∑[𝑥[𝑛] cos(2𝜋𝑓0 𝑛 + 𝜙) − 𝐴 cos(4𝜋𝑓0 𝑛 + 2𝜙)]
𝜕𝜙 𝜎
𝑛=0

substituting 𝑥[𝑛] produces

𝑁−1
𝜕 2 ln 𝑝(𝐱; 𝜙) 𝐴
2
= − 2 ∑[(𝐴 cos(2𝜋𝑓0 𝑛 + 𝜙) + 𝑤[𝑛]) cos(2𝜋𝑓0 𝑛 + 𝜙)
𝜕𝜙 𝜎
𝑛=0
− 𝐴 cos(4𝜋𝑓0 𝑛 + 2𝜙)]
 𝑁−1
𝐴
= − 2 ∑[𝐴 cos 2 (2𝜋𝑓0 𝑛 + 𝜙) + 𝑤[𝑛] cos(2𝜋𝑓0 𝑛 + 𝜙) − 𝐴 cos(4𝜋𝑓0 𝑛 + 2𝜙)]
𝜎
𝑛=0
𝑁−1
𝐴 1 1
= − 2 ∑ [𝐴 ( + cos(4𝜋𝑓0 𝑛 + 2𝜙)) + 𝑤[𝑛] cos(2𝜋𝑓0 𝑛 + 𝜙) − 𝐴 cos(4𝜋𝑓0 𝑛 + 2𝜙)]
𝜎 2 2
𝑛=0

after taking the negative expected value, we have

𝑁−1
𝜕 2 ln 𝑝(𝐱; 𝜙) 𝐴2 1 1 𝐴2 𝑁
−𝐸 [ ] = 2 ∑ [ − cos(4𝜋𝑓0 𝑛 + 2𝜙)] ≈
𝜕𝜙 2 𝜎 ⏟2 2 2𝜎 2
𝑛=0 1
≈ for 𝑓0 not near 0 or 1/2
2

Therefore,

2 𝜎2 2
var(𝜙̂) ≥ 2
=
𝑁𝐴 𝑁SNR

• An estimator which is unbiased and attains CRLB is called efficient.

• A MVU estimator may not be efficient.
• Figure 4.2 shows the difference between efficient and MVU estimators.

and CRLB
CRLB

(a) efficient and MVU (b) not efficient but MVU

Figure 4.2. Efficiency vs. minimum variance.

The CRLB may also be expressed in a slightly different form
2
𝜕 2 ln 𝑝(𝐱; 𝜃) 𝜕 ln 𝑝(𝐱; 𝜃)
−𝐸 [ 2
] = 𝐸 [( ) ]
⏟ 𝜕𝜃 ⏟ 𝜕𝜃
𝐼(𝜃):Fisher Inf. 𝐼(𝜃):Fisher Inf.

so that
 1 1
var(𝜃̂) ≥ 2 =
𝜕 ln 𝑝(𝐱; 𝜙) 𝜕 2 ln 𝑝(𝐱; 𝜃)
𝐸 [( ) ] −𝐸 [ ]
𝜕𝜃 𝜕𝜃 2

The CRLB is
1. Nonnegative
2. Additive for independent observations → CRLB for 𝑁 iid observations is 1/𝑁 𝛼 times
that for one observation

4.3 General CRLB for Signals in White Gaussian Noise

Assume that a deterministic signal with an unknown parameter 𝜃 is observed in WGN as
𝑥[𝑛] = 𝑠[𝑛; 𝜃] + 𝑤[𝑛] 𝑛 = 0,1, ⋯ 𝑁 − 1
The likelihood function is
𝑁−1
1 1
𝑝(𝐱; 𝜃) = 2 𝑁/2
exp {− 2 ∑(𝑥[𝑛] − 𝑠[𝑛; 𝜃])2 }
(2𝜋𝜎 ) 2𝜎
𝑛=0

Differentiating log-likelihood function once yields

𝑁−1
𝜕 ln 𝑝(𝐱; 𝜃) 1 𝜕𝑠[𝑛; 𝜃]
= 2 ∑(𝑥[𝑛] − 𝑠[𝑛; 𝜃])
𝜕𝜃 𝜎 𝜕𝜃
𝑛=0

second differentiation will give

𝑁−1 2
𝜕 2 ln 𝑝(𝐱; 𝜃) 1 𝜕 2 𝑠[𝑛; 𝜃] 𝜕𝑠[𝑛; 𝜃]
= ∑ (𝑥[𝑛]
⏟ − 𝑠[𝑛; 𝜃] ) − ( )
𝜕𝜃 2 𝜎2 𝜕𝜃 2 𝜕𝜃
𝑛=0 =𝑤[𝑛]

Taking the expectation produces

𝑁−1 2
𝜕 2 ln 𝑝(𝐱; 𝜃) 1 𝜕𝑠[𝑛; 𝜃]
𝐸[ ] = − ∑ ( )
𝜕𝜃 2 𝜎2 𝜕𝜃
𝑛=0

therefore

𝜎2
var(𝜃̂) ≥ 2 = CRB
𝜕𝑠[𝑛; 𝜃]
∑𝑁−1
𝑛=0 ( )
𝜕𝜃

• This form demonstrates the importance of the signal dependence on 𝜃.

• Signals that change rapidly as the unknown parameter changes result in more
accurate estimators.

Example: Sinusoidal Frequency Estimation

Suppose now that we would like to estimate the frequency of the signal represented by
 1
𝑠[𝑛; 𝜃] = 𝐴 cos(2𝜋𝑓0 𝑛 + 𝜃), 0 < 𝑓0 <
2
where the amplitude and the phase are known. The CRLB becomes

𝜎2
var(𝑓̂0 ) ≥
𝐴2 ∑𝑁−1
𝑛=0 (2𝜋𝑛 sin(2𝜋𝑓0 𝑛 + 𝜙))
2