Classical Estimation Classical Estimation

Chapter 2
Estimator Performance Metrics Estimator Performance Metrics
Mean of the estimator (bias)
Variance of the estimator
Minimum Variance Unbiased
(MVU) Estimation
UNBI ASED O h i ill i ld UNBI ASED On t he average est imat or will yield
t rue value or
O O = O
|
all for )

( E
EXAMPLE: DC level in WGN ( zero- mean)
|
value true
A - on values can take A
1 - 0,1,...N n WGN ] [ ] [
< <
= + = n W A n X
X[n] 1/N A

1 - N
0 n
=

=
( ) UNBIASED A all for X[n] E 1/N ) A

E(
1 - N
0 n
= =

=
A
Bias Bias
Example : same setup as previous example but p p p p
consider the estimator
2 / ] [ ) (
1
1
A A E n x A
N
= =
v

Bias of estimator :
2 / ] [ ) (
2
0
A A E n x
N
A
n
= =
=

Bias of estimator :
u u u =
.
) ( ) ( E b
Generally seek unbiased estimator (necessary but not
sufficient for good estimator)
Minimum Variance Estimation Minimum Variance Estimation
Mean Square Error (MSE) : Mean Square Error (MSE) :
] ) [( ) (
2
u u u =
. .
E MSE
Results in unrealizable estimator (for classical
but not Bayesian estimators more later )
] ) [( ) ( u u u E MSE
but not Bayesian estimators, more later.)
] )] ) ( ( )) ( [[( ) (
2
u u u u u + =
. . . .
E E E MSE
)] )

( ))(

( [( 2 ] ) ) ( [( ] )) ( [(
2 2
u u u u u u u u + + =
. . . .
E E E E E E E
u u )

( E is constant and comes out of the Expectation, hence last term is 0

Minimum Variance Unbiased Estimation
2
OF FUNCTION STRONG
) ( )

( b VAR
O =
O + O =
k ! t d ' hi h
on dependent

a produce ation will Differenti

OF FUNCTION STRONG
O O
O =
1 - N
X[n]
N
1
X X a A : Example
know! t don' which we
= =
v

0 n
? a" " best Find
[ ]
N
p
=

(allowing biased estimators)

2 2 2
2 2
) ( ) / ( ) (
/N a ) A Var( aA ) A E( = =
v
v v
o
2 2 2
) ( ) / a ( ) A MSE( A aA N + = o
Minimum Variance Unbiased Estimation
0 A)A - 2(aA /N 2
2
a
da
dMSE
o = + =
A! on depends a
2
2
2
opt
A
A
da
o
=
2
N
A
o
+
Generally, in classical est imat ion, we const rain t he
bias t o be zero ( t o get a realizable est imat or) and
t hen minimi e it s a iance ( hich is same as t hen minimize it s variance ( which is same as
minimizing MSE)

Minimum Variance Unbiased ( MVU) Est imat or

Existence of MVU Estimator

DOES MVU EXI ST?
values all for variance
smallest have must

u
)

var(u
2

u
MVU =

u
Estimator)
MVU Uniformly ( . of
values all for variance
u
1
u
MVU =
3
u
u
exist. not will general, In
1

u
0 1) N( X[1]
,1) N( X[0]
: EXAMPLE
> ~
~
u u
u
No MVU !
)

var(u
2

u
3

u
are X[1] and X[0]
0 ,2) N(
0 ,1) N( X[1]
<
> ~
u u
u u
3
u
0
u
t Independen
[ ] [ ]
Example (Contd)
( ) ] 1 [ ] 0 [
2
1

1
+ = u X X
Consider
] 1 [
3
1
] 0 [
3
2

2
+ = u X X
Unbiased ? Yes !
1 4
]) 1 [ (
4
1
]) 0 [ (
4
1
)

(
1
+ = u X VAR X VAR VAR
1 ]) 0 [ (
]) 1 [ (
9
1
]) 0 [ (
9
4
)

(
2
=
+ = u
X VAR
X VAR X VAR VAR
0 2
0 1 ]) 1 [ (
1 ]) 0 [ (
< =
> =
u
u X VAR
X VAR
0 2 < u
Example (Contd)
)

(u VAR
36 / 27
36 / 24
2

u
36 / 24
36 / 20
36 / 18
1

u
36 / 18
u
BETTER

2
u BETTER

1
u
0? for

and 0 for

choose not Why

2 1
< > u u u u 0? for and 0 for choose not Why
2 1
< > u u u u
Example (Contd)
possible smallest are 0 for
)

VAR( and 0 for )

VAR( : that show also Can

2 1
u
u u u
<
>
ance lower vari with

other No
possible smallest are 0 for
u
u

<
Exercise : prove this result for the class of estimators
HOW DO WE FIND MVU?
(IF IT EXISTS)
APPROACHES:
1) Comput e Cramer Rao lower Bound on Variance.
Find t he Est imat or t hat sat isfies it ( Chapt er 3)

u
)

(u VAR
2
u
1

u
u
CRLB
When CRLB sat isfied wit h equalit y for all
MVU Est imat or
u

MVU Est imat or

Note : it may happen that no estimator exists whose variance equals CRLB
yet MVU still exists (as in picture above)
HOW DO WE FIND MVU?
2) Apply Rao Blackwell Lehmann Scheffe Theorem 2) Apply Rao- Blackwell- Lehmann- Scheffe Theorem
Find sufficient st at ist ic and make it Unbiased
MVU Est imat or ( Chapt er 5)
3) Rest rict Est imat or t o be Linear funct ion of dat a
Best Linear Unbiased Est imat or or MVU for All

Linear Est imat ors ( Chapt er 6)

MVU FOR VECTOR PARAMETER
parameters of vector 1 p
2
1

(
(
(

= u
u
u
Letting or )

E( if Unbiased
p p
2
(
(
(

p
u u
u

) (
) (
Y
X
E
Letting or ) E( if Unbiased
(

=
(

=
i i
Y E
X E
u u
( )

) ( Y
1 1
(
(
(

(
(
(

E
Y E
u u
( ) D UNBIASE
)

(
) (
2 2
=
(
(
(

=
(
(
(

=
p
p
E
E E u
u
u
u
u u

Minimum is 1,2,...p i )

VAR( means variance Minimum =

i
u