Statistics 512 Notes 21: SuIIiciency and Its

Implications

SuIIiciency DeIinition: A statistic
1
( , , )
n
Y u X X =
is said
to be suIIicient Ior

iI the conditional distribution oI

1
, ,
n
X X
iven
Y y =
does not depend on

Ior any value

oI
y


Example 1:
1
, ,
n
X X
a sequence oI independent Bernoulli
random variables with
( 1)

! X = =

1
n

Y X
=
=
is suIIicient Ior

Example 2:
Let
1
, ,
n
X X
be iid UniIorm(
0,
) Consider the statistic
1
max
n
Y X
A A
=

We have shown beIore (see Notes 1) that
1
0
( )
0 elsewhere
n
n
Y
ny
y
f y

|
A A
=
,

For Y , we have
1 1
1 1
1 1
( , , , )
( , , , )
( )
1
1

n n
n n
Y n
n n
Y n
! X x X x Y y
! X x X x Y y
! Y y
I
ny ny
I

A

A
= = =
= = = =
=
= =

which does not depend on

For
Y
,
1 1
( , , , ) 0
n n
! X x X x Y y

= = = =


Thus, the conditional distribution does not depend on

and
1
max
n
Y X
A A
=
is a suIIicient statistic O

It is oIten hard to veriIy or disprove suIIiciency oI a
statistic directly because we need to Iind the distribution oI
the suIIicient statistic The Iollowin theorem is oIten
helpIul

Factorization Theorem: Let
1
, ,
n
X X
denote the a random
sample oI size n Irom a distribution that has pdI or pmI
( ; ) f x
,
O
A statistic
1
( , , )
n
Y u X X =
is suIIicient
Ior

iI and only iI we can Iind two nonneative Iunctions,

1

and
2

such that Ior all sample points

1
( , , )
n
x x
1 1 1 2 1
( ; ) ( ; ) | ( , , ); | |( , , )|
n n n
f x f x u x x x x =

where
2 1
|( , , )|
n
x x
does not depend upon

ProoI: We prove this Ior the discrete case (The prooI Ior
the eneral case is more subtle and requires reularity
conditions, but the basic ideas are the same) First, suppose
that the probability mass Iunction Iactors as iven in the
theorem We have
1
1
1 1 1
( , , )
1 2 1
( , , )
( ( , , ) ; ) ( , , ; )
| ; | |( , , )|
n
n
n n n
u x x u
n
u x x
! u X X u ! X x X x
u x x

=
= = = =
=

We then have
1
1 1 1
1 1 1
1
2 1
2 1
( , , )
|( , , ) , ( , , ) ; |
(( , , ), ( , , ) ; )

( ( , , ) ; )
|( , , )|
|( , , )|
n
n n n
n n n
n
n
n
u x x u
! X x X x u X X u
! X x X x u X X u
! u X X u
x x
x x

=
= = = =
= = =
=
=

Thus, the conditional distribution does not depend on

so
that
1
( , , )
n
Y u X X =
is suIIicient Ior

Now suppose that
1
( , , )
n
Y u X X =
is suIIicient Ior

,
meanin that the distribution oI
1
( , , )
n
X X
iven
1
( , , )
n
u X X
is independent oI

Let
1 1
| , | | ( , , ) ; |
n
u ! u X X u = =

2 1 1 1 1
|( , , )| |( , , ) , ( , , ) ; |
n n n n
x x ! X x X x u X X u = = = =

We then have
1 1
1 1 1 1 1 1
1 1 2 1
|( , , ); |
| ( , , ) ( , , ); | |( , , ) , ( , , ) ( , , )|
| ( , , ), | |( , , )|
n n
n n n n n n
n n
! X x X x
! u X X u x x ! X x X x u X X u x x
u x x x x

= = =
= = = = =

as was to be shown O

Example 1 Continued:
1
, ,
n
X X
a sequence oI
independent Bernoulli random variables with
( 1)

! X = =
To show that
1
n

Y X
=
=
is suIIicient Ior

, we Iactor the probability mass Iunction as Iollows:

1 1
1
1
1 1
1
( , , ; ) (1 )
(1 )
(1 )
1

n n


n

n
x x
n n

x n x
x
n
! X x X x

= =
=

= = =

=

+
=

' '
|

The pmI is oI the Iorm 1 2 1
1
( , ) ( , , )
n
n

x x x
=

where
2 1
( , , ) 1
n
x x =


Example 2 continued: Let
1
, ,
n
X X
be iid UniIorm(
0,
)
To show that
1
max
n
Y X
A A
=
is suIIicient, we Iactor the pdI
as Iollows:
1 1
1 0 max max 0
1
1 1
( , , ; )
n n
n
n x X X
n

f x x I I I

A A A A
A A A K
=
= =
|

The pdI is oI the Iorm
1
1 max 2 1
( , ) ( , , )
n
X n
I x x

A A
A

where
1 1
1 max 2 1 max 0
1
( , , ; ) , ( , , )
n n
n X n X
n
x x I x x I

A A A A
A K
= =


Example 3: Let
1
, ,
n
X X
be iid Normal (
2
, 3 9
) Althouh
the Iactorization theorem was stated explicitly Ior a one-
dimensional suIIicient statistic, it also applies to
multidimensional suIIicient statistics The pdI Iactors as
2 2
1 2
1
2
/ 2 2
1
2 2
/ 2 2
1 1
1 1
( , , ; , ) exp ( )
2
2
1 1
exp ( )
(2 ) 2
1 1
exp 2
(2 ) 2
n
n

n n

n n

n n

f x x x
x
x x n
3 9 3
9
9 6
3
9 6 9
3 3
9 6 9
=
=
= =

=

|

=

|

= +

|
|

The pdI is thus oI the Iorm

2 2
1 2 1
1 1
( , , , ) ( , , )
n n
n

x x x x 3 9
= =


where
2 1
( , , ) 1
n
x x =

Thus,
2
1 1
( , )
n n


x x
= =
is a two-dimensional suIIicient
statistic Ior
2
( , ) 3 9
, ie, the distribution oI
1
, ,
n
X X
is
independent oI
2
( , ) 3 9
iven
2
1 1
( , )
n n


x x
= =


Rao-Blackwell Theorem:

Theorem 731 (stated a little diIIerently): Let
1
, ,
n
X X
be
an iid sample Irom the pdI or pmI
( ; ) f x
,
O
Let
1
( , , )
n
u X X
be a suIIicient statistic Ior

Let
1
`
( , , )
n
W X X =
be an estimator oI

Because
1
( , , )
n
u X X
is a suIIicient statistic Ior

,
1
`
( , ( , , ) ; )
n
E u X X u = =

is a Iunction oI
1
, ,
n
X X

that is independent oI

The theorem is that Ior all

,
`
( ) ( ) $E $E

A

The inequality is strict unless
`
=

ProoI:
By the property oI iterated conditional expectations
(Theorem 231 (a)),
1
`
( ) ( , ( , , )) ( )
n
E E E u X X E



= =
|

ThereIore, to compare the two mean square error oI the two
estimators, we need only compare their variances From
the prooI oI Theorem 231(b), we have
1 1
` ` `
( ) ( , ( , , )) | ( , ( , , ))|
n n
Jar Jar E u X X E Jar u X X



= +
|


or
1
` `
( ) | ( , ( , , ))|
n
Jar Jar E Jar u X X



= +
|

Thus,
`
( ) ( ) Jar Jar

>

unless
1
`
( , ( , , )) 0
n
Jar u X X

=
Ior all
1
( , , )
n
u X X
, which is
the case only iI
`

is a Iunction oI
1
( , , )
n
u X X
, which
would imply
`
=

O

Since
1
`
( , ( , , ) ; )
n
E u X X u = =

is a Iunction oI the
suIIicient statistic
1
( , , )
n
u X X
, the Rao-Blackwell
theorem ives a stron rationale Ior basin estimators on
suIIicient statistics iI they exist II an estimator is not a
Iunction oI a suIIicient statistic, it can be improved

Further theoretical support Ior the maximum likelihood
estimator is provided by the Iact that it is a Iunction oI any
suIIicient statistic:

Theorem 732: Let
1
( , , )
n
X X
be an iid sample Irom a
distribution that has pdI or pmI
( ; ), f x O
II a
suIIicient statistic
1
( , , )
n
Y u X X =
Ior

exists and iI a
maximum likelihood estimator
`

oI

also exists uniquely,

then
`
E

is a Iunction oI the suIIicient statistic

1
( , , )
n
Y u X X =


ProoI: Usin the Iactorization theorem, the likelihood can
be written as
1 1 1 2 1
( ) ( , , ; ) | ( , , ), | |( , , )|
n n n
f x x u x x x x = =

To maximize this quantity, we need only maximize
1 1
| ( , , ), |
n
u x x
; thus
`
E

is a Iunction oI
1
( , , )
n
u X X
O