Professional Documents
Culture Documents
STAT 426: Fall 2012
STAT 426: Fall 2012
Lecture 20
Fall 2012
Arash A. Amini
1 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
2 / 37
X1 , . . . , Xn Ber()
I
n
Y
i=1
xi (1 )1xi =
xi
(1 )n
xi
P
= g ( i xi , ).
I
By factorization theorem, T (X ) =
Pn
i=1
Xi is sufficient for .
3 / 37
4 / 37
P (Xi = 1) = 1 (1 )2 = (2 ).
We have Xi Ber (2 ) for i = 2, 4, . . . , n.
The joint pmf is
Y
Y
xi
1xi
p (x) =
xi (1 )1xi
(2 )
(1 )2
.
i even
i odd
Let t1 :=
i odd xi
and t2 :=
i even xi .
Then,
We have
p (x) = t1 +t2 (2 )t2 (1 )3kt1 2t2
= g (t1 , t2 , )
We conclude
that (T1 , T2 ) is sufficient where T1 =
P
T2 = i even Xi .
i odd
Xi and
X = (0 1 1 1 1 0 0 1 0 1 1 1 0 0 1 1 1 0 0 0)
It is enough to keep (T1 , T2 ) = (5, 6) and discard the sequence, if we only
care about estimating .
I
In an intuitive sense, with this model, there is more information in the same
sequence for estimating than just its number of ones.
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
7 / 37
I
I
X1 ,
n
X
Xi
i=2
I
I
8 / 37
Minimal sufficiency
I
I
I
I
I
9 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
10 / 37
11 / 37
n
Y
i=1
f (xi ) =
n
Y
f (x(i) )
i=1
Concrete example:
f (2, 0.4, 1.5) = f (2)f (0.4)f (1.5)
12 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
13 / 37
I
I
I
14 / 37
I
I
I
h(x) = 1{1x1}
2 sinh()
The density is
h
2 sinh i
f (x) = exp x log
1{1x1}
exp(x)1{1x1}
15 / 37
0.5
0
2
16 / 37
Say we take
T (x) = (x, x 2 ),
h(x) = 1{1x1}
17 / 37
2
1.5
1
0.5
0
2
18 / 37
I
I
T (x) = (u 2 , v 2 , u, v , uv ),
I
h(x) = 1{0u,v 1}
The density is
f (x) exp 1 u 2 + 2 v 2 + 3 u + 4 v + 5 uv 1{0u,v 1}
19 / 37
0
1
1
0.5
v
0 0
(a)
0.5
u
0
1
1
0.5
v
0 0
0.5
u
(b)
20 / 37
= (2, -2, 2, 2, 1)
= (-2, -2, 2, 2, 1)
1.5
5
1
0.5
1
0.5
0 0
v
(c)
0.5
u
0
1
1
0.5
v
0 0
0.5
u
(d)
21 / 37
6
10
0
1
1
0.5
0 0
v
(e)
0.5
u
0
1
1
0.5
v
0 0
0.5
u
(f)
22 / 37
= (1, 0, 0, 0, -0.5)
1
0.5
0.5
1
01
11
v
(g)
1
1
11
(h)
23 / 37
= (0, 0, 0, 0, -1.5)
0.4
0.5
0.2
1
1
11
v
(i)
1
01
11
(j)
24 / 37
= (0, -0.5, 0, 0, 1)
0.4
0.5
0.2
1
1
11
v
(k)
1
01
11
(l)
25 / 37
Can be rewritten as
x = 0, 1, 2, . . .
1
p (x) = exp x log
x!
|{z}
h(x)
1
exp(x e )
x!
26 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
27 / 37
MT (t) := E [e T (X ) ] = e A(t+)A() .
I
28 / 37
Cumulants m (T ) =
dm
dt m KT (t) |t=0 .
1 (T ) = E [T ] = A0 ()
2 (T ) = var [T ] = A00 ()
I
29 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
30 / 37
Consider the 1-param canonical case f (x) = h(x) exp[T (x) A()].
Joint density base on n samples,
n
n X
oY
f (x) = exp
T (xi ) nA()
h(x)
i=1
I
I
T (Xi ),
MLE coincides with MOM
31 / 37
Uniform[0, ].
32 / 37
Outline
Sufficiency
I
I
I
I
Exponential families
I
I
I
Back to sufficiency
I
I
33 / 37
34 / 37
Rao-Blackwell Theorem
Theorem (Rao-Blackwell)
I
I
I
I
35 / 37
Proof of Rao-Blackwell
I
I
e = E [].
b
E []
Since MSE = var + (bias)2 , need to compare variances only.
Law of total variance implies
b = var E(|T
b ) .
b ) + E var(|T
var ()
| {z }
|
{z
}
e
We are done.
36 / 37
Rao-Blackwell example
I
I
I
I
I
I
I
iid
Now T = E [T |T ] =
Pn
j=1
E [Xj |T ] = nh(T ).
We have
3
e = E [X1 |T ] + 2E [X5 |T ] = 3h(T ) = T .
n
37 / 37