Statistics 550 Notes 21

Reading: Section 4.2

I. Neyman-Pearson Lemma (Section 4.2)
Deine t!e "i#e"i!ood ratio statistic $y
1
0 1
0
( % )
( & & )
( % )
p
L
p

X
X
X
&
'!ere
( % ) p x
is t!e (ro$a$i"ity mass )nction or
(ro$a$i"ity density )nction o t!e data X.
*!e statistic L ta#es on t!e +a")e

'!en
1 1
( % ) 0& ( % ) 0 p p > X X
and $y con+ention e,)a"s 0
'!en $ot!
1 1
( % ) 0& ( % ) 0 p p X X
.
-e ca""
k

a "i#e"i!ood ratio test i

0 1
0 1
0 1
1 i ( & & )
( ) i ( & & )
0 i ( & & )
k
L k
c L k
L k



>

'

<

x
x x
x
*!eorem 4.2.1 (Neyman-Pearson Lemma): .onsider
testing
0 0 0 0
: ( ( % ) ) H P X P
+s.
1 1 1 1
: ( ( % ) ) H P X P
(a) I 0 > and
k

is a si/e

"i#e"i!ood ratio test& t!en

k

is
a most (o'er)" "e+e"

test
1
($) 0or eac! 0 1 & t!ere e1ists a most (o'er)" si/e

"i#e"i!ood ratio test.

(c) I

is a most (o'er)" "e+e"

test& t!en it m)st $e a

"e+e"

"i#e"i!ood ratio test e1ce(t (er!a(s on a set A

satisying
0 1
( ) ( ) 0 P A P A

X X
21am("e 1:
1
& &
n
X X K
iid
( &1) N
.
0 0
: H
&
1 1
: H
& '!ere
0 1
<
.
*!e "i#e"i!ood ratio statistic is
2
1
1
0 1
2
0
1
2 2
1 0 1 0
1
2 2
1 0 1 0
1
1 1
e1( ( )
2
2
( & & )
1 1
e1( ( )
2
2
1
e1( ( ) ( )
2
e1( ( ) ( )
2
n
i
i
n
i
i
n
i
i
n
i
i
x
L
x
x
n
x

' ;

' ;



' ;



' ;

X
Re3ecting t!e n)"" !y(ot!esis or "arge +a")es o
0 1
( & & ) L X
is e,)i+a"ent to re3ecting t!e n)"" !y(ot!esis or
"arge +a")es o
1
n
i
i
X

()sing t!e act t!at

0 1
<
).
-!at s!o)"d t!e c)to $e4 *!e distri$)tion o
0
1
( )
n
i
i
X

)nder t!e n)"" !y(ot!esis is

(0& ) N n
so t!e
most (o'er)" "e+e"

tests re3ects or
2
0
1
( )
(1 )
n
i
i
X
n

>

'!ere is t!e .D0 o a

standard norma".
0or t!e data considered in Notes 20&
1 5
( & & ) X X K
(1.1054&
1.1556& -0.1502& 1.0747& -0.1089)& 'it!
0
: 0 H
+s.
1
: 1 H
& t!e most (o'er)" "e+e" 0.05 test re3ects or
1
( 0) : (1 0.05) 1.54
n
i
i
X n

>

. -e !a+e
1
( 0) : 1.75 1.54
n
i
i
X n

<

so 'e acce(t (retain)

0
: 0 H
.
21am("e 2:
P(X;x)
0 1 2 7 4
0
P
0.1 0.1 0.1 0.2 0.5
1
P
0.7 0.7 0.2 0.1 0.1
0 1
( & & ) L x
7 7 2 0.5 0.2
*!e most (o'er)" "e+e" 0.2 test re3ects i and on"y i X;0
or 1.
*!ere are m)"ti("e most (o'er)" "e+e" 0.1 tests& e.g.& 1)
re3ect t!e n)"" !y(ot!esis i and on"y i X;0< 2) re3ect t!e
n)"" !y(ot!esis i and on"y i X;1< 7) "i( a coin to decide
'!et!er to re3ect t!e n)"" !y(ot!esis '!en X;0 or X;1.
Proo o Neyman-Pearson Lemma:
7
(a) -e (ro+e t!e res)"t !ere or contin)o)s random
+aria$"es X. *!e (roo or discrete random +aria$"es
o""o's $y re("acing integra"s 'it! s)ms.
Let
=
$e t!e test )nction o any ot!er "e+e"

test $esides
k

. >eca)se
=
is "e+e"

&
0
=
P
E
. -e 'ant to s!o'
t!at
=
1 1
( ) ( ) ( ) ( )
k
dP dP

x x x x
.
-e e1amine
=
1 0
( ( ) ( ))( ( ) ( ))
k
p kp d

x x x x x
and s!o'
t!at
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
p kp d

x x x x x
. 0rom t!is&
'e conc")de t!at
= =
1 0
( ( ) ( )) ( ) ( ( ) ( )) ( ))
k k
p kp d

x x x x x x x
*!e "atter integra" is 0 $eca)se
=
0 0
( ) ( ) & ( ) ( )
k
p d p d

x x x x x x
?ence& 'e conc")de t!at
=
1
( ( ) ( )) ( ) 0
k
p

x x x
as
desired.
*o s!o' t!at
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
p kp d

x x x x x
& "et
=
=
0 =
@ : ( ) ( )A
@ : ( ) ( )A
@ : ( ) ( )A
k
k
k
S
S
S



+

>
<

x x x
x x x
x x x
S)((ose S
+
x . *!is im("ies
( ) 0
k
> x
'!ic! im("ies
t!at
1 0
( ) ( ) p kp x x
. *!)s&
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
S
p kp d
+

x x x x x
.
B"so& simi"ar"y&
4
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
S
p kp d

x x x x x
and
0
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
S
p kp d

x x x x x =
(since
1 0
( ) ( ) p kp x x
or
0
S x ).
*!)s&
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
p kp d

x x x x x
and t!is s!o's t!at
=
1
( ( ) ( )) ( ) 0
k
p

x x x
as arg)ed a$o+e.
($) Let
0 1 0 0
( ) C ( ) ( )D 1 ( ) c P p cp F c > x x
'!ere
0
F
is t!e
cd o
1
0
( )
( )
p
p
x
x
)nder
0
P
. >y t!e (ro(erties o .D0s&
( ) c
is
nonincreasing in c and rig!t contin)o)s.
>y t!e rig!t contin)ity o
( ) c
& t!ere e1ists
0
c
s)c! t!at
0 0
( ) ( ) c c


. So deine
1
0
0
0 1
0
0 0 0
1
0
0
( )
1 i
( )
( ) ( )
( ) i
( ) ( ) ( )
( )
0 i
( )
p
c
p
c p
x c
c c p
p
c
p

>

'

<

x
x
x
x
x
x

*!en&
5
0
0 1 1
0 0 0 0
0 0 0 0
0
0 0 0
0 0
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( ) ( )
( )
( ) C ( ) ( )D
( ) ( )

P
c p p
E P c P c
p c c p
c
c c c
c c

> +

x x
x
x x
So 'e can ta#e k to $e
0
c
.
(c) Let
=
$e t!e test )nction or any most (o'er)" "e+e"

test. >y (arts (a) and ($)& a "i#e"i!ood ratio test

k

'it!
si/e

can $e o)nd t!at is most (o'er)". Since

=
and
k

are $ot! most (o'er)"& it o""o's t!at

=
1
( ( ) ( )) ( ) 0
k
p

x x x
0o""o'ing t!e (roo in (art (a)& im("ies t!at
=
1 0
( ( ) ( ))( ( ) ( )) 0
k
p kp d

x x x x x
'!ic! can $e t!e case i and on"y i
=
( ) ( ) 0
k
x x
'!en
1 0
( ) ( ) p kp < x x
(i.e.&
0 1
( & & ) L k < X
) and
=
( ) ( ) 1
k
x x
'!en
1 0
( ) ( ) p kp > x x
(i.e.&
0 1
( & & ) L k > X
) e1ce(t on a set A satisying
0 1
( ) ( ) 0 P A P A

X X
.
.onnection o "i#e"i!ood ratio tests to >ayes tests: .onsider
t!e 0-1 "oss or !y(ot!esis testing. *!e >ayes test c!ooses
5
1 1
: H
o+er
0 0
: H
i
1
0
( % )
1
( % )
p
p

>
X
X
& '!ic! is
e,)i+a"ent to
0
0 1
0
( )
( & & ) 1
1 ( )
L



>

x
. *!e "i#e"i!ood ratio
test
0 1
0 1
0 1
1 i ( & & )
( ) i ( & & )
0 i ( & & )
k
L k
c L k
L k



>

'

<

x
x x
x
is a >ayes test or t!e (rior
0
1
( )
1 k

+
& $eca)se or t!is
(rior
1
0 1
0
( % ) 1
( & & )
( % )
p
L
p k

X
x
X
is greater t!an 1 i and on"y
i
0 1
( & & ) L k > x
.
*!e dierence $et'een t!e >ayes a((roac! and t!e
Neyman-Pearson a((roac! is t!at t!e >ayes a((roac! starts
'it! a (rior and t!is determines t!e c)to k or t!e
"i#e"i!ood ratio test& '!i"e t!e Neyman-Pearson a((roac!
starts 'it! a signiicance "e+e" (a ma1im)m acce(ta$"e
*y(e I error rate) and t!is determines t!e c)to k or t!e
"i#e"i!ood ratio test.
II. Eniorm"y Fost Po'er)" *ests (Section 4.7)
-!en t!e a"ternati+e !y(ot!esis is com(osite&
1 1
: H
&
t!en t!e (o'er can $e dierent or dierent a"ternati+es.
0or eac! (artic)"ar a"ternati+e
1

& a test is t!e most

8
(o'er)" "e+e"

test or t!e a"ternati+e

1

i t!e test is most

(o'er)" or t!e sim("e a"ternati+e
1 1
: H
.
I a (artic)"ar test )nction
=
( ) x is t!e most (o'er)" "e+e"

test or a"" a"ternati+es

1

& t!en 'e say t!at
=
( ) x is
a uniformly most powerful (EFP) "e+e"

test and 'e

s!o)"d c"ear"y )se
=
( ) x as o)r test )nction )nder t!e
Neyman-Pearson (aradigm.
In notation& "et
( & )
denote t!e (o'er o a test

at t!e
a"ternati+e . B "e+e"

test
=
is EFP i
=
1
( & ) ( & ) or a""
or any ot!er "e+e"

test

.
21am("e o )niorm"y most (o'er)" test:
Let
1
& &
n
X X K
$e iid
( &1) N
and s)((ose 'e 'ant to test
0 0
: H
+ers)s
1 0
: H >
. 0or eac!
1 0
>
& t!e most
(o'er)" "e+e"

test o
0 0
: H
+ers)s
1 1
: H

re3ects t!e n)"" !y(ot!esis or
0
1
( )
(1 )
n
i
i
X
n

>

.
Since t!is same test )nction is most (o'er)" or eac!
1 0
>
& t!is test )nction is EFP.
>)t s)((ose 'e consider t!e a"ternati+e !y(ot!esis
1 0
: H
. *!en t!ere is no EFP test. *!e most
(o'er)" test or eac!
1 1
: H
& '!ere
1 0
>
& re3ects
6
t!e n)"" !y(ot!esis or
0
1
( )
(1 )
n
i
i
X
n

>

& $)t t!e

most (o'er)" test or eac!
1 1
: H
& '!ere
1 0
<
&
re3ects or
0
1
( )
( )
n
i
i
X
n

<

. Note t!at t!e test t!at

re3ects t!e n)"" !y(ot!esis or
0
1
( )
(1 )
n
i
i
X
n

>

cannot $e most (o'er)" or an a"ternati+e
1 0
<
$y (art
(c) (necessity) o t!e Neyman-Pearson Lemma since it is
not a "i#e"i!ood ratio test or
1 0
<
.
It is rare or EFP tests to e1ist. ?o'e+er& or one-sided
a"ternati+e& t!ey e1ist in some (ro$"ems.
B condition )nder '!ic! EFP tests e1ist is '!en t!e
ami"y o distri$)tions $eing considered (ossesses a
(ro(erty ca""ed monotone likelihood ratio.
Deinition: *!e one-(arameter ami"y o distri$)tions
@ ( % ) : A p x
is said to $e a monotone "i#e"i!ood ratio
ami"y in t!e one-dimensiona" statistic
( ) T x
i or
0 1
<
(a)
0 1
( % ) ( % ) p p x x
or a"" x (identiia$i"ity)
($)
1
0
( )
( )
p
p

x |
x |
is an increasing )nction o
( ) T x
.
21am("es o ami"ies 'it! monotone "i#e"i!ood ratio:
(a) 0or
1
& &
n
X X K
iid 21(onentia" ( )&
9
1
0
1
1 1 1
1 0
1
0 0
0
1
( )
e1( ( )
( )
i
i
n
X
n
n
i
i
n
X
i
i
e
p
X
p
e

_
_



,
,

x |
x |
0or
1 0
>
&
1
0
( )
( )
p
p

x |
x |
is an increasing )nction o
1
n
i
i
X

so t!e ami"y !as monotone "i#e"i!ood ratio in

1
( )
n
i
i
T X

x
.
($) .onsider t!e one-(arameter e1(onentia" ami"y mode"
( ) ( ) e1(@ ( ) ( ) ( )A p h T x | x x
I
( )
is strict"y increasing in & t!en t!e ami"y is
monotone "i#e"i!ood ratio in
( ) T x
. I
( )
is strict"y
decreasing in & t!en t!e ami"y is monotone
"i#e"i!ood ratio in
( ) T x
.
21am("e 2: Let
G >inomia"( & )& 0 1 X n < <
.
( % ) (1 )
e1(C "og "og(1 )D
1
x n x
n
p x
x
n
x n
x

_


,
_
_
+

,
,

*!is is a one-(arameter e1(onentia" ami"y 'it!
10
( ) "og & ( ) "og(1 )& ( ) & ( )
1
n
n T x x h x
x

_
_

,
,
Since
( )
is strict"y increasing in & t!e ami"y is
monotone "i#e"i!ood ratio in
( ) T x x
.
*!eorem (4.7.1 and .oro""ary): I t!e one-(arameter ami"y
o distri$)tions
@ ( % ) : A p x
!as monotone "i#e"i!ood
ratio in
( ) T x
& t!en t!ere e1ists a EFP "e+e"

test or
testing
0 0
: H
+ers)s
1 0
: H >
and it is gi+en $y
=
1 i ( )
( ) i ( )
0 i ( )
T c
T c
T c

>

'

<

x
x x
x
'!ere c and

are determined so t!at

0
=
( ) E

x
.
Proo: 0i1 an a"ternati+e
1 0
>
. >eca)se t!e ami"y is
monotone "i#e"i!ood ratio in
( ) T x
& t!e "i#e"i!ood ratio
statistic
1
0 1
0
( % )
( & & )
( % )
p
L
p

x
x
x
is an increasing )nction o
( ) T x
. *!e Neyman-Pearson
"emma gi+es t!at
=
( ) x is a most (o'er)" test or testing
0 0
: H
+ers)s
1 1
: H
. Since t!is !o"ds or a""
1 0
>
& 'e !a+e t!at
=
( ) x is EFP.
Note: Not a"" one-(arameter ami"ies !a+e monotone
"i#e"i!ood ratio. 0or e1am("e& t!e .a)c!y distri$)tion is
not monotone "i#e"i!ood ratio in x:
11
2
1 1
( % )
1 ( )
p x
x

+
.
0or 0 > &
2
2
( % ) 1
( % 0) 1 ( )
p x x
p x x

+
&
'!ic! is not increasing in x.
No' consider
0 0
: H
+ers)s
1 0
: H >
. Is t!e a$o+e
test
=
( ) x sti"" EFP4
0act: *!e test
=
( ) x is "e+e"

or
0 0
: H
(i.e.& or
e+ery
0

& t!e (ro$a$i"ity o re3ection is at most

).
>eore (ro+ing t!is act& 'e (ro+e .oro""ary 4.2.1.
.oro""ary 4.2.1: I
=
is a most (o'er)" "e+e"

test o
0 0
: H
+s.
1 1
: H
& t!en t!e (o'er o
=
at
1

is
greater t!an or e,)a" to

'it! e,)a"ity i and on"y i

0 1
( % ) ( % ) p p x x
'it! (ro$a$i"ity one ()nder $ot!
0
H

and
1
H
).
Proo: *!e test
( ) x
or a"" x (i.e.& re3ect 'it!
(ro$a$i"ity

regard"ess o x) !as "e+e"

and (o'er

.
>y t!e necessity (art o t!e Neyman-Pearson "emma& i
( ) x
'ere most (o'er)"& t!en
1
0
( % )
( % )
p
k
p

X
X
'it!
12
(ro$a$i"ity one. *!ereore&
1 0
( % ) ( % ) p kp X X
'it!
(ro$a$i"ity one '!ic! im("ies t!at k;1 and conse,)ent"y
t!at
1 0
( % ) ( % ) p p X X
'it! (ro$a$i"ity one.
Proo o act: Let
= =
( & ) C ( )D E

x
. -e 'ant to s!o'
=
( & ) or
0

. 0i1
2 0
<
. *!en
( ) x
is t!e
most (o'er)" test or testing
0 2
: H
+s.
1 0
: H
at
"e+e"
=
2
( & )
. >)t rom .oro""ary 4.2.1&
= =
0 2
( & ) ( & )
.
>ac# to t!e ,)estion o is
=
( ) x EFP "e+e"

or testing
0 0
: H
+ers)s
1 0
: H >
4
Hes& it is EFP. Proo: .onsider a "e+e"

test

or
0 0
: H
.

m)st a"so $e a "e+e"

test or
0 0
: H
. *!en& $eca)se
=
( ) x is EFP "e+e"

or testing
0 0
: H
+ers)s
1 0
: H >
&
=
0
( & ) ( & ) or a""
.
S)mmary statement o *!eorem 4.7.1 and .oro""ary:
I t!e one-(arameter ami"y o distri$)tions
@ ( % ) : A p x
!as monotone "i#e"i!ood ratio in
( ) T x
&
t!en t!ere e1ists a EFP "e+e"

test or testing
0 0
: H

+ers)s
1 0
: H >
and it is gi+en $y
17
=
1 i ( )
( ) i ( )
0 i ( )
T c
T c
T c

>

'

<

x
x x
x
'!ere c and

are determined so t!at

0
=
( ) E

x
.
21am("e 2 contin)ed: 0or
G >inomia"( & ) X n
and testing
0 0
: H
+s.
1 0
: H >
& t!e EFP "e+e"

test is
=
1 i
( ) i
0 i
x c
x x c
x c

>

'

<

'!ere
c
is t!e sma""est integer s)c! t!at
0
( ) P X c

> <
and
0 0
C ( )D : ( ) P X c P X c

>
.
14