6.

2 Scheffes method for multiple contrasts

The Bonferroni adjustment applies very generally,
but it suffers from the limitation that the number of
simultaneous confidence intervals must be specified
in advance in order to maintain the validity of the
simultaneous coverage.

The Scheffe method is an alternative that controls the
simultaneous coverage over all contrasts in a
subspace. If a large number of contrasts are of
interest, or it is not known in advance which ones are
of interest, the Scheffe method provides a way to
snoop through all the possibilities while
maintaining control on the coverage of confidence
intervals and the false positive rates of tests.

This method is intimately connected with the F-test:
the F-test for the model is significant if and only if
some contrast in the model space has a significant
Scheffe confidence interval.

In particular, if the F-test is not significant, then none
of the Scheffe confidence intervals will be
significant.

Consider the linear model e y + = X with ) I , 0 ( ~
2
N e .

We will obtain simultaneous confidence intervals for

} | ' {

where is any set of vectors in the row space of X
(so all the ' are estimable).

Let

a denote the vector such that y a '

'

= (the least
squares estimate of ' . Note that all such vectors are
in (X) C = M , the column space of X.

Define the vector space M) M =

( } | {

a span

Then the Scheffe confidence intervals have the form

)

' (

'

se C
with
) , , 1 ( p n q F q C =

and ) (

= M rank q

Remark: Here the crucial multiplier of the standard
error depends not on the number of contrasts under
consideration but on the dimension of the space they
define.

The following result implies that the Scheffe intervals
control the coverage over all contrasts in the space.

Theorem 6.1: Under the linear model with
I) , 0 ( ~
2
N e and ) , , 1 ( p n q F q C =

we have


1 ) )

' ( | )

( ' (| all for se C P .

Proof: Let X = and note that

a a M =

. Then

) ( ' ) ( ' )

( '

= = M y a y a
and
|| || )

' (

a se = .

Consequently, using the Cauchy-Schwarz inequality,

2 2
2
2
2
|| ||
)} ( ' {
)

' (
)}

( ' {

a
M y a
se

) , ( ~

|| ||
|| ||
|| || || ||
2
2
2 2
2 2
p n q F q
M y
a
M y a

Since the upper bound is independent of ,

. 1 )

|| ||
( )
)

' (
| )

( ' |
max (
) )

' ( | )

( ' (|
2
2
2

=



C
M y
P C
se
P
all for se C P

Example: In the additive model for the r by c table,
suppose we are interested in controlling the level for
all contrasts in the row effects. Then M

is the space
of the columns of X for the row effects and has rank
r-1. In this case
2 / 1
)} 1 , 1 , 1 ( ) 1 {( + = c r n r F r C

.

Included in this set of contrasts would be all the
pairwise differences. Also included are any other
linear combinations the row effects with coefficients
summing to zero, i.e. any

r r
c c c + + + = ...
2 2 1 1
with 0 ...
2 1
= + + +
r
c c c .

For this reason the Scheffe intervals can be quite
conservative if we are only interested in a small
number of pairwise differences.



Example: Compare Scheffe and Bonferroni intervals
for all the pairwise differences in the Chicken feed
data.

> ## compare Bonferroni and Scheffe intervals
>
> sval <- sqrt(5*qf(.95, 5, 65))
> sval
[1] 3.432221
>
> round(
+ data.frame(C, s.lo=diff-sval*diffse,
+ s.hi=diff+sval*diffse,
+ bonf.lo=diff-bbval*diffse,
+ bonf.hi=diff+bbval*diffse),
+ digits=2)

X1 X2 X3 X4 X5 X6 s.lo s.hi bonf.lo bonf.hi
1 -1 1 0 0 0 0 -22.06 139.16 -13.02 130.12
2 -1 0 1 0 0 0 8.28 164.17 17.02 155.44
3 -1 0 0 1 0 0 34.45 198.97 43.67 189.75
4 -1 0 0 0 1 0 82.78 243.99 91.81 234.96
5 -1 0 0 0 0 1 88.11 249.32 97.14 240.29
6 0 -1 1 0 0 0 -46.38 101.74 -38.08 93.44
7 0 -1 0 1 0 0 -20.42 136.74 -11.62 127.94
8 0 -1 0 0 1 0 27.98 181.69 36.59 173.08
9 0 -1 0 0 0 1 33.31 187.02 41.92 178.41
10 0 0 -1 1 0 0 -45.37 106.33 -36.87 97.83
11 0 0 -1 0 1 0 3.09 151.22 11.39 142.91
12 0 0 -1 0 0 1 8.43 156.55 16.73 148.25
13 0 0 0 -1 1 0 -31.91 125.26 -23.10 116.45
14 0 0 0 -1 0 1 -26.58 130.59 -17.77 121.78
15 0 0 0 0 -1 1 -71.52 82.19 -62.91 73.58
>
> names(feedmn[ord])
[1] "feedhorsebean" "feedlinseed" "feedsoybean"
[4] "feedmeatmeal" "feedcasein" "feedsunflower"




With the Scheffe bounds we can consider other
contrasts as well and still maintain overall control on
the confidence level. E.g.,

> # a couple more contrasts (no cost
> # using Scheffe, but it would cost us
> # using Bonferroni
>
> C2 <- rbind(
+ c(-1,-1,-1,1,1,1),
+ c(-1,-1,-1,2,2,-1))
>
> C2
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] -1 -1 -1 1 1 1
[2,] -1 -1 -1 2 2 -1
>
> est <- C2 %*% feedmn[ord]
> se <- sqrt(diag(
+ C2 %*% feedcov[ord,ord] %*% t(C2)))
>
> data.frame(C2,
+ s.lo=est-sval*se,
+ s.hi=est+sval*se)

X1 X2 X3 X4 X5 X6 s.lo s.hi
1 -1 -1 -1 1 1 1 169.27944 438.7816
2 -1 -1 -1 2 2 -1 55.15912 438.2201


6.3 Tukey's method for paired comparisons

There are a variety of other more specialized methods
for multiple comparisons and multiple testing.
Among these Tukey's Honest Significant
Difference (HSD) method may be the most well
known.

This method applies to paired comparisons between
effect levels in balanced anova designs, either one-
way designs or additive multi-way designs.

The method requires a function to compute
percentiles of the following distribution.

Definition: Suppose
g
Z Z ,...,
1
are independent N(0,1)
and
2
~

U independently of the Z
i
's. Then

,
~
/
} min{ } max{
g
i i
T
U
Z Z
T

=

the studentized range distribution with parameters g
and .

Percentiles of this distribution can be computed in R
using the qtukey function.

Then in the balanced one-way design the Tukey
intervals for paired differences are

} / {
, , * *
N T y y
g i i i i




with g n = .

Exercise: show that in the balanced one-way anova,
with independent Gaussian errors and constant
variance, the Tukey pairwise confidence intervals
have simultaneous coverage of at least 1 .


6.4 False Discovery Rate

Rather than control the overall false positive rate of
multiple tests, the False Discovery Rate (FDR)
controls the rate of false positives among the
positives, i.e.

=
rejections of # Total
true is null when Rejections #
E FDR

So, if 05 . 0 FDR then approximately 5% at most of
the rejected hypotheses are falsely rejected.

Versions of this approach have become popular in the
context of gene expression microarray analysis where
thousands of genes are tested for expression under
varying experimental conditions.

In comparison with experimentwise error rate control
such as Bonferroni, the FDR approach is much less
conservative, allowing more false rejections in order
to allow more "discoveries" to emerge. The most
complete treatment of FDR is in the case of
independent tests. Dealing with correlated tests is an
area of ongoing research.