Multiple Comparison Procedures: The Practical Solution

Author(s): D. J. Saville
Source: The American Statistician, Vol. 44, No. 2 (May, 1990), pp. 174-180
Published by: American Statistical Association
Stable URL: http://www.jstor.org/stable/2684163 .
Accessed: 17/06/2014 13:51

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to The
American Statistician.

http://www.jstor.org

This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM

All use subject to JSTOR Terms and Conditions
 OMENT
Commentariesare informativeessays dealing with viewpoints of statis-
tical practice, statistical education, and other topics considered to be
of general interest to the broad readershipof The American Statistician.
Commentaries are similar in spirit to Letters to the Editor, but they
Multiple Comparison Procedures: The Practical Solution
D. J. SAVILLE*
A practicing statistician looks at the multiple comparison
controversyand relatedissues throughthe eyes of the users.
The concept of consistency is introducedand discussed in
relation to five of the more common multiple comparison
procedures.All of the proceduresare found to be inconsis-
tent except the simplest procedure, the unrestrictedleast
significant difference (LSD) procedure(or multiple t test).
For this and other reasons the unrestrictedLSD procedure
is recommended for general use, with the proviso that it
should be viewed as a hypothesis generatorratherthan as
a method for simultaneoushypothesis generationand test-
ing. The implicationsfor Scheff6's test for generalcontrasts
are also discussed, and a new recommendationis made.
KEY WORDS: Comparisonwiseerrorrate;Duncan's mul-
tiple rangetest; Experimentwiseerrorrate;Power;Teaching
of statistics; Tukey's honest significant difference proce-
dure; Waller-Duncan k-ratiotest.
1. INTRODUCTION
Proceduressuch as the least significant difference (LSD)
test, Duncan's multiplerangetest, and Wallerand Duncan's
k-ratio LSD test are used by applied researchersfor the
analysis of data from experimentsin which the treatments
have no easily definable structure.These and similar pro-
cedures, referredto generally as multiple comparisonpro-
cedures, have been the subject of controversyfor over half
a century.
Many statisticiansdislike the idea of simultaneouslytest-
ing a multitude of unplannedand interrelatedhypotheses,
and some question the usefulness of multiple comparison
procedures(e.g., Nelder 1971; Plackett 1971; Preece 1982).
In spite of this, multiple comparison procedurescontinue
to be widely used and sometimes misused by researchers.
*D. J. Saville is Biomnetrician,Ministry of Agricultureand Fisheries,
P.O. Box 24, Lincoln, Canterbury,New Zealand. The authoris grateful
to F. JacksonHills for stimulatingthe work and to PeterHeffernan,Harold
Henderson, Mike Ryan, Chris Dyson, and Karen Baird for constructive
criticism. Samuel G. Carmeris also thankedfor his early encouragement
and for assistance with the preparationof this article, including the com-
putation of the equivalent significance levels in Section 5. The ideas in
the article were presented at the thirteenthInternationalBiometric Con-
ference in Seattle in 1986.
174 The American Statistician, May 1990, Vol. 44, No. 2
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
involve longer discussions of background, issues, and perspectives. All
commentarieswill be refereed for their merit and compatibilitywith these
criteria.
Theirmisuse in experimentsin which the treatmentspossess
a factorial structure,or are quantitativein nature,has been
highlighted by many excellent papers in applied science
journals(e.g., Chew 1980; Cousens 1988; Little 1978;Perry
1986; Petersen 1977). These writers have rightly pointed
out the greaterappropriatenessof the methodsof orthogonal
contrasts and regression analysis for the analysis of data
from such experiments.
In most researchwork, the objectives are sufficientlywell
defined that the questions of interestcan be answeredusing
appropriatecontrasts. In a minority of cases, such as pes-
ticide screening trials or cultivar evaluation trials, a least
significant difference is useful as a yardstickwith which to
assess the strengthof evidence for any particularpairwise
difference. In these cases I recommendusage of the simplest
multiple comparison procedure, the multiple t test, or un-
restrictedLSD procedure.
The purpose of this article is to outline some of the rea-
soning behind this recommendation.Particularattentionis
paid to the "inconsistency" of the alternativesto the un-
restrictedLSD procedure.This refersto the fact thata given
procedurecan returna verdict of not significant for a given
difference in one experiment, but return a verdict of 1%
significant for the same difference in a second experiment,
with no change in the standarderrorof the differenceor the
numberof errordegrees of freedom.
The formatof the article is to define the unrestrictedLSD
procedure(Sec. 2), set the scene by discussing its practical
usage (Sec. 3), and discuss the common objections to its
use (Sec. 4). The notion of inconsistencyis then introduced
and discussed, with examples of the inconsistency of Fish-
er's restricted LSD procedure, Tukey's honest significant
difference (HSD) procedure, and Waller and Duncan's k-
ratioLSD procedure(Sec. 5). In a discussion (Sec. 6), three
main points are covered. First, the advantagesof the un-
restrictedLSD procedureare summarized.Second, the dis-
tinction between hypothesis generation and testing is
highlighted, leading to the suggestion that multiple com-
parison procedures should be treated as hypothesis gener-
atorsratherthan simultaneousgeneratorsand testers. Third,
the implicationsfor the analysis of general, unplannedcon-
trasts are spelled out, leading to a recommendationof the
usual F or t test instead of the inconsistentScheffe test. To
summarize, the "practicalsolution" is given in Section 7.
(C 1990 AmericanStatistical Associationi
 2. DEFINITION OF LSD
The 5%-level "multiple t test" consists of all pairwise
comparisonsof the populationmeans using 5%-level t tests
based on a pooled variance estimate, s2. That is, for each
pair of populations i andj, the absolute value of
t = (Yi- )N2s2n = difference/SE(difference)
is compared with the 97.5 percentile, or 5% critical value,
of the tv distribution, tv(.975), where v is the number of
degrees of freedom associated with the variance estimate
s2, ji and y are the ith andjth sample means, and n is the
common sample size; SE representsstandarderror.
This is equivalentto comparing jY - yj with the quantity
tv(.975) x SED, where SED is the standarderror of the
difference between two means. This quantityis thus known
as the 5%-level unrestrictedleast significant difference, or
LSD(5%). The name unrestrictedLSD procedure is used,
since there is no restrictionrequiringa significant prelim-
inary overall F test.
3. PRACTICAL USAGE
In some teaching institutions, proceduresfor the testing
of hypotheses are presentedas "black and white" in char-
acter; the significance level for the test is decided before
the data are examined, and the test yields a clear-cut de-
cision-yes, the hypothesis is true, or no, the hypothesis
is false.
In practice, most statisticiansrecognize thatthereare also
shades of gray, and many adopt the alternativestrategyof
measuringthe strengthof the evidence againsta hypothesis
by varying the significance level of the test and quoting the
most stringenttest for which the hypothesis is rejected. In
the present context, this means that critical values such as
the LSD(10%), LSD(5%), LSD(1%), and LSD(.1%) are
used as distances on a yardstickthat measure the strength
of the evidence against a hypothesisHo: 0 i = /.
To put the usage of the unrestrictedLSD procedurein
perspective, we now presenttwo sample data sets. The first
sample data set consists of the results from a typical weed-
control experiment (Table 1). In this case, the main ques-
tions of interest are answered by the testing of three or-
thogonal contrasts.The LSD(5%)is includedto indicatethe
Table 1. Report on a Typical Weed-Control Experiment
Treatment
1. Control
2. Coded chemical, half-normal rate (.5N)
3. Coded chemical, normal rate (N)
4. Coded chemical, twice-normal rate (2N)
5. Standard chemical, normal rate (N)
LSD(5%)
Contrasts
Control versus treated (treatment 1 versus treatments 2-5)
Linear trend within coded chemical (treatments 2-4)
Coded (N) versus standard (N) (treatment 3 versus treatment 5)
NOTE: Treatment means, LSD(5%), and the significance of the contrasts of interest are presented for an herbicide experiment conducted
using a randomized complete block design with four replicates.
a1% significance.
b5% significance.
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
level of variability in the experiment and to allow an in-
dication of the strengthof evidence for unanticipatedques-
tions that may be asked by fellow researchers,such as "Did
the coded chemical depress yield when applied at twice
normal instead of the normal rate?"
The second sample data set consists of data from a series
of wheat cultivar evaluation trials in an agriculturaldistrict
(Table 2). In this case the LSD(5%)'s are providedto give
an indicationof the level of variabilitywithin each trial and
to provide an indication of the strength of evidence con-
cerning any particularpairwise difference within a trial. If,
however, the researcherwishes to make an overall com-
parisonbetween, say, the cultivarsRongotea and Orouafor
the agriculturaldistrict, the LSD's are not especially useful.
A more appropriatemethod of analysis is to calculate the
differences (Rongotea - Oroua) = .23, .40, .30, .23, .49,
and .39 tonnes/hectarein the respective trials and then test
the hypothesis Ho: Rongotea = Oroua using a simple t
test. With a t value of 8.0 with 5 degrees of freedom, there
would be little difficulty in concluding that Rongotea is on
average a higher-yieldingcultivar than Oroua.
In both of these data sets, the LSD is just one piece of
informationused in arrivingat a sensible interpretationof
the results. In all cases the most importantinformationis
provided by the treatmentmeans and the observed differ-
ences between these means. These provide the best guesses
as to the truedifferencesbetween the means. If the observed
differences are large enough and importantenough to be of
potential interest, the researcherwould be wise to conduct
furtherwork to establishmore accuratelythe truemagnitude
of the differences. [A good way of expressing the uncer-
tainty associated with the estimate of a particulardifference
is to quote the 95% confidence interval for the true differ-
ence, which is given by observed difference + LSD(5%).]
4. OBJECTIONS TO LSD
In the statisticalliterature,the main objections to the use
of the unrestrictedLSD procedureappearto be as follows.
1. Too many of the pairwise differences are correlated.
2. There are too many chances of spuriouslydeclaringa
zero difference to be nonzero (i.e., of committinga Type 1
error).
Wheat yield
(tonneslhectare)
5.2
7.7
8.8
8.9
8.3
.9
**a
*b
Not significant
The AmericanStatistician, May 1990, Vol. 44, No. 2 175
 Table 2. A Typical Series of Six Wheat Cultivar Evaluation For example, suppose in an alfalfa cultivarevaluationtrial
Trials in an Agricultural District that the cultivar Rere proved significantly higher yielding
than Wairau, Caliverde, and Atra. The analyst would then
Grain yields
Trial Cultivar (tonnes/hectare) bear in mind that the three comparisonsof Rere with each
of the other cultivars were correlated, so three wrong de-
Marshall,1981 Aotea 3.15
Oroua 3.81 cisions could have been made "for the price of one"; for
Rongotea 4.04 example, the field plots assigned to Rere may have been
LSD(5%) .25 unusuallyfertile, leading to a spuriousconclusion thatRere
McCraw,1982 Durum1 3.05 was a superior cultivar. Apart from this sort of informed
Durum2 3.11
Oroua 4.10 doubt about the results, it is a fact of life that a statistician
Rongotea 4.50 cannot use statistical reasoning to decide when or where
LSD(5%) .31 these "bunches of errors" occur.
Reed, 1982 Oroua 5.20 The second objection is that the unrestrictedLSD pro-
CRD 1015 5.36
CRD 1018 5.37 cedure allows too many chances of making a Type I error.
Rongotea 5.50 In this context, statisticiansknow that the more work they
LSD(5%) .27 do, or the greater the number of comparisons made, the
Lochhead,1983 Durum1 2.27 greaterthe chance of making at least one Type I error.This
Durum2 2.32 is another fact of life for statisticians, who must learn to
Hilgendorf 2.37
Oroua 2.98 live with the uncertaintythat is the very reason for their
Rongotea 3.21 existence. In the case of, say, a wheat cultivar evaluation
Karamu 3.52 trial in which the aim is to compareeach cultivarwith every
LSD(5%) .25
other cultivar, the chance of making at least one Type I
Kyle, 1983 Durum2 3.07
Durum1 3.24 errorwill increase with the numberof cultivarsincluded in
Hilgendorf 3.75 the trial. So too will the chance of making at least one
Oroua 4.33 Type II error. However, no redesign of the analysis pro-
Rongotea 4.82
Karamu 5.10 cedure can overcome this problem.
LSD(5%) .30 How common are Type I errors?The answerdependson
Whittaker,1984 Oroua 2.68 the true set of means, so I shall consider a few cases. First,
Rongotea 3.07 what if all treatmentmeans are truly equal? On average,
Karamu 3.22 how many Type I errorswould we expect to make in each
LSD(5%) .35
experiment?The answer is 5% of the numberof pairwise
NOTE: Wheat grain yields are means for each cultivar in each trial. Trials are labeled by
farmer and year. All trials were randomized complete block designs with four replicates. comparisons. Thus, in an experimentwith five treatments,
the average numberof Type I errorsis 10 x (.05) = .50,
or one Type I error in every second experiment. Second,
3. The procedureis based on comparisonwiseerrorrates what if, say, only 4 of the 10 pairs are truly equal? Then
ratherthan experimentwiseerrorrates. (This is a less com- the average numberof Type I errors is 4 x (.05) = .20,
mon objection.) or one Type I error in every five experiments. Last, what
How serious is the first objection?Or, more specifically, if no two treatmentmeans are truly equal?In this case there
how large is the correlation,and what proportionof differ- is absolutely no chance of a Type I error!
ences is correlated?The first answer is that the correlation In real experiments the case "all ui's unequal" occurs
is precisely ?/2for pairs of differences with one treatment more frequentlythanthe case [u = ,2 = . = l-k. Thus

in common to both differences and 0 for pairsof differences
with no treatmentin common. The second answer is that
for an experimentinvolving k treatments,the proportionof
correlatedpairs of pairwise differences is 4/(k + 1), which
decreases rapidly as k increases. Hence, when there are 7
treatments,half of the pairs of pairwise differences are cor-
related, but when there are 39 treatmentsonly 10% of the
pairs are correlated.
Does the unrestrictedLSD procedureallow more corre-
lated comparisonsthan the alternativeprocedures?The an-
swer is no! All proceduresare equal in this respect. This is
because the problem of correlationcan only be solved at
the design stage, by prespecifyingorthogonalcontrastscor-
respondingto questions of interestto the researcher.At the
analysis stage, the best that the data analyst can do is to
bear in mind the fact that each comparisonis correlatedto
certain other comparisons, so Type I errors tend to occur
in bunches, being more frequentthan usual in some exper-
iments but less frequent than usual in other experiments.
in general there is more opportunityfor Type II errors to
occur than Type I errors. This means that from a practical
viewpoint it is desirable to put more weight on minimizing
Type II errors than Type I errors. Since most of the alter-
native procedureshave a higher Type II errorrate than the
unrestrictedLSD procedure (Carmer and Swanson 1971,
1973), this points to the unrestrictedLSD procedureas the
procedureof choice.
The preoccupationwith Type I errorsamong theoretical
statisticianspresumablyarises from the comparativemath-
ematical simplicity of the null case ,? = A2 = =k
which has led to a predominanceof work on this case at
the expense of equally importantalternativehypotheses.
The third objection sometimes raised is that the unre-
stricted LSD procedureis wrongly based on a comparison-
wise Type I error rate and should instead be based on an
experimentwiseType I errorrate. The problemhere is that
holding the experimentwiseType I errorrate constant, say
at 5%o,causes a rapidincrease in the probabilityof Type II

176 The American Statistician, May 1990, Vol. 44, No. 2

This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM

All use subject to JSTOR Terms and Conditions
error with increasing size of experiment. This is highly
undesirable;it takes the control of Type I and IIerrorsout
of the hands of the researcherin such a way that an un-
desirablebalanceof Type I and IIerrorsis usually obtained.
The natural conceptual unit is the comparison, not the
experiment. An experimentis no more a naturalunit than
a project consisting of several experiments or a research
programconsisting of several projects. Clearly, it is unsat-
isfactory to have the size of the experiment,or the number
of experiments in a project, influencing the probabilityof
detecting a particularpairwise difference.
5. INCONSISTENCY
A basic defect in most multiple comparisonprocedures
is the lack of consistency in the verdicts they returncon-
cerning whethertwo populationmeans differ. In this section
I define the term inconsistencyand then consider several of
the better-known alternative procedures, showing by ex-
ample that each procedureis inconsistent.
5.1 Formal Definition
A procedureis called inconsistent if for any two popu-
lation means gi and i,j the probabilityof judging them to
be differentdependson either the numberof populationsor
the values of the sample means for the other populations.
Thus a procedureis consistentif the decision it generates
as to whethertwo populationmeans are differentis depen-
dent only on (a) the difference between the two sample
means, (b) the standarderrorof this difference, (c) the num-
ber of error degrees of freedom, and (d) the significance
level at which the procedureis operated.
5.2 Tukey's HSD Procedure
Tukey's HSD procedure (Tukey 1949) is based on the
distributionof the range. The proceduredeclares two pop-
ulations to be different, say at p = .05, if their sample
means differ by more than HSD(5%) = qk,,(.95) x SEM,
where qk,,(.95) denotes the 95th percentile of the distri-
bution of the range of k N(O, 1) randomvariables, k is the
numberof populationsin the study, v is the numberof error
degrees of freedom, and SEM is the standarderrorof the
mean.
In Figure 1 we may suppose that three research col-
leagues-Alison, Sue, and Graeme-have each carriedout
an experiment. The experiments include 2, 4, and 8 treat-
ments, respectively, but each experiment includes treat-
ments A and B. All designs were completely randomized,
with 13, 7, and 4 replicates, respectively, so the numberof
error degrees of freedom was 24 in each case. The SEM
was 1.00, and the observed difference between treatments
A and B was 4.3 in all cases.
In our hypotheticalsituationnone of the researchershave
any knowledge to enable them to prespecify meaningful
contrasts, so their statisticianadvises them to use Tukey's
procedure to analyze their data. Imagine the statistician's
embarrassmentwhen they comparenotes and discover that
the significance of the difference between treatmentsA and
B has varied from not significant, to 5% significant, to 1%
significant. This means that the evidence concerningtreat-
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
Significance
A vs. B
Alison HSD = 2.92
A B
20 25
Sue t - = 3.90
HSD
., B111 *
20 25
Graeme
HSD = 4.68
fLtJLL zL2J1 ns
20 25
Figure 1. Significance of the Difference Between Populations
Receiving Treatments A and B in Three Studies Statistically Ana-
lyzed Using Tukey's HSD Procedure. The horizontal bar represents
the HSD(5%), ns is not significant, a single asterisk denotes 5%
significance, and a double asterisk denotes 1% significance.
ments A and B has varied from "no proof of a difference,"
to "suggestive of an effect," to "convincing evidence of a
difference" [to use the words of O'Brien (1983)].
The "inconsistency" in the decisions returned by Tukey's
procedure is in practice unacceptable. The phenomenon is
a reflection of the increase in the critical HSD value with
an increasing number of experimental treatments; this is well
known and is the main reason why Tukey's procedure is
not widely used.
5.3 Restricted LSD Procedure
The restricted (or Fisher's) LSD procedure is a two-stage
procedure. First, the overall F value is examined. If the
overall F value is not significant, say at p = .05, all pop-
ulation means are declared to be identical, and no pairwise
tests are conducted. If the overall F value is significant, all
5%-level pairwise t, or LSD, tests are conducted. Similarly,
if the overall F value is 1% significant, all 1%-level pairwise
t, or LSD, tests are conducted.
In Figure 2 we suppose that another three research col-
leagues-Ron, Mary, and Dave have carried out exper-
iments. These experiments all have four treatments, including
a common pair, A and B. All designs were randomized
complete block designs with six replicates, so the number
of error degrees of freedom was 15 in each case. The SEM
was 1.00, and the observed difference between treatments
A and B was 4.3 in all cases.
In this example the statistician advises his or her col-
leagues to use the restricted LSD procedure to analyze their
data sets. As displayed in Figure 2, this advice also leads
to inconsistent results and embarrassment for the consulting
statistician. The phenomenon in this case arises because the
significance of the difference between treatments A and B
is tied to the significance of the overall F test, with the
latter varying from 5.62 (**) to 4.21 (e) to 3.14 (ns).
The restrictedLSD procedurereceived favorablemention
in the reviews carriedout by Carmerand Swanson (1971,
1973) and Chew (1976, 1977), and has many followers.
However, it is clearly not a consistent procedure.-
The AmericanStatistician, May 1990, Vol. 44, No. 2 177
 Significance
A vs. B
Ron
LSD(1%) = 4.17 AL **
20 25
Mary
LSD(5%) = 3.01 A B
20 25
Dave
LSD = oc) A B ns
20 25
Figure 2. Significance of the Difference Between Populations
Receiving Treatments A and B in Three Studies Statistically Ana-
lyzed Using the Restricted LSD Procedure. ns is not significant, a
single asterisk denotes 5% significance, and a double asterisk de-
notes 1% significance.
5.4 Waller and Duncan's Procedure
In Waller and Duncan's k-ratio LSD procedure(Waller
and Duncan 1969) the overall F value is used in calculating
the LSD. If the overall F value is large the calculatedLSD
is smaller than if the F value is small. A k-ratio of 100
approximatesa 5%-level test and a k-ratioof 500 approx-
imates a 1%-level test. In general, critical values must be
obtained from tables (Waller and Duncan 1969). It is in-
formative, however, to note thatfor large experiments,with
?15 treatmentsand ?30 degrees of freedom for error, the
LSD(5%) = LSD(k = 100) can be approximated by 1.72 x
\F/(F - 1) x SED (Duncan 1965). In this expression,
as F -> ? the LSD(5%) approaches 1.72 x SED, and as
F -- 1 the LSD(5%) approachesoo.
In Figure 3 anotherthree colleagues Harry, John, and
Skip-have each conducted experimentswith 7 treatments
and 20 replicates, laid out in a completely randomizedfash-
ion. Each researcherincludeda common pairof treatments,
A and B, and each observed a difference of 3.6 between
treatmentsA and B. The SEM was 1.00, so the SED was
1.414, and the numberof errordegrees of freedomwas 133
in all cases.
In this hypotheticalexample the statisticianadvises his or
her colleagues to use Waller and Duncan's procedureas a
method of analysis, using a k ratio of 100 to approximate
a 5%-level test and a k ratio of 500 to approximatea 1%-
level test. As shown in Figure 3, the same inconsistency
arises as with the other two procedures. The explanation
here is thatthe critical value dependson the overall F value;
small F values mean large criticalvalues, and largeF values
mean small critical values (in Fig. 3 the F values are 25.25,
3.01, and 1.40).
This procedureis regardedby the reviewers mentionedin
the last section as one of the better procedures. In agricul-
turalresearchit has received some acceptanceas a successor
to Duncan's multiple range test.
5.5 Duncan's Multiple Range Test
Duncan's multiple range test (Duncan 1955) arises from
two modifications to Tukey's HSD procedure. These are
(a) to use the critical value for the distributionof the range
for the number of treatments,p, in a reduced group, not
178 The American Statistician, May 1990, Vol. 44, No. 2
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
Significance
LSD = 2.52 A vs. B
Harry
1 * IA BH *
15 20 25
LSD = 3.07
John
.JAI2LE LIL ,
20 25
*
LSD 3.87
Skip ,
IAfflFLlBI
20
ns
25
Figure 3. Significance of the Difference Between Populations
Receiving Treatments A and B in Three Studies Statistically Ana-
lyzed Using Wallerand Duncan's k-Ratio LSD Procedure (and their
tabular values). A k ratio of 100 was used to derive LSD(5%), which
is shown as a horizontal bar. ns is not significant, a single asterisk
denotes 5% significance, and a double asterisk denotes 1% sig-
nificance.
the total numberof treatments,and (b) to varythe protection
level with the numberof treatmentsso that it is consistent
with the structuredcase in which (p - 1) orthogonalcon-
trastscan be specified. Criticalvalues are tabulatedin many
statisticaltextbooks;these values inflate quite slowly as the
group size (p) increases, so in practice the procedureoften
yields results similarto those obtainedfrom the unrestricted
LSD procedure.
With Duncan's multiple range test it is more difficult to
construct examples as extreme as those shown in Figures
1-3; these do exist, but only for relatively large experi-
ments. This indicates that Duncan's multiple range test is
less inconsistent than the three preceding procedures.
5.6 Unrestricted LSD Procedure
In each of Figures 1-3 the unrestrictedLSD procedure
is entirely consistent in the verdict it returnsas to whether
the populationsreceiving treatmentsA and B have different
means. In fact, the definition of the word consistent means
that the unrestrictedLSD is always a consistent procedure
and is the only consistent procedure.
5.7 Comparison of Procedures
In Figures 1-3 the decision returnedby a particularpro-
cedure has been shown to vary from experimentto exper-
iment, with no change in the observed difference, the SED,
or the numberof errordegrees of freedom. Similarexamples
can be constructedfor all proceduresexcept the unrestricted
LSD procedure. Some procedures, however, are more in-
consistent than others.
Tukey's procedure is the most inconsistent of the alter-
native proceduresmentioned in this article. With this pro-
cedure the significance can vary from not significantto . 1%
significant in studies of only modest size. In Figure 1, the
quoted HSD(5%) values are equivalent to the following
LSD values: HSD(Alison) = LSD(5%); HSD(Sue) =
LSD(1.1%); HSD(Graeme) = LSD(.3%). In other words,
each researcher has in effect used the unrestrictedLSD
procedurebut has also allowed the numberof treatmentsto
select the significance level. Tukey's procedure,however,
is at least predictable in its inconsistency, since the HSD
depends simply on the numberof treatmentsincludedin the
study and not on the observed sample means.
The restrictedLSD procedurecan be very inconsistent,
since examples of studies of modest size can be constructed
where the significance varies from not significant to .1%
significant. In practice, however, the inconsistency of this
procedure is much less serious than that of Tukey's pro-
cedure, since in many instances the overall F value is sub-
stantial.
With Waller and Duncan's procedure the limit of the
variation in the significance is from not significant to 1%
significant. In Figure 3, the quoted LSD(5%) values are
equivalent to the following LSD values: LSD(Harry) =
LSD(7.7%); LSD(John) = LSD(3.2%); LSD(Skip) =
LSD(.7%). In other words, each researcherhas in effect
used the unrestrictedLSD procedure, but at different sig-
nificant levels determinedby the data. The example shown
in Figure 3, however, had to be carefully constructedto
exhibit this level of variation.This suggests that the incon-
sistency of Waller and Duncan's procedureis usually only
moderate.
With Duncan's multiple range test it is the number of
treatmentmeansthatare intermediatebetweenthe two means
being compared that determines the effective significance
level of the particularcomparison. With this test it is even
more difficult to constructexamples with the level of vari-
ation shown in Figures 1-3. In practice, this procedure
appearsto be the most consistent of the alternativesto the
unrestrictedLSD procedure.
It is interesting that it is Duncan's multiple range test
that, until recently, enjoyed the greatestacceptanceamong
agriculturalresearchers. In addition to its greater level of
consistency, Duncan's test is the least conservative of the
alternatives to the unrestrictedLSD procedureand is the
procedurethat is most similar to the latter. In fact, one can
speculate that the t test, or unrestrictedLSD, is subcon-
sciously accepted by researchersas the "standard,"so any
procedure that is too different is unacceptable. More re-
cently, many researchershave changed back to using the
unrestrictedLSD procedure.
6. DISCUSSION
In practice, an applied statisticiancannot afford to rec-
ommend an inconsistentprocedure,so the unrestrictedLSD
procedureis the only procedurethat can be safely recom-
mended to researchers.In additionto its consistency, how-
ever, the unrestrictedLSD procedure has many practical
advantagesover the alternativeprocedures(Table 3). It is
simple, provides a naturalextension to the two-population
case, and is flexible enough to cater easily for unequal
Table 3. Comparison of Unrestricted LSD Procedure WithAlternative
Multiple Comparison Procedures
Characteristics
Consistency?
Simplicity?
Flexibility?
Type I error rate?
Power?
Required sample size?
Easy to check?
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
replicationor heterogeneityof variance.The calculatedLSD
can also be used to derive a confidence interval for the
difference between two populationmeans, namelyobserved
difference + LSD. It has a known andconstantType I error
rate that is "true to label." If conservatism is required, it
can be operatedat a 1% or . 1% level of significance rather
than at a 5% level. Its power is determinedby the number
of replications and the underlyinglevel of variability, and
the numberof replicationsrequiredto achieve a certainlevel
of power for a given truedifferencecan be easily computed.
The unrestrictedLSD procedureoften has maximumpower
among the procedures, exceeded only by Waller and Dun-
can's test under some assumed sets of true means (Carmer
and Swanson 1971, 1973). The calculations are easier to
check than with most of the alternative procedures, and
usage of these latter procedureshas undoubtedlyled to an
increase in the numberof wrong conclusions based on un-
detectedcomputationalerrors.In summary,if a cost-benefit
analysis were to be performed, the unrestrictedLSD pro-
cedure would shine throughas the clear leader in the field.
Carmer and Walker (1982, 1985) also presented argu-
ments for the conclusion that use of the LSD is appropriate
whenever a pairwise multiple comparison procedureis in
order. They arguedthat the researcheris in the best position
to fix the significance level prior to the hypothesis test, to
achieve the best balancebetween Type I and Type II errors.
This is an alternativeview to thatput forwardin this article;
I preferto carryout the test at a rangeof significance levels,
in effect using the LSD's to measure the strength of the
evidence, ratherthan to set up a "black versus white" type
of hypothesis test.
In the more general hypothesis testing context, the sce-
nariothat is most acceptableto statisticiansis thatof a well-
designed study in which orthogonal contrasts are prespe-
cified, correspondingto a "vision of reality" that will, it is
hoped, be supportedby the data. If this vision is not sup-
ported by the data, however, it is sometimes found that
anotherset of orthogonalcontrastsprovides a good descrip-
tion of the data. This descriptiongeneratesa new vision of
reality, which will then need to be confirmedin subsequent
studies.
Multiple comparisonproceduresgo againstthis basic phi-
losophy in that they appearto formulateand test hypotheses
in the same study simultaneously.In fact, the multiplecom-
parisoncontroversyis resolved if the proceduresarethought
of as hypothesis generatorsratherthan as methods for si-
multaneous generation and testing. When viewed in this
light, the 5%-level unrestrictedLSD procedureis seen to
have the following simple characteristics:
Unrestricted Alternative
LSD procedure procedures
Consistent Inconsistent
Simple More complex
Flexible Less flexible
Constant Variable
Maximum power Lower power
Easy to calculate Hard to calculate
Easily checked Harder to check
The AmericanzStatistician, May 1990, Vol. 44, No. 2 179
 1. If in fact 1xi = s,j, the hypothesis HA: 1,i # i,j will
be generatedwith probability .05.
2. If ,ij =$ ,j, this probabilitydepends only on the size
of the standardizeddifference (ui - tj)/SED.
In any field of researchan individualstudy is merely one
componentto be interpretedin the light of other studies and
otherknowledge. The unrestrictedLSD proceduremay gen-
erate a false hypothesis in one study, but this is unlikely to
be confirmedby subsequentstudies or may be recognizable
as false from previous studies or previous knowledge. In
other words, the problem of false hypotheses is not partic-
ularly serious for researchworkers, who are attunedto the
problemsof working with variablematerialsand are aware
of the need for confirmation of unexpected results using
independentdata sources.
Scheff6's test (Scheffe 1953), which has been advocated
for the analysis of general unplannedcontrasts, is in a po-
sition analogous to multiple comparisonproceduresin that
it also representsan attemptat simultaneouslyformulating
and testing hypotheses. Scheffe's test is inconsistent, since
the significance of a given contrastvalue with a given stan-
dard errorcan vary from not significant to . I% significant,
dependingsimply on the numberof populationsincludedin
the study. Forthis reasonScheffe's test is widely recognized
as excessively conservative and is seldom used in practice.
The consistent method of analysis is to use the same F or
t test as for plannedcontrasts. I would again suggest, how-
ever, treatingthis as a hypothesisgeneratingprocedurerather
than a procedurefor simultaneousgenerationand testing.
In summary,the only consistentway to analyzeunplanned
contrasts, both pairwise and general, is to use the ordinary
single-degree-of-freedomF test or the equivalentt test. That
is, one should use the same proceduresas in the planned
case but should treat them as hypothesis generating pro-
cedures rather than hypothesis generating and confirming
procedures.
The implications for teachers of statistics are important.
These new, simplifiedrecommendationsmeanthatthe large
and confusing body of material on multiple comparison
procedures, and the smaller amount on Scheffe's test, can
be replacedby materialthat stresses the distinctionbetween
hypothesis formulationand testing. This makes a much ti-
dier and more consistent teaching package than is currently
available.
7. THE PRACTICAL SOLUTION
My recommendationis to use only the simplest formal
proceduresand to rely upon improvedcommon sense, better
statistical education, and the incorporationof information
from other sources as safeguardsagainst errorsof interpre-
tation.
7.1
Multiple Comparisons
1. Do not attemptto simultaneouslyformulateandtest
hypothesesconcerningpairwisedifferencesin a single study.
2. When analyzing the data from a study in which the
populations have no discernible structure, seek out inter-
esting or importantdifferences between populations. Use
180 The American Statistician, May 1990, Vol. 44, No. 2
This content downloaded from 138.26.188.186 on Tue, 17 Jun 2014 13:51:16 PM
All use subject to JSTOR Terms and Conditions
the unrestrictedLSD's for, say, 10%-, 5%-, 1%-, and . 1%-
level tests, to determine the strength of the evidence for
each of these differences;this gives an indicationas to which
of the observed differences are likely to be real.
3. Confirmthese differences in subsequentstudies.
7.2 General Unplanned Contrasts
1. Do not attemptto simultaneouslyformulateand test
hypotheses concerning contrastsin a single study.
2. When analyzingthe datafrom a study in which several
sets of contrasts could have been prespecified, formulate
hypotheses in terms of those contraststhat best describe the
datausing the ordinarysingle-degree-of-freedomF or t tests
(not Scheffe's more conservative test).
3. Confirm any interesting hypotheses in subsequent
studies.
[ReceivedJuly 1987. Revised September1989.]
REFERENCES
Carmer, S. G., and Swanson, M. R. (1971), "Detection of Differences
Between Means: A Monte Carlo Study of Five Pairwise Multiple Com-
parison Procedures,"AgronomyJournal, 63, 940-945.
(1973), "An Evaluation of Ten Pairwise Multiple Comparison
Proceduresby Monte Carlo Methods," Journal of the AmericanStatis-
tical Association, 68, 66-74.
Carmer, S. G., and Walker, W. M. (1982), "Baby Bear's Dilemma: A
StatisticalTale," AgronomvJournal, 74, 122-124.
(1985), "Pairwise Multiple Comparisonsof TreatmentMeans in
Agronomic Research," Journal of Agronomic Education, 14, 19-26.
Chew, V. (1976), "Comparing Treatment Means: A Compendium,"
HortScience, 11, 348-357.
(1977), "ComparisonsAmong TreatmentMeans in an Analysis
of Variance," AgriculturalResearch Service Technical Bulletin H-6,
U.S. Departmentof Agriculture,Washington, D.C.
(1980), "Testing Differences Among Means: Correct Interpreta-
tion and Some Alternatives,"HortScience, 15, 467-470.
Cousens, R. (1988), "Misinterpretationsof Results in Weed Research
ThroughInappropriateUse of Statistics," WeedResearch, 28, 281-289.
Duncan, D. B. (1955), "Multiple Range and Multiple F Tests," Biomet-
rics, 11, 1-42.
(1965), "A Bayesian Approachto Multiple Comparisons,"Tech-
nometrics, 7, 171-222.
Little, T. M. (1978), "If Galileo Publishedin HortScience,"HortScience,
13, 504-506.
Nelder, J. A. (1971), Discussion of "The Present State of Multiple Com-
parison Methods," by R. O'Neill and G. B. Wetherill, Journal of the
Royal Statistical Society, Ser. B, 33, 244-246.
O'Brien, P. C. (1983), "The Appropriatenessof Analysis of Varianceand
Multiple ComparisonProcedures,"Biometrics, 39, 787-794.
Perry,J. N. (1986), "Multiple-Comparison Procedures:A DissentingView,"
Journal of Economic Entomology, 79, 1149-1155.
Petersen, R. G. (1977), "Use and Misuse of Multiple ComparisonPro-
cedures," AgronomyJournal, 69, 205-208.
Plackett, R. L. (1971), Discussion of "The PresentStateof MultipleCom-
parison Methods," by R. O'Neill and G. B. Wetherill, Journal of the
Royal Statistical Society, Ser. B, 33, 242-244.
Preece, D. A. (1982), "The Design and Analysis of Experiments:What
Has Gone Wrong?" Utilitas Mathematica, Ser. A, 21, 201-244.
Scheffe, H. (1953), "A Method for JudgingAll Contrastsin the Analysis
of Variance," Bometrika, 40, 87-104.
Tukey, J. W. (1949), "ComparingIndividual Means in the Analysis of
Variance," Biometrics, 5, 99-114.
Waller, R. A., and Duncan, D. B. (1969), "A Bayes Rule for the Sym-
metric Multiple Comparison Problem," Journal of the American Sta-
tisticalAssociation, 64, 1484-1503; Corrigendum(1972), 67, 253-255.