Special Pair-Wise Comparison

Techniques used in Multiple

Comparison Hypotheses

INDE-8210 / MECH-8290
 Pair-wise Comparison of All
Treatments
H 0 : 1 =  2 = 3 = .... =  k vs.
H1 : not H 0
While comparing all treatment means in a single
factor experiment, it may be necessary to compare
each individual treatment mean with each of the
other treatment means using pair-wise comparisons.
The parameters of interest are all pair-wise
differences among the treatment means,
 i − j for all i  j, resulting in k(k-1)/2 comparisons.
Most frequently, applications of these methods have
an objective to detect significant inequalities,  i   j
for all i  j .
 Pair-wise Comparison of All
Treatments
H 0 : 1 =  2 = 3 = .... =  k vs.
H1 : not H 0

In practice, we would like to conduct these pair-

wise comparison between different treatment
means when the null hypothesis H0 has
been rejected. As it is clear that if the null
hypothesis is not rejected, there is no point to
conduct pair-wise comparison as all of them
are found to be equal.
 1. The Tukey Method: Honestly
Significant Difference (HSD) Test
Means are different if: yi − y j  HSD
se2 MSe
where HSD = qa ,k , = qa ,k ,
n n
qa ,k , is the Studentized range statistic for a Range
of k treatment means in an ordered array.
Critical values for an experiment-wise rate a
and  degrees of freedom can be found in
the following table.
Note: Experiment-wise error rate = a
Number of replications (n) is the same for all treatments
This MCP controls experiment-wise error rate! Comparison-wise
error rates is thus very low.
Studentized Range Table for Tukey’s Test
Example: Tukey HSD Test
In an experiment conducted to compare the strength of welds
produced by four different welding techniques, each welding
technique was used to weld five pairs of metal plates in a
random fashion (completely randomized design). The average
strengths for the five welds of each technique were:
Technique: A B C D
Mean strength 69 83 75 71
The estimate of experimental error variance for the
experiment was MSe=15 with 16 degree of freedom.
An One-Way ANOVA was conducted on the comparison of the
four treatment means and the null hypothesis has been
rejected, i.e., the four welding techniques will not yield equal
welding strength. Now, we are interested in pair-wise
comparison using the Tukey HSD technique.
Example: Tukey HSD Test
Applying the Tukey method, the Studentized Range statistic
with an experiment-wise error of alpha=0.05 can be found
from the table as q0.05,4,16=4.046 where k=4 treatment means
in the ordered array and d.o.f.  = 16, MSe=s2=15. The
standard error is MS / n = 15 / 5 = 1.73
e

The computed HSD is therefore HSD=4.046*1.73 = 7.0

 Example: Tukey HSD Test
Comparison | yi − y j | HSD Different?

A vs. B 14 > 7.0 Yes

A vs. C 6 < 7.0 No
A vs. D 2 < 7.0 No
B vs. C 8 > 7.0 Yes
B vs. D 12 > 7.0 Yes
C vs. D 4 < 7.0 No
 Unequal Replications and Complex
Models with the Tukey Method
For Unequal replications in the Completely
Randomized Design, Tukey and Kramer proposed
approximation for the HSD calculation.

Means are different if: yi − y j  HSD

se2 1 1 MS e 1 1
where HSD = qa ,k , ( + ) = qa ,k , ( + )
2 ni n j 2 ni n j

qa ,k , is the Studentized range statistic for a Range

of k treatment means in an ordered array.
Critical values for an experiment-wise rate a
and  degrees of freedom can be found in
the previous table.
 2. Fisher’s Least Significant
Difference (LSD) Test
For any pair of observed treatment means, yi and y j
the Least Significant Difference (LSD) is defined as:
1 1 1 1 2
LSD(a ) = ta ,
s [ + ] = ta , MS e [ + ] = ta , MS e
2
2 ni n j 2 ni n j 2 n

if all groups have same size n.

The null hypothesis H 0 : i =  j is rejected if

| yi − y j |  LSD(a )
Type I (comparison-wise) error rate = a
This procedure controls Comparison-wise Error. Experiment-wise
error control comes from requiring a significant overall F test prior to
performing any means comparisons.
 Example: Fisher’s LSD Test
In an experiment conducted to compare the strength of welds
produced by four different welding techniques, each welding
technique was used to weld five pairs of metal plates in a
random fashion (completely randomized design). The average
strengths for the five welds of each technique were:
Technique: A B C D
Mean strength 69 83 75 71

The estimate of experimental error variance for the experiment

was MSe=15 with 16 degree of freedom.

An One-Way ANOVA was conducted on the comparison of the

four treatment means and the null hypothesis has been
rejected, i.e., the four welding techniques will not yield equal
welding strength. Now, we are interested in pair-wise
comparison using the Fisher’s LSD technique.
 Example: Fisher’s LSD Test
Applying the Fisher’s LSD method, the computation of
the LSD requires the critical value for the Student-t
test, t0.025,16 =2.12
The standard error of the difference between two
treatment means is
2MS e / n = 2(15) / 5 = 2.45

The computed LSD is therefore LSD=2.12*2.45=5.2

 Example: Fisher’s LSD Test

Comparison | yi − y j | LSD Different?

A vs. B 14 5.2 Yes

A vs. C 6 5.2 Yes
A vs. D 2 5.2 No
B vs. C 8 5.2 Yes
B vs. D 12 5.2 Yes
C vs. D 4 5.2 No
Another Example for Both Tukey HSD and
Fisher’s LSD Tests

Rank Variety Mean Yield

5 A 50.3
4 B 69.0
6 C 24.0
2 D 94.0
3 E 75.0
n1=n2=n3=n4=n5=n=4
1 F 95.3

ANOVA Table
Source df MeanSquare F
Variety 5 2976.44 24.80**
Error 18 120.00
 H0 : 1 =  2 =  3 =  4 =  5 =  6

HA : i  i for some i  i

Overall F-test indicates that we reject H0 and assume HA

Which mean is not equal to which other means.

Consider all possible comparisons between varieties: yi − y j

First sort the treatment levels such that the level with the smallest
sample mean is first down to the level with the largest sample mean.

Then in a table (matrix) format, compute the differences for all of the
t(t-1)/2 possible pairs of level means.
t(t − 1) 6(5)
= = 15
2 2
 Differences for all of the t(t-1)/2=15 possible pairs of level
means
Arrange in order
Largest Difference
y A − yC
C A B E D F
24.0 50.3 69.0 75.0 94.0 95.3
C 24.0 -- 26.3 45.0 51.0 70.0 71.3
A 50.3 -- 18.7 24.7 43.7 45.0
B 69.0 -- 6.0 25.0 26.3
E 75.0 -- 19.0 20.3
D 94.0 -- 1.3
F 95.3 --

Smallest difference

Question: How big does the difference have to be

before we consider it “significantly big”?
 Fisher’s LSD
F=24.8 > F5,18,.05=2.77 --> F is significant
LSD = t18,a MSE • ( n2 ) = t18,0.025 120 • ( 24 ) = 2.101 • 7.746 = 16 .27
2

C A B E D F
24.0 50.3 69.0 75.0 94.0 95.3
C 24.0 -- 26.3 ‡ 45.0 ‡ 51.0 ‡ 70.0 ‡ 71.3 ‡
A 50.3 -- 18.7 ‡ 24.7 ‡ 43.7 ‡ 45.0 ‡
B 69.0 -- 6.0 25.0 ‡ 26.3 ‡
E 75.0 -- 19.0 ‡ 20.3 ‡
D 94.0 -- 1.3
F 95.3 --
‡ Implies that the two treatment level means are statistically
different at the a = 0.05 level.
a b c c d d Alternate ways to
C A B E D F
24.0 50.3 69.0 75.0 94.0 95.3 indicate grouping
of means.
 Tukey’s (Honestly Significant Difference)
Not protected hence no preliminary F test required.
MS e 120
HSD = qa ,k , = 4.49 = 24.59
n 4
From table

C A B E D F
24.0 50.3 69.0 75.0 94.0 95.3
C 24.0 -- 26.3 ‡ 45.0 ‡ 51.0 ‡ 70.0 ‡ 71.3 ‡
A 50.3 -- 18.7 24.7 ‡ 43.7 ‡ 45.0 ‡
B 69.0 -- 6.0 25.0 ‡ 26.3 ‡
E 75.0 -- 19.0 20.3
D 94.0 -- 1.3
F 95.3 --
‡ Implies that the two treatment level means are statistically different at the a = 0.05 level.

a b bc cd d d
C A B E D F
24.0 50.3 69.0 75.0 94.0 95.3
 There are many other pair-wise comparison
tests such as:
• Student-Newman-Keuls (SNK) Multiple
Range test
• Duncan’s New Multiple Range Test,
• Waller-Duncan K-Ratio Test,
• Scheffé’s S Method, …

In MINITAB, you can select what pair-wise

comparison to be conducted. We will not
discuss these tests.
 One-Way ANOVA and
One-Factor Experiment
• One-way ANOVA is the analysis technique
for experiments involving only one factor
(or one independent variable) at more than
2 levels. (What do you do with one factor
with only two levels?)
More on One-factor Experiment
• One-Factor, at two independent levels → t
test.
• One, at two dependent levels → paired t-test.
• One factor, at more than two independent
levels → CRD (Completely Randomized
Design)
• One-factor, at more than two dependent
levels → Randomized Block Design. (RBD)