Analysis of Variance

(ANOVA)-I
INDE-8210 / MECH-8290
 Multiple Comparison
• When an experiment (comparison) has
– A single factor with 3 or more levels, or
• (e.g. Testing the percentage of dosage; quench
schedule of metal parts; cutting speed of a CNC
machine, etc.)
– Two or more factors
• (e.g. mix of two/multiple chemicals for a new clue,
cutting speed and machine feed for a CNC
machine, etc.
• Statistical Test to be used for comparison of
means from different populations: ANOVA, as
the t-test or z-test is no longer capable of
handling this kind of comparison.
 ANOVA
• Stands for ANalysis Of VAriance
• Used for hypothesis testing in
– Simple Regression
– Multiple Regression
– Comparison of Means
• The heart of the ANOVA is a comparison
of variance estimates between your
conditions (factors, groups, treatments,
etc.)
• Often call the effects as treatment effects
 Why do ANOVA
• For two levels of a Independent Variable,
ANOVA produce same results as the t-test.
• However, when more than 2 levels of treatment
(or experiment), MUST use ANOVA.
– Often, it is necessary to have more than two
levels of treatments ( in most experiments)
• Non-linear relationship, two levels will
always produce a straight line, or linear
relationship. But, three or more can
describe a non-linear function.
 Problems with Multiple t-tests
• Each time you perform a statistical test, you reject the
null hypothesis if the probability that your sample data
was obtained is less than 5% given that the null
hypothesis is true.
• You are potentially wrong 5% of the time that you reject
a null hypothesis (this is another reason why we never
use the word PROVE)!
• Perform three t-tests, each test has a 5% chance of
erroneously rejecting null hypothesis –implies that the
probability that you are wrong on at least one of the t-
tests increases to ~15%
• Family-wise error rate: chance of making at least 1 type-I
error in a family (or sequence) of statistical tests.
Reasons and Assumptions of ANOVA

• ANOVA tests all conditions simultaneously,

while keeping experiment-wise error rate
equal to  (usually 0.05 or 5%).
• Assumptions (as in any parametric test):
– Independent samples;
– Normally distributed data;
– Variance of all populations are homogenous
• Each one of the experimental groups must have
approximately equal variance.
 Sources of Variation
• There is variation anytime that all of the
data values are not identical
• This variation can come from different
sources such as the model or the factor
• There is always the left-over variation
that can’t be explained by any of the other
sources. This source is called the error.
(Natural variation; within-treatment
variation; random variation…)
 Variation
• Variation is the sum of squares of the
deviations of the values from the mean of
those values
• As long as the values are not identical,
there will be variation
• Abbreviated as SS for Sum of Squares
 Degrees of Freedom
• The degrees of freedom are the number of
values that are free to vary once certain
parameters have been established (e.g.
mean, …)
• Usually, this is one less than the sample
size, but in general, it’s the number of
values minus the number of parameters
being estimated
• Abbreviated as df (or d.o.f.)
 degrees of freedom
• total d.o.f = N-1
• for k treatments, treatment d.o.f = k-1
• for n data points per treatment, error d.o.f
= k(n-1) (i.e. the sum of the d.o.f’s for each
group) (Note: error can be considered as
the natural variation, or the uncontrollable
variation that can not be avoided, as we
discussed back in SPC about variation
classification.)
 Variance
• The sample variance is the average
squared deviation from the mean
• Found by dividing the variation by the
degrees of freedom
• Variance = Variation / df
• Abbreviated as MS for Mean of the
Squares
• MS = SS / df
 THE ANALYSIS OF VARIANCE (ANOVA)
One Sample T-test:
H0: =0

Two-Sample T-test:
H0: 1 = 2
(Two populations have the same mean)

Single Factor Analysis of Variance (more than 2 levels)

(a.k.a. One-way ANOVA)
H0: 1 =2=3=.....=k
Test if the mean of several populations are all the same
H1: at least one mean is different
 Decomposition of Variance
(Variation)
Example:
An experiment with the electrical plating (gold) of the
memory board connector. The thickness is the quality
measure. The electric current is the control parameter.
In the experiment, we tested the process with 5 different
levels of electric current: (10A, 15A, 20A, 25A, 30A). For
each current level, two memory boards are tested and the
thickness of the connector are as follows:
Current (Amps) 10 15 20 25 30
Thickness 15.5 15.4 15.3 15.5 15.2
(μm) 15.3 15.6 15.5 15.3 15.4
 Current (Amps) 10 15 20 25 30
Thickness 15.5 15.4 15.3 15.5 15.2
(μm) 15.3 15.6 15.5 15.3 15.4
Average 15.4 15.5 15.4 15.4 15.3

Average of 5 Levels
= (15.4+15.5+15.4+15.4+15.3)/5 = 15.4
Total Variation
ssTotal = (15.5-15.4)2+(15.3-15.4)2+(15.4-15.4)2+(15.6-15.4)2+(15.3-15.4)2+(15.5-15.4)2+
(15.5-15.4)2+(15.3-15.4)2 +(15.2-15.4)2+(15.4-15.4)2
= 0.140
=
ssBetween-Level = 2(15.4-15.4)2+2(15.5-15.4)2+2(15.4-15.4)2+2(15.4-15.4)2 +2(15.3-15.4)2
= 0.04 Between-Treatment-Variation (due to different levels of Amps)

SSe
+
ssWithin-Level = (15.5-15.4)2+(15.3-15.4)2+(15.4-15.5)2+(15.6-15.5)2 +(15.3-15.4)2+
(15.5-15.4)2 +(15.5-15.4)2+(15.3-15.4)2 + (15.2-15.3)2+(15.4-15.3)2
= 0.100 Within-Treatment-Variation (due to Errors or natural causes)
One-way ANOVA: thickness versus current

Source DF SS MS F P
Current 4 0.0400 0.0100 0.50 0.739
Error 5 0.1000 0.0200
Total 9 0.1400

S = 0.1414 R-Sq = 28.57% R-Sq(adj) = 0.00%

Individual 95% CIs For Mean Based on Pooled StDev

Level N Mean StDev --------+---------+---------+---------+-
10 2 15.400 0.141 (------------*------------)
15 2 15.500 0.141 (------------*------------)
20 2 15.400 0.141 (------------*------------)
25 2 15.400 0.141 (------------*------------)
30 2 15.300 0.141 (------------*------------)
Example 2A
Does cutting fluid type affects the surface finish of
machined parts? Note: We can still use t-test
for this experiment
- Factor = “cutting fluid type"
- Factor levels ( treatments) = types "1“ and "2“
H0: 1 = 2
H1: 1 ≠ 2
- Y(i,j) = surface finish from the j-th part treated
with cutting fluid i
Cutting Fluid 1 Cutting Fluid 2
Y(1,1)=18.95 Y(2,1)=14.95
Y(1,2)=19.05 Y(2,2)=15.05
Average 19.00 15.00
(within Type)
Grand Average = 17.00
Example 2A
Assuming without cutting Fluid (BASE) = 20.00

Effect of Applying Fluid Type 1: - 1

Effect of Applying Fluid Type 2: - 5

Cutting Fluid 1 Cutting Fluid 2

Y(1,1)=19.00 Y(2,1)=15.00
Y(1,2)=19.00 Y(2,2)=15.00

Variation due to the Natural Causes : -0.05 or + 0.05

Cutting Fluid 1 Cutting Fluid 2
Y(1,1)=18.95 Y(2,1)=14.95
Y(1,2)=19.05 Y(2,2)=15.05
Average 19.00 15.00
(within Type)
 σ2 σ2 ഥ
SStotal = 𝑖=1 𝑗=1 𝑌𝑖,𝑗 − 𝑌
=(18.95-17)2+(19.05-17)2+(14.95-17)2+(15.05-17)2
= 16.01
SScutting-fluid-type=(2*(19-17)2)+(2*(15-17)2)=16.00
SSnatural-variation=(18.95-19)2+(19.05-19)2+
(14.95-15)2+(15.05-15)2=0.01

Notice that
SStotal = SScutting-fluid + SSnatural-variation
Example 2B
Does cutting fluid type affects the surface finish of
machined parts?
- Factor = “cutting fluid type"
- Factor levels ( treatments) = types "1“, "2“, “3”, and “4”
H0: 1 = 2 = 3 = 4
H1: At least one is different
Y(i,j) = surface finish from the j-th part treated with cutting fluid i
Cutting fluid 1 Cutting fluid 2 Cutting fluid 3 Cutting fluid 4
Y(1,1)=18.95 Y(2,1)=14.95 Y(3,1)=18.05 Y(4,1)=19.95
Y(1,2)=19.05 Y(2,2)=15.05 Y(3,2)=17.95 Y(4,2)=20.05
Average 19.00 15.00 18.00 20.00
(within
Type) Grand Average = 18.00
 σ4 σ2 ഥ
SStotal = 𝑖=1 𝑗=1 𝑌𝑖,𝑗 − 𝑌
=(18.95-18)2+(19.05-18)2+(14.95-18)2+(15.05-18)2
+(18.05-18)2+(17.95-18)2+(19.95-18)2+(20.05-18)2
= 28.02
SScutting-fluid-type=(2*(19-18)2)+(2*(15-18)2)+
(2*(18-18)2)+(2*(20-18)2)= 28.00

SSnatural-variation=(18.95-18)2+(19.05-18)2+(14.95-15)2
+(15.05-15)2+(18.05-18)2+(17.95-18)2
+(19.95-19)2+(20.05-19)2 = 0.02
Notice that the following is still true!
SStotal = SScutting-fluid + SSnatural-variation
 Basic Model of a data set:
• Assume the factor is at “k” levels, and that n
observations are taken within each level (balanced!)
• Let y(i,j) be the response for the j-th observation within
the i-th level of the factor.
FACTOR
1 2 3 ....... k
-----------------------------------------
y(1,1) y(2,1) y(3,1) ...... y(k,1)
y(1,2) y(2,2) y(3,2) y(k,2)
. .
Observation . .
y(1,n) y(2,n) y(3,n) ..... y(k,n)
• Each y(i,j) is a random variable.
• Once the expt has been conducted, we will have numbers
that are outcomes of the random variables.
• If we repeated the expt, we would get different outcomes
• Sampling is assumed to be RANDOM

• Statistical Models:
Two Basic Types depending on the situation
FIXED vs. RANDOM Effects models
Example:
y(i,j) = spot welding quality of randomly parts from electric
current level at i
• Factor = welding current at levels “10000A", “15000A",
“9000A"
• Randomly select parts from each level, and record their
welding quality in the dataset.
FIXED Effects Model
Interested only in the 3 levels, 10000A, 15000A, and 9000A
H0: 10000 = 15000 = 9000

RANDOM Effects Model

Suppose you wanted to test whether or not the mean
welding quality were the same at all electric current levels.
How would you do it?
• From the population of all different levels, you might
randomly select “k" different levels to study.
• From each of these levels, you might randomly select
welded parts, and record their welded quality in the
dataset.
H0: Mean welded quality are equal at all electric current
levels
Fixed Effects:
Interested in making inference only about
those treatments (factor levels) included in
the study.

Random Effects:
Interested in making inference about a large
population of treatments from which those studied
are a representative random sample.
Basic Relevance:
For the One-way ANOVA
i) The statistical tests are the same for Fixed
and Random
ii) The power of the tests is different

For Multi-factor ANOVA the tests are

different.
(Fixed Effects)
Statistical Model of an Observation yij
yij = i + ij
ij ~ NID(0 , 2)
H0: 1 = 2 = ..... = k

Note that the i are unknown CONSTANTS

Thus yij are Normally distributed with mean

i and variance 2
Alternative Model:
yij =  + i + ij
where  + i = i

i is the deviation of i from the overall mean 

Implies the additional constraint:
H0: 1 = 2 = ..... = k = 0
Note that  and the i are unknown CONSTANTS
RANDOM Effects Statistical Model
yij =  + i + ij

Where the i are RANDOM VARIABLES since

the treatments are selected randomly
Assume i ~ NID(0 , 2)
Thus yij are Normally distributed with mean 
and Variance (2 + 2)
H0: 2 = 0 !!!!!!
• The Random Effects Model is a.k.a the
Components of Variance model.
• Classic example is in gauge R&R studies
with variance due to both the instrument
and the inspector.
 "F-Test" for the ANOVA
Define three "Sums of Squares"
• Total Corrected Sum of Squares (SST): Squared
deviations of observations from overall average
• Error Sum of Squares (SSE): Squared deviations
of observations from treatment averages
• Treatment Sum of Squares (SStrt): Squared
deviations of treatment averages from overall
average (times n)
Dot Notation
𝑘 𝑛

𝑦.. = ෍ ෍ 𝑦𝑖𝑗 The Overall Sum

𝑖=1 𝑗=1

𝑦..
𝑦ധ = ൗ𝑁 The Overall Average

𝑦𝑖. = ෍ 𝑦𝑖𝑗 The Sum within Treatment Level i

𝑗=1

𝑦𝑖.ൗ The Average for Treatment Level i

𝑦𝑖. = 𝑛
 DECOMPOSITION OF SS
𝑛𝑖
𝑆𝑆𝑇𝑜𝑡𝑎𝑙 = σ𝑘𝑖=1 σ𝑗=1 𝑦𝑖,𝑗 − 𝑦ത 2

k 2
SStreatment =  n y − y






i i.  
i =1

SSerror = SS − SS
Total Treatment
 Proof
k n 2 k n 2
SS =   y − y =   y − y + y − y
 
   
   
T ij ij
 i. i. 





i =1 j =1 i =1 j =1

k n 2
  
=   y −y + y −y
  
  
    
ij i. 
i.    
 
i =1 j =1
k n 2 k n 2 k n   
=   y −y +   y −y +   2 y −y y −y
   
     
       
ij 
i. i.  ij i. i.      
   
i =1 j =1 i =1 j =1 i =1 j =1

k 2 k n   
= SS +  n y − y +   2 y − y y − y
   
   
     
E i. ij i. i.  


  
i =1 i =1 j =1
 ( )
k n
  
= SS + SS +  2  yij − yi.  yi. − y 
E TRT   
i =1 j =1
= SS + SS +0
E TRT

Must show last term is zero

 ( ) ( )
k n k n
    
  2  yij − yi.  yi. − y  = 2  yi. − y   yij − yi. 
 
i =1 j =1 i =1 j =1

 n  
= 2  (yi. − y ) 
k  
  yij  − nyi. 
i =1  j = 1  
  

 
= 2  (yi. − y )nyi. − nyi. = 2  (yi. − y )0 = 0
k k

i =1 i =1
 k
SStrt = n  ( yi. - y.. )2
i =1

Note the distribution of the treatment average

yi . is NID(i , 2/n)

Under H0 all have the same mean 

And all yi . are NID( ,  /n)

2
SSE: Each term of SSE divided by n-1
is an estimate of 2.
Add them all up for an overall estimate of 2
Note that SSE/(N-k) = MSE estimates 2
whether or not H0 is true
 F-Test
• There will be an F test statistic for each
source except for the error and total
• F is the ratio of two sample variances
• The MS column contains variances
• The F test statistic for each source is the
MS for that row divided by the MS of the
error row
 F
• F requires a pair of degrees of freedom,
one for the numerator and one for the
denominator
• The numerator df is the df for the source
• The denominator df is the df for the error
row
• F is always a right tail test
 F-test

• If H0 is true, the ratio MStrt/MSE should be

around 1 since both are estimates of 2
• If H0 is false, MStrt/MSE should be bigger
than 1.
• Our test statistic F0 = MStrt/MSE thus
"measures something significant about H0.
Analysis of Variance : Graphic
View
 one-way ANOVA Ho

2(total)
2(error)

No difference among different treatment levels

5 treatment levels
 one-way ANOVA H1

2(total)
2(error)

Difference exists among different treatment levels

 sources of variance

Remember, variances add, so  total

2
=  treatment
2
+  error
2

So if, all means are equal:

 2
 treatment
2
+  error
2
total
= ~ 1.0
 2
error   error
2

(called “F” which is just one of several tests we could form)



 the underlying model
5

4

3
2

1

Xij =  + (j-) + ij

=  + j + ij
 computational formula
• goal: estimate the aforementioned
variances in an “easy” way
• compute a bunch of SSs (summed
squared differences from the mean)
• compute MS (mean-squares, i.e.
variances) and make our ratio, F
 degrees of freedom
• total d.o.f = N-1
• for k treatments, treatment d.o.f = k-1
• for n data points per treatment, error d.o.f
= k(n-1) (i.e. the sum of the d.o.f’s for each
group) (Note: error can be considered as
the natural variation, or the uncontrollable
variation that can not be avoided, as we
discussed back in SPC about variation
classification.)
 F
• F is the F test statistic
• There will be an F test statistic for each
source except for the error and total
• F is the ratio of two sample variances
• The MS column contains variances
• The F test statistic for each source is the
MS for that row divided by the MS of the
error row
 F
• F requires a pair of degrees of freedom,
one for the numerator and one for the
denominator
• The numerator df is the df for the source
• The denominator df is the df for the error
row
• F is always a right tail test
 The ANOVA Table
• The ANOVA table is composed of rows,
each row represents one source of
variation
• For each source of variation …
– The variation is in the SS column
– The degrees of freedom is in the df column
– The variance is in the MS column
– The MS value is found by dividing the SS by
the df
 ANOVA Table
• The complete ANOVA table can be
generated by most statistical packages
and spreadsheets
• We’ll concentrate on understanding how
the table works rather than the formulas
for the variations
 The ANOVA Table
Source SS df MS F
(variation) (variance)
Explained*
(treatment)
Error
(natural)
Total
The explained* variation has different names depending on the particular type
of ANOVA problem
 Example 1
Source SS df MS F

Explained 18.9 3

Error 72.0 16

Total

The Sum of Squares and Degrees of Freedom

are given. Complete the table.
 Example 1 – Find Totals
Source SS df MS F

Explained 18.9 3

Error 72.0 16

Total 90.9 19

Add the SS and df columns to get the totals.

 Example 1 – Find MS
Source SS df MS F

Explained 18.9 ÷3 = 6.30

Error 72.0 ÷ 16 = 4.50

Total 90.9 ÷ 19 = 4.78

Divide SS by df to get MS.

 Example 1 – Find F
Source SS df MS F

Explained 18.9 3 6.30 1.40

Error 72.0 16 4.50

Total 90.9 19 4.78

F = 6.30 / 4.50 = 1.4

 Notes about the ANOVA
• The MS(Total) isn’t actually part of the
ANOVA table, but it represents the sample
variance of the response variable, so it’s
useful to find
• The total df is one less than the sample
size
• You would either need to find a Critical
F value or the p-value to finish the
hypothesis test
 Example 2
Source SS df MS F

Explained 106.6 21.32 2.60

Error 26

Total

Complete the table

 Example 2 – Step 1
Source SS df MS F

Explained 106.6 5 21.32 2.60

Error 26 8.20

Total

SS / df = MS, so 106.6 / df = 21.32. Solving for df gives df = 5.

F = MS(Source) / MS(Error), so 2.60 = 21.32 / MS. Solving gives MS = 8.20.
 Example 2 – Step 2
Source SS df MS F

Explained 106.6 5 21.32 2.60

Error 213.2 26 8.20

Total 31

SS / df = MS, so SS / 26 = 8.20. Solving for SS gives SS = 213.2.

The total df is the sum of the other df, so 5 + 26 = 31.
 Example 2 – Step 3
Source SS df MS F

Explained 106.6 5 21.32 2.60

Error 213.2 26 8.20

Total 319.8 31

Find the total SS by adding the 106.6 + 213.2 = 319.8

 Example 2 – Step 4
Source SS df MS F

Explained 106.6 5 21.32 2.60

Error 213.2 26 8.20

Total 319.8 31 10.32

Find the MS(Total) by dividing SS by df. 319.8 / 31 = 10.32

 Example 2 – Notes
• Since there are 31 df, the sample size was
32
• Since the sample variance was 10.32 and
the standard deviation is the square root of
the variance, the sample standard
deviation is 3.21
Example3:
The tensile strength of Portland cement is being studied. Four
different mixing techniques can be used. A completely
randomized experiment was conducted and the following data
were collected.
Mixing Tensile Strength (lb/in2)
Technique

1 3129 3000 2865 2890

2 3200 3300 2975 3150
3 2800 2900 2985 3050
4
2600 2700 2600 2765

Test the hypothesis that

mixing techniques affect
the strength of the
cement. Use  = 0.05.
Example 4
A product developer is investigating the tensile strength of a new synthetic fiber
that will be used to make cloth for men’s shirts. Strength is usually affected by
the percentage of cotton used in the blend of materials for the fiber. The
engineer conducts a completely randomized experiment with five levels of
cotton content and replicated the experiment five times. The data are shown in
the following table.

Cotton
Weight
Percentage
Observations

15 7 7 15 11 9

20 12 17 12 18 18

25 14 19 19 18 18

30 19 25 22 19 23

35 7 10 11 15 11

Is there evidence to support the claim that cotton content affects the mean
tensile strength? Use  = 0.05.
Example4: Solution
Minitab Output
One-way ANOVA: Tensile Strength versus Cotton Percentage

Analysis of Variance for Tensile

Source DF SS MS F P
Cotton P 4 475.76 118.94 14.76 0.000
Error 20 161.20 8.06
Total 24 636.96

Yes, the F-value is 14.76 with a corresponding P-

value of 0.000. The percentage of cotton in the
fiber appears to have an affect on the tensile
strength.