Design of Experiments: New Methods and How to Use Them in Design, Development and Decision-making

Douglas C. Montgomery Regents Professor of Industrial Engineering & Statistics ASU Foundation Professor of Engineering Arizona State University doug.montgomery@asu.edu

Gore Lecture - UD 16 March 2011

Four Eras in the History of DOX

The agricultural origins, 1918 1940s R. A. Fisher & his co-workers Profound impact on agricultural science Factorial designs, ANOVA The first industrial era, 1951 late 1970s Box & Wilson, response surfaces Applications in the chemical & process industries The second industrial era, late 1970s 1990s Quality improvement initiatives in many companies Taguchi and robust parameter design, process robustness The modern era, beginning early 1990s
Gore Lecture - UD 16 March 2011

R. A. Fisher (1890 1962)

George E. P. Box

Gore Lecture - UD 16 March 2011

Design Optimality
One of the truly great paradigm shifts in DOX Can create custom designs for almost any situation Modern software makes this easy for at least some optimality criteria What optimality criteria should we use?

Gore Lecture - UD 16 March 2011

Regression and Design (ANOVA) Models are the Same Thing

Gore Lecture - UD 16 March 2011

Design Optimality Criteria:

X' X The D-criteria: M = N
Maximize the determinant of M

| XX | Deff = Max[| XX |]

1/ p

The G-criteria: Minimize the maximum SPV over R

(x)] NVar[ y 1 ( m ) (m) ( ) SPV = = N x ' X' X x 2

Geff =

p max( SPV )
xR

The I-criteria: Minimize the average prediction variance over R

(x)] 1 NVar[ y I= dx 2 AR
Gore Lecture - UD 16 March 2011

I eff =

Min( APV ( D)) APV ( D)

Example E-Commerce
Web page design is important Companies experiment with the design regularly This can be an ideal application of optimal designs Many factors Factors often have different number of levels Often lots of categorical factors
Gore Lecture - UD 16 March 2011

Example E-Commerce
k = 5 categorical factors One 5-level factor, one 4-level factor, one 3level factor, and two 2-level factors Consider a full factorial design: All possible combinations of factor levels 240 runs Thats a lot of web pages! What about a fractional factorial?
Gore Lecture - UD 16 March 2011

This is a D-optimal fractional factorial design constructed from JMP with 57 runs. It is nearly orthogonal No main effects are aliased with any two-factor interactions This is still a large experiment Lets consider a smaller design

Gore Lecture - UD 16 March 2011

30-run D-optimal fractional factorial design constructed from JMP It is nearly orthogonal

Gore Lecture - UD 16 March 2011

The 15-run design is nearly orthogonal

Gore Lecture - UD 16 March 2011

A Standard Factor Screening Problem Resolution IV Screening Designs in 16 Runs These are designs for 6, 7, and 8 factors Very widely used The generators for the standard designs are:
For six factors, E = ABC and F = BCD; for seven factors, E = ABC, F = BCD, and G = ACD; for eight factors, E = BCD, F =ACD, G = ABC, and H = ABD.
Gore Lecture - UD 16 March 2011

Alias Relationships: Six Factors

Gore Lecture - UD 16 March 2011

Alias Relationships: Seven and Eight Factors

Gore Lecture - UD 16 March 2011

Alias Relationships
In each design, there are seven alias chains involving only twofactor interactions These are regular fractions These are the minimum aberration fractions Because two-factor interactions are completely confounded experimenters often experience ambiguity in interpreting results
Resolve with process knowledge Additional experimentation
Gore Lecture - UD 16 March 2011

Example: Montgomery (2009), Design and Analysis of Experiments, 7E

This experiment was conducted to study the effects of six factors on the thickness of photoresist coating applied to a silicon wafer. The design factors are A = spin speed, B = acceleration, C = volume of resist applied, D = spin time, E = resist viscosity, and F = exhaust rate.

Gore Lecture - UD 16 March 2011

Interpretation: A, B, C, and E are probably real effects AB and CE are aliases Either some process knowledge or additional experimentation is required to complete the interpretation

Gore Lecture - UD 16 March 2011

Additional Experimentation: Fold-Over

Reverse the signs in column A, add another 16 runs to the original design This single-factor fold over allows all the two-factor interactions involving the factor whose signs are switched to be separated:

Gore Lecture - UD 16 March 2011

Results:

Gore Lecture - UD 16 March 2011

Other Design Augmentation Tricks

Partial fold-over, requiring only eight additional runs Optimal augmentation (using the D-optimality criterion) But what if additional experimentation isnt an option?

Gore Lecture - UD 16 March 2011

An Alternative to Additional Experimentation

While strong two-factor interactions may be less likely than strong main effects, there are many more interactions than main effects in screening situations. It is not unusual to find that between 15 and 30% of the effects in a screening design are active As a result, the likelihood of at least one significant interaction effect is high. There is often substantial institutional reluctance to commit additional time and material (and sometimes its impossible). Consequently, experimenters would like to avoid the need for a follow-up study. We show can lower the risk of analytical ambiguity by using a specific orthogonal but non-regular fractional factorial design. The proposed designs for six, seven, and eight factor studies in 16 runs have no complete confounding of pairs of two-factor interactions.
Gore Lecture - UD 16 March 2011

Background
Plackett and Burman (1946) introduced non-regular orthogonal designs for sample sizes that are a multiple of four but not powers of two. Hall (1961) identified five non-isomorphic orthogonal designs for 15 factors in 16 runs. Our proposed six through eight factor designs are projections of the Hall designs created by selecting specific sets of columns. Box and Hunter (1961) introduced the regular fractional factorial designs that became the standard tools for factor screening. Sun, Li and Ye (2002) catalogued all the non-isomorphic projections of the Hall designs. Li, Lin and Ye (2003) used this catalog to identify the best designs to use in case there is a need for a fold-over. For each of these designs they provide the columns to use for folding and the resulting resolution of the combined design. Loeppky, Sitter and Tang (2007) also used this catalog to identify the best designs to use assuming that a small number of factors are active and the experimenter wished to fit a model including the active main effects and all two-factor interactions involving factors having active main effects.
Gore Lecture - UD 16 March 2011

Metrics for Comparing Screening Designs

Consider the model y=X+e where X contains columns for the intercept, main effects and all two-factor interactions, is the vector of model parameters, and is the usual error vector. For six through eight factors in 16 runs, the matrix, X, has more columns than rows. Thus, it is not of full rank and the usual least squares estimate for does not exist because the matrix XX is singular. With respect to this model, every 16 run design is supersaturated. Booth and Cox (1962) introduced the E(s2) criterion as a diagnostic measure for comparing supersaturated designs:
2 x x ( ) i j < i j = E (s 2 ) = ,k k (k 1)

number of columns in X

Minimizing the E(s2) criterion is equivalent to minimizing the sum of squared off-diagonal elements of the correlation matrix of X.
Gore Lecture - UD 16 March 2011

The correlation is zero between all main effects and two-factor interactions (because the design is resolution IV) and the correlation is +1 between every two-factor interaction and at least one other twofactor interaction. If another member of the same design family had been used at least one of the generators would have been used with a negative sign in design construction and some of the entries of the correlation matrix would have been -1.

The Correlation Matrix for the Regular 26-2 Resolution IV Fractional Factorial Design

Gore Lecture - UD 16 March 2011

Alias Matrix
y =X+ 1 1
= y X+ 1 X 1
2

The model we fit

2 +

The true model

1 )= X 2 2 = + X X X1 1 + A 2 E ( ( ) 1 1 1 1

In a regular design, all entries in A are either 0 or 1. In a non-regular design, some entries will be 0 < |aij| < 1. A few non-regular designs have no 1 entries. The trace of AA is a measure of the total bias in a design.

Gore Lecture - UD 16 March 2011

Number of 16-Run Orthogonal Nonisomorphic Designs

Number of Factors 6 7 8 Number of Designs 27 55 80

Gore Lecture - UD 16 March 2011

Recommended 16-Run Six-Factor NoConfounding Design

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 B 1 1 -1 -1 1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 C 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 D 1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 -1 1 E 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 -1 1 1 -1 F 1 -1 -1 1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1

Gore Lecture - UD 16 March 2011

How Did We Find this Design?

The brute force approach complete enumeration Selected the design that minimized the trace of AA The trace is 6 Can also find using an optimal design algorithm:

min trace( AA) subject to: Deff > lD

Gore Lecture - UD 16 March 2011

Correlation Matrix (a) Regular 26-2 Fractional Factorial (b) the No-Confounding Design

Gore Lecture - UD 16 March 2011

Recommended 16-Run Seven Factor No-Confounding Design

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 B 1 1 1 1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 C 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 D 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 E 1 -1 1 -1 -1 1 -1 1 1 -1 -1 1 -1 1 1 -1 F 1 -1 -1 1 1 -1 -1 1 1 -1 1 -1 -1 1 -1 1 G 1 -1 -1 1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1

Gore Lecture - UD 16 March 2011

Correlation Matrix (a) Regular 27-3 Fractional Factorial (b) the No-Confounding Design
Gore Lecture - UD 16 March 2011

Recommended 16-Run Eight Factor No-Confounding Design

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 B 1 1 1 1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 C 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 D 1 1 -1 -1 -1 -1 1 1 1 -1 1 -1 1 -1 1 -1 E 1 -1 1 -1 1 -1 1 -1 1 1 -1 -1 -1 -1 1 1 F 1 -1 1 -1 -1 1 -1 1 1 -1 -1 1 -1 1 1 -1 G 1 -1 -1 1 1 -1 -1 1 1 -1 1 -1 -1 1 -1 1 H 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1 1

Gore Lecture - UD 16 March 2011

Correlation Matrix (a) Regular 28-4 Fractional Factorial (b) the No-Confounding Design
Gore Lecture - UD 16 March 2011

Design Comparison on Metrics

N Factors 6 Design Recommended Resolution IV Confounded Effect Pairs 0 9 E(s2) 7.31 10.97 Trace(AA) 6 0

Recommended Resolution IV

0 21

10.16 14.20

6 0

Recommended Resolution IV

0 42

12.80 17.07

10.5 0

Gore Lecture - UD 16 March 2011

Example Revisited
The No-Confounding Design for the Photoresist Experiment
Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 -1 B 1 1 -1 -1 1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 C 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 D 1 -1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 -1 1 E 1 -1 -1 1 1 -1 -1 1 1 -1 -1 1 -1 1 1 -1 F 1 -1 -1 1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 Thick ness 4494 4592 4357 4489 4513 4483 4288 4448 4691 4671 4219 4271 4530 4632 4337 4391

Gore Lecture - UD 16 March 2011

Lock X

Entered X X X X

Parameter Intercept A B C D

Estimate 4462.812 5 85.3125 -77.6825 -34.1875 0 21.5625 0 0 0 0 0 0 0 0 0 0 0 54.8125 0 0 0 0

nDF 1 1 1 2 1 2 1 1 1 2 1 2 1 2 1 2 2 1 2 2 0 2

SS 0 77634.37 64368.76 42735.84 31.19857 31474.34 2024.045 395.8518 476.1781 3601.749 119.4661 4961.283 60.91511 938.8809 3677.931 2044.119 1655.264 24035.28 2072.497 79.65054 0 5511.275

"F Ratio" 0.000 53.976 44.753 14.856 0.020 10.941 1.474 0.255 0.308 1.336 0.075 2.106 0.038 0.279 3.092 0.663 0.520 16.711 0.673 0.022 . 2.485

"Prob>F" 1 2.46e-5 5.43e-5 0.00101 0.89184 0.00304 0.25562 0.6259 0.59234 0.31571 0.78986 0.18413 0.84923 0.76337 0.11254 0.54164 0.61321 0.00219 0.53667 0.97803 . 0.14476

E F A*B A*C A*D A*E A*F B*C B*D B*E B*F C*D

C*E C*F D*E D*F E*F

Gore Lecture - UD 16 March 2011

If I only knew then what I know now

Gore Lecture - UD 16 March 2011

A personal experience with DOX: Wine-making. Original vineyard property purchased in 1983. Objectives were to begin as a grape supplier to other winemakers, then develop a winemaking process. Big problem: none of the partners were winemakers (how do you make a small fortune in wine). However, some partners knew the power of designed experiments. Wine-making is a chemical processI didnt know how to make polymer, either, when I took my first job as a chemical engineer.

Gore Lecture - UD 16 March 2011

There are many factors involved. One experiment per year is all that is feasible. Focus on Pinot Noir (Burgundy). Factors considered for one year (1985):

Gore Lecture - UD 16 March 2011

The experimental design for 1985 is a fractional factorial:

Gore Lecture - UD 16 March 2011

Results for 1985:

Taste

p
99

Normal % probability

A: PN CloneB: Oak Type C: Barrel Age B: Oak Type D: Yeast E: Stems C: Barrel Age F: Barrel Toast G: Whole Cluster D: Yeast H: Temp E: Stems F: Barrel Toast G: Whole Cluster H: Temp

A: PN Clone

95 90 80 70

50

A
30 20

AD
10 5

F D

[AD] = AD + CF + BH + EG
-3. 9 8 -2. 0 4 -0. 1 0 1. 84 3. 77

E ffe c t

Gore Lecture - UD 16 March 2011

Some of the results are interesting and useful, such as 1. toasting the barrel a little more seems like a good idea, and 2. it doesnt seem to matter much where the oak comes from. Some results are surprising such as no temperature effect! There is an interaction: [AD] = AD + CF + BH + EG Which effect(s) are real? CF and EG are more intuitive than AD, but we really dont have any process knowledge How would we normally resolve this? 1. Fold-over (is this even possible)? 2. Partial fold-over? 3. Would the recommended non-regular designs have been better?
Gore Lecture - UD 16 March 2011

How did we do? First commercial release, the1990 Pinot Noir, won a gold medal at the American Wine Competition 1991 release won a silver medal 1992 release won gold a medal, sixth best wine overall (of 2000 entries), best Oregon Pinot Noir Consistently ranked by The Wine Spectator as among the best Pinot Noir available, 90+ ratings:
Wine Spectator, Dec. 2010 93 - Corral Creek vineyards, 93 - Stoller Vineyards, 92 - Wind Ridge Vineyards, 91 - 3 Vineyard

Consistently highly rated in the International Pinot Noir Competition

Gore Lecture - UD 16 March 2011

Gore Lecture - UD 16 March 2011

Chehalemwines.com
Gore Lecture - UD 16 March 2011

Gore Lecture - UD 16 March 2011

Other Work
Appropriate analysis methods (other than stepwise regression)? Whats the power of these designs? How many two-factor interactions can we detect? What about the resolution III case (9-15 factors in 16 runs)?

Gore Lecture - UD 16 March 2011

References
Booth, K.H.V., Cox, D.R., (1962). Some systematic supersaturated designs. Technometrics 4, 489495. Box, G. E. P. and Hunter, J. S. (1961). The 2k-p Fractional Factorial Designs. Technometrics 3, pp.449458. Bursztyn, D. and Steinberg, D. (2006). Comparison of designs for computer experiments Journal of Statistical Planning and Inference 136, pp. 1103-1119. Hall, M. Jr. (1961). Hadamard matrix of order 16. Jet Propulsion Laboratory Research Summary, 1, pp. 2126. Jones, B. and Montgomery, D.C. (2010), Alternatives to Resolution IV Screening Designs in 16 Runs, International Journal of Experimental Design and Process Optimisation, Vol. 1, No. 4, pp. 285-295. Li, W., Lin, D.K.J., Ye, K. (2003) Optimal Foldover Plans for Two-Level Non-regular Orthogonal Designs Technometrics 45, pp.347351. Plackett, R. L. and Burman, J. P. (1946). The Design of Optimum Multifactor Experiments. Biometrika 33, pp. 305325. Loeppky, J. L., Sitter, R. R., and Tang, B.(2007) Nonregular Designs With Desirable Projection Properties. Technometrics 49, pp.454466. Montgomery, D.C. (2009). Design and Analysis of Experiments 7th Edition. Wiley Hoboken, New Jersey. Sun, D. X., Li, W., and Ye, K. Q. (2002). An Algorithm for Sequentially Constructing NonIsomorphic Orthogonal Designs and Its Applications. Technical Report SUNYSB-AMS-02-13, State University of New York at Stony Brook, Dept. of Applied Mathematics and Statistics.

Gore Lecture - UD 16 March 2011

Thank You!!
Questions?

Gore Lecture - UD 16 March 2011