American Society for Quality

A Systematic Approach to Planning for a Designed Industrial Experiment

Author(s): David E. Coleman and Douglas C. Montgomery
Source: Technometrics, Vol. 35, No. 1 (Feb., 1993), pp. 1-12
Published by: American Statistical Association and American Society for Quality
Stable URL: http://www.jstor.org/stable/1269280 .
Accessed: 17/03/2014 09:15

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Statistical Association and American Society for Quality are collaborating with JSTOR to digitize,
preserve and extend access to Technometrics.

http://www.jstor.org

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
? 1993American Statistical Association and FEBRUARY1993,VOL. 35, NO. 1
TECHNOMETRICS,
theAmericanSocietyforQualityControl

Editor's Note: This article and the firsttwo discussionswere presentedorally at the Technometricssession of the 36th Annual Fall
Technical Conferenceheld in Philadelphia,Pennsylvania,October 8-9, 1992. The conferencewas cosponsoredby the Chemical and
Process Industries,the StatisticsDivisions of the AmericanSociety forQuality Control,and the Section on Physicaland Engineering
Sciences of the AmericanStatisticalAssociation.

A Systematic
Approach to Planning
fora
Industrial
Designed Experiment
David E. Coleman Douglas C. Montgomery
Alcoa Laboratories Industrial Department
Engineering
Alcoa Center,PA 15069 ArizonaState University
Tempe,AZ 85287

Design of experimentsand analysisof data fromdesignedexperimentsare well-established

methodologiesin whichstatisticians are formallytrained.Anothercriticaland rarelytaught
skillis the planningthatprecedesdesigningan experiment.This articlesuggestsa set of tools
forpresentinggenerictechnicalissues and experimentalfeaturesfoundin industrialexperi-
ments.These tools are predesignexperimentguidesheetsto systematizetheplanningprocess
and to produceorganizedwrittendocumentation.They also help experimenters discusscom-
plex trade-offs betweenpracticallimitationsand statisticalpreferencesin the experiment.A
case studyinvolvingthe(computernumericalcontrol)CNC-machiningofjet engineimpellers
is included.

KEY WORDS: Industrialexperimentaldesign; Measurementerror;Nuisance factors;Sta-

tisticalconsulting.

1. INTRODUCTION periment,or a stepin sequentialexperimentationon

1.1 A Consulting Scenario
Considerthe followingscenario:An experimenter
fromthe process engineeringgroup comes to you
and says: "We are manufacturing impellersthatare
used in a jet turbineengine.To achieve the claimed
performanceobjectives,we mustproducepartswith
blade profilesthatcloselymatchthe engineeringde-
sign requirements.I want to studythe effectof dif-
ferenttool vendorsand machineset-upparameters
on the dimensionalvariabilityof the partsproduced
on the machinesin our CNC-machinecenter."
Many experimentaldesignapplicationsin industry
begin with such a statement.It is well recognized
that the planningactivitiesthat precede the actual
experimentare criticalto successfulsolutionof the
experimenters' problem(e.g., see Box, Hunter,and
Hunter 1978; Hahn 1977, 1984; Montgomery1991;
Natrella1979). Montgomery (1991) presenteda seven-
for
stepapproach planningexperiments, summarized
in Table 1. The firstthreeof these steps constitutes
the preexperimentplanning phase. The detailed,
specificactivitiesin thisphase are the focusof this
article.The emphasisis planningfora screeningex-
an existingproduct/process,
off-line
or on-line.Many
of the issues addressed,however,also applyto new
products/processesor research and development
(R&D) and to variousadditionalexperimentalgoals,
such as optimizationand robustnessstudies.
1.2 A Gap
It is oftensaid thatno experimentgoes exactlyas
planned, and this is true of most industrialexperi-
ments. Why? One reason is that statisticianswho
designexperiments withscientistsand engineers(the
"experimenters")usually have to bridge a gap in
knowledgeand experience.The consequencesof not
bridgingthisgap can be serious.
The statistician'slack of domain knowledgecan
lead to:
1. Unwarrantedassumptionsof process stability
duringexperimentation
2. Undesirable combinationsof control-variable
levels in the design
3. Violationor lackofexploitationofknownphys-
ical laws
4. Unreasonablylarge or small designs

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
2 DAVID E. COLEMAN AND DOUGLAS C. MONTGOMERY

Table 1. Steps ofExperimentation (1984) listedmostof these issues and made the rec-
ommendation,"The major mode of communication
1. Recognitionof and statementofthe problem betweentheexperimenter and thestatisticianshould
2.* Choice of factorsand levels
3.* Selectionofthe responsevariable(s) be face-to-facediscussion.The experimenter should,
4. Choiceof experimental design however,also be encouragedto documentas much
5. Conductionofthe experiment of the above information as possible ahead of time"
6. Data analysis (p. 25). Unfortunately, as Hahn observed,"Not all
7. Conclusionsand recommendations
experimentersare willingto prepare initial docu-
*In some situations, steps 2 and 3 can be reversed. mentation"(p. 26). Moreover,notall oftherelevant
issues may be thoroughlythoughtout-hence, the
5. Inappropriateconfounding
need for face-to-facediscussions,duringwhich,as
6. Inadequate measurement precision of re-
Hahn advised,"The statistician'smajorfunctions are
sponses or factors to help structurethe problem,to identifyimportant
7. Unacceptable predictionerror
issues and practicalconstraints,and to indicatethe
8. Undesirablerunorder
effectof variouscompromiseson the inferencesthat
The experimenter'slack of statisticalknowledgecan can be validly drawn for the experimentaldata"
lead to: (p. 21).
The guide sheetsproposedin thisarticleoutlinea
1. Inappropriate control-variablesettings(e.g.,
systematic"script"forthe verbalinteractionamong
rangetoo smallto observean effector rangeso large the people on the experimentation team. When the
thatirrelevantmechanismsdrivethe responsevari-
guide sheetsare completed,the team shouldbe well
able)
2. Misunderstanding of the natureof interaction equipped to proceed withthe task of designingthe
effects,resultingin unwiselyconfoundeddesigns experiment,takinginto accountthe needs and con-
straintsthusidentified.
3. Experimentaldesign or resultscorruptedby
measurementerroror settingerror 2. PREDESIGN MASTER GUIDE SHEET AND
4. Inadequate identification of factorsto be "held SUPPLEMENTARY SHEETS
constant"or treatedas nuisancefactors,causingdis-
The guidesheetsconsistofa "MasterGuide Sheet,"
tortedresults
5. Misinterpretation plus supplementary sheets and two tutorials.These
of past experimentresults,
are schematicallyillustratedin Figure 1. The sup-
affecting selection of response variables or control
variablesand theirranges plementary sheetsare oftenmoreconvenient foritems
3-7.
6. Lack of appreciationof different levels of ex-
The Master Guide Sheet is shown in Figure2. It
perimentalerror,leading to incorrect testsof signif-
icance is strippedof theblankspace usuallyprovidedto fill
in the information. Blank copies willbe providedby
This article attemptsto help bridge the gap by the authorson request.
providinga systematicframeworkforpredesignin- Discussion of issues relatedto differentpieces of
formationgatheringand planning.Specifically,we theMasterGuide Sheetand thesupplementary sheets
presentguide sheetsto directthiseffort.The use of follows.
the guide sheetsis illustratedthroughthe (computer Writingtheobjective(item2, Fig. 2) is harderthan
numericalcontrol)CNC-machineryexample briefly it appears to mostexperimenters. Objectivesshould
presentedpreviously.This articleis a consolidation be (a) unbiased,(b) specific,(c) measurable,and (d)
and extensionof thediscussionby Hahn (1984), Box of practicalconsequence. To be unbiased, the ex-
et al. (1978), Montgomery(1991), Natrella (1979), perimentation team mustencourageparticipationby
Bishop, Petersen,and Traysen(1982), and Hoadley knowledgeableand interestedpeople with diverse
and Kettenring(1990). perspectives.The data will be allowed to speak for
The guide sheetsare designedto be discussedand themselves.To be specificand measurable,the ob-
filledout bya multidisciplinary team
experimentation jectives should be detailed and stated so that it is
consistingof engineers,scientists,technicians/oper- clear whethertheyhave been met.To be ofpractical
ators, managers,and process experts.These sheets consequence,thereshould be somethingthatwillbe
are particularlyappropriateforcomplexexperiments done differently as a resultof the outcome of the
and forpeople withlimitedexperiencein designing experiment.This mightbe a change in R&D direc-
experiments. tion,a changein process,or a new experiment.Con-
The sheets are intendedto encourage the discus- ductingan experimentconstitutesan expenditureof
sion and resolutionofgenerictechnicalissuesneeded resourcesfor some purpose.
beforethe experimentaldesign is developed. Hahn Thus experimentalobjectivesshouldnotbe stated

TECHNOMETRICS,FEBRUARY1993,VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
 3
PLANNINGFOR A DESIGNED INDUSTRIALEXPERIMENT

GuideSheet
Master
Pre-design
1. Name, Organization,Title
2. Objectives
3. RelevantBackground
4. Response variables
5. Controlvariables
6. Factorsto be "held constant"
7. Nuisance factors
8. Interactions
9. Restrictions
10. Design preferences
11. Analysis& presentationtechniques
12. Responsibilityforcoordination
13. Trial run?
, H,

Figure1. Structure GuideSheets.

ofPredesignExperiment

as, "To show that catalystz14 works betterthan
catalystd12, if the technicianadjusts the electrode
voltagejust right."A betterobjectivewouldbe: "To
quantifythe efficiency difference,A, betweencata-
lystsz14 and d12 forelectrodevoltages7, 8, and 9
in the ABC conversionprocess-and assess statis-
ticalsignificance(compareto 95% ) and practicalsig-
nificance(A > 3C), perhapseconomicallyjustifying
one catalystover the other."
As Box et al. (1978, p. 15) put it (paraphrased),
the statisticianor othermembersof the experimen-
tationteamshould"ensurethatall interestedparties
agree on the objectives,agree on what criteriawill
determinethattheobjectiveshave been reached,and
arrangethat,ifthe objectiveschange, all interested
partieswillbe made aware of thatfactand willagree
on the new objectives and criteria." Even experi-
mentersin the physicalsciences-who have been
trained in the scientificmethod-sometimes need
proddingin this.
The objectiveof the experimentcan be metifthe
predesignplanningis thorough,an appropriatede-
sign is selected, the experimentis successfullycon-
ducted,the data are analyzedcorrectly,and the re-
sultsare effectively reported.By usinga systematic
approachto predesignplanning,thereis greaterlike-
lihood thatthe firstthreeconditionswilloccur. This
increasesthe likelihoodof the fourth.Then the ex-
perimentis likelyto produce its primaryproduct-
new knowledge.
2.1 Relevant Background
The relevantbackgroundsupportingthe objec-
tives should include information frompreviousex-
periments,routinelycollected observationaldata,
physicallaws, and expertopinion. The purposesof
providingsuchinformation are (a) to establisha con-
textfor the experimentto clearlyunderstandwhat
new knowledgecan be gained; (b) to motivatedis-
cussionabout the relevantdomainknowledge,since

FEBRUARY1993,VOL. 35, NO. 1

TECHNOMETRICS,

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
4 DAVIDE. COLEMANAND DOUGLASC. MONTGOMERY

1. Experimenter'sName and Organization:

BriefTitle of Experiment:
2. Objectivesof the experiment(shouldbe unbiased,specific,measurable,
and
ofpractical
consequence):
3. Relevant background on responseand controlvariables:(a) theoretical
relationships; (c) previousexperiments.Where
(b) expertknowledge/experience;
doesthisexperimentfitintothestudy
oftheprocessorsystem?:
4. List:(a) each response variable,(b) thenormal
responsevariablelevelat which
theprocessruns,thedistribution or rangeof normal (c) theprecision
operation, or
rangetowhichitcan be measured(andhow):
5. List:(a) each controlvariable,(b) thenormal controlvariablelevelat whichthe
processis run,andthedistribution
orrangeofnormal operation,(c) theprecision
(s) or
rangetowhichitcan be set (fortheexperiment, notordinaryplantoperations)andthe
precision to whichitcan be measured, (d) theproposedcontrolvariablesettings,and
(e) the predictedeffect(at least qualitative)thatthe settingswillhave on each
responsevariable:
6. List:(a) each factor
to be "held constant"intheexperiment, (b) itsdesiredlevel
and allowables or rangeof variation, (c) the precisionor rangeto whichit can
measured(andhow),(d) howitcan be controlled, and (e) itsexpectedimpact,ifany,
oneachoftheresponses:
7. List:(a) each nuisancefactor(perhapstime-varying),
(b) measurementprecision,
(c) strategy (e.g.,blocking, orselection),
randomization, and(d) anticipated
effect:
8. Listand labelknown
orsuspectedinteractions:
9. Listrestrictionson the experiment, e.g., ease of changingcontrolvariables,
methods ofdataacquisition,
materials,duration,number ofruns,typeofexperimental
unit(needfora split-plot orirrelevant
design),"illegal" experimental to
limits
regions,
randomization,runorder,
costofchanging a controlvariable etc.:
setting,
10. Givecurrent ifany,and reasonsforpreference,
design preferences, including
blockingand randomization:
11. Ifpossible,proposeanalysis and presentationtechniques, e.g., plots,
ANOVA, regression, t tests,etc.:
plots,
12. Who will be responsibleforthe coordinationof the experiment?
13. Should trialruns be conducted? Why/ whynot?

Figure2. PredesignMaster Guide Sheet. Thisguide can be used to help plan and design an experiment.
It serves as a
checklistto accelerateexperimentation
and ensuresthatresultsare notcorrupted
forlackofcarefulplanning.Notethatitmay
notbe possible to answerall questionscompletely.Ifconvenient,
use thesupplementarysheets for4-8.

suchdiscussionmaychangetheconsensusofthegroup, rate,a concentration,or a yield.Whatmakes a good

hence the experiment;and (c) to uncoverpossible
experimentalregionsofparticularinterestand others
that should be avoided. With this background,we
reduce the risksof naive empiricismand duplication
of effort.
For the CNC-matchingproblem introducedear-
lier, we have the guide sheet shownin Figure3.
3.
RESPONSE VARIABLES
As mentionedpreviously,items4-8 on the guide
sheet are most convenientlyhandled usingthe sup-
plementarysheets.The firstone is forresponsevari-
ables, as shownin Table 2.
Response variablescome to mindeasily formost
experimenters,at least superficially;
theyknowwhat
outcomestheywantto change-a strength, a failure
TECHNOMETRICS,FEBRUARY1993,VOL. 35, NO. 1
response variable? The answer to this question is
A
complex. complete answer is beyondthe scope of
thisarticle,but some guidelinescan be given.A re-
sponse variable
1. Is preferablya continuousvariable. Typically,
thiswill be a variablethatreflectsthe continuumof
a physicalproperty,such as weight,temperature,
voltage,length,or concentration.Binaryand ordinal
variableshave muchless information content-much
as the raw values are more informative than histo-
gramsthat have wide bins. Note thatbeing contin-
uous withrespectto a controlvariable may be im-
portant.If a responsevariablehas, perhaps,a steep
sigmoidalresponseto a controlvariable,it is effec-
tivelybinaryas thatvariablechanges.For example,

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
 PLANNINGFOR A DESIGNED INDUSTRIALEXPERIMENT 5

I ._ __
1.Experlmenter's Name and Organization: John Smith, Process Eng. Group
Brief Title of Experiment: CNC MachiningStudy
2. Objectives of the experiment (should be unbiased, specific, measurable, and
of practicalconsequence):
For machinedtitaniumforgings, the effectsof tool vendor;shiftsin a-axis, x-
quantify
axis, y-axis,and z-axis; spindlespeed; fixtureheight;feed rate;and spindlepositionon
the average and variability in blade profileforclass X impellers,such as shown in
Figure4.
3. Relevant background on response and control variables: (a) theoretical
relationships;(b) expert knowledge/experience; (c) previous experiments. Where
does thisexperiment fitintothestudyoftheprocess orsystem?:
(a) Because of tool geometry,x-axis shiftswould be expected to produce thinner
blades, an undesirablecharacteristic ofthe airfoil.
of
(b) This family parts has been producedforover 10 years; historicalexperience
indicatesthatexternally regroundtools do notperform as well as those fromthe
"internal"vendor(our own regrindoperation).
(c) Smith (1987) observed in an internalprocess engineeringstudy that current
spindle speeds and feed rates workwell in producingparts that are at the
nominalprofilerequiredbythe engineeringdrawings- but no studywas done of
to variationsin set-upparameters.
the sensitivity
Results of thisexperimentwillbe used to determinemachineset-up parametersfor
impellermachining.A robustprocess is desirable;thatis, on-targetand lowvariability
performance reoardlessofwhichtoolvendoris used.

Figure3. Beginningof Guide Sheet forCNC-MachiningStudy.

weightofprecipitate as a function
ofcatalyst
maybe
nearzero fortheselectedlow levelsofcatalyst and
nearmaximum forthehighlevels.
2* Shouldcapture,as muchas possible,a quantity
or qualityof interestfortheexperimental unit.For
example, ifthe experimentalunitis an ingotand a
responseis T = temperature, itmaymatter whether
T is takenata singlepointoraveragedovera surface
region,theentiresurfacearea, or theentireingot
volume.
3. Shouldbe in appropriate units.The unitsmay
be absolute,suchas pounds,degreescentigrade, or
meters.Theymaybe relativeunits,suchas percent
ofconcentration byweightor byvolumeor propor-
tionaldeviationfroma standard.Whatis "appro-
priate"maybe determined byan empirical or first-
principlesmodel,suchas usingabsoluteunitsin E
=mc2, or it maybe determined by practicallimi-
tations,such as usingpercentof concentration by
weightbecausetheexperimental samplesare notall
thesameweight.
4. Shouldbe associatedwitha targetor desirable

Table 2. Response Variables

Measurement Relationshipof
Response variable Normaloperating precision,accuracy- responsevariableto
(units) level and range how known? objective
Blade profile Nominal(target) crE- x 10-5 inches Estimatemean
(inches) ? 1 x 10-3 inchesto froma coordinate absolutedifference
? 2 x 10-3 inchesat measurement fromtargetand
all points machinecapability standarddeviation
study ofdifference
Surfacefinish Smoothto rough Visualcriterion Should be as smooth
(requiringhand (compareto as possible
finish) standards)
Surfacedefect Typically0 to 10 Visualcriterion Mustnotbe
count (compareto excessive in
standards) numberor
magnitude

TECHNOMETRICS, FEBRUARY 1993, VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
 6 DAVIDE. COLEMANAND DOUGLAS C. MONTGOMERY
condition(whichmotivatesthe experiment).Such a
comparisonmightbe used to derive "performance
measures"fromresponse-variable outcomes.For ex-
ample, with CNC-machining, blade profileis a re-
sponse variable, and it is compared to the target
profilebycomputing differences at certainlocations.
Mean absolutedifference and thestandarddeviation
of the differences are performancemeasuresforthe
various experimentalconditions.They can be ana-
lyzed separatelyor by usingthe standarddeviations
to computeweightsforthe mean analysis.
5. Is preferablyobtained by nondestructiveand
nondamagingmethodsso thatrepeatedmeasurescan
be made and measurementerrorcan be quantified.
6. Should not be near a naturalboundary.Other-
wise, the variablewill not discriminate well. For ex-
ample, it is hard to a of
distinguish yield 99.5% from
99.8%, and it is hard to detectand distinguishcon-
taminationlevels near 0.
7. Preferablyhas constantvarianceovertherange
of experimentation.
There are other importantcharacteristicsof re-
sponsevariablesthattheexperimenters maynothave
considered or communicatedto the whole experi-
mentationteam. This sheet helps to drawthemout:
(a) currentuse, ifany (col. 2); (b) abilityto measure
(col. 3); and (c) the knowledgesoughtthroughex-
perimentation(col. 4).
It is helpfulto know the currentstate of use, and
if it is unknown,the experimentersare advised to
include some trial runs priorto the experimentor
"checkpoint" runs duringthe experiment(perhaps
these data have not been previouslyacquired). The
currentdistribution servesas one of severalpossible
referencedistributions forjudgingthepracticalmag-
nitudeof the effectsobserved. Given a typicalstan-
dard deviationfor a response variable of a, a low-
to-highcontrol-variableeffectof 0(/2may be of no
practicalsignificance,but one of 4ormay be impor-
tant.Anotheradvantageto knowingthecurrentstate
of use is a check on credibility.Process or design
limitationsmay constraina responsevariable to be
boundedon one or twosides. An experimentalresult
outside that range may be erroneousor due to an
abnormal mechanism(which may, however, be of
interest).
Measurementprecision(and, in some cases, bias)
and how to obtain it (i.e., choice of measurement
systemor repeated measurements)is a thornin the
flesh for many experimenters.The admonitionof
Eisenhart(1962) servesas a relevant(if overstated)
warning,"until a measurementoperation . . . has
attaineda state of statisticalcontrolit cannotbe re-
garded in any logical sense as measuringanythingat
all" (p. 162). It has been our experiencethatmany
TECHNOMETRICS,FEBRUARY1993,VOL. 35, NO. 1
This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM
All use subject to JSTOR Terms and Conditions
experimenters do not know the state of controlnor
the precisionand bias of mostmeasurementsystems
measuringa responseor a controlvariable.The mea-
surementsystemswere not useless,theywere just of
unknownutility.Importantand poorly understood
systemsshouldbe evaluatedwitha measurementca-
pabilitystudy,a designedexperiment ofthemeasure-
mentprocess.As a compromise,one mightbe forced
to resortto historicaldata and experienceor weaken
the experimentalobjectiveto obtain ranking,selec-
tion,or a binaryresponseinsteadof quantification.
The relationshipof a responsevariableto the ob-
jective may be direct.An objective may be defined
in termsof a response variable-for example, "to
quantifytheeffectthatthermalcycleB has on tensile
strength measuredon customerqualifying testerX."
In the case of CNC-machining,a responsevariable
is blade profile(see Fig. 4). This is related to the
objectivethroughtwomeasurement-performance in-
dicators-mean absolute difference of blade profile
and the target,and standarddeviationof the differ-
ence. Sometimesa responsevariable may be a sur-
rogateforthe trueresponseof interest.This is often
the case in destructivetesting,in whicha standard
stress-to-fracture test, for example, representsper-
formanceunderconditionsof use. Anotherexample
is yield rate or failurerate, which are inferiorre-
sponses that oftenrepresentwhere a specification
fallsrelativeto a distribution ofcontinuous-scaleval-
ues (the collectionof whichprovidessuperiorinfor-
mation).
As discussedpreviously,the relationshipof a re-
sponse variableto the objectivemaybe throughper-
formancemeasuresthatinvolvea comparisonof the
responseto a targetor desirableoutcome.
4. CONTROL VARIABLES
As withresponsevariables,mostinvestigators can
easily generatea listof candidatecontrolvariables.
Controlvariablescan be attributeor continuous.
They can be narrowlydefined,such as "percentof
copper, by weight," or broadly defined, such as
"comparablyequipped pc: Apple or IBM." In either
case, controlvariablesshould be explicitlydefined.
When discussingpotentialcontrolvariables with
experimenters, it may be helpfulto anticipatethat
held-constant factorsand nuisancefactorsmustalso
be identified.Figure 5 is a Venn diagramthat can
be used to help select and prioritizecandidate fac-
tors. It illustratesdifferent
categoriesof factorsthat
affectresponsevariables,based on threekey char-
acteristics-magnitude of influence on response
variables,degree of controllability,and measurabil-
ity(e.g., precisionand accuracy).Each typeoffactor
is discussedin detailin followingsections.A descrip-
 PLANNINGFORA DESIGNED INDUSTRIALEXPERIMENT 7

Figure4. JetEngineImpeller(side view;z axis is vertical, and yaxis is intothepage): 1.HeightofWheel;

x axis is horizontal,
2. Diameterof Wheel;3. InducerBlade Height;4. ExducerBlade Height;5. Z HeightofBlade.

tionofthediagramis as follows: 4.1 CurrentUse (col. 2)

1. Controlvariablesaremeasurable, controllable, Therearetworeasonsithelpstoknowtheallowed
and thought to be (very)influential. rangesandnominal valuesofcontrol
variablesunder
2. Held-constant factors are controlled. current use. First,thedegreetowhichhistoricalpro-
3. Nuisancefactors areuncontrolled factors(either cess data can be used to gain relevantknowledge
theycannotbe controlled, or theyare allowedto maybe revealed.This is discussedin Section4.2.
vary). Second,theexperimenter shouldselecta rangelarge
In discussing different variablesand factorsthe enoughto producean observableeffect andto span
a good proportion of theoperatingrange,yetnot
teammaychoosetoreassign variablesfrom onegroup
chooseso greata rangethatno empirical modelcan
to another,and thisis partof theordinary process be postulated fortheregion,as discussedin Section
forplanninga designedexperiment. For theCNC-
4.3. In some,less matureexperimental situations,
machining problem, thecontrol-variable information
theremaybe nowell-defined "current
use,"inwhich
wasdevelopedas showninTable 3; thosebelowthe
case trialrunsbeforeor duringexperimentation are
spaceareconsidered to be ofsecondary importance. are with variables.
Similartotheresponsevariablessheet,thecontrol helpful-as they response
variablessheetsolicitsinformation about(a) current
4.2 Abilityto Measure and Set (col. 3)
use (col. 2), (b) abilityto measureand set (col. 3),
and (c) knowledge soughtthrough experimentation Withcontrolvariables,
thereis an additional
con-
(cols. 4-5). siderationrarelymentionedin the literature.
The
experimentationteamnotonlyneedsto knowhow
"heldconstant"
factors measurementswill be obtainedand the precisionof
measurement,O'm,but also how thecontrolvariable
|JinfluentialK controllable willbe obtainedand "settingerror,"er.These
settings
different
typesofdeviationfromtheidealhavedif-
control
ferenteffects
on experimentation.
Large o-mwillmean
: .':S!
:--
:.variables
thateithererrors-in-variables
methodswillhaveto
be used (e.g., methodsthatwillallowestimation
of
biasineffects or,alternatively,
estimates) manysam-
nuisance measurable
factors pleswillhavetobe collectedformeasurementduring
experimentation to get an acceptablysmall Olm/li,
Figure 5. Different
Categoriesof Factors Affecting
Re- especiallyif e,slis also large.If IEs is large,traditional,
sponse Variables. class-variable-based
analysisofvariancewillhaveto

TECHNOMETRICS, FEBRUARY 1993, VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
8 DAVIDE. COLEMANAND DOUGLASC. MONTGOMERY

Table 3. ControlVariables

Measurement
precisionand Proposedsettings, Predictedeffects
Controlvariable Normallevel settingerror- based on (forvarious
(units) and range how known? predictedeffects responses)
x-axisshift* 0-.020 inches .001 inches 0, .015 inches Difference
/
(inches) (experience)
y-axisshift* 0-.020 inches .001 inches 0, .015 inches /
Difference
(inches) (experience)
z-axisshift* 0-.020 inches .001 inches ? Difference
?
(inches) (experience)
Tool vendor external -Internal,
Internal, external Externalis more
variable
a-axis shift* 0-.030 degrees .001 degrees 0, .030 degrees Unknown
(degrees) (guess)
Spindlespeed 85- 115% -1% 90%, 110% None?
(% of (indicator
nominal) on control
panel)
Fixtureheight 0-.025 inches .002 inches 0, .015 inches Unknown
(guess)
Feed rate(% of 90--110% -1% 90%, 110% None?
nominal) (indicator
on control
panel)
*Thex, y,and z axes are used to referto thepartand theCNCmachine.Thea axis refersonlyto themachine.

be replacedbyregression Theresultoflarge
analysis. variable,low/high settingsshouldbe selectedto cause
settingvariation
may be unwanted aliasing,greater a predictedeffect(maineffect)forthe "key" re-
error,violation
prediction ofexperiment constraints, sponsevariableequal to one standarddeviationof
and difficulty
conducting analyses.
split-plot itsvariationinordinary use,ap (ifthereis "ordinary
Often,one findsthatarm |Esl, such as when the use"). Thisis a largeenoughchangein responseto
measurement systemispartofa controller,andequi- havepractical consequenceandalso largeenoughto
libriumconditionscan be achieved.Measurement likelybe detectedifmeasurement erroris negligible
precisionand settingerrorare not alwayscompa- and theexperiment has enoughruns.If theruleof
rable,however.Forexample,am< |Es| isnotunusual thumbis followed, everycontrol variablehas"equal
fora continuous-batchmixingprocess.Supposethat opportunity" to affect theresponsevariable.
theconcentrationofconstituent A is at 10% and is Naturally, it is harderto suggestsucha rulefor
reducedtowards
continuously a targetof5%. Batches immature processes.Moreover, otherissuesandcon-
might be producedwithconcentrations of10%,7%, straintsmustbe takenintoaccountwhensettings are
and4%. In thiscase,perhapsIeJl 1%, buta spec- selected-safety,discreteness of settings,process
trographmay measurewitha-, s .1%. Anotherex- constraints,ease ofchanging a setting,andso forth.
ampleisa thermostat, whichoftenprovidesrm< |Esl, Theseare solicitedin item8 oftheguidesheet.
especiallyifit has a "dead zone" in itslogic. Predictedeffects forthe responsevariablesmay
Alternatively,one may findarm > Esl. For exam- be availablefromtheknowledge sourcespreviously
lawsmaymakeitpossibleto accurately
ple,physical listed-theory,experts,and experiments. Quanti-
setgaspressure
ina sealedcavitybysetting
gastem- tativepredictedeffectsare preferable, butexperi-
buttheremaybe noprecisewaytodirectly
perature, menters maynotbe able to providemorethanqual-
measurepressure. itativeindications. Evenifuncertain, theexerciseof
to
attemptingpredict the outcome ofthe experiment
4.3
beforeitis runcanfoster goodinteraction within
the
Knowledge Sought Through team and oftenleads to revised
Experimentation experimentation
choicesof settings. An additionaladvantageis that
In thedesignofexperiments classeshe teachesat thepredictions willalwaysbe wrong,so it is easier
Alcoa, J. S. Huntergivesa ruleof thumbforex- to see whatknowledge has beengainedthrough ex-
periments on existingprocesses.For each control perimentation.
TECHNOMETRICS,FEBRUARY1993,VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
 PLANNINGFORA DESIGNED INDUSTRIALEXPERIMENT 9

5. HELD-CONSTANT FACTORS ing mayinteractwithexperimental variables.The

Held-constant factorsarecontrollable factors operator's role in this automated
highly processis
whose
effectsare notof interest in thisexperiment. small, and material of
properties the blanktitanium
Most
thinkintheseterms:"Forthisexper- forgings are carefully
controlledbecauseofthecrit-
experimenters
iment,I wanttostudytheeffect offactors A, B, and icalityofthepart.
C on responses y, andY2,butall othercontrolvari- 6. NUISANCE FACTORS
ablesshouldbe heldat theirnominalsettings, andI
do not wantextraneousfactorsdistorting the re- Processesvaryovertime.Experimental conditions
sults."Thissheetwas developedto ensurethe"all varyover time."Identicalsamples"differ.Some
othercontrol variablesheldattheirnominal settings" variations are innocuous,someare pernicious. Ex-
condition.(Thenextsheetis usedtohelpensurethat amplesincludecontamination ofprocessfluidsover
"thereare no extraneous factorsdistorting there- time,equipment wear,build-upofoxideson tools,
sults.")For theCNC-machining the
example, held- and so forth.Some of thesecan be measuredand
constant factorsare as shownin Table 4. monitored to at least ensurethattheyare within
The sheetin Table 4 can forcehelpfuldiscussion limits;othersmustbe assessedsubjectively by ex-
aboutwhichfactorsare adequatelycontrolled and perts;stillothersare unmeasured. Nuisancefactors
whichfactorsare not. In so doing,it is oftennec- are notcontrolled, and are notof primary interest
essaryto consultexpertsto helpprioritize factors, in thisexperiment. Theydiffer fromheld-constant
recommend preexperiment studies to assess control, factorsin thattheycannotbe deliberately set to a
or developcontrolstrategy. colstantlevelforall experimental units.Ifthelevel
Forexample, intheCNC-machining case,thissheet can be selectedforanyexperimental unit,however,
resultedin therecognition of thefactthatthema- blocking or randomization be
might appropriate. If
chinehad to be fullywarmedup beforecutting any levelscannotbe selected(i.e., thelevelsofthefactor
blade forgings. The actualprocedureused was to are unpredictable, perhapscontinuous),thenthe
mountthe forgedblankson the machinespindles nuisancefactorbecomesa covariatein theanalysis.
and runa 30-minute cyclewithoutthecutting tool If a nuisancefactoris notmeasurableand thought
engaged. This would allow all machine and
parts the to be veryinfluential, it mayalso be calledan ex-
lubricantto reach normal,steady-state operating perimental risk factor. Such factorscan inflateex-
temperature.Theuseofa "typical" (i.e., "mid-level") perimental error,makingit moredifficult to assess
operatorand theblockingof theblankforgings by thesignificance of controlvariables.Theycan also
lot numberweredecisionsmade forexperimental bias the results.For the CNC-machining example,
insurance,although neither variablewasexpectedto thenuisancefactors are as shownin Table 5.
haveimportant effects.Notthatitis notpractical or Experiment designers havea setof passivestrat-
desirableto holdsomefactors constant.For exam- egies (randomization, blocking,analysisof covari-
ple,although itmight be idealto haveexperimental ance,stratifiedanalysis)to reducetheimpactofnui-
materialfromonlyone titanium forging, theremay sancefactors. Thesestrategies canhavea majoreffect
notbe enoughmaterial within oneforging, andforg- ontheexperimental design.Theymaybe constrained

Table 4. Held-Constant
Factors

Desired experi- Measurement

Factor mental level and precision-how How to control Anticipated
(units) allowablerange known? (inexperiment) effects
Typeof cutting Standardtype Notsure,but Use one type None
fluid thoughtto be
adequate
Temperature of 100-110?F.when 1-2? F. (estimate) Do runsafter None
cuttingfluid machineis machinehas
(degrees F.) warmedup reached100?
Operator Severaloperators Use one "mid- None
normally work level"
in the process operator
Titanium Material Precisionof lab Use one lot Slight
forgings propertiesmay testsunknown (or blockon
varyfromunit forginglot,
to unit onlyif
necessary)

TECHNOMETRICS, FEBRUARY 1993, VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
10 DAVIDE. COLEMANAND DOUGLASC. MONTGOMERY

Table 5. NuisanceFactors

Measurement Strategy(e.g.,
Nuisancefactor precision-how randomization,
(units) known? blocking,etc.) effects
Anticipated
Viscosityof Standardviscosity Measureviscosityat None to slight
cuttingfluid startand end
Ambient 1-2? F. by room Make runsbelow Slight,unlessvery
temperature (?F.) thermometer 80?F. hotweather
(estimate)
Spindle -Block or randomize Spindle-to-spindle
on machinespindle variationcould be
large
Vibrationof ? Do notmove heavy Severe vibrationcan
machineduring objectsin CNC introducevariation
operation machineshop withinan impeller

by limitson the numberof observations,costs of
changingcontrol-variablesettings,and logisticcon-
siderations.In the CNC-machining example,theonly
nuisancefactorto have potentially
seriouseffectsand
forwhichblockingseems appropriateis the machine
spindle effect(though it may be necessaryto also
block on titaniumforgings).The machinehas four
spindles,requiringa designwithfourblocksor ran-
domizingon all four.Blockingwill introducea bias
in the estimatesconfoundedwiththe blockingvari-
able(s), whereas randomizationwill inflatethe ex-
perimentalerror. The other two factorsare dealt
withby ensuringthattheystaybelow levelsat which
problemsmay be encountered.
7. INTERACTIONS
The interactionssheet is self-explanatory. Unfor-
tunately,the concept of interactionsis not self-ex-
planatory-even among intelligent,mathematically
inclinedpeople in thesciences.Hence, as partof the
package of guide sheets, it is helpfulto include a
tutorial.The graphicalportionof the tutorialis pre-
sented in Figure 6. An additional, expositoryde-
scriptionof interactionsis sometimesincluded,but
it is notshownhere. The interactions table explicitly
recognizesonlypairwiseinteractions of linearterms.
It providesan opportunityforthe experimenters to
capture knowledgeor speculationthatcertainpair-
wise interactionsmay be presentand othersare un-
likelyto be present.This inputis helpfulwhen the
experimentis laterdesigned-to choose resolution,
or more generallyto choose whicheffectsshould or
should not be confounded.Higherordereffectsmay
also be importantbut are not capturedin the guide
sheets.The interactionsheetfortheCNC-machining
example is shownin Table 6.
A helpfulway to use this matrixis to avoid dis-
cussingeverypossible pairwiseinteractionone at a
time but instead use the process of eliminationor
inclusion; that is, if interactionsare generallyim-
portant,a question can be posed: "Are there any
interactionsthatare arguablynot present?"If main
effectsdominateinteractions,a questioncan be posed:
"Are thereany interactionsthatmustbe estimated
clear of maineffects?"Alternatively, a secret-ballot
vote on potentially
important interactionscan be held
among experimenters and other knowledgeable in-
vestigators,with each receiving,say, 100 votes to
spread among the interactions.
The remaining itemsare foundon theMasterGuide
Sheet.
8. RESTRICTIONS, PREFERENCES FOR THE
DESIGN, ANALYSIS, AND PRESENTATION
Box, Hunter, and others have repeatedly ex-
horted,"Attentionto detail can determinethe suc-
cess or failureof the experiment."Item 8 in Figure
2 is partof heedingthatadvice. Theoreticaloptimal
experimentaldesign and practicalexperimentalde-
sign are oftenworlds apart, and restrictions often
make the difference.Since a singleunknownrestric-
tion can renderworthlessan otherwisewell-consid-
ered, laboriouslydeveloped design,the statistician
should encourageexperimenters to be quick to put
these limitationsand pitfallson the table. In partic-
ular, thereappears to be a lack of awarenessin the
applied statisticscommunity of theprevalenceof ex-
perimentswithunidentified split-plotstructure.Be-
cause it is unidentified(differentexperimentalunits
used fordifferent partsof the experiment),the anal-
ysisis oftendone incorrectly-usingthewrongerror
termsto test statisticalsignificance.The discussion
of the issues surrounding the choice of experimental
unitand analysisstrategiesgoes beyondthescope of
thisarticlebut should take place on the experimen-
tationteam.
Items 10 and 11 of Figure2 are intendedforthe
followingthree circumstances:First,when experi-
mentersare statistically
sophisticated and have a good
idea of appropriatedesignsor analysistechniques;

TECHNOMETRICS, FEBRUARY 1993, VOL. 35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
 PLANNINGFORA DESIGNEDINDUSTRIALEXPERIMENT 11

Interactions
(Tutorial)
* Taylor
seriesapproximation:
f(x,y)= ao + aix + bly + c11xy+ a2x2+ b2y2+..
Plot la Plot lb
R No AoB Interaction R A B Interaction I
e e
s Low B
P ps Low B

n High BI B
s
$ ~- High
e e
I i
IowI A HIg
Low A High Low A High
A .A

Simple, linear, additive Factors A and B interact,but no

model is sufficient. quadratic term is present (assumed).

Table la - No A-B Interaction Table lb - A-B Interaction

Low A High A Low A High A

Low B 5 6 Low B 5 5.2

HighB 3 4 HighB 3 4

response

Figure6. Graphical(tutorial)
Presentation
ofInteractions.

second, when the experimenthas been preceded by
experimentsinwhicha particulardesignor technique
provedto be useful;third,when,on consideringde-
signs, analyses, and plots, the experimentersmay
want to change informationin items2-7-for ex-
ample, narrowingthe scope of the objective or in-
creasingthenumberofsettingsfora controlvariable.
9. THE NEXT STAGE
By the time the experimentation team has come
to a consensusconcerningthe information collected
in items1-10 of the guide sheet, the statistician(or
surrogate)will have had the opportunity to step be-
yond the generic confines of the guide sheet and
discuss more problem-specific issues thatwill affect
the experimentaldesign,such as multilevelfactors,
Table 6. Interactions
different sizes of experimentalunits,and logistics.
Then, it may be usefulto (a) choose candidate de-
signs,(b) reviewthemwiththe experimenters in the
contextof the collectedinformation to determineif
any of the designsshould be dropped fromfurther
consideration,and (c) writean experimentaldesign
proposal that contains(at least) one or more pro-
posed designs;a comparativeanalysisof the designs
withrespectto numberofruns,resolution(or aliased
effects),numberof distinctcontrolvariable combi-
nations,predictionerrorstandarddeviation,and so
forth;a design recommendationwithjustification;
and copies of the completedguide sheets.
When an experimentaldesign has been selected,
the sheets are used to help launch supportivetasks
requiredfor the experimentto be successful.This

Control
variable y shift z shift Vendor a shift Speed Height Feed
x shift P
y shift -
z shift - -
Vendor - -
a shift
Speed - F, D
Height --

NOTE: Response variables are P = profiledifference,F - surface finish,and D = surface defects.

TECHNOMETRICS,
FEBRUARY1993,VOL.35, NO. 1

This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM

All use subject to JSTOR Terms and Conditions
12 DAVIDE. COLEMANAND DOUGLAS C. MONTGOMERY
involvesissues addressed in items 11 and 12 on the
guide sheet. Additionally,there will be logistical
planningand planningfor measurementcapability
studies, process capabilitystudies, preexperiments
to quantifythe effectsof various factors(held-con-
stantand nuisance) on responsevariables,and trial
runs.
In regardto item 12 of Figure 2, an experiment
withouta coordinatorwill probablyfail. Though a
statisticiancan playthisrole, itis oftenbetterplayed
by another memberof the experimentationteam,
who can "champion" the experimentamong peers.
The statistician(or surrogate)can play a strongsup-
port role and be primarilyresponsiblefor that in
whichhe or she is professionally trained-the design
and analysisoftheexperimentand nottheexecution.
Finally,consideringitem13 oftheguidesheet,the
teamshouldentertaintheidea oftrialrunsto precede
the experiment-especially if this is the firstin a
series of experiments.A trial run can consistof a
centerpointrunor a small part (perhaps a block) of
the experiment.The firstand most importantpur-
pose of trialrunsis to learn and refineexperimental
procedureswithoutriskingthe loss of time and ex-
pensiveexperimentalsamples. Most experimentsin-
volve people (and sometimesmachines)doingthings
thattheyhave neverdone before.Usuallysome prac-
tice helps.
A second importantreason fortrialrunsis to es-
timateexperimentalerrorbefore expendingmajor
resources.An unanticipatedlarge experimentaler-
ror could lead to cancelingor redesigningthe ex-
periment,wideningtherangesof settings,increasing
the numberof replicates,or refining the experimen-
tal procedure.An unanticipatedsmall experimental
error(does thisever reallyhappen?) could have op-
posite effectson plans or cause theexperimenters to
reassess whetherthe estimateis rightor complete.
A thirdreason is thattrialrunsare also excellent
opportunitiesto ensurethatdata-acquisitionsystems
are functioning and willpermitexperimentalrunsto
be conductedas fastas had been planned.
Last, a fourthreason is that trialruns may yield
resultsso unexpectedthatthe experimenters decide
to change theirexperimentalplans.
Naturally,the feasibilityand advisabilityof con-
ductingtrial runs depends on the context,but the
experimentteams in whichwe have been involved
have neverregretted conductingtrialruns.Some trial
runshave saved experimentsfromdisaster.
10. SUMMARY
To conductcomplexexperiments, carefulplanning
withattentionto detailis critical.Predesignplanning
TECHNOMETRICS,FEBRUARY1993,VOL. 35, NO. 1
This content downloaded from 134.148.29.34 on Mon, 17 Mar 2014 09:15:43 AM
All use subject to JSTOR Terms and Conditions
is one partof the processby whichexperimentsare
conceived,planned,executed,and interpreted.It is
oftenthe part claimed by no one, hence it is often
done informally-andsloppily.The use ofpredesign
experimentguide sheets providesa way to system-
atize the processby whichan experimentation team
does thisplanning,to help people to (a) moreclearly
definetheobjectivesand scope of an experimentand
(b) gatherinformation needed to design an experi-
ment.
ACKNOWLEDGMENTS
We much appreciatethe patienttoleranceof the
people who cooperated in the firstuse of this sys-
tematicapproach,especiallyE. Malecki,R. Sanders,
G. S. Smith,and R. Welsh,who providedthe initial
opportunity.G. Hahn, membersof the Alcoa Lab-
oratoriesStatisticsgroup (L. Blazek, M. Emptage,
A. Jaworski,K. Jensen,B. Novic, and D. Scott),P.
Love, and M. Peretic also provided useful insight
and comments.The thoughtful commentsprovided
by the and
referees editorconsiderablyimprovedthe
article.
[ReceivedJuly1991. RevisedApril 1992.]
REFERENCES
Bishop,T., Petersen,B., and Trayser,D. (1982), "AnotherLook
at theStatistician'sRole inExperimentalPlanningand Design,"
The AmericanStatistician, 36, 387-389.
Box, G. E. P., Hunter,W. G., and Hunter,J. S. (1978), Statistics
for Experimenters, New York: JohnWiley.
Eisenhart,C. (1962), "Realistic Evaluation of the Precisionand
Accuracyof InstrumentCalibrationSystems,"Journalof Re-
searchof theNationalBureau of Standards,67C, 161-187.
Hahn, G. (1977), "Some Thing EngineersShould Know About
ExperimentalDesign," Journalof QualityTechnology,9, 13-
20.
(1984), "Experimental Design in a Complex World,"
Technometrics, 26, 19-31.
Hoadley, A., and Kettenring,J. (1990), "CommunicationsBe-
tween Statisticiansand Engineers/Physical Scientists" (with
commentary),Technometrics, 32, 243-274.
Hunter,W. G. (1977), "Some Ideas About Teaching Design of
ExperimentsWith25 Examples of ExperimentsConducted by
Students,"The AmericanStatistician, 31, 12-17.
McCulloch, C. E., Boroto, D. R., Meeter, D., Polland, R., and
Zahn, D. A. (1985), "An Expanded Approach to Educating
StatisticalConsultants," The American Statistician,39, 159-
167.
Montgomery,D. C. (1991), Design and Analysisof Experiments
(3rd ed.) New York, JohnWiley.
Natrella,M. G. (1979), "Design and Analysisof Experiments,"
in Quality ControlHandbook, ed. J. M. Juran,New York:
McGraw-Hill,pp. 27-35.