BARBARA C. PERDUE and JOHN O.

SUMMERS*

The authors discuss several issues in the timing, construction, and analysis of
manipulation and confounding checks in marketing experiments. A review of 34
experiments involving latent independent variables reported in the Journal of Mar-
keting Research over the past decade suggests that most researchers are familiar
with the concept of manipulation checks but few systematically evaluate potential
sources of confounding in experimental manipulations. Three alternative approaches
for assessing the construct validity of experimental manipulations also are discussed.

•
Checking the Success of Manipulations In
Marketing Experiments

The identification of cause and effect relationships is
the raison d' etre of experimentation. In experimental re-
search the investigator attempts to discover the causal
relationship between two variables by (1) the manipu-
lation (systematic variation) of the independent variable
(also referred to as a factor or a treatment variable) and
(2) the subsequent measurement of the dependent vari-
able. If differences are observed in the dependent vari-
able, the investigator would like to conclude that the in-
dependent variable of interest was responsible. Random
assignment of the test units to treatment conditions cer-
tainly facilitates causal interpretation by eliminating po-
tential systematic differences across treatment conditions
due to extraneous factors associated with characteristics
of the test units (Keppel 1982). What if, however, the
manipulations themselves are confounded (i.e., manip-
ulations that are meant to represent a particular indepen-
dent variable can be interpreted plausibly in terms of more
than one construct, each at the same level of reduction)?
In such a situation confidence in the investigator's causal
explanation (expressed in theoretical terms) of the ex-
perimental results is greatly reduced because the con-
struct validity of the manipulations as operationaliza-
tions of the intended independent variables would be
questionable.
*Barbara C. Perdue is Assistant Professor of Marketing, University
of Georgia. John O. Summers is Professor of Marketing, Indiana Uni-
versity.
The authors acknowledge the many helpful suggestions of three
anonymous JMR reviewers.
Cook and Campbell (1979, p. 60) have noted that fit-
ting the constructs of interest begins with a "careful
preexperimental explication of constructs so that the def-
initions are clear and in conformity with public under-
standing of the words being used." Failure to adhere to
this principle can lead to the manipulation of the wrong
"thing" and/or a redefining of the independent variable
to match whatever the manipulation appears to repre-
sent. For example, source credibility has been defined
as a function of both expertise and trustworthiness (Hov-
land, Irving, and Kelley 1953). Without a preexperi-
mental recognition of this definition, the researcher may
conclude that source credibility has been manipulated
successfully when the treatment groups differ signifi-
cantly on perceptions of the source's expertise and at-
tractiveness but not on perceptions of the source's trust-
worthiness.
When ali of the independent variables to be manipu-
lated in a particular experiment are concrete, observable
variables (e.g., price and/or advertising expenditures in
dollars, color, number of sales calls), it is relatively sim-
ple to confirm that the independent variables were ma-
nipulated as intended. In addition, inadvertent confound-
ing of the manipulations often can be avoided by
maintaining ceteris paribus conditions across treatments.
However, experimental studies in marketing frequently
involve "higher order," unobservable independent vari-
ables (e.g., perceptions of a salesperson's expertise, fear
arousal, group cohesiveness). Because latent variables
such as these cannot be altered directly, the researcher
attempts to manipulate them indirectly by changing se-
lected aspects of the subject's environment. Unfortu-
nately, it is rarely safe to assume that the operations used

317

Journal of Marketing Research

Vol. XXIII (November 1986), 317-26
318
to manipulate psychological and sociological variables
will represent the precise concepts the researcher has in
mind (cf. Frornkin and Streufert 1976). It is therefore
usually prudent for the researcher to perform manipu-
lation checks for these types of independent variables.
Festinger (1953) was among the first to emphasize the
systematic use of manipulation checks; however, there
is evidence the notion was used in experimental research
at least as early as the 1930s (cf. Farnsworth and Misumi
1931; Saadi and Farnsworth 1934). It seems likely that
the concept developed from a number of independent
sources.
Ideally, the experimenter would like to be able to
demonstrate that (1) the treatment manipulations are re-
lated to "direct" measures of the latent variables they
were designed to alter and (2) the manipulations did not
produce changes in measures of related but different
constructs. The first condition relates to Cook and
Campbell's (1979) idea of assessing the convergence of
measures and manipulations of the same "thing" and is
the most widely reported form of manipulation check.
For example, in an experiment on the effects of a sales-
person's expertise on purchasing agents' buying inten-
tions, the researcher might be expected to determine
whether subjects in the "high expertise" condition rate
the salesperson as more knowledgeable about various as-
pects of the presumed area of expertise than do those in
the "low expertise" condition. This procedure is often
referred to as assessing the "take" of the independent
variable and serves to establish a sort of convergent va-
lidity (or lack thereof) between the manipulation and an
effort to measure directly the independent variable of in-
terest. The latter condition incorporates Cook and Camp-
bell's suggestion of testing for a divergence of measures
and manipulations of related but distinct "things." For
example, in the preceding selling experiment, it might
be desirable for the experimenter to investigate whether
the manipulation designed to alter expertise inadvertent-
ly varied perceptions of the salesperson's trustworthiness
or friendliness rather than, or in addition to, expertise.
This special type of manipulation check, which serves
to assess discriminant validity, is referred to as a con-
founding check (Wetzel 1977). Of course, this proce-
dure allows increased confidence in the labeling of the
manipulation as an operationalization of a particular con-
struct only when these checks measure the most plau-
sible rival interpretations of what other constructs the
manipulation might be varying. In summary, both types
of checks can provide important evidence about the con-
struct validity of the putative cause (i.e., the independent
variable) in an experiment.
Manipulation and confounding checks also can be used
to investigate the plausibility of demand characteristics.
In a study of the effects of source credibility, Dholakia
and Sternthal (1977) used manipulation checks for trust-
worthiness and expertise (the dimensions of credibility)
along with confounding checks for attractiveness, dy-
namic/not dynamic, and aggressiveness to investigate
JOURNAL OF MARKETING RESEARCH, NOVEMBER 1986
the validity of their manipulation. Only the treatment ef-
fects for their manipulation checks were statistically sig-
nificant. This finding suggested that source credibility
was manipulated successfully and that the manipulation
was not contaminated by any of the three confounding
variables considered. Later, in a reply to Stanley (1978),
Sternthal and Dholakia (1978) argued that Stanley's de-
mand characteristic explanation for their experimental
results would also predict similar treatment effects for
both the manipulation and confounding checks. Because
this did not occur, they rejected the demand character-
istics explanation.
Finally, reliable manipulation checks are required for
conducting internal analyses (e.g., within-cell analyses
of the relationships among the independent, confound-
ing, and dependent variable measures). The case for a
particular interpretation of the experimental results often
can be strengthened by this form of analysis. Biehal and
Chakravarti (1986) provide an excellent example of this
approach in their study of consumers' use of memory
and external information in choice behavior. Of interest
was whether subjects used prior brand evaluations to guide
choice instead of attempting to retrieve specific brand-
attribute information as suggested by the authors' ac-
cessibility explanation. To resolve this issue they first
determined which subjects in their high and low memory
accessibility conditions attempted to retrieve target brand
feature information and which were successful. Then they
related the successful retrieval of this information to choice
of the target brand. The case for the accessibility expla-
nation was supported by the findings that (1) most sub-
jects within each condition attempted to retrieve the tar-
get brand-attribute information and (2) successful retrieval
was associated strongly with choice of the target brand
within each condition.
A review of 34 experiments reported in the Journal of
Marketing Research from 1975 to 1984 and involving
latent independent variables was conducted to gain a per-
spective on the use of manipulation and confounding
checks in marketing experiments;' That many of these
articles contain a separate section on manipulation checks
and the majority report an attempt to demonstrate a con-
vergence of the manipulation with an independent effort
to measure directly the independent variable suggests most
researchers in marketing are aware of manipulation
checks. However, that only two of these articles report
the use of confounding checks, together with the ten-
dency for authors to provide insufficient information for
the reader to determine exactly when and how their ma-
JThe theory on which a study was based, the hypotheses to be tested,
and statements about what was being intentionally varied by the ma-
nipulations were used to determine whether a particular study was
manipulating latent variables. All studies that could be identified as
involving at least one latent independent variable were included in the
analysis. A tabular summary of the results is available from the au-
thors.
CHECKING THE SUCCESS OF MANIPULATIONS
nipulation and confounding checks were conducted, raises
doubts about the adequacy with which these checks are
being conducted in marketing experiments.
In the following sections we discuss several issues
pertaining to the appropriate timing, construction, anal-
ysis, and reporting of manipulation and confounding
checks. Though our focus is mainly on checks related to
manipulations of latent independent variables, most of
the issues discussed are also relevant to checks on hy-
pothesized intervening processes that have a critical role
in the theory being tested. As Aronson and Carlsmith
(1968, p. 17) suggest, "To get a better idea of whether
some conceptual variable is producing the observed re-
suits, one can sometimes measure the intervening pro-
cesses involved." In this context, they use the term
"measure" to refer to an attempt to obtain "some direct
indication" of whether the hypothesized process is ac-
tually occurring. For example, mental states that are be-
lieved to be a function of the process of interest often
can be assessed.
TIMING OF MANIPULATION AND CONFOUNDING
CHECKS
Pretest Versus the Main Experiment
Manipulation and confounding checks appear to have
their greatest value during the pretest and/or pilot-test-
ing phases of an experiment when an inadequately de-
signed manipulation can still be modified and the main
experiment saved (Aronson and Carlsmith 1968; Wetzel
1977).2 Furthermore, the cost to the researcher associ-
ated with a negative result at this stage (i.e., the time
and effort involved in refining the manipulation and run-
ning an additional pretest or pilot test) is relatively small.
In contrast, the investigator has much to lose should ma-
nipulation and/or confounding checks in the main ex-
periment reflect unfavorably on the manipulation. Fur-
thermore, as we discuss subsequently, including these
checks in the main experiment can present problems in-
dependent of whether they come before or after the de-
pendent variable measures. Fortunately, extensive test-
ing of the manipulations in the pretest and/or pilot-testing
phases will lessen the need for manipulation and con-
founding checks in the main experiment. Of course, when
one is extrapolating manipulation and confounding check
results from a pilot test, it is important that this testing
be conducted with the same procedures, experimental in-
struments, subject types, etc., as the main experiment.
In summary, it seems advisable that the major experi-
ment be run only after the checks included in a pilot test
have suggested the manipulation is successful.
'The term "pretest" is used to refer to those activities designed to
assess the appropriateness of selected parts of the experimental pro-
cedures and/or instruments. The term "pilot test" applies to those
procedures involved in exposing subjects to the total experimental ex-
perience under conditions like those of the main experiment, with the
possible exception of the measurement of the dependent variables.
319
Pretest and pilot test subjects should be interviewed
immediately after exposure to the manipulation (Aron-
son and Carlsmith 1968). Waiting until after the depen-
dent variables have been assessed may reduce the sub-
jects' abilities to describe fully their reactions to the
manipulation and could bias their reports. For example,
in certain cases the desired mental or emotional state (e.g.,
anxiety, source credibility) to be produced by a partic-
ular manipulation may be very temporary and/or might
be altered by the process associated with measuring the
dependent variables.
Usually, the experimenter's initial design of the ma-
nipulation will require several revisions before the main
experiment is run. In addition to signaling that some-
thing is wrong, an effective pretest should suggest cor-
rective changes to be made in the manipulation. This is
particularly important early in the pretesting stage when
substantial revisions are most likely to occur. Aronson
and Carlsmith's (1968) suggestion that long, probing in-
terviews be conducted with subjects after their exposure
to the manipulation seems appropriate at this point in the
study. In addition, experimenters might want to consider
alternative qualitative research techniques such as col-
lecting and analyzing concurrent verbal protocols, par-
ticularly when the theory being tested postulates the oc-
currence of an intervening process. These methods can
give the investigator a better perspective of how, if at
all, the manipulation is operating to create the desired
variance in the intended independent variable and/or
permit the investigation of any theoretically important
intervening processes.
Perhaps more importantly, the preceding methods can
contribute to the identification of potential confounding
variables before the main experiment is conducted (if an
additional highly plausible confounding variable is iden-
tified after the main experiment has been run, the re-
searcher might also find it useful to perform a confound-
ing check on a separate pool of subjects). However, that
so few of the experimental studies reviewed included the
systematic use of confounding checks suggests many ex-
perimenters may not be sufficiently sensitive to these types
of problems. Notable exceptions to this observation are
studies conducted by Holbrook (1978) and Miniard and
Cohen (1979). In an experiment designed to study the
effects of the "factualness/evaluativeness" of a persua-
sive message on beliefs and affect, Holbrook (1978) as-
sessed the "perceived message favorability" (a plausible
confounding variable) as well as the factualness/evalu-
ativeness of his "advertising-like messages." The ma-
nipulation checks for factualness/evaluativeness but not
the confounding checks for message favorability were
statistically significant.
Miniard and Cohen (1979) used an experimental method
to examine aspects of the construct validity of the Fish-
bein behavioral intentions model. In addition to includ-
ing multiple checks within their main experiment to as-
sess the validity of their manipulations of "manipulative
intent" and "agree-disagree" (i.e., the consistency be-
320
tween the hypothetical person's attitude and her belief
about what the referent thinks the person should do), the
authors used separate subject pools to investigate pos-
sible confounding variables. One such group was used
to demonstrate that the agree-disagree manipulation did
not inadvertently affect the perceived attractiveness of
the referent.
In spite of the apparent advantages of establishing the
credibility of the manipulation during the pretest and/or
pilot-testing phases, authors reported doing so in only a
third of the articles reviewed. However, it seems likely
that such tests may be underreported because of space
limitations imposed on authors.
Checks Within the Main Experiment
Potential problems. In reference to the inclusion of
manipulation and confounding checks in the main ex-
periment, Wetzel (1977, p. 89) states, "One of the car-
dinal rules of experimentation is to measure the major
dependent variables first." This position is based on the
potential for these checks, particularly those involving
self-reports or other forms of obtrusive measurement, to
introduce demand characteristics when they precede the
dependent variable measures. However, measuring the
dependent variables before conducting an assessment of
the success of the manipulation presents its own set of
potentially serious problems. First, as noted before, the
manipulation may cause only a temporary change in the
level of the independent variable and/or any confound-
ing variables. Hence, when the manipulation and con-
founding checks follow the dependent measures, impor-
tant effects of the manipulation may already have
dissipated substantially. Also troublesome is the possi-
bility that the subjects' own responses to the dependent
measures may bias their reactions to the subsequent ma-
nipulation and confounding checks, particularly when
these checks involve self-report measures (Kidd 1976).
However, the possibility that the process of measuring
the dependent variables affected the later assessment of
the success of the manipulations seems less plausible
whenever both manipulation and confounding checks are
used and only the manipulation checks demonstrate sig-
nificant treatment effects. The extent to which the ex-
perimenter can exclude such an explanation under the
conditions described will depend on the degree to which
the prior assessment of the dependent variables is ex-
pected to affect the confounding checks at least as much
as the manipulation checks. Unfortunately, as noted be-
fore, use of confounding checks has been very limited
in marketing experiments.
Alternative solutions. At least three approaches have
been proposed for coping with the potential problems in
the ordering of manipulation and confounding checks and
dependent variable measures. First, Kidd (1976) sug-
gests the creation of manipulation check groups (one for
each treatment condition) whose sole purpose is the as-
sessment of the success of the manipulation. The de-
pendent variables are not assessed in the manipulation
JOURNAL OF MARKETING RESEARCH, NOVEMBER 1986
check groups, and the manipulation and confounding
checks are omitted from the experimental groups. This
approach avoids the potential for one set of measure-
ments (i.e., those associated with either the checks or
the dependent variables) to bias the other at the cost of
increasing the number of test units required.
Though Kidd (1976) criticizes a counterbalancing ap-
proach, in which half of the subjects receive the checks
before the dependent measures and the other half after-
ward, as not eliminating the problem but only permitting
its detection, it is not clear that his position is justified.
The manipulation check group approach is identical to
counterbalancing with the last set of measurements (i.e.,
the checks or the dependent measures, depending on which
come last) deleted. Because a measurement cannot bias
responses to other measures that precede it, one can
achieve the same benefits with the counterbalancing ap-
proach as with manipulation check groups by merely ig-
noring the final set of measurements when counterbal-
ancing is used. Furthermore, when the results obtained
from counterbalancing suggest there are no order-bias
problems, one can combine the two sets of groups for
the purpose of testing the theoretical hypotheses of in-
terest, thereby doubling the effective sample size and in-
creasing the power of the related statistical tests. In this
regard, the researcher should compare the means and
variances of all measures (those relating to both the checks
and the dependent variables), as well as the intercorre-
lations among these measures, across the two sets of
treatment groups.
Wetzel (1977) proposed that the subjects in a final pilot
test serve the function of Kidd's (1976) manipulation
check groups. Implicit in this suggestion is that the sub-
jects, manipulations, and environmental setting of this
pilot test be as similar as possible to those of the main
experiment. This alternative is consistent with the po-
sition that the main experiment should be conducted only
after checks in a pilot test have suggested that the ma-
nipulation is successful and no additional changes in the
experimental procedures appear to be required. The total
number of subjects required may be smaller than in the
preceding two approaches, but only if the final pilot test
would have been conducted for reasons other than the
need to conduct manipulation and confounding checks.
One potentially important disadvantage of running a fi-
nal pilot test that includes the manipulation checks is the
difficulty of ensuring the equivalence of the pilot test and
the main experiment on subject and setting factors. The
final pilot test typically is run before the main experi-
ment, and subjects usually are not assigned randomly to
the two groups.
The studies reviewed failed to reveal a general sen-
sitivity to the timing of manipulation and confounding
checks involving self-reports. In the majority of cases it
was not possible to determine whether the checks or the
dependentmeasures were taken first. Furthermore, in three
experiments the manipulation checks were reported as
preceding part or all of the measures for the dependent
CHECKING THE SUCCESS OF MANIPULATIONS
variables. None of the authors reported using either the
manipulation check group or counterbalancing tech-
niques suggested by Kidd (1976), and it was not evident
that any of the experimenters consciously attempted to
employ Wetzel's (1977) "pilot-test adaptation" of Kidd's
manipulation check group approach.
SCALES FOR THE MANIPULATION AND
CONFOUNDING CHECKS
The measures for the manipulation and confounding
checks should be constructed with the same care as those
for the dependent variables. It is important to establish
that the manipulation produced a large enough variance
in the intended independent variable to provide for a
meaningful test of the hypotheses of interest. Further-
more, Aronson and Carlsmith (1968, p. 46) contend that
". . . it is extremely important for all subjects to be
brought to the identical point by the manipulation. . .. "
An assessment of the degree to which these conditions
are met in a particular experiment obviously requires a
valid measure of the independent variable of interest.
Furthermore, valid measures of plausible confounding
variables are needed to permit a clear evaluation of
whether and to what extent the manipulation is con-
founded. The finding of nonsignificant results for the
confounding checks contributes little, if anything, to the
credibility of a manipulation when the measures used are
of questionable validity.
The construction of manipulation and confounding
checks as well as measures for the dependent variables
in an experiment involves basically the same set of gen-
eral issues as the measurement of any concept (see
Churchill 1979 for an extensive discussion of the various
procedures for developing better measures of marketing
constructs). However, when the data used to assess the
quality of these measures are collected solely from ex-
perimental subjects (i.e., those individuals exposed to
the experimental treatments) rather than from a random
sample of individuals from some well-defined popula-
tion (as in a survey), some unique problems arise, par-
ticularly in the interpretability of reliability estimates and
the meaningfulness of estimated factor structures. These
problems derive from the fact that the "across-treat-
ments" covariance matrix for the items included in these
measures will necessarily be a function of the strength
of the manipulations designed by the experimenter.
With respect to reliability, strong (weak) manipula-
tions will tend to produce high (low) reliability esti-
mates. As Peter (1979, p. 7) observes, the reliability
coefficient "is nothing more than the ratio of true vari-
ance to observed variance." A strong (weak) manipu-
lation will produce a large (small) true variance for the
associated manipulation check while, in most cases, not
materially affecting the error variance, thus producing a
high (low) reliability coefficient. Hence, "reliability es-
timates" based on the preceding covariance matrix should
not be interpreted as reflecting the inherent properties of
the related measure.
321
Whenever a large portion of the variance in a partic-
ular manipulation check is "explained" by the related
manipulation, the interpretability issue decreases in im-
portance. In this case it is apparent that the manipulation
check was sufficiently reliable to detect that a meaning-
ful variance in the intended independent variable was
achieved. However, the use of "reliability estimates,"
based on data from experimental subjects, outside the
context of the particular study in which these estimates
were derived seems highly questionable.
It is when the experimenter fails to find a satisfactory
convergence between a manipulation and its associated
manipulation check that the reliability issue becomes most
prominent. In particular, one may have difficulty deter-
mining whether this lack of convergence is due to a weak
manipulation, an "unreliable manipulation" (i.e., one for
which the within-cell variance of the intended indepen-
dent variable is high), an unreliable manipulation check
measure, or some combination ofthese factors (note that
the within-cell variance of a manipulation check measure
will be a function of both the reliability of the manip-
ulation and the reliability of the manipulation check).
Hence, it will be difficult to assess the relative impact
of these three factors without an estimate of the reli-
ability of the manipulation check that is independent of
the strength and reliability of the manipulation itself.
A factor analysis of the across-treatments covariance
matrix seems likely to produce misleading results be-
cause the covariances will be a function of the strength
of the experimental manipulations. One can choose in-
stead to factor the pooled within-treatments covariance
matrix. This approach avoids allowing the manipulations
to affect the factor structure and therefore appears to be
a more defensible procedure. Calder and Stemthal (1980),
in a study of television commercial wearout, success-
fully analyzed the factor structure of their dependent
measures using this within-treatments approach. It should
be noted, however, that the success of this approach de-
pends on the presence of a substantial amount of within-
cell variance on the underlying constructs of interest.
When the within-cell covariance matrix is dominated by
random measurement error the results will tend to be un-
stable and difficult to interpret.
As noted before, the use of self-reports for both the
checks and the dependent variables can result in biased
responses to whichever measures are taken last. Fur-
thermore, Aronson and Carlsmith (1968, p. 50) suggest
that ". . . too often subjects are unable or unwilling to
explain just what the effects of some manipulation have
been," and suggest that" . . . the best solution is to ob-
serve some other behavior which we expect to covary
directly with our theoretical variable and see whether it
does. . . . " Most frequently, this can be done best in a
pretest or pilot test, particularly if these measures require
some special intervention by the experimenter. When-
ever such behavioral checks are included in the main ex-
periment, it seems important that the behaviors in ques-
tion occur naturally as a function of the manipulation
322
itself and that the observation be unobtrusive. Subjects
can use their memories of their past behaviors as a basis
for their responses to the dependent measures and are
most likely to do so when those behaviors are made sa-
lient (cf. Sherman et al. 1978). Though a few experi-
menters have been able to develop such behavioral mea-
sures for their particular studies (cf. Reingen and Kernan
1977), most apparently find this a difficult if not im-
possible task, and self-reports are the dominant method
for assessing the "take" of the independent variable in
marketing experiments.
ANALYSIS OF MANIPULATION AND
CONFOUNDING CHECKS
Conducting a statistical test to determine whether the
manipulation had either no effect or the wrong effect on
an independent variable can be relatively straightforward
for single-factor designs. However, when multiple fac-
tors are involved, directional r-tests and/or one-way
ANOVA followed by multiple contrasts may not be suf-
ficient for adequate?, analyzing the manipulation and
confounding checks. As Wetzel (1977, p. 88) observes,
However, in some situations we may be able to falsify a
manipulation in the sense of demonstrating that the ma-
nipulation was properly carried out. For example, in a 2
x 2 (AB) design, manipulation A must be independent
of manipulation B. If the researcher discovers that ma-
nipulation A significantly influences the manipulation
checks for independent variable B, then the null hypoth-
esis that the two manipulations are independent can be
rejected, and manipulation A can be considered to be fal-
sified.
The practical significance of Wetzel's remarks is that they
indicate an adequate analysis of a manipulation check for
a given factor (manipulation) within a multiple-factor
design requires the use of the full-factorial ANOVA model
whenever it is plausible that one manipulation may have
'Though Bagozzi's (1977) structural equation approach to analyzing
experimental data enables the experimenter both to assess error in the
measures of the independent and dependent variables and to estimate
the relationships among these variables within the context of a single
comprehensive model, it has yet to be widely adopted by marketing
researchers. Only two of the studies reviewed here (Bearden and Shimp
1982; Churchill and Surprenant 1982) utilized structural equation
models. Two factors may partially account for this situation. First,
Bagozzi's approach requires that multiple manipulation checks and
multiple measures of the dependent variables be taken on the same
subjects during the main experiment, which can create methodological
problems. Furthermore, researchers often find it difficult to develop
a single reliable and valid measure for each of their variables, much
less multiple measures. Second, many, if not most, experimental stud-
ies in marketing involve factorial designs, and interactions between
factors are difficult to handle with USREL (Joreskog and Sorbom
1981), a particularly popular program for analyzing structural models.
Neither of the studies reviewed that used structural equation models
included interaction effects in those models. ANDYA appears likely
to remain the dominant approach for the analysis of intervally scaled
dependent variables in experimental studies for the immediate future
and much of our discussion implicitly assumes its use.
JOURNAL OF MARKETING RESEARCH, NOVEMBER 1986
inadvertently affected an independent variable associated
with a different manipulation. Furthermore, researchers
must be concerned with the statistical significance of all
main and interaction effects, not just those involving the
factor corresponding to the manipulation check measure
being analyzed. A statistically significant main effect for
the manipulation (factor) corresponding to the manipu-
lation check being analyzed provides evidence in favor
of the convergent validity of that particular manipula-
tion. To the extent that other main and/or interaction
effects are statistically significant, the discriminant va-
lidity of the associated manipulations becomes suspect.
Ideally, only one effect, the main effect of the factor
(manipulation) of interest, will be statistically signifi-
cant. If effects associated with other manipulations prove
to be statistically significant, these manipulations will
have been "falsified" in the sense that they have not had
their intended effects. That is, these manipulations have
had an effect on an independent variable other than the
one they were individually intended to manipulate, and
the independent variables, as manipulated, will not be
orthogonal. Under these conditions, misleading results
can occur when the researcher analyzes the dependent
measures with the ANOVA model corresponding to the
planned orthogonal research design.
Whenever confounding checks are analyzed, the only
condition strictly favorable to the construct validity of
the manipulations is that of statistically nonsignificant
results for all main and interaction effects. Any signif-
icant main or interaction effects for the confounding
checks reflect negatively on the discriminant validity of
the associated manipulation(s). However, even when
confounding is present, it may still be possible in some
cases to detect whether the intended independent vari-
able has had its hypothesized directional effects-for
example, when the pattern of the confounding cannot
plausibly explain the results for the dependent measures.
Consider an experiment in which videotaped sales pre-
sentations are used to manipulate the perceived expertise
(factor A) and friendliness (factor B) of a salesperson in
a (2 x 2) AB factorial design. A confounding check for
trustworthiness is included along with manipulation checks
for both expertise and friendliness. If when analyzing the
manipulation check for expertise (A) one finds statisti-
cally significant results for the main effect of Band/or
the AB interaction, one must conclude that the friendli-
ness manipulation had an unintended effect on perceived
expertise. In this case, the treatment cells labeled "high
expertise" would not contain the same level of expertise.
In the analysis of the confounding check for trustwor-
thiness, a statistically significant main effect for A (B)
would suggest that the manipulation for expertise
(friendliness) is confounded, and a significant AB inter-
action would suggest a more complex confounding of
both manipulations.
It is important to demonstrate not only that the in-
tended effects of the manipulations did occur, but also
that these effects are of sufficient magnitude to provide
CHECKING THE SUCCESS OF MANIPULATIONS
for a meaningful test of the hypotheses of interest. What
constitutes an acceptably large intended effect for a given
manipulation check will depend on such factors as the
desired or predicted effect size for the dependent vari-
able and the expected strength of the relationship be-
tween the independent and dependent variables. Fur-
thermore, if the significance tests suggest that the
manipulations are confounded, the researcher should
evaluate whether the degree of confounding present is
serious enough to impair an unambiguous evaluation of
the results of the main experiment.
In the context of the aforementioned ANOVA model,
an appropriate indicator of effect size might be CJ)2 (Saw-
yer and Ball 1981; Sawyer and Peter 1983).4 Omega
squared represents the proportion of variance in the "de-
pendent" variable (in this case, the particular manipu-
lation or confounding check measure being analyzed) ac-
counted for by a given main or interaction effect.5 Ideally,
for any given manipulation check measure, the experi-
menter would like to find a sufficiently large CJ)2 asso-
ciated with the main effect of the factor (manipulation)
corresponding to the manipulation check measure being
analyzed and a near-zero CJ)2 for each of the other main
and interaction effects." For any given confounding check
measure, the desired result is that the CJ)2,S for all main
and interaction effects be close to zero.
Miniard and Cohen (1979) provide an excellent ex-
ample of the proper analysis of manipulationchecks. They
used the appropriate ANOVA model in analyzing each
of several manipulation checks and compared the size of
the intended and unintended effects within each of their
analyses whenever the latter were statistically significant
(also see Baker and Churchill 1977; Burnkrant and How-
ard 1984).
If, in analyzing the results of the manipulation check
for A, the experimenter obtains a statistically significant
main effect for A but an insufficiently high -associated
CJ)2, the revision of A should be contemplated if the main
experiment has not yet been conducted. However, it is
also possible that the manipulation check measure itself
is confounded and/or unreliable. If the experiment al-
ready has been completed and A's manipulation check
demonstrated to be reliable, some form of internal anal-
ysis might be conducted if there is sufficient within-cell
variance; however, causal statements are no longer war-
ranted (Aronson and Carlsmith 1968).
4It would also be desirable to examine the distribution of manipu-
lation check scores within treatments to ensure that what appears to
be a "successful" manipulation is not primarily due to a few subjects
showing a very large effect of the treatment.
'Omega squared will be a function of the reliability of the measure
employed, the heterogeneity of the subjects, and the heterogeneity of
the treatment implementation as well as the differences between cell
means (LaTour 1981).
'When this is the case, w 2 for the intended effect will be roughly
equivalent to the partial w 2 for this same effect (Keren and Lewis
1979).
323
When in the analysis of the manipulation check for A
the effect sizes for B and AB are much smaller than that
for A, their statistical significance probably should not
be of great concern. However, whenever the total of the
CJ)2,S for B and AB are of the same order of magnitude
as the CJ)2 for A, analyzing the dependent measures in the
main experiment with the ANOVA model corresponding
to the planned orthogonal design is likely to produce dis-
torted results (e.g., unexpected interaction effects that
are difficult to explain in theoretical terms). In this sit-
uation a statistical model that does not assume orthog-
onality (e.g., some form of the general linear model)
might be used to analyze the dependent variable mea-
sures.
When, in comparison with the independent variables
of interest, a confounding variable is considered to have
a substantially greater theoretical impact on the primary
dependent variable, even a small effect size for the cor-
responding check measure can indicate serious problems
with confounding. Conversely, moderate effect sizes for
the confounding checks can be tolerated when the cor-
responding confounding variables are thought to have only
a very slight impact on the dependent variable. How-
ever, checks seem most likely to be included for those
potentially confounding variables that are thought to have
a moderate to strong causal link to the primary depen-
dent variable. In those cases where the experimenter is
uncertain about the relative theoretical impact of the con-
founding and independent variables, an internal analysis
may help resolve the issue when within-cell variance is
moderate to high and the manipulation and confounding
checks have high reliability.
If the confounding checks suggest that the manipula-
tions are seriously confounded, it may be difficult to sep-
arate satisfactorily the main and interaction effects of the
intended independent variables from those of the con-
founding variables. However, the researcher might wish
to test whether the manipulation checks explain a sub-
stantial additional portion of the variance in the depen-
dent measures beyond the part that can be accounted for
by the confounding checks alone. The models compar-
ison approach (cf. Chapter 2 of Green 1978) is one method
that might be used for this purpose. However, analyses
of this type are likely to be very effective only when the
confounding checks are reliable and moderately corre-
lated with the manipulation checks. Even in this case,
the results should be interpreted with caution because the
manipulations designed to alter the intended independent
variables also produce changes in the confounding vari-
ables.
Another simpler, though perhaps not as demanding,
test of the confounding variables involves entering the
confounding checks as covariates in the ANOVA model
used to analyze the dependent measures. Burnkrant and
Howard (1984) used this approach successfully in in-
vestigating whether the effects of their grammatical form
manipulation on "total thoughts" could be explained by
the reduced confidence (the confounding variable) de-
324
tected in their "question" condition. Because the con-
fidence covariate was nonsignificant in this analysis and
the statistical significance of the main effect of the gram-
matical form manipulation was not noticeably affected,
they rejected the confidence explanation.
ARE MANIPULATION AND CONFOUNDING
CHECKS NECESSARY?
The use of manipulation and confounding checks is
only one of several methods for assessing the construct
validity of the experimental manipulations. Given the
potential problems with conducting these checks within
the main experiment, it is worthwhile to consider the
available options. One alternative approach involves in-
cluding multiple dependent variables in the experimental
study. Should the pattern of results be completely con-
sistent with the broad theory underlying the study, one
might claim that evidence for the construct validity of
the manipulation has been provided by demonstrating a
degree of nomological validity. As Campbell (1960, p.
547) suggests, nomological validity relates to "the pos-
sibility of validating tests by using the scores from a test
as interpretations of a certain term in a formal theoretical
network and, through this, to generate predictions which
would be validating if confirmed when interpreted as still
other operations and scores" (also see Peter 1981; Cron-
bach and Meehl 1955). However, the strength of this
claim would depend on the absence of other plausible
rival interpretations of the manipulation, as well as de-
mand artifacts, which might also explain the results.
Furthermore, when the expected pattern of results is not
fully achieved, the experimenter is in the awkward po-
sition of having to decide whether some part of the the-
ory (e.g., the relationship between the intended inde-
pendent variable and a particular dependent variable),
the manipulation itself, or some other aspect of the ex-
perimental procedures (e.g., the dependent variable
measures) is at fault. Without manipulation and con-
founding checks, finding a convincing solution to this
potential dilemma would be difficult.
Much of the analysis performed by Miniard and Cohen
(1979) involved tests of theoretical linkages among the
independent variables they attempted to manipulate and
other constructs associated with the Fishbein behavioral
intentions model. Therefore their study could be inter-
preted as employing the preceding "nomological valid-
ity" approach.
A second possibility involves the use of what Gamer,
Hake, and Eriksen (1956, p. 150) refer to as "converg-
ing operations," that is, "any set of two or more ex-
perimental operations which allow the selection or elim-
ination of alternative hypotheses or concepts which could
explain an experimental result." Their approach is not
unlike the "technique of purification" proposed by Aron-
son and Carlsmith (1968). This alternative represents an
application of Platt's (1964) "strong inference" approach
in that it suggests the systematic devising of alternative
hypotheses and the development and execution of crucial
JOURNAL OF MARKETING RESEARCH, NOVEMBER 1986
experiments. It therefore seems appropriate for investi-
gating hypothesized intervening processes as well as the
construct validity of the manipulation as an operation-
alization of the intended independent variable. The prac-
tical value of this method is heavily dependent on the
number of plausible alternative hypotheses as well as the
experimenter's ability to develop credible converging
operations.
The study by Kisielius and Sternthal (1984) on the ef-
fects of vividness might be interpreted as employing
convergent operations. In a pilot study the authors found
that their verbal message condition produced more fa-
vorable product judgments than their picture condition.
Though these results could be interpreted in availability-
valence terms, other interpretations (e.g., that informa-
tion unique to the picture condition caused the judgment
effect) were plausible. The authors conducted a series of
three experiments to rule out alternative explanations and/
or to otherwise strengthen and clarify their availability-
valence explanation. For example, the first of these ex-
periments was designed to distinguish between two al-
ternative explanations for the initial results, (1) that the
effects were "due to the fact that cognitive elaboration
in response to the pictures reduced persuasion" and (2)
that the effects were "due to intertreatment differences
in stimulus information."
A third alternative, suggested by Kidd (1976) among
others, is to rely on the emergence of convergent find-
ings from a large number of replications of an experi-
ment to strengthen the case for a particular interpretation
of the manipulation. When the design of the manipula-
tion varies in important ways across experiments, dif-
ferent sets of plausible interpretations will be relevant for
the various studies. If a particular alternative interpre-
tation of the manipulation is reasonable in some studies
but not in others, it may be possible to eliminate this
interpretation for the experimental results in much the
same manner as if convergent operations had been pur-
posely employed.
In their review of dissonance theory, Brehm and Cohen
(1962, p. 311) discuss the significance of the convergent
results found across dissonance experiments. In partic-
ular, they claim that, "While each experiment or at least
a number of experiments can be explained in alternative
ways, there appears to be no theory that at present can
more easily explain the phenomena dealt with by all dis-
sonance experiments."
It is when a series of replications contain different ir-
relevancies with respect to both the manipulations and
measurement methods employed that convincing evi-
dence is provided about the construct validity of the pu-
tative cause (Cook and Campbell 1979). Unfortunately,
as the heterogeneity of the irrelevancies contained in a
series of replications increases, more divergent findings
can be expected. Without effective manipulation and
confounding checks, resolving these potential inconsis-
tencies is difficult.
Manipulation checks also may be useful in explaining
CHECKING THE SUCCESS OF MANIPULATIONS
an apparent "lack of convergence" within an experi-
ment. For example, if when analyzing a key dependent
variable one found that a presumed theoretically irrele-
vant background variable (e.g., age or sex) interacted
with the manipulation, one might wish to use manipu-
lation checks to determine whether the strength of the
manipulation varied as a function of the background
variable (e.g., whether the manipulation caused a greater
change with respect to the independent variable of in-
terest for some age groups than for others). To the extent
the manipulation-background variable interaction effect
on the dependent variable can be explained by the man-
ner in which the strength of the manipulation varied with
the background variable, the original theory escapes
damage even though the interaction was not predicted
initially. Hence, omitting manipulation and confounding
checks and relying entirely on experimental replications
appears to be satisfactory only in those seemingly rare
cases when the results of several widely diverse repli-
cations are in agreement and no unexpected manipula-
tion-background variable interactions are encountered.
Though each of the alternative approaches for assess-
ing construct validity might be sufficient in particular
situations, one can rarely be certain of this outcome in
advance. Furthermore, the case for a particular interpre-
tation of a manipulation is always more credible when
supported by empirical evidence developed through a
variety of methodological approaches. Hence, manipu-
lation and confounding checks can have an important role
in establishing the construct validity of the manipulation
even when these alternative approaches are employed.
DISCUSSION AND CONCLUSIONS
The design of an experiment involving one or more
latent independent variables is one of the most difficult
tasks a researcher can undertake. It requires substantial
methodological training and an in-depth knowledge of
the theory to be tested. Creativity is also critical because
often no obvious credible alternatives are available for
manipulating the independent variables and, conse-
quently, experimenters must develop their own manip-
ulations. It is frequently difficult and time-consuming to
develop even one apparently reasonable procedure for
manipulating a particular independent variable, and when
several variables are to be manipulated within a single
vehicle (e.g., expertise and friendliness within a sales
presentation), the complexity of the experimenter's task
can increase geometrically. Many things can go wrong
with an experimental design: (1) the manipulations may
not be strong enough to allow for a meaningful test of
the theory, (2) a manipulation designed to vary one par-
ticular independent variable may also affect other inde-
pendent variables in the study, (3) other latent variables
(i.e., confounding variables) may be inadvertently af-
fected by the manipulations, (4) the experimental pro-
cedures might create demand artifacts (Sawyer 1975),
and (5) the design of the manipulations may suggest that
the generalizability of the findings to real-world mar-
325
keting situations, that is, the study's external validity
(Cook and Campbell 1979; Lynch 1982), is highly ques-
tionable.
The appropriate time to look for problems in an ex-
perimental design is in the pretest and/or pilot stages,
not after the main experiment has been conducted and
has failed to produce the expected results. An aggressive
effort should be made in the early pretest stages to iden-
tify plausible problems with the manipulations. This step
is difficult, not only because there are so many possi-
bilities, but also because it places experimenters in the
role of critiquing the results of their own efforts in de-
signing the initial manipulations. Critical reviews of the
proposed manipulations by other researchers working in
the same or related areas, the use of unstructured inter-
views with pretest subjects, the collection of concurrent
verbal protocols during pretests, and other such quali-
tative methods can be helpful. However, such ap-
proaches do not ensure that all, or even most, of the
potential problems will be identified.
Insufficient attention to assessing rigorously the suc-
cess of experimental manipulations, particularly in the
pretestand pilot-testing phases, is likely to result in lengthy
discussions of alternative post hoc explanations for un-
expected experimental findings and/or a series of ex-
periments that provide little solid evidence about the re-
search hypotheses of interest. The appropriate design,
execution, analysis, and reporting of manipulation and
confounding checks are frequently essential to achieving
a convincing interpretation of the results of experimental
studies involving latent independent variables.
REFERENCES
Aronson, Elliot and J. Merrill Carlsmith (1968), "Experimen-
tation in Social Psychology," in The Handbook of Social
Psychology, 2nd ed., Vol. 2, Gardner Lindzey and Elliot
Aronson, eds. Reading, MA: Addison-Wesley Publishing
Company, 1-79.
Bagozzi, Richard P. (1977), "Structural Equation Models in
Experimental Research," Journal of Marketing Research,
14 (May), 209-26.
Baker, Michael J. and Gilbert A. Churchill, Jr. (1977), "The
Impact of Physically Attractive Models on Advertising Eval-
uations," Journal of Marketing Research, 14 (November),
538-55.
Bearden, William O. and Terence A. Shimp (1982), "The Use
of Extrinsic Cues to Facilitate Product Adoption," Journal
of Marketing Research, 19 (May), 229-39.
Biehal, Gabriel and Dipankar Chakravarti (1986), "Con-
sumers' Use of Memory and External Information in Choice:
Macro and Micro Perspectives," Journal of Consumer Re-
search, 12 (March), 382-405.
Brehm, Jack W. and Arthur R. Cohen (1962), Exploration in
Cognitive Dissonance. New York: John Wiley & Sons, Inc.
Burnkrant, Robert E. and Daniel J. Howard (1984), "Effects
of the Use of Introductory Rhetorical Questions Versus
Statements on Information Processing," Journal of Person-
ality and Social Psychology, 47 (December), 1218-30.
Calder, Bobby J. and Brian Sternthal (1980), "Television
326
Commercial Wearout: An Information Processing View,"
Journal of Marketing Research, 17 (May), 173-86.
Campbell, Donald T. (1960), "Recommendations for APA Test
Standards Regarding Construct, Trait, or Discriminant Va-
lidity," American Psychologist, 15 (August), 546-53.
Churchill, Gilbert A., Jr. (1979), "A Paradigm for Developing
Better Measures of Marketing Constructs, " Journal of Mar-
keting Research, 16 (February), 64-73.
- - - and Carol Surprenant (1982), "An Investigation Into
the Determinants of Customer Satisfaction, " Journal of
Marketing Research, 19 (November), 491-504.
Cook, Thomas D. and Donald T. Campbell (1979), Quasi-
Experimentation: Design and Analysis Issues for Field Set-
tings. Boston: Houghton-Mifflin Company.
Cronbach, Lee J. and Paul E. Meehl (1955), "Construct Va-
lidity in Psychological Tests," Psychological Bulletin, 52
(July), 281-302.
Dholakia, Ruby and Brian Sternthal (1977), "Highly Credible
Sources: Persuasive Facilitators or Persuasive Liabilities?"
Journal of Consumer Research, 3 (March), 223-32.
Farnsworth, Paul R. and Issei Misumi (1931), "Further Data
on Suggestion in Pictures," Americal Journal of Psychol-
ogy, 43 (October), 632.
Festinger, Leon (1953), "Laboratory Experiments," in Re-
search Methods in the Behavorial Sciences, Leon Festinger
and Daniel Katz, eds. New York: Holt, Rinehart and Win-
ston, 136-72.
Frornkin, Howard L. and Siegfried Streufert (1976), "Labo-
ratory Experimentation," in Handbook ofIndustrial and Or-
ganizational Psychology, Marvin D. Dunnette, ed. Chicago:
Rand McNally, 415-65.
Garner, Wendell R., Harold W. Hake, and Charles W. Erik-
sen (1956), "Operationism and the Concept of Perception,"
Psychological Review, 63 (May), 149-59.
Green, Paul E. (1978), Analyzing Multivariate Data. Hins-
dale, IL: The Dryden Press.
Holbrook, Morris B. (1978), "Beyond Attitude Structure: To-
ward the Informational Determinants of Attitude," Journal
of Marketing Research, 15 (November), 545-56.
Hovland, Carl I., Janis L. Irving, and Harold H. Kelley (1953),
Communication and Persuasion: Psychological Studies of
Opinion Change. New Haven, CT: Yale University Press.
Joreskog, Karl G. and Dag Sorbom (1981), LISREL V. Chi-
cago: National Educational Resources, Inc.
Keppel, Geoffrey (1982), Design and Analysis: A Research-
er's Handbook, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall,
Inc.
Keren, Gideon and Charles Lewis (1979), "Partial Omega
Squared for ANOVA Designs," Educational and Psycho-
logical Measurement, 39 (Spring), 119-28.
Kidd, Robert F. (1976), "Manipulation Checks: Advantage or
JOURNAL OF MARKETING RESEARCH, NOVEMBER 1986
Disadvantage," Representative Research in Social Psychol-
ogy, 7 (2), 160-5.
Kisielius, Jolita and Brian Sternthal (1984), "Detecting and
Explaining Vividness Effects in Attitudinal Judgments,"
Journal of Marketing Research, 21 (February), 54-64.
LaTour, Stephen A. (1981), "Variance Explained: It Measures
Neither Importance nor Effect Size," Decision Sciences, 12
(January), 150-60.
Lynch, John G., Jr. (1982), "On the External Validity of Ex-
periments in Consumer Research," Journal of Consumer
Research, 9 (December), 225-39.
Miniard, Paul W. and Joel B. Cohen (1979), "Isolating At-
titudinal and Normative Influences in Behavioral Intentions
Models," Journal of Marketing Research, 16 (February),
102-10.
Peter, J. Paul (1979), "Reliability: A Review of Psychometric
Basics and Recent Marketing Practices," Journal of Mar-
keting Research, 16 (February), 6-17.
- - - (1981), "Construct Validity: A Review of Basic Issues
and Marketing Practices," Journal of Marketing Research,
18 (May), 133-45.
Platt, John R. (1964), "Strong Inference," Science, 146 (Oc-
tober), 347-53.
Reingen, Peter H. and Jerome B. Kernan (1977), "Compli-
ance with an Interview Request: A Foot-in-the-Door, Self-
Perception Interpretation," Journal of Marketing Research,
14 (August), 365-9.
Saadi, Mitchel and Paul R. Farnsworth (1934), "The Degrees
of Acceptance of Dogmatic Statements and Preferences for
Their Supposed Makers," Journal of Abnormal and Social
Psychology, 29 (2), 143-50.
Sawyer, Alan G. (1975), "Demand Artifacts in Laboratory
Experiments in Consumer Research," Journal of Consumer
Research, 1 (March), 20-30.
- - - and A. Dwayne Ball (1981), "Statistical Power and
Effect Size in Marketing Research," Journal of Marketing
Research, 18 (August), 275-90.
- - - and J. Paul Peter (1983), "The Significance of Statis-
tical Significance Tests in Marketing Research," Journal of
Marketing Research, 20 (May), 122-33.
Sherman, Steven J., Karin Ahlm, Leonard Berman, and Ste-
ven Lynn (1978), "Contrast Effects and Their Relationship
to Subsequent Behavior," Journal of Experimental Social
Psychology, 14 (July), 340-50.
Stanley, Thomas J. (1978), "Are Highly Credible Sources Per-
suasive?" Journal of Consumer Research, 5 (June), 66-7.
Sternthal, Brian and Ruby Roy Dholakia (1978), "Rejoinder,"
Journal of Consumer Research, 5 (June), 67-9.
Wetzel, Christopher G. (1977), "Manipulation Checks: A Re-
ply to Kidd," Representative Research in Social Psychol-
ogy, 8 (2), 88-93.