Experimental Designs

CHAPTER 9

1
Chapter Objectives
Distinguish between causal and
correlational analysis
Explain the difference between lab
and field experiments.
Explain the following terms:
nuisance variables, manipulation,
experimental and control groups,
treatment effect, matching and
randomization.
2
Chapter Objectives
Discuss internal and external
validity in experimental designs.
Discuss the seven possible threats
to internal validity in experimental
designs.
Describe the different types of
experimental designs.
Apply what has been learned to
class assignments and exams.
3
Experimental Designs
Experimental designs fall into two
categories:
Experiments done in an artificial or
contrived environment, known as lab
experiments, and
Experiments done in the natural
environment in which activities
regularly take place, known as the field
experiment.
4
Experimental Designs
Experimental designs are set up
to examine possible cause and
effect relationships among
variables.
Correlational studies examine
the relationships among variables
without necessarily trying to
establish if one variable causes
another. 5
Experimental Designs
To establish that variable X causes
variable Y, all three of the following
conditions should be met:
1. Both X and Y should covary (when
one goes up, the other should also go
up or down).
2. X (the presumed causal factor)
should precede Y. In other words,
there must be a time sequence in
which the two occur.
3. No other factor should possibly cause
the change in the dependent variable
Y. 6
Experimental Designs
To establish causal relationships
between two variables, several
variables that might covary with
the dependent variable have to be
controlled.
This control would allow us to say
that variable X alone causes the
dependent variable Y.
7
Experimental Designs
establishing cause-and-effect
relationships is not easy because:
Several other variables that covary with
the dependent variable have to be
controlled.
It is not always possible to control all
the variables while manipulating the
causal factor (the independent variable
that is causing the dependent variable)
in organizational settings, where events
flow naturally and normally.
8
Experimental Designs
It is possible to first isolate the
effects of a variable in a tightly
controlled artificial setting (the lab
setting), and
After testing and establishing the
cause-and-effect relationship
under these tightly controlled
conditions, see how
generalizable such 9
Example 1
Suppose a manager believes that
staffing the accounting department
completely with personnel with M.Acc
(Master of Accountancy) degrees will
increase its productivity.
Thus, he wants to examine the
hypothesis that possession of a
M.Acc degree would cause
increases in productivity.

10
Example 1 (Cont.)
The hypothesis can be tested in an
artificially created setting (not at
the regular workplace) because if it was
tested at the regular workplace, the
manager should transfer all those
without the M.Acc degree currently
in the department to other departments
and recruit fresh M.Acc degree holders
to take their place. Such action will
disrupt the work of the entire
organization.
11
Example 1 (Cont.)
The work of the organization will
be disrupted because of the
following factors:
The new people will have to be
trained.
Employees will get upset.
Work will slow down.

12
Example 1 (Cont.)
The hypothesis that possession
of a M.Acc degree would cause
increases in productivity can be
tested not at the regular
workplace but in an artificial
created setting in which an
accounting job can be given to
three groups of people.
13
Example 1 (Cont.)
An artificial created setting in which
an accounting job can be given to three
groups of people:
The first group contains those with a
M.Acc degree.
The second group contains those
without a M.Acc degree.
The third group contains a mixed
group of those with and without a
M.Acc degree (as in the case in the
present work setting)
14
Example 1 (Cont.)
If the first group performs
exceedingly well, the second
group poorly, and the third
group falls somewhere in the
middle, there will be evidence to
indicate that the M.Acc degree
qualification might indeed cause
productivity to rise.
15
Example 1 (Cont.)
If such evidence is found, then
planned and systematic efforts can be
initiated to gradually transfer those
without the M.Acc degree in the
accounting department to other
departments and recruit others with this
degree to this department.
It is then possible to see to what extent
productivity does, in fact, go up in the
department because all the staff
members are M.Acc degree holders.
16
The Lab Experiment
When a cause-and-effect relationship
between an independent and a dependent
variable of interest is to be clearly established,
then all other variables that might
contaminate or confound the relationship have
to be tightly controlled so that the actual
causal effects of the investigated independent
variable on the dependent variable can be
determined.
It is also necessary to manipulate the
independent variable so that the extent of its
causal effects can be established.
17
The Lab Experiment
The controls and manipulations are
best done in an artificial setting (the
laboratory), where the causal effects
can be tested.
When controls and manipulations
are introduced to establish cause-and-
effect relationships in an artificial
setting, we have laboratory
experimental designs or lab
experiments.
18
Control of the Dependent
Variable
When we assume cause-and-effect
relationships between two variables X and Y,
it is possible that some other factor, say A,
might also influence the dependent variable
Y.
In such a case, it will not be possible to
determine the extent to which Y occurred only
because of X, since we do not know how
much of the total variation of Y was caused
by the presence of the other factor A.
So we have to control the contaminating
factor, A.
19
Example 2
A Human Resource Development manager
might arrange for special training to a set of
newly recruited secretaries in creating web
pages to prove that such training would
cause them to function more effectively.
However, some of the secretaries might
function more effectively than others because
they have had previous experience with
the web.
In this case, the manager cannot prove that
the special training alone caused greater
effectiveness, since the previous
experience of some secretaries with the
web is a contaminating factor.
20
Example 2 (Cont.)
If the true effect of the training on
learning is to be assessed, then
the learners previous
experience has to be
controlled.
This might be done by not
including in the experiment those
who already have had some
experience with the web. 21
Manipulation of the
Independent Variable
In order to examine the causal
effects of an independent variable
on a dependent variable, certain
manipulations need to be tried.
Manipulation means that we
create different levels of the
independent variable to assess the
impact on the dependent variable.
22
Example 3
If we want to test the theory that
depth of knowledge of various
manufacturing technologies is
caused by rotating the employees
on all the jobs on the production
line and in the design department,
over a 4-week period.

23
Example 3 (Cont.)
To test this theory, we can manipulate the
independent variable, rotation the
employees, by:
- rotating one group of production workers and
exposing them to all the systems during the 4-
week period.
- rotating the second group of workers and
exposing them to only half of the
manufacturing technologies during the 4-
weeks.
- leaving the third group to continue to do
what they are currently doing, without any
special rotation. 24
Example 3 (Cont.)
By measuring the depth of
knowledge of these groups both
before and after the
manipulation (also known as
treatment), it would be possible to
assess the extent to which the
treatment caused the effect, after
controlling the contaminating
factors.
25
Example 3 (Cont.)
If deep knowledge is indeed caused by
rotation and exposure, the results
would show that:
- the third group had the lowest
increase in depth of knowledge.
- the second group had some
significant increase, and
- the first group had the greatest
gains.
26
Example 4
We want to test the effects of
lighting on worker production
levels among sewing machine
operators.
To establish cause-and-effect
relationship, we must follow the
following steps:

27
Example 4 (Cont.)
First measure the production levels of all the
operators (60 operators) over a 15-day
period with the usual amount of light they
work with- say 60 watt lamps.
Split the operators into three groups of 20
members each:
1. Allowing one subgroup to continue to work
under the same conditions as before (60-
watt lambs)
2. Manipulate the intensity of the light for the
second subgroup by working with 75-watt
lambs, and
3. Manipulate the intensity of the light for the
third subgroup by working with 100-watt
lambs. 28
Example 4 (Cont.)
After the different groups have worked
with these varying degrees of light
exposure for 15 days, each groups total
production for these 15 days may be
analyzed to see if the difference
between the pre-experimental and the
post-experimental productions among
the groups is directly related to the
intensity of the light to which they
have bees exposed.
29
Example 4 (Cont.)
If the hypothesis that better lighting
increases the production levels is
correct, then the subgroup that did not
have any change in the lighting (called
the control group), should have no
increase in production and the other
two groups should show increases:
- the group with the 100-watt lambs
showing the greatest increase, and
- the group with the 75-watt lambs
showing increase lower than the 100-
watt group.
30
Example 4 (Cont.)
The independent variable
(lighting) has been manipulated
by exposing different groups to
different degrees of changes in
light.
This manipulation of the
independent variable is also
known as the treatment, and
the results of the treatment
31
are
Example 7.1: Using both
Controlling and Manipulation in a
Lab Setting
The owner of a toy shop is
disappointed with the number of
imitation Ninja turtles (which is
greatly in demand) produced by
his workers, who are paid wages at
an hourly rate. He might wonder
whether paying them piece
rates would increase their
production levels.
32
Example 7.1: Using both
Controlling and Manipulation in a
Lab Setting
Before implementing the piece-rate
system, he would want to make sure
that switching over to the new system
would indeed achieve the objective.
The researcher might first want to test
the causal relationships in a lab
setting, and if the results are
encouraging, conduct the experiment
later in a field setting.
33
Example 7.1: Designing the
Lab Experiment
The researcher should first think of
possible factors that would affect the
production level of the workers, and then try
to control these factors.
The factors that would influence the
production levels of the employees,
other than piece rates, are:
1. Previous job experience.
2. Gender differences.
3. Age
The researcher needs to control these three
variables.

34
Example 7.1: Designing the
Lab Experiment
To control these three variables,
the researcher intends to set up
four groups of 15 people each, for
the lab experiment.
one to be used as the control
group
the other three subjected to
three different pay
manipulations.
35
Example 7.1: Designing the
Lab Experiment
The variables that may impact
on the cause-and-effect
relationship can be controlled in
two different ways:
1. Either by matching the groups or
2. Through randomization.

36
Controlling the Contaminating
Exogenous or Nuisance
Variables
Matching Groups
Is done by matching the various groups
by picking the confounding
characteristics and deliberately
spreading them across groups.
In our example, if there are 20 women
among the 60 members, then each
group will be assigned 5 women.
Likewise, age and experience factors
can be matched across the four groups,
such that each group has a similar mix
of individuals in terms of gender, age ,
and experience. 37
Controlling the Contaminating
Exogenous or Nuisance
Variables
Because the suspected
contaminating factors are matched
across the groups, we may take
comfort in saying that variable X
alone causes variable Y.
But here, we are not sure that
we have controlled all the nuisance
factors, since we may not be
aware of them all. 38
Controlling the Contaminating
Exogenous or Nuisance
Variables
Randomization
Another way of controlling the contaminating
variables is to assign the 60 members
randomly to the four groups.
Every member would have a known and equal
chance of being assigned to any of these four
groups. We might throw the names of all the
60 members into a box, and draw their names.
The first 15 names drawn may be assigned to
the first group, the second 15 to the second
group, and so on.

39
Controlling the Contaminating
Exogenous or Nuisance
Variables
In randomization the process by which
individuals are drawn and their
assignment to any particular group are
both random.
Thus, the confounding variables, age,
sex, and previous experience (the
controlled variables) will have an
equal probability of being distributed
among the groups.

40
Controlling the Contaminating
Exogenous or Nuisance
Variables
Randomization would ensure that
all variables that have effects
(known or unknown factors) on the
dependent variable will be
distributed equally among all
groups. Any causal effects found
would be over and above the
effects of the confounding
variables.
41
The Difference between
Matching and Randomization
We expect that the process of
randomization would distribute the
inequalities among the groups, based
on the laws of normal distribution.
Thus, we need not be concerned about
any known or unknown confounding
factors.
In matching groups, individuals are
deliberately and consciously matched to
control the differences among group
members.
42
The Difference between
Matching and Randomization
Matching might be less effective,
since we may not know all the factors
that could possibly contaminate the
cause-and-effect relationship in any
given situation, and hence fail to match
some critical factors across all groups
while conducting an experiment.
Randomization will take care of this,
since all the contaminating factors will
be spread across all groups.
43
The Difference between
Matching and Randomization
Even if we know the confounding
variables, we may not be able to find
a match for all such variables. For
instance, if we have only 2 women in a
four-group experimental design, we will
not be able to match all the groups with
respect to gender.
Thus, lab experimental designs
involve control of the contaminating
variables through the process of either
matching or randomization, and the
manipulation of the treatment.
44
Table 7.1: Cause and Effect
Relationship after Randomization
Groups Treatment Treatment effect
(% increase in
production over pre-
piece rate system)
Experimental group $1.00 per piece 10
1

Experimental group $1.50 per piece 15

2

Experimental group $2.00 per piece 20

3

Control group (no 0

treatment)
45
Table 7.1 (Cont.)
Note that because the effects of experience,
sex, and age have been controlled in all the
four groups by randomly assigning the
members to them, and the control group had
no increase in productivity, it can be
concluded from the result that the
percentage increases in production are
a result of the piece rate (treatment
effect).
Here we have a high internal validity or
confidence in the cause-and-effect
relationship.
46
Example 5

47
48
49
50
Internal Validity
Internal validity refers to the
confidence we place in the cause-
and-effect relationship.
Internal validity addresses the
question, To what extent does the
research design permit us to say
that the independent variable A
causes a change in the dependent
variable B? 51
Internal Validity
In research with high internal
validity, we are relatively better able to
argue that the relationship is causal,
whereas in studies with low internal
validity, causality can not be inferred
at all.
In lab experiments where cause-and-
effect relationships are substantiated,
internal validity can be said to be high.

52
External Validity or
Generalizability or Lab
Experiments
If we do find a cause-and-effect
relationship after conducting a
lab experiment, can we then
confidently say that the same
cause-and-effect relationship will
also hold true in the
organizational setting?
The answer is NO.
53
External Validity or
Generalizability of Lab
Experiments
The tasks in organizational settings
are far more complex, and there
might be several confounding
variables that cannot be
controlled. Under such
circumstances, we cannot be
sure that the cause-and-effect
relationship found in the lab
experiment is necessarily likely to
hold true in the field setting.
54
The Field Experiment
The field experiment is an experiment
done in the natural environment in
which work goes on as usual, but
treatments are given to one or more
groups.
In the field experiment, even though
it may not be possible to control all the
nuisance variables because members
cannot be either randomly assigned to
groups, or matched, the treatment
can still be manipulated.
55
The Field Experiment
If there are three different shifts
in a production plant, and the
effects of the piece-rate system
are to be studied, one of the shifts
can be used as the control
group, and the two other shifts
given two different treatments or
the same treatment,that is,
different piece rates or the same
56
piece rate.
The Field Experiment
Any cause-and-effect relationship
found under these conditions would
have wider generalizability to other
similar production settings, even
though we may not be sure to what
extent the piece rates alone were the
cause of the increase in productivity,
because some of the other confounding
variables could not be controlled.

57
External Validity
External validity refers to the
extent of generalizability of the
results of a causal study to other
settings, people, or events.
Internal validity refers to the
degree of our confidence in the
causal effect (that variable X
causes variable Y).
58
External Validity
Field experiments have more external
validity (the results are more
generalizable to other similar
organizational settings), but less
internal validity (we cannot be certain of
the extent to which variable X alone
causes variable Y).
In the lab experiment, the reverse
is true. The internal validity is high
but the external validity is rather
59
law.
Trade-Off Between Internal and
External Validity
If we want high internal validity, we
should be willing to settle for lower
external validity and vice versa.
To ensure both types of validity,
researchers usually try first to test the
causal relationships in a tightly
controlled artificial or lab setting and
once the relationship has been
established, they try to test the causal
relationship in a field experiment.
60
Factors Affecting Internal
Validity
Lab experiments could be
influenced by factors that might
affect the internal validity.
These possible confounding factors
pose a threat to internal
validity.

61
Factors Affecting Internal Validity

62
History Effects
Certain events or factors that
would have an impact on the
independent variable-dependent
variable relationship might
unexpectedly occur while the
experiment is in progress, and this
history of events would confound
the cause-and-effect relationship
between the two variables, thus
affecting the internal validity.
63
Example 7
Let us say that the manager of a Dairy
Products Division wants to test the
effects of the buy one, get one free
sales promotion on the sale of the
company-owned brand of packaged
cheese, for a week.
The manager carefully records the sales
of the packaged cheese during the
previous 2 weeks to assess the effect of
the promotion.
64
Example 7 (Cont.)
On the first day the sales
promotion goes into effect, the
Dairy Farmers Association
unexpectedly launches a
multimedia advertisement on the
benefits of consuming dairy
products, especially cheese. The
sales of all dairy products,
including cheese, go up in all the
65
stores.
Example 7 (Cont.)
Here, because of unexpected
advertisement, one cannot be sure how
much of the increase in sales of the packaged
cheese in question was due to the sales
promotion and how much to the advertisement
of the Dairy Farmers Association.
The effects of history have reduced the
internal validity or the faith that can be placed
on the conclusion that sales promotion caused
the increase in sales.
The history effects in this case are illustrated
in Figure 7.1
66
Figure 7.1

67
Maturation Effects
Other uncontrollable variable is the
passage of time which called
maturation effect.
The maturation effects are a function
of the processes operating within the
respondents as a result of the passage
of time.
Examples of maturation processes
could include growing older, getting
tired, feeling hungry, and getting bored.
68
Example 8
Let us say that an R & D director contends that
increases in the efficiency of workers would
result within 3 months time if advanced
technology is introduced in the work setting.
If at the end of the 3 months increased
efficiency is indeed found, it will be difficult to
claim that the advanced technology (and it
alone) increased the efficiency of workers,
because with the passage of time, employees
would also have gained experience,
resulting in better job performance and
therefore in improved efficiency.

69
Example 8 (Cont.)
Thus, the internal validity also
gets reduced owing to the effects
of maturation inasmuch as it is
difficult to pinpoint how much of
the increase is attributable to the
introduction of the enhanced
technology alone.
Figure 7.2 illustrates the
maturation effects in the example.
70
Figure 7.2

71
Testing Effects
The respondents were exposed to the
pretest might influence their responses
on the posttest, which would adversely
impact on internal validity.
For example, if a challenging job is
expected to cause increases in job
satisfaction, and a pretest on job
satisfaction is administered asking for
employees level of satisfaction with
their current jobs.
72
Testing Effects
When a challenging job is introduced
and a further job satisfaction
questionnaire administered
subsequently, the respondents might
now react and respond to the posttest
with a different frame of reference than
if they had not originally been
sensitized to the issue of job satisfaction
through the pretest..
This kind of sensitization through
previous testing is called the testing
effect, which affects the internal
validity of experimental designs.73
Instrumentation Effects
The instrumentation effects might
arise because of a change in the
measuring instrument between pretest
and posttest.
In organizations, instrumentation
effects in experimental designs are
possible when the pretest is done by
the experimenter, treatments are
given to the experimental groups, and
the posttest on measures such as
performance is done by different
managers. 74
Instrumentation Effects
One manager might measure
performance by the final units of
output, a second manager might
take into account the number of
rejects as well, and a third
manager might also take into
consideration the amount of
resources expended in getting the
job done.
75
Selection Bias Effects
It comes from improper or unmatched
selection of subjects for the
experimental and control groups.
For example, if a lab experiment is set
up to assess the impact of working
environment on employees attitudes
toward work, and if one of the
experimental conditions is to have a
group of subjects work for about 2 hours
in a room with some high temperature.
76
Selection Bias Effects
If the researcher select a group of
volunteers who are poor and
unemployed, those will be quite
different from the other workers whom
are not poor or unemployed, and their
responses to the treatment might be
quite different. Such bias in the
selection of the subjects might
contaminate the cause-and-effect
relationships and pose a threat to
internal validity. 77
Statistical Regression
The effect of statistical regression
are brought about when the members
chosen for the experimental group have
extreme scores on the dependent
variable to begin with.
For example, if a researcher wants to
test the understanding of students for
Research Methods classes, he should
not choose those with extremely low
or extremely high ability students for
the experiment.
78
Statistical Regression
This is because we know from the laws of
probability that those with very low scores on
a variable have a greater probability of
showing improvement and scoring closer to
the mean on the posttest after being exposed
to the treatment. Likewise, those with very
high abilities would also have a greater
tendency to regress toward the mean- they
would score lower on the posttest than on the
pretest.
Thus, those who are at either end of the
continuum with respect to a variable would
not truly reflect the cause-and-effect
relationship.
79
Mortality
The mortality of the members in the
experimental or control group or both, is
another confounding factor on the
cause-and-effect relationship.
When the group composition changes
over time, comparison between the
groups becomes difficult, because those
who dropped out of the experiment may
confound the results.

80
Factors Affecting External
Validity
External validity raises issues about
the generalizability of the findings to
other settings. The reasons are:
- the effects of the treatment in lab
experiments are not the same in the
field.
- the selection of the subjects in lab
setting could be very different from the
types of subjects selected by the
organizations.
81
Example 10
Students in a university might be
given a task that could be
manipulated to study the effects
on their performance. The findings
from this experiment cannot be
generalized to the real world of
work, where the employees and
the nature of the jobs would both
be quite different.
82
Factors Affecting External
Validity
Maximum external validity can
be obtained by ensuring that the
lab experimental conditions are
as close to and compatible with
the real-world situation.
Thus, field experiments have
greater external validity than lab
experiments.
83
Simulation
Simulation is an alternative to
lab and field experimentation.
Simulation uses a model-building
technique to determine the
effects of changes.
Computer-based simulations are
becoming popular in business
research.
84
Simulation
A simulation is an experiment
conducted in a specially created setting
that very closely represents the natural
environment in which activities are
usually carried on.
The simulation lies somewhere
between a lab and a field experiment
since the environment is artificially
created but not far different from reality.

85
Simulation
Two types of simulations can be
done:
One in which the nature and timing of
simulated events are totally determined
by the researcher (called
experimental simulation).
The other where the course of activities
is at least partly governed by the
reaction of the participants to the
various stimuli as they interact among
themselves (called free simulation).
86
Simulation
Experimental and free simulations
are both expensive, since
creating real-world conditions in an
artificial setting and collecting data
over extended periods of time
involve a high costs.

87
Simulation
Causal relationships can be
tested since both manipulation
and control are possible in
simulations.
Cause-and-effect relationships
are better established in
experimental simulations where
the researcher exercises greater
control. 88
Areas Where Simulation can
be Used
The effectiveness of various analytic
review procedures in detecting errors in
account balances has been tested
through simulations.
In the finance area, risk management
has been studied through simulations.
Simulations have also been used to
understand the complex relationships in
the financing of pension plans and
making important investment decisions.
89
Simulation
Simulation has also been used by
many companies to test the robustness
and efficacy of various products.
It is quite likely that we will see
simulation being used as a
managerial tool to enhance
motivation, leadership, and the like, in
the future.
Simulation can also be applied as a
problem-solving managerial tool in
other behavioral and administrative
areas. 90
Ethical Issues in
Experimental Design
Research
The following practices are considered
unethical:
Putting pressure on individuals to
participate in experiments through
coercion, or applying social pressure.
Giving menial tasks and asking
demeaning questions that diminish their
self respect.
Deceiving subjects by deliberately
misleading them as to the true purpose
of the research. 91
Ethical Issues in
Experimental Design
Research
Exposing participants to physical or
mental stress.
Not allowing subjects to withdraw from
the research when they want to.
Using the research results to
disadvantage the participants, or for
purposes not to their liking.
Not explaining the procedures to be
followed in the experiment.

92
Ethical Issues in
Experimental Design
Research
Exposing respondents to
hazardous and unsafe
environments.
Not debriefing participants fully
and accurately after the
experiment is over.
Not preserving the privacy and
confidentiality of the information
given by the participants. 93