Educational Research

Chapter 9
Experimental Research

Gay, Mills, and Airasian

Topics Discussed in this Chapter
 Defining characteristics of experimental
research
 Conducting experimental research
 Manipulation and control
 Threats to validity
 Internal validity
 External validity
 Group designs
Defining Characteristics
 Research designed to investigate cause and effect
relationships through the direct manipulation of an
independent variable and control of extraneous
variables
 Independent variable – the variable being manipulated
 Dependent variable – the variable in which the effect of the
manipulation of the independent variable is observed
 Researcher manipulation and control – choice of treatments,
choice of a research design, use of specific procedures, etc.

Objectives 1.1 & 1.2

Experimental Process
 Six steps to conducting experimental research
1. Selection and definition of the problem
 Statement of a hypothesis indicating a causal
relationship between variables
2. Selection of participants and instruments
 Random selection of a sample of subjects from a larger
population
 Random assignment of members of the sample to each
group
 Selection of valid and reliable instruments
Objectives 2.1 & 2.2
Experimental Process
 Six steps to conducting experimental research
(cont.)
3. Selection of a research plan
 Three types of comparisons
 Comparison of two different approaches
 Comparison of new and existing approaches
 Comparison of different amounts of a single approach
4. Execution of the research plan
 Two concerns
 Sufficient exposure to the treatment
 Substantively different treatments

Objectives 2.1, 2.3, & 2.6

Experimental Process
 Six steps to conducting experimental
research (cont.)
5. Analysis of data
6. Formulation of conclusions

Objective 2.1
Manipulation and Control
 Manipulation
 The researcher’s decisions related to what
constitutes the independent variable
 Active and assigned variables
 Active variables are those the researcher actively
manipulates
 Choice of an instructional strategy
 A particular counseling approach
 Assigned variables are those that cannot be manipulated
by the researcher but are of interest
 Gender
 Race
Objectives 3.1 & 3.2
Manipulation and Control
 Control
 The researcher’s efforts to remove the influence of any
extraneous variables that might have an effect on the
dependent variable
 The goal is to be assured the only differences between
groups is that related to the independent variable
 Participant variables – characteristics of the subjects
 Pre-existing achievement levels
 Differences in attitudes
 Environmental variables – characteristics of the context
 Learning materials
 Differences in the time available for treatment between groups

Objectives 3.3 & 3.4

 Experimental Validity
 Internal validity – the degree to which the
results are attributable to the independent
variable and not some other rival explanation
 External validity – the extent to which the
results of a study can be generalized
 Population validity – generalizations related to
other groups of people
 Ecological validity – generalizations related to
other settings, times, contexts, etc.
Objectives 4.1, 4.2, 6.1, & 6.2
 Experimental Validity
 Relative importance of internal and external
validity
 Internal and external validity are related
reciprocally
 Controlling internal validity decreases external validity
 Controlling external validity decreases internal validity
 First demonstrate an effect in a highly controlled
environment (i.e., prioritize internal validity)
 Second replicate the study in a more realistic,
natural setting (i.e., prioritize external validity)
Objective 4.3 & 4.4
Threats to Internal Validity
 History
 Maturation
 Testing
 Instrumentation
 Statistical regression
 Differential selection of participants
 Mortality
 Selection-maturation interaction, etc.
Objectives 5.1 – 5.8
Threats to External Validity
 Pre-test treatment interaction
 Multiple treatment interference
 Selection treatment interaction
 Specificity of variables
 Participants
 Operational definition of the treatment
 Operational definition of the dependent variable
 Specific times
 Specific circumstances
 Treatment diffusion

Objectives 6.3 – 6.7

Threats to External Validity

 Experimenter effects
 Reactive arrangements (Participant Effects)
 Artificial environment
 Hawthorne effect (attention)
 John Henry effect (compensatory rivalry)
 Placebo effect
 Novelty effect

Objectives 6.8 & 6.9

Controlling for Extraneous Variables
 Extraneous variables must be controlled
to be able to attribute the effect to the
treatment
 Group equivalency must be assured
 Four major means to achieve control
 Randomization
 Selection – controls for representation
 Assignment – controls for group equivalency
Objectives 2.5, 7.1, & 8.1
Controlling for Extraneous Variables
 Matching
 Identifying pairs of subjects “matched” on specific
characteristics of interest
 Randomly assigning subjects from each pair to different
groups
 Difficulty with subjects for whom no match exists
 Comparing homogeneous groups
 Restricting subjects to those with similar characteristics
 Restricting subjects results in problems related to
generalization

Objectives 8.4 & 8.5

Controlling for Extraneous Variables
 Using subjects as their own controls
 Multiple treatments across time
 Problem with carry-over effects
 Analysis of covariance (ANCOVA)
 Statistically adjusting the posttest scores for
the subjects in each group for pretest
differences that existed at the beginning of the
study
 Creates statistically equivalent groups

Objectives 8.6 & 8.7

Controlling for Extraneous Variables
 Other ways to control extraneous
variables
 Holding variables constant
 Using only males rather than males and
females
 Selecting teachers with only similar levels of
experience
 Selecting only one grade level
 Stipulating the specific length of a treatment
Objective 8.2
Group Designs

 Two major classes of group designs

 Single-variable designs – one independent
variable
 Factorial designs – two or more independent
variables
 Three types of experimental designs
 Pre-experimental designs
 Experimental designs
 Quasi-experimental designs
Objectives 9.1, 9.2, 10.1, & 10.2
DESIGNS
Experiments
If X, then Y

or, in more colloquial terms:

If the program is given, then the

outcome occurs
 If X, then Y
and
If not X, then not Y

Or, once again more colloquially:

If the program is given, then the outcome occurs

and
If the program is not given, then the outcome does not
occur
NOTATION
 Pre-Experimental Designs
 Three types
 One-shot case study
X O
 One-group pretest-posttest design
O X O
 Static group comparison
X1 O
X2 O
 Threats to internal validity – see Figure 9.1
Objective 9.2
 True Experimental Designs
 Three types
 Pretest-posttest control group design
R O X O
R O O
 Posttest only control group design
R X O
R O

Objective 9.4
 True Experimental Designs
 Three types (cont.)
 Solomon four-group comparison
R O X O
R O O
R X O
R O
 Threats to internal validity – see Figure
9.2
Objective 9.4
 Quasi-Experimental Designs
 Three types
 Non-equivalent control group design
O X O
O O
 Time series design
O O O O X O O O O
 Counterbalanced design
O X1 O X 2 O X 3 O
O X3 O X 1 O X 2 O
O X2 O X 3 O X 1 O
 Threats to internal validity – see Figure 9.2
Objective 9.5
Factorial Designs
 Two independent variables and one
dependent variable
 The effect of teaching strategy and gender on
students’ achievement
 The effect of a particular counseling technique
and the clients’ ethnicity on the success of the
treatment
 The effect of a specific coaching approach and
children in three age groups on the ability to
perform certain physical tasks
Objective10.1
Factorial Designs
 This design increases explained variance and
reduces unexplained variance
 Explained variance is that which can be accounted
for by the independent variable(s)
 By adding an additional variable into the design the
explained variance is likely going to increase
 Unexplained variance is that which cannot be
accounted for by the independent variable(s)
 Because it is unexplained, the researcher assumes this is
due to error
 This is also known as error variance

Objectives 10.1 & 10.2

Factorial Designs
 Interaction
 The degree to which changes in the dependent
variable are different depending on the levels of
each of the independent variables
 A particular instructional strategy is more effective
for males than females
 A particular counseling technique is more effective
when the ethnicity of the counselor and client are
similar

Objective 11.1
Factorial Designs
 Interaction
 Visually explained by a graph of
performance of all levels of both
independent variables
 Parallel lines indicate no interaction
 Non-parallel lines indicate an interaction
 Statistically examined with an F-test of the
statistical significance of the interaction

Objective 11.1
Factorial Designs
 Interaction
 Visual presentation of a significant interaction
14

12

10

TEST
T est Scores

Small Tes ting Room

4 Large Testing Room

Small Lec ture R oom Large Lecture Room

LECTURE

 Statistical analysis of this significant interaction:

F1,16 = 4.88 (p = .0420)
Objective 11.1
Factorial Designs
 Interaction
 Visual presentation of a non-significant interaction
18

16

14

12
T est Scores

10
TEST

Small Tes ting R oom

8 Large Testing R oom

Small Lec ture R oom Large Lec ture Room

LECTURE

 Statistical analysis of this non-significant

interaction: F1,16 = 0.97 (p = .340)
Objective 11.1
 GROUP WORK
Group of 3 to 4 students
Answer the following questions

1. Explain the similiraties and differences

between pre-experiment, true
experiment, and quasi-experiment
2. Explain 3 designs of pre-experiments, 3
design of quasi-experiments, and 3
designs of true experiment.