Two General

Criteria of Research
Designs
Razmen R. Pinto
 OVERVIEW

Here’s what you’ll find in my report:

 Internal Validity vs. External Validity
 Trade-off between Internal and External Validity
 Threats to Internal Validity and how to counter them
 Types of External Validity
 Threats to External Validity and how to counter them
INTERNAL VALIDITY vs. EXTERNAL VALIDITY

Internal validity External validity

 refers to the degree of  refers to the extent to
confidence that the causal which results from a
relationship being tested study can be applied
is trustworthy and not (generalized) to other
influenced by other situations, groups or
factors or variables. events.
Trade-off between Internal and
External Validity

There is an inherent trade-off between internal and

external validity; the more you control extraneous factors
in your study, the less you can generalize your findings to
a broader context.
 Example:
You want to test the hypothesis that drinking a cup of coffee improves memory.
You schedule an equal number of college-aged participants for morning and
evening sessions at the laboratory. For convenience, you assign all morning
session participants to the treatment group and all evening session
participants to the control group.

Once they arrive at the laboratory, the treatment group participants are given a cup
of coffee to drink, while control group participants are given water. You also
give both groups memory tests. After analyzing the results, you find that the
treatment group performed better than the control group on the memory test.
How to check whether your
study has internal validity?
 Example:
You want to test the hypothesis that drinking a cup of coffee improves memory.
You schedule an equal number of college-aged participants for morning and
evening sessions at the laboratory. For convenience, you assign all morning
session participants to the treatment group and all evening session
participants to the control group.

Once they arrive at the laboratory, the treatment group participants are given a cup
of coffee to drink, while control group participants are given water. You also
give both groups memory tests. After analyzing the results, you find that the
treatment group performed better than the control group on the memory test.
 Conditions

1 2 3

Your treatment and response Your treatment precedes No confounding or extraneous

variables change together. changes in your response factors can explain the results
variables of your study.

 Drinking coffee and  Drinking coffee × The time of day of the

memory performance happened before the sessions is an extraneous
increased together. memory test. factor that can equally
explain the results of the
study.
Threats to Internal
Validity and How to
counter them
The management of company X wants to know if flexible
working hours will improve job satisfaction among
employees. They set up an experiment with two groups: 1)
control group of employees with fixed working hours 2)
experiment group with employees with flexible working
hours. The experiment will run for six months. All employees
fill in a survey measuring their job satisfaction before the
experiment (pre-test) and after the experiment (post-test).

—Research example
 Threats

History Maturation Testing

An unrelated event The outcomes of the study The pre-test (used to
influences the outcomes. vary as a natural result of establish a baseline)
time. affects the results of the
post-test.

Participant selection
Participants in the control and experimental
group differ substantially and can thus not
be compared.
 Threats

Attrition Regression towards Instrumentation

Over the course of a (longer) mean There is a change in how the
study, participants may drop Extreme scores tend to be dependent variable is
out. If the drop out is caused closer to the average on a measured during the study.
by the experimental treatment second measurement.
(as opposed to coincidence) it
can threaten the internal
validity.

Social interaction
Interaction between
participants from different
groups influences the
outcome.
 How to
counter
threats to
Internal
Validity
 Threats

History
Evaluators should do their
best to identify any external
events or changes that may
impact their program results
Maturation
If feasible within your
evaluation questions, reducing
the amount of time between
the pretest and posttest can
limit maturation threats
Testing
Keep an eye out for this threat
whenever there is a pretest-
posttest design and no
comparison group to help
control for the learning curve
of taking the pretest.

Participant selection
Be alert for this potential threat if you are
working with a nonequivalent comparison
group.
 Threats

Attrition Regression towards Instrumentation

Evaluators should seek to mean Keep an eye out for this if
employ intensive strategies to Avoid selecting participants there are multiple
minimise study attrition such based on extreme performance observation/test points in your
as through offering incentives, or scores study
alternative locations for data
collection and repeat
reminders via mail

Social interaction
The threats described here can
often be minimized by
constructing multiple groups
that are not aware of each
other
Types of External
Validity

There are two main types of external

validity: population validity and
ecological validity.
1. Population validity
Population validity refers to whether you
can reasonably generalize the findings
from your sample to a larger group of
people (the population).
Example
You want to test the hypothesis that people tend to
perceive themselves as smarter than others in terms
of academic abilities. Your target population is the
10,000 undergraduate students at your university.
You recruit over 200 participants. They are science
and engineering majors; most of them are Filipino,
male, 18–20 years old and from a high
socioeconomic background. In a laboratory setting,
you administer a mathematics and science test and
then ask them to rate how well they think performed.
You find that the average participant believes they
are smarter than 66% of their peers.

2. Ecological validity
Ecological validity refers to whether
you can reasonably generalize the
findings of a study to other situations
and settings in the ‘real world’.
Example
You want to test the hypothesis that driving
reaction times become slower when people pay
attention to others talking.
In a laboratory setting, you set up a simple
computer-based task to measure reaction times.
Participants are told to imagine themselves
driving around the racetrack and double-click
the mouse whenever they see an orange cat on
the screen. For one round, participants listen to a
podcast. In the other round, they do not need to
listen to anything. After assessing the results,
you find that reaction times are much slower
when listening to the podcast.
 Threats to
External Validity
and How to
counter them
 Example
A researcher wants to test the hypothesis that
people with clinical diagnoses of mental
disorders can benefit from practicing mindfulness
daily in just two months time. They recruit people
who have been diagnosed with depression for at
least a year, are aged between 20–29, and live
locally.
Participants are given a pretest and a post-test
measuring how often they experienced anxiety in
the past week. During the study, all participants
are given an individual mindfulness training and
asked to practice mindfulness daily for 15
minutes in the morning.

Since the levels of anxiety decreased between

the pre- and post-test, the researcher concludes
that all clinical populations can benefit from
mindfulness.
 Threats to External Validity
Experimenter
Sampling Bias History
effect

The characteristics or
The sample is not behaviors of the
An unrelated event
representative of the experimenter(s)
influences the outcomes.
population. unintentionally influence
the outcomes.
 Threats to External Validity

Hawthorne Aptitude- Situation

Testing effect
effect treatment effect

The tendency for

participants to Interactions between Factors like the
The administration characteristics of the setting, time of day,
change their
of a pre- or post- group and individual location, researchers’
behaviors simply variables together characteristics, etc.
test affects the
because they know influence the limit generalizability
outcomes.
they are being dependent variable. of the findings.
studied.
 How to counter
Threats to
External Validity
 How to counter threats to external validity

Field Probability
Replications Recalibration
Experiments Sampling

Probability sampling
Replications counter Recalibration or
Field experiments counters selection
almost all threats by reprocessing also
counter testing and bias by making sure
enhancing counters selection
everyone in a
generalizability to situation effects by population has an
bias using algorithms
other settings, using natural to correct weighting
equal chance of being
populations and contexts. of factors (e.g., age)
selected for a study
conditions. within study samples.
sample.