Research Methods in Psychology II

Chapter 6: Independent Groups Designs – Part III

 Outline and the Learning Outcomes

• RANDOM GROUPS DESIGN

– Threats to Internal Validity
– Block Randomization
• MATCHED GROUPS DESIGN
• NATURAL GROUPS DESIGN
• ANALYSIS AND INTERPRETATION OF EXPERIMENTAL FINDINGS
• ESTABLISHING THE EXTERNAL VALIDITY OF EXPERIMENTAL FINDINGS

The students who succeeded in this course;

• Will be able to explain the key concepts of the experimental
method and identify these concepts in experimental studies.
• Will be able to identify different types of experimental research
used in research.
2
Threats to Internal Validity

• Ability to make causal inferences is threatened when

– Intact groups of subjects are used
– Extraneous variables are not controlled
– Selective subject loss occurs
– Demand characteristics and experimenter effects are not controlled

3
Testing Intact Groups
• Random assignment is used to form comparable groups in the random
groups design.
• There are times, however, when noncomparable groups are formed even
when random assignment appears to have been used.
• This problem occurs when intact groups (not individuals) are randomly
assigned to the conditions of an experiment.

4
Balancing Extraneous Variables
• Potential variables that are not directly of interest to the researcher but
that could still be sources of confounding in the experiment are called
extraneous variables.
– For example, to complete an experiment more quickly, a researcher might decide to
have several different experimenters test small groups of participants.
– If all the individuals in the experimental group were tested by one experimenter and all
of those in the control group were tested by another experimenter, the levels of the
intended independent variable would become confounded with the two experimenters.

• Block randomization controls extraneous variables by balancing them

across groups.
– For example, if there were four different experimenters, entire blocks of the block-
randomized schedule would be assigned to each experimenter.

5
Subject Loss
• When subjects begin an experiment but fail to complete it successfully,
subject loss occurs.
Mechanical Subject Loss
• Occurs when a subject fails to complete
the experiment because of an
equipment failure
• Mechanical loss can also reasonably be
understood as the result of chance
events that should occur equally across
groups.
• It should be documented.
• The name or subject number of the
dropped subject and the reason for the
loss should be recorded.
• The lost subject can then be replaced by
the next subject tested.
Selective Subject Loss
• (1) when subjects are lost differentially
across the conditions of the experiment;
• (2) when some characteristic of the
subject is responsible for the loss; and
• (3) when this subject characteristic is
related to the dependent variable used
to assess the outcome of the study.
• Destroys the comparable groups
• Preventive steps can be taken:
• To administer a pretest and screen out
subjects who are likely to be lost
• Researchers can give all subjects a
pretest but then simply randomly assign
participants to conditions. Then, if a
subject is lost from the experimental
group, a subject with a comparable
pretest score can be dropped from the
control group.
6
Placebo Control and Double-Blind Experiments

• Demand characteristics refer to the cues and other information that

participants use to guide their behavior in a psychological study.
• Potential biases can also arise due to the expectations of the
experimenters called experimenter effects.

7
Placebo Control and Double-Blind Experiments
• Researchers use a placebo control group as one way to control demand
characteristics.
– A placebo is a substance that looks like a drug or other active substance but is actually
an inert, or inactive, substance.

• The use of placebo control groups in combination with a double-blind

procedure can control for both demand characteristics and experimenter
effects.
• In a double-blind procedure, both the participant and the observer are
blind to (unaware of) what treatment is being administered.

https://www.youtube.com/watch?v=_jyCZ8S3Eco

8
How many subjects are
required for this averaging
process to work as it should?

9
 Matched Groups Design
• More subjects will be needed to average out individual differences when
samples are drawn from a heterogeneous population than from a
homogeneous one.
– Random assignment will not be effective in balancing the differences
among subjects when small numbers of subjects are tested from
heterogeneous populations.

10
 Matched Groups Design
• Using a repeated measures design
• Nevertheless, some independent variables require separate groups of
subjects for each level.
– For instance, suppose researchers wish to compare two types of postnatal care for
premature infants and it is not possible to give both types of care to each infant.

• In this situation, and many others, researchers will need to test separate
groups in the experiment.
• The matched groups design is a good alternative when neither the random
groups design nor the repeated measures design can be used effectively.

11
 Matched Groups Design
• Procedure
– Select matching variable
• related to outcome or dependent variable
– Match pairs (or triples, quadruples, etc. depending on number of
conditions) of identical or similar scores.
– Randomly assign participants within each match to the different IV
conditions.

12
 Matched Groups Design
• The most preferred matching task is one that uses the same task that will
be used in the experiment itself.
– For example, if the dependent variable in the experiment is blood pressure, participants
should be matched on blood pressure prior to the start of the experiment.
– The matching is accomplished by measuring the blood pressure of all participants and
then forming pairs or triples of participants (depending on the number of conditions in
the experiment) who have identical or very similar blood pressures.
– Thus, at the start of the experiment, participants in the different groups will have, on
average, equivalent blood pressure.

13
 Matched Groups Design
• In some experiments, the primary dependent variable cannot be used to
match subjects.
– For example, consider an experiment that teaches participants different approaches to
solving a puzzle.
– If a pretest were given to see how long it took individuals to solve this puzzle, the
participants would likely learn the solution to the puzzle during the pretest.
– In this situation the next best alternative for a matching task is to use a task from the
same class or category as the experimental task.

14
 Natural Groups Design
• Researchers in many areas of psychology are interested in independent
variables that are called individual differences variables, or participant
variables.
• An individual differences variable is a characteristic or trait that varies
across individuals.
– Religious affiliation is an example of an individual differences variable.
– Researchers can’t manipulate this variable by randomly assigning people to
Catholic, Jewish, Muslim, Protestant, or other groups.
– Instead, researchers “control” the religious affiliation variable by systematically
selecting individuals who naturally belong to these groups.

15
 Natural Groups Design
• Experiments involving independent variables whose levels are selected—
like individual differences variables—are called natural groups designs.
• The natural groups design is frequently used in situations in which ethical
and practical constraints prevent us from directly manipulating
independent variables.
– If we were interested in the relationship between divorce and emotional disorders, we
could not randomly assign some people to get divorced.

16
 Natural Groups Design
• Researchers use natural groups designs to meet the first two objectives of
the scientific method: description and prediction.
• Improve causal inferences: study individual differences variables in
combination with manipulated IVs.

17
Analysis and Interpretation of Experimental Findings

• Use statistical analysis to

– Claim IV produced an effect on DV
– Rule out the alternative explanation that chance produced any
observed effect
• Replication
– Best way to determine whether findings are reliable
– Repeat experiment and see if same results are obtained

18
Analysis of Experimental Designs

• Three steps
– Check the data
• Errors? outliers?
– Describe the results
• Descriptive statistics such as means, standard deviations, effect size
• Confirm what the data reveal
– Inferential statistics

19
Analysis of Experimental Designs

• Descriptive Statistics
– Mean (central tendency)
• Average score on DV, computed for each condition
• Not interested in each individual score, but how people responded on
average in a condition
– Standard deviation (variability)
• Average distance of each score from the mean of a group
• Not everyone responds the same way to an experimental condition

20
Analysis of Experimental Designs
• Effect size
– the strength of the relationship between the independent and
dependent variables.
– Cohen’s d
difference between treatment and control means
average variability for all participants’ scores

Guidelines for interpreting Cohen’s d:

small effect of IV: d = .20
medium effect of IV: d = .50
large effect of IV: d = .80

21
Confirming What the Results Reveal

• Researchers use inferential statistics to determine whether an

independent variable has a reliable effect on a dependent variable.
• Researchers use null hypothesis testing to determine whether mean
differences among groups in an experiment are greater than the
differences that are expected simply because of error variation.
• Effect of an IV on the DV is statistically significant when the probability of
the observed results being due to chance is low.

22
Null Hypothesis Significance Testing (NHST)
Steps for Null Hypothesis Testing
• (1) Assume the null hypothesis is true.
– The population means for groups in the experiment are equal.
• (2) Use sample means to estimate population means.
– Example
mean body dissatisfaction for Barbie = -.76
mean body dissatisfaction for Emme = 0.00
mean body dissatisfaction for neutral = 0.00
difference between Barbie and Emme/neutral = -.76
• Is the observed mean difference (-.76) greater than what is expected
when we assume the null hypothesis is true (zero difference)?

23
Null Hypothesis Significance Testing (NHST)

Steps for Null Hypothesis Testing

• (3) Compute the appropriate inferential statistic.
• (4) Identify the probability associated with the inferential statistic
• (5) Compare the observed probability with the predetermined level of
significance (alpha), which is usually p < .05

24
Null Hypothesis Significance Testing (NHST)
Confidence intervals
• Confidence interval for a mean provides the range of values that contains
the true population mean.
– with some probability, usually .95
• Compute confidence interval around sample mean in each condition.

25
Null Hypothesis Significance Testing (NHST)
Confidence intervals
• Compute confidence interval around sample mean in each
condition.
– If CIs do not overlap, we gain confidence that the population means
for the conditions are different
– If CIs overlap slightly, we are uncertain about the true mean
difference.
– If CIs overlap such that the mean of one group lies within interval of
another group, we conclude the population means do not differ.

26
External Validity
• Extent to which findings from an experiment can be generalized to
describe individuals, settings, and conditions beyond the scope of a
specific experiment.
– Any single experiment has limited external validity
– External validity of findings increases when findings are replicated in a new
experiment

• Questions of external validity

– Would the same findings occur
• In different settings?
• In different conditions?
• With different participants?

27
External Validity
• Increasing external validity
– Include characteristics of situations, settings, and population to which
researchers seek to generalize
– Partial replications
• A partial replication can help to establish external validity by showing that a similar
experimental result occurs when slightly different experimental procedures are
used.

– Field experiments

28
More experimental research concepts

• Floor effect
• Ceiling effect
?
• Manipulation check

29