EXP FINALS REVIEWER
are insufficient subjects to
Within subject designs small
N
What is a large n design?
A large N design compares
the performance of groups of
subjects.
How does a large N design
differ from a small N design?
A small N design studies one
or two subjects, often using
variations of the ABA reversal
design (A = baseline
condition; B = experimental
condition; A = verify the
change in behavior).
Aggregate effects
Aggregate effects are the
pooled findings from many
subjects
Why do small N research
challenge large N
experiments?
They argue that large N
studies ignore individual
subject responses to the IV
and instead report aggregate
results or trends. When
subjects vary greatly in their
response to the IV, this can
create the appearance of no
difference between the
groups.
Why might a clinical
psychologist use small N
designs?
A clinical psychologist could
use a small N design to
test a treatment when there
decision of many clinical
conduct a large N study and
when she wants to avoid the
ethical problem of an
untreated control group.
how about animal
researcher?
Animal researchers prefer
small N designs to minimize
the acquisition and
maintenance cost, training
time, and possible sacrifice of
their animal subjects.
Which historical
development caused the
shift to large N design?
Sir Ronald Fisher’s (1935)
creation of the analysis of
variance allowed inferential
testing of large N data.
Where have small N designs
been most extensively used?
Small N designs have been
most extensively used in
operant conditioning
research.
B. F. Skinner examined the
continuous behavior of
individual subjects in
preference to analyzing
discrete measurements from
separate groups of subjects.
Baseline
In both large and small N
designs, baseline are control
conditions that allow us to
measure behavior without
the influence of the IV.
How did Kazdin explain the
researchers to end without a
return to baseline?
It would be ethically
indefensible to cause a
patient to relapse by
returning to baseline after
treatment appeared to
improve behavior.
When is this most
important?
When relapse threatens the
health or safety of the
patient or others, as in in
self-injurious, and suicidal or
homicidal behavior
What price do researchers
pay when they can’t return
to baseline?
They can’t rule out the
possibility that the patient’s
clinical improvement was
caused by an extraneous
variable.
Multiple baseline design
In a multiple baseline design,
a series of baselines and
treatments are compared
within the same subject, and
once treatments are
administered, they are not
withdrawn.
This approach could also be
used to evaluate the effect of
a treatment administered to
different individuals after
baselines of different
lengths. A researcher can
evaluate the effects of a
treatment on two or more
behaviors or on the same
behavior in different settings.
How might this design
overcome the ethical
problem of withdrawing an
effective treatment?
In a multiple baseline design,
an experimenter never
withdraws treatments after
administering them.
How do researchers analyze
data from small N
experiments?
Researchers often visually
inspect changes in the
dependent variable across
treatment conditions. The
independent variable’s effect
is often apparent.
They may also use statistics
to analyze small N data.
Why is statistical analysis of
small N data controversial?
Critics are concerned about
generalizing from a single
subject to a population.
Unless 50 measurements are
taken during each baseline
and treatment phase,
important assumptions
underlying inferential tests
may be violated.
Changing criterion designs
In changing criterion designs,
the criteria for reinforcement
are incrementally increased
as
participants succeed. For
example, initially, a subject
might receive a reward for 30
minutes of daily exercise,
later, for 45 minutes, and
finally, for 60 minutes.
Reinforcement for successive
approximations of the target
behavior is central to athletic
training, behavior
modification, and
biofeedback and neuro
feedback.
Discrete trials design
A discrete trials design is a
small N design without
baselines used in
psychophysical research.
Instead, the impact of
different levels of the
independent variable is
averaged across 100s to
1000s of trials.
How does a discrete trials
design differ from a typical
experiment?
A discrete trials design has
no baselines and administers
the levels of the independent
variable 100s to 1000s of
times to each subject.
What are a discrete trails
designs benefits?
The large number of data
points produced by 100s to
1000s of trials provides a
very reliable measurement of
the effect of the independent
variable.
The similarity of human
sensory systems allows
researchers to generalize
from a small number of
subjects.
When is a small N designs
appropriate?
When studying a clinical
subject (a self-injurious child)
or when very few subjects
are available.
When would we prefer a
large N design?
A large N design would be
desirable when we have
sufficient subjects and want
to increase generalizability.
The generalizability of a large
N study depends on how we
select our sample since a
seriously biased sample will
not represent the
population.
Why doesn’t a large N study
always have greater
generalizability than a small
N study?
If a large N study’s sample is
biased, we will be unable to
generalize its findings to a
larger population. Also, if it is
poorly controlled, there will
be no valid findings to
generalize.
Why doesn’t a large N study
always have greater
generality a small N study?
In contrast, a well-controlled
small N experiment
using a single subject might
be successfully replicated
across sufficient subjects to
generalize its results to the
population from which they
were drawn.
Why we need statistics?
The Argument for Using
Statistics
Statistics are quantitative
measurements of samples.
Through statistics we can
quantify the phenomena we
observe. It provides
researchers to with objective
and consensual techniques
for describing their results.
What do statistics tell us?
Statistical Inference is about
making a statement about
the population and all its
samples based on what we
see in the data that we have.
Descriptive statistics
describe sample central
tendency and variability.
Inferential statistics allow us
to draw conclusions about a
parent population from a
sample. In research, we do
not outright conclude a
statement; but rather proves
a certain probability.
Define sample and
population
A population is a set of
people, animals, or objects
that
share at least one
characteristic in common
(like passengers on a car).
A sample is a subset of the
population that we use to
draw inferences about the
population.
What does it mean when we
conclude that our scores
probably came from same
population?
The differences in scores
obtained from separate
treatment groups are not
significantly greater than
what we might expect
between any samples
randomly drawn from this
population. When
researchers report this
outcome, it means that were
was no treatment effect.
What is variability?
For a set of dependent
variable measurements,
there is variability when the
scores are different.
Variability “spreads out” a
sample of scores drawn from
a population.
What is null hypothesis?
The null hypothesis (H0) is
the statement that the
scores came from the same
population and the
independent variable did not
significantly affect the
dependent variable.
What is statistical
significance?
Results are statistically
significant when the
difference between our
treatment groups exceeds
the normal variability of
scores on the dependent
variable.
Statistical significance means
that there is a treatment
effect at an alpha level we
have preselected, like .01 or
.05.
Alternative hypothesis
The alternative hypothesis
(H1) is the statement that
the scores came from
different populations the
independent variable
significantly affected the
dependent variable.
When may we reject the null
hypothesis?
We may reject the null
hypothesis when the
differences between
treatment groups exceed the
normal variability in the
dependent variable at our
chosen level of significance.
What does a frequency
distribution of scores
reveal?
The frequency distribution
displays the number of
individuals contributing a
specific value of the
dependent variable in a
sample.
What does a frequency
distribution of score reveal?
The values of the dependent
variable are indicated
on the horizontal X-axis
(abscissa) and the
frequencies of these values
are indicated on the vertical
Y-axis (ordinate). You can
calculate the total number of
participants by adding the
frequencies.
Why does rejecting the null
hypothesis depend on data
variability?
The decision to accept or
reject the null hypothesis
depends on whether the
differences we measure
between treatment groups
are significantly greater than
the normal variability among
people in the population.
The greater the normal
variability in the population,
the larger the difference
between groups required to
reject the null hypothesis.
Contrast directional and
non-directional hypothesis
A directional hypothesis
predicts the “direction” of
the difference between two
groups on the dependent
variable.
For example: The
experimental group will
lower their systolic blood
pressure more than the
control group.
A non-directional hypothesis
predicts that the two groups
will have different values on
the dependent variable:
For example: The
experimental group and
control
group will achieve different
systolic blood pressure
reductions.
What is a significance level
and how do we select one?
The significance level (alpha)
is our criterion for deciding
whether to accept or reject
the null hypothesis.
Psychologists do not use a
significance level larger than
.05.
What is significance level
and how do we select one?
A significance level of .05
means that a pattern of
results is so unlikely that it
could have occurred by
chance fewer than 5 times
out of 100.
What are type 1 and type 2
errors?
A Type 1 error (a) is rejecting
the null hypothesis when it is
correct. The experimenter
determines the risk of a Type
1 error by selecting the alpha
level.
A Type 2 error (b) is
accepting the null hypothesis
when it is false.
How should we support null
hypothesis testing?
An American Psychological
Association task force
recommended that
researchers include
estimates of effect size and
confidence intervals, in
addition to p values.
When you calculate a p value
that is statistically significant,
this means that your results
are unlikely to be due to
chance (are probably real).
Effect size
Effect size estimates the
strength of the association
between the independent
and dependent variable—
the percentage of the
variability in the dependent
variable is due to the
independent variable.
How should we support null
hypothesis testing?
A confidence interval is a
range of values above and
below a sample mean that is
likely to contain the
population mean (usually
95% or 99% of the time).
What is critical region?
A critical region is a region of
the distribution of a test
statistic sufficiently extreme
to reject the null hypothesis.
For example, if our criterion
is the .05 level, the critical
region consists of the most
extreme 5% of the
distribution.
To reject the null hypothesis,
the test statistic would have
to fall within the shaded
critical region.
What are the one tailed and
two tailed test?
A one-tailed test has a
critical region at one tail of
the distribution. We use a
one-tailed test with a
directional hypothesis.
A two-tailed test has two
critical regions, found at
opposite ends of the
distribution. We use a two-
tailed test with a non-
directional hypothesis.
What is the function of
inferential statistics?
Inferential statistics allow us
to predict the behavior of a
population from a sample.
Examples of inferential
statistics are the t test and F
test.
Analyzing results
The parameters of data
analysis
 How many
independent
variables are there?
 How many treatment
conditions are there?
 Is the experiment
run between or
within subjects?
 What is the level of
measurement of the
dependent variable
 Are the subjects
matched?
Summarize the properties of
the four levels of
measurement
A nominal scale assigns tines
to two or more distinct
categories that can be
named using a shared
feature, but does not
quantify items.
Example: you can sort
pictures into attractive and
unattractive categories.
An ordinal scale measures
the magnitude of the DV
using ranks. This scale allows
us to make statements about
contestants’ relative speed.
Example: marathon
contestants are assigned to
places from first place to last
place.
An interval scale measures
the magnitude of the DV
using equal intervals
between values with no
absolute zero point.
Example: Fahrenheit or
Centigrade temperatures,
and Sarnoff and Zimbardo’s
0-100 scale.
A ratio scale measures the
magnitude of the DV using
equal intervals between
values and
an absolute zero. This scale
allows us to state that 2
meters are twice as long as 1
meter.
Example: distance in meters
or time in seconds.
How does the level of
measurement affect the test
we sue?
Nonparametric tests use
nominal or ordinal data.
Parametric tests require
interval or ratio data.
When should we use the chi
square test?
When the data are nominal
and the groups are
independent, which means
the experimenter assigns
different subjects to them.
What does it reveal?
The chi square test
determines whether the
frequency of sample
responses represents the
frequencies we would expect
in the population.
Explain the meaning of the
obtained (C2 obt) and critical
values, and cramer’s
coefficient delta?
The c2 obt is the actual
frequency of responses.
The critical value is the
minimum value required to
reject the null hypothesis.
Cramer’s coefficient Φ is
analogous to r2 and indexes
the degree of association
between priming and the
number of incorrect
responses.
If our sample included every
member of the population,
we would have the maximum
possible degrees of freedom
and would know the exact
population values of the
mean and variance.
How do we decide whether
to accept or reject the null
hypothesis?
If x2 obt > x2 critical, reject
the null hypothesis.
How does sample size affect
the t-test?
The sample size determines
the degrees of freedom
There is a different t
distribution for each value of
degrees of freedom.
How does t distribution
change as sample size
increases?
The t distribution approaches
a normal curve as sample
size increases.
What does robustness
mean?
The t test provides a valid
test of the hypothesis when
assumptions like normal
distribution of population
values are slightly to
moderately violated.
When may we reject the null
hypothesis?
We reject the null hypothesis
when tobt > tcrit. For 9 df, if
tobt > 2.262, we would reject
the null hypothesis.
Calculate an effect size for a
t test for independent
groups.
First, we calculate the t
statistic (2.47) and then we
enter it into the following
formula:
What is a t test for matched
groups?
A t test for matched groups
either assigns the same
subjects to both conditions
or matches subjects and then
randomly assigns them to
either condition.
A t test for matched groups
may use fewer subjects and
achieve greater control over
individual differences than a
t test for independent
groups.
This makes a t test for
matched groups potentially
more powerful.
When do we use an analysis
of variance?
We use an analysis of
variance when data are
interval or ratio level and
there is at least one
independent variable with
three or more levels.
What are within-groups
variability and between-
groups variability?
Within-groups variability is
the degree to which the
scores of subjects in the
same treatment group differ
from each other.
Within-groups variability
consists of error due to
individual differences and
extraneous variables.
Between-groups variability is
the degree to which the
scores of different treatment
groups differ from one
another or the grand mean.
Between-groups variability
consists of error due to
individual differences and
extraneous variables and
treatment effects.
What does it mean when an
F ratio is statistically
significant?
Across all group means,
there is a significant
difference due to the
independent variable.
We compare the relative
sizes of the components of
ANOVA by computing a ratio
between them
When is post hoc test
appropriate?
When an overall ANOVA is
significant and you have
made no specific predictions,
you may perform
post hoc tests on all pairs of
treatment groups. This is a
test done after the overall
analysis indicates a
significant difference.
How many comparisons may
we perform?
You may perform all possible
pairwise comparisons
without increasing the risk of
Type 1 error.
When may we use a prior
test?
We may use a priori tests to
test predictions of
differences between groups,
such as between two groups
or between one group and
the others.
The maximum number of
comparisons = p – 1, where p
is the number of treatment
groups.
What is the advantage of a
priori test over a post hoc
test?
A priori tests are more
powerful than post hoc tests;
but you may perform fewer a
priori tests.
Effect size
Effect size measured by η2 is
the proportion of the
variability in the dependent
variable that can be
accounted for by the
independent variable.
η2 indexes the strength of
the relationship between the
independent and dependent
variables.
Drawing conclusions, the
search for the elusive
bottom line
What is internal validity?
An experiment is internally
valid when the effects on the
dependent variable are due
to the independent variable.
An internally valid
experiment is free of
confounding.
What is a manipulation
check?
A manipulation check
evaluates how well the
experimenter manipulated
the experimental situation.
A manipulation check
determines whether subjects
followed directions and were
appropriately affected by our
treatments.
What did Orne (1969) mean
by a pact of ignorance?
Subjects expect their data to
be discarded if they guess
the experimental hypothesis,
and don’t volunteer this
information to the
experimenter.
Note: Debrief, to counter
this, subjects need to believe
that you really need to
believe that you really want a
truthful answer.
Experimenters don’t want to
test additional subjects and
may take subject reports at
face value instead of
requesting additional
information.
Note: when you interpret,
always be open to other
potential explanation for
your findings.
Which mistakes could
produce this threat to
internal validity?
Selecting the wrong
statistical test Using a t-test
to analyze ordinal data.
Improperly using a statistical
test Calculating multiple t-
tests.
Drawing the wrong
conclusions from the test
Reporting p = .07 as a trend.
Statistical Conclusion
Validity – the validity of
drawing conclusions about a
treatment effect from the
statistical results that were
obtained.
External Validity
An experiment is externally
valid when its findings can be
extended to other situations
and populations.
What two requirements
must an externally valid
study satisfy?
1. The experiment must be
internally valid.
2. The experimental findings
can be replicated.
What does it mean to
generalize across subject?
why is this important?
The findings can be extended
to a larger group than our
sample.
Generalizing across subjects
is critical to the external
validity and usefulness of
experimental findings.
Which problems prevent us
from generalizing across
subjects?
The samples used in
psychological research are
often biased and may not
represent the larger
population.
The samples may not always
represent even college
sophomores since we heavily
depend on volunteers.
Note: volunteers can be
quite different from non-
volunteers.
Explain generalizing from
procedures to concepts
Experimental variables like
anger may have multiple
operational definitions.
When we generalize from
our experimental results, we
move from discussing our
specific operational
definition of anger to
discussing the concept of
anger itself.
Why is this a problem in
research?
It is dangerous to generalize
from a single
experiment’s operational
definition of anger.
We cannot be sure of the
reliability or validity of our
procedures.
What is research
significance?
A study achieves research
significance when its findings
clarify or extend knowledge
gained from previous studies
and raise implications for
broader theoretical issues.
When should we question
novel findings?
We should question novel
findings when they
contradict prior findings that
have been successfully
replicated.
The burden of proof is on the
experimenter who claims
novel findings to explain this
discrepancy.
Why do we want to
generalize beyond the
laboratory?
We want to generalize
beyond the laboratory to
increase the external validity
of our findings.
What is the problem in
extending laboratory
findings?
Since extraneous variables
are uncontrolled in real
world setting and operate in
complex combinations, they
can modify the influence of
our individual variables.
What is the trade-off
between laboratory and
field experiments?
The trade-off is between the
laboratory’s more precise
control of extraneous
variables and the field
experiment’s greater realism
and external validity.
What did Hanson (1980) find
about the agreement
between laboratory and
field studies?
Hanson (1980) found that
more laboratory than field
studies reported a positive
correlation between
reported attitudes and
behavior.
When can we extend results
from an experiment to
everyday life?
We can’t confirm external
validity until additional
studies are completed in field
settings.
Researchers can increase and
verify the external validity of
laboratory findings using
aggregation, multivariate
designs, nonreactive
measurements, field
experiments, and naturalistic
observation.
What is aggregation? What
is its logic?
Aggregation is the grouping
together and averaging
of data to increase external
validity.
Combining the results of
experiments with different
subjects and methodologies
increases the generality and
external validity of our
findings.
How is this logic shared by
meta-analysis?
Meta-analysis uses statistical
analysis to combine and
quantify data from many
comparable experiments to
calculate an average effect
size.
How does aggregation
establish external validity?
Aggregation establishes
external validity by
combining the results of
experiments performed
using different subjects,
stimuli and/or situations,
trials or occasions, and
measures.
What is a multivariate
design?
A multivariate design studies
multiple DVs.
For example, a study of
repetitive strain places a
computer keyboard at
different distances from the
subject IV) and measures the
effect on three different
muscle groups (3 DVs).
What is the advantage of
multiverse design?
Multivariate designs allow us
to study the effect of an
independent variable on
combinations of dependent
variables.
These designs better
simulate the complexity of
the real world than
univariate designs and
provide more detailed
information.
How are multivariate
experiments analyzed?
We analyze multivariate
experiments with a
multivariate analysis of
variance (MANOVA).
How should researchers
handle a non-significant
outcome?
Accept the outcome, don’t
reframe your result as
“almost significant.”
Examine the experimental
procedures for design flaws.
If the design appears sound,
decide whether the
hypothesis was reasonable.
How should we handle the
possibility of faulty
procedures?
Check for possible causes of
a non-significant
outcome like:
1. confounding

2. extraneous variables that

increase within- subject’s
variability

3. weak manipulation of the

IV

4. inconsistent or flawed
procedures

5. ceiling and floor effects

6. insufficient power

How should we handle the

possibility of a faulty
hypothesis?

If previous studies supported

the hypothesis and ours did
not, look for differences in
experimental design or
sample.

If there was no previous

support and our design and
execution were good, we
may have to revise or discard
our hypothesis.

GOOOO YOURE SO GOOD AT

THIS! WE GOT THIS!!!! ACE
THE FINALS!!