Copyright © 2011. Taylor & Francis Group. All rights reserved.

210
Concept 7.3 Questions to ask about the internal validity of an experiment or a quasi-experiment, with possible L2
learning strategy research examples
Main question: To what degree is the study free of extraneous variables that could cloud the interpretation of
results? Subsidiary question: Are any potentially relevant variables not considered that should have been?
Column A Column B Column C
Name of possible Questions to ask about this possible Possible examples in L2 learning strategy

Created from vu on 2021-09-09 03:01:26.

internal validity threat threat
(Note: These threats can
also be combined.)
History threat Did any unusual or unexpected events
occur between the first and second
measurements that could have affected
the results?
Maturation threat Did any normal processes (e.g., growing
older, more fatigued, more overloaded, or
hungrier) occur for participants, unrelated
to any specific study events but potentially
influencing the results?
Reactivity threat Did the effect of taking the measurement
instrument (e.g., questionnaire or
experiments or quasi-experiments
A teacher’s strike occurred in the middle of
an experimental study on strategy instruction,
and that influenced the study’s results.
The control group classes were held at the
end of the day, when the students were tired,
but the treatment group classes were held
at 10 a.m., when students were feeling fresh
and alert.
Several students remembered their earlier
strategy answers and used the same answer

Oxford, R. L. (2011). Teaching and researching : Language learning strategies. ProQuest Ebook Central <a
interview) the first time affect the scores the second time.
the second time?
TEACHING AND RESEARCHING LANGUAGE LEARNING STRATEGIES

Instrumentation threat Was there any difference in how the
instrument (e.g., questionnaire or
interview) was calibrated or employed
between the first or second times it was
used? Were there changes in the
observers or the rating scales?
The first time the strategy questionnaire was
used during the study, the directions were not
clear enough, but the directions were much
clearer the second time. Results are therefore
not comparable between time 1 and time 2.

onclick=window.open('http://ebookcentral.proquest.com','_blank') href='http://ebookcentral.proquest.com' target='_blank' style='cursor: pointer;'>http://ebookcentral.proquest.co

 Copyright © 2011. Taylor & Francis Group. All rights reserved.
Regression to the mean
(a statistical fact) threat
Selection threat
Created from vu on 2021-09-09 03:01:26.
Attrition (also called research
mortality) threat
Researcher bias threat
Design implementation threat
Oxford, R. L. (2011). Teaching and researching : Language learning strategies. ProQuest Ebook Central <a
Procedural threat
Threat to comprehensiveness
Source: R. Oxford’s (2008) research methodology course notes, University of Maryland.
onclick=window.open('http://ebookcentral.proquest.com','_blank') href='http://ebookcentral.proquest.com' target='_blank' style='cursor: pointer;'>http://ebookcentral.proquest.co
Did the groups chosen for extremely high
scores or extremely low scores tend to
move toward the overall mean when they
took the instrument again?
Were groups selected in a way that would
affect the scores? Were the treatment and
control groups different from the outset?
Did members of one of the groups
drop out of the study more often than
members of the other group?
Was the researcher unconsciously
influenced by his or her beliefs or
hypotheses so that he or she unintentionally
biased the way the groups were treated?
Did the treatment group and the control
group realize they were being treated
differently?
Are the statistical procedures misapplied?
Are any potentially important variables
ignored or forgotten that might influence
the results?
The highest-proficiency students in the
strategy experiment tended to move toward
the proficiency mean when retested.
The better teacher’s classes were chosen
to be the control group, while the less
effective teacher’s classes were chosen to
be the treatment group.
For reasons unrelated to the strategy study,
three control group members dropped out of
the study, while none of the treatment group
participants dropped out.
The same researcher taught both the
experimental (strategy instruction) and
control groups but was far more enthusiastic
when teaching the treatment group.
The control group realized that the treatment
group was receiving strategy instruction, and
the control group started sharing strategy ideas
and competing against the treatment group.
The researcher used the wrong statistical
procedure for nominal (categorical) data in
one of the analyses.
The researcher did not consider gender or
socialization as a factor in the strategy results.
A GUIDE TO CONDUCTING RESEARCH ON SELF-REGULATED
211
 Copyright © 2011. Taylor & Francis Group. All rights reserved.

212
Concept 7.4 Questions to ask about the external validity of an experiment or a quasi-experiment, with possible L2
learning strategy research examples
Main question: To what degree are the results of the experiment (or quasi-experiment) generalizable to a
larger population or to different environments, conditions, persons, and times?
Column A Column B Column C

Created from vu on 2021-09-09 03:01:26.

Name of possible external
validity threat (Note:
These threats can also be
combined.)
Population validity threat
(threat to generalizability to
larger population)
Ecological generalizability:
treatment replicability threat
Questions to ask about this possible
threat
To what extent can the results be
generalized to the larger population?
(This relates to the question: Was the
sampling done in a principled way or
purely by convenience?)
To what extent can the results be
generalized if the treatment is unique to
the initial setting and cannot be
reproduced elsewhere?
Possible examples in L2 learning strategy
experiments or quasi-experiments
The strategy experiment used convenience
sampling only, so results are not generalizable
to the larger population of L2 learners.
The strategy experiment supported strategy
instruction because the school head was very
supportive and dropped by frequently, but a
different result might be found for schools
without such a school head.

Oxford, R. L. (2011). Teaching and researching : Language learning strategies. ProQuest Ebook Central <a
Ecological generalizability:
threat from multiple
treatment interference
To what extent can the results be Results of a single part of the strategy
generalized if the treatment actually instruction treatment, such as colour-coding
included several sub-treatments? instruction, might not be generalizable
TEACHING AND RESEARCHING LANGUAGE LEARNING STRATEGIES

without the other parts.

Ecological generalizability: To what extent can the results be generalized The excitement of learning new L2 strategies
threat from novelty effect if the treatment was in fact novel (new, might not generalize to other places where
exciting, refreshing) in the original study strategies are already discussed a lot.
but will not be novel in a later study?

onclick=window.open('http://ebookcentral.proquest.com','_blank') href='http://ebookcentral.proquest.com' target='_blank' style='cursor: pointer;'>http://ebookcentral.proquest.co

 Copyright © 2011. Taylor & Francis Group. All rights reserved.
Ecological generalizability:
threat from Hawthorne effect
(effect of being observed)
Created from vu on 2021-09-09 03:01:26.
Ecological generalizability:
threat from interaction
between treatment and
group characteristics
(called “treatment × group
interaction”)
Time durability/time effect
threat
Pretest or posttest sensitization To what extent can the results be
threat
Oxford, R. L. (2011). Teaching and researching : Language learning strategies. ProQuest Ebook Central <a
Dependent measure threat
Source: Typology in Columns A and B is adapted by R. Oxford from Martella, Nelson, and Marchand-Martella (1999). Column C (strategy
applications) is original.
onclick=window.open('http://ebookcentral.proquest.com','_blank') href='http://ebookcentral.proquest.com' target='_blank' style='cursor: pointer;'>http://ebookcentral.proquest.co
To what extent can the results be
generalized if the study participants
changed their behaviour simply because
of being observed?
To what extent can the results be
generalized if differential treatment
results occurred based on group
characteristics?
To what extent are the results durable
across time for the same group? To what
extent are the results generalizable to the
future for the same group?
generalized if a pretest only or a posttest
only is used?
To what extent can the results be
repeated if a different measure of the
dependent variable is used?
In the experiment, the control group (not
receiving strategy instruction) changed its
behaviour and paid more attention to learning
because of being observed. This would limit
generalization of the results.
Results of an L2 learning strategy experiment
might not generalize across different cultural
groups.
More talented L2 learners often learn
strategies very quickly, make them automatic
(habits), and no longer use them consciously
6 weeks later. Other learners might still use
them consciously.
Study results might differ if a proficiency
pretest were not used and if students were not
sensitized implicitly to a connection between
strategies and proficiency.
If the dependent variable in a strategy
instruction experiment (or other strategy
study) is frequency of strategy use, the results
might be different if a different strategy
questionnaire were used.
A GUIDE TO CONDUCTING RESEARCH ON SELF-REGULATED
213