Educational Measurement: Issues and Practice

Spring 2010, Vol. 29, No. 1, pp. 3–13

Does an Argument-Based Approach

to Validity Make a Difference?

Carol A. Chapelle, Iowa State University, Mary K. Enright, Educational Testing

Service, Joan Jamieson, Northern Arizona University

Drawing on experience between 2000 and 2007 in developing a validity argument for the
high-stakes Test of English as a Foreign LanguageTM (TOEFL R
), this paper evaluates the differences
between the argument-based approach to validity as presented by Kane (2006) and that described
in the 1999 AERA/APA/NCME Standards for Educational and Psychological Testing. Based on an
analysis of four points of comparison—framing the intended score interpretation, outlining the
essential research, structuring research results into a validity argument, and challenging the
validity argument—we conclude that an argument-based approach to validity introduces some new
and useful concepts and practices.

Keywords: claim, construct, high-stakes, inference, interpretive argument, Standards, TOEFL, validity argument

A ttempts to improve professional practice in validation

have resulted in seemingly new perspectives appearing
in the fourth edition of Educational Measurement. In the
introduction to the volume, Brennan (2006) contrasts the
presentation on validity given by Kane (2006) with that in
the previous edition: in 1989 the chapter was an extensive
scholarly treatment rather than one “that provides much
specific guidance to those who would undertake validation
studies” (p. 2). In contrast, the chapter in the 2006 edi-
tion “aims at making validation a more accessible enterprise
for educational measurement practitioners” (p. 3). Practi-
tioners reading the chapter will notice a change from what
appears in both the third edition of Educational Measure-
ment (Linn, 1989) and the 1999 AERA/APA/NCME Standards
for Educational and Psychological Testing. The introduction
of new concepts and frameworks in the 2006 chapter raises
the question of whether or not Kane really offers new insights
into validation, and achieves the aim of making validation
more accessible. We address this question in this paper by
drawing on our experience between 2000 and 2007 in de-
veloping a validity argument for the Test of English as a
Foreign LanguageTM (TOEFL R
), a test used for high-stakes
decisions about admissions to English-medium universities,
which requires substantial validity evidence supporting score
interpretation and use.
A revision of the TOEFL was undertaken by Educational
Testing Service (ETS) from 1990 through 2005. ETS has
standards for validity evidence that are in line with those
of the AERA/APA/NCME Standards for Educational and
Carol A. Chapelle is a Professor, Department of English, Iowa
State University, 203 Ross Hall, Ames, IA 50011; carolc@iastate.edu.
Mary K. Enright is Research Director, Center for Validity Research,
Educational Testing Service, Rosedale Road MS10-R, Princeton, NJ
08541. Joan Jamieson is a Professor, English Department, Northern
Arizona University, Box 6032, Flagstaff, AZ 86011.
Psychological Testing (1999), and therefore throughout the
TOEFL revision, research was conducted that would yield
the appropriate evidence for a validity argument in line with
the perspectives presented in the Standards. By the late
1990s, a considerable amount of such research needed to be
synthesized into a validity argument, and so we turned to
the Standards for guidance. This process yielded consider-
able insight into shortcomings in the guidance provided by
the Standards for synthesizing research results into a validity
argument. Because the project began with an attempt to use
the Standards and ended up instead drawing on the perspec-
tives of Kane (1992, 2001, 2002, 2006) and Kane, Crooks, and
Cohen (1999), it revealed contrasts in the two approaches
that may be informative in future attempts to update the
chapter on validity in the Standards. The validity argument
resulting from this process is available elsewhere (Chapelle,
Enright, & Jamieson, 2008), but this paper describes the con-
trasts as four points of comparison between the Standards
and Kane’s approach as summarized in Table 1.
Framing the Intended Score Interpretation
The Standards frame the intended interpretation of the test
as a construct which can be defined from many different
perspectives. Multiple perspectives were brought to bear on
the discussion of construct definition in the TOEFL project
as the developers attempted to define a construct of aca-
demic English language proficiency based on work in applied
linguistics. A strong construct theory—which would place
academic English proficiency within a nomological network
that could be used to generate clear hypotheses about test
performance relative to other constructs and behaviors—did
not result from this process and therefore the construct itself
was not a good basis for subsequent research. However, Kane’s

Copyright 
C 2010 by the National Council on Measurement in Education 3
Table 1. Key Aspects in the Process of Validation in the Standards (1999) and in Educational
Measurement (Kane, 2006)
Four Aspects Characterizing
Approaches to Validity Standards (1999) Kane (2006)
Framing the intended score interpretation A construct An interpretive argument
Outlining the essential research Propositions consistent with the Inferences and their assumptions
intended interpretation
Structuring research results into a validity Listing types of evidence Series of inferences linking grounds with
argument conclusions
Challenging the validity argument Counterevidence for propositions Refuting the argument

organizing concept of an “interpretive argument,” which does
not rely on a construct, proved to be useful.
A Construct
In the Standards, the term “construct” refers broadly to “the
concept or characteristic that a test is designed to measure”
(1999, p. 5) and “proposed interpretation” is used inter-
changeably with “construct” (e.g., 1999, p. 9). This approach
is consistent with views in educational measurement in the
early 1990s that a theoretical construct should provide the ba-
sis for score interpretation for a large-scale, high-stakes test
(e.g., Messick, 1994). Like experts in other areas, language
assessment experts have attempted to capture the dynamic
and context-mediated nature of the construct measured by
a test such as the TOEFL. Discussions of how to do so (e.g.,
Stansfield, 1986) ultimately led to the launch of a project
to develop a revised TOEFL that would be based on modern
conceptions of language proficiency. Designers of the new
TOEFL agreed that theoretical rationales underlying score
interpretation needed to come from a construct of language
proficiency and that therefore this would serve as a basis for
test design.
The idea that a language proficiency construct should un-
derlie test development and validation has strong support in
language testing. For example, Alderson, Clapham and Wall
(1995) tie the construct to both the test specifications and
validation. “For validation purposes, the test specifications
need to make the theoretical framework which underlies the
test explicit, and to spell out relationships among its con-
structs, as well as the relationship between the theory and
the purpose for which the test is designed” (p. 17). Bach-
man and Palmer (1996) include definition of the construct
in the fourth step of language test development: “ . . . a the-
oretical definition of the construct . . . provides the basis for
considering and investigating the construct validity of the
interpretations we make of test scores. This theoretical def-
inition also provides a basis for the development . . . of test
tasks” (p. 89). Weir (2005) ties the construct to validation:
“The more fully we are able to describe the construct we
are attempting to measure at the a priori stage the more
meaningful might be the statistical procedures contribut-
ing to construct validation that can subsequently be applied
to the results of the test . . . . We can never escape from the
need to define what is being measured, just as we are obliged
to investigate a test in operation” (p. 18).
Despite agreement on the need to define the construct as
a basis for test development, no agreement exists concerning
a single best way to define constructs of language proficiency
to serve as a defensible basis for score interpretation (e.g.,
Bachman, 1990; Bachman & Palmer, 1996; Chapelle, 1998;
Chalhoub-Deville, 1997, 2001; Oller, 1979; McNamara, 1996).
Nevertheless, most would agree that limiting a construct of
language proficiency to a trait such as knowledge of vocabu-
lary or listening is too narrow for the interpretations that test
users want to make for university admissions decisions. In-
stead, test users are typically interested in examinees’ ability
to use a complex of knowledge and processes to achieve par-
ticular goals. Therefore, strategies or processes of language
use have been included in constructs of language proficiency,
called communicative competence (Canale & Swain, 1980)
or communicative language ability (Bachman, 1990). Applied
linguists would also agree that language proficiency needs to
be defined in terms of contexts of performance because sec-
ond language learners can be proficient in some contexts
but lack proficiency in others (Cummins, 1983). A concep-
tualization of language proficiency that recognizes one trait
(or even a complex of abilities) as responsible for perfor-
mance across all contexts fails to account for the variation in
performance observed across these different contexts of lan-
guage use (Bachman & Palmer, 1996; Chalhoub-Deville, 1997;
Chapelle, 1998; McNamara, 1996; Norris, Brown, Hudson, &
Bonk, 2002; Skehan, 1998). As a result, language proficiency
constructs of interest are difficult to define in a precise way
(e.g., as construct representation; Embretson, 1983). Bach-
man (2007) provides an analysis of the issues and the various
perspectives on construct definition that have been presented
over the past fifty years in applied linguistics.
This historical perspective reveals the challenge the
TOEFL project faced in attempting to use a construct def-
inition developed by applied linguists as a basis of the va-
lidity argument. But this challenge is not unique to con-
structs in applied linguistics. Borsboom (2006) points out
that “psychological theories are often simply too vague to mo-
tivate psychometric models” (p. 437), and without a specified
psychometric model underlying score interpretation, what
kind of evidence should be sought for the validity argument?
When the construct underlying test score interpretation is so
complex, how can a validity argument be formulated? After
struggling with these questions, we welcomed the different
perspective offered by Kane for approaching the problem of
score interpretation.
An Interpretive Argument
Kane’s approach does not require a construct per se but rather
an explicitly stated interpretation called an interpretive ar-
gument (Kane, 1992, p. 527; 2001; Kane et al., 1999). He
describes the interpretive argument as consistent with the
general principles accepted for construct validity that appear
in the Standards: “Validation requires a clear statement of
the proposed interpretations and uses” (Kane, 2006, p. 23).

4 Educational Measurement: Issues and Practice


CLAIM: The student’s
English speaking abilities are
inadequate for study in an
English-medium university.
SO
GROUNDS: A student’s presentation
to the class on an assigned topic was
characterized by hesitations and
mispronunciations.
FIGURE 1. Illustration of an interpretive argument comprising an
inference about ability based upon grounds.
Rather than relying on formal theories, however, the inter-
pretive argument “specifies the proposed interpretations and
uses of test results by laying out the network of inferences
and assumptions leading from the observed performances
to the conclusions and decisions based on the performances”
(Kane, 2006, p. 23). “The validity argument provides an eval-
uation of the interpretive argument” (Kane, 2006, p. 23). In
the simplest terms, then, a validity argument is an interpre-
tive argument in which backing has been provided for the
assumptions. In view of the complexity of the construct the-
ory underlying the TOEFL interpretation, we were open to
exploring a means of establishing a basis for score interpre-
tation that did not rely solely on the concepts of tasks and
abilities from psychology or on applied linguistics’ discussion
of language ability constructs. Instead, the building blocks
of interpretive arguments are the types of inferences iden-
tified by Kane. Test developers and researchers wishing to
support score interpretation need to identify the inferences
upon which score interpretation is to be based.
To illustrate the meaning of inference, a simplified inter-
pretive argument for speaking performance in an English
language classroom is shown in Figure 1. It begins with
grounds such as the observation that a student’s presentation
to a class on an assigned topic was characterized by hesita-
tions and mispronunciations. “Grounds” is the term used by
Toulmin, Rieke and Janik (1984) to denote the basis for mak-
ing a claim; “data” was used by Toulmin (2003) and has been
used by others to refer to the same functional unit of the argu-
ment. The claim one might make on the basis of that perfor-
mance is that the student’s speaking abilities are inadequate
for study in an English-medium university. The arrow extend-
ing from the grounds to the claim represents an inference.
The inference allows for a conclusion, which, in the example,
is the claim. The point is that the observation itself cannot
mean that the student is unprepared. Instead, an interpretive
argument specifies the interpretation to be drawn from the
grounds to a claim by an inference. Such inferences play a
critical role in current approaches toward developing inter-
pretive and validity arguments (Kane, 1992, 2001; Mislevy,
Spring 2010
Steinberg, and Almond, 2003), which are based on Toulmin’s
(2003) description of informal or practical arguments, such
as those used in nonmathematical fields like law.
Interpretive arguments for test scores are constructed
through the use of particular types of inferences such as
the evaluation inference shown in Figure 2. In this example,
the grounds consist of the observations made of a student’s
speaking performance on a task in class. The evaluation in-
ference is made in awarding a score of “2” to the student for
that performance. The interpretive argument states explic-
itly that such an inference is the basis for awarding a score
of “2.” This statement about the inference provides the basis
for planning and interpreting validity evidence. Validity ev-
idence is needed to support such inferences, which, in this
example, could come from evidence showing the rationales
for developing the scoring rubric and the consistency with
which it was applied.
This simple, hypothetical example illustrates the basic ap-
proach to an interpretive argument that states the basis of
an interpretation without defining a construct. Rather than
terms referring to examinee characteristics, the tools of the
interpretive argument are inferences that are typically made
in the process of measurement. When we pursued this ap-
proach in the TOEFL project, we did not entirely eliminate
the construct but rather it became one part of an overall chain
of inferences. The set of inferences, which we were able to
specify in terms that had direct implications for research, be-
came the organizing units underlying score interpretation. If
test developers can work with a set of such inferences rather
than solely with the complex constructs that can be defined
in many different ways, the basis for score interpretation
becomes more manageable.
Outlining the Essential Research
In a general sense, the construct or interpretive argument
provides a starting point for planning research to be used in
a validation argument, but more specifically, within the Stan-
dards framework, how does one move from a construct defini-
tion of something like “reading comprehension” or “speaking
ability” to the design of a study whose results will show that
the scores should be interpreted as intended? The Standards
advise researchers to begin to consider validation research
by generating propositions that would be expected to be true
if test scores did in fact reflect the intended construct. In the
TOEFL project we attempted to follow this advice, ending up
with a list of propositions. Taking Kane’s approach, in con-
trast, propositions, called warrants, were also generated but
the important difference lies in how those propositions are
generated and what they are likely to consist of.
Propositions Consistent with the Intended Interpretation
The Standards direct test developers and researchers to
gather “kinds of evidence” that are needed to evaluate the
“intended interpretation” of test scores. This guidance, of
course, is recognizable as a way of summarizing the dominant
views on test validation throughout the 1990s that assumed
that multiple types of evidence should support score interpre-
tation (e.g., Cronbach, 1988; Messick, 1989). The Standards
point out “many lines of evidence can contribute to an under-
standing of the construct meaning of test scores” (p. 5) and
suggests that those lines of evidence can consist of the familiar
5
 Observed Score: The student’s spoken response received a score of 2.

Evaluation

Observations: When asked to discuss the relationship between information

presented in a brief lecture and a short reading passage a student replies:
“Oh,xxx he xxx the author xxx that alter energy xxx needed in United States and
the woman on the film he spoke about new kind of automobile. xxx It use fewer
gasoline.” Note: xxx represents unintelligible speech.

FIGURE 2. Example grounds and conclusion for an evaluation inference. (Adapted from Chapelle, Enright, and Jamieson, 2008, p. 11.
Reprinted courtesy of Taylor and Francis.)

categories of evidence based on test content, response pro-
cesses, internal structure, relations to other variables, and
consequences, as outlined by Messick (1989). Each of these
lines of evidence suggests methodologies, and there are plenty
of examples of these in language testing and elsewhere. How-
ever, as Shepard (1993) pointed out, the idea that “many lines
of evidence can contribute” offers a large set of options rather
than guidance to an efficient and effective path for validation.
According to the Standards, “The decision about what types
of evidence are important for validation in each instance can
be clarified by developing a set of propositions that support the
proposed interpretation for a particular purpose of testing”
(AERA/APA/NCME, 1999, p. 9). For example, if one wishes to
make the proposition that the test score distinguishes among
examinees at different English ability levels, the validation
research must provide data indicating that this is actually
the case. The proposition guides the researcher to produce
supporting evidence. These propositions are to serve as hy-
potheses about score interpretations, which would provide
guidance about the types of validity evidence required. In
the Standards, six examples of propositions are given for a
mathematics achievement test used to test readiness for an
advanced course. They include statements such as “that cer-
tain skills are prerequisite for the advanced course” and “that
test scores are not unduly influenced by ancillary variables”
(p. 9).
Drawing on these examples, we developed the following
propositions:
1. Certain language skills defined as listening, reading, speak-
ing and writing both independently and in combination are
necessary (but not sufficient) for students to succeed in
advanced academic settings.
2. The content domain of the tasks on the TOEFL requires
the English language skills students need to succeed in
English-speaking North American university settings.
3. Each of the skills—listening, reading, speaking, and
writing—is composed of a set of subskills.
4. Test tasks comprising each skill score exhibit internal con-
sistency.
5. Each of the four skills is distinct enough from each other
to be measured independently, but the skills are related by
some core competencies.
6. Test performance is not affected by test-taking processes
irrelevant to the constructs of interest.
7. Test scores are arrived at through judgments of appropriate
aspects of learners’ performance.
8. Test performance is not affected by examinees’ familiarity
with computer use.
9. Test performance is not affected inappropriately by back-
ground knowledge of the topics represented on the test.
10. The test assesses second language abilities independent of
general cognitive abilities.
11. Criterion measures can validly assess the linguistic aspects
of academic success.
12. Test scores are positively related to criterion measures of
success.
13. Use of the test will result in positive washback in ESL/EFL
instruction, such as increased emphasis on speaking and
writing and focus on academic language.
We stopped at number 13, recognizing that the list could
go on and on unless we gained a better sense of what a
proposition should consist of and how many one would like
to have for a good validation argument. Moreover, in the
absence of such guidelines we found that the propositions we
were generating were influenced by the validation research
that had been completed, and were therefore unlikely to help
identify areas where more research was needed. On the one
hand, contextualizing research was precisely what needed to
be done, but on the other hand this process seemed to start
from the perspective of completed research rather than from
the perspective of score meaning. In short, our examination
of the Standards and the materials that had led up to them
demonstrated the need for more explicit guidance on how
to formulate an intended interpretation and the propositions
that are supposed to point to the types of evidence that would
ultimately contribute to the TOEFL validity argument.
Inferences and Their Assumptions
Kane’s approach to identifying framing statements is to con-
nect them to the inferences in the interpretive argument
through the use of two types of statements: warrants and as-
sumptions. Taking the example in Figure 2, the evaluation
inference would be supported by a warrant such as “Observa-
tions of performance on the speaking task are evaluated to
provide a score reflective of the relevant language abilities.”
Here, the warrant is the generally held principle that hesita-
tions and mispronunciations are characteristics of students
with low levels of speaking ability who would have trouble
at an American university. Such a warrant is a statement
which rests on assumptions that need to be supported in
order for the inference to be made. A warrant is a law, a

6 Educational Measurement: Issues and Practice

Table 2. Summary of the Inferences, Warrants in the TOEFL Validity Argument with Their
Underlying Assumptions
Inference Warrant Licensing the Inference
Domain description Observations of performance on the TOEFL reveal
relevant knowledge, skills, and abilities in
situations representative of those in the target
domain of language use in the English-medium
institutions of higher education.
Evaluation Observations of performance on TOEFL tasks are
evaluated to provide observed scores reflective
of targeted language abilities.
Generalization Observed scores are estimates of expected scores
over the relevant parallel versions of tasks and
test forms and across raters.
Explanation Expected scores are attributed to a construct of
academic language proficiency.
Extrapolation The construct of academic language proficiency as
assessed by TOEFL accounts for the quality of
linguistic performance in English-medium
institutions of higher education.
Utilization Estimates of the quality of performance in the
English-medium institutions of higher education
obtained from the TOEFL are useful for making
decisions about admissions and appropriate
curricula for test takers.
Adapted from Chapelle, Enright, and Jamieson, 2008, pp. 19–21. Reprinted courtesy of Taylor and Francis.
generally held principle, rule of thumb, or established proce-
dure. Assumptions would be, for example, that the rubric for
scoring the responses was appropriate for providing the rel-
evant evidence of ability. Assumptions prompt research that
focuses on particular issues. In this case, the research would
need to provide evidence for the accuracy and relevance of
the rating of the student’s performance.
As shown in Table 2, we identified six inferences, each
with a warrant and assumptions, that form the basis for the
TOEFL interpretive argument. Each of the inferences is used
to move from grounds to a claim; each claim becomes grounds
for a subsequent claim. For example, a generalization in-
ference connects the grounds of an observed score which
reflects the relevant aspects of performance with a claim
Spring 2010
Assumptions Underlying Inferences
1. Critical English language skills, knowledge, and
processes needed for study in English-medium
colleges and universities can be identified.
2. Assessment tasks that require important skills and
are representative of the academic domain can be
simulated.
1. Rubrics for scoring responses are appropriate for
providing evidence of targeted language abilities.
2. Task administration conditions are appropriate for
providing evidence of targeted language abilities.
3. The statistical characteristics of items, measures,
and test forms are appropriate for
norm-referenced decisions.
1. A sufficient number of tasks are included on the
test to provide stable estimates of test takers’
performances.
2. Configuration of tasks on measures is appropriate
for intended interpretation.
3. Appropriate scaling and equating procedures for
test scores are used.
4. Task and test specifications are well defined so
that parallel tasks and test forms are created.
1. The linguistic knowledge, processes, and
strategies required to successfully complete tasks
vary across tasks in keeping with theoretical
expectations.
2. Task difficulty is systematically influenced by task
characteristics.
3. Performance on new test measures relates to
performance on other test-based measures of
language proficiency as expected theoretically.
4. The internal structure of the test scores is
consistent with a theoretical view of language
proficiency as a number of highly interrelated
components.
5. Test performance varies according to amount and
quality of experience in learning English.
Performance on the test is related to other criteria of
language proficiency in the academic context.
1. The meaning of test scores is clearly interpretable
by admissions officers, test takers, and teachers.
2. The test will have a positive influence on how
English is taught.
that the observed score reflects the expected score across
tasks, occasions and raters. Rather than state all of these
grounds and claims which are linked in a formulaic way to
types of inferences, Table 2 focuses on the warrants and as-
sumptions which need to be generated by the researcher to
guide the validity research. Discussion of the inferences as
the central building blocks for the interpretive argument ap-
pears in Kane et al. (1999), and the specific statements used
as grounds and claims in the TOEFL validity argument ap-
pear in Chapelle (2008). The intended score interpretation
is based on a domain description inference, which has a war-
rant that observations of performance on the TOEFL reveal
relevant knowledge, skills, and abilities in situations repre-
sentative of those in the target domain of language use in the
7
English-medium institutions of higher education. Kane does
not define domain description as an inference, but he points
out that “. . .if the test is intended to be interpreted as a mea-
sure of competence in some domain, then efforts to describe
the domain carefully and to develop items that reflect the do-
main (in terms of content, cognitive level, and freedom from
potential sources of systematic errors) tend to support the
intended interpretation” (Kane, 2004, p. 141). Because this
inference was critical to score interpretations for the TOEFL,
we included it in the TOEFL interpretive argument. The valid-
ity of that inference rests on the assumptions that assessment
tasks that are representative of the academic domain can be
identified, that critical English language skills, knowledge,
and processes needed for study in English-medium colleges
and universities can be identified, and that assessment tasks
that require important skills and are representative of the
academic domain can be developed.
The second inference in the argument is evaluation, which
has the warrant that observations of performance on TOEFL
tasks are evaluated to provide observed scores reflective of
targeted language abilities. The quality of evaluation infer-
ences rests on the assumption that “the criteria used to score
the performance are appropriate and have been applied as
intended and second, that the performance occurred under
conditions compatible with the intended score interpreta-
tion” (Kane et al., 1999, p. 9). In other words, if the test
administration conditions are not favorable or if they vary for
test takers, then the intended interpretation of a test taker’s
score may not be supported. The assumptions underlying this
inference are that rubrics for scoring responses are appropri-
ate for providing evidence of targeted language abilities, that
task administration conditions are appropriate for providing
evidence of targeted language abilities, and that the statis-
tical characteristics of items, measures, and test forms are
appropriate for norm-referenced decisions.
Third, the interpretive argument includes an inference of
generalization with the warrant that observed scores are es-
timates of expected scores over the relevant parallel versions
of tasks and test forms and across raters, as presented by
Kane et al. (1999, p. 10). In the example shown in Figure 2,
the expected score on similar test tasks would be a “2.” The
“2” across all tasks was not observed, but instead was in-
ferred on the basis of the single observed “2.” Of course, in
most test settings, one would not attempt to generalize on
the basis of one task. In the TOEFL interpretive argument,
generalization rests on the assumptions that (1) a sufficient
number of tasks are included on the test to provide stable
estimates of test takers’ performances, (2) the configuration
of the measures in terms of the tasks included is appropriate
for the intended interpretation, (3) appropriate scaling and
equating procedures for test scores are used, and (4) task
and test specifications are well defined so that parallel tasks
and test forms are created.
Fourth, the explanation inference in the TOEFL interpre-
tive argument has a warrant that the expected scores are
attributed to a construct of academic language proficiency.
Kane (2001) maintains that an interpretive argument can be
developed without including explanation, but suggests that
in cases where a construct is used in score interpretation,
as is often the case in language assessment (e.g., Bachman,
2002), it should be specified as an explanation inference. An
explanation inference is included in the interpretive argu-
ment so that assumptions associated with it can be identified
and investigated. The assumptions identified were that (1)
8 Educational Measurement: Issues and Practice
the linguistic knowledge, processes, and strategies required
to successfully complete tasks vary across tasks in keeping
with theoretical expectations; (2) task difficulty is systemat-
ically influenced by task characteristics; (3) performance on
new test measures relate to performance on other test-based
measures of language proficiency as expected theoretically;
(4) the internal structure of the test scores is consistent
with a theoretical view of language proficiency as a number
of highly interrelated components; and (5) test performance
varies according to the amount and quality of experience in
learning English.
Fifth, the interpretive argument includes an extrapolation
inference with the warrant that the construct of academic
language proficiency as assessed by TOEFL accounts for the
quality of linguistic performance in English-medium institu-
tions of higher education. This is consistent with the Kane
et al. (1999) description of extrapolation: the inference that
is made when the test takers’ expected scores are interpreted
as indicative of performance and scores that they would re-
ceive in the target setting. The assumption underlying this
inference in the TOEFL interpretive argument is that per-
formance on the test is related to other criteria of language
proficiency in the academic context.
Finally, utilization links the target score to the decisions
about examinees for which the score is used. This “decision-
making” is viewed by Kane as an inference, which underscores
the need for evidence to support it in an overall validity ar-
gument. Kane distinguishes the “descriptive interpretations”
(Kane, 2002, p. 32) of the other inferences from the policy-
based interpretations inherent in test score use. The latter
“involve assumptions supporting the decision procedure’s
suitability as a policy [e.g., requiring a TOEFL score as one
part of the application of international students at American
universities], and policies are typically justified by claims
about their consequences” (Kane, 2002, p. 32). Bachman
(2005) referred to decision-making as “utilization,” which
is the term we adopted in view of its grammatical parallelism
with the terms used for the other inferences. The utilization
inference in the TOEFL interpretive argument is made on the
basis of assumptions that the meaning of the test scores is
clearly interpretable by admissions officers, test takers, and
teachers, and that the test will have a positive influence on
how English is taught.
This interpretive argument with six inferences helped to
surmount the challenges presented by our attempt to gener-
ate relevant propositions as advised by the Standards. Based
on the six inferences shown in Table 2, we were able to
identify the assumptions in need of support in the TOEFL
validity argument. The assumptions in the third column of
Table 2 are similar in type and form to the propositions we
had brainstormed under the guidance of the Standards, but
the important difference is that each assumption is tied to
a particular inference. In other words, assumptions are de-
veloped in a more principled way as a piece of an inter-
pretive argument. In the initial brainstorming session of
propositions, we tended to look to the existing evidence,
and work backwards to formulate a proposition. Working
backwards from the existing evidence is not a good way of
ending up with a fair and balanced validity argument which
can identify areas where additional research is needed. The
number of possible assumptions associated with each in-
ference can vary, but the issues they should focus on are
constrained more productively than propositions guided
by construct theory alone. Moreover, a finite number of
inferences are typically made in measurement, and there-
fore the inferences and warrants that a researcher has to
work with have bounds.
Structuring Research Results into a Validity Argument
“I invite you to think of ‘validity argument’ rather than ‘val-
idation research’” (p. 4). These important framing words of
Cronbach (1988) over 20 years ago were used to highlight
the importance of the many audiences for which a validity
argument might be made. Audiences interested in, for exam-
ple, the functional, political, and explanatory dimensions of
tests place demands on some of what should be included in a
validity argument or how it might best be presented. However,
none of these audiences explicitly informs the internal logic
of the argument itself. In the 1999 Standards, researchers
are advised to develop a “sound validity argument,” but the
technical issue of how one actually puts together a validity
argument with all of the relevant pieces of research was not
developed, leaving researchers to produce lists rather than
arguments.
Listing Types of Evidence
The validity chapter in the Standards enumerates various
sources of validity evidence including those based on con-
tent, responses processes, internal structure, relations to
other variables, and consequences of testing. The Standards
specify that “a sound validity argument integrates various
strands of evidence into a coherent account of the degree
to which existing evidence and theory support the intended
interpretation of test scores for specific uses” (p. 17). How-
ever, the source of the coherence is not spelled out. Thus,
this approach identifies the types of research that would typ-
ically be done, but the validity evidence remains a listing of
research results, which researchers need to integrate in a nar-
rative, and which others need to evaluate in view of their own
experience.
Working within these perspectives, the TOEFL program
maintains a taxonomy consisting of types of research that
have been conducted (Educational Testing Service, 2007).
One could look at such a taxonomy and argue that strong
evidence supported various propositions that one might make
about the test. Such a list could also be used to identify areas
where more research is needed. However, a taxonomy is not
an argument, and in working with a taxonomy one is not
prompted to look at the strength of the evidence or to organize
it in a way that presents a validity argument.
Series of Inferences Linking Grounds with Conclusions
Kane’s approach depicts the validity argument as a step-by-
step movement across “bridges” (the metaphor referring to
inferences) to a conclusion about test score use. To extend
this bridge metaphor, one can think of a valid token being re-
quired to cross each bridge, and tokens being obtained when
adequate backing is provided for the assumptions associated
with each inference. The idea of a chain of inferences and
implications is consistent with Toulmin, Rieke, and Janik’s
(1984) conception of an argument, which is “liable to become
the starting point for a further argument; this second argu-
ment tends to become the starting point for a third argument,
and so on. In this way, arguments become connected together
in chains” (p. 73).
Spring 2010
Figure 3 shows the structure of the validity argument
for the TOEFL. It consists of the inferences summarized in
Table 2 as well as the backing for their assumptions. In the di-
agram, the inferences are used as short-hand for referring to
the inference and its warrant. For example, the evaluation in-
ference is made on the basis of the warrant that observations
of performance on TOEFL tasks are evaluated to provide ob-
served scores reflective of targeted language abilities. Such
a warrant is made on the basis of assumptions, including
that rubrics for scoring responses are appropriate for provid-
ing evidence of targeted language abilities. These warrants
and assumptions outlined in Table 2 prompt the selection of
backing that is shown in Figure 3. In Kane’s terminology, the
interpretive argument lays out the intended inferences and
warrants. The process of building a validity argument requires
the researcher to examine the assumptions underlying each
of the warrants in order to focus on obtaining the relevant
backing. Only inferences and backing are noted in the sum-
mary of the validity argument appearing in Figure 3, but these
are the result of the process of laying out inferences, war-
rants and assumptions, as well as gathering the appropriate
backing.
The diagram summarizing the TOEFL validity argument is
read starting at the bottom of the page with the grounds—
in this case the target language use domain, the academic
contexts, where the examinees will have to use English in
order to succeed. The backing required to move across the
domain description inference comes from a combination of
systematic domain analysis conducted by applied linguists,
empirical studies of stake holders’ views of important tasks
and academic language, and task modeling. These are summa-
rized in Chapelle et al. (2008), which refers to the documents
describing how applied linguists identified academic domain
tasks and language abilities required to complete them, re-
search that showed teachers and learners thought these tasks
were important, as well as the systematic process of task de-
sign and modeling engaged by experts. The conclusion, that
a relevant observation of examinee performance could be ob-
tained from the test tasks, serves as the grounds for the next
inference, evaluation.
The backing for the evaluation inference was obtained from
systematic rubric development, prototyping studies, and item
and test analysis, as described by Chapelle et al. (2008).
Rubrics and multiple task administration conditions were
developed, trialed, and revised based on expert consensus.
Statistical characteristics of tasks and measures were moni-
tored throughout the test development and modifications in
tasks and measures were made as needed. This process pro-
vided the backing needed to move to the conclusion that the
observed score is appropriate.
The observed score is the grounds for the generalization
inference, which was supported by backing consisting of gen-
eralizability and reliability studies as well as scaling and
equating studies. Because generalization is a way of concep-
tualizing reliability, in some cases assumptions underlying
generalization inferences can be supported through reliabil-
ity estimates. Other support of generalization comes from
standardization of task characteristics and test administra-
tion conditions (Kane et al., 1999) and from score equating
(Kane, 2004). In the TOEFL validity argument, results from
reliability and generalizability studies were used to determine
the number of tasks required. A variety of task configurations
was tried to find a stable configuration. Various rating sce-
narios were examined to maximize efficiency. An equating
9
 CONCLUSION: The test score reflects the ability of the test taker to use and understand
English as it is spoken, written, and heard in college and university settings. The score is useful
for aiding in admissions and placement decisions and for guiding English-language instruction.

1. Educational Testing Service has produced materials and held

test user information sessions.
2. Educational Testing Service has produced materials and held Utilization
information sessions to help test users set cut scores.
3. The first phases of a washback study have been completed.
Target Score

Results indicate positive relationships between test performance

and students’ academic placement, test takers’ self-assessments of
their own language proficiency, and instructors’ judgments of Extrapolation
students’ English language proficiency.

1. Examination of task completion processes and discourse Construct

supported the development of and justification for specific tasks.
2. Expected correlations were found among TOEFL measures and
other tests.
3. Correlations were found among measures within the test and Explanation
expected factor structure.
4. Results showed expected relationships with English learning.

Expected Score
1. Results from reliability and generalizability studies indicated the
number of tasks required.
2. A variety of task configurations was tried to find a stable
configuration. Generalization
3. Various rating scenarios were examined to maximize efficiency.
4. An equating method was identified for the listening and the
reading measures.
5. An ECD process yielded task shells for producing parallel tasks.
Observed Score
1. Rubrics were developed, trialed, and revised based on expert
consensus.
2. Multiple task administration conditions were developed, trialed,
and revised based on expert consensus. Evaluation
3. Statistical characteristics of tasks and measures were monitored
throughout the test development and modifications in tasks and
measures were made as needed.
Observation
1. Applied linguists identified academic domain tasks. Research
showed teachers and learners thought these tasks were important.
2. Applied linguists identified language abilities required for
academic tasks. Domain Description
3. A systematic process of task design and modeling was engaged
by experts.

GROUNDS: The
target language use
domain

FIGURE 3. Structure of the validity argument for the TOEFL.

method was identified for the listening and the reading mea-
sures. The speaking and writing measures are not equated
statistically due to technical and test security constraints.
Other methods of monitoring comparability of these test sec-
tions across forms are used (Educational Testing Service,
2008). In addition, a specific procedure was developed and
implemented for carrying out evidence-centered design (Mis-
levy et al., 2003) in a manner intended to produce parallel
tasks. These types of support led to the conclusion that the ob-
served scores reflect the expected scores across tasks, forms,
and raters.
Moving from this conclusion, the explanation inference,
which links the expected score to the construct of academic
English proficiency, is supported through several types of
studies typically associated with construct validation: studies
analyzing discourse and cognitive processes, task characteris-
tic and item difficulty studies, concurrent correlational stud-
ies, analysis of reliability and factor analysis, and comparison
studies of group differences. Examination of task completion
processes and discourse supported the development of and
justification for specific tasks. The correlations among TOEFL
measures and other tests found expected relationships with
the other measures. Correlations among measures within the
test and expected factor structure confirmed expectations
as well. Results showing expected relationships with English
learning were also found. In the TOEFL validity argument,
these results linked to the test construct were not considered
the whole validity argument, but rather were the backing for
one inference.
The extrapolation inference links the construct to the
target score which signifies examinee’s performance in
the academic contexts of interest. In the TOEFL validity

10 Educational Measurement: Issues and Practice

argument this inference is supported by criterion-related
validity studies. Results indicated positive relationships be-
tween test performance and students’ academic placement,
test takers’ self-assessments of their own language profi-
ciency, and instructors’ judgments of students’ English lan-
guage proficiency.
Finally, the utilization inference, which links the target
score to the conclusion that the score is useful for aiding in
admissions and placement decisions and for guiding English-
language instruction, is supported by backing that comes from
standard-setting studies, score interpretation materials, the
availability of instructional materials, and washback studies.
As a step toward the needed research, Educational Testing
Service has produced materials and held test user information
sessions about the test and about setting cut scores. The first
phase of a washback study (Wall & Horák, 2006) has been
completed to examine test effects of the former version of
the TOEFL as a point of comparison. The second phase (Wall
& Horák, 2008) monitored six teachers in five countries as
they learned about changes in the TOEFL and began to think
about how these might impact their teaching in the future.
In a report on Phases 3 and 4, Wall and Horák (submitted for
publication) analyze the impact of the new TOEFL on test
preparation materials and describe how three of the teachers
modified their teaching in response to changes in the TOEFL
test. More research is needed to provide additional backing
for this inference.
The structure and content of the TOEFL validity argument
is based on the interpretive argument that specifies the in-
tended test score interpretation and use. It includes both the
theoretical and empirical backing for the intended interpre-
tation and use. This presentation provides a framework for
synthesizing existing research conducted throughout the re-
vision and development and points to areas where additional
research efforts should be directed.
Challenging the Validity Argument
In developing the validity argument for the TOEFL, we focused
on the backing required to support intended interpretations.
However, part of the impetus for constructing such a validity
argument is that it provides sufficient clarity to create a space
for productive discussion of weakness in the argument. In
the Standards, the issue of challenging a validity argument
is described in terms of counterevidence for propositions,
whereas Kane’s approach refers to refuting the argument.
Counterevidence for Propositions
The Standards advise those who are conducting validation
research as follows: “Identifying the propositions implied by a
proposed test interpretation can be facilitated by considering
rival hypotheses that may challenge the proposed interpre-
tation” (p. 10). The Standards advise that a good way to
look for rival hypotheses is by considering the possibilities of
construct underrepresentation and construct irrelevant vari-
ance. Thus, the development of counterevidence is also tied
to the construct from this perspective. Moreover, it is not
clear how counterevidence should be interpreted relative to
positive evidence, or how one might systematically seek coun-
terevidence pertaining to particular aspects of the construct
definition.
Spring 2010
Refuting the Argument
Kane’s approach to the validity argument provides a place
for counterevidence. Figure 4 illustrates the counterevi-
dence as a rebuttal which is added to the argument shown
in Figure 1. In the example, the warrant that allows the
inference is that hesitations and mispronunciations are
characteristic of students with low levels of English speak-
ing abilities, and it is supported by the experience of an ESL
teacher with such students. The ESL teacher’s experience
is used in the interpretive argument as backing for the as-
sumption. The argument can also include the rebuttal, which
appears in the example as the fact that the topic required
highly technical and unfamiliar vocabulary. Such a fact could
weaken the argument by providing additional information
that would suggest that an inference about the student’s lack
of preparedness should not be made on the basis of a class-
room presentation on a particular topic.
Even when the warrant is supported with backing, excep-
tions may be relevant or other circumstances may undermine
the inference, thereby rebutting the force of the interpretive
argument. For example, the rebuttal in the example is that
the assigned topic for the oral presentation required the stu-
dent to use highly technical and unfamiliar vocabulary. This
rebuttal weakens the inferential link between the grounds—
that the oral presentation contained many hesitations and
mispronunciations—and the claim that the student’s speak-
ing ability was at a level that would not allow him to succeed at
an English-medium university. Such a rebuttal would pertain
to an important link in the argument for everyone perform-
ing the task. Rebuttals can also pertain to particular cases,
thereby delimiting the conditions under which the argument
would hold.
Development of the TOEFL validity argument thus far has
not yet reached the stage at which we can evaluate its utility
for discussion of either type of rebuttal. The validity argument
to date represents the first stage, what Briggs (2004) called
“design validity,” which necessarily reflects a confirmationist
approach. Our aim was to express the intended interpretation
and use of the TOEFL scores, as well as the relevant backing
as clearly as possible. This presentation is open to challenges
to “the appropriateness of the proposed goals and uses of the
testing program, the adequacy of the interpretive argument,
or the plausibility of the interpretive argument” (Kane, 2004,
p. 166). Time will tell whether future researchers will be able
to pick up the validity narrative and add to it with additional
backing or challenge it with rebuttals.
Did the Argument-Based Approach Make a Difference?
The timing and mission of the TOEFL project offered a unique
opportunity to compare the approach to validation argumen-
tation presented in the Standards with the more recent pre-
sentation in the fourth edition of Educational Measurement,
which had unfolded in papers by Kane over the past decade.
As a high-stakes test, the TOEFL required the most serious at-
tention in constructing a validity argument and therefore we
attempted to follow the guidelines provided in the Standards.
An understanding of those guidelines had proven useful in
codifying a shared understanding of what validity evidence
could consist of throughout test development. However, when
it came to putting that evidence together, the approach sug-
gested in the Standards lacked the structure required to
move forward with decisiveness. In particular, the reliance
11
 CLAIM: The student’s
English speaking abilities are
inadequate for study in an
English-medium university.

SO

REBUTTAL: The
UNLESS
topic required highly
technical and unfamiliar
vocabulary.
WARRANT:
Hesitations and SINCE
mispronunciations are
characteristic of students
with low levels of English
speaking abilities.

BECAUSE OF

BACKING: This is based

on teacher’s training and
previous experience.

GROUNDS: A student’s presentation

to the class on an assigned topic was
characterized by hesitations and
mispronunciations.

FIGURE 4. The structure for a validity argument including a rebuttal. (Adapted from Chapelle, Enright, and Jamieson, 2008, p. 7. Reprinted
courtesy of Taylor and Francis.)

on the test construct as a starting point in the Standards
was problematic in the TOEFL project, not because the test
was solely focused on performance without any interest in a
construct, but because the construct was extremely complex.
The argument-based approach to validity provided a place for
the construct, without relying on it as a starting point or the
central piece. The interpretive argument provides a means
of specifying the multiple inferences underlying score inter-
pretation and use, thereby removing the enormous burden
that might otherwise be placed on an imprecise theoretical
construct.
The discussion of theory and practice of validation in ed-
ucational measurement since at least the 1950s (Cronbach
& Meehl, 1955) has been concerned with establishing princi-
pled procedures for evaluating the test-based inferences and
uses. Discussions about how much and what kind of validity
evidence is needed to support inferences and uses of test
scores has been ongoing since that time as has validation
research in practice. Our attempt to put the discussion and
guidelines into practice revealed areas of tension between the
guidelines in the Standards and our specific case in practice
which was better addressed by the approach offered in Kane’s
article in the fourth edition of Educational Measurement.
In the case of the TOEFL, the argument-based approach to
validation represented a difference from that presented in
the Standards and a clear improvement as well. We hope that
this and other examples of validity arguments will be consid-
ered by the committee charged with the next revision of the
Standards.
References
AERA, APA, & NCME (1999). Standards for educational and psycho-
logical testing. Washington, DC: AERA.
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language testing
construction and evaluation. Cambridge: Cambridge University
Press.
Bachman, L. F. (1990). Fundamental considerations in language test-
ing. Oxford: Oxford University Press.
Bachman, L. F. (2002). Alternative interpretations of alternative assess-
ments: Some validity issues in educational performance assessments.
Educational Measurement: Issues and Practice, 21(3), 5–18.
Bachman, L. F. (2005). Building and supporting a case for test use.
Language Assessment Quarterly, 2(1), 1–34.
Bachman, L. F. (2007). What is the construct? The dialectic of abilities
and contexts in defining constructs in language assessment. In J. Fox,
M. Wesche, & D. Bayliss (Eds.), What are we measuring? Language
testing reconsidered (pp. 41–71). Ottawa: University of Ottawa Press.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice.
Oxford: Oxford University Press.
Borsboom, D. (2006). The attack of the psychometricians. Psychome-
trika, 71(3), 425–440.
Brennan, R. L. (Ed.) (2006). Perspectives on the evolution and
future of educational measurement. In R. Brennan (Ed.),

12 Educational Measurement: Issues and Practice

 Educational Measurement (4th ed., pp. 1–16). Westport, CT: Green-
wood Publishing.
Briggs, D. C. (2004). Comment: Making an argument for design va-
lidity before interpretive validity. Measurement: Interdisciplinary
Research and Perspectives, 2(3), 171–174.
Canale, M., & Swain, M. (1980). Theoretical bases of communicative
approaches to second language teaching and testing. Applied Lin-
guistics, 1, 1–47.
Chapelle, C. A. (1998). Construct definition and validity inquiry in
SLA research. In L. F. Bachman & A. D. Cohen (Eds.), Interfaces
between second language acquisition and language testing research
(pp. 32–70). Cambridge, UK: Cambridge University Press.
Chapelle, C. A. (2008). The TOEFL validity argument. In C. A. Chapelle,
M. K. Enright, & J. Jamieson (Eds.), Building a validity argument
for the Test of English as a Foreign Language (pp. 319–352). London:
Routledge.
Chapelle, C. A., Enright, M. K., & Jamieson, J. (Eds.) (2008). Building
a validity argument for the Test of English as a Foreign Language.
London: Routledge.
Chalhoub-Deville, M. (1997). Theoretical models, assessment frame-
works, and test construction. Language Testing, 14, 3–22.
Chalhoub-Deville, M. (2001). Task-based assessments: Characteristics
and validity evidence. In M. Bygate, P. Skehan, & M. Swain (Eds.),
Researching pedagogical tasks: Second language learning, teaching
and testing (pp. 210–228). Harlow, UK: Longman.
Cronbach, L. (1988). Five perspectives on validity argument. In H.
Wainer & H. Braun (Eds.), Test validity (pp. 3–17). Hillsdale, NJ:
Lawrence Erlbaum.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological
tests. Psychological Bulletin, 52, 281–302.
Cummins, J. P. (1983). Language proficiency and academic achieve-
ment. In J. Oller (Ed.), Issues in language testing research
(pp. 108–126). Rowley, MA: Newbury House.
Educational Testing Service (2007). TOEFL research: Ensuring
test quality. Retrieved August 13, 2008, from http://www.ets.org/
Media/Research/pdf/Framework Recent TOEFL Research.pdf
Educational Testing Service (2008). Reliability and comparability of
TOEFL R
iBT scores. Retrieved October 2, 2008, from http://www.
ets.org/Media/Tests/TOEFL/pdf/TOEFL_iBT_Reliability.pdf
Embretson, S. (1983). Construct validity: Construct representa-
tion versus nomothetic span. Psychological Bulletin, 93(1), 179–
197.
Kane, M. T. (1992). An argument-based approach to validity. Psycho-
logical Bulletin, 112, 527–535.
Kane, M. T. (2001). Current concerns in validity theory. Journal of
Educational Measurement, 38, 319–342.
Kane, M. T. (2002). Validating high-stakes testing programs. Educa-
tional Measurement: Issues and Practice, 21, 31–41.
Spring 2010
Kane, M. (2004). Certification testing as an illustration of argument-
based validation. Measurement, 2(3), 135–170.
Kane, M. (2006). Validation. In R. Brennan (Ed.), Educational Measure-
ment (4th ed.) (pp. 17–64). Westport, CT: Greenwood Publishing.
Kane, M., Crooks, T., & Cohen, A. (1999). Validating measures of per-
formance. Educational Measurement: Issues and Practice, 18(2),
5–17.
Linn, R. L. (Ed.) (1989). Educational measurement (3rd ed.). New
York: Macmillan.
McNamara, T. (1996). Measuring second language performance. Lon-
don: Longman.
Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measure-
ment (3rd ed., pp. 13–103). New York: Macmillan.
Messick, S. (1994). The interplay of evidence and consequences in
the validation of performance assessments. Educational Researcher,
23(2), 13–23.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure
of educational assessments. Measurement: Interdisciplinary Re-
search and Perspectives, 1, 3–62.
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee
abilities and task difficulty in task-based second language perfor-
mance assessment. Language Testing, 19, 395–418.
Oller, J. (1979). Language tests at school. London: Longman.
Shepard, L. A. (1993). Evaluating test validity. Review of Research in
Education, 19, 405–450.
Skehan, P. (1998). A cognitive approach to language learning. Oxford:
Oxford University Press.
Stansfield, C. W. (Ed.) (1986). Toward communicative competence
testing: Proceedings of the second TOEFL invitational conference.
TOEFL Research Reports, No. 21. Princeton, NJ: Educational Testing
Service.
Toulmin, S. E. (2003). The uses of argument (updated ed.). Cambridge,
UK: Cambridge University Press.
Toulmin, S. E., Rieke, R., & Janik, A. (1984). An introduction to reason-
ing. New York: Macmillan.
Wall, D., & Horák, T. (2006). The impact of changes in the TOEFL exam-
ination on teaching: Phase I, The baseline study. TOEFL-Monograph
Series Report No. 34. Princeton, NJ: Educational Testing Service.
Wall, D., & Horák T. (2008). The impact of changes in the TOEFL exam-
ination on teaching and learning in Central and Eastern Europe:
Phase 2, Coping with change. TOEFL iBT Report No. TOEFLiBT-05.
Princeton, NJ: Educational Testing Service.
Wall, D., & Horák, T. (submitted for publication) The impact of changes
in the TOEFL R
examination on teaching and learning in a sample
of countries in Europe: Phase 3, The role of the course book, and Phase
4, Describing change. Princeton, NJ: Educational Testing Service.
Weir, C. J. (2005). Language testing and validation: An evidence-based
approach. Basingstoke, UK: Palgrave Macmillan.
13