Educational Measurement: Issues and Practice

Winter 2020, Vol. 39, No. 4, pp. 30–40

Rubric Rating with MFRM versus Randomly Distributed

Comparative Judgment: A Comparison of Two Approaches
to Second-Language Writing Assessment

Maureen E. Sims, Troy L. Cox , Grant T. Eckstein, K. James Hartshorn,

Matthew P. Wilcox, and Judson M. Hart, Brigham Young University

The purpose of this study is to explore the reliability of a potentially more practical approach to
direct writing assessment in the context of ESL writing. Traditional rubric rating (RR) is a common
yet resource-intensive evaluation practice when performed reliably. This study compared the
traditional rubric model of ESL writing assessment and many-facet Rasch modeling (MFRM) to
comparative judgment (CJ), the new approach, which shows promising results in terms of
reliability. We employed two groups of raters—novice and experienced—and used essays that had
been previously double-rated, analyzed with MFRM, and selected with fit statistics. We compared the
results of the novice and experienced groups against the initial ratings using raw scores, MFRM,
and a modern form of CJ—randomly distributed comparative judgment (RDCJ). Results showed that
the CJ approach, though not appropriate for all contexts, can be as reliable as RR while showing
promise as a more practical approach. Additionally, CJ is easily transferable to novel assessment
tasks while still providing context-specific scores. Results from this study will not only inform
future studies but can help guide ESL programs in selecting a rating model best suited to their
specific needs.

Keywords: comparative judgment (CJ), ESL writing assessment practicality, ESL writing assessment reliability,
many-facet Rasch measurement model (MFRM), rubric rating

S coring reliability is a necessary condition in any validity

argument (Knoch & Chapelle, 2017). Writing assessment
scholars agree that direct measures such as essay tests are
better than discrete response tasks such as multiple-choice
tests when measuring writing ability (Greenberg, 1992; Huot,
1990; Yancey, 1999) because direct measures are more valid;
that is, evaluating a student’s actual writing can lead to a
better measurement of writing ability than a multiple-choice
exam about writing. Despite the simplicity of this argument,
assessment scholars are sensitive to the fact that discrete re-
sponse tasks are easier to score and produce much higher reli-
ability coefficients than scores emerging from rater-mediated
assessments (Eckes, 2011).
Direct writing tasks are prone to unreliable ratings due
to interpretations that can be made by individual raters
(Breland, 1983), thus reducing the credibility, generaliz-
ability, and validity of the assessment (Kane, 2013; Knoch &
Chapelle, 2017). To reduce divergent ratings, assessors use
holistic and analytic rubrics (Barkaoui, 2011; Huot, O’Neill,
& Moore, 2010), rater training (McNamara, 1996), and
algorithms like many-facet Rasch measurement (MFRM)
that adjust raw scores to account for rater biases (Eckes,
2011; Engelhard, 1992).
While these practices help increase the reliability of
direct writing assessment by focusing raters on what is being
measured (an issue of validity), each has unique drawbacks.
Rubrics and training, for instance, can be costly and resource-
intensive to develop and employ, yet do not statistically
control for rater bias. MFRM offers statistical adjustment of
rater bias, yet the calculations require appropriate software
and sufficient technical skill to generate and interpret the
modified scores. These conditions have led to the investi-
gation of a reemerging scoring approach to direct writing
assessment called Comparative Judgment (CJ; Fechner, 1965;
Laming, 2004; Pollitt, 2004; Thurstone, 1927). In this process,
raters repeatedly select the better of two essays within a given
pool that computer algorithms then organize from strongest
to weakest.
Researchers have advocated for wider adoption of CJ based
on its high reliability as measured in a range of assessment
domains across a number of studies (Bramley, 2007; Jones &
Inglis, 2015; Pollitt, 2012; Steedle & Ferrara; 2016). None of
these studies, however, examined CJ reliability when rating
texts composed by English as a second language (ESL) writ-
ers, whose language skills and organizational patterns may
make it difficult for raters to agree on relative essay quality.

30 
C 2020 by the National Council on Measurement in Education

Furthermore, CJ has not been adequately compared to tradi-
tional MFRM-adjusted scores to determine whether it has an
advantage over the more commonly accepted reliability pro-
cedure. Finally, it is unknown if CJ is reliable when performed
by minimally trained raters, though a number of studies in-
volving CJ have revealed high reliability results among both
trained and untrained raters in other domains (Heldinger &
Humphrey, 2013; Jones & Alcock, 2014; McMahon & Jones,
2015; Metzgar, 2016). Moreover, a comprehensive comparison
among rubric-rated and CJ-rated essays by both experienced
and inexperienced raters of ESL writing has not yet been con-
ducted. It is possible that highly reliable CJ scores will fail to
emerge when compared to rubric rating (RR) of ESL writing,
especially by untrained raters. Since resources to conduct
CJ are becoming more accessible and may be an attractive
approach to assessing ESL writing, CJ must be studied more
thoroughly.
This study’s purpose is to compare inferences that can
be made in a validity argument through examining reliability
measures of experienced and inexperienced raters among raw
rubric scores, MFRM-adjusted rubric scores, and CJ scores on
ESL writing. The emphasis on reliability is a necessary condi-
tion of this research, which takes as its theoretical framework
an assumption that reliability is a necessary (but not suffi-
cient) condition of validity (Kane, 2013; Knoch & Chapelle,
2017). Thus, we must first determine whether raters can pro-
duce consistent scores utilizing CJ on ESL writing before
examining claims about CJ’s validity.
Background
Writing assessors value reliability because low reliability is
evidence that raters lack consistency in measuring intended
writing objectives—a threat to validity. Increased reliability
reduces measurement error (Wiliam, 2001), which is an es-
sential consideration in high-stakes testing, where test results
have substantial consequences and must be valid (O’Neill,
2011; Wiliam, 2001). Yet, it is well established in the writing
community that productive tasks (e.g., essays) are a pre-
ferred method for validly assessing writing skill over discrete
response tasks (e.g., multiple choice; Greenberg, 1992; Huot,
1990; Yancey, 1999) and that a valid assessment is one in
which inferences made and actions taken are adequately
supported by theoretical and empirical evidence (Kane, 2013;
Messick, 1995). Such direct assessment approaches are dif-
ficult to rate reliably; Breland (1983) explained that “as far
back as 1880 it was recognized that the essay examination
was beset with the curse of unreliability” (p. 1). Rating incon-
sistency undermines the generalizability and credibility of as-
sessment, which calls into question the validity and decisions
made based on its results (Kane, 2013; Knoch & Chapelle,
2017).
Validity Framework
Concerned with investigating the validity of the rating pro-
cess of CJ, we have framed the findings of this study in terms
congruent with Knoch and Chapelle’s (2017) argument-based
framework for rating process validation. Argument-based ap-
proaches have been used successfully throughout the assess-
ment literature, and their benefits stem from delineating
“principled connections . . . in an argument that demon-
strates the logic for making interpretations . . . of test takers’
performance” (p. 4). The framework developed by Knoch and
Winter 2020
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Chapelle is comprehensive and incorporates six inferences,
each with warrants backed by assumptions that are, in turn,
supported by evidence. These are: evaluation inference, gen-
eralization inference, explanation inference, extrapolation
inference, decision inference, and consequences inference.
It is beyond the scope of this article to address all six in-
ferences as they are rigorous and comprehensive; however,
as “reliability is taken to be a necessary but not sufficient
condition for validity,” and “the validity of a test with re-
spect to any criterion cannot exceed the index of reliability”
(Kane, 2013, p. 29), we, therefore, focus our efforts on ad-
dressing evaluation inference as it relates directly to rating
reliability.
Evaluation inference incorporates two important war-
rants: rating scale properties and rater reliability. These two
warrants, in turn, involve a total of 13 combined assump-
tions that, if supported by evidence, meet the criteria for
the evaluation inference in validating the rating process.
Warrant 1 asserts that the scale functions as intended by the
development team and consists of the following assumptions:
(1) the scale criteria are well-defined, (2) the steps are
adequate for distinguishing different levels of performance
within the scale, and (3) the scale can spread examinees
into different levels. Warrant 2 asserts that the raters rate
reliably and includes the assumptions that the raters: (4)
can identify different levels of performance among different
scores, (5) can apply the scale consistently, (6) are comfort-
able and confident using the scale, (7) are well-trained, (8)
have adequate support documentation, and (9) are qualified.
In addition, assumption 10 posits that rating sessions are de-
signed to optimize rater performance, and assumptions 11–13
presuppose the existence of evidence that rater bias is not
systematically introducing construct irrelevant variance into
the scores through rater characteristics, use of sub-scales, or
rating of tasks. For each of these, Knoch and Chapelle (2017)
listed types of evidence needed to support the assumptions
and thereby the warrants. They further argued that the use
of MFRM can provide strong evidence for a number of these
assumptions. For example, MFRM provides scale diagnostic
data regarding rater use of the scale in terms of both consis-
tency and consensus (see Knoch & Chappelle, 2017, for a full
description of the framework).
Increasing Reliability
Reliability, which contributes to a case for the valid inter-
pretations of a score, measures the consistency of rater judg-
ments and is reported on a scale of 0 (significant error) to
1 (no error). Scores of .80 and above are considered highly
reliable and acceptable for high-stakes testing. In order to
produce reliable essay scores, writing assessors have im-
plemented numerous measures including creating rubrics,
training raters on the rubrics, and utilizing software to com-
pensate for rater error and bias (Knoch & Chappelle, 2017).
Rubrics provide descriptions of expectations of quality, and
raters assign examinees to locations on the scale to rep-
resent their rating in relation to the traits being assessed
(Myford & Wolfe, 2003). Holistic scoring rubrics include rat-
ing scale levels with descriptors for each level (Huot, O’Neill,
& Moore, 2010). Analytic rubrics go one step further and in-
clude descriptors for varying criteria that can be measured
individually and aggregated to reach a final score (Barkaoui,
2011).

C 2020 by the National Council on Measurement in Education 31

While rubrics can act as a “regulatory device” (Jonsson &
Svingby, 2007, p. 136), they do not guarantee high reliability
(Eckes, 2011). Raters, notwithstanding extensive training,
often internalize and apply the same rubric differently
(Eckes, 2011; McNamara, 1996; Myford & Wolfe, 2003)
and don’t always utilize the rubric as intended (Winke &
Lim, 2015). To counter this problem, raters are trained
and normed to have essentially the same mental picture
of each rubric descriptor (Wolfe, 2005). Although training
and norming on rubrics has demonstrated a positive effect
on intrarater reliability (Eckes, 2011; McNamara, 1996),
rater bias can still prove to be disproportionately influential
(Wilson & Case, 1997). In response, sophisticated statistical
measures, including MFRM, have been employed in the
last three decades to compensate for rater effects (Eckes,
2011; Engelhard, 1992). MFRM is widely accepted as a
robust statistical mechanism that adjusts for rater effects
and identifies outlying judges or examinees, resulting in a
modified “fair average score” that represents a more accurate
assessment.
Unlike other models in which item discrimination is esti-
mated freely on an item-by-item basis, the Rasch model scales
the discrimination index for all items to 1. This restriction
allows the model to place both item difficulty and person abil-
ity on the same equal-interval log-odds (logit) scale. MFRM
extends this model by permitting the addition of facets, or
sources of variability. In the present study, we are primar-
ily interested in rater self-consistency in judging examinee
ability; however, other facets such as task difficulty, rating
occasion, etc., can be added. MFRM models all these facets
jointly and analyzes the pattern of examinee responses and
rater scores in the form of fit statistics that help detect anoma-
lous behavior. MFRM, therefore, is an effective solution for
greater reliability in direct writing assessment, but it can be
resource-intensive and potentially cost-prohibitive. CJ sug-
gests similar reliability through the use of built-in statistical
modeling and is offered at no cost by a proprietary system
available through the No More Marking website: nomore-
marking.com. Since this system is widely accessible and the
underlying processes are proprietary, it needs to be carefully
scrutinized.
Comparative Judgment
Comparative Judgment (CJ) was first proposed by Thurstone
(1927) in the early 1900s and is based on the theory that
humans are innately predisposed to successful comparisons
but less apt when attempting absolute judgments in isolation
(Fechner, 1965; Laming, 2004). Although the theory has been
well researched in the field of psychology, CJ has only recently
been used in education (Pollitt, 2004).
We can conceptualize CJ by way of analogy in which a sports
league ranks teams from best to worst. CJ would have every
team compete against every other team over the course of
a season to determine that ranking. When applied to direct
writing assessment, raters compare two essays side by side,
choose which is better, then move to the next set. Comparison
data is aggregated and eventually rank-ordered on a scale
representing the relative distance between each essay.
Proponents argue that the main advantage of CJ is its
ability to estimate the subjective distance of objects that can-
not otherwise be objectively arranged (Vasquez-Espinosa &
Conners, 1982). Pollitt, an advocate for its use in educational
32 
C 2020 by the National Council on Measurement in Education
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
assessment, claims: “Probably the most immediate advantage
of [CJ] is in making reliable the assessment of skills that are
currently problematic” (2012, p. 292). This characteristic can
be particularly useful in the assessment of essays that reflect
actual writing ability but can be difficult to assess reliably
(Huot, 1990; O’Neill, 2011; Yancey, 1999). Jones and Inglis
(2015) illustrated this potential through a multistage study
wherein they investigated the flexibility of CJ in scoring pro-
ductive tasks. Experienced test designers first developed a
productive math assessment of higher order problem-solving
skills for which experts then created what amounted to a
16-page rubric. After administering the exam to 750 students,
it was marked traditionally by four experienced markers and
judged using CJ by 20 math experts. Traditional marking re-
sulted in high interrater reliability, r = .91, and a strong
correlation with predicted General Certificate of Secondary
Education (GCSE) grades, r = .73. CJ interrater reliability
was r = .86, yet correlated more closely with GCSE predicted
grades (r = .76). The same task resulted in comparable reli-
ability and validity evidence using CJ without the need for an
extensive rubric, exemplifying the potential advantage of CJ
over RR in productive assessment design and rating. Steedle
and Ferrara (2016) further reported on multiple CJ stud-
ies in which all interrater reliability indicators were greater
than .73, and most were greater than .93. Such high levels of
reliability may result from the method itself: the forced com-
parisons inherent in CJ can mitigate rater bias, and problems
typical in writing assessment, such as central and extreme
tendencies or lenience and strictness, are moderated by the
very nature of the method (Bramley, 2007; Pollitt, 2012).
A number of CJ studies have further shown high inter-
rater reliability scores among trained and untrained raters
(Heldsinger & Humphrey, 2013; Jones & Alcock, 2014; Jones
& Inglis, 2015; Jones, Swan, & Pollitt, 2015; McMahon & Jones,
2015; Metzgar, 2016). Pollitt (2012) indicated untrained
raters competently assessed writing quality and achieved ab-
solute reliability (an equivalent of Cronbach’s α) estimates of
.96 of the objects rated. Heldsinger and Humphry (2013) em-
ployed a form of CJ involving calibrated exemplars in which
teacher ratings were compared with those of trained raters;
they reported high interrater reliability for the teachers (.92)
and a strong correlation between trained rater and teacher
ratings (.90). Therefore, the amount of training required to
perform CJ rating may be significantly less than that for rubric
rating (Jones & Wheadon, 2015).
Although promising, CJ is not without drawbacks. It is
cumbersome when applied to lists, easily measured outcomes,
or longer tasks (McMahon & Jones, 2015). Additionally, it
does not provide a feedback mechanism and is therefore
unsuited for use in certain pedagogical contexts. The lack of
rubric also removes the instrument by which a consensus of
quality expectations is communicated (Brown, 2012). While
rubric-rated benchmarks can be used as anchors in score
translations, this means CJ is not completely independent
of RR. There may also be a potential for raters to use proxy
surface features, such as word count, spelling, or essay length
when making judgments. These potential shortcomings can
conceivably enervate the arguments of both the scale and
rater warrants of the validation framework.
Traditional CJ exhibits a further drawback in terms of
practicality. Studies involving traditional CJ suggest raters
need to make anywhere from 25 to 50 judgments per essay to
assure quality results (Steedle & Ferrara, 2016; Whitehouse &
Educational Measurement: Issues and Practice

Pollitt, 2012), as each essay needs to be compared with almost
every other essay in order to accurately assign its place in the
rank order (Vasquez-Espinosa & Conners, 1982).
Adaptive comparative judgment (ACJ) can conceiv-
ably increase reliability and practicality simultaneously by
leveraging initial Swiss rounds1 to fuel an algorithm that
generates the most informative pairings (Whitehouse &
Pollitt, 2012). ACJ is a system whereby raters make inferences
concerning an individual essay’s ranking within a group of es-
says based on a subset of known judgments between multiple
linked pairs. In this approach, to further the sports league
analogy used earlier, each team only plays a subset of all the
teams in the league but is linked to all other teams through
the pairings (i.e., team A never plays team C, but team B
plays both team A and team C, thus teams A and C are linked
through pairings with team B). Similarly, ACJ works with
essay rating in such a way that each essay is compared to a
subset of other essays in the group; in each match, one essay
is judged to be better than the other and provides additional
information that eventually results in the ranking of the es-
says from best to worst. The algorithms generating ACJ data
are based on the Rasch logistic model (Andrich, 1978) and
the Bradley–Terry–Luce model (Turner & Firth, 2012). Using
these algorithms, which follow similar statistical models to
those underlying MFRM analyses, the predictive nature of ACJ
minimizes the number of pairs required to complete the scale
while maintaining reliability scores above .80 in most cases
(Heldsinger & Humphry, 2013; Pollitt, 2012; Steedle & Fer-
rara, 2016; Whitehouse & Pollitt, 2012). This is an acceptable
level for high-stakes testing (Wiliam, 2001). Even in its adap-
tive form, the number of pairings is greater with ACJ than RR.
However, proponents of CJ suggest that the binary nature of
the decisions made within the CJ model reduce the cognitive
load required to perform evaluations (Goffin & Olson, 2011),
thus making rating decisions not only faster but easier as well.
Critics question the efficacy of the adaptive nature of ACJ,
however, arguing that preemptive pairings may lead to po-
tential overinflation of ACJ reliability data (Bramley, 2015;
Bramley & Wheadon, 2015). In acknowledgement, nomore-
marking.com generated a more robust version with randomly
distributed pairings (C. Wheadon, personal communication,
August 5, 2017). This version, which we refer to as randomly
distributed comparative judgment (RDCJ), also reduces the
number of required judgments; however, unlike ACJ, which
narrows the pairing possibilities as the algorithm advances,
RDCJ maintains random pairings throughout the judging pro-
cess, conceivably providing a more equitable judging frame-
work for comparisons (Wheadon, 2015). These randomly dis-
tributed pairs may be less likely to overinflate reliability.
RDCJ is made available through nomoremarking.com. It is
important to note, however, that there is a clear trade-off in
the use of a proprietary system. While the software is free
and easy to use, the calculations are opaque and preclude
open scrutiny. Nevertheless, as RDCJ is a readily available
resource currently being used extensively in the educational
marketplace by novice and experienced raters alike, it is this
version of CJ we have chosen to investigate.
Moreover, whereas RDCJ shows potential as a reliable and
practical approach to direct writing assessment, the broad
applicability of CJ is still unexplored. For instance, while
studies of CJ suggest that rater training is not necessary for
achieving acceptable levels of reliability (greater than .80),
this claim has yet to be explored in the linguistically complex
Winter 2020
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
context of ESL writing assessment. Research so far has looked
primarily at native English contexts, in areas ranging from
peer assessment (Jones & Wheadon, 2015) to mathematical
problem-solving (Jones, Swan, & Pollitt, 2015). Thus, there
is a need for further investigation into the reliability of RDCJ
and its applicability in relatively new contexts.
Furthermore, there is an increasing need for practical ap-
proaches to L2 writing assessment. Reports estimate the num-
ber of English-language learners will exceed two billion by
the year 2020 (Beare, 2017). Many of these learners will be
required to take direct writing assessments such as those
in the Test of English as a Foreign Language (TOEFL) to
demonstrate their language proficiency. Given the potential
emergence of RDCJ as an ESL rating method, determining
the extent to which it meets the criteria of an assessment
validity argument is valuable. Thus, this study assesses the
reliability of RDCJ relative to the reliability of rubric rating
with MFRM when applied to ESL writing using trained and
untrained raters. The primary research aim is to examine how
novice and experienced raters compare in terms of reliability
when utilizing traditional RR, MFRM, and RDCJ in an ESL
setting.
Method
For this study, we used essays that had been double-rated with
a rubric, analyzed with MFRM, and selected with fit statistics.
Novice and experienced raters scored congruous sets of these
essays using RDCJ and RR with MFRM (see Figure 1).
Sample Essays
We collected a stratified sample of 60 essays from a pool
of 30-minute ESL placement essays in an intensive English
program (IEP) that had previously been rubric-rated by a
group of experienced and rubric-normed IEP teachers. We
made efforts to select essays evenly from each of the rubric
levels and to representatively sample the language groups
involved in the testing (see Table 1). To control for prompt
effect (Evans, Hartshorn, Martin, & Cox, 2014), we collected
essays from four prior rating sessions that used the same essay
prompt and were further divided into two congruous sets of
37 and 38 essays, with 12 essays in common.
The strata are based on the rubric levels of 0 (little to
no language or a reliance on simple, memorized words and
phrases) to 7 (university-level writing). The holistic rubric
included three criterion categories (text type, content, and
accuracy) that were combined to produce a single rating. The
rubric used for all rubric-rating sessions associated with this
study has been consistently utilized by the IEP over several
years with satisfactory results in terms of scale properties.
We selected essays that typified models of each of the seven
levels being tested. For rubric levels 1–5, all essays had been
rated with 100% rater agreement based on observed scores and
MFRM infit scores between 0 and 1.36 (2–13 raters per essay)
and represented a full-level difference between each rating
level. Only two essays had a rating of 0, so both were included
in the study. Additionally, the small number of available level
6 essays resulted in less rigorous selection standards and,
consequently, a little less than a full-level difference in some
instances. Fair average scores ranged from 5.62 to 6.67, and
infit scores exceeded 1.5 on three of the essays (1.58, 2.35,
and 3.22). Level 6 rater agreement, however, remained 100%

C 2020 by the National Council on Measurement in Education 33

Note. ANOVA run to test for effects on rating time, and Spearman’s rho used to correlate between MFRM adjusted fair average, the study
rubric rating fair averages, and RDCJ true scores to show evidence of accuracy. Pearson correlation run to compare surface features with
MFRM fair averages and RDCJ true scores.
FIGURE 1. Study design to compare traditional rubric rating (RR) to many-facet Rasch modeling (MFRM) and randomly distributed
comparative judgment (RDCJ).
with two raters per essay. Level 7 was not included because
none of the available essays had been rated a 7.
Raters
Two groups of evaluators (novice and experienced) rated
congruous sets of the essays using RDCJ and RR with MFRM.
The novice and experienced raters were further divided
into two groups in order to measure method and order
for both levels of rater experience; each group, A and B,
was composed of four novice raters and four experienced
raters. Although Knoch and Chapelle’s (2017) evaluation
inference requires that raters are thoroughly and regularly
trained, we included novice, untrained raters in order to
investigate training effect. Eight undergraduates minoring
in the Teaching English to Speakers of Other Languages
(TESOL) program at the university where this research was
conducted formed the novice group.
The experienced raters met the training and qualification
requirement and exhibited confidence using the scale
(Knoch & Chapelle, 2017). This group comprised eight
individuals who self-selected from a pool of trained raters
34 
C 2020 by the National Council on Measurement in Education
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
currently working or having previously worked at the same
IEP. They had from 2 to 7 years’ experience rating congruent
placement essays within the same context while using the
rubric chosen for this study.
Rating Methods
Most MFRM rubric-rating systems use incomplete yet linked
rating schedules (Eckes, 2011) where at least two raters
judge each essay. For the rubric-rating portion of this study,
we employed a fully crossed design in which each essay set
was rated by all of the raters within the rating group to which
it was assigned. Further, a portion of the essays (20%) were
included in both Essay Set 1 and Essay Set 2 so that they were
rated by all the raters employing both methods. The design of
the CJ portion of the study was also conservative. RDCJ has
demonstrated acceptable levels of reliability, .80 or higher,
with a minimum of nine judgments for each essay (Jones &
Wheadon, 2015). Full essay sets were assigned to each of four
distinct groups: Group A Novice, Group A Experienced, Group
B Novice, and Group B Experienced for RDCJ rating. For each
set of 37 or 38 essays, raters were assigned an average of 100
judgments each, or approximately 11 judgments per essay.
Educational Measurement: Issues and Practice
 17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Table 1. Essay Levels and Language the accuracy of RR scoring alone and could only be conducted
Background in a fully crossed design in which every rater rated every essay,
Essay rating levels hence only the 12 essays included in both essay sets were used.
We further grouped raters by experience (see Table 2).
Languages 0 1 2 3 4 5 6
The major observation from this analysis is that adjacent
Arabic – – – – – – 1 agreement was high among experienced raters and low among
Chinese – 1 2 – – – – novice. For instance, one of the level 3 essays received six level
French – – – – 1 1 – 2 ratings and two level 3 ratings from experienced raters,
Japanese – – 1 1 – – – while three novice raters gave it a level 1, four a level 2,
Korean – 3 1 1 – – 1 zero a level 3, and one a level 4. Globally, experienced raters
Mongolian – 1 – 1 – – –
Portuguese – – 1 1 1 2 2
exhibited 100% adjacent agreement across all essays in con-
Russian – – – – 1 – – trast with the novice raters achieving 100% adjacent agree-
Spanish 2 3 5 5 5 6 6 ment for four essays. Moreover, experienced raters displayed
Thai – – – – 2 1 – exact agreement above 63% in seven instances while novice
Turkish – 1 – – – – – rater exact agreement did not exceed 38% except in one
case and equaled 0% in two cases. If RR without MFRM ad-
Totals 2 9 10 9 10 10 10
justments were used to assign essay scores, novice raters
would present obvious threats to reliability, which corrobo-
Process rates Weigle’s (1999) findings that novice raters exhibited
greater severity on graph essays than experienced raters, a
To control for order effect, Group A was assigned to RR and difference that disappeared after training.
Group B used RDCJ during the first rating session using Essay Despite the advantage of training, both groups, without
Set 1. They then switched rating modes for the following ses- MFRM, call into question the reliability of using a single rater
sion one week later using Essay Set 2. Both groups received (Kane, 2013). The low percentages of exact agreement of
video rating instructions and were told to complete the rat- experienced and novice rating groups, 53% and 28%, respec-
ings independently in one session per rating method. For RR, tively, suggest student placement based on this data would
scores were entered into a computer-based system. For RDCJ, be accurate at best only about half the time with experienced
raters used an online proprietary system called nomoremark- raters. Such variance does little to assure assessment quality
ing.com. Raters were instructed to log out of each system and deliver fair results to examinees as McNamara (1996)
during breaks. During orientation, raters completed three observed: “Raw scores . . . are no reliable guide to candidate
RR practice ratings and two RDCJ practice comparisons. RR ability” (p. 118).
raters used a paper copy of the rubric to rate the essay pre-
sented on the screen. RDCJ raters were asked to compare Evidence on the Scale, Rater Consistency,
two essays, displayed side-by-side on the computer screen, and Examinee Reliability
and “choose the better response.” Upon completion of the
practice sessions, the raters were instructed to begin rating. We chose to distinguish between rater reliability and exam-
To examine raw RR without MFRM, the ratings of the inee reliability, though both kinds of reliability measure the
12 essays that were judged by all 16 raters were compared reproducibility of results. Rater reliability reflects an ability to
to the original MFRM ratings. We used the software program consistently distinguish between levels of a trait. Raters can
FACETS to run MFRM on the RR sessions from which we de- be trained to be more self-consistent in their interpretation
rived measure scores, fair averages, fit statistics, and separa- and implementation of a rating scale as manifested through
tion reliability results for both examinees and raters (Linacre, fit statistics; however, it is often difficult to achieve perfect
2017). RDCJ data analysis was completed by the proprietary unity among raters in terms of absolute agreement, and thus
system operating from nomoremarking.com, which provided if the raters fit the model (are self-consistent), rater sever-
reliability scores, fit statistics, and true scores. ity differences can be modelled mathematically. Examinee
In examining the scoring accuracy of the newly rated reliability reflects differences in the level of the trait the per-
data, we used Spearman’s rank-order correlation coefficient son possesses. MFRM was conducted using FACETS (Linacre,
(Spearman’s rho) to identify the level of association between 2018) in which a four-facet rating scale model was specified,
(1) the original fair average and the new MFRM as well as including: Examinee, Rater, Occasion (First, Second), and
(2) the original fair average and the RDCJ true score (see Rater Group (Novice, Experienced). The 16 raters assigned
Table 5). To investigate the relationship between essay length 802 RR scores to the essays from the 60 examinees and made
and assigned ratings, we ran a Pearson’s correlation, compar- 1,609 RDCJ judgments through nomoremarking.com.
ing word count with MFRM fair average scores and RDCJ true
scores. Further, we compared the initial MFRM-adjusted rat- Rater Reliability. The MFRM results in Table 3 indicate that
ings with the raw scores assigned in this study to assess the ratings progressed monotonically without any mis-ordering of
impact of MFRM on study data (see Table 2). We calculated categories or thresholds. However, several indices point to is-
statistical analyses using SPSS R
version 25.0. sues with category 7. For instance, the count is extremely low
compared to the other groups: only 12 essays were given a
Results rating of 7 (coincidentally, none were given this score by the
original raters). Further, the outfit mean square of 2.1 (which
RR Raw Scores has an expected value of 1) and the relatively large discrep-
The initial analysis compared the raw RR scores of 12 essays ancy between the expected measure and the average measure
rated by all participants in this study against the original indicate a high degree of unpredictability in how raters as-
MFRM-adjusted ratings. This analysis was meant to reveal signed a score of 7. Finally, results from the MFRM analysis

Winter 2020 
C 2020 by the National Council on Measurement in Education 35
 17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Table 2. Comparison of Initial and Study RR Raw Scores
Experienced raters Novice raters

Adjacent agreement

Adjacent agreement
Original rating level

Exact agreement

Exact agreement
(original rating)

(original rating)

(original rating)

(original rating)
Range

Range
Essay

0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
0 35 2 6 2 25% 100% 6 2 2 75% 88%
1 28 2 5 1 3 63% 100% 7 1 2 13% 100%
32 6 2 2 75% 100% 3 3 2 3 38% 100%
2 27 5 3 2 63% 100% 6 1 1 3 13% 100%
31 5 3 2 63% 100% 5 3 2 38% 100%
3 36 6 2 2 25% 100% 3 4 1 3 0% 63%
4 34 1 6 1 3 75% 100% 3 5 2 0% 63%
37 6 2 2 75% 100% 2 3 3 3 38% 75%
5 30 5 2 1 3 25% 100% 1 3 2 1 1 5 13% 50%
33 3 3 2 3 38% 100% 3 1 1 3 4 38% 50%
6 26 1 7 2 88% 100% 1 2 3 2 4 38% 88%
29 5 2 1 3 25% 100% 1 3 3 1 4 38% 88%
2.42 53% 100% 3.08 28% 80%

= Original rating

Table 3. Rating Scale Effectiveness

Quality control Rasch–Andrich thresholds
Response Ave. Exp. Outfit
category Count % measure measure MNSQ Measure SE
0 62 8% –8.18 –8.04 .9 – –
1 126 16% –5.38 –5.31 .7 –7.37 .20
2 156 19% –2.80 –2.78 .8 –4.21 .15
3 137 17% -.63 -.68 .8 –1.60 .14
4 146 18% 1.42 1.29 .8 .28 .14
5 100 12% 2.78 2.70 1.3 2.41 .14
6 63 8% 4.15 4.15 1.1 3.84 .18
7 12 1% 4.65 5.74 2.1 6.65 .35
MNSQ = mean squared; SE = standard error.

estimate a rater strata of 6.94, which indicates that raters are
able to distinguish approximately seven distinct levels of per-
formance rather than the eight response categories available
to them.
Likely due to the binary decisions inherent in RDCJ, the
company that provides RDCJ does not supply information
regarding scale quality. This was a nonissue for this study
as all the rating samples were rated by both methods. How-
ever, it could inhibit researchers in addressing the assertions
that the scale functions as intended. Nevertheless, if trained
raters were to agree on rubric-rated exemplars to include
in an RDCJ rating session, a post hoc analysis could provide
information relative to the effectiveness of the RDCJ rating
scale.
Infit measures were used to provide insight into the con-
sistency of each of the raters when performing both RR with
MFRM and RDCJ. These measures, infit and outfit, are sensi-
tive to inliers and outliers, respectively, and have an expected
value of 1. Linacre (2018) noted that values between .5 and
1.5 are “productive for measurement” (2018, p. 272), and
further argued that values between 1.5 and 2.0, while not pro-
ductive, do not degrade the measurement. For the RR with
MFRM, none of the raters had mean-square estimates above
2.0; however, four of the novice raters had infit, outfit, or both
mean square estimates between 1.5 and 2.0 (see Table 4).
RDCJ infit values less than 1 provide evidence of overall rater
consistency, infit values ranging from 1 to 1.3 indicate some
rating inconsistency, and infit values greater than 1.3 indicate
rating inconsistency (No More Marking Ltd, 2019). Fifteen of
the raters had fit statistics less than 1; however, one novice
rater exhibited inconsistent rating with an infit of 1.46 (see
Table 4). Thus, the scale for RR with MFRM functioned pri-
marily as intended. Furthermore, there is evidence for both
RR and RDCJ that the raters can apply the scale consistently,
especially among experienced raters.
Examinee Reliability. Examinee reliability was measured
in terms of how reliably MFRM and RDCJ differentiated
between the essays. This study reports a common reliability
estimate calculated for both MFRM and RDCJ analyses:
examinee separation reliability. This type of reliability
is analogous to the more well-known Cronbach’s α and
indicates the extent to which the scores of a measure are
consistent, or the degree to which they provide reproducible
results. Reliabilities reported, therefore, are based on
examinee separation: examinee separation reliability for

36 
C 2020 by the National Council on Measurement in Education Educational Measurement: Issues and Practice
 17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Table 4. Rater Infit Data for MFRM and RDCJ Table 6. Word Count Correlations According
MFRM RDCJ to Background and Rating Mode
Infit Outfit Infit Pearson’s r
Experience Rater MNSQ MNSQ Statistic Group Background Mode N Word count
Experienced 1 .78 .72 .67 A Novice RR 36 .79
Experienced 2 .75 .73 .60 RDCJ 38 .90
Experienced 3 .65 .74 .70 Experienced RR 36 .83
Experienced 4 .53 .50 .61 RDCJ 38 .89
Experienced 5 .66 .74 .76 B Novice RR 37 .89
Experienced 6 .72 .72 .74 RDCJ 37 .82
Experienced 7 .51 .53 .52 Experienced RR 37 .89
Experienced 8 1.00 .88 .65 RDCJ 37 .80
Novice 9 1.35 1.33 .82
Novice 10 .76 .82 .56 RR = rubric rating; RDCJ = randomly distributed comparative
judgment.
Novice 11 1.86 1.60 .94
Novice 12 1.89 1.98 1.46
Novice 13 1.51 1.19 .51
Novice 14 .52 .49 .40 raters in Group B, the reliability of both groups exceeded
Novice 15 1.90 1.46 .53 industry standards for high-stakes testing. Thus, novice and
Novice 16 1.00 .96 .67 experienced raters were able to produce highly reliable re-
MNSQ = mean squared. sults when employing the fully crossed RR with MFRM or
Bolded values represent fit statistics outside the accepted range of .5 to RDCJ.
1.5.
Evidence of Concurrent Validity. Evidence of concurrent
Table 5. Reliability and Accuracy Indicators validity was primarily derived from a Spearman’s rho corre-
Reliability Original lation between (1) the initial MFRM-adjusted fair average
Examinee rating scores and the study MFRM fair average scores as well as (2)
Group Experience Mode n separation rho the initial MFRM-adjusted fair average scores and the RDCJ
A Novice RR 36 .96 .94
true scores. The RDCJ true score is analogous to the MFRM
RDCJ 38 .91 .90 fair average; in other words, the true score is an objective
Experienced RR 36 .98 .95 measurement of an examinee’s latent ability. The correla-
RDCJ 38 .92 .94 tions for both methods and each rating group are presented
B Novice RR 37 .96 .96 in Table 5. Spearman’s rank correlation coefficients involv-
RDCJ 37 .89 .92 ing the original MFRM fair average score were significant,
Experienced RR 37 .96 .94 ranging from .90 to .96, p < .001.
RDCJ 37 .94 .94 The Spearman’s rank-order correlation coefficient pro-
RR = rubric rating; RDCJ = randomly distributed comparative vides correlational indicators of data quality. Results echo
judgment. that of reliability as again no significant difference is evident
between novice and experienced raters; between 92% and 96%
of scores generated by participating raters, novice and expe-
RR and an analogous RDCJ reliability indicator (Wheadon, rienced, can be explained by the original writing placement
personal communication, August 5, 2017). score. Novice RR demonstrated slightly higher correlations
Although reliability evidence may not, of itself, speak to with the original scores than novice RDCJ, whereas expe-
the quality of the data, it provides important information rienced RR and RDCJ correlations with the original scores
relative to the consistency of the data collected and the sub- were essentially the same. Further, in all but one instance
sequent potential for drawing valid conclusions. In addition (Group B Novice raters), RDCJ correlations with the original
to mixed-rater group analyses, we calculated the examinee MFRM fair average were slightly lower than RR correlations.
separation reliability estimates for each rating group—Group However, in every case the data was highly correlated—rs
A Novice, Group A Experienced, Group B Novice, and Group >.85, p < .01—suggesting that both novice and experienced
B Experienced—revealing that within each of the aforemen- raters, utilizing either method, achieved similar results in
tioned homogenous groups, examinees judged by raters of terms of final scores: MFRM fair averages and RDCJ true
similar experience had reproducible relative locations on a scores.
measure scale for the same essays. A comparison of exam- We addressed the potential for surface features to intro-
inee separation reliability among all rating groups for both duce construct-irrelevant variance through investigating the
methods is presented in Table 5. To be acceptable for high- possibility that essay length acted as a proxy for writing
stakes assessments, reliability should be above .80 (Nunnally quality. Using a Pearson’s r, we compared word counts with
& Bernstein, 1994). We found both rating methods (MFRM RR measure scores and RDCJ true scores. Results (see Ta-
and RDCJ) and rater backgrounds (novice and experienced) ble 6) indicated statistically significant correlations between
exceeded this standard. Examinee separation reliability mea- word count and both RR and RDCJ scores. Nevertheless, it
sures ranged from .89 to .98 and tended to be slightly higher is important to note that high correlations between essay
for rubric rating and experienced raters. Close reliability in- length and rater score are also common throughout the lit-
dicators within each method for Group A show no significant erature in more traditional rating contexts (Barkaoui, 2011;
difference in rating background. And although experienced Kobrin, Deng, & Shaw, 2011; Wolfe, Song, & Jiao, 2016). In-
raters demonstrated slightly greater reliability than novice terestingly, word count exhibited higher correlations with

Winter 2020 
C 2020 by the National Council on Measurement in Education 37

rating order than rating method; word count correlations
were higher with the second rating session (Group B for
RR; Group A for RDCJ) regardless of rating mode or rater
background.
The lack of distinct variance between the two rating meth-
ods, RR with MFRM and RDCJ, provides evidence that RDCJ is
not uniquely prone to this potential for construct irrelevance.
Yet the high correlation with word count for both methods is
indicative of a strong connection. However, high word count
correlations with scores may not necessarily equate to the
scoring of nonconstruct-relevant indicators. In all writing,
but perhaps especially so in an ESL setting, the amount
of language the examinee can produce, particularly under
time constraints, is a clear indicator of language and writing
fluency.
Raw score rating disparity between experienced and novice
raters supports what others have found: training minimizes
rater effects (Eckes, 2011; McNamara, 1996). However, the
remaining gap between experienced rater raw scores and ini-
tial MFRM-adjusted ratings ratings points to the need for sta-
tistical procedures that account for rater bias. MFRM, under
heightened coverage conditions of eight raters per examinee,
essentially eliminated the distinction between the two groups,
demonstrating the strength of the model in accounting for
rater effect. Under typical double-rating conditions with ex-
perienced raters, MFRM has consistently demonstrated its
ability to successfully model out rater variance (Eckes, 2011;
Myford & Wolfe, 2003). However, under similar double-rating
conditions, the distinction between novice and experienced
raters would likely be much greater; it is unlikely that MFRM
would be able to model out all of the rater variance in novice
raters. Interestingly, RDCJ is not subject to the same limita-
tions and demonstrated comparable reliability for both novice
and experienced raters.
Limitations
While this study provides important information relative to
necessary conditions for thoroughly assessing the validity of
an assessment approach, it nevertheless is not broad enough
to encompass all essential aspects of argument-based validity
theory. As such, it is but one step in a larger sequence of stages
necessary for establishing the appropriateness of a particular
assessment approach in any given context.
Additionally, despite the irregularities in the novice rat-
ings, the MFRM analysis produced highly reliable results.
It is likely, however, that the atypical fully crossed rubric-
rating model, as well as the carefully selected stratified es-
says utilized in the study, overinflated the RR reliability in-
dicators to some degree. A modified rating schedule with
more typical rating samples may achieve different results.
Further, this study investigated RDCJ in a new but poten-
tially high-use context; however, in light of the concerns
raised by Bramley (2015) regarding the reliability of ACJ,
further research is needed. It is possible that claims made
regarding the greater reliability of RDCJ are unfounded. This
potential for unreliability and the undisclosed underlying al-
gorithms of RDCJ within nomoremarking.com introduce pos-
sible sources of inaccuracy that could weaken the validity
argument.
Finally, while the highly significant correlations between
the original MFRM fair average scores and both the MFRM
38 
C 2020 by the National Council on Measurement in Education
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
and RDCJ study-generated scores indicate they are likely as-
sessing similar traits, it is not clear what those traits are. The
word count correlations of this study provide some evidence;
however, further analysis involving construct relevant com-
ponents such as T-units would provide important information
relative to the construct validity of RDCJ.
Conclusions
Several important conclusions can be drawn from this study,
not the least of which is that the resulting evidence indicates
the consistency and potential generalizability of the data,
thus furthering the investigation of CJ as a viable alterna-
tive to RR in terms of reliability. Additional steps need to be
taken, however, within the argument-based validity frame-
work before adopting CJ as a primary scoring method. As
per the Knoch and Chapelle (2017) framework, these steps
include, among other things, an investigation of the various
potential interpretations of the data, a look at context-based
decisions that can be made, and a review of the resulting
consequences.
Further, while training was shown to increase intrarater
reliability in RR, it was still inadequate. MFRM under typi-
cal double-rating conditions, though adequate for use with
experienced raters, would also likely be unequal to the task
of modeling out all novice rater variations. In contrast, un-
der typical conditions, RDCJ produced analogous results re-
gardless of rater background and proved appropriate for use
in an ESL direct writing assessment context in terms of
reliability.
Similar reliability indicators between experienced raters
employing MFRM and experienced or novice raters using
RDCJ spotlights practicality as a point of discrimination.
While not the focus of this article, we did record the time
it took to rate using each method. The time per essay to
rate using RDCJ was just under 52 seconds faster than
RR. A Mann–Whitney U test indicated that rating using RR
(Mdn = 63.0) took significantly longer than RDCJ (Mdn =
11.3), U = 293081.500, p < .001. The difference in rating
time and the additional time and resources required for rater
training suggest that RDCJ might be a more practical rat-
ing method. However, more research is needed as part of a
validity argument to make this claim.
Finally, whereas RR with MFRM automatically incorporates
the rating scale in data reports, RDCJ true score conversion
requires the inclusion of examinee essays already rated us-
ing RR to act as anchors against which the relationship of
new examinees can be measured. Should RDCJ prove to be
a valid assessment approach, perhaps a marriage of the two
methods would capitalize on the strengths of each: smaller
numbers of highly trained raters could identify benchmarks
to anchor RDCJ rankings while larger numbers of less expe-
rienced raters would then perform RDCJ on the bulk of the
examinees, delivering quality results that are both reliable
and practical.
Note
1
Swiss rounds are typically used when the number of competitors makes
the inclusion of all potential pairings infeasible. After an initial random
round, subsequent rounds pair according to wins and losses and match
pairs with similar scores.
Educational Measurement: Issues and Practice

References
Andrich, D. (1978). Relationships between the Thurstone and Rasch
approaches to item scaling. Applied Psychological Measurement,
2(3), 449–460.
Barkaoui, K. (2011). Effects of marking method and rater experience on
ESL essay scores and rater performance. Assessment in Education:
Principles, Policy & Practice, 18(3), 279–293.
Beare, K. (2017, May). How many people learn English? ThoughtCo. Re-
trieved August 4, 2017, from https://www.thoughtco.com/how-many-
people-learn-english-globally-1210367
Bramley, T. (2007). Paired comparison methods. In P. Newton, J. Baird,
H. Goldstein, H. Patrick, & P. Tymms (Eds.), Techniques for moni-
toring the comparability of examination standards (pp. 246–294).
London: Qualifications and Curriculum Authority.
Bramley, T. (2015). Investigating the reliability of adaptive comparative
judgment. Cambridge Assessment Research Report. Cambridge, UK:
Cambridge Assessment.
Bramley, T., & Wheadon, C. (2015, November). The reliability of
adaptive comparative judgment. AEA-Europe Annual Conference.
Cambridge, UK: Cambridge Assessment. Retrieved from http://www.
cambridgeassessment.org.uk/Images/296241-the-reliability-of-
adaptive-comparative-judgment.pdf, (accessed March 26, 2020).
Breland, H. M. (1983). The direct assessment of writing skill: A mea-
surement review (College Board Report No. 83.6; ETS RR No. 83.82).
Brown, J. (2012). Developing, using, and analyzing rubrics in lan-
guage assessment with case studies in Asian and Pacific languages.
Honolulu, HI: National Foreign Language Resource Center.
Eckes, T. (2011). Introduction to many-facet Rasch measurement:
Analyzing and evaluating rater-mediated assessments. Frankfurt,
Germany: Lang.
Engelhard, G., Jr. (1992). The measurement of writing ability with
a many-faceted Rasch model. Applied Measurement in Education,
5(3), 171–191.
Evans, N. W., Hartshorn, K. J., Martin, T., & Cox, T. (2014). Measuring
written linguistic accuracy with weighted clause ratios: A question
of validity. Journal of Second Language Writing, 24, 33–50.
Fechner, G. T. (1965). Elements of psychophysics. In R. J. Herrnstein,
& E. G. Boring, (Eds.), A source book in the history of psychology
(pp. 66–75). Cambridge, MA: Harvard University Press.
Goffin, R. D., & Olson, J. M. (2011). Is it all relative? Comparative
judgments and the possible improvement of self-ratings and ratings
of others. Perspectives on Psychological Science, 6(1), 48–60.
Greenberg, K. L. (1992). Validity and reliability issues in the direct
assessment of writing. Writing Program Administration, 16(1–2),
7–22.
Heldsinger, S. A., & Humphry, S. M. (2013). Using calibrated exemplars
in the teacher-assessment of writing: An empirical study. Educa-
tional Research, 55(3), 219–235.
Huot, B. (1990). Reliability, validity, and holistic scoring: What we know
and what we need to know. College Composition and Communica-
tion, 41(2), 201–213.
Huot, B., O’Neill, P., & Moore, C. (2010). A useable past for writing
assessment. College English, 72(5), 495–517.
Jones, I., & Alcock, L. (2014). Peer assessment without assessment
criteria. Studies in Higher Education, 39(10), 1774–1787.
Jones, I., & Inglis, M. (2015). The problem of assessing problem solving:
Can comparative judgement help? Educational Studies in Mathe-
matics, 89(3), 337–355.
Jones, I., Swan, M., & Pollitt, A. (2015). Assessing mathematical prob-
lem solving using comparative judgement. International Journal of
Science and Mathematics Education, 13(1), 151–177.
Jones, I., & Wheadon, C. (2015). Peer assessment using comparative and
absolute judgement. Studies in Educational Evaluation, 47, 93–101.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Relia-
bility, validity and educational consequences. Educational Research
Review, 2, 130–144.
Kane, M. T. (2013). Validating the interpretations and uses of test
scores. Journal of Educational Measurement, 50(1), 1–73.
Winter 2020
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Knoch, U., & Chapelle, C. A. (2017). Validation of rating processes
within an argument-based framework. Language Testing, https://
doi.org/10.1177/0265532217710049
Kobrin, J. L., Deng, H., & Shaw, E. J. (2011). The association be-
tween SAT prompt characteristics, response features, and essay
scores. Assessing Writing, 16(3), 154–169. https://doi.org/10.1016/
j.asw.2011.01.001.
Laming, D. (2004). Human judgment: The eye of the beholder. London,
UK: Thomson.
Linacre, J. M. (2017). FACETS computer program for many-facet
Rasch measurement (Version 3.80.0) [Software]. Beaverton, OR:
Winsteps.com
Linacre, J. M. (2018). FACETS computer program for many-facet Rasch
measurement (Version 3.81.0). Beaverton, OR: Winsteps.com
Metzgar, M. (2016). Using adaptive comparative judgment to assess stu-
dent work in an MBA course. International Journal for Infonomics,
9(3), 1217–1219.
McMahon, S., & Jones, I. (2015). A comparative judgement approach to
teacher assessment. Assessment in Education: Principles, Policy &
Practice, 3, 368–389.
McNamara, T. F. (1996). Measuring second language performance.
New York: Longman.
Messick, S. (1995). Validity of psychological assessment: Validation of
inferences from person’s responses and performances as scientific
inquiry into score meaning. American Psychologist, 50(9), 741–749.
Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater
effects using many-facet Rasch measurement: Part I. Journal of
Applied Measurement, 4(4), 386–422.
No More Marking Ltd. (2019). No More Marking. Retrieved
November 15, 2018, from https://www.nomoremarking.com/judges?
tab=MonitorJudges
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd
ed.). New York: McGraw-Hill.
O’Neill, P. (2011). Reframing reliability for writing assessment. The
Journal of Writing Assessment, 4(1), n.p.
Pollitt, A. (2004). Let’s stop marking exams. Paper presented at the
meeting of the IAEA Conference, Philadelphia. Retrieved from
http://www.cambridgeassessment.org.uk/Images/109719-let-s-stop-
marking-exams.pdf, (accessed March 26, 2020).
Pollitt, A. (2012). The method of adaptive comparative judgement.
Assessment in Education: Principles, Policy & Practice, 19(3), 281–
300.
Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgment as
an approach to essay scoring. Applied Measurement in Education,
29(3), 211–223.
Thurstone, L. L. (1927). A law of comparative judgment. Psychological
Review, 34(4), 273–286.
Turner, H., & Firth, D. (2012). Bradley-Terry models in R: The
BradleyTerry2 package. Journal of Statistical Software, 48(9), 1–
21.
Vasquez-Espinosa, R. E., & Conners, R. W. (1982). The law of
comparative judgment: Theory and implementation (RSIP TR
403.82). Baton Rouge, LA: Louisiana State University. Retrieved
from http://www.dtic.mil/dtic/tr/fulltext/u2/a136169.pdf, (accessed
March 26, 2020).
Weigle, S. C. (1999). Investigating rater/prompt interactions in writ-
ing assessment: Quantitative and qualitative approaches. Assessing
Writing, 6(2), 145–178.
Wheadon, C. (2015, February 10). The opposite of adaptivity?
[Web log post]. Retrieved August 7, 2017 from https://blog.
nomoremarking.com/the-opposite-of-adaptivity-c26771d21d50
Whitehouse, C., & Pollitt, A. (2012). Using adaptive comparative judge-
ment to obtain a highly reliable rank order in summative assess-
ment. Manchester, UK: Centre for Education Research and Polic.
Wiliam, D. (2001). Reliability, validity, and all that jazz. Education 3–13,
29(3), 17–21.
Wilson, M., & Case, H. (1997). An examination of variation in rater
severity over time: A study in rater drift (Berkeley Evaluation and
Assessment Research Center report). UC Berkley.

C 2020 by the National Council on Measurement in Education 39

Winke, P., & Lim, H. (2015). ESL essay raters’ cognitive processes in
applying the Jacobs et al. rubric: An eye-movement study. Assessing
Writing, 25, 37–53.
Wolfe, E. W. (2005). Uncovering rater’s cognitive processing and focus
using think-aloud protocols. Journal of Writing Assessment, 2(1),
37–56.
40 
C 2020 by the National Council on Measurement in Education
17453992, 2020, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/emip.12329 by Victoria University Of Wellington, Wiley Online Library on [05/05/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Wolfe, E. W., Song, T., & Jiao, H. (2016). Features of difficult-to-
score essays. Assessing Writing, 27, 1–10. https://doi.org/10.1016/
j.asw.2015.06.002.
Yancey, K. B. (1999). Looking back as we look forward: Historiciz-
ing writing assessment. College Composition and Communication,
50(3), 483–503.
Educational Measurement: Issues and Practice