Factors Affecting the Assessment of ESL Students’ Writing
Jinyan Huang, Ph.D.
Assistant Professor in TESOL and Assessment
Niagara University
Abstract: Due to the different linguistic
and cultural backgrounds of ESL students,
the assessment of their English writing
becomes a problematic area. On the one
hand, many factors affect ESL students’
writing, including their English
proficiency, mother tongue, home culture,
and style of written communication. In
rating ESL students’ writing, raters may
differentially consider these factors. On
the other hand, empirical studies have
found that many factors (e.g., raters’
linguistic backgrounds, previous
experiences, and prior training in
assessment) affect the rating of ESL
students’ writing. The impact of these
factors leads to questions about the
accuracy, precision and ultimately, the
fairness of the assessment of ESL
students’ writing. This paper reviews 20
major empirical studies in which the
factors affecting the rating of ESL writing
in North American school contexts were
investigated. These factors are
categorized into two types: rater-related
and task-related. Rater-related factors
include the rating methods used, rating
criteria, raters’ academic disciplines,
professional experiences, linguistic
backgrounds, tolerance for error,
perceptions and expectations, and rater
training. Task-related factors include the
types of and difficulty levels of writing
tasks. The paper also identifies research
gaps and proposes directions for future
research in the assessment of ESL writing.
Introduction
The assessment of writing has long
been considered a problematic area for
educational assessment professionals. As
1
stated by Speck and Jones (1998), “there
are more problems than solutions -
problems of inter-grader reliability,
single-grader consistency, and ultimate
accountability for the grades we assign”
(p. 17). Due to the different linguistic and
cultural backgrounds of English-as-a-
second-language (ESL) students, the
assessment of their English writing is
more problematic than the assessment of
NE students’ writing (Connor-Linton,
1995; Hamp-Lyons, 1991a; Sakyi, 2000;
Sweedler-Brown, 1993). On the one hand,
many factors affect ESL students’ writing,
including their English proficiency,
mother tongue, home culture, and style of
written communication (Casanave &
Hubbard, 1992; Hinkel, 2003; Shaw &
Liu, 1998; Yang, 2001). On the other
hand, raters may differentially consider
these factors when rating ESL students’
writing. Empirical studies have noted
differences in rater behavior for ESL
students (Bachman, 2000). Rater
background, mother tongue, previous
experience, amount of prior training, and
types and difficulty of writing tasks have
been found to affect the rating of the
written responses of ESL students (Brown,
1991; Kobayashi, 1992; Sakyi, 2000;
Santos, 1988; Weigle, 1994, 1999; Weigle,
Boldt, & Valsecchi, 2003). The impact of
these factors leads to questions about the
accuracy, precision and ultimately, the
fairness of the scores obtained from the
ratings of written work produced by ESL
students.
Fairness is a priority in the field of
educational assessment. Educational
organizations, institutions, and individual
professionals should make assessments as
IJAES – Vol. 5 No. 1 Pg. No. 2
fair as possible for test takers of different
races, genders, and ethnic backgrounds
(American Educational Research
Association, American Psychological
Association, National Council on
Measurement in Education, 1999; Cole &
Zieky, 2001; Joint Advisory Committee,
1993). Due to a significant growth in the
number of ESL students being educated in
North American schools in the past two
decades (CBIE, 2002; IIE, 2001), fairness
issues in ESL writing assessments have
been of growing interest and importance
(Connor-Linton, 1995; Hamp-Lyons,
1991a; Kunnan, 2000; McDaniel, 1985;
Sweedler-Brown, 1993; Vaughan, 1991).
Increasingly, writing-proficiency
standards are being established for both
secondary school and university students
in North America without regard to
students’ native languages (EQAO, 2002;
Johnson, Penny, & Gordon, 2000;
Thompson, 1990). ESL students have to
write the same tests as native English
(NE) students. Like NE students, ESL
students are expected to successfully
demonstrate their English writing skills or
complete high-stakes essay examinations
(Casanave & Hubbard, 1992; Hayward,
1990, Wiggins, 1993).
However, research shows that ESL
students face considerable challenges
passing institutional or provincial/state
competency examinations of writing
(Blackett, 2002; Johns, 1991; Ruetten,
1994; Thompson, 1990). Further, these
challenges may be due to more than
language deficiencies. As an example,
leniency or severity within raters could
underestimate or overestimate ESL
students’ performance on these writing
examinations. Previous studies have also
found that raters with different teaching
experience assign different scores to the
same piece of ESL writing (Cumming,
1990b; Hamp-Lyon, 1996; Rubin &
William-James, 1997). Such variability
due to raters, therefore, may threaten the
fairness of assessments of ESL writing.
Assessing and evaluating ESL
writing involves both assigning a score or
grade to an essay and commenting on it
(Perkins, 1983). This review paper
discusses only the scoring of ESL essays.
Therefore, terms “grading”, “rating”, and
“scoring” are used interchangeably in this
paper, referring to the process raters use
to arrive at the scores students will receive
(Speck & Jones, 1998). Given the
increasing number of ESL students being
educated in the North American context
(IIE, 2001; CBIE, 2002), the investigation
of the factors that affect the accuracy,
reliability, and validity of the scoring of
English compositions written by ESL
students in North American secondary
schools and colleges/universities is the
focus of this paper. The central argument
of this paper is that to some extent ESL
compositions are not fairly scored as they
should be because various factors have
been found to affect the accuracy,
reliability, and validity of the scoring of
ESL compositions at North American
schools. Studies conducted in North
American school contexts and related to 1)
secondary school teachers’ or
college/university professors’ rating of
ESL students’ writing; and 2) raters’
rating of ESL compositions in large-scale
state/provincial and institutional
assessment contexts are reviewed. A rater,
depending on the assessment context, can
be a teacher/professor or a grader/marker.
The writing tasks can be compositions,
short essays, timed essays, or constructed
written responses in state/provincial (e.g.,
diploma exams, literacy tests) and
institutional (e.g., placement, exit exams)
assessment contexts. They can include
writing assignments, term papers, and
course papers.
This review paper begins with a brief
explanation of reliability and validity in
the context of the rating of students’
writing. It then reviews the empirical
studies on factors affecting the accuracy,
reliability, and validity of the rating of

ESL students’ writing in North American
school contexts. Finally, it identifies
research gaps and proposes directions for
future research.
Reliability and Validity of the Rating of
Writing
Inter- and intra-rater reliability or
rating consistency is particularly
important in the rating of students’
writing since there may exist unwanted
variations in scores due to variations
among raters and within rater (Bachman,
1990; Gamaroff, 2000; Huot, 1990;
Johnson et al., 2000). Different raters
often assign different scores to the same
piece of writing, and the same rater may
assign different scores to the same
composition at different times. Both of
these variations are problematic as they
adversely affect the reliability, validity,
and fairness of the scores assigned to
students.
Reliability
In classical test theory (CTT), an
examinee’s observed score consists of a
“true score” and an “error score” (Huot,
1990). Within the CTT framework, the
variance of the observed score across
persons is equal to the sum of the
variances of the persons’ true scores and
error scores. Variation among raters and
within raters of writing constitutes
random measurement error if there is no
predictability in the variation. As such,
these sources of error contribute to the
variance of the error scores. Spearman
(1904) defined reliability as the ratio of
the true score variance to the total score
variance. Therefore, if the error score
variance is small, then reliability will be
high and close to 1.0. In contrast, if the
error score variance is large, then
reliability will be low. Given that random
variation among and within raters of
writing contributes to the error score
variance, this random variation
contributes to lower reliability.
IJAES – Vol. 5 No. 1 Pg. No. 3
Consequently, the study of random inter-
and intra-rater reliability and attempts to
lower these two unwanted sources of
variability are warranted. Rater reliability
or consistency is important because it
indicates the precision of the rating of
students’ writing in North American
school contexts. Thus consistency is
related to fairness for test-takers (Johnson
et al., 2000). Therefore, it is vital to
ensure a high level of rater reliability in
the rating of writing.
Validity
The validity of scoring is also a major
concern in the rating of student writing
(Connor-Linton, 1995; Hamp-Lyons,
1991a). Messick (1989) describes validity
as “an integrated evaluative judgment of
the degree to which empirical evidence
and theoretical rationales support the
adequacy and appropriateness of
inferences and actions based on test
scores” (p. 13). In this sense, a student
who scores high on a writing test would
be considered a good or competent writer,
and a student who scores low would be
labeled as a poor or incompetent writer
(Huot, 1990).
Of concern are the sources of
systematic measurement error that, if
present, would serve to confound the
interpretation of a writing score. If the
nature of the systematic error raises
observed scores, then the observed scores
would systematically overestimate true
scores. Alternatively, if the nature of the
systematic error is to lower the observed
scores, then the true scores would be
underestimated. Variations due to
systematic sources lower the validity of
the interpretation of observed scores as
estimates of the corresponding true scores.
Many factors may contribute to the
systematic rater error of writing. In the
case of ESL writing, these factors include
the raters’ linguistic and academic
backgrounds, rater severity, raters’
tolerances for errors, rater training, types
IJAES – Vol. 5 No. 1 Pg. No. 4
and difficulty of writing tasks, and rater
interactions with tasks (Ferris, 1994;
Janopoulos, 1992; Russikoff, 1995;
Santos, 1988; Song & Caruso, 1996;
Sweedler-Brown, 1993; Vaughan, 1991;
Weigle, 1994, 1998).
Like reliability of rating, the study of
validity is important because it indicates
the absence or presence of bias in the
rating of writing. Likewise, validity is
closely tied to fairness. Cole and Moss
(1989) argue that fairness is an aspect of
validity. Messick (1989) clearly ties
fairness to validity in his “Validity”
chapter in the third edition of Educational
Measurement (Linn, 1989). An invalid
writing score for a student would mean
unfairness to him/her and produce
consequences (Huot, 1990; Johnson et al.,
2000). Therefore, it is important to ensure
valid ratings of students’ compositions.
Both reliability and validity are
closely related to fairness, therefore we
need to consider fairness as a critical
central component and give primacy to it
(Kunnan, 1997). In talking about
reliability, validity, and fairness in
language assessment, Kunnan (2000)
argues that “if a test is not fair there is
little or no value in it being valid and
reliable or even authentic and interactive”
(Kunnan, 2000, p. 10).
Factors Affecting the Rating of ESL
Writing
Table 1 contains a summary of 20
major empirical studies in which the
factors affecting the accuracy, reliability,
and validity of the rating of ESL writing
in North American schools were
investigated. The nature of the writing
tasks, number of students or papers (ESL
versus NE), number of raters, rater
qualifications and training, and rating
methods of each empirical study are listed.
As shown in Table 1, many factors
affecting the rating of ESL compositions
in North American school contexts have
been studied. These factors can be
categorized into two types: rater-related
and task-related. Rater-related factors
include the rating methods used, rating
criteria, raters’ academic disciplines,
professional experiences, linguistic
backgrounds, tolerance for error,
perceptions and expectations, and rater
training. Task-related factors include the
types of and difficulty levels of writing
tasks. Each of these factors is reviewed
below, beginning with rating method.
Rating Methods. Holistic and
analytical rating have gained wide
acceptance in writing assessment
practices (Jacobs, Zinkgraf, Wormuth,
Hartfiel, & Hughey, 1981; Perkins, 1983).
In holistic rating, each rater rates the
overall writing proficiency on a single
rating scale, whereas in analytical rating
the writing performance is categorized
into identifiable component parts, such as
organization and content, and then each
component is separately marked using a
rating scale (Hamp-Lyons, 1991b;
Stiggins & Bridgeford, 1983).
 IJAES – Vol. 5 No. 1 Pg. No. 5

Table 1.
A Summary of 20 Major Empirical Studies on ESL Writing Assessment
# of # of Rater
Author(s) Year Writing Task(s) Rater Qualification Rating Methods
Students Raters Training
Essay 56 ESL English & ESL Holistic
Brown (1991) 16 Yes
(institutional) 56 NE faculties (6-point)
Essay Novice & expert Analytical* 3
Cumming (1990b) 12 ESL 13 Yes
(institutional) ESL teachers (4-point)
Essay Experienced in Holistic
Engber (1995) 66 ESL 10 No
(institutional) holistic rating (6-point)
Persuasive essay 30 ESL Composition Holistic
Ferris (1994) 3 No
(institutional) 30 NE instructors (6-point)
Hamp-Lyons & 64 writing tasks Experienced in ESL Holistic
8583 ESL 2 Yes
Mathias (1994) (commercial) writing assessment (10-point)
Intara-prawat & Persuasive essay Experienced in Holistic
12 ESL 5 No
Steffensen (1995) (institutional) teaching ESL (5-point)
Essay Professors, grad. & Analytical 4
Kobayashi (1992) 2 ESL 269 No
(institutional) undergrad. students (10-point)
11 ESL Trained English Analytical 3
McDaniel (1985) Essay (provincial) 2 Yes
9 NE teachers (9-point)
Mendel-sohn & Essay University Holistic
8 ESL 26 No
Cumming (1987) (institutional) professors (8-point)
Reid & O’Brien Essay Holistic
11 ESL 10 Graduate students No
(1981) (institutional) (9-point)
Essay 89 ESL Some with ESL Holistic &
Russikoff (1995) 8 No
(institutional) 11 NE writing expertise Analytical
12 Experienced in
Essay Holistic
Sakyi (2000) university 6 writing, teaching & No
(institutional) (5-point)
students testing
Essay University Analytical 6
Santos (1988) 2 ESL 178 No
(institutional) professors (10- point)
Holistic &
Song & Caruso Essay 2 ESL English & ESL
62 No Analytical
(1996) (institutional) 2 NE professors
(6-point)
Sweedler-Brown Essay 6 ESL Experienced in Holistic
6 No
(1993) (institutional) 6 NE writing (6-point)
Essay 6 university Experienced in Holistic
Vaughan (1991) 9 No
(institutional) students holistic assessment (6-point)
Essay Analytical 3
Weigle (1994) 4 ESL 16 Graduate students Yes
(institutional) (10-point)
Essay Analytical 3
Weigle (1998) 60 ESL 16 Graduate students Yes
(institutional) (10-point)
Essay Analytical 3
Weigle (1999) 60 ESL 16 ESL faculty & TAs Yes
(institutional) (10-point)
Weigle, Boldt & Essay University
6 ESL 16 No Analytical
Valsecchi (2003) (institutional) professors & TAs
* For analytical rating, 3 (4-point) indicates that the writing performance is categorized into 3
identifiable component parts and each component part is separately rated using a 4-point scale.
IJAES – Vol. 5 No. 1 Pg. No. 6
These two rating methods have both
advantages and disadvantages (Perkins,
1983). Holistic rating has the highest
construct validity when overall attained
writing proficiency is the construct to be
assessed. It is frequently used as a tool for
“certification, placement, proficiency, and
research testing” (Perkins, 1983). But it
has “threats to reliability” because it can
be highly subjective due to “bias, fatigue,
internal lack of consistency, previous
knowledge of the student, and/or shifting
standards from one paper to the next”
(Perkins, 1983, p. 653). Analytical rating
allows students to see how they have
performed on each component considered
and, as such, provides diagnostic
information. Although analytical rating
produces higher inter-rater reliability than
holistic rating, it is more time-consuming
(Perkins, 1983).
Empirical evidence indicates that the
two rating methods may affect the
reliability and validity of the rating of
ESL compositions. For example, Song
and Caruso (1996) examined the degree to
which differences existed between the
holistic rating and analytical ratings of
four compositions written by two ESL
speakers and two NE speakers. The
ratings were completed by 32 English and
30 ESL professors in the English
Department of an American university.
The two faculty groups were divided into
two subgroups, one rating the four essays
holistically on a 1 to 6 scoring rubric and
the other rating the same four essays
analytically using 1 to 6 scale for each of
10 features, 6 comprising rhetorical
features and 4, language features.
Statistical analyses indicated that both the
holistic and analytical methods produced
no significant differences between the
scores assigned to ESL and NE essays.
However, there was a significant
difference (p < .05) between the holistic
scores awarded by the English faculty and
the holistic scores awarded by the ESL
faculty, with the English faculty assigning
higher scores to all four essays. The
English faculty raters seemed to give
more weight to the overall content and
quality of the rhetorical features in the
writing samples than they did to language
use when they used the analytic method.
In this study, the two ESL compositions
were written by Russian speakers. This
factor might limit the generalizability of
the findings.
Rating Criteria. Rating criteria are
specified in scoring rubrics and rating
scales to guide the rating of the written
responses of the students who were tested.
Generally, rhetoric (text organization,
style, register, discourse functions, or
genre), language (grammar/mechanics,
syntax, and vocabulary), and content
(knowledge of subject, thesis
development, topic relevance, original or
factual support) are considered as the
three main criteria for rating both ESL
and NE compositions in North American
school contexts (Cumming, 1990a;
Mendelsohn & Cumming, 1987; Reid &
O’Brien, 1981; Russikoff, 1995; Sakyi,
2000).
Empirical research indicates that
raters weight or emphasize the criteria
differently while rating compositions
written by ESL and NE students
(McDaniel, 1985). By analyzing a large-
scale essay test administered to all
graduating high school students in British
Columbia, McDaniel (1985) found that, in
analytical rating, raters (people teaching
or qualified to teach English in high
school, college, university, or as graduate
teaching assistants, or employed as
graders in English departments of public
schools) rated essays written by ESL and
NE students differently in terms of the
following three specific criteria: “content
development and organization,”
“sentences,” and “words.” For example,
sentence-level errors had a stronger
negative effect on the scores of ESL
writers than on the scores of the NE
writers.

Further, raters may unfairly focus on
one or two criteria outlined in the rating
guide or use their own internalized criteria
when rating ESL compositions. For
example, Russikoff (1995) found that
when the raters rated ESL compositions
holistically, they paid attention only to
“language use,” which was the ESL
students’ weakness. However, when the
same raters rated the same ESL
compositions analytically, they were
surprised to see how strong the “content”
and “organization” of these compositions
were.
Sakyi (2000) found that six raters,
who had experience in teaching and
testing writing ranging from 6 to 37 years,
focused on different criteria or used
different criteria while holistically rating
12 essays written by first year university
students attending a Canadian university.
He found that the raters tended to focus
on three criteria not in the rating guide:
errors in the text; essay topic, and
presentation of ideas; and on their own
personal reaction to the text. Further, he
also found that raters who tried to follow
the rating guide assigned scores to essays
using mostly one or two of the particular
features outlined in the rating guide.
Other researchers have also found
similar results to those reported by Sakyi
(2000). For example, syntactic and lexical
proficiency have become major criteria in
the rating of ESL writing. Essay raters
often consider assigning a lower rating to
an essay due to its simple construction
and lexicon (Vaughan, 1991). However,
investigations into ESL writing have
established that, in large-scale testing and
university-level assessments of student
writing, syntactic and lexical simplicity
are often considered to be a severe
handicap for ESL students (Casanave &
Hubbard, 1992; Hinkel, 2003). Further,
Engber (1995) investigated the
relationship between lexical proficiency
(lexical richness and lexical errors) and
the quality of compositions written by
IJAES – Vol. 5 No. 1 Pg. No. 7
ESL students in an intensive English
program at an American university. The
results showed high and significant
correlations (p < .05) for lexical variation
and lexical variation minus error,
suggesting that the “diversity of lexical
choice and the correctness of lexical
form” have a significant benign effect on
the ratings of timed essays written by ESL
students at an intermediate to high-
intermediate level of proficiency (Engber,
1995, p. 150).
Similarly, the use of rhetorical
strategies by students in their writing is
considered an important criterion by some
raters of compositions. By analyzing 60
persuasive compositions written by both
ESL and NE freshman composition
students at an American university, Ferris
(1994) found that in the rating of
persuasive/argumentative writing, raters
often considered assigning good scores to
writers who used rhetorical strategies.
However, the ESL students were at a
disadvantage because of their linguistic
and rhetorical deficits for the task of
persuasion in English (Hinds, 1987;
Ostler, 1987). Intaraprawat and Steffensen
(1995) further investigated the effect of
meta-discourse features, "facets of a text
which make the organization of the text
explicit, provide information about the
writer's attitude toward the text content,
and engage the reader in the interaction”
(p. 253) on the rating of ESL
compositions. After analyzing 12 good
and poor essays written by ESL students
at an American university, they found that
ESL essays which contained a variety of
meta-discourse features were considered
“good” essays and received high scores.
Intaraprawat and Steffensen’s findings
also indicated that ESL students were at a
disadvantage because they had
considerable challenges in learning how
to apply English meta-discourse features
when writing in English. In this study, the
criterion used to rate the essays affected
the rating of ESL students’ essays.
IJAES – Vol. 5 No. 1 Pg. No. 8
Lastly, different criteria may be used
by different raters in the rating of ESL
compositions of different types and
quality. Weigle et al. (2003) found that
professors from different disciplines
tended to use different criteria for
assessing ESL essays. For example, the
ESL and English department raters chose
grammar more frequently as the most
important reason for failing essays on
general topics in contrast to essays that
were based on such specific texts as
reading passages and lecture notes. This
result suggests that ESL students may be
“penalized” for poor linguistic control in
essays on general topics, which are
typically used in large-scale assessment
writing tasks. In contrast, the psychology
raters chose content as the most important
aspect of writing for both essays on
general topics and essays that were based
on such specific texts as reading passages
and lectures notes. Similarly, Sweedler-
Brown (1993) found that raters with no
ESL training placed far more emphasis on
the essays’ linguistic features (sentence-
level errors) than on the essays’ strong
rhetorical features.
Raters’ Discipline. Several
researchers have investigated the effects
of a rater’s discipline (e.g., English,
History, ESL, and Engineering) on the
rating of compositions written by ESL
students at North American universities
(Brown, 1991; Mendelsohn & Cumming,
1987; Santos, 1988; Song & Caruso, 1996;
Weigle et al., 2003). Mendelsohn and
Cumming (1987) found that Engineering
professors attributed more importance to
language use than to rhetorical
organization in rating the effectiveness of
ESL papers. ESL professors attributed
more importance to rhetorical
organization when rating the same paper.
Santos (1988) investigated 178 non-ESL
professors’ (96 in the humanities/social
sciences and 82 in the physical sciences)
ratings of two compositions written by
two Asian students at an American
university. The results showed that all
raters rated the content (holistic
impression, development, and
sophistication) more severely than the
language (comprehensibility, acceptability,
and irritation). However, raters in the
humanities/social sciences were more
lenient in rating than raters in the physical
sciences.
Brown (1991) and Song and Caruso
(1996) investigated if raters from different
disciplines rated ESL speakers’
compositions the same way as they rated
NE students’ compositions. Brown (1991)
investigated whether significant
differences existed in the holistic writing
scores of 56 ESL and 56 NE students at
the end of their first-year composition
courses at an American university. The
compositions were marked by eight ESL
and eight English raters. The results
showed that there were no significant
differences (p < .05) between the scores
assigned to ESL and NE compositions or
by ESL and English raters. Brown (1991)
explained that the raters from both
faculties seem to “assign very similar
scores - regardless of differences in
background or training,” although they
may “arrive at those scores from
somewhat different perspectives (as
indicated by the features analysis)” (p.
601). Similarly, as previously reported,
Song and Caruso (1996) did not find any
differences between the scores assigned to
ESL and NE essays by 30 ESL and 32
English faculty members at an American
university. However, they found
significant differences (p < .05) between
ESL and English faculty raters.
A feature of the papers reviewed thus
far is that the essays were on general
topics. Recently, Weigle et al. (2003)
extended this area of research to the rating
of essays that were based on specific texts
such as reading passages and lecture notes
by ESL teachers and teachers in English,
history, and psychology departments at an
American university. They found that

scores differed (p < .05) among raters
from different disciplines. The instructors
from the English department were the
most severe raters and the instructors
from the history department the most
lenient. The ESL raters, in contrast to the
raters from other disciplines,
demonstrated consistency across all tasks.
Raters’ Professional Experience. The
ESL writing assessment literature has
shown that raters’ professional experience,
such as the number of years of teaching
and rating ESL writing, influences their
rating of ESL writing (Cumming, 1990a;
Hamp-Lyon, 1996; Rubin & William-
James, 1997; Vaughan, 1991). Song and
Caruso (1996) found that raters with more
years of experience in teaching tended to
be more lenient than raters with fewer
years of teaching experience when they
used holistic scoring. In contrast, when
they employed analytical ratings,
experience was not a significant factor
that affected their rating of ESL
compositions.
Reid and O’Brien (1981) examined
the reliability of the ratings of 11
compositions written by 11 ESL
university students who were enrolled in
an intensive ESL program. Ten raters
holistically marked the compositions. The
results showed that the more experienced
the raters were in teaching ESL writing
and/or the more practice the raters had
with holistic scoring, the higher the inter-
and intra-rater reliability.
Cumming (1990b) investigated the
ratings of seven novice and six expert
ESL raters who scored 12 compositions
written by adult students with differing
levels of ESL proficiency at a Canadian
university. The results showed that there
was a significant difference (p < 0.05)
between the two groups of raters’ ratings
for “content” and “rhetorical
organization,” but not for “language use.”
The novice raters consistently rated
content and rhetorical organization higher
than the expert raters. Further, their
IJAES – Vol. 5 No. 1 Pg. No. 9
ratings for language use were significantly
different from their ratings of content and
rhetorical organization. However, the
expert raters did not show any significant
differences among their ratings of the
three aspects of the compositions.
Raters’ Linguistic Background.
Raters’ linguistic background (native
language) can be a factor that affects their
ratings of ESL compositions. Kobayashi
(1992) investigated how 145 NE raters
and 124 native Japanese raters at the
professorial, graduate, and undergraduate
levels rated two compositions written by
ESL students at an American university.
The results showed that NE raters were
stricter about grammar than the native
Japanese raters, and that NE professors
and graduate students gave more positive
ratings for the aspects of clarity of
meaning and organization for both
compositions than did the comparable
native Japanese raters. However, the
native Japanese undergraduates rated both
compositions much more positively than
did the NE undergraduates. The study
ignored such factors as raters’ age and
gender, which have been found to affect
the rating of ESL compositions (Santos,
1988; Vann, Lorenz, & Meyer, 1991).
Raters’ Perceptions And
Expectations. Raters’ perceptions and
expectations of ESL writing may have an
effect on the rating of ESL compositions
(Janopoulos, 1995). Casanave and
Hubbard (1992) surveyed graduate faculty
at an American university to obtain
specific information about the writing the
faculty required of first-year doctoral
students, the criteria they use to evaluate
students’ written work, and the writing
problems of both ESL and NE students.
Eighty-five graduate faculty representing
the humanities/social sciences and
science/technology academic fields
responded to the survey. The findings
revealed that faculty from both fields
claimed to use similar criteria for
evaluating students’ writing. In terms of
IJAES – Vol. 5 No. 1 Pg. No. 10
importance, discourse-level criteria (e.g.,
quality of content, development of ideas,
and adequate treatment of topic) were
ranked high, and word- and sentence-level
criteria (e.g., accuracy of grammar, size of
vocabulary, and spelling and punctuation)
were ranked low. However, faculty from
the humanities/social sciences gave high
ratings to “development of ideas, quality
of overall paper organization,
appropriateness of vocabulary, and
appropriateness of tone/style” than
science and technology faculty (p. 38). In
terms of their students’ writing problems,
while all participants agreed that ESL
students had more problems than NE
students on most features of writing, they
perceived ESL students as “having only
minor or moderate problems meeting the
constraints of particular assignments” (pp.
38-39). In addition, two differences were
found between the two groups of faculty
members. First, appropriateness of
vocabulary usage was considered to be
the biggest problem for ESL students by
the humanities/social sciences faculty, but
was considered less of a problem by the
science and technology faculty. Second,
humanities/social sciences faculty thought
that both ESL students and NE students
had greater problems with overall paper
organization than did the science and
technology faculty.
Raters’ Tolerance for Error.
Research suggests that grammatical errors
made in a written composition lead to
lower ratings of ESL students’
compositions (Homberg, 1984; Sweedler-
Brown, 1993). But there are different
types of errors, and raters from different
disciplines have different tolerances for
errors and may assign different weights to
grammatical errors while rating ESL
compositions (Janopoulos, 1992; Santos,
1988; Vann, Meyer, & Lorenz, 1984;
Vann et al., 1991).
Four studies in which faculty
members’ responses to the writing errors
of ESL students were examined
(Janopoulos, 1992; Santos, 1988; Vann et
al., 1984; Vann et al., 1991). The data
from all four studies revealed that
professors considered such errors as word
order, verb form, and relative clause to be
among the most serious and placed such
errors as incorrect article and preposition
usage at the “high end of the spectrum of
tolerance” (Janopoulos, 1992, p. 116). In
addition, these four studies also revealed
that, among faculty members in social
sciences, education, humanities,
biological and agricultural sciences,
physical and mathematical sciences, and
engineering, those from the social
sciences were the most tolerant of the
ESL writing errors in general. However,
the biggest limitation of the two studies
completed by Vann et al. (1984) and
Janopoulos (1992) is that the errors were
considered at the sentence-level rather
than in extended discourse.
Rater training can also change the
raters’ tolerance for grammatical errors.
McDaniel (1985) found that “the
judgments of graders who were not
trained in evaluating ESL writing were
dominated by error when they graded ESL
essays” (quoted in Sweedler-Brown, 1993,
p. 5). Sweedler-Brown (1993)
investigated the rating of ESL papers by
experienced English writing instructors
with no training in evaluating ESL writing
and found that “sentence-level error was
the only significant influence on holistic
score” and was the “critical factor” in
causing ESL essays to fail (p. 12).
These studies suggest that a “double
standard” may exist in terms of faculty
tolerance of ESL writing errors, with
some faculty apparently willing to make
allowances for the efforts of ESL writers
(Janopoulos, 1992). Ruetten (1994)
suggested that, to provide more accurate
holistic ratings, officials responsible for
university composition programs need to
clarify their standards regarding ESL
students, especially on the “acceptability
of error, and to train faculty members to

read accordingly” (p. 94).
Rater Training. Rater training is an
essential component of the essay rating
process (Davidson, 1991; Weigle, 1994).
Training provides raters with a clear
conception of what a piece of quality
writing looks like, and thus promotes rater
consensus (Homburg, 1984). Training can
also minimize differences caused by
raters’ different backgrounds (Jacobs et
al., 1981; Reid & O’Brien, 1981) and
modify expectations of good writing by
clarifying for the raters both the task
demands and writer characteristics (Huot,
1990). In fact, rater training is an issue
that lies at the heart of both reliability and
validity in ESL essay rating (Weigle,
1994). Homburg (1984) commented that
holistic rating of ESL compositions, “with
training to familiarize readers with the
types of features present in ESL
compositions, can be considered to be
adequately reliable and valid” (p. 103).
Weigle (1994) investigated the
effects of training on eight experienced
and eight inexperienced raters of the ESL
placement compositions written by four
ESL students at an American university.
The results showed that four of the
inexperienced raters gave scores after
training that were different from the
scores they had given to the same
compositions before training. The think-
aloud protocols of these four raters
indicated that the changes in their scores
could be attributed to the following three
factors: 1) clarification of the rating
criteria, 2) revision of expectations, and 3)
concern for rater agreement. Analyses of
raters’ think-aloud protocols can be an
approach to the study of rater training
effects; however, it is never a complete
record of one’s cognitive activity (Ericson
& Simon, 1993). Hence the changes in
raters’ scores from before to after training
could also be attributed to other factors.
Weigle (1998) conducted a second
similar study to explore differences in
rater severity and consistency among
IJAES – Vol. 5 No. 1 Pg. No. 11
inexperienced and experienced raters of
ESL placement compositions in an
American university both before and after
rater training by using the Item Response
Theory (IRT) program FACETS (Linacre,
1989). The analysis showed the changes
in rater behavior from before- to after-
training. First, the spread of rater severity
estimates was reduced, particularly for the
inexperienced raters, indicating the raters
were more like each other after training
than before training. Second, the training
seemed to have brought the extreme raters
“within a more tolerable range of
severity,” but still had not eliminated all
differences in rater severity. Third, the fit
statistics from before- to after-training
were improved. For example, the three
raters who were highly inconsistent before
training became quite consistent after
training. Finally, the results showed that
before training the inexperienced raters
were more extreme in their severities than
the experienced raters, and for most part
they were more severe as well. These
results suggest that rater training can be
more successful in helping essay raters to
give more predictable scores than in
getting them to give identical scores.
Types and Difficulty of Writing Task.
Writing tasks of different types should be
of equal difficulty; and different raters
should rate essays of different difficulty
levels consistently. However, in reality
they may not (Brown, Hilgers, & Marsella,
1991). The types and difficulty of writing
prompts, then, may affect raters’ ratings
of ESL compositions. Weigle (1999)
investigated how experienced and
inexperienced raters scored placement
essays written by ESL students on two
different prompts. One prompt was a
graph while the second prompt was a
chart or table. In the case of the graph, the
students were required to interpret the
graph and make predictions based on the
information contained in the graph. The
chart or table prompt required the students
to make and defend a choice based on
IJAES – Vol. 5 No. 1 Pg. No. 12
information contained in the chart or table.
The results showed that there were no
significant differences in severity of
marking between the experienced and
inexperienced raters before training on the
chart or table essay. However, the
inexperienced raters were significantly
more severe on the graph essays than
were the experienced raters before
training. After training no significant
differences were found between the two
rater groups on either prompt.
In responding to the effects of the
difficulty of writing prompts on raters’
ratings of ESL compositions, Hamp-
Lyons (1990) indicated that raters could
compensate for more difficult questions
by giving higher scores. In later research,
Hamp-Lyons and Mathias (1994)
confirmed that topics which were judged
more difficult by experts received higher
scores, and those which were judged
easier received lower scores, indicating
essay raters were “consciously or
unconsciously compensating” in their
rating for relative prompt difficulty
“based on their own, internalized,
difficulty estimates” (p. 59).
Identified Research Gaps
Based upon the above review of 20
empirical studies on ESL writing
assessments in North American school
contexts, three major research gaps are
evident. First, there was an imbalance by
both school levels and countries. Most
studies examined the factors affecting the
rating of ESL compositions written by
undergraduate and graduate students in
colleges and universities. Very few
studies, however, examined the factors
affecting the rating of ESL compositions
written by secondary school students.
Among these reviewed studies, the study
by McDaniel (1985) is the only one
involving secondary school students.
Undergraduate and graduate students
usually come to study in North America
universities on their own and they need to
pass English language proficiency tests
before studying; hence they are quite
proficient in English. However, secondary
school students come to study in North
America with their immigrated parents.
Further, their English skills are not as well
developed as compared to university
students (Blackett, 2002). Thus the
secondary ESL students likely have very
different language exposure to English
and English writing skills. Further, most
studies were conducted in American
contexts. Only a few studies were
conducted within Canadian contexts
(Cumming, 1990a; McDaniel, 1985;
Mendelsohn & Cumming, 1987; Sakyi,
2000). ESL students in these two
countries are generally different in terms
of their linguistic backgrounds. For
example, in America, there are a large
number of Spanish-speaking ESL students;
whereas there are a large number of
French-speaking students and Asian
students in Canada (CBIE, 2002; IIE,
2001). ESL students’ native language
affects their English writing (Shaw & Liu,
1998; Yang, 2001).
Second, the generalizability of most
studies was limited due to their
unrepresentative sampling (McDaniel,
1985; Sakyi, 2000; Santos, 1988; Song &
Caruso 1996). For example, Song and
Caruso (1996) sampled two ESL
compositions written by Russian speakers
in their study. Similarly, Santos (1988)
sampled two compositions written by two
Asian students in order to investigate the
effects of a rater’s discipline on the rating
of ESL compositions. Such sampling was
not intended to be representative, limiting
the interpretations that can be made and
affecting the generalizability of their
results.
Finally, due to the quantitative nature
of analyzing and reporting ESL writing
scores, most reviewed studies used
quantitative approaches. However, in the
investigation of empirical evidence for
rater variation, most reviewed quantitative

studies were based on the framework of
CTT. In CTT, the ratio of true score
variance to observed score variance is
defined as reliability (Crocker & Algina,
1986). The reliability estimates of scores
derived from various holistic and
analytical scoring methods reported in the
literature have relied on CTT (Brown,
1991; Reid & O’Brien, 1981), which
accounts for only a single source of
variance within a given analysis.
Generalizability (G-) theory (Cronbach,
Gleser, Nanda, & Rajaratnam, 1972) is a
more powerful quantitative approach for
detecting rater variation in the field of
performance assessment (Linn & Burton,
1994). It extends the framework of CTT in
order to take into account the multiple
sources of variability that can have an
effect on test scores. It can identify the
sources of variance and error and estimate
the impact of these variance components
on scoring accuracy, allowing the
investigator to consider numerous
applications of an instrument (Shavelson
& Webb, 1991). Therefore, G-theory
provides a comprehensive conceptual
framework and methodology for
analyzing more than one measurement
facet simultaneously in investigations of
assessment error and score dependability
(Brennan, 2001). However, none of the
reviewed studies has used G-theory for
detecting rater variation.
Conclusion
This review paper explores various
factors that affect the accuracy, reliability
and validity of the scoring of ESL
compositions in North American school
contexts. The reviewed empirical studies
indicate that scoring criteria have become
a major concern about the construct
validity and reliability of the scoring of
ESL writing (Ferris, 1994; Mendelsohn &
Cumming, 1987; Sakyi, 2000). No matter
what scoring methods they use, different
raters use different criteria to rate ESL
compositions (Russikoff, 1995; Song &
IJAES – Vol. 5 No. 1 Pg. No. 13
Caruso, 1996). Therefore, rater is a factor
that has important impact on the rating of
ESL writing. Rater-related factors include:
rater’s linguistic background, discipline,
perceptions and expectations, professional
experience, tolerance for the error, and
rater training (Brown, 1991; Janopoulos,
1992; Mendelsohn & Cumming, 1987;
Santos, 1988; Weigle, 1994, 1998; Weigle
et al, 2003). Finally, the type and
difficulty of writing tasks, and the rater-
task interaction also affect raters’ rating of
ESL writing (Sakyi, 2000; Weigle, 1999).
The results of these studies have
confirmed the central argument of this
paper: To some extent ESL compositions
are not fairly scored as they should be.
In order to thoroughly examine the
factors that influence raters as they assess
ESL students’ writing in North American
schools, future empirical investigations
need to keep a balance between school
levels and assessment types. For example,
factors affecting classroom teachers’
assessment of ESL writing in secondary
and graduate school settings and raters’
rating of ESL writing in large-scale
provincial/state assessment contexts are
under-researched. As increasing number
of ESL students are educated in North
American schools, it is important to
explore these factors that have an impact
on the assessment of their writing and
ensure that the assessment of their writing
is fair.
References
American Educational Research
Association (AERA), American
Psychological Association (APA),
and National Council on
Measurement in Education (NCME).
(1999). Standards for educational
and psychological testing.
Washington, DC: American
Psychological Association.
Bachman, L. (2000). Modern language
testing at the turn of the century:
Assuring that what we count counts.
IJAES – Vol. 5 No. 1 Pg. No. 14
Language Testing, 17(1), 1-42.
Bachman, L. (1990). Fundamental
considerations in language testing.
Oxford: Oxford University Press.
Blackett, K. (2002). Ontario schools
losing English as a second language
programs - despite increase in
immigration. Retrieved August 31,
2004, from
http://www.peopleforeducation.com/r
eleases/2003/oct24_02.html
Brennan, R. L. (2001). Statistics for social
science and public policy:
Generalizability theory. New York:
Springer-Verlag.
Brown, J. D. (1991). Do English and ESL
faculties rate writing samples
differently? TESOL Quarterly, 25(4),
587-603.
Brown, J. D., Hilgers, T., & Marsella, J.
(1991). Essay prompts and topics.
Written Communication, 8, 533-556.
Casanave, C. P., & Hubbard, P. (1992).
The writing assignments and writing
problems of doctoral students:
Faculty perceptions, pedagogical
issues, and needed research. English
for Specific Purposes, 11, 33-49.
Canadian Bureau for International
Education. (2002, April 15).
International student numbers hit
record high, but Canada offers
dwindling support for African
students. Retrieved October 28, 2002,
from
http://www.cbie.ca/news/index_e.cfm
?folder=releases&page=rel_2002-
04-15_e
Cole, N. S., & Moss, P. A. (1989). Bias in
test use. In R. L. Linn (Ed.),
Educational Measurement (pp. 201-
219). New York: Macmillan.
Cole, N. S., & Zieky, M. J. (2001). The
new faces of fairness. Journal of
Educational Measurement, 38, 369-
382.
Connor-Linton, J. (1995). Looking behind
the curtain: What do L2 composition
ratings really mean? TESOL
Quarterly, 29(4), 762-765.
Crocker, L., & Algina, J. (1986).
Introduction to classical and modern
test theory. New York: Holt, Rinehart
and Winston.
Cronbach, L. J., Gleser, G. C., Nanda, H.,
& Rajaratnam, N. (1972). The
dependability of behavioral
measurements: Theory of
generalizability for scores and
profiles. New York: Wiley.
Cumming, A. (1990a). Application of
contrastive rhetoric in advanced ESL
writing. Paper presented at the 24th
Annual TESOL Conference, San
Francisco, CA.
Cumming, A. (1990b). Expertise in
evaluating second language
composition. Language Testing, 7,
31-51.
Davidson, F. (1991). Statistical support
for training in ESL composition
rating. In L. Hamp-Lyons (Ed.),
Assessing second language writing
(pp. 155-165). Norwood, NJ: Ablex.
Education Quality and Accountability
Office. (2002). Ontario Secondary
School Literacy Test, February 2002:
Report of provincial results. Toronto:
Queen’s Printer for Ontario.
Engber, C. A. (1995). The relationship of
lexical proficiency to the quality of
ESL compositions. Journal of Second
Language Writing, 4, 139-155.
Ericsson, K. A., & Simon, H. (1993).
Protocol analysis: Verbal reports as
data. Cambridge, MA: MIT Press.
Ferris, D. (1994). Rhetorical strategies in
student persuasive writing:
Differences between native and non-
native English speakers. Research in
the Teaching of English, 28, 45-65.
Gamaroff, R. (2000). Rater reliability in
language assessment: The bug of all
bears. System, 28, 31-53.
Hamp-Lyons, L. (1990). Second language
writing: Assessment issues. In B.
Kroll (Ed.), Second language writing:
Research insights for the classroom

(pp. 69-87). Cambridge: Cambridge
University Press.
Hamp-Lyons, L. (1991a). Issues and
directions in assessing second
language writing in academic
contexts. In L. Hamp-Lyons (Ed.),
Assessing second language writing in
academic contexts (pp. 323-329).
Norwood, NJ: Ablex.
Hamp-Lyons, L. (1991b). Rating
procedures for ESL contexts. In L.
Hamp-Lyons (Ed.), Assessing second
language writing in academic
contexts (pp. 241-277). Norwood, NJ:
Ablex.
Hamp-Lyons, L. (1996). The challenges
of second language writing
assessment. In E. White, W. Lutz and
S. Kamusikiri (eds.), Assessment of
writing: Policies, politics, practice
(pp. 226-240). New York: Modern
Language Association.
Hamp-Lyons, L., & Mathias, S. P. (1994).
Examining expert judgments of task
difficulty on essay tests. Journal of
Second Language Writing, 3(1), 49-
68.
Hayward, M. (1990). Evaluations of essay
prompts by nonnative speakers of
English. TESOL Quarterly, 24, 753-
758.
Hinds, J. (1987). Reader versus writer
accountability: A new typology. In U.
Connor & R. Kaplan (Eds.), Writing
across languages: Analysis of L2 text
(pp. 141-152). Reading, MA:
Addison-Wesley.
Hinkel, E. (2003). Simplicity without
elegance: Features of sentences in L1
and L2 academic texts. TESOL
Quarterly, 37, 275-301.
Homburg, T. (1984). Holistic evaluation
of ESL compositions: Can it be
validated objectively? TESOL
Quarterly, 18(1), 87-107.
Huot, B. A. (1990). Reliability, validity,
and holistic rating: What we know
and what we need to know. College
Composition and Communication, 41,
IJAES – Vol. 5 No. 1 Pg. No. 15
201-213.
Institute of International Education. (2001,
May 16). 98/99 open doors on the
Web. Retrieved June 15, 2002, from
http://www.opendoorsweb.org/Lead
%20Stories/international_studs.htm
Intaraprawat, P., & Steffensen, M. S.
(1995). The use of metadiscourse in
good and poor ESL essays. Journal
of Second Language Writing, 4(3),
253-272.
Jacobs, H. L., Zingraf, S. A., Wormuth, D.
R., Hartfiel, V. F., & Hughey, J. B.
(1981). Testing ESL composition: A
practical approach. Rowley, MA:
Newbury House.
Janopoulos, M. (1992). University faculty
tolerance of NS and NNS writing
errors: A comparison. Journal of
Second Language Writing, 1(2), 109-
121.
Janopoulos, M. (1995). Writing across the
curriculum, writing proficiency
exams, and the NNS college student.
Journal of Second Language Writing,
4, 43-50.
Johns, A. M. (1991). Interpreting an
English competency examination:
The frustrations of an ESL science
student. Written Communication, 8(3),
379-401.
Johnson, R. L., Penny, J., & Gordon, B.
(2000). The relation between score
resolution methods and interrater
reliability: An empirical study of an
analytic rating rubric. Applied
Measurement in Education, 13(2),
121-138.
Joint Advisory Committee. (1993).
Principles for Fair Student
Assessment Practices for Education
in Canada. Edmonton, AB.
Kobayashi, T. (1992). Native and
nonnative reactions to ESL
compositions. TESOL Quarterly, 26,
81-112.
Kunnan, A. J. (1997). Connecting fairness
and validation. In A. Huhta, V.,
Kohonen, L. Kurki-Suomo, & S.
IJAES – Vol. 5 No. 1 Pg. No. 16
Luoma (Eds.), Current developments
and alternatives in language
assessment (pp. 85-105). Jyvaskyla,
Finland.
Kunnan, A. J. (Ed.) (2000). Fairness and
validation in language assessment.
Cambridge: Cambridge University
Press.
Linacre, J. M. (1989). Many-facet Rasch
measurement. Chicago, IL: MESA
Press.
Linn, R. L. (1989). Educational
measurement (3rd ed.). New York:
Macmillan.
Linn, R. L., & Burton, E. (1994).
Performance-based assessment:
Implications of task specificity.
Educational Measurement: Issues
and Practice, 13, 5-8.
McDaniel, B. A. (1985). Ratings vs.
equity in the evaluation of writing.
Paper presented at the annual meeting
of the Conference on College
Composition and Communication,
Minneapolis, MN. (ERIC Document
Reproduction Service No. ED 260
459)
Mendelsohn, D., & Cumming, A. (1987).
Professors' ratings of language use
and rhetorical organization in ESL
compositions. TESL Canada Journal,
5, 9-26.
Messick, S. (1989). Validity. In R. L.
Linn (Ed.), Educational measurement
(pp. 13-103). New York: Macmillan.
Ostler, S. (1987). English in parallels: A
comparison of English and Arabic
prose. In U. Connor & R. Kaplan
(Eds.), Writing across languages:
Analysis of L2 text (pp. 169-185).
Reading, MA: Addison-Wesley.
Perkins, K. (1983). On the use of
composition rating techniques,
objective measures, and objective
tests to evaluate ESL writing ability.
TESOL Quarterly, 17(4), 651-671.
Reid, J., & O’Brien, M. (1981). The
application of holistic grading in an
ESL writing program. Paper
presented at the annual convention of
Teachers of English to Speakers of
Other Languages, Detroit, MI. (ERIC
Document Reproduction Service No.
ED 221 044)
Rubin, D. L., & Williams-James, M.
(1997). The impact of writer
nationality on mainstream teachers'
judgments of composition quality.
Journal of Second Language Writing,
6(2), 139-153.
Ruetten, M. K. (1994). Evaluating ESL
students' performance on proficiency
exams. Journal of Second Language
Writing 3, 85-96.
Russikoff, K. A. (1995). A comparison of
writing criteria: Any differences?
Paper presented at the annual meeting
of the Teachers of English to
Speakers of Other languages, Long
Beach, CA.
Sakyi, A. (2000). Validation of holistic
rating for ESL writing assessment:
How raters evaluate ESL
compositions. In A. Kunnan (Ed.),
Fairness and validation in language
assessment (pp. 129-152). Cambridge:
Cambridge University Press.
Santos, T. (1988). Professors' reactions to
the writing of nonnative-speaking
students. TESOL Quarterly, 22(1),
69-90.
Shavelson, R. J., & Webb, N. M. (1991).
Generalizability theory: A primer.
Newbury Park, CA: Sage.
Shaw, P., & Liu, E. T.-K. (1998). What
develops in the development of
second language writing. Applied
Linguistics, 19, 225-254.
Song, B., & Caruso, I. (1996). Do English
and ESL faculty differ in evaluating
the essays of Native English-
Speaking, and ESL students? Journal
of Second Language Writing, 5(2),
163-182.
Spearman, C. (1904). The proof and
measurement of association between
two things. American Journal of
Psychology, 15, 72-101.

Speck, B. W., & Jones, T. R. (1998).
Direction in the grading of writing?
In F. Zak and C. C. Weaver (Eds.),
The theory and practice of grading:
Problems and possibilities (pp. 17-
29). Albany: SUNY Press.
Stiggins, R. J., & Bridgeford, N. J. (1983).
An analysis of published tests of
writing proficiency. Educational
Measurement: Issues and Practices,
2(1), 6-19.
Sweedler-Brown, C. O. (1993). ESL essay
evaluation: The influence of
sentence-level and rhetorical features.
Journal of Second Language Writing,
2, 3-17.
Thompson, R. (1990). Writing-
proficiency tests and remediation:
Some cultural differences. TESOL
Quarterly, 24, 99-102.
Vann, R., Lorenz, F., & Meyer, D. (1991).
Error gravity: Faculty response to
errors in the written discourse of non-
native speakers of English. In L.
Hamp-Lyons (Ed.), Assessing second
language writing in academic
contexts (pp. 181-195). Norwood, NJ:
Ablex.
Vann, R., Meyer, D., & Lorenz, F. (1984).
Error gravity: A study of faculty
opinion of ESL errors. TESOL
Quarterly, 18, 427-440.
IJAES – Vol. 5 No. 1 Pg. No. 17
Vaughan, C. (1991). Holistic assessment:
what goes on in the raters' minds? In
L. Hamp-Lyons (Ed.), Assessing
second language writing in academic
contexts (pp. 111-126). Norwood, NJ:
Ablex.
Weigle, S. C. (1994). Effects of training
on raters of ESL compositions.
Language Testing, 11, 197-223.
Weigle, S. C. (1998). Using FACETS to
model rater training effects.
Language Testing, 15(2), 263-287.
Weigle, S. C. (1999). Investigating
rater/prompt interactions in writing
assessment: Quantitative and
qualitative approaches. Assessing
Writing, 6(2), 145-178.
Weigle, S. C., Boldt, H., & Valsecchi, M.
I. (2003). Effects of task and rater
background on the evaluation of ESL
writing: A pilot study. TESOL
Quarterly, 37(2), 345-354.
Wiggins, G. (1993). Assessing student
performance: Exploring the purpose
and limits of testing. San Francisco,
CA: Jossey-Bass.
Yang, Y. (2001). Chinese interference in
English writing: Cultural and
linguistic differences. (ERIC
Document Reproduction Service No.
ED 461 992).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.