Studies in Educational Evaluation 73 (2022) 101140

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.elsevier.com/locate/stueduc

Evaluating peer feedback as a reliable and valid complementary aid to

teacher feedback in EFL writing classrooms: A feedback giver perspective
Wenyan Wu a, Jinyan Huang b, *, 1, Chunwei Han c, Jin Zhang d
a
Beijing Institute of Graphic Communication, China
b
Jiangsu University, China
c
Henan University of Engineering, China
d
Xi’an International University, China

A R T I C L E I N F O A B S T R A C T

Keywords: This study evaluated peer feedback as a reliable and valid complementary aid to teacher feedback in Chinese
EFL writing classrooms college English writing classrooms. Thirty college students as peer feedback givers and four experienced college
Peer feedback English teachers as teacher feedback givers participated in the study. The participants scored 30 English essays
Teacher feedback
holistically and provided written comments on the content, format, language, and organizational aspects of each
Holistic scoring
Written comments
essay. The results indicated that the reliability of up to three peer feedback givers’ holistic scoring is equivalent
Generalizability theory to that of one teacher feedback giver’s scoring; further, peers performed almost as well as teachers in making
comments on the content and organizational aspects of English essays. Implications for Chinese college students
and their English teachers are discussed.

1. Introduction examined the effectiveness of peer feedback in contrast to teacher

1.1. Research Background
During the past couple of decades, the positive impact of feedback on
students’ learning has been well documented (Alqassab, Strijbos, & Ufer,
2017, 2018; Carless, 2015; Carless & Boud, 2018; van Ginkel, Gulikers,
Biemans & Mulder, 2017; Hyland & Hyland, 2006; Ruegg, 2015; Yao,
Guo, Li & McCampbell, 2020; Xu & Carless, 2017). Feedback involves
the learners’ appreciation of the information from a variety of sources
and then using it to polish their work (Carless, 2015; Carless & Boud,
2018). Teachers and peers become the major sources of information, and
the feedback they provide facilitates students’ effective learning through
multiple revisions (Carless & Boud, 2018; Yao et al., 2020). With the
mass expansion of higher education coupled with limited resources, the
feedback processes are difficult to implement effectively (Evans, 2013).
One solution to the problem is to shift from teacher feedback to peer
feedback (Boud & Molloy, 2013; Price, Handley, & Millar, 2011) to in­
crease the utilization of feedback. As a result, an increasing number of
researchers in higher education focused on applying peer feedback to
the instruction of diverse subject areas including science, math, English,
and English as a second/foreign language (ESL/EFL); and they often
feedback (Alqassab et al., 2017; Alqassab et al., 2018; van Ginkel et al.,
2017; Hyland & Hyland, 2006; Ruegg, 2015; Yu & Lee, 2016a, 2016b,
2016c; Yao et al., 2020).
ESL/EFL researchers stated that teacher feedback plays a dominant
role in students’ English writing process (Hyland & Hyland, 2006; Niu &
Zhang, 2018). Although students consider teacher feedback to be more
useful and reliable to revise drafts, they are also willing to have peer
feedback together with teacher feedback to increase the diversity of
feedback about their writing (Lei, 2017; Yao et al., 2020). Peer feedback
provides a good opportunity of learning and useful information for
improvement in the EFL writing classroom for both peer feedback givers
and receivers (Berg, Admiraal, & Pilot, 2006; Chang, 2015). By
analyzing the peers’ writing, the feedback givers have an opportunity to
understand their own process of EFL writing development and get new
perspectives about ESL/EFL writing (Yu, 2019; Zimmerman & Kitsantas,
2002). Similarly, the feedback receivers could in turn integrate the peer
feedback to improve the capability to detect, diagnose, and remedy their
writing problems (Yu & Hu, 2017; Yu & Lee, 2016a).
Furthermore, peer feedback is found to be a helpful complementary
aid to teacher feedback in ESL/EFL writing classrooms (Gibbs & Simp­
son, 2004; Hu & Zhang, 2014; Wang, 2014; Yao et al., 2020). For one

* Correspondence to: The Evidence-based Research Center for Educational Assessment, Jiangsu University, Zhenjiang, Jiangsu Province 212013, China.
E-mail address: huangniagara@hotmail.com (J. Huang).
1
Joint first author.

https://doi.org/10.1016/j.stueduc.2022.101140
Received 10 May 2021; Received in revised form 30 January 2022; Accepted 27 February 2022
Available online 6 March 2022
0191-491X/© 2022 Elsevier Ltd. All rights reserved.
W. Wu et al.
thing, peer feedback boasts some advantages teacher feedback does not
have. Unlike teacher feedback on which students passively depend to
improve the quality of the writing, peer feedback activities provide
students with an audience (Caulk,1994), involves students actively in
the feedback process and gives them more control and autonomy
(Mendoca & Johnson, 1994), which makes writing a more authentic task
(Caulk,1994). In addition, students always believe that their similar
learning experiences allow them to better understand each other’s dif­
ficulties, which reduces their fear and pressure as writers and reviewers
and help them to develop their self-confidence (Curtis, 2001; Cotterall &
Cohen, 2003). For another, peer feedback can solve practical problems
in real classroom settings. Given that teacher feedback can sometimes
delay in large-size classrooms, peer feedback could be its complement or
even replacement so that students receive timely feedback for writing
improvement (Min, 2006; Zhao, 2010).
Actually, many college English classrooms are large in Chinese
higher education; as a result, most English teachers are supposed to
teach two or three classes of over 50 students in each class in one se­
mester (Hu & Zhang, 2014). Under this circumstance, it is challenging
for classroom teachers to give all the students timely feedback about
their English writing. Without such timely feedback, students will be less
motivated to develop and improve their writing skills (Gibbs & Simpson,
2004; Hu & Zhang, 2014). Therefore, it is important to evaluate peer
feedback to see if it is an effective complementary aid to teacher feed­
back in college English writing classrooms in China. This study could
provide some useful insights into EFL writing classrooms for other
countries such as Japan and Iran, which also emphasize EFL learning
and many researchers from these countries had examined peer feedback
in the writing classrooms (Azarnoosh, 2013; Allen & Mills, 2014;
Rahimi, 2013; Ruegg, 2015; Séror, 2011; Tsui & Ng, 2000).
2. Research problem
This study aimed to examine whether Chinese college students are
able to provide reliable and valid feedback to their peers’ essay writing
as their English teachers often do, what challenges students face in
giving feedback to their peers’ essays, and what benefits students
perceive they can get from this practice. Given the fact that rubric-based
scoring, an important part of writing assessment, is informative about
the quality of a piece writing (Jonsson & Svingby, 2007; Patchan,
Schunn, & Clark, 2018), the term “feedback” in this study is opera­
tionally defined as both the holistic scores and comments provided by
classroom teachers and peers that would lead to the improvement of
Chinese college students’ English writing. Holistic scoring is the quan­
titative evaluation and comments are the qualitative evaluation of a
student’s writing (Patchan et al., 2018). The two compose a complete
assessment of the student’s writing. A close investigation into both offers
adequate evidence about whether peer feedback could be an effective
complementary aid to teacher feedback in Chinese college English
writing classrooms.
3. Literature review
3.1. Efficacy and benefits of peer feedback
Educational researchers have examined the efficacy and benefits of
peer feedback in the domains of mathematics, science, English, and ESL/
EFL (e.g., Alqassab et al., 2017; Chang, 2015; Hansen & Liu, 2005;
Huisman, Saab, van Driel & van den Broek, 2018; Rahimi, 2013; Wang,
2014; Yu, 2019). For example, Alqassab et al. (2017) examined peer
feedback in the domain of mathematics and reported that the feedback
given by students with high and medium domain knowledge stimulated
their peers to act and reflect on their learning, whereas those low in
domain knowledge provided more peer feedback on accuracy of the
solution or the content knowledge used to solve the task.
Researchers in the domain of ESL/EFL found that students were able
2
Studies in Educational Evaluation 73 (2022) 101140
to generate peer feedback on both global and local issues regarding
peers’ essays in accordance with a rubric as long as systematic training
or teacher modeling was offered (Chang, 2015; Rahimi, 2013; Wang,
2014). Feedback givers’ comments can facilitate feedback receivers to
perfect their writing from the first drafts to their final drafts (Hansen &
Liu, 2005; Huisman et al., 2018). However, the feedback givers’ im­
provements in their writing were manifested in all aspects of content,
structure, and style (Huisman et al., 2018).
Furthermore, several studies indicated that providing feedback
benefited students more than receiving feedback (Cho & MacArthur,
2011; Hislop & Stracke, 2017; Lundstrom & Baker, 2009; Yu, 2019). For
example, after analyzing writing samples at the beginning and end of
one academic semester, Lundstrom and Baker (2009) found that ESL
university students who offered feedback improved their writing more
than those who just received feedback. Further, Cho and MacArthur
(2011) presented that the students who reviewed, rated, and com­
mented on three sample papers performed better in a subsequent writing
task than those who just read the same sample papers without rating and
commenting on the papers, indicating the importance of providing
feedback. Several years later, Hislop and Stracke (2017) claimed that
reading their peers’ essays gave them an opportunity to get new per­
spectives, raised their awareness of their own writing shortcomings, and
developed their writing skills. Similarly, Yu (2019) found that post­
graduates, as feedback givers, raised their awareness of the thesis genre,
improved their writing skills, and became more skillful learners and
more reflective academic writers.
To optimize the efficacy and benefits of peer feedback, factors
affecting its effectiveness were explored by many researchers. The
following is a brief review of these factors.
3.2. Factors affecting the effectiveness of peer feedback
Many factors are found to affect the effectiveness of peer feedback (e.
g., Allen & Mills, 2014; Hu, 2005; Liu & Hansen, 2002; Min, 2006; Tsui
& Ng, 2000; Wu, 2019; Zhao, 2010). First, training can have a positive
impact on students’ peer feedback performance. Peer feedback is usually
a new practice to most students. They are often not confident about their
ability to offer feedback on the work of a peer and wish to have more
support (Alqassab et al., 2017). In the training process, clarifying the
goals and benefits of peer feedback, designing peer feedback guidance
sheets, and modeling different types of feedback and making specific
suggestions were found to be effective in helping students’ peer feed­
back performance (Chang, 2015; Hu, 2005; Liu & Hansen, 2002).
Notably, peer feedback guidance sheets are expected to involve items
used to help students succeed in providing feedback on all aspects of a
peer’s work. Besides, error-prone grammar and vocabulary points
should be clearly listed down in the guidance sheets and elaborately
explained through examples since domain knowledge scaffolds can help
students provide better peer feedback (Alqassab et al., 2017). Without
appropriate training, students might not be able to offer useful feedback
or to distinguish useful from useless peer feedback and revise their
writing accordingly (Min, 2006).
Second, the cultures in which students learn and grow up affect the
effectiveness of peer feedback. Students from teacher-centered cultures
tend to believe that their peers are not competent enough to comment
their essays and will doubt their suggestions (Tsui & Ng, 2000; Zhao,
2010). Students from collectivist cultures are more likely to avoid giving
critical comments on their peers’ essays to maintain harmonious per­
sonal relationship (Nelson & Carson, 1998; Liu & Hansen, 2002). Ac­
cording to Hu and Lam (2010), however, adult Chinese students were
able to give and incorporate many helpful suggestions about each
other’s writing; furthermore, most of them were willing to have peer
suggestions as a valuable type of feedback; and finally, most of them
claimed that peer feedback did not conflict with Chinese cultural beliefs
and practices. Therefore, it can be an effective pedagogical tool in Chi­
nese writing classrooms.
W. Wu et al.
Third, students with different domain knowledge levels provide peer
feedback on different aspects (Alqassab et al., 2017). For instance, stu­
dents with high and medium domain knowledge on geometric con­
struction provided more peer feedback on related tasks at the
self-regulation level (i.e., feedback stimulating peers to act and reflect
on their learning), whereas those low in domain knowledge provided
more peer feedback at the task level (i.e., feedback on the correctnes­
s/incorrectness of the solution or the content knowledge used to solve
the task) (Alqassab et al., 2017). When it comes to peer feedback in
language studies, ESL/EFL students’ English proficiency levels affect the
effectiveness of peer feedback. Research shows that high English profi­
ciency (HEP)students offer more suggestions and are more actively
engaged in peer feedback activities than low English proficiency (LEP)
students, especially when the writer is poor in English proficiency (Allen
& Katayama, 2016; Allen & Mills, 2014; Lundstrom & Baker, 2009). One
possible reason is that HEP students tend to play the role of “expert” in
the activities while the LEP peers are more likely to take on the role of
“novice” (Allen & Katayama, 2016). Besides, LEP students are less
capable of giving feedback on their HEP peer’s writing (Allen & Mills,
2014). Wu (2019), nevertheless, discovered that LEP students performed
better in peer feedback because they were given enough time and
allowed to use the first language (L1). One explanation for this
discrepancy may lie in the different implementation of peer feedback
adopted in these studies.
Although Allen and Mills (2014) found no significant difference in
the type of feedback made by the two proficiency groups, Allen and
Katayama (2016) and Wu (2019) revealed that HEP reviewers tended to
give more language-related feedback rather than content feedback while
LEP reviewers did the opposite. This finding was confirmed by Lund­
strom and Baker (2009), who reported that LEP reviewers made more
progress in global aspects like content and organization than in local
aspects like grammar and vocabulary after one-semester practice of
feedback-giving to their peers’ essays.
Fourth, students’ perceptions can impact their engagement in peer
feedback process, which in turn influences the effectiveness of peer
feedback activities (Yao et al., 2020). For example, peer feedback pro­
vided by 53 preservice mathematics teachers demonstrated that feed­
back givers’ perceptions of their feedback message positively predicted
peer feedback accuracy (Alqassab et al., 2018). Amores (1997) indicated
that students who perceived themselves as lower in proficiency than
their peers tended to accept their peers’ suggestions uncritically;
moreover, students who perceived themselves as higher in proficiency
were more likely to give more feedback on their peers’ writing. Hu and
Lam (2010) further stated that students who perceived that their English
proficiency would affect their peer feedback behaviors offered fewer
suggestions for their peers’ essays and incorporated fewer suggestions in
their revisions than those who did not perceive so. Yu and Lee (2016b)
revealed that LEP students who regarded peer feedback as a useful
learning practice felt more motivated to engage themselves in peer
feedback activities, and they could also contribute to their group
members in peer feedback process. According to Yu and Lee (2016b),
HEP students who viewed peer feedback as an activity only bringing
rewards to feedback receivers would be less motivated to engage in the
activity seriously than those who viewed peer feedback as a learning
activity primarily benefiting feedback givers.
In fact, the effectiveness of peer feedback was not affected by a single
factor (Wu, 2019; Yu & Hu, 2017). Yu and Hu (2017) examined two
Chinese EFL university students’ peer feedback practice and revealed an
abundance of factors affecting the practice, including such individual
factors as beliefs in good writing, motives and goals, previous learning
and feedback experience, as well as such contextual factors as teacher
feedback practice, feedback training, group dynamics (e.g., positive or
negative comments, interpersonal relationships), and the learning and
feedback culture. This could account for the different findings in the
above studies.
Researchers often examine the effectiveness of peer feedback in
3
Studies in Educational Evaluation 73 (2022) 101140
contrast to teacher feedback (e.g., Lee, 2015; Lei, 2017; Séror, 2011;
Yang, Badger, & Yu, 2006). The following section briefly reviews the
literature on the effectiveness of peer in contrast to teacher feedback.
3.3. Effectiveness of peer versus teacher feedback
The effectiveness of peer versus teacher feedback could be different
as perceived by students (Carson & Nelson, 1994; Chen, 2010; Jacobs,
Curtis, Braine & Huang, 1998; Nelson & Carson, 1998; Lee, 2015; Lei,
2017; Yang et al., 2006; Zhang, 1995). For example, Zhang (1995)
compared students’ views on peer and teacher feedback and reported
that over 90% of his EFL students preferred teacher feedback to peer
feedback. His finding was supported by other researchers. For example,
Carson and Nelson (1994) and Nelson and Carson (1998) indicated all
the participants in their studies favored teacher feedback since they
assumed that teachers were more proficient in English language and
thus could offer more helpful feedback than peers. Yang et al. (2006)
further confirmed that Chinese EFL students regarded teacher feedback
as more important than peer feedback although they acknowledged the
importance of peer feedback. Recent studies (Séror, 2011; Lee, 2015;
Lei, 2017) demonstrated that students regarded teacher feedback as
more useful and reliable than peer feedback to revise their writing, but
they were also willing to have peer feedback to increase the amount and
diversity of feedback on their writing.
However, Jacobs et al. (1998) found that their EFL students from
Hong Kong and Taiwan preferred to have peer feedback on their writing.
Similarly, Chen (2010) reported that his Taiwanese master’s students
felt quite positive about providing and receiving peer feedback in their
English writing. Whether students were forced to make a choice between
peer and teacher feedback might explain the different results of the
studies above (Jacobs et al., 1998). Peer and teacher feedback should
not be mutually exclusive (Jacobs et al., 1998), and students would
embrace both types of feedback when they were not asked to make a
choice between the two (Tsui & Ng, 2000).
Furthermore, the feedback provided by peers and teachers focuses on
different aspects and thus produces diverse effects on the improvement
of the work or on their ability. A quasi-experimental study (van Ginkel
et al., 2017) examined the effectiveness of peer, teacher, and
self-feedback on 144 first-year undergraduate students’ progression in
oral presentation skills, knowledge of the presentation, and attitudes
towards the presentation. Results demonstrated that teacher feedback
encouraged students’ presentation skills, while all the three feedback
sources significantly developed students’ knowledge and attitudes to­
wards presentation.
When the revisions made by students in their writing redrafts were
investigated after they received peer or teacher feedback, it was found
that students incorporated more teacher feedback in their redrafts
(Connor & Asenavage, 1994; Paulus, 1999;Tsui & Ng, 2000; Yang et al.,
2006; Zhao, 2010). According to Zhao (2010), students integrated more
teacher feedback not because they fully understood teacher feedback
but because they considered teacher feedback to be more trustworthy
than peer feedback. Incorporating feedback without fully understanding
would not facilitate students’ development of writing ability. Instead,
the application of L1 enabled students to easily understand peer feed­
back and integrating such peer feedback in redrafts could promote stu­
dents’ writing development. This study highlighted the value of peer
feedback (Zhao, 2010). Moreover, students made more meaning-level
revisions (i.e., idea-related) after receiving peer feedback and more
surface-level revisions (i.e., grammar-related) after receiving teacher
feedback (Connor & Asenavage, 1994; Yang et al., 2006). However, they
did not clarify whether different types of revisions were resulted from
different types of feedback.
Recently, Ruegg (2015) conducted a one-year study in her EFL
writing classes to examine the relative effects of peer and teacher
feedback on students’ writing ability. She divided her students into peer
and teacher feedback groups. Pre- and post-writing tests showed that
W. Wu et al.
there was no significant difference between gains for organization, vo­
cabulary, content, or the total essay scores; the teacher feedback group,
however, gained significantly more in grammar scores than the peer
feedback group. Feedback analysis demonstrated that teacher feedback
was significantly more related to meaning-level issues and content. The
analysis results of writing tests and feedback seemed contradictory but
made sense according to other researchers (e.g., Biber, Nekrasova, &
Horn, 2011), who stated that feedback on both content and form was
likely to enhance grammatical accuracy, while feedback focusing on
language form alone did not.
More recently, Dressler, Chu, Crossman, and Hilman (2019) exam­
ined the quantity and quality of uptake of surface-level and
meaning-level feedback provided by peers and a teacher on academic
writing assignments in an online graduate-level research course. Ana­
lyses of the students’ drafts and feedback they received indicated that
compared with the teacher, students were more likely to give
surface-level feedback while teachers gave more meaning-level feed­
back. That was mainly because peers viewed the writing first and offered
much surface-level feedback, and thus the teacher did not necessarily
repeat comments on that aspect. Besides, students incorporated slightly
higher percentage of teacher feedback than peer feedback. Further,
students dealt with surface-level feedback a little more often than
meaning-level feedback. The reason behind it was that it was easy to
incorporate surface-level feedback while for meaning-level feedback,
referring to the literature, rewriting a section, or reorganizing the paper
(Baker, 2016) might be involved, which was troublesome and chal­
lenging. Based on the findings, it was concluded that training should be
enhanced so that students were more capable of providing and incor­
porating meaning-level feedback.
Most recently, Tian and Zhou (2020) investigated five Chinese stu­
dents’ engagement with automated, peer and teacher feedback in an
online EFL writing course and explored the driving force behind their
decision-making in integrating these sources of feedback into their
redrafts. The analyzes of their essay drafts indicated that the automated
writing evaluation (AWE) offered the most feedback in total but was
least incorporated by the five students; however, the results for the
teacher feedback were just the opposite of the automated feedback; and
the results for the peer feedback were between them. More specifically,
the AWE gave surface-level but not meaning-level feedback; the peers
gave twice of the surface-level than the meaning-level feedback; how­
ever, the teachers gave twice of the meaning-level than the surface-level
feedback. Further, students incorporated more automated feedback on
grammar and mechanics, more peer feedback on the lexical meaning
and grammar, but more teacher feedback on the sentence and paragraph
levels in their redrafts.
3.4. Research gaps
In view of the studies above, this study is significant in the following
ways. First, given the contradictory results of the effectiveness of peer
versus teacher feedback and limited research on the effectiveness of peer
in contrast to teacher feedback in the EFL writing classrooms, more
research is needed to find out on what aspects of an essay EFL students
are able to provide feedback to their peers’ writing as their English
teachers often do. Unlike previous studies (e.g., Paulus, 1999; Ruegg,
2015; Zhao, 2010), this study would directly compare the effectiveness
of peer in contrast to teacher feedback on the same essay simulta­
neously. Second, given that the existing literature mainly focused on the
qualitative comments offered by peers, more studies are needed to
investigate the variability and reliability evidence of quantitative peer
feedback in terms of holistic scoring since rubric-based scoring is
informative about the quality of a piece writing (Patchan et al. 2018)
and thus able to reflect effectiveness of peer versus teacher feedback.
Finally, given the large writing classes, uneven development of college
students’ English writing skills as well as writing teachers’ hesitation to
incorporate peer feedback into their teaching process in China (Yu,
4
Studies in Educational Evaluation 73 (2022) 101140
2019; Yu & Lee, 2016b, 2016c), more research is needed to explore ways
in involving students in peer feedback tasks effectively in the EFL
writing classrooms.
3.5. The present study
This study aimed to examine whether Chinese college students are
able to provide reliable and valid feedback to their peers’ essay writing
as their English teachers often do. The following five research questions
guided the study: 1) what is the variability of the holistic scores assigned
to English essays by peer in contrast to teacher feedback givers? 2) What
is the reliability of the holistic scores assigned to English essays by peer
in contrast to teacher feedback givers? 3) How valid are the written
comments on EFL essays made by peer in contrast to teacher feedback
givers? 4) What are peer feedback givers’ major challenges in providing
feedback on peers’ English essays? And 5) what are peer feedback
givers’ perceived benefits of peer feedback in writing English essays?
The purpose of the last two questions was to further provide evidence for
and explain the results of the peer feedback givers’ written comments.
4. Methodology
4.1. Methodological considerations
Due to the nature of peer feedback on EFL writing, both quantitative
and qualitative approaches were adopted in the design. The variability
and reliability of holistic scores assigned by peer in contrast to teacher
feedback givers were investigated quantitatively by using generaliz­
ability (G-) theory (Cronbach, Gleser, Nanda & Rajaratnam, 1972) as a
guiding framework. This is because G-theory has become a powerful
quantitative approach for detecting the score variability and reliability
(Brennan, 2001; Zhao & Huang, 2020; Liu & Huang, 2020; Shavelson &
Webb, 1991); further, it provides a comprehensive conceptual and
methodological framework for analyzing more than one measurement
facet simultaneously in investigations of score variability and reliability
(Huang, 2008, 2012; Huang & Foote, 2010; Brennan, 2001).
In addition, the qualitative approach was used to analyze peer and
teacher feedback givers’ written comments on the four aspects of con­
tent, format, language, and organization of the selected EFL essays as
well as peer feedback givers’ responses to the open-ended questions
about their major challenges in providing feedback on peers’ English
essays and perceived benefits of peer feedback in writing English essays.
The purpose of including these qualitative investigations was to provide
in-depth information about the validity of peer feedback in the context
of writing assessment of the EFL classrooms.
4.2. Research context and participants
The study was conducted at a university in northern China. At the
university, non-English majors were placed into three levels (i.e.,
advanced, intermediate, and elementary) based on their performance in
the English language placement test before they started to take the two-
year compulsory college English course, with two sessions of 100-minute
classes each week for 16 weeks each semester. It is important to note
that there was variation in students’ English proficiency within each
level.
The non-English major students who participated in this study were
in their second semester of the second year and were preparing for the
College English Test Band-4 (CET-4), one of the national high-stakes
standardized English proficiency tests for non-English major students
administered by the Department of Higher Education of the Ministry of
Education in China (National College English Testing Committee, 2016).
Given the nature of the CET-4, both norm-referenced and
criterion-referenced frameworks are evident (Li & He, 2015). One of the
teaching objectives in that semester was to facilitate students to meet the
writing requirement of the CET-4, i.e., being able to write a 120-word
W. Wu et al.
essay or practical writing based on a given outline, picture, situation or
title, with ideas accurately expressed, sentences well connected, and no
serious grammatical errors. During this specific semester, these students
were required to write one CET-4 essay every two or three weeks on an
automatic writing assessment platform in China.
Before the deadline, the students could revise their writing based on
the comments given by the automatic assessment platform (comments
are usually on some local issues such as spelling, grammar, and vocab­
ulary) for as many times as they wanted. After the deadline, the teacher
selected a few students’ writing samples to instruct students how to
organize the structure of an essay, what to be included in the intro­
duction, body, and conclusion parts, how to connect paragraphs and
sentences with transitional words/sentences, connectives or signal
words, how to make the ideas clearer and more convincing, and how to
improve grammar and vocabulary accuracy. Students were suggested to
revise their essays after the teacher’s instruction. Due to the large
number of students and their writing samples, it was challenging for the
teacher to provide each student with specific comments on each writing
sample. Thus, the teacher intended to encourage students to get
involved in the feedback providing process to explore the effectiveness
of peer feedback provided for their peers’ essays.
For the convenience of the research, the first researcher, a college
English teacher at this university, invited all her three intermediate level
college English classes (118 students altogether) to write one short En­
glish essay (see Appendix A) in a normal class session within 30 min
required by the CET-4. The writing task was the authentic writing item
used in the previous CET-4 administration.
In order to better focus on the research objective (i.e., evaluating the
reliability and validity of peer feedback as an aid to teacher feedback in
the EFL writing classrooms), this study did not involve all 118 students
and their essays in the study. Instead, only 30 representative writing
samples and 30 peer feedback givers were selected for this study.
4.3. Research procedures
4.3.1. The selection of writing samples
After the 118 essays written by all students were collected, the first
researcher, together with other researchers who are experienced in EFL
writing assessment, reviewed all the 118 essays and then selected a total
of 30 essays for this study. Since the first researcher was also the teacher
of the participants in the study and she knew them well, she completed
the selection task together with other researchers to avoid the potential
bias functioning in being both a teacher and a researcher and also to
make sure the essays selected represented a variety of different qualities.
Although these 30 essays were similar in length, they were evaluated
and carefully selected in terms of the structure and discourse clarity,
argument sufficiency, as well as the lexical and grammatical accuracy.
4.3.2. The selection of feedback givers
Students’ writing proficiency was considered in the selection of peer
feedback givers. From the three participating intermediate level EFL
classes, 30 students with slightly different English writing proficiency
levels (as measured by their writing scores in previous writing tasks and
final examinations) were invited to serve as peer feedback givers of this
study. None of them had the experience providing feedback to their
peers’ writing. Among them, ten majored in printing and packaging
engineering, ten in information engineering, and the remaining ten in
business management. Twenty-one of them (70%) were females, and
nine of them (30%) males. Each peer feedback giver received a financial
reward for this feedback activity due to the massive amount of work
required. It was worth noting that peer feedback givers did not know to
whose essays they gave feedback since all the writers’ names were
replaced by number codes.
Furthermore, the first researcher invited four of her colleagues to
serve as teacher feedback givers. All the four teachers were females.
Three of them had been teaching college English reading and writing in the
5
Studies in Educational Evaluation 73 (2022) 101140
institution for over 15 years and one for over ten years. All of them had
served as the official CET-4 writing raters.
4.3.3. The training of feedback givers
The training of the peer and teacher feedback givers was conducted
online separately, following the same procedures. The feedback giver
training was essential to obtain reliable results. Therefore, both peer and
teacher feedback givers were thoroughly trained so that they could
apply the same standards to their feedback of the EFL essays (Han &
Huang, 2017). The following was a detailed description of the peer
feedback givers’ training.
Before they assessed the writing samples independently, all feedback
givers had received three-hour formal training administered by the first
researcher. The purpose of the study was first described to the feedback
givers. Further, the CET-4 scoring criteria (see Appendix B) was
reviewed and explained by the trainer using the CET-4 writing samples
of different qualities. Following that, the feedback givers were asked to
assess four writing samples independently and then discussed their
feedback outcomes until they had finally reached an agreement. As soon
as the training session was completed, all feedback givers were invited to
assess the selected 30essays independently, first assigning a holistic
score to each essay, and then providing approximately 100-word written
comments in Chinese on each essay.
To avoid low reliability caused by exhaustion, both peer and teacher
feedback givers were suggested to assess three essays after studying the
rating criteria each day. The 30 essays were dispersed to peer and
teacher feedback givers three times, with 10 each time, to prevent them
from rushing to assess the essays within a limited time before the
deadline, with the reliability sacrificed.
4.3.4. Data collection
As shown in Fig. 1, all 30 peer feedback givers and four teacher
feedback givers were invited to quantitatively and qualitatively assess
the 30 essays independently first, and then peer feedback givers were
invited to answer follow-up open-ended questions about their percep­
tions of peer feedback in the EFL writing classrooms (see Appendix C).
The data collection involved two steps. The first was for peer and teacher
feedback givers to assign each essay a holistic score on a 15-point scale
and to provide approximately 100-word Chinese written comments on
the essay. The quantitative feedback resulted in 30 independent samples
of EFL essays or EFL papers (student papers, p), scored by 30 peer
feedback givers (peer feedback giver, r1) and four teacher feedback
givers (teacher feedback giver, r2). The holistic scoring strictly followed
the authentic CET-4 scoring criteria (see Appendix B). The written
comments were made on the content, format, language, and organiza­
tion aspects of each essay. The second step was for peer feedback givers
to answer the open-ended questions, which were used to examine their
challenges in providing feedback on peers’ English essays as well as their
Fig. 1. Data collection process.
W. Wu et al.
perceived benefits of peer feedback in English writing classrooms.
4.3.5. Data analysis
The G-theory computer program GENOVA (Crick & Brennan, 1983)
was used to analyze the quantitative data. Within G-theory framework,
data were analyzed in the following three stages: a) a person2-by-peer
feedback giver (p × r1) random effects G-study for peer feedback givers;
b) a person-by-teacher feedback giver (p × r2) random effects G-study;
c) two person-by-feedback giver random effects D-studies for peer
feedback givers (P × R1) and teacher feedback givers (P × R2), respec­
tively. The variance components obtained from these G-studies were
used to examine the variability of the holistic scores assigned to the 30
EFL essays. The D-studies results were used to examine the reliability of
the writing scores assigned to these 30 EFL essays by peer in contrast to
teacher feedback givers.
It is important to mention that many other facets could also
contribute to the score variation, e.g., essay quality, peer feedback
giver’s English writing proficiency level, gender, major, and their in­
teractions. Since the focus of this study’s G-theory analyses was to make
comparisons of their score variability and reliability between peer and
teacher feedback givers as raters, the rater facet then became the most
important facet that we are interested in investigating.
Further, qualitative written comments were first coded and sorted,
then organized, and finally grouped and categorized according to the
recurring themes (Creswell, 2014). Specifically, the written comments
were coded by the first, third, and fourth researchers independently as
valid and invalid. Valid comments were expected to improve the quality
of the writing once incorporated. With the rating criteria of CET-4
writing item for reference, a four-level coding scheme was used to
code the comments on content, format, language, and organization.
Each of the four levels is further divided into several detailed facets. In
the case of problematic coding, the three coders negotiated with each
other until an agreement was reached.
After the coding process, the data were put into Excel to calculate the
total number of recorded comments and valid comments on each facet
assigned to the 30 essays by each peer and teacher feedback giver. Then
descriptive analyses were conducted to examine the quantitative dif­
ferences in the written comments on each level between peer and
teacher feedback givers. Further, the quantity of valid written comments
on each level given by peer and teacher feedback givers, respectively,
were compared with the number of written comments agreed by the four
researchers. They were considered as the gold standard because they
have similar background to the four teacher participants; more impor­
tantly, they had worked together to make comments on each essay and
reached an agreement in terms of its strengths and weaknesses. It is
assumed that the researchers’ group work would have been more valid
than the four teacher participants’ individual work. Further, sample
essays with written comments were analyzed for similarities and dif­
ferences between peer and teacher feedback givers.
Finally, peer feedback givers’ responses to follow-up open-ended
questions were analyzed in a similar manner. The data were entered into
an Excel spreadsheet by the four researchers who have rich experience in
qualitative data analysis including coding. After that, they indepen­
dently color-coded the responses under each open-ended question and
sorted them into different categories, and then grouped and categorized
the conceptually similar responses collaboratively according to the
recurring themes (Creswell, 2014).
2 a
Since each person (i.e., p, the object of measurement) in this study
completed only one writing task (i.e., essay), it was impossible to consider task
as a facet of analysis in these G-studies.
6
Studies in Educational Evaluation 73 (2022) 101140
5. Results
5.1. RQ1: what is the variability of the holistic scores assigned to English
essays by peer in contrast to teacher feedback givers?
To answer the first research question, the following two G-studies
were conducted: a) a person-by-peer feedback giver (p × r1) random
effects G-study for peer feedback givers; and b) a person-by-teacher
feedback giver (p × r2) random effects G-study for teacher feedback
givers. Each G-study yielded the following three variance components:
person (p), feedback giver (r), and person-by-feedback giver (pr). The
results are presented in Table 1.
As presented in Table 1, the results for peer feedback givers show
that the residual yielded the largest variance (44.56% of the total vari­
ance). The residual contains the variability due to the interaction be­
tween peer feedback givers, person, and other unexplained systematic
and unsystematic sources of error. The person (p), the object of mea­
surement, yielded the second largest variance component (43.23% of
the total variance), suggesting that the 30 college students were quite
different in their written abilities as measured by the writing task. The
feedback giver (r1) variance component yielded the third largest vari­
ance component (12.21% of the total variance), which demonstrates
that the 30 peer feedback givers differed greatly from one another in
terms of leniency of scoring the 30 EFL essays.
Also as shown in Table 1, the person (p), the object of measurement,
yielded the largest variance component (64.85% of the total variance),
indicating that the 30 college students were extremely different in their
writing abilities. The residual yielded the second largest variance
component (28.43% of the total variance). Again, the residual contains
the variability due to the interaction between person, feedback givers,
and other unexplained systematic and unsystematic sources of error.
Teacher feedback giver (r2) yielded the smallest variance component
(6.72% of the total variance), suggesting that the four teacher feedback
givers were only slightly different in terms of their leniency of scoring
these 30 essays.
5.2. RQ2: what is the reliability of the holistic scores assigned to English
essays by peer in contrast to teacher feedback givers?
To answer the second research question, the following two D-studies
were conducted: a) a person-by-peer feedback giver (P × R1) random
effects D-study for peer feedback givers; and b) a person-by-teacher
feedback giver (P × R2) random effects D-study for teacher feedback
givers. Given both the norm-referenced and criterion-referenced nature
of the CET-4, both G- and Phi-coefficients were provided. The results of
both D-studies are presented in Table 2.
As shown in Table 2, the G- and Phi-coefficients obtained for the 30
essays for the current 30 peer feedback givers were.97 and.96, respec­
tively. In real classroom assessment context, where it is possible to have
two to four peer feedback givers involved, the G-and Phi-coefficients can
then range from .60 to .80, suggesting acceptable reliability coefficients.
Table 1
Variance components for random effects p × r1 and p × r2 G-studies.
Feedback giver group Source of variability df σ2 %
Peer feedback givers pa
29 2.1001 43.23
r1 29 0.5928 12.21
pr1 841 2.1648 44.56
Total 899 4.8577 100
Teacher feedback givers pa 29 3.5718 64.85
r2 3 0.3703 6.72
pr2 87 1.5658 28.43
Total 119 5.5079 100.00
Since each person (i.e., p, the object of measurement) in this study completed
only one writing task (i.e., essay), it was impossible to consider task as a facet of
analysis in these G-studies.
W. Wu et al. Studies in Educational Evaluation 73 (2022) 101140

Table 2
Summary of G- and Phi-coefficients for peer and teacher feedback givers.
Numberof peer feedback givers Peer feedback givers Numberof teacher feedback givers Teacher feedback givers

G-coefficients Phi-coefficients G-coefficients Phi-coefficients

1 .50 .43 1 .70 .65

2 .66 .60 2 .82 .79
3 .74 .70 3 .87 .85
4 .80 .75 4 .90 .88
5 .83 .79 5 .91 .90
6 .85 .82 6 .93 .92
7 .87 .84 7 .94 .93
8 .89 .86 8 .95 .94
9 .90 .87 9 .95 .94
10 .91 .88 10 .96 .95
11 .91 .89 11 .96 .95
13 .93 .91 12 .96 .96
15 .94 .92 13 .97 .96
17 .94 .93 14 .97 .96
19 .95 .93 15 .97 .97
21 .95 .94 16 .97 .97
23 .96 .95 17 .97 .97
25 .96 .95 18 .98 .97
27 .96 .95 19 .98 .97
30 .97 .96 20 .98 .97

Also as shown in Table 2, the G- and Phi-coefficients obtained for the
30 essays for the current four teacher feedback givers were .90 and .88,
respectively. In real classroom assessment context, where it is possible to
have only one teacher feedback giver, the G-and Phi-coefficients are .70
and .65, respectively, suggesting barely acceptable reliability co­
efficients in the context of classroom assessment.
5.3. RQ3: how valid are the written comments on EFL essays made by
peer in contrast to teacher feedback givers?
To answer the third question, the comments by both groups were
analyzed both quantitatively and qualitatively. The quantitative results
were presented first, followed by the qualitative results. Table 3 presents
the descriptive statistics (i.e., means and standard deviations) of the
total number of recorded comments and valid comments on the 30 EFL
essays made by peer and teacher feedback givers. It further includes the
percentage of valid comments as well as the percentage of expected valid
comments made by the two feedback giver groups.
As shown in Table 3, with all the comments of the 30 essays counted
altogether, teacher feedback givers made more comments than peer
feedback givers on content(teacher mean = 63, teacher standard
deviation = 19.24; peer mean = 52.2, peer standard
deviation = 14.77), language (teacher mean = 192.25, teacher stan­
dard deviation = 94.5; peer mean = 99.4, peer standard
deviation = 41.87), and organization(teacher mean = 134.25, teacher
standard deviation = 17.25; peer mean = 117.57, pear standard
deviation = 12.57). However, peer feedback givers made more com­
ments than teacher feedback givers on format (peer mean = 14.93, peer
standard deviation = 10.59; teacher mean = 13.75, peer standard
deviation = 12.97). Overall, the descriptive statistical results indicate
that comments made on content, format, and organization did not vary
substantially between teacher and peer feedback givers, whereas
teacher feedback givers made substantially more comments on language
than peer feedback givers.
As also shown in Table 3, the comments made on content, format,
language, and organization by teacher feedback givers were 100% valid,
but the comments made on the same four areas by peer feedback givers
were 92.54%, 86.2%, 76.56%, and 93.17% valid, respectively. The
percentage of valid comments over the total number of expected com­
ments agreed by the four researchers of this study demonstrated that

Table 3
Quantitative results of the written comments.
Total number of expected Feedback giver N Total number of Total number of Percentage of valid Percentage of expected valid
commentsa group commentsb valid commentsc commentsd(%) commentse(%)

Mean SD Mean SD

Content (183) Teacher feedback 4 63.00 19.24 63.00 19.24 100 34.43
givers
Format (94) Peer feedback givers 30 14.93 10.59 12.87 9.83 86.20 13.69
Teacher feedback 4 13.75 12.97 13.75 12.97 100 14.63
givers
Language (802) Peer feedback givers 30 99.40 41.87 76.10 35.92 76.56 9.49
Teacher feedback 4 192.25 94.50 192.25 94.50 100 23.97
givers
Organization (269) Peer feedback givers 30 117.57 12.57 109.54 13.57 93.17 40.72
Teacher feedback 4 134.25 17.25 134.25 17.25 100 49.91
givers
a
Total number of expected comments refers to the total number of comments in the four areas of content, format, language, and organization which should be
identified in the 30 EFL essays as agreed by the four researchers of this study.
b
Total number of comments refers to the total number of comments on the 30 EFL essays made by peer and teacher feedback givers.
c
Total number of valid comments refers to the total number of comments made by peer and teacher peer feedback givers on the 30 EFL essays which were valid.
d
Percentage of valid comments refers to the percentage of valid comments among the total number of comments made by peer and teacher feedback givers.
e
Percentage of expected valid comments refers to the percentage of valid comments among the total number of expected comments agreed by the four researchers of
this study.

7
W. Wu et al.
both teacher and peer feedback merely covered part of the problems of
the essays. Offering 100-word feedback in a separate paragraph instead
of directly in the essays might limit the feedback to comments on more
important problems. Still, there were some differences between teacher
and peer feedback givers in the areas of on content (teacher
percentage = 34.43%; peer percentage = 26.4%), format (teacher
percentage = 14.63%; peer percentage = 13.69%), language (teacher
percentage = 23.97%; peer percentage = 9.49%), and organization
(teacher percentage = 49.91%; peer percentage = 40.72%).
Qualitative results of the written comments on content, format,
language, and organization are presented in the following section. The
comments on sample essays by both feedback giver groups were
compared with the comments agreed by the four researchers to
demonstrate the similarities and differences in the comments made by
the peer and teacher feedback givers on these four aspects of the essay.
The sample essays (or parts of the sample essays) were selected
randomly for the presentation of the typical problems in the aspects of
content, format, language, or organization.
First, comments on content involved whether the essay was pertinent
to the topic, whether adequate arguments were provided to support the
thesis statement or the topic sentence in corresponding paragraph, and
whether the ideas were clearly expressed. For example, comments on
essay Sample #8 (see below) agreed by the four researchers included
that the argument in the fourth paragraph failed to support its topic
sentence, and then a suggestion was offered to either change the topic
sentence or modify the details to ensure the consistency of the para­
graph. The results indicated that only one teacher and one peer feedback
giver pointed it out. Surprisingly, almost half of the peer feedback givers
praised the essay by saying “adequate/well-founded arguments”.
(Essay Sample #8)
Furthermore, the underlined sentences and phrases are not clearly
expressed in this essay, which were identified and suggested for modi­
fication by the four researchers. The results showed that all four teacher
feedback givers provided valid feedback. They either offered general
comments such as “too many invalid expressions lead to unclear ideas,”
“what do you mean by…? What does … sentence mean?” to get the writer’s
attention to the problem, or offered suggestions for modification, or
8
Studies in Educational Evaluation 73 (2022) 101140
directly modified them. Only four peer feedback givers, however,
identified some of the problems and then commented “the idea in para­
graph…is not clear/is hard to understand,” “the ideas in the essay are not
clearly expressed,” and “some sentences in the essay are not easy to under­
stand.” Apparently, peer feedback givers were more implicit in pointing
out idea-expression problems, and they did not offer suggestions for
modifications. Even worse, many peer feedback givers praised this essay
for conveying clear ideas.
Second, comments on format included the spelling, capitalization,
and punctuation errors identified in the essay. For example, comments
on essay Sample #29 (see below) agreed by the four researchers
included three underlined punctuations, and one spelling and two
capitalization errors in the essay as well as the corrections of these er­
rors. The results showed that two teacher feedback givers identified and
corrected all punctuation errors, but no teacher feedback giver com­
mented on the capitalization and spelling errors. In contrast, about ten
peer feedback givers identified and corrected the spelling and capitali­
zation errors, but only four peer feedback givers identified and corrected
one of the punctuation errors.
(Essay Sample #29)
Third, comments on language mainly included both vocabulary and
grammatical errors. For example, comments on essay Sample #18 (see
below) agreed by the four researchers included nine identified vocab­
ulary (circled and numbered from 1 to 9) and seven grammatical errors
(underlined and numbered from A to G) as well as their suggestions for
modification. The results showed that the first teacher feedback giver
just pointed out vocabulary errors #1 and #2 as well as the grammar
error #B without offering suggestions for modification; the second
teacher feedback giver pointed out and corrected three vocabulary (i.e.,
#1, #5, and #9) and two grammatical (i.e., #B and #C) errors; the third
teacher pointed out and corrected all the errors underlined except vo­
cabulary error #4; and the fourth teacher identified and corrected all the
errors underlined but grammatical error #F.
(Essay Sample #18)
In contrast, for an EFL essay with so many language errors, three peer
feedback givers praised it for diverse sentence patterns and vocabularies
in their written comments. Further, a few peer feedback givers made no
W. Wu et al. Studies in Educational Evaluation 73 (2022) 101140

comments on language at all. Most peer feedback givers identified only underlined sentences and suggested changing the sentence order and
one to three vocabulary or grammatical errors highlighted in the essay, add connection words to make it more coherent. Awkwardly, nine peer
with vocabulary errors #2 and #5 as well as grammatical errors #A, #D, raters praised the essay by saying “the sentences in the essay were fluent
#E, and #G most frequently identified and corrected. and logical”.
Finally, comments on organization included sentence coherence,
paragraph organization, and passage structure. The underlined senten­
ces and circled connective phrases in Samples # 24 and # 27 (see below)
were commented as incoherent in the comments by the four researchers.
Then very explicit suggestions for modifications were provided. For
instance, in Sample #24 “so” was suggested to be replaced by “because”,
turning the sentence into a subordinate cause to the former sentence. For (An Excerpt from Essay Sample # 24)
the underlined sentences in Sample #27, it was suggested to move the
sentence “I am more like a child now that I afraid everything new” after
the second sentence of this paragraph and apply connective words like
“so…that” or “and” to make it clearer and more concise. Also, one of the
circled connection phrases is suggested to be modified to make them
parallel.
Most teacher and peer raters praised sample # 24 for its fluent,
coherent and logic sentences, ignoring the problem underlined above.
Only one teacher and one peer rater pointed out the problem in sample
# 24, yet without giving any modification suggestions. As for sample # (An Excerpt from Essay Sample # 27)
27, three teacher and 11 peer raters pointed out the unparallel The four researchers agreed that the essay structure of sample #9
connection phrases and offered ways of modifications; two teacher and (see below) was good for its consistency, with the conclusion corre­
four peer raters made general comments like “influent sentences of the sponding to the thesis statement; however, the essay was not well-
essay” and “lack of connection words between sentences”; only one organized. Specifically, the introduction included some arguments that
teacher and two peer raters identified the specific problem in the should have been in the body of the essay, and the conclusion was too

9
W. Wu et al. Studies in Educational Evaluation 73 (2022) 101140

short to sum up the key ideas. The four researchers also offered several to overcome these challenges. Whenever they encountered difficulties in
specific suggestions for the reorganization of the essay. The results offering feedback, they looked up dictionaries and English grammar
indicated that almost all teacher and peer feedback givers pointed out books, or searched the Internet, or even asked friends, classmates, and
the problems in passage structure and paragraph organizations, but only teachers for assistance. They believed that they had made the greatest
one teacher and 13 peer feedback givers provided one or two sugges­ effort to give their peers more useful feedback.
tions for the improvement of the essay organization. When asked whether their feedback could help peers improve the
(Essay Sample #9) quality of their English essays, 27 peer feedback givers (90%) made a
positive response and three (10%) made a negative response. The three
peer feedback givers who made a negative response admitted that they
5.4. RQs 4 & 5: what are peer feedback givers’ major challenges in
were poor in English writing, which had prevented them from giving
providing feedback on peers’ English essays? And what are their perceived
quality feedback on their peers’ essays. One peer feedback giver who
benefits of peer feedback in writing English essays?
made a positive response commented that “my feedback could only help
peers to improve poor-quality essays but not the good-quality essays.”
To answer the fourth and fifth questions, all 30 peer feedback givers
Another peer feedback giver who made a positive response added that
responded to six open-ended questions. Their responses to the first few
“even if my feedback is invalid, it can be a warning to the peer writer, who will
open-ended questions were used to further prove the qualitative analysis
avoid making a similar error in the future.”
results, i.e., they are not proficient in providing feedback on language
All peer feedback givers believed that they had benefitted from of­
aspects; their responses to the remaining open-ended questions were
fering feedback on their peers’ essays. Their perceived benefits included
used to further explain their perceptions about the benefits of peer
a) the errors their peers made could be a reminder to them in future
feedback as well as their attitudes towards peer feedback, which would
English essay writing practices; b) seeking assistance from all kinds of
increase the validity of the written comments provided by the peer
channels could help further enhance their English proficiency; c) their
feedback givers and further suggest that peer written comments could be
awareness of the importance of essay organization was strengthened
practiced in the real EFL classroom setting. The findings are as follows.
through peer feedback activities; and d) their ability to write essays with
Among the 30 peer feedback givers, 28 of them (93.3%) felt easy to offer
grammatical language and sufficient content could be also increased.
feedback on the content and organization aspects of an essay. They
Taking these benefits into consideration, 26 peer feedback givers
could easily confirm whether an essay was pertinent to the topic and the
(86.7%) expressed their willingness to have peer feedback as a routine
ideas were clear by reading the thesis statement and conclusion as well
activity in their future English writing classrooms. Some of them further
as the major points in the body part. In addition, they could easily find
suggested that the peer feedback activity be practiced in small groups to
out whether an essay was well-organized or not by checking the signal
provide effective feedback to those peers with relatively poor English
words of each paragraph. These results were generally consistent with
writing skills. Further, other peer feedback givers recommended that
qualitative findings of the written comments. Finally, 15 peer feedback
peer feedback should be treated as a complementary to teacher’s feed­
givers (50%) mentioned that they could easily make comments on idea-
back in the college English writing classrooms. However, four peer
expression by examining whether the sentences were fluent or not.
feedback givers (13.3%) objected to incorporating it in their future
However, qualitative analysis of the written comments demonstrated
English writing classes because they thought that the peer feedback was
that only a very few peer feedback givers were able to properly identify
time-consuming although it did yield benefits for feedback givers and
problems on this aspect. Apparently, these peer feedback givers over­
receivers as well.
estimated their ability to give feedback on this aspect.
However, 27 (90%) and 12 (40%) peer feedback givers felt they had
6. Discussion
difficulty in providing feedback on grammar and vocabulary, respec­
tively. They reported to have challenges in identifying the grammatical
6.1. The variability of the holistic scores
and vocabulary (e.g., some Chinglish expressions) errors in an essay;
furthermore, even sometimes they could identify these errors, but they
The first research question attempted to examine the variability of
were not sure about how to modify them. These findings were also
the holistic scores assigned by peer in contrast to teacher feedback
consistent with qualitative findings of the written comments.
givers. The score variation did exist between the two feedback giver
More importantly, 29 peer feedback givers (96.7%) reported that
groups. The desired variance associated with the object of measurement
they had tried their best to give feedback on their peers’ essays in order

10
W. Wu et al.
for the teacher feedback givers was considerably larger than that for the
peer feedback givers. Furthermore, the undesired feedback giver and
residual variance components for the peer feedback givers were almost
twice as much as for the teacher feedback givers. These results suggested
that there were considerably more score variations in peer feedback
givers than in teacher feedback givers. The possible reason is that
teacher feedback givers are definitely more experienced in scoring EFL
essays than the peer feedback givers; further, it was the first time for all
peer feedback givers to score EFL essays following the CET-4 writing.
Further study is needed to find out whether more practice will be
effective in narrowing the score variation between peer and teacher
feedback givers.
6.2. The reliability of the holistic scores
The second question attempted to examine the reliability of the ho­
listic scores assigned to the 30 EFL essays by peer in contrast to teacher
feedback givers. The results suggested that the G- and Phi-coefficients
obtained for the 30 essays for one teacher feedback giver in the real
classroom assessment context were.70 and.65, respectively. Such reli­
ability coefficients could be easily achieved if the number of peer
feedback givers is increased to three. These results indicate that small-
group quantitative peer feedback in Chinese college English writing
classrooms could be as reliable as teacher’s feedback. In other words,
four students are expected to be involved in one group to perform peer
feedback activities in a real classroom setting, with each essay given
feedback by three peers, which is practicable. This was inconsistent with
what other researchers suggested (e.g., Mittan, 1989; Paulus, 1999).
They claimed that pairs were more likely to discuss their writing in a
more intensive way. This would deserve further investigation given that
the conclusion this current study arrived at was based on a quantitative
analysis, and no discussion of the written essays was involved in this
study. Despite a summative purpose, being able to offer a reliable
rubric-based holistic scoring does demonstrate that students have a good
understanding of the criteria, which would help them in giving quality
written comments to their peers’ essays and benefit them greatly in
writing their own essays.
6.3. The validity of the written comments by peer and teacher feedback
givers
The third research question aimed to examine the validity of the
written comments made by peer in contrast to teacher feedback givers.
The quantitative results suggested that all the written comments made
by teacher feedback givers were valid and appropriate, whereas some of
the written comments made by peer feedback givers were invalid and
inappropriate. With this taken into consideration, the total number of
correct comments made by teacher feedback givers on all four aspects of
content, format, language, and organization exceeded the total number
of valid comments made by peer feedback givers, especially on the
language aspect of EFL essays. This did make sense because teacher
feedback givers are more proficient in English; besides, considering that
giving feedback to 30 peers’ essays was a “time-consuming” task to some
peer feedback givers, both the quantity and quality of the peer feedback
could have been negatively influenced.
Surprisingly, this finding was different from the result of the study by
Tian and Zhou (2020). The latter found that peers offered more feedback
than the teacher. The difference might be caused by the different
research designs. In the present study, teachers and peers gave feedback
on the essays simultaneously. In the study by Tian and Zhou (2020),
nevertheless, the teacher provided feedback after the students had
revised their essays based on their peers’ feedback, and thus the teacher
might not find many problems with these essays. Besides, peer feedback
givers in the current study were expected to provide written comments
on 30 essays over a period often days while the participants in the study
by Tian and Zhou (2020) were assigned into peer feedback pairs and
11
Studies in Educational Evaluation 73 (2022) 101140
each was supposed to offer feedback to one piece of essay after each
writing task. Heavy workload might inhibit peers from offering suffi­
cient feedback on the essays. Further, peers’ proficiency might also play
a role. The participants in the study by Tian and Zhou (2020) were from
a privileged university and had passed CET-4, so they were more pro­
ficient than the participants in the present study. High English profi­
ciency (HEP) students did offer more feedback than low English
proficiency (LEP) students (Allen & Katayama, 2016; Allen & Mills,
2014; Lundstrom & Baker, 2009).
Furthermore, peer feedback givers generally performed almost as
well as teacher feedback givers in making comments on most of the
content and organization aspects of EFL essays, with more than 92%
being valid, suggesting that most of the comments on content and or­
ganization made by peer feedback givers were useful and effective. Peer
feedback givers’ comments on format and language, however, were
comparatively lower in quality, with 86.20% and 76.57%being valid,
respectively. These results were similar to the study by Caulk (1994),
who found that 89% of the comments made by the students were valid.
What should be noted is that although teachers offered more valid
written comments than peers, how much feedback receivers could learn
from each source of comments deserves further investigation. Zhao
(2010) found that students accepted teachers’ feedback without fully
understanding while the application of L1 and similar learning experi­
ence allowed students to easily understand peer feedback. In view of
this, feedback receivers benefited more from peer feedback, which
would be more likely to promote students’ writing development.
Despite lower percentage of valid feedback on format and language,
students performed relatively well in offering feedback to their peers’
essays. One probable reason is that students were trained to provide
feedback on their peers’ essays by checking a feedback guidance sheet,
which clearly listed down and exemplified on what aspects of the essays
should be commented. This type of training or scaffold can help students
perform peer feedback better (Alqassab et al., 2017; Min, 2006).
Another possible reason is that peers were paid to assess the essays
(despite a small pay), which might have motivated them to perform the
task more conscientiously. Extrinsic stimuli like compliments, extra
scores, and financial rewards do play a role in motivating students to
perform well in their schoolwork, particularly when it is the first time for
them to do that work and no intrinsic motivation is developed (Cameron
and Pierce, 1994; Lepper, Keavney, & Drake, 1996).
The findings above were also partly consistent with earlier research
which showed that teacher feedback givers gave more feedback on
grammar and vocabulary than peer feedback givers while peer feedback
givers tended to provide overall comments or make comments on global
issues like organization (Chen, 2010); but it conflicted with the finding
by Ruegg (2015), Dressler et al. (2019), and Tian and Zhou (2020), who
stated that teacher feedback givers provided more feedback on
meaning-level issues.
Such a difference could be attributed to students’ different language
proficiencies. The students in the study by Dressler et al. (2019) were
graduates, and the students in the study by Tian and Zhou (2020) were
from a privileged university and had passed CET-4 before the study.
Thus, they might have a higher English proficiency, and there were few
grammar and vocabulary errors worthy of the teacher’s attention. In
Ruegg’s (2015) study, the students were English major sophomores.
Considering the latter spending more time in learning English, they were
more proficient in English and less likely to make more grammar and
vocabulary errors in their writing, which resulted in less feedback on
this perspective. Another possible reason was whether the teacher
shared the same L1 with the students. In the current study, the teachers
shared the same L1 with the students while it was not the case in Ruegg’s
(2015) study. The same L1 could reduce difficulties in meaning-level
comprehensibility (Ruegg, 2015), thus the teachers made no more
feedback on this aspect than peers in this study.
Another noticeable point that might have an impact on the kind of
feedback given is the format required to provide feedback. In the current
W. Wu et al.
study, the feedback was given in written paragraphs rather than track
changes in the essays directly. If feedback givers could offer feedback
directly in the essays, they might be more inclined to correct each
mistake. However, they were asked to provide 100-word written com­
ments on a separate sheet in the study, which confined their feedback to
more important aspects.
According to the qualitative analysis of the written comments, peer
feedback givers had more trouble identifying problems in sentence
coherence, idea-expression, vocabulary, grammar, and punctuation.
Also, some peer feedback givers sometimes identified improper errors in
these aspects. Compared with the teachers, students were less qualified
to give valid comments on language aspects (Ruegg, 2015). These boiled
down to peer feedback givers’ insufficient mastery of English gram­
matical rules and vocabulary knowledge. Due to the L1 influence, they
were not capable of distinguishing words of the same root, like “grad­
uation versus graduate”, “suit versus suitable” in the samples above;
they did not pay attention to diverse verb-noun collocations; they
expressed their ideas in Chinese word order; they frequently use
commas. In view of this, teachers could be suggested to provide feed­
back on sentence coherence, idea-expression, and grammar (or give
students more help in giving feedback on these aspects), while peers
provide feedback on organization and format, which was agreed by
Ruegg (2015).
6.4. The challenges and perceived benefits reported by peer feedback
givers
The fourth and fifth research questions asked about peer feedback
givers’ major challenges in providing feedback on peers’ English essays
and their perceived benefits of peer feedback in writing English essays,
respectively. They had major challenges in first identifying the gram­
matical and vocabulary errors in an essay and then providing the writer
with appropriate suggestions for modification, which was consistent
with previous studies (Chen, 2010, Connor & Asenavage, 1994; Yang
et al., 2006) and was also proved by the qualitative analyses of peers’
written comments. Furthermore, almost all of the peer feedback givers
claimed that they sought assistance from the Internet, dictionary, or
grammar books to make sure their feedback on their peers’ essays was as
valid as possible. As a result, they perceived that their feedback could
help their peers improve essays to some extent regardless of their
imperfection in providing revision suggestions on the language prob­
lems. More importantly, they agreed that offering feedback on their
peers’ essays could also help them enhance their reader awareness,
involving them in self-reflection on and regulation of their own writing,
and thus improve their ability to write English essays with grammatical
language, convincing content, and good organization. This was also
proved by other studies (Berg et al., 2006; Chang, 2015; Wang, 2014;
Yao et al., 2020, Yu, 2019; Yu & Hu, 2017; Zimmerman & Kitsantas,
2002).
Based on these benefits, peer feedback givers stated that they were
willing to have peer feedback as a routine activity of their English
writing class as long as the number of essays assessed was no more than
three or four after each writing task. Such a finding echoed the result of
the quantitative analyses of holistic scores, which suggested that the
scores given to the same essay by three peer feedback givers and the
teacher would be equally reliable. Previous studies also demonstrated
that students were willing to have peer feedback to increase the amount
and diversity of feedback on their writing (Lee, 2015; Lei, 2017; Séror,
2011).
7. Limitations
This study was limited in the following six ways. These limitations
may limit the generalization of the findings to the larger context of
feedback activities in the college English writing classrooms across the
country. First, the student participants only represented intermediate
12
Studies in Educational Evaluation 73 (2022) 101140
proficiency level peer feedback givers. As much research suggested
(Allen & Mills, 2014; Allen & Katayama, 2016; Lundstrom & Baker,
2009; Wu, 2019; Yu & Hu, 2017; Yu & Lee, 2016b), peer feedback
effectiveness is influenced by peers’ proficiency. The results would be
different if students of advanced proficiency level were involved.
Second, the participants were from a single college and taught by the
same English teacher. The results might be different if participants from
different colleges or taught by different teachers were engaged in the
study. Research indicated that the cultures in which EFL students learn
did play a role in their peer feedback practice (Tsui & Ng, 2000).
Third, feedback receivers’ views towards the effectiveness of peer
and teacher feedback and their incorporation of the feedback into their
revisions were not considered in the study. These are important in­
dicators of the effectiveness of peer feedback in contrast to teacher
feedback, especially considering the fact that incorporating feedback
does not necessarily amount to learning and benefiting from that feed­
back (Zhao, 2010). Also, it is necessary to compare the types of feedback
given with the types of revisions made (Ferris, 1997). One type of
feedback does not necessarily lead to the same type of revisions or the
same type of improvement (Biber et al., 2011; Paulus, 1999; Rueggs,
2015).
Fourth, the 30 peer raters of this study did receive three-hour formal
training before assessing these essays. However, it seemed not as
effective as expected since it was conducted online, and a few peer raters
missed some important information due to the unstable network and
some other personal reasons. As suggested in the literature, training is
commonly recognized as an indispensable factor exerting positive
impact on students’ peer feedback effectiveness (Alqassab et al., 2017;
Hu, 2005; Liu & Hansen, 2002; Min, 2006). Moreover, both teacher and
peer raters were trained to give 100-word comments. This hindered
them from giving as many comments as they could. Another important
issue was that since this study was conducted in the last four weeks of
the semester, both teacher and peer raters were so busy that they might
be distracted by exams and papers, with the validity and reliability of the
feedback more or less sacrificed. According to Wu (2019), time avail­
ability is one of the influential factors of peer feedback effectiveness.
Given that this was the first time for peer feedback givers to do the job,
more training should have been given to them to improve the feedback
quality (Min, 2006).
Fifth, this study focused on the peer feedback reliability and validity
as compared with teacher feedback. It might cause misunderstanding
that peer feedback could replace teacher feedback since they do not
interact in an either-or relationship. Such comparison was to provide
evidence for evaluating peer feedback as a reliable and valid comple­
mentary aid to teacher feedback in EFL writing classrooms, which was
exactly the purpose of this study. Therefore, the results of this study
should be interpreted with caution.
Finally, the study was explorative and conducted in a research
setting instead of a real classroom setting. In the current study, peer
feedback givers offered feedback individually while in a real classroom
setting, students are put into a group to assess each other’s essays, and
they have opportunities to discuss and offer feedback to the essays.
Therefore, the results of this study could not be totally generalized to a
real classroom setting.
8. Conclusions
In light of these limitations, the following conclusions were reached.
First, peer feedback givers did a relatively satisfactory job in assessing
the 30 EFL essays, and peer feedback could be applied in EFL writing
class as an effective complementary aid to teacher feedback. The reli­
ability of the holistic scores by up to three peer feedback givers could
reach that of the holistic scores by one teacher, which was the real
classroom assessment context. Besides, most peer feedback givers per­
formed almost as well as teacher feedback givers in most content and
organization aspects despite their difficulties in feedback provision of
W. Wu et al. Studies in Educational Evaluation 73 (2022) 101140

grammar and vocabulary. Most importantly, the analyzes of open-ended
questions demonstrated the students’ willingness to have peer feedback
as a routine activity of their writing class as long as the number of essays
assessed was no more than three or four after each writing task.
Admittedly, the effectiveness of peer feedback depends on students’
positive perceptions of peer feedback to some extent since they facilitate
students’ engagement in the activity (Yao et al., 2020).
Second, there were large residual variance components in the G-
studies, suggesting that other hidden facets were not considered in the
analysis (Brennan, 2001). These possible hidden facets might include
peer feedback givers’ gender, English proficiency, time spent on each
essay, and the quality of each essay (Brennan, 2001; Huang, 2012;
Shavelson & Webb, 1991). More studies are needed in the future to
integrate these facets into the design to minimize the residual variance
components.
9. Implications for practice
The results of this study have important implications for Chinese
college English teachers. They can make full use of peer feedback in the
writing classrooms. The explorative study provided evidence that the
reliability of three peers’ feedback was the same as that of a teacher.
Consequently, teachers can assign four peers to each assessment group
in a real classroom setting because they should have the ability to check
peers’ writing and then provide effective feedback, particularly at the
levels of content and organization. At the same time, teachers could
concentrate on certain global problems such as inadequate arguments
and illogically coherent sentences as well as other local problems such as
unclear expressions, ungrammatical sentence structures, and vocabulary
errors. Above all, teachers should make sure students can fully under­
stand their feedback since only a full understanding the feedback can
promote students’ writing development (Zhao, 2010).
The results of this study have implications for Chinese college stu­
dents as well. They should be actively engaged in peer feedback activ­
ities in their English writing classrooms. It is suggested that peer
feedback givers learn to provide appropriate and effective feedback in
all aspects of an essay at both the global and local levels, which could be
improved through frequent training (Min, 2006). Likewise, peer feed­
back receivers should trust their peers’ feedback and incorporate it in
the revision of English essays, considering the findings of this study and
previous studies (e.g., Yu & Lee, 2016b; Wu, 2019), which showed that
even LEP students could make valuable comments on their peers’
writing with the help of L1, reference tools, enough time availability,
and separation from their peers.
10. Recommendations for future research
Future research is needed to investigate into the effectiveness of four-
person group peer feedback as a reliable and valid complementary aid to
teacher feedback in a real classroom setting. Feedback receivers’ views
towards the effectiveness of peer and teacher feedback and their
incorporation of the feedback into their revisions also need to be studied
in the future since these are important indicators of the effectiveness of
peer feedback in contrast to teacher feedback. Besides, such facets as
peer feedback givers’ gender, English proficiency, time spent on
assessing each essay, and the quality of each essay should be integrated
into the G-theory study design to minimize the residual variance com­
ponents. Above all, training, rather than a once-for-all practice, should
be conducted throughout the entire process to help students practice the
peer feedback activities.
Disclaimers
None.
Source(s) of support
This research did not receive any specific grant from funding
agencies in the public, commercial, or not-for-profit sectors.

Appendix A. The writing task

You are allowed 30 min to write an essay. Suppose you have two options upon graduation: one is to find a job somewhere and the other to go to a
graduate school. You are to make a choice between the two. Write an essay to explain the reasons for your choice. You should write at least 120 words
but no more than 180 words.

Appendix B. The CET-4 writing scoring criteria

The maximum score point of a CET-4 essay is 15, with the following six score ranges specified: 13–15, 10–12, 7–9, 1–3 and 0. The table below
presents the scoring criteria of each score range (Zhao & Huang, 2020, p. 8).

Score Scoring criteria

range

13–15 The essay is/has a) pertinent to the topic, b) clear idea-expression, c) coherent, and d) basically no language errors except a few minor errors.
10–12 The essay is/has a) pertinent to the topic, b) clear idea-expression, c) fairly coherent, and d) several language errors.
7–9 The essay is/has a) basically pertinent to the topic, b) not clear idea-expression in some sentences, c) barely coherent, and d) many language errors with several major
errors.
4–6 The essay is/has a) basically pertinent to the topic, b) unclear idea-expression, c) not coherent, and d) many major language errors.
1–3 The essay is/has a) not pertinent to the topic, b) disorganized idea-expression, c) fragmented sentences, and d) many incorrect sentences with major errors.
0 They essay is/has left blank, or only several isolated words, or not relevant to the topic.

Appendix C. Follow-up open-ended questions about peer raters’ reflections on the assessment activities

1. On what aspect could you give feedback easily? Why was it easy?
2. On what aspect did you have difficulty in offering feedback and why?

13
W. Wu et al. Studies in Educational Evaluation 73 (2022) 101140

3. Did you seek any assistance from the internet, dictionary or other people when giving feedback on your peers’ essays? If yes, did you think it
enabled you to give more effective feedback on your peers’ essays?
4. Do you think your feedback could help your peers improve the quality of their essays? Why or why not?
5. Did you benefit from offering feedback for your peers’ essays? If yes, what were they? If not, why?
6. Are you willing to have peer feedback as a routine activity of your writing class? Why or why not?

References Curtis, A. (2001). Hong Kong student teachers’ responses to peer group process writing.
Asian Journal of English Language Teaching, 11, 129–143.
Dressler, R., Chu, M. W., Crossman, K., & Hilman, B. (2019). Quantity and quality of
Allen, D., & Katayama, A. (2016). Relative second language proficiency and the giving
uptake: Examining surface and meaning-level feedback provided by peers and an
and receiving of written peer feedback. System, 56, 96–106.
instructor in a graduate research course. Assessing Writing, 39, 14–24.
Allen, D., & Mills, A. (2014). The impact of second language proficiency in dyadic peer
Evans, C. (2013). Making sense of assessment feedback in higher education. Review of
feedback. Language Teaching Research, 20(4), 498–513.
Educational Research, 83(1), 70–120.
Alqassab, M., Strijbos, J. W., & Ufer, S. (2017). Training peer-feedback skills on
Ferris, D. R. (1997). The influence of teacher commentary on student revision. TESOL
geometric construction tasks: Role of domain knowledge and peer-feedback levels.
Quarterly, 31, 315–339.
European Journal of Psychology of Education, 33(1), 11–30. https://doi.org/10.1007/
Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’
s10212-017-0342-0
learning. Learning and Teaching in Higher Education, 1, 18–19.
Alqassab, M., Strijbos, J. W., & Ufer, S. (2018). Preservice mathematics teachers’ beliefs
Hansen, J. G., & Liu, J. (2005). Guiding principles for effective peer response. ELT
about peer feedback, perceptions of their peer feedback message, and emotions as
Journal, 59(1), 31–38.
predictors of peer feedback accuracy and comprehension of the learning task.
Hislop, J., & Stracke, E. (2017). ESL students in peer review: An action research study in
Assessment & Evaluation in Higher Education, 44(1), 139–154. https://doi.org/
a university English for Academic Purposes course. University of Sydney Papers in
10.1080/02602938.2018.1485012
TESOL, 12, 9–44.
Amores, M. J. (1997). A new perspective on peer-editing. Foreign Language Annals, 30(4),
Hu, C., & Zhang, Y. (2014). A study of college English writing feedback system based on
513–522. https://doi.org/10.1111/j. 1944-9720.1997.tb00858.x
M-learning. Modern Educational Technology, 7, 71–78. https://doi.org/10.3969/j.
Huang, J. (2008). How accurate are ESL students’ holistic writing scores on large-
issn.1009-8097.2014.07.010
scaleassessments? A generalizability theory approach. Assessing Writing, 13,
Hu, G. (2005). Using peer review with Chinese ESL student writers. Language Teaching
201–218.
Research, 9(3), 321–342.
Huang, J. (2012). Using generalizability theory to examine the accuracy and validity of
Hu, G., & Lam, S. T. E. (2010). Issues of cultural appropriateness and pedagogical
large-scale ESL writing assessment. Assessing Writing, 17, 123–139.
efficacy: Exploring peer review in a second language writing class. Instructional
Huang, J., & Foote, C. J. (2010). Grading between lines: What really impacts professors’
Science, 38, 371–394.
holistic evaluation of ESL graduate student writing? Language Assessment Quarterly,
Huisman, B., Saab, N., van Driel, J., & van den Broek, P. (2018). Peer feedback on
7, 219–223.
academic writing: Undergraduate students’ peer feedback role, peer feedback
Azarnoosh, M. (2013). Peer assessment in an EFL context: Attitudes and friendship bias.
perceptions and essay performance. Assessment & Evaluation in Higher Education, 43,
Language Testing in Asia, 3, 1–10.
955–968. https://doi.org/10.1080/02602938.2018.1424318
Baker, K. M. (2016). Peer review as a strategy for improving students’ writing process.
Hyland, K., & Hyland, F. (2006). Feedback on second language students’ writing.
Active Learning in Higher Education, 17(3), 170–192.
Language Teaching, 39, 83–101. https://doi.org/10.1017/S0261444806003399
Berg, I. V. D., Admiraal, W., & Pilot, A. (2006). Peer assessment in university teaching:
Jacobs, G. M., Curtis, A., Braine, G., & Huang, S. Y. (1998). Feedback on student writing:
Evaluating seven course designs. Assessment and Evaluation in Higher Education, 31
Taking the middle path. Journal of Second Language Writing, 7(3), 307–317.
(1), 19–36.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and
Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English
educational consequences. Educational Research Review, 2, 130–144.
and, L2-writing development: A meta-analysis. TOEFL iBT Research Report, 14.
Lee, M.-K. (2015). Peer feedback in second language writing: Investigating junior
Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: The challenge
secondary students’ perspectives on inter-feedback and intra-feedback. System, 55
of design. Assessment & Evaluation in Higher Education, 38(6), 698–712.
(Suppl. C), S1–S10.
Brennan, R. L. (2001). Statistics for social science and public policy: Generalizability theory.
Lei, Z. (2017). Salience of student written feedback by peer-revision in EFL writing class.
New York: Springer-Verlag.
English Language Teaching, 10(2), 151–157.
Cameron, J., & Pierce, W. D. (1994). Reinforcement reward and intrinsic motivation: A
Lepper, M. R., Keavney, M. M., & Drake, M. (1996). Intrinsic motivation and extrinsic
meta-analysis. Review of Educational Psychology, 35, 459–477.
rewards: Acommentary on Cameron and Pierce’s meta-analysis. Review of
Carless, D. (2015). Excellence in university assessment: Learning from award-winning
Educational Research, 66, 5–22.
practice. London: Routledge.
Li, H., & He, L. (2015). A comparison of EFL raters’ essay-rating processes across two
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling
types of rating scales. Language Assessment Quarterly, 12(2), 178–212.
uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325.
Liu, J., & Hansen, J. G. (2002). Peer response in second language writing classrooms. Ann
Carson, J. G., & Nelson, G. L. (1994). Writing groups: Cross-cultural issues. Journal of
Arbor, MI: University of Michigan Press.
Second Language Writing, 3(1), 17–30.
Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer.
Caulk, N. (1994). Comparing teacher and student responses to written work. TESOL
review to the reviewer’s own writing. Journal of Second Language Writing, 18(1),
Quarterly, 28(1), 181–188.
30–43.
Chang, C. Y. (2015). Teacher modeling on EFL reviewers’ audience-aware feedback and
Mendoca, C., & Johnson, K. (1994). Peer review negotiations: Revision activities in ESL
affectivity in L2 peer review. Assessing Writing, 25, 2–21.
writing instruction. TESOL Quarterly, 28(4), 745–768.
Chen, C. (2010). Graduate students’ self-reported perspectives regarding peer feedback
Min, H. T. (2006). The effects of training peer review on EFL students’ revision types and
and feedback from writing consultants. Asia Pacific Education Review, 11, 151–158.
writing quality. Journal of Second Language Writing, 15, 118–141.
Cho, K., & MacArthur, C. (2011). Learning by reviewing. Journal of Educational
Mittan, R. (1989). The peer review process: harnessing students’ communicative power.
Psychology, 103(1), 73–84.
In D. Johnson, & D. Roen (Eds.), Richness in writing: Empowering ESL students (pp.
Zhao, C., & Huang, J. (2020). The impact of the scoring system of a large-scale
207–219). New York: Longman.
standardized EFL writing assessment on its score variability and reliability:
National College English Testing Committee. (2016). Syllabus for college English test –
Implications for assessment policy makers. Studies in Educational Evaluation, 67,
Band 4. Shanghai, China: Shanghai Language Education Press.
Article 100911.
Nelson, G. L., & Carson, J. G. (1998). ESL students’ perceptions of effectiveness in peer
Liu, Y., & Huang, J. (2020). The quality assurance of a national English writing
response groups. Journal of Second Language Writing, 7(2), 113–131. https://doi.org/
assessment: Policy implications for quality improvement. Studies in Educational
10.1016/S1060-3743(98)90010-8
Evaluation, 67, Article 100941.
Niu, R., & Zhang, R. (2018). A case study of focus, strategy and efficacy of anL2 writing
Han, T., & Huang, J. (2017). Examining the impact of scoring methods on the
teacher’s written feedback. Journal of PLA University of Foreign Languages, 41(3),
institutional EFL writing assessment: A Turkish perspective. PASAA, 53, 1–36.
91–99.
Connor, U., & Asenavage, K. (1994). Peer response groups in ESL writing classes: How
Patchan, M. M., Schunn, C. D., & Clark, R. J. (2018). Accountability in peer assessment:
much impact on revision. Journal of Second Language Writing, 3(3), 256–276.
Examining the effects of reviewing grades on peer ratings and peer feedback. Studies
Cotterall, S., & Cohen, R. (2003). Scaffolding for second language writers: Producing an
in Higher Education, 43(12), 2263–2278.
academic essay. ELT Journal, 57(2), 158–166.
Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods
of Second Language Writing, 8(3), 265–289.
approaches (4th ed.). Thousand Oaks, CA: SAGE Publications.
Price, M., Handley, K., & Millar, J. (2011). Feedback: Focusing attention on engagement.
Crick, J. E., & Brennan, R. L. (1983). GENOVA: A general purpose analysis of variance
Studies in Higher Education, 36(8), 879–896.
system. Version 2.1. Iowa City, JA: American College Testing Program.
Rahimi, M. (2013). Is training student reviewers worth its while? A study of how training
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of
influences the quality of students’ feedback and writing. Language Teaching Research,
behavioral measurements: Theory of generalizability for scores and profiles. New York:
17, 67–89. https://doi.org/10.1177/1362168812459151
Wiley.

14
W. Wu et al.
Ruegg, R. (2015). The relative effects of peer and teacher feedback on improvement in
EFL students’ writing ability. Linguistics and Education, 29, 73–82.
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park,
CA: Sage.
Séror, J. (2011). Alternative sources of feedback and second language writing
development in university content courses. Canadian Journal of Applied Linguistics, 14
(1), 118–143.
Tian, L., & Zhou, Y. (2020). Learner engagement with automated feedback, peer
feedback and teacher feedback in an online EFL writing context. System, 91, 1–14.
Tsui, A. B. M., & Ng, M. (2000). Do secondary L2 writers benefit from peer comments?
Journal of Second Language Writing, 9, 147–170.
van Ginkel, S., Gulikers, J., Biemans, H., & Mulder, M. (2017). The impact of the
feedback source on developing oral presentation competence. Studies in Higher
Education, 42(9), 1671–1685. https://doi.org/10.1080/03075079.2015.1117064
Wang, W. (2014). Students’ perceptions of rubric-referenced peer feedback on EFL
writing: A longitudinal inquiry. Assessing Writing, 19, 80–96.
Wu, Z. (2019). Lower English proficiency means poorer feedback performance? A mixed-
methods study. Assessing Writing, 41, 14–24.
Xu, Y., & Carless, D. (2017). ‘Only true friends could be cruelly honest’: Cognitive
scaffolding and social-affective support in teacher feedback literacy. Assessment &
Evaluation in Higher Education, 42(7), 1082–1094.
Yang, M., Badger, R., & Yu, Z. (2006). A comparative study of peer and teacher feedback
in a Chinese EFL writing class. Journal of Second Language Writing, 15, 179–200.
15
Studies in Educational Evaluation 73 (2022) 101140
Yao, Y., Guo, N. S., Li, C., & McCampbell, D. (2020). How university EFL writers’ beliefs
in writing ability impact their perceptions of peer assessment: Perspectives from
implicit theories of intelligence. Assessment & Evaluation in Higher Education, 1–17.
https://doi.org/10.1080/02602938.2020.1750559
Yu, S. (2019). Learning from giving peer feedback on postgraduate theses: Voices from
Master’s students in the Macau EFL context. Assessing Writing, 40, 42–52.
Yu, S., & Hu, G. (2017). Understanding university students’ peer feedback practices in
EFL writing: Insights from a case study. Assessing Writing, 33, 25–35.
Yu, S., & Lee, I. (2016a). Peer feedback in second language writing (2005–2014).
Language, Teaching, 49(4), 461–493.
Yu, S., & Lee, I. (2016b). Understanding the role of learners with low English language
proficiency in peer feedback of second language writing. TESOL Quarterly, 50(2),
483–494.
Yu, S., & Lee, I. (2016c). Exploring Chinese students’ strategy use in a cooperative peer
feedback writing group. System, 58, 1–11.
Zhang, S. (1995). Reexamining the affective advantage of peer feedback in the ESL
writing class. Journal of Second Language Writing, 4(3), 209–222.
Zhao, H. (2010). Investigating learners’ use and understanding of peer and teacher
feedback on writing: A comparative study in a Chinese English writing classroom.
Assessing Writing, 15(1), 3–17.
Zimmerman, B. J., & Kitsantas, A. (2002). Acquiring writing revision and self-regulatory
skill through observation and emulation. Journal of Educational Psychology, 94,
660–668.