See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/340939343

The Affordances and Constraints of Automatic Writing Evaluation (AWE)

Tools: A Case for Grammarly

Article · December 2019

CITATIONS READS

12 1,997

1 author:

Babak Khoshnevisan
University of South Florida
42 PUBLICATIONS 280 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

design analysis and comparison of HE and E shaped micro strip patch antenna View project

Exploration of the developmental stages of pre/in-service teachers (NEST & NNEST) View project

All content following this page was uploaded by Babak Khoshnevisan on 27 April 2020.

The user has requested enhancement of the downloaded file.

 The Affordances and Constraints of Automatic Writing Evaluation (AWE) Tools:
A Case for Grammarly
Babak Khoshnevisan
University of South Florida

Abstract
Technology has permeated different aspects of language education. While our understanding
regarding the use of technology in language education is well developed, the use of emerging
technologies such as automated writing evaluation (AWE) tools in writing tasks seems to be
missing. In this article, I will introduce automated writing evaluation (AWE) tools as both an
assessment and instruction tool to develop writing skills of English language learners.
Beginning with an introduction of the background of AWE tools, I will critically analyze the
affordances and constraints of these tools. I will then introduce Grammarly as an AWE tool
that can help students develop and hone their writing skills. Finally, I will report the results of
a case study that I conducted with international students learning English for academic
purposes (EAP) courses. The results suggest that Grammarly not only motivates language
learners but also serves as a catalyst to develop the learners’ writing skills.
Keywords: Automated writing evaluation (AWE) tool, English for academic purposes (EAP),
grading writing tasks

Introduction
As an EFL/ESL language educator, my two-decade experience in teaching English
indicates that educators—by and large—consider grading a cumbersome task. I am exposed to
educators from different nationalities, backgrounds, and educational levels that complain
about the amount of grading. They have found grading learners’ papers an interminable task
that does not help learners. Grading English for academic purposes (EAP) classes at the
graduate level has exacerbated the situation as learners are required to follow American
Psychological Association (APA) style referencing and accordingly need to comply with more
rules. On the other end of the scale, giving feedback to students while grading their papers is a
challenging task to implement. To complicate the situation, “...academics regularly complain
that students do not engage with feedback by citing uncollected coursework, students
repeatedly protest about the timeliness and quality of feedback, citing illegible, overly critical
and a lack of verbal feedback” (Blair, Curtis, Goodwin, & Shields, 2013, p. 66).
Ongoing technological advances have extensively altered the landscape of language
education worldwide. Khoshnevisan (2019, p.85) posits that the question “whether media
and/or technology-infused pedagogy makes technology-mediated learning more or less
influential” is yet to be thoroughly scrutinized. Multiple studies have investigated the role of
12
technology in language education: literacy and augmented reality (Park &
Khoshnevisan,2019); augmented reality and teacher education (Khoshnevisan, 2019c);
animated pedagogical agents (Khoshnevisan, 2018a); augmented reality (Khoshnevisan & Le,
2018); audiotaped dialogue journals (Rashtchi & Khoshnevisan, 2008); AR-infused apps
(Hadid, Mannion, Khoshnevisan, 2019).

Automated Writing Evaluation (AWE) Tools

The use of automated writing evaluation (AWE) Tools dates back to 1973 when Ellis
Page primarily programmed the first automated writing evaluation system—called Project
Essay Grade (PEG). PEG could help educators reduce the grading load. The results of PEG
were impressive at the time. However, the use of AWEs was—for some reason—postponed to
the 1990s. At the time, AWE tools were not well received for daily pedagogical implications
in the classrooms. One reason could be that technological devices do not afford teachers and
learners with the right tools to implement grading.

Feedback
Feedback is characterized as “information provided by an agent regarding one’s
performance or understanding – feedback thus is a sequence of performance” (Blair et al.,
2013, p. 67). Blair et al. (2013) argue that feedback provides information about the gaps
between actual performance and desired performance. Feedback seems to be the backbone of
formative assessment—a type of assessment in which “the performance of the student is
evaluated providing feedback, and instructional strategies are adjusted accordingly”
(Maclellan, 2011, p.99).
One may argue that educators are bound to supply learners with feedback. It is,
however, worth reviewing the compelling reasons why educators are held accountable to
provide learners with feedback. Blair et al. (2013) note that feedback clarifies good
performance, facilitates self-assessment, encourages dialogue between learner and instructor,
allows for positive motivation, reduces gaps, and effects a change in instruction.
Notwithstanding the type and mode of feedback (either oral or written), it seems to be an
inseparable part of instruction. When providing feedback, educators are required to take a
number of points into account, so as to make it effective. The major features of an effective
feedback are:
- be specific, immediate and spontaneous,
- indicate progress,
- provide corrective action.

13
 Parallel to the growing recognition of the role of feedback in the development of
writing skills, different types of technology-infused feedback have emerged (written and oral).
Automated Writing Evaluation (AWE) Tools are amongst the technology-infused tools that
provide learners with feedback. AWEs fall under two categories: business and free tools. Both
educators and learners have the opportunity to employ either business or free versions of these
tools. Educators can introduce these tools for learners to edit their essays, since such tools not
only enhance the awareness of learners regarding their writing process but also help educators
reduce the load of grading assignments. Table 1 introduces a list of business tools for which
they will be charged if they use them. However, there is a myriad of free tools that both
educators and learners can employ free of charge. Table 2 showcases free tools.

Table 1. Business AWE Tools URL

Grammarly http://www.grammarly.com
Criterion® Online Writing Evaluation https://criterion.ets.org/student/
Service
MY Access! http://www.myaccess.com
RightWriter http://www.right-writer.com/
WritePlacer http://professionals.collegeboard.com/higher-
ed/placement/accuplacer/write

Table 2. Free AWE tools

Ginger Software http://www.gingersoftware.com/grammarcheck/
Grammark http://grammark.org/
Grammarcheckme http://www.grammarcheck.me/
LanguageTool http://www.languagetool.org/
PaperRater http://www.paperrater.com/free_paper_grader
Queequeg http://sourceforge.net/projects/queequeg/
Spellchecker.net http://www.spellchecker.net/grammar/
SpellCheckPlus http://spellcheckplus.com/
WhiteSmoke http://www.whitesmoke.com/free-online-checker

Ferster et al. (2012) set out a number of reasons why educators use AWE tools. These
tools provide educators with objective results as they have no biases concerning the topic,
content, or learners. They primarily focus on the writing content as well as the final product.
These tools give constructive feedback to students, highlight the errors made, present the
reasons for these errors, and in some cases, they also provide more instruction about the type
of error. With that in mind, these tools—according to my experience—have the potential to
give immediate feedback to students so that students feel motivated. Immediate feedback,
coupled with a positive impact on the students’ gain, is key in grading students’ writings.
As discussed above, AWE tools can supply learners with correct feedback as soon as
possible. In the development of writing skills, one of the thorny issues for both students and
educators is that they do not develop a comprehensive picture of the learners’ improvement.
14
To obtain a detailed report of the learners’ development in writing tasks, portfolio assessment
has proved to immensely contribute to shape a more comprehensive picture of learners’
improvement. Learners can learn from their errors and observe their improvement in a tangible
way when completing their portfolios. Hornby (2006) posits that a portfolio is “a collection of
photographs, drawings, etc. that you use as an example of your work, especially when
applying for a job” (p. 1172). Richards and Schmidt (2002) state that a portfolio can be
defined as a purposeful collection of artifacts to portray a comprehensive picture of a person’s
efforts, progress, and achievement. Consequently, owing to the advantages of portfolios,
educators can exploit this tool as a learning or assessment tool. On this account, AWE tools
can help both language learners and educators develop a portfolio using AWE tools for
feedback. From this perspective, Khoshnevisan (2018b, p.8) explains that the major features of
portfolios are:
• A learner has the final decision to save a work in his portfolio (learner-
centered)
• It is viable for a learner to revise his portfolio based on the feedback
received from the teacher or peers.
• Learners assess and reflect on their portfolio. Accordingly, learners are
aware of their progress and academic gains.
• A portfolio can be taken into account as the mastery of idioms by teachers
and learners.
• A portfolio can encompass different modes of learning such as audio,
video, and written works.
The use of metacognitive strategies may enable a learner to self-regulate his advances
and complete a portfolio for the betterment of his writing. Learners can use all type of
feedback they receive (audio, video, or written) to improve their future texts. Collecting essays
in a portfolio affords students with a unique opportunity to examine and revisit the accuracy
and appropriacy of their written language in context.

Literature Review
Comparison between AWE Tools and Human Graders
Research indicates that there is a high correlation between human and machine grading
(Ferster et al., 2012). Accordingly, machine grading, contrary to public opinion, is not too
distant from human grading. However, there are some conflicting findings regarding machine
grading. Wang (2013) stresses that automated graders assign higher scores. On the other hand,
when machine grading is used, many students improve their performance when the pre- and
15
post-writing tests are compared (Wang, 2013). Regarding the benefits of automated grading,
research shows that this type of grading allows for recognition and detection of issues in
grammar, sentence structure, word usage, and style. (Ferster, Hammond, Alexander, &
Lyman, 2012; Shermis, Burstein, Higgins, & Zechner, 2010).
Despite the affordances of the automated graders, there are multiple constraints that
should caution educators. First and foremost, the feedback that these tools provide is not
specific (Ferster, Hammond, Alexander, & Lyman, 2012). Misinterpretation (e.g., the door is
ajar or the door is a jar) seems to be an indispensable part of feedback for automated graders.
These tools cannot provide feedback on students’ ability to answer a prompt. Accordingly, the
only feedback provided is the correct response rather than a progressive correction. It then
comes as no surprise that higher scores are assigned to longer essays (Ferster, Hammond,
Alexander, & Lyman, 2012).
Figure 1. juxtaposes AWE tools with human graders. Casting a cursory look at these
two shots, it turns out that AWE tools leave no room for a second thought. Said another way,
learners have no opportunity to contemplate their errors and make corrections. However,
machine grading can be immediate, correct, and organized.

Figure 1. A Comparison between AWE Tools with Human Graders

Prior studies have examined the impact of AWE tools on the development of writing
skills. However, exploring the perceptions and experiences of language learners about AWE
tools is vital to gain a more comprehensive picture about the effect of AWE tools on the
process of writing. Accordingly, to explore the perceptions of the students about one of the
Automated Writing Evaluation (AWE) tool—Grammarly. a qualitative method (case study)
was employed. It is worth mentioning that case study design enables researchers to gain an in-
depth and rather comprehensive description of learners’ approach to learning.

16
 The following research question guided my work:
1. How do four international students describe their experiences using Grammarly as
an automated writing evaluation (AWE) tool?

Methodology
Participants
Since this article attempts to analyze the results of a case study, for it explores the
participants’ perceptions and their experiences in using an AWE tool—Grammarly, EAP level
students were requested to use Grammarly to edit their essays at the beginning of the semester.
Twelve students, aged between 21 and 25, voluntarily participated in this study. They were
from a variety of majors such as mechanical engineering, financial management, management,
and education. It is worth mentioning that all the participants are graduate level students
studying EAP courses at an upper intermediate level of English proficiency.

Materials
Grammarly is a free, easy-to-use tool, which provides English language learners with
instant feedback for their writing. Users can also pay to upgrade their account and receive
more detailed corrections and explanations concerning the errors made. Another unique
feature of Grammarly is that it underlines errors and allows users to find out the type of error
to be able to modify their writing. Grammarly—as shown in Figure 2—supplies learners with
the correct format as well as a quick instructional tip to reduce the number of errors of the
same nature. There is no cap to the number of pages or texts being graded by this tool.
Figure 2. Quick instructional tips by Grammarly

At the end of the semester, I delved into the perceptions of the participants through an
online questionnaire (Appendix A) to explore their perceptions. I employed the constant
comparative method to analyze the data.

17
Procedure
The participants of this study used Grammarly to check their papers at the editing stage
of their writing from the outset of the semester. During the first session of the course, those
students who had used this tool to write at least 4 major papers went over the features of the
free version of Grammarly.

Results and Discussion

The data collected from the online questionnaire was analyzed to explore the major
perceptions of the students based on their experiences with Grammarly as an AWE tool to
develop their writing skill. The participants stated that completing a task in English is hard for
EFL students. One reason for the hardship is the existing differences between the participants’
L1 and English.
A challenging aspect of the English language is the fact that there are several
grammatical rules that must be followed when writing a paper. This is the aspect some
participants complained about:
Chen:
Writing has become my weakness since I learned English. I feel confused about
some English grammars because there are lots of differences between English
and my native language-Chinese. For example, I usually forget to add “s” to the
verb, when I use “he, she and it.
Muhammad:
You know, there are some difficult English grammars, such as clause,
independent genitive, subjunctive mood, etc.

In addition to observing differences between L1 and English, some participants

mentioned that Grammarly is a useful tool since it can help students build self-
confidence when writing papers:
Lee:
When I wrote academic papers or essays at …, I faced a[n] enormous challenge
in grammar and academic writing style. We often learn MLA and APA in
English courses. However, the English rules are difficult for secondary English
learners. According, I often used Grammarly to check my grammar mistakes. I
am not confident for English rules because my IELTS writing grades are only
6. Grammarly is very useful tool for me.

18
Ali:
After using Grammarly, I feel better when I submit my essay because it can
help me to check whether I have grammar mistakes. At the same time, I usually
learn some new grammars.

Similarly, Lee confirmed that AWE tools are helpful for international students:
A large number of automated writing tools is prevalent in the world. Because
international students should attend writing courses, humanity courses and social
courses. Accordingly, many students prefer to harness these tools for their
writing assignments. These tools are beneficial to enhance the quality of
writing…. Grammarly has a beneficial effect on improving vocabularies
correction and diversities. Grammarly often give some advices that include
reasons and specific vocabularies. I recommend it for international students…
Firstly, it helps me build my confidence. Secondly, it often helps check some
basis grammar mistakes such as punctuation rules, noun rules and so on.

All in all, the participants, arguably, learned from Grammarly. More

importantly, Grammarly increased the learners’ their awareness about grammatical
rules. Also, this tool successfully helped build the students’ self-confidence when
writing their papers, as it was stated by some of the participants:
Fatemeh:
I feel that I have made progress in the past few months.
Shantel:
I use Grammarly to check my writing, it helped me avoid fragments and
improve my punctuations. Grammarly helped improve my writing as it suggests
better vocabulary and correct punctuations. Grammarly is free, however, if you
need more than spell check and punctuations one needs to buy the premium
package for advanced writing feedback. I would positively recommend it to
everyone as it helps with small improvements in your paper.

However, it must be acknowledged that Grammarly is not a tool without deficiency.

There are few issues with Grammarly that need to be dealt with by its designers. For example,
Grammarly fails to effectively function beyond the sentence level. In this regard, Lee
mentioned:

19
 Grammarly, however, has disadvantages in grammar mistakes. For example, it
is hard to find logic prediction mistakes. This is the obvious disadvantage…

Although Grammarly has limited disadvantages, it can reduce grammar mistakes

and these students almost always have improved their writing grades.
The participants were, by and large, satisfied with their experience. The findings
of this study indicate that above 90% of them enjoyed working with Grammarly as a free
AWE tool. The following graph shows the details of the first question (Appendix A)
regarding the extent to which the participants are convinced that working with
Grammarly was a pleasant experience.

Along with the last diagram, the next diagram indicates that the participants deem that
Grammarly effectively impacted on their learning attainment. Also, more than 90% of the
participants stated that Grammarly was effective in learning grammar. The data collected from
the online questionnaire suggest that Grammarly immensely contributes to learning
grammatical tips and their applications for international students. The following diagram hows
the details of participants’ responses.

Learning vocabulary and collocations is a prominent feature of AWE tools such as

Grammarly. Accuracy and appropriacy are two challenging aspects of using words in

20
academic writing. Although the questionnaire did not explicitly state the ideas of the
participants, the responses to the third question suggest that approximately 80% confirm that
Grammarly can help them learn the accurate use of vocabulary and collocations. The
following diagram shows the responses of the participants concerning the impact of
Grammarly on learning vocabulary. A determining factor in the responses is the learners’
focus on grammar for EAP courses rather than on vocabulary and collocations.

Punctuation is a long-standing topic in developing writing skill. This often-neglected

topic should be discovered and examined in future studies. In this research work, the
participants reported that punctuation is important for them. Surprisingly, the participants did
not reiterate on this aspect of writing as much as on the other aspects discussed above. The
following diagram portrays the participants’ responses regarding the impact of Grammarly on
learning punctuation. On this account, approximately 63% of the participants verify that
Grammarly is effective in learning punctuation for international students.

The results of this online questionnaire coupled with informal chats with the
participants imply that Grammarly—as an AWE tool—can motivate language learners in

21
writing essays in academic courses such as EAP courses. Grammarly, in this sense,
contributed to building the students’ self-confidence to write essays. Furthermore, this tool
increases the academic achievement of learners.

Pedagogical Implications
Though Grammarly certainly offers multiple affordances, it may impose certain
constraints. Considering both affordances and limitations it is advisable for teachers to let
learners use automated writing evaluation (AWE) tools. AWE tools provide learners with
feedback on many aspects. This feedback is both instructional and motivational. Besides,
AWE tools help to minimize the amount of grading and therefore reduce the educators’
workload. It is, however, worth mentioning that AWE tools are not 100% accurate. This last
drawback can be well covered through final revisions by peers and educators.
One of the prominent features of AWE tools is that they can provide learners with
prompt feedback. Instant feedback cannot only increase the awareness of the students but also
individualize learning grammar, vocabulary, and punctuation. AWE tools, together with
portfolio assessment, can contribute to a broader and deeper learning for language learners and
assessment for language educators.

Conclusion
This article analyzed automated writing evaluation (AWE) tools, as well as their
affordances and limitations. The results of this study show that Grammarly not only motivated
learners to develop their writing skills but also helped learners produce more accurate essays.
As discussed earlier, the participants felt more self-confident and gained much by using
Grammarly as an AWE tool. The findings suggest that Grammarly facilitates the development
of writing skills by introducing tips about grammar, vocabulary, and punctuation. It is hoped
that this article can help educators reduce their workload in writing courses while motivating
international students via Grammarly. Though from my perspective, and according to the
results analyzed in this study that was carried out with a limited number of students (twelve)
and calls for further exploration, I would recommend educators to use Grammarly in their
writing courses because it helps learners scaffold learning and build confidence in a less
threatening ambience.

22
The Author
Babak Khoshnevisan is a PhD candidate in the Technology in Education and Second
Language Acquisition (TESLA) Program at the University of South Florida (USF). He is an
instructor at INTO USF. He is also the coeditor of CALL-IS newsletter at TESOL. His
research interests include but are not limited to CALL, VR, AR, teacher education, writing
skill, and idiomaticity.

References
Blair, A., Curtis, S., Goodwin, M., & Shields, S. (2013). What feedback do students
want? Politics, 33(1), 66-79.
Ferster, B., Hammond, T. C., Alexander, R. C., & Lyman, H. (2012). Automated formative
assessment as a tool to scaffold student documentary writing. Journal of Interactive
Learning Research, 23(1), 81-99.
Hadid, A., Mannion, P., & Khoshnevisan, B. (2019). Augmented reality to the rescue of
language learners. Florida Journal of Educational Research, 57(2), 81-89.
https://feraonline.org/article/2-augmented-reality-to-the-rescue-of-language-learners/
Hornby, A.S. (2006). Oxford Advanced Learner’s Dictionary (7th Ed.). Oxford: Oxford
University Press.
Khoshnevisan, B. (2019). To integrate media and technology into language education: For and
against. In W. B. James, & C. Cobanoglu (Eds.) Proceedings of the Global Conference
on Education and Research (GLOCER) Conference (Vol. 3). (pp. 85-92). Tampa, FL:
ANAHEI Publishing, LLC.
https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1021&context=anaheipub
lishing
Khoshnevisan, B. (2019). Spilling the Beans on Understanding English Idioms Using
Multimodality: An Idiom Acquisition Technique for Iranian Language Learners.
International Journal of Language, Translation and Intercultural Communication, 8,
128-143.
http://dx.doi.org/10.12681/ijltic.20281
Khoshnevisan, B. (2019). Teacher education meets emerging technologies: Augmented
Reality, (AR). TEIS Newsletter — TESOL International Association. Retrieved from
http://newsmanager.commpartners.com/tesolteis/issues/2019-03-04/4.html
Khoshnevisan, B. (2018). The Effect of Incorporating Animated Pedagogical Agents in Apps
on L2 Idiom Acquisition and Retention. In Proceedings of the Global Conference on
Education and Research (GLOCER) Conference (pp. 75-83). Sarasota, FL: ANAHEI

23
 Publishing, LLC.
https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1017&context=anaheipub
lishing
Khoshnevisan, B. (2018). Idiom Assessment: To Go off the Beaten Path. In Proceedings of the
Global Conference on Education and Research (GLOCER) Conference (pp. 2-9).
Sarasota, FL: ANAHEI Publishing, LLC.
https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1017&context=anaheipub
lishing
Khoshnevisan, B., & Le, N. (2018). Augmented Reality in Language Education: A Systematic
Literature Review. In Proceedings of the Global Conference on Education and
Research (GLOCER) Conference (pp. 59-74). Sarasota, FL: ANAHEI Publishing,
LLC.
https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1017&context=anaheipub
lishing
Maclellan, E. (2001). Assessment for learning: The differing perceptions of tutors and
students. Assessment & Evaluation in Higher Education, 26(4), 307-318.
Park, S. & Khoshnevisan, B. (2019). Literacy meets augmented reality (AR): The use of AR in
literacy. In W. B. James, & C. Cobanoglu (Eds.) Proceedings of the Global Conference
on Education and Research (GLOCER) Conference (Vol. 3). (pp. 93-99). Tampa, FL:
ANAHEI Publishing, LLC.
https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1021&context=anaheipub
lishing
Rashtchi, M., & Khoshnevisan, B. (2008). Audio-taped dialogue journal: A technique to
improve speaking skill of Iranian EFL learners. Journal of English Language
Pedagogy and Practice, 1(3), 164-176.
Richards, J.C., & Schmidt, R. (2002). Longman Dictionary of Language Teaching & Applied
Linguistics (3rd ed.). London: Longman (Pearson Education).
Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring:
Writing assessment and instruction. International encyclopedia of education, 4(1), 20-
26.
Wang, P. L. (2013). Can automated writing evaluation programs help students improve their
English writing? International Journal of Applied Linguistics and English
Literature, 2(1), 6-12.

24
 Appendix A

Grammarly as an Automated Writing Evaluation (AWE) Tool

This research attempts to explore the perceptions and experiences of international students
regarding using Grammarly as both an assessment and instruction tool. Accordingly, I would
appreciate it if you can invest some time and complete this online questionnaire.

Name:
Email:
Best phone number:

1. From the scale of 1-10, how do you describe your experience?

1 2 3 4 5 6 7 8 9 10

2. From the scale of 1-10, to what extent did Grammar impact on learning grammar?

1 2 3 4 5 6 7 8 9 10

3. From the scale of 1-10, to what extent did Grammar impact on learning vocabulary?

1 2 3 4 5 6 7 8 9 10

4. From the scale of 1-10, to what extent did Grammar impact on learning punctuation?

1 2 3 4 5 6 7 8 9 10

5. Which aspects of Grammarly helped you more? You may choose more than one option.

• Grammar
• Vocabulary
• Punctuation
• Word order
• Parts of speech (noun, verb, adjective, adverb)
• Sentence writing

6. What are the strong points of Grammarly?

7. What were the weak points of Grammarly?

8. Final comments?

25

View publication stats