2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA)
Investigating the Effect of Chatbot-to-User Questions
2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA) | 978-1-6654-1511-8/20/$31.00 ©2021 IEEE | DOI: 10.1109/CAIDA51941.2021.9425208
and Directives on Student Participation
Daniel Bailey
English Language and Culture Department,
Konkuk University Glocal Campus, South Korea
268 Chungwon-daero, Dalcheon-dong, Chungju-si,
Chungcheongbuk-do
dbailey056@kku.ac.kr
ORCID ID: 0000-0003-0278-40
Abstract— Pedagogical chatbots are used to elicit information
from students. Yet, how the amount of student-generated content
differs according to the type of chatbot-delivered questions is
unknown. To this end, 19 South Korean English majors completed
six chatbot assignments through an in-house developed Facebook
Messenger chatbot. The chatbot activity entailed creating original
stories for class presentations. In addition to directives requesting
plot details, the chatbot used closed-ended button reply questions,
open-ended questions, and fill-in-the-blank template statements to
help students create stories. Results indicated that button reply
questions allowed for pacing, recall and content assessment and
required low levels of critical thinking. Next, open-ended questions
and fill-in-the-blank template statements resulted in similar word
count production but different levels of creativity and critical
thinking. Lastly, directives requesting user input resulted in 35%
more output, indicating students took more action when told to do
something than when asked. Regarding the novelty effect, fewer
students volunteered to do the sixth and final chatbot activity, but
those who did produced word count on par with their initial
chatbot activity.
Keywords— Chatbot, conversational agent, chatbot-assisted
language learning, computer-mediated communication.
I. INTRODUCTION
With the outstanding expansion of educational technology
throughout the past decade, chatbots are evolving into an
inexorably prominent alternative method for communicating
with students, and their appropriation is rapidly spreading [1].
Innovation leads to improved chatbot platforms that can be
trained to respond to user-input with reasonable accuracy,
resulting in user satisfaction and flexibility in functions,
however, testing methods to mitigate miscommunication and
optimize learner engagement need attention. Further, research
exploring the persuasive power chatbots have on eliciting
participation is an opening frontier.
Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
Norah Almusharraf
Applied Linguistics Department,
College of Humanities, Prince Sultan University
Kingdom of Saudi Arabia, Riyadh, P.O. Box 66833
nmusharraf@psu.edu.sa
ORCID ID: 0000-0002-6362-4502
This study sought to compare the different levels of user
output according to the chatbot-to-user question type and
directive by implementing a conversational agent that scaffolds
learners through their own storytelling process. Further, a
comparison to user output between the first and final chatbot
activity was made to check for previously recognized novelty
effects with chatbot participation [2]. Specifically, this paper
centers on the types of questions and directives chatbots deliver
to students, how those questions and directives influence
participation, and how that participation changes over time.
With these aims in mind, this paper asks the following questions:
1. What are the rates of participation with a story
generating chatbot activity?
2. How do differences in chatbot-delivered questions and
directives influence participation?
3. How do user responses change over the period of an
academic semester?
II. LITERATURE REVIEW
A. Theoretical Framework: Bloom’s Taxonomy
Bloom’s Taxonomy [3] was termed after Benjamin Bloom
and initiated some educational objectives and assessments for
institutions to utilize [4]. Bloom's revised taxonomy of
educational purposes [5] is a tool that can aid the determining
learners' cognitive development from lower-order thinking, at
the fact and concept levels, to higher-order thinking. The revised
taxonomy entails six levels of increasing cognitive complexity
by which learners comprehend content: knowledge,
understanding, application, analysis, evaluation, and creation.
The first two levels reflect lower-order thinking (i.e.,
remembering and understanding), while the other four reflect
progressively higher-order thinking, including analyzing,
evaluating, and creating [6].
 Previous studies have spectacled the usage of chatbots in
learning English. In particular, a study by Chocarro et al. [7]
examined teachers’ attitudes towards chatbots in education by
considering the effect of social language, bot proactiveness, and
users’ characteristics on learner perceptions. The results
confirmed a positive and substantial influence of the perceived
practicality for using chatbots in educational settings. Similarly,
Chen et al. [8] conducted a study with 58 students, which found
chatbots significantly improved the students’ learning
attainment and found perceived usefulness as the predictor of
behavioral intention; however, participants agreed that while
chatbots benefited their learning of Chinese vocabulary, several
adjustments to functionality, including ease-of-use, need to be
made for further progress.
B. Challenges with Chabot-User Interactions
Although chatbots are a great approach to getting a closer
look into any language that the user hopes to learn and develop
over time, some argue that they cannot be reliable without
human supervision [9]. Further, another recognized problem
with current chatbot interactions is the lack of cohesion during
the session due to incorrect pair-matching. In EFL, chatbots like
Mistuku (www.mitsuku.com) and Tutor Mike (www.rong-
chang.com/tutor_mike.htm) have been used as communication
channels, often resulting in limited and superficial interactions
that lack cohesion from one statement to the next. And while the
accuracy of the chatbot responses to user-input continues to
improve over time, the session topics are random, short-lived,
and often have no point. Too often chatbots go through system
fails, giving wrong suggestions to the student searching for a
specific topic or giving the student unreliable information.
Strategic use of bot-to-user questions is a proposed solution
to superficial, meaningless, chatbot interactions. When
messaging with a chatbot, a combination of question types is
recommended so that the student is not either bored with simple
recall questions or overwhelmed with cognitively demanding
questions requiring higher-order thinking skills [10]. Through
higher-level questions, learners create original content by
processing beforehand learned material or information [11].
Questions igniting higher-order thinking are more practical for
reassuring students to think deeper and inspire them to hunt for
information autonomously. Still, lower-level questions are
suitable for assessing learners’ preparation or reviewing content.
Bailey et al. [23] administered a series of chatbot activities
for digital storytelling. Students would interact with a Facebook
Messenger bot that controlled the pace and plot of a story, asking
students to predict plot details and recall story information.
Results indicated cohesion throughout the chatbot sessions with
easily corrected instances of miscommunication [23]. Overall,
the digital storytelling chatbots proved to be appropriate for EFL
reading practice but were limited in their writing practice
capacity and this was attributed to limited use of bot-to-user
questions.
To support a chatbot-assisted learning environment
conducive to writing practice, we propose a framework,
supported by Bloom’s revised taxonomy, for a type of chatbot
interaction, which functions as a framework for the analysis of
content emanating from chatbot-to-user questions directives.
Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
III. METHODS
A. Overview
This single sample exploratory study followed a descriptive
analysis method to first measure student-generated output
according to chatbot questions and directives to EFL students.
Then, how that output changes over time was observed. The
chatbot used button reply questions, open-ended questions,
close-ended template statements, and directives to elicit the
target language's use (i.e., English).
The chatbot activities were administered during the fall
semester of 2020 when universities in South Korea were taught
online due to the COVID-19 pandemic. Chatbot logs were
collected from 19 South Korean (male= 9, female=10) junior
and senior English majors attending a multimedia
communication course. Participation was defined as content
generated as measured by word count, which is a useful metric
for calculating chatbot-user interactions [12], [13]. Overall,
students completed six chatbot activities aimed to help them
create original content for class presentations and small group
discussions. The chatbot would scaffold students through
sessions by asking them questions related to the task. The
chatbot guided students with various questions and directive
statements to construct an original story with the main character,
setting, conflict, and resolution. At the end of the chatbot
session, student responses were returned to them in a single
collocated message, which would be used as notes for a class
presentation.
B. Materials
The chatbot activities were created with the ActiveChat.ai
chatbot platform and then integrated with Facebook Messenger.
All students had Facebook Messenger installed on their phones,
and chatbot participation occurred during regular class hours..
1) Question types and directives
The purpose of the chatbot questions and directives was to
collect information from the user. Questions types included
button responses (e.g., yes/no), open-ended questions seeking
specific information (e.g., name, location, time, and weather),
open-ended questions asking for plot details, directives
requesting plot details, and directives telling students to fill out
template statements for plot details that students can model their
responses or replace completely with original details. Table 1
displays examples of questions asked and directives given
during the “Create Story” activity.
TABLE I. QUESTIONS ASKED ACCORDING TO CHATBOT REQUEST
Example button-reply questions (Recall Questions)
1 Have you visited my storytelling library and read some of my stories?
2 Do you understand characters, setting and conflict?
Example open-ended questions eliciting specific information (Analyzing
Questions)
1 How old is your character?
2 Is your character a male, female or something else?
Open-ended question (Creating Questions)
1 How else would you describe your character?
2 What problem in life does your character have?
Fill-in-the-blanks question (Fill in the blank analyzing question)
 1 Now it is your turn. Fill in the blank. "I am going to tell you a story
about ________________"
2 Now it's your turn. Fill in the blank with your own answer. "I went to
___________ to
__________________________________________."
Chatbot-to-user directives
1 Now it is your turn. Write the 4th line of your story. I'm not going to
give you any help. Do this line by yourself.
2 Now it is your turn. Write a line in your story that introduces conflict.
a.
Note: a Close-ended button reply statement led to an open-ended response by students with
considerable output.
C. Data Analysis
The statistical package for social science (SPSS; Version 25)
was used to calculate descriptive statistics. For research question
1, data were sourced from 19 chatbot session logs, the first
“Create Story” activity. The participation rate was measured as
total bot messages, non-question button replies, bot-to-user
questions and directives, and user-to-bot response. For research
question 2, student responses to the corresponding question
types were tagged as button response questions (BRQ), specific
information questions (SIQ), open-ended questions (OEQ),
directive statements (DS), and template style questions (TSQ).
Next, analysis of variance (ANOVA) followed by Bonferroni
post hoc analyses were used to check for significant differences
between the amount of content produced by the different chatbot
questions and directives. For research question 3, 10 chatbot
session logs from activity six, the second “Create Story”
activity, were coded following steps taken for the initial 19
session logs. Then pre/post-analysis of word count for the
different question types and directives were measured between
the second chatbot activity (i.e., first create story activity) in
week three to the sixth chatbot activity (i.e., second create story
activity) in week 14.
IV. RESULTS
For research question one, rates of participation were
measured. Table 2 displays descriptive statistics for
participation metrics emanating from the initial “Create Story”
chatbot activity (n=19). Overall, approximately 35% of
messages were sent from students accounting for 21% of the
total word count produced during the session. Chatbots sent 65%
of the messages resulting in the remaining 79% of the content.
Further, the bot sent 709 total questions averaging 37 questions
per activity. Of the 709 questions, 477 required written, open-
ended, user responses, and 244 entailed closed-ended choices.
Non-question button choices were used for pacing the flow of
information to the student. In contrast to the research leading up
to the current stud [23], students produced more words and sent
more messages, and this was attributed to the increased
frequency of questions asked and directives given.
TABLE II. DESCRIPTIVE STATISTICS FOR PARTICIPATION METRICS
(N=19)
Message
M SD
s course; however, differences were not statistically significant.
Bot Messages 1854 17.81
Questions 709 15.29
User Messages 1009 8.62
Open-ended questions 477 16.83
Button Questions 244 1.21
Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
Research question 1 revealed that students wrote more when
meet
asked more questions. Now, question 2 asks how participation
differs according to question type, fill-in-the-blank statements,
and directives. Table 3 displays ANOVA results for word count
concerning the bot's different types of questions to the students.
Stark differences were revealed between the content
produced according to question-type. Directives resulted in the
most student-generated content (M=34.55, SD=19.41). When
told, not asked, students committed to more work resulting in
more product. Button response questions (BRQ) and specific
information questions (SIQ) overwhelmingly resulted in
answers with a single word or short phrase but rarely a complete
sentence. These questions often inquired towards background
information about characters (e.g., age and name) and settings
(e.g., coffee shop). When given the opportunity, students
avoided writing complete sentences and instead wrote a single
word or number. With an average output of approximately 19
words per question (SD=16.94), open-ended questions resulted
in substantially more content generated by students than button
response questions and questions asking for specific
information. Further, open-ended questions were answered in
complete sentences. Template style questions shared similar
output levels as open-ended ones with approximately 17 words
per question (SD =13.02). Students also wrote in complete
sentences; however, less word variation was recognized because
students often copied the given template instead of creating
original content.
TABLE III. ANOVA RESULTS FOR WORD COUNT FOR DIFFERENT
QUESTION TYPES
N M SD SE
BRQ 247 1.23 0.48 0.03
SIQ 149 3.32 3.83 0.31
OEQ 113 19.21 16.94 1.59
DS 105 34.55 19.41 1.89
TSQ 96 16.93 13.02 1.33
Total 997 8.63 14.61 0.46
b.
Note: BRQ, button response questions; SIQ, specific information question; OEQ, open-ended
questions; DS, directive statements; TSQ, template style question.
Research question 3 addresses possible novelty effects
resulting in decreased participation levels between the first and
last “Create Story” activity. The first bot activity resulted in all
19 students completing the activity. This number dropped by the
last chatbot activity, which resulted in only 10 of the 19 students
participating. Once engaged with the activity, the amount of
writing by the student was on par with the first “Create Story”
activity. Students wrote similar amounts of content, which was
consistent for the questions and directives being addressed but
not for the fill-in-the-blank statements. Table 5 shows
descriptive statistics of content generation between the pre and
post-activity. For open-ended questions and directive
statements, participation trended upward at the end of the
9.42
7.89
The opposite trend was observed for template style questions,
14.38 which showed a 25% drop in words per question.
17.58
To test for significance in change between the first and
0.41
second “Create Story” activity, a series of independent t-tests
were performed. While the mean count differences were robust, V. DISCUSSION
the differences were not statistically significant, attributed to Several findings emanated from this study. In particular,
high variation in student responses. findings contributed to missing attention concerning chatbot-
delivered questions and directives in chatbot dialogues [14].
TABLE IV. BONFERRONI POST HOC ANALYSIS OF STUDY QUESTION
TYPE RESPONSES Student content varied according to whether they were
answering a particular question type or responding to a directive.
MD SE p Low High Evidence for both lower and higher levels of critical thinking
BRQ SIQ -2.08 0.99 0.53 -4.99 0.82 were apparent in the students’ interactions in the form of shorter
OEQ -17.98 1.08 0.00 -21.16 -14.80
and more extended responses. Open-ended questions and
directives required analyzing and creating original content,
DS -33.32 1.11 0.00 -36.58 -30.05
therefore, relying on deeper critical thinking levels. Contrarily,
TSQ -15.69 1.14 0.00 -19.06 -12.32 button-response questions and questions asking for specific
SIQ BRQ 2.08 0.99 0.53 -0.82 4.99 details required recall of specific information, therefore, relying
OEQ -15.90 1.19 0.00 -19.39 -12.40 on lower levels of critical thinking and a faster pace in the
DS -31.24 1.21 0.00 -34.81 -27.67 activity. Fill-in-the-blank template style questions resulted in
TSQ -13.61 1.25 0.00 -17.28 -9.95 similar levels of words per question as open-ended questions.
OEQ BRQ 17.98 1.08 0.00 14.80 21.16
Low word variation with fill-in-the-blank template style
questions was attributed to students borrowing language directly
SIQ 15.90 1.19 0.00 12.40 19.39
from the template.
DS -15.34 1.29 0.00 -19.14 -11.54
DS BRQ 33.32 1.11 0.00 30.05 36.58 In response to research question 1: What are the rates of
SIQ 31.24 1.21 0.00 27.67 34.81
participation with a story generating chatbot activity?
participation resulted in increased levels of writing by students
OEQ 15.34 1.29 0.00 11.54 19.14
compared to past chatbot studies that relied on less variation and
TSQ 17.63 1.34 0.00 13.67 21.58 frequency in chatbot-to-user questioning behavior [15]. Chatbot
TSQ BRQ 15.69 1.14 0.00 12.32 19.06 activities requiring students to create original stories resulted in
SIQ 13.61 1.25 0.00 9.95 17.28 a substantial amount of user output, and this was partly
DS -17.63 1.34 0.00 -21.58 -13.67 attributed to questions asking students to analyze, evaluate, and
c.
create content. There was a combination of both long and short
Note: BRQ, button response questions; SIQ, specific information question; OEQ, open-ended
questions; DS, directive statements; TSQ, template style question. answers depending on whether students were addressing
questions or directives requiring lower or higher critical thinking
levels. A similar pattern of results was obtained in Rahman and
TABLE V. MEAN SCORE COMPARISON BETWEEN PRE AND POST Abdul Manaf’s [6] study that indicated lower-order thinking is
“CREATE STORY” ACTIVITIES a precondition for higher-order thinking to analyze, evaluate, or
generate information. Button reply options served the purpose
Question Pre and
Type Post
N M SD SE of both guiding low-level thinking skill questions and pacing
through information flow, but perhaps most importantly,
Pre 45 3.87 4.74 0.71 provided the foundation for deeper and more critical level
SIQ
Post 41 4.00 4.33 0.68 thinking later on in the activity.
Pre 33 18.58 16.43 2.86
OEQ Further, button reply options quickened the story's pace,
Post 31 20.13 16.413 2.95 which Tallyn et al. [16] recognized as a valuable attribute to
Pre 28 34.54 20.36 3.85 button reply questions. The next finding concerning
DS participation pertained to communication breakdown and added
Post 31 37.32 19.17 3.44
cohesion throughout the chatbot session. Giving control of the
Pre 32 20.56 16.04 2.84
TSQ conversation to the chatbot system to amend instances of
Post 25 15.20 12.75 2.55 miscommunication through the use of a closed-system dialog
d.
Note: BRQ, button response questions; SIQ, specific information question; OEQ, open- chatbot mitigated communication breakdown recognized in past
ended questions; DS, directive statements; TSQ, template style question. studies [10], [17]. Further, having the activity relate to a single
topic, such as creating a story, added the missing cohesion in
TABLE VI. INDEPENDENCE T-TEST FOR PRE AND POST RESPONSES TO chatbot-user interactions recognized in past chatbot research [2].
DIFFERENT QUESTION TYPES
Research question 2: How do differences in the types of
p t df p MD SE chatbot-delivered questions and directives influence
SIQ .533 -.136 84 0.890 -.130 .980 participation? observed the different influence types of
OEQ .545 -.378 62 0.710 -1.55 4.11 questions and directives have on word count. A key finding
DS .413 -.541 57 0.590 -2.79 5.15 emerged from our study indicated that directives resulted in the
TSQ .486 1.367 55 0.177 5.36 3.92 most extended messages and provided the most details for the
created stories in comparison to button reply questions, open-
ended questions, and fill-in-the-blank template style questions.
More words equated to more ideas, cohesion between ideas, and

Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
this finding indicated directives are responsible for more
profound levels of critical thinking than actual questions.
Directives, in the form of the chatbot telling the student what
to do, resulted in the most action taken. This result ties well with
previous studies wherein imperative statements resulted in the
most word count in responses, and longer messages resulted in
more detailed stories [14], [18]. This result of directives having
a more considerable impact on work than questions has
significance in EFL because content generation equates to target
language awareness and consequential second language
acquisition [19]. Further, directives' greater use can compensate
for the richness lacking in current chatbot-human
communication [20].
Both open-ended and fill-in-the-blank question styles had
similar word count levels; however, students borrowed language
from the fill-in-the-blank template statements resulting in less
creative storylines than if students were responsible for
producing an initial response without an example (i.e., help).
Open-ended questions seeking specific information resulted in
short phrases (e.g., my house) or single words (e.g., daytime)
with a few students writing simple sentences (e.g., in my house).
Consequently, the story's pace moved much faster and required
less cognitive load with button-reply responses and short answer
open-ended questions. To this end, directives, open-ended
questions and directives are recommended to be placed at the
end of chatbot activities, while short answer and button-reply
questions should be used earlier in the chatbot session in attempt
to grab the user’s interest.
Research question 3: How do user responses change over
an academic semester? compared participation levels between
the second and sixth chatbot activity over the course of the
semester. Both activities were identical in design and guided
students through the process of writing an original story with
characters, settings, conflict, and resolution. Findings echo Yin
and Satar’s [13] chatbot study that found lower proficient EFL
students engaged less with the chatbot system. In the current
study, all students were asked to participate through Facebook
Messenger. In five cases, students interacted with the chatbot
with less than five messages, out of 50+ interaction
opportunities. In four other cases, students never responded at
all. This activity was administered during class time, and their
instructor requested students to participate. The class was
conducted through the video conferencing platform zoom
during the fourteenth week of the semester, and the
observational data recognized participation drop-off by several
of the same students who chose to either underperform or not
participate in the sixth and final chatbot activity. Students that
chose not to participate in the chatbot activity were many of the
same students who displayed low or no participation in other
multimedia activities during the semester, indicating student
characteristics, such as those related to L2 proficiency, self-
efficacy, or intrinsic motivation may have influenced the
novelty effect rather than the value chatbots have towards
language learning. Future research is needed to better
understand the phenomenon of continued chatbot use.
A. Pedagogical Implications and Future Research
This study was exploratory in nature, however, resulted in
findings that apply to pedagogical chatbot systems and future
Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
research. These findings add to the growing body of evidence
supporting the use of chatbot activities in flipped, blended, and
online learning environments [21], [22]. In the EFL context,
mainly, chatbots can be used to preview vocabulary and key
phrases that can be later practiced with a live partner [10]. In
other academic domains, the possibilities are only constrained
by one’s imagination. Chatbots are, in essence, nothing more
than an interactive website that builds itself around the user,
through mapping reference points, knowledge level, and
participation patterns.
In classroom settings, the chatbot activity's content has
added value when it can be utilized in real-world settings. In
comparison to the Bailey et al.’s [23] preceding study on
storytelling chatbots, students produced significantly more
content, which was directly attributed to the increase in
questions and question types. With storytelling, students
answered hypothetical (e.g., what would you do if this happened
to you?) and predictive (e.g., what would you do next) style
questions but did not create content for their own story,
producing no original content to present in the classroom after
finishing the chatbot activity. In contrast, students answered a
variety of questions to help create an original story for
presentation purposes.
Chatbots that ask various questions types and directives
contribute to the current set of chatbot activity in EFL that have
shown benefits to learning [10], [24], [25], [26]. A combination
of question types is recommended for pedagogical chatbot
systems [10]. The student is neither bored with simple recall
questions nor overwhelmed with open-ended questions and
directives. Closed-ended button reply questions can then be
followed by open-ended questions that require recall
information like names, locations, or specific information
requiring one-word to single phrase answers. Once engaged,
chatbot activities can use fill-in-the-blank questions to scaffold
the behavior of writing complete sentences. Open-ended
questions that require students to generate original ideas should
be introduced. In the second half of the chatbot activity,
directives requesting students to produce original content should
be given. Button reply questions can be used more liberally. In
contrast, general open-ended questions and directives should be
used strategically, with each use contributing in a significant
way to the final product.
Fryer [2] states that “teachers should be aware that learners’
motivation to interact with chatbots may decrease as the novelty
effects diminish.”, therefore, a variety of chatbot activities need
to be explored and categorized. In essence, the chatbot activity
can be viewed as an interactive worksheet, and just as worksheet
activities are specific to learning objectives, so are chatbot
activities. In the future, chatbot activities will become more and
more idiosyncratic to the learners’ needs as designing activities
becomes more manageable for teachers to use.
VI. CONCLUSIONS
In this study, it was recognized that students produced
original stories with the assistance of an in-house developed
chatbot. A few significant takeaways emanated from this study.
Moving forward, continuing awareness should be placed on how
questions and directives elicit varying levels of user-generated
output in chatbot learning activities. Content written by students
was proportional to the level in which the question appeared on
Bloom’s revised taxonomy, with less time spent on recall
questions and the most time dedicated to questions asking
students to use creative thinking skills. Chatbot-to-user
directives resulted in the most content produced. These
directives explicitly asked students to create original story
content and therefore were placed at the tip of Bloom’s
taxonomy.
Several limitations were recognized with this study,
including a small sample-size of junior and senior South Korean
university English majors. Therefore, further research can
examine the factors affecting students’ engagement by
comparing chatbots with other activities and with different types
of students. A larger sample of participants needs to be gathered
in future studies in order to provide more valuable and widely-
utilized insights into chatbot technology in the blended and fully
online learning environment. Comparisons between questions
should be observed in L1 settings, with different age groups and
geographic locations. Further, consistency in findings with
educational research is imperative before making any
procedure-base conclusion.
Further work is certainly required to disentangle these
complexities in exploring functional websites for additional
evidence to enhance the existing knowledge base. In addition,
research is required to confirm these original findings regarding
chatbots’ usefulness. Cultivating chatbots' performance and
usefulness is an essential determinant for users when
considering adopting chatbot technology [7]. Merging relevant
data from all the stored documents may empower the system to
yield answers to complex questions. Features in communication,
such as the quality of user responses, apply across different bot
designs because it is not the bot's sophistication in question but
instead bot communication strategies, including the analysis of
bot-to-user question types and commands.
ACKNOWLEDGEMENT
The authors would like to thank Prince Sultan University
(PSU) for funding this research project under grant number
IRB-CH-2021-6-1 and in affiliation with the Applied
Linguistics Research Lab.
REFERENCES
[1] D. Laurillard and A. Masulis, Rethinking university teaching: A
conversational framework for the effective use of learning technologies.
Routledge, 2013.
[2] L. K. Fryer, M. Ainley, A. Thompson, A. Gibson, and Z. Sherlock,
“Stimulating and sustaining interest in a language course: An
experimental comparison of Chatbot and Human task
partners,” Computers in Human Behavior, vol. 75, pp. 461–468, 2017.
[3] B. S. Bloom (1956), Taxonomy of educational objectives. Allyn & Bacon.
[4] D. R. Krathwohl, “A Revision of Blooms Taxonomy: An
Overview,” Theory Into Practice, vol. 41, no. 4, pp. 212–218, 2002.
[5] B. S. Bloom, L. W. Anderson, and D. R. Krathwohl, A taxonomy for
learning, teaching, and assessing a revision of Blooms Taxonomy of
educational objectives. New York: Longman, 2001.
[6] S. A. Rahman and N. F. A. Manaf, “A Critical Analysis of Bloom’s
Taxonomy in Teaching Creative and Critical Thinking Skills in Malaysia
through English Literature,” English Language Teaching, vol. 10, no. 9,
p. 245, 2017.
[7] R. Chocarro, M. Cortiñas, and G. Marcos-Matás, “Teachers’ attitudes
towards chatbots in education: a technology acceptance model approach
Authorized licensed use limited to: Carleton University. Downloaded on May 28,2021 at 15:14:49 UTC from IEEE Xplore. Restrictions apply.
considering the effect of social language, bot proactiveness, and users’
characteristics,” Educational Studies, pp. 1–19, 2021.
[8] H.-L. Chen, G. V. Widarso, and H. Sutrisno, “A ChatBot for Learning
Chinese: Learning Achievement and Technology Acceptance,” Journal
of Educational Computing Research, vol. 58, no. 6, pp. 1161–1189, 2020.
[9] A. Veglis and T. A. Maniou, “Chatbots on the Rise: A New Narrative in
Journalism,” Studies in Media and Communication, vol. 7, no. 1, p. 1,
2019.
[10] Y. Goda, M. Yamada, H. Matsukawa, K. Hata, and S. Yasunami,
“Conversation with a Chatbot before an Online EFL Group Discussion
and the Effects on Critical Thinking,” The Journal of Information and
Systems in Education, vol. 13, no. 1, pp. 1–7, 2014.
[11] J. Pereira, “Leveraging chatbots to improve self-guided learning through
conversational quizzes,” in Proceedings of the fourth international
conference on technological ecosystems for enhancing multiculturality ,
pp. 911–918.
[12] S. H. Qashoa, “Effects of teacher question types and syntactic structures
on ESL classroom interactions,” The International Journal of Social
Science, vol. 7, no. 1, pp. 52–62, 2013.
[13] Q. Yin and M. Satar, “English as a foreign language learner interactions
with chatbots: Negotiation for meaning,” International Online Journal of
Education and Teaching (IOJET), pp. 390–410, 2020.
[14] R. G. Athreya, A. C. Ngonga Ngomo, and R. Usbeck, “Enhancing
Community Interactions with Data-Driven Chatbots--The DBpedia
Chatbot,” in Companion Proceedings of the The Web Conference 2018,
pp. 143–146.
[15] D. R. Bailey. Chatbots as conversational agents in the contxt of language
learning. Proceedings of The Fourth Industrial Revoloution and
Education 2020, pp. 32- 120.
[16] E. Tallyn, H. Fried, R. Gianni, A. Isard, and C. Speed, “The Ethnobot:
Gathering Ethnographies in the Age of IoT,” in proceedings of the 2018
CHI Conference on Human Factors in Computing Systems, 2018, pp.1–
31, doi: 10.1145/3173574.3174178.
[17] S. Ghose and J. J. Barua, "Toward the implementation of a topic specific
dialogue based natural language chatbot as an undergraduate advisor,"
2013 International Conference on Informatics, Electronics and Vision
(ICIEV), Dhaka, Bangladesh, 2013, pp. 1-5, doi:
10.1109/ICIEV.2013.6572650.
[18] N. Y. Kim, Y. S. Cha, and H. undefined Kim, “Future English learning:
Chatbots and artificial intelligence,” Multimedia-Assisted Language
Learning, vol. 22, no. 3, pp. 32–53, 2019.
[19] M. Swain, “The output hypothesis and beyond: Mediating acquisition
through collaborative dialogue,” in Sociocultural theory and second
language learning, Oxford University Press, pp. 97–114.
[20] J. Hill, W. R. Ford, and I. G. Farreras, “Real conversations with artificial
intelligence: A comparison between human–human online conversations
and human–chatbot conversations,” Computers in human behavior, vol.
49, pp. 245–250, 2015.
[21] W. Huang, K. F. Hew, and D. E. Gonda, “Designing and evaluating three
chatbot-enhanced activities for a flipped graduate course,” International
Journal of Mechanical Engineering and Robotics Research, vol. 8, no. 5,
pp. 1–6, 2019.
[22] J. Jia, “CSIEC: A Computer Assisted English Learning Chatbot Based on
Textual Knowledge and Reasoning,” Knowledge-Based Systems, vol. 22,
no. 4, pp. 249–255, 2009.
[23] D. R. Bailey, A. Southam, & J. Costley (2021). Digital storytelling with
chatbots: Mapping L2 participation and perception patterns. Interactive
Technology and Smart Education, online journal: DOI 10.1108/ITSE-08-
2020-0170
[24] D. Coniam, “Evaluating the language resources of chatbots for their
potential in English as a second language,” ReCALL, vol. 20, no. 1, pp.
98–116, 2008.
[25] L. Fryer and R. Carpenter, “Emerging Technologies Bots as Language
Learning Tools,” Language Learning & Technology, vol. 10, no. 3, pp.
8–14, 2006.
[26] Höhn, Artificial Companion for Second Language Conversation.
Springer International Publishing.