2018 4th International Conference on Computer and Information Sciences (ICCOINS)

An Assessment of Game Elements in

An Assessment of Game
Language-Learning Elements
Platform in
Duolingo
Language-Learning Platform Duolingo
Duy Huynh1, Long Zuo2 and Hiroyuki Iida3
Japan Huynh ,Long
DuyAdvanced 1 Zuo
Institute of ,Hiroyuki
Science
2 Iida
and Technology.
3

Japan Advanced Institute

1 duyhuynh, of Science
2 zuolong, and Technology
3 iida3@jaist.ac.jp
duyhuynh1 , zuolong2 , iida3 @jaist.ac.jp

Abstract—This paper explores a way to assess the effects
of game elements in the domain of education. Educational
gamified language-learning platform ‘Duolingo’ is chosen as
a testbed, whereas game refinement measure is employed as
an assessment tool. Two game elements ‘Badge’ and ‘Winning
Streak’ with respect to the course structure of three language
courses in Duolingo are analyzed in this study. The results from
the analysis with about 1000 users show that Winning Streak
successfully helps users enhance their regular learning activity
into serious gaming activity. Moreover, Winning Streak increases
the motivation of advanced users when the attractiveness of
Badge decreases. Thus this study confirmed the effectiveness of
game refinement measure to identify the merit of the appropriate
division of learning process to maintain the attractiveness of
learning activity.
Keywords: Duolingo, Language learning platform, Game re-
finement theory, Game elements’ attractiveness, Badge, Winning
Streak
positive effects until they conduct an assessment. In case where
the use of the game is not sufficiently effective, it is challenging
to know exactly what game elements and what reasons
caused that. Therefore, this study aims at finding a common
measurement which reflects effectiveness or entertainment of
game elements. Using this measure, we assess and analyze
effects of a particular game element. According to conducted
analyses, an effective strategy can be applied for using game
elements in an existing system.
In this paper, we first aim at finding an assessment method
for the entertainment aspect. The method for our assessment is
to figure out from game refinement theory, which is a unique
theory that shows a particular way to quantify the attractiveness
and sophistication of the game under consideration. This theory
was proved and applied by several studies such as [6] [7].

I. I NTRODUCTION II. R ELATED W ORKS

Nowadays, digital technologies have changed the ways of Early works [8] [9] mainly used the game refinement measure
human working, socializing, communicating and studying. to quantify the attractiveness of game element Badge and
The game, which has been showcased as one of the digital figure out the contribution of game element Winning Streak for
technologies, is an important part of leisure lives for the young the improvement from entertainment aspect in Duolingo. In a
generation. Along with the development of technology, the gamified platform, one or a bundle of game elements is used for
game has been used to encourage people to achieve their a specific purpose. For this purpose, a reward or achievement
goals in almost every aspect of life, and support real-life such as certificate, badge, trophy, rank, or title is given to users
objectives many times in the past. In the recent years, this for encouraging them to complete the given tasks. Hence, an
idea has become especially popular. Gamification as a term attempt to complete learning tasks for obtaining rewards is
originated in the digital media industry. It entered widespread considered as a game action in the educational environment.
adoption in the second half of 2010 [1], when several industry Therefore, the game progress model has two factors R and
players and conferences popularized it. Unlike serious games, T which represents the average number of achievements or
gamification applications do not have gameplay designed for rewards and the average number of efforts or tasks respectively.
a specific purpose. It is only the gather of game elements The values R and T correspond to x(tk ) and tk in the
which are used for different purposes without changing the original idea of game refinement theory. The general formula
practice of existing systems. Most of the recent studies in the for deriving game refinement GR in a gamified platform is
domain of gamification aim to figure out the practical way expressed in Eq. (1).
for applying game elements into their learning systems. Thus, √
R
many gamification design models and frameworks [2] [3] are GR = (1)
figured out for pursuing that target. T
However, most gamification products are assessed via a In the previous study [8], the attractiveness of Badge in
survey or usability test [4] [5]. Those assessment methods a language course of Duolingo was analyzed. The milestone
spend many resources and only give general results. The game technique, which is used to design a desirable course structure
designers do not know whether the applied game elements make for a language course, was highlighted. The effect of a course
structure was examined by making some checkpoints for a long
R
“Duolingo” and all other trademarks, service marks, graphics and logos and tedious learning road. Using game refinement measure, the
used in connection with the Service are trademarks or service marks of
D UOLINGO or their respective owners, and certain of them are registered with attractiveness of Badge through each milestone was quantified.
the United States Patent and Trademark Office. The decrease of game refinement value through each milestone

978-1-5386-4744-8/18/$31.00 ©2018 IEEE ~ 1 ~

 2018 4th International Conference on Computer and Information Sciences (ICCOINS)

Milestone Bk Lk GRBk Wk GRW Sk PoSU

1 6 21 0.1166 2.2 0.0047 9%
2 16 67 0.0597 6.4 0.0053 41%
3 29 160 0.0337 21.9 0.0124 60%
4 61 317 0.0246 47.8 0.026 67%

Table I
T HE RESULTS OF ANALYZING W INNING S TREAK AND BADGE IN ‘S PANISH FOR E NGLISH S PEAKERS ’ COURSE

shows the increase of challenge in a language course which B. Data Collection and Analysis
tries to adapt the advancement of learners. Although the effect To observe the effects of Winning Streak, we collected many
of a course structure is indicated carefully in the previous work, data from 1000 users, who enrolled the “Spanish for English
we are not certain about whether different course structures Speakers” course (SFES) in Duolingo. The criterion to select
make any different effect on users in Duolingo. candidates is that they must be active on the system within one
Moreover, another work [9] suggests that Duolingo users week before the data collecting date. Because some information
may be attracted by Winning Streak when the attractiveness of fields of users should be kept in secret, so we only collected
Badge decrease. Therefore, in this study, we first aim at figuring public information of users. The collected information, which
out the attractiveness of Winning Streak individually to make is indicated on user’s profile page of Duolingo, includes some
sure that Duolingo users are attracted by this game element at skills, length of a Winning Streak and the languages course.
some last milestone. Then based on the results of analysis we After collecting the data, Eq. (2) and Eq. (3) are applied to
try to figure out the effect of the course structure obviously derive a measurement to assess the attractiveness of Winning
by comparing three popular language courses: “Spanish for Streak and Badge respectively.
English speakers”, “French for English speakers” and “Italian √
for English speakers.” Bk
GRBk = (3)
Lk
III. ATTRACTIVENESS OF W INNING S TREAK
In Eq. (3) which is derived from Eq. (1), Bk and Lk denote the
A. Game Refinement Measure for Winning Streak average number of obtained badges and the average number
of learned lessons. The explanation is detailed in the previous
The term ‘Winning Streak’ was initially used in sports [10]. study [8]. The results are shown in Table I.
It refers to a consecutive number of games won, which begins Table I shows that the game refinement value of Winning
with the third consecutive victory. In Duolingo, Winning Streak Streak increases through each milestone. It starts from 0.0047,
has been used as a measurement of how consistently learners increases after that and reaches 0.026 at the last milestone.
study. A winning streak of users starts at zero and increases by Similar to GR of Badge, GR of Winning Streak falls into
one as long as they complete to study their required number of the appropriate range in the end. This is another evidence
lessons. The winning streak will be reset to zero if the number which proves that the range between 0.02 and 0.03 should be
of studied lessons is missed. considered as the appropriate range for gamification domain.
For measuring the attractiveness of Winning Streak, we Whereas obtaining a badge is used as a gaming activity for
figure out the game progress model by extending the streak’s attracting beginner to acquire new knowledge, extending the
length of diligent learners. A Winning Streak can be obtained length of a Winning Streak is considered as enhancing normal
by completing lessons to reach the learning goal. Although it learning activity into serious gaming activity.
is easy to obtain, extending the length of Winning Streak is According to Figure 1, GR of Badge starts from the appro-
more difficult. For example, if we want to extend our Winning priate range of sophisticated games which means that users
Streak’s length to ten, we must pass the learning goal in ten are attracted to learn as playing a fun game at the beginning.
consecutive days. That is the reason why the longer winning However, GR decreases and maintains an appropriate range
is more precious. of gamification domain after that. Obtaining badges is an
Therefore, the main game progress model can be given by exciting activity but doing so frequently will become boring.
two factors: the number of learning consecutive days and the That explains why the attractiveness of Badge decrease after a
total number of days within a year. As the total number of certain period.
days in a year is 365 and the number of learning consecutive On the contrary, language learning is a normal activity.
days is the length of a Winning Streak. Thus, the measure of For making it enjoyable, Duolingo has been enhanced while
game refinement for game element Winning Streak at each introducing Winning Streak as a reward for diligent learners.
milestone is given by Eq. (2) where Wk is the average length The length of Winning Streak reflects user’s efforts. Therefore,
of a Winning Streak at a milestone of k. the longer Winning Streak is more precious. This streak is
√ shown on the learner’s profile, which reflects the ability of
Wk
GRW Sk = (2) learners in language learning. In consequence, the experimental
365

~ 2 ~
 2018 4th International Conference on Computer and Information Sciences (ICCOINS)

results indicated that Winning Streak is effective to help Milestone k Bk Lk GRBk PoSU
learning process. It helps Duolingo improve entertainment 1 7 31 0.085 10%
aspect when attractiveness of Badge decreases as suggested in 2 18 67 0.063 25%
[9]. 3 37 154 0.039 45%
4 78 358 0.024 67%

Table II
T HE RESULTS OF ANALYZING W INNING S TREAK IN ‘F RENCH FOR E NGLISH
S PEAKERS ’ COURSE

Milestone k Bk Lk GRBk PoSU

1 8 36 0.079 9%
2 17 76 0.054 34%
3 38 209 0.029 51%
4 66 405 0.020 67%

Table III
T HE RESULTS OF ANALYZING W INNING S TREAK IN ‘I TALIAN FOR E NGLISH
S PEAKERS ’ COURSE

Figure 1. Measures of game refinement at each milestone: Badge and Winning

Streak compared whereas that GR value is not maintained until the end. They
decrease sharply and fall into the range between 0.02 and 0.03.
Although three-course structures all include four milestones,
IV. M ILESTONE T ECHNIQUE IN L ANGUAGE C OURSE
the different divisions of learning content and number of
In this section, we compare the course structures between learning material in each course can affect the enjoyment
selected language courses to clarify the effects of well organized aspect. The shorter learning material and the small division in
course structure and the use of milestone technique. the first milestone of Spanish course make it more attractive
than other two courses (0.1166 in the beginning and 0.0246 in
A. Analyzing Course Structures of Three Language Courses
the end). That leads to some learners in this course is higher
Duolingo has more than 29 language courses in total. than other courses (109 million users compared to 64 million
However, this does not mean that we can compare any couple users in French course and 25 million users in Italian course).
language courses together. For instance, “Japanese for Chinese Thus, we claim that it is important to find appropriate settings
speakers” and “Japanese for English speakers” cannot be of course structure or learning process, which we call milestone
compared because it is easier for Chinese speakers to learn technique, to maintain the learner’s motivation.
Japanese than English speakers. Hence it is assumed that
Chinese speakers may be interested in learning Japanese rather
than English speakers, which is so-called ‘bias of a language
in learning’.
Therefore, to overcome this bias issue we carefully choose
the courses with the same difficulty in learning for our experi-
ment. The language difficulty ranking for English speakers
indicates that Spanish, French and Italian have the same
difficulty [11]. For that reason, we have chosen three courses:
“Spanish for English speakers”, “French for English speakers”
and “Italian for English speakers” as experiment candidates
in this study. To analyze the structure of these three courses
the data are collected in the same way as in [8]. Eq. (3) is
used to obtain the measurement of game refinement for each
milestone of the three courses.
B. Results and Discussion Figure 2. Measures of game refinement at each milestone: three different
courses compared
The results are shown in Tables I, II and III. The results
show that GR at the first milestone in Spanish course is 0.1166,
which is higher than other two courses (0.085 and 0.079), Conjecture 1: Different ways to split learning process make
and that GR of three courses are not much different at other different degrees of attractiveness for learners.
milestones. GR at the first milestone of three courses are Although the attractiveness of Badge in Spanish course is
equal to or greater than the zone value of sophisticated games, higher than others, its GR trend is less smoothly than French

~ 3 ~
 2018 4th International Conference on Computer and Information Sciences (ICCOINS)

and Italian courses as shown in Figure 2. The slope from the that Winning Streak successfully helps users enhance their
first to the second milestone in Spanish course is too sloping. regular learning activity into serious gaming activity. Moreover,
The large decrease of GR in that period would make users feel Winning Streak increases the motivation of advanced users
anxiety. From the game designer’s point of view, this is not when the attractiveness of Badge decreases. Thus this study
good for users. To prove that viewpoint, we analyze the slope of confirmed the effectiveness of game refinement measure to
GR trend. It is assumed that GRBk presents the attractiveness identify the merit of the appropriate division of learning process
of Badge. The strength of this attractiveness is derived from in order to maintain the attractiveness of learning activity.
how much users focus on this game element. Then, the effect of
ACKNOWLEDGEMENT
game element Badge decreases as users are attracted by other
game elements. In Duolingo, when users complete their lessons, This research is funded by a grant from the Japan Society
they not only achieve Badges but also have an opportunity to for the Promotion of Science, within the framework of the
obtain or extend their Winning Streak. We call such users, who Grant-in-Aid for Challenging Exploratory Research.
aim at obtaining or extending their Winning Streak, “streaking- R EFERENCES
users” and the percentage of streaking-users or PoSU in short
[1] J. McGonigal, Reality is broken: Why games make us better and how
reflect the decrease of users’ engagement with the game element they can change the world. Penguin, 2011.
Badge. We collect some statistics to figure out the PoSU, as [2] J. Brito, V. Vieira, and A. Duran, “Towards a framework for gamification
shown in Tables I, II and III. design on crowdsourcing systems: the game approach,” in Information
Technology-New Generations (ITNG), 2015 12th International Conference
Conjecture 2: When GR slope is more sloping, users may on, pp. 445–450, IEEE, 2015.
be easily attracted by other game elements. [3] O. B. Gené, M. M. Núñez, and Á. F. Blanco, “Gamification in mooc:
challenges, opportunities and proposals for advancing mooc model,” in
For example, the slope, which starts from the first milestone to Proceedings of the Second International Conference on Technological
the second milestone in Spanish course, is too sloping (GRBK Ecosystems for Enhancing Multiculturality, pp. 215–220, ACM, 2014.
slope value is 0.057). This implies that there are 41% streaking- [4] M. D. Hanus and J. Fox, “Assessing the effects of gamification in
the classroom: A longitudinal study on intrinsic motivation, social
users at the second milestone in Spanish course. However, there comparison, satisfaction, effort, and academic performance,” Computers
is 25% and 34% in French and Italian course respectively. & Education, vol. 80, pp. 152–161, 2015.
[5] L. Moccozet, C. Tardy, W. Opprecht, and M. Léonard, “Gamification-
Conjecture 3: The percentage of streaking-users is affected based assessment of group work,” in Interactive Collaborative Learning
by the game element Badge. (ICL), 2013 International Conference on, pp. 171–179, IEEE, 2013.
[6] H. Iida, N. Takeshita, and J. Yoshimura, “A metric for entertainment
For example, at the first milestone GR in Italian course is of boardgames: its implication for evolution of chess variants,” in
lowest among the three courses, whereas its PoSU is highest. Entertainment Computing, pp. 65–72, Springer, 2003.
In French course, GR at the first milestone is high enough to [7] H. Iida, K. Takahara, J. Nagashima, Y. Kajihara, and T. Hashimoto, “An
application of game-refinement theory to mah jong,” in International
attract the beginners, but its PoSU does not much differ from Conference on Entertainment Computing, pp. 333–338, Springer, 2004.
Spanish course. However, GR trend of French course is more [8] D. Huynh, L. Zuo, and H. Iida, “Analyzing gamification of “duolingo”
stable than others. That is a reason why the PoSU of French with focus on its course structure,” in Games and Learning Alliance,
pp. 268–277, Springer, 2016.
course is still lower than 50% at the third milestone, whereas [9] D. Huynh and H. Iida, “An analysis of winning streak’s effects in language
other courses are over 50%. This implies that users in French course of “duolingo”,” Asia-Pacific Journal of Information Technology
course are engaged with the game element Badge for a longer and Multimedia, vol. 6, no. 2, 2017.
[10] “Wiki–winning streak.” https://en.wikipedia.org/wiki/Winning streak
period than others. (sports),2017.
Conjecture 4: The increase of streaking-users is related to [11] “Language difficulty ranking.” http://www.effectivelanguagelearning.com/
language-guide/language-difficulty,2017.
the decrease of the attractiveness of Badge.
The PoSU in the three courses reaches 67% when GRBK falls
into the appropriate range. This means that the attractiveness
of the considered game element becomes low if its GR goes
down. This implies that users are no longer engaged with those
game elements. Therefore, the game refinement measure works
well in presenting the attractiveness of the game element in
the educational environment.
V. C ONCLUSION
In this paper, we investigated a way to assess the effects
of game elements in the domain of education. Educational
gamified language-learning platform ‘Duolingo’ is chosen as
the testbed, whereas game refinement measure is employed
as an assessment tool. Two game elements ‘Badge’ and
‘Winning Streak’ with respect to the course structure of three
language courses in Duolingo are analyzed in this study.
The results from the analysis with about 1000 users show

~ 4 ~