Professional Documents
Culture Documents
Effects of Gamification On Students' English Language Proficiency
Effects of Gamification On Students' English Language Proficiency
Review
Effects of Gamification on Students’ English Language
Proficiency: A Meta-Analysis on Research in South Korea
Je-Young Lee 1 and Minkyung Baek 2, *
1 Department of English Education, Jeonju University, Jeonju 55069, Republic of Korea; jylee@jj.ac.kr
2 Department of Home Economics Education, Jeonju University, Jeonju 55069, Republic of Korea
* Correspondence: bmk0419@jj.ac.kr
Abstract: This study presents a meta-analysis of research on the impact of gamification on English
language proficiency among South Korean students. Through an examination of 11 cases involving
610 participants, the study reveals a medium effect size (g = 0.517), suggesting that gamification
can significantly enhance English language learning outcomes. The analysis also reveals that theses
(g = 0.799) reported higher effect sizes than journal articles (g = 0.298), and that the absence of
technology in gamified learning interventions could potentially lead to larger effect sizes (g = 0.932).
Furthermore, the incorporation of points/scores and badges/rewards showed statistically significant
effects on student learning. The study found no significant differences in effect sizes when considering
grade, number of participants, weeks, sessions, sessions per week, and the number of gaming
elements. The results demonstrate varying impact of gamification across different subcomponents
of English proficiency, particularly in the learning of vocabulary, listening, and writing skills. The
findings underscore the potential of gamification as a tool for English language learning, but also call
for careful consideration in its design and implementation to maximize learning outcomes. Lastly,
we offer suggestions for future research and discuss the pedagogical implications of this study.
study perform better and retain the language for longer [5,6]. Hence, gamification can
significantly enhance not only the efficiency but also the overall amount of learning.
Nevertheless, the efficacy of gamification in foreign language education is not unan-
imously agreed upon. Numerous studies (e.g., [7–9]) have suggested that gamification
positively impacts learning outcomes and improves foreign language skills. Conversely,
other studies have highlighted no significant improvements or even potential negative
impacts on foreign language learning and behavioral aspects (e.g., [10–12]).
With this background, our study aspires to conduct a comprehensive meta-analysis on
the influence of gamification on English learning as a foreign language, with a particular
focus on experimental studies from South Korea. We aim to evaluate whether English
classes employing gamification demonstrate positive effects, and if so, quantify the extent
of these benefits. Simultaneously, we intend to scrutinize factors, particularly gaming
elements, that moderate the impact of gamification on learning. Prior to presenting the
methodology and results of our research, we will delve into the gaming elements, known for
significantly influencing the success or failure of gamification, and underline the importance
and uniqueness of the present study.
Table 1. Taxonomy of gaming elements (summarized from the findings of Toda et al. [17]).
Elements Description
Rewards learners for specific tasks, e.g., badges for completed
1. Acknowledgement
problems.
2. Level Hierarchical system providing new advantages as learners progress.
Performance/measurement 3. Progression Guides users about their advancement in the environment.
4. Point Basic feedback method, usually through scores or experience points.
Visual information about the learner’s performance or overall
5. Stats
environment.
1. Chance Involves uncertainty in outcomes.
2. Imposed Choice Requires users to make decisions for progress.
Ecological 3. Economy Represents transactions within the environment.
4. Rarity Involves limited resources to stimulate specific goals.
5. Time Pressure Applies time constraints but can disengage users.
Sustainability 2023, 15, 11325 3 of 19
Table 1. Cont.
Elements Description
1. Competition Involves user challenges to attain common goals.
2. Cooperation Encourages collaboration towards shared objectives.
Social
3. Reputation Relates to social status titles within a community.
4. Social Pressure Reflects the influence of social interactions on behavior.
1. Novelty Updates within the environment to maintain user engagement.
2. Objectives Goals providing a purpose for task completion.
3. Puzzle Cognitive challenges within the environment.
Personal
4. Renovation Opportunities for learners to redo tasks.
Enhances the experience using sensory stimulation. Lack of these
5. Sensation
may lead to demotivation.
1. Narrative Describes event sequences, influenced by user decisions.
Fictional
2. Storytelling Conveys the environment’s story, supporting the narrative.
2. Methods
2.1. Selection of Studies for Analysis
The scope of this meta-analysis is defined by studies that have investigated the influ-
ence of gamification on English proficiency, identified through a search on the Research
Information Sharing Service (RISS). RISS, a well-acknowledged Korean academic database,
provides a broad platform to perform meta-searches across prime academic databases in
Korea, such as KCI, DBpia, e-article, KISS, Kyobo Scholar, etc. Furthermore, RISS accom-
modates almost all of Master’s theses and Doctoral dissertations from universities in South
Korea. The search, conducted in April 2023, utilized keywords such as ‘gamification’ and
‘gamified learning’, along with their Korean equivalents.
Sustainability 2023, 15, 11325 5 of 19
The initial collection phase accumulated a total of 237 studies: 154 from academic
Sustainability 2023, 15, x FOR PEER REVIEW 5 of 19
journals and 83 theses. A series of refining steps, as depicted in Figure 1, were then applied
to this primary collection, following the PRISMA statement [33].
Figure 1. PRISMA
Figure1. PRISMA flowchart
flowchart of
of the
the article
article screening
screening process.
process.
In
In the
the selection
selection process,
process, inclusion and exclusion
inclusion and exclusion criteria
criteriawere
wereestablished
establishedwith
withrefer-
refer-
ence to previous studies [2,34–36], creating a definitive framework for this meta-analysis:
ence to previous studies [2,34–36], creating a definitive framework for this meta-analysis:
(a)
Onlystudies
Only studies that
that empirically
empirically investigated
investigated gamified
gamified practices
practices werewere included,
included, ex-
exclud-
cluding any that simply discussed or described gamification without
ing any that simply discussed or described gamification without empirical backing. empirical
backing.
The focus was placed on studies conducted in K-12 or higher education settings. Any
(b) The focus was placed on studies conducted in K-12 or higher education settings.
studies where interventions were held in out-of-school environments, such as private
Any studies where interventions were held in out-of-school environments, such as
tutoring or extracurricular academies, were omitted.
private tutoring or extracurricular academies, were omitted.
Included studies were required to objectively measure students’ English proficiency
(c) Included studies were required to objectively measure students’ English proficiency
after the treatment. Studies were excluded if they solely relied on self-reported data
after the treatment. Studies were excluded if they solely relied on self-reported data
about students learning achievements.
about students learning achievements.
Studies had to explicitly mention the usage of at least one game element. Those not
(d) Studies had to explicitly mention the usage of at least one game element. Those not
specifying the game elements used were excluded.
specifying the game elements used were excluded.
Finally, studies were excluded if their datasets or results lacked sufficient information
(e) Finally, studies were excluded if their datasets or results lacked sufficient information
for effect size calculations, such as missing sample size data or mean scores without
for effect size calculations, such as missing sample size data or mean scores without
corresponding standardized deviation values.
corresponding standardized deviation values.
2.2. Codebook
The objective of this study is to explore the impact of gamification on EFL learners’
English proficiency. In order to do so, each study is classified according to analysis criteria,
as displayed in Table 2. Two independent researchers conducted the analysis, reconciling
any differences in coding outcomes via discussion and consensus.
Sustainability 2023, 15, 11325 6 of 19
2.2. Codebook
The objective of this study is to explore the impact of gamification on EFL learners’
English proficiency. In order to do so, each study is classified according to analysis criteria,
as displayed in Table 2. Two independent researchers conducted the analysis, reconciling
any differences in coding outcomes via discussion and consensus.
Category Variables
1. Publication Type (1) journal article (2) MA thesis
2. Experimental Design (1) quasi-experimental (2) pre-experimental
3. School Level (1) primary school (2) secondary school (3) university
4. Technology Use (1) used (2) not used
(1) avatar/character (2) point/score (3) leaderboard/scoreboard (4) feedback
5. Gaming Element
(5) level (6) collaboration (7) mission/challenge (8) story/fiction (9) badge/reward
6. Grade Raw data
7. Number of Participants Raw data
8. Weeks Raw data
9. Sessions Raw data
10. Sessions per Weeks Raw data
11. Number of Gaming Elements Raw data
12. Dependent Variables (1) listening (2) speaking (3) reading (4) writing (5) vocabulary (6) achievement
To begin with, let us explain the moderating variables, which are classified under a
nominal scale. The ‘publication type’ is the first criterion, differentiating between studies
published in academic journals and those presented as master’s theses. Notably, no doctoral
dissertation was included in the final meta-analysis.
The second criterion, ‘experimental design’, is employed to categorize the study
designs investigating the effects of gamification. As per Brown’s classification [37] for the
L2 field, experimental study designs are divided into: (1) true-experimental design, using
both random sampling and control groups, (2) quasi-experimental design, only employing
control groups without random sampling, and (3) pre-experimental design, which lacks
both random sampling and control groups. This study adopts these categories. However,
it should be noted that this study is confined to experimental studies carried out in K-12
and universities, where random sampling is infrequently applied. Therefore, none of the
studies featured random sampling.
The ‘school level’ is the third criterion, categorizing the setting of the experiment
into primary school, secondary school, and university. Initially, a distinction was planned
between middle and high schools, but no middle school studies were included in the final
meta-analysis.
The fourth criterion, ‘technology use’, differentiates between studies that incorpo-
rated technology-related elements, such as computers, tablets, or applications, in their
gamification process, and those that did not.
The fifth criterion, ‘gaming element’, classifies the elements utilized in each gami-
fied learning experiment, only recording those explicitly mentioned in each study, and
recategorizing these according to the classifications of prior studies [2,17,34,38–40].
Items 6–11 correspond to continuous scale (interval scale) moderators, with the results
for each case entered as raw data (numerical data). In the ‘grade’ category, the grades of
experiment participants are recorded. In studies where multiple grades are involved in the
experiment [17,41], an average grade is calculated based on the ratio of students by grade
and recorded.
Sustainability 2023, 15, 11325 7 of 19
For ‘number of participants’, the total number of students from both the experimental
and control groups are added up. While conducting a meta-analysis, both approaches
can be utilized, depending on the research question and the context of the study. Yet, it is
typically more common to use the total number of participants from both groups, especially
when determining effect sizes such as standardized mean differences. These effect sizes
rely on the variability within both groups, and larger sample sizes usually offer more
dependable estimates of the effect size [27].
The set of items—‘weeks’, ‘sessions’, and ‘sessions per weeks’—evaluates the influence
of treatment duration on gamified learning. Although all studies in this meta-analysis
provided information about the week, some studies [17,42] did not provide session infor-
mation. There are various methods for handling missing values, such as the likewise or
pairwise deletion [43], model-based method [44], and imputation method [45,46]. In this
study, regression imputation, which can predict missing values based on other variables
and maintain the relationship between variables as much as possible, was used.
The final set of items, ‘dependent variables’, assesses the English-proficiency-related
variables under examination for the effects of gamification. There were a total of six de-
pendent variables among the studies targeted in this meta-analysis: listening, speaking,
reading, writing, vocabulary, and achievement. The results of each study’s coding, classified
by these criteria, are presented in Appendices A and B.
(Q = 30.848, df = 10, p = 0.001). Notably, about two-thirds of the observed variability in the
results across studies (indicated by the I2 statistic of 67.583) was due to true differences in
effect sizes rather than merely sampling errors. Considering the typical interpretation of
Sustainability 2023, 15, x FOR PEER REVIEW 8 of 19
I2 values—25%, 50%, and 75%, corresponding to low, moderate, and high heterogeneity,
respectively [53], our study unveiled a moderate to high level of heterogeneity.
Card provides three potential options for instances where significant heterogeneity is
Card
identified: provides three potential
(1) disregarding options for instances
the heterogeneity where significant
and proceeding heterogeneity
with analysis as if the data
iswere
identified: (1) disregarding the heterogeneity and proceeding with
homogeneous, an approach that is typically considered least justifiable; analysis as if the
(2)data
undertak-
were homogeneous, an approach that is typically considered least justifiable;
ing moderator analyses, which leverage the coded characteristics of the studies, such as (2) under-
taking moderator analyses,
methodological features orwhich leverage
sample the coded
attributes, characteristics
to predict variances of in
theeffect
studies, such
sizes between
asstudies;
methodological
or (3) adopting an alternative to the Fixed Effect Model, specifically thebe-
features or sample attributes, to predict variances in effect sizes Random
tween studies; or (3) adopting an alternative to the Fixed Effect Model, specifically the
Effect Model, which conceptualizes the population effect size as a distribution rather than
Random Effect Model, which conceptualizes the population effect size as a distribution
a fixed point (pp. 184–185) [30]. In response to the detected heterogeneity in this study,
rather than a fixed point (pp. 184–185) [30]. In response to the detected heterogeneity in
we applied the Random Effects Model to compute the mean effect size, thus mitigating
this study, we applied the Random Effects Model to compute the mean effect size, thus
potential errors attributed to the heterogeneity. We also pursued further investigation into
mitigating potential errors attributed to the heterogeneity. We also pursued further inves-
the heterogeneity sources through a subgroup analysis and meta-regression analysis, based
tigation into the heterogeneity sources through a subgroup analysis and meta-regression
on the characteristics of the moderator.
analysis, based on the characteristics of the moderator.
Subsequently,
Subsequently, wewe employed
employed Hedges’
Hedges’ g togdetermine
to determine the individual
the individual effecteffect
size forsize for each
each
study. As shown in Figure 2, the resulting individual effect sizes displayed a range from from
study. As shown in Figure 2, the resulting individual effect sizes displayed a range
−0.26
−0.26 to to 1.26.
1.26. Among
Among these,
these, we we observed
observed ninenine instances
instances of positive
of positive effect effect sizestwo
sizes and andoftwo of
negative, indicating that most studies reported beneficial
negative, indicating that most studies reported beneficial outcomes. outcomes.
Forest
Figure2.2.Forest
Figure plot
plot [11,18,19,41,42,54–56].
[11,18,19,41,42,54–56].
Theoverall
The overall mean
mean effect
effect size,
size, computed
computed via random-effects
via the the random-effects model, model, was 0.517,
was 0.517,
whichsignifies
which signifiesa medium
a medium effect
effect size
size (N (N = 610,
= 610, k =g11,
k = 11, g = 0.517,
= 0.517, se = 0.139,
se = 0.139, CI = 0.245–0.790,
CI = 0.245–0.790,
ZZ= =3.718,
3.718,p =p0.000).
= 0.000).
ThisThis
impliesimplies that
that the the employment
employment of gamification-based
of gamification-based English in-English
instruction
struction can can
leadlead
to a to a medium-to-large
medium-to-large improvement
improvement in English
in English proficiency.
proficiency.
As
Asnoted
notedininthe
theprevious
previous discussion
discussion ononrelated literature,
related nono
literature, quantitative
quantitativemeta-anal-
meta-analysis
ysis
hashas
beenbeen conductedsolely
conducted solelyon onthe
the impact
impact of ofgamification
gamificationonon foreign
foreignlanguage
languagelearning
learning to
todate.
date.Nonetheless,
Nonetheless,Bai Baietetal.
al.[2]
[2]have
have documented
documented that gamificationin
that gamification indiverse
diverseeduca-
educational
tional contexts
contexts correlates
correlates withwith a medium
a medium effect
effect sizesize
(g =(g = 0.504)
0.504) on academic
on academic achievements.
achievements. Interest-
Interestingly, they reported a slightly smaller effect size (g = 0.377) for language-related
ingly, they reported a slightly smaller effect size (g = 0.377) for language-related disciplines.
disciplines.
While theirWhile
studytheir study
did not did not
isolate isolate
foreign foreign languages,
languages, particularlyparticularly English,
English, a larger a size
effect
larger effect size emerged from the findings of the present study. In reviewing the collec-
tive findings of past meta-analyses that investigated the influence of gamification on aca-
demic achievements across a range of subjects, inclusive of foreign languages, the
Sustainability 2023, 15, 11325 9 of 19
emerged
Sustainability 2023, 15, x FOR PEER REVIEW from the findings of the present study. In reviewing the collective findings 9 of of
19 past
meta-analyses that investigated the influence of gamification on academic achievements
across a range of subjects, inclusive of foreign languages, the reported effect sizes typically
fall into the medium effect size ([34]: g = 0.464; [35]: g = 0.49; [36]; g = 0.557). Remarkably,
reported effect sizes typically fall into the medium effect size ([34]: g = 0.464; [35]: g = 0.49;
these reported effect sizes are closely aligned with the findings from the current study. As
[36]; g = 0.557). Remarkably, these reported effect sizes are closely aligned with the find-
such, the evidence suggests that the application of gamification could result in a moder-
ings from the current study. As such, the evidence suggests that the application of gami-
ate increase in academic achievements across various disciplines, including English as a
fication could result in a moderate increase in academic achievements across various dis-
Foreign
ciplines,Language.
including English as a Foreign Language.
3.2. Publication Bias
3.2. Publication Bias
Publication bias was
Publication bias wasassessed
assessedusing
usingvarious
various statistical
statistical analyses,
analyses, including
including a funnel
a funnel
plot with Egger’s Regression Test, Begg’s Rank Correlation Test, Trim
plot with Egger’s Regression Test, Begg’s Rank Correlation Test, Trim and Fill Method,and Fill Method,
and
and fail-safe N analysis. Each study’s effect size and standard error were derived, and and
fail-safe N analysis. Each study’s effect size and standard error were derived, the the
resulting
resulting funnel plot (Figure
funnel plot (Figure3)3)showed
showedconsiderable
considerable symmetry.
symmetry. TheThe results
results fromfrom Egger’s
Egger’s
Regression Test were
Regression Test werenotnotstatistically
statistically significant
significant (z =(z = −0.5410,
−0.5410, p = 0.589),
p = 0.589), suggesting
suggesting no ev- no
evidence
idence ofof funnel
funnel plot
plot asymmetry
asymmetry or publication
or publication bias bias inmeta-analysis
in this this meta-analysis
[31]. [31].
Figure3.
Figure 3. Funnel
Funnel plot.
plot.
The Begg’s
The Begg’s Rank
RankCorrelation
Correlation Test measured
Test measured the correlation
the correlationbetween the ranks
between theofranks
ef- of
fect sizes
effect sizesand
andtheir
theirvariances.
variances. AA negative
negativeKendall’s
Kendall’stautau (−0.2727)
(−0.2727) suggests smaller
suggests studies
smaller studies
tendedtotoshow
tended showsmaller
smallereffects,
effects,but
butthe
therelationship
relationshipisisnot
notstrong.
strong.The
Thep-value
p-valueofof0.283,
0.283,
being
being greater
greater thanindicates
than 0.05, 0.05, indicates the asymmetry
the asymmetry observed
observed in the
in the funnel
funnel plot
plot is isnot
notstatistically
statis-
tically significant.
significant. Thus, is
Thus, there there is insufficient
insufficient evidence
evidence of publication
of publication biasbias based
based onon this
this test
test [57].
[57].The Trim and Fill method, a non-parametric technique, estimates potentially missing
studiesThe Trim
due to and Fill method,
publication biasaby
non-parametric
assessing thetechnique,
funnel plot’sestimates potentially
symmetry missing
[58]. The method
identified no missing studies on the right side of the funnel plot (SE = 2.1765),method
studies due to publication bias by assessing the funnel plot’s symmetry [58]. The suggesting
identified
no no missing
substantial studies
publication on the right side of the funnel plot (SE = 2.1765), suggesting
bias.
no substantial publication bias.
Finally, the fail-safe N was calculated using the Rosenthal approach. The fail-safe N
Finally, the fail-safe N was calculated using the Rosenthal approach. The fail-safe N
method estimates the number of unpublished studies with null findings that would need
method estimates the number of unpublished studies with null findings that would need
to exist to make the meta-analysis finding statistically non-significant. The fail-safe N of
to exist to make the meta-analysis finding statistically non-significant. The fail-safe N of
this meta-analysis is 197, exceeding Rosenthal’s threshold of 5k + 10 (65 for this study) [59].
this meta-analysis is 197, exceeding Rosenthal’s threshold of 5k + 10 (65 for this study)
Therefore, given the observed significance level of less than 0.0001, the meta-analysis results
[59]. Therefore, given the observed significance level of less than 0.0001, the meta-analysis
appear robust and not overly influenced by publication bias.
results appear robust and not overly influenced by publication bias.
Sustainability 2023, 15, 11325 10 of 19
1. Publication type
Upon assessing the impact of gamification by the type of publication, it was observed
that master’s theses (g = 0.799) exhibited a higher effect size than that of journal articles
(g = 0.289). Statistical significance was confirmed for these differences via a homogeneity
test (Q = 6.426, df = 1, p = 0.011). Interestingly, these findings are in stark contrast to those
presented by Huang et al. [34], where journal articles (g = 0.662) and conference proceedings
(g = 0.666) recorded a medium-to-large effect size, while dissertations/theses exhibited a
negative effect size (g = −0.170).
Numerous studies concerning publication bias warn of an increased propensity for
published studies to exhibit positive effects in comparison to their unpublished counter-
parts, often attributing such tendencies to journals’ bias towards positive results [60,61],
researchers’ bias towards positive results [62], and the phenomenon of ‘salami slicing’ [63].
However, in an interesting deviation from these widely held perspectives, the findings of
this study contradict the commonly perceived pattern.
A potential explanation for these findings might be rooted in the research settings.
Specifically, journal articles, which tended to report relatively lower effects, were predomi-
nantly conducted in university environments. On the other hand, the theses that reported
larger effect sizes were chiefly carried out in elementary school settings. This disparity in
research settings across different types of sources may account for the observed differences.
2. Experimental design
The current study, following Brown’s classification [37], divided the experimental
design into two types: quasi-experimental and pre-experimental. In the quasi-experimental
Sustainability 2023, 15, 11325 11 of 19
design, while no random sampling was conducted, both control and experimental groups
were set up. In contrast, the pre-experimental design only included an experimental group,
without implementing any random sampling. The results displayed a medium-to-large
effect size (g = 0.778) in the pre-experimental design and a small-to-medium effect size
(g = 0.475) in the quasi-experimental design. However, this difference was not statistically
significant (Q = 2.953, df = 1, p = 0.103).
Sailer and Hommer’s study [35] utilized the presence of randomization, differentiating
between the true-experimental design and quasi-experimental design, as a moderating
variable in their meta-analysis on the impact of gamification on academic achievement.
Their findings revealed a larger effect size in the quasi-experimental design (g = 0.56) than
in the true-experimental design (g = 0.29). Upon synthesizing the findings of both studies,
a trend became evident: as experimental designs become stricter, the effect sizes reported
tend to decrease.
3. School level
A comparison of the effects of gamification was conducted in the educational envi-
ronments of primary school, secondary school, and university. The results revealed that
gamification had the largest effect size in primary schools, with a value of 0.801. The effect
size was slightly lower in secondary schools (g = 0.624) and further reduced in universities
(g = 0.270). Although there was a decreasing trend in effect sizes as the school level in-
creased, these differences were not statistically significant (Q = 5.589, df = 2, p = 0.061). This
finding aligns with a previous meta-analytic study that examined the influence of gami-
fication on academic achievement [2,34,36], which also found no statistically significant
differences among school levels.
The observed pattern of decreasing effect sizes from primary schools to universities
could be attributed to various factors. To begin, primary school students generally demon-
strate a keen interest in play and games, which makes them potentially more receptive
to gamification in education. The familiarity and engagement with digital games outside
school settings might facilitate the transference of this enthusiasm into the classroom when
gamified methods are adopted [64].
In contrast, as we ascend the educational ladder to secondary schools and universities,
students’ learning preferences and motivations may undergo significant shifts. Older
students, for instance, could perceive gamification elements, such as rewards, points,
and badges, as somewhat infantile or even as distractions from their main educational
objectives. Intrinsically motivated, these students often place greater value on autonomous
and self-directed learning experiences, exhibiting a tendency to favor traditional teaching
methodologies over gamified ones [65,66].
While these considerations offer insight into the decreasing trend of effect sizes, they
also highlight the importance of adapting educational strategies to cater to the evolving
needs and preferences of students at different stages of their academic journey. Therefore,
the effective implementation of gamification in education, across all levels, may require a
nuanced approach tailored to the age and preferences of the learners.
4. Technology use
In this moderator, we examined the impact of gamification on learning outcomes,
particularly differentiating between instances where technology such as computers, tablets,
or applications were utilized and those where no technology was used. The outcomes
yielded a noticeable distinction between the two groups. In scenarios where technology
was absent, we observed an effect size of g = 0.932, surpassing the range defined as a large
effect size. However, when technology was incorporated, the effect size (g = 0.383) fell
within the range of a small-to-medium effect size. The difference in these results was found
to be statistically significant (Q = 4.171, df = 1, p = 0.041).
The results in this moderator analysis, demonstrating higher effect sizes in non-
technology gamified interventions, appear to be in opposition to the findings of Yıldırım’s
research, which did not identify a significant difference between technology-based and
Sustainability 2023, 15, 11325 12 of 19
Among the nine subvariables, statistically significant differences were found for
point/score and badge/reward. First, in the case of point/score, the use of this ele-
ment corresponded to a small-to-medium effect size of 0.340, while the absence of its
use corresponded to a large effect size of 0.840 (Q = 6.235, df = 1, p = 0.013). Secondly,
Sustainability 2023, 15, 11325 13 of 19
for ‘badge/reward’, there was a difference of about 0.48 between cases where it was ap-
plied (g = 0.832) and not applied (g = 0.351), which was statistically significant (Q = 5.088,
df = 1, p = 0.024). On the other hand, no statistically significant differences were found
for the other variables, such as avatar/character, leaderboard/scoreboard, feedback, level,
collaboration, mission/challenge, and story/fiction.
The findings of Hamari, Koivisto, and Sarsa [68], which highlighted the capacity
of various gaming elements, notably points, badges, and leaderboards, to boost student
motivation and subsequently enhance academic performance. The prominent effects of
these distinct gaming elements suggest that they potentially stimulate students’ intrinsic
motivation by furnishing palpable markers of progress and accomplishments. Contrast-
ingly, Huang and colleagues [34] did not identify any significant differences among 14
gaming elements, including leaderboards, badges/awards, points/experiences, and ad-
vancements/levels. Similarly, Bai et al. [2] found no statistical significance when comparing
effects across types of gaming elements.
In view of these divergent findings, while points/scores and badges/rewards seem to
be influential components within gamified learning environments, educators are urged to
consider the comprehensive design of the learning experience. This includes contemplating
the synergistic effects among various gaming elements and aligning them with the specific
needs and preferences of learners. It is imperative to underscore that the effectiveness of
these gaming elements may fluctuate depending on the context and the characteristics of
the learners [14].
Variable b SE 95% CI Z p
Intercept −13.8558 44.5029 −101.0800~73.3683 −0.3113 0.756
Grade 0.0115 0.3317 −0.6387~0.6616 −0.0345 0.973
No. of participants 0.0034 0.0313 −0.0579~0.0648 0.1098 0.913
Weeks 1.1266 3.2314 −5.2069~7.4601 0.3486 0.747
Sessions −0.7337 2.1939 −5.0336~3.5662 −0.3344 0.738
Sessions per week 9.1047 28.205 −45.8144~64.0238 0.3249 0.745
No. of gaming elements 0.1789 0.2491 −0.3093~0.6672 0.7183 0.473
to 1 month, and g = −0.278 for 1 semester or longer. Similarly, Sailer and Hommer [35]
did not find any statistically significant difference when conducting a subgroup analysis
according to the duration of intervention. The fact that the largest effect sizes were not
found in the largest sample size or the longest intervention duration groups implies that
‘more’ is not necessarily ‘better’ in the context of gamified learning. These non-linear
relationships concerning the effect size, sample size, intervention duration, and other
variables may point to the existence of ‘sweet spots’ that might be optimal for gamified
learning implementations.
suggesting that more research is needed to fully understand this result. To sum up, more
nuanced research could investigate further how different elements of gamified learning
might be more or less effective for different language skills [71].
Author Contributions: Conceptualization, J.-Y.L. and M.B.; methodology, J.-Y.L. and M.B.; software,
J.-Y.L.; validation, J.-Y.L. and M.B.; formal analysis, J.-Y.L. and M.B.; investigation, J.-Y.L. and M.B.;
resources, J.-Y.L. and M.B.; data curation, J.-Y.L. and M.B.; writing—original draft preparation, J.-Y.L.
and M.B.; writing—review and editing, J.-Y.L. and M.B.; visualization, J.-Y.L.; supervision, J.-Y.L.;
project administration, J.-Y.L. and M.B. All authors have read and agreed to the published version of
the manuscript.
Funding: This research received no external funding.
Data Availability Statement: The data presented in this study are available on request from the
corresponding author.
Conflicts of Interest: The authors declare no conflict of interest.
References
1. Zimmerman, E. Position statement: Manifesto for a ludic century. In The Gameful Word: Approaches, Issues, Applications; Walz, S.P.,
Deterding, S., Eds.; The MIT Press: Cambridge, MA, USA, 2015; pp. 19–22.
2. Bai, S.; Hew, K.F.; Huang, B. Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis
of qualitative data in educational contexts. Educ. Res. Rev. 2020, 30, 100322. [CrossRef]
3. Gee, J.P. What Video Games Have to Teach Us about Learning and Literacy; Palgrave Macmillan: New York, NY, USA, 2003.
4. Burke, B. Gamify: How Gamification Motivates People to Do Extraordinary Things; Bibliomotion: New York, NY, USA, 2014.
5. Le-Thi, D.; Dörnyei, Z.; Pellicer-Sánchez, A. Increasing the effectiveness of teaching L2 formulaic sequences through motivational
strategies and mental imagery: A classroom experiment. Lang. Teach. 2020, 26, 136216882091312. [CrossRef]
6. Philp, J.; Duchesne, S. Exploring engagement in tasks in the language classroom. Annu. Rev. Appl. Linguist. 2016, 36, 50–72.
[CrossRef]
Sustainability 2023, 15, 11325 17 of 19
7. Berns, A.; Isla-Montes, J.; Palomo-Duarte, M.; Dodero, J. Motivation, students’ needs and learning outcomes: A hybrid game-based
app for enhanced language learning. SpringerPlus 2016, 5, 1305. [CrossRef] [PubMed]
8. Hung, H. Clickers in the flipped classroom: Bring your own device (BYOD) to promote student learning. Interact. Learn. Environ.
2017, 25, 983–995. [CrossRef]
9. Purgina, M.; Mozgovoy, M.; Mozgovoy, M. WordBricks: Mobile technology and visual grammar formalism for gamification of
natural language grammar acquisition. J. Educ. 2019, 58, 073563311983301. [CrossRef]
10. Cardoso, W.; Rueb, A.; Grimshaw, J. Can an interactive digital game help French learners improve their pronunciation? In CALL
in a Climate of Change: Adapting to Turbulent Global Conditions—Short Papers from EUROCALL 2017; Borthwick, K., Bradley, L.,
Thouësny, S., Eds.; Research-Publishing.Net: Dublin, Ireland, 2017; pp. 67–72.
11. Lee, J. Effects of using gamification-based quiz on recalling formulaic sequences. Int. Promot. Agency Cult. Technol. 2022, 8,
589–596. [CrossRef]
12. Munday, P. The case for using Duolingo as part of the language classroom experience. RIED 2016, 19, 83–101. [CrossRef]
13. Koivisto, J.; Hamari, J. The rise of motivational information systems: A review of gamification research. Int. J. Inf. Manag. 2019,
45, 191–210. [CrossRef]
14. Sailer, M.; Hense, J.U.; Mayr, S.K.; Mandl, H. How gamification motivates: An experimental study of the effects of specific game
design elements on psychological need satisfaction. Comput. Hum. 2017, 69, 371–380. [CrossRef]
15. Dignan, A. Game Frame: Using Games as a Strategy for Success; Free Press: New York, NY, USA, 2011.
16. Hanus, M.D.; Fox, J. Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social
comparison, satisfaction, effort, and academic performance. Comput. Educ. 2014, 80, 152–161. [CrossRef]
17. Toda, A.M.; Klock, A.C.T.; Oliveira, W.; Palomino, P.T.; Rodrigues, L.; Shi, L.; Bittencourt, I.; Gasparini, I.; Isotani, S.; Cristea, A.I.
Analysing gamification elements in educational environments using an existing Gamification taxonomy. Smart Learn. Environ.
2019, 16, 6. [CrossRef]
18. Kim, S. Effects of a gamified learning environment on learning experiences: A case study of a general English course using
relative evaluation policy. MALL 2014, 17, 68–94.
19. Laffey, D. Gamification and EFL writing: Effects on student motivation. ETAK 2022, 28, 23–42. [CrossRef]
20. Toda, A.M.; Valle, P.H.D.; Isotani, S. The dark side of gamification: An overview of negative effects of gamification in education.
In Higher Education for All: From Challenges to Novel Technology-Enhanced Solutions; Cristea, A., Bittencourt, I., Lima, F., Eds.;
Springer: Cham, Switzerland, 2018; pp. 143–156.
21. Lee, J.; Hammer, J. Gamification in education: What, how, why bother? Acad. Exch. Q. 2011, 15, 146.
22. Kwon, B.; Lyou, C. The meta-analysis or domestic gamification research: Status and suggest. Humancon 2015, 39, 97–124.
[CrossRef]
23. Ministry of Education. 2022 Revised English Language Curriculum; Ministry of Education: Sejong, Republic of Korea, 2022.
24. Kim, S.; Lee, Y. Development of TPACK-P education program for improving technological pedagogical content knowledge of
pre-service teacher. J. Korea Soc. Comput. Inf. 2017, 22, 141–152.
25. Yi, S.; Lee, Y. The effects of software education teaching efficacy (SE-TE) of in-service teachers on backward design based TPACK-P
teachers’ training program. KACE 2019, 22, 113–121.
26. Kim, Y.M. Pre-service English teachers’ mobile information and communication technology-technological pedagogy and content
knowledge. FLE 2018, 25, 1–25. [CrossRef]
27. Borenstein, M.; Hedges, L.V.; Higgins, J.P.T.; Rothstein, H.R. Introduction to Meta-Analysis; John Wiley & Sons: West Sussex,
UK, 2009.
28. Morrison, A.; Polisena, J.; Husereau, D.; Moulton, K.; Clark, M.; Fiander, M.; Mierzwinski-Urban, M.; Clifford, T.; Hutton, B.;
Rabb, D. The effect of English-language restriction on systematic review-based meta-analyses: A systematic review of empirical
studies. Int. J. Technol. Assess. Health Care 2012, 28, 138–144. [CrossRef]
29. Pieper, D.; Puljak, L. Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility
criteria if necessary. J. Clin. Epidemiol. 2021, 132, 146–147. [CrossRef] [PubMed]
30. Card, N.A. Applied Meta-Analysis for Social Science Research; Guildord Press: New York, NY, USA, 2012.
31. Egger, M.; Smith, G.D.; Schneider, M.; Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997, 315,
629–634. [CrossRef] [PubMed]
32. Sutton, A.J.; Duval, S.J.; Tweedie, R.L.; Abrams, K.R.; Jones, D.R. Empirical assessment of effect of publication bias on meta-
analyses. BMJ 2000, 320, 1574–1577. [CrossRef] [PubMed]
33. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA
statement. BMJ 2009, 339, b2535. [CrossRef]
34. Huang, R.; Ritzhaupt, A.D.; Sommer, M.; Zhu, J.; Stephen, A.; Valle, N.; Hampton, J.; Li, J. The impact of gamification in
educational settings on student learning outcomes: A meta-analysis. Educ. Technol. Res. Dev. 2020, 68, 1875–1901. [CrossRef]
35. Sailer, M.; Hommer, L. The gamification of learning: A meta-analysis. Educ. Psychol. Rev. 2020, 32, 77–112. [CrossRef]
36. Yıldırım, İ.; Şen, S. The effects of gamification on students’ academic achievement: A meta-analysis study. Interact. Learn. Environ.
2021, 29, 1301–1318. [CrossRef]
37. Brown, J.D. Understanding Research in Second Language Learning: A Teacher’s Guide to Statistics and Research Design; Cambridge
University Press: Cambridge, UK, 1988.
Sustainability 2023, 15, 11325 18 of 19
38. Dichev, C.; Dicheva, D. Gamifying education: What is known, what is believed and what remains uncertain: A critical review. Int.
J. Educ. Technol. High. Educ. 2017, 14, 9. [CrossRef]
39. Kim, J.; Castelli, D.M. Effects of gamification on behavioral change in education: A meta-analysis. Int. J. Environ. Res. Public
Health 2021, 18, 3550. [CrossRef]
40. Landers, R.N. Developing a theory of gamified learning: Linking serious games and gamification of learning. Simul. Gaming 2014,
45, 752–768. [CrossRef]
41. Ahn, Y. The Effects of a Phonemic Awareness Activity Class with Gamification on English Phonemic Awareness and Affective
Domains in Elementary School Students, and Observation for the Influence of the Native Language on Phonemic Awareness.
Master’s Thesis, Cyber Hankuk University of Foreign Studies, Seoul, Republic of Korea, 2019.
42. Lee, S. The effects of Gamification-based Artificial Intelligence Chatbot activities on elementary English learners’ speaking
performance and affective domains. Korean Soc. Elem. Engl. Educ. 2019, 25, 75–98. [CrossRef]
43. Little, R.J.A.; Rubin, D.B. Statistical Analysis with Missing Data, 3rd ed.; Wiley: Hoboken, NJ, USA, 2019.
44. Schafer, J.; Graham, J.W. Missing data: Our view of the state of the art. Psychol. Methods 2002, 7, 147–177. [CrossRef]
45. Allison, P.D. Missing Data; Sage: Thousand Oaks, CA, USA, 2002.
46. Rubin, D.B. Multiple Imputations for Nonresponse in Surveys; John Wiley & Sons: New York, NY, USA, 1987.
47. Cummings, G. Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis; Routledge: New York, NY,
USA, 2012.
48. Cohen, J. Statistical Power for the Behavioral Sciences, 2nd ed.; Academic Press: New York, NY, USA, 1988.
49. Ferguson, C.J. An effect size primer: A guide for clinicians and researchers. Prof. Psychol. Res. Pract. 2009, 40, 532–538. [CrossRef]
50. Lakens, D. Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs.
Front. Psychol. 2013, 4, 863. [CrossRef]
51. Plonsky, L.; Oswald, F.L. How big is “big”? Interpreting effect sizes in L2 research. Lang. Learn. 2014, 64, 878–912. [CrossRef]
52. Sawilowsky, S.S. New effect size rules of thumb. J. Mod. Appl. Stat. Methods 2009, 8, 597–599. [CrossRef]
53. Higgins, J.P.T.; Thompson, S.G.; Deeks, J.J.; Altman, D.G. Measuring inconsistency in meta-analyses. BMJ 2003, 327, 557–560.
[CrossRef]
54. Kim, H. A Study of the Improvement of Elementary Students’ English Vocabulary Through Gamification Using Kahoot Applica-
tion. Master’s Thesis, Woosuk University, Jeonbuk, Republic of Korea, 2023.
55. Baek, J. The Effects of Using Gamification on Primary School Students’ Learning English: Based on Students’ Acquisition
of English Vocabulary and Their Affective Attitudes on English. Master’s Thesis, Chinju National University of Education,
Gyeongnam, Republic of Korea, 2021.
56. Jeon, W. The Effects of Gamification Using Classcard and Class123 on the English Vocabulary Proficiency and the Affective
Domain for High School Students. Master’s Thesis, Cyber Hankuk University of Foreign Studies, Seoul, Republic of Korea, 2021.
57. Begg, C.B.; Mazumdar, M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994, 50, 1088–1101.
[CrossRef]
58. Duval, S.; Tweedie, R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in
meta-analysis. Biometrics 2000, 56, 455–465. [CrossRef]
59. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 1979, 86, 638–641. [CrossRef]
60. Dickersin, K. The existence of publication bias and risk factors for its occurrence. JAMA 1990, 263, 1385–1389. [CrossRef]
61. Dwan, K.; Gamble, C.; Williamson, P.R.; Kirkham, J.J. Systematic review of the empirical evidence of study publication bias and
outcome reporting bias: An updated review. PLoS ONE 2013, 8, e66844. [CrossRef] [PubMed]
62. Franco, A.; Malhotra, N.; Simonovits, G. Publication bias in the social sciences: Unlocking the file drawer. Science 2014, 345,
1502–1505. [CrossRef] [PubMed]
63. Norman, G. Data dredging, salami-slicing, and other successful strategies to ensure rejection: Twelve tips on how to not get your
paper published. Adv. Health Sci. Educ. 2014, 19, 1–5. [CrossRef] [PubMed]
64. Hamari, J.; Shernoff, D.J.; Rowe, E.; Coller, B.; Asbell-Clarke, J.; Edwards, T. Challenging games help students learn: An empirical
study on engagement, flow and immersion in game-based learning. Comput. Hum. Behav. 2016, 54, 170–179. [CrossRef]
65. Landers, R.N.; Bauer, K.N.; Callan, R.C.; Armstrong, M.B. Psychological theory and the gamification of learning. In Gamification
in Education and Business; Reiners, T., Wood, L.C., Eds.; Springer International Publishing: Cham, Switzerland, 2015; pp. 165–186.
66. Hew, K.F.; Huang, B.; Chu, K.W.S.; Chiu, D.K.; Lo, C.K. Engaging Asian students through game mechanics: Findings from two
experiment studies. Comput. Educ. 2016, 92, 221–236. [CrossRef]
67. Warschauer, M. Information literacy in the laptop classroom. Teach. Coll. Rec. 2008, 109, 2511–2540. [CrossRef]
68. Hamari, J.; Koivisto, J.; Sarsa, H. Does gamification work? A literature review of empirical studies on gamification. In Proceedings
of the 47th Hawaii International Conference on System Sciences, Waikoloa, HI, USA, 6–9 January 2014.
69. Nation, I.S.P. Learning Vocabulary in Another Language; Cambridge University Press: Cambridge, UK, 2001.
70. Stoller, F.L. Establishing a theoretical foundation for project-based learning in second and foreign language contexts. In Project-
Based Second and Foreign Language Education: Past, Present, and Future; Beckett, G.H., Miller, P.C., Eds.; Information Age Publishing:
Greenwich, CT, USA, 2006; pp. 19–40.
71. Peterson, M. Computer Games and Language Learning; Palgrave Macmillan: New York, NY, USA, 2013.
Sustainability 2023, 15, 11325 19 of 19
72. Song, F.; Hooper, L.; Loke, Y.K. Publication bias: What is it? How do we measure it? How do we avoid it? Open Access J. Clin.
Trials. 2013, 5, 71–81. [CrossRef]
73. Bower, M. Technology-mediated learning theory. Br. J. Educ. Technol. 2019, 50, 1035–1048. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.