Effect of Gamification of Practice Activities and Assessments

on Academic Performance of Middle School Students

in an Online Grammar Class

Sandra Chambers

EDF6481: Foundations of Educational Research

Research Proposal

Dr. Peggy Jones

November 20, 2020

 Abstract

The current study proposal aims to investigate the effect of gamification of practice activities and

assessments on the academic performance of middle school and high school students in an online

grammar course. While similar studies have been done, most of these studies focus on the motivational

impact of gamification rather than the effect on academic performance. This study attempts to control

motivation of students by using heterogeneity of subjects (all students have previously shown

motivation to not only succeed, but to excel). Participants are enrollees of a voluntary online grammar

course during the four seasons of 2021. These participants will be randomly assigned to either the

control group (no gamification) or the experimental group (gamification). The inferences obtained from

the results may impact instructional design and pedagogical policies, theoretical implications, and

practical recommendations for this grammar course (and any future courses to come).

Keywords: gamification, teaching strategies, education

PREPARED BY SANDRA CHAMBERS

2
EDF 6481
 Contents

Abstract.......................................................................................................................................................3

Introduction.................................................................................................................................................5

Background.............................................................................................................................................5

Research Gap..........................................................................................................................................6

Research Question..................................................................................................................................6

Significance.............................................................................................................................................7

Literature Review........................................................................................................................................8

Article 1...................................................................................................................................................8

A Qualitative Investigation of Student Perceptions of Game Elements in a Gamified Course: by T.

Aldemir, B. Celik, and G. Kaplan..........................................................................................................8

Article 2.................................................................................................................................................10

An Empirical Study on the Use of Gamification on IT Courses at Higher Education: by B. Barna and

S. Fodor.............................................................................................................................................10

Article 3.................................................................................................................................................12

Using Gamification to Motivate Children to Complete Empirical Studies in Lab Environments: by R.

Brewer, L. Anthony, Q. Brown, G. Irwin, J. Nias, and B. Tate............................................................12

Article 4.................................................................................................................................................14

Establishing the Impact that Gamified Homework Portals Can Have on Students’ Academic

Motivation: by Brittany Butler and Cheryl Bodnar............................................................................14

PREPARED BY SANDRA CHAMBERS

3
EDF 6481
 Article 5.................................................................................................................................................16

Gamifying English Language Learning: A Quasi-Experimental Study Examining Middle School EFL

Learners’ Vocabulary Learning Motivation: by Zeynep Turan and Birgil Akdag Cimen.....................16

Article 6.................................................................................................................................................17

Learning and Engagement in a Gamified Course: Investigating the Effects of Student Characteristics:

by K. Davis, H. Sridharan, L. Koepke, and R. Boiko............................................................................17

Article 7.................................................................................................................................................19

An Empirical Study Comparing Gamification and Social Networking on E-learning: by Luis de-

Marcos, Adrian Dominguez, Joseba-de-Navarrette, and Carmen Pages...........................................19

Proposed Methodology.............................................................................................................................22

Research Design....................................................................................................................................22

Research Participants............................................................................................................................22

Variables...............................................................................................................................................23

Data Collection and Analysis.................................................................................................................24

References.................................................................................................................................................26

Introduction

Background

A massive rising trend for teenagers (as well as children and adults) in this technological age is

collaborative and interactive gaming. It is no surprise, then, that adding gamification to education “has

PREPARED BY SANDRA CHAMBERS

4
EDF 6481
recently become more popular and prominent in everyday life,” (Aldemir et al., 2018, p. 235). The

popularity of the gaming industry is certainly a prime factor for upward trends of gamifying educational

content, however there are other reasons that instructional designers implement games into curriculum

and training. In fact, De-Marcos et al. (2014) suggest that “gamification is currently driven by the success

and momentum of videogames but it also draws on different psychological theories, mostly using

motivational models,” (De-Marcos et al., 2014, p. 82).

While gamification technically can include intensive player interaction like virtual and

augmented reality most of the prominent research literature, when referring to gamification, writes of

more simplistic methods of gamification and game elements such as leaderboards, points, and badges.

According to Aldemir et al. (2018), gamification is “an umbrella term focusing on the use of game

elements instead of full-fledged games to improve user experience and engagement in non-game

contexts including education.” For the purpose of the current study, gamification will use the latter

definition including simple game play through 2D arcade style assignments and assessments.

Research Gap

Research regarding the effects of gamification on learning is prolific (at least for a recently developed

educational movement), however, most of the studies focus on the motivational aspects of learning, or

how motivation improves student performance rather than the actual effect of gamification on the

academic improvement of students. According to Davis et al. (2018) “motivational views of game‐based

learning represent a sizable portion of existing scholarship on games and learning.” Barna and Fodor

accentuate that point when they stated that “gamification has recently become one of the most popular

strategies to improve the methodology of promoting motivation and engagement in education.”

While gamification is popular and “used to promote the engagement and motivation of the

involved individuals in business, education, health industry, societal responsibilities and many other

PREPARED BY SANDRA CHAMBERS

5
EDF 6481
fields of everyday life,” (Barna & Fodor, 2017, p. 2) it has not always proven effective at increasing

student performance. According to Turan and Cimen (2018) “gamification of education may offer a new

and more enjoyable way for students to learn, however, gamified courses do not guarantee academic

success on their own”.

While acknowledging the importance of integrating motivational design into instructional

design, this study intends to veer away from the motivational aspects of gamification. The research for

this study will focus solely on whether gamification of assignments and assessments in an online

grammar class for middle school and high school students has a significant effect on academic

performance.

Research Question

To what extent, if any, does gamification of practice activities and assessments affect the academic

performance of middle school and high school students in an online grammar course?

Significance

Understanding the true effect of gamification on academic performance (not on motivation), if there is

indeed an effect, can help instructional designers to better plan curriculum that will improve student

output. In regard to policy, if this study discovers a positive relationship between gamification and

academic performance, instructional designers will know to invest time and energy into constructing

gamification concepts into academics as an improvement tool for student performance. If there is no

relationship or a negative relationship between gamification and academic performance, designers will

know to avoid the extra expense of creating gamification in course curriculum, or to plan the

gamification solely as a method for improving student motivation.

The current theories revolving around gamification in learning are variant and at times even

contradictory. Barna and Fodor (2017) are of the opinion that “gamification may improve various

PREPARED BY SANDRA CHAMBERS

6
EDF 6481
aspects of higher education such as student engagement and motivation, interaction between students

and teachers, providing regular feedback to students, and optimal use of available infrastructure and

human resources.” And Brewer et al. (2013) concluded that “the points and prizes were successful in

their goal of motivating the children to complete all the tasks.” While many researchers hail gamifying

curriculum as a boost to student motivation and therefore as a method to improve academic

performance, not everyone agrees. Butler and Bodnar reported that “overall, the implementation of the

gamification platform had relatively neutral impact on student’s academic motivation towards

homework.” And, according to Aldemir et al. (2018), “students in a gamified course had less motivation,

satisfaction, and lower exam grades than those in a non-gamified class”. The results of this research

study will contribute to the developing instructional design and pedagogical theories centralized around

combining gamification with education.

The results of the current research will be used in a practical way to implement changes to the

online grammar course that will improve the overall student knowledge gain in the course. In the event

that there is a positive correlation between gamification and educational performance, the course

designers will have the data they need to better design the gamification of the current course and to

develop future courses in a similar manner. If the correlation between the two is negative, the course

designers will know not to invest time, money, and effort into gamifying courses in the future. If there is

no significant correlation between the two, the designers may choose to eliminate the gamification,

modify the gamification, or leave the gamification as it is currently.

PREPARED BY SANDRA CHAMBERS

7
EDF 6481
 Literature Review

Article 1

A Qualitative Investigation of Student Perceptions of Game Elements in a Gamified Course: by T.

Aldemir, B. Celik, and G. Kaplan

In their qualitative study, A Qualitative Investigation of Student Perceptions of Game Elements in a

Gamified Course, Aldemir, Celik, and Kaplan (2018) explore how game elements affect student

perceptions in a gamified instructional technology and material development course. In the study they

suggest that game elements such as badges, leaderboards, points, ranks, progress bars, and quests

increase student motivation, satisfaction, and achievement. However, they assert that gamification

methods can be diverse and therefore application, extent, and effectiveness will also be diverse (Aldemir

et al., 2018, p. 236).

In order to understand gamification in an educational context, the authors collaborated with an

instructor to develop a course that was fully gamified in a Harry-Potter-like atmosphere. The basic

premise was that students start as apprentice wizards and progress to a master wizard level by

obtaining points through course work and through collaboration with their wizard house. Gamification

was integrated throughout the entire course including pre-class welcome letters, instructor PowerPoint

presentations, syllabus, assignments, group activities, and lectures (Aldemir et al., 2018, p. 240).

Results of the study were separated into nine categories of gamification: challenge, narrative,

leaderboard, reward, badge, teams, win-state, points, and constraints. Aldemir et al. (2018) summarized

data according to each of the nine categories of gamification and gave percentages of participants that

liked and/or disliked each category (Aldemir et al., 2018, p. 243). The study did not include any other

descriptive statistics. It did not mention causality.

PREPARED BY SANDRA CHAMBERS

8
EDF 6481
 The researchers worked hard to strengthen the study in a variety of methods. Participants were

chosen through a combination of convenience sampling (the instructor was willing to try a new method

in her class) and purposeful sampling (interview participants were chosen who had information rich

experiences). The researchers gathered data using triangulation through interviews, observation, and

documentation in order to determine how game elements should be designed and implemented from a

student perspective (Aldemir et al., 2018, p. 242).

Additionally, Aldemir et al. (2018) consulted with outside experts who analyzed and coded the

data. The researchers reported that “triangulation, peer examination, and an audit trail were followed

to ensure consistency, whereas triangulation, member checking, engagement in data collection, peer

examination, and the search for negative findings were followed to fulfill credibility” (Aldemir et al.,

2018, p. 243). In order to improve external validity, the course was also delivered to participants from a

variety of departments and was repeated in two cycles (Aldemir et al., 2018, p. 242).

Aldemir et al. (2018) attempted to emphasize the strength of the study, but they also succinctly

indicated the limits of the study. While they worked to improve external validity, the researchers openly

state that the study participants may not represent a more generalized target group (Aldemir et al.,

2018, p. 250). They also identify that more than two repetitions may be needed to eliminate extraneous

variable affect (particularly as they did nothing to address differences in the two groups of students

between the two cycles) (Aldemir et al., 2018, p. 250). As with many qualitative studies, the results of

the data are reliant on honest and sincere responses from participants and unbiased interpretations of

observations.

This study determines student preference of gamification methods for this sample population.

However, it has low external validity and while it can be used as a starting point for other studies, it

cannot be generalized to other populations without considerable care. Additionally, this study had too

PREPARED BY SANDRA CHAMBERS

9
EDF 6481
many independent variables to effectively measure results. Any or all of the independent variables could

have been affecting student preference. There were too many variables and not enough constants.

While this study focuses on the motivational aspects of gamification in multiple aspects of a

course, the present study focuses on solely the impact of gamification used in practice activities and

assessments only. All other curriculum and content will be identical between the experimental group

and the control group. Additionally, while Aldemir and his associates concentrated on student

perception and motivation, the present study is solely concerned with academic performance.

Article 2

An Empirical Study on the Use of Gamification on IT Courses at Higher Education: by B. Barna and S.

Fodor

In their paper, An Empirical Study on the Use of Gamification on IT Courses at Higher Education, B. Barna

and S. Fodor (2017) focused on the effect of using gamification to teach Information Technology to

higher education students in Budapest. The participants, enrolled in a compulsory Information

Technology course, had access to a gamified version of the course in an online platform called Modular

Object-Oriented Dynamic Learning Environment (MOODLE). The students participated in the four-

module online curriculum to complete weekly tests, “life-boat” tests, and optional assignments for

which they were rewarded points and badges. In turn, students could use these points and badges to

earn extra points on the midterm and final exams. However, participation in these tests was not

mandatory (Barna & Fodor, 2017, p. 4).

The data collected from the MOODLE platform (points, failure rates, course evaluations, and

student feedback) were analyzed and coded. Barna and Fodor (2017) assessed course quality by student

willingness to participate, exam scores, and student satisfaction surveys. Based on the results that they

PREPARED BY SANDRA CHAMBERS

10
EDF 6481
received, Barna and Fodor (2017) suggested that while gamification cannot solve every problem, it can

lead to a better course experience.

The researchers differentiated results into two groups: full-time students and part-time students

(Barna, & Fodor, 2017, p. 4). The statistics were displayed with graphs showing percentage of each

student group that participated in gamification activities each week. Student satisfaction was obtained

through course evaluation system questionnaire (1-5 scale) and compared with a questionnaire filled

out by students in a similar Information Technology class that did not have access to the online MOODLE

gamification (Barna, & Fodor, 2017, p. 8). (While both courses had the same syllabus, no mention was

made of whether teacher was the same or whether other extraneous variables were taken into

consideration by the researchers). The authors did acknowledge that lack of a comparable course is a

weakness to the study and have plans to split each course into two groupings one with the gamification

and one without during the next rendition of the experiment (Barna, & Fodor, 2017, p. 9).

This qualitative study was intended to be a correlational study but lacked the necessary data to

determine a correlational relationship. The researchers indicated that they wanted to determine the

effectiveness of teaching using gamification in a higher level Information Technology course, however,

they only reported data about the gamified course and did not report data about the non-gamified

course. They did, however, have participants from both courses take the student satisfaction survey and

compared the results. Student satisfaction surveys from the gamification course scored roughly 10%

higher.

Unfortunately, the study includes very little reference to either reliability or validity and does

not offer replicable methods. They did not provide any descriptive statistics other than percentage of

full-time and part-time students who participated in the voluntary online platform each week.

PREPARED BY SANDRA CHAMBERS

11
EDF 6481
 As is common in research regarding gamification in education, this study through qualitative

surveys, attempted to describe the effect of gamification on student satisfaction. While motivation and

student satisfaction are important elements to education, they do not necessarily lead to higher

academic performance. While the study participants in this paper are undergraduates, the participants

in the current study are middle schoolers. Additionally, the current study differs in that it is attempting

to pinpoint the effect of gamification on academic performance in a middle school grammar class and

not considering student motivation or enjoyment.

Article 3

Using Gamification to Motivate Children to Complete Empirical Studies in Lab Environments: by R.

Brewer, L. Anthony, Q. Brown, G. Irwin, J. Nias, and B. Tate

Brewer et al. (2013), in their study Using Gamification to Motivate Children to Complete Empirical

Studies in Lab Environments, discuss the use of gamification to help increase motivation and completion

rates of child study participants aged 5 to 7. In their original research work on determining how children

engage with educational technology, the researchers encountered difficulty working with younger

children. To remedy this situation, they researched how introducing gamification components to the

research study protocol affected the participation and completion rates for younger children (Brewer et

al., 2013, p. 2).

In order to obtain data on touch interaction tasks and gesture interaction tasks in younger

children, Brewer et al. developed two study protocols (Brewer et al., 2013, p. 3). This correlational

research compares the completion rates of the first protocol, performed without gamification, and the

second protocol, performed with gamification added. In the first protocol, seven participants were asked

to use their fingers to draw various gestures (like letters) on the screen. Unfortunately, many of the

young children grew bored and requested permission to quit (Brewer et al., 2013, p. 3). The second

PREPARED BY SANDRA CHAMBERS

12
EDF 6481
protocol was similar to the first but added gamification elements (points that could be used to purchase

small trinkets) (Brewer et al., 2013, p. 4).

The completion rate of the gamified protocol was much higher than the completion rate for the

non-gamified protocol. Descriptive statistics were used to identify completion rate differences (mean,

range, and standard deviation). The mean completion rate of the first protocol was 73% whereas the

mean completion rate for the second protocol was 97%. From these results, the researchers inferred

that gamification (points and prizes) motivates young children to complete tasks in empirical studies.

They also determined that these gamification elements did not negatively impact the study (Brewer et

al., 2013, pp. 4-5).

This study suggests that gamification used with children does not negatively influence study

results. This is encouraging to the current study because the participants are students (though slightly

older than these students) and ideally, adding gamification to activities and assessments in a middle

school grammar course will similarly not present negative impacts to learning and academic

achievement. However, while this study focused on task completion rates, the present study will focus

on different quantitative data, namely test scores and academic performance.

Article 4

Establishing the Impact that Gamified Homework Portals Can Have on Students’ Academic Motivation:

by Brittany Butler and Cheryl Bodnar

In their study, Establishing the Impact that Gamified Homework Portals Can Have on Students’ Academic

Motivation, Butler and Bodnar (2017) investigated how the implementation of a gamification platform

impacted student motivation to complete homework. The homework platform was created with nine

levels that Freshman Engineering students could move through to complete quests which began with

more basic concepts and increased in complexity and difficulty (Butler & Bodnar, 2017, p. 5). Each

PREPARED BY SANDRA CHAMBERS

13
EDF 6481
completed quest (similar to traditional activities or homework assignments) provided students with

experience points, badges, and special rewards on the platform (Butler & Bodnar, 2017, p. 6).

Completion of quests on the platform counted for 15% of the students’ final grade in the course (Butler

& Bodnar, 2017, p. 8).

Butler and Bodnar (2017) specified the operational definition of motivation in this study as

results from a voluntary MUSIC Mosel Academic Motivation Survey as well as voluntary participation in

an end of semester focus group. Both the survey and participation in the focus group were analyzed

using a coding framework by two coders (a faculty member and an undergraduate student) and

prominent themes were identified. Inter-rater reliability was assessed using Cohen’s Kappa with an

agreement level of 0.68 (Butler & Bodnar, 2017, p. 10).

According to the results reported by the researchers, there were many discrepancies in the data

set between the two class sections. Each motivational aspect measured by the MUSIC survey

(Empowerment, Useful, Success, Interest, Caring) had means and standard deviations calculated for

each class section. Section 1 means were significantly lower than Section 2 means, and Section 1

standard deviations were much higher than Section 2 standard deviations (Butler & Bodnar, 2017, pp.

10-11). The paper identifies many possibilities for these discrepancies, one of which was to suggest that

the differences might be due to the variation in percent of completed surveys (section 1 had 21 surveys

out of 23 students whereas section 2 had only 6 surveys out of 18 students) (Butler & Bodnar, 2017, p.

11).

While the researchers studied motivation in two sections of the course, the sample size was too

small to generalize results to wider population without more iterations of the study. Additionally, the

focus group questions were specific to the platform used and would not be able to be generalized to a

great population. The external validity of this study is low. The course sections were taught by two

PREPARED BY SANDRA CHAMBERS

14
EDF 6481
different professors adding an extraneous variable that also decreases the internal validity of the study.

The implications of the study showed that the gamified platform had a neutral effect on academic

motivation (Butler & Bodnar, 2017, pp. 13-14).

Due to lack of both internal and external validity, the results of this study are not particularly

valid in regard to the current study. However, the gamification platform design in this study is similar to

the platform design of the present study, where participants will earn experience points by playing

games to fulfill assignment requirements. In contrast to this study, however, the current research design

will incorporate game play as the means to obtain experience (and therefore gain assignment points).

Article 5

Gamifying English Language Learning: A Quasi-Experimental Study Examining Middle School EFL

Learners’ Vocabulary Learning Motivation: by Zeynep Turan and Birgil Akdag Cimen

Turan and Cimen (2018), in Gamifying English Language Learning: A Quasi-Experimental Study

Examining Middle School EFL Learners’ Vocabulary Learning Motivation, investigated the impact of

gamification on student learning motivation. The researchers studied two classrooms of seventh grade

students. Through convenience sampling, Turan and Cimen (2018) assigned one class as the control

group (non-gamified) and the other as the experimental group (gamified). While the control group

studied language vocabulary in a traditional way, the experimental group was taught language

vocabulary through the use of gamification tools (Turan & Cimen, 2018, p. 2).

The data were collected through a Likert-type questionnaire which was filled out prior to the

course as a pre-test survey and again after the course as a post-test survey (Turan & Cimen, 2018, p. 2).

The results of the pre-test for the control group and the experimental group were compared via t-test,

and the results (t = 0.359, p =.72 >.05) showed that there was no significant difference between the two

group pre-test survey scores. In contract, the post-test survey comparison showed a significant

PREPARED BY SANDRA CHAMBERS

15
EDF 6481
difference (t = -2.180, p =.033>.05). The difference in the significance between the pre-test and post-test

surveys seems to indicate that the gamification in this study affects student motivation (Turan & Cimen,

2018, pp. 4-5).

Unfortunately, the researchers did not include a copy of the Likert scale survey that they used

for their study and did not refer to whether they created the survey or modified an existing survey. If the

survey was indeed a good measurement tool for student motivation, then the conclusions drawn by the

researchers may be accurate. If, however, the measurement tool did not assess motivation, but

assessed something else or nothing at all, then the study conclusions would be in error.

Once again, the focus of this study was on student motivation, contrary to the current study.

However, this research design was also focused on language learning, which is in the same general field

as learning grammar. Additionally, the game-play features of the gamification process are similar. While

Turan et al. (2018) reported to have found a significant improvement of student motivation when these

gamification treatments are utilized, they do not report on academic achievement. The current study

intends to extend this research to include information regarding effect on student performance.

Article 6

Learning and Engagement in a Gamified Course: Investigating the Effects of Student Characteristics: by

K. Davis, H. Sridharan, L. Koepke, and R. Boiko

In their study, Learning and Engagement in a Gamified Course: Investigating the Effects of Student

Characteristics, Davis et al. (2018) investigated how gamification in an undergraduate informatics course

impacted student learning, achievement, and engagement with course material. The researchers

surveyed student participants three times during the year in order to note student attitudes of both

gamification and the course, specifically focusing on engagement, learning, and achievement. The

PREPARED BY SANDRA CHAMBERS

16
EDF 6481
authors examined the differences in effects of gamification on gamers versus non-gamers and on males

versus females (Davis et al., 2018, pp. 1-2).

Participant students attended a combined “Big Class” with the instructor once a week where

they engaged in highly interactive activities and combined play. They also attended a “Small Class” twice

a week with a teaching assistant where they would take quizzes and perform class activities (Davis et al.,

2018, p. 4). All student work (tests, quizzes, activities, class participation, etc.) was worth a certain

amount of XP points, and student grades were based on accumulation of XP. The top ten XP holders

would be listed on a leaderboard and given a small prize periodically throughout the semester.

Additionally, students were able to earn badges and level-up (Davis et al., 2018, pp. 4-5).

The researchers used two different ordinal measurement scales to identify which students to

classify as “gamers”. The first was based on gameplaying hours per week (0 hr - nonplayers, 1-9 hr- light

players, 10+ hr - heavy players). The second was student self-identification as (non-gamer, casual gamer,

avid gamer). Unfortunately, due to variance in attendance, many participants did not take all three

surveys (Davis et al., 2018, p. 5).

Because the data did not have “enough response levels to satisfy the normal assumptions of

common statistics”, Davis et al. (2018) used nonparametric tests for statistical significance (the Mann-

Whitney U test for comparing two groups and the Kruskal and Wallis test for comparing more than two

groups). For count and non-ordinal data chi-squared tests were used. Participants that did not have

complete data sets were omitted from the data analysis (Davis et al., 2018, p. 5).

Davis et al. (2018) reported that their results indicated that student attitude toward gamification

was mostly positive and maintained similar percentages throughout the semester. Roughly 60% of

participants said that they learned and achieved more with the gamification than they would have

PREPARED BY SANDRA CHAMBERS

17
EDF 6481
otherwise (Davis et al., 2018, p. 6). Approximately two-thirds of the participants felt that the

gamification in the course enhanced student engagement in the material (Davis et al., 2018, p. 7).

The effects of gender did not seem to play any significant difference in relation to engagement

or learning achievement. The one exception was that female students were slightly more likely to

indicate that the gamification had a positive impact on their grade. In contrast, there was a much larger

difference of the effect of gamification in the course from the perspective of non-gamers. Approximately

70% of non-gamers reported that they had less motivation to do well in the gamified course compared

to the 30% of heavy gamers and the 15% of light gamers (Davis et al., 2018, p. 7).

One of the strengths of the study was that the researchers were able to obtain survey results

from participants three different times during the study. These survey results followed student

perceptions and attitudes through the course beginning, middle, and conclusion. A weakness in the

study was the mortality of participant involvement due to varying levels of student attendance in the

class and omitting participants without complete data decreased internal validity.

This study indicates that gamification in this study did not affect males and females differently,

which will be useful information in the currently study. However, it does indicate that “non-gamers”

may be negatively affected by gamification used as part of the course. Because the current research

proposal is targeting an alternate age group, it is possible that this will not be an issue. Nevertheless, in

order to avoid this possible negative impact, the present study intends to limit the intensity of the

gamification and only integrate the gamification as part of learning activities and assessments (not

learning curriculum).

PREPARED BY SANDRA CHAMBERS

18
EDF 6481
Article 7

An Empirical Study Comparing Gamification and Social Networking on E-learning: by Luis de-Marcos,

Adrian Dominguez, Joseba-de-Navarrette, and Carmen Pages

In their study, An Empirical Study Comparing Gamification and Social Networking on E-learning, de-

Marcos et al. compare the effect of social networking and gamification on undergraduate student

academic achievement, participation, and attitude in an introductory information technology course.

The researchers designed and tested two motivational instruments, a gamification plugin and a

networking site that provided opportunities for collaboration and interaction among students.

The gamification plugin (which was integrated into Blackboard LMS) allowed students to

complete course activities while competing and/or collaborating with classmates. Individual activities

were divided into levels, and upon level completion students were awarded trophies. Additionally,

students were awarded badges for accomplishing specific tasks. The second supplemental treatment

was a social networking system that allowed students to view videos, interact, ask/answer questions,

comment/ like one another’s progress, post on blogs, rate questions/answers, etc.

In this quasi-experimental study, three groups of students (convenience samples) were selected

from undergraduate students in a variety of majors. Group A (economics, business admin, accounting,

finance, international business majors) received the gamification plugin, Group B (life science, nursing,

economics, business admin) the social networking site, and Group C (nursing, infant education, primary

education, business admin, tourism, construction engineering) was the control group and received

neither treatment. Experimental groups could choose to participate through the given treatment, the

regular course, or a combination of both. All three groups received traditional e-learning curriculum and

lectures. However, each group was located in a different city and had separate e-learning courses.

PREPARED BY SANDRA CHAMBERS

19
EDF 6481
 Quantitative data were gathered and analyzed in both the spring and fall of 2012. The

researchers gave participants a pre-test and post-test to compare student performance before and after

treatment. Student participation scores were computed based on lecture attendance, contributions in

the e-learning course, and contributions in the experimental treatments. All scores were normalized on

a 0-100 scale, and pre-test and post-test data were compared using ANOVA tests. Finally, a Likert scale

attitude survey was administered to both of the experimental groups to determine student perceptions

of the treatments.

Led by de-Marco, the research group reported that while pre-test scores between the three

groups were not significantly different, both Group A (gamification plugin) and Group B (social

networking) outperformed Group C (control group) in all four practical assignment categories. Group B

had higher participation scores than both Group A and Group C. Additionally, Group B outperformed

Group A in word processing and spreadsheets. Interestingly, however, the Group C outperformed both

Group A and Group B on the final written exam.

The authors suggested that the higher final exam scores for the control group may have been

due to material overload in the experimental groups. The experimental treatments focused on the

practical activities and skill acquisition rather than on knowledge acquisition. From this knowledge

discrepancy, de-Marco et al. suggested that both social networking and gamification plugins may create

student bias towards practical learning rather than knowledge acquisition.

The study strengths include using appropriate descriptive statistics, normalizing scores, and

using the ANOVA test. The study also included participants from a wide range of majors, however, the

validity would have been much stronger if participants had not been clustered according to majors

within the groups. Additionally, the study did not account for extraneous variables such as alternate

teachers, alternate locations, and alternate curriculums. Without eliminating these variables, it is

PREPARED BY SANDRA CHAMBERS

20
EDF 6481
difficult to verify that the independent variables (gamification plugin and social networking) had an

effect on the dependent variables (academic achievement, participation, and attitude).

Another serious concern with the results of this study is that participation rates in the

experimental treatments were significantly low – only 38% for social networking and only 24% for

gamification. While the sample size began at an acceptable size, the lack of participants actually

engaging with the experimental treatments overly reduced the sample size. This sampling mortality rate

limits the inferences that can be made from the study.

The results from this research study indicate that there may be a significant difference in the

effect of gamification on practical skills versus the effect of gamification on knowledge skills. It is

probable that this discrepancy is due to the type of gamification used in this study. In order to avoid a

similar issue in the current study, the design includes gamification features that will alternate between

practical and knowledge skills so that students will be able to practice both.

Proposed Methodology

Research Design

This quantitative study will implement a true experimental design including manipulation of the

independent variables (gamification of assignments and gamification of assessments) and randomly

assigning participants to the experimental group through probability sampling. The investigation of

gamification in the online grammar course will be conducted in 2021 in four iterations, winter (Jan-Mar),

spring (Apr-Jun), summer (Jul-Sep), fall (Oct-Dec). Additional renditions will be done in future semesters,

as necessary. Repeating the research cycle through four renditions and lengthening the study will

improve the reliability. A single blind study (assigning random IDs to each student) will be used in order

PREPARED BY SANDRA CHAMBERS

21
EDF 6481
to preserve student anonymity, protect the results from bias, and also increase reliability. All personal

information will be protected.

Research Participants

Participants will be late middle school and early high school students (ages 13 to 17) in the Miami-Dade

School District. Most of these students will be taking the grammar course in order to prepare for the

English section of the PERT test so that they can enroll as a dual enrollment student. It is probable that

they were referred to the program by their current English teachers, by school counselors, from

informative emails sent to parents, or by dual enrollment specialists. Almost without exception, these

students are highly ambitious and motivated students who have demonstrated their ability to excel

academically and their desire to progress toward higher education. The heterogeneity of the

participants increases reliability of the research. Each student who enrolls in the online grammar course

(estimated to be approximately 40 students each season) will be randomly assigned to either the control

group (no gamification) or the experimental group (gamification). While diffusion among participants

poses a small threat to internal validity, most of the students will not know each other as the course is

online and district wide.

Variables

The independent variables for the research proposal are gamification of practice activities and

gamification of assessments. The constitutive definition of gamification is educational activities

programmed to teach through interactive game play. The operational definition of gamification in this

study will be game play through 2D arcade style assignments and assessments.

The dependent variable of the study is the academic performance of middle school students in

an online grammar course. The operational definition of academic performance will be measured in two

ways. First, with a comparison of each student’s pre-test and post-test scores. Second, each student’s

PREPARED BY SANDRA CHAMBERS

22
EDF 6481
scores on the English section of the PERT college readiness test. These are both preexisting

measurement tools.

The constants in the research include curriculum, teacher, learning management system (LMS),

approximate student age, course length, and time of year. The participants in both the experimental

group and participants in the control group will be treated identically with the exception of the

independent variable treatments of gamifying assignments and assessments.

Data Collection and Analysis

Understanding the direct relationship between gamification in a course and student academic

performance is a necessity for instructional designers in this technological age. While it is not possible to

remove the motivational aspects that might be involved, this study attempts to eliminate the impact of

motivation by studying students who have all demonstrated a higher level of motivation regularly in

academic pursuits. Thus, attempting to control for the extraneous variable of motivation.

All enrolled students, both the experimental group and the control group, will participate in the

online grammar class as though no research was being done. While the participants will be aware that

they are involved in a research study, and will consent to participating in the research, the study itself

should have no influence over their performance in the course. Both groups will be taught by the same

teacher and with the same curriculum. All content, discussions, as well as assignments and assessment

content will be mirrored between the two groups. However, the assignments and assessments for the

control group will be standard grammar class assignment media (worksheets, readings, etc) whereas the

media for the experimental group assignments and assessments will be gamified versions of the same

content.

All students will be required to take a pre-test before beginning the curriculum, and a post-test

once finished with the curriculum. For each season, a comparison of the pre-test and post-test scores

PREPARED BY SANDRA CHAMBERS

23
EDF 6481
will be used to compare the academic performance growth of each student. Scores will be compared

using analysis of variance (ANOVA) tests with p<.05 showing statistically significant differences. Because

this study is a correlational study, analyses will also include collecting and calculating group mean,

standard deviation, standard error, and Pearson product moment correlation (r) on pre-test and post-

test scores.

In addition, each season, participating students will be required to take the English section of

the PERT test prior to the course and retake the test again after the course is finished. Similar statistical

measures and analyses will be conducted with the student PERT test scores, comparing the student

scores prior to the class and those after the class. Both PERT tests attempts will be provided free of

charge for all students participating in the study. The double measurement of student academic

performance through both course scores and also PERT scores provides concurrent validity in the study.

After the data are collected and calculated for each season individually, the data from all four

seasons of the course will be combined and data will be analyzed with similar statistical tests. The

descriptive statistics and correlation tests will increase validity and determine what type of correlation, if

any, exists between gamification of assignments and assessments on student academic performance.

The inferential statistics from the ANOVA comparison results will indicate if the data are statistically

significant.

PREPARED BY SANDRA CHAMBERS

24
EDF 6481
 References

Aldemir, T., Celik, B., & Kaplan, G. (2018). A qualitative investigation of student perceptions of game

elements in a gamified course. Computers in Human Behavior, 78, 235-254.

Barna, B., & Fodor, S. (2017, September). An empirical study on the use of gamification on IT courses at

higher education. International Conference on Interactive Collaborative Learning (pp. 684-692).

Springer, Cham.

Brewer, R., Anthony, L., Brown, Q., Irwin, G., Nias, J., & Tate, B. (2013, June). Using gamification to

motivate children to complete empirical studies in lab environments. Proceedings of the 12th

international conference on interaction design and children (pp. 388-391).

Butler, B., & Bodnar, C. (2017). Establishing the Impact that Gamified Homework Portals Can Have on

Students' Academic Motivation.

Davis, K., Sridharan, H., Koepke, L., Singh, S., & Boiko, R. (2018). Learning and engagement in a gamified

course: Investigating the effects of student characteristics. Journal of Computer Assisted

Learning, 34(5), 492-503.

De-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014). An empirical study comparing

gamification and social networking on e-learning. Computers & Education, 75, 82-91.

Turan, Z., & Çimen, B. A. (2018). Gamifying English language learning: A quasi-experimental study

examining middle school EFL learners’ vocabulary learning motivation. PREFACE OF THE

EDITORS, 40.

PREPARED BY SANDRA CHAMBERS

25
EDF 6481