2020 International Conference on Promising Electronic Technologies (ICPET)
Automatic Analytics Model for Learning Skills Analysis Using Game Player Data
and Robotic Process Automation in a Serious Game for Education
Radwan Qasrawi1*, Malak Amro1, Rashid Jayousi1
1
Department of Computer Science, Al-Quds University, Jerusalem, Palestine.
radwan@staff.alquds.edu*, malak.sharif@gmail.com, rjayousi@staff.alquds.edu
Abstract— Utilizing serious games as a learning support
tool can motivate students’ interest in learning and
improving their educational skills. The effectiveness of
serious games in learning is usually assessed with
traditional paper-based pre-posttests tools and predefined
learning outcomes. However, serious game player data
have proven to be a useful tool to measure the effect of
games on student knowledge, learning skills and to
evaluate their academic performance and achievements. In
this paper, we introduce a new model for automatically
read game player data and provide an analysis of game
player learning skills. The model is composed of game
learning analytics, robotic process automation, and the
statistical analysis package to assess the effect based on
game players' data. The model used the plant kingdom
unit in the science book of grade eight students. The
learning outcomes, content, and learning skills are
considered in developing game content. The model tested
on a sample of 15 students aged 13-14 years from
Palestinian public schools. The students played the game
four times, and the game's player data sent automatically
into the game database and analyzed. The model provides
an efficient pre-posttest assessment tool based on the game
players dataset. The results showed a significant
improvement in learning skills performance measures
after comparing the results of the first session with the
fourth session.
Keywords— Analytics model, learning skills, serious game,
robotic process automation, game analytics,
game player data.
I. INTRODUCTION
A serious game is a popular term used for an application that
is developed using video game technology and game design
principles but is used for a non-entertainment purpose [1], [2].
Serious games have been linked to higher academic
achievements and used to increase the engagement, attraction,
hold the interest and attention of players for a specific purpose
such as to promote new knowledge or skills [3]–[5]. It is used
to help in learning via interaction, there are many other
advantages to serious games, for example, they allow users to
live situations that are difficult to try in reality because of
several factors including cost, time, safety, personal
circumstances, and ethical concerns [2], [6]. Moreover,
978-0-7381-1139-1/20/$31.00 ©2020 IEEE
DOI 10.1109/ICPET51420.2020.00026
Authorized licensed use limited to: b-on: Universidade de Trás-os-Montes e Alto Douro. Downloaded on April 05,2023 at 16:58:47 UTC from IEEE Xplore. Restrictions apply.
researches showed that after regular use of educational games
and apps, students’ abilities had been promoted important
knowledge and math skill increases, compared to the use of
non-educational apps [7]–[9]. Findings also showed games
may enhance some of the core cognitive skills, such as
selective attention that contain the ability to obtain educational
content or skills - a process that has been named “learning to
learn” [2], [6], [9], [10]. Meanwhile, few researchers have
highlighted the positive advantages of cognitive games among
adolescents students, such as their role in improving
interactions between students and have more friendships,
which improve personal skills and abilities of communication
[5], [11], [12].
Currently, the traditional evaluation method (pre-Posttests) is
used for evaluating the effect of serious games on players’
learning skills development. This method is commonly used in
public health and medical research. However, serious game
player data have proven to be a powerful tool for assessing the
SG effect on players' knowledge development. The game
learning analytics that combines the SG educational goals with
the learning analytics techniques were used in capturing,
storing, analyzing, and reporting the game results from
players' data [12]–[16].
In this paper, we proposed a new automatic analytics model
based on players' data using the game learning analytics and
the robotic process automation in collecting and analyzing
players learning skills development.
II. METHODS
The automatic analytics model was developed in a serious
game for enhancing adolescents learning attentions and
understanding. The game is designed in a series of steps
starting from defining game objectives to results analysis and
evaluation. Figure 1 illustrates the gam design architecture.
The serious game was developed as an educational tool for
teaching grade eight students in the plant kingdom unit in the
science book. The game objects and characters were selected
from the teaching materials. The human-computer interaction
standards were considered in designing the game scena rios,
including colors, shapes, time, structure…etc.
94

Fig. 1. The serious game architecture design
Fig.2. Screenshot from level 1 and level 2
The video game design method was used as a tool to deliver
the game objectives. It was built based on two fundamental
pillars: instructional and interactive design to provide students
with the necessary instructions to navigate and interact
through the game levels. The game contents were designed
according to the textbook flow to follow the knowledge
development cycle set by the curriculum. The game design
combines educational content, thinking, and cognitive skills.
The Maze and Mixed and Match approaches were used in the
gameplay levels. The levels of both approaches were
presented in a sequential order where the difficulties increased
by the levels forward sequence. The display duration,
sensitivity, touch control, task time, status, speed, distance,
and stimuli were considered in gameplay activities.
A convenient sample of 20 students from grade 8 aged 13-14
years from public schools was selected for serious game
testing and validation. The selected students were asked to fill
the consent form before the gameplay indicating their parents’
95
Authorized licensed use limited to: b-on: Universidade de Trás-os-Montes e Alto Douro. Downloaded on April 05,2023 at 16:58:47 UTC from IEEE Xplore. Restrictions apply.
approval. Participants are required to be able to use mobile
phones and have an internet connection.
The game was uploaded on the google store, where students
managed to download and play the game using their android
smartphones. Due to the Covid19 lockdown and the school
closure, students were asked to play the game in their homes.
The schoolteacher and the science course supervisor were
supervising and mentoring the students playing activities. The
player game data was sent automatically to a cloud database
that was developed for the data collection purpose.
Automatic analytics model
The development of game software uses the Unity 3D mobile
App development environment, in addition to free software
development tools such as PHP and MySQL to maintain user
records and feedback database. The web-based platform
developed for managing students' game data and to enhance
the game visibility and provide information about the game.
The intended development SG was built by experts in the field
of game design and programming according to the
international standards and guidelines.
An embedded software component was developed in the plant
kingdom SG. The software uses the game analytics library for
tracking players' interactions with the game. The data
automatically sent to a cloud server and stored on the game
database. The following variables were collected from the
game player: The session number, level number, player ID,
gender, number of trails, playing tool( Maze or Mix and
Match), position, drag and drop time, status (Success or Fail),
the reason of fail (bumped into the maze border or the answer
was wrong), distance, time, and speed.
The robotic process automation using the automation
anywhere software development environment was used to
develop Bot programs for data processing and analysis[17].
The RPA bots were used to build the players' database records
from the players' text files. Also, the bots were used for
cleaning and analyzing the players' data based on game
learning analytics. The extracted variables from game players'
data were introduced into a statistical model that includes
descriptive analysis and basic inferential analysis using
JavaScript statistical libraries.
Experiment Setting
The experiment was conducted under the ministry of
education (MOE) supervision. The ministry had approved the
game testing and nominated two science supervisors and
teachers to supervise the experiment in collaboration with the
study researchers. The list of participants including their
contact information was provided by the MOE. The
supervisors contacted the participants' parents and asked them
to sign the participation approval on the game consent form.
After receiving the parents’ approval, students were contacted
and asked to join one to one orientation session. The
orientation session explained the serious game purpose and
objectives, download and installation instructions, and brief
instructions of the game playing procedure.
 The participants were left to play the game without any
interference from the supervisors, however, they were
mentored through the automatic play data recorder in the
cloud system. The students were asked to play one complete
session per day. The minimum required sessions were four
sessions to achieve the game objectives in enhancing attention
and understanding. Out of 20 participants, 15 students
managed to complete the required sessions. The 5 students
that did not complete the sessions, managed to complete 2 or 3
sessions.
III. RESULTS
The gameplay data were collected and analyzed using the
automatic game player learning analytics model. The game
was played by 15 players aged 14 years, who completed 4
playing sessions. The players were 9 girls and 6 boys.
The study examined the learning analytic model by providing
a basic descriptive analysis of game players' data. Table 1
shows the total number of trials per session. Results indicated
that session one has a higher number of trails than session 4
among girls (58%, 52.6%) respectively. While boys reported
more trails in session 4 than session one (42%, 47.4%)
respectively. However, girls reported a greater number of
trials than boys (54.4%,45.6%) respectively, as the sample has
more girls than boys (9:6).
The participants' status was further investigated by collecting
data on the reasons for success and failure for each player in
each stage and level. The results in figure 3 (a) and (b)
indicated that two main reasons were reported, either touching
the maze border during the play activity or achieving the
activity with the wrong answer. A comparison between girls
and boys was reported as illustrated in figure 3 (a) and (b). In
session one (pre-test) girls reported a greater percentage of
maze touch than boys (49.1%,35%) respectively. While for
session four (post-test) boys reported a greater percentage of
border tough than girls (14.3%, 11.2%) respectively. For the
wrong answer reason, the girls reported a higher percentage of
the wrong answer in session one and a lower percentage in
session 4 than boys (38% to11.2%; 26.8% to 18.1%)
respectively.

Table I Total number of trials per session by gender

Girls Boys
Sessions
n % n %
1 1194 58.0 865 42.0
2 898 51.5 847 48.5
3 857 54.4 718 45.6
4 720 52.6 650 47.4
Total 3669 54.4 3080 45.6
The game player status (success or fail) was also reported.
Table 2 shows the percentage of success and failure by session
for the 15 participants. The results indicated that the
percentage of fail was decreased from 40% to 23.2%, while
the success percentage was increased from 60% to 76.8%.
This result indicated that there is a positive enhancement in
the player status score by 16.8%.

Table II Players status during the game sessions

Fail Success
Sessions
n % n %
1 823 40.0 1236 60.0 Fig. 3. (a) Shows the percentage of failure by gender caused
2 628 36.0 1117 64.0 by the Maze border, (b) Shows the percentage of failure by
gender caused by the wrong answer.
3 509 32.3 1066 67.7
4 318 23.2 1052 76.8
The players' failure data were analyzed by the gameplay tool,
Total 2278 33.8 4471 66.2 the results in Table 3 shows the percentage of failures by
playing tool and by gender. The Maze playing tool reported
the highest percentage of failures. While the two baskets tool

96

Authorized licensed use limited to: b-on: Universidade de Trás-os-Montes e Alto Douro. Downloaded on April 05,2023 at 16:58:47 UTC from IEEE Xplore. Restrictions apply.
 reported the lowest failure percentage (55.8%; 3.3%)
respectively. Boys almost reported a higher percentage of
failures in all playing tools than girls (54.1%; 46%)
respectively.
Table III. Percentage of deviation in attention and
understanding using pre-posttest analysis by player status and
gender.
Boys Girls
Fail Success Fail Success Fail
Attention -22.9 -15.1 22.8 22.3 27.6
Understanding 14.2 2.1 31.4 5.5 20.7
IV. DISCUSSION
As indicated in the analysis of the results the automatic
analytics model was able to automatically and accurately
analyze and evaluate the serious game effect on game players
learning skills. The use of embedded software components
facilitates the data collection and managed to replace the
traditional pre-posttest with an automatic pre-post
measurement tool. The use of the Unity game analytics tool
provides us with quantitative measures, metrics, and tools for
tracking players' interactions with the serious game. This was
consistent with other studies that used game analytics in
evaluating the player's knowledge [12], [13]. The use of
robotic process automation facilitates the process of
converting the text file data into database records and helped
us in cleaning and analyzing the players' dataset. The RPA
bots managed to perform complete data processing and
analysis without human interaction.
The inclusion of basic statistical packages in our model helped
us in automatically analyzing the game players' data and
provides a descriptive analysis of players learning skills.
The results showed in Table 1 indicated the total number of
trials by players between boys and girls and the results showed
that, in general, the number of these trials decreased between
each session, and that followed the session. Table 2, showing
the percentage of success and failure in general for the
participants and shows that the number of failed trials
decreased after each session from the session that followed,
and consequently the number of successful trials between the
sessions increased.
With the indicator of the decrease in the number of trials
between each session and that followed, which is directly
proportional to the failed trials in these sessions, thus this
indicates that the cognitive skills and experience increased
when the players were learning from their mistakes, as they
need fewer trails to end the game session.
The results show that the girls needed more trials than the
boys to end the sessions, however, the improvement in the
success rate for the girls was higher than the Boy when
compared to the first session with the last session. The results
also indicated that two main reasons were reported, either
touching the maze border during the play activity or achieving
the activity with the wrong answer. Overall, the results
showed that the player's collision with the border of the maze
Authorized licensed use limited to: b-on: Universidade de Trás-os-Montes e Alto Douro. Downloaded on April 05,2023 at 16:58:47 UTC from IEEE Xplore. Restrictions apply.
decreased within the sessions, and a comparison between the
results of the first and fourth sessions from 41.2% to 12.9%,
these results indicate that the skills of focus and sustained
attention and selective attention increased among players
within the sessions, as they became more attentive to the
borders of the maze and more ability to hold in the object, and
not drop it to get it to destination position (basket) correctly.
The experiment results confirmed the findings of [1], [3], [4],
Total
[9], [18] who indicated that serious games could be used as an
educational tool for enhancing students' knowledge and
Success cognitive skills. In terms of attention and understanding, the
20.7 finding of the study is consistent with other previous
4.1 research[1], [3], [6], [18]. This research shows a case study
of an automatic analytics model for measuring players
learning skills in a serious game and the model managed to
replace the traditional pre-posttest with a fully automatic pre-
posttest. We have tested the model on a validated serious
game and the results indicated the efficiency of this model is
replacing the traditional (paper-based) testing methods.
The promising results obtained in this case study recommend
that the integration of game analytic tools with robotic process
automation can be generalized to other serious games for
measuring the player's knowledge and skills.
This research has some limitations related to the limitation on
the game analytics tool and the difficulties of integrating
robotic programming into the Unity games development
platform. The RPA works very well on the server-side while
integrating the RPA development environment with unity will
be extremely helpful. Another limitation, the model was
applied only in one case and on a limited sample of students,
which could partially affect the results. However, we will
consider these limitations in our future work.
Future work could be further testing this model on other
serious games and with larger samples and datasets. Also
further developing the analytical model for inferential
statistics is highly recommended.
V. CONCLUSION
This research presents a case study of a new model in
gathering and analyzing game player data using robotic
process automation and automatic analytics tools. The model
was tested using a serious game developed for enhancing
students' cognitive and learning skills. The efficiency of data
gathering and management obtained on this model show that
the automatic analytics tools and the integration of robotic
process automation can indeed predict the improvement in
players' cognitive and learning skills development. The case
study results showed that the suggested model can replace
traditional pre-post testing methods.
This research has some limitations. The automatic analytics
model was limited to descriptive analysis, and the robotic
process automation was limited to data management.
Furthermore, the model was tested on a small sample from
two schools, which could limit the statistical analysis.
However, we believe this approach could be applied to a
wider range of game players.
97
 Future work on this research includes the development of an
inferential statistical analysis tool and testing the model on
more advanced serious games and on bigger datasets.
VI. ACKNOWLEDGMENT
The authors acknowledge participants, families, and
institutions that collaborated to make this study possible.
Special thanks to the Palestinian ministry of education and the
INTETECH company for facilitating the development and
implementation of this research project.
VII. REFERENCES
[1] P. García-Redondo, T. García, D. Areces, J. C. Núñez, and C.
Rodríguez, “Serious games and their effect improving attention in
students with learning disabilities,” Int. J. Environ. Res. Public
Health, vol. 16, no. 14, pp. 1–12, 2019, doi:
10.3390/ijerph16142480.
[2] L. J. C. Stege and C. Communication, “SERIOUS GAMES IN
LEARNING Bachelor thesis.”
[3] H. Chi, E. Agama, and Z. G. Prodanoff, “Developing serious games
to promote cognitive abilities for the elderly,” 2017 IEEE 5th Int.
Conf. Serious Games Appl. Heal. SeGAH 2017, 2017, doi:
10.1109/SeGAH.2017.7939279.
[4] A. B. Saavedra, F. J. Á. Rodríguez, J. M. Arteaga, R. S. Salgado,
and C. A. C. Ordoñez, “A serious game development process using
competency approach. Case Study: Elementary School Math,” ACM
Int. Conf. Proceeding Ser., vol. 10-12-September-2014, no. April
2016, 2014, doi: 10.1145/2662253.2662352.
[5] P. García-Redondo, T. García, D. Areces, J. C. Núñez, and C.
Rodríguez, “Serious games and their effect improving attention in
students with learning disabilities,” Int. J. Environ. Res. Public
Health, vol. 16, no. 14, 2019, doi: 10.3390/ijerph16142480.
[6] M. H. Hussein, S. H. Ow, L. S. Cheong, and M. K. Thong, “A
Digital Game-Based Learning Method to Improve Students’ Critical
Thinking Skills in Elementary Science,” IEEE Access, vol. 7, pp.
96309–96318, 2019, doi: 10.1109/ACCESS.2019.2929089.
[7] S. Abdelali, B. Othman, E. L. A. Lotfi, B. Amine, and E. Fatiha,
“Serious Games in Business Field, the Challenges and the
Perspectives Serious Games in Business Field, the Challenges and
the Perspectives,” no. MAY, 2014.
[8] L. Stege, “Serious Games in Learning Processes,” 2011.
[9] O. Allal-Chérif and M. Makhlouf, “Using serious games to manage
knowledge: The SECI model perspective,” J. Bus. Res., vol. 69, no.
5, pp. 1539–1543, 2016, doi: 10.1016/j.jbusres.2015.10.013.
[10] V. H. Carvalho, T. Martins, F. Soares, and M. Araújo, “Total
Challenge: A Serious Game for Stimulating Cognitive Abilities,”
Int. J. Adv. Corp. Learn., vol. 9, no. 1, p. 4, 2016, doi:
10.3991/ijac.v9i1.4903.
[11] Y. Wang, P. Rajan, C. S. Sankar, and P. K. Raju, “Let Them Play:
The Impact of Mechanics and Dynamics of a Serious Game on
Student Perceptions of Learning Engagement,” IEEE Trans. Learn.
Technol., vol. 10, no. 4, pp. 514–525, 2016, doi:
10.1109/tlt.2016.2639019.
[12] N. A. A. Zaki, N. Z. M. Zain, N. A. Z. M. Noor, and H. Hashim,
“Developing a conceptual model of learning analytics in serious
games for stem education,” J. Pendidik. IPA Indones., vol. 9, no. 3,
pp. 330–339, 2020, doi: 10.15294/jpii.v9i3.24466.
[13] C. Alonso-Fernández, I. Martínez-Ortiz, R. Caballero, M. Freire,
and B. Fernández-Manjón, “Predicting students’ knowledge after
playing a serious game based on learning analytics data: A case
study,” J. Comput. Assist. Learn., vol. 36, no. 3, pp. 350–358, 2020,
doi: 10.1111/jcal.12405.
[14] D. B. Clark, M. M. Martinez-Garza, G. Biswas, R. M. Luecht, and
P. Sengupta, “Driving assessment of students’ explanations in game
dialog using computer-adaptive testing and hidden Markov
Modeling,” in Assessment in game-based learning, Springer, 2012,
pp. 173–199.
[15] B. Manero, J. Torrente, M. Freire, and B. Fernández-Manjón, “An
instrument to build a gamer clustering framework according to
gaming preferences and habits,” Comput. Human Behav., vol. 62,
pp. 353–363, 2016.
[16] G. Siemens and P. Long, “Penetrating the fog: Analytics in learning
and education.,” Educ. Rev., vol. 46, no. 5, p. 30, 2011.
[17] T. Kowsalya, S. Pratheba, K. Punithavarshini, and S. S. Ram,
“Robotic Process Automation in Social Innovation for Education
System.”
[18] D. K. Ramos and H. M. Melo, “Can digital games in school
improve attention? A study of Brazilian elementary school
students,” J. Comput. Educ., vol. 6, no. 1, pp. 5–19, 2019, doi:
10.1007/s40692-018-0111-3.

98

Authorized licensed use limited to: b-on: Universidade de Trás-os-Montes e Alto Douro. Downloaded on April 05,2023 at 16:58:47 UTC from IEEE Xplore. Restrictions apply.