Democratizing Game Learning

Analytics for Serious Games

Pérez-Colado, Víctor Manuel; Pérez-Colado, Iván José; Martínez-Ortiz, Iván;
Freire-Morán, Manuel; Fernández-Manjón, Baltasar

victormp@ucm.es

Facultad de Informática, Universidad Complutense de Madrid

GALA 2021 | 1-2 December 2021 | Online

 Serious games as assessment tools
Use of Serious games for education have growth due to their multiple benefits.
Black box (mainly): just scores, times, etc.
White box: report everything for analysis.

Greater
Immediate attention span Only a few serious games
feedback include assessment

Real-process Evidence based

simulation assessment

Formal validation: e.g. pre-post tests to
measure effectiveness.
✘ Game-specific/ad-hoc; small scale.
Learning Analytics:. e.g. trace all player
interactions and analyze to correlate
scores.
✓ Better for replicability and
scalability.
 LA usefulness & benefits vs Low
Current GLA situation application

Causes:

Missing LA roles in small game studios Complex analytics infrastructure

+
Satisfy LA educational and regulatory needs

Solution: systematize and democratize GLA so responsibilities are delimited

(e.g. Developers only focus in selecting the in game events relevant for analysis)
Approach: Standards to democratize GLA
Robust GLA centered in e-Learning standards:
Trace
● Tracker: Software tool used to generate player traces
○ Simplifies the trace creation providing a higher level API.
○ Manages the delivery of the traces, protocols and connections.
Format
● xAPI: Format for describing student interactions with the SG
○ xAPI Format: an actor, a verb, an object and a context (all timestamped).
○ Customized through profiles and templates => xAPI for Serious Games.
● Learning Record Storage: API for storing xAPI traces Store
○ Use off-the-shelf xAPI standard-based LRSs to store the interaction data
○ The LRS is used by stakeholders to assess and analyze the gameplays

Systematize developers task to use the tracker = Simpler GLA.

 Our scalable Game Learning Analytics ecosystem
Systematizing GLA requires SIMVA validation tool
reliable software to manage the
● Orchestrates LA activities and LTI.
traces. ● Manages users, groups (A/B), pre-post,
analysis and trace storage.
e-UCM current open-code scalable
● Unity plugin for easy setup with OAuth2.0.
GLA ecosystem consist in:

e-UCM xAPI tracker TxMon analysis tool

● C#, JS and Unity. ● Analyzes xAPI traces.
● High-level interface for devs. ● Default visualizations.
● Compatible with xAPI for SGs.
Our current standards-based scalable GLA ecosystem
The e-UCM xAPI tracker
The most relevant piece for developers:

● High-level interface to create xAPI for

SGs traces and send to endpoint
● Save traces locally in multiple formats
● Designed for SGs and multiplatform
● Authorization with up to OAuth2
● Resilient:
○ Different working modes for stability.
○ Policies to avoid missing traces.

It missed default xAPI LRS support New Full xAPI & LRS support + Working modes
A standard-based scalable generic GLA infrastructure

Creating a GLA infrastructure requires different components

depending on the goals (e.g. small single game analysis vs
Tracker
multi-game systems).

GLA processes are complex and fragile and achieving a fully

integrated approach is not always possible. Format

We propose an strategy using existing third-party software

that focuses on the collection and storage process in xAPI LRS

format using xAPI trackers and LRS.

Learning Record Consumer

Off-the-shelf xAPI LRS xAPI-enabled trackers can use OTS LRS

Reduce: Storage Analytics & Dashboards Notes

ADL LRS Apache 2.0 None Reference impl. model

● Costs
● Technical Learning GPL 3.0 Included in SaaS paid Oldest LRS
knowledge Locker version

Flexibility: Rustici Private Included as part of SCORM Inside mediator tool SCORM
LRS Cloud. Cloud; Supports CMI-5
● Licenses
Yet Analytics Apache 2.0 None Works locally (offline). Has a
● Formats LRS (SQL ver.) cloud version for AWS
● Analysis
Watershed Private Simple reports and Supports CSV import/export
● Price
LRS (SaaS only) dashboard. Rest is Paid

Apereo ECL-2.0 Available when connected Also supports IMS Calyper

OpenLRW (educational) to Apereo OpenDashboard and OneRooster
Off-the-shelf xAPI Unity Trackers
Unity trackers supporting LRS

TinCan.NET UnityGBLxAPI Unity-xAPIWrapper Compared to e-

Developer Rustici Software Dit-iT! Games ADL ( Trey Hayden) UCM tracker:

Asynchronous No Yes Yes ● Lower level

interface.
Cross-platform Depends on .NET Partial* Yes
● Only basic
Unity support Partial Yes Yes OAuth1.
xAPI Support Complete xAPI API Misses responses* Only Statements API
● Only work
online.
Notes .NET 4.0 Uses TinCan.NET Only depends on ● No backups.
Uses HttpWebRequest underneath but with JSON.NET.
causing cross-platform UnityWebRequest More effort to ● No resiliency.
and Unity issues. and a async queue. implement other APIs Next step:
Complex excels. (e.g. to support cmi5) Supporting CMI-5
* Can be fixed easily.
Adaptation of the e-UCM tracker for generic use
New tracker features:
● Improved xAPI compliance
● CMI-5 launch - LMS
forwards authentication,
status and more to
launched game
● 3 Modes
∙ LMS mode (CMI-5 launch)
∙ LRS mode preconfigured
∙ (Old) SIMVA Mode
Conclusions and future work
LA is decisive for the adoption of SGs in the learning process as analytics can provide
better evidence of learning than just scores, but they are too complex for small or medium
developers.

To reduce the complexity, we propose a systematization and democratization of the tools

using standards in a xAPI-tracker-LRS strategy. To democratize our tools we had to
include full xAPI support and new working modes that support CMI-5.

- xAPI is in the process of becoming a new official IEEE open standard

Developers that want to include LA in their games using this strategy can use any of the
existing COTS software to setup a simple architecture, including our tracker, that is
simpler and provides more flexibility and resiliency.
 Thanks!
Contact: victormp@ucm.es
Our publications: https://www.e-ucm.es/publications/all/
Our open code: https://github.com/e-ucm/

This work has been partially funded by Regional Government of Madrid (eMadrid S2018/TCS4307, co-funded by the European
Structural Funds FSE and FEDER), by the Ministry of Education (TIN2017-89238-R, PID2020-119620RB-I00), by MIT-La Caixa
(MISTI program, LCF/PR/MIT19/5184001) and by Telefonica-Complutense Chair on Digital Education and Serious Games.