Learning Analytics

Mahesh Abnave
maheshabnave@cse.iitb.ac.in

Abstract course content and providing learners with extra guidance

whenever needed.
With increasing digitization, e-learning Section 2 and section 3 of this paper discuss two summary
platforms have emerged as a major shift in papers in LA. Section 4 discusses the paper exploring
the learning process. With ever-increasing various machine learning techniques for predicting
online courses and online learners, student performance in online learning platforms. Section
understanding learners' behavior, tracking 5 discusses the paper surveying various visualization
their progress, and predicting their success in approaches followed by LA systems for conveying
the online course becomes a new challenge insights.
faced by the instructors in these e-learning
platforms. The field of learning analytics
caters to these challenges and empowers
instructors with key insights into online
learners’ behavior and their learning
progress. This paper explores existing
research done in Learning Analytics (LA),
right from different kinds of inputs processed
by LA systems, various tools, datasets used,
different insights obtained, and different
visualizations used for effectively conveying
these insights. Figure 1. Growth of MOOCs
2 Predicting academic performance: a
1 Introduction systematic literature review by Hellas et
In today's digital world, all activities and services are
al.
facing the challenge of digitalization. Everyone has to Paper by Hellas et al. [2] surveys 357 research papers
upgrade their products and services to keep up with the
and articles from the year 2010 to 2018 that focus
digitalization and provide online versions of their product
and services to the end-users. The education sector is also
mainly on student performance in various courses. It
not left behind. Exponential growth in the Massive Open divides papers into four streams: computer science,
Online Courses (MOOCs) has given a new boost to the STEM (science, technology, engineering, and
global EdTech industry. The COVID19 pandemic has also mathematics), multi-disciplinary and other. It also
forced worldwide educational institutes to switch to the notes that for some papers, the stream of courses
online environment. For example, the report by class examined is unclear. The paper summarizes the types
central [1] notes that in 2020, providers launched over of performance being predicted, and the factors and
2800 courses, 19 online degrees, and 360 micro- methods used to perform the predictions.
credentials. Table 1 shows how the top MOOC providers
look in terms of users and offerings in 2020 [1]:
Lerners Courses Micro Degrees
credentials
Coursera 76 m 4600 610 25
edX 35 m 3100 385 13
FutureLearn 14 m 1160 86 28
Swayam 16 m 1130 0 0

Table 1. MOOCs in 2020

However, in online education, it becomes difficult for the
instructor to gauge the learner’s progress and learning
behavior. Learning analytics has emerged as a capable tool
to help the instructor with this information. Learning Table 2. Use of factors to predict student performance
analytics provides instructors with a wide range of by year
information regarding student learning behavior, struggle, For example, as shown in Table 2 above, the paper
and progress and even helps in predicting students' course notes that the course performance and course
grades. This further helps the instructor in updating the
engagement are increasingly used by various papers
1
as an input metric to predict the course performance
in the final exam. Paper also notes that many papers
did not consider metrics such as team cohesion for
tasks such as team assignments and projects and
learners' physical and mental conditions, such as
depression, etc.
Authors have also have divided the methods used for
predictions into a few broad categories:
classification (supervised learning, Naïve Bayes,
decision trees), clustering (unsupervised learning),
mining (finding frequent patterns and/or feature
extraction), and statistical (correlation, regression, t-
testing, etc). They have listed down the number of
times these methods are used for predictions over the
years as shown in Table 3:
Table 3. Use of methods to predict student performance
by year
However, authors have noted that it won't be
appropriate to compare the accuracies of these
models as the context of their studies varied
significantly and the quality of reporting also varied
widely.
Similarly, Table 4 depicts that the papers mainly
focused on predicting course grades.
Table 4. The aspects of student performance predicted
over the years
It also notes that a significant number of papers have
vague or unspecified target performance metrics.
Paper also maps the inputs used to their
corresponding predictions (Table 5). It can be seen
that previous and existing course performance and
engagement are used frequently for predicting
assignment performance and exam and course
grades.
Finally, the authors point out the ethical risks
involved in the data collection and suggest that
2
consent should be taken from the students before
collecting data. They also comment on the model
risks and points out the extra care required to ensure
that models don’t get biased toward ideal students
(blood group O, Asian, educated parents)
discouraging other non-ideal students.
Authors have also appealed to the community to
report more details in their publications, share their
data set, collaborate more with each other, help each
other replicate existing results, and also report details
of their failed experiments in their publications.
Table 5. Cross-tabulation of features (inputs) and
performance values (outputs)
3 Educational Data Mining and Learning
Analytics in Programming: Literature
Review and Case Studies by Ihantola et al.
Ihantola et al. [3] surveyed 76 papers from 2005 to
2015 with a focus on Learning Analytics for
programming courses. The paper divides metrics used
in the literature for analytics into different granularities.
It also discusses various automated tools used for
collecting these metrics. Finally, the authors have
mentioned various prediction goals and different
methods employed in the literature.
To start with, the paper divides the metrics used for
predictions in the granularities as shown in Figure 2.
Metrics at different granularities imply different
collection frequencies and data set sizes.
Figure 2. Input metrics at different levels of granularity
Metrics such as source code editing behavior, syntax
errors on the compilation, and test case success rate
can be collected through Integrated Development
Environment (IDE) instrumentation, whereas other
metrics such as assignment submissions can be
collected by the means of other assessment systems
as shown in Figure 3. The paper further lists
programming data collection tools, supported
programming languages, the UI clients in which
those tools operate, and the kind of data collection
done as listed in Table 6. For example, TestMyCode
is a Netbeans plugin that records keystrokes and
supports Java programming language, whereas
Problets is a web-based IDE.
The authors note that 63 studies (18%) provided
descriptive statistics. Most of them reported at least
basic counts and percentages. 22 studies (29%) also
conducted the more detailed statistical analysis, such
as inferential, Bayesian, and t-test, etc, while 14
studies (18%) conducted some form of exploratory
statistical analysis, such as correlation, regression, or
factor analysis.

Table 8. Comparing datasets (D = Duke University,

Figure 3. Data instrumentation and collection system
The Paper further compares three publicly available
datasets for learning analytics as summarized in
Table 7. Authors have compared the separate set of
datasets and compared them on the attributes of
programming language supported, corresponding
course, the university where it was collected, number
of tasks performed by students, etc as shown in Table
8.
Table 6. Programming data collection tools
H = University of Helsinki, T = Toronto University,
Y - York College. Assignment metadata: a =
assignment description, t = test cases, p = pedagogy,
f = feedback)
Finally, to check if the conclusions of different
papers can be reproduced or not, authors have tried
to implement some papers. They implemented one
paper as it is, one with new data set and one with both
new data set and analysis methods. They note that it
is possible to recreate the results of the original
paper. However, they also noted that dataset
metadata is absolutely necessary (as different
datasets require different types of pre-processing).
Also, it is important for the authors to clearly state
when a certain metric is calculated using the only
subset of data, for example, the best score of each
exercise can be calculated both by including or
excluding the students that attempted exercise but
never compiled it successfully. Also, different terms
(for example, “successful compilation”) might have
different meanings depending upon the tool and
programming language used in the study.

For each paper surveyed, the authors have extracted

their research goals and motivation. Research goals
are further divided into three categories: student-
focused, environment-focused and programming
focused as explained in Table 9.
Dataset #students #tasks #data Nature of
points Data
Blackbox 1M NA 830M Compilation
[9] events
CodeHunt 258 24 13K Code
[8] submission
in Java and Table 9. Categorization of the research goals
C#
Authors found it difficult to recreate the conclusions
code.org 500k 2 2.4M Test results
[11] of the original paper included in the survey when a
new dataset is used. They were not able to replicate
Table 7. Comparison of publicly available datasets
the paper’s conclusion that successful compilations
3
increased with more practice because the new
dataset contained smaller exercises at the beginning.
Similarly, when both a new dataset and analysis
method was used, the authors were not able to
replicate the paper’s conclusion. The original dataset
involved exercises in Java whereas the new dataset
involved exercises in Python. This was mainly
because Java errors are more specific than Python
(missing semicolon, parenthesis vs Syntax error) and
Python doesn’t report typing errors.
The authors concluded that the replication of
results in existing LA literature is a challenging task.
Hence, they suggested the community provide a
complete description of the experimental setup,
algorithms, parameters, and datasets. They also
noted that an online log system can be utilized for
this purpose, due to the lack of space in paper
publications.
4 An overview and comparison of
supervised data mining techniques for
student exam performance prediction
by Tomasevic et al.
This paper aims to provide a comprehensive analysis
and comparison of state-of-the-art supervised
machine learning models for student exam
performance prediction, i.e. discovering students at a
“high risk” of dropping the course and predicting
their future achievements, such as the final exam
scores.
The authors have started by discussing the
implementation issues:
• The problem of cold start: refers to the lack
of sufficient amount of adequate data during
the initial phase. This can be resolved by
importing already existing data from external
Learning Management Systems (LMSs) if
available or by creating student profiles (for
which one can utilize the five-factor model
(FFM) for capturing personality traits as
suggested by Coast et al. [4] and Goldberg
[5]).
• Scalability Issues: With the increase in the
number of students and the amount of
available learning data, the complexity of
analytics modules and computational
requirements is also increasing. This issue
can be resolved by employing dimensionality
reduction and by introducing adequate time
constraints (such as considering the most
recent data).
4
• Data sparsity: Often there is not enough
information about student activities and their
engagement, a large number of students are
passively interacting with the learning
platform (e.g. using the offline learning
materials, etc.) and only a small number of
students are truly interactive. This can be
resolved by appropriate means for improving
student interaction with the system: for
example letting students store lecture videos
within the app so that any interaction with the
video can be captured by the app, in-app
discussions requiring students to comment or
share their thoughts, in-app quizzes, and
tests.
Authors have used Open University Learning
Analytics Dataset (OULAD). It contains information
about 22 module-presentation (that is, courses and their
offerings) and 32,593 students. The dataset includes
different student-related data such as their assessment
results and logs of their interactions with the Virtual
Learning Environment (VLE) represented by daily
summaries of student clicks on different “resources”,
i.e. the learning material (10,655,280 entries). The
entire dataset has been anonymized but properly
annotated with unique identifiers so the cross-
correlation of various data is possible to be made across
the dataset. Figure 4 shows various tables and their
relation in OALAD dataset.
Figure 4. OULAD database schema
Authors preprocessed the dataset and logically
divided its various fields into three categories of
features: past student performance, student
engagement, and student demographics as explained in
Table 10:

Table 10. Data preprocessing & logical categorization of
features
Features 4 to 9 are the number of clicks in each of
the six intermediary assessment tests. Features 10 to
15 are scores of these intermediary tests. Feature 16
is the number of attempts in the final exam.
For the classification task, a predicted score less than
40% was interpreted as a fail (model outputs 0),
whereas a score greater than equal to 40% was
interpreted as a pass (model outputs 1). For the
regression task, the score was normalized in the range
of 0 to 100.
Table 11 compares the F1 score of various models
on student final exam pass / fail classification task for
a different combination of input features. As can be
seen, by the red highlighted cell, the best score is
obtained by Artificial Neural Network (ANN) for the
combination of engagement and performance metrics.
assessments using all features (D+E+P). For
classification, ANN reported a maximum average F1
for the first four assessments whereas, for the 5th and 6th
assessments, SVM gave the best results, as shown in
Figure 5. For the regression task, ANN reported
minimum RMSE after all assessments (Figure 6).
Thus, for both classification and regression tasks, the
overall highest precision was obtained with artificial
neural networks by feeding the student engagement
data and past performance data, while the usage of
demographic data did not show significant influence on
the precision of predictions. To exploit the full potential
of the student exam performance prediction, it was
concluded that adequate data acquisition functionalities
and the student interaction with the learning
environment are a prerequisite to ensuring a sufficient
amount of data for analysis.

Figure 5. F1 score comparison for final exam pass / fail

Table 11. F1 scores of different models for final exam classification
pass / fail classification task with different input feature
combinations
Similarly, Table 12 compares the root mean square
error (RMSE) of various models on the final exam
score prediction (regression) task for different
combinations of input features. Again, as highlighted
by the red cell, the least RMSE was obtained for ANN
for the combination of engagement and performance
metrics.

Figure 6. RMSE comparison for final exam score

prediction (regression)

Table 12. RMSE of different models for final exam score

prediction (regression) task with different input feature
combinations
Authors also carried out final exam pass / fail
classification and exam score prediction (regression)
and prediction, after each of six intermediary

5
5 A Systematic Review of Empirical Studies
on Learning Analytics Dashboards (LAD)
– A Self-Regulated Learning (SRL)
Perspective by Matcha et al.
In this paper, Matcha et al. have surveyed the literature
on Learning Analytics Dashboards (LADs) research
with the goal to understand how much the design of
LADs is grounded in the literature on self-regulated
learning (SRL).
Self-Regulated Learning (SRL) theory focuses to
enhance students learning skills by exploring cognitive
and metacognitive processes. It allows students to
actuate and sustain cognition behaviors and effects that
are systematically directed towards their goals. There Figure 7. Type of reference frames used in LADs for each
are various SRL models in the literature. Authors have target user group
considered the Information Processing Theory of SRL
by Winne and Hadwin. It involves four cyclical phases:
task definition, goal setting and planning, enactment of
tactics and strategies, and adaptation.
Within these four phases, five components are running
recursively to form what is called as COPES model:
• conditions: in which learners define the tasks,
set learning goals, and plan learning constraints
• operations: this phase involve putting learning
plan and strategies into practice
• products: operations should lead to some
products such as memory recall or essay
Figure 8. Research approaches used for data collection
• evaluation: this involves evaluating the
learning products against the standards and The authors examined how many LADs are
goals set in the first step developed with some educational theories or models in
• standards: evaluation may lead to revising the mind and found that only 8 papers out of 29 considered
standards and goals some form of theory for developing LADs. Also, none
Authors shortlisted 29 papers for literature review. of them considered self-regulated-learning theory as
They have reported statistics such as the number of shown in Table 13.
indicators used for visualizing information in different
reference frames as shown in Figure 7. Individual
reference frame means students can only see their own
activities. An average comparison reference frame
provides students with the average comparison against
their peers; whereas a course-wide reference frame
simply provides all information to all students.

Table 13. Educational theories and models used to derive

the developments of LADs
Thus, the authors conclude that LADs are rarely
based on the learning theory. They neither provide
insights into effective learning methods and strategies

6
nor provide any evaluation of such techniques. Thus,
the authors say that LADs in their current form, cannot
be recommended to support metacognition.
For this reason, the authors suggest that any future
research in the domain of LAD should not start with
any presumption about the representation of data and
analytics results. Finally, to formalize this notion, the
authors have proposed the model of user-centered
learning analytics systems (MULAS). It involves
four cyclical and recursive dimensions for developing
LADs.
• Theory: asks to avoid prior design decisions
about the representation of data and analytics.
Instead, the focus should be learner-centric and
inform learners of effective learning techniques
and strategies.
• Design: should be informed by theory. It should
be iterative and cyclic.
• Feedback: should be dialogical and not
unidirectional from educators to learners.
Learners should be allowed to update their user
models when they find discrepancies in data or
analytics results
• Evaluation: It was found that students who
received feedback through the dashboard
showed significantly higher final scores than
those who did not.
The paper concludes by restating the need for
strong grounding of learning analytics systems and
LADs in the literature on effective study methods and
feedback. It also suggested the need for
interdisciplinary teams with the expertise of learning
sciences, human-information interaction, design, and
research methods.
6 Conclusion
With increasing data collection tools and online
learning platforms, Learning Analytics will continue to
prove key ingredient in ensuring the success of e-
learning platforms. Despite decades of research
Learning Analytics still faces difficulties mainly
because of a lack of standardization in tools and
datasets to readily replicate research experiments in
different Virtual Learning Environments (VLE) and
poor reporting of research details to facilitate
replication of research experiments. Learning
Analytics is also challenging because every VLE is
unique and different from others. So, LA
methodologies applied in one environment cannot be
replicated in other VLEs without making any changes.
Nevertheless, it is well proved by Tomasevic et al. that
with proper data engineering and using sophisticated
machine learning techniques, one can still predict
7
learners' progress and obtain useful actionable insights.
Especially Neural Networks are found to deliver
promising results. Also, past student grades and
learners’ engagement proved to be the most useful
metrics in gauging learners' progress. Student
demographic metrics did not show any significant
improvement in grade prediction performance scores.
Automated tools demonstrate great importance in
capturing student engagement data such as clicks and
video views. From the insights visualization
perspective, it was found that LADs were rarely
designed with the goal of optimizing students learning
strategies in mind. Instead, most LAD designs are
influenced by the nature of data and visualization tools
available and LAD designs in the existing literature.
However, as noted by Matcha et al., an important
challenge in LADs development is learner-centric
designing to better communicate effective learning
techniques and strategies with students.
7 Acknowledgments
I would like to express deep gratitude to Prof.
Kameswari Chebrolu for giving me the opportunity to
work on the topic of Learning Analytics. I would also
like to share my deep appreciation for the continuous
guidance and direction she provided without which I
would have found myself fumbling in the vast
literature.
References
[1] D. Shah, "By The Numbers: MOOCs in 2020,"
30 Nov 2020. [Online]. Available:
https://www.classcentral.com/report/mooc-
stats-2020/.
[2] A. Hellas, P. Ihantola, A. Peterson, V. Ajanovski
and M. Gutica, "Predicting academic
performance: a systematic literature review,"
ITiCSE 2018 Companion: Proceedings
Companion of the 23rd Annual ACM
Conference on Innovation and Technology in
Computer Science Education, pp. 175-199, July
2018.
[3] P. Ihantola, A. Vihavainen, A. Ahadi, M. Butler,
J. Börstler, S. Edwards and E. Isohanni,
"Educational Data Mining and Learning
Analytics in Programming: Literature Review
 and Case," Proceedings of the 2015 ITiCSE on
Working Group Reports, pp. 41-63, July 2015.

[4] P. J. &. M. ,. R. R. CCoast, "Revised NEO

personality inventory (NRO-PI-R) and NEO five-
factor inventory (NEO-FFI) manual," Odessa,
Psyclogical Assessment Resources, 1992.

[5] L. Goldberg, "The structure of phenotypic

personality traits," The American psychologist,
vol. 48(1), no. 26-34, 1993.

[6] N. Tomasevic, N. Gvozdenovic and S. Vranes,

"An overview and comparison of supervised
data mining techniques for student exam,"
Computers & Education, vol. 143, January
2020.

[7] "A Systematic Review of Empirical Studies on

Learning Analytics Dashboards: A Self-
Regulated Learning Perspective," IEEE
Transactions on Learning Technologies, vol.
13, no. 2, pp. 226-245, April 2020.

[8] A. Kumar, "Problets," [Online]. Available:

http://problets.org/.

[9] "BlueJ Blackbox Data Collection Project,"

[Online]. Available: http://blackbox.bluej.org/.

[10] "CodeHunt - A serious education game,"

[Online]. Available:
https://github.com/Microsoft/Code-Hunt.

[11] "code.org - Free Computer Science courses for

all grades, from K12 grade to high school,"
[Online]. Available: https://code.org/.