Emotion Detection through Smartphone’s Accelerometer and
Gyroscope Sensors
Orestis Piskioulis Katerina Tzafilkou
School of Science and Technology, School of Science and Technology,
International Hellenic University, International Hellenic University,
Thessaloniki, Greece Thessaloniki, Greece
piskioulisorestis@gmail.com katerinatzaf@gmail.com
ABSTRACT
Emotion recognition is essential for assessing human emotional
states and predicting user behavior to provide appropriate and
personalized feedback. The wide range of Smartphones with ac-
celerometers, microphones, GPSs, gyroscopes, and more motivate
researchers to explore the automatic emotion detection through
Smartphone sensors. To this end, mobile sensing can facilitate the
data retrieval process in a non-intrusive way without disturbing
the user’s experience. This study seeks to contribute to the field of
non-intrusive mobile sensing for emotion recognition by detecting
user emotions via accelerometer and gyroscope sensors in Smart-
phones. A prototype gaming app was designed and a sensor log app
for Android OS was used to monitor the users’ sensor data while
interacting with the game. The recorded data from 40 users was
processed and used to train different classifiers for two emotions: a
positive (enjoyment) and a negative (frustration) one. The valida-
tion study demonstrates a high prediction of 87.90% for enjoyment
and 89.45% for frustration. Our findings indicate that by analyzing
accelerometer and gyroscope data, it is possible to make efficient
predictions of a user’s emotional state. The proposed model and its
empirical development and validation are described in this paper.
CCS CONCEPTS
• Human-centered computing; • Smartphones; • Non-
intrusive emotion detection;
KEYWORDS
Mobile sensing, Mobile emotion sensing, Emotion Detection, Smart-
phone Sensing
ACM Reference Format:
Orestis Piskioulis, Katerina Tzafilkou, and Anastasios Economides. 2021.
Emotion Detection through Smartphone’s Accelerometer and Gyroscope
Sensors. In Proceedings of the 29th ACM Conference on User Modeling, Adap-
tation and Personalization (UMAP ’21), June 21–25, 2021, Utrecht, Netherlands.
ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3450613.3456822
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8366-0/21/06. . . $15.00
https://doi.org/10.1145/3450613.3456822
Corrected Version of Record. V.1.1. Published July 20, 2021.
Anastasios Economides
SMILE lab (smile.uom.gr),
Information Systems IPPS, University
of Macedonia, Thessaloniki, Greece
economid@uom.gr
1 INTRODUCTION
Emotion recognition is essential for assessing human emotional
states and predicting user behavior to provide appropriate and
personalized feedback [1]. The automatic recognition of emotions
is an interesting but difficult problem and various researchers tried
to solve. For example, it is interesting to find out what types of
music and movies cause what emotions to people. In addition, robots
will soon be used in various areas of our lives. The robots will be
more intelligent and humanized if they can ’feel the mood’ of the
person with whom they interact. It is also important to recognize
the people’s emotional state and its development to determine their
psychological health and mental welfare [2], [3].
Smartphones are now commonly used in industry for business,
social and entertainment purposes in people’s everyday life. To-
day, Smartphone devices embed several sensors, like microphone,
accelerometer, electronic compass, GPS, proximity, etc. The data
obtained from the sensors has profound information and can be
used to evaluate the social habits of the user, such as physical ac-
tivity, social communications, location, etc. Intuitively, mobile use
and background knowledge are associated with the emotions of
users. For instance, when they are happy, people may play mobile
phone games, or they may feel depressed if they live in a noisy en-
vironment. Several works were inspired by this, using Smartphone
sensing data for emotion detection.
For instance, a multifactorial statistical model to understand
everyday stress by comprehensively analyzing cell phone data and
weather conditions has been proposed in [4]. For cold-start emotion
prediction using transfer learning, the authors in [5] used sensor
data, APP usage information, and SMS content. However, their
approach is content-based involving privacy-sensitive user’s data
such as SMS content. In [6] the authors suggested a mood detection
system using the period of email, SMS, venue, and app use. Previous
studies have mainly investigated the identification of only one
emotion assuming that a person perceives only one emotional state
every moment. This study extends previous studies by relaxing this
assumption and considers that a user may be in multiple co-existed
emotions called ‘compound emotions’. According to the theory of
Plutchik [7], a mixture of fundamental emotions would co-exist in
a person at any moment. Compound facial feelings in human facial
expressions were also seen in the analysis of [8], [9]. For example,
a compound emotional expression that incorporates basic feelings
of happiness and surprise is "happily surprised".
Furthermore, although there are several research studies on emo-
tion detection through mobile devices, many of them apply obtru-
sive techniques of data collection (e.g., through external devices
or wearables), eliminating their easy and practical implementation
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
in everyday user-Smartphone interactions [1], [10]. The literature
review has shown that the accelerometer is the most frequently
used modality in non-intrusive mobile sensing methodologies. The
front camera is frequently used as well; however, face tracking in
mobile devices has raised several functional limitations (based on
the difficulty to efficiently capture the user’s full face) that might
affect the user experience during the data collection process [11].
Although researchers have implemented several prediction models
to identify human emotions through mobile sensor data, only a
few models have used solely non-intrusive methodologies and have
managed to achieve high accuracy scores [12]. Overall, the limited
number of studies, and their limitations imply that further research
is needed.
Motivated by the increased need for further investigation of
the prediction of emotions during the user interaction with mo-
bile devices, this study aims to investigate the recognition of two
basic human emotions (one positive and one negative) using ac-
celerometer and gyroscope sensors. Overall, our objective is to
apply a multimodal and non-intrusive methodology to predict user
enjoyment and frustration during the use of a prototype gaming
app.
2 THEORETICAL BACKGROUND
2.1 Models for emotion and mood
measurement
There has been extensive research with regards to emotion and
mood in psychology, sociology, and neuroscience [13]. In general,
“emotion” refers to the present instantaneous feeling, and “mood”
refers to the longer-term normal feelings [14]. Such models can be
used to quantify both instantaneous and long-term sensations, and
they can be used to quantify emotion and mood without limiting
differentiation. The Positive and Negative Affect Schedule (PANAS)
model is a widely used model to measure mood and broad affective
states [15]. Participants completing the PANAS are asked to rate
the degree to which each of the 20 emotions they encountered on a
5-point Likert scale range from “very slightly” to “very strongly”.
Participants can be asked how they feel right now or over longer
periods of time (e.g., over the past month), depending on the intent
of the measured emotion or mood state.
One popular model is Ekman’s six basic categories of emotions
[16]: happiness, sadness, anger, surprise, fear, and disgust. The Ek-
man’s model is intuitive and understandable by the average users,
and it allows multiple emotions to coexist with various intensity lev-
els. The Ekman’s emotion model has been widely adopted by many
studies due to its high applicability [9], [17]. In different approach,
Lövheim [18] compiled a list of 15 emotions that derive from the
nine fundamental emotions of interest, joy, surprise, sorrow, fear,
guilt, disdain, rage, and disgust.
Several emotions and affective states have been pointed out as
‘determinants’ in gaming or game-based learning experiences, since
they tend to affect a user’s overall experience and performance [19].
Overall, the emotions of enjoyment, frustration, confusion and anx-
iety are outlined among the most frequently experienced emotions
during learning with a range of technologies, such as intelligent
tutoring systems, simulations and serious games [20][21].
Orestis Piskioulis et al.
2.2 Emotion Recognition in Smartphones
Mobile Sensing has been attracting research in recent years. The
productivity plateau is anticipated to be reached in around two or
five years, according to Gartner’s hype period of ‘’Emotion Detec-
tion/Recognition” [22]. Nowadays, most people use Smartphones
and researchers have shown that mobile sensing data from Smart-
phones can be adopted to infer and detect human mental well-being
such as stress, anxiety, depression, emotion and mood [9], [23]–
[26]. Several recent studies have managed to predict user emotions
through Smartphones in a non-obtrusive manner. Many of them
used accelerometer data in combination to other modalities like
touch screen [27]–[29] or microphone, [30], [31]. However, their
collection of data lasted for several days or weeks and the feedback
modalities included several more secondary ones like pedometer,
network, GPS, light sensor, keyboard etc.
In a relevant study [12] implemented a multimodal methodology
using accelerometer and keyboard data through a soft sensor to
detect the emotions of happiness, anger, and neutral state. Their
model implementation was based on Decision Trees, achieving
almost 93% accuracy.
In [29] the authors used a combination of accelerometer, gyro-
scope, and touch-screen data to predict two generic affective states
(positive and neutral). Their implemented support vector machine
model achieved a high accuracy score of about 87%. Most of the
works that used an accelerometer unimodal approach were limited
in the detection of stress [32] or pleasantness [33] and generally
achieved lower scores of accuracies compared to the multimodal
approaches.
As shown in [23], DeepMood detected bipolar affective disorder
on a multi-view neural network-based Smartphone using typing
dynamics and accelerometer sensor data. In [34] the author aimed to
assist mobile individuals when they need emotional help to connect
offline with friends. A multifactorial statistical model was proposed
by [4] to understand everyday stress by comprehensively analyzing
cell phone data and weather conditions.
Using mobile mobility trace data, the authors in [35] tracked
user depression states. The identification of positive and negative
emotions using wild mobile product sensors was suggested by
[29]. Also, sensor data, APP usage information and SMS content
were used by the iSelf system [5] for emotion prediction using
transfer learning. However, these previous studies consider only
single emotion detection ignoring the fact that multiple emotions
can co-exist in time [36]. In our paper, we are trying to predict
two basic emotions (enjoyment and frustration), by making use of
Smartphones data from a collection of sensing information [37].
Overall, we notice there are not any methodologies based only
on non-intrusive sensors of accelerometer and gyroscope. However,
motivated by the above-mentioned studies, it seems worthy to
explore the success of accelerometer and gyroscope as the only
inputs to predict both positive and negative emotions.
2.3 Sensors
Modern Smartphones have a high number of sensors available, in
addition to software-based recordings of how the phone is used. For
Android phones, the following sensors might be relevant for the
task of identifying emotions. In the present research and using the
Emotion Detection through Smartphone’s Accelerometer and Gyroscope Sensors
Figure 1: Research Framework
appropriate technology we chose to record data from the following
sensors:
⃝ Accelerometer (with gravity, motion and position detection);
⃝ Accelerometer Linear (without gravity, motion and position
detection);
⃝ Gyroscope (affected by other vibrations, rotation detection).
Both the accelerometer and the gyroscope provide measurements
for each axis in a 3-dimensional coordinate system, and for this
project we will focus on accelerometer readings. The output is a
vector consisting of the acceleration, in each of three directions x,
y, and z.
Ideally, the accelerometer of a phone lying on a flat surface
outputs the values A = {0, 0, ±g}. However, the accelerometer data
is not fully reliable. Even in an immovable phone, its accelerometer
data does not remain constant, but they shift from one reading
to another. For this reason, accelerometers are usually not used
for monitoring movement and orientation to detect emotions or
affective data in general. Although affective data cannot be accessed
directly through the accelerometer, the forces exerted on the system
are clearly generated. This enables us to determine the device’s
orientation and movement [38].
3 MATERIAL AND METHODS
Our research methodology followed a multi-step approach. First,
a prototype psychometric gaming app was developed, and a free
sensor log app was selected to be installed on the users’ Smartphone
devices. Second, a self-reported measurement was designed accord-
ing to the literature review to measure the examined emotions.
Then, an experimental study was conducted on a set of volunteers
who were provided with all the necessary software (i.e., the pro-
totype gaming app installation, the sensor log app installation for
Android OS and the questionnaire link). After the data collection,
the sensor recorded data was processed and the selected features
were extracted. Finally, a set of prediction models was developed,
and their accuracy scores were compared. Figure 1 summarizes
the research methodology to collect feedback from a sample of
participants who interact with the prototype Android app, and then
process the data, extract the features to be fed into the model and
finally train and evaluate the emotion recognition (ER) model.
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
3.1 Prototype App and Sensor Recording
We have developed a gaming app to serve as the prototype app
that participants will interact with during the experimental study.
The prototype app has been developed in a way to be like business
games that big firms are now using to provide an overview of their
employees’ psychometric profile.
Overall, the designed game is a prototype framework that aims
to explore its users’ basic psychometric features. For this reason,
this app is based on similar commercial games (i.e., Skyrise City)
that are used massively in order to build psychometric profiles of
company’s potential employees.
For the creation and successful development of the prototype
app we looked for a platform that could enable us to develop a fully
compatible game with the recording of the sensors data. We finally
decided to use the open-source game development platform, the
MIT App Inventor.
Using MIT App Inventor, we developed a game in the standards
of Skyrise City, capable of giving us a simple and short psycho-
metric game that will arouse the interest of users. Those kinds
of game-based assessments are grounded in validated personality
and cognitive neuroscience frameworks and use real behavior data
to create a user’s psychometric profile. Through the challenges
faced by the user, a user profile can be created along six psycho-
logical characteristics: i) Aptitude, ii) Personal Style, iii) Cognition,
iv) Drive, v) Interpersonal Style, and vi) Thinking Style. Figure 2
depicts some interface snippets from the gaming app.
To record the accelerometer and gyroscope data, a sensor record-
ing app was used. The MIT App Inventor and the Sensor Record
app ware both installed in a mobile phone. The app we used to
record data is a free Android application, Sensor Record, and gives
the possibility to record 11 mobile sensors. You can easily monitor
your Smartphone’s sensor data with this app and save it on your
SD card or your computer for evaluation in csv-format.
3.2 Survey Instrument
We developed a short questionnaire to collect users’ demographic
data, and previous gaming experience counting 10 items (four items
about demographics and six about the users’ familiarity with games
and technology). A second questionnaire (counting ten items in
total) was designed to measure the user’s emotions (on a five-point
Likert scale) after playing the game. Five items were used to mea-
sure enjoyment and five items were used to measure frustration.
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
Figure 2: Prototype gaming app- interface snippets
The items were selected based on relative literature [39], and their
wordings were carefully examined by two reviewers who are ex-
perts in the field of Technology Enhanced Learning, to eliminate
any typos or inconsistencies. Among the questions, there was also
a concentration question to filter and ultimately exclude those who
answered by chance and gave contradictory answers. After col-
lecting the data, the instrument was validated in terms of internal
consistency, according to the criteria of [40].
As depicted in Table 1, the Cronbach’s Alpha value based on
standardized items for enjoyment, equals to 0.755 which is higher
than 0.7. Therefore, the data of our research about enjoyment are
considered reliable in terms of internal consistency. Similarly, the
variable of frustration received a value of 0.709 or 70.9%, which is
considered reliable since it exceeds the reliability base of 0.7. Finally,
we can conclude that none of our questionnaire items predispose
the user to select or direct in one direction of answers and therefore
the questions are considered inviolable and impartial, not guiding
the user to certain pairs of answers.
3.3 Participants and Procedure
The experimental study was conducted on a set of users who volun-
tarily offered to participate. The volunteers were provided with all
the necessary software (i.e., the prototype gaming app installation,
the sensor log app installation for Android OS and the question-
naire link). Out of 75 users that were invited to participate (through
email –sent and direct invitation), a total of 42 users accepted to
participate and finally 40 users (21 males, 17 females and 2 people
who preferred not to state their gender) successfully completed all
Table 1: Cronbach’s Alpha (CA) values for the measured emotions
Reliability Statistics: CA
Enjoyment
Frustration
Orestis Piskioulis et al.
parts of the survey. Two users discarded from the concentration
question and were excluded from the final sample. At this point,
we must point out that we have obtained data from 34 users from
the Huawei device while only 6 users have used their cell phones
for the game and the sensor recording system.
Regarding the users’ age, there were no significant differences,
since 30%, were 25-34 and 35-44 years old, 20% of the users were in
the age group 45-54, 17.5% in the group of 18-24 and finally 2.5%
of the participants were 55-64 years old. In terms of educational
level, 52.5% of the participants hold a postgraduate degree while
27.5% of the participants gained a bachelor’s degree. The remaining
percentage corresponds to a 15% for holders of a doctoral degree
and 5% for high school graduates. Regarding the time that the
participants spend in gaming activities, 42.5% of them spend from 4
to 10 hours a week, while 7.5 stated that spend more than 10 hours
a week. Also, 30% stated that they spend 1 to 3 hours, per week or
not at all. Finally, 7.5% stated that they spend more than 10 hours
in this activity. Finally, 27.5% of the participants stated that they
prefer sports games, 12.5% strategy games and 7.5% of users spend
time on action, adventure, war, and car games.
In the beginning of the experimental process, the participants
were asked to manually install the required apps (game and sen-
sor log) in their Smartphones, or to use a given Smartphone (see
the device characteristics in Table 2) that was already hosting the
required apps. Then, the users were asked to consent for their par-
ticipation and answer the first part of the online questionnaire and
play the prototype game. During playing the game, the "Sensor
Record" application was running to monitor accelerometer and
gyroscope sensing data. In the end, the participants answered the
second part of the questionnaire to provide a self-reported feedback
on their perceived enjoyment and frustration during playing the
game. Table 2 shows the characteristics of the mobile phone that
was used for the survey.
3.4 Data Overview
A total of 1.367.049 sensor-based records were collected and pro-
cessed for the classification of emotions, as depicted in Table 3.
Regarding the self-reported emotional data, the participants had
to choose one of five levels for each dimension of emotions. While
such a data resolution might be interesting, to increase the classifi-
cation accuracy, the two highest levels and the two lowest levels
were combined resulting to three different classes on each emo-
tional dimension for enjoyment and frustration: low, medium, and
high as depicted in Table 4
3.5 Data (pre)Processing and Feature Selection
The sensor data presents some difficulties when it comes to data
quality:
Cronbach’s Alpha N of items
.755 5
.704 5
Emotion Detection through Smartphone’s Accelerometer and Gyroscope Sensors UMAP ’21, June 21–25, 2021, Utrecht, Netherlands

Table 2: Mobile Phone Characteristics

Mobile Phone Huawei P10

Android 8.0.0
RAM 3G
ROM 64GB
CPU 4x2.1GHz ARM Cortex-A53, 4x1.7GHz ARM Cortex-A53, Cores 8
GPU ARM Mali-T830 MP2, 900MHz, Cores 2
Dimensions 72x146.5x7.2 mm

Table 3: Amount of collected sensor data.

Sensor Amount of data (expressed in excel rows)

Accelerometer 450,886
Accelerometer Linear 455,549
Gyroscope 460,614
Total 1,367,049

Table 4: User-reported data & emotional classes

User-reported data [1, 5] Class

1-2 Low
3 Medium
4-5 High

⃝ The accelerometer (linear or not) and gyroscope are not
perfectly accurate, adding a little noise.
⃝ When a user moves and plays with the phone, this causes
confusion to the sensors and it is difficult to compare the
readings from individual axes.
⃝ If the mobile is not stable, this could make it more possible
to jerk.
⃝ The user will influence the recordings with gait features.
⃝ It cannot determine whether the user moved or not during
the entire recording period.
Firstly, we have cleaned out data by isolating:
⃝ Standard Types which the pandas recognized as out of the
box.
⃝ Non-standard Types, that pandas have not automatically
recognize as missing values.
⃝ Unexpected Types when a column of names contains unex-
pected data types.
⃝ Summarizing
⃝ Replacing, when making simple replacements
To accurately identify the accelerometer and gyroscope features
that were essential to be included in the prediction model, we first
had to make following definitions. Given that we had to export
features from a huge amount of data, we set up a specific computing
database. This was introduced as the calculation of the average of
every twenty observations so that we can proceed to the following
calculations. Practically, these 20 observations are made within a
range of 200 milliseconds. In this way, we have been able to extract
accurate results for a variety of features.
For the success rate of a classifier, the choice of good features is
essential. In this study, the selected features were extracted based on
suggestions of previous research studies [33]. The selected features
and their calculated values are depicted in Table 5. Three different
axes were measured by each sensor. Since in this work, we do not
intend to consider the position of the telephone, it is reasonable
to consider the total movement instead of the movement on the
individual axes.
4 PREDICTION MODELS
4.1 Measured Emotions
In this step we will try to predict two basic human feelings (Enjoy-
ment, Frustration) based on the mobile sensor data from accelerom-
eter, accelerometer linear and gyroscope that we have tracked in
this research. To this effort we have constructed the following de-
tecting emotion recognition models for the two basic emotions that
we have targeted.
Before doing that, in this research we will use the term emotion
mean value. We will define this term as the average value we re-
ceived from the users of our research regarding the emotional state
that they perceived during playing the game. Next, we examine to
what extent it has occur changes in the two basic emotional states
that we try to predict.
To give a simple example, when we receive a mean value of 3.8
for enjoyment, we concluded that our game had a positive impact
on this user because it makes them feel joy/entertainment. In simple
terms, their emotional state about this variable has changed from
neutral, i.e., level of 3 (which we assume all users start in our
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands Orestis Piskioulis et al.

Table 5: Extracted features from accelerometer and gyroscope sensors

Name Enjoyment Frustration

Mean Acceleration 3.77 2.61
Std Acceleration 0.034 0.031
Mean Peak Acceleration 0.12 0.23
Std peak Acceleration 0.0009 0.0012
Mean Jerk 0.002 0.005
Std Jerk 0.0001 0.0003
Mean Peak Jerk 0.07 0.13
Std peak Jerk 0.0004 0.0012
Mean step duration 0.01 0.024
Std step duration 0.0002 0.0005
Skewness -1.18 0.32
Kurtosis 1.46 2.00
Mean PSD 0.02 0.04
Std PSD 0.001 0.003

Figure 3: Self-reported emotion mean values, measured from 1 to 5 (N=40)

research purposes) to 3.8. Finally, this whole process has caused
a positive change to the term of enjoyment which is equal to 0.8.
Conversely, when a user’s frustration reported mean value 2.6, we
assume that there is a decrease from 3 to 2.6. This value reflects
that this user did not feel any frustration after the whole process.
Figure 3 below depicts the self-reported mean values of perceived
enjoyment and frustration for the sample of 40 participants. As
depicted, frustration meets lower values (of low or medium class
according to the labels in Table 4) compared with enjoyment (high
class according to the labels in Table 4), revealing an overall positive
emotional experience of the participants.
4.2 Performance of Prediction Models
The model development was conducted via the KNIME 4.2 Ana-
lytics Platform. We developed a list of machine learning (ML) algo-
rithms as depicted in Table 6. We used 30% of the sample dataset to
train the model and 70% of the dataset to test their prediction perfor-
mance scores. We have selected 70/30 as a training model based on
other researchers that have also used a high volume of data to train
emotion recognition models. Table 6 depicts the obtained results
regarding the prediction success for the two targeted emotional
variables. More specifically, it was observed that by running the
Decision Tree at KNIME, we obtained the optimal prediction rates
for both emotions of enjoyment and frustration. In fact, with this
method we exceeded the average percentage of accuracy for the
detection of emotions using the sensors of the mobile and which
is 77% for this method [41]. Also, we must mention that the use
of decision trees has been mainly used by researchers to predict
negative emotions like stress and positive emotions associated with
enjoyment and happiness.
We realize that the decision tree reached a high accuracy score
of about 90% and is the most accurate model we have set up. It is
important to note that previous models found in the literature were
based on an accelerometer unimodal methodology, a multimodal
approach including several modalities or an approach including
camera feedback. In this research we have used only the accelerom-
eter and gyroscope and maybe this is one reason why some fore-
casting models did not provide us with the expected accuracy that
was met in previous emotion detection models [41].
4.2 Emotion Recognition Model
From the above analysis and presentation of the evaluation percent-
ages of the models, we have selected the Decision Tree Algorithm
for the prediction of both basic emotions. We have chosen this
Emotion Detection through Smartphone’s Accelerometer and Gyroscope Sensors
Table 6: Prediction models and performance scores
ML Algorithm Enjoyment
Linear Regression 65.13%
Decision Trees 87.90%
Logistic Regression 61.20%
Support Vector Machine 72.08%
Neural Network 77.16%
method because it gives us the most accurate and continuous train-
ing of the model that we have managed to achieve satisfactory
accuracy in the data of this research.
This model needs two types of data to work:
→X, Y, Z axis data
→Self-reported emotion values
It was observed that the more data we fed the model with, the
more accuracy it returned. Secondly, this model uses the Decision
Tree Predictor to try to predict the emotion value with the high-
est possibly accuracy. This value is based on the change in the
mental state of the user in relation to the variables of enjoyment
and frustration. We conclude that this model can predict the self-
reported emotion value for enjoyment at 87.90% accuracy and the
self-reported emotion value for frustration at 89.45% accuracy.
This model was created with the ability to be continuously fed
back and try to increase its accuracy on a regular basis if it is
equipped with the suggested sensor-extracted features. Thus, one
realizes that it can be used in similar research works that need
to estimate human emotion to provide with appropriate feedback.
Finally, one realizes that mobile sensing emotion recognition models
need a lot of data to be successful, something that we need to
consider when dealing with the emotional change of users.
5 DISCUSSION AND CONCLUSIONS
Overall, this study managed to apply a multimodal and non-
intrusive methodology to predict user enjoyment and frustration
during the use of a prototype gaming app. The results revealed
that the decision tree algorithm performs on the highest predic-
tion model accuracies for both measured emotions. Our prediction
model accuracy is 87.90% for enjoyment and 89.45% for frustration.
The model was trained and evaluated based on sensor and self-
reported data gathered from 40 users. This is a satisfactory sample
size compared to previous studies, but a larger sample could be
chosen in the future, as the data size is shown to add value to the
model.
One limitation of this work is that we assume that in a user
session the emotion of the user does not generally change. While
we observe that 80% of the sessions are less than 4 minutes in
our dataset, this assumption is probably untrue, but this may not
always be true. The design of advanced experience sampling method
schedules to trace the emotional changes in the session can mitigate
this problem [42].
In addition, people often do not have a pure kind of emotion.
For instance, an individual can be both excited and nervous when
a new process is applied. For a more accurate and comprehensive
emotional inference service it is therefore essential to describe
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
Frustration
57.88%
89.45%
42.12%
74.22%
79.10%
and infer complex emotional statements considering multiple basic
emotions at one time.
Moreover, the performance of the model depends heavily on
training data of individual users. However, because of the wild
nature of the study, a balanced dataset may not be possible. While
some filters and some techniques have been used to overcome the
imbalance problem, this should be carefully considered to obtain
balanced data during such application deployment. Machine learn-
ing algorithms can be used specifically for unbalanced datasets [43],
[44].
It is crucial to say that the deployment of emotion-sensitive ap-
plications on personal mobile devices challenges privacy issues. To
train prediction models, applications must record the sensor-data
and collect information from their users. Because of their limited
computational capability or battery power, mobile devices can dis-
charge the inferring task into cloud or another device. This means
that user data may be revealed. Location, device use, data from the
activity, calls and communications contain all the information to
reveal the identity of a user. For this reason, we have anonymized
the personal data of our users.
Our model contributes significantly towards the non-intrusive
detection of user emotions through Smartphone sensors. The sug-
gested model can be used in several ways. For instance, once we can
detect the emotional state of the, we can change the look and feel at
the interface in our effort to generate to the emotional state. More-
over, emotion-based personalized services or empathetic agents can
be provided. As a fact, empathetic agents have been shown to be
effective in persisting positive emotions and altering an emotional
state of fear to a neutral one [45].
Overall, this study reveals the efficiency of analyzing Smart-
phone sensor data to recognize user emotions of enjoyment and
frustration in a non-intrusive way. Our outcomes can contribute
to breakthroughs in practical applications in several fields, includ-
ing affective user modeling and design of intelligent interfaces for
learning, advertising/marketing, industrial engineering, and more.
REFERENCES
[1] P. Wang, L. Dong, Y. Xu, W. Liu, and N. Jing, “Clustering-Based Emotion Recog-
nition Micro-Service Cloud Framework for Mobile Computing,” IEEE Access, vol.
8, 2020, doi: 10.1109/ACCESS.2020.2979898.
[2] E. Diener, S. Oishi, and R. E. Lucas, “Personality, Culture, and Subjective Well-
being: Emotional and Cognitive Evaluations of Life,” Annual Review of Psychology,
vol. 54. 2003, doi: 10.1146/annurev.psych.54.101601.145056.
[3] J. J. Gross and O. P. John, “Individual Differences in Two Emotion Regulation
Processes: Implications for Affect, Relationships, and Well-Being,” J. Pers. Soc.
Psychol., vol. 85, no. 2, 2003, doi: 10.1037/0022-3514.85.2.348.
[4] A. Bogomolov, B. Lepri, M. Ferron, F. Pianesi, and A. Pentland, “Daily stress
recognition from mobile phone data, weather conditions and individual traits,”
2014, doi: 10.1145/2647868.2654933.
UMAP ’21, June 21–25, 2021, Utrecht, Netherlands
[5] B. Sun, Q. Ma, S. Zhang, K. Liu, and Y. Liu, “ISelf: Towards cold-start emotion
labeling using transfer learning with smartphones,” ACM Trans. Sens. Networks,
vol. 13, no. 4, 2017, doi: 10.1145/3121049.
[6] S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,”
Proc. Natl. Acad. Sci. U. S. A., vol. 111, no. 15, 2014, doi: 10.1073/pnas.1322355111.
[7] R. PLUTCHIK, “A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION,”
in Theories of Emotion, 1980.
[8] X. Bao, S. Fan, A. Varshavsky, K. A. Li, and R. R. Choudhury, “Your reactions
suggest you liked the movie: Automatic content rating via reaction sensing,” 2013,
doi: 10.1145/2493432.2493440.
[9] Y. Zhou, H. Xue, and X. Geng, “Emotion distribution recognition from facial
expressions,” 2015, doi: 10.1145/2733373.2806328.
[10] A. Wahid and F. Rasheed, “Emotion Detection Using Unobtrusive methods: An
Integrated Approach,” 2019, doi: 10.1109/ICICICT46008.2019.8993118.
[11] M. Khamis, F. Alt, and A. Bulling, “The Past, Present, and Future of Gaze-enabled
Handheld Mobile Devices: Survey and lessons learned,” MobileHCI 2018 - Beyond
Mob. Next 20 Years - 20th Int. Conf. Human-Computer Interact. with Mob. Devices
Serv. Conf. Proc., pp. 1–17, 2018, doi: 10.1145/3229434.3229452.
[12] I. Zualkernan, F. Aloul, S. Shapsough, A. Hesham, and Y. El-Khorzaty, “Emotion
recognition using mobile phones,” Comput. Electr. Eng., vol. 60, pp. 1–13, 2017,
doi: 10.1016/j.compeleceng.2017.05.004.
[13] J. Posner, J. A. Russell, and B. S. Peterson, “The circumplex model of affect:
An integrative approach to affective neuroscience, cognitive development, and
psychopathology,” Dev. Psychopathol., vol. 17, no. 3, pp. 715–734, 2005, doi:
10.1017/S0954579405050340.
[14] X. Zhang, W. Li, X. Chen, and S. Lu, “MoodExplorer: Towards Compound Emotion
Detection via Smartphone Sensing,” Proc. ACM Interactive, Mobile, Wearable
Ubiquitous Technol., vol. 1, no. 4, 2018, doi: 10.1145/3161414.
[15] J. R. Crawford and J. D. Henry, “The Positive and Negative Affect Sched-
ule (PANAS): Construct validity, measurement properties and normative data
in a large non-clinical sample,” Br. J. Clin. Psychol., vol. 43, no. 3, 2004, doi:
10.1348/0144665031752934.
[16] P. Ekman et al., “Universals and Cultural Differences in the Judgments of Facial
Expressions of Emotion,” J. Pers. Soc. Psychol., vol. 53, no. 4, pp. 712–717, 1987,
doi: 10.1037/0022-3514.53.4.712.
[17] R. Reisenzein, M. Studtmann, and G. Horstmann, “Coherence between emotion
and facial expression: Evidence from laboratory experiments,” Emotion Review,
vol. 5, no. 1. 2013, doi: 10.1177/1754073912457228.
[18] H. Lövheim, “A new three-dimensional model for emotions and
monoamine neurotransmitters,” Med. Hypotheses, vol. 78, no. 2, 2012, doi:
10.1016/j.mehy.2011.11.016.
[19] R. Pekrun, “The control-value theory of achievement emotions: Assumptions,
corollaries, and implications for educational research and practice,” Educ. Psychol.
Rev., vol. 18, no. 4, pp. 315–341, 2006, doi: 10.1007/s10648-006-9029-9.
[20] A. C. Graesser, S. K. D’Mello, and A. C. Strain, “Emotions in advanced learning
technologies.,” in International handbook of emotions in education., New York, NY,
US: Routledge/Taylor & Francis Group, 2014, pp. 473–493.
[21] S. D’Mello and A. Graesser, “Dynamics of affective states during com-
plex learning,” Learn. Instr., vol. 22, no. 2, pp. 145–157, Apr. 2012, doi:
10.1016/j.learninstruc.2011.10.001.
[22] W. Goertz, “Hype Cycle for Human-Machine Interface, 2018,” Gartner, 2018. .
[23] B. Cao et al., “DeepMood: Modeling mobile phone typing dynamics for mood
detection,” Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., vol. Part F1296,
no. October, pp. 747–755, 2017, doi: 10.1145/3097983.3098086.
[24] S. Servia-Rodríguez, K. K. Rachuri, C. Mascolo, P. J. Rentfrow, N. Lathia, and G.
M. Sandstrom, “Mobile Sensing at the Service of Mental Well-being,” 2017, doi:
10.1145/3038912.3052618.
Orestis Piskioulis et al.
[25] Y. Suhara, Y. Xu, and A. S. Pentland, “DeepMood: Forecasting depressed mood
based on self-reported histories via recurrent neural networks,” 2017, doi:
10.1145/3038912.3052676.
[26] T. Stütz et al., “Smartphone based stress prediction,” in Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture
Notes in Bioinformatics), 2015, vol. 9146, doi: 10.1007/978-3-319-20267-9_20.
[27] M. C. and K. Wac, “7515181 @ ieeexplore.ieee.org.” pp. 51–65, 2018, [Online].
Available: https://ieeexplore.ieee.org/document/7515181.
[28] S. Ghosh, S. Sahu, N. Ganguly, B. Mitra, and P. De, “EmoKey: An Emotion-aware
Smartphone Keyboard for Mental Health Monitoring,” 2019, doi: 10.1109/COM-
SNETS.2019.8711078.
[29] A. Mottelson and K. Hornbæk, “An affect detection technique using mobile
commodity sensors in the wild,” 2016, doi: 10.1145/2971648.2971654.
[30] M. L. Barron-Estrada, R. Zatarain-Cabada, and C. G. Aispuro-Gallegos, “Multi-
modal recognition of emotions with application to mobile learning,” 2018, doi:
10.1109/ICALT.2018.00104.
[31] M. Gjoreski, H. Gjoreski, M. Lutrek, and M. Gams, “Automatic Detection
of Perceived Stress in Campus Students Using Smartphones,” 2015, doi:
10.1109/IE.2015.27.
[32] Maxhuni, Lecture Notes of the Institute for Computer Sciences, Social Informatics
and Telecommunications Engineering. 2017.
[33] A. F. Olsen, “Detecting Human Emotions Using Smartphone Accelerometer Data,”
Univ. Oslo Dep. Informatics, 2016.
[34] K. C. Herdem, “Reactions: Twitter based mobile application for awareness of
friends’ emotions,” 2012.
[35] L. Canzian and M. Musolesi, “Trajectories of depression: Unobtrusive monitoring
of depressive states by means of smartphone mobility traces analysis,” 2015, doi:
10.1145/2750858.2805845.
[36] Y. Gao, N. Bianchi-Berthouze, and H. Meng, “What does touch tell us about
emotions in touchscreen-based gameplay,” ACM Trans. Comput. Interact., vol. 19,
no. 4, 2012, doi: 10.1145/2395131.2395138.
[37] S. Shah, J. N. Teja, and S. Bhattacharya, “Towards affective touch interaction:
predicting mobile user emotion from finger strokes,” J. Interact. Sci., vol. 3, no. 1,
2015, doi: 10.1186/s40166-015-0013-z.
[38] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, “Human emotion recognition:
Review of sensors and methods,” Sensors (Switzerland), vol. 20, no. 3, pp. 1–41,
2020, doi: 10.3390/s20030592.
[39] T. Ding, Yan; Zhao, “Emotions, Engagement, and Self-Perceived Achievement
in a Small Private Online Course,” J. Comput. Assist. Learn., vol. v36, no. n4, pp.
449–457.
[40] D. Gefen, “E-commerce: The role of familiarity and trust,” Omega, vol. 28, no. 6,
pp. 725–737, Dec. 2000, doi: 10.1016/S0305-0483(00)00021-9.
[41] K. Tzafilkou, N. Protogeros, and A. Chouliara, “Experiential learning in web devel-
opment courses: Examining students’ performance, perception and acceptance,”
Educ. Inf. Technol., vol. 25, no. 6, 2020, doi: 10.1007/s10639-020-10211-6.
[42] H. Vachon, W. Viechtbauer, A. Rintala, and I. Myin-Germeys, “Compliance and
retention with the experience sampling method over the continuum of severe
mental disorders: Meta-analysis and recommendations,” J. Med. Internet Res., vol.
21, no. 12, 2019, doi: 10.2196/14475.
[43] C. Chen, A. Liaw, and L. Breiman, “Using Random Forest to Learn Imbalanced
Data | Department of Statistics,” 2004.
[44] Y. Tang, Y. Q. Zhang, and N. V. Chawla, “SVMs modeling for highly imbalanced
classification,” IEEE Trans. Syst. Man, Cybern. Part B Cybern., vol. 39, no. 1, 2009,
doi: 10.1109/TSMCB.2008.2002909.
[45] C. N. Moridis and A. A. Economides, “Affective learning: Empathetic agents with
emotional facial and tone of voice expressions,” IEEE Trans. Affect. Comput., vol.
3, no. 3, pp. 260–272, 2012, doi: 10.1109/T-AFFC.2012.6.