Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109

Contents lists available at ScienceDirect

Journal of King Saud University –

Computer and Information Sciences
journal homepage: www.sciencedirect.com

A new emotion–based affective model to detect student’s engagement

Khawlah Altuwairqi a,⇑, Salma Kammoun Jarraya a,c, Arwa Allinjawi a, Mohamed Hammami b,c
a
Department of Computer Science, King Abdul-Aziz University, Jeddah, Saudi Arabia
b
Department of Computer Science, Faculty of Science, Sfax University, Sfax, Tunisia
c
MIRACL-Laboratory, Sfax, Tunisia

a r t i c l e i n f o a b s t r a c t

Article history: Detecting student’s engagement is an important key to improve an e-learning system. An e-learning sys-
Received 9 September 2018 tem adapted to learner emotions is considered as an innovative system. Among the challenges that face
Revised 15 November 2018 researcher is how to measure student’s engagement depending on their emotions. During the few years,
Accepted 27 December 2018
several solutions were proposed to measure student’s engagement, but few solutions detect engagement
Available online 29 December 2018
level without consider if the student is learning or not. In this paper, we reviewed the current works of
emotions and engagement level of student. According to that, we built our engagement level and linked
Keywords:
them with the appropriate emotions. Then, we propose an affective model and a new process to detect
Face expressions
Emotions
final engagement level. The efficiency of the proposed Affective Model is shown experimentally by con-
Engagement levels ducting a series of experiments. Firstly, we compute the Matching Score (MS) and Miss-matching Score
Academic emotions (MisMS) for each engagement level. Secondly, we apply the new engagement level detection process on
Affective model severe cases. Thirdly, we analyze all emotions in each level of engagement to detect strong emotions. We
record matching score (MS) in range [71.2%, 100%]. Finally, we proposed some suggestions to improve the
affective model.
Ó 2018 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an
open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction Some researchers highlight the importance of monitoring stu-

The detection of student’s engagement is an important factor to
improve the learning process. Engagement is one of the qualitative
indicators in learning process (D’Errico et al., 2016). It has three-
dimensional structure during learning (Fredricks et al., 2004).
There is behavioral engagement as stay on task, emotional engage-
ment as being bored, and cognitive engagement as focused atten-
tion (Whitehill et al., 2014). All these dimensions are important
to measure engagement level. Many of researchers (Monkaresi
et al., 2017; Whitehill et al., 2014; Jang et al., 2014; Li et al.,
2016; Schlechty, 2011; D’Mello and Graesser, 2012) have proposed
levels to classify students depend on engagement state. They
started by measuring students’ engagement to detect their levels.
⇑ Corresponding author at: Department of Computer Science, King Abdul-Aziz
University, 21589 Jeddah, Saudi Arabia.
E-mail addresses: Kaltowairqi0003@stu.kau.edu.sa (K. Altuwairqi), smoha-
mad1@kau.edu.sa (S.K. Jarraya), aallinjawi@kau.edu.sa (A. Allinjawi), Mohamed.
Hammami@fss.rnu.tn (M. Hammami).
Peer review under responsibility of King Saud University.
dent’s emotions and how can detect if they engaged (D’Errico
et al., 2016). Typically, emotions are classified into basic emotions
and multi-dimensional emotions. There are many of models of
emotions, the popular models are mentioned as following. The
basic emotions know as Paul Ekman’s emotions (Griffiths, 2003).
The multi-dimensional models are Russell Model, PANA Model,
etc. In the next section, we reviewed and discussed many works
that are related to our work.
2. Related work
In this section, we reviewed basic emotions, multi-dimensional
emotions, and academic emotions. This review helps us to under-
stand which most emotions that indicate if students engaged or
not. Also, we reviewed engagement levels of student in several
studies. Finally, we linked between academic emotions and
engagement levels to detect level of student’s engagement depend-
ing on their emotions.

2.1. Emotions and academic emotions

Emotions are a major area of affective computing. It can define

Production and hosting by Elsevier
as multifaceted internal states. It is encompassing feelings and

https://doi.org/10.1016/j.jksuci.2018.12.008
1319-1578/Ó 2018 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
100 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109
cognitive, physiological, expressive, and motivational aspects.
Some researchers classified the emotions based on their require-
ments on the research (Bakhtiyari et al., 2015). Typically, emotions
are classified into basic emotions and multi-dimensional emotions.
There are many of models of emotions, the popular models are
mentioned as following. The basic emotions know as Paul Ekman’s
emotions. There are anger, disgust, fear, happiness, sadness and
surprise emotion (Griffiths, 2003).
The circumplex model is the popular model in multi-
dimensional emotion. It has two dimensions: valence/pleasure
and activation/arousal (Russell and Lemay, 2000). The vector
model is a different dimensional emotion. The arousal dimension
determines its vector, while valence determines its direction
(Finch et al., 2015). The PANA model is another well-respected
model created by Watson and Tellegen. PANA stands for ‘‘Positive
Affect-Negative Affect”. The PANA model has two major-
dimensional emotions and two minor-dimensional emotions. The
high positive affect/low positive affect and high negative affect/
low negative affect defined as a major-dimensional. The strong
engagement/disengagement and pleasantness/unpleasantness
defined as a minor-dimensional as showed in Fig. 1.a (Watson
and Tellegen, 1985).
Yik et al. (1999) have adapted Watson and Tellegen model with
45° rotational hypothesis. So, major dimensions emotions are
strong engagement/ disengagement and pleasantness / unpleas-
antness as showed in Fig. 1.b.
Some of previous emotions related to learning context. These
emotions are belonging to term of academic emotions. Emotion
research in education during the last century mainly focused on
test anxiety or motivation-based traits. Many researchers in the
last decade have investigate about more set of academic emotions.
In 2002, Pekrun et al. have defined hope, pride, anger, anxiety,
shame, hopelessness, and boredom as academic emotions of stu-
dent (Pekrun et al., 2002). In 2006, Pekrun et al. have defined clas-
sified academic emotions depending on the control-value theory of
achievement emotions. The academic emotions classified in four
categories that included positive activating, positive deactivating,
negative activating, and negative deactivating. The positive activat-
ing included enjoyment, hope, pride emotions. The positive deacti-
vating included relief, relaxation emotions. The negative activating
included anger, anxiety, shame emotions. The negative deactivat-
ing included boredom, hopelessness emotions (Pekrun, 2006). In
2009, Bernardo, Ouano, & Salanga have defined the academic emo-
(a)
HIGH
POSITIVE AFFECT
active
content elated
happy enthusiastic
excited
kindly
NEGATIVE AFFECT
pleased peppy
satisfied strong
warmhearted
LOW
at rest
calm
placid
relaxed
quiescent
still drowsy
quiet dull
sleepy
sluggish
LOW
POSITIVE AFFECT
Fig. 1. (a) Positive affect and negative affect model (Watson and Tellegen, 1985), (b) The 45° – rotation model (Yik et al., 1999).
tions insight from Filipino bilingual students. The Table 1 included
all emotions that classified as academic emotions (Bernardo et al.,
2009).
In 2012, D’Mello and Graesser have proposed dynamics of affec-
tive states during complex learning. The affective states included
boredom, engagement/flow, confusion, frustration, anxiety, curios-
ity, delight, and surprise (D’Mello and Graesser, 2012). In 2015,
Finch et al. have managing emotions by case study that exploring
the relationship between experiential learning, emotions, and stu-
dent performance. Their academic emotions classified in three cat-
egories that included positive, negative, and activity. The positive
emotions included joy, hope, pride, gratitude. The negative emo-
tions included relief, anxiety, hopelessness, sadness, shame, anger.
The activity emotions included enjoyment, anger, frustrations,
boredom (Finch et al., 2015). In the next section, we reviewed
engagement levels of student in several studies.
2.2. Engagement level
Engagement defined as basic protocol. The people use it to man-
age their perceived connection with each other while in interaction
(Jang et al., 2014). According to our review, many researchers have
built the levels of student engagement as shown in Fig. 2.
Whitehill et al. (2014) have four levels. The first level is not
engaged at all (e.g., looking away from a computer and obviously
not thinking about the task, eyes completely closed). The second
level is nominally engaged (e.g., eyes barely open, clearly not ‘‘into”
the task). The third level is engaged in the task (e.g. student
requires no admonition to ‘‘stay on task”). The fourth level is very
engaged (e.g. student could be ‘‘commended” for his/her level of
engagement in the ‘‘Set” game task).
Monkaresi et al. (2017) have two levels. The first level is not
engaged student (e.g. when a student is distracted by noise or stu-
dent not engaged in the writing task). The second level is an
engaged student (e.g. when a student is reviewing his writing or
student is typing). Jang et al. (2014) have two levels engaged or
disengaged. They classified engagement level depending on social
signals, duration of math quiz and the answer correctness. The
social signals used are smile, laughter and neutral.
Li et al. (2016) have three levels high, medium and low atten-
tion. They detected the attention level of a user when reading an
article. When a student’s face express anger, it indicates to a high
level of attention. The behavior of eye gaze (e.g. eye fixation) is
(b)
Engagement
aroused HIGH
astonished
HIGH
NEGATIVE AFFECT POSITIVE AFFECT
NEGATIVE AFFECT
surprise
distressed
fearful
HIGH
hostile
jittery
nervous Unpleasantness Pleasantness
blue scornful
grouchy
lonely
sad
sorry Low
Low
unhappy
Positive Affect Negative Affect
Disengagement
 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109 101

Table 1
The academic emotions insight from Filipino bilingual students (Bernardo et al., 2009).

Happy Bore Enjoy Lazy Sad Excite Proud Tire Angry Confident
Fulfill Frustrate Satisfy Good Stress Challenge Fear Eager Content Interest
Pressure Confuse Curious Determined Love Anxiety Depress Hurt Inspire Persevere
Disappoint Hope Nervous Patience Willing Enthusiasm Hard Work Hate

Rebellion Engagement
Dis- Productive
Not Very engagement Confusion
Low High Engaged Engaged
Attention Attention
not Strategic
Engaged Retreatism
Engaged Nominally Engaged in Hopeless Dis- Compliance
Engaged Task Confusion equilibrium
Medium
Attention
Ritual
Compliance

(Monkaresi et al., 2017; (Li et al., 2016) (Whitehill et al., 2014) (D’Mello and Graesser, 2011) (Schlechty, 2011)
Jang et al., 2014)

Fig. 2. Summary of current engagement levels.

very indicative of the human attention level. It is notable that a
reader tends to have long fixations while paying high attention
to reading. In contrast, short fixations happen when the reader’s
attention level is low.
Schlechty (2011) have five levels. The first level is engagement.
In this level, the student will persist in the face of difficulty and will
learn at high profound levels. The second level is strategic compli-
ance. In this level, the student will abandon work if extrinsic goals
are not realized and will not retain what is learned. The third level
is ritual compliance. In this level, the student is willing to expend
whatever effort is needed to avoid negative consequences. The
fourth level is retreatism. In this level, the student is disengaged
from the tasks but does not try to disrupt the work or substitute
other activities for it. The fifth level is rebellion. In this level, the
student refuses to do the work, acts in ways to disrupt others, or
substitutes tasks and activities to which he or she is committed.
D’Mello and Graesser (2012) have 4 hypotheses/levels. The first
hypothesis is Productive Confusion. The student in this hypothesis
resolves the impasse and goal attained. Productive Confusion
hypothesis predicts appearance of transitions from confusion to
engagement/flow. The second hypothesis is Disequilibrium. The
student in this hypothesis detects the impasse. Disequilibrium
hypothesis predicts appearance of transitions from engagement/
flow to confusion. The third hypothesis is Hopeless Confusion.
The student in this hypothesis fails to achieve the goal. Hopeless
Confusion hypothesis predicts appearance of transitions confusion
to frustration. The fourth hypothesis is Disengagement. The stu-
dent in this hypothesis persists failure. Disengagement hypothesis
predicts appearance of transition from frustration to boredom.
Depending on these hypotheses, they have 6 levels. The levels
are novelty, equilibrium, achievement, disequilibrium, stuck, dis-
engagement. The student in novelty be surprised, and in equilib-
rium be engaged. Student in achievement have positive emotions
like content or delighted, and in disequilibrium be confused. Stu-
dent in stuck be frustrated, and in disengagement be hopelessness
and boredom.
In the next section, we built new levels of student engagement.
That done by selecting the most relevant levels to face expressions.
Also, we discussed how mapped engagement levels of student with
academic emotions. In addition, we proposed affective model.
2.3. Linking academic emotions with engagement levels
Depending on our reviewed of engagement levels and academic
emotions. We reclassified these levels from different studies
(Whitehill et al., 2014; Jang et al., 2014; Li et al., 2016; Schlechty,
2011; Watson and Tellegen, 1985; D’Mello and Graesser, 2012;
Yik et al., 1999) depending on similarities on each level.
As shown in Table 2. five levels were appeared after reclassified.
In the first level, student has high attention and high commitment
(Schlechty, 2011). In addition, student be surprised when s/he
detected something novelty (Watson and Tellegen, 1985; D’Mello
and Graesser, 2012; Yik et al., 1999). So, student has strong engage-
ment in this level (Watson and Tellegen, 1985; Yik et al., 1999). In
the second level, student has high attention and low commitment
(Schlechty, 2011). When the student feels angry, his/her attention
is high and has long eye fixations (Li et al., 2016). The long fixations
mean student feels enthusiastic and excited. Also, the student is in
a state of equilibrium in this level (D’Mello and Graesser, 2012).
That meaning s/he does not have any problems or difficulties in
accomplishing the tasks. So, we can say student has high engage-
ment in this level.
In the third level, student has medium to low attentions, med-
ium eye fixations, or low commitment (Li et al., 2016; Schlechty,
2011). Student be content, happy, joyful or delight (Jang et al.,
2014; Li et al., 2016). Medium eye fixations indicate sometime to
student felt sad or content (Li et al., 2016). So, we can say student
has medium engagement in this level. In the fourth level, student
has low to no attention, or be nominally engaged (Whitehill
et al., 2014; Schlechty, 2011). When his eyes barely open, or has
short eye fixations, that’s mean he feels tire, bored or relaxed
(Whitehill et al., 2014; Li et al., 2016). Also, he feels tire and bored
when he be stuck (D’Mello and Graesser, 2012). So, we can say stu-
dent has low engagement in this level. In the fifth level, student has
diverted attention and no commitment (Schlechty, 2011). Student
be not engaged when, his eyes completely closed (Whitehill et al.,
2014), he feels hopelessness boredom (D’Mello and Graesser,
2012), or feels still, quiet (Watson and Tellegen, 1985; Yik et al.,
1999). So, student has disengagement in this level (Whitehill
et al., 2014; Watson and Tellegen, 1985; D’Mello and Graesser,
2012; Yik et al., 1999).
102 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109

Table 2
Engagement levels from different studies.

Ref First Level Second Level Third Level Fourth Level Fifth Level
Whitehill et al., – Very Engaged student could Engaged in the Task student Nominally Engaged eyes Not Engaged eyes
2014 be ‘‘commended” for his/her requires no admonition to ‘‘stay barely open (tire, completely closed
level on task” relaxed) (sleep)
Jang et al., 2014 – – Engaged Smile, Laughter (happy, – Disengaged no
joyful, delight) commitment
Li et al., 2016 – High Attention (anger), long Medium Attention have medium Low Attention has short –
fixations (Enthusiastic, fixations (content, sad) fixations (bored)
Excited)
Schlechty, 2011 Engagement high Strategic compliance high Ritual compliance low attention Retreatism no attention Rebellion diverted
attention and high attention and low and low commitment and no commitment attention and no
commitment. commitment. commitment
D’Mello and Novelty (Surprise) Equilibrium Engaged – Stuck Frustration (tire, Disengagement
Graesser, 2012 bored) Hopelessness
Boredom
Watson and Strong Engagement – – – Disengagement Still,
Tellegen, 1985; (Surprise) Quiet
Yik et al., 1999
Our Levels Strong Engagement High Engagement Medium Engagement Low Engagement Disengagement

3. Proposed affective model
Depending on previous discussion, we built new levels of stu-
dent engagement. These levels are: Strong Engagement, High
Engagement, Medium Engagement, Low Engagement, and Disen-
gagement. Each level should contain specific emotions of face
expressions. As we mentioned earlier in academic emotions sec-
tion. Watson and Tellegen (1985) model have two engagement
levels mapped them with specific emotions as shown in Table 2.
Thereafter, Russel and Feldman have adapted Watson and Tellegen
model with 45° rotational hypothesis. So, we adapted Russel and
Feldman model that shown in Fig. 1.b by the following: (1) We
kept all academic emotions and removed other emotions. (2) We
added new academic emotions that have estimated their location
(ranging from 0° to 360°) by Remington et al. (2000). Fig. 3 shown
the model after the step 1 and 2 done.
(3) We mapped our engagement levels on this model. The
strong engagement level and disengagement have remained in
the same location that were in Watson and Tellegen model. The
high engagement level contained angry, enthusiastic, and excited
emotions. This level has high positive affect and some high nega-
tive affect. Thus, student when s/he has high positive affect or high
negative affect may be in high engagement level. So, we mapped
high engagement level with emotions in high positive affect and
high negative affect. The medium engagement level contained
content, happy, joyful, delight, and sad emotions. This level has
pleasantness emotions and unpleasantness emotion. Thus, student
when s/he has pleasantness or unpleasantness emotions may be in
medium engagement level. So, we mapped medium engagement
level with emotions in pleasantness and unpleasantness. The low
engagement level contained tire, bored, and relaxed emotions.
This level has low positive affect and some low negative affect.
Thus, student when s/he has low positive affect or low negative
affect may be in low engagement level. So, we mapped low engage-
ment level with emotions in low positive affect and low negative
affect. The Proposed Affective Model shown in Fig. 4.
In our model, we take into consideration what the engagement
levels of student during learning process. However, we didn’t take
into consideration whether the student learned or not. Therefore,
we can recognize the student’s engagement depending on stu-
dent’s emotion in relation to e-learning. In the next section, we
conducted an experiment to verify the proposed Affective Model.
4. Method
4.1. Participants
In this study, we conducted a series of experiments to validate
the proposed Affective Model. We recorded 100 videos from 50
volunteers (two videos for each volunteer). In session 1, 50 videos
were recorded, and another 50 videos recorded in session 2. The
videos recorded by MacBook Pro camera. In addition, the recorded
video takes between 7 and 12 mins in each session. The volunteers
were 5 students (males) in Electrical Engineering Fields, and 45
students (females) in Computer Science Field. Their ages are rang-
ing between 19 and 22.
4.2. Procedure

Strong Engagement Each volunteer selected his/ her interesting task to do in session
90 1, and boring task to do in session 2 as shown in Table 3. After each
High Negative Affect High Positive Affect session, we write our observations about face expressions of volun-
135 45
Angry
Surprise
teers. In addition, we asked the volunteers two questions. The first
Ashamed Enthusiastic
Excited
question: what emotions they have in each expression that
Fearful
Nervous appeared on their face. The second question: what the engagement
Happy
Disgusted Content level they have depending on the five proposed levels (Strong
Unpleasantness 180 Disappointed Delighted 0 Pleasantness
Sad Joyful engagement, High Engagement, Medium Engagement, Low
Satisfied
Bored Engagement, and Disengagement). The number of video frames
Depressed Calm
Tired Relaxed of all volunteers in each session shown in Fig. 5. In Table 4, the total
Still
Quiet number of video frames that depended on 100 videos were shown.
225 315
Low Positive Affect Low Negative Affect

270
4.3. Analysis
Disengagement
We conducted three experiments in this study. In first experi-
Fig. 3. The Russel and Feldman model after adaptation. ment, we compute the Matching Score (MS) and Miss-Matching
 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109 103

Strong Engagement

90 Strong Engagement
High Negative Affect High Positive Affect
135 Surprise 45

High Engagement
Angry
Ashamed Enthusiastic
Fearful Excited
Nervous
Happy
Disgusted Content Pleasantness
Unpleasantness 180 Disappointed Delighted
0
Medium Engagement
Sad Joyful
Satisfied
Bored Calm
Depressed Relaxed
Tired
Low Engagement
Still
Quiet 315
225
Low Positive Affect Low Negative Affect
270 Disengagement
Disengagement

Fig. 4. The proposed affective model.

Score (MisMS) for each engagement level. The MS and MisMS score
Table 3
The task of each session for all volunteers. depended on our proposed affective model. Three cases have been
defined to detect matching. (1) Matching can be existing if emotion
Volunteer ID Session 1 Session 2
found in the proposed affective model and compatible with the
1 Playing League of Legends Shopping Online engagement level. (2) Matching cannot be existing if emotion is not
Game
compatible with the engagement level. (3) Matching can be as a
2 Playing Crash Game Shopping Online
3, 8, 9, 13, 15, 16, 17, Playing Brain stimulation Watching Video Clip new emotion if volunteer felt about new emotion not found in the pro-
18, 20, 22, 23, 24, games and IQ test (Shoalah about Scientific posed affective model. Furthermore, we compute Matching Score
27 Game) Miracle (MS) of emotions that did get the match status. Also, we compute
4, 10, 14 Playing Dancing Line Game Watching Video Clip the Miss-Matching Score (MisMS) of emotions that didn’t get the
about Football Match
5 Playing Dancing Line Game Watching Video Clip
match status (emotions belong to other engagement level MisMS1
about Scientific or new emotion not covered in our model MisMS2). So, MS can be
Miracle denoted by Eq. (1), MisMS1 can be denoted by Eq. (2), and MisMS2
6 Playing Number Link Game Watching Video Clip can be denoted by Eq. (3):
about Scientific
PNeðOÞ
Miracle ek
ELðiÞ
7, 12, 19, 28, 30 Playing Shoalah Game Watching Video Clip MSELðiÞ ¼ k¼1
 100 ð1Þ
about Football Match NeðOÞ
11 Playing Peak Game Watching Video Clip
about Makeup PNeðOÞ
1 ek R ELðiÞ
Tutorial MisMSELðiÞ ¼ k¼1
 100 ð2Þ
21 Watching Video Clip about Watching Video Clip NeðOÞ
Football Match about Cooking
25 Watching Video Clip about Playing Shoalah PNeðOÞ
Scientific Miracle Game 2 k¼1 = EL5j¼1
ek 9
26 Playing Peak Game Watching Video Clip
MisMSELðiÞ ¼  100 ð3Þ
NeðOÞ
about Football Match
29 Playing Dancing Line Game Watching Video Clip We sum the number of emotions (ek ½k ¼ 1 to N eðOÞ ) that got the
match status (ek
ELði)). Furthermore, may got miss status (not
+ Shoalah Game about Scientific
Miracle
32, 33, 35, 36, 37, Flap Flap Game + Touch Fly Watching Video Clip belong to other engagement level) (ek R ELði) or new status
38, 41, 45, 46, 49 Free Game + Impossible about Scientific = EL5j¼1 ). In second experiment, we propose a new process to
(ek 9
Twisty Dots Miracle
detect final engagement level in both sessions. The final engage-
31, 34, 39, 40, 42, Flap Flap Game + Touch Fly Watching Video Clip
43, 44, 47, 48, 50 Free Game + Impossible about Football Match ment level for each volunteer in both sessions is detected based
Twisty Dots on video-based analysis. The final engagement level can be com-
puted based on Eqs. (4) and (5):

800
Number of Frames

600

400

200

0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Volunteers ID
Session 1 Session 2

Fig. 5. The number of video frames of all volunteers in each session.

104 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109

Table 4 observed emotions or behaviors in this session. The engagement

The total number of video’s frames in each session. level that have greatest value in this session becomes the final
Session Number of all videos Number of all video’s frames engagement level based on Eq. (5).
1 50 29,788 In all equations, the number of observed emotions denoted by
2 50 22,091 N eðOÞ , and ELðiÞ[i = 1–5] is the engagement level. These levels sorted
Total 100 51,879 as 1: Strong Engagement, 2: High Engagement, 3: Medium Engage-
ment, 4: Low Engagement and 5: Disengagement. In Eq. (4), Imek
ELðiÞ
denoted by impact value. In Eq. (5), ELId;sessionx can be denoted by
1 the final engagement level. In addition, N oeðELðiÞÞ is the number of
Imek
ELðiÞ ¼ PN ð4Þ
eðELðiÞÞ
k¼1 ek
ELðiÞ observed emotions or behaviors belong to ELðiÞ, and N eðsessionxÞ is
the total number of the observed emotions or behaviors in this
! ! session.
X
NoeðELðiÞÞ
NoeðELðiÞÞ
ELId;sessionx ¼ max Imoek
ELðiÞ  ð5Þ In third experiment, we analysis all emotions/behavior in each
ImELðiÞ
k¼1
NeðsessionxÞ level of engagement. The purpose is to detect strong emotion/
behavior in each engagement level. The emotion/ behavior that
In each session the volunteer has different engagement levels.
felt/done by the largest number of volunteers indicate that it is
These levels rely on different emotions of face expression. First,
stronger than others. All results and discussion of these experiment
we compute an impact value for emotions of face expression
explain in next.
belong to an engagement level based on Eq. (4). We assigned value
1 for each engagement level. Then, this value is divided on the
number of emotions in each level. For example, in high engage- 5. Results and discussion
ment level, there are 6 emotions, so the impact value is 0.166 for
each emotion. After that, all observed emotions or behaviors 5.1. First experiment
impact value that have the same level were summed and then
multiplied by, the number of observed emotions or behaviors In this experiment, we compute the MS and MisMS for each
belong to engagement level divided with the total number of the engagement level depended on our proposed affective model. Each
engagement level in Figs. 6.1–6.5 describe if there a matching
between emotion and engagement level. The data of strong
Session 1 engagement level show in Fig. 6.1. There are 28 volunteers felt Sur-
"Matching Emotions" prise in the first session. They had a strong engagement when they
had surprise emotion. So, there is a matching between surprise and
Surprised 28
strong engagement. Depending on these volunteers, the MS =
0 5 10 15 20 25 30 35 40 45 50 (28/28)  100 = 100%. So, MisMS1 = MisMS2 = 0%. All surprise emo-
tion appeared in first session.
Fig. 6.1. The number of surprised-emotion from analysis of the recorded video of all The data of high engagement level show in Fig. 6.2. There are 39
volunteers in strong engagement level.
volunteers felt Enthusiastic, 24 volunteers felt Nervous, 27

Session 1 Session 2
"Matching Emotions" "Matching Emotions"
Ashamed 6
Fearful 20
Angry 31
Excited 27 Angry 1
Nervous 40 Nervous 2
Enthusiastic 38 Enthusiastic 1

0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50

Fig. 6.2. The numbers of each emotion from analysis of the recorded video of all volunteers in high engagement level.

Session 1 Session 2
"Matching Emotions" "Matching Emotions"
Content 1
Delighted 10
Disgusted 2
Happy 23
Sad 12
Joyful 18
Satisfied 16 Disgusted 9
Disappointed 36 Disappointed 3
0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50

Fig. 6.3. The numbers of each emotion from analysis of the recorded video of all volunteers in medium engagement level.
 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109 105

Session 1 Session 2
"Matching Emotion" "Matching Emotions"
Calm 21
Tired 15
Depressed 8
Bored 2
Bored 43

0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50
"Not Matching Emotion" "New Emotion"
Calm 13 Sleepy 23

0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50

Fig. 6.4. The numbers of each emotion from analysis of the recorded video of all volunteers in low engagement level.

Session 2
"New Behaviors"
Talking with other 4
Playing with her hair 3
Seeing the remaining time of the video 3
Looking at her phone 2
Played with the chair 7
Stopped the video 3
Not looking at the computer 30

0 5 10 15 20 25 30 35 40 45 50

Fig. 6.5. The numbers of each behavior from analysis of the recorded video of all volunteers in disengagement level.

volunteers felt Excited, 32 volunteers felt Angry, 20 volunteers felt
Fearful, and 6 volunteers felt Ashamed. Each volunteer may have
one or more from previous emotions. All these emotions appeared
in first session only, except Enthusiastic, Nervous, Angry emotion
appeared 1, 2, 1 respectively also in second session. Volunteers
had a high engagement when they had one of these emotions.
So, there are a matching between all these emotions and high
engagement level. Depending on these volunteers, the MS =
(166/166)  100 = 100%. So, MisMS1 = MisMS2 = 0%.
The data of medium engagement level show in Fig. 6.3. There
are 39 volunteers felt Disappointed, 16 volunteers felt Satisfied,
18 volunteers felt Joyful, 12 volunteers felt Sad, 23 volunteers felt
Happy, 11 volunteers felt Disgusted, 10 volunteers felt Delighted,
and 1 volunteer felt Content. Each volunteer may have one or more
from previous emotions. Most of these emotions appeared in first
session only, except Disappointed and Disgusted emotions
appeared also in second session. Volunteers had a medium engage-
ment when they had one of these emotions. So, there are a match-
ing between all these emotions and medium engagement level.
Depending on these volunteers, the MS = (130/130)  100 = 100%.
So, there are no miss-matching status, MisMS1 = MisMS2 = 0%.
The data of low engagement level show in Fig. 6.4. There are 45
volunteers felt Bored, 34 volunteers felt Calm, 23 volunteers felt
Sleepy, 8 volunteers felt Depressed, 15 volunteers felt Tired, and
no volunteer felt Relaxed. Each volunteer may have one or more
from previous emotions except Relaxed. Most of these emotions
appeared in second session only, except Bored and Calm emotions
appeared also in first session. Volunteers had a low engagement
when they had one of these emotions expect some cases of calm
emotion. Depending on these volunteers, there are 125 emotions
observed from recorded video. The number of emotions were
matched with our proposed affective model is 112 emotions. So,
the MS = (89/125)  100 = 71.2%. We detected 13 cases of clam
emotion belong to other engagement level. Also, 23 cases of sleepy
emotion not covered in our model, it’s a new emotion. So, there are
miss-matching status in this level. We calculated miss-matching
score, MisMS1 = (13/125)  100 = 10.4% and MisMS2 =
(23/125)  100 = 18.4%.
The data of disengagement level show in Fig. 6.5. There are no
emotions detected in this level. No volunteer felt Still or Quiet.
But there are several behaviors detected in this level. The number
of volunteers not looking at the computer during a task were 30.
Three volunteers stopped the video. Seven of volunteers were play-
ing with the chair. Two of volunteers were looking at their phone.
Three volunteers were checking the remaining time of the video.
Three volunteers were playing with their hair. Four volunteers
were talking with other. Each volunteer may have one or more
from previous behavior. All these behaviors appeared in second
session only. Volunteers had a disengagement when they had
one or more of these behaviors. There are no emotions/behaviors
matching in this level. So, the MS is 0% depending on these volun-
teers. There are no cases of emotions/behaviors belong to other
engagement level. But we detected 52 cases of behaviors not cov-
ered in our model (new emotions). So, the MisMS1 = 0% and
MisMS2 = (52/52)  100 = 100%. As shown in Fig. 7, all emotions
of each volunteer assigned on affective model by their id, and
Table 5 shown MS and MisMS for each engagement level.
5.2. Second experiment
In second experiment, we detect final engagement level in both
session by proposed a new process. This new process depended
on an impact value for emotions in each level and number of
observed emotions in both sessions. Table 6 presents the impact
value for emotions/behaviors in each engagement level. The
impact value of emotion in strong engagement level is 1/(1 emo-
tion). So, this emotion has 1 impact value. In high engagement
level there are 6 emotions, the impact value of each emotion in this
106 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109

Surprise

90

Angry 47 46 Enthusiastic
48 42 16
Ashamed 43 41
44
Excited
Fearful 135 31 33 37
39 45 45 Strong
18 27 25 19 14 23
Nervous 21 Engagement
2 1 12
11 6 7
15
47 48 50 49 44 40
47 42 42 43 49 43 46 48 48
41
41 33 33 40 17 40 34 32 42 15 43 31 34 32 42
45 35 14 15 30 25 45 31 36 37 38 48 High
47 39 36 33 32 2 4 20 27 40 2
39 42
15
3 22 13 38 26 23 6 7 3 45
44 36 34 31 3 1 4 8 33 26 Engagement
3 5 9 12 23 2634 30 24 19 12 7 4 3
8 10 45
45 38
35 31 1 5 7 6 4 710 10 16 24 29 23 19 12 11 11 10 13 37 43 48
46 43 36 32 7 8 11 10 23 30 22 27 20 18 4 14 13
33 46 50
34 13 13 8 18 24 20 16 21 1 49
45 37 37 33 39 23 38 11 40 33 20 18 15 28
41 45 48 47 46 45 46 42 39 45
46 50 49
Happy
41
4 10 22 24 25 34 32 40 43 32 33 38 16 20 36 37
Disgusted 40 43 48 47 Content
180 43 37 33 1 6 10 11 18 19 27 38 38 30 26 22 18 11 7 6
46
0 Medium
Disappointed 31 42 42 28 25 22 17 11 8 7 6 34 37 41 Delighted
42 34 3 5 8 11 16 18 22 29 42 45 Engagement
33 3 34 27 23 18 13 12 8 3 34 41 Joyful
Sad 44 35 7 9 15 17 20 23 42 31
32 25 22 19 15 12 40 43
48 36 33 24 10 15 18 20 23 32 31 24 34 47 12 36 41 42 48 Satisfied
43 39 38 28 26 32 31
14 19 20 23 30
40 36 14 12 13 33
50 37 31 7 2 13 15 16 22 27 29 19 44 40
4 6 11 17 23 28 45 16 21
40 35 31 5
1 2 4 10 18 27 34 24 9 8 36 49
2 41 48
Low
47 39 34 32 32 1 3 4 3 32 43 25 17 5 37 50
41 37 35 Engagement
48 47 35 34 33 33 33 42 44 45 38
39
41 34
49 45 42 42 43 46

No
emotions
only Disengagement
Behaviors
225 270
Bored
Depressed Relaxed
Tired Calm
Sleepy
315
Still
Quiet

Fig. 7. The distribution of volunteers on our affective model based on their detected engagement level.

Table 5 level is 1/(6 emotions) = 0.166. Medium engagement level has 8

Matching score (MS) and miss-matching score (MisMS) for each engagement level. emotions, the impact value of each emotion in this level is 1/(8
Engagement Levels MisMS1 MisMS2 MS emotions) = 0.125. Low engagement level has 5 emotions, the
impact value of each emotion in this level is 1/(5 emotions) = 0.2.
Strong Engagement 0% 0% 100%
High Engagement 0% 0% 100% Disengagement level has 2 Emotions, 7 Behaviors, so the impact
Medium Engagement 0% 0% 100% value of each emotion/behavior in this level is 1/(9) = 0.111.
Low Engagement 10.04% 18.04% 71.2% Table 7 shows an example of the final engagement level detec-
Disengagement 0% 100% 0% tion (EL12;session1 ; EL12;session2 ) for the volunteer 12. In first session, this
volunteer felt surprised, enthusiastic, excited, nervous, satisfied,
and Joyful. Impact value for each emotion was computed. Then,
we sum the impact value of all emotions that have the same
Table 6 engagement level. The number of observed emotions in strong
The impact value for emotions/behaviors in each engagement level.
engagement level is 1 emotion, in high engagement level is 3 emo-
Engagement Levels (EL) Number of emotion/EL Impact Value tions, and in medium engagement level is 3 emotions. In addition,
Strong Engagement 1 Emotion 1 the number of all emotions that observed in first session is 7 emo-
High Engagement 6 Emotions 0.166 tions. So, then we compute engagement in each level and take the
Medium Engagement 8 Emotions 0.125 maximum value as a final engagement. Strong engagement got
Low Engagement 5 Emotions 0.2
Disengagement 2 Emotions, 7 Behaviors 0.111
0.14 engagement value, high engagement got 0.2 engagement
value, and medium engagement got 0.16 engagement value. The

Table 7
The detection of final engagement level for the volunteer 12.

Volunteer ID Session Emotion / Behavior Impact Value Engagement Level Engagement level Value Final Engagement Level
12 1 Surprised 1 Strong 0.14 High Engagement
Enthusiastic 0.166 High 0.2
Excited 0.166 High
Nervous 0.166 High
Satisfied 0.125 Medium 0.16
Joyful 0.125 Medium
Happy 0.125 Medium
2 Bored 0.2 Low 0.06 Disengagement
Not looking at the computer 0.111 Disengagement 0.148
Playing with her hair 0.111 Disengagement
 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109 107

Session 1

Engagement Level Value

Strong High Meduim Low Disengagement
0.8

0.6

0.4

0.2

0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Volunteer ID

Fig. 8.1. The detection of final engagement level in session 1 for all volunteers.

Session 2
Engagement Level Value

Strong High Meduim Low Disengagement Unknown

0.8

0.6

0.4

0.2

0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Volunteer ID

Fig. 8.2. The detection of final engagement level in session 2 for all volunteers.

Table 8 final engagement in first session of volunteer 12 is a high engage-

Number of final engagement level in each session. ment, because got the maximum engagement value rather than
Levels First Session Second Session other levels.
Interesting Tasks Boring Tasks Figs. 8.1 and 8.2 shows the detection of final engagement level
Strong Engagement 8 0
for all volunteers in session 1 and 2 respectively. In first session
High Engagement 35 0 (interesting tasks), 8 volunteers got a strong engagement as final
Medium Engagement 7 2 engagement. The volunteers they got a high engagement as final
Low Engagement 0 33 engagement were 35. In addition, 7 volunteers got a medium
Disengagement 0 12
engagement as final engagement. In second session (boring
Unknown 0 3
tasks), 2 volunteers got a medium engagement as final engage-

(a) Enthusiastic (b) Disappointed

60 40
40 Content 30 Satisfied
Ashamed Nervous 20
20 10
0 Delighted 0 Joyful

Fearful Excited
Disgusted Sad
Angry Happy

Bored
(c) 50 (d) Not looking at
40 the computer
30
30
Talking with Stopped the
20 20
Relaxed Sleepy other video
10 10
0 0
Playing with her Played with the
hair chair

Seeing the
Looking at her
Tired Depressed remaining time
phone
of the video

Fig. 9. The analysis of emotions in each engagement level. (a) High engagement level. (b) Medium engagement level. (c) Low engagement level. (d) Disengagement level.
108 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109

Table 9
Emotions and degree angle for each engagement level in proposed affective model.

Our Levels Degrees in Emotions

Engagement
Strong

Surprised Models
Strong Engagement Emotions in Surprise
90°
Engagement

High Engagement Emotions in Enthusiastic, Excited, Angry,

High

Enthusiastic
45° + 135° Ashamed, Fearful, Nervous
Medium Engagement Emotions in Happy, Content, Delighted,
Engagement

0° + 180° Joyful, Satisfied, Disgusted,

Medium

Disappointed Disappointed, Sad

Low Engagement Emotions in Bored, Depressed, Tired,
225° + 315° Sleepy, Relaxed
Disengagement Engagement
Low

Bored Disengagement Emotions in Still, Quiet

270°

Not looking at the computer

has 5 emotions. As showed in Fig. 9.c, the largest number of volun-

0 10
Fig. 10. Strong emotions/behaviors in each engagement level.
ment. The volunteers they got a low engagement as final engage-
ment were 33. In addition, 12 volunteers got a disengagement as
final engagement. The last 3 volunteers couldn’t detect their final
engagement level, because it was a calm emotion only (Note, the
calm emotions were not taken into account in calculating the final
engagement level, because it has contradictory values). When looking
to Table 8, we noticed most volunteers have a high engagement in
the first session because it is an interesting task for them. Further-
more, the most volunteers have a low engagement in the second
session because it is a boring task for them. The strong and high
engagement detected only in interesting tasks (session 1). The
low engagement and disengagement detected only in boring tasks
(session 2). The medium engagement detected in both sessions.
5.3. Third experiment
In each engagement level there are some emotions/behaviors
have appeared on volunteers more than other emotions/behaviors.
The strong engagement level has only one emotion ‘‘surprise” and
the 28 volunteers have felt of this emotion. The high engagement
level has 6 emotions. As showed in Fig. 9.a, the largest number of
volunteers were enthusiastic, and least number of volunteers were
ashamed. The medium engagement level has 8 emotions. As
showed in Fig. 9.b, the largest number of volunteers were disap-
pointed, and one volunteer was content. The low engagement level
20 30 40 50 teers were bored, and no volunteers were relaxed. The disengage-
ment level has 7 behaviors. As showed in Fig. 9.d, the largest
number of volunteers were not looking at the computer, and one
volunteer was talking with other. Fig. 10 shown the strong emo-
tion/ behavior in each engagement level that appears more than
others.
Based on our experiments, we propose several modifications to
improve the proposed Affective Model as showed in Fig. 11 and
some details in Table 9. These updates in proportion to the results
that emerged from the experiment. The results as the following:
(1) All emotions in strong, high and medium engagement level
appeared as what we proposed so, we kept them. (2) These three
emotions Relaxed, Still, Quiet are not appeared at all, but may
appear with other volunteers so, we kept them. (3) The Calm emo-
tion appeared in one session as low engagement but in other ses-
sion as high engagement. This is a problem that will hinder us in
determining the right level of engagement so, it has been removed.
(4) The Sleepy emotion is a new emotion appeared on 23 students
and it represent as a low engagement level so, we added it. (5)
Finally, in disengagement level there are no emotions but new
behaviors. Disengagement level recognized if not detect any emo-
tions from other level. This level needs more investigation. So, we
need to analysis the behavior, not just face expression.
6. Conclusion
In this paper, we proposed Affective Model that measure stu-
dent’s engagement based on their emotions. All engagement levels
and emotions in this model were selected depending on previous

Strong Engagement

90 Strong Engagement
High Negative Affect High Positive Affect
135 Surprise 45
High Engagement
Angry
Ashamed Enthusiastic
Fearful Excited
Nervous
Happy
Disgusted Content Pleasantness
Unpleasantness 180 Disappointed Delighted
0
Medium Engagement
Sad Joyful
Bored Satisfied
Depressed Calm
Tired Relaxed
Sleepy
Low Engagement
Still
Quiet 315
225
Low Positive Affect Low Negative Affect
270 Disengagement
Disengagement

Fig. 11. Proposed affective model.

 K. Altuwairqi et al. / Journal of King Saud University – Computer and Information Sciences 33 (2021) 99–109
related works. We have five engagement levels, strong, high, med-
ium, low, and disengagement. Each level has specific emotion/s
that indicated of specific level. We validate the proposed Affective
Model by conducted a series of experiments. We recorded 100
videos for 50 volunteers in two sessions for each of them and anal-
ysis these videos. In experiment 1, we computed the Matching
Score (MS) and Miss-Matching Score (MisMS) for each engagement
level. The strong, high and medium engagement level got 100%
matching score (MS). But the low engagement level got 71.2%
MS. The disengagement level got 100% miss-matching score, due
to their no emotions observed only behaviors. In experiment 2,
we proposed and apply a new process to detect final engagement
level in both sessions. The most volunteers have a high engage-
ment in the first session because it is an interesting task for them.
Furthermore, the most volunteers have a low engagement in the
second session because it is a boring task for them. In experiment
3, we analyze all emotions in each level of engagement to detect
strong emotions. The surprise, enthusiastic, disappointed, bored,
and not looking at the computer were the strong emotions/
behavior.
Finally, we proposed some suggestions to improve the affective
model. In disengagement level we need more investigation. So, as a
future work, we will analysis the body activities of student with
emotions to define a complete behavior for each engagement level.
Acknowledgements
The authors, therefore, gratefully acknowledge the Deanship of
Scientific Research (DSR), King Abdul-Aziz University, Jeddah,
Saudi Arabia, under grant No (KEP-3-612-1439).
Funding
This work was supported by the DSR under grant No (KEP-3-
612-1439).
Conflict of interest
Khawlah Altuwairqi declares that she has no conflict of interest.
Salma Jarraya declares that she has no conflict of interest. Arwa
109
Allinjawi declares that she has no conflict of interest. Mohamed
Hammami declares that he has no conflict of interest.
References
Bakhtiyari, K., Taghavi, M., Husain, H., 2015. Hybrid affective computing—keyboard,
mouse and touch screen: from review to experiment. Neural Comput. Appl. 26
(6), 1277–1296.
Bernardo, A.B., Ouano, J.A., Salanga, M.G.C., 2009. What is an academic emotion?
Insights from Filipino bilingual students’ emotion words associated with
learning. Psychol. Stud. 54 (1), 28–37.
D’Errico, F., Paciello, M., Cerniglia, L., 2016. When emotions enhance students’
engagement in e-learning processes. J. e-Learn. Knowl. Soc. 12 (4).
D’Mello, S., Graesser, A., 2012. Dynamics of affective states during complex learning.
Learn. Inst. 22 (2), 145–157.
Finch, D., Peacock, M., Lazdowski, D., Hwang, M., 2015. Managing emotions: a case
study exploring the relationship between experiential learning, emotions, and
student performance. Int. J. Manage. Educ. 13 (1), 23–36.
Fredricks, J.A., Blumenfeld, P.C., Paris, A.H., 2004. School engagement: potential of
the concept, state of the evidence. Rev. Educ. Res. 74 (1), 59–109.
Griffiths, P.E., 2003. III. Basic emotions, complex emotions, Machiavellian emotions
1. R. Inst. Philos. Suppl. 52, 39–67.
Jang, M., Park, C., Yang, H.S., Kim, J.H., Cho, Y.J., Lee, D.W., et al., 2014. Building an
automated engagement recognizer based on video analysis. In: Proceedings of
the 2014 ACM/IEEE international conference on Human-robot interaction. ACM,
pp. 182–183.
Li, J., Ngai, G., Leong, H.V., Chan, S.C., 2016. Multimodal human attention detection
for reading from facial expression, eye gaze, and mouse dynamics. ACM SIGAPP
Appl. Comput. Rev. 16 (3), 37–49.
Monkaresi, H., Bosch, N., Calvo, R.A., D’Mello, S.K., 2017. Automated detection of
engagement using video-based estimation of facial expressions and heart rate.
IEEE Trans. Affective Comput. 8 (1), 15–28.
Pekrun, R., 2006. The control-value theory of achievement emotions: assumptions,
corollaries, and implications for educational research and practice. Educ.
Psychol. Rev. 18 (4), 315–341.
Pekrun, R., Goetz, T., Titz, W., Perry, R.P., 2002. Academic emotions in students’ self-
regulated learning and achievement: a program of qualitative and quantitative
research. Educ. Psychol. 37 (2), 91–105.
Remington, N.A., Fabrigar, L.R., Visser, P.S., 2000. Reexamining the circumplex
model of affect. J. Pers. Soc. Psychol. 79 (2), 286.
Russell, J., Lemay, G., 2000. Emotion concepts. In: Lewis, M., Haviland-Jones, M.J.
(Eds.), Handbook of Emotion. second ed. Guilford Press, New York, pp. 491–503.
Schlechty, P.C., 2011. Engaging Students: The Next Level of Working on The Work.
Wiley, John & Sons, San Francisco.
Watson, D., Tellegen, A., 1985. Toward a consensual structure of mood. Psychol.
Bull. 98 (2), 219.
Whitehill, J., Serpell, Z., Lin, Y.C., Foster, A., Movellan, J.R., 2014. The faces of
engagement: automatic recognition of student engagement from facial
expressions. IEEE Trans. Affective Comput. 5 (1), 86–98.
Yik, M.S.M., Russell, J.A., Barrett, L.F., 1999. Structure of self-reported current affect:
Integration and beyond. J. Pers. Soc. Psychol. 77 (3), 600–619.