See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/337252943

Subject Independent Emotion recognition from EEG using VMD and Deep
Learning

Article  in  Journal of King Saud University - Computer and Information Sciences · November 2019
DOI: 10.1016/j.jksuci.2019.11.003

CITATIONS READS

27 710

2 authors:

Pallavi Pandey Seeja K.R.

Indira Gandhi Delhi Technical University for Women Indira Gandhi Delhi Technical University for Women
5 PUBLICATIONS   61 CITATIONS    40 PUBLICATIONS   444 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

EEG signal analysis View project

All content following this page was uploaded by Pallavi Pandey on 28 November 2019.

The user has requested enhancement of the downloaded file.

 Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx

Contents lists available at ScienceDirect

Journal of King Saud University –

Computer and Information Sciences
journal homepage: www.sciencedirect.com

Subject independent emotion recognition from EEG using VMD and deep
learning
Pallavi Pandey, K.R. Seeja ⇑
Department of Computer Science & Engineering, Indira Gandhi Delhi Technical University for Women, Kashmere Gate, Delhi 110006, India

a r t i c l e i n f o a b s t r a c t

Article history: Emotion recognition from Electroencephalography (EEG) is proved to be a good choice as it cannot be
Received 30 July 2019 mimicked like speech signals or facial expressions. EEG signals of emotions are not unique and it varies
Revised 27 September 2019 from person to person as each one has different emotional responses to the same stimuli. Thus EEG
Accepted 5 November 2019
signals are subject dependent and proved to be effective for subject dependent emotion recognition.
Available online xxxx
However, subject independent emotion recognition plays an important role in situations like emotion
recognition from paralyzed or burnt face, where EEG of emotions of the subjects before the incidents
Keywords:
are not available to build the emotion recognition model. Hence there is a need to identify common
Variational Mode Decomposition
Valence-Arousal model
EEG patterns corresponds to each emotion independent of the subjects. In this paper, a subject
Deep Neural Network independent emotion recognition technique is proposed from EEG signals using Variational Mode
Affective computing Decomposition (VMD) as a feature extraction technique and Deep Neural Network as the classifier.
Intrinsic-mode functions The performance evaluation of the proposed method with the benchmark DEAP dataset shows that the
combination of VMD and Deep Neural Network performs better compared to the state of the art
techniques in subject-independent emotion recognition from EEG.
Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an
open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction
Emotion recognition is associated to the field of affective
computing. Affective computing is the study of how computers
process and recognize emotions. The major part of the emotions
in human’s daily life is the management of attention (Dolan,
2002) and decision making (Lerner et al., 2015). An emotionally
imbalance person may be less responsive to perform daily tasks.
So it is clear that emotions play important role in one’s life.
EEG recording is a non invasive method and hence researchers
are using it to study the neural activity of the brain related to
emotional responses. Emotion recognition from EEG has already
been explored successfully by the researchers in subject dependent
case (Liu and Sourina, 2014). In subject dependent approaches, EEG
data from same user is used to train as well as to test the emotion
recognition system. These types of emotion recognition work well
⇑ Corresponding author.
E-mail address: seeja@igdtuw.ac.in (K.R. Seeja).
Peer review under responsibility of King Saud University.
Production and hosting by Elsevier
for applications like online gaming where EEG data of the player
can be captured to train the model and later on that model is used
to evaluate the emotion/mood of the user while game playing.
However, for recognizing emotions from people whose face is
burned or paralyzed, the recorded EEG signal of the person before
the incident needs to be available for performing the subject
dependent emotion recognition. Most of the time, the need of
the emotion recognition based on EEG data, comes after the inci-
dent when it becomes difficult to capture their facial expressions.
In this situation a subject Independent EEG based emotion recogni-
tion system is required. Likewise, if a patient with depression
comes for diagnosis, whose earlier data is unavailable when he
was healthy, then also subject independent approach can be
adopted. There are various applications of emotion detection like
stress management (Kalas and Momin, 2016), anger management
(Mohamed et al., 2012) and depression detection (Cai et al.,
2018). Similarly driver’s state detection whether he is in anger/
stress, sleepy or calm state (Dabbu et al., 2017). If the driver is
not in calm state, some kind of alarm would be generated. Another
is fear detection (Masood and Farooq, 2019) for ATM machine
users. If user is in fear state, this means there is a threat to her/him
and then ATM will not dispatch the money.
In literature, we can find that not only EEG but other physiolog-
ical signals like electro-myogram, facial muscle tension, blood

https://doi.org/10.1016/j.jksuci.2019.11.003
1319-1578/Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
2 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
pressure volume and a skin conductance (Picard et al., 2001;
Chanel et al.,2011) are also used to detect emotions. Various tech-
niques like Short Time Fourier Transform (STFT) (Lin et al., 2010;
Ackermann et al., 2016), Multifractal detrend fluctuation analysis
(Paul et al., 2015) and Empirical mode decomposition (EMD)
(Zhuang et al., 2017; Mert and Akan, 2018) are used to extract var-
ious frequency bands from EEG. Several other features like power
spectral density (PSD) (Lin et al., 2010; Ackermann et al., 2016)
and discrete wavelet transform (Aydin et al., 2016; Pandey and
Seeja, 2019a,b) are also extracted from the EEG bands for emotion
recognition. In a comparative study (Jatupaiboon et al., 2013) on
various features of EEG for emotion recognition, PSD performed
best as compared to other features like spectral power asymmetry,
higher order spectra, higher order crossing, common spatial pat-
terns and asymmetric spatial patterns for classifying two emotions
- happy and unhappy. The authors reported that beta and gamma
frequency bands and channel T7 and T8 give good results as com-
pared to other bands and other pairs of electrodes.
Various classifiers (Alarcao and Fonseca, 2017) are used for clas-
sifying emotions with EEG data. Out of which Support Vector
Machine (SVM) (Wang et al., 2011; Shahabi and Moghimi, 2016)
is the most reported in the literature. Most of them are subject
dependent emotion classifiers. In some subject dependent
approaches, linear SVM (Wang et al., 2014) performed well while
in some other studies RBF SVM (Atkinson and Campos, 2016;
Chanel et al., 2009), polynomial SVM (Lan et al., 2016) and adaptive
SVM (Liu et al., 2013) have given good results. K-Nearest Neighbor
(KNN) classifier (Xu and Plataniotis, 2012; Mohammadi et al.,
2017) is also found to be effective in emotion classification with
EEG data. There were few (Jirayucharoensak et al., 2014; Zheng
and Lu, 2015) emotion classification attempts with deep learning
too.
Most of the existing work (Petrantonakis and Hadjileontiadis,
2010; Soleymani et al., 2012; Lin et al., 2014) on subject indepen-
dent emotion classification is on self produced databases and
hence it is very difficult to perform a comparative study on their
performance. (Petrantonakis and Hadjileontiadis, 2010) used
higher order crossing to extract features on a self-produced data-
base with 16 subjects. They have classified six basic emotions
using SVM classifier and obtained an accuracy of 83.33%. In
another research (Soleymani et al., 2012), an accuracy of 62.1%
on three level of arousal and 50% on three level of valence has been
obtained on a self created database using SVM. Lin et al. (2014) cre-
ated an EEG database with 26 subjects and worked on both
approaches- subject dependent and independent. For subject inde-
pendent, they got an accuracy of 61.09% for valence and 57.33% for
arousal with SVM.
Most of the emotion detection researches on DEAP databases
are subject dependent. Zhuang et al. (2017) have proposed a sub-
ject dependent emotion recognition method using eight electrodes
data and they achieved an accuracy of 69.10% for valence and
71.99% for arousal with DEAP database. Atkinson and Campos
(2016) have used various statistical features of EEG and with
SVM classifier on DEAP database and achieved an accuracy of
73.14% for valence and 73.16% for arousal. Further, with three
levels of valence and arousal classification, they got an accuracy
of 60.7% for arousal and 62.33% for valence. Zhang et al. (2016) per-
formed both subject dependent and independent classification for
four emotions with SVM. They have taken data of 16 subjects from
30 electrodes from DEAP database. They got an accuracy of 62.59%
for subject dependent classification and 58.75% for subject inde-
pendent classification. Li et al. (2018) have worked on two data-
bases - DEAP and SEED. On DEAP, they achieved an accuracy of
59.06% for positive and negative emotions and on SEED data they
got an accuracy of 83.33% with SVM.
Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
In the proposed method, Empirical Mode Decomposition (EMD)
and Variational Mode Decomposition (VMD) are used to obtain
intrinsic mode functions (IMF) from EEG data. For each IMF, two
features namely Peak value of Power Spectral Density and First dif-
ference of the signal are calculated and then these features are fed
into a Deep Neural Network for classification.
2. Material and methods
2.1. EEG data
EEG is a noninvasive procedure used to track electrical func-
tions of the brain along the scalp and hence it is the first choice
of researchers for several applications (Read and Innis, 2017).
Human emotion recognition is one of those applications. A repre-
sentative EEG signal may have amplitude of 10 mV to 100 mV and
frequency range of 1 Hz to 100 Hz approximately. There are mainly
five types of EEG frequency bands: Delta band which is less than
4 Hz, Theta band which is greater than or equal to 4 and less than
8 Hz, Alpha band which is greater than or equal to 8 and less than
or equal to 14, Beta band which is greater than 14 and less than
40 Hz and Gamma band is greater than or equal to 40 Hz.
EEG signals can be recorded by two ways - mono-polar record-
ing and bipolar recording. Mono-polar recording can be obtained
by the voltage difference between the electrode placed on the scalp
and reference electrode situated at ear lobe. In bipolar recording,
voltage difference of two scalp electrodes is recorded. Subject will
wear an electrode cap while watching the stimuli for fixed dura-
tion of time and the EEG recording will take place with EEG record-
ing software. The electrodes in the electrode cap would be placed
as suggested by 10/20 international electrode placement system
(Acharya et al., 2016) as shown in Fig. 1. The numbers 10/20 are
used to establish the constraint on the distances between neigh-
boring electrodes. Constraint is that contiguous electrodes ought
to be far off each other either ten percent or twenty percent of
the total front to back or left to right distance of the skull. Head
is partitioned into various lobes. Several lobe positions are repre-
sented by letters.
2.2. Emotion representation
Emotion can be represented using either categorical model or
dimensional model (Mauss and Robinson, 2009). Emotions are
labeled in categorical model like ‘surprised’ or ‘anger’ emotions.
In dimensional-model (Lang, 1995) emotions are expressed in
terms of several dimensions like Valence, Arousal, Dominance
and Liking/Disliking. The two dimensional model using Valence
and Arousal is shown in Fig. 2. Emotions are labeled as discrete
points in this valence-arousal space. Valence axis goes from nega-
tive to positive and arousal axis goes from passive to active. In this
modal, the space is divided into four quadrants. Valence and arou-
sal values are rated on the scale which contains values 1 to 9. If the
rating of valence scale is greater than 5 and arousal is greater 5
then the emotion could be ‘Excited’ or ‘Happy’ and falls in first
quadrant. If the Valence is less than 5 and Arousal greater than 5
then the emotion could be ‘Angry’ or ‘Afraid’ and falls in second
quadrant and so on.
2.3. DEAP database
To the best of our knowledge, DEAP database (Koelstra et al.,
2012) used in this work is the publically available EEG database
for emotion recognition which contains maximum number of sub-
jects. SEED and MAHNOB are other publically available EEG data-
base for emotion recognition containing data from 15 and 30
 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx 3

Fig. 1. 10–20 Electrode-Placement system (Acharya et al., 2016).

data. EEG was recorded at 512 Hz with 32 active AgCl electrodes

placed on the subjects scalp according to the international 10/20
electrode placement system. Then the data is down sampled to
128 Hz. The statistics of the database is as follows:
Number of electrodes – 40 (32 to record EEG, 8 to record other
physiological signals)
Number of files – 32 .mat files (one for each subject) and a
metadata file containing participants rating.
Data dimension – 40
40
8064 for one subject (here first
dimension shows video, second dimension shows electrodes posi-
tion and third shows voltages.)

2.4. Feature extraction

Intrinsic mode functions (IMF) of an EEG signal provide mean-

Fig. 2. Valence-Arousal model of emotions (Jirayucharoensak et al., 2014).
subjects respectively. The DEAP database contains data of 16 male
and 16 female subjects and therefore the data is not gender biased.
The data is collected by allowing the participants to watch a one
minute duration video by wearing an electrode cap and asking
them to rate their emotions on valence and arousal scale. In this
way, the EEG is recorded when the subjects were watching videos
and they have rated how much valence and arousal they felt on
valence arousal scale of 1 to 9 using self assessment manikin
(Morris, 1995) as shown in Fig. 3.
There were forty videos and for every video, this database holds
readings of forty electrodes. Authors have already preprocessed the
ingful time-frequency information about the signal. In the pro-
posed method, EMD and VMD techniques are used to compute
the IMFs of EEG signal. When the signal is decomposed into its var-
ious oscillatory components, these components are called IMFs.
After getting IMF signals of EEG data using EMD or VMD, two fea-
tures namely peak value of PSD and first difference of the signal are
calculated.
2.4.1. Empirical mode decomposition (EMD)
In EMD (Huang et al., 1998), for a signal sðtÞ, its IMFs are
obtained by using a repetitive process called sifting.
Each IMF must satisfy two conditions. First is that the number
of zero crossing and extrema must be same or differ by at most
one. Next, the mean value of the upper envelop defined by local
maxima and lower envelop defined by local minima is zero. The
complete process to find IMF involves the following steps:

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
4 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx

Fig. 3. Self-assessment-manikin used to rate videos (Morris, 1995).

1) Find out all minima and maxima in sðtÞ. sum of bandwidths of IMFs such that the sum of all the uk is equal
2) Using interpolation, get the upper envelop env max ðtÞ and to main signal. This optimization problem can be formulated
lower envelop env min ðtÞ by connecting maxima and minima (Dragomiretskiy and Zosso, 2014) as given in Eq. (2).
respectively. ( )
X
3) Compute the mean of env max ðtÞand env min ðtÞ as. min jj@t ½dðtÞ þ j=pt  uk ðtÞejwk t jj22 ð2Þ
mðtÞ ¼ env max ðtÞþen
2
v min ðtÞ
. ained mean value is subtracted from uk ;wk
k
the original signal s ðt Þ. to get the details as P
Such that k U k ¼ S where sis the original signal which has to be
dðtÞ ¼ sðt Þ  mðt Þ.
decomposed into k IMFs.
4) cide whether dðt Þ. satisfy the two basic conditions of IMF.
A sample of three IMFs obtained using VMD is shown in Fig. 5.
5) To obtain first IMF Iðt Þ, repeat the above steps from (a) to
(e) until it satisfies the required two conditions of IMF.
Now this obtained details signal d(t) will be the first IMF i. 2.4.3. Power-spectral-density (PSD)
Power-spectral-density describes how power of a signal is dis-
eIðtÞ ¼ dðt Þ. . .or next IMF, calculate the residue
tributed over frequency. In this research, Welch method (Rahi
xðtÞ ¼ sðtÞ  Iðt Þ. This residue will be termed as a new signal.
and Mehra, 2014) is used to calculate PSD of IMF.
Now above steps will be repeated again.
6) The whole process will be continued until the residue signal
will satisfy some stopping criteria (Say, it becomes 2.4.3.1. Welch method.
constant). 1. Partition the signal s[0], s[1],. . .,s[N  1] into k segments.
2. For each segment 1 to k, Compute a windowed discrete fourier
The original signal in terms of its decomposed IMF components transfer at a frequency v ¼ i=M where ðM=2  1Þ  i  M=2 as
is shown in Eq. (1) shown in Eq. (3).
X
X
k
Sk ð v Þ ¼ S½mw½mexpðj2pv mÞ ð3Þ
sðtÞ ¼ Ii ðtÞ þ xk ðtÞ ð1Þ m
i¼1
where m varies from (k  1)L to M+(k  1)L  1, w[m] is the win-
In the above equation k represents the number of IMFs obtained
dow function, M represents segment size and L is the number of
and Ii ðtÞis the ith IMF. A sample of six IMFs obtained using EMD is
points that is shifted between the segments.
shown in Fig. 4.
3. For each segment, compute the modified periodogram value
2.4.2. Variational-Mode-Decomposition (VMD): using Eq. (4).
VMD (Dragomiretskiy and Zosso, 2014) is a time-frequency
decomposition approach. It is made to overcome the limitations Pk ðv Þ ¼ 1=wabsðSk ðv ÞÞ2 ð4Þ
present in EMD, listed as follows: PM
where w ¼ m¼0 w ½m
2

i. EMD uses a recursive approach that does not allow back-

ward error correction. 4. Estimate of the power spectral density by finding the average of
ii. It is unable to handle noise properly. the obtained periodogram values using Eq. (5).

VMD does not use recursive approach; instead it uses concur- X

K

rent approach to extract intrinsic mode functions from the signal. Ls ðv Þ ¼ 1=K Pk ðv Þ ð5Þ
k¼1
This is an adaptive method and decomposes the signal into k IMFs
and gives set of modes uk with their respective centre frequencies The number of points common to 2 adjacent segments is equal
wk. The sum of these modes represents the original signal. VMD is to (M-L) i.e. two adjacent segments will be overlapped by (M-L)
less susceptive to noise in comparison to EMD and it does not leave points. After getting power spectral density signal, the peak value
residual noise (Jiang et al., 2019). of it is selected as a feature. Fig. 6(a) and (b) shows a sample PSD
In VMD, identification of IMFs is considered as an optimization identified using Welch method for low valence class and high
problem. The aim of optimization algorithm is to minimize the valence class respectively.

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx 5

Fig. 4. First six IMFs using EMD (electrode fp1 of subject 1 for high valence).

Fig. 5. First three IMFs using VMD (electrode fp1 of subject 1 for high valence class).

Visualizing both the plots, it is clear that peak value of PSD of the voltage variation of EEG during a period of time (ie, during var-
IMFs can discriminate both the classes and hence it is selected as ious emotional stimuli), we have selected First Difference of IMF
feature for emotion recognition. also as a feature.

2.4.4. First difference of IMF 2.5. Support vector machine

First difference represents the changes in time from one period
to the next. If yt denotes the value at time t of the time series y then SVM is a machine learning technique that classifies the data by
First Difference¼ yt  yt1 . Since in this research, we are analyzing drawing maximal marginal hyper planes which segregates the data

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
6 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx

Fig. 6. PSD of first IMF (method-VMD, electrode fp1, subject 1) (a) Low valence class (b) high valence class.

very well. For non-separable data, it finds the hyper plane that
maximizes the margin and minimizes the misclassification by
introducing penalty term for misclassification. It maps the data
by using kernel function. As described in Section 1, various studies
have suggested SVM as a better classifier for EEG based emotion
classification tasks and hence it has been selected in this research
for performance comparison with DNN.
2.6. Deep Neural Network
It is a machine learning technique that can be used to learn fea-
tures as well as to perform classification tasks. The major attraction
towards DNN is its ability to work well with large amount of data.
If a neural network consists of more than three layers including
input and output layer, this network comes under the category of
deep neural network. A simple model of deep neural network with
three hidden layers is shown in Fig. 7. The number of hidden layers
and the number of neurons at each hidden layers can be varied.
DNN also has the capability to detect stronger features from the
data and hence it is selected in this research for the emotion recog-
nition task.
2.7. Proposed methodology
The schematic diagram of the proposed emotion recognition
methodology using EEG signals is shown in Fig. 8. The subjects will
wear an electrode cap during the recording of EEG and they will
watch videos. In this way the emotions will be induced and corre-
sponding EEG will be recorded. Then the signals are preprocessed
and artifacts are removed. Finally the data base of preprocessed
EEG Data for emotion recognition is created. In this study the
bench mark DEAP database is used that is already preprocessed.
Then features of EEG are extracted by using EMD and VMD
techniques. Obtained features are fed into the deep neural network
to classify high/low valence and high/low arousal. These high/low
valence and arousal indicators are used to obtain emotion label as
described in the Section 2.2.
The proposed methodology of subject independent emotion
classification is outlined as follows:
1. Data collection
EEG data corresponds to the electrodes fp1, fp2, F3 and F4 of all
the 40 videos are selected from the DEAP database. Out of the
32 subjects, the data corresponds to 30 subjects are selected
for training and 2 subjects are selected for testing in order to
make it a pure subject independent approach.
2. Feature extraction
Two features are extracted from EEG signals – Peak value of PSD
and First difference. These features are calculated from the IMFs
of EEG signal. For each EEG signal, extract the IMFs using EMD
or VMD. Then by using Welch method, the PSD of each IMF is
calculated. Select the peak value of the PSD as a feature. Calcu-
late the first difference of IMF as another feature as described in
Section 2.4.4.
3. Classification
The two features calculated from the top IMFs are combined
together to form the feature vector. Then the DNN classifier is
trained with the feature vectors correspond to 30 subjects
selected for training and then tested with the feature vectors
correspond to the 2 subjects selected for testing.
3. Experiments and results
The proposed methodology has been implemented in Python
with tensor-flow platform. Gradient-descend based optimization
algorithm is used and the activation function is Relu. At the output
layer soft max function is used. DEAP database contains EEG
recordings of 32 electrodes corresponds to 40 videos and 32

Fig. 7. Deep Neural Network.

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx 7

Fig. 8. Schematic diagram of the proposed method for emotion recognition from EEG.

subjects. We have selected the EEG recording from four electrodes
namely fp1,fp2,F3 and F4 (Petrantonakis and Hadjileontiadis,
2010) since these electrode positions are already identified as emo-
tion specific. Also, we have considered the valence/arousal rating
greater than 5 as High and less than or equal to 5 as LOW.
3.1. Experiment 1: EMD based features
In this experiment, EEG signals were decomposed in to several
IMFs using EMD. For some EEG signals, the total number of IMFs
obtained was 11 and for some signals it was 12 based on the stop-
ping criteria (i.e., when the IMF becomes constant). Then the top
IMFs were selected for feature calculation. For each of the selected
IMFs, the Peak value of PSD and First difference of IMF are calcu-
lated. The total number of features obtained for one electrode is
6.(i.e.,3
2) and thus for 4 electrodes (fp1, fp2, F3 and F4) it is
24. (i.e.,4
3
2). The features of the selected IMFs are then given
to the two selected classifiers – SVM and DNN. The number of top
IMFs per EEG signal was selected based on exhaustive experimen-
tation. After trying different options like top one IMF, top two IMFs,
top three IMFs and so on, we got best results with top three IMFs as
given in Table 1.
As shown in Table 1, the first three IMFs are highly correlated
with EEG signal and hence for the second experiment only the
top three IMFs are considered for feature extraction.
3.2. Experiment 2: VMD based features
In this experiment, EEG signals were decomposed in to several
IMFs using VMD. Then the top three IMFs of each signal are
selected for feature extraction. The peak value of PSD and first dif-
ference of each of the selected IMFs are calculated as features.
Then, the obtained feature vector is fed into SVM as well as DNN.
In the case of SVM, linear, polynomial and radial basis function
(RBF) kernel are examined and it is found that RBF is best suited
for VMD based features. Similarly, different topologies of DNN
are examined by changing the number of hidden layers and num-
ber of neurons in hidden layers. The best results were obtained at
three hidden layers as shown in Table 2.
Table 3 shows the performance comparison of the classifiers
(SVM with RBF kernel and DNN with three hidden layers) with
VMD based features. We did the experiment with data from 2 elec-
trodes (fp1 and fp2) also in order to reduce the number of features
as shown in Table 3. However the best results were obtained with
features from four electrodes.
4. Discussion
From the experiments, it is found that
1. Features of IMFs like peak value of PSD and first differences are
effective for emotion classification.
2. Features of top IMFs, especially the first three IMFs, are more
informative for the emotion recognition task.
3. EEG recordings of electrodes fp1, fp2, F3 and F4 are related to
emotions as found in literature.
4. VMD based features with DNN perform best for both Arousal
and Valence classification.
The comparison of EMD and VMD based features in terms of
classification accuracy for arousal and valence is shown in the
Fig. 9.
In DEAP database, the valence and arousal are rated in a 1–9
(low to high) scale. With this valence–arousal model (Russell,
1980), emotions can be classified as shown in Table 4. Thus, with
the proposed binary classification model, four different emotions
can be identified.
Various attempts have been made to develop human emotion
recognition systems as explained in Section 1. However most of
them are subject dependent approaches and have used self created

Table 1
SVM vs. DNN (EMD based features).

Arousal or Valence No of IMFs selected Train data size (Data Test data size (Data Classifier Kernal function Accuracy on Accuracy on
for each signal from 30 subjects) from 2 subjects) train Data (%) test data (%)
Arousal 3 1200
24 80
24 SVM RBF 83 58.4
Polynomial 92 56
Deep Neural Network 94 60
Arousal 6 1200
48 80
48 SVM RBF 86.5 56.8
Polynomial 94 50
Deep Neural Network 92 57.6
Valence 3 1200
24 80
24 SVM RBF 80 57.6
Polynomial 90.5 52
Deep Neural Network 91.75 56
Valence 6 1200
48 80
48 SVM RBF 82 56
Polynomial 86.5 48
Deep Neural Network 90.5 52.8

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
8 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx

Table 2
Accuracy at different DNN Topologies.

No. of Hidden Layers No. of neurons at hidden layers for Valence No. of neurons at hidden layers for Arousal Accuracy (%) Valence Accuracy (%) Arousal
3 (12, 24, 12) (7, 14, 7) 62.50 61.25
4 (12, 24, 12, 3) (12, 24, 12, 3) 55.00 59.75
5 (12, 24, 12, 21, 12) (12, 24, 12, 21, 12) 60.00 56.75
6 (100, 200, 100, 200, 100, 200) (100, 200, 100, 200, 100, 200) 56.25 52.50

Table 3
SVM vs. DNN (VMD based features).

Valence/ No. of Classifier No. of neurons at 3 hidden Train data size (Data from 30 Test data size (Data from 2 Accuracy on test
Arousal electrodes layers subjects) subjects) data (%)
Arousal 4 (fp1, fp2, F3, DNN (7, 14, 7) 1200
24 80
24 61.25
F4) SVM – 57.50
Valence DNN (12, 24, 12) 62.50
SVM – 58.75
Arousal 2 (fp1, fp2) DNN (4, 8, 4) 1200
12 80
12 57.50
SVM – 55.00
Valence DNN (10, 20, 10) 60.00
SVM – 56.25

Fig. 9. Comparison of different methods with respect to accuracy.

database with less number of subjects to evaluate the model. We

Table 4
have compared our results with the state of the art subject inde-
Emotion classification based on Valence-Arousal. pendent techniques, found in literature, on the bench mark DEAP
database and is shown in Table 5.
Valence Arousal Emotion
Positive (6–9) Passive (1–5) Calm/Content
Positive (6–9) Active (6–9) Happy/Excited 5. Conclusion
Negative (1–5) Passive (1–5) Sad/Depressed
Negative (1–5) Active (6–9) Angry/Afraid
In this paper, a subject independent emotion recognition
system based on EEG signals is proposed. In the proposed

Table 5
Performance comparison with state of the art techniques.

Article Approach Database Feature Extraction Classifier Accuracy (%)

(Li et al., 2018) Subject DEAP STFT SVM 59.06
Independent Positive and Negative
Emotions
(Lan et al., 2019) Subject DEAP Differential Entropy Domain adaption 48.93
Independent technique For Three level valence
(Rayatdoost and Soleymani, Subject DEAP Spectral topography maps of different Convolutional Arousal- 55.70
2018) Independent bands Neural Network Valence- 59.22
Proposed Subject DEAP VMD Deep Neural Network Arousal-61.25
Independent Valence-62.50

Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
 P. Pandey, K.R. Seeja / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
methodology, two features namely peak value of PSD and first dif-
ference were derived from the IMFs are used for emotion classifi-
cation. The IMFs are obtained from EEG signals using two
methods namely EMD and VMD. The experimental evaluation of
the proposed methodology is done on benchmark DEAP dataset
and it is found that the VMD based features from the top three
IMFs are better compared to EMD based features. It is also found
that the Deep Neural Network classifier performs better in case
of subject-independent emotion recognition compared to SVM
classifiers. In the proposed method, the EEG data of subjects used
for building and testing the model are different and hence it is a
generalized emotion recognition system. Thus, the proposed
method can be utilized for emotion recognition from the EEG of
any person whose data is not used for training of the classifier.
Declaration of Competing Interest
The authors declare that they have no known competing
financial interests or personal relationships that could have
appeared to influence the work reported in this paper.
References
Acharya, J.N., Hani, A.J., Cheek, J., Thirumala, P., Tsuchida, T.N., 2016. American
Clinical Neurophysiology Society guideline 2: guidelines for standard electrode
position nomenclature. Neurodiagnostic J. 56 (4), 245–252.
Ackermann, P., Kohlschein, C., Bitsch, J.A., Wehrle, K., Jeschke, S., 2016. EEG-based
automatic emotion recognition: feature extraction, selection and classification
methods. e-Health Networking, Applications and Services (Healthcom), 2016
IEEE 18th International Conference on (pp. 1-6). IEEE.
Alarcao, S.M., Fonseca, M.J., 2017. Emotions recognition using EEG signals: a survey.
IEEE Trans. Affective Comput.
Atkinson, J., Campos, D., 2016. Improving BCI-based emotion recognition by
combining EEG feature selection and kernel classifiers. Expert Syst. Appl. 47,
35–41.
Aydin, S.G., Kaya, T., Guler, H., 2016. Wavelet-based study of valence–arousal model
of emotions on EEG signals with LabVIEW. Brain Inf. 3 (2), 109–117.
Cai, H., Han, J., Chen, Y., Sha, X., Wang, Z., Hu, B., et al., 2018. A pervasive approach to
EEG-based depression detection. Complexity.
Chanel, G., Kierkels, J.J., Soleymani, M., Pun, T., 2009. Short-term emotion
assessment in a recall paradigm. Int. J. Hum. Comput. Stud. 67 (8), 607–627.
Chanel, G., Rebetez, C., Bétrancourt, M., Pun, T., 2011. Emotion assessment from
physiological signals for adaptation of game difficulty. IEEE Trans. Syst., Man,
Cybernetics-Part A: Syst. Hum. 41 (6), 1052–1063.
Dabbu, S., Malini, M., Reddy, B.R., Vyza, Y.S.R., 2017. ANN based Joint Time and
frequency analysis of EEG for detection of driver drowsiness. Defence Life Sci. J.
2 (4), 406–415.
Dolan, R.J., 2002. Emotion, cognition, and behavior. Science 298 (5596), 1191–1194.
Dragomiretskiy, K., Zosso, D., 2014. Variational mode decomposition. IEEE Trans.
Signal Process. 62 (3), 531–544.
Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zheng, Q., Liu, H.H., 1998. The
empirical mode decomposition and the Hilbert spectrum for nonlinear and non-
stationary time series analysis. Proc. R. Soc. London A 454 (1971), 903–995.
Jatupaiboon, N., Pan-ngum, S., Israsena, P., 2013. Real-time EEG-based happiness
detection system. Sci. World J.
Jiang, L., Zhou, X., Che, L., Rong, S., Wen, H., 2019. Feature extraction and
reconstruction by using 2D-VMD based on carrier-free UWB Radar
application in human motion recognition. Sensors 19 (9), 1962.
Jirayucharoensak, S., Pan-Ngum, S., Israsena, P., 2014. EEG-based emotion
recognition using deep learning network with principal component based
covariate shift adaptation. Sci. World J.
Kalas, M.S., Momin, B.F., 2016. Stress detection and reduction using EEG signals. In:
2016 International Conference on Electrical, Electronics, and Optimization
Techniques (ICEEOT), pp. 471–475.
[Database] Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T.,
Patras, I., 2012. Deap: A database for emotion analysis; using physiological
signals. IEEE Transactions on Affective Computing, 3(1), 18–31.
Lan, Z., Sourina, O., Wang, L., Liu, Y., 2016. Real-time EEG-based emotion monitoring
using stable features. Visual Comput. 32 (3), 347–358.
Lan, Z., Sourina, O., Wang, L., Scherer, R., Müller-Putz, G.R., 2019. Domain adaptation
techniques for EEG-based emotion recognition: a comparative study on two
public datasets. IEEE Trans. Cognitive Dev. Syst. 11 (1), 85–94.
Lang, P.J., 1995. The emotion probe: studies of motivation and attention. Am.
Psychol. 50 (5), 372.
Please cite this article as: P. Pandey and K. R. Seeja, Subject independent emotion recognition from EEG using VMD and deep learning, Journal of King Saud
University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.11.003
View publication stats
9
Lerner, J.S., Li, Y., Valdesolo, P., Kassam, K.S., 2015. Emotion and decision making.
Annu. Rev. Psychol. 66, 799–823.
Li, X., Song, D., Zhang, P., Zhang, Y., Hou, Y., Hu, B., 2018. Exploring EEG features in
cross-subject emotion recognition. Front. Neurosci. 12, 162.
Lin, Y.P., Wang, C.H., Jung, T.P., Wu, T.L., Jeng, S.K., Duann, J.R., Chen, J.H., 2010. EEG-
based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57 (7),
1798–1806.
Lin, Y.P., Yang, Y.H., Jung, T.P., 2014. Fusion of electroencephalographic dynamics
and musical contents for estimating emotional responses in music listening.
Front. Neurosci. 8, 94.
Liu, Y., Sourina, O., 2014. Real-time subject-dependent EEG-based emotion
recognition algorithm. In: Transactions on Computational Science XXIII.
Springer, Berlin, Heidelberg, pp. 199–223.
Liu, Y.H., Wu, C.T., Kao, Y.H., Chen, Y.T., 2013. Single-trial EEG-based emotion
recognition using kernel Eigen-emotion pattern and adaptive support vector
machine. In: Engineering in Medicine and Biology Society (EMBC), 2013 35th
Annual International Conference of the IEEE, pp. 4306–4309.
Masood, N., Farooq, H., 2019. Investigating EEG patterns for dual-stimuli induced
human fear emotional state. Sensors 19 (3), 522.
Mauss, I.B., Robinson, M.D., 2009. Measures of emotion: a review. Cogn. Emot. 23
(2), 209–237.
Mert, A., Akan, A., 2018. Emotion recognition from EEG signals by using multivariate
empirical mode decomposition. Pattern Anal. Appl. 21 (1), 81–89.
Mohamed, M., Quan, L.R., Ahmad, I.L., Chuan, L.C., bt Hamid, S.H., 2012.
Determination of Angry Condition based on EEG, Speech and Heartbeat. Int. J.
Comput. Sci. Eng. 4 (12), 1897.
Mohammadi, Z., Frounchi, J., Amiri, M., 2017. Wavelet-based emotion recognition
system using EEG signal. Neural Comput. Appl. 28 (8), 1985–1990.
Morris, J.D., 1995. Observations: SAM: the Self-Assessment Manikin; an efficient
cross-cultural measurement of emotional response. J. Advertising Res. 35 (6),
63–68.
Pandey, P., Seeja, K.R., 2019. Emotional state recognition with EEG signals using
subject independent approach. In: Mishra, D., Yang, X.S., Unal, A. (Eds.), Data
Science and Big Data Analytics. Lecture Notes on Data Engineering and
Communications Technologies. Springer, Singapore.
Pandey, P., Seeja, K.R., 2019b. Subject-independent emotion detection from EEG
signals using deep neural network. In: Bhattacharyya, S., Hassanien, A., Gupta,
D., Khanna, A., Pan, I. (Eds.), International Conference on Innovative Computing
and Communications. Lecture Notes in Networks and Systems. Springer,
Singapore.
Paul, S., Mazumder, A., Ghosh, P., Tibarewala, D.N., Vimalarani, G., 2015. EEG based
emotion recognition system using MFDFA as feature extractor. In: Robotics,
Automation, Control and Embedded Systems (RACE), International Conference
on, pp. 1–5.
Petrantonakis, P.C., Hadjileontiadis, L.J., 2010. Emotion recognition from EEG using
higher order crossings. IEEE Trans. Inf Technol. Biomed. 14 (2), 186–197.
Picard, R.W., Vyzas, E., Healey, J., 2001. Toward machine emotional intelligence:
analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell.
23 (10), 1175–1191.
Rahi, P.K., Mehra, R., 2014. Analysis of power spectrum estimation using welch
method for various window techniques. Int. J. Emerging Technol. Eng. 2 (6),
106–109.
Rayatdoost, S., Soleymani, M., 2018. Cross-corpus eeg-based emotion recognition.
In: 2018 IEEE 28th International Workshop on Machine Learning for Signal
Processing (MLSP), pp. 1–6.
Read, G.L., Innis, I.J., 2017. Electroencephalography (Eeg). Int. Encyclopedia
Commun. Res. Methods, 1–18.
Russell, J.A., 1980. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178.
Shahabi, H., Moghimi, S., 2016. Toward automatic detection of brain responses to
emotional music through analysis of EEG effective connectivity. Comput. Hum.
Behav. 58, 231–239.
Soleymani, M., Pantic, M., Pun, T., 2012. Multimodal emotion recognition in
response to videos. IEEE Trans. Affective Comput. 3 (2), 211–223.
Wang, X.W., Nie, D., Lu, B.L., 2011. EEG-based emotion recognition using frequency
domain features and support vector machines. In: International Conference on
Neural Information Processing. Springer, Berlin, Heidelberg, pp. 734–743.
Wang, X.W., Nie, D., Lu, B.L., 2014. Emotional state classification from EEG data
using machine learning approach. Neurocomputing 129, 94–106.
Xu, H., Plataniotis, K.N., 2012. Affect recognition using EEG signal. Multimedia
Signal Processing (MMSP), 2012 IEEE 14th International Workshop on (pp. 299-
304).
Zhang, J., Chen, M., Zhao, S., Hu, S., Shi, Z., Cao, Y., 2016. ReliefF-based EEG sensor
selection methods for emotion recognition. Sensors 16 (10), 1558.
Zheng, W.L., Lu, B.L., 2015. Investigating critical frequency bands and channels for
EEG-based emotion recognition with deep neural networks. IEEE Trans.
Autonomous Mental Dev. 7 (3), 162–175.
Zhuang, N., Zeng, Y., Tong, L., Zhang, C., Zhang, H., Yan, B., 2017. Emotion recognition
from EEG signals using multidimensional information in EMD domain. BioMed
Res. Int.