Biomedical Signal Processing and Control 58 (2020) 101756

Contents lists available at ScienceDirect

Biomedical Signal Processing and Control

journal homepage: www.elsevier.com/locate/bspc

EEG-based emotion recognition using simple recurrent units network

and ensemble learning
Chen Wei, Lan-lan Chen ∗ , Zhen-zhen Song, Xiao-guang Lou, Dong-dong Li
Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology,
Shanghai, 200237, PR China

a r t i c l e i n f o a b s t r a c t

Article history: The purpose of this research is to develop an EEG-based emotion recognition system for identification of
Received 25 June 2019 three emotions: positive, neutral and negative. Up to now, various modeling approaches for automatic
Received in revised form 16 October 2019 emotion recognition have been reported. However, the time dependency property during emotion pro-
Accepted 5 November 2019
cess has not been fully considered. In order to grasp the temporal information of EEG, we adopt deep
Available online 8 January 2020
Simple Recurrent Units (SRU) network which is not only capable of processing sequence data but also
has the ability to solve the problem of long-term dependencies occurrence in normal Recurrent Neural
Keywords:
Network (RNN). Before training the emotion models, Dual-tree Complex Wavelet Transform (DT-CWT)
EEG
Emotion recognition
was applied to decompose the original EEG into five constituent sub-bands, from which features were
DT-CWT then extracted using time, frequency and nonlinear analysis. Next, deep SRU models were established
SRU using four different features over five frequency bands and favorable results were found to be related to
Ensemble learning higher frequency bands. Finally, three ensemble strategies were employed to integrate base SRU models
to get more desirable classification performance. We evaluate and compare the performance of shallow
models, deep models and ensemble models. Our experimental results demonstrated that the proposed
emotion recognition system based on SRU network and ensemble learning could achieve satisfactory
identification performance with relatively economic computational cost.
© 2019 Published by Elsevier Ltd.

1. Introduction visual based approaches, physiological signals show relatively high

Emotion is an essential component of being human, and has a
significant impact on people’s daily activities, including communi-
cation, interaction, learning, etc. It is important to distinguish the
emotional states of people around us for natural communication
[1]. Besides, in the field of human-machine interaction (HMI), auto-
matic emotion recognition is also an important and challenging task
[2].
A large number of literatures on technique approaches for emo-
tion recognition have been reported over the past years, and can
mainly be classified into three categories: (1) facial expressions
and voice; (2) periphery physiological signals; (3) brain signals
generated by central nervous system [3]. In these measures, the
audio-visual based detectors that interpret facial expression and
voice realize noncontact recognition of emotion, but they cannot
always return reliable results because people may easily disguise
their emotions without being noticed [4]. Compared with audio-
∗ Corresponding author.
E-mail address: llchen@ecust.edu.cn (L.-l. Chen).
identification accuracy because the users cannot control it. There
are two kinds of physiological changes related to emotion: one is
relevant to Peripheral Nervous System (PNS) and the other is rele-
vant to Central Nervous System (CNS). Usually, features extracted
from peripheral physiological signals such as electrocardiogram
(ECG), skin conductance (SC) and pulse could provide detailed
and complex information for recognizing emotional states [5].
Compared with other peripheral physiological signals, electroen-
cephalogram (EEG) signals captured from central nervous system,
can directly reflect the activity of the brain, and have an intrinsic
relationship with human emotional states [6].
With the rapid progress of wearable devices, flexible dry elec-
trode, machine learning and wide applications of human-machine
interaction, more and more researches and approaches on EEG-
based emotion recognition have been developed. In the previous
studies, researchers extracted various features from raw EEG
signals to evaluate the correlation between the EEG data and
emotional states, such as time analysis, wavelet coefficients and
nonlinear dynamical analysis [7–9]. To reduce the dimension of fea-
tures, Principal Component Analysis (PCA) and Linear Discriminant
Analysis (LDA) are the most widely used methods [10–12]. Besides,

https://doi.org/10.1016/j.bspc.2019.101756
1746-8094/© 2019 Published by Elsevier Ltd.
2 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756

Minimum Redundancy Maximum Relevance (mRMR), Singular

Value Decomposition and QR Factorization have been employed
so as to eliminate the weak and redundant features [13,14]. Some
modeling techniques, such as Naive Bayes (NB), Support Vector
Machine (SVM) and Random Forest (RF) are usually used with man-
ual feature extraction and selection processes together [15,16].
Recent deep learning techniques enable automatic feature
extraction and selection, and have a great influence on signal and
information processing [17]. Many deep learning models, such as Fig. 1. The experimental process of SEED.
Auto-Encoder (AE), Deep Belief Network (DBN) and Convolution
Neural Network (CNN), have been successfully applied into physi-
2. SEED dataset
ological signals processing and achieve promising results compared
with conventional shallow models [10,11]. adopted multimodal
In this paper, emotion recognition research was conducted on
deep learning approach to classify different emotions based on two
SEED dataset [14], in which emotions were stimulated by film clips.
standard emotion recognition datasets (i.e., SEED and DEAP). They
These emotional films were all Chinese films containing scene and
demonstrated that high level representation features extracted by
audio, and could place the subjects in a more realistic scene and
the Bimodal Deep Auto-Encoder (BDAE) were significant for emo-
evoke strong physiological changes. There were totally 15 clips in
tion recognition. Wang et al. [18] used the data augmentation
one experiment, and each clip lasted for about 4 min. There were
method which help mitigate overfitting by generating modified
3 classes of emotions (positive, neutral and negative) estimated in
samples from the training data. After comparing the performance
the dataset, and each emotion had 5 corresponding film clips. In one
of SVM and CNN models (i.e., LeNet and ResNet) on SEED and
session, there was a 5 s hint before each clip, 45 s self-assessment
MAHNOB-HCI datasets, they found that deep models could get bet-
and 15 s rest after each clip. For the self-assessment, subjects were
ter effect with the data augmentation method.
required to complete the questionnaires to feedback their emo-
In most cases, the subject’s emotional states are often treated as
tional experience. Fig. 1 shows the detailed experimental process.
independent points. However, emotion is a changing process with
There were totally 15 subjects (7 males and 8 females;
the time dependency property, which should be taken into con-
23.27 ± 2.37 years old) participating in the experiments. All par-
sideration [19]. Deep learning methods like CNN or AE could not
ticipants were native Chinese students with self-reported normal
grasp the temporal information of EEG. There is another deep learn-
vision and hearing. Each subject executed the experiments 3 times,
ing algorithm called Recurrent Neural Network (RNN), in which
and the experiment interval was one week or longer. EEG was
connections between units form a loop that enables it to process
recorded from a 62-channel electrode cap according to the 10–20
sequence data. RNN has been widely used on machine translation
system at a sampling rate of 1000 Hz. For signal processing, the orig-
and speech recognition [20]. However, standard RNN has the inher-
inal EEG data were first downsampled to 200 Hz and then filtered
ent problem of long-term dependencies. Some special kinds of RNN
to 0.5–70 Hz.
such as Long Short-Term Memory [21] show the ability of solving
this problem and have been applied to process and analyze physi-
ological signals. Tang et al. [22] introduced a Bimodal-LSTM model 3. Methodology
for emotion recognition with multimodal signals. When evalu-
ated on SEED dataset, the Bimodal-LSTM model used EEG and eye The general block diagram of the proposed system is presented
movement features as inputs, while it analyzed EEG and peripheral in Fig. 2. Firstly, signals are pre-processed by filtering, then decom-
physiological signals when examined on DEAP dataset. Li et al. [9] posed into sub-bands using DT-CWT. Secondly, four kinds of feature
first encapsulated the multi-channel signals in DEAP dataset into extraction methods are executed from time, frequency and non-
grid-like7-15: frames by using wavelet and scalogram transform. linear analysis. Thirdly, SRU models are established on different
Then they designed a hybrid deep learning model combining CNN feature sets. Finally, three different ensemble strategies are adopted
and LSTM, aiming to extract task-related features, explore inter- and compared to select the best ensemble model.
channel correlation and integrate contextual information from the
frames. 3.1. Dual-tree complex wavelet transform (DT-CWT)
Recently, an improved version of RNN, namely Simple Recurrent
Units (SRU) has been proposed, which can simplify the computation Put forward by Kingsbury [23], DT-CWT utilizes a dual tree of
and expose more parallelism, so the training time can be reduced wavelet filters to attain the real and imaginary parts of complex
at a large degree. In this paper, we adopt SRU to construct classifi- coefficients. Compared with Discrete Wavelet Transform (DWT),
cation model, which both considers the temporal changing of EEG DT-CWT has the prominent attributes of approximate shift invari-
and computing efficiency to help with the recognition of emotion. ance and superior anti-aliasing [24].
Furthermore, ensemble learning methods are applied to integrate Fig. 3 presents the decomposition for a 3-level DT-CWT. Tree
multiple SRU models so as to obtain better results than using any A and Tree B represent real and imaginary parts of the transform
base model alone. The current work is designed to predict future respectively. The h0 and h1 are the low- and high-pass filter pair
emotion for a single subject after customizing a person-specific for Tree A, while the g0 and g1 are that for Tree B. Besides, (↓2)
model. To validate the efficiency of the proposed machine learning is the down-sampling operator. The sampling location of Tree B is
algorithms, we test the stability of our emotion recognition system always kept in the middle position of Tree A, thus Tree B can exactly
within and cross experimental days. collect the missing information of Tree A. Therefore, Tree A and Tree
The layout of the paper is as follows. Section 2 introduces the B can complement and fulfill each other. This could weaken the
SEED dataset studied in this paper. Section 3 gives a description effect of alternately sampling, restrain the problem of frequency
of our methodology, including DT-CWT, feature extraction meth- aliasing, and make the reconstruction components closer to real
ods, SRU network and ensemble strategies. Section 4 shows the results [25,26].
experimental results. Section 5 reviews similar researches and dis- EEG has five common rhythms, which are Delta (0–4 Hz),
cusses the contributions and limitations of the study. In addition, Theta (4–8 Hz), Alpha (8–13 Hz), Beta (13–30 Hz) and Gamma
conclusions have been made in this section as well. (30–50 Hz). These five rhythms range from 0 to 50 Hz, and con-
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756 3

Fig. 2. General block diagram of emotion recognition based on SRU.

Fig. 3. Schematic of 3-level DT-CWT decomposition.

tain abundant information about basic rhythms of EEG. In this 3.2.2. Frequency analysis
study, 5-level decomposition and reconstruction on EEG are per- On the basis of fast Fourier transform (FFT), the power spectral
formed by DT-CWT, thus six components of z5 (0–3.125 Hz), y5 density (PSD) approach is adopted to obtain the characteristics of
(3.125–6.25 Hz), y4 (6.25–12.5 Hz), y3 (12.5–25 Hz), y2 (25–50 Hz) EEG signals in the frequency domain [7]. The definition of PSD is
and y1 (50–100 Hz) can be acquired. Among them, the ranges of z5, proposed by Wiener-Khintchine theorem. Regarding the signal as
y5, y4, y3 and y2 are close to the frequency ranges of Delta, Theta, a stationary random process, the signal autocorrelation function is
Alpha, Beta and Gamma respectively. Hence, these five sub-bands calculated as:
could represent the corresponding rhythms of EEG.
1
N−1
ˆ
3.2. Feature extraction Rx (m) = x(i)x(i + m) (2)
N
i=0

The aim of feature extraction is to gain the salient features of EEG

under different emotional states [7]. For a comprehensive study, we where m = −(M − 1), ..., −1, 0, 1, ..., M − 1, M ≤ N. Then the PSD
investigate four different kinds of features from the aspects of time, is the Fourier transform of the signal autocorrelation function, and
frequency and nonlinear analysis. could be computed as:
The segment length ought to be chosen reasonably. The longer
the segment is, the more temporal information it reflects, but the 
M−1
ˆ jw ˆ −jwm
fewer samples we obtain in the data pool. To balance the num- S x (e ) = Rx (m)e (3)
ber and length of the segments, we first divided the EEG signals m=−(M−1)

into segments with a length of 5 s. Four feature extraction methods

were carried out every 0.5 s with an overlap rate of 50% respectively,
3.2.3. Nonlinear analysis
thus the sequence length of each segment was 19. In addition,
Fractal dimension (FD) is suitable for measuring the complexity
because 62-channel electrodes were used in this experiment, the
of biological signals [28]. Higuchi method shows an excellent ability
input structure of each segment was 62*19. In the 3-class emo-
in estimating FD, and is used for nonlinear analysis of EEG signals in
tion recognition task, the number of segments in each category was
this paper [25,26]. From the time-series points x(1), x(2), ..., x(N),
approximately equal. The data structure is shown in Fig. 4.
a new time series Xm k can be constructed as follows:

3.2.1. Time analysis N−m

The mean absolute value (MAV) method is employed to extract Xm
k : x(m), x(m + k), x(m + 2k), ..., x(m + int ( )k)
k (4)
the time-domain features of EEG signals [27]. Given a finite set of
time-series points x(1), x(2), ..., x(N), the MAV refers to the average m = 1, 2, ..., k
of the absolute value of the segmented signal, formulated in Eq. (1):
  where m and k represent the initial time and discrete interval time,
1    respectively. The curve length Lm (k) of Xm
N
k is calculated as:
M = log x(i) (1)
N
i=1 int ( N−m )
1 k   N−1
where N is the number of points in the time-windowed signal, x(i) Lm (k) = ( x(m + ik) − x(m + (i − 1)k)) × (5)
k int ( N−m )k
is the value of the ith point, and M is the log-transformed value. i=1 k
4 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756

Fig. 4. The data structure of samples.

Mean value of the curve length L(k) is computed by averaging which have the ability to learn long-term dependencies because of
Lm (k) over k sets for all m. After that, FD could be obtained by: the exclusive design of units [25,26].
For sequences of symbols, recurrent networks process one sym-
log L(k)
FD = − (6) bol at a time. In common RNN architectures, such as LSTM and GRU,
log k the computation in each step is based on completing the previous
Another nonlinear analysis method used in this research is dif- step. So recurrent computations are less suitable to parallelization
ferential entropy (DE). DE is the continuous version of Shannon [32]. Besides, gating method is used in most recurrent architectures
entropy, and could be computed as: to control the information flow to alleviate vanishing and exploding
 gradient problems. In this process, the computation of the network,
especially the matrix multiplication, is the most expensive opera-
h(x) = − f (x) log(f (x))dx (7)
tion. Lei et al. [32] proposed an improved version of RNN named
Simple Recurrent Units (SRU). The key design in SRU is making the
DE feature is simple and efficient for the complexity evalua- gate computation only rely on the current input of the recurrence. In
tion of a continuous random variable. Previous studies have shown this way, only the point-wise multiplication is dependent on pre-
the advantage of DE in characterizing EEG time series [14]. For a vious steps. Thus the matrix multiplications in the feed-forward
fixed length EEG sequence, DE is equal to the logarithm of PSD in network can be easily parallelized. The structure of SRU is shown
a certain frequency band. If a random variable obeys the Gaussian in Fig. 6.
distribution N(,  2 ), the DE feature can simply be computed by The basic form of SRU includes a single forget gate. Given an
the following formulation: input xt at time t, a linear transformation x̃t and the forget gate ft
 ∞ are computed as:
1 (x − )2 1 (x − )2 1
h(x) = − √ exp log √ exp dx = log 2e 2 (8)
−∞ 2 2 2 2
2 2 2 2 2 x̃t = Wxt (10)

where e is the Euler’s constant, and  is the standard deviation of a ft = (Wf xt + bf ) (11)
sequence.
This computation only depends on xt , which realizes computing
it in parallel across all time steps. The forget gate is used to modulate
3.3. Simple recurrent units (SRU) the internal state ct :

Recurrent Neural Network (RNN) has loops inside, and the
information is transmitted from the present loop to the next. The
chain-like property indicates that RNN is the natural structure of
neural network applied to sequences and lists [25,26]. The structure
of standard RNN is shown in Fig. 5.
Given a general input sequence [x1 , x2 , ..., xk ], where xi ∈ Rd , at
each time-step of RNN modeling, a hidden state is generated which
produces a hidden sequence of [c1 , c2 , ..., ck ]. The hidden state at
time-step t is computed from an activation function f of the current
input xt and previous hidden state ct - 1 as:
ct = f (xt , ct−1 )
Then an optional output can be produced by ht = g(ct ), result-
ing in an output sequence [h1 , h2 , ..., hk ], which can be used for
sequence-to-sequence tasks [29]. However, standard RNN cannot
avoid the problem of long-term dependencies, which implies RNN
would lose the capacity to connect information as the gap between
loops increases [30]. Thus, special kinds of RNN are proposed, such
as Long Short-Term Memory [21] and Gated Recurrent Units [31],
ct = ft  ct−1 + (1 − ft )  x̃t (12)
The reset gate is employed to calculate the output state ht as a
combination of ct and xt :
rt = (Wr xt + br ) (13)
ht = rt  tanh(ct ) + (1 − rt )  xt (14)
The complete algorithm also utilizes skip connections to
improve training of deep networks with numerous layers. While
even a naive implementation of the approach brings improvements
(9) in performance, one of the merits is that it can realize optimization
especially for the existing hardware architectures. Eliminating the
dependencies between time steps for the most expensive opera-
tions enables parallelization across different dimensions and time
steps.
When using the artificial neural network to process data, a mul-
tilayer network can usually get more satisfactory effects than a
single-layer network. Deep RNN is a kind of deep network in which
depth is added via a recurrent connection in the hidden layer [33].
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756 5

Fig. 5. The structure of standard RNN.

Fig. 6. The structure of SRU.

Fig. 7. The structure of the 2-level SRU model.

Usually a single-layer network has limited ability to extract abstract reasonably. The more layers the network has, the more training
features. While a multilayer network could produce more easily time it will take. With the comprehensive consideration of effec-
learned representations of input sequences, leading to a better clas- tiveness and efficiency, this paper designs a 2-level SRU model. As
sification accuracy [34]. But the depth of network should be chosen shown in Fig. 7, the deep SRU model consists of an input layer,
6 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756
a sequence-to-sequence SRU layer, a many-to-one SRU layer, a
Fully-Connected (FC) layer and a final soft-max layer for emotion
recognition. We should have adjusted the number of nodes and
hidden layers using grid cross-validation, but in consideration of
the limited computational resources, the hidden layer sizes were
tested on typical parts of data. It was found the network appeared
to work well when the first SRU layer used 200 units while the sec-
ond SRU layer contained 100 units and the FC layer had 50 units.
Meanwhile, the dropout probability after it was set to 50% in order
to avoid overfitting.
3.4. Ensemble learning
Considering that there are four different kinds of features across
five different frequency bands, then twenty groups of base models
are obtained. To further improve the recognition effect, ensemble
learning methods are adopted to make full use of these base models.
Ensemble learning can prevent the overfitting problem of a sin-
gle base classifier, and many ensemble learning algorithms have
been developed so far, such as Boosting, Bagging and Stacking [35].
The base classifiers ought to be designed rather different from each
other. In general, base classifiers can be constructed by using dif-
ferent input features, training sets, and learning algorithms [36].
In this paper, different input features and different frequency
bands are utilized to design base SRU models. Then multiple SRUs
are integrated in order to obtain better classification results than
any single model. We propose three ensemble strategies: (1) Select
the best feature for each of the five frequency bands for ensemble
(i.e., 5 models from the 20 candidate models); (2) Select the best
two frequency bands for each of the four features for ensemble (i.e.,
8 models from the 20 candidate models); and (3) Ensemble all the
20 candidate SRU models. In these strategies, the outputs of the
individual models are combined by majority voting and weighted
averaging methods [37]. The ensemble framework is depicted in
Fig. 8.
Let yi (x) i = 1, 2, ..., K, represents the output of the i-th base
model, and Y (x) is the ensemble result.
(1) Voting
The output class label is determined through majority voting
among base models. That is:

K
Y (x) = sign( yi (x))
i=1
(2) Weighted Averaging
The ensemble result is computed by assigning different weights
to different base models. That is:

K
Y (x) = wi yi (x)
i=1

T a little larger than that of Negative and Neutral states for most
where wi is the weight of yi , and wi ≥ 0 wi = 1. In general, the
i=1
individual model with higher recognition accuracy should be more
weighted. In this paper, wi is chosen based on the accuracy ranking
of the base models:
ki
wi =

K
ki
i=1
In Eq. (15), ki represents the ranking index of the model accu-
racy on validation data, e.g., ki = K when the model has the highest
validation accuracy, while ki = 1 when the model has the lowest
validation accuracy.
4. Results
The proposed system was designed to predict future emotion
for a specific subject in two manners: One was within-one-day
manner where the emotion model was trained and tested using
the data obtained from the same experimental day of one par-
ticipant. According to the experimental sequence, the first nine
sessions were used for model training while the last six sessions
were for testing. The other was cross-day manner where the
emotion model was trained and tested using the data obtained
from different experimental days of one participant. Under both
manner, the validation rate was set to 10% from the training
data.
Each model was trained with 200 epochs, and the optimization
algorithm used for SRU model was Adam [38] with a mini-batch
size of 50. The initial learning rate was 0.001, while the first
and second momentum decay rates were 0.90 and 0.99 respec-
tively. The proposed deep learning method was implemented
on Keras library with TensorFlow backend and run on a PC
with Intel(R) Core(TM) i7-7700HQ CPU @2.80 Hz, 16GB memory,
NVIDIA GeForce GTX 1060 GPU, and 64-bit Windows10 operating
system.
In this section, we aim to examine the SRU model using dif-
ferent features on different frequency bands. After DT-CWT and
feature extraction, raw EEG data were transformed into four dif-
ferent features (MAV, PSD, FD and DE) over five typical frequency
bands (Delta, Theta, Alpha, Beta and Gamma). Then base SRU mod-
els were trained on these samples respectively. Sections 4.1–4.3
report the results evaluated in within-one-day manner and Sec-
tion 4.4 presents the stability of emotion recognition model cross
days.
4.1. Feature analysis with ROC
Receiver operating characteristic (ROC) curve is usually
employed as a visual technique to reveal the association between
true positive rate and false positive rate in pace with the change
of a threshold parameter [39]. Three emotion categories, i.e., neg-
ative, neutral and positive, were divided into three 2-class cases,
i.e., negative to others, neutral to others, and positive to others, as
shown in Fig. 9.
Fig. 10(a)–(d) are ROC curves for MAV, PSD, FD and DE features,
(15)
respectively. Moreover, the area under ROC curve (AUC) was cal-
culated to make comparisons among multiple features and bands.
First, we compared the ROC curves of the MAV feature on different
frequency bands. As shown in Fig. 10(a), the AUCs of Gamma and
Beta bands were larger than other frequency bands, while Gamma
band performed the best. These results reveal that Gamma and
(16) Beta bands are more related with emotion activities than other
frequency bands [40].
Besides, it could be seen that the AUCs of Positive state were
of the frequency bands. Take the MAV feature for instance, for
Negative, Auc Gamma = 0.78, Auc Beta = 0.74, Auc Alpha = 0.63; for
Neutral, Auc Gamma = 0.83, Auc Beta = 0.78, Auc Alpha = 0.65; for
Positive, Auc Gamma = 0.90, Auc Beta = 0.86, Auc Alpha = 0.68. In
some sense, positive class was more recognizable compared with
the other two classes in this experiment. Moreover, when using the
(17) PSD, FD and DE features, we could get the similar conclusion as that
from the MAV feature.
4.2. Classification performance of emotion recognition models
The classification confusion matrices are illustrated in Fig. 11,
which uses the results from MAV feature for demonstration. In
a confusion matrix, each row donates the target class, while
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756
Fig. 8. Detailed framework of the ensemble model.
Fig. 9. Three emotional states divided into three 2-class cases.
each column donates the predicted class. The value in element
(i, j) is the percentage of samples in class i that is classi-
fied as class j. It can be seen that the SRU model achieved
relatively competitive recognition performance on higher fre-
quency bands, such as the best classification accuracy of 79.22%
on Gamma band and 75.91% on Beta band. When tested on
lower frequency bands, the classifier obtained lower accuracy
such as 69.04%, 69.07% and 68.11% on Alpha, Theta and Delta
bands respectively. In addition, an accuracy of 68.67% was got-
ten when trained on samples extracted directly from raw EEG
signals.
When using the PSD, FD and DE features, similar phenomenon
could be observed as that of MAV feature, that was higher frequency
bands had stronger discriminative capacity for emotion recognition
tasks than lower frequency bands. Unlike MAV and PSD features,
the FD and DE features captured from raw signals led to a reason-
able accuracy of 74.33 and 74.35%, which showed the suitability of
FD and DE features to deal with raw signals. In addition, positive
class showed higher correct identification percentage than other
two classes on all the frequency bands. In terms of computational
cost, the average training time of the SRU model was 60.1 s. When
the SRU layers were replaced by LSTM, the training time became
136.5 s. Obviously, the computation speed of SRU was much supe-
rior to that of LSTM.
Further, we compared the SRU with traditional classification
algorithms, such as K-Nearest Neighbor (KNN), Naive Bayes (NB)
and Support Vector Machine (SVM). For these methods, the input
sample structure (62∗19) was first transformed into a column vec-
tor (1178∗1). Then PCA was adopted to reduce the dimension of
features (50∗1). In KNN, the number of nearest neighbors K was
searched in the space [1:20] with a step of one for the optimal value.
In NB, we assumed that each feature obeyed Gaussian distribution,
and the default prior probability was the appearance frequency
of each class. In SVM, we chose linear kernel, and searched the
optimal value for C in the parameter space 2[−10:10] with a step
7
of one. The average classification results of all subjects are shown
in Table 1.
Similar to the results from confusion matrices, the SRU model
obtained obviously better performance on higher frequency bands.
Besides, SRU achieved relatively stable results on features extracted
from raw signals and lower frequency bands. When using the DE
feature, the SRU model led to the best classification performance
on Gamma band with an average accuracy of 80.02%. Compared
with other classifiers, the SRU model attained the best recogni-
tion performance on all the five frequency bands. For the MAV
feature, it obtained the best classification accuracy of 79.22%, which
was 11.34%, 10.98% and 10.04% higher than KNN, NB and SVM
respectively. For the PSD feature, its best classification accuracy
was 78.29%, which was 10.71%, 10.03% and 9.16% higher than KNN,
NB and SVM respectively. For the FD feature, the best classification
accuracy was 77.22%, which was 10.53%, 12.14% and 8.01% higher
than KNN, NB and SVM respectively. For the DE feature, the best
classification accuracy was 80.02%, which was 8.63%, 28.45% and
6.33% higher than KNN, NB and SVM respectively. These results
demonstrated the superiority of the SRU models based on EEG for
the emotion recognition.
4.3. Ensemble results
The final results of the three ensemble strategies are presented
in Table 2. From it, we could learn that the voting or weighted
method of Strategy2, and the weighted method of Strategy3 outper-
formed all individual base models. Strategy1 did not lead to a good
classification performance due to the interference of a few lower
frequency bands; Strategy2 achieved more desirable results than
Strategy1. Specifically, the weighted method of Strategy2 led to the
best classification accuracy of 83.13% among all these three strate-
gies, which was 3.11% higher than the best individual SRU model
(80.02%); Strategy3 using the weighted method also achieved good
classification effects, but the training time was much longer than
8 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756

Fig. 10. The ROC curves of four different features.

the other two strategies. Therefore, the weighted method of Strat-
egy2 was selected as the final ensemble strategy. Different subjects
may have different optimal features and frequency bands, the
ensemble method could lead to a better average value and a smaller
standard deviation for all subjects by combining several base
models.
4.4. Stability of emotion recognition model cross days
In SEED dataset, each participant executed the experiments
three times on different experimental days. By using SEED, we could
evaluate the stability of emotion recognition model cross days. We
splitted the data in three different ways:
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756 9

Fig. 11. The confusion matrices of MAV feature.

Table 1
Classification accuracy (%) comparison of different classifiers.

Feature Frequency band Classifiers

KNN NB SVM SRU

Gamma 67.88 ± 4.78 68.24 ± 6.85 69.18 + 10.84 79.22 ± 8.09

Beta 60.14 ± 2.55 63.55 ± 7.43 66.76 ± 7.27 75.91 ± 8.15
Alpha 44.92 ± 8.28 39.46 ± 6.48 49.23 ± 7.37 69.04 ± 5.35
MAV
Theta 44.02 ± 7.53 43.73 ± 1.36 45.91 ± 7.46 69.07 ± 3.23
Delta 44.61 ± 7.34 43.76 ± 2.63 45.33 ± 4.45 68.11 ± 2.94
Raw Signal 49.04 ± 5.97 42.57 ± 7.63 57.76 ± 9.92 68.67 ± 7.77
Gamma 67.58 ± 10.47 68.26 ± 6.76 69.13 ± 12.10 78.29 ± 8.01
Beta 60.26 ± 12.42 61.43 ± 9.46 67.27 ± 11.15 76.42 ± 7.91
Alpha 45.45 ± 9.13 40.82 ± 6.85 49.09 ± 9.28 69.29 ± 4.96
PSD
Theta 44.70 ± 10.66 45.59 ± 2.12 46.10 ± 7.14 68.32 ± 2.56
Delta 44.82 ± 9.85 43.78 ± 3.32 46.13 ± 7.74 67.46 ± 8.62
Raw Signal 51.10 ± 8.48 48.01 ± 0.36 59.95 ± 11.26 68.09 ± 2.33
Gamma 66.69 ± 7.93 65.08 ± 6.63 69.21 ± 6.42 77.22 ± 7.31
Beta 59.73 ± 3.65 55.26 ± 6.79 65.74 ± 3.58 75.52 ± 6.45
Alpha 44.63 ± 2.79 39.74 ± 9.36 48.04 ± 2.03 69.11 ± 4.12
FD
Theta 44.43 ± 9.63 44.43 ± 3.52 45.57 ± 8.37 68.14 ± 2.75
Delta 44.87 ± 1.69 41.27 ± 2.41 44.84 ± 2.06 68.15 ± 2.34
Raw Signal 58.86 ± 10.25 63.16 ± 3.46 68.29 ± 3.05 74.33 ± 7.77
Gamma 71.39 ± 4.56 50.76 ± 4.63 73.69 ± 7.44 78.86 ± 8.09
Beta 68.52 ± 9.75 51.57 ± 5.12 71.43 ± 1.48 80.02 ± 8.15
Alpha 59.89 ± 5.28 46.56 ± 6.31 64.68 ± 8.13 71.52 ± 5.35
DE
Theta 61.54 ± 4.69 46.07 ± 2.12 63.72 ± 4.36 69.85 ± 3.23
Delta 62.08 ± 7.65 44.19 ± 6.57 66.23 ± 9.58 70.05 ± 2.94
Raw Signal 63.69 ± 4.39 46.18 ± 3.42 66.95 ± 9.36 74.35 ± 7.77

The best performance for each feature and each frequency is in bold.

Type I: The data from the first experimental day of one par-
ticipant was used as training data and the data from the second
experimental day was used as testing data.
Type II: The data from the first experimental day of one par-
ticipant was used as training data and the data from the third
experimental day was used as testing data.
Type III: The data from the first and second experimental day
of one participant was used as training data and the data from the
third experimental day was used as testing data.
The average results from 15 participant were presented in
Table 3, where differential entropy (DE) in Gamma band was used
as feature for comparison. Initially, we intuitively thought that the
10 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756
Table 2
Classification comparison of different ensemble strategies.
Ensemble Strategy Method Accuracy (%)
Voting 77.12 ± 3.22
Strategy1
Weighed 78.84 ± 1.88
Voting 81.82 ± 2.13
Strategy2
Weighted 83.13 ± 1.67
Voting 80.21 ± 4.23
Strategy3
Weighed 81.53 ± 2.22
The best performance for different ensemble strategies is in bold.
Table 3
The average accuracies (%) of our emotion model within and cross days.
Stats. Type I
Mean 79.03
Std. 7.53
Differential entropy (DE) in Gamma band is used for comparison in Table 3.
identification accuracy in within-one-day manner would be higher
than that in the cross-day manner. In fact, we could find that the
recognition accuracy was relatively stable within and cross days,
which reflected that the relation between the variation of emo-
tional states and the EEG signal was relatively stable for one person
over time. There is a reason why the recognition accuracy under
within-one-day manner is not very high. The movie chips used to
stimulate emotion in the same experimental day originated from
different films. So using the first nine movie sessions to model and
the last six to test is a cross-film modeling in some sense. Consid-
ering EEG signal is task-related and sensitive to different emotion
stimulus, the cross-task recognition accuracy of 78.86% within sin-
gle day is acceptable.
5. Discussions and conclusions
5.1. Comparisons with similar work
This research presented an analysis of multi-domain and multi-
frequency features of EEGs related to emotion changes. Simple
Recurrent Units Network and ensemble learning methods were uti-
lized to construct an automatic recognition system to distinguish
three different emotional states. Several fundamental conclusions
can be drawn: 1) Higher frequency bands like Gamma and Beta, are
more favorable for emotion recognition than other lower frequency
bands. 2) Stimulated by scene and audio materials in the experi-
ments, positive emotion is relatively more recognizable compared
with the other two emotional states by using EEG measurements.
3) As an improved kind of RNN, SRU Network is good at grasping
the temporal changing property under different emotions by using
sequential data. 4) Efficient SRU Network and ensemble learn-
ing method show high identification performance and acceptable
processing efficiency in automatic recognition of emotion. 5) The
performance of our emotion system shows that the neural patterns
are relatively stable within and cross days.
A brief summary of emotion recognition based on EEG is shown
in Table 4, and the numbers in parenthesis denote the number of
levels for each emotional dimension. These researches present the
feasibility and availability of establishing emotion models using
EEG. In these studies, the stimuli materials used in the experi-
ments contain images, music and videos. Early studies often used
pictures as emotion elicitation materials. For example, Heraz and
Frasson [41], Brown et al. [42] and Jenke et al. [43] used IAPS
(International Affective Picture System) to implement emotion
experiments. Later, researchers applied music to stimulate corre-
sponding emotional states. Lin et al. [44] and Hadjidimitriou and
Hadjileontiadis [45] operated emotion recognition based on music
Precision (%) Recall (%) F-score(%) Time (s)
76.21 ± 4.34 76.43 ± 3.43 75.51 ± 1.76
315.4
76.91 ± 5.32 78.12 ± 3.73 77.44 ± 4.38
80.71 ± 4.23 80.94 ± 6.48 80.21 ± 2.89
378.5
82.24 ± 3.64 81.53 ± 3.25 81.24 ± 3.65
79.54 ± 3.34 79.12 ± 3.95 78.81 ± 1.96
936.1
81.11 ± 1.58 80.93 ± 3.86 80.54 ± 3.86
Type II Type III Within-one -day
77.35 79.62 78.86
6.73 7.04 8.09
stimulus. Compared with pictures and music, videos are composed
of scenes and audio and can supply more real-life scenarios to sub-
jects. Thus, videos have become popular in emotion recognition
researches. Some famous datasets for emotion recognition, such
as DEAP [14–16,25,46] and SEED [14,40,47] have been established,
where the emotions are all elicited using videos.
Traditional classification algorithms along with feature selec-
tion or dimensionality reduction methods are extensively used to
classify different emotional states, and have achieved fine classifi-
cation effects. Zheng et al. [14] operated their research on DEAP and
SEED datasets based on traditional machine learning algorithms.
They applied Differential Entropy (DE), Linear Dynamic System
(LDS), Minimal Redundancy Maximal Relevance (MRMR), and Dis-
criminative Graph regularized Extreme Learning Machine (GELM)
to feature extraction, smoothing, selection and pattern recogni-
tion, respectively. Besides, they investigated the stability of neural
patterns over time. In our proposed framework, we adopted deep
learning method, which did not need the step of feature selec-
tion or dimensionality reduction, thus could simplify the process
of emotion recognition. We also investigated the stability of neural
patterns over different sessions and different experimental days.
In recent years, deep learning algorithms are gradually used to
solve emotion recognition problems. Zheng and Lu [40] introduced
a deep learning algorithm named Deep Belief Network (DBN) for
emotion recognition on SEED dataset. DBN models were trained
with differential entropy (DE) features, then critical frequency
bands and channels were selected according to the weight dis-
tributions of the trained DBN models. At last, they designed four
different profiles, and obtained the best effect with the profile of
the 12 channels. Li et al. [47] first organized DE features from dif-
ferent channels to form two-dimensional maps. Then they trained
the Hierarchical Convolutional Neural Network (HCNN) to identify
different emotions. HCNN yielded the highest accuracy of 88.2% on
Gamma band. They also confirmed that the high-frequency bands
Gamma and Beta were the optimum bands for emotion processing.
In our experimental settings, we also investigated critical frequency
bands, but we did not select critical channels. DT-CWT technique
was adopted to decompose EEG into five sub-bands, so as to study
the discriminative capacity of different frequency bands. In addi-
tion, we have considered the temporal characteristics of EEG, and
transformed the features into a sequence format. Compared with
DBN and HCNN, our SRU model is another deep learning method
with different specialties.
Considering the temporal, spatial and frequency characteristics
of EEG signals, Alhagry et al. [46] and [25,26] took the advan-
tage of LSTM to distinguish different emotion classes on DEAP
dataset, respectively. Alhagry et al. [46] used LSTM to learn features
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756 11

Table 4
Comparison of the existing emotion recognition systems.

Authors Stimuli Channels Subjects Method Description Emotional States Performance (Accuracy)

Heraz and Frasson IAPS 2 17 Amplitudes of four frequency Valence (12), arousal (12) Valence: 74%, arousal: 74%
[41] bands, evaluated KNN, Bagging and dominance (12) and dominance: 75%
Brown et al. [42] IAPS 8 11 Spectral power features, KNN Positive, negative and 85%
neutral
Jenke et al. [43] IAPS 64 16 Higher Order Crossings, Higher Happy, curious, angry, sad 36.8%
Order Spectra and and quiet
Hilbert-Huang Spectrum
features, QDA
Lin et al. [44] Music 24 26 Power spectral density and Joy, anger, sadness, and 82.29%
asymmetry features of five pleasure
frequency bands, evaluated
SVM
Hadjidimitriou and Music 14 9 Time-frequency analysis, KNN, Like and dislike 86.52%
Hadjileontiadis QDA and SVM
[45]
Koelstra et al. [15] Video 32 32 Spectral power features of four Valence (2), arousal (2) and Valence:57.6%,
frequency bands, Gaussian liking (2) arousal:62.0% and liking:
naive Bayes classifier 55.4%
Gupta et al. [16] Video 32 32 Graph-theoretic features, Valence (2), arousal (2), Valence: 67%, arousal: 69%,
SVM/RVM dominant (2) and liking (2) dominant: 65% and liking:
65%
Zheng et al. [40] Video 62 15 Differential entropy features of Negative, neutral and 86.65%
five frequency bands, DBN positive
32 32 Differential entropy features of Quadrants of VA space (4) 69.67%
Zheng et al. [14] Video
62 15 four/five frequency bands, LDS, Negative, neutral and 91.07%
MRMR, GELM positive
Alhagry et al. [46] Video 32 32 Raw EEG signal, LSTM Valence (2), arousal (2) and Valence: 85.45%, arousal:
liking (2) 85.65% and liking: 87.99%
Li et al. [25] Video 32 32 Rational asymmetry features of Valence (2) 76.67%
four frequency bands, LSTM
Li et al. [47] Video 62 15 Organize the differential Negative, neutral and 88.2%
entropy features as positive
two-dimensional maps, HCNN

from every 5 s-length EEG data, then the dense layer was applied
to classify these features. This method led to an average accu-
racy of 85.65%, 85.45% and 87.99% with low/high arousal, valence
and liking classes, respectively [25,26]. extracted Rational Asym-
metry (RASM) features from every 63s-length signal to extract
the frequency-space domain features of EEG signals. Then LSTM
was constructed as the classifier and achieved a mean accuracy of
76.67% with low/high valence classes. In our research, SRU models
were established instead of LSTM networks, which could realize the
computation in parallel and achieved obviously improved compu-
tational efficiency.
• Regard emotion as a changing process with the time dependency
property.
Most of the shallow learning and deep learning methods
treated emotional states as independent points and ignored the
time dependency property during emotion process. In order
to grasp the temporal information of EEG, we adopted Sim-
ple Recurrent Units (SRU) Network which was not only capable
of processing sequence data but also had the ability to solve
the problem of long-term dependencies occurrence in normal
RNN network.

5.2. Merits of the proposed system
Our research constructed an automatic emotion recognition
system based on Simple Recurrent Units Network and ensemble
learning. The main contributions of this paper can be summarized
as follows:
• Comprehensively extract physiological features of EEG from
multi-domains and multi-frequency bands.
Considering the brain as a sort of highly complex system, the
current study evaluated the characteristics of brain signals from
various aspects: time analysis from mean absolute value (MAV),
frequency analysis from power spectral density (PSD) and nonlin-
ear analysis from fractal dimension (FD) and differential entropy
(DE). Besides, raw EEG signals were decomposed and reconstructed
using dual-tree complex wavelet transform (DT-CWT), which could
be viewed as an improved kind of wavelet transform and had better
expression capacity for EEG due to its superiorities of approx-
imate shift invariance and excellent reconstruction. Our results
demonstrated that the features extracted from sub-bands espe-
cially higher frequency bands provided more accurate information
than those from original signals.
• Obtain competitive classification performance with low compu-
tational cost.
Superior to common RNN, SRU realizes computing in parallel,
thus it can accelerate the calculation at a large degree. Furthermore,
ensemble learning methods using three different strategies were
applied to integrate multiple SRU models so as to obtain better
results than using any base model alone.
• Explore the stability of our emotion recognition system over time.
The dataset used in this paper consisted of 15 participants
and each one performed the experiments three times on dif-
ferent experimental days. The performance of the proposed
subject-specific emotion models were evaluated in with-one-
day and cross-day two manners. The neural patterns of
EEG signals over time for emotion recognition were fully
explored.
5.3. Limitations and future work
The limitations of the current work and corresponding direc-
tions of future research could be summarized as follows:
12 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756

• The SRU network parameters, such as the number of nodes References

and parameters of training algorithms, were mostly selected
depending on experience or trial-and-error method. Thus, a [1] R.W. Picard, Affective Computing, MIT Press, 2000.
[2] J. Wagner, N.J. Kim, E. Andre, From physiological signals to emotions:
more systematic method for selecting the appropriate param- implementing and comparing selected methods for feature extraction and
eters would better to be developed. classification, in: 2005 IEEE International Conference on Multimedia and
• The machine learning model constructed in this paper was Expo, IEEE Computer Society, 2005, pp. 940–943.
[3] K.H. Kim, S.W. Bang, S.R. Kim, Emotion recognition system using short-term
subject-specific. The well-learned parameters are appropriate for monitoring of physiological signals, Med. Biol. Eng. Comput. 42 (3) (2004)
one subject, but may not suitable for others. This may limit the 419–427.
applicability of the proposed emotion recognition methods. So [4] M.J. Black, Y. Yacoob, Recognizing facial expressions in image sequences using
local parameterized models of image motion, Int. J. Comput. Vis. 25 (1) (1997)
it is worth developing a generic (subject-independent or cross- 23–48.
subject) classifier for all subjects. [5] J.F. Brosschot, J.F. Thayer, Heart rate response is longer after negative
• Compared with traditional machine learning methods, deep emotions than after positive emotions, Int. J. Psychophysiol. 50 (3) (2003)
181–187.
learning is more dependent on powerful computation and will
[6] X.W. Li, B. Hu, T.S. Zhu, J.Z. Yan, F. Zheng, Towards affective learning with an
spend much more time on training in general. In order to EEG feedback approach, in: Acm International Workshop on Multimedia
save the computational cost, we can consider introducing some Technologies for Distance Learning, ACM, 2009, pp. 33–38.
pre-training models or combining transfer learning methods to [7] X.W. Wang, D. Nie, B.L. Lu, Emotional state classification from EEG data using
machine learning approach, Neurocomputing 129 (4) (2014) 94–106.
accelerate the model training. [8] L.L. Chen, J. Zhang, J.Z. Zou, C.J. Zhao, G.S. Wang, A framework on
wavelet-based nonlinear features and extreme learning machine for epileptic
seizure detection, Biomed. Signal Process. Control 10 (2014) 1–10.
Acknowledgments [9] X. Li, D. Song, P. Zhang, G.L. Yu, Y.X. Hou, B. Hu, Emotion recognition from
multi-channel EEG data through convolutional recurrent neural network, in:
This work is partly supported by National Natural Science Foun- 2016 IEEE International Conference on Bioinformatics and Biomedicine
(BIBM), IEEE, 2016, pp. 352–359.
dation of China (Nos. 61976091 and 61806078) and Fundamental [10] J.X. Liu, H.Y. Meng, A.K. Nandi, M.Z. Li, Emotion detection from EEG
Research Funds for the Central Universities (Nos. 222201917006). recordings, in: International Conference on Natural Computation, IEEE, 2016,
pp. 1722–1727.
[11] W. Liu, W.L. Zheng, B.L. Lu, Emotion recognition using multimodal deep
Declaration of Competing Interest learning, in: International Conference on Neural Information Processing,
Springer, 2016, pp. 521–529.
[12] L.L. Chen, Y. Zhao, P.F. Ye, J. Zhang, J.Z. Zou, Detecting driving stress in
There is no conflict of interest in this work.
physiological signals based on multimodal feature analysis and kernel
classifiers, Expert Syst. Appl. 85 (2017) 279–291.
Appendix A. A list of acronyms [13] S.N. Daimi, G. Saha, Classification of emotions induced by music videos and
correlation with participants’ rating, Expert Syst. Appl. 41 (13) (2014)
6057–6065.
A summary of acronyms used in this research is listed in [14] W.L. Zheng, J.Y. Zhu, B.L. Lu, Identifying Stable Patterns over Time for Emotion
Table A1. Recognition from EEG, arXiv preprint arXiv:1601.02197, 2016.
[15] S. Koelstra, C. Muhl, M. Soleymani, DEAP: a database for emotion analysis
using physiological signals, IEEE Trans. Affect. Comput. 3 (1) (2012) 18–31.
[16] R. Gupta, K.U. Laghari, T.H. Falk, Relevance vector classifier decision fusion
Table A1
and EEG graph-theoretic features for automatic affective state
A summary of acronyms.
characterization, Neurocomputing 174 (2015) 875–884.
Acronym Full name [17] X. Chai, Q.S. Wang, Y.P. Zhao, X. Liu, O. Bai, Y.Q. Li, Unsupervised domain
adaptation techniques based on auto-encoder for non-stationary EEG-based
AE Auto-encoder emotion recognition, Comput. Biol. Med. 79 (2016) 205–214.
AUC The area under ROC curve [18] F. Wang, S.H. Zhong, J.F. Peng, J.M. Jiang, Y. Liu, Data augmentation for
BDAE Bimodal deep auto-encoder EEG-based emotion recognition with deep convolutional neural networks, in:
CNN Convolution neural network Conference on Multimedia Modeling, Springer, 2018, pp. 82–93.
CNS Central nervous system [19] N. Zhang, W.L. Zheng, W. Liu, B.L. Lu, Continuous vigilance estimation using
LSTM neural networks, in: International Conference on Neural Information
DBN Deep belief network
Processing, Springer, 2016, pp. 530–537.
DE Differential entropy
[20] L. Deng, J.Y. Li, J.T. Huang, K.S. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X.D. He,
DT-CWT Dual-tree complex wavelet transform
J. Williams, Y.F. Gong, A. Acero, Recent advances in deep learning for speech
DWT Discrete wavelet transform research at microsoft, in: International Conference on Acoustics, Speech, and
ECG Electrocardiogram Signal Processing (ICASSP), IEEE, 2013, pp. 8604–8608.
EEG Electroencephalogram [21] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8)
FC Fully-connected (1997) 1735–1780.
FD Fractal dimension [22] H. Tang, W. Liu, W.L. Zheng, B.L. Lu, Multimodal emotion recognition using
FFT Fast Fourier transform deep neural networks, in: International Conference on Neural Information
GELM Discriminative graph regularized extreme learning machine Processing, Springer, 2017, pp. 811–819.
HCNN Hierarchical convolutional neural network [23] N.G. Kingsbury, The dual-tree complex wavelet transform: a new technique
HMI Human-machine interaction for shift invariance and directional filters, IEEE Digital Signal Processing
IAPS International affective picture system Workshop 98 (1998) 2–5.
KNN K-nearest neighbor [24] U. Bal, Dual tree complex wavelet transform based de-noising of optical
microscopy images, Biomed. Opt. Express 3 (12) (2012) 3231–3239.
LDA Linear discriminant analysis
[25] M.Y. Li, W.Z. Chen, T. Zhang, Automatic epileptic EEG detection using
LDS Linear dynamic system
DT-CWT-based non-linear features, Biomed. Signal Process. Control 34 (2017)
MAV Mean absolute value 114–125.
MRMR Minimum redundancy maximum relevance [26] Z.Q. Li, X. Tian, L. Shu, X.M. Xu, B. Hu, Emotion recognition from EEG using
NB Naive Bayes RASM and LSTM, in: International Conference on Internet Multimedia
PCA Principal component analysis Computing and Service, Springer, 2017, pp. 310–318.
PNS Peripheral nervous system [27] H.M. Shim, H. An, S. Lee, E.H. Lee, H.K. Min, S. Lee, EMG pattern classification
PSD Power spectral density by split and merge deep belief network, Symmetry 8 (12) (2016)
RASM Rational asymmetry 148.
RF Random forest [28] U.R. Acharya, S.V. Sree, C.A.A. Peng, R. Yanti, J.S. Suri, Application of non-linear
RNN Recurrent neural network and wavelet based features for the automated identification of epileptic EEG
ROC Receiver operating characteristic signals, Int. J. Neural Syst. 22 (2) (2012) 565–579.
SC Skin conductance [29] X.Y. Zhang, F. Yin, Y.M. Zhang, C.L. Liu, Y. Bengio, Drawing and recognizing
Chinese characters with recurrent neural network, IEEE Trans. Pattern Anal.
SRU Simple recurrent units
Mach. Intell. 40 (4) (2018) 849–862.
SVM Support vector machine
 C. Wei, L.-l. Chen, Z.-z. Song et al. / Biomedical Signal Processing and Control 58 (2020) 101756
[30] Y. Bengio, P.Y. Simard, P. Frasconi, Learning long-term dependencies with
gradient descent is difficult, IEEE Trans. Neural Netw. 5 (2) (1994)
157–166.
[31] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk,
Y. Bengio, Learning Phrase Representations Using RNN Encoder-Decoder for
Statistical Machine Translation, arXiv preprint arXiv:1406.1078,
2014.
[32] T. Lei, Y. Zhang, S.I. Wang, H. Dai, Y. Artzi, Simple Recurrent Units for Highly
Parallelizable Recurrence, arXiv preprint arXiv:1709.02755, 2017.
[33] R.G. Hefron, B.J. Borghetti, J.C. Christensen, C.M. Kabban, Deep long short-term
memory structures model temporal dependencies improving cognitive
workload estimation, Pattern Recognit. Lett.
94 (2017) 96–104.
[34] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016.
[35] A. Ozcift, A. Gulten, Classifier ensemble construction with rotation forest to
improve medical diagnosis performance of machine learning algorithms,
Comput. Methods Programs Biomed. 104 (3) (2011)
443–451.
[36] C. Padilha, D.A. Barone, A.D. Neto, A multi-level approach using genetic
algorithms in an ensemble of least squares support vector machines, Knowl.
Based Syst. 106 (2016) 85–95.
[37] J.H. Zhang, S.N. Li, R.B. Wang, Pattern recognition of momentary mental
workload based on multi-channel electrophysiological data and ensemble
convolutional neural networks, Front. Neurosci. 11 (2017) 1–16.
[38] D.P. Kingma, J.L. Ba, Adam: a Method for Stochastic Optimization, arXiv
preprint arXiv:1412.6980, 2014.
13
[39] L.L. Chen, Y. Zhao, J. Zhang, J.Z. Zou, Automatic detection of
alertness/drowsiness from physiological signals using wavelet-based
nonlinear features and machine learning, Expert Syst. Appl. 42 (21) (2015)
7344–7355.
[40] W.L. Zheng, B.L. Lu, Investigating critical frequency bands and channels for
EEG-based emotion recognition with deep neural networks, IEEE Trans.
Auton. Ment. Dev. 7 (3) (2015) 162–175.
[41] A. Heraz, C. Frasson, Predicting the three major dimensions of the learner’s
emotions from brainwaves, World Acad. Sci. Eng. Technol. Int. J. Comput.
Electr. Autom. Control Inf. Eng. 1 (7) (2007) 1988–1994.
[42] L. Brown, B. Grundlehner, J. Penders, Towards wireless emotional valence
detection from EEG, in: International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC), IEEE, 2011, pp. 2188–2191.
[43] R. Jenke, A. Peer, M. Buss, Feature extraction and selection for emotion
recognition from EEG, IEEE Trans. Affect. Comput. 5 (3) (2014) 327–339.
[44] Y.P. Lin, C.H. Wang, T. Jung, T.L. Wu, S. Jeng, J. Duann, J. Chen, EEG-Based
emotion recognition in music listening, IEEE Trans. Biomed. Eng. 57 (7) (2010)
1798–1806.
[45] S. Hadjidimitriou, L.J. Hadjileontiadis, Toward an EEG-based recognition of
music liking using time-frequency analysis, IEEE Trans. Biomed. Eng. 59 (12)
(2012) 3498–3510.
[46] S. Alhagry, A.A. Fahmy, R.A. ElKhoribi, Emotion recognition based on EEG
using LSTM recurrent neural network, Int. J. Adv. Comput. Sci. Appl. (IJACSA) 8
(10) (2017) 355–358.
[47] J.P. Li, Z.X. Zhang, H.G. He, Hierarchical convolutional neural networks for
EEG-based emotion recognition, Cognit. Comput. 10 (2) (2018) 368–380.