Computers in Biology and Medicine 146 (2022) 105606

Contents lists available at ScienceDirect

Computers in Biology and Medicine

journal homepage: www.elsevier.com/locate/compbiomed

EEG emotion recognition based on enhanced SPD matrix and manifold

dimensionality reduction
Yunyuan Gao a, *, Xinyu Sun a, Ming Meng a, Yingchun Zhang b
a
College of Automation, Hangzhou Dianzi University, Hangzhou, China
b
Department of Biomedical Engineering, University of Houston, Houston, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Recently, Riemannian geometry-based pattern recognition has been widely employed to brain computer inter­
EEG face (BCI) researches, providing new idea for emotion recognition based on electroencephalogram (EEG) signals.
Emotion recognition Although the symmetric positive definite (SPD) matrix manifold constructed from the traditional covariance
Riemannian manifold
matrix contains large amount of spatial information, these methods do not perform well to classify and recognize
SPD matrix
Dimensionality reduction
emotions, and the high dimensionality problem still unsolved. Therefore, this paper proposes a new strategy for
EEG emotion recognition utilizing Riemannian geometry with the aim of achieving better classification perfor­
mance. The emotional EEG signals of 32 healthy subjects were from an open-source dataset (DEAP). The wavelet
packets were first applied to extract the time-frequency features of the EEG signals, and then the features were
used to construct the enhanced SPD matrix. A supervised dimensionality reduction algorithm was then designed
on the Riemannian manifold to reduce the high dimensionality of the SPD matrices, gather samples of the same
labels together, and separate samples of different labels as much as possible. Finally, the samples were mapped to
the tangent space, and the K-nearest neighbors (KNN), Random Forest (RF) and Support Vector Machine (SVM)
method were employed for classification. The proposed method achieved an average accuracy of 91.86%,
91.84% on the valence and arousal recognition tasks. Furthermore, we also obtained the superior accuracy of
86.71% on the four-class recognition task, demonstrated the superiority over state-of-the-art emotion recognition
methods.

1. Introduction
Emotion is a complex psychological and physiological state [1],
which plays an important role in daily life, especially for interpersonal
communication. In recent years, with the continuous improvement of
human-computer communication requirements [2], many researchers
began to focus on emotion recognition technology [3]. Through emotion
recognition, we can establish an automatic system that can provide
correct feedback according to different emotions, so as to make
human-computer interaction more friendly [4]. In the past decades,
many emotion recognition methods based on different
non-physiological patterns have been proposed, such as facial expres­
sion, gesture, and sounds [5]. Compared to the non-physiological sig­
nals, physiological signals [6] such as electroencephalogram (EEG),
electromyogram (EMG), and electrocardiogram (ECG) can be detected
by wearable devices. Among these physiological signals, EEG signals
have been widely utilized for researches of emotion recognition. The
low-cost, portable, and high temporal resolution EEG signals make it
possible to capture the emotion immediately once the participant re­
ceives the emotional stimulus [7].
Lately, various classification strategies have been proposed for
emotion recognition based on EEG signals [8]. These methods usually
extract features from the EEG signals and construct classifiers to group
them into different emotions. Conventional EEG feature extraction
methods usually extract features from the time domain or frequency
domain of EEG signal, such as event-related potential (ERP) [9], statis­
tical features (fractal dimension and first-order difference) [10], power
spectral density, autoregressive coefficient, etc. In the construction of
classifiers, machine learning algorithms such as naïve Bayes, support
vector machine (SVM), linear discriminant analysis (LDA) [11], and
bagging ensemble [12] have been widely used. Subasi et al. [13]
extracted features by tunable Q wavelet transform (TQWT), and
employed a rotation forest ensemble classifier on emotion classification.
J. Atkinson et al. [14] extracted features of statistical features,

* Corresponding author. College of Automation, Hangzhou Dianzi University, Hangzhou, China.

E-mail address: gyy@hdu.edu.cn (Y. Gao).

https://doi.org/10.1016/j.compbiomed.2022.105606
Received 26 March 2022; Received in revised form 8 May 2022; Accepted 9 May 2022
Available online 13 May 2022
0010-4825/© 2022 Elsevier Ltd. All rights reserved.
Y. Gao et al.
bandwidth power and fractal dimension, and used SVM to recognize
emotions. Tuncer et al. [15] proposed a novel fractal pattern feature
extraction approach, and established a high-performance emotion
recognition model by using tunable Q-factor wavelet transform.
Abdullah et al. [16] proposed a new prime pattern network to generate
features. Excellent classification performance is achieved on multiple
datasets by using prime pattern and statistical feature generators on the
generated sub-bands and raw signals to obtain multiple features. Wang
et al. [17] investigated brain-directed connectivity (BDC) networks for
emotion recognition using electroencephalographic (EEG) source sig­
nals estimated from high-density sensor EEG signals. Effectively
improve the performance of emotion recognition by combining BDC
with Differential entropy (DE) features.
In recent years, Riemannian geometry-based classification algo­
rithms have been applied in brain computer interface (BCI) studies [18].
Barachant et al. [19] first used the Riemannian method to classify
MI-EEG data and proposed the minimum distance to Riemannian mean
(MDRM) algorithm to classify MI tasks. By mapping the symmetric
positive definite (SPD) matrix to the Euclidean space, the existing vector
methods can be used. Barachant et al. [20] mapped the SPD matrix onto
the tangent space, and used LDA in the space, the results showed that it
could significantly improve the accurate of MI recognition. Wang et al.
[21] combined the SPD matrix with domain adaptation to capture the
internal emotional representation shared by different subjects, effec­
tively avoiding interindividual variability. Tang et al. [22] proposed to
extend the generalized learning vector quantization (GLVQ) method to
the Riemannian manifold. Experimental research on simulated data and
MI-EEG data showed better performance of the Riemann GLVQ. For the
high dimensionality problem of samples in Riemannian space, Xu et al.
[23] used filters in the tangent space to effectively reduce the sample
dimensionality and the calculation time, and avoid the complexity of
high dimensionality matrix optimization.
Riemannian geometry provides some Riemannian metrics that can
directly apply the spatial information in the covariance matrix without
source separation [25]. Riemannian metric can suppress noise well, and
it has been proved to be an effective method to measure the distance
between samples in the field of brain-computer interface. In particular,
the Riemannian mean is very robust to outliers. These properties are
suitable for processing emotional EEG features. However, the changes of
emotional state involve complex brain activities, and the characteristics
of EEG signals are distributed in the time-frequency spatial domain at
the same time. Using only the spatial features provided by the SPD
matrix for EEG emotion recognition results in poor classification per­
formance [26]. Few studies combine time-frequency domain and spatial
domain features and use the Riemannian geometric framework for
emotion recognition, so the main contributions of this paper are as
follows:
1. Aiming at the complex EEG signals generated by the changes of
different emotions, we propose an EEG recognition method based on
Riemannian geometry. It takes the SPD matrix obtained from the
original EEG signal as samples and use the Riemannian geometric
properties of the SPD matrix for emotion recognition.
2. Most researches on emotion recognition are based on the time-
domain or frequency-domain signal extraction of EEG features,
while spatial features that contain significant information related to
emotional states are usually ignored. We design an enhanced SPD
matrix that effectively fuses time-frequency and spatial features to
provide more information for emotion recognition.
3. The SPD matrix dimensionality will be higher as the number of
channels increases, especially when the SPD matrix is mapped to the
tangent space, which will cause the curse of dimensionality [24].
Thus, we propose a supervised dimensionality reduction strategy on
the Riemannian manifold to solve the high dimensionality problem
of the SPD matrix. While reducing the dimension, the same labels of
samples are gathered together and different labels of samples are
2
Computers in Biology and Medicine 146 (2022) 105606
Fig. 1. Tangent space of the manifold Tp (M) at point.P
separated. Meanwhile, the reduced samples still maintain symmetric
positive definite structure, which can be directly applied to the
classification algorithm in Riemannian space.
The rest of this article is organized as follows: Section 2 introduces
the mathematical background of the Riemannian geometry; Section 3
shows the emotion recognition method proposed in this article and its
key components, including the construction of the enhanced SPD
matrices and the supervised dimensionality reduction methods; Section
4 describes the data set used in the experiment and the experimental
results; Section 5 discusses the proposed method. Section 6 concludes
our study.
2. Background
This section introduces the background of the Riemannian geometry,
Riemannian metric, tangent space, and Riemannian mean, which will be
used in the next section.
2.1. Riemannian geometry
Manifold is locally homeomorphic in Euclidean space and has a
globally defined differential structure [27]. Riemannian geometry
mainly studies smooth manifolds with Riemannian metric. Riemannian
manifolds are smooth differential manifolds which define the inner
product of Euclidean space at the tangent space of any point on manifold
M. Let S PD(N) = {P ∈ Rn×n |∀x ∕ = 0, x ∈ Rn , x⊤ Px > 0, P = P⊤ } and the
set of d × d SPD matrix is denoted by Sym+ , which is located on the
smooth differentiable SPD manifold.
2.2. Riemannian metric
The distance between two points on the Riemannian manifold is
defined as the length of the shortest path along a smooth Riemannian
manifold, which is called geodesic distance. The commonly used Rie­
mannian metric is the affine invariant Riemannian metric (AIRM), δR :
⃒⃒
⃒⃒ (
⃒⃒
)⃒⃒⃒⃒
[
∑N
]12
⃒⃒
δR (P1 , P2 ) = ⃒⃒log P1 − 1 P2 ⃒⃒ = log 2 ηi (1)
⃒⃒ ⃒⃒
F i=1
where ηi are the real eigenvalues of P−1 1 P2 . The metric of AIRM was
proposed by Forstner W [28], and Pennec [29] proved that the metric
satisfies affine invariance:
( )
δR W T P1 W, W T P2 W = δR (P1 , P2 ) (2)
W is any invertible matrix.
2.3. Tangent space
Riemannian manifold is a smooth differentiable surface, P ∈ Sym+ .
The linear space, which formed by all tangent vectors at point P, is called
tangent space, denoted as Tp (M) , as shown in Fig. 1. A point pi is
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

Fig. 2. The overall technical route of this article.

projected to tangent space Tp (M) using the logarithmic mapping Logp (pi ) 3. Methods
as:
( − 1 − 1) 1 In this section, we first describe the construction of enhanced SPD
(3)
1
si = log p (pi ) = p2 log p 2 pi p 2 p2 matrices as new samples, and then, we propose a supervised dimension
reduction method on Riemannian manifold as well as an optimization
where si ∈ Tp (M) and p is the tangent point. The inverse mapping: si → pi , method. The framework of our method is shown in Fig. 2.
is defined by the exponential mapping Expp (si ):
( − 1 − 1) 1
(4) 3.1. Wavelet packet transform
1
pi = Expp (si ) = p2 exp p 2 si p 2 p2

2.4. Riemannian mean EEG signals have time-varying and non-stationary characteristics.
Only analyzing EEG signals int time domain or frequency domain can
The mean of Riemannian space is different from that of Euclidean not accurately reflect its characteristics [32]. Time-domain and
space. It is not arithmetic mean, but geometric mean [30]. The Rie­ frequency-domain characteristics need to be considered together.
mannian mean is defined as: Wavelet transform is a widely used time-frequency analysis tool [33,34].
Compared with the Fourier transform [35] and the short-time Fourier
∑ transform [36], the wavelet transform has the advantage of
n
DR (P1 , …, Pn ) = argmin δ2R (P, Pi ) (5)
P∈P(n) i=1 multi-resolution analysis and can reflect the local details of the signal in
multiple scales. However, wavelet transform only decomposes the
There is no closed-form expression to compute the mean. An efficient
low-frequency part downwards, and cannot satisfy high time resolution
iterative algorithm to compute the Riemannian mean of SPD matrices is
and high frequency resolution at the same time. The wavelet packet
given in Ref. [31].
transform (WPT) is an extension of the wavelet transform. It divides the
signal into high-frequency and low-frequency parts at multiple levels
through a set of filters.
which is different from the low frequency resolution of wavelet

Fig. 3. The enhanced SPD matrix sample. a) Contains covariance matrix blocks between frequency bands. b) The local diagonal blocks of enhanced SPD matrix.

3
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

transform at high frequencies [37]. WPT can subdivide the high fre­ on SPD matrix is proposed, which maps high dimensional SPD manifold
quency part of the signal, and is widely used in processing random to low dimensional SPD manifold, and can effectively separate different
signals. classes of samples.

3.2. Enhanced SPD matrix 3.3.1. Supervised dimensionality reduction

EEG signals record the multi-channel time series of the brain. Let Xi ∈
RN×T be the EEG signal of the i − th trial, where N is the numbers of
channels and T is the numbers of samples. The sample covariance matrix
(SCM) of trials is denoted by P ∈ RN×N , which is defined by
1 matrix P, in order to maintain its matrix structure, we define a bilinear
P= XX T (6)
T− 1
The covariance matrix is a symmetric positive definite matrix, so we
can use the tool of Riemannian geometry to deal with it. However, the
covariance matrix only contains the spatial information between chan­
nels, and does not contain the time domain and frequency domain in­
formation of signals. Therefore, the original covariance matrix does not
contain enough features that can be used to classify emotional EEG
signals. For this purpose, we design an enhanced SPD matrix, which
contains both time-frequency and spatial information. For each trial Xi ∈
RN×T , we first extract four frequency bands of the signal using wavelet
packet, θ, α, β, γ, and Xθ , Xα , Xβ , Xγ are the corresponding reconstructed
signals. Then arrange them in turn to construct a new sample X∗ ∈ R4N×T
in (7):
⎡ ⎤
Xθ
⎢ Xα ⎥
X* = ⎢ ⎥ (7)
⎣ Xβ ⎦
Xγ
Learning a low dimensional manifold from a high one can not only
reduce the amount of computation, but also remove some redundant
information. By using label information, the samples of sample class can
be gathered as much as possible, and different samples can be separated.
The traditional dimensionality reduction methods, whose samples are
vectors, are not suitable for SPD matrix. For symmetric positive definite
mapping matrix W, the reduced dimensionality matrix Pm = WT PW. The
mapping matrix can be directly used to reduce the dimensionality of test
samples.
Compared with unsupervised dimensionality reduction algorithms,
supervised algorithms can use the label information to increase the
separability of the samples. Class center usually represents the charac­
teristics of a class, so the dimensionality is reduced by moving the
sample closer to its class center and away from other class centers. To
this end, we define a weight function u( ⋅, ⋅) to represent the relationship
between samples and class centers. For the weight of each sample, we
use the heat kernel based on Riemannian distance to calculate. More
specifically, for each sample xi , we denoted its class label by yi . Through
these labels, we can divide the samples into different sets Cj , and
calculate the Riemannian mean points of each class sj as the class cen­
ters, where 1 ≤ j ≤ c is the number of labels. If xi ∈ Cj , its weight will be
positive, otherwise it will be negative.
{ 2
( ) e− δ (xi ,sj )/2σ , xi ∈ cj
2

u xi , sj = (9)
− e− δ (xi ,sj )/2σ , otherwise ​
2 2
These new samples X* are used to calculate the covariance matrix by
eq. (6), and finally we get the enhanced SPD matrix P* ∈ R4N×4N :
where δ is the Riemannian metric, σ is the width parameter of the kernel
P* =
1 [ * * T]
X (X ) function. Based on the above ideas, we propose the following objective
(T − 1) function:
⎛ ⎞
T T ∑
N ∑
c
( ) ( )
⎜ Xθ Xθ … Xθ Xγ ⎟ (8) F = argmin δ2 W T xi W, W T sj W u xi , sj
1 ⎜ ⎜⋮ ⋱ ⋮
⎟
⎟ i=1 j=1 (10)
=
(T − 1) ⎜
⎝
⎟
⎠ s.t.W T W = In
T T
Xγ Xθ ⋯ Xγ Xγ
In order to ensure that the new samples WT XW > 0, ∀X ∈ Sym+ , we
We selected EEG signals of some channels to form the enhanced SPD add the constraint WT W = In . There is no closed-form solution for eq
matrix, as shown in Fig. 3. (10), so we design an iterative optimization method to solve the
problem.
3.3. Dimensionality reduction
3.3.2. Optimization
The SPD matrix is usually in high-dimensional space. When the SPD Affine invariant metric can accurately calculate geodesic distance, so
matrix of size n × n is mapped to Euclidean space, the obtained vector AIRM metric is chosen in this paper. equation (10) will be rewritten as
dimensionality is as high as n(n + 1)/2. It will cause the curse of follows:
dimensionality [38]. Therefore, most researches usually choose SPD c ⃒⃒ ⃒⃒2
N ∑⃒⃒
matrix with smaller dimension, which will lead to the loss of many ∑ ⃒⃒ ( T −1 )⃒⃒⃒⃒ ( )
F = argmin T
⃒⃒ log W x WW sW ⃒⃒ u xi , sj (11)
useful features. In fact, this is also a very important reason to limit the i=1
⃒ ⃒ ⃒ ⃒
F
application of Riemannian geometry to multi-channel signals, such as j=1

emotional EEG signals. Sheung [39] proved that logMk ≈ Mk − I under the condition of
There are three main ways to reduce the dimensionality of SPD allowable error, and as the number of iterations increases, the error
matrix. The first method is to map SPD matrix to Euclidean space, and gradually decreases. Therefore, we can write
then use the common methods to reduce dimensionality, such as linear ( ) ( )
dimension reduction method PCA, nonlinear dimension reduction log W T x− 1 WW T sW ≈ W T x− 1 WW T s − I W
( ) (12)
method Isomap, LLE and so on. However, these methods do not provide ≈ W T log x− 1 WW T s W
explicit mapping, and the new samples need to be recalculated. The
The objective function of eq (11) can then be written as:
second is to map the SPD matrix to Hilbert space by using kernel method
and finding the kernel function suitable for Riemannian manifold. The
third is to map a high-dimensional SPD manifold to a low dimensional
SPD manifold, and the obtained low dimensional SPD matrix samples
can be directly used in the Riemannian space classification algorithm.
In this paper, a supervised dimensionality reduction algorithm based

4
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

⃒⃒ ⃒⃒2
∑N ∑ c ( )⃒⃒⃒⃒ ( )⃒⃒⃒⃒
u xi , sj ⃒⃒log WT x− 1 WWT sW ⃒⃒
⃒⃒ ⃒⃒
4. Experiments and results
i=1 j=1
⃒⃒ ⃒F⃒2
∑ N ∑ c
( )⃒⃒⃒⃒ T ( − 1 T
) ⃒⃒⃒⃒ (13)
= u xi , sj ⃒⃒W log x WW s W⃒⃒
i=1 j=1
⃒⃒ ⃒⃒ In this part, we first introduce the public database used in this study.
( ( ) F)
∑∑ (
N c
) Then, we give a detailed description of the related experimental settings.
= u xi , sj Tr WT log x− 1 WWT s W Finally, the experimental results on DEAP database [41] are given and
i=1 j=1
( ) analyzed.
= Tr WT LW

N ∑
∑ c 4.1. DataSet
where L = u(xi , sj )log(x− 1 WWT s).
i=1 j=1
Obviously, this is a trace optimization problem [40], we can itera­ The data used in this paper was from DEAP database. It is a multi­
tively optimize through eigenvalue decomposition. The pseudo code of modal dataset based on physiological signals to analyze emotional state,
Dimensionality Reduction is given in Algorithm 1. and it is one of the most widely used data sets in emotional computing
research. In the data collection experiments, 32 subjects participated in
Algorithm 1. Supervised Dimensionality Reduction on Riemannian 40 Music Video experiments with different emotion tendencies and each
manifold. experiment lasted for 1 min. After watching each video, self-assessment

Fig. 4. The valence-arousal model of emotions.

Fig. 5. Electrode distribution diagram of the international 10–20 system. The
electrode pairs used in this paper are marked with blue circles.
5
Y. Gao et al.
Fig. 6. Performance comparison of each subject using different classifiers for valence.
makings (SAM) scale including four dimensions of pleasure, arousal,
dominance and liking were filled in for self-evaluation. The score range
of each dimension was from 1 to 9. The data recorded 32 channels of
EEG signals. In this paper, a preprocessed EEG dataset was used. The
sampling frequency of the original EEG signal is reduced to 128 Hz. The
electro-optic artifacts have been removed, and a band-pass filter of
4.0–45.0 Hz has been applied.
4.2. Data processing
We analyzed DEAP data from the dimensions of valence and arousal,
which have been divided into 9 levels [42]. A self-score of 5 was used as
6
Computers in Biology and Medicine 146 (2022) 105606
the dividing line, with arousal and valence above 5 being high arousal
and positive valence, and below 5 being low arousal and negative
valence. Therefore, we further segmented the four quadrants of the
valence-arousal space according to the score as shown in Fig. 4.
At the beginning of data collection, some subjects could not induce
emotional state well, so the last 30s data was used as our experimental
data. It was reported that the duration of emotion was about 0.5–4s
[43], so we use the sliding window of 4s, overlap for 2s, and get 14
samples in one session. Neuroscience research shows that there are
differences between the left and right hemispheres of the brain, which
can regulate different emotional states [44]. The brain activities related
to emotions are mainly distributed in the temporal and frontal lobes
Y. Gao et al.
Fig. 7. Performance comparison of each subject using different classifiers for arousal.
[45–47]. Therefore, using the difference in human brain response is very
important for emotion recognition [48,49]. For our study, we used the
seven symmetrical pairs of electrodes on the right (O2、P8、T8、FC6、
F4、F8、AF4) and left hemisphere (AF3、F7、F3、FC5、T7、P7、O1)
from the DEAP dataset. Fig. 5 illustrates the EEG electrodes placement.
Each subject’s EEG data was divided into 560 samples, four fre­
quency bands of each sample were extracted by wavelet packet. Follow
the method mentioned in Section 3 to generate the enhanced SPD matrix
as a new sample. The SPD matrix usually leads to high dimensional
problems, especially enhanced SPD matrix embedded with more time-
frequency information. The dimensionality reduction method we pro­
posed can map a high-dimensional SPD manifold into a low-dimensional
SPD manifold. We determined the best dimension for each subject
through the gird search method.
4.3. Classification and evaluation metrics
In this work, K-nearest neighbors (KNN), Random forest (RF) and
7
Computers in Biology and Medicine 146 (2022) 105606
Support vector machine (SVM) classifiers are selected to identify
different emotional states. For KNN, we search the best k value in the
range of 3–10. For RF, the maximum tree depth was optimized in the
range of 10–100, and the number of estimators was optimized in the
range of 100–500 with step of 100. For SVM, linear kernel is used as
kernel function on LIBSVM software. We use the grid search method to
search for the optimal parameter C in the range of 2− 10 − 210 . The pa­
rameters of the above model are selected through 5-fold cross valida­
tion. We trained the model for each subject and used the mean results of
5 times 5-fold cross-validation to access the model performance.
Accuracy (Acc) and F1-score are used to measure and evaluate the
performance of our proposed method. The Acc is defined in formula
(14):
TP + TN
Acc = (14)
TP + TN + FP + FN
where TP is the number of true positives, TN is the number of true
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

Fig. 8. Performance comparison of each subject using different classifiers for four-class.

negatives, FN is the number of false negatives, and FP is the number of Finally, KNN, RF and SVM were used to classify them in binary classi­
false positives. fication of valence and arousal. The experimental results of different
F1-score combines the results of precision (Pre) and recall rete (Rec) subjects are shown in Fig. 6 – Fig. 7.
at the same time. Their definitions are as follows: For the binary classification of valence, as shown in Fig. 6, the
highest accuracy rate and F1-score for 32 subjects were 96.81% and
TP
Pre = (15) 96.82%, which were obtained by SVM. Overall average accuracies of all
TP + FP
subjects were calculated as 91.86% ± 3.09%, 85.22% ± 5.82%, 82.77%
TP ± 5.63% for SVM, KNN, RF, respectively. In addition, the average of F1-
Rec = (16) score of SVM, KNN, and RF were 90.34%, 82.32%, 80.77%, respectively.
TP + FN
From Fig. 7, the classification results on the binary classification of
2 × Pre × Rec arousal showed that the proposed method achieve the highest accuracy
F1 − score = (17)
Pre + Rec rate of 96.04% and the highest F1-score of 97.03% for 32 subjects. For
SVM, KNN and RF, the average of accuracies were 91.81% ± 3.03%,
4.4. Binary classification of valence and arousal 85.96% ± 5.51% and 83.75% ± 6.54%, the average of F1-score were
89.07% ± 4.36%, 81.63% ± 7.4%, 79.91% ± 7.19% respectively.
We took the geometric mean point of the training sample as the
tangent point to establish the tangent plane, and mapped the reduced-
dimensional matrix to the tangent space to obtain the feature vector.

8
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

Fig. 9. The enhanced SPD matrix in different valence-arousal states. Each enhanced SPD matrix contains covariance matrix blocks in θ, α, β, γ bands. a) HVHA: high-
valence/high-arousal b) LVHA: low-valence/low-arousal c) LVLA: low-valence/low-arousal d) HVLA: high-valence/low-arousal.

Fig. 10. Compare different dimensionality reduction methods.

4.5. Four-class classification
For the four-class classification task, all data processing steps were
the same as the binary classification task. The difference was their labels,
LALV, HALV, LAHV, and HAHV. The results of each subject are shown in
Fig. 8. In the four-class classification tasks, the highest accuracy we
achieved was 97%, and the highest F1-score was 96.88%. The average
accuracies were 86.71% ± 4.98%, 77.12% ± 7.57%, 76.63 ± 7.3%%,
and the average F1-score were 85.60% ± 5.29%, 73.81% ± 8.13%,
72.82% ± 8.54% for SVM, KNN and RF, respectively.
5. Discussion
In this paper, we constructed the enhanced SPD matrices and
designed a supervised dimensionality reduction on Riemannian mani­
fold. The enhanced SPD matrix, which contains more time-frequency

9
Y. Gao et al. Computers in Biology and Medicine 146 (2022) 105606

Fig. 11. The accuracy corresponding to SPD matrices of different sizes.

Table 1
Details from various previous researches.
Emotion recognition task Method Features and Classifier Accuracy (%)

Valence Arousal

Binary classification Gao et al. [50] Hjorth, PSD, Differential Entropy, Sample Entropy; SVM 80.52 75.22
Chen et al. [51] amplitude and PSD; SVM 87.07 86.98
Piho et al. [52] HOC, SF, PSE, HOS; SVM 87.11 84.84
An et al. [53] Differential Entropy, spatiotemporal features; CNN-SAE 89.49 90.76
Chen et al. [51] amplitude and PSD; CVCNN 88.76 85.57
Ganapathy et al. [54] spatial and temporal features; CNN 72 75
Our time-frequency and spatial features; SVM 91.86 91.81
Four-class Classification Kong et al. [55] the EEG phase synchronization 67.04
indexes (PSI); SRC
Kwon et al. [56] Wavelet transformed spectrogram of EEG, Short time zero Crossing rate of GRS; CNN 73.43
Mei et al. [57] Connection matrix of the brain structure; CNN 73.10
Our time-frequency and spatial features; SVM 86.71

and spatial information, was used as features. Fig. 9 shows the enhanced
SPD matrix for different valence-arousal states, and each enhanced SPD
matrix contains the covariance matrix of multiple sub-bands. In the
high-valence/high-arousal state, channels from the frontal and temporal
lobes have strong effects in all four frequency bands. Channels in the
right hemisphere were more active in the low-valence/ow-arousal state.
In the low-valence/high-arousal state, the frontal and temporal lobes
from the left hemisphere were more active in the theta and alpha bands.
In the high valence low arousal state, the interaction between the left
and right hemispheres in the theta and alpha bands was significantly
enhanced. The enhanced SPD matrices in different states capture the
features of different brain regions, which will make the SPD matrices
distributed at different locations on the Riemannian manifold. So we can
use the differences in the distribution of these SPD matrices to better
identify different emotions.
Moreover, the supervised dimensionality reduction algorithm we
proposed effectively increased the separability of samples while
reducing the dimensionality. Sample labels were used to bring similar
samples closer to the center of the class, while samples of different types
were far away from each other. We compared the classical linear
dimensionality reduction method PCA and the nonlinear dimensionality
reduction method Isomap with our method. The classification experi­
ments of valence, arousal and four-class on subject 1 were conducted
through different dimensionality reduction methods. As shown in
Fig. 10, the accuracy of our dimensionality reduction method is signif­
icantly better than PCA and Isomap. In addition, we conducted a paired
t-test on the results, and the p values obtained were less than 0.001,
indicating that the results were statistically significant.
It is worth noting that our dimensionality reduction method maps the
SPD matrix to the low dimensional SPD matrix in Riemannian space, and
the samples obtained are suitable for both classifiers in Riemann space
and Euclidean space. Fig. 11 shows the classification accuracy for
different matrix sizes. The result show that when the matrix size is
reduced by half, the accuracy is still higher than the SPD of the original
size. As the matrix becomes smaller, the accuracy decreases slowly,
indicating that the reduced dimension matrix still retains most of the
effective features. If it is classified in Riemannian space, the appropriate
matrix size can be selected according to the accuracy. If the classification
is carried out in Euclidean space, a smaller SPD matrix size can be
selected, which can reduce the training time while maintaining a certain
accuracy.
By using the SVM classifier, we have obtained the best model per­
formance, which is significantly higher than the KNN and RF classifiers.
To verity the superiority of the proposed method, we compared the
latest methods with our proposed method. Table 1 details the features
and classifiers used in the comparison researches. As shown in Table 1,
using the SVM classifier [50–52], our proposed method improved at
least 4.75% in binary classification of valence and at least 3.32% in
binary classification of arousal. We also compared with other methods
that combined time or frequency domain features with spatial features
[51,53,54]. Compared with these methods, our method improved
2.37%–19.86% in binary classification of valence and 1.05%–16.81% in
binary classification of arousal.
Our methods were also compared with other four-class emotion
recognition methods and the specific details can be found in Table 1.
Compared with these studies, our method is at least 13.28% higher than
the comparative methods in four-class classification. Experimental re­
sults indicated that our method could be well used in four-class emotion
classification tasks. Because our method could gather multi-class sam­
ples near multiple class centers and away from other class centers, which
increased the separability of samples.
6. Conclusion
To conclude, we utilized the enhanced SPD matrix to improve the
performance of EEG emotion recognition. We designed a supervised

10
Y. Gao et al.
dimensionality reduction method in Riemann space to reduce the
dimensionality of the enhanced SPD matrix. While reducing the
dimensionality, it can also reduce the intra-class distance and increase
the inter-class distance between samples. In order to verify the effec­
tiveness of our method, we used three classifiers to conducted the binary
classification of valence/arousal and the four-class classification ex­
periments on DEAP. Compared with other state-of-the-art methods in
recent years, the performance of our proposed method is superior.
Currently, our method needs to map the SPD matrix to the Euclidean
space for classification, which increases the computation complexity. In
the next work, we will try to design a classifier that can be directly
applied in Riemannian space to better distinguish the emotion states. In
addition, emotional EEG signals have large individual differences
among different subjects, and the Riemannian metric has a strong
generalization ability. Therefore, in future work, we will explore the
common emotional patterns among different subjects under the Rie­
mannian geometry framework and improve the performance of emotion
recognition across subjects.
Declaration of competing interest
The authors declared that they have no conflicts of interest to this
work.
Acknowledgments
This work was supported by the National Nature Science Foundation
of China (61971168, 61871427), the Open Research Projects of Zhijiang
Lab (No. 2021MC0AB04), the Zhejiang Provincial Key R&D Program of
China (No.2021C03031) and the Zhejiang Provincial Natural Science
Foundation of China (LZ22F010003).
References
[1] J.J. Gross, Emotion Regulation: Conceptual and Empirical Foundations, Handbook
of Emotion Regulation, 2014, pp. 3–20.
[2] J.A. Russell, A circumplex model of affect, J. Pers. Soc. Psychol. 39 (1980) 1161.
[3] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, J.
G. Taylor, Emotion recognition in human-computer interaction, IEEE Signal
Process. Mag. 18 (2001) 32–80.
[4] T.J. Brigham, Merging technology and emotions: introduction to affective
computing, Med. Ref. Serv. Q. 36 (2017) 399–407.
[5] Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition
methods: audio, visual, and spontaneous expressions, IEEE Trans. Pattern Anal.
Mach. Intell. 31 (2008) 39–58.
[6] R.A. Calvo, S. D’Mello, Affect detection: an interdisciplinary review of models,
methods, and their applications, IEEE Trans. Affect Comput. 1 (2010) 18–37.
[7] C. Li, W. Tao, J. Cheng, Y. Liu, X. Chen, Robust multichannel eeg compressed
sensing in the presence of mixed noise, IEEE Sensor. J. 19 (2019) 10574–10583.
[8] M.R. Islam, M.A. Moni, M.M. Islam, M. Rashed-Al-Mahfuz, M.S. Islam, M.K. Hasan,
M.S. Hossain, M. Ahmad, S. Uddin, A. Azad, Emotion recognition from EEG signal
focusing on deep learning and shallow learning techniques, IEEE Access 9 (2021)
94601–94624.
[9] C.A. Frantzidis, C. Bratsas, C.L. Papadelis, E. Konstantinidis, C. Pappas, P.
D. Bamidis, Toward emotion aware computing: an integrated approach using
multichannel neurophysiological recordings and affective visual stimuli, IEEE
Trans. Inf. Technol. Biomed. 14 (2010) 589–597.
[10] S. Tripathi, S. Acharya, R.D. Sharma, S. Mittal, S. Bhattacharya, Using deep and
convolutional neural networks for accurate emotion classification on deap dataset,
in: Twenty-ninth IAAI Conference, 2017, pp. 4746–4752.
[11] R. Nawaz, K.H. Cheah, H. Nisar, V.V. Yap, Comparison of different feature
extraction methods for EEG-based emotion recognition, Biocybern. Biomed. Eng.
40 (2020) 910–926.
[12] K.S. Kamble, J. Sengupta, Ensemble machine learning-based affective computing
for emotion recognition using dual-decomposed EEG signals, IEEE Sensor. J. 22 (3)
(1 Feb.1, 2022) 2496–2507.
[13] A. Subasi, T. Tuncer, S. Dogan, D. Tanko, U. Sakoglu, EEG-based emotion
recognition using tunable Q wavelet transform and rotation forest ensemble
classifier, Biomed. Signal Process Control 68 (2021), 102648.
[14] J. Atkinson, D. Campos, Improving BCI-based emotion recognition by combining
EEG feature selection and kernel classifiers, Expert Syst. Appl. 47 (2016) 35–41.
[15] T. Tuncer, S. Dogan, A. Subasi, A new fractal pattern feature generation function
based emotion recognition method using EEG, Chaos, Solit. Fractals 144 (2021),
110671.
[16] A. Dogan, M. Akay, P.D. Barua, M. Baygin, S. Dogan, T. Tuncer, A.H. Dogru, U.
R. Acharya, PrimePatNet87: prime pattern and tunable q-factor wavelet transform
11
Computers in Biology and Medicine 146 (2022) 105606
techniques for automated accurate EEG emotion recognition, Comput. Biol. Med.
138 (2021), 104867.
[17] H. Wang, X. Wu, L. Yao, Identifying cortical brain directed connectivity networks
from high-density EEG for emotion recognition, IEEE Trans. Affect. Comput.
(2020) 1, 1.
[18] L. Xu, M. Xu, T. Jung, D. Ming, Review of brain encoding and decoding
mechanisms for EEG-based brain–computer interface, Cognit. Neurodynamics 15
(2021) 569–584.
[19] A. Barachant, S. Bonnet, M. Congedo, C. Jutten, Riemannian geometry applied to
BCI classification, in: International Conference on Latent Variable Analysis and
Signal Separation, 2010, pp. 629–636.
[20] A. Barachant, S. Bonnet, M. Congedo, C. Jutten, Multiclass brain–computer
interface classification by Riemannian geometry, IEEE (Inst. Electr. Electron. Eng.)
Trans. Biomed. Eng. 59 (2011) 920–928.
[21] Y. Wang, S. Qiu, X. Ma, H. He, A prototype-based SPD matrix network for domain
adaptation EEG emotion recognition, Pattern Recogn. 110 (2021), 107626.
[22] F. Tang, M. Fan, P. Tiňo, Generalized learning riemannian space quantization: a
case study on riemannian manifold of spd matrices, IEEE Transact. Neural
Networks Learn. Syst. 32 (2020) 281–292.
[23] J. Xu, M. Grosse-Wentrup, V. Jayaram, Tangent space spatial filters for
interpretable and efficient Riemannian classification, J. Neural. Eng. 17 (2020),
026043.
[24] F. Wu, A. Gong, H. Li, L. Zhao, W. Zhang, Y. Fu, A new subject-specific
discriminative and multi-scale filter bank tangent space mapping method for
recognition of multiclass motor imagery, Front. Hum. Neurosci. 15 (2021) 104.
[25] M. Congedo, A. Barachant, R. Bhatia, Riemannian geometry for EEG-based brain-
computer interfaces; a primer and a review, Brain Comput. Interface 4 (2017)
155–174.
[26] T. Zhang, W. Zheng, Z. Cui, Y. Zong, Y. Li, Spatial–temporal recurrent neural
network for emotion recognition, IEEE Trans. Cybern. 49 (2018) 839–847.
[27] A.P. Shirokov, Structures on differentiable manifolds, in: R.V. Gamkrelidze (Ed.),
Progress in Mathematics: Algebra and Geometry, Springer US, Boston, MA, 1971,
pp. 137–207.
[28] W. Förstner, B. Moonen, "A metric for covariance matrices." Geodesy-the Challenge
of the 3rd Millennium, Springer, Berlin, Heidelberg, 2003, pp. 299–309.
[29] X. Pennec, P. Fillard, N. Ayache, A Riemannian framework for tensor computing,
Int. J. Comput. Vis. 66 (2006) 41–66.
[30] T. Ando, C. Li, R. Mathias, Geometric means, Lin. Algebra Appl. 385 (2004)
305–334.
[31] P.T. Fletcher, S. Joshi, Principal geodesic analysis on symmetric spaces: statistics of
diffusion tensors, in: Computer Vision and Mathematical Methods in Medical and
Biomedical Image Analysis, Springer, 2004, pp. 87–98.
[32] A.S. Al-Fahoum, A.A. Al-Fraihat, Methods of EEG signal features extraction using
linear analysis in frequency and time-frequency domains, Int. Sch. Res. Notices
2014 (2014) 1–7, 02-13 2014.
[33] M.R. Islam, M. Ahmad, Wavelet analysis based classification of emotion from EEG
signal, in: 2019 International Conference on Electrical, Computer and
Communication Engineering (ECCE), 2019, pp. 1–6.
[34] Z. Mohammadi, J. Frounchi, M. Amiri, Wavelet-based emotion recognition system
using EEG signal, Neural Comput. Appl. 28 (2017) 1985–1990.
[35] J.W. Cooley, J.W. Tukey, An algorithm for the machine calculation of complex
Fourier series, Math. Comput. 19 (1965) 297–301.
[36] J. Allen, Short term spectral analysis, synthesis, and modification by discrete
Fourier transform, IEEE Trans. Acoust. Speech Signal Process. 25 (1977) 235–238.
[37] S. Chinara, Automatic classification methods for detecting drowsiness using
wavelet packet transform extracted time-domain features from single-channel EEG
signal, J. Neurosci. Methods 347 (2021) 108927.
[38] M. Congedo, P.L.C. Rodrigues, F. Bouchard, A. Barachant, C. Jutten, A closed-form
unsupervised geometry-aware dimensionality reduction method in the Riemannian
Manifold of SPD matrices, in: 2017 39th Annual International Conference of the
IEEE Engineering in Medicine and Biology Society (EMBC), 2017, pp. 3198–3201.
[39] S.H. Cheng, N.J. Higham, C.S. Kenney, A.J. Laub, Approximating the logarithm of a
matrix to specified accuracy, SIAM J. Matrix Anal. Appl. 22 (2001) 1112–1125.
[40] E. Kokiopoulou, J. Chen, Y. Saad, Trace optimization and eigenproblems in
dimension reduction methods, Numer. Lin. Algebra Appl. 18 (2011) 565–602.
[41] S. Koelstra, C. Muhl, M. Soleymani, J. Lee, A. Yazdani, T. Ebrahimi, T. Pun,
A. Nijholt, I. Patras, Deap: a database for emotion analysis; using physiological
signals, IEEE Trans. Affect. Comput. 3 (2011) 18–31.
[42] J.A. Russell, A. Mehrabian, Evidence for a three-factor theory of emotions, J. Res.
Pers. 11 (1977) 273–294.
[43] V. Gupta, M.D. Chopda, R.B. Pachori, Cross-subject emotion recognition using
flexible analytic wavelet transform from EEG signals, IEEE Sensor. J. 19 (2018)
2266–2274.
[44] K.M. Alfano, C.R. Cimino, Alteration of expected hemispheric asymmetries:
valence and arousal effects in neuropsychological models of emotion, Brain Cognit.
66 (2008) 213–220.
[45] B. Bogert, T. Numminen-Kontti, B. Gold, M. Sams, J. Numminen, I. Burunat,
J. Lampinen, E. Brattico, Hidden sources of joy, fear, and sadness: explicit versus
implicit neural processing of musical emotions, Neuropsychologia 89 (2016)
393–402.
[46] J.P. Ramirez-Mahaluf, J. Perramon, B. Otal, P. Villoslada, A. Compte, Subgenual
anterior cingulate cortex controls sadness-induced modulations of cognitive and
emotional network hubs, Sci. Rep. 8 (2018) 1–16.
[47] N. Zhuang, L. Jiang, B. Yan, L. Tong, J. Shu, K. Yang, D. Yao, P. Xu, Y. Zeng, Neural
mechanism of affective perception: evidence from phase and causality analysis in
the cerebral cortex, Neuroscience 461 (2021) 44–56.
Y. Gao et al.
[48] D. Huang, S. Chen, C. Liu, L. Zheng, Z. Tian, D. Jiang, Differences first in
asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for
EEG emotion recognition, Neurocomputing 448 (2021) 140–151.
[49] J. Zhang, M. Chen, S. Zhao, S. Hu, Z. Shi, Y. Cao, ReliefF-based EEG sensor
selection methods for emotion recognition, Sensors 16 (2016) 1558.
[50] Q. Gao, C. Wang, Z. Wang, X. Song, E. Dong, Y. Song, EEG based emotion
recognition using fusion feature extraction method, Multimed. Tool. Appl. 79
(2020) 27057–27074.
[51] J.X. Chen, P.W. Zhang, Z.J. Mao, Y.F. Huang, D.M. Jiang, Y.N. Zhang, Accurate
EEG-based emotion recognition on combined features using deep convolutional
neural networks, IEEE Access 7 (2019) 44317–44328.
[52] L. Piho, T. Tjahjadi, A mutual information based adaptive windowing of
informative EEG for emotion recognition, IEEE Trans. Affect. Comput. 11 (2020)
722–735.
12
Computers in Biology and Medicine 146 (2022) 105606
[53] Y. An, S. Hu, X. Duan, L. Zhao, C. Xie, Y. Zhao, EEG emotion recognition based on
3D feature fusion and convolutional autoencoder, Front. Comput. Neurosci. (2021)
83.
[54] N. Ganapathy, Y.R. Veeranki, R. Swaminathan, Convolutional neural network
based emotion classification using electrodermal activity signals and time-
frequency features, Expert Syst. Appl. 159 (2020), 113571.
[55] W. Kong, X. Song, J. Sun, Emotion recognition based on sparse representation of
phase synchronization features, Multimed. Tool. Appl. 80 (2021) 21203–21217.
[56] Y. Kwon, S. Shin, S. Kim, Electroencephalography based fusion two-dimensional
(2D)-convolution neural networks (CNN) model for emotion recognition system,
Sensors 18 (2018) 1383.
[57] H. Mei, X. Xu, EEG-based emotion classification using convolutional neural
network, in: 2017 International Conference on Security, Pattern Analysis, and
Cybernetics (SPAC), 2017, pp. 130–135.