A Novel Multivariate-Multiscale Approach for Computing

EEG Spectral and Temporal Complexity for Human Emotion

Recognition

Journal: IEEE Sensors Journal

Manuscript ID Sensors-35517-2020

Manuscript Type: Regular Paper

Date Submitted by the

04-Sep-2020
Author:

Complete List of Authors: Bhattacharyya, Abhijit; NIT Hamirpur, ECE

Tripathy, RK; BITS Pilani Hyderabad, EEE
Garg, Lalit; University of Malta
Pachori, R. B.; Discipline of Electrical Engineering

Keywords: DATP
Page 1 of 16 Classification of positive, neutral, and negative human emotions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 Page 2 of 16
1

1
2
3 A Novel Multivariate-Multiscale Approach for
4
5
6
Computing EEG Spectral and Temporal Complexity
7
8
for Human Emotion Recognition
9
Abhijit Bhattacharyya, Rajesh Kumar Tripathy, Lalit Garg, and Ram Bilas Pachori
10
11
12
13
14 Résumé—This work proposes a novel multivariate-multiscale than emotional intelligence, emotions have direct linkage with
approach for computing the spectral and temporal entropies many mental disorders such as depression, attention deficit
15
from the multichannel electroencephalogram (EEG) signal. This hyperactivity, autism, and game addiction [4], [5]. In this
16 facilitates the recognition of three human emotions : positive,
17 neutral, and negative. The proposed approach is based on the regard, affective computing emerges as an interdisciplinary
18 application of the Fourier-Bessel series expansion based empirical research area aiming to build human-aware artificial intelli-
19 wavelet transform (FBSE-EWT). We have extended the existing gence capable to recognize, comprehend, and control emotions
FBSE-EWT method for multichannel signals and derived FBSE- [6]. The affective computing is primarily focused on detection
20 EWT based multivariate Hilbert marginal spectrum (MHMS)
21 and modelling human emotions using pattern recognition and
for computing spectral Shannon and K-nearest neighbor (K-
22 NN) entropies. The multivariate FBSE-EWT decomposes the machine learning techniques. The emotion recognition using
23 multichannel EEG signals into narrow band subband signals. The electroencephalogram (EEG) signals is found more convin-
24 multiscaling operation adapted in the spectral domain is based cing as compared to other approaches relying on external
on the selection of successive joint instantaneous amplitude and appearances like as facial expression, gesture, speech signal
25 frequency functions of the subband signals. On the other hand,
26 [7] which might be pretended emotions [8]. Despite the fact
the time domain multiscale K-NN entropy is computed from
27 the cumulatively added multidimensional subband signals. The that EEG has low spatial resolution, the use of EEG provides
28 extracted spectral and temporal entropy features are smoothed great time resolution which can be effective to measure the
29 and fed to the sparse autoencoder based random forest (ARF) changes in the signal characteristics due to emotional stimuli.
classifier architecture for emotion classification. The proposed In addition, EEG is inexpensive, fast, and noninvasive which
30 approach is tested using multichannel EEG signals available in
31 make it a preferred choice to study the brain responses
a public database (SJTU emotion EEG dataset (SEED)). The
32 bivariate EEG signals from different channel pairs with distinct subjected to emotional stimuli [9]. The emotion recognition
33 spatial locations over the scalp are considered as input to our task can be carried out after performing several steps such
34 proposed system. The obtained overall classification accuracy of as the feature computation, and classification. The feature
94.4% reveals that the proposed approach is useful in classifying computation from the EEG signal using signal processing
35 human emotions. The method is also validated using DREAMER
36 approaches is the crucial stage for the emotion classification.
emotion EEG public database. The method outperforms the
37 existing state-of-the-art methods evaluated in these databases. In literature, various methods based on the EEG signal
38 features and classifiers were reported for automated human
39 emotion classification [10]–[21]. In [10], authors used fuzzy
Index Terms—multivariate-multiscale, emotion recognition,
40 spectral entropy, temporal entropy, ARF classifier. c-mean and fuzzy k-mean clustering algorithms for the classi-
41 fication of emotion EEG signals. In [11] authors used EEG
42 signals, gaze distance, and pupillary response for emotion
I. I NTRODUCTION recognition using modality fusion strategy and support vector
43
MOTIONS play an imperative role in human daily life
44
45 E and have an influence on our normal day to day activities
such as human cognition, decision making, human intelligence
machine (SVM) classifier. The common spatial patterns (CSP)
and linear SVM classifier were explored together for the
classification of EEG emotions in [13]. The SVM classifier
46
etc [1]. Apart from logical intelligence, emotional intelligence based emotion recognition is also carried out in [14]. In [15],
47
is also considered as crucial part of human intelligence [2]. various time-frequency (TF) distributions namely Zhao-Atlas-
48
Developing emotional artificial intelligence is a recent trend Marks transform, spectrogram, amd Hilbert-Huang spectrum
49
of research in human-computer interactions (HCIs) [3]. Other were used for time-windowing based feature extraction and
50
51 classification of emotion EEG signals. The fast Fourier trans-
Abhijit Bhattacharyya, is with the Department of Electronics and
52 Communication Engineering, National Institute of Technology Hamirpur, form (FFT) based features were fed to a probabilistic classifier
53 Hamirpur-177005, India. e-mail : abhijit@nith.ac.in (based on Bayes’ theorem) for EEG emotion recognition in
Rajesh Kumar Tripathy, is with the Department of Electrical and Elec- [16]. The characteristics of emotion EEG signals were studied
54 tronics Engineering, Birla Institute of Technology and Science (BITS) Pilani,
55 Hyderabad Campus, India. e-mail : rajeshiitg13@gmail.com using differential entropy feature in [17]. In [18], authors
56 Lalit Garg, is with the Department of Information and Communication introduced deep belief networks (DBNs) for investigating
Technology, University of Malta, Msida. e-mail : lalit.garg@um.edu.mt critical frequency bands and channels in order to perform
57 Ram Bilas Pachori is with Discipline of Electrical Engineering, In-
58 dian Institute of Technology Indore, Indore-453552, India. e-mail : pa- EEG emotion recognition for three emotions namely positive,
59 chori@iiti.ac.in neutral, and negative. The deep learning framework namely
60
Page 3 of 16
2

1
2 spatial–temporal recurrent neural network is used in [19] for experimental results obtained and discusses about the effec-
3 emotion recognition. In [20], 18 different linear and non-linear tiveness of the proposed method, respectively. Finally, Section
4 features were used with SVM classifier for EEG emotion clas- VI concludes the paper.
5 sification. The information potential (IP) based features were
6 extracted from flexible analytic wavelet transform (FAWT)
2 100

Frequency
7 decomposed subband signals for EEG emotion recognition in a

(Hz)
8 recent study [21]. In [22], authors used multivariate empirical 0 (a)
9 mode decomposition (MEMD) based time and frequency 50 (d)
-2
10 domain features with artificial neural network (ANN) classifier
0 5 10

Amplitude
11 for human emotion recognition. 1 0
12 Here, we propose multivariate-multiscale spectral and tem- (b) 0 5 10
0 Time in seconds
13 poral entropy features for analyzing the inherent characteristics
-1

Magnitude
14 of emotion EEG signals. The entropy features and their 0 5 10 200
15 multiscale versions have been found useful in many signal 2
16 processing applications [23]–[26]. However, most of these 0 (c) 100 (e)
17 multiscale entropy measures are focused on measuring the
-2 0
18 complexity of univariate signals. In recent days, the mul- 0 5 10 0 50 100
19 tivariate signals are more frequently recorded and analyzed Time in seconds Frequency (Hz)
20 in the area of biomedical engineering, neuroscience [27]. In
21 this work, the Fourier-Bessel series expansion (FBSE) based F IGURE 1: (a) Multi-component sinusoidally modulated FM
22 empirical wavelet transform (FBSE-EWT) [28] is explored for signal of channel 1. (b) Multi-component AM signal of
23 decomposing multivariate emotion EEG signal into subband channel 2. (c) Multi-component linear chirp signal of
24 signals. The FBSE-EWT is an improved version of its pre- channel 3. (d) The multivariate time-frequency representation
25 decessor empirical wavelet transform (EWT) [29] which was of the considered trivariate signal. (e) The MHMS of the
26 applied previously for the analysis of biomedical signals [30]– considered trivariate signal obtained using FBSE-EWT.
27 [32]. The multivariate multi-scaling operation is performed in
28 the spectral domain by deriving multivariate Hilbert marginal
29 spectrum (MHMS) from the successive joint instantaneous am-
Amplitude

30 plitude (JIA) and frequency (JIF) functions (represent distinct 2 2

31 oscillatory levels). The Hilbert marginal spectrum (HMS) was 0 (a) 0 (b)
32 introduced in [33] for the spectral analysis of length-of-day -2 -2
33 data. Afterwards, Shannon and K-nearest neighbor (K-NN) 0 5 10 0 5 10
entropies are computed from the multiscaled MHMSs. On the Time in seconds
34
other hand, multiscale entropy is computed in the time domain 150
Frequency (Hz)

35
36 after cumulatively summing the multivariate subband signals 60
Magnitude

37 and computing their K-NN entropies. Finally, the multiscale 100

entropy features are smoothed and fed to a novel classifier (c) 40 (d)
38
39 architecture which is formed by stacking hidden layer of sparse 50 20
40 autoencoder and random forest (RF) classifier. Our major
contributions in this paper are as follows: 0 0
41 0 5 10 0 50 100 150
42 . The adaptive signal decomposition property of FBSE- Time in seconds Frequency (Hz)
43 EWT is explored for the analysis of nonstationary emo-
44 tion EEG signals. F IGURE 2: (a) Multi-component negative exponential chirp
45 signal of channel 1. (b) Multi-component positive
. The multiscale MHMS is proposed based on the selection
46 exponential chirp signal of channel 2. (c) The multivariate
of JIA and JIF functions.
47 time-frequency representation of the considered bivariate
48 . The Shannon and K-NN entropies are computed in order signal. (d) The MHMS of the considered bivariate signal
49 to capture the spectral and temporal characteristics of obtained using FBSE-EWT.
50 emotion EEG signals.
51 . The sparse autoencoder based RF classifier is proposed
52 for classifying positive, neutral, and negative emotion
53 EEG signals. II. M ETHODS
54 A. Multivariate Hilbert marginal spectrum based on FBSE-
55 The remaining of the paper is organised as follows : Sec-
tion II describes the MHMS based on FBSE-EWT, proposed EWT
56
57 multiscale spectral and temporal entropy computation method, The EWT is a suitable method for the analysis of non-
58 and proposed ARF classifier. Section III discusses about the stationary signals. The EWT generates wavelet based filters
59 database used in this work. Section IV and V present the which are adaptive to the analyzed signals. These adaptively
60
 Page 4 of 16
3

1
2 built wavelet based filters have supports in the information decomposed with the built filter bank in order to obtain N
3 location of the spectrum. In a recent work [28], authors number of subband signals with same frequency support at
4 presented improved time-frequency representation with better every oscillatory level [30]. The Hilbert transform [36] has
5 estimated signal components after including FBSE with the been applied for determining instantaneous amplitude (IA)
6 existing EWT method. The FBSE-EWT method is summarised and instantaneous frequency (IF) functions of the subbands of
7 by the following steps: multivariate signals. Then IA and IF of multivariate subband
8 1) The spectral representation of the analyzed signal in the signals falling in same oscillatory level have been combined in
9 frequency range [0, π] is obtained using FBSE method. order to compute the JIA and JIF functions at each oscillatory
10 The zero-order Bessel functions are used for the FBSE of level. The JIA and JIF functions are mathematically expressed
11 y(n) in terms of Fourier–Bessel (FB) series coefficients as [30], [37],
12 Ci as follows [34]: M
13
X
V   [Am, SBn (t)]2 Ωm, SBn (t)
14 y(n) =
X
Ci J0
δi n
, n = 0, 1, . . . , V − 1 (1)
15 V Ωmulti
n (t) = m=1
M
(5)
i=1
16
X
[Am, SBn (t)]2
17 The coefficients Ci can be computed using the first-order
m=1
18 J1 (.) and zero-order J0 (.) Bessel functions as [35], v
19
u M
V  
2 δi n multi
uX
2
X
20 Ci = 2 ny(n)J 0 (2) A n (t) = t [A m, SBn (t)] (6)
V (J1 (δi ))2 i=1 V
21 m=1

22 The parameter δi ; with i = 1, 2, . . . V denotes the posi- where, n = 1, 2, · · · , N represents the distinct oscillatory
23 tive roots (in ascending order) of the equation J0 (δ) = 0. levels. The functions Am, SBn (t) and Ωm, SBn (t) represent
24 The order i of FB series coefficients, analog frequency the IA and IF functions of subband n of channel m respecti-
25 (in Hz), and sampling frequency fs are related by the vely.
26 following equation [34], [35]: The time-frequency representation of the multivariate si-
27 gnals is mathematically expressed as,
2πfi V
28 δi ≈ , where δi ≈ δi−1 + π ≈ iπ (3)
29 fs
N
30 The equation (3) can be approximated as [35] i ≈ 2ffisV . TFmulti (Ω, t) = Amulti (t)δ[Ω − Ωmulti
X
n n (t)]. (7)
31 For covering the entire signal bandwidth, the order i n=1
32 should be varied from 1 to discrete time signal length The MHMS is defined as,
33 V . Finally, the FBSE spectrum is the plot of magnitude
34 of FB coefficients (|Ci |) versus frequencies fi .
Z
35 y(Ω) = TFmulti (Ω, t)dt (8)
2) Then, EWT boundary detection method [29] has been T
36 applied and N + 1 number of boundary frequencies
37 In order to support the theoretical MHMS derivation pre-
have been found in order to segment the FBSE spectrum sented above, we have considered the following two examples.
38 into N contiguous segments. The segments of the FBSE
39 1) Example 1: The synthetically generated trivariate multi-
spectrum are expressed as [0 ∅1 ], [∅1 ∅2 ], · · · [∅N −1 π]. component signal considered in this example can be mathe-
40
41 3) Finally, the empirical bandpass wavelet filters have been matically expressed as follows:
42 defined on each segment of the Fourier spectrum. The x(t) = [x1 (t) + x2 (t); x3 (t) + x4 (t); x5 (t) + x6 (t)]
43 detailed mathematical expressions of empirical bandpass where
44 wavelet filters can be found in [29]. x1 (t) = cos[2π(50t + 4sin(t))]
45 x2 (t) = cos[2π((50 + σ)t + 4sin(t))]
In [30], the authors applied EWT for multivariate signals
46 x3 (t) = [0.2 + 0.06cos(2πt)]sin(2π80t)
using the concept of mean spectrum magnitude of multi-
47 x4 (t) = [0.2 + 0.06cos(2πt)]sin[2π(80 + σ)t]
channel signals. In similar manner, here we define mean
48 x5 (t) = sin[2π(10t + 0.8t2 )]
FBSE spectrum for a given multivariate signal x(t) =
49
 x6 (t) = sin[2π((10 + σ)t + 0.8t2 )]
x1 (t); x2 (t); · · · ; xM (t) , as,
50 In Fig.1 (a), we have shown the considered multi-component
M sinusoidally frequency modulated (FM) signal with compo-
51 C(Ω) =
1 X m
|C (Ω)| (4)
52 M m=1 nents x1 (t) and x2 (t). The signal present in the second channel
53 is a multi-component amplitude modulated (AM) signal with
54 where, M is the number of channels, |C m (Ω)| is the components x3 (t) and x4 (t), shown in Fig.1 (b) . The third
55 individual channel FBSE spectrum, and variable Ω denotes channel contains multi-component linear chirp signal with
56 the frequency of the signal. Then the conventional boundary components x5 (t) and x6 (t) as shown in Fig.1 (c). The
57 detection method of EWT has been applied on the mean frequency deviation (σ) and sampling rate is considered as 0.4
58 FBSE spectrum C(Ω) and adaptive filter bank has been Hz and 204 Hz for all the channels. Fig.1 (d) and (e) show the
59 generated. After that, each of the channel signals has been time-frequency representation and the computed MHMS of the
60
Page 5 of 16
4

1
2 M-variate signal
3 Multivariate FBSE-EWT
4
5 …
…
6 SB11 SB12 SB1M SB21 SB22 … SB2M SBL1 SBL2
… SB M
L

7
8
… IA M
9 IA11 IA12 1 JIA1 JIF1 IF11 IF12 … IF1M
10
τ1

…
11 IA21 IA22 … IA2M JIA2 JIF2 IF21 IF22 … IF2M
12 …
τ2

…
13 IAL1 IAL2 … IALM JIAL
τL JIFL IFL1 IFL2 … IFLM
14
15 Multivariate time-frequency representation for a given scale factor IJ
16
17 Hilbert marginal spectrum
18
Computation of entropy features
19
20 F IGURE 3: Block diagram of the proposed FBSE-EWT based multivariate multiscale spectral entropy.
21
22
23
considered multivariate signal, respectively. It can be observed where τ varies from 1 to N . For lower values of τ the low
24
from Fig.1 (e) that the derived MHMS provides clear spectral frequency components will be absent in the time-frequency
25
representation of the considered trivariate signal. plane, where as for τ = N , the time-frequency plane contains
26
2) Example 2: The bivariate multi-component synthetic all the frequency components.
27
signal in this example is mathematically written as follows Step 2 : Computation of the MHMS from the scaled time-
28
: frequency representation TFmulti (Ω, t) can be performed as
29 τ
x(t) = [x1 (t) + x2 (t); x3 (t) + x4 (t)] follows:
30 where
31 k1t −1
x1 (t) = cos[2πf1 ( log(k )] ; x (t) = cos[2π(f + σ)(
k1t −1
)] ;
Z
32 1 ) 2 1 log(k 1)
τ
y (Ω) = TFmulti
τ (Ω, t)dt (10)
kt −1 kt −1 T
33 x3 (t) = cos[2πf2 ( log(k
2
2)
)] ; x4 (t) = cos[2π(f2 + σ)( log(k
2
2)
)].
34 The scale factor dependent MHMS of multichannel EEG
35 The multi-component negative exponential chirp signal in signals is discussed in detail in Section IV.
36 channel 1 (see Fig. 2(a)) has two components x1 (t) and x2 (t) Step 3 : Estimation of spectral Shannon and K-NN entropy can
37 with f1 = 35 Hz and frequency deviation σ = 0.2 Hz, and be obtained from the MHMS corresponding to scale factor τ .
38 exponential frequency rate k1 = 0.8. The channel 2 (see The computation of multivariate multiscale spectral entropy is
39 Fig. 2(b)) of the bivariate signal is a multicomponent positive presented in Fig. 3.
40 exponential chirp signal with components x3 (t) and x4 (t) 1) K-NN entropy estimate: The K-NN estimate I(Y ) of the
41 with f2 = 55 Hz and frequency deviation σ = 0.2 Hz, and differential entropy for a d-dimensional variable Y is defined
42 exponential frequency rate k2 = 1.1. The sampling rate of as [38], [39],
43 the bivariate signal is fixed at 304 Hz. Fig.2 (c) and (d) show V
the time-frequency representation and the computed MHMS d X
44 I(Y ) = φ(V ) − φ(L) + log(Gd ) + log(L
i ) (11)
45 of the considered multivariate signal, respectively. V i=1
46
where V represents sample length of the random variable
47 B. Computation of multiscale spectral entropy
Y , and diagama function φ(x) mathematically expressed as
48 The computation algorithm of multiscale spectral entropy is follows [39]:
49 summarised in the following steps: 1 dΓ(x)
50 Step 1 : For a given scale factor τ , the first τ number of φ(x) =
Γ(x) dx
(12)
51 JIAs and their corresponding JIFs have been considered in
52 order to generate time frequency representation using equation The parameter L i represents the distance in the d-
53 (7). Thus, the time-frequency representation using τ number dimensional space between the sample yi of the considered
54 of JIAs and JIFs is named as scaled time-frequency represen- variable Y and its L nearest neighbors. Gd represents d-
55 tation which is expressed as, dimensional unit ball volume. Considering Euclidean norm,
56 Gd is defined as follows [38]:
57 τ d
π2
58 TFmulti Amulti (t)δ[Ω − Ωmulti
X
τ (Ω, t) = n n (t)] (9) Gd = (13)
59 n=1 Γ(1 + d2 )
60
 Page 6 of 16
5

1
2 Here, we have computed K-NN entropy of the MHMS y τ (Ω) 50

3 defined in equation (10). 0

4
-50
2) Shannon entropy: The Shannon entropy is defined as 20

Amplitude
5 follows : 0

6 K 
X Ek
  
Ek
-20

7 ShEnτ = − log (14) 20

E E 0
8 k=1
-20
9
where Ek and E are the energy of the k th frequency
0 0.2 0.4 0.6 0.8 1

10 Time in second
component and the total energy of y τ (Ω), respectively.
11 F IGURE 4: Sample EEG signals for positive emotion (top),
12 neutral emotion (middle), and negative emotion (bottom).
13 C. Computation of multiscale temporal entropy
14 Multiscale temporal entropy is computed as follows:
15 Step 1 : For a specific scale factor τ , cumulatively sum τ 1
16 2
subband signals of each channel as follows: 3
17 4
5
τ

Subband signals
18 X 6
CumlτCh (n) = SBjCh (n) (15) 7
19 8
9
j=1
20 10
11
21 The parameter Ch represents the channel index of the 12
13
22 multivariate signals. 14
15
23 Step 2 : K-NN entropy is computed from the recons- 16
17
24 tructed multivariate (M-channel) multiscaled subband signals 18
25 [CumlτCh1 (n); CumlτCh2 (n); . . . ; CumlτChM (n)]. 19
20
26 0 0.2 0.4 0.6 0.8 1
27 D. Emotion recognition using ARF classifier Time in second
28
In this work, we have formulated the sparse autoencoder F IGURE 5: Plot of the FBSE-EWT subband signals for the
29
based random forest (ARF) classifier for the emotion recog- positive emotion.
30
nition. The ARF consists of the input layer, the hidden layer
31
and the random forest (RF) [40] based classification layer,
32
respectively. The training and test instances for the proposed
33 exploit the correlation between the desired sparsity and the
ARF classifier are chosen based on the 10-fold cross-validation
34 average activation of each hidden neuron in sparse autoen-
approach. The input layer comprises of the 60 dimensional
35 coder. The other parameters such as α and λ are denoted
feature vector which is the concatenation of the 20 dimensional
36 as the sparsity regularizer and weight regularizer for the
temporal K-NN entropy feature vector (F VT KE ), 20 dimen-
37 sional spectral K-NN entropy feature vector (F VSKE ), and 20 sparse autoencoder [42]. In this work, the hidden layer from
38 dimensional spectral Shannon entropy feature vector (F VSSE ), sparse autoencoder acts as a non-linear dimension reduction
39 respectively. These 20 dimensional feature vectors (formed by technique which projects the 60 dimensional feature vector
40 varying scale factor τ from 1 to 20) are evaluated for each to a lower dimensional hidden layer space. In the last layer,
41 pair of EEG channels. The second layer of ARF consists of the RF classifier is used for the classification of positive
42 hidden neurons and the activations of these hidden neurons emotion, negative emotion and neutral emotion from the
43 are computed based on the training of sparse autoencoder. hidden layer feature vector. The proper selection of the sparse
44 The training of sparse autoencoder is performed based on autoencoder parameters (α, λ, ρ, iterations) and the random
45 the estimation of weight matrix and bias values based on the forest parameters (number of trees, number of splits and the
46 solution of the optimization problem as [41], depth of each tree) is important for the classification using
47 ARF classifier [43]. In this study, we have considered the
m p
48 λ ρ
2
X ˜ i i 2
X grid-search [44] to estimate the optimal training parameters
J(W, b) = FV − FV + kWk2 +α KL( ) (16)
49 i=1
2 2 j=1
ρ̃ j for the ARF classifier. The test instances in each fold are then
50 evaluated using the trained ARF classifier. In this study, the
51 where J, W, b are denoted as the cost function, weight ma- individual class accuracy and the total accuracy are considered
52 trix and bias values between the input layer to the hidden layer for the evaluation of the performance of ARF classifier [45].
53 in sparse autoencoder [31]. FVi = [F VSSE F VSKE F VT KE ]
54 is termed as the 60 dimensional feature vector for ith training
instance, and the total number of instances is denoted as m III. DATABASE USED
55
56 [42]. The important parameters in sparse autoencoder are the We have used publicly available SJTU emotion EEG dataset
57 desired sparsity (ρ) and the average activation value for the (SEED) EEG database for emotion recognition [18], [46]. The
58 j th hidden neuron (ρj ) (total number of hidden neurons as database holds EEG signal recordings acquired from 8 male
59 ‘p’), respectively. The Kullback–Leibler divergence is used to and 7 female subjects. Each individual participated in three
60
Page 7 of 16
6

1
2 1 col was followed while recording 62 channel EEG signals at
2
3 3 1000 Hz sampling rate. The recorded EEG signals were down
4
4 5 sampled at 200 Hz and further band-pass filtered between 0.5-
Subband signals

6
5 7 70 Hz in order to remove noise and artefacts. For detailed
6 8 information about the database see [46]. The database is
9
7 10 summarised in Table I. Our selection of the EEG channels in
11
8 12
13
order to facilitate emotion recognition is based on the results
9 14
15
obtained in a previous study [18]. In [18], authors found
10 16
17
the following channels most appropriate for EEG emotion
11 18 classification: C5, C6, CP5, CP6, FT7, FT8, P7, P8, T7, T8,
19
12 20 TP7, and TP8. From this list we have considered the sym-
13 0 0.2 0.4 0.6 0.8 1
metric bivariate channels resulting into six bivariate signals
14 Time in seconds namely C5-C6, CP5-CP6, FT7-FT8, P7-P8, T7-T8, and TP7-
15 TP8. We have analyzed last 30 seconds of each EEG signal
F IGURE 6: Plot of the FBSE-EWT subband signals for the recording (following the previous literature standard [21])
16
neutral emotion. and segmented them into one second duration epochs. Fig.
17
18 4 presents the sample EEG signals for positive, neutral, and
19 negative emotions. It can be noticed that the visual analysis of
1 different emotion EEG waveforms is not discriminatory. In this
20 2
3 context, advanced signal processing and analysis techniques
21 4
5
Subband signals

22 6
could provide useful information which is detailed in the
23 7 following section.
8
24 9
10
25 11
12 IV. E XPERIMENTAL R ESULTS
26 13
14
27 15 In this section we present the obtained results using propo-
16
28 17 sed entropy features and ARF classifier for classification of
18 positive, neutral, and negative emotions. We have segmented
29 19
20
30 each 30 second duration bivariate EEG signals into one second
0 0.2 0.4 0.6 0.8 1 duration epochs for further analysis. The multivariate FBSE-
31
Time in second EWT (MFBSE-EWT) method has been applied for obtaining
32
33 F IGURE 7: Plot of the FBSE-EWT subband signals for the the subband signals from each bivariate EEG epochs. The
34 negative emotion. number of subband signals has been fixed to 20 irrespective
35 of the type of EEG signals. The subband signals for positive,
36 neutral, and negative emotion EEG signals of channel T7 are
37 presented in Figs. 5, 6, and 7, respectively. These subband
experiments held at an interval of one week or longer. Fifteen signals are narrow band in nature and carry information in
38 Chinese film clips (containing both scene and audio) were
39 multiple oscillatory levels of the EEG signals. In the next step,
used as visual stimuli in order to evoke positive, neutral, and we have computed JIA and JIF functions from multivariate
40 negative emotions. In an experiment, each emotion comprises
41 subband signals in order to derive MHMS corresponding to
five film clips with each of 4 minutes duration. The film clips different scale factors (see Section II-A for the computation
42 were selected based on following criteria : (a) The whole
43 of MHMS). Fig. 8 presents MHMS of positive, neutral, and
length of the experiment should not make subjects fatigue ; (b) negative emotion EEG signals for scale factors 2, 5, 10, and
44 The videos are self-explanatory and easy to understand ; (c)
45 20, respectively. It can be observed that for low scale factor,
The videos should induce desired target emotion. Moreover, MHMS is present only in the high frequency region whereas
46 each film clip was edited in order to elicit coherent emotions
47 with increase of scale factors the low frequency components
and maximize emotional sense. appear gradually. The reason for such can be understood by
48
49 TABLE I: Information about the SEED emotion EEG observing the obtained subbands using MFBSE-EWT (see
50 database Figs. 5, 6, 7) and analyzing equations 9 and 10. For lower scale
51 factor, the high frequency subbands are selected for generating
Number of Sampling 62 available
52 Subjects/film rate Channels the scaled time-frequency representation. As an example, when
53
clips shown/ FP1 FPZ FP2 AF3 AF4 F7 F5 F3 F1 FZ F2 scale factor is 2, the first two highest oscillatory subbands
clip duration F4 F6 F8 FT7 FC5 FC3 FC1 FCZ FC2 FC4
54 FC6 FT8 T7 C5 C3 C1 CZ C2 C4 C6 T8 TP7
are used for time-frequency representation. As a result, the
55 15 (8 Male &
200 Hz
CP5 CP3 CP1 CPZ CP2 CP4 CP6 TP8 signal energy distribution will be limited to only high fre-
7 Female)/ P7 P5 P3 P1 PZ P2 P4 P6 P8 PO7 PO5
56 15/4 minutes PO3 POZ PO4 PO6 PO8 CB1 O1 OZ O2 CB2
quency region of time-frequency plane and subsequently the
57 derived MHMS will contain only high frequency spectral
58 components. Further increment of the scale factor increases
59 The International 10-20 system electrode positioning proto- the low frequency components in the derived MHMS. This
60
 Page 8 of 16
7

1
2 1000 1000
3
4
5 500 500
6 Positive
7
Magnitude

emotion
8 0 Neutral 0
emotion
9 1000
Negative
2000

10 emotion 1500
11
12 500 1000
13 500
14
15 0 0
20 40 60 80 100 20 40 60 80 100
16
Frequency in Hz
17
18
F IGURE 8: Plots of the FBSE-EWT based MHMSs for scale factors 2 (top left), 5 (top right), 10 (bottom left), and 20
19
(bottom right), respectively.
20
21
22 TABLE II: Computed TP rates and accuracy (Acc) values using spectral and temporal entropy features and RF classifier for
23 EEG emotion classification.
24 Channel Temporal Spectral
25 pair
Emotion
K-NN entropy Shannon entropy K-NN entropy
26 TP (%)
Acc (%)
TP (%)
Acc (%)
TP (%)
Acc (%)
27 Positive 81.3 86.9 73.8
28 FT7-FT8 Neutral 80.0 78.1 85.8 84.5 70.6 71.5
29 Negative 73.1 80.9 70.2
Positive 91.4 90.6 79.8
30 T7-T8 Neutral 82.3 83.9 86 86.7 74.8 74.6
31 Negative 77.9 83.5 69.2
32 Positive 85.0 87.4 74.8
C5-C6 Neutral 82.3 80.9 86.8 85.2 72.0 72.3
33 Negative 75.3 81.5 70.1
34 Positive 86.7 86.1 74.4
35 TP7-TP8 Neutral 85.5 82.5 86.7 84.4 74.8 72.2
Negative 75.4 80.3 67.2
36 Positive 83.7 83.5 72.4
37 CP5-CP6 Neutral 82.0 79.6 88.1 83.4 72.5 70.5
38 Negative 73.1 78.6 66.6
Positive 86.8 85.6 77.1
39 P7-P8 Neutral 85.2 82.6 87.3 84.5 73.2 73.3
40 Negative 75.8 80.5 69.6
41
42
43 helps us to compute the entropy features in multiple scales of size of the moving average filter is chosen in order to achieve
44 the MHMS. From the figure it is also evident that magnitude optimal classification performance. The feature smoothing has
45 for positive emotion EEG signal MHMS is significantly higher been proved to be successful and recommended in the previous
46 than its neutral and negative emotion counterpart. The inter- research works on determining human emotional states [17],
47 class distribution of the spectral and temporal entropy features [21]. We have tested the performance of the individual feature
48 for scale factor 10 have been depicted in Fig. 9. It is evident vectors namely spectral K-NN entropy (R20×1 ), spectral Shan-
49 that the mean of the spectral and temporal entropy features for non entropy (R20×1 ), and temporal K-NN entropy (R20×1 )
50 positive emotion EEG are significantly high as compared to when fed to RF classifier. The obtained results using these
51 other two classes. The mean values of the computed features features have been presented in Table II for each considered
52 are highest for positive emotion EEG signals followed by channel pair. It can be observed from Table II that the highest
53 negative and neutral emotions. However, a significant overlap classification accuracy of 86.7% is reported for channel pair
54 can be observed in the range of feature values for neutral T7-T8 among all considered channel pairs with spectral Shan-
55 and negative emotions. The features computed from each 30 non entropy feature. The accuracy values reported for spectral
56 second EEG signals have been smoothed by a moving average K-NN entropy features have been found significantly less in
57 filter with a window size of 10 samples for getting rid of the comparison to other two feature vectors. While looking at the
58 rapid fluctuations in the extracted raw features. The window true positive (TP) rates in Table II it is evident that positive
59
60
Page 9 of 16
8

1
TABLE III: Computed TP rates and Acc values using RF and ARF classifiers for the classification of positive, neutral, and
2
negative emotions.
3
4 Channel RF Classifier ARF classifier
pair 30 Hidden neurons 50 Hidden neurons
5 Emotion
Acc (%) Acc (%) Acc (%)
6 TP (%) TP (%) TP (%)
7 Positive 92.6 93.7 93.7
FT7-FT8 Neutral 90.9 89.8 93.3 91.8 94.2 92.2
8 Negative 85.8 88.4 88.6
9 Positive 96.1 96.1 96.3
10 T7-T8 Neutral 91.4 92.1 94.3 94.1 94.8 94.4
Negative 88.6 91.8 91.9
11
Positive 93.0 93.5 94.0
12 C5-C6 Neutral 91.1 90.5 93.6 92.1 94.0 92.3
13 Negative 87.3 89.1 88.8
14 Positive 92.7 93.3 93.5
TP7-TP8 Neutral 91.5 90.5 94.0 92.0 94.7 92.7
15 Negative 87.4 88.6 90.1
16 Positive 89.9 91.4 92.1
17 CP5-CP6 Neutral 91.6 88.8 92.8 90.4 93.6 91.2
Negative 84.9 87 87.9
18 Positive 91.5 91.3 91.8
19 P7-P8 Neutral 92.4 90.4 92.9 90.9 92.7 91.1
20 Negative 87.4 88.5 88.7
21
22
23 4 100
15
24
Time domain K-NN entropy

30
Spectral Shannon entropy

Spectral K-NN entropy

Acc and TP rate (%)

3.5
25 25
95
26 3
10 20
27 2.5
15 90
28 2 Acc
10
29 5 85 Positive
1.5
30 5
Neutral
31 1 0
80 Negative
32 Pos Neu Neg Pos Neu Neg Pos Neu Neg
33 5 10 15 20 25 30 35 40 45 50
34 F IGURE 9: Boxplots of the computed multivariate entropy
35 features for scale factor 10. The abbreviations Pos, Neu, and
Neg stand for positive, neutral and negative emotions F IGURE 10: Plot of Number of hiddenofneurons
the variations the classification
36
respectively. accuracy and TP rate with increased number of hidden
37
neurons in ARF classifier.
38
39
40
41 emotion recognition is performed with high accuracy whereas
42 the accuracy of recognizing negative emotion is significantly are considered in the sparse autoencoder layer, the overall
43 low. We have further increased the dimension of the feature accuracy value is further improved for each EEG channel pair,
44 vector (R60×1 ) by concatenating spectral and temporal multis- and the maximum accuracy value of 94.4% is obtained for
45 cale entropy features. The high dimensional feature vector has T7-T8 channel. The variation of TP rate and classification
46 been tested for classifying different emotional states using RF accuracy with respect to number of neurons in the hidden
47 and proposed ARF classifier. Table III presents the obtained layer of the sparse autoencoder is depicted in Fig. 10. It is
48 TP rates and classification accuracies for all the considered evident that even with 15 hidden neurons the classification
49 channels pairs. It is evident that ARF classifier achieved higher accuracy of ARF classifier is almost comparable with the RF
50 classification accuracies using high dimensional feature vector. classifier. However, with further reduction in number of hidden
51 The accuracy values of ARF classifier with 30 hidden neurons neurons the TP rate and accuracy value dropped significantly.
52 are improved more than 1.5% for FT7-FT8, T7-T8, C5-C6, The improvement/variation of the performance parameters
53 and CP5-CP6 channel pairs as compared to the performance has not been found significant with the increase of hidden
54 of RF classifier. The rationale behind this is straight forward neurons beyond 50 and hence not reported. The confusion
55 as the sparse autoencoder layer in ARF classifier projects the matrices using ARF classifier (with 50 hidden neurons) for
56 correlated features of the EEG channel pairs into a new di- each bivariate channel have been presented in Fig. 11. The
57 mensional space due to the nonlinearity of its output activation rows and columns of the confusion matrix represent the true
58 function. In this new space (hidden layer space), the features class and predicted class, respectively. The element aij in each
59 become uncorrelated [31]. Moreover, when 50 hidden neurons confusion matrix denotes number of EEG epochs originally
60
 Page 10 of 16
9

1
Channel pair FT7-FT8 Channel pair T7-T8
2 Channel pair C5-C6

3 Negative 5982 478 290 Negative 6205 373 172 Negative 5992 481 277
4 88.6% 91.9% 88.8%
True class

True class
5

True class
6 Neutral 197 6356 197 Neutral 234 6402 114 Neutral 236 6343 171
7 94.2% 94.8% 94%

8
Positive 193 229 6328 Positive 151 100 6499 Positive 208 196 6346
9 93.7% 96.3% 94%
10 Negative Neutral Positive Negative Neutral Positive Negative Neutral Positive
11 Predicted class Predicted class Predicted class
12 Channel pair CP5-CP6 Channel pair P7-P8
Channel pair TP7-TP8
13
14 Negative 6079 429 242 Negative 5934 522 294 Negative 5984 480 286
15 90.1% 87.9% 88.7%

True class
True class
True class

16
Neutral 365 6258 127
17 Neutral 230 6392 128 Neutral 237 6320 193
93.6% 92.7%
18 94.7%

19 Positive 253 188 6309 Positive 238 292 6220 Positive 334 220 6196
20 93.5% 92.1% 91.8%
21 Negative Neutral Positive Negative Neutral Positive Negative Neutral Positive
22 Predicted class Predicted class Predicted class
23 F IGURE 11: Confusion matrices for the considered channel pairs using ARF classifier for classification of emotions with 50
24 hidden neurons.
25
26
27 TABLE IV: Performance evaluation using SVM classifier and
28 belonging to class i but classified as class j (i 6= j). It
MEMD method in SEED emotion database. can be observed that ARF classifier in most of the occasions
29
30 Channel MFBSE-EWT MEMD correctly predicts the positive emotions. However, the classifier
pair Emotion and SVM and RF confuses to correctly classify negative emotions. Therefore,
31
TP (%) Acc (%) TP (%) Acc (%) it can be stated that recognition of positive emotion can be
32
33
Positive 80.2 80.6 carried out with high accuracy, whereas negative emotion
FT7-FT8 Neutral 70.6 74.4 79.7 79.3
34 Negative 72.3 77.6
recognition is a challenging task. This fact was also revealed
35 Positive 89.5 86.1 in the previous emotion recognition research [18]. However,
T7-T8 Neutral 67.7 77.5 78.7 80.8 the correctly classified instances for all three emotional states
36 Negative 75.2 77.6
37 Positive 78.8 81.7 using ARF classifier are significantly higher than the existing
38
C5-C6 Neutral 67.1 74 79 79.8 RF classifier.
Negative 76 78.9
39 Positive 86.1 81.1 TABLE VI: Comparison of the proposed emotion recognition
40 TP7-TP8 Neutral 64.1 71.4 78.3 78.2
method with existing methods evaluated in SEED EEG
Negative 64 75.1
41 Positive 73.8 82.2 database.
42 CP5-CP6 Neutral 67.1 65.9 76.3 76
Negative 56.7 69.6 Authors Computed features Classifier Acc (%)
43 Positive 84.4 81.2 Time-frequency
Li et al [20] SVM 83.33
44 P7-P8 Neutral 70.1 70 75.3 75.1 features
45 Negative 55.5 68.7
Gupta et al [21]
FAWT based
RF 93.46
IP features
46 MFBSE-EWT
47 TABLE V: Performance evaluation using mono-channel EEG Proposed work based spectral ARF 94.4
48 signals. and temporal entropy
49 FBSE-EWT and
50 Channel Emotion ARF classifier
51 TP (%) Acc (%)
A. Emotion recognition with MEMD and SVM classifier
Positive 94.2
52 T7 Neutral 92.5 91.9
53 In this subsection, we have presented the performance of
Negative 89.2
54 Positive 92.3 MEMD method and SVM classifier for recognition of human
55 T8 Neutral 92.3 90.7 emotions. The intrinsic mode functions (IMFs) are extracted
Negative 87.4 from multichannel signals after decomposition using MEMD
56
57 method. The temporal and spectral entropies are computed
58 from the multivariate IMFs in the time domain and MHMS
59 domain (see Section II-A for MHMS), respectively. As the
60
Page 11 of 16
10

1
2 multivariate IMFs differed in number across different epochs signals. Therefore, in this work we have proposed MFBSE-
3 of the signals, minimum 8 IMFs are considered for feature EWT based MHMS and used it for spectral analysis of
4 extraction. This resulted into a feature vector of dimension emotion EEG signals. The spectral Shannon entropy is an
5 R24×1 by considering the temporal and spectral features. useful measure for computing the signal energy distribution
6 Finally, RF classifier is used for classification of different over a group of frequency bands [50]. In this work, we
7 emotional states. The performance of the MEMD-RF method have found significant discrimination with spectral Shannon
8 for emotion recognition is compared with MFBSE-EWT me- entropy features in distinguishing emotion EEG signals. The
9 thod coupled with SVM classifier and results are presented in K-NN entropy is a distance based entropy measure which
10 Table IV. It can be observed that MEMD-RF method achieves have been found useful in numerous research works [26],
11 higher classification accuracies for all six considered channel [53]. Here, K-NN entropy is estimated for measuring spectral
12 pairs. However, the performance of the proposed MFBSE- and temporal complexity of emotion EEG signals. Further, we
13 EWT method coupled with ARF classifier provides superior have verified that the ARF classifier coupled-with the spectral
14 performance among all the methods discussed here in the and temporal entropy features of EEG signal has significant
15 recognition of human emotions. classification performance for the discrimination of positive,
16 neutral, and negative categories of emotions. Moreover, we
17 have compared the performance of the proposed multivariate
B. Emotion recognition with mono-channel EEG signals multiscale entropy features with other cross-subject based
18
19 In this subsection, we have experimentally demonstrated emotion recognition methods in Table VI. Li et al. [20] used
20 the higher efficacy of multichannel EEG signal analysis over different nonlinear features with automatic feature selection
21 mono-channel EEG signals for human emotion recognition. methods and reported emotion recognition accuracy of 83.33%
22 For this purpose, two single channel EEG signals from two with SVM classifier. Gupta et al. [21] have extracted in-
23 different electrodes (T7 and T8) are considered. The FBSE- formation potential (IP) features from the subband signals
24 EWT method decomposes the single channel EEG signal which are evaluated using the FAWT of EEG signal. They
25 into narrow-band subband signals which are used for time- have reported an average accuracy value of 93.46% using RF
26 frequency representation. The temporal and spectral entropies classifier for T7 EEG channel. However, the proposed method
27 are computed in the time domain and HMS domain of the has higher performance as compared to the IP based features.
28 generated subband signals. Finally, ARF classifier classifies The major developments in the proposed work can be stated
29 the extracted features into different emotional states. The as follows. The novel multivariate multiscale entropy features
30 obtained classification accuracies are reported in Table V. are proposed for the analysis and classification of multichannel
31 It can be noticed that T7 channel has provided the highest EEG signals. The ARF classifier which is constructed based
32 classification accuracy of 91.9% which is lesser than the on the hidden layer from sparse autoencoder and RF is used
33 classification accuracy of 94.4% achieved from T7-T8 channel for the classification of emotions. The method is evaluated
34 pair. This supports the advantage of multichannel EEG signal for different EEG channel pairs and results are presented
35 analysis over it single channel counterpart in human emotion for every distinct channel pair. The method attains highest
36 recognition. classification accuracy as compared to the other approaches.
37 It should be noted that the method can be easily extended
38 for higher number of channels such as 6-channels or 12-
V. D ISCUSSION
39 channels with higher computation burden. Therefore, the entire
40 The objective of this research work is the classification experiment is not repeated for increased number of channels.
41 of positive, neutral, and negative emotions from multichannel However, we have analyzed the computational difference for
42 EEG signals using novel MFBSE-EWT based approach. The 2 and 12-channel signals. The decomposition time for 12-
43 negative emotions such as anxiety, depression are the major channels using MFBSE-EWT is approximately 5 time higher
44 risk factors for the development of cardiac and neurological than the 2-channel case. Further, the computation time of
45 diseases [47], [48]. The positive emotional states play an MHMS for 12-channels is 2.6 time higher than the 2-channel
46 important role in maintaining better health and quality life case. It should be noted that, the reliability of emotion recogni-
47 [49]. Therefore, the classification of positive and negative tion depends on several factors namely measurement method,
48 emotions from physiological signals is of prime important sensors employed, developed signal processing, and analysis
49 in affective computing. In our work, the multivariate signal technique [54]. The decrease in reliability and quality of
50 decomposition using FBSE-EWT provides narrow band signal emotion recognition may be observed with increase in number
51 adaptive components/subbands which carries information in of subjects and recognized emotions [55]. This proposes a new
52 multiple oscillatory levels. The prime objective of the proposed direction for the setup of new protocols and planning in the
53 method is to compute the temporal and spectral complexity of research of emotion recognition.
54 those subbands. It should be noted that the discrete Fourier
55 transform based spectrum is not well suited for the spectral
analysis of non-stationary signals [50], [51]. In this context, A. Validation of the method
56
57 the use of HMS for spectral analysis of non-stationary signals We have validated the performance of the proposed method
58 is not uncommon in the literature [50]–[52]. However, most of in publicly available DREAMER emotion recognition data-
59 the HMSs defined in the literature are intended for univariate base [56]. The database is a multi-modal database containing
60
 Page 12 of 16
11

1
2 electrocardiogram (ECG) and EEG recordings acquired during time domain complexity of EEG signals are measured with
3 audio-visual stimuli based affect elicitation. The physiological proposed multiscale K-NN entropy measure. The ARF classi-
4 signals were recorded from 23 volunteers who self assessed fier has been formulated using the combination of hidden layer
5 their emotional states for each of the 18 selected film clips of sparse autoencoder and RF classifier for the recognition of
6 shown to them. The participants rated their perceived arousal, positive, neutral, and negative emotions. The proposed method
7 dominance, and valance on five point scales. The International achieved higher classification accuracy as compared to the
8 10-20 system electrode positioning protocol was followed for existing state-of-the-art methods for human emotion recogni-
9 recording the multichannel (AF3, F7, F3, FC5, T7, P7, O1, O2, tion using EEG signals. In this study, we have only computed
10 P8, T8, FC6, F4, F8, AF4) EEG signals at a sampling rate of KNN-entropy and spectral Shannon entropy features in the
11 128 Hz using Emotive EPOC system. On the other hand, ECG proposed multivariate multiscale framework. The non-linear
12 signals were recorded using SHIMMERTM wireless sensor entropy features such as sample entropy, permutation enrtropy
13 at a sampling rate of 256 Hz. The details of this database is etc. [23]–[25] can also be employed to evaluate the temporal
14 summarised in Table VII. In this work, we have used bivariate and spectral domain complexity in the proposed framework
15 EEG signals from the channel pair T7-T8 for the recognition for the analysis of multichannel EEG and other multivariate
16 of felt emotions. physiological signals. In the present work, we have used
17 two public databases namely SEED and DREAMER emotion
TABLE VII: Information about the DREAMER emotion databases for validation of our method. In future, we plan
18 database
19 to record physiological signals using low cost and portable
20 Number of Modality system for recognition of human emotions considering large
Subjects/film EEG ECG
21 clips shown/ sampling sampling number of subjects.
128 Hz 256 Hz
22 clip duration
23/18/
rate
AF3 F7 F3 FC5 T7 P7 O1
rate
2
23 (65–393 s)
Channels
O2 P8 T8 FC6 F4 F8 AF4
Channels
Channels R ÉFÉRENCES
24
[1] S. M. Alarcao and M. J. Fonseca, “Emotions recognition using EEG
25 signals : a survey,” IEEE Transactions on Affective Computing, 2017.
26 Three classification tasks have been carried out namely
[2] M. Ptaszynski, P. Dybala, W. Shi, R. Rzepka, and K. Araki, “Towards
27 high/low arousal, high/low dominance, high/low valence. The context aware emotional intelligence in machines : computing contextual
28 felt emotions are considered as high in case when users appropriateness of affective states,” in Twenty-First International Joint
rated their felt arousal, dominance, and valence as four or Conference on Artificial Intelligence, 2009.
29 [3] I. B. Mauss and M. D. Robinson, “Measures of emotion : A review,”
30 more, low otherwise. The bivariate EEG signals are segmen- Cognition and Emotion, vol. 23, no. 2, pp. 209–237, 2009.
31 ted into epochs of two seconds duration and decomposed [4] A. M. Al-Kaysi, A. Al-Ani, C. K. Loo, T. Y. Powell, D. M. Martin,
32 using MFBSE-EWT method followed by the computation M. Breakspear, and T. W. Boonstra, “Predicting tDCS treatment out-
of multivariate-multiscale spectral and temporal entropies. comes of patients with major depressive disorder using automated EEG
33 classification,” Journal of Affective Disorders, vol. 208, pp. 597–603,
34 Finally, ARF classifier is used in order to perform the binary 2017.
35 classification tasks as mentioned earlier. Table VIII presents [5] A. V. Bocharov, G. G. Knyazev, and A. N. Savostyanov, “Depression
36 the classification performance of the proposed method and and implicit emotion processing : an EEG study,” Neurophysiologie
compares the same with previous works carried out in this Clinique/Clinical Neurophysiology, vol. 47, no. 3, pp. 225–230, 2017.
37 [6] W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, “Emotionmeter :
38 database. We have achieved highest classification accuracy A multimodal framework for recognizing human emotions,” IEEE
39 of 86.2% in classifying high and low valence states. The Transactions on Cybernetics, no. 99, pp. 1–13, 2018.
40 proposed method outperforms both the existing works in terms [7] S. Deb and S. Dandapat, “Multiscale amplitude feature and significance
of classification accuracy for all the classification tasks. of enhanced vocal tract information for emotion classification,” IEEE
41 Transactions on Cybernetics, no. 99, pp. 1–14, 2018.
42 TABLE VIII: Performance comparison of the proposed and [8] G. L. Ahern and G. E. Schwartz, “Differential lateralization for positive
43 existing methods in DREAMER emotion database.
and negative emotion in the human brain : EEG spectral analysis,”
Neuropsychologia, vol. 23, no. 6, pp. 745–755, 1985.
44 [9] C. Niemic, “Studies of Emotion : A Theoretical and Empirical Review
45 Emotion MFBSE-EWT PSD features and Deep neural
of Psychophysiological Studies of Emotion.” 2004.
recognition and ARF SVM [56] networks [57]
46 (Acc %) (Acc %) (Acc %) [10] M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, I. Zunaidi, and
47 Arousal 85.4 62.17 79.95 D. Hazry, “EEG feature extraction for classifying emotions using FCM
and FKM,” International Journal of Computers and Communications,
48 Valence 86.2 62.49 79.95
vol. 1, no. 2, pp. 21–25, 2007.
Dominance 84.5 61.84 —
49 [11] M. Soleymani, M. Pantic, and T. Pun, “Multimodal emotion recognition
50 in response to videos,” IEEE Transactions on Affective Computing,
51 vol. 3, no. 2, pp. 211–223, 2012.
[12] V. Bajaj and R. B. Pachori, “Detection of human emotions using features
52 VI. C ONCLUSION based on the multiwavelet transform of EEG signals,” in Brain-Computer
53 Interfaces. Springer, 2015, pp. 215–240.
54 In this paper, novel multivariate multiscale entropy features [13] M. Li and B.-L. Lu, “Emotion classification based on gamma-band
55 are proposed for emotion recognition using multichannel EEG EEG,” in 2009 Annual International Conference of the IEEE Enginee-
signals. The FBSE-EWT based MHMS is proposed based on ring in medicine and biology society. IEEE, 2009, pp. 1223–1226.
56 [14] D. Iacoviello, A. Petracca, M. Spezialetti, and G. Placidi, “A classi-
57 successive selection of JIA and JIF functions. Further, K-NN
fication algorithm for electroencephalography signals by self-induced
58 and Shannon entropies are computed from the multivariate emotional stimuli,” IEEE Transactions on Cybernetics, vol. 46, no. 12,
59 multiscaled HMSs for measuring spectral complexity. The pp. 3171–3180, 2016.

60
Page 13 of 16
12

1
2 [15] S. K. Hadjidimitriou and L. J. Hadjileontiadis, “EEG-based classification [36] N. E. Huang et al., “The empirical mode decomposition and the Hilbert
of music appraisal responses using time-frequency analysis and familia- spectrum for nonlinear and non-stationary time series analysis,” in
3 rity ratings,” IEEE Transactions on Affective Computing, vol. 4, no. 2, Proceedings of the Royal Society of London A : Mathematical, Physical
4 pp. 161–172, 2013. and Engineering Sciences, vol. 454, no. 1971, 1998, pp. 903–995.
5 [16] H. J. Yoon and S. Y. Chung, “EEG-based emotion estimation using [37] A. Ahrabian et al., “Synchrosqueezing-based time-frequency analysis of
6 Bayesian weighted-log-posterior function and perceptron convergence multivariate data,” Signal Processing, vol. 106, pp. 331 – 341, 2015.
algorithm,” Computers in Biology and Medicine, vol. 43, no. 12, pp. [38] K. A. Veselkov, V. I. Pahomov, J. C. Lindon, V. S. Volynkin, D. Crock-
7 2230–2237, 2013. ford, G. S. Osipenko, D. B. Davies, R. H. Barton, J.-W. Bang, E. Holmes
8 [17] R.-N. Duan, J.-Y. Zhu, and B.-L. Lu, “Differential entropy fea- et al., “A metabolic entropy approach for measurements of systemic me-
9 ture for EEG-based emotion classification,” in 2013 6th International tabolic disruptions in patho-physiological states,” Journal of Proteome
IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 2013, Research, vol. 9, no. 7, pp. 3537–3544, 2010.
10 pp. 81–84. [39] A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual
11 [18] W.-L. Zheng and B.-L. Lu, “Investigating critical frequency bands information,” Physical review E, vol. 69, no. 6, p. 066138, 2004.
12 and channels for EEG-based emotion recognition with deep neural [40] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp.
13 networks,” IEEE Transactions on Autonomous Mental Development, 5–32, 2001.
vol. 7, no. 3, pp. 162–175, 2015. [41] J. Xu, L. Xiang, Q. Liu, H. Gilmore, J. Wu, J. Tang, and A. Madabhushi,
14 “Stacked sparse autoencoder (SSAE) for nuclei detection on breast
[19] T. Zhang, W. Zheng, Z. Cui, Y. Zong, and Y. Li, “Spatial-temporal
15 recurrent neural network for emotion recognition,” IEEE Transactions cancer histopathology images,” IEEE Transactions on Medical Imaging,
16 on Cybernetics, no. 99, pp. 1–9, 2018. vol. 35, no. 1, pp. 119–130, 2016.
17 [20] X. Li, D. Song, P. Zhang, Y. Zhang, Y. Hou, and B. Hu, “Exploring [42] A. Ng et al., “Sparse autoencoder,” CS294A Lecture notes, vol. 72, no.
2011, pp. 1–19, 2011.
18 EEG features in cross-subject emotion recognition,” Frontiers in Neu-
[43] R. Tripathy, L. Sharma, and S. Dandapat, “Detection of shockable
roscience, vol. 12, p. 162, 2018.
19 [21] V. Gupta, M. D. Chopda, and R. B. Pachori, “Cross-Subject Emotion ventricular arrhythmia using variational mode decomposition,” Journal
20 Recognition Using Flexible Analytic Wavelet Transform From EEG of Medical Systems, vol. 40, no. 4, p. 79, 2016.
21 Signals,” IEEE Sensors Journal, vol. 19, no. 6, pp. 2266–2274, 2019. [44] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern classification. John
Wiley & Sons, 2012.
22 [22] A. Mert and A. Akan, “Emotion recognition from eeg signals by
[45] M. Sokolova and G. Lapalme, “A systematic analysis of performance
using multivariate empirical mode decomposition,” Pattern Analysis and
23 Applications, vol. 21, no. 1, pp. 81–89, 2018. measures for classification tasks,” Information Processing & Manage-
24 [23] M. Costa et al., “Multiscale entropy analysis of complex physiologic
ment, vol. 45, no. 4, pp. 427–437, 2009.
25 time series,” Physical Review Letters, vol. 89, no. 6, p. 068102, 2002. [46] W.-L. Zheng, J.-Y. Zhu, and B.-L. Lu, “Identifying stable patterns over
time for emotion recognition from EEG,” IEEE Transactions on Affective
26 [24] J. F. Valencia et al., “Refined multiscale entropy : Application to 24-h Computing, 2017.
holter recordings of heart period variability in healthy and aortic stenosis
27 subjects,” IEEE Transactions on Biomedical Engineering, vol. 56, no. 9,
[47] D. F. Smith, “Negative emotions and coronary heart disease : causally
28 pp. 2202–2213, 2009.
related or merely coexistent ? A review,” Scandinavian Journal of
Psychology, vol. 42, no. 1, pp. 57–69, 2001.
29 [25] H. Amoud et al., “Intrinsic mode entropy for nonlinear discriminant [48] K. M. Heilman, D. Bowers, and E. Valenstein, “Emotional disorders
30 analysis,” IEEE Signal Processing Letters, vol. 14, no. 5, pp. 297–300, associated with neurological diseases,” Clinical Neuropsychology, vol. 3,
2007.
31 pp. 461–97, 1993.
[26] A. Bhattacharyya et al., “Tunable-Q wavelet transform based multiscale
32 entropy measure for automated classification of epileptic EEG signals,”
[49] M. D. Avia, “Personality and positive emotions,” European Journal of
Personality, vol. 11, no. 1, pp. 33–56, 1997.
33 Applied Sciences, vol. 7(4), no. 385, pp. 1–18, 2017.
[50] A. Humeau-Heurtier et al., “Refined Multiscale Hilbert–Huang Spectral
34 [27] A. Bhattacharyya, R. Ranta, S. Le Cam, V. Louis-Dorr, L. Tyvaert, Entropy and Its Application to Central and Peripheral Cardiovascular
35 S. Colnat-Coulbois, L. Maillard, and R. B. Pachori, “A multi-channel Data,” IEEE Transactions on Biomedical Engineering, vol. 63, no. 11,
approach for cortical stimulation artefact suppression in depth EEG pp. 2405–2415, 2016.
36 signals using time-frequency and spatial filtering,” IEEE Transactions [51] K. Fu et al., “Hilbert marginal spectrum analysis for automatic seizure
37 on Biomedical Engineering, 2018. detection in EEG signals,” Biomedical Signal Processing and Control,
38 [28] A. Bhattacharyya, L. Singh, and R. B. Pachori, “Fourier–Bessel series vol. 18, pp. 179–185, 2015.
expansion based empirical wavelet transform for analysis of non-
39 stationary signals,” Digital Signal Processing, vol. 78, pp. 185–196,
[52] A. Bhattacharyya, V. Gupta, and R. B. Pachori, “Automated iden-
tification of epileptic seizure EEG signals using empirical wavelet
40 2018. transform based Hilbert marginal spectrum,” in 2017 22nd International
41 [29] J. Gilles, “Empirical wavelet transform,” IEEE Transactions on Signal Conference on Digital Signal Processing (DSP). IEEE, 2017, pp. 1–5.
42 Processing, vol. 61, no. 16, pp. 3999–4010, 2013. [53] M. Kumar, R. B. Pachori, and U. R. Acharya, “An efficient automated
43 [30] A. Bhattacharyya and R. B. Pachori, “A multivariate approach for patient technique for CAD diagnosis using flexible analytic wavelet transform
specific EEG seizure detection using empirical wavelet transform,” and entropy features extracted from HRV signals,” Expert Systems with
44 IEEE Transactions on Biomedical Engineering, to be published, DOI Applications, vol. 63, pp. 165–172, 2016.
45 10.1109/TBME.2017.2650259. [54] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, “Human Emotion
46 [31] R. K. Tripathy, A. Bhattacharyya, and R. B. Pachori, “A Novel Approach Recognition : Review of Sensors and Methods,” Sensors, vol. 20, no. 3,
for Detection of Myocardial Infarction from ECG Signals of Multiple p. 592, 2020.
47 Electrodes,” IEEE Sensors Journal, 2019. [55] K. Takahashi et al., “Remarks on emotion recognition from bio-potential
48 [32] A. Bhattacharyya, M. Sharma, R. B. Pachori, P. Sircar, and U. R. signals,” in 2nd International conference on Autonomous Robots and
49 Acharya, “A novel approach for automated detection of focal EEG Agents, 2004, pp. 186–191.
50 signals using empirical wavelet transform,” Neural Computing and [56] S. Katsigiannis and N. Ramzan, “Dreamer : A database for emotion
Applications, vol. 29, no. 8, pp. 47–57, 2018. recognition through eeg and ecg signals from wireless low-cost off-
51 the-shelf devices,” IEEE Journal of Biomedical and Health Informatics,
[33] N. E. Huang, M.-L. C. Wu, S. R. Long, S. S. Shen, W. Qu, P. Gloersen,
52 and K. L. Fan, “A confidence limit for the empirical mode decomposition vol. 22, no. 1, pp. 98–107, 2017.
53 and hilbert spectral analysis,” Proceedings of the Royal Society of [57] S. Siddharth, T.-P. Jung, and T. J. Sejnowski, “Utilizing deep learning
towards multi-modal bio-sensing and vision-based affective computing,”
54 London. Series A : Mathematical, Physical and Engineering Sciences,
IEEE Transactions on Affective Computing, 2019.
vol. 459, no. 2037, pp. 2317–2345, 2003.
55
[34] J. Schroeder, “Signal Processing via Fourier-Bessel Series Expansion,”
56 Digital Signal Processing, vol. 3, no. 2, pp. 112–124, 1993.
57 [35] R. B. Pachori and P. Sircar, “EEG signal analysis using FB expansion
58 and second-order linear TVAR process,” Signal Processing, vol. 88,
no. 2, pp. 415–420, 2008.
59
60
 Page 14 of 16

1
2
3
4
Manuscript ID: Sensors-33433-2020
5
6 A Novel Multivariate-Multiscale Approach for Computing EEG Spectral and Temporal Complexity for
7
Human Emotion Recognition
8
9
10
11 The authors would like to thank the reviewers for their valuable comments. We have addressed all
12
the comments in the revised manuscript. Please find below comments of the reviewer and our
13
14 responses. We have also highlighted the changes in blue coloured font in the revised manuscript.
15
16
17
18
19
20 Reviewer: 1
21
22 The authors have proposed novel multivariate-multiscale approach for classifying human emotions
23 based on the EEG signal. In my opinion this work has sufficient novelty for publication. But few
24 grammatical correction and clarification before the publication.
25
26 Comment 1: While describing the contributions of paper the author has done typing mistake and written
27 " temoral characteristics in of emotion EEG signal" in page 3, line no. 39. In my opinion temoral should
28 be temporal characteristics of emotions in the EEG signal.
29
30 Response: Thank you for pointing the typing mistake. We have corrected it in the revised paper.
31
32
Comment 2: In page no. 3 and line no. 23, Cite an appropriate pioneering reference paper for the work
33 on multivariate Hilbert marginal spectrum (MHMS).
34
Response: In the revised manuscript, we have cited the research article proposed Hilbert marginal
35
36 spectrum (HMS). We have extended HMS for multichannel signals and proposed multivariate Hilbert
37 marginal spectrum (MHMS).
38
39 Comment 3: In subsection C of section II named as "Computation of multiscale temporal entropy",
40 provide the appropriate spacing for writing Step 1 and Step 2. It seems that the authors has missed to
41 change the paragraph.
42
43 Response: In the revised manuscript we have taken care that change of paragraph.
44
Comment 4: In experiment section the author has chosen the window size of 30 for moving average
45
46 filter. Provide the appropriate reason for this.
47 Response: The features computed from each 30 second EEG signals have been smoothed by a moving
48
49
average filter with a window size of 10 samples in order to remove the rapid fluctuations in the extracted
50 raw features. The window size of the moving average filter is chosen in order to achieve optimal
51 classification performance. We have clearly mentioned this in the revised manuscript.
52
53
54
55
Reviewer 2:
56 It is an interesting paper for emotion recognition (positive, neutral, and negative). Fourier-Bessel series
57
expansion based empirical wavelet transform (FBSE-EWT), Hilbert spectrum, and entropy-based
58
59 features of SEED database are applied to sparse autoencoder based RF classifier (ARF)
60
Page 15 of 16

1
2
3 ARF structure and the multivariate extension of the FBSE-EWT can be considered to be contributions
4
of the paper, but the following issues should be clearer
5
6 Comment 1: Signal processing methods were not included to the simulation, and not compared. I’m
7 not sure that rates are caused by FBSE-EWT. For this reason, another multivariate decomposition
8
method should be added.
9
10 Response: Thank you for your suggestion. In the revised manuscript we have shown the performance
11
of multivariate empirical mode decomposition (MEMD). The results using MEMD are described in
12
13 Section IV-A and also summarised in Table IV.
14 Comment 2: Other classifiers have to be added.
15
16 Response: In the revised manuscript we have shown the performance of support vector machine
17 classifier (SVM) in human emotion recognition. The results using SVM classifier are described in
18
Section IV-A and also summarised in Table IV.
19
20 Comment 3: The advantage of bipolar chs should be explored by simulating the same approaches on
21
the mono-channels.
22
23 Response: Thank you for your advice. In the revised manuscript we have presented the performance of
24 the method using mono-channels. This is presented in Section IV-B and also summarised in Table V.
25
26 Comment 4: Please apply your method to DEAP or another multi-channel EEG to prove yours is
27 successful.
28
29 Response: Thank you for your suggestion. As we did not receive the access to the DEAP database after
30 multiple attempts, we have chosen DREAMER emotion database for the validation of our method. The
31 validation of our method in DREAMER emotion database is presented in Section V-A and also
32
summarised in Table VIII.
33
34
35
36 Comment 5: Please explain the aim of the selected channels. In other papers, frontal and pre-frontal
37 channels are generally used, but yours are different.
38
39 Response: Our selection of the EEG channels in order to facilitate emotion recognition is based on the
40 results obtained in a previous study [18]. In [18], authors found the following channels most appropriate
41 for EEG emotion classification: C5, C6, CP5, CP6, FT7, FT8, P7, P8, T7, T8, TP7, and TP8. From this
42
list we have considered the symmetric bivariate channels resulting into six bivariate signals namely C5-
43
44 C6, CP5-CP6, FT7-FT8, P7-P8, T7-T8, and TP7-TP8. We have mentioned this in the revised paper in
45 Section III.
46
47
Comment 6: It is not correct way to say 94.4%. Your paper is based multi-channel & multi-variate
48 processing, but the results are channel based. Please describe it.
49
Response: The method is evaluated for different EEG channel pairs and results are presented for every
50
51 distinct channel pair. The method attains highest classification accuracy as compared to the other
52 approaches. It should be noted that the method can be easily extended for higher number of channels
53 such as 6-channels or 12-channels with higher computation burden. Therefore, the entire experiment is
54 not repeated for increased number of channels. However, we have analyzed the computational
55 difference for 2 and 12- channel signals. The decomposition time for 12-channels using multivariate
56 FBSE-EWT is approximately 5 time higher than the 2-channel case. Further, the computation time of
57
MHMS for 12-channels is 2.6 time higher than the 2-channel case. We have mentioned this in
58
59 Discussion Section.
60
 Page 16 of 16

1
2
3
4
5 Comment 7: Results and discussion are difficult to follow. Please re-write it.
6
7 Response: We have rewritten the Results and Discussion section in the revised manuscript.
8
9
10
11
12
13
14 Reviewer 3
15
16
17
18 Comment 1: The section on EWT is over stressed.
19
20 Response: Thank you for your suggestions. In the revised manuscript we have not mentioned EWT
21 scaling and wavelet function and referred the base paper for mathematical expressions.
22
23
24
25 Comment 2: fig 4, 5 are not required.
26 Response: In the revised manuscript we have excluded those figures.
27
28
29
30 Comment 3: A table on available database is necessary.
31
32 Response: As per your suggestion, In the revised manuscript we have included Table I and VII for
33 SEED and DREAMER emotion databases, respectively.
34
35
36
37
Comment 4: Language needs improvement.
38 Response: In the revised paper we have corrected the grammatical mistakes and improved the language
39
of the paper.
40
41
42
43 Comment 5: Some measurement setup for recording real time EEG will add value to the paper.
44
45 Response: In the revised manuscript, we have also included and described DREAMER emotion
46 database. In the present work, we have used two public databases namely SEED and DREAMER
47 emotion databases for validation of our method. In future, we plan to record physiological signals using
48 low cost and portable system for recognition of human emotions considering large number of subjects.
49
We have mentioned this in the conclusion section.
50
51
52
53
54
55
56
57
58
59
60