Expert Systems With Applications 149 (2020) 113285

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Motor imagery EEG recognition based on conditional optimization

empirical mode decomposition and multi-scale convolutional neural
network
Xianlun Tang a, Wei Li a,∗, Xingchen Li a, Weichang Ma a, Xiaoyuan Dang b
a
Chongqing Key Laboratory of Complex Systems and Bionic Control, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
b
College of Mobile Telecommunications, Chongqing University of Posts and Telecom, Chongqing 401520, China

a r t i c l e i n f o a b s t r a c t

Article history: Electroencephalogram (EEG) signals classification plays a crucial role in brain computer interfaces (BCIs)
Received 19 August 2019 system. However, the inherent complex properties of EEG signals make it challenging to get them ana-
Revised 15 November 2019
lyzed and modeled. In this paper, a novel method based on conditional empirical mode decomposition
Accepted 5 February 2020
(CEMD) and one-dimensional multi-scale convolutional neural network (1DMSCNN) is proposed to rec-
Available online 6 February 2020
ognize motor imagery (MI) EEG signals. In the CEMD algorithm, the correlation coefficient between the
Keywords: original EEG signal and each intrinsic modal component (IMF) is used as the first condition to select IMFs,
Empirical mode decomposition and the relative energy occupancy rates between the IMFs are the second condition. The CEMD algorithm
Convolutional neural network is applied to remove the noise of EEG signals. Then, an EEG signals combination method is proposed to
Motor imagery EEG encode event-related synchronization/de-synchronization (ERS/ERD) information between the channels.
Feature extraction Finally, a model called 1DMSCNN is built to classify the processed EEG signals. The proposed method is
Intelligent wheelchair
applied to the dataset collected in our laboratory and BCI competition IV dataset 2b. The results indi-
cate that the proposed method can achieve higher accuracy for EEG signals classification, compared with
other state-of-the-art works. In addition, the proposed algorithm is applied to the online recognition of
EEG signals, a BCI system that directly interacts with brain and wheelchair is designed and implemented.
This system can directly command wheelchair to turn left and right through EEG signals. The online ex-
perimental results indicate that the designed intelligent wheelchair system is a feasible BCI application.
It verifies the proposed algorithm can be used in expert and intelligent systems. Our method can provide
a stimulus to the development of human-robot interaction.
© 2020 Elsevier Ltd. All rights reserved.

1. Introduction
Brain-computer interfaces (BCIs) convert brain activity recorded
from human scalp into computer control commands to control ex-
ternal devices, thus helping disabled people recover some motor
abilities (Sun, Feng, Lu, Wang & Zhang, 2019; Van Erp, Lotte &
Tangermann, 2012; Wolpaw, Birbaumer, McFarland, Pfurtscheller &
Vaughan, 2002). There have been studies on the use of electroen-
cephalography (EEG) to control intelligent wheelchair (zhang et al.,
2015), robotic arm (Wang, Dong, Chen & Shi, 2015) and other
external equipment (Liao et al., 2012; LaFleur et al., 2013). In a
BCI system, the feature extraction and classification of EEG sig-
∗
Corresponding author.
E-mail addresses: tangxl@cqupt.edu.com (X. Tang), cqyddxliwei@foxmail.com
(W. Li), starchen.li@foxmail.com (X. Li), weichangma@foxmail.com (W. Ma),
dxy831110@126.com (X. Dang).
nals form important parts. The commonly used EEG signals include
event-related P300 potentials, steady-state visually evoked poten-
tials (SSVEPs) and motor imagery (MI) related mu/beta rhythms
(Yu et al., 2015).
Compared with other types of signals, there are some dis-
tinct features in EEG signals. The collected EEG signals vary with
the structure of the human brain and the subjects’ mental states
(Gui, Jin, Xu, Ruiz-Blondet & Laszlo, 2015). Therefore, each sub-
ject’s EEG signal is unique. EEG signals are nonlinear and non-
stationary, which means the characteristics of EEG signals change
over time. Moreover, as the collected EEG signals are usually mixed
with noise, posing a challenge to the analysis of EEG signals. There-
fore, effective measures should be taken to improve the signal-to-
noise ratio (SNR) of EEG signals.
Different MI tasks inspire different activation states in the sen-
sorimotor cortex of the brain (Tabar & Halici, 2016), and MI can be
collected in a convenient, non-invasion, cost-saving way. Therefore,

https://doi.org/10.1016/j.eswa.2020.113285
0957-4174/© 2020 Elsevier Ltd. All rights reserved.
2 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285
many researchers have focused on the feature extraction and clas-
sification of MI EEG signals. The commonly used feature extraction
algorithms include wavelet transform (WT) (Ma, Guo, Su & Liang,
2017; Pattnaik, Dash & Sabut, 2016), common spatial patterns (CSP)
(Lotte & Guan, 2010; Wu et al., 2013; Zhang et al., 2018), Extreme
energy difference (EED) (Sun, 2010), principal component analysis
(PCA) (Kottaimalai, Rajasekaran, Selvam & Kannapiran, 2013), em-
pirical mode decomposition (EMD) (Gaur, Pachori, Wang & Prasad,
2015; Kevric & Subasi, 2017; Park, Looney, ur Rehman, Ahrabian
& Mandic, 2012) and so on. As the EMD algorithm can decom-
pose signals adaptively, it has been proved to be a very suitable
candidate to analyze nonlinear and non-stationary EEG signals. For
instance, in (Kevric & Subasi, 2017), the EMD algorithm is used
to filter out the noise in EEG signals. However, the typical EMD
algorithm generally select the intrinsic modal components (IMFs)
by researchers’ experience, which will result in some EEG samples
mixing redundant information or losing some useful information.
In addition, the features extracted by these traditional methods are
usually hand-designed, requiring a high degree of expertise. There-
fore, it is of great significance to automatically extract effective fea-
tures from EEG signals.
Deep learning can powerfully deal with nonlinear and non-
stationary data and automatically extract the effective feature rep-
resentation from the original data. In recent years, some deep
learning methods (Chu et al., 2018; Dose, Møller, Iversen &
Puthusserypady, 2018; Li, Struzik, Zhang & Cichocki, 2015; Sun, Lo
& Lo, 2019; Schirrmeister et al., 2017; Tabar & Halici, 2016) are em-
ployed for the classification of EEG signals. Convolutional neural
network (CNN) (LeCun, Bottou, Bengio & Haffner, 1998) is one of
the typical representatives. In a study by Dose, Møller, Iversen and
Puthusserypady (2018), an end-to-end CNN model is proposed for
the classification of EEG signals, and the model is proved to be
effective in classifying small sample EEG data. To adapt to the
nonlinear and non-stationary features of EEG signals, CNN and
stacked autoencoders (SAE) are combined to classify the EEG sig-
nals, and the results show the recognition accuracy of EEG sig-
nals is improved (Tabar & Halici, 2016). There are also some meth-
ods combining traditional feature extraction methods with deep
learning methods (Li, Zhang, Luo & Yang, 2016; Xu, et al., 2018;
Yang, Sakhavi, Ang & Guan, 2015). For instance, in Xu et al. (2018),
the wavelet transform (WT) is used to convert the one-dimensional
(1D) EEG signals into the Two-dimensional (2D) time-frequency
images, and a CNN is built as the classifier.
In this paper, to overcome the weakness of traditional EMD al-
gorithm in terms of selecting effective IMFs based on researcher’s
experience, the conditional empirical mode decomposition (CEMD)
algorithm is proposed. The correlation coefficient between the
original EEG signal and each IMF is used as the first condition to
select IMFs, and the relative energy occupancy rates between the
IMFs are the second condition. The CEMD algorithm is used to re-
duce the noise of EEG signals. EEG signals are multivariate time
series signals with strong anterior-posterior dependence, and the
characteristics of EEG signals are mainly changed over time. To
protect the structure of EEG time series from being destroyed, 1D
convolution is performed to extract features along the time axis.
However, previous studies have shown that there are event-related
synchronization (ERS) and event-related de-synchronization (ERD)
phenomena exist (Yu et al., 2015) between the EEG signals chan-
nels, and performing 1D convolution along the time axis cannot
capture this information. Therefore, an EEG signals combination
method is proposed to encode this information between the chan-
nels. Finally, to extract effective features of EEG signals from mul-
tiple scales, a one-dimensional multi-scale convolution neural net-
work (1DMSCNN) is built to classify the processed EEG signals.
The dataset collected in our laboratory and BCI competition IV
dataset 2b are carried out to validate the performance of the pro-
posed method, and as it shows, compared with other state-of-the-
art works, the proposed method can achieve superior performance.
It proves that the proposed method can extract the features from
complex EEG signals more effectively, and it can be used in a BCI
system.
Finally, a BCI system that directly interacts with brain and
wheelchair is designed and implemented. In the system, the pro-
posed algorithm is applied to classify left hand and right hand MI
EEG signals, and then the system generates corresponding control
commands for the wheelchair to turn left and right. The online
experiments show that the designed intelligent wheelchair system
can accurately judge subject’s instructions. It demonstrates that the
designed intelligent wheelchair system is a feasible BCI application,
and that our approach provides an effective strategy to design ex-
pert and intelligent wheelchair system.
The main highlights of this paper are list as follow:
(1) The EMD algorithm is improved by using two conditions to se-
lect IMFs, and the improved EMD (CEMD) algorithm is used to
reduce the noise of EEG signals;
(2) An EEG signals combination method is proposed to encode the
ERS/ERD information between the channels;
(3) A model called 1DMSCNN is built to classify EEG signals;
(4) An expert and intelligent wheelchair system based on the pro-
posed algorithm is designed and implemented, and the online
experiments show that the proposed algorithm is feasible for
BCI application.
2. Conditional empirical mode decomposition
2.1. Empirical mode decomposition
The EMD (Li, Zhou, Yuan, Geng & Cai, 2013) is an adaptive and
intuitive signal-dependent decomposition algorithm for the analy-
sis of nonlinear and non-stationary signals. The purpose of EMD
is to decompose a complex signal into a set of band-limited IMFs
and a residual component. Each IMF should satisfy two basic con-
ditions: (1) In a full data set, the number of extreme points and
the number of zero crossings should be equal or at most differ by
one; (2) For any given point, the mean value of the envelops de-
fined by local minima and local maxima should be zero.
Using the EMD method, the EEG signal x(t) can be recon-
structed as:

n
x(t ) = im fl (t ) + res (1)
l=1
where imf1 (t),imf2 (t),imfl (t) are all the IMFs decomposed from
the original EEG signal, and n stands for the number of IMFs. The
number of IMFs obtained by different EEG signals may be different,
which fully demonstrates that the EMD algorithm can adaptively
decompose the EEG signals. res is the residue signal.
2.2. The conditions of selecting IMFs
To overcome the problem caused by empirically selecting IMFs,
the correlation coefficient between the original EEG signal and
each IMF is used as the first IMFs selection condition, and the rela-
tive energy occupancy rates between the IMFs are proposed as the
second condition.
(1) Condition 1: correlation coefficient
The definition of correlation coefficient is as follows:

N _ _
(im fi − im f )(xi − x )
i=1
r=  (2)

N _ 2
N _ 2
(im fi − im f ) ( xi − x )
i=1 i=1
 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285
where r is the correlation coefficient between the original EEG sig-
_ 
nal and an IMF, and its range is [−1, 1]. im f = 1n Ni=1 im f i , x is the
_  N
original EEG signal, x = 1n i=1 xi , N represents the sampling fre-
quency of the signal.
The larger the correlation coefficient is, the more effective infor-
mation the IMF contains. The correlation coefficient between the
original EEG signal and each IMF is calculated by formula (2). And
the condition to satisfy the correlation coefficient is defined as:
|r| > α
where α is the set threshold and0 ≤ α ≤ 1.
(2) Condition 2: relative energy occupancy rate
Firstly, the energy of each IMF is defined as:

N
im fl (i )
2
El =
i=1
where El is the energy of theimfl , N represents the sampling fre-
quency of the signal. So the energy of each IMF can be expressed
as:E1 ,E2 ,, En .
Then, the relative energy occupancy rate is defined as:
El
q=
E1 + E2 + · · · + En
where q is the relative energy occupancy rate of theimfl .
Higher relative energy occupancy rate means more obvious
fluctuations of the IMF, indicating that it encompasses more effec-
tive information. The relative energy occupancy rate of each IMF
is calculated according to formula (5), and the selecting condition
that satisfies the relative energy occupancy rate is defined as:
q≥β
whereβ is the set threshold and0 ≤ β ≤ 1.
And then, selected IMFs are used to reconstruct the EEG signal
by the following formula:

m CNN (Cecotti & Graser, 2008) can directly input the original
y= im fi
i=1
where y is the reconstructed EEG signals, m is the number of ef-
fective IMFs after selecting.
2.3. Conditional empirical mode decomposition algorithm
The steps of the conditional empirical mode decomposition al-
gorithm are as follows:
Step1: Perform an EMD process on the EEG signal of one chan-
nel, obtain the IMFs and residual signal;
Step2: Remove the residual signal;
Step3: Retain the first five IMFs for analysis. According to pre-
vious research, the information related to the left and right
hand motor imagery is mainly scattered in the first five
IMFs.
Step4: Use the correlation coefficient condition to select the
IMFs. Calculate the correlation coefficient between each IMF
and the EEG signal before the EMD process according to for-
mula (2), and then the IMFs are selected according to for-
mula (3);
Step5: Use the relative energy occupancy rate condition to se-
lect the IMFs. For the IMFs selected by the correlation coeffi-
cient condition, the energy of each IMF is calculated accord-
ing to formula (4). And then the relative energy occupancy
rates of each IMF in the channel is calculated according to
formula (5). Finally, the IMFs are selected according to for-
mula (6);
Step6: Reconstruct the EEG signals using the selected IMFs.
3
3. EEG signals combination method
1DCNNs are often used to extract features of time series sig-
nals and have achieved good results (Li, Zhang, Zhang & Wei, 2017;
Ullah, Hussain & Aboalsamh, 2018). EEG signals are multivariate
time series signals, and their characteristics are mainly changed
over time. So we perform 1D convolution to extract features along
the time axis. Furthermore, to utilize the information (ERD and
ERS phenomena) between the EEG signal channels, the following
(3) method is proposed.
The EEG signal from a left sensorimotor channel can be repre-
sented as[A1 ,A2 ,, Ae ], and the EEG signal from a right sensori-
motor channel is represented as[B1 ,B2 ,, Be ]. So the subtraction
between the EEG signal channels is defined as follows:
Ci = Ai − Bi , i = 1, 2, · · · , e (8)
(4)
where e denotes the length of the EEG signal in a channel, and the
obtained EEG signal is expressed as[C1 ,C2 ,, Ce ].
The schematic diagram of the EEG signals combination method
is depicted in Fig. 1. And the steps of the EEG signals combination
method are as follows:
Step1: Determine the EEG signals of the corresponding channels
(5)
(In the same sensorimotor position on left and right sides of
the brain);
Step2: For each set of corresponding channels, the new EEG sig-
nals can be obtained by subtracting the EEG signals of the
right channel from those of the left channel;
Step3: The new EEG signal and the original signal are combined
in parallel way.
(6)
4. 1D Multi-scale convolutional neural network
4.1. Convolutional neural network
(7) samples and automatically learn the feature representation of the
samples, thus facilitating the classification of the samples. After
years of development and improvement, CNNs have been widely
used in classification, target detection and other fields. Compared
with fully connected networks, shared weights, local receptive
fields and sub-sampling are the three main distinct features of
CNNs. A CNN model is generally composed of five building blocks:
input layer, convolution layer, pooling layer, fully connected (FC)
layer and output layer.
Convolution layer, one of the most important parts of CNNs,
plays a key role in extracting features from the input data. The
convolution formula is as follows:
 

sdj = f sd−1
i
∗ wdi j + bdi (9)
i∈M j
where sdj denotes the jth feature graph in the dth convolution layer,
wdi j is the connection weight between the jth feature of the dth
layer and the ith feature of the d-1 layer. ∗ indicates the con-
volution operator, Mj is a collection of input features, b is the
bias term, f(•) denotes the activation function, sigmoid ( f (x ) =
1
1+e−x
), hyperbolic tangent (f(x) = tanh(x)) and rectified linear units
(ReLU,f(x) = max(0, x)) are widely employed as activation func-
tions.
Pooling layer is done by aggregating neighboring values in a
feature map into one value by taking the average (average pool-
ing) or maximum (max pooling). Its main functions are to reduce
the network parameters and the dimension of feature maps while
preserving useful information. In most of the related research, the
4 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285
Fig. 1. The schematic diagram of the EEG signals combination method.
Fig. 2. The structure of multi-scale convolution block.
max pooling is chosen in CNN models, for it can better preserve
the main features of the previous layer.
In a CNN model, the convolution and pooling layers are usu-
ally followed by one or more fully connected layer. The fully con-
nected layer converts the feature maps of the previous layer into a
1D vector, and connects the Softmax layer to separate the various
categories.
4.2. 1D multi-scale convolution
The features extracted by convolution kernels of different scales
are different. Convolution with large kernel size can capture rela-
tively holistic features, but it is not sensitive to capture detail fea-
tures. While convolution with small kernel size can capture de-
tail features more effectively (Szegedy, Ioffe, Vanhoucke & Alemi.,
2017). To make the extracted features contain more complete in-
formation, a 1D multi-scale convolution block is proposed, and its
structure is shown in Fig. 2.
In this block, convolution1-1 with large kernel size can cap-
ture relatively holistic features from EEG signals, but it cannot cap-
ture detail features effectively. Convolution1-2 with small kernel
size can capture detail features from EEG signals. The two parts
are followed by max pooling to reduce the parameters. In addi-
tion, max pooling1-3 is used directly to preserve the main features
of the previous layer, and a convolution operation with the ker-
nel size of 1 is used to change the dimension of the feature maps
(Szegedy et al., 2015). Finally, the feature maps obtained from the
three parts are combined by concatenation operation.
The block can extract features from EEG signals on multiple
scales, thus ensuring the validity of extracted features and improv-
ing the classification accuracy of EEG signals.
4.3. 1D Multi-scale convolutional neural network
The proposed structure of 1D multi-scale convolutional neural
network is illustrated in Fig. 3. The network is designed to accept
EEG signals of NEEG × Nchan shape (where NEEG refers to the length
of EEG signal and Nchan refers to the number of EEG channels
used). 1D multi-scale convolution is used to extract multi-scale
features from the input EEG signals. The large kernel size is re-
ferred asKL , and the small kernel size is referred asKS . The main pa-
rameters of 1D multi-scale convolution layer are listed in Table 1.
Then, the 1D multi-scale convolution layer is followed by a con-
volution layer, whose kernel size isK. Max pooling operation can
reduce the size of feature maps by using KMP filters with a stride
KMP (where KMP is the size of filters). The extracted features are the
input of an FC layer with 300 hidden units. The output of network
is generated by a Softmax layer with the number of neurons deter-
mined by the types of EEG signals. In addition, ReLU is chosen as
activation function, because it can speed up the optimization pro-
cess of the network. The main parameters of 1D multi-scale con-
volutional neural network are listed in Table 2.
In the network, the cross-entropy is chosen as the loss function,
which is defined as:

loss = − pi log qi (10)
i
where p indicates the expected value, and q represents the pre-
dicted value. The training strategy for our model is stochastic gra-
dient descent (SGD) algorithm which has proven to be effective.
The process of training is to minimize the value of loss. However,
when the learning rate is too large, ReLU will cause some neurons
in the network to die (Goh et al., 2018). So the learning rate we set
is η=1 × 10−4 .
When training the 1D multi-scale convolutional neural network,
it may over fit with a few train samples. So we add dropout
in the fully connected layer to reduce over-fitting. In the experi-
ment, we set the dropout rate at 0.5 based on the empirical value
(Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov, 2014)
to regulate the network.
5. Experiment on motor imagery EEG classification
5.1. EEG signal acquisition based on Emotiv EPOC+
The Emotiv EEG acquisition instrument is used in this experi-
ment, which is shown in Fig. 4. Its system comprises an EPOC+
neuroheadset, a wireless USB receiver, an electrode box, 16 elec-
trode caps and conductive liquid. The EPOC+ neuroheadset is
equipped with 14 effective electrodes and two reference electrodes.
The electrodes are placed in accordance with the international 10–
20 electrode placement standard, as shown in Fig. 5. CMS and DRL
are reference electrodes, and other electrodes are effective elec-
 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285 5

Table 1
Main parameters of 1D multi-scale convolution layer.

Layer name Output Shape Kernel size Numbers of kernel Stride Padding

Convolution1-1 NEEG × 16 KL 16 1 SAME

Max pooling1-1 (NEEG /KMP ) × 16 KMP - KMP SAME
Convolution1-2 NEEG × 16 KS 16 1 SAME
Max pooling1-2 (NEEG /KMP ) × 16 KMP - KMP SAME
Max pooling1-3 (NEEG /KMP ) × Nchan KMP - KMP SAME
Convolution1-3 (NEEG /KMP ) × 16 1 16 1 SAME
Concat (NEEG /KMP ) × 48 - - - -

Table 2
Main parameters of 1D multi-scale convolutional neural network.

Layer name Output Shape Kernel size Numbers of kernel Stride Padding

Multi-scale convolution (NEEG /KMP ) × 48 - - - -

Convolution (NEEG /KMP ) × 96 K 96 1 SAME
Max pooling (NE E G /KMP
2
) × 96 KMP - KMP SAME
FC 300 - - - -
Output (Softmax) 2 - - - -

Fig. 3. The structure of 1D multi-scale convolutional neural network.

Fig. 5. The placement position of Emotiv electrodes.

Fig. 4. Emotiv EEG signal acquisition instrument.

trodes. The sample rate is 128 Hz. The EEG signals are collected
by the EPOC+ neuroheadset and recorded by the EmotivPRO soft-
ware.
The EEG signal acquisition experiment is conducted in a rela-
tively quiet environment. 5 healthy postgraduates (denotes S1, S2,
…, S5) are selected to collect the left and right hand motor im-
agery EEG signals. As we can see in Fig. 6, a subject is collecting
the EEG data. The experimental process of one trial is shown in
Fig. 7.
Prior to the experiment, the subject sits in a chair for 40s to re-
lax, and then enters the acquisition process. One trial of collection
experiment requires 10 s, with the first six seconds seeing the sub-
ject calm down and ready for the collection. When t = 6 s, there
is a reminder sound to prompt the subject to begin imagining left
or right hand movement. When t = 10 s, there is also a reminder
sound to prompt the subject to stop imagining. And the EEG data
recorded at the 8s and 9s are taken as samples, so two EEG signal
samples can be obtained at one trial. For each subject, each type
of imagination task is repeated 90 times, so we get 180 samples of
left hand motor imagery EEG signal and 180 samples of right hand
motor imagery EEG signal. To reduce the computational complex-
ity, only the data of six channels (F3, FC5, T7, F4, FC6, T8) from the
sensorimotor are retained for analysis. And the dataset is divided
into a training set and a testing set according to the ratio of 4:1.
6 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285

Fig. 6. A subject is conducting a collection experiment.

Fig. 7. The experimental process of one trial.

5.2. Data preprocessing

EEG signals usually contain a lot of background noise such as

electro-oculogram (EOG), electromyogram (EMG) and power fre-
quency clutter, etc. To reduce the background noise, three effective
measures are taken to process EEG signals. Firstly, the abnormal
samples are removed. Secondly, each sample is subtracted from the
average amplitude to obtain a zero-mean signal. Thirdly, a band-
pass filtering of 8–30 Hz for the EEG signals is used. Studies sug-
gest that the event related to synchronization/de-synchronization
phenomenon is mainly manifested in the mu rhythm (8–13 Hz)
and beta rhythm (14–30 Hz) (Yu, et al., 2015).

5.3. Experimental process

The EEG signals of each channel are processed separately using

CEMD algorithm. The threshold of correlation coefficient condition
and relative energy occupancy rate condition are obtained from the
following two experiments. The experiments are performed on the
Fig. 8. The recognition accuracy of EEG signal when setting different threshold α .
first subject’s data.
(1) Determination of threshold α
respectively. As can be seen from Fig. 8, when the threshold is at
In the experiment, only the correlation coefficient condition is 0.1, the recognition accuracy of EEG signal is the highest, so the
used to select IMFs, and the threshold α is uniformly taken at in- threshold α is set at 0.1.
tervals of 0.05 in [0, 1]. Then the processed EEG data are input
(1) Determination of threshold β
into 1DCNN to be classified. The experimental result shows that
the recognition accuracy of EEG signals decrease rapidly when the In the experiment, the correlation coefficient condition and rel-
value of α exceeds 0.2. Fig. 8 shows the recognition accuracy of ative energy occupancy rates condition are taken into account
EEG signal when the threshold α is 0, 0.05, 0.1, 0.15, 0.2 and 0.25 when selecting IMFs. On the basis of the above experiments, the
 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285 7

Table 3
The recognition accuracy (%) of different algorithms for 5 subjects.

Subject CSP DBN DWT-LSTM 1DCNN 1DMSCNN Proposed

S1 80.56 83.33 81.94 81.94 83.33 86.11

S2 95.83 93.06 94.44 95.83 95.83 97.22
S3 72.22 73.61 75.00 73.61 76.39 77.78
S4 76.39 77.78 80.56 79.17 80.56 83.33
S5 75.00 80.56 83.33 80.56 81.94 84.72
Average 80.00 81.67 83.05 82.22 83.61 85.83

Table 4
The P-values between the proposed method and other five algorithms.

Method CSP DBN DWT-LSTM 1DCNN 1DMSCNN

P-values <0.001 <0.001 0.005 0.001 0.020

Fig. 9. The recognition accuracy of EEG signal when setting different threshold β .
threshold α is set at 0.1, and the threshold β is evenly taken at
intervals of 0.05 in [0, 1]. The processed EEG data are input into
1DCNN to be classified. The experimental result shows that the
recognition accuracy of EEG signal is the highest when the thresh-
old β is 0.1, so the threshold β is set at 0.1. Fig. 9 shows the recog-
nition accuracy of EEG signal when the threshold β is 0, 0.05, 0.1,
0.15, 0.2 and 0.25 respectively.
As can be seen in Fig. 5, F3 and F4, FC5 and FC6, T7 and T8
are symmetrically distributed on the left and right sides of the
brain, so they are mutually corresponding channels. Then, EEG sig-
nals are processed by the EEG signals combination method pro-
posed in Section 3. Finally, the processed EEG signals are in-
put into 1DMSCNN to be classified, and in this part, we set the
parameterKL = 10, KS = 3, KP = 2, K = 5.
5.4. Experimental results and analysis
In this section, detailed experiments are conducted to verify the
effectiveness of the proposed method for MI EEG signals recogni-
tion.
To demonstrate the validity of CEMD algorithm and the pro-
posed EEG signals combination method, EMD (Take the first 3
IMFs), CEMD and EEG signal combination method are employed to
process EEG signals, respectively, and the same structure of 1DCNN
is adopted to extract features of EEG signals. For each subject,
the experiment is executed for 10 times. The average classification
accuracy and standard error (SE) of each subject are depicted in
Fig. 10 (ESCB represents the EEG signals combination method).
As can be observed in Fig. 10 (a), the CEMD method can achieve
the highest accuracy among 5 subjects, thus validating the effective
of the proposed conditions for selecting IMFs. In addition, it can be
observed that EEG signals processed by EMD method achieve bet-
ter results than EEG signals without any treatment, indicating that
the EMD method is a feasible method for processing EEG signals.
From Fig. 10 (b), we learn the recognition rate of EEG signals is im-
proved after being processed by EEG signals combination method.
It shows that EEG signals combination method takes into account
the effective information between channels, and thus improves the
recognition accuracy of EEG signals.
In the next experiment, the 1DCNN, 2DCNN and 1DMSCNN are
used to classify the EEG signals. The input EEG signals are also
processed by the CEMD algorithm and EEG signal combination
method. The Receiver Operating Characteristic (ROC) Curve is a
graphical representation used to evaluate the quality of the classi-
fication model. The effect of the model can be judged by analyzing
the value of the area under the curve (AUC). As can be observed in
Fig. 11, in most instances, 1DCNN shows the higher AUC value than
2DCNN, which proves that 1D convolution is more effective than
2D convolution in analyzing EEG signals. 1DMSCNN can achieve
the best performance among 5 subjects, indicating that the fea-
tures of EEG signals extracted from multiple scales contain more
valid information, thus it can improve the recognition accuracy of
EEG signals.
The performance of the proposed algorithm is compared
with that of other feature extraction algorithms, including CSP
(Wu et al., 2013), DBN (Chu et al., 2018), long short-term mem-
ory with Discrete Wavelet Transform (DWT-LSTM) (Li, Zhang, Luo
& Yang, 2016), 1DCNN and 1DMSCNN. Based on the features ex-
tracted by CSP, a linear SVM is employed to classify the EEG sig-
nals. Table 3 lists the recognition accuracy of these algorithms for
5 subjects. It can be seen that the proposed algorithm performs
better than other algorithms among 5 subjects.
To validate the statistical significance of the differences be-
tween the proposed algorithm and the other five algorithms, two-
way analysis of variance (ANOVA2) and multiple comparisons tests
are used to calculate the P-values between the proposed algorithm
and these algorithms. The subject and the method are two inde-
pendent variables of the test, and accuracy is dependent variable
of the test. The least significant difference (LSD) method is used
for multiple comparisons. The P-values between the proposed al-
gorithm and other five algorithms are listed in Table 4. It is gen-
erally considered that there is a significant difference between the
two comparison algorithms when the P-value is less than 0.05. As
can be seen from Table 4, the P-values between the proposed al-
gorithm and CSP, DBN, DWT-LSTM, 1DCNN and 1DMSCNN are all
less than 0.05. Therefore, the recognition accuracy of the proposed
algorithm is considered to be significantly improved.
5.5. Experiment on BCI IV dataset 2b
BCI Competition dataset 2b is also used to evaluate the perfor-
mance of the proposed algorithm. The Dataset 2b is provided by
the Technical University of Graz (TUG), and it includes two classes
MI EEG signals involving left hand and right hand. The dataset
comprised of EEG signals are collected from 9 subjects, with each
subject going through 5 sessions of MI experiments, the first two
sessions without feedback and the other three sessions with feed-
back. During the collection process, three bipolar recordings are
(C3 , Cz and C4 ) used to record the EEG signals with a sampling
frequency of 250 Hz, and the recorded EEG signals are band-pass
filtered between 0.5 Hz and 100 Hz. Each of the first two sessions
and the rest three sessions includes 120 trials and 160 trials, re-
spectively.
As for preprocessing, the operations are employed as the former
dataset. And the EEG signals are processed by CEMD algorithm and
8 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285

Fig. 10. Average recognition accuracy of five subjects obtained by different EEG signal processing methods.

Table 5
The recognition accuracy (%) of different algorithms for 9 subjects.

Subject EED CSP ACSP DBN CNN-SAE 1DMSCNN Proposed

B1 56.56 66.56 67.50 66.56 76.00 76.39 80.56

B2 52.19 57.81 55.36 62.50 65.80 61.03 65.44
B3 58.13 61.25 62.19 60.00 75.30 60.41 65.97
B4 89.69 94.06 94.69 96.87 95.30 98.65 99.32
B5 64.69 80.63 76.88 82.02 83.00 85.14 89.19
B6 71.88 75.00 75.94 77.44 79.50 80.56 86.11
B7 56.25 72.50 71.25 76.56 74.50 78.47 81.25
B8 80.00 89.38 89.38 88.75 75.30 84.21 88.82
B9 70.63 85.63 81.25 86.06 73.30 81.94 86.81
Average 66.69 75.87 74.93 77.42 77.60 78.53 82.61

EEG signal combination method. In this dataset, the corresponding Table 6

channels are C3 and C4. Then, the EEG signals are input into 1DM-
SCNN to be classified, and we set the parameterKL = 15, KS = 5,
KP = 5, K = 5.
To evaluate the performance of the proposed method on BCI
competition IV dataset 2b, we compare the proposed method with
other state-of-the-art approaches, including EED (Sun, 2010), CSP
(Wu et al., 2013), ASCP (Sun & Zhou, 2014), DBN (Chu et al., 2018),
CNN-SAE (Tabar & Halici, 2016) and 1DMSCNN, with the recogni-
tion accuracy of those algorithms listed in Table 5.
As can be seen from Table 5, the classification performance of
proposed method is more effective than that of other compari-
son methods for most subjects, and the proposed method can get
the best average recognition accuracy. Moreover, the deep learning
methods DBN, CNN-SAE and 1DMSCNN can achieve higher recog-
nition accuracy than the traditional feature extraction algorithms
EED, CSP and ACSP. And it also can be seen that all of the algo-
rithms have poor recognition accuracy on subject B2 and B3. There
are three possible reasons that for this. Firstly, the subject is poorly
concentrated. Secondly, the noise level in the environment may be
differ during the time of collection, resulting in the collected EEG
signals containing different amount of useless information. Thirdly,
each subject’s EEG signals are unique, with some subject’s EEG sig-
nals easier to classify, while some difficult to classify.
ANOVA2 and multiple comparisons tests are also used to val-
idate the statistical significance of the differences between the
proposed algorithm and the other eight algorithms. The P-values
between the proposed algorithm and other eight algorithms are
shown in Table 6. It can be seen that the P-values between the
proposed algorithm and EED, CSP, ASCP, DBN, CNN-SAE and 1DM-
SCNN are all less than 0.05.
The P-values between the proposed method and other eight algorithms.
Method EED CSP ACSP DBN CNN-SAE 1DMSCNN
P-value <0.001 0.002 <0.001 0.013 0.015 0.047
6. Online experiment
To further verify the practicability of the proposed algorithm,
an online experiment is carried out on the self-designed intelli-
gent wheelchair system. The intelligent wheelchair system mainly
includes: an Emotiv EEG acquisition instrument, a laptop computer,
wireless communication module, control system and a wheelchair.
The structure of the intelligent wheelchair system is shown in
Fig. 12.
Emotiv EEG acquisition instrument can record EMG and EEG
signals simultaneously. Through previous observations, we know
that gritting the teeth can make the F8 channel produce obvious
voltage changes, so this signal is used to start and stop the online
experiment. Blinking the eye can make the FC4 channel produce
obvious voltage changes, and it is used to command wheelchair
to go straight. Furthermore, the left hand and right hand MI EEG
signals of F3, F4, FC5, FC6, T7, and T8 channels are collected, and
the proposed method is employed to classify the EEG signals. The
left hand and right hand MI EEG signals are used to command
wheelchair to turn left and right, respectively.
The first three subjects in the former dataset conduct the on-
line experiments, they command the intelligent wheelchair to go
straight, turn left and turn right. All kinds of experiments are car-
ried out in a crossover way. Each type of experiment is performed
140 times, with a five-minute break after 20 experiments and an
 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285 9

Fig. 11. The ROC curves of 5 subjects.

interval of 20 seconds between each experiment. The online recog-
nition accuracy is shown in Table 7.
As can be seen from Table 7, the recognition accuracy of
the proposed method is higher than the recognition accuracy of
the 1DCNN model. And the online recognition accuracy of the
EMG signal is higher than that of the EEG signal, because the
EMG signals have more obvious characteristics than the EEG sig-
nals. In addition, after comparing the experimental results of
Table 3 and those of Table 7, it can be found that the online
recognition accuracy is lower than the offline recognition accu-
racy. This is due to more factors need to be considered when
conducting an online experiment. For example, the subjects may
10 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285

ference information, and it will affect the recognition accuracy of

EEG signals.
To design a better expert and intelligent wheelchair system, fur-
ther work will be carried out in the following aspects: (1) Collect-
ing and classifying more complex EEG signals to further improve
the functions of the wheelchair, such as using EEG signals to con-
trol the speed of the intelligent wheelchair. (2) Studying how to
further improve the performance of convolutional neural networks,
such as reducing network parameters and reducing network train-
ing time. (3) Integrating EEG signals with other biological signals to
control wheelchair is an interesting method as it allows for more
precise control of the wheelchair and improves the anti-jamming
capability of the designed wheelchair system.

Declaration of Competing Interest

There are no conflicts of interest.

Fig. 12. The structure of the intelligent wheelchair system.
Credit authorship contribution statement
Table 7
The online recognition accuracy (%) of three subjects. Xianlun Tang: Conceptualization, Methodology, Resources,
Subject Straight 1DCNN Proposed Project administration, Funding acquisition, Writing - review &
editing. Wei Li: Conceptualization, Methodology, Software, Inves-
Left Right Average Left Right Average
tigation, Writing - original draft, Writing - review & editing.
S1 94.29 77.86 75.71 76.78 82.86 82.14 82.50 Xingchen Li: Validation, Investigation, Writing - review & editing.
S2 96.43 91.43 92.14 91.78 93.57 94.29 93.93
Weichang Ma: Investigation, Writing - review & editing. Xiaoyuan
S3 86.43 71.43 68.57 70.00 75.00 73.57 74.28
Dang: Validation, Supervision.

Acknowledgments
be susceptible to the surrounding environment and prone to
fatigue. This work is supported by the National Natural Science
Foundation of China under Project 61673079, 61703068 and
the Natural Science Foundation of Chongqing under Project
7. Conclusion cstc2018jcyjAX0160.

In this paper, a method based on CEMD and 1DMSCNN is pro- References

posed to improve the performance of MI EEG signals classification.
The classic EMD algorithm is optimized by the strategy of adding
condition to select effective IMFs. The EEG signals combination
method is proposed to encode the effective information between
channels. To extract effective features of EEG signals from mul-
tiple scales, the structure of convolutional neural network is im-
proved by multi-scale convolution feature extraction strategy. The
dataset collected in our laboratory and BCI competition IV dataset
2b are used to validate the effectiveness of the proposed method.
Compared with other existing feature extraction algorithms such
as CSP, DBN, LSTM and CNN, the recognition accuracy of the pro-
posed method is significantly improved. The experimental results
show that the proposed algorithm can extract more effective fea-
tures from EEG signals and can be used to classify EEG signals in a
BCI system.
With the proposed algorithm as the basis, an intelligent
wheelchair system featuring live interactions between brain and
wheelchair is designed and developed. The system consists of five
parts: an Emotiv EEG acquisition instrument, a laptop computer,
wireless communication module, control system and a wheelchair.
In this system, the left hand MI EEG and right hand MI EEG are
used to command the wheelchair to turn left and right, respec-
tively. The online experimental results show that the intelligent
wheelchair system can effectively identify the EEG signals and give
precise control commands, which further validates the applicability
of the proposed algorithm.
The designed intelligent wheelchair system needs to be used in
a relatively quiet environment. The subject is easily affected by the
external factors, noise in particular, during the collection of EEG
signals. So the recorded EEG signals are usually blend with inter-
Chu, Y., Zhao, X., Zou, Y., Xu, W., Han, J., & Zhao, Y. (2018). A decoding scheme for
incomplete motor imagery EEG with deep belief network. Frontiers in Neuro-
science, 12, 680.
Cecotti, H., & Graeser, A. (2008). Convolutional neural network with embedded
Fourier transform for EEG classification. In 2008 19th International Conference
on Pattern Recognition (pp. 1–4).
Dose, H., Møller, J. S., Iversen, H. K., & Puthusserypady, S. (2018). An end-to-end
deep learning approach to MI-EEG signal classification for BCIs. Expert Systems
with Applications, 114, 532–542.
Gui, Q., Jin, Z., Xu, W., Ruiz-Blondet, M. V., & Laszlo, S. (2015). Multichannel
EEG-based biometric using improved RBF neural networks. In 2015 IEEE Signal
Processing in Medicine and Biology Symposium (SPMB) (pp. 1–6).
Gaur, P., Pachori, R. B., Wang, H., & Prasad, G. (2015). An empirical mode decom-
position based filtering method for classification of motor-imagery EEG signals
for enhancing brain-computer interface. In 2015 International Joint Conference on
Neural Networks (IJCNN) (pp. 1–7).
Goh, S. K., Abbass, H. A., Tan, K. C., Al-Mamun, A., Thakor, N., Bezerianos, A.,
et al. (2018). Spatio–spectral representation learning for electroencephalo-
graphic gait-pattern classification. IEEE Transactions on Neural Systems and Re-
habilitation Engineering, 26(9), 1858–1867.
Kottaimalai, R., Rajasekaran, M. P., Selvam, V., & Kannapiran, B. (2013). EEG sig-
nal classification using principal component analysis with neural network in
brain computer interface applications. In 2013 IEEE International Conference
on Emerging Trends in Computing, Communication and Nanotechnology (ICECCN)
(pp. 227–231).
Kevric, J., & Subasi, A. (2017). Comparison of signal decomposition methods in clas-
sification of EEG signals for motor-imagery BCI system. Biomedical Signal Pro-
cessing and Control, 31, 398–406.
LaFleur, K., Cassady, K., Doud, A., Shades, K., Rogin, E., & He, B. (2013). Quadcopter
control in three-dimensional space using a noninvasive motor imagery-based
brain–computer interface. Journal of Neural Engineering, 10(4), 046003.
Liao, L. D., Chen, C. Y., Wang, I. J., Chen, S. F., Li, S. Y., Chen, B. W., et al. (2012).
Gaming control using a wearable and wireless EEG-based brain-computer in-
terface device with novel dry foam-based sensors. Journal of Neuroengineering
and Rehabilitation, 9(1), 5.
Lotte, F., & Guan, C. (2010). Regularizing common spatial patterns to improve BCI
designs: unified theory and new algorithms. IEEE Transactions on Biomedical
Engineering, 58(2), 355–362.
 X. Tang, W. Li and X. Li et al. / Expert Systems With Applications 149 (2020) 113285
Li, J., Struzik, Z., Zhang, L., & Cichocki, A. (2015). Feature learning from incomplete
EEG with denoising autoencoder. Neurocomputing, 165, 23–31.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied
to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.
Li, M., Zhang, M., Luo, X., & Yang, J. (2016). Combined long short-term memory
based network employing wavelet coefficients for MI-EEG recognition. In 2016
IEEE International Conference on Mechatronics and Automation (pp. 1971–1976).
Li, S., Zhou, W., Yuan, Q., Geng, S., & Cai, D. (2013). Feature extraction and recog-
nition of ictal EEG using EMD and SVM. Computers in Biology and Medicine,
43(7), 807–816.
Li, D., Zhang, J., Zhang, Q., & Wei, X. (2017). Classification of ECG signals based on
1D convolution neural network. In 2017 IEEE 19th International Conference on
e-Health Networking, Applications and Services (Healthcom) (pp. 1–6).
Ma, M., Guo, L., Su, K., & Liang, D. (2017). Classification of motor imagery EEG
signals based on wavelet transform and sample entropy. In 2017 IEEE 2nd
Advanced Information Technology, Electronic and Automation Control Conference
(IAEAC) (pp. 905–910).
Pattnaik, S., Dash, M., & Sabut, S. K. (2016). DWT-based feature extraction and clas-
sification for motor imaginary EEG signals. In 2016 International Conference on
Systems in Medicine and Biology (ICSMB) (pp. 186–201).
Park, C., Looney, D., ur Rehman, N., Ahrabian, A., & Mandic, D. P. (2012). Classi-
fication of motor imagery BCI using multivariate empirical mode decomposi-
tion. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 21(1),
10–22.
Sun, L., Feng, Z., Lu, N., Wang, B., & Zhang, W. (2019). An advanced bispectrum fea-
tures for EEG-based motor imagery classification. Expert Systems with Applica-
tions, 131, 9–19.
Sun, S. (2010). Extreme energy difference for feature extraction of EEG signals. Ex-
pert Systems with Applications, 37(6), 4350–4357.
Sun, Y., Lo, F. P. W., & Lo, B. (2019). EEG-based user identification system using 1D–
convolutional long short-term memory neural networks. Expert Systems with Ap-
plications, 125, 259–267.
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. A. (2017). Inception-v4, incep-
tion-resnet and the impact of residual connections on learning. Thirty-First AAAI
Conference on Artificial Intelligence (pp. 4278–4284).
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., et al. (2015). Go-
ing deeper with convolutions. In Proceedings of the IEEE conference on computer
vision and pattern recognition (pp. 1–9).
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M.,
Eggensperger, K., Tangermann, M., et al. (2017). Deep learning with convolu-
tional neural networks for EEG decoding and visualization. Human Brain Map-
ping, 38(11), 5391–5420.
11
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014).
Dropout: a simple way to prevent neural networks from overfitting. The Journal
of Machine Learning Research, 15(1), 1929–1958.
Sun, S., & Zhou, J. (2014). A review of adaptive feature extraction and classification
methods for EEG-based brain-computer interfaces. In 2014 International Joint
Conference on Neural Networks (IJCNN) (pp. 1746–1753).
Tabar, Y. R., & Halici, U. (2016). A novel deep learning approach for classification of
EEG motor imagery signals. Journal of Neural Engineering, 14(1), 016003.
Ullah, I., Hussain, M., & Aboalsamh, H. (2018). An automated system for epilepsy de-
tection using EEG brain signals based on deep learning approach. Expert Systems
with Applications, 107, 61–71.
Van Erp, J., Lotte, F., & Tangermann, M. (2012). Brain-computer interfaces: Beyond
medical applications. Computer, 45(4), 26–34.
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., &
Vaughan, T. M. (2002). Brain–computer interfaces for communication and
control. Clinical Neurophysiology, 113(6), 767–791.
Wang, H., Dong, X., Chen, Z., & Shi, B. E. (2015). Hybrid gaze/EEG brain computer
interface for robot arm control on a pick and place task. In 2015 37th Annual
International Conference of the IEEE Engineering in Medicine and Biology Society
(EMBC) (pp. 1476–1479).
Wu, S. L., Wu, C. W., Pal, N. R., Chen, C. Y., Chen, S. A., & Lin, C. T. (2013). Common
spatial pattern and linear discriminant analysis for motor imagery classification.
In 2013 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind,
and Brain (CCMB) (pp. 146–151).
Xu, B., Zhang, L., Song, A., Wu, C., Li, W., Zhang, D., et al. (2018). Wavelet trans-
form time-frequency image and convolutional network-based motor imagery
EEG classification. IEEE Access, 7, 6084–6093.
Yu, T., Xiao, J., Wang, F., Zhang, R., Gu, Z., Cichocki, A., et al. (2015). Enhanced motor
imagery training using a hybrid BCI with feedback. IEEE Transactions on Biomed-
ical Engineering, 62(7), 1706–1717.
Yang, H., Sakhavi, S., Ang, K. K., & Guan, C. (2015). On the use of convolutional neu-
ral networks and augmented CSP features for multi-class motor imagery of EEG
signals classification. In 2015 37th Annual International Conference of the IEEE En-
gineering in Medicine and Biology Society (EMBC) (pp. 2620–2623).
Zhang, R., Li, Y., Yan, Y., Zhang, H., Wu, S., Yu, T., et al. (2015). Control of a
wheelchair in an indoor environment based on a brain–computer interface and
automated navigation. IEEE Transactions on Neural Systems and Rehabilitation
Engineering, 24(1), 128–139.
Zhang, Y., Wang, Y., Zhou, G., Jin, J., Wang, B., Wang, X., et al. (2018). Multi-kernel
extreme learning machine for EEG classification in brain-computer interfaces.
Expert Systems with Applications, 96, 302–310.