Master’s Thesis

Brain Activity During Motor Imagery

and Mental Arithmetic: Deep Learning
Approach for EEG

Min-Kyung Jung

Department of Brain and Cognitive Engineering

Graduate School
Korea University

August 2023
 Brain Activity During Motor Imagery
and Mental Arithmetic: Deep Learning
Approach for EEG

by
Min-Kyung Jung

_____________________________________

under the supervision of Professor Dong-Joo Kim

A thesis submitted in partial fulfillment of

the requirements for the degree of
Master of Engineering

Department of Brain and Cognitive Engineering

Graduate School
Korea University

August 2023

ii
The thesis of Min-Kyung Jung has been approved
by the thesis committee in partial fulfillment of
the requirements for the degree of
Master of Engineering

June 2023

__________________________
Committee Chair: Dong-Joo Kim

__________________________
Committee Member: Jae-Ho Han

__________________________
Committee Member: Hakseung Kim

iii
 Brain Activity during Motor Imagery and Mental
Arithmetic: Deep Learning Approach for EEG

by Min-Kyung Jung

Department of Brain and Cognitive Engineering

under the supervision of Professor Dong-Joo Kim

ABSTRACT

Brain-computer interface (BCI) is an emerging communication technology that

facilitates interaction between humans and external devices based on the user's thoughts.
Motor imagery (MI) classification, using electroencephalography (EEG) to detect user's
movement intentions, is a primary BCI research area. Mental arithmetic (MA) classification
is also employed in the analysis of brain activity related to concentration and working
memory in simple arithmetic operations such as addition and subtraction. In addition, recent
advancements in data analysis techniques have enabled the active use of deep learning-based
artificial intelligence models to analyze EEG signals. Accordingly in this thesis, a method
for analyzing brain activity during MI and MA using a deep learning model based on EEG
signals is proposed.

This thesis is composed of contents related to MI and MA classification. In MI

classification, a method of filtering out data that is considered to indicate a decrease in user's
concentration was proposed, and it was confirmed that this led to an improvement in the
model's classification performance. In the case of the data excluded from the training, there
was a tendency for the motor-related potentials to not be observed. In MA classification, a

i
graph neural network (GNN) model was proposed that achieves high classification
performance using spatial brain activity and functional connectivity, and the explanation of
the trained model operation based on explainable artificial intelligence was examined. The
GNN utilized connectivity between the left centroparietal-left frontal regions and the
frontoparietal connection as an important feature for mental arithmetic classification. The
findings of this thesis are expected to provide valuable insights to future researchers who
aim to advance brain-computer interface technology based on a thorough investigation of
brain activity.

Keywords: Brain-computer interface (BCI), Electroencephalography (EEG), Motor

imagery (MI), Mental arithmetic (MA), Artificial intelligence (AI), Explainable artificial
intelligence (XAI)

ii
 동작 상상 및 암산 동안의 뇌 활동에 대한 연구:
딥러닝 기반의 뇌파 분석을 통해

정민경

뇌공학과

지도교수: 김 동 주

국문 초록

뇌-컴퓨터 인터페이스는 생각만으로 사람과 외부의 장비 간의 상호작용을

가능하게 하는 커뮤니케이션 기술이다. 뇌파를 통해 상상으로 움직이고자 하는

의도를 파악하는 동작 상상 분류는 뇌-컴퓨터 인터페이스의 주요한 연구 분야 중

하나이다. 암산 분류 또한 집중도 및 작업 기억에 대한 뇌파 분석에 적극적으로

활용되고 있으며, 간단한 덧셈, 뺄셈 등의 과제를 수행하는 동안의 뇌 활동 변화를

탐지한다. 한편, 최근 데이터 분석 방법의 발전으로 인해, 딥러닝 기반의 인공지능

모델이 뇌전도 분석에 적극적으로 활용되고 있다. 따라서, 본 학위 논문에서는

뇌전도를 분석하는 딥러닝 모델을 통해, 동작 상상 및 암산 중의 뇌 활동을 분석하는

iii
방법에 대해 논하고자 한다.

본 학위 논문은 동작 상상 분류 및 암산 과제 분류에 대한 내용으로 구성되어 있다.

동작 상상 분류에서는 사용자의 집중력이 떨어진 것으로 간주되는 데이터를

필터링하는 방법을 제안했으며, 이를 통해 모델의 분류 성능이 향상되는 경향을

확인했다. 이때, 학습에서 제외된 데이터들의 경우, 운동 관련 전위가 현저히 낮게

나타나는 경향을 보였다. 암산 분류에서는 공간적 뇌 활동 및 뇌 연결성을 입력으로

높은 분류 성능을 달성하는 그래프 신경망 모델을 제안했으며, 설명가능한

인공지능을 기반으로 학습된 모델의 동작 방식을 살펴보았다. 그래프 신경망은

두정엽-좌측 전두엽 연결성과 전두정엽 내의 연결성을 암산 분류에 있어 중요한

특징으로 활용했다. 본 학위 논문의 발견은 뇌 활동의 면밀한 조사를 기반으로 뇌-

컴퓨터 인터페이스 기술을 고도화하고자 하는 향후 연구자들에게 가치 있는 통찰을

제공할 수 있을 것으로 사료된다.

중심어: 뇌-컴퓨터 인터페이스, 뇌파, 동작 상상, 암산, 인공지능, 설명가능한

인공지능

iv
 PREFACE

Part of this work has been presented in the following publications:

Jung, Min-Kyung, et al. "Phase Transition in previous Motor Imagery affects Efficiency of
Motor Imagery based Brain-computer Interface." 2021 9th International Winter Conference
on Brain-Computer Interface (BCI). IEEE, 2021.

Jung, Min-Kyung, et al. "Interpretability of Hybrid Feature Using Graph Neural Networks
from Mental Arithmetic Based EEG." 2023 11th International Winter Conference on Brain-
Computer Interface (BCI). IEEE, 2023.

v
 TABLE OF CONTENTS

ABSTRACT ........................................................................................................................... i

국문 초록 .............................................................................................................................iii

PREFACE.............................................................................................................................. v

TABLE OF CONTENTS ..................................................................................................... vi

LIST OF TABLES .............................................................................................................viii

LIST OF FIGURES .............................................................................................................. ix

Part I. Introductory Materis ................................................................................................... 1

CHAPTER 1. Overview of Brain-Computer Interface.......................................................... 2

1.1 Brain-computer Interface ....................................................................................... 2
1.2 Electroencephalography......................................................................................... 3
1.3 Motor Imagery ....................................................................................................... 5
1.4 Mental Arithmetic .................................................................................................. 6
1.5 Thesis Objectives ................................................................................................... 7

Part II. Main Study ................................................................................................................ 8

CHAPTER 2. Phase Transition in Previous Motor Imagery Affects Efficiency of Motor

Imagery Based Brain-Computer Interface ............................................................................. 9
2.1 Summary ................................................................................................................ 9
2.2 Introduction.......................................................................................................... 10
2.3 Materials and Methods ........................................................................................ 11
2.4 Results.................................................................................................................. 14
2.5 Discussion and Conclusion .................................................................................. 22

CHAPTER 3. Interpretability of Hybrid Feature Using Graph Neural Networks from

Mental Arithmetic Based EEG ............................................................................................ 23
3.1 Summary .............................................................................................................. 23
vi
 3.2 Introduction.......................................................................................................... 24
3.3 Materials and Methods ........................................................................................ 25
3.4 Results.................................................................................................................. 29
3.5 Discussion and Conclusion .................................................................................. 35

Part III. Conclusions ............................................................................................................ 37

CHAPTER 4. Concluding Remarks .................................................................................... 38

4.1 Summary and Major findings .............................................................................. 38
4.2 Limitations and Suggestions ................................................................................ 39
4.3 Conclusions.......................................................................................................... 40

REFERENCES .................................................................................................................... 41

vii
 LIST OF TABLES

Table 1.1 Detailed comparison of major control signals used in brain-computer interface
systems .................................................................................................................................. 4

Table 2.1 Cue detection accuracy ........................................................................................ 16

Table 2.2 Classification performance of selected-base model MI....................................... 18

viii
 LIST OF FIGURES

Figure 2.1 ERSP of all subjects (use STFT, window size 2 seconds, update period 0.04
seconds). (A) ERSP of selected trials, (B) ERSP of non-selected trials.............................. 15

Figure 2.2 Comparison of the MI classification accuracy between the total-based and upper
-2σ selection-based model ................................................................................................... 19

Figure 2.3 ERSP Difference plot (A) ERSP Difference plot of selected left class trials, (B)
ERSP Difference plot of selected right class trials, (C) ERSP Difference plot of non-
selected left class trials, (D) ERSP Difference plot if non-selected right class trials .......... 21

Figure 3.1. Model performance trained on both datasets. (A) is about connectivity (CON)
and (B) is about connectivity and PSD features (CON-PSD). Red, green, and blue donate
accuracy, recall, and precision, respectively ....................................................................... 30

Figure 3.2. Circular graphs depicting the connectivity of envelope correlation between
nodes, and topographic visualization of each PSD band. (A), (B), and (C) were obtained
from 20, 21, 24 subjects in CON-PSD dataset. The upper and lower rows represented each
baseline (BL), mental arithmetic (MA). Delta, theta, alpha, beta, and gamma PSD bands
were colored purple, blue, green, orange, and red, respectively ......................................... 32

Figure 3.3. Average of subgraphs that found as important in task classification by the
GNNExplainer in (A) subject 20, (B) subject 21, and (C) subject 24 ................................. 34

Figure 2.1 ERSP of all subjects (use STFT, window size 2 seconds, update period 0.04
seconds). (A) ERSP of selected trials, (B) ERSP of non-selected trials.............................. 15

ix
Part I:
Introductory Materials

1
 CHAPTER 1. Overview of Brain-Computer Interface

1.1 Brain-computer Interface

Brain-computer interface (BCI) technology refers to the direct connection between the
human brain and computer, enabling the control of computers through
electroencephalography (EEG), and it can be considered as the ultimate interface between
humans and computers [1]. BCI technology is broadly categorized as human-computer
interface (HCI) technology, and it is also called brain-machine interface (BMI) because it
allows the manipulation of machines such as wheelchairs and robots through EEG. BCI
technology analyzes brain activity to understand the intentions of humans or animals, and
applies it to control robots or computers. The implementation of BCI technology involves
acquiring EEG through devices and then processing the signals to analyze them before
issuing commands to the input and output devices [2].

2
1.2 Electroencephalography

The BCI technology is classified into invasive and non-invasive methods depending on
the location of the EEG measurement, and into evoked signals and spontaneous signals
depending on the EEG measurement paradigm, as shown in Table 1.1 [3]. Invasive methods
in BCI involve the insertion of a microchip into the scalp to measure brainwaves, while non-
invasive methods measure brainwaves externally through helmets or headset devices.
Although non-invasive methods are easier to use, they are susceptible to signal noise,
making accurate measurement difficult. Evoked signal methods require training to induce
the appearance of specific EEG, while spontaneous signal methods analyze EEG to
recognize simple thoughts or actions, enabling the direct transmission of user intent to a
machine. [4]. If evoked-based BCI is like using Morse code to communicate, then the pattern
recognition-based BCI can be compared to having a simple conversation over the phone.
Even though it involves training the user to generate specific EEG patterns, if the training is
consistent with the user's intention, it falls into the category of EEG recognition.

3
 Table 1.1 Detailed comparison of major control signals used in brain-computer
interface systems

Classification criteria Method Feature

- Microchips are implanted into the scalp to measure
EEG.
Invasive
- Although precise measurement is possible, surgery is
location of the EEG required and there may be surgical side effects.
measurement - Brain waves can be easily measured using a helmet or
Non- headset, making it convenient.
invasive - Inevitable noise in the signal makes it difficult to
obtain accurate measurements.
- The method induces the appearance of specific EEG
for application.
Evoked - As the actual user intention and the occurrence of
EEG may not match, training to produce specific EEG is
EEG measurement necessary.
paradigm
- Extracting simple intentions or actions through
analyzing EEG signals.
Spontaneous
- Transmits user's intention directly to the computer or
machine

EEG used in the analysis of BCI are obtained by measuring the subtle signals of the
brain surface, which are synthesized by the electrical signals generated by numerous neurons
in the brain. EEG signals vary spatiotemporally depending on brain activity, measurement
state, and brain function, and show frequencies of 1-50 Hz and amplitudes of approximately
10-200 uV. All analysis methods that can be applied to time series are available for brain
wave analysis [5], including 1) time analysis: mean, standard deviation, autocorrelation
function, cross-correlation function, etc., 2) frequency analysis: power spectrum, etc., 3)
time-frequency analysis: short-time Fourier transform (STFT), wavelet transform, Wigner
distribution, etc., 4) statistical models: autoregressive, etc., and 5) spatiotemporal analysis:
principal component analysis (PCA), independent component analysis (ICA), etc. In
addition, with the advancement of signal processing, machine learning and deep learning
methodologies are actively being utilized in the analysis of EEG [6, 7].

4
1.3 Motor Imagery

BCI based on EEG sensorimotor rhythms is known as MI-BCI, which is a type of

endogenous EEG-based BCI that involves imagining specific actions without actually
performing them. MI is defined as a conscious mental simulation of actions related to the
brain's motor representations that are similar to those performed during actual movement [8].
This has led to the suggestion that MI depends on similar neural structures and processes as
actual motor execution [8, 9]. In MI-BCI, operator is instructed to haptically imagine
movements of specific limbs, such as left or right arm or leg. Then, operators voluntarily
switch between the corresponding mental tasks in synchronous (cue-paced) or asynchronous
(self-paced) mode to generate commands. EEG rhythms and propagation can provide
valuable evidence about the brain's information processing mechanisms [10]. Previous
studies have shown that specific frequency components of EEG, such as mu and central beta
rhythms, are (de)synchronized in contralateral (ipsilateral) sensorimotor areas when subjects
perform or even imagine limb movements [11-13]. Therefore, early research on pattern
discrimination of MI movements was based on quantification of event-related
synchronization/desynchronization (ERS/ERD) using band power.

5
1.4 Mental Arithmetic

MA is a complex mental activity that requires the integration of multiple cognitive

processes, such as activation of number representations, number recognition, memory search
(e.g., direct access to a stored solution), mental calculation, decision making, response
selection, or responding. Dehaen et al. [14] assumed that the horizontal intraparietal sulcus
in humans is systematically involved in all number tasks. The prefrontal and inferior frontal
cortex are also activated during the subject's MA performance [14]. The activity in these
areas generally reflects the involvement of working memory processes [15]. Therefore, MA
tasks are frequently used to induce mental workload and investigate working memory. The
number calculation task is particularly suitable for inducing working memory load as it can
easily manipulate the arithmetical features of number tasks (e.g., the number of digits or
operands) to change the amount of load. EEG-theta has been found to be enhanced during
all kinds of MA tasks. This enhancement is primarily observed in the frontal cortex and is
induced by working memory load [16, 17]. The left parietal cortex and frontal lobe have
been shown to be involved in MA. Kong et al. [18] found that simple arithmetic performance
is associated with the right inferior parietal lobule, left precuneus, and left superior parietal
gyrus (especially for subtraction).

6
1.5 Thesis Objectives

BCI research has been primarily focused on the ultimate goal of manipulating external
machines solely through thoughts, thus often involving mental simulations without actual
physical actions [19, 20]. Unlike mapping brain responses to external sensory-motor stimuli,
the mapping of brain regions activated by such tasks remains a central challenge in BCI
research due to the distributed nature of brain networks involved in cognitive tasks and
limitations of various imaging techniques [21]. The high temporal resolution of EEG is
inherently suitable for mapping the dynamic engagement of brain regions during mental
processes, leading to increased interest in EEG within recent BCI research [22].
Representative BCI tasks include MI and MA, which predominantly elicit brain activation
around the central area and the prefrontal cortex covering the forehead, respectively.
Moreover, recent advancements in data analysis methodologies have led to the active
application of deep learning approaches in EEG-BCI research, achieving high performance.
However, deep learning algorithms exhibit characteristics that make it difficult to directly
examine the inferential process, resembling a black box. Consequently, various attempts are
being made to overcome these limitations and interpret the functioning of models. In this
thesis, deep learning algorithms were applied for MI and MA classification, and through
additional analysis, the operating mechanisms of the models were investigated to examine
brain activity during the tasks. The contributions of this thesis can be summarized as follows:

1) Proposing a trial rejection methodology to quantitatively evaluate the subject's focus

during the task.

2) Confirming the advantages of integrating the analysis of spatial neural activity and
functional connectivity during classification tasks for BCI.

3) Investigating the functioning of the trained models and discussing brain activity during
mental tasks.

7
Part II:
Main Study

8
 CHAPTER 2. Phase Transition in previous Motor Imagery
affects Efficiency of Motor Imagery based Brain-computer
Interface

The contents of this chapter are approved for publications in: 2021 9th International Winter
Conference on Brain-Computer Interface (BCI).

DOI: 10.1109/BCI51272.2021.9385321

2.1 Summary

An electroencephalography (EEG) based brain-computer interface (BCI) provides a

communication channel to operate the external environment by decoding brain patterns.
Movement-related cortical potentials (MRCPs) are one particular type of EEG pattern during
movement of peripheral limbs. The performance of motor imagery (MI) is related to pattern
of MRCPs by planning simulation. In resent decade, MI-based BCI have shown potential as
its performance significantly improved. In this study, the feasibility of selected-based
method was proposed by compared with conventional method. The detection accuracy
overall performances were 72.42 ± 3.12%. When a top 97.8% trial is selected, overall
performance improved approximately 3.15 % compared to baseline. When MI were
analyzed in non-selected trials, C3 and C4 channel showed no different aspects in left and
right class respectively. The brain state was changed after the cue appeared, and these power
of delta band appeared in all subjects. The performance of classification was improved by
rejecting trials with no difference between the state before and after cue.

9
2.2 Introduction

Brain-computer interface (BCI) provides a communication channel between human and

external devices by decoding brain activity patterns [23]. A wide range of modality exists
for monitoring brain activity pattern; among those, the electroencephalogram (EEG) remains
as the de facto primary measure of brain activity pattern for BCI, due to its high temporal
resolution, non-invasiveness, usability, and low cost [24].

Movement-related cortical potentials (MRCPs) refer to the EEG patterns during

movement of peripheral limbs [25]. The MRCPs comprises three events, namely; readiness
potential (RP), motor potential (MP), and movement-monitoring potential (MMP), which
reflect the movement planning/preparation, execution, and control of performance,
respectively [26]. Existing literatures also indicate that the RP explains volitional behaviors
because the featured pattern at δ band (0-5 Hz) on the central region is revealed before 0.5-
2s of movement [27]. The RP can also be observed during the imagination of movement,
i.e., without actual movement [28]. The behaviors of MRCPs raises a particular research
interest for their potential use for motor imagery (MI)-based BCI application [25];
nonetheless, despite the rapid advancements in MI-BCI performance [29], an attempt to
apply the changes in EEG patterns during movement-related state transition reflected as
MRCPs has yet to be made. This study hypothesized that the degree of change in motor
cortex activity during the preparation phase before MI could be associated with MI
performance, and set out to test the hypothesis using an open-access MI-BCI dataset.

10
2.3 Materials and Methods

EEG Signal Acquisition

The open-access MI dataset from BCI competition IV 2b [30] was used for this study.
The dataset includes EEG recordings from 9 subjects during binary MI task (i.e., left hand
and right hand). The training data, which consisted of the first 2 sessions and the 3rd session,
comprised 240 trials without visual feedback and 160 trials with visual feedback,
respectively. The validation data consisted of 2 sessions of EEG data that comprised a total
of 320 trials. EEG data were recorded at a 250 Hz sampling rate from 3 bipolar (C3, Cz, and
C4) and 3 monopolar EOG channels. Details of data acquisition protocol are stated in [30].

Signal Data Preprocessing

The actual recorded data were re-referenced using common average reference (CAR)
and filtered with bandpass of 1 to 40 Hz (FIR filter). The baseline set at -3 to -1 seconds was
removed due to contamination by beep sound and rejected trials was eliminated. The
preprocessing of EEG signals data was conducted using MATLAB R2018b (MathWorks,
MA, USA) and EEGLAB toolbox [31].

Cue Detection Dataset

In order to reflect the change in the state of the brain, the dataset was configured as
follows. In the data after signal processing, the window size is 2.25 seconds, and the update
period is composed of 0.25 seconds. Window size and update period were selected
empirically to match the data balance in learning the model. The EEGNet [32] used in this
study does not perform feature extraction because it uses the temporal signal as it is, and
details are described in Model and training part. Two classifications were performed using
the dataset. The first classifies whether a cue exists in the window, and the second classifies
whether the left or right in the 3~7 seconds MI section.
11
Model and Training

EEGNet was used for both cue detection to detect changes in brain state and MI
classification to distinguish MI of both hands. EEGNet is a model that can be trained with
minimal pre-processing of EEG signals, and the model can be trained with relatively little
trial compared to other artificial intelligence models for EEG analysis.

Model was trained with Adam [33] optimizers and label smoothing cross-entropy loss.
Cosine annealing learning rate scheduler [34] and early stopping [35] were also used. A 5-
fold cross-validation was performed. The EEG signals for the construction (i.e., training and
testing data) of the subject-specific model were collected only within a single subject.

Data Selection

Before MI was performed, data was selected in the following way to determine the
effect of changes in brain state on the classification performance of the model. Using the
softmax values of the model that detected changes in brain states, the difference between the
mean softmax values before and after cue is calculated to compare the distribution across
the entire trial. In order to calculate the optimal threshold of the data selection algorithm
proposed in this study, the criteria of 0 σ(median), 1σ (top 84.1%) and 2σ (top 97.2%) are
used.

12
Spectral Perturbation

The event-related spectral perturbation (ERSP) [36] analysis was performed to confirm
the differences appearing in the three channels. ERSP is a time-frequency analysis of EEG
signals and is an analysis method used to observe the frequency change over time. In this
study, a short-time Fourier transform (STFT) [37] is used to perform time-frequency analysis.
The calculated STFT is analyzed in two ways. First, frequency analysis according to cue was
conducted. Using the grand average on the estimated STFT, we analyzed the change in
power appearing in all subjects. Second, in order to compare the MI of the right hand and
the left hand, ERSP is calculated by the following formula.

(2.1)

Through this calculated ERSP difference, it is possible to observe the change in spectral
power during MI in each channel. The following formula was applied equally in three
channels.

Statistical Analysis

Non-parametric analysis was used in this study due to the small cohort size (each dataset
evaluated 5 times). The differences in the classification accuracy between the models were
assessed by the Wilcoxon rank-sum test. Statistical significance was based on p < 0.05.

ERS
Statistical analysis was performed using SPSS 24 (IBM Corp., Armonk, NY, United States).

13
2.4 Results

Detection of Brain State Transition

In our study, we verified that the brain state transition is detected according to the
change of time (Figure 2.1). Table 2.1 presents the detection accuracy for each of the 5 folds
and the overall performances was 72.42 ± 3.12%. To identify these transition, we performed
supplementary analysis. When trials exceed certain threshold, change of ERSP appeared
before and after cue, as shown in Figure 2.2. Especially, before cue high δ band power was
appeared, after cue the power was decreased 9.95%. Unlike, ERSP of trial that cannot exceed
threshold No noticeable change has emerged at δ band Units.

14
Figure 2.1 ERSP of all subjects (use STFT, window size 2 seconds, update period 0.04 seconds). (A) ERSP of selected
trials, (B) ERSP of non-selected trials

15
 Table 2.1 Cue detection accuracy

Accuracy for each of the 5 folds (%) Overall

Subject
1 2 3 4 5 performance

B01 78.33 76.93 78.2 78.18 77.43 77.81 ± 0.54

B02 68.94 68.71 68.76 68.92 68.71 68.81 ± 0.10
B03 71.89 71.59 72.52 72.39 71.39 71.96 ± 0.44
B04 74.19 71.84 72.79 74.22 68.96 72.40 ± 1.94
B05 78.44 77.55 76.68 77.51 76.45 77.33 ± 0.71
B06 71.63 71.99 72.35 72.43 70.72 71.82 ± 0.62
B07 69.85 71.7 70.37 70.93 71.21 70.81 ± 0.65
B08 68.87 69.85 67.93 68.91 69.78 69.07 ± 0.70
B09 71.76 70.86 72.55 72.18 71.78 71.83 ± 3.12
Total 72.43 ± 3.12

16
Model Performance Improvement Through Optimal Threshold

Optimal MI classification performance analysis according to threshold was performed

using state transition. Because no golden standard exists for threshold, performance changes
were observed for each sigma (Table 2.2, Figure 2.2). When trial selection threshold is
median value, model accuracy was 76.02 ± 9.74% without any significant difference from
the baseline. Also, when top of 85.4% trials is selected significant difference was not
appeared. However, when a top 97.8% trial is selected, it is confirmed that overall
performance is improved by approximately 3.15 % compared to baseline, and that there is a
performance improvement of up to 6.38% for B06 (Wilcoxon rank-sum test, p =0.031).

17
 Table 2.2 Classification performance of selected-base model MI

Accuracy for each trial selection criteria (%)

Subject
Baseline Upper 0 σ Upper -1 σ Upper -2 σ
B01 67.54 ± 1.14 67.37 ± 0.81 70.35 ± 1.43 72.11 ± 0.60
B02 74.29 ± 0.93 73.06 ± 0.63 75.43 ± 1.38 78.20 ± 1.43
B03 83.30 ± 1.37 76.70 ± 1.62 82.26 ± 1.08 84.35 ± 0.99
B04 67.36 ± 0.56 66.84 ± 0.24 67.04 ± 1.45 67.62 ± 1.30
B05 95.09 ± 0.55 94.87 ± 0.80 98.17 ± 0.46 97.14 ± 0.48
B06 79.28 ± 0.98 79.36 ± 1.41 84.94 ± 0.99 85.66 ± 0.50
B07 85.17 ± 1.11 88.28 ± 0.74 89.92 ± 0.21 89.48 ± 0.59
B08 74.52 ± 1.05 74.96 ± 0.52 77.39 ± 0.87 78.70 ± 0.67
B09 59.96 ± 1.33 62.78 ± 1.62 60.73 ± 0.94 61.39 ± 1.41
Total 76.26 ± 10.20 76.02 ± 9.84 78.47 ± 11.09 79.41 ± 10.58

18
Figure 2.2 Comparison of the MI classification accuracy between the total-based and
upper -2σ selection-based model

19
Difference of Each Class

Based on these results, right and left EEG signal of a top 97.8% trial was compared. As
shown Figure 2.3, When left and right MI were performed, C3 and C4 channel showed
different aspects respectively. When imagining the left-hand MI, the power appears to be
10.3 % larger in the α band on the C3 channel and 5.8 % greater in the β band. On the other
hand, right hand MI can see 8.2 percent larger than the high β band of C4. However, in both
hemispheres of the unselected trials, there is no difference in spectral power.

20
 Figure 2.3 ERSP difference plot (A) ERSP difference plot of selected left class trials,
(B) ERSP difference plot of selected right class trials, (C) ERSP difference plot of non-
selected left class trials, (D) ERSP difference plot if non-selected right class trials

21
2.5 Discussion and Conclusion

Previous studies have speculated that MRCPs reflect important information in

predicting the timing of movement by using the pattern in RP [38, 39]. In particular, the
power of the delta band has a significant feature [25]. According to the ERSP results in this
study (Figure 2.1), it was observed that the brain state was changed after the cue appeared,
and these patterns of delta band power appeared in all subjects. The results indicate that the
subjects participating in the experiment were sufficiently aware of the task before MI [28].

Further, by rejecting trials that had no difference between the state before and after cue,
notable increase in classification performance was achieved (Figure 2.2). The rejected trials
could not distinguish between right and left MI ERSP (Figure 2.3). In other words, there is
no significant difference in both hemispheres of brain activity pattern between the right-hand
and the left-hand MI, which could indicate that the subject was not sufficiently prepared to
proceed with the MI task. In this context, selecting appropriate (i.e., well-prepared) trials
might be improving the performance of the classification models.

Some limitations should be mentioned; this study was not able to analyze the overall
changes in brain activity patterns since the EEG was measured using only three electrodes
(C3, C4, and Cz). Second, though the change in delta band power before and after the cue
was observed, it is unclear how this change could affect the MI. Follow-up investigation
using multidimensional MI-BCI dataset is warranted.

22
 CHAPTER 3. Interpretability of Hybrid Feature Using Graph
Neural Networks from Mental Arithmetic Based EEG

The contents of this chapter are approved for publications in: 2023 11th International Winter
Conference on Brain-Computer Interface (BCI).

DOI: 10.1109/BCI57258.2023.10078648

3.1 Summary

A high cognitive load could significantly impair problem-solving skills.

Electroencephalography (EEG)-based real-time assessment of mental workload is feasible,
and graph neural networks (GNN) can classify brain activity patterns during cognitively
demanding tasks with high accuracy. However, previous GNN studies pertaining to mental
workload classification lack explainability. This study utilized a state-of-the-art GNN
variant with GNNexplainer to find relevant connectivity during mental arithmetic (MA)
tasks. In this endeavor, MA EEG recordings were retrieved from an open-access database.
The signals were transformed to graph data through the envelope correlation and power
spectral density (PSD), and subjected to GNN with hierarchical graph pooling with a
structure learning model to classify MA and baseline (BL). The model accuracy was 85.57
± 6.27 and 96.26 ± 4.14% for the connectivity dataset and the PSD and the connectivity
feature, respectively. Among the connections between nodes identified as important by
GNNExplainer, two notable edge patterns were found as 1) from the left centro-parietal
region to left frontal regions, and 2) the frontoparietal connection. The results indicate 1) the
GNN model performance could be improved using the connectivity and PSD feature
together, and 2) characteristic patterns of the connectome and PSD could be important for
MA classification. The connectivity analysis by the “explainable” GNN model could be
beneficial in future brain activity pattern studies.
23
3.2 Introduction

An increased cognitive load could lead to increased human error and decreased
performance in workplaces [40, 41], and temporary attention impairment [42]. Given that
an adequate level of attention and cognitive capacity is crucial for problem-solving [43],
monitoring and evaluation of cognitive load could be helpful in a wide variety of situations.

Previous studies pertaining to real-time evaluation of mental workload utilized

electroencephalogram (EEG) to monitor changes in brain activity patterns during cognitively
demanding task [44-46]. In particular, the EEG bands of theta (4-8 Hz) and alpha (8-12 Hz)
reflect mental workload status [47, 48]; in motor imagery studies, the difference of left/right
hemisphere alpha band powers was found to be associated with the level of attention during
the task [49, 50]. Most previous EEG studies pertaining to mental workload focused on
finding specific EEG features that are associated with cognitive status; however, the changes
in brain activity patterns during tasks could be better assessed by investigating the
connectivity across various brain regions.

The network organization of the brain can be understood via graph theory [51]. With
graph neural network (GNN), a deep learning method designed to be applied to graphs,
recent studies have reported notable classification performance in emotion recognition [52],
identifying reading tasks [53], and mental arithmetic tasks [54]. Nonetheless, the existing
GNN models do not offer an explanation of brain regions and spectral components that were
more relevant in model output.

This study aims to utilize a GNN for the identification of the mental arithmetic (MA)
task. In this endeavor, the connectivity and spectral information were extracted from EEG
recordings during MA task and subjected to a state-of-the-art GNN model. Further, the
GNNExplainer was applied to grant the model an explainability on its output. The methods
described in this study are expected to yield high classification performance, and specify the
brain regions and EEG bands that were relevant to MA tasks.

24
3.3 Materials and Methods

Dataset Description

An open-access EEG dataset that contains recordings of two mental states, i.e., MA and
baseline (BL) were assessed and subjected to analyses [55]. The dataset comprises twenty-
nine healthy subjects (14 males, 15 females) with an average age of 28.5 +- 3.7 years. The
EEG recordings were acquired from 30 electrodes placed according to the international 10-
5 system at a sampling frequency of 200 Hz. The MA experiment consisted of 3 sessions,
each comprised of 20 trials. MA and BL trials were presented randomly 10 times in each
session. During MA tasks, subjects were instructed to repeatedly perform simple arithmetic
calculations. Each subject underwent a total of 60 trials.

Preprocessing

All data processing was conducted with MATLAB R2020a (MathWorks, Natick, MA,
USA) and BBCI toolbox [56]. First, the raw EEG signal was re-referenced using the
common average reference. The recordings of each trial were segmented into 10-seconds
epochs that contained a task period. The dataset was filtered by a band-pass of 1 to 40 Hz
(fifth-order Butterworth filter). Electrooculography (EOG) channels were rejected for
analysis.

25
EEG Connectivity and Power Spectral Density

Edge and node features were extracted from EEG raw signals, because the GNN model
receives graph-type data. For computing edge features, connectivity was utilized. Then, for
extracting node features, the node strength and power spectral density were used. The
connectivity was derived by the envelope correlation method to generate a connectivity
matrix that represents the relationship of inter-node [57]. To compute envelope correlation,
analytic signals were obtained by applying the Hilbert transform, and the mean of Pearson
correlation between orthogonalizing values of two signals was calculated. The inter-node
relations were evaluated via adjacency matrices that represent connectivity, distance, and
degree information. In addition, the node strength was used as node feature. Power spectral
density values of delta (1-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), beta (12-30 Hz), and gamma
(30-40 Hz) frequency bands were utilized to derive node features. Ultimately, each graph
yielded 435 edges above the main diagonal of the square matrix, and the dimension of node
features was 30 * 6.

To compare the effect of PSD features on model performance, a dataset with

connectivity (CON) only and a dataset with connectivity and PSD features (CON-PSD) were
generated.

Experiments

GNN model was trained by optimizing the negative log-likelihood loss function using
Adam [33] and backpropagation. The 5-fold cross-validation method is followed to evaluate
the performance of the trained model; the 5-folds average was calculated to evaluate the final
model performance. To avoid possible overfitting problems, data augmentation was
executed to increase the size of the training dataset size to ten times. Augmented data was
generated by adding Gaussian noise with μ = 0, and σ = 0.001 to the original data. As for the
model evaluation, only the non-augmented original test dataset was used. Experiments were
conducted once for each CON and CON-PSD features.
26
Classification using Graph Neural Networks

To identify and classify the MA task-related brain activity patterns, this study utilized
hierarchical graph pooling with a structure learning model (HGP-SL) [58]. HGP-SL is the
state-of-the-art method for enhancing the effectiveness of GNN, and has shown notable
performance on open datasets for graph classification (e.g., Mutagenicity [59], PROTEINS
[60], etc.). HGP-SL enhances GNN by 1) graph pooling that effectively preserves a subset
of informative nodes, the 2) structure learning that performs graph structure learning for the
pooled subgraph.

Graph pooling operation by HGP-SL generates a subgraph by assigning an information

score, which defined as:

p =  ( i ) = (I −(D ) A ) H
i i
−1
i i
1
(3.1)

, where Ai and Hi were the adjacent and node representations matrices, and || · ||1 performs
L1 norm row-wisely. Di denotes the diagonal degree matrix of Ai, and Ii the identity matrix.
Subsequently, the pooled subgraph is generated by selecting a pooling ratio that of high-
scored nodes. Edge and node features of the subgraph were imported as of selected nodes.

The other characteristic of HGP-SL, i.e., structure learning, derives edges by evaluating
attention score. for nodes that disengage from each other after pooling, the graph structure
information vanishes. To avoid this problem, the HGP-SL employs an attention mechanism
that permutes the similarity between preserved nodes and can be formulated as follows:

(
Ei ( p, q ) =  a  H i ( p,:) H i ( q,:)  ) +   A ( p, q )
i (3.2)

, where σ(·) is the activation function and || represents the concatenation operation. Hi (p, :)
and Hi (q, :) indicate the p-th and q-th row of matrix Hi, which denote the representations of
node vp and vq, respectively.

Next, the similarity score was normalized to better compare the similarity between
27
edges. Normalization was conducted via sparemax function that returns the Euclidean
projection of input onto the probability simple, and can be formulated as follows:

Si ( p, q ) = sparsemax ( Ei ( p, q ) ) (3.3)

sparsemax ( Ei ( p, q ) ) =  Ei ( p, q ) −  ( Ei ( p,:) ) (3.4)

+

, where [x]+ = max [61], and τ(·) was the threshold function that returned a threshold
according to the procedure. Thus, sparsemax preserves the values above the threshold, and
the other values will be truncated to zeros.

Graph level of hierarchical representation value was extracted from each layer by
readout method that performed global mean pooling and global max pooling. The last layer
(e.g., multi-layer perceptron (MLP) and softmax layer) estimates the label using the
concatenated value of representation after the HGP-SL layer.

GNNExplainer

The GNNExplainer identified a subgraph structure that has a crucial role in trained
GNN’s classification [62]. The primary objective for the GNNExplainer is to generate a
subgraph that explains the decision. This process is maximizing the mutual information
between the subgraph and the computation graph. The secondary objective was the subgraph
needs to be minimal. The GNNExplainer is literally the combination of prediction loss and
edge size loss. The GNNExplainer was performed in the following order. In first, the
computation graph which is the k-hops neighbor for node classification, or the entire graph
for graph classification was extracted. Then, a mask for each edge in the computation graph
was initialized. And then, a neural network that learns the mask with loss was constructed.
Lastly, the mask was used, to reduce the computation graph to a minimal graph.

28
3.4 Results

Model Performance

The HGP-SL enhanced GNN yielded better classification performance on CON-PSD

compared to CON feature as shown in Figure 3.1. The mean accuracy was 85.57 ± 6.27 and
96.26 ± 4.14% for the CON feature and the CON-PSD feature, respectively. In CON feature,
the best accuracy was found in subject 21 (95.00 ± 6.67%), whereas the lowest accuracy was
from subject 20 (73.33 ± 16.99%) presented in Figure 3.1A. In CON-PSD feature, the best
classification accuracy was found in subject 24 (100.00%), and the lowest performance was
from subject 20 (80.00 ± 13.54%) illustrated in Figure 3.1B.

29
Figure 3.1 Model performance trained on both datasets. (A) is about connectivity (CON) and (B) is about connectivity and
PSD features (CON-PSD). Red, green, and blue donate accuracy, recall, and precision, respectively

30
PSD and Envelope Connectivity Pattern

During MA trials, certain trends were observed in connectivity and PSD. As for the
connectivity, the connectome within frontal channels (AFp1, AFp2, AFF1h, AFF2h, AFF5h,
AFF6h, F3, F4, F7, and F8) was increased or maintained, as illustrated in Figure 3.2. In
addition, the connection within the parieto-occipital lesion (P4, P8, PPO2h, POO2, Pz,
POO1, PPO1h, P3, and P7) was stronger than baseline. In terms of PSD, theta-band power
at the frontal electrode had increased, whereas beta-band power had decreased in the central
region.

31
Figure 3.2 Circular graphs depicting the connectivity of envelope correlation between nodes,
and topographic visualization of each PSD band. (A), (B), and (C) were obtained from 20,
21, 24 subjects in CON-PSD dataset. The upper and lower rows represented each baseline
(BL), mental arithmetic (MA). Delta, theta, alpha, beta, and gamma PSD bands were colored
purple, blue, green, orange, and red, respectively.

32
Explanation of Model using GNNExplainer

Certain subgraph patterns that derived by the GNNExplainer, such as Figure 3.3A, C in
BL, and Figure 3.3B in MA, were observed in all subjects. Two prominent patterns were 1)
from CCP5h to the frontal regions (AFF1h, AFp1, AFF5h, and F3), and 2) from AFF5h to
POO1.

33
Figure 3.3 Average of subgraphs that found as important in task classification by the
GNNExplainer in (A) subject 20, (B) subject 21, and (C) subject 24

34
3.5 Discussion and conclusion

Adding PSD to CON yielded a significant increase in classification performance; even

for subject 20, where both models returned the lowest accuracy, CON-PSD still yielded a
slightly better accuracy (73.33 ± 16.99 vs. 80.00 ± 13.54 %). The results suggest that by
considering not only the connectivity but also the power density of the frequency bands,
higher performance can be achieved for MA classification using the GNN model.

Previous studies regarding the connectivity during MA tasks reported that the
connectivity of the frontal and the left parietal regions are associated with cognitive
workload, and active involvement of left parietal area during MA [63-65]. The findings of
this study are consistent with previous studies; the connection within the frontal region was
sustained or increased, and the parieto-occipital lesion (P4, P8, PPO2h, POO2, Pz, POO1,
PPO1h, P3, and P7) was included in the connectivity. Further, there was an incremental
tendency in the theta band power of the frontal region, similar to a previous MA study which
suggested that high frontal theta activities are related to cognitive control and concentration
[63]. In addition, there was a tendency that the beta band power in the central cortex
decreased, which is suspected to associated with mental workload [63, 66].

Among the connections between nodes identified as important by GNNExplainer, two

notable patterns were found in Figure 3.3: edges from CCP5h to frontal regions (AFF1h,
AFp1, AFF5h, and F3), and the frontoparietal connection from AFF5h to POO1. The former
could be reflecting left superior parietal gyrus activity during performing subtraction [18],
whereas the latter reflecting frontoparietal connectivity which is associated with effective
mental arithmetic tasks [67]. The findings of this study indicate that the connectivity patterns
reported here could be important in MA classification, and that the connectivity analysis by
“explainable” GNN could be beneficial in investigating brain activity patterns during
cognitively demanding tasks.

Limitations of this study should be mentioned. First, this research was performed in an
offline environment with retrospective data. Second, the adding PSD information resulted
35
in significant increase in classification performance, however the explainability measure
utilized in this study was not applicable to PSD features. Future studies with similar design
should aim to validate the proposed methods, preferably with a real-time, prospective design
in a larger cohort.

36
Part III:
Conclusions

37
 CHAPTER 4. Concluding Remarks

4.1 Summary and Major Findings

In the study, the potential for enhancing the understanding of spatial neural activity and
brain functional connectivity patterns during MI and MA tasks was demonstrated through
the use of deep learning-based EEG-BCI applications.

In Chapter 2, a quantitative trial rejection methodology was proposed that could

improve the performance of the MI classification model. Furthermore, it was confirmed that
the deep learning model used for MI classification can learn features related to MRCPs. The
proposed methodology partially overcomes the limitations of conventional trial rejection
techniques that rely on visual inspection, and it could be an effective alternative for future
researchers who wish to develop EEG-BCI systems based on MI with high performance.

Chapter 3 presented a GNN model that can reasonably classify spatial neural activity
and functional connectivity during MA. Additionally, an explainable artificial intelligence
(XAI) technique was used to investigate the model's explanation. The input to the GNN
model, consisting of PSD and connectivity patterns, showed features related to attention and
mental workload, indicating that it contained rich features for classification. Moreover, GNN
XAI identified the frontoparietal connection from the centroparietal region to the left frontal
region as an important feature for MA classification. The findings of this study indicate that
the connectivity patterns reported here could be important in MA classification, and that the
connectivity analysis by “explainable” GNN could be beneficial in investigating brain
activity patterns during cognitively demanding tasks.

38
4.2 Limitations and Suggestions

Several shared limitations of the studies introduced in this thesis should be mentioned.
First, the study designs and data analysis were conducted in offline environments. To apply
BCI system in real-life device control, an online system that has low computational
complexity, enables real-time analysis, and having simplified measurement systems is
required. Second, the results of the above-mentioned studies may have insufficient statistical
significance due to the small number of subjects. Hence, validation of the results with
additional data is required to ensure statistical reliability. Third, all participants in the
experiment are healthy. Although a major objective of the BCI application is to help patients
with motor disabilities whose brain activity might be different from healthy individuals,
external validation with data from patients with neuromotor disabilities is required to
compensate such differences. Future works will validate the results with more sample size
data from motor disabled patients.

39
4.3 Conclusions

This thesis investigates deep learning methods for MI and MA classification systems.
For MI classification, a methodology is proposed to quantitatively reject trials based on
changes that occur before and during mental tasks. The proposed method achieved higher
performance on a relatively small dataset, contrary to the common belief that the
performance of deep learning models improves with more data. Based on the absence of
MRCPs in the rejected data, it can be inferred that the lack of changes in EEG signals with
respect to cues indicates decreased subject concentration. For MA classification, a GNN
model is developed that reasonably classifies spatial neural activity and functional
connectivity. The developed model achieves high performance by integrating node and edge
features for analysis. In addition, XAI verifies that the GNN model highlights features
related to attention and working memory for the purpose of classification. The investigated
methods in all processes of MI and MA BCI hold promise for enhancing the practical utility
of high performance BCIs.

40
 REFERENCES

[1] J. R. Wolpaw, "Brain-computer interfaces (BCIs) for communication and control,"

in Proceedings of the 9th international ACM SIGACCESS conference on Computers
and accessibility, 2007, pp. 1-2.
[2] M. Rashid, N. Sulaiman, A. PP Abdul Majeed, R. M. Musa, B. S. Bari, and S.
Khatun, "Current status, challenges, and possible solutions of EEG-based brain-
computer interface: a comprehensive review," Frontiers in neurorobotics, p. 25,
2020.
[3] E. H. Houssein, A. Hammad, and A. A. Ali, "Human emotion recognition from
EEG-based brain–computer interface using machine learning: a comprehensive
review," Neural computing and applications, vol. 34, no. 15, pp. 12527-12557, 2022.
[4] S. Wainio-Theberge, A. Wolff, and G. Northoff, "Dynamic relationships between
spontaneous and evoked electrophysiological activity," Communications biology,
vol. 4, no. 1, p. 741, 2021.
[5] J. Kaur and A. Kaur, "A review on analysis of EEG signals," in 2015 International
conference on advances in computer engineering and applications, 2015: IEEE, pp.
957-960.
[6] A. Craik, Y. He, and J. L. Contreras-Vidal, "Deep learning for
electroencephalogram (EEG) classification tasks: a review," Journal of neural
engineering, vol. 16, no. 3, p. 031001, 2019.
[7] M.-P. Hosseini, A. Hosseini, and K. Ahi, "A review on machine learning for EEG
signal processing in bioengineering," IEEE Reviews in biomedical engineering, vol.
14, pp. 204-218, 2020.
[8] M. Jeannerod and J. Decety, "Mental motor imagery: a window into the
representational stages of action," Current opinion in neurobiology, vol. 5, no. 6, pp.
727-732, 1995.
[9] J. Munzert, B. Lorey, and K. Zentgraf, "Cognitive motor processes: the role of motor
imagery in the study of motor representations," Brain research reviews, vol. 60, no.
2, pp. 306-326, 2009.
[10] R. Kus and J. Ginter, "Propagation of EEG activity during finger movement and its
imagination," Acta neurobiologiae experimentalis, vol. 66, no. 3, pp. 195-206, 2006.
[11] C. Neuper and G. Pfurtscheller, "Event-related dynamics of cortical rhythms:
frequency-specific features and functional correlates," International journal of
psychophysiology, vol. 43, no. 1, pp. 41-58, 2001.
[12] C. Neuper and G. Pfurtscheller, "Evidence for distinct beta resonance frequencies in
human EEG related to specific sensorimotor cortical areas," Clinical
neurophysiology, vol. 112, no. 11, pp. 2084-2097, 2001.
[13] J. R. Wolpaw, D. J. McFarland, and T. M. Vaughan, "Brain-computer interface
research at the Wadsworth Center," IEEE Transactions on rehabilitation
engineering, vol. 8, no. 2, pp. 222-226, 2000.
[14] S. Dehaene, N. Molko, L. Cohen, and A. J. Wilson, "Arithmetic and the brain,"
Current opinion in neurobiology, vol. 14, no. 2, pp. 218-224, 2004.
41
[15] R. Cabeza and L. Nyberg, "Imaging cognition II: An empirical review of 275 PET
and fMRI studies," Journal of cognitive neuroscience, vol. 12, no. 1, pp. 1-47, 2000.
[16] M. Rugg and A. Dickens, "Dissociation of alpha and theta activity as a function of
verbal and visuospatial tasks," Electroencephalography and clinical
neurophysiology, vol. 53, no. 2, pp. 201-207, 1982.
[17] A. Mecklinger, A. F. Kramer, and D. L. Strayer, "Event related potentials and EEG
components in a semantic memory search task," Psychophysiology, vol. 29, no. 1,
pp. 104-119, 1992.
[18] J. Kong, C. Wang, K. Kwong, M. Vangel, E. Chua, and R. Gollub, "The neural
substrate of arithmetic operations and procedure complexity," Cognitive brain
research, vol. 22, no. 3, pp. 397-405, 2005.
[19] D. J. Crammond, "Motor imagery: never in your wildest dream," Trends in
neurosciences, vol. 20, no. 2, pp. 54-57, 1997.
[20] M. Jeannerod, "The representing brain: Neural correlates of motor intention and
imagery," Behavioral and brain sciences, vol. 17, no. 2, pp. 187-202, 1994.
[21] J. R. Wolpaw et al., "Brain-computer interface technology: a review of the first
international meeting," IEEE Transactions on rehabilitation engineering, vol. 8, no.
2, pp. 164-173, 2000.
[22] T. N. Lal et al., "A brain computer interface with online feedback based on
magnetoencephalography," in Proceedings of the 22nd international conference on
machine learning, 2005, pp. 465-472.
[23] U. Chaudhary, N. Birbaumer, and A. Ramos-Murguialday, "Brain–computer
interfaces for communication and rehabilitation," Nature reviews neurology, vol. 12,
no. 9, pp. 513-525, 2016.
[24] G. Pfurtscheller et al., "Graz-BCI: state of the art and clinical applications," IEEE
Transactions on neural systems and rehabilitation engineering, vol. 11, no. 2, pp.
1-4, 2003.
[25] K. Dremstrup, Y. Gu, O. F. d. Nascimento, and D. Farina, "Movement‐related
cortical potentials and their application in brain–computer interfacing," Introduction
to neural engineering for motor rehabilitation, pp. 253-266, 2013.
[26] O. F. d. Nascimento, K. D. Nielsen, and M. Voigt, "Movement-related parameters
modulate cortical activity during imaginary isometric plantar-flexions,"
Experimental brain research, vol. 171, pp. 78-90, 2006.
[27] D. J. Wright, P. S. Holmes, and D. Smith, "Using the movement-related cortical
potential to study motor skill learning," Journal of motor behavior, vol. 43, no. 3,
pp. 193-201, 2011.
[28] H. H. Kornhuber and L. Deecke, "Hirnpotentialänderungen bei Willkürbewegungen
und passiven Bewegungen des Menschen: Bereitschaftspotential und reafferente
Potentiale," Pflüger's archiv für die gesamte physiologie des menschen und der tiere,
vol. 284, pp. 1-17, 1965.
[29] M. Arvaneh, I. Robertson, and T. E. Ward, "Subject-to-subject adaptation to reduce
calibration time in motor imagery-based brain-computer interface," in 2014 36th
Annual International Conference of the IEEE Engineering in medicine and biology
society, 2014: IEEE, pp. 6501-6504.
42
[30] R. Leeb, C. Brunner, G. Müller-Putz, A. Schlögl, and G. Pfurtscheller, "BCI
Competition 2008–Graz data set B," Graz university of technology, Austria, vol. 16,
pp. 1-6, 2008.
[31] A. Delorme and S. Makeig, "EEGLAB: an open source toolbox for analysis of
single-trial EEG dynamics including independent component analysis," Journal of
neuroscience methods, vol. 134, no. 1, pp. 9-21, 2004.
[32] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J.
Lance, "EEGNet: a compact convolutional neural network for EEG-based brain–
computer interfaces," Journal of neural engineering, vol. 15, no. 5, p. 056013, 2018.
[33] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv
preprint arXiv:1412.6980, 2014.
[34] I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts,"
arXiv preprint arXiv:1608.03983, 2016.
[35] Y. Yao, L. Rosasco, and A. Caponnetto, "On early stopping in gradient descent
learning," Constructive approximation, vol. 26, no. 2, pp. 289-315, 2007.
[36] S. Makeig, "Auditory event-related dynamics of the EEG spectrum and effects of
exposure to tones," Electroencephalography and clinical neurophysiology, vol. 86,
no. 4, pp. 283-293, 1993.
[37] D. Griffin and J. Lim, "Signal estimation from modified short-time Fourier
transform," IEEE Transactions on acoustics, speech, and signal processing, vol. 32,
no. 2, pp. 236-243, 1984.
[38] I. K. Niazi, N. Jiang, M. Jochumsen, J. F. Nielsen, K. Dremstrup, and D. Farina,
"Detection of movement-related cortical potentials based on subject-independent
training," Medical & biological engineering & computing, vol. 51, pp. 507-512,
2013.
[39] I. K. Niazi, N. Jiang, O. Tiberghien, J. F. Nielsen, K. Dremstrup, and D. Farina,
"Detection of movement intention from single-trial movement-related cortical
potentials," Journal of neural engineering, vol. 8, no. 6, p. 066009, 2011.
[40] R. Parasuraman, "Neuroergonomics: Brain, cognition, and performance at work,"
Current directions in psychological science, vol. 20, no. 3, pp. 181-186, 2011.
[41] S. Shappell, C. Detwiler, K. Holcomb, C. Hackworth, A. Boquet, and D. A.
Wiegmann, "Human error and commercial aviation accidents: an analysis using the
human factors analysis and classification system," in Human error in aviation:
routledge, 2017, pp. 73-88.
[42] S. Mun, M. Whang, S. Park, and M.-C. Park, "Effects of mental workload on
involuntary attention: A somatosensory ERP study," Neuropsychologia, vol. 106,
pp. 7-20, 2017.
[43] T. Radüntz, "The effect of planning, strategy learning, and working memory
capacity on mental workload," Scientific reports, vol. 10, no. 1, pp. 1-10, 2020.
[44] G. F. Wilson and C. A. Russell, "Real-time assessment of mental workload using
psychophysiological measures and artificial neural networks," Human factors, vol.
45, no. 4, pp. 635-644, 2003.
[45] P. Gerjets, C. Walter, W. Rosenstiel, M. Bogdan, and T. O. Zander, "Cognitive state
monitoring and the design of adaptive instruction in digital environments: lessons
43
 learned from cognitive workload assessment using a passive brain-computer
interface approach," Frontiers in neuroscience, vol. 8, p. 385, 2014.
[46] S. Puma, N. Matton, P.-V. Paubel, É . Raufaste, and R. El-Yagoubi, "Using theta and
alpha band power to assess cognitive workload in multitasking environments,"
International journal of psychophysiology, vol. 123, pp. 111-120, 2018.
[47] D. Grimes, D. S. Tan, S. E. Hudson, P. Shenoy, and R. P. Rao, "Feasibility and
pragmatics of classifying working memory load with an electroencephalograph," in
Proceedings of the SIGCHI conference on human factors in computing systems,
2008, pp. 835-844.
[48] A. Gundel and G. F. Wilson, "Topographical changes in the ongoing EEG related
to the difficulty of mental tasks," Brain topography, vol. 5, no. 1, pp. 17-25, 1992.
[49] S. Lee, C.-H. Lee, H. Kim, and D.-j. Kim, "Lateralization of alpha oscillation under
preparation Lead to Efficiency of Motor Imagery: Related with Performance of
Classification," in 2020 IEEE International conference on systems, man, and
cybernetics (SMC), 2020: IEEE, pp. 2502-2505.
[50] S. Lee, Y.-T. Kim, S.-O. Hwang, H. Kim, and D.-J. Kim, "Importance of reliable
EEG data in motor imagery classification: Attention level-based approach," in 2020
8th International winter conference on brain-computer interface (BCI), 2020: IEEE,
pp. 1-4.
[51] E. Bullmore and O. Sporns, "Complex brain networks: graph theoretical analysis of
structural and functional systems," Nature reviews neuroscience, vol. 10, no. 3, pp.
186-198, 2009.
[52] P. Zhong, D. Wang, and C. Miao, "EEG-based emotion recognition using
regularized graph neural networks," IEEE Transactions on affective computing,
2020.
[53] P. Mathur, T. Mittal, and D. Manocha, "Dynamic Graph Modeling Of Simultaneous
EEG And Eye-Tracking Data For Reading Task Identification," in ICASSP 2021-
2021 IEEE International conference on acoustics, speech and signal processing
(ICASSP), 2021: IEEE, pp. 1250-1254.
[54] A. Shalbaf and A. Maghsoudi, "Mental Arithmetic Task Recognition Using
Effective Connectivity and Hierarchical Feature Selection from EEG Signals,"
Basic and clinical neuroscience, pp. 0-0, 2020.
[55] J. Shin et al., "Open access dataset for EEG+ NIRS single-trial classification," IEEE
Transactions on neural systems and rehabilitation engineering, vol. 25, no. 10, pp.
1735-1745, 2016.
[56] J. F. Hipp, D. J. Hawellek, M. Corbetta, M. Siegel, and A. K. Engel, "Large-scale
cortical correlation structure of spontaneous oscillatory activity," Nature
neuroscience, vol. 15, no. 6, pp. 884-890, 2012.
[57] S. Khan et al., "Maturation trajectories of cortical resting-state networks depend on
the mediating frequency band," NeuroImage, vol. 174, pp. 57-68, 2018.
[58] Z. Zhang et al., "Hierarchical graph pooling with structure learning," arXiv preprint
arXiv:1911.05954, 2019.
[59] K. Riesen and H. Bunke, "IAM graph database repository for graph based pattern
recognition and machine learning," in Joint IAPR International workshops on
44
 statistical techniques in pattern recognition (SPR) and structural and syntactic
pattern recognition (SSPR), 2008: Springer, pp. 287-297.
[60] K. M. Borgwardt, C. S. Ong, S. Schönauer, S. Vishwanathan, A. J. Smola, and H.-
P. Kriegel, "Protein function prediction via graph kernels," Bioinformatics, vol. 21,
no. suppl_1, pp. i47-i56, 2005.
[61] J. Leppink, F. Paas, T. Van Gog, C. P. van Der Vleuten, and J. J. Van Merrienboer,
"Effects of pairs of problems and examples on task performance and different types
of cognitive load," Learning and instruction, vol. 30, pp. 32-42, 2014.
[62] R. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, "Gnnexplainer:
Generating explanations for graph neural networks," Advances in neural
information processing systems, vol. 32, p. 9240, 2019.
[63] G. N. Dimitrakopoulos et al., "Task-independent mental workload classification
based upon common multiband EEG cortical connectivity," IEEE Transactions on
neural systems and rehabilitation engineering, vol. 25, no. 11, pp. 1940-1949, 2017.
[64] Y. Kitaura et al., "Functional localization and effective connectivity of cortical theta
and alpha oscillatory activity during an attention task," Clinical neurophysiology
practice, vol. 2, pp. 193-200, 2017.
[65] K. Katahira, Y. Yamazaki, C. Yamaoka, H. Ozaki, S. Nakagawa, and N. Nagata,
"EEG correlates of the flow state: A combination of increased frontal theta and
moderate frontocentral alpha rhythm in the mental arithmetic task," Frontiers in
psychology, vol. 9, p. 300, 2018.
[66] J. Shin, J. Kwon, and C.-H. Im, "A ternary hybrid EEG-NIRS brain-computer
interface for the classification of brain activation patterns during mental arithmetic,
motor imagery, and idle state," Frontiers in neuroinformatics, vol. 12, p. 5, 2018.
[67] T. Hinault, K. Larcher, L. Bherer, S. M. Courtney, and A. Dagher, "Age-related
differences in the structural and effective connectivity of cognitive control: A
combined fMRI and DTI study of mental arithmetic," Neurobiology of aging, vol.
82, pp. 30-39, 2019.

45