412 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO.

3, JUNE 2021

Hardware Acceleration of EEG-Based Emotion

Classification Systems: A Comprehensive Survey
Hector A. Gonzalez , Student Member, IEEE, Richard George, Senior Member, IEEE,
Shahzad Muzaffar , Student Member, IEEE, Javier Acevedo , Student Member, IEEE, Sebastian Höppner ,
Christian Mayr , Member, IEEE, Jerald Yoo , Senior Member, IEEE, Frank H.P. Fitzek , Senior Member, IEEE,
and Ibrahim M. Elfadel , Senior Member, IEEE

Abstract—Recent years have witnessed a growing interest in
EEG-based wearable classifiers of emotions, which could enable
the real-time monitoring of patients suffering from neurological
disorders such as Amyotrophic Lateral Sclerosis (ALS), Autism
Spectrum Disorder (ASD), or Alzheimer’s. The hope is that such
wearable emotion classifiers would facilitate the patients’ social
integration and lead to improved healthcare outcomes for them
and their loved ones. Yet in spite of their direct relevance to
neuro-medicine, the hardware platforms for emotion classification
have yet to fill up some important gaps in their various approaches
to emotion classification in a healthcare context. In this paper, we
present the first hardware-focused critical review of EEG-based
wearable classifiers of emotions and survey their implementation
perspectives, their algorithmic foundations, and their feature ex-
traction methodologies. We further provide a neuroscience-based
analysis of current hardware accelerators of emotion classifiers and
Manuscript received March 31, 2021; revised May 28, 2021; accepted June
6, 2021. Date of publication June 14, 2021; date of current version August 17,
2021. This work was partially supported by the Khalifa University Center for
Cyber Physical Systems (C2PS) and by the German Research Foundation (DFG,
Deutsche Forschungsgemeinschaft) as part of Germany’s Excellence Strategy
- EXC 2050/1 - Project ID 390696704 - Cluster of Excellence “Centre for
Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität
Dresden. This paper was recommended by Associate Editor Prof. Chul Kim.
(Corresponding author: Hector A. Gonzalez.)
Hector A. Gonzalez, Richard George, and Sebastian Höppner are
with the Chair for Highly-Parallel VLSI-Systems and Neuromorphic
Circuits, Technische Universität Dresden, 01062 Dresden, Germany (e-
mail: hector.gonzalez@tu-dresden.de; richard_miru.george@tu-dresden.de;
sebastian.hoeppner@tu-dresden.de).
Javier Acevedo is with the Deutsche Telekom Chair of Communication
Networks, Technische Universität Dresden, 01187 Dresden, Germany (e-mail:
javier.acevedo@tu-dresden.de).
Christian Mayr is with the Chair for Highly-Parallel VLSI-Systems and Neu-
romorphic Circuits, Technische Universität Dresden, 01062 Dresden, Germany,
and also with the Centre for Tactile Internet (CeTi) with Human-in-the-Loop,
Cluster of Excellence, Dresden, Technische Universität Dresden, 01187 Dres-
den, Germany (e-mail: christian.mayr@tu-dresden.de).
Jerald Yoo is with the Department of Electrical and Computer Engineer-
ing, National University of Singapore, Singapore 117583, Singapore, and
also with the N.1 Institute for Health, Singapore 117456, Singapore (e-mail:
jyoo@nus.edu.sg).
Frank H.P. Fitzek is with the Deutsche Telekom Chair of Communication
Networks, Technische Universität Dresden, 01187 Dresden, Germany, and also
with the Centre for Tactile Internet (CeTi) with Human-in-the-Loop, Cluster of
Excellence, Dresden, Technische Universität Dresden, 01187 Dresden, Germany
(e-mail: frank.fitzek@tu-dresden.de).
Shahzad Muzaffar and Ibrahim M. Elfadel are with the Department of Elec-
trical Engineering and Computer Science, and the Center for Cyber Physical
Systems (C2PS), Khalifa University, Abu Dhabi 127788, United Arab Emirates
(e-mail: shahzad.muzaffar@ku.ac.ae; ibrahim.elfadel@ku.ac.ae).
Color versions of one or more figures in this article are available at https:
//doi.org/10.1109/TBCAS.2021.3089132.
Digital Object Identifier 10.1109/TBCAS.2021.3089132
1932-4545 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
use it to map out several research opportunities, including multi-
modal hardware platforms, accelerators with tightly-coupled cores
operating robustly in the near/supra-threshold region, and pre-
processing libraries for universal EEG-based datasets.
Index Terms—Emotion detection and classification, EEG,
hardware acceleration, machine learning, monitoring of
neurological disorders.
I. INTRODUCTION
MOTION classification using EEG features is a promising
E path to achieving social integration of patients suffering
from neurological disorders that involve locked-in emotional
states. In contrast to face-based emotion detectors whose ac-
curacy depends on external muscular responses, EEG-based
classifiers provide a non-intrusive window into the fluctuations
of the internal organismic subsystems, which are measurable
despite the presence of a paralytic condition. Recent years have
witnessed sustained efforts to design EEG-based software and
hardware classifiers for emotions. The focus of this paper is to
critically survey the hardware classifiers as they are the linchpins
of wearable, robust, real-time medical devices with small foot-
print and long battery life that aim at improving the healthcare
outcomes for patients with neurological disorders. Inspired by
the state-of-the-art software solutions, the paper will establish
a common framework to analyze the emerging hardware ones
and will use it to identify unexplored paths for novel hardware
designs. The paper will further bring a Neuroscience perspective
to bear on the existing and emerging engineering approaches to
the hardware acceleration of emotion classifiers.
To the best of our knowledge, this is the first extensive
hardware-focused review of EEG-based emotion classification
systems. Most of the prior research related to EEG-based emo-
tion detection has focused on analyzing multi-modal contri-
butions [1], surveying generic EEG-based Deep Learning ap-
proaches [2], reviewing the available transfer learning mech-
anisms for general EEG classification tasks [3], and analyzing
existing solutions [4]–[6], all from a software perspective, which
unfortunately excludes the hard challenges of designing a robust,
wearable hardware emotion classifier under severe constraints
of area, power and stringent specifications on accuracy and
reliability. A recent survey [7] with focus on hardware design for
biomedical applications discusses the area of emotion detection
in the context of “AI-based EEG processing algorithm design”

Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
and briefly presents three of the hardware emotion classifiers,
listing only their methods and accuracies. Despite some similar-
ities, this survey has several distinguishing features, including:
1) Review focus: In this paper, we are laser-focused on the
hardware acceleration challenges of EEG-based emotion
classification systems as we believe it is the domain that
is most in need of research toward the development of
wearable healthcare solutions.
2) Integrated in-depth analysis: The paper includes and inte-
grates the most recent hardware contributions in one single
framework that facilitates their detailed, comparative in-
depth analysis, while overlaying them with the software
state of the art.
3) Neuroscience awareness: The paper brings a Neuroscience
perspective to bear on the engineering aspects of emotion
classification, and in the process, identifies several gaps
in existing approaches and unveils unexplored paths to
narrow them.
4) Thorough dataset analysis: Taking healthcare as the major
objective, the paper analyzes the existing datasets and
benchmarks in terms of their suitability for enabling highly
accurate, and clinically specific emotion classification.
5) Holistic perspective: In contrast to the mainstream
accuracy-driven, machine-learning paradigm, this paper
emphasizes the importance of other metrics that are rel-
evant to wearable emotion classifiers such as footprint,
low-power, and implementation complexity. The hope is
that such holistic approach would lead to a more system-
atic search of the design space for hardware accelerators
of emotion classifiers.
The remainder of this article is organized as follows. Section II
is devoted to the methodology we have used in our bibliographic
selections from the state of the art. Section III describes the
existing hardware solutions and contrasts them with the software
ones. In particular, this section contains a comprehensive survey
of pre-processing and feature extraction methods in EEG-based
emotion detection hardware as well as a wearability metric for
such hardware. Given the stringent constraints that hardware
imposes on classification algorithms, Section IV has been de-
voted to a detailed review of hardware-friendly classification
algorithms and their performance in the context of EEG-based
emotion detection hardware systems. In Section V, a close
look is taken at the hardware system specifications of wearable
EEG-based emotion detection systems with particular emphasis
not only on traditional metrics such as power, area, and latency,
but also on the specifications that are relevant to the problem
domain of EEG-based emotion detection. Issues such as train-
ing, electrode engineering, and verification are discussed in In
Section V as well. In Section VI, challenges and opportunities
in the field of wearable EEG-based emotion detection systems
are identified, with interlock with the neurosciences being one
of the major challenges while presenting the neurosciences with
a hardware platform for the real-time monitoring of neurolog-
ical disorders being a major opportunity. Section VII presents
recommendations for further research. The recommendations
are based on the multi-dimensional analysis and evaluation of
existing hardware for EEG-based emotion detection we have
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
413
Fig. 1. Hardware (HW) and Software (SW) contributions per publication year.
given in the earlier sections. The paper is closed with a brief
conclusion in Section VIII.
II. SURVEY METHODOLOGY
Before introducing the hardware solutions for emotion clas-
sification, we briefly review, in this section, the bibliographic
methodology of our survey, especially in relation to article
selection and evaluation from the perspective of the EEG-based
emotion classification problem. Our hardware-focused survey is
not intended to replace the multiple excellent software-focused
surveys [1]–[6], but it rather leans on the various software
approaches as reference points to contextualize the EEG-based
emotion classification problem and guide the analysis of the
hardware-based solutions.
The survey presented by S. M. Alarcão in [4] selects 63
software-oriented papers based on their feature extraction meth-
ods and algorithmic analysis, as well as their compliance with
standard practices in neuroscience experiments as spelled out
in [8]. All these 63 papers are included in our software ref-
erences. We further include the 13 papers of [2] where recent
work on deep neural networks in relation to EEG-based emo-
tion detection was reviewed. The pool of 86 papers is further
extended using articles directly extracted form the IEEE Xplore
database. Out of the 540 hits from the IEEE Xplore database for
the 2017–2020 period, following the 2017 publication of [4], 94
papers have been selected from journals and conferences with
an impact factor above 1. This initial list of 180 papers was
further down-selected to 151 papers based on topical pruning,
i.e., whether the paper addressed EEG-based emotion detection
or not. This final collection of 151 papers is used in the following
sections to contextualize the emerging hardware solutions and
contrast them with more established software.
As shown in Fig. 1, the EEG-based emotion detection field has
gained tremendous popularity over the past decade. Numerous
algorithms, a wide variety of feature extraction methods, several
input stimuli, extensive datasets, and a whole gamut of training
paradigms, have been reported in the recent literature and are
evidence of the significant progress already made. However, up
until 2018 or so, the contributions were mainly software-focused
414 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021

TABLE I
FEATURE EXTRACTION METHODS IN SW AND HW

STFT: Short-time Fourier transform. HOC: Higher Order Crossings. DE: Differential Entropy. CSP: Common Spatial Pattern.
HFD: Higuchi Fractal Dimension. AR: Autoregressive model. FBCSP: Filter Bank Common Spatial Pattern. DCT: Discrete Cosine Transform.
FD: Fractal Dimension. GP: Grassberger-Procaccia algorithm. MFCC: Mel-frequency Cepstral Coefficients. PCA: Principal Component Analysis.
ARRC: Autoregression Reflection Coefficients. ASP: Asymmetric Spatial Pattern. BDGLS: Broad Dynamical Graph Learning System.
CS: Compressed Sensing. DTF: Directed Transfer Function. EMD: Empirical Mode Decomposition. GCB-Net: Graph Convolutional Broad Network.
HAF: Hybrid Adaptive filtering. HHS: Hilbert-Huang Spectrum. HOS: Higher Order Spectral analysis. LBP: Local Binary Patterns.
LFCC: Linear frequency Cepstral Coefficients. LPP: Late Positive Potential. MSCE: Magnitude Squared Coherence Estimate. D.† : Distance.
NEE: Narrow band energy event. NTSPP: Neural Time series Prediction Pre-processing. SPWV: Smoothed Pseudo-Wigner-Ville distribution.

with the standing assumptions being the availability of abundant
hardware resources and off-line operation. The lack of an early
hardware perspective may be easily inferred from the high
computational complexity of the feature extraction methods and
algorithms of the software approaches listed in Tables I and II.
Since 2018, a growing trend in hardware-focused approaches
has been noticed as is clear from the black curve in Fig. 1. This
trend has helped in highlighting the many challenges and the
upside potential of wearable devices for the EEG-based emotion
classification problem. Such devices have the potential to enable
near real-time, robust, clinically relevant emotion classification,
under very small form factors and for extended periods of oper-
ation in a medical environment. The near real-time operation
could improve healthcare outcomes and facilitate the social
re-integration of patients suffering from neurological diseases
such as Alzheimer’s, Amyotrophic Lateral Sclerosis (ALS),
and Autism Disorder (AD). It is one of the major aims of this
paper to give a detailed account of the potential of the hardware
approaches in contrast to the software ones, while identifying
the opportunities and the unexplored paths that can guide future
research and development in this important biomedical domain.
III. EEG-BASED HARDWARE SOLUTIONS
In this section, the hardware solutions to EEG-based emotion
detection are compared in terms of wearability, pre-processing
techniques, and feature extraction methods. In Section IV, they
will be compared in terms of their classification algorithms,
while in Section V, they will be compared based on hardware
implementation and verification metrics, including power, area,
and latency. The main criterion for inclusion in the hardware
survey list is for the publication to report on an online inference
implemented in either FPGA or ASIC, regardless of the classifier
used. Two hardware contributions [9], [10], which are part of the

Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY 415

TABLE II
ALGORITHMS IN SW AND HW PLATFORMS

AI-VAE: Attribute Invariance - Variational Auto-encoder. BLS: Broad Learning System. CCA: Canonical Correlation Analysis. DA: Discriminative Analysis.
CNN: Convolutional Neural Networks. ConvLSTM: Convolutional Long Short-Term Memory (LSTM). DBN: Deep Belief Network. DT: Decision Trees.
FCM: Fuzzy Cognitive Map. FCN: Fully Connected Network. GAN: Generative Adversarial Network. GCNN: Graph Convolutional Neural Network.
GLCM: Grey Level Co-occurrence Matrices. SVM: Support Vector Machines. GRU: Gated Recurrent Unit. HMM: Hidden Markov Models.
KNN: K-nearest neighbors algorithm. MLP: Multi-layer Perceptron. NB: Naive Bayes. PNN: Probabilistic Neural Network. RF: Random Forest.
RNN: Recurrent Neural Network. SAE: Sparse Auto-encoder. GELM: Graph Regularized Extreme Learning Machine.
‡
: [30], [41], [42], [50], [50], [102], [126], [127], [129], [131], [149], [187]–[191].
¶
: [44], [45], [53], [56], [58], [62], [64], [68], [70]–[72], [74], [76], [78], [79], [81], [89], [91]–[94]
¶
: [96], [99], [103], [104], [109], [110], [114], [115], [118], [120], [122], [124], [133]–[136], [138], [140], [144], [148], [192]–[195]
Ψ: [60], [65], [69], [73], [77], [88], [95], [108], [123], [132], [196]

growing trend in Fig. 1, were not as extensively analyzed as the

rest of the list because they only describe implementation details
of a specific component (e.g., Softmax) of the full system [11].
Additionally, in cases where two publications are addressing the
same implementation such as [12] and [13], only the first chrono-
logically available paper is described to avoid redundancy, but
their relation is preserved in Table III. In our analysis, we have
given preference to full end-to-end implementations. We first
evaluate these implementations from the wearability viewpoint.

A. Wearability Index
The high-level block diagram of a complete EEG-based
hardware emotion classifier is presented in Fig. 2. The violet
color shows the offline processing flow used to train the classi-
fier, whereas the purple color highlights the components most
amenable for a hardware implementation (FPGA or ASIC) as a
standalone online inference. Depending on the number of online
blocks, a wearability index is assigned to each implementation.
Fig. 2. High-level block diagram of a generic EEG-based hardware classifier.
Whereas this index is based on the level of hardware integra-
tion, it is being used as a wearability metric due to hardware
integration being an essential requirement for any standalone
wearable System-on-Chip (SoC) solution. A maximum index of

Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
416 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021

TABLE III
FIGURE OF MERIT FOR THE EEG-BASED CLASSIFIERS OF EMOTIONS IN HARDWARE

VAM: Valence/Arousal Model.  : Accuracy for 3 classes. LGE: Logic Gate Equivalent. | : Full CNN system. ∦ : Normalized to 180 nm.
HOC: Higher Order Crossings. SK: Skewness.  : Normalized energy per Class during inference defined in equation. Nclasses : Number of classes.

four is given in case the pre-processing, feature extraction (FE),
training, and inference blocks are all supported in hardware.
A colored symbol representing each of the hardware imple-
mentations reported in the prior art is used to highlight the
supported blocks. As per the survey selection criterion, all the
hardware classifiers have in common the online inference block.
The articles of [14] (ISCAS’19), [15] (CICC’20), [16] (IS-
CAS_b’20) and [17] (TBioCAS’20) are the only ones where the
pre-processing and feature extraction blocks are implemented
in hardware, whereas the remaining two studies outsource these
components to the offline processing. This is clearly an ad-
vantage for [14]–[16], and [17], which brings them closer to
a standalone, independent solution with an index of 3 out of
4. Although [11] (JETCAS’19) is missing an on-board feature
extraction component, resulting in a wearability index of 2 out of
4, it is the only implementation that provides on-board training,
which is essential for enabling on-the-fly tuning of the classifier.
The implementation reported in [12] (ISCAS_a’20) is focused
entirely on the proof of concept of a low-footprint classifier to
the exclusion of other features, which resulted in an index of 1
out of 4. All the surveyed hardware platforms share a common
supervised training approach for the offline processing blocks.
B. Pre-Processing
The signals recorded via the EEG electrodes are susceptible
to external interference, which requires the implementation of
filtering techniques to remove artefacts. EEG-based classifiers
that do not remove such artefacts are introducing biases into
their classifications and should therefore implement methods to
compensate for such biases.
The Pre-processing blocks in Fig. 2, whether operating on-
board or offline, represent the signal processing used in the hard-
ware approaches to condition the EEG data. Some datasets [18]
provide EEG data where certain artefacts have been removed.
While this may ease offline learning and verification, it is the
actual hardware implementation of the pre-processing block
that provides the more complete and realistic assessment of the
hardware resources required to achieve a fully wearable solution
for EEG-based emotion classification.
As indicated in Fig. 2, the HW platforms of [11] (JETCAS’19)
and [12] (ISCAS_a’20) do not include on-board pre-processing.
Both platforms rely on off-chip components to condition the
EEG data, which unfortunately prevents a detailed assessment of
the complete HW resources required to achieve a full HW solu-
tion. Additionally, [11] (JETCAS’19) uses a rather simple band-
pass filter to remove those artefacts whose spectra are within the
5 to 45 Hz frequency range, but such filter does not remove all
artefacts. As summarized in Fig. 3, the spectrum of the EEG
signal spans the 0 to 100 Hz range and overlaps with the spectra
of non-physiological [19] and physiological [20]–[22] artefacts
that are mingled with the EEG signal. The non-physiological
category of artefacts is composed by Electrode pop (Ep), ca-
ble movement (Cable), incorrect placement of the reference
electrode, and electrical interference (AC). The physiological
category of artefacts corresponds to myogenic (EMG) signals

Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
Fig. 3. EEG artefacts and their frequency bands.
(e.g., jaw clenching, head, neck and facial muscle contraction),
ocular (EOG) activity (e.g., eye lateral movement and blinking),
cardiac (ECG) perturbations (e.g., Heartbeat and electrical arte-
facts), perspiration (P), and respiration (Rs) [23]–[26]. These
contaminating factors coexist with the emotion-related activity
in the same frequency bands [18], which renders the reliance
on a single band-pass filter for EEG conditioning quite limit-
ing [27]–[29].
On the other hand, the approach in [12] (ISCAS_a’20)
provides a more complex conditioning strategy, involving a
Kurtosis-based Independent Component Analysis (ICA), which
facilitates the extraction of artefacts coexisting in the same
frequency bands as EEG. However, the technique is only applied
off-line, and its feasibility in a hardware implementation is
still unproven due to a lack of automation. Indeed, the off-line
procedure involves a manual inspection step of the indepen-
dent components (IC), in which the IC frequency content, the
stimulated regions in the IC scalp plots, and the Event Related
Potential (ERP) images are simultaneously checked for random-
ness across trials. These blind source separation methods are
commonly used by the software approaches to the EEG-based
emotion detection problem to achieve appropriate EEG signal
conditioning [30]–[34], but due to the direct dependency of the
number of obtained independent components on the number
of signal sources (electrodes), they are not very effective with
a low number of sources. It is clear that the less sources the
pre-processing block has to handle, the less complex is the HW
implementation. To achieve a holistic and robust HW solution,
low-cost alternatives [35], [36] for incorporating the ICA engine
need to be explored, possibly using matched filters for the
real-time identification of “artefact-rich” ICs in combination
with threshold-crossing checks on filter performance as function
of the number of electrodes.
Among the publications proposing an on-board pre-
processing block of the EEG signals, three [14] (ISCAS’19) [15]
(CICC’20) [16] (ISCAS_b’20) employ straightforward band-
pass filtering, which as previously mentioned does not account
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
417
Fig. 4. Robustness of the classifier against Eye blinking ([15], Fig. 9).
for complex interference scenarios involving the realistic arte-
facts described in Fig. 3. In [16], a band-pass filter between 8
and 45 Hz is used, which removes some of the low frequency
artefacts in Fig. 3 but at the cost of attenuating EEG activity
in the θ and δ brainwaves. On the other hand, [14] (ISCAS’19)
and [15] (CICC’20) use a low cut-off frequency of 2 Hz that lets
slow eye-movement artefacts to coexist with the EEG record-
ings. Interestingly, the implementation of [15] (CICC’20), as
illustrated in Fig. 4, seems to drastically reduce the impact of
eye blinking episodes on the EEG classifier. A further analysis
of the confusion matrix of [15] (CICC’20) could provide even
more insights into the nature of misclassified artefacts and the
conditioning methods needed to reduce misclassification errors.
The EEG pre-processing approach of [17] (TBioCAS’20),
seems to have the highest potential among all the surveyed
contributions toward a wearable EEG-based emotion classifica-
tion. Their pre-processing block comprises three steps, involving
down-sampling, band-pass filtering, and a Mean Average Ref-
erenced (MAR) filter block. The down-sampling is by factor
of 2 and has no effect on the frequency content except for
halving the power spectrum amplitude. The band-pass filter acts
on the range of 2 Hz to 60 Hz. As for MAR, it is based on
the idea of re-referencing the electrode signals with respect to
their overall average instead of the reference electrode in order
to reduce signal dependency on its position [21]. The method is
implemented by subtracting the average over all electrodes from
each signal. Other linear combinations of electrode signals may
also be used as electrode reference proxies [37]. This method
results in the elimination of the artefacts at the physical reference
electrode while averaging may help out in reducing the noise that
is re-injected into the EEG signals through the MAR operation.
A robust approach to pre-processing the EEG data is a pre-
requisite for successful EEG-based emotion classification. A
reliable approach should incorporate robust detrending, robust
re-referencing, outlier detection, interpolation, step detection
and correction, and ringing artefact removal [38]. Our survey
indicates that no single HW approach has fully addressed this
important task so far. If we are to provide a pre-processing index
the same way we have provided a wearability index, then the
418 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
Fig. 5. Main feature categories and their number of occurrences in the sur-
veyed publications. [9]–[17], [30], [31], [33], [39]–[149].
indices would be distributed as follows on a scale of 1 to 3. The
maximum score of 3 would be given to [17] (TBioCAS’20), due
to the fairly complex on-chip pre-processing realizations. The
approach in [12] (ISCAS_a’20), which was already penalized
in the wearability index, would receive a score of 2 due to the
complexity involved in using on-board blind source separation
methods. The approaches in [14] (ISCAS’19), [15] (CICC’20),
and [16] (ISCAS_b’20) would all get a score of 1 due to the
restrictions inherent in band-pass filtering.
C. Feature Extraction
EEG feature extraction (EFE) refers to the methods employed
to highlight latent and relevant information in the pre-processed
EEG data in order to facilitate the classification process. The
performance of the classifier is highly dependent on the EFE
stage, and hence every HW platform must incorporate an EFE
block to realistically assess its impact on the overall performance
of the EEG emotion classifier. Both EFE and the classifier
are subject to the quantization and round-off errors that are
inherent to the hardware. The impact of such errors on overall
performance cannot be realistically evaluated without the inte-
gration of the EFE block into the HW platform. Additionally,
the implementation of any EFE block entails an area, power,
and complexity cost whose evaluation is tied up to an actual
HW implementation. As indicated in Fig. 5, feature extraction
in EEG-based emotion classification problem is divided into five
categories depending on the kind of transformation applied to
the EEG data to extract its features. These categories will be
explained next.
1) Frequency-Based Features: As shown in Fig. 5, the
frequency-based feature category is the most popular in both
SW and HW platforms. This is not a surprising finding as several
previous publications provided enough evidence of the correla-
tion between the spectral components and brain emotions [150],
[151]. In fact, recent studies inspired by the biological integra-
tion across populations of neurons in the human cortex [152],
[153], have shown the important role of correlation indices
in describing coordinated activity between different frequency
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
regions of the EEG waveforms. Such indices can be used as
features for EEG-based emotion detection [154], [155]. The
frequency-based feature extraction methods compute the spec-
trum of the EEG signals or the subject-specific cross frequency
coupling (CFC) indices, and present them to the classifier for
the generation of labels.
The methods used by the software approaches of Table I
differ in terms of their complexity but are operated under the
premise of unlimited resources. Generally, their complexity
order is high, which makes most of them unsuitable for a direct
hardware implementation. On the other hand, methods such as
Power Spectral Distribution (PSD), Fourier Transform, Discrete
Cosine Transform (DCT), Singular Spectrum Analysis (SSA),
and combinations of filter banks of different spatial patterns,
are achievable in hardware due to the maturity of the hardware
designs in these traditional signal processing blocks (e.g., FFT
accelerators with low footprint [156]). In contrast, methods such
as Higher Order Spectral (HOS) Analysis, Linear-Frequency
Cepstral Coefficients (LFCC) and Mel-Frequency Cepstral Co-
efficients (MFCC), the Burg method, and Auto-Regressive Re-
flection Coefficients (ARRC), might require more resources and
design effort to incorporate them into a hardware solution.
Alternatively, the CFC features, which exploit synchronized
changes amongst different frequency bands [154], represent
an important alternative for future hardware platforms due to
their robustness and to their potential to reduce the EEG data
rates and memory footprint. The CFC features are embodied
in a connectivity graph [155] or index, and are therefore a
compact EEG representation that will reduce the need to store or
process raw EEG data in the classifier. However, the complexity
of the CFC algorithm, which requires a Coordinate Rotation
Digital Computer (CORDIC) processor to compute the Hilbert
transforms in the various frequency bands [157], presents a sig-
nificant challenge to ultra low-power hardware design. The same
is the case for the connectivity estimators and subject-based
correlation tuples obtained through surrogate analysis.
Four of the publications in Fig. 2, extract their frequency-
based features using PSD [17] (TBioCAS’20), [12] (IS-
CAS_a’20), [14] (ISCAS’19), [11] (JETCAS’19). PSD is also
the most popular method among the SW emotion classification
field as illustrated in Table I. However, the exact method em-
ployed to compute the PSD spectrum distribution differs among
these four HW papers. For instance, the implementation of [12]
(ISCAS_a’20) uses a Fast Fourier Transform (FFT) with 128
frequency bins, which provides a resolution that is suitable for
capturing fluctuations within and among brainwave frequency
bands.
As for the work reported in [17] (TBioCAS’20) and [14]
(ISCAS’19), it uses an approximation of the power spectrum
with only one power value per channel calculated with the
average activity in the β band (12 to 30 Hz). This averaging
approach is definitely more hardware friendly, as it removes
the need for an FFT accelerator. However, the consideration of
only one frequency band and its further reduction to a single
average value are bound to result in information losses within
the band itself. Fig. 6, which is taken from [11], presents an
average of the scalp plot activation per frequency band in each
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
Fig. 6. Scalp plots with frequency activity for different emotion labels in the
DEAP dataset [18] ([11], Fig. 3).
of the classes generated by combining the Low (L) and High
(H) values of the Arousal (A) and Valence (V), respectively.
Although the figure does not show the fluctuations within the
β band, it does highlight the additional brain activity that takes
place in the other frequency bands that were excluded in [17]
(TBioCAS’20) and [14] (ISCAS’19).
2) Time-Based Features: In contrast to the frequency-based
features, the time-based features are generated directly from
the EEG time series. Among the SW-based approaches of
Table I, statistical features are the most used in this category
(24 papers). These features include Kurtosis, Skewness, mean,
maxima, and slope variations, among many others. The second
most used method is Higher Order Crossing (HOC), whose
general principle is to keep track of zero crossings and their
successive differences. Higuchi’s Fractal Dimension (HFD) is
also among the popular methods, and its non-linear measures of
the EEG signal complexity make it suitable for feature extraction
under various emotional episodes. Other EFE methods based on
time series include Hjorth’s (activity, mobility, and complexity),
Grassberger-Procaccia (GP), Hybrid Adaptive Filtering (HAF),
Hurst exponents, Local Binary Patterns (LBP), and the event-
driven Late Positive potential (LPP). There are other time-series
extraction methods that compute features based on predictive
approaches such as Neural Time Series Prediction Processing
(NTSPP), and the Wasserstein metric of distances between prob-
ability distributions. In general, the time series methods require
less hardware resources than the frequency-domain methods
and are therefore more desirable. Naturally, methods involving
non-linear transformations such as HFD or Hjorth result in an
increase in HW design complexity.
Only one of the hardware-focused publications in Fig. 2 uses
time-based methods for feature extraction [15] (CICC’20). Its
approach is to extract the Higher Order Crossings and Skewness
(SK) information from the time-series to compose the feature
vector. The Skewness, which is used as an amplitude regularity
metric, is traditionally defined by
N of Hjorth parameters, which seems to be more advantageous
 (Xk − μ)2
Skewness =
(N − 1) ∗ σ 3
k=0
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
419
Fig. 7. (a) Accuracy plot for time-based methods. (b) Area and power of the
time-based methods. The legend in Figure (a) is for both (a) and (b). ([17], Table
I).
where Xk denotes the k-th sample the time series, N the number
of samples, μ the mean, and σ the standard deviation.
In order to achieve a hardware friendly implementation, [15]
proposes an approximate SK indicator (ASKI), whose definition
is given as
ASKI = μ3 ∗ k (2)
and whose overall trend is bounded by the true SK curve. The
design decision of using ASKI leads to a reduction in Gate Count
by a factor of 86.
As for the zero crossing metric (HOC), it corresponds to
a simple comparator-based implementation in which a given
sample is compared with its previous value that is monitored
through a 3-bit SR flip-flop. The identification of a sign change
from one sample value to the next triggers a simple counter to
accumulate the zero crossings. The two metrics are normalized
and passed to a dense neural network, as will be discussed
Section III.
As no other hardware design in Fig. 2 uses time-based
methods for feature extraction, the reader is referred to Fig. 7,
prepared using the feature exploration performed in [17] (TBio-
CAS’20), for a summary on the hardware implications of some
of these methods. In particular, Fig. 7(b) provides an insight
as to the area and power required for each of the considered
time-based methods when synthesized in 0.18 μm technology.
Fig. 7(a) presents the accuracy achieved in each of them.
As shown in Fig. 7(b), the method with the best accuracy is
the Common Spatial Pattern (CSP), which is a mathematical
algorithm to separate a multivariate signal into its individual
sub-components. The separation takes place by maximizing
the differences in variance between two windows and requires
the generation of co-variance matrices, which are not hard-
ware friendly. This explains the fact that this method has the
highest power and area consumption. The method with the
second worst power and area combination is the calculation
(1) to the Valence dimension. Hjorth parameters are indicators of
statistical properties used in the analysis of EEG signals [158].
420 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
Their hardware computation is also not trivial, and involves a
variance computation for the Activity parameter, the square root
of a ratio for the Mobility parameter, and a division together with
a first derivative in the Complexity parameter.
The next time-based method with a relatively acceptable accu-
racy for both emotion dimensions is the Sample Entropy (SE).
Entropy-based metrics are also among the top ten algorithms
used to extract features in the software approaches summarized
by Table I. SE is a measure of the complexity in a time se-
ries [159], and is defined as the logarithm of a ratio, which makes
its implementation relatively simple in hardware. In reference
again to Fig. 7(b), the method with the smallest HW resource
utilization and power consumption is the statistical mean with
a hardware accuracy of about 50%, 7(a). A similar accuracy
is achieved for the Root Mean Square (RMS), which has a
higher area and power consumption than the Mean. The standard
deviation (SD) has a similar area utilization as the RMS method,
although with a relatively higher power consumption, but its
accuracy advantages are relatively better. Finally, the method
with the worst accuracy metrics in Fig. 7(a) is the Variance (Var)
method, which also requires a higher area utilization and power
consumption than the Mean, RMS, or SD.
3) Time-Frequency Features: Time-frequency features com-
bine both frequency-domain and time-series features, which
enables the capture of the time dependence of spectral fluc-
tuations. Methods describing local transient characteristics in
both frequency and time domain are suitable to deal with the
non-stationary properties of the EEG signals.
Among the software methods summarized by Table I, the main
methods for extracting time-frequency features are Wavelet
transforms (WT) and their variations (e.g., Discrete Wavelet
Transform, Continuous Wavelet Transform, and Adaptive Tun-
able Q Wavelet Transform), Short-Time Fourier Transform
(STFT), and Hilbert-Huang Transform (HHT) with its stan-
dalone Empirical Mode Decomposition (EMD). In contrast to
the Fourier and Wavelet transforms, HHT does not involve the
selection of a mother wavelet or a basis function. It rather uses
EMD to decompose the signal using a data-driven approach that
sequentially extracts quasi-harmonics from a signal, starting
from the highest frequencies. The HHT method leads to a
robust analysis of non-stationary and non-linear time series as
is the case in EEG signals. However, the extraction of the HHT
Intrinsic Mode Functions (IMF) uses iterative computations that
cannot be performed in parallel [160], which makes its real-time
hardware implementation non-trivial.
In their basic form, the WT and STFT approaches are two pos-
sible alternatives for hardware implementations. The Wavelet
Transform applies a filter bank to the full signal at once,
whereas the STFT divides it into multiple smaller windows for
individual FFT applications. Both approaches lead to similar
results, although WT is traditionally well-known for dealing
with non-periodic and fast transient features more efficiently.
On the other hand, STFT is known to be more applicable in
real-time processing, due to its relatively short processing time
windows [161]. Additionally, the advances in FFT hardware
acceleration [162], and the numerous techniques to compress
the FFT accelerator footprint [156], [163]–[167], make them
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
Fig. 8. Spectrograms produced using the STFT ([11], Fig. 5).
particularly attractive for hardware solutions to the EEG-based
emotion detection problem.
Five hardware designs of EEG-based emotion classifiers of
the list shown in Table III use STFT for time-frequency feature
extraction. [9]–[11], [16], [117]. These designs come from the
same research group, and in terms of feature extraction, they
all share the same Spectrogram approach described in [11]
(JETCAS’19). The STFT is applied to the signals collected
from six electrodes, which are then mixed based on asymmetry
indices, which will be described under the next EFE category
on “Spatial Features.” The resulting combinations are used to
generate the final EEG frames for the classifier. Fig. 8 illustrates
the spectrograms produced by six EEG channels and the com-
bination process used to create the final EEG frame.
4) Spatial Features: Spatial features are meant to capture
coordinated neural activity between different brain regions. They
are not standalone features as their computation relies on the
availability of time-based, frequency-domain, or time-frequency
EEG data. In fact, one of the spatial-features methods, Common
Spatial Pattern, was already presented in the time-based features,
as part of Fig. 7.
Among the spatial-features methods used by the solutions in
Table I, Asymmetry Exploitation is one of the most popular
techniques in the emotion detection context due to its very low
overhead. It is based on the computation of Asymmetry Indices
which are coefficients relating opposite sides of the brain and
highlighting asymmetrical and coordinated neural activity [114].
These coefficients are always calculated in pairs of electrodes
and can be used with frequency, time, or time-frequency features.
The hardware designs in [14] (ISCAS’19) and [17] (TBio-
CAS’20) apply asymmetry indices on top of the channel-based
PSD averages already described under the frequency-features
category. The two approaches, which use 8 channels, compute
two indices from the PSD averages in their asymmetrical pairs of
electrodes. The first index is called the Inter Hemispheric Power
Spectral Ratio (IHP R) and is expressed as
P SDLef t
IHP R = (3)
P SDright
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
whereas the second one is called Inter Hemispheric Power
Asymmetry (IHP A), and it is given by
IHP A = |P SDLef t − P SDright |
The asymmetry index approaches of [14] (ISCAS’19)
and [17] (TBioCAS’20) are from the same research group and
use the same indices as in Eq. (3) and (4), but are implemented
differently. The design in [14] (ISCAS’19) is based on a scaled
version of IHPR with the required hardware division imple-
mented using a customized LookUp Table (LUT) approach,
which is more advantageous than using standard IP cores. On the
other hand, the approach in [17] (TBioCAS’20) slightly changes
the IHPR index into the following logarithmic expression
LIHP R = log2 {P SDLef t } − log2 {P SDRight }
whose hardware implementation is more convenient. Indeed,
The LIHP R index reduces the gate counts and power consump-
tion by a factor of 4.7 and 1.5, respectively, when compared with
IHP R. These indices then compose the feature vector passed
to the Support Vector Machine classifier (see Section III).
As already mentioned, the five hardware designs applying
the STFT rely extensively on asymmetry properties [9]–[11],
[16], [117]. As shown by Fig. 8, their reconstruction of the
EEG frames is based on combining asymmetrical pairs from the
six-electrode configuration using the Differential Asymmetry
expression
DASM = Spectrogramlef t − Spectrogramright
The six Spectrograms are reduced to 3 DASM frames, which
are then combined using
Reconstruction = W1 ∗ DASMF3 −F4 +
W2 ∗ DASMFp1 −Fp2 + W3 ∗ DASMF7 −F8
where W1 , W2 , and W3 are pre-defined weights. The Recon-
struction in Eq. (7) corresponds to the EEG frame of Fig. 8 and is
passed as input to the classifier with the asymmetry information
already embedded in its composition.
Besides CSP and asymmetry indices, another set of spatial-
based features that have been exclusively used by the software-
based EEG-based emotion detection systems is the Connectiv-
ity features, and in particular the Directed Transfer Function
(DTF). DTF features are causality measures that are used in
the determination of brain connectivity patterns [168], which
characterize specific brain states. The DTF is used to describe
casual influence of one EEG channel over another at a particular
frequency. Although a hardware implementation of DTF is
possible, it is not as simple as that of asymmetry indices, as
it entails the computation of the transfer matrix of a multivariate
auto-regressive model, a square root, and a division.
5) Raw Signals: Lastly, the Raw Signals category of EEG
features corresponds to a new paradigm that has recently
emerged as a result of using Deep Neural Network (DNN)
classifiers. Under this paradigm, the classifier is directly applied
to the raw EEG signal with the underlying assumption being
that the classifier is strong enough to capture the variations
embedded in the different emotional classes. There are currently
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
421
no hardware approaches using this new paradigm, as it normally
requires complex DNN’s to compensate for the absence of
specific features [127], [131]. As will be discussed in Section III,
(4)
the hardware solutions to the EEG-based emotion detection
problem tend to opt for shallow neural networks to achieve
low-footprint implementations. These networks are unlikely to
work directly on raw EEG data. Future research on new circuits
paradigms, such as near/supra-threshold techniques [169] or
adaptive body bias operation [170], [171] in Fully Depleted
Silicon on Insulator (FDSOI) [172] technologies (e.g., 22FDX)
may potentially enable the HW implementation of raw EEG
signals DNN’s into wearable classifiers.
A possibly more promising approach for the use of raw data
may be based on dedicating the initial layers of a deep CNN to
(5) feature extraction from raw EEG signals. In [126], coincidence
filtering along with Differential Entropy (DE) and PSD are used
to find appropriate kernels that will guide the automated feature
extraction from raw EEG data in the first layers of a CNN. Such
approach has the advantage of implicitly accounting for spectral
and entropy information while processing the raw EEG data for
learning and classification.
6) FE Score: As illustrated by Fig. 2, only four of the HW
publications include the EFE engine in their on-board implemen-
tation. The remaining two use a software interface to provide the
classifier with the features already extracted. Such combined
SW/HW platform serves well a a proof of concept but does
not allow a realistic assessment of the resources required in a
(6) fully wearable solution. No additional score is assigned in the
EFE case, as there is no preferred feature extraction method, the
major requirement being that its combination with the classifier
results in acceptable classification accuracy. Such combination
should also satisfy the HW design specifications on power, area,
(7) and latency. For these reasons, we only provide a binary EFE
index with a score of 1 assigned to the HW platfoms with
on-board feature extraction and a score of 0 assigned to those that
don’t have on-board. Other hardware-related aspects of feature
extraction performance will be addressed in the next section
devoted to hardware-friendly classification methods.
IV. HARDWARE-FRIENDLY CLASSIFICATION
The classification block is the most important component
of an EEG-based emotion detection system. It is in charge of
transforming the extracted features into useful emotion labels.
As presented in Fig. 2, the training of these inference engines
for the emotion detection problem is typically based on super-
vised learning, involves large volumes of data, and is therefore
performed offline. Supervised training require that input-output
pairs be passed to the classifier, which implies that the emotion
classes for a given EEG input need to be known ahead of
time, as it is the case for any other supervised learning task.
However, in the context of emotion detection, the output labels
are obtained through psycho-physical experiments in which
participants are asked to categorize their evoked response in
a graphical representation of the Valence-Arousal scales [173].
Whereas the graphical representation is quite helpful in lowering
the language barriers, there are still subjective and cultural
422
Fig. 9. SVM classifier in [17] highlighting its main difference with [14]. ([17],
Fig. 15).
factors that make it difficult to standardize the labeled datasets
across a given population in a given geography across time, or
across geographic locations and cultural domains. Further bias
may be introduced during the discretization of the continuous
scales, as multiple benchmarks exclude a neutral emotion class,
while the boundaries between low and high along the Arousal
or Valence axes can be quite fuzzy or even arbitrary.
Regarding the specific algorithms employed to identify emo-
tions using EEG, the two most popular machine learning models
on both SW and HW platforms are Support Vector Machines
(SVM) and Convolutional Neural Networks (CNNs) as illus-
trated in Table II. SVMs are shallow models with large margins
in their decision boundaries, which can perform non-linear
classification using kernel transformations with a relatively low
overhead. The Figure of Merit in Table III shows that the smallest
silicon area has been achieved in an SVM-based approach,
namely, 5.4 mm2 in [17]. Additionally, the mathematical sim-
plicity of the SVM classifier allows the compute operations to
take place with a very low clock frequency (e.g., 1 KHz in [14]
and [17]), which also leads to low power consumption. On the
other hand, Table III shows that the SVM-based approaches do
not lead to the highest classification accuracies. The hardware
designs of the classifiers in [14] and [17] are almost identical but
corresponds to different development stages with the only fun-
damental difference being the implementation of time division
multiplexing in [17] (TBioCAS’20) to reuse the SVM engine for
both the Arousal and Valence modes. This aspect is highlighted
with red color in Fig. 9. The HW design of [14] has two SVM
engines as shown in Fig. 9, one for Arousal and another for
Valence.
In contrast to the SVM algorithm, the Convolutional Neural
Network (CNN) is primarily used in a Deep Learning context
with multiple CNN layers, each one of which is responsible
for extracting a specific class of features. The extracted CNN
features are commonly arranged in a two-dimensional structure
treated as an image, which is then fed into the multi-layered
topology. In each layer, the receptive fields are designed to
respond to oriented patterns within the data such as edges, lines
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
or more complex compositions, which in the EEG context, cor-
respond to the emotion-related patterns whether in the frequency
or time domain. The SW approaches, particularly in the CNN
domain, tend to work under the premise of unlimited resources.
These SW approaches involve 3D stacking of convolutions ap-
plied to multi-dimensional EEG structures [30], [127] [131], the
utilization of a large number of layers [50], and a large number
(e.g., 32 in [51]) of electrodes to build the EEG frames. Despite
the high accuracy reported in some of these CNN models (e.g.,
99.72% for DEAP dataset in [50]), their excessive complexity
makes them unsuitable for a wearable hardware implementation.
In contrast to the CNN-based software platforms, the CNN
models used in hardware tend to implement shallower networks,
which can fit in an FPGA or ASIC without consuming too much
area. The CNN models in the surveyed HW platforms have a
total of 2 layers [12], [13], 3 layers [117], [11], or 4 [16] layers.
The first one corresponds to a shallow but wide CNN with 1
convolutional block (i.e., 1 convolutional layer, 1 pooling layer,
and 1 ReLU) of 100 kernels, and 1 dense layer having two output
neurons [12], [13]. The second platform uses 2 convolutional
blocks of 6 and 8 kernels, and 1 dense layer with 2 output
neurons [11], [117]. The third platforms uses 3 convolutional
blocks having 16, 32, and 32 kernels, and a dense layer of two
output neurons [16]. The sizes of the feature frames, which are
passed to the first layer in each network, are 22x22 for the
platforms using the STFT spectrogram [11], [117] [16], and
16x139 for the platform using the PSD+Welch frames [12],
[13]. The Welch method in the last two approaches is intended
to smooth discontinuities between consecutive frequency bins,
in order to apply the CNN engine, which operates under the
premise of spatial correlation in the input images.
The HW CNN platforms for EEG-based emotion classifica-
tion implement the HW inference engine using two different
objectives. The first objective is aligned with the traditional
paradigm of high bandwidth and low latency, which is inherited
from the highly demanding applications of video processing
and computer vision. The HW platforms in [11], [16], [117]
implement their convolutional neural network in line with that
paradigm, where a Multiply-Accumulate (MAC) array with 54
processing elements is allocated to compute all the kernels in
all layers with high parallelism. On the other hand, the HW
CNN platforms in [12], [13] are less conventional, and aim for a
serialized execution of the different layers in the CNN, in order to
reduce the logic resources required for the implementation. The
latter case is possible because the EEG-based emotion detection
context has lower bandwidth requirements than other computer
vision tasks.
Fig. 10 highlights the differences between these objectives.
Fig. 10(a) corresponds to the CNN design of [12], [13], in which
the convolutional layer is implemented via a pipelined hierar-
chichal tree that produces the rectified outputs. The rectified
outputs are then received by a special First-Input First-Output
(FIFO) Maxpool module, which operates on-the-fly to avoid
waiting until all these intermediate results are available, which
also reduces the heap and helps to maintain a high throughput.
This CNN design results in very low resource usage. In contrast,
the design of Fig. 10(b) [11], [16], [117] requires the buffering
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
Fig. 10. Comparison between the Convolutional layer in [12] (a) and [11] (b).
Figure [197] is used as starter template.
of the intermediate results in order to compute the maxpooling
layer. The dense layer in [12], [13] is also completely redesigned
to cope with the serialized maxpooling values, which results in
further reduction of logic resources. This is because computing
the serialized values in the dense layer translates into fewer
synapses in need of concurrent processing, thus relaxing the
number of required parallel multipliers.
Naturally, the latency of the design in Fig. 10(a) [12], [13]
is higher than that of the traditional CNN design in Fig. 10(b).
However, as shown in Table III, the latency in [12], [13] is still
within human interaction time, which is about 150 ms. Besides
the higher latency, another fundamental drawback in the low-
resource approaches in [12], [13] is the poor flexibility, and the
difficulty to scale the model up to deeper neural networks. The
utilization of a MAC array [11], [16], [117] is much more flexible
as it can be adapted to any deep multi-layer neural architecture.
The MAC array can also help to implement other functions on-
chip such as dot products and matrix multiplication, which are
quite useful in many machine learning models.
Another representative algorithm in EEG-based classification
of emotions is the Fully Connected Network (FCN), which is
ranked among the top ten most used algorithms in the soft-
ware domain, and provides a neural network alternative that
is simpler than CNN but more complex than SVM. An FCN
is also more amenable to hardware design due the absence of
a sliding receptive field. On the other hand, FCN hardware
implementation leads to a denser interconnect network due to
the fully connected topology, and with it, to a larger number
of weights, whose training and storage are required ahead of
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
423
Fig. 11. Dense network implemented in [15] with time multiplexing ([15],
Fig. 6).
inference. In the EEG-based context, an FCN has a trainable
weight between every input node and the extracted features.
The same applies to the connections between latent layers. It is
interesting that the HW implementation in [15] (CICC’20) uses a
shallow model of 4 layers, whose inference is highly optimized.
The model of Fig. 11 employs an Arithmetic Logic Unit (ALU)
that is reused by all layers. The control unit of the dense network
manages a time division multiplexing scheme across the 4 layers
so that all calculations are executed within the same ALU. Such
architecture significantly reduces the overall logic resources.
Section V is devoted to a detailed account on the evaluation
of the above HW implementations of classification algorithm
where traditional metrics such as area and power will be used.
In this section, the HW implementations are evaluated and
compared in terms of their classification accuracy. We use a
comparison scale from 1 to 6 with two caveats. The first is that
accuracy very much depends on the labeling quality if the EEG
data, and as was mentioned above, such quality varies from one
dataset to another. The second caveat is that score assignment is
not a value judgment but a comparative assessment as may be
used in a comparison table.
According to Table III, the highest accuracy (84%) is achieved
by the FCN model [15] (CICC’20), which is using a Va-
lence/Arousal combined model to predict four classes. The
design of [15] presents the best trade-off between complexity (4
layers) and area, especially that it is using only 2 channels. The
HW design with the second highest accuracy (83.36% only for
Valence) corresponds to the CNN model of [11] (JETCAS’19),
which is the most complex CNN topology among the all the
surveyed designs. The HW implementation of [11] is particu-
larly attractive because the back-propagation engine is designed
to support on-board training, which should enable the tuning
of the classifier after its deployment. In that regard, the design
of [11] is unique amongst the surveyed HW solutions.
The third highest score is given to the CNN model of [16]
(ISCAS_b’20), which claims an accuracy of 76.94% for three
classes. The account in [16] does not report on the traditional HW
metrics such as area, power, or latency, and the accuracy outcome
of the design should be considered on its own in comparison
with the other accuracy scores. The next score is also for the
CNN model of [12] (ISCAS_a’20) where the accuracy achieved
424
is 77.57% and 71.25% for Valence and Arousal, respectively.
The last two scores correspond to the SVM models, which as
described earlier, tend to trade-off high accuracy for low power
and small footprint. The SVM model of [17] (TBioCAS’20)
achieves an accuracy of 73.14% and 72.96% for Arousal and
Valence, respectively, whereas that of [14] (ISCAS’19) achieves
60.0% and 63.0%, respectively.
Besides the hardware-friendly classification algorithms de-
scribed above, there is a wide variety of other algorithms that
were used only on SW platforms and never mapped onto hard-
ware. These algorithms provide opportunities for novel HW
designs in line with the wearability objectives of EEG-based
emotion classification systems. Table II may be read as a list of
such opportunities with the most notorious being that of Spiking
Neural Networks (SNN) [186]. Known HW implementations of
SNN have excellent power and energy-efficiency metrics [198]
due to their short-pulse operation which mimics the on-demand
firing rates of the neurons, and enables the implementation
of biologically plausible models [199], [200] with high levels
of parallelism [201]. This on-demand behavior in SNNs have
also inspired other Artificial Neural Network approaches that
can be ported to the EEG time series, such as the event-based
GRU described in [202], which exploits these properties to
reduce power consumption while classifying Arrhythmia in
ECG signals. Other algorithms in Table II that are very much
worth exploring for HW design include the various recurrent
models such as the mentioned GRU [183], LSTMs [174], [175],
ConvLSTMs [40], [178], [179], and most of the recurrent neural
networks [128]. Such models are well adapted to the capture
of the spatio-temporal variations of the EEG signal during the
emotional episodes.
On the other hand and in contrast to the feed forward net-
works and their HW implementations, recurrent networks are not
amenable to parallelization due to their temporal dependencies,
which forces the HW engine to operate serially. Hardware paral-
lelism cannot therefore be used to improve throughput. Another
issue encountered in the HW implementations of the GRU
is the number of weights in the recurrent connections, which
grows quadratically with the GRU output dimension. In other
words, doubling the number of outputs results in quadrupling
the number of GRUs weights. A similar behavior is present in
LSTMs, with the latter having a more regular topology than
the GRUs [205]. As illustrated in Fig. 12(a), a HW design
of the LSTM architecture can be achieved by fusing a dot
product into one single operation where all the independent
multiplications (yellow) are executed in parallel. On other hand,
the multiplication operations in the GRU have dependencies as
illustrated by the yellow and red multipliers in Fig. 12(b). These
dependencies prevent their fusion into a single operation using
matrix multiplication in MAC arrays.
V. THE HARD FACTS OF HARDWARE IMPLEMENTATION AND
VERIFICATION
In this section, the various EEG-based hardware designs are
compared using the standard hardware metrics of power, area,
and latency. Two other important aspects are also considered
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
Fig. 12. (a) LSTM architecture. (b) GRU architecture. The figures from [203]
and [204] are adapted to highlight parallel multiplier stages.
in the comparison for the specific case of EEG-baed emotion
classifiers, namely, EEG electrode engineering, and verification
testbenches.
A. Power Consumption
One of the fundamental metrics to assess hardware designs
is power consumption. This is even more so in the context
of a wearable medical device such as the EEG-based emo-
tion classifiers where battery life and continuous operation are
of paramount importance. Unfortunately, not all the surveyed
hardware designs provide a power metric, which has hindered
their fair comparison. Another hindrance to fair comparison is
the diversity of platforms used as is clear from Table III. The
power figures of ASICs and FPGAs are not directly comparable
since the FPGA typically consumes more power than its ASIC
counterpart. In addition, the power figure of an FPGAs is highly
dependent on its evaluation board. In other words, even the same
FPGA designs may not be comparable.
These limitations notwithstanding, the power figures of the
surveyed ASIC designs have been compared across various tech-
nologies using a normalized energy per class metric computed
as follows:
P ower × ExecT ime
N ormalizedEnergy = (8)
Nclasses (VDD /Vref )2
where the normalized energy is measured in nJ per classifica-
tion. The number of classes, Nclasses , is an important variable
in (8), as not all surveyed designs support the full 4 classes in
the estimation of the Valence and Arousal. Also in (8), Vref =
1 V, which corresponds to the voltage used in the 0.18 μm
technologies of [15], [17]. Since [15] reports the energy per
classification as 10.13 μJ/classif ication, its design latency
of 48 s is used to estimate the average power and calculate the
normalized Energy per class as in Equation (8). An important
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
distinction between the N ormalizedEnergy metric in (8) and
the energy-per-classification metric provided by [15] is that the
former considers as Nclasses the total number of classes the
classification model supports whereas the latter accounts only
for the energy spent during one feed-forward inference. The
purpose of the metric in (8) is to provide a fair comparison
between the HW implementations of the classification models,
as “small” classifiers supporting only two classes will naturally
consume less power than a “large” classifier supporting the four
valence and arousal classes.
As shown in Table III, the best average power for the ASIC
designs is achieved by the 2-channels architecture in [15] (168
μW), followed by the SVM design in [17] (12.7 mW). The next
design is the complex CNN in [11], which consumes an average
of 29.5 mW during inference. The training power is also reported
in [11] but it is not included in the comparison, as this is the
only design with on-board training. These power figures are not
normalized, yet their power ratings are indicative of the energy
efficiency of the designs themselves. Indeed, the largest power
consumption is that of [11] which uses the smallest VDD and
the smallest technology (28 nm). Its normalization would only
widen the gap between the designs since the first two are already
in 0.18 μm and are using the same VDD . Note that the power
metric is only an indicator and has to be augmented with the
number of classes and the latency, which has a direct impact on
the energy consumption, and therefore battery life.
The differences in the normalized energy per class for the
ASIC designs are also presented in Table III. The design with
the smallest normalized energy per class is in fact the complex
CNN in [11]. This is because of its extremely small latency. The
following designs are the FCN of [15] and the SVM of [17].
In regard to the FPGA designs, the smallest average power
(12.7 mW) is that of the SVM in [14], which also has the smallest
normalized energy per class (2.54 μJ/class). The next design is
the low-resources CNN in [13], [12], which consumes 150 mW,
and achieves a normalized energy per class of 34.84 μJ/class.
The last design in this rating is the CNN of [16], which did not
report any power figures.
One last disclaimer regarding the power comparisons is that
not all designs in Table III have implemented all the stages of the
EEG-based emotion classification system, which as discussed
earlier, will impact the power figures. However, the best power
figures amongst the surveyed designs have been achieved by
those that have implemented the most complete systems [11],
[15] [17], [14].
B. Area
The previous paragraphs have already qualitatively described
some of the area-related optimizations performed by the hard-
ware accelerators of the EEG-based emotion classification sys-
tems. In this section, more quantitative assessments and com-
parisons are provided. The area figures have the same caveats as
the power figures. For one, it is not possible to directly compare
the areas of an ASIC and an FPGA design. One reasonable way
to make a comparison is to fetch the area of a single ASIC cell
(e.g., NOR gate) from the Library Exchange Format (LEF) files,
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
425
and use it to calculate the Logic Gate Equivalent (LGE), which
in turn can be used in conjunction with the FPGA utilization
reports to estimate the FPGA area in relation to that of the ASIC
one. However, such estimation only leads to approximate area
comparisons rather than exact ones, and in this particular case,
the cell information from all contributions is unknown, which
prevents that comparison.
The area comparisons amongst ASIC designs implemented
in different technologies have a similar caveat. However, it can
be addressed using normalized metrics. The normalized area is
defined as [206]:
Area
N ormalizedarea = (9)
(T echnology/180 nm)2
where the reference technology is 0.18 μm as it is the oldest
amongst the ones used for EEG-based ASIC emotion classifica-
tion platforms.
The smallest normalized area (5.4 mm2 ) among the ASIC
designs is achieved by the SVM of [17]. The second best design
is the FCN in [15] with 16 mm2 , followed by the complex CNN
in [11]. Regarding the FPGA designs, the only one reporting
utilization is the low-resource CNN in [12], [13]. Although its
figure cannot be compared with those of the ASIC designs, an
approximate comparison with the complex CNN [11] is partially
possible, as the latter reports an LGE metric. A direct inspection
of Table I shows the large area differences due to the logic cells
used. This is expected as the low-resource CNN is even smaller
than multiple state-of-the-art FPGA implementations of other
CNN models [12], [13]. However, it is important to note that
the complex CNN design of [11] provides a full system whereas
that of [12], [13] is focused on compressing the CNN classifier.
Additionally, the complex CNN [11] provides a more flexible
and complete solution, including on-board training support.
An additional observation regarding the top designs is that
all of them use time division multiplexing to reduce the area,
taking advantage of the relaxed timing constraints in EEG-based
emotion detection. Another observation regarding the second
best normalized area of [15] is that despite using only 2 channels,
the use of dense 4-layer network has resulted in an area that is
three times larger than the SVM design, which has achieved the
smallest area.
C. Latency
The latency is the time taken by the system to produce a
classification result once an EEG epoch is received. The EEG
classification latency is particularly important in the context
of a wearable EEG-based emotion classification device, where
the objective is to track, within timing constraints of human
interaction (150 ms), the patient’s emotional state. This is in stark
contrast with EEG-based seizure identification devices, in which
longer detection time windows are more effective because the
relevant episodes are sparsely distributed over time [207]. Emo-
tional experiences are short and highly dynamic. and patients
under emotional stimuli are unlikely to manifest a consistent
response over large time intervals [61]. Tracking such dynamic
emotional states require a short classification latency.
426 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021

Fig. 13. Training paradigms and the number of occurrences in the surveyed
state of the art [9]–[17], [30], [31], [33], [39]–[42], [44]–[51], [53]–[58], [60]–
[84], [88]–[106], [108]–[112], [114]–[119], [121]–[149], [187].

In terms of their classification latency, the surveyed designs

are ranked as follows. The first one is the Complex CNN [11],
[117] with 0.026 ms and a clock frequency of 70 MHz. This Fig. 14. Electrodes used in SW [4] (a) and HW (b) approaches. The 10-20
latency rating is expected as the design is based on a large system figure is adapted from [219].
MAC on-chip array. The next best design is the low-resource
CNN [12], [13], which, in spite of its complex logic reutilization
schemes, can provide classification times that are smaller than 1
ms with a clock frequency of 100 MHz. The following designs
use slower clock frequencies to trade-off low power with longer
latency. Thus, the SVM design [14], which is also implemented
in an FPGA, uses a clock frequency of 1KHz, and achieves a
latency of 800 ms. The next design is also an SVM model and
achieves a latency of 48 s with a 1KHz of clock frequency [17].
As discussed previously in relation to Fig. 9, the design in [17]
switches the SVM engine between the Arousal and Valence
mode, which leads to longer latency. Fig. 15. Experimental setup for [11] showing: (a) the 28-nm CNN chip;
The longest latency amongst the surveyed designs is that of (b) the Spartan 3E FPGA controller; and (c) the Bluetooth RF module (see
Fig. 15 therein).
FCN in [15], which achieves a latency of 60 seconds. Recall that
this design has ranked first for its average power consumption,
and second for its normalized energy efficiency per class. Such
ratings have been enabled through ALU and Analog Front
End (AFE in Fig. 17) re-utilization as shown in Fig. 11. In
Subsection V-E, more details will be given regarding the FCN
design, especially its time division multiplexing, the reduction
of the effective impedance connected to its electrodes, and its
re-use of the acquisition AFE. Naturally, with such loads along
the datapath, the trade-offs between area, power, and latency
are in favor of area and power over latency. Finally, the FPGA
design in [16] does not include any of the relevant utilization
metrics and is therefore rated last in this survey.

D. Training Paradigms
In contrast to the EEG-based seizure identification prob-
lem [207], the EEG-based emotion detection task, has the
potential of using large volumes of data to fully exploit the
capabilities of Deep Neural networks in generic classifiers. The
evoked experiences in the emotion detection problem are short
and can provide sufficient amount of data to train state of the
Fig. 16. Experimental setup for [17] (See Fig. 9 therein).
art machine learning algorithms such as Deep Neural Networks.
In this section, the hardware contributions are assessed in terms
of the training paradigm used as well as the number of training
paradigms supported.

Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
Fig. 17. Experimental setup for [15](see Fig. 1 therein).
As shown in Fig. 13, the preferred training paradigms in the
state of the art are the Subject Dependent (SD) approaches, in
which the data employed for training and testing comes from
the same participant. The second most used approach is the
Subject Independent (SI) paradigm, in which the training data
comes from all subjects, and is employed for testing any subject.
A newly emerging paradigm is the Subject Semi-independent
(SSI) training which corresponds to the use of transfer learning
techniques to pre-train the classifier with data from a group of
subjects and deploy it for inference after fine-tuning with new
data from the target session or subject. In contrast to the direct
concatenation of new data, the fine-tuning process is applied on a
pre-trained model with the learning taking place iteratively on a
relatively small amount of data. Despite using the same number
of samples as its data concatenation counterpart, the fine-tuned
generic classifier tends to show a superior performance for gen-
eral applications [208] as well as EEG-related tasks [209], [210].
Network initialization [210] and assisted fine-tuning in which
the model capacity is allowed to increase either by widening
existing layers, or deepening the pre-trained network, further
enhance the transfer learning of new tasks [208].
In the specific EEG-based emotion detection context, the
study in [42] uses the unsupervised transfer learning approach
of [211], which advocates training by similarity, i.e., using a
group of proxy subjects sharing certain similarities with the
target subject. The similarity metric is based on clustering tech-
niques and is used to certify the similarities amongst the subjects
on which the inference engine will be used. Alternative ap-
proaches have also shown improvements during the exploitation
of transfer learning without similarity metrics [30]. Regardless
of the method used, the overall SSI goal is to augment the training
data and learn from it efficiently. Studies with similar objectives
make use of Generative Adversarial Networks (GAN) [182]
to increase the number of samples for the same subject via
the artificial generation of data. In the EEG context, GAN is
hampered by the fact that the problem of EEG signal synthesis
is not entirely settled and that it is still very challenging to assess
the quality of synthetic EEG data.
Another learning paradigm that is also geared toward im-
proving model performance as old data is updated or new data
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
427
becomes available is task-progressive [212], continual [213],
or lifelong learning [214]. Some of these incremental learn-
ing paradigms have already been explored for emotion detec-
tion [215] but their translation into hardware platforms is yet
to be explored. Their usefulness in the context of wearable
EEG-based emotion detection is two-fold:
1) Not only do they improve patient specificity throughout
the usage of the wearable device but also they help adapt
to the evolution of the disease in a given patient. This
is especially helpful in degenerative diseases such as
Alzheimer’s.
2) The tuning and growth paradigms preempt a major short-
coming in fixed-architecture neural networks, namely,
catastrophic forgetting [216], [217]. At the very least,
wearables based on machine learning paradigms should
incorporate, in their qualification for clinical usage, crite-
ria for predicting the onset of catastrophic forgetting [218]
as more model tuning is introduced or more tasks are added
to the model’s prediction portfolio.
Returning back to the surveyed hardware designs, 8 of them
have used the traditional subject dependent (SD) approach, 2
have attempted to train entirely subject-independent (SI) clas-
sifiers, and only one has explored the subject semi-independent
(SSI) paradigm. In order to provide a quantitative metric for this
part of the survey, the number of training paradigms listed in
Table III is used as training paradigm (TRP) score. The HW
design of [17] achieves a score of 2 because it considers both
SI and SD. A similar score is assigned to [12], [13] due to their
utilization of both SSI [42] and SD. The remaining designs have
been assigned a score of 1, as they only consider one training
paradigm.
E. Electrode Engineering
One important design consideration for wearable EEG-based
devices is the number of input ports connecting the process-
ing unit to the EEG sensing unit. In order to reduce on-chip
processing and connectivity overhead, it has been typical for
the EEG-based emotion classification devices to focus only on
the most representative EEG channels. In this subsection, the
number of EEG channels and their source electrodes as used
in the HW designs are surveyed. A related issue that is also
addressed in this subsection is the preferred placement of the
electrodes in the 10–20 system.
In line with the EEG electrode coding conventions described
in [4], a color code indicating the usage of a particular electrode
has been adopted. Fig. 14(a) is based on the SW emotion
classification survey of [4] and indicates the percentage of SW
classifiers that have used a given electrode in the scalp placement
pattern. Similarly, in this HW survey, Fig. 14(b) is given to
convey similar information about the electrodes of the HW
designs. For instance, a green electrode indicates that it has
been used by less than 25% of the designs; a yellow electrode
by 25% to 50%; an orange electrode color by 50 to 75%; and
a red electrode color by more than 75%. In contrast to the
electrode usage by the SW classifiers 14(a) [4], those of the HW
classifiers are concentrated mainly in the frontal and temporal
428 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
lobes with some minor utilization overlap in the posterior and
occipital regions. Interestingly, the highest hardware accuracies
for the DEAP dataset in Table III, were reported by the 2-channel
implementation of [15] (CICC’20), in which the two electrodes
used were not in the frontal lobe. Rather they were placed at the
junction between the posterior and temporal regions (TP7 and
TP8).
Aside from algorithmic differences between the various hard-
ware designs, Table III shows that including a larger number of
electrodes does not necessarily result in an accuracy improve-
ment. As mentioned above, the highest accuracies were achieved
by designs using 2 [15] (CICC’20) and 6 electrodes [14] (IS-
CAS’19) [17] (TBioCAS’20). This aspect of electrode count and
placement is well-documented in the EEG-based SW emotion
classifiers [99] and has immediate relevance to the HW designs.
Such designs can reduce processing and input connectivity over-
head without sacrificing accuracy by using a minimal number
of electrodes.
The utilization of a high number of electrodes has far-reaching
impact on hardware implementations that is translated into a
significant increase in the hardware metrics of area, power,
and design complexity. Since these metrics will be accounted
as a matter of course when evaluating the HW designs, no
particular score is assigned to the electrode engineering aspects
of the EEG-based emotion classification systems. The main
recommendation is to carefully select the number and placement
of EEG channels channel as they impact not only hardware itself
but also the accuracy of the classification system. One possible
venue of further development in EEG-based HW classifiers of
emotions is the incorporation of newer, more wearable technolo-
gies for EEG signal acquisition [220]. Such technologies are
particularly relevant to patients suffering from emotion lock-in
disorders that are accompanied by difficulties in maintaining
stable head postures.
F. Verification Methodology
In addition to the algorithmic and hardware criteria used to
assess the EEG-based hardware designs of Fig. 2, this subsection
presents an overview of the experimental settings that have been
employed for testing and verification. Setting up the testing and
verification environment for the EEG-based hardware emotion
classifiers is indeed a challenging task that sets the HW approach
apart from SW approach.
Fig. 15 shows the experimental setup used in [11], which is
the one design involving significant ports and interfacing effort.
As shown in Fig. 15, a front-end circuit and a micro-controller
are attached to the back of the 16-channel EEG cap to acquire
and send the data via a Bluetooth module. The Bluetooth pack-
ets are received by a laptop, in which the feature extraction
engine is implemented. The laptop then forwards the data to
the Printed Circuit Board (PCB) hosting the 28-nm CNN chip.
The PCB also includes a bluetooth module, a data conversion
stage implemented in an embedded Spartan-3E FPGA, and has
the possibility to enable the classification either in an on-board
Virtex-7 FPGA or the 28-nm CNN engine. The classification
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
loop is then closed by sending back the result via Bluetooth into
the Laptop, where a graphical user interface (GUI) displays the
result. This complete setup with such a complex communication
environment is currently the state of the art in EEG-based
hardware for emotion classification, but is still few steps away
from a low-footprint, wearable device that tightly couples the
pre-processing front-end to the EEG sensors on the one hand
and to the classifier on the other. However, it is clear that this
was not the original intent of the design. The aim was rather
to achieve maximum reconfiguration flexibility to test multiple
algorithms and explore various design options.
The second experimental setup is from the same research
group [16] (ISCAS_b’20), involving an FPGA platform with a
less complex but more compact implementation than the ASIC
CNN classifier. The platform operates based on multi-modal
sensors such as EEG, electrocardiogram (ECG) and photo-
plethysmogram (PPG), with the various elements centralized
around two evaluation boards closely connected via a serial
periphery interface (SPI). The primary board hosts a RISC-V
core synthesized in a Kintex-7 FPGA, whereas the secondary
board hosts the classifier implemented in a Spartan-6 FPGA. The
sensory data is sent via Bluetooth from each of the three sensors,
and it is acquired on the primary board, where the pre-processing
is performed in the RISC-V core. The pre-processed signals
are then transmitted via SPI to the secondary board, where the
classification takes place. Lastly, the output is transmitted via
Bluetooth to an external laptop, which has a GUI to display the
classification result. The setup provides an interesting proof of
concept of a flexible combination of an embedded core acting
as feature extractor with a machine learning (ML) accelerator.
In contrast to the previously described hardware designs, the
design in [12] (ISCAS_a’20), which also employs an FPGA
as the implementation platform, uses a simpler experimental
setup that is dedicated to a proof-of-concept for a low-footprint
CNN accelerator. The design of [12] does not address the issue
of bench-marking a holistic HW system to classify emotions.
It uses a laptop for the data acquisition, pre-processing, and
feature extraction. Upon the extraction of the frequency-based
features, the EEG frequency frames are transmitted to the FPGA
using a universal asynchronous receiver-transmitter (UART)
interface. The FPGA hosting the CNN accelerator generates the
classification labels.
Toward achieving full wearability, the designs of [14], [17],
present a System-on-Chip (SoC) involving all the stages required
for the EEG-based emotion detection. Their system acquires
the EEG data directly from the electrodes, which entails a
larger design effort due to the inclusion of a customized Analog
Front-End (AFE). The designs in [14] (ISCAS’19) and [17]
(TBioCAS’20) employ one AFE block for each of the 8 EEG
channels. The AFE blocks in both designs include a low-noise,
capacitively coupled instrumentation amplifier (CCIA) and a
programmable gain amplifier (PGA). The fundamental AFE dif-
ference between the two designs lies in the number of Analog-to-
Digital Converters (ADC) used. The design in [14] (ISCAS’19)
uses one ADC per channel, whereas that of [17] (TBioCAS’20)
shares one successive-approximation (SAR) ADC between all
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
TABLE IV
SUMMARY OF SCORES ACCORDING TO THE PROPOSED ASSESSMENT METRICS
channels, as shown in Fig. 16. After the digital data is ac-
quired, both approaches use time division multiplexing for pre-
processing and feature extraction to reduce on-chip resources.
The features are then passed to the SVM classifier with both
designs using the same SVM algorithm. Lastly, the output of
the classifier is sent to an external Bluetooth master, where the
accuracy assessments take place.
In addition to the previous two SoC designs, another SoC
design with an FCN classifier and including all the processing
stages on-chip, is presented in [15]. It is the one with the
smallest footprint and most efficient utilization of a compact
two-electrode sensing interface. As such, it is representative
of the state-of-the-art in EEG-based HW emotion classifiers.
Their customized AFE uses a Capacitively Coupled Low Noise
Amplifier (C2LNA) followed by the 2nd stage of a Continuous-
time, Digitally-Assisted Capacitively Coupled Instrumentation
Amplifier (CTDAC2IA). The output of the IA is passed to a
low-pass filter with a cut-off frequency of 2KHz, then digitized
using an SAR ADC with 10-bit resolution. One design concern
of the two-electrode interface is to reduce the on-resistance of
the front end. This is done through a carefully designed parallel
configuration multiplexer that is used to share one AFE between
the two channels, thus resulting in more area savings. The
acquired digital data is passed to the on-chip feature extraction
engine for subsequent use by the fully connected network (FCN)
classifier.
As is evident from the above review, the verification setup
EEG-base HW classifiers requires a significant amount of effort,
involving the design of reliable communication links, complex
mixed-signal front ends, and robust digital data paths. Given
that a fully wearable, clinically approved, EEG-based solution
is still few years down the road, all the surveyed designs have
received a score of 2 for their experimental setups, except for [12]
(ISCAS_a’20), which received a score of 1 due to its streamlined
environment with a mandatory external host to pre-process and
extract features.
G. Scoring of Surveyed Hardware Designs
In our assessment of the various EEG-based hardware de-
signs, we have considered a number of metrics, including:
Wearability, Pre-processing, Feature Extraction, Classification,
Online Training support, Normalized Energy, Area, Latency, and
number of training types. Each design has received a score in
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
429
each and everyone of these metrics with the convention that a
high score implies a successful design for that particular category
the metric is meant to measure. The scores for Area, Latency
and Energy, range from 1 to 6, except when a specific design
has failed to report on one or another of these metrics, in which
case the assigned score is zero. In order to ensure a balanced
weight distribution between the assessed metrics, a normalized
score calculated with the corresponding maximum value is also
reported. The summary of all scores is presented in Table IV,
in which the best two designs correspond to the CNN chip
described in [11], [117], and the SoC implementing the SVM
classifier in [17]. However, the normalized score assigns the
top place to the SVM-based SoC [17] due to its consistently
good performance in all metrics. A more detailed summary
including the full technical specifications of the various designs
was already shown in Table III.
VI. CHALLENGES AND OPPORTUNITIES
In this section, we highlight two main challenges facing the
adoption of clinically-approved EEG-based emotion monitors.
The first is the absence of standardized, multi-modal, datasets
for training, validating and benchmarking the HW emotion clas-
sification engines. The second is the interlock not just with the
physio-psychological community but also with neuro-scientific
community where functional studies of brain activities may
provide a more solid foundation to the EEG-base platforms
for emotion classification. We also highlight a couple of re-
search opportunities. The first one is short-term and is related
to the leveraging of the latest ultra-low-power design tech-
niques for the design of medium-to-high complexity EEG-based
HW platforms for emotion classification. The second is to use
such platforms as tools in support of neuro-scientific research
on neurological disorders, including Alzheimer’s, Autism, and
Amyotrophic Lateral Sclerosis (ALS).
A. Challenge: EEG Datasets
Comprehensive and high-quality EEG data is a key condition
for the robustness of emotional recognition systems. In light of
this condition, available datasets are assessed to provide guide-
lines for closing the gap between existing EEG-based emotion
classifiers and next-generation, clinically-tested, robust classi-
fiers. The EEG-based datasets currently available for emotion
classification of emotions include the following:
430 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
1) Food, image-induced dataset for single-channel EEG
emotion classification [221].
2) DREAMER: EEG and ECG database for emotion
recognition using low-cost, wireless, off-the-shelf sen-
sors. [150].
3) DEAP: This is an often-used database for emotion anal-
ysis using physiological signals [18].
4) SEED: A dataset using emotional movie clips as stim-
uli [113].
5) K-EmoCon: A multi-modal sensor dataset for continuous
emotion recognition in naturalistic conversations [222].
6) GAMEEMO: An EEG database for emotion recognition
in the context of computer games. [223].
7) The listen and imagine dataset described in [224].
8) The driver’s affect dataset for autonomous vehicles de-
scribed in [82].
9) AMIGOS: A dataset for multimodal research on af-
fect, personality traits, and mood in individuals and
groups [225].
10) The FEXD dataset described in [42], which is a merger
of DEAP and DREAMER augmented with 5 additional
subjects monitored with 14-channel EEG helmets.
Unfortunately, none of the above datasets have been clinically
approved as a benchmark for validating EEG-based, wearable
emotion classifiers for medical use. In particular, none of them
have collected data from patients suffering neurological disor-
ders whose monitoring is one of the most relevant applications
of EEG-based emotion classification. It would be imperative
to involve such patients in dataset development as it would
naturally lead to the development and deployment of more
advanced and better-structured emotion recognition systems in
both software and hardware. Furthermore, it is noteworthy that
most of the currently employed EEG-based emotion datasets
are recorded in highly subjective environments, which of course
hinders their generalization.
In addition, existing EEG datasets tend to be “sanitized” prior
to their release. Whereas some form of signal conditioning
is highly desirable, excessive filtering may present the HW
platform with signals that are quite different from the EEG
waveforms that will be collected by a wearable device. It is
therefore important that raw EEG data be made available in the
datasets along with full data acquisition information.
Specifically in [18], [221], the EEG signals are passed through
a filtering stage in which a band-pass Hamming window is
applied. The data in [223] makes use of a built-in Sinc fil-
ter to discard frequency components above certain threshold,
eliminating artefacts related to hands, head, and arms motion.
In [224], a high-pass filter is employed to remove bad event
codes and noisy channels from the raw input data. The dataset
in [113] is down-sampled, band-pass filtered, and segmented
according to the duration of each movie clip. Since the FEXD
data in [42] corresponds to a unified data collection of two
publicly available datasets augmented with local experiments,
it employs a three stage ICA-based artefact removal process,
and a frequency conversion to deal with the heterogeneity of
the data. Although the goal of the frequency transformation
is to standardize data from multiple sources, it narrows down
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
TABLE V
DATASETS CURRENTLY AVAILABLE FOR TRAINING EEG-BASED EMOTION
CLASSIFIERS
P: Pictures. V: Video. A: Audio. VG: Video-games. S: Single Modality.
Subj: Subjects. M: Multiple Modalities. DS: Driving Simulation.
∗
: Variable number per subject. ¶ : Below 250 s. †: Above 840 s
Length‡ : Duration given in seconds
the research exploration into only frequency-based features. In
contrast to the previously presented datasets, the data collections
in [82], [222], do not report any pre-processing applied to the
raw input data.
The type of stimuli is another important element that has
to be decided ahead of EEG data acquisition, and whose stan-
dardization is highly desired to define a clear path for merging
and consolidating independent efforts. Normally, there are three
kinds of stimuli: aural, visual, and hybrid aural/visual. Table V
presents the variety of stimuli used in various references with the
visual stimulus approach being the most common. The stimuli
sources range from emotional or general movie clips and music
videos to social interaction, video gaming, and computerized
driving, but are unfortunately not sufficient to define a common
benchmark for advancing automatic emotion classification in
clinical applications. This is because the stimuli used are highly
subjective, and the classification results are difficult to replicate.
Besides the number of subjects and the duration of each
trial, subject ethnicity, occupation, personality traits [226], and
physiological conditions need to be reported in the EEG datasets,
along with the full details of the experimental setup so as
to be able to determine external influences if any [225]. One
particularly important detail is whether the recording has been
conducted in group or in isolation. Reporting such information
will enable researchers to further explore generic classifiers,
while the inclusion of a large number of subjects enables the
application of Deep Neural Networks to emotion classification.
Table V ranks the datasets in a descending order based on the
number of subjects involved. Although having several subjects
in the dataset does not necessarily imply a large number of
training samples, it does provide a quick categorization of the
size differences among the datasets. The data collection in [42]
has a large number of participants due to the frequency-based
combination of DEAP [18], DREAMER [150], and in-house
experiments. However, despite having a large collection, [18]
only reports frequency-domain data, which limits its usefulness
to research based only on frequency-domain features.
Another important aspect in EEG-based data collection is the
inclusion of reference sensors to provide anchor data points. Ta-
ble V shows the datasets using a single modality approach [221],
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
[223], [224], in which only the EEG data is collected without any
reference sensor, and those including a multi-modal recording
testbed composed by other sensors besides EEG, including ECG,
facial video clips, peripheral physiological signals (e.g., EOG,
galvanic skin response, etc), and infrared spectroscopy [18],
[113], [150], [222], [225]. Of course from a modeling and
machine-learnin viewpoint, the higher the number of sensor
modalities, the better, as more robust pre-processing and classi-
fication strategies can be explored.
B. Challenge: Neuroscience Interlock
The research we have surveyed so far on the EEG-based
emotion classification problem is engineering-focused. Such
research is essential if we are to achieve tangible solutions with
a potential of being deployed in a clinical context. It is however
biased toward existing technologies and may not be informed
with the latest advances in the neurosciences. In this subsection,
we approach the EEG-based emotion classification problem
from a Neuroscience perspective with a hope of identifying novel
approaches toward its solution.
In the early days of affective neuroscience, basic emotions
such as anger, fear and disgust, have been treated as basic
affects that arise from the limbic system, comprising structures
such as the thalamic nucleii, hippocampus, hypothalamus and
the cingulate cortex [227]. This view has been particularly
supported by the finding that there are shared facial expressions
for these emotions independent of cultural context as was already
postulated by Charles Darwin [228]. Based on the criticism
that subjectively, emotional states seem to be at times diffuse
and merge one into another, contemporary dimensional models
break down the emotional spectrum into two, sometimes three,
elemental components that correspond with well understood
signalling pathways within the brain-stem and cortex and whose
combination give rise to emotional phenomenologies [229].
Evidence from a variety of EEG and fMRI studies [230]
points out that within these pathways, cortical structures are
crucially involved in the processing of emotions, which opens
the possibility of an at least partial electro-physiological investi-
gation through non-invasive methods such as EEG and MEG. In
turn, the fMRI evidence provides significant justification for the
noninvasive, wearable EEG-based approaches to emotion clas-
sification. Indeed, such approaches primarily acquire cortical
signals.
Different authors have used a variety of terms for these
elemental emotion components, and in this paper, we have made
use of the terminology proposed by [229], according to which
the term valence describes the continuum between euphoria and
dysphoria, thought to be originating in the mesolimbic dopamine
system, and the serotonergic projections from the dorsal Nucleus
Raphe to the Ventral Striatum.
An orthogonal component is arousal (also called Ap-
proach/Withdrawl or Behavioral Activation among several other
terminologies). Arousal plays an important role in the formation
of declarative memory where it seems to modulate the ability to
commit a given situation to memory. As shown in [231] by the
use of gustatory stimuli, amygdala activation directly correlated
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
431
with stimulus intensity rather than its valence. Emotional stimuli
activate the reticular formation and amygldala as well [232].
The amygdala activation can be directly correlated with skin
conductive response measurements [233].
Of course, the involvement of deeper brain areas in the
processing of emotions poses a stiff challenge for their direct,
non-invasive electrophysiological observation since only their
cortical correlates can be acquired. However, areas of the limbic
system control autonomic nervous system responses, that can
be quantified by measuring skin impedance, heart-rate, and
breathing frequency, among other vital signs [234]. In particular,
these secondary markers have been used to quantify valence as
early as 1993 [235]. Whereas such quantification techniques
form a useful approach to the non-invasive monitoring of the
limbic system, it has however been noted that the secondary
responses are significantly influenced by factors besides the
subjects emotional state [236].
From an electrophysiological standpoint, it is often over-
looked that autonomic responses can in fact be acquired as part
of the EEG artefacts and are usually rejected as “noise” in the
early feature extraction stages of the EEG signals. The EEG
artefacts contain the following autonomic responses:
r Heart Rate: In the EEG signals, cardiac-related artefacts
appear in amplitude and frequency ranges that are sim-
ilar to those due to neuronal activity. They are however
highly stereotypical and periodic, which facilitates their
identification. In order to separate these signals from EEG,
methods such as Independent Component Analysis can be
employed effectively [237].
r Breathing Rate: It is well known that the respiratory fre-
quency modulates the spectral power of the alpha EEG
band [238], which may allow the inference of breathing
frequency, and from the latter, the arousal state of the
subject.
r Eye Blinking Rate: Occulomotor activity can be observed
on the frontal electrodes as muscle artefacts. Whereas some
EEG databases tend to filter out this artefact, it is interesting
to observe that the rate of eye blinks correlates with the
sympathetic arousal of a subject [239], which may lead to
a new secondary marker of emotions in EEG recordings.
r Skin Impedance: Many commercially available EEG se-
tups include an electrode-skin impedance test in order to fa-
cilitate the correct placement of wet electrodes. Wearables
for continuous skin impedance measurements have also
been well researched by the affective computing commu-
nity [234], [240]. In an EEG-based, emotion classification
system, they may provide important secondary information
that will significantly improve the accuracy of the emotion
inference engine.
One typical psycho-physiological paradigm for the investi-
gation of autonomic responses is to “Startle” the subject with
an unexpected stimulus such as an acoustical burst of white
noise. The intensity of the involuntary reaction to the sudden
stimulus is modulated by the emotional state of the subject. Even
EEG responses to startle-stimuli could be considered a form of
sensory evoked potential [241] and find use cases in clinical
diagnoses of psychological disorders such as Alexithemia and
432 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
Schizophrenia [242], [243]. The EEG reponses may also be used
as a tool to estimate the emotional capacities of patients with
neurodegenerative diseases such as Alzheimer’s.
Another important paradigm of the neuro-anatomical basis of
emotion classification is to augment biomarkers in the frequency
domain representations of EEG signals with the spatial informa-
tion of cortical activity [244], [245]. The traditional spectral
decomposition into alpha, beta, and gamma bands is further
analyzed in light of the frontal alpha-band power asymmetry
between hemispheres, and the results are used to measure the
activated approach/withdrawal responses. Following the 10/20
system, the electrode pairs F3-F4, F6-F5, F8-F7 are investigated
and their power spectral density in the alpha band (8–13 Hz)
compared according to the IHPA given in Eq. (5).
Whereas these approaches yield promising results [246], the
neurological basis that motivates the selection of electrode pairs
in IHPA may need to be investigated further. It is clear that
the availability of a hardware emotion classifier that can collect
real-time EEG data as function of electrode placement and selec-
tion is bound to facilitate the investigation of the neuroscientific
research questions.
In such context, EEG acquisition platforms should offer
the capability to log raw signals while enabling reproducible
electrode placement. This can be accomplished by using, for
instance, optical markers on the electrodes to photographically
document any deviation from the clinical 10–20 standard. More
will be said on the use of the EEG hardware platform in neuro-
science research in Subsection VI-D.
C. Opportunity: Ultra Low-Power Hardware Design
As already mentioned in Section III, robust System-on-Chip
(SoC) solutions addressing the EEG-based emotion classifica-
tion problem opt for simple classifiers to meet a low-power
budget. The HW designs we have surveyed use traditional,
low-power and power management circuit techniques, and the
problem of trading off classifier complexity with more aggres-
sive power reduction strategies is yet to be fully explored.
The recent near/supra-threshold circuit design methodologies
should enable such trade-off exploration under robust operation
conditions. The new hardware classifiers of emotions can now
exploit near/supra-threshold designs to close the gap between a
high-accuracy complex classifier and a stringent power envelop
constraint. Although there are extensive research efforts to oper-
ate with sub-threshold voltages under process variations [247]–
[259], the combination of low-voltage and robustness benefits
is achieved best in the near/supra-threshold region.
New mechanisms, such as Adaptive Body Biasing (ABB) in
FDSOI technology [172], enable a highly controlled operation
in the near/supra-threshold region, as it prevents the worst cases
from going deep into the sub-threshold regime. This ensures
very reliable and robust designs with high yield and ultra-low
voltages. These designs can still achieve reasonable performance
while operating at ultra low voltages. This is in fact the case
for forward designs (flipped wells) in the near/supra-threshold
operation [170], [171]. Alternatively, complex hardware classi-
fiers having for objective power consumption minimization, can
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
actively control performance and leakage by using 22FDX ABB
designs in reverse bias configurations (conventional well) [260].
The latter designs result in extremely efficient implementations
with low to medium performance, as required for real-time bio-
signal processing, and allows direct SRAM integration within
the ultra low voltage domain. It is worth noting that the winning
design (ZEN) of the 2021 German national competition on
“Energy-efficient AI system” in the ASIC category was such
a design, in which an ECG classifier based on gated-recurrent
units (GRU) has been implemented [202]. This ultra low voltage
design paradigm is very much worth exploring for the classifica-
tion of EEG signals as well. We believe that it represents a signif-
icant opportunity for achieving complex HW emotion classifiers
operating in the ultra low power domains with tightly-coupled
cores to enable flexible pre-processing, and SRAM blocks fully
integrated in support of adaptive and subject specific wearables.
D. Opportunity: Research Platform for EEG-Based
Monitoring of Neurological Disorders
The challenge already mentioned in VI-B regarding the inter-
lock with neuro-scientific research can be turned into a signif-
icant research opportunity as the ultra-low-power, near/supra-
threshold EEG platforms may be used to enable the social
integration and improve the medical treatments of those patients
suffering from certain neurological disorders. Such opportu-
nity is presented in the context of three important disorders:
Alzheimer’s, Autism, and Amyotrophic Lateral Sclerosis.
1) Alzheimer’s Disease: Despite the degenerative nature of
the disease, there is enough evidence to support that Alzheimer’s
patients are still able to identify and process emotions. Multiple
studies have been conducted to prove that the Alzheimer subjects
have a reasonable understanding of emotions [261]–[263]. As
presented in [264], Alzheimer’s patients in the moderate stage of
the disease encounter difficulties only when processing second
order emotion inferences as they require a cognitive process hav-
ing a representation in working memory of embedded clauses.
Otherwise, the amygdala of the Alzheimer’s patient does not
deteriorate, and as such, it remains an active agent in emotion
processing throughout the disease, even after the patient loses
the language capabilities [265].
One attempt to classify emotions using SVM for patients
with Alzheimer’s is documented in [266], where functional
magnetic resonance imaging (fMRI) is used as the brain com-
puter interface. The reported classification accuracy for pleasant
and unpleasant emotions was reported to be 71% on average,
reaching a maximum of 83%, which are the typical accuracies
displayed by healthy subjects in state-of-the-art emotion clas-
sification datasets. Despite the intrusive nature of fMRI, this
research confirms the presence and relevance of brain-related
physiological markers in Alzheimer’s patients.
2) Amyotrophic Lateral Sclerosis (ASL): Functional mag-
netic resonance imaging (fMRI) has also been used to examine
the brain activity of patients suffering from Amyotrophic Lateral
Sclerosis (ALS) under controlled emotion elicitation experi-
ments. An analysis is conducted in [267] on the fMRI data of
volunteeers diagnosed with ALS while they were being exposed
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
to visual stimuli from the International Affective Picture System
(IAPS) dataset. The two experimental sessions were recorded
with a time lapse of six months. An important conclusion of
the analysis is that the ALS subjects displayed lower arousal
levels after the six-month gap. This “history effect” is a good
candidate for further EEG-based research as it could impact the
development of reference data for the emotion classification of
ALS patients.
3) Autism Spectrum Disorder: The convergence of affective
computing and autism has been recognized and explored in [268]
and [269]. Early work on autism monitoring proposed multi-
modal affective sensing technologies to facilitate the socio-
emotional integration of autistic patients [269]. The early de-
tection of Autism Spectrum Disorder (ASD) in newborns is
a key factor to avoid the irreversible consequences of ASD
during adulthood. However, the lack of an approved medical
procedure has made the early diagnosis of ASD very difficult as
the signs and symptoms are very much subject-dependent, and
their severity varies from one patient to another. It is therefore
important to develop subject-specific, custom neuro-feedback
systems to track socio-emotional responses to stimuli in real-
time. Although conventional EEG systems have been vastly used
for affect recognition in adults and children under controlled en-
vironments, they are still short of providing day-to-day monitor-
ing because of their high cost, limited wearability, and subject-
independent training. In [15], a patient-centric, FPGA-based
sensor is proposed that could be integrated into an inexpensive
and low-power EEG headset to furnish seamless signal recording
that should facilitate the treatment of ASD patients. In [270],
a deep-learning approach based on multi-modal sensing has
been proposed to address patient-specific variations in autistic
children.
VII. RECOMMENDATIONS FOR FUTURE DIRECTIONS
This article has provided a comprehensive, up-to-date survey
of hardware-based classifiers of emotions using EEG signals.
The more widely researched software-based classifiers have
been used to identify the challenges and opportunities facing the
emerging hardware classifiers. In this final section of our survey
article, we give recommendations for promising, high-impact
research directions in the area of EEG-based, wearables for
emotion classification. They are concerned with embedded pre-
processing, high-resolution feature extraction, HW platforms
with tightly-coupled cores, and multi-modal, cross-cultural EEG
datasets.
A. Embedded Pre-Processing
As mentioned earlier, removing artefacts is highly desired in
single-mode EEG-based classifiers. Artefacts commonly mask
interesting activity [38]. For example, power line noise obscures
cortical activity around 50 Hz (or 60 Hz), whereas drifting
of electrodes interfere with slow cortical potentials. Ocular or
myogenic activity can be interpreted as cortical activity [271],
especially if they correlate with external stimulation [272],
which is quite common in emotion-based experiments.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
433
Artefact mitigation strategies sometimes trigger additional
undesired effects such as the multipolar excursions caused by
the application of high-pass filters [273] and that are commonly
mistaken for neural activity [274], [275]. To address these prac-
tical issues, the EEG-based hardware design must include low-
cost but robust pre-processing techniques for “smart” artefact
mitigation. As shown in Fig. 3, filter-based pre-processing does
not remove all artefacts present in the EEG data. Furthermore,
ICA-based techniques such as the one proposed in [12] are not
entirely hardware friendly. Nor are they sufficient in themselves
to deal with all artefacts. For instance, ICA is not effective in
removing artefacts due to electrode-specific or sensor-specific
sources as such sources cannot be suppressed linearly as in blind
source separation or beam-forming techniques. [276], [277].
B. High-Resolution Feature Extraction
The concept of brainwave bands has been a useful model
for extracting frequency-domain features but is not entirely sup-
ported by solid neurophysiological evidence. The average power
in a single frequency band is normally assumed to be a represen-
tative measure of the underlying activity. However, such measure
has proven to be quite coarse, and future frequency-domain brain
activity models should be based on higher frequency resolutions
and more fundamental, non-invasive approaches for capturing
brain activity. Besides the high frequency resolution, small fre-
quency windows may help better capture power excursions over
the full frequency range. Furthermore, the inclusion of Cross-
Frequency Coupling (CFC) features in the EEG-based hardware
classifiers of emotions represents an important alternative path
to wearable emotion detection with the potential to improve
robustness [155] and reduce memory footprint by focusing on
correlation graphs [157], rather than on the full raw EEG dataset.
Another improvement in the frequency-domain features is
the systematic inclusion of spatial asymmetry as is done in the
10/20 EEG system (see “Spatial Features” in Section IV) and
its extension to other EEG electrode patterns, especially those
with a minimal number of electrodes.
C. Hardware Platforms With Tightly-Coupled Cores
The hardware designs we have surveyed have all been the
subject of several engineering trade-offs, the most important
of which is the one trading off inference engine accuracy and
complexity for low power and small footprint. The lightweight
SVM classifier of [17] is an extreme example of such trade-
off, especially that the SVM inference engine has very limited
margin to adapt to changes in the subject’s emotional signaling
over time. Such changes are common in patients suffering from
neurological diseases in which their emotional capabilities are
likely to deteriorate in the late stages of the disease [267]. On
the other hand, the designs of [16] (ISCAS_b’20) and [11] (JET-
CAS’19) have successfully shown that some degree of adapta-
tion is achievable with complex classifiers. Despite its modest
complexity, the approach in [16] (ISCAS_b’20) presents an
interesting combination of an embedded core with an ML accel-
erator that can provide sufficient margins to implement multiple
DNNs with flexible feature extraction. However, both power and
434 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
footprint are increased as a result of this added flexibility. One
promising approach to improve the trade-off between adaptation
and energy efficiency is to compute the DNN inference within
the accelerator, while keeping a tightly-coupled core within the
low-power domain to achieve flexible pre-processing and feature
extraction as was done in [278]. The programmability of the low-
power core should provide the flexibility needed for adapting
the pre-processing needs to subject and environment variations
with the configurable accelerator ensuring short inference times
without compromising wearability or power. Another promising
direction is to combine such core-based platforms with network
weight binarization [279].
D. Comprehensive EEG Datasets and Benchmarks
Unfortunately, the field of EEG-based emotion classification
is still lacking a comprehensive EEG dataset where all stim-
ulation modalities are included. Existing datasets use at best
a very small number of multimodal signals with the machine
learning engines limited to training and inference with those
signals only. Such restrictions on sensing modalities lead to
sensor biases that are detrimental to network training. Future
datasets for EEG-based emotion classification should have the
following characteristics:
1) Multi-modality: The dataset should cover a broader range
of multi-modal markers to minimize the impact of sensing
biases.
2) Universality: The dataset should be universal in that it
should include subjects from various regions, ethnic back-
grounds, occupations, and health conditions, in order to
account for inter-cultural differences and similarities in
response to emotion stimuli.
The dataset that comes closest to the above two requirements
is [18], which unfortunately still needs a well-established pro-
tocol to eliminate distortions during emotion elicitation. One
such distortion is the impact of trail timing and duration on
collected data. Indeed, [150] has shown that during an extended
period of trial time, the emotional state of a person may vary,
introducing bias in the collected data. In the 60 seconds allocated
to data collection, several emotions may arise in response to the
same stimulus, thus introducing ambiguity as to the correlation
between the stimulus and the collected data. Further recommen-
dations for dataset improvements include:
1) Use of sudden stimuli: The use of sudden stimuli such as
white noise bursts, can help reduce the bias of more com-
mon stimuli such as song snippets, film clips, and video
games. The latter stimuli are known to induce emotions
with a history effect, in that they depend on the subject’s
emotions prior to the initiation of the stimuli.
2) Consistent labeling: Methods and protocols should be
developed and adopted so that EEG data is labeled consis-
tently across subjects and throughout the EEG collection
time frame, not just for the collection session itself but
for the entire duration of dataset construction. This con-
sistency is especially important around the intermediate
regions of the Valence-Arousal plane, where the value of
the data is the least. Consistency will increase the value
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
of such data and make it more usable for training and
validation.
3) Pre-processing library: The multi-modal data which
would include EEG, visual, aural, temperature, pulse
rate, skin impedance, and breathing rate information
need to be pre-processed to ensure consistency and re-
duce undesired interferences. So far, such pre-processing
has been fragmented with each research group em-
ploying its own algorithms to pre-process data and ex-
tract features from the EEG sub-band signals. Algo-
rithmic and pre-processing differences result in vari-
ations in the training data. The multi-modal, univer-
sal datasets we are advocating should come with their
own standardized libraries of pre-processing tools so
as to insure input uniformity across all classification
platforms.
4) Standardized benchmarking: It is important that the com-
munity adopts a set of universal benchmarks for evaluating
emotion classification engines. Such benchmarks will be
essential for validating the clinical use of EEG-based
wearable devices for emotion classification.
VIII. CONCLUSIONS
In this paper, we have presented the very first critical review
of EEG-based wearable classifiers of human emotions. We have
surveyed their algorithmic foundations, their feature extraction
methodologies, and their implementation details. We have also
assessed their performance based on a variety of metrics that we
have condensed into a single score assigned to all the HW ac-
celerators included in this survey. In addition, We have provided
a neuroscience-based analysis of existing hardware accelerators
of emotion classifiers and used it to map out several research
opportunities, including multi-modal hardware platforms, accel-
erators with tightly-coupled cores, and pre-processing libraries
for universal EEG-based datasets.
REFERENCES
[1] Z. He et al., “Advances in multimodal emotion recognition based on
brain-computer interfaces,” Brain Sci., vol. 687, no. 10, pp. 1–29, 2020.
[2] A. Craik, Y. He, and J. L. Contreras-Vidal, “Deep learning for elec-
troencephalogram (EEG) classification tasks: A review,” J. Neural Eng.,
vol. 16, no. 3, Apr. 2019, Art. no. 031001.
[3] K. Zhang et al., “Application of transfer learning in EEG decoding based
on brain-computer interfaces: A review,” Sensors, vol. 6321, no. 21, pp.
1–25, Jul. 2020.
[4] S. M. Alarcão and M. J. Fonseca, “Emotions recognition using EEG sig-
nals: A survey,” IEEE Trans. Affect. Comput., vol. 10, no. 3, pp. 374–393,
Jul.-Sep. 2019.
[5] Y. Zhao, W. Zhao, C. Jin, and Z. Chen, “A review on EEG based emotion
classification,” in Proc. IEEE 4th Adv. Inf. Technol., Electron. Automat.
Control Conf., vol. 1, 2019, pp. 1959–1963.
[6] F. Lotte et al., “A review of classification algorithms for EEG-based
brain-computer interfaces: A 10 year update,” J. Neural Eng., vol. 15,
no. 3, Feb. 2018, Art. no. 031005.
[7] Y. Wei et al., “A review of algorithm hardware design for AI-based
biomedical applications,” IEEE Trans. Biomed. Circuits Syst., vol. 14,
no. 2, pp. 145–163, Apr. 2020.
[8] A. M. Brouwer, T. O. Zander, J. B. van Erp, J. E. Korteling, and A. W.
Bronkhorst, “Using neurophysiological signals that reflect cognitive or
affective state: Six recommendations to avoid common pitfalls,” Front.
Neurosci., vol. 9, no. 136, pp. 1–11, 2015.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
[9] Y. Huang, K. Wang, Y. Ho, C. He, and W. Fang, “An edge AI system-on-
chip design with customized convolutional-neural-network architecture
for real-time EEG-based affective computing system,” in Proc. IEEE
Biomed. Circuits Syst. Conf., 2019, pp. 1–4.
[10] K.-Y. Wang, Y.-D. Huang, Y.-L. Ho, and W.-C. Fang, “A customized
convolutional neural network design using improved softmax layer for
real-time human emotion recognition,” in Proc. IEEE Int. Conf. Artif.
Intell. Circuits Syst., 2019, pp. 102–106.
[11] W. Fang, K. Wang, N. Fahier, Y. Ho, and Y. Huang, “Development and
validation of an EEG-based real-time emotion recognition system using
edge AI computing platform with convolutional neural network system-
on-chip design,” IEEE Trans. Emerg. Sel. Topics Circuits Syst., vol. 9,
no. 4, pp. 645–657, Dec. 2019.
[12] H. A. Gonzalez, S. Muzaffar, J. Yoo, and I. A. M. Elfadel, “An in-
ference hardware accelerator for EEG-based emotion detection,” in
Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), 2020, pp. 1–5, doi:
10.1109/ISCAS45731.2020.9180728.
[13] H. A. Gonzalez, S. Muzaffar, J. Yoo, and I. M. Elfadel, “BioCNN: A hard-
ware inference engine for EEG-based emotion detection,” IEEE Access,
vol. 8, pp. 140896–140914, 2020, doi: 10.1109/ACCESS.2020.3012900.
[14] A. R. Aslam and M. A. B. Altaf, “An 8 channel patient specific neuro-
morphic processor for the early screening of autistic children through
emotion detection,” in Proc. IEEE Int. Symp. Circuits Syst., 2019,
pp. 1–5.
[15] A. R. Aslam, T. Iqbal, M. Aftab, W. Saadeh, and M. A. B. Altaf,
“A 10.13uJ/classification 2-channel deep neural network-based SoC for
emotion detection of autistic children,” in Proc. IEEE Custom Integr.
Circuits Conf., 2020, pp. 1–4.
[16] C. J. Yang, N. Fahier, C. Y. He, W. C. Li, and W. C. Fang, “An AI-edge
platform with multimodal wearable physiological signals monitoring
sensors for affective computing applications,” in Proc. IEEE Int. Symp.
Circuits Syst., 2020, pp. 1–5.
[17] A. R. Aslam and M. A. B. Altaf, “An on-chip processor for chronic neu-
rological disorders assistance using negative affectivity classification,”
IEEE Trans. Biomed. Circuits Syst., vol. 14, no. 4, pp. 838–851, Aug.
2020.
[18] S. Koelstra et al., “DEAP: A database for emotion analysis using physi-
ological signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31,
Jan.–Mar. 2012.
[19] M. Sazgar and M. G. Young, “EEG artifacts,” in Absolute Epilepsy EEG
Rotation Review. Berlin, Germany: Springer, 2019, pp. 149–162.
[20] J. W. Clark Jr., “The origin of biopotentials,” Med. Instrum.: Appl. Des.,
vol. 3, pp. 121–182, 1998.
[21] P. L. Nunez et al., Electric Fields of the Brain: The Neurophysics of EEG.
London, U.K.: Oxford Univ. Press, USA, 2006.
[22] L. Sörnmo and P. Laguna, Bioelectrical Signal Processing in Cardiac
and Neurological Applications. New York, NY, USA: Academic Press,
2005, vol. 8.
[23] B. Grundlehner and V. Mihajlović, “Ambulatory EEG monitoring,” in
Encyclopedia Biomedical Engineering, R. Narayan, Ed., Oxford, U.K.:
Elsevier, 2019, pp. 223–239.
[24] D. B. Stone, G. Tamburro, P. Fiedler, J. Haueisen, and S. Comani,
“Automatic removal of physiological artifacts in EEG: The optimized
fingerprint method for sports science applications,” Front. Hum. Neu-
rosci., vol. 12, no. 96, pp. 1–15, 2018.
[25] M. K. Islam, A. Rastegarnia, and Z. Yang, “Methods for artifact detec-
tion and removal from scalp EEG: A review,” Neurophysiologie Clin-
ique/Clinical Neuriophysiol., vol. 46, no. 4/5, pp. 287–305, 2016.
[26] C. Marque, C. Bisch, R. Dantas, S. Elayoubi, V. Brosse, and C. Perot,
“Adaptive filtering for ECG rejection from surface EMG recordings,” J.
Electromyogr. Kinesiol., vol. 15, no. 3, pp. 310–315, 2005.
[27] K. T. Sweeney, T. E. Ward, and S. F. McLoone, “Artifact removal
in physiological signals-practices and possibilities,” IEEE Trans. Inf.
Technol. Biomed., vol. 16, no. 3, pp. 488–500, May 2012.
[28] J. A. Urigüen and B. Garcia-Zapirain, “EEG artifact removal-state-of-the-
art and guidelines,” J. Neural Eng., vol. 12, no. 3, 2015, Art. no. 031001.
[29] G. L. Wallstrom, R. E. Kass, A. Miller, J. F. Cohn, and N. A. Fox,
“Automatic correction of ocular artifacts in the EEG: A comparison of
regression-based and component-based methods,” Int. J. Psychophysiol.,
vol. 53, no. 2, pp. 105–119, 2004.
[30] D. Wu, X. Han, Z. Yang, and R. Wang, “Exploiting transfer learning
for emotion recognition under cloud-edge-client collaborations,” IEEE
J. Sel. Areas Commun., vol. 39, no. 2, pp. 479–490, Feb. 2021.
[31] X. Tao, Z. Chen, M. Xu, and J. Lu, “Rebuffering optimization for dash
via pricing and EEG-based QoE modeling,” IEEE J. Sel. Areas Commun.,
vol. 37, no. 7, pp. 1549–1565, Jul. 2019.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
435
[32] T. Nguyen, T. Zhou, T. Potter, L. Zou, and Y. Zhang, “The cortical network
of emotion regulation: Insights from advanced EEG-fMRI integration
analysis,” IEEE Trans. Med. Imag., vol. 38, no. 10, pp. 2423–2433,
Oct. 2019.
[33] P. Li et al., “EEG based emotion recognition by combining functional
connectivity network and local activations,” IEEE Trans. Biomed. Eng.,
vol. 66, no. 10, pp. 2869–2881, Oct. 2019.
[34] R. M. Mehmood, H. J. Yang, and S. H. Kim, “Children emo-
tion regulation: Development of neural marker by investigating hu-
man brain signals,” IEEE Trans. Instrum. Meas., vol. 70, 2021,
Art. no. 4000411.
[35] Y. H. Chen, S. W. Chen, and M. X. Wei, “A VLSI implementation of
independent component analysis for biomedical signal separation using
CORDIC engine,” IEEE Trans. Biomed. Circuits Syst., vol. 14, no. 2,
pp. 373–381, Apr. 2020.
[36] Y. H. Chen and S. P. Wang, “Low-cost implementation of independent
component analysis for biomedical signal separation using very-large-
scale integration,” IEEE Trans. Circuits Syst. II: Exp. Briefs, vol. 67,
no. 12, pp. 3437–3441, Dec. 2020.
[37] D. Yao, “A method to standardize a reference of scalp EEG recordings
to a point at infinity,” Physiol. Meas., vol. 22, no. 4, pp. 693–711, 2001.
[38] A. de Cheveigné and D. Arzounian, “Robust detrending, rereferencing,
outlier detection, and inpainting for multichannel data,” NeuroImage,
vol. 172, pp. 903–912, 2018.
[39] T. Song, W. Zheng, P. Song, and Z. Cui, “EEG emotion recognition using
dynamical graph convolutional neural networks,” IEEE Trans. Affect.
Comput., vol. 11, no. 3, pp. 532–541, Jul.-Sep. 2020.
[40] B. H. Kim and S. Jo, “Deep physiological affect network for the recog-
nition of human emotions,” IEEE Trans. Affect. Comput., vol. 11, no. 2,
pp. 230–243, Apr.-Jun. 2020.
[41] J. Chen, P. Zhang, Z. Mao, Y. Huang, D. Jiang, and Y. Zhang, “Accu-
rate EEG-based emotion recognition on combined features using deep
convolutional neural networks,” IEEE Access, vol. 7, pp. 44317–44328,
Mar. 2019.
[42] H. A. Gonzalez, J. Yoo, and I. M. Elfadel, “EEG-based emotion
detection using unsupervised transfer learning,” in Proc. 41st Annu.
Int. Conf. IEEE Eng. Med. Biol. Soc., 2019, pp. 694–697, doi:
10.1109/EMBC.2019.8857248.
[43] A. H. Setianingrum and B. S. Budhi, “Model prediction of psychoanalysis
trend of radical emotional aggressiveness using EEG and GLCM-SVM
method,” in Proc. 6th Int. Conf. Cyber IT Serv. Manage., 2018, pp. 1–7.
[44] M. Alsolamy and A. Fattouh, “Emotion estimation from EEG signals
during listening to Quran using PSD features,” in Proc. 7th Int. Conf.
Comput. Sci. Inf. Technol., 2016, pp. 1–5.
[45] S. Liu, J. Tong, M. Xu, J. Yang, H. Qi, and D. Ming, “Improve the
generalization of emotional classifiers across time by using training
samples from different days,” in Proc. 38th Annu. Int. Conf. IEEE Eng.
Med. Biol. Soc., 2016, pp. 841–844.
[46] M. Stikic, R. R. Johnson, V. Tan, and C. Berka, “EEG-based classification
of positive and negative affective states,” Brain-Comput. Interfaces,
vol. 1, no. 2, pp. 99–112, 2014.
[47] H. Ullah, M. Uzair, A. Mahmood, M. Ullah, S. D. Khan, and F. A. Cheikh,
“Internal emotion classification using EEG signal with sparse discrimi-
native ensemble,” IEEE Access, vol. 7, pp. 40144–40153, Mar. 2019.
[48] Z. Wang, Y. Tong, and X. Heng, “Phase-locking value based graph
convolutional neural networks for emotion recognition,” IEEE Access,
vol. 7, pp. 93711–93722, Jul. 2019.
[49] W. Zhao, Z. Zhao, and C. Li, “Discriminative-CCA promoted by EEG
signals for physiological-based emotion recognition,” in Proc. 1st Asian
Conf. Affect. Comput. Intell. Interaction, 2018, pp. 1–6.
[50] S. Moon, S. Jang, and J. Lee, “Convolutional neural network approach for
EEG-based emotion recognition using brain connectivity and its spatial
information,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.,
2018, pp. 2556–2560.
[51] Y. Li, J. Huang, H. Zhou, and N. Zhong, “Human emotion recognition
with electroencephalographic multidimensional features by hybrid deep
neural networks,” Appl. Sci., vol. 7, no. 10: 1060, pp. 1–20, 2017.
[52] H. Xu and K. N. Plataniotis, “Affective states classification using EEG
and semi-supervised deep learning approaches,” in Proc. IEEE 18th Int.
Workshop Multimedia Signal Process., 2016, pp. 1–6.
[53] J. Atkinson and D. Campos, “Improving BCI-based emotion recognition
by combining EEG feature selection and kernel classifiers,” Expert Syst.
Appl., vol. 47, pp. 35–41, 2016.
[54] A. M. Bhatti, M. Majid, S. M. Anwar, and B. Khan, “Human emotion
recognition and analysis in response to audio music using brain signals,”
Comput. Hum. Behav., vol. 65, pp. 267–275, 2016.
436 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
[55] E. Kroupi, J.-M. Vesin, and T. Ebrahimi, “Subject-independent odor
pleasantness classification using brain and peripheral signals,” IEEE
Trans. Affect. Comput., vol. 7, no. 4, pp. 422–434, Oct.–Dec. 2016.
[56] R. M. Mehmood and H. J. Lee, “A novel feature extraction method based
on late positive potential for emotion recognition in human brain signal
patterns,” Comput. Elect. Eng., vol. 53, pp. 444–457, 2016.
[57] N. Thammasan, K.-i. Fukui, and M. Numao, “Application of deep belief
networks in EEG-based dynamic music-emotion recognition,” in Proc.
Int. Joint Conf. Neural Netw., 2016, pp. 881–888.
[58] Z. Lan, O. Sourina, L. Wang, and Y. Liu, “Real-time EEG-based emo-
tion monitoring using stable features,” Vis. Comput., vol. 32, no. 3,
pp. 347–358, 2016.
[59] S. Jirayucharoensak, S. Pan-Ngum, and P. Israsena, “EEG-based emotion
recognition using deep learning network with principal component based
covariate shift adaptation,” Sci. World J., vol. 2014, pp. 1–10, 2014.
[60] S. Hatamikia, K. Maghooli, and A. M. Nasrabadi, “The emotion recogni-
tion system based on autoregressive model and sequential forward feature
selection of electroencephalogram signals,” J. Med. Signals Sensors,
vol. 4, no. 3, pp. 194–201, 2014.
[61] Y.-Y. Lee and S. Hsieh, “Classifying different emotional states by means
of EEG-based functional connectivity patterns,” PLoS one, vol. 9, no. 4,
2014, Art. no. e95415.
[62] Y.-P. Lin, Y.-H. Yang, and T.-P. Jung, “Fusion of electroencephalographic
dynamics and musical contents for estimating emotional responses in
music listening,” Front. Neurosci., vol. 8, no. 94, pp. 1–14, 2014.
[63] X.-W. Wang, D. Nie, and B.-L. Lu, “Emotional state classification from
EEG data using machine learning approach,” Neurocomputing, vol. 129,
pp. 94–106, 2014.
[64] Y.-H. Liu, C.-T. Wu, Y.-H. Kao, and Y.-T. Chen, “Single-trial EEG-based
emotion recognition using kernel eigen-emotion pattern and adaptive
support vector machine,” in Proc. 35th Annu. Int. Conf. IEEE Eng. Med.
Biol. Soc., 2013, pp. 4306–4309.
[65] M. Murugappan and S. Murugappan, “Human emotion recognition
through short time electroencephalogram (EEG) signals using fast
Fourier transform (fft),” in Proc. IEEE 9th Int. Colloq. Signal Process.
Appl., 2013, pp. 289–294.
[66] H. J. Yoon and S. Y. Chung, “EEG-based emotion estimation using
bayesian weighted-log-posterior function and perceptron convergence
algorithm,” Comput. Biol. Med., vol. 43, no. 12, pp. 2230–2237, 2013.
[67] T. D. Pham and D. Tran, “Emotion recognition using the emotiv EPOC
device,” in Proc. Int. Conf. Neural Inf. Process., 2012, pp. 394–399.
[68] D. Nie, X.-W. Wang, L.-C. Shi, and B.-L. Lu, “EEG-based emotion
recognition during watching movies,” in Proc. 5th Int. IEEE/EMBS Conf.
Neural Eng., 2011, pp. 667–670.
[69] R. Khosrowabadi, H. C. Quek, A. Wahab, and K. K. Ang, “EEG-based
emotion recognition using self-organizing map for boundary detection,”
in Proc. 20th Int. Conf. Pattern Recognit., 2010, pp. 4242–4245.
[70] Y.-P. Lin et al., “EEG-based emotion recognition in music listening,”
IEEE Trans. Biomed. Eng., vol. 57, no. 7, pp. 1798–1806, Jul. 2010.
[71] M. Mikhail, K. El-Ayat, J. A. Coan, and J. J. Allen, “Using minimal
number of electrodes for emotion detection using brain signals produced
from a new elicitation technique,” Int. J. Auton. Adaptive Commun. Syst.,
vol. 6, no. 1, pp. 80–97, 2013.
[72] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal
database for affect recognition and implicit tagging,” IEEE Trans. Affect.
Comput., vol. 3, no. 1, pp. 42–55, Jan.-Mar. 2012.
[73] T. F. Bastos-Filho, A. Ferreira, A. C. Atencio, S. Arjunan, and
D. Kumar, “Evaluation of feature extraction techniques in emotional state
recognition,” in Proc. 4th Int. Conf. Intell. Hum. Comput. Interaction,
2012, pp. 1–6.
[74] R.-N. Duan, X.-W. Wang, and B.-L. Lu, “EEG-based emotion recognition
in listening music by using support vector machine and linear dynamic
system,” in Proc. Int. Conf. Neural Inf. Process., 2012, pp. 468–475.
[75] S. Nasehi, H. Pourghassem, and I. Isfahan, “An optimal EEG-based emo-
tion recognition algorithm using Gabor,” WSEAS Trans. Signal Process.,
vol. 3, no. 8, pp. 87–99, 2012.
[76] M. Soleymani, M. Pantic, and T. Pun, “Multimodal emotion recognition
in response to videos (extended abstract),” in Proc. Int. Conf. Affect.
Comput. Intell. Interaction, 2015, pp. 491–497.
[77] L. Brown, B. Grundlehner, and J. Penders, “Towards wireless emotional
valence detection from EEG,” in Proc. Annu. Int. Conf. IEEE Eng. Med.
Biol. Soc., 2011, pp. 2188–2191.
[78] X.-W. Wang, D. Nie, and B.-L. Lu, “EEG-based emotion recognition
using frequency domain features and support vector machines,” in Proc.
Int. Conf. Neural Inf. Process., 2011, pp. 734–743.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
[79] S. Koelstra et al., “Single trial classification of EEG and peripheral
physiological signals for recognition of emotions induced by music
videos,” in Proc. Int. Conf. Brain Inform., 2010, pp. 89–100.
[80] H. Hu, Z. Zhu, Z. Gao, and R. Zheng, “Analysis on biosignal characteris-
tics to evaluate road rage of younger drivers: A driving simulator study,”
in Proc. IEEE Intell. Veh. Symp., 2018, pp. 156–161.
[81] P. Arnau-González, S. Katsigiannis, M. Arevalillo-Herráez, and N.
Ramzan, “Image-evoked affect and its impact on EEG-based biometrics,”
in Proc. IEEE Int. Conf. Image Process., 2019, pp. 2591–2595.
[82] V. Govindarajan, K. Driggs-Campbell, and R. Bajcsy, “Affective driver
state monitoring for personalized, adaptive ADAS,” in Proc. 21st Int.
Conf. Intell. Transp. Syst., 2018, pp. 1017–1022.
[83] M. Alex, U. Tariq, F. Al-Shargie, H. S. Mir, and H. Al Nashash, “Discrim-
ination of genuine and acted emotional expressions using EEG signal and
machine learning,” IEEE Access, vol. 8, pp. 191080–191089, Oct. 2020.
[84] V. Gupta, M. D. Chopda, and R. B. Pachori, “Cross-subject emotion
recognition using flexible analytic wavelet transform from EEG signals,”
IEEE Sensors J., vol. 19, no. 6, pp. 2266–2274, Mar. 2019.
[85] K. Guo, H. Yu, R. Chai, H. Nguyen, and S. W. Su, “A hybrid physiological
approach of emotional reaction detection using combined FCM and SVM
classifier,” in Proc. 41st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2019,
pp. 7088–7091.
[86] M. Mohammadpour, S. M. R. Hashemi, and N. Houshmand, “Classifica-
tion of EEG-based emotion for BCI applications,” in Proc. Artif. Intell.
Robot., 2017, pp. 127–131.
[87] K. Guo, H. Candra, H. Yu, H. Li, H. T. Nguyen, and S. W. Su, “EEG-based
emotion classification using innovative features and combined SVM and
HMM classifier,” in Proc. 39th Annu. Int. Conf. IEEE Eng. Med. Biol.
Soc., 2017, pp. 489–492.
[88] Z. Mohammadi, J. Frounchi, and M. Amiri, “Wavelet-based emotion
recognition system using EEG signal,” Neural Comput. Appl., vol. 28,
no. 8, pp. 1985–1990, 2017.
[89] H. Candra, M. Yuwono, R. Chai, H. T. Nguyen, and S. Su, “EEG emotion
recognition using reduced channel wavelet entropy and average wavelet
coefficient features with normal mutual information method,” in Proc.
39th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2017, pp. 463–466.
[90] X. Li, D. Song, P. Zhang, G. Yu, Y. Hou, and B. Hu, “Emotion recognition
from multi-channel EEG data through convolutional recurrent neural
network,” in Proc. IEEE Int. Conf. Bioinf. Biomed., 2016, pp. 352–359.
[91] M. Ali, A. H. Mosa, F. Al Machot, and K. Kyamakya, “EEG-based
emotion recognition approach for e-healthcare applications,” in Proc.
8th Int. Conf. Ubiquitous Future Netw., 2016, pp. 946–950.
[92] N. Jatupaiboon, S. Pan-Ngum, and P. Israsena, “Subject-dependent and
subject-independent emotion classification using unimodal and multi-
modal physiological signals,” J. Med. Imag. Health Inform., vol. 5, no. 5,
pp. 1020–1027, 2015.
[93] N. Jatupaiboon, S. Pan-ngum, and P. Israsena, “Emotion classification
using minimal EEG channels and frequency bands,” in Proc. 10th Int.
Joint Conf. Comput. Sci. Softw. Eng., 2013, pp. 21–24.
[94] A. E. Vijayan, D. Sen, and A. Sudheer, “EEG-based emotion recognition
using statistical measures and auto-regressive modeling,” in Proc. IEEE
Int. Conf. Comput. Intell. Commun. Technol., 2015, pp. 587–591.
[95] H. Xu and K. N. Plataniotis, “Affect recognition using EEG signal,”
in Proc. IEEE 14th Int. Workshop Multimedia Signal Process., 2012,
pp. 299–304.
[96] S. A. Hosseini and M. B. Naghibi-Sistani, “Emotion recognition method
using entropy analysis of EEG signals,” Int. J. Image, Graph. Signal
Process., vol. 3, no. 5, pp. 30–36, 2011.
[97] M. Murugappan, M. R. B. M. Juhari, R. Nagarajan, and S. Yaacob, “An
investigation on visual and audiovisual stimulus based emotion recogni-
tion using EEG,” Int. J. Med. Eng. Inform., vol. 1, no. 3, pp. 342–356,
2009.
[98] S. Issa, Q. Peng, and X. You, “Emotion classification using EEG brain
signals and the broad learning system,” IEEE Trans. Syst., Man Cybern.
Syst., to be published, doi: 10.1109/TSMC.2020.2969686.
[99] S. K. Khare, V. Bajaj, and G. Sinha, “Adaptive tunable Q wavelet
transform-based emotion identification,” IEEE Trans. Instrum. Meas.,
vol. 69, no. 12, pp. 9609–9617, Dec. 2020.
[100] H.-C. Yang and C.-C. Lee, “An attribute-invariant variational learning
for emotion recognition using physiology,” in Proc. ICASSP IEEE Int.
Conf. Acoust., Speech Signal Process., 2019, pp. 1184–1188.
[101] H. Chao and L. Dong, “Emotion recognition using three-dimensional
feature and convolutional neural network from multichannel
EEG signals,” IEEE Sensors J., vol. 21, no. 2, pp. 2024–2034,
Jan. 2021.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
[102] Z. Gao, X. Wang, Y. Yang, Y. Li, K. Ma, and G. Chen, “A channel-
fused dense convolutional network for EEG-based emotion recog-
nition,” IEEE Trans. Cogn. Develop. Syst., to be published, doi:
10.1109/TCDS.2020.2976112.
[103] T. Matlovič, “Emotion detection using EPOC EEG device,” in Proc.
Student Res. Conf. Inform. Inf. Technol., 2016, pp. 1–6.
[104] A. T. Sohaib, S. Qureshi, J. Hagelbäck, O. Hilborn, and P. Jerčić, “Eval-
uating classifiers for emotion recognition using EEG,” in Proc. Int. Conf.
Augmented Cogn., 2013, pp. 492–501.
[105] G. Chanel, C. Rebetez, M. Bétrancourt, and T. Pun, “Emotion assessment
from physiological signals for adaptation of game difficulty,” IEEE Trans.
Syst., Man Cybern.-Part A: Syst. Humans, vol. 41, no. 6, pp. 1052–1063,
Nov. 2011.
[106] A. Yazdani, J.-S. Lee, and T. Ebrahimi, “Implicit emotional tagging of
multimedia using EEG signals and brain computer interface,” in Proc.
1st SIGMM Workshop Social Media, 2009, pp. 81–88.
[107] J. Huang, X. Xu, and T. Zhang, “Emotion classification using deep
neural networks and emotional patches,” in Proc. IEEE Int. Conf. Bioinf.
Biomed., 2017, pp. 958–962.
[108] J. Chen, B. Hu, P. Moore, X. Zhang, and X. Ma, “Electroencephalogram-
based emotion assessment system using ontology and data mining tech-
niques,” Appl. Soft Comput., vol. 30, pp. 663–674, 2015.
[109] Y. Liu and O. Sourina, “Real-time subject-dependent EEG-based emo-
tion recognition algorithm,” in Proc. Trans. Comput. Sci. XXIII, 2014,
pp. 199–223.
[110] Y. Liu and O. Sourina, “EEG-based valence level recognition for real-time
applications,” in Proc. Int. Conf. Cyberworlds, 2012, pp. 53–60.
[111] T.-Y. Chai, S. Woo, M. Rizon, and C. Tan, “Classification of human
emotions from EEG signals using statistical features and neural network,”
Int. J. Integr. Eng., vol. 1, pp. 71–79, 2010.
[112] Z. Khalili and M. H. Moradi, “Emotion recognition system using brain
and peripheral signals: Using correlation dimension to improve the results
of EEG,” in Proc. Int. Joint Conf. Neural Netw., 2009, pp. 1571–1575.
[113] W. Zheng and B. Lu, “Investigating critical frequency bands and chan-
nels for EEG-based emotion recognition with deep neural networks,”
IEEE Trans. Auton. Mental Develop., vol. 7, no. 3, pp. 162–175,
Sep. 2015.
[114] R.-N. Duan, J.-Y. Zhu, and B.-L. Lu, “Differential entropy feature for
EEG-based emotion classification,” in Proc. 6th Int. IEEE/EMBS Conf.
Neural Eng., 2013, pp. 81–84.
[115] Y.-P. Lin, C.-H. Wang, T.-L. Wu, S.-K. Jeng, and J.-H. Chen, “EEG-based
emotion recognition in music listening: A comparison of schemes for
multiclass support vector machine,” in Proc. IEEE Int. Conf. Acoust.,
Speech Signal Process., 2009, pp. 489–492.
[116] A. D. Bigirimana, N. Siddique, and D. Coyle, “Emotion-inducing im-
agery versus motor imagery for a brain-computer interface,” IEEE Trans.
Neural Syst. Rehabil. Eng., vol. 28, no. 4, pp. 850–859, Apr. 2020.
[117] K. Wang, Y. Ho, Y. Huang, and W. Fang, “Design of intelligent EEG sys-
tem for human emotion recognition with convolutional neural network,”
in Proc. IEEE Int. Conf. Artif. Intell. Circuits Syst., 2019, pp. 142–145.
[118] A. Jalilifard, E. B. Pizzolato, and M. K. Islam, “Emotion classification
using single-channel scalp-EEG recording,” in Proc. 38th Annu. Int. Conf.
IEEE Eng. Med. Biol. Soc., 2016, pp. 845–849.
[119] T. Zhang, W. Zheng, Z. Cui, Y. Zong, and Y. Li, “Spatial-temporal
recurrent neural network for emotion recognition,” IEEE Trans. Cybern.,
vol. 49, no. 3, pp. 839–847, Mar. 2019.
[120] G. Zhao, Y. Ge, B. Shen, X. Wei, and H. Wang, “Emotion analysis for
personality inference from EEG signals,” IEEE Trans. Affect. Comput.,
vol. 9, no. 3, pp. 362–371, Jul.-Sep. 2018.
[121] J. Teo, C. L. Hou, and J. Mountstephens, “Deep learning for EEG-based
preference classification,” AIP Conf. Proc., vol. 1891, no. 1, 2017,
Art. no. 020141.
[122] P. Ackermann, C. Kohlschein, J. Á. Bitsch, K. Wehrle, and S. Jeschke,
“EEG-based automatic emotion recognition: Feature extraction, selection
and classification methods,” in Proc. IEEE 18th Int. Conf. Netw., Appl.
Serv., 2016, pp. 1–6.
[123] S. K. Hadjidimitriou and L. J. Hadjileontiadis, “Toward an EEG-based
recognition of music liking using time-frequency analysis,” IEEE Trans.
Biomed. Eng., vol. 59, no. 12, pp. 3498–3510, Dec. 2012.
[124] G. Chanel, J. J. Kierkels, M. Soleymani, and T. Pun, “Short-term emotion
assessment in a recall paradigm,” Int. J. Hum.-Comput. Stud., vol. 67,
no. 8, pp. 607–627, 2009.
[125] H. Huang, Z. Hu, W. Wang, and M. Wu, “Multimodal emotion recognition
based on ensemble convolutional neural network,” IEEE Access, vol. 8,
pp. 3265–3271, Dec. 2019.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
437
[126] Z. Gao, Y. Li, Y. Yang, N. Dong, X. Yang, and C. Grebogi, “A
coincidence-filtering-based approach for CNNs in EEG-based recogni-
tion,” IEEE Trans. Ind. Inform., vol. 16, no. 11, pp. 7159–7167, Nov.
2020.
[127] Y. Wang, Z. Huang, B. McCane, and P. Neo, “EmotioNet: A 3-D con-
volutional neural network for EEG-based emotion recognition,” in Proc.
Int. Joint Conf. Neural Netw., 2018, pp. 1–7.
[128] S. Alhagry, A. Aly, and R. El-Khoribi, “Emotion recognition based on
EEG using LSTM recurrent neural network,” Int. J. Adv. Comput. Sci.
Appl., vol. 8, no. 10, pp. 355–358, 2017.
[129] M. Yanagimoto and C. Sugimoto, “Recognition of persisting emotional
valence from EEG using convolutional neural networks,” in Proc. IEEE
9th Int. Workshop Comput. Intell. Appl., 2016, pp. 27–32.
[130] Y. Gao, H. J. Lee, and R. M. Mehmood, “Deep learninig of EEG signals
for emotion recognition,” in Proc. IEEE Int. Conf. Multimedia Expo
Workshops, 2015, pp. 1–5.
[131] E. S. Salama, R. A. El-Khoribi, M. E. Shoman, and M. A. W. Shal-
aby, “EEG-based emotion recognition using 3D convolutional neural
networks,” Int. J. Adv. Comput. Sci. Appl., vol. 9, no. 8, pp. 329–337,
2018.
[132] N. Liu, Y. Fang, L. Li, L. Hou, F. Yang, and Y. Guo, “Multiple feature
fusion for automatic emotion recognition using EEG signals,” in Proc.
IEEE Int. Conf. Acoust., Speech Signal Process., 2018, pp. 896–900.
[133] N. Kumar, K. Khaund, and S. M. Hazarika, “Bispectral analysis of EEG
for emotion recognition,” Procedia Comput. Sci., vol. 84, pp. 31–35,
2016.
[134] X. Jie, R. Cao, and L. Li, “Emotion recognition based on the sample
entropy of EEG,” Bio-Med. Mater. Eng., vol. 24, no. 1, pp. 1185–1192,
2014.
[135] P. C. Petrantonakis and L. J. Hadjileontiadis, “Emotion recognition from
EEG using higher order crossings,” IEEE Trans. Inf. Technol. Biomed.,
vol. 14, no. 2, pp. 186–197, Mar. 2010.
[136] P. C. Petrantonakis and L. J. Hadjileontiadis, “Emotion recognition from
brain signals using hybrid adaptive filtering and higher order crossings
analysis,” IEEE Trans. Affect. Comput., vol. 1, no. 2, pp. 81–97, Jul.–Dec.
2010.
[137] X. Wang, T. Zhang, X. Xu, L. Chen, X. Xing, and C. L. P. Chen,
“EEG emotion recognition using dynamical graph convolutional neural
networks and broad learning system,” in Proc. IEEE Int. Conf. Bioinf.
Biomed., 2018, pp. 1240–1244.
[138] J. Pan, Y. Li, and J. Wang, “An EEG-based brain-computer interface
for emotion recognition,” in Proc. Int. Joint Conf. Neural Netw., 2016,
pp. 2063–2067.
[139] S. Makeig, G. Leslie, T. Mullen, D. Sarma, N. Bigdely-Shamlo, and C.
Kothe, “First demonstration of a musical emotion BCI,” in Proc. Int.
Conf. Affect. Comput. Intell. Interact., 2011, pp. 487–496.
[140] M. Li and B.-L. Lu, “Emotion classification based on gamma-band EEG,”
in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2009, pp. 1223–1226.
[141] D. Huang, C. Guan, K. K. Ang, H. Zhang, and Y. Pan, “Asymmetric
spatial pattern for EEG-based emotion detection,” in Proc. Int. Joint Conf.
Neural Netw., 2012, pp. 1–7.
[142] S. A. Hosseini and M. A. Khalilzadeh, “Emotional stress recognition
system using EEG and psychophysiological signals: Using new labelling
process of EEG signals in emotional stress state,” in Proc. Int. Conf.
Biomed. Eng. Comput. Sci., 2010, pp. 1–6.
[143] R. M. Mehmood, R. Du, and H. J. Lee, “Optimal feature selection and
deep learning ensembles method for emotion recognition from human
brain EEG sensors,” IEEE Access, vol. 5, pp. 14797–14806, 2017.
[144] H. Shahabi and S. Moghimi, “Toward automatic detection of brain
responses to emotional music through analysis of EEG effective con-
nectivity,” Comput. Hum. Behav., vol. 58, pp. 231–239, 2016.
[145] S. Koelstra and I. Patras, “Fusion of facial expressions and EEG for im-
plicit affective tagging,” Image Vis. Comput., vol. 31, no. 2, pp. 164–174,
2013.
[146] F. Abtahi, T. Ro, W. Li, and Z. Zhu, “Emotion analysis using audio/video,
EMG and EEG: A dataset and comparison study,” in Proc. IEEE Winter
Conf. Appl. Comput. Vis., 2018, pp. 10–19.
[147] T. Zhang, X. Wang, X. Xu, and C. L. P. Chen, “GCB-Net:
Graph convolutional broad network and its application in emotion
recognition,” IEEE Trans. Affect. Comput., to be published, doi:
10.1109/TAFFC.2019.2937768.
[148] J. Jiang, Y. Zeng, L. Tong, C. Zhang, and B. Yan, “Single-trial
ERP detecting for emotion recognition,” in Proc. 17th IEEE/ACIS Int.
Conf. Softw. Eng., Artif. Intell., Netw. Parallel/Distrib. Comput., 2016,
pp. 105–108.
438 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
[149] S. K. Khare and V. Bajaj, “Time-frequency representation and convo-
lutional neural network-based emotion recognition,” IEEE Trans. Neu-
ral Netw. Learn. Syst., vol. 32, no. 7, pp. 2901–2909, Jul. 2021, doi:
10.1109/TNNLS.2020.3008938.
[150] S. Katsigiannis and N. Ramzan, “DREAMER: A database for emo-
tion recognition through EEG and ECG signals from wireless low-cost
off-the-shelf devices,” IEEE J. Biomed. Health Inform., vol. 22, no. 1,
pp. 98–107, Jan. 2018.
[151] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of
EEG signals and facial expressions for continuous emotion detection,”
IEEE Trans. Affect. Comput., vol. 7, no. 1, pp. 17–28, Jan.–Mar. 2016.
[152] S. Vanhatalo, J. M. Palva, M. Holmes, J. Miller, J. Voipio, and K. Kaila,
“Infraslow oscillations modulate excitability and interictal epileptic ac-
tivity in the human cortex during sleep,” Proc. Nat. Acad. Sci., vol. 101,
no. 14, pp. 5053–5057, 2004.
[153] P. Fries, “A mechanism for cognitive dynamics: Neuronal communi-
cation through neuronal coherence,” Trends Cogn. Sci., vol. 9, no. 10,
pp. 474–480, 2005.
[154] G. Knyazev et al., “Cross-frequency coupling in developmental perspec-
tive,” Front. Hum. Neurosci., vol. 13, no. 158, pp. 1–10, 2019.
[155] A. D. Marimpis, S. I. Dimitriadis, and R. Goebel, “A multiplex connec-
tivity map of valence-arousal emotional model,” IEEE Access, vol. 8,
pp. 170928–170938, 2020.
[156] H. A. Gonzalez et al., “Ultra-high compression of twiddle factor ROMs
in multi-core DSP for FMCW radars,” in Proc. IEEE Int. Symp. Circuits
Syst. (ISCAS), 2021, pp. 1–5, doi: 10.1109/ISCAS51556.2021.9401547.
[157] P. Davis, C. D. Creusere, and W. Tang, “ASIC implementation of the
cross frequency coupling algorithm for EEG signal processing,” in Proc.
Int. Symp. Integr. Circuits, 2014, pp. 248–251.
[158] B. Hjorth, “EEG analysis based on time domain properties,” Electroen-
cephalogr. Clin. Neuriophysiol., vol. 29, no. 3, pp. 306–310, 1970.
[159] J. S. Richman and J. R. Moorman, “Physiological time-series analysis
using approximate entropy and sample entropy,” Amer. J. Physiol.-Heart
Circulatory Physiol., vol. 278, no. 6, pp. H2039–H2049, 2000.
[160] Y. Hong and Y. Bao, “FPGA implementation for real-time empirical
mode decomposition,” IEEE Trans. Instrum. Meas., vol. 61, no. 12,
pp. 3175–3184, Dec. 2012.
[161] M. K. Kıymık, İ. Güler, A. Dizibüyük, and M. Akın, “Comparison of
STFT and wavelet transform methods in determining epileptic seizure
activity in EEG signals for real-time application,” Comput. Biol. Med.,
vol. 35, no. 7, pp. 603–616, 2005.
[162] C. Yu and M. Yen, “Area-efficient 128- to 2048/1536-point pipeline FFT
processor for LTE and mobile WiMAX systems,” IEEE Trans. Very Large
Scale Integration (VLSI) Syst., vol. 23, no. 9, pp. 1793–1800, Sep. 2015.
[163] H. Kang, J. Lee, and J. Kim, “Low-complexity twiddle factor generation
for FFT processor,” Electron. Lett., vol. 49, no. 23, pp. 1443–1445, 2013.
[164] M. Hasan and T. Arslan, “Scheme for reducing size of coefficient memory
in FFT processor,” Electron. Lett., vol. 38, no. 4, pp. 163–164, 2002.
[165] H. Lee and I. Park, “Balanced binary-tree decomposition for area-efficient
pipelined FFT processing,” IEEE Trans. Circuits Syst. I: Regular Papers,
vol. 54, no. 4, pp. 889–900, Apr. 2007.
[166] H. Kang, B. Yang, and J. Lee, “Low complexity twiddle factor multipli-
cation with ROM partitioning in FFT processor,” Electron. Lett., vol. 49,
no. 9, pp. 589–591, 2013.
[167] W. Tsai, S. Chen, and S. Huang, “A low-complexity mixed-radix FFT
rotator architecture,” in Proc. IEEE Asia Pacific Conf. Circuits Syst.,
2018, pp. 183–186.
[168] M. J. Kaminski and K. J. Blinowska, “A new method of the description
of the information flow in the brain structures,” Biol. Cybern., vol. 65,
no. 3, pp. 203–210, 1991.
[169] K. Singh and J. P. de Gyvez, “Twenty years of near/sub-threshold design
trends and enablement,” IEEE Trans. Circuits Syst. II: Exp. Briefs, vol. 68,
no. 1, pp. 5–11, Jan. 2021.
[170] S. Höppner et al., “Adaptive body bias aware implementation for ultra-
low-voltage designs in 22FDX technology,” IEEE Trans. Circuits Syst.
II: Exp. Briefs, vol. 67, no. 10, pp. 2159–2163, Oct. 2020.
[171] S. Höppner et al., “How to achieve world-leading energy efficiency
using 22FDX with adaptive body biasing on an arm Cortex-M4 IoT
SoC,” in Proc. ESSDERC 49th Eur. Solid-State Device Res. Conf., 2019,
pp. 66–69.
[172] R. Carter et al., “22 nm FDSOI technology for emerging mobile, Internet-
of-Things, and RF applications,” in Proc. IEEE Int. Electron. Devices
Meeting, 2016, pp. 2.2.1–2.2.4.
[173] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment
manikin and the semantic differential,” J. Behav. Ther. Exp. Psychiatry,
vol. 25, no. 1, pp. 49–59, 1994.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
[174] X. Du et al., “An efficient LSTM network for emotion recognition from
multichannel EEG signals,” IEEE Trans. Affect. Comput., to be published,
doi: 10.1109/TAFFC.2020.3013711.
[175] L.-Y. Tao and B.-L. Lu, “Emotion recognition under sleep deprivation
using a multimodal residual LSTM network,” in Proc. Int. Joint Conf.
Neural Netw., 2020, pp. 1–8.
[176] Y. Yang, Q. J. Wu, W.-L. Zheng, and B.-L. Lu, “EEG-based emotion
recognition using hierarchical network with subnetwork nodes,” IEEE
Trans. Cogn. Develop. Syst., vol. 10, no. 2, pp. 408–419, Jun. 2018.
[177] X. Li, B. Qian, J. Wei, A. Li, X. Liu, and Q. Zheng, “Classify EEG
and reveal latent graph structure with spatio-temporal graph convolu-
tional neural network,” in Proc. IEEE Int. Conf. Data Mining, 2019,
pp. 389–398.
[178] S. Sheykhivand, Z. Mousavi, T. Y. Rezaii, and A. Farzamnia, “Recog-
nizing emotions evoked by music using CNN-LSTM networks on EEG
signals,” IEEE Access, vol. 8, pp. 139332–139345, 2020.
[179] Y. Yang, Q. Wu, M. Qiu, Y. Wang, and X. Chen, “Emotion recognition
from multi-channel EEG through parallel convolutional recurrent neural
network,” in Proc. Int. Joint Conf. Neural Netw., 2018, pp. 1–7.
[180] W. Zheng, “Multichannel EEG-based emotion recognition via group
sparse canonical correlation analysis,” IEEE Trans. Cogn. Develop. Syst.,
vol. 9, no. 3, pp. 281–290, Sep. 2017.
[181] J. Cheng et al., “Emotion recognition from multi-channel EEG via deep
forest,” IEEE J. Biomed. Health Inform., vol. 25, no. 2, pp. 453–464, Feb.
2021.
[182] Y. Luo and B.-L. Lu, “EEG data augmentation for emotion recognition
using a conditional wasserstein GAN,” in Proc. 40th Annu. Int. Conf.
IEEE Eng. Med. Biol. Soc., 2018, pp. 2535–2538.
[183] J. Chen, D. Jiang, and Y. Zhang, “A hierarchical bidirectional GRU model
with attention for EEG-based emotion classification,” IEEE Access,
vol. 7, pp. 118530–118540, 2019.
[184] R. Al-Fahad and M. Yeasin, “Micro-states based dynamic brain con-
nectivity in understanding the commonality and differences in gender-
specific emotion processing,” in Proc. Int. Joint Conf. Neural Netw., 2019,
pp. 1–8.
[185] W. Huang, Y. Xue, L. Hu, and H. Liuli, “S-EEGNet: Electroen-
cephalogram signal classification based on a separable convolution
neural network with bilinear interpolation,” IEEE Access, vol. 8,
pp. 131636–131646, 2020.
[186] Y. Luo et al., “EEG-based emotion classification using spiking neural
networks,” IEEE Access, vol. 8, pp. 46007–46016, 2020.
[187] S. Liu, X. Wang, L. Zhao, J. Zhao, Q. Xin, and S. Wang, “Subject-
independent emotion recognition of EEG signals based on dynamic
empirical convolutional neural network,” IEEE/ACM Trans. Comput.
Biol. Bioinf., to be published, doi: 10.1109/TCBB.2020.3018137.
[188] Y. Zhao, J. Yang, J. Lin, D. Yu, and X. Cao, “A 3D convolutional neural
network for emotion recognition based on EEG signals,” in Proc. Int.
Joint Conf. Neural Netw., 2020, pp. 1–6.
[189] X. Jia et al., “Multi-channel EEG based emotion recognition using
temporal convolutional network and broad learning system,” in Proc.
IEEE Int. Conf. Syst., Man, Cybern., 2020, pp. 2452–2457.
[190] L. Shen, W. Zhao, Y. Shi, T. Qin, and B. Liu, “Parallel sequence-channel
projection convolutional neural network for EEG-based emotion recog-
nition,” IEEE Access, vol. 8, pp. 222966–222976, 2020.
[191] K.-J. Wang and C. Y. Zheng, “Toward a wearable affective robot that de-
tects human emotions from brain signals by using deep multi-spectrogram
convolutional neural networks (deep MS-CNN),” in Proc. 28th IEEE Int.
Conf. Robot Hum. Interactive Commun., 2019, pp. 1–8.
[192] B. Krisnandhika, A. Faqih, P. D. Pumamasari, and B. Kusumoputro,
“Emotion recognition system based on EEG signals using relative wavelet
energy features and a modified radial basis function neural networks,” in
Proc. Int. Conf. Consum. Electron. Devices, 2017, pp. 50–54.
[193] G. Li, Z. Zhang, and G. Wang, “Emotion recognition based on low-
cost in-ear EEG,” in Proc. IEEE Biomed. Circuits Syst. Conf., 2017,
pp. 1–4.
[194] Y. Kumagai, M. Arvaneh, H. Okawa, T. Wada, and T. Tanaka, “Classi-
fication of familiarity based on cross-correlation features between EEG
and music,” in Proc. 39th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.,
2017, pp. 2879–2882.
[195] S. Liu et al., “EEG-based emotion estimation using adaptive tracking
of discriminative frequency components,” in Proc. 39th Annu. Int. Conf.
IEEE Eng. Med. Biol. Soc., 2017, pp. 2231–2234.
[196] B. Hu, X. Li, S. Sun, and M. Ratcliffe, “Attention recognition
in EEG-based affective learning research using CFS KNN algo-
rithm,” IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 15, no. 1,
pp. 38–45, Jan./Feb. 2018.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
[197] Cecbur, “CC-BY-SA-4.0,” Feb. 2019. [Online]. Available: https:
//commons.wikimedia.org/wiki/File:3_filters_in_a_Convolutional_
Neural_Network.gif
[198] Y. Yan et al., “Low-power low-latency keyword spotting and adaptive
control with a spinnaker 2 prototype and comparison with loihi,” 2020,
arXiv:2009.08921.
[199] Y. Yan et al., “Efficient reward-based structural plasticity on a SpiN-
Naker 2 prototype,” IEEE Trans. Biomed. Circuits Syst., vol. 13, no. 3,
pp. 579–591, Jun. 2019.
[200] C. Liu et al., “Memory-efficient deep learning on a SpiNNaker 2 proto-
type,” Front. Neurosci., vol. 12, no. 840, pp. 1–15, 2018.
[201] C. Mayr, S. Hoeppner, and S. Furber, “SpiNNaker 2: A 10 million
core processor system for brain simulation and machine learning,” 2019,
arXiv:1911.02385.
[202] M. Jobst et al., “Event-based neural network for ECG classification with
delta encoding and early stopping,” in Proc. 6th Int. Conf. Event-Based
Control, Commun., Signal Process. (EBCCSP), 2020, pp. 1–4, doi:
10.1109/EBCCSP51266.2020.9291357.
[203] Fdeloche, “CC-BY-SA-4.0,” Jun. 2017. [Online]. Available: https://
commons.wikimedia.org/wiki/File:Gated_Recurrent_Unit.svg
[204] Long Short-Term Memory, “CC-BY-SA-4.0,” Ixnay, Jun. 2017. [On-
line]. Available: https://commons.wikimedia.org/wiki/File:Long_Short-
Term_Memory.svg
[205] K. Cho et al., “Learning phrase representations using RNN encoder-
decoder for statistical machine translation,” 2014, arXiv:1406.1078.
[206] B. M. Baas, “A low-power, high-performance, 1024-point FFT pro-
cessor,” IEEE J. Solid-State Circuits, vol. 34, no. 3, pp. 380–387,
Mar. 1999.
[207] J. Yoo et al., “An 8-channel scalable EEG acquisition SoC with
fully integrated patient-specific seizure classification and recording
processor,” in Proc. IEEE Int. Solid-State Circuits Conf., 2012,
pp. 292–294.
[208] Y.-X. Wang, D. Ramanan, and M. Hebert, “Growing a brain: Fine-tuning
by increasing model capacity,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2017, pp. 3029–3038.
[209] C.-S. Wei, T. Koike-Akino, and Y. Wang, “Spatial component-wise
convolutional network (SCCNet) for motor-imagery EEG classification,”
in Proc. 9th Int. IEEE/EMBS Conf. Neural Eng., 2019, pp. 328–331.
[210] H. Wu et al., “A parallel multiscale filter bank convolutional neural
networks for motor imagery EEG classification,” Front. Neurosci., vol.
13, no. 1275, pp. 1–9, 2019.
[211] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans.
Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
[212] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic
intelligence,” in Proc. 34th Int. Conf. Mach. Learn., Sydney, Australia,
Aug. 2017, pp. 3987–3995.
[213] G. Zeng, Y. Chen, B. Cui, and S. Yu, “Continual learning of context-
dependent processing in neural networks,” Nat. Mach. Intell., vol. 1,
no. 8, pp. 364–372, 2019.
[214] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual
lifelong learning with neural networks: A review,” Neural Netw., vol. 113,
pp. 54–71, 2019.
[215] J. Gideon, S. Khorram, Z. Aldeneh, D. Dimitriadis, and E. M. Provost,
“Progressive neural networks for transfer learning in emotion recogni-
tion,” in Proc. INTERSPEECH, 2017, pp. 1098–11102.
[216] J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural
networks,” Proc Nat. Acad. Sci., vol. 114, no. 13, pp. 3521–3526,
2017.
[217] R. Kemker, M. McClure, A. Abitino, T. L. Hayes, and C. Kanan, “Mea-
suring catastrophic forgetting in neural networks,” in Proc. AAAI Conf.
Artif. Intell., 2018, pp. 3390–3398.
[218] R. R. Karn, P. Kudva, and I. M. Elfadel, “Criteria for learning without
forgetting in artificial neural networks,” in Proc. IEEE Int. Conf. Cogn.
Comput., 2019, pp. 90–97.
[219] International 10-20 system for EEG-MCN, “CC0 1.0,” B. C. Oxley,
2017. [Online]. Available: https://commons.wikimedia.org/wiki/File:
International_10-20_system_for_EEG-MCN.svg
[220] T. Tang et al., “An active concentric electrode for concurrent EEG
recording and body-coupled communication (BCC) data transmission,”
IEEE Trans. Biomed. Circuits Syst., vol. 14, no. 6, pp. 1253–1262, Dec.
2020.
[221] K. Zhao and D. Xu, “Food image-induced discrete emotion recognition
using a single-channel scalp-EEG recording,” in Proc. 12th Int. Congr.
Image Signal Process., Biomed. Eng. Inform., 2019, pp. 1–6.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
439
[222] C. Y. Park et al., “K-EmoCon, a multimodal sensor dataset for continuous
emotion recognition in naturalistic conversations,” May 2020. [Online].
Available: https://doi.org/10.5281/zenodo.3814370
[223] T. B. Alakus, M. Gonen, and I. Turkoglu, “Database for an emo-
tion recognition system based on EEG signals and various com-
puter games - Gameemo,” Biomed. Signal Process. Control, vol. 60,
2020, Art. no. 101951.
[224] J. Onton, “Imagined emotion study,” Jul. 2020. [Online]. Available: https:
//openneuro.org/datasets/ds003004/versions/1.0.0
[225] J. A. M. Correa, M. K. Abadi, N. Sebe, and I. Patras, “AMIGOS: A dataset
for affect, personality and mood research on individuals and groups,”
IEEE Trans. Affect. Comput., vol. 12, no. 2, pp. 479–493, Apr.–Jun. 2021.
[226] S. B. Eysenck, H. J. Eysenck, and P. Barrett, “A revised version of the
psychoticism scale,” Pers. Individual Differences, vol. 6, no. 1, pp. 21–29,
1985.
[227] J. W. Papez, “A proposed mechanism of emotion,” Arch. Neurol. Psychi-
atry, vol. 38, no. 4, pp. 725–743, Oct. 1937.
[228] C. Darwin, The Expression of Emotions in Man and Animals. London,
U.K.: John Murray, 1872.
[229] J. Posner, J. A. Russel, and B. S. Peterson, “The circumplex model
of affect: An integrative approach to affective neuroscience, cog-
nitive development, and psychopathology,” Develop. Psychopathol.,
vol. 17, no. 3, Sep. 2005. [Online]. Available: https://doi.org/10.1017/
s0954579405050340
[230] K. Phan, T. Wager, S. F. Taylor, and I. Liberzon, “Functional neu-
roanatomy of emotion: A meta-analysis of emotion activation studies
in pet and fMRI,” NeuroImage, vol. 16, no. 2, pp. 331–348, 2002.
[231] J. Gläscher and R. Adolphs, “Processing of the arousal of subliminal and
supraliminal emotional stimuli by the human amygdala,” J. Neurosci.,
vol. 23, no. 32, pp. 10 274–10 282, 2003.
[232] D. M. Small, M. D. Gregory, Y. Mak, D. Gitelman, M. Mesulam, and
T. Parrish, “Dissociation of neural representation of intensity and affec-
tive valuation in human gustation,” Neuron, vol. 39, no. 4, pp. 701–711,
Aug. 2003.
[233] K. H. Wood, L. W. Ver Hoef, and D. C. Knight, “The amygdala mediates
the emotional modulation of threat-elicited skin conductance response,”
Emotion, vol. 14, no. 4, pp. 693–700, 2014.
[234] R. R. Fletcher et al., “iCalm: Wearable sensor and network archi-
tecture for wirelessly communicating and logging autonomic activ-
ity,” IEEE Trans. Inf. Technol. Biomed., vol. 14, no. 2, pp. 215–223,
Mar. 2010.
[235] P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, “Looking
at pictures: Affective, facial, visceral, and behavioral reactions,” Psy-
chophysiology, vol. 30, no. 3, pp. 261–273, May 1993.
[236] I. B. Mauss and M. D. Robinson, “Measures of emotion: A review,” Cogn.
Emotion, vol. 23, no. 2, pp. 209–237, Feb. 2009.
[237] G. Tamburro, D. B. Stone, and S. Comani, “Automatic removal of cardiac
interference (ARCI): A new approach for EEG data,” Front. Neurosci.,
vol. 13, no. 441, pp. 1–17, 2019.
[238] P. Bušek and D. Kemlink, “The influence of the respiratory cycle on the
EEG,” Physiol. Res., vol. 54, pp. 327–33, 2005.
[239] P. J. D. Jong and H. Merckelbach, “Eyeblink frequency, rehearsal activity,
and sympathetic arousal,” Int. J. Neurosci., vol. 51, no. 1/2, pp. 89–94,
Jan. 1990.
[240] M. Poh, N. C. Swenson, and R. W. Picard, “A wearable sensor for un-
obtrusive, long-term assessment of electrodermal activity,” IEEE Trans.
Biomed. Eng., vol. 57, no. 5, pp. 1243–1252, May 2010.
[241] K. Arikan et al., “EEG correlates of startle reflex with reactivity to
eye opening in psychiatric disorders: Preliminary results,” Clin. EEG
Neurosci., vol. 37, no. 3, pp. 230–234, 2006.
[242] G. Panayiotou and E. Constantinou, “Emotion dysregulation in alex-
ithymia: Startle reactivity to fearful affective imagery and its relation to
heart rate variability,” Psychophysiology, vol. 54, no. 9, pp. 1323–1334,
2017.
[243] D. C. Gooding, R. J. Davidson, K. M. Putnam, and K. A. Tallent,
“Normative emotion-modulated startle response in individuals at risk for
schizophrenia-spectrum disorders,” Schizophrenia Res., vol. 57, no. 1,
pp. 109–120, 2002.
[244] V. De Pascalis, G. Cozzuto, G. V. Caprara, and G. Alessandri, “Relations
among EEG-alpha asymmetry, BIS/BAS, and dispositional optimism,”
Biol. Psychol., vol. 94, no. 1, pp. 198–209, 2013.
[245] J. A. Coan and J. J. Allen, “Frontal EEG asymmetry and the behavioral
activation and inhibition systems,” Psychophysiology, vol. 40, no. 1,
pp. 106–114, 2003.
440 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
[246] R. J. Davidson, “Affective neuroscience and psychophysiology: To-
ward a synthesis,” Psychophysiology, vol. 40, no. 5, pp. 655–665,
Sep. 2003.
[247] S. Clerc et al., “8.4a0.33v/ − 40◦ c process/temperature closed-loop
compensation SoC embedding all-digital clock multiplier and dc-dc
converter exploiting FDSOI 28 nm back-gate biasing,” in Proc. IEEE
Int. Solid-State Circuits Conf. Dig. Tech. Papers, 2015, pp. 1–3.
[248] D. Bol et al., “19.6 A 40-to-80 MHz sub-4 µW/MHz ULV Cortex-M0
MCU SoC in 28 nm FDSOI with dual-loop adaptive back-bias generator
for 20 µs wake-up from deep fully retentive sleep mode,” in Proc. IEEE
Int. Solid- State Circuits Conf., 2019, pp. 322–324.
[249] M. Pons et al., “A 0.5 V 2.5 µW/MHz microcontroller with analog-
assisted adaptive body bias PVT compensation with 3.13nW/kB SRAM
retention in 55 nm deeply-depleted channel CMOS,” in Proc. IEEE
Custom Integr. Circuits Conf., 2019, pp. 1–4.
[250] F. u. Rahman, R. Pamula, A. Boora, X. Sun, and V. Sathe, “19.1
computationally enabled total energy minimization under performance
requirements for a voltage-regulated 0.38-to-0.58V microprocessor in
65 nm CMOS,” in Proc. IEEE Int. Solid- State Circuits Conf., 2019,
pp. 312–314.
[251] J. Lee et al., “19.2 a 6.4pJ/Cycle self-tuning Cortex-M0 IoT processor
based on leakage-ratio measurement for energy-optimal operation across
wide-range PVT variation,” in Proc. IEEE Int. Solid- State Circuits Conf.,
2019, pp. 314–315.
[252] Y. Pu, J. P. de Gyvez, H. Corporaal, and Y. Ha, “An ultra-low-energy
multi-standard JPEG co-processor in 65 nm CMOS with sub/near
threshold supply voltage,” IEEE J. Solid-State Circuits, vol. 45, no. 3,
pp. 668–680, Mar. 2010.
[253] H. Mostafa, M. Anis, and M. Elmasry, “A novel low area overhead
direct adaptive body bias (D-ABB) circuit for die-to-die and within-die
variations compensation,” IEEE Trans. Very Large Scale Integr. (VLSI)
Syst., vol. 19, no. 10, pp. 1848–1860, Oct. 2011.
[254] N. Kamae, A. K. M. M. Islam, A. Tsuchiya, and H. Onodera, “A body
bias generator with wide supply-range down to threshold voltage for
within-die variability compensation,” in Proc. IEEE Asian Solid-State
Circuits Conf., 2014, pp. 53–56.
[255] M. Blagojevic, M. Cochet, B. Keller, P. Flatresse, A. Vladimirescu, and
B. Nikolic, “A fast, flexible, positive and negative adaptive body-bias
generator in 28 nm FDSOI,” in Proc. IEEE Symp. VLSI Circuits, 2016,
pp. 1–2.
[256] A. Quelen, G. Pillonnet, P. Flatresse, and E. Beigne, “A 2.5 µw
0.0067 mm2 automatic back-biasing compensation unit achieving 50%
leakage reduction in FDSOI 28 nm over 0.35-to-1 V VDD range,” in
Proc. IEEE Int. Solid - State Circuits Conf., 2018, pp. 304–306.
[257] M. Saligane et al., “An adaptive body-BiasIna SoC using in situ slack
monitoring for runtime replica calibration,” in Proc. IEEE Symp. VLSI
Circuits, 2018, pp. 63–64.
[258] X. Wang, M. Tehranipoor, S. George, D. Tran, and L. Winemberg, “De-
sign and analysis of a delay sensor applicable to process/environmental
variations and aging measurements,” IEEE Trans. Very Large Scale
Integration (VLSI) Syst., vol. 20, no. 8, pp. 1405–1418, Oct. 2012.
[259] A. K. M. M. Islam and H. Onodera, “On-chip monitoring and compensa-
tion scheme with fine-grain body biasing for robust and energy-efficient
operations,” in Proc. 21st Asia South Pacific Des. Automat. Conf., 2016,
pp. 403–409.
[260] D. Walter et al., “A 0.55v 6.3uW/MHz ARM Cortex-M4 MCU with
adaptive reverse body bias and single rail SRAM,” in Proc. IEEE Symp.
Low-Power High-Speed Chips, 2020, pp. 1–3.
[261] D. Fernandez-Duque and S. Black, “Impaired recognition of negative
facial emotions in patients with frontotemporal dementia,” Neuropsy-
chologia, vol. 43, pp. 1673–1687, 2005.
[262] R. Bucks and S. Radford, “Emotion processing in Alzheimer’s disease,”
Aging Ment. Health, vol. 8, pp. 222–232, 2004.
[263] R. Hargrave, R. Maddock, and V. Stone, “Impaired recognition of facial
expressions of emotion in Alzheimer’s disease,” J. Neuropsychiatry Clin.
Neurosci., vol. 14, no. 1, pp. 64–71, 2002.
[264] D. Zaitchik, E. Koff, H. Brownell, E. Winner, and M. Albert, “Inference
of beliefs and emotions in patients with Alzheimer’s disease,” Neuropsy-
chology, vol. 20, pp. 11–20, 2006.
[265] M. Downs and B. Bowers, Excellence in Dementia Care: Research
Into Practice: Principles and Practice. McGraw-Hill Education, Maid-
enhead/Berkshire England, 2008. [Online]. Available: https://books.
google.de/books?id=Bgdxu4lnTCIC
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
[266] G. Liberati et al., “Development of a binary fMRI-BCI for Alzheimer
patients: A semantic conditioning paradigm using affective uncondi-
tioned stimuli,” in Proc. Humaine Assoc. Conf. Affect. Comput. Intell.
Interaction, 2013, pp. 838–842.
[267] D. Lulé et al., “Brain responses to emotional stimuli in patients with
amyotrophic lateral sclerosis (ALS),” J. Neurol., vol. 254, pp. 519–527,
2007.
[268] R. El Kaliouby, R. Picard, and S. Baron-Cohen, “Affective computing
and autism,” Ann. New York Acad. Sci., vol. 1093, no. 1, pp. 228–248,
2006.
[269] R. W. Picard, “Future affective technology for autism and emotion
communication,” Philos. Trans. Roy. Soc. London. Ser. B., Biol. Sci.,
vol. 364, no. 1535, pp. 3575–3584, 12 2009.
[270] O. Rudovic, J. Lee, M. Dai, B. Schuller, and R. W. Picard, “Personalized
machine learning for robot perception of affect and engagement in autism
therapy,” Sci. Robot., vol. 3, no. 19, pp. 1–11, 2018.
[271] N. Yeung, R. Bogacz, C. B. Holroyd, and J. D. Cohen, “Detection of
synchronized oscillations in the electroencephalogram: An evaluation of
methods,” Psychophysiol., vol. 41, no. 6, pp. 822–832, 2004.
[272] S. Yuval-Greenberg, O. Tomer, A. S. Keren, I. Nelken, and L. Y. Deouell,
“Transient induced gamma-band response in EEG as a manifestation of
miniature saccades,” Neuron, vol. 58, no. 3, pp. 429–441, 2008.
[273] D. J. Acunzo, G. MacKenzie, and M. C. van Rossum, “Systematic biases
in early ERP and ERF components as a result of high-pass filtering,” J.
Neurosci. Methods, vol. 209, no. 1, pp. 212–218, 2012.
[274] D. Tanner, K. Morgan-Short, and S. J. Luck, “How inappropriate high-
pass filters can produce artifactual effects and incorrect conclusions in
ERP studies of language and cognition,” Psychophysiology, vol. 52, no. 8,
pp. 997–1009, 2015.
[275] A. Widmann, E. Schröger, and B. Maess, “Digital filter design for
electrophysiological data-a practical approach,” J. Neurosci. Methods,
vol. 250, pp. 34–46, 2015.
[276] L. C. Parra, C. D. Spence, A. D. Gerson, and P. Sajda, “Recipes for the
linear analysis of EEG,” Neuroimage, vol. 28, no. 2, pp. 326–341, 2005.
[277] F. C. Viola, S. Debener, J. Thorne, and T. R. Schneider, “Using ICA
for the analysis of multi-channel EEG data,” Simultaneous EEG fMRI:
Recording, Anal., Appl., vol. 1, pp. 121–133, 2010.
[278] F. Kelber et al., “Mapping deep neural networks on SpiNNaker2,” in
Proc. Neuro-Inspired Comput. Elements Workshop, ser. NICE ’20. New
York, NY, USA: Association for Computing Machinery, 2020. [Online].
Available: https://doi.org/10.1145/3381755.3381778
[279] Y. Umuroglu et al., “FINN: A framework for fast, scalable binarized
neural network inference,” in Proc. ACM/SIGDA Int. Symp. Field-
Programmable Gate Arrays, ser. FPGA, 2017, pp. 65–74.
Hector A. Gonzalez (Student Member, IEEE) re-
ceived the B.Sc. degree in electronics engineering
from the National University of Colombia, Bogota,
Colombia, in 2011, and the M.Sc. degree in microsys-
tems engineering from Khalifa University (Masdar
Institute Campus), in collaboration with MIT, Abu
Dhabi, UAE, in 2017. He is currently a Scientific Staff
and working toward the Ph.D. degree with the Chair
of highly-parallel VLSI-systems and neuromorphic
circuits with Technische Universität Dresden, Dres-
den, Germany. From 2011 to 2015, he held multi-
ple industrial positions as a Senior Engineer of instrumentation and control
electronics. His research interests include neuromorphic computing, hardware
design for machine learning algorithms, biomedical applications, and digital
signal processing for radar systems. His honors and awards include multiple
Enrolments of Honor from the National University of Colombia, a Richard
Newton fellowship at the Design Automation Conference (2018), San Francisco,
United States of America, an international scholarship for an immersion training
at the Institut Teknologi Petroleum Petronas, and the Petronas Leadership Centre,
Kuala Terengganu and Kuala Lumpur, Malaysia, and a full Graduate studies
scholarship in Abu Dhabi, United Arab Emirates.
GONZALEZ et al.: HARDWARE ACCELERATION OF EEG-BASED EMOTION CLASSIFICATION SYSTEMS: A COMPREHENSIVE SURVEY
Richard George (Senior Member, IEEE) received
the M.Sc. degree in medical engineering from HTW
and the University of Saarland, Saarbrcken, Germany,
in 2013, with a specialization Neural Engineering,
and the Ph.D. degree in computational neurosciences
from the Institute of Neuroinformatics of UZH and
ETH Zürich, Zürich, Germany, in 2018, for his work
on structural plasticity in neuromorphic systems. He
is currently a Postdoctoral Fellow with the Chair
of highly-parallel VLSI-systems and neuromorphic
circuits, Technische Universität Dresden, Dresden,
Germany. His aim of research is the creation of active and intelligent neuro-
prosthetic devices. His particular focus is in the creation of energy efficient
computational architectures capable of processing electrophysiological signals
and forming electrical response stimuli within biohybrid closed-loop systems.
Shahzad Muzaffar (Student Member, IEEE) re-
ceived the B.S. degree in telecommunication engi-
neering from the Electrical Engineering Department,
National University of Computer and Emerging Sci-
ences, Lahore, Pakistan, in 2008, the M.Sc. degree
in microsystems engineering, and the Ph.D. degree in
interdisciplinary engineering from Khalifa University
(Masdar Campus), in collaboration with MIT, Abu
Dhabi, UAE, in 2015 and 2018, respectively. He is
currently a Postdoctoral Fellow with Electrical and
Computer Engineering Department, Khalifa Univer-
sity, Abu Dhabi, UAE. From 2011 to 2013, he was a Senior Development Engi-
neer with the ESD Division of Mentor Graphics, Lahore, Pakistan. From 2009
to 2011, he was a Design Engineer (Team Lead) with the Center for Excellence
in FPGA/ASIC Research, National University of Sciences and Technology,
Islamabad, Pakistan. Since he joined the Masdar Institute, Abu Dhabi, UAE,
in 2013, he has been a Graduate Research Assistant with the Abu Dhabi -
SRC Center of Excellence on Energy-Efficient Electronics Systems (ACE 4 S),
where he has been the System Design Lead of one of two ACE 4 S technology
demonstrators. He is the author of more than ten refereed publications and book
chapters and is the inventor of five US patents (pending). His awards include the
Best M.Sc. Thesis Award at the Masdar Institute for the 2014–2015 academic
year and a third Best-Paper Prize at the 2017 Cyber Security Awareness Week,
New York University, Abu Dhabi Campus.
Javier Acevedo (Student Member, IEEE) received
the B.Sc. degree in electronics engineering from the
National University of Colombia, Bogota, Colombia,
in cooperation with the Grenoble Institute of Tech-
nology, where he coursed the M2 Master Program for
the M.Sc. degree in signal processing and the M.Sc.
degree in nanoelectronics systems from Technische
Universität Dresden, Dresden, Germany. He is cur-
rently a Scientific Staff and working toward the Ph.D.
degree with the Telekom Chair of Communication
Networks, Technische Universität Dresden. His par-
ticular focus is in the design and implementation of hardware accelerators for
low-latency applications applied to the Tactile Internet.
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
441
Sebastian Höppner received the Dipl.-Ing. (M.Sc.)
in electrical engineering and the Ph.D. degree from
Technische Universität Dresden, Dresden, Germany,
in 2008 and 2013, respectively. He is currently a
Research Group Leader and Lecturer with the Chair
of Highly-Parallel VLSI-Systems and Neuromorphic
Circuits. He is the author or co-author of more than
56 publications, and five issued and five pending
patents in below fields. His research interests include
circuits for low-power systems-on-chip in advanced
technology nodes, with special focus on clocking,
data transmission, and power management. He has experience in designing full-
custom circuits for multiprocessor systems-on-chip, like ADPLLs, register files,
and high-speed on-chip and off-chip links, in academic and industrial research
projects. He is managing the full-custom circuit design and SoC integration for
more than 12 MPSoC chips in 65 nm, 28 nm, and 22 nm CMOS technology.
He leads the chip design of the SpiNNaker2 neuromorphic computing system
within the Human Brain Project. He was the recipient of the Barkhausen Award.
Christian Mayr (Member, IEEE) received the Dipl.-
Ing. (M.Sc.) in electrical engineering, and the Ph.D.
and Habilitation degrees from Technische Univer-
sität Dresden, Dresden, Germany, in 2003, 2008,
and 2012, respectively. He is currently a Professor
of electrical engineering with Technische Universität
Dresden. From 2003 to 2013, he was with Technische
Universität Dresden, with a secondment to Infineon
(2004–2006). From 2013 to 2015, he was a Postdoc
with the Institute of Neuroinformatics, University of
Zurich and ETH Zurich, Zurich, Switzerland. Since
2015, he has been the Head of the Chair of Highly-Parallel VLSI-Systems and
Neuromorphic Circuits with Technische Universität Dresden. He is the author
or co-author of more than 80 publications and holds four patents. His research
interests include bio-inspired circuits, brain-machine interfaces, AD converters,
and general mixed-signal VLSI-design. He was the Editor or Reviewer for
various IEEE and Elsevier journals. His work has received various awards.
Jerald Yoo (Senior Member, IEEE) received the B.S.,
M.S., and Ph.D. degrees from the Department of
Electrical Engineering, Korea Advanced Institute of
Science and Technology, Daejeon, South Korea, in
2002, 2007, and 2010, respectively.
From 2010 to 2016, he was with the Department
of Electrical Engineering and Computer Science,
Masdar Institute, Abu Dhabi, UAE, where he was
an Associate Professor. From 2010 to 2011, he was
also with the Microsystems Technology Laboratories,
Massachusetts Institute of Technology, Cambridge,
MA, USA, as a Visiting Scholar. Since 2017, he has been with the Department of
Electrical and Computer Engineering, National University of Singapore, Singa-
pore, where he is currently an Associate Professor. He has pioneered research on
low-energy body-area-network transceivers and wearable body sensor network
using planar-fashionable circuit board for continuous health monitoring system.
He has authored book chapters in Biomedical CMOS ICs (Springer, 2010) and
in Enabling the Internet of Things - From Circuits to Networks (Springer, 2017).
His current research interests include low-energy circuit technology for wearable
bio signal sensors, flexible circuit board platform, BAN transceivers, ASIC
for piezoelectric Micromachined Ultrasonic Transducers, and System-on-Chip
design to system realization for wearable healthcare applications.
Dr. Yoo is an IEEE Circuits and Systems Society Distinguished Lecturer.
From 2017 to 2018, he was also the IEEE Solid-State Circuits Society (SSCS)
Distinguished Lecturer. He was the recipient or co-recipient of various awards,
including the IEEE International Symposium on Circuits and Systems (ISCAS)
2015 Best Paper Award, the ISCAS 2015 Runner-Up Best Student Paper
Award, the Masdar Institute Best Research Award in 2015, and the IEEE Asian
Solid-State Circuits Conference (A-SSCC) Outstanding Design Award (2005).
He was the Vice Chair of IEEE SSCS United Arab Emirates Chapter. He is a
Technical Program Committee Member of the IEEE A-SSCC, IEEE Custom
Integrated Circuits Conference, and the IEEE International Solid-State Circuits
Conference Student Research Preview. He is also an Analog Signal Processing
Technical Committee Member of IEEE Circuits and Systems Society.
442 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS, VOL. 15, NO. 3, JUNE 2021
Frank H. P. Fitzek (Senior Member, IEEE) received
the Diploma (Dipl.-Ing.) degree in EE from RWTH
Aachen, Aachen, Germany, in 1997, and the Ph.D.
(Dr.-Ing.) degree in EE from the Technical University
Berlin, Berlin, Germany, in 2002. In 2002, he became
an Adjunct Professor with the University of Ferrara,
Ferrara, Italy. He is currently a Professor and Head
of the Deutsche Telekom Chair of Communication
Networks, Technische Universität Dresden, Dresden,
Germany, and has been coordinating the 5G Lab Ger-
many since 2014. Since 2019, he has been a Speaker
of the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft)
Cluster of Excellence Centre for Tactile Internet with Human-in-the-Loop. In
2003, he joined Aalborg University, Aalborg, Denmark, as a Professor. He
has visited various research institutes, including the Massachusetts Institute
of Technology, Cambridge, MA, USA, VTT, Espoo, Finland, and Arizona
State University, Tempe, Arizona. Since 1999, he co-founded various start-up
companies. He was the recipient of various awards, including the NOKIA
Champion Award and the Nokia Achievement Award. In 2011, he was also the
recipient of the SAPERE AUDE research grant from the Danish government and
in 2012 the Vodafone Innovation Prize. In 2015, he was awarded the honorary
degree Doctor Honoris Causa from the Budapest University of Technology and
Economics (BUTE).
Authorized licensed use limited to: PSG COLLEGE OF TECHNOLOGY. Downloaded on July 04,2024 at 16:39:10 UTC from IEEE Xplore. Restrictions apply.
Ibrahim (Abe) M. Elfadel (Senior Member, IEEE)
received the Ph.D. degree from the Massachusetts In-
stitute of Technology, Cambridge, MA, USA, in 1993.
He is currently a Professor of electrical engineering
and computer science with Khalifa University, Abu
Dhabi, UAE. Prior to his current academic position,
he was a Research Staff Member and then Senior
Scientist with the corporate CAD organizations at
IBM Research, and the IBM Systems and Technology
Group, Yorktown Heights, NY, USA, where he was
involved in the research, development, and deploy-
ment of CAD tools and methodologies for IBM’s high-end microprocessors. He
is the inventor or co-inventor of 50 issued US patents with several more pending.
His current research interests include IoT platform prototyping, energy-efficient
edge and cloud computing, secure IoT communications, embedded digital-signal
processing, and computer-aided design for VLSI, MEMS, and silicon photonics.
He was the recipient of six invention achievement awards, one Outstanding
Technical Achievement Award and one Research Division Award, all from
IBM, for his contributions in the area of VLSI CAD, and in 2018, (with Prof.
Mohammed Ismail) the SRC Board of Director Special Award for pioneering
semiconductor research in Abu Dhabi. In 2014, he was the co-recipient of
the D. O. Pederson Best Paper Award from the IEEE TRANSACTIONS ON
COMPUTER-AIDED DESIGN FOR INTEGRATED CIRCUITS AND SYSTEMS, in 2019,
the Best Paper Award from the IEEE Conference on Cognitive Computing,
Milan, Italy, 2019. He is an Associate Editor for the IEEE TRANSACTIONS
ON VLSI SYSTEMS and was also on the Technical Program Committees of
several leading conferences, including DAC, ICCAD, ASPDAC, DATE, ISCAS,
VLSI-SoC, ICCD, ICECS, and MWSCAS. He was the General Co-chair of
the IFIP/IEEE 25th International Conference on Very Large Scale Integration
(VLSI-SoC 2017), Abu Dhabi, UAE.