See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/319284555

Cognitive load classification in learning tasks from hemodynamic responses

using type-2 fuzzy sets

Conference Paper · July 2017

DOI: 10.1109/FUZZ-IEEE.2017.8015659

CITATIONS READS
11 1,926

6 authors, including:

Amiyangshu De Amit Konar

Indian Institute of Technology Kharagpur Jadavpur University
20 PUBLICATIONS   32 CITATIONS    599 PUBLICATIONS   8,663 CITATIONS   

SEE PROFILE SEE PROFILE

Amalesh Samanta Souvik Biswas

Jadavpur University Indian Institute of Technology Kharagpur
120 PUBLICATIONS   1,686 CITATIONS    12 PUBLICATIONS   21 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Kinect Sensor based Gesture Recognition View project

Evolution of Language and Cognition View project

All content following this page was uploaded by Amiyangshu De on 25 December 2017.

The user has requested enhancement of the downloaded file.

 Cognitive Load Classification in Learning Tasks from
Hemodynamic Responses Using Type-2 Fuzzy Sets
Amiyangshu De1 , Amit Konar1 ,
Amalesh Samanta2 , Souvik Biswas3
1 University of Cincinnati, USA
Department of Electronics and
Telecommunication Engineering
2 Liverpool Hope University, United Kingdom
Department of Pharmaceutical Technology
3 anca.ralescu@uc.edu, nagara@hope.ac.uk
School of Bio-Science and Engineering
Jadavpur University
Kolkata, India
amiyangshu de@yahoo.com, konaramit@yahoo.co.in,
asamanta61@yahoo.co.in, souvikbiswas.res@gmail.com
Abstract—Although there exist recent works on fMRI based
cognitive learning, there is a dearth of literature on fNIRs
based studies on learning and memory. This paper provides
a novel study on the cognitive load detection of subjects en-
gaged in symbol-meaning associative learning tasks from the
direct measurement of the hemodynamic response of the brain.
The hemodynamic response collected during symbol-meaning
associative learning tasks by subjects are pre-processed (filtered
from artifacts) for extraction of 112-dimensional features, which
are reduced to 20 dimensions by a meta-heuristic optimization
algorithm for subsequent transfer to a interval type-2 fuzzy
classifier to classify three levels of cognitive loads (High, Low
and Moderate) borne by the subjects at different time slots of
the learning task. Analysis undertaken reveals that the type-2
fuzzy classifier with the proposed feature selection mechanism
has a high performance in classification of the cognitive loads
over 89%. Experimental analysis further reveals that the transfer
of brain activation from orbitofrontal to ventrolateral prefrontal
cortex takes place during transition of cognitive load from high to
low. In addition, the activation of dorsolateral prefrontal cortex is
also reduced during low cognitive load of subjects. These findings
would offer justification of inability to handle high cognitive
loads by people with under-developed/damaged orbitofrontal and
dorsolateral prefrontal cortex.
Keywords—fNIRs, Hemodynamic response, Type -2 fuzzy set
induced classifier, Meta-heuristic feature selection.
I. INTRODUCTION
Brain-computer interfacing (BCI) is currently receiving in-
creasing interest for its ability to uncover the biological basis
of cognition in brain-actuated tasks. Depending on the com-
plexity of the cognitive tasks, the engagement of brain-areas
also vary, which in turn fixes the blood-supply requirement
of the tissues. The brain has its natural control mechanism to
supply required level of oxygenated blood to the right tissue
at the desired time point [1]. Measurement of oxygenated
and deoxygenated blood in a given brain area indicates the
level of activity of that area/lobe. A Functional Near Infra-
Red Spectroscopy (fNIRs) machine offers the measurements
of oxygenated and deoxygenated blood in selected brain lobes,
over which the device is mounted. The present study attempts
Anca L. Ralescu4 , Atulya K. Nagar5
4
EECS Department,
5
Mathematics and Computer Science Department,
to predict the cognitive load of subjects engaged in symbol-
meaning associative learning tasks using the fNIRs response
of the subject. There exist traces of works on the relationship
between prefrontal hemodynamics and motor activities and/or
learning. For instance, Ono et al., Lu and Yuan and Gentilli
et al. demonstrate cortical changes during learning of motor
skills [2], [3], [4]. Ma et al. and White et al. reveal the role of
prefrontal cortex hemodynamics in fear learning and cognitive
control during gaming using fNIRs data [5], [6]. Rossi et al.
demonstrate the role of language training to modulate word
learning in young infants, whereas, Ferreri et al. present the
role of music in verbal memory encoding with decreasing
prefrontal activity [7], [8]. Vermeij et al. indicate the role of
ageing in working memory [9]. Blumenfeld and Ranganath
claim that working memory plays a crucial role in long term
memory formation and in learning new skills [10]. Vogan et al.
claim using fMRI analysis that cognitive load among children
with brain disorders differs from normal subjects in visual
working memory task [11].
However, recent studies have not explored the relationship
between changes in cognitive load in prefrontal cortex at
different learning stages. In brief, how neurovascular coupling
changes during successive learning trials up to the completion
of learning still remains a mystery. Hence, a near infrared
response study may offer an insight to the detailed steps
of learning mechanism. The essence of studying learning
mechanism is important as it offers a guideline to select
the right person for the right memory-related cognitive task.
The paper aims at classifying the cognitive load of subjects
at different stages of learning into 3 classes: HIGH, LOW
and MODERATE using the hemodynamic features of fNIRs
signals. The captured signals are pre-processed to keep them
free from artifacts. Next, certain hemodynamic features are
extracted from the pre-processed fNIRs data. A particle swarm
optimization induced feature selection algorithm is used to
reduce high dimensional features and the reduced features
are fed to an interval type 2 fuzzy classifier to classify the
cognitive load at different levels of symbol-meaning associa- C. Feature Extraction
tive learning. Fuzziness appears in the present context because The following features are extracted from the 16-channel
of interpersonal differences in learning levels, which can be fNIRs data, to obtain seven features/channel (for a total of
better captured by type-2 fuzzy sets. Here lies the importance 16 × 7 = 112 features):
of a type- 2 fuzzy classifier. The novelty of this paper is to F1: Mean values of oxyhemoglobin (HbO) concentration;
design a interval type-2 fuzzy classifier (IT2FC) for the present F2: Mean values of total hemoglobin (THb) concentration;
application. Additional merit of this paper lies in the meta- F3: Mean values of oxygen demand (HbO-HbR) concentration;
F4: Standard deviation of HbO concentration;
heuristic feature selection. The paper is organized as follows: F5: Standard deviation of THb concentration;
Section II provides the theoretical framework of interval type-2 F6: Skewness of HbO concentration;
fuzzy classifier and PSO-induced feature selection. Section III F7: Skewness of HbO-HbR concentration.
includes experiments and main results. Conclusions are listed
D. Feature Reduction
in section IV.
The 112-dimensional features are reduced to vectors of
II. P RINCIPLES AND M ETHODOLOGY dimension 20 using a novel meta-heuristic feature selection
algorithm based on optimizing the following two objectives.
This section provides the tools and techniques developed for
The first objective is to minimizing the (city block) distance
the given problem. The following basic steps of operations are
between components of i-th vector with j-th class centroid.
required for the cognitive load detection problem: (a) normal-
The second maximize the intra-class distance of each pair of
ization of the raw data, (b) preprocessing and noise filtering,
classes.
(c) meta-heuristic feature selection and (d) classification and
More precisely, given that → a `i,1 , . . . , →
a `i = {→
− − −
a `i,R }
cognitive load in different learning stages using interval type
is the i-th feature vector having R components falling in
2 fuzzy sets. The above mentioned approaches are discussed
class `, b`j and bmj denote the component of class centroids
below:
for class ` and m respectively, the two objective functions are
as shown in equations (3) and (4) below:
A. Normalization of the Raw Data
Let CHbO and CHbR be the concentration of oxyhe- X Q X
P X R

moglobin (HbO) and deoxyhemoglobin (HbR) respectively, J1 = |a`i,j − b`j | (3)

measured in mili-mol/litre. We take maximum and minimum c=1 i=1 j=1

of CHbo and CHbR in each session. Let M ax CHbO and P P X

R
M in
CHbO be the maximum and minimum concentration of
X X
J2 = |b`j − bm
j | (4)
HbO in a session. Then we normalize CHbO by: m=1,m6=c c=1 j=1

CHbO −M in CHbO where P denotes the number of classes, Q the number

C
bHbO =
M ax C M in C
(1) of data points, and R is the number of features. To jointly
HbO − HbO
optimize J1 and J2 the objective function J is defined as
Similarly, we normalize CHbR by: shown in equation (5):

CHbR −M in CHbR J = J1 − J2 =
C
bHbR =
M ax C M in C
(2)
HbR − HbR
PP PQ PR
where the parameters used in the last equation has meaning c=1 i=1 j=1 |a`i,j − b`j |− (5)
similar to those defined for CHbO . It is needless to mention PP PP PR
here that such transformation returns normalized C bHbO and m=1,6=c c=1 j=1 |b`j − bm
j |
CHbR in [0, 1]. We also verified that the data set obtained
b
Best r of R features are obtained from equation (5) by using
within a session are normally distributed, i.e., all data points Particle Swarm Optimization [13].
lie within µ ± 3σ, (i.e., within three standard deviations of the
mean values). E. Cognitive Load Classification using Interval Type-2 Fuzzy
Sets
B. Preprocessing and Noise Removal
An Interval Type-2 Fuzzy Set induced classifier is developed
A digital filter with a pass band of (0.1-0.4 Hz) is used to classify 20 dimensional data points into three classes: LOW,
to eliminate the undesirable signals in the above band. The MODERATE and HIGH. An example of a classifier rule is
Chebyshev filter from well-known digital filters realized with given below:
Butterworth and Chebyshev technology is selected for it has If
the faster roll-off than Butterworth. In addition, an independent For all channels F 1, F 2, F 3 are HIGH
component analysis (ICA) is performed to remove physiolog- and F 4, F 5, F 6, and F 7 are LOW
ical noise from mixed signals allowing the restoration of the THEN
primary cortical activity related signals [12]. For all the channels then cognitive load is HIGH
 f1 , f2 , ..., fN ∈ set of 20 features selected by feature selection algorithm.

Fig. 1. Computing firing strength of an activated IT2FS induced rule.

Each feature is represented by a Gaussian membership func- for symbol-meaning association is shown in Fig. 2 and the
tion (MF), the mean and variance of which are obtained from scheme of stimulus presentation is shown in Fig. 3
experimental instances. Repeated experiments with different
subjects result in a set of Gaussian MFs over individual feature
axis with shifted mean values. The collection of the obtained
Gaussians membership functions is summarized as a Interval
Type-2 Fuzzy Set (IT2FS): their minimum/maximum yield
the lower(LMF)/upper(UMF) membership functions. Thus, for
seven features there will be seven IT2FS.
Let, µA˜j (fi ) , i = 1, . . . , N, j = 1, . . . , 7, N Gaussian
Interval Type-2 membership functions for features f1 , . . . , fN .
Suppose, f1 = f10 , f2 = f20 , . . . , fN = fN
0
be the measurement
points. Define now:
Zmin,j = min{µA˜ (fi0 ) | i = 1, . . . , N } (6)
j

Zmax,j = min{µ̄A˜j (fi0 ) | i = 1, . . . , N } (7)

where µA˜j (fi0 ) and µA˜ (fi0 ) denote the jth UMF and the
j
LMF. Define,
Zmax,j + Zmin,j
Zj =
2
the average degree of firing strength (See Fig. 1) of a rule for
the given measurement points, f10 , f20 , . . . , fN0
.
After computing the firing strength of all the rules for the
measurements f10 , f20 , . . . , fN0
, the rule with the highest firing Fig. 2. Sample of symbol-meaning association used in learning and cognitive
load analysis.
strength is selected to produce the output class listed in its
consequent [14].
III. E XPERIMENTS AND R ESULTS
This section outlines the experimental set-up, the experi-
mental results and conclusions.
A. Experimental Set-up
Ten right handed student volunteers, age between 20-26
years are selected for this experiment. All of them have
normal or corrected to normal vision. A symbol-meaning
associative learning stimulus containing 15 American Indian Fig. 3. Sample of symbol-meaning association used in learning and cognitive
Symbols [15] is presented over 15 seconds for learning and load analysis.
is followed by a 30 seconds recall task. This symbol-meaning
association learning task is repeated 5 times. One minute rest is A continuous wave near infrared spectroscopy (fNIR 1100)
provided between two consecutive trials. The sample stimulus is used for recording with a sampling rate of 2 Hz and
 TABLE I
a light penetration depth of 1.25 cm. Data acquisition and M EAN ACCURACY AND STANDARD DEVIATION OF THE CLASSIFIERS
visualization are performed using COBI studio software. For Cognitive PCA PCA PCA PCA PFS
each instance, hemodynamic data is recorded after removal Load /Class LDA LSVM kNN IT2FC IT2FC
of the baseline. Fig. 4 shows capturing of prefrontal hemody- 72.24 77.201 75.462 85.604 89.77
High
(5.2222) (5.7495) (6.39216) (8.72738) (5.5305)
namic response by fNIRs during symbol-meaning associative 76.267 74.402 70.49 82.905 87.90
Moderate
learning task. The raw oxyhemoglobin (HbO) concentration (8.556) (8.4235) (8.8434) (10.1077) (9.8089)
75.569 73.791 69.635 83.014 85.17
data for first, third and fifth learning trials are plotted for all Low
(6.2131) (9.5537) (10.7174) (9.9352) (8.3108)
the 16 channels which can be observed in Fig. 5. The plotted
raw HbO concentration data reports most prominent change TABLE II
in blood oxygenation in channel 12 at different time slots of C ONFUSION MATRIX OF F UZZY T YPE 2 CLASSIFIER AND PSO BASED
learning, which is anatomically related to orbitofrontal cortex F EATURE S ELECTION (FS) TECHNIQUE
(OFC).
Predicted Class
High Moderate Low
Actual Class
High 89.77 5.41 4.82
Moderate 7.83 87.90 4.27
Low 8.33 6.50 85.17

Fig. 4. Cognitive load analysis of a subject performing symbol-meaning
associative learning task.
B. Cognitive Load Detection
Table-I provides the results of classification accuracy by
the proposed feature selection based Interval Type-2 Classifier
(PFS-IT2FC) with other existing techniques. The existing
feature selector and classifiers included in the Table I are Prin-
cipal Component Analysis (PCA) based Linear Discriminant
Analysis (LDA), Linear Support Vector Machine (LSVM),
k-Nearest Neighbor (kNN), Interval Type 2 Fuzzy Classifier
(IT2FC). It is apparent from Table I that the proposed meta-
heuristic feature selection plus IT2FS classifier (PFS-IT2FC)
outperforms others with respect to classification accuracy.
Table. II reports individual class performance from confu-
sion matrix of three separate classes LOW, MODERATE and
HIGH by implementing proposed meta-heuristic optimization
induced feature selection and Interval Type 2 Fuzzy classifier.
Table II shows that the classification accuracy for each class
exceeds 85%, reaching close to 90% (89.77%) for the HIGH
class.
C. Biological Implications
Cognitive load transfer is observed in different prefrontal
regions during learning tasks. In order to determine the cog-
nitive load distribution in prefrontal cortex in the time domain
considering different learning stages, a voxel plot approach
is performed using MATLAB 2015b version. The mean HbO
concentration for each voxel is plotted as a 2 × 8 voxel plot
demonstrating the 16 channels of the fNIRs system as shown
in Fig. 6, where, the red color means highest oxygenation,
yellow means intermediate and blue for low oxygenation.
Fig. 6 clearly demonstrates cognitive load shift from the
orbitofrontal cortex (OFC) to ventrolateral prefrontal cortex
(VLPFC) during learning trials. In addition, activation of Dor-
solateral Prefrontal Cortex (DLPFC) falls significantly when
cognitive load shifts from HIGH to LOW. The underlying
brain anatomy related to the activated orbitofrontal cortex is
associated to Broadmann area 11, which has role in planning,
decision making and encoding new information into long term
memory [16], [17], [18]. Hence, we can conclude that initial
learning trials need higher executive function and it is related
to high cognitive load. Our experimental results indicate that
the orbitofrontal cortex activation goes down when the task is
learned. Reduced activity of DLPFC correlated with low learn-
ing ability is possibly related to improper brain development,
which is still an open research problem [19], [20].
D. Ranking Classifiers using Friedman Statistical Test
The Friedman test is considered as the non-parametric
equivalent of repeated measures ANOVA. Friedman test ranks
the algorithms for each data sets [21].
k X
12 X
χ2 = [ ( Ri2 )] − 3p(d + 1) (8)
pd(d + 1) i=1
The Friedman statistics score is computed as 10.40 which
is greater than χ24,0.05 = 9.488. Thus, null hypothesis, which
assumes that the proposed classifier is no better than existing
ones used for comparison, is rejected at 4 degree of freedom
with 95% confidence level, which suggests that the classifiers
Fig. 5. HbO concentration plot obtained from raw time-domain data which is obtained from symbol meaning association learning, X axis represents time
in sec and Y axis represents HbO concentration in mili-mol/litre. Here, (a) 0-15 seconds of learning, (b) 30-45 seconds of learning, and (c) 60-75 seconds of
learning. When learning is over, cognitive load reduces which is indicated by near zero HbO concentration in (c).

of the work lies in the design of an intelligent algorithm

for feature selection and type-2 fuzzy logic induced classi-
fier for automatic classification of cognitive loads in three
distinct levels: HIGH, MODERATE and LOW. Experimental
results indicate that the proposed feature selector-classifier
combination outperforms existing techniques by a large mar-
gin. Biological implication of the research results are also
discussed in this paper. It is apparent from the voxel plots
that dorsolateral (DLPFC) and orbitofrontal cortex (OFC) are
primarily engaged in HIGH cognitive load tasks. They are less
influenced when cognitive load is reduced. The identification
of cognitive load at different phase of the problem still remains
an open arena of research. It could not be solved in this study,
Fig. 6. Shift of cognitive load in prefrontal cortex with time during (a) 0-15s using fNIRs, mainly due to the sluggish response of the device.
(b) 30-45s and (c) 60-75s of learning. Fig. (d) represents the voxel positioning A combined EEG-fNIRs approach might help in this regard.
related to the 16 fNIRs channels. Here, cognitive load shifts from OFC and
right DLPFC to right VLPFC when learning is complete. Engagement of ACKNOWLEDGMENT
OFC and DLPFC, which are more active in initial learning trials, go down in
learning completion. Financial support offered by University Grant Commission
(UGC), India to University with Potential for Excellence
TABLE III
R ANKING TABLE OF THE CLASSIFIERS ACCORDING TO THEIR % program (UPE) Phase II in Cognitive Science granted to
ACCURACY Jadavpur University, India is gratefully acknowledged.
PFS PCA PCA PCA PCA
Class Dataset
IT2FC IT2FC LSVM LDA kNN R EFERENCES
High 1 2 3 5 4
Moderate 1 2 4 3 5 [1] A.B. Scholey, S. Harper and D. O. Kennedy, ”Cognitive demand
PLow 1 2 4 3 5 and blood glucose,” Psychology and Behaviour, vol. 73, pp. 585-
Ri 3 6 11 11 14 592, 2001.
[2] Y. Ono, J. A. Noah, X. Zhang and e. al., ”Motor Learning and
Modulation of Prefrontal Cortex: An fNIRs Assesment,” Journal
of Neural Engg, p. 12pp, 2015.
can be ranked according to their percentage classification [3] F. Lu and Z. Yuan, ”Cortical changes of hemodynamic signals
accuracy. Here, p is number of class dataset, d is number of during motor skill learning: a functional NIRS study,” in TEN-
CON 2015 - 2015 IEEE Region 10 Conference, 2015.
algorithms, Ri is the rank of the classifier for i-th class dataset. [4] R. J. Gentilli, P. A. Shewokis, H. Ayaz and J. L. Contreres-
Table. III shows classifiers ranking. Vidal, ”Functional Near Infrared Spectroscopy based correlates of
Prefrontal Cortical Dynamics during a Cognitive Motor Executive
IV. C ONCLUSION Adaptation Task,” Frontiers in Human Neuroscience, vol. 7, p.
This paper proposes a new approach to detect cognitive 277, 2013.
[5] Q. Ma, Y. Huang and L. Wang, ”Left Prefrontal Activity Reflects
loads in different learning stages using fNIRs signals from the Ability of Vicarious Fear Learning: A Functional Near-
the prefrontal cortex. A pool of seven features is considered Infrared Spectroscopy Study,” The Scientific World Journal, p.
for cognitive load detection during learning. The originality Article ID 652542, 2013.
[6] M. White, M. Ninaus , S. E. Kober, C. Neuper and G. Wood,
”Neuronal Correlates of Cognitive Control during Gaming Re-
vealed by Near-Infrared Spectroscopy,” Plos One, 2015.
[7] S. Rossi, I. B. Jurgenson, A. Hanulikova, S. Telkemeyer, I.
Wartenburger and H. Obrig, ”Implicit Processing of Phono-
tactic Cues: Evidence From Electrophysiological and Vascular
Responses,” Journal of Cognitive Neuroscience, pp. 1752-1764,
2010.
[8] L. Ferreri, J.-J. Aucouturier, M. Muthalib, E. Bigand and A.
Bugaiska, ”Music improves verbal memory encoding while de-
creasing prefrontal cortex activity: an fNIRS study,” Frontiers in
Human Neuroscience, vol. 7, p. 779, 2013.
[9] A. Vermeij, A. H. E. A. van Beek, . M. G. Olde Rikkert ,
J. A. H. R. Claassen and R. P. Kessels, ”Effects of Aging
on Cerebral Oxygenation during Working-Memory Performance:
A Functional Near-Infrared Spectroscopy Study,” Plos One, p.
e46210, 2012.
[10] R. S. Blumenfeld and C. Ranganath, ”Dorsolateral Prefrontal
Cortex Promotes Long Term Memory Formation Through its Role
in Working Memory Organization,” The Journal of Neuroscience,
vol. 26, pp. 916-925, 2006.
[11] M. V. Vogan, B. R. Morgan, W. Lee, T. L. Powell, M. L. Smith
and M. J. Taylor, ”The neural correlates of visuo-spatial working
memory in children with autism spectrum disorder: effects of
cognitive load,” Journal of Neurodevelopmental Disorders, 2014.
[12] N. Naseer and K. Hong, ”fNIRS-based brain-computer inter-
faces : a review,” Frontiers in Human Neuroscience, 2015.
[13] S. Das, A. Abraham and A. Konar, ”Particle Swarm Optimiza-
tion and Differential Evolution Algorithms: Technical Analysis,
View publication stats
Applications and Hybridization Perspective,” Studies in Compu-
tational Intelligence, 2008.
[14] A. Halder, A. Konar, R. Mandal, A. Chakraborty, P. Bhowmik,
N. R. Pal and A. K. Nagar, ”General and Interval Type-2 Fuzzy
Face-Space Approach to Emotion Recognition,” IEEE Transac-
tions on Systems, Man and Cybernatics: Systems, pp. 587-605,
2013.
[15] ”Native Indian Tribes,” [Online]. Available:
https://www.warpaths2peacepipes.com/. [Accessed 08 02 2017].
[16] R. D. Rogers, A. M. Owen, H. C. Middleton, E. J. Williams, J.
D. Pickard, B. J. Sahakian and T. W. Robbins, ”Choosing between
small, likely rewards and large, unlikely rewards activates inferior
and orbital prefrontal cortex,” The Journal of Neuroscience, pp.
9029-9038, 1999.
[17] S. Frey and M. Petrides, ”Orbitofrontal cortex: A key prefrontal
region for encoding information,” PNAS, pp. 8723-8727, 2000.
[18] A. K. Barbey, M. Koening and J. Grafman, ”Dorsolateral
prefrontal contributions in human working memory,” Cortex, vol.
49, pp. 1195-1205, 2013.
[19] T. G. Lee, R. S. Blumenfeld and M. dEsposito, ” Disruption
of Dorsolateral but Not Ventrolateral Prefrontal Cortex Improves
Unconscious Perceptual Memories,” Journal of Neuroscience,
vol. 33, pp. 13233-13237, 2013.
[20] D. T. Stuss and R. K. Knight , Priciples of Frontal Lobe
Function, Oxford University Press, 2002.
[21] J. D. Sheskin, Handbook of Parametric and Nonparametric
Statistical Procedures, Chapman & Hall/CRC, 2000.