International Journal of Machine Learning and Cybernetics

https://doi.org/10.1007/s13042-019-00967-w

ORIGINAL ARTICLE

A robust multilayer extreme learning machine using kernel

risk‑sensitive loss criterion
Xiong Luo1,2,3 · Ying Li1,2,3 · Weiping Wang1,2,3 · Xiaojuan Ban1,2 · Jenq‑Haur Wang4 · Wenbing Zhao5

Received: 23 October 2017 / Accepted: 22 May 2019

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Abstract
More recently, extreme learning machine (ELM) has emerged as a novel computing paradigm that enables the neural network
(NN) based learning to be achieved with fast training speed and good generalization performance. However, the single hidden
layer NN using ELM may be not effective in addressing some large-scale problems with more computational efforts. To avoid
such limitation, we utilize the multilayer ELM architecture in this article to reduce the computational complexity, without
the physical memory limitation. Meanwhile, it is known to us all that there are a lot of noises in the practical applications,
and the traditional ELM may not perform well in this instance. Considering the existence of noises or outliers in training
dataset, we develop a more practical approach by incorporating the kernel risk-sensitive loss (KRSL) criterion into ELM,
on the basis of the efficient performance surface of KRSL with high accuracy while still maintaining the robustness to outli-
ers. A robust multilayer ELM, i.e., the stacked ELM using the minimum KRSL criterion (SELM-MKRSL), is accordingly
proposed in this article to enhance the outlier robustness on large-scale and complicated dataset. The simulation results on
some synthetic datasets indicate that the proposed approach SELM-MKRSL can achieve higher classification accuracy and
is more robust to the noises compared with other state-of-the-art algorithms related to multilayer ELM.

Keywords Extreme learning machine (ELM) · Multilayer perceptron · Kernel risk-sensitive loss (KRSL) · Deep learning

1 Introduction are proved to be well-performed [1–3]. At the same time,

With the rapid development in the field of artificial intel-
ligence, there have been lots of practical applications
using machine learning approaches and these applications
This manuscript is recommended by the 8th International
Conference on Extreme Learning Machines (ELM2017).
* Xiong Luo
xluo@ustb.edu.cn
1
School of Computer and Communication Engineering,
University of Science and Technology Beijing (USTB),
Beijing 100083, China
2
Beijing Key Laboratory of Knowledge Engineering
for Materials Science, Beijing 100083, China
3
Key Laboratory of Geological Information Technology,
Ministry of Land and Resources, Beijing 100037, China
4
Department of Computer Science and Information
Engineering, National Taipei University of Technology,
Taipei 10608, Taiwan
5
Department of Electrical Engineering and Computer Science,
Cleveland State University, Cleveland, OH 44115, USA
many researchers continue making an intensive study in
new and effective algorithms. In the last decade, an emerg-
ing machine learning algorithm, named extreme learning
machine (ELM) [4, 5], has drawn many researchers’ atten-
tions. The hidden node parameters of neural network (NN)
using ELM, including the input weights connecting input
layer and hidden layer, and the biases of hidden neurons,
are assigned randomly without being adjusted manually.
Consequently, ELM is an effective method with extremely
fast training speed and good generalization performance [6].
There have been a lot of variants of ELM addressing dif-
ferent practical issues [7–18]. Meanwhile, some theoreti-
cal improvements on ELM are also achieved. For instance,
a sparse ELM was proposed, and it can realize a highly
compact NN to reduce the storage space and testing time
[19, 20]. An enhanced bidirectional ELM was proposed to
reduce the network complexity [21]. In this network, only
the hidden nodes with large residual error are reserved and
the unimportant nodes are discarded. The online sequential
ELM was incorporated into the ensemble NNs, while pre-
senting an improved algorithm called ensemble of online

13
Vol.:(0123456789)

sequential ELM [22]. However, when solving some prob-
lems with large-scale datasets, shallow structure is not
efficient enough even with a large quantity of hidden neu-
rons. Hence, the study on multilayer architecture and deep
learning is extraordinarily important and necessary. Then,
an ELM-based hierarchical learning framework was devel-
oped for multilayer perceptron [23]. It includes two main
components, which are unsupervised multilayer feature rep-
resentation and supervised feature classification separately.
In addition, a stacked ELM was designed, while dividing an
ELM-based large single NN into some NNs with multiple
small ELMs [24]. These sub-ELMs are connected serially
and stacked on multiple layers.
In addition to the challenge of multilayer architecture, one
of the most common issues is the outliers or noises existed
in the training set [25]. On this condition, the poor gener-
alization performance may be achieved by traditional ELM
[26], partly because the mean square error (MSE) is used in
ELM as cost function. MSE is a global similarity measure
and it applies to Gaussian distributed error [27]. Neverthe-
less, it is very common for the data which are non-Gaussian
distributed in many real-world applications. In order to over-
come the poor robustness to noisy datasets, a probabilistic
regularized ELM was proposed [28]. It constructs a new
objective function to minimize both mean and variance of
the modeling error, and it is proved to be more robust com-
pared with traditional ELM. Moreover, based on the fuzzy
ELM (F-ELM), a ridge regression based extreme learning
fuzzy system (RR-EL-FS) was developed to enhance the
robustness to small and noisy datasets [29]. In RR-EL-FS,
the strategy of ridge regression is incorporated into F-ELM
to substitute the least square which is used for computing the
output weight. In order to handle the noisy data regression
problems, the constrained optimization method based ELM
for regression (CO-ELM-R) was proposed [30]. It combines
two Lagrange multipliers that mimic support vector regres-
sion (SVR) into the basis of ELM to cope with infeasible
constraints of the regression optimization problem. Recently,
a local similarity measure named correntropy was presented
[31]. Correntropy is a robust nonlinear generalized correla-
tion in kernel space and uses the concept like probability to
evaluate the similarity of two variables [32]. It is flexible
and easy to be implemented, thus it has been utilized in
various applications [33, 34]. Considering the advantages
of correntropy, a regularized correntropy criterion for ELM
was designed to address the datasets with noises [35]. In
this method, the Gaussian kernel function is utilized to
substitute Euclidean norm of the MSE criterion, which can
enhance the anti-noise ability of ELM [35, 36]. Moreover,
a variant of stacked ELM was proposed to reduce the train-
ing time and be robust for outliers rejection, which used the
correntropy-optimized principle component analysis (PCA)
to replace the original PCA method [37]. However, owing
13
International Journal of Machine Learning and Cybernetics
to the non-convexity of the performance surface for corren-
tropic loss (C-loss), it is very flat when it is far away from the
optimal solution, and it is sharp around the optimal solution,
resulting in poor accuracy and slow convergence [38]. Con-
sequently, an alternative to the correntropy in kernel space,
named kernel risk-sensitive loss (KRSL), was proposed
[39]. The performance surface of KRSL is more convex in
comparison with that of C-loss, hence it can achieve better
convergence performance in some applications [40].
Motivated by it, in this article, we propose a robust multi-
layer ELM, i.e., the stacked ELM using the minimum KRSL
(SELM-MKRSL), which utilizes KRSL as loss function by
substituting MSE in the S-ELM framework to measure the
similarity between the desired output and the learned model
output. Here, the multilayer ELM architecture is used in an
effort to tackle the issue of large-scale and complicated
dataset. The dimensionality of feature space is fixed, that
is, the number of hidden neurons is user specified at first,
without the problem of memory overflow. Meanwhile, the
multilayer structure makes it capable to train extensive data
and achieve good generalization. Differing from it in [37],
we use KRSL instead of correntropy as cost function in this
article to achieve more robustness to noises. Meanwhile,
on the basis of it in [40], we extend KRSL not just in the
field of adaptive filtering, but in classification applications.
Moreover, considering that the L1-norm and L2-norm are
used in the traditional ELM algorithm to acquire structural
risk minimization and do achieve a good performance [11],
we combine KRSL and the L2 penalty as a loss function in
S-ELM architecture in our work. The main contributions of
this article are further summarized in two aspects as follows:
1. We utilize minimum KRSL (MKRSL) criterion to
replace the MSE in ELM framework. The KRSL is
robust to non-Gaussian distributed noises and outliers,
so it can remarkably improve the anti-noise ability of
NN-based learning algorithm. Furthermore, some theo-
retical analyses are provided.
2. In order to further enhance the stability and generaliza-
tion performance of our proposed approach, we add a L2
regularization term to the cost function.
The structure of this article is organized as follows. In
Sect. 2, the original ELM, the stacked ELM as well as the
basic concept and properties of KRSL are described. In
Sect. 3, we detail the algorithm SELM-MKRSL proposed
in this article. In Sect. 4, the simulation results are shown to
evaluate the algorithmic performance and noise robustness
of our proposed SELM-MKRSL. The comparison between
SELM-MKRSL and original S-ELM as well as correntropy-
based S-ELM is also given in this section. Finally, we con-
clude this article.
International Journal of Machine Learning and Cybernetics
2 Related work
2.1 Extreme learning machine (ELM)
Extreme learning machine is a kind of generalized learning
algorithm for single hidden layer feedforward NN [4–6].
For ELM, the input weights and the biases of hidden layer
are initialized randomly and
{( need)}Nnot be tuned. Given N
arbitrary training samples xi , ti i=1, where xi ∈ Rd is the
input data vector, ti ∈ Rm is the corresponding target label.
The output of the original ELM with L hidden neurons can
be written as:
∑ 𝐈
L
ti = fL (xi ) = 𝜷 j g(wj ⋅ xi + bj ), i = 1, … , N,
j=1
where βj is the output weight vector connecting the jth hid-
den node with the output layer, g(·) is a nonlinear activation
function, wj ∈ Rd is the input weight vector connecting the
input layer with the jth hidden unit, and bj denotes the bias
term of the jth hidden neuron.
Then, (1) can be represented in the matrix form:
𝐇𝜷 = 𝐓, (2)
[ []T ]T
where 𝜷 = 𝜷 1 , 𝜷 2 , … , 𝜷 L , 𝐓 = t1 , t2 , … , tN , and the
hidden layer output matrix H is:
⎡ h(x1 ) ⎤ ⎡ g(w1 ⋅ x1 + b1 ) ⋯ g(wL ⋅ x1 + bL ) ⎤
𝐇=⎢ ⋮ ⎥=⎢ ⋮ ⋱ ⋮ ⎥.
⎢ ⎥ ⎢ ⎥ C
⎣ h(x N ⎦
) ⎣ g(w1 ⋅ xN + b 1 ) ⋯ g(w L ⋅ x N + b L ⎦
)
The output weight β can be acquired by solving
Minimize ∶ ‖𝐇𝜷 − 𝐓‖2F ,
where ‖⋅‖F represents the Frobenius norm.
The least square solution of (4) is given as:
𝜷 = 𝐇† 𝐓, (5)
†
where H is the Moore–Penrose generalized inverse of
matrix H [41, 42]. We can use the orthogonal projection
method to compute H† when HTH is nonsingular [4]:
( )−1
𝐇† = 𝐇T 𝐇 𝐇T ,
and the output weight β can be obtained by:
( )−1
𝜷 = 𝐇T 𝐇 𝐇T 𝐓.
Considering the existence of noises in real-world appli-
cations, the equality constrained optimization method-
based ELM was further proposed [43] to improve the gen-
eralization performance and stability of the standard ELM.
The core idea of this method is to minimize the training
error as well as the norm of output weights by:
Minimize ∶ ‖𝐇𝜷 − 𝐓‖2F + 𝜇‖𝜷‖2F , (8)
where ‖⋅‖F can be defined as ‖⋅‖2 which denotes the L2-
norm, μ is a tuning parameter.
According to the scale of the training dataset, we can
obtain different solutions of (8) in an effort to decrease the
computational complexity.
In the case that the number of training samples is less
than the hidden neurons, that is N < L, the output weight
vector can be represented as [24]:
( )−1
𝜷 = 𝐇T + 𝐇𝐇T 𝐓, (9)
(1) C
and the corresponding output of ELM is:
( )−1
𝐈
f (x) = h(x)𝜷 = h(x)𝐇T + 𝐇𝐇T 𝐓, (10)
C
where I is the identity matrix and C is a constant.
In another case that the number of training samples is
much larger than the hidden neurons, that is N ≫ L, the out-
put weight vector can be represented as [24]:
( )−1
𝐈
𝜷= + 𝐇T 𝐇 𝐇T 𝐓, (11)
C
and the corresponding output of ELM is:
( )−1
f (x) = h(x)𝜷 = h(x)
𝐈
+ 𝐇T 𝐇 𝐇T 𝐓. (12)
(3)
2.2 Stacked extreme learning machine (S‑ELM)
(4) When solving some problems with large-scale dataset, we
often need plenty of hidden nodes to train a learning model,
so that good generalization performance could be achieved.
However, too many hidden neurons may make the train-
ing NN become very complicated and incur the issue with
out of memory. In response to this limitation, an improved
algorithm called stacked extreme learning machine (S-ELM)
was developed [24]. S-ELM is a multilayer NN which is
composed of multiple sub-ELMs lying in distinct layers. It
(6) decomposes a large single ELM network into multiple small
ELMs, and these sub-ELMs are connected serially. The
hidden layer output of the lower layer is transmitted to the
higher layer. It should be noted that the hidden layer outputs
(7) are not passed directly but extracted several key components
via the PCA dimension reduction method [44].
Assumed that the dimensionality of the feature space is
fixed with L hidden neurons, where L can be appointed by
users. In the lower layer of S-ELM, we first conduct a standard
ELM network and acquire the hidden layer output matrix H.
The corresponding output weight β can be obtained by (9)
13
 International Journal of Machine Learning and Cybernetics

or (11). However, owing to the randomness of the generated expectation, and FAB(a, b) represents the simultaneous distri-
hidden layer parameters, i.e., input weights and biases, the bution of (A, B). In this article, we only take Gaussian kernel
connections between hidden layer and output layer may have into consideration and the corresponding kernel function is
redundant information, and will not be linearly independent. expressed as:
Hence, we perform a PCA method on β and the dimension of � �
β is cut down from L to L′, where L′ < L. 1
𝜅𝜎 (a − b) = √ exp −
(a − b)2
During the implementation of PCA dimension reduction, a 2𝜎 2
. (18)
𝜎 2𝜋
series of eigenvectors can be obtained, and they are arranged in
descending order on the basis of their eigenvalues. We select However, due to the high non-convexity of the C-loss per-
the first L′ eigenvectors as Q ∈ RL×L′. Then, the output weight formance surface, which is given by Closs(A, B) = 1 − Vσ(A,
after dimension reduction can be expressed as 𝜷 � = 𝜷 T 𝐐. B), it is extremely steep around the optimal solution, and it
Similarly, the hidden layer output is obtained as: becomes very flat when it is away from the optimal solution.
This will result in slow convergence and poor performance.
𝐇� = 𝐇𝐐. (13)
To address this issue, a new measure criterion, i.e., KRSL,
Then, the dimension-reduced hidden layer output H′ is was proposed [39].
passed to the higher layer to represent the whole information With two random variables A and B, KRSL is defined as:
of the lower hidden layer. In the next hidden layer of S-ELM,
(L−L′) new hidden neurons will be generated randomly. And L𝜆 (A, B) =
1
E[exp(𝜆(1 − 𝜅𝜎 (A − B)))]
then we have the output matrix Hnew of these new nodes as: 𝜆
(19)
𝜆∫
1
= exp(𝜆(1 − 𝜅𝜎 (a − b)))dFAB (a, b),
⎡ hnew (x1 ) ⎤ ⎡ h1 (x1 ) ⋯ hL−L� (x1 ) ⎤
𝐇new = ⎢ ⋮ ⎥=⎢ ⋮ ⋱ ⋮ ⎥. (14)
⎢ ⎥ ⎢ ⎥ where λ is the risk-sensitive parameter and λ > 0. In prac-
⎣ hnew (xN ) ⎦ ⎣ h1 (xN ) ⋯ hL−L� (xN ) ⎦
tical applications,
{( )}N with a finite number of data instances
Thus, the hidden layer output of this layer is formulated as: ai , bi i=1, we often calculate empirical KRSL to approx-
[ ] imately substitute above equation as:
𝐇 = 𝐇� , 𝐇new . (15)
1 ∑
N
We repeat these procedures mentioned above on the sub- L̂ 𝜆 (A, B) = exp(𝜆(1 − 𝜅𝜎 (ai − bi ))). (20)
sequent hidden layer until the last one, and the final output of N𝜆 i=1
S-ELM network is calculated as (10) or (12). We describe the
total process of the combination as follows: Here, the empirical KRSL [ measures]T the similarity
between[ the vector
]T 𝐀 = a 1 , a2 , … , aN and the vector
[ � 𝐇] → 𝐇� 1
� 𝐁 = b1 , b2 , … , bN . When there is no ambiguity, we denote
[𝐇� 1 , 𝐇new 2 ] → 𝐇� 2 L̂ 𝜆 (A, B) as L̂ 𝜆 (𝐀, 𝐁).
𝐇 2 , 𝐇new 3 → 𝐇 3
. (16) Generally, there are some significant properties of KRSL.
[ � ] ⋮ � They are listed as follows, and they have been proved in [39]:
𝐇 N−2 , 𝐇new (N−1) → 𝐇 [ N−1 ]
𝐇� N−1 , 𝐇new N
Property 1 KRSL is symmetric: L𝜆 (A, B) = L𝜆 (B, A).

2.3 Kernel risk‑sensitive loss (KRSL) Proper ty 2 K R S L i s p o s i t i ve a n d b o u n d e d :

1
𝜆
≤ L𝜆 (A, B) ≤ 𝜆1 exp(𝜆), and when A = B, L𝜆 (A, B) reaches
The KRSL is a modified local similarity measure based on the the minimum.
correntropy [39]. Both correntropy and KRSL are effective in
processing the data which have large outliers, because they are Property 3 L𝜆 (A, B) ≈ 1
𝜆
+ Closs (A, B) when λ is small
not sensitive to noises. enough.
Given two arbitrary variables A and B, the correntropy of
them can be defined by: Property 4 L𝜆 (A, B) ≈ 1
𝜆
+ 1
2𝜎 2
E[(A − B)2 ] when σ is large
enough.

∫ (17) [ ]T
V𝜎 (A, B) = E[𝜅𝜎 (A − B)] = 𝜅𝜎 (a − b)dFAB (a, b),
Property 5 Let e = 𝐀−𝐁 = e1 , e2 , … , eN , where
ei = ai − bi and i = 1, 2, …, N. The empirical KRSL L̂ 𝜆 (𝐀, 𝐁)
is the function of e, and when ‖e‖∞ = maxi=1,2,…,N ��ei �� ≤ 𝜎 ,
where σ is kernel band width, ĸσ(·) represents the Mer-
cer kernel function [45], E(·) denotes the mathematical
it is convex.

13
International Journal of Machine Learning and Cybernetics

Property 6 In the case that ‖e‖∞ > 𝜎 , the empirical KRSL where λ is the risk-sensitive parameter satisfying λ > 0, η is
L̂ 𝜆 (𝐀, 𝐁) will be convex as long as the risk-sensitive param- the regularization coefficient, and ĸσ(·) is the kernel function
eter λ is large enough. defined as (18) with bandwidth σ.
Using the MKRSL criterion in SELM-MKRSL, for the
τth iteration, the optimal solution of above equation can be
3 Stacked ELM using the minimum KRSL expressed as:
criterion
𝜷 𝜏+1 = (𝐇T 𝐀𝐇 + 𝜂𝐈)−1 𝐇T 𝐀𝐓, (24)
In this section, we present the implementation of our algo- where A is a diagonal matrix and its diagonal element Aii
rithm SELM-MKRSL. Furthermore, we also provide the is defined as:
theoretical analysis for the convergence and computational
1 ∑ ( ( ))
N
complexity of SELM-MKRSL. Aii = 𝛼i𝜏+1 = exp 𝜆 1 − 𝜅𝜎 (ti − hi 𝜷 𝜏 ) 𝜅𝜎 (ti − hi 𝜷 𝜏 ).
2
N𝜎 i=1
3.1 The proposed algorithm (25)
Actually, it is easy to deduce
The KRSL is used as the loss function of S-ELM to replace 𝜕
the objective function as (4). Then, the problem can be J
𝜕𝜷 KRSL
(𝜷) = 0
solved by minimizing the KRSL between the target output
1 ∑ ( ( ( )))
N
and the predicted output. The new criterion is given as: ⇒− exp 𝜆 1 − 𝜅𝜎 ti − hi 𝜷
2
N𝜎 i=1
∑
N ( )( )
J(𝜷) = min L𝜆 (ti − yi ), (21) × 𝜅𝜎 ti − hi 𝜷 ti − hi 𝜷 hi + 2𝜂𝜷̃ =0
𝜷 (N )
i=1 ∑ T ∑N
⇒ hi 𝛼i hi + 𝜂 𝜷 = hTi 𝛼i ti
where N is the number of training samples, ti denotes the i=1 i=1
target output of the ith sample xi. In addition, yi denotes the ( )−1 (N )
∑
N
∑
calculated output of xi in training network, and it can be ⇒𝛽= hTi 𝛼i hi +𝜂 × hTi 𝛼i ti ,
expressed as: i=1 i=1
(26)
yi = hi 𝜷, (22)
where hi is the hidden layer output vector for xi, and β is the
hidden layer output weight, which is given by
where 𝜂 = 2𝜂̃ represents the regularization coefficient, and
[ ]T αi can be obtained by (25).
𝜷 = 𝜷 1, 𝜷 2, … , 𝜷 L .
We implement the MKRSL to calculate the output weight
Moreover, in order to improve the stability and learning β as described above in each hidden layer during the training
performance, a L2 regularization term is added to (21). As of S-ELM network. The architecture of our proposed SELM-
a result, we can obtain a new objective function defined as: MKRSL is illustrated in Fig. 1 and the whole process of train-
ing SELM-MKRSL is summarized in Algorithm 1. It should
� �
1 �
N
� � � ��� be pointed out that, Lτ mentioned in Algorithm 1 is the KRSL
2
JKRSL (𝜷) = min
𝜷 N𝜆 i=1
exp 𝜆 1 − 𝜅𝜎 ti − hi 𝜷 + 𝜂‖𝜷‖F , of the desired output and the model output in the τth iteration,
(23) which is calculated as:

1 ∑ ( ( ( )))
N
L𝜏 = exp 𝜆 1 − 𝜅𝜎 ti − yi . (27)
N𝜆 i=1

13
 International Journal of Machine Learning and Cybernetics

13
International Journal of Machine Learning and Cybernetics

(a) (b)

The initial β βT

Dimension reduction
1 ... j ... L

The τ-th β
1 ... L’

No
Lτ-Lτ-1< ε ? β’

Yes
T

(c)

1 1

1 1 1
...
...

...
...

...
xi L’ L’ ti
...
...

...

...

...
L L L
d Input m
weight
Input matrix wj Random Output
layer hidden nodes Randomly generated nodes layer

Fig. 1  The architecture of SELM-MKRSL: a MKRSL; b PCA; c S-ELM

3.2 Convergence analysis Hence, this sequence is non-increasing finally. In

addition, it has been proved that the KRSL is bounded,
Motivated by [35], in accordance with the above analysis, that is 𝜆1 ≤ L𝜆 (A, B) ≤ 𝜆1 exp(𝜆) [39]. As a consequence,
two theorems for the convergence of our approach are pro- {JKRSL (𝛽 ,𝜏 𝛼 )𝜏 is bounded} as well. So the sequence
𝜏 𝜏

vided as follows. JKRSL (𝛽 , 𝛼 ), 𝜏 = 1, 2, ⋯ is verified to be converged.

{ }
Theorem 1 The sequence JKRSL (𝛽 𝜏 , 𝛼 𝜏 ), 𝜏 = 1, 2, ⋯ con- Theorem 2 When A = I and η = 0 in (24), the MKRSL cri-
verges after iteratively calculating by (24) and (25). terion is equivalent to MSE.

Proof According to (23) and (24), it can be seen that Proof If A = I and η = 0, we have:
𝜷 ∗ = (𝐇T 𝐀𝐇 + 𝜂𝐈)−1 𝐇T 𝐀𝐓 = (𝐇T 𝐇)−1 𝐇T 𝐓. (34)
JKRSL (𝛽 𝜏 , 𝛼 𝜏 ) ≥ JKRSL 𝛽 𝜏+1 , 𝛼 𝜏 ≥ JKRSL 𝛽 𝜏+1 , 𝛼 𝜏+1 .
( ) ( )
We can observe that this equation is the same as (7).
(33) Consequently, the theorem turns out to be true.

13

3.3 Computational complexity
In Algorithm 1, the main computational cost is the update of
output weight β in Step 2. The computational complexity
of vector α in each iteration for one hidden layer is O(NL)
according to (30). Furthermore, the calculation of the
output weight using (24) takes the computational cost as
O(L3 + L2N + L2m + 2LN2 + LNm) in each iteration of one
layer, where m is the number of output labels. As a conse-
quence, the total computational cost for Step 2 is O(kNIter
(L3 + L2N + L2m + 2LN2 + LNm)) with total k hidden layers
and NIter iterations in each layer. Generally speaking, the
number of training samples N is larger than the number of
hidden neurons L, i.e., N > L, and the number of labels m is
usually small. Hence, the overall computational complexity
of Algorithm 1 is O(kNIterLN2).
It should be pointed out that the iteration times NIter will
not be so large if the parameters in SELM-MKRSL are set
appropriately. Hence, the MKRSL criterion used in multi-
layer ELM may not significantly increase the computational
cost compared with MSE.
4 Performance evaluation and analysis
In this section, the simulation results are presented to evalu-
ate the performance as well as the noise robustness of our
proposed approach on some synthetic datasets. Meanwhile,
we compare the SELM-MKRSL proposed in this article with
the original S-ELM [24] and the S-ELM using correntropy
(SELM-correntropy), in terms of the performance in dealing
with contaminated training data. As to SELM-correntropy,
we replace the MSE criterion in the same way as [35], just in
the S-ELM architecture instead. Therefore, these three meth-
ods all have the same multilayer architecture in the learning
network, so that we can evaluate the effect of MSE, corren-
tropy and KRSL criterions on classification performance.
All the simulations in this article are conducted on the con-
ditions as follows: Intel-i7 2.8G CPU, 16G RAM, Windows
7, MATLAB R2012a.
4.1 Datasets
In this article, we implement the simulations on some data-
sets which are described in detail below. With regard to the
selection of these datasets, we choose those whose distribu-
tion is relatively balanced. Meanwhile, the number of data
samples ranges from 200 to 10,000 and the dimensionality
of features ranges from single digit to four figures. Note that
the number of features will not be more than the correspond-
ing number of samples.
USPS is a handwritten digits dataset transformed from
8-bit grayscale images of digits “0” to “9”. This dataset
13
International Journal of Machine Learning and Cybernetics
is provided by the U.S. Postal Service and it has 7291
training samples and 2007 testing samples, each of which
contains 256 features [46].
ISOLET is a dataset used for classification task from
UCI machine learning repository [47]. It has 7797
instances in total with 617 attributes and is trained for
predicting which letter name is spoken. In our simulations,
we select 6238 samples randomly as training data and the
rest 1559 samples as testing data.
SPECT heart is a dataset that describes diagnosing
of cardiac single proton emission computed tomography
(SPECT) images [48]. Each of the patients is classified
into two categories: normal and abnormal. There are 267
instances in total and 80 of them are used for training as
well as the remainder 187 instances are for testing with
22 attributes.
Internet advertisements dataset represents a set of pos-
sible advertisements on Internet pages [49]. It is used to
predict whether an image is an advertisement (“ad”) or not
(“nonad”) with 3279 samples and 1558 features. We take
70% of the samples as training data and the remaining 30%
as testing data.
Balance scale dataset was generated to model psychologi-
cal experimental results [50]. Each example is classified as
having the balance scale tip to the right, tip to the left, or be
balanced. There are 625 samples in total with four attributes.
We take 75% of it to train the model and the rest 25% to test
the performance.
Drug consumption is a dataset used for estimating when
people use the drug lastly [51]. The dataset contains 1885
instances and each sample has 30 features. We choose 1320
samples to train randomly and the rest to test.
Contraceptive method choice dataset is a subset of the
1987 National Indonesia Contraceptive Prevalence Survey
[52]. It was collected to predict the current contraceptive
method choice (no use, long-term methods, or short-term
methods) of a woman based on her demographic and socio-
economic characteristics. The dataset has 1473 instances
with 9 attributes, about 67% of which is used as training
data and the rest 33% as testing data.
Parkinson speech dataset is used to estimate whether peo-
ple have Parkinson’s disease (PD) or not [53]. The training
data were collected from 40 people, half of which are PD
patients and the other half are healthy individuals. Each
person has 26 voice samples, so the training dataset con-
tains 1040 samples in all. As to the collection of the testing
dataset, 28 PD patients were asked to say only the sustained
vowels ‘a’ and ‘o’ three times respectively, which made a
total of 168 recordings. Each sample in training dataset or
test dataset has 26 features extracted from voice samples.
QSAR biodegradation dataset contains 1055 instances
with 41 attributes to discriminate ready (356) and not ready
(699) biodegradable molecules [54]. We choose 750 samples
International Journal of Machine Learning and Cybernetics
randomly as training data and the remaining 305 samples as
testing data.
Seismic bumps dataset describes the problem of high
energy (higher than ­104 J) seismic bumps forecasting in a
coal mine [55]. There are 2584 samples in total and each
sample has 19 features. 70% of the data is used for training
and the rest 30% is used for testing.
In order to estimate the algorithmic robustness with outli-
ers, we add a mixed noise model to the training data, which
is assumed to be n(i) = rP(i) + (1 − r)Q(i), where 0 < r < 1
[39]. Here, P(i) and Q(i) are two noise processes both inde-
pendent with each other, and the variances of them are 𝜎P2
and 𝜎Q2 , respectively. In the simulations below, we set Q(i)
as Gaussian distribution with zero-mean and r = 0.06. Mean-
while, we change the 𝜎Q2 to observe the variation of classifi-
cation performance. With respect to the noise P(i),
� √we √ con-
�
sider three cases: (a) Uniform distribution over − 5, 5 ,
(b) Gaussian distribution with zero-mean and 𝜎P2 = 1, and
(c) Binary distribution with Pr{A(i) = 1} = Pr{A(i) = − 1} =
0.5. Usually the variance of Q(i) is much larger than that of
P(i), so Q(i) represents larger outliers here. The generated
noises above are overlapped with the original datasets.
4.2 Simulation results and discussion
We first demonstrate the selections of the parameters in our
proposed SELM-MKRSL, and then show the impacts of
them on classification performance and the outlier robust-
ness. The number of parameters in SELM-MKRSL is
slightly more than that of original S-ELM, due to the use
of KRSL in our proposed approach. Considering that the
distribution of the datasets used in this article is balanced,
and what we concern is the correct classified ratio on all cat-
egories, not on only one specific type, we choose accuracy
to evaluate the classification performance of the different
methods. Specifically, for those datasets used in the binary
classification problem, we use other two criterions, including
precision and recall, to evaluate the performance addition-
ally. As for multi-class classification datasets, the precision
and recall may not have significant meaning to evaluate the
classification performance, hence we just use accuracy as
the measure criterion.
4.2.1 Parameters selection and analysis
For S-ELM, there are only three parameters required to be
adjusted. They are the number of total hidden layers k, the
number of hidden nodes in each layer L, and the equality
constrained optimization-based regularization parameter C.
While for SELM-MKRSL, in addition to these parameters
in S-ELM, there are more parameters to be tuned, including
risk-sensitive parameter λ, kernel bandwidth σ, and another
regularization coefficient η. Here, we just take dataset USPS
for instance, and we set the noise P(i) as Uniform distribu-
tion to demonstrate the process of selecting various param-
eters. In our simulations, we choose the iterative termination
threshold as ε = 10−3, and the number of hidden layers is set
to k = 3 for all the NNs.
Firstly, we estimate the effect of risk-sensitive parameter
λ and bandwidth σ on classification performance. Actually,
there is no efficient way to determine the appropriate param-
eter values, so we just find a relatively good parameter set-
ting within the following bounds: λ ∈ {1, 3, 5} and σ ∈ {0.2,
0.4, 0.6, …, 8}. Figure 2 shows the testing classification
accuracy of SELM-MKRSL with the increase of bandwidth
σ, where the risk-sensitive parameter λ is prefixed. We can
observe that, as σ increases, the classification accuracy
first increases steeply, then tends to converge, and slightly
decreases at last. In terms of λ, the testing accuracy val-
ues have similar variation tendency and nearly overlap with
different λ. Hence, the SELM-MKRSL is not very sensitive
to the risk-sensitive parameter λ. According to the simula-
tion designed above, we set λ as 5 and σ as 1 in the following
simulations on dataset USPS when P(i) is set as Uniform
distribution. Similarly, as shown in Figs. 3 and 4, we can
determine the parameters λ and σ in the same way when
P(i) is set as Gaussian distribution and Binary distribution
respectively.
Then, we need to adjust the regularization coefficient η
in our proposed SELM-MKRSL. The testing accuracy and
training time with the change of parameter η are shown in
Fig. 5. It can be seen that as η increases, the testing accu-
racy does not change very much, just decreases a little. As
for training time, it spends much time to train the learning
network when η becomes large. As a consequence, we set
parameter η as 0.1 in the simulations below on dataset USPS
when P(i) is set as Uniform distribution.
Subsequently, the effect of equality constrained opti-
mization-based regularization parameter C is evaluated.
In the simulations, C is set within {10−20, ­10−19, …, 1­ 019,
­1020}, and the result is illustrated in Fig. 6a. It can be seen
that, the testing accuracy of S-ELM increases rapidly at
first, and then becomes stable and converges in a wide
range of C. Nevertheless, in terms of SELM-MKRSL and
SELM-correntropy, the testing accuracy is almost invari-
able regardless of the variation of C. The reason of this
phenomenon is that parameter C is just used for calcu-
lating the initial output weight β in SELM-MKRSL and
SELM-correntropy. Whether the initial β approximates to
the optimal solution or not will not influence the classifica-
tion performance of these two algorithms, because the iter-
ative updating rule will make the output weight approach
the optimal solution gradually. However, the nearer the
distance between the initial output weight and the opti-
mal one is, the less iterations may conduct to reach to the
13
 International Journal of Machine Learning and Cybernetics

Fig. 2  The relationship between 90

the bandwidth σ and testing
accuracy with different λ in
SELM-MKRSL on dataset
USPS when P(i) is set as uni- 80
form distribution

70

Testing Accuracy (%)

60

50

40

=1
30
=3
=5

20
0 1 2 3 4 5 6 7 8

Fig. 3  The relationship between 90

the bandwidth σ and testing
accuracy with different λ in
SELM-MKRSL on dataset
USPS when P(i) is set as Gauss- 80
ian distribution

70
Testing Accuracy (%)

60

50

40 =1
=3
=5

30
0 1 2 3 4 5 6 7 8

13
International Journal of Machine Learning and Cybernetics

Fig. 4  The relationship between 90

the bandwidth σ and testing
accuracy with different λ in
SELM-MKRSL on dataset
USPS when P(i) is set as binary 80
distribution

70

Testing Accuracy (%) 60

50

40 =1
=3
=5

30
0 1 2 3 4 5 6 7 8

84 160

82
140

80

120
78

76
Testing Accuracy (%)

100
Training Time (s)

74

80
72

70 60

68
40
66
SELM-MKRSL SELM-MKRSL

64 20
0 0.5 1 1.5 2 2.5 3 3.5 4 0 0.5 1 1.5 2 2.5 3 3.5 4

(a) Testing accuracy (b) Training time

Fig. 5  The relationship between the regularization coefficient η and testing accuracy as well as training time in SELM-MKRSL on dataset USPS

13

90
80
70
Testing Accuracy (%)
60
50
40 SELM-MKRSL
SELM-Correntropy
S-ELM
30
-20 -15 -10 -5 0 5 10 15
log C
(a) Regularization parameter C
Fig. 6  The relationship between two parameters and testing accuracy in three algorithms
optimal solution, in an effort to reduce the training time. In
the SELM-correntropy, it replaces the MSE by correntropy
and conducts the iterations to update the output weight β
[45]. In the proposed SELM-MKRSL, the process of itera-
tions using MKRSL criterion makes it attain the optimal
solution for β. Meanwhile, we can observe that the testing
accuracy of SELM-MKRSL converges in a higher position
than S-ELM and SELM-correntropy.
Afterwards, we study the impact of the number of hidden
neurons L on the classification performance. As shown in
Fig. 6b, the testing accuracies of all three algorithms first
slightly increase and then become stable with the increasing
of L. In addition, as we all know, the complexity of training
network will grow large gradually if the number of hidden
nodes increases. Considering the situation mentioned above,
we determine the number of hidden nodes L as 500 on data-
set USPS. Furthermore, it is noted that the testing classifica-
tion accuracy of the proposed SELM-MKRSL is higher than
that of other two approaches as shown in Fig. 6a, b.
4.2.2 Simulation results
In order to demonstrate the outlier robustness of our pro-
posed approach, we conduct the simulations on SELM-
MKRSL with different amplitudes and types of noises. We
use the same noise model as described in Sect. 4.1 and
change the noise variance 𝜎Q2 to observe the impacts of out-
lier variances on classification performance and algorithmic
robustness. Before we conduct the simulations with noises,
13
International Journal of Machine Learning and Cybernetics
82
80
78
76
Testing Accuracy (%)
74
72
70
68
SELM-MKRSL
66 SELM-Correntropy
S-ELM
64
20 0 100 200 300 400 500 600 700 800 900 1000
L
(b) Number of hidden nodes L
we first give the initial results with zero noise, in order to
provide a better insight of the baseline performance. Fig-
ure 7a, b show the comparison results of three algorithms on
datasets USPS and ISOLET, respectively. We can see that,
when there is no extra artificial noise on dataset, the testing
accuracy values of S-ELM and SELM-correntropy
are almost similar and are slightly higher than that of the
proposed SELM-MKRSL with the increase of hidden nodes’
number L. However, when there are noises in datasets, the
classification performance of our proposed method is better
than that of other two algorithms. Figure 8a, b show the
relationship between the outlier variance and testing accu-
racy on datasets USPS and ISOLET, respectively,
� √ √ when� the
noise P(i) is set as uniform distribution over − 5, 5 . The
experimental results on datasets SPECT Heart and Internet
Advertisements are also given in Fig. 8c, d. It is obvious that
the classification performance of our proposed SELM-
MKRSL is nearly stable with a relatively high testing accu-
racy in comparison with S-ELM and SELM-correntropy,
regardless of the changes of outlier variance. As a result, the
proposed algorithm SELM-MKRSL is applicable to the
training of the datasets with noises.
In addition, we also conduct the simulations in distinct
noise settings to evaluate the algorithmic robustness on
different types of noise. Figure 9 shows the relationship
between outlier variance and testing accuracy when the
noise P(i) is set as zero-mean Gaussian distribution with
𝜎P2 = 1, and Fig. 10 shows the relationship when P(i) is set
as Binary distribution with Pr{A(i) = 1} = Pr{A(i) = − 1} =
0.5. As illustrated in these figures, we change the variance
International Journal of Machine Learning and Cybernetics

95 96

94 94

93
92

92
90

Testing Accuracy (%)

Testing Accuracy (%)

91
88

90

86
89

84
88

SELM-MKRSL SELM-MKRSL
82
87 SELM-Correntropy SELM-Correntropy
S-ELM S-ELM

86 80
100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000

L L

(a) USPS dataset (b) ISOLET dataset

Fig. 7  The comparison results of three algorithms on two datasets with zero noise when increasing L (the number of hidden nodes)

of noise Q(i) from 5 to 40, and observe the impact on
testing accuracy. It can be seen that our proposed SELM-
MKRSL is robust to the noise type and can achieve good
classification accuracy compared with original S-ELM and
SELM-correntropy.
The reason why our proposed approach can achieve
good performance is not because SELM-MKRSL starts at a
higher point, but due to the robustness and insensitivity of
KRSL to the noises or outliers. In fact, the SELM-MKRSL
algorithm may not outperform other two methods when the
data is not contaminated with noises. However, while the
data contains noises, SELM-MKRSL performs better com-
pared with S-ELM and SELM-correntropy.
The parameter settings of the three distinct approaches
on four datasets are summarized in Table 1. Furthermore,
the comparison of corresponding testing classification accu-
racy is represented in Table 2, where the highest accuracy
values are in bold. In order to present the generalization of
the proposed algorithm, we conduct the simulations on more
datasets when the noise P(i) is Uniform distribution. All
these datasets are downloaded from the UCI machine learn-
ing repository [56], and the simulation results as well as the
corresponding parameter settings are given in Tables 3 and
4, respectively. As shown in Tables 2 and 4 where the bold
values are the highest accuracy values, we can see that the
testing accuracy values of our proposed algorithm are higher
than those of other two methods, regardless of the types and
variances of the noises. Therefore, the proposed algorithm
in this article is effective and can achieve better classifi-
cation performance. For the binary classification datasets,
the corresponding precision and recall are also presented in
Table 5 where the best values are in bold, when the noise
P(i) is uniform distribution and the variance of the noise
Q(i) is set as 10. We can see that the precision of the original
S-ELM is slightly higher than that of SELM-correntropy
and the proposed SELM-MKRSL. However, the recall of
our proposed SELM-MKRSL is better than that of other two
methods. This may be because that the S-ELM only classi-
fies the sample who has very high probability to be positive
sample as positive one, and mistakenly estimates the sample
as negative one whose probability is not extremely high.
With regard to SELM-MKRSL, we notice that it can achieve
better recall with relatively high precision compared with
other methods. Among those simulation results, the reason
why the precision of Parkinson speech dataset can achieve
100% for all three approaches is just that all the instances in
testing dataset come from the PD patients and they are all
positive samples.
As a conclusion, the simulation results demonstrate that
SELM-MKRSL proposed in this article is more robust to
large noises and can achieve better performance compared
with other state-of-the-art algorithms, including S-ELM
[24] and SELM-correntropy.
4.2.3 Discussion
It is well known that the loss function for a training model
is usually a similarity measure between the predicted out-
put and the desired one. Meanwhile, MSE is one of the

13
 International Journal of Machine Learning and Cybernetics

84 84

82 82

80 80

78 78

Testing Accuracy (%)

Testing Accuracy (%)

76 76

74 74

72 72

SELM-MKRSL SELM-MKRSL
70 70
SELM-Correntropy SELM-Correntropy
S-ELM S-ELM

68 68
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(a) USPS (b) ISOLET

95 100

90
90

85
80
80

70
75
Testing Accuracy (%)
Testing Accuracy (%)

70 60

65
50

60
40
55
SELM-MKRSL SELM-MKRSL
30
50 SELM-Correntropy SELM-Correntropy
S-ELM S-ELM

45 20
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(c) SPECT Heart (d) Internet Advertisements

Fig. 8  The relationship between outlier variance and testing accuracy in three algorithms on four datasets when the noise P(i) is uniform distri-
bution

most common methods to measure the similarity between
two variables owing to its simplicity. Nevertheless, as
researched in [57, 58], second-order statistics (such as
MSE) is not suitable for dealing with non-Gaussian data,
which is very usual in practical applications. Recent stud-
ies in information theoretic learning (ITL) indicate that
the ITL costs can obtain higher-order statistics of data
and achieve better performance than MSE especially on
non-Gaussian data [57, 59]. The correntropy, as a local
similarity measure in ITL, can effectively eliminate the
bad influence of outliers and has been used in many adap-
tive algorithms [60]. However, the performance surface
of the C-loss can be highly non-convex, which is very flat
away from the optimal solution and very steep around the
solution, resulting in poor convergence performance [38].
The KRSL used in algorithm SELM-MKRSL can solve the
problem mentioned above. Its performance surface can be
more convex than that of C-loss, leading to higher accu-
racy and faster convergence speed, and at the same time
maintaining the robustness to noises [39]. In consequence,
the proposed algorithm SELM-MKRSL can outperform
S-ELM and SELM-correntropy on the dataset with noises.

13
International Journal of Machine Learning and Cybernetics

86 86

84 84

82 82

80 80

Testing Accuracy (%)

Testing Accuracy (%)

78 78

76 76

74 74

SELM-MKRSL SELM-MKRSL
72 72
SELM-Correntropy SELM-Correntropy
S-ELM S-ELM

70 70
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(a) USPS (b) ISOLET

100

90
90

80 80

70
70
Testing Accuracy (%)
Testing Accuracy (%)

60

60
50

50 40

SELM-MKRSL SELM-MKRSL
30
SELM-Correntropy SELM-Correntropy
40
S-ELM S-ELM

20
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(c) SPECT Heart (d) Internet Advertisements

Fig. 9  The relationship between outlier variance and testing accuracy in three algorithms on four datasets when the noise P(i) is Gaussian distri-
bution

5 Conclusion and future work
In this article, a novel criterion of training the multilayer
ELM is proposed, and a new robust stacked ELM, i.e.,
SELM-MKRSL, is accordingly designed. Compared with
the MSE used in the traditional ELM, the KRSL is more
efficient to measure the similarity between the target out-
put and the predicted output when the training data are
contaminated with noises, because the KRSL is insensitive
to large outliers and may be little influenced by noises.
The simulation results indicate that the proposed SELM-
MKRSL is superior to other methods and is robust to
various noises with higher classification accuracy. Hence,
the algorithm SELM-MKRSL proposed in this article
enhances the noise robustness characteristic of multilayer
ELM, and it is verified that this approach is applicable for
large-scale data problems with outliers.
Nevertheless, in this article, the noises in the datasets are
artificial and may have limitations in evaluating the actual
performance. In the future, we will continue the study and
apply the proposed approach to some other application
fields, e.g., human action recognition. As we all know, there
are many noises or outliers in the data collected from human
movements, and we will use these practical datasets to fur-
ther evaluate the effectiveness of our proposed approach.

13
 International Journal of Machine Learning and Cybernetics

86 86

84 84

82 82

80 80

Testing Accuracy (%)

Testing Accuracy (%)

78 78

76 76

74 74

SELM-MKRSL SELM-MKRSL
72 72
SELM-Correntropy SELM-Correntropy
S-ELM S-ELM

70 70
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(a) USPS (b) ISOLET

100

90
90

80 80

70
70
Testing Accuracy (%)
Testing Accuracy (%)

60

60
50

50 40

SELM-MKRSL SELM-MKRSL
30
SELM-Correntropy SELM-Correntropy
40
S-ELM S-ELM

20
5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40

Outlier Variance Outlier Variance

(c) SPECT Heart (d) Internet Advertisements

Fig. 10  The relationship between outlier variance and testing accuracy in three algorithms on four datasets when the noise P(i) is binary distri-
bution

Table 1  Parameter settings of Datasets Noise type S-ELM SELM-correntropy SELM-MKRSL

three different approaches on P(i)
four datasets L σ η λ σ η

USPS Uniform distribution 500 3.0 3.0 5.0 1.0 0.1

Gaussian distribution 500 3.0 2.8 5.0 1.0 0.2
Binary distribution 500 2.5 2.0 5.5 1.5 0.6
ISOLET Uniform distribution 200 2.0 1.4 3.0 2.0 1.2
Gaussian distribution 200 3.5 2.2 5.5 2.0 1.6
Binary distribution 200 2.0 1.8 5.5 2.5 0.2
SPECT Heart Uniform distribution 20 4.5 3.6 1.5 0.2 0.8
Gaussian distribution 20 0.5 0.2 1.5 2.2 2.0
Binary distribution 20 0.5 1.0 1.5 0.6 2.4
Internet Adver- Uniform distribution 250 1.5 0.4 1.5 0.6 3.4
tisements Gaussian distribution 250 1.0 1.2 1.5 0.5 3.8
Binary distribution 250 0.5 2.8 1.5 0.6 3.6

13
International Journal of Machine Learning and Cybernetics

Table 2  Performance Datasets Noise type Noise Accuracy (%)

comparison among three P(i) variance
different approaches on four 𝜎Q2 S-ELM SELM-cor- SELM-MKRSL
datasets rentropy

USPS Uniform distribution 5 77.98 76.38 79.32

15 77.13 79.02 81.96
25 79.07 79.12 79.87
35 77.48 77.28 80.02
Gaussian distribution 5 82.46 81.12 82.96
15 81.61 81.12 83.81
25 81.76 81.27 83.96
35 81.91 81.81 83.36
Binary distribution 5 81.51 82.21 84.26
15 81.96 80.47 83.26
25 81.17 80.12 82.96
35 81.17 80.22 82.16
ISOLET Uniform distribution 10 77.16 79.73 82.68
20 77.81 78.51 82.10
30 76.52 78.13 82.62
40 81.14 80.24 82.68
Gaussian distribution 10 83.19 83.19 83.26
20 82.23 82.75 83.13
30 81.85 82.94 84.99
40 79.86 81.08 83.58
Binary distribution 10 83.32 82.49 83.90
20 81.91 82.38 82.49
30 81.23 80.50 81.33
40 81.31 79.15 81.33
SPECT heart Uniform distribution 10 74.87 79.14 89.84
20 87.17 87.70 91.98
30 90.37 91.44 91.98
40 74.33 77.54 91.02
Gaussian distribution 10 71.12 59.89 78.98
20 72.19 67.43 74.33
30 75.40 88.77 89.14
40 77.01 88.24 89.84
Binary distribution 10 80.75 80.21 91.98
20 74.33 59.36 91.98
30 85.03 78.61 91.98
40 77.54 86.10 91.98
Internet advertisements Uniform distribution 10 71.94 72.20 83.27
20 72.86 71.15 82.35
30 70.22 70.36 84.72
40 70.75 71.01 83.40
Gaussian distribution 10 76.81 72.20 82.61
20 74.70 73.25 84.45
30 74.84 71.94 84.32
40 71.41 75.23 85.64
Binary distribution 10 71.54 73.91 84.72
20 73.25 77.60 83.40
30 71.28 69.17 84.06
40 74.04 73.25 85.90

13
 International Journal of Machine Learning and Cybernetics

Table 3  Parameter settings Datasets S-ELM SELM-correntropy SELM-MKRSL

among three different
approaches on six more datasets L σ η λ σ η
when the noise P(i) is Uniform
distribution Balance scale 250 1.0 0.2 2.5 1.0 0.4
Drug consumption 250 0.5 1.0 2.5 0.4 0.6
contraceptive method choice 200 1.5 1.2 2.5 1.6 0.2
Parkinson speech 300 1.0 0.8 0.5 1.4 0.2
QSAR biodegradation 300 1.2 0.6 2.5 1.6 0.2
Seismic bumps 300 1.2 0.8 2.5 1.6 0.2

Table 4  Performance Datasets Noise variance Accuracy (%)

comparison among three 𝜎Q2
different approaches on six S-ELM SELM-corren- SELM-MKRSL
more datasets when the noise tropy
P(i) is uniform distribution
Balance scale 10 74.67 88.00 90.67
20 83.33 87.33 91.33
30 74.00 89.33 89.67
40 75.33 86.00 90.00
Drug consumption 10 72.92 75.04 77.70
20 75.40 78.05 79.29
30 75.75 77.70 78.05
40 74.16 76.81 77.35
Parkinson speech 10 77.98 70.83 85.12
20 64.29 52.38 94.05
30 70.24 58.93 86.90
40 72.02 68.45 83.33
Contraceptive method choice 10 52.01 52.64 53.49
20 51.59 52.22 53.28
30 50.74 52.43 53.07
40 53.49 54.12 54.76
QSAR biodegradation 10 78.69 77.70 80.66
20 77.70 78.03 82.30
30 79.34 76.72 81.31
40 80.98 80.00 81.64
Seismic bumps 10 92.47 92.86 94.01
20 92.60 92.73 93.88
30 91.58 91.84 94.01
40 92.09 93.24 93.37

Table 5  Performance Datasets Metric (%) S-ELM SELM-correntropy SELM-MKRSL

comparison on precision and
recall among three different QSAR biodegradation Precision 82.92 82.73 83.61
approaches for the binary
Recall 85.66 89.53 89.56
classification datasets
Parkinson speech Precision 100.00 100.00 100.00
Recall 47.44 39.41 97.17
SPECT heart Precision 97.25 97.15 93.06
Recall 80.35 86.13 96.98
Seismic bumps Precision 93.64 93.62 93.62
Recall 99.25 99.90 99.96
Internet advertisements Precision 92.01 91.79 84.45
Recall 70.71 71.00 98.24

13
International Journal of Machine Learning and Cybernetics
Acknowledgements This work is funded by the National Key Research
and Development Program of China under Grant 2016YFC0600510,
the National Natural Science Foundation of China under Grants
U1836106 and U1736117, the Key Laboratory of Geological Infor-
mation Technology of Ministry of Land and Resources under Grant
2017320, and the University of Science and Technology Beijing—
National Taipei University of Technology Joint Research Program
under Grant TW201705.
References
1. Serengil SI, Ozpinar A (2017) Workforce optimization for bank
operation centers: a machine learning approach. Int J Interact Mul-
timed Artif Intell 4(6):81–87
2. Elvira C, Ochoa A, Gonzalvez JC, Mochón F (2018) Machine-
learning-based no show prediction in outpatient visits. Int J Inter-
act Multimed Artif Intell 4(7):29–34
3. Alasadi AHH, Alsafy BM (2017) Diagnosis of malignant mela-
noma of skin cancer types. Int J Interact Multimed Artif Intell
4(5):44–49
4. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine:
theory and applications. Neurocomputing 70(1–3):489–501
5. Huang GB, Wang DH, Lan Y (2011) Extreme learning machines:
a survey. Int J Mach Learn Cybern 2(2):107–122
6. Cervellera C, Maccio D (2017) An extreme learning machine
approach to density estimation problems. IEEE Trans Cybern
47(10):3254–3265
7. Iosifidis A, Gabbouj M (2015) On the kernel extreme learning
machine speedup. Pattern Recognit Lett 68:205–210
8. Huang GB, Bai Z, Kasun LLC, Vong CM (2015) Local receptive
fields based extreme learning machine. IEEE Comput Intell Mag
10(2):18–29
9. Zhu H, Tsang ECC, Wang XZ (2016) Monotonic classification
extreme learning machine. Neurocomputing 225:205–213
10. Cao JW, Zhang K, Luo MX, Yin C, Lai XP (2016) Extreme learn-
ing machine and adaptive sparse representation for image clas-
sification. Neural Netw 81:91–102
11. Luo X, Yang X, Jiang C, Ban XJ (2018) Timeliness online regu-
larized extreme learning machine. Int J Mach Learn Cybern
9(3):465–476
12. Mozaffari A, Azad NL (2016) Self-controlled bio-inspired
extreme learning machines for scalable regression and classifi-
cation: a comprehensive analysis with some recommendations.
Artif Intell Rev 46(2):167–223
13. Zhai JH, Shao QY, Wang XZ (2016) Architecture selection of
ELM networks based on sensitivity of hidden nodes. Neural
Process Lett 44(2):471–489
14. Balasundaram S, Gupta D (2016) On optimization based
extreme learning machine in primal for regression and classifi-
cation by functional iterative method. Int J Mach Learn Cybern
7(5):707–728
15. Zhu H, Tsang ECC, Wang XZ, Aamir Raza Ashfaq R (2017)
Monotonic classification extreme learning machine. Neurocom-
puting 225:205–213
16. Ding S, Zhang N, Zhang J, Xu X, Shi Z (2017) Unsupervised
extreme learning machine with representational features. Int J
Mach Learn Cybern 8(2):587–595
17. Alom MZ, Sidike P, Taha TM, Asari VK (2017) State preserving
extreme learning machine: a monotonically increasing learning
approach. Neural Process Lett 45(2):703–725
18. Luo X, Jiang C, Wang W, Xu Y, Wang JH, Zhao W (2019) User
behavior prediction in social networks using weighted extreme
learning machine with distribution optimization. Future Gener
Comput Syst 93:1023–1035
19. Bai Z, Huang GB, Wang D (2014) Sparse extreme learning
machine for classification. IEEE Trans Cybern 44(10):1858–1870
20. Cao JW, Zhao YF, Lai XP, Ong MEH, Yin C, Koh ZX, Liu
N (2015) Landmark recognition with sparse representation
classification and extreme learning machine. J Franklin Inst
352(10):4528–4545
21. Cao WP, Ming Z, Wang XZ, Cai SB (2017) Improved bidirec-
tional extreme learning machine based on enhanced random
search. Memet Comput 5:1–8
22. Lan Y, Soh YC, Huang GB (2009) Ensemble of online sequential
extreme learning machine. Neurocomputing 72:3391–3395
23. Tang JX, Deng CW, Huang GB (2016) Extreme learning machine
for multilayer perceptron. IEEE Trans Neural Netw Learn Syst
27(4):809–821
24. Zhou HM, Huang GB, Lin ZP, Wang H, Soh YC (2015) Stacked
extreme learning machines. IEEE Trans Cybern 45(9):2013–2025
25. Luo X, Deng J, Liu J, Wang W, Ban X, Wang JH (2017) A quan-
tized kernel least mean square scheme with entropy-guided learn-
ing for intelligent data analysis. China Commun 14(7):127–136
26. Miche Y, Bas P, Jutten C, Simula O, Lendasse A (2008) A meth-
odology for building regression models using extreme learning
machine: OP-ELM. In: Proc 16th Eur symposium artif neural
netw—adv comput intell learn, pp 247–252
27. Guo D, Shamai S, Verdu S (2005) Mutual information and mini-
mum mean-square error in Gaussian channels. IEEE Trans Inf
Theory 51(4):1261–1282
28. Lu XJ, Ming L, Liu WB, Li HX (2017) Probabilistic regularized
extreme learning machine for robust modeling of noise data. IEEE
Trans Cybern 48(8):2368–2377
29. Zhang T, Deng ZH, Choi KS, Liu JF, Wang ST (2017) Robust
extreme learning fuzzy systems using ridge regression for small
and noisy datasets. In: Proc IEEE int conf fuzzy syst, pp 1–7
30. Wong SY, Yap KS, Yap HJ (2016) A constrained optimization
based extreme learning machine for noisy data regression. Neu-
rocomputing 171:1431–1443
31. Santamaria I, Pokharel PP, Principe JC (2006) Generalized cor-
relation function: definition, properties, and application to blind
equalization. IEEE Trans Signal Process 54(61):2187–2197
32. Liu W, Pokharel PP, Principe JC (2007) Correntropy: properties
and applications in non-Gaussian signal processing. IEEE Trans
Signal Process 55(11):5286–5298
33. He R, Zheng WS, Hu BG (2011) Maximum correntropy criterion
for robust face recognition. IEEE Trans Pattern Anal Mach Intell
33(8):1561–1576
34. Chen BD, Xing L, Liang JL, Zheng N, Principe JC (2014) Steady-
state mean-square error analysis for adaptive filtering under
the maximum correntropy criterion. IEEE Signal Process Lett
21(7):880–884
35. Xing HJ, Wang XM (2013) Training extreme learning machine
via regularized correntropy criterion. Neural Comput Appl
23(7–8):1977–1986
36. Luo X, Sun J, Wang L, Wang W, Zhao W, Wu J, Wang JH,
Zhang Z (2018) Short-term wind speed forecasting via stacked
extreme learning machine with generalized correntropy. IEEE
Trans Ind Inf 14(11):4963–4971
37. Luo X, Xu Y, Wang WP, Yuan MM, Ban XJ, Zhu YQ, Zhao
WB (2018) Towards enhancing stacked extreme learning
machine with sparse autoencoder by correntropy. J Franklin
Inst 355(4):1945–1966
38. Syed MN, Pardalos PM, Principe JC (2014) On the optimization
properties of the correntropic loss function in data analysis.
Optim Lett 8(3):823–839
39. Chen BD, Xing L, Xu B, Zhao H, Zheng N, Principe JC (2017)
Kernel risk-sensitive loss: definition, properties and applica-
tion to robust adaptive filtering. IEEE Trans Signal Process
65(11):2888–2901
13

40. Luo X, Zhang D, Yang LT, Liu J, Chang X, Ning H (2016) A
kernel machine-based secure data sensing and fusion scheme in
wireless sensor networks for the cyber-physical systems. Future
Gener Comput Syst 61:85–96
41. Serre D (2010) Matrices: theory and applications. Springer,
New York
42. Dwyer PS, Rao CR, Mitra SK (1973) Generalized inverse of
matrices and its applications. J Am Stat Assoc 68:239
43. Huang GB, Zhou HM, Ding X, Zhang R (2012) Extreme learn-
ing machine for regression and multiclass classification. IEEE
Trans Syst Man Cybern Part B Cybern 42(2):513–529
44. Candes EJ, Li X, Ma Y (2011) Robust principal component
analysis? J ACM 58(3):11
45. Schölkopf B, Smola AJ (2002) Learning with kernels: support
vector machines, regularization, optimization, and beyond. MIT
Press, Cambridge
46. Hull JJ (1994) A database for handwritten text recognition
research. IEEE Trans Pattern Anal Mach Intell 16(5):550–554
47. Cole R, Fanty M (1994) UCI machine learning repository. https​
://archi​ve.ics.uci.edu/ml/datas​ets/ISOLE​T
48. Kurgan LA, Cios KJ, Tadeusiewicz R, Ogiela M, Goodenday
LS (2001) Knowledge discovery approach to automated cardiac
SPECT diagnosis. Artif Intell Med 23(2):149–169
49. Kushmerick N (1999) Learning to remove Internet advertise-
ments. In: Proc int conf autonom agents, pp 175–181
50. Klahr D, Siegler RS (1978) The representation of children’s
knowledge. Adv Child Dev Behav 12:61–116
51. Fehrman E, Muhammad AK, Mirkes EM, Egan V, Gorban AN
(2017) The five factor model of personality and evaluation of
drug consumption risk. In: Palumbo F, Montanari A, Vichi M.
(eds) Data science. Studies in classification, data analysis, and
knowledge organization. Springer, Cham
52. Lim TS, Loh WY, Shih YS (2000) A comparison of prediction
accuracy, complexity, and training time of thirty-three old and
new classification algorithms. Mach Learn 40(3):203–228
13
International Journal of Machine Learning and Cybernetics
53. Sakar BE, Isenkul ME, Sakar CO, Sertbas A, Gurgen F, Delil S,
Apaydin H, Kursun O (2013) Collection and Analysis of a Parkin-
son speech dataset with multiple types of sound recordings. IEEE
J Biomed Health 17(4):828–834
54. Mansouri K, Ringsted T, Ballabio D, Todeschini R, Consonni
V (2013) Quantitative structure-activity relationship mod-
els for ready biodegradability of chemicals. J Chem Inf Model
53(4):867–878
55. Sikora M, Wrobel L (2010) Application of rule induction algo-
rithms for analysis of data collected by seismic hazard monitoring
systems in coal mines. Arch Min Sci 55(1):91–114
56. Dua D, Karra TE (2019) UCI machine learning repository. Uni-
versity of California, School of Information and Computer Sci-
ence, Irvine. http://archi​ve.ics.uci.edu/ml
57. Principe JC (2010) Information theoretic learning: Renyi’s entropy
and kernel perspectives. Springer, New York
58. Chen BD, Zhu Y, Hu JC, Principe JC (2013) System parame-
ter identification: information criteria and algorithms. Elsevier,
Amsterdam
59. Chen M, Li Y, Luo X, Wang W, Wang L, Zhao W (2019) A
novel human activity recognition scheme for smart health using
multilayer extreme learning machine. IEEE Internet Things J
6(2):1410–1418
60. Chen LJ, Qu H, Zhao JH, Chen BD, Principe JC (2016) Efficient
and robust deep learning with correntropy-induced loss function.
Neural Comput Appl 27(4):1019–1031
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.