See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/328353465

Securing Big Data Ecosystem with NSGA-II and Gradient Boosted Trees based
NIDS using Spark-3

Conference Paper · October 2018

CITATIONS READS

0 292

1 author:

Gita Donkal
Chandigarh University
4 PUBLICATIONS 20 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Big Data Security View project

All content following this page was uploaded by Gita Donkal on 28 December 2019.

The user has requested enhancement of the downloaded file.

 Journal of Information Security and Applications 43 (2018) 1–11

Contents lists available at ScienceDirect

Journal of Information Security and Applications

journal homepage: www.elsevier.com/locate/jisa

A multimodal fusion based framework to reinforce IDS for securing

Big Data environment using Spark
Gita Donkal, Gyanendra K. Verma∗
Computer Engineering Department, National Institute of Technology Kurukshetra, India

a r t i c l e i n f o a b s t r a c t

Article history: Securing Big Data has become one of the major issues of the exponentially pacing computing world,
where data analysis plays an integral role, as it helps data analysts to figure out the interests and detailed
Keywords: information of organizational and industrial assets. Acts like cyber espionage and data theft lead to the
Hadoop MapReduce inappropriate use of data. In order to detect the malicious content, we propose a model that ensures the
Spark security of heterogeneous data residing in commodity hardware. To verify the correctness of our model,
IDS (Knowledge Data Discovery) NSL KDD Cup 99 dataset is used that has been used by various researchers
Multimodal fusion for working on (Intrusion Detection System) IDS. We incorporate decision-based majority voting multi-
NSGA-II modal fusion that combines the results of different classifiers and facilitates better performance in terms
Multiple classifiers
of accuracy, detection rate and false alarm rate. Moreover, (Non-dominated Sorting Genetic Algorithm)
NSGA-II plays its integral role for the selection of most promising features. Additionally, to reduce the
computational complexity which is again a crucial aspect while processing Big Data, we incorporate the
concepts of Hadoop MapReduce and Spark to ensure the fast processing of Big Data in a parallel com-
putational environment. Our proposed model is able to achieve 92.03% accuracy, 99.38% detection rate
and a testing time of 0.32 seconds. Additionally, we have achieved advantages in terms of accuracy and
testing time of data over the existing techniques that use IDS as a security mechanism.
© 2018 Elsevier Ltd. All rights reserved.

1. Introduction with a lot of variations that confronts difficulties of storing, ana-

The term Big Data is used for the immense collection of
differently-structured data obtained from variety of heterogeneous
sources piled up on storage devices for which data is measured in
peta-bytes and zeta-bytes. The five distinct dimensions associated
with Big Data, termed as 5 Vs are: Volume, Variety, Velocity, Value,
and Veracity [1]. Each one of these dimensions has a key role to
play in the Big Data Management Systems (BDMSs) which is a re-
cent term for managing Big Data Systems (BDSs) [2]. Nowadays,
data can be analyzed for unraveling the covert patterns, unascer-
tained correlations, market-trends, and a lot other beneficial infor-
mation that can help industries and organizations in establishing
more-cognizant commercial decisions [3]. BDSs confront lots of dif-
ficulties in processing and analyzing massive data to retrieve vital
information that could be useful in medical field, sports industry,
business and so forth. Big Data deals with voluminous data stored
on servers and in data centers having a highly complex structure
lyzing and visualizing the data for further results [4].
Enforcing security in Big Data is indeed a challenging task, as
over the past few years, size of data has shot up astronomically.
Also, massive amount of data is generated from heterogeneous
sources and comes in different forms, such as structured, semi-
structured and unstructured that results in crashing of commodity
hardware. Cryptography, tokenization, de-duplication of data are
some of the ways that assist in securing the confidential data from
getting hacked. However, the way in which new cutting edge tools
and techniques are being developed to ensure the safety of our
data, in the similar manner, adversaries are coming up with new
techniques to exploit the susceptibilities present in BDS including
its infrastructure. Security in Big Data scenario includes the secu-
rity of the data generated as well as the infrastructure security,
data availability, user authentication and communication security
[5]. Most of these issues can be resolved using (Intrusion Detec-
tion System) IDS. In this paper, we consider two important aspects
of Big Data.

1.1. Computational challenges in Big Data environment

∗
Corresponding author. The exponential growth in the size of Big Data with variations
E-mail address: gyanendra@nitkkr.ac.in (G.K. Verma). in data structure requires huge computational power and process-

https://doi.org/10.1016/j.jisa.2018.10.001
2214-2126/© 2018 Elsevier Ltd. All rights reserved.
2 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11

Fig. 1. Apache spark components.

Table 1
Comparison between Hadoop MapReduce and Apache Spark [2,8,11].

Functionalities Hadoop MapReduce Apache spark

Data processing
Near real-time processing
Graph processing
Machine learning
Joining datasets
If dataset is larger than available RAM, then
Hadoop MapReduce may outperform Spark
Not good for obtaining immediate insights
Hadoop MapReduce supports HBase for graph
computation
Hadoop MapReduce needs a third-party for the
same
Hadoop MapReduce is better for joining a large no.
of datasets that require a lot of shuffling and
sorting
Up to 100 times fast processing of data in
(random access memory) RAM up to 10 times
fast processing of data in storage
If a business needs immediate insights then
Spark is the best option
Apache Spark has GraphX—an (application
programming interface) API for graph parallel
computation
Spark has (machine learning library) MLlib- a
built in machine learning library
Spark can create all combinations faster

ing time that makes it quite difficult to gain the insights of user’s
data in a few seconds. Moreover, storage for streaming data which
is characterized by huge variation in data rates can go beyond its
capacity. But with the advent of technologies like Hadoop MapRe-
duce [6,7], Hadoop Distributed File System (HDFS) and latest pro-
cessors like i7, Macintosh, supercomputers and mainframes, it is
possible to process Big Data in seconds. As there are many data
models that are not only complex in structure but also associated
with heterogeneous sources that makes the processing and analy-
sis of data even more difficult. Therefore, we need BDS like Spark
[8,9], Hadoop MapReduce that supports data-scalability, facilitate
resilience to failure, fast processing of voluminous data and pro-
vides flexibility, while at the same time cost effective too [2]. A
typical structure of Apache Spark exhibiting its integral compo-
nents is shown in Fig. 1 [10].
Hadoop also includes (Yet Another Resource Negotiator) YARN
in its system which is entirely responsible for the parallel process-
ing of data stored in HDFS. Comparison between Hadoop MapRe-
duce and Apache Spark is exhibited in Table 1.
As BDSs need more processing nodes to handle the load, so a
buffer or a queue is used to process data in chunks. With large size
datasets, evolution of BDS becomes more complicated and less ro-
bust that demands for a fault tolerance system with infrastructure
management along with fast processing of data. To serve this pur-
pose, we use Spark in our model [8,11].
1.2. Security of Big Data
Security attacks are launched almost every day and executed
in various forms like virus, worms, rootkits, spywares, ransom-
ware, cross-site scripting, (Structured Query Language) SQL injec-
tion, buffer overflow, masquerading, Denial of Service (DoS) and
much more. The non-deterministic nature of attacks has made
them a lot complicated to detect and react. A number of systems
and techniques are designed for resolving these security issues
including firewall, IDS, antivirus, packet sniffers, encryption tech-
niques and anti-rootkits [12]. In our model, we choose IDS over
other cyber-security tools for many reasons among which the top-
notch reason is that it covers almost all the attributes of an (In-
ternet Protocol) IP packet and network-traffic that are necessary
to distinguish between malicious and genuine packets including
matching of pre-stored signatures of attacks that reduces the false
alarm rate to an acceptable rate [1,13].
IDS has always been of major concern and a crucial topic of
research when it comes to cyber attacks on computers, worksta-
tions and networks. It is of two types -Signature-based IDS and
Anomaly-based IDS. Researchers are working on each one of them
as well as Hybrid-IDS for the enhancement of their performance
with respect to accuracy, false alarm rate and detection rate that
helps in constructing a more robust and reliable system. In our
proposed model, we train our classifiers using machine learning
for verifying the strings, regular expressions and digital signatures
in case of Signature-based IDS and authenticating the traffic of in-
coming packets for Anomaly-based IDS [14].
To make reinforced IDS a success, we use in-built classifiers in
Apache Spark that are Support Vector Machine (SVM), Gradient
Boosted Trees (GBT), Decision Tree (DT), Logistic Regression (LR)
and Random Forest (RF). We also use NSL KDD cup 99 dataset
[15] which is derived from the KDD cup 99 dataset [16]. We also
pre-process it for our convenience, for example in terms of chang-
ing string values to numerical values for conveniently performing
feature selection procedure. In feature selection phase use NSGA-II
[17,18] which is a Pareto optimal solution-based approach.
Multimodal fusion [9,20] is a process of integrating two or
more input modalities in order to combine them as a complete
command. The integration mechanism can be based on statistical
technique, artificial neural networks, Hidden Markov model, finite-
state transducers or time-stamped lattices, depending upon the
approach. There are basically three approaches to execute multi-
modal fusion including decision-based, recognition-based and hy-
 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11
brid multi-level fusion. We used Decision-based Multimodal fusion
that categorizes the information gained on the basis of decision
from majority.
Following are the noteworthy contributions of our proposed
scheme:
• A multimodal fusion based framework for reinforcing IDS in or-
der to achieve high detection rate and false alarm rate.
• Use of NSGA-II for the extraction of prominent features out of
feature set of 41 attributes in order to improve feature selection
procedure. Feature selection is taken into consideration in order
to improve the overall efficiency of the proposed systems.
Rest of the paper is organized as follows. Section 2 highlights
some of the related work from the past. Section 3 discusses our
proposed model in detail along with the system entities and work-
ing mechanism. Section 4 covers the experimental results of the
same. Finally, Section 5 concludes the paper with future work.
2. Related work
Dietrich et al. [2] mentioned the entire process of handling and
managing Big Data. They performed a detailed and useful analysis
over such a massive data, embedding Hadoop MapReduce archi-
tecture to incorporate fast processing mechanism in Big Data, us-
ing classifiers like Logistic Regression (LR), Random Forest (RF) and
Decision Tree (DT) for the classification of input dataset, and also
enforcing clustering on data chunks level to minimize the load. Ap-
parently, after gaining the insights of Big Data in depth from Diet-
rich et al. [2] work, led to the development of our proposed model.
Giacinto et al. [21] proposed a model which reflects the advantage
of using fusion based pattern recognition approach to network de-
tection when implemented over the collaborative results of multi-
ple classifiers. The experimental results of this scheme motivated
us to work in a similar manner However, we assimilated Giacinto
et al. [21] scheme in a Big Data environment, where we can secure
Big Data environment using multiple IDS.
Zaharia et. al [8] emphasized the significant and efficient use
of Apache Spark for the processing of massive data along with the
concept of parallel computation that assists in the faster process-
ing of Big Data, which is proved by the implementation results.
Apache Spark utilizes the concept of composability which is the
inter-relationships of components in programming libraries for Big
Data, and also encourages the development of conveniently inter-
operable libraries. Sun et al. [22] incorporated their improved ap-
proach on Spark system. Their approach is based on feature se-
lection for Random Forests classifier that utilizes UCI dataset for
proving their propounded approach. They studied the two impor-
tant issues of feature selection that includes—elimination of noisy
features which have no relevance with classification, and elimina-
tion of redundant features.
Kalyanmoy et al. [17] proved the benefits of embodying NSGA-II
for a better feature selection approach which is an improved ver-
sion of the existing NSGA that takes into account the three very
important issues in NSGA viz. high computational complexity of
non-dominated sorting, need for specifying the sharing parame-
ter, and lack of elitism. Tamimi et al. [18] proposed a method to
generate rules for IDS using NSGA-II that utilizes multi-objective
method. DARPA dataset is used to evaluate the experimental re-
sults that eventually lead to the acceleration of rule generation,
and elevating the security factor in rule-based IDS. Sung et al.
[23] used KDD cup 99 dataset and considered the most promis-
ing attributes out of the 41 attributes for carrying out the exper-
iments, which led to the degradation in performance of IDS with
respect to accuracy and detection rate. Thus, we used NSL KDD cup
99 dataset, an upgraded version of KDD cup 99 dataset, for a bet-
3
ter detection of novel attacks on computer systems as there are no
redundant records and no missing values in NSL KDD dataset.
From the experimental results conducted by Thomas et al. [24],
it can be concluded that fusion of multiple sensors can produce
much more accuracy and performance rate than considering their
outputs individually. Kaliappan et al. [25] proposed a scheme that
performs fusion of multiple IDS including both signature-based
and anomaly-based. The experiments show that performance and
accuracy can be improved with reduction of false positive rate by
utilizing the multimodal fusion. C. Ramachandran [20] elaborated
the taxonomies of IDS along with data mining, feature selection
and data preprocessing. He did experimentation using KDD cup 99
dataset, on which he performed pre-processing and feature selec-
tion that included learning and testing phases. He proved in his ex-
perimentation results that fusion IDS structure provides more ac-
curacy and better false positive rate comparatively.
3. The proposed system
Considering the bottleneck of handling computational chal-
lenges in BDSs, we propose a framework where tremendously huge
data can be processed in a very satisfactory interval of time along
with the security of massive data using IDS. Fig. 2 represents our
proposed model for securing a Big Data environment using a mul-
timodal fusion based framework on Apache Spark BDS. A brief de-
scription of entities of the proposed model is explained in forth-
coming sub-sections.
3.1. Proposed system entities
• Hadoop MapReduce [2,26]—Hadoop is popularly known as “Big
Data Handler” as it is a system designed especially for han-
dling the data generated in bulk. We utilize Hadoop 2.6.0 for
the implementation of our model. Collectively, Hadoop refers to
several different techniques such as HDFS, MapReduce, HBase,
Hive, Pig etc. In Hadoop, YARN is responsible for managing
all the associated resources and job scheduling tasks. In cer-
tain scenarios, the data is needed immediately and this data is
stored in HDFS. Hadoop can store data in peta-bytes and zeta-
bytes with no limitation on storage, and when used in collabo-
ration with MapReduce then provides faster computation and
processing of data. MapReduce is a batch-oriented program-
ming model in Hadoop that performs management on data and
scheduling of jobs. MapReduce first splits the data into inde-
pendent chunks that are processed entirely by map tasks in
parallel and later sorted by the framework. Then the output is
given as input to the reduced task. Hadoop and MapReduce col-
lectively manage data.
• Apache Spark [8,7,27]—Spark is a BDS built on Hadoop and
runs on Hadoop Yarn. We use Apache Spark version 2.1.0. Spark
can be standalone or can work in collaboration with the frame-
work of MapReduce and HDFS, out of which we use the lat-
ter one. Spark’s components include SQL, Streaming and ML-
lib (Machine Learning library). Classifiers in Apache Spark are
still evolving. Hadoop clusters are capable of running interac-
tive query, streaming data, and much more simultaneously on
Apache Spark.
• Spark RDD [8,11,28]—Spark supports a programming model
that is very similar to MapReduce however extends it with
“Resilient Distributed Datasets” (RDD) that is a data-sharing
abstraction. Using this extension, Spark is able to capture a
broader range of processing workloads that previously needed
separate engines, including streaming, machine learning, SQL,
and graph processing. It also provides fault-tolerance without
requiring replication. Spark exposes RDD through an API that is
a functional programming in Scala, Java, Python and R.
4 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11

Fig. 2. The Proposed framework.

• Data Preprocessing [15,29]—We use NSL KDD cup 99 dataset
which is preprocessed to some extent. We further preprocess it
using Parse-labeled-point. In MLlib, labeled points are used in
supervised learning algorithms. We use a double to store a la-
bel, so that labeled points can be used in both regression and
classification. Spark.ml package provides machine learning API
built on the data frames that are also becoming the core part of
Spark SQL library. Spark.ml package can be used for developing
and managing the machine learning pipelines, feature extrac-
tion, as transformers and selectors. Moreover, machine learning
lection which is NSGA-II [18] that uses fast elitist NSGA that is
techniques like classification and regression, and clustering are
also provided by this library.
• NSGA-II [17,18]—We use Information Gain (IG) [19] and NSGA-
II for feature selection phase. As in the proposed framework of
Kaliappan et al. [19], IG has been derived using some formulas
and Genetic Algorithm; we also apply the same methodology.
Feature selection is done to filter out less promising and irrele-
vant features for IDS from our dataset.
We incorporate the upgraded version of NSGA for feature se-
 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11
Fig. 3. Output showing best string selected.
Table 2
NSGA-II parameters.
Modeling description Setting
Population size 40
Crossover rate 0.8
Mutation rate 0.2
Binary chromosome length 41
Crowding distance sorting Dynamic
Pareto front Dynamic
Non-domination rank Dynamic
best suited for our Big Data environment. Because of large size
of columnar data and with diversification in attributes of the NSL
KDD cup 99 dataset, NSGA-II can perform feature selection with
a faster rate [17]. Table 2 represents all the parameters of Genetic
Algorithm (GA) and non-dominated algorithm combined. Since GA
provides solutions to all constrained and unconstrained optimiza-
tion problems and it is entirely based on theory of Charles Darwin
that all organisms develop through natural selection of a small por-
tion of inherited variations that boost up the individual’s ability to
compete, reproduce and survive. We have considered population
size to be 40 and the length of binary chromosome is 41 that rep-
resents the 41 features of our dataset. Crossover rate is kept 0.8
and mutation 0.2 because recombination is mandatory to generate
new offspring, however we do not want the whole new popula-
tion with large diversity, therefore crossover to mutation ratio is
kept 4:1.
As we have 41 attributes in our dataset with 1, 25,973 instances,
so it will affect the computational speed of system and complexity
will escalate too, that’s why NSGA-II is the best choice to perform
feature selection. Fig. 3 shows one of the outputs of our program-
ming model, where 17169 rows are tested and chromosomes con-
taining binary values are the resulting outputs, out of which the
best chromosome is selected by NSGA-II.
● Classifiers: Below are the five classifiers used by our proposed
model. We train these classifiers using in-built machine learn-
ing libraries of Apache Spark according to our needs for better
results [30,31].
(i) Support vector machine [23]—SVM is a supervised learning
model that examines data and acknowledges the existence of
patterns in data which is used for linear and non-linear clas-
sification and regression analysis. Besides, it is commonly used
for transduction, novelty detection and semi-supervised learn-
ing.
(ii) Gradient boosted trees [25,32]—There are three elements in
GBT; a loss function to be optimized, a weak learner to make
predictions and an additive model to add weak learners to min-
imize the loss function. There are a few constraints in GBT
that include the number of trees, tree depth, number of nodes,
number of observations per split, and minimum improvement
to loss.
5
(iii) Decision tree [25,33,34]—A rule-based system classifier that
works as follow:
• Best attribute of the dataset is placed at the root of the tree
and then the training set is split into subsets.
• Subsets should contain data with the same value for an at-
tribute.
• Step 1 and step 2 are repeated on each subset until leaf nodes
are found in all the branches of the tree.
(iv) Random forest [12,27,35,34]—RF is a supervised classification al-
gorithm which is similar to bootstrapping algorithms that grow
bootstrap samples from the original data and grow an un-
pruned classification tree and predict the new data by aggre-
gating the predictions of the new tree. Using RF in collabora-
tion with MapReduce for a Big Data scenario produces better
results.
(v) Logistic regression [11]—LR algorithm is used for machine
learning and predictive modeling. It works in scenarios, where
the dependent variable is Binary or Dichotomous. As lots of val-
ues in our dataset are binary, so we use this algorithm to obtain
successful results. It doesn’t require any pick learning rate and
also runs faster.
● Intrusion detection system [20,23,36]—We train our classifiers
in a manner where they can differentiate between malicious
and non-malicious packets. In Signature-based IDS, the patterns
including hash values, signatures, string values and regular ex-
pressions are matched with the values already stored in the
database. Anomaly-based IDS analyzes the traffic and informs
about the nature of traffic that is further distinguished as legit-
imate and illegitimate. Examining the incoming/outgoing traf-
fic, one can determine whether it is DoS attack in which the
traffic of Bots goes from Master to Slave (Zombie machines). In
addition, Probe, R2L (Remote to Local) and U2R (User to Root)
attacks can also be identified using this technique. Anomaly-
based IDS is used for the detection of DoS and R2L attacks,
whereas Signature- based IDS is used for U2R and Probe at-
tacks.
● Multimodal fusion [19–21]—We use multimodal fusion in our
scheme to enhance the performance of the entire IDS, making
it ready for detecting novel attacks and to classify attacks in-
cluding DoS, U2R, R2L, and Probe. As singular-modality won’t
be sufficient to achieve a better accuracy and detection rate,
therefore we instill the idea of merging all the three techniques
using decision-based multimodal fusion which gets its founda-
tion from the approach used by Kaliappan et al. [19] proving
that the false alarm rate and accuracy can be improved by us-
ing this technique. Decision-based multimodal fusion incorpo-
rates the majority voting rule on the outputs generated from
individual classifiers and then decides the output. In our case,
the output is either “Attack” or “Not Attack”.
● NSL KDD 99 dataset [15,18]—This dataset is a newly proposed
dataset that overcomes the issues of existing KDD Cup 99
6 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11
Table 3
Class-wise attack in NSL_KDD cup 99 dataset [15].
Class of attack Train file Test file
Normal 9711 9711
DoS 45927 7456
Probe 11656 2421
R2L 995 2756
U2R 52 200
Total instances 125973 22544
Table 4
Types of attacks in NSL_KDD cup 99 dataset [15].
Attack type Attack pattern
Probe ipsweep, nmap, portsweep, satan, mscan, saint
DoS Back, land, neptune, pod, smurf, teardrop, apache2,
worm, udpstorm, mailbomb, processtable
U2R buff_overflow, loadmodule, rootkit, perl, sqlattack,
xterms, ps
R2L guess_password imap, multihop, phf, spy, warezclient,
warezmaster, ftp_write, xlock, xsnoop, snmpgue,
snmpgetattack, httptunnel, sendmail, named
dataset, such as, it does not include redundant records in the
training set and there is no duplicity of records in the test sets.
Apparently, we can say that it is a reduced version of KDD cup
99 dataset. There are 1, 25,973 records in the training set and
22,544 records in the test set, with forty one attributes. It is the
current upgraded version of KDD Cup 99 dataset which is used
for working on IDS. Table 3 shows the total number of attacks
and Normal packet instances that are present in NSL KDD 99
dataset. Table 4 represents the types of attacks available in NSL
KDD dataset along with their patterns.
3.2. Working mechanism
We will be working on NSL KDD cup 99 dataset that is ap-
parently suitable for implementation of the proposed model. Af-
ter setting the environment for Hadoop MapReduce and Spark we
execute the following steps:
Step I: Initialization phase [37]—System’s configuration is
checked initially, as only upgraded systems are able to sup-
port the parallel processing of massive dataset. First of all,
NetBeans is initiated on JDK8.0 and all the required libraries
are imported to NetBeans. Now, Hadoop [38] is initialized,
that further initiate Yarn, MapReduce and HDFS. Then we
initialize Apache Spark [39] which exists in the same cluster
with Hadoop MapReduce. Hadoop MapReduce and Spark are
in running state now, which further initiates parallel com-
putation. Finally, the initialization phase gets over with NSL
KDD Dataset importation to NetBeans.
Step II: Data pre-processing phase—NSL KDD cup dataset al-
ready comes as a processed dataset as it is being used since
1999 for IDS and have been upgraded as well with no redun-
dant values in training set and no duplicate records in test
set. However, we pre-process in order to achieve a dataset
with fast processing, no redundant values, and no string val-
ues in any column. We use JavaRDD and parseRDD that per-
form index mapping, parsing and labeling, and thus yield
clean data.
Step III: Feature selection phase—In this phase, NSGA-II is
used to extract the best features out of all the 41 features
with diversity in its selection. Using NSGA-II that uses Pareto
optimality [17] and is faster with no over-fitting problem,
select optimum features can be selected that that also pro-
vides multi-objective functions in the execution of an algo-
rithm. We implement our model using both feature selection
and without feature selection in order to distinguish the re-
sults in terms of false alarm rate, detection rate and accu-
racy.
Step IV: Classification phase—Our preprocessed and selected
dataset now goes through classification phase, where classi-
fiers like SVM, GBT, DT, RF, and LR perform the required clas-
sification on the basis of training strategies applied on them.
Then their results are inputted to the next phase. Their out-
puts basically include the classification of an input catego-
rizing it as attack and not-attack. For example, classifying
rootkit under attack category.
Step V: Multimodal fusion phase—In this phase, decision-
based fusion is applied on the inputs received from the pre-
vious phase. As we are considering Big Data scenario, we
need an efficient and highly accurate classification process
which could perform faster computations and facilitates pre-
cise results with low false alarm rate and higher values for
detection rate. So instead of combining all the classifiers, we
combine SVM, GBT, DT, LR, and RF. Outputs from classifi-
cation phase are of two categories; attack and not-attack,
where majority wins; pseudo code is shown in Fig. 4.
Step VI: Output phase—The categorizer (Fig. 4) will decide the
final output on the basis of majority, that further decides
whether it is an attack or not on the basis of number of fea-
tures obtained in Attack (A) and Not-Attack (NA). In Fig. 2,
outputs from different classifiers are denoted by X and Y,
where X represents “it’s an attack” and Y represent “it’s not
an attack” for all the patterns and packets matched in the
NSL KDD dataset. Now, function of a categorizer comes into
action, when it has to decide on the basis of majority of the
result, whether it’s an attack or not. Let us assume if 3 out
of 5 classifiers say it is an Attack and 2 classifiers say it is
Not-Attack, then majority will win facilitating the final out-
put of our model.
4. Experimental results and discussions
We carried out all the experiments on Windows10 platform
on a 64-bit OS and a system with configuration of i5 proces-
sor/2.70 GHz, with 8 GB installed RAM for executing our experi-
ment. NetBeans 8.2 is used as our main platform to run all our
applications including Hadoop MapReduce task (with RDD), and
Spark (RDD). NetBeans assimilated with Hadoop MapReduce and
Spark RDD is used for simulation and analysis of experimental re-
sults.
Firstly, we perform data pre-processing on NSL KDD cup 99
dataset using Java RDD and parse-labeled-point that are in-built
functions provided by Java and Scala. This step lets us work with
our dataset with no setbacks of redundancy, string conversion is-
sues, and chaotic adjustments of instances. We utilize all the at-
tacks which are categorized as malicious packets including DoS,
Probe, R2L and U2R to train our classifiers to predict which packet
is malicious and which one is legitimate. Secondly, IG values are
obtained for different features available in the train set and test
set which is apparently the basis for proceeding with feature se-
lection step are represented in Fig. 5. Reckoning IG is an integral
part of the implementation of this entire model as it helps in fea-
ture selection procedure and then later classifying a given sample.
Declination in the IG values corresponding to the attributes rang-
ing from 1 to 41 for training set can be observed, whereas there
are fluctuations in IG values of testing set.
We employ NSGA-II to perform feature selection on our dataset
utilizing the parameters as mentioned in Table 2 that selects the
most promising features which assist us in procuring the enhanced
performance in respect with accuracy and detection rate. Table 5
provides the description of all the forty one features that are avail-
 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11 7

Fig. 4. Function of a categorizer.

Fig. 5. Information gain (IG) values for (i) Train set, and (ii) Test set.

Table 5
Description of 41 features in NSL KDD dataset.

S.no Feature name Description Type

1. Duration Length (number of seconds) of the connection Continuous

2. Protocol_type Type of the protocol, e.g. tcp, udp, etc. Discrete
3. Service Network service on the destination e.g. http, telnet, etc. Discrete
4. Src_bytes Number of data bytes from source to destination Continuous
5. Dst_bytes Number of data bytes from destination to source Continuous
6. Flag Normal or error status of the connection Discrete
7. Land 1 if connection is from/to the same host/port; 0 otherwise Discrete
8. Wrong_fragment Number of wrong fragments Continuous
9. Urgent Number of urgent packets Continuous
10. Hot Number of hot indicators Continuous
11. Num_failed_logins Number of failed login attempts Continuous
12. Logged_in 1 if successfully logged in; 0 otherwise Discrete
13. Num_compromised Number of compromised conditions Continuous
14. Root_shell 1 if root shell is obtained; 0 otherwise Discrete
15. Su_attempted 1 if su root command attempted; 0 otherwise Discrete
16. Num_root Number of root accesses Continuous
17. Num_file_creat ions Number of file creation operations Continuous
18. Num_shells Number of shell prompts Continuous
19. Num_access_files Number of operations on access control files Continuous
20. Num_outbound_cmd s Number of outbound commands in an ftp session Continuous
21. Is_host_login 1 if the login belongs to the host list; 0 otherwise Discrete
22. Is_guest_login 1 if the login is a guest login; 0 otherwise Discrete
23. Count Number of connections to the same host as the current connection in the past two seconds Continuous
24. Serror_rate Number of connections that have SYN errors Continuous
25. Rerror_rate Number of connections that have REJ errors Continuous
26. Same_srv_rate Number of connections to the same service Continuous
27. Diff_srv_rate number of connections to different services Continuous
28. Srv_count number of connections to the same service as the current connection in the past two seconds Continuous
29. Srv_serror_rate Number of connections that have SYN errors Continuous
30. Srv_rerror_rate Number of connections that have REJ errors Continuous
31. Srv_diff_host_rate Number of connections to different hosts Continuous
32. Dst_host_count count of connections having the same destination host Continuous
33. Dst_host_srv_count count of connections having the same destination host and using the same service Continuous
34. Dst_host_same_srv_r ate Number of connections having the same destination host and using the same service Continuous
35. Dst_host_diff_srv_ra te Number of different services on the current host Continuous
36. Dst_host_same_src_ port_rate Number of connections to the current host having the same src port Continuous
37. Dst_host_srv_diff_ho st_rate Number of connections to the same service coming from different hosts Continuous
38. Dst_host_serror_rate Number of connections to the current host that have an S0 error Continuous
39. Dst_host_srv_serror_ rate Number of connections to the current host and specified service that have an S0error Continuous
40. Dst_host_rerror_rate Number of connections to the current host that have an RST error Continuous
41. Dst_host_srv_rerror_ rate Number of connections to the current host and specified service that have an RST error Continuous
8 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11

Table 6
Selected features based on NSGA-II providing best accuracy.

Classifiers Total features Selected features Selected features Accuracy

SVM 41 30 1, 2, 3, 4, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 35, 36, 37, 39, 40 90.58%
GBT 41 30 1, 3, 4, 6, 7, 8, 10, 12, 14, 16, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 39, 40,41 90.36%
DT 41 29 1, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 29, 33, 34, 39, 40,41 91.04%
LR 41 32 1, 3, 4, 6, 7, 8, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 25, 26, 28, 29, 30, 31, 34, 35, 36, 37, 38, 39, 40,41 89.39%
RF 41 28 1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 18, 19, 20, 21, 22, 24, 25, 26, 30, 34, 35, 36, 37, 38, 39 90.13%
Fusion 41 30 1, 2, 3, 4, 5, 6, 8, 9, 12, 13, 14, 15, 17, 18, 19, 20, 21, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 37, 38, 40 92.03%

able in NSL KDD cup 99 dataset [39]. NSGA-II selects the most Table 7
Calculated accuracy, precision and F-measure.
prominent features from these forty one features. Some of the
parameters of NSGA-II are dynamic and some are set static, for Classifiers Feature selection Accuracy (%) Precision F-measure
instance the crossover and mutation rate in which the ratio of SVM With FS 90.58 0.916 0.903
crossover and mutation rate is kept 8:2, as we do not want es- Without FS 83.38 0.859 0.827
calation in complexity in diversity that will be affected with the GBT With FS 90.36 0.914 0.901
soaring scalability of Big Data. If scalability soars in Big Data envi- Without FS 88.79 0.902 0.885
DT With FS 91.04 0.919 0.908
ronment, it is very likely that complexity will increase too, due to
Without FS 88.68 0.900 0.884
which BDSs become less robust and more complicated to evolve. LR With FS 89.39 0.907 0.891
Table 6 shows the features selected by NSGA-II algorithm, along Without FS 90.69 0.909 0.906
with the accuracy obtained so far in percentage. This experiment is RF With FS 90.13 0.913 0.899
Without FS 90.37 0.915 0.901
carried out for all the attack patterns available in NSL KDD Cup
Fusion With FS 92.03 0.928 0.919
99 dataset that are mentioned in Table 4. We carried out this Without FS 89.95 0.912 0.897
experiment iteratively with various combinations of features, and
we were able to achieve maximum accuracy, detection rate, false
alarm rate and processing speed with some of the most promi- Table 8
nent features filtered out by NSGA-II. It can be articulated from Calculated detection rate (%) for different classifiers.
Table 6 that how different combinations of distinguishable features Classifiers With feature selection (%) Without feature selection (%)
of our considered dataset can generate different accurate values
SVM 99.26 98.02
and allow us to choose a limited number of features out of all 41
GBT 99.14 99.04
features selecting only the most promising features and thus re- DT 99.03 98.85
ducing the burden of all features. Apparently, the number of fea- RF 99.42 99.33
tures selected by multimodal-fusion is 30 out of 41 and these fea- LR 99.29 95.78
tures can be looked up in Table 5. Fusion 99.38 99.34

4.1. Performance evaluation

graph generated corresponding to the data given for precision and
We use confusion matrix to evaluate the performance of our
F-measure in Table 7. Similarly, Fig. 7 can be referred to skim data
implemented proposed scheme on Apache Spark. Confusion matrix
related to accuracy of all the five classifiers.
is created from the total number of True Positive (TP), False Posi-
Fig. 6 shows the experimental results of our proposed scheme,
tive (FP), True Negative (TN) and False Negative (FN) values. The
where the elevation in precision and F-measure can be observed
overall classification performance of IDS is measured by accuracy,
which is obtained after the successful incorporation of feature se-
false alarm rate, precision, F-measure, and detection rate. More-
lection in our scheme using NSGA-II. Eqs. (3)–(5) represents the
over, accuracy is defined by the ratio between numbers of correct
formula for calculating accuracy, precision and F-measure. Table 8
predictions to the total number of predictions made by classifica-
represents the detection rate with and without feature selection.
tion mode [19].
Detection rate can be computed using formula in Eq. (2). Detection
FP rate is computed using formula provided in Eq. (2). Moreover, it is
Falsealarmrate = (1)
TN + FP quite lucid that using feature selection the detection rate achieved
is not only high consolidating fusion but also the DR achieved in-
TP dividually by all the five classifiers is higher using FS.
Detectionrate = ∗ 100 (2)
FN + TP Fig. 7 depicts the increase in accuracy percentage when fusion
is performed utilizing NSGA-II for feature selection procedure. This
TP + TN elevation in accuracy and false alarm rate cannot be observed in
Accuracy = ∗ 100 (3)
TP + TN + FP + FN Kaliappan et al. [17] and Gupta et al. [11] proposed works. Al-
though Kaliappan et al. [17] work is not Big Data environment cen-
TP
Precision = ∗ 100 (4) tric, but the defense mechanism he introduced with the fusion of
TP + FP five IDS has proved that better efficiency and performance can be
achieved.
P recision ∗ Recall
F − measure = 2 ∗ (5) To recapitulate all our experimental results, including the classi-
P recision + Recall fiers used, tools/programming models used along with false-alarm
Table 7 shows the precision and F-measure/F1 score with and rate (formula in Eq. (1)), detection rate and testing time of the en-
without feature selection (FS) up to three decimal values for all the tire execution are depicted in Table 9. Also, after exploring our op-
five individual classifiers and for the fusion. Additionally, it shows tions to choose the best implementation environment, we found
the accuracy for all the five classifiers for both the scenarios “with that eclipse provides a smooth and effective execution of our pro-
FS” and “without FS” in percentage format. Fig. 6 represents the gramming model.
 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11 9

0.94
0.92
0.9
0.88
0.86
0.84 Precision
0.82
0.8
F-measure
0.78
0.76
With FS Without With FS Without With FS Without With FS Without With FS Without With FS Without
FS FS FS FS FS FS
SVM GBT DT LR RF Fusion

Fig. 6. Measurements of precision and F-measure.

94%
92%
90%
88%
ACCURACY

86%
84%
82%
80%
78%
SVM GBT DT RF LR Fusion
With Feature Selecon Without Feature Selecon

Fig. 7. Measurement of accuracy with and without feature selection.

Table 9
Comparison of results of different approaches used in building an Enhanced IDS.

Approaches Model description Big Data Tool/language used Classifiers used Detection FAR Testing
considered rate time(s)

Kaliappan et al. – Decision-based No – Weka tool SVM 95.4 0.7 126.0

[17] multi-modal fusion – Java IBK 99.5 0.3 0.25
– Feature selection- – NSL-KDD Dataset J48 99.6 0.1 1.69
GA RF 99.7 0.2 12.88
BayesNet 93.7 0.3 0.69
Fusion (All) 99.0 1.0 NA
Gupta et al. – No fusion Yes – Apache Spark and LR 60.01 6.6 1.336
[11] – Correlation-based it’s MLlib SVM 58.45 83.1 1.53
and Chi-squared – DARPA KDD’99 Naïve Bayes 0.65 18.6 0.345
feature selection Dataset RF 69.28 3.6 2.242
– NSL-KDD cup 99 GBT 60.85 2.67 1.674
Dataset
Proposed – Decision-based Yes – Apache Spark and SVM 99.26 0.35 0.19
model multi-modal fusion it’s MLlib GBT 99.14 0.21 0.14
– Feature selection- – Scala language DT 99.03 0.19 0.11
NSGA-II – NSL-KDD cup 99 LR 99.42 0.21 0.43
Dataset RF 99.29 0.23 0.22
Fusion 99.38 0.17 0.32
10 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11
Since data is the most important asset these days, therefore,
BDSs developed for securing the entire Big Data environment must
be designed in a manner where the confidentiality, integrity and
availability of data can be maintained. Large networks and data
centers that supports the communication of titanic data needs
to be facilitated with full-fledge security, so that user’s data do
not get tampered by any anonymous user or impersonated device
or person. In Table 9, our scheme is compared with the existing
schemes (Kaliappan et al. [19] and Gupta et al [11]) which exhibits
that the proposed scheme is better in terms of detection rate and
false alarm rate. In addition, a slight improvement in overall test-
ing time is also achieved in our proposed model.
In [19], by adopting multimodal fusion and GA, the model was
able to achieve a 99.0% of detection rate, on the other hand, our
proposed model achieved 99.38% detection rate. Additionally, in
comparison to the model proposed in [18], detection rate achieved
by our proposed model is much higher since the authors did not
use fusion-based approach. Our proposed scheme clearly signifies
that using this methodology, the overall efficiency of the model is
approved significantly. In addition, a slight improvement in overall
testing time is also achieved in our proposed model.
5. Conclusion and future scope
The pace with which data is being generated and transferred
across the globe has inclined the world towards a more cognitive
and knowledgeable era. But with bulk generation of data, hackers
seek to work on new opportunities in order to breach security sys-
tems built with cutting-edge techniques and algorithms. Therefore,
keeping in mind the value and importance of data in our day-to-
day lives, its security should be the primary concern of data scien-
tists.
Our proposed work was inspired by the disadvantages we ex-
plored in some of the existing techniques that include the predic-
tion time and detection rate of Kaliappan et al. [19] approach and
accuracy parameter of other few approaches. We have merged the
benefits of five classifiers available in Apache Spark BDS that led to
the development of our proposed model. Consolidating the outputs
of five different classifiers using multimodal fusion assimilating on
BDS Spark helped us in achieving:
• Better performance of the entire system in terms of false alarm
rate, detection rate, accuracy and faster processing of data.
• We were able to achieve 92.03% accuracy and the highest accu-
racy achieved individually is by DT which is 91.04%.
• Additionally, we were able to procure the high computational
speed by performing the entire execution in 0.32 s.
To proceed with resolving bottlenecks of security and compu-
tation issues in a Big Data environment, maneuvers that optimize
the entire voluminous ecosystem must be taken into consideration.
Since we used Apache Spark which is still under development and
thus do not support many classifiers. Therefore, more classifiers
can be introduced in Spark BDS to obtain more accurate results
and also to have diverse options for better performances of classi-
fiers. For future work, this entire implementation can be executed
on Flink which is faster than Hadoop Map-Reduce and Spark.
Supplementary materials
Supplementary material associated with this article can be
found, in the online version, at doi:10.1016/j.jisa.2018.10.001.
References
[1] Zuech R, Khoshgoftaar TM, Wald R. Intrusion detection and big heterogeneous
data: a survey. J. Big Data 2015;2(1):3.
[2] Dietrich D, Heller B, Yang B. Data science & big data analytics: discovering.
Analyzing, visualizing and presenting data; 2015.
[3] Chen M, Mao S, Liu Y. Big data: A survey. Mobile Netw. Appl.
2014;19(2):171–209.
[4] Ordiano JÁG, Bartschat A, Ludwig N, Braun E, Waczowicz S, Renkamp N,
et al. Concept and benchmark results for Big Data energy forecasting based
on Apache Spark. J. Big Data 2018;5(1):11.
[5] Moreno J, Serrano MA, Fernández-Medina E. Main issues in big data security.
Future Internet 2016;8(3):44.
[6] Wang L, Jones R. Big data analytics for network intrusion detection: a survey.
Int J Netw Commun 2017;7(1):24–31.
[7] Download link (Hadoop): https://archive.apache.org/dist/hadoop/core/. Ac-
cessed on December 2017.
[8] Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, et al. Apache spark:
a unified engine for big data processing. Commun ACM 2016;59(11):56–65.
[9] Download link (Spark): https://www.apache.org/dyn/closer.lua/spark/. Ac-
cessed on December 2017.
[10] https://spark.apache.org/. Accessed on December 2017.
[11] Gupta GP, Kulariya M. A framework for fast and efficient cyber secu-
rity network intrusion detection using apache spark. Procedia Comput Sci
2016;93:824–31.
[12] Suthaharan S. Big data classification: Problems and challenges in network in-
trusion prediction with machine learning. ACM SIGMETRICS Perform Eval Rev
2014;41(4):70–3.
[13] Jonnalagadda SK, Reddy RP. A literature survey and comprehensive study of
intrusion detection. Int J Comput Appl 2013;81(16):40–7.
[14] Terzi DS, Terzi R, Sagiroglu S. Big data analytics for network anomaly detec-
tion from netflow data. In: Computer science and engineering (UBMK), 2017
international conference on. IEEE; 2017. p. 592–7. October.
[15] Dhanabal L, Shantharajah SP. A study on NSL-KDD dataset for intrusion detec-
tion system based on classification algorithms. Int J Adv Res Comput Commun
Eng 2015;4(6):446–52.
[16] Aggarwal P, Sharma SK. Analysis of KDD dataset attributes-class wise for in-
trusion detection. Procedia Comput Sci 2015;57:842–51.
[17] Deb K, Pratap A, Agarwal S, Meyarivan TAMT. A fast and elitist multiobjective
genetic algorithm: NSGA-II. IEEE Trans Evol Comput 2002;6(2):182–97.
[18] Tamimi A, Naidu DS, Kavianpour S. An Intrusion detection system based
on NSGA-II Algorithm. In: Cyber security, cyber warfare, and digital forensic
(CyberSec), 2015 fourth international conference on. IEEE; 2015. p. 58–61.
October.
[19] Kaliappan J, Thiagarajan R, Sundararajan K. Fusion of heterogeneous intrusion
detection systems for network attack detection. Sci World J 2015;2015:1–9.
[20] Ramachandran C. An advanced data processing based fusion IDS structures. Int
J Appl Eng Res 2017;12(21):10929–37.
[21] Giacinto G, Roli F, Didaci L. Fusion of multiple classifiers for intrusion detection
in computer networks. Pattern Recog Lett 2003;24(12):1795–803.
[22] Sun K, Miao W, Zhang X, Rao R. An improvement to feature selection of ran-
dom forests on spark. In: Computational science and engineering (CSE), 2014
IEEE 17th international conference on. IEEE; 2014. p. 774–9.
[23] Sung AH, Mukkamala S. Identifying important features for intrusion detection
using support vector machines and neural networks. In: Applications and the
internet, 2003. Proceedings. 2003 symposium on. IEEE; 2003. p. 209–16.
[24] Thomas C, Balakrishnan N. Improvement in intrusion detection with advances
in sensor fusion. IEEE Trans Inf Forensics Secur 2009;4(3):542–51.
[25] Mulay SA, Devale PR, Garje GV. Intrusion detection system using support vec-
tor machine and decision tree. Int J Comput Appl 2010;3(3):40–3.
[26] Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R,
et al. Apache hadoop yarn: Yet another resource negotiator. In: Proceedings
of the 4th annual symposium on cloud computing. ACM; 2013. p. 5.
[27] Jeya PG, Ravichandran M, Ravichandran CS. Efficient classifier for R2L and U2R
attacks. Int J Comput Appl 2012;45(21):29.
[28] Ezin EC, Djihountry HA. Java-based intrusion detection system in a wired net-
work. Int J Comput Sci Inf Secur 2011;9(11):33.
[29] Hamed T, Ernst JB, Kremer SC. A survey and taxonomy on data and pre-pro-
cessing techniques of intrusion detection systems. In: Computer and Network
Security Essentials. Springer; 2018. p. 113–34.
[30] Singh SP, Jaiswal UC. Machine learning for big data: a new perspective. Int J
Appl Eng Res 2018;13(5):2753–62.
[31] Meng X, Bradley J, Yavuz B, Sparks E, Venkataraman S, Liu D, Xin D. Mllib:
machine learning in apache spark. J Mach Learn Res 2016;17(1):1235–41.
[32] Kevric J, Jukic S, Subasi A. An effective combining classifier approach us-
ing tree algorithms for network intrusion detection. Neural Comput Appl
2017;28(1):1051–8.
[33] Rai K, Devi MS, Guleria A. Decision tree based algorithm for intrusion detec-
tion. Int J Adv Netw Appl 2016;7(4):2828.
[34] Liu H, Gegov A, Cocea M. Rule based systems for big data: a machine learning
approach, 13. Springer; 2015.
[35] Río Del, S López, V Benítez, M J, Herrera F. On the use of MapReduce for im-
balanced big data using Random Forest. Inf Sci 2014;285:112–37.
[36] Landset S, Khoshgoftaar TM, Richter AN, Hasanin T. A survey of open source
tools for machine learning with big data in the Hadoop ecosystem. J Big Data
2015;2(1):24.
[37] Hashem IAT, Yaqoob I, Anuar NB, Mokhtar S, Gani A, Khan SU. The rise of
“big data” on cloud computing: Review and open research issues. Inf Syst
2015;47:98–115.
 G. Donkal, G.K. Verma / Journal of Information Security and Applications 43 (2018) 1–11 11

[38] Shanahan JG, Dai L. Large scale distributed data science using apache spark. In: [39] Dhanabal L, Shantharajah SP. A study on NSL-KDD dataset for intrusion detec-
Proceedings of the 21th ACM SIGKDD international conference on knowledge tion system based on classification algorithms. Int J Adv Res Comput Commun
discovery and data mining. ACM; 2015. p. 2323–4. Eng 2015;4(6):446–52.

View publication stats