Explainable Intrusion Detection For Cyber Defences in The Internet of Things Opportunities and Solutions

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 25, NO.
3, THIRD QUARTER 2023 1775
Explainable Intrusion Detection for Cyber

Defences in the Internet of Things:
Opportunities and Solutions
Nour Moustafa , Senior Member, IEEE, Nickolaos Koroniotis , Marwa Keshk,
Albert Y. Zomaya , Fellow, IEEE, and Zahir Tari
Abstract—The field of Explainable Artificial Intelligence (XAI) I. I NTRODUCTION

has garnered considerable research attention in recent years,
OWADAYS, due to the Fourth Industrial Revolution
aiming to provide interpretability and confidence to the inner
workings of state-of-the-art deep learning models. However, XAI-
enhanced cybersecurity measures in the Internet of Things (IoT)
N (Industry 4.0), communication and manufacturing tech-
nologies have been revolutionised, enabling complex services
and its sub-domains, require further investigation to provide powered by interconnected sub-networks of near-ubiquitous
effective discovery of attack surfaces, their corresponding vec- sensors and actuators, and giving rise to new branches of
tors, and interpretable justification of model outputs. Cyber
defence involves operations conducted in the cybersecurity field industry that aim to reduce production costs and improved
supporting mission objectives to identify and prevent cyberat- efficiency [1]. As everyday life grows increasingly more
tacks using various tools and techniques, including intrusion reliant on digital appliances, it becomes a priority to safe-
detection systems (IDS), threat intelligence and hunting, and guard the confidentiality, integrity and availability of indus-
intrusion prevention. In cyber defence, especially anomaly-based
IDS, the emerging applications of deep learning models require
trial systems/networks, while also ensuring that privacy is
the interpretation of the models’ architecture and the explanation maintained [2]. The Internet of Things (IoT), consists of
of models’ prediction to examine how cyberattacks would occur. interconnected sensors and actuators that are managed locally
This paper presents a comprehensive review of XAI techniques by coordinator devices and allow users to interface with them
for anomaly-based intrusion detection in IoT networks. Firstly, via the Internet. IoT deployments promote dynamic, sense-
we review IDSs focusing on anomaly-based detection techniques
in IoT and how XAI models can augment them to provide trust based and automation which lead to improvements in quality
and confidence in their detections. Secondly, we review AI mod- of life. However, the benefits of IoT are offset by a wide range
els, including machine learning (ML) and deep learning (DL), for of inherent vulnerabilities that can be remotely exploited by
anomaly detection applications and IoT ecosystems. Moreover, malicious actors via advanced cyberattack techniques [3].
we discuss DL’s ability to effectively learn from large-scale
Although several studies [4], [5] have been conducted on the
IoT datasets, accomplishing high performances in discovering
and interpreting security events. Thirdly, we demonstrate recent detection, analysis, identification and mitigation of cybersecu-
research on the intersection of XAI, anomaly-based IDS and rity events resulting in safeguarding non-IoT environments,
IoT. Finally, we discuss the current challenges and solutions of these solutions are not ideal for IoT deployments, due to
XAI for security applications in the cyber defence perspective of the nature of these smart environments. To begin with, by
IoT networks, revealing future research directions. By analysing
our findings, new cybersecurity applications that require XAI design, IoT environments often adopt a distributed architecture
models emerge, assisting decision-makers in understanding and consisting of multiple sub-networks of sensors and actua-
explaining security events in compromised IoT networks. tors, resulting in complex, interconnected and near-ubiquitous
Index Terms—Cyber defence, intrusion detection system (IDS), implementations [4]. Furthermore, due to the heterogeneous
artificial intelligence (AI), explainable AI (XAI), Internet of nature of IoT systems, with deployments consisting of diverse
Things (IoT). device types and due to the lack of common standards that
lead to vendor-specific device implementations, IoT systems
Manuscript received 18 June 2021; revised 20 December 2021, generate high-dimensional and multimodal data, which in
30 May 2022, 17 October 2022, and 3 April 2023; accepted 24 May 2023. turn require big data techniques for effective analysis to
Date of publication 26 May 2023; date of current version 23 August 2023. take place. To address this high-dimensionality challenge,
This work was supported by the Australian Research Council’s Discovery
Early Career Researcher Award (DECRA) under Project DE230100116. Artificial Intelligence (AI), especially Machine Learning (ML)
(Corresponding author: Nour Moustafa.) and Deep Learning (DL) algorithms can be effectively
Nour Moustafa, Nickolaos Koroniotis, and Marwa Keshk are with
the University of New South Wales at Canberra, Canberra, ACT 2612,
utilised across industries to achieve high performances with
Australia (e-mail: nour.moustafa@unsw.edu.au; n.koroniotis@unsw.edu.au; large scales of data sources, as has been illustrated in the
marwa.keshk@unsw.edu.au). literature [6].
Albert Y. Zomaya is with the Centre for Distributed and High Performance
Computing, School of Information Technologies, The University of Sydney,
Sophisticated cyberattacks can cause substantial financial,
Sydney, NSW 2006, Australia (e-mail: albert.zomaya@sydney.edu.au). political and social damage to organisations and entities that
Zahir Tari is with the Centre of Cyber Security Research and Innovation, maintain and manage IoT-based critical infrastructure and
School of Computing Technologies, RMIT University, Melbourne, VIC 3000,
Australia (e-mail: zahir.tari@rmit.edu.au). intelligent environments. According to the cybersecurity ven-
Digital Object Identifier 10.1109/COMST.2023.3280465 tures statistics, global cybercrime costs grow by 15% per year
1553-877X
c 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Tallinn University of Technology. Downloaded on September 04,2023 at 18:38:25 UTC from IEEE Xplore. Restrictions apply.
1776 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 25, NO. 3, THIRD QUARTER 2023
and will reach USD 10.5$ trillion annually by 2025 [7]. For which have evolved to be dynamic and complex, constantly
instance, over the period of the 2020-2021 financial year, improving and seeking to thwart detection by employing vari-
the Australian Cyber Security Centre (ACSC) received 67 500 ations and stealth strategies, such as zero-day and multi-stage
cybercrime reports at an average of one report every 8 min- attacks [16].
utes. The report found that the self-reported financial loss due An IDS is a prominent cyber defence method that identi-
to cybercrime exceeded AUD $33 billion [8]. Microsoft esti- fies diverse cyber attacks, including zero-day and multi-stage
mated that the annual impact of cybercrime on the Australian attacks. As a result, in recent years, considerable focus has
economy reaches $29 billion, which is equivalent to almost 2% been placed on the research and development of IDSs. Due to
of Australia’s GDP [9]. The economic impact of cybercrime the application of AI techniques, AI-based IDSs can achieve
around the globe clearly illustrated the need for countries to high performances with benchmark datasets. Deep learning
invest billions of dollars in enhancing and maintaining cyber- techniques can analyse complex data and learn attack patterns
security infrastructure. For example, according to a report by from datasets to detect zero-day and multi-stage attacks [17].
Atlas VPN, the U.S. government was expected to invest $18.78 Therefore, many ML algorithms have been incorporated into
billion USD for cybersecurity purposes in 2021 [10]. the design of efficient IDSs with high accuracy and low false-
As a result of the potential benefits of compromising positive rates. However, the lack of transparency of the inner
network systems, malicious actors invest considerable amounts workings of DL models results in a lack of confidence by
of money, time and effort in planning, devising and launching administrators and security experts in the decision-making
sophisticated cyber attacks. For instance, zero-day and multi- process of automated DL-based IDS, which can impact the
stage attacks are examples of sophisticated cyberattacks that analysis of the root causes of the detected cyberattacks. The
pose significant challenges to ensuring the security of digital field of explainable artificial intelligence (XAI) has emerged
assets. They have the capacity to bypass traditional security in recent years to address this concern. The primary pur-
mechanisms, especially in the case of IoT, due to their hetero- pose of XAI techniques is to interpret black-box models
geneous, complex and vulnerable nature and a general lack of of ML by providing explanations of their functioning and
coupled with a lack of device design standards. While zero-day predictions [18], [19].
attacks utilise malicious techniques that rely on unknown and Research Motivation – Designing XAI for anomaly-based
generally undisclosed vulnerabilities, multi-stage attacks com- IDSs as a cyber defence mechanism in IoT is a combina-
bine multiple attacking phases, that are not easily detected as tion of difficulties in developing an effective anomaly-based
malicious if considered separately [11]. Advanced Persistent IDS, obtaining explainability by AI techniques and deploy-
Threats (APTs) are an instance of multi-stage cyberattacks, ing them at IoT nodes. The challenges for designing effective
where complex techniques are employed to achieve a stealthy IDSs consist of a comprehensive data source and real-time
take-over of IoT networks [12]. detection. The data source is crucial in anomaly-based IDS
IoT systems, consisting of millions of sensor and actuator as it is used to train the ML model and significantly affects
devices, constitute a considerable percentage of daily global its performance in real-time deployment. However, collecting a
Internet communications, part of which can be classified as comprehensive and high-quality data source, including teleme-
cyberattack network exchanges [13]. As a result, the develop- try data of IoT devices and their network traffic, which reflects
ment of cyber defence techniques for existing networks and all possible types of normal activities, is a considerable chal-
IoT deployments is crucial to ensure the protection of data lenge [3]. Moreover, the development of real-time detection
and information systems against cyberattacks. Cyber defence capabilities is a non-trivial activity, as it requires high com-
is the process of developing cyber threat intelligence and hunt- putational resources to deal with IoT data heterogeneity that
ing that can be combined and orchestrated with Intrusion may result in high false alarm rates [20]. Developing an effi-
Detection Systems (IDS). Cyber threat hunting refers to the cient XAI-based IDS method in IoT that can be accessed and
proactive search for cyber threats in a particular environ- evaluated is another considerable challenge. XAI needs to gen-
ment based on an expert’s knowledge, and forensics [14]. erate explanations for the structure of models and attack types
Although it would be ideal to develop cyber defence mech- inferred by those models.
anisms that could be directly deployed on the IoT, because Existing literature has concentrated on increasing the
IoT systems are often affected by numerous vulnerabili- performance of detection accuracy [21]. However, there is
ties, found in their firmware, hardware or network protocol limited research in developing explainable models for IDSs
implementations, in practice, this is challenging due to IoT’s in IoT for several reasons [22]. First, an XAI-enabled IDS
heterogeneous nature and constrained processing, memory and requires pre-processing actions, such as cleaning data (e.g.,
power resources. Traditional cyber security mechanisms, such removing missing values, normalisation) and fusing records
as anti-malware, firewalls, authentication and encryption, are from multiple data sources, such as telemetry data of IoT
all well-documented and have been shown to be effective in devices and their network communications (i.e., network
conventional non-IoT systems [15]. However, it is insufficient traffic). Features from such data sources would carry mas-
to deploy these mechanisms in IoT systems, as they consist sive values and some may not contribute to the detection
of multiple lightweight IoT devices with inherent vulnerabili- engine [23], thus their inclusion may result in high false
ties that vary greatly from the conventional systems that these alarm rates (i.e., false positives and negatives). The expla-
techniques were originally designed for [5]. Moreover, these nation of the detection engine still focuses on understanding
mechanisms are ineffective against contemporary cyberattacks, the potential structure and hyperparameters of models with
MOUSTAFA et al.: EXPLAINABLE INTRUSION DETECTION FOR CYBER DEFENCES IN IoT 1777
manual adjustments [24] and thus there is a need for optimi- domains and provide a comparison with our study to illustrate
sation techniques to be incorporated that would automate the the value of investigating the trends of XAI-based IDS in IoT
learning process. Additionally, the interpretation of predictions settings.
by trained models is still limited to the adoption of feature
relevance and correlation of security events. However, the
interpretation should be associated with the IoT data sources, A. Related Studies
and their attack surfaces and events [25]. This would be In this subsection, we provide a survey of recent studies
designed using a generic IoT architecture that demonstrates in the multi-disciplinary field of XAI-based cyber defences
cyber attacks against IoT physical elements and application for IoT settings. Specifically, we have explored studies pub-
elements at a network level. To sum up, XAI can be used in lished in top-tier conferences and journals, in the domains of
intrusion detection for IoT devices to provide a clear under- XAI, IDS, cyber defence and IoT between the years 2017 and
standing of the decision-making process of AI techniques. XAI 2022. We review the significant features of XAI-based cyber
can also help identify and correct biases in the system, and defence methods from various published articles as outlined in
ultimately increase the accuracy and effectiveness of IDSs. Table I. Table I lists the contemporary and relevant research
This offers a transparent and interpretable explanation of the studies in XAI-based cyber defence methods in IoT environ-
reasoning behind certain decisions. ments. This table includes four features that can be used to
Research Contribution – This paper presents a comprehen- describe the research studies and how their implementation dif-
sive survey that discusses XAI methods and techniques for fers from each other. The features involve detection methods,
developing explainable cyber defences, especially anomaly- explainability methods, data sources and targeted domains.
based IDSs, in IoT networks. The significant contributions of The detection methods focus while the explainability meth-
this study are structured as follows: ods clarify the XAI methods employed to explain the model’s
1) We demonstrate IDSs focusing on anomaly-based detec- output. The data sources indicate the type of data used for
tion in IoT networks. Different types of IDS are cate- training and evaluating the results of models, while the tar-
gorised and analysed. Their utilisation of these systems geted domains refer to the intended settings and environment
for various IoT environments is discussed, demonstrat- used for implementation.
ing their advantages and disadvantages. Although the presented methods would leverage various
2) We review AI techniques, including ML and DL, and data sources for training and testing purposes, with result-
explain their types. We also discuss how these tech- ing models demonstrating high performance and low false
niques can effectively learn from large-scale datasets to alarm rates, it is challenging to analyse and assess their
discover and explain cyberattack events. efficiency comprehensively without utilising realistic environ-
3) We survey recent studies in the intersection of XAI, ments for real-world executions. For instance, Khan et al. [22]
anomaly-based IDSs and IoT networks. We show presented an XAI technique called transparency relying upon
insights into the intersection of those topics and analyse statistical theory (TRUST) tested using three datasets to
each piece of research. detect cyber attacks in industrial IoT (IIoT) systems. The
4) We also identify current challenges and provide future results demonstrated that the technique could explain new
research directions related to XAI for cyber defences in random examples with an average success rate of 98%, but
IoT networks. The application of XAI in cybersecurity it suffers from explaining zero-day attacks. In [33], [34],
presents several challenges and offers new opportuni- [35], the authors developed explainable cyber defence meth-
ties for further research to develop explainable AI-based ods for recognising cyberattacks in IoT environments. They
IDSs in current IoT networks. developed their models using DL approaches and utilised
This paper is structured as follows. The overview of different explainability methods to explain the prediction
the recently published studies and surveys is presented in decisions. Although they managed to achieve reasonable accu-
Section II. Section III introduces an overview of explain- racy rates, the generated explanations are not enough to
able AI and IoT technologies. Section IV discusses the key understand the complete internal workings of the developed
terms of cyber defences. Section V discusses IDSs and their models.
types. Section VI discusses AI and its role in cybersecurity, Khan et al. [23] developed a recurrent units-based cyber-
along with XAI. Section VIII introduce AI in IoT and secu- attack detection model for the discovery of threats in con-
rity aspects. Section XI discusses research challenges, lessons nected medical systems. They also employed skip connections
learnt and future directions related to XAI for IDS-based cyber to lessen the gradient vanishing issue in recurrent networks.
defences in IoT. Lastly, Section XII concludes the paper. Although their developed model is reliable, it needs to be
tested on medical data from various networks. In another work
in the smart medical domain, Alkhariji et al. [28] investigated
II. R ELATED S TUDIES AND S URVEYS methods for producing and incorporating privacy elements,
This section presents the relevant literature on the main extracted from collected privacy patterns, in IoT-enabled med-
domains that are investigated in this survey. In the first subsec- ical applications, supported by explanations, in a process
tion, we present studies that were found to combine XAI and known as explainable privacy. In their work, the researchers
AI-based IDS, developed for the IoT domains. In the second investigated several privacy-by-design schemes from the liter-
subsection, we review existing surveys in the aforementioned ature and extracted privacy patterns that were then applied to
TABLE I
C OMPARISON OF R ECENTLY P UBLISHED XAI-BASED I NTRUSION D ETECTION IN I OT S YSTEMS
medical IoT application designs to promote privacy. Although Dang [43] proposed an explainable model to understand the
this work incorporated elements of security in the form of pri- predictions of Ml-based models in IDSs. They used different
vacy assurance in the design of IoT in the medical sector, it techniques to explain their model; however, the effectiveness
lacks a cyber defence perspective. of their model can not be easily gauged, as they relied on an
Rahman et al. [29] developed an ML-based model to detect outdated dataset for assisting its performance. Oyekanlu [35]
abnormalities in medical images. They also used visualisa- developed an explainable DL model for IoT networks using the
tion techniques to explain their prediction model. However, osmotic computing method. Although the presented model is
they did not discuss the potential of their work in IoHT- efficient and accurate, the complexity of its utilisation in real-
based healthcare systems. Monroe et al. [31] designed an world setups is very high. Amarasinghe et al. [36] developed
explainable Internet of Health Things (IoHT) framework, a DL-based cyberattack detection model and incorporated
for monitoring health from a smart cities perspective. The explanation methods for its predictions. To achieve prediction
research team employed several techniques to provide explana- explanations, the authors leveraged a layer-wise relevance
tions about their results, however, the experimental accuracy propagation method. Although the method is interesting, this
that their proposed framework achieved was relatively low. research has a similar shortcoming as [43], as the assessment
Nguyen et al. [24] proposed a data-driven model to monitor process was based on an outdated dataset. Liu et al. [38] and
user sentiment in IoT environments. In their implementa- Dang [40] proposed explainable models aiming to improve the
tion, the researchers combined IoT and enterprise data with accuracy and predictability of IDS methods. To achieve XAI
explainability methods to enable the interpretation of their and improve the performance of IDSs, several explainability
results. Although the research compared the performance of techniques were leveraged, such as EDA, BRCG and CEM,
three models (random forest, XGB and LSTM), it neglected however, the applicability of their work in real-world IoT envi-
to address the class imbalance issue in their synthesised ronments is questionable, as this was not investigated by the
dataset. Das and Birant [25] developed an activity recognition researchers. Keneni et al. [37], developed a two-stage model
system based on ML-based methods, and leveraged IoT sen- to enable the interpretation of the decision-making process
sors to assess the efficiency of the resulting model. Although of unmanned aerial vehicle systems. The first stage generates
the presented work is promising, the researchers neglected the rules using the Mamdani fuzzy method, and the second
to investigate the generalisation potential of their method. stage generates a reverse method using the Sugeno approach.
De Luca and Chen [17], proposed a rule-based explainable Although their developed model is robust and accurate, they
model applicable in IoT and robotics programming environ- did not discuss the potential of their work from a cybersecurity
ments. The presented work achieves explainability of student perspective for IoT settings.
programming tasks using the VIPLE language and thus is not It can be observed from Table I that the authors employed
applicable for cybersecurity applications in vast IoT-powered different kinds of explainability methods to explain their model
deployments. decisions. It is also worth mentioning that the most often
TABLE II
C OMPARISON OF R ECENTLY P UBLISHED S URVEY A RTICLES
employed XAI method in the presented work was LIME, explored various studies related to XAI in the social science
as it was more versatile and easier to implement than other field. In [51], [52], [53], the authors described the essen-
methods. Although the inner workings of employed ML tial requirements, objectives and stakeholders for employing
and DL-based detection methods remain opaque, XAI-based core XAI concepts. It is obvious that most of the surveys
explainability modules are shown to be promising methods for have not focused on the IoT domain, which is why we have
comprehending the decisions made by these detection meth- elected to focus on IoT-based and XAI-powered cyber defence
ods. To sum up, XAI-based IDSs have certain limitations when applications.
they are constructed for IoT devices and their networks. It is By reviewing the surveys relating to XAI in the field of
difficult in interpreting the data generated by IoT devices due cyber security, it is clear that research interest is shifting
to the heterogeneous nature of those devices. Another limi- towards developing XAI-enabled cybersecurity applications
tation is related to the lack of standardized protocols, which for IoT deployments [50]. Multiple deep learning algorithms
could limit the interoperability of XAI-based IDSs with IoT have been considered and their architectures adjusted for
devices. Additionally, the limited processing power available the incorporation of XAI approaches to interpret their inner
on many IoT devices can make it challenging to execute processes [55]. Several XAI methods address the challenge
advanced XAI algorithms. Lastly, there is a potential lack of of lacking transparency and interpretability in black box deep
expertise in configuring XAI-based IDSs, which could limit learning models [52]. The need for interpretability arises due
their effectiveness in real-world IoT environments. Therefore, to the complicated internal structures of most DL model archi-
while XAI-based IDS systems reveal promise in enhancing tectures, along with a tendency to prioritise high performance
cyber defences in IoT environments, these limitations have to without ensuring transparency of the prediction process. The
be considered in their implementation. consideration of the opaque norm of complex models has
obstructed their future applications in generating critical deci-
sions in the complex environment of IoT systems, which
B. Existing Surveys has the capacity to place the life and health of humans in
This subsection presents the recently published studies that danger. The reason for investigating and developing accurate
discuss research in the domains of XAI, DL, IoT and cyber explanation methods tailored for cyber defence mechanisms
defence. Table II illustrates the area of study of each survey, is to provide system administrators and decision-makers with
followed by the strengths and weaknesses of the study, to enough confidence to accept or reject the output of such
demonstrate the diverse aspects of XAI models and techniques. predictive models and assist them in measuring the risks
It is observed that research attention towards XAI in various of cyberattacks in IoT systems. This study aims to review
diverse domains is on the rise, as there is an interest in moving and evaluate vital areas of the field of XAI-enabled cyber
away from a “black box” view of AI, towards a more reliable defence systems for IoT settings, provide an analysis of the
and interpretable version of AI, where biases can be more read- most recent literature, unearth existing open challenges that
ily identified, and even non-technical users can understand the affect the development of XAI-powered IDS systems for IoT
process that led to a particular output. By reviewing Table II, networks and propose future directions for research in this
most existing surveys have focused on general IoT and neglect multidisciplinary domain.
to study novel emerging fields of IoT (e.g., smart airports,
smart agriculture, etc.) and investigate methods for ensuring
their cyber defences. For instance, in [47], [54], [55], general III. A N OVERVIEW OF XAI AND I OT T ECHNOLOGIES
concepts and implementations of XAI methods were explored. This section provides a discussion on background concepts
In [48], [49], the authors explained perturbation-based meth- of eXplainable Artificial Intelligence (XAI) and IoT systems.
ods for different input types, specifically for video, natural Furthermore, the value of XAI-based methods and techniques
language, software code, and XAI models for configurations applied in an IoT context is explored. Methods of applying
of wireless networks. Similarly, the authors of [50] discussed XAI models for cyber defence systems in IoT deployments
explainable techniques for healthcare systems to interpret deep are discussed. The incorporation of XAI models can enable
learning architectures and their predictions. In [19], the authors decision-makers to understand how attack surfaces and vectors
TABLE III
S UMMARY OF XAI P ROSPECT IN R ECENT S TUDIES
are discovered on lightweight IoT devices, and which features “black-box” models. The term “black-box” refers to the lack
contributed to this discovery, thus improving the transparency of transparency of many complex ML and DL models, with
of the detection process. With a successful cyber attack, such regard to their internal mechanisms and processes. For exam-
as ransomware or botnet-enabled Denial of Service (DoS), ple, in the case of Deep Neural Networks (DNNs) that may
having a considerable financial impact on organisations, reli- consist of hundreds of neurons and hidden layers, interpreting
able explanations about AI-based detection techniques could the contribution of a single neuron or the role each input fea-
assist in decision-making and the selection of appropriate ture played in the model’s output can be challenging. The
countermeasures. goal of XAI is to augment existing ML and DL models
and improve their robustness [54], according to the principles
of Accountability, Responsibility and Transparency (ART) as
A. An Overview of XAI Methods defined by Dignum [65]. The transparency dimension of ART
XAI is an extension of the AI field, which defines meth- dictates that values, morals and ethical concerns will be taken
ods for explaining the predictions of ML models and renders into consideration when training, evaluating and debugging AI
them interpretable. In the context of XAI, the two terms models.
“interpretability” and “explainability” are often used inter- The growing attention to the XAI field is owing to the con-
changeably by the researcher community [54] however, a temporary shift of research interest, where XAI has increas-
distinction can be made. Miller [19] defined “interpretability” ingly become an indispensable theme in conferences such as
as the degree to which a user, regardless of their technical ICCV [66], BMVC [61], and ICML [59]. Besides, it has
competence and level of expertise in any given field, under- also been the main area of discussion for numerous special
stands a decision or prediction of any problem in a field. issues in top-tier conferences and journal venues between 2019
Interpretability implies the interpretation of model processes and 2021, listed in Table III. XAI is primarily tasked with
to the extent that users can discern the cause-and-effect rela- developing white-box models that enable decision-makers to
tionship between changes in input and the resulting output. understand how models already classified events such as cyber-
On the other hand, explainability corresponds to the capac- attacks. Undeniably, at the time of writing, the two main
ity to understand how the internal mechanisms of a model contributors in the XAI domain are the DARPA [67] and ACM
affect its output, thus understanding what role internal nodes FAT [68]. DARPA sponsored a project in the XAI field to
play in the construction of a model’s output. The first appear- implement novel models that can clarify AI schemes, contain-
ance of explainability in the context of AI was in the 80s, ing the interpretability of complex security applications. ACM
with the publication by Moore and Swartout [18] however, FAT endorses the development of AI models, which have fair-
the majority of research interest at the time was on improv- ness and interpretability via analyzing those models’ economic
ing model performance rather than developing explainability and social impacts.
methods. With the recent proliferation of AI in a variety of
domains, including cybersecurity, and the increased reliance on
AI-powered automated services, more research attention has B. IoT Ecosystems and XAI Importance in IoT
been placed on developing XAI-related topics and providing Although lacking an official and universally accepted defi-
confidence in the performance of AI. nition, the IoT can be thought of as a network of networks of
In contemporary literature, it is common practice to refer portable, near-ubiquitous devices, classified as sensors, actu-
to existing AI models that lack XAI-derived explanations as ators or a hybrid of the two, that can be leveraged to enable
sensory-based automation and enable their users to remotely IV. A N OVERVIEW OF C YBER D EFENCE
manage them. The rise of the IoT has resulted in the emer- Cyber defence is a broad term that refers to the set of prac-
gence of the novel smart environment domain, with prominent tices, processes, and technologies used to protect computer
sub-domains including smart airports, and agricultural and systems, networks, and data from unauthorized access, attack
industrial settings. In these new fields, rooms, buildings or or damage. It involves the use of various security measures
even entire cities can be interconnected via IoT deployments, such as firewalls, intrusion detection and prevention systems,
with constant streams of data generated, exchanged, collected encryption, and antivirus software to prevent cyber threats
and analysed to improve services and processes. Evolving such as malware, phishing, and hacking attacks. Additionally,
technologies, which are used to construct IoT devices, have cyber defence involves continuous monitoring and analysis
resulted in the development of low-powered, low-cost proces- of network traffic and system logs to identify, contain, and
sors, wireless networking, artificial intelligence, and mobile respond to potential security incidents or breaches. Overall,
computing [69]. Together, these technologies make commu- cyber defence plays a critical role in ensuring the security and
nication between people, processes, and things much more integrity of digital assets in today’s interconnected world.
accessible, improving the quality of life for humans and Over the last decades, there has been a significant increase in
enhancing efficiency, reducing costs and promoting interoper- computing and digital appliances, as they are quite versatile,
ability for existing business processes. Therefore, technologies providing among other things effective means of communi-
related to IoT systems have advanced rapidly, making them cation, information management, research/development, data
one of the fastest-growing sectors in computing [3]. analytics and more. Ultimately, these devices achieve an
As AI-based models are becoming an integral part of increase in efficiency and promote interoperability by con-
commonly accessible services and everyday processes, and necting the virtual and physical worlds, enabling users to
humans are relying more on such automated decision-making review collected data and remotely manage monitored systems.
automata, it becomes evident that accountability needs to be Consequently, humans’ daily lives heavily depend on these
shared between users and these automated models. However, computing networks, applications, or devices [71]. As people
defining accountability in the context of ML become more dependent on the Internet and digital appliances,
DL model is non-trivial, as they rely on fundamental math- security issues become more popular and have severe con-
ematical principles and the quality of training data during sequences. The privacy and security of information systems
training, and by default, they lack clear-cut processes for need to be a priority for any organization. In [72], the authors
justifying their output. Further, it is not easy to define the com- defined cybersecurity as the security and privacy of dig-
plex learning process of an AI-based method using existing ital assets, which are everything from computer networks
tools [54]. to mobile devices and data that is processed, stored and
There is a growing need for new XAI-based technologies transferred by interconnected information systems.
to comprehend how and why a particular AI model makes
the prediction decisions and how they compensate for the het-
erogeneous nature of IoT deployments and their respective
networks. This explanatory process is vital in critical environ- A. Cyber Security Concepts
ments, such as IoT-enabled medical, automotive, industrial and Cyber security’s goal is to preserve the integrity, con-
critical infrastructure settings. For instance, a woman was hit fidentiality, and availability of information and systems in
by an unmanned vehicle (car) which caused her death [70]. cyberspace. Cybersecurity refers to the processes, guide-
This example refers to an Uber incident in which a car hit lines, technologies, and practices of defending cyberspace
and killed a pedestrian, initiating the first human death that from malicious activities. Cyberspace is a global domain in
happened due to an AI-powered vehicle. The need for AI which electronic and electromagnetic spectrum combined with
models to be explainable and their actions accountable can interconnected and dependent networks help create, edit, store
be illustrated through a real-world example from 2018 [70]. and transfer information [73], [74]. The term cyber defence
This incident, during which an Uber unmanned vehicle was can be defined as the procedure of developing threat hunting
involved in an accidental collision with a pedestrian, result- and intrusion detection methods and techniques to protect crit-
ing in her death, constitutes the first instance of human death ical infrastructures of organisations [20]. In this essence, cyber
caused by an AI-powered vehicle. Incorporating XAI mod- defences should discover and prevent security risks in IoT by
els in real-world unmanned and IoT-enabled systems could following, and achieving, general security objectives, includ-
assist in answering the following questions: 1) was the inci- ing confidentiality, integrity and availability (CIA triad) [75].
dent caused by an error in the AI-enabled decision-making We have summarised the key terminologies involved in the
core of the automated vehicle?, when vehicles are involved in cyber security field as follows:
accidents causing fatality?, 2) is the car operator at fault?, 3) is • Vulnerabilities are flaws or weaknesses in a system
the car manufacturer at fault?, and 4) is the pedestrians or the that attackers can exploit by executing malicious com-
passengers at fault?. It is an upsetting process to resolve who is mands, accessing data without authorization, or conduct-
accountable for this intensely weighty and ethical setup. This ing denial-of-service attacks [76], [77]. More generally,
is why an understanding of the conclusion basis of the AI vulnerabilities refer to any components of the exploitable
system is essential to assure the reliability and accountability information system and threaten the whole system’s
of a system. security.
Fig. 1. Seven phases of cyber kill chain, and their need for cyber defences.
• Threats are actions that can be taken from existing vulner- delivery, exploitation, installation, command and control, and
abilities in a system to gain benefit [78]. Different from actions on the objective, as depicted in Figure 1.
vulnerabilities, threats would involve outside elements. In the reconnaissance phase, attackers engage in information
• Cyberattack/intrusion is a set of intentional actions gathering, identifying open ports in target machines, vulner-
taken to exploit an information system using differ- able services, network misconfigurations and user credentials
ent techniques to compromise the system’s confiden- through social engineering. Next, in the weaponisation phase,
tiality, integrity, and availability. It achieves malicious malware is constructed to exploit vulnerabilities and secu-
goals [79]. Methods to launch a cyber attack can be rity weaknesses that were detected during the reconnaissance
malware, phishing, SQL injection, Man-In-The-Middle phase. The malware is then paired with a benign-looking pay-
(MITM), etc. load in the delivery phase before it is transferred to the target
• Zero-day attacks can be defined as variants of known machine. Once the payload has reached the target machine,
attacks [80] or unknown attacks that exploit zero-day vul- the exploitation phase is initiated, where the malware exploits
nerabilities in the system. Zero-day vulnerabilities are a vulnerability and is afterwards capable of executing arbitrary
unknown to the public and are not yet discovered by code. During the installation phase, a program is installed that
software vendors or network defenders. can establish a persistent channel of communication with the
• Multi-stage attack (MSA) is a type of intrusion consisting attacker. Command and control are established after a suc-
of a sequence of correlated techniques [11]. MSA uti- cessful installation phase, enabling the attacker to access the
lizes more complex attack techniques over a long period, target machine. The final phase of the cyber kill chain is action
where each step taken is insufficient to be recognized as on the objective which, depending on the attacker’s motives,
aggression to compromise the victim. may result in activities such as data exfiltration, keylogging,
• According to the National Institute of Standards and credential theft, loss of data, installation of additional mal-
Technology (NIST), APTs are adversaries which ulti- ware on target, pivot point to compromise other machines
mately aim to steal information, undermine or impede in the vicinity, privilege escalation, orchestration of DoS and
critical aspects of a mission, program, or organization, or DDoS attacks and damaging the target machine (file system,
place themselves in a position to do so in the future. OS, hardware). This model supports security experts in com-
To do that, APTs establish and extend their presence prehending and fighting against different cyber attacks and
in information systems by using a wide range of attack discovering them at any stage using cyber defences.
surfaces with sophisticated levels of expertise and signif-
icant resources. APTs typically launch the attack through
multiple steps; they are persistent, targeted attacks on a B. Summary and Insights
particular organization [81]. Therefore, APT attacks are This section introduced fundamental cybersecurity concepts,
a complex version of multi-stage attacks. that underpin the discussion and development of cyber defence
Numerous cyber defence frameworks exist in the literature, solutions focusing on the detection of cyberattacks in complex
with some prioritising cyber resilience and others mitiga- networked environments such as the IoT. Initially, the basic
tion. However, in this work, we primarily focus on cyber goals of cybersecurity and cyber defence are provided, focus-
defence measures based on IDS. One prominent example of ing on thwarting malicious actors that seek to compromise
a cyber defence model is cyber kill chain [82], which illus- the confidentiality, integrity and availability of a comput-
trates the common activities of advanced cyberattacks and erised system. Furthermore, basic background concepts such
is the fundamental security feature of any cyber defence as vulnerabilities, threats, zero-day and multi-stage attacks
framework to defend against vulnerabilities and their attack are introduced in this section, as they are integral concepts
vectors. According to the cyber kill chain model, as defined by to understand for the rest of this manuscript due to their
Lockheed, advanced malicious adversaries that seek to com- importance in devising defence solutions.
promise a target machine, such as an IoT device or network, Building upon these concepts, the section next introduces
rely on seven stages, including reconnaissance, weaponization, the concept of cyber defence frameworks and models from
Fig. 2. Methods of IDSs and their environments.
an IDS perspective, focusing on the cyber kill chain as a A. Intrusion Detection Techniques
case study of an APT multi-stage threat. According to the Intrusion detection is defined as the process of monitoring
structured approach that APTs display, designed so that each the events occurring in a computer system or network and
phase enables the next one by providing it with services and analysing them for signs of intrusions [83]. According to the
information, it is surmised that for an IDS to be effective Bace and Mell [83], an intrusion can be considered any activ-
in safeguarding IoT environments, it needs to be capable of ity that seeks to compromise a system’s CIA triad or bypass
detecting such organised cyber threats at any stage where con- security mechanisms that protect digital assets, by employ-
tact is established between the attacker and the victim. As ing strategies that exploit security weaknesses, logical flaws
such, to be able to detect suspicious behaviour or malicious and misconfigurations in devices and software. IDS aims to
activities taking place in an IoT deployment during the phases observe data audits and identify possible threats in the network
of the cyber kill chain, excluding the weaponisation phase as and computer system by supervising, identifying and evaluat-
this is carried out on the attacker’s side, an IDS needs to be ing their violations of the security principles [84], [85]. The
configured to receive data from various and diverse sources, methods through which intrusion detection is achieved, IDS
including telemetry data from the IoT sensors and actuators, designs based on ML/DL models and their deployment envi-
network data, computer logs and process activities. ronments are presented in Figure 2 and briefly defined in this
section.
• Anomaly-based detection – This detection method mon-
V. I NTRUSION D ETECTION S YSTEMS itors the activities of data sources to construct a baseline
Due to the heavy reliance on information systems, vari- of legitimate behaviours. An anomaly detection system
ous approaches have been deployed to protect information constructs a profile of normal activities and considers any
systems, such as access control, firewall, anti-malware, sand- variations as anomalous behaviour. These systems would
box and cryptography [15]. Those are traditional mechanisms be static or dynamic, and could be based on a variety
that have been widely adopted nowadays; however, due to the of mechanics, including counting the number of packets
rapid evolution of attack techniques and zero-day vulnerabil- sent, the number of failed attempts to log in, and many
ities, such methods are insufficient for securing information others [86]. Whenever any activities have deviated from
systems. Thus an IDS, the primary method capable of identi- the constructed normal baseline, an alarm would be gen-
fying a wide range of cyberattacks, has become a necessary erated to alert the admin [87]. Therefore, if the baseline
addition to any organisation’s security infrastructure. of normal behaviours is built carefully, anomaly-based
IDS can detect any type of attack, including both known • Network-based IDS (NIDS) – monitors and inspects
and zero-day attacks [88]. activities in the network by reading all inbound pack-
• Signature-based detection – This detection method recog- ets in the entire network where it is deployed [95].
nises possible intrusions by comparing patterns against NIDS do not need to use system resources [91] and can
captured signatures of known attacks. Each type of monitor network activities over a particular network seg-
attack is associated with a specific pattern called sig- ment regardless of the type of the operating system [96],
nature [86]. The signature-based detection method relies which renders them platform-independent and portable.
on the database consisting of the signature of existing However, one major drawback of NIDS is that they can-
attacks to detect them. As a result, all the existing attacks not process encrypted or obfuscated payloads since it
stored in the database would be detected with high con- primarily leverages information extracted from packet
fidence. However, zero-day attacks or even variants of headers [97].
known malicious behaviours would easily bypass the • Cloud-based IDS – typically has several different places
signature-based IDS. Moreover, the database containing of deployment. Host-based IDS can be deployed in
the signature of known attacks requires to be updated virtual machines (VMs) hosted on a cloud server or
repeatedly by network security experts; otherwise, this placed in the hypervisor to monitor network traffic and
method is not effective [89]. information transfer between VMs. Network-based IDS
• Hybrid-based detection– this detection method combines can be deployed to detect abnormalities in the virtual
anomaly-based and signature-based detection methods. network traffic or to monitor unencrypted network traffic
Therefore, this method of detection combines the ben- between VMs [98].
efits of both anomaly and signature-based detection. • IoT-based IDS – can be classified into two approaches
The hybrid-based method depends on the signature-based based on the IDS place of deployment. Centralised IDS is
module to detect known attacks, lowering the false alarm the most widely-used [99] method in which a dedicated
rate. At the same time, the anomaly-based module con- central unit such as a cloud is utilised to monitor and
structs the baseline of normal behaviours in network determine the traffic data in IoT. Centralised IoT-based
traffic; thus, it helps detect zero-day attacks [90]. IDS offer the advantages of powerful computation capa-
bilities and centralised management of network traffic;
however, it degrades the network performance by gener-
B. IDS Deployment Types ating significant communication overhead [100]. Due to
Due to the increasing complexity of intrusion techniques, the increasing data traffic that IoT devices generate, this
signature-based IDS can be easily bypassed by attackers util- method gradually becomes unsuitable and is replaced. To
ising zero-day attacks. On the other hand, anomaly-based IDS address these challenges, decentralised IDS distributes
are capable of detecting zero-day attacks without prior knowl- the centralised computing and computational tasks to
edge, a significant advantage over the signature-based model. local fog nodes. Therefore, the heavy load of monitor-
Thus, this paper focuses on anomaly-based IDSs to build effi- ing traffic data is decreased, and the processing capacity
cient, explainable AI-based IDSs with high detection accuracy is increased [101].
and low false-positive rates. After the development of an IDS • Mobile-based IDS – has gained popularity in recent
is completed, the next step is to define the location where the years due to the emergence of smartphones. Mobile-
system will be deployed. Based on the place of deployment of based IDS can be further classified into three categories,
IDS and the type of system the IDS protects, multiple types of 1) host-based: the IDS is deployed on the mobile device,
IDS have been developed and improved through considerable 2) centralised: IDS that is deployed within the cloud and
research effort: monitors the mobile devices, 3) distributed: IDS is partly
• Host-based IDS (HIDS) – is attached to the operating deployed on the cloud and partly on the device [102].
system kernel of a specific host and protects the host by Cloud computing allows centralised data collection and
forming a layer that allows only legitimate (identified as processing; thus, deploying IDS on the cloud is con-
non-malicious) requests to go through. In other words, venient and practical to employ powerful computation
HIDS is attached to a single host and watches for mali- power and memory capacity. However, relying on a cloud
cious activities. In HIDS, computational resources that server has two limitations, including continuous con-
power the host-based system are taken from the attached nectivity to the central server and the risk of sensitive
host. Host-based methods are reactive, which means that information leakage [103].
they alert the host after an attack has occurred [91]. Defence responses are actions taken after an attack is
Moreover, it might be exposed if the host server is com- detected. Within the context of IDS, there are two types
promised and it may not be compatible with different of defence responses, including passive response and active
platforms [92]. However, HIDS is useful for identifying response. In passive response, the network administrator must
malicious activities in an organisation’s internal equip- take manual action, after the IDS raise an alert about mali-
ment by monitoring system calls, processes, file-system cious behaviour. The form of the alarm would be a popup
changes and application logs [93]. Moreover, it can also window or an onscreen alert [20]. The active responses refer
analyse the encrypted or obfuscated payloads on its to a set of actions taken automatically by the IDS that changes
network interfaces [94]. the behaviour of an intrusion, such as terminating connections
and attacks [104]. The most popular and widely adopted cyber being explicitly programmed to perform the task [109]. ML
defence mechanisms rely on IDS for the initial attack detection models have been employed for a variety of tasks in multiple
and triggering of secondary defensive response actions. fields, including cyber defence. ML techniques have been
investigated to optimise different real-world systems, includ-
C. Summary and Insights ing image recognition, virtual reality, object detection, natural
This section introduces IDS, providing a discussion of the language processing, malware filtering, and many others [110].
basic techniques that are employed for their design and the Based on the nature of training data or the learning techniques,
various deployment schemes that have emerged in recent years. ML algorithms can be classified into different types briefly
Initially, to the underline techniques that are employed for the defined below.
• Supervised learning – The training data has labelled
detection of intrusions IDS are classified as either: signature,
anomaly or hybrid based. Due to the increased complexity inputs and their desired outputs. After being trained, the
of contemporary cyberattacks and the tendency of hackers to ML model can capture the relationships and dependen-
bypass signature-based detection methods by altering the source cies between the prediction output and the input features,
code of their malware and employing other obfuscation tech- giving correct labels for the features of unknown samples.
niques, anomaly-based IDS are identified as a viable option for Supervised learning problems grouped into classification
an IDS designed for deployment in IoT settings. Specialisations and regression problems.
• Unsupervised learning – The training set only includes
of IDS have been proposed, with regards to their placement
in an active environment, including host, network, IoT, Cloud inputs without the desired output. An unsupervised model
and mobile. For an effective IDS to be designed for a complex is tasked with identifying the relationships and patterns
and interconnected IoT setting, a combination of data sources in data, without relying on explicitly provided labels.
from the aforementioned settings is required, to ensure that an Specifically, the system arranges data into categories or
intrusion is captured as swiftly as possible. clusters from the offered training figures and input pat-
terns, according to some strategy that often relies on
VI. A RTIFICIAL I NTELLIGENCE FOR C YBER S ECURITY calculating similarity metrics among the provided data
points. Therefore, it is also considered as self-organizing
Artificial Intelligence (AI) is a branch of computer science and adaptive learning [111]. There are two main unsu-
referring to algorithms that seek to simulate human-like intel- pervised learning problems, including clustering and
ligence by leveraging advanced mathematical concepts and association problems.
observations from nature (human brain structure). In the last • Semi-supervised learning – This is a combination
decade, the field of AI has seen such rapid growth that between supervised learning and unsupervised learning
AI-based algorithms have been developed in every techno- in which the training set includes both unlabeled data
logical and industrial domain, transforming in the process the and labelled data. Semi-supervised learning is helpful
way we approach real-world tasks. In the context of cyberse- in many scenarios when data collecting is expensive,
curity, the utilisation of ML/DL techniques has resulted in the time-consuming or even unrealistic [112]. This approach
development of powerful and versatile cybersecurity defence can utilize the unlabeled data to improve the learning
tools, with two such examples that have become common- accuracy [112].
place in recent years being malware/intrusion detection and • Reinforcement learning – There is no training dataset in
cyber events classification. reinforcement learning. The model is trained in a dynamic
With regards to malware detection, an important task car- environment where it interconnects with its surroundings
ried out by anti-malware software, ML has been demonstrated by employing trials and then receiving rewards or penalties
to be capable of generalising to previously-unknown mal- depending on its actions and the selected objective function.
ware families and polymorphic strains, while also enabling The data feedback is crucial for the model to learn from
IDS to detect zero-day attacks [105]. Similarly, it allows experience and improve performance. This type of learning
anomaly-based IDS to detect zero-day attacks. Lavanya and is motivated by behaviourist psychology [111].
Sriram [106] used DL techniques for insider threat detection
by analysing system logs to detect malicious activities. In the
field of digital forensics, ML methods have been employed B. Deep Learning
for file fragment classification [107] and kernel rootkits detec- Deep learning is a subset of machine learning whose archi-
tion in virtual machines [108], thus assisting experts during a tecture is motivated by the structure and functioning of the
forensic investigation. Moreover, Sarker et al. [15] proposed a human brain. Deep learning architecture consists of multiple
security intelligence modelling using the combination of var- layers each of which is made up of numerous neurons. Each
ious AI methods and other techniques. The modelling can be neuron is considered a fundamental computational component,
used in multiple cybersecurity challenges to protect against and the whole network presents the computation of the learn-
phishing attacks and malicious code. ing process [113]. Typically, a neural network consists of a
single input layer, a variable number of hidden layers and a
A. Machine Learning single output layer. An input layer is comprised of a num-
ML is a subset of AI that builds a mathematical model ber of neurons, equal to the number of input features [114].
based on training data to make decisions or predictions without Depending on the classification problem, the output layer may
consist of multiple neurons, in case of multi-label or multi- followed by the XAI taxonomy and popular XAI models. The
class classification problems (output is a vector indicating a details of post-hoc methods are also covered. This discussion
class or labels), or a single neuron in the case of binary classi- illustrates how XAI methods for automating cyber defences
fication (output being either ‘0’ or ‘1’). What distinguishes DL can be developed. Cyber defences concentrate on discovering,
models from their shallow counterparts, is the former incor- identifying and then mitigating cyberattacks to assure that the
porate multiple hidden layers between their input and output intended mission of an attacker is not achieved and to evade
layers. In [115], the authors classify deep learning models into malicious events. This includes the design of various cyber
two categories based on the architectures as follows: security techniques, including intrusion detection, threat hunt-
• Generative – Generative models depict independence/ ing, threat intelligence and vulnerability analysis. Thus, any
dependence for distribution by computing joint prob- of these techniques can be augmented using XAI-based cyber
ability distributions from data with their labels [20]. defence methods.
Models, which can utilize generative architecture, are
Recurrent Neural Network (RNN), Deep Auto Encoder
(DAE), Deep Boltzmann Machine (DBM) and Deep A. The Need for XAI and Its Benefits
Belief Network (DBN). ML models are often considered to be black-boxes, with
• Discriminative – Discriminative models directly esti- their inner mechanics not easily interpreted and their out-
mate the posterior distributions of classes conditioned on put being difficult to be justified. These limitations hinder
the input data [115]. Thus, the discriminative approach the development of responsible AI and prevent its effec-
is more efficient since it only focuses computational tive application in other critical fields, where accountability
resources on a given task, which is classification, without of such automated systems would be crucial. Responsible
modelling underlying probability distributions. There are AI establishes accountability for AI systems by defining
two types of discriminative architecture, including RNN interpretability features for decisions that are based on AI
and Convolution Neural Networks (CNN). predictions. Responsible AI is a model framework of gov-
• Hybrid – The hybrid models combine different DL ernance that describes how a particular AI model addresses
models, for example, the Convolutional-LSTM model. the tasks around AI from both the legal and ethical points of
Although the performance of the hybrid model for a spe- view [117]. By design, ML-based black-box models compro-
cific problem is much better than the other two categories, mise ART principles and run the risk of leading to unethical
they are hard to formulate. use, lack of responsibility, accountability, and potential biases
Generally, DL algorithms demonstrate data as a nested hier- in critical cases of decision-making.
archy of concepts within their multi-layer neural network Moreover, research in XAI is becoming vital, due to laws
architecture. Therefore, deep learning techniques can learn recently applied by the governments [118]. Arrieta et al. [52]
the computational process in depth [20] and achieve good defined two causes that make the incapability of explaining
performance and flexibility that outperforms the traditional decisions such a barrier that AI is facing. First, relying on
ML in data with large scale [116]. Thus, deep learning black-box models to make predictions is impossible in some
approaches are the most suitable for building an efficient IDS. critical sectors due to the enormous gap between the research
community and the business operators. Such sectors have strict
C. Summary and Insights regulations, and the daily decisions are of such importance,
that unnecessary risks, such as relying on an untrustworthy
By reviewing the existing body of literature, it has become
model that produces vague predictions are not an option.
apparent that the cybersecurity domain is becoming increas-
Second, the desire to acquire knowledge and improve under-
ingly reliant on automated methods for incident detection and
standing promotes XAI development. The reason is that every
response. The main motivation behind this trend is the ever-
field benefits from AI; the results are important, and having
evolving efficiency and sophistication of cyberattacks, which
the capacity to interpret and explain them is paramount.
renders a manual expert-based detection approach unfeasible.
In a study by Adadi and Berrada [54], the benefits of
Relying on numerous sources, with millions of exchanged
utilising XAI were separated into four categories: explain
network packets, generated logs and files to assess, AI-
to justify, control, improve and discover. To further inves-
powered attack detection approaches have been the focus of
tigate the benefits of XAI, a concise user-centric model is
the research community, especially solutions that leverage DL
provided in Figure 3, where benefits are prioritised accord-
models, that are best suited to swiftly processing vast volumes
ing to five types of users. To begin with, in environments
of data [116]. This section presented the most prominent sub-
where AI is ever-present and a crucial aspect of systems,
types of ML and DL models, based on their training strategy
such as unmanned vehicles, explanations would benefit regu-
(for ML models) and how they model the relationship between
lar users that are affected by a model’s decision. Explanations
dependant and independent features, corresponding to the class
in such environments would allow users to better understand
and the data features respectively.
the process that led to a decision, enabling them to accept
or question the output of a model and the behaviour of an
VII. XAI IN C YBER D EFENCES AI-based system. Understanding the decision-making process
In this section, XAI concepts and applications are presented. through XAI and being able to discern how alterations in the
First, a discussion of XAI and its benefits is presented. This is data, for example, telemetry measurements from sensors in
Fig. 3. XAI models for assisting various audience profiles.
intelligent environments and automated vehicles, affect the • Human Decision-making – In critical sectors where deci-
output, domain experts and users are able to gain intuition sions are so important and sensitive, humans must make
about the inner workings of black-box models and gain trust them. XAI can serve as a tool to support humans by
in their operations. providing its predictions with reasonable explanations.
From a regulatory perspective, assessing compliance for AI- • Trust Building – This is arguably the ultimate aim of
powered applications and systems is challenging, as it relies on XAI [120], [121]. By getting an insight into the black-
exhaustive studies and assessments that require realistic data box model, people can verify basic facts or identify errors
that is representative of the intended deployment environment. to avoid. Therefore, XAI builds trust between ML models
Managers and regulatory agencies can take benefit from XAI and humans.
and the explanations that it provides, shifting somewhat the In the cyber defence domain, and specifically in IDS
assessment focus from a data-driven approach to one based applications, the incorporation of XAI would greatly benefit
on cause-and-effect inference. The compliance of a model to both non-technical decision makers, as-well-as cybersecurity
regulations and audits can be evaluated according to finite experts, as not only would such systems enable the identifi-
predictions and the resulting explanations. From the perspec- cation of malicious behaviour (basic operation of IDS), but
tive of developers, XAI can provide benefits towards improv- it would also provide explanations of the reasons for, and a
ing a model’s efficiency and performance. Understanding the justification why an event was flagged as malicious, which is
effect that data features have on a model’s output can help crucial for the effective integration of cyber defence solutions
guide data scientists to take actions that would improve the into the cybersecurity strategies of organisations. Assessing
model’s capabilities, by removing redundant features, con- features and their contribution to the output of a decision
structing new ones through feature engineering, flagging data engine can help detect models that may exhibit erroneous
records that hinder the model and replacing them. In other behaviour, due to mislabelled data or biases. For non-technical
words, the appearance of XAI methods can enhance the use end-users such as corporate decision makers, the value of
of ML techniques for application in different industries by XAI in automated solutions in domains such as cyber security
introducing the following benefits: solutions, is to build trust on the output and performance of
• Model Debugging – During the training process, ML such automated solutions, gain a high-level understanding of
models would take biases from the training dataset, which attack behaviour and facilitate successful discussions between
are not easily detectable or preventable. Thus, they tend to them and more technical security experts. From more technical
provide discriminated decisions against underrepresented users, an insight into the decision helps administrators iden-
groups [119]. XAI can present the patterns constructed tify the segment of the network, features, and security policies
by the model, which helps data scientists and develop- compromised by attackers [122]. With the information pro-
ers analyse and erase irrelevant patterns to renovate the vided by XAI, the IDS operator can select the correct actions,
dataset. whether to debug the IDS model or apply new security poli-
• Data Collection – XAI would give an insight into ML cies to prevent the same attacks in the future. Considering the
models and a good understanding of the value of the benefits of XAI discussed above, the need for XAI in IDS is
feature in the training set. Thus, the data scientist can of utmost necessity.
evaluate the importance of features and adjust the process Existing AI models, assuming we are referring to non-
of data collection. interpretable ML, do not explicitly provide insights into the
process that resulted in a particular output. For example, to explain how a host was compromised. Thus, the benefit of
after training a GRU-RNN model, a type of neural network XAI in cybersecurity settings is found to be a many-fold one,
designed for processing temporal sequences of data, if two with benefits including the identification of key input features
network flows are flagged as DoS attacks, one being a slow that identify certain types of attacks, the elimination of biases
loris flood, while the other being a DNS amplification attack, from the model, introduced through errors during data collec-
the model does not include an inherent method for justifying tion and pre-processing and drawing insights about how an
what prompted it to identify them as DoS attacks, rather than attack was achieved, to name a few.
a slow-connection user and a DNS resolution query respec- To sum up, XAI is necessary for cyber defences, such as
tively. Additionally, the model can not justify the distinction IDSs and threat intelligence, because it allows for the identifi-
between the two types of attacks, according to input feature cation of potential security breaches and vulnerabilities. XAI
values. Furthermore, XAI could be used as a tool for detecting algorithms are designed to analyse data patterns and make
false alarms, triggered by a DL model. Let us assume that a predictions, but without explainability, it can be hard to under-
neural network model detects two flows as being part of an stand how the algorithm came to its conclusions. This lack of
SYN flood. For one flow, the XAI method identifies that flags transparency can make it difficult to detect errors, biases, and
are one of the influential features that resulted in a positive other issues that would impact the effectiveness of the pro-
(attack-indicative) output for the model. For the other flow, the gram. In the case of cyber defences, an XAI technique helps
XAI method identifies the number of bytes but not the flags analyse network traffic, and telemetry data of IoT devices,
as an influential feature. Without utilising XAI, both flows identify potential threats, and provide clear explanations of its
would have been considered equally as likely to be attacked. findings, allowing defenders to take action effectively. XAI
By using XAI, insights can be drawn about the output of the is different from traditional AI in that it is designed to pro-
model, and a user may place more weight on the first flow, vide transparency and understanding of how a decision or
where the flag feature indicated a series of SYN packets in the prediction is made. In cyber defences, XAI can be used to
flow, rather than the second, where a large volume of bytes explain how a decision is made in detecting and responding
was exchanged. to potential cyber threats. Traditional AI-based cyber defence
Rule-based expert systems, although inherently inter- techniques would provide accurate results, but the lack of
pretable, face challenges when applied to contemporary transparency and understanding of how decisions are made
networks and systems, due to the volume and speed of gener- creates a challenge for cyber defence teams in interpreting
ated data, the complexity of attacks, the complex dependence and responding to potential cyber threats. XAI-based tech-
of input features and attack polymorphism. These challenges, niques can assist in tackling these challenges by providing
along with the fact that expert systems require considerable clear explanations for the basis of the decision, which can
manual intervention as it is highly-reliant on expert knowl- be crucial in identifying and responding to cyber threats in a
edge, render them inefficient for contemporary networks and timely and effective manner.
systems [123]. A subset of the existing ML model landscape
consists of inherently interpretable models, with some promi-
nent examples including regressors and decision trees. By B. Taxonomy of XAI
design, such models provide inherent methods for interpret- The ML and DL model landscape is a diverse one, with
ing their results, including the contribution of features to a models employing distinct mechanisms to perform a number
particular output and their importance. However, research in of activities, including classification, regression, clustering and
the cybersecurity domain has exhibited a trend towards util- data generation. Similarly, development in XAI methods has
ising complex models, including random forests and DL, due given rise to numerous types of explanations, which can be
to their improved performance when compared to simpler classified according to the techniques used to provide expla-
models [124]. nations, the stage during training or deployment of the ML/DL
As further justified by surveying relevant literature, model where explanations are obtained or the reliance of the
Capuano et al. [125] identified an increasing trend in the vol- explanations to particular models. In this subsection, a tax-
ume of publications focusing on IDS, with the most recent onomy of XAI methods is provided, elements of which are
ones leveraging ML and DL models for detecting intrusions. depicted in Figure 4.
By reviewing the relevant literature, the authors identified that • Intrinsic vs. post-hoc methods – Intrinsic XAI refer to
the efficacy of DL models to accurately detect cyberattacks in a ML models that are transparent and by design have
multitude of real-world settings is affected by weaknesses such reduced complexity, thus enabling users to understand
as the inability of domain experts to accurately understand the what caused a particular output, considering the input.
process that drove a model to a particular result, concluded However, other models that are characterised by com-
by the black-box design of most DL models. Similar observa- plex architectures or opaque internal mechanics such as
tions were reported for Malware analysis, with recent research SVM and Deep Neural Networks (DNN) require sec-
efforts leveraging post-hoc methods to explain the output of ondary, external methods to be employed, in order to
ML/DL models for the detection of malware binaries and sus- justify their outcomes. To explain the results of such
picious system calls. Similarly, the need for XAI in the field models, after they have been successfully trained, post-
of Botnet detection was also emphasised through the findings hoc methods are employed, where usually a simpler
of the team’s investigation, with post-hoc methods leveraged and easier-to-understand (surrogate) model is employed.
Fig. 4. Taxonomy of XAI methods for cyber defences.
These simpler, surrogate models are trained to model the of diversly structured models. Model-specific methods
behaviour of the more complex model, with results being have the advantage of providing an in-depth under-
reproducible. What differentiates intrinsic from post-hoc standing of the prediction process, however, they limit
XAI methods, is that the former can be applied to mod- the available choices for selecting decision models, as
els whose structure is simple and thus supports intrinsic these types of explanations are highly dependent on the
explainable methods, whereas post-hoc methods are more underline decision model. Additionally, model-specific
versatile and are not model-locked, thus they can be methods face performance issues as they often traverse
applied to a wide range of models [126]. These methods the entire structure of the model to provide accurate jus-
construct self-explanatory systems with interpretability tifications of model outputs [54]. Model-specific methods
directly at their potential structures. Although post-hoc are considered to be intrinsic, while post-hoc methods are
methods are more flexible than intrinsic XAI, growing predominantly model-agnostic.
support for intrinsic XAI has been observed, as post-hoc • Local explanation vs. global explanation – Local expla-
methods introduce overhead and added complexity to the nations are built around individual input records, and
explanation process. seek to provide explanations by modifying features and
• Model-specific vs. model-agnostic – With regards to assessing how the output is affected, often describing
the dependence of XAI models on the models they the impact of each data feature on the output class fea-
explain, there are two categories: model-specific and ture. Global explanations on the other hand assess the
model-agnostic. Model-specific methods are built for one features and their overall importance to the classifica-
specific model or a group of similarly structured mod- tion process, thus indicating how each feature contributes
els, relying on the characteristics of their architecture to to the decision process, leveraging most if not all the
provide explanations. Model-agnostic XAI methods on available data points. As such, local explanations can be
the other hand are designed to use mathematical and leveraged to explain a particular model decision, while
statistical approaches that do not rely on the internal global explanations are utilised to assess the model in its
structure of a model, thus are applicable to a wider range entirety [126].
C. Explainability Models explanations in different formats. Therefore, the best tech-

There are ML models that can be intrinsically interpreted niques can be chosen to explain the black-box models
due to their simple structure. This allows researchers and depending on the type of end-users and audience. For
developers to impose constraints directly on the ML model to example, a visualization explanation would be adequate
explain its decision based on internal functioning. A notable for the lay audience, while feature importance may be
example of this approach is proposed by Letham et al. [127]. more suitable for data scientists.
The authors introduce a generative model named Bayesian • Low or no cost to switch – Switching the underlying
Rule Lists based on a decision tree to produce interpretable model for a new one is convenient without any mod-
medical scoring systems. Experimental results show that the ification to the presentation of the explanations. This
proposed model is concise and convincing and authors suggest is a significant advantage over intrinsically interpretable
applying similar models in other fields. models. Post-hoc methods allow developers to choose
Xu et al. [128] proposed an attention-based approach for the best-performing ML model as the underlying model,
a model that can learn to describe images. In the experi- enhancing accuracy.
ments using three benchmark datasets, the method has good Table IV lists and Figure 4 depicts all the model-agnostic
performance and can effectively explain the results to users methods that will be discussed. The advantages and disadvan-
through visualization [128]. The authors also suggested visual tages of each method and their applications are summarised
attention for future work. In another work by Ustun and in the table which presents useful knowledge about state-of-
Rudin [129] a Supersparse Linear Integer Model (SLIM) for the-art techniques. Visual-based explanations are the easiest to
creating data-driven medical scoring systems was introduced. understand, and explain a model’s behaviour regarding how
Due to the high level of sparsity and small integer coefficients, a provided input affects the produced output. Visualisations
the model can interpret the results with qualitative understand- are the most natural way to demonstrate complex interac-
ing [54]. Other interpretable ML models include linear/logistic tions within input features or the effect of each feature on
regression, general additive models, general linear models, the model’s prediction to users who might not have expert
decision trees, k-nearest neighbours, and rule-based learners knowledge about AI techniques, such as domain experts and
listed in [52], [119]. managers. XAI model-agnostic visualisation methods are com-
Although this approach provides intuitive explanations that plex to implement; therefore, it is usually coupled with feature
are easy to understand, it can only be applied to models that relevance explanations techniques to improve the understand-
do not perform very well with a large scale of data. Therefore, ing and provide comprehensive information to the recipient of
the trade-off between explainability and performance hindered the XAI model’s results [52].
this approach’s adoption [130]. As discussed previously, deep In the scope of this paper, some notable techniques of
learning can achieve good performance and flexibility that this type will be discussed, including the Partial Dependence
outperforms traditional ML in data with large scale [116]. Plot (PDP), the Accumulated Local Effect (ALE) plot and
However, DL’s network consisting of multiple layers is con- the Individual Conditional Expectation (ICE) curves. Partial
sidered a black box due to the difficulties in explaining Dependence Plot (PDP) visualises the average marginal effect
its functioning. Therefore, the post-hoc approach is used to on the global level of a subset of features on the model’s
interpret the complex black-box models. This approach can be predictions with all other features fixed [153]. Each row of
considered reverse engineering that explains predictions with- data is considered and passed through the trained ML model
out any modification or knowledge about the model’s internal to produce outputs. Then, the value of the features of interest
functioning [54]. is altered repeatedly to make a series of predictions. After iter-
ating through all the data instances, the visualisation method
can represent the relationship between the output and the fea-
D. Post-Hoc Methods tures. This technique has two major drawbacks: assumption
When it comes to ML models that cannot be interpreted of independence and hidden heterogeneous effect [119]. The
intrinsically due to their sophisticated structures, post-hoc assumption of independence implies that features are assumed
techniques can be employed to provide explanations in a post- to be independent, which may be an erroneous premise. The
prediction mode. Most post-hoc methods are model-agnostic hidden heterogeneous effect may occur since the PDP can only
and thus are more versatile as they can be applied to any visualise the average marginal effect. Each data point can have
ML model. Such an approach has been observed in numerous a positive or negative association with the model’s prediction;
works of research recently due to its desirable advantages as thus, only presenting the average effects of all points would
followings [152]: hide such relationships.
• Model flexibility – The interpretation techniques are not A solution to prevent the assumption of independence is the
tied to a specific type of ML model. It is up to developers Accumulated Local Effect (ALE) plot proposed by Apley et al.
to choose the most suitable post-hoc methods without The ALE plot was designed to perform well with highly cor-
changing the black-box model or compromising its high related input features; therefore, there is no need to assume
performance. that the features are independent. Another advantage of the
• Explanation flexibility – Explanations can take many ALE plot over PDP is that it requires less computational
forms, such as visual, graphical or numerical. The post- resources [134]. Goldstein et al. [135] introduced a method
hoc approach introduces various techniques to generate called Individual Conditional Expectation curves to uncover
TABLE IV
C OMPARISON OF M ODEL -AGNOSTIC M ETHODS
heterogeneous relationships. ICE plots show one line per data assumes the independence of features. Consequently, invalid
instance, representing the relationship between a feature of data points may be generated and learned by local explanation
interest and the fitted model’s decision. Therefore, comparing models.
ICE curves with PDP would produce interesting insights into In [155], the authors showed the instability of the expla-
ML models as ICE curves of each instance might be signifi- nations produced by LIME. Moreover, Slack et al. [156]
cantly different from the average of all instances (PDP) [154]. described how LIME explanations could be manipulated to
ICE plots also have some limitations. Firstly, the ICE plot is hide biases in the dataset. Although LIME is promising, it
only useful for one feature since drawing two features will is still under development and needs to be improved before
create overlaying surfaces [119]. Secondly, ICE curves suffer it can be effectively applied in real-world scenarios. There
from the assumption of independence, just like PDP. Lastly, have been several pieces of research that investigated ways of
the plot might get overcrowded, hence becoming impossible overcoming the limitations of LIME and analysing its prop-
to analyse. erties [142]. In [143], the authors of LIME introduced an
Surrogate models are interpretable models that mimic the extension method of LIME using a high-precision rules class
behaviour of the black-box models. This post-hoc method is called Anchors. Anchors’ explanation is intuitive and easy to
especially flexible due to the free choice of surrogate mod- understand; however, they require a configurable setup like
els. However, the generated explanations would not fit all most other perturbation-based explainers.
records in the dataset, and there is no valid theory that sup- Relevance-based methods explain the model by ranking the
ports the surrogate model’s representation of the complex most relevant input features that impact the model’s prediction.
model [54]. Moreover, this method suffers from the drawbacks The Permutation Feature Importance (PFI) was first intro-
of the interpretable model chosen as the surrogate model. duced in 2001 by Breiman [144] and measures the increase in
In [120], the author proposed Local Interpretable Model- prediction error of a fitted model after permuting the feature’s
agnostic Explanations (LIME), which derives from the concept values. This technique shows the importance of each feature by
of local surrogate models. LIME’s key concept is fitting an breaking the relationship between the feature and the desired
interpretable model around a specific instance to visualize sig- output. Therefore, the model’s error would increase if the
nificant features of that data instance. The interpretable model feature of interest is considered important by the model. In
can be of any type if it is a good local approximation of the 2018, Fisher et al. [145] proposed Model Class Reliance which
ML model predictions. LIME is flexible as it can perform is a model-agnostic version of the PFI. PFI provides global
well with tabular data, text and images. However, the correct insight into the model’s behaviour and automatically consid-
definition of the neighbourhood is questionable when using ers all interactions with other features. However, PFI requires
LIME with tabular data because improper kernel settings can shuffling the features, which adds randomness to the compu-
lead to non-sense explanations [119]. In LIME, data points tation and makes results vary greatly after repeated training.
are sampled from a Gaussian distribution, which means LIME Moreover, PFI suffers from the assumption of independence.
Lundberg and Lee [146] proposed the Shapley Additive explanations for the model and demonstrate the relation-
exPlanations (SHAP) method, which is based on the game’s ship between values of input features and the model’s
theoretically optimal Shapley Values as introduced in [157]. decisions.
SHAP connects LIME and Shapley values and has several • Visual explanation – This technique utilises the input
advantages over LIME. The behaviour of the ML model is features and model’s prediction to visualise the model’s
assumed to be linear locally in LIME, yet the effects are dis- output, usually by plotting a graph. A visualisation-based
tributed fairly in SHAP. Moreover, SHAP allows contrastive method effectively explains a black-box model to end-
explanations by comparing a prediction to a subset or a sin- users with varying levels of AI expertise. Visualisation
gle data instance. However, SHAP has to use all the features, as a model-agnostic method is a complex task; therefore,
hence being computationally expensive. Assumption of inde- it is usually coupled with feature relevance explanations
pendence is also a big problem with SHAP, like many other techniques to improve the understanding and provide
permutation-based interpretation methods. Another method comprehensive information to the recipient of the XAI
named saliency maps (or pixel attribution map) [147] is a model’s results [52].
type of both relevance-based methods and visualization expla- A number of XAI-based tools have been developed, accord-
nations as a pixel of an input image can also be considered ing to the concepts previously discussed, such as DeepVis
as a feature. These approaches generate intuitive explanations Toolbox, TreeInterpreter, Keras-vis, Microsoft InterpretML,
by highlighting the most relevant pixels on the final classi- MindsDB, SHAP, Tensorboard WhatIf, Tensorflow’s Lucid,
fication. Importance scores of individual pixels are computed Tensorflow’s Cleverhans and many others. Most of these tools
using occlusion techniques, or calculations with gradients [54]. are model-agnostic methods, and a few are model-specific.
However, the saliency map would be fragile [158], and very For instance, DeepVis, kerasvis, and Lucid are for a neural
highly unreliable [159]. network’s explainability, and TreeInterpreter is for a tree-
Unlike other model-agnostic methods, example-based expla- based model’s explainability [118]. Each proposed approach
nation approaches generate explanations from particular has similar concepts at a high level, such as relevance-based,
instances of the dataset instead of creating summaries of the Shapely values, partial dependence plots, surrogate models,
features [54]. Example-based explanation methods have two counterfactual, prototype, and criticism.
main techniques: counterfactual explanations and prototype
and criticisms. Wachter et al. [148] introduce counterfactual E. Summary and Insights
explanations as a novel model-agnostic XAI method. The tech-
This section discusses the value that XAI methods intro-
nique explains the ML model by describing the change needed
duce to automated ML and DL solutions. At first, a model
to change or flip the prediction. Explanations generated by
is presented that illustrated the benefits of XAI to end-users,
this technique are straightforward for humans to understand as
according to their technical level and interaction with the
often users question why a particular prediction was produced,
model (end-user, regulator, security expert). Benefits resulting
instead of another [160]. However, each instance can have
from the incorporation of XAI include the capacity to debug
multiple counterfactual explanations, and they may contradict
AI models, performance-driven data collection, informed deci-
each other [119].
sion making and trust building. Next, a taxonomy of XAI
Prototypes are representative data instances of the
methods, according to their methodologies, including intrinsic,
dataset [149], [150] and criticisms are data points that are
post-hoc methods, model-specific and agnostic, and local-
not well represented by the prototypes [151]. Together with
based or global-based explanations. This subsection primarily
criticism, prototypes can provide meaningful insights into the
focused on post-hoc and mostly model-agnostic methods, as
ML model. Kim et al. [151] develop MMD-critic that selects
they are more flexible and can be effortlessly applied to a
prototypes and criticism for a dataset to aid human under-
wider range of ML/DL models. Some post-hoc approaches
standing and reasoning. In addition, this technique requires
that were discussed include PDP, ALE, ICE. The concept of
a meaningful data-processing module to select only relevant
surrogate models was also discussed, focusing on LIME and
features because prototypes and criticisms are generated by
SHAP local and global explanations respectively. Furthermore,
considering the entire feature set, which may be misleading
two major XAI post-hoc technique categories were presented:
due to irrelevant features [119]. In the scope of this paper, the
feature importance and visual explanation methods. Feature
two following classes of XAI techniques are discussed: feature
importance provides a numerical value indicating the contri-
relevance and visual explanations
bution of each feature to a decision, whilst visual explanations
• Feature relevance – The technique is a relevance-based
produce a graphical representation of the relationship between
method that explains the model’s decisions by calcu-
the input features and a model’s output. Although visual expla-
lating an importance score for each feature. A com-
nations may be more impactful to end-users, their construction
parison among different features’ scores would reveal
is a complex process, whereas feature importance is easier to
the importance of each feature, which is granted
produce but maybe, comparatively, more difficult to interpret.
by the model when making decisions [52]. A pop-
ular XAI post-hoc method using feature relevance
techniques mentioned above - SHAP proposed by VIII. AI IN I OT AND C YBER S ECURITY P ERSPECTIVES
Lundberg and Lee in [146]. The proposed method This section gives insights into utilising AI in IoT for cyber
combines LIME with Shapley value to generate local security application purposes. It also compares conventional
Fig. 5. Proposed architecture of Internet of Things (IoT) for cyber defences.
and contemporary AI with the explanation capability. It then data in transit. It purifies the data and only extracts valuable
describes the common cyber threats in IoT ecosystems. information. Cloud computing and big data processing mod-
ules are examples of technologies used in this layer [162].
A. Proposed IoT Architecture for Cyber Defences Forth, the Application layer refers to IoT applications that
The domain of IoT typically consists of a wide range provide services to users, such as smart homes, smart health
of advanced technologies; thus, there is no single reference and many others. Fifth, the business layer is responsible for
architecture that can represent all possible implementations controlling IoT applications and business models [163].
of a secure IoT ecosystem. However, for research purposes, Although the IoT structure looks simple, there are essen-
the five-layer IoT architecture is utilised to focus on the tially a group of fragments that move and work together for
finest aspects of IoT, which can fulfil security and privacy the proper functioning of the IoT as planned. From IoT’s
requirements. This architecture can also assist in developing viewpoint, the architecture of IoT is defined as a framework
and evaluating new applications of cyber defences, such as that describes the functioning events, the physical appara-
intrusion detection and threat hunting [161]. tuses, network configuration, the functional grouping, and
Figure 5 presents a proposed IoT architecture linked with the formats of data to be utilised. Nevertheless, there’s no
cyber defences. To effectively shield and provide robustness sole typical reference framework for the IoT as it involves
for an IoT deployment and its five layers against multistage various components. We can comprehend from this notion
and zero-day attacks, cyber defence methods such as intru- that there’s not a single easy design that can be consulted
sion detection and threat intelligence need to be adjusted and for all the potential applications. Therefore, the architecture
assimilated. To give insights into the functions of these IoT of IoT can essentially differ considerably, depending on the
layers, first, the perception layer is the sensor layer that col- application; it requires to be sufficiently exposed to open
lects information through the sensors attached to physical procedures so that it can provision manifold applications of
objects such as cars, robots, surveillance cameras, phones and the network. The ecosystem of a sustainable IoT has dif-
many others. Second, the network layer connects devices and ferent significant design blocks: maintainability, scalability,
servers and transmits collected data from the perception layer obtainability, and functionality. Scalability is considered vital
using Wifi, Bluetooth, Near-Field-Communication (NFC) and because the scheme requires to be capable of enhancing
other communication protocols and technologies. Third, the together with the application requirements set forth by an
middleware layer is responsible for processing and analysing organisation.
Fig. 6. Deployment of XAI-based anomaly detection for Cyber defences in IoT systems.
Maintainability refers to the ability of an IoT system to processing and then sends back signals to IoT devices for fur-
continue functioning as originally intended under both normal ther actions. This approach is also referred to as the Cloud
operations and less-than-ideal circumstances. It is a dimension of Things (CoT) paradigm, and it is flexible and robust in
of a smart system’s resilience and involves both automated managing IoT devices. However, this centralised architec-
internal methods found in IoT deployments that render them ture commonly suffers from high communication latency and
resilient to intentional (malicious) or unintentional (bugs) errors power consumption; thus, large-scale deployment of CoT is
and device failures, as well as routine maintenance procedures almost impossible in real life [165]. The conventional approach
such as firmware updates and software patches. Obtainability was replaced with the emergence of fog computing (also
refers to the capability of a smart system to deliver IoT-powered known as edge computing). Fog computing is an extension
services and data obtained from an environment, to the end- of cloud computing that distributes the benefits of the cloud
users and owners of the system, regardless of the relative closer to the IoT and across five layers of the IoT [1]. Fog
location of the end-user to the smart system in question. Users nodes can be any devices with computing, storage and network
are thus able to access the data and services of their IoT connectivity, such as industrial controllers, switches, routers
deployment, effortlessly and from anywhere, through the use and surveillance cameras. Generally, the fog was developed to
of Web and smartphone interfaces. To guarantee that obtain- address two significant problems in IoT networks, including
ability is achieved, it should be ensured that traversing between high network latency and sensitive data leakage [166]. IoT
networks and environments does not affect a user’s access to applications are written for fog nodes at the network edge,
their IoT-derived data. Functionality corresponds to the practical allowing fog nodes to ingest IoT data from multiple devices.
augmentations that IoT sensors and actuators would introduce to IoT-derived data is forwarded to a valid destination in the
a mundane environment. IoT functionality defines a breadth of IoT architecture, such as fog nodes, aggregator devices or the
services, starting from simple reporting of environmental con- cloud back-end, depending on how swiftly it needs to be pro-
ditions by sensors to the combination of such telemetry streams, cessed, and further analysed [167]. Therefore, fog nodes offer
to the creation of complex connections between sensors and an advantage for IoT networks by allowing distributed secu-
actuators that enable automated sensory-based adjustments to rity services due to their ability to offload computational tasks
be made to the smart environment by the environment itself. from IoT devices [168].
IoT application tasks are usually delegated to cloud com- In recent years, considerable work [99], [169] has focused
puting due to the limited computational resources on IoT on utilising fog or edge computing to design distributed IDSs
devices [164]. The cloud server receives IoT data for in IoT systems. Figure 6 illustrates a proposed architecture for
TABLE V
anomaly-based IDS in IoT networks. In the proposed envi- C OMPARISON OF XAI VS T RADITIONAL AI
ronment, IoT devices are placed in the fog layer along with
fog nodes that are functional gateways for the IoT system
and are responsible for collecting, aggregating, pre-processing
and forwarding telemetry and network data that is generated
from the IoT deployment to the cloud backend. While data
is sent to the cloud backend for persistent storage, where
users are able to access, review and act on the collected
information, the intrusion detection process is carried out in
the fog layer. Before any intrusions can be detected, it is
for AI were Fuzzy Logic and Expert Systems having Lisp
required to pre-process the aggregated IoT data to make it suit-
and Prolog being the topmost selection as implementation
able for ML/DL models to process it. Pre-processing activities
languages amongst C/C++, where there was no concept of
include normalisation, one-hot encoding, discretisation, impu-
accountability and explainability in AI models.
tation, denoising and feature extraction. Next, an AI-based
The introduction of modern AI is considered to have trig-
decision engine processes the data and classifies it as either
gered the new boom in the AI world when the “Data science”
benign, involving normal operations of the IoT deployment, or
term was devised in early 2008 by the two leading groups
malicious, which indicates the presence of an ongoing cyber-
from Facebook and Linkedin that relied on different statisti-
attack. The proposed XAI-enhanced AI-based IDS assumes
cal, algebraic and calculus methods. Regardless of this rapid
an active approach to cybersecurity, as normal operations are
development, a major drawback of AI methods that is valid
allowed to continue uninterrupted, while abnormal traffic is
to this day, is a lack of understanding of how decisions are
initially terminated and assessed through the XAI method for
made by AI models internally, lending the term “black box”
further processing. The output of the XAI method is then
to such models. One of the breakthroughs in modern AI is
forwarded to both end-users, to inform them of the detected
moving away from this black box nature and acquiring a
security incident and the reasons why it was flagged and ter-
reasonable interpretation of the decision-making procedures.
minated, as well as to security specialists with access to the
This innovative idea is now termed XAI. Once XAI is accom-
cloud layer, to further analyse the incidents, according to the
plished comprehensively, the community of AI researchers and
XAI explanations.
scholars community will have access to a new trend in AI,
The model generates different explanations directed to
and more resilient and robust AI platforms could be possible.
inform audiences or stored in cloud servers for further
The comparison of XAI and traditional AI is summarised in
research. IDS would be trained in a cloud environment and
Table V.
then executed on a fog node; thus, it prevents the latency
problem apparent in centralised-based solutions. Moreover, fog
nodes can analyse and explain each network activity gener- C. Cyber Threats in IoT Ecosystem
ated by IoT devices as they are directly connected to sensor IoT has evolved to encompass multiple domains and aspects
devices [1]. An AI-based IDS that is deployed at a fog node of society, including industry, healthcare, homes, sport, peer-
would be handy for processing and obtaining confidential to-peer networks [172], entertainment, and others [173]. It is
information from big IoT data [170]. Regarding the advan- gradually replacing many conventional computer systems in
tages of this approach, we recommend utilising fog computing those fields. However, in comparison with traditional com-
to design a distributed explainable AI-based IDS, as it enables puting systems, this engagement has introduced new security
fast and decentralised data processing and safeguards the sen- challenges [3] for several reasons:
sitive data of end-users. Additionally, different explanations • Complex and diverse environments – An IoT is diverse
generated from the IDS will be directed to the optimal place as it is connected with various devices, platforms, com-
for analysis and to inform multiple audiences. munication means and protocols. The diversity enhances
the usability and convenience of IoT technologies yet at
the cost of numerous potential targets and attack vectors.
B. Traditional AI vs Modern AI • Undefined boundary – The IoT system does not have a
In contemporary technology sectors and industry domains, well-defined boundary since it changes very often due to
AI has become a very popular buzzword. Most leading the mobility of users and devices. Due to the undefined
research companies, such as PWC, Gartner, and McKinsey, boundary, it is not easy to design effective security mech-
have overvalued the growth of AI with astonishing numbers anisms for the IoT system. Moreover, the IoT systems
and forthcoming estimates. In a 2018 report by PWC, [171] would suffer from an extensive attack surface.
predictions indicated that AI would return $15.7 trillion • Connection between virtual and physical systems – IoT
towards the worldwide economy sector by 2030, and the devices would be a potential target since the attacker
general GPD and efficiency will rise by 14% and 55%, respec- can access them physically in unattended working envi-
tively. The progress in AI research and development has been ronments. Moreover, IoT devices can function on the
quite tumultuous throughout the years. For instance, the British received data, which optimises the connection between
Government seized AI’s funding of several universities after virtual and physical systems yet allows the attacker to
an investigation in 1973. Back then, the well-known methods convert the potential physical consequences quickly.
TABLE VI
M ACHINE L EARNING -BASED IDS
• Limited energy and computational resources on devices – factors that caused Mirai-originating attacks to be so destruc-
Most IoT devices have limited energy and computational tive [3]. This example has demonstrated the value of secure
resources, making it hard to implement decentralised IDS authentication mechanisms and traffic classification techniques
and advanced security techniques on physical objects. towards securing digital assets around the Internet. Thus, new
For example, IDS that utilises deep neural networks defence mechanisms should be developed to prevent cyber
require GPU to perform well in real-time, which most threats in IoT systems, with IDS being considered the primary
IoT devices cannot afford. method to attain these requirements.
D. Summary and Insights IX. C ONTEMPORARY DL-BASED IDS

According to the challenges that were introduced in this Since IoT deployments and their component smart devices
subsection, IoT is a compelling target that garners the atten- have been found to be susceptible to cyberattacks, as a result
tion of cyber attackers. Many attack techniques are utilised of vulnerabilities inherently found in their firmware and pro-
to exploit the security issues at different layers of the IoT tocol implementations, AI has been identified as an enabling
environment, including sensing, network, middleware, appli- factor, towards securing them from a diverse range of cyber
cation and business layer. According to [187], 41% of attacks threats. This section briefly discusses challenges that affect the
exploit vulnerabilities found in IoT devices, bringing them performance of DL-based models, relating to a state known
under the control of an attacker that can then leverage them as overfitting. Furthermore, recent DL-based IDS, both for
to launch large-scale attacks, such as DDoS. For example, a IoT and non-IoT environments are presented in this section,
Mirai-based attack compromised a French cloud computing in Tables VI and VII, along with the base DL model that
company in 2016 and became the most significant distributed the researchers employed and the dataset that was leveraged
denial of service attack (DDoS) recorded up to that date [4]. during training and testing.
Attackers were reported to instrument zombified IoT devices Research that aims to provide holistic cyber defence mecha-
and use them as a pivot point to launch a DDoS attack on nisms in the context of IDS, threat intelligence, threat hunting,
the French Web host. According to researchers that investi- privacy preservation, and digital forensics, has employed ML
gated the Mirai malware, default and weak security credentials, algorithms as the underline decision-making component, to
along with incorrect device configurations were the enabling great effect. With a particular focus on developing dynamic
TABLE VII
D EEP L EARNING -BASED IDS FOR I OT
IDS, researchers have produced several studies that leverage performance. In regards to the solutions for balancing the
ML/DL models for the detection of cyberattacks, comparing available dataset and other characteristics, Table VII sum-
and evaluating the performance of several types of mod- marises the advantages and disadvantages of recent literature
els on a series of widely employed datasets. Unlike other on DL-based IDS for IoT systems. In the smart environment
attack/abnormality detection methods, such as rule-based IDS, context, 37.5% of the presented work neglected to perform
ML-based IDS face data-related challenges during training, model optimisation, with 25% presenting overfitting character-
such as overfitting, underfitting and model optimisation. The istics, while other issues, such as poor pre-processing/feature-
challenge of overfitting relates to the performance of a super- selection, hyperparameter tuning and lack of real-time design
vised model to differentiate between the available classes, were present in 50%.
characterising a model that maps to training data (near)- This section has focused on reviewing IDS-related literature
perfectly (low bias) but fails to perform equally as accurately and related applications in a cyber defence context. The recent
when faced with previously unseen data (high variability). development in ML-based IDSs is discussed; most researchers
Underfitting models, similar to their Overfitting counterparts, utilized CNN and RNN architectures. Overfitting is identi-
face similar high variability, rendering them inapplicable in fied as a primary deterrent for the real-world applicability
real-world scenarios, however, they also fail to effectively of some of these works, while a lack of optimisation of the
capture the associations in the training dataset, thus per- employed models further hinders their effectiveness and timely
forming poorly when evaluating on previously seen records deployment.
(high bias). As a means of training ML models, often a loss
function is defined, which indicates the performance of the
model, after the completion of a single batch. Training of X. R ESPONSIBLE A RTIFICIAL I NTELLIGENCE
such models is carried out by attempting to minimise this loss The impact of AI and its applications to everyday life
function, thus reducing the errors of the model, in a process is ever-increasing and spans multiple domains, including the
called model optimisation, with examples including Stochastic medical sector, where it enhances the diagnostic capabilities
gradient descent (SGD) and Adam. of medical practitioners [195], to the agricultural and automo-
In regards to those characteristics and a few others, includ- tive industries, where it enables automated applications such as
ing accuracy rate and performance comparison, Table VI monitoring for changes in environmental conditions that may
summarises the advantages and disadvantages of existing lit- affect crops and automated vehicle construction [196], [197].
erature on DL-based IDSs. Briefly, it is evident from the table However, societal reliance on such automated solutions, cou-
that, although some researchers investigated the performance pled with their opaque nature that hinders interpretability, has
of several types of DL models, while others studied a range brought into question the trustworthiness of AI models.
of model configurations, to detect optimised performance, In light of these concerns and the challenges of unearthing
approximately 61.5% of the presented work suffered from hidden biased in the model, as illustrated by real-world inci-
overfitting, while 69.2% based their work on outdated datasets, dents where automated AI-based applications have been shown
which would result in IDSs with negatively affected (degraded) to suffer from biases that affect their operations, introduced
to the AI models by improperly curated data, the need for can assist end-users in selecting an appropriate AI-enabled
responsible AI arose. Being an emerging concept, and an application for their respective problem.
open research area, the existing literature lacks a commonly To ensure that AI is robust and AI-enabled applications
accepted definition for responsible AI and what distinguishes operate as expected and to reduce the number of bugs,
it from conventional AI applications however, some organisa- Google AI proposes a rigorous phase of preliminary testing,
tions have produced their own strategies and terminologies. where each autonomous component of the application/model
is initially tested in a controlled and isolated environment.
A. Microsoft-Based Definition and Practices Integration assessments are then conducted, where the inter-
actions between individual components and other application
According to Microsoft [198], a responsible AI approach
parts are assessed, prior to assembling them into a unified
should not only consider the data that is leveraged for training
application. Additionally, input drift testing should be imple-
and evaluation, ensuring that no hidden biases and misrepre-
mented to ensure that the data does not change in unexpected
sentations are detected and addressed effectively but also the
ways, while also incorporating background quality checks is
end-users that will be affected by the AI solution. Microsoft
crucial, to ensure that, in case a component of the AI appli-
defines responsible AI as the ethical design of AI-based
cation fails, potentially faulty predictions are not going to be
solutions, guided by the principles of fairness, inclusiveness,
generated.
reliability, safety, transparency, accountability and privacy.
Microsoft approaches fairness and inclusiveness in their AI
applications by focusing on the impact that flawed AI systems C. Literature-Based Definition and Practices
may have on end-user groups, focusing on racial, gender, age
According to the literature, it is evident that responsible
and disability biases.
AI is a collection of best practices and recommendations that
AI systems are reliable, if they display consistency dur-
guide the design of AI applications, to prevent states of unfair-
ing both testing in a controlled environment, and also in the
ness, with results being uninterpretable and unexplainable and
real world; unreliable behaviour displayed by AI systems may
to ensure that the privacy and safety of users are not compro-
introduce safety risks. To accommodate for such reliability
mised. To that effect, Wang et al. [200] identified four primary
and safety challenges, Microsoft has focused on investigat-
practices that govern the implementation of responsible AI,
ing dataset blindspots and the variations between testing and
pertaining to data governance, ethics, user safety and employee
deployment environments. Transparency and accountability
education. The best practices regarding data governance are
are approached by developing methods, based on existing XAI
separated into three characteristics: transparency, explainabil-
approaches and by investigating the characteristics and origins
ity and trust. Transparency is built by enabling end-users and
of datasets that are employed during training. To establish a
stakeholders in organisations to understand how their data is
better understanding of datasets, the usage of datasheets that
being processed by an AI model to reach an outcome.
would describe a dataset, key components of its creation and
Explainability is built by providing easy-to-interpret expla-
its strengths/weaknesses is advised.
nations that are tailored for each end-user type (according to
their levels of technological competence). Trust is built by
B. GoogleAI-Based Definition and Practices leveraging high-quality data that has been carefully curated,
According to Google AI [199], responsible AI is achieved pre-processed and gathered with consent from end users.
by focusing on a similar approach to Microsoft, in that both Ethics should be introduced into the design of AI by adopting
organisations have elected to follow a human-centred design, an ethical mindset (AI engineers) and culture (organisations),
however, Google emphasises a continuously monitored and ensuring that the goal of an AI application is to fairly bene-
updated design, to ensure an up-to-date and adaptable respon- fit society. Furthermore, the AI algorithms that are applied for
sible AI system. During the training process of ML and the development of automated applications should be selected,
DL models, one important factor that greatly affects their modified or designed with ethics in mind, ensuring that a
performance is the quality of data. Google AI recommends a subset of users does not receive unethical or unlawful benefits.
thorough pre-processing phase, where data analysts can ensure Additionally, AI engineers need to be able to identify,
that the collected data is representative of the intended user through manual testing and evaluation, ethical challenges that
population, missing values are handled, redundancy in input lead to unethical or socially discriminating automata (AI,
features is addressed and data biases are detected and mitigated robots, unmanned vehicles). User safety is another crucial
prior to training. aspect of responsible AI, and it needs to be ensured, through
Furthermore, Google AI asserts that the limitations of AI the application of risk control mechanisms. Risks in this con-
models need to be clearly communicated to end-users, to text can include security risks, pertaining to cybersecurity, user
ensure that they are applied in the intended fashion to build privacy and AI adversarial attacks, economic risk, pertaining
confidence. For instance, a linear regression model trained on to job losses and reputation harm and performance risk, per-
a dataset comprised of sales data from the U.S. between the taining to unreliable performance due to high false alarm rates
years 1995 and 2001, may not produce accurate prices for and a lack of transparency which could help AI engineers in
homes in the year 2022. The disclosure of such information, identifying errors in their models.
pertaining to the limitations of an AI model including descrip- Finally, for responsible AI to be realised, a crucial prac-
tive information about the dataset that was used to train it, tice is to provide educational programs for individuals in an
organisation, from managers to employees, and help them the exact investigational settings to make the acquired
build a solid foundation of ethics in the AI context and the role attributes profoundly connected to the task being con-
data plays. These programs dictate ethical design, development sidered. In contrast, the statement is correct when the
and use principles of AI in real-world implementations, along investigation task is more complex, and massive quanti-
with best practices for ethically acquiring representative and ties of data are available to train a complex black-box
high-quality data to be used for AI training. method. This holds true when the model’s accuracy is
expressively enhanced by feeding more input data into a
D. Summary and Insights DL-based method with additional hidden computational
In this section, the concept of responsible AI was inves- layers. Nevertheless, the interpretability of the model’s
tigated, providing definitions and best practices from major decisions becomes difficult to ascertain [202]. So, in such
companies and the literature. According to our research, we a situation, the balance between explainability and detec-
conclude that responsible AI is a set of principles that governs tion performance needs to be well thought out. The need
the development of real-world, problem-solving applications to ensure that this balance is maintained has motivated
that are powered by AI models and provide users with trans- scholars to sacrifice interpretability to expand the effi-
parency into their internal mechanics and a human-friendly ciency of the employed model. Consequently, electing to
explanation of output according to input data. Such responsi- employ a more complicated approach, results in a more
ble AI models are built on unbiased data that fairly represent precise answer to the given problem, while at the same
the population of the intended target group(s) of users. Their time, interpretability suffers. This phenomenon drives
design should enable lawmakers to effortlessly gauge compli- many researchers to over-correct or lessens this trend
ance with laws and regulations and in case of wrongdoing, the at least. For example, the authors of [203] investigated
process for deciding accountability should be straightforward. the balance between the interpretability and efficiency
It should be a fundamental characteristic of such respon- of a model using a simplified method. They focused
sible AI models, to discern which input feature, data point on lessening the binary classification risks and consid-
or stream of telemetry data (in the case of IoT settings), ered interpretability as a restriction imposed throughout
was responsible for leading the AI system into causing harm the training stage. Their empirical outcomes verified that
and whether the model carries biases introduced during train- the developed approach accurately analysed the influen-
ing, in which case the AI data scientist incur blame. An tial factors and the attributes that affected the model’s
integral part of ensuring that responsible AI is maintained performance. Generally, the capability of interpretability
is to design a feedback/update loop in the application life- of the model is estimated to rise meaningfully, in accor-
cycle, to address any emerging concerns, such as arising dance with the model’s efficiency, a fact that is supported
user-date privacy concerns and flaws in the model’s predicting by a growing number of studies that suggested novel
capabilities. approaches for XAI in IoT in past years.
• Assessment Methods and XAI Concepts in IoT: To ensure
XI. R ESEARCH C HALLENGES , L ESSONS L EARNED AND that the domain of XAI flourishes, particularly in the
F UTURE D IRECTION context of IoT, it is crucial to construct the funda-
mental literature and to ensure that research builds on
This section assesses the mentioned studies and provides the prior and mutually accepted knowledge, thus avoiding
concluding observations on the emerging topics, achievements, the pitfall of re-inventing the wheel. The existing studies
and the existing complications that involve extra consideration disclosed the two thoughts recommended by two key bod-
in the XAI-based cyber defences in the IoT field. Further, we ies (the FAT and the DARPA). The researchers have also
also propose some new research directions in the XAI field as explored other concepts to define the concepts of XAI.
future directions. For example, the authors of [52] identified XAI as the
model’s ability to make its internal procedures more see-
A. Lessons Learned and Future Directions through to the viewers by employing the post-modelling
• Balance and Trade-Off Between Explainability and approaches. Although the concepts of XAI offered in this
Efficiency: Although the balance between efficiency and study could be deficient since the research in the XAI
explainability has conversed many times, this problem field is still in the initial progressing phase, it assumes a
is still plagued by misinformation as is the case with significant role, as a reference point for future research.
other difficulties in XAI implementations in IoT. The Moreover, it is well thought that the research commu-
writers of [201] confirmed in their study that it is not nity focusing on XAI advances in the IoT domain will
completely right that the most multifaceted black-box sooner or later grasp an integrated notion for XAI by link-
approaches constantly produce a greater accuracy rate ing the fragmented contributions via rising XAI research
for any given detection task. For example, this state- trends. The metrics are essential for assessing a specific
ment is proven wrong in a case where the training data method [204]. In the XAI perspective of the IoT domain,
is preprocessed and prepared well by having only the the assessment metrics allow a systematic calculation of
vital features. The issue generally occurs in applications performance, i.e., the standard metrics, recall, accuracy,
related to industrial sectors, as the limitations are the and precision. Several studies have been conducted to
group for the features and data being inspected within converse the assessment metrics for XAI, e.g., [52], [205].
Generally speaking, the proposed metrics allow schol- – explainability at the post-modelling level delivers
ars to assess the efficiency of the XAI-based model and supplementary interpretations to increase the under-
expand the sureness and belief of end-users and secu- standability of the prediction decisions taken by the
rity experts. These two types of research focused on the employed AI model.
assessment metrics for XAI models and demonstrate ade- – Explainability can assist the developers in certifying
quately the process of assessing XAI approaches in IoT. that the scheme is working properly as anticipated.
Nevertheless, more quantifiable and standard assessment Traditional security systems lack these features, making
metrics are required to calculate IoT networks’ rising XAI them less trustworthy for employing the cyber defence
methods and tools. This idea could be addressed more in model in critical infrastructures.
future, and more efforts are needed to develop new and
efficient methods for assessing XAI models related to IoT.
B. Challenges of Building an Efficient Anomaly
• XAI Concepts for DL: Although considerable work has
Detection in IoT
been done to interpret the DL method in past years,
numerous challenges need to be resolved before acquir- Despite many pieces of research in anomaly detection
ing the complete interpretability for the DL methods. showing good performance with benchmarking datasets, it is
The standard definition of XAI concepts concerning DL challenging to build an effective IDS with a high detection
models is still unavailable, as the XAI research is still rate, scalability, robustness and protection against all attack
in the early stage. During our research, it was observed vectors [3], especially zero-day attacks and multi-stage attacks.
that feature importance and feature relevance terms are This section presents challenges for designing an efficient IDS
used indistinguishably. Additionally, the task of explana- in IoT.
tion is unavoidable for the multifaceted methods, i.e., DL • Data source is a crucial anomaly detection component in
methods having numerous non-linear functions for acti- the training and testing process. However, it is impos-
vation, which are dissimilar from linear methods that are sible to construct a dataset that involves all normal
entirely explainable. Scholars from diverse areas (such as and malicious behaviours in a heterogeneous environ-
medicine, banking, cyber security, and law) need to focus ment of IoT. Many existing datasets suffer from missing
on this challenge, to produce understandable explanations labels and poor attack diversity. Moreover, most of
for DL-based models. them collect an incomplete set of features, and network
• Explainable IDS as Cyber Defence: As witnessed in lit- information is captured without including headers and
erature, organisations and companies rely heavily on IDS payloads. In addition, the IDS may only perform well
to protect against cyberattacks and monitor devices and on a limited number of devices because collected data
network packets to detect anomalous activities. But, the only contains network activities of a few types of IoT
generation of false alarms in high volume from such devices [3].
security tools makes it challenging for security experts • Real-time detection is also a challenge. Collecting and
and administrators to isolate the valid alarms from false monitoring network traffic in real-time would cause
positives for additional research. The explainability of a long processing time and a high false alarm rate,
the employed algorithm in an IDS model becomes vital, which can degrade the network performance signifi-
which in turn helps security professionals to understand cantly. Therefore, the data processing module and the
underlying evidence and causal reasoning. The explain- detection method must be adopted carefully to mitigate
able units in any IDS-based cyber defence system will this [20].
enable the administrators to trust their employed system • Although the IDS using DL-based techniques shows good
and make more knowledgeable decisions driven by the performance, applying such techniques in IoT environ-
rapid abolition of false-positive rates. ments is difficult due to the limited computation resources
• Explainable IDS vs Traditional IDS for IoT: For years, the on IoT devices. Moreover, ML and DL-based tech-
ML and DL models have been used for the protection of niques are computationally expensive, which leads to
IoT-based environments. However, the black-box nature network latency issues and hence, become impossible to
of ML/DL models makes it hard for the researchers to be used in critical sectors such as the Internet of Vehicles
grasp the internal workings of the employed models. The (IoVs) [206].
XAI field has emerged to highlight the importance of • Defending against multi-stage attacks (MSA) remains a
interpretability in securing networks against cyber threats, challenge for existing IDS techniques. IDS effectively
especially IoT-based platforms. The literature studied in detects single-stage attacks as malicious activities are
this paper indicates that XAI-based IDS methods have conducted quickly. MSA utilizes complex attack tech-
numerous advantages over traditional IDSs. For example: niques over a long period [207] in which each step taken
– explainability at the pre-modelling level enhances is insufficient to be recognized as aggression. Therefore,
network traffic’s data quality through different data the MSA avoids being detected by IDS. In [208], the
cleaning techniques. authors point out several challenges for detecting MSA,
– explainability at the modelling level facilitates the including: (1) modelling MSAs, (2) building a system
security experts to gain insights into the internal to detect and track the progress of interleaved MSA, and
functioning of the AI model. (3) constructing datasets with interleaved MSA scenarios.
C. Challenges of Developing Effective XAI-Based To sum up, the field of XAI has emerged with the need
Cyber Defences to understand the predictions and functioning of ML models
While XAI is being focused on and efforts have been put on, and develop robust XAI models that benefit various types of
many limitations have to be improved to obtain explainability audiences [52]. XAI methods have witnessed many studies and
in AI. This section presents challenges for XAI-based cyber applications across a wide range of subjects, such as image
defences. recognition [212], [213], [214] and natural language process-
• Accessible Explanations: instead of focusing on the ing [215]. However, there have been very few studies in IDS.
demands of different types of audiences as in Figure 3, Nowadays, as human life significantly depends on computer
most current XAI methods produce results that only make systems, IDS is crucial for the security of any organisation’s
sense to researchers. In addition, some XAI methods pro- system. Additionally, there is a growing need for explainable
duce explanations as important feature vectors, which AI-based IDS since the model’s recipients require explana-
are inherently low-level explanations. This format would tions for many valid purposes. Throughout the literature, there
be useful for developers and researchers to debug or are many examples of ML-based IDS (Section VIII-C) and
design the models. However, other audience types may various XAI methods, with very few addressing XAI-based
find such an explanation format complex, confusing and IDS challenges.
useless [21]. Ideally, the XAI method should provide This paper identifies a gap in existing research that is of
accessible explanations for all types of audiences, which utmost importance. The field of XAI is relatively young, and
means they should be easily utilised by society, espe- there is limited existing literature addressing the explainable
cially policymakers and the law [52]. This remains a big AI-based IDS. There is a lack of studies to create new XAI
challenge in the field of XAI. methods for IDS and evaluate existing XAI approaches for
• Standardised Terminology and Evaluation of ML-based IDS. This provides a research opportunity to under-
Explanations: the XAI research community does stand existing XAI methods for IDS and design a novel,
not have standardised terminology. Terminologies in explainable AI-based IDS. Current challenges in Section XI
XAI are defined differently by researchers as there are are recommended as future research to tackle and develop
no standardised definitions, and vocabularies [52], which XAI-enabled cyber defences in IoT ecosystems.
would lead to confusion. This makes it impossible to
XII. C ONCLUSION
evaluate and compare the performance of different XAI
methods as the ground truth is unknown. Moreover, Although XAI is a newly emerging concept that attracts
no criteria have been standardised that consider the growing interest, this paper has examined the value of
subjective measures used in human-centred evaluations. Explainability in Cyber Defences and IoT networks. This
It is also impossible to evaluate all XAI methods by a survey focused on various features of XAI in the form of
defined evaluation metric [209]. All the above aspects recent works and existing challenges in network security,
need to be considered carefully to develop a practical Intrusion Detection Systems (IDS), Artificial Intelligence (AI)
evaluation of XAI explanations. and security issues in the Internet of Things (IoT) networks.
• Statistical Uncertainty: Many XAI methods (e.g., PFI and Essential concepts of XAI concerning cyber defences are dis-
SHAP) are subject to uncertainty as they provide explana- cussed; since digital transformation is taking place across
tions by computing from data. However, the uncertainty many industries, the security, privacy and interpretation of
of the explanation is not addressed as explanations are modelled systems arise as a considerable concern. Recent lit-
given. Consequently, the explanations are not reliable and erature shows that anomaly-based IDS, which uses machine
compromise the responsible AI. To fix this, a rigorous learning algorithms and deep learning as an underlying detec-
approach should be adopted to study the uncertainty of tion method, achieves excellent results in preventing unknown
XAI methods [210]. Otherwise, XAI has to face statistical attacks. The heterogeneous nature of IoT also encourages the
testing problems such as p-hacking [211]. utilisation of AI techniques in IDS due to AI’s capability of
• Systematic Instability: This characteristic affects the analysing and learning attack patterns from a large scale of
performance of XAI methods that analyse the internal data. This paper has surveyed recent works in ML-based IDS,
structure of the models. Many ML models can perform especially DL-based IDS. It is observed that many proposed
well on a specifically labelled dataset; however, their models suffer from outdated or unbalanced datasets, which can
internal pathways might differ due to the complexity of degrade the model’s efficiency in real-life implementation. In
ML techniques. Such differences would lead to changes this work, we have presented the basic ways to clutch some
in generated explanations across multiple models [54]. elementary knowledge of XAI in cyber defences. This will
• Feature Dependence: Many XAI methods (e.g., PDP, assist researchers in exploring the challenges and future stud-
LIME and PFI) suffer from the assumption of indepen- ies of explainable AI for developing context-aware anomaly
dence; thus, they automatically create invalid data points. detection methods in IoT as cyber defence systems.
Using such data points probably degrades the reliabil-
ity of XAI methods as explanations are given unreal R EFERENCES
data instances and will never happen in real-life [210]. [1] K. Tange, M. De Donno, X. Fafoutis, and N. Dragoni, “A systematic
survey of Industrial Internet of Things security: Requirements and fog
Technically, when features in the dataset are correlated, computing opportunities,” IEEE Commun. Surveys Tuts., vol. 22, no. 4,
the explanations would be misleading. pp. 2489–2520, 4th Quart., 2020.
[2] J. Srinivas, A. K. Das, and N. Kumar, “Government regulations in [24] A. Nguyen et al., “System design for a data-driven and explainable cus-
cyber security: Framework, standards and recommendations,” Future tomer sentiment monitor using IoT and enterprise data,” IEEE Access,
Gener. Comput. Syst., vol. 92, pp. 178–188, Mar. 2019. vol. 9, pp. 117140–117152, 2021.
[3] A. Nisioti, A. Mylonas, P. D. Yoo, and V. Katos, “From intrusion detec- [25] D. B. Das and D. Birant, “XHAC: Explainable human activity classi-
tion to attacker attribution: A comprehensive survey of unsupervised fication from sensor data,” in Emerging Trends in IoT and Integration
methods,” IEEE Commun. Surveys Tuts., vol. 20, no. 4, pp. 3369–3388, With Data Science, Cloud Computing, and Big Data Analytics.
4th Quart., 2018. New York, NY, USA: IGI Global, 2022, pp. 146–164.
[4] M. A. Al-Garadi, A. Mohamed, A. K. Al-Ali, X. Du, I. Ali, and [26] B. Hou et al., “Mitigating the backdoor attack by federated filters
M. Guizani, “A survey of machine and deep learning methods for for Industrial IoT applications,” IEEE Trans. Ind. Informat., vol. 18,
Internet of Things (IoT) security,” IEEE Commun. Surveys Tuts., no. 5, pp. 3562–3571, May 2022.
vol. 22, no. 3, pp. 1646–1685, 3rd Quart., 2020. [27] J. Rožman, H. Hagras, J. Andreu-Perez, D. Clarke, B. Müller, and
[5] H. Habibzadeh, B. H. Nussbaum, F. Anjomshoa, B. Kantarci, and S. Fitz, “A type-2 fuzzy logic based explainable AI approach for the
T. Soyata, “A survey on cybersecurity, data privacy, and policy issues easy calibration of AI models in IoT environments,” in Proc. IEEE Int.
in cyber-physical system deployments in smart cities,” Sustain. Cities Conf. Fuzzy Syst. (FUZZ-IEEE), 2021, pp. 1–8.
Soc., vol. 50, Oct. 2019, Art. no. 101660. [28] L. Alkhariji, N. Alhirabi, M. N. Alraja, M. Barhamgi, O. Rana, and
[6] S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y. Zomaya, C. Perera, “Synthesising privacy by design knowledge toward explain-
“Edge intelligence: The confluence of edge computing and artificial able Internet of Things application designing in healthcare,” ACM
intelligence,” IEEE Internet Things J., vol. 7, no. 8, pp. 7457–7469, Trans. Multimedia Comput. Commun. Appl., vol. 17, no. 2s, pp. 1–29,
Aug. 2020. 2021.
[7] S. Morgan, Cybercrime to Cost the World $10.5 Trillion Annually by [29] M. A. Rahman, M. S. Hossain, A. J. Showail, N. A. Alrajeh, and
2025, Cybercrime Mag., Northport, NY, USA, 2020. M. F. Alhamid, “A secure, private, and explainable IoHT framework
[8] ACSC. “Australian cyber security centre annual cyber threat report to support sustainable health monitoring in a smart city,” Sustain. Cities
2020–21.” 2021. [Online]. Available: https://www.cyber.gov.au/acsc/ Soc., vol. 72, Sep. 2021, Art. no. 103083.
view-all-content/reports-and-statistics/acsc-annual-cyber-threat-report- [30] M. Zolanvari, Z. Yang, K. Khan, R. Jain, and N. Meskin, “TRUST XAI:
2020-21 Model-agnostic explanations for AI with a case study on IIoT security,”
[9] S. Das. “Direct costs associated with cybersecurity incidents costs IEEE Internet Things J., vol. 10, no. 4, pp. 2967–2978, Feb. 2023.
Australian businesses $29 billion per annum.” 2018. [Online]. [31] W. S. Monroe, F. M. Skidmore, D. G. Odaibo, and M. M. Tanik, “HihO:
Available: https://news.microsoft.com/en-au/features/direct-costs- Accelerating artificial intelligence interpretability for medical imag-
associated-with-cybersecurity-incidents-costs-australian-businesses-29- ing in IoT applications using hierarchical occlusion,” Neural Comput.
billion-per-annum/ Appl., vol. 33, no. 11, pp. 6027–6038, 2021.
[10] “U.S. government to spend over $18 billion on cybersecurity—Atlas [32] Y. Chai, Y. Zhou, W. Li, and Y. Jiang, “An explainable multi-
VPN.” 2020. [Online]. Available: https://www.globalsecuritymag. modal hierarchical attention model for developing phishing threat
com/US-government-to-spend-over-18,20200622,99884.html#::text= intelligence,” IEEE Trans. Depend. Secure Comput., vol. 19, no. 2,
According%20to%20Atlas%20VPN%20investigation%2C%20the%20 pp. 790–803, Mar./Apr. 2022.
US%20government,government%20and%20all%20critical%20infrastr [33] A. K. Sarica and P. Angin, “Explainable security in SDN-based IoT
ucture%20from%20cyber%20threats networks,” Sensors, vol. 20, no. 24, p. 7326, 2020.
[11] J. Navarro, A. Deruyver, and P. Parrend, “A systematic survey on [34] I. García-Magariño, R. Muttukrishnan, and J. Lloret, “Human-centric
multi-step attack detection,” Comput. Security, vol. 76, pp. 214–249, AI for trustworthy IoT systems with explainable multilayer percep-
Jul. 2018. trons,” IEEE Access, vol. 7, pp. 125562–125574, 2019.
[12] I. Ghafir et al., “Detection of advanced persistent threat using machine- [35] E. Oyekanlu, “Distributed osmotic computing approach to implemen-
learning correlation analysis,” Future Gener. Comput. Syst., vol. 89, tation of explainable predictive deep learning at industrial iot network
pp. 349–359, Dec. 2018. edges with real-time adaptive wavelet graphs,” in Proc. IEEE 1st Int.
[13] Y.-D. Lin, “Editorial: Second quarter 2020 IEEE communications sur- Conf. Artif. Intell. Knowl. Eng. (AIKE), 2018, pp. 179–188.
veys and tutorials,” IEEE Commun. Surveys Tuts., vol. 22, no. 2, [36] K. Amarasinghe, K. Kenney, and M. Manic, “Toward explainable deep
pp. 790–795, 2nd Quart., 2020. neural network based anomaly detection,” in Proc. IEEE 11th Int. Conf.
[14] S. Mohammadi, H. Mirvaziri, M. Ghazizadeh-Ahsaee, and Human Syst. Interact. (HSI), 2018, pp. 311–317.
H. Karimipour, “Cyber intrusion detection by combined feature [37] B. M. Keneni et al., “Evolving rule-based explainable artificial
selection algorithm,” J. Inf. Security Appl., vol. 44, pp. 80–88, intelligence for unmanned aerial vehicles,” IEEE Access, vol. 7,
Feb. 2019. pp. 17001–17016, 2019.
[15] I. H. Sarker, M. H. Furhad, and R. Nowrozy, “AI-driven cybersecurity: [38] H. Liu, C. Zhong, A. Alnusair, and S. R. Islam, “FAIXID: A framework
An overview, security intelligence modeling and research directions,” for enhancing AI explainability of intrusion detection results using data
SN Comput. Sci., vol. 2, no. 3, pp. 1–18, 2021. cleaning techniques,” J. Netw. Syst. Manag., vol. 29, no. 4, pp. 1–30,
[16] T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for cyber 2021.
security,” IEEE Trans. Neural Netw. Learn. Syst., early access, Nov. 1, [39] S. Mane and D. Rao, “Explaining network intrusion detection system
2021, doi: 10.1109/TNNLS.2021.3121870. using explainable AI framework,” 2021, arXiv:2103.07110.
[17] G. De Luca and Y. Chen, “Explainable artificial intelligence for [40] Q.-V. Dang, “Improving the performance of the intrusion detection
workflow verification in visual IoT/robotics programming language systems by the machine learning explainability,” Int. J. Web Inf. Syst.,
environment,” J. Artif. Intell. Technol., vol. 1, no. 1, pp. 21–27, 2021. vol. 17, no. 5, pp. 537–555, Jun. 2021.
[18] J. D. Moore and W. R. Swartout, “Explanation in expert systemss: [41] G. Andresini, A. Appice, F. P. Caforio, D. Malerba, and G. Vessio,
A survey,” Marina del Rey Inf. Sci., Univ. Southern California, “ROULETTE: A neural attention multi-output model for explainable
Los Angeles, CA, USA, Rep. ADA206283, 1988. network intrusion detection,” Exp. Syst. Appl., vol. 201, Sep. 2022,
[19] T. Miller, “Explanation in artificial intelligence: Insights from the social Art. no. 117144.
sciences,” Artif. Intell., vol. 267, pp. 1–38, Feb. 2019. [42] C. Wu, A. Qian, X. Dong, and Y. Zhang, “Feature-oriented design
[20] N. Moustafa, J. Hu, and J. Slay, “A holistic review of network anomaly of visual analytics system for interpretable deep learning based intru-
detection systems: A comprehensive survey,” J. Netw. Comput. Appl., sion detection,” in Proc. IEEE Int. Symp. Theory Aspects Softw. Eng.
vol. 128, pp. 33–55, Feb. 2019. (TASE), 2020, pp. 73–80.
[21] M. Du, N. Liu, and X. Hu, “Techniques for interpretable machine learn- [43] Q.-V. Dang, “Understanding the decision of machine learning based
ing,” Commun. ACM, vol. 63, no. 1, pp. 68–77, Dec. 2019. [Online]. intrusion detection systems,” in Proc. Int. Conf. Future Data Security
Available: https://doi.org/10.1145/3359786 Eng., 2020, pp. 379–396.
[22] I. A. Khan, N. Moustafa, D. Pi, K. M. Sallam, A. Y. Zomaya, and [44] N. L. Tsakiridis et al., “Versatile Internet of Things for agriculture: An
B. Li, “A new explainable deep learning framework for cyber threat explainable AI approach,” in Proc. IFIP Int. Conf. Artif. Intell. Appl.
discovery in industrial iot networks,” IEEE Internet Things J., vol. 9, Innov., 2020, pp. 180–191.
no. 13, pp. 11604–11613, Jul. 2022. [45] I. T. Christou, N. Kefalakis, A. Zalonis, and J. Soldatos, “Predictive and
[23] I. A. Khan et al., “XSRU-IoMT: Explainable simple recurrent units explainable machine learning for Industrial Internet of Things applica-
for threat detection in Internet of Medical Things networks,” Future tions,” in Proc. 16th Int. Conf. Distrib. Comput. Sensor Syst. (DCOSS),
Gener. Comput. Syst., vol. 127, pp. 181–193, Feb. 2022. 2020, pp. 213–218.
[46] A. Y. Al Hammadi et al., “Explainable artificial intelligence to evaluate [71] S. P. Portillo, J. Manuel Estévez, and T. Leganés, “Attacks against
industrial internal security using EEG signals in IoT framework,” intrusion detection networks: Evasion, reverse engineering and optimal
Ad Hoc Netw., vol. 123, Dec. 2021, Art. no. 102641. countermeasures,” Comput. Sci. Eng. Dept., Universidad Carlos III
[47] D. Minh, H. X. Wang, Y. F. Li, and T. N. Nguyen, “Explainable artifi- de Madrid, Madrid, Spain, Rep. TR-2014, 2014. [Online]. Available:
cial intelligence: A comprehensive review,” Artif. Intell. Rev., vol. 55, https://core.ac.uk/download/pdf/29406661.pdf
pp. 3503–3568, Nov. 2021. [72] B. von Solms and R. von Solms, “Cybersecurity and information
[48] M. Ivanovs, R. Kadikis, and K. Ozols, “Perturbation-based methods for security—What goes where?” Inf. Comput. Security, vol. 26, no. 1,
explaining deep neural networks: A survey,” Pattern Recognit. Lett., pp. 2–9, 2018.
vol. 150, pp. 228–234, Oct. 2021. [73] D. Kuehl, “From cyberspace to cyberpower: Defining the problem,” in
[49] W. Guo, “Explainable artificial intelligence for 6G: Improving trust Cyberpower National Security, vol. 30. Washington, DC, USA: Nat.
between human and machine,” IEEE Commun. Mag., vol. 58, no. 6, Defense Univ. Press, 2009.
pp. 39–45, Jun. 2020. [74] D. J. Gunkel, “Hacking cyberspace,” J. Antimicrobial Chemotherapy,
vol. 20, no. 4, pp. 797–823, 2000.
[50] E. Tjoa and C. Guan, “A survey on explainable artificial intelligence
[75] H. Ghadeer, “Cybersecurity issues in Internet of Things and coun-
(XAI): Toward medical XAI,” IEEE Trans. Neural Netw. Learn. Syst.,
termeasures,” in Proc. IEEE Int. Conf. Ind. Internet (ICII), 2018,
vol. 32, no. 11, pp. 4793–4813, Nov. 2021.
pp. 195–201.
[51] C. Meske, E. Bunde, J. Schneider, and M. Gersch, “Explainable [76] M. Abomhara and G. M. Kien, “Cyber security and the Internet
artificial intelligence: Objectives, stakeholders, and future research of Things: Vulnerabilities, threats, intruders and attacks,” J. Cyber
opportunities,” Inf. Syst. Manag., vol. 39, no. 11, pp. 53–63, 2020. Security Mobility, vol. 4, no. 1, pp. 65–88, Jan. 2015.
[52] A. B. Arrieta et al., “Explainable artificial intelligence (XAI): Concepts, [77] S. Mittal, P. K. Das, V. Mulwad, A. Joshi, and T. Finin, “CyberTwitter:
taxonomies, opportunities and challenges toward responsible AI,” Inf. Using Twitter to generate alerts for cybersecurity threats and vulner-
Fusion, vol. 58, pp. 82–115, Jun. 2020. abilities,” in Proc. IEEE/ACM Int. Conf. Adv. Soc. Netw. Anal. Min.
[53] M. Langer et al., “What do we want from explainable artificial intel- (ASONAM), Nov. 2016, pp. 860–867.
ligence (XAI)—A stakeholder perspective on XAI and a conceptual [78] H. Zhang, P. Cheng, L. Shi, and J. Chen, “Optimal denial-of-service
model guiding interdisciplinary XAI research,” Artif. Intell., vol. 296, attack scheduling with energy constraint,” IEEE Trans. Autom. Control,
Jul. 2021, Art. no. 103473. vol. 60, no. 11, pp. 3023–3028, Nov. 2015.
[54] A. Adadi and M. Berrada, “Peeking inside the black-box: A sur- [79] T. Rid and B. Buchanan, “Attributing cyber attacks,” J. Strategic
vey on explainable artificial intelligence (XAI),” IEEE Access, vol. 6, Stud., vol. 38, pp. 4–37, Jan. 2015. [Online]. Available: https://www.
pp. 52138–52160, 2018. tandfonline.com/doi/abs/10.1080/01402390.2014.977382
[55] S. Chakraborty et al., “Interpretability of deep learning mod- [80] J. Zhao, S. Shetty, J. W. Pan, C. Kamhoua, and K. Kwiat, “Transfer
els: A survey of results,” in Proc. IEEE Smartworld Ubiquitous learning for detecting unknown network attacks,” EURASIP J. Inf.
Intell. Comput. Adv. Trust. Comput. Scalable Comput. Commun. Security, vol. 2019, no. 1, pp. 1–13, 2019.
Cloud Big Data Comput. Internet People Smart City Innov. [81] Q. Zou, X. Sun, P. Liu, and A. Singhal, “Advanced persistent threat
(Smartworld/SCALCOM/UIC/ATC/CBDcom/IOP/SCI), 2017, pp. 1–6. attack detection: An overview,” Int. J. Adv. Comput. Netw. Security,
[56] J. Franco, A. Aris, B. Canberk, and A. S. Uluagac, “A survey of honey- vol. 4, no. 4, p. 5054, 2014.
pots and honeynets for Internet of Things, Industrial Internet of Things, [82] L. Martin. “Cyber kill chain.” Accessed: Jul. 17, 2021. [Online].
and cyber-physical systems,” IEEE Commun. Surveys Tuts., vol. 23, Available: https://www.lockheedmartin.com/en-us/capabilities/cyber/
no. 4, pp. 2351–2383, 4th Quart., 2021. cyber-kill-chain.html
[57] CIM. “Explainable and trustworthy artifcial intelligence.” 2021. [83] R. Bace and P. Mell, Intrusion Detection Systems, Nat. Inst. Stand.
[Online]. Available: https://sites.google.com/view/specialissue-xai- Technol., Gaithersburg, MD, USA, 2001.
ieee-cim [84] R. Wazirali, “An improved intrusion detection system based on KNN
hyperparameter tuning and cross-validation,” Arabian J. Sci. Eng.,
[58] FGCS. “Explainable AI for healthcare.” 2021. [Online]. Available:
vol. 45, no. 12, pp. 10859–10873, 2020.
https://www.journals.elsevier.com/future-Gener.-Comput.-Syst./call-
[85] P. Sharma, J. Sengupta, and P. Suri, “Survey of intrusion detection
papers/explainable-Artif.-Intell.--healthcare.
techniques and architectures in cloud computing,” Int. J. High Perform.
[59] ICML. “Theoretic foundation, criticism, and application trend of Comput. Netw., vol. 13, no. 2, pp. 184–198, 2019.
explainable AI.” 2021. [Online]. Available: https://icml2021-xai.github. [86] H.-J. Liao, C.-H. R. Lin, Y.-C. Lin, and K.-Y. Tung, Intrusion Detection
io/ System: A Comprehensive Review. Amsterdam, The Netherlands:
[60] AI. “Explainable artificial intelligence.” 2020. [Online]. Available: Elsevier, 2012.
https://www.journals.elsevier.com/Artif.-Intell./callfor-papers/special- [87] T. Saranya, S. Sridevi, C. Deisy, T. D. Chung, and M. K. Khan,
issue-explainable-Artif.-Intell. “Performance analysis of machine learning algorithms in intru-
[61] BMVC. “Interpretable & explainable machine vision.” 2020. [Online]. sion detection system: A review,” Procedia Comput. Sci., vol. 171,
Available: https://arxiv.org/html/1909.07245 pp. 1251–1260, Jun. 2020.
[62] NIPS. “Interpreting, explaining and visualizing deep learn- [88] W. Li, W. Meng, and L. F. Kwok, “Surveying trust-based collaborative
ing.” 2019. [Online]. Available: http://www.interpretable-ml.org/ intrusion detection: State-of-the-art, challenges and future directions,”
nips2017workshop/ IEEE Commun. Surveys Tuts., vol. 24, no. 3, pp. 280–305, 1st Quart.,
[63] PR. “Explainable deep learning for efficient and robust pattern recog- 2022.
nition.” 2019. [Online]. Available: https://www.journals.elsevier.com/ [89] A. P. Kuruvila, X. Meng, S. Kundu, G. Pandey, and K. Basu,
pattern-Recognit./call-papers/call-paper-special-issue-explainabledeep- “Explainable machine learning for intrusion detection via hardware
Learn. performance counters,” IEEE Trans. Comput.-Aided Design Integr.
[64] IJCAI. “Explainable artificial intelligence (XAI).” 2019. [Online]. Circuits Syst., vol. 41, no. 11, pp. 4952–4964, Nov. 2022.
Available: https://sites.google.com/view/xai2019/home [90] N. Chaabouni, M. Mosbah, A. Zemmari, C. Sauvignac, and P. Faruki,
[65] V. Dignum, Responsible Artificial Intelligence: Designing AI for “Network intrusion detection for IoT security based on learning tech-
Human Values, ITU, Geneva, Switzerland, 2017. niques,” IEEE Commun. Surveys Tuts., vol. 21, no. 3, pp. 2671–2701,
3rd Quart., 2019.
[66] ICCV. “Interpretating and explaining visual artifcial intelligence mod-
[91] S. A. Ludwig, “Intrusion detection of multiple attack classes using a
els.” 2019. [Online]. Available: http://xai.unist.ac.kr/workshop/2019
deep neural net ensemble,” in Proc. IEEE Symp. Comput. Intell. (SSCI),
[67] Darpa. “Explainable artificial intelligence (XAI).” 2020. [Online]. Feb. 2018, pp. 1–7.
Available: https://www.darpa.mil/program/explainable-Artif.-Intell. [92] H. A. Kholidy and F. Baiardi, “CIDS: A framework for intrusion detec-
[68] ACM. “ACM conference on fairness, accountability, and transparency.” tion in cloud systems,” in Proc. 9th Int. Conf. Inf. Technol. New Gener.,
2020. [Online]. Available: https://fatconference.org 2012, pp. 379–385.
[69] F. Firouzi and B. Farahani, Architecting IoT Cloud. Cham, Switzerland: [93] L. Santos, C. Rabadao, and R. Gonçalves, “Intrusion detection systems
Springer Int., 2020, pp. 173–241. in Internet of Things: A literature review,” in Proc. 13th Iber. Conf. Inf.
[70] UberAccident. “What happens when self-driving cars kill peo- Syst. Technol. (CISTI), 2018, pp. 1–7.
ple.” 2019. [Online]. Available: https://www.forbes.com/sites/ [94] D. Moon, S. B. Pan, and I. Kim, “Host-based intrusion detection system
cognitiveworld/2019/09/26/what-happens-self-driving-cars-kill-people/ for secure human-centric computing,” J. Supercomput., vol. 72, no. 7,
#4b798bcc405c pp. 2520–2536, 2016.
[95] M. D. Singh, “Computer network and information security,” Comput. [121] B. Kim, E. Glassman, B. Johnson, and J. Shah, “iBCM: Interactive
Netw. Inf. Security, vol. 8, pp. 41–47, May 2014. [Online]. Available: Bayesian case model empowering humans via intuitive interaction,”
http://www.ossec.net/files/ossec-hids- Comput. Sci. Artif. Intell. Lab., Massachusetts Inst. Technol.,
[96] J. Peng, K.-K. R. Choo, and H. Ashman, “User profiling in intru- Cambridge, MA, USA, Rep. MIT-CSAIL-TR-2015-010, 2015.
sion detection: A review,” J. Netw. Comput. Appl., vol. 72, pp. 14–27, [122] B. Mahbooba, M. Timilsina, R. Sahal, and M. Serrano, “Explainable
Sep. 2016. artificial intelligence (XAI) to enhance trust management in intrusion
[97] M. Kumar, M. Hanumanthappa, and T. V. S. Kumar, “Encrypted traffic detection systems using decision tree model,” Complexity, vol. 2021,
and IPsec challenges for intrusion detection system,” in Proc. Int. Conf. Jan. 2021, Art. no. 6634811.
Adv. Comput., 2012, pp. 721–727. [123] M. Abdullahi et al., “Detecting cybersecurity attacks in Internet of
[98] E. Besharati, M. Naderan, and E. Namjoo, “LR-HIDS: Logistic regres- Things using artificial intelligence methods: A systematic literature
sion host-based intrusion detection system for cloud environments,” J. review,” Electronics, vol. 11, no. 2, p. 198, 2022.
Ambient Intell. Humanized Comput., vol. 10, no. 9, pp. 3669–3692, [124] D. Kaushik et al., “Application of machine learning and deep learn-
2019. ing in cybersecurity: An innovative approach,” in An Interdisciplinary
[99] M. A. Rahman, A. T. Asyhari, L. Leong, G. Satrya, M. H. Tao, Approach to Modern Network Security. Hoboken, NJ, USA: CRC
and M. Zolkipli, “Scalable machine learning-based intrusion detec- Press, 2022, pp. 89–109.
tion system for IoT-enabled smart cities,” Sustain. Cities Soc., vol. 61, [125] N. Capuano, G. Fenza, V. Loia, and C. Stanzione, “Explainable arti-
Oct. 2020, Art. no. 102324. ficial intelligence in cybersecurity: A survey,” IEEE Access, vol. 10,
[100] M. F. Elrawy, A. I. Awad, and H. F. Hamed, “Intrusion detec- pp. 93575–93600, 2022.
tion systems for IoT-based smart environments: A survey,” J. Cloud
[126] A. Das and P. Rad, “Opportunities and challenges in explainable
Comput., vol. 7, no. 1, pp. 1–20, 2018.
artificial intelligence (XAI): A survey,” 2020, arxiv.abs/2006.11371.
[101] B. B. Zarpelão, R. S. Miani, C. T. Kawakani, and S. C. de Alvarenga,
[127] B. Letham et al., “Interpretable classifiers using rules and bayesian
“A survey of intrusion detection in Internet of Things,” J. Netw.
analysis: Building a better stroke prediction model,” Ann. Appl. Stat.,
Comput. Appl., vol. 84, pp. 25–37, Apr. 2017.
vol. 9, no. 3, pp. 1350–1371, 2015.
[102] J. Ribeiro, F. B. Saghezchi, G. Mantas, J. Rodriguez, and
R. A. Abd-Alhameed, “HIDROID: Prototyping a behavioral host-based [128] K. Xu et al., “Show, attend and tell: Neural image caption genera-
intrusion detection and prevention system for Android,” IEEE Access, tion with visual attention,” in Proc. Int. Conf. Mach. Learn., 2015,
vol. 8, pp. 23154–23168, 2020. pp. 2048–2057.
[103] J. Ribeiro, F. B. Saghezchi, G. Mantas, J. Rodriguez, S. J. Shepherd, [129] B. Ustun and C. Rudin, “Supersparse linear integer models for opti-
and R. A. Abd-Alhameed, “An autonomous host-based intrusion detec- mized medical scoring systems,” Mach. Learn., vol. 102, no. 3,
tion system for android mobile devices,” Mobile Netw. Appl., vol. 25, pp. 349–391, 2016.
no. 1, pp. 164–172, 2020. [130] S. Sarkar, T. Weyde, A. Garcez, G. G. Slabaugh, S. Dragicevic, and
[104] D. Li, L. Deng, M. Lee, and H. Wang, “IoT data feature extraction C. Percy, “Accuracy and interpretability trade-offs in machine learning
and intrusion detection system for smart cities based on deep migration applied to safer gambling,” in Proc. CEUR Workshop, vol. 1773, 2016,
learning,” Int. J. Inf. Manag., vol. 49, pp. 533–545, Dec. 2019. pp. 1–9.
[105] H. S. Anderson, A. Kharkar, B. Filar, and P. Roth, “Evading machine [131] J. H. Friedman, “Greedy function approximation: A gradient boosting
learning malware detection,” in Proc. Black Hat, 2017, p. 9. machine,” Ann. Stat., vol. 29, no. 5, pp. 1189–1232, 2001.
[106] P. Lavanya and V. S. S. Sriram, “Detection of insider threats using [132] Q. Zhao and T. Hastie, “Causal interpretations of black-box models,”
deep learning: A review,” in Proc. Comput. Intell. Data Min., 2022, J. Bus. Econ. Stat., vol. 39, no. 1, pp. 272–281, 2021.
pp. 41–57. [133] B. M. Greenwell, “pdp: An R package for constructing partial depen-
[107] Q. Chen et al., “File fragment classification using grayscale image con- dence plots,” R J., vol. 9, no. 1, p. 421, 2017.
version and deep learning in digital forensics,” in Proc. IEEE Security [134] D. W. Apley and J. Zhu, “Visualizing the effects of predictor variables
Privacy Workshops (SPW), 2018, pp. 140–147. in black box supervised learning models,” J. Roy. Stat. Soc. B (Stat.
[108] X. Wang, J. Zhang, A. Zhang, and J. Ren, “TKRD: Trusted kernel Methodol.), vol. 82, no. 4, pp. 1059–1086, 2020.
rootkit detection for cybersecurity of VMs based on machine learning [135] A. Goldstein, A. Kapelner, J. Bleich, and E. Pitkin, “Peeking inside
and memory forensic analysis,” Math. Biosci. Eng., vol. 16, no. 4, the black box: Visualizing statistical learning with plots of individ-
pp. 2650–2667, 2019. ual conditional expectation,” J. Comput. Graph. Stat., vol. 24, no. 1,
[109] X.-D. Zhang, “Machine learning,” in A Matrix Algebra Approach to pp. 44–65, 2015.
Artificial Intelligence. Singapore: Springer, 2020, pp. 223–440. [136] A. Goldstein, A. Kapelner, and J. Bleich, “ICEbox: Individual condi-
[110] Y. Liu, F. R. Yu, X. Li, H. Ji, and V. C. M. Leung, “Blockchain and tional expectation plot toolbox,” 2017. [Online]. Available: https://rdrr.
machine learning for communications and networking systems,” IEEE io/cran/ICEbox/
Commun. Surveys Tuts., vol. 22, no. 2, pp. 1392–1431, 2nd Quart.,
[137] C. Chen, K. Lin, C. Rudin, Y. Shaposhnik, S. Wang, and T. Wang, “A
2020.
holistic approach to interpretability in financial lending: Models, visu-
[111] H. U. Dike, Y. Zhou, K. K. Deveerasetty, and Q. Wu, “Unsupervised
alizations, and summary-explanations,” Decis. Support Syst., vol. 152,
learning based on artificial neural network: A review,” in Proc. IEEE
Jan. 2022, Art. no. 113647.
Int. Conf. Cyborg Bionic Syst. (CBS), 2018, pp. 322–327.
[138] Y. Ming, H. Qu, and E. Bertini, “RuleMatrix: Visualizing and under-
[112] F. Zhuang et al., “A comprehensive survey on transfer learning,” Proc.
standing classifiers with rules,” IEEE Trans. Vis. Comput. Graphics,
IEEE, vol. 109, no. 1, pp. 43–76, Jan. 2021.
vol. 25, no. 1, pp. 342–352, Jan. 2019.
[113] I. J. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep
Learning, vol. 1. Cambridge, MA, USA: MIT Press, 2016. [139] N. Frosst and G. Hinton, “Distilling a neural network into a soft
[114] S. Raschka and V. Mirjalili, Python Machine Learning—Second decision tree,” 2017, arXiv:1711.09784.
Edition. London, U.K.: Packt, 2017. [140] Q. Huang, M. Yamada, Y. Tian, D. Singh, and Y. Chang, “Graphlime:
[115] E. Hodo, X. Bellekens, A. Hamilton, C. Tachtatzis, and R. Atkinson, Local interpretable model explanations for graph neural networks,”
“Shallow and deep networks intrusion detection system: A taxonomy IEEE Trans. Knowl. Data Eng., early access, Jun. 30, 2022,
and survey,” Jan. 2017. [Online]. Available: arXiv:1701.02145. doi: 10.1109/TKDE.2022.3187455.
[116] R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: [141] J. Rabold, H. Deininger, M. Siebers, and U. Schmid, “Enriching visual
A survey,” Jan. 2019. [Online]. Available: arXiv:1901.03407. with verbal explanations for relational concepts—Combining LIME
[117] M. Ghallab, “Responsible AI: Requirements and challenges,” AI with Aleph,” in Proc. Joint Eur. Conf. Mach. Learn. Knowl. Disc.
Perspect., vol. 1, no. 1, pp. 1–7, 2019. Databases, 2019, pp. 180–192.
[118] S. R. Islam, W. Eberle, S. K. Ghafoor, and M. Ahmed, “Explainable [142] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling lime
artificial intelligence approaches: A survey,” Jan. 2021. [Online]. and shap: Adversarial attacks on post hoc explanation methods,” in
Available: arXiv:2101.09429. Proc. AAAI/ACM Conf. AI Ethics Soc., 2020, pp. 180–186.
[119] C. Molnar. “Interpretable machine learning.” 2019. [Online]. Available: [143] M. T. Ribeiro, S. Singh, and C. Guestrin, “Anchors: High-precision
https://christophm.github.io/interpretable-ml-book/ model-agnostic explanations,” in Proc. AAAI Conf. Artif. Intell., vol. 32,
[120] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why should i trust you?’ 2018, pp. 1527–1535.
Explaining the predictions of any classifier,” in Proc. 22nd ACM [144] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32,
SIGKDD Int. Conf. Knowl. Disc. Data Min., 2016, pp. 1135–1144. 2001.
[145] A. Fisher, C. Rudin, and F. Dominici, “All models are wrong, but many [171] A. S. Rao and G. Verweij, Sizing the Prize, What’s the Real Value of
are useful: Learning a variable’s importance by studying an entire class AI for Your Business and How Can You Capitalise?. London, U.K.:
of prediction models simultaneously,” J. Mach. Learn. Res., vol. 20, PwC, 2018.
no. 177, pp. 1–81, 2019. [172] R. Pecori, “A PKI-free key agreement protocol for P2P VoIP applica-
[146] S. Lundberg and S.-I. Lee, “A unified approach to interpreting model tions,” in Proc. IEEE Int. Conf. Commun. (ICC), 2012, pp. 6748–6752.
predictions,” 2017, arXiv:1705.07874. [173] H. Tahaei, F. Afifi, A. Asemi, F. Zaki, and N. B. Anuar, “The rise
[147] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolu- of traffic classification in IoT networks: A survey,” J. Netw. Comput.
tional networks: Visualising image classification models and saliency Appl., vol. 154, Mar. 2020, Art. no. 102538.
maps,” 2013, arXiv:1312.6034. [174] V. K. Rahul, R. Vinayakumar, K. Soman, and P. Poornachandran,
[148] S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual explanations “Evaluating shallow and deep neural networks for network intrusion
without opening the black box: Automated decisions and the GDPR,” detection systems in cyber security,” in Proc. 9th Int. Conf. Comput.
Harvard J. Law Technol., vol. 31, no. 2, p. 841, 2017. Commun. Netw. Technol. (ICCCNT), Oct. 2018, pp. 1–6.
[149] J. Bien and R. Tibshirani, “Prototype selection for interpretable clas- [175] R. Vinayakumar, K. P. Soman, and P. Poornachandran, “Applying con-
sification,” Ann. Appl. Stat., vol. 5, no. 4, pp. 2403–2424, 2011. volutional neural network for network intrusion detection,” in Proc. Int.
[150] K. S. Gurumoorthy, A. Dhurandhar, G. Cecchi, and C. Aggarwal, Conf. Adv. Comput. Commun. Inf. (ICACCI), 2017, pp. 1222–1228.
“Efficient data representation by selecting prototypes with impor- [176] J. Kim, J. Kim, H. L. Thi Thu, and H. Kim, “Long short term memory
tance weights,” in Proc. IEEE Int. Conf. Data Min. (ICDM), 2019, recurrent neural network classifier for intrusion detection,” in Proc. Int.
pp. 260–269. Conf. Platform Technol. Service (PlatCon), 2016, pp. 1–5.
[151] B. Kim, R. Khanna, and O. Koyejo, “Examples are not enough, learn to [177] T. Le, J. Kim, and H. Kim, “An effective intrusion detection classifier
Criticize! Criticism for interpretability,” in Proc. 30th Int. Conf. Neural using long short-term memory with gradient descent optimization,” in
Inf. Process. Syst. (NIPS), 2016, pp. 2288–2296. Proc. Int. Conf. Platform Technol. Service (PlatCon), 2017, pp. 1–6.
[152] M. T. Ribeiro, S. Singh, and C. Guestrin, “Model-agnostic interpretabil- [178] J. Kim and H. Kim, “Applying recurrent neural network to intru-
ity of machine learning,” 2016, arXiv:1606.05386. sion detection with Hessian free optimization,” in Information Security
[153] A. Stojić et al., “Explainable extreme gradient boosting tree-based Application, H.-W. Kim and D. Choi, Eds. Cham, Switzerland: Springer
prediction of toluene, ethylbenzene and xylene wet deposition,” Sci. Int., 2016, pp. 357–369.
Total Environ., vol. 653, pp. 140–147, Feb. 2019. [179] C. Xu, J. Shen, X. Du, and F. Zhang, “An intrusion detection system
[154] M. Kłosok and M. Chlebus, Towards Better Understanding of Complex using a deep neural network with gated recurrent units,” IEEE Access,
Machine Learning Models Using Explainable Artificial Intelligence vol. 6, pp. 48697–48707, 2018.
(XAI): Case of Credit Scoring Modelling, Faculty Econ. Sci., Univ. [180] C. Yin, Y. Zhu, J. Fei, and X. He, “A deep learning approach for intru-
Warsaw, Warsaw, Poland, 2020. sion detection using recurrent neural networks,” IEEE Access, vol. 5,
[155] D. Alvarez-Melis and T. S. Jaakkola, “On the robustness of inter- pp. 21954–21961, 2017.
pretability methods,” 2018, arXiv:1806.08049. [181] A. Andalib and V. T. Vakili, “An autonomous intrusion detection system
[156] Z. Zhao, P. Xu, C. Scheidegger, and L. Ren, “Human-in-the-loop using an ensemble of advanced learners,” in Proc. 28th Iran. Conf.
extraction of interpretable concepts in deep learning models,” IEEE Elect. Eng. (ICEE), 2020, pp. 1–5.
Trans. Visualization Comput. Graph., vol. 28, no. 1, pp. 780–790, 2021. [182] N. Chaibi, B. Atmani, and M. Mokaddem, “Deep learning approaches
to intrusion detection: A new performance of ANN and RNN on NSL-
[157] L. S. Shapley, “A value for n-person games,” Contributions Theory
KDD,” in Proc. 1st Int. Conf. Intell. Syst. Pattern Recognit. (ISPR),
Games, vol. 2, no. 28, pp. 307–317, 1953.
2020, pp. 45–49.
[158] A. Ghorbani, A. Abid, and J. Zou, “Interpretation of neural networks
[183] H. Gwon, C. Lee, R. Keum, and H. Choi, “Network intrusion detection
is fragile,” in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019,
based on LSTM and feature embedding,” 2019, arxiv.abs/1911.11552.
pp. 3681–3688.
[184] B. Roy and H. Cheung, “A deep learning approach for intrusion detec-
[159] W. Samek and K.-R. Müller, “Towards explainable artificial intelli- tion in Internet of Things using bi-directional long short-term memory
gence,” in Explainable AI: Interpreting, Explaining and Visualizing recurrent neural network,” in Proc. 28th Int. Telecommun. Netw. Appl.
Deep Learning. Cham, Switzerland: Springer, 2019, pp. 5–22. Conf. (ITNAC), 2018, pp. 1–6.
[160] P. Lipton, “Contrastive explanation,” Roy. Inst. Philosophy [185] B. A. Tama and K. H. Rhee, “Attack classification analysis of IoT
Supplements, vol. 27, pp. 247–266, Dec. 1990. network via deep learning approach,” in Proc. ReBICTE, Nov. 2017,
[161] M. Burhan, R. A. Rehman, B. Khan, and B.-S. Kim, “IoT elements, p. 3.
layered architectures and security issues: A comprehensive survey,” [186] S. A. Althubiti, E. M. Jones, and K. Roy, “LSTM for anomaly-based
Sensors, vol. 18, no. 9, p. 2796, 2018. network intrusion detection,” in Proc. 28th Int. Telecommun. Netw.
[162] M. A. J. Jamali, B. Bahrami, A. Heidari, P. Allahverdizadeh, and Appl. Conf. (ITNAC), 2018, pp. 1–3.
F. Norouzi, “IoT architecture,” in Towards Internet of Things. Cham, [187] “2020 unit 42 IoT threat report.” 2020. [Online]. Available: https://
Switzerland: Springer, 2020, pp. 9–31. unit42.paloaltonetworks.com/iot-threat-report-2020/
[163] M. Mukherjee, I. Adhikary, S. Mondal, A. K. Mondal, M. Pundir, and [188] M. A. Ferrag and L. Maglaras, “DeepCoin: A novel deep learning and
V. Chowdary, “A vision of IoT: Applications, challenges, and opportu- blockchain-based energy exchange framework for smart grids,” IEEE
nities with Dehradun perspective,” in Proc. Int. Conf. Intell. Commun. Trans. Eng. Manag., vol. 67, no. 4, pp. 1285–1297, Nov. 2020.
Control Devices, 2017, pp. 553–559. [189] S. Aldhaheri, D. Alghazzawi, L. Cheng, B. Alzahrani, and
[164] J. Almutairi and M. Aldossary, “A novel approach for IoT tasks offload- A. Al-Barakati, “Deepdca: Novel network-based detection of iot
ing in edge-cloud environments,” J. Cloud Comput., vol. 10, no. 1, attacks using artificial immune system,” Appl. Sci., vol. 10, no. 6,
pp. 1–19, 2021. p. 1909, 2020.
[165] Y.-D. Lin, “Editorial: Fourth quarter 2020 IEEE communications sur- [190] Z. K. Maseer, R. Yusof, N. Bahaman, S. A. Mostafa, and
veys and tutorials,” IEEE Commun. Surveys Tuts., vol. 22, no. 4, C. F. M. Foozy, “Benchmarking of machine learning for anomaly based
pp. 2130–2135, 4th Quart., 2020. intrusion detection systems in the CICIDS2017 dataset,” IEEE Access,
[166] A. Yousefpour et al., “All one needs to know about fog computing vol. 9, pp. 22351–22370, 2021.
and related edge computing paradigms: A complete survey,” J. Syst. [191] Y. N. Soe, P. I. Santosa, and R. Hartanto, “DDoS attack detection based
Architect., vol. 98, pp. 289–330, Sep. 2019. on simple ANN with SMOTE for IoT environment,” in Proc. 4th Int.
[167] Fog Computing and the Internet of Things: Extend the Cloud to Where Conf. Inf. Comput. (ICIC), 2019, pp. 1–5.
the Things Are, Cisco, San Jose, CA, USA, 2016. [192] A. Derhab, A. Aldweesh, A. Z. Emam, and F. A. Khan, “Intrusion
[168] S. Venticinque and A. Amato, “A methodology for deployment of IoT detection system for Internet of Things based on temporal convolution
application in fog,” J. Ambient Intell. Humanized Comput., vol. 10, neural network and efficient feature engineering,” Wireless Commun.
no. 5, pp. 1955–1976, 2019. Mobile Comput., vol. 2020, Dec. 2020, Art. no. 6689134.
[169] P. Kumar, G. P. Gupta, and R. Tripathi, “A distributed ensemble design [193] M. Ge, X. Fu, N. Syed, Z. Baig, G. Teo, and A. Robles-Kelly,
based intrusion detection system using fog computing to protect the “Deep learning-based intrusion detection for IoT networks,” in Proc.
Internet of Things networks,” J. Ambient Intell. Humanized Comput., IEEE 24th Pac. Rim Int. Symp. Depend. Comput. (PRDC), 2019,
vol. 12, pp. 9555–9572, Nov. 2020. pp. 256–25609.
[170] D. C. Nguyen et al., “Enabling AI in future wireless networks: A data [194] N. Moustafa, “A new distributed architecture for evaluating AI-based
life cycle perspective,” IEEE Commun. Surveys Tuts., vol. 23, no. 1, security systems at the edge: Network TON_IoT datasets,” Sustain.
pp. 553–595, 1st Quart., 2021. Cities Soc., vol. 72, Sep. 2021, Art. no. 102994.
[195] S. K. Zhou et al., “A review of deep learning in medical imaging: Nour Moustafa (Senior Member, IEEE) received
Imaging traits, technology trends, case studies with progress high- the bachelor’s and master’s degrees in computer sci-
lights, and future promises,” Proc. IEEE, vol. 109, no. 5, pp. 820–838, ence from the Faculty of Computer and Information,
May 2021. Helwan University, Egypt, in 2009 and 2014, respec-
[196] V. Meshram, K. Patil, V. Meshram, D. Hanchate, and S. Ramkteke, tively, and the Ph.D. degree in cyber security
“Machine learning in agriculture domain: A state-of-art survey,” Artif. from the University of New South Wales (UNSW)
Intell. Life Sci., vol. 1, Feb. 2021, Art. no. 100010. at Canberra, Canberra, Australia, in 2017. He is
[197] S. M. Ammal, M. Kathiresh, and R. Neelaveni, “Artificial intelligence a Leader of Intelligent Security Group, a Senior
and sensor technology in the automotive industry: An overview,” in Lecturer, and an ARC DECRA Fellow with UNSW
Automotive Embedded Systems. Cham, Switzerland: Springer, 2021, Canberra, where he was a Postdoctoral Fellow from
pp. 145–164. June 2017 till December 2018. His areas of interest
[198] S. Amershi, E. Kamar, K. Lauter, J. W. Vaughan, and H. Wallach. include cyber security, in particular, network security, IoT security, intru-
“Research collection: Research supporting Responsible Ai.” Apr. 2020. sion detection systems, statistics, deep learning, and machine learning
[Online]. Available: https://www.microsoft.com/en-us/research/blog/ techniques. He has been awarded the 2020 Prestigious Australian Spitfire
research-collection-research-supporting-responsible-ai/ Memorial Defence Fellowship Award. He has served his academic commu-
[199] “Responsible AI practices.” 2021. [Online]. Available: https://ai.google/ nity as an Guest Associate Editor of IEEE Transactions journals, including
responsibilities/responsible-ai-practices/ IEEE T RANSACTIONS ON I NDUSTRIAL I NFORMATICS, IEEE I NTERNET
[200] Y. Wang, M. Xiong, and H. Olya, “Toward an understanding of respon- OF T HINGS J OURNAL , as well as the journals of IEEE ACCESS , Future
sible artificial intelligence practices,” in Proc. 53rd Hawaii Int. Conf. Internet, and Information Security Journal: A Global Perspective. He has
Syst. Sci., 2020, pp. 4962–4971. also served over seven conferences in leadership roles, involving the Vice-
[201] S. Guo, J. Yu, X. Liu, C. Wang, and Q. Jiang, “A predicting model Chair, the Session Chair, a Technical Program Committee Member, and the
for properties of steel using the industrial big data based on machine Proceedings Chair, including the IEEE TrustCom from 2020 to 2021 and the
learning,” Comput. Mater. Sci., vol. 160, pp. 95–104, Apr. 2019. 33rd Australasian Joint Conference on Artificial Intelligence in 2020. He is
[202] M. Moradi and M. Samwald, “Post-hoc explanation of black-box clas- also an ACM Distinguished Speaker and a Spitfire Fellow.
sifiers using confident itemsets,” Exp. Syst. Appl., vol. 165, Mar. 2021,
Art. no. 113941.
[203] G. Karolina Dziugaite, S. Ben-David, and D. M. Roy, “Enforcing inter-
pretability and its statistical impacts: Trade-offs between accuracy and
interpretability,” 2020, arXiv:2010.13764.
[204] C. Xu, J. Shen, and X. Du, “A method of few-shot network intru-
sion detection based on meta-learning framework,” IEEE Trans. Inf.
Forensics Security, vol. 15, pp. 3540–3552, 2020.
[205] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, “Machine learning
interpretability: A survey on methods and metrics,” Electronics, vol. 8,
Nickolaos Koroniotis received the bachelor’s degree
no. 8, p. 832, 2019.
in informatics and telematics in 2014, the mas-
[206] L. Xiao, X. Wan, X. Lu, Y. Zhang, and D. Wu, “IoT security techniques
ter’s degree in web engineering and applications
based on machine learning: How do IoT devices use AI to enhance
in 2016, and the Ph.D. degree in cyber security
security?” IEEE Signal Process. Mag., vol. 35, no. 5, pp. 41–49,
from the University of New South Wales (UNSW)
Sep. 2018.
at Canberra with a particular interest in Network
[207] J. Shin, S.-H. Choi, P. Liu, and Y.-H. Choi, “Unsupervised multi-stage
Forensics and the IoT in June 2020. He is a Research
attack detection framework without details on single-stage attacks,”
Associate with the School of Engineering and
Future Gener. Comput. Syst., vol. 100, pp. 811–825, Nov. 2019.
Information Technology, UNSW Canberra, ADFA.
[208] T. Shawly, M. Khayat, A. Elghariani, and A. Ghafoor, “Evaluation
He has been working as a Research Associate in
of HMM-based network intrusion detection system for multiple multi-
projects relating to AI-enabled vulnerability and
stage attacks,” IEEE Netw., vol. 34, no. 3, pp. 240–248, May/Jun. 2020.
attack detection in IoT environments since 2020. His interests lie in the fields
[209] J. Zhou, A. H. Gandomi, F. Chen, and A. Holzinger, “Evaluating the
of cyber security, artificial intelligence, and the Internet of Things.
quality of machine learning explanations: A survey on methods and
metrics,” Electronics, vol. 10, no. 5, p. 593, 2021.
[210] C. Molnar, G. Casalicchio, and B. Bischl, “Interpretable machine
learning—A brief history, state-of-the-art and challenges,” in Proc.
Joint Eur. Conf. Mach. Learn. Knowl. Disc. Databases, 2020,
pp. 417–431.
[211] M. L. Head, L. Holman, R. Lanfear, A. T. Kahn, and M. D. Jennions,
“The extent and consequences of P-hacking in science,” PLOS Biol.,
vol. 13, no. 3, pp. 1–15, Mar. 2015.
[212] I. Kim, S. Rajaraman, and S. Antani, “Visual interpretation of con-
volutional neural network predictions in classifying medical image Marwa Keshk received the bachelor’s degree in
modalities,” Diagnostics, vol. 9, no. 2, p. 38, 2019. computer science from the Faculty of Computer
[213] Q. Zhang, Y. Yang, H. Ma, and Y. N. Wu, “Interpreting cnns via deci- and Information, Helwan University, Egypt, in
sion trees,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2012, the master’s degree from the UNSW of
2019, pp. 6261–6270. Canberra in 2016, and the Ph.D. degree from the
[214] T. G. Dietterich and A. Guyer, “The familiarity hypothesis: Explaining School of Engineering and Information Technology
the behavior of deep open set methods,” Pattern Recognit., vol. 132, (Australian Centre for Cyber Security), UNSW of
Dec. 2022, Art. no. 108931. Canberra. She is currently working as a Lecturer
[215] S. M. Mathews, “Explainable artificial intelligence applications in NLP, of Cyber Security with UNSW Canberra, City
biomedical, and Malware classification: A literature review,” in Adv. Campus. Her areas of interest include cyber secu-
Intell. Syst. Comput., vol. 998. Heielberg. Germany: Springer-Verlag, rity, privacy preservation, evolutionary computation,
Jul. 2019, pp. 1269–1292. artificial intelligence techniques, and statistical methods.
Albert Y. Zomaya (Fellow, IEEE) is a Peter Nicol Zahir Tari is a Full Professor with RMIT
Russell Chair Professor of Computer Science and University, Australia, and the Research Director of
the Director of the Centre for Distributed and RMIT Cyber Security Research and Innovation. His
High-Performance Computing with the University of expertise is in the areas of system performance (e.g.,
Sydney. Up to date, he has published more than 700 P2P, cloud, and edge/the IoT) as well as system secu-
scientific papers and articles and is (co)author/editor rity (e.g., SCADA, smart grid, cloud, and edge/the
of more than 30 books. His research interests lie in IoT).
parallel and distributed computing, networking, and
complex systems.
Prof. Zomaya’s recent awards include the
Research Innovation Award, the IEEE Technical
Committee on Cloud Computing in 2021, the Technical Achievement and
Recognition Award, and the IEEE Communications Society’s IoT, Ad Hoc and
Sensor Networks Technical Committee in 2022. He is a Clarivate 2022 Highly
Cited Researcher. A sought-after speaker, he has delivered >250 keynote
addresses, invited seminars, and media briefings. He is the past Editor-in-
Chief of the IEEE T RANSACTIONS ON C OMPUTERS from 2010 to 2014
and the IEEE T RANSACTIONS ON S USTAINABLE C OMPUTING from 2016
to 2020. He is a Decorated Scholar with numerous accolades, including
Fellowship of the American Association for the Advancement of Science
and the Institution of Engineering and Technology. He is also a Fellow of
the Australian Academy of Science and Royal Society of New South Wales,
a Foreign Member of Academia Europaea, and a member of the European
Academy of Sciences and Arts.

Explainable Intrusion Detection For Cyber Defences in The Internet of Things Opportunities and Solutions

Uploaded by

Copyright:

Available Formats

You might also like

Explainable Intrusion Detection For Cyber Defences in The Internet of Things Opportunities and Solutions

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Explainable Intrusion Detection For Cyber Defences in The Internet of Things Opportunities and Solutions

Uploaded by

Copyright:

Available Formats

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 25, NO.

3, THIRD QUARTER 2023 1775

Explainable Intrusion Detection for Cyber

Abstract—The field of Explainable Artificial Intelligence (XAI) I. I NTRODUCTION

Fig. 2. Methods of IDSs and their environments.

Fig. 3. XAI models for assisting various audience profiles.

Fig. 4. Taxonomy of XAI methods for cyber defences.

C. Explainability Models explanations in different formats. Therefore, the best tech-

Fig. 5. Proposed architecture of Internet of Things (IoT) for cyber defences.

D. Summary and Insights IX. C ONTEMPORARY DL-BASED IDS

You might also like