A REPORT -

“Explainable Artificial Intelligence for Cybersecurity”

by Deepak Kumar Sharma, Jahanavi Mishra, Aeshit Singh,
Raghav Govil,Gautam Srivastava, Jerry Chun-Wei Lin

Submitted to: Prof. Dr. Tarek R. Besold (Eindhoven University of Technology,

Netherlands and Sony AI Lab, Barcelona)

Submitted by: Syed Fahad Hassan (Masters in CS) – ID# 2044060

email: syedfahad.hassan@studenti.unipd.it

Submission purpose: OTHER

Department of Mathematics, University of Padua

Course: ADVANCED TOPICS IN COMPUTER SCIENCE
 "Explainable Artificial Intelligence for Cybersecurity"

1. Overview of the paper:

Introduction:

The research article titled "Explainable Artificial Intelligence for Cybersecurity" explores the application of
explainable artificial intelligence (XAI) techniques in the context of cybersecurity. XAI aims to make AI
solutions more transparent and interpretable, enabling humans to understand the decisions and actions taken by
AI models. The paper highlights the importance of XAI in the domain of cybersecurity and presents a
methodology that utilizes an adversarial approach to explain misclassifications by data-driven AI models in IoT
applications. The research also focuses on developing black-box attacks using XAI techniques to evaluate the
security properties of the methods used.

Motivation:

The paper is motivated by the need to address certain challenges in the field of cybersecurity, specifically related
to the misclassification of samples in Data-driven Intrusion Detection Systems (IDS). When faced with a lack of
data, the IDS could erroneously classify an attack as normal, leading to a gap in the system's performance.
Understanding the reasons behind such misclassifications is crucial for debugging and diagnosing the system. The
paper aims to provide understandable explanations for misclassifications, enabling the identification of necessary
observations to prevent future attacks. This forms the primary goal of the research, which is to correctly classify
misclassified samples using an adversarial approach.

The motivation is further driven by the lack of research in the cybersecurity domain that analyzes the security
resilience of explainable techniques using realistic threat models. While there have been efforts to improve
explainability in machine learning models and generate feature-rich explanations, little research has been
conducted to evaluate the security properties of explainable techniques in the context of cybersecurity. Therefore,
the paper undertakes a security evaluation of Explainable AI (XAI) techniques in opportunistic networks, utilizing
the "Six Ws" of Explainable Security (Who? What? Where? When? Why? and How?) framework. The goal is to
assess the security characteristics of XAI techniques and analyze how an attacker can mislead target classifiers
and explainable methods using an explainable model to achieve similar classifier outputs.

The paper acknowledges the existing focus in the literature on various domains, such as producing adversarial
examples for supervised machine learning and exploring defenses against attacks. However, these areas have not
been adequately addressed in the context of the anomaly detection task in a black-box context. Therefore,
investigating strategies for black-box attacks that minimize the search effort to find adversarial examples with a
high likelihood becomes crucial. The proposed methodology aims to provide insightful and logical explanations
for sample misclassifications, aligning with expert knowledge to enhance the understanding of the system's
behavior.

Technical Contributions:

The paper emphasizes three key characteristics of XAI algorithms: explainability, transparency, and
interpretability. It distinguishes between black-box and white-box ML algorithms, with white-box systems
providing interpretable results that can be easily understood by professionals. The proposed methodology
leverages an adversarial approach to detect misclassifications in data-driven intrusion detection systems and
compute the minimum number of changes required to correctly classify misclassified data. The authors use visual
diagrams to illustrate the results, facilitating interpretation and understanding.
 "Explainable Artificial Intelligence for Cybersecurity"

It introduces a data-driven classifier that approximates the likelihood of a given sample belonging to a particular
class. The classifier is trained using supervised learning, and the paper focuses on explaining misclassifications
made by the model. An adversarial approach is employed to modify misclassified samples, finding the minimum
changes necessary to achieve correct classification. The differences between the modified samples and the original
misclassified samples provide explanations for the incorrect outputs, highlighting the key features that lead to
misclassifications.

Additionally, the paper explores the design of black-box attacks using XAI techniques. Adversarial samples are
created to alter the results of the model, aiming to mislead both the classifier and the explanation report without
affecting the classifier's results. The proposed model evaluates the stability, accuracy, and confidence in the
security properties of XAI methods by testing the effectiveness of these attacks on security-related datasets.

View and Relevance:

The research article addresses the growing need for XAI techniques in the field of cybersecurity. With the
increasing complexity and potential vulnerabilities of AI systems, it is crucial to understand the decisions made
by these systems to ensure their trustworthiness and effectiveness. The paper presents a practical methodology
that can be applied to detect misclassifications and explain the outputs of data-driven AI models in IoT
applications.

The use of an adversarial approach provides insights into the decision boundaries of the trained model, enabling
the identification of key features that contribute to misclassifications. This can help in refining the model,
enhancing its performance, and gaining a better understanding of its limitations. Moreover, the development of
black-box attacks using XAI techniques contributes to evaluating the security properties of XAI methods, ensuring
their robustness against adversarial attacks.

The research article's findings are relevant not only in the domain of cybersecurity but also in other fields where
interpretability and explainability are crucial, such as medicine, finance, law, and defense. The proposed
methodology can enhance user trust and confidence in AI systems by providing explanations for their decisions,
facilitating human-AI collaboration, and enabling the detection of vulnerabilities or biases.

Proposed Methodology:

The research article introduces a methodology for explaining misclassifications by data-driven AI models in IoT
applications. The approach involves an adversarial approach to detect misclassifications and determine the
minimum number of changes required to correctly classify misclassified data. Additionally, the article explores
the development of black-box attacks to evaluate the stability, accuracy, and trustworthiness of XAI methods in
the cybersecurity domain. The proposed model (shown in figure 1) is evaluated using benchmark datasets and
algorithms such as multilayer perceptron, k-nearest neighbors, support vector machine, and random forest.
 "Explainable Artificial Intelligence for Cybersecurity"

Evaluation and Results:

The proposed methodology is evaluated using the NSL-KDD99 benchmark and PDF datasets with various ML
algorithms such as multilayer perceptron (MLP), k-nearest neighbors (KNN), support vector machine (SVM), and
Random Forest (RF). Visual diagrams are provided to facilitate the interpretation of the results. The effectiveness
of the adversarial approach is demonstrated by identifying the minimal changes required to correctly classify
misclassified data. The black-box attack shows the vulnerabilities and potential risks associated with XAI
techniques in cybersecurity.

2. Trustworthy AI context

Overview of XAI Techniques:

As mentioned, XAI algorithms are characterized by three key attributes: explainability, transparency, and
interpretability. Explainability refers to the ability to provide explanations for the decisions made by AI models.
Transparency focuses on the extent to which the processes involved in model training and decision-making can
be defined and understood. Interpretability pertains to the ability to interpret the underlying machine learning
model itself. These characteristics enable XAI to confirm existing knowledge, challenge information, and generate
new assumptions.

Importance of XAI in Cybersecurity:

XAI holds significant importance in cybersecurity, particularly in domains such as medicine, finance, law, defense,
and IoT applications. In these fields, it is essential to interpret AI model results and establish confidence in their
accuracy and reliability. The Department of Defense (DoD) recognizes the need for explainable AI to ensure war
fighters can understand and trust the decisions made by intelligent systems. XAI techniques provide the necessary
explanations and insights for cybersecurity practitioners to detect vulnerabilities, assess robustness, and analyze
the impact of attacks on AI models.

Explainable Interface and Explanations of Incorrect Output:

The proposed methodology employs an explainable interface to provide insights into misclassifications. It utilizes
an adversarial technique to modify misclassified samples and determine the minimum changes needed for correct
classification. The differences between the original misclassified samples and the modified samples highlight the
key features that contribute to misclassifications. Explanations are generated by computing the minimum changes
required to correctly classify samples and visualizing the deviations between the original and modified samples.

Designing Attacks on Consistency, Correctness, and Confidence Properties:

The research article also discusses designing attacks to evaluate the stability and trustworthiness of XAI methods.
The attacks aim to disrupt either the classifier or the interpreter, affecting trust in the underlying AI system. The
article presents methodologies for I-attacks (affecting the interpreter) and CI-attacks (affecting both the classifier
 "Explainable Artificial Intelligence for Cybersecurity"

and interpreter) to manipulate explanations while keeping the classifier's output almost constant. These attacks
help assess the vulnerabilities and robustness of XAI methods in cybersecurity applications.

Trade-offs and Trustworthiness:

When considering the wider context of trustworthy AI, the presented technique has several trade-offs that need to
be acknowledged. One trade-off is between accuracy and interpretability. While black-box ML algorithms may
achieve high accuracy, they lack transparency and interpretability, making it challenging to understand their
decision-making process. The proposed XAI approach prioritizes interpretability, allowing humans to
comprehend and verify the decisions made by the AI model, even if it may come at a slight cost to accuracy.

Another trade-off is between security and interpretability. Adversarial attacks, although useful for evaluating the
trustworthiness of XAI methods, raise ethical concerns when used maliciously. Striking a balance between these
dimensions is crucial to ensure that XAI techniques are both effective and trustworthy.

In terms of trustworthiness, the paper's methodology contributes positively to the dimensions of explainability,
transparency, and interpretability. By providing explanations for misclassifications, the methodology enhances the
explainability of AI models. The visual diagrams further improve transparency, allowing users to gain insights
into the decision-making process. Moreover, the adversarial approach helps identify vulnerabilities and assess the
robustness of the AI system, contributing to its reliability and trustworthiness.

3. Personal evaluation (advantages/disadvantages, etc.)

Personal Evaluation:
As mentioned, the research article presents a methodology that utilizes XAI techniques to address
misclassifications in data-driven AI models and develop black-box attacks in IoT applications. The article
discusses the concept of XAI, the importance of explainability in AI models, and the use of adversarial approaches
to detect misclassifications and explain the underlying features leading to those misclassifications.

One of the advantages of the article is its focus on the application of XAI in the cybersecurity domain.
Cybersecurity is a critical field where the transparency and interpretability of AI models are crucial. The article
recognizes the need for understanding and trusting AI systems in this context, particularly in detecting
vulnerabilities and analyzing robustness to noise. By explaining misclassifications and identifying key features,
XAI can provide valuable insights into the decision-making process of AI models, enabling better analysis and
refinement.

The article also highlights the use of adversarial samples to reveal blind spots and vulnerabilities in ML algorithms.
Adversarial samples are designed to alter the results of the model and can be used both by attackers to evade
detection and by defenders to assess the robustness of the system. By leveraging adversarial approaches, the
proposed methodology aims to determine the minimal changes required to correctly classify misclassified data,
providing explanations for the incorrect outputs of the classifier.

Furthermore, the article discusses the importance of explainability in the context of black-box attacks. It
emphasizes the need to evaluate the stability, accuracy, and confidence in the security properties of XAI methods.
By developing black-box attacks, the researchers demonstrate the potential vulnerabilities in XAI techniques and
the impact on both the classifier and the explanation report. This evaluation helps in identifying potential
weaknesses and improving the security of AI systems.

However, there are a few limitations in the article that should be considered. First, while the article provides an
overview of XAI and its importance, it lacks a comprehensive literature review. Including a more extensive review
 "Explainable Artificial Intelligence for Cybersecurity"

of existing research and methodologies in XAI for cybersecurity would have strengthened the article's foundation
and provided more context for the proposed methodology.

Second, the article focuses on specific ML algorithms (MLP, KNN, SVM, and Random Forest) and uses the NSL-
KDD99 benchmark and PDF datasets for evaluation. While these choices may be suitable for the purpose of the
research, a broader analysis involving different types of datasets and algorithms would have enhanced the
generalizability of the proposed methodology.

Lastly, the article briefly mentions the development of black-box attacks but does not provide detailed insights
into the attack techniques and their potential implications. Expanding on the attack methodologies and discussing
their impact on the security properties of XAI methods would have added depth to the research.

Conclusion:

In conclusion, the research article on "Explainable Artificial Intelligence for Cybersecurity" presents a valuable
contribution to the field of XAI and its application in cybersecurity. The methodology introduced in the paper
offers a practical approach to explaining misclassifications and evaluating the trustworthiness of AI models in IoT
applications. The research highlights the importance of XAI techniques in gaining user trust and confidence,
particularly in critical domains such as medicine, finance, law, and defense.

The proposed methodology addresses the dimensions of explainability, transparency, and interpretability in
trustworthy AI. By providing explanations for incorrect outputs, the methodology enhances the explainability of
AI models. The visual diagrams improve transparency, allowing users to understand the decision-making process.
The adversarial approach aids in identifying vulnerabilities and assessing the robustness of the AI system, thereby
contributing to its reliability and trustworthiness.

However, it is essential to consider the trade-offs involved, such as the potential sacrifice of accuracy or the ethical
concerns raised by black-box attacks. Achieving a balance between trustworthiness and performance is crucial in
the broader context of trustworthy AI. Further research and exploration of XAI techniques in the field of
cybersecurity are encouraged to advance the understanding and application of trustworthy AI principles.