DRAFT
A Review on Explainable Artificial Intelligence
for Healthcare: Why, How, and When?
Subrato Bharati, M. Rubaiyat Hossain Mondal, Member, IEEE, and Prajoy Podder

Abstract—Artificial intelligence (AI) models are increasingly
finding applications in the field of medicine. Concerns have been
raised about the explainability of the decisions that are made by
these AI models. In this article, we give a systematic analysis of
explainable artificial intelligence (XAI), with a primary focus on
models that are currently being used in the field of healthcare. The
literature search is conducted following the preferred reporting
items for systematic reviews and meta-analyses (PRISMA)
standards for relevant work published from 1 January 2012 to 02
February 2022. The review analyzes the prevailing trends in XAI
and lays out the major directions in which research is headed. We
investigate the why, how, and when of the uses of these XAI models
and their implications. We present a comprehensive examination
of XAI methodologies as well as an explanation of how a
trustworthy AI can be derived from describing AI models for
healthcare fields. The discussion of this work will contribute to the
formalization of the XAI field.
Impact Statement — XAI is emerging in various sectors and the
literature has recognized numerous concerns and problems.
Security, performance, vocabulary, evaluation of explanation, and
generalization of XAI in the field of healthcare are the most
significant problems recognized and discussed in the literature.
The paper has identified a number of obstacles associated with
XAI in the healthcare area, including System Evaluation,
Organizational, Legal, socio-relational, and Communication
issues. Although a few studies in the literature have proposed some
answers for the stated problems, others have characterized it as a
future research gap that must be filled. In healthcare, the
organizational difficulty posed by XAI can be mitigated via AI
deployment management plans. The doctor's understanding of
how patients would interpret the system and double-checking
health facts with patients might help alleviate communication
problems. The socio-organizational issue or difficulty can be
overcome through patient instruction that facilitates the use of
artificial intelligence. The applications of AI in healthcare,
including diagnosis and prognosis, medication discovery,
population health, healthcare organization, and patient-facing
apps, have been reviewed critically. This article addresses a
number of AI's limitations, including the difficulties of
authenticating AI's output. Active collaboration between
physicians, theoretical researchers, medical imaging specialists,
and medical image analysis specialists will be crucial for the future
development of machine learning and deep learning techniques.
This paper contributes to the body of knowledge in identifying
different techniques for XAI, in determining the issues and
difficulties associated with XAI, and finally in reviewing XAI in
healthcare.
Subrato Bharati was with the Institute of Information and Communication
Technology, Bangladesh University of Engineering and Technology, Dhaka-
1205, Bangladesh, and now with the Department of Electrical and Computer
Engineering, Concordia University, Montreal, QC, Canada. (e-mail:
subratobharati1@gmail.com) (Corresponding Author).
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current
or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective
works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Index Terms— Explainable Artificial Intelligence, healthcare,
deep learning, medical care, medicine, medical imaging
I. INTRODUCTION
A dvances in Artificial Intelligence (AI) have taken place in
recent years, with the technology becoming more
complicated, more autonomous, and more sophisticated.
As data grows exponentially, so does the speed at which
computer power advances. This combination gives a massive
boost to the development of AI. People's lives have already
been impacted by AI's practical accomplishments in a range of
fields (e.g., voice recognition, self-driving cars, and
recommendation systems) [1-4]. AI can improve people's well-
being and health by increasing clinicians’ diagnostic work,
highlighting prevention options, and delivering tailored
treatment suggestions on electronic health records (EHRs).
Despite the employment of several useful technologies in health
care [5, 6], AI is not widely deployed [7, 8]. As a result, AI,
machine learning (ML) and deep learning (DL) algorithms
continue to be mysterious in many cases [9-11]. Decision-
making processes are generally hard to explain. Moral and
practical challenges arise. Without understanding AI diagnoses,
it is impossible to tell if diagnostic disparities between patients
reflect diagnostically important variations or bias, diagnostic
errors, or under/over-diagnosis.
A number of governmental and commercial organizations
have recently released recommendations and guidelines for the
appropriate use of AI in response to these issues. The study,
however, reveals a considerable discrepancy in the alleged
transparency characteristics. Many critical difficulties in
medicine may be resolved with the use of AI. Numerous cases
have gained pace in recent years as a result of significant
research on automated prognosis, diagnosis, testing, and
medicine formulation [12-14]. Along with genetic data,
biosensors, and medical imaging, these sources create a large
quantity of data [15-17]. A critical objective of AI in medicine
is to personalize medical decisions, health practices, and
medications for unique people. Nevertheless, the current state
of AI in medicine has been described as "strong on promise but
rather lacking in evidence and demonstration". Real-world
clinical settings have been tested to detect wrist fractures,
M. Rubaiyat Hossain Mondal is with the Institute of Information and
Communication Technology, Bangladesh University of Engineering and
Technology, Dhaka-1205, Bangladesh (e-mail: rubaiyat97@iict.buet.ac.bd).
Prajoy Podder is with the Institute of Information and Communication
Technology, Bangladesh University of Engineering and Technology, Dhaka-
1205, Bangladesh (e-mail: prajoypodder@gmail.com).

diabetic retinopathy, cancer metastases, histologic breast,
congenital cataracts, and colonic polyps; however, many of the
systems that have been demonstrated to be equivalent to or
superior to experts in experimental settings have demonstrated
high false-positive rates in real-world clinical settings [18].
Additionally, prejudice, security, privacy, and a deficit of
transparency exist, as do trust, fairness, informativeness,
transferability, and causality [19]. Numerous critical difficulties
in medicine may be resolved with the use of AI [3, 20-25].
Together with medical imaging, biosensors, genetic data, and
electronic medical records, these sources create a large quantity
of data [18]. To make a precise diagnosis in precision medicine,
doctors need substantially more information than a simple
binary prediction [19, 27]. ML in medicine has grown more
concerned with explainability, interpretability, and openness
[19]. There is sufficient evidence that ML-based methods are
helpful; however, their widespread adoption is unlikely until
these issues are solved, most likely by providing systems with
adequate explanations for their judgments [28]. Because of this,
it is difficult to develop generalizable solutions that can be used
in a variety of contexts. For example, various applications often
have distinct interpretability and explainability demands [19,
26-28].
Explainable AI (XAI) and interpretable AI (IAI) are not the
same concepts. Although some studies used the terms XAI and
IAI interchangeably, the difference and relations between these
two are described in the literature in multiple ways. This is
presented in the following. XAI outlines why the choice was
made but not how the decision was reached. The term IAI
outlines how the choice was made but not why the criteria used
were reasonable [171]. Explainability means that a ML model
and its results can be explained in a way that a person can
understand. XAI permits the explanation of learning models
and focuses on why the system made a given decision,
analyzing its logical paradigms. On the other hand, the
interpretability of ML enables users to comprehend the results
of the learning models by revealing the rationale for its
decisions [171, 172]. According to some literature [173],
explainability is the concept that a ML model and its findings
may be communicated to a person in words that make sense. In
one work, Adadi and Berrada et al. [173] described that
interpretable systems are explainable if their processes can be
comprehended by humans, demonstrating that the idea of
explainability is closely related to that of interpretability.
Nonetheless, Gilpin et al. The authors of [174] noted that
interpretability and fidelity are both essential components of
explainability. They stated that a good explanation must be
human-comprehensible (interpretability) and accurately
characterize model behavior over the whole feature space
(fidelity). Interpretability is necessary to handle the social
interaction of explainability, whereas fidelity is necessary to
assist in confirming other model prerequisites or discovering
new explanations. In other words, the fidelity of an explanation
relates to how reliably and exactly the behavior of a model is
explained. So, an explanation is explainable if it is easily
understood by humans and accurately explains the model.
Bharati et al.
Figure 1 depicts the literature search conducted for this
article in accordance with the PRISMA (preferred reporting
items for systematic reviews and meta-analyses) standards.
PRISMA guidelines and the expanded version of PRISMA
scoping reviews were used to create this scoping review (PRIS-
MA-ScR). The number of records considered, records included,
and records deleted for further evaluation are shown in a flow
diagram in Figure 1. After conducting a thorough literature
review, all relevant research publications were found. The dates
1 January 2012 to 02 February 2022 were used as the unit of
measurement. Records in English were considered, while those
in other languages were eliminated. Records were examined to
see whether they met the requirements of the PICO (patient,
intervention, comparator, and outcome) study. Initial attention
was given to studies that met all four of the following criteria:
Using XAI for screening or diagnosing patients, as well as
comparing it to conventional approaches, was shown to be
effective. Two Boolean operators, OR and AND, were used to
find the most important terms. Many terms were searched for,
including XAI, explainable AI, interpretable AI, medical
medicine, medical, etc. In addition, the publications' titles,
abstracts, and keywords were all taken into account while doing
the search. No papers on XAI fundamental scientific,
epidemiology, or clinical aspects were reviewed. It was then
decided to eliminate any content that covered the same topic
twice. Extracting information and synthesizing data elements
such as research objectives, and performance outcomes were
the next step in the process.
Elsevier (ScienceDirect), IEEE, MDPI, SpringerNature,
Bentham Science, Wiley, Medrxiv, Arxiv and many more were
among the reputable publishers and preprint servers that yielded
a total of 17,300 publications on XAI for medical care or
medicine purposes. The Appendix contains the search syntax
used in this article. Out of a total of 17,300 documents, 16,850
were manually excluded because they did not correspond to our
primary research area. Therefore, the healthcare or medicine or
medical treatment application submitted by XAI did not match
the necessary criteria. In total, 450 papers were evaluated;
however, only 195 were chosen for inclusion in the research and
the report. From the 195 papers assessed, 110 were chosen for
inclusion in the summary of the findings as these reported the
performance measurements and the performance outcomes of
XAI for healthcare or medicine. To put it another way, any of
the following problems have been covered by any of these 110
publications. Our classification of XAI in healthcare and
medicine into five categories synthesized by the method of the
authors of [135], which includes (1) dimension reduction using
XAI, (2) feature selection using XAI, (3) attention mechanism
using XAI, (4) knowledge distillation using XAI, and (5)
surrogate representations using XAI, is presented in this review.
Although some recent studies have reviewed XAI, ongoing
surveys and studies are necessary because the field of XAI in
healthcare is always changing. With this in consideration, this
review paper extends the findings reported by previous survey
papers in the field of XAI [19, 26, 73, 80, 81, 87, 195].
A Review on Explainable Artificial Intelligence for Healthcare
Fig. 1. PRISMA guideline for manuscript selection for this review
The research mainly focuses on the emerging field of XAI
and its challenges in health care. This paper also reports some
IAI models, as some literature uses the terms XAI and IAI
interchangeably. The main contributions of this paper is in
presenting the answers to the following questions:
 Why use XAI in health care and medicine, and what
problems cannot be solved with standard AI but can only
be solved using XAI?
 What are the best ways to accomplish our XAI?
 When is the best time to perform XAI?
The remaining sections are organized as follows. Background
information is provided in Section II of this review work. In
Section III, we go through the benefits of utilizing XAI as well
as the appropriate times to do so. Section IV presents the XAI
methods describing how it is used. In Section V, several
different XAI application situations are broken down and
examined. The overview of various XAI aspects and the most
recent research developments in XAI are covered in the next
part, Section VI. In Section VII, the final thoughts are
presented.
II. BACKGROUND
By mimicking human mental processes, AI presents a
paradigm shift in healthcare data-driven by increasing data
availability in healthcare as well as fast developments in
mathematical methodologies. This article examines healthcare
AI applications that are currently being developed. General AI
techniques for healthcare are discussed in the literature [29],
[30], [31], while medical image analysis is described in [32-34].
The consequences of AI have been disputed in the medical
3
literature [35-37]. An AI-powered device [38] may assist in
clinical decision-making by delivering recent medical
information from conferences, manuals, and journals. AI
systems may also assist in eliminating therapeutic and medical
errors that are inescapable (such as those that are more
repeatable and objective) in human clinical practice [38-40]. An
AI system may also use data from a large number of patients to
provide real-time insights for outcome estimation and health
risk warning. For decades, rule-based algorithms have made
great strides and have been used to interpret ECGs, detect
disorders, choose the best treatment option for each patient, and
support clinicians in establishing diagnostic theories and
hypotheses in difficult instances of patients. Because they need
the explicit specification of necessitating human-authored
updates, decision rules and rule-based systems are both time
and cost-intensive to construct. Thus, the effectiveness of the
structures is limited by the extent to which past medical
knowledge encompasses higher-order relationships between
material published in different specialties [20, 41]. Likewise, it
was challenging to include a strategy that combines
probabilistic and deterministic reasoning methods to minimize
the suitable recommended therapy, prioritize medical theories,
and psychological environment [42-43]. In contrast to the 1st
generation of AI programs, which depended only on medical
information curation by specialists as well as the construction
of state-of-the-art AI research, strict decision rules have utilized
ML methodologies, which can be used to explain complex
relationships [44]. By studying the patterns produced from all
the labeled input-output pairings, ML algorithms learn to offer
the appropriate output for a provided input in new cases [45].
Algorithms for supervised ML are designed to find the model
parameters that minimize the discrepancy from their predicted
impacts in their training cases to the observed effects in these
situations, in the hopes that the contexts established will be
transferable to other datasets. The test dataset may be used to
determine the generalizability of the model. For an unlabeled
collection of data, unsupervised learning may help to uncover
sub-clusters of the main dataset or recognize outliers in the
dataset. As a result, it is worth noting that low-dimensional
depictions of labeled datasets may be recognized in a
supervised way. It is possible to design applications of AI that
allow for the earlier study of unrecognized patterns in data apart
from the requirement to specify decision-making rules for every
task or take into consideration sophisticated relations between
input elements. Because of this, ML has become the most
commonly used approach for constructing AI applications [2,
46-48].
The current resurgence of AI is largely due to the active
application of DL, which comprises training a deep neural
network on large datasets [49]. Fortunately, the processing
resources necessary for DL have been enabled by recent
improvements in computer processor design [50]. Due to the
establishment of a range of large-scale research [51, 52], data
collection platforms [53], these algorithms have only recently
become available. A wide range of DL-based models has been
created for prediction and diagnosis, image classification,
biomarker identification, genomic interpretation, and

monitoring via automated robotic surgery wearables and life-
logging devices to improve digital healthcare [54]. Due to the
quick expansion of AI, healthcare data may be used to construct
DL-based strong algorithms that can automate diagnosis and
provide a more precise technique of medicine and health by
customizing pharmaceuticals and aiming for services in a
dynamic and timely manner. A non-exhaustive list of potential
uses is shown in Figure 2. Medical AI malpractice claims will
continue to get precise instructions from the legal system as to
which agency is liable if they arise. As part of the AI
framework, healthcare decisions are made in part by healthcare
practitioners with malpractice insurance [31, 55, 56]. It is
important to minimize bias while maximizing patient
satisfaction with the implementation of AI modules in these
areas [57, 58]. It is shown in Figure 2 that an auxiliary
advancement of XAI might possibly solve the problem of
limited sample learning by removing clinically insignificant
characteristics. DL models, on the other hand, may often
provide results that are beyond the comprehension of non-
experts. A DL model may have millions of parameters, which
might make understanding what the machine monitors in
clinical health data, such as CT scans, challenging [3, 48 59].
Fig. 2. Applications of XAI
Researchers have noted specifically that being a major
restriction for AI in dermatology due to its inability to perform
a tailored assessment by doctors [60]. There must be proof that
a high-performance DL algorithm genuinely recognizes the
correct region of the image and does not overemphasize
insignificant discoveries. In recent years, new ways to describe
AI models have been created, including methods for visualizing
them. Attention maps [61], class activation maps [62-63],
salience maps [64], and occlusion maps [65] are some of the
most frequently used levers in the field of visual perception.
Because the result is a picture, localization and segmentation
techniques are easier to understand. A key problem in
understanding deep neural network (DNN) based models
trained on non-imaging medical data other than images is yet
Bharati et al.
unsolved [66]. It can be noted that risk and obligations are only
two examples of concerns that are not recognized in other
domains when it comes to clinical interpretations [30, 67].
Letting AI algorithms make life-or-death medical choices
without providing enough transparency and accountability is
irresponsible [68]. A number of recent studies [6, 69-73] have
examined the issue of explainability in medical AI. COVID-19
classification and detection, emotion analysis, chest
radiography, as well as the relevance of interpretability in the
medical industry have all been studied in detail [74-75]. Figure
3 depicts an overview of XAI system. Some research works rely
mostly on keeping the interpretability of AI models intact while
enhancing their performance via optimization and refining
procedures [76-79]. The authors of [80] studied XAI in
healthcare, whereas the authors of [81] studied XAI for
electronic health records. Therefore, the purpose of this study is
to provide a synthesis of the categories of XAI in healthcare and
medicine by analyzing them in terms of dimension reduction,
feature selection, attention mechanism, knowledge distillation,
and surrogate representations utilizing XAI.
Fig. 3. An overview of XAI system step by step
III. REASONS FOR USING XAI
Recently, AI methods like DL have played a revolutionary
role in healthcare, particularly in the areas of diagnostics and
surgery. Some deep-learning-based diagnosis jobs are even
more accurate than those performed by doctors. However, the
DL model's lack of transparency makes it difficult to explain
their results and implement them in clinical settings. A growing
number of experts in the intersection of AI and healthcare have
concluded that an AI model's capacity to be explained to
humans is more important than the model's accuracy when it
comes to its practical application in a clinical setting. It is
important to understand how medical AI applications work
before they are adopted and used. As a result, there is an
incentive to conduct a survey of the medical XAI, and XAI is
necessary for the acceptability of AI applications in medicine.
IV. XAI METHODS
This section describes the methods of XAI described in the
literature [12, 84, 85-87]. For example-based, attribution-based,
and model-based explanations, the qualities of explainability,
A Review on Explainable Artificial Intelligence for Healthcare
such as interpretability characterized by simplicity and clarity,
as well as fidelity characterized by soundness and
completeness, are evaluated in this section.
Figure 3 depicts an overview of XAI system. Some research
works rely primarily on keeping the interpretability of AI
models intact while enhancing their performance via
optimization and refining procedures [76-79]. The work in[80]
and [81] studied XAI for healthcare and electronic health
records, respectively. The works of [82] and [83] focus on
specialized applications, such as digital pathology.
A. Analyzing explanations based on models
Model-based descriptions meet the completeness criteria
since they offer enough information to calculate the outcome
for a given input. Fidelity can be quantified by looking at the
proportion of predictions that are consistent with the task
model's post-hoc explanations. Global explanations, which give
a single reason for the whole model, are more likely to meet the
criterion of clarity. Because of this, an example-based,
attribution-based, or global model-based approach is the best
option for providing clarity. Using rule overlap and feature
space coverage, the researchers of [132] established a measure
of unambiguity that may be applied to this particular scenario.
On the other hand, local explanations are plagued by the issue
of the great variation in explanations for the same or
comparable cases. Local model-based explanations' clarity
could not be measured using any existing metric. As a measure
of model interpretability and as an estimate of model
complexity [85, 94, 133], model size is often utilized. For
example, the number of features (the number of rules, non-zero
weights utilized in splits), as well as the complexity of
relationships (the depth of the tree, the length of rules,
interactions), are examples of metrics, however, they are model
dependent. For example, the authors of [91] suggested
assessing model complexity by counting the number of
arithmetic and boolean operations required for the given input.
They also examined how complexity affected model accuracy.
Molnar et al. [95] calculated the level of model complexity by
looking at the number of features, the intensity of interactions,
AI APPROACHES THAT CAN BE EXPLAINED IN TERMS OF THE CHARACTERISTICS OF EXPLAINABILITY
Method Explanation Scope Fidelity
type Soundness
Post-hoc Attribution Local General quantitative metric
explanation
Attribution Global General quantitative metric
Model Local General quantitative metric
Model Global General quantitative metric
Example Local Unavailable matrices
Example Global Unavailable matrices
5
and complexity of the principal effects.
B. Analyzing explanations based on attribution
Attribution-based descriptions only give a limited
description of the model, and hence do not meet the requirement
of completeness. There were no metrics available to measure
the completeness of an explanation based on attribution.
However, the majority of attribution-based explanation
evaluation approaches may be utilized to quantify soundness.
The effectiveness of the attribution technique may be
empirically evaluated. In certain cases, this is accomplished via
the use of ground truth created artificially [66, 96]. Other
approaches use perturbation analysis to measure performance
decline [97,98]. It is the goal of axiomatic evaluations to
establish what an ideal attribution technique should accomplish,
and then to assess whether this has been done. In order to see
whether an explanation is affected by a nonsensical change in
the input data, the authors of [98] offer a test. Sensitivity and
implementation invariance are also examined by Sundararajan,
Taly, and Yan [99]. When a feature's non-zero attribution leads
to various forecasts for a given baseline, they designate an
attribution approach as sensitive. Implementation invariant
means that an attribution method always returns the same
attribution for functionally similar models (i.e., the similar
output is produced by the similar input). Explanation continuity
and selectivity are essential qualities discussed by Montavon et
al. [100]. The authors of [101] report that an attribution
approach meets sensitivity-N when the total of the attributions
for any subset of cardinality N features is equal to the change in
the output produced by eliminating the features in the subset.
C. Analyzing explanations based on example
An evaluation methodology for describing examples may be
insufficient because there are fewer tools for obtaining
influential cases and prototypes. Global examples are clear and
example-based explanations are efficient if the case is
understandable. Other conditions were not met, and no
assessment techniques were found. Table I summarizes the
results of this section.
TABLE I
Interpretability
Completeness Parsimony Clarity
Unavailable Satisfied if an instance or General quantitative metric
matrices feature is human-intelligible.
Unavailable Satisfied if an instance or Satisfied
matrices feature is human-intelligible.
Satisfied General quantitative metric Unavailable matrices
Satisfied General quantitative metric If the model is incapable of
providing various rationales,
the model is satisfied.
Unavailable Satisfied if an instance or Unavailable matrices
matrices feature is human-intelligible.
Unavailable Satisfied if an instance or Satisfied
matrices feature is human-intelligible.

V. APPLICATIONS SCENARIOS
This section presents the current and potential application
scenarios of XAI.
A. Dimension reduction using XAI
Deciphering AI models by depicting their most essential
aspects is regularly and customarily accomplished via the use
of dimension reduction methods such as Laplacian Eigenmaps
[102, 103], independent component analysis (ICA) [104], and
principal component analysis (PCA) [105]. The ideal
dimensions of the input characteristics may be used to estimate
side effects pharmacologically by merging multi-label as well
as k-nearest-neighbor approaches [106]. Unsupervised
classification of 1H MR spectroscopy brain tumor data was
improved using a nonlinear dimension reduction approach
suggested by the authors of [107], which used Laplacian
Eigenmaps to identify the most significant characteristics.
Patients with stage-one lung cancer were categorized by the
authors of [108] using a supervised learning approach, which
defined the most informative cases. A hybrid technique for
dimension reduction, integrating pattern recognition and
regression analytics, was presented to recognize a collection of
"exemplars" in order to build a "dense data matrix." In the end,
they used the most accurate instances in their model. On the
basis of their expertise in the field of cell-type-specific
enhancers, Kim and his colleagues [109] used DL to identify
which traits were most significant and ranked them in order of
importance in the model. Researchers of [110] presented a
sparse DL approach with pathway-associated to investigate
gene pathways as well as their correlations in brain tumor
patients. Using dimension reduction techniques, it is possible to
understand the model's underlying behavior by reducing the
information to a tiny subset.
B. Rule extraction and knowledge distillation using XAI
Learning a student model that is easy to understand with a
trainer model that is difficult to explain is called knowledge
distillation. This is a special kind of model-specific XAI that
involves extracting information from a complex model and
simplifying it. Dimension reduction and tree regularization
[111] or a combination technique of model compression and
dimension reduction [112] may be used to achieve this goal
[113]. Hinton et al. [114] are among the researchers who have
studied this strategy for some time, however, recent
advancements in XAI and IAI [115-117] have given it new life.
In addition to knowledge distillation, rule extraction is a
commonly applied XAI technology that is well-suited for usage
in digital healthcare. For example, the Model Understanding via
Subspace Explanations (MUSE) technique [118] has been
created to characterize the projections of the global model by
examining several subgroups of instances. For example, rule
sets or decision sets have been explored for interpretability [89].
In the context of XAI and IAI, mimic-learning is a fundamental
strategy for distilling information that uses gradient-boosting
trees to learn interpretable structures and make the original
model understandable. An interpretable model for ICU
outcomes, such as death or ventilator use, was built using the
Bharati et al.
distilled information from this technique. An explainable
description of the risk of pneumonia-related mortality was
proposed by Caruana et al. in the work of XAI and IAI [119].
Bayesian rule lists were introduced by Letham et al. as IAI
[120] and claimed to be able to predict strokes. A model
induction technique developed by the authors of [121] applied
visualization to create rules by the probability of a complex
algorithm in a variety of tasks, including the classification of
diabetes and the detection of breast cancer. By converting EHR
occurrences into embedded clinical principles, DL was used by
the authors of [122] to break dynamic connections between
potential risk factors and hospital readmission for patients. An
XAI system based on rule extraction was created by Davoodi
and Moradi [123] to predict ICU mortality, while the authors of
[124] employed a comparable XAI technique for Alzheimer's
disease diagnosis. Textual reasoning was applied to the LSTM-
based breast mass categorization by the authors of [125].
Argumentation theory was used in [126], where the XAI
algorithm was applied to the training process for risk and stroke
prediction.
C. Feature importance or selection using XAI
Using feature importance as a tool, researchers have been
able to better describe the properties and importance of
extracted features, as well as the relationships between those
features and the outcomes they predict [12, 113, 127]. The
authors of [128] employed weights in features to identify the
top 10 extractable characteristics or features for predicting
mortality in the critical care unit. The researchers of [129]
developed an algorithm for clinically relevant and risk-
calculating prostate cancer [130] by estimating the importance
of features by applying Shapley's value. The relevance of a
feature may be gauged by doing a sensitivity analysis on the
extracted features; the more sensitive the result, the more
significant the feature. To identify the most important elements
of a microbiota-based diagnostic job, Eck et al. [131] reduced
the features and tested their influence on model performance.
The other authors in [132] used a backpropagation-based
technique to achieve interpretability by using DL important
features (DeepLIFT). As a result of the backpropagation
method, it is possible to determine the importance of a feature.
By calculating the nucleotide contribution score for each
nucleoside, the authors of [133] created interpretable DL that
are IAI models for predicting splice sites using the DeepLIFT
approach. Layer-wise relevance propagation (LRP) [134],
guided backpropagation (GBP) [135], SHapley Additive
exPlanations (SHAP) [136], DeepLIFT [132], and other XAI
and IAI methods, were compared for diagnosing ophthalmic
[137]. By extracting feature significance, XAI not only
provides an explanation of features but also reflects their
relative relevance to clinical interpretation. However,
numerical weights can be either misinterpreted or difficult to
grasp.
D. Attention mechanism in XAI
Global and local attention mechanisms use all words to
produce context, and self-attention mechanisms use several
A Review on Explainable Artificial Intelligence for Healthcare
mechanisms simultaneously to locate every link between words
[138]. There is evidence that the attention mechanism aids in
both the improvement of interpretability and the advancement
of visualization technology [139]. It has been shown in the
patients of ICU utilizing the attention mechanism by the work
of [140] that some input characteristics have a greater impact
on clinical event predictions than others. Patients in the
intensive care unit (ICU) may be assessed for severity using an
interpretable acuity score framework developed by Shickel et
al. [141] that makes use of attention and DL-based evaluation
on sequential organ failure. HIV genome integration sites were
accurately predicted by Hu and colleagues [142] using
mechanistic theories. In addition, using a similar XAI strategy,
the authors of [143] constructed a system for learning how to
represent EHR data in a way that might explain the relationship
between clinical results for individual patients. The Reverse
Time Attention Model (RETAIN) was built by Choi et al. [144]
and comprised two sets of attention weights: one at the visit
level to account for the effect of each visit, and another at the
variable level to account for the influence of each variable. In
order to retain interpretability, reproduce clinical activities, and
integrate sequential information, RETAIN was developed as a
reverse attention mechanism. Predictive models for heart
failure and cataract risk have been suggested by Kwon and
colleagues [145] based on RETAIN (RetainVis). DL models'
interpretability may be improved by identifying certain
positions (e.g., DNA, visits, time) in a sequence (e.g., DNA,
visits, time) where those input attributes might impact the
output of prediction. Since 2016, XAI researchers have been
studying the complementary and alternative medicine (CAM)
[146] approach and its modifications [147], which have since
been put to use in digital healthcare, particularly in medical
picture analysis. According to the authors of [148], a limited
dataset of CT scans was utilized for training Grad-CAM to
detect appendicitis. To diagnose hypoglycemia, Porumb et al.
[149] used a combination of RNN and CNN for an
electrocardiogram (ECG) experiment with Grad-CAM to
identify the heartbeat segments that were most relevant. The
coronavirus classification method was applied with a multiscale
camera to identify the diseased regions in a work [150]. These
saliency maps are suggested because of their ease of visual
XAI APPROACHES USED IN MEDICINE AND HEALTHCARE AS WELL AS THEIR TYPES
Type Method Post-hoc Intrinsic
Dimension Optimum feature ✘ ✔
reduction selection
using XAI Laplacian Eigenmaps ✘ ✔
Sparse-balanced SVM ✘ ✔
Cluster analysis and ✘ ✔
LASSO
Sparse DL ✘ ✔
Optimal feature ✘ ✔
selection
Rule Decision rules ✘ ✔
extraction and Rule-based system ✘ ✔
knowledge Mimic learning ✔ ✘
7
interpretation. Clinical end users are not given precise
responses by attention-based XAI approaches, however, these
methods do indicate regions of increased concern, making it
simpler for them to make decisions. However, the primary
drawbacks, such as information overload and warning fatigue.
It may not be desirable to highly convey this information to a
clinical end user.
E. Surrogate representation using XAI
Individual health-related factors that lead to disease
prediction are effectively recognized by using XAI in the
medical field by applying the local interpretable model-agnostic
explanation (LIME) method [66]. This method provides
explanations for any classifier by approximating it with a
surrogate interpretable and locally faithful representation.
Linear models in the neighborhood are learned by LIME in
order to explain the disruption of an instance [151]. LIME was
utilized by Pan et al. [152] to examine the impact of additional
cases on the prediction of central precocious puberty in
children. When it comes to AI survival models, Kovalev et al.
[153] came up with an approach called SurvLIME to describe
it. According to the authors of [154], local post-hoc explanation
model was applied in their investigation of the segmented lung
suspicious items. An AI system built by Panigutti and his
colleagues [155] can forecast the patient's return to the hospital
as well as their diagnosis and prescription order. LIME's
implemented system uses a rule-based explanation to train a
local surrogate model, which may be mined employing a
decision tree with many labels. This system is similar to LIME's
implementation. For the purpose of predicting acute critical
illness via the use of EHRs, Lauritsen et al. [156] investigated
an XAI technique employing Layer-wise Relevance
Propagation [157]. The white-box estimation needs to precisely
reflect the black-box algorithm in order to provide a trustworthy
description, which is why XAI uses surrogate representation so
often. Comprehension by the physician may be compromised if
the surrogate algorithms are very complex. Table II depicts the
overview of the above methods. In addition, Table III presents
different XAI algorithms and Table IV describes particularly
SHAP XAI algorithm reported in the literature.
TABLE II
Local Global Application Ref.
✘ ✔ Prediction of enhancers that are cell type specific [109]
✘ ✔ Classification of Brain tumor from MRI [107]
✘ ✔ Diagnosis of type 2 diabetes at an early stage [158]
✘ ✔ Classification of lung cancer patients [108]
✘ ✔ Glioblastoma multiforme long-term survival prediction [110]
✘ ✔ Estimation of side effect for drug [106]
✘ ✔ Stroke Prediction [126]
✘ ✔ Forecasts for 30-day readmissions and pneumonia risk. [14]
✔ ✔ Predicting outcomes in the ICU for severe lung injury [120]
 Bharati et al.

Type Method Post-hoc Intrinsic Local Global Application Ref.

distillation Textual or visual ✔ ✘ ✔ ✔ Classification of breast mass [125]
using XAI justification
Visualization rules ✔ ✘ ✘ ✔ Clinical diagnosis of diabetes and breast cancer [121]
Lists of Bayesian rule ✘ ✔ ✘ ✔ Stroke prediction [120]
Fuzzy rules ✘ ✔ ✘ ✔ Prediction of in-hospital mortality for all cases [123]
Feature DeepLIFT ✔ ✘ ✘ ✔ Detection of splice site [133]
importance or Feature weighting ✔ ✘ ✘ ✔ Prediction of ICU mortality for all-causes [128]
selection
using XAI
DeepLIFT ✔ ✘ ✔ ✔ Ophthalmic diagnosis [137]
Feature marginalization ✔ ✘ ✔ ✔ Diagnosis of inflammatory or microbiota bowel diseases [131]
from skin and gut
Attention Attention ✘ ✔ ✔ ✘ Prediction of future hospitalization from EHR [143]
mechanism in ✘ ✔ ✔ ✔ ICU clinical events predictions [140]
XAI
✘ ✔ ✔ ✘ HIV genome integration site prediction [142]
✘ ✔ ✔ ✔ Prediction of clinical risk for cardiac cataract or failure [145]
✘ ✔ ✔ ✘ Prediction of heart failure [144]
MLCAM ✘ ✔ ✔ ✘ Localization of brain tumor [145]
Grad-CAM ✘ ✔ ✔ ✘ Appendicitis diagnosis [147]
Grad-CAM ✘ ✔ ✔ ✘ Hypoglycaemia detection using ECG [149]
Surrogate LIME ✔ ✘ ✔ ✘ Prediction of early puberty in the central region [152]
representation ✔ ✘ ✔ ✘ Construction of survival models [153]
using XAI
✔ ✘ ✔ ✘ Diagnosis of autism spectrum disorder [159]
Rule-based XAI ✔ ✘ ✔ ✘ Patient medications, diagnosis and readmission prediction [155]

TABLE III
APPLICATIONS OF XAI ALGORITHMS IN DIFFERENT SCENARIOS
Algorithm Ref. Location Mode Remarks
LRP [175] Brain MRI employs LRP to locate Alzheimer's disease-causing brain areas, where LRP is proved to be
more specific than guided backpropagation in detecting the disease areas.
Multiple instance [176] Dental Eye Fundus Image Presents a cutting-edge DL-based diabetic retinopathy grading system based
learning on a medically comprehensible explanation; and infers an image grade.
Trainable attention [177] Dental Eye Fundus Image proposes an attention-based CNN for removing high redundancy in fundus images for
glaucoma detection.
Trainable attention [178] Chest X-ray Proposes two novel neural networks for the detection of chest radiographs containing
pulmonary lesions; both the architectures make use of a large number of weakly-labelled
images combined with a smaller number of manually annotated X-rays.
Grad-CAM , LIME [179] Chest X-ray Presents several examples of CXRs with pneumonia infection, together with the results from
the grad-CAM and LIME; LIME's explanatory output is presented as the superpixels with
the highest positive weights superimposed on the original input.
CAM [180] Prostate Histology constructs a patch-wise predictive model using convolutional neural networks to detect
cancerous Gleason patterns, and examines the interpretability of trained models using Class
Activation Maps (CAMs).
Saliency propagation [181] Skin Vitiligo image Proposes a unique weakly supervised approach to segmenting vitiligo lesions with preserved
fine borders; the activation map can be paired with superpixel-based saliency propagation to
generate segmentation with well-preserved edges.
Guided [182] Gastrointestinal Endoscopy Uses Guided Backpropagation based Fully Convolutional Networks to investigate the pixels
backpropagation necessary for identifying polyps (FCNs).
TCAV [183] Cardiovascular MRI using TCAV (Testing with Concept Activation Vectors) to show which biomarkers are
already known to be linked to cardiac disease.
TCAV [184] Breast and Eye histopathology and proposing the addition of regression concept vector to TCAV in identifying tumor size, while
retinopathy TCAV represented binary ideas, regression concept vectors indicated continuous-valued
measurements.

TABLE IV
DISCUSSION OF SHAP XAI ALGORITHMS IN THE LITERATURE
Ref Disease Diagnosis ML Models Used Work Summary
[185] Chronic Obstructive Gradient boosting machine (GBM) The work ranks the attributes crucial to the GBM model according to their average
Pulmonary Disease absolute SHAP values that reflect the impact of each feature on a prediction.
[186] Parkinson's disease deep forest (gcForest), extreme The classifiers determine how much the SHAP value contributes to the features.
diagnosis gradient boosting (XGBoost), light When 150 features are considered, SHAP-gcForest achieves 91.78 percent
gradient boosting machine classification accuracy. With 50 features, SHAP-LightGBM combined with
(LightGBM) and random forest (RF) LightGBM yields 91.62% accuracy.
[187] predicting acute XGboost , Logistic Regression (LR) The SHAP value increases as creatinine levels climb until it reaches about 5 mg/dL.
kidney injury As the Furosemide Stress Test (FST) is increased until it reaches 100 ml/min, the
progression SHAP result decreases.
A Review on Explainable Artificial Intelligence for Healthcare 9

[188] COVID-19 Prediction XGboost model, LGBM, gradient Harris hawks optimization (HHO) approach is employed to fine-tune the
boosting classifier (GBC), hyperparameters of a few cutting-edge classifiers, including the HHO-RF, HHO-
categorical boosting (CatBoost), RF XGB, HHO-LGB, and ensemble methods, in order to increase classification
accuracy.
[189] Prediction of COVID- GBM A gradient-boosting machine model created with decision-tree base-learners was
19 diagnosis based on used to produce predictions. SHAP values calculate the contribution of each feature
symptoms to overall model predictions by average across samples.
[190] Efficient analysis of Support Vector Machine (SVM), Simple classification techniques like random forest can outperform keras with
COVID-19 Naive Bayes (NB), Multiple Linear Boruta for feature selection when the dataset size is not too large. Computing the
Regression (MLP), K-Nearest information gain values for each attribute in Clinical Data1 shows the importance of
Neighbors (KNN), RF, LR, and each attribute.
Decision Tree (DT), Keras Classifier
[191] COVID-19 Diagnosis SqueezeNet, LIME LIME and SHAP are compared for COVID diagnosis using SqueezeNet to recognize
pneumonia, COVID-19, and normal lung imaging. Results show LIME and SHAP
can boost the model's transparency and interpretability.
[192] COVID-19 vaccine Random Forest, XGBoost classifiers, CovidXAI predicts a user's risk group using Random Forest and XGBoost classifiers.
prioritization XAI CovidXAI uses 24 criteria to define an individual's risk category and vaccine
urgency.
[193] COVID-19 XGBoost, Random Forest The goal is to provide grounds for understanding the distinctive COVID-19
Pneumonia radiographic texture features using supervised ensemble ML methods based on trees
Classification through the interpretable SHAP approach. SHAP recursive feature elimination with
cross-validation is used to select features. The best classification model was
XGBoost, with an accuracy of 0.82 and a sensitivity of 0.82.
[194] Lung cancer hospital Random Forest, logistic regression This paper introduces a predictive LOS framework for lung cancer patients utilizing
length of stay ML models. Using SHAP, the output of the predictive model (RF) with SMOTE
prediction class balancing approach is understood demonstrating the most relevant clinical
factors that contributed to predicting lung cancer LOS using the RF model.

VI. DISCUSSION ON DIFFERENT ASPECTS OF XAI A. Developing XAI to create trustworthy AI

It is still a work in progress to design and govern AI systems
that can be trusted. We looked at the function that explainability
plays in constructing trustworthy AI since the two concepts are
intertwined. A framework with actual suggestions for choosing
between classes of XAI approaches is provided by us as an
extension to previous recent surveys [12, 19, 85, 92, 113]. As
a result, we presented useful definitions and made contributions
to the literature on quantitative assessment measures. XAI is
being discussed as a part of a broader effort to build confidence
in AI.
Much work on automated diagnosis and prognosis has been
done in the medical area using ML techniques [29]. We can see
from grand-challenge.org that a number of medical issues have
surfaced and energized ML and AI researchers. DL models
[160], [161] that use U-Net for medical segmentation are
effective. U-Net, on the other hand, is still a mystery since it is
a DL neural network. There are several more approaches and
special transformations (denoising, for example) that have been
described in MICCAI papers, such as [162]. The issue of
interpretability in medicine goes much beyond idle speculation.
Medical interpretability contains risks and duties that other
areas do not take into consideration [67] . There is a risk of
death associated with medical decisions that are made. The
pursuit of medical explainability has therefore resulted in
several further publications [71]. To raise awareness about the
significance of interpretability in the medical sector, they
summarize prior studies [163], consisting of subfield-specific
reviews, i.e., [74] for sentiment analysis and chest radiography
in medicine [75]. As stated explicitly in [60], AI in dermatology
is severely limited because it cannot deliver individualized
assessments [60].
The reason for requiring explainability is that it dictates what
ought to be communicated as well as a step-by-step guide for
developers with real diagram suggestions was offered. The
development of post-hoc explanation techniques that contain
argumentation evidence for their assertions [164] might be a
solution to the problem of misleading post-hoc explanations.
Model-based explanations can be suitable when looking for a
post-hoc explanation since they are full and contain quantitative
proxy criteria to assess their soundness. Many more studies are
required to better understand the performance of explainable
models in the healthcare industry [165] and to develop
explainable modeling approaches such as rule-based systems or
GAMs with or without interaction terms. Thus, the importance
of feature engineering in the context of feature interpretation is
further highlighted. Another potential research approach is to
create hybrid techniques that combine knowledge-driven and
data-driven aspects for feature engineering or feature selection.
The quality of explanations is critical when utilizing XAI to
build trustworthy AI. According to Poursabzi-Sangdeh et al.
[84], improved results from human-machine tasks are not
always accompanied by greater performance. We discovered
that evaluating the clarity of local explanations is challenging
and that there are currently no quantitative assessment measures
for example-based methodologies. Even while interpretability
is commonly recognized to be user-dependent, it has not been
measured as such when assessing the quality of explanations
[85]. Future research should focus on establishing a criterion
for identifying when AI systems can be explained to a wide
range of people.
B. Complementary approaches to the development of
trustworthy healthcare AI
In some cases (e.g. [165]), explanations are not essential nor

adequate to build faith in AI. Perceived system capabilities,
control, and predictability are other crucial factors [166].
Because of this, even while XAI has the potential to contribute
to reliable AI, it is not without its limitations [167]. The
following are some additional approaches that may be
implemented in conjunction with AI in healthcare to ensure its
trustworthiness:
The first one is regarding the quality of data. It is important
for providing information on the quality of the data. Real-world
data may include biases, inaccuracies, or be incomplete since it
was not gathered for study reasons. This means that knowing
the quality of the data obtained and how it was acquired is at
least as crucial to the final model's constraints as its
explainability [168]. Kahn et al. [169] provide a generally
recognized approach for determining whether EHR data is
appropriate for a given use case. The second aspect is about
conducting comprehensive testing. The use of standard data
formats in health care has made external validation more viable,
even if it is still considered an area for the development of
models of clinical risk prediction [87, 170]. Stability, fairness,
and privacy are only a few of the topics being studied in this
field. Another aspect is the governing law.
C. Trends in Existing Research
Some interesting trends are visible in the works reported in
the literature. These studies have been described in detail in the
previous section, and now some of the insights are described in
the following. Literature indicates post-hoc and model-specific
explanations can be used to categorize XAI model explanations.
Gradient-based, perturbation-based, and rule-based attribution
are three different methods. XAI can assist in explaining how
the reduced features are connected to the original variables and
how they impact the model's performance in the context of
dimension reduction. Laplacian Eigenmaps, ICA, and PCA
techniques for dimension reduction utilizing XAI are available.
Other similar techniques are feature importance, and partial
dependence plots. Studies so far indicate that in the field of
XAI, rule extraction and knowledge distillation are two crucial
concepts. The permutation importance method is one of the
most widely used XAI strategies for feature importance.
Another XAI technique for determining feature importance is
the SHAP, partial dependence plots, and methods based on
decision trees, including the Gini index and information gain.
An electrocardiogram experiment with Grad-CAM can be used
to diagnose hypoglycemia by combining RNN and CNN. In
XAI, examples of surrogate models found in the literature are
decision trees and linear regression models.
Several XAI techniques are difficult to compute with and
may be challenging to scale to large datasets or complex
models. There is still a shortage of XAI methods interaction
with useful systems. There is not enough information on how
humans interpret and apply XAI explanations. However, some
recent studies discuss the idea of federated learning of XAI
models, which was created specifically to fulfill these two
criteria simultaneously. Recent studies also consider the
application of XAI in the field of medical data standardization
which is a still a challenging issue in many underdeveloped or
Bharati et al.
developing countries. Moreover, some studies are focused to
developing hybrid approaches of XAI where individual XAI
techniques are combined for better performance. Furthermore,
some recent works are concentrating on research intersecting
quantum computing and XAI as quantum computing has the
potential to improve the performance of XAI for healthcare.
D. Challenges and Future Prospects
Intractable systems may be simplified using models. A lack
of utilization of DNNs with a large number of parameters may
result in under-utilized ML capabilities. It is feasible to take
advantage of the excitement around predictive science by
examining if it is viable to add new, more complex components
to an existing model, such that sections of these components
may be recognized as fresh insights. The enhanced model must,
of course, be similar to earlier demonstrations and models, with
a clear interpretation of why the additional components
correlate to formally ignored insights.
Some approaches that are deeply embedded in particular
industries are still widely used, while powerful ML algorithms
are finding new uses. Fragmented and experimental
implementations of existing or custom-made interpretable
techniques are common in medical ML, which is still in its
infancy. We may still have large undiscovered prospects in
interpretability research despite the current emphasis on
increasing accuracy and performance in feature selection and
extraction.
VII. CONCLUSIONS
This review paper summarizes the state-of-the-art, evaluates
the level of quality of recent research, identifies the research
gap, and makes recommendations for future studies that can
significantly develop XAI. In particular, this review focuses on
the application of XAI techniques to the healthcare industry.
Discussions are provided on the importance of using XAI, the
scenarios where XAI is suitable, and the methods of
implementing XAI. In this regard, this paper provides an
overview of the literature, connecting various perspectives and
advocating specific design solutions. It is discussed here that AI
has the potential to alter medical practice and the role of
physicians. Moreover, explainable models may be favored over
posthoc explanations when XAI is used to develop trustworthy
AI for healthcare. Rigorous regulation and validation may be
required in addition to reporting data quality and extensive
validation, as limited evidence supports the practical utility of
explainability. In the future, efforts will be necessary to address
the problems and concerns that have been raised with XAI.
APPENDIX
((‘‘Explainable AI’’) OR (‘‘Interpretable AI’’) OR
(‘‘Explainable Artificial Intelligence’’) OR (‘‘Interpretable
Artificial Intelligence’’) OR (‘‘XAI’’)) AND ((‘‘medical’’) OR
(‘‘healthcare’’))
ACKNOWLEDGMENT
This work has been carried out at the Institute of Information
A Review on Explainable Artificial Intelligence for Healthcare
and Communication Technology (IICT) of Bangladesh
University of Engineering and Technology (BUET),
Bangladesh. Hence, the authors would like to thank BUET for
providing the support.
REFERENCES
[1] A. Gupta, A. Anpalagan, L. Guan, and A. S. Khwaja, "Deep learning
for object detection and scene perception in self-driving cars:
Survey, challenges, and open issues," Array, vol. 10, p. 100057,
2021.
[2] S. Bharati, P. Podder, and M. R. H. Mondal, "Artificial Neural
Network Based Breast Cancer Screening: A Comprehensive
Review," International Journal of Computer Information Systems
and Industrial Management Applications, vol. 12, pp. 125-137,
2020.
[3] M. R. H. Mondal, S. Bharati, and P. Podder, "CO-IRv2: Optimized
InceptionResNetV2 for COVID-19 detection from chest CT
images," PloS one, vol. 16, no. 10, p. e0259179, 2021.
[4] S. Bharati, P. Podder, and M. R. H. Mondal, "Hybrid deep learning
for detecting lung diseases from X-ray images," Informatics in
Medicine Unlocked, p. 100391, 2020.
[5] A. Rajkomar et al., "Scalable and accurate deep learning with
electronic health records," NPJ Digital Medicine, vol. 1, no. 1, pp.
1-10, 2018.
[6] S. Tonekaboni, S. Joshi, M. D. McCradden, and A. Goldenberg,
"What clinicians want: contextualizing explainable machine
learning for clinical end use," presented at the Machine learning for
healthcare conference, 2019, 2019.
[7] E. D. Peterson, "Machine learning, predictive analytics, and clinical
practice: can the past inform the present?," Jama, vol. 322, no. 23,
pp. 2283-2284, 2019.
[8] J. He, S. L. Baxter, J. Xu, J. Xu, X. Zhou, and K. Zhang, "The
practical implementation of artificial intelligence technologies in
medicine," Nature medicine, vol. 25, no. 1, pp. 30-36, 2019.
[9] Z. C. Lipton, "The Mythos of Model Interpretability: In machine
learning, the concept of interpretability is both important and
slippery," Queue, vol. 16, no. 3, pp. 31-57, 2018.
[10] J. Burrell, "How the machine ‘thinks’: Understanding opacity in
machine learning algorithms," Big Data & Society, vol. 3, no. 1, p.
2053951715622512, 2016.
[11] F. Doshi-Velez and B. Kim, "Considerations for evaluation and
generalization in interpretable machine learning," in Explainable
and interpretable models in computer vision and machine learning:
Springer, 2018, pp. 3-17.
[12] E. J. Hwang et al., "Development and validation of a deep learning–
based automated detection algorithm for major thoracic diseases on
chest radiographs," JAMA network open, vol. 2, no. 3, pp. e191095-
e191095, 2019.
[13] J. G. Nam et al., "Development and validation of deep learning–
based automatic detection algorithm for malignant pulmonary
nodules on chest radiographs," Radiology, vol. 290, no. 1, pp. 218-
228, 2019.
[14] K. J. Geras et al., "High-resolution breast cancer screening with
multi-view deep convolutional neural networks," arXiv preprint
arXiv:1703.07047, 2017.
[15] A. Khamparia et al., "Diagnosis of Breast Cancer Based on Modern
Mammography using Hybrid Transfer Learning," Multidimensional
Systems and Signal Processing, 2020.
[16] M. R. H. Mondal, S. Bharati, and P. Podder, "Diagnosis of COVID-
19 Using Machine Learning and Deep Learning: A Review,"
Current Medical Imaging, 2021.
[17] S. Bharati, P. Podder, M. Mondal, and V. B. Prasath, "Medical
Imaging with Deep Learning for COVID-19 Diagnosis: A
Comprehensive Review," International Journal of Computer
Information Systems and Industrial Management Applications, vol.
13, pp. 91 - 112, 2021.
[18] E. J. Topol, "High-performance medicine: the convergence of
human and artificial intelligence," Nature medicine, vol. 25, no. 1,
pp. 44-56, 2019.
11
[19] A. B. Arrieta et al., "Explainable Artificial Intelligence (XAI):
Concepts, taxonomies, opportunities and challenges toward
responsible AI," Information fusion, vol. 58, pp. 82-115, 2020.
[20] R. Basnet, M. O. Ahmad, and M. N. S. Swamy, "A deep dense
residual network with reduced parameters for volumetric brain
tissue segmentation from MR images," Biomedical Signal
Processing and Control, vol.70, p. 103063, 2021.
[21] S. Bharati and M. R. Hossain Mondal, "12 Applications and
challenges of AI-driven IoHT for combating pandemics: a
reviewComputational Intelligence for Managing Pandemics," A.
Khamparia, R. Hossain Mondal, P. Podder, B. Bhushan, V. H. C. d.
Albuquerque, and S. Kumar, Eds.: De Gruyter, 2021, pp. 213-230.
[22] P. Podder, S. Bharati, M. R. H. Mondal, and U. Kose, "Application
of Machine Learning for the Diagnosis of COVID-19," in Data
Science for COVID-19: Elsevier, 2021, pp. 175-194.
[23] E. Jabason, M. O. Ahmad, and M. N. S. Swamy, "Classification of
Alzheimer’s Disease from MRI Data Using a Lightweight Deep
Convolutional Model," In 2022 IEEE International Symposium on
Circuits and Systems (ISCAS), pp. 1279-1283. IEEE, 2022.
[24] S. Bharati, P. Podder, M. Mondal, and V. B. Prasath, "CO-ResNet:
Optimized ResNet model for COVID-19 diagnosis from X-ray
images," International Journal of Hybrid Intelligent Systems, vol.
17, pp. 71-85, 2021.
[25] M. R. H. Mondal, S. Bharati, P. Podder, and P. Podder, "Data
analytics for novel coronavirus disease," Informatics in Medicine
Unlocked, vol. 20, p. 100374, 2020.
[26] G. Yang, Q. Ye, and J. Xia, "Unbox the black-box for the medical
explainable AI via multi-modal and multi-centre data fusion: A
mini-review, two showcases and beyond," Information Fusion, vol.
77, pp. 29-52, 2022/01/01/ 2022.
[27] B. Mahbooba, M. Timilsina, R. Sahal, and M. Serrano, "Explainable
artificial intelligence (xai) to enhance trust management in intrusion
detection systems using decision tree model," Complexity, vol.
2021, 2021.
[28] S. R. Islam, W. Eberle, and S. K. Ghafoor, "Towards quantification
of explainability in explainable artificial intelligence methods," In
The thirty-third international flairs conference 2020, 2020.
[29] F. Jiang et al., "Artificial intelligence in healthcare: past, present and
future," Stroke and vascular neurology, vol. 2, no. 4, pp. 230-243,
2017.
[30] T. Panch, H. Mattie, and L. A. Celi, "The “inconvenient truth” about
AI in healthcare," NPJ digital medicine, vol. 2, no. 1, pp. 1-3, 2019.
[31] K.-H. Yu, A. L. Beam, and I. S. Kohane, "Artificial intelligence in
healthcare," Nature biomedical engineering, vol. 2, no. 10, pp. 719-
731, 2018.
[32] D. Shen, G. Wu, and H.-I. Suk, "Deep learning in medical image
analysis," Annual review of biomedical engineering, vol. 19, pp.
221-248, 2017.
[33] G. Litjens et al., "A survey on deep learning in medical image
analysis," Medical image analysis, vol. 42, pp. 60-88, 2017.
[34] J. Ker, L. Wang, J. Rao, and T. Lim, "Deep learning applications in
medical image analysis," Ieee Access, vol. 6, pp. 9375-9389, 2017.
[35] S. E. Dilsizian and E. L. Siegel, "Artificial intelligence in medicine
and cardiac imaging: harnessing big data and advanced computing
to provide personalized medical diagnosis and treatment," Current
cardiology reports, vol. 16, no. 1, p. 441, 2014.
[36] V. L. Patel et al., "The coming of age of artificial intelligence in
medicine," Artificial intelligence in medicine, vol. 46, no. 1, pp. 5-
17, 2009.
[37] S. Jha and E. J. Topol, "Adapting to artificial intelligence:
radiologists and pathologists as information specialists," Jama, vol.
316, no. 22, pp. 2353-2354, 2016.
[38] E. Strickland, "IBM Watson, heal thyself: How IBM overpromised
and underdelivered on AI health care," IEEE Spectrum, vol. 56, no.
4, pp. 24-31, 2019.
[39] N. S. Weingart, R. M. Wilson, R. W. Gibberd, and B. Harrison,
"Epidemiology of medical error," Bmj, vol. 320, no. 7237, pp. 774-
777, 2000.
[40] M. L. Graber, N. Franklin, and R. Gordon, "Diagnostic error in
internal medicine," Archives of internal medicine, vol. 165, no. 13,
pp. 1493-1499, 2005.
[41] I. Doroniewicz et al., "Computer-based analysis of spontaneous
infant activity: A pilot study," in Information Technology in
Biomedicine: Springer, 2021, pp. 147-159.

[42] S. Sachan, F. Almaghrabi, J.-B. Yang, and D.-L. Xu, "Evidential
reasoning for preprocessing uncertain categorical data for
trustworthy decisions: An application on healthcare and finance,"
Expert Systems with Applications, vol. 185, p. 115597, 2021.
[43] D. Li et al., "Semi-supervised variational reasoning for medical
dialogue generation," presented at the Proceedings of the 44th
International ACM SIGIR Conference on Research and
Development in Information Retrieval, 2021, 2021.
[44] A. Rajkomar, J. Dean, and I. Kohane, "Machine learning in
medicine," New England Journal of Medicine, vol. 380, no. 14, pp.
1347-1358, 2019.
[45] K.-H. Yu and M. Snyder, "Omics profiling in precision oncology,"
Molecular & Cellular Proteomics, vol. 15, no. 8, pp. 2525-2536,
2016.
[46] K. Roberts et al., "Biomedical informatics advancing the national
health agenda: the AMIA 2015 year-in-review in clinical and
consumer informatics," Journal of the American Medical
Informatics Association, vol. 24, no. e1, pp. e185-e190, 2017.
[47] W. Rogers et al., "Radiomics: from qualitative to quantitative
imaging," The British journal of radiology, vol. 93, no. 1108, p.
20190948, 2020.
[48] S. Bharati, P. Podder, M. R. H. Mondal, and N. Gandhi, "Optimized
NASNet for Diagnosis of COVID-19 from Lung CT Images,"
presented at the 20th International Conference on Intelligent
Systems Design and Applications (ISDA 2020), 2020.
[49] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT
press, 2016.
[50] Y. E. Wang, G.-Y. Wei, and D. Brooks, "Benchmarking tpu, gpu,
and cpu platforms for deep learning," arXiv preprint
arXiv:1907.10701, 2019.
[51] K. Tomczak, P. Czerwińska, and M. Wiznerowicz, "The Cancer
Genome Atlas (TCGA): an immeasurable source of knowledge,"
Contemporary oncology, vol. 19, no. 1A, p. A68, 2015.
[52] C. Sudlow et al., "UK biobank: an open access resource for
identifying the causes of a wide range of complex diseases of middle
and old age," PLoS medicine, vol. 12, no. 3, p. e1001779, 2015.
[53] E. Williams et al., "Image Data Resource: a bioimage data
integration and publication platform," Nature methods, vol. 14, no.
8, pp. 775-781, 2017.
[54] A. Esteva et al., "A guide to deep learning in healthcare," Nature
medicine, vol. 25, no. 1, pp. 24-29, 2019.
[55] D. Schönberger, "Artificial intelligence in healthcare: a critical
analysis of the legal and ethical implications," International Journal
of Law and Information Technology, vol. 27, no. 2, pp. 171-203,
2019.
[56] D. Higgins and V. I. Madai, "From bit to bedside: a practical
framework for artificial intelligence product development in
healthcare," Advanced intelligent systems, vol. 2, no. 10, p.
2000052, 2020.
[57] M. DeCamp and C. Lindvall, "Latent bias and the implementation
of artificial intelligence in medicine," Journal of the American
Medical Informatics Association, vol. 27, no. 12, pp. 2020-2023,
2020.
[58] P. Esmaeilzadeh, "Use of AI-based tools for healthcare purposes: a
survey study from consumers’ perspectives," BMC medical
informatics and decision making, vol. 20, no. 1, pp. 1-19, 2020.
[59] J. R. England and P. M. Cheng, "Artificial intelligence for medical
image analysis: a guide for authors and reviewers," American
journal of roentgenology, vol. 212, no. 3, pp. 513-519, 2019.
[60] A. Gomolin, E. Netchiporouk, R. Gniadecki, and I. V. Litvinov,
"Artificial intelligence applications in dermatology: where do we
stand?," Frontiers in medicine, vol. 7, p. 100, 2020.
[61] D. C. Somers and S. L. Sheremata, "Attention maps in the brain,"
Wiley Interdisciplinary Reviews: Cognitive Science, vol. 4, no. 4,
pp. 327-340, 2013.
[62] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and
D. Batra, "Grad-cam: Visual explanations from deep networks via
gradient-based localization," presented at the Proceedings of the
IEEE international conference on computer vision, 2017, 2017.
[63] B. N. Patro, M. Lunayach, S. Patel, and V. P. Namboodiri, "U-cam:
Visual explanation using uncertainty based class activation maps,"
presented at the Proceedings of the IEEE/CVF International
Conference on Computer Vision, 2019, 2019.
Bharati et al.
[64] C. Roggeman, W. Fias, and T. Verguts, "Salience maps in parietal
cortex: imaging and computational modeling," Neuroimage, vol. 52,
no. 3, pp. 1005-1014, 2010.
[65] M. Aminu, N. A. Ahmad, and M. H. M. Noor, "Covid-19 detection
via deep neural network and occlusion sensitivity maps," Alexandria
Engineering Journal, vol. 60, no. 5, pp. 4829-4855, 2021.
[66] M. T. Ribeiro, S. Singh, and C. Guestrin, "Why should i trust you?"
Explaining the predictions of any classifier," presented at the
Proceedings of the 22nd ACM SIGKDD international conference on
knowledge discovery and data mining, 2016, 2016.
[67] P. Croskerry, K. Cosby, M. L. Graber, and H. Singh, Diagnosis:
Interpreting the shadows. CRC Press, 2017.
[68] T. P. Quinn, S. Jacobs, M. Senadeera, V. Le, and S. Coghlan, "The
three ghosts of medical AI: Can the black-box present deliver?,"
Artificial Intelligence in Medicine, p. 102158, 2021.
[69] Z. Zhang, Y. Xie, F. Xing, M. McGough, and L. Yang, "Mdnet: A
semantically and visually interpretable medical image diagnosis
network," presented at the Proceedings of the IEEE conference on
computer vision and pattern recognition, 2017, 2017.
[70] A. Holzinger, C. Biemann, C. S. Pattichis, and D. B. Kell, "What do
we need to build explainable AI systems for the medical domain?,"
arXiv preprint arXiv:1712.09923, 2017.
[71] A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Müller,
"Causability and explainability of artificial intelligence in
medicine," Wiley Interdisciplinary Reviews: Data Mining and
Knowledge Discovery, vol. 9, no. 4, p. e1312, 2019.
[72] M. S. Hossain, G. Muhammad, and N. Guizani, "Explainable AI and
mass surveillance system-based healthcare framework to combat
COVID-I9 like pandemics," IEEE Network, vol. 34, no. 4, pp. 126-
132, 2020.
[73] E. Khodabandehloo, D. Riboni, and A. Alimohammadi,
"HealthXAI: Collaborative and explainable AI for supporting early
diagnosis of cognitive decline," Future Generation Computer
Systems, vol. 116, pp. 168-189, 2021.
[74] K. Kallianos et al., "How far have we come? Artificial intelligence
for chest radiograph interpretation," Clinical radiology, vol. 74, no.
5, pp. 338-345, 2019.
[75] C. Zucco, H. Liang, G. Di Fatta, and M. Cannataro, "Explainable
sentiment analysis with applications in medicine," presented at the
2018 IEEE International Conference on Bioinformatics and
Biomedicine (BIBM), 2018, 2018.
[76] G. Stiglic, S. Kocbek, I. Pernek, and P. Kokol, "Comprehensive
decision tree models in bioinformatics," PloS one, vol. 7, no. 3, p.
e33812, 2012.
[77] X. Peng, Y. Li, I. Tsang, H. Zhu, J. Lv, and J. T. Zhou, "XAI Beyond
Classification: Interpretable Neural Clustering," Journal of Machine
Learning Research, vol. 22, pp. 1-28, 2021.
[78] D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, and G.-Z.
Yang, "XAI—Explainable artificial intelligence," Science Robotics,
vol. 4, no. 37, p. eaay7120, 2019.
[79] G. Valdes, J. M. Luna, E. Eaton, C. B. Simone, L. H. Ungar, and T.
D. Solberg, "MediBoost: a patient stratification tool for interpretable
decision making in the era of precision medicine," Scientific reports,
vol. 6, no. 1, pp. 1-8, 2016.
[80] E. Tjoa and C. Guan, "A survey on explainable artificial intelligence
(xai): Toward medical xai," IEEE transactions on neural networks
and learning systems, vol. 32, no. 11, pp. 4793-4813, 2020.
[81] S. N. Payrovnaziri et al., "Explainable artificial intelligence models
using real-world electronic health record data: a systematic scoping
review," Journal of the American Medical Informatics Association,
vol. 27, no. 7, pp. 1173-1185, 2020.
[82] M. Pocevičiūtė, G. Eilertsen, and C. Lundström, "Survey of XAI in
digital pathology," in Artificial intelligence and machine learning
for digital pathology: Springer, 2020, pp. 56-88.
[83] A. B. Tosun, F. Pullara, M. J. Becich, D. Taylor, S. C.
Chennubhotla, and J. L. Fine, "Histomapr™: An explainable ai (xai)
platform for computational pathology solutions," in Artificial
Intelligence and Machine Learning for Digital Pathology: Springer,
2020, pp. 204-227.
[84] F. Doshi-Velez and B. Kim, "Towards a rigorous science of
interpretable machine learning," arXiv preprint arXiv:1702.08608,
2017.
[85] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and
D. Pedreschi, "A survey of methods for explaining black box
A Review on Explainable Artificial Intelligence for Healthcare
models," ACM computing surveys (CSUR), vol. 51, no. 5, pp. 1-42,
2018.
[86] M. Nazar, M. M. Alam, E. Yafi, and M. S. Mazliham, "A Systematic
Review of Human-Computer Interaction and Explainable Artificial
Intelligence in Healthcare with Artificial Intelligence Techniques,"
IEEE Access, 2021.
[87] A. Das and P. Rad, "Opportunities and challenges in explainable
artificial intelligence (xai): A survey," arXiv preprint
arXiv:2006.11371, 2020.
[88] J. Huysmans, K. Dejaeger, C. Mues, J. Vanthienen, and B. Baesens,
"An empirical evaluation of the comprehensibility of decision table,
tree and rule based predictive models," Decision Support Systems,
vol. 51, no. 1, pp. 141-154, 2011.
[89] I. Lage et al., "An evaluation of the human-interpretability of
explanation," arXiv preprint arXiv:1902.00006, 2019.
[90] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W.
Wortman Vaughan, and H. Wallach, "Manipulating and measuring
model interpretability," presented at the Proceedings of the 2021
CHI conference on human factors in computing systems, 2021,
2021.
[91] D. Slack, S. A. Friedler, C. Scheidegger, and C. D. Roy, "Assessing
the local interpretability of machine learning models," arXiv
preprint arXiv:1902.03501, 2019.
[92] S. Mohseni, N. Zarei, and E. D. Ragan, "A survey of evaluation
methods and measures for interpretable machine learning," arXiv
preprint arXiv:1811.11839, vol. 1, 2018.
[93] H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec, "Interpretable
& explorable approximations of black box models," arXiv preprint
arXiv:1707.01154, 2017.
[94] M. Barandas, D. Folgado, R. Santos, R. Simão, and H. Gamboa,
"Uncertainty-Based Rejection in Machine Learning: Implications
for Model Development and Interpretability," Electronics, vol. 11,
no. 3, p. 396, 2022.
[95] C. Molnar, G. Casalicchio, and B. Bischl, "Quantifying
interpretability of arbitrary machine learning models through
functional decomposition," 2019.
[96] L. Arras, A. Osman, K.-R. Müller, and W. Samek, "Evaluating
recurrent neural network explanations," arXiv preprint
arXiv:1904.11829, 2019.
[97] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R.
Müller, "Explaining nonlinear classification decisions with deep
taylor decomposition," Pattern recognition, vol. 65, pp. 211-222,
2017.
[98] S. Hooker, D. Erhan, P.-J. Kindermans, and B. Kim, "A benchmark
for interpretability methods in deep neural networks," Advances in
neural information processing systems, vol. 32, 2019.
[99] M. Sundararajan, A. Taly, and Q. Yan, "Axiomatic attribution for
deep networks," pp. 3319-3328: PMLR.
[100] G. Montavon, W. Samek, and K.-R. Müller, "Methods for
interpreting and understanding deep neural networks," Digital
Signal Processing, vol. 73, pp. 1-15, 2018.
[101] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross, "Towards better
understanding of gradient-based attribution methods for deep neural
networks," arXiv preprint arXiv:1711.06104, 2017.
[102] S. Zhang, J. R. Leistico, R. J. Cho, J. B. Cheng, and J. S. Song,
"Spectral clustering of single-cell multi-omics data on multilayer
graphs," bioRxiv, 2022.
[103] M. Belkin and P. Niyogi, "Laplacian eigenmaps and spectral
techniques for embedding and clustering," Advances in neural
information processing systems, vol. 14, 2001.
[104] P. Comon, "Independent component analysis, a new concept?,"
Signal processing, vol. 36, no. 3, pp. 287-314, 1994.
[105] R. Bro and A. K. Smilde, "Principal component analysis,"
Analytical methods, vol. 6, no. 9, pp. 2812-2831, 2014.
[106] W. Zhang, F. Liu, L. Luo, and J. Zhang, "Predicting drug side effects
by multi-label learning and ensemble learning," BMC
bioinformatics, vol. 16, no. 1, pp. 1-11, 2015.
[107] G. Yang, F. Raschke, T. R. Barrick, and F. A. Howe, "Manifold
Learning in MR spectroscopy using nonlinear dimensionality
reduction and unsupervised clustering," Magnetic resonance in
medicine, vol. 74, no. 3, pp. 868-878, 2015.
[108] L. P. Zhao and H. Bolouri, "Object-oriented regression for building
predictive models with high dimensional omics data from
translational studies," Journal of biomedical informatics, vol. 60,
pp. 431-445, 2016.
13
[109] S. G. Kim, N. Theera-Ampornpunt, C.-H. Fang, M. Harwani, A.
Grama, and S. Chaterji, "Opening up the blackbox: an interpretable
deep neural network-based classifier for cell-type specific enhancer
predictions," BMC systems biology, vol. 10, no. 2, pp. 243-258,
2016.
[110] J. Hao, Y. Kim, T.-K. Kim, and M. Kang, "PASNet: pathway-
associated sparse deep neural network for prognosis prediction from
high-throughput data," BMC bioinformatics, vol. 19, no. 1, pp. 1-13,
2018.
[111] M. Wu, M. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F. Doshi-
Velez, "Beyond sparsity: Tree regularization of deep models for
interpretability," presented at the Proceedings of the AAAI
conference on artificial intelligence, 2018, 2018.
[112] A. Polino, R. Pascanu, and D. Alistarh, "Model compression via
distillation and quantization," arXiv preprint arXiv:1802.05668,
2018.
[113] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, "Machine learning
interpretability: A survey on methods and metrics," Electronics, vol.
8, no. 8, p. 832, 2019.
[114] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a
neural network," arXiv preprint arXiv:1503.02531, vol. 2, no. 7,
2015.
[115] N. Frosst and G. Hinton, "Distilling a neural network into a soft
decision tree," arXiv preprint arXiv:1711.09784, 2017.
[116] X. Yang, H. Zhang, and J. Cai, "Auto-encoding and distilling scene
graphs for image captioning," IEEE Transactions on Pattern
Analysis and Machine Intelligence, 2020.
[117] X.-H. Li et al., "A survey of data-driven and knowledge-aware
explainable ai," IEEE Transactions on Knowledge and Data
Engineering, vol. 34, no. 1, pp. 29-49, 2020.
[118] H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec, "Faithful and
customizable explanations of black box models," presented at the
Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and
Society, 2019, 2019.
[119] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad,
"Intelligible models for healthcare: Predicting pneumonia risk and
hospital 30-day readmission," presented at the 21th ACM SIGKDD
international conference on knowledge discovery and data mining,
2015.
[120] B. Letham, C. Rudin, T. H. McCormick, and D. Madigan,
"Interpretable classifiers using rules and bayesian analysis: Building
a better stroke prediction model," The Annals of Applied Statistics,
vol. 9, no. 3, pp. 1350-1371, 2015.
[121] Y. Ming, H. Qu, and E. Bertini, "Rulematrix: Visualizing and
understanding classifiers with rules," IEEE transactions on
visualization and computer graphics, vol. 25, no. 1, pp. 342-352,
2018.
[122] C. Xiao, T. Ma, A. B. Dieng, D. M. Blei, and F. Wang,
"Readmission prediction via deep contextual embedding of clinical
concepts," PloS one, vol. 13, no. 4, p. e0195024, 2018.
[123] R. Davoodi and M. H. Moradi, "Mortality prediction in intensive
care units (ICUs) using a deep rule-based fuzzy classifier," Journal
of biomedical informatics, vol. 79, pp. 48-59, 2018.
[124] D. Das, J. Ito, T. Kadowaki, and K. Tsuda, "An interpretable
machine learning model for diagnosis of Alzheimer's disease,"
PeerJ, vol. 7, p. e6543, 2019.
[125] H. Lee, S. T. Kim, and Y. M. Ro, "Generation of multimodal
justification using visual word constraint model for explainable
computer-aided diagnosis," in Interpretability of Machine
Intelligence in Medical Image Computing and Multimodal Learning
for Clinical Decision Support: Springer, 2019, pp. 21-29.
[126] N. Prentzas, A. Nicolaides, E. Kyriacou, A. Kakas, and C. Pattichis,
"Integrating machine learning with symbolic reasoning to build an
explainable ai model for stroke prediction," presented at the 2019
IEEE 19th International Conference on Bioinformatics and
Bioengineering (BIBE), 2019, 2019.
[127] P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis,
"Explainable ai: A review of machine learning interpretability
methods," Entropy, vol. 23, no. 1, p. 18, 2021.
[128] W. Ge, J.-W. Huh, Y. R. Park, J.-H. Lee, Y.-H. Kim, and A. Turchin,
"An Interpretable ICU Mortality Prediction Model Based on
Logistic Regression and Recurrent Neural Networks with LSTM
units," presented at the AMIA Annual Symposium Proceedings,
2018, 2018.

[129] M. Stojadinovic, B. Milicevic, and S. Jankovic, "Improved
predictive performance of prostate biopsy collaborative group risk
calculator when based on automated machine learning," Computers
in Biology and Medicine, vol. 138, p. 104903, 2021.
[130] S. M. Lundberg et al., "From local explanations to global
understanding with explainable AI for trees," Nature machine
intelligence, vol. 2, no. 1, pp. 56-67, 2020.
[131] A. Eck et al., "Interpretation of microbiota-based diagnostics by
explaining individual classifier decisions," BMC bioinformatics,
vol. 18, no. 1, pp. 1-13, 2017.
[132] A. Shrikumar, P. Greenside, and A. Kundaje, "Learning important
features through propagating activation differences," presented at
the International conference on machine learning, 2017, 2017.
[133] J. Zuallaert, F. Godin, M. Kim, A. Soete, Y. Saeys, and W. De Neve,
"SpliceRover: interpretable convolutional neural networks for
improved splice site prediction," Bioinformatics, vol. 34, no. 24, pp.
4180-4188, 2018.
[134] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and
W. Samek, "On pixel-wise explanations for non-linear classifier
decisions by layer-wise relevance propagation," PloS one, vol. 10,
no. 7, p. e0130140, 2015.
[135] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller,
"Striving for simplicity: The all convolutional net," arXiv preprint
arXiv:1412.6806, 2014.
[136] H. Chen, S. Lundberg, and S.-I. Lee, "Explaining models by
propagating Shapley values of local components," in Explainable AI
in Healthcare and Medicine: Springer, 2021, pp. 261-270.
[137] A. Singh, A. R. Mohammed, J. Zelek, and V. Lakshminarayanan,
"Interpretation of deep learning using attributions: application to
ophthalmic diagnosis," presented at the Applications of machine
learning 2020, 2020.
[138] L. Putelli, A. E. Gerevini, A. Lavelli, and I. Serina, "Applying self-
interaction attention for extracting drug-drug interactions,"
presented at the International Conference of the Italian Association
for Artificial Intelligence, 2019, 2019.
[139] D. Mascharka, P. Tran, R. Soklaski, and A. Majumdar,
"Transparency by design: Closing the gap between performance and
interpretability in visual reasoning," presented at the Proceedings of
the IEEE conference on computer vision and pattern recognition,
2018, 2018.
[140] D. A. Kaji et al., "An attention based deep learning model of clinical
events in the intensive care unit," PloS one, vol. 14, no. 2, p.
e0211057, 2019.
[141] B. Shickel, T. J. Loftus, L. Adhikari, T. Ozrazgat-Baslanti, A.
Bihorac, and P. Rashidi, "DeepSOFA: a continuous acuity score for
critically ill patients using clinically interpretable deep learning,"
Scientific reports, vol. 9, no. 1, pp. 1-12, 2019.
[142] H. Hu et al., "DeepHINT: understanding HIV-1 integration via deep
learning with attention," Bioinformatics, vol. 35, no. 10, pp. 1660-
1667, 2019.
[143] J. Zhang, K. Kowsari, J. H. Harrison, J. M. Lobo, and L. E. Barnes,
"Patient2vec: A personalized interpretable deep representation of
the longitudinal electronic health record," IEEE Access, vol. 6, pp.
65333-65346, 2018.
[144] E. Choi, M. T. Bahadori, J. Sun, J. Kulas, A. Schuetz, and W.
Stewart, "Retain: An interpretable predictive model for healthcare
using reverse time attention mechanism," Advances in neural
information processing systems, vol. 29, 2016.
[145] B. C. Kwon et al., "Retainvis: Visual analytics with interpretable
and interactive recurrent neural networks on electronic medical
records," IEEE transactions on visualization and computer
graphics, vol. 25, no. 1, pp. 299-309, 2018.
[146] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,
"Learning deep features for discriminative localization," presented
at the Proceedings of the IEEE conference on computer vision and
pattern recognition, 2016, 2016.
[147] H. Lee et al., "An explainable deep-learning algorithm for the
detection of acute intracranial haemorrhage from small datasets,"
Nature biomedical engineering, vol. 3, no. 3, pp. 173-182, 2019.
[148] P. Rajpurkar et al., "AppendiXNet: Deep learning for diagnosis of
appendicitis from a small dataset of CT exams using video
pretraining," Scientific reports, vol. 10, no. 1, pp. 1-7, 2020.
[149] M. Porumb, S. Stranges, A. Pescapè, and L. Pecchia, "Precision
medicine and artificial intelligence: a pilot study on deep learning
Bharati et al.
for hypoglycemic events detection based on ECG," Scientific
reports, vol. 10, no. 1, pp. 1-16, 2020.
[150] S. Hu et al., "Weakly supervised deep learning for covid-19
infection detection and classification from ct images," IEEE Access,
2020.
[151] Y. Liang, S. Li, C. Yan, M. Li, and C. Jiang, "Explaining the black-
box model: A survey of local interpretation methods for deep neural
networks," Neurocomputing, vol. 419, pp. 168-182, 2021.
[152] L. Pan et al., "Development of prediction models using machine
learning algorithms for girls with suspected central precocious
puberty: retrospective study," JMIR medical informatics, vol. 7, no.
1, p. e11728, 2019.
[153] M. S. Kovalev, L. V. Utkin, and E. M. Kasimov, "SurvLIME: A
method for explaining machine learning survival models,"
Knowledge-Based Systems, vol. 203, p. 106164, 2020.
[154] A. Meldo, L. Utkin, M. Kovalev, and E. Kasimov, "The natural
language explanation algorithms for the lung cancer computer-aided
diagnosis system," Artificial Intelligence in Medicine, vol. 108, p.
101952, 2020.
[155] C. Panigutti, A. Perotti, and D. Pedreschi, "Doctor XAI: an
ontology-based approach to black-box sequential data classification
explanations," presented at the Proceedings of the 2020 conference
on fairness, accountability, and transparency, 2020, 2020.
[156] S. M. Lauritsen et al., "Explainable artificial intelligence model to
predict acute critical illness from electronic health records," Nature
communications, vol. 11, no. 1, pp. 1-11, 2020.
[157] G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R.
Müller, "Layer-wise relevance propagation: an overview,"
Explainable AI: interpreting, explaining and visualizing deep
learning, pp. 193-209, 2019.
[158] M. Bernardini, L. Romeo, P. Misericordia, and E. Frontoni,
"Discovering the type 2 diabetes in electronic health records using
the sparse balanced support vector machine," IEEE Journal of
Biomedical and Health Informatics, vol. 24, no. 1, pp. 235-246,
2019.
[159] S. Ghafouri-Fard, M. Taheri, M. D. Omrani, A. Daaee, H.
Mohammad-Rahimi, and H. Kazazi, "Application of single-
nucleotide polymorphisms in the diagnosis of autism spectrum
disorders: a preliminary study with artificial neural networks,"
Journal of Molecular Neuroscience, vol. 68, no. 4, pp. 515-521,
2019.
[160] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional
networks for biomedical image segmentation," pp. 234-241:
Springer.
[161] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O.
Ronneberger, "3D U-Net: learning dense volumetric segmentation
from sparse annotation," presented at the International conference
on medical image computing and computer-assisted intervention,
2016, 2016.
[162] C. Ulas, G. Tetteh, S. Kaczmarz, C. Preibisch, and B. H. Menze,
"DeepASL: Kinetic model incorporated loss for denoising arterial
spin labeled MRI via deep residual learning," presented at the
International conference on medical image computing and
computer-assisted intervention, 2018, 2018.
[163] Y. Xie, G. Gao, and X. A. Chen, "Outlining the design space of
explainable intelligent systems for medical diagnosis," arXiv
preprint arXiv:1902.06019, 2019.
[164] B. Mittelstadt, C. Russell, and S. Wachter, "Explaining explanations
in AI," presented at the Proceedings of the conference on fairness,
accountability, and transparency, 2019, 2019.
[165] P. Hall, N. Gill, and N. Schmidt, "Proposed guidelines for the
responsible use of explainable machine learning," arXiv preprint
arXiv:1906.03533, 2019.
[166] D. Holliday, S. Wilson, and S. Stumpf, "User trust in intelligent
systems: A journey over time," presented at the Proceedings of the
21st international conference on intelligent user interfaces, 2016,
2016.
[167] M. Sendak et al., "" The human body is a black box" supporting
clinical decision-making with deep learning," presented at the
Proceedings of the 2020 conference on fairness, accountability, and
transparency, 2020, 2020.
[168] Q. V. Liao, M. Singh, Y. Zhang, and R. Bellamy, "Introduction to
explainable AI," presented at the Extended Abstracts of the 2021
CHI Conference on Human Factors in Computing Systems, 2021,
2021.
A Review on Explainable Artificial Intelligence for Healthcare
[169] M. G. Kahn et al., "A harmonized data quality assessment
terminology and framework for the secondary use of electronic
health record data," Egems, vol. 4, no. 1, 2016.
[170] B. A. Goldstein, A. M. Navar, M. J. Pencina, and J. Ioannidis,
"Opportunities and challenges in developing risk prediction models
with electronic health records data: a systematic review," Journal of
the American Medical Informatics Association, vol. 24, no. 1, pp.
198-208, 2017.
[171] V. Vishwarupe, P. M. Joshi, N. Mathias, S. Maheshwari, S.
Mhaisalkar, V. Pawar, “Explainable AI and Interpretable Machine
Learning: A Case Study in Perspective,” Procedia Computer
Science, Volume 204, 2022, Pages 869-876.
[172] T. Miller, Explanation in Artificial Intelligence: Insights from the
Social Sciences. Artif. Intell. 2019, 267, 1–38.
[173] A. Adadi, M. Berrada, "Peeking Inside the Black-Box: A Survey on
Explainable Artificial Intelligence (XAI)," IEEE Access 2018, 6,
52138–52160.
[174] L.H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, L. Kagal,
"Explaining Explanations: An Overview of Interpretability of
Machine Learning," In Proceedings of the 2018 IEEE 5th
International Conference on Data Science and Advanced Analytics
(DSAA), Turin, Italy, 1–4 October 2018; pp. 80–89.
[175] M. Böhle, et al., "Layer-wise relevance propagation for explaining
deep neural network decisions in MRI-based Alzheimer's disease
classification," Frontiers in aging neuroscience 11 (2019): 194.
[176] T. Araújo et al., "DR| GRADUATE: Uncertainty-aware deep learning-
based diabetic retinopathy grading in eye fundus images," Medical
Image Analysis 63 (2020): 101715.
[177] L. Li et al., "A large-scale database and a CNN model for attention-
based glaucoma detection," IEEE transactions on medical imaging
39.2 (2019): 413-424.
[178] E. Pesce et al., "Learning to detect chest radiographs containing
pulmonary lesions using visual attention networks," Medical image
analysis 53 (2019): 26-38.
[179] S. Rajaraman et al., "Visualizing and explaining deep learning
predictions for pneumonia detection in pediatric chest radiographs,"
Medical Imaging 2019: Computer-Aided Diagnosis. Vol. 10950.
SPIE, 2019.
[180] J. Silva-Rodríguez et al., "Going deeper through the Gleason scoring
scale: An automatic end-to-end system for histology prostate
grading and cribriform pattern detection," Computer Methods and
Programs in Biomedicine 195 (2020): 105637.
[181] Z. Bian et al., "Weakly supervised Vitiligo segmentation in skin
image through saliency propagation," 2019 IEEE International
Conference on Bioinformatics and Biomedicine (BIBM). IEEE,
2019.
[182] K. Wickstrøm, M. Kampffmeyer, and R. Jenssen, "Uncertainty and
interpretability in convolutional neural networks for semantic
AUTHORS’ BIOGRAPHIES
Subrato Bharati was with the Institute of
Information and Communication Technology,
BUET, Bangladesh, and now with the Department
of Electrical and Computer Engineering, Concordia
University, Montreal, QC, Canada. He authored/co-
authored over 50 Journal articles, Conference
Proceedings, and Book Chapters published by IEEE, Elsevier,
Springer, and others. He is an Associate Editor of Journal of the
International Academy for Case Studies and a Guest Editor of a special
issue in Journal of Internet Technology. His research interests include
health informatics, medical imaging, digital signal and image
processing, eXplainable AI, machine and deep learning.
M. Rubaiyat Hossain Mondal obtained the PhD
degree in 2014 from the Department of Electrical
and Computer Systems Engineering, Monash
University, Melbourne, Australia. From 2005 to
2010, and from 2014 to date, he has been working
15
segmentation of colorectal polyps," Medical image analysis 60
(2020): 101619.
[183] J. R. Clough et al., "Global and local interpretability for cardiac MRI
classification," International Conference on Medical Image
Computing and Computer-Assisted Intervention. Springer, Cham,
2019.
[184] M. Graziani et al., "Concept attribution: Explaining CNN decisions
to physicians," Computers in biology and medicine 123 (2020):
103865.
[185] C.-T. Kor, Y.-R. Li, P.-R. Lin, S.-H. Lin, B.-Y. Wang, and C.-H. Lin,
“Explainable Machine Learning Model for Predicting First-Time
Acute Exacerbation in Patients with Chronic Obstructive Pulmonary
Disease,” Journal of Personalized Medicine, vol. 12, no. 2, p. 228,
Feb. 2022, doi: 10.3390/jpm12020228.
[186] Y. Liu, Z. Liu, X. Luo, H. Zhao, "Diagnosis of Parkinson's disease
based on SHAP value feature selection," Biocybernetics and
Biomedical Engineering, Volume 42, Issue 3, 2022, Pages 856-869.
[187] C. Wei, L. Zhang, Y. Feng et al., "Machine learning model for
predicting acute kidney injury progression in critically ill patients,"
BMC Med Inform Decis Mak 22, 17 (2022).
[188] K. Debjit et al., “An Improved Machine-Learning Approach for
COVID-19 Prediction Using Harris Hawks Optimization and
Feature Analysis Using SHAP,” Diagnostics, vol. 12, no. 5, p. 1023,
Apr. 2022.
[189] Y. Zoabi, , S. Deri-Rozov, N. Shomron, "Machine learning-based
prediction of COVID-19 diagnosis based on symptoms," npj Digit.
Med. 4, 3 (2021).
[190] S. Ali, Y. Zhou, M. Patterson, "Efficient analysis of COVID-19
clinical data using machine learning models," Med Biol Eng Comput
60, 1881–1896 (2022).
[191] J. H. Ong, K. M. Goh and L. L. Lim, "Comparative Analysis of
Explainable Artificial Intelligence for COVID-19 Diagnosis on
CXR Image," 2021 IEEE International Conference on Signal and
Image Processing Applications, 2021, pp. 185-190.
[192] D. Chowdhury, S. Poddar, S. Banarjee et al., "CovidXAI: explainable
AI assisted web application for COVID-19 vaccine prioritization,"
Internet Technology Letters, 2022; 5(4):e381.
[193] L. V. De Moura et al., "Explainable Machine Learning for COVID-
19 Pneumonia Classification With Texture-Based Features
Extraction in Chest Radiography," Frontiers in digital health 3
(2021).
[194] B. Alsinglawi, O .Alshari, M. Alorjani et al., "An explainable machine
learning framework for lung cancer hospital length of stay
prediction," Sci Rep 12, 607 (2022).
[195] A. F. Markus, J. A. Kors, and P. R. Rijnbeek, "The role of explainability
in creating trustworthy artificial intelligence for health care: a
comprehensive survey of the terminology, design choices, and
evaluation strategies," Journal of Biomedical Informatics, vol. 113,
2021.
as a Faculty Member at the Institute of Information and
Communication Technology (IICT) in BUET, Bangladesh. He has
published a number of papers in journals, conferences and book
chapters, and edited several books published by reputed publishers.
His research interests include artificial intelligence, image processing,
bioinformatics, wireless communications, and cryptography.
Prajoy Podder is currently a Researcher at the
Institute of Information and Communication
Technology, Bangladesh University of Engineering
and Technology. He authored/co-authored over 45
Journal articles, Conference Proceedings, and Book
Chapters published by IEEE, Elsevier, Springer, and
others. His research interests include Digital Image Processing, Data
Mining, IoT, Machine Learning, and Health Informatics.