*&&&
UI
*OUFSOBUJPOBM
$POGFSFODF
PO
$PMMBCPSBUJPO
BOE
*OUFSOFU
$PNQVUJOH

$*$
What Information is Required for Explainable AI? :
A Provenance-based Research Agenda and Future
Challenges
2020 IEEE 6th International Conference on Collaboration and Internet Computing (CIC) | 978-1-7281-4146-6/20/$31.00 ©2020 IEEE | DOI: 10.1109/CIC50333.2020.00030
Fariha Tasmin Jaigirdar
Dept. of Software Systems and Cybersecurity
Monash University
Melbourne, Australia
fariha.jaigirdar@monash.edu
Gillian Oliver David Watts
Dept. of Human Centred Computing La Trobe Law School
Monash University La Trobe University
Melbourne, Australia Melbourne, Australia
Gillian.Oliver@monash.edu D.Watts@latrobe.edu.au
Abstract—Deriving explanations of an Artificial Intelligence-
based system’s decision making is becoming increasingly essen-
tial to address requirements that meet quality standards and
operate in a transparent, comprehensive, understandable, and
explainable manner. Furthermore, more security issues as well
as concerns from human perspectives emerge in describing the
explainability properties of AI. A full system view is required
to enable humans to properly estimate risks when dealing with
such systems. This paper introduces open issues in this research
area to present the overall picture of explainability and the
required information needed for the explanation to make a
decision-oriented AI system transparent to humans. It illustrates
the potential contribution of proper provenance data to AI-
based systems by describing a provenance graph-based design.
This paper proposes a six-Ws framework to demonstrate how
a security-aware provenance graph-based design can build the
basis for providing end-users with sufficient meta-information
on AI-based decision systems. An example scenario is then
presented that highlights the required information for better
explainability both from human and security-aware aspects.
Finally, associated challenges are discussed to provoke further
research and commentary.
Index Terms—artificial intelligence, data provenance, explain-
able AI, decision-oriented systems, cybersecurity, human-centric
policy
I. I NTRODUCTION
Artificial Intelligence (AI) has become a topic of central
importance because of the ways in which it impacts on various
sectors, including transportation, finance, legal, military, and
medicine [1]. With machine learning (ML) and deep learn-
ing (DL) algorithms, AI provides the core technology for
autonomous decision-making platforms, from mission-critical
services (e.g., autonomous driving) to life-critical tasks (e.g.,
remote healthcare monitoring, criminal justice). All of these
have a direct influence on different areas in people’s lives and

¥
*&&& 
%0*
$*$
Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
Carsten Rudolph
Dept. of Software Systems and Cybersecurity
Monash University
Melbourne, Australia
carsten.rudolph@monash.edu
Chris Bain
Monash University Digital Health
Monash University
Melbourne, Australia
chris.a.bain@monash.edu
often induce human decisions. Thus, AI has a considerable
impact on peoples’ everyday lives.
That said, there is a growing body of research that highlights
problems associated with AI-based decision-making. These
arise in particular with ‘second wave’ AI [2] that is based
on statistical learning and probability theory where statistical
models are created for solving specific problem domains.
These are ‘trained’ on big data and analyzed using machine
learning and deep neural networks to produce highly nuanced
classification and prediction capabilities. Almost invariably,
this interplay of data and algorithmic-based analysis occurs
within what has been referred to as a ‘black box’, where the
reasons why a decision or recommendation is made, or an
action is taken are opaque and unknown. Further, the relation
between input data, training data, and resulting classifications,
as well as the provenance of these various inputs, are not
obvious to the human user.
One core issue is that autonomous decision-making systems
follow post-hoc methods, where the roots of included artifacts
are not made explicit. Without documenting the detailed
connection of the training data-set with the decision gener-
ated, the overall system’s acceptability is questionable. Thus,
the autonomous recommendations require strong ‘explainable
properties’ at each step of processing. Without this explana-
tion, the training data-set used and the overall system may face
several vulnerabilities. For example, a woman walking across
a road in Arizona on March 18, 2018, was struck and killed
by a self-driven and it showed to be difficult to identify the
cause of the accident [3]. Moreover, a recent report highlights
various incidents of misunderstandings for unclear sourcing of
data, particularly when one is the CEO, or an owner, or even
an operator of a decision-making software/system platform [4].
Therefore, AI-based decision-support systems [5] and their in-
 depth explanation of knowledge is vital. Even if the overall
system works perfectly, it is essential to know the very root
of performing the decision, especially when the decision or
prediction is crucial [6] [7]. In this paper, we adopt a human-
centric perspective, looking at the data used for decision
generation in an AI-based system. We identify and discuss
four open issues (OI) in this paper. The first open issue focuses
on developing a better understanding of requirements from a
human perspective.
OI1. What are the necessary requirements for explainability
that need to be included in an AI-based decision support
system? What kind of information would an end-user
need to get a proper idea of the system’s behavior in
a decision support system?
Interestingly, even if we were to know the ‘explainability’ of
the result in a system, there are still some remaining concerns.
For example, deliberate bias might be built into a system
that influences the result. Training data has been shown to
affect AI outcomes significantly. This can be unintentional
(e.g., the bias introduced or accuracy depending on different
factors) or intentional (e.g., introducing back-doors or favoring
particular decisions). Debugging or actual reasoning in AI
based on statistical processing or probabilistic analysis is
very difficult or nearly impossible in a classical sense [8].
Furthermore, adaptive systems continue learning, and hence,
they can also change behavior. Therefore, adding explanations
in a system does not guarantee that the explanation is reliable,
as there are other factors that can change the explanation,
and a deliberate or intentional bias or system fault is always
possible. There are repeated examples [9] of the problematic
nature of this type of black-box decision-making that points
to inaccurate and unsafe decision-making as well as instances
of bias, discrimination, lack of due process, and breaches of
human rights. Moreover, while expecting ‘explanations’, it is
possible that the ‘black-box’ nature of an AI system may
create an illogical explanation without proper documentation.
Therefore, if the system could document the security evidence
on the security controls involved in a system, it could have an
overview of the overall system security/safety specifications.
Thus, with these discussions, we sketch our second and third
open questions as follows.
OI2. Has the end-user any means to check that the generated
decision is correct, authentic from the right ML or
DL algorithms, and all processing steps have been as
expected?
OI3. How to add security evidence in an AI-based system,
where approaches to detect or prevent ‘misrepresen-
tation’, ‘deliberate bias’, ‘safety-mismatch’, or ‘back-
doors’ are yet to discover?
Policy and lawmakers are increasingly focusing on how to
best address potential harm induced by AI-based systems. One
response is the EU General Data Protection Regulation’s [10]
right to an explanation of automated decision-making. It
requires that individuals affected by AI-based decisions be
provided with ‘meaningful information about the logic in-

Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
volved and the significance and envisaged consequences’ of
the automated decision-making (see Articles 13, 21, and 22).
Other responses take narrower, sector-specific approaches. For
example, safety and quality requirements for autonomous ve-
hicles, medical devices, or civil aviation. Whichever approach
is taken, the key problem is to identify suitable, necessary,
desirable and practicable steps to open the AI black box in
a way that humans can digest and understand. Therefore, the
fourth open issue adds an ethical and legal perspective on
human rights for critical decision making.
OI4. How should requirements be added in a critical decision-
based system from an ethical and policy sectors so that
policies are checked appropriately?
Data provenance documents and explains the data stream
by answering ‘who-what-when-where’ of each step of data
propagation [11] [12]. It identifies the data origin as well as
the full data transmission procedure and helps in determining
the data trustworthiness by portraying data transparency. In
this paper, we discuss the four open research issues from a
user-centric provenance-based security perspective. We discuss
the role of implementing provenance graphs for explaining
AI properties so that information from data origin to data
modification via data processing can be documented in an
AI-based decision system. We emphasize the need to add
security and legal information in a provenance graph to achieve
transparency and comprehensibility of data propagation in
the system. Our specific contributions can be summarised as
follows.
1) We discuss the roles of adding information to a layer-
wise data provenance graph in explaining AI in a
system-wide view.
2) We propose a ‘Six Ws’ structure to present the infor-
mation required for a security-aware data provenance
graph.
3) We propose to include evidence of explanation of AI
system from both the human and security point of view,
including evidence for secure data propagation, and
highlight the required information/metadata to be added
in the provenance graphs.
The rest of the paper is organized as follows. Section II
demonstrates necessary evidence for explanation both from
human and security point of view; The role of provenance
information in decision-oriented AI system with the proposed
framework is described in Section III; Section IV illustrates an
AI-based loan processing scenario and discusses the required
information for explanation; Challenges and future research
directions are included in Section V, and finally, the paper
concludes in Section VI.
II. E VIDENCE FOR E XPLANATION
In this section, we discuss the evidence (information) re-
quired for decision-based AI system’s explainability from
human perspectives and security points of view.
A. The Human Point of View provenance graph so that a user-centric view can be generated
at the end to show relevant risks (if any).
AI brings various critical sectors under the umbrella of tech-
nological sophistication. For example, criminal justice, foren-
sic analysis, clinical health research, and remote healthcare III. T HE ROLE OF P ROVENANCE I NFORMATION
monitoring [13] [14]. With the immense growth of AI in these
A standard for data provenance is presented in the PROV-
dynamic sectors, certain risk factors, including maintaining
DM model [12]. It provides the history of data records by
human values and human rights, also come under consid-
documenting a set of metadata that describes agents, activities,
erations [15] [16]. The challenge is that machine learning
and entities involved in creating, manipulating, and delivering
operations are often not transparent even for the researchers
a piece of data, a document, and/or an automated decision in a
involved in system design [15]. While it may seem not very
data provenance graph. Certain relationships among different
problematic in some areas of applied machine learning, for
attributes of a provenance graph are also added in the graph to
the critical cases of judicial settings, or healthcare analysis,
illustrate their inter-connections. The entities can be physical
evidence for the transparency of the reasoning (from human-
or conceptual things, which can be digital or virtual and are
centric perspectives), explainability of decision operations
usually identified by a unique ID. Activities can be described
(process-explanation perspective), and documentation of secu-
as generating, processing, using, or transforming entities and
rity evidence (security-perspective) are significant. Therefore,
are related to time involved for that action. Documentation of
when discussing and referring to the information necessary
activities includes the start time and end time of an activity and
for explainable AI (XAI), documenting evidence relating to
is identified by a unique ID. Agents are related to activities,
human-centric values and human rights is vital.
entities, and other agents according to their responsibilities.
To illustrate further, while discussing implementing algo-
Provenance linking the input and output of AI systems
rithms in the ‘fair trial’ in a judicial decision, the present
is commonly identified as an important issue, as analyzing
articles raise questions of not following law/policy. Article 6
provenance data is a valuable source for deriving the explana-
of the European Convention on Human Rights [17] guarantees
tions about decisions/recommendations made by algorithmic
the accused the right to participate effectively in the trial and
systems. In these systems, the resulting data, as seen by the
include the presumption of innocence. Thus, implementing AI-
end-user (e.g., a medical doctor using a decision support
based decision procedures in such systems provokes serious
system or a lawyer working with the metadata for any forensic
concerns, and the procedures need a proper justification on
analysis or criminal verdict), is derived from a classification
whether this criterion is maintained or not in the overall
resulting from the AI system. Hence, provenance metadata
explainability of the system procedure. Researchers are con-
plays a vital role in describing ‘explainable’ properties of AI-
cerned about not maintaining the law/policy from human-
based decision oriented systems. Although some researchers
centric perspectives in these critical domains and identify
have highlighted provenance as an emerging research area
that the ‘black-box’ nature of statistical analysis of AI-based
to explain AI-based systems (see [19] [20] [4]), no details
systems makes the system even more complex in terms of
or research road map are identified to present the required
transparency. However, no research has indicated ‘adding and
information for a transparent and comprehensive XAI system.
checking these policies’ as explainability properties. There-
An important consideration while considering ‘provenance’ as
fore, in this paper, we focus on adding information regarding
a measure of ‘explainable AI’ is not only to document the
governance and policy in different applications. While repre-
metadata involved, but to confirm that the metadata collected
senting a data provenance for information flow, we propose
is authentic and comes from an authorized source. A user’s
to add it as a required information of explaining AI-based
trust and confidence in an interpretable learning system is a
decision systems.
function of the user’s capability to understand the machine’s
input/output mapping behavior, which is broadly dependent
B. The Security Point of View
on the AI system to be ‘explainable’ [21]. Therefore, the
One noteworthy point about investigating ways for ‘expla- information supporting explainability also needs to be ex-
nation’ in an AI-based system is that generated explanations tended with security information about the security controls
can be unreliable and misleading. Deliberate manipulation of in the system to enable users to analyze the risks. While some
data sets for specific motivation in different applications of AI- existing provenance models [12] [22] describe complete data
based systems is not new [18]. Moreover, the training data and transformations by semantic relations among entities in data
even the model used to draw a prediction in a decision-based histories, using provenance in AI and detailed descriptions
system can be manipulated. Currently, there is no way to find including the source of ‘security metadata’ or ‘legal metadata’
out whether there is any ‘deliberate’ or ‘intentional’ bias in to be used to generate clarifications about automated decisions
an AI black-box, and any ‘unexpected’ result is the outcome that affect users are yet to be developed. Thus, approaching
of that bias or not. Furthermore, as the AI’s development ‘Explainable AI’, i.e., documenting and understanding how
depends on input data, the security properties guaranteed for the output is derived from the input and the system used in
this data need to be included in the evaluation. In this paper, between, is an essential consideration and needs to be covered
we propose to add ‘security-related information’ in a data in the provenance graph [11].



Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
 Which security/safety information is added
for explanation?
Which?
(security • Security protocol
• System specification
metadata)
• Security controls on system configuration,
authentication mechanisms
Who?
(agent)
Who is responsible for the data
Provenance Graph based
generation/modification/explanation?
• Users/actors/auditors involved XAI
• Machine/automated system
Where?
(activity)
When/Where is the explanation given?
• System overflow
• Overall process
When?
(activity)
• Design/modification/implementation
Fig. 1. The Six Ws of Provenance-based Explainable AI (XAI)
A. Proposed ‘Six Ws’ Framework for Provenance Graph
based XAI Design
In this section, we propose a ‘Six Ws’ structure (see
Fig. 1), which represents the necessary attributes to present
provenance-based explainable AI properties. While the PROV-
DM model illustrates the basic definition that can be repre-
sented as ‘who-what-when-where’ in a data provenance graph,
we propose to use it to identify the information required for
the explainable properties of AI along with two new ‘Ws’:
‘Which’ and ‘Why’.
We consider decision-oriented AI systems for this paper,
which are generally based on training data [18]. The prove-
nance graph of the system should document the information
needed to understand the overall system so that an end-user at
any point in the system can get the overall idea and estimate
risks of the system. The six Ws are shown in Fig 1. Firstly,
‘who’ represents an ‘agent’ in the PROV-DM model, where
this agent can be an actor, user, machine involved in the
decision-based system. Thus, this agent is responsible for data
generation, modification, or action involved in an appropriate
explanation. Secondly, ‘what’ represents ‘entity’ in the PROV-
DM model to delineate the data and/or document included in
the system, which generates the ‘decision’ after certain manip-
ulation in different layers in training based AI system. Thirdly,
‘when’ and ‘where’ represent specific dimensions of the
‘activity’ used in the PROV-DM model to illustrate the work-
flow involved at a layer or design/modification/implementation
involved in the overall process.
We argue that providing information only on these factors
(who-what-when-where) in a data provenance graph is not
sufficient as the black box and the overall transmission of
data in an AI-based system may have certain security holes or
system limitation in between. Therefore, these security holes
may have different impacts according to various applications.
A provenance graph should include information on the security
of all layers in a decision-support system from the source of
the data via training to decision-making. This training data
explanation is essential in order to stop blindly believing in
Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
black box or hidden-layer decision generation of a machine
learning algorithm. Moreover, a detailed analysis of the com-
plete system, including all internal components, is essential
in order to understand the appropriate security controls (if
What is being processed/
What? explained? any) placed in the overall system for detecting and mitigating
(entity) • Data/document/
decision system vulnerabilities. Hence, as a vital attribute of describing
• System information
the ‘second wave’ of AI, we propose to include ‘which’
security information is necessary to add in the provenance
graph to represent the security controls involved in an AI-
Why?
(law/policy) based decision system. This ‘security metadata’ may represent
specific system specifications, security protocols used for
Why is the explanation
generated?
data propagation, security controls on system configuration,
• Legislation/policy
• Governance
authentication mechanisms, safety measures equipped by the
• Human-centric
system, etc. So, ‘which’ provides security evidence in the
provenance graph involved in a decision-oriented AI system.
This inclusion in a provenance graph helps a user to get an
overview of the system and estimate risks/system fault/status.
The big open issue for collecting information in which security
and explainability information are available is on the side of
AI. While we have established best-practice for other security
controls, the AI itself’s security and explainability are work
in progress. Examples for options currently available are as
follows.
• Collect sufficient information for auditing the AI’s be-
havior to ensure a lack of bias or backdoors. The result
of audits and some guarantees can be included with
provenance information.
• Document which features are actually used in the AI and
remove unwanted features (e.g., gender) from the input
data before processing.
• Use and document the use of privacy-enhancing technolo-
gies, such as differential privacy.
The last ‘w’ in the proposed structure is the ‘why’ that
represents the mandate for, or reasoning of, the explainability
from a law/policy perspective in an AI system. It can be de-
signed for specific applications so that the related legislation or
policy can be reported in the provenance graph. This attribute
enables the option to design a system from a domain-specific
and/ or person-centered perspective of why this explanation
is generated,and whether the specific policy or governance
requirements have been adhered to when generating a decision.
This ‘why’ attribute aims to empower users against any
undesired design of a black-box automated decision-oriented
system, which may violate their ethical rights and freedom.
B. Provenance Graph-based XAI properties in Decision-
oriented AI system
In this section, we present how our proposed structure of
‘six Ws’ is relevant to a currently operational paradigm of an
AI-based decision support system and big data analysis.
Fig. 2 shows different steps for big data analysis in AI,
where each step demonstrates data generation to visualization
via processing steps. The data is generated from an actor in
a decision-based AI system and is visualized by an end-user
after applying the ML algorithm. Actors can be any source
for data generation, and end-users can be a machine or a

person working on visualized data or decision generated. If
we try to plot the provenance graph concept in a decision
support system, we can see that the actors and users in the
system are the ‘agents’ in the graph. The workflows involved
in the overall processing of the system represent the steps of
various ‘activities’ , Datax and Datay represent ‘entities’ at
different stages with different data representations. Step one
and two in Fig. 2 indicate that the actor involved in the system
starts working with the initial data. The relevant provenance
questions are introduced in the figure to indicate the open is-
sues by the authors. These questions highlight the significance
of including the ‘which’ and ‘why’ in the provenance graph.
For example, ‘Is the exact representation of data reported?’,
or ‘Is the process transparency reported?’, or ‘Is the exact
prediction mechanism documented?’ - can be addressed from
documenting ‘security evidence’ or ‘security metadata’ for
the certain layers. Moreover, the question ‘Is the appropriate
law/policy maintained?’ can be answered by reporting whether
the appropriate law/policy has been followed from human-
centric perspectives.
Step three in Fig. 2 constitutes a ML algorithm, which
does not reveal the hidden layer processing. Moreover, deep
learning algorithms or neural network designs on decision-
making or recommendation generation act as a black-box,
where one may include the service to know where the AI is
running on, but unable to keep the knowledge from where
the neural network originates and propagates. The fact is,
provenance is usually exploited in a relatively coarse-grained
manner, in which whole algorithms or data transformations
are just described by semantic relations. As a result, with the
above discussion, whole pipelines may be documented with
provenance, but individual algorithms remain black boxes. We
need the details about the system used, how it was trained,
and additional information required for the system to make
the explanation clear both from human and security points of
view. Therefore, although we may generate provenance based
explanations, back-door attacks, or deliberate or misrepresent-
ing ‘bias’ of training dataset is hard to determine.
IV. A N E XAMPLE S CENARIO AND R EQUIRED
I NFORMATION FOR E XPLANATION
This section discusses a bank loan processing scenario and
designs its associated provenance graph to demonstrate the
essential information for an explanation. AI support has been
available for these loan processing for quite some time [23],
and the need for explainable AI has been identified in this
context. Very first solutions providing some explanation have
been suggested in [24]. Although this paper provides some
explanation, it does not link the explanation to any input data.
Therefore, it does not deliver the information to humans to
fully backtrack the decision to the input data and verify both.
Let us consider a hypothetical scenario of an AI-based
decision-oriented loan processing system, where a female
customer of a bank requests a loan to buy a new car. A
provenance graph is able to track the connection between
application data (input) and decision (prediction outcome) by

Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
getting information from each processing (activities) steps
with data (entity) and from agents involved. Fig. 3 illus-
trates related agents, activities, and entities in the provenance
graph. The meanings of the shapes used are indicated in the
lower-left corner of the figure. The graph also indicates the
relationships among different attributes (entity, activity, and
agent). For ease of representation we indicate the relevant
relationships [12] as WasGeneratedBy (WGeB), WasAssociat-
edWith (WAsW), WasInformedBy (WInB), and WasDerived-
From (WDeF). While details of agents, activities, and entities
involved in the graph delineate the explanation on ‘who-what-
when-where’ of the system, we include ‘security controls’, and
‘legal controls’ to document the security-related information
and law/policy related information, respectively. Therefore, it
creates the opportunity of adding ‘which’ and ‘why’ related in-
formation for security-aware and human-centric explanations.
Fig. 3 illustrates the internal attributes involved in a loan
processing provenance graph. After the customer requests
for a loan, the ‘loan processing’ activity evaluates the loan
application according to some machine learning model and
based on the training data of the system., The result is a
recommendation to approve or reject based on the risk of
the loan to default. We assume that the loan is not approved
as the system model is designed in a way that it does not
sufficiently value financial benefits to a female customer, while
the underlying assumption in the system model is ‘female =
less creditworthy’. From the providers design view and purely
mathematical risk estimation, such a bias might be logical. For
example, if data (used for training the AI) showed that women
had a record of lower income and thus pose higher chances
of financial instability. Therefore, the system would reject
loans to women with a higher probability. Such a behavior
is currently not transparent for an individual applicant. With
a provenance based explanation, it would be possible to get
the justification of the decision. In particular, it would link
input data parameters to the decisions. Combined with XAI
data, such as statistics on previous decisions and the ability
to audit, there would be higher transparency on the causes
of the decisions. Therefore, from a human-centric view, one
could actually analyze the results and conclude ‘the logic’ is
improper and unacceptable. Such an explanation should be
added to justify the decision to the customer, as the decisive
factor should actually be ‘income’, not ‘gender’. Another
angle to be considered is security. If a data forgery attack
is generated by an adversary between the loan processing
steps, the decision could be changed, and the user would
get a skewed view of the AI’s behavior. Therefore, if we
do not document the security information for the complete
system, starting from the input data through the different layers
of the system, there would be no possibility of determining
whether there has been any ‘deliberate bias’, ‘system fault’,
or manipulations.
V. C HALLENGES AND F UTURE R ESEARCH D IRECTIONS
Working with data provenance generates several challenges
as the provenance graph, and the provenance records are
 Step 1
Gathering data from Actor
various sources (Datax)

Workflow(s)
Step 2
Cleaning data (Is the exact representation
of data reported?)
Big Data Analysis in AI

Workflow(s)
Step 3 (Is the process transparency
Model building-selecting

Process
reported?)
the right ML algorithm (Is the appropriate law/policy
maintained?)
Workflow(s) Input Output
Step 4 (Is the exact prediction mechanism layer layer
Getting insight/AI documented?)
prediction Hidden
(Is the appropriate law/policy
layer
maintained?)

End-user Machine Learning Algorithm

Step 5
Data visualization (Datay)

Fig. 2. Data provenance based explanation schema in a decision-oriented AI system

generating any decision. While in communication architecture,

Bank, B
WAsW we know what protocols to document for further validation
Security controls
of communications, in the case of an AI-based system, it is
Security controls Requesting Bank Loan
BID, Cinfo, unknown. Therefore, a crucial challenge is where to include
WGeB CID, Datax
and document the relevant security metadata.
Customer, C Another challenge of making a fully designed data
WAsW WInB WDeF provenance-based system is that it should produce inter-
pretable explanations by the target community of users. This
Bank, B
WAsW
Loan Processing
WGeB
Datax
target community may differ for different applications, and the
system administrator must design specific requirements.
Security controls Security controls Since we are also focusing on human-centric values of
explanation in a provenance graph, considerations should be
entity designed from the human and policy levels. Social scientists,
Training Data Model
activity psychologists, policymakers, and lawyers should determine
agent how the practice needs to be imposed for better efficiency
security Legal controls among different applications/scenarios. The fact is, working
evidence
with these diverse groups may introduce further challenges.
Law/policy
For example, the information essential from a lawyer’s per-
Prediction
Used shapes spective may affect certain human-rights described by a so-
cial scientist. A plan for recognizing and reconciling these
End-user
WAsW
Approval outcome
WGeB Datay, diverse agendas is vital. Therefore, this work reveals an inter-
yes/no
disciplinary research opportunity and creates prospects for
industry and academic collaboration.
Fig. 3. An Example Provenance Graph representing an AI-based Loan
Processing System Provenance data in any framework should include the
granularity policy [26] so that users can have sufficient
documentation at any point. As a result, firstly, the system can
restrict/limit the use of bias/poor training set. Secondly, with
themselves vulnerable to various security threats. Hence, main- proper documentation of security metadata, it can estimate the
taining a secured record of data provenance is an important risk (if any). However, if AI does not know what information
issue [25] [11]. We are working on a comprehensive data to include and where, there are always possibilities of miscon-
provenance model and introducing the concept of adding secu- duct.
rity and legal-related metadata in each step of data processing. If we get all the relevant explanations for an AI-based
Such metadata can include evidence on active security con- system, the next challenge is how we can present it to an
trols, various authorization and/or authentication mechanisms end-user to get an overall view of the system. Therefore,
for users or software running on a device, legal perspectives of



Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
visualization is another issue to explore the implementation
of a provenance-based explainable AI system. Researchers
need to maintain a trade-off between retaining any required
‘secrecy’ and ‘transparency’ to the target group of users to
maintain security and trust and useful data privacy.
VI. C ONCLUSION
Several researchers have highlighted AI’s vulnerabilities
associated with its development processes, or the chain of
reasoning in AI-based systems. Hence, maintaining the trans-
parency and explainability of AI-based systems, particularly
in the second wave of AI, is crucial. Solutions to also add
meaningful security and legal metadata are required for AI-
based systems in any critical scenarios. This paper introduces
four open issues and discusses possible contributions of prove-
nance graphs in big data analysis and AI interpretability both
from a security-aware and human-centric view. In this paper,
we first demonstrate how the basic design of a provenance
graph can help describe the overall system. Afterward, we
propose a framework representing the six necessary elements
of information to be added in a provenance graph. This frame-
work presents the information needed for the explainability
and transparency of an overall AI-based decision system. Our
research identifies some essential points for the future design
of explainable AI. Finally, we identify specific challenges
in this area that would provoke interesting future research
directions.
Acknowledgement
Special thanks to the ICT Division, Ministry of Posts,
Telecommunication and IT, Peoples Republic of Bangladesh,
for the student research fellowship.
R EFERENCES
[1] S. A. Bini, “Artificial intelligence, machine learning, deep learning, and
cognitive computing: what do these terms mean and how will they
impact health care?” The Journal of arthroplasty, vol. 33, no. 8, pp.
2358–2361, 2018.
[2] G. Hurlburt, “How much to trust artificial intelligence?” IT Professional,
vol. 19, no. 4, pp. 7–11, 2017.
[3] “Autonomous vehicle,” accessed: 2020-02-04. [Online]. Avail-
able: https://www.nytimes.com/interactive/2018/03/20/us/self-driving-
uber-pedestrian-killed.html
[4] L. Frost, “Explainable AI and other questions where provenance
matters,” IEEE IoT Newsletter: https://iot.ieee.org/newsletter/january-
2019/explainable-ai-and-other-questions-where-provenance-matters, ac-
cessed: 2020-03-04.
[5] A. S. Keshavarz, T. D. Huynh, and L. Moreau, “Provenance for
online decision making,” in International Provenance and Annotation
Workshop. Springer, 2014, pp. 44–55.
[6] P. Buneman and W.-C. Tan, “Data provenance: What next?” ACM
SIGMOD Record, vol. 47, no. 3, pp. 5–16, 2019.
[7] J. Shaw, F. Rudzicz, T. Jamieson, and A. Goldfarb, “Artificial intelli-
gence and the implementation challenge,” J Med Internet Res, vol. 21,
no. 7, p. e13659, Jul 2019.
[8] A. Preece, “Asking ‘why’in ai: Explainability of intelligent systems–
perspectives and challenges,” Intelligent Systems in Accounting, Finance
and Management, vol. 25, no. 2, pp. 63–72, 2018.
[9] D. Castelvecchi, “Can we open the black box of ai?” Nature News, vol.
538, no. 7623, p. 20, 2016.
[10] P. Voigt and A. Von dem Bussche, “The EU general data protection
regulation (GDPR),” A Practical Guide, 1st Ed., Cham: Springer Inter-
national Publishing, 2017.

Authorized licensed use limited to: Carleton University. Downloaded on June 14,2021 at 16:48:21 UTC from IEEE Xplore. Restrictions apply.
[11] F. T. Jaigirdar, C. Rudolph, and C. Bain, “Can I Trust the Data
I See?” in Proceedings of the Australasian Computer Science Week
Multiconference. Sydney, NSW, Australia: ACM, 2019, pp. 1–10.
[12] L. Moreau and P. Missier, “PROV-DM: The prov data model. W3C
recommendation,” World Wide Web Consortium, 2013.
[13] C. Rigano, “Using artificial intelligence to address criminal justice
needs,” National Institute of Justice. Issue, no. 280, 2019.
[14] C. Kuziemsky, A. J. Maeder, O. John, S. B. Gogia, A. Basu, S. Meher,
and M. Ito, “Role of artificial intelligence within the telehealth domain:
Official 2019 yearbook contribution by the members of imia telehealth
working group,” Yearbook of medical informatics, vol. 28, no. 1, p. 35,
2019.
[15] A. Završnik, “Criminal justice, artificial intelligence systems, and human
rights,” in ERA Forum, vol. 20, no. 4. Springer, 2020, pp. 567–583.
[16] M. Latonero, “Governing artificial intelligence: Upholding human rights
& dignity,” Data & Society, pp. 1–37, 2018.
[17] R. Crawshaw and L. Holmström, “7. convention for the protection of
human rights and fundamental freedoms (european convention on human
rights),” in Essential Texts on Human Rights for the Police, 2008, pp.
323–340.
[18] D. Pedreschi, F. Giannotti, R. Guidotti, A. Monreale, S. Ruggieri, and
F. Turini, “Meaningful explanations of black box ai decision systems,” in
Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33,
2019, pp. 9780–9784.
[19] S. F. Jentzsch and N. Hochgeschwender, “Don’t forget your roots! using
provenance data for transparent and explainable development of machine
learning models,” in 2019 34th IEEE/ACM International Conference on
Automated Software Engineering Workshop (ASEW). IEEE, 2019, pp.
37–40.
[20] S. Liu, X. Wang, M. Liu, and J. Zhu, “Towards better analysis of
machine learning models: A visual analytics perspective,” Visual In-
formatics, vol. 1, no. 1, pp. 48–56, 2017.
[21] D. Doran, S. Schulz, and T. R. Besold, “What does explainable AI really
mean? A new conceptualization of perspectives,” in CEUR Workshop
Proceedings, vol. 2071, 2018.
[22] S. Salmin, G. Gabriel, B. Elisa, and S. Mohamed, “A Lightweight Secure
Provenance Scheme For Wireless Sensor Networks,” in Proceedings
of the International Conference on Parallel and Distributed Systems
(ICPADS), vol. 6, no. 1, 2012.
[23] S. F. Eletter, S. G. Yaseen, and G. A. Elrefae, “Neuro-based artificial
intelligence model for loan decisions,” American Journal of Economics
and Business Administration, vol. 2, no. 1, p. 27, 2010.
[24] S. Sachan, J.-B. Yang, D.-L. Xu, D. E. Benavides, and Y. Li, “An
explainable ai decision-support-system to automate loan underwriting,”
Expert Systems with Applications, vol. 144, p. 113100, 2020.
[25] A. Alkhalil and R. A. Ramadan, “IoT Data Provenance Implementation
Challenges,” Procedia Computer Science, vol. 109, no. 2014, pp. 1134–
1139, 2017.
[26] K. W. Hamlen, L. Kagal, and M. Kantarcioglu, “Policy enforcement
framework for cloud data management.” IEEE Data Eng. Bull., vol. 35,
no. 4, pp. 39–45, 2012.