CSCM23

DESIGNING-IN TRUST,
UNDERSTANDING, AND
NEGOTIATION
Towards Explainable AI
CSCM23
Designing-In Trust, Understanding, and Negotiation

XAI
A Brief

Based on slides by Xiuyi Fan

Overview of
 3 XAI

DEFINITIONS

Interpretability is the degree to which a human can understand the

cause of a decision.

Tim Miller, Explanation in arti cial intelligence: Insights from the social sciences,
Arti cial Intelligence, 2019
Designing-In Trust, Understanding, and Negotiation

Interpretability is the degree to which a human can consistently

predict the model’s result.

Kim, Been, Rajiv Khanna, and Oluwasanmi O. Koyejo. “Examples are not enough,
learn to criticize! Criticism for interpretability.” Advances in Neural Information
Processing Systems (2016).
CSCM23
fi
fi
CSCM23
Designing-In Trust, Understanding, and Negotiation 4 XAI

(DARPA XAI PROJECT)

A MACHINE LEARNING PERSPECTIVE …
 5 XAI

XAI AS A FIELD

Tim Miller, Explanation in arti cial intelligence:

Insights from the social sciences

Arti cial Intelligence, 2019

fi
fi
 6 XAI

TWO MAIN DIRECTIONS

▪︎ Knowledge Driven

▪︎ Starting point: knowledge representation (logic, ontology, Bayesian

nets, etc. - structured)

▪︎ Outputs: key elements / core inference processes

▪︎ Open questions: reaching explanation / usefulness / knowledge

sources
Designing-In Trust, Understanding, and Negotiation

▪︎ Data Driven

▪︎ Starting point: data (tabular forms, images, time series, etc. -

unstructured)

▪︎ Outputs: key features / core inputs

CSCM23

▪︎ Open questions: shallow explanation / ground truth for correctness /

overall effectiveness
Plan for this mini-module
Lecture 1: Introduction

Lecture 2: Assumption-based Argumentation

Lecture 3: Argumentation-based Explanation

Lecture 4: Shapley Value and SHAP

Lecture 5: Practical session on SHAP (??)

Zoom Poll
 8 XAI

WHY XAI? – TRUST

& TRANSPARENCY
Explanation in Artificial Intelligence: Insights from the
Social Sciences

Tim Miller, 2019 ▪︎ The running hypothesis is that by building

more transparent, interpretable, or
explainable systems, users will be better
equipped to understand and therefore trust
Designing-In Trust, Understanding, and Negotiation

the intelligent agents.

→reliable

→responsible
CSCM23
 9 XAI

WHY XAI? – INSIGHT

& KNOWLEDGE
Interpretable Machine Learning, A Guide for Making
Black Box Models Explainable

Christoph Molnar, 2019

▪︎ Knowing the ‘why’ can help you learn more

about the problem, the data and the reason
why a model might fail.
Designing-In Trust, Understanding, and Negotiation

→reliable

→resilient
CSCM23
 10 XAI

WHY XAI?
▪︎ Human curiosity and learning

▪︎ Why is something the case?

(Christoph Molnar, 2019) ▪︎ Find meaning in the world

▪︎ Resolve inconsistencies between elements of

our knowledge structures

▪︎ Goal of science
Designing-In Trust, Understanding, and Negotiation

▪︎ To gain knowledge, which does not always exist

in black box methods

▪︎ Safety measures

▪︎ Understanding the internal mechanisms of

algorithms
CSCM23
 11 XAI

WHY XAI?
▪︎ Detecting bias

(Christoph Molnar, 2019) ▪︎ Identify biases from the training data

▪︎ Increase social acceptance

▪︎ Make algorithms more human-friendly and

acceptable
Designing-In Trust, Understanding, and Negotiation

▪︎ Debugging and auditing

▪︎ Verify and validate algorithms

▪︎ …
CSCM23
 12 XAI

▪︎ Algorithm Transparency
SCOPE OF
▪︎
INTERPRETABILITY How does the algorithm create the model?

▪︎ Focus on speci c algorithms

(algorithmic perspective)
▪︎ Global Model Interpretability

▪︎ How does the trained model make predictions?

▪︎ Which features are important in the model?

▪︎ Model Interpretability on a Modular Level

Designing-In Trust, Understanding, and Negotiation

▪︎ How do parts of the model affect predictions?

▪︎ Which “components” in a model are responsible for a

decision?

▪︎ Instance Interpretability for a Single Prediction

CSCM23

▪︎ Why did the model make a certain prediction for an instance?

▪︎ Instance level explanation

fi
 13 XAI

TWO MAIN TYPES OF X-ML

Interpretable Models
vs.
Designing-In Trust, Understanding, and Negotiation

Model-Agnostic Methods
CSCM23
 14 XAI

INTERPRETABLE
MODELS
Just to drop some names … ▪︎ Standard simple / weak models …

▪︎ Linear Regression

▪︎ Logistic Regression

▪︎ Decision Trees
Designing-In Trust, Understanding, and Negotiation

▪︎ Naïve Bayes

▪︎ K-nearest Neighbours
CSCM23
 15 XAI

MODEL-AGNOSTIC
METHODS:

Partial Dependence Plot (PDP) ▪︎ Global Method

▪︎ The partial dependence plot shows the marginal

e ect of one or two features have on the
predicted outcome of a machine learning model.

▪︎ For classi cation where the machine-learning model

Designing-In Trust, Understanding, and Negotiation

outputs probabilities, the partial dependence plot

displays the probability for a certain class
given di erent values for feature(s) in the set of
all features.
CSCM23

Friedman, Jerome H.
Greedy function approximation: A gradient boosting
machine.”
Annals of statistics (2001): 1189-1232
ff
ff
fi
 16 XAI

MODEL-AGNOSTIC
METHODS: ▪︎ Instance version of PDP

▪︎ Individual Conditional Expectation (ICE) plots display

one line per instance that shows how the
Individual Conditional Expectation
instance’s prediction changes when a feature
changes.
Designing-In Trust, Understanding, and Negotiation
CSCM23

Goldstein, Alex, et al.

Peeking inside the black box: Visualizing statistical
learning with plots of individual conditional expectation.
Journal of Computational and Graphical Statistics 24.1
(2015): 44-65
 17 XAI

MODEL-AGNOSTIC
METHODS
PERMUTATION
Feature Importance
Designing-In Trust, Understanding, and Negotiation

▪︎ Global Method

▪︎ Permutation feature importance measures the

increase in the prediction error of the model after
we permuted the feature’s values, which breaks the
relationship between the feature and the true
outcome.
CSCM23

Fisher, Aaron, Cynthia Rudin, and Francesca Dominici.

Model Class Reliance: Variable importance measures for
any machine learning model class, from the ‘Rashomon’
perspective.
arxiv 2018
 18 XAI

MODEL-AGNOSTIC
METHODS
GLOBAL SURROGATE
▪︎ Global Method

▪︎ A global surrogate model is an interpretable

model that is trained to approximate the
predictions of a black box model.

▪︎ We can draw conclusions about the black box

Designing-In Trust, Understanding, and Negotiation

model by interpreting the surrogate model.

Decision Tree → SVM, etc.

CSCM23
 19 XAI

MODEL-AGNOSTIC
METHODS: ▪︎ Instance Method
LOCAL SURROGATE (LIME) ▪︎ Local surrogate models are interpretable models
that are used to explain individual predictions
of black box machine learning models.

▪︎ LIME generates a new dataset consisting of

permuted samples and the corresponding
predictions of the black box model. On this new
dataset LIME then trains an interpretable model,
which is weighted by the proximity of the sampled
Designing-In Trust, Understanding, and Negotiation

instances to the instance of interest.

CSCM23

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin.

Why should I trust you?: Explaining the predictions of any
classi er.
Proceedings of the 22nd ACM SIGKDD international conference
on knowledge discovery and data mining. ACM (2016)
fi
 20 XAI

MODEL-AGNOSTIC
METHODS:
SHAPLEY VALUES
▪︎ Instance Method

▪︎ Features are viewed as “players”.

▪︎ Shapley values – a method from coalitional game

theory tells us how to fairly distribute the
“payout” among the features.
Designing-In Trust, Understanding, and Negotiation

▪︎ The Shapley value is the average marginal

contribution of a feature value across all possible
coalitions.
CSCM23
 21 XAI

MODEL-AGNOSTIC
METHODS:
SHAPLEY VALUES

▪︎ Features are viewed as “players”.

Designing-In Trust, Understanding, and Negotiation

▪︎ Consider all possible coalitions.

CSCM23
 22 XAI

EXAMPLE-BASED
EXPLANATIONS
▪︎ Counterfactual Explanations
▪︎ If X had not occurred, Y would not have
occurred

▪︎ Adversarial Examples
▪︎ Adversarial examples are counterfactual
examples with the aim to deceive the
Designing-In Trust, Understanding, and Negotiation

model, not interpret it.

▪︎ Identify cases where the model fails.

▪︎ …
CSCM23
 23 XAI

VISUALISING
NEURAL NETWORKS

▪︎ Very important & interesting area

▪︎ See e.g. papers in Interpretable Machine Learning,

Christoph Molnar, 2019
Designing-In Trust, Understanding, and Negotiation
CSCM23
 24 XAI

RECENT WORK
On Understanding the Influence of Controllable
Factors with a Feature Attribution Algorithm:
a Medical Case Study
Veera Raghava Reddy Kovvuri Siyuan Liu Monika Seisenberger (a) A lung cancer instance randomly selected from the Simulacrum dataset. Uncontrollable features are: Age, Ethnicity, Sex, and Height.
Swansea University Nanyang Technological University Swansea University (a) A lung cancer instance randomly selected from the Simulacrum dataset. Uncontrollable features are: Age, Ethnicity, Sex, and H
Swansea,United Kingdom Singapore Swansea,United Kingdom
v.r.r.kovvuri@swansea.ac.uk syliu@ntu.edu.sg m.seisenberger@swansea.ac.uk

Xiuyi Fan Berndt Müller Hsuan Fu

Nanyang Technological University Swansea University Université Laval
Singapore Swansea,United Kingdom Québec, Canada
xyfan@ntu.edu.sg berndt.muller@swansea.ac.uk hsuan.fu@fsa.ulaval.ca

Abstract—Feature attribution XAI algorithms enable their treat all features homogeneously when computing their relative
users to gain insight into the underlying patterns of large datasets importances. Such homogeneity may not always give desirable
through their feature importance calculation. Existing feature
Designing-In Trust, Understanding, and Negotiation

interpretations when feature attribution algorithms are used

attribution algorithms treat all features in a dataset homoge-
neously, which may lead to misinterpretation of consequences of for data mining purposes. Consider the following hypothetical
changing feature values. In this work, we consider partitioning example.
features into controllable and uncontrollable parts and propose
the Controllable fActor Feature Attribution (CAFA) approach to Suppose we want to estimate the chance for some
compute the relative importance of controllable features. We individual having breast cancer, with features like
carried out experiments applying CAFA to two existing datasets age, gender, weight, alcohol intake, smoking habits,
and our own COVID-19 non-pharmaceutical control measures family history, etc. A predictive model estimates the
dataset. Experimental results show that with CAFA, we are likelihood of the person having breast cancer; and
able to exclude influences from uncontrollable features in our
explanation while keeping the full dataset for prediction. a feature attribution algorithm gives attributions like
Index Terms—Explainable AI, Feature Attribution, Medical age: 0.3, gender: 0.13, weight: 0.27, alcohol intake:
Application 0.15, smoking: 0.3, family history: 0.36, etc.
From these calculated values, we notice that certain features,
I. I NTRODUCTION such as age, gender and family history, while being influential (b) A breast cancer instance randomly selected from the UCI breast cancer dataset. Uncontrollable features are Age and Menopause.
Feature attribution algorithms [17] are a popular class of to the prediction, are uncontrollable risk factors [5]. Knowing
Explainable AI (XAI) algorithms. Given a prediction instance, the relative importance of these features makes little contribu-
Fig. 2: Illustration of CAFA vs. SHAP on two explanation instances selected from two medical datasets. We observe that (1)
▪︎
they tell the relative “importance” of each feature in the tion to clinical decision making. On the other hand, features
instance. In addition to “explaining” the prediction model,
importance measures also reveal insight about the instance
representing controllable risk factors such as weight, alcohol
intake and smoking habits are vital to clinical interventions (b) Awith CAFA,
breast cancer Uncontrollable
allinstance
uncontrollable features
randomly are from
selected features
assigned
the UCI should
importance
breast cancer not
0; (2) for
dataset.be considered
controllable features,features
Uncontrollable CAFA produces results
are Age and that
Menopa
being explained, e.g., [4] shows that XAI can help “gener- [5]. Thus, from an intervention perspective, it is necessary
are agreeable with the ones given by SHAP.
when attempting to nd the biggest contributor.
ating the hypothesis about causality” in developing decision to distinguish these two classes of factors and compute their
support systems. In this sense, feature attribution algorithms influences accordingly. We raise the question:
What are the influences of controllable factors used in a
Fig. 2: Illustration of CAFA vs. SHAP on two explanation instances selected from two medical datasets. We observe th
are considered as a data mining tool for extracting and discov-
ering patterns in large datasets. For instance, [6] uses feature prediction? with CAFA, all uncontrollable features are assigned importance 0; (2) for controllable features, CAFA produces resul
CSCM23

attribution algorithms to understand important factors affecting To answer this question, a naive approach would be to build (a) Global views of lung cancer cases in the Simulacrum (left: SHAP; right: CAFA). Uncontrollable features are: Age, Ethnicity, Sex, and
and Humber, Northern Ireland, Scotland and Wales. To remove • SHAP considers High Restriction on Cafes and Restau-
cancer patient survivability; [7] employs feature attribution al- another predictive model, which only considers controllable
are agreeable with the ones givenHeight.
by SHAP.
gorithms to study factors affecting the transmission of SARS-
CoV-2; and [16] uses feature attribution to analyse factors
factors, and apply feature attribution algorithms to that model.
However, as explained in [8] and [26], dropping features from
noise and achieve a more accurate Rt estimation, we drop data rants Access (CR H), High Restriction on Pubs and
affecting foreign exchange markets. However, existing feature
attribution algorithms (see e.g., [2], [19], [24] for overviews)
models can negatively impact the model performance as we
will show in Table I in experimental study. Thus, instead points with cumulative cases less than 20 for each region and Bars Access (PB H), Number of Daily Infections (Cases),
of building models with fewer features, we suggest creating
algorithms that are able to treat controllable factors differently keep 3,936
and Humber, instances.
Northern A sliding-window
Ireland, Scotland and mean filter
Wales. To of size 3
remove • Number
SHAP considers of Daily Infections (Cases), Medium
High Restriction Restriction
on Cafes and R
978-1-6654-9810-4/22/$31.00©2022 IEEE from uncontrollable ones.
hasachieve
noise and been used to filter
a more noise inRdaily
accurate cases. we drop data
estimation, on PubsAccess
rants and Bars(CR AccessH), (PB M), Restriction
High and High Restriction
on Pub
t
Wecumulative
split the dataset as 70% Sport and Leisure Facilities (SL H) as the top five
points with cases less thanfor 20 training
for eachand region 30%and for Bars Access (PB H), Number of Daily Infections (C
testing, and use a random forest classifier. We achieve a high effective control measures; whereas
keep 3,936 instances. A sliding-window mean filter of size 3 Number of Daily Infections (Cases), Medium Rest
• CAFA considers CR H, PB H, PB M, Medium Restric-
prediction accuracy of 94.4%. Since
(b) Globalwe
views aim toBreast
obtain a bird’s-
has been used to filter noise in daily cases. of the UCI Cancer dataset
on Pubs and Bars and Access
Nursing (PB
(left: SHAP; right: CAFA). Uncontrollable features are: Age, and Menopause.
tion on Hospital HomeM), and(HV
Visits High
M) Rest
and
eye view of how control measures are affecting the disease, we
fi
 Using Explainable AI to Understand
Impacts of Non-pharmaceutical Control
Measures on COVID-19 Transmission for
Evidence-based Policy
An Example
Funded by Welsh Government
2020-2021
 “This project will gain deep insights into the effects of
COVID measures used to control the spread of COVID-19. … We will
further explore XAI techniques to reveal dynamics between
Project control measures and disease transmission.”
 Question: Which control measures are more effective?

Two Objectives:

(O1) develop and maintain a streamlined dataset capturing

COVID COVID-19 transmission information and implemented control
measures, and
Project
(O2) adapt and apply multiple XAI techniques on the dataset
constructed to fulfil O1 to obtain easily interpretable relations
between control measures and disease transmission while
actively interacting with the policy community.
Publication
 Current Work

Multiple
Information
Sources Data in Machine
Machine Learning
(E.g., Wikipedia, Learning Tabular XAI Results
Models
Government Format
Websites, Kaggle,
etc.)
Current Work
Current Work
Current Work

Important Features
Current Work
Current Work
 35 XAI

ANTI-XAI

▪︎ Cynthia Ruin
Stop explaining black box machine learning models
for high stakes decisions and use interpretable
models instead
Designing-In Trust, Understanding, and Negotiation

Nature Machine Intelligence, 1, 206–215, (May

2019).
CSCM23
 36 XAI

IN SHORT …

▪︎ Motivated by issues with AI and ML in real-world applications

(many of them)

▪︎ Many approaches, no dominant method

▪︎ New area, still large scope to explore

Designing-In Trust, Understanding, and Negotiation
CSCM23