Tutorial On Methods For Interpreting and Understanding Deep Neural Networks

Tutorial on Methods for Interpreting and
Understanding Deep Neural Networks
Wojciech Samek
Grgoire Montavon
Klaus-Robert Mller
(Fraunhofer HHI) (TU Berlin) (TU Berlin)
1:30 - 2:00 Part 1: Introduction
2:00 - 3:00 Part 2a: Making Deep Neural Networks Transparent
3:00 - 3:30 Break
3:30 - 4:00 Part 2b: Making Deep Neural Networks Transparent
4:00 - 5:00 Part 3: Applications & Discussion
ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller

Before we start
We thank our collaborators !
Alexander Binder
Sebastian Lapuschkin
(SUTD) (Fraunhofer HHI)
Lecture notes will be online soon at:
Please ask questions at any time !
ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller /29

Tutorial on Methods for Interpreting and
Understanding Deep Neural Networks

W. Samek, G. Montavon, K.-R. Mller
Part 1: Introduction
ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller

Recent ML Systems achieve superhuman Performance
Deep Net outperforms humans
AlphaGo beats Go in image classification DeepStack beats
human champ professional poker players
Autonomous search-and-rescue
drones outperform humans
Computer out-plays Deep Net beats human at

humans in "doom" recognizing traffic signs
IBM's Watson destroys

humans in jeopardy
ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 4 /29

From Data to Information
Huge volumes of data
Solve task
Interpretable
Information
ex
tra
Computing power
ct
Deep Nets / Kernel Machines / Information (implicit)

From Data to Information
Interpretability
AlexNet
Clarifai
VGG
GoogleNet
ResNet
(16.4%) (11.1%) (7.3%) (6.7%) (3.57%)
Performance
Interpretable
Data Information
for human
Crucial in many applications

(industry, sciences )

Interpretable vs. Powerful Models ?
Linear model Non-linear model
vs.
Poor fit, but easily

interpretable Can be very complex
global explanation individual explanation

Linear model Non-linear model
vs.
Poor fit, but easily

interpretable Can be very complex
global explanation individual explanation
60 million parameters
650,000 neurons We have techniques to interpret and

explain such complex models !

train best
train interpretable
interpret it vs.
model model
suboptimal or biased due to
assumptions (linearity, sparsity )

Dimensions of Interpretability
Different dimensions
prediction
of interpretability Explain why a certain pattern x has
been classified in a certain way f(x).
model
What would a pattern belonging
to a certain category typically look
like according to the model.
data
Which dimensions of the data
are most relevant for the task.

Why Interpretability ?
1) Verify that classifier works as expected
Wrong decisions can be costly

and dangerous
Autonomous car crashes, AI medical diagnosis system

because it wrongly recognizes misclassifies patients disease

2) Improve classifier
Generalization error Generalization error + human experience

3) Learn from the learning machine
It's not a human move. I've

never seen a human play this Old promise:
move. (Fan Hui) Learn about the human brain.

4) Interpretability in the sciences
Stock market analysis:

In medical diagnosis:
Model predicts share value Model predicts that X will

with __% accuracy. survive with probability __
What to do with this

Great !!!
information ?

4) Interpretability in the sciences
Learn about the physical / biological / chemical mechanisms.
(e.g. find genes linked to cancer, identify binding sites )

5) Compliance to legislation
European Unions new General
right to explanation
Data Protection Regulation
Retain human decision in order to assign responsibility.
With interpretability we can ensure that ML models

work in compliance to proposed legislation.

Interpretability as a gateway between ML and society

Make complex models acceptable for certain applications.
Retain human decision in order to assign responsibility.
Right to explanation
Interpretability as powerful engineering tool

Optimize models / architectures
Detect flaws / biases in the data
Gain new insights about the problem
Make sure that ML models behave correctly

Techniques of Interpretation

Interpreting models better understand

(ensemble) internal representation
- find prototypical example of a category

- find pattern maximizing activity of a neuron
Explaining decisions crucial for many

(individual) practical applications
- why does the model arrive at this

particular prediction
- verify that model behaves as expected

In medical context
Population view (ensemble)
Which symptoms are most common for the disease
Which drugs are most helpful for patients
Patients view (individual)
Which particular symptoms does the patient have
Which drugs does he need to take in order to recover
Both aspects can be important depending on who you are
(FDA, doctor, patient).

Interpreting models

cheeseburger
goose
car

Interpreting models

cheeseburger
goose
car
simple regularizer
(Simonyan et al. 2013)

Interpreting models

cheeseburger
goose
car
complex regularizer
(Nguyen et al. 2016)

Explaining decisions
- why does the model arrive at a certain prediction

Explaining decisions
- why does the model arrive at a certain prediction
- Sensitivity Analysis
- Layer-wise Relevance Propagation (LRP)

Sensitivity Analysis
(Simonyan et al. 2014)

Layer-wise Relevance Propagation (LRP)
(Bach et al. 2015)
every neuron gets its share of

relevance depending on activation
and strength of connection.
Theoretical interpretation
Deep Taylor Decomposition
(Montavon et al., 2017)

Sensitivity Analysis: LRP / Taylor Decomposition:

what makes this image what makes this image
less / more scooter ? scooter at all ?

More to come
Part 2
Part 2
Part 3 quality of explanations, applications,

interpretability in the sciences, discussion

Tutorial On Methods For Interpreting and Understanding Deep Neural Networks

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tutorial On Methods For Interpreting and Understanding Deep Neural Networks

Uploaded by

Copyright:

Available Formats

Tutorial on Methods for Interpreting and

Understanding Deep Neural Networks

(Fraunhofer HHI) (TU Berlin) (TU Berlin)

1:30 - 2:00 Part 1: Introduction

2:00 - 3:00 Part 2a: Making Deep Neural Networks Transparent

3:00 - 3:30 Break

3:30 - 4:00 Part 2b: Making Deep Neural Networks Transparent

4:00 - 5:00 Part 3: Applications & Discussion

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller

We thank our collaborators !

(SUTD) (Fraunhofer HHI)

Lecture notes will be online soon at:

Please ask questions at any time !

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller /29

Understanding Deep Neural Networks

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller

Computer out-plays Deep Net beats human at

IBM's Watson destroys

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 4 /29

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 5 /29

(16.4%) (11.1%) (7.3%) (6.7%) (3.57%)

Crucial in many applications

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 6 /29

Linear model Non-linear model

Poor fit, but easily

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 7 /29

Linear model Non-linear model

Poor fit, but easily

650,000 neurons We have techniques to interpret and

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 8 /29

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 9 /29

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 10 /29

Wrong decisions can be costly

Autonomous car crashes, AI medical diagnosis system

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 11 /29

Generalization error Generalization error + human experience

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 12 /29

It's not a human move. I've

move. (Fan Hui) Learn about the human brain.

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 13 /29

Stock market analysis:

Model predicts share value Model predicts that X will

What to do with this

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 14 /29

Learn about the physical / biological / chemical mechanisms.

(e.g. find genes linked to cancer, identify binding sites )

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 15 /29

European Unions new General

Retain human decision in order to assign responsibility.

With interpretability we can ensure that ML models

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 16 /29

Interpretability as a gateway between ML and society

Retain human decision in order to assign responsibility.

Interpretability as powerful engineering tool

Detect flaws / biases in the data

Gain new insights about the problem

Make sure that ML models behave correctly

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 17 /29

ICASSP 2017 Tutorial W. Samek, G. Montavon & K.-R. Mller 18 /29

Interpreting models better understand

- find prototypical example of a category

Explaining decisions crucial for many