A Unified Approach to

Interpreting Model Predictions

Scott Lundberg, Su-In Lee
University of Washington

NIPS 2017
 Why?!

chance John will have

repayment problems

model 55%
No loan

John, a bank customer Why?! AI magic!

 Interpretable Accurate

Complex model ✘ ✔
Simple model ✔ ✘

⚖️
Interpretable or accurate: choose one.
💰

3
 Output value
Input value

Complex models are But a single prediction involves only

inherently complex! a small piece of that complexity.

4
data .
. model prediction
.
data .
. prediction
.
data .
. prediction
.
 LIME DeepLIFT
Ribeiro et al. 2016 Shrikumar et al. 2016

Shapley reg. values Relevance prop.

Lipovetsky et al. 2001 Bach et al. 2015

QII Shapley sampling Path expectations

Datta et al. 2016 Štrumbelj et al. 2011 Saabas 2014

8
 Additive feature attribution methods

LIME DeepLIFT
Ribeiro et al. 2016 Shrikumar et al. 2016

Shapley reg. values Relevance prop.

Lipovetsky et al. 2001 Bach et al. 2015

QII Shapley sampling Path expectations

Datta et al. 2016 Štrumbelj et al. 2011 Saabas 2014

9
 Additive feature attribution methods

data .
. prediction
.

Only one way to assign feature attributions given two properties!

 Additive feature attribution methods

data .
. prediction
.

1 Local accuracy
 data .
.
.

2 Consistency
 data .
.
.

2 Consistency
 data .
.
.

2 Consistency
 Additive feature attribution methods

LIME DeepLIFT

Shapley reg. values Relevance prop.

QII Shapley sampling Path expectations

 LIME DeepLIFT

SHAP
Shapley reg. values Relevance prop.

QII Shapley sampling Path expectations

SHapley Additive exPlanation (SHAP)
values

Base rate Prediction for John

20% 55%

How did we get here?

21
SHapley Additive exPlanation (SHAP)
values

20% 35% 55% 70% 90%

Base rate Day trader Open accounts = 1

Age = 20
Married

22
SHapley Additive exPlanation (SHAP)
values
The order matters!
SHAP values result from averaging over all N! possible orderings.

Day trader
Age = 20

23
 LIME DeepLIFT

SHAP
Shapley reg. values Relevance prop.

QII Shapley sampling Path expectations

LIME Objective

The loss function to force g to well approximate f

Optional regularization of g

Kernel specifies what ‘local’ means

A class of interpretable models (linear models)
But how do we pick L, , and ?
25
L, , and are forced under local accuracy and
consistency !

Kernel weight
Interpretable input space

26
 Improved consistency with human intuition

cou
gh?

fever?

data treatment treatment

model priority
tion?
conges

27
 Improved consistency with human intuition

cou
gh?

fever? cough AND fever → 2

data treatment
cough XOR fever → 5
treatment
model priority
tion? otherwise → 0
conges

28
Improved consistency with human intuition

cou
gh
treatment priority
fever cough AND fever → 2
treatment
cough XOR fever → 5 2
model
tion otherwise → 0
conges

29
Improved consistency with human intuition

treatment priority

30
 Faster estimation than classic Shapley
Feature attribution value
methods

# of model evaluations # of model evaluations

31
 Faster estimation than classic Shapley
Feature attribution value
methods

# of model evaluations # of model evaluations

32
 Faster estimation than classic Shapley
Feature attribution value
methods

# of model evaluations # of model evaluations

33
 Faster estimation than classic Shapley
Feature attribution value
methods

# of model evaluations # of model evaluations

34
Improved separation of MNIST classes

35
Improved separation of MNIST classes

36
Unified the additive feature attribution methods

Proved intuitive uniqueness results

Proposed SHAP values + estimation methods

github.com/slundberg/shap

Poster is tonight #34

github.com/slundberg/shap

Poster is tonight #2493

Scratch slides
 github.com/slundberg/shap

Shapley sampling

Relevance prop.

Orig. DeepLIFT
Kernel SHAP

Linear SHAP
Deep SHAP

regression
Max SHAP

Tree SHAP

Tree exp.
Shapley
LIME

QII
Local
accuracy ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕

Missingness ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕ ➕

Consistency ➕ * ➕ ➕ ➕ ➕ ➕ ➕ ➕

Sample
efficiency ➕ ➕➕ ➕➕ ➕➕ ➕➕ ➕ ➕➕ ➕➕ ➕➕

Model
agnostic ➕ ➕ ➕ ➕ ➕