Professional Documents
Culture Documents
Hima Lakkaraju
Assistant Professor
Harvard Business School + Computer Science
Agenda
2
Course Staff
4
Course Overview
▪ Introduction & overview (Week 1)
5
Course Readings
6
Who should take this course?
8
Course Components
▪ We will release:
▪ a list of problem statements next week,
▪ all the final project reports from last year’s course
▪ Proposal (10%)
▪ 2 page overview of your project direction
▪ What is the problem?
▪ Why haven’t prior works solved it?
▪ How do you propose to solve it?
▪ How will you measure success?
11
Project Milestones
12
Background
13
Questions??
Motivation
[ Weller 2017 ]
Motivation: Why Model Understanding?
This prediction is
Defendant Details biased. Race and
gender are being
used to make the
prediction!!
Model Understanding
Model understanding facilitates bias detection.
Race
Crimes
Gender
Predictive
Prediction = Risky to Release
Model
Increase
Loan Applicant Details
salary by I have some means
50K + pay for recourse. Let me
credit card go and work on my
bills on time promotion and pay
my bills on time.
Model understanding helps
for next 3 provide recourse to individuals
months
who are adversely to
affected by model predictions.
get a loan
Predictive
Prediction = Denied Loan
Model
Loan Applicant
Motivation: Why Model Understanding?
Model Understanding
Patient Data If gender = female, This model is using
irrelevant features when
if ID_num > 200, then sick predicting on female
25, Female, subpopulation. I should
If gender = male, not trust its predictions
Cold for that group.
if cold = true and cough = true,
32, Male, No Model understanding helps assess if and when to trust
then sick
31, Male,
model predictions when making decisions.
Cough Predictions
Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy 19
Sick
Motivation: Why Model Understanding?
If gender = female,
This model is using
if ID_num > 200, then sick irrelevant features when
25, Female, predicting on female
Cold If gender = male, subpopulation. This
32, Male, No cannot be approved!
if cold = true and cough = true,
31, Male, then sick
Cough
. Predictions
Healthy
Sick
Predictive Sick
.
Model
.
Healthy
Healthy 20
Sick
Motivation: Why Model Understanding?
Utility Stakeholders
Explainer
Example
[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]
Inherently Interpretable Models vs.
Post hoc Explanations
30
Motivation for Interpretability
31
Prior Work: Defining and Measuring
Interpretability
▪ Little consensus on what interpretability is and
how to evaluate it
32
Prior Work: Defining and Measuring
Interpretability
▪ Evaluate in the context of an application
▪ If a system is useful in a practical application or a
simplified version, it must be interpretable
33
Lack of Rigor?
▪ Yes and No
▪ Previous notions are reasonable
Important to formalize these notions!!!
▪ However,
34
What is Interpretability?
35
When and Why Interpretability?
36
When and Why Interpretability?
▪ Incompleteness ≠ Uncertainty
▪ Uncertainty can be quantified
▪ E.g., trying to learn from a small dataset (uncertainty)
37
Incompleteness: Illustrative Examples
▪ Scientific Knowledge
▪ E.g., understanding the characteristics of a large dataset
▪ Goal is abstract
▪ Safety
▪ End to end system is never completely testable
▪ Not possible to check all possible inputs
▪ Ethics
▪ Guard against certain kinds of discrimination which are too
abstract to be encoded
▪ No idea about the nature of discrimination beforehand
38
Taxonomy of Interpretability Evaluation
39
Application-grounded evaluation
▪ Potential experiments
▪ Pairwise comparisons
▪ Simulate the model output
▪ What changes should be made to input to change the
output?
41
Functionally-grounded evaluation
▪ Potential experiments
▪ Complexity (of a decision tree) compared to other other
models of the same (similar) class
▪ How many levels? How many rules?
42
Open Problems: Design Issues
43
Taxonomy based on applications/tasks
▪ Degree of Incompleteness
▪ What part of the problem is incomplete? How
incomplete is it?
▪ Incomplete inputs or constraints or costs?
▪ Time Constraints
▪ How much time can the user spend to understand
explanation?
44
Taxonomy based on applications/tasks
45
Taxonomy based on methods
46
Taxonomy based on methods
▪ Level of compositionality:
▪ Are the basic units organized in a structured way?
▪ How do the basic units compose to form higher order
units?
▪ Uncertainty:
▪ What kind of uncertainty is captured by the methods?
▪ How easy is it for humans to process uncertainty?
47
Questions??
Relevant Conferences to Explore
▪ ICML
▪ NeurIPS
▪ ICLR
▪ UAI
▪ AISTATS
▪ KDD
▪ AAAI
▪ FAccT
▪ AIES
▪ CHI
▪ CSCW
▪ HCOMP
49
Breakout Groups
▪ Say hi to your neighbors! Introduce yourselves!