Disparate Impact

• Distributional Unfairness
• Ossification
• Surveillance
• Asymmetry
Distributional Unfairness
 Street Bump

An app that automatically reports potholes to the city

(using the phone’s accelerometer and GPS).
 Street Bump
• Deployed in Boston since 2012,
– many other cities since more recently.
• Crowd-sourced reporting by citizens.
• With focus on roads driven by people rich enough to have
a car and a smartphone, and the time to download the app.
• Compensate by having city employees driving around in
poorer neighborhoods.
 Unfairness to Groups

• Data derivations can have disparate impact on some

social groups.
– E.g. the poor
• Disproportionate differences between demographic
groups in how they view privacy
 Customs
Inspection

All travelers are

subject to search
when they enter
a country
 Customs Inspection
• Most travelers are not smugglers.
• Most travelers are not searched
– It is inconvenient and time-consuming for the traveler
– It is expensive to allocate customs agents to this task
• How to choose which travelers to search?
• Search the travelers most likely to be smugglers.
– Even if the odds of “most likely” are very low.
 Customs Inspection (contd.)
• What if algorithm chooses “most likely” based
primarily on country of origin?

• Travelers from “chosen” country feel discriminated

against
• They exchange stories of customs “harassment” on
social media
 Unfairness to Individuals

• Stereotypes
– Common human failing
– Often have a grain of truth behind them
– But completely unfair to the individual who is being typecast
because of the group she belongs to
• Algorithms can do the same thing
– Unfair to individuals who are classified based on their membership
in a group based on the value of some attribute
 Muslims and Terrorists

• Suppose 1000 terrorist sympathizers in the US.

– 900 Muslim, 100 non-Muslim.
• Terrorist => Muslim (with 90% probability)
• But this does NOT mean Muslim => Terrorist!!
• 3 million Muslims in the US.
– Less than one in 3000 Muslims is a terrorist sympathizer.
 Predict a Crime??
• Preemptively arrest person
with criminal intent?

• In general, NO.

• But possibly yes for

terrorism?
 Prediction is Probabilistic
• Probabilistic – only
indicates likelihood.
• Suggests greater
surveillance
– More audits of tax returns for
people more likely to have
cheated.
 Predictive Policing
• Deployment of police forces
to higher crime areas in
greater numbers.
• Vicious cycle?
– More surveillance can lead to more
detected crime.
• Police car follows known bad
drivers.
 China Revolt Prediction

• Goal is to prevent another “Tiananmen Square.”

• Predictive analytics, similar to the “pre-crime unit”
in the film Minority Report.
• Based on “dang’an”
• To be deployed initially in troubled regions: Xinjiang
and Tibet.
 Own the Consequences
• Predictions are probabilistic
– Will sometimes be wrong
• Understand societal cost of errors
– Usually different for type I and type II errors
• Factor into algorithm tuning
• We know how to do this