Experimentation

Algorithmic Bias

19th July 2021

Causality and Potential Outcomes
 Causal Question

• What is the effect of the treatment on the outcome?

• If the opposite treatment had been received, how would
the outcome differ?
• Example:
• Treatment: choosing organic produce.
• Outcome: get cancer? (yes or no).
• Question: Does choosing organic produce decrease risk
of cancer?
Potential Outcomes

• Key question: what would have happened, under the

opposite treatment?
• A potential outcome is the value of the outcome
variable for a given value of the treatment.
• Outcome variable: Y
• Y(treatment): outcome under treatment.

• Y(control): outcome under control

 Causal Effect
• The causal effect is the comparison of the potential
outcome under treatment to the potential outcome under
control.
• For quantitative outcomes, we often take a difference:
Causal Effect = Y(treatment) – Y(control)
 Causal Effect

• Possibility 1:
Y(organic) = no cancer, Y(non-organic) = cancer.

Causal effect for this individual: choosing organic produce prevents

cancer, which he otherwise would have gotten eating non-organic.

• Possibility 2:
Y(organic) = cancer, Y(non-organic) = cancer.

Causal effect for this individual: he would get cancer regardless, so

no causal effect.
 • Y = Test score:
• Y(study) – Y(don’t study).
• Example: Y(study) = 90, Y(don’t study) = 60.
• Causal effect = 90 – 60 = 30. Studying causes a
Causal Effect 30 - point gain in test score for this unit.
• Y = accident (Y/N)
• Y(texting) vs Y(not texting).
• Y = energy during the day
• Y(morning exercise) – Y(no morning exercise).
• Y = hook-up (y/n)
• Y(alcohol) vs Y(no alcohol).
 • These are unit-level causal effect
• Unit: the person, place, or thing upon which
the treatment will operate, at a particular
point in time
Units • Note: the same person at different points in
time = different units
• The causal effect will probably not be the
same for each unit
Causal Effects
• The definition of a causal effect does not depend on
which treatment is observed.
• Causal effects, in their definition, do not relate to
probability distribution for subjects “who got different
treatments”, or to coefficients of models.
• Not a before-after comparison. Potential outcomes are a
“what-if” comparison.
• Potential outcomes may depend on other variables (not
just the treatment).
Estimation

For the definition of a causal effect: compare potential outcomes for a

single unit.

However, in reality, we can only see one potential outcome for each unit.

For estimation of a causal effect we will need to consider multiple units,

some who have been exposed to the treatment, and some to the control.
Multiple Units

NEED TO COMPARE SIMILAR UNITS, CAN BE SAME UNITS AT DIFFERENT

SOME EXPOSED TO ACTIVE TREATMENT, POINTS IN TIME, OR DIFFERENT UNITS
AND SOME TO CONTROL. AT THE SAME POINT IN TIME.
Summary

Potential outcomes represent

Causality is tied to an action
the outcome for each unit
(treatment).
under treatment and control.

A causal effect compares the

In reality, only one potential
potential outcome under
outcome observed for each
treatment to the potential
unit, so need multiple units to
outcome under control for each
estimate causal effects.
unit.
Stable Unit Treatment Value Assumption
 Experiment 1

Sahni, N. S., & Nair, H. S. (2020). Sponsorship disclosure and consumer deception: Experimental evidence from native advertising in mobile search. Marketing
Science, 39(1), 5-32.
Experiment 2
Blake, T., Moshary, S., Sweeney,
K., & Tadelis, S. (2021). Price
salience and product
choice. Marketing Science.
Experiment 3

Abhishek, V., Jerath, K., &

Sharma, S. (2019). Advertising
on Online Marketplaces:
Information Asymmetry and
the Relevance of Sponsored
Listings. Available at SSRN
3013468.
 Experiment 4

Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large‐scale field experiment. Econometrica, 83(1), 155-174.
Experiment 5
Fong, J. (2020). Search,
Selectivity, and Market
Thickness in Two-Sided
Markets: Evidence from Online
Dating. Selectivity, and Market
Thickness in Two-Sided Markets:
Evidence from Online Dating
(June 12, 2020).
Experiment 6
Lambrecht, Anja, and Catherine
Tucker. "When does retargeting
work? Information specificity in
online advertising." Journal of
Marketing research 50.5 (2013):
561-576.
Multi-Armed Bandit (MAB) – A/B Testing
https://vwo.com/blog/multi-armed-bandit-algorithm/
 MAB has • Opportunity cost of lost conversions is too
high
shown that • Optimizing Click-Through Rates For News
it’s clearly Outlets That Cover Time-Sensitive Events
• Continuous Optimization
superior to • Optimizing Revenue with Low Traffic
A/B Testing
A/B Testing is clearly the better choice

When you are aiming for Statistical Optimizing for multiple metrics Post experiment analysis Incorporating learnings from all
Significance variations (including the poor ones)
into further business decisions
Causation - Correlation
https://towardsdatascience.com/why-correlation-does-not-imply-causation-5b99790df07e
Algorithmic Bias
Lambrecht, A., & Tucker, C. (2019). Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management Science, 65(7), 2966-2981.
https://www.wsj.com/articles/SB10001424052702304458604577488822667325882
https://www.newscientist.com/article/2246202-uber-and-lyft-pricing-algorithms-charge-more-in-non-white-areas/
https://www.nytimes.com/2019/12/19/technology/facial-recognition-bias.html
https://www.theguardian.com/business/2020/jan/16/black-owned-firms-are-twice-as-likely-to-be-rejected-for-loans-is-this-discrimination
https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
WhenA/B TestingDoesn’tTell You theWhole
Story
Imagine you work for a large arts organization that is
concerned about declining retention rates among its
members.
You’re thinking about sending a small gift along with the
renewal notification to the members you’ve determined are
at a higher risk of canceling their memberships, but since that
comes at a cost, you want to make sure the intervention is
effective before rolling it out more broadly.
So, you decide to run a small pilot campaign, randomly
choosing one group of “at risk” members to receive a gift and
one not to, in order to see if those who receive the gift are
more likely to renew.