RMDA Final Review

2020-2021
Semester 1
Prof. Sally Hudson
Agenda
• Letters + Formulas
• Concepts
• STE vs. Beta
• Clustering + its impacts
• MDE + Power
• Regressions: understanding the baseline Positive Note: you will NOT
• Adding coefficients need to do STATA on the
• TOT vs ITT vs LATE final!!
• Concept
• Math
• Picture version
• Practice Exam
• Practice sheet we were given
• 2019 Final
• Practice Problems
 Letters and Formulas
Yi: The outcome for the individual  E[Yi|Zi=1]-E[Yi|Zi=0]: ITT
• Causal effect of offering treatment
Di: The treatment for the individual 
• Difference in outcomes for those who were offered the
Zi: The instrument for the individual treatment and those who weren’t
Xi: Baseline covariates (addtl variable) E[Yi1 - Yi0 | Di = 1] : TOT
Yi1 : Potential outcome of receiving the treatment • Average outcomes for those who got treated 
on the individual  E[Yi1 - Yi0 | Di1 > Di0 ]: LATE 
Yi0: Potential outcome of not receiving the • Average treatment effect for compilers 
treatment on the individual  • Difference from TOT: includes a fitted (weighted) D variable
• ITT/FS
Di1: Treatment if offered instrument
E[Di|Zi = 1] – E[Di|Zi = 0]: First Stage  
Di0 : Treatment if not offered instrument
E[Yi | Di = 1] - E[Yi | Di = 0]: naive difference
Yi1 - Yi0: Causal effect of the treatment on the
individual  E[Yi0 | Di = 1] - E[Yi0 | Di = 0]: selection bias 
• If 0, then the naive difference equals the TOT
 Letters and Formulas
Reading Greek Reading STATA
Yi = outcome Regressions

Di = binary treatment Base Code: regress A B, i.strata cluster(var)

• A = outcome coefficient applies to
Xi = baseline trait 
• B = thing you want to turn on and off
• Yi = α +βDi +εi (regress Yi on Di = regress Yi Di)
Covariate
• α=?
• regress covariate_var treatment_var
• β=?
• Want: high p-value
• Yi = α +βDi +γXi +εi (regress Yi on Di w/ covariate = Treatment Impact
regress Yi Di Xi)
• regress outcome_var treatment_var
• α=?
• Want: low p-value 
• β=?
• γ = what does it do to your regression?
LATE
Base Code: ivregress 2sls Yi (Di=Zi) Xi i.strata, cluster(var)
STE vs β
β > 2(STE)
= Coefficient is significant at
the 95% level

STE β
• Measurement of the variation in a sample • Coefficient on the treatment variable
• How much “white noise” is common place • What’s the additional impact of the treatment
on the outcome

2 STE 0 2 STE

2 STEs
 Blue = Treatment

Clustering Orange = Control

Letter = Household (everyone in the household gets the treatment)

A A
A
D

B A
B

E
C
C C
B
C
D B B
E B C
C

n = 10
Strata: The variable that impacts the likelihood of someone receiving the
treatment

If anyone in your
A A household gets the
treatment, you also get A
D
the treatment 2x as
likely 1
B A
B chance

E
C
C C 1
B
chance
B B C
D 3x as C C
E B likely
3x as
likely
Check-In
• What’s this: E[Yi1 - Yi0 | Di = 1]
• What’s the difference between the TOT and LATE?
• What’s this and where do we use it: E[Di|Zi = 1] – E[Di|Zi = 0]
• What’s α?
• What’s β?
• β what goes here/why? 2STE
• Why is the cluster the household and the strata the household size?
 Minimum Detectable Effect (MDE)
The smallest change you're able to detect.
It determines how "sensitive" an
experiment is. Generally, we want a low
MDE
Decrease in MDE = you can pick up smaller
differences
Increase = you can only detect big
differences
Levers
↑ n = ↓ MDE
↑ willingness to accept errors = ↓ MDE
Power
• Probability of rejecting the null hypothesis
when it is false (good thing!)
• How likely is it that you’ll detect an effect when it is
there.
• Well powered = lower MDE or findings are at MDE =
more likely to notice a significant effect
• Underpowered: relatively poor probability of
detecting a specified effect size (MDE)
• Increases likelihood of “No evidence of effect” (aka
smaller than MDE)
• Why bad: you may need to run the test again
(costly, boring, so last season)
• Substantive vs. Statistical Significance
• Substantive: you care for people reasons
• Statistical: you care for number reasons
 What’s the Difference?
“Evidence of No Effect” “No Evidence of (Any) Effect”
• You trust your data and test • You’re not sure about your data and test
• Low STE • High STE
• High p-value because the treatment • High p-value because the data was messy
didn’t work (evidence of no effect) • High chance of selection bias
• Low chance of selection bias • The data doesn’t prove anything
• The data shows something

Not Statistically Different from Zero

 High p-value
 Not statistically significant
 Coefficient (Beta) < 2STE
Powered

2 STE

Evidence of No evidence Evidence of

no effect of effect effect
ITT vs TOT vs LATE
But first!
 Foundation Building
GOOD CONTROLS BAD CONTROLS
• Measured before random • Measured after random
assignment assignment (aka an outcome)
• Strong predictors of outcome • Potentially correlated with
assignments status

Non-Random Take-up Problem: you can’t randomly assign and force people to take the
treatment (Di), but you can randomly assign access to the treatment (Zi). *Enter the
instrumental variable (IV)*
Instrumental Variables: way of studying the chain reactions where you can’t actually randomly
assign the treatment that you care about
4 Kinds of People
Always Takers Never Takers
• will get treatment even if denied instrument (Di1 • will not get treatment even if they get the
= Di0 = 1) or (Diz =1) instrument (Di1 = Di0 = 0) or (Diz=0)
• People who buck the system • Never takes us up on the offer even if we give
them it

Compliers Defiers
• will get the treatment if offered and not if not • will get treatment if denied it and won’t if given
Di = Zi (rare, generally assume they don’t exist)
Diz = z Di = 1 - Zi
Di1 > Di0 Diz = 1 - z
Di1 < Di0
 ITT TOT
E[Yi|Zi=1]-E[Yi|Zi=0] E[Yi1 - Yi0 | Di = 1]
Causal effect of Zi on Yi by Average outcomes for those
comparing avg outcomes for who got treated
people who did and did not
Similar to LATE, but includes
receive the instrument
everybody  
Charter Schools
-ITT: effect of being offered admission in the charter school on test scores
-TOT: effect of enrolling in the charter school on test scores

Bednets
-ITT: effect of being offered a bed net
-TOT: effect of using the bed net on rates of malaria

Deworming
-ITT: effect of being offered the medication
-TOT: effect of taking the medication
LATE
E[Yi1 - Yi0 | Di1 > Di0 ] = ITT/FS = (Reduced Form)/(Takeup Rate)

• Bridge between ITT and effect of the treatment on the outcome

• ITT = LATE when takeup rate is 100%

• Closest you can get to TOT when there is non-random compliance

• LATE = TOT when there are no always takers (only access to the treatment is through the instrument)

• Average treatment effect for compliers

• Only looks at people who’s treatment status is changed by the instrument and compliers are the only
people for who that is true
• Local = small subset of people
• LATE only focuses on causal effects for compliers
 LATE Details
E[Yi1 - Yi0| Di1 > Di0] = (E[Yi|Zi =1 ] – E[Yi|Zi = 0])/(E[Di|Zi = 1] – E[Di|Zi = 0])

• Average Treatment Effect: E[Yi1 - Yi0|

• Compliers: Di1 > Di0
• First Stage Effect
• The instrument changes treatment rates
• Intent to Treat Effect (ITT):
• Effect of the offer on the outcome (the instrument on the outcome)
• Reduced Form Effect

• = ITT/FS = ITT/(takeup rate)

• = “(scale changes in Y for changes in Z) by (changes in D for changes in Z)”
• = “Scale ITT effect by the first-stage”
 4 Conditions for Late
First Stage: instrument changes rates of treatment
• Z changes D
• Test: regression of the treatment on the instrument
(regress D Z)
• Is avg D different when Z is turned on and off? (Ex:
regress enrolled admitted)
• Want: statistically significant difference (low p-value
• Speak: “If effect statistically significant, than there is a
causal effect of the instrument on the treatment and
therefore the first-stage is non-zero”
Exclusion Restriction: instrument only affects outcome
through treatment
• One causal chain from Z to Y and it runs through D
• Why: Gives permission to do: (change in Y)/(change in D)
• Argue: receiving/not receiving the instrument impacts the
outcomes without them receiving the treatment
Independence: instrument is randomly assigned
• Captures causal effect of Z on Y
• Test: covariate balance test to assess whether the
instrument was randomly assigned
• Regression of the covariate on the
instrument (regress X Z)
• Is the covariate different when Z is
turned on and off? (Ex: regress
special_ed admitted)
• Want: no statistically significant difference in the
baseline covariates for those who received the
instrument and those who did not (high p-value)
No Defiers: the instrument only makes individuals
more like to get the treatment (not less)
• Why: makes complier component of LATE
theorem hold
• Argue: against receiving the instrument makes
you less likely to get the treatment
What’s Random?
INSTRUMENT

Calculate ITT: causal effect of instrument on outcome

• E[Yi|Zi=1] – E(Yi|Zi=0]
• ex) Does ADMITTING people to charter schools improve performance absent if
they go?
• Includes: effects on always-takers, never-takers, and compliers
• When It’s Interesting: when you’re not sure if the instrument/offer is working
(even if the treatment is)
TREATMENT
Calculate LATE: causal effect of treatment on outcome for compliers
Calculate TOT: causal effect of treatment on outcome • E[Yi1-Yi0|Di1 > Di0]
• E[Yi1=Yi0|D=1] • Di1 > Di0 = compliers
• *Be able to explain why this doesn’t limit it to only the outcomes you see as a
result of the instrument: why is this outcome expandable to just the treatment
and outcome absent the instrument?
• ex) Does going to a charter school improve performance?
• The instrument changes the math, BUT not the interpretation of the
result
• Includes: effects on compliers (people whose potential outcomes were changed
as a result of the instrument)
• When It’s Interesting: when the policy will be administered in a similar way to
how the instrument is applied (good reflection of reality)
Practice Practice Practice
This is the team participation part of the evening
 About 10% of the public 9th grade
students are Asian. About 3% of the
charter school applicants were Asian.
There was no statistically significant
difference between the share of Asian
students between those who were
admitted to the charter school and those
that weren’t. Lottery winners were no
more likely to be Asian than lottery
losers.
Regression of Di (enrollment) on Zi (admission offer)
Nonoffered Mean = E[Di|Zi = 0] (got treatment despite not receiving it through the instrument)
• 10.4% of student who did not win the lottery enrolled in the charter school
• Verbalizing It: work it backwards % of Zi=0 that got Di=1

First Stage – Immediate Offer (Di = π + μ(Zi) + Mi

• Students who won the lottery were 37.3 percentage points more likely to enroll in charter schools
o Because it’s a statistically significant difference and Z was randomly assigned, you can say: “Admission offers increased
enrollment rates by 37.3 percentage points”
 Combine together
 47.3% of students who won the lottery enroll
o Lead with the %
 Number not zero -> checks first stage
 Reduced Form: Yi = ϐ+ μ(Zi) + ϗ= ITT
 Noncharter Mean: students who did not enroll in a charter school
o Students who did not enroll in charter schools scored about .3 Combine Together
standard deviations below the state average  Reduced Form & First-Stage
 Immediate Offer o RF = .15 STEs – effect of the offer
o Students who did enroll in charter schools scored about .15 o FS = 0.37 percentage points
standard deviations higher than those who didn’t enroll • LATE = Effect of Going to Charter School = Charter
o Because Z is randomly assigned, can use causal language: Effect
Admission offers (Z) to a charter school increased MCAS  (Reduced Form)/(FS) = 0.15/0.37 = 0.4
scores (Y) by about 0.15 standard deviations

https://economics.mit.edu/files/9799