Impact Evaluation Workshop

IRSA Annual Conference, Solo, 2018

Causal Inference

Dr. Sarah Dong

Fellow, Indonesia Project
Australian National University
sarah.dong@anu.edu.au
Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 2
The counterfactual
What would have happened to
beneficiaries had they not received the
intervention?
Outcome
Observed from
Effect = outcome - counterfactual

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 3

The missing counterfactual
Outcome A: Outcome B:
Effect = receive - do not
PKH receive PKH

Recipient: A - B

Non-recipient: A - B

but never observe both A and B for the same

household!

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 4

Problem with OLS regression
Y = αTreat+ε
Treat =1 if receiving program
Treat = 0 if not receiving program

How do we interpret α?
• α is the average difference in Y between people who
receive the program and people who do not receive the
program
• These two groups of people may be different and not
comparable

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 5

Approximate for the counterfactual?
• Find a household in non-recipients that is
similar to the household in recipients
• The similar household will be the
counterfactual

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 6

What is a similar household in the non-
recipient pool?
• Similar in observed characteristics
– OLS, Matching
• Similar in both observed and unobserved
characteristics
– RCT, RDD, IV
• Similar in change overtime
– DID
• Same household in a pre-program time
– Panel fixed effect
Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 7
Exercise: good counterfactual vs. bad
counterfactual vs.

Program: PKH in Papua

Outcome: education of children in the recipient household
Unit: household
• Counterfactual 1: Non-recipient households in the same
village
• Counterfactual 2: Non-recipient household with similar
size, income, household composition in Jakarta
• Counterfactual 3: Education of children in the same
household 5 years ago before introduction of PKH
• Ideal counterfactual?

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 8

Exercise: good counterfactual vs. bad
counterfactual vs.
Program: Dana desa
Outcome: average per capita expenditure of the village
Unit: village
• Counterfactual 1: Villages that receive a lower amount of
village fund
• Counterfactual 2: Urban areas that do not receive village
fund
• Counterfactual 3: The same village when there was no
village fund
• Ideal counterfactual?

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 9

Exercise: good counterfactual vs. bad
counterfactual vs.

Program: Tax amnesty

Outcome: tax revenue
Unit: national
• Counterfactual 1: tax revenue in the same months in the
year before the amnesty
• Counterfactual 2: tax revenue in the same months in the
year after the amnesty
• Ideal counterfactual?

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 10

The treatment effects
• Average treatment effect (ATE)

• Average treatment effect on the treated

(ATT)

• Local average treatment effects (LATE)

(for later: IV and RDD)

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 11

Average treatment effect
Outcome B1:
Outcome A1: do not
Individual 1: treatment effect1 = receive receive
program program

Individual 2: treatment effect2 = Outcome A2 Outcome B2

Individual N: treatment effectN = Outcome AN Outcome BN

Average treatment effect = average(treatment effect1, …, treatment effectN)

Note: N is the population

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 12

 Average treatment effect on the treated
Individual 1: treatment effect1 = Outcome A1: Outcome B1:

Treated …
Individual N1: treatment effectN1 = Outcome AN1 Outcome BN1

Outcome Outcome
Individual N1+1: treatment effectN1+1 = BN1+1
AN1+1
Not
Treated …
Individual N: treatment effectN = Outcome AN Outcome BN

Average treatment effect on the treated =

Average(treatment effect1, …, treatment effectN1)

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 13

ATE vs. ATT
• ATE is usually unobtainable, unless
– Sample is representative from population
– Random allocation between treated and
untreated
• ATT is usually what we get
• Therefore should state for what group is
the estimated treatment effect for

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 14

Exercise: defining treatment effects
For PKH, define:
1. The ATE
2. The ATT
3. The potential difference between ATE and
ATT

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 15