Applying the Synthetic Control Method

Illustrating with the Public Resource Management Program for Assam

Arnab Mukherji Hiranya Mukhopadhyay
Conference on Impact Evaluation: Methods, Practices, and Lessons 11 July 2012, ADB Auditorium Zone A

Section 1
Evaluation and the Synthetic Control Method

Evaluation is about Developing Parallel States of the world

Before

After - Restructuring Road

Implicitly: the missing counterfactual is ?

3

Public Policy Evaluations

Have traditionally used the Before/After Design frequently. (e.g. See RBIs State-Wise Analysis of Fiscal Performance) Issues Easy to collect time-series data Identification rests on pre-intervention acting as a counterfactual for post-intervention Concern: Does the pre-intervention status really give a good sense of what would have happened in the absence of the intervention?

Before/After unable to disentangle other effects .

FD/GSDP

Program Works

FD/GSDP

Program makes things worse!

Impact < 0 Impact > 0

T= Pre T= Post

Time (T)

T= Pre T= Post

Time (T)

Counterfactual Observed Treatment Group

Difference in Difference tells us

FD/GSDP

Our best estimate of what would have happened in the absence of the intervention C Impact = T -C

T= Pre

T= Post

Time (T)

Control Group Observed Treatment Group

A DiD Analysis raises

A number of concerns centered on the validity of the control group: Are the treatment and control groups similar on key dimensions?- Selection Bias Or, do we use all untreated units in the control group symmetrically? Strengths Clearly identified control group Omitted Variable Bias less of a concern Needed micro data until recently
7

Synthetic Control Group- Effect of Terrorism in Basque County

Discuss basic question AER 2003 paper Critical question 1-on which dimensions would we like synthetic Basque to be identical? Critical question 2 -Which convex combination of states can be used to construct a statistically indistinguishable version of pre-intervention Basque? Solve for
Variable weights Control State weights

Goal is
reproduce pre-intervention outcome trends Statistical balance on key predictors across Basque and Synthetic Basque
8

Algorithm works as follows:

Let (YitI,YitN) denote an outcome in state i in year t; I, N indicate if the state received PRMA or not. T0 is the intervention year thus, for all t in {1, , T01} and for i in {1, , N}, we want YitI = YitN. Goal is identify a set of weight for each control state such that

is minimizing: where is the set of predictors of Y for Assam and for other states and V is a diagonal matrix with specific weights for X.
9

An Example

Suppose there are three states Assam (Program state), Maharashtra and Tamil Nadu There are two predictors of the outcome variable (Y) : X1 and X2 The Minimization exercise is: Min: V1(X1A- X1MWM)2 + V2(X2A- X2TWT)2 Subject to WT,WM>0 and WT + WM = 1

But we still need to determine Vs Vs are determined by minimizing the prediction error of the outcome variable (Y) for Assam from the synthetic Assam during the pre-intervention period.

W vs. V

W gives us which states are to contribute to the synthetic control unit V ensures that the chosen synthetic unit matches Assam in it its pre-intervention outcomes. W is a function of V we have a two stage simultaneous optimization problem computationally intensive Currently exists as a package in R a statistical programming language.

11

Main Advantages
It is a data-driven process- reduces selection bias. Unlike Propensity Score Matching, does not require large data set. Ideal for Macro interventions. This method provides safeguard against extrapolation.

Basque and Synthetic Basque

Section 2:
Case Study: Public Resource Management in Assam

14

PRMPA: Policy Response for Assam

15

Focus on Two Key Components

Key Evaluation Challenge: What would have happened in the absence of the PRMPA?
16

Section 3
Findings

17

Two Outcomes
Own Tax Revenues/GSDP Pre- Intervention Indicators
Variable Log(GSDP) Non-Agriculture/GSDP Per Capita # of Factories Synthetic Assam Assam 10.594 0.661 0.057 10.587 0.663 0.059

Interest Payment to GSDP Pre- Intervention Indicators

Variable Assam log(GSDP) 10.5937 Debt/ GSDP 29.46% Average Effective Interest Rate 10.45% Synthetic Assam 10.60007 31.10% 10.22%

Synthetic Control Karnataka, Meghalaya, Tamil Nadu, Chhattisgarh, and Mizoram

Synthetic Control Meghalaya, Uttar Pradesh, Orissa, West Bengal

18

Interest Payments/GSDP Ratio

19

Own Revenues/GSDP Ratio

20

Time Trends with Fiscal Deficit

21

Broad Inferences
With the three years of follow-on data Short-term gains have been achieved with respect to own tax to GSDP ratio - an increase in the order of 0.35% to 1.1% in the post-intervention period, averaging about 0.71% No evidence of short-term gains achieved with respect to a decline in interest payments to GSDP ratio Future trends will confirm however, it would be interesting to learn about the pattern and nature of debt swap Longer term horizon needed for full evaluation.
22

Thank You

23