Impact Evaluation (Prof Lee Jong Wook)

Clara Joan Joachim (2021-26084)

# Question 1

1.1 Discuss why you use per capita household expenditure amount instead of per capita household
income amount to analyze the program impact on household income.

In my opinion, per capita household income amount is still a good measure for the outcomes of the
program. However, an overall impact evaluation of the program is much more reasonable by utilizing on
the outcome of per capita household expenditure. This is because the changes of household incomes could
possibly be affected by vary resources and not only due to the program that we have implemented. Hence,
it will be much difficult for investigator to detect on the program impact. While household expenditure has
a long-term effect to the participants and show absolute differences prior and post program implication.

1.2 Conduct a balance test for the baseline variables.

To my understanding, we can do baseline test either by using command t-test, t-test by command global
or regression. Regardless, they still show the same result.

For instance, t-test for agehead_pre on the left, we could see at the bottom analysis, none of the
P-values are less than 0.05, hence they are not significant in 95% confidence interval. Therefore, we cannot
reject the null hypothesis and we could say that the age head for both treatment and control group have
very random values. Besides, when we look at the difference between the mean for both groups, the
treatment does not give any impact to the treatment group by age of household. All the other 14 variables
are not significant in 95% confidence interval but the egg_pre. On the right-side image on t-test of egg_pre,
the corresponding two tailed P-value is 0.0465, which is less than 0.05. Hence, we could conclude that the
difference of means between treatment and control group is not different from 0, and that the price of the
egg in both groups are not randomized.
 When we do a regression (left picture), the analysis will come out that one variable, expnfd_pre is
being omitted. This is due to collinearity, in another words, there is dependency of expnfd_pre to the other
independent variables. This dependency can be identified by regressing expnfd_pre as dependent variable
and the remaining variables as the independent variables. Right picture, it shows that expnfd_pre is
correlated with exptot_pre and lnexptot_pre.
There was argument on whether balance test is required or otherwise. Balance test is informative
firstly, when we concern the randomization of our observation, and secondly, when we want to look at
baseline characteristics for a sample with attrition. However, some economists stated that the test arises
issues such as correlation on baselines with outcome interest are more important than statistical
significance for imbalances. Besides, it was argued that researchers might use this test improperly by
reporting fewer significance results than expected by any chance.1

1.3 In ideal RCTs, all households in the treatment group participate in the program, and all households in
the control group do not participate in the program. In this perfect compliance case, we estimate ATE by
comparing average outcomes and by using OLS regression.

1D. Mckenzie, 2017, World Bank Blog. Retrieved from, https://blogs.worldbank.org/impactevaluations/should-we-require-

balance-t-tests-baseline-observables-randomized-experiments
The estimation of ATE among the potential beneficiaries is a simple difference of mean between the
treatment group and the control group. The program impact is 0.1682131. As the p-value is close to 0, this
microfinance program impact is significant at 0.0001 significant level.

1.4 If the RCT is implemented perfectly, adding additional control variable does not change the estimated
effect size. When you add additional control variables and estimate ATE, compare the results with the
results in 1-(3). What is the benefit of adding control variables in RCTs?

After adding control variables, sexhead, educhead and famsize, the result shows that P-value on
sexhead is larger than 0.05, therefore it shows no significant impact to per capita total expenses based on
gender of household head. Besides, it shows that educhead has P-value of 0.000. Therefore, we can
conclude that higher education of the family head gives strong significancy towards the program impact.
While for famsize, the P-value is 0.007, means that family size gives significant effect to the program impact.
We could say that the larger the family, they will benefit less from the program. The confidence interval of
the impact in microfinance program becomes smaller as more additional of constant variables being added.

We could also compare the differences of constant variable effect to the effect by bivariate
correlation using command pwcorr. The result shows that sexhead is not correlated to the lnexptot but
correlated to the treatment. On the other hand, educhead shows correlated value to lnexptot but not to
treatment. When we control educhead, the relation between lnexptot and treatment will be weakened.
When we add a control variable, we could make our estimate more precise by reducing the
variance. Including a control variable that are strongly correlated with the outcome variable can reduce the
amount of unexplained variance and sample size needed to detect the program effect. However, if we
include a variable that are influenced by the treatment, it will create bias to our estimate. The negative side
including many control variables will otherwise reduce the precision of our estimate. It is always a good
choice to collect control variables before randomization.
1.5 In your study, you found imperfect compliance. The actual program participation is observed with a
variable P. Compare the likelihood of program participation in the treatment group and control group.

Imperfect compliance is a condition when we offer the microfinance to everyone, the households
that assigned to treatment group are not receiving the program, while the households in control group
might accept the treatment. In this case, instead of ATE, we will be estimating the Intention to Treat (ITT).
Here, we will have another variable P, P=1 when they take the credit and P=0 when they did not take the
credit.

As we can see from the tabulation, from the control group of 150 households, there are 11
households that took the credit. While in the treatment group of 150 households, there are 15 households
that did not take the credit. We could also have an overview of this result in term of proportion by using
command bysort.

From here, we could see at the mean values that 0.07 of the observed households in the control
group took the credit while 0.9 of the observed households in the treatment group really took the credit.
We could estimate the ITT by looking at the difference of their mean values.

ITT = E(𝒀𝟏 − 𝒀𝟎 | 𝑷 = 𝟏) 𝐏𝐫(𝑷 = 𝟏)

ITT = 0.9 – 0.0733333 = 0.827

1.6 Estimate the reduced form and the share of compliance. Then estimate LATE using the reduced form
and the share of compliance. Compare the result with your LATE estimation using 2SLS. Also, estimate LATE
using 2SLS with additional control variables and adjust the standard errors appropriately.

We first calculate the reduced form or ITT form (left image), we had in question 1.5, the coefficient of the
treatment is 0.6821. Then, we found that the share of compliance (right image) the coefficient is 0.8266.
By manually calculation, the local average treatment effect is 0.2034.

#$%&'$% )*+, 4.67896

LATE = = = 0.2034
-./+$ *) '*,012/3'$ 4.8977

From this estimation, we got the LATE value at coefficient of participating which is 0.2034.
# Question 2

From the table above, we could say that the number of observations from both years are the same,
4011. Those who participated in the program are only 2051 from 4011 observations during the
implementation. If the implementation of the program begins in 2003, we create a dummy variable “time”
with value of 0 before 2003 and 1 for 2003 onward.
In this estimation, we will have two linear models:

1) 𝐲𝐫𝐬𝐜𝐡𝒕 = 𝐏𝐱 𝒕 + 𝛆𝒕
2) 𝐞𝐧𝐫𝐨𝐥𝐥𝒕 = 𝐏𝐱 𝒕 + 𝛆𝒕

2.1 Cross Section Estimator

In this program, we will have two outcomes, years of schooling (yrsch) and school enrollment (enroll), both
with a dummy variable P for participations in the program and otherwise. Let’s first see on our first
outcomes by using OLS.
 After clustering them, for each household that accepted the treatment, the year of schooling is
decreases. Therefore, in this model we could say that the program is not impactful. We will then do the
same regression again with another outcome interest, the enrollment of child in school for 2004. Since
enrollment is a dummy variable, in this regression, we will use command “logit”.

After clustering them, and adjusting the standard error, we could see that the coefficient value is
negative for the treatment and therefore, we could say that for each household that accept the treatment,
the enrollment of child into school is decrease. At this point, we could not conclude yet whether this
program gives positive impact or not. By using only participation as the independent variable, it will be
more difficult to predict on the dependent variable (impact of the program). Therefore, we might need to
check on other independent variables to make the prediction more precise.

2.2. Before-After Estimator

 From the regression estimation of yrsch, we will see the positive coefficient which is different from
the estimation from our previous estimation by using cross section estimation. By using before-after
estimator, we could say that every participation will increase the time of schooling in each household.
Meanwhile, for enroll, the negative coefficient shows that household that participating the program will
have more possibility of not attending school in 2004. Therefore, it shows a negative impact of the program
to the enrollment of child to school.

2.3 Difference-in-difference Estimation

While, we have dummy variable P to identify the group exposed to the treatment. Then, we create an
interaction between time and treated. We will call this interaction DID.

The result on the left shows that the program has no positive impact for the year of schooling
while on the left, the program also has no positive impact to the enrollment of child in the year 2004.

2.4 Compare the result. Which one is the most reliable and credible? Result for Simple Regression:

Estimated program impact on year of schooling

-0.113
Cross Section Estimator
(0.093)
1.353***
Before-After Estimator
(0.018)
-0.113
Difference-in-difference Estimator
(0.093)

Estimated program impact on enrollment status of child in school

-0.0938
Cross Section Estimator
(0.099)
-0.3187***
Before-After Estimator
(0.046)
-0.0938
Difference-in-difference Estimator
(0.099)
Standard errors are clustered. *** = 0.001 significant level.
 Based on the estimation, we could see that before-after estimator has very small standard error
and the estimations are significant under 99.9% confident level. Therefore, I would say that by using only
treatment as the independent variable, before-after estimator is so far the best estimator.

2.5 Regress by adding another control variables, per capita household income (pchhinc = hhtotinc/tothh)
and maximum level of education (maxedu).

Now we have two new multiple regression models for our estimator:

1) 𝐲𝐫𝐬𝐜𝐡𝒕 = 𝐏𝐱 𝒕 + 𝐩𝐜𝐡𝐡𝐢𝐧𝐜𝒕 + 𝐦𝐚𝐱𝐞𝐝𝐮𝒕 + 𝛆𝒕

2) 𝐞𝐧𝐫𝐨𝐥𝐥𝒕 = 𝐏𝐱 𝒕 + 𝐩𝐜𝐡𝐡𝐢𝐧𝐜𝒕 + 𝐦𝐚𝐱𝐞𝐝𝐮𝒕 + 𝛆𝒕
2.5.1 Cross Section Estimator

2.5.2 Before-After Estimator

2.5.2 Difference-in-difference Estimator

Following is the result of the Multiple Regression is tabulated together with Simple Regression

Estimated program impact on year of schooling

-0.113
Simple regression
(0.093)
Cross Section Estimator
1.0055***
Multiple regression
(0.054)
1.3530***
Simple regression
(0.018)
Before-After Estimator
1.3530***
Multiple regression
(0.018)
0.1382
Simple regression
(0.079)
Difference-in-difference Estimator
-0.113
Multiple regression
(0.093)

Estimated program impact on enrollment status of child in school

-0.0938
Simple regression
(0.099)
Cross Section Estimator
-0.2436***
Multiple regression
(0.071)
-0.3187***
Simple regression
(0.046)
Before-After Estimator
-0.3196***
Multiple regression
(0.046)
-0.0938
Simple regression
(0.099)
Difference-in-difference Estimator
-0.0516
Multiple regression
(0.100)
Standard errors are clustered. *** = 0.001 significant level.

After regressing them with additional control variables, comparing all three estimators, we could
conclude that Before-After estimator has the most consistent, precise, and significant estimator comparing
to the other two estimators. Analyzing from Before-After estimation, we could say that PROGRESA program
does give impact to the year of schooling, as child in household gain the benefit of the program and attend
school longer. Meanwhile, for the outcome of child enrollment to school, PROGRESA program apparently
did not give a positive outcome as once they are participating, every child in the household enrolling to
school is decreasing in 2004.
2.6 Repeat 2.1 through 2.5 to estimate separate effects for boys and girls.

In this program as we assumed implemented on year 2003, from 4011 participants, there are 1969
girls (gender = 0) and 2042 for boys (gender = 1). Now, we wanted to see the evaluation impact for the
program from each estimator for both genders.

2.6.1 Program impact for girls:

Estimated program impact on year of schooling

0.7888***
Simple regression
(0 .085)
Cross Section Estimator
0.00005**
Multiple regression
(0.00002)
1.3730***
Simple regression
(0.0231)
Before-After Estimator
1.3737***
Multiple regression
(0.023)
-0.2065*
Simple regression
(0.125)
Difference-in-difference Estimator
0.1074
Multiple regression
(0.111)
Estimated program impact on enrollment status of child in school
-0.3018**
Simple regression
(0.095)
Cross Section Estimator
-0.2621**
Multiple regression
(0.097)
-0.2910***
Simple regression
(0.067)
Before-After Estimator
-0.2924***
Multiple regression
(0.067)
-0.3018**
Simple regression
(0.099)
Difference-in-difference Estimator
-0.1048
Multiple regression
(0.136)

Standard errors are clustered. *** = 0.001, ** = 0.01 and * = 0.05 significant.
2.6.2 Program impact for boys:

Estimated program impact on year of schooling

0.8803***
Simple regression
(0 .085)
Cross Section Estimator
-0.00021**
Multiple regression
(0.00002)
1.335***
Simple regression
(0.023)
Before-After Estimator
1.3341***
Multiple regression
(0.023)
-0.0236
Simple regression
(0.124)
Difference-in-difference Estimator
0.1695
Multiple regression
(0.109)
Estimated program impact on enrollment status of child in school
-0.2405*
Simple regression
(0.097)
Cross Section Estimator
-0.2237*
Multiple regression
(0.098)
-0.3438***
Simple regression
(0.062)
Before-After Estimator
-0.3445***
Multiple regression
(0.062)
-0.2406*
Simple regression
(0.095)
Difference-in-difference Estimator
-0.0019
Multiple regression
(0.133)

Standard errors are clustered. *** = 0.001, ** = 0.01 and * = 0.05 significant.

In this section, the discussion is generally be done based on the overview of estimation using
Before-After estimator. From the result above, we could see that comparing both girls and boys, the
outcome interest of years children going to school, PROGRESA program is slightly more impactful to girls
than boys. Comparing to the coefficient of multiple regression of yrsch, the girls gain about 3% more benefit
than the boys. In other words, the girls attend school a little longer than the boys after receiving the
treatment.
On the other hands, for the outcome interest of enrollment child to school in 2004, none of both
genders receive any benefits from PROGRESA program but otherwise. The boys have more negative
coefficient compared to girls. We could say that, at the time of 2004, there are more boys that is not
enrolling to school. After the program was implemented, the percentage of boys not going to school has
dropped about 34% while girls, 29%.