Impact Evaluation Methods in Public Finance

Impact evaluation assesses the changes that can be attributed

to a particular intervention, such as a project, program or policy.
Before Jumping onto to methods there are few basic concepts
that are needed to be defined.
 Casual Effect is the impact of a program that on to a specific
situation or scenario.
 The Policy which is being analyzed is known as Treatment.
 Counterfactual is the imaginary situation which depicts the
situation when the Treatment is not present.
 Control Group is a group of to be treated people that helps in
estimation of Counterfactual. They are basically the people who
do not take part in the program
 Treatment Group is group of people who have participated in the
program
Methods:

The simple difference method.

It is one of the most frequently used methods to measure impacts.
Analysis based on simple differences measure the impact by comparing
the post situation of those people who participated in a program with
group of people that did not. The main assumption of this method is
that those in the comparison group are totally same to those that
participated in the program. A key advantage, and reason for its
frequent use, is that this method does not require data on the situation
prior to the treatment. However, there is a big drawback that if the
treated and comparison groups are different in any way prior to the
program, the method may be biased and may under- or overestimate
the real impact of a policy; that is, selection bias.

Pre- vs. Post-Treatment Period

It is the simplest type of evaluation method. Instead of using a
comparison group it only focuses on the treated group. In this methods
the treated group is analyzed before the program and after the
program. The data gathered after the analysis is then compared to
come up to a solution. Situation of treated group before the
implementation of program is considered as the counterfactual. The
only benefit of this method is it does not require information on people
that did not participate in the program. But this method also has a
drawback that many factors that vary over time can affect an outcome.
Difference-in-Differences Estimations
A difference-in-differences evaluation is a combination of simple
difference and pre-post evaluation method. The effect is calculated by
measuring the change over time for the treated group and for the
comparison group and then taking the difference between these two
differences. The key assumption of this method is the assumption of
common trends. The benefit of this method is it controls for all the
characteristics that do not change over time (both observable and
unobservable) and for all the changes over time that affect the treated
and untreated group in the same manner. The drawback is that it is
typically impossible to assess whether the two groups would have
developed in the same way in the absence of the program. If this is not
the case, the analysis will be biased.

Matching methods

Matching Method compare outcomes of treated individuals with those of similar

individuals that were not treated. participants are matched with individuals that
are identical along selected characteristics but that did not participate in the
treatment. The key assumption of this method is that those who participate in
the program are, on average, identical to their matched peers, except for having
participated in the program. The benefit of this method is it controls for observed
characteristics that do not change over time. The drawback is that it is typically
impossible to rule out that there are not also other, unobserved characteristics
that differ between the groups, which would bias the impact estimation.