DWM ECON335 MEASURING Treatment Effects 2-15-24

ECON 335
Measuring Treatment Effects
We will now turn to an econometric approach which has become popular over the last two
decades; namely, measuring Treatment Effects using Natural Experiments. The analysis will
include, for the most part, materials which are not in the textbook. We will, however, cover
materials in Chapter 13, sections 1 & 4.

☞ Primary focus is on the impact of programs, policies or interventions on economic outcomes.

We refer to the program, policy … etc. as a Treatment.

Ex. Impact of a College Degree on Income - Treatment = Degree

☯ Econometric Ideal is an Experiment

Don’t see them very often in economic settings but see them often in a medical context.

Ex. Efficacy of a drug. Have a treatment group and a control group. The former gets the
pharmaceutical, while the latter doesn’t. Compare differences in outcomes.

Natural (or quasi-) experiments are situations which act like an actual experiment (7 th: 434).

Our analysis will start by describing a general theoretical model concerning measurement of a
Treatment Effect. We will then turn to the typical experimental situation and why it’s desirable.
After that, we will consider Natural Experiments.

Theoretical Modelling of a Treatment Effect (TE)

☞ Ideal Measure: we first consider what we are interested in in the context of the efficacy of a
pharmaceutical. We want to estimate the effect of the drug on an individual.

Our ideal measure would compare the person’s health when they take the drug with their health
when they don’t take the drug.

1
DWM ECON335 MEASURING Treatment Effects 2-15-24

Let y1 be their health when they take the drug & y0 be their health when they don’t take the drug.
We call each of y1 and y2 Potential Outcomes.

For an individual person, we are interested in (y1 – y0) (1).

We cannot estimate the TE for an individual, however, because an individual is in only ever in
one of the two states: they either take the drug or they don’t take the drug.

Focus on “Mean” Treatment Effect: So, what can we do? Is all lost? For the individual, the
answer is Yes. We never observe the person in both states.

All is not necessarily lost, however. Suppose that, instead of seeking to measure the effect for an
individual, we instead decide to focus on the average effect across all people in a population. It is
possible to calculate several measures if we’re willing to focus on an average (or mean) effect.
Let’s take a look at the base model in this context, called the …

Roy Rubin Causal Model

Let’s start by defining the basic elements of the model (A&P Mostly Harmless: 13).
Let D = 1 if a person has received the treatment and = 0 if they have not.

Ex. 1 D = 1 if take the pharmaceutical, and = 0 if don’t.

Ex 2 Wages and College Degree. D = 1 if have a college degree.

Note that we may think of D as defining two distinct groups , where people in one group
received the Treatment and people in the other group didn’t receive the Treatment. Of course,
members of each group might have received the opposite in terms of the treatment.

2
DWM ECON335 MEASURING Treatment Effects 2-15-24

Let y1 represent the outcome of someone if they received the Treatment (e.g., they received the
drug) and
y0 represent the outcome of someone if they did not receive the Treatment (e.g., they did not
receive the drug).

☞ Note that these are measures regardless of whether they are in group D = 1 or D = 0.

People in D = 1 might not have received the treatment, and people in D = 0 might have received
the treatment. Thus, we may use them to represent the counterfactual outcomes which we cannot
observe. To see how, let’s look at some expectations.

☸ Consider the following expectations E(y1|D=1) and E(y0|D=0). What are they?
The first is the expected outcome of a person who has received the treatment given that they
have received the treatment. The second is the expected outcome of a person who hasn’t
received the treatment given that they haven’t received the treatment. Both of these make sense
(in fact, you might be wondering why we have the redundancy; wait a second & you’ll see why).

Can we measure these? Absolutely … Now, let’s turn to the following:

☯ How about E(y1|D=0) and E(y0|D=1)?

The first is the expected outcome of a person who received the treatment (received a college
degree) for an individual who is in the Control group (they didn’t receive the treatment in
reality). Stated in other words, think of this as asking you to take a look at a person in the Control
group and asking what their expected outcome would have been if they had received the
treatment. That’s the counterfactual which we will never observe.

The second: the expected outcome of a person in the Treatment group if they hadn’t received the
Treatment.

Ex. Wages and Education yc = wages with college degree, yhs = wages with a high school
diploma, D = 1 if college degree, and D = 0 if high school diploma.

3
DWM ECON335 MEASURING Treatment Effects 2-15-24

E(yc|D = 0) is … E(yhs|D = 1) is …

☞ Why are we considering these counterfactuals? They help us define what we can do with
available information.

☞ Average Treatment Effect on the Treated (ATET): A TE in which we might be interested

is
E(y1|D = 1) - E(y0|D = 1)

In other words, we look at the people who are in the Treatment group & consider the differences
in their outcomes if they received and didn’t receive the treatment. We call this the ATET.

☯ The problem is that we don’t have the second measure: E(y0|D = 1). We can, however, relate
this measure to what we do observe. Add and subtract E(y0|D = 0) [need to get the 2nd
observable in there] and rearrange, to get

E(y1|D=1) - E(y0|D=1) = [E(y1|D=1) - E(y0|D = 0)] + [E(y0|D = 0) - E(y0|D=1)]

._____________________.
Observe This

We have seen that the first element on the right-hand side is what we observe in the real world.
Let’s represent everything in terms of it. We may rearrange to get

E(y1|D=1) - E(y0|D = 0) = [E(y1|D=1) - E(y0|D=1)] + [E(y0|D = 1) - E(y0|D=0)] (V)

.___________________. ._______________
____.
ATET Selection Bias

Interpretation: what we can observe has two components: (i) the ATET, and (ii) what we call a
Selection Bias measure.

4
DWM ECON335 MEASURING Treatment Effects 2-15-24

Interpret the Selection Bias measure: difference between the two groups in expected outcomes
if they had not received the Treatment. Since members of both groups receive the same
hypothetical treatment this measure captures inherent differences between the two groups.

Ex. Wages and Education: the SB measure is the difference in the mean wages college grads
would have received had they not gone to college and the mean wages of people who didn’t go
to college. Effectively, we might think of it as measuring the difference in innate abilities
between the two groups; they have the same levels of education and we are looking at the
difference in their expected incomes.

Experiments (A&P: 15)

Transition: equation (V) tells us that if we use the observed outcomes to measure the TE it will
capture both the ATET and a Bias measure. If we could just get rid of the bias measure, we
would be in good shape.

☞ So, the question is “is there a way to get rid of the bias measure?”

If we think of the bias measure, we see that it captures inherent differences between the Non-
Treatment and Treatment groups.

So, we might rephrase the question as “is there a way to ensure that the two groups are
effectively the same in terms of unobservable differences?”

☯ Random Sampling: suppose that we obtain two random samples from the population – i.e.,
we randomly sample two groups – and place one into the Treatment group and the other into the
Control group.

Since both groups have been randomly sampled from the same population we have no reason to
believe that they would have systematic differences between them.

So, E(y0|D = 1) = E(y0|D=0) and the bias term will equal zero.

5
DWM ECON335 MEASURING Treatment Effects 2-15-24

☞ This finding is the basis for randomized experiments in the medical field. In economics, we
rarely observe actual experiments. That is why we look for what are called natural experiments.

☞ Examples of the potential benefits of experiments in Economics (A&P: 16).

Ex 1: Government Subsidized training programs for long-term unemployed, drug addicts and
ex-offenders. They are designed to make participants capable of holding jobs & getting higher
wages.

Studies which compared people in the programs and those out of the programs typically found
that the programs reduce earnings! That’s problematic.

In light of our discussion above, what is a possible SB problem inherent in such a comparison.
We might expect that the people most in need of such a program (and, thus, least likely to be
employable) enter the programs.
Studies which have used actual experiments for such programs have found a positive effect.

Ex 2. Kindergarten-12 Education Class size and student learning

In the context of K-12 education, policy makers are often interested in producing the best
outcomes (student learning) at the lowest cost. One of the largest components of cost is class
size, because with smaller class sizes more teachers are needed (and their salaries are a large
component of cost). It is also generally believed that smaller class sizes are associated with better
learning outcomes in students.

Non-experimental studies which compared student outcomes in smaller classes and those in
larger classes generally found no statistically (or economically) significant differences in
outcomes between the students in smaller classes and larger classes, suggesting that spending
money for smaller classes would not produce a desired outcome of better student learning.

What is a potential SB problem? Students in smaller classes tended to be there because they
were having trouble in school; they were lower academic achievers.
An important experiment in this area (the STAR experiment in TN) suggested “a strong and
lasting payoff to smaller classes” (A&P 17).

6
DWM ECON335 MEASURING Treatment Effects 2-15-24

Natural (Quasi-) Experiments (7th: 434. A&P 21)

Actual experiments are expensive and, in economic contexts, not very practical (at times). As a
result, economists often look for situations we call Natural (or quasi-) experiments.
As you might imagine, these are situations which “act like” an actual experiment.

Definition: it is “the result of a policy change or event that occurs completely outside the context
of [a] study ….”

The key factor is that the change is not related to the subject matter of the experiment and, as a
result, the objects of the study (e.g., individuals, cities, states …) are not able to change their
behavior in such a way that composition of the groups changes.

In analyzing NE’s it is important to keep in mind that experiments are beneficial because they
get rid of the selection biases (which capture inherent differences between the people in the
proposed Treatment group and the proposed Control group).

So, in analyzing a given situation proposed as a NE it is important to keep in mind the need to
ensure that the two groups are similar in all other ways.

Examples of Natural Experiments

Ex 1 Hurricane, maternal stress and fetal death

Issue: fetal death (FD) and maternal stress (MS). Does MS result in fetal problems? There might
be unobserved factors which drive both results. So, the following regression

FD = β0 + β1•MS + u

to infer the impact might be biased because u might include (unobserved) factors which affect
FD and MS: MS might be correlated with u. So, the parameter estimate on MS will be biased.
How can we get rid of the bias?

7
DWM ECON335 MEASURING Treatment Effects 2-15-24

Suppose that a hurricane comes through, creating maternal stress. That is a stress not related to
unobserved factors in the mother.

So, we might look at two geographic areas which are otherwise substantially similar, where one
was affected by a hurricane and the other not: New Orleans vs. …. We might then compare mean
FB in these two areas to see if there were differences.

Graphically … [look at areas just outside the path of the hurricane and areas within the path of
the hurricance]

Ex 2 Ex. Card and Kreuger Change in NJ minimum wage law.

April 1, 1992 NJ raises its minimum wage from $4.25 to $5.05.
Issue: did it create unemployment? We’ll look at this study later.

Had data for Feb/Mar and Nov/Dec. Problem, other factors might have changed in the NJ
economy between those two periods. What to do? Look to neighboring areas.

Econometric Methods for Analyzing Experimental Data [S&W: 478. A&P 22]

We will start by considering how we measure a TE when we have experimental data (i.e.,
random assignment to the two groups). It’s actually pretty easy.
Let y = the outcome of interest and D = 1 if one receives the Treatment & = 0 otherwise.

We estimate the following model y = β0 + δ1•D + u (X)

using the OLS estimator. Call it the Difference Estimator.

Control Variables: Differences in Observable Variables

When measuring the TE it is possible (we might want) include other Control variables in the
regression. Note that we call them “Control” variables. In terms of an equation …

y = β0 + δ1•D + β1•x1 + … + βk•xk + u.

8
DWM ECON335 MEASURING Treatment Effects 2-15-24

Notes:
(1) Even though we technically do not have to include Control variables in the regression to
get an estimate the TE, they reduce the standard error of our estimate of δ1 (A&P: 23-24),
allowing us to make more accurate inferences about the TE.

(2) The Control variables should not include experimental outcomes.

Ex. Wages and Education: can’t include hours worked.

(3) The parameter estimates on the Control Variables should not be given a causal
interpretation. They’re designed to reduce the standard errors of our estimates; they
aren’t necessarily included for causal reasons.

☯ While these quasi-experimental examples look “as if” they’re experiments, there is often
reason to believe that the control groups and treatment groups had some pre-existing differences.

We know from our analysis that pre-existing differences can produce bias in the Difference
estimator.

So, we need some way of trying to control for them. One possibility is to look at the differences
between the groups before the treatment. Let’s see what such an approach would imply.

Let ygt be the expected outcome variable for group g = {treatment, control} in time
T = {pre-treatment, post-treatment}.

The Difference estimator may be represented as (ytreatment,post – ycontrol,post)

The pre-treatment difference between the two groups is (ytreatment,pre – ycontrol,pre).

If we subtract the latter from the former we get

DD = (ytreatment,post – ycontrol,post) - (ytreatment,pre – ycontrol,pre) (1).

9
DWM ECON335 MEASURING Treatment Effects 2-15-24

If you had to give this a name, you would call it … the Difference-in-Differences Estimator!!

Second Interpretation of the DD estimator: note that we can rearrange (1) as follows:

DD = (ytreatment,post – ytreatment,pre ) – (ycontrol,post – ycontrol,pre) (2)

The difference in the treatment groups pre and post and the difference in the control group pre
and post. The first difference captures the change in the treatment group, while the second
difference captures the change in the control group. The latter change, presumably, represents
and change that would have happened to the treatment group had it not undergone the treatment.

Note that this is an important assumption: we assume that the change in the Control group
between the two periods would have happened to the Treatment group had it not received the
treatment.

This is called the Common Trends Assumption. We will look at this point when we consider a
[Card-Kreuger) example.

☞ The DD Estimator in a Linear Regression: this interpretation allows us to represent the DD

estimator in terms of a simple linear regression. There are two possible representations: one not
used often and the other more standard.

☸ Standard representation of the DD estimator. Let D = 1 if in the treatment group and = 0 if a

person is in the control group. Suppose that we have two periods (pre-treatment and post-
treatment). Let T = 1 for the post-treatment period and = 0 for the pre-treatment period.
Consider the following regression:

y = β0 + β1•D + β2•T + β3•D•T + u.

↑
Note the interaction term.

10
DWM ECON335 MEASURING Treatment Effects 2-15-24

☞ It turns out that the DD estimator is the parameter on the interaction term: D•T. Let’s derive
this result:

ytreatment,pre = E(y|D=1,T=0)) = β0 + β1•1 + β2•0 + β3•1•0 = β0 + β 1

ytreatment,post = E(y|D=1,T=1) = β0 + β 1 + β 2 + β 3

→ (ytreatment,post – ytreatment,pre ) = β2 + β3.

Note that this should include the TE plus any bias.

ycontrol,pre = E(y|D=0,T=0) = β0 ycontrol,post = E(y|D=0,T=1) = β0 + β2

→ (ycontrol,post – ycontrol,pre) = β2

This is supposed to be the bias term (in terms of the second interpretation of the DD estimator, it
represents the change between the two periods for the control group).

If we take the difference between the two differences, we get (β2 + β3) - β2 = β3
the parameter on the interaction term.

13-2 Policy Analysis with Pooled Cross Sections (7th: 431)

Pooled Cross Section: have random samples taken from a population a different points in time
(typically different years).

Ex. Current Population Survey: It provides a comprehensive body of data on the labor
force, employment, unemployment, persons not in the labor force, hours of work, earnings, and
other demographic and labor force characteristics. Sample around 60,000 households.
So, have it for 2016, 2015, 2014, etc.
Note that it differs from panel data in that you don’t have the same individuals every year. The
exact individuals will vary from year to year. So, we don’t necessarily have the same
econometric advantages of this data as we have with panel data (can’t get rid of the individual-
specific effects).

11
DWM ECON335 MEASURING Treatment Effects 2-15-24

We can, however, still gain insights into the impact of public policies on “mean outcomes in the
population.” Note this last point: since we’re focusing on mean outcomes in natural experiments,
we can gain insight into those means with random samples from the population at different
points in time.

Let’s take a look at how we can use Pooled Cross Sectional data to gain insights in the context of
natural experiments. I will do it with the following example(s).

Minimum Wage Laws: Card and Kreuger (A&P: 228. S&W: 497)
April 1, 1992 NJ raises its minimum wage from $4.25 to $5.05.

Have heard the argument that increasing the minimum wage will  unemployment. Is that true?
That’s the issue this article seeks to address.

We might think of comparing the employment in NJ before the change and right after the change
to determine the impact. In the study they looked at the February/March before the change and
the November/December after the change. Econometrically, let D = 0 for the February and D =
1 for the November, we might estimate
yit = β0 + β1∙Dt + εit

where yit is employment at fast food store i in period t,

and interpret β1 as the impact of the change in minimum wage law in NJ on employment.
(We would expect it to be negative if an increase in the minimum wage caused unemployment.)

 Such a comparison might be problematic, however. Why? Other things, like the economy,
might have changed between the points of comparison.

Thus, β1 would pick up not only the impact of the change in minimum wage laws but also
changes over time.
So, what might we do? … from our earlier examples we know that we might look for another
substantially similar region which didn’t undertake such a change.

12
DWM ECON335 MEASURING Treatment Effects 2-15-24

That’s where southeastern PA comes in. Note that NJ and the Eastern PA region considered are
contiguous [MAP]. E.g., the Philadelphia-Camden area. So, we might view them as being
substantially similar.

During this same period, the minimum wage in Pennsylvania was constant at $4.25.
This type of change sets up a nice natural experiment in that we have a relatively homogeneous
area where a change in policy affects only a subset of the area.

Their data: fast food restaurants in New Jersey and eastern Pennsylvania from the Burger King,
KFC, Wendy's, and Roy Rogers chains

Treatment Group: we can think of the fast food restaurants in NJ as having received a treatment:
the change in law.
Control Group: we can think of the fast food restaurants in PA as not having received a
treatment.

Econometric Model: letting x1 = 1 if a store was in NJ, and = 0 if it was in PA, we might think
of estimating
yit = β0 + β1∙ Dit + β2∙x1it + β3∙Dit∙x1it + uit.

Let’s calculate the DD estimate using mean values of the data (A&P: 230). Note that this works
only if there are no other regressors. Stated in another way, this works when we only have the
two dummy variables: D and x1.

Variable PA NJ Difference
FT Employment, Pre 23.33 20.44 -2.89
FTE, Post 21.17 21.03 -0.14
Change -2.16 .59 2.75
*FTE: average FT employment at a restaurant.

The change in NJ: 0.59. Opposite of what we’d expect.

13
DWM ECON335 MEASURING Treatment Effects 2-15-24

This does include the potential bias due to changes in the economy. We estimate the latter using
the PA data.
The change in PA was -2.16, suggesting a general drop off in the economy between the two
periods.
DD Estimate: 0.59 – (-2.16) = 2.75.

The estimate suggests that the increase in the minimum wage resulted in an increase in
employment; the opposite of what some people argue.

Potential Problem with the estimate: the control group isn’t representative of the NJ
counterfactual.

I have noted that implicit in the estimator is what is called a “Common Trends” assumption;
basically, that the two economies follow the same trend (making PA representative of the
counterfactual NJ). If this assumption doesn’t hold, then the estimates will be biased.

Graphically, we have something like … We assume that the Treatment group would have
dropped by the same amount as the Control group did. The difference between what was actually
true for the Treatment group and this counterfactual is called the Treatment Effect.

Subsequent analyses of changes in employment in the two areas over time indicated that
employment in the two economies (A&P), suggesting that SE PA might not have been an
appropriate control region.
14
DWM ECON335 MEASURING Treatment Effects 2-15-24

Ex Our Change in Oregon Law – Defensive Medicine Article

Issue: do physicians perform more cesarean sections (vs. vaginal births) out of fear of being
subject to a lawsuit?

Ex. If make a mistake could be subject to damages in the millions of $US. (Will have insurance
but still have damage to reputation and costs of going thru a lawsuit.)

Policy: “Tort Reform” limits on the amount of damages a person can recover in court.

Natural Experiment: The Portland, OR/WA area. Covers two states. One state (OR) overturned
a law limiting damages. As a result, doctors faced greater potential damages in a lawsuit.

[rough map of the area]

Doctors in OR were subject to the change while doctors in WA were not subject to the change.
Like the Card and Kreuger case.

Let
T = 0 for time period before the change, and = 1 for the period after the change.
G = 1 if in OR and = 0 if in WA.
y = 1 if have a c-section.

y = β0 + β1∙G + β2∙T + β3∙G∙T + u.

15