You are on page 1of 80

Ch.

13: Experiments and


Quasi-Experiments
Econ 141 Spring 2014
Lecture: April 28 & 30, 2014
Bart Hobijn
4/28&30/2014 Econ 141, Spring 2014 1
The views expressed in these lecture notes are solely those of the instructor and do not necessarily
reflect those of the UC Berkeley, or other institutions with which he is affiliated.
Sources of exogenous variation
Chapter 12 showed that successful identification of
causation requires exogenous variation in the
explanatory variable in the regression.
Ch.13 introduces two types of such variation
1. Experiments
Situation in which exogenous variation is explicitly
generated by the researcher.
2. Quasi-experiments
Situation in which particular circumstances generated
exogenous variation in the data.
Practical examples of experiments and quasi
experiments.
4/28&30/2014 Econ 141, Spring 2014 2
Experiment setup
Estimate causal effect of treatment on
outcome (Jargon comes from medical sciences)

Potential outcome
Outcome,

, for an individual under potential


treatment,

.
Causal effect is difference in outcome if treatment is
received and when it is not received.
Average treatment (causal) effect
Average causal effect across population of interest
4/28&30/2014 Econ 141, Spring 2014 3
Difference estimator
Regression model

=
0
+
1

outcome

treatment

1
average treatment effect
Parameter that we want to consistently estimate

control variables
effect of control variables not
necessarily consistently estimated

residual of the regression


4/28&30/2014 Econ 141, Spring 2014 4
Randomized controlled experiment
Researcher controls whether an individual is
subjected to treatment.
Treatment can be randomly assigned to
participants in experiment.
Fully random or
Based on observables to better approximate
population.

Treatment assigned to assure exogenous
variation in

.

4/28&30/2014 Econ 141, Spring 2014 5
Quasi controlled experiment
Whether an individual is subjected to
treatment or not is not determined by the
researchers.
Particular circumstances resulted in random
assignment of treatment to participants in
experiment.

(Part of) variation in treatment across
individuals exogenous.

4/28&30/2014 Econ 141, Spring 2014 6
Five studies as examples
Experiment
Duration Dependence and Labor Market Conditions: Evidence from a
Field Experiment
Kroft, Lange, and Notowidigdo (QJE, 2013)
Quasi-experiment
Household Expenditure and the Income Tax Rebates of 2001
Johnson, Parker, and Souleles (AER, 2006)
Using Electoral Cycles in Police Hiring to Estimate the Effect of Police
on Crime
Levitt (AER, 1996), comment by McCrary (AER, 2002)
Minimum Wages and Employment: A Case Study of the Fast Food
Industry in New Jersey and Pennsylvania
Card and Krueger (AER, 1994)
Did Securitization Lead to Lax Screening? Evidence from Subprime
Loans.
Keys, Mukherjee, Seru, and Vig (QJE, 2010)
4/28&30/2014 Econ 141, Spring 2014 7
Focus on same points across papers
1. Research question
2. Regression specification
3. Potential threats to internal validation
4. Instrument/experiment to resolve threats
5. OLS and IV results
6. Hypothesis tests
7. Interpretation
4/28&30/2014 Econ 141, Spring 2014 8
Experiment
Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment
Kroft, Lange, and Notowidigdo (QJE, 2013)

4/28&30/2014 Econ 141, Spring 2014 9
Duration dependence of finding job
4/28&30/2014 Econ 141, Spring 2014 10
Causes of duration dependence
Sample selection / unobserved heterogeneity
The best workers get hired first and contribute to high job-
finding rate of unemployed at short durations. The pool of
unemployed workers at longer durations just consist of ones
that are less likely to get hired.
Pure duration dependence
The longer an individual worker remains unemployed the
less likely she or he is to find a job in subsequent months.
4/28&30/2014 Econ 141, Spring 2014 11
Research question
Sample selection / unobserved heterogeneity
The best workers get hired first and contribute to high job-
finding rate of unemployed at short durations. The pool of
unemployed workers at longer durations just consist of ones
that are less likely to get hired.
Pure duration dependence
The longer an individual worker remains unemployed the
less likely she or he is to find a job in subsequent months.

Do potential employers hold the time someone has been
out of a job against them and are they less likely to call
back those who have been unemployed longer for an
interview when they apply for a job?
4/28&30/2014 Econ 141, Spring 2014 12
Regression specification
Linear probability model

,
=
0
+
1

,
+
2
log
,
+
,
+
,

individual, city (MSA) person lives in

,
= 1 if gets called back for interview after
applying for a job. Is 0 otherwise

,
Dummy if employed at other job at
time of application.

,
Duration of unemployment if not
employed at other job.

,
Set of conditioning variables
4/28&30/2014 Econ 141, Spring 2014 13
Regression specification
Linear probability model

,
=
0
+
1

,
+
2
log
,
+
,
+
,

individual, city (MSA) person lives in

,
= 1 if gets called back for interview after
applying for a job. Is 0 otherwise

,
Dummy if employed at other job at
time of application.

,
Duration of unemployment if not
employed at other job.

,
Set of conditioning variables
4/28&30/2014 Econ 141, Spring 2014 14
Research question
boils down to asking
whether

< or not.
Why not apply OLS?
Linear probability model

,
=
0
+
1

,
+
2
log
,
+
,
+
,

individual, city (MSA) person lives in

,
= 1 if gets called back for interview after
applying for a job. Is 0 otherwise

,
Dummy if employed at other job at
time of application.

,
Duration of unemployment if not
employed at other job.

,
Set of conditioning variables
4/28&30/2014 Econ 141, Spring 2014 15
Unobserved Heterogeneity:
Workers with undesirable unobserved
characteristics will be unemployed
longer and be less likely to be called
back for an interview when they apply
for a job.
cov
,
,
,
< 0
Threat to internal validation
Experiment is a solution
Linear probability model

,
=
0
+
1

,
+
2
log
,
+
,
+
,

individual, city (MSA) person lives in

,
= 1 if gets called back for interview after
applying for a job. Is 0 otherwise

,
Dummy if employed at other job at
time of application.

,
Duration of unemployment if not
employed at other job.

,
Set of conditioning variables
4/28&30/2014 Econ 141, Spring 2014 16
Send out a set of, otherwise identical,
resumes to apply for job openings where the
only difference is the treatments
,
and
log

.
Estimate whether there is a significant
difference in call back rates as a function of
the duration of unemployment,

.
Description of treatments
4/28&30/2014 Econ 141, Spring 2014 17
Description of sample
4/28&30/2014 Econ 141, Spring 2014 18
Table describes the
composition of the
sample of 12,054
resumes that
researchers sent
out, the call back
rate they got, and
the types of jobs
they applied for.
Show treatment is randomly assigned
4/28&30/2014 Econ 141, Spring 2014 19
Experiment is a solution
Linear probability model

,
=
0
+
1

,
+
2
log
,
+
,
+
,

individual, city (MSA) person lives in

,
= 1 if gets called back for interview after
applying for a job. Is 0 otherwise

,
Dummy if employed at other job at
time of application.

,
Duration of unemployment if not
employed at other job.

,
Set of conditioning variables
4/28&30/2014 Econ 141, Spring 2014 20
Because this is an experiment we have
generated all variation in
,
and
,
as
exogenously.
Result is that we can estimate equation using
OLS for the data we obtain from the
experiment.
Table with main result
4/28&30/2014 Econ 141, Spring 2014 21
Table with main result
4/28&30/2014 Econ 141, Spring 2014 22
Figure illustrates the point
4/28&30/2014 Econ 141, Spring 2014 23
Interpretation
Very convincing evidence that employers are
less likely to call back persons who have
been unemployed for more than 6 moths for
an interview when they apply for a job.

Call back rate drops by about 3 percentage
points.

4/28&30/2014 Econ 141, Spring 2014 24
Quasi-Experiment
Household Expenditure and the Income Tax Rebates of 2001
Johnson, Parker, and Souleles (AER, 2006)

4/28&30/2014 Econ 141, Spring 2014 25
Research question
Permanent Income Hypothesis (PIH)
the hypothesis states that a change in permanent income,
rather than a change in temporary income, affects the choices
that determine a consumer's consumption patterns. The key
conclusion of this theory is that transitory, temporary changes
in income have little effect on consumer spending behavior,
whereas permanent changes can have large effects on
consumer spending behavior.
4/28&30/2014 Econ 141, Spring 2014 26
Research question
Permanent Income Hypothesis (PIH)
the hypothesis states that a change in permanent income,
rather than a change in temporary income, affects the choices
that determine a consumer's consumption patterns. The key
conclusion of this theory is that transitory, temporary changes
in income have little effect on consumer spending behavior,
whereas permanent changes can have large effects on
consumer spending behavior.

Did consumers change their consumption level upon receipt of
the one-time 2001 federal income tax rebates?
4/28&30/2014 Econ 141, Spring 2014 27
Regression equation

,+1

,
=
0,

,
12
=1
+

1
+
2

,+1
+
,+1


,
Household s consumption in month

,
Month dummy for which month time falls in

Set of conditioning variables



,+1
Size of the one-time tax rebate a household r
receives at time +1.

4/28&30/2014 Econ 141, Spring 2014 28
Permanent Income Hypothesis

,+1

,
=
0,

,
12
=1
+

1
+
2

,+1
+
,+1


,
Household s consumption in month

,
Month dummy for which month time falls in

Set of conditioning variables



,+1
Size of the one-time tax rebate a household r
receives at time +1.

4/28&30/2014 Econ 141, Spring 2014 29
Permanent Income Hypothesis
For households that satisfy the PIH the one-time
tax rebate which affects their current income but
barely their permanent income it should be the
case that they do not change their consumption
in response to the tax rebate. That is, the PIH
implies the null hypothesis

= .
Why OLS might be inconsistent

,+1

,
=
0,

,
12
=1
+

1
+
2

,+1
+
,+1


,
Household s consumption in month

,
Month dummy for which month time falls in

Set of conditioning variables



,+1
Size of the one-time tax rebate a household r
receives at time +1.

4/28&30/2014 Econ 141, Spring 2014 30
Observed and unobserved characteristics that affect the
level of consumption, like the level of different types of
income, a households preference for leisure, etc., might
not only affect the level of consumption (and the size of
its change) but also the size of the tax rebate a household
receives.
If that is the case then cov
,+1
,
,+1
0
and OLS not consistent.
Timing of Rebate Instrumental Variable
Timing of month in which households received
the tax rebate depended on their social security
number.
This choice of timing meant random timing of
rebate across households.
Let
,+1
= 1 if household received tax
rebate in month +1 and 0 otherwise.

,+1
is positively correlated with
,+1
(valid)
and random across households (exogenous).
4/28&30/2014 Econ 141, Spring 2014 31
Table with main results from OLS
4/28&30/2014 Econ 141, Spring 2014 32
Reduced form regression
4/28&30/2014 Econ 141, Spring 2014 33
Putting instrument

in as regressor in
IV regression rather
than fitted values of
endogenous
variables

is called
reduced form
regression.
It always has a
better fit than the IV
regression
Table with main results from OLS
4/28&30/2014 Econ 141, Spring 2014 34
Note the footnote to this table
4/28&30/2014 Econ 141, Spring 2014 35
PIH rejected
4/28&30/2014 Econ 141, Spring 2014 36
OLS and 2SLS give similar results
4/28&30/2014 Econ 141, Spring 2014 37
Test whether OLS
estimates are valid
using Hausman test
Problem set
Additional results in paper
Estimate the timing of the spending response
with respect to the rebate.
Show that it is mainly low income households
and those with a low level of liquid assets
whose consumption responds most to the tax
rebates.

Replication files with STATA do file to
generate the results are on bSpace.
Problem 3 on Problem set on Chapters 12 and 13.
4/28&30/2014 Econ 141, Spring 2014 38
Interpretation
Spending of a large fraction of U.S.
households responds to temporary changes
in income.

Important for understanding
Permanent income hypothesis
Importance of liquidity constraints
Aggregate demand effect of temporary fiscal
stimulus.
4/28&30/2014 Econ 141, Spring 2014 39
Quasi-Experiment
Using Electoral Cycles in Police Hiring to Estimate the Effect of Police
on Crime
Levitt (AER, 1996), comment by McCrary (AER, 2002)

4/28&30/2014 Econ 141, Spring 2014 40
Crime rates peaked in early 90s
4/28&30/2014 Econ 141, Spring 2014 41
Research question
Does putting more cops on the street reduce
the crime rate in a city?
4/28&30/2014 Econ 141, Spring 2014 42
Regression equation
4/28&30/2014 Econ 141, Spring 2014 43
Why OLS might be a problem
4/28&30/2014 Econ 141, Spring 2014 44
Cities are likely to increase the size of their police force
as a direct response to an increase in crime.
So cov ln

> .
OLS inconsistent and likely to understate the degree to
which increasing the police force reduces crime.
Election cycles also drive changes
in police force
Timing of Mayoral and Gubernatorial
elections is determined by law and not
affected by changes in crime rates.
Use election-driven changes in size of police
forces as exogenous variation in ln

to
identify causal effect of size of police force on
crime.
4/28&30/2014 Econ 141, Spring 2014 45
1
st
stage regression to show relevance
4/28&30/2014 Econ 141, Spring 2014 46
First-stage regression
Regress endogenous variable on instruments
and all the explanatory variables that are also
in the second-stage regression.
1
st
stage regression to show relevance
4/28&30/2014 Econ 141, Spring 2014 47
1
st
stage regression to show relevance
4/28&30/2014 Econ 141, Spring 2014 48
1
st
stage regression to show relevance
4/28&30/2014 Econ 141, Spring 2014 49
Columns (4) and (5) reduced forms
4/28&30/2014 Econ 141, Spring 2014 50
Columns (4) and (5) reduced forms
4/28&30/2014 Econ 141, Spring 2014 51
2
nd
stage regression:
police reduces crime
4/28&30/2014 Econ 141, Spring 2014 52
2
nd
stage regression:
police reduces crime
4/28&30/2014 Econ 141, Spring 2014 53
Levitt did weighted least squares
4/28&30/2014 Econ 141, Spring 2014 54
But he reversed the weights
4/28&30/2014 Econ 141, Spring 2014 55
McCrary (2002) points out that Levitt made a
programming error and gave unreliable
observations more weight rather than less.
Main results not robust to correction of this error.
McCrarys correction table
4/28&30/2014 Econ 141, Spring 2014 56
Heteroskedasticity matters!
4/28&30/2014 Econ 141, Spring 2014 57
Quasi-experiment
Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and
Pennsylvania
Card and Krueger (AER, 1994)

4/28&30/2014 Econ 141, Spring 2014 58
Minimum wage debate continues
WASHINGTONPresident Barack Obama's proposal
Tuesday to raise the federal minimum wage is likely to
rekindle debates over whether the measure helps or hurts
low-income workers.
White House officials say the move to boost the wage to
$9 an hour, from $7.25, is aimed at addressing poverty
and helping low-income Americans.
But the proposal likely will be opposed by Republicans
and business groups, which have traditionally said raising
the minimum wage discourages companies from hiring
low-skilled workers.
Wall Street Journal

4/28&30/2014 Econ 141, Spring 2014 59
Research question
How does raising the minimum wage affect
labor demand (employment levels)?

Quasi experiment
On April 1, 1992, New Jersey's minimum wage rose from $4.25 to $5.05
per hour. To evaluate the impact of the law we surveyed 410fast-food
restaurants in New Jersey and eastern Pennsylvania before and after the
rise. Comparison of employment growth at stores in New Jersey and
Pennsylvania (where the minimum wage was constant) provide simple
estimates of the effect of the higher minimum wage.



4/28&30/2014 Econ 141, Spring 2014 60
Regression equation
4/28&30/2014 Econ 141, Spring 2014 61
Regression equation
4/28&30/2014 Econ 141, Spring 2014 62
Sample of fast food restaurants
4/28&30/2014 Econ 141, Spring 2014 63
Fast food restaurants in study
4/28&30/2014 Econ 141, Spring 2014 64
NJ & PA restaurants similar
4/28&30/2014 Econ 141, Spring 2014 65
NJ & PA restaurants similar
4/28&30/2014 Econ 141, Spring 2014 66
Pre-treatment wages similar
4/28&30/2014
Econ 141, Spring 2014
67
Post-treatment wages different
4/28&30/2014 Econ 141, Spring 2014 68
Raising minimum wage
increased employment?
4/28&30/2014 Econ 141, Spring 2014 69
Puzzling result questioned
Internal validity:
Did NJ restaurants already adjust their hiring
before the first wave of survey?
Was there another NJ vs PA state-specific
shock/policy that affected relative demand for fast
food?
Were restaurants sampled in both states really
similar?
External validity:
How does NJ/PA fast food to U.S. labor market?
4/28&30/2014 Econ 141, Spring 2014 70
Quasi-experiment
Did Securitization Lead to Lax Screening? Evidence from Subprime Loans.
Keys, Mukherjee, Seru, and Vig (QJE, 2010)
4/28&30/2014 Econ 141, Spring 2014 71
Mortgage crisis
2008 financial crisis was largely due to default
(delinquency) rates on mortgages that were
part of mortgage backed securities increasing.

Research question:
Did financial institutions lower their standards
on the mortgages they issued when they knew
they could offload them into mortgage backed
securities?
4/28&30/2014 Econ 141, Spring 2014 72
Regression discontinuity
Discontinuity is treatment:
Loans with a credit score above 620 were
much easier securitized and offloaded.

Research question
Was there more scrutiny on loans with a score
just below 620 than on loans just above?

Regression discontinuity analysis
4/28&30/2014 Econ 141, Spring 2014 73
Regression discontinuity
Discontinuity is treatment:
Loans with a credit score above 620 were
much easier securitized and offloaded.

Research question
Was there more scrutiny on loans with a score
just below 620 than on loans just above?

Regression discontinuity analysis
4/28&30/2014 Econ 141, Spring 2014 74
Degree of scrutiny determined by amount of
background documentation on loan applicant
and on loan
Low documentation
vs
Full documentation
4/28&30/2014 Econ 141, Spring 2014 75
Regression discontinuity equation
4/28&30/2014 Econ 141, Spring 2014 76
Picture says more than regression
4/28&30/2014 Econ 141, Spring 2014 77
Regression confirms picture
4/28&30/2014 Econ 141, Spring 2014 78
Interpretation
Size of discontinuity suggests that financial
firms were less diligent with loans they could
more easily offload and securitize.
For loans they were more likely to hold on
their own book they required more
documentation.
4/28&30/2014 Econ 141, Spring 2014 79
Where to go after this class
This class taught main statistical theory behind
regression analysis. Logical topics beyond what
was covered are
Alternative estimation methods
Method of Moments, Generalized Method of Moments, Maximum Likelihood, Bayesian
Estimation.
Time series analysis
Observations correlated over time
Extended panel data techniques
Find applications
Labor Economics, Industrial Organization, Macroeconomics, Development Economics, ect.
4/28&30/2014 Econ 141, Spring 2014 80