03 Randomization

Program Evaluation

Instructor: Dániel Horn

(slides: Gábor Kézdi)
Content
 Randomized Controlled Trials
 Random selection provides good control groups
 That give good distributions of the counterfactual

 Case study: NSW

ProgEval 03 Randomization 2
Randomized Controlled Trial (RCT)
 Random assignment of people into a treated group
and a non-treated group
 Think of a medical experiment

 Identify a group of potential participants

 Randomly assign each into one of two groups
 Group 1 receives the treatment (“the treatment group”)
 Group 0 does not receive it (the “control group”)
 After treatment, compare the average outcome of the
two groups
 Why does this work?
ProgEval 03 Randomization 3
Randomized Assignment
 Whoever participates or does not participate
 “assignment” or “treatment”

 is independent of potential outcomes

Wi ǁ (Y(0)i, Y(1)i)
 Where sign ǁ here means independence
 The distribution of (Y(0)i, Y(1)i)
 their joint distribution

 is the same for those assigned to treatment and those not

assigned
 for Wi=1 and Wi=0

ProgEval 03 Randomization 4
Randomized Assignment
 Assignment independent from both Y(0) and Y(1)
 Participation is unrelated to whether someone would
have higher or lower outcome if treated
 Participation is unrelated to whether someone would
have higher or lower outcome if untreated

 Assignment independent from Y(1)i − Y(0)i

 Participation is unrelated to whether someone would
experience a higher or lower treatment effect

ProgEval 03 Randomization 5
Randomized Assignment
 Whatever “randomness” means is not really
important: what we need here is independence
 The idea of “true” randomness is controversial
 Is the result of a coin flip random or is it predictable if we
know the forces, the properties of the coin, the wind, etc.?
 In practice randomization is usually carried out using
computer-generated “pseudo-random sequences”
 But that is not really important for us

 We need a rule that makes sure

 Assignment independent of potential outcomes

ProgEval 03 Randomization 6
 Independence of What from What?
 Independence of treatment status from potential
outcomes
 Of course, that does not mean that treatment is
independent of observed outcomes
 If the program has positive effect, observed outcomes for
the treated may be better than the observed outcomes of
the untreated
 Observed outcomes for the treated are Y(1) and observed
outcomes for the untreated are Y(0)

ProgEval 03 Randomization 7
Randomization In Practice
 Create a group from the potential participants
 List the potential participants
 Split them into two groups
 using randomization device (coin, computer-generated pseudo-
random number etc.)

 One group becomes the treated group

 The other group becomes the untreated (control) group
 Note: there can be more than one control groups, depending
on what we want to measure as treatment (cf. drug vs. placebo
vs. no-drug)

ProgEval 03 Randomization 8
 Consequences of Randomization
 Potential outcomes are similar in the treatment group
and the control group
 The average potential outcomes are the same
E[Y(1)i|Wi=1] = E[Y(1) i|Wi=0]
and E[Y(0) i|Wi=1] = E[Y(0) i|Wi=0]
 So that
E [Y(1) i] = P(Wi=1)E[Y(1)i|Wi=1] + P(Wi=0)E[Y(1) i|Wi=0]
= E[Y(1) i|Wi=1] = E[Yi|Wi=1]
E [Y(0) i] = P(Wi=1)E[Y(0) i|Wi=1] + P(Wi=0)E[Y(0) i|Wi=0]
= E[Y(0) i|Wi=0] = E[Yi|Wi=0]
ProgEval 03 Randomization 9
 Consequences of Randomization
 So that the average treatment effect is identified
 by the difference of observed outcomes in the
treated group vs. the untreated group

ATE = E[Y(1)i − Y(0)i] = E[Y(1)i] − E[Y(0)i]

= E[Yi|Wi=1] − E[Yi|Wi=0]

 These are all observable averages!

What would be the ATET?
ProgEval 03 Randomization 10
 Consequences of Randomization
 The average treatment effect on the treated is also
identified
 by the difference of observed outcomes in the
treated group vs. the untreated group
ATET = E[Y(1)i − Y(0)i|Wi=1]
= E[Y(1)i|Wi=1] − E[Y(0)i|Wi=1]
= E[Y(1)i|Wi=1] − E[Y(0)i|Wi=0]
= E[Yi|Wi=1] − E[Yi|Wi=0]
So here ATET= ATE
ProgEval 03 Randomization 11
 Consequences of Randomization
 Same is true within any pre-defined group
 So CATE and CATET are also identified
 by comparing average outcomes
 between treated and untreated
 in the same group defined by Xi=x

ProgEval 03 Randomization 12
The RCT Movement
 The gold standard of evaluations
 High internal validity
 Easy to communicate

 Yet they are (still) rare. Why?

 Randomization needs to be built into the design of the
program
 You can’t do randomized evaluation if you are called to
evaluate after the program is over
 Randomization may be problematic in practice
 Program administrators often object the idea

ProgEval 03 Randomization 13
The RCT Movement
 There is a movement out there to propagate
randomized evaluations
 The JPAL group
http://www.povertyactionlab.org/

 Many donors have started to require randomized

evaluations
 The Gates Foundation
 The World Bank
 Even some divisions within the EU bureaucracy

ProgEval 03 Randomization 14
RCTs Beyond Program Evaluation
 “Field experiments” vs. “lab experiments”
 See the field experiment site by John List
www.fieldexperiments.com
 Most experiments in econ are done in labs

 “Mechanism experiments” vs. “policy experiments”

 (Experimental) evaluation of some policies can uncover
theoretically interesting phenomena
 But most don’t
 Researchers interested in “mechanisms” have to organize
their own experiments

ProgEval 03 Randomization 15
Case Study: NSW
 The National Supported Work demonstration
 U.S., 1970’s
 Run and evaluated by non-profit organization Manpower
Demonstration Research Corporation (MDRC)

 Target group: low employment prospects

 AFCD (Aid to Families with Dependent Children) recipients (single
mothers), marginalized men

 A subsidized employment program

 At private sector employers
 Program covers total employment costs for 12 months
 Starting wage low but steep increase to market level
 Intensive consulting for participants throughout
ProgEval 03 Randomization 16
Randomization in the NSW
 Started with an applicant pool of 6000
 4000 men, 2000 women
 In two cities

 Randomly assigned them into treatment and

control groups
 Members of the treatment groups were offered the
job
 Data was then collected on everyone
 Retrospective information on past employment
 Followed through for time
ProgEval 03 Randomization 17
Evidence on Randomization
 Suppose we are not sure whether randomization was
carried out properly
 We can check the implications of randomization
 By comparing variables unaffected by the program
 e.g., compare means across treatment vs. control
 Why? Because randomization should result in similar
distribution of everything
 Except of course things that the program could influence

 In fact, we should check, always

 With small chance, substantial differences may remain even I
randomization is properly done
ProgEval 03 Randomization 18
Evidence on Randomization, NSW

 Source:
LaLonde (1986)

ProgEval 03 Randomization 19
Effect on Employment (Women)

Source:
Ham &
LaLonde
(1996)

ProgEval 03 Randomization 20
Things to Note On Graph
 Treatment & control very similar before 0
 Employment rates at months 26 are different
 38% vs. 28%, cca. 10 pctage points difference

 Decreasing trend before 0, immediate increase

afterwards even in control group
 “Ashenfelter’s dip”
 The dip is asymmetric here
 Likely due to improving macro conditions

 Take-up of program is imperfect

 And there is continuous attrition
ProgEval 03 Randomization 21
Effects On Earnings
Women Men

Source: LaLonde (1986)

ProgEval 03 Randomization 22
The Effect of What?
 We are comparing assigned vs. non-assigned people
 The effect of being assigned to the treatment group
 Is what’s identified by this difference

 All individuals here are volunteers

 All wanted to participate

 Yet not all assigned to participate do participate

 88% of assigned women started working in subsidized jobs
 65% of assigned women worked in subsidized jobs in month 12

 Is this an effect that is interesting for policy?

ProgEval 03 Randomization 23
Conclusions from the NSW
 The program lead to an increase in employment
among women by 10 percentage points
 It lead to an increase in earnings by about $800
 (Earnings of non-employed are calculated as zero)
 Increase mostly due to increase in employment

 High internal validity

 Because of randomization

 Estimates not very precise

 Sample relatively small, variation very large esp. among men

ProgEval 03 Randomization 24
Take-away
 Randomized assignment of treatment
 Treatment status independent of potential outcomes
Wi ǁ (Y(0)i, Y(1)i)

 If treatment randomized ATE is identified by comparing

means of treated vs. untreated
 Also equals to ATET
 CATE, CATET is also identified by comparing the mean of
treated vs. untreated in subgroups

 Conclusions from the NSW

 Randomization identifies the impact of the program
 More precisely the effect of being offered participation
ProgEval 03 Randomization 25