Kennedy School of Government HKS448

Case Program CR14-09-1903.0

Designing Impact Evaluations:

Assessing Jamaica’s PATH Program

Introduction

It is March 2003, and the Government of Jamaica’s Ministry of Labour and Social Security
is in the midst of a major reform of its social safety net. The World Bank helped finance this reform
and has required the government to evaluate the cornerstone of the social safety net reform, PATH
(Programme Advancement through Health and Education). The government has selected a firm to
evaluate the program, and is now in discussions with this firm about how best to evaluate the
impact of the program. The firm has presented three possible evaluation designs, and the Minister
of Labour and Social Security has assigned you, as the Director of the Social and Manpower
Planning Division, to select a design.

Background on PATH

In early 2000, the Government of Jamaica undertook a reform of its social safety net
system, refocusing the system around PATH, a conditional cash transfer program. Through the
conditional cash transfer program, eligible families received cash assistance conditional on regular
attendance at school and regular checkups at health centers. This meant that once a family started
receiving cash transfers, they would continue to receive them as long as they would continue to
meet the program’s conditions.
1

1
The conditions for receiving benefits are as follows: Children 0-6 years old need to visit a health clinic every two
months during the first year and twice a year thereafter. Children 7-17 years old need to attend school at least 85
percent of school days.

This case was written by Dan Levy, Harvard Kennedy School (HKS) Lecturer in Public Policy, and HKS students
Michael McCreless and Daniel Bjorkegren. It is based on an evaluation conducted by Mathematica Policy Research
(Dan Levy and Jim Ohls) for Jamaica’s Ministry of Labour and Social Security that was funded by The World Bank.
The case was funded by Harvard Kennedy School’s Strengthen Learning and Teaching Excellence (SLATE)
initiative. It does not necessarily reflect the views of any of these institutions. (0409)

Copyright © 2009 by the President and Fellows of Harvard College. No part of this publication may be
reproduced, revised, translated, stored in a retrieval system, used in a spreadsheet, or transmitted in any
form or by any means (electronic, mechanical, photocopying, recording, or otherwise) without the written
permission of the Case Program. For orders and copyright permission information, please visit our website
at www.ksgcase.harvard.edu or send a written request to Case Program, John F. Kennedy School of
Government, Harvard University, 79 John F. Kennedy Street, Cambridge, MA 02138

This document is authorized for use only in Ivan Dario Lobo Romero's UNIANDES MDP GOBERNANZA INSTITUCIONES POLITICAS PUBLICAS S2 ONLINE (2022-10) IDLR ng at
Universidad de Los Andes - Colombia (UniAndes) from Feb 2022 to May 2022.
 Designing Impact Evaluations ________________________________________________ CR14-09-1903.0

The primary objective of the program was to link social assistance with human capital
accumulation. Another aim was to improve the targeting of welfare benefits over previous social
assistance programs in Jamaica. With nearly 20 percent of the Jamaican population under the
poverty line, targeting of previous social assistance programs was seen as inadequate. For example,
only 36 percent of the benefits of the Food Stamps program were received by poor households.

The PATH program sought to improve targeting of benefits to poor households by

gathering detailed information about applicants and ranking them by level of poverty. Each
household interested in the program would fill out an application form, reporting information on
socio-economic and demographic characteristics. This information was entered into a computer
system that used an algorithm to analyze each household’s responses and calculate a household
eligibility score. If the score was below a pre-determined threshold, then the household was
considered eligible, and invited to register in the program. If the score exceeded the threshold, then
the household was not eligible for the program. Applicants were generally informed of their
eligibility status 2-5 months after applying.

The program was piloted in the parish of St. Catherine in 2001, and was expanded Island-
wide in 2002. The Ministry of Labour and Social Security received most applications in the summer
of 2002, and continued to receive applications until the summer of 2003. The number of individual
beneficiaries who registered for the program was 245,000, which fell short of the program’s
planned enrollment. The cash transfer amount given per eligible member of a household was about
J$400 per month (later increased to J$530) and the average amount received per household was
2

J$1,800 per month.

Is PATH Effective?

Even though many in the ministry were certain that PATH would help the poor, there
were many unanswered questions. Would children in participating households attend school more
often? Would participating households become healthier? In order to answer these questions, the
Minister asked the firm to evaluate the causal impact of the program, using the most rigorous and
credible evaluation design possible. In particular, this involves choosing an evaluation design that
meets the following methodological criteria:
• The design should compare a participant group representative of some
population of policy-interest, to a comparison group similar to PATH
participants in:
o Observable characteristics such as demographics and socio-
economic status
o Unobservable characteristics such as motivation or need for PATH

2
In 2002, one Jamaican dollar was worth approximately two US cents.

This document is authorized for use only in Ivan Dario Lobo Romero's UNIANDES MDP GOBERNANZA INSTITUCIONES POLITICAS PUBLICAS S2 ONLINE (2022-10) IDLR ng at
Universidad de Los Andes - Colombia (UniAndes) from Feb 2022 to May 2022.
 Designing Impact Evaluations ________________________________________________ CR14-09-1903.0

(This is crucial to ensure the differences in outcomes between participant

and comparison groups can be attributed to PATH and not to something
else.)
• It is important to use large enough sample sizes to be able to reliably detect
policy-relevant program impacts. The sample sizes needed to reliably detect
PATH’s impacts were 2,500 participant group households and 2,500
comparison group households.
• The evaluation should measure the effect of PATH 1-2 years after the
participant group enrolled in the program, on at least two key outcomes of
interest:
o School attendance in the last 4 weeks
o Attendance at a health center for preventive reasons in the last six
months

Data

The following data were available for use in the evaluation:

• PATH data on socio-economic and demographic characteristics for all
applicants at the time they applied to the program.
• 2002 Jamaica Survey of Living Conditions (SLC): Household survey on a
representative sample of Jamaican households. Total Sample Size: 8,000
households. Conducted in the summer of 2002 (just before the program
started in the non-pilot areas).
• Survey of PATH participants: household survey similar to SLC 2002, but
on a representative sample of households who had been declared eligible
for the PATH program. Total Sample Size: 1,200 households. Conducted in
November 2002.

In addition, there are resources available to conduct two more household surveys (each
surveying a total of 5,000 households).

This document is authorized for use only in Ivan Dario Lobo Romero's UNIANDES MDP GOBERNANZA INSTITUCIONES POLITICAS PUBLICAS S2 ONLINE (2022-10) IDLR ng at
Universidad de Los Andes - Colombia (UniAndes) from Feb 2022 to May 2022.
 Designing Impact Evaluations ________________________________________________ CR14-09-1903.0

Evaluating Design Proposals

The firm selected to evaluate PATH has proposed the following three evaluation design
options, from which the Director of the Social and Manpower Planning Division must choose one:

Design 1 Design 2 Design 3

PATH participants from
PATH Participant Survey
Participant Group
Subset of households in the
Summer 2002 SLC who are
Comparison Group poor and are similar to the
participant group on a set of
observable characteristics
Using regression analysis,
compare change in
outcomes for participants to
change in outcomes for
Impact Estimation
comparison group and
control statistically for a host
of household and individual
characteristics
Applicants who scored just
below the eligibility
threshold
Applicants who scored just
above the eligibility
threshold
Using regression analysis,
compare outcomes for
participants to outcomes for
comparison group and
control statistically for a host
of household and individual
characteristics including the
eligibility score
Randomly selected group of
applicants who scored just
above the threshold
(normally ineligible) who will
be offered to participate in
the program
Randomly selected group of
applicants who scored just
above the threshold and who
will not be offered to
participate in the program
Using regression analysis,
compare outcomes for
participants to outcomes for
control group and control
statistically for a host of
household and individual
characteristics

Discussion Questions

1. Imagine yourself in the role of the Director of the Social and Manpower Planning
Division. Create a table or list in which you assess the strengths and weaknesses of
each design. This table can be written in a technical language. Be sure to consider:
a. The scientific quality of the design, i.e. its ability to estimate the true
impact of PATH on the key outcomes of interest.
b. The political feasibility of implementing the design
c. The logistical implications of the design, in terms of ensuring that
findings from the evaluation are available in a timely manner for
policymakers
d. The financial implications of the design, in particular if it involves
more resources than those already budgeted.
2. Write a one-page single-spaced memo to the Minister of Labour and Social
Security recommending which design should be selected to evaluate PATH.
Justify your recommendation using the strengths and weaknesses you
identified above, but write the memo in non-technical language. Attach your
table of strengths and weaknesses to the memo.

This document is authorized for use only in Ivan Dario Lobo Romero's UNIANDES MDP GOBERNANZA INSTITUCIONES POLITICAS PUBLICAS S2 ONLINE (2022-10) IDLR ng at
Universidad de Los Andes - Colombia (UniAndes) from Feb 2022 to May 2022.