CHAPTER # 4

RANDOMIZED ASSIGNMENT
Impact Evaluation in Practice
2nd Edition
 CONTENTS OF THE CHAPTER
Evaluating Programs Based on the Rules of Assignment

Randomized Assignment of Treatment

Why Does Randomized Assignment Produce An Excellent Estimate Of The Counterfactual

External and Internal Validity?

When Can Randomized Assignment Be Used?

How Do You Randomly Assign Treatment?

At What Level Do You Perform Randomized Assignment?

Estimating Impact under Randomized Assignment

EVALUATING PROGRAMS BASED ON THE RULES OF ASSIGNMENT
There are two common but potentially biased methods used for estimating the counterfactual
in impact evaluations: before-and-after comparisons and enrolled-non-enrolled comparisons.

Estimating program impacts is not always straightforward because programs operate in

complex and ever-changing environments where various factors can affect both participants
and non-participants.

These factors include natural disasters (droughts, earthquakes), economic downturns

(recessions), political changes (changes in government), and shifts in policies (international and
local).

Key parameter for determining the appropriate impact evaluation method is the program's
rules for selecting participants.

In most cases, the evaluation methods should align with the program's operational rules rather
than forcing the program to adapt to a specific evaluation method.
EVALUATING PROGRAMS BASED ON THE RULES OF ASSIGNMENT
There is great importance of fair and transparent rules for program assignment, especially
emphasizing the idea of providing everyone eligible with an equal opportunity to participate in
the program, such as through a lottery system.

So randomized assignment method, also known as randomized controlled trials (RCTs) is
introduced in this chapter, as a way to decide who enters a program and who does not, similar
to a lottery.

This method is presented as a fair and transparent way to allocate resources among deserving
populations.

The use of the randomized assignment method, RCTs, for evaluating the impacts of social
programs has been on the rise in recent years due to its effectiveness in providing fairness and
producing strong impact assessments
 RANDOMIZED ASSIGNMENT OF TREATMENT
• Randomized assignment of treatment is considered the gold standard of impact evaluation.

• It uses a random process, a chance, to decide who is granted access to the program and who is
not.

Fair and Transparent Allocation:

• Randomized assignment is a fair and transparent way to assign scarce program resources.

• It ensures that every eligible unit (e.g., individuals, households, businesses, schools, hospitals,
or communities) has an equal chance of receiving the program.

• It prevents the program from being assigned based on arbitrary or subjective criteria,
patronage, or unfair practices.
 RANDOMIZED ASSIGNMENT OF TREATMENT
Advantages of Randomized Assignment:
• It can be easily explained by program managers and understood by key stakeholders.

• When conducted openly and transparently, it is less susceptible to manipulation, protecting

program managers from accusations of favoritism or corruption.

Deriving Randomized Assignment from Operational Rules:

• In many programs, the population of intended participants exceeds the program's capacity to
accommodate everyone simultaneously. Reasons for this limitation include budget constraints
and capacity constraints.

• As a result, program administrators must decide who will enter the program and who will not
i.e. must define a rationing mechanism to allocate the program’s services.
 RANDOMIZED ASSIGNMENT OF TREATMENT
• To allocate program services, administrators can use various mechanisms, such as first-come,
first-served, observed characteristics (e.g., serving the poorest areas first), unobserved
characteristics (e.g., based on individual motivation), or lotteries.

Use of Lotteries in Allocation:

• Even when potential participants can be ranked based on need (e.g., targeting the poorest
households), lotteries may still be desirable.

• For example, if income is an imperfect measure, households near the eligibility threshold may
have uncertain eligibility. If program targets poorest 20% households then undoubtedly bottom
15% will be included but near to 20% (below and above) households may or may not be poor if
income was measured perfectly.

• In such cases, running a lottery among households around threshold (15%-20%) in a certain
income range can be a fair way to allocate benefits within that group.
WHY DOES RANDOMIZED ASSIGNMENT PRODUCE AN EXCELLENT
ESTIMATE OF THE COUNTERFACTUAL
In impact evaluations, it's ideal for the comparison group to be as similar as possible to the
treatment group in all aspects except their participation in the evaluated program.

When units are randomly assigned to treatment and comparison groups, this random process
naturally results in two groups that are highly likely to be statistically identical, especially when
applied to a large number of units.

In other words, with a sufficiently large number of units, random assignment creates groups
with statistically equivalent averages for all their characteristics.

The following figure illustrates why randomized assignment produces a comparison group that
is statistically equivalent to the treatment group.
WHY DOES RANDOMIZED ASSIGNMENT PRODUCE AN EXCELLENT
ESTIMATE OF THE COUNTERFACTUAL
WHY DOES RANDOMIZED ASSIGNMENT PRODUCE AN EXCELLENT
ESTIMATE OF THE COUNTERFACTUAL
Suppose we have 1,000 eligible individuals, half are randomly assigned to the treatment group,
and the other half are randomly assigned to the comparison group

e.g. write names on paper and blindly choose 500 names and include them in treatment group
the rest of 500 will be in comparison group.

Randomly dividing them into a treatment group and a comparison group results in both groups
being statistically similar.

For instance, if 40% of the original population were women, approximately 40% of both groups
will also be women.

If 20% of the original population had blue eyes, approximately 20% of both groups will also
have people with blue eyes
WHY DOES RANDOMIZED ASSIGNMENT PRODUCE AN EXCELLENT
ESTIMATE OF THE COUNTERFACTUAL
Similarly, other characteristics, both observed (e.g., hair color) and unobserved (e.g., motivation
or personality traits), will also be distributed evenly between the two groups.

This ensures that the estimated counterfactual, representing the outcome without treatment,
closely approximates reality, and the estimated impacts of the program are not biased due to
group selection.

In summary, when a sufficiently large number of units is used randomized assignment creates
groups that are similar in every way, leading to unbiased impact assessments.

Baseline data is used to empirically verify that there are no systematic differences in observed
characteristics between the treatment and comparison groups before the program begins.

Any differences in outcomes observed after the program launch can be attributed solely to the
program's impact since the two groups were identical at the start and experienced the same
external factors.
WHY DOES RANDOMIZED ASSIGNMENT PRODUCE AN EXCELLENT
ESTIMATE OF THE COUNTERFACTUAL
• To estimate the program's impact under randomized assignment, the difference between the
treatment group's mean outcome and the counterfactual (mean outcome of the comparison
group) is taken.

• This estimation is reliable because it accounts for all factors, both observed and unobserved,
that could explain outcome differences.

• It's noted that in some cases, not all eligible units need to be included in the evaluation.

• A representative random sample can be selected, and the same principles of random
assignment and equivalence preservation apply within this sample.

• Then randomly assign half to treatment group and half to comparison group, ensuring that the
estimated impact accurately represents the program's effect.
 EXTERNAL AND INTERNAL VALIDITY?
• The steps outlined above for randomized assignment of treatment will ensure both the internal
and the external validity of the impact estimates

Internal Validity:
• Internal validity ensures that the estimated impact of a program accurately reflects the true
impact, considering all potential confounding factors.

• Randomized assignment creates a comparison group that is statistically equivalent to the

treatment group at the beginning (baseline) before the program starts.

• Any differences in outcomes between the two groups after the program launch can be
attributed solely to the program's existence in the treatment group, ensuring internal validity.
 EXTERNAL AND INTERNAL VALIDITY?
External Validity:
• External validity means that the evaluation sample, or the group of units being studied,
accurately represents the entire population of eligible units.
• The results of the evaluation should be applicable or generalizable to this broader population.

• Random sampling is used to ensure that the evaluation sample is representative of the
population.

Two Purposes of Randomization:

• Randomization is employed for two distinct purposes in impact evaluation.

• First, it's used for random selection of a sample to achieve external validity, ensuring that the
sample reflects the population.
• Second, it's used as an impact evaluation method for internal validity, making sure that the
estimated impacts are attributable to the program itself.
 EXTERNAL AND INTERNAL VALIDITY?
Balancing Internal and External Validity:
• An impact evaluation can produce internally valid estimates of impact through randomized
assignment of treatment, However, if the evaluation is performed on a nonrandom sample of
population, the estimated impacts may not be generalizable to the population of eligible units.
• Conversely, if the evaluation uses a random sample of the population of eligible units, but
treatment is not assigned in a randomized way, then the sample would be representative, but
the comparison group may not be valid, thus jeopardizing internal validity.
• There can be situations where programs face constraints that force a trade-off between internal
and external validity.
• For example, if a program targets the bottom 20% of households based on income but conducts
a randomized assignment impact evaluation among a random sample of households in the 15th
to 25th percentiles.
• It achieves internal validity for that specific group but limits external validity since the results
cannot be applied directly to all beneficiaries, including those below the 15th percentile.
EXTERNAL AND INTERNAL VALIDITY?
 WHEN CAN RANDOMIZED ASSIGNMENT BE USED?
Scenario 1: Program Demand Exceeds Supply:
• Randomized assignment is used when the eligible population is larger than the available
program slots.

• A lottery is conducted to select the treatment group from the eligible population, ensuring each
unit has an equal chance of being selected.

• The group that wins the lottery becomes the treatment group, while the remaining population
serves as the comparison group.

• This approach allows for the measurement of program impacts over different time frames, and
there are no ethical dilemmas since capacity constraints necessitate leaving out some units.

• Example: If the Ministry of Education has funds for only one-third of public schools to receive
libraries, a lottery determines which schools get libraries (treatment group), and the other two-
thirds serve as the comparison group to measure program impacts.
 WHEN CAN RANDOMIZED ASSIGNMENT BE USED?
Scenario 2: Gradual Program Phasing:
• Randomization is employed when a program needs to be rolled out gradually to cover the
entire eligible population.

• Randomization determines the order in which eligible units receive the program.

• The last group not yet phased into the program serves as a valid comparison group, allowing
the estimation of counterfactuals for the groups that have already received the program.

• This setup can also allow for the evaluation to pick up the effects of differential exposure to
treatment: that is, the effect of receiving a program for more or less time.

• Example: If the Ministry of Health plans to train all 15,000 nurses in a new healthcare protocol
over three years, random assignment determines which nurses receive training in each year.
This enables evaluation of the program's impact by comparing the group trained in the first
year (treatment) with the group assigned to training in the third year (comparison).
 HOW DO YOU RANDOMLY ASSIGN TREATMENT?
Step 1
Define Eligible Units:
1. Start by defining the units eligible for the program. These units could be individuals, health
centers, schools, businesses, or even entire communities.

2. The eligible units are those for which you want to assess the program's impact. Units not in
this category should be excluded.

Evaluate Sample Size:

3. Determine whether the size of the eligible population is sufficient for the evaluation or if
it's necessary to select a sample.

4. The size of the evaluation sample is determined through power calculations, based on the
specific evaluation questions.

5. If the eligible population is small, all units may be included. Otherwise, proceed to the next
step.
 HOW DO YOU RANDOMLY ASSIGN TREATMENT??
Step 2
Select an Evaluation Sample:
• Select a sample of units from the eligible population. This is often done to manage data
collection costs.

• If existing monitoring systems cover the entire eligible population and provide relevant data, a
separate evaluation sample may not be needed.

• The sample should be representative of the entire eligible population to ensure external
validity.

• Collecting data on this sample of 1,000 teachers out of 1 million teachers in 200 schools out of
10,000 schools will be much cheaper than collecting data on every teacher in all schools in the
country.
 HOW DO YOU RANDOMLY ASSIGN TREATMENT??
Step 3
Randomly Assign Treatment and Comparison Groups:
• Form the treatment and comparison groups from the units in the evaluation sample through
randomized assignment.

• If randomized assignment is conducted publicly, methods like flipping a coin or drawing names
from a hat may be used.

Examples
• The following examples assume that the unit of randomization is an individual person, but the
same logic applies to randomizing more aggregated units of observation such as schools,
businesses, or communities:
 HOW DO YOU RANDOMLY ASSIGN TREATMENT??
1.Coin Flip (50/50 Assignment):
• To assign 50% of individuals to the treatment group and 50% to the comparison group, use
a coin.
• Decide in advance whether "heads" or "tails" on the coin will assign a person to the
treatment group. Flip the coin for each individual accordingly.

2.Die Roll (One-Third Assignment):

• To assign one-third of the evaluation sample to the treatment group, use a die (dice).
• Establish a rule in advance; for instance, you might decide that rolling a 1 or 2 on the die
assigns a person to the treatment group, while 3, 4, 5, or 6 assigns them to the comparison
group.
• Roll the die once for each person in the evaluation sample and assign them based on the
outcome.
 HOW DO YOU RANDOMLY ASSIGN TREATMENT??
3.Drawing from a Hat:
• Write the names of all individuals on identical pieces of paper and fold them so that the
names cannot be seen.
• Mix these pieces of paper thoroughly in a hat or container.

• Determine a clear rule in advance, specifying how many pieces of paper you will draw.

• Drawing a name corresponds to assigning that person to the treatment group.

• Have someone unbiased (e.g., a child) draw the required number of names to form the
treatment group.

These methods ensure a randomized assignment of individuals to treatment and

comparison groups, maintaining fairness and eliminating bias in the selection
process.
 HOW DO YOU RANDOMLY ASSIGN TREATMENT??
Need for automated methods for Randomized Assignment
• When assigning a substantial number of units (e.g., over 100), manual methods like coin flips or
drawing names become impractical so automated processes are required.

• An automated process involves deciding on a rule for assignment based on random numbers
generated by a computer or specialized software.

• For example, if 40 out of 100 units need to be assigned to the treatment group, you might choose
to assign the 40 units with the highest random numbers to the treatment group.

• The key is to determine the assignment rule before generating the random numbers to maintain
the integrity of randomized assignment.

• Transparency is crucial, so the assignment rule must be communicated in advance, followed

rigorously, and documented for auditing purposes. Methods like videotaping lotteries or
providing computation logs for computer-generated random numbers can ensure transparency.
HOW DO YOU RANDOMLY ASSIGN TREATMENT??
 AT WHAT LEVEL DO YOU PERFORM RANDOMIZED ASSIGNMENT?
Various Assignment Levels:
• Randomized assignment can occur at different levels, such as individual, household, business,
community, or region, depending on the program's implementation.

Choosing the Level:

• The level of assignment depends on how and where the program is being implemented. For
instance, a health program at the clinic level might involve random selection of clinics, followed by
random assignment to treatment or comparison groups.

Challenges with High-Level Assignment:

• Assigning treatment at higher, more aggregate levels (e.g., regions or provinces) can make impact
evaluation difficult. Small numbers of regions or provinces may not ensure balanced treatment and
comparison groups.

• Also, it's harder to balance time-bound external factors (like weather or local elections) at higher
levels.
 AT WHAT LEVEL DO YOU PERFORM RANDOMIZED ASSIGNMENT?
Balancing External Factors:
It's crucial to ensure that external factors affecting treatment and comparison groups are balanced.
For instance, if evaluating an agriculture program, droughts should affect both groups equally.

Benefits of Lower-Level Assignment:

Lowering the assignment level (e.g., to municipalities) increases the likelihood of balancing
external factors over time.

Risks Associated with Lower-Level Assignment:

• There are two kinds of risks associated with Lower-Level Assignment.

• These are Spillovers and Imperfect Compliance:

• Spillovers occur when the treatment group directly or indirectly affects outcomes in the
comparison group (or vice versa).
 AT WHAT LEVEL DO YOU PERFORM RANDOMIZED ASSIGNMENT?
• Example of Spillovers: Providing deworming medicine to children can lead to spillovers if treated
households are close to untreated ones, reducing the chances of contracting worms. Distancing
households can mitigate this but may increase program and survey costs.

• Imperfect compliance occurs when some members of the comparison group participate in the
program, or some members of the treatment group do not.

Managing Risks of spillovers and imperfect compliance:

• To minimize spillovers, assignments can be done in groups or clusters (e.g., students or households).
Distancing can also be beneficial.

• To reduce imperfect compliance, choose the assignment level according to the program's capacity to
maintain a clear distinction between groups.

• Maximizing Unit Numbers: If spillovers can be ruled out, it's best to assign treatment at the lowest
possible level. This maximizes the number of units in treatment and comparison groups, enhancing the
evaluation's statistical power.
 ESTIMATING IMPACT UNDER RANDOMIZED ASSIGNMENT
• After randomly assigning treatment and comparison groups, you need to measure outcomes for
both groups.

Impact Calculation:
• To estimate the program's impact, calculate the difference between the average outcome (Y) for
the treatment group and the average outcome (Y) for the comparison group by assuming that all
units in the treatment group receive treatment, and none in the comparison group do.

For example, if the treatment group’s

average outcome is 100 & comparison
group's average outcome is 80, the
impact is 20.
Thank You