Randomization (CLASS NOTES-4)

Last week, we mentioned that avoiding sample selection problem can be handled in two
ways: to get rid of selection bias or to find a ways to account for it, so that we can calculate
program effect in properly.

However, we mentioned that finding a proper counterfactual to treatment is the main

challenge of impact evaluation.

There are two approaches to get imitated counterfactual: statistical design method, and
wipe out differences that would have existed between the treated and non-treated groups
before comparing outcomes across the two groups.

In some disciplines, such as medical science, evidence about counterfactuals is generated

through randomized trials.

Allocating a program or intervention randomly across a sample of observations is one

solution to avoiding selection bias, provided that program impacts are examined at the level
of randomization.

Randomization can be conducted purely randomly, where treated and control units have the
same expected outcome in absence of the program. Or

Randomization can be conducted partially randomly, where treatment and control samples
are chosen randomly, conditional on some observable characteristics for example income. If
these programs are exogenously placed, conditional on these observed characteristics, an
unbiased program estimate can be made.

Pure Randomized Assignment

In randomized assignment, each eligible unit has the same probability of being selected for
treatment, ensuring equivalence between the treatment and comparison groups in both
observed and unobserved characteristics.

Following Figure illustrates the case of randomization graphically.

14
Consider a random distribution of two “similar” Groups of households or individuals—one
group is treated and the other group is not treated.

They are similar or “equivalent” in that both groups prior to a project intervention are
observed to have the same level of income (in this case, Y0).

After the treatment is carried out, the observed income of the treated group is found to be
Y2 while the income level of the control group is Y1. Therefore, the effect of program
intervention can be described as (Y2 − Y1).

Statistical Design of Randomization

In practice, it can be very difficult to ensure that a control group is very similar to project
areas,

It must provide the following two priorities:

 That treatment effects observed in the sample are generalizable-external validity.

 That effects themselves are a function of only the program itself- internal validity.

Randomized assignment process will produce two groups that have statistically equivalent
averages for all their characteristics, if we have large enough number of units.

Statisticians have proposed a two-stage randomization approach outlining these priorities.

In the first stage, a sample of potential participants is selected randomly from the relevant
population. This sample should be representative of the population, within a certain
sampling error. This stage ensures external validity of the experiment.

15
In the second stage, individuals in this sample are randomly assigned to treatment and
comparison groups, ensuring internal validity in that subsequent changes in the outcomes
measured are due to the program instead of other factors.

An evaluation is internally valid if it provides an accurate estimate of the counterfactual

through a valid comparison group.

An evaluation is externally valid if the evaluation sample accurately represents the

population of eligible units. The results of the evaluation can then be generalized to the
population of eligible units.

=============================================================

we have used a randomization process for two different purposes:

random selection of a sample (for external validity), and

randomized assignment of treatment as an impact evaluation method (for internal validity).

When Can Randomized Assignment Be Used?

Randomized assignment can be used as a program allocation rule in one of two specific
scenarios:

1. When the eligible population is greater than the number of program spaces available.

When the demand for a program exceeds the supply, a lottery can be used to select the
treatment group within the eligible population. The group that wins the lottery is the
treatment group, and the rest of the population that is not offered the program is the
comparison group.

16
 2. When a program needs to be gradually phased in until it covers the entire eligible
population.

When a program is phased in, randomization of the order in which participants receive the
program gives each eligible unit the same chance of receiving treatment in the first phase or
in a later phase of the program. As long as the last group has not yet been phased into the
program, it serves as a valid comparison group from which the counterfactual for the groups
that have already been phased in can be estimated.

How Do You Randomly Assign Treatment?

There are 3 steps to successfully assign treatment in a randomized way.

Step 1 is to define the units that are eligible for the program. A unit could be a person, a
health center, a school, a business, or even an entire village or municipality. The population
of eligible units consists of those for which you are interested in knowing the impact of your
program.

For example, if you are implementing a training program for primary school teachers in rural
areas, then primary school teachers in urban areas or secondary school teachers would not
belong to your population of eligible units.

Step 2 is to determine the number of observations required for the evaluation.

Once you have determined the population of eligible units, it will be necessary to compare
the size of the group with the number of observations required for the evaluation. The size
of the evaluation sample is determined through power calculations and is based on the
types of questions you would like answered.

Power calculation is related to hypothesis testing.

H0: Impact or difference between outcomes in treatment and comparison groups
equals zero
H1: Impact or difference between outcomes in treatment and comparison groups
does not equal zero
A type I error occurs when an evaluation concludes that a program has had an
impact, when in reality it had no impact.

A type II error occurs when an evaluation concludes that the program has had no
impact, when in fact it has had an impact.

Power is the probability of detecting an impact, when in fact one exists. An impact
evaluation has high power if there is a low risk of not detecting real program impacts:
that is, of committing a type II error.
This second step is done mainly to limit data collection costs. If it is found that data from
existing monitoring systems can be used for the evaluation, and that those systems cover
17
the full population of eligible units, then you may not need to draw a separate evaluation
sample.

=================================

Step 3 is to form the treatment and comparison groups from the units in the evaluation
sample through randomized assignment.

Randomized assignment needs to be done in several ways or techniques such as flipping a

coin or picking names out of a hat or in a public forum.

Steps in Randomized Assignment to Treatment

Using excel to randomize assignment to treatment.

=S_SAYI_ÜRET()

=EĞER(column row number <0.5;1;0)

18
19