kSome of the standard questions that come up in the field of development economics,

which is concerned with determinants of poverty and policies to alleviate it, are as
follows: Does microcredit alleviate poverty? Are policies of financial inclusion
effective in helping the poor who are self-employed to save, invest, and raise their
incomes? Did the Mahatma Gandhi National Rural Employment Guarantee Act
(MNREGA)1 raise wages by providing an alternative source of employment for rural
labourers in India? Is it availability of textbooks or mid-day meals or better health and
sanitation that can improve the educational attainment of children from poor families
in rural areas?

The main challenge in answering these questions is establishing a connection between

cause and effect. If the questions sound straightforward and the approach sounds
simple, then you have not encountered the word that is to academic seminars what
spells are to Harry Potter and his friends: ‘identification’. The moment this comes up,
a hushed silence descends on the room and the speaker launches an intense defence of
how their analysis establishes a robust path from cause to effect. The problem is that
in the real world everything changes at the same time and so it is hard to identify what
is a cause and what is an effect. For example, if the poor save less, is it because low
incomes cause low savings or is it the case that low savings cause low incomes?
Similarly, it is hard to figure out the effect of one cause from that of another: maybe
expansion of bank branches does facilitate saving but simply establishing a correlation
between the two is not sufficient as some third factor (such as rising wages in the
region) could be driving both, creating a spurious correlation. Theory can justify all
these lines of arguments but cannot tell whether we should focus on policies that will
raise income and therefore boost savings, or whether we should prioritise policies that
will enhance savings opportunities for the poor and thereby eventually raise income.

The strength of randomised controlled trials (RCTs)

Standard empirical methods try to find some externally driven change in the
environment that changes one factor, and then follows the line of causality. To
continue with the example of savings, the Government of India has enthusiastically
pushed the Jan Dhan Yojana2 in the last few years and one could try to see if those
who were brought under this scheme were able to save more than those who were not.
The trouble is that the government may have chosen to prioritise some areas over
others for a reason (for example, they were poorer) and so this is not a clean
comparison. Similarly, those who chose to have an account may be thriftier and so we
cannot use their behaviour to judge how the average person would react to such a
scheme. Finally, even if we see a positive effect on the savings and incomes of those
who signed up for these accounts, it could well be that something else was going on
that drove both trends – maybe wages were rising due to a rise in export demand, or
due to a government programme of road construction.

This is where the strength of randomised controlled trials (RCTs) lies. RCTs is a
technical phrase only heard within the confines of academic and policy worlds until
14 October 2019, when the Nobel Prize in economics was awarded to Abhijit
Banerjee, Esther Duflo, and Michael Kremer for pioneering the use of RCTs in
development economics. Following experimental trials in medicine, RCTs use a key
insight that can be traced back to The Design of Experiments (1935) by Ronald Fisher,
an eminent British statistician and geneticist: you select two groups that are similar
and then randomly select one to receive the treatment (a drug, or a policy) being tested
and then compare the outcome of this group (called the ‘treatment’ group) with that of
the other group (called the ‘control’ group). If the difference is statistically significant,
that is attributed to the treatment. The very design of the study eliminates the standard
problems mentioned above. 

The key innovation here is not coming up with the idea of randomisation – but
applying it in real life with programmes and interventions that directly affect the lives
of the poor. From testing drugs to placing government programmes as well as those
carried out by NGOs (non-governmental organisations) on a randomised basis across
villages, households, and organisations, takes quite a leap of imagination.   

Using this method in economics has altered our views about what policies work and
what do not. Take the example of microfinance, which serves more than 100 million
people, mostly women, belonging to the poorer sections of society worldwide.
Muhammad Yunus of Bangladesh is viewed as the leader of the microfinance
movement for singlehandedly creating the most famous and successful microfinance
institution (MFI) of the modern era, the Grameen Bank of Bangladesh. In 2006,
Yunus and the Grameen Bank jointly won the Nobel Peace Prize for their contribution
to reducing world poverty.

But, is microfinance effective in reducing poverty? If we merely compared those that

have access to microfinance and those that do not, we would not get a satisfactory
answer for the reasons mentioned above. Banerjee and Duflo, together with their
colleagues, studied the impact of access to microfinance on the creation and
profitability of small business as well as various measures of standard of living by
working with Spandana, an MFI. They randomly selected half of around 100 slums of
Hyderabad where a new branch was opened (the ‘treatment’ group), while in the
remaining half of the slums no branch was opened (the ‘control’ group).

Before the programme was carried out, the control and treatment slums looked very
similar in terms of population, average debt outstanding, businesses per capita, per
capita expenditure, and literacy. What about the effect of the programme on the
treatment slums? Small business investment and profits of pre-existing businesses
increased, but consumption did not significantly rise. Durable goods expenditure
increased, which suggests that loans were mostly used to purchase these. The study
found no significant changes in health, education, or women’s empowerment. This
research and a set of other studies in different countries have changed our views about
the role of microfinance in alleviating poverty. While access to small loans is
undoubtedly useful for expanding existing businesses and funding consumer durable
goods, and may also help recipients to tide over temporary gaps between income
flows and consumption needs, it is no longer seen as a magic bullet for solving the
problem of poverty.
Where do RCTs fit into the broad scope of the field of
development economics?
Development economics is concerned with a much broader set of issues than
evaluating specific programmes relating to health, education, or credit, where RCTs
have been most frequently applied.

A central concern has been the process of structural transformation of an economy –

how the population moves from agriculture to industry and services – and
accordingly, how the sectoral composition of national income changes. This process
involves not just a movement of resources (land, labour, and capital) but also a
process of institutional change – from informal personalised transactions to more
formal contractual arrangements and markets, and associated changes in social norms.
These are the kind of issues that Simon Kuznets and Arthur Lewis, two earlier
recipients of the Nobel Prize, dealt with.

RCTs, however, can mostly be applied to study problems at the micro-level where the
implementation of an individual programme – whether it is by the government or a
private organisation (like a MFI or an NGO) – can be done in a randomised way that
allows for a statistically satisfactory evaluation of the programme’s impact, as
outlined earlier. Clearly, as with any other tool of analysis, RCTs cannot be applied to
every question of interest within the field. And, as with any new method that attracts
young researchers and research funding, there are grounds to worry that this will push
out important research that uses other methods, including theory and empirical work
that does not use RCTs. By their very nature, RCTs cannot be applied to broad macro-
level issues or the more long-run aspects of development and institutional change.

However, one should note that a new generation of RCTs have come up that goes
beyond evaluating programmes, and suggests that the frontier of their applicability can
be pushed forward in creative ways. For example, a major focus of research in
development economics has been to understand the contractual terms that prevail in
land, labour, and credit markets in developing countries. A number of recent research
papers have applied the tools of RCTs to vary terms of credit or tenancy, and have
overcome some of the limitations of earlier work. Take the case of tenancy. My own
work with Abhijit Banerjee and Paul Gertler showed how Operation Barga, a tenancy
reform programme carried out in West Bengal in the late 1970s and early 1980s,
changed tenancy arrangements and improved agricultural productivity. However,
despite our best efforts, given the data we could not rule out the role of other policies
that were carried out at the same time such as empowering the panchayats. In a recent
RCT carried out in Uganda, the research team collaborated with the Bangladeshi NGO
BRAC (Building Resources Across Communities) to induce randomised variation in
real-life tenancy contracts. As part of their operations, BRAC leased plots of land to
women from low socioeconomic levels who were interested in becoming farmers,
effectively acting as the landlord. In the experiment, some tenants received a higher
crop share (75%) and some a lower crop share (50%). The study, which was carried
out by a group of researchers that included two of my former Ph.D. supervisers from
the London School of Economics (LSE), Konrad Burchardi and Selim Gulesci, found
that tenants with higher output shares used more inputs, cultivated riskier crops, and
produced 60% more output relative to those in the control group. While these effects
are reassuringly similar to those that we had found earlier, the nature of the new
evidence ensures that the new study is not subject to the methodological limitations
ours had to face.  

Criticisms of RCTs from inside and outside the world

of academic research
The main ‘inside’ criticisms of RCTs – from within the world of academic research
(for example, by recent Nobel Laureate Angus Deaton) – are as follows.

First, while RCTs overcome some problems of evaluating individual programmes, the
typically small sample size of these studies implies that the conclusions cannot be
generalised to the whole population or extended to other environments. Moreover,
there is the possibility that these studies may also be partly picking up the sheer effect
of being observed by the researchers and the surveyors, which creates a bit of an
artificial environment and therefore may give a biased picture of how the programme
will work out when it is not being surveyed (the so-called ‘Hawthorne effect’).

Second, if some programme works well, we do not know if there is another

programme that would have worked better.

Third, if a policy worked well, it is hard to infer the exact mechanism by which it
worked – for example, does microfinance work by making credit more available or is
it something that empowers women, or both?

There is some validity to each of these criticisms. However, every method has some
limitations and to find a way forward one has to either come up with a better method
or improve the existing method. Another promising direction is to harness the synergy
of different methods – for example, it may be worth exploring how RCTs can be
combined with other tools of economics, such as theory and simulation. Theory is
good at coming up with alternative narratives that connect cause and effect, but it is
not very good at determining what may be going on in a given environment. This is
exactly as in medical science – theory gives us a first hunch as to what has happened
while empirics are diagnostic tests which may confirm or disprove or modify the
original hunch. A recent research trajectory that combines theoretical models with
RCT evidence to carry out policy simulations that estimate the effect of hypothetical
alternative policies tells us what else could work even better, as well as what the likely
effect will be in a different environment.

Then there are ‘outside’ criticisms of RCTs.

Some wonder why academic economists should do policy evaluation. Should that not
be left to policymakers? After all, as economists, we know the value of comparative
advantage and specialisation. As much as science and engineering are different fields,
should research not be separate from policy work, whether it is formulation of policy
or its evaluation?

There is also some concern that, because RCTs require lots of funding, the missions of
certain donor agencies and philanthropic organisations may distort the direction of
research – as much as the profit motive of pharmaceutical companies can influence
the agenda of medical research.

Then there are ethical considerations regarding experimenting on human subjects.

These range from depriving those in the control group of a beneficial programme, to
manipulating the behaviour of individuals in the treatment group, which raises
questions of transparency and informed consent. 

Another criticism is that, since policymaking happens in a political framework, to take

a purely technocratic view about evidence-based policy and incremental
improvements may be misguided at best, and at worst, the equivalent of putting band-
aid on a serious injury.

Once again, there is some validity to each of these criticisms. But they provide a
partial picture. Policymaking may be too important to be left to policymakers only.
After all, we have seen too many instances of policy formulation that oversimplify
problems and take a centralised one-size-fits-all approach. In the Indian context, some
of the major policy shifts, such as demonetisation or goods and services tax (GST)
implementation, or making the Aadhaar3  card mandatory, were done without any
grounding in evidence or without first testing the waters. Yes, there are ethical
considerations regarding the design of experiments, as well as the need for
accountability regarding how well the research agenda fits the development priorities
of a country. But that points to the need for developing a legal and ethical framework
that governs research, not to abandon a particular method. It is also true that the kind
of programmes that are studied offer incremental improvements but it is not the case
that stopping doing these would unleash more major initiatives, whether on the part of
the government or by other actors, including the people themselves.

To me one of the most significant legacies of the RCT research agenda is to put the
importance of evidence at the centre of the table in the context of policy. Knowledge
consists of knowing both what we know and what we don’t know. The demands of
rigorous evidence make us acutely aware of the boundary between the two. Another
welcome aspect of this research agenda is its emphasis on a bottom-up rather than top-
down approach towards policymaking. The same policy may not work equally well
everywhere or for everyone in the same place. Only evidence can help improve the
effectiveness of policies by making them better suited to the specific needs of an area
or a group of people. This can provide a much-needed corrective to the top-down,
one-size-fits-all approach that, sadly, is a feature of centralised policymaking, whether
in contemporary India or in the failed model of central planning.
Maitreesh Ghatak,
London School of Economics; IGC India