ACADEMIA Letters

Evaluate government policies holistically - don’t judge a

book by its cover alone
Amod Kumar, Indian Institute of Management Lucknow
Sanjay Kumar Singh, Indian Institute of Management Lucknow

1. Introduction
Government welfare policies are complex, having possible unanticipated and unexpected ef-
fects – either positive or negative. These could be attributed to the fact that any social phe-
nomenon can have multiple, cyclical, and volatile causes and effects, which may vary as per
the dynamic state of socio-political, cultural, or scientific determinants in the society (Oliver
et al., 2019; Anderson, 2004; Merton, 1936). Recent research identifies three key reasons that
lead to unanticipated effects - limited pre-implementation testing, lack of evidence due to cost
and time limitations, and absence of robust and explicit theoretical foundation (Oliver et al.,
2019).
Policymakers are constantly faced with this dilemma of balancing pre-defined or specu-
lated intended and unintended effects of a policy at the time of policy formulation, scale-up,
alteration, and termination. The ex-ante evaluation of the policy at the formation stage is
rooted primarily in available evidence, state of resources, and socio-political needs. It is be-
lieved to lead to desired outcomes. Limited availability of data, evidence, and the complex
nature of the outcomes make it very difficult to predict or foresee all major unintended effects
of the policy. However, subsequent ex-post evaluations of the policy may have a greater possi-
bility of examining these unintended effects along with the stated primary effect of the policy.
Moreover, knowledge of these unintended effects – positive or negative - become important
in scenarios when the policy is under artificial stress or is unpopular among specific segments

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.

1
of polity, business, or society due to the state of the direct effects of the policy.
The Right to Education Act, 2009 in India is one of the largest affirmative action pol-
icy interventions globally. The Act mandates free and compulsory education for all children
between 6-14 years of age or 20% of India’s population, with a focus on creating a holistic,
inclusive, and enabling learning environment through improved quality of schools, education
delivery, and educational outcomes (Shah and Steinberg, 2019). The clause 12.1.(c) of the
Act requires neighborhood private unaided schools to reserve 25% of their entry-level classes
for disadvantaged or economically weaker section children. The clause could potentially im-
pact 20 million children over the next decade (Sarin et al., 2015), with 30% of them coming
from Uttar Pradesh, India’s largest state with a population of 230 million. Prior research sug-
gests that making good quality schooling accessible to disadvantaged children improves their
educational outcomes - a key driver for long-term poverty alleviation, besides creating inclu-
sive classrooms by overcoming discriminate access to quality schools (Dreze & Sen, 2013;
Hanshuek & Zhang, 2006; Tilak, 2002).
Recent studies on the Act have primarily focused on the intended effects of the policy,
mainly using input (Kar, 2019; Bhattacharjee et al., 2016; Malik, 2015) or outcome-oriented
lenses (Bhattacharjee, 2019; Mansoor et al., 2019). The input research has focused on imple-
mentation, policy, and technological challenges and interventions, while outcome-oriented
research has focused on immediate school-level learning and behavioral outcomes. Section
12.1.(c) of the RTE Act, in particular, has been debated among policymakers and stakeholders
alike since its inception for its applicability and effectiveness and has been subjected to severe
resistance.
This paper intends to examine the unintended effects of RTE section 12.1.(c) and in par-
ticular looks at the effect of the policy on child’s household environment, and provide further
evidence to the above argument.
The remainder of the paper is organized as follows. Section 2 consists of a literature
review and theoretical foundation on the topic of interest, discussing various aspects of the
study. Section 3 explains the research design and methodology, followed by results in section
4 and discussion and conclusion in section 5.

2. Literature Review and Theoretical Foundation

The concept of unintended effects of government policies is not new, and is defined as the
law of unintended consequences (Merton, 1936; Hargreaves et al., 2002), which states that
government actions always have unanticipated or “unintended” effects. These unintended
effects arise out of gaps in formulation or implementation of these policies leading to conflicts

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.

2
and consequences not aligned to the original intent. Adam smith’s “invisible hand” is in fact
a reference to the same (Smith, 1761). Frederic Bastiat (1850) in his famous essay “What Is
Seen and What Is Not Seen” differentiated a good economist from a bad one by the ability to
account for unforeseen unintended effects.
It was Robert K. Merton (1936) who in his famous work, the unanticipated consequences
of purposive social action, first mentioned five sources of such actions – ignorance, error,
imperious immediacy of interest, lack of basic values, and self-defeating principles. Recent
research has highlighted that the unintended effects of policies are caused when these poli-
cies produce: a) counterproductive or opposite effect, b) no effect, or c) effects other than
those intended or desired. Hacker (2006) argues that policies which are formed as an emo-
tional response to social issues, or with significant incoherence between policy formulators
and implementors, or which ignore the views of those impacted directly by it are most prone to
unintended effects. These unintended effects may not be totally avoidable but can be mitigated
by explicit and sound theoretical grounding, greater stakeholder engagement, and concurrent
monitoring and evaluation.
Unintended effects are different from the main effects of a policy, and can be either positive
– which helps in achieving overall social and human development goals, or negative – which
impedes achievement of these goals in any manner.
As an example of positive unintended effect, Duflo (2000) in her study found out that a
pension scheme meant for older women in South Africa had a positive effect on the grand-
daughters, who were healthier compared to those of women not covered by the pension scheme.
Similarly, as an example of negative unintended effect, the restricted steel import policy of
USA which aimed to protect local steel manufacturers against low cost imported steel, did
help the steel companies but rendered USA automobile-makers less competitive compared to
their global peers (Norton, 2020).
Prior research has treated unintended effects of policies in a mostly negative manner.
There is scarcity of literature on unintended positive effects, which is important especially for
the evaluation of important policies ex-post.

3. Research Design
3.1 Sampling, Data, Variables, and Data Collection
To test any unintended effect (positive or negative), we have used RTE 12.1.(c) lottery data of
Uttar Pradesh. A district wise lottery is conducted annually to choose economically weaker
or disadvantaged children for admissions against the RTE 12.1.(c) quota. The district wise

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.

3
availability of seats under the quota are displayed on the government website at the begin-
ning of an academic year. Applicants apply with ordered preference of the schools in their
neighbourhood, which are allocated through a random lottery. Additional rounds of lottery
are conducted in case of any vacant seats. RTE lottery data from 2017 and 2018 are used.
During this period, a cumulative of 121,824 applications were made, out of which 69,236 ap-
plicants were selected and 52,588 rejected. As we found that 20 districts constituted majority
(90%) of the applicants during the selected time period, we adopted convenience sampling
and randomly selected 12 districts of which a sample of 1,047 applicants was drawn in the
ratio of 75 (selected) : 25 (rejected).
The survey tool to measure study outcomes was adopted from the National Council of Ap-
plied Economic Research (NCAER) and questions from Human Development tool of NCAER
were modified to assess the unintended effects at household level on selected and rejected ap-
plicants. Four dependent variables were used to measure the same: a) change in school of
other household children subsequent to the focal child’s selection or rejection in lottery; b)
total number of books available in the household (no books, less than five books, five to ten
books, and more than ten books); c) use of newspapers and radio at home ( yes, no); d) use
of mobile communication applications and social media. The independent variable being the
selection or rejection of the applicant in RTE lottery.
The data collection was done using phone based surveys, as participants were dispersed
across a large geography. The survey was conducted by trained surveyors who were blind to the
actual status of applicants being interviewed by them, using an interactive and conversational
approach.

3.2 Empirical model

The empirical model used is shown below. Yi is the dependent variable for applicant i, RTE
12.1.(c) (Rejected or Selected) will take the values of 0 or 1 in case of rejected or selected
applicants. Xi is the control variable (sex, number of siblings, caste, religion, HH income).
Bd represents cumulative district level effects, and E is the error term. Effect of RTE selection
or rejection on dependent variable will be determined by B1
Yi = Bd + B1 RTE 12.1.(c) (Rejected or Selected) + B2 Xi + Ei

4. Results
The study results show that selection or rejection in the RTE 12.1.(c) lottery had a significant
positive effect on households decision to change the school of other children in the family ( B1

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.

4
= 1.006, p < 0.05, se = 0.240, buy more books for studies ( B1 = 0.121, p < 0.05, se = 0.0432),
and to greater use of radio and newspaper at home ( B1 = 0.367, p < 0.05, se = 0.0471), while
a negative effect was seen in use of mobile communication and social media ( B1 = -0.380, p
< 0.05, se = 0.106).

5. Discussion and conclusion

This study aimed to assess the unintended household level effects of RTE 12.1.(c) implemen-
tation, and examined if access to better schools leads to household level changes and if these
changes are positive or negative. We examined our hypothesis by using RTE 12.1.(c) appli-
cant data from Uttar Pradesh. The results from our study show that a policy intervention such
as 12.1.(c) can have far reaching household level effects (unintended positive). These positive
effects not only ensure creation of enabling learning environment at home for the beneficiary
child but also for other household children, while making parents put in greater efforts to en-
sure the same. Moreover, any evaluation of RTE 12.1.(c) without considering these effects
could mean underestimating the cost-benefits of the policy. Our study is an attempt to gen-
erate evidence for policymakers and implementers to follow a holistic approach in ex-post
policy evaluation with an eye on unintended effects of the policies.

References
Anderson, B. A. (2004). Unintended population consequences of policies. Population and
Environment, 25(4), 377-390.

Bastiat, F. (2001). What is seen and what is not seen. Ideas on Liberty, 51, 12-16.

Bhattacharjee, M., & Mysoor, D. (2016). “Unredressed” grievances under RTE: navigating
the state Labyrinth. Governance, 29(1), 31-45.

Bhattacharjee, S. (2019). Ten Years of RTE Act: Revisiting Achievements and Examining
Gaps. ORF Issue Brief, Issue No. 304.

Drèze, J., & Sen, A. (2013). An uncertain glory. Princeton University Press.

Duflo, E. (2000). Child health and household resources in South Africa: evidence from the
old age pension program. American Economic Review, 90(2), 393-398.

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.

5
Hacker, A. J. (2006). Unintended consequences in public policy: Formulation and imple-
mentation of Michigan’s Safe Delivery of Newborns law. Western Michigan University.

Hargreaves, A., Earl, L., Moore, S., & Manning, S. (2002). Learning to change: Teaching
beyond subjects and standards. John Wiley & Sons.

Hanushek, E. A., & Zhang, L. (2006). Quality-consistent estimates of international returns

to skill. NBER Working Paper No. 12664.

Kar, N. (2019). A Study on Right to Education (RTE) Act, 2009 and its compliance in Schools
of Golaghat District of Assam. Humanities & Social Sciences Reviews, 7(6), 570-584.

Malik, B. B. (2015). RTE and marginal communities: A perspective from the field. Economic
and Political Weekly, 50(5), 25-27.

Mansoor, S., Dharangutti, Y., & Mane, A. S. (2019). The Fundamental Right to Education
as against its implementation in the contrarian settings of Naxalism in India. Journal of
Critical Reviews, 6(6), 199-204.

Merton, R. K. (1936). The unanticipated consequences of purposive social action. American

sociological review, 1(6), 894-904.

Norton, Rob. 2020. Unintended Consequences. Online at: https://www.econlib.org/library/

Enc/UnintendedConsequences.html

Oliver, K., Lorenc, T., Tinkler, J., & Bonell, C. (2019). Understanding the unintended conse-
quences of public health policies: the views of policymakers and evaluators. BMC public
health, 19(1), 1-9.

Sarin, A., Kuhn, S., Singh, B. D., Khanghta, P., Dongre, A. A., Joshi, E.,…& Rahman, F.
(2015). State of the Nation: RTE Section 12 (1)(c). Available at SSRN 2637817.

Shah, M., & Steinberg, B. (2019, May). The right to education act: Trends in enrollment,
test scores, and school quality. In AEA Papers and Proceedings (Vol. 109, pp. 232-38).

Smith, Adam (1761). Theory of Moral Sentiments. Oxford: Oxford University Press.

Tilak, J. B. (2002). Education and poverty. Journal of Human Development, 3(2), 191-207.

Academia Letters, August 2021 ©2021 by the authors — Open Access — Distributed under CC BY 4.0

Corresponding Author: Amod Kumar, efpm06006@iiml.ac.in

Citation: Kumar, A., Singh, S.K. (2021). Evaluate government policies holistically - don’t judge a book by its
cover alone. Academia Letters, Article 2946. https://doi.org/10.20935/AL2946.