UU-EDUD-803

Educational Policy: Theory and

Practice
UU-EDUD-
803

Week 5
Evidence-
based Policy-
making

1
 UU-EDUD-803

Table of Contents

5.1. What is evidence based policy? 4

5.2 What is evidence? Using data for policy making 6
References 11

2
UU-EDUD-803

Learning Outcomes

At the end of this week, students will be able to:

• Understand the use of data for policy-making

• Understand how to select and analyse data for policy- making

• Compare educational systems at international level

• Report the outcomes of the data analysis

• Identify educational issues at international level

• Formulate national overviews for developing policy

UU-EDUD-803

5.1 What is evidence based policy?

We are normally told that information is power, or that knowledge is power,

however, the only way to have good quality information and knowledge is by
having strong foundations to build information and knowledge, i.e. evidence.
Therefore, we can say, that evidence is power.

What do we mean when we talk about evidence? For example, as portrayed by

Sutherland in Zajda (2005) [p.346]:

“Many educators were concerned to find that girls’ school performance was
than that of boys, especially in maths and science. Various historical reasons
for the discrepancy were proposed – lack of ambition by girls or even fear of
being seen to be too clever, too academic. Whatever the reasons adduced for
girls’ ‘under-performance,’ determined efforts were made in a number of
developed countries to improve girls’ attainment, especially in the ‘masculine’
subjects of maths and science (notably the ‘hard’ sciences). Efforts of various
kinds were made (e. g., Whyte, 1986), to improve girls’ progress in science
and technology – for instance, teaching in single-sex classes! But almost at
the same time, other evidence was published, for instance OECD (1986) and
Sutherland (1988), that in some cases girls’ general performance at the end of
secondary school was superior to that of their male compeers”.

For Fowler (1962) evidence is the descriptive material that should be

presented as evidence to support at least some of the most important
claims made about a problem. The best forms of evidence are dramatic
anecdotes, atrocity stories, and statistics – especially big statistics drawn from
official sources.

To develop evidence-based policy, it is important to understand the Principles

of Evidence-Based Policymaking. Evidence-based policymaking has two
UU-EDUD-803

objectives: to use what we already know from programme evaluation to make

policy decisions and to build more knowledge to better inform future decisions,
prioritising rigorous research findings, data, analytics, and evaluation of new
innovations instead of anecdotes, ideology or marketing.

This can be achieved by using research findings to inform new policies or

improve effectiveness of existing programmes, supporting data collection and
analysis for research and management.

The core principles of evidence-based policymaking are guidelines that apply at

every level and reflect an agreement among researchers across the
ideological spectrum. The principles can be seen as follows:

Principle One: Build and compile rigorous evidence about what works, including

costs and benefits [2iv & 3v].

UU-EDUD-803

Principle Two: Monitor program delivery and use impact evaluation to

measure program effectiveness [5vi].

Principle Three: Use rigorous evidence to improve programs, scale what works, and
redirect funds away from consistently ineffective programs.

Principle Four: Encourage innovation and test new approaches.

These four principles support the development of policy by using elements that
constitute evidence which are basically: Data and Research

6
UU-EDUD-803

5.2 What is evidence? Using data for policy making

Evidence is the core element for identifying and analysing a problem and it has
two main sources: Data and Research. Educational Data can be obtained
by several means, before start using data. The key is to learn to identify
the problem, to check how the problem affects a group, and how to draw
and develop strategies using the data collected and analysis to address a
problem. For OECD (2015), “it is important to point out that measuring policy
impact through consistent quantitative and qualitative indicators would allow
for more accountability and strengthened knowledge for policy-makers to
make more evidence-based policy choices” (p.17).

Data driven policy-making, is the process in which data is considered the key
element that provides evidence to resolve a problem. For Mandinach, Honey,

& Light (2006)viii, “when people look at data from a number of different groups
all together, they can miss differences among the groups. Understanding
what constitutes a significant difference among groups and how to
interpret interactions are also critical skills. It is important to define the groups
and then examine the distributions of the groups separately to discern
potential differences. Understanding that there is normal variability in every
process is yet another core skill. Small changes from one time to the next,
such as those that occur when retaking the same test, can indicate
nothing or something significant. The key is deciding when to pay attention to
the differences”.

Davies (1999) defines evidence-based policy-making as an approach that

“helps people make well informed decisions about policies, programmes and
projects b y putting the best available evidence at the heart of
policy development and implementation”.

According to Niemi (2007), ”the aim of research/evidence-based or

research/evidence informed practice is promoting economic competitiveness
and social cohesion by improving educational resources, structures, and
practices. For promoting these two objectives, we need an educational
infrastructure that provides all learners with opportunities to obtain an
7
UU-EDUD-803

education at the highest level commensurate with their own growth and
growth potential”.

As stated in the EU Eurydice Report (2017), “while there is a wide range of

evidence available, the challenge is to ensure that the evidence used in
policy- making is pertinent to current policy issues. There are two
important challenges here: timeliness and relevancy. Firstly, constant
societal change implies that policies must be adapted to address new and
emerging issues (such as coping with migrants, radicalisation, youth
unemployment etc.). As a consequence, the evidence to support policy-
making often does not exist when it is needed. Secondly, the complexity
and multidimensional nature of education system poses challenges to
produce relevant and reliable evidence using an appropriate methodology”
(p.9).

Figure 1
Eurydice Sources of evidence used in policy-making in the field of education, 2016

For OECD (2016), “the policymaker tasked with developing a response to a

particular issue, it is often not fully clear what kinds of evidence are needed in
order to address key policy issues – and in fact there may be multiple paths to
a particular evidence-based solution. Apart from the use of knowledge by
policy-makers, important questions are how and where knowledge is produced
8
UU-EDUD-803

and how it is transmitted to policy-makers. In this context, the role of brokerage

agencies is becoming increasingly important” (pp. 19-20).

One of the elements that can provide us with evidence is data, however,

as stated in the Eurydice report (2017)xiii ,“descriptive system data (for

example on achievement or graduation) needs to be combined with
research findings that can determine whether something is working or not.
In these processes, for example, knowledge brokers can play a role,
combining different sources of data to produce evidence, and also tailoring
research findings so they may be more easily used by policy-makers”

(p.11). As for Evidence.Org (2016)xiv, “several other elements are critical

to the evidence-building process. One is independence: evaluators must be
impartial to ensure the validity and rigor of new evidence.

Agency leaders can help evaluation staff identify the most relevant questions
to examine, but the evaluation itself and the publication of results must be
free of influence from program or political leadership that might have a
vested interest in a particular outcome. Privacy must also be protected
when using or sharing data, particularly administrative and statistical data
containing personally identifiable information” (p.5).

As for UNESCO (2013), “founding policy action on solid and reliable evidence
is crucial not only for ensuring that policies are effective and efficient, but also
for ensuring that they are acceptable and implementable at the practical level.
Yet, what constitutes sound evidence can be debatable, particularly
where there are multiple sources and disagreement regarding interpretation
of key data and information. Policy analysis must therefore pay close
attention to verifying the degree to which a policy has incorporated clear and
unambiguous assessment and research evidence about the issues and
needs within the country’s context, and about existing implementation
capacity” (p.24). However, ‘the reliability and comparability of indicators
depends on the availability and quality of the underlying data. A vital task in
education policy analysis is to verify whether student-level data have been
collected from all schools throughout the country’s territory. It is equally
important that all such data are available disaggregated by the subgroups
9
UU-EDUD-803

mentioned above (gender, administrative area, geographical location, socio-

cultural groupings, education level and type of providers). Such
disaggregated data is necessary for comparing access and participation
by different population groups and for identifying those who are
educationally disadvantaged, as well as the degree of equity and the scale of
disparities in access to education. If available on a year- to-year basis, such
data can enable the monitoring of changes over time and the identification of
development trends. These trends can point to future policy priorities and
guide more balanced development of the education system” (p.28). It is
also important to note that “data should be compiled from each provider
about their methods, and the providers should be asked to use the same
data collection methods, so that the data from each can be easily
compared” (p.57), because “data and statistics are essential as a basis
for formulating evidence-based policies and implementing and monitoring
result- oriented programmes. However, data can only be useful if they are
relevant, reliable, consistent and available in a timely manner” (p.73).

According to the UNESCO Institute for Statistics, the quality of data depends
on several dimensions including: Relevance to Policy; Validity and
reliability; Potential for disaggregation; Timeliness and punctuality; Clarity
and Transparency; Comparability through standards; Accessibility and
Affordability; Consistency over time and space.

UNESCO (2013) also notes that it is important to note the quality of data:
Accurate, reliable and consistent information can enable a decision- maker to
make appropriate and timely decisions or policy choices. Weaknesses on
the part of data providers – from inadequate technical skills to lack of
motivation and genuine commitment – often result in much of the collected
data not being used, which is a waste of the resources invested into
compiling that data and ‘inkages between planners and statisticians: Statistics
play a major role in every step of the policy cycle. Lack of effective co-
ordination between policy-makers, planners and statisticians can lead to
difficulties in formulating sound policies that are based on relevant and
accurate information and data. Planners often prepare plans that include
various targets and monitoring indicators without much involvement from

10
UU-EDUD-803

statisticians and without their professional validation as to whether the

indicators they have identified are feasible, meaningful and realistic to
measure. At the same time, statisticians are often busy collecting data without
much consultation with the education planners, assuming that all the data will
be useful for the planners (p.74).

Figure 2
Linkages between planners and statistics

Finally, it is noted that is necessary to focus into transforming data into relevant
information and useful policy: a huge amount of data is collected in each
country, but not enough attention is paid to how to transform this data into
useful information and use this information as a basis for in decision-making.
In relation to this, it is worthwhile to note the following:

We have more information now than we can use, and less knowledge and
understanding than we need. Indeed, we seem to collect information
because we have the ability to do so, but we are so busy collecting it that we
haven’t devised a means of using it. The true measure of any society is not
what it knows but what is does with what it knows.

11
UU-EDUD-803

References

Burns, T. and F. Köster (eds.) (2016). Governing Education in a Complex World,

OECD Publishing, Paris.

Davies, P. (1999). What is evidence based education? British Journal of Educational

Studies, 47(2), 108–121.

European Commission/EACEA/Eurydice. (2017). Support Mechanisms for

Evidence-based Policy- Making in Education - Eurydice Report. Luxembourg:

Publications Office of the European Union.

European Commission/EACEA/Eurydice. (2017). Support Mechanisms for

Evidence-based Policy- Making in Education - Eurydice Report. Luxembourg:

Publications Office of the European Union.

Evidence.Org (2016). Principles of Evidence-Based Policymaking, (September).

Fowler, F. C. (1962). Policy Studies for Educational Leaders: An Introduction.

Mandinach, E. B., Honey, M., & Light, D. (2006). A Theoretical Framework for Data-

Driven Decision Making. AERA, 1–18.

Milner, J. and Eldridge, M. (2016). From Evidence to Outcomes: Using Evidence to

Inform Pay for Success Project Design. Washington, DC: Urban Institute.

Niemi, H. (2007). Equity and Good Learning Outcomes. Reflections on Factors

Influencing Societal, Cultural and Individual Levels - The Finnish Perspective.

Zeitschrift Für Pädagogik - PeDocs - Deutsches Institut Für Internationale

Pädagogische Forschung (DIPF), 53(1), 92–107.

OECD. (2015). Education Policy Outlook 2015: Making Reforms Happen. OECD

Publishing.

Oxford Dictionaries Online, s.v. “evidence,” accessed August

26, 2016.

Tatian, P. (2016). Performance Measurement to Evaluation. Washington, DC: Urban

Institute.
12
UU-EDUD-803

UNESCO. (2013). UNESCO Handbook on Education Policy Analysis and

Programming. Volume 1: Education Policy Analysis. Bangkok: UNESCO

Bangkok Asia and Pacific Regional Bureau for Education.

UNESCO. (2013). UNESCO Handbook on Education Policy Analysis and

Programming. Volume 1: Education Policy Analysis. Bangkok: UNESCO

UNESCO. (2013). UNESCO Handbook on Education Policy Analysis and

Programming. Volume 1: Education Policy Analysis. Bangkok: UNESCO

Bangkok Asia and Pacific Regional Bureau for Education.

Zajda, J. (2005). International Handbook on Globalisation, Education and Policy.

Research Global Pedagogies and Policies. Amsterdam.

13