Measuring the effects of environmental policies: an issue of methodology and legitimacy

EPA Evaluation, Washington DC, June 2006

Point of departure
Quantitative methods are only one type of method to evaluate policy
They work well under certain assumptions and For certain purposes

Policy analysis often requires qualitative approach in addition to ‘data’/numbers oriented approaches The issue is not quantitative or qualitative, but which method, when and how?
EPA Evaluation, Washington DC, June 2006

Policy is ...
A complex, multi-faceted proces including multiple actors and institutions, multiple fases, multiple measures and instruments, multiple secondary processes,... In other words, there is no such thing as a simple, straightforward policy evaluation from the perspective of the social policy scientist/evaluator
EPA Evaluation, Washington DC, June 2006

Policy evaluation: basic questions
Why evaluate?
What are the results of policy making? (judgement) Legitimacy through results (political) Pragmatic improvements (learning)

Many evaluations answer this question in a very restricted way. This obscures the debate about policy, about the environment and also about the value of the discipline of evaluation
EPA Evaluation, Washington DC, June 2006

Plan evaluation
Mostly ex ante: will this policy/plan work, why, how, with what effects?
Importance of the policy theory Instrument choice Context elements

Increasingly done in multiple policy fields Strong demand/activity from civil society actors
EPA Evaluation, Washington DC, June 2006

Process evaluation
Process: the interactions between actors,
institutions with the goal of environmental policy making and implementation

Relevant questions:
who was involved, why and how? status of diff. elements of the process, legitimacy of the process link between the process and the outcomes
EPA Evaluation, Washington DC, June 2006

Instrument evaluation
Instruments: broad definition
Steering instruments: carrot, stick, preech or legal, economic, social Direct instrument: actions by the government: ex: infrastructure

Relevant questions:
does the instrument work? circumstances: context, prerequisites instrument mix
EPA Evaluation, Washington DC, June 2006

Shift towards effectiveness evaluation Why?
At the end of the day, the results are what matters link with legitimacy! State of policy making vs. state of the environment

Attention from:
historically: environmentalists scholars studying environmental policy policy makers: EU, OECD
EPA Evaluation, Washington DC, June 2006

Effects evaluation: basics
Effectiveness
Causal relationship between the policy and goal attainment

Side effects evaluation
Causal relationship between the policy and side effects (=unintended effects)

Effects evaluation
Combined appraisal of effectiveness and side effects evaluation
EPA Evaluation, Washington DC, June 2006

Societal effectiveness

NEEDS

MEANS -

SOCIETAL

OUTCOME

IMPACT

INSTRUMENTS OBJECTIVES OUTPUT

MEANS –

POLICY PROCESS

INPUTS

Institutional effectiveness Target group effectiveness Impact effectiveness EPA Evaluation, Washington DC, June 2006

Why include side-effects?
More complete cost/benefit analysis of environmental policies Attention for the intrinsic interconnectedness of policy goals and outcomes in different policy domains Important in the public discourse and appreciation Legitimacy of the policy
EPA Evaluation, Washington DC, June 2006

Methodology (1/2)
‘Modus Narrandi’ (cf. Modus Operandi) 4 steps
A. Reconstruction of the effect spectrum
Reconstruction of the policy theory Reconstruction of the blind spot How? Documents, interviews, literature, experience

B.Valueing effects
Observed changes: linked to goals and side-effects How? Data, if available; proxi-variables; qualitative data
EPA Evaluation, Washington DC, June 2006

Methodology (2/2)
C. Establishing causality
Causal chains: Logical linkages between output, outcome, impact Causal proximity Causal contribution How? Analytical models, interviews, expert judgements

D. Causal narrative story:
‘Putting things together’ How: logic, triangulation, ‘craftsmanship’

Shifting the burden of proof
EPA Evaluation, Washington DC, June 2006

Modus Narrandi Reconstruction of the effects spectrum
Establi shing the effects

E stablishin g th e un derla ying principles to achie ve th e goal
Reconstruction of the causal and impact model

Judging the merit of the policy theory

Valuing of effects

Reconstruction of the ca usal narrati ve sto ry

Em pirical da ta

Triangulation

Causality
Pol icy the ory

Illu strate cau sal links Ca usal proxim ity

Ca usal con tribution

Conclude and make statement on (side) effectiveness