PUBLIC MANAGEMENT

SESSION 3
Guillem Ripoll

24/01/2023
 WHAT IS
PUBLIC POLICY EVALUATION?

2
One of the great mistakes is to judge policies
and programs by their intentions rather than
their results (Milton Friedman, 1975)

3
 WHAT DOES ‘EVALUATION’ MEAN?

 How many times per day do you evaluate?

 Things and people
 It is also possible to evaluate policies, projects and programmes …
 What’s the difference between the two?
4
 DEFINING PUBLIC POLICY EVALUATION

 Rigorous assessment of the interventions of public agencies according to their

products and impacts on the needs they intend to satisfy (European Commission,
2007)
 Main elements:
 Academic / Scientific rigor (common scientific principles apply)
 Evidence-based information for decision-making
 (social) Science research methods, both quantitative and qualitative
 Practical usefulness

5
WHY DO WE EVALUATE PUBLIC POLICIES?

Scientific learning

Public learning

Compulsory accountability

Voluntary/External accountability
6
 PUBLIC POLICY EVALUATION:
A TYPOLOGY (I)

Moment Ex-ante Focus Needs

In-itinere Design

Expost Implementation

Impact

Economic 7
 PUBLIC POLICY EVALUATION:
A TYPOLOGY (II)

Needs Design Implementation Impact Economic

• Nature of the • Inspecting the • Practice and • Experiment/causal • Cost-benefit
problem, needs to logic of theory: resources, relationships • Are the costs of
attend intervention activities and • Does it improve the policy
• Does the current (science-based, outputs the past situation? justified? Could the
situation justifies needs…) • Has it been applied • (in-itinere, ex-post) same benefits
the need of a/the • What’s the logic of as designed? have been
program? intervention? Is it • (in-itinere, ex-post) obtained with
• (ex-ante, ex-post) well-designed? fewer costs?
• (ex-ante, in-itinere, • (in-itinere, ex-post)
ex-post)

8
 Problem

Problem
Evaluation
definition
Evaluation
Economic Needs Evaluation
ex-post
Impact ex-ante

Implementation
Identify
Implementation
solutions
Evaluation
Design Evaluation
In-itinere
Select solution ex-ante
Evaluation
and design
ex-ante 9
 PUBLIC POLICY EVALUATION:
A TYPOLOGY (III)

• All right… but, which is the good one?

• It’s better to ask
• Who wants the evaluation? What are their needs?
• Which is the evaluation that will bring a greater added value?
• What are the possibilities to develop the evaluation?
• What are your resources?

10
 PRACTICAL ISSUES IN
PUBLIC POLICY EVALUATION
 The socio-political context is there
 History of the problem and interventions,
actors and stakeholders, people
 Manage the acceptance of the evaluation
 The final product is for the others, not for us
 Do they have interest in the evaluation?
 Do they see us (evaluators) as ‘foreigners’?
 Do we have legitimation to develop the
evaluation?
 Who has to be involved in the evaluation?
IN/OUT dilemma
 Stakeholders (interested/involved in…)
 Experts, ‘public’ workers , consultors,
advisors, bureaucrats
 Politicians
 NGO’s, think tanks, pressure groups
 Independent entities (e.g. consumers’
protection organization)
 Citizens
 Common problems
 Lack of resources (time, data, support,
expertise, money…)
 No available information/data/docs

11
 HOW TO PLAN A PUBLIC POLICY
EVALUATION?

 How to do the evaluation? – What are our resources?

 Expertise in the problem being tackled by the public policy
 Expertise in the public policy
 Knowledge, personal contacts and affinities
 What does justify the public policy evaluation? → learning / accountability +
context
 What are the aims of the public policy evaluation? → which one?
 Who will find it useful?

12
BASIC NEEDS EVALUATION

13
 A RULE FOR SUCCESS IN BUSINESS

▪ If you are the owner of NESTLÉ, and want to make more money by launching new
KitKat products in Spain… What do you need to do before going into action?
Market analysis
14
 LET’S SEE A MORE CLOSE EXAMPLE

▪ How many old people can’t take care of themselves?

▪ … and how are they?
▪ … and what services do they need?
▪ … and how to define the public interventions to improve their situation?
15
SOCIAL PROBLEMS AND PUBLIC INTERVENTIONS

Current Desired
state of state of
situation X situation X

Public
intervention

16
 THE RAISON D’ETRE OF
BASIC NEEDS EVALUATION
▪ How does a public intervention work?
Social problem Public policy

▪ Policies can be…

▪ Effective

A good assessment of the social

▪ Ineffective issue or need partly determines the
effectiveness of a policy

17
 WHO EVALUATES?
AND HOW?

Citizens
Experts
▪ Objective measures
(e.g. obesity)
Evidence- Clients ▪ Subjective measures
based (e.g. fertility)
▪ A combination of the
two (e.g. poverty)

Evaluation 18
 PREVIOUS YEAR, 3rd ELG

▪ Students / future Experts

▪ Objective and subjective measures

19
 USEFULNESS OF BASIC NEEDS
EVALUATION

1) Focus on improvement of current resources

2) Development of new public policies
3) Cumulative logic across years: control variations

20
 TIMING

Ex-ante / plan. In-itinere / imp. Ex-post / account.

Needs
Design
Implementation
Impact
Economic

▪ Usually ex-ante, but also in parallel to current public interventions

▪ Logic: to see if the picture changes

21
 PREVIOUS YEAR, 3rd ELG

▪ In parallel to… / In-itinere

▪ Focus on 1 (current resources) and open to 3 (time series)

22
SPECIFYING THE EXTENT OF THE PROBLEM:
WHERE?
Target population

Reference population

Geopgraphical area

23
GEOGRAPHICAL AREA, REFERENCE POPULATION
AND TARGET POPULATION
▪ In which geographical area are we going to develop our basic needs evaluation?
▪ A specific area: EU, Spain, France, Germany, Navarra, Pamplona, Madrid, Chueca,
Iturrama…
▪ Several areas: South-european countries, Latin America, Pamplona-Tudela-Estella,
Ermitagaña-Iturrama
▪ Reference population
▪ Eerybody with X conditions
▪ Issue: school failure in Pamplona and Cizur Mayor
▪ Reference population: all the kids between 4-16 years old from Pamplona and Cizur Mayor
▪ Everybody
▪ Issue: Car accidents
▪ Who suffers the issue? Who recieves the public intervention? Target population
▪ Car accidents: young people who drives after drinking alcohol beverages
▪ School failure: kids with economic disadvantaged families

24
PREVIOUS YEAR, 3rd ELG

Geographical area:
Pamplona

Reference population:
Everybody
Target population:
Homeless outside local
shelters

25
 COMMON QUESTIONS TO BEGIN
A BASIC NEEDS EVALUATION

▪ What are the nature and magnitude of the problem to be addressed?

▪ What are the characteristics of the population in need?
▪ What are the needs of the population?
▪ What services are needed?
▪ How much service is needed, over what time period?
▪ What service delivery arrangements are needed to provide those services to
the population?
26
 PREVIOUS YEAR, 3rd ELG

▪ Four things were estimated:

▪ Public opinion: What does citizens think about homelessness?
▪ Population: How many homeless?
▪ Characteristics: What are the characteristics of the homeless? Profile
27
▪ Needs: What are the needs of the homeless?
 TWO ELEMENTS IN NEEDS ASSESSMENTS

• Explanation and estimation of the social problem

1) Performance
needs
Which is the
one we should
do first?
• What is needed to solve the social problem?
2) Treatment
needs

28
 THREE MAIN STAGES…
TO EXPLORE THE THREE ELEMENTS
1) Issue definition, causes and outcomes
E.g. Young ▪ What’s the issue? When, where and how big
people
problems to
▪ From geographical area to the target of intervention
buy a first ▪ Causes of the issue (and current outcomes)
home
▪ Useful at this stage:
▪ Academic/Scientific literature review
▪ Discourse of main actors
▪ Public opinion
▪ Not so easy… issues can be (and indeed usually are) multidimensional
▪ E.g. PIRMI (Programa Interdepartamental de la Renta Mínima de Inserción): related to
poverty, but… no single cause: no job, no family, mental health problems, addictions…
29
 THREE MAIN STAGES

1) Issue definition, causes and outcomes

2) Current needs and how to satisfy these needs
▪ How many?
▪ What do they need? / What is needed?
▪ How these needs can be satisfied?
▪ Set priorities (needs and interventions)
▪ Look at previous research: effectiveness and cost-effectiveness of different public
interventions, expert groups, comparisons…
▪ Current resources
▪ E.g. Dependency needs evaluation cannot simply say how many dependants do we
have, but also inform about how many need a residence, a day-care center… 30
 THREE MAIN STAGES

1) Issue definition, causes and outcomes

2) Current needs and how to satisfy these needs
3) Comparison between current needs and current use of
resources (needs, resources/offer, and demand)
▪ Not covered needs without being asked (7)
▪ Unattended demand (6 / 5)
▪ Attended demand without needs (4)
▪ Unnecessary offer (1)
▪ Useful offer (2 / 3) Which ones
seem
problematic?
31
 THREE MAIN STAGES

1) Issue definition, causes and outcomes

2) Current needs and how to satisfy these needs
3) Comparison between current needs and current use of resources (needs,
resources/offer, and demand)

1) EXPLAIN

32
 PREVIOUS YEAR, 3rd ELG

1 1)Problem definition (homelessness definition, causes, public concern,

good responses)
1 2)Review of homelessness in Pamplona (present situation (regulations and
resources), public opinion, politics and homelessness, profile)
2 3)Methodology (geographical area, work camp/methods, limitations)
2-4 4)Volunteers recruiting , marketing and initial strategy projection
5)Data analysis and results (how many? where? how are they? what are
2-3 their needs?

4 6)Reflection
problem)
(service-based learning, volunteering orientation, experience,
33
SOME RESULTS (I)

34
SOME RESULTS (II)

35
 SOME RESULTS (III)

▪ Method:
▪ Identification
▪ Approximation
▪ Classification questions
▪ Talk verification & incentive
▪ Survey
≈100 volunteers, 5-6 ▪ Close
members per group

36
SOME RESULTS (IV)

37
SOME RESULTS (V)

38
SOME RESULTS (VI)

39
SOME RESULTS (VII)

40
METHODS TO DO A BASIC NEEDS EVALUATION
(AND MORE)

Quantitative Qualitative Which are the

good ones?

41
 METHODS:
QUICK OVERVIEW

QUANTITATIVE QUALITATIVE
• Interviews
• Surveys • Group dynamics
OBTAIN • Observations • Observations
• Administrative data • Case studies
• Document analysis
• Content analysis
ANALYSE • Statistical analysis
• Discourse analysis
42
 HOW TO OBTAIN DATA

INE, IDESCAT, ESS Build your own

… • Limitations:
• Limitations: time,
variables list, Pre- resources
concept New
operationalization existing

43
 QUANTITATIVE METHODS:
HOW TO OBTAIN DATA (II)
▪ Surveys:

▪ Observations:

44
 QUANTITATIVE METHODS:
HOW TO OBTAIN DATA (II)
▪ Surveys:
▪ Sample of key actors (administration, clients, people with needs, key
informants…)
▪ Online, by phone, or by hand
▪ Standard questions and content
▪ One point in time vs. Time-series

▪ Observations:
▪ Sample of key actors (administration, clients, people with needs, key
informants …)
▪ Physical presence
▪ Standard questions and content
▪ One point in time vs. Time-series
45
 QUANTITATIVE METHODS:
HOW TO ANALYSE DATA
▪ Econometrics …
▪ Statistical analysis:
▪ Univariate (interpret one variable)
▪ Bivariate (interpret one variable as explained by another one, or their
interaction)
▪ Multivariate (interpret one variable as explained by others, or their
interaction)
▪ Techniques:
▪ Descriptives
▪ Contingency tables (simple and controlled)
▪ Regression analyses
▪ Structural equation models
46
 QUALITATIVE METHODS:
HOW TO OBTAIN THE DATA
▪ Interviews
▪ Talk with key actors (administration, clients, people with needs, key informants …)
▪ Aim: comprehend perceptions, ideas, values, opinions, present state of things…
▪ Structured, semi-structured, open, informal
▪ Issues:
▪ Collaboration
▪ Designing questions
▪ Record or not

47
 QUALITATIVE METHODS:
HOW TO OBTAIN THE DATA

▪ Group dynamics
▪ Talk with a group about an issue
▪ Aim: comprehend perceptions, ideas, values, opinions, present state of things…
specially when conflicting (or delicate) thoughts may arise
▪ Three types:
▪ Group discussions/Focus group (Presentation+discussion)
▪ Nominal group (Presentation+inner reflection+personal solutions+discussion+voting)
▪ Delphi (Presentation+several rounds)
▪ Issues:
▪ Collaboration
▪ Select participants (Homogeneous vs. Heterogeneous, Profile)
▪ Designing dynamics
▪ Record or not

48
 QUALITATIVE METHODS:
HOW TO OBTAIN THE DATA
▪ Document analysis
▪ Analyse reality using pre-existing materials
▪ Aim: comprehend perceptions, ideas, values, opinions, present state of things…
▪ Materials: written or video
▪ Issues:
▪ Select material

49
 QUALITATIVE METHODS:
HOW TO OBTAIN THE DATA
▪ Direct observation
▪ To gaze at a specific reality
▪ Systematic observation of routines, ways of doing, opinions…
▪ Aim: comprehend perceptions, ideas, values, opinions, present state of things…
▪ Types:
▪ Role playing
▪ Rating by qualified observers
▪ ‘Independent’ observation
▪ Issues:
▪ Collaboration
▪ Conscious or unconscious
▪ Structured or not
▪ Designing dynamics
▪ Record or not, diaries…
50
 QUALITATIVE METHODS:
HOW TO OBTAIN THE DATA
▪ Interviews
▪ Group dynamics
▪ Group discussions
▪ Delphi groups
▪ Nominal groups
▪ Document analysis
▪ Direct observation
▪ Role playing
▪ Rating by qualified observers
▪ ‘Independent’ observation

51
 QUALITATIVE METHODS:
HOW TO ANALYSE THE DATA

▪ Narrative analysis
▪ Life histories
▪ Conversation analysis
▪ Analytical induction
▪ Content analysis
▪ Discourse analysis

52
 PREVIOUS YEAR, 3rd ELG

Quantitative
Qualitative

53
 PLANNING A BASIC NEEDS EVALUATION
IN 6 STEPS

1 • Policy type and aim of the evaluation

2 • Problem, target population and services
3 • Set the evaluation questions
4 • Data sources
5 • Data analysis
6 • Resources (time, skills, …) 54