Monitoring?

Monitoring and Evaluation (M&E)

Design Monitoring and Evaluation (DME)

Participatory Monitoring and Evaluation (PME)

Vision Statement of DSK

DSK seeks a country of social justice,

where poverty has been overcome and
people live in dignity and security. DSK
aims to be a partner of choice within a
worldwide movement dedicated to
ending poverty.
THE
DME
CYCLE
 AN OVERVIEW OF DESIGN
PROCESS

PROBLEM
IDENTIFICATION
How Partners, donors, GoB
were involved (different
stakeholders)
Conduct Assessment
Was it Holistic
HLS, Gender, etc.
How rigorous it was
 Elements of Household Livelihood Security
Priority Needs
Education Community
Copping Strategy Shelter
Empowerment
Health /Participation

Food Water

Economic
Environment
Security

Context:
Means:
Social, Political and
Income, skills, time Cultural
WORLD Bi-lateral /
Multi-lateral

NATION National
Government.

STATE NGOs

DISTRICT CBOs

COMMUNITY
Social Networks
 Additional perspectives added by
Benefits-Harms Analysis
INSTITUTIONAL SOCIAL, CULTURAL,
CAPACITY ATTITUDINAL

HOUSEHOLD LIVELI-
HOOD SECURITIES

PERSONAL PERSONAL
SECURITY PARTICIPATION
 Determining appropriate precision and mix of multiple methods
High rigor, high quality, more time & expense
Extractive --- Quantitative

Nutritional
measurements
HH
surveys Focus

Participatory ---
Groups

Nutritional
measurements Focus
HH Groups
surveys
Key
Informant
interviews
Large
group

Low rigor, questionable quality, quick and cheap

 Different levels of rigor
depends on source of evidence; level of confidence; use of information
Objective, High precision, More time & expense
Level 5: A thorough research project is undertaken to conduct in-depth
analysis of situation; P= +/- 1% Book published!

Level 4: Good sampling and data collection methods used to gather data
which is representative of target population; P= +/- 5% Decision maker reads
full report
Level 3: A rapid survey is conducted on a convenient sample of
participants; P= +/- 15% Decision maker reads 10-page summary of report

Level 2: A fairly good mix of people are asked their perspectives about
project; P= +/- 25% Decision maker reads at least executive summary of report

Level 1: A few people are asked their perspectives about project;

P= +/- 40% Decision made in a few minutes
Level 0: Decision-maker’s impressions based on anecdotes and sound
bytes heard during brief encounters (hallway gossip), mostly intuition;
Level of confidence +/- 50%; Decision made in a few seconds
Subjective , Sloppy, Quick & cheap
 AN OVERVIEW OF DESIGN
PROCESS

PROBLEM ANALYSIS

IDENTIFYING TARGET
FORMULATION OF GROUPS- Characteristics of
PROBLEM TREE Target Groups:
Land
(converting need
Income
assessment to Age
problem tree) Illiteracy
Consequences Consequences Consequences

PROBLEM

PRIMARY PRIMARY PRIMARY

CAUSE 1 CAUSE 2 CAUSE 3

Secondary Secondary Secondary

cause 2.1 cause 2.2 cause 2.3

Tertiary Tertiary Tertiary

cause 2.2.1 cause 2.2.2 cause 2.2.3
 High infant mortality rate

Children are malnourished

Insufficient Diarrheal Poor quality

food disease of food

Contaminated Unsanitary Lack of sanitary

water practices facilities

Flies and Do not use People do not

rodents facilities wash hands
correctly before eating
 AN OVERVIEW OF DESIGN
PROCESS

STRATEGY FORMUALTION

Delivery Strategies:
Integrated/Coordinated Direct delivery
Approach
Partnership
 HLSA

One option Selection of target beneficiaries Another option

and proposed interventions

Single Project Design Program Design

M&E plan Multisectoral Logframe

Project Baseline Program Baseline

Single Project Implementation of
Implementation Multiple Projects
Single Sector Program
Project Evaluation Evaluation
 AN OVERVIEW OF DESIGN
PROCESS

FORMUALTION OF LOGICAL FRAMEWORK

LOGICAL FRAMEWORK
GOALS AND LOGICAL Terminologies
LINKAGES Cause & Effect
Relationship
Project Logical
Framework
Transition from problem tree to logframe
Health Project Final Goal: Diarrheal disease among
children in target community decreased from X%
(baseline) to Y% by 2005 (final evaluation) .

Effect Goal #1: Q% of community HHs will have access

to potable water

Effect Goal #2: R% of community HHs will have access

to sanitation facilities

Effect Goal #3: S% of a random sample of members of

community will demonstrate a defined set of sanitary
practices including I) anti fly and rodent measures; ii)
use of sanitary facilities; and III) washing hands

Intermediate Goals at effect level

 AN OVERVIEW OF DESIGN
PROCESS

CONCEPT LOGICAL FRAMEWORK

PAPER

CAUSE & EFFECT

NOT ACCEPTED ACCEPTED
RELATIONSHIP

TERMINOLOGIES
REFINE PROPOSAL WRITING

PROJECT
LOGFRAME
 Design Logic

Sound project design involves a clear

and logical cause-effect framework.

INPUT ACTIVITIES OUTPUT EFFECT IMPACT

 DSK’s terminology for project hierarchy
OUTCOMES
--- RESULTS ---

IMPACT Sustainable improvements in human

conditions or well-being

EFFECTS Changes in individual behavior or

systemic capacity
OUTPUTS Products of project activities

ACTIVITIES Interventions / processes implemented by project

INPUTS Resources needed by project (e.g. funds, staff, commodities)

 Example for childhood malnutrition
project
PROGRAM IMPACT Reduction in malnutrition rate among
children under five

PROJECT IMPACT Decrease in the incidence of diarrhea

EFFECT Parents of children practice sanitary behaviors
OUTPUT Parents acquire knowledge about sanitary practices
ACTIVITIES Parents given training in sanitary practices
INPUTS Funding obtained, staff trained, training center set up
 The quality of each level is
measured by the next higher
level.
IMPACT !
EFFECTS which, if our assumptions hold true,
should be shown to lead to ...

OUTPUTS, which, if our hypothesis is correct, the

effectiveness can be measured by...

ACTIVITIES (interventions) which should lead to ...

If sufficient INPUTS are received , then we will be able to do ...

Comparison of logical framework terms used by different agencies
End Outcomes Intermediate Outcomes Outputs Interventions

Needs-based Problem Cause Solution Process

CARE terminology Impact Effects Outputs Activities Inputs
CARE logframe Final Goal Intermediate Goals Outputs Activities Inputs
PC/LogFrame Goal Purpose Outputs Activities
USAID Results Strategic Objective Intermediate Results Outputs Activities Inputs
Framework
USAID Logframe Final Goal Strategic Goal/ Objective Intermediate results Activities 202E
DANIDA + DFID Goal Purpose Outputs Activities
CIDA Overall goal Project purpose Results/outputs Activities Inputs
GTZ Overall goal Project purpose Results/outputs Activities Inputs
European Union Overall Objectives Project Purpose Results Activities
FAO and UNDP Development Objective Immediate Objectives Outputs Activities Inputs
NORAD Development objectives Intermediate Objectives Outputs Activities Inputs
World Bank Long-term Objectives Short-term Objectives Outputs Inputs
AusAID Scheme Goal Major Development Outputs Activities Inputs
Objectives

This has been called the “Rosetta Stone of LogFrames”

 AN OVERVIEW OF DESIGN
PROCESS (SUMMARIZATION)

OPPORTUNITY
DIAGONISIS
ANALYSIS

ASSESSMENT PROBLEM PROBLEM

ANALYSIS TREE

STRATEGY GOAL CONCEPT PROJECT

FORMULATION FORMULATION PAPER PROPOSAL

THE PROCESS VARIES DUE TO SEVERAL REASONS

 Determining
Appropriate Monitoring
& Evaluation
(Research) Designs
 AN OVERVIEW OF M&E
PROCESS
OVERALL M&E SYSTEM & THE NEED

M&E Plan &

Implementation M&E Terminologies M&E Methodologies

Quantitative Qualitative

Indicators Counting Data Use of

Beneficiary Processing Information
System Review & Learning Application
 The Project Implementation Process

Was our hypothesis Are we doing well?

valid? Project Efficiency

Impact Effects Outputs Activities Inputs

Project Effectiveness

Did we do the right thing in a worthwhile way?

 Project concept
Diagnosis Project Design  Logframe  M&E plan
The problem Project hypothesis: outputs + valid assumptions will
lead to outcomes
Target group

ASSUMPTIONS

Impact Effects INDICATORS

INDICATORS INDICATORS Outputs Activities Inputs

INDICATORS INDICATORS INDICATORS

End
(Evaluations) Means
(Monitoring)

A good M&E plan will show all of these dimensions and how they relate
 We need to recognize which evaluative
process is most appropriate for
measurement at various levels
• Impact PROGRAM EVALUATION

• Effect PROJECT EVALUATION

• Output PERFORMANCE
• Activities MONITORING

• Inputs
 An introduction to various evaluation designs
Illustrating the need for quasi-experimental
longitudinal time series evaluation design
Project participants

Comparison group

baseline end of project post project

evaluation evaluation
scale of major impact indicator
 rs
to
DESIGNING A SURVEY IS LIKE

ica
nd
LAYING A PIPELINE

n
i

io

n
of

ty

ct

tio
e
si z
ty

a li

l le

iz a
di

is
qu

co
&
li

il
s
va

ut
aly
n
ir e

a
si g

at

&
y&

an
a

fd
nn

ng
de

of
li t

yo
tio

rti
e
bi

h
pl

t
es

po
l ia

pt
ali
m
Qu

De

Re
Re

Qu
Sa

QUALITY OF INFORMATION GENERATED BY AN EVALUATION

DEPENDS UPON LEVEL OF RIGOR OF ALL COMPONENTS
 Re
l ia
bi
li t
y&
va
Qu li di
es ty
tio of
nn i nd
a ir e ica
Sa to
m qu rs
pl
e de
a li
ty
si g
Qu n
ali &
t yo si z
e
fd
at
De a co
pt
h l le
of ct
an io
Re aly n
po
rti s is
ng
&
ut
il iz a
tio
n
AMOUNT OF “FLOW” (QUALITY) OF INFORMATION IS LIMITED
TO THE SMALLEST COMPONENT OF THE SURVEY “PIPELINE”
 Determining appropriate levels of precision for
events in a project’s evaluation plan
High rigor

Same level of precision

Baseline Final
study evaluation

Mid-term
evaluation
Needs Special
assessment Study

Annual
self-evaluation

Low rigor Time during project life cycle

What’s needed is the right combination of
BOTH QUALITATIVE methods
AND QUANTITATIVE methods
 Indicators

A variable, measure or criterion

used to assist in determining to
what degree change has been
taken place
Indicator Selection Criteria

• Technically Feasible
• Reliable
• Relevant
• Sensitive
• Cost Effective
• Timely
• Ethical/Acceptable
 Indicator should be

 Relevant to the goals they

represent, qualitatively or
quantitatively measurable,
objectively verifiable, reliable, meet
international professional
standards ...
 and yet be understandable and
appreciated by project participants
and other stakeholders.
 Indicator should be

Clear on appropriate levels of

indicators:
 Indicators that represent different
levels of goals and objectives in the
logframe hierarchy
 Different levels of precision from
broad issues to objectively
verifiable indicators and specific
raw data variables to be collected.
 Look for the depth of evidence (level of rigor) behind decision-making.

The role of managers is to base

their decision-making on the
evidence provided

The role of M&E staff is to provide

relevant and timely information
with the appropriate degree of
rigor (precision)
How solid is the foundation upon which your decisions are made?
 ACCOUNTABILITY
Consider how many Then consider
resources we put how many
into accounting for resources we put
how funds have been into monitoring and
spent. evaluating results
achieved by our
projects.

FUNDS
ACCOUNTING
M&E of
RESULTS
Over-all Principles of good
DME
Holistic diagnosis of needs and
opportunities which includes
community participation
Logical project designs
Systematized monitoring systems
Quality evaluations that measure
impact, are credible and useful
Decision makers “think evaluatively”
 Logical
Framework

Detailed M&E
Plan
Good DME

Systematic
DME-IS
Three Major Components of
In these ways we can assure
that
all of us working together
will help make the world a
better place for all

Nagarkot, Nepal; photo by Richard Caldwell