Monitoring and Evaluation

Charles Katende PhD.

Director of Monitoring, Evaluation and Research
JHPIEGO (An affiliate of John Hopkins University)
 Session Objective

• To increase participants understanding of

the concepts used in designing M&E
Frameworks and Plans

• To build participants competence in

designing Program M&E Plans
 Expected Results
• At the end of the sessions participants will know
about Program frameworks, M&E frameworks
and the difference between the two frameworks

• Participants will be able to identify and select

appropriate indicators for a program.

• Participants will be able to produce a program

monitoring and evaluation framework.
 Introduction…
• Write the health problem addressed by a major
public health program in your country

• Write at up to three specific objectives of a

public health program that addresses the above
mentioned health problem.

• Write down two indicators the program

mentioned above uses to monitor it progress or
performance towards its objectives.
 What is Program Monitoring, Evaluation?

Monitoring is the Evaluation is the use of

social research
routine process of methods to
data collection and systematically
measurement of investigate a
progress toward achievement of a
program objectives. program’s results
 Key Questions
• What is the purpose of carrying out M&E
• Who needs, uses M&E Information
• Who carries out M&E?
• How is M&E carried out?
• When should M&E be carried out?
 What is the purpose….?
• Improve program implementation
– Data on program progress and implementation
– Improve program management and decision making

• Inform future programming

• Inform stakeholders
– Accountability (donors, beneficiaries)
– Advocacy
 Who needs, uses M&E Information?
• Managers
To Improve program
implementation…
• Donors
To Inform and improve • Governments
future programs • Technocrats

• Donors
Inform stakeholders • Governments
• Communities
• Beneficiaries
 Who conducts M&E….?

Program implementer
Stakeholders
Beneficiary

Remember ..
M&E Technical skills
Participatory process
 How to carry out M&E…?
Key Features
1. Program Framework: Analyze and systematically lay out
program elements

2. Identify key elements to monitor and evaluate.

3. Determine and describe the measures to be used for

monitoring and evaluation

4. Develop M&E Framework and action plans, including

data collection and analysis, reporting and dissemination
of findings.
Program Framework

What do you know about your program….?

 Program Framework

• Systematic lay out of the program

elements and path showing what
the program plans to:

do ……………..achieve!
 Program Framework

• Based on a theoretical, empirical model, or

general understanding

Public health Problem

Population, system level factors that cause the public

health problem

Action/interventions that can change the factors and ultimately

alleviate/eliminate the problem
 Results Framework
Improved Health Status
Impact:

Improved (Sustained) Use of Key Health Services

Strategic Objective: and Practices/Behaviors

Increased Improved social /

Increased quality availability/ access policy
Intermediate Result: of… to… environment…

Strategies: Strategies: Strategies:

Strategies (Sub IR):
 Example: Result Framework for a
Family Planning Project
GOAL: REDUCED FERTILITY
SO: Increased FP use and improved FP/RH practices
Increased knowledge of, Increased quality of FP Increased Improved social and
improved attitudes toward, counseling and services availability/access to policy environment for
  and acceptance of key for FP/RH FP
  services and behavior
 
  Strategies:
 
   Increase availability of Design/ implement Strengthen logistics Advocate for community
  educational materials at supportive supervision management based distribution of
  clinic and community System • Mobilize private sector pills
  level • Train service providers providers
  • Community (in-service and • Mobilize CHWs/CBDs
 
  mobilization (using preservice in FP • Encourage socially Promote addition of
  PRA and PDI) counseling and marketed pills Depo injections to EPI
  including men management of side outreach strategy
  • Implement mass media effects
  strategy • Remodel clinic to allow
 
 
• Mobilize opinion for privacy Pilot social marketing of
  leaders at national and • Design and implement pills
  local level quality improvement
  program
 
 
 
 
Basic Logic Model
VCT Program Implementation Logic Model

INPUTS ACTIVITIES IMMEDIATE INTERMEDIATE IMPACTS

OUTPUTS
OUTCOMES OUTCOMES

Nurses & lab

Train nurses & Quality of VCT
Nurses techs trained
lab techs in VCT increased
Lab techs
Clients are
Provide pre-test Access to VCT
counseled for
counseling increased HIV transmission
Govt. funds HIV testing
Risk behaviors rates decreased
GAP funds
Other donor Conduct HIV test decreased
Clients are Knowledge of
funds tested HIV status
Provide post-test increased
HIV test kits counseling to all Clients receive HIV prevalence
clients tested results and post- decreased
Knowledge
test counseling
about & access
Counseling to prevention HIV treatment
protocol Refer pregnant increased
HIV+ women to Pregnant HIV+ resources
PMTCT svcs women referred increased
Referral to PMTCT svcs HIV morbidity &
system for Access to HIV mortality
prevention & Refer HIV+ HIV+ clients treatment decreased
Tx services clients to Tx referred to ARV, resources
services support & HBC increased

26
 PROJECT DESIGN FRAMEWORK LABELS
Labels Used By Various Organizations

Project Design Level USAID Others

Level A:
Improvement Impact Impact Goal
in Health
Status

Level B:
Use of Strategic Objectives Purpose
Services

Level C: Specific Objectives

Demand for Intermediate Results (IR)
Services
Capacity to
Deliver
Services
Level D: Startegies (Sub-IR) Outputs
Interventions Outputs Activity Clusters
Activities Activities
Inputs
Case 1: To decrease maternal mortality, a 10-year program plan to
improve to train midwifes to Delivery and ANC services at health
facilities, and to train and deploy CHWs to increase the community’s
awareness about, and use of the improved services at the health
facilities.

Case 2: To reduce high fertility, a 5-year program plans to work with

the Government to change policies in order to allow and promote
use of modern family planning methods, train family planning
providers to provide better FP services, and to launch public
campaigns that promote family planning methods.

Case 3: To reduce HIV infection among adolescents, a five-year

program plans to implement income generation activities for the
youth, provide and promote universal secondary education, and
build adolescent-friendly reproductive health service delivery points.
 Exercise
• Identify and state is the Public Health problem
implied in the case study.

• What are population level factors will the

program target to change in order to alleviate
the public health problem

• Prepare a Program Framework for the scase

study
Monitoring and Evaluations
Framework
 M&E Questions

• Monitoring questions
– What is being done?
– By whom?
– Target population?
– When?
– How much?
– How often?
– Additional outputs?
– Resources used? (Staff,
funds, materials, etc.)
 M&E Questions
• Evaluation Questions?

– Is the content of the

intervention or the activity
being delivered as
planned?

– Does the content of the

intervention or the activity
reflect the requisite
standards?

– Have the intervention

achieved the expected
results?
What do we need to answer these questions…?

INDICATORS …to take measurements.

 Indicators: Definition
• Markers that help to measure change by
showing progress towards meeting
objectives

• Observable, measurable, and agreed upon

as valid markers of a less well-defined
concept or objective

• Indicators differ from objectives in that they

address specific criteria that will be used to
judge the success of the project or program.

See comment for examples

Type and Level of Each Indicator
• Type
– Input/Process (Monitoring)
– Outcome / Impact (Evaluation)

• Level
– Global level
– Country level
– Program level
 Exercise: Group work
• Use your case study and identify at least
two indicators for program monitoring and
two indicators for program evaluation.
 What Is a Good Indicator?
• Valid: Measures the effect it is supposed to measure

• Reliable: Gives same result if measured in the same way

• Precise: Is operationally defined so people are clear about what

they are measuring

• Timely: Can be measured at an interval that is appropriate to the

level of change expected

• Comparable: Can be compared across different target groups or

project approaches
 Criteria for Indicator Selection

• Consistent with project design—measure the desired

result

• Useful—contributes to project design, management, and

evaluation

• Available

• Affordable
 Standard Indicators

Where possible, a project should select

standard indicators.
• They have been tested for validity and
reliability.
• They allow comparison between projects
or sites.
• They tend to be available for SOs and
some IRs.
 How Many Indicators?
• Choose at least one or two indicators per intermediate
result, as well as the SO for evaluation purposes.

• Choose one or two indicators per result for program

monitoring.

• Choose indicators that may be able to “cover” more than

one element.

• For management, think about basic activities that you

need to monitor to judge if you are implementing activities
as planned; include indicators that help you make
decisions.
 Exercise: Group work
Refer to the indicators you selected..

• Were the good indicators ?

• Did you select a minimum number

recommended given the type and size of
your program?
M&E Framework
 Sample M&E Framework
Preventing Post Partum Hemorrhage :
Increase Active Management of the Third Stage of Labor
Result Indicator Definition Data Collection Freq- Respons-
source Method uency ible Party

Active Proportion # of AMTSL Clinical Annual Zambia

Management of trained trained observa- observation JHPIEGO
of the Third clinicians midwives tion staff
Stage of performing performin checklist
labor AMTSL to g all steps
increased standard of AMTSL
on all
patients/

See comment for examples

 M&E Plan
• The plan is a managerial tool that specifies the schedule, resources,
responsibilities, for your M&E activities (data collection, data quality
control, reporting, dissemination and use of data)

• Note:
– The plan should specify the time points when evaluations will be carried
out, for example: Midterm, and End term.

– Outcome/Impact evaluation is reserved for large longer term programs

that can make impact at public health status level.

– Your plan should include activities to monitor and evaluate the

implementation, as planned, of the M&E plan.
 Strategic Planning for M&E: Setting Realistic
Expectations

Monitoring and Evaluation Pipeline

Number of All Most Some Few *

Projects
Impact Monitoring/
Outcome
Process Evaluation
Input/Output Monitoring/
Evaluation Evaluation * Supplemented with impact
Monitoring
indicators from surveillance data.

Levels of Monitoring & Evaluation Effort

Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001

Source: CDC. Global AIDS program monitoring

and evaluation (M&E) field guide
 Question
If funding for your case study program was
cut off and the program closed in two years.
What changes would you make to you M&E
Plan?
 Sources of Information
• http://www.cpc.unc.edu/measure
• http://www.unaids.org/DocOrder/OrderFor
m.aspx
• http://www.fhi.org/en/Publications/index.ht
m
THE END