Programme

Monitoring and Evaluation

in Health and Clinical
Settings
Divisi Manajemen Kedokteran
Dept. Ilmu Ked. Komunitas
Fak. Kedokteran UI
 What is Evaluation? (AEA)
l Everyone has basically done some forms of
evaluation, either formally or not
- What should I wear today? Should I buy that?
Which one should I prioritise?
l Evaluation: a systematic process to determine
merit, worth, value or significance
- Programme evaluation: Has the programme
achieved its goals? Can it be improved? Should it
continue? Are the results worth the cost?
 Programme Evaluation
l Formative and summative evaluation (Scriven,
1991)
- Formative: Support improvement process
- Summative: Valuative conclusions besides development
l “When the cook tastes the soup, that’s formative
evaluation. When the guest tastes it, that’s
summative evaluation. (Stake)
l This module focuses on formative evaluation
 So, What is Monitoring?
l Systematic and routine collection of information
from projects and programmes
l Tracks progress against set plans and check
compliance to established standards
l Monitoring data can be used as part of an
evaluation
What is being monitored/evaluated?
https://www.csiro.au/~/media/About/Images/Impact-images/Framework.jpg
Why do we monitor/evaluate?
• Work conducted in
compliance to plans
Accountability • Transparent to
beneficiaries
• Are we doing things
well?
Learning • Are we doing the
right things
Evaluation Criteria (OECD)
• Is there (still) a need?
Relevance
• Were resources used
Efficiency wisely?
• Did we deliver as planned?
Effectiveness
• Did it change the lives of
Impact target group?
• What remains after
Sustainability leaving?
 There are a lot of approaches
l Theory driven evaluation, such as Theory of
Change, Realist evaluation, etc
l Analytical thinking, using Problem Solving Cycle
- A thinking process in which an individual scrutinise
and break down facts and thoughts
l This module will focus on using Problem
Solving Cycle
Realist Evaluation
 Problem Solving Cycle
Analisis
Situasi Identifikasi
Evaluasi Masalah
Prioritas
Masalah
Pelaksanaan
Problem
Solving Penyebab
Cycle Masalah
Prioritas
Alternatif
Pemecahan Prioritas
Masalah Penyebab
Alternatif Masalah
Pemecahan
Masalah
 Steps of Programme Evaluation
l Identify and define the problem
- Identify the output indicators and look for discrepancies
between targets and achievements
l Prioritise the problem
l There are various prioritising method
- Decision matrix (ex. I × T × R)
- Hanlon technique
- Multivoting
- Nominal Group Technique (NGT)
- Pareto chart
 I×T×R matrix for prioritising
problem
l Dept. CM FMUI has routinely conducted
programme evaluation using two specific form
of decision matrix
l I×T×R: Importance, Technical Feasibility, and
Resources Availability
- I = P+S+RI+DU+SB+PB+PC
 I×T×R matrix for prioritising
problem (2)
l All of them are given 1-5 score
- Prevalence (how big is the problem)
- Severity (how bad is the impact)
- Rate of Increase (how fast it grows)
- Degree of Unmet Need (how much it affect the
community)
- Social Benefit
- Public Concern
- Political Climate
 I×T×R matrix for prioritising
problem (3)
No. Problem Importance T R I×T×R
list
P S RI DU SB PB PC Sum
Prioritising Tool: Hanlon Method for
Prioritizing Health Problems
l Advantageous when the desired outcome is an
objective list based on baseline data and
numerical values
l Rate against specific criteria:
Prioritising Tool: Hanlon Method (2)
Prioritising Tool: Hanlon Method (3)
l Apply ‘PEARL’ test:
- Propriety – Is a programme suitable?
- Economics – Does it make economic sense? Economic
consequences if a problem is not carried out?
- Acceptability – Will community accept? Is it wanted?
- Resources – Funding available or potentially available?
- Legality – Current laws allow implementing?
l Eliminate any health problems with ‘No’ answer
Prioritising Tool: Hanlon Method (4)
l Calculate Priority Score:
l D = [A + 2xB] x C
l D = Priority Score
l A = Size of health problem ranking
l B = Seriousness of health problem ranking
l C = Effectiveness of intervention ranking
l Rank the health problems based on the Priority
Score
 Prioritising Tool: Multivoting
l Display the list
l Decide how many items on the final list, and
choices each member will vote for (usually five,
but longer list may need 10)
l Each member selects then ranks the choices in
order of priority
- Write each one on a separate paper, with the
ranking
- It is possible not to introduce rank, all votes are
equal
 Prioritising Tool: Multivoting (2)
l Tally votes. Look at dramatic voting differences,
such as item received both 5 and 1
- No pressure to change vote
l Repeat the voting process with reduced number
of choice (usually half of the remaining list)
Prioritising Tool: Multivoting (3)
You rank your votes by assigning score five to one
http://asq.org/img/learn-about-quality/edmt_multivoting.jpg
Assigning no particular rank/order to the votes
http://www.cqeacademy.com/wp-content/uploads/2015/03/Multivoting.png
 Prioritising Tool: NGT
l NGT is actually a brainstorming tool, altough it
can be combined with other tool, such as
multivoting for prioritising method
l Useful in the early phases of prioritization when
there exists a need to generate a lot of ideas in
a short amount of time and when input from
multiple individuals must be taken into
consideration
 Prioritising Tool: NGT (2)
l State and clarify the subject until everyone
understands
l Silently thinks of and writes down ideas (5 to 10
minutes) as many as possible
l In turn, state aloud one idea. Facilitator records
it on the flipchart
l No discussion, not even questions for
clarification.
l Ideas do not need to be from the written list
 Prioritising Tool: NGT (3)
l A member may “pass”, but may then add an
idea on a subsequent turn.
l Continue until all pass or for an agreed-upon
length of time.
l Discuss each idea in turn.
- Wording change only when the originator agrees
- Ideas stricken only by unanimous agreement
- Clarify meaning, explain logic, Q&A, or state
agreement or disagreement
l Prioritise the ideas
 Prioritising Tool: Pareto Chart
Approximately 20% of the diagnoses (coronary disease, heart failure, and diabetes)
account for over 80% of the total number of hospital admissions and ED visits.
This exemplifies the 80/20 rule and provides direction regarding which diseases should be
considered for disease management strategies.
Steps of Programme Evaluation (2)
l Creating conceptual framework to link the
prioritised problem with the possible cause
- Fish bone diagram
- Problem tree diagram
Cause-Effect Tool: Problem Tree
 Cause-Effect Tool: Fishbone
diagram
METHOD MATERIAL MAN
PROBLEM
PLANNING ORGANIZING ACTION/CONTROLLING
Steps of Programme Evaluation (3)
l Identify and define the root cause of the problem
- Identify the input and process indicators and look for
discrepancies between targets and achievements
l Prioritise the root cause of the problem, if needed
- I×T×R
- Nominal Group Technique (NGT), Multivoting, Decision
Matrix, Pareto Chart
Steps of Programme Evaluation (4)
l Make the problem solving alternatives
l Again, prioritise the alternatives (if needed)!
- (M×I×V)/C
- Nominal Group Technique (NGT), Multivoting,
Decision Matrix, Pareto Chart
l Conclude the evaluation
l Throughout all the evaluation process, we will
need data, data, data. The source can be
primary, secondary, or tertiary
 (M×I×V)/C for prioritising problem
solution
l Effectivity
- Magnitude (how big the solution can influence
the problem)
- Importance (how permanent the solution can be)
- Vulnerability (how quick the solution can
influence the problem)
l Efficiency
- Cost (how big the cost for the implementation of
the solution is)
 (M×I×V)/C for prioritising problem
solution (2)
!×#×$
No. Solution Alternatives Effectivity C
% !×#×$
M I V %
 References
AEA Administrator. What is Evaluation? [Internet]. American Evaluation
Association: Blogs. 2014 [cited 2017 Jul 6]. Available from:
http://www.eval.org/p/bl/et/blogid=2&blogaid=4
Chen H-T. A Comprehensive Typology for Program Evaluation. Am J
Eval. 1996 Jun 1;17(2):121–30.
Hitchcock S. Monitoring Versus Evaluation: A brief overview of these
terms and their application for development actors [Internet]. Council for
International Development; 2014. Available from:
http://www.cid.org.nz/assets/Key-issues/Good-Development-
Practice/Factsheet-17-Monitoring-versus-evaluation.pdf
ASQ. Learn About Quality [Internet]. ASQ – Quality Resources. 2004
[cited 2017 Jul 7]. Available from: http://asq.org/learn-about-quality/
Azwar A. Pengantar Administrasi Kesehatan. 3rd ed. Jakarta: Binarupa
Aksara; 1996.
NACCHO. First Things First: Prioritizing Health Problems. NACCHO;
Thank You