You are on page 1of 19

Module 9 Quality Improvement

and Evaluation

Unit 3 Evaluation Methods


Sylvia Sax, Barbara Kloss-Quiroga

Facilitators Manual
Dr. Barbara Kloss-Quiroga (Ed.), Berlin 2003
Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Evaluation
To enable participants
to understand and apply methods for
evaluation and
to improve their personal and technical
skills in evaluation

District Health Management Tools, T 931

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Specific Objectives
At the end of the unit the participants will be able to
identify concepts key to evaluation
list key questions for evaluation
describe the differences and linkages between monitoring
and evaluation activities
identify three types of evaluation methods used within the
health sector for evaluating quality
plan and practice an evaluation at the health district level
using one of these methods
describe the strengths and weaknesses inherent in these
evaluation methods

District Health Management Tools, T 932

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

History and Introduction


Quantitative basis of many evaluations
General nature of evaluation
Judgement and value

District Health Management Tools, T 933

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Evaluation: A Working Definition


Attributing value to an intervention
Gathering reliable and valid information in a
systematic way
by making comparisons (i.e. to standards,
guidelines, pathways)
for the purposes of making more informed
decisions
and/or understanding causal mechanisms or
general principles
and/or assessing if goals have been met

District Health Management Tools, T 934


Source: Adapted from Ovretveit, John: Evaluating health interventions, Open University Press,
Buckingham, 1998

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Evaluation

A process which attempts to determine,


as systematically and objectively as
possible, the relevance, adequacy,
effectiveness, efficiency, and/or health
impact of health services or their
components in the light of predetermined
objectives (standards, criteria or the like)

District Health Management Tools, T 935


Source: Adapted from Schrettenbrunner, A., Harpham, T.: A different approach to evaluating PHC
projects in developing countries, Health Policy and Planning 8/1993, pp. 128-135

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Evaluation

Pre-determined purpose for evaluation


Often one-off process, clear beginning and end
Involves collection and analysis of set of
information that leads to a decision
Usually involves some form of comparison
(i.e. to standards, guidelines, pathways)
For the purposes of making more informed
decisions

District Health Management Tools, T 936


Source: Adapted from Schrettenbrunner, A., Harpham, T.: A different approach to evaluating PHC
projects in developing countries, Health Policy and Planning 8/ 1993, pp. 128-135

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Monitoring
Regular and routine collection
Information to be collected has clear definitions/
parameters
Can be multiple uses for the information
Tends to be quantitative data (numbers, units etc.)
A purpose for the data is often for comparing similar
organisations/units
Assess achievement of activities of the operational plan

District Health Management Tools, T 937

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

The Planning (Management) Cycle


Evaluating

Implementing

Situational Analysis

MONITORING

Priority Setting

uncertain events
Programming

1. Situational Analysis

Planning

2. Priorit y Setting

INPUT

Income/
Vision

Input/
Structure

3. Planning

4. Progr amming

5. Implementing

6. Evaluating

PROCESS

Direction

Supervision

Targets

OUTPUT

Activites

Output/
Results

Outcome/
Impact

MONITORING

Morbidity
Mortality
Image of
Services

Accessibility
Availability

Baseline

Productivity
Use
Utilisation
Quality

Progress control

Coverage
Efficiency
Efficacy

Objectives based
evaluation

Effectiveness
Morbidity
Mortality
Image

Impact
evaluation

District Health Management Tools, T 938


Source: Barbara Kloss-Quiroga

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Questions for Evaluation Design


Why undertake the evaluation?
Who is the evaluation for?
When do we evaluate?
How do we evaluate?
Who evaluates?
What will become of the results?

District Health Management Tools, T 939

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Why Evaluate?
Needs assessment
Situational analysis
Evaluation of objectives
Change management
Quality assurance
Comparison of results and impacts of health programs
Comparison of health care structures
Establishment of new initiatives
Suggest more purposes for evaluation in health.

District Health Management Tools, T 9310

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Who Evaluates?
Peers (including organisations and professionals)
Private professional auditors/consultants
Purchasers/Funders/Insurance
Consumers/Patients
Governments
The organisation or group itself

District Health Management Tools, T 9311

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

How Do We Evaluate?
Multiple Methods
Administrative Cycle is one method
1. identifying the problem
2. clarifying and analysing the problem
3. collecting the necessary information
4. implementing judgment-based changes
5. controlling and advising the changes

District Health Management Tools, T 9312

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Evaluation Framework
Types of Evaluation

Questions Answered by the


Different Types of Evaluation

Formative Evaluation

Is an intervention needed?

(Determines Concept and Design)

Who needs the intervention?


How should intervention be carried out?

Process Evaluation
(Monitors Inputs and Outputs;
Assesses Service Quality)

To what extent are planned activities


actually realised?
How well are the services provided?

Effectiveness Evaluation

What outcomes are observed?

(Assesses Outcome and Impact)

What do the outcomes mean?


Does the program make a difference?

Cost-Effectiveness Analysis
(Including Sustainability Issues)

Should program priorities be changed or


expanded?
To what extent should resources be
reallocated?

District Health Management Tools, T 9313


Source: Rehle, T., Saidel, T., Mills, S., Magnani, R. (editors): Evaluating Programs for HIV/AIDS
prevention and care in developing countries, Family Health International, Washington D.C., 2003

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Who Evaluates?
Individuals
Technical knowledge and skills
Professional and personal qualifications
Evaluation knowledge and skills

Teams
Complimentarity
Team building

District Health Management Tools, T 9314

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Four Phases of Evaluation

District Health Management Tools, T 9315

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Desk Study
An examination of key provider documents relevant to
the standards/criteria being assessed
Conducted by the evaluation team prior to the site visit
Provides an understanding of the organisation,
programme or persons activity
Provides theoretically adequate assessment as a basis
for an on-site visit but incomplete as an assessment tool
in itself
Documents should show evidence that they are
approved, implemented, reviewed, and change
controlled
District Health Management Tools, T 9316

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Other Important Terminology:


Policy
Provides direction and/or guidance for
the actions of those working in the
organisation/system

District Health Management Tools, T 9317

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.

Other Important Terminology:


Procedure
Outlines a process for key activities, often includes the
following headings:
Purpose:

objective/intent of the procedure

Scope:

application/boundary of what the


procedure will accomplish

References:

other documents, people/links


relevant to this procedure

Definition of key terms used in the description of the procedure


Procedure/method:

who, what, where, when, how; the


step-by-step description of the procedure

Associated documents: names of important forms, records etc.


associated with the procedure

District Health Management Tools, T 9318

Please purchase 'e-PDF Converter and Creator' on http://www.e-pdfconverter.com to remove this message.