Monitoring and Evaluation:

Indicators
M&E Indicators:
Module Objectives

At the end of the session, participants will be able

to:
• Critique indicators
• Identify criteria for selection of sound indicators
• Understand how indicators are linked to the
frameworks covered in the Frameworks Module
• Select indicators and complete an Indicator
Reference Sheet
An Indicator is…
• a variable
• that measures
• one aspect of a program/project or health
outcome

An appropriate set of indicators includes at least one

indicator for each significant aspect of the program or
project
(i.e. at least one per box in an M&E framework)
Characteristics of Good Indicators

• Valid: accurate measure of a behavior, practice or task

• Reliable: consistently measurable in the same way by
different observers
• Precise: operationally defined in clear terms
• Measurable: quantifiable using available tools
and methods
• Timely: provides a measurement at time intervals
relevant and appropriate in terms of program goals and
activities
• Programmatically important: linked to a public health
impact or to achieving the objectives that are needed
for impact
Characteristics of
Good Indicators: Valid
Accurate measure of a behavior, practice or task

• Indicator measures what it is supposed to

measure
– Direct measures
– Proxy measures
– Straightforward interpretation: change in
value signals a change in focal concept or
behavior
Validity: Class Activity
• Is ideal family size a valid indicator of fertility
demand?
• Is the maternal mortality ratio a valid indicator of
the impact of a family planning program on
women’s health?
• Is “% Willing to buy vegetables from HIV+
shopkeeper” a valid measure of AIDS stigma?
• Is “More than one non-marital partner in past year”
a valid measure of “risky sex”?
Characteristics of
Good Indicators: Reliable
Consistently measurable in
the same way by different observers

• Types of measurement error

– Sampling Error: over-representation of urban
populations because access is easier
– Non-Sampling Error: survey estimates of abortion
incidence, due to response bias
– Subjective Measurement: indicators that ask for
personal judgment such as “quality,” “environment”
and “progress”
Characteristics of
Good Indicators: Precise
Operationally defined in clear terms

Activity: Develop definitions for

• New user
• Knowledge of AIDS
• Quality of care
• Trained provider
Characteristics of
Good Indicators: Measurable
Quantifiable using available tools and methods

• Examples of measures
– Days of drug stock outs
– Total Fertility Rate
– Indicators related to Poverty Reduction Strategy
Characteristics of
Good Indicators: Timely
Provides a measurement over periods of time of
interest
with data available for all appropriate intervals

• Timeliness Considerations
– Reporting schedules
– Recall periods
– Survey schedules
– Length of time over which change can be detected
Characteristics of Good Indicators:
Programmatically Important
Linked to a public health impact or
to achieving the objectives needed for impact

• Example: Condom distribution program

– Indicator: # of sex partners in past 4 weeks
• Example: Program to increase access to oral
rehydration salt for childhood diarrhea through
community based distributors
– Indicator: # of ORS packets distributed in past
month
Factors to Consider When
Selecting Indicators
• Logic/link to framework
• Programmatic needs/information for
decision making
• Resources
• External requirements (government, donor,
headquarters)
• Data availability
• Standardized indicators
Operationalizing Indicators
• To operationalize an indicator is to identify how a
given concept or behavior will be measured:

• Challenges
– Subjective judgment
– Local conditions
– Unclear yardsticks
Common Indicator Metrics
• Counts
– Number of providers trained
– Number of condoms distributed
• Calculations: percentages, rates, ratios
– % of facilities with trained provider
– Maternal mortality ratio, Total fertility rate
• Index, composite measures
– Quality index comprising the sum of scores on
six quality outcome indicators
– DALY (Disability Adjusted Life Years)
• Thresholds
– Presence, absence
– Pre-determined level or standard
Anatomy of an Indicator Metric
Indicator 1: # of ANC care providers trained
“providers” include any clinician providing direct clinical
services to clients seeking ANC at public health facilities
“trained” refers to attending every day of a five-day
training course offered by the program
Indicator 2: % of facilities with a provider trained in ANC
Numerator: # of public facilities with a provider who
attended all 5 days of ANC training offered by the
program
Denominator: total # of public facilities providing ANC
services
Always Specify the Details!
Threshold indicator for post-abortion family planning
• Facility provides post-abortion FP if
– 90% PAC clients received FP counseling
– 90% PAC clients who want no more children were informed
about sterilization services
– Facility has all of 6 relevant IEC materials
Composite Indicators
• How to measure, “Infection Prevention at facilities providing
obstetric care” ?
Could construct an index, have various measurement options:
– “Facility Readiness,” from facility audit
• Has regulation sharps container
• Up to date guidelines on hazardous and medical waste
disposal
• Leak-proof lidded container for medical waste, etc.
– Knowledge, from Provider interview
– Practice, from Provider-client observation
Issues with Composite
Indicators
• How to “scale” the index
– Weighted v. un-weighted components
– Value of each component: yes/no,
present/absent, count of sub-components
(e.g., drugs in stock)
– Zero/one total indicator value? % of
maximum possible score? At least 8 out of
10 positive (=1) scores?
– How to interpret and report?
 Example of Composite Indicator:
Proper Disposal of Medical Waste
Percent of Facilities

100

80 74
69
64 64
60

39 41
37
40
27
23
20 15

0
Infection Regulation Leakproof, lidded Lidded plastic All Items
Prevention or Container for container for bucket for CL Available
Hazardous disposal of Medical Waste solution
Waste Protocol sharps

Government Private
Sources of Indicators:
Using Pre-Defined Indicators
• From past years of the program
• From related or similar programs
• From lists of global or recommended
indicators
– Millennium Development Goals
– HIV/AIDS: UNAIDS/WHO
– Reproductive Health: Compendium of Indicators
for Evaluating Reproductive Health Programs
– Poverty Reduction Strategy Goals (World Bank)
Indicator Pyramid

Decreases

Global Number of
Compare countries
Overview world-wide situation Indicators

Increases
National/Sub-national
Assess effectiveness of response
Reflect goals/objectives of national/sub-national response

District or Facility
Identify progress, problems, and challenges
Indicator Matrix

Indicator Data Frequency Decision

Source Points/Comments

Outputs

% of facilities with a Program Quarterly Disaggregate by district.

provider trained in ANC records
Outcomes

% of pregnant women HMIS and Annual Does not provide

attended at least once during census information about quality
pregnancy by skilled health of care provided.
personnel for reasons
related to the pregnancy
Indicator Reference Sheet
Compile detailed documentation for each
indicator:
• Basic information
• Description
• Plans for data collection
• Plans for data analysis, reporting and review
• Data quality issues
• Performance data table (baseline and targets)
Indicator Strengths & Limitations
All indicators have limitations, even those
commonly used:
• Low birthweight: cannot use in rural areas
where few births are in facilities
• Sexual behavior (e.g. condom use, number of
partners): self reporting bias
• Maternal Mortality Ratio: maternal death is
infrequent event
• Couple Years Protection: information on
program activity but not on number of people
benefiting
Setting Indicator Targets:
Useful Information Sources
• Past trends
• Client expectations
• Donor expectations
• Expert opinion
• Research findings
• What has been accomplished elsewhere
Common Pitfalls in Indicator
Selection
• Indicators not linked to program activities
• Poorly defined indicators
• Indicators that do not currently exist and
cannot realistically be collected
• Process indicators to measure outcomes &
impacts
• Indicators that are not very sensitive to
change
• Too many indicators
Pitfalls with Selecting Indicators
Indicator not linked to program activities
IR: Expanded access to AIDS prevention & treatment services
Activities: train providers in current clinical protocols
Inappropriate Indicator: % of facilities with adequate
conditions to provide care
Better indicators: # of clinicians trained, # of facilities with a
trained provider

The program is not aiming to affect facility conditions,

only provider skills.
Pitfalls with Selecting Indicators
Indicator poorly defined
Activity: radio campaign to dispel myths about AIDS
transmission and prevention
Inappropriate Indicator: % of population with AIDS knowledge
Better indicators: % of population with no incorrect beliefs
about AIDS (defined as: % correctly rejecting the two most
common local misconceptions about AIDS and who know that a
healthy-looking person can transmit AIDS)

AIDS knowledge does not indicate what is desirable:

knowledge that AIDS exists? How it is transmitted?
Pitfalls with Selecting Indicators

Data needed for indicator not available

Inappropriate Indicator: % of days per quarter that service
delivery points have stock-out of drugs
Data issue: Information on stock-outs may not be
collected daily
Better indicators: % of service delivery points that had a stock
out of drugs at some time during the last quarter

If relying on routine data, indicator definition must

depend on how data are collected.
Pitfalls with Selecting Indicators
Indicator does not accurately represent desired
outcome
IR: Expanded access to antiretroviral treatment (ARVs) for
pregnant women to prevent mother to child transmission of HIV
Inappropriate Indicators: % of women on ARVs who are
pregnant; % of people on ARVs who are pregnant women
Better indicator: % of HIV positive pregnant women who
are on ARVs

What does it mean if inappropriate indicators increase?

Decrease? Do they reflect the desired program effect?
Indicator systems -- How much
is enough?
Rule of thumb
• At least one or two indicators per key activity or
result (ideally, from different data sources)
• At least one indicator for every core activity (e.g.,
training, BCC, CSM)
• No more than 8-10 indicators per area of significant
program focus
• Use a mix of data collection strategies/source
Not everything that can be
counted counts, and not
everything that counts can be
counted.
Albert Einstein
References

• Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating
Family Planning Programs, with Adaptations for Reproductive Health, Chapel
Hill, N.C.: The EVALUATION Project.
• Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of Indicators for
Evaluating Reproductive Health Programs, vols. 1 and 2, Chapel Hill, N.C.:
MEASURE Evaluation.
• Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute,
University of North Carolina, Chapel Hill.
• Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute,
University of North Carolina, Chapel Hill.
• UNICEF. 1998. State of the World’s Children.
• USAID/Tanzania Country Strategic Plan, 2005-2014.
• WHO, 1999. The Evolution of Diarrhoeal and Acute Respiratory Disease
Control at WHO: Achievement 1980-1995 Research, Development and
Implementation (WHO/CHS/CAH/99.12).
Group Project
• Form groups
• For your project:
– Identify indicators and define metrics
– If frameworks are not finished, continue working on
frameworks
– For two indicators, complete indicator reference sheet