Module 9 Part A:

Monitoring and Evaluation

*Image courtesy of: World Lung Foundation

 Learning Objectives
• Describe the difference between monitoring and
evaluation

• Explain why we monitor programmes

• Explain how to monitor a TB programme

– What to monitor
– Principles and techniques

• Describe how to conduct a supervisory visit

What is M&E?

*Image courtesy of: World Lung Foundation

Differences between Monitoring and Evaluation
 Why Monitor and Evaluate?

1. Collect accurate information

about the TB Programme

2. Use that information to improve

the TB Programme
Monitoring serves several purposes

 Management
 Reporting
 Accountability
 Advocacy
 Evaluation
 Why Evaluate?
• Episodic assessment of specific indicators
– determine effectiveness or impact of services or
activities
– during a given interval
• Determine whether goals are being met
• Assess impact of a specific service or
intervention
- HIV testing among TB patients
• Advocacy
 Benefit for BNTP
• Determine if staff activities follow BNTP guidelines

• Measure key indicators related to case detection, quality

of diagnosis, and TB treatment

• Identify problems in implementing TB control activities

(e.g., laboratory reporting delays)

• Inform the annual evaluation of TB Programme

• Use the findings to modify programme goals and

strategies (e.g., implementing RHT)
Surveillance Data and Programme
Monitoring within BNTP
National-level: ETR

District-level: District TB Register and ETR

Clinic-level: TB Suspect and TB Case Registers

How should I monitor?
The first step is observation, but talking and
analysis help you understand the cause

Observe Talk Analyse

Actions Attitudes Consistency

Processes Understanding Accuracy
Conditions Morale Effectiveness
Observe
 Take a look around the clinic

 Are there signs of

disorganization?
 Are the patient treatment
cards for all registered
patients available?
 Are the cards in a binder
in numerical order?
Are the meds organized?
 Other Visual Indicators

 Are there IEC

materials?
 Are there masks for
patients and
respirators for staff?
 Are patients being
triaged appropriately?
Is the condom
dispenser full?
Communication with
Clinic Staff
 Find Out More from People
Through communicating we can:
 Test level of knowledge

 Gauge attitude and morale

 Seek guidance on priority areas

 Some Communication Tips

DO’s:


 Listen and be prepared to
learn

 Take notes – it’s your job.
Don’t be embarassed!
 Ask follow-up questions and
explore the issues
 Compare one story against
another! “Triangulate”

DONT’s:
 Don’t worry if the person
knows more than you
 Don’t take anything at face
value – CHECK all verbal
information against the data
 Don’t threaten or intimidate
the people you speak to
 Apply this Approach to
Determine How Register is Used
 Level of knowledge:
 ‘Who should be entered into this register?’
 ‘Explain the process of how the register is used’
 Attitude / Perception
 ‘Whose job is it to enter the data?’
 ‘How important is it that this register is
used properly?’
 ‘Are you comfortable with using it?’
 Seek guidance
 ‘What needs to be done to improve how it is used?’
Analysing the Suspect and
Sputum Dispatch Register
 Indicator: Quality of
Programme Management
• % of new pulmonary TB (NPTB) suspects who have 3
initial sputums collected consecutively

• Formula:

Number of who have

3 initial sputums collected consecutively
Number of NPTB suspects worked up for TB

• Tip: compare the result you get to last time. You can learn
a lot by comparing indicators over time
The TB Register
 How does the data ‘look’?
 Use your experience and common sense
 Are the entries recorded correctly?
– Proper chronological sequence
– Proper identification of class and type
 Are there signs that the register is being
completed in “batches”?
– Multiple entries on the same date with different
treatment start dates (all in the same pen!)
Analysing the District
TB Register
 Recording and reporting
 Calculating Indicators
 How does the data ‘look’?
 Use your experience and common sense
 Are the entries recorded correctly
– Proper chronological sequence
– Proper identification of class and type
 Are there signs that the register is being
completed in “batches”
– Multiple entries on the same date with different
treatment start dates (all in the same pen!)
Example: Review of District TB Register

 Is there a report for each facility in this quarter?

 For cases registered 3 months ago, are there follow-up
sputum examination results (that is, for those collected
at month 2 / end of the intensive phase?)
 For cases registered > 6 months ago, are there more
follow-up sputum exam results, or blanks in these
columns?
 For cases registered 12 months ago, are there
treatment outcomes, or gaps in that area of the
register?
 Are there cases who were registered on the basis of
sputum exam results but never started treatment?
Analysing the District
TB Register via the ETR
 Calculating Indicators
 Recording and reporting
 Calculating TB Rates
Death Rate = deaths / proportion of incident cases
that die
Case Detection Rate = annual new smear-positive
notifications (country) / estimated annual new
smear positive incidence (country)
Other Useful TB Rates Include:
Notification Rate, Cure Rate, Treatment Success
Rate, Default Rate, Treatment Failure Rate, Transfer
Out Rate
 AFB Diagnosis Rate Botswana Tuberculosis Programme
Case Finding Report
The proportion of notified
Report on New and Retreatment Cases of Tuberculosis
cases diagnosed with sputum (WHO)
smear microscopy 15 – GABORONE Quarter 1 of 2004

Pulmonary
TB Cases EP Total %
Formula New cases
Smear +
91
Smear -
15
No Smear
105
Total
211 45 256 97%

SS+ cases Relapses 3 2 3 8 0 8 3%

After default 0 0 0 0 0 0 0%

Registered cases After failure 0 0 1 1 0 1 0%

Total 94 17 109* 220 45 265 100%
% 35% 6% 41% 83% 17% 100%
* of which children aged 0-7: 6
Example TB Cases 0-14 15-24 25-34 35-44 45-54 55-64 65-74 >75 Total %

94 M 9 16 66 44 17 5 2 0 159 60%

265
= 35% All
TB
Cases
F
Total
5
14
17
33
39
105
33
77
9
26
3
8
0
2
0
0
106
265
40%
100%
% 5% 12% 40% 29% 10% 3% 1% 0% 100%
M 1 7 23 15 6 2 0 0 54 59%
All F 0 7 15 13 3 0 0 0 38 41%
Smear +
Total 1 14 38 28 9 2 0 0 92 100%
Cases
% 1% 15% 41% 30% 10% 2% 0% 0% 100%

43%
M 0 0 2 0 0 0 0 0 2 67%
All F 0 1 0 0 0 0 0 0 1 33%
Smear +
Total 0 1 2 0 0 0 0 0 3 100%
Re-treat
% 0% 33% 67% 0% 0% 0% 0% 0% 100%
0 record(s) with missing age
 What can it mean if cases diagnosed through AFB is
under 50%?
% Cases diagnosed with
AFB Microscopy 1. Too many cases being diagnosed
90 through x-ray
80 2. Too many extra-pulmonary cases
70 3. In comparison with previous
60
quarters an increase may imply
50
increasing reliance by clinicians on
40
30
smear microscopy…
20
• Is that good or bad?
10 • What could have caused the
0 change in District A?
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr • Take a look at District C – if you
District A District B District C
were monitoring this province
what would you think if you saw
this trend?
• What about District B?
• Which is probably the best
result?
 Botswana Tuberculosis Programme
Program Default Rate: Treatment Outcome Report
Report on the Outcome of Tuberculosis Treatment

How many patients are defaulting 15 – GABORONE Quarter 1 of 2004

from treatment. In other words, Category Outcome Status No %

Treatment completed
how many people stop taking their Smear negative at completion 28 8%

medications. Smear positive at completion

Smear results not available
0
67
0%
19%
All
Treatment not completed
TB
Died during treatment 12 3%
Formula Cases
Transferred to another unit 26 7%
Defaulted from treatment 20 6%
No of registered cases that default Treatment outcome not evaluated 195 56%
All tuberculosis cases 347 100%
Total no of registered cases Treatment completed
Smear negative at completion 21 18%
Smear positive at completion 0 0%
Smear results not available 16 13%
Example All
Smear +
Treatment not completed
New Died during treatment 3 3%
20 Transferred to another unit 11 9%

347
= 6% Defaulted from treatment
Treatment outcome not evaluated
6
63
5%
53%
All tuberculosis cases 120 100%
Treatment completed
Smear negative at completion 0 0%
Smear positive at completion 0 0%
Less than All
Smear results not available
Treatment not completed
1 25%

Smear +

5%
Died during treatment 0 0%
Re-treatment
Transferred to another unit 0 0%
Defaulted from treatment 0 0%
Treatment outcome not evaluated 3 75%
All tuberculosis cases 4 100%
 Less than

5%
It’s important to understand what
‘default’ means:

“A TB Patient is classified as a ‘default’

when their treatment is interrupted for
2 consecutive months or more.”

This is an easy indicator to get from

the data, but it can be deceptive!

The
Every
What
Something
patient
defaulting
do weis know
quite
is wrong
patient
likely
for in
sure
=sill
the
a risk
sick
if
program
ofwe’re
MDR-TB
and
missing
andcontagious.
inneeds
thethe
community.
totarget?
be fixed.
 What does it mean if the program
default rate is not on target?
1. Is it to do with the patient?
• Is it embarassing for the patient?
• Are patients scared of the
side-effects?
• Does the patient know that you have to KEEP taking
the tablets?
2. Is it to do with the nurse?
• Are nurses doing the DOT?
• Is it dangerous / difficult to get to the patient?
3. Is it a problem of drug supply?
The defaulter rate is an important indicator for measuring quality of
treatment. But what are its limitations? Take a closer look at the
definition of ‘default’.
The Sputum Conversion Rate Botswana Tuberculosis Programme
Sputum Conversion Report
Report on Response to Initial Phase Tuberculosis
Are enough patients converting to smear Treatment

negative at the end of 2 months of 15 – GABORONE Quarter 1 of 2004

At 2 months At 3 months
treatment? Category Sputum Conversion Status
No % No %
Treatment still ongoing
To do this we use a conversion Rate Converted to smear negative
Remaining smear positive
29
9
23%
7%
49
7
40%
6%
All Smear results not available 75 60% 54 44%
Formula Smear + Treatment discontinued
Cases Died during treatment 1 1% 1 1%
SS+ converting to smear negative Transferred to another unit 9 7% 10 8%
Defaulted from treatment 1 1% 3 2%

SS+ registered cases All smear + cases

Treatment still ongoing
124 100% 124 100%

Converted to smear negative 28 23% 46 38%

Remaining smear positive 8 7% 7 6%
All Smear results not available 73 61% 53 44%
Example Smear +
New
Treatment discontinued
Died during treatment 1 1% 1 1%
29
= 23%
Transferred to another unit 9 8% 10 8%
Defaulted from treatment 1 1% 3 3%
124 All smear + new cases 120 100% 120 100%
Treatment still ongoing
Converted to smear negative 1 25% 3 75%
Equal to or Remaining smear positive
Smear results not available
1
2
25%
50%
0
1
0%
25%
All
greater than Smear + Treatment discontinued
Re-treatment Died during treatment 0 0% 0 0%

85% Transferred to another unit

Defaulted from treatment
All smear + new cases
0
0
4
0%
0%
100%
0
0
4
0%
0%
100%
What can it mean if sputum conversion rate is under 85%?

Equal to or 1. The drug treatment regimen

greater than may not be working
85% • Are the correct regimens being
used?
• Could MDR-TB be an issue?
100
90
2. Patients are not taking their
80
70
TB medications
60 • Are the nurses doing their job?
50 • Is Directly Observed Treatment
40 actually happening?
30 • Are patients scared of taking
20 the tablets?
10
0
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr What kind of detective work
would you do to find the
Sputum Conversion (%)
cause?
Proportion of notified cases HIV Status

Registered
(mmddyy)
Reg Result

Date
tested for HIV: Number Date

What % of TB cases were tested for Test 1 Test 2

HIV? N
001/05 01/01/05

P
Formula 002/05 01/02/05

No of registered cases with test taken N

003/05 01/03/05
Total no of registered cases
N
004/05 01/04/05
Example
N
6 005/05 01/05/05
7
= 84%
N
006/05 01/06/05

007/05 01/07/05

100%
7 6
What can it mean if less than 100% of patients are tested for HIV?

1. Lack of training among

100%
Low testing rates may indicate that the
policy of routine HIV testing has not been
implemented properly in a site or district.
clinical staff
2. Shortage of test kits
3. High refusal rates among
patients
4. Other factors requiring
consultation with the TBFP or
relevant clinical staff
Analysing the treatment card
 Reviewing the Treatment Card

• Patient’s information is complete

• Patient’s DOT and weights correctly recorded
• All HIV Status results are correctly recorded
• For HIV-infected patients
– Receipt of ART noted?
– Receipt of IPT?
 By now (hopefully)…

We have:
 An understanding of the principles of M&E
 Some experience and guidelines you can take
away with you on good ways to:
 Observe
 Communicate
 Analyse
 Next steps

In the next session we will apply these practices to

the supervision checklist, which means we have to:
 Use the whole supervision checklist
 Practice communicating with actual field staff
 Interpret actual data to develop indicators
 Observe the environment
 Develop a list of action-items
Thank you
and
Good Luck!