Selecting a reference standard for evaluating respiratory rate devices 569

to detect symptoms of pneumonia in children under five:

Lessons from resource-poor settings in sub-Saharan Africa and Asia
Charlotte Ward1, Kevin Baker1,2, Sarah Marks1, Karin Källander1,2
1Malaria Consortium, London, UK; 2Karolinska Institutet, Stockholm, Sweden

Key messages Results

• We recommend that video panel review remains as the best practice • A total 120 of 146 videos (82%) were readable, of Continuous monitor versus video panel rates
reference standard for evaluating respiratory rate (RR) devices. which 97 (81%) had video expert panel agreement 30

RR difference video panel vs. continuous

• Further studies should aim to reduce human counting error in the video between any two of three panel members. Of 97 20 20.3 Mean differerence
10 (95% CI)
panel, for example by assisting counters with video annotation software. video panel rates, corresponding comparator 0 Upper LOA
information could be analysed for 90 (93%). For the -10
-3.6

continuous monitor, 20 comparisons were excluded

monitor
-20
Lower LOA
Introduction due to connection failures. For the EC, nine -30 -27.5
-40
• Manually counting a child’s RR for 60 seconds using an acute respiratory comparisons were excluded due to failure to get a -50 Continuous monitor
difference ±2 bpm
infection (ARI) timer is the WHO-recommended method for diagnosing reading. -60
-70 Continuous monitor
symptoms of pneumonia in resource-poor settings. • Agreement ±2 bpm between references is lowest 0 20 40 60 80 100 difference >2 bpm

• Evaluating new respiratory rate diagnostic aids is challenging due to the for the continuous monitor versus EC (29%). Average RR (bpm)

absence of a gold standard. Agreement ±2 bpm with the video panel is higher at Expert clinician versus video panel rates
~40% for both comparators. Agreement ±5 bpm is 30
Mean
• The study objective was to compare RR agreement between different similar between all references (55-59%) (Table 1).

RR difference video panel vs. EC

differerence
20
reference standards. 17.0 (95% CI)
Upper LOA
Table 1: Agreement between reference standards 10

Methods Continuous monitor EC versus Continuous 0 -0.1

Lower LOA
Level of
• Four RR devices were tested by community health workers (CHWs) on agreement
versus video panel video panel monitor versus EC -10 EC difference ±2
children 0-59 months across four countries in sub-Saharan Africa and (n=77) (n=88) (n=75) bpm
-17.2
-20
Southeast Asia. N; Freq. (%) N; Freq. (%) N; Freq. (%) EC difference >2
bpm
-30
• Three reference standards were used: 1) simultaneous video recording of ± 2 bpm 31 (40.3%) 34 (38.6%) 22 (29.3%)
0 20 40 60 80 100
child’s chest movements with independent RR assessment by a ± 5 bpm 45 (58.4%) 48 (54.6%) 42 (56.0%) Average RR (bpm)
three-person expert panel; 2) contemporaneous expert clinician (EC) Continuous monitor versus expert clinician rates
30
counting RR with a manual stopwatch; and 3) simultaneous continuous • Agreement between reference standards tends to

RR difference EC vs. continuous monitor

Mean
RR with a Masimo capnography CO2 module on a patient monitor. decrease as average RR between reference standards 20 20.5 differerence (95%
increases (Figures 1, 2 and 3). On average, CI)
• Agreement was calculated using the proportion of observations that were continuous monitor RRs were lower than the video 10
Upper LOA

±2 and ±5 breaths per minute (bpm). Bland-Altman plots with limits of panel rate and EC rate (mean difference -3.6 bpm 0 Lower LOA
agreement (LOA) were used for analysis of agreement between methods. and -3.1 bpm, respectively) (Figures 1 and 3). -3.1
-10
Continuous
monitor
Conclusion -20 difference ±2 bpm
Continous
-26.6
monitordifference
Agreement between all reference standards was low. -30
>2 bpm
0 20 40 60 80 100
Evaluating the accuracy of selected RR devices in Continuous monitor RR readings are consistently Average RR (bpm)
the hands of CHWs in Mpigi Hospital, Uganda, lower in comparison to EC and video panel RR Figures 1 (top), 2 (middle) and 3 (bottom): Bland-Altman plots
using three reference standards. readings. of the absolute difference (with limits of agreement) between
Photo: Tine Frank reference standards

For more information Acknowledgements

Charlotte Ward, c.ward@malariaconsortium.org This study has been funded by Bill & Melinda Gates Foundation. However, the views expressed do not necessarily reflect the organisations official
Malaria Consortium, Development House, 56-64 Leonard Street, London, EC2A 4LT policies. The authors thank all organisations and individuals who participated in and supported this project. We are particularly thankful to the Malaria
www.malariaconsortium.org Consortium country teams in South Sudan, Ethiopia, Uganda and Cambodia for their support in data collection.