See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/5440015

Evaluation of syndromic surveillance for the early detection of outbreaks

among military personnel in a tropical country

Article in Journal of Public Health · May 2008

DOI: 10.1093/pubmed/fdn026 · Source: PubMed

CITATIONS READS

35 79

8 authors, including:

Henry Jefferson Jean-Paul Boutin

University of Liverpool GISPE - Groupe d'intervention en épidémiologie et santé publique
5 PUBLICATIONS 108 CITATIONS 188 PUBLICATIONS 2,051 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Jean-Paul Boutin on 07 February 2017.

The user has requested enhancement of the downloaded file.

 Journal of Public Health | Vol. 30, No. 4, pp. 375 –383 | doi:10.1093/pubmed/fdn026 | Advance Access Publication 15 April 2008

Evaluation of a syndromic surveillance for the early detection

of outbreaks among military personnel in a tropical country
Henry Jefferson1, Bruce Dupuy2, Hervé Chaudet3, Gaetan Texier4, Andrew Green5,
Guy Barnish1, Jean-Paul Boutin4, Jean-Baptiste Meynard 2
1
Liverpool School of Tropical Medicine, Liverpool, UK
2
Institut Pasteur de la Guyane, 23 avenue Pasteur, BP 6010, 97306 Cayenne Cedex, French Guiana
3
Université de la Méditerranée, Marseille, France
4
Institut de Médecine Tropicale du Service de santé des armées, Marseille, France
5
Defence Medical Services Department, London, UK
Address correspondence to Dr Jean-Baptiste Meynard, E-mail: jbmeynard@pasteur-cayenne.fr

A B S T R AC T

Background To evaluate a new military syndromic surveillance system (2SE FAG) set up in French Guiana.

Methods The evaluation was made using the current framework published by the Centers for Disease Control and Prevention, Atlanta, USA. Two
groups of system stakeholders, for data input and data analysis, were interviewed using semi-structured questionnaires to assess timeliness, data
quality, acceptability, usefulness, stability, portability and flexibility of the system. Validity was assessed by comparing the syndromic system with
the routine traditional weekly surveillance system.

Results Qualitative data showed a degree of poor acceptability among people who have to enter data. Timeliness analysis showed excellent case
processing time, hindered by delays in case reporting. Analysis of stability indicated a high level of technical problems. System flexibility was found
to be high. Quantitative data analysis of validity indicated better agreement between syndromic and traditional surveillance when reporting on
dengue fever cases as opposed to other diseases.

Conclusions The sophisticated technical design of 2SE FAG has resulted in a system which is able to carry out its role as an early warning system.
Efforts must be concentrated on increasing its acceptance and use by people who have to enter data and decreasing the occurrence of the
frequency of technical problems.

Keywords early warning, evaluation, real-time surveillance, syndromic surveillance

Introduction Despite this increase in use, evidence that these systems are
effective in their objective of early outbreak detection is still
Syndromic surveillance systems gather pre-diagnostic
being gathered, mainly through the evaluation of the existing
health-related data with the objective of detecting earlier
systems. There are an increasing number of opinions in the
disease outbreaks and epidemics than would be possible
literature, showing that investment in these systems is not
using traditional surveillance data, e.g. confirmed diagnoses
justified without a better evidence base for their use.7 – 11
and laboratory reports.1,2 Many different types of data may
The Centers for Disease Control and Prevention (CDC)
be collected, either by manual input or by automatic collec-
tion. These may include recording numbers of school or
work absenteeism, over the counter medication sales, phone
calls to hotlines such as National Health Service Direct and Henry Jefferson, MD Student

hospital presenting complaint reporting.3 – 6 Bruce Dupuy, Information Technology Manager

Hervé Chaudet, Associate Professor of Public Health
The use of syndromic surveillance throughout the world
Gaetan Texier, Consultant of Public Health
is increasing. At the start of 2000, very few syndromic sur-
Andrew Green, Consultant of Public Health
veillance systems were in use, yet by the end of 2003 100 Guy Barnish, Senior Fellow in Parasitology
different states and local health authorities in the United Jean-Paul Boutin, Professor of Public Health
States were using some form of syndromic surveillance.7 Jean-Baptiste Meynard, Consultant of Public Health

# The Author 2008, Published by Oxford University Press on behalf of Faculty of Public Health. All rights reserved. 375
376 J O U RN A L O F P U B L I C H E A LTH
have issued several guidelines and revisions on the
evaluation of public health surveillance systems,8,12,13 with
the latest focusing on surveillance for early detection of epi-
demics of a natural or aggressive nature. These guidelines
provide a semi-standardized method of evaluation of syn-
dromic surveillance systems. Other guidelines have also
been published,14,15 but are far less in-depth than those
published by CDC. Although several evaluations have been
published using the CDC guidelines,16 – 19 many evaluations
have been carried out which have not.20 – 25 The results of
these evaluations have varied from finding the systems to be
successful in carrying out their objectives17 – 19 to failing
almost completely.20,25 However, it is clear that further
evaluations of syndromic surveillance systems are required
to understand their place in public health disease surveil-
lance, for early detection of outbreaks but also for other
uses and values.
The ‘Surveillance Spatiale des épidémies au Sein des
Forces Armées en Guyane’ (2SE FAG) system has been in
operation since October 2004. It is a prototype, near real-
time syndromic surveillance system operating among some
3000 armed forces in French Guiana. The system is
designed to allow, in near real time, geo-location and epide-
miological analysis of cases of fever (temperature, 388C)
occurring in members of the armed forces anywhere in this
South America French overseas department,25,26 where tro-
pical diseases responsible for outbreaks exist, such as
dengue fever and malaria. This system has already demon-
strated successful outbreak detection; it detected the dengue
epidemic which occurred in French Guiana in 2006 several
weeks before the traditional civilian surveillance system,
based on the weekly surveillance of biologically confirmed
cases within the 200,000 general population, issued an
alert.27
The aim of this study was to evaluate the syndromic
system 2SE FAG using the CDC guidelines ‘Framework for
Evaluating Public Health Surveillance Systems for Early
Detection of Outbreaks’.8
Methods
This study used both quantitative and qualitative methods.
Qualitative data
Two main groups of stakeholders were identified and inter-
viewed using separate semi-structured questionnaires: the
data input and the data analysis stakeholders. Separate ques-
tionnaires were used due to different areas of expertise of
these two groups of stakeholders. Twenty-one people who
have to enter data were identified, all doctors, nurses and
paramedics. Questions were devised to assess the system
using the CDC themes of timeliness, data quality, acceptabil-
ity, usefulness and stability. Seven data analysis stakeholders
were identified, all epidemiologists working in France and
French Guiana. These stakeholders were interviewed to
assess portability and flexibility as well as data quality,
acceptability, usefulness and stability. Every person inter-
viewed was given an information sheet on the study, and
any questions were answered. They were then asked to sign
a consent form confirming their willingness to participate in
the project. All information was kept strictly anonymous;
this was made clear to all interviewees to avoid any
unnecessary bias in their answering of questions. People
who have to enter data were interviewed in person by visit-
ing the various armed forces regiments based in French
Guiana. Data analysis stakeholders were emailed their
questionnaire.
Quantitative data
The quantitative part of the study was concerned solely with
the assessment of validity. The frequency and timing of
reporting cases of fever by syndromic surveillance was com-
pared with the traditional surveillance system ‘Message
Epidémiologique Hebdomadaire’ (MEH), which has been in
use among the French armed forces for some time.28 Data
concerning all cases of fever, suspected and confirmed cases
of dengue fever and confirmed cases of malaria dating from
January 2005 to December 2006 were taken from both
systems and the performance of syndromic surveillance was
compared with that of the traditional one. A case of sus-
pected dengue fever, for syndromic surveillance, was
defined as a sudden onset of fever with no evidence of
other infection ( particularly malaria, when rapid diagnostic
tests and/or thick blood smears were negative), associated
with one or more non-specific symptoms including head-
ache, myalgia, arthralgia and/or retro-orbital pain. As tra-
ditional surveillance is a weekly reporting system and
syndromic surveillance is a real-time reporting system, data
from syndromic surveillance was converted into a weekly
format to allow comparison. Statistical assessment was
carried out to ascertain agreement between the two systems
when reporting on three different categories—all cases of
fever, dengue fever and malaria. For each category, an initial
paired Student’s t-test and correlation coefficient were calcu-
lated to assess the presence of a relationship between the
two sets of data, respectively. A method of assessing agree-
ment between data sets was then used.29 This method
allows ‘limits of agreement’ to be calculated, giving an
 E VA LUAT I O N O F A R E A L TI M E SU RV E IL L A N C E S Y S T E M 377

indication of how wide discrepancies between the two data Table 1 People who contribute to the syndromic surveillance system 2SE
sets will be in 95% of cases. These limits are calculated as FAG
ratios, if the difference between the ratios is small, the
systems are in good agreement, thus if it is large the systems Data input Data analysis
are not. For instance, if the limits of agreement are between
0.5 and 3, syndromic surveillance reports between 0.5 and 3 Types of people Doctors Epidemiologists
who contribute
times the number of cases of traditional surveillance in 95%
Nurses Public Health Consultants
of cases; this would be a poor result as the limits are very
Paramedics
wide. All data used were supplied by the Institut de
Functions To record the data To collect the data from the
Médecine Tropicale du Service de Santé des Armées data input stakeholders
(Marseilles) and the Direction Interarmées du Service de To send the data to To analyse the data
Santé (Cayenne). Data analysis was carried out using SASw, the data analysis
version 9.1 (SAS Institute Inc., Cary, NC, USA). stakeholders
To send the results to the
commanders
Results To send a feed-back to the
data input stakeholders
Nineteen of the 21 people who have to enter data (11
Organisations French Forces in Institut de Médecine
doctors and 8 nurses) and 6 of the 7 data analysis stake-
French Guiana (FAG) Tropicale du service de santé
holders were interviewed. The remaining stakeholders were des armées (IMTSSA)
unavailable for interview. The average time of involvement Service de santé des Université de la Méditerranée
with the system was 1.5 years for the people who have to armées (SSA)
enter data and 3.0 years for the data analysis stakeholders. Direction Interarmées Institut Pasteur de la Guyane
du Service de santé (IPG)
(DIASS)
System description Subsidiary of Centre National
des Etudes Spatiales (CNES)
System purpose and stakeholders
French Ministry of Defense
The aims of 2SE FAG were to allow the operational study
of a real-time surveillance system using a specific military
prototype, to evaluate the value of such a system before its The analysis network of syndromic surveillance, also
generalization and to develop an interoperable system for known as ‘Communauté de Services Internet pour la
allied cooperation.30 The stakeholders are shown in Table 1. Surveillance Syndromique’ (CS3), is based around four
system modules (Fig. 2). Each system module is made up of
System operation web servers which host web services dedicated to that
The system consists of two networks30: the declaration modules function. Computers hosting these servers may be
network and the analysis network (Fig. 1). Whenever a based anywhere, as the communication backbone is based
member of the armed forces in French Guiana presents on a virtual private network (VPN) which operates via the
with a symptom of fever, the medical staff member attend- Internet.30 Automated alarms are issued from the syndromic
ing is required to record specific information regarding the system based on a current past experience graph (CPEG).31
patient into an available IT system with syndromic surveil- There is no automated alarm for traditional surveillance,
lance software installed. Depending on the location of the which also uses CPEG for its non automated weekly analy-
consultation, this IT system may be a computer in a perma- sis of data.
nent medical unit or a personal digital assistant (PDA) on a
mission. Later, when biological results are available, these
Outbreak detection
are added to the system. Finally, a clinical closure form is
completed at the conclusion of each case, recording both Timeliness
the final diagnosis and the final outcome. The declaration Under ideal circumstances, i.e. prompt reporting and trans-
network is heavily dependent on data input by the medical mission of cases by medical officers, there were 60 min
staff within the armed forces, without whose contribution between the presentation of a case and the detection by the
the system would have nothing to analyse. system post-analysis. However, qualitative results showed
378 J O U RN A L O F P U B L I C H E A LTH

Fig. 1 General organization of the military syndromic surveillance system ‘2SE FAG’ (including a declaration network and analytic network).

Fig. 2 2SE FAG analysis network, indicating the four modules involved in the system (CS3).

that this ideal scenario is often not the case (Table 2). These
results showed that although the system has the ability to
report cases of fever in near real time, this often did not
occur due to delays in reporting by medical officers at the
start of the process, and in several instances there was no
reporting at all.
Data quality
The response of people who have to enter data to questions
regarding completeness of data entry were: very often, 26.4%
(5/19); often, 31.5% (6/19); sometimes, 10.5% (2/19); never,
10.5% (2/19); and non-appreciated 21.1% (4/19). The results
showed that although the majority of people who have to
enter data were completing report forms in full when they
used the system, there were still a large number who were not
utilizing the system in the correct manner by not fully com-
pleting report forms. This adversely affected the quality of
data supplied to the analysis network.
Validity
In both 2005 and 2006, traditional surveillance reported
more cases of fever than syndromic surveillance by 12.6%
 E VA LUAT I O N O F A R E A L TI M E SU RV E IL L A N C E S Y S T E M 379

Table 2 Questionnaire responses by people who enter the data regarding (Fig. 3). This striking difference, especially in 2005, is prob-
case logging and transmission ably due to the fact that traditional surveillance reports con-
firmed cases only and syndromic surveillance reports
Cases are logged/ Case logging into Case transmission to suspected cases. Correlation between the systems was weak
transmitted system (% of IPG server (% of in 2005 (P ¼ 0.9002) and higher in 2006 (P ¼ 0.0001).
respondents) respondents) The t-test showed evidence of relationship in both years
(P ¼ 2.098  1029 in 2005 and P ¼ 0.007 in 2006), indicat-
Within 1 h 7 (36.7) 7 (36.7)
ing consistently higher reporting by syndromic surveillance
At the end of the day 4 (21.1) 2 (10.5)
in both years. The limits of agreement were narrower than
Within 1 week 4 (21.1) 5 (26.4)
Never 4 (21.1) 5 (26.4)
all cases of fever data in both years (Table 3), indicating
better agreement between the systems when reporting on
dengue fever cases.
(196 – 152) and 8.4% (230 – 206), respectively (Table 3). Only malaria data comparison was carried out for 2006.
Significant correlation coefficients were calculated in both Traditional surveillance reported 49% more cases of malaria
years (P ¼ 0.0431 in 2005 and P ¼ 0.0001 in 2006). than syndromic surveillance (99 –34), although correlation
The t-test result showed evidence of relationship in 2005 between the data sets was high (P  0.0001) and peaks in
(P ¼ 0.028) but not in 2006 (P ¼ 0.387). The limits of cases between the systems appear to have coincided (Fig. 4).
agreement were wide in both years (Table 3), indicating However, the limits of agreement were wide (Table 3), indi-
poor agreement between the two systems. cating poor agreement between the systems.
Dengue fever case reporting in 2005 and 2006 was higher
by syndromic surveillance compared with traditional surveil-
System experience
lance by 67% (151 –23) and 29% (162 –89), respectively
System costs
Table 3 Statistical results comparing syndromic surveillance (2SE FAG) and The initial development of the system cost 275 000E,
traditional surveillance (MEH), 2005/2006 which included the purchase of all necessary hardware, soft-
ware development, subscription to communication services
All cases of fever 2005 2006 and salaries of those involved. Annual costs are 235 000E.
MEH 2SE MEH 2SE
FAG FAG System flexibility and portability
Sixty-seven percent of data analysis stakeholders (four of six
No. cases reported 196 152 230 206 stakeholders) were of the opinion that the system would
t-Test P-value 0.028 0.387 cope well with a change in disease conditions in French
Correlation 0.282 0.611 Guiana, and all agreed (six of six stakeholders) that the
coefficient (r) (P ¼ 0.0431) (P  0.0001)
system could be adapted to detect syndromic information
Limits of agreement 20.73 to 5.07 20.74 to 3.59
other than fever. This adaptation has already occurred in
(ratios)
Djibouti, where an updated version of 2SE FAG has been
Dengue fever
No. cases reported 29 151 89 162
used among the French armed forces for the past 6
t-Test P-value 2.098  1029 0.007 months. This system uses an altered interface to allow
Correlation 0.018 0.548 recording of 69 different symptoms and signs, including
coefficient (r) (P ¼ 0.9002) (P  0.0001) those of a non-infectious nature. Although it is too early to
Limits of agreement 20.91 to 1.00 20.88 to 1.85 assess how successful the adaptation to the Djibouti system
(ratios) has been, 83% of data analysis stakeholders (five of six sta-
Malaria keholders) thought that the system could be transferred to
No. cases reported – – 99 34 other areas without significant change in personnel and
t-Test P-value – 2.179  1027 resources.
Correlation – 0.741
coefficient (r) (P  0.0001)
Limits of agreement – 20.36 to 3.26 System usefulness
(ratios) Ninety-six percent of all stakeholders (24 of 25 stakeholders)
agreed that alarms issued by the system-stimulated activities
380 J O U RN A L O F P U B L I C H E A LTH

Fig. 3 Suspected cases (syndromic surveillance—2SE FAG) and cases (traditional surveillance—MEH) of dengue incidence rates, 2005/2006.

Fig. 4 Malaria case counts by traditional surveillance (MEH) and syndromic surveillance (2SE FAG), 2006.

related to prevention and control of disease (strengthening
of individual and collective vector control measures, adap-
tion of health education, etc.) and 84% (21 of 25 stake-
holders) agreed it was better at detecting febrile episodes
than traditional surveillance. All the data analysis
stakeholders (six out of six stakeholders) stated that the
system consistently and adequately detected disease out-
breaks whose primary symptom was fever. Eighty-three
percent of data analysis stakeholders (five of six stake-
holders) said that the introduction of a standardized

response protocol would increase the system’s ability to aid
the control of epidemics detected.
System acceptability
Some lack of acceptability of the system is already evident
from areas such as data quality and timeliness. Eighty
percent of all stakeholders (20 of 25 stakeholders) either
agreed or strongly agreed that the system was an important
tool for the armed forces, 84% (21 of 25 stakeholders)
agreed or strongly agreed that they had received adequate
training in the use of the system and 72% (18 of 25 stake-
holders) agreed or strongly agreed that they felt their contri-
bution was important for the effective functioning of the
system. However, 48% (12 of 25 stakeholders) thought that
the time taken to use the system was not proportional to its
benefit and 24% (6 of 25 stakeholders) believed the system
was not easy to use.
System stability
The reliability of the system was one of the main problems
that people who have to enter data had with the system:
68% of all them (13/19) said that the system was not always
available when needed, and many commented on the pro-
blems concerning case logging and transmission. These pro-
blems were of sufficient severity that PDAs were never and
satellite phones rarely used on missions.
Discussion
The results indicated that although there was a belief that
the system is important among its stakeholders, specific
areas of acceptability, such as the time taken to use the
system and actual use of the system by people who have to
enter data, were areas which could be significantly improved.
The importance of technical issues was also highlighted, as
previous studies had shown problems with the use of PDAs
while on missions.32
The high sensitivity and low specificity of syndromic sur-
veillance noted in the dengue fever results is characteristic of
these types of systems,33 the detection of syndromic infor-
mation generally leads to many false positives. This lack of
specificity may lead to costly false alarms.34 However, the
sensitivity of the system means that if utilized correctly by
its stakeholders, it is unlikely to miss any disease epidemics
where the primary symptom is fever.35 A historical record-
ing of when alarms occur, whether they are true or false
and what action is taken in each case will greatly help the
adjustment of such parameters.
E VA LUAT I O N O F A R E A L TI M E SU RV E IL L A N C E S Y S T E M 381
The timeliness of the system was one of its major strong
points. It had a true ability to report in near real-time any
case of fever occurring within the armed forces in French
Guiana, with the only major limits on the time taken to
process a case being the interval before an individual presents
to a medical officer and how long it takes the officer to log
and transmit the case. One of the main drawbacks of this
study was that it did not completely investigate the benefits of
the near real-time nature of syndromic surveillance, as all data
were converted into a weekly format for comparison with tra-
ditional surveillance. In reality, cases reported by syndromic
surveillance may be entered and detected by the system at any
time, whereas traditional surveillance only collects data weekly.
Delays between the two systems within a week were not
accounted for by this study. Despite this ability, this swiftness
was often not due to delays in data input. Many users
believed that the logging of patients into the system took too
much time, outweighing its potential benefit. It seemed unli-
kely that these stakeholders would take the time to use the
system in busy or emergency situations. This finding may
indicate a need to simplify the data input forms, preventing
too much time being taken on the system, also potentially
increasing acceptability. Also important to timeliness and use-
fulness of the system is the introduction of a standardized
protocol for responding to system alarms, to ensure thorough
and swift investigation of all alarms.
The use of the CDC guidelines to evaluate such military
systems is by no means perfect. Specific elements of the
systems may be missed using the guidelines. Ongoing updat-
ing of the guidelines will be required in the future as new
systems need to be evaluated. The military specificities will
have to be integrated in the guidelines, as proposed by some
authors,36 e.g. variability of geographical conditions for mis-
sions, high mobility of the armed forces in the field, import-
ant turn-over of the military people and increasing
multi-nationality of the external operations.
2SE FAG is a prototype system under development. The
study comprised part of that development. Despite the
system having given an early warning of the dengue epi-
demic of 2006, it has yet to be given the opportunity to
repeat this feat. The technical design of 2SE FAG has
resulted in a system that is sophisticated enough to carry out
its role as an early-warning system based on syndromic sur-
veillance. Efforts must now be concentrated on increasing
its acceptance and use by people who have to enter data as
well as minimizing the occurrence of hindering technical
problems, so that the system is able to fulfil its potential.
The use of syndromic surveillance for early-warning of
epidemics is still a relatively new concept, with many poten-
tial public health applications. The development and
382 J O U RN A L O F P U B L I C H E A LTH
modification of systems such as 2SE FAG is critical to
advance the understanding of what place syndromic surveil-
lance has in a military public health setting and how it may
be most effectively deployed.
Acknowledgements
The authors are grateful to the French Armed Forces in
French Guiana and the Health Directorate of the French
Forces.
References
1 Henning KJ. What is syndromic surveillance? MMWR Morb Mortal
Wkly Rep 2004;53(Suppl):5 – 11.
2 Wagner MM, Tsui FC, Espino JU et al. The emerging science of
very early detection of disease outbreaks. J Public Health Manag Pract
2001;7(6):51 – 9.
3 Bravata DM, McDonald KM, Smith WM et al. Systematic review:
surveillance systems for early detection of bioterrorism-related dis-
eases. Ann Intern Med 2004;140(11):910 – 22.
4 Edge VL, Pollari F, Lim G et al. Syndromic surveillance of gastroin-
testinal illness using pharmacy over-the-counter sales. A retrospec-
tive study of waterborne outbreaks in Saskatchewan and Ontario.
Can J Public Health 2004;95(6):446– 50.
5 Magruder SF, Lewis SH, Najmi A et al. Progress in understanding
and using over-the-counter pharmaceuticals for syndromic surveil-
lance. MMWR Morb Mortal Wkly Rep 2004;53(Suppl):117 – 22.
6 Pavlin JA. Investigation of disease outbreaks detected by “syndromic”
surveillance systems. J Urban Health 2003;80(2 Suppl 1):107–14.
7 Buehler JW, Berkelman RL, Hartley DM et al. Syndromic surveil-
lance and bioterrorism-related epidemics. Emerg Infect Dis
2003;9(10):1197 – 204.
8 Buehler J, Hopkins R, Overhage J et al. Framework for evaluating
public health surveillance systems for early detection of outbreaks:
recommendations from the CDC Working Group. MMWR Morb
Mortal Wkly Rep 2004;53(RR-5):1 – 11.
9 Koo D. Leveraging syndromic surveillance. J Public Health Manag
Pract 2005;11(3):181 – 3.
10 Reingold A. If syndromic surveillance is the answer, what is the
question? Biosecur Bioterror 2003;1(2):77 – 81.
11 Sosin DM, DeThomasis J. Evaluation challenges for syndromic
surveillance – making incremental progress. MMWR Morb Mortal
Wkly Rep 2004;53(Suppl):125 – 9.
12 Centers for Disease Control and Prevention. Guidelines for
evaluating surveillance systems. MMWR Morb Mortal Wkly Rep
1998;37(suppl 5):1 – 18.
13 Centers for Disease Control and Prevention. Updated guidelines for
evaluating public health surveillance systems: recommendations
from the guidelines working group. MMWR Morb Mortal Wkly Rep
2001;50(RR-13):1 – 35.
14 Sosin DM. Draft framework for evaluating syndromic surveillance
systems. J Urban Health 2003;80(2 Suppl 1):8 – 13.
15 Chretien JP, Lewis SL, Burkom HS et al. Evaluating pandemic influ-
enza surveillance and response systems in developing countries:
framework and pilot application. Adv Dis Surveill 2007;2:146.
16 Cooper DL, Verlander NQ, Smith GE et al. Can syndromic surveil-
lance data detect local outbreaks of communicable disease? A
model using a historical cryptosporidiosis outbreak. Epidemiol Infect
2006;134(1):13 – 20.
17 Doroshenko A, Cooper D, Smith G et al. Evaluation of syndromic
surveillance based on National Health Service Direct derived
data—England and Wales. MMWR Morb Mortal Wkly Rep
2005;54(Suppl):117 – 22.
18 Jones NF, Marshall R. Evaluation of an electronic
general-practitioner-based syndromic surveillance system—
Auckland, New Zealand, 2000 – 2001. MMWR Morb Mortal Wkly
Rep 2004;53(Suppl):173 – 8.
19 Lombardo J, Burkom H, Pavlin J, Centers for Disease Control and
Prevention (CDC). ESSENCE II and the framework for evaluating
syndromic surveillance systems. MMWR Morb Mortal Wkly Rep
2004;53(Suppl): 159 – 65.
20 Balter S, Weiss D, Hanson H et al. Three years of emergency department
gastrointestinal syndromic surveillance in New York City: what have we
found? MMWR Morb Mortal Wkly Rep 2005;54(Suppl): 175–80.
21 Gesteland PH, Gardner RM, Tsui FC et al. Automated syndromic
surveillance for the 2002 Winter Olympics. J Am Med Inform Assoc
2003;10(6):547 –54.
22 Hadler JL, Siniscalchi A, Dembek Z. Hospital admissions syndro-
mic surveillance—Connecticut, October 2001 – June 2004. MMWR
Morb Mortal Wkly Rep 2005;54(Suppl):169– 73.
23 Pavlin JA, Murdock P, Elbert E et al. Conducting population behavioral
health surveillance by using automated diagnostic and pharmacy data
systems. MMWR Morb Mortal Wkly Rep 2004;53(Suppl):166–72.
24 Ritzwoller DP, Kleinman K, Palen T et al. Comparison of syndromic
surveillance and a sentinel provider system in detecting an influenza
outbreak—Denver, Colorado, 2003. MMWR Morb Mortal Wkly Rep
2005;54(Suppl):151 – 6.
25 Terry W, Ostrowsky B, Huang A. Should we be worried?
Investigation of signals generated by an electronic syndromic sur-
veillance system—Westchester County, New York. MMWR Morb
Mortal Wkly Rep 2004;53(Suppl):190 – 5.
26 Meynard JB, Chaudet H, Texier G et al. Real time epidemiological
surveillance within the armed forces; concepts, realities and pro-
spects in France. Rev Epidemiol Sante Publiq 2008;56:11– 20.
27 Meynard J, Dussart P, Lamy M et al. Interest of syndromic surveil-
lance within the Armed Forces in French Guiana for early warning.
Adv Dis Surveill 2007;2:111.
28 Spiegel A, Haus R, Berger F et al. La surveillance épidémiologique
des maladies transmissibles dans les armées. Bull Soc Fr Microbiol
2004;19:156 – 162.
29 Bland JM, Altman DG. Statistical methods for assessing agreement
between two methods of clinical measurement. Lancet
1986;1(8476):307 – 10.
30 Chaudet H, Pellegrin L, Meynard JB et al. Web services based syn-
dromic surveillance for early warning within French Forces. Stud
Health Technol Inform 2006;124:666 –71.

31 Stroup D, Wharton M, Kafadar K et al. Evaluation of a method for
detecting aberrations in public health surveillance data. Am J
Epidemiol 1993;137:373– 80.
32 Daudens E, Langevin S, Pellegrin L et al. Assessment of a military
real-time epidemiological surveillance system by its users in French
Guiana. Public Health 2008;122:729 – 32.
33 Centers for Disease Control and Prevention. Annotated bibliography
for syndromic surveillance. http://www.cdc.gov/EPO/DPHSI/
syndromic/index.htm (13 January 2006, date last accessed)
View publication stats
E VA LUAT I O N O F A R E A L TI M E SU RV E IL L A N C E S Y S T E M 383
34 Hope K, Durrheim DN, d’Espaignet ET et al. Syndromic
Surveillance: is it a useful tool for local outbreak detection?
J Epidemiol Community Health 2006;60(5):374– 5.
35 Hopkins R. Design and operation of state and local infectious
disease surveillance systems. J Public Health Manag Pract
2005;11:184– 90.
36 Meynard JB, Chaudet H, Jefferson H et al. Proposal of a framework
for evaluating military surveillance systems for early detection of
outbreaks on duty areas. Adv Dis Surveill 2007;4:257.