You are on page 1of 13

Pathogens and Global Health

ISSN: 2047-7724 (Print) 2047-7732 (Online) Journal homepage: https://www.tandfonline.com/loi/ypgh20

Achieving accurate laboratory diagnosis of typhoid


fever: a review and meta-analysis of TUBEX® TF
clinical performance

Reynaldo Bundalian Jr., Madonna Valenzuela & Raphael Enrique Tiongco

To cite this article: Reynaldo Bundalian Jr., Madonna Valenzuela & Raphael Enrique
Tiongco (2019): Achieving accurate laboratory diagnosis of typhoid fever: a review and
meta-analysis of TUBEX® TF clinical performance, Pathogens and Global Health, DOI:
10.1080/20477724.2019.1695081

To link to this article: https://doi.org/10.1080/20477724.2019.1695081

Published online: 28 Nov 2019.

Submit your article to this journal

Article views: 18

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at


https://www.tandfonline.com/action/journalInformation?journalCode=ypgh20
PATHOGENS AND GLOBAL HEALTH
https://doi.org/10.1080/20477724.2019.1695081

Achieving accurate laboratory diagnosis of typhoid fever: a review and meta-


analysis of TUBEX® TF clinical performance
a
Reynaldo Bundalian Jr. , Madonna Valenzuelab and Raphael Enrique Tiongco c

a
Center for Research and Development, Angeles University Foundation, Angeles City, Philippines,; bPublic Health Program, Graduate
School, Angeles University Foundation, Angeles City, Philippines; cDepartment of Medical Technology, College of Allied Medical
Professions, Angeles University Foundation, Angeles City, Philippines

ABSTRACT KEYWORDS
This review discusses currently available serological diagnostic methods for typhoid fever with Serological test; Widal test;
a focus on the clinical utility of TUBEX® TF as an alternative to the Widal or Typhidot test. TUBEX® TF; typhoid fever;
A literature search was conducted in PubMed for related publications written in English. Salmonella Typhi; sensitivity;
A qualitative analysis was done to determine various serological tests used for typhoid fever specificity; meta-analysis
diagnosis with emphasis on TUBEX® TF in comparison to the Widal of Typhidot test. Further,
a meta-analysis was performed to obtain a pooled estimate of diagnostic accuracy (sensitivity
and specificity) using different analysis models. A total of sixteen studies was included in the
qualitative analysis. Further screening of these studies yielded ten studies that were used for
the meta-analysis. The sensitivity/specificity range of different commonly used serological tests
in typhoid patients is between 55-100%/58-100% for TUBEX® TF, 54-67%/54-95% for Typhidot,
and 32-95%/4-98% for the Widal test. As for the pooled meta-analysis estimates, the TUBEX® TF
showed superior results when differentiating individuals with febrile illness of unknown origin
from those with typhoid fever. Overall, the results of this review and meta-analysis suggest that
the TUBEX® TF is more advantageous to use as a serological test for typhoid fever diagnosis due
its accuracy and simplicity. However, further studies are still needed to validate our results.

1. Introduction The definitive diagnosis of typhoid fever is achieved


by isolating the S. Typhi bacteria from different speci-
Typhoid fever, also known as enteric fever, is an infectious
mens such as blood, bone marrow, and other body
disease caused by Salmonella Typhi and characterized by
fluids in the laboratory. In many of the developing
clinical symptoms of high fever, fatigue, abdominal pain,
countries where trained technicians and laboratory facil-
diarrhea, headache and complications of bleeding and
ities are limited, this method remains impractical. Due to
intestinal perforation [1]. It is transmitted through the
this, laboratory diagnosis of typhoid fever in developing
ingestion of contaminated food or drinking water.
or underdeveloped countries is primarily performed
Some factors that cause typhoid fever constitute an
through the detection of serum antibodies by applying
essential health problem in developing countries and
the Widal test (agglutination latex slide test) and in
continue to be a significant public health problem world-
some cases, using bacterial culture. The Widal test is
wide [2–4]. Typhoid fever is treatable with antibiotics.
the standard serological method for diagnosis of
However, without antibiotic therapy, the illness may last
typhoid fever in many countries, due to practical rea-
up to one month, with a high fatality rate. Reported
sons. However, both culture methods and the Widal test
surveillance studies provide critical information for guid-
demonstrate relatively low sensitivity and specificity and
ing public health decisions related to typhoid fever.
are time-consuming [5–7]. Furthermore, culture meth-
Typhoid fever is currently diagnosed based on clin-
ods require good laboratory facilities, which are lacking
ical signs, symptoms, and by applying various labora-
in many developing countries, where testing for typhoid
tory tests. The diagnosis of typhoid fever based on
fever if performed. Many hospitals lack facilities for
clinical signs and symptoms is often not adequate.
blood culture, and approximately 90% of typhoid
Usually, symptoms are nonspecific and may present
patients are treated as outpatients [8,9].
similarities with other acute febrile illnesses. Because
In Asia, disease burden estimates have relied on
of this, it is often impossible to find a specific clinical
clinically diagnosed cases of typhoid fever compiled by
symptom for typhoid fever, particularly during the first
governmental facilities and hospitals, usually with
week of illness. This is an indication of the need for
uncertain denominators [10]. Accurate estimates of
laboratory investigations to confirm the clinical diag-
rates of compilation and death at the population level
nosis of typhoid fever.
are not available. Population-based estimates of blood

CONTACT Raphael Enrique Tiongco tiongco.raphael@auf.edu.ph Department of Medical Technology, College of Allied Medical Professions,
Angeles University Foundation, Angeles City 2009, Philippines
© 2019 Informa UK Limited, trading as Taylor & Francis Group
2 R. BUNDALIAN ET AL.

culture-confirmed typhoid cases are sparse. Though, meta-analysis. The additional inclusion criteria used
recent reports of decreased susceptibility to diagnostic are: (1) studies that contain a case group (individuals
agents might suggest a reemergence of untreatable with known typhoid fever) vs. control group compar-
typhoid fever and an increasing global burden [11,12]. ison (either with febrile illness of known origin aside
The primary objective of this review and meta-analysis from S. Typhi, febrile illness of unknown origin, culture
is to define the present outlook in terms of the laboratory negative, or healthy individuals); and (2) studies that
confirmation of typhoid fever using serological tests, spe- contain the total number of positive and negative
cifically the clinical performance of TUBEX® TF. Rapid cases in both the case and control group.
diagnostic tests with acceptable levels of sensitivity, spe-
cificity, and accuracy for the definitive diagnosis of
typhoid cases are essential for better estimates of disease 2.5. Diagnostic test accuracy meta-analysis
burden and prompt treatment of infection. Estimation of the overall diagnostic test accuracy was
carried out using Meta-DiSc ver. 1.4. Pooled estimates
2. Materials and methods of sensitivity, specificity, and area under the curve
2.1. Search strategy (AUC) were estimated, and a summary receiving opera-
tion characteristic (SROC) curve were drafted. Meta-
A literature search in PubMed was carried out using analysis was performed only for categories where
a combination of the following key search terms: enough data was available to produce average sensi-
‘TUBEX’ and ‘typhoid fever,’ for related studies written tivity and specificity estimates. Three analysis models,
in English. The titles and abstracts of the resulting namely: (1) Model 1: healthy controls vs. typhoid fever;
publications were screened to filter relevant studies. (2) Model 2: febrile illness of known origin aside from
Full texts were then evaluated carefully, and references S. Typhi vs. typhoid fever; and (3) Model 3: febrile
cited were manually screened for additional eligible illness of unknown origin + culture negative controls
articles. vs. typhoid fever, was performed.

2.2 Study selection for the qualitative synthesis 3. Results and disucssion
The following inclusion criteria were used for the initial 3.1. Characteristics of the included studies
qualitative synthesis: (1) studies that used TUBEX-TF in
determining typhoid fever (or cultures positive for S. Summary of the literature search is presented in Figure 1,
Typhi); (2) studies that contain a control group either in whereas the characteristics of the included studies are
the form of febrile illness with known origin (any infec- presented in Table 1. Overall, a total of 16 studies were
tious disease aside from S. Typhi), febrile illness of included in the qualitative analysis, and 12 in the meta-
unknown origin (maybe infectious or noninfectious in analysis. Year of publication of articles ranged from 1998
nature), culture negative controls, and healthy con- to 2016. Of the total number of publications retrieved, ten
trols; and (3) studies that used other serological tests were conducted in Asian countries [13–22], five in the
such as the Widal test or Typhidot test as a comparison African region [23–27] and the remaining study in Papua
to TUBEX® TF. All studies were investigated indepen- New Guinea [28]. The reported findings focused mainly
dently for eligibility by two of the authors (R Bundalian on sensitivity, specificity, positive predictive value (PPV),
and RE Tiongco). and negative predictive value (NPV) of TUBEX® TF, Widal,
and Typhidot with their clinical outcome.

2.3. Data extraction for the qualitative synthesis


3.2. Results of the qualitative analysis
Two authors (R Bundalian and RE Tiongco) indepen-
dently extracted the data and reached an agreement Different studies have been made to evaluate the clin-
on all the items. For each study included, the following ical utility of serological tests for typhoid fever diag-
data were obtained: (1) first author’s last name; (2) year nosis. In Table 1, a summary of the major findings in
of publication; (3) country where the study was con- different evaluation studies of TUBEX® TF in Asian
ducted; (4) sample size; (5) method of typhoid fever countries are highlighted. Several studies were also
testing; (6) controls used; and (7) sensitivity and speci- made in the African region that evaluated the useful-
ficity of TUBEX-TF and Widal of Typhidot test. ness of TUBEX® TF in diagnosing typhoid fever. Details
of the clinical performance of TUBEX® TF are further
described in Table 1 as well. The sensitivity/specificity
2.4. Study selection and data extraction for the
range of different commonly used serological tests in
meta-analysis of diagnostic test accuracy
typhoid patients is between 55–100%/58–100% for
Studies included in the qualitative synthesis were TUBEX® TF, 54–67%/54–95% for Typhidot, and
further screened to obtain articles to be used for the 32–95%/4–98% for the Widal test.
PATHOGENS AND GLOBAL HEALTH 3

Key search terms:


“TUBEX” and “typhoid fever”

Records identified through Additional records identified


PubMed searching through other sources
(n = 29) (n = 1)

Records identified after review of titles and abstracts as well as


removal of duplicates (n = 24)

Abstracts screened Records excluded (n = 2)


(n = 24) Due to incomplete or irrelevant
data

Full-text articles Full-text articles excluded (n = 6)


assessed for eligibility Deemed irrelevant to our study
(n = 22)

Studies included in the


qualitative analysis
(n = 16)

Studies included in the


quantitative analysis
(n = 10)

Figure 1. Summary of the literature search.

3.2.1. Laboratory diagnosis of typhoid fever Different forms of serological procedures have been
There are at present several strategies to diagnose for used in diagnosing for typhoid fever. These methods
typhoid fever in the laboratory. Approaches can be are based on the detection of specific IgM and IgG
categorized into (1) direct methods and (2) indirect antibodies against either the somatic O-antigen or
methods. Of these, isolation of S. Typhi from clinical flagellar H-antigen of S. Typhi. Currently, the Widal
specimen remains as the gold standard in the diagno- test is still used in laboratories for the diagnosis of
sis of typhoid fever. This section describes some of the typhoid fever. This serological test is dependent on
conventional serological diagnostic methods used for the production of IgM and IgG antibodies and agglu-
typhoid fever. tination reaction using patient’s sera. Both the O- and
Most enteric fevers occur in low and middle-income H-antigens used in the Widal test are nonspecific and
countries where culture methods are seldom available, may cross-react with other Salmonella species as well
unaffordable, or inconsistently applied. Furthermore, as species of Plasmodium. This may lead to false posi-
several members of the genus Salmonella may share tive test results for those with non-Salmonella infection
the same antigenic composition. Thus confirmatory and even malaria. It may also lead to false negative
testing by means of biochemical tests such as cases if tested during the early phase of typhoid fever.
Analytical Profile Index (API) is often necessary. However, due to its simplicity and cost-effectivity,
4 R. BUNDALIAN ET AL.

Table 1. Characteristics of the included studies and summary of the sensitivity and specificity of TUBEX® TF compared with Widal
test or Typhidot test.
First Sample Sensitivity Specificity Ref.
Author Year Location Size (%) (%) Control Group No.
Lim 1998 Hong Kong/ Total: 90 TUBEX® TF Total: 65 [14]
Malaysia Bacteriologically 100 100 Healthy individuals;
confirmed: 14 Widal Test Patients with autoimmune diseases; Typhus positive
Clinically 81 43 patients; Septicemic patients
diagnosed:11
House 2001 Vietnam Total: 127 TUBEX® TF Febrile hospitalized cases [16]
Culture 87 76
confirmed: 64 Widal Test
32 98
Olsen 2004 Vietnam Total: 79 TUBEX® TF Laboratory confirmed febrile illness other than typhoid [20]
Culture 78 94
confirmed: 59 Widal Test
64 76
Dutta 2006 India Total: 459 TUBEX® TF Paratyhoid fever cases and malaria cases [15]
Culture 55 81
confirmed: 176 Widal Test
67 85
Khanna 2015 India Total: 200 TUBEX® TF Other febrile illness and healthy controls [19]
Culture 76 98
confirmed: 50 Widal Test
68 98
Rahman 2007 Bangladesh Total: 243 TUBEX® TF Blood culture negative cases [18]
Culture 91 82
confirmed: 34 Widal Test
82 69
Dong 2007 China Total: 33 TUBEX® TF S. Paratyphi [22]
Culture 69 95 cases
confirmed: 13 Typhidot
54 91 Blood culture negative cases [17]
Kawano 2007 Philippines Total: 177 TUBEX® TF
Culture 95 80
confirmed: 75 Typhidot
55 65
Naheed 2008 Bangladesh Total: 867 TUBEX® TF Blood culture negative cases and other bacteremia [21]
Culture 60 58
confirmed: 43 Typhidot
67 54
Islam 2016 Bangladesh Total:127 TUBEX® TF Other febrile illness and healthy controls [13]
Culture 75 89
confirmed: 28 Typhidot
64 80
Ley 2011 Tanzania Total: 139 TUBEX® TF Non S. Typhi infection cases [23]
Culture 79 89
confirmed: 33 Widal Test
Not
performed Not performed
Keddy 2011 South Africa/ Total: 92 TUBEX® TF Non S. Typhi infection cases [24]
Tanzania Culture 73 69
confirmed: 28 Widal Test
95 4
Fadeel 2011 Egypt Total: 388 TUBEX® TF Healthy individuals [25]
Culture 77 85
confirmed: 67 Typhidot
63 95
Neil 2012 Uganda Total: 76 TUBEX® TF Blood or stool culture negative cases [26]
Culture 88 84
confirmed: 27 Widal Test
Not
performed Not performed
Tarupiwa 2015 Zimbabwe Total: 136 TUBEX® TF Unspecified [27]
Culture 100 94
confirmed: 131 Widal Test
Not
performed Not performed
Siba 2012 Papua New Total: 500 TUBEX® TF Malaria-negative healthy controls [28]
Guinea Culture 77 87
confirmed: 22 Widal Test
86 95

many clinical laboratories still use the test even if infection of typhoid fever as an alternative to the old
considered less specific [5]. Widal test and as a complement to blood culture.
Presently, serological methods still suffer from lim- A serologic assay is clinically useful if it is characterized
itations of sensitivity and specificity emphasizing the by high sensitivity and specificity. Advances in the
need for a quick and reliable serologic test for acute development of serological assays for the diagnosis
PATHOGENS AND GLOBAL HEALTH 5

of typhoid fever have recently demonstrated that malaria. This makes it difficult to establish a relevant
a rapid test such as TUBEX® TF assay might be of baseline titer for the population [5].
clinical value for the acute diagnosis of typhoid fever
and surveillance of treatment. Such assay provides
critical clinical information to public health decision- 3.2.1.2. TUBEX® TF (inhibition magnetic binding
makers, regarding clinical management, disease pre- immunoassay). TUBEX® TF (Inhibition Magnetic
vention, and infection control strategies [1]. Binding Immunoassay) is a semi-quantitative in-vitro
Serologic tests that have been applied for the diag- diagnostic rapid test for the detection of acute S. Typhi.
nosis of typhoid fever in clinical practices are TUBEX® TUBEX® TF is based on the detection of S. Typhi IgM anti-
TF (IgM), Typhidot (IgM/IgG), IgM/IgG ELISA, IgM dip- O9 antibodies in patient serum [14,30]. The inhibition of
stick and the Widal test (hemagglutination). Technical binding between an anti-O9 IgM monoclonal antibody
characteristics of the most applied serological tests for conjugated to colored latex particles and S. Typhi lipopo-
the diagnosis of typhoid fever (IgM antibody determi- lysaccharide (LPS) O9 antigen coated to magnetic latex
nation) are presented in Table 2. particles is measured. The latex reagent and the magnetic
beads (blue and brown particles) are mixed in a specially
designed reaction well for 2 minutes together with sam-
3.2.1.1. Widal test. The Widal test is a whole cell
ple or test control, and the magnetic beads are separated
agglutination reaction carried out on serum samples
from the solution on a magnet. The resultant color of the
using commercially available reagents evaluated about
supernatant is recorded and compared with color stan-
the saline reaction (negative control) to demonstrate
dards (color scale). A TUBEX® TF score between 0 (nega-
the presence of agglutinins (antibodies) in the serum
tive), and 10 (positive) is assigned, and a TUBEX® TF score
of infected patients. It can either be a slide agglutina-
of 4–10 is positive for typhoid fever, while TUBEX® TF
tion test (using undiluted serum) and tube agglutina-
scores in the range above 2 and below 4 are inconclusive,
tion methods (serially diluted patient sera). Serum
after which the tests should be repeated.
samples are collected 2–10 days from the appearance
of symptoms. Several limitations of this test are nota-
ble. The result of the Widal test is difficult to interpret 3.2.1.3. Typhidot. Typhidot is a qualitative dot
unless the sensitivity and specificity for the specific enzyme immunoassay that detects either IgM or IgG
laboratory and patient population are known [29]. antibodies against a specific antigen (50 kDa) on the
Also, the sensitivity of the Widal test may require rela- outer membrane protein of serotype S. Typhi. Patient
tively high antibody concentrations (more than 500 serum and control are incubated with the strips fol-
ng). lowed by washing. After further washing, the strips are
Furthermore, the value of the Widal test depends incubated with color reagent, and the intensity of the
upon the standardization and maintenance of the anti- sample dots are compared with dots from the positive
gen to produce consistent results [25]. Applying the control strip. The detection of IgM antibodies reveals
Widal test in typhoid-endemic regions may also be acute typhoid disease in the early phase of infection,
problematical due to patients’ repeated exposure to while the detection of both IgG and IgM suggests
S. Typhi or due to cross-reactivity to S. Paratyphi A or acute typhoid in the middle phase of infection. In

Table 2. Comparison of the commonly used serological test for typhoid fever.
TUBEX ® TF Typhidot ELISA IgM Dipstick Widal
Test principle One-step Dot blot EIA, multi-step Enzymatic detection in Immuno- chromato- One-step
Rapid test, magnetic microtiter plates graphic agglutination
separation test
Result Semi- Qualitative Quantitative Qualitative Semi- quantitative
quantitative titration
Specimen/ Serum, Serum, 4 drops Serum, Blood, serum Serum, 50 µL
volume 45 µL 50–100 µL
Antigen LPS O-9 50 kDa antigen O- and H- antigens LPS antigen O-and H- antigens,
detected antigen Whole organism
Antibody IgM IgM, IgG IgM, IgG IgM IgM, IgG
detected
Hands on time 20 minutes 60 minutes 3 hours 3 hours Overnight,
37°C
Result/ Color scale Color Color Staining Type of
interpretation developed, developed, of test line agglutination
compared to control compared to standard
Current Yes (IgM) Yes (IgM, IgG) Yes (IgM, IgG) Yes (IgM) Yes
infection
Instrumentation Magnetic stand Shaker Shaker, washer, reader No Incubator
Controls Positive and negative Positive and negative Positive and negative Negative control Local standard
control included control included control included line- mandatory
IgM
6 R. BUNDALIAN ET AL.

areas highly endemic for typhoid fever, the detection 3.3. Results of the meta-analysis
of typhoid specific IgG antibodies increases.
Based on the results of the pooled diagnostic accuracy
estimates (Table 3), the sensitivity and specificity of the
3.2.1.4. IgM/IgG ELISA. IgM/IgG ELISA is a solid- TUBEX® TF range between 68–86% and 84–97%,
phase immunochemical assay that measures antibo- respectively. Meta-analysis performed using Model 1
dies against defined epitopes on S. Typhi antigens. showed an average sensitivity of 76% (95% CI: 65–85)
Serum samples or controls are added to wells coated and specificity of 97% (95% CI: 92–99). No AUC were
with purified typhoid LPS antigen (S. Typhi somatic drafted for this association since there are fewer than
O-antigen and flagellar H-antigen). After blocking the three studies included. For Model 2, average sensitivity
solid-phase for unspecific binding, samples are per- and specificity are at 68% (95% CI: 63–73) and 90%
mitted to react with the immobilized solid-phase anti- (95% CI: 87–93), respectively, with an AUC of 0.8752
gen. If present, IgM or IgG specific antibodies bind to (Figure 2). On the other hand, Model 3 showed super-
the antigen. After a specified incubation period, ior results compared to the other two models with an
unbound material is washed away, and the enzyme- average sensitivity and specificity of 86% (95% CI:
conjugate is added in order to bind to the antibody- 80–91) and 84% (95% CI: 80–87), respectively with an
antigen complex. After another washing step, the AUC of 0.9205 (Figure 3).
substrate is added, and the absorbance is quantified
(proportional to the amount of IgM or IgG specific
3.3.1. TUBEX® TF studies from Asian countries
antibodies in the sample). The detection limit for
All studies conducted used the culture from either
ELISA assays applied for typhoid testing is, in general,
blood or bone marrow sample to confirm for typhoid
very high, (0.05 ng, according to data reported by
infection. The confirmed test was defined by the isola-
Fadeel et al. in 2011). However, ELISA assay is time-
tion of S. Typhi from the sample. The majority (7 out
consuming and requires several instrumentations
of 9) of the studies conducted in Asia evaluating the
such as a microplate reader making it challenging to
performance of TUBEX® TF, reported higher sensitivity
be routinely used for typhoid diagnosis.
and specificity compared with either Widal test and
Typhidot. Highest sensitivity (100%) and specificity
3.2.1.5. Typhoid IgM dipstick assay. The typhoid (100%) was reported compared with Widal test using
IgM dipstick assay is an immunochromatographic the first version of TUBEX® TF test that utilizes a one-
method for qualitative determination of IgM antibodies step procedure based upon particle separation, using
to LPS antigens released by the S. Typhi bacteria. The anti-O9 LPS IgM antibody as the detector reagent [31].
patient sample, containing S. Typhi LPS antibodies, will Among the different studies conducted, only the study
react with specific gold-labeled anti-human IgM. The performed in India [15], reported lower sensitivity for
antigen and antibody complex will migrate to the reac- TUBEX® TF compared with the Widal test. A possible
tion zone where the complex reacts with another spe- reason for this observation may be due to the selection
cific antibody forming a sandwich. The antigen-
antibody complexes are captured by anti-IgM antibo-
dies immobilized on the test strip and will be detected Table 3. Summary of the pooled sensitivity and specificity
as a visible line. The excess of gold-labeled antibodies estimates.
Analysis Model TP FP FN TN SN% (95% CI) SP% (95% CI)
will migrate further into a second reaction zone creating
Model 1 (Healthy controls vs. Typhoid fever)
a control line. The dipstick method provides a rapid and Khanna, 2015 38 1 12 99 76 (62, 87) 99 (95, 100)
simple alternative for the diagnosis of typhoid fever. Islam, 2016 21 3 7 17 75 (55, 89) 85 (62, 97)
Overall - - - - 76 (65, 85) 97 (92, 99)
Model 2 (Febrile illness of known origin aside from S. Typhi vs. Typhoid fever)
Dong, 2002 9 1 4 20 69 (39, 91) 95 (76, 100)
3.2.1.6. TPTest. TPTest or Typhoid and Paratyphoid Dutta, 2006 58 14 45 99 56 (46, 66) 88 (80, 93)
test is a recently reported platform for the serodiagno- Islam, 2016 21 1 7 14 75 (55, 89) 93 (68, 100)
Khanna, 2015 38 2 12 48 76 (62, 87) 96 (86, 100)
sis of typhoid fever. This test uses a heparinized blood Ley, 2011 26 12 7 94 79 (61, 91) 89 (81, 94)
sample from which peripheral blood mononuclear Naheed, 2008 26 3 17 21 60 (44, 75) 88 (68, 97)
Olsen, 2004 43 1 12 17 78 (65, 88) 94 (73, 100)
cells (PBMC) are separated through density gradient Overall - - - - 68 (63, 73) 90 (87, 93)
centrifugation. This procedure is then followed by cul- Model 3 (Febrile illness of unknown origin + culture negative controls vs. Typhoid
ture of the PBMCs in RPMI medium at 37°C and 5% fever)
Fadeel, 2011 50 17 17 126 75 (63, 84) 88 (82, 93)
CO2. Culture supernatants are tested for S. Typhi spe- Kawano, 2007 71 20 4 82 95 (87, 99) 80 (71, 88)
cific IgA 48 hours after incubation using ELISA. Rahman, 2007 31 37 3 172 91 (76, 98) 82 (76, 87)
Overall - - - - 86 (80, 91) 84 (80, 87)
Measurements of >10 ELISA units are considered posi-
TN: true positive; FP: false positive; FN: false negative; TN: true negative;
tive for the test based on established cutoff value. SN: sensitivity; SP: specificity; CI: confidence interval
PATHOGENS AND GLOBAL HEALTH 7

Figure 2. SROC curve for the Model 2 comparison.

Figure 3. SROC curve for the Model 3 comparison.

of the control group, S. Paratyphi cases were chosen as were used as controls, higher sensitivity (69%) and
the negative control in the study. Also, the knowledge specificity (95%) was observed using TUBEX® TF when
about the high degree of cross-reactivity between S. compared with the sensitivity (54%) and specificity
Typhi and S. Paratyphi A when using TUBEX® TF was (91%) of Typhidot. Looking at the different studies,
not previously known [32]. In contrary, in another the choice of control group might be critical in evalu-
study conducted in China where S. Paratyphi cases ating the clinical usefulness of different serological
8 R. BUNDALIAN ET AL.

tests for typhoid fever. The sensitivity and specificity of 3.3.2. TUBEX® TF studies from African countries
the test are affected by the choice of control groups in A total of five studies have been performed in the
the studies. In one of the studies conducted, the spe- African region evaluating the clinical usefulness of
cificity for TUBEX® TF increased from 82% to 90% when TUBEX® TF in typhoid fever diagnosis. Similar with
healthy non-febrile subjects or cases with dengue previous studies in Asian countries; typhoid fever was
fever instead of blood culture negative cases were confirmed using blood culture in all studies reported.
used as the control group [18]. The first TUBEX® TF study was performed in Tanzania,
Compared with Typhidot, only the study performed involving children hospitalized with symptoms of feb-
in Bangladesh [21] reported a lower sensitivity (60%) rile illness [23]. In this study, controls were categorized
for TUBEX® TF. However, slightly higher specificity into two groups. The first group includes only non-
(58%) was observed for TUBEX® TF compared with Typhi Salmonella cases, while the second group
Typhidot (54%) in this study. One of the probable includes all blood culture positive non-Salmonella
cause of this observation is due to a large number of cases. No significant difference in the sensitivity (79%)
indeterminate results for both tests. At least, 139 and specificity (89%) using the first group as control
patients could not be properly read against the compared with the sensitivity (79%) and specificity
TUBEX® TF color scale (indeterminate borderline (97%) when the second group of controls was used in
cases), and 186 patients could not be properly evalu- the analysis. Although, a notable increase in the speci-
ated with Typhidot (only IgG positive). The number of ficity (89% to 97%) of TUBEX® TF is observed.
inconclusive cases included in the study group will In another study performed, commercially available
likely have a significant influence on the calculated serologic tests for typhoid fever diagnosis were ana-
sensitivity and specificity for TUBEX® TF and Typhidot. lyzed for clinical efficacy. Efficacy was defined as the
In most studies conducted, the stage of the febrile ability of the test to detect IgM antibodies against S.
illness at which the blood sample was drawn for test- Typhi in patients suspected of having typhoid fever.
ing was not specified. Only the length of febrile epi- A total of 28 patients were confirmed for typhoid fever
sodes was described before the inclusion in the study. using blood culture, while 52 patients with negative
It has been observed that the sensitivity (55% to 57%) blood culture for S. Typhi or other pathogens were
and PPV (72% to 85%) of TUBEX® TF improve when used as control. In this study, both Typhidot and
only cases with more than five days of febrile episodes TUBEX® TF demonstrated lower sensitivity to the Widal
are assessed [15]. This indicates that the sensitivity and test. However, while high sensitivity (95%) for the Widal
specificity of the test are affected by the kinetics of the test was reported, specificity (4%) of Widal was very low.
antibody response in the patient. Also, it is notable that The reported low sensitivity (77%) and specificity (85%)
in some of the studies made, patients who have for TUBEX® TF can be attributed to the selection of
initiated antibiotic therapy were not excluded among control cases that were suspected of being typhoidal
the subjects [20]. Antimicrobial therapy before blood [16,17]. Furthermore, the control group may also have
collection is known to affect the results of the culture. included S. Paratyphi A positive patients, which may
Seven out of the nine studies performed also high- cross-react with S. Typhi and thus be cross-detected
lighted equal or better specificity when TUBEX® TF is using TUBEX® TF [17,20,23] decreasing the specificity.
used compared to either the Widal test or Typhidot in An alternative to the conventional serologic test for
diagnosing typhoid fever. Only the studies in India [15] typhoid, an in-house ELISA assay, measuring IgG, IgM,
and Vietnam [16] reported higher specificity for Widal and IgA antibodies against bacterial culture antigens,
test compared with TUBEX® TF. Lower specificity for was evaluated in Egypt [25]. Efficacy of an ELISA assay
TUBEX® TF in the study made in India [15] is mainly was compared with TUBEX® TF, Widal test, and
a result of the use of S. Paratyphi cases as control, since Typhidot among typhoid fever patients. Higher sensi-
TUBEX® TF may cross-react with S. Paratyphi. In the tivity (77%) for TUBEX® TF was reported compared with
study performed in Vietnam [16], although higher spe- the sensitivity of the Widal test (71%) and Typhidot
cificity (98%) (at 1/400 titer) was observed for Widal (63%). The sensitivity (88%) of the experimental IgM,
test, reduced sensitivity (32%) is reported when com- IgG, IgA ELISA assay was higher compared with all
pared with the sensitivity (87%) of TUBEX® TF. In sev- conventional serological tests.
eral studies, poor clinical outcome for the Widal test is On another case, a laboratory surveillance study and
notable [5,16–18,20]. There is significant relative varia- a community survey were performed in Uganda to
tion in the Widal data [20], mostly due to lack of local determine the origin of typhoid fever outbreak [26].
standardization and difficulties in the interpretation of TUBEX® TF assay was used as a rapid laboratory test for
generated data. When repeated in a local laboratory measuring IgM antibodies against S. Typhi antigens,
(Pasteur Institute) a significant increase in specificity where blood and stool cultures were also performed.
(from 76% to 100 %) with maintained sensitivity [5] can Using TUBEX® TF in 76 of the patients with samples
be observed indicating the necessity to establish collected four days after the onset of illness, sensitivity
a local baseline and cutoff values when using this test. (88%) and specificity (84%) was established. In this
PATHOGENS AND GLOBAL HEALTH 9

study, the lack of microbiological laboratory capacity, specificity toward O9 antigens were detected, and
equipment, supplies as well as trained personnel, led the addition of IgG antibodies did not influence the
to difficulties in isolating S. Typhi consequently delay- detection of IgM antibodies. Thus, the direct inhibition
ing the confirmation of etiology. of IgG antibodies could not be demonstrated, and the
More recently, TUBEX® TF and OnSite Typhoid IgM/ IgG antibodies appear to be non-reactive in the
IgG rapid test were evaluated among patients sus- TUBEX® TF assay. This is clinically important for any
pected of typhoid fever in Zimbabwe [27]. Using cul- form of serologic test in order to easily differentiate
ture as the reference standard, the sensitivity (100%) acute infection from previous exposure. In addition,
and specificity of (94%) of TUBEX® TF was comparable the enhancement effect of IgG on the detection of
with the sensitivity (100%) and specificity (94%) of IgM antibodies assayed with TUBEX® TF was also illu-
OnSite Typhoid IgM/IgG rapid test. Both test also strated [18].
reported similar PPV (63%) and (100%). Another main advantage of TUBEX® TF compared
with other available serologic test is its simplicity.
3.3.3. TUBEX® TF studies from other countries Compared with other serological tests, TUBEX® TF is
PCR-based techniques are an interesting development simple, easy to use, and requires short test time with
in molecular diagnostics as a detection tool for patho- very minimal instrumentation and equipment
gens. Coupled with Salmonella-specific PCR, this required. This allows the potential use of the test in
method could be used instead of conventional sero- decentralized platforms where limited laboratory facil-
typing methods to identify the presence S. enterica ities are available in communities.
serovar accurately. Real-time PCR was applied in In addition to the previous advantages mentioned
a study from Papua New Guinea together with the the semi-quantitative nature of the TUBEX® TF further
serological methods TUBEX® TF, Typhidot, and the adds value to its clinical usefulness. The TUBEX® TF
Widal test on 530 outpatients (fever >2 days) from score read against the color scale, showed some corre-
two different hospitals [33]. Of these patients, 500 lation between the semi-quantitative TUBEX® TF score
were tested negative for malaria, and these subjects and the stage of the disease. Low scores denote either
were included in the study with suspicion of typhoid an early or a very late stage, while high scores generally
fever. Typhoid fever was diagnosed in patients using denote late phase or convalescence. However, more
real-time PCR and blood culture. Some of the often than not, many patients can mount a good
PCR-positive patients were blood culture negative response early in the infection and maintain the same
and further, some blood culture positive subjects response (unchanged TUBEX®TF score) for many weeks
were PCR-negative, demonstrating the limited value thereafter. The TUBEX® TF scores like the Widal titer or
of applying the molecular identification of S. Typhi the ELISA titer, add value (confidence) to the diagnosis
infection. Blood culture and PCR testing were used as of S. Typhi. A high score generally denotes a result
a reference standard in 47 typhoid positive patients. undisputable positive, whereas a low score raises the
TUBEX®TF showed sensitivity (51%) similar to the sen- possibility that the result might be spurious due to
sitivity (51%) of the Widal test. Sensitivity (70%) of technical reasons. In the latter case, repeating the
Typhidot was higher compared with both TUBEX®TF test on the same patient sample or another specimen
and Widal test. On the other hand, the specificity (96%) can help to resolve the dilemma. More often, many
of the Widal test was higher compared to the specifi- patients who yielded a low TUBEX® TF score,
city (88%) of TUBEX®TF and the specificity (80%) of a subsequent specimen obtained a week or so later
Typhidot. It was concluded in the study that either produce a more definitive TUBEX®TF result. In order to
TUBEX® TF or Typhidot was sufficiently sensitive to be improve POC testing for typhoid fever, it has been
used in routine clinical practice [33]. proposed that the use of paired samples might
improve the outcome of the laboratory testing [6].
On the other end, a limitation of the TUBEX® TF test,
3.4. TUBEX® TF: advantages and limitations
which uses a colorimetric reaction, is the potential
Several advantages and limitations of the TUBEX® TF difficulty in interpreting the results accurately, compar-
test are notably based on this review and meta- ing the color of the test solution with a color scale (gray
analysis. One of the significant advantages of TUBEX® zone or borderline cases). Assessing the color change
TF is its selectivity for S. Typhi IgM antibodies. It has in the reaction well strip and comparing the color of
been assumed that TUBEX® TF is a selective assay that the test solution with a color scale (results graded from
measures only IgM antibodies in typhoid patient 0 (negative) to 10 (positive) requires some experience
serum [17,32]. This was later confirmed in a study per- and good lighting conditions. Earlier studies using an
formed where IgM antibodies were separated from IgG old version of TUBEX® TF showed that the gray zone is
antibodies in a typhoid fever patient serum and tested too great when reading the color scale leading to
with TUBEX® TF [32]. Only IgM antibodies with indeterminate results.
10 R. BUNDALIAN ET AL.

Another identified concern when using TUBEX® TF individual readers (i.e. visual interpretation). This,
assay is the interpretation of hemolyzed/icteric sam- therefore, may represent a possible area requiring
ples when reading against the color scale. Hemolyzed/ further improvement.
icteric samples may produce false positive signals due As based on the review of the clinical studies per-
to discoloration. The current manufacturer recommen- formed for TUBEX® TF selection of appropriate control
dation for hemolyzed/icteric samples is to take a new is clearly critical. For the evaluation of statistical para-
sample and re-test with TUBEX® TF. However, this issue meters for serological assays such as TUBEX® TF, the
was already resolved. Recently, a new version of selection of control (negative) cases is critical. Studies
TUBEX® TF assay that contains an extra wash step revealed that this might be one of the factors that
with glycine buffered saline to eliminate serum disco- account for the wide range of variation in sensitivity,
loration of hemolyzed/icteric samples was presented specificity, PPV, and NPV of TUBEX® TF. In a recent
[33]. The introduction of wash buffer, as a complement meta-analysis performed in Belgium, the overall accu-
to TUBEX® TF assay, is a quick and simple modification racy of TUBEX® TF and Typhidot for the diagnosis of
of the TUBEX® TF protocol and enables detection of typhoid fever was evaluated, and extensive range esti-
acute typhoid fever in discolored serum samples. The mates for sensitivity and specificity were noted [34].
wash procedure decreases the gray zone with border- The calculated pooled sensitivity for TUBEX® TF in the
line or inclusive results and allows a more accurate meta-analysis ranged between 56–95% (average sen-
reading of the color scale. This, in turn, increases the sitivity 69%) and the pooled specificity between
sensitivity of TUBEX®TF assay. 72–95% (average specificity 88%). In a recent diagnos-
tic test accuracy review, TUBEX® TF had an average
sensitivity of 78% (95% confidence interval (CI) 71%
3.5. TUBEX® TF: future research perspectives
to 85%) and specificity of 87% (95% CI 82% to 91%).
Serologic laboratory testing of typhoid fever is essen- The overall diagnostic accuracy score was moderate,
tial and can be a beneficial complement to blood with no significant difference compared with Typhidot
culture in order to increase the clinical information. and Test-it Typhoid test [13].
Laboratory tests play an integral role in confronting Selection of control cases is very critical to establish
typhoid fever infection and can help to establish the better estimates of accuracy. In some of the studies
primary diagnosis. Early detection of typhoid fever by conducted for TUBEX® TF, febrile patients with patho-
TUBEX® TF assay can help in the clinical management gens other than Salmonella were used as control.
of the disease and patient surveillance while reducing Furthermore, in many reported studies blood culture,
the diagnostic confusion. This review and meta- negative patient samples are applied as negative con-
analysis substantiate that in comparison with other trol even if there is a possibility, that these patients are
commercially available serologic methods, TUBEX® TF true typhoidal, as blood culture demonstrates low
assay can be a useful laboratory tool for diagnosis of sensitivity compared to serological assays [18]. These
typhoid fever infection especially, within the first three false negative cases might influence the calculation of
to seven days of febrile illnesses. sensitivity and specificity. In addition, the utilization of
TUBEX® TF test procedure has been improved dur- typhoid negative blood culture with the presence of
ing the past few years – new type of latex particles, other pathogens as control generally results in low
a more precise reagent volume, a careful and specified sensitivity and specificity [17,21,24]. Protocol harmoni-
LPS (O9 antigen), a standardized coupling procedure zation may be required in order to come up with the
of LPS-antigen to the magnetic particles, improved comparable interpretation of clinical studies for sero-
indicator reagents (IgM antibody-coupled particles), logical assays.
and more stable color scale, a better defined TUBEX® Another area of concern in the selection of control is
TF control reagent and a specific washing step to the inclusion of S. Paratyphi A patients in the control
reduce the negative influence of hemolyzed samples/ group. In reported studies where a number of S.
icteric samples has been integrated. The improved Paratyphi A patients were included in the control
version of TUBEX® TF test has made the assay more together with cases with another type of febrile ill-
applicable for the clinical diagnosis of typhoid fever. If nesses, the sensitivity and specificity of TUBEX® TF
used in a proper way on a single serum sample of were low [15,22]. S. Typhi and S. Paratyphi A show
a suspected case, TUBEX® TF assay has the ability to a cross-reactivity of about 50% with TUBEX® TF [20]
improve the diagnostic algorithm, contributing signifi- leading to false positive cases that reduce the overall
cantly to the clinical treatment and control of typhoid sensitivity and specificity of the test. Most authors
bacterial infection. However, according to several stu- recommend the use of non-Salmonella cases (other
dies, interpretation of TUBEX® TF results has been diffi- febrile illnesses or those with other bacterial diseases)
cult in many health centers. It has been recommended as control subjects when evaluating sensitivity, specifi-
that the evaluator should be trained in interpreting city, PPV and NPV characteristics of TUBEX® TF which
TUBEX® TF results because of variation among can be consequently made in larger studies.
PATHOGENS AND GLOBAL HEALTH 11

4. Conclusion burden and implications for controls. Bull World


Health Organ. 2008;86(4):260–268.
As compared to other existing serological test for typhoid [11] Crump JA, Mintz ED. global trends in typhoid and
fever detection, the TUBEX® TF is noted to be more paratyphoid fever. Clin Infect Dis. 2010;50(2):241–246.
advantageous due to its selectivity for S. Typhi IgM and [12] Buckle GC, Walker CLF, Black RE. Typhoid fever and
its simplicity. Also, the TUBEX® TF is also easy to use, paratyphoid fever: systematic review to estimate glo-
bal morbidity and mortality for 2010. J Glob Health.
would require less time and instruments to perform. In
2012;2:1.
terms of diagnostic accuracy, the test shows reasonable [13] Islam K, Sayeed MA, Hossen E, et al. Comparison of the
sensitivity and specificity, especially when differentiating performance of the TPTest, tubex, typhidot and Widal
individuals with a febrile illness that are negative for immunodiagnostic assays and blood cultures in
culture versus those with typhoid fever. It also showed detecting patients with typhoid fever in Bangladesh,
including using a Bayesian latent class modeling
a high level of specificity when differentiating healthy
approach. Baker S, editor. PLoS Negl Trop Dis. 2016
individuals from those with typhoid fever. However, Apr 8;10(4):e0004558.
further studies are still needed, especially with the use [14] Lim PL, Tam FCH, Cheong YM, et al. One-step 2-minute
of an appropriate control group, are still needed to verify test to detect typhoid-specific antibodies based on
our claims. particle separation in tubes. J Clin Microbiol. 1998;36
(8):2271–2278.
[15] Dutta S, Sur D, Manna B, et al. Evaluation of
new-generation serologic tests for the diagnosis of
Disclosure statement typhoid fever: data from a community-based surveil-
lance in Calcutta, India. Diagn Microbiol Infect Dis.
No potential conflict of interest was reported by the authors.
2006;56(4):359–365.
[16] House D, Wain J, Ho VA, et al. Serology of typhoid fever
in an area of endemicity and its relevance to diagnosis.
ORCID J Clin Microbiol. 2001;39(3):1002–1007.
[17] Kawano RL, Leano SA, Agdamag DMA. Comparison of
Reynaldo Bundalian Jr. http://orcid.org/0000-0003-2377- serological test kits for diagnosis of typhoid fever in
006X the Philippines. J Clin Microbiol. 2007;45(1):246–247.
Raphael Enrique Tiongco http://orcid.org/0000-0002- [18] Rahman M, Siddique AK, Tam FCH, et al. Rapid detec-
7793-6558 tion of early typhoid fever in endemic community
children by the TUBEX® O9-antibody test. Diagn
Microbiol Infect Dis. 2007;58(3):275–281.
References [19] Khanna A, Khanna M, Gill KS. Comparative evaluation
of tubex TF(inhibition magnetic binding immunoas-
[1] WHO. The diagnosis, treatment and prevention of say) for typhoid fever in endemic area. J Clin Diagn
typhoid fever. Commun Dis Surveill Response Res. 2015;9(11):DC14–7.
Vaccines Biol [internet]. 2003:2–48. Available from: [20] Olsen SJ, Pruckler J, Bibb W, et al. Evaluation of rapid
www.who.int/vaccines-documents/ diagnostic tests for typhoid fever. J Clin Microbiol.
[2] Crump J, Luby S, Mintz E. The global burden of typhoid 2004;42(5):1885–1889.
fever. Bull World Health Organ. 2004;82(5):346–353. [21] Naheed A, Ram PK, Brooks WA, et al. Clinical value of
[3] Crump JA. Typhoid fever and the challenge of nonma- TubexTM and Typhidot® rapid diagnostic tests for
laria febrile illness in Sub-Saharan Africa. Clin Infect typhoid fever in an urban community clinic in
Dis. 2012 Apr 15;54(8):1107–1109. Bangladesh. Diagn Microbiol Infect Dis. 2008;61
[4] Parry CM, Hien TT, Dougan G, et al. Typhoid fever. (4):381–386.
N Engl J Med. 2002;347(22):1770–1782. [22] Dong B, Galindo CM, Shin E, et al. Optimizing typhoid
[5] Wain J, Diep T, Ho V, et al. Quantitation of bacteria in fever case definitions by combining serological tests in
blood of typhoid fever patients and relationship a large population study in Hechi City, China.
between counts and clinical features, transmissibility, Epidemiol Infect. 2007;135(6):1014–1020.
and antibiotic resistance. J Clin Microbiol. 1998;36 [23] Ley B, Thriemer K, Ame SM, et al. Assessment and
(6):1683–1687. comparative analysis of a rapid diagnostic test
[6] Parry CM, Nguyen TTH, Diep TS, et al. Value of a (Tubex®) for the diagnosis of typhoid fever among
single-tube Widal test in diagnosis of typhoid fever in hospitalized children in rural Tanzania. BMC Infect
Vietnam. J Clin Microbiol. 1999;37(9):2882–2886. Dis. 2011;11:147.
[7] Bakr WMK, El Attar LA, Ashour MS, et al. The dilemma [24] Keddy KH, Sooka A, Letsoalo ME, et al. Sensitivity and
of widal test - which brand to use? a study of four specificity of typhoid fever rapid antibody tests for
different widal brands: a cross sectional comparative laboratory diagnosis at two sub-Saharan African sites.
study. Ann Clin Microbiol Antimicrob. 2011;10(1):7. Bull World Health Organ. 2011;89(9):640–647.
[8] Peacock SJ, Newton PN. Public health impact of estab- [25] Fadeel MA, House BL, Wasfy MM, et al. Evaluation of
lishing the cause of bacterial infections in rural Asia. a newly developed ELISA against widal, TUBEX-TF and
Trans R Soc Trop Med Hyg. 2008;102:5–6. Typhidot for typhoid fever surveillance. J Infect Dev
[9] Parry CM, Wijedoru L, Arjyal A, et al. The utility of Ctries. 2011;5(3):169–175.
diagnostic tests for enteric fever in endemic [26] Neil KP, Sodha SV, Lukwago L, et al. A large outbreak of
locations. Expert Rev Anti Infect Ther. 2011;9:711–725. typhoid fever associated with a high rate of intestinal
[10] Ochiai RL, Acosta CJ, Danovaro-Holliday MC, et al. perforation in Kasese district, Uganda, 2008–2009. Clin
A study of typhoid fever in five Asian countries: disease Infect Dis. 2012;54(8):1091–1099.
12 R. BUNDALIAN ET AL.

[27] Tarupiwa A, Tapera S, Mtapuri-Zinyowera S, et al. [31] Cherian T, Sridharan G, Mohandas V, et al. Prevalence
Evaluation of TUBEX-TF and onsite typhoid IgG/IgM of salmonella typhi O and H antibodies in the serum of
combo rapid tests to detect salmonella enterica sero- infants and pre-school children. Indian Pediatr.
var typhi infection during a typhoid outbreak in 1990;27:293–294.
Harare, Zimbabwe. BMC Res Notes. 2015;8:1. [32] Tam FCH, Wang M, Dong B, et al. New rapid test for
[28] Siba V, Horwood PF, Vanuga K, et al. Evaluation of paratyphoid a fever: usefulness, cross-detection, and
serological diagnostic tests for typhoid fever in solution. Diagn Microbiol Infect Dis. 2008;62
Papua New Guinea using a composite reference (2):142–150.
standard. Clin Vaccine Immunol. 2012 Nov;19 [33] Tam FCH, Leung DTM, Ma CH, et al. Modification of the
(11):1833–1837. TUBEX typhoid test to detect antibodies directly from
[29] Olopoenia LA, King AL. Widal agglutination test - 100 haemolytic serum and whole blood. J Med Microbiol.
years later: still plagued by controversy. Postgrad Med 2008;57(11):1349–1353.
J. 2000;76:80–84. [34] Thriemer K, Ley B, Menten J, et al. A systematic review
[30] Tam FCH, Lim PL. The TUBEXTM typhoid test based on and meta-analysis of the performance of two point of
particle-inhibition immunoassay detects IgM but not care typhoid fever tests, tubex tf and typhidot, in
IgG anti-O9 antibodies. J Immunol Methods. 2003;282 endemic countries. Way SS, editor. PLoS One. 2013
(1–2):83–91. Dec 16;8(12): e81263.

You might also like