Defining pediatric sepsis by different criteria: Discrepancies in

populations and implications for clinical practice

Scott L. Weiss, MD; Brandon Parker, MD; Maria E. Bullock, RN, MSN, CPNP-AC/PC; Sheila Swartz, BS;
Carolynn Price, BS; Mark S. Wainwright, MD, PhD; Denise M. Goodman, MD, MS

Objective: Pediatric patients with sepsis are identified using re-
lated but distinct criteria for clinical, research, and administrative
purposes. The overlap between these criteria will affect the valid-
ity of extrapolating data across settings. We sought to quantify the
extent of agreement among different criteria for pediatric severe
sepsis/septic shock and to detect systematic differences between
these cohorts.
Design: Observational cohort study.
Setting: Forty-two bed pediatric intensive care unit at an aca-
demic medical center.
Patients: A total of 1,729 patients ≤18 yrs-old.
Interventions: None.
Measurements and Main Results: All patients were screened
for severe sepsis or septic shock using consensus guidelines
(research criteria), diagnosis by healthcare professionals (clini-
cal criteria), and International Classification of Diseases, Ninth
Revision, Clinical Modification codes (administrative criteria).
Cohen’s κ determined the level of agreement among criteria, and
patient characteristics were compared between cohorts. Ninety
(5.2%) patients were identified by research, 96 (5.6%) by clinical,
and 103 (6.0%) by administrative criteria. The κ ± standard error
for pair-wise comparisons was 0.67 ± 0.04 for research-clinical,
0.52 ± 0.05 for research-administrative, and 0.55 ± 0.04 for clini-
cal-administrative. Of the patients in the clinical cohort, 67% met
research and 58% met administrative criteria. The research cohort
exhibited a higher Pediatric Index of Mortality-2 score (median,
interquartile range 5.2, 1.6-13.3) than the clinical (3.6, 1.1-6.2) and
administrative (3.9, 1.0-6.0) cohorts (p = .005), an increased re-
quirement for vasoactive infusions (74%, 57%, and 45%, p < .001),
and a potential bias toward an increased proportion with respira-
tory dysfunction compared with clinical practice.
Conclusions: Although research, clinical, and administrative
criteria yielded a similar incidence (5%-6%) for pediatric severe
sepsis/septic shock, there was only a moderate level of agree-
ment in the patients identified by different criteria. One third of
patients diagnosed clinically with sepsis would not have been in-
cluded in studies based on consensus guidelines or International
Classification of Diseases, Ninth Revision, Clinical Modification
codes. Differences in patient selection need to be considered when
extrapolating data across settings. (Pediatr Crit Care Med 2012;
13:e219–e226)
KEY WORDS: biomedical research; clinical trials; intensive care
units; International Classification of Disease; pediatric; sepsis;
septic shock

T he lack of a true gold standard

to define the pediatric sepsis
syndrome poses a challenge
to the study and implementa-
tion of novel therapies for children with
sepsis, septic shock, and sepsis-induced
From the Divisions of Critical Care (SLW, MEB, MSW,
DMG), Medical Education (BP), and Neurology (MSW),
Department of Pediatrics, Children’s Memorial Hospital,
Northwestern Feinberg School of Medicine (SS, CP),
Chicago, IL.
Listen to the Critical Care podcasts for an in-
depth interview on this article. Visit www.sccm.
org/iCriticalCare or search “SCCM” at iTunes.
Supported, in part, by the Division of
Critical Care, Department of Pediatrics at Children’s
Memorial Hospital.
The authors have not disclosed any potential con-
flicts of interest.
For information regarding this article, E-mail:
dgoodman@childrensmemorial.org
Copyright © 2012 by the Society of Critical
Care Medicine and the World Federation of Pediatric
Intensive and Critical Care Societies
DOI: 10.1097/PCC.0b013e31823c98da
organ dysfunction (1). The diagnosis of
sepsis is based on clinical and labora-
tory abnormalities in the presence (or
suspicion) of infection, and several re-
lated but distinct criteria are used to
identify children with sepsis for clinical,
research, and administrative purposes.
Inconsistencies within and between
these criteria have made comparisons
across studies difficult and complicate
the extrapolation of data from research
to clinical practice (1, 2).
In recognition of these concerns, con-
sensus definitions for pediatric sepsis,
severe sepsis, septic shock, and sepsis-in-
duced organ dysfunction were published
in 2005 (3). While these were intended
to facilitate uniformity across research
studies, many of the criteria have also
been variably adopted into clinical prac-
tice (4, 5). However, the strict cut points
necessary for enrollment in clinical tri-
als are not always practical at the bed-
side, where rapid diagnosis and therapy
require a more operational definition of
pediatric sepsis often based on incom-
plete information and clinical acumen
(1, 4). Therefore, the criteria used for the
clinical diagnosis of sepsis can be expect-
ed to deviate from consensus research
definitions.
The International Classification of
Diseases, Ninth Revision, Clinical Modi-
fication (ICD-9-CM) codes are also com-
monly used to identify patients with the
sepsis syndrome. These codes, intended
for administrative purposes, offer a fea-
sible approach for retrospective, epide-
miologic, and health services-related
studies and enable tracking of resource
utilization and costs. However, assign-
ment of specific ICD-9-CM codes is not
consistent within or across institutions,
and multiple codes may be associated
with the same disorder. Overlapping
ICD-9-CM codes are therefore commonly
used to identify patients, although exact
strategies may differ. For example, the

Pediatr Crit Care Med 2012 Vol. 13, No. 4 e219

epide-miologic study of severe sepsis in
children by Watson et al (6) utilized ICD-
9-CM codes for bacterial or fungal infec-
tions and organ dysfunction, whereas a
more recent study focused on a subset
of codes specific for sepsis (7). Unfortu-
nately, neither approach may accurately
reflect clinical diagnoses of sepsis or or-
gan failure (8).
To appropriately apply data from one
setting to another, it is important to un-
derstand the extent to which research,
clinical, and administrative criteria iden-
tify the same patients with the sepsis syn-
drome. A lack of sufficient agreement may
limit the external validity and applicabil-
ity of research or administrative data to
clinical practice, and thus has significant
implications for translating efficacy in
clinical trials into effective clinical prac-
tice (9, 10).
We hypothesized that the application
of consensus guidelines (research crite-
ria), diagnosis by healthcare professionals
(clinical criteria), and ICD-9-CM codes
(administrative criteria) to the same
population would identify overlapping,
but distinct, cohorts of children with se-
vere sepsis or septic shock. We sought to
quantify the extent of agreement among
these three criteria and to detect system-
atic differences between septic children
identified in each cohort.
MATERIALS AND METHODS
We performed an observational cohort
study of all patients ≤18 yrs-old admitted to
a 42-bed pediatric intensive care unit (PICU)
at an academic medical center between May
2009 and June 2010. This study was approved
by the institutional review board and a waiver
of consent was granted to perform the chart
review.
Patient Selection and
Definitions
All patients were prospectively evalu-
ated for severe sepsis or septic shock based
on criteria outlined by the International
Consensus Conference on Pediatric Sepsis
(3) as part of a previous study (11). The con-
sensus guidelines define severe sepsis as the
presence of two or more systemic inflam-
matory response syndrome criteria (one of
which must be abnormal temperature or
leukocyte count), a suspected or proven in-
fection, and either cardiovascular dysfunc-
tion or acute respiratory distress syndrome
or two or more other organ dysfunctions.
Septic shock is defined as two or more sys-
temic inflammatory response syndrome
criteria, suspected or proven infection,
e220
and cardiovascular dysfunction (3). Initial
screening was completed within 24 hrs of
admission and daily surveillance was con-
tinued until PICU discharge for interval de-
velopment of severe sepsis or septic shock.
Those patients who met consensus crite-
ria for severe sepsis or septic shock at any
time were classified as the research cohort.
Although the consensus guidelines specify
that cardiovascular and respiratory dysfunc-
tion should be present for the purposes of
enrolling children with severe sepsis into
clinical trials, patients with two or more
other organ dysfunctions were included in
the research cohort in this study.
To determine the clinical cohort, the elec-
tronic medical records of all patients were ret-
rospectively reviewed for a diagnosis of sepsis,
severe sepsis, or septic shock as documented
by a member of the care team (i.e., attending,
fellow, or resident physician or critical care
nurse practitioner) at any time during the
PICU admission. Patients were only included
if there was an explicit diagnosis of sepsis,
severe sepsis, or septic shock recorded in the
medical record. Patients with a written diag-
nosis of “sepsis” without the qualifying terms
of “severe sepsis” or “septic shock” were in-
cluded because these terms are differentiated
primarily for research purposes and are of-
ten used interchangeably in clinical practice.
Furthermore, since this study was limited to
the PICU, patients with “sepsis” were likely
to have concurrent organ dysfunction (i.e.,
severe sepsis) or septic shock. Patients with a
“rule out” or “differential diagnosis” of sepsis
were only included if there was a subsequent
explicit diagnosis of sepsis recorded in the
chart.
For both the research and clinical cohorts,
patients were included if they were identi-
fied with sepsis by their respective criteria on
any PICU day, even if this was subsequently
thought to be unlikely. This permitted iden-
tification of patients with sepsis as they would
have been prospectively enrolled in research
studies or diagnosed and treated in clinical
practice.
To identify patients in the administrative
cohort, we selected all PICU admissions over
the same time period with ICD-9-CM codes for
a bacterial or fungal infection (Appendix Table
1) and acute organ dysfunction (Appendix Table
2), as previously described for adult (12) and
pediatric (6) sepsis epidemiology studies. We
additionally included patients with ICD-9-CM
codes specific for septicemia, sepsis with acute
organ dysfunction, and septic shock (Appendix
Table 3) similar to that of Martin et al (7).
Data Collection
A review of the medical record was com-
pleted for all patients in the research, clinical,
and administrative cohorts. Data collection
began on the calendar day on which the pa-
tient was first identified as meeting either
research or clinical criteria (first septic day).
For the administrative criteria cohort, the
first septic day was considered to be the day
of PICU admission regardless of when the pa-
tient developed sepsis since ICD-9-CM codes
could not be timed to a particular day within
the hospital course. If patients were identified
with sepsis by more than one criterion corre-
sponding to different calendar days, then data
were collected starting from a distinct first
septic day for each criterion. Information was
obtained on demographics, comorbid condi-
tions, laboratory and microbiological data,
use of vasoactive infusions and mechanical
ventilation, length of stay following the first
septic day, disposition, and mortality. The
most experienced provider (in the order of
attending, fellow, nurse practitioner, and resi-
dent) who first identified the patient as having
sepsis was determined for the clinical cohort.
Total hospital and PICU charges generated
for each admission were also determined.
The Pediatric Index of Mortality (PIM)-2 (13)
and inotrope scores (14) were calculated for
the first septic day as a measure of severity of
illness and vasoactive infusion requirement,
respectively.
Data extracted from the medical chart
were recorded onto a standardized case re-
port form developed by collaborative input
from all of the authors. Five of the authors
(SLW, BP, MEB, SS, and CP) were trained to
identify patients using clinical criteria and
extract other data elements. Periodic meet-
ings were held to resolve discrepancies in
coded information and monitor the perfor-
mance of the chart abstractors. A glossary of
terms and tips for items more difficult to find
was also developed. Before performing the
chart review, inter-rater reliability testing
was undertaken. To ensure congruent iden-
tification of septic patients for the clinical
cohort, a random selection of 40 charts was
chosen with the requirement that >20% be
from patients already identified with sepsis
by research criteria to ensure sufficient prev-
alence of clinical sepsis to prevent bias (15).
All abstractors reviewed these same charts
for identification of patients in the clinical
cohort. To test for consistent extraction of
other variables, all abstractors reviewed a
second random selection of charts for repre-
sentative data elements.
Statistical Analysis
Statistical analysis was performed using
Statistical Package for the Social Sciences
(SPSS Version 12.1, Chicago, IL). Pair-wise
comparisons between the three cohorts were
determined using percentage agreement and
Cohen’s κ, with the mean κ value calculated
to assess general agreement among all three
cohorts. For continuous variables, the mean
± sd was reported for normally distributed
and medians with interquartile range (IQR)
for non-normally distributed data and were
Pediatr Crit Care Med 2012 Vol. 13, No. 4
compared using two-way analysis of variance
and Kruskal-Wallis tests, respectively. The
chi-square test was used to compare categori-
cal variables. Although we anticipated that a
subset of patients would be identified as hav-
ing sepsis by more than one criterion or might
have multiple PICU admissions over the study
period, we treated each cohort as indepen-
dent for purposes of the statistical analysis.
For inter-rater reliability testing, we com-
pared Cohen’s κ for dichotomous variables
and Pearson’s correlation (r) for continuous
variables between each pair of abstractors and
calculated the mean value for κ or r. For the
identification of patients in the clinical co-
hort, a minimum κ of 0.95 was a priori deter-
mined to be the quality threshold. For other
data elements, a mean κ or r < .80 identified
opportunities for additional training of the
abstractors.
RESULTS
Inter-Rater Reliability
The mean κ between all pairs of ab-
stractors for the identification of pa-
tients in the clinical criteria cohort was
0.96. For other data elements, the mean
κ ranged between 0.73 and 1.0 and the
mean r ranged between .33 and 1.0. The
one variable for which the mean κ was
<0.80 was sex and the two for which the
mean r was < .80 were first septic day for
patients identified using clinical criteria
and PIM-2 score. Additional training was
provided in the abstraction of these (and
similar) data elements before undertaking
further data collection.
Agreement between Research,
Clinical, and Administrative
Cohorts
Over the 13-month study period, there
were 1,729 patients ≤18 yrs old admitted to
the PICU. In total, 159 (9.2%) met at least
one criterion for severe sepsis or septic
shock, with 90 (5.2%) patients identified
by research criteria, 96 (5.6%) by clinical
criteria, and 103 (6.0%) by administrative
criteria (Fig. 1). Forty-four (27.7%) were
identified by all three criteria. The most
experienced provider who first identified
the patients as having clinical sepsis was
the attending for 62 (64%), fellow for 20
(21%), nurse practitioner for three (2%),
and resident for 12 (13%). In the adminis-
trative cohort, 48 (47%) were identified us-
ing ICD-9-CM codes for both a bacterial or
fungal infection and acute organ dysfunc-
tion, and 65 (66%) using ICD-9-CM codes
for septicemia, sepsis with acute organ
Pediatr Crit Care Med 2012 Vol. 13, No. 4
dysfunction, or septic shock (13 [13%] had
both) . The pair-wise comparisons among
the different criteria are shown in Table 1.
Since the incidence of sepsis was low, we
determined the percentage agreement us-
ing all patients as the denominator, as well
as only septic patients to avoid “prevalence
bias.” The mean κ (± se) across all three
cohorts was 0.58 (± 0.04).
To determine the proportion of pe-
diatric patients diagnosed with severe
sepsis or septic shock in clinical practice
for which it would be appropriate to gen-
eralize results obtained from a study en-
rolling patients according to consensus
guidelines or ICD-9-CM codes, we cal-
culated the percentage of patients in the
clinical cohort also identified by research
and administrative criteria, respectively.
For a study based on research criteria,
results would be appropriately applied to
67% (64 of 96) of the clinical cohort. For
a study based on administrative criteria,
results would be appropriately applied to
58% (57 of 96) of the clinical cohort.
Figure 1.  Venn diagram. The overlap of patients
identified as having severe sepsis or septic shock
by research, clinical, and administrative criteria
is shown in this Venn diagram. Of the total 1,729
patients admitted to the pediatric intensive care
unit, 159 (9.2%) were identified as septic by at
least one criterion and 44 (2.5%) were identified
as septic by all three criteria.
Table 1.  Agreement between sepsis definitions
% Agreementa
Pair-wise Comparison (All Patients)
Research – clinical 96.6%
Research – administrative 95.0%
Clinical – administrative 95.1%
a
Percentage of patients identified as either having or not having sepsis by both criteria divided
by the total number of patients screened; bPercentage of patients identified as having sepsis by both
criteria divided by the total number of patients identified as having sepsis by either criteria; cCohen’s
κ (± se).
Comparison of Research,
Clinical, and Administrative
Cohorts
Patient characteristics and labora-
tory values for the research, clinical,
and administrative cohorts are shown in
Tables 2 and 3, respectively. There were no
differences in age, sex, race, proportion
with cancer or neutropenia, or source
of infection between groups. Patients in
the research cohort had a higher median
PIM-2 score (5.2, IQR 1.6-13.3) compared
with patients in the clinical (3.6, 1.1-6.2)
and administrative (3.9, 1.0-6.0) cohorts
(p = .005). There was also an increased
frequency of cardiovascular dysfunction
(defined as receiving vasoactive infusions)
on the first septic day in the research co-
hort. However, the proportion of patients
who required vasoactive infusions for >1
or 2 days was similar (Table 4). There
were no differences in noncardiovascular
organ dysfunction. The trend toward lon-
ger length of stay for the administrative
cohort likely reflects that the first septic
day, by definition, was the initial day of
PICU admission for this group. However,
the median (IQR) first septic day was
also day 1 (1-2) and day 1 (1-1) for the
research and clinical cohorts, respective-
ly. PICU and total hospital charges were
lowest for patients in the clinical cohort
(Table 2).
Mortality for patients identified with
severe sepsis or septic shock by any cri-
teria was 18% (29 of 159), and 20% (nine
of 44) for patients identified by all three
criteria. Although there was a lower per-
centage of nonsurvivors in the clinical
cohort (13%) compared with the research
(21%) and administrative (22%) cohorts,
this difference did not reach significance
(p = .16). There were no differences in
laboratory values related to organ func-
tion or global perfusion (arterial pH, lac-
tate) on the first septic day between the
three cohorts (Table 3).
% Agreementb
(Septic Patients) κ (±se[r])c
52.5% 0.67 ± 0.04
37.8% 0.52 ± 0.05
40.1% 0.55 ± 0.04
e221
Table 2.  Characteristics for research, clinical, and administrative cohorts

Research Clinical Administrative

Variablea (n = 90) (n = 96) (n = 103) pb

Age (years), mean ± sd 8.86 ± 6.34 8.19 ± 6.26 8.39 ± 6.27 .84
Male, n (%) 41 (46) 42 (44) 47 (46) .96
Non-Caucasian race, n (%) 50 (56) 57 (59) 62 (60) .79
% Households below poverty linec 11 (6-19) 12 (6-21) 12 (6-20) .93
Pediatric Index of Mortality-2 score 5.2 (1.6- 13.3) 3.6 (1.1-6.2) 3.9 (1.0-6.0) .005
Comorbidity        
  Previously healthy 15 (17) 20 (21) 17 (17) .74
 Oncologic 34 (38) 28 (29) 34 (33)  
Neutropenic (absolute neutrophil count 26 (32) 22 (26) 21 (22) .35
  <500/µL), n (%)d
Positive culture (any source), n (%) 65 (72) 62 (65) 71 (69) .53
Source of infection, n (%)e        
 Blood 30 (49) 30 (52) 35 (51) .10
 Respiratory 23 (38) 11 (19) 17 (25)  
Organ dysfunction, n (%)f        
 Cardiovascular 67 (74) 55 (57) 46 (45) <.001
 Respiratory 43 (48) 37 (39) 42 (41) .42
  Renal (creatinine ≥2.0 mg/dL) 10 (12) 12 (13) 11 (11) .91
 Hepatic        
  Alanine aminotransferase ≥100 IU/L 8 (19) 6 (18) 13 (26) .61
  Total bilirubin ≥4.0 mg/dL 2 (5) 1 (3) 5 (10) .42
 Hematologic        
  Platelets <80,000/µL 33 (40) 34 (39) 42 (43) .83
   International normalized ratio ≥2.0 9 (19) 7 (14) 9 (15) .71
Inotrope score (day 1 maximum)g 12 (9-22) 11 (10-29) 15 (10-26) .89
Vasoactive infusion-free days (out of 28) 25 (21-27) 26 (23-28) 26 (20-28) .06
Ventilator-free days (out of 28) 24 (9-28) 28 (18-28) 27 (12-28) .13
Nonsurvivors, n (%) 19 (21) 12 (13) 23 (22) .16
Discharge to rehabilitation, n (%)h 7 (10) 6 (7) 6 (8) .80
Pediatric intensive care unit length of stay, days 7 (4-15) 6 (3-13) 10 (4-23) .05
Hospital length of stay, days 16 (8-35) 13 (7-33) 25 (10-47) .08
Pediatric intensive care unit charges ($)i 44,149 (14,492-97,918) 26,120 (12,470-76,4610) 59,107 (18,604-45,997) .04
Total hospital charges ($)i 245,487 (76,700-562,806) 158,129 (67,574-438,375) 360,166 (118,332-842,778) .03

a
Values are median (interquartile range), unless indicated; btwo-way analysis of variance or Kruskal-Wallis tests for continuous and chi-square test for categorical
variables; ccensus tract data (Source: U.S. Census Bureau, 2005-2009 American Community Survey); donly those who had laboratory values measured on first septic day
were included; eonly those who had microbiologically confirmed pathogens were included; forgan dysfunction present on first septic day; cardiovascular dysfunction
defined as receiving ≥1 vasoactive infusion; respiratory dysfunction defined as noninvasive or invasive mechanic ventilation; gonly those receiving vasoactive infusions
on first septic day were included; Inotrope score = dopamine + dobutamine + (norepinephrine × 100) + (epinephrine × 100) + (milrinone × 10) + (vasopressin × 10)14;
h
survivors only; icharges accrued may not reflect actual billing to clarify that these represent charges accrued rather than what was billed to the patient.

Table 3. Laboratory valuesa source of infection (50% vs. 16%, p = .01),

a higher frequency of respiratory dysfunc-
Research Clinical Administrative tion (65% vs. 41%, p = .03), and trends
toward fewer ventilator-free days (16, IQR
Laboratory n Median (IQR) n Median (IQR) n Median (IQR) pb 0-28 vs. 27, IQR 15-28, p = .052), less car-
diovascular dysfunction (62% vs. 80%,
White blood cell count (thou/µL) 83 6.4 (0.7-17.2) 88 6.7 (1.8-17.7) 98 7.3 (2.4-15.3) .87
Hemoglobin (g/dL) 83 10.3 (9.0-11.3) 88 10.0 (8.8-11.3) 98 9.8 (8.6-11.3) .61 p = .06), and higher mortality (31% vs.
Platelet (thou/µL) 83 131 (38-272) 88 168 (46-303) 98 138 (38-306) .66 17%, p = .13). These findings suggest that
Blood urea nitrogen (mg/dL) 87 16 (10-28) 92 17 (9-28) 99 16 (10-28) .87 research criteria may more frequently
Creatinine (mg/dL) 87 0.59 (0.37-0.97) 92 0.54 (0.33-1.0) 99 0.56 (0.32-0.99) .98 categorize pediatric patients with pulmo-
Glucose (mg/dL) 86 121 (102-160) 92 118 (101-145) 97 119 (97-142) .76
Alanine aminotransferase (IU/L) 43 45 (23-78) 34 46 (22-89) 51 45 (21-100) .99
nary pathology as septic compared with
Total bilirubin (mg/dL) 42 1.1 (0.5-1.7) 33 0.9 (0.45-1.5) 52 0.9 (0.5-2.0) .63 clinical practice.
International normalized ratio 47 1.45 (1.17-1.86) 52 1.41 (1.17-1.65) 62 1.38 (1.11-1.72) .53 In contrast, the 19 patients identified
pH, arterial 36 7.32 (7.22-7.40) 34 7.32 (7.26-7.40) 37 7.32 (7.26-7.39) .99 solely by clinical criteria exhibited a low
Lactate, arterial (mEq/L) 38 1.9 (1.0-4.9) 36 1.5 (0.9-4.5) 39 1.8 (1.0-2.8) .68 median PIM-2 score (1.5, IQR 1.1-4.5)
IQR, interquartile range.
and, compared with patients identified
a
Only those who had laboratory values measured on first septic day were included; bcomparisons by clinical plus a second criteria, had a
made using the Kruskal-Wallis test. lower incidence of culture-positive sep-
sis (42% vs. 70%, p = .03), less cardio-
Of the patients in the research co- both research and clinical criteria, these vascular dysfunction (11% vs. 69%, p <
hort, 29% (26 of 90) were not diagnosed 26 patients had significantly higher PIM- .001), shorter PICU length of stay (4, 2-5
by the care team as having sepsis. When 2 scores (10.9, 4.2-34 vs. 4.8, 1.3-6.7, p = days vs. 7, 3-14 days, p = .02), and fewer
compared to the 64 patients identified by .003), increased likelihood of a respiratory nonsurvivors (0% vs. 16%, p = .12). The

e222 Pediatr Crit Care Med 2012 Vol. 13, No. 4

Table 4. Duration of vasoactive infusions
Research
Vasoactive Infusion (n = 90)
Durationb
≤1 day 27 (30)
>1 day 63 (70)
≤2 days 41 (46)
<2 days 49 (54)
a
Comparisons made using the chi-square test; bdata are presented as n (%).
37  patients identified only by adminis-
trative criteria were less likely to have
the sepsis-specific ICD-9 codes listed in
Appendix Table 3 (38%) compared with
patients identified by both administra-
tive and clinical criteria (72%, p < .001).
DISCUSSION
We found that 5%-6% of patients ad-
mitted to a large, tertiary PICU over a
13-month time period were identified
with severe sepsis or septic shock re-
gardless of whether research, clinical,
or administrative criteria were utilized.
However, the modest degree of overlap
revealed that, in many cases, distinct pa-
tients were identified by different criteria.
Although the overall percentage agree-
ment between criteria was ≥95% when in-
cluding all 1,729 patients, the agreement
when considering only septic patients
and the κ statistics was less robust. While
there is no standard κ value for a minimal
acceptable level of agreement between
groups, a common interpretation is that
values <0.60 constitute low, 0.60-0.80
moderate, and >0.80 high levels of agree-
ment (15). Thus, there was moderate
agreement between research and clinical
criteria, but only low levels of agreement
between research-administrative and
clinical-administrative criteria.
Approximately one third of patients
diagnosed clinically with sepsis would
not have been included in studies based
on consensus guidelines or ICD-9-CM
codes. Such exclusions could have criti-
cal implications for the extrapolation of
data from clinical trials and epidemio-
logic studies of pediatric sepsis to clini-
cal practice. Similar findings have been
reported in other disorders (16-18). In
a survey of ischemic stroke in adults,
only 25%-67% of patients would have
fulfilled enrollment criteria for seven
landmark antiplatelet trials, leading the
authors to conclude that patients with
ischemic stroke in randomized trials
Pediatr Crit Care Med 2012 Vol. 13, No. 4
Clinical Administrative
(n = 96) (n = 103) pa
44 (46) 42 (41) .08
52 (54) 61 (59)  
55 (57) 53 (51) .28
41 (43) 50 (49)  
are only partially representative of those
treated in clinical practice (17). Another
study found that 44% of potentially
eligible patients with the index disease
were excluded from study participation
in 16 randomized trials (16). Such dis-
crepancies between patient selection for
research and diagnosis in clinical prac-
tice, while perhaps unavoidable to some
degree, complicate the translation of
efficacy in clinical trials into effective-
ness at the bedside (10). Consequently,
Dougherty and Conway (9) have proposed
the “3T’s” roadmap in which evaluation
of efficacy (“what works” or T1), effective-
ness (“who benefits” or T2), and delivery
of care (“how to deliver” or T3) are all
deemed necessary to successfully dissem-
inate new knowledge into practice for the
right patients. Based on the modest level
of agreement among criteria used in dif-
ferent settings in this study, research in
pediatric sepsis would benefit from appli-
cation of this model.
Notably, the three cohorts were
similar with regards to demographics,
comorbid conditions, source of infec-
tion, noncardiovascular organ dysfunc-
tion, discharge to rehabilitation centers,
and survival. Although prior studies
have shown that patients recruited into
clinical trials often differ from those eli-
gible but not enrolled in terms of age,
sex, and race/ethnicity (10), we did not
find evidence for such biases in this
study. While the similarities in patient
profiles strengthens the external valid-
ity of consensus guidelines and ICD-9-
CM codes for studying pediatric sepsis,
comparing only these variables would
have obscured the extent of disagree-
ment in patients identified by different
criteria. For example, in the epidemio-
logic studies of severe sepsis performed
by Angus et al (12) and Watson et al (6),
the researchers validated their ICD-9-
CM-based methodology by comparing
a subset of their population to a cohort
from the same hospitals that had been
prospectively diagnosed with severe sep-
sis using consensus criteria (19) and did
not find important differences in age,
sex, intensive care unit admission, or
site of infection. Whether the different
criteria would have identified the same
patients, however, could not be deter-
mined, and their validation cohort did
not include children.
The research cohort had the highest
median PIM-2 score and proportion of
patients with cardiovascular dysfunction,
suggesting a predisposition toward en-
rollment of septic children with a higher
severity of illness and frequency of fluid-
refractory shock into clinical trials. In
addition, the duration of vasoactive infu-
sions trended lower for the clinical co-
hort and the patients identified solely by
clinical criteria had a lower severity of ill-
ness than patients who also met research
and/or administrative criteria. Given that
multiple studies have demonstrated the
greatest benefit of novel therapeutics in
the sickest patients with sepsis (20-22),
these findings support the appropriate-
ness of enrolling septic children with a
high severity of illness into clinical trials
and yet also reinforce the need to judi-
ciously extrapolate research findings to
the most appropriate pediatric patients.
The lower severity of illness in the clini-
cal cohort may also reflect caretaker
vigilance to diagnose sepsis in its earli-
est stages, underscoring the success of
recent initiatives, such as the Surviving
Sepsis Campaign (5) and practice guide-
lines for pediatric shock (4), which strive
to improve the early recognition and
rapid reversal of sepsis before the onset
of hemodynamic deterioration. The lower
charges and shorter PICU length of stay
in the clinical cohort further support
early recognition and rapid reversal of
clinically diagnosed sepsis. The consen-
sus guidelines, in contrast, may preferen-
tially identify patients in more advanced
stages of sepsis (1).
There was also a predisposition to-
ward identifying patients with pulmonary
pathology as having sepsis by research
criteria compared with clinical practice.
This could relate to the inclusion of acute
respiratory distress syndrome as one de-
terminant for severe sepsis in the con-
sensus guidelines (3), whereas clinicians
may be more likely to distinguish acute
respiratory distress syndrome from sep-
sis at the bedside. A higher enrollment
of patients with acute respiratory distress
syndrome into clinical trials for sepsis
compared with bedside diagnosis could
e223
affect the external validity of the results
from these studies.
Several factors likely contributed to
the overlapping, yet distinct, cohorts of
patients identified using different criteria.
The consensus guidelines define specific
cut points for systemic inflammatory re-
sponse syndrome and organ dysfunction
and, recognizing that tachycardia and
tachypnea are common symptoms in a
variety of pediatric disorders, require at
least one systemic inflammatory response
syndrome criterion to be abnormal tem-
perature or leukocyte count (3). In con-
trast, distinctions between “normal” and
“abnormal” vital signs and indices of or-
gan failure are subjective in clinical prac-
tice. Furthermore, since microbiological
evidence for infection is typically delayed,
both research and clinical criteria allow
for suspicion of infection, which may be
interpreted differently by clinicians and
researchers.
The largest discrepancy in this study
was between patients identified by admin-
istrative and the other two criteria. In our
institution, multiple coders assign ICD-
9-CM codes, which may lead to inconsis-
tencies. Additionally, ICD-9-CM codes are
unlikely to reflect “working diagnoses”
since these codes are often retrospectively
assigned at discharge. In contrast, we
included patients in the clinical and re-
search cohorts even if the sepsis diagnosis
was later found to be incorrect since these
patients would have been considered for
sepsis-related therapy or study enroll-
ment with “real-time” decision making.
Finally, while our use of ICD-9-CM codes
for infection and acute organ dysfunction
was based on previous, well-designed epi-
demiologic studies (6, 12), this approach
does not ensure temporal overlap between
infection and organ dysfunction and re-
quires only a single organ dysfunction.
Patients identified using these ICD-9-CM
combinations were also less likely to have
been concurrently diagnosed clinically
as septic. Thus, this approach may have
been overly inclusive. However, reliance
on sepsis-specific ICD-9-CM codes alone
is also an imperfect reflection of clinical
practice (7, 8).
Strengths of this study include the
application of three different criteria to
a large, unbiased population, prospec-
tive determination of the research cohort
(which involves the most explicit defi-
nitions), and utilization of established
combinations of ICD-9-CM codes (6, 7,
12). There were also several limitations.
e224
First, we retrospectively identified the
clinical cohort, and while electronic
medical records provide comprehensive
data, their accuracy for use in research
is difficult to quantify (23). Second, we
utilized multiple chart abstractors. Since
this was necessary due to the sample size,
we employed several methods, includ-
ing an inter-rater reliability analysis, to
ensure accuracy and consistency of data
extraction (24). Third, hospital charges
and mortality were based on crude data
and were not directly attributable to sep-
sis. Finally, the inclusion of patients re-
gardless of whether the sepsis diagnosis
subsequently proved correct could have
inflated the number of patients identi-
fied by clinical and research criteria.
Similarly, querying the medical record
for a diagnosis of “sepsis” – in addition to
“severe sepsis” and “septic shock” – may
have included some patients who had
sepsis without organ dysfunction in the
clinical cohort. We also caution that our
findings reflect those of a single institu-
tion, and a prospective, multicentered
study is necessary to fully understand
differences in how pediatric patients with
sepsis are identified in clinical, research,
and administrative settings, including
variability in genetics, treatment, and
outcomes.
CONCLUSIONS
Although research, clinical, and ad-
ministrative criteria yielded a similar
incidence of 5%-6% for pediatric se-
vere sepsis/septic shock, there was only
a moderate level of agreement in the
specific patients identified by different
criteria. Approximately one third of pa-
tients diagnosed clinically with sepsis
would not have been included in studies
based on consensus criteria or ICD-9-CM
codes, and while the cohorts were simi-
lar in many aspects, patients identified
by research criteria exhibited a higher
severity of illness and incidence of flu-
id-refractory shock, as well as a poten-
tial bias toward respiratory dysfunction
compared with clinical practice. Despite
the importance of consensus guidelines
in standardizing enrollment for pediatric
sepsis research, differences in patient se-
lection still need to be considered when
extrapolating data across settings, and
attention to studies of efficacy, effective-
ness, and delivery will be important for
implementation of novel therapeutic
strategies in pediatric sepsis.
ACKNOWLEDGMENTS
We thank George Lales, MS, for assis-
tance with ICD-9-CM coding and determi-
nation of hospital charges, and Tracie L.
Smith, MPH, for assistance with census
tract data.
REFERENCES
1. Brilli RJ, Goldstein B: Pediatric sepsis defini-
tions: Past, present, and future. Pediatr Crit
Care Med 2005; 6:S6–S8
2. Fischer JE: Physicians’ ability to diagnose
sepsis in newborns and critically ill children.
Pediatr Crit Care Med 2005; 6:S120–S125
3. Goldstein B, Giroir B, Randolph A, et al:
International pediatric sepsis consensus con-
ference: Definitions for sepsis and organ dys-
function in pediatrics. Pediatr Crit Care Med
2005; 6:2–8
4. Brierley J, Carcillo JA, Choong K, et al:
Clinical practice parameters for hemody-
namic support of pediatric and neonatal sep-
tic shock: 2007 update from the American
College of Critical Care Medicine. Crit Care
Med 2009; 37:666–688
5. Dellinger RP, Levy MM, Carlet JM, et al:
Surviving Sepsis Campaign: International
guidelines for management of severe sepsis
and septic shock: 2008. Crit Care Med 2008;
36:296–327
6. Watson RS, Carcillo JA, Linde-Zwirble WT, et
al: The epidemiology of severe sepsis in chil-
dren in the United States. Am J Respir Crit
Care Med 2003; 167:695–701
7. Martin GS, Mannino DM, Eaton S, et al: The
epidemiology of sepsis in the United States
from 1979 through 2000. N Engl J Med 2003;
348:1546–1554
8. Ollendorf DA, Fendrick AM, Massey K, et al:
Is sepsis accurately coded on hospital bills?
Value Health 2002; 5:79–81
9. Dougherty D, Conway PH: The “3T’s” road
map to transform US health care: The “how” of
high-quality care. JAMA 2008; 299:2319–2321
10. Rothwell PM: External validity of randomised
controlled trials: “To whom do the results of
this trial apply?”. Lancet 2005; 365:82–93
11. Weiss S, Haymond S, Ralay-Ranaivo H, et al:
Decreased plasma asymmetric dimethylargi-
nine (ADMA) and altered arginine metabo-
lism in pediatric sepsis. Crit Care Med 2010;
38:A61
12. Angus DC, Linde-Zwirble WT, Lidicker J, et al:
Epidemiology of severe sepsis in the United
States: Analysis of incidence, outcome, and
associated costs of care. Crit Care Med 2001;
29:1303–1310
13. Leteurtre S, Martinot A, Duhamel A, et al:
Validation of the paediatric logistic organ
dysfunction (PELOD) score: Prospective, ob-
servational, multicentre study. Lancet 2003;
362:192–197
14. Gaies MG, Gurney JG, Yen AH, et al: Vasoactive-
inotropic score as a predictor of morbidity and
Pediatr Crit Care Med 2012 Vol. 13, No. 4
 mortality in infants after cardiopulmonary by-
pass. Pediatr Crit Care Med 2010; 11:234–238
15. Liddy C, Wiens M, Hogg W: Methods to
achieve high interrater reliability in data col-
lection from primary care medical records.
Ann Fam Med 2011; 9:57–62
16. Charlson ME, Horwitz RI: Applying results of
randomised trials to clinical practice: Impact
of losses before randomisation. BMJ 1984;
289:1281–1284
17. Maasland L, van Oostenbrugge RJ, Franke
CF, et al: Patients enrolled in large random-
ized clinical trials of antiplatelet treatment
for prevention after transient ischemic attack
or ischemic stroke are not representative of
patients in clinical practice: The Netherlands
Stroke Survey. Stroke 2009; 40:2662–2668
18. Stein BD, Charbeneau JT, Lee TA, et al:
Hospitalizations for acute exacerbations of
chronic obstructive pulmonary disease: How
you count matters. COPD 2010; 7:164–171
19. Sands KE, Bates DW, Lanken PN, et al:
Epidemiology of sepsis syndrome in 8 academ-
ic medical centers. JAMA 1997; 278:234–240
20. Bernard GR, Vincent JL, Laterre PF, et al:
Efficacy and safety of recombinant human
activated protein C for severe sepsis. N Engl
J Med 2001; 344:699–709
21. Abraham E, Laterre PF, Garg R, et al:
Drotrecogin alfa (activated) for adults with
severe sepsis and a low risk of death. N Engl
J Med 2005; 353:1332–1341
22. Tidswell M, Tillis W, Larosa SP, et al: Phase
2 trial of eritoran tetrasodium (E5564), a
toll-like receptor 4 antagonist, in patients
with severe sepsis. Crit Care Med 2010;
38:72–83
23. Wasserman RC: Electronic medical records
(EMRs), epidemiology, and epistemology:
Reflections on EMRs and future pediatric clin-
ical research. Acad Pediatr 2011; 11:280–287
24. Gilbert EH, Lowenstein SR, Koziol-McLain J,
et al: Chart reviews in emergency medicine
research: Where are the methods? Ann Emerg
Med 1996; 27:305–308

Appendix Table 1.  International Classification of Dis- Appendix Table 1.—Continued Appendix Table 1.—Continued
eases, Ninth Revision, Clinical Modification codes
used to identify bacterial or fungal infection. Codea Description Codea Description

Codea Description 097 Other and unspecified syphilis 562.11 Diverticulitis of colon without
098 Gonococcal infections hemorrhage
100 Leptospirosis 562.13 Diverticulitis of colon with hemorrhage
001 Cholera
101 Vincent’s angina 556 Anal and rectal abscess
002 Typhoid/paratyphoid fever
102 Yaws 567 Peritonitis
003 Other salmonella infection
103 Pinta 569.5 Intestinal abscess
004 Shigellosis
104 Other spirochetal infection 569.83 Perforation of intestine
005 Other food poisoning
110 Dermatophytosis 572 Abscess of liver
008 Intestinal infection not otherwise
111 Dermatomycosis not otherwise 572.1 Portal pyemia
classified
classified or specified 575.0 Acute cholecystitis
009 Ill-defined intestinal infection
112 Candidiasis 590 Kidney infection
010 Primary tuberculosis infection
114 Coccidioidomycosis 597 Urethritis/urethral syndrome
011 Pulmonary tuberculosis
115 Histoplasmosis 599.0 Urinary tract infection not otherwise
012 Other respiratory tuberculosis
116 Blastomycotic infection specified
013 Central nervous system tuberculosis
117 Other mycoses 601 Prostatic inflammation
014 Intestinal tuberculosis
118 Opportunistic mycoses 614 Female pelvic inflammation disease
015 Tuberculosis of bone and joint
320 Bacterial meningitis 615 Uterine inflammatory disease
016 Genitourinary tuberculosis
322 Meningitis, unspecified 616 Other female genital inflammation
017 Tuberculosis not otherwise classified
324 Central nervous system abscess 681 Cellulitis, finger/toe
018 Miliary tuberculosis
325 Phlebitis of intracranial sinus 682 Other cellulitis or abscess
020 Plague
420 Acute pericarditis 683 Acute lymphadenitis
021 Tularemia
421 Acute or subacute endocarditis 686 Other local skin infection
022 Anthrax
451 Thrombophlebitis 711.0 Pyogenic arthritis
023 Brucellosis
461 Acute sinusitis 730 Osteomyelitis
024 Glanders
462 Acute pharyngitis 790.7 Bacteremia
025 Melioidosis
463 Acute tonsillitis 996.6 Infection or inflammation of device/
026 Rat-bite fever
464 Acute laryngitis/tracheitis graft
027 Other bacterial zoonoses
465 Acute upper respiratory infection of 998.5 Postoperative infection
030 Leprosy
031 Other mycobacterial disease multiple sites/not otherwise specified 999.3 Infectious complication of medical care
032 Diphtheria 481 Pneumococcal pneumonia not otherwise classified
033 Whooping cough 482 Other bacterial pneumonia
034 Streptococcal throat/scarlet fever 485 Bronchopneumonia with organism not a
Where 3 or 4 digit codes are listed, all associ-
035 Erysipelas otherwise specified ated subcodes were included.
036 Meningococcal infection 486 Pneumonia, organism not otherwise
037 Tetanus specified
038 Septicemia 491.21 Acute exacerbation of obstructive
039 Actinomycotic infections chronic bronchitis
040 Other bacterial diseases 494 Bronchiectasis
041 Bacterial infection in other diseases not 510 Empyema
otherwise specified 513 Lung/mediastinum abscess
090 Congenital syphilis 540 Acute appendicitis
091 Early symptomatic syphilis 541 Appendicitis not otherwise specified
092 Early syphilis latent 542 Other appendicitis
093 Cardiovascular syphilis 562.01 Diverticulitis of small intestine without
094 Neurosyphilis hemorrhage
095 Other late symptomatic syphilis 562.03 Diverticulitis of small intestine with
096 Late syphilis latent hemorrhage

Pediatr Crit Care Med 2012 Vol. 13, No. 4 e225

Appendix Table 2.  International Classification of
Diseases, Ninth Revision, Clinical Modification
codes used to identify organ dysfunction

Codea Description

785.5 Shock without trauma

458 Hypotension
96.7 Mechanical ventilation
348.3 Encephalopathy
293 Transient organic psychosis
348.1 Anoxic brain damage
287.4 Secondary thrombocytopenia
287.5 Thrombocytopenia, unspecified
286.9 Other/unspecified coagulation defect
286.6 Defibrination syndrome
570 Acute and subacute necrosis of liver
573.4 Hepatic infarction
584 Acute renal failure

a
Where 3 or 4 digit codes are listed, all associ-
ated subcodes were included.

Appendix Table 3.  International Classification

of Diseases, Ninth Revision, Clinical Modification
codes for septicemia, severe sepsis, and septic shock

Codea Description

003.1 Septicemia, Salmonella

004.9 Septicemia, Shigella
020.2 Septicemia, plague
022.3 Septicemia, with anthrax
027.0 Septicemia, Listeria
monocytogenes
027.1 Septicemia, with Erysipelothrix
036.2 Septicemia, meningococcal
038 Septicemia (suppurative), with
organism
054.5 Septicemia, herpes/herpetic
098.89 Septicemia, gonococcal
639.0 Septicemia following abortion or
pregnancy
659.3 Septicemia, complicating labor
670.0 Septicemia, postpartum
771.81 Septicemia, newborn
785.52 Septic, shock
995.91 Sepsis (generalized), without acute
organ dysfunction
995.92 Sepsis, with acute organ
dysfunction/multiple organ
dysfunction/severe
998.59 Septicemia, postoperative
999.39 Septicemia, following infusion,
injection, transfusion, or
vaccination

a
Where 3 or 4 digit codes are listed, all associ-
ated subcodes were included.

e226 Pediatr Crit Care Med 2012 Vol. 13, No. 4