Professional Documents
Culture Documents
Pediatric brain injury is a significant public health may survive, the resulting morbidity can be devas-
problem in the United States.1,2 Such brain injury may tating.
be traumatic, anoxic (e.g., near-drowning), or in- Many approaches for improving the neurologic
fectious. With advances in therapy, many infants outcome of children after brain injury are under
and children who have suffered potentially fatal investigation, including accurate triage to specialized
insults are now surviving. Although these children centers, medical and surgical therapies, and rehabil-
itation. The efficacy of interventions cannot be
evaluated, however, unless neurologic outcome can
From the Division of Emergency Medicine, Department of be reliably measured. A recent review of existing
Pediatrics, University of Texas Southwestern Medical Center of outcome measures for pediatric brain injury identified
Dallas, Dallas, TX (PJO); Department of Emergency Medicine,
a need for versatile, practical, and reliable clinical
Harbor-UCLA Medical Center, Torrance, CA (KDY, RJL); De-
partment of Pediatrics, Harbor-UCLA Medical Center, Torrance, instruments for measuring neurologic outcome in
CA (KDY, KRH); Harbor-UCLA Research and Education Institute, infants and children.3
Torrance, CA (KDY, KRH, RJL); Department of Emergency A number of complex neuropsychological tests to
Medicine, University of Pennsylvania School of Medicine, Phila- measure deficits in cognitive or motor function have
delphia, PA (JMB); Department of Pediatrics, Children’s Hospital
Central California, Fresno, CA (SSR); Division of Neurology, been developed and validated.3–8 The need for
Department of Pediatrics, Harbor-UCLA Medical Center, Torrance, specialized expertise on the part of the examiner, the
CA (KRH); Denton Regional Medical Center, Denton, TX (JBW); length and complexity of these instruments, and their
Division of Psychology, Department of Psychiatry, Children’s focus on a narrow range of ages or levels of neurologic
Medical Center of Dallas, Dallas, TX (PLS); and Division of
Psychology, Department of Psychiatry, University of Texas South-
function make these instruments largely impractical
western Medical Center of Dallas, Dallas, TX (PLS). for use in clinical pediatric brain injury research. On the
Received June 9, 2003; revision received June 9, 2003; accepted June other end of the spectrum, many investigators have
11, 2003. used simple, multilevel scales such as the Glasgow
Supported by an individual NRSA fellowship grant (5 F32 HS00091)
from the Agency for Health Care Policy and Research (now the
Outcome Scale (GOS) or the Pediatric Cerebral
Agency for Healthcare Research and Quality) to Dr. Okada. Performance Scale (PCPS) to measure neurologic out-
Address for correspondence and reprints: Pamela J. Okada, MD, come in children.9–14 These scales are easy to use by the
Department of Pediatrics, Division of Emergency Medicine, nonspecialist and cover a broad range of outcomes, but
University of Texas Southwestern Medical Center at Dallas, 5323
Harry Hines Boulevard, Dallas, TX 75390-9063. E-mail: pokada@
they do not account for the developmental level of the
childmed.dallas.tx.us. child and are often not sensitive enough to detect small,
doi:10.1197/S1069-6563(03)00364-6 but important, differences in outcome.
ACAD EMERG MED d October 2003, Vol. 10, No. 10 d www.aemj.org 1035
The purpose of this study was to develop a reliable Instrument Revision. The reliability data were sent
and valid neurologic outcome measure for infants and to an expert panel for review along with the NOSIC
children, appropriate over a broad range of ages and instrument and an explanation of the scoring system.
levels of neurologic outcome, which could be applied The expert panel evaluated the face validity and
in a brief period of time by nonspecialist examiners, content validity, and identified any test items that
and which would account for developmental differ- were unclear, developmentally inappropriate, or
ences in cognitive and motor skills by age. needed modification. Based on the input of the expert
panel, a number of modifications were made. Mental
METHODS status criteria were separated into narrower age
groups. Criteria for deficits in the areas of vision,
The Neurologic Outcome Scale for Infants and speech and hearing, learning, attention, social disin-
Children (NOSIC) was developed in 4 steps: 1) hibition, and activities of daily living were defined
creation of a draft instrument; 2) reliability testing of separately according to age group. Finally, definitions
the draft instrument using a test population of of several key terms were clarified, and the scoring
children with a wide range of ages and levels of system was modified slightly.
neurologic function; 3) revision of the draft instru-
ment after expert review of reliability data; and 4) Validity and Reliability Testing of the Revised
reliability and validity testing of the revised instru- Instrument. The revised instrument was adminis-
ment in a new population of children. tered to 157 additional children with or without prior
neurologic abnormalities recruited from the pediatric
Creation of the Draft Instrument. The following emergency department, pediatric clinics, and the
domains of neurologic function were included: inpatient wards at Children’s Medical Center of
general level of consciousness, current mental status Dallas, a large academic children’s hospital, by one
relative to age-appropriate norms, motor function investigator (PJO). Fluent translators were used for
(cranial nerves, truncal tone, ability to sit, gait, children and parents whose primary language was
extremity strength), visual attention, speech and Spanish. The complete instrument, along with a scor-
hearing, learning and schooling, task attention, social ing guide, is available as a Data Supplement (Adobe
disinhibition, activities of daily living, and the care- Acrobat .pdf format) in the electronic version of this
takers’ perception of the impact the child’s neurologic article (at www.aemj.org). A second investigator
status has on his or her activities. (SSR), masked to the results of the first examination,
Test items were weighted according to their applied the revised instrument to 84 of these children.
perceived importance, based on input from a pediatric Trained psychometrists supervised by a neuropsy-
neurologist (KRH), a neuropsychologist (JBW), and chologist administered age-specific, previously-vali-
a child development specialist. Scoring of items on the dated neuropsychological tests to the children as part
NOSIC is adjusted for items that cannot be assessed in of their usual clinical care. These tests were used as
individual children. For example, if a child with head criterion standards in assessing the validity of the
trauma and a long-bone fracture cannot be assessed NOSIC. Different instruments were used depending
for motor activity in the fractured limb, the maximum on the age and level of neurologic function of the
raw NOSIC score possible is reduced accordingly. The child (Table 1).
total score is then normalized to a maximum of 100,
taking into account the maximum possible raw score Data Management and Statistical Analysis. All
based on all evaluable items. Thus, a score of 100 data were collected on a closed-response data
represents normal neurologic status (a perfect score collection form and entered into an electronic data-
on all evaluable items), even if some components base (Paradox version 3.5 or 7.0, Borland, Scotts
could not be tested for non-neurologic reasons. Valley, CA). Data were translated into native SAS
format using a translation program (DBMS/Copy,
Reliability Testing of the Draft Instrument. The Conceptual Software, Inc., Houston, TX) and analyzed
draft instrument was administered to 100 children using SAS Version 8.1 (SAS Institute, Inc., Cary, NC).
with and without prior neurologic diagnoses re- Descriptive statistics were calculated for patient
cruited from the pediatric emergency department demographics, and for results of the NOSIC and the
and pediatric clinics at Harbor-UCLA Medical Center, criterion neuropsychological tests. Results of all crite-
an academic county hospital, by one investigator rion standards were transformed so that a score of 100
(PJO). A second investigator (KDY or JMB), masked to represented normal neurologic status, with an ex-
the results of the first examination, also administered pected standard deviation of 15. A composite criterion
the draft instrument to 86 of the children. Interrater score was created by combining the Bayley Scales of
reliability was evaluated using the Spearman rank Infant Development–II (BSID-II) motor and mental
correlation coefficient and descriptive statistics of raw scores for infants and toddlers \3 years old,4 the score
differences between paired scores. on the Wechsler Preschool and Primary Scale of
1036 Okada et al. d NEUROLOGIC OUTCOME SCORES
Intelligence–Revised (WPPSI-R) for 3- to 5-year-olds,7 a pre-existing neurologic diagnosis. The observed in-
and the score on the Wechsler Intelligence Scale for terrater reliability for all children was r ¼ 0.87; for
Children, Third Edition (WISC III) for 6-year-olds and children with a neurologic diagnosis, it was r ¼ 0.93.
older.6 This allowed the creation of a single criterion Numerical differences between the scores of paired
standard over the full age range in the patient popu- examinations also were examined, revealing a mean
lation. Interrater reliability and validity were quanti- difference of 0.1 (SD ¼ 4.3). Although no formal
fied using the Spearman rank correlation coefficient validity assessment was performed, it was noted that
and by describing the differences between paired the neurologically normal children had a mean score
measurements. Bootstrap methods using 2,500 resam- of 99 (range, 84 to 100, SD ¼ 2.8), whereas those with
ples were used to calculate confidence intervals (CIs) a pre-existing neurologic condition had a mean score
around the Spearman rank correlation coefficients.15 of 79 (range, 30 to 100, SD ¼ 22).
Institutional Review Board Approval. The Institu- Results from the Revised Instrument. The ages of
tional Review Boards at the Harbor-UCLA Research the 157 children tested with the revised instrument
and Education Institute and at the University of Texas ranged from 2 weeks to 20 years, with a mean of 65
Southwestern School of Medicine approved this months (interquartile range [IQR] 18–109 months).
study. Written informed consent, in either English or Forty-six percent of the children were female; pre-
Spanish, was obtained from parents or legal guard- dominant ethnicities were African American (35%),
ians before the first examination. Written assent was non-Hispanic white (34%), and Hispanic (28%). Forty-
obtained from older children at Harbor-UCLA Med- seven percent (74/157) of the children had a history of
ical Center, as appropriate. prior developmental delay: 38 carried a diagnosis of
motor delay, 34 had learning disability, 21 had
language delay, 12 had cerebral palsy, and 12 had
attention-deficit disorder.
RESULTS The NOSIC scores ranged from 32 to 100, with
Results from Reliability Testing of the Draft a median score of 97 and a mean score of 90 (SD ¼ 16).
Instrument. The 100 children examined had a mean Descriptive statistics for the NOSIC scores and criterion
age of 6 years (range, 1 day to 21 years); 19 were standards are given in Table 2. Children with a history
younger than 1 year of age. Forty-eight were con- of motor delay, language delay, or cerebral palsy were
sidered neurologically without deficit, and 52 had considered neurologically abnormal (n ¼ 52).
TABLE 2. Descriptive Statistics for the Neurologic Outcome Scale for Infants and Children (NOSIC) and
Criterion Neuropsychological Test Scores
Total Neurologically Normal Neurologically Abnormal
Interrater reliability assessed by the Spearman rank by PJO, who also had neuropsychological testing with
correlation of the paired NOSIC scores in the 84 one of the criterion standards (Table 3). Overall, the
patients rated by both raters demonstrated excellent correlation between the NOSIC and criterion stand-
reliability ( r ¼ 0.77, 95% CI ¼ 0.62 to 0.88). Correlation ards was r ¼ 0.63, 95% CI ¼ 0.50 to 0.74 (Table 3,
coefficients for subgroups of patients are shown in Figure 1). Besides correlation with criterion standards,
Table 3. The mean difference between the two raters’ another indication of the validity of the NOSIC is its
scores was 0.02 (SD ¼ 2.57, range 5.6 to 11.1), and the ability to differentiate neurologically normal and
median difference was 0. abnormal children. The interquartile range (IQR) of
Despite the lack of a common criterion standard the first rater’s NOSIC scores for neurologically
across the range of ages and domains covered by the normal children did not overlap with the IQR for
NOSIC, validity was assessed by Spearman rank abnormal children (Table 2).
correlation of the NOSIC scores of 127 patients rated
DISCUSSION
The development of a practical, reliable, valid neuro-
logic outcome instrument is an important step in
forwarding research to improve the outcomes of
children with brain injury. Previous research studies
have used a variety of instruments ranging from
complex neuropsychological tests to simple outcome
scales with a few levels.8,12,16–20 Simple measures are
easy to apply and are applicable to children with
a wide range of ages and neurologic diagnoses, but
provide only broad categories of outcomes. Several
commonly used measures from the psychological
Figure 1. Each point in the scatterplot represents a patient’s literature offer finer gradations of outcome.3 These
Neurologic Outcome Scale for Infants and Children (NOSIC) measures, however, are often time-consuming, com-
score by the first or second rater, and the patient’s criterion plex, and specially trained personnel are required to
score. The intended ceiling effect of the NOSIC, by which all administer them. Also, they are often only applicable
neurologically normal children are intended to score in the 95–
100 range, is evident. The criterion standards used include to children from a narrow range of ages or with
intelligence tests, which differentiate children of average a narrow range of neurologic function and address
intelligence from those of above-average intelligence. The only one or a few of the domains of neurologic out-
NOSIC is not designed to differentiate these two groups of
come: cognitive, motor, behavioral, and function in
children, but to incorporate multiple domains (cognitive,
motor, behavioral, and activities of daily living functioning) activities of daily living. There has been a need for
into an overall outcome score. a neurologic outcome instrument that is easy to use,
1038 Okada et al. d NEUROLOGIC OUTCOME SCORES
10. Jennett B, Snoek J, Bond MR, Brooks N. Disability after severe 16. Chiaretti A, Piastra M, Pulitano S, et al. Prognostic factors and
head injury: observations on the use of the Glasgow Outcome outcome of children with severe head injury: an 8-year
Scale. J Neurol Neurosurg Psychiatry. 1981; 44:285–93. experience. Childs Nerv Syst. 2002; 18:129–36.
11. Hall K, Cope DN, Rappaport M. Glasgow Outcome Scale and 17. Donders J. Validity of two short forms of the WISC-R in
Disability Rating Scale: comparative usefulness in following children with traumatic brain injury. J Clin Psychol. 1992;
recovery in traumatic head injury. Arch Phys Med Rehabil. 48:364–70.
1985; 66:35–7. 18. Ewing-Cobbs L, Fletcher JM, Levin HS, Iovino I, Miner ME.
12. Fiser DH. Assessing the outcome of pediatric intensive care. Academic achievement and academic placement following
J Pediatr. 1992; 121:68–74. traumatic brain injury in children and adolescents: a two-year
13. Crouchman M, Rossiter L, Colaco T, Forsyth R. A practical longitudinal study. J Clin Exp Neuropsychol. 1998; 20:769–81.
outcome scale for paediatric head injury. Arch Dis Child. 2001; 19. Fay GC, Jaffe KM, Polissar NL, Liao S, Rivara JB, Martin
84:120–4. KM. Outcome of pediatric traumatic brain injury at three
14. Barlow KM, Spowart JJ, Minns RA. Early posttraumatic years: a cohort study. Arch Phys Med Rehabil. 1994; 75:
seizures in non-accidental head injury: relation to outcome. 733–41.
Dev Med Child Neurol. 2000; 42:591–4. 20. Gausche M, Lewis RJ, Stratton SJ, et al. Effect of out-of-hospital
15. Jackknife and Bootstrap Analyses. SAS Institute, Inc. June 22, pediatric endotracheal intubation on survival and neurological
2000. Available at: http://ftp.sas.com/techsup/download/ outcome: a controlled clinical trial. JAMA. 2000; 283:
stat/jackboot.html. Accessed April 23, 2003. 783–90.