Research

JAMA Pediatrics | Original Investigation | CARING FOR THE CRITICALLY ILL PATIENT

Association of Nurse Workload With Missed Nursing Care

in the Neonatal Intensive Care Unit
Heather L. Tubbs-Cooley, PhD, RN; Constance A. Mara, PhD; Adam C. Carle, MA, PhD; Barbara A. Mark, PhD, RN;
Rita H. Pickler, PhD, RN

Author Audio Interview

IMPORTANCE Quality improvement initiatives demonstrate the contribution of reliable Supplemental content
nursing care to gains in clinical and safety outcomes in neonatal intensive care units (NICUs);
when core care is missed, outcomes can worsen.

OBJECTIVE To evaluate the association of NICU nurse workload with missed nursing care.

DESIGN, SETTING, AND PARTICIPANTS A prospective design was used to evaluate associations
between shift-level workload of individual nurses and missed care for assigned infants from
March 1, 2013, through January 31, 2014, at a 52-bed level IV NICU in a Midwestern academic
medical center. A convenience sample of registered nurses who provided direct patient care
and completed unit orientation were enrolled. Nurses reported care during each shift for
individual infants whose clinical data were extracted from the electronic health record.
Data were analyzed from January 1, 2015, through August 13, 2018.

EXPOSURES Workload was assessed each shift with objective measures (infant-to-nurse
staffing ratio and infant acuity scores) and a subjective measure (the National Aeronautics
and Space Administration Task Load Index [NASA-TLX]).

MAIN OUTCOMES AND MEASURES Missed nursing care was measured by self-report of
omission of 11 essential care practices. Cross-classified, multilevel logistic regression models
were used to estimate associations of workload with missed care.

RESULTS A total of 136 nurses provided reports of shift-level workload and missed nursing
care for 418 infants during 332 shifts of 12 hours each. When workload variables were
modeled independently, 7 of 12 models demonstrated a significant worsening association of
increased infant-to-nurse ratio with odds of missed care (eg, nurses caring for ⱖ3 infants
were 2.51 times more likely to report missing any care during the shift [95% credible interval,
1.81-3.47]), and all 12 models demonstrated a significant worsening association of increased
NASA-TLX subjective workload ratings with odds of missed care (eg, each 5-point increase in
a nurse’s NASA-TLX rating during a shift was associated with a 34% increase in the likelihood
of missing a nursing assessment for his or her assigned infant[s] during the same shift
[95% credible interval, 1.30-1.39]). When modeling all workload variables jointly, only 4 of 12
models demonstrated significant association of staffing ratios with odds of missed care,
whereas the association with NASA-TLX ratings remained significant in all models.
Few associations of acuity scores were observed across modeling strategies.

CONCLUSIONS AND RELEVANCE The workload of NICU nurses is significantly associated with
missed nursing care, and subjective workload ratings are particularly important. Subjective
workload represents an important aspect of nurse workload that remains largely unmeasured
despite high potential for intervention.
Author Affiliations: Author
affiliations are listed at the end of this
article.
Corresponding Author: Heather L.
Tubbs-Cooley, PhD, RN, Center for
Women, Children, and Youth,
The Ohio State University College of
Nursing, 1585 Neil Ave, 338 Newton
JAMA Pediatr. 2019;173(1):44-51. doi:10.1001/jamapediatrics.2018.3619 Hall, Columbus, OH 43210
Published online November 12, 2018. (tubbscooley.1@osu.edu).

44 (Reprinted) jamapediatrics.com

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021

 Association of Nurse Workload With Missed Nursing Care in the NICU
Q
uality improvement initiatives consistently demonstrate
the positive effect of bedside nursing care on clinical and
safety outcomes in neonatal intensive care units (NICUs).
For example, high levels of nurse adherence to central line main-
tenance bundles substantially reduce neonatal bloodstream
infections,1,2 and consistent nurse adherence to precise oxygen
titration protocols reduces risk of retinopathy of prematurity.3-6
Given the importance of nursing care to infant outcomes in
NICUs, a clear understanding of the working conditions influenc-
ing nurses’ capability to deliver care in a highly reliable way is criti-
cal for sustaining improvement gains.
Nurse workload, defined as the “the amount of performance
required to carry out nursing activities”7(p464) and frequently
measured using resource-based objective metrics, such as
infant-to-nurse staffing ratios or patient acuity scores, has been
correlated with infant outcomes in NICUs,8 including risk of
hospital-acquired infection,9 adverse events,10 and in-hospital
mortality.11-13 However, explanation of the mechanisms under-
lying these associations is lacking. Does higher workload in the
form of more patients, higher patient acuity, increased cognitive
or temporal demands, or some combination of these interfere
with a nurse’s ability to deliver highly reliable care?
Missed nursing care or necessary patient care that is omit-
ted or substantially delayed14 is a commonly theorized out-
come of high nurse workload and an emerging measure of nurs-
ing care process reliability that may partly explain the effects
of workload on outcomes. Researchers have correlated nurse
workload with missed nursing care in adult populations; higher
nurse workload measured as perceived staffing adequacy,15
staffing ratio,16-19 and nursing care hours per patient day20,21
is frequently associated with increased missed care. Evalua-
tion of a relationship between nurse workload and missed care
in NICUs is minimal22 despite evidence that neonatal nurses
miss or have lapses in providing essential care.23 A 2015 study
of missed care among NICU nurses in 7 states identified oral
care for infants undergoing mechanical ventilation, atten-
dance at team rounds, and engaging parents in care as missed
most frequently among 35 essential activities, and early evi-
dence suggests that missed care in NICUs is associated with
infant outcomes, including increased time to achieve full feed-
ings and length of stay.24,25
This study’s objective was to evaluate the association of
NICU nurse workload with missed nursing care by analyzing
multilevel data from individual nurses reporting on missed
nursing care for individual infants during 332 shifts of 12 hours
each. We hypothesized that nurses exposed to higher shift-
level workload in the form of more assigned patients, higher
acuity scores of assigned patients, and/or higher subjective rat-
ings of workload on a validated scale would be significantly
more likely to report missing care for their assigned infants dur-
ing the same shift.
Methods
Study Design
We conducted a prospective study during the 11 months from
March 1, 2013, through January 31, 2014. Each quarter we ini-
jamapediatrics.com
© 2018 American Medical Association. All rights reserved.
Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021
Original Investigation Research
Key Points
Question Does the workload of neonatal intensive care unit
nurses influence the likelihood that a nurse will miss necessary
care for assigned infants?
Findings In this study of 136 nurses caring for 418 infants during
332 shifts, increased infant-to-nurse ratio during a shift was
associated with increased missed nursing care in about half of the
measured missed care items. When a measure of subjective
workload was considered, the associations of ratios were mostly
attenuated; increased subjective workload was consistently
associated with increased missed care.
Meaning Focusing exclusively on infant-to-nurse ratios to address
missed care may be limiting; nurses’ subjective workload is
typically unmeasured but has promise for tailored workload
interventions.
tiated a continuous 6-week data collection cycle for a total of
4 cycles; at the end of every shift we collected confidential
nurse reports of staffing ratio, subjective workload, and missed
care that was specific to each assigned infant. We also col-
lected infant acuity data from the electronic health record that
corresponded to each infant-specific report of missed care
based on the medical record number, date, and time. The in-
stitutional review board at the first author’s institution re-
viewed and approved the study protocol. Written informed
consent was obtained from participating nurses; a waiver of
informed consent was granted for the use of deidentified in-
fant data obtained through an honest broker. A Certificate of
Confidentiality was obtained as an added measure of protec-
tion.
Setting, Sample, and Recruitment
The study occurred in a 52-bed level IV NICU in a Midwestern
academic medical center. Registered nurses were eligible if they
had completed unit orientation, provided direct patient care
for more than 80% of their clinical effort, and were perma-
nently employed in the NICU. Lists of eligible nurses were
obtained from hospital administrators. Participants were re-
cruited via email solicitations, brochures, flyers, and informa-
tion sessions. Enrollment remained open until the fourth cycle;
our goal was to obtain as many corresponding nurse-infant
shifts of data as possible, with an enrollment target of 80% of
eligible nurses participating. Research incentives were pro-
vided to compensate participants for their time.
We included data from all infants during active cycles be-
cause the missed care measures were not condition specific
and represented standard nursing care. However, we only in-
cluded an infant’s data in an analytic model if the infant was
at risk for the missed care outcome on the particular shift as
indicated by nurse report.
Outcomes
Missed nursing care was assessed by asking nurses to report
on omission of 11 essential neonatal nursing care practices using
items developed in earlier work.24 Self-report is the prevail-
ing method for obtaining information about care that was not
(Reprinted) JAMA Pediatrics January 2019 Volume 173, Number 1 45
 Research Original Investigation
delivered and therefore is absent from clinical documenta-
tion and other objective data sources. Nurses reported on the
frequency of care missed for each infant assigned to them based
on a Likert-type scale (never, rarely, occasionally, or fre-
quently missed or not applicable). Because the frequency of
each care item varied each shift according to an individual in-
fant’s needs, we relied on nurses’ estimates of missed care fre-
quencies. Responses were dichotomized into missed or not
missed for each item because of low endorsement of higher
levels of missed care response options, and we created a global
dichotomous missed care indicator for each infant on each shift
if a nurse reported missing any of the 11 items during the shift.
Primary Exposures and Covariables
Nurse workload was assessed using objective and subjective
measures.26 Objective measures included infant-to-nurse staff-
ing ratio and infant acuity scores. Nurses reported on the maxi-
mum number of infants cared for at a single point during the
shift. Ratios were verified by a count of unique missed care
forms received each shift and by validation against adminis-
trative data sources. We categorized the staffing ratio as 1, 2,
or 3 or more infants to 1 nurse. Infant acuity scores were esti-
mated each shift by nurses as part of an electronic health re-
cord–based patient classification system for determining an
infant’s required nursing care hours per day.27 The score in-
corporates clinical indicators of nursing care intensity such as
ventilation modality, frequency and mode of feedings, num-
ber and type of infusions, and procedures. The range for each
indicator varies depending on the number and type of items
assessed, with higher scores indicative of higher nursing care
intensity.
Subjective workload, an emerging measure of ICU nurse
workload,28,29 was assessed each shift using a paper version
of the National Aeronautics and Space Administration Task
Load Index (NASA-TLX), a scale developed to measure
domains of subjective workload in high-risk industries such
as aviation30,31 and nuclear energy32 and, increasingly, in
health care.33 The NASA-TLX is primarily a measure of how
an individual experiences the situational demands of work,
including cognitive and mental demands, physical demands,
time pressure, and overall required effort to accomplish
goals. Overall workload scores were calculated as a raw sum
of scores from these 4 items; scores for each item ranged
from 1 (low) to 20 (high), and overall workload scores ranged
from 4 (low) to 80 (high).28 Covariables included nurse edu-
cational degree (bachelor’s degree or higher), nurse certifica-
tion status (holds a professional certification or not), new
graduate status (<1 year since obtaining nursing licensure),
unit census at shift start, and shift start time (7 AM or 7 PM).
Data Sources and Collection
Data collection procedures, summarized herein, are de-
scribed in detail elsewhere.34 At each shift’s end, nurses re-
ported subjective workload experienced during the course of
the shift, infant-to-nurse ratio, and missed care for assigned
infants. Shift-level infant acuity data were retrospectively
extracted from the electronic health record and merged with
missed care data. Forms were reviewed for completeness, and
46 JAMA Pediatrics January 2019 Volume 173, Number 1 (Reprinted)
© 2018 American Medical Association. All rights reserved.
Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021
Association of Nurse Workload With Missed Nursing Care in the NICU
data were entered into a secure database maintained by data
managers. Nurses who did not return expected forms were
contacted within 24 hours; data were considered missing af-
ter 48 hours. Data were deidentified before release to inves-
tigators.
Statistical Analysis
Data were analyzed from January 1, 2015, through August 13,
2018. Descriptive statistics were used to describe the sample,
shift characteristics, and frequency of missed nursing care
items. Pearson product moment and Spearman rank correla-
tions were calculated to assess baseline associations among
workload variables, and tetrachoric correlations were calcu-
lated for missed care item associations.
Modeling associations between workload and missed
care required advanced analytic techniques owing to the
data’s hierarchical complexity. Our data included multiple
nested and nonnested contexts simultaneously: infants
nested within nurses within a single shift and infants nested
within different nurses during the course of multiple shifts.
In this analysis, we focus on associations at the level of the
nurse and infant within a shift; shifts represented a sequen-
tial time variable. A logistic cross-classified multilevel model
(CCMM) was fit for each of the 11 missed care items and to
the global missed care measure. In a CCMM, the repeated
observations (shifts) simultaneously belong to 2 inconsis-
tently nested contexts,35-38 in this case, infants and nurses.
To test for effects of workload on missed care, we esti-
mated multiple CCMMs. In the first set of models, we tested
for an association of each individual workload measure with
an infant’s odds of missed care for each of the 12 missed care
outcomes. Next, we tested effects of objective workload
measures—staffing ratios and acuity scores—on each out-
come. Finally, we tested a set of models that incorporated ob-
jective and subjective workload measures as associated with
missed care outcomes. We restricted analyses to infants with
acuity scores of less than 191, the 99th percentile, because most
infants with scores above 191 were extreme outliers with in-
verted maximum ratios (eg, 1 infant receiving extracorporeal
circulation membrane oxygenation and cared for by 2 nurses).
To aid interpretation, we rescaled acuity and subjective work-
load variables to indicate change in odds for each 5-unit in-
crease in score.
We first ran iterative generalized least squares models using
the estimates as start values for the CCMMs to reduce compu-
tation time.39 Then, Bayesian estimation procedures were
implemented via Markov chain Monte Carlo methods using the
Metropolis-Hastings algorithm.40 Bayesian estimation was
favored to reduce bias in parameters and standard errors
(SEs).40-42
Odds ratios and 95% credible intervals (CrIs), which are the
confidence intervals generated using Bayesian procedures (spe-
cifically, 2.5th and 97.5th quantiles of the posterior distribu-
tions) are presented for fixed-effect parameters. Model diag-
nostics on null models showed no evidence of assumption
violations, outliers, or influential points. The false discovery
rate was addressed using the Benjamini-Yekutieli procedure
for correlated data.43 Analyses were conducted using MLwiN
jamapediatrics.com
 Association of Nurse Workload With Missed Nursing Care in the NICU Original Investigation Research

Table 1. Characteristics of Nurse Participants Table 2. Workload Characteristics

Characteristic Data (N = 136) Summary Statistics Data

Baccalaureate degree or higher, No. (%) 114 (83.8) Infant acuity score
Specialty certification, No. (%) 44 (32.4) Mean (SD) 62.51 (56.51)
Primary shift, No. (%) Median (IQR) 49 (36-75)
Day 49 (36.0) NASA-TLX overall workload scorea
Evening 2 (1.5) Mean (SD) 45.54 (19.21)
Night 45 (33.1) Median (IQR) 45 (29-60)
Rotating 40 (29.4)
Maximum ratio of infants to nurse, No. (%) of shifts
Nursing experience, mean (SD), y 9.0 (9.5)
1:1 1022 (9.8)
Tenure in current NICU, mean (SD), y 6.4 (7.6)
2:1 8730 (83.5)
Weekly hours worked, mean (SD) 32.4 (6.2)
3:1 675 (6.4)
No. of shifts of data contributed during study, mean (SD) 41.8 (17.3)
4:1 25 (0.2)
No. of unique infants cared for during study, mean (SD) 39.9 (15.8)
Correlation coefficient (P value)
Abbreviation: NICU, neonatal intensive care unit. Staffing ratio and acuity score −0.32 (<.001)
Staffing ratio and NASA-TLX score 0.19 (<.001)
multilevel modeling software (version 2.30) within Stata Acuity score and NASA-TLX score 0.19 (<.001)
(version 15; StataCorp), and α was corrected to .004.41 Acuity score by maximum staffing ratio, mean/medianb
1 137.0/104
2 69.8/62

Results ≥3 62.7/54
NASA-TLX score by maximum staffing ratio, mean/median
Descriptive Findings
1 38.3/38
Of 202 eligible nurses, 136 (67.3%) reported on the care of 418
2 44.5/44
infants during 332 shifts of 12 hours each. A mean of 17 nurses
reported on care of 32 infants for each shift. All participants ≥3 60.0/65

submitted more than 93% of expected surveys, resulting in less
than 1% missing survey data. In total, 10 428 nurse-infant shifts
of workload and missed care data were available for analysis.
Characteristics of nurse participants are detailed in Table 1. No
significant differences in key characteristics were found be-
tween eligible nurses who did and did not enroll (n = 66) with
the exception of experience; enrolled nurses had a mean (SE)
of 3.4 (1.3) fewer years of nursing experience compared with
nonenrolled peers (P = .01).
The mean infant-to-nurse ratio on a shift was nearly 2:1
and ranged from 1 to 4 infants per nurse (Table 2); most shifts
were characterized by a 2:1 ratio. Infant acuity scores ranged
from a low of 7 to a high of 800 for infants receiving extracor-
poreal circulation membrane oxygenation; the median score
was 49, and the 99th percentile was 191 (the cut point for in-
clusion in models). The mean NASA-TLX subjective work-
load rating per shift was 46 (range, 4-80; median, 45). Corre-
lations among workload variables were low to moderate and
statistically significant, including staffing ratio and acuity score
(−0.32; P < .001), staffing ratio and NASA-TLX score (0.19;
P < .001), and acuity score and NASA-TLX score (0.19; P < .001).
Missed care was reported on 326 (98.2%) of the 332 shifts
and in 2502 (24.0%) of 10 428 corresponding nurse-infant shifts
of missed care data. All missed care items were significantly
correlated (eTable in the Supplement). When nurses reported
missing care for an infant during a shift, the amount of appli-
cable care missed ranged from 9% to 100%, with a median
value of 22% and 90th percentile of 50%. Frequency distri-
butions of the types of care missed by nurses are shown in
Table 3. When care was reported as missed, it was most often
Abbreviations: IQR, interquartile range; NASA-TLX, National Aeronautics
and Space Administration Task Load Index.
a
Scores range from 4 (low) to 80 (high).
b
Scores range from 7 to 191, with higher scores indicating higher nursing care
intensity.
missed rarely or occasionally. Overall, nurses most fre-
quently missed hourly checks of intravenous line sites (1066
[20.4%] of applicable shifts) and adherence to the central line–
associated bloodstream infection prevention bundle (695
[15.5%] of applicable shifts). Nurses least frequently reported
missing standard safety checks of alarms and equipment (188
[1.8%] of applicable shifts).
Workload and Missed Care
Table 4 displays effects of workload measures on missed care
items when modeled separately. Seven of 12 ratio models dem-
onstrated a statistically significant worsening effect of an in-
creased infant-to-nurse ratio on the odds of missed care, with
effects most pronounced when nurses cared for 3 or more in-
fants during a shift compared with a 1:1 assignment (eg, nurses
caring for ≥3 infants were 2.51 times more likely to report miss-
ing any care during the shift [95% CrI, 1.81-3.47]). Small in-
creases in acuity scores were significantly associated with in-
creased odds of missed care in 3 of 12 models (any missed care,
1.06 [95% CrI, 1.04-1.07]; missed parent involvement, 1.04
[95% CrI, 1.02-1.07]; and missed verification of 6 rights dur-
ing medication administration (6 rights), 1.03 [95% CrI,
1.10-1.06]). All 12 NASA-TLX models indicated significant wors-
ening effects of increased subjective workload ratings on the

jamapediatrics.com (Reprinted) JAMA Pediatrics January 2019 Volume 173, Number 1 47

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021

 Research Original Investigation Association of Nurse Workload With Missed Nursing Care in the NICU

Table 3. Distributions of Missed Care When Item Was Applicable to an Infant's Care
No. of
Applicable
Nurse- Response, No. (%) of Shifts
Infant Occasionally Frequently
Missed Care Item Shifts Rarely Missed Missed Missed Missed
Hourly IV site assessment 5224 869 (16.6) 173 (3.3) 24 (0.5) 1066 (20.4)
Adherence to central venous catheter 4476 582 (13.0) 95 (2.1) 18 (0.4) 695 (15.5)
infection prevention bundle
Oral feedings 4278 240 (5.6) 57 (1.3) 23 (0.5) 320 (7.5)
Patient assessment 10 397 643 (6.2) 78 (0.8) 30 (0.3) 751 (7.2)
Obtain laboratory samples 2838 163 (5.7) 28 (1.0) 8 (0.3) 199 (7.0)
Parent involvement 5662 303 (5.4) 54 (1.0) 22 (0.4) 379 (6.7)
Adherence to ventilator-associated 2978 131 (4.4) 20 (0.7) 8 (0.3) 159 (5.3)
respiratory infection prevention bundle
Thorough handoff 10 301 303 (2.9) 19 (0.2) 24 (0.2) 346 (3.4)
Verification of 6 rights during 8564 240 (2.8) 20 (0.2) 19 (0.2) 279 (3.2)
medication administration
Double-check of high-risk medications 3090 88 (2.8) 7 (0.2) 5 (0.2) 100 (3.2)
Standard safety checks of equipment 10 351 141 (1.4) 23 (0.2) 25 (0.2) 188 (1.8)
and alarms Abbreviation: IV, intravenous.

Table 4. Effects of Workload Variables on Missed Care Outcomes When Modeled Separately

Workload Measures
Staffing Ratio
No. of
Infants per Infant Acuity Score, NASA-TLX Rating,
Outcome Nursea OR (95% CrI)b OR (95% CrI)b,c OR (95% CrI)b,c
Any missed care 2:1 1.23 (0.97-1.57)
1.06 (1.04-1.07)d 1.22 (1.19-1.24)d
≥3:1 2.51 (1.81-3.47)d
Missed hourly IV site assessment 2:1 2.98 (2.17-4.02)d
0.99 (0.97-1.01) 1.22 (1.18-1.25)d
≥3:1 7.25 (4.16-11.84)d
Missed central venous catheter 2:1 2.44 (1.60-3.41)d
infection prevention bundle 0.99 (0.97-1.01) 1.21 (1.16-1.25)d
≥3:1 6.39 (3.31-10.87)d
Missed oral feeding 2:1 2.06 (0.71-5.12)
1.05 (1.01-1.09) 1.21 (1.16-1.27)d
≥3:1 4.58 (1.48-11.70)d
Missed patient assessment 2:1 2.65 (1.65-4.23)d
1.01 (0.99-1.03) 1.34 (1.30-1.39)d
≥3:1 7.15 (4.05-12.21)d
Missed obtaining laboratory 2:1 1.22 (0.73-1.98)
samples 1.00 (0.97-1.02) 1.16 (1.10-1.22)d
≥3:1 1.37 (0.45-3.07)
Missed parent involvement 2:1 0.77 (0.47-1.21)
1.04 (1.02-1.07)d 1.16 (1.12-1.21)d
≥3:1 1.61 (0.83-2.90) Abbreviations: IV, intravenous;
NASA-TLX, National Aeronautics and
Missed ventilator-associated 2:1 1.47 (0.84-2.50)
respiratory infection prevention 1.03 (1.00-1.07) 1.18 (1.10-1.26)d Space Administration Task Load
bundle ≥3:1 3.51 (1.11-8.22) Index; OR, odds ratio; 95% CrI,
Missed handoff 2:1 1.07 (0.64-1.69) 95% credible interval.
1.00 (0.98-1.03) 1.23 (1.18-1.29)d a
Reference category was 1 infant per
≥3:1 3.59 (1.85-6.47)d
nurse (1:1 ratio).
Missed 6 rights during 2:1 1.55 (0.93-2.75) b
medication administration 1.03 (1.01-1.06)d 1.23 (1.17-1.29)d Each model controlled for nurse
≥3:1 1.70 (0.69-3.53) education, certification, experience,
Missed double-check of high-risk 2:1 0.91 (0.50-1.58) unit census, and shift start time.
medications 1.03 (0.99-1.06) 1.13 (1.04-1.21)d c
≥3:1 0.83 (0.08-2.79) Rescaled to indicate change in odds
per 5-unit increase in score.
Missed standard safety checks 2:1 1.75 (0.83-3.71)
of equipment and alarms 1.00 (0.98-1.03) 1.20 (1.14-1.26)d d
P < .004, α corrected for false
≥3:1 4.26 (1.65-9.71)d
discovery rate.

odds of missed care (eg, each 5-point increase in a nurse’s
NASA-TLX rating during a shift was associated with a 34% in-
crease in the likelihood of missing a nursing assessment for his
or her assigned infant[s] during the same shift [95% CrI,
1.30-1.39]) (Table 4).
Twelve models estimating joint effects of objective work-
load measures—staffing ratios and acuity scores—on the odds
of missed care are highlighted in Table 5. The pattern of re-
sults was generally consistent with independent modeling,
with a significant worsening effect of increased ratio in 9 of

48 JAMA Pediatrics January 2019 Volume 173, Number 1 (Reprinted) jamapediatrics.com

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021

 Association of Nurse Workload With Missed Nursing Care in the NICU Original Investigation Research

Table 5. Effects of Workload Variables on Missed Care Outcomes

Objective Workload Variables All Workload Variables

Staffing Ratio Infant Acuity Staffing Ratio Infant Acuity
No. of Infants Score, OR (95% No. of Infants Score, OR NASA-TLX Rating,
Outcome per Nursea OR (95% CrI)b CrI)b,c per Nursea OR (95% CrI) b
(95% CrI)b,c OR (95% CrI)b,c
Any missed care 2:1 2.01 (1.57-2.55)d 1.07 2:1 1.30 (1.04-1.66) 1.05
(1.06-1.09)d 1.20 (1.17-1.23)d
≥3:1 4.53 (3.15-6.22)d ≥3:1 1.66 (1.18-2.30)d (1.04-1.06)d
Missed hourly IV 2:1 3.36 (2.34-4.64)d 1.02 (1.00-1.03) 2:1 2.29 (1.55-3.17)d
site assessment 0.99 (0.98-1.02) 1.20 (1.16-1.24)d
≥3:1 8.41 (4.67-13.78)d ≥3:1 3.28 (1.76-5.54)d
Missed central 2:1 2.67 (1.76-3.88)d 1.01 (0.99-1.03) 2:1 1.82 (1.22-2.66)d
venous catheter
≥3:1 6.99 (3.54-12.35)d ≥3:1 2.79 (1.39-4.93)d 0.99 (0.97-1.01) 1.19 (1.14-1.24)d
infection
prevention bundle
Missed oral 2:1 2.42 (0.75-7.69) 1.06 2:1 1.06 (0.39-2.54)
feeding (1.02-1.10)d 1.04 (1.01-1.08) 1.20 (1.14-1.26)d
≥3:1 5.66 (1.60-17.55)d ≥3:1 1.51 (0.47-4.08)
Missed patient 2:1 3.54 (2.27-5.52)d 1.03 2:1 2.02 (1.23-3.18)d
assessment d (1.02-1.05)d 1.00 (0.98-1.02) 1.34 (1.29-1.38)d
≥3:1 10.04 (5.65-16.79) ≥3:1 2.70 (1.47-4.67)d
Missed obtaining 2:1 1.27 (0.75-2.14) 1.00 (0.97-1.03) 2:1 0.88 (0.45-1.56)
laboratory 0.98 (0.94-1.01) 1.18 (1.12-1.26)d
samples ≥3:1 1.46 (0.47-3.40) ≥3:1 0.69 (0.20-1.60)
Missed parent 2:1 1.10 (0.68-1.80) 1.05 2:1 0.77 (0.44-1.25) 1.03
involvement (1.02-1.08)d 1.15 (1.10-1.20)d
≥3:1 2.53 (1.23-4.71) ≥3:1 1.20 (0.53-2.32) (1.01-1.06)d
Missed 2:1 1.93 (1.03-3.36) 1.05 (1.01-1.09) 2:1 1.42 (0.71-2.55)
ventilator-
associated ≥3:1 4.87 (1.45-11.92)d ≥3:1 2.56 (0.71-6.38)
1.03 (0.99-1.06) 1.16 (1.09-1.24)d
respiratory
infection
prevention bundle
Missed handoff 2:1 1.15 (0.63-1.99) 1.01 (0.98-1.04) 2:1 0.89 (0.50-1.50)
0.99 (0.97-1.02) 1.22 (1.15-1.27)d
≥3:1 3.96 (1.90-7.63)d ≥3:1 1.70 (0.79-3.24)
Missed 6 rights 2:1 2.38 (1.31-4.02)d 1.05 2:1 1.50 (0.84-2.53)
during medication (1.02-1.07)d 1.02 (1.00-1.05) 1.23 (1.18-1.28)d
administration ≥3:1 2.88 (1.19-5.65) ≥3:1 1.06 (0.41-2.22)
Missed 2:1 1.04 (0.54-1.78) 1.03 (0.99-1.07) 2:1 0.86 (0.40-1.63)
double-check
≥3:1 1.02 (0.09-3.59) ≥3:1 0.68 (0.06-2.41) 1.01 (0.97-1.05) 1.13 (1.05-1.23)d
of high-risk
medication
Missed standard 2:1 1.94 (0.95-3.70) 1.02 (0.98-1.05) 2:1 1.18 (0.61-2.21)
safety checks
≥3:1 5.10 (2.07-10.87)d ≥3:1 1.80 (0.70-3.86) 0.99 (0.96-1.02) 1.19 (1.13-1.26)d
of equipment
and alarms
Abbreviations: IV, intravenous; NASA-TLX, National Aeronautics and Space unit census, and shift start time.
Administration Task Load Index; OR, odds ratio; 95% CrI, 95% credible interval. c
Rescaled to indicate change in odds per 5-unit increase in score.
a
Reference category was 1 infant per nurse (1:1 ratio). d
P < .004, α corrected for false discovery rate.
b
Each model controlled for nurse education, certification, experience,

12 models and significant effects of small increases in acuity
in 5 models, including any missed care (1.07; 95% CrI,
1.06-1.09), missed oral feeding (1.06; 95% CrI, 1.02-1.10), missed
patient assessment (1.03; 95% CrI, 1.02-1.05), missed parent
involvement (1.05; 95% CrI, 1.02-1.08), and missed verifica-
tion of 6 rights of medication administration (1.05; 95% CrI,
1.02-1.07). Compared with models in Table 4, effect sizes and
significance levels of ratios were increased when modeled with
acuity scores.
Full models estimating associations between objective and
subjective workload variables and the odds of missed care are
juxtaposed in Table 5. Four of 12 ratio models demonstrated a
significant worsening effect of increased ratio on the odds of
missed care during a shift, particularly for a ratio of at least 3:1
compared with 1:1; all ratio effect sizes were substantially de-
creased and CrIs were narrowed. Effects of subjective work-
load ratings were consistent with independent modeling and
demonstrated statistically significant worsening effects of in-
creased ratings on the occurrence of missed care for all items.
For example, each 5-point increase in a nurse’s NASA-TLX rat-
ing during a shift remained associated with a 34% increase in
the likelihood of missing a nursing assessment for his or her
assigned infant(s) during the same shift (95% CrI, 1.29-1.38).
Few meaningful effects of acuity were observed, and post hoc
testing for nonlinear acuity effects did not change results.
Discussion
Consistent with our hypothesis, we found a statistically sig-
nificant association between nurse workload and the odds of
missed nursing care for individual infants, although the ef-
fects varied across workload measures and modeling strate-
gies. Staffing ratios appeared to be strongly associated with

jamapediatrics.com (Reprinted) JAMA Pediatrics January 2019 Volume 173, Number 1 49

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021

 Research Original Investigation Association of Nurse Workload With Missed Nursing Care in the NICU

some missed nursing care items when modeled alone, and ra-
tio effects were amplified when modeled with acuity. How-
ever, when modeled with nurses’ subjective workload rat-
ings, the association between staffing ratios and missed nursing
care was mostly attenuated. Subjective workload ratings were
repeatedly and significantly associated with all missed nurs-
ing care items, irrespective of modeling strategies, and dem-
onstrated consistency across models. We observed few mean-
ingful associations between acuity scores and missed nursing
care in any modeling approach.
Like many studies of missed care,16-19 we demonstrated a
significant association between increased ratios and in-
creased odds of missed care, but this association was stron-
gest when subjective workload was not included in the mod-
els. One potential explanation for this pattern is theoretical:
subjective workload mediates, or explains, the effects of ra-
tios on missed care. We did not examine mediation because
formal tests in CCMMs are not yet developed, but partial me-
diation is possible given the low but significant correlation we
found between workload measures. Subjective workload may
be farther along the causal pathway between staffing ratios and
missed care; how one experiences workload may directly in-
fluence actions and response.43 Although this relationship
requires clarification, the literature offers insight on the in-
fluence of other system factors on nurses’ subjective ratings.
In a mixed-methods study of workload challenges associated
with falls prevention efforts in adult inpatient units, Lopez
and colleagues 44 found that prevention efforts were not
substantially affected by staffing ratios but were associated
with NASA-TLX ratings, which were correlated with physical
environment layout and work processes. Moving forward, rig-
orous evaluation of the causal pathway between nurse work-
load and missed care will be important for tailoring interven-
tions to varying unit and shift contexts.
The changing results across modeling strategies under-
score the importance of measuring multiple domains of work-
load in the same analysis. Most NICU workload studies lack ad-
justment for time-varying acuity, which in this analysis
strengthened staffing ratio associations when these variables
were modeled jointly. In addition, some missed care items ap-
pear to be differentially sensitive to workload measures. For
example, ratio associations vary across models for some items
but are consistently high and statistically significant for rou-
tine assessment, hourly intravenous line checks, and adher-
ence to central line infection prevention protocols, possibly be-
cause these care needs are scheduled and time sensitive. In
these instances, a 3:1 compared with a 1:1 infant-to-nurse ra-
tio substantially increases the risk of missed care.
Strengths and Limitations
Study strengths include high rates of enrollment and re-
sponse, repeated measures of workload and missed care, use
of innovative techniques to model data as closely to its com-
plex hierarchical structure as possible, and use of a widely dis-
seminated and empirically supported measure of subjective
workload. Limitations include self-reported data from a con-
venience sample and potential social desirability bias owing
to the study topic, although the latter is likely to make our re-
sults underrepresentative of actual effects. We cannot rule out
reverse causality or the possibility that nurses’ reports of missed
care influenced their subjective workload ratings. The site was
part of an academic medical center serving a primarily surgi-
cal neonatal population and therefore may not represent con-
ditions in most NICUs. Replication in a larger and more di-
verse sample of units is needed to ascertain generalizability.
Conclusions
Nurse workload is associated with missed nursing care in
NICUs. However, interventions focused exclusively on staff-
ing ratios and/or acuity estimation may not substantially
reduce missed nursing care. Subjective workload is an
emerging aspect of nurse workload that is largely unmea-
sured yet presents new possibilities for tailored workload
interventions.

ARTICLE INFORMATION University College of Nursing, Columbus, Ohio September 30, 2015 (principal investigator,
Accepted for Publication: August 21, 2018. (Pickler). Dr Tubbs-Cooley).

Published Online: November 12, 2018.
doi:10.1001/jamapediatrics.2018.3619
Author Affiliations: Center for Women, Children,
and Youth, The Ohio State University College of
Nursing, Columbus (Tubbs-Cooley); Department of
Patient Services, Cincinnati Children’s Hospital
Medical Center, Cincinnati, Ohio (Tubbs-Cooley);
Division of Behavioral Medicine and Clinical
Psychology, Cincinnati Children’s Hospital Medical
Center, Cincinnati, Ohio (Mara); Department of
Pediatrics, College of Medicine, University of
Cincinnati, Cincinnati, Ohio (Mara, Carle); James M.
Anderson Center for Health Systems Excellence,
Cincinnati Children’s Hospital Medical Center,
Cincinnati, Ohio (Carle); Department of Psychology,
College of Arts and Sciences, University of
Cincinnati, Cincinnati, Ohio (Carle); School of
Nursing, University of North Carolina at Chapel Hill,
Chapel Hill (Mark); Martha S. Pitzer Center for
Women, Children, and Youth, The Ohio State
Author Contributions: Dr Tubbs-Cooley had full
access to all the data in the study and takes
responsibility for the integrity of the data and the
accuracy of the data analysis.
Concept and design: Tubbs-Cooley, Carle, Mark,
Pickler.
Acquisition, analysis, or interpretation of data:
All authors.
Drafting of the manuscript: All authors.
Critical revision of the manuscript for important
intellectual content: All authors.
Statistical analysis: Tubbs-Cooley, Mara, Carle.
Obtained funding: Tubbs-Cooley.
Administrative, technical, or material support: Mark,
Pickler.
Supervision: Tubbs-Cooley, Carle, Mark, Pickler.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by a
Cincinnati Children’s Hospital Research Scholar in
Patient Services Award from July 1, 2012, through
Role of the Funder/Sponsor: The sponsor had no
role in the design and conduct of the study;
collection, management, analysis, and
interpretation of the data; preparation, review, or
approval of the manuscript; and decision to submit
the manuscript for publication.
REFERENCES
1. Zachariah P, Furuya EY, Edwards J, et al.
Compliance with prevention practices and their
association with central line-associated
bloodstream infections in neonatal intensive care
units. Am J Infect Control. 2014;42(8):847-851.
doi:10.1016/j.ajic.2014.04.020
2. Schulman J, Stricof R, Stevens TP, et al; New York
State Regional Perinatal Care Centers. Statewide
NICU central-line-associated bloodstream infection
rates decline after bundles and checklists. Pediatrics.
2011;127(3):436-444. doi:10.1542/peds.2010-2873

50 JAMA Pediatrics January 2019 Volume 173, Number 1 (Reprinted) jamapediatrics.com

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021

 Association of Nurse Workload With Missed Nursing Care in the NICU
3. Lau YY, Tay YY, Shah VA, Chang P, Loh KT.
Maintaining optimal oxygen saturation in
premature infants. Perm J. 2011;15(1):e108-e113.
4. Chow LC, Wright KW, Sola A; CSMC Oxygen
Administration Study Group. Can changes in clinical
practice decrease the incidence of severe
retinopathy of prematurity in very low birth weight
infants? Pediatrics. 2003;111(2):339-345. doi:10.
1542/peds.111.2.339
5. Sink DW, Hope SA, Hagadorn JI. Nurse:patient
ratio and achievement of oxygen saturation goals in
premature infants. Arch Dis Child Fetal Neonatal Ed.
2011;96(2):F93-F98. doi:10.1136/adc.2009.178616
6. van Zanten HA, Pauws SC, Beks EC, Stenson BJ,
Lopriore E, Te Pas AB. Improving manual oxygen
titration in preterm infants by training and guideline
implementation. Eur J Pediatr. 2017;176(1):99-107.
doi:10.1007/s00431-016-2811-x
7. Morris R, MacNeela P, Scott A, Treacy P, Hyde A.
Reconsidering the conceptualization of nursing
workload: literature review. J Adv Nurs. 2007;57(5):
463-471. doi:10.1111/j.1365-2648.2006.04134.x
8. Sherenian M, Profit J, Schmidt B, et al.
Nurse-to-patient ratios and neonatal outcomes:
a brief systematic review. Neonatology. 2013;104
(3):179-183. doi:10.1159/000353458
9. Rogowski JA, Staiger D, Patrick T, Horbar J,
Kenny M, Lake ET. Nurse staffing and NICU
infection rates. JAMA Pediatr. 2013;167(5):444-450.
doi:10.1001/jamapediatrics.2013.18
10. Lamy Filho F, Silva AA, Lopes JM, Lamy ZC,
Simões VM, Dos Santos AM. Staff workload and
adverse events during mechanical ventilation in
neonatal intensive care units. J Pediatr (Rio J). 2011;
87(6):487-492. doi:10.2223/JPED.2140
11. Callaghan LA, Cartwright DW, O’Rourke P,
Davies MW. Infant to staff ratios and risk of
mortality in very low birthweight infants. Arch Dis
Child Fetal Neonatal Ed. 2003;88(2):F94-F97.
doi:10.1136/fn.88.2.F94
12. Hamilton KE, Redshaw ME, Tarnow-Mordi W.
Nurse staffing in relation to risk-adjusted mortality
in neonatal care. Arch Dis Child Fetal Neonatal Ed.
2007;92(2):F99-F103. doi:10.1136/adc.2006.102988
13. Watson SI, Arulampalam W, Petrou S, et al;
Neonatal Data Analysis Unit (NDAU) and the
Neonatal Economic, Staffing, and Clinical Outcomes
Project (NESCOP) Group. The effects of a
one-to-one nurse-to-patient ratio on the mortality
rate in neonatal intensive care: a retrospective,
longitudinal, population-based study. Arch Dis Child
Fetal Neonatal Ed. 2016;101(3):F195-F200. doi:10.
1136/archdischild-2015-309435
14. Kalisch BJ, Landstrom GL, Hinshaw AS. Missed
nursing care: a concept analysis. J Adv Nurs.
2009;65(7):1509-1517. doi:10.1111/j.1365-2648.2009.
05027.x
15. Bragadóttir H, Kalisch BJ, Tryggvadóttir GB.
Correlates and predictors of missed nursing care in
hospitals. J Clin Nurs. 2017;26(11-12):1524-1534.
doi:10.1111/jocn.13449
16. Cho SH, Kim YS, Yeon KN, You SJ, Lee ID.
Effects of increasing nurse staffing on missed
nursing care. Int Nurs Rev. 2015;62(2):267-274.
doi:10.1111/inr.12173
jamapediatrics.com
© 2018 American Medical Association. All rights reserved.
Downloaded From: https://jamanetwork.com/ by Kevin Wladimir Chanchicocha on 05/19/2021
17. Ausserhofer D, Zander B, Busse R, et al;
RN4CAST Consortium. Prevalence, patterns and
predictors of nursing care left undone in European
hospitals: results from the multicountry
cross-sectional RN4CAST study. BMJ Qual Saf.
2014;23(2):126-135. doi:10.1136/bmjqs-2013-002318
18. Ball JE, Murrells T, Rafferty AM, Morrow E,
Griffiths P. “Care left undone” during nursing shifts:
associations with workload and perceived quality of
care. BMJ Qual Saf. 2014;23(2):116-125. doi:10.1136/
bmjqs-2012-001767
19. Friese CR, Kalisch BJ, Lee KH. Patterns and
correlates of missed nursing care in inpatient
oncology units. Cancer Nurs. 2013;36(6):E51-E57.
doi:10.1097/NCC.0b013e318275f552
20. Kalisch BJ, Tschannen D, Lee KH. Do staffing
levels predict missed nursing care? Int J Qual Health
Care. 2011;23(3):302-308. doi:10.1093/intqhc/
mzr009
21. Dabney BW, Kalisch BJ. Nurse staffing levels
and patient-reported missed nursing care. J Nurs
Care Qual. 2015;30(4):306-312. doi:10.1097/NCQ.
0000000000000123
22. Lake ET, Staiger D, Edwards EM, Smith JG,
Rogowski JA. Nursing care disparities in neonatal
intensive care units. Health Serv Res. 2018;53(suppl
1):3007-3026. doi:10.1111/1475-6773.12762
23. Pillay T, Nightingale P, Owen S, Kirby D, Spencer
A. Neonatal nurse staffing and delivery of clinical care
in the SSBC Newborn Network. Arch Dis Child Fetal
Neonatal Ed. 2012;97(3):F174-F178. doi:10.1136/adc.
2011.300224
24. Tubbs-Cooley HL, Pickler RH, Younger JB,
Mark BA. A descriptive study of nurse-reported
missed care in neonatal intensive care units. J Adv
Nurs. 2015;71(4):813-824. doi:10.1111/jan.12578
25. Tubbs-Cooley HL, Pickler RH, Meinzen-Derr JK.
Missed oral feeding opportunities and preterm
infants’ time to achieve full oral feedings and
neonatal intensive care unit discharge. Am J Perinatol.
2015;32(1):1-8. doi:10.1055/s-0034-1372426
26. Carayon P, Gürses AP. A human factors
engineering conceptual framework of nursing
workload and patient safety in intensive care units.
Intensive Crit Care Nurs. 2005;21(5):284-301.
doi:10.1016/j.iccn.2004.12.003
27. Daraiseh NM, Vidonish WP, Kiessling P, Lin L.
Developing a patient classification system for a
neonatal ICU. J Nurs Adm. 2016;46(12):636-641.
doi:10.1097/NNA.0000000000000419
28. Tubbs-Cooley HL, Mara CA, Carle AC,
Gurses AP. The NASA Task Load Index as a measure
of overall workload among neonatal, paediatric and
adult intensive care nurses. Intensive Crit Care Nurs.
2018;46:64-69. doi:10.1016/j.iccn.2018.01.004
29. Hoonakker P, Carayon P, Gurses A, et al.
Measuring workload of ICU nurses with a
questionnaire survey: the NASA Task Load Index
(TLX). IIE Trans Healthc Syst Eng. 2011;1(2):131-143.
doi:10.1080/19488300.2011.609524
30. Hart SG, Staveland LE. Development of
NASA-TLX (Task Load Index): results of empirical
and theoretical research. In: Hancock PA, Meshkati
N, eds. Human Mental Workload. Amsterdam, the
Netherlands: North Holland Press; 1988. doi:10.1016/
S0166-4115(08)62386-9
(Reprinted) JAMA Pediatrics January 2019 Volume 173, Number 1
Original Investigation Research
31. Hart SG. NASA Task Load Index (NASA-TLX):
20 years later. In: Human Factors and Ergonomics
Society Annual Meeting Proceedings. Thousand
Oaks, CA: Sage Publications; 2006;50(9):
904-908.
32. Hwang SL, Yau YJ, Chen JH, et al. Predicting
work performance in nuclear power plants. Saf Sci.
2008;46(7):1115-1124. doi:10.1016/j.ssci.2007.06.005
33. Jacobson CJ Jr, Bolon S, Elder N, et al. Temporal
and subjective work demands in office-based
patient care: an exploration of the dimensions of
physician work intensity. Med Care. 2011;49(1):
52-58. doi:10.1097/MLR.0b013e3181f53542
34. Tubbs-Cooley HL, Pickler RH, Mark BA,
Carle AC. A research protocol for testing
relationships between nurse workload, missed
nursing care and neonatal outcomes: the Neonatal
Nursing Care Quality Study. J Adv Nurs. 2015;71
(3):632-641. doi:10.1111/jan.12507
35. Rasbash J, Goldstein H. Efficient analysis of
mixed hierarchical and cross-classified random
structures using a multilevel model. J Educ Behav Stat.
1994;19(4):337-350. doi:10.3102/
10769986019004337
36. Raudenbush SW. A crossed random effects
model for unbalanced data with applications in
cross-sectional and longitudinal research. J Educ
Behav Stat. 1993;18(4):321-349. doi:10.3102/
10769986018004321
37. Bhat CR. A multi-level cross-classified model for
discrete response variables. Transp Res, Part B:
Methodol. 2000;34(7):567-582. doi:10.1016/S0191-
2615(99)00038-7
38. Browne WJ, Goldstein H, Rasbash H. Multiple
membership multiple classification (MMMC)
models. Stat Model. 2001;1(2):103-124. doi:10.1177/
1471082X0100100202
39. Leckie G. Cross-classified multilevel models
using Stata: how important are schools and
neighbourhoods for students’ educational
attainment? In: Garson D, ed. Hierarchical Linear
Modeling: Guide and Applications. Thousand Oaks,
CA: SAGE Publications Ltd; 2012:311-332.
40. Browne WJ. An illustration of the use of
reparameterisation methods for improving MCMC
efficiency in crossed random effect models.
Multilevel Modelling Newsletter. 2004;16(1):13-25.
41. Leckie G, Charlton C. runmlwin: A program to
run the MLwiN multilevel modelling software from
within Stata. J Stat Softw. 2013;52(11):1-40.
42. Goldstein H, Rasbash J, Plewis I, et al. A User’s
Guide to MLwiN. 2nd ed. London, UK: Institute of
Education; 2000:286.
43. Benjamini Y, Yekutieli D. The control of the false
discovery rate in multiple testing under
dependency. Ann Stat. 2001;29(4):1165-1188.
doi:10.1214/aos/1013699998
44. Lopez KD, Gerling GJ, Cary MP, Kanak MF.
Cognitive work analysis to evaluate the problem of
patient falls in an inpatient setting. J Am Med Inform
Assoc. 2010;17(3):313-321. doi:10.1136/jamia.2009.
000422
51