PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006 1466

CONSEQUENTIAL ANALYSIS OF INFORMATION SYSTEM

CRITICALITY IN A HEALTHCARE ORGANIZATION
Perry SJ1, Wears RL1,2, Chozos N3, Johnson CW3, Smith KF4
1
University of Florida, Jacksonville, FL
2
Clinical Safety Research Unit – Imperial College, London
3
University of Glasgow, Scotland
4
Shands Jacksonville Medical Center, Jacksonville, Florida

Implementation of information technology (IT) in healthcare has increased with little

attention paid to the consequences of system failures. This qualitative study assesses the
organizational understanding of IT vulnerabilities, the potential consequences of failure
and system recovery capabilities within a large healthcare facility. Fifty nine percent
(59%) of identified software applications were rated mission critical by participants, 46%
were medium impact and 1 application was a non-factor. Downtime procedures were in
place for only 39% of applications with 30% of those deemed “mission-critical” lacking
downtime procedures. Expected recovery time objectives (RTO) and recovery point
objectives (RPO) for users were not consistent with those projected by the IT department.
A sub-analysis of the emergency department showed a high percentage of mission critical
software but only 36% had downtime procedures. Continued inattention to the risks and
hazards associated with widely disseminated IT within healthcare represents a continuing
and little discussed vulnerability.

INTRODUCTION METHODS

Advanced information technology has been
gradually increasing its presence in the clinical
work areas of healthcare. Often the introduction of
IT functionality begins as an optional ‘luxury’ that
will enhance or improve some aspect of clinical
care or information management, and therein is not
viewed as mission-critical. The new IT gradually
becomes a necessity, because the old systems have
withered, are not maintained, or get in the way of
current IT-based systems. The result is that
mission-critical (and sometimes, life- or limb-
critical) functionality is in use with little or no
formal consideration by it users or the organization
of its physical or logical security, its reliability, or
the consequences of failure - unavailability,
unreliability (eg, data corruption), or data loss.
The objective of this paper was to use
qualitative research methods to assess
organizational understanding of IT vulnerabilities,
the potential consequences of failure and system
recovery capabilities in a large healthcare facility.
Downloaded from pro.sagepub.com at University of Exeter on August 14, 2015
Setting. An urban 653 bed teaching hospital
that is part of an 8 hospital network. This medical
facility has over 300,000 patient visits annually and
is staffed by 4500 employees and clinicians.
Data collection. A data criticality survey was
administered to units and services within the
hospital in conjunction with the ITS department of
the study site. Individual interviews were
conducted with representative personnel using
standard questions to acquire more detail.
Respondents were asked to assess the following: 1)
mission criticality (on a 10-point Likert scale) of
the software currently being utilized in their areas,
2) the recovery time objective (RTO), defined as
the time goal for the reestablishment of IT
functions in the event of a system failure,
3) recovery point objective (RPO), defined as the
period of time for which work may be lost within
the software program once the application is
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006
restored and 4) documentation of down-time
procedures.
Additional data were obtained in the ED by
interviews with selected users to assess their
perceptions of consequences of failure as they
relate to the clinical, research, and educational
missions, and to critical administrative functions.
The types of system failure were not specified
to the participants (eg, unavailability, inaccuracy,
unreliability, data loss). To obtain the point of
view end users, application layer functionality
rather than hardware and software subcomponents
were the focus of this project.
All surveys were anonymous and no subject
identifiers were recorded in the interviews.
Data analysis. Simple descriptive statistics
were used to summarize the data. For ease of
summarization, the 10-point criticality scale was
collapsed into three categories: 1-2 mission critical,
3-8 medium impact, 9-10 non-factor.
RESULTS
Surveys were distributed to 25 units and
services within the hospital with 11 units
responding (44%). One hundred forty-eight (148)
servers were identified to be in use within by the
facility per survey results and ITS logs with total of
130 software applications in operation. Eighty (80)
of these applications were supported by hospital
IT, with the remaining 50 maintained by individual
departments or outside sources.
Data criticality ratings. Eighty-seven (87) of
the 130 identified software applications (67%)
were rated by participants on its importance to
departmental or area operations and services.
Forty-six (46) of 87 applications rated were
designated as “mission critical” (53%), 40 as
medium impact (46%) and only 1 application as a
non-factor. The remaining 43 applications received
no rating from participants.
Downtime procedures. Two hundred and forty-
four (244) responses were received when asked
whether individual units maintained documented
downtime procedures for each of the applications
utilized in their area. Fifty-nine percent ( 59%)
Downloaded from pro.sagepub.com at University of Exeter on August 14, 2015
1467
reported that they maintained downtime procedures
for some or all of the applications used by their
service area while 39% reported that they did not
any such procedures in place. Two percent (2%)
did not know if downtime procedures existed or
did not answer. Of those who stated no down time
procedure existed, 30% of their applications had
been designated as mission-critical (rating 1 or 2).
Recovery Time Objective (RTO). Assessment
of Recovery Time Objective ranged from 1 - 168
hrs with a median of 11 hrs. Significant
divergence was noted between subjects expected
RTO and the current capability for recovery
projected by the facility’s IT department for the
applications it supported (range 1.5 - 82 hrs,
median= 17 hrs).
Recovery Point objective (RPO). Responses to
the question of what Recovery Point Objective
would be acceptable for data loss ranged from 0-
168 hrs, median =1 hr. A difference was found
between the expected RPO of the participants and
the projected period for data loss from the IT
department for the applications it supported (range:
0-24 hrs, median=16 hrs).
Emergency Department (ED) sub-analysis
Fifteen key ED personnel were surveyed and/or
interviewed regarding IT applications used in the
department Twenty-two (22) separate systems were
identified in the ED with 14 rated mission-critical
(64%), 8 rated medium impact (36%), and 0 rated
non-critical. Downtime procedures were
identified for only 6 applications (27%), with most
respondents only vaguely familiar with those
identified. (e.g, “I know we start writing on this
form but I do not know where to get them”).
DISCUSSION
The growing dependence on IT-based tools in
clinical work has been insidious and relatively
untracked inside healthcare organizations. Little
attention has been directed to the vulnerabilities
that are growing in parallel with this increasing
dependence. There are numerous publications
within main stream medical journals looking at
discrete IT based tools for “sexy” topics such as
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 50th ANNUAL MEETING—2006
medication safety and computerized physician
order entry (CPOE) but only a handful that
question the widespread unexpected hazards
associated with use of IT systems in general.(Ash
2004, Wears and Berg, 2005)
This failure to attend to the risks of IT is due in
part to the misconception that hazards and failures
in healthcare will manifest in obvious and easily
discernible ways, such as with a wrong site
surgery. As a result, there is even less attention
being paid to the possibility that failures can
become manifest at some distance in time and
space from the initiating fault, or that faults may
arise from the unexpected interactions of normally
functioning components The risk of operating
mission-critical applications on commercial off-
the-shelf (COTS) hardware and software is also
being largely overlooked.
Healthcare organizations remain unaware of
these risks due to their lack of experience with the
majority of these hazards. Even vicarious
experience is scant, as those who have experienced
significant failures seldom share their experience
publicly. Previous work by these authors is one of
a handful to describe a system wide IT failure from
a still unidentified fault in the software of an
automated medication dispensing system.(Wears
and Perry, 2005) Additionally, there is limited
understanding of the highly complex domain of IT
and its unique hazards by organizational leaders
and end-users rooted a naïve faith that the
problems are all under control.(Nemeth, 2005)
Continued inattention to the risks and hazards
associated with widely disseminated IT within
healthcare will undermine recognition of failure
and the ability to cope with it.
These days we adopt innovations in large
number, and put them to extensive use, faster than
we can even hope to know their
consequences…which tragically removes our
ability to control the course of events….
Patrick Lagadec, Major Technology Risk(1982)
CONCLUSION
Despite recognition of a large number of
mission-critical IT applications, a significant
Downloaded from pro.sagepub.com at University of Exeter on August 14, 2015
1468
proportion (~60%) lack documented downtime
procedures in case of system failure. The
expectations of users for recovery from system
failure and subsequent data recovery are
incongruent with the current capabilities of the
system. Inattentiveness to these issues by
healthcare organizations and users decreases their
ability to cope or maintain patient safety when
system failure occurs.
LIMITATIONS
The use of survey methodology in this project
limits its utility in several ways. Department chairs
and unit directors assigned survey completion to
the person most IT knowledgeable in their area, but
respondents varied a great deal in their level of
sophistication and understanding. In addition,
respondents were asked to estimate the effect of
system outage on unit operations, but their ability
to envision such conditions and how they might
affect operations in unknown. Telephone, cell
phone and pager systems were not included in this
study because they were being reconfigured at that
time. Mission criticality was not consistently
defined but was locally interpreted. Finally, the
effectiveness of down time procedures was not
assessed, only their existence.
REFERENCES
Ash JS, Berg M, Coiera E. Some unintended
consequences of information technology in health care: the
nature of patient care information system-related errors. J Am
Med Inform Assoc 2004; 11(2):104-12.
Lagadec, Patrick. Major Technological Risk: Assessment
of Industrial Disasters. 1982.
Nemeth C, Nunnally M, O'Connor M, et al. Getting to
the point: developing IT for the sharp end of healthcare.
Journal of Biomedical Informatics 2005;38(1):18-25.
Wears RL, Perry SJ, Cook RI. The role of automation in
complex system failures. J Patient Safety. 2005:1(1):56-61.
Wears RL, Berg M. Computer Technology and Clinical
Work: Still Waiting for Godot. JAMA 2005;293(10):1261-63.