See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/234082419

Human Factors and Ergonomics in Health Care and Patient Safety

Chapter · January 2012

CITATIONS READS

86 3,343

4 authors:

Tommaso Bellandi Sara Albolino

Azienda Ospedaliero Universitaria Careggi Centre for Patient Safety, Tuscan Region
52 PUBLICATIONS   305 CITATIONS    65 PUBLICATIONS   400 CITATIONS   

SEE PROFILE SEE PROFILE

Riccardo Tartaglia Sebastiano Bagnara

BSDesign National Research Council,
101 PUBLICATIONS   413 CITATIONS    102 PUBLICATIONS   1,084 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

safety and quality in surgery View project

Handover View project

All content following this page was uploaded by Riccardo Tartaglia on 29 August 2014.

The user has requested enhancement of the downloaded file.


1
Agency for Healthcare
Research and Quality, US
Department of Health and
Human Services, Rockville,
Maryland, USA
2
Clinical Risk Management and
Patient Safety Centre, Tuscany
Region, Florence, Italy
Correspondence to
Dr K Henriksen, Agency for
Healthcare Research and
Quality, 540 Gaither Road,
Rockville, MD 20850, USA;
kerm.henriksen@ahrq.hhs.gov
Accepted 20 July 2010
Qual Saf Health Care 2010;19(Suppl 3):i1ei2. doi:10.1136/qshc.2010.045849
Towards a safer healthcare system
Kerm Henriksen,1 Sara Albolino2
While the terms ergonomics and human factors have
distinctive origins (in Europe, ergonomics referred to
the laws of work or how work conditions affect
people such as leading to physiological stress or
musculoskeletal injury; in the USA, human factors
originally focused on the useresystem interface or
how people interact with equipment, workplaces
and their environment), today the two terms are
used interchangeably. A basic premise shared by the
authors of the present supplement is that the
creation of safer healthcare involves taking into
account human strengths and limitations as we
design and interact with open dynamic systems
that comprise patients and providers, processes of
care, tools and technology, the physical environ-
ment, the organisational environment and a host of
external influences. The supplement actually has its
origins in an international conference, Healthcare
Systems Ergonomics and Patient Safety (HEPS),
held in Florence in 2005.1 The idea of bridging the
worlds of the varied healthcare professions and
the ergonomics and human factors community in
the form of a conference arose from the Italian
Ergonomic Society and was spearheaded by S
Bagnara and R Tartaglia.
A second international HEPS conference was
organised collaboratively by Italian, French and
German ergonomic societies. Held in Strasbourg in
2008, it highlighted the role and experiences of
patients and providers. Representatives from over
20 countries participated. The conference gained
the endorsement of the International Ergonomics
Association and is now recognised by those with
a serious interest in patient safety and quality of
care as a much anticipated meeting at the inter-
national level. At the time of writing, a third HEPS
conference is in the organising stages and will be
held in Oviedo, Spain, on 22e24 June 2011.
The majority of articles in the supplement stem
from papers that were submitted to the second
conference. A smaller handful of the articles were
received by invitation. Collectively, the articles
represent a diverse range of issues, research
methods and settings of care, as might be expected
when bridging two communitiesdergonomics/
human factors and healthcaredthat already are
highly heterogeneous and multidisciplinary. The
guest editors launch the supplement with two
articles; the first by Henriksen is a sense-making
exercise and examines cherished notions that are
frequently regarded as whole truths, but at best are
merely partial truths in our pursuit of patient
safety; and the second by Albolino et al surveys the
opinions and experiences of Italian healthcare
workers in reporting adverse events to incident-
reporting systems. Lichtner et al then focus on the
identification of patients and their records in
a London walk-in centre, underscoring the role of
Introduction
context for both formal and informal practices. The
article shines a light on the somewhat unrecognised
yet pervasive influence that context has in shaping
our understanding as we communicate and process
patient information. It points to how a context-
dependent frameworkdreferred to as pragma-
ticsdcan help keep patients safe.
The hospital is the setting of interest for the next
set of papers. The emergency department repre-
sents the first line of care for many patients. It also
is likely the most unsafe, given variable patient
acuity, lack of information, handoffs and distrac-
tions. Using an active surveillance procedure, Hall
et al gain a better understanding of the types and
incidence of non-ideal care events in a busy, urban
emergency department. Moyer et al take us to the
neonatal intensive care unit and describe the use of
Health Care Failure Modes and Effects Analysis to
proactively assess the risk of transitioning neonates
from the neonatal intensive care unit to the
ambulatory care setting. The paediatric intensive
care setting also is known for its thin margins of
safety. In this setting, Russell et al examine the
discrepancies and their contributing factors
between medication orders for infusions as entered
into a computer order physician entry system and
the medication actually infused as indicated by
infusion pump programming. Melles et al continue
the intensive care focus by recognising the impor-
tance of quality of work life in intensive care
nursing and how it can be potentially aided by
technology that takes into account the different
roles of ICU nurses. To derive design implications,
the research team assessed the suitability of the
roles with the aid of a digital prototype during
interviews with ICU nurses from six Dutch
hospitals.
Interest in simulation has increased dramatically
in the past decade as evidenced by the next set of
papers. With a focus on insertion of central venous
catheters, Evans and Dodge address the need for
valid and reliable assessment of provider perfor-
mance, examine studies on the use of checklists and
rating scales as evaluation tools, and provide guid-
ance on checklist design. In an article with both
theoretical and practical nuances, Kneebone
provides a contextually rich view of surgical simu-
lation that selectively recreates crucial elements of
the clinical environment necessary for the suspen-
sion of disbelief while remaining cognisant of the
limited resources typically available for training.
Riley and colleagues describe a team-based in situ
approach (ie, simulation that occurs in the patient
care setting) for identifying the active and latent
breaches to defensive barriers in obstetric emergency
care. And Jeffs et al remind us that open disclosure of
medical error and how it is disclosed remains
a problematic process for healthcare professionals.
i1
 Introduction
To better understand the appropriateness of a team approach, the
perceptions of nurses and physicians participating in a team-
based error disclosure simulation intervention were examined.
Health information technology (health IT) also receives
attention in two of the articles. Given the steady migration of
medical devices and health IT into the home, Zayas-Cabán and
Dixon analyse a diverse set of ergonomic and human factors
issues encountered during the design and implementation of
home-based health IT applications. Sittig and Singh then intro-
duce an eight-dimensional model to address sociotechnical
challenges associated with the design, development, imple-
mentation, use and evaluation of health IT within complex
adaptive healthcare systems. For the final paper in the supple-
ment, Olsen and Aase conduct a comparative study of safety
climate differences in healthcare and the petroleum setting, and
suggest how the safety management systems found in the
petroleum industry could benefit healthcare.
For many countries, the first decade of the 21st century can be
characterised as a period of uncertainty and needed change in
healthcare. While much of the healthcare reform discussion has
i2
focused on cost and access issues, the authors of the supplement
(representing six countries) are united in the belief that funda-
mental to the needed change is a dedicated and sustained focus
on improving patient safety and quality of care. Contributions
from the ergonomics and human factors community are essen-
tial components of this focus. The articles that follow bear
witness to this period of needed change and raise our confidence
that in the coming decade, we will be able to generate and more
proactively shape safer and higher-quality healthcare systems.
The supplement is made possible by support from the Agency
for Healthcare Research and Quality in the US and the Tuscany
RegiondClinical Risk Management and Patient Safety Centre in
Italy.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Tartaglia R, Bagnara S, Bellandi T, et al, eds. Healthcare systems ergonomics and
patient safety. London: Taylor & Francis, 2005.
Qual Saf Health Care 2010;19(Suppl 3):i1ei2. doi:10.1136/qshc.2010.045849

Correspondence to
Dr Kerm Henriksen, Agency for
Healthcare Research and
Quality, 540 Gaither Road,
Rockville, MD 20850, USA;
kerm.henriksen@ahrq.hhs.gov
Disclaimer No official
endorsement of this paper by
the Agency for Healthcare
Research and Quality,
Department of Health and
Human Services, is intended or
should be inferred.
Accepted 5 May 2010
Qual Saf Health Care 2010;19(Suppl 3):i3ei7. doi:10.1136/qshc.2010.041293
Partial truths in the pursuit of patient safety
Kerm Henriksen
ABSTRACT
The paper explores several issues in the form of partial
truths that dominate current thinking as investigators
continue their pursuit of patient safety. Among the partial
truths examineddcast as bipolar orientationsdare
evidence-based medicine versus quality improvement,
‘knowledge in the head’ versus ‘knowledge in the world’,
sharp end versus blunt end, reporting systems versus
local knowledge, changing beliefs versus changing
behaviour and system components versus system
interdependencies. The paper provides a cautionary note
regarding the downside of creating dichotomies that
tend to assert too much. An enhanced understanding of
patient safety will likely result from rising above bipolar
orientations and valuing them as partial approaches to
a complex and dynamic problem space.
The last couple of decades have witnessed consid-
erable intellectual vigour in the pursuit of patient
safety. While still in its infancy, patient safety has
spawned a diversity of subject matter worthy of
exploration and a wealth of contrasting theoretical
and pragmatic approaches. A newcomer to the field
is likely to be simultaneously inspired yet under-
standably confused. Is it possible that two
contrasting views or approaches could both be
right, or put another way, is it possible that in
patient safety there are no whole truths, only
partial truths? As a sense-making exercise, the
purpose of the paper is to explore some of these
partial truthsdthe bipolar orientations of our
cherished views and established paradigms that
dominate our thinking, at times unwisely, as we
pursue patient safety and seek to understand
complex socio-technical clinical phenomena. The
paper is guided by a philosophical observation by
Alfred North Whitehead:
‘There are no whole truths; all truths are half-truths.
It is trying to treat them as whole truths that plays
the devil.’1
EVIDENCE-BASED MEDICINE VERSUS QUALITY
IMPROVEMENT
A central tenet of evidence-based medicine is that
clinical interventions should be based on a solid
foundation of scientific evidence. Randomised
clinical trials (RCTs) and dependent measures
directly linked to morbidity and mortality are
considered the gold standard when it comes to
experimental designs and outcomes. Like many
fields, medicine’s past is replete with erroneous
expert opinion and authoritative misinformation.
When asked to explain his low opinion of the
necessity for hand-washing, a prominent physician
in the first half of the twentieth century is known
to have proclaimed, ‘gentlemen do not carry germs
Original research
on their hands.’ Medicine has not always done the
right thing or done the right thing right. It is thus
understandable when evidence-based colleagues in
medicine cast a wary eye on the latest safety or
quality improvement initiative and, in turn, ask
‘What is the evidence for all this human factors
engineering, device evaluation, technology adop-
tion, decision support, crew resource management,
teamwork, incident reporting, root-cause analysis,
structured communication, use of checklists, safety
culture and resilience stuff?’
Such a question begs another question: what
constitutes acceptable evidence? Research and
evaluation efforts start with a clear specification of
the question or issue of interest and then generate
research techniques that form the basis for an
appropriate investigation. The research questions
drive the research design, not the other way
around. For questions of clinical efficacy, it is likely
that RCTs best address many questions; however,
other issues, such as overcoming resistance to
technology adoption or improving the design of
a confusing interface, are likely best addressed by
qualitative techniques and test and evaluation
procedures. It is one thing to demonstrate the
efficacy of a practice such as prophylactic anti-
coagulation for venous thromboembolism with
RCTs; it is another matter to ensure its effective-
ness in the clinical setting; that is, aligning the
relevant system factors to ensure that the anti-
coagulation gets to the right patients at the right
time in the right dose.2
When it comes to demonstrating effectiveness of
many quality improvement initiatives in complex
clinical settings, where there are likely to be
considerable variance in organisational contexts,
looseness in the internal integrity of the interven-
tion itself and hence replication, varying skill levels
among investigators applying the intervention, and
use of multiple process and outcome measures,
experimental designs have generally rendered
conflicting or equivocal results. In the practice of
human factors engineering, with process or product
improvement as the goal, the predominant paradigm
has been an iterative specifyedesignetesteevaluate
and redesign set of activities.3 Physical mock-ups,
rapid prototyping, computer simulations and
quasi-experimental designs where they make
sense are employed. Iterative cycles of design,
testing with representatives from the user popu-
lation, evaluative assessment and redesign occur
typically in the absence of higher experimental
designs using control groups.
Individuals who are charged with implementing
quality and safety improvement initiatives in their
healthcare settings do not have the luxury of
waiting for the last scintilla of evidence before
taking meaningful action. Uncertainty, incomplete
i3
 Original research
information, and unknown consequences are part of the fuzzy
context in which decisions are made. Should one accept the risk
of being an ‘early adopter’ of an innovation whose effects and
consequences are not fully known, or should one accept the
comfort of the ‘skeptical late majority’ and not take any action
until there is substantial evidence?4 How clear-sighted are we
about our end goals? Are we interested in questions of clinical
efficacy for which RCTs are likely to be appropriate, or in quality
improvement efforts which typically employ methods of lower
internal validity yet higher contextual sensitivity? In patient
safety, as in other applied fields, what constitutes acceptable
evidence has very little to do with a gold standard, but instead
depends on a clear understanding of the purpose and nature of
one’s goals.
‘KNOWLEDGE IN THE HEAD’ VERSUS ‘KNOWLEDGE IN THE
WORLD’
Precluding the occurrence of adverse events through purposeful
design is a time-honoured approach for enhancing system safety.
Reflecting a distain for educational methods for enhancing
safety, a common retort of early human factors practitioners in
aviation was ‘it’s easier to bend metal than twist arms.’ Some-
time later, a distinction was made between ‘knowledge in the
head’ and ‘knowledge in the world.’5 The three holes or narrow
horizontal gap at the top of your bathroom washbasin that
prevents overflow is an example of ‘knowledge in the world.’ We
could choose to design washbasins without the holes or gap, and
instead educate people about the safe way to fill washbasins, but
not many of us would choose to do this. There is nothing wrong
with ‘knowledge in the head.’ It is just that our lives can be
made easier by putting more knowledge in the world in the form
of affordances, making things visible, natural mapping and
providing clear feedback to users when tasks are successfully
executed. A useful compendium of mistake-proofing approaches,
based on the ‘knowledge in the world’ concept, has been
recently applied to the design of healthcare processes.6
For many years, medicine relied too heavily on the ‘knowledge
in the head’ model with the omniscient physician enjoying
unquestioned autonomy in determining what was best for the
patient. The tacit and pervasive assumption that physicians and
providers practise with the latest scientific evidence and are at
the top of their game at the particular moment they see us as
patients is a little naive today. Providers are like the rest of us;
they are subject to the same cognitive limitations and vary
considerably in executing the skill-based tasks they are called
upon to perform. Despite Wennberg’s earlier research on medical
practice variations that shed light on questionable surgeries and
hospitalisations,7 8 there are still many routine nursing, medical
and surgical practices that are subject to the whim of the indi-
vidual provider rather than the available evidence. The results
are inefficiencies, disappointing outcomes and unnecessary risks
to patients. While there have been an increasing number of
initiatives to put ‘knowledge in the world’ in the form of best
practice guidelines, greater standardisation of clinical procedures
and adoption of health information technology, their successful
implementation has been somewhat uneven and largely
confined to those institutions with sufficient resources and
champions to ensure success. It is noted, however, that
successful use of surgical checklists in reducing complications
and mortality has been reported in less well-resourced facilities
in addition to well-resourced facilities.9 10 Still other investiga-
tors have commented on the folly of rushing to implement new
technology and standardised procedures too quickly, running the
i4
risk of being insensitive to important contextual factors and not
fully understanding what is gained and what is lost as medical
practices are transformed.11e13
At the same time, there is no shortage of at-risk situations
that arise in the moment-to-moment delivery of healthcare for
which there is no evidence, where best-practice guidelines are
yet to be scripted (or may never be) and for which designers lack
sufficient foresight in anticipating. Because of the complexity,
uncertainty and rapidly emerging conditions in many clinical
settings, the provider becomes less of an instrument for carrying
out carefully deliberated guidelines and more of a problem solver.
In facing the unexpected, providers are left to improvise on the
spot and create safety when the margins of safety become
precarious. Rather than focussing on error and design efforts to
preclude it, resilient providers tap into the dynamic aspects of
risk management, exploring how they and their organisations
anticipate and adapt to changing conditions and recover from
system anomalies. Building on insights gleaned from complex
adaptive systems, high-reliability organisations and the actions
of resourceful providers at the point of patient care, resilience is
viewed as a critical system property, reflecting an organisation’s
adaptive capacity to bounce back in the face of continuing
pressures and challenges.14
Aiding the adaptive capacity of an organisation to bounce
back is a certain level of underlying structure and stand-
ardisation, skill automaticity and second-nature swiftness in
executing certain tasks which, in turn, free up the necessary
cognitive, technical and organisational resources to address and
mange the unexpected. A singular reliance on either a design or
resilience perspective, or a ‘knowledge-in-the-head’ or ‘knowl-
edge-in-the-world’ orientation, without taking into consider-
ation their reciprocal relationship to one another, can be
limiting.
SHARP END VERSUS BLUNT END
Patient safety and quality improvement efforts to date have
focused predominately on the sharp end or micro-system level
(emergency departments, ICUs, labour and delivery) where there
is sufficient complexity to occupy the attention of researchers
and outcomes-minded providers alike. Some of the safety and
quality gains at the micro-system level are indeed impressive,
and there is no doubt that the magnitude of the challenges will
keep providers busy for years to come. Focussing so predomi-
nantly on the micro-system level, however, has resulted in an
interesting tapestry with respect to overall quality at the insti-
tutional level. Institutionally, quality of care is frequently a hit-
or-miss affair with high-performing units coexisting and
adjoining low-performing units. There is considerable variation
in the social and organisational environment of hospital units
that is subject to the leadership style of clinical directors and
nurse managers. Significant differences have been found in
attitudes about the consequences of speaking up on sensitive
topics such as medical error.15 With some teams, errors were
openly acknowledged and discussed to decrease their chances of
happening again in the future. Other teams, however, main-
tained silence with respect to errors. Terms such as ‘cultural
censorship’ and ‘organisational silence’ have been used to
describe the duplicitous side of organisational life where unto-
ward events paradoxically are simultaneously recognised yet
neglected.16 17 Such differences at the micro-system level suggest
that teams learn different underlying assumptions or beliefs as
to the appropriate way to perceive, think and feel about sensi-
tive patient safety issues. A key influence at the local micro-
Qual Saf Health Care 2010;19(Suppl 3):i3ei7. doi:10.1136/qshc.2010.041293

system level is whether the clinical director or nurse manager is
a strong patient safety advocate. Unfortunately, patients have to
transition through the hit-or-miss quality landscape as they
undergo successive phases of the acute care cycle. The best
intensive care available can be undermined quickly if the patient
leaves the ICU with excellent prospects for recovery only to
receive substandard care on the unit floor or during discharge. In
too many hospitals, the likelihood of not experiencing a mishap or
complication throughout the care cycle is not what it should be.
Compared with sharp-end providers at the micro-system
level, institutional administrators and leaders at the blunt-end
are in a better position to do something about the problems
behind the problem. Although they may not be as close to the
clinical action as that of their sharp-end colleagues, they are in
a position to work across the separate units of care, gain a basic
understanding of the complex interdependencies and begin to
address areas of vulnerability in their institutions. Yet, too many
institutional leaders have remained aloof and silent on critical
safety and quality of care issues.17 Few data have been collected
on the practices of leaders and the role of organisational factors
in creating consistent and high-quality care across all the units in
institutions. An interesting exception is a recent study that
devised a composite scoring system using quality and safety
measures derived from patient discharge data from 79 academic
medical centres in the US for 2003e2004.18 Based on the scoring
system results, three top performing institutions and three
average-performing institutions were visited by a site visit team
that was blinded as to the institution’s performance status.
Abstraction of findings from the site visit interviews reveals
some key differences between high-performing and average-
performing institutions.
Among the high-performing institutions was a shared sense of
purpose of putting safety and quality of care for patients first.
The shared sense of purpose was sustained through announce-
ments, town meetings, performance measures and leadership
walk rounds. Executive personnel at the top-performing orga-
nisations provided a sense of urgency, were not satisfied with the
status quo and were passionate about what they were doing.
Comparisons were made not with peer organisations, but
instead between the current state and the ideal state to which
they were striving. Strategic priorities were set by a central
quality committee, while departmental chairs and unit leaders
were responsible for the improvement tactics, giving rise to
spontaneous innovation and commitment at the local unit level.
While measurement and a focus on results played key roles,
measurement was not an end in itself but a tool in support of
work redesign to improve outcomes. Likewise, technology was
used in service of redesign rather than as a replacement for
troublesome processes of care. A spirit of collaboration existing
among administrators, physicians, nurses and other staff was
noted, and the culture was such that employees were valued for
their expertise and resourcefulness.18
REPORTING SYSTEMS VERSUS LOCAL KNOWLEDGE
How well do centralised error-reporting systems serve providers
at the sharp end and form the basis for systemic safety
improvements? Or would providers be better off by exploring
more expeditious ways of pooling what they already know
implicitly, making it more explicit, and undertake systemic
improvements on the basis of that more personal, local knowl-
edge? The use of error-reporting systems is based on the premise
that before well targeted and systemic improvements can be
made, a certain amount of reporting, aggregating of incidents
Qual Saf Health Care 2010;19(Suppl 3):i3ei7. doi:10.1136/qshc.2010.041293
Original research
and reporting back of incident trends needs to occur first. The
development, implementation and maintenance of reporting
systems represent a significant, resource-intensive undertaking.
Yet, among users and supporters of these systems, an under-
reporting of errors is a common concern. Under-reporting is
viewed as undermining efforts to bring about system-level
improvements. A recently reported survey of over 1000 physi-
cians and surgeons in Missouri and Washington State in the USA
found that the respondents considered the current systems to
report and share information about errors to be inadequate, and
instead they relied on informal discussions with their
colleagues.19 Forty-five per cent did not know if a reporting
system existed in their healthcare organisation. When asked what
would increase their willingness to report, physicians wanted
the information to be kept confidential and non-discoverable;
evidence that the information would be used for system
improvements; the system to be non-punitive; the error
reporting process to take less than 2 min; and the findings and
resulting recommendations to be relevant to their local unit or
department.
With these expressed concerns, it is easy to see how a host of
important factorsdlegal, trust, technical, cultural, usability and
relevancedneed to converge favourably for reporting systems to
perform the function for which they are intended. For many
users of reporting systems, the important factors have not been
aligned favourably, leading some to question whether the
benefits realised to date justify the effort put into creating them.
With the knowledge sharing that occurs at the local level (where
physicians share their concerns and frustrations with physicians,
and nurses do likewise with one another), an alternative
approach would be to search for efficient ways of capturing
what already seems to be known locally, make it less tacit and
more open, have a way of organising, analysing and prioritising
events to a manageable subset, obtain the required leadership
backing to make the system changes and track their effects.
Such an approach has the advantage of getting closer to the
many contextual factors to which large-scale reporting systems
might be less sensitive. It bases safety improvement efforts on
the way clinical work is actually performed rather than
a prescriptive model that might be wide of the mark. A potential
disadvantage of locally generated systems (in addition to not
benefiting from or sharing findings with other systems) is that
without strong transformational leadership, they might not
have the same impact in terms of their potential to ‘rock the
organisational boat’ that a larger, centralised system might have.
Many improvement efforts, based on what is known locally,
start with good intentions and high levels of enthusiasm, only to
‘fizzle out’ a few months later. While the possibility of
combining the best of both approachesdof benefiting from the
feedback and impact of a centralised system while optimising
the use of local knowledgedmay make sense conceptually,
convincing demonstrations have been relatively few.
CHANGING BELIEFS VERSUS CHANGING BEHAVIOUR
It has been a decade since the publication of To Err is Human in
the USA and An Organization with a Memory in the UK helped to
launch the international patient safety movement.20 21
According to commentaries providing a 10-year retrospective
analysis, much more needs to be done despite some successes
and promising projects.22 23 The patient safety movement has
not been the rising tide that has floated all boats evenly. A
significant cultural change that places safety of patients as one
of its foremost underlying values has not taken root in many
i5
 Original research
healthcare facilities. For those who are ever mindful of the
recurring harm that patients experience, a critical question
remains: why is it so difficult to bring about change?
The traditional way of thinking about change is that one first
needs to change underlying attitudes and beliefs. These changed
beliefs will then serve as springboards to meaningful behavioural
change. However, changing these underlying attitudes and
beliefs is not easy to achieve. Providing a clue as to why is
Schein’s rich and comprehensive definition of culture: ‘a pattern
of basic assumptions, invented, discovered, or developed by
a given group as it learns to cope with its problems of external
adaptation and internal integration, that has worked well
enough to be considered valid, and therefore is to be taught to
new members as the correct way to perceive, think, and feel in
relation to those problems.’24 It is these deep-seated, underlying
assumptions, values and beliefs that a given group has come to
learn and share so extensively that it shapes the way they view
the world. The glib expression ‘it’s the way we do things around
here’ reflects the deeply ingrained and shared acceptance of
doing things a certain way without a moment’s reflection. To
tell a group that they need to change their culture is to tell them
that their shared underlying assumptionsdtheir learnt ways of
viewing their worlddare no longer valid. This is why cultural
change can be so threatening and anxiety-provoking when
organisations are called upon to ‘change’ their culture or ‘rein-
vent’ themselves. Truly changing an institution’s culture with
respect to patient safety is not a short-term process.
Do we really need to wait until attitudes and beliefs change
before taking meaningful action? The civil rights legislation in
the USA in the 1960s and the public smoking restrictions of the
1990s did not wait until beliefs changed before undertaking
action. Would school integration and lower lung cancer rates
have been so readily achieved without these deliberate actions?
Cognitive dissonance theory allows us to consider that it may be
easier to act your way into a new way of thinking than think
your way into a new way of acting. In other words, behavioural
change drives cultural change rather than the other way around,
as is commonly assumed. Here, the thinking is that behaving in
a manner that is inconsistent or incompatible with one’s beliefs
creates a state of dissonance, and if the instilled behaviours are
seen as irrevocable, the only recourse for reducing cognitive
dissonance is to alter one’s beliefs.25 26 New safety interventions
that instil new behaviour and that carry the force of regulatory
or institutional dictum (so that there is no going back) are likely
to stand a good chance of influencing the attitudes and beliefs
that underlie the organisation’s culture. Given the reciprocal
and intricately intertwined nature of beliefs and behaviour in
any organisational setting, it is often difficult to determine
which factor is the driving agent or the extent to which these
processes are subject to the influence other internal and external
contextual factors.
SYSTEM COMPONENTS VERSUS SYSTEM
INTERDEPENDENCIES
Efforts aimed at improving the delivery of care stand the best
chance of working when they are based not only on a sound
knowledge of the clinical processes of care, but also on an
appreciation of the interacting work system components that
comprise the clinical specialty areas. These interacting work
system components are made up of people (providers)
performing tasks using various tools and technology within
a physical environment in pursuit of organisational goals.27
Should one study the singular main effects of individual system
components or the interdependencies among these five
i6
components? When phrased in such a loaded fashion, the
author’s bias is clearly showing. Despite the amount of ink and
paper attesting to the virtue of a systems approach to patient
safety, there appears to be very little research that addresses the
interdependencies of a system’s components.
The interacting impact of the physical environment, the nature
of the tasks performed and human sensory capabilities on visual
performance illustrates the importance of knowing something
about the interdependencies of a system’s components. For
example, it is known that medication-dispensing errors among
pharmacists are lower when work-surface lights are at higher
illumination levels.28 At the same time, visual performance
depends upon the nature of the tasks performed. Performance can
suffer when providers are executing tasks that involve small
visual elements and when contrast between figure and ground is
low, as would be the case in dimly lit rooms. Furthermore, visual
performance depends on changes that take place in our eyes as we
age. As we reach 40 years and beyond, changes in the lens and
optic pathways reduce the amount of light that reaches the
retina. At the time of writing, the average age of registered nurses
in US hospitals is close to 50. The results of an earlier study on the
relationship between illumination and young and aged subjects
performing difficult (poor print quality) and easy (good print
quality) versions of the same proofreading task are quite infor-
mative.29 While increases in illumination had a positive impact on
performance, and younger subjects overall outperformed older
subjects, the difference between the two age groups was most
pronounced for the poor-quality print condition and considerably
reduced for the good-quality print condition. Older subjects
performed almost as well across the four illumination levels when
print quality conditions were good. In addition to demonstrating
the effects of illumination, the study shows how a frequently
neglected workplace feature such as the quality of written
material as can be found on medication labels and infusion pumps
can have a significant impact on the performance of older
workers, who, in turn, are making up a steadily increasing
proportion of the workforce. The lesson for designers is that
design features do not always have uniform effectsdthat the
effects of a given variable frequently depend upon a particular
level of another variabledand that it is important to know
something not just about the main effects, but how system
factors, such as lighting, the nature of the tasks performed and
capabilities of people interact in less than obvious ways.30
Systems thinking has not come easily to healthcare profes-
sionals.31 The dynamic and multiple interdependencies among
personnel, technology, work processes, and internal and external
organisational factors frequently result in consequences that are
unanticipated and unintended. Witness the difficulties experi-
enced in implementing new information technology systems.
All of a sudden, the many work process, organisational and
personnel factors that had been conveniently neglected come to
the forefront and take on immediate significance. It is not the
technology, but the failure to consider its impact on other
system components, that sinks many implementation efforts.
Rather than seek to maximise the singular effects of individual
components, healthcare systems need to optimise the relation-
ships and interactions harmoniously among their components in
order to realise safe and reliable system performance. More is not
better with respect to each component’s output. Different and
multiple ways of measuring and analysing a system’s output
and that of its components should help sharpen our fore-
sightdperhaps with the aid of simulation and modelling exer-
cisesdin anticipating some of the less recognised
interdependencies that operate in complex healthcare systems.
Qual Saf Health Care 2010;19(Suppl 3):i3ei7. doi:10.1136/qshc.2010.041293
 Original research

In the final analysis, each of the above bipolar orientations as
framed represents a somewhat overly simplified, limited way of
viewing the world. Complex system phenomena do not arrange
themselves neatly along bipolar orientations. There is danger in
creating dichotomies that tend to assert too much. The dichot-
omies are our creations and not the phenomena we study. The
goal is safety, not missionary work for particular orientations,
paradigms or tools. It is not difficult to find sets of conditions that
will amply demonstrate both the sheer folly and exalted wisdom
of any particular orientation (especially when we indulge in
selection bias and have the after-the-fact luxury of doing so). The
partial truths we come to cherish stem, in part, from our own
bounded rationality in dealing with a complex problem space and
with our eager delusions that they account for more than they
actually do. Too frequently, the consequences are over-
simplification, neglect of subtlety and interdependencies, and
short-term appeal. In brief, investigators of patient safety are
likely to make the most sense when they rise above bi-polar
orientations and value them as partial truths of a dynamic,
complex problem space in their pursuit of a fuller understanding.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Price L, ed. Dialogues of Alfred North Whitehead (Original printing, 1954). Jaffrey,
NH: David R Godine Publisher, 2001 (Reprinted).
2. Leape LL, Berwick DM, Bates DW. What practices will most improve safety?
Evidence-based medicine meets patient safety. JAMA 2002;288:501e7.
3. Henriksen K. Human factors and patient safety: Continuing challenges. In: Carayon
P, ed. Handbook of Human Factors and Ergonomics in Health Care and Patient Safety.
Hillsdale, NJ: Erlbaum, 2007:21e37.
4. Rogers EM. Diffusion of Innovations. 5th edn. New York: Free Press, 2003.
5. Norman DA. The Design of Everyday Things. New York: Doubleday, 1988.
6. Grout J. Mistake-Proofing the Design of Health Care Processes. AHRQ Publication
No. 07-0020. Rockville, MD: Agency for Healthcare Research and Quality, 2007.
7. Wennberg JE, Gittelsohn A. Small area variations in health care delivery. Science
1973;182:1102e8.
8. Wennberg JE. Dealing with medical practice variations: a proposal for action.
Health Aff 1984;3:6e32.
9. Haynes AB, Weiser TG, Berry WR, et al. A surgical checklist to reduce morbidity and
mortality in a global population. N Engl J Med 2009;360:491e9.
10. Gawande A. A Checklist Manifesto: How to Get Things Right. New York:
Metropolitan Books, 2009.
11. Berg M. Rationalizing Medical WorkdDecision-Support Techniques and Medical
Practice. Cambridge, MA: The MIT Press, 1997.
12. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry
systems in facilitating medication errors. JAMA 2005;293:1197e203.
13. Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot.
JAMA 2005;293:1261e3.
14. Hollnagel E, Woods DD, Leveson N, eds. Resilience EngineeringdConcepts and
Precepts. Aldershot, UK: Ashgate Publishing Limited, 2006.
15. Edmondson A. Learning from failure in health care: Frequent opportunities,
pervasive barriers. Qual Saf Health Care 2004;13(suppl 2):ii3e9.
16. Hart E, Hazelgrove J. Understanding the organizational context for adverse events in
the health services: the role of cultural censorship. Qual Saf Health Care
2001;10:257e62.
17. Henriksen K, Dayton E. Organizational silence and hidden threats to patient safety.
Health Serv Res 2006;41:1539e54.
18. Keroack MA, Youngberg BJ, Cerese JL, et al. Organizational factors associated with
high performance in quality and safety in academic medical centers. Acad Med
2007;82:1178e86.
19. Garbutt J, Waterman AD, Kapp JM, et al. Lost opportunities: how physicians
communicate about medical errors. Health Aff 2008;27:246e55.
20. Institute of Medicine. To Err Is Human: Building a Safer Health System.
Washington, DC: National Academy Press, 2000.
21. Department of Health. An Organization with a Memory: Report of an Expert
Working Group on Learning from Adverse Events in Nhs. Norwich, UK: The Stationery
Office, 2000.
22. Clancy CM. Ten years after to err is human. Am J Med Qual 2009;24:525e8.
23. Leape LL. Errors in medicine. Clin Chim Acta 2009;404:2e5.
24. Schein EH. Organizational culture. Am Psychol 1990;45:109e19.
25. Festinger L. A Theory of Cognitive Dissonance. Evanston, IL: Row, Peterson, 1957.
26. Brehm JW, Cohen AR. Explorations of Cognitive Dissonance. New York: Wiley,
1962.
27. Carayon P, Smith MJ. Work organization and ergonomics. Appl Ergon
2000;31:649e62.
28. Buchanan TL, Barker KN, Gibson JT, et al. Illumination and errors in dispensing. Am
J Hosp Pharm 1991;48:2137e45.
29. Smith SW, Rea MS. Proofreading under different levels of illumination. J Illum Eng
Soc 1978;8:27e52.
30. Henriksen K, Isaacson S, Sadler BL, et al. The role of the physical environment in
crossing the quality chasm. Jt Comm J Qual Patient Saf 2007;33:68e80.
31. Schyve PM. Systems thinking and patient safety. In: Henriksen K, Battles JB, Marks
ES, et al (Eds) Advances in Patient Safety: from Research to Implementation, Vol 2.
Concepts and Methodology, 1e4. AHRQ Publication No. 05-0021-2. Rockville, MD:
Agency for Healthcare Research and Quality, 2005.

Qual Saf Health Care 2010;19(Suppl 3):i3ei7. doi:10.1136/qshc.2010.041293 i7

 Original research
1
Center for Clinical Risk
Management and Patient
Safety, Florence, Italy
2
Age.na.s, National Agency for
the Regional Healthcare
Services, Rome, Italy
3
Department of Biostatistics,
Cancer Prevention and Research
Institute, Florence, Italy
Correspondence to
Dr Sara Albolino, Center for
Clinical Risk Management and
Patient Safety, Tuscany Region
Italy, Via Taddeo Alderotti 26n,
50126 Florence, Italy;
sara.albolino@regione.toscana.it
Accepted 5 May 2010
i8
Patient safety and incident reporting: survey of Italian
healthcare workers
Sara Albolino,1 Riccardo Tartaglia,1 Tommaso Bellandi,1
Anna Maria Vincenza Amicosante,2 Elisa Bianchini,3 Annibale Biggeri3
ABSTRACT
Background Incident-reporting systems (IRS) are tools
that allow front-line healthcare workers to voluntary
report adverse events and near misses. The WHO has
released guidelines that outline the basic principles on
how to design and implement successful IRS in
healthcare organisations.
Methods A written survey was administered with an
assisted self-assessment technique to a representative
sample of healthcare workers in Italian hospitals with
and without IRS. Data were collected using two different
16-item questionnaires. The questionnaires targeted two
issues: (1) workers’ experience of patient safety
incidents and (2) their expectations on incident reporting.
Results 70% of respondents confirmed involvement in
a patient safety incident, but only 40% utilised an IRS to
formally report the event. The data indicate that
information regarding patient safety incidents is not
communicated throughout the entire organisation.
Conclusions Research findings are consistent with the
available evidence on healthcare workers’ experience of
patient safety incidents.
Incident-reporting systems (IRS) are tools that
allow front-line healthcare workers to voluntarily
report adverse events and near misses. An IRS
achieves two primary purposes: (1) to provide
individual workers with a systematic methodology
to promote continuous experiential learning; and
(2) to provide organisations with a record of inci-
dents for clinical risk assessment and prevention.1 2
Even if reporting of adverse events is mandatory, it
is critical that clinicians learn from the experience.
The WHO has clearly outlined the basic princi-
ples on how to design and implement successful
IRS in healthcare organisations.3 A successful IRS is
one in which the information it yields is used to
improve patient safety.4
The following characteristics of a successful IRS
are well established in published literature: wide
consensus between professionals and managers on
the need to collect data on existing problems, time
spent to design and start-up the system, use of
well-recognised experts to manage the system and
continuous marketing activities to ‘sell’ the system
to the users.1
Each one of these characteristics has the prereq-
uisite of an active commitment of both head
clinicians and top managers in patient safety. Also,
leadership is crucial, in particular for those who are
in charge of running the daily operations of the IRS
because the IRS needs workers to provide honest
reports. Finally, an understanding of the basics of
the system’s approach is an important requirement
Qual Saf Health Care 2010;19(Suppl 3):i8ei12. doi:10.1136/qshc.2009.036376
to spread the safety culture, where an incident
report is considered a mode in which to learn from
errors and for improving the organisation rather
than a self-claim.12e15
After the incident is reported, organisations must
put in place a quantitative and qualitative analysis
of the report, then follow-up with an in-depth
investigation into the incident and finally develop,
apply and monitor an action plan to anticipate and
control the assessed risks. For front-line workers, it
is particularly useful to have bounce-back (auto-
matic response produced by the software) and then
a structured feedback after the report.5 In health-
care, considering the high level of expertise and
specialisation of the workforce, involving those
who reported in the investigation process can be
a first step to fostering learning and improvement,
with the condition that confidentiality and
deidentification be granted in the final report of the
investigation.3 5
After all, the effectiveness of an IRS is still
questioned, mostly when an IRS has been devel-
oped and implemented with large investments,
personal effort and political support.7e9
The most valuable uses for an IRS are the rapid
identification of incidents at the front-line level and
the communication of the event to the managerial
level of the organisational, or to an external agency
dedicated to the analysis of reports and eventually
to the investigation of major events or clusters.
The use of an IRS in daily clinical practice in
many countries around the world can now offer the
possibility to reflect on some myths accompanying
their development. The daily practice with an IRS
and concurrent studies of the effectiveness of
patient safety provide more evidence on their
shortcomings rather than on their benefits.10 11 The
main point is that each report is more like a short
novel than a systematic summary of a clinical
process with negative results. These results are
more valuable than the details of the individual
stories. According to these considerations, the
difficulty in coming to a standard classification of
patient safety incidents can be attributed to the
nature of an adverse event that is unexpected and
therefore difficult to categorise a priori. The IRS is
based on voluntary reports as stated previously. As
a result, the statistical elaboration of data cannot
provide generalisations about results or the devel-
opment of provisional models.1 8 10 Studies on the
healthcare workers’ opinions and perceptions on
IRS are needed.
In this paper, the authors present results of
a survey conducted in 2006 and 2007 that examined
healthcare workers’ opinions and perceptions of
 Original research

patient safety incidents and the use of IRS in Italian hospitals. In A list of reserves composed of 30% of the defined sample size
hospitals without IRS, healthcare workers’ opinions on patient was elaborated in order to integrate the sample in case of a high
safety incidents and their expectations for an IRS were assessed. rate of non-responders (higher than 20%).
In settings where an IRS had been in use for at least 1 year, The specific dimension of each hospital and professional
workers’ opinions on patient safety incidents, expectations for category is described in table 1.
IRS and the effects of an IRS on patient safety were sought.
Analysis of the differences between these two settings can reveal Survey design
ways to improve and promote the use of IRS to improve patient Two questionnaires were designed to evaluate workers’ opinions
safety in Italian Hospitals. and perceptions of patient safety incidents and their expecta-
tions for incident reporting and learning systems, and to explore
METHODS differences in opinion between workers in hospitals with IRS
Sample and hospitals without IRS. The surveys were modelled after
This research is part of a national project on clinical risk items in AHRQ’s Patient Safety Culture Survey16 and the Safety
management (http://www.agenas.it/monitor2007.html). Hospi- Attitude Questionnaire17 18 (previously known as IHI Patient
tals participating in the national project were invited to partic- Safety Climate toolkit). The two questionnaires differed from
ipate in this study. Of those invited, four hospitals with an IRS each other only by two items (contact the lead author for more
(defined below) and 14 hospitals without an IRS elected to information about the two questionnaires). Workers in hospitals
participate (see table 1). These hospitals are located in seven with IRS received one version, and workers in hospitals without
regions of Italy. Within each hospital, a representative sample of IRS received the other.
healthcare workers were selected to participate in the survey.
The surveys were conducted between April/May 2006 and Survey collection
January 2007. The written surveys were administered in person during
To be defined as having an IRS, a hospital must have met the scheduled sessions. The surveys were administered according to
following criteria: (1) the IRS must have been in use for at least an ‘assisted self-assessment’ technique, in which respondents
1 year; (2) formal training of healthcare workers on the basics of filled in the questionnaire independently in the presence of
patient safety and incident reporting must have preceded IRS a researcher who could provide further explanation of the
implementation; (3) near misses and unsafe acts must be questions, if needed.
reported; (4) reporting must be anonymous or made anonymous
by the system; and (5) data from incident reports must be used Statistical analysis
to analyse contributing factors and to plan and implement Descriptive statistics were provided for each of the two samples
corrective actions in the short and long term. (with and without IRS). In each sample, respondents’ gender,
Survey participants were selected through a sampling proce- profession and clinical specialty were described as a percentage
dure proportional to the size of the organisational unit (the the group. Professions were categorised as either physicians or
hospital) they belonged to and to the professions that charac- other healthcare workers, which included nurses and techni-
terised the organisation. The professions were divided into two cians. The clinical specialties category was composed of indi-
main categories: physicians and other healthcare workers (nurses viduals from the following fields: surgery, medicine, obstetrics
and technicians). The size of the two samples (hospitals with and and genecology, intensive care, radiology, laboratory and reha-
without IRS) was calculated with a random sampling procedure, bilitation.18 The results of the survey were organised into two
without repetition, applied to the two different settings (margin parts according to the two different investigated issues:
of error 0.025). The sample for each setting was then subdivided ‘personal experience on patient safety issues’ and ‘opinion of
in two ways: (1) proportionally respect to the dimension of each incident reporting and learning systems.’
hospital participating in the survey, and (2) in each hospital Differences between sample groups in the response of indi-
proportionally with respect to each profession. The sample size vidual items and the difference within the two parts of the
required for the study in hospitals without IRS and in hospitals questionnaire were tested through the use of the c2 test and the
with IRS was 503 and 439 healthcare workers, respectively. Fisher exact test. The possible confounding effects of profes-
sional type and clinical specialties was evaluated using the
Table 1 Sample CochraneManteleHaenszel (CMH) test. All statistical analyses
Hospital setting were conducted using the software SAS System for Windows
Without incident-reporting With incident-reporting (V8; SAS Institute, Cary, North Carolina).
systems systems
Workers Workers
included included
RESULTS
Healthcare in Healthcare in Surveys were completed by 820 (87.0%) of 942 eligible workers,
Region Hospitals workers sample Hospitals workers sample with 466 respondents working in hospitals without IRS and 374
Lombardia 2 2843 98 2 4629 216 working in hospitals with IRS (table 2). The response rate of
Piemonte 2 2212 76 1 3622 167 workers in hospitals without IRS was slightly higher than that
Venito 1 1342 46 1 1196 56 of workers in hospitals with IRS (92.6% vs 85.2%). In hospitals
Friuli- 2 7.19 25 e e e with IRS, 30.5% of workers were physicians, while 69.1% were
Venezia nurses and technicians, in hospitals without an IRS, 26% were
Giulia physicians, and 73.8% were other healthcare workers. In both
Lazio 2 3912 135 e e e
settings, the specialties with the most sampled individuals were
Campania 4 1377 47 e e e
surgery and medicine, which is in line with the larger presence of
Puglia 1 2219 76 e e e
these specialties in most hospitals. The responders were
Total 14 14624 503 (3.4%) 4 9447 439 (4.6%)
predominantly women in both settings.

Qual Saf Health Care 2010;19(Suppl 3):i8ei12. doi:10.1136/qshc.2009.036376 i9

 Original research

Table 2 Characteristics of the 820 respondents* The way in which healthcare workers received communica-
Hospital setting tion on these patient safety incidents is described in table 4. In
Without incident-reporting With incident-reporting general, communication happened mainly during daily informal
Characteristic systems (n[503) systems (n[439) discussions with colleagues. This is more evident in hospitals
Response ratey 92.6 85.2 with an IRS (74.3 vs 68.7).
Gender With respect to the personal experience of either one or more
Male 40.8 35.3 patient safety incidents, 30.8% and 36.7% of the healthcare
Female 58.6 64.2 workers in hospitals with and without an IRS declared having
Not listed 0.6 0.5 not experienced a patient safety incident during their individual
Clinical training careers, and more than 40% of the participants of the survey in
Physician 30.5 25.9 both settings reported having experienced between one and
Other healthcare workers 69.1 73.8 three patient safety incidents during their career.
Not listed 0.4 0.3 Among healthcare workers who declared having not reported
Clinical specialty an adverse event, there was no significant difference between
Surgery 22.1 26.7 settings in the reasons as to why they did not. Nevertheless, in
Medicine 29.4 31.8 hospitals without an IRS, healthcare workers more frequently
Obstetrics & gynaecology 9.7 4.3 did not report patient safety incidents through a formal report
Intensive care 11.2 11.5 because they were afraid of mistrust from colleagues and
Radiology & laboratory 11.8 8.0 because they were sceptical towards any possible improvement
Rehabilitation 2.2 2.1
after the report would have been filed (table 5).
Other 11.6 13.4
In both settings, we found that the most frequent response
*Data are given as a percentage of each group unless otherwise indicated. was ‘none of the above.’ We then tried to understand if this kind
yFor hospitals with incident-reporting systems, this rate was based on 466 respondents of
503 possible; and for hospitals without incident-reporting systems, 374 respondents of 439 of answer was due to a bias in the design of the questionnaires or
possible. in the characteristics of the responders. Data analysis showed
that more than 50% of the interviewees who either did not
Personal experience with patient safety issue answer or answered ‘none of the above’ also declared not to have
There were statistically significant differences between the two experienced any patient safety incident. On the other hand,
settings around workers’ personal experiences in clinical risk those who identified any of the proposed barriers to reporting
management (CRM). Workers were asked to reveal the methods had previously experienced at least one patient safety incident.
by which they gained exposure to CRM (table 3). Workers in
hospitals with IRS were more likely to report reading material in Opinion on incident reporting and learning systems
their unit and learning practices from colleagues, and to partic- Healthcare workers differed on who should be the recipient of
ipate in clinical audits and meetings for CRM than workers in the formal report of an adverse event, either between the two
hospitals without IRS. settings (table 6) and among different professional figures
Healthcare workers involved in incident investigation (formal (figure 1). There was a significant difference in the way of
in-depth analysis of the event) were a minority in both settings, communicating adverse events between the two settings
35.2% for hospitals with IRS and 40% hospitals without IRS. No (p¼0.001). In particular, in hospitals with an IRS system, the
significant differences were found between hospitals adjusting horizontal communication prevailed even for formal reports of
for professional profiles and clinical specialty categories patient safety incidents (38.6 vs 32.7), while in settings without
(CMH¼1.46, p¼0.23). an IRS the vertical communication was preferred (38.8 vs 26.0).
In both settings, almost 90% of the workers who participated According to the general opinion of the interviewed healthcare
in a formal incident investigation agreed on the effectiveness of workers, fear of punishment as a result of reporting a patient
this activity with respect to a positive change on patient safety. safety incident did not seem to be a barrier for using an incident
In both settings, 10% of interviewees had never been informed reporting system because more than 50% of the total sample
of a patient safety incident that occurred to a colleague, and the said that was ‘never or hardly ever a problem’ (CMH¼3.91,
other 90% declared that they received information about an p¼0.27; data adjusted for the different professions). Quite
incident which occurred to a colleague at least once in their interestingly, this answer was the same in hospitals with or
careers.
Table 4 Percentage of response on the communication flow of patient
Table 3 Response rate and rate of affirmative responses for questions safety incidents (question: For the most part, in what manner do you
relating to the personal experience on clinical risk management learn about adverse events?)
(question: Please, answer the following statement on the basis of your
Hospital setting*
personal experience related to the clinical risk management)
Without With
Without With
incident-reporting incident-reporting
incident- incident-
Source of information systems (n[399) systems (n[331)
reporting reporting
Survey item systems systems p Value* Daily informal discussions 68.7 74.3
I participated in training courses, 60.5 58.1 0.52 Confidential report from a colleague 22.3 16.0
conferences, events Media 6.5 3.6
I read articles on the topic 56.5 54.0 0.52 During a formal meeting (clinical audit 1.0 4.5
I read some material in my unit 37.6 66.0 >0.001 or mortality and morbidity review)
I learnt this practice from colleagues 35.4 44.7 0.019 Notice from the clinical risk manager 0.75 0.2
I participated in clinical audits and 29.7 38.6 0.020 Total responders 399 331
meetings for clinical risk management Non-responders 67 43
*c2 test. *c2 test Fisher exact p¼0.0026.

i10 Qual Saf Health Care 2010;19(Suppl 3):i8ei12. doi:10.1136/qshc.2009.036376

 Original research

Table 5 Percentage of response on barrier to reporting in case of never

41.6
reporting of adverse events (question: If you did not report an adverse Physicians 32.6
event, what was the reason?) IRs 2.25
23.6
37.5
Hospital setting* Other healthcare workers 24.6
35.9
Without incident- With incident- 2.02
reporting systems reporting systems 24.4
Barrier (n[215) (n[160) Physicians 27.5
42
6.11
Fear of mistrust of colleagues 14.4 5.6 No IRs 26.8
32
It is not considered a priority 12.6 11.9 Other healthcare workers 37.6
Fear of punishment 4.2 4.4 3.59

Does not help to improve safety 4.2 2.5

0 5 10 15 20 25 30 35 40
Lack of time 2.8 2.5
% of response
Total responders 215 160
Close colleagues
Non-responders 54 44 Professional community
*CochraneManteleHaenszel (CMH) test: CMH¼9.37, p¼0.095 (data adjusted for Head of department
professional profiles). None of the previous

Figure 1 Percentage of response on who should be the first recipient

without an IRS, while we expected to find more fear for a blame
of the formal report of an adverse event for professional figures.
culture in settings where clinical risk management had not yet
been formally implemented in daily practice. One of these factors might be a general increase in the interest
More than half of the total healthcare workers involved in the among clinicians in clinical risk management in the last decade.
survey, 60.44% in hospitals without IRS system and 59.29% Indeed, incident reporting and reflection on patient safety inci-
hospitals with an IRS system still believed that an IRS can be dents were only a fraction of the activities for patient safety at
used to assess professional responsibility and to blame individ- the hospital level.
uals because of eventual errors. This was true both for physi- In any case, from this survey, we can make some general
cians and for the other healthcare professionals. Despite this consideration about IRS:
perceived inappropriate use of an incident report, more than 60% 1. Experience of patient safety incidents and expectation of IRS.
of the interviewees believed this system could help to reduce Regardless of whether a hospital had an IRS, workers reported
patient safety incidents, and it was not a fruitless effort. No similar rates of involvement in patient safety incidents.
difference between settings was found after adjusting for Seventy per cent of the respondents, in both settings (with
professional profiles. or without an IRS) affirmed that they were involved in an
The main factors contributing to the patient safety incidents incident. Of those with an available IRS, only 40% formally
were workload, stress, fatigue and organisation of work. reported an event. This confirmed the presence, in general, of
Communication was not yet perceived as a major contributing different reporting barriers in the IRS systems.15 19e22
factor, despite the evidence shown in published literature.5 14 2. Professional community and an IRS. The communication of
risk remained in the community to which the professional
DISCUSSION belonged. The data showed that information on patient
The statistical analysis did not show any important differences safety incidents was not expanded to the entire organisation.
between hospitals with or without an IRS. In particular, in The frequency with which people knew about an incident
hospitals with an IRS, participation in clinical audits and was very rare (never or almost never). Informal communica-
meetings was higher. tion of patient safety incidents was more integrated in daily
Thus, the opinions and expectations on an IRS were very practice, and horizontal communication seems to prevail.
similar for healthcare workers, regardless of whether or not they The perception of risks stayed local and very rarely (only
belonged to a hospital with or without an IRS, or whether or when disasters occurs) became part of the organisational
not they belonged to a professional profile or a clinical specialty. memory.
There were probably other factors we did not explore in the 3. Value of an IRS. In both settings, workers expressed
survey that might have affected the development of the overall agreement that an IRS is useful for patient safety. Addition-
safety culture within a hospital and a professional community. ally, we found a general agreement on the effectiveness of
patient safety incident investigation after formally reporting
an IRS, particularly when front-line clinicians were involved
Table 6 Percentage of response on the first recipient of the formal in the analysis and reflection during formal clinical audits or
report of an adverse event (question: If you report an adverse event, who
mortality and morbidity meetings.
would you like to talk to first?)
Settings*
LIMITS OF THE STUDY
Without incident- With incident-
As with any retrospective survey, one must take a respondent’s
reporting systems reporting systems
recall bias into consideration when evaluating the study’s reli-
Recipient n Percentage n Percentage
ability. Respondents were asked to recall incidents throughout the
Head of department 170 38.8 110 32.7 life of their careers. Additionally, the experience and consequences
Professional community 133 30.4 90 26.7 negative experiences can modify the memory of the event itself
Close colleagues 114 26.0 130 38.6 and make it more difficult for the individual to give an accurate
None of the above 21 4.8 7 2.1 depiction of the number of incident experiences. This is especially
Total responders 438 337 true when incidents are related to self-esteem and social trust.
Non-responders 28 37
Differences between professions should also be better explored
*c2 test p¼0.001. in the settings of the survey, because we did not find any

Qual Saf Health Care 2010;19(Suppl 3):i8ei12. doi:10.1136/qshc.2009.036376 i11

 Original research
significant differences, even though we know from other studies
that physicians and nurses usually show different behaviours
towards an IRS.
The longevity of an IRS has not been considered in this
survey. In fact, hospitals with an IRS had one requirement: to
have had the system in place for at least 1 year. Finally, an IRS is
quite a novel idea in Italy. Therefore, in the future, we need to
better explore the relationships between healthcare workers’
opinions of IRS, the number of IRS and the outcomes in terms
of quality and safety of the hospitals with and without an IRS.
CONCLUSIONS
These research findings are in agreement with existing research
on healthcare workers’ experiences of patient safety incidents, as
well as their positive opinions and expectations of an IRS, even
though there are still barriers to formal reporting using the
IRS.23 24 26 We shall consider developing further studies and
interventions to appraise safety culture, in order to have more
effective reports and occasions for learning from errors.24 25 After
all, at the time of this survey, an IRS in Italy was still a fresh
innovation, and as a result, it was difficult to provide an in-
depth analysis on its impact in the world of patient safety. We
can conclude that this research, as part of a wider national
project aimed at spreading methods and tools for patient safety,
has contributed to developing IRS in Italy, considering the
sociocultural context of Italian hospitals.
Acknowledgements The authors would like to thank all the healthcare operators
who accepted to participate in the research, the Italian Ministry of Health and
Gutenberg Sicurezza in Sanità, which funded this project. The research in all its
phases (planning, conducting and reporting the work) has been realized with the
contributorship of Agenas and all the members of the group of the research project:
A dell’Erba, University of Bari, Bari; L Sommella, S.Filippo Neri Hospital, Rome;
G Luongo, Regional Agency for Healthcare of Campania; G Simon, Regional Agency
for Healthcare of Friuli Venezia Giulia; R Russo, University of Turin, Turin; M Bosio,
Lecco Hospital, Lecco; AM Longo, Regional Agency for Healthcare of Rovigo.
Funding The Italian Ministry of Health and Gutenberg Sicurezza in Sanità.
Competing interests None.
Provenance and peer review Not commissioned; not externally peer reviewed.
REFERENCES
1. Billings C. Incident reporting systems in medicine and experience with the aviation
safety reporting system. In: Cook RI, Woods DD, Miller C. eds. A tale of two stories:
contrasting views on patient safety. Report from a workshop on assembling the
scientific basis for progress on patient safety. McLean, VA: National Patient Safety
Foundation at the AMA, 1998. http://www.npsf.org/ exec/billings.html.
i12
2. Pronovost PJ, Thompson DA, Holzmueller CG, et al. Toward learning from patient
safety reporting systems. J Crit Care 2006;21:305e15.
3. WHO. WHO draft guidelines for adverse event reporting and learning systems.
Geneva: WHO Press 2005.
4. Kreckler S, Catchpole K, McCulloch P, et al. Factors influencing incident reporting in
surgical care. Qual Saf Health Care 2009;18:116e20.
5. Benn J, Koutantji M, Wallace L, et al. Feedback from incident reporting: information
and action to improve patient safety. Qual Saf Health Care 2009;18:11e21.
6. National Patient Safety Agency. Building a memory: preventing harm, reducing
risks and improving patient safety. The first report of the National Reporting and
Learning System and the Patient Safety Observatory July 2005.
7. Cook RI, Miller C, Woods DD. A tale of two stories: contrasting views of patient
safety report from a Workshop on Assembling the Scientific Basis for Progress on
Patient Safety National Health Care Safety Council of the National Patient Safety
Foundation at the AMA 1998.
8. Wald H, Shojania KG. Incident reporting. In: Making Health Care Safer: A Critical
Analysis of Patient Safety Practices. AHRQ, 2001:41-50. Evidence Report/Technology
Assessment No 43. AHRQ Publication No 01-E058. http://archive.ahrq.gov/clinic/
ptsafety/chap4.htm.
9. Blais R, Bruno D, Bartlett G, et al. Can we use incident reports to detect hospital
adverse events? J Patient Saf 2008;4:9e12.
10. Rosenthal J, Riley T, Booth M. State reporting of medical errors and adverse events:
results of a 50-state survey. National Academy for State Health Policy 2000.
11. Hutchinson A, Young TA, Cooper KL, et al. Trends in healthcare incident
reporting and relationship to safety and quality data in acute hospitals: results
from the National Reporting and Learning System. Qual Saf Health Care
2009;18:5e10.
12. Kohn LT, Corrigan JM, Donaldson MS. To err is human. Washington, DC: Institute of
Medicine, National Academy Press 2000.
13. Altman DE, Clancy C, Blendon RJ. Improving patient safety dfive years after the
IOM Report. N Engl J Med 2004;351:20.
14. Vincent C. Patient safety. London: Elsevier, 2006.
15. Schectman JM, Plews-Ogan ML. Physician perception of hospital safety and
barriers to incident reporting. Jt Comm J Qual Patient Saf 2006;32:337e43.
16. Agency for Healthcare Research and Quality. Patient safety culture surveys.
http://ahrq.hhs.gov/qual/patientsafetyculture/ (accessed 7 Jul 2009).
17. Sexton JB, Helmreich RL, Neilands TB, et al. The safety attitudes questionnaire:
psychometric properties, benchmarking data, and emerging research. BMC Health
Serv Res 2006;6:44.
18. McDonald JH. Handbook of biological statistics. 2nd edn. Baltimore, MD: Sparky
House Publishing, 2008.
19. Joint Commission on Accreditation of Healthcare Organizations. Sentinel
event policy and procedure. Revised edn. Illinois, USA: The Joint Commission, 1998.
20. Barach P, Small SD. Reporting and preventing medical mishaps: lessons from
non-medical reporting systems. BMJ 2000;320:759e62.
21. Firth-Cozens J. Barriers to incident reporting. Qual Saf Health Care 2002;11:7.
22. Stanhorpe N, Crowley-Murphy M, Vincent C. An evaluation of adverse incident
reporting. J Eval Clin Pract 1999;5:5e12.
23. Wu AW, Pronovost P, Morlock L. ICU incident reporting systems. J Crit Care
2002;17:86e94.
24. Leape L. Why should we report adverse incidents? J Eval Clin Pract
1999;3:213e22.
25. Pizzi LT, Goldfarb NI, Nash DB. Promoting a culture of safety. In: Shojania KG,
Duncan BW, McDonald KM, et al, eds. Making health care safer: a critical analysis of
patient safety practices. AHRQ Publication No 01-E058, Rockville, MD: Agency for
Healthcare Research and Quality, 2001.
26. Ahluwalia J, Marriott L. Critical incident reporting systems. Semin Fetal Neonatal
Med 2005;10:31e7.
Qual Saf Health Care 2010;19(Suppl 3):i8ei12. doi:10.1136/qshc.2009.036376

1
Information Systems and
Innovation Group, Department
of Management, London School
of Economics and Political
Science, London, UK
2
Centre for HCI Design, City
University London, Northampton
Square, London, UK
Correspondence to
Dr Valentina Lichtner,
Information Systems and
Innovation Group, Department
of Management, London School
of Economics and Political
Science, Houghton Street,
London WC2A 2AE, UK;
v.lichtner@lse.ac.uk
Accepted 2 December 2009
Published Online First
31 May 2010
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400
A pragmatics’ view of patient identification
Valentina Lichtner,1 Julia R Galliers,2 Stephanie Wilson2
ABSTRACT
Background Patient identification is a central safety
critical aspect of healthcare work. Most healthcare
activities require identification of patients by healthcare
staff, often in connection with the use of patient records.
Indeed, the increasing reliance on electronic systems
makes the correct matching of patients with their
records a keystone for patient safety. Most research on
patient identification has been carried out in hospital
settings. The aim was to investigate the process of
identification of patients and their records in the context
of a primary healthcare clinic.
Method A qualitative field study was carried out at
a Walk-In Centre in London (UK).
Results The identification of patients and their records
was found to be a context-dependent process, both
when formalised in procedures and when relying on
informal practices. The authors discovered a range of
formal and informal patient identifiers were used in this
setting, depending on the task at hand. The theoretical
lens of Pragmatics was applied to offer an explanation of
this identification process.
Conclusions Context provides the cognitive scaffolding
for a process of ‘suitably constrained guesswork’ about
the identity of patients and their records. Implications for
practice and for system design are discussed.
Practitioners and technology designers should be aware
of the risk for misidentifications inherent in this natural
information processing activity.
INTRODUCTION
Patient identificationdthat safety-critical activity
of matching patients with their care, treatment and
records1dis a matter of context-dependent ‘suit-
ably constrained guesswork.’ This conclusion
becomes apparent when observing the daily work
of healthcare staff, reading the literature and even
just reflecting on our daily first-hand experience of
recognising people in everyday settings. Identifica-
tion of people is an everyday ubiquitous activity,
and similarly, patient identification is a ubiquitous
activity in healthcare settings. Most healthcare
activities involve, explicitly or implicitly, identifi-
cation of patients by healthcare staff, when regis-
tering patients, giving medication, discussing
diagnoses, etc. Computerised or paper records may
be required for these activities. The use of multiple
identifiers such as names and date of birth is
recommended for safe practice.2 3
Errors occur in patient identification, sometimes
with serious consequences. The variety of reporting
systems and categorisations, and the fact that
identification mix-ups may be perceived as ‘too
trivial to merit reporting,’4 hinders the collection of
comprehensive statistical evidence of misidentifi-
cations of patients or their records across settings
and countries. Data collected by the National
Original research
Patient Safety Agency (NPSA) in the UK show
how, in the period February 2006 to January 2007,
24 382 cases of patients being mismatched to their
care were reported, with an estimate of more than
2900 cases related to the use of wristbands.5 In
addition, there is qualitative or anecdotal evidence
presented in studies, reports or the news. For
instance, cases of registration errors are described in
Hakimzada et al,6 a barcoding near-miss in McDo-
nald7 and a case of misidentification in the paedi-
atric emergency department in O’Neill et al.8 The
classic paper ‘The Wrong Patient,’9 is the detailed
analysis of the case of a patient who was mistak-
enly taken for an invasive electrophysiology
procedure scheduled for another patient with
a similar name. At least 17 times before beginning
the procedure, clinicians had the chance to check
the patient’s identity and correct the mistake but
failed to do so, always assuming they had the right
patient. It is also not uncommon to read about
cases of identification errors in the news, for
instance in relation to medication errors (medica-
tions meant for another patient),10 switching of
newborn babies11 or a CT scan carried out on the
wrong pregnant woman.12
Solutions (technologies and protocols) exist to
support correct patient identification.13 In partic-
ular, the effectiveness of wristbands and barcodes in
hospital settings is widely recognised;5 ‘auto-iden-
tification’ technologies such as radio frequency
identification (RFID) are also being increasingly
applied. However, these solutions are not without
their problems. Full compliance with protocols may
be difficult to achieve,14 and there are risks or
unintended consequences associated with the use
of technology, such as data entry errors when
registering patients, the ‘reliance on imperfect
technical solutions as if they were perfect’ and the
‘elimination of human checking processes when
automated systems are implemented.’2
Identification errors are not uncommon, but they
happen in the complex context of healthcare
provision where million of patients are identified
correctly every dayddespite their names being
imperfect identifiers and their care fragmented
among many practitioners. We argue that the
process of patient identificationdwhen formalised
in protocols or procedures, or when relying on
informal practicesddepends on context. In this
paper, we provide examples of how patient identi-
fication happens in practice, provide reasons for
why it happens this way and suggest some impli-
cations for safety and for technology design.
METHODOLOGY
To investigate the practice of patient identification,
we carried out a case study of an NHS Walk-in
Centre in London (UK), with a field study
i13
 Original research
conducted in the period June to September 2006. The study was
based on observations of the daily activities of the Walk-In
Centre, shadowing receptionists and clinicians, and their
consultations with patients. Whenever possible, the members of
staff were engaged in informal conversations to clarify and
confirm our understanding of their work. Written field notes
were supplemented by sketches and copies of paper and
computer systems in use. The anonymity of all participants was
maintained: patients’ identifiers were not recorded and/or were
changed when necessary. The project received Ethical Approval
from the local NHS Ethics Committee. The data were analysed
with the support of qualitative research software Atlas-ti, used
to sort and label the data, and highlight the contextual elements
affecting patient identification. More on the field study and the
method for the analysis can be found in Lichtner.15
RESULTS
Setting
Walk-In Centres have been introduced in the NHS in the UK to
facilitate access to healthcare. They offer ambulatory care
without the need for an appointment. They are nurse-led and
usually open 7 days a week. They offer treatment for minor
injuries and illnesses, emergency contraception, medical
prescriptions for minor ailments and advice on which other
healthcare services are availabledall services that do not usually
require continuity of care and that can be solved with one visit.
The Walk-In Centre in this study is located in an area char-
acterised by multiple ethnicities. It treats about 3000 patients
a month and shares its space with other healthcare services.
Receptionists, nurses and doctors, manage the flow of patients.
The typical workflow starts with the triaging, streaming and
‘booking in’ of patients at reception, moves to the consultation
with clinicians, the referral to other services and request for tests
if necessary, until the patients leave and are ‘booked out’
(figure 1). The work is fast-paced, often interrupted and, as we
will explain in the next section, involves several patient-identi-
fication related tasks, usually with the support of paper artefacts
or computer records.
Context-dependent identification: the main workflow
Patient-identification related tasks are necessary key steps in the
main workflow. These steps are centred on the production and
use of a patient registration formdthe ‘Booking-In Form’dand
the patient electronic healthcare record. At their arrival at the
Walk-In Centre, patients are asked to fill in the Booking-In Form
with their biographical information, contact details and
healthcare problem (this step in the process is discussed more in
depth in Lichtner et al16). On the basis of the information
provided by the patient in their paper Booking-In Form, the
receptionist searches for an existing electronic patient record and
Figure 1 Walk-In Centre workflow and main patient identification steps.
i14
creates a new one if no matching record is found (Step 1 in the
workflow). The clinician later uses the same Booking-In Form
to invite the patient to the consultation room (Step 2) and to
retrieve the electronic patient record added by the receptionist to
the list of ‘booked-in patients’ (the electronic queue of patients
waiting to be seen) (Step 3): the clinician will select the patient
record that shows the same name and date of birth, among
those showing in the electronic list of booked-in patients. This
electronic record is used as official documentation of the
patient’s visit at the Walk-In Centre. When necessary, the
patient details are copied from the Booking-In Form to
prescription forms or forms to request laboratory tests (Step 4).
Each one of the four steps requires matching the patient with
their electronic and/or paper record. These steps are carried out
following the standard rule of always applying multiple official
patient identifiers when matching patients with their records:
gender, date of birth, name and address. Different combinations
of these identity attributes are used at the different steps. For
instance, when booking in patients (Step 1), the receptionist
compares the identifiers provided by the patient in the Booking-
In Form with those recorded on the electronic patient record
system, and accepts an existing record as the ‘right’ one only if
all four of the identity attributes match. However, a range of
other contextual elements arising from the interaction and the
activity support the matching of patients with their records. For
instance, for the receptionist, it is the process itself of comparing
the information provided by the Booking-In form with the
records on the computer screen that supports the retrieval of the
correct record. The case below, observed at the Walk-In Centre,
shows how, if the patient’s writing is difficult to read and the
receptionist is not sure of the correct spelling of the patient’s
name, the list of possible names appearing once a search is done
with the date of birth can help:
Receptionist Ra searches for the patient record in the system. The
patient name is difficult to read, but an existing record was on the
system, and this helps the recognition of the right word. Ra passes
the form to receptionist Rb for data entry in another electronic
system. Rb has the same difficulties reading the name. Ra says to
Rb: ‘you’ll find it there.’
Most patients visiting this Walk-in Centre come from an ethnic
community where people often share the same name (eg, the
name Begum is very common), and large families live at the same
address. Therefore, it is not uncommon for clinicians to view very
similar patient records. Table 1 shows a section of the computer
screen: of the list of 11 patients, two of them, close together,
appeared to be very similar. The similarity of the official patient
identifiers (surname and age) does not facilitate the identification
of the correct patient record. The patient’s medical case, showing
in the ‘reason for visiting’ provides more meaningful information.
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400
 Original research

Table 1 Section of computer screen showing patient records in a queue of waiting patientsdsimilarity of data*
Time (min)
Surname Age Reason for visiting Unit in queue
Begum 31 years, 6 months, 16 days Sprained ankle 0 Minor Illness Unit 34
Begum 33 years, 4 months, 14 days Sore throat 0 Minor Illness Unit 30
*Based on field note of 4 July 2006dpatients’ age has been changed.

Also, the order of the records may help: clinicians expect the record
for the ‘next patient’ to be on top of the list (this being consistent
with the Walk-In Centre rule of seeing patients on a first-come
first-served basis). A match exclusively based on official identifiers
would make the identification of the correct record more difficult,
since patient identity attributes, like the surname Begum, are not
necessarily ‘unique.’
Yet, patient identity attributes may be considered ‘unique’ in
a specific context. For instance, there is only one patient called
Begum on the list of patient records shown in table 2, appearing
on a specific date and time at the Walk-In Centre. The context
constrains the range of potentially correct records and facilitates
the identification. Incidentally, it may be interesting to note the
extra elements added in the field ‘Reason for visiting,’ shown in
table 2: for example ‘FT 16.15.’ These are the initials of the
clinician expected to call the patient for the consultation, and
the time when this is expected to happen. They have been added
to the electronic record as a consequence of a change in the
workflow introduced in the Walk-In Centre during the field
study, now requiring allocation of patients to named nurses and
doctors. This contextual information constitutes an extra iden-
tifier for the identification of the correct patient record, as the
clinician would now look for and match not only the patient
details but also their own initials.
Other patient identification activities
In addition to the patient identification activities described
above, there are several other occurrences of identification of
patients and their records at the Walk-In Centre. These show
how ubiquitous patient identification is and how it often
happens in an undetected manner. Again, in these cases, too,
context plays a key role in the identification of patients.
Occurrences of patient identification outside the main workflow
were observed, for example, in association with:
< clinicians discussing patients’ cases or asking for advice
(within the boundaries of the Walk-In Centre);
< clinicians requesting lab results or asking for confirmation of
treatments from family doctors (communication with
external services, outside the boundaries of the Walk-In
Centre).
Discussing and remembering patients
The identity attributes used for talking about patients and
exchanging information about them within the Walk-In Centre
are usually not the patient’s name but the patient’s illness, the
reason for visiting or in general any event that occurred during
the interaction with the patient. For instance, a nurse asking for
advice would show a patient record to the doctor and say: ‘This
is that bloke I drained the finger of ’ (Field Note); or a doctor
having problems with a patient’s record would ask a colleague:
‘did you see a patient, the one with long complicated history,
diabetes’ (Field Note).
At the Walk-In Centre, patients are rarely known, or referred to,
by name. Receptionists may remember patients’ names, but this is
also rare. Patients do not usually come often enough to the Walk-
In Centre for the members of staff to learn their names. Two
episodes were observed where the patient was known by name. A
receptionist was observed remembering an elderly woman’s
name, as she used to visit very frequently the doctor’s surgery
where the receptionist used to work. Similarly, after a few months
of repeated visits by a patient going to the ‘Fast Response Unit’
based in the same clinic, a receptionist also learnt the patient’s
name and was able to announce the patient by his name without
asking. However, the same receptionist suggested that is easier to
remember patients by their reason for visiting:
Receptionist: ‘What was her complaint (reason for visiting)? If I
remember her complaint I remember.’
Names are necessary to call the patient for the consultation
and retrieve patient’s recorded information, but they seem
superfluous when other tasks are to be carried out. For instance:
The nurse is discussing with the receptionist the list of waiting
patients, to help with the queue.
They are checking the Booking-In forms:
Receptionist: ‘the stitches is at 15.20’
Nurse: ‘that’s in 5 min’
In this exchange what is relevant is the type of healthcare
problem patients were presenting, so as to allocate patients on
the basis of the difficulty of the case and the time needed for the
consultation. Referring to the patient’s problem provides enough
information to identify the patient.
Communication with outside services
A patient’s case often requires interacting with outside services,
usually requiring the correct identification of the patient in
question: for example, for checking test results from the lab, or
obtaining permission from a patient’s family doctor to prescribe
a repeat medication. Communication with these services happens
either by phone, fax or letter, possibly with preset paper forms.
Multiple patient identifiers are always used. In most cases, the

Table 2 Section of computer screen showing patient records in a queue of waiting patientsdall ‘unique’ names*
Time (min)
Surname Age Reason for visiting Unit in queue
Begum 31 years, 6 months, 16 days Swollen fingerdFT 16.15 0 Minor Illness Unit 39
Ross 53 years, 16 days Ankle problemdECd16.40 0 Minor Illness Unit 25
Jones 25 years, 30 days Mapdtod16.40 0 Minor Illness Unit 20
Smith 39 years, 9 months, 8 days Prolonged bruisingdFT 16.45 0 Minor Illness Unit 8
*Field note of 23 August 2006dpatients’ details have been changed.

Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400 i15

 Original research
member of staff would support the communication with the
patient’s paper and/or electronic record. In the following example,
a nurse supports their request for lab tests results with paper
notes, previous lab results and computer records:
The nurse is asked to check lab results for a patient.
She is holding lab test results with a small piece of paper attached
(on it: patient’s date of birth, telephone number, and notes such as
‘called’).
She calls the lab: ‘I am (explains her role and where she is calling
from). I am chasing results for a patient seen here’ (at Walk-In
Centre).
She is asked for/answers with: Date of birth, Type of test. For the
Hospital numberd‘No we don’t have it’dFor the NHS
numberd‘No we don’t have it.’ She gives the patient’s surname.
(She waits).
She is asked for more confirmation, passed onto to another person.
She stretches to the computer behind her with the phone and paper
on her hands. She looks on the electronic patient record for the
patient’s history of consultation.. describes history to the lab.
Pieces of paper, preset forms, computer records and printouts
are props always used when formal patient identifiers are the
types of identity attributes required for retrieving patient
recorded information. Formal identifiers need to be written
down for the member of staff to use them. Communication
with services outside the Walk-In Centre always relies on official
patient identifiers, yet also in this case the context supports the
identification of the patient record: that is, the two NHS services
share the context of ‘patients sent by the Walk-In Centre’
(without NHS numbers), ‘patient’s type of test’ and patient’s
‘medical case’ that makes the type of test relevant to the patient
in question. We can expect the person on the other end of the
phone to make use of all this information to match it with the
lab results available on their systemdlimiting the list of
potential matches and easing identification of the right record.
DISCUSSION
There is a series of formal and informal ways in which patients
and their records are identified at the Walk-In Centre. Coding of
the field notes from this study revealed different identity attri-
butes used for patient and record identification. Not all these
identifiers are actual ‘data’: the identification may rely on the
person’s appearance, or the appearance of the Booking-In Form,
or rather the Booking-In form or the electronic record being
absent (not ready yet): ‘. his record would be towards the end
(of the list), because the form was not ready’ said a clinician of
a patient invited to the consultation without the support of the
Booking-In form. Elements belonging to the history of the
interactions with the patients are also used, especially their
medical history.
Patient identification involves matching identity attributes,
and this is possible when the matched attributes are of the ‘same
type.’ Formal identifiers (for instance, a patient’s name) would
not assist in patient identification unless they are compared
with the data shown on a supporting tool (eg, a piece of paper or
the Booking-In Form in this context, or wristbands in a hospital
context). As shown at a shift handover in an emergency
department in Australia, patient names did not afford recogni-
tion of patients:
On several occasions, we observed how incoming staff needed to
ask several times ‘Who is he/she?’ when a patient’s name was
i16
mentioned, because they could not match a name with a person. In
order to help out, others refreshed their memories by referring to
specific characteristics such as ‘The confused woman’ or ‘The girl
with the big family.’17
Conversely, matching ‘informal’ identity attributes with
formal identifiers also does not afford recognitiondthe problem
authorities had with The Piano Man, an unknown confused
patient, found on a beach in the UK, who played piano in the
hospital:
(The co-founder of the National Missing Persons Helpline) said
a European database for missing people would not have helped in
(The Piano Man’s) case because ‘you need a name and strong
descriptive details of the person, something the authorities did not
have.’18
In all casesdwhen sharing patients’ information with people
or through electronic systems, when relying on formal or
informal identity attributesdthe context, relevance and simi-
larity of the attributes used for recognition are central to the
process. Pragmatics can explain how and why this is the case.
Pragmatics is a rich field of study that spans from linguistics
to psychology and philosophy. Broadly speaking, pragmatics ‘is
an approach to description, to information processing, thus to
the construction, interpretation and communication of experi-
ence’19 It offers a context-dependent theoretical explanation of
human behaviour and use of language. One of the main princi-
ples of this approach is that: ‘The meaning of an expression
cannot be fully understood without understanding the context
in which the expression is used.’19
Context
From a pragmatics’ point of view, context is constituted by the
‘outside world’ or a person’s specific situation but also consists of
a person’s assumptions and expectations and these will influence
the interpretation of the information:
A context . is not limited to information about the immediate
physical environment or the immediately preceding utterances:
expectations about the future, . anecdotal memories, general
cultural assumptions, beliefs about the mental state of the speaker,
may all play a role in interpretation.20
Context then also implies a variety of frames of reference,
a multiplicity of ‘map coordinates’ of one’s vantage point.’19
Following this approach, ‘(t)he description of an entity is
incomplete, indeed un-interpretable, unless it specifies the point
of view from whence the description was undertaken.’19 A
sentence such as: ‘the stitches is at 15.20’ makes sense only in
the context of the space and the activity where it is expressed.
Here is another example, from a different healthcare context,
taken from the literature:
During morning rounds . HS (the on-call resident), who will be
admitting the patients, checked his patient summary sheet and
asked TK (a fellow), ‘can the trauma (patient) go to the floor?’ TK
told him if they need room, then the patient can be discharged.21
No matter how many other trauma patients were in the
hospital at the time, the expression ‘can the trauma go to the
floor’ made sense to the clinicians involved in the exchange, and
the patient was successfully recognised by the single word
‘trauma.’ For the same reasons, when in a conversation we refer
to someone who is present in the same room, sharing a glance
may be enough to understand who the person is we are talking
aboutdthe context of our discussion as well as the glance would
remove, or at least reduce, ambiguity in identification.
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400

Relevance
Another reason why an utterance such as ‘the stitches is at
15.20’ makes sense is because the information appears to be
relevant to the people involved in the interaction. Sperber and
Wilson (1986) argue that human communication involves
capturing someone’s attention, ‘hence to imply that the infor-
mation communicated is relevant,’20 and that the property of
‘relevance’ is the key to understand both human communication
and cognition. Again, the idea is that the human being has to
find an efficient solution to cope with the huge amount of
information available in the environment:
In the immense, Herculean task of naturaldbiologicaldinformation
processing, the bulk of the input is in fact blocked, that is,
deemeddin the appropriate contextdto be either irrelevant or not
urgent. Only small morsels of the input, judged to be either relevant
or urgent in context, are let through for further processing. The
selective exercise of the mind’s contextual judgement . is the sinae
qua non of natural biological information processing, which is
undertaken under severe limits: finite time, finite storage capacity,
finite means.19
In any communication exchange the hearer will expect the
information communicated to be relevant, and they will inter-
pret this information on the basis of what ‘best satisfies this
expectation.’20 The same is true when communication is not
face to face, as in the case of a telephone conversation22 or
computer-mediated communication. In the case of telephone
calls, auditory information may not be sufficient for the person
on the receiving side of the phone line to recognise who is
calling, especially if the call appears to be ‘out of context,’ or ‘not
relevant’ to the person called. Schegloff explains this phenom-
enon with the impact of expectations (the expected potential
event): the answerers solve the recognition problem by searching
not for ‘all those they know or all recognizables’ but for those
who are ‘potential callers.’22
Relevance implies that human understanding is a form of
‘suitably constrained guesswork’20dthe constraints constitute
a form of cognitive scaffolding23 and they are offered by the
personal and interactional context, the human activity, the
subjects’ intentions. Patient identity and record recognition
based on statements such as ‘.one called Smith, got yellow
marking everywhere’ (Field note) is possible because the
elements used for recognition are based on shared contextual
knowledge, and they are relevant to the task at hand. The
sentence only makes sense to the member of staff because of
the shared knowledge, the common understanding of the
nurses’ work and the relevance to the expected workflow. The
description of a patient identitydor of their Booking-in form
with yellow marking everywheredwould be too ambiguous, or
meaningless, outside the specific situation at the Walk-In
Centre. Clinicians and receptionists share the same assump-
tions and expectationsdthe same ‘psychological’
contextdcollaborating on the Walk-In Centre work-
flowdsharing the same physical and working context. The
exchanges are only meaningful in this context, and they are
only ‘contextually relevant’ to this particular activity. Indeed,
as seen in table 1, patient identity information alone can be
ambiguous even within the physical context of the Walk-In
Centre. While instead table 2 shows how, at a given day and
time at the Walk-In Centre, context provided the tacit and
necessary constraints to facilitate recognition of a patient called
Begum. It reduced ambiguity in identification, reducing the
number of ‘possible candidates.’
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400
Original research
Similarity
In patient identification, the matching between attributes is
done on the basis of their ‘similarity.’ This is another concept
used in pragmatics, where similarity is known to be ‘a matter of
degree’: ‘In principle anything can be similar to anything else .
provided the appropriate context, frame, or point of view is
construed.’19 Even more then is identification a ‘matter of
probability,’ the ‘likelihood’ that the matching is correct. In the
case shown before, the receptionists Ra and Rb judged that,
given the context and the matching of all other attributes
available, the similarity between the patient name scribbled on
the Booking-in Form and that shown on the electronic record
system meant that this was the correct patient record.
Potential for identification errors
Because of the role of context in human communication, ‘a
mismatch between the context envisaged by the speaker and the
one actually used by the hearer may result in a misunder-
standing.’20 A doctor recounted in a court hearing in the UK:
. I was leaving the ICU, and I was approached by the surgical
senior house officer who said, ‘We’ve seen the patient and we’ll be
taking over his care,’ and as he was the only referral I had made, I
assumed she was talking about him. (Doctor’s statement reported
in Pinch.24)
The senior house officer was in fact talking about another
patient. The case is reported in a news article entitled: ‘Death of
a patient lost in the hospital system,’ as the misunderstanding
meant that the patient in question was not taken care of and later
died. The assumption over the identity of the patient was made on
the basis of context and relevance of the interaction: the exchange
happened in the ICU, where the doctor had made a referral for one
patient only, and they expected this to be acted upon. Similar
reasoning is an everyday practice, when we refer to things or people
by pronouns assuming they have been ‘previously identified.’25
Mix-ups with the use of formal identifiersdespecially
namesdare associated with the fact that given the context
(a ward at a given time, a waiting room, etc), we expect patient
names to be unique even though they are not. There are
numerous cases of patients mix-ups due to similarity of names
reported in the literature and in the news: ‘Patients with similar
names present challenges to the best-functioning healthcare
systems.’9 As pointed out in Cummins,26 the ‘extraordinary’
coincidences of patients with similar names are in fact ‘ordinary.’
At the Walk-In Centre, the similarity of two patients’ names led
one doctor to mistake one patient record for the other:
The doctor goes to reception to get the form for the next patient.
The doctordrushing back to his room with the form in his hand: ‘I
got the wrong form! Because they have the same name.’
Researcher: How did you realise?
Doctor: ‘I didn’t realise. It’s just the next patient is the same
patient!’
Relying on the patient’s surname for matching patients with
their forms and for selecting patients’ records can lead to mix-ups
as we assess the similarity of the identifiers available and think we
have the right record when we have not. The wider range and
greater number of identity attributes used (eg, age, but also reason
for visiting, allocated time and allocated clinician, the position of
the record in the list, etc), the more elements are provided for
a correct ‘guess’ and for detection of record mix-ups.
i17
 Original research
CONCLUSIONS
In patient identification, the similarity and contextual relevance
of the identifying attributes are the basis for the matching of
patients with their care, treatment, samples and records. The
matching is a process of ‘educated guesswork’ supported by
contextual constraints and expectationsdfor example, the range
of potential patients for matching is limited to ‘those waiting to
be seen’ at any given time and the relevance of the information
to the task at hand. Furthermore, the correctness of the
matching is assessed on the basis of the overall set of formal and
informal identifiers in context, rather than each specific one on
its own. Awareness should be raised of the dangers inherent in
this natural information processing activity, among both prac-
titioners and technology designers.
From a healthcare delivery perspective, these findings support
a recommendation that practitioners should be familiar with
their patients. It is possible that the better the clinician knows
a patient, the more identity attributes and contextual cues they
would have at their disposal for the probabilistic matching that
takes place in the identification process. Hence, the less likely it
is that the patient would be the subject of misidentification.
Conversely, the less a patient is known (for instance when
patients become ‘invisible’), the more limited the set of attri-
butes used for the matching. In particular, there is then an
increased reliance on formal identifiers not supported by other
contextual cues, leading to an increased likelihood of identifica-
tion errors. A study commissioned by the NPSA found that ‘all
interventions to improve the personal knowledge of patients,
such as the named nurse principle, or a reduced number of
patients per nurse appear to be useful.’27
There is an established literature on safety checks, redundancy
and other techniques aimed at safeguarding against ‘normal
errors’ 28 in complex, highly coupled environments. These safety
checks are often already inbuilt in today’s healthcare practice
and contribute to the prevention of hazards. It might also be
possible to apply a reliability approach based on ‘distributed
knowledge’ and ‘shared awareness’ of the patient identity.
Falzon proposes building reliability of the system as a collective
development process based on ‘the permanent sharing, discus-
sion and construction of knowledge about work, work situa-
tions, and rules for decision-making.’29 In the distributed
healthcare environment, the problem of guaranteeing correct
patient identification may find a further solution in strength-
ening group awareness and shared knowledge. It is not just the
single practitioner’s knowing of the patient that supports
correct identification but the shared knowledge of that patient.
A limited example of this was observed at the Walk-In Centre
when the receptionists intervened to make sure that clinicians
called the correct patient to the consultation. The unplanned
collaboration between receptionist and clinician in the task of
‘calling the patient’ allowed for the detection of an identification
error (calling the wrong patient with respect to the specific
Booking-In Form).
In terms of design of electronic record systems, this study
suggests that technology should be designed to support patient
identification as a context-dependent cognitive process. Since
context is not just the outside environment but also an indi-
vidual’s assumptions, and since patients and their information
travel across different contexts, this issue may be more complex
than at first appears. How can we build computerised systems
that support contextual patient identification across changes in
contexts: both ‘physical context,’ for instance when the patient
is moved from ward to ward, and ‘technical context,’ when the
i18
patient is known in person but it is then their disembodied
record that is recognised on a computer screen?
The NHS undertook an initiative, in collaboration with
Microsoft, for a standard common user interface design that
should provide a well-designed, always-visible, ‘patient banner’ in
electronic patient records.30 This is a greatly needed improvement
over the design of some healthcare systems that we have observed
in practice: some show very limited patient identifiers; others hide
them all together once the record has been selected. The need for
contextual patient identification in electronic records was
acknowledged in the first release of the design guidelines devel-
oped by Microsoft for the patient banner (later modified),
although this was described only as ‘useful information’:
In certain contexts, the user should be able to discover other
‘temporary’ information that could be used as supplementary
identification information, such as ward, bed number, consultant
responsible, and nurse caring for them this shift. This however,
might be seen as ‘useful’ information as opposed to patient
‘identification’ information.’30
We argue that this information is not only useful but indeed
necessary to support the identification of the correct record and
to detect and correct any record mix-ups.
Acknowledgements Special thanks to the members of staff and the patients of
the Walk-in Centre. We are also grateful to the reviewer for her constructive
suggestions.
Funding This work was supported by a 3 year PhD research scholarship granted to
the first author by the School of Informatics, City University, London, UK.
Competing interests None.
Patient consent Obtained.
Ethics approval Ethics approval was provided by the East London and the City NHS
Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed
REFERENCES
1. NPSA. Right patientdright care. London: National Patient Safety Agency, 2004.
2. WHO. Patient identificationdpatient safety solutions. Vol 1. Solution 2, May 2007.
World Health Organization, 2007. http://www.ccforpatientsafety.org/common/pdfs/
fpdf/presskit/PS-Solution2.pdf (accessed 3 Aug 2009).
3. Sevdalis N, Norris B, Ranger C, et al. Designing evidence-based patient safety
interventions: the case of the UK’s National Health Service hospital wristbands.
J Eval Clin Pract 2009;15:316e22.
4. O’Kane MJ, Lynch PLM, McGowan N. The development of a system for the
reporting, classification and grading of quality failures in the clinical biochemistry
laboratory. Ann Clin Biochem 2008;45:129e34.
5. NPSA. Standardising wristbands improves patient safety. UK: National Patient Safety
Agency, 2007. http://www.npsa.nhs.uk/public/display?contentId¼6073 (accessed
28 Jul 2007).
6. Hakimzada AF, Green RA, Sayan OR, et al. The nature and occurrence of
registration errors in the emergency department. Int J Med Inform 2008;77:169e75.
7. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss.
Ann Intern Med 2006;144:510e16.
8. O’Neill KA, Shinn D, Starr KT, et al. Patient misidentification in a pediatric
emergency department: patient safety and legal perspectives. Pediatr Emerg Care
2004;20:487e92.
9. Chassin MR, Becher EC. The wrong patient. Ann Intern Med 2002;136:826e33.
10. BBC News. Woman died after wrong medicine. UK: BBC News, 2006. http://news.
bbc.co.uk/1/hi/scotland/4707534.stm (accessed 14 Feb 2006).
11. Booth J. Heartache over babies mixed up at hospital. Times Online UK, 2007. http://
www.timesonline.co.uk/tol/news/world/europe/article2629613.ece (accessed 15
Oct 2007).
12. Cohen E. How to avoid falling victim to a hospital mistake, 2009. http://www.cnn.
com/2009/HEALTH/11/11/hospital.mistakes/ (accessed 15 Nov 2009).
13. WHO. WHO launches ‘Nine patient safety solutions’. World Health Organzation,
2007. http://www.who.int/mediacentre/news/releases/2007/pr22/en/index.html
(accessed 10 May 2007).
14. Garnerin P, Ares M, Huchet A, et al. Verifying patient identity and site of surgery:
improving compliance with protocol by audit and feedback. Qual Saf Health Care
2008;17:454e8.
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400

15. Lichtner V. Patient identity, identification and technology [Phd thesis]. London, City
University, 2009.
16. Lichtner V, Wilson S, Galliers J. The challenging nature of patient identifiers: an
ethnographic study of patient identification at a London walk-in centre. Health Informatics
J 2008;14:141e50. http://jhi.sagepub.com/cgi/content/abstract/14/2/141 (accessed 25
Nov 2009). (Special IssuedIntegrated Health Records: Practice and Technology).
17. Mueller FF, Kethers S, Alem L, et al. From the certainty of information transfer to
the ambiguity of intuition. In: Kjeldskov J, Paay J, eds. Proceedings of the 18th
Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and
Environments. Sydney, Australia: ACM Press, 2006:63e70. http://portal.acm.org/toc.
cfm?id=1228175&coll=GUIDE&dl=GUIDE&type=proceeding&CFID=89042704&
CFTOKEN=86521638.
18. BBC News. ‘Piano Man psychotic’ says lawyer. BBC News 2005. http://news.bbc.
co.uk/2/hi/uk_news/england/kent/4182926.stm (accessed 3 Aug 2009).
19. Givon T. Mind, code, and context: essays in pragmatics. Hillsdale, NJ: Lawrence
Erlbaum Associates, 1988.
20. Sperber D, Wilson D. Relevance: communication and cognition. Oxford: Blackwell, 1986.
21. Reddy M, Dourish P. A finger on the pulse: temporal rhythms and information seeking in
medical work. In: Proceedings of the ACM Conference on Computer-Supported
Cooperative Work CSCW 2002. New Orleans, LO: New York: ACM, 2002: 344e53.
22. Schegloff E. Identification and recognition in telephone conversation openings. In:
Psathas G, ed. Everyday language studies in ethnomethodology. New York: Irvington
Publishers, 1979: 23e78.
Qual Saf Health Care 2010;19(Suppl 3):i13ei19. doi:10.1136/qshc.2009.036400
Original research
23. Clark A. Being there: putting brain, body, and world together again. Cambridge,
Mass; London: MIT Press, 1997.
24. Pinch E. Death of a patient lost in the hospital system. icBirmingham, Birmingham
Post, UK, 2006. http://icbirmingham.icnetwork.co.uk/0100news/0100localnews/
tm_objectid¼17062582&method¼full&siteid¼50002&headline¼death-of-a-patient-
lost-in-the-hospital-system-name_page.html (accessed 3 Aug 2009).
25. Kent W. Data and reality. Amsterdam, New York: North-Holland Publishing
Company, 1978.
26. Cummins D. Patient identification: hybrids and doppelgängers. Ann Clin Biochem
2007;44:106e10.
27. Sujan MA, Henderson J, Embrey D. Mismatching between planned and actual
treatments in medicinedmanual checking approaches to prevention: human reliability
associates, for the NPSA. 2004.
28. Perrow C. Normal accidents: living with high-risk technologies. Basic Books, 1984.
29. Falzon P. Reliability management as an individual/collective development process:
a constructive ergonomics viewpoint. In: Activity analyses for developing work,
second symposium of the technical committee ‘activity theories for work analysis and
design’ (ATWAD) of the International Ergonomics Association (IEA); Helsinki, Finland,
12e14 May 2008. http://www.iscar.org/it/Activity2008_CfP.pdf (accessed
May 2010).
30. Microsoft. Design guidance: patient bannerdfirst release, version 1.0.0.0e5 July
2007. Microsoft Health Common User Interface, 2007. http://www.mscui.org/
(accessed 28 Aug 2008).
i19
 Original research
1
Agency for Healthcare
Research and Quality, U.S.
Department of Health and
Human Services, Rockville,
Maryland, USA
2
Department of Emergency
Medicine, University of
Maryland School of Medicine
and Mercy Medical Center,
Baltimore, Maryland, USA
Jon Mark Hirshon - University of
Maryland School of Medicine,
Baltimore, Maryland, USA
3
Department of Emergency
Medicine, University of
Maryland School of Medicine,
Baltimore, Maryland, USA
4
Department of Emergency
Medicine, National Study Center
for Trauma and EMS, University
of Maryland School of Medicine,
Maryland, USA
5
Department of Epidemiology
and Preventive Medicine,
National Study Center for
Trauma and EMS, University of
Maryland School of Medicine,
Maryland, USA
6
Patient Safety Research, Baylor
Health Care System, Dallas,
Texas, USA
7
Feinberg School of Medicine
Northwestern University, and
Northwestern Memorial
Hospital, Chicago, Illinois, USA
8
Healthcare Epidemiology and
Quality, Northwestern Memorial
Hospital, Chicago, Illinois, USA
9
Northwestern Center for
Patient Safety, Northwestern
Memorial Hospital, Chicago,
Illinois, USA
Correspondence to
Dr Kendall K Hall, Center for
Quality Improvement and
Patient Safety, Agency for
Healthcare Research and
Quality, 540 Gaither Road,
Rockville, MD 20850, USA;
kendall.hall@ahrq.hhs.gov
Accepted 17 May 2010
Published Online First
19 August 2010
i20
Incidence and types of non-ideal care events in an
emergency department
Kendall K Hall,1 Stephen M Schenkel,2,3 Jon Mark Hirshon,4,5 Yan Xiao,6
Gary A Noskin7,8,9
ABSTRACT
Aim To identify and characterise hazardous conditions in
an Emergency Department (ED) using active surveillance.
Methods This study was conducted in an urban,
academic, tertiary care medical centre ED with over
45 000 annual adult visits. Trained research assistants
interviewed care givers at the discharge of
a systematically sampled group of patient visits across
all shifts and days of the week. Care givers were asked
to describe any part of the patient’s care that they
considered to be ‘not ideal.’ Reports were categorised by
the segment of emergency care in which the event
occurred and by a broad event category and specific
event type. The occurrence of harm was also
determined.
Results Surveillance was conducted for 656 h with 487
visits sampled, representing 15% of total visits. A total of
1180 care giver interviews were completed (29
declines), generating 210 non-duplicative event reports
for 153 visits. Thirty-two per cent of the visits had at
least one non-ideal care event. Segments of care with
the highest percentage of events were: Diagnostic
Testing (29%), Disposition (21%), Evaluation (18%) and
Treatment (14%). Process-related delays were the most
frequently reported events within the categories of
medication delivery (53%), laboratory testing (88%) and
radiology testing (79%). Fourteen (7%) of the reported
events were associated with patient harm.
Conclusions A significant number of non-ideal care
events occurred during ED visits and involved failures in
medication delivery, radiology testing and laboratory
testing processes, and resulted in delays and patient harm.
Of all types of healthcare settings, the emergency
department (ED) is likely to be the most unsafe for
patients.1 By nature, ED care is transitional and
more likely than other locations to be influenced by
multiple factors such as variable patient acuity,
handoffs, distractions and a lack of information.2e4
Emergency care is complex as a result of the many
people, decision points, uncertainties and over-
lapping processes of care that must be integrated
both within and outside the ED. The care activities
are also tightly coupled, with many time-depen-
dent, critical processes requiring sequential perfor-
mance to achieve a safe, successful outcome.5 An
error that occurs in the ED can compound and
propagate to subsequent care in both the hospital
and ambulatory settings. The combination of
complexity and tight coupling makes the ED
a unique system highly prone to adverse events.6
In the US, the ED represents the first line of care
for over 119 million patients annually.7 This is due
in part to the US governmental mandate that
Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246
anyone presenting to an ED requires an appropriate
medical screening exam regardless of ability to pay
(Emergency Medical Treatment and Labour Act).8
Surprisingly, current knowledge about the
frequency and types of errors and adverse events
occurring within the ED is limited. The majority of
previous studies conducted in the ED have focused
on missed diagnoses of high-stakes conditions,9e11
with only a small number of studies evaluating the
epidemiology of errors and adverse events.12 13 For
hospitals, the method most often used to collect
information about hazards within the ED is the
staff-initiated ‘incident report.’ Many hospitals
implement incident reporting systems with a clas-
sification scheme for types and causes; some have
broadened this approach to develop an all-purpose
safety reporting mechanism. These systems can
provide valuable information for process improve-
ment and identification of hazards, but the quality
of these incident reports is highly variable.14 The
types of reports generated are likely to be non-
systematic sampling of risks and hazards. Further-
more, incident reports provide the so-called
numerator representing the number of reported
incidents that have occurred but lack the denomi-
nator which allows for the determination of the
prevalence.15 The reports are prone to bias as the
decision to classify an event as an incident is made
by the reporter, and does not necessarily reflect
important contributing factors that are precursors
to harm. This type of information is critical to
improving the safety of emergency care.16e18
In this study, we sought to determine the inci-
dence and types of non-ideal care events in an
urban, academic medical centre through the use of
active surveillance on a systematically sampled
group of patient visits.
METHODS
Setting
The study was conducted in the ED of a large
urban, academic medical centre located in the Mid-
Atlantic region of the US, with approximately
45 000 adult visits annually. Paediatric patients,
defined as those 18 years of age or younger, are
treated by paediatric specialists in a separate area of
the ED.
This study was evaluated by the University of
Maryland Institutional Review Board and deter-
mined to be exempt.
Selection of visits
Patient visits to the ED were systematically
sampled based on sequentially generated financial
 Original research

numbers. Patient visits were eligible for inclusion if they met the each RA had at least a Bachelor of Science in nursing or Master’s
following criteria: (1) the sequentially generated financial level degree in a healthcare related field (eg, Master’s in Public
number for the visit ended in a 0 or 5; (2) the visit occurred in Health) and experience collecting data within the ED environ-
the adult ED (patient over the age of 18 years); and (3) the visit ment. RAs were trained in patient safety principles and practised
ended (patient discharged home, transferred, or admitted and querying care givers prior to the initiation of data collection.
moved to an inpatient unit) during the data-collection periods.
This final criteria ensured that all patient safety data obtained Data collection
referred to the entire ED evaluation, from arrival to discharge. RAs contacted all available care givers for sampled patient visits
within 1 h of the patient’s discharge. Care givers contacted for
Definitions interviews were those who were responsible for the patient at
The objective of this project was to identify and quantify the time of discharge. For patients whose visits spanned more
non-ideal care events that occur for patients being treated in the than one shift, only the care givers who were present at the time
ED through active surveillance. Based on the model used in of discharge were interviewed. In this academic hospital with an
infection control, active surveillance is the continuous and Emergency Medicine (EM) residency, there were typically three
systematic collection, analysis and interpretation of informa- care givers identified for each patient: an attending physician,
tion.19 In contrast to passive surveillance, which relies on the a resident physician and a nurse.
initiative of the staff to report events, active surveillance For the interviewed care givers, RAs asked the following
involves direct solicitation of event reports. Active surveillance is questions:
likely to be the most accurate and precise method of measuring What was your role in the patient’s care (eg, nurse)?
events within the ED because it minimises various limitations Was this patient’s care handed-off to you by another staff
found in other methodologies, such as selection bias.20 21 member?
A non-ideal care event was described as any event in the Was there any part of this patient’s care that was not ideal?
patient’s care that the care giver judged to be less than ideal. The If yes, please describe how it was not ideal, and why do you
term ‘non-ideal care event,’ rather than ‘error,’ was developed in think it might have happened?
order to reduce reporting bias by the care givers. This termi- In addition to the interviews, the RAs collected basic demo-
nology was determined, through pilot testing of the surveillance graphic data and visit characteristics (listed in table 1) for the
process, to be less fraught with negative connotations and less sampled patients. No information was collected that could
blame-oriented than the term ‘error.’ Harm was defined as ‘any identify an individual patient or care giver. Care givers were
physical or psychological injury or damage to the health of informed that they could withdraw from participation at any
a person, including both temporary and permanent injury.’22 time during the interview.
Two types of event schemes were developed to facilitate
identification and categorisation of events. These were designed Data analysis
for both face validitydimportant for communication and buy- All reports were entered into a database and were independently
indand the potential link to targeted improvement projects. reviewed by two of the investigators (KKH, SMS). The review
First, the research team outlined the basic steps in the process of involved categorising the events into the Segment of Care during
delivering emergency care. These segments of care are as follows: which the event occurred, and the Event Category and Specific
triage, registration, evaluation, diagnostic testing, treatment and Event Type. Discrepancies between the two reviewers were
disposition. An additional category, transitions, was added to resolved through discussion. Throughout the review, continuous
account for periods where an event occurred during the transfer reference was made to previously evaluated events in order to
of care from one clinician to another. Other safety researchers ensure consistency. The categorisation of any event could be
have successfully used similar systems for categorising errors questioned at any point in the analysis. The conflict would then
into the care delivery segment during which they occurred.23 be resolved through further discussion.
The second categorisation scheme was developed to describe the
broad event category (eg, medication delivery; laboratory Table 1 Patient demographics and visit characteristics
testing) and specific event type (eg, administrationdwrong Patient demographics Visit characteristics
drug; resultsddelay or failure to report results). This taxonomy Sex Primary location
was derived from the hospital’s online event reporting system Male 46% Acute care 67%
and iteratively refined as the study progressed and reports Female 54% Urgent care 32%
became available. Age range Hallway 1%
19e29 years 26% Discharge disposition
Care giver and research assistant education 30e39 years 20% Home 66%
Staff members were educated to the goals of the study and the 40e49 years 27% Medical/surgical unit 21%
definition of non-ideal care events through regular staff meetings, 50e59 years 16% Intensive care unit 1%
memoranda and emails. Residents, through their regular weekly 60e69 years 5% Against medical advice 3%
conference, were educated as part of a quality and patient safety 70+ years 6% Rapid diagnostic unit 2%
lecture series. Emphasis was placed on the non-punitive nature of Race/ethnicity Other 5%
the study and the appreciation of systems factors, rather than Black 77% Time of discharge
individual blame, as related to events in the ED. Care givers were White 20% Day shift 32%
instructed to report any events through the standard hospital Hispanic 1% Evening shift 40%
event-reporting system, as they would through their normal Asian 1% Night shift 28%
course of work. Cards were given to all staff with information on Other 1% Length of stay
Language Median 341 min
how to access the online reporting system.
English 99% Range 30e2964 min
Four research assistants (RAs), from a cohort of trained RAs at
Spanish 1%
the medical centre, participated in the study. For this project,

Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246 i21

 Original research

The same investigators (KKH, SMS) reviewed all reports and, Table 2 Events by category and type (N¼210)
when there was sufficient information, determined whether or No. (no. with
not any events had associated harm. Both reviewers had to agree Category Type known harm)
on the occurrence of harm, discussing each event to resolve any Care transfer or discharge Inpatient bed availabilityddelay or failure 31
conflict. Patient left ED before completion of visit 11
Delay in transfer of care not related to bed 7
availability
RESULTS
Incomplete discharge 2
A total of 656 h of RA surveillance was conducted with 487
Inadequate follow-up 2
visits sampled for inclusion, representing approximately 15% of Laboratory testing Resultsddelay or failure to report 20
the total visits during the study period. The surveillance repre- Performanceddelay or failure to obtain 4
sented 82 shifts (26 day, 28 evening and 28 night shifts) over specimen
a 15-week period. Of the 487 visits, there were five visits for Orderingddelay or failure to order 3
which it was not possible to obtain care-giver interviews. The Performancedlost or misplaced 2 (1)
basic patient demographics and visit characteristics for the 482 specimen
visits with associated interviews are described in table 1. Performancedhaemolysed specimen 2
There were 1180 interviews conducted, with a median of Resultsddelay or failure to act on results 2
three interviews conducted for each of the 482 sampled visits. Radiology testing Performanceddelay or failure to perform 13
Resultsddelay or failure to report results 5
Interviews were conducted with nurses (39% of interviews),
Performancedwrong study performed 3 (1)
resident physicians (32%), attending physicians (28%) and
Orderingddelay or failure to order 1
technicians (1%). There were 27 attempted interviews for which
Orderingdwrong study ordered 1
the care giver declined to participate (2%). Since no individual
Resultsddiscrepancy in reading 1
identifying information was collected for the care givers, it is not ED evaluation Delay or failure to evaluate 10
known how many unique individuals these 27 interview Mis-triage (includes over- and 5
attempts represent. undertriage)
Through the 1180 interviews, 263 reports of non-ideal care Inaccurate or incomplete triage evaluation 4 (1)
events were generated (53% by nurses, 28% by resident physi- Prolonged time in waiting room 3
cians, 19% by attending physicians, 0% by technicians). The 263 Inaccurate evaluation 1 (1)
reports represented 210 non-duplicative events occurring in 153 Other 1
of the 482 (32%) visits. Approximately one-third of visits had at ED consultation Consultant evaluationddelay or failure 18
least one associated non-ideal care event reported with 13% of Consultant communicationddelay or 2
failure
the visits having more than one event.
Inadequate resident or fellow supervision 1 (1)
Of the 482 visits with interviews, 187 (39%) had at least one
Medications Dispensingddelay or wrong time 5
interview of a care giver who received the patient through
Medication not available in ED 3
a handoff. For the 153 visits with reported events, 90 (59%) had
Orderingddelay or failure 2
at least one interview with a care giver who received the patient Administrationddelay or failure 2
through a handoff. Adverse drug event 1 (1)
The non-ideal care events were categorised by the segment of Medication not available from pharmacy 1
care in which they occurred (figure 1). These events occurred Orderingdwrong dose 1
most commonly during the diagnostic phase (29%), which Orderingdwrong time 1
included laboratory and radiology testing. Events during the Dispensingdwrong route 1
disposition phase (eg, no inpatient bed availability) accounted Procedures Intravenous accessdunanticipated 7 (7)
for 21% of the discrete events. difficulty obtaining access
The results as categorised by event category and specific event Other proceduresdunanticipated 2
difficulty
type are listed in table 2. A significant portion of events reported
Intravenous accessdintravenous 1 (1)
were related to delays or failures, particularly for laboratory and malfunction
radiology study and result acquisition. Some examples of reports Communication 6
are listed in table 3. Documentation 6
Patient factors (eg, intoxication, behavioural issues) 5
Patient complaints 4
Other testing 1
Not enough information to classify 5
ED, Emergency Department.

Thirteen visits had a total of 14 events that were judged to be

Figure 1 Non-ideal care events by segment of emergency care.
associated with harm. The majority of these harms were related
to repeated attempts at obtaining intravenous access. Other
events that led to recognised harm included a case of an
incomplete triage evaluation that resulted in a significant delay
in definitive care for a patient with an acute coronary syndrome;
an incomplete initial ED evaluation that led to a missed diag-
nosis that was discovered by the subsequent shift; an allergic
reaction to a medication; and a case where definitive treatment
was delayed because of a reported lack of resident supervision.

i22 Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246


Table 3 Examples of non-ideal care event reports
Event category and type
Care transfer or discharge: inpatient bed availabilityddelay or
failure
Care transfer or discharge: inpatient bed availabilityddelay or
failure
Laboratory testing: resultsddelay or failure to report
Laboratory testing: resultsddelay or failure to report
Radiology testing: performanceddelay or failure to perform
DISCUSSION
The method of proactively querying care givers regarding non-
ideal care events used in this project is a feasible method for
identifying safety issues within the emergency department. The
method was deployed successfully to systematically sample
patient visits, with success rates of more than 97% in interview
attempts. Of the 482 visits for which clinician interviews were
conducted, 32% had at least one non-ideal care event, with 13%
having more than one reported event. Three per cent of the visits
had an event which caused known harm. The segments of care
incurring the greatest risks to patient safety were the diagnostic
testing phase, followed by the disposition phase. The specific
event type, delay or failure of bed availability for admitted
patients, had the greatest number of reports.
The method most often used to collect information about
hazards within the ED is the staff-initiated voluntary reporting
system. These systems, whether paper-based or electronic, are
widely accessible in many US hospitals and provide a standard
classification scheme for events across the individual institution.
A significant limitation of these systems is the lack of consis-
tency in reporting: what one individual considers a reportable
event may not match the impression of another. Implementa-
tion of incident reporting systems is variable, leading to incon-
sistent data potentially with significant biases. These systems
cannot provide reliable information regarding the incidence or
prevalence of errors and adverse events,21 making assessments of
the success or failure of patient safety interventions unreliable.
In addition, these types of reporting systems tend to focus
primarily on adverse events causing harm and may miss
important factors that are precursors to errors.
Studies evaluating ED events often ask clinicians to provide
information about ‘errors’ and ‘adverse events.’ However
reported, whether through in-person interviews or through
standardised reporting mechanisms, the clinician must make
a judgement as to what constitutes an error or adverse event.
This is particularly difficult in an environment where reporting
is associated with negative consequences for the individuals
involved. Missing lab results, delayed consultation or even a lack
of inpatient beds necessitating holding patients in the ED
(‘boarding’) may not be reported, yet each of these events has
the ability to harm the patient.24e27
As demonstrated in this study, ED staff clearly have their own
sense of what is a non-ideal event. As might be expected in
a highly time-dependent location, such as the ED, many of the
reported events had to do with delays in care, specifically delays
to diagnostic tests or delays in provision of a bed for an admitted
patient. These are far more subtle findings than the typically
Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246
Original research
Person reporting event Non-ideal care event description
Nurse ‘The patient has been waiting for a bed all day. He was to be
admitted to a (telemetry) bed, but since he’s been here so long and
doing better, the docs decided to change the bed request to
a general medicine bed. He just started the cycle again, and is still
waiting for a bed.’ (Total length of stay in ED: 22 h, 24 min.)
Attending ‘Patient was to be a direct admission to the Internal Medicine
service, but the bed was not available and was therefore sent to the
ED. She later got a bed and was transferred to the floor.’ (Total
length of stay in ED: 5 h, 27 min)
Nurse ‘Delay in lab resultsd(they were) ordered from the day shift, but
weren’t processed until (the night shift nurse) asked them’
Resident ‘The lab results took 3e4 h. I kept having to call.’
Nurse ‘Delay in CT scandpatient got contrast at 17:00, but CT scan was
not done until 20:30. We’re waiting for the CT scan results now
(21:41).’
reported medical errors, and more common than even medica-
tion errors. In pointing out these non-ideal events, ED staff are
suggesting the degree to which interventions to improve daily
ED care also represent interventions that would bolster patient
safety. The reported events demonstrate to what degree activi-
ties of daily care and patient safety are closely intertwined:
when processes are broken, the quality of care has the potential
to suffer.
Event categorisation and taxonomy of reporting remain
a substantial challenge for any reporting system. Through iter-
ative evaluation of events, we developed a system of catego-
risation that worked for this series of events and that maintains
face validity within the work stream of the ED. One of the
reasons for the development of our system of categorisation was
that our institutional event reporting system provided no clear
strategy for categorising a number of the reported events. The
ED provides a unique environment, as, we suspect, do many
other areas of the healthcare system. A significant challenge is
presented in balancing the need for an institution-wide reporting
system with a system that meets the needs, both in mechanism
and in content, of a local environment. Categorisation affects
how we think about and respond to events. The choice between
local categorisation with face value and broad categories that
may not fit local circumstances may influence the ability to
develop and track interventions in a meaningful and effective
way.
In a majority of cases, there was no clear report of harm
associated with events. This appears to be partly a result of the
encouragement to report on any non-ideal event. It also indi-
cates another challenge in the collection of patient safety data in
a location where there is a relatively transient interaction with
the patient along the entire continuum of care. ED care typically
represents early care in what may be a prolonged episode of care
or hospitalisationdfollow-up on this more extended care was
not available to the study investigators and is not typically
available to ED staff. For patients who are discharged and return
home, clinical follow-up is also rarely available to ED staff. The
study was based on concurrent evaluation of non-ideal events,
and so reflects staff perception of the state of patient safety
while the patient was still in the ED. Therefore, it is perhaps not
surprising that when attempting to identify harm, the vast
majority of events fell into an ‘uncertain’ categorydcare givers
had no way of knowing whether harm might be in the process
of developing. Simply stated, from active surveillance in the ED,
it is very difficult to tell whether a non-ideal event leads to any
definitive harm. This raises a challenge for any reporting system
in an ED environment. Active reporting through surveillance, as
i23
 Original research
practised in this study, is likely to provide the fullest stories and
therefore the most actionable results, but it appears to bias the
results towards a collection of events that led to less harm, not
because harm did not occur, but because the staff simply do not
know at the time the question is asked. This may also be
a reason that incident-reporting systems are used so infrequently
in transient settings such as the EDdif the threshold for
reporting is harm, then that threshold will only rarely be
definitively met.
Limitations
Despite the large number of reports, as with any reporting
system relying on the care giver to determine if an event has
occurred, this method is likely to underestimate the true number
of events. In addition, since the care givers were queried at the
end of the patient visit, it could be that the patient had been in
the ED across multiple shifts with multiple care givers who were
more aware of events occurring earlier in the course of care. A
goal of this study was to establish the prevalence of events by
knowing the number of patients for whom events were reported
(the numerator) and the number of patients for whom the
clinicians were interviewed (the denominator). A large number
of events were identified, but the limitations noted here suggest
that this could still be an underestimate. To increase reporting
would require tackling the challenge from both sides by devel-
oping more continuous methods that would query providers
throughout the patients’ course while also tracking the patient
in follow-up to see whether non-ideal ED events became known
later in the patients’ course of illness.
The agreement between professions (eg, nurse perception vs
physician perception) as to what constitutes non-ideal care was
not addressed in this study. Only 37 of the 153 visits with events
had duplicate reports of the same event. This phenomenon may
be a result of the individual’s role in the care of the patient:
a physician who is waiting for a radiology study to diagnose
a patient is likely to be more sensitive to issues with process
delays or failures originating in the Radiology Department.
Similarly, a nurse who is responsible for transferring a patient to
the inpatient unit is likely to be more sensitive to delays related
to a lack of inpatient beds.
Reporting will naturally be biased by the nature of the ques-
tions asked and the environment in which it is asked. ED staff
constantly work under the pressure of time. The pressure is both
real, as that faced when there are sick patients and a full waiting
room, and artificial, as found in standard ED management
variables such as length of stay and time to evaluation. To the
degree that staff are always thinking about time, delays in care
are more likely to be reported as non-ideal events. The mecha-
nism used here to ask about eventsdspecifically avoiding any
mention of error or harmdmay have biased responses towards
a larger number of less potentially dangerous reports. Respon-
dents may have assumed that ‘big’ eventsdthat is, events
causing harmdshould be reported by the more traditional
incident reporting system or not at all.
The simplification of adverse events into categories results in
a trade-off between a loss of detail that categorisation invokes
and the broader understanding that such categorisation allows.
We used iterative analysis to develop categories for events while
consistently reviewing all of the event details. The two mech-
anisms developed here, with categorisation based on the stage of
ED care and by nature of the event, made sense within the
context of the events reported and the clinical experience of the
investigators. While these mechanisms allowed for the produc-
tion of appropriate categories that reflected this data set, there is
i24
hesitance to recommend them as a rigid taxonomy.28 Ultimately,
any system of categorisation must survive first based on its face
validity and subsequently, and most importantly, on its useful-
ness in directing interventions.
Conclusions
One in three patients experienced a non-ideal care event during
their course of ED care. The majority of these events had to do
with delays and failures in the process of care, from delays and
failures in the reporting of laboratory results to delays in
obtaining inpatient beds for admitted patients. Despite the large
number of reported events, relatively few were identified as
causing patient harm, though the assessment of harm appears to
be severely limited in the ED. This limitation sets emergency
care apart from other more discrete episodes of hospital care
such as surgical procedures. Much of emergency care occurs in
a grey zone where the outcomes are reserved for another time
and place. While likely an underestimate, these results provide
an estimate for the total number of non-ideal events that occur
in the course of ED care.
Acknowledgements The authors would like to thank B Browne (University of
Maryland) and JG Adams (Northwestern University), for their support and guidance
with the project, and K Henriksen and JB Battles, for their review of the manuscript.
Funding This project was supported by grant no P20HS017111 from the Agency for
Healthcare Research and Quality.
Competing interests None.
Ethics approval Ethics approval was provided by the University of Maryland IRB.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Biros MH, Adams JG, Wears RL. Errors in emergency medicine: a call to action.
Acad Emerg Med 2000;7:1173e4.
2. Chisholm CD, Dornfeld AM, Nelson DR, et al. Work interrupted: a comparison of
workplace interruptions in emergency departments and primary care offices. Ann
Emerg Med 2001;38:146e51.
3. Clancy CM. Care transitions: a threat and an opportunity for patient safety. Am J
Med Qual 2006;21:415e17.
4. Kovacs G, Croskerry P. Clinical decision making: an emergency medicine
perspective. Acad Emerg Med 1999;6:947e52.
5. Perrow C. Normal Accidents: Living with High Risk Technologies. New York: Basic
Books, 1984.
6. Cook R, Woods D. Operating at the sharp end: the complexity of human error. In:
Bogner M, ed. Human Error in Medicine. Hillsdale, NJ: Lawrence Erlbaum Associates,
1994.
7. Pitts SR, Niska RW, Xu J, et al. National Hospital Ambulatory Medical Care Survey:
2006 emergency department summary. National health statistics reports; no 7.
Hyattsville, MD: National Center for Health Statistics, 2008.
8. Emergency Medical Treatment and Active Labor Act (EMTALA). 42 USC 1395dd,
1986.
9. Kachalia A, Gandhi TK, Puopolo AL, et al. Missed and delayed diagnoses in the
emergency department: a study of closed malpractice claims from 4 liability insurers.
Ann Emerg Med 2007;49:196e205.
10. Pope JH, Aufderheide TP, Ruthazer R, et al. Missed diagnoses of acute cardiac
ischemia in the emergency department. N Engl J Med 2000;342:1163e70.
11. Wei CJ, Tsai WC, Tiu CM, et al. Systematic analysis of missed extremity fractures in
emergency radiology. Acta Radiol 2006;47:710e17.
12. Fordyce J, Blank FS, Pekow P, et al. Errors in a busy emergency department. Ann
Emerg Med 2003;42:324e33.
13. Thomas M, Mackway-Jones K, Boreham N. Derivation of a typology for the
classification of risks in emergency medicine. Emerg Med J 2004;21:464e8.
14. Cullen DJ, Bates DW, Small SD, et al. The incident reporting system does not
detect adverse drug events: a problem for quality improvement. Jt Comm J Qual
Improv 1995;21:541e8.
15. Hall KK, Xiao Y. Safety in numbers: the value of counting adverse events. Crit Care
Med 2008;36:2196e7.
16. Battles JB, Stevens DP. Adverse event reporting systems and safer healthcare.
Qual Saf Health Care 2009;18:2.
17. Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med
1989;320:53e6.
18. Vincent C. Understanding and responding to adverse events. N Engl J Med
2003;348:1051e6.
Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246
 Original research

19. Lee TB, Baker OG, Lee JT, et al. Recommended practices for surveillance.
Association for Professionals in Infection Control and Epidemiology, Inc. Surveillance
Initiative Working Group. Am J Infect Control 1998;26:277e88.
20. Nuckols TK, Bell DS, Paddock SM, et al. Contributing factors identified by hospital
incident report narratives. Qual Saf Health Care 2008;17:368e72.
21. Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen
Intern Med 2003;18:61e7.
22. Anon. Standardizing a Patient Safety Taxonomy. Washington, DC: National Quality
Forum, 2006.
23. Schiff GD, Kim S, Abrams R, et al. Diagnosing Diagnosis Errors: Lessons from
a Multi-Institutional Collaborative Project. Advances in Patient Safety. Vol. 2.
Rockville, MD: Agency for Healthcare Research and Quality, 2005:255e78.
24. Bernstein SL, Aronsky D, Duseja R, et al. The effect of emergency
department crowding on clinically oriented outcomes. Acad Emerg Med
2009;16:1e10.
25. Chalfin DB, Trzeciak S, Likourezos A, et al. Impact of delayed transfer of critically ill
patients from the emergency department to the intensive care unit. Crit Care Med
2007;35:1477e83.
26. Hwang U, Richardson L, Livote E, et al. Emergency department crowding and
decreased quality of pain care. Acad Emerg Med 2008;15:1248e55.
27. Richardson DB. Increase in patient mortality at 10 days associated with emergency
department overcrowding. Med J Aust 2006;184:213e16.
28. Nemeth CP, Cook RI, Wears RL. Studying the technical work of emergency care.
Ann Emerg Med 2007;50:384e6.

Qual Saf Health Care 2010;19(Suppl 3):i20ei25. doi:10.1136/qshc.2010.040246 i25

 Original research
1
Baylor College of Medicine,
Texas Children’s Hospital,
Houston, Texas, USA
2
Houston VA Health Services
Research and Development
(HSR&D) Center of Excellence,
Michael E DeBakey Veterans
Affairs Medical Center and the
Section of Health Services
Research, Department of
Medicine, Baylor College of
Medicine, Houston, Texas, USA
3
Texas Children’s Hospital,
Houston, Texas, USA
4 paediatricians, nurses, a social worker and a parent of
Baylor College of Medicine,
Texas Children’s Health Plan,
Houston, Texas, USA
Correspondence to
Dr Virginia A Moyer, Baylor
College of Medicine, Texas
Children’s Hospital, 6621 Fannin
Street, Suite CCC 1540,
Houston, TX 77030 USA;
moyer@bcm.edu
Accepted 20 June 2010
i26
Transitions from neonatal intensive care unit to
ambulatory care: description and evaluation of the
proactive risk assessment process
Virginia A Moyer,1 Hardeep Singh,2 Karen L Finkel,3 Angelo P Giardino4
ABSTRACT
Objective Over 20 000 US neonates annually make the
potentially risky transition from the neonatal intensive
care unit (NICU) to the care of primary care physicians
whom they have never met. The authors describe the
use of Health Care Failure Modes and Effects Analysis
(HFMEA) to proactively assess the risks of this transition,
and present a qualitative evaluation of the HFMEA
process.
Methods The HFMEA team, led by a patient safety
specialist, included neonatologists, general
a premature infant. The authors conducted a facilitated
debriefing session with the team, interviews of key
informants, and a content analysis of documentation
generated throughout the project.
Results The authors identified 40 high-risk failure modes
and 75 associated high-risk causes. Clear thematic
categories included poor communication among care
providers in the hospital, between care providers and
parents/care givers, or between the hospital-based and
ambulatory care providers, as well as a lack of
knowledge and skills among community-based providers
to care for fragile infants. Evaluation of the HFMEA
process revealed a high level of involvement, with over
250 h of professional time devoted to the process,
agreement that the interaction itself was valuable and
concerns about the limitations of the HFMEA method in
capturing the complexity of the transition from NICU to
ambulatory care.
Conclusion While HFMEA holds promise for improving
the safety of care transitions, the full effort required to
realise the potential benefit requires additional evaluation
to confirm its value over less intensive means of
achieving safer care transitions.
INTRODUCTION
The patient safety literature increasingly acknowl-
edges the potential risks of the transitions that
patients make between healthcare providers and
settings as their care needs evolve.1 2 Infants born
prematurely or with complex congenital abnor-
malities are surviving to discharge in growing
numbers. Over 20 000 neonates make the risky
transition from the neonatal intensive care unit
(NICU) to home each year in the USA, often to the
care of primary care physicians (PCPs) they have
never met.3 Efforts to minimise lengths of stay in
the NICU contribute to increased complexity of
healthcare needs at the time of discharge.4 To
ensure safe and effective monitoring and coordina-
tion of care for fragile neonates, an efficient
Qual Saf Health Care 2010;19(Suppl 3):i26ei30. doi:10.1136/qshc.2010.040543
‘handoff ’ from the intensive care specialist to the
ambulatory care provider is critical. Despite their
potential significance, little is known about the
specific vulnerabilities that involve care transitions
from the NICU to home.
Studies from the adult literature reveal signifi-
cant vulnerabilities around the point of discharge
from the hospital. About half (49%) of hospitalised
adults experience at least one medical error
following hospital discharge,5 and approximately
one-fifth (19e23%) of discharged patients experi-
ence an adverse event.6e8 Most errors and adverse
events in this transition involve communication
breakdowns in handoffs between inpatient care
teams, patients (or their care givers) and/or PCPs.7
However, few published studies describe the risks
of care transitions in care in the paediatric popu-
lation. Previous studies have measured attendance
at follow-up appointments using a conceptual
approach that implies that the root of the problem
is lack of compliance on the part of care givers,
rather than systematic difficulties that lead to
delayed or inadequate follow-up. One study found
that 28% of children discharged from paediatric
(not neonatal) intensive care did not receive timely
medical follow-up.9 Most published research on
this transition in paediatric patients focuses on the
effect of insurance on follow-up or describes early
discharge programmes. To prevent potential errors
and harm in these care transitions, a thorough
understanding of the risks of the discharge process
of high-risk neonates is needed. Like many patient
safety issues that deal with provider and patient
interactions, adverse events related to care transi-
tions can be viewed from a human factors
perspective in that poorly designed systems permit
less than ideal interactions to occur between the
people in the care environment as it shifts from one
venue to another.
Proactive evaluation of error-prone healthcare
processes may inform interventions to prevent
adverse patient outcomes. Health Care Failure
Mode and Effect Analysis (HFMEA) is one such
approach to improving patient safety. HFMEA is
a systematic, team-based method of identifying
potential errors and risks within healthcare
processes and prioritising targets for intervention or
redesign. The method is well described by DeRosier
et al and has been successfully implemented in the
Veterans Affairs health system.10 To our knowl-
edge, HFMEA has been used in only two published
paediatric studies, neither of which involved
neonates.11 12 Furthermore, HFMEA has not been
used in any setting to study care transitions during

the discharge process. We applied HFMEA to identify risks in
care transitions involving the discharge of newborns with
complex conditions. We also assessed the team’s perceptions of
advantages and challenges of using HFMEA in the high-risk
neonatal setting.
METHODS
We conducted the HFMEA at Texas Children’s Hospital (TCH),
a freestanding paediatric hospital currently licenced for 639 beds,
including a 76-bed Level III NICU and a 62-bed Level II nursery.
The 10-member HFMEA team was led by a hospital-based
patient safety specialist and included neonatologists, general
paediatricians, nurses, discharge planners, a social worker and
a parent of a premature infant. The team met from October
2007 to April 2008. The institutional review board of Baylor
College of Medicine approved the study.
Application of the HFMEA methodology
Using the HFMEA methodology outlined by DeRosier et al., the
team initially developed a flow diagram that identified the main
processes and subprocesses involved in the discharge of a high-
risk infant.10 Once members agreed that all pertinent steps were
listed, the team brainstormed for all potential errors that might
occur (failure modes) at each step of the process. Each failure
mode was then scored on two parameters: the probability that it
could occur (frequency score) and the severity of the potential
outcome if it did occur (severity score). Frequency and severity
were rated on a numerical scale and decided by group consensus.
For each identified failure mode, the product of the frequency
and severity scores was computed to generate a hazard score.
Potentially ‘high-risk’ failure modes were defined as those with
hazard scores of at least 50% of the maximum possible score.
The HFMEA Decision Tree tool was used to identify oppor-
tunities for intervention.13 Each low-scoring failure mode was
evaluated to determine whether it was a single-point weakness,
that is, a step in the process that is so critical that its failure
would result in system failure or an immediate adverse event.
Single point weaknesses were reclassified as ‘high risk.’ Next, we
eliminated vulnerabilities for which effective control mecha-
nisms were already in place and failure modes that were so
obvious that no mitigation plan would be needed. The team
then listed potential contributory factors (‘causes’) for each of
the remaining high-risk failure modes and scored these factors
using an approach similar to that of the failure modes. The
resulting final list of high-risk failure modes and causes was used
to guide the development of a comprehensive mitigation plan.
Qualitative assessment of HFMEA
Because the HFMEA methodology had not been used previously to
study the neonatal discharge process, qualitative data (described
below) were collected from participants at the end of the project to
learn about their experiences. A list of challenges that were
encountered by the team and their possible solutions was gener-
ated by synthesising data from the following three sources:
1. A facilitated debriefing session was held with the entire team
to assess team members’ perceptions of the components of
the HFMEA method and the application of HFMEA to care
transitions within the NICU population. Team members’
opinions were solicited about what went well during the
HFMEA project and what did not, their perceptions of the
value of HFMEA after its evolution over time and suggestions
for improving the process.
2. Key informants (principal investigator, patient safety
specialist, social worker, general paediatrician, neonatologist
Qual Saf Health Care 2010;19(Suppl 3):i26ei30. doi:10.1136/qshc.2010.040543
Original research
and a parent) were then interviewed to expand upon issues
identified in the team debriefing. Additional issues were
explored related to the time commitment required from team
members, the comfort level of team members with their skills
in this area, the resources needed and technical aspects of the
HFMEA method.
3. A content analysis of all documentation maintained by the
patient safety specialist, including meeting agendas and
detailed minutes, was conducted to enrich findings from
the debriefing and the interviews.14
In keeping with qualitative methodology, data collection and
analysis were overlapping processes.15 Transcribed notes were
analysed to identify and organise content themes that emerged
from an inductive reading and rereading of the notes from all
three data sources by one investigator (AG). These themes were
compiled into a list that was iteratively revised as patterns
within the data became more apparent and consensus among
the investigators was achieved.
RESULTS
Application of HFMEA
The team held twenty 1.5 h meetings from 1 October 2007 to 30
April 2008. The team began by developing a high-level flow
diagram of the entire discharge process, which the team defined
as starting with identifying the patient for discharge and ending
when a follow-up appointment takes place. The team broke
down each of the high-level steps into multiple individual
substeps. Figure 1 shows the resulting diagram of the discharge
process from the NICU to a primary care paediatrician.
The team identified 114 potential failure modes in the
discharge process as mapped in the flow diagram. To score failure
modes and causes, we initially used the definitions published by
DeRosier et al.10 However, the team found the original HFMEA
scoring system unsuitable to grade events in the NICU care
transition. Members found the subjective descriptions anchoring
each severity score to be not relevant to the neonatal population,
and rewrote the descriptions using language that better reflected
their experience in the NICU. Similarly, the team found the
frequency scores at the more frequent end of the scale to inad-
equately distinguish among frequency differences that were felt
to be important in this population. For example, the lowest
frequency score in the original scoring system is ‘Frequent.
may happen several times in 1 year’; the team agreed that many
of the events described in the flow diagram were likely to occur
as often as daily. The team added three additional frequency
categories to distinguish among these very frequent events.
The revised, NICU-specific definitions (see tables 1, 2) were
felt by the team to improve the clarity of severity ratings and
specificity of probability ratings such that the process of scoring
generated less discussion and moved much more quickly after
adoption of the revised system.
Because the hazard scores are simply the product of the severity
and probability scores, the team recognised that simultaneously
scoring the severity and the probability of an event might result
in team members adjusting some of the scores to make sure that
a certain failure mode or cause would be included in the final
model. To explore that possibility, the team rescored one step of
the process independently. Figure 2 shows an example of
a comparison of the scores obtained when one substep scored for
severity and probability simultaneously versus those obtained
when the same substep was scored independently at a later
meeting. The team rescored Step 1 of the process and found that
although most of the Hazard Scores changed somewhat, few
changed enough to reclassify the failure mode from low risk to
i27
 Original research
Figure 1 High-level process map with
substeps.
high risk or vice versa. For consistency, the team decided to use the
original scoring to develop the final model.
The final model included 40 high-risk failure modes and 75
high-risk causes. Although some, such as insurance-related
causes, were eliminated as events not under clinician or hospital
control, the following issues were present across most of the
identified failure modes and causes:
1. Healthcare providers (attending physicians, consulting physi-
cians and other healthcare providers) in the NICU tend to act
in isolation, which results in lack of a standardised, coordi-
nated and comprehensive plan of care.
2. Parents/care givers may be inadequately prepared for home
care and management of fragile neonates due to a lack of
consistent and early communication between parents and
NICU staff and lack of coordinated educational and social
services support programmes prior to discharge.
3. Community providers (including PCPs, home health nurses,
pharmacists, and community emergency rooms) may lack
the required knowledge and skills to manage complex infants,
leading to suboptimal office-based care and perceived over-
utilisation of the emergency system.
i28
Once these care transitions issues were identified, the team
developed a comprehensive plan for corrective action. The plan
addresses coordination of information sharing between and
among care providers within the NICU, improving parent/care
giver preparation to assume responsibility for the child’s care
upon discharge, and enhancing skill and knowledge of PCPs who
receive these infants into their practices after discharge from the
NICU.
Process evaluation
Analysis of qualitative data revealed several consistent themes.
Overall, the team members thought that the group functioned
extremely well, with a high level of involvement from most
members and many new insights gained in the process. All team
members agreed that the transition from the NICU to ambu-
latory care was an important and often risky process that
warranted proactive risk assessment. Further, there was broad
agreement that the time spent during the group activity of
defining all of the various steps in the discharge process led to
novel discussions in which knowledge about discharges and care
transitions emerged in a new way, and information was shared
Qual Saf Health Care 2010;19(Suppl 3):i26ei30. doi:10.1136/qshc.2010.040543
 Original research

Table 1 Revised severity categories

Category (score) Outcome type Description of outcome(s) within category
Catastrophic (4) Patient outcomes Original HFMEA: Death or major permanent loss of function, suicide, rape, haemolytic transfusion reaction, surgery/
procedure on wrong patient or body part, or infant abduction. Revised: It is realistic to think that this failure mode could
result in death or serious disability.
Equipment or facility outcomes Damage equal to or more than $250 000
Major (3) Patient outcomes Original HFMEA: permanent lessening of body functioning (sensory, motor, physiologic, or intellectual), disfigurement,
surgical intervention required, or increased length of stay or increased level of care for three or more patients. Revised: this
failure mode could (but is unlikely to) really hurt someone or result in a much longer length of stay.
Equipment or facility outcomes Damage equal to or more than $100 000
Moderate (2) Patient outcomes Original HFMEA: increased length of stay or increased level of care for one or two patients. Revised: this failure mode
could extend lenght of stay but is unlikely to be fatal, or could adversely affect health, but not severe.
Equipment or facility outcomes Damage more than $10 000 but less than $100 000
Minor (1) Patient outcomes Original HFMEA: no injury, not increased length of stay, nor increased level of care. Revised: slight increase in length of
stay, no effect on health outcomes.
Equipment or facility outcomes Damage less than $10 000 or loss of any utility without adverse patient outcomes (eg, power, natural gas, electricity,
water, communications, transport, heat/air conditioning)
HFMEA, Health Care Failure Modes and Effects Analysis.

for the first time among the various participants in the discharge
process.
Concerns about the HFMEA process included the substantial
time commitment required to adequately apply the HFMEA
method and the relevance of the original HFMEA scoring system
to the NICU setting, discussed previously. Moreover, team
members uniformly felt that the HFMEA method was limited in
capturing the high degree of clinical complexity inherent in the
transition from the NICU to a primary care paediatrician.
The total time commitment from all team members for the
HFMEA was substantial. The median number of meetings
attended by each member was 16, and no member attended
fewer than half of the meetings, for a total of approximately
230 h of professional time spent in the HFMEA meetings. The
patient safety specialist and the principal investigator together
spent an additional 40e50 professional hours outside the group
sessions in preparation for meetings. We estimated the value of
the professional hours involved in the HFMEA meetings at
approximately $20 000.
DISCUSSION
We used the HFMEA technique to conduct a proactive risk
assessment of the complex process of discharge to ambulatory
follow-up for high-risk neonates, and identified a large number
of high-risk failure modes and associated causes. Many of these
related to lack of communication among care providers in the
hospital, between care providers and parents/care givers, or
between hospital-based and ambulatory care providers. A second
common theme was a lack of knowledge and skills among
community-based care providers responsible for the care of
fragile infants after discharge. To our knowledge, this report is
among the first to focus on a systematic risk assessment in the
inpatient-to-outpatient transition for fragile infants.
Our findings are of potentially great significance to the
increasing number of infants who survive the neonatal period
and are discharged to the community with ongoing complex
healthcare needs. Survival of extremely low-birth-weight infants
(birth weight <1000 g) increased by 35% between the 1980s and
the 1990s, with a concomitant rise in disabilities such as chronic
lung disease and neurodevelopmental impairment.3 In one large
study, the incidence of chronic lung disease (defined as receiving
supplemental oxygen at 36 weeks’ postmenstrual age) was 23%,
severe intracranial haemorrhage 11% and proven necrotising
enterocolitis 7%. Poor postnatal growth was seen in almost
100% of infants born at <1000 g.16 Each of these outcomes
presents challenges in long-term management of babies
following NICU discharge. Given the increasingly complex
healthcare needs that community paediatricians must address, it
is critical to understand and improve the transition from NICU
to ambulatory care.
Potential risks associated with the transition of patients from
hospital to ambulatory follow-up are increasingly recognised.5 17
Unlike the usual inpatient ‘handoff,’ the time period between
discharge and ambulatory follow-up is prolonged, and lines of
responsibility for the patient in the interim are unclear. Despite
the increasing availability of electronic health records, follow-up
providers may not have complete or accurate information about
the patient’s condition. Although most studies of adverse events
around discharge are from the adult literature, many of these are
potentially relevant to the NICU setting. Forster et al, using
a telephone interview, found that 19% of adult patients had
identifiable adverse events in the 3 weeks following discharge
from a general medicine service, with adverse drug events being
most common. Similar results were found in a study of patients
discharged from Canadian hospitals.6 Beers et al found that 73%
of older patients failed to use at least one medication as it was
ordered at the time of hospital discharge.18 Coleman found

Table 2 Revised probability categories

Score Description in original Health Care Failure Modes and Effects Analysis Revised description
7 NA Daily
6 NA Greater than once a week
5 NA Greater than once a month but less than once a week
4 Frequent: likely to occur immediately or within a short period (may happen Greater than once a year but less than once a month
several times in 1 year)
3 Occasional: probably will occur (may happen several times in 1e2 years) Less than twice a year
2 Uncommon: possible to occur (may happen sometime in 2e5 years) Once every 2e5 years
1 Remote: unlikely to occur (may happen sometime in more than 5e30 years) Once every greater than 5e30 years

Qual Saf Health Care 2010;19(Suppl 3):i26ei30. doi:10.1136/qshc.2010.040543 i29

 Original research
Figure 2 Care giver identified and notified about potential discharge.
a lower but still alarming rate of medication discrepancies.19 In a
review of written discharge summaries, McMillan found an
average of more than one medication error per discharge
summary.20 Neonates are discharged with many high-risk
medications, and thus a similar potential for adverse events
exists. In addition, medical conditions often are not completely
resolved, and patients or care givers may be unprepared for
significant medical care responsibilities at home. When the
primary care follow-up visit occurs, it is likely that the provider
who sees the infant is not the same provider who cared for the
infant during the hospitalisation. Hence, the transition period
between the inpatient and outpatient settings has many
vulnerabilities to errors that may result in severe consequences.
The HFMEA method has been used in a variety of healthcare
settings and has been applied in paediatrics to medication-
related processes.21 22 The application of the HFMEA method to
care transitions in the NICU discharge process was viewed as
potentially valuable but arduous in terms of time and effort.
Although the team was sceptical of the overall value of the
HFMEA method for studying these care transitions, the team
uniformly found value in the opportunity to discuss the
discharge process among the disparate groups involved in the
process.
Our study had several limitations. We studied the NICU
discharge process only at a single institution and adapted the
scoring matrix from adult literature. The specific findings of our
HFMEA may not be generalisable to other institutions; however,
our findings regarding the HFMEA process itself should be
widely applicable. As is always the case in a group process,
judgements of one or two team members can bias the scoring.
We took steps to minimise this through careful discussion with
frequent clarification of each team member’s input. Although
we found the HFMEA method useful, leadership involvement
and commitment are essential to enable participants to devote
the required time.
In summary, the HFMEA process enabled us to improve our
understanding of the critical error points that introduce risk
during the transition from NICU to the ambulatory environ-
ment. We identified risks related to lack of communication
between care providers in the hospital, between care providers
and parents/care givers, and between hospital-based and
ambulatory care providers. Additional identified risks included
potential lack of knowledge and skills among community-based
care providers who are responsible for caring for vulnerable
infants after discharge. While the HFMEA holds promise for
improving patient safety during care transitions, the value of
i30
applying this tool to the complex transitions from the NICU to
community settings requires additional study.
Acknowledgements We would like to acknowledge the members of the HFMEA
team and S Mathew for assistance with project coordination.
Funding This work was supported by a P-20 grant from AHRQ (P20 HS017122-01
‘Crossing An Invisible Quality Chasm: From NICU To Ambulatory Care’) and in part by
the Houston VA HSR&D Center of Excellence (HFP90-020).
Competing interests None.
Ethics approval Ethics approval was provided by the Baylor College of Medicine.
Contributors VAM contributed to the conceptualisation, execution and analysis, and
drafted and revised the manuscript. HS contributed to the conceptualisation, analysis,
drafting and revision of the manuscript. KLF contributed to the conceptualisation and
execution of the project, and the revision of the manuscript. APG contributed to the
conceptualisation, execution and analysis of the research, and to drafting and revision
of the manuscript.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Coleman EA, Berenson RA. Lost in transition: challenges and opportunities for
improving the quality of transitional care. Ann Intern Med 2004;141:533e6.
2. Clancy CM. Care transitions: a threat and an opportunity for patient safety.
Am J Med Qual 2006;21:415e17.
3. Wilson-Costello D, Friedman H, Minich N, et al. Improved survival rates with
increased neurodevelopmental disability for extremely low birth weight infants in the
1990s. Pediatrics 2005;115:997e1003.
4. Kotagal UR, Perlstein PH, Gamblian V, et al. Description and evaluation of a program
for the early discharge of infants from a neonatal intensive care unit. J Pediatr
1995;127:285e90.
5. Moore C, Wisnivesky J, Williams S, et al. Medical errors related to discontinuity
of care from an inpatient to an outpatient setting. J Gen Intern Med
2003;18:646e51.
6. Forster AJ, Clark HD, Menard A, et al. Adverse events among medical patients after
discharge from hospital. CMAJ 2004;170:345e9.
7. Forster AJ, Murff HJ, Peterson JF, et al. The incidence and severity of adverse
events affecting patients after discharge from the hospital. Ann Intern Med
2003;138:161e7.
8. Forster AJ, Murff HJ, Peterson JF, et al. Adverse drug events occurring following
hospital discharge. J Gen Intern Med 2005;20:317e23.
9. McPherson ML, Lairson DR, Smith EO, et al. Noncompliance with medical follow-up
after paediatric intensive care. Pediatrics 2002;109:94.
10. DeRosier J, Stalhandske E, Bagian JP, et al. Using health care Failure Mode and
Effect Analysis: the VA National Center for Patient Safety’s prospective risk analysis
system. Comm J Qual Improv 2002;28:248e67, 209.
11. Kunac DL, Reith DM. Identification of priorities for medication safety in neonatal
intensive care. Drug Saf 2005;28:251e61.
12. van Tilburg CM, Leistikow IP, Rademaker CM, et al. Health Care Failure Mode and
Effect Analysis: a useful proactive risk analysis in a pediatric oncology ward. Qual Saf
Health Care 2006;15:58e63.
13. U.S. Department of Veterans Affairs. National Center for Patient Safety, 2009.
http://www.va.gov/NCPS/CogAids/HFMEA/index.html (accessed 1 May 2009).
14. Krippendorff K. Content analysis: an introduction to its methodology. Thousand
Oaks, CA: Sage Publications, 2004.
15. Denzin NK. The interpretive process. In: Huberman AM, Miles MB, eds, The
qualitative researcher’s companion. Thousand Oaks, CA: Sage Publications,
2002:349e66.
16. Lemons JA, Bauer CR, Oh W, et al. Very low birth weight outcomes of the National
Institute of Child Health and Human Development Neonatal Research Network,
January 1995 through December 1996. NICHD Neonatal Research Network.
Pediatrics 2001;107:E1.
17. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results
that return after hospital discharge. Ann Intern Med 2005;143:121e8.
18. Beers MH, Sliwkowski J, Brooks J. Compliance with medication orders among the
elderly after hospital discharge. Hosp Formul 1992;27:720e44.
19. Coleman EA, Smith JD, Raha D, et al. Posthospital medication discrepancies:
prevalence and contributing factors. Arch Intern Med 2005;165:1842e7.
20. McMillan TE, Allan W, Black PN. Accuracy of information on medicines in hospital
discharge summaries. Intern Med J 2006;36:221e5.
21. Kunac DL, Reith DM. Identification of priorities for medication safety in neonatal
intensive care. Drug Saf 2005;28:251e61.
22. van Tilburg CM, Leistikow IP, Rademaker CM, et al. Health Care Failure Mode and
Effect Analysis: a useful proactive risk analysis in a pediatric oncology ward. Qual Saf
Health Care 2006;15:58e63.
Qual Saf Health Care 2010;19(Suppl 3):i26ei30. doi:10.1136/qshc.2010.040543

Department of Pediatrics,
Division of Critical Care, Medical
College of Wisconsin,
Milwaukee, Wisconsin, USA
Correspondence to
Dr Rebecca A Russell, Division
of Critical Care, Department of
Pediatrics, Medical College of
Wisconsin, PO Box 1997, MS
681, 9000 West Wisconsin
Avenue, Milwaukee, WI 53201,
USA; rarussel@mcw.edu
Accepted 24 February 2010
Qual Saf Health Care 2010;19(Suppl 3):i31ei35. doi:10.1136/qshc.2009.036384
Discrepancies between medication orders and
infusion pump programming in a paediatric intensive
care unit
Rebecca A Russell, Kathy Murkowski, Matthew C Scanlon
ABSTRACT
Background Errors and the incorrect use of medications
are significant sources of risk and harm to children in US
hospitals. The risk associated with medication infusions
has led to recommendations for the adoption of
technologies including computer order physician entry
(CPOE) and ‘smart’ infusion pumps despite a paucity of
evidence demonstrating the ability of these technologies
to reduce harm to paediatric inpatients.
Objective To measure discrepancies between
medication orders for infusions entered into a CPOE
system and the medication being infused as measured
by the programmed settings of the smart infusion pump
within a paediatric intensive care unit.
Methods This study used a prospective, observational
design in a 30-bed paediatric intensive care unit. Data
were simultaneously collected from the medication
orders in the CPOE system and the bedside smart
infusion pumps by trained observers. Analysis consisted
of a line-by-line comparison of order observation data
with the pump observation data.
Conclusions Of 296 observations of medication
infusions and 231 observations of intravenous fluid
infusions, the frequency of discrepancies between orders
entered and pumps programming ranged from 24.3% for
observed medications to 42.4% for observed fluids. Anti-
infectives (100%), concentrated electrolytes (46.7%) and
anticoagulants (46.2%) were associated with greatest
discrepancy between orders and programmed doses.
INTRODUCTION
Errors and the incorrect use of medications are
significant sources of risk and harm to children in
US hospitals.1e3 Medication errors are classically
defined as ‘any error occurring in the medication
use process.’4 A study of medication errors among
children found that the rate of preventable adverse
drug events (ADE) was five per 10 778 (0.05%),
similar to adult studies.5 However, the rate of
potential adverse drug events (errors that occurred
yet did not harm the patient) was three times
higher in children when compared with adult rates
(1.1% vs 0.35%). These potential ADEs were most
common in neonatal intensive care units, a finding
that is consistent with previous work demon-
strating a higher number of medication errors in
neonatal and paediatric intensive care units.6 7 The
reported incidence of medication errors in paedi-
atric intensive care units (ICUs) is between 22 and
55 per 1000 doses.5 8 While not all medication
errors cause harm, they are sources of risk and of
potential harm to children.
Original research
The use of intravenous medications has been
identified as a specific source of medical risk. In one
adult ICU setting, 83% of ADEs were related to
medications in intravenous form.9 The risk related
to intravenous medications includes those medica-
tions administered via an infusion pump. A study
of infusion errors in an adult setting found that
66.9% of intravenous medications had one or more
error associated with their administration, and
infusion pumps accounted for up to 35% of all
medication errors resulting in significant harm to
patients in one children’s hospital.10 11
In response to the errors and harm associated
with medication infusions, recommendations have
called for the adoption of a range of healthcare
information technologies (HIT) including
computer order physician entry (CPOE), bar-coded
medication administration (BCMA) and ‘smart’
infusion pumps.10 12 13 Unfortunately, while these
technologies have been associated with the elimi-
nation of errors, the available studies have yet to
demonstrate that the technologies reduce harm. For
instance, studies of CPOE suggest that the errors
most commonly prevented are those that are least
likely to cause harm.14 Furthermore, while there
has been a national call for the use of CPOE based
on its ability to decrease prescribing errors, exami-
nation of CPOE by both quantitative and qualita-
tive measures has revealed its potential association
with new and unanticipated errors.15e18 Similarly,
‘smart’ infusion pump technology has been identi-
fied as a means of preventing errors with little
evidence to support the prevention of harm.19 20
Like adult studies, there is a paucity of evidence
demonstrating the ability of these technologies to
reduce harm to paediatric inpatients.21
Applying insight into work systems from the
science of human factors engineering (HFE) may
provide a helpful perspective through which to
view the limited realised benefits of these medica-
tion safety technologies. One such model, the
SEIPS model, illustrates how people (both health-
care providers and patients) use tools and technol-
ogies to perform tasks in various environments
within the context of an organisation.22 23 These
five components and their interactions result in the
emergent properties of safety and quality of the
system of care delivery. In light of such a model,
what may be viewed as the simple introduction of
technology (infusion pumps) may actually result in
a fundamental change to systems of care delivery
that impact providers and their tasks.
Further, work into the multidimensional nature
of nursing workload suggests that there may be at
i31
 Original research
least three types of nursing workload that exist within the
context of a specific care unit. These include unit-level, job-level
and task-level workloads.24 25 While the unit-level workload can
be thought of as staffing ratios on a nursing unit, the job-level
workload includes both general and specific demands of a given
job, and the task-level work is includes the aspects of the work
that is specific to a given activity. Additionally, changes to one
level of workload may have unintended and negative conse-
quences for other levels of workload. Thus, the task-level
activities associated with using a new technology, such as an
infusion pump, may have an impact on unit- and job-level work.
Conversely, the job- and unit-level workload may influence the
ability to perform task-level work.
With a model of a system of healthcare delivery and the
concepts of multidimensional work in mind, the primary
objective of this study was to measure discrepancies between
medication orders for infusions entered in to a CPOE system and
the medication (or fluid) actually being infused as indicated by
the programmed settings of the smart infusion pump, in the
setting of a PICU. A second objective was to interpret the
findings from the study in using HFE concepts in the hopes of
better understanding why any observed discrepancies may exist.
MATERIALS AND METHODS
Study design
This study used a prospective, observational design in a 30-bed
paediatric intensive care unit (PICU), averaging 1820 admissions
per year, in a free-standing children’s hospital located in the
Midwest. The patient population in the PICU includes children
ranging from 1 day to over 18 years of age, with a wide range of
disease processes including trauma, cardiothoracic surgery,
respiratory failure, metabolic disease and sepsis. Trained PICU
nurses perform infusion pump programming, while orders for
medication infusions and intravenous fluids may be entered into
the CPOE system by attending physicians, trainees (fellows and
resident physicians), as well as nurse practitioners and physi-
cians assistants. Data were gathered over a period of 24 week-
days, and approval from the hospital Institutional Review Board
was obtained prior to any data collection.
Technologies
The CPOE system, Sunrise Clinical Manager Version 4.5 by
Eclipsys Technologies Corporation, was implemented in
September of 2000, with more than 90% of orders directly
entered into the system. The ‘smart’ infusion pumps (Smith-
Medex) were implemented in 2006.
Data collection
For this study, infused medications were defined as medications
requiring the use of an infusion pump. These included both
continuous infusions over multiple hours or days as well as
single-dose infusions. Intravenous fluids were defined as total
parenteral nutrition (TPN), lipids and crystalloid infusions
requiring the use of an infusion pump.
Data were simultaneously collected from the medication
orders in the CPOE system and the bedside smart infusion
pumps by trained observers, once a day during weekdays over
a period of 24 days. One observer went from bed space to bed
space recording date, time, bed number and each infused medi-
cation or fluid, along with the corresponding dose or rate. A
second observer reviewed existing orders by bed space, capturing
the date and time of observation, and each ordered medication or
fluid with its respective dose or rate. All data were then entered
into a spreadsheet, which was stored in a secured manner.
i32
No patient related-information was collected during this study.
Information was simultaneously collected only from medication
orders and infusion pumps, using the patient bed space as the basis
for linking orders and pumps. Therefore, patients not receiving
infused medications (as evidenced by the absence of pumps) were
excluded from the pump observation component.
Analysis
Analysis consisted of a line-by-line comparison of order observa-
tion data with the pump observation data, matched by time and
bed number. Any observations which occurred more than 15 min
apart were excluded to minimise the effect of any interim changes
to order or pump. For each observed medication infusion, analysis
began with verifying the presence of a corresponding order. For
each observed medication that had a corresponding order, the
doses that were ordered and programmed were compared.
Observed intravenous fluids were compared with the ordered
fluids, and when corresponding orders and fluids were identified,
the rate of administration was compared. The frequency of
discrepancies by medication and fluid type was established.
Discrepancies were defined as follows: a medication or fluid
found to be infusing without a corresponding order was cate-
gorised as an unauthorised medication or unauthorised fluid.
When no medication or fluid infusion was observed despite the
presence of an active order, this discrepancy was categorised as
an omitted medication or omitted fluid. Finally, when the
medication or fluid infusing at the bedside was observed to differ
in dose or rate when compared with the active order in CPOE,
this discrepancy was categorised as a wrong medication dose or
wrong fluid rate.26
RESULTS
A total of 296 observations of medication infusions and 231
observations of intravenous fluid were completed during the
study period of 24 days. No observations required exclusion based
on the predetermined 15 min repugnant time window for orders
and programming. Seventy-two of the 296 (24.3%) observations
of medications revealed order-programming discrepancies, while
97 of the 231 (42%) observations of intravenous fluid revealed
order-programming discrepancies (tables 1, 2).
Of the 72 medication observations that revealed order-
programming discrepancies, 62 (86.1%) of the discrepancies were
due to either unauthorised medications or omitted medications,
while 10 (13.9%) of the detected discrepancies were due to
wrong medication dose (table 3). When the medication obser-
vations were classified by medication category, those categories
with the most frequent order-programming discrepancies
included anti-infectives (100%), concentrated electrolytes
(46.7%) and anticoagulants (46.2%). Within the anti-infective
and concentrated electrolyte groups, every discrepancy was
identified as an unauthorised medication, whereas anticoagulant
discrepancies were almost evenly split between unauthorised
and omitted medications. None of the discrepancies observed
within the anti-infective medications, concentrated electrolytes
or anticoagulants were found to be due to a wrong dose.
Similar to the observed medications, 56.7% of the observed
order-programming discrepancies among intravenous fluids were
due to unauthorised and omitted fluids. The observed order-
programming discrepancies among intravenous fluids also
revealed a high frequency of incorrect rate of infusion (43.3%),
whereas there were fewer observed wrong doses among the
medications. The greatest number of observed intravenous fluid
discrepancies was in dextrose-containing fluids (48.3%) and TPN
(50%) (table 2). For both crystalloids and TPN, discrepancies
Qual Saf Health Care 2010;19(Suppl 3):i31ei35. doi:10.1136/qshc.2009.036384
 Original research

Table 1 Discrepancies by medication type and subcategory Table 2 Discrepancies by fluid type and subcategory
No of discrepancies No of observations by No of discrepancies per
per observations within intravenous fluid observations within a fluid
No of observations by a medication subcategory Intravenous fluid subcategory (percentage subcategory (percentage of
medication subcategory (percentage of type of total) discrepancies)
Medication type (percentage of total) discrepancies)
All 231 (100) 97 (42.0)
All 296 (100) 72 (24)
Crystalloid 123 (53.2) 60 (48.7)
Cardiovascular 100 (33.8) 10 (10) D5W 83 (35.9) 35 (42.1)
medications
D10W 19 (8.2) 15 (78.9)
Milrinone 49 (16.6) 1 (2)
D20W 18 (7.8) 8 (44.4)
Epinephrine 29 (9.8) 5 (17.2)
Lactated ringers 1 (0.4) 1 (100)
Norepinephrine 9 (3) 1 (11.1)
Normal saline 2 (0.8) 1 (50)
Phenoxybenzamine 9 (3) 2 (22.2)
Nicardipine 2 (0.7) 1 (50)
Hyperalimentation 108 (46.8) 37 (34.3)
Nitroprusside 1 (0.3) 0
Total parenteral 44 (19) 22 (50)
Vasopressin 1 (0.3) 0 nutrition
Lipids 64 (27.7) 15 (23.4)
Analgesia/sedation 58 (19.6) 6 (10.3)
Fentanyl 20 (6.8) 4 (20)
Sufentanil 11 (3.7) 1 (9.1) programming discrepancies ranged from 40 to 100%. This is of
Midazolam 9 (3) 1 (11.1) particular concern because concentrated electrolytes and anti-
Ketamine 8 (2.7) 0
coagulants have been identified as high-risk medications.27
Dexmedetomidine 7 (2.4) 0
Why were these discrepancies present? The exact cause for
Morphine 3 (1) 0
any specific observed discrepancy is unclear; however, a number
of potential origins have been identified. Problems with order
Anticoagulants 65 (22) 30 (46.2)
Heparin 65 (22) 30 (46.2)
entry include omission of entry as well as incorrect order entry.
Alternatively, pump programming may be the source of the
Concentrated 15 (5.1) 7 (46.7) discrepancy and could be related to a lack of recognition of active
electrolytes orders, failure to execute programming, and incorrect program-
Calcium gluconate 12 (4) 4 (33.3) ming. Regardless of the origin, it should be noted that there are
Magnesium 2 (0.7) 2 (100) inherent risks and potential for harm within the process of
Potassium 1 (0.3) 1 (100) reconciling these discrepancies. Specifically, there is a very real
risk that a nurse may discover the discrepancy between order
Anti-infective 14 (4.7) 14 (100) and infusion pump, and decide to reconcile it by changing one to
medications
match the other. As long as the nurse correctly reconciles to the
Vancomycin 5 (1.7) 5 (100)
accurate item (order or pump), the patient does well. However,
Fluconazole 3 (1) 3 (100)
reconciliation of the incorrect item may create additional risk of
Metronidazole 3 (1) 3 (100)
patient harm.
Cefepime 1 (0.3) 1 (100)
Micafungin 1 (0.3) 1 (100)
Due to the limitations of this study, several questions remain.
Oxacillin 1 (0.3) 1 (100)
First, which is correct: the medication order in the CPOE system
or the actual pump settings? Our study cannot answer this, as it
Miscellaneous 44 (14.9) 5 (11.4) was impossible to tell which technology reflects the intent of
Furosemide 34 (11.2) 1 (2.9) the prescribing physician and which technology was incorrect.
Vecuronium 7 (2.7) 1 (14.3) Anecdotally, asking the bedside providers or physicians may not
Other 3 (1) 3 (100) help clarify whether the order or the pump is correct because
these providers typically go to the pump or order to aid their
recall. A second question that remains unanswered by this study
is, why might this phenomenon be occurring in the first place?
This question is perhaps easier to answer by considering the flow
of actions and information associated with titration of infusions
were divided fairly equally between unauthorised/omitted fluids of medications.
and wrong fluid rates (table 3). For medications that are titrated for clinical effect, such as
epinephrine, the assumed process by the engineers designing
DISCUSSION CPOE and infusion technologies is as follows: Provider conceives
We observed a high frequency of discrepancies between medi- of order / Provider enters order / Provider communicates
cation and intravenous fluid orders within a CPOE system and order to nurse / Nurse reprogrammes infusion pump.
programmed settings on ‘smart’ infusion pumps at the beside. While this may happen, there are other variations of this
The observed frequency of discrepancy (24%) between medica- process that appear to occur related to the patient being the
tion order and the medication infusions in our study is similar to point of initiating care and NOT the computer. For instance, the
the frequency observed by Barker et al (19%) in 2002.26 There are sequence may be: Provider conceives of order / Provider enters
no published frequencies of fluid discrepancies to compare with order / Provider fails to communicate order to nurse / Nurse
our observed frequency of intravenous fluid discrepancies (42%). does not act or acts only after later discovering order.
Among the medication classes of anti-infectives, concentrated More commonly, an evaluation of the patient at the bedside
electrolytes and anticoagulants, the observed frequency in order- leads to one of the following occurrences: Provider exams patient

Qual Saf Health Care 2010;19(Suppl 3):i31ei35. doi:10.1136/qshc.2009.036384 i33

 Original research
Table 3 Count of discrepancies by cause and medication/fluid type
Cause of discrepancy
Medication type Omitted medication Unauthorised medication
Cardiovascular 3 3
Analgesia/sedation 0 1
Anticoagulants 16 14
Concentrated electrolytes 0 7
Anti-infectives 0 14
Miscellaneous 0 4
Fluid type Omitted fluid Unauthorised fluid
Crystalloid 8 26
Hyperalimentation 16 5 16
/ Provider conceives of order / Provider communicates order
to nurse / Nurse reprogrammes infusion pump / Provider
may or may not later enter order / Nurse may or may not
enter order as verbal order in the event they realise the provider
failed to enter the order. Or: Provider exams patient / Provider
conceives of order / Provider reprogrammes infusion pump /
Provider may or may not later enter order.
In this last example, the nurse may not recognise that a pump
change was made for some time.
An additional factor that likely contributed to the increased
frequency of discrepancies between CPOE orders and pump
programming relates to vascular access and the number and
relative timing of infusions, and compatibility of medications.
For example, orders within a CPOE system may indicate that
multiple medications are to be given within a narrow time
frame. Depending on the duration of infusion and compatibility
of the various medications, it may be necessary for the bedside
nurse to delay administration of one medication while another is
infusing. This situation could have led to observation of either
omitted or unauthorised medications under our study design,
and it may help explain the high frequency of discrepancies
within the anti-infective and concentrated electrolyte medica-
tion groups.
This study was based on a several concepts from the science of
human factors engineering including a model of understanding
systems in healthcare as well as the multidimensional nature of
nursing work. The issue of nursing work is critical to this study,
as nurses are the providers who program infusion pumps in the
studied PICU. This task-level activity (programming a pump) is
performed in a system that involves multiple prescribing
physicians and clinicians who are charged with entering orders
into the CPOE system. The nurses in the PICU act as one of
several ‘people’ in the care delivery system who must perform
the task-level action of order discovery and pump programming
while also performing their job- and unit-level activities.
An additional concept that is important to this study is the
understanding that errors and harm are both manifestations of
flawed systems of care delivery.19 These system flaws can be
identified as risks, which can then be analysed for potential
impact and reduced, when appropriate. Using this concept, this
project did not attempt to identify harm from medication errors.
How harm might manifest in any given patient would vary by
the medication or fluid involved in a discrepancy and by the
cause of the error. An unauthorised medication might create
harm that would be different from an omitted medication or
wrong dose. The heterogeneity of diagnoses, severity of illness,
comorbidities, ages and other therapies that exist in a PICU
setting confound the detection of harm. Thus, the goal was to
identify any discrepancy between a medication order and the
actual programmed infusion that created the potential for harm
i34
Wrong medication dose Total discrepant by medication type
4 10
5 6
0 30
0 7
0 14
1 5
Wrong fluid rate Total discrepant
26 60
37
to patients. Only through identifying and analysing risks can
the process of risk elimination and mitigation begin.
This leads to the final question that remains unanswered by
this study: how best to eliminate or mitigate this risk? The
question is most difficult to answer. Because the expectation of
information and work flow as envisioned by designers of tech-
nologies used in medication delivery is very different from the
reality at the bedside, any solutions may serve as ‘workarounds’
that increase risk. Thus, the introduction of technologies
intended to reduce risk and harm may, by virtue of inadequate
understanding of the pre-existing and resultant systems of
medication delivery, create increased risk. This is particularly
true in ICU environments where the changes are often made at
the patient’s bedside. Ironically, these patients happen to be
among the most ill and most complex, receiving large numbers
of potentially dangerous medications.
CONCLUSIONS
Our study revealed potential risks that result from the interac-
tions between different providers performing the tasks of ordering
intravenous medications and fluids, and the programming of the
infusion pumps in a paediatric intensive care unit environment
that has adopted two safety technologies, CPOE and ‘smart’
infusion pumps. Both healthcare providers and designers of
technology might benefit from more intensive understanding of
such interactions before tampering with existing systems.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Takata GS, Mason W, Taketomo C, et al. Development, testing, and findings of
a paediatric-focused trigger tool to identify medication-related harm in US children’s
hospitals. Pediatrics 2008;121:e927e35.
2. Hicks RW, Becker SC, Cousins DD. Harmful medication errors in children: a 5-year
analysis of data from the USP’s MEDMARX program. J Pediatr Nurs 2006;21:290e8.
3. Slonim AD, LaFleur BJ, Ahmed W, et al. Hospital-reported medical errors in children.
Pediatrics 2003;111:617e21.
4. Bates DW, Boyle DL, Vander Vliet MB, et al. Relationship between medication errors
and adverse drug events. J Gen Intern Med 1995;10:199e205.
5. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events
in paediatric inpatients. JAMA 2001;285:2114e20.
6. Folli HL, Poole RL, Benitz WE, et al. Medication error prevention by clinical
pharmacists in two children’s hospitals. Pediatrics 1987;79:718e22.
7. Raju N, Kecskes S, Thornton JP, et al. Medication errors in neonatal and paediatric
intensive-care units. Lancet 1989;2:374e6.
8. Potts AL, Barr FE, Gregory DF, et al. Computerized physician order entry and
medication errors in a pediatric critical care unit. Pediatrics 2004;113:59e63.
9. Nuckols TK, Bower AG, Paddock SM, et al. Programmable infusion pumps in ICUs:
an analysis of corresponding adverse drug events. J Gen Intern Med 2008;23:41e5.
10. Husch M, Sullivan C, Rooney D, et al. Insights from the sharp end of intravenous
medication errors: implications for infusion pump technology. Qual Saf Health Care
2005;14:80e6.
11. Tourville J. Automation and error reduction: how technology is helping Children’s
Medical Center of Dallas reach zero-error tolerance. US Pharm 2003;28:80e6.
Qual Saf Health Care 2010;19(Suppl 3):i31ei35. doi:10.1136/qshc.2009.036384

12. Bates DW, Gawande AA. Improving safety with information technology. N Engl J
Med 2003;348:2526e34.
13. Stucky ER. American Academy of Pediatrics Committee on Drugs, American
Academy of Pediatrics Committee on Hospital Care. Prevention of medication errors
in the pediatric inpatient setting. Pediatrics 2003;112:431e6.
14. Wang JK, Herzog NS, Kaushal R, et al. Prevention of pediatric medication errors by
hospital pharmacists and the potential benefit of computerized physician order entry.
Pediatrics 2007;119:e77e85.
15. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry
systems in facilitating medication errors. JAMA 2005;293:1197e203.
16. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after
implementation of a commercially sold computerized physician order entry system.
Pediatrics 2005;116:1506e12.
17. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential
adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA
1995;274:29e34.
18. Milstein A, Galvin RS, Delbanco SF, et al. Improving the safety of health care: the
leapfrog initiative. Eff Clin Pract 2000;3:313e16.
19. Rothschild JM, Keohane CA, Cook EF, et al. A controlled trial of smart infusion pumps
to improve medication safety in critically ill patients. Crit Care Med 2005;33:533e40.
Qual Saf Health Care 2010;19(Suppl 3):i31ei35. doi:10.1136/qshc.2009.036384
Original research
20. Leape LL. ‘Smart’ pumps: a cautionary tale of human factors engineering. Crit Care
Med 2005;33:679e80.
21. Walsh KE, Landrigan CA, Adams EF, et al. Effect of computer order entry
on prevention of serious medication errors in hospitalized children. Pediatrics
2008;121:e421e7.
22. Carayon P, Alvarado CJ, Brennan P, et al. Work system and patient safety.
In: Luckzak H, Zink KJ, eds. Human factors in organizational design and
managementeVII. Santa Monica, CA: IEA Press, 2003:583e8.
23. Carayon P, Schoofs Hundt A, Karsh BT, et al. Work system design for patient
safety: the SEIPS model. Qual Saf Health Care 2006;15:50e8.
24. Holden RJ, Scanlon MS, Patel NR, et al. A human factors framework and study of
the effect of nursing workload on patient safety and employee quality of working life.
Qual Saf Health Care In press.
25. Holden RJ, Patel NR, Scanlon MS, et al. Effects of mental demands during
dispensing on perceived medication safety and employee well being:
A study of workload in pediatric hospital pharmacies. Res Social Adm Pharm
In Press.
26. Barker KN, Flynn EA, Pepper GA, et al. Medication errors observed in 36 health care
facilities. Arch Intern Med 2002;162:1897e903.
27. http://www.ismp.org/Tools/highalertmedications.pdf.
i35
 Original research
1
Department of Industrial
Design, Delft University of
Technology, Delft, The
Netherlands
2
Department of Nursing Affairs,
University Medical Center
Groningen, Groningen, The
Netherlands
Correspondence to
Marijke Melles, Department of
Industrial Design, Delft
University of Technology,
Landbergstraat 15, Delft 2628
CE, The Netherlands;
m.melles@tudelft.nl
Received 10 August 2009
Revised 16 March 2010
Accepted 27 March 2010
i36
Coping with different roles in intensive care nursing:
design implications for digital support
Marijke Melles,1 Adinda Freudenthal,1 Addie Bouwman,2 Chris Snijders,1
Huib de Ridder1
ABSTRACT
Objectives The aim of this study is to arrive at design
implications for the digital support of intensive care
nurses, with a focus on supporting them in their roles as
practitioners, as scholars and in their human response to
their work.
Methods Seventeen nurses from six different Dutch
hospitals were interviewed using a newly developed
digital prototype designed around these roles. The nurses
were questioned about the suitability of four functions:
two supporting the practitioner role (‘web of actors’ and
‘timeline and notes’); one supporting the scholar role
(‘assessment of the patient’); and one supporting the
role of human (‘perception of the shift’). The remarks of
the nurses about these functions were grouped by their
relevance to design (ie, the perceived added value of the
function, suggestions for other uses, and the perceived
negative impact of the function) and by topic.
Results A total of 23 topics emerged: nine topics
described the added values of the functions, nine were
suggestions for other uses, and five concerned the
negative impact of the functions as perceived by the
participants. The main points of each topic were
summarised, resulting in 23 design implications for digital
role-based support.
INTRODUCTION
Recently the European Society of Intensive Care
Medicine released a statement on patient safety in
intensive care medicine, known as the Declaration
of Vienna.1 This statement focuses particularly on
the impact of the care process and includes issues
such as teamwork, collaboration, work culture and
professional demeanour. The declaration also refers
to the problems of communication between the
physicians and nurses2 and the importance of job
satisfaction and the well-being of intensive care
nurses in relation to the quality of their work.3 Our
research attempts to make a contribution to these
last points of concern. We focus on the work of
intensive care nurses (ICU nurses) and how tech-
nology can help support them in their day-to-day
experiences. In the study presented here, we
investigate the functions these future digital
support systems might offer. An interactive proto-
type was used to collect insights from ICU nurses.
The results are design implications for digital role-
based support for ICU nurses.
The prototype used in this study is designed
around a theoretical framework for supporting
ICU nursing. In this framework, based on insights
from previous studies, the cognitive activities of
Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483
interpretation, anticipation and reflection are
connected to the three roles an ICU nurse fulfils
(also illustrated in figure 1). First is the role of the
practitioner, referring to the strategies nurses
undertake to implement and execute the nursing
process. Examples of such activities are autono-
mously planning all work-related actions, operating
the required equipment and reporting on the status
of the patient. Second is the role of the scholar,
referring to the strategies nurses undertake to
obtain and maintain their own required knowledge
and skills. This can include training, reflecting on
experiences, but also putting thought processes
into words or having to account for actions when
tutoring trainee nurses. Third is the role of the
human, referring to the strategies nurses undertake
to cope with stress and the emotional consequences
of working with seriously ill or dying patients and
their relatives. Examples are sharing experiences
with colleagues and the use of humour. The aim of
our role-based framework is to offer handles to
both product developers and ICU staff to discuss
the nursing process and related informational
problems and needs in a way that is understandable
and useful to both parties. Furthermore, the
framework is expected to support product devel-
opers in defining product functions for digital
products.
The concept of roles for medical practitioners has
been used in the CanMeds Roles Framework, as
defined by the Royal College of Physicians and
Surgeons of Canada.4 This framework, consisting
of seven roles including scholar, communicator and
collaborator, aims to assist medical specialists in
developing the skills associated with the various
roles. Our framework includes the addition of the
role of the human. Based on interviews and focus
group sessions we concluded that coping with
stress and emotions is essential to the well-being of
nurses and therefore essential for ICU nurses to be
able to carry out their work safely. Furthermore,
while the CanMeds Roles Framework is mainly
used for training and assessment purposes, we
want to explore the extent to which the concept of
roles can serve as a blueprint for product develop-
ment in daily support. Although making use of
roles is a well-known design approach in informa-
tion technology practice,5 6 it does not yet seem to
be fully exploited in medical product development.
The aim of the prototype is to try out new role-
based functions and to trigger reactions from ICU
nurses about the suitability of the functions
offered. The prototype is not an end product but
a ‘vehicle for exploration,’ intended to give rise to
interesting design ideas and concepts.7 8 We expect

Figure 1 Schematic representation of the different roles an ICU nurse
fulfils (based on Melles et al9).
that the implications for future design that follow from
discussion of the prototype will be generic and also applicable to
other applications. The functionality of the prototype is based
on conclusions from previous studies that investigated the
opportunities for enhancing patient safety by means of tech-
nology.9 The following findings from these studies shaped our
prototype: (1) supplying nurses with enhanced insights into
their working environment contributes to the quality of their
work; (2) the information that nurses use is provided by the
different actors in their working environment, namely the
patient, equipment, nursing team, physician team, relatives of
the patient and other medical staff (eg, physiotherapists or social
workers); and (3) although nurses continually interpret a current
situation and anticipate future events, there is little reflection
(and little support for reflection) upon past events, even though
reflection is important for building skills and experience. These
findings were translated into different potential functions to
support the work of ICU nurses. In the study presented in this
paper, the following four functions are investigated: (1) a repre-
sentation of the (dynamic) working environment (relating to the
practitioner role); (2) an overview of past and future events
(relating to the practitioner role); (3) support for reflecting on
events, aiming to learn from them, technically or otherwise
(relating to the scholar role); and (4) support for reflecting on
events with the aim of coping with the emotions it generates or
learning from it from an emotional perspective (relating to the
human role). These functions are described in more detail in the
Methods.
METHODS
Participants
Seventeen ICU nurses from six different Dutch hospitals took
part in this study. Three of the six hospitals were teaching
hospitals (11 participants); the other three were regional
hospitals (six participants). Six participants were male, and 11
were female. Their experience ranged from none (having just
started as an ICU nurse) to 25 years, with a mean of 11 years
and a median of 7 years.
Materials
The prototype used during the interviews is a software simu-
lation, made in Macromedia Director and running on a laptop
computer. It has three mutually excluding modes: the practi-
tioner mode, the scholar mode and the human mode. Each mode
includes several functions. For this study, four functions were
used: two from the practitioner mode (‘web of actors’ and
‘timeline and notes’), one from the scholar mode (‘assessment of
the condition of the patient’) and one from the human mode
(‘perception of the shift’). The functions offered are envisioned
to be embedded in a patient data management system, meaning
Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483
Original research
that the system would be linked to a bed (ie, a patient) and
personalised to the nurse who is currently responsible for the
patient in that bed. All data are envisioned to be stored in
a networked database, so a nurse can retrieve information from
previous shifts, and plans made by others. The following
describes the implementation of the four functions in
more detail.
Function 1: practitioner modedweb of actors
The web of actors is a visualisation of the working environment
of the ICU nurse. All actors who are related to the patient and
therefore influence the work of an ICU nurse are represented by
icons. Clicking on an actor icon provides more information
about the specific actor, such as patient data measured with the
various devices, phone numbers of physicians, or names of
relatives. This information is displayed in a box along with the
web of actors. The distance between the actor icons and the
patient icon or the nurse icon visualises the priority of those
actors at a particular moment. Hence, this distance can change
over time. For example, when a doctor’s visit is scheduled
within 30 min, the physician icon moves towards the patient
icon. The nurse is represented with a flower-like shape. The
three petals of the flower represent the three aspects of the
nurse’s job. When the mouse moves over a petal, the application
switches to the corresponding role (ie, mode) and stays there.
Each mode has a different colour: the practitioner mode is blue,
the scholar mode is dark green, and the human mode is
light green.
Function 2: practitioner modedtimeline and notes
Another function in the practitioner mode is a timeline. On this
timeline, important tasks and events in the (near) future as well
as from the past are arranged, such as meetings in the future, but
also stressful events which happened in the past (eg, a cardiac
arrest). By clicking on an envelope displayed on the timeline for
each event, the nurse can view more information about that
particular event. Notes are automatically generated by the
system or added by the nurses themselves and can be placed on
the part of the timeline representing the future as well as on the
part representing the past. The timeline shows one 8 h shift and
can be scrolled backward or forward in time to see past or future
shifts. The timeline is also functional in the scholar and human
mode. However, in this study, only its functionality in the
practitioner mode is investigated.
Function 3: scholar modedassessment of the condition of the
patient
The function that is part of the scholar mode needs to be
prepared during the actual shift. Every 2 h, the nurse has to
indicate their assessment of the condition of their patient. This
is done by clicking on a star displayed on the timeline under the
current time. When clicking on this star, the nurse can make
a choice between routine (represented by a green star), vigilance
(yellow star) and emergency (red star) (based on Groen).10 At
a later moment in time (in the scholar role), this ‘snapshot’ of
the situation can be compared with the data measured at that
same moment. In this way, a learning situation is created,
comparing the subjectively experienced situation with objective
data from the same situation.
Function 4: human modedperception of the shift
The function chosen to explore the possibilities of supporting
the human role is a personal diary. After each shift, the nurse can
fill in whether they felt pleasantly calm, unpleasantly calm,
i37
 Original research
pleasantly stressed or unpleasantly stressed by choosing an
emotion icon (based on studies by Desmet).11 In the personal
diary this emotion icon is displayed for each shift. Also displayed
are coloured stars indicating the nurse’s assessment of the
condition of the patient in their role as scholar (see function 3),
and whether it was a day, evening or night shift. This provides
a rough indication of the context of a shift, giving the nurse the
means by which to relate her emotional well-being to the actual
circumstances during a shift. Furthermore, there is space to type
a personal note. The diary provides a weekly overview, making it
possible to look back in time at passed weeks.
Procedure
The interview began with an explanation of the goal of the
study (ie, eliciting responses and ideas for future nursing infor-
matics applications). The interviewer explained that the
functions offered could be envisioned as embedded in a patient
data management system, personalised to the nurse and linked
to the patient for whom the nurse would be currently respon-
sible. The interviewer demonstrated the functions. Each time
after a function was demonstrated, the questions regarding that
function were discussed before continuing with the other
functions. Participants were asked whether they considered the
demonstrated function to be useful or not and why. When the
function was considered to be useful they were asked how and
when they would use it. They were encouraged to think about
other situations in which they could use the particular function
or about elaborations of the function. In this way, a form of
participative design was adopted during the interview. Each
participant saw the various functions in the same order: web of
actors, timeline, assessment of the condition of the patient and
perception of the shift. Sometimes a function was skipped due
to time limitations or because there was another course of
events during the interview. In such cases, the interview began
with the function about assessing the condition of the patient,
followed by the perception of the shift, because these functions
were the most debatable.
The interview took place at the ICU department where the
participant worked. It was combined with a second interview
about the appropriateness of the three roles.12 The participant
was interviewed first about the roles and then about the
prototype. The entire interview lasted about 1 h, with the part
about the prototype taking 15 min. All interviews were audio-
recorded with the permission of the participants.
Analysis
Transcripts of all interviews were made. Each function was
assessed in terms of whether or not the participant considered it
to be useful. Situations in which the participant indicated that
the function could be useful if some changes were made to the
current implementation were marked as ‘potential.’ Remarks
about how, why or when the function woulddor would
notdbe used were noted down. This resulted in an overview of
all the relevant remarks for each function. These remarks were
then divided into three groups, each with a specific relevance to
product design: (1) the perceived added value of the function;
remarks on why participants found a function useful (this could
be the added value as envisioned by the developers but could also
be a way of using the function that was unforeseen by the
developers); (2) suggestions for elaboration of the function; and
(3) negative remarks about the function. The different types of
remarks can be used to optimise the implementation of a design.
For example, knowledge about what (future) users perceive as an
added value can be used to emphasise that feature or to
i38
implement the same idea in other types of applications. All
remarks were then grouped by topic. The grouping process
began by looking at the first remark and coming up with a topic
for it. Subsequent remarks were then classified either as
belonging to an existing topic or as a new topic. Where possible,
the same topics were used across the four functions. A total of
23 topics emerged from this process. The main points of each
topic were summarised briefly, resulting in 23 design implica-
tions.
RESULTS
Usefulness of the functions
Table 1 shows the participants’ assessments of the four different
functions. The practitioner-related functions (ie, web of actors
and timeline) were considered to be useful by most of the
participants. The web of actors function was assessed as useful
by 81% of the participants, and the timeline was assessed as
useful by all participants. Both functions mainly triggered
reactions about how they would be used (perceived added value;
see table 2). The functions related to the scholar and human
roles (ie, assessment of the patient and perception of the shift)
elicited more doubt and discussion. One-quarter of the partici-
pants indicated that both functions have potential, but they
questioned the current implementation. Another quarter
assessed both functions as not useful. We believe that the most
likely explanation for this is the more personal character of these
functions. Making these personal issues explicit can be perceived
as threatening, as illustrated by the following quotation from
a participant about ‘assessing the condition of the patient’
(scholar role) in relation to his upcoming yearly evaluation:
.Because anybody can make a wrong assessment. Even when it’s
a routine job. Say a patient comes in from the OR, without any
complications. Everything went well, the patient regains
consciousness quickly. But then he starts to bleed, his vital signs
start to deteriorate and then you’re faced with a problem. You had
assessed the situation as routine. And sure, you know why the
routine job suddenly became an emergency, but will everybody else
know that.
Usefulness of prototyping
Table 2 shows the number of design-relevant remarks (ie, added
value, suggestions for other use and perceived negative impact)
per function. A total of 116 remarks were made by the 17
participants, which means an average of nearly seven design
relevant remarks per participant. Based on these results we
conclude that the prototype does trigger discussion and
encourages nurses to participate in the process with the designer.
For example, 27% of all the remarks made were suggestions for
other uses of the functions.
Table 1 Numbers and percentages of participants who assessed the
different prototype functions in general as useful, not useful, or ‘having
potential’
Prototype mode and function Useful Not useful Potential
Practitioner
Web of actors 13/16 (81%) 2/16 (13%) 1/16 (6%)
Timeline and notes 15/15 (100%) 0/15 (0%) 0/15 (0%)
Scholar
Assessment of the patient 9/17 (53%) 4/17 (23.5%) 4/17 (23.5%)
Human
Perception of the shift 8/16 (50%) 4/16 (25%) 4/16 (25%)
The denominator indicates the number of participants who responded to the specific
functions.
Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483
 Original research

Table 2 Numbers and percentages of design-relevant remarks made with a particular design implication. Although the implications
per function are based on remarks made with reference to the prototype, the
Remarks about. findings are generic and also applicable to other applications. In
Perceived Suggestions Perceived the following, a selection of design implications will be discussed
Prototype mode added for other negative per function. The codes in parentheses in the text refer to the
and function value uses impact Total numbering of the design implications in table 3.
Practitioner
Web of actors 17 (15%) 5 (4%) 3 (3%) 25 (22%)
Timeline and notes 28 (24%) 6 (5%) 1 (1%) 35 (30%) Practitioner mode: web of actors
Scholar As a main added value of the web of actors function, the
Assessment of the 10 (9%) 9 (8%) 10 (8%) 29 (25%) participants mentioned that it provides a convenient arrange-
patient ment of information generated by their working situation,
Human something which most current patient data-management
Perception of the 8 (7%) 12 (10%) 7 (6%) 27 (23%) systems lack. The actors (ie, patient, equipment, physician
shift team, nursing team, relatives of the patient and other medical
Total 63 (54%) 32 (27%) 21 (18%) 116 (100%)
staff) were considered to reflect an ICU working domain
appropriately by all participants. The participants foresee using
DESIGN IMPLICATIONS this survey for two reasons: (1) as checklist (A2); and (2) as
Table 3 lists the 23 design implications that emerged from the a way to retrieve extra information about the different actors in
assessment of the four functions. The figures in table 3 refer to the working environment (A5)dfor example, protocols related
the number of participants who made comments associated to the diagnosis or maintenance information for the equipment

Table 3 Twenty-three design implications that emerged from grouping the remarks made by ICU nurses during the prototype study
Practitioner Scholar Human
Web of Assessment Perception
Design relevance Design implication actors Timeline of the patient of the shift
A. Perceived added value 1. The function provides insight into future tasks, hence providing 13
of the function support for anticipating tasks
2. The function serves as a checklist for fully comprehending the 11 2
work situation
3. The function makes the personal work process explicit, yielding 2 6 4
a personal learning experience
4. The function provides the possibility of reflecting on a chain of 7 2
events, yielding a learning experience
5. The function provides additional, easy to retrieve information 6
about each actor in the working environment
6. The function acknowledges the emotional load of nursing 4
7. The function provides the possibility of personalising the timeline 3
using notes
8. The function makes the nurses’ intuition concrete, formalising 2
intuitive patient assessments
9. The function provides insight into the context of tasks and to 1
why certain tasks are needed, thereby improving safety
B. Suggestions for other uses 1. Use this function during formal team evaluations to discuss 2 6
of the function (emotion-related) issues
2. Include this function in patient assessment tools for 5 1
management purposes, in particular to assess the degree of care
3. Include this function in the shift handover 1 2 1
4. Include alterations of the medical policy in this function; 4
differentiate between alterations and recurring tasks
5. Include this function in management tasks: how employees 3
experience their work
6. Use this function to visualise the care plan in context 2
7. Use this function as a training tool: assess the patient from the 2
perspective of another discipline and learn from this different
opinion
8. Include this function in the personal digital portfolio for reflection 2
9. Use this function to encourage nurses to develop views on 1
medical policy
C. Perceived negative impact 1. The information provided by the function is unnecessary, 2 4 1
of the function because nurses know the information provided from memory or
other sources
2. The requested information can be (mis-)used by managers 3 3
3. The use of technology is inappropriate 2 2
4. Using the function is too cumbersome 1 1
5. The function has no added value, hence no incentive to use it 1 1
For each implication, the number of participants who made remarks regarding that implication is shown, divided into the four functions. An empty cell means that no remarks were made.

Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483 i39

 Original research
used (eg, replacing filters). An interesting suggestion for another
use of the function is to view the patient’s condition from the
perspective of another discipline (such as the physician) by
clicking on an actor icon (B7). This may provide a valuable
learning experience. Two participants mentioned that they
thought the information supplied by the web of actors is
redundant; they could also observe it by just looking around, at
the equipment or in the patient file (C1). However, they agreed
it could provide a useful overview when looking back at events.
Note that the changing distance between the actor icons, as
a feature part of the web of actors, was assessed as ‘not useful’
by 15 of the 16 participants who assessed this function.
Practitioner mode: timeline and notes
The main added value of the timeline as noted by the partici-
pants is its reminder function, providing support to anticipate
tasks (A1). Four nurses suggested including alterations in
medical policy in the timeline (B4)dfor example, weaning
patients from the ventilator or cutting back on certain medica-
tions. These alterations and other tasks that are part of the
specific patient schedule (including medical as well as nursing
tasks) should be clearly different visually from the daily (recur-
ring) nursing tasks. Another interesting comment about the
timeline is that it shows the context within which tasks have to
be performed or events have taken place (A9). Current task lists
often show only the tasks that have to be done, sometimes even
only the task that has to be performed at that very moment.
Making the context of a task explicit in some way clarifies why
a certain task has to be performed. This contextuality can
contribute to patient safetydfor example, by allowing the nurse
to check whether a medication is appropriate.
Scholar mode: assessment of the condition of the patient
All participants considered the categorisation of patients into
cases requiring routine, vigilant and emergency care to be
appropriate. Two participants described recording personal
assessments of the patient’s condition as a way of making the
intuition of nurses explicit (A8). This function can be used, for
example, to back up the nurse’s opinion about a patient when
communicating with physicians (B1), or be included in patient
assessment tools (B2), as suggested by five participants. Three
participants find it threatening to share a personal assessment
with a computer because management can have access to this
information (C2) and use it when evaluating staff. Even though
the interviewer suggested that certain information would be
confidential, some participants did not want to record this type
of information on a computer, as they are not confident that it
would not be used by their managers.
Human mode: perception of the shift
The categorisation used in the human mode to assess a shift
(pleasantly calm, pleasantly stressed, unpleasantly calm,
unpleasantly stressed) was recognised and considered to be
appropriate by all participants. The main added value of this
function is that it acknowledges the emotional load of ICU
nursing (A6). A suggestion for another potential use of this
function is to couple the (personal) perception of the shift to the
(personal) assessment of the condition of the patient (B2). In
this way it can provide a more accurate picture of the patient
than current patient assessment tools do. Routine cases can be
very labour-intensive because they need a lot of psycho-social
attention, which is not measured with the current tools. Two
participants indicated that they find it inappropriate to share
emotional issues with a computer (C3). They prefer to share
i40
their emotions with real people, in particular colleagues or
relatives.
As illustrated by table 3, the study resulted in a wide variety
of design implications. Which implications are more promising
directions than others to pursue depends on the area of appli-
cation. The application area of this research is the digital support
of ICU nurses in their day-to-day experiences. Promising direc-
tions for this area are implications that relate to actual daily
working strategies which have the potential of being enhanced
by digital technology. Additionally, promising directions should
address a critical problem in the ICU work domain and improve
the current situation, not just change it (see Melles et al,12 also
based on Patterson et al13). For example, a promising direction
seems to be the idea of making the intuition of nurses tangible
by recording intuitive assessments of the condition of the
patient in a digital system. The recorded assessments can show
the patient assessments over time and can be used to back up
the nurse’s opinion about a patient when communicating with
physicians. This may contribute to the daily communication
and collaboration between physicians and nurses.
CONCLUSIONS
This study demonstrates the potential for implementing the
roles of practitioner, scholar and human in daily digital support
systems for ICU nurses. The requirements for support in all
three roles should be considered for each task. The necessity of
supporting nurses in their roles as practitioners is obvious, but
the scholar and human roles should be considered and supported
as well. More insight into the working routines and needs
related to these two roles is needed. A study of the needs related
to the human role has been conducted, and more design impli-
cations for this role have recently been published.12
This study also demonstrates that using a prototype is
a useful strategy for encouraging medical practitioners to
participate in generating design ideas in the early phase of the
product development process. Discussing functions with
medical practitioners, such as ICU nurses, provides the product
developer with insights about perceptions of added value and
the negative impact of ideas. Furthermore, this study shows that
a prototype can also trigger suggestions from the participants
about other uses, hence generating a participative design
approach between product developer and medical practitioner.
All of the practitioners’ remarks help the product developer to
obtain ideas that fit the actual working routines of the medical
practitioner. In addition, medical practitioners are given an idea
of the extent of technological possibilities. They are encouraged
to critically consider their own work situation and what they
might expect from future systems supporting their work. This is
particularly important with the current regulations concerning
early user involvement in the product development process and
postmarketing surveillance. At the moment, product developers
are responsible for involving users, but medical practitioners
should also take responsibility for recognising their needs and
make them known. Ensuring patient safety is the responsibility
of both product developers and medical practitioners.
Acknowledgements The authors gratefully acknowledge the participating
hospitals for their hospitality and in particular the ICU nurses who kindly consented
to take part in this study. Special thanks to P Craig and A Peeters for their help in
building the prototype. Also special thanks to the reviewers of this paper who
contributed greatly to its clarity and positioning and to other related work.
Competing interests None.
Provenance and peer review Not commissioned; not externally peer reviewed.
Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483

REFERENCES
1. Moreno RP, Rhodes A, Donchin Y. Patient safety in intensive care medicine: the
Declaration of Vienna. Intensive Care Med 2009;35:1667e72.
2. Donchin Y, Gopher D, Olin M, et al. A look into the nature and causes of human
errors in the intensive care unit. Qual Saf Health Care 2003;12:143e8.
3. Le Blanc PM, de Jonge J, de Rijk AE, et al. Well-being of intensive care nurses
(WEBIC): a job analytic approach. J Adv Nurs 2001;36:460e70.
4. Frank JR. The CanMEDS 2005 Physician Competency Framework. Ottawa: The
Royal College of Physicians and Surgeons of Canada, 2005.
5. Beyer H, Holtzblatt K. Contextual Design; Defining Customer-Centered Systems. San
Francisco: Morgan Kaufmann, 1998.
6. van Welie M. Task-based user interface design. Amsterdam: Vrije Universiteit, 2001.
7. Holmquist LE. Prototyping: generating ideas or cargo cult designs? Interactions
2005;12:48e54.
Qual Saf Health Care 2010;19(Suppl 3):i36ei41. doi:10.1136/qshc.2009.036483
Original research
8. Preece J, Rogers Y, Sharp H. Interaction design: beyond humanecomputer
interaction. New York: John Wiley & Sons, 2002.
9. Melles M, Freudenthal A, de Ridder H, et al. Designing for enhanced interpretation,
anticipation and reflection in the intensive care unit. In: Thissen W, Wieringa P,
Pantic M, Ludema M, eds. IEEE Conference on Systems, Man, & Cybernetics. The
Hague, The Netherlands: IEEE Systems, Man & Cybernetics Society, 2004:809e15.
10. Groen MAH. Technology, Work and Organisation: a Study of the Nursing Process in
Intensive Care Units. Maastricht: Rijksuniversiteit Limburg, 1995.
11. Desmet PMA. Designing Emotions. Delft: Delft University of Technology, 2002.
12. Melles M, Freudenthal A, de Ridder H. Supporting the Human Role in Intensive Care
Nursing. 17th World Congress on Ergonomics. Beijing, China: IEA, 2009.
13. Patterson ES, Woods DD, Tinapple D, et al. Using Cognitive Task Analysis (CTA) to
Seed Design Concepts for Intelligence Analysts under Data Overload. Human Factors
and Ergonomics Society 45th Annual Meeting. Minneapolis: HFES, 2001:439e43.
i41
 Original research
Department of Emergency
Medicine, Yale University
School of Medicine, New
Haven, Connecticut, USA
Correspondence to
Dr Leigh V Evans, Department of
Emergency Medicine, Yale
University School of Medicine,
464 Congress Avenue, Suite 260,
New Haven, CT 06519, USA;
leigh.evans@yale.edu
Received 4 March 2010
Revised 25 May 2010
Accepted 20 June 2010
i42
Simulation and patient safety: evaluative checklists
for central venous catheter insertion
Leigh V Evans, Kelly L Dodge
ABSTRACT
In the advent of concerns for patient safety, simulation
training is emerging as a method to train healthcare
providers to perform invasive procedures such as central
venous catheter (CVC) insertion while minimising harmful
complications to the patient. New technologies in
medical simulation have begun to shift research attention
to the performance component of clinical competency.
Accurate assessment of healthcare provider competence
is a major priority in medical education necessitating the
development of valid and reliable assessment tools. In
the past year alone, nine evaluative tools, both global
rating scales and procedural checklists, have been
published in the research literature to evaluate the
insertion of CVCs. A review of the advantages of
published evaluation tools helps inform users with regard
to the critical components necessary for a checklist.
Ease of use, ability to be completed by a non-expert,
categorical breakdown of critical actions involved in CVC
insertion and the need for a comprehensive stepwise
procedural checklist are discussed. The development of
an ideal checklist may improve future competency-based
training and performance evaluation in the clinical
setting. A more thorough understanding of the status of
checklists as evaluation tools in assessing performance
of invasive procedures will lead to better training
protocols and ultimately to improved patient safety.
In 2009, an explosion in American public interest in
the role of checklists began in January with the
crash landing of US Airways Flight 1549 on the icy
Hudson River and culminated in December with
the publication of The Checklist Manifesto: How To
Get Things Right1 by the writer and surgeon Atul
Gawande. While being hailed the world over as the
hero of the ‘miracle on the Hudson’ and the pilot of
the flight a National Transportation Safety Board
official described as the ‘most successful ditching in
aviation history,’2 Captain Chelsey Sullenberger
frequently reiterated that the successful landing of
the plane was the result of teamwork and adher-
ence to procedural checklists.3 In The Checklist
Manifesto, Gawande formulates an argument that
the use of a simple checklist can enable the surgeon
to reduce complications and mortality at virtually
no cost. He describes the capabilities and limita-
tions of checklists and how they can be used in
fields beyond medicine. Aviation checklists are brief
with just a few lines on each page, and each
checklist is applied to a different situation. There
are ‘normal’ checklists which are the routine lists
used for everyday flight operations and
‘non-normal’ checklists covering every conceivable
emergency situation. The checklists address situa-
tions most pilots never encounter during their
Qual Saf Health Care 2010;19(Suppl 3):i42ei46. doi:10.1136/qshc.2010.042168
entire career. Pilots turn to checklists when faced
with catastrophe because they are trained to do so
and because they believe that they work. The goal
of all of these checklists is to guide behaviour, by
either confirming routines or directing rarely
needed behaviour in the setting of a disaster.
While checklists have been widely used to guide
behaviour, they also have utility as evaluation tools.
In 2009, in the field of medical simulation, nine
articles4e11 were published with evaluative check-
lists designed to assess performance of central
venous catheter (CVC) insertion alone. Of human
factors and ergonomics concerns, new technologies
continue to introduce procedures that are difficult
to learn, have a thin margin of safety and have the
potential to inflict injury on patients. Simulation
training with partial task trainers for invasive
procedures has emerged as a method to develop
clinical competency and improve patient safety.
Simulators address the learning of intricate invasive
procedures by allowing the physician to repeat
motions in a stepwise fashion. The Accreditation
Council for Graduate Medical Education (ACGME)
Toolbox of Assessment Methods recommends the
use of simulation to assess competence and ranks
simulation as the ‘most desirable method’ for
assessing medical procedural skills in the patient
care domain.12
With advances in simulation technology, there is
a need for objective and reliable measures to eval-
uate the effectiveness of a teaching curriculum and
to provide a basis for assessment. Does the ideal
checklist for evaluation of procedural competency
have the same features as the behavioural checklists
described by Gawande and used in the aviation
industry? How are assessment tools designed,
validated and implemented? Does the rater using
the checklist as an evaluation tool need to be an
expert or can they be inexperienced? Is the check-
list used to assess performance in the simulation
laboratory transferable to the evaluation of
competency in invasive procedures on actual
patients in the clinical setting? It is the premise of
this article that a more thorough understanding of
the status of checklists as evaluation tools in
assessing performance of invasive procedures will
lead to better training protocols and ultimately to
improved patient safety. The article will focus on
checklists to evaluate CVC insertion.
STANDARDISED EVALUATIVE MEASURES: GLOBAL
RATING SCALES VS EVALUATIVE CHECKLISTS
Professional societies, accreditation bodies and
hospitals now recognise the growing importance of
training in invasive procedures and the need to
institute competency requirements for technical

skills. CVC insertion is one of the most commonly performed
invasive bedside procedures, with over five million CVCs
inserted annually in the USA.13 CVC insertion is also a common
source of morbidity and mortality with complication rates
reported as high as 15e25%.14 15 Despite the pressing need for
measures to determine competence in invasive procedures, there
are no standardised measures or data-driven recommendations.
For CVC insertion, the only quantitative evidence for compe-
tence suggests that operators who have inserted 50 or more
CVCs have half the complication rate of operators who have
performed fewer than 50 CVC insertions.14 However, it is also
now recognised that experience alone is an insufficient proxy for
competence.16 Studies suggest that neither medical students nor
residents are comfortable with their procedural skills. Based on
a series of surveys of medical students and recent graduates, it
became clear that many important medical procedures are
encountered far less frequently than clinical staff would have
believed necessary.17e20 Self-assessment of procedural compe-
tence has been demonstrated to be inaccurate.21 22 The need for
objective and reliable measures to evaluate effectiveness of
teaching curricula and to provide a basis for assessment of
procedural performance has led experts to recommend that
direct observation be combined with validated checklists.23 24
The effectiveness and impact of any training method require
an assessment strategy that yields valid, acceptable, reliable,
accurate and sensitive measures with which to quantify learning
and performance. What are the best assessment instruments for
measuring procedural skills? There are two general approaches
to rating technical performance: global rating scales and check-
lists. Global rating scales of the 5-point Likert scale variety are
subjective and have been validated for clinical skills examina-
tions involving patient assessment, decision-making and team
management25 26 Global rating scales are easy to use and do not
require much time to administer or specific training to imple-
ment but have poor inter-rater reliability and do not address
specific procedural shortcomings.27 Procedures tend to be
sequential and predictable and therefore may be better suited to
assessment with checklists. Checklists allow for the structured
assessment of the component skills of a procedure.28 Checklists
can discriminate between differing levels of performance,
particularly in settings where specific solutions are present.29e31
At a 2008 Academic Emergency conference in simulation, the
consensus of experts on the metrics and methodology of
assessing procedural skills using simulation stated that ‘check-
lists derived through task analysis appear to be well suited for
describing procedures, but it is not clear whether a combination
of checklists and global assessments is preferable.’32
McKinley and colleagues argue that global ratings of compe-
tence have advanced, and an evidence base for the reliability and
validity of assessments based upon them is emerging.8 33e35
They attempted to develop generic criteria for the assessment of
clinical procedural competence by performing a systematic
review of procedural skills assessment checklists published from
1990 to June 2005.8 Seven themes in 88 checklists were identi-
fied: procedural competence, preparation, safety, communication
and working with patient, infection control, postprocedural care
and team working. A total of 37 subthemes were also identified.
‘Infection control’ and ‘safety’ were assessed in checklists far less
frequently than the other themes.
CHECKLIST VALIDATION AND RELIABILITY
Validity implies that the results of an assessment represent what
the researcher intended for them to represent. Validity is very
Qual Saf Health Care 2010;19(Suppl 3):i42ei46. doi:10.1136/qshc.2010.042168
Original research
difficult to assess in a single study, so educators often settle for
proxy measures, such as construct validity.36 For high-fidelity
simulation in crisis resource management, formal evaluation
instruments known as the Ottawa Global Rating Scale (GRS)
and the Ottawa Crisis Resource Management (CRS) checklist
demonstrated construct validity in multiple domains.29 37 38
Both rating instruments were based on the work by Gaba and
colleagues.39 Construct validity was determined for both
instruments through a Delphi process, an iterative process
designed to reach consensus among experts on critical deci-
sions.40 An investigator administers repeated surveys to experts
who are unaware of each other’s identities; after analysing and
incorporating the experts’ responses from the first survey, the
investigator conducts a second survey round with the experts for
further feedback. Repeat rounds of surveys are administered
until the experts reach full consensus. For procedural checklist
validation, Goff and colleagues used construct validity to
distinguish between residency levels in the surgical skill perfor-
mance of obstetric and gynaecology residents with a one-way
analysis of variance with residence year as the independent
variable. For all seven surgical procedures evaluated, blinded
evaluators found significant differences between postgraduate
year (PGY) performance (PGY1 < PGY2, PGY3 < PGY4).36
Huang and colleagues sought to create a valid assessment tool
for faculty to use to assess resident performance in CVC inser-
tion.7 The authors convened experts to create an assessment tool
for CVC insertion using a modified Delphi method and then
applied the Angoff method to a second set of experts to deter-
mine minimum passing scores (MPSs) for the borderline trainee
and the competent trainee. With the Angoff method, content
experts are provided with a checklist of criteria and asked to
judge, item by item, whether or not a borderline trainee or
competent trainee would perform each step correctly. The MPS
required completion of 10 major and two minor criteria for
a trainee to show borderline proficiency with CVC insertion
under supervision. The MPS for competence was 17 major and
five minor criteria. The instrument demonstrated some evidence
of validity for the borderline trainee in that it correlated with
low-normal global rating scores and was consistent with the
authors’ hypothesis that the borderline trainee would have basic
proficiency under supervision but not be proficient yet without
supervision.
Reliability is an estimate of the dependability of a measurement
or device or test. The different types of reliability of assessment
tools include testeretest reliability, parallel forms of reliability,
internal consistency reliability and inter-rater reliability.41 For
assessments depending on human raters, the consistency
between raters is the most pertinent estimate of the reliability of
a procedural checklist and can be assessed with the Kappa
statistic.42 43
PROCEDURAL CHECKLISTS FOR CENTRAL VENOUS CATHETER
INSERTION
In 2009, nine procedural assessment tools were published for
central venous catheter insertion: seven checklists and two
global rating scales table 1.
Huang and colleagues created a consensus-driven procedural
assessment tool with data-driven standards for basic proficiency
and competency to be used by faculty experts to assess residents
as they perform CVC insertion.7 The authors created an
assessment tool for both the internal jugular (IJ) and subclavian
approach to assess resident performance on simulators. The IJ
checklist used a finder needle rather than ultrasound guidance,
i43
 Original research
Table 1 Checklists for central venous catheter insertion published in 2009
Global rating
Author Study type scale/checklist
44
Barsuk et al Cohort study Checklist
Britt et al5 Prospective, randomised controlled Checklist
Evans et al4 45 Prospective, randomised controlled Checklist
Huang et al7 Delphi and Angoff method Checklist
Millington et al10 Prospective beforeeafter design Global rating scale
and checklist
Rosen et al9 Prospective observational Checklist
Woo et al11 Prospective beforeeafter design Checklist and global
rating scale
and the checklists ended with insertion of the guidewire rather
than insertion of the CVC. A global rating scale (1e5) demon-
strated some construct validity for the determination of the
borderline trainee with a global rating of three. The authors
hypothesised that there would be a correlation between the
minimum passing score for competence and the global rating
scale of 5, but instead found that none of the trainees met this
minimum passing score. A low number of performed procedures
by the cohort of residents in this study may have been inade-
quate to confer confidence, since competence in the actual
patient setting as defined by decreased complication rates is not
achieved until operators have performed 50 or more CVC
insertions.14
As part of an observational, prospective study, Rosen and
colleagues used a procedural checklist as part of a curriculum to
provide trainees with the necessary skills to make a smooth and
competent transition from a non-human tissue model (tunnelled
0.2 mm thick rubber tubing vessels through raw, whole
chickens) to live patients.9 Their checklist is divided into five
categories: preprocedure preparation, preparing the site, vascular
access using ultrasound guidance, central line placement and
safety. Rather than the typical ‘yes/no’ checklist, this checklist
used a 0e5 grading scale, with 0 reflecting resident complete
unfamiliarity with the step necessitating visual and verbal
instruction in order to perform the step (‘stumped’), to 5,
reflecting procedural independence with total confidence and
without error. The checklist was used as both a guide through all
of the steps of CVC insertion during hands-on personalised
training and also as an assessment tool for immediate post-
training evaluation and follow-up assessment at 3e4 weeks.
Unlike the Huang checklist, this checklist included all steps of
the procedure from preprocedure preparation to disposing of
sharps and confirmatory chest x-ray.
The efficacy of simulators in teaching CVC insertion in an
internal medicine residency programme was studied by Mill-
ington and colleagues using both a global rating scale and
a checklist.10 Primary outcome of CVC insertion was assessed
based on a global rating score, and secondary measures included
checklist score, knowledge score and self-reported confidence.
Pre- and postsimulation training CVC insertions on a simulator
were videotaped and assessed by two blinded expert raters who
were not aware of whether the performances were recorded
during presimulation or postsimulation training sessions. The
global rating scale with scores of 1 (poor) to 5 (excellent) was
divided into four categories (time and motion, instrument
handling, flow of operation and forward planning, knowledge of
instruments) and an overall score. The 10 item procedural
checklist began with site preparation and ended with securing
the catheter. A skill assessment checklist included number of
i44
Ultrasound- Simulator versus
Anatomical location guided actual patients Rater/evaluator
Internal jugular subclavian Yes Simulator Expert authors
Subclavian No Actual patients Surgeon experts
Internal jugular subclavian Yes Actual patients Non-physician
Internal jugular subclavian No Simulator Faculty experts
Internal jugular No Simulator Faculty experts
Non-human tissue model Yes Non-human Senior medicine
tissue model residents
Internal jugular Yes Simulator Expert reviewers
attempts to locate the vein, number of attempts to insert the
catheter and total time of the procedure. Thirty participants
completed the study protocol. The median global rating scale
scores, checklist scores, mean knowledge scores and participant
confidence ratings all increased. In addition, the time required for
CVC insertion decreased. Woo and colleagues designed a similar
prospective beforeeafter study examining the efficacy of
a simulator training programme in ultrasound guided IJ CVC
insertion for emergency medicine residents.11 (REF) Residents
gained confidence and felt their technical skills in ultrasound
guided CVC insertion improved, and all participants felt satisfied
with the training programme. Their checklist was divided into
five categories: cleansing and draping the skin, probe prepara-
tion, needle insertion, Seldinger technique and postprocedure
care. The authors further delineated between major (14) steps
and minor (four) steps. As examples, ‘putting on gloves’ was
considered a major step and ‘warning the patient about cold
chorhexidine’ was considered a minor step. The assessment tool
concluded with a global rating scale from one (poor) to seven
(excellent).
Two prospective, randomised controlled studies examined the
impact of simulation training on performance in the clinical
setting on actual patients. Britt and colleagues randomised
junior residents on the trauma rotation to standard (n¼21)
versus simulation (n¼13) training.5 Subclavian CVC insertions
were observed in the intensive care unit by a trauma fellow or
critical care attending. The simulated training group had
a significantly higher level of comfort than the training group.
There were significantly more complications in the standard
group. An objective structured checklist was completed with 10
performance variables from hand washing to inserting the CVC
over the wire. The performance measures did not include steps
beyond catheter insertion such as flushing the ports, suturing or
ordering a postprocedure chest x-ray. The authors included the
need for senior residents to take over the procedure or inability
to insert the catheter as complications rather than performance
measures. The expert raters were not blinded to the groups
which may have introduced bias into the study. Evans and
colleagues utilised a 50-point procedural checklist in a prospec-
tive randomised controlled study of PGY1 and PGY2 residents
(n¼185) examining the impact of a hands-on competency-based
ultrasound-guided simulation training programme on CVC
insertion performance in the hospital setting.45 Blinded inde-
pendent raters observed 495 CVC insertions by 115 residents
over a 21-month period. Residents with the hands-on compe-
tency based training had increased rates of successful first
cannulation and overall CVC insertion success. There was no
significant difference between groups related to technical errors
or mechanical complications.
Qual Saf Health Care 2010;19(Suppl 3):i42ei46. doi:10.1136/qshc.2010.042168

CHECKLIST DESIGN
Based on over 30 years of experience in designing evaluative
checklists, Stuffelbeam created a ‘checklist for developing
checklists’ with 12 steps.46 47 Based on our area of expertise, we
will use our 50-point checklist for ultrasound guided CVC
insertion4 as an example of developing an evaluative checklist
based on Stuffelbeam’s 12 steps:
1. Define the focus of the checklist task by determining the area
of interest and the checklist’s intended uses, studying the
relevant literature and engaging in conversations with
experts in the content area. We created our checklist to be
used by trained non-expert independent raters to observe in-
hospital CVC insertion as part of a prospective, randomised
controlled study evaluating the impact of hands-on compe-
tency-based simulation training on successful CVC insertion.
We consulted with surgeons and intensivists at our
institution who supervise CVC insertions performed by
resident physicians.
2. Create a candidate list of checkpoints based on well-
established criteria. Each checkpoint needs to be clearly
defined with descriptors and definitions. We observed CVC
insertions and created checkpoints using a kinesiology
approach48 for each individual step from consent of the
patient to ordering a chest x-ray at the end of the procedure.
3. Classify the checklist into categories. We created 13 different
categoriesdsix of which included observable steps in the
procedure and seven of which were descriptive such as
cannulation site and catheter type.
4. Define a rationale for each category and then review the
checkpoints within each category for inclusiveness and
clarity. Checkpoints can then be added, subtracted or
rewritten. We rationalised the categories we chose by
grouping related components of the procedure together in
case the operator chose to perform a ‘category’ in a different
chronological sequence than if all 50 steps were included in
a stepwise fashion. For instance, the procedural steps for
‘Ultrasound Preparation’ and ‘CVC Kit Preparation’ were
separate categories, but which preparation category was
performed first by the operator was irrelevant.
5. Order the categories and determine whether the order
sequence is an important consideration regarding the
intended uses of the checklist. For CVC insertion, certain
steps of the procedure require a specific order. For instance,
we considered the incorrect order of placing the dilator over
the wire prior to nicking the skin with a scalpel to be
a technical error.
6. Engage potential users to review and critique the checklist.
We used the training of independent raters with our checklist
as an opportunity to critique the checklist. During training,
independent raters completed the checklist while observing
CVC insertion videotapes with choreographed errors. This
phase of checklist development allowed us to clarify
definitions of individual checkpoints and determine inter-
rater reliability.
7. Revise and rewrite checklist content. Items which seemed
ambiguous to the raters were revised and rewritten.
8. Format the checklist to serve the intended use. Should the
checklist be scored with individual checklist items receiving
differential weight? Should items be measured using
nominal or ordinal data? Because we used non-expert
independent raters, we attempted to avoid the need for the
independent raters to make any judgement during the
observation of the CVC insertion. Therefore, our checklist
Qual Saf Health Care 2010;19(Suppl 3):i42ei46. doi:10.1136/qshc.2010.042168
Original research
included only ‘Yes,’ ‘No’ and ‘Comment’ columns, rather
than Likert-type scales.
9. Evaluate the checklist. We measured the reliability of the
independent raters’ observational assessment of each data
point on the checklist by crude agreement with the standard
answer for each videotape with choreographed errors. The
hired group had 100% crude agreement for 30 of the 50
checklist data points and over 88% for all remaining checklist
items except for keeping hands on the guidewire at all times
during CVC insertion (77%) and determining the number of
times hands were removed from the guidewire (73%).4
10. Finalise the checklist. We submitted the checklist to the
hospital IRB for approval.
11. Apply and disseminate the checklist. The checklist was used
to observe CVCs inserted in the hospital setting. It was
sealed in an envelope and attached to the CVC kits to assure
rapid availability to the IR observing the procedure.
12. Periodically review and revise the checklist. In January 2010,
we conducted a workshop at the International Meeting for
Simulation in49 attended by 50 participants, the majority of
whom were physicians and several of whom were directors
of medical simulation programmes at US academic centres.
Of the attendees, there were none utilising a uniform
published checklist for CVC competency and procedural
evaluation.
During the initial stage of the workshop, participants were
randomly divided into small groups, with each group having
access to one of the procedural checklists (Barsuk,6 Millington
global rating scale,10 Millington checklist,10 Evans,4 Rosen).9
They then observed a CVC insertion under ultrasound guidance
that was performed with choreographed technical errors. Each
participant utilised the assigned checklist to assess performance.
After completion of the CVC insertion, each group discussed
inter-rater reliability and the pros and cons of the assigned
checklist. In an entire group discussion, the 50 participants
believed that the global rating scale was inferior to the itemised
checklists, with regards to assessing performance and compe-
tency. Each group was then given copies of the remaining
checklists. They observed a novice operator in the insertion of
the CVC utilising these checklists as an evaluation tool. Once
again, we then discussed the checklists as a whole group. This
workshop format enabled participants to come to a consensus
with regards to the critical components necessary for a checklist,
which included: ease of use, ability to be completed by a non-
expert, categorical breakdown of critical actions involved in
CVC insertion, and the need for a comprehensive stepwise
procedural checklist. We used this interdisciplinary discussion
for the revision of our existing checklist in order to improve
future competency-based training and performance evaluation
in the clinical setting.
LOOKING AHEAD
It has become increasingly clear that real-world practice in
invasive procedures is no longer appropriate because of safety
concerns, and therefore simulation offers the opportunity to
optimise the interaction of cognitive, technical and ergonomic
factors that are needed for safe and reliable performance. Just as
in the aviation industry, healthcare providers will need to have
reached an acceptable level of performance in a simulated setting
in order to reduce exposure of patients to the risks associated
with operator inexperience and vulnerability. Over the next
decade, it is reasonable to assume that the training of physicians
will become much more performance oriented. The advantages
i45
 Original research
of simulation to both providers and patients have yet to be fully
realised; however, the emerging efforts at standardisation and
demonstration of competency in technical skills are encouraging.
Demonstration of proficiency in performance of invasive proce-
dures will require the development of both valid and reliable
evaluation tools. A sound evaluation checklist clarifies the criteria
that should be considered when evaluating performance and aids
the evaluator in following the procedure in a stepwise fashion. A
good checklist enhances the assessment’s objectivity, credibility
and reproducibility. The recent work to create evaluative check-
lists to assess CVC insertions reflects the growing effort to
address the previously neglected performance component of
clinical competency. As more precise evaluation tools are devel-
oped, improved curricula can be developed to train healthcare
providers to competency in the controlled, simulated environ-
ment prior to performing the procedure on actual patients.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Gawande A. The Checklist Manifesto: How to Get Things Right. New York:
Metropolitan Books, 2009.
2. Olshan J, Livingston I. ‘Quiet air hero is captain America.’ New York Post 2009.
3. Phillips M. ‘Sully, Flight 1549 Crew Receiver Keys to New York City.’ The Middle
Seat, blog. Wall Street J 2009.
4. Evans LV, Morse JL, Hamann CJ, et al. The development of an independent rater
system to assess resident competence in invasive procedures. Acad Med
2009;84:1135e43.
5. Britt RC, Novosel TJ, Britt LD, et al. The impact of central line simulation before the
ICU experience. Am J Surg 2009;197:533e6.
6. Barsuk JH, McGaghie WC, Cohen ER, et al. Simulation-based mastery learning
reduces complications during central venous catheter insertion in a medical intensive
care unit. Crit Care Med 2009;37:2697e701.
7. Huang GC, Newman LR, Schwartzstein RM, et al. Procedural competence in internal
medicine residents: validity of a central venous catheter insertion assessment
instrument. Acad Med 2009;84:1127e34.
8. McKinley RK, Strand J, Ward L, et al. Checklists for assessment and certification of
clinical procedural skills omit essential competencies: a systematic review. Med Educ
2008;42:338e49.
9. Rosen BT, Uddin PQ, Harrington AR, et al. Does personalized vascular access
training on a nonhuman tissue model allow for learning and retention of central line
placement skills? Phase II on the Procedural Patient Safety Initiative (PPSI-II). J Hosp
Med 2009;4:423e9.
10. Millington SJ, Wong RY, Kassen BO, et al. Improving internal medicine residents’
performance, knowledge, and confidence in central venous catheterization using
simulators. J Hosp Med 2009;4:410e16.
11. Woo MY, Frank J, Lee A, et al. Effectiveness of a novel training program for
emergency medicine residents in ultrasound-guided insertion of central venous
catheters. CJEM 2009;11:343e8.
12. Accreditation Council on Graduate Medical Education. General Competency
Standards. http://www.acgme.org/ outcome/comp/GeneralCompetenciesStandards
21307.pdf (accessed 11 Feb 2010).
13. Raad I. Intravascular-catheter-related infections. Lancet 1998;351:893e8.
14. Sznajder JI, Zveibil FR, Bitterman H. Central venous catheterization. Failure and
complication rates by three percutaneous approaches. Arch Intern Med
1986;146:259e61.
15. Merrer J, De Jonghe B, Golliot F. Complications of femoral and subclavian venous
catheterization in critically ill patients: A randomised controlled trial. JAMA
2001;286:700e7.
16. American Board of Internal Medicine. Procedures required for internal medicine.
http://www.abim.org/certification/policies/imss/im.aspx (accessed 28 Feb 2010).
17. Hanskar S, Seim S. Medical student’s experiences in medical emergency
procedures upon qualification. Med Educ 1985;19:294e8.
18. Wakeford R, Roberts S. An evaluation of medical student’s practical experience
upon qualification. Med Teach 1982;4:140e3.
19. Jolly BC, Macdonald M. Education for practice: the role of practical experience in
undergraduate and general clinical training. Med Educ 1989;23:189e95.
i46
20. O’Connor HM, Davidson J. Emergency medicine skills: are primary physicians
adequately prepared? Can Fam Phys 1992;38:1789e93.
21. Duffy FD, Holmboe E. Self-assessment in lifelong learning and improving
performance in practice: Physicians know thyself. JAMA 2006;296:1137e9.
22. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment
compared with observed measures of competence: a systematic review. JAMA
2006;296:1094e102.
23. Reznick R. Teaching and testing technical skills. Am J Surg 1993;165:358e61.
24. Reznick RK, MacRae H. Teaching surgical skillsdchanges in the wind. N Engl J
Med 2006;355:2664e9.
25. Rothman A, Cohen R. A comparison of empirically- and rationally-defined standards
for clinical skills checklists. Acad Med 1996;71(10 Suppl):S1e3.
26. Bardes C, Colliver J, Alonso D, et al. Validity of standardized-patient examination
scores as an indicator of faculty observer ratings. Acad Med 1996;71(1 Suppl):
S82e3.
27. Brasel KJ, Bragg D, Simpson DE, et al. Meeting the Accreditation Council for
Graduate Medical Education competencies using established residency training
program assessment tools. Am J Surg 2004;188:9e12.
28. Rosethal R, Gantert W, Hamel C, et al. Assessment of construct validity of a virtual
reality laparoscopic simulator. J Lap Adv Surg Tech 2007;17:407e13.
29. Kim J, Neilipovitz D, Cardinal P, et al. A comparison of global rating scale and
checklist scores in the validation of an evaluation tool to assess performance in the
resuscitation of critically ill patients during simulated emergencies. Sim Healthcare
2009;4:6e16.
30. Issenberg SB, Gordon MS, Greber A. Bedside cardiology skills training for the
osteopathic internist using simulation technology. J Am Osteopath Assoc
2003;103:603e7.
31. Wayne DB, Fudala MJ, Butter J, et al. Comparison of two standard-setting methods
for advanced cardiac life support training. Acad Med 2005;80:S63eS6.
32. Lammers RL, Davenport M, Korley F, et al. Teaching and assessing procedural
skills using simulation: metrics and methodology. Acad Emerg Med
2008;15:1079e87.
33. van Luijk SJ, Vleuten Cvd. A comparison of checklists and rating scales in
performance based testing. In: Hart IR, Harden RM, Marchais JD, eds. Current
Developments in Assessing Clinical Competence. Ottawa: IRH Medical Ltd.
1992:357e62.
34. Regehr G, Freeman R, Robb A, et al. OSCE performance evaluations made by
standardised patients: comparing checklists and global rating scores. Acad Med
1999;74:135e7.
35. Keynan A, Friedman M, Benbassat J. Reliability of global rating scales in the
assessment of clinical competence of medical students. Med Educ
1987;21:477e81.
36. Goff B, Mandel L, Lentz G, et al. Assessment of resident surgical skills: is testing
feasible? Am J Obstet Gynecol 2005;192:1331e40.
37. Adler M, Ziglio E. Gazing into the Oracle: the Delphi Method and Its Application to
Social Policy and Public Health. London: Jessica Kingsley Publishers, 1996.
38. Downing S. Validity: on the meaningful interpretation of assessment data. Med
Educ 2003;37:830e7.
39. Gaba DM, Fish KJ, Howard S. Crisis Management in Anesthesia. London: Churchill
Livingstone, 1994.
40. Clayton M. A technique to harness expert opinion for critical decision-making tasks
in education. Educ Psychol 1997;17:373e86.
41. Peyre SE, Peyre CG, Hagen JA, et al. Reliability of a procedural checklist as a high-
stakes measurement of advanced technical skill. Am J Surg 2010;199:110e14.
42. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible
absolute passing scores on performance examinations in health professions
education. Teach Learn Med 2006;18:50e7.
43. O’Connor HM, McGraw R. Clinical skills training: developing objective assessment
instruments. Med Educ 1997;31:359e63.
44. Barsuk JH, Cohen ER, Feinglass J, et al. Use of simulation-based education to
reduce catheter-related bloodstream infections. Arch Intern Med 2009;169:1420e3.
45. Evans LV, Dodge KL, Shah TD, et al. Simulation training in central venous catheter
insertion: improved performance in the clinical setting. Acad Med 2010. (accepted).
46. Stufflebeam D. Guidelines for developing evaluation checklists: the checklists
development checklist (CDC). Evaluation Checklists Project 2000. http://www.
wmich.edu/evalctr/checklists (accessed 1 Mar 2010).
47. Stufflebeam D. Evaluation checklists: practical tools for guiding and judging
evaluations. Am J Eval 2001;22:71e9.
48. Martin M, Scalabrini B, Berard D. Training for competence in critical surgical skills:
the impact of kinesiology-based instruction on outcome. The Society of University
Surgeons Sixty-First Annual Meeting, Abstract paper no 40.
49. Evans LV, Dodge KL, Shah TD, et al. Evaluation Methods for Invasive Procedures: the
Use of Checklists and Global Rating Scales. Phoenix, AZ: International Meeting of
Simulation in Healthcare, 2010.
Qual Saf Health Care 2010;19(Suppl 3):i42ei46. doi:10.1136/qshc.2010.042168

Correspondence to
Dr R Kneebone, Imperial College
London, Department of Surgery
and Cancer, 2nd Floor, Paterson
Centre, St Mary’s Hospital,
Praed Street, London W2 1NY,
UK; r.kneebone@imperial.ac.uk
Accepted 20 June 2010
Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424
Simulation, safety and surgery
Roger Kneebone
ABSTRACT
Objectives This paper explores the place of simulation in
contemporary healthcare education and training,
highlighting the challenges of recreating complex clinical
settings which can support the development of
competent, rounded and caring practitioners, and
address issues around human factors as well as
technical skill. It frames the relationship between clinical
and simulation-based practice as a mutually dependent,
two-way process.
Discusssion According to this view, simulation is less
like a photograph of clinical care than a painting of itda
process that requires selection and interpretation. The
paper presents simulation as a canvas on which to paint
this picture. To be effective, simulation must mirror the
essentials of a clinical setting without reproducing every
detail. After highlighting key issues with current
approaches to simulation, the paper considers how
authenticity and perceived realism can be heightened
through innovative uses of technology and design,
putting forward a conceptual framework based on the
notion of ‘circles of focus.’ The paper then outlines the
concept of Distributed Simulation, using low-cost,
portable yet immersive environments to address
limitations of access to dedicated facilities.
Conclusion The paper concludes by considering
theoretical and practical implications of these
innovations, focussing especially on surgery and other
craft specialties.
INTRODUCTION
This paper explores the place of simulation in
contemporary healthcare education and training. It
highlights the challenges of recreating complex
clinical settings which can support the develop-
ment of competent, rounded and caring practi-
tioners and address issues around human factors as
well as technical skill. It frames the relationship
between clinical and simulation-based practice as
a mutually dependent, two-way process.
According to this view, simulation is less like
a photograph of clinical care than a painting of
itda process that requires selection and interpre-
tation. The paper presents simulation as a canvas
on which to paint this picture. To be effective,
simulation must mirror the essentials of a clinical
setting without reproducing every detail. The paper
will consider some theoretical and practical impli-
cations of this proposition, focussing especially on
surgery and other specialties which demand high
levels of manipulative dexterity.
BACKGROUND
In recent years, the landscape of clinical care has
undergone profound upheaval. Time-honoured
ways of thinking and doing are being challenged by
Original research
widespread change, with new techniques for
investigation, diagnosis and treatment emerging
constantly. At the same time, the duration of
training programmes is being drastically reduced,
and work hours restrictions are bringing about
fundamental organisational changes on both sides
of the Atlantic. One obvious effect of shortened
training is a curtailment of opportunities for clin-
ical experience, especially as shift working rather
than continuous care becomes the norm.
The professional climate is also changing radi-
cally. It is no longer ethically acceptable (if it ever
was) for clinicians in training to learn ‘on’ patients.
This reduction in clinical exposure (previously built
upon a prolonged apprenticeship extending over
many years) poses obvious challenges. Already
there is widespread concern that doctors
completing their specialist training in areas such as
surgery and the interventional specialties may
simply not gain the breadth of skill and experience
to deal confidently with any clinical problem they
may encounter.1 2
At first sight, simulation offers attractive solu-
tions to many of these issues. Practising on inani-
mate models in the safety of a skills centre, the
argument goes, allows learners at all stages to gain
component skills without endangering patients.
This argument is especially compelling in the face
of dwindling opportunities for clinical exposure,
framing simulation as an alternative means of
gaining experience. The case is bolstered by clear
parallels with other safety-critical professions, such
as aviation, where simulation-based training is
accepted as an article of faith.3 Simulation therefore
offers a proxy for those elements of education and
training which are no longer available within
a clinical setting.3 4
This line of reasoning is powerful, and simulation
has now moved to centre stage in many countries.
The UK’s Chief Medical Officer, for example,
explicitly states that simulation will be of central
importance in healthcare education, especially for
surgery and related craft specialties.5 Yet, on closer
examination, the picture is less clear than it first
appears. What exactly are the benefits and limita-
tions of simulation? What is the evidence that
experience gained within simulation is transferable
to the real world? What might be the drawbacks of
an increasing reliance upon simulation-based
education? And if simulation is as valuable as it
seems, how can it be made available to all those who
would benefit?
A key issue concerns risk.6 For many people,
simulation equates safety with absence of risk. This
reflects a growing climate within healthcare of
aversion to risk generally, and a philosophy of risk-
free training. In reality, however, all clinical care
entails risk, and its effective management is key to
i47
 Original research
becoming a mature clinician. Understanding the impact of risk
or danger on clinical judgement and skill is a crucial element of
becoming expert.7 Yet if risk is stripped away from training,
clinicians may be ill-prepared when things go wrong with their
patients. To be an effective proxy for clinical experience, there-
fore, simulation should recreate the sensations and perceptions
of danger while still ensuring that patients are not jeopardised.
Achieving this balance poses considerable challenges.
This paper will explore these issues and put forward possible
solutions. In it, I will argue that simulation’s major benefit is as
an adjunct to clinical practice, that its value and relevance
depend upon a close alignment with the clinical world, but that
the world of simulation and the clinical world it purports to
represent are frequently divided. To be effective, simulation
should:
< recreate authentic conditions of the clinical work environ-
ment;
< reflect the relationship of care between patient and clinician;
< address the needs of individual learners;
< be perceived as locally owned;
< optimise the level of ‘imaginative work’ required for effective
engagement;
< allow participants to develop expertise while experiencing
danger safely;
< contribute to personal development and professional identity.
PROBLEMS WITH CURRENT SIMULATION
Two approaches to simulation are commonly encountered. The
first addresses so-called ‘simple’ procedural skills such as wound
closure, intravenous infusion and urinary catheterisation. Else-
where, I have argued that simplicity is in fact a contestable
concept, and that the determination of what is ‘simple’ lies in
the hands of expert clinicians whose conception of simplicity
may be quite different from a novice’s.8 The experience of
working on a model arm, for example, is qualitatively entirely
different from performing a procedure on a living person. For
a novice, performing this apparently straightforward procedure
clinically may in fact be extremely challenging, especially in the
context of a sick or demanding patient where a variety of
complex skills must be integrated under pressure.
Especially in the early stages of learning, students commonly
focus on technical aspects such as needle placement and knot
tying, using isolated benchtop models which are widely avail-
able in many institutions. Yet such practice overlooks the crucial
importance of context in applying any skill. To be effective,
simulation must be aligned with the needs of each learner and
their level of development. If a simulation is too simple and
undemanding, it will be perceived as irrelevant and of little
value. If too complex and taxing, it may either overwhelm
learners and damage their confidence, or waste resources by
providing unnecessary sophistication. Yet this dynamic of
adjustment is difficult to achieve, especially when dealing with
large numbers of participants with varying levels of experience.
At the other pole lie complex, high-fidelity and immersive
simulations, allowing clinical teams to practise managing
emergency situations. Simulation of this kind was pioneered by
anaesthetists, achieving high levels of realism and value for
anaesthetic teams and allowing a range of complex or
uncommon conditions to be addressed. In such simulations, the
responses of a patient under general anaesthetic are conveyed by
a mannekin connected to an anaesthetic machine. Patients and
their pathophysiological responses are ‘represented’ by traces on
monitor screens and haemodynamic responses to drugs, creating
a highly engaging experience.9e12
i48
For anaesthetic teams, such proxies for a real person closely
approximate the conditions of actual practice and provide high
levels of authenticity. For surgeons, however, the picture is
different. Existing models simply do not recreate the look and
feel of living human tissue with a similar level of convincing-
ness; this presents a real problem in terms of willing suspension
of disbelief and the amount of ‘imaginative work’ required of
participants.
AUTHENTICITY
How can we ensure that simulation mirrors real life for
surgeons? As long ago as 1993, Grant Wiggins wrote:
If we want competent performance later, we need to introduce
novices to that performance from day one. Only a deep and ancient
prejudice about academic learning keeps us thinking that
intellectual competence is achieved by accretion of knowledge and
movement through simple logical elements to the complex
wholedinstead of movement from a crude grasp of the whole to
a sophisticated grasp of the whole.13
For Wiggins, the notion of authenticity is central. Authen-
ticity is about replicating the diverse and rich contexts of
performance, meeting ‘an obligation to make the student expe-
rience [.] tasks under constraints as they typically and ‘natu-
rally’ occur, with access to the tools that are usually available for
solving such problems.’13 Wiggins offers key characteristics of
authenticity as:
1. Engaging in worthy problems and questions of importance;
2. Faithfully representing contexts encountered in the field;
3. Addressing non-routine and multistage tasks;
4. Requiring the student to produce quality performance;
5. Providing concurrent feedback and allowing self-adjustment;
6. Requiring trained assessor judgement;
7. Searching for patterns of response in diverse settings.
Although initially framed in terms of intellectual competence,
Wiggins’ arguments hold especial force for specialties such as
surgery, where craftsmanship and workmanship are key to
clinicians’ identity as well as to the outcomes of their work.
Sennett, for example, refers to craftsmanship as ‘an enduring,
basic human impulsedthe desire to do a job well for its own
sake,’14 while Pye distinguishes between the ‘workmanship of
risk’ (where the quality of the result is uncertain during the
process of making) and the ‘workmanship of certainty’ (where
the quality of the result is exactly determined in advance, as
with a factory-made object).15 Clinical practice clearly involves
workmanship of risk, whose outcome is the product of
a dynamic relationship between clinicians (skilled craftsmen, in
Pye’s terms), a living patient and a complex environment with
multiple variables.
ACHIEVING REALISM
So, how can such authenticity be achieved within simulation? A
central challenge is how to recreate the essence of real-life
experience. Clinical practice is built upon the relationship
between two peopleda patient and a clinician. The complexity,
richness, uniqueness and challenge of clinical care depend on this
human connection. And making sense of this complexity lies at
the heart of being a clinician. If simulation is to engage with this
complexity, it must somehow mirror the unruliness of real
lifedit must allow Wiggins’ progression ‘from a crude grasp of
the whole to a sophisticated grasp of the whole.’13
A crucial distinction is between simulations (recreating
a whole clinical event) and simulators (physical models,
mannekins or computer programs). Often, both in simple and in
Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424
 Original research

complex simulations, a simulator is the focal point of the

experience. In this case, the primary relationship lies between
a person (the learner) and a machine (the simulator). However
sophisticated the machine, it remains a machine. This imposes
inescapable constraints on authenticity.

HYBRID SIMULATION
So, how can this central relationship between two people (rather
than between a person and a machine) be provided within
simulation? A logical starting-point is to place a real person
within the simulation. Of course Simulated/Standardised
Patients (SPs) have been central to healthcare education for
decades, especially when addressing consultation and history-
taking skills. In such scenarios, trained actors play the role of
patient. Until recently, however, invasive procedures were treated
in a different fashion, placing inanimate models at the focus.
Work by our group has pioneered the concept of hybrid
simulation, where inanimate models are aligned with SPs
playing patient roles to create a setting which requires an inte- Figure 2 Hybrid simulation using seamless prosthetic wound.
grated approach. Participants have to interact with the patient
as they perform a procedure, avoiding any split between Some, but not necessarily all, of these elements would involve
so-called technical and non-technical skills. This recreates the a human ‘patient.’ During the operative phase, for instance, the
integrated nature of clinical care, where such splits are both anaesthetised patient might be represented by an inanimate
artificial and unhelpful.8 16e21 model. At present, realistic models for operative surgery are
Our initial work aligned SPs with existing models, covering conspicuous by their absence. Extension of our work with
the join with a drape (figure 1). Although technically crude, this prosthetics is leading to custom made models aimed at surgeons
provided surprisingly high levels of perceived realism and and their immediate teams (figure 3).
engagement. Yet there are obvious limitations to scenario design
if models have to be contrived so as to conceal a join. Current WIDENING ACCESS
work within our group is using prosthetics expertise from film Most current immersive simulation relies upon dedicated static
and television to create ‘seamless simulation’dhighly realistic facilities which are scarce, expensive and resource-intensive, and
models which are attached to a person in such a way that the which lie beyond the reach of many clinicians in training. In
join cannot be seen (figure 2). Preliminary studies have demon- such centres, activities are driven by forces (such as ensuring
strated extremely high levels of engagement by participants, and sufficient throughput to survive financially) which may be out
further work is exploring this concept systematically. of alignment with the educational needs of those who attend. In
consequence, there may be a built-in lack of responsiveness to
SEQUENTIAL SIMULATION individual learners and a mismatch with learning goals. This
Simulation activity typically revolves around a single encounter may result in a perception that simulation-based education is
or episode of care. Yet, actual practice is a continuum. In surgery, being imposed from without, rather than being ‘owned’ by
for example, an evolving relationship is built up, which extends participants themselves. The dominant agenda should therefore
from outpatient referral, discussion of management options, be clinical rather than organisational, framing simulation as
preoperative assessment, the operation itself and postoperative a means of strengthening elements of clinical practice rather
recovery, to discharge into the community. than functioning in a self-contained universe of its own.
Within our group, the concept of sequential simulation aims If immersive simulation is to be of value, it should be available
to recreate this longitudinal aspect of care by sampling ‘frames’ to all who need it. At present, however, immersive facilities are
from a patient’s trajectory. In the case of surgery, this might
include a preoperative encounter between surgeon and patient
(exploring options and gaining consent), an operation under
general anaesthetic, and postoperative assessment and detection
of complications.

Figure 1 Hybrid simulation using strap-on suture pad. (Posed by

a model). Figure 3 Simulated laparotomy.

Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424 i49

 Original research
restricted to a relatively small number of dedicated simulation
centres, usually located in major cities. Such centres provide
realistic, quasi-clinical settings which closely resemble real
environments, while digital recording technology allows activi-
ties to be recorded and used for feedback and debriefing. In
addition, specialised facilities such as control rooms and
debriefing spaces allow activities that would not be possible in
clinical practice.
Much attention is paid to the indiscriminate reproduction of
a whole environment, ensuring that all components are
provided. Such facilities commonly provide faithful reproduc-
tions of clinical environments (such as an operating theatre or
ITU). As many elements as possible of the original setting are
provided, including equipment such as operating lights and
tables, anaesthetic machines and storage facilities. Based closely
on real settings, these allow clinical teams to take part in
scenarios based around common or important clinical situations.
Providing and maintaining such environments is costly and
resource-intensive. Largely because of their cost and scarcity,
such centres are only available to a limited number of potential
users. How then might it be possible to retain the essence of
immersive simulation, yet make it more accessible? An inter-
esting alternative is ‘in situ simulation,’ where simulators and
simulated scenarios are taken to actual clinical settings.22 23
Although this offers obvious attractions, the practical difficul-
ties of aligning such simulations with service demands of
the UK National Health Service have led us to explore other
approaches.
DISTRIBUTED SIMULATION
We have developed the concept of Distributed Simulation
(DS).24 The underlying philosophy of DS is to provide simula-
tion facilities that are ‘good enough’ to engage participants and
achieve learning goals, yet are low-cost, portable and able to be
erected in a variety of clinical or non-clinical locations (figure 4).
Instead of the faithful reproduction of an actual setting, only
salient features are selected and recreated.
The first function of DS is to establish a boundary. To draw on
terminology from theatre studies, simulated clinical activity
occurs within a conceptual enclosure (a ‘space’) which is inde-
pendent of its actual geographical location (the ‘place’).25 26 In
order to function effectively, this space must be delimited from
its surroundings, so that those within it can perform without
distraction from the world outsidedas they would within the
Figure 4 Distributed Simulation showing inflatable operating theatre.
i50
walls of a real operating theatre, where access is restricted and
authorisation required.
The concept underpinning DS is to separate space from place.
A portable simulation space can be quickly erected in any
available location, using an inflatable ‘igloo’ to provide a circular
enclosure whose ambience provides an ‘impression’ of a clinical
environment (figure 4). This can then be populated by a variety
of scenarios, people and ‘props,’ depending on specific need.
Having established a simulation ‘space,’ the next requirement is
to determine what should take place within it and how this can
be recreated as effectively as possible.
CIRCLES OF FOCUS
Attention is not uniform and unselectivedclinicians see most
clearly what is most important to them, and the rest becomes
blurred. Where this focus is directed will depend on the clinician’s
specialty. An image of concentric ‘circles of focus’ is used to
describe a gradient of perceived realism (figure 5). Applying this
model to an operation, the surgeon’s primary focus of attention is
the operative field. In this central circle, every detail is of interest
and importance. The clinical operating theatre environment is
designed to support this heightened awarenessdthe field is
brightly lit by multiple overhead lights which eliminate shadows;
instruments are placed in the surgeon’s hand by a member of the
team without any need for distracting eye contact; every nuance
of anatomy or drop of blood is sharply defined.
Around this central circle is another circledthe setting within
which the operation is taking place. This too is crucial but takes
place at a lower level of awareness. Here is the context of the
proceduredthe setting where it occurs and the people who take
part in it. Within this second circle, a surgeon’s general sense of
being in an operating theatre is supported by a complex combi-
nation of sights, sounds and sensations: the noises of the
monitor and the buzz of muted conversation; an awareness of
the anaesthetic machine and the team around it; the bright light
overhead; and the sensation of being gowned and gloved. Because
the surgeon is focussing so intently on the primary circle,
however, events and objects in this second circle are less distinct.
This blurring is both physical and metaphorical. Components of
this circle register at a less conscious leveldsome elements
indeed are only noticeable if they are not there.
These two circles are embedded within a thirddthe wider
picture of the clinical scenario that is unfolding, the tapestry of
events from which the operation is constituted. Anaesthetic
decisions are made, drugs are fetched and administered, instru-
ments are requested, and sometimes problems arise and stressors
are introduced. But again, this activity takes place outside the
surgeon’s primary focus.
Processes of care
Practitioner
Patient &
Boundary
procedure
Place & people
Figure 5 Circles of Focus.
Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424

REPRODUCTION, RECREATION AND SELECTIVE ABSTRACTION
If this model of circles of focus has authenticity, then it can form
the basis for a different approach to simulation design. Instead
of simple reproduction of a operating theatre, the process
becomes one of recreation. And at the heart of this lies selective
abstraction, the identification of what are the crucial elements
required for belief in a simulation as a mirror of reality. Posi-
tioning these elements along a realism gradient becomes key to
creating an effective simulation experience.
Paradoxically, however, the component that is of most
interest to the surgeondthe operative fielddis often the least
realistically portrayed within current simulations. Models are
crude and stylised; anatomical variation is absent; and tissues do
not bleed. For the surgeon, therefore, the effort of believing in
the simulationdthe ‘imaginative work’ requireddis unaccept-
ably high. As described above, the introduction of prosthetics
expertise promises radical changes, by creating models with high
levels of perceived authenticity, in both look and feel.
In order to develop the DS framework, we gave a team of
design engineers the task of identifying and recreating key trig-
gers for perceived realism (from the surgeon’s perspective).
During an extended period of observation in actual operating
theatres and in-depth discussions with surgical teams, the
engineers (who had no previous exposure to clinical settings)
selected key components which constitute a surgical setting (eg,
operating lamp; ambient sounds; monitor beep; anaesthetic
machine; equipment trolleys). Preliminary studies with surgeons
confirmed our belief that equipment and activity beyond
the first circle were perceived as real, even when represented by
low-cost models and pictorial representations.
The creation of a convincing environment for simulated care,
although necessary, is not a sufficient condition for authentic
simulation. The next requirement is to provide experiences that
reflect clinical practice, and allow educational goals and
outcomes to be achieved. The construction of simulations
(scenarios) must of course be based in actual clinical experience.
As discussed above, this requires the relationship of care between
patient and clinician to be established. But again, this is
a process of recreation rather than reproductiondalthough in
this case a functional rather than a structural recreation. Here
the process of selective abstraction results in a dynamic
‘performance.’ If successful, the means by which this perfor-
mance is achieved will fall out of conscious awareness, and
participants will experience it ‘as if ’ it were the real thing.27 To
return to my opening metaphor, they are seeing the painting
itself rather than the canvas on which it is painted.
REHEARSAL
If we can create convincing simulated environments that allow
authentic performance to be reproduced, how might we use these
systematically in the service of clinical education? The concept of
systematic rehearsal offers considerable potential, and there are
analogies between surgical teams and ensemble musicians.28
When starting to prepare for a public performance of an
unfamiliar piece, musicians become proficient in their individual
parts through solitary practice. Once they have mastered these
parts, however, the players come together for rehearsal. This
crucial stage allows them to work together in conditions that
approximate those of the eventual performance. Here they reach
a negotiated agreement of how they will perform as an
ensemble, addressing issues of interpretation and integration as
well as planning for possible mishaps on the night and ‘experi-
encing danger safely.’ By the time they come to the performance,
where the piece must be played from beginning to end with no
Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424
Original research
opportunity for backtracking, the ensemble members are playing
as a single unit.
In surgery, by contrast, most time is spent upon performance
(the actual operation). Practice of component skills (in simula-
tion and skills centres) is becoming increasingly evident.
However, formal rehearsal is still conspicuously lacking. Part of
the reason is that facilities for effective rehearsal are seldom
available. By providing access to immersive yet low-cost simu-
lation facilities, DS offers the potential for developing rehearsal
as a key plank of healthcare education, allowing clinical teams to
‘experience danger safely’ and to build up a repertoire of
authentic responses to complex clinical situations which will
stand them and their patients in good stead (figure 6).
CONCLUSION
A simplistic perspective frames simulation as a means of gaining
component skills (usually relating to invasive clinical proce-
dures). A more nuanced view frames it differentlydas a stepping
stone to gaining mastery within a complex clinical world. This
paper suggests how this might be achieved, exploring how
a selective approach might allow high levels of immersion and
perceived realism to be provided within the constraints of
a resource-limited healthcare system. At the heart of this
argument lies the relationship between clinical care and
simulation.
Simulation should allow participants to behave not only as
technicians but also as rounded clinicians, recreating the condi-
tions of practice without imposing unnatural constraints. To
achieve this, the artifice through which simulation’s illusions are
achieved should fade into invisibility, leaving participants free to
function authentically. To return to the metaphor of a painting,
simulation should allow clinicians to see the picture, not just the
canvas and the paint.
Considered in this way, simulation becomes not just an
activity but a way of seeing, which can change how we
perceive our clinical practice. Merleau-Ponty describes a parallel
experience when looking at a painting:
I would be at great pains to say where is the painting I am looking
at. For I do not look at it as I do at a thing: I do not fix it in its
place. It is more accurate to say that I see according to it, or with it,
than that I see it [my italics].29
SKILLS CENTRE SIMULATED OR REAL OR
Practice Rehearsal Performance
Figure 6 Three stage model for procedural training.
i51
 Original research

Used imaginatively, simulation can serve a similar function, 11. Gaba D, Howard S, Fish K, et al. Simulation-based training in anaesthesia crisis
allowing us to see clinical practice according to what takes place resource management (ACRM): a decade of experience. Simul Gaming
2001;32:175e93.
in a simulated setting and vice versa. By using simulation as 12. Gaba DM. Anaesthesiology as a model for patient safety in health care. BMJ
a lens for shaping our perception of clinical practice, rather than 2000;320:785e8.
as an end in itself, we will ensure that caring for patients 13. Wiggins G. Assessing Student Performance Exploring the Purpose and Limits of
Testing. San Francisco: Jossey-Bass, 1993.
remains our primary concern. 14. Sennett R. The Craftsman. London: Allen Lane, 2008.
Acknowledgements I wish to acknowledge the invaluable contributions of 15. Pye D. The Nature and Art of Workmanship. Cambridge: Cambridge University Press,
numerous colleagues at Imperial College London, in particular, the contributions of 1968.
F Bello, D Nestel, N Sevdalis, E Kassab, D King, S Arora, J Kyaw-Tun, A Cope, 16. Kneebone R, Nestel D, Yadollahi F, et al. Assessing procedural skills in context:
J Bezemer, M Campbell, C Plumbe, M Harrison and A Granados. exploring the feasibility of an Integrated Procedural Performance Instrument (IPPI).
Med Educ 2006;40:1105e14.
Funding The work on Distributed Simulation referred to in this paper was funded by 17. Kneebone R, Nestel D, Wetzel C, et al. The human face of simulation:
the London Deanery Simulation and Technology Enhanced Learning Initiative (STeLI). patient-focused simulation training. Acad Med 2006;81:919e24.
18. Kneebone R, Kidd J, Nestel D, et al. Blurring the boundaries: scenario-based
Competing interests None. simulation in a clinical setting. Med Educ 2005;39:580e7.
Provenance and peer review Not commissioned; externally peer reviewed. 19. Kneebone RL. Clinical simulation for learning procedural skills: a theory-based
approach. Acad Med 2005;80:549e53.
20. Kneebone RL, Scott W, Darzi A, et al. Simulation and clinical practice: strengthening
REFERENCES the relationship. Med Educ 2004;38:1095e102.
1. Chikwe J, de Souza AC, Pepper JR. No time to train the surgeons. BMJ 21. Kneebone R, Kidd J, Nestel D, et al. An innovative model for teaching and learning
2004;328:418e19. clinical procedures. Med Educ 2002;36:628e34.
2. Bell RH Jr. Keeping an open (surgery) mind. Ann Surg 2010;251:213e14. 22. Rall M, Stricker E, Reddersen S, et al. Mobile ‘in situ’ simulation crisis resource
3. Henriksen K, Moss F. From the runway to the airway and beyond. Qual Saf Health management training. In: Kyle R, Murray W, eds. Clinical simulation: operations,
Care 2004;13(Suppl 1):i1. engineering and management. Elsevier, 2008:565e81.
4. Henriksen K, Patterson M. Simulation in health care: setting realistic expectations. 23. LeBlanc D. Situated simulation: taking simulation to the clinicians. In: Kyle R, Murray
J Patient Saf 2007;3:127e34. W, eds. Clinical Simulation: Operations, Engineering and Management. Elsevier:
5. CMO. 150 Years of the Annual Report of the Chief Medical Officer: on the State of 2008:553e57.
Public Health. 2008. 2009. 24. Kneebone R, Arora S, King D, et al. Distributed simulationdAccessible immersive
6. Amalberti R, Vincent C, Auroy Y, et al. Violations and migrations in health care: training. Med Teach 2010;32:65e70.
a framework for understanding and management. Qual Saf Health Care 25. Balme C. The Cambridge introduction to theatre studies. Cambridge: Cambridge
2006;15(Suppl 1):166e71. University Press, 2008.
7. Kneebone RL, Nestel D, Vincent C, et al. Complexity, risk and simulation in learning 26. McAuley G. Space in performance making meaning in the theatre. Ann Arbor:
procedural skills. Med Educ 2007;41:808e14. University of Michigan Press, 2000.
8. Kneebone R. Perspective: simulation and transformational change: the paradox of 27. Dieckmann P, Gaba D, Rall M. Deepening the theoretical foundations of patient
expertise. Acad Med 2009;84:954e7. simulation as social practice. Simul Healthc 2007;2:183e193.
9. Gaba DM. The future vision of simulation in health care. Qual Saf Health Care 28. Kneebone RL. Practice, rehearsal, and performance: an approach for
2004;13(Suppl 1):2e10. simulation-based surgical and procedure training. JAMA 2009;302:1336e8.
10. Rall M, Gaba D. Patient simulators. In: Miller R, ed. Miller’s Anesthesia 29. Merleau-Ponty M. Eye and mind. In: Edie J, ed. The Primacy of Perception.
2004:3073e103. Chicago: Northwestern University Press, 1964:159e90.

i52 Qual Saf Health Care 2010;19(Suppl 3):i47ei52. doi:10.1136/qshc.2010.042424


1
Department of Health Policy &
Management and Associate
Dean, School of Public Health,
University of Minnesota,
Minneapolis, Minnesota, USA
2
Teamwork and Simulation,
Fairview Health Services,
Minneapolis, Minnesota, USA
3
Clinical Safety, Fairview Health
Services, Minneapolis,
Minnesota, USA
4
School of Nursing Emeritus,
University of Minnesota,
Minneapolis, Minnesota, USA
5
Department of Urologic
Surgery, University of
Minnesota, Minneapolis,
Minnesota, USA
Correspondence to
Dr William Riley, School of
Public Health, University of
Minnesota, 420 Delaware
Street SE, Mayo Mail Code 197,
Minneapolis, MN 55455-0381,
USA
Accepted 5 May 2010
Published Online First
19 August 2010
Qual Saf Health Care 2010;19(Suppl 3):i53ei56. doi:10.1136/qshc.2010.040311
Detecting breaches in defensive barriers using in situ
simulation for obstetric emergencies
William Riley,1 Stan Davis,2 Kristi M Miller,3 Helen Hansen,4 Robert M Sweet5
ABSTRACT
Background In Reason’s safety model, high-reliability
healthcare organisations are characterised by multiple
layers of defensive barriers in depth associated with
increased levels of safety in the care delivery system.
However, there is very little empirical evidence
describing and defining defensive barriers in healthcare
settings or systematic analysis documenting the nature
of breaches in these barriers. This study uses in situ
simulation to identify defensive barriers and classify the
nature of active and latent breaches in these barriers.
Methods An in situ simulation methodology was used to
study team performance during obstetrics emergencies.
The authors conducted 46 trials of in situ simulated
obstetrics emergencies in two phases at six different
hospitals involving 823 physicians, nurses and support
staff from January 2006 to February 2008. These six
hospitals included a university teaching hospital, two
suburban community hospitals and three rural hospitals.
The authors created a high-fidelity simulation by
developing scenarios based on actual sentinel events.
Results A total of 965 breaches were identified by
participants in 46 simulation trials. Of the 965 breaches,
461 (47.8%) were classified as latent conditions, and
494 (51.2%) were classified as active failures.
Conclusions In Reason’s model, all sentinel events
involve a breached protective layer. Understanding how
protective layers breakdown is the first step to ensure
patient safety and establish a high reliability. These
findings suggest where to invest resources to help
achieve a high reliability. In situ simulation helps
recognise and remedy both active failures and latent
conditions before they combine to cause bad outcomes.
INTRODUCTION
Landmark reports from the Institute of Medicine1 2
have led to an unprecedented interest in under-
standing the nature of error in medical care
organisations and in transforming hospitals into
high-reliability organisations (HROs) that provide
safe care and minimise errors.3 Reason’s accident
model has become one of the dominant paradigms
for analysing medical errors and patient safety
incidents.4
High-reliability healthcare organisations are
characterised by multiple layers of defensive
barriers associated with increased levels of safety
in the care delivery system.5 However, there is
very little empirical evidence describing and
defining defensive barriers in healthcare settings or
systematic analysis documenting the nature of
breaches in these barriers. Simulation training has
emerged as an important strategy for improving
patient safety and quality in healthcare.6 This
article describes a study using in situ simulation to
Original research
identify defensive barriers and classify the nature
of active and latent breaches in these barriers.
In situ simulation is a team-based simulation
strategy that occurs in patient care units involving
interdisciplinary team members working within
their own environments. In contrast to simulation
laboratories, in situ simulation methods can be
useful in uncovering latent conditions in a hospital
setting that would not be detectible in a labora-
tory setting.7 In situ simulation aims to achieve
high-fidelity realism by performing the training in
the setting where patient care is delivered, where
organisational processes exist and real errors
occur.8 Fidelity is the clinical and psychological
degree to which the simulator or simulation
scenario represents the real system.6
METHODS AND DESIGN
Methods
An in situ simulation methodology was used to
study team performance during obstetrics emer-
gencies.7 We created a high-fidelity simulation by
developing scenarios based on actual sentinel
events. All simulations were video-recorded.
Study design: scenarios and event sets
A total of three sentinel event clinical scenarios were
developed based on medical record review, postevent
investigation and interviews with selected persons
involved in the actual event. These were placental
abruption, ruptured uterus and postpartum
haemorrhage. The scenarios were segmented into
scenes or ‘event-sets,’9 a method used to stress the
care delivery team and to interrogate the system for
both active failures and latent conditions. The event
set methodology was developed in aviation in the
mid-1990s and was introduced to healthcare
a decade later.9 Four to six event sets are embedded
in a scenario and occur in chronological sequence.
Each event set has a specific trigger (sudden clinical
change) and distracter (element designed to divert
the team’s attention). The scenario and event sets
were designed to prompt specific observable behav-
ioural markers including situational awareness,
standardised communication techniques of situa-
tion, background, assessment, recommendation,
response (SBAR-R) or closed loop communication,
shared mental model and leadership. The beginning
of each event set is distinguished by critical junc-
tures (especially a change in team composition)
characterised by a greater chance for a breach in
communication or teamwork behaviour.7 8 The
average time per simulation was 3 h, with the three
components averaging 15 min for the briefing,
45 min for the simulation event and 2 h for the
facilitated debriefing.
i53
 Original research

Research setting
We conducted 46 trials of in situ simulated obstetrics emergencies
in two phases at six different hospitals within one healthcare
system serving a comparable rural/suburban population in the
Midwest. The participants were 823 obstetric, anaesthesia,
family practice, neonatal and paediatric physicians, nurses and
support staff from January 2006 to February 2008. These six
hospitals included a university tertiary hospital, two level II
suburban community hospitals and three rural primary care
hospitals. Together, the six hospitals represent about 11 000
deliveries per year. IRB approval was secured from the University
of Minnesota to conduct this study.
In situ simulation
We used in situ simulation and two directional feedback to
assess the performance of healthcare systems and individual
team members in the patient care setting at the microsystem
level.10 Each simulation session for perinatal critical events
consisted of four components: (1) a briefing for participants
and staff in the care unit immediately prior to the simulation;
(2) a simulation scenario event that utilised confederates for
a live patient (standardised patient) accompanied by a family
member and later a manikin for physiological fidelity; (3) an
expert-facilitated, structured debriefing designed to identify
latent process issues and active human factors failures self-
described by the participants; (4) follow-up to correct systems
issues identified in the simulation trial. In each simulation, the
patient was followed from triage through labour and the oper-
ating room. The simulation began with a handoff consisting of
a brief history from one provider to another. The simulation
continued with clinical conditions (triggers) transmitted to the
receiving team by confederates, a manikin or the participating
physician being fed information through a remote headset.
Video cameras captured all interactions of the surgical and
paediatric teams. Immediately following the simulation
scenario, a debriefing session was held. The participants
discussed what went well, what did not go well and why, and
what could have been better. The debriefing closed the gap
between the experience and the making sense of it, and
allowed for self discovery for individuals to express their own
experiential learning. Follow-up from the lessons learnt about
communication, teamwork, safety and any changes made to the
unit were communicated as a key outcome of in situ simulation.
Data collection and analysis
All data were collected during the facilitated debriefing stage of
the simulation with participants deconstructing their own
experiences using the grounded theory approach, a qualitative
research methodology that emphasises the iterative nature of
discovery, especially in the study of human performance.
Grounded theory is based on a textual database (such as field
notes or video recording) to identify construct categories and
their interrelationship 11 with a focus on discovering themes
that emerge from this research rather than hypothesis
testing.12 Selected segments of the video were used by the
facilitators during the debriefing to clarify certain behaviours or
to assist when recollections by participants were vague or
uncertain.
The debriefing team consisted of two facilitators and one note
taker. The note taker captured the real-time comments from
participants. A breach was enumerated when a participant
identified a team behaviour necessary for safety that was not
performed or performed poorly, or when a system process did
not perform to the fullest extent and thus could contribute to an
injury. Each element contributed to the qualitative database of
breaches and was subsequently analysed and categorised as
either (a) an active failure or (b) a latent condition as described
by Reason’s organisational accident model.13 14 We classified an
active failure as an unsafe act committed by those at the
patient/provider interface where the damaging consequences are
immediately apparent, and a latent condition as something
created as a result of decisions at higher echelons whose
damaging consequences may lie dormant for a long time until
triggered by local conditions. A defensive barrier is a system
elementdfacility, equipment, policy, procedure, human behav-
iour or other that is either intentionally designed or uninten-
tionally evolves over time. In either circumstance, the defensive
behaviour is a metaphor that serves to prevent hazard from
contributing to injurydwhich serves to prevent system errors
from causing injurydto interdict the ‘accident trajectory.’ A
breach in a defensive barrier occurs when no defensive barrier
existed or when the existing defensive barrier did not work as

Table 1 Categories of active and latent breaches in safety10 15

Breach Definition Behavioural example

Active failure
Situational awareness
Standardised communication
< Closed loop communication
< Situation, background, assessment,
recommendation response
Shared mental model
Latent condition
Policy or protocol
Equipment/environment
Process issue
Failure to have conscious, mindful observation of one’s own
environment or recognition of patient condition
Failure to communicate to a specific person that is
acknowledged by the receiver and then affirmed by the sender.
(eg, Verb order reed back) Failure to use technique of
communication about a critical situation that involves clear
specification of SituationeBackgroundeAssessmente
RecommendationeResponse
Failure of a team to articulate common understanding of the
problem and/or the plan. Everyone is not ‘in the same movie’
Policydprocedure not followed (lack of role definition,
knowledge, skills or training)
Technical, equipment or environment failure or not available
System process failure (interdepartment or unit services/
support/communication)
Obstetrician became task saturated upon entering room, did
not ‘hear’ correct data from nurse, distracted by family
questions
Physician called lab orders ‘in the air,’ nurse did not write them
down or acknowledge and forgot them by time she got to the
phone
Anaesthesiologist entered operating room and neither
physician nor nurse told him what was going on with the
urgency of the patient condition
Pharmacist had difficulty in finding the epinephrine during infant
resuscitation, but did not share his concern with the neonatal
team
No patient identification band was placed on patient
People showing up for infant code did not know their roles
No infant isolette or stethoscope in operating room for
resuscitation
Very loud in the operating room
Lab did not know where to go when they heard the code, when
they arrived in operating room no orders and no paperwork to
obtain the emergency blood

i54 Qual Saf Health Care 2010;19(Suppl 3):i53ei56. doi:10.1136/qshc.2010.040311

 Original research

Table 2 Raw numbers and proportion of breaches by hospital and per simulation: latent conditions and active breaches
Hospital 1 Hospital 2 Hospital 3 Hospital 4 Hospital 5 Hospital 6 All hospitals
Latent conditions N 101 58 60 69 40 133 461
Mean/trial 10.1 7.3 10.0 17.3 10.0 9.5 10.0
Percentage of hospitals 53.0 48.0 46.0 66.0 45.0 43.2 47.8
95% CI (47.5 to 61.6) (33.6 to 54.3) (35.4 to 56.2) (56.5 to 76.2) (35.1 to 55.9) (32.8 to 53.5) (37.8 to 58.7)
Active breaches Facility 91 74 71 35 48 175 494
Mean/trial 9.1 9.3 11.8 8.8 12.0 12.5 10.7
Percentage 47.0 52.0 54.0 34.0 55.0 56.8 51.2
95% CI (37.0 to 57.8) (45.7 to 66.4) (43.8 to 64.6) (23.8 to 43.5) (44.1 to 64.9) (46.5 to 67.2) (41.3 to 62.2)
Total n 192 142 131 104 88 308 965
Mean/trial 19.2 17.8 21.8 26.0 22.0 22.0 20.8
Percentage of hospitals 100.0 100.0 100.0 100.0 100.0 100.0 100.0
Total simulations 10 8 6 4 4 14 46

intended by the system design. A breach is judged to have the
potential to allow the progress of accident trajectory towards
patient injury. Six separate categories of breaches are listed and
defined in table 1.
RESULTS
Table 2 shows the results of trials in six hospitals with two types
of breaches: latent conditions and active failures. There were
a total of 965 breaches identified by participants over 46 simula-
tion trials, ranging from a low of four trials (Hospitals 4 and 5) to
a high of 14 trials (Hospital 6). There was an average of 20.8
breaches per trial, ranging from a low of 17.8 breaches (Hospital 2)
to a high of 26 breaches (Hospital 4). Of the 965 breaches,
461 (47.8%) were classified as latent conditions, and 494 (51.2%)
were classified as active failures.
We calculated the proportion of active breaches and latent
conditions for each facility, as well as for the combined hospital
condition. Examination of the 95% CI for the proportion of
active failures indicates that while five of the six hospitals
showed no significant difference in the proportion of active
breaches and latent conditions, one hospital (Hospital 4) had
significantly more latent conditions than active breaches
(95% CI for active failures is 56.5e76.2%).
Table 3 shows the percentage of breaches for three categories
of latent conditions: (a) failure to comply with policy and
procedure (such as no hard stop prior to incision), (b) breaches
caused by equipment or the physical environment (such as two
operating rooms in different locations of the hospital with the
same numbering system) and (c) breaches caused by estab-
lished system processes (such as inadequate blood ordering
protocols during emergencies). The three latent condition
categories of policy, equipment, and system processes account
for 23.2%, 9.3% and 16.3%, respectively, of total breaches. The
three categories of active failures are: (a) loss of situational
awareness (such as failing to note non-reassuring fetal heart
sounds), (b) failure to establish a shared mental model (not
communicating a decision to the care team) and (c) commu-
nication breakdown (not performing a closed-loop communi-
cation) respectively of the total breaches associated with active
failures in the six hospitals. The three active failure categories
of situational awareness, shared mental model, and commu-
nication breakdown accounted for 13.0%, 16.0% and 22.3%,
respectively, of total breaches.
DISCUSSION
This study shows a high number of participant reported
breaches in defensive barriers to protect patients from injury
during simulated obstetrics emergencies conducted in six sepa-
rate hospitals. A total of 955 breaches were reported by 823
persons from seven professions and occupations who partici-
pated in 46 separate healthcare teams with an average of 20.8
breaches identified by the participants during the debriefings.
The active failures and latent conditions occurred with compa-
rable frequency (51.7% and 48.3% respectively) with no signifi-
cant difference between these two categories. Two components
of the six categories accounted for approximately 45% of the
breaches: (1) failure to comply with organisational policies (a
latent condition), which implies inadequate training, orientation
or disregard of known procedure under the circumstances, and
(2) ineffective communication among team members (an active
failure), which implies mistakes or errors of omission and
commission by the individual. In order to make these findings
actionable, additional follow-up research is needed to determine
why policies and procedures were not followed. For example,
they may be outdated, irrelevant or in need of change.
Differences between hospitals are noteworthy. The average
number of breaches reported for all hospitals is 20.8 (ranging
from a low of 17.8 to a high of 26). This compact range clustered
around the overall average. It is not clear whether differences in
the number of breaches reported between hospitals are related to

Table 3 Relative proportion of breaches by category by hospital

Category Hospital 1 Hospital 2 Hospital 3 Hospital 4 Hospital 5 Hospital 6 All Hospitals
Latent conditions
Policy/procedure (%) 29.2 23.2 26.0 34.6 25.0 14.0 23.2
Equipment (%) 10.4 2.8 10.7 15.4 11.4 8.4 9.3
System processes (%) 13.0 21.8 9.2 16.3 9.1 20.8 16.3
Active failures
Situational awareness (%) 5.2 20.4 9.2 9.6 15.9 16.2 13.0
Shared mental model (%) 8.3 13.4 27.5 10.6 14.8 19.2 16.0
Communication (%) 33.9 18.3 17.6 13.5 23.9 21.4 22.3
Total (%) 100.0 100.0 100.0 100.0 100.0 100.0 100.0

Qual Saf Health Care 2010;19(Suppl 3):i53ei56. doi:10.1136/qshc.2010.040311 i55

 Original research
hospital structural features, team dynamics or both influences.
Moreover, only Hospital 4 had significant differences between
the proportion of latent conditions (66%) and active failures
(44%). The constant proportion of latent conditions and active
failures in five hospitals underscores the importance of both
factors, as predicted by Reason’s model. The larger proportion of
latent conditions in hospital four needs further exploration.
In Reason’s model, all sentinel events involve a breached
protective layer. Understanding how protective layers break-
down is the first step to ensure patient safety and establish high
reliability.13 Defences in depth have successive layers of protec-
tion, one behind the other, each guarding against the possible
breakdown of the one in front. The holes in Reason’s Swiss
cheese model represent breaches or weaknesses in the defensive
barriers. The findings from this study can be used to give
meaning to the type and context of the breaches in defensive
barriers described by Reason. There have been no systematic
investigations of defensive barriers in depth in healthcare
settings, and we advance understanding of Reason’s accident
model in healthcare settings by: (1) identifying the defensive
barriers in place, (2) exposing the weaknesses in the defensive
layers and (3) identifying where error proofing and safeguarding
may be most needed.
Implications
These findings suggest two areas in which to invest resources to
help achieve high reliability: interdisciplinary teamwork and
removing latent conditions. In situ simulation helps recognise
and remedy both active and latent conditions before they
combine to cause bad outcomes. It is their combined lack of
recognition and discoverability that make it important for
healthcare organisations that want to approach high reliability.
While building strong defensive barriers is considered the best
protection against patient injury,4 there has been very little
empirical evidence or systematic analysis about how to identify
defensive barriers, the nature and location of defensive barriers
as well as the breaches. We selected voluntary reporting of active
failures and latent conditions by participants in order to inter-
rogate the care delivery process. Many of the breaches are not
observable unless identified by the participants, and the in situ
simulation method facilitates the study of breaches as experi-
enced by the clinician.
The findings from this study can be used to understand better
how injury occurs and develop more rigorous safety design. The
findings support Reason’s model that neither errors nor breaches
in defensive barriers necessarily work in a linear two-dimensional,
progressive, accident trajectory. Our study underscores the
dynamic and unpredictable interplay of multiple dimensions of
errors and breaches. While Reason’s model is highly relevant for
healthcare and has been widely adopted, there has been little
evidence to identify the frequency and nature of breaches in
defensive barriers. Identifying and reducing breaches in defensive
i56
barriers requires interdisciplinary team training to create high-
performing care teams16 17 and of well-designed care processes18
to achieve highly reliable care in healthcare settings.6 Our findings
provide further understanding of the defensive barriers in depth
needed for HROs and their breaches with respect to system
design and team performance. The in situ simulation method-
ology allows organisations to identify breaches as well as classify
communication and teamwork skills to help prevent breaches
that may lead to injury.
Acknowledgements We would like to acknowledge P Ranallo and M McCullough
for their assistance in data analysis and preparation of the manuscript.
Funding Agency for Healthcare Research and Quality, Fairview Health Services, and
the University of Minnesota Academic Health Center.
Competing interests None.
Ethics approval Ethics approval was provided by the University of Minnesota
Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Institute of Medicine. To err is human: building a safer health system. Washington,
DC: National Academy Press, 1999.
2. Institute of Medicine. Crossing the quality chasm: a new health System for the 21st
Century. Washington, DC: National Academy Press, 2001.
3. Gauthier AK, Davis K, Schoenbaum SC. Achieving a High Performance Health Care
System: High Reliability Organizations within a Broader Agenda. Health Serv Res
2006;41:1710e20.
4. Perneger TV. The Swiss cheese model of safety incidents: Are there holes in the
metaphor? BMC Health Serv Res 2005;5:71e7.
5. Institute of Medicine. Keeping patients safe: transforming the work environment of
nurses. Washington, DC: National Academies Press, 2004.
6. Gaba DM, Howard SK, Fish KJ, et al. Simulation-based training in anesthesia
crisis resource management: A decade of experience. Simulation & Gaming
2001;32:175.
7. Riley W, Hansen HE, Gürses AP, et al. The nature, characteristics and patterns of
perinatal critical events teams. In: Advances in Patient Safety: New Directions and
Alternative Approaches, Vol. 3. Rockville MD: Agency for Healthcare Research and
Quality, 2008:131e44.
8. Miller K, Riley W, Davis S, et al. In situ simulation: a method of experiential learning
to promote safety and team behaviour. J Perinat Neonatal Nurs 2008;22:105e13.
9. Hamman WR. The complexity of team training: What we have learned from aviation
and its applications to medicine. Qual Saf Health Care 2004;13:i72e9.
10. Fanning RM, Gaba DM. The Role of Debriefing in Simulation-Based Learning. Simul
Healthc 2007;2:115e25.
11. Moustakas C. Phenomenological Research Methods. Thousand Oaks, CA: Sage
Publishing, 1994.
12. Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and
techniques. Newbury Park, CA: Sage Publishing, 1990.
13. Reason J. Managing the risks of organizational accidents. 9th edn. Aldershot, UK:
Ashgate Publishing Ltd, 2005.
14. Reason J. Human error: models and management. BMJ 2000;320:768e70.
15. Agency for Healthcare Research and Quality. TeamSTEPPS: National
Implementation. http://teamstepps.ahrq.gov/index.htm (accessed 2 Oct 2007).
16. Wilson KA, Burke CS, Priest HA, et al. Promoting health care safety through training
high reliability teams. Qual Saf Health Care 2005;14:303e29.
17. Resar R, Pronovost P, Haraden C, et al. Using a bundle approach to improve
ventilator care process and reduce ventilator associated pneumonia. Jt Comm J Qual
Patient Saf 2005;31:243e8.
18. Berwick D, Leape L. Forward. In: Leonard M, Frankel A, Simmonds T, et al, eds.
Achieving safe and reliable healthcare. Chicago, IL: Health Administration Press, 2004.
Qual Saf Health Care 2010;19(Suppl 3):i53ei56. doi:10.1136/qshc.2010.040311

1
Keenan Research Centre of the
Li Ka Shing Knowledge Institute,
St Michael’s, Toronto, Canada
2
Ryerson University, Toronto,
Canada
3
University of Washington,
Washington, USA
4
University of Toronto, Toronto,
Canada
5
University of Western Ontario,
Centre for Education Research &
Innovation, London, Canada
Correspondence to
Miss Lianne P Jeffs, St
Michael’s Hospital, Room 5-060
Bond Wing, 30 Bond Street,
Toronto M5B 1W8, Canada;
jeffsl@smh.toronto.on.ca
Accepted 18 January 2010
Qual Saf Health Care 2010;19(Suppl 3):i57ei60. doi:10.1136/qshc.2009.036418
A new way of relating: perceptions associated with
a team-based error disclosure simulation intervention
Lianne Jeffs,1 Sherry Espin,2 Sarah E Shannon,3 Wendy Levinson,1,4
Melanie Kazman Kohn,1 Lorelei Lingard5
ABSTRACT
Background Despite the call for open and team-based
approaches to error disclosure, the participation beyond
physicians and managers is not a common practice in
health care settings. Moreover, within the growing
literature base on error disclosure, team-based error
disclosure is an emerging concept. To address this
knowledge gap, a study was undertaken to explore the
perceptions associated with an educational simulation
intervention for team-based error disclosure.
Methods A qualitative study that involved analysis of
data obtained from semi-structured interviews with
a sample of 6 physicians, 6 surgeons, and 12 nurses
recruited from the three participating hospitals.
Results Perceptions from study participants elucidated
a tension between team-based error disclosure as an
unrealistic, forced practice and as a realistic, beneficial
practice. This tension was highly contextual and
differentiated by study participants’ perceptions of the
nature of the error; patient’s preferences; and prevailing
cultural and professional norms. Regardless of the view,
study participants described the simulation experience
as a new way of relating that departed from existing
practice.
Conclusions Study findings revealed that a team-based
approach to disclosure is not realistic or necessary for all
error situations, such as when the error involves a single
discipline. However, when the error involves a variety of
health care professionals interacting with the patient,
a team-based approach is beneficial to them and the
patient. Further work is required by researchers and
administrators to develop and test out interventions that
enable health care professionals to practice team-based
error disclosure in a safe and supported environment.
Despite multilevel efforts, open disclosure of
healthcare errors continues to be the exception
rather than the rule where only approximately 30%
of the errors that cause harm are disclosed to
patients.1e4 Moreover, how an error is disclosed
varies among physicians, including the information
they would report to the patient, how the error is
described, whether and how an apology is offered,
acknowledgement of patients emotions and discus-
sion of how errors are to be prevented.4e7 Further
physicians reported that they chose to disclose less
information to the patient for less apparent errors
than respondents provided with more apparent
errors.4 Disclosure practices have traditionally been
conceptualised as a physicianepatient dyad, yet the
delivery of healthcare is a team function. As such,
other healthcare providers may have played a role in
the error and thus should be involved in communi-
cating the error to the patient.8 In a study utilising
Original research
focus groups, nurses reported wanting to be
involved in the error disclosure process.8 These
nurses also reported fearing being blamed for the
error to the patient if they were not present during
the disclosure. However, having nurses or other
healthcare professionals participate in an error
disclosure process is not common practice.8 Some
scholars posit that the lack of participation of
healthcare professionals may be a result of lack of
disclosure training that renders them unprepared for
engaging in difficult conversations with each other
and with patients.4e8 Although there is a growing
literature on error disclosure, team-based error
disclosure is an emerging concept. To address this
knowledge gap, we explored the perceptions of
nurses and physicians participating in a team-based
error disclosure simulation intervention.
METHODS
We conducted a qualitative study to explore study
participants’ perceptions of team-based error
disclosure as part of a simulation intervention. The
study received ethics approval from the Research
Ethics Boards of the participating institutions and at
an academic institution. The simulation interven-
tion consisted of a nurse, a physician or surgeon, and
a standardised team member being given two sce-
narios. See table 1 for the four scenario descriptions.
Each team was asked to discuss the first scenario
with each other and then discuss with a standardised
patient, including potentially disclosing the error. A
disclosure coach then provided supportive feedback
on their communication. The disclosure coaches
(n¼3) were nurses who functioned either as a risk
manager or director from each of the Quality and
Risk Departments of participating hospital sites.
Prior to participating in the simulation intervention,
the principal investigator and research coordinator
met with the disclosure coaches and reviewed the
standardised disclosure checklist developed by the
Canadian Patient Safety Institute.9 The team was
provided with the second scenario and repeated the
experience. Box 1 provides an overview of the
simulation intervention. Completion of Steps 1e4
ranged between 45 min and 1 h in length.
Following the simulation experience, participants
were interviewed separately using a semistructured
approach with open-ended questions. See box 2 for the
interview protocol. The range of the interviews was
20e30 min. All interviews were tape-recorded and
transcribed. Data were analysed using a content anal-
ysis approach that identified codes and categories
leading to emergent themes through saturation.10 Four
researchers independently reviewed the transcripts
line by line to identify codes and then met to determine
i57
 Original research
Table 1 Simulated scenario description
Clinical specialty Nature of error Outcome
Surgical case 1 Lost specimen Psychological distressdunknown diagnosis
Surgical case 2 Retained sponge Infection and second operation
Medicine case 1 Insulin overdose Diabetic shock
Medicine case 2 Lovenox overdose Lower GI bleed
categories through consensus. Threats to the applicability and
transferability of qualitative research were minimised through
attention to rigorous verification strategies integrated into the
enquiry. A final coding schema was developed that linked conceptual
codes and categories.10 11 As a verification strategy, one of the
investigators reviewed all the transcripts using the coding schema.
The narrative dataset analysed in this study drew from
participants who worked at the three healthcare organisations (a
550-bed and a 521-bed teaching hospital and a 365-bed
community hospital). Specifically, a total of 24 healthcare
workers engaged in the simulation experience including six
surgeons (four men, two women), six general internists (five
men, one woman) and 12 registered nurses (all women) were
interviewed. The physicians and nurses were drawn from
a clinical setting in which they might have worked together as
a team, such as pairing internal medicine physicians and nurses
or surgeons with perioperative nurses. Table 2 provides an
overview of the sample characteristics.
RESULTS
Participating nurses and physicians described team-based error
disclosure as a tension between an unrealistic and forced prac-
tice, and a realistic and beneficial practice. This tension was
differentiated by these study participants’ perceptions of: (1) the
nature of the error; (2) patient’s preferences in the disclosure
Box 1 Team-based error disclosure simulation process
Step 1: Team discussion of nature of the error. Each team
member was given a specialty-specific description of the event.
The research project coordinator summarised the facts of the
case, and then each team member shared their perspectives.
Step 2: Team planning of disclosure conversation. The second
step involved the team (surgeon or general internal medical
doctor; nurse; and standardised healthcare professional) partici-
pating in a discussion on whether and how to disclose the event.
Specific details included how much information to provide the
patient, what to disclose about why the error happened, how
recurrences could be prevented and whether to apologise. The
team also considered what role each would play in the disclo-
sure, what questions the patient would likely ask and subsequent
responses to these questions.
Step 3: Disclosure conversation. The disclosure conversation
took place with the Standardised Patient (SP) who presented
with a variety of disclosure challenges to the team (eg, trying to
affix blame, reacting to the disclosure with anger, disbelief, and
sarcasm).
Step 4: Team debriefing with disclosure coach. Following the
disclosure of the first case to the SP, the team debriefed with
their disclosure coach, who used a standardised checklist
adapted to guide discussion on what went well and what they
might have done differently. After this debriefing, Steps 2e4
were repeated with another clinical case scenario.
i58
Impact of error
No specimen to send to pathology and need for another biopsy to
confirm diagnosis
Patient length of stay extended due to infection and need to schedule
second operation to retrieve sponges
Patient admitted to ICU overnight to stabilise her blood glucose
Patient length of stay extended to manage lower GI bleed
process; and (3) cultural and professional norms. Regardless of
their view, the majority of study participants described team-
based error disclosure as a new way of relating that departed
from existing practice.
Nature of the error
Whether or not team-based error disclosure was viewed as
realistic was influenced by the study participants’ perception of
the nature of the error. The nature of the error was characterised
by the perceived magnitude or seriousness of the error and its
consequences, and by whom on the team was involved in the
error. On one hand, team-based error disclosure was perceived as
unrealistic, forced practice when the error was considered to be
small with no potential for adverse outcome. Specifically,
physicians expressed that disclosing ‘every little error that
happens’ would occur daily and was seen as impractical. On the
contrary, team-based error disclosure was perceived as a reason-
able, beneficial practice and worthwhile when the magnitude of
the error was considered a ‘serious medical error’ or ‘a significant
error.’ As one medical doctor noted: ‘But you know when it’s
a high stakes situation where somebody really needs an expla-
nation you need to find time for everybody to be present. I think
that it was worthwhile to do it that way.’
Participants also identified that when the error involved
a single discipline, it may not be necessary to involve other team
Box 2 Interview protocol
Prescenario discussion
1. What did you think of the process of sitting down as
a team and talking about the event?
2. Have you ever had discussions like this in your practice?
3. What are the advantages of this kind of team discussion?
4. What are some of the disadvantages of this kind of team
discussion?
5. From your perspective, how did you participate?
6. From your perspective, why did you participate that way?
7. Do you think this kind of team discussion could become
part of standard practice when errors occur? Why or why
not?
8. Did the discussion make you feel prepared?
Disclosure discussion
1. How do you think the team-based disclosure went?
Strengths/positiveseWeakness/barriers or challenges
(eg, did you follow the plan or not?)/negatives
2. What was your role during the disclosure to the patient?
3. Did you feel comfortable with your participation/role?
4. How do you think the team disclosure was for the patient
in each of the two scenarios?
5. Overall, how does your experience today impact the way
you will approach error disclosure in the future?
6. What were your impressions regarding the scenarios?
Were they realistic? Could they be improved?
Qual Saf Health Care 2010;19(Suppl 3):i57ei60. doi:10.1136/qshc.2009.036418
 Original research

Table 2 Sample characteristics

Hospital description Cohort Sex Position
Community healthcare centre Medicine team 1 Male Surgeon
General internist
Nursing team 1 Female Circulating nurse
Registered nursedgeneral internal
medical unit
Medicine team 2 Female Surgeon
Male General internist
Nursing team 2 Female Circulating nurse
Female Registered nursedgeneral internal
medical unit
Academic healthcare centre 1 Medicine team 1 Male Surgeon
General internist
Nursing team 1 Female Circulating nurse
Registered nursedgeneral internal
medical unit
Medicine team 2 Male Surgeon
General internist
Nursing team 2 Female Circulating nurse
Registered nursedgeneral internal
medical unit
Academic healthcare centre 2 Medicine team 1 Male Surgeon
Female General internist
Nursing team 1 Female Circulating nurse
Registered nursedgeneral internal
medical unit
Medicine team 2 Female Surgeon
Male General internist
Nursing team 2 Female Circulating nurse
Registered nursedgeneral internal
medical unit

members. On the other hand, a team-based approach to error
disclosure was supported when the nature of the error was
‘multifactorial’ or viewed as a ‘team event.’ Specifically, many
participants emphasised that ‘whoever was involved in the case
should be in the disclosure process.’ The value of involving those
who were part of the error in the disclosure process included the
opportunity to enhance disclosure. As one medical nurse stated,
‘.a RN should be part of the team because we’ve been with the
patient the whole time,’ while a medical doctor observed,
‘there’s things that the RN would pick up on that I wouldn’t
pick up on.’ Moreover, nurses expressed wanting to be there to
provide their perspective on how nursing contributed to the
error and not be solely blamed for it.
Patient preferences
Some participants within the medical cohort held the view that
patients would initially only ‘want to talk to the doctor’ as one
surgical nurse noted: ‘but maybe they would like to see the surgical
team coming and apologising for error if there was an obvious
error’but not at the initial stage because it’s just too over-
whelming for them.’ However, others viewed that team-based
error disclosure was a way to meet ‘information needs’ and ‘offer
support’ to the patient during the error disclosure process. As one
medical nurse stated, ‘a nurse being there again, can offer support
and stay a few extra minutes with the patient.’ This view was also
expressed by a medical doctor who noted: ‘The three of us being
there was much better than just one individual; it offers more
support. The patient can relate or ask questions to any of the three
and they can interject or comment.’
Cultural and professional norms
Participating in team-based error disclosure was viewed by some
as an ‘effective’ way to gain a deeper understanding of what
occurred in the error process, what details and how the error
should be disclosed, and what steps are being taken to ‘prevent
(similar error) from happening again.’ However, for some
participants, team-based disclosure was described as ‘a bit
forced’ or ‘rehearsed.’ Some study participants suggested that
team-based error disclosure would not happen in actual clinical
practice often appealing to a traditional, hierarchical view of the
surgeon or physician as having the ultimate responsibility on
a healthcare team. Study participants described the physicians as
the ‘top guy’, ‘leader of the team’ and ‘ultimate one responsible’
for error disclosure, regardless of their involvement in the error.
As one surgeon commented: ‘I consider my position as being sort
of the bridge between what happened and the patient, in terms
of the common denominator.whether I was responsible for it
or not.’ Nurse participants, particularly surgical, supported this
view. As one participant stated: ‘Usually if anything happens the
MDs are probably [best to disclose] given the relationship that
they have with their patient.’ The view of the physician in the
disclosure process influenced how some study participants
perceived whether team-based disclosure would actually happen
in practice. This is evident by the following two narratives:
‘It’s a very foreign concept that they (nurse) would come to the
patient bedside and apologise for something. It just in the past it
never happened, I don’t even know if it happens now.’ (Surgeon)
‘I don’t think it (team-based disclosure) could, I don’t think it
would happen, I can’t see it happening’ (Surgical nurse).
DISCUSSION
The delivery of healthcare is complex and involves interactions
among different clinicians; hence understanding how healthcare
teams approach error disclosure is essential.8 Our study is the first

Qual Saf Health Care 2010;19(Suppl 3):i57ei60. doi:10.1136/qshc.2009.036418 i59

 Original research
to explore physicians’ and nurses’ perceptions associated with
a team-based error disclosure simulation intervention. Many of
the findings support what has been previously identified as key
concerns with disclosure perceived by clinicians1e5 7 8 13 and
patients.14 Our findings provide greater insight into the tension of
team-based error disclosure as an unrealistic, forced practice versus
a realistic, beneficial practice. This tension was highly contextual
whereby study participants would determine whether disclosing
an error should be solely the physician’s responsibility or should
involve other healthcare team members based on the specific
situation. For these nurse and physician participants, team-based
error disclosure was seen as more appropriate in more serious error
situations. This finding is consistent with existing research that
has found that physicians, and in some cases nurses, are more apt
to disclose an error that had significant or major harm compared
with a minor or no harm event.1 8 13 14
The nature of the error also contributed to the participants’
perceptions as to the appropriateness of team-based error
disclosure. When errors involved multiple disciplines, these
participants believed that team-based error disclosure would lead
to increased communication among the team and hence, clearer
and more consistent communication with the patient. This may
parallel studies finding that proactive communication appears to
improve outcomes for patients and patients’ families.15 16 A study
exploring nurses’ perceptions of error disclosure reported that
nurses expected physicians to lead the disclosure process but that
nurses welcomed the opportunity to communicate directly to the
patient about nursing’s role in the event.8 These findings suggest
that team-based error disclosure may offer an opportunity for
interprofessional collaboration and communication.
Regardless of whether team-based error disclosure was viewed
as beneficial or unrealistic, study participants identified the
process as a new way of relating that departed from the tradi-
tional view of the physicians and surgeons as the primary
clinician to disclose errors to patients. While most participants
described their experience in the simulation as a worthwhile
learning experience, the surgical cohort in particular did not see
team-based error disclosure as a current or future practice. Many
surgical study participants noted that the common practice was
for the surgeon to disclose errors to patients as they are
perceived to be the only clinician that is constant to the patient
during the pre-, peri- and postoperative phases. Thus, team-
based error disclosure for the surgical cohort was viewed as
unrealistic and problematic. This latter finding is consistent
with what is known about introducing change, as it tends to be
fraught with uncertainty and uneven success.17 For example,
people resist change that has the potential to undermine their
expertise, status and professional identity.18 Moreover, some find
it difficult to accommodate information that conflicts with their
current beliefs and methods.19 Further attention is required to
explore the factors that influenced team members roles including
perception of communication skills, prior experiences with error
disclosure, and attitudes and beliefs around professional roles.
An important limitation of these results is that the sample
was from only three hospitals in one urban city. However, our
sample did include professionals from both teaching and
community-based hospitals and both medical and surgical
specialities. The finding that there were no differences between
the two teaching hospitals and one community hospital may
not be transferable to other healthcare settings, and thus further
enquiry is required. Second, our vignettes described errors that
were relatively clear and involved moderate patient harm.
Ambiguous errors and those involving devastating harm such as
patient death may evoke different responses.
i60
Our findings demonstrate that clinicians’ views as to who
should be involved in the disclosure are influenced by the perceived
nature of the error, patient preferences, and prevailing cultural and
professional norms. At an organisational level for healthcare
administrators and policy makers, these findings suggest that
a team-based approach to disclosure is not realistic or necessary for
all error situations, such as when the error involves a single
discipline. However, when the error involves a variety of health-
care professionals interacting with the patient, a team-based
approach may be beneficial to both the team and patient. In this
context, team-based error disclosure will require ethos and struc-
tures that involve new patterns in the organisation of professional
work where collaborative interdependence is enacted.20 Thus,
organisations need to provide opportunities for healthcare
professionals to practise team-based error disclosure in a safe and
supported environment. At the discipline level, clinicians may be
required to construct and change the rules they follow and how
they engage in disclosure in their daily practice. Finally, researchers
can build on this empirical base to measure the potential outcomes
to patients of team-based error disclosure including satisfaction
with communication with the team.
Funding Canadian Patient Safety Institute.
Competing interests None.
Ethics approval Ethics approval was provided by the Research Ethics Board at
Ryerson University and the institutional Research Ethics Board at the three
participating study sites.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Levinson W, Gallagher TH. Disclosing medical errors to patients: a status report in
2007. CMAJ 2007;177:265e7.
2. Blendon RJ, DesRoches CM, Brodie M, et al. Views of practicing physicians and the
public on medical errors. N Engl J Med 2002;347:1933e40.
3. Fein SP, Hilborne LH, Spiritus EM, et al. The many faces of error disclosure: A
common set of elements and a definition. J Gen Intern Med 2007;22:755e61.
4. Gallagher TH, Garbutt JM, Waterman AD, et al. Choosing your words carefully: how
physicians would disclose harmful medical errors to patients. Arch Intern Med
2006;166:1585e93.
5. Chan DK, Gallagher TH, Reznick R, et al. How surgeons disclose medical errors to
patients: a study using standardized patients. Surgery 2002;138:851e8.
6. Schoen C, Osborn R, Huynh PT, et al. Taking the pulse of health care systems:
Experiences of patients with health problems in six countries. Health Aff (Millwood)
2005;4:W5509e25.
7. Moskop JC, Geiderman JM, Hobgood CD, et al. Emergency physicians and
disclosure of medical errors. Ann Emerg Med 2006;48:523e31.
8. Shannon SE, Foglia MB, Hardy M, et al. Disclosing errors to patients: perspectives
of Registered Nurses. Jt Comm J Qual Saf 2009;35:5e12.
9. Canadian Patient Safety Institute. Disclosure Working Group. Canadian disclosure
guidelines. Edmonton, Alberta, AB: Canadian Patient Safety Institute; 2008. Author.
10. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services
research. Health Serv Res 2007;42:1758e72.
11. Ryan GW, Bernard HR. Techniques to identify themes. Field Methods
2003;15:85e109.
12. Gallagher TH, Waterman AD, Ebers AG, et al. Patient’s and physicians’ attitudes
regarding the disclosure of medical errors. JAMA 2003;289:1001e7.
13. Espin S, Regehr G, Levinson W, Baker R, et al. ‘Error or ‘act of god?’: a study of
patients’ and operating room team members’ perceptions of error definition,
reporting and disclosure. Surgery 2006;139:6e14.
14. Iedema R, Sorensen R, Manias E, et al. Patient’s and family members’ experiences
of open disclosure following adverse events. International Journal for Quality in Health
Care 2008;20:421e32.
15. Lautrette A, Darmon M, Megarbane B, et al. A communication strategy and
brochure for relatives of patients dying in the ICU. N Engl J Med 2007;356:469e78.
16. Zhang B, Wright AA, Huskamp HA, et al. Health care costs in the last week of life:
associations with end-of-life conversations. Arch Intern Med 2009;169:480e8.
17. Stensaker I, Falkenberg J, Gronhaug K. Implementation activities and organizational
sensemaking. J Appl Behav Sc 2008;44:162e85.
18. Carroll JS, Quijada MA. Redirecting traditional professional values to support safety:
changing organisational culture in health care. Qual Saf Health Care 2004;13:16e21.
19. Akgun AE, Byrne JC, Lynn GS, et al. Organizational unlearning as changes in beliefs
and routines in organizations. J Org Change 2007;20:794e812.
20. Adler PS, Kwon SW, et al. Professional work: the emergence of collaborative
community. Org Science 2008;19:359e76.
Qual Saf Health Care 2010;19(Suppl 3):i57ei60. doi:10.1136/qshc.2009.036418

1
Agency for Healthcare
Research and Quality, Rockville,
Maryland, USA
2
Regenstrief Institute, Indiana
University-Purdue University
Indianapolis, Indianapolis,
Indiana, USA
Correspondence to
Dr Teresa Zayas-Cabán, Agency
for Healthcare Research and
Quality, 540 Gaither Road,
Room 6115, Rockville, MD
20850, USA; teresa.
zayascaban@ahrq.hhs.gov
Accepted 5 May 2010
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897
Considerations for the design of safe and effective
consumer health IT applications in the home
Teresa Zayas-Cabán,1 Brian E Dixon2
ABSTRACT
Introduction Consumer health IT applications have the
potential to improve quality, safety and efficiency of
consumers’ interactions with the healthcare system. Yet
little attention has been paid to human factors and
ergonomics in the design of consumer health IT,
potentially limiting the ability of health IT to achieve
these goals. This paper presents the results of an
analysis of human factors and ergonomics issues
encountered by five projects during the design and
implementation of home-based consumer health IT
applications.
Methods Agency for Healthcare Research and Quality-
funded consumer health IT research projects, where
patients used the IT applications in their homes, were
reviewed. Project documents and discussions with
project teams were analysed to identify human factors
and ergonomic issues considered or addressed by
project teams. The analysis focused on system design
and design processes used as well as training,
implementation and use of the IT intervention.
Results A broad range of consumer health IT
applications and diverse set of human factors and
ergonomics issues were identified. The design and
implementation processes used resulted in poor fit with
some patients’ healthcare tasks and the home
environment and, in some cases, resulted in lack of use.
Clinician interaction with patients and the information
provided through health IT applications appeared to
positively influence adoption and use.
Conclusions Consumer health IT application design
would benefit from the use of human factors and
ergonomics design and evaluation methods. Considering
the context in which home-based consumer health IT
applications are used will likely affect the ability of these
applications to positively impact the quality, safety and
efficiency of patient care.
INTRODUCTION
Over the last decade, the healthcare industry has
recognised patients’ changing role as consumers of
health information and their increased use of health
IT to support patienteprovider communication
and patient-centred care.1 Consumer health IT
applications have the potential to improve care
delivery, empower patients to be active participants
in their care and support health management acti-
vitiesdkey components of patient-centred care,
which the Institute of Medicine has identified as
important to ensuring quality and patient safety.2
While health IT has been demonstrated to improve
patient safety and quality of healthcare provided in
clinical settings, realising the potential quality and
safety benefits of consumer health IT interventions
Original research
rests on several factors: (a) the interventions are
designed to ‘fit’ consumers and the environment in
which they will use them, (b) consumers adopt and
effectively use the interventions, and (c) structured
evaluation measures exist to assess the impact of
the interventions on quality and safety.3 4
Consumer health IT applications are electronic
information and communication technology appli-
cations that individuals use to improve their health
outcomes and/or participate in healthcare decision-
making processes.5 Examples include in-home moni-
toring, disease management, self-management and
web-based portals that give patients access to infor-
mation from their provider’s medical record system.6
Similar to clinical health IT applications,
consumer health IT applications have been devel-
oped using a technology-push approach, which
does not allow developers to design applications
appropriate for the home environment or under-
stand their impact following implementation.7e10
Lack of fit between clinical health IT systems and
their users, as well as their users’ work environ-
ment, can create inefficiencies and facilitate error
risks that could lead to decreased patient safety.11 12
In addition, poor design processes affect adoption
and appropriate use of consumer health IT appli-
cations, and create barriers that may hinder user
acceptance and limit overall use.13 Jimison et al, for
example, found several barriers to consumer use of
interactive health IT, including lack of a perceived
benefit; lack of fit with users’ normal daily routines;
overly cumbersome data entry; low clinician
participation; lack of fit between the application’s
recommendation and the patient’s mental model;
technical issues; and limited access to the tech-
nology needed to use the application.6 Barriers to
the adoption and appropriate use of clinical health
IT systems that have led to negative impact on
clinician workflow, quality and patient safety have
been well documented.11 14e17
The field of human factors and ergonomics
provides a structured evaluation framework to
guide the design and implementation of consumer
health IT applications, which can then influence
the adoption and appropriate use of these systems
and ensure that they can be used to improve
healthcare quality.18 Specifically, human factors and
ergonomics methods and approaches can assist
with understanding the tasks required for individ-
uals to take care of themselves and use the IT
solution; understanding user needs and preferences;
assessing users’ physical and cognitive abilities; and
ensuring the context and environment in which the
technology is being used are taken into account.19
For example, human factors and ergonomics
methods and techniques are increasingly applied to
i61
 Original research

the study of the impact of clinical health IT systems to better
understand healthcare patterns of communication and work-
flow that contribute to system inefficiencies, poor quality and
safety risks.14 20
This article focuses on the grants funded by the Agency for
Healthcare Research and Quality (AHRQ) through its Health IT
Portfolio that developed a consumer health IT intervention for
use by patients and/or care givers in the home.21 22 The authors
used multiple methods to capture human factors and ergo-
nomics issues considered or faced by AHRQ grantees during the
design and implementation stages of home-based consumer
health IT applications.
METHODS
The authors examined AHRQ grants funded under the Trans-
forming Health Care Quality through the Health IT (THQIT)
initiative.21 22 Project abstracts and applications were reviewed to
determine which projects included an IT intervention intended for
use by patients and/or their families and care givers. Grants were
further screened to identify projects where patient use of the IT
intervention occurred primarily in the home. The authors defined
the home environment as private residences, such as apartment-
type dwellings and freestanding units, but not more structured
facilities such as nursing homes and assisted living communities.23
For selected projects, the authors conducted an in-depth
review of the original grant application, subsequent interim
status and final reports, and available publications. The grant
documents were reviewed for background information on the
project; system design and design processes used; and reported
information on training, implementation and use of the IT
intervention. Background review focused on the type of health
IT intervention, its purpose and platform, the target population
for the project, whether the system was currently live and the
setting in which the system was expected to be or was used.
System design and process review focused on IT application
development (in house or vendor), customisability, requirements
criteria, IT application piloting or prototype testing, and the
extent of contextual influence on the design or purchase of the
application. Finally, the authors reviewed whether training or
educational materials were provided to users, application
implementation and system use.
Following the document review, open-ended discussions were
scheduled with project investigators and/or study coordinators
to clarify items abstracted from the grant documents. During
the telephone discussions with grantees, the researchers took
detailed notes, which were enhanced and annotated following
the conversations.
Documents and discussion notes were analysed to understand
which human factors issues were considered or faced by grantees
during design, implementation and use of the health IT appli-
cations. In particular, the authors looked for commonalities and
differences across projects and the extent to which they might
be affected by the type of application and target population.
RESULTS
Projects, health IT interventions and user populations
Five AHRQ-funded projects implemented consumer health IT
applications that targeted patients’ home use. Of these, two
were developed specifically for home use as part of the patients’
usual health monitoring and management. One was developed
for patients to share medical information with physicians after
leaving the hospital. The remaining two were web-based and
could potentially be used from the home as well as from other
remote locations with internet access. Four of the applications
were commercial solutions, while a fifth application started as
a manual, homegrown process for information gathering and
was later integrated with a commercial electronic health record
(EHR) system. Table 1 summarises the health IT applications
used in the selected projects.
Projects 1 and 2 in table 1 included clinician feedback. Infor-
mation entered into the IT system was sent to the clinician for
either immediate response or review with the patient during the
next visit. In particular, the interactive voice response (IVR)
system would generate reports at the clinician’s office every time
the patient submitted personal data (ie, blood pressure, weight).
Projects 1, 2 and 4 included care management components as
part of the intervention.
As also shown in table 1, all but one of the projects targeted
patients with chronic diseases. Project 5 focused on premature
infants with complex medical needs and at risk for neuro-
developmental disorders. Projects 2 and 5dIVR and
PDHRdfocused specifically on patients in rural settings.
Human factors and ergonomics considerations
Design considerations
Because all five grants were research projects, the investigators had
specific functionality in mind when selecting vendor products or

Table 1 Consumer health IT interventions, target population, users and healthcare settings
Medical Healthcare
Project IT application devices used Target population Home users Other users settings involved
1 Videophone and Camcorder < Glucometer Elderly, Medicare < Patients < Wound-care < Home health agencies
or Digital Camera < Blood pressure beneficiaries with < Care givers specialists < Long-term care facilities
monitor chronic wounds (spouse or son/daughter) < Diabetes
< Scale < Home health staff educators
2 Interactive Voice Response < Glucometer Older, medium- and < Patients < Clinicians < Primary care clinics
System and Modem < Blood pressure high-risk congestive < Care givers (across four states)
monitor heart-failure patients
< Scale
3 Tethered Personal Health General adult patient < Patients < Clinicians < Primary care practices
Record (focus on secure population and subsamples < Care givers or proxies (affiliated with integrated
messaging) for: (1) depression, (2) diabetes delivery service networks)
and (3) congestive heart failure
4 Tethered Personal Health < Glucometer Adult patients with diabetes < Patients < Nurse < Primary care practices
Record with care managers (affiliated with integrated
management component < Primary care delivery service networks)
providers
5 Portable Personal Infants born prematurely < Patient representatives < Physicians < Neonatal intensive care unit
Developmental Health at risk for neurodevelopmental (parents or guardians) < Nurse (public hospital)
Record CD or DVD problems practitioners < Family medicine clinics

i62 Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897


developing the intervention. If working with a vendor, design
considerations focused primarily on the functions offered by the
vendor’s product, the level of support offered and the ability to
modify functions or features if needed. For example, one of the
personal health record projects sought the capability for proxies to
access information on the behalf of children under 13 years of age
as well as geriatric patients. All of the projects focused on the
content that could be stored or transmitted by the technology.
For example, one project team asked for the health IT system to
include information such as birth records, primary care records,
hospital records and medications.
While all projects targeted patient populations or care givers
with unique healthcare needs, two of them targeted older
populations. These teams considered the unique needs of their
users when interacting with devices and computers. For
example, both project teams wanted their devices to automati-
cally transmit patient information with the push of a single
button. They understood from prior research that older popu-
lations may not be computer-savvy and would be unwilling to
use a computer-like device. To incorporate users’ needs, both
teams worked closely with vendors to modify initial system
designs. One team knew from prior research that their patients
found early videophones frightening, so the team and the vendor
developed a videophone that operated ‘like a regular phone’ and
included a flip-down lid for privacy. Finding usable devices,
however, was not always easy. One project team had not found
a spirometer, used to measure pulmonary function, that was
easy for patients with arthritis to operate. Another team was
unable to locate extra large or extra small (eg, paediatric) blood
pressure cuffs.
Project teams looked for flexibility from vendors. Patients used
different models of glucometers, so teams needed to ensure the
information from different models could easily be sent to the
information system. One team also needed the commercial
system to adjust for variance in care plans, ensuring that
patients could transmit data no matter what combination of
devices were installed in the home.
Design approach, testing and prototyping
Study designs were examined to understand the design approach
and to assess whether patients and/or care givers were involved
in the selection or design of the system. Patient involvement
varied from project to project, but most projects tried to gather
feedback before deploying the technology. For example, when
difficulties arose during initial use of the system, the IVR project
stopped the study and used volunteers to test both the devices
(eg, scale and blood pressure monitor) and the IVR system itself.
Volunteers pilot-tested the equipment at home, revealing several
issues with the system and the reports it generated. The project
team subsequently asked the vendor to make revisions to
minimise user error and provide better feedback.
Two of the projects used focus groups that influenced some of
the technology’s features (eg, ability to enter glucose readings or
refine alerts provided to patients). Another project used patient
focus groups in the early development of the system, focussing
on the information being provided and usability testing of the
interface. One of the project teams tested the telemonitoring
technology during prior work and found video cameras were less
expensive and worked better than medical device cameras. Two
of the projects used continuous evaluation and iteration based
on feedback from both patients and clinicians, which, for one of
the personal health record (PHR) projects, resulted in the addi-
tion of a new feature to satisfy requests from patients,
employers and purchasers.
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897
Original research
Implementation
Approaches for enroling patients and/or offering patients the
technology varied across projects. One of the PHR projects
employed many strategies to boost enrolment, including making
brochures and CDs available from clinic staff, purchasing
advertisements on buses and prime-time television, instituting
an incentive programme to enrol patients and asking clinicians
to encourage patients to use secure messaging. The other PHR
project enroled patients through kiosks. Parents of premature
infants at risk for neurodevelopmental disorders were asked at
the time of discharge from the neonatal intensive care unit
(NICU) if they wanted a personalised CD created for them.
Patients participating in the wound care project were enroled
through a home health service, but the wound care centre
installed the equipment, including needed devices (ie, gluc-
ometer, blood pressure monitor, or scale), and provided training
on use. The IVR system project enroled patients discharged
following an episode of care related to chronic heart failure
(CHF). Equipment was sent to patients’ homes after enrolment
for self-installation. However, some participants encountered
significant barriers, such as a lack of compatibility between the
IVR and voice-over internet protocol (VOIP) or cellular phones.
The project team found that some patients did not have the
device installed 30e60 days after receiving it. Because patients
were dispersed over a large geographic area, home visits to install
the devices were delayed, and researchers eventually contracted
for that service.
Training and support
All projects provided patients with training or user manuals.
The approaches varied from project to project. One of the
projects using a PHR allowed patients to view a demonstration
from the provider ’s website to learn how the portal worked.
Another project developed training guides for patients, providers
and patient service representatives, and distributed training
materials to the patients directly. The vendor for a third project
sent the device to the home and provided installation instruc-
tions. A fourth project provided training to family members on
how to use, update and access the technology. The last project
provided training at the point of installation; furthermore,
patients were given training on how to use a glucometer and
were referred to a diabetes educator.
All projects provided IT support for their patients to address
questions or problems while using the technology. IT support
methods included providing a 24 h help desk, secure messaging
to clinic staff, technical assistance by project (study team) staff
and phone support through a vendor.
Use
Four of the five projects reported information on adoption and
use. The two PHR projects examined which patients used secure
messaging or portal features, respectively. Both found that use
varied with patient characteristics and was influenced by clini-
cians’ engagement.24 Applications were used more commonly by
female patients who were chronically ill. While elderly patients
did use secure messaging, in some instances their proxies or care
givers were the main users of the PHR. The second PHR study
found that race, education and prior internet use influenced
adoption and use. Access to broadband internet also influenced
use. Information often did not download fast enough for dial-up
users, which resulted in fewer dial-up adopters than broadband.
In the two telemonitoring projects, the context of home use
influenced people’s ability to operate the technology and deter-
mined how it was used or whether any changes were needed.
i63
 Original research
For the IVR project, researchers found some participants had no
phone outlets in the bedroom or bathroom, or had rotary phones
that could not be used to transmit weight and other data.
Outlets and phone lines were installed where needed or touch-
tone phones provided to accommodate those participants. Other
patients were homeless or transient, so phones were set up
where they worked or at other locations. Finally, Native Amer-
ican participants shared common homes; therefore, the phone
system was set up at a clinic where participants would go daily
to use the devices.
The way in which some daily health-management tasks were
conducted affected how the technology was set up and used. For
example, depending on the location of the phone and the room
in which patients kept their medical device, (eg, scale), some
patients had to tape the modem’s cable along the wall from the
kitchen to the bedroom. One patient in the wound care study
wrote down weights and called them into the provider later
because of the distance between the scale and the phone outlet.
In addition, when using scales, several patients failed to recog-
nise that scales perform differently when placed on carpet than
on flat surface flooring. Elderly patients who needed assistance
to walk often had trouble standing still long enough to record an
accurate weight measurement.
Patient responses to problems while using the technology
varied. Some refused the home devices as soon as they received
them. For some participants, if the technology failed once or
twice, they no longer wanted the device. Others vented their
frustration regarding the need to frequently replace batteries
(and pay for them) or interact with unresponsive vendors.
Finally, in some cases, equipment was replaced multiple times,
and disillusioned patients did not give the technology a fifth or
sixth chance.
Other design, implementation and use factors
Provider interaction
Provider interaction affected application design and use. All
projects depended on clinicians interacting with the application
in some way; therefore, their needs were incorporated as part of
the design of the intervention, and in some cases, their level of
use influenced the patients’ use. The institution affiliated with
one of the PHR projects implemented a philosophy that secure
messaging had become a daily part of patient care and provided
clinicians with training on its use. The second PHR project
created institutional policies about the use of the clinician
interface with the application; physicians were given an hour-
long training session on the portal’s functionality. The PDHR
project also incorporated clinicians’ feedback once a mock CD
had been developed. The team conducted focus groups with
clinicians, NICU nurses and primary care practitioners to obtain
feedback on the CD’s content. All pilot testing involved families
as well as primary care practitioners, whose feedback influenced
revisions to the PDHR’s format.
Clinicians’ use of or interaction with the application had an
effect on patient use. For the wound care project, the study team
found increased compliance with patients who were aware that
someone was ‘watching’ the information flow. They also found
the perceived engagement of the clinician providing care
instructions influenced how closely patients followed instruc-
tions. For example, wound healing time increased when patients
or their care givers used remedies recommended by friends or
others. However, having a clinician reassert during tele-
consultations that patients follow instructions provided by
home health workers increased patient adherence. Finally,
a practice effect occurred during the course of the IVR study.
i64
Reports were sent to clinicians after each data entry. The prac-
tice found the increased volume of information overwhelming.
Patients were subsequently educated on how to send readings
once a day.
Vendor support
All five projects worked with vendors, and in some instances, the
vendor’s level of support affected the participants’ ability to use
the system. For one project, the vendor’s unreliable response was
frustrating to patients and researchers alike. Furthermore,
vendor support was only available weekdays from 9:00 until
17:00 EST. This affected patients’ and clinicians’ ability to get
help outside those time limits.
Compatibility of devices with the information system in use
was another issue. One project received a new model of gluc-
ometers for patient participants, but the vendor system did not
have the appropriate protocols to recognise the newer device.
Project researchers obtained the needed protocol from the
manufacturer of the new device; in the interim, older gluc-
ometers were used, or data were input by patients manually.
Finally, the PDHR project team began developing the CD by
uploading data directly on site at the hospital. As they began to
work with a vendor to automate the creation of PDHRs, they
encountered several delays because the PDHR was not an orig-
inal component of their system. Getting vendor agreement on
the categories of information for the CD presented one barrier.
The vendor solution also took longer than expected, so the
project team developed an intermediate solution and hired
a consultant.
DISCUSSION
The projects included diverse interventions and environments
that raise several human factors and ergonomic issues. First, the
projects involved a range of end users (patients, care givers and
providers) with distinct physical and cognitive capabilities and
limitations. In addition, the applications were intended for use
in homes with different physical characteristics. The studies
carefully examined and tailored system design to meet the needs
of care providers. A similar approach is also needed in system
design to recognise patients’ different capabilities, limitations
and varying constraints of the home environment.
For example, gradually introducing technology to an older
population has been shown to promote health IT use.6 Accord-
ingly, the two projects that targeted older patients considered
needs such as patient comfort with devices and technology.
These teams tried to find technologies that were easy to install
and use, although they were constrained by what was available
in the market. Another project explicitly considered the care
giver’s role by allowing them proxy access to the patient’s
information.
If users’ needs and capabilities are not carefully considered,
less than optimal installation or use of the application may
result. For example, two studies found differential use of the
consumer health IT application due to prior internet use and age,
which suggests, consistent with literature reviews, that some
patients’ needs may not have been met by this application.6 In
these two studies, patients who did not use the system likely
had a higher disease burden.
To maximise the benefit of consumer health IT applications,
system designers need to understand how to support the needs,
abilities and limitations of patient populations and how to
complement health IT applications with other interventions. A
successful intervention must accommodate providers and
patients to enhance the patient interface and to promote
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897

patients’ ability to use the system. Designers will need to
consider how to design for these distinct sets of users and to
strike a balance so the system is easy to use and useful for all end
users. Furthermore, as future health IT applications for home use
develop, it is unclear how much input purchasers will have over
the design. Therefore, project teams should conduct robust
needs assessments and have very specific design requirements
when selecting a commercial product.
User testing with the target population in the five projects
was limited. A few projects used focus groups or developed
a mock product. One project conducted a large pilot test,
although the diversity and number of users involved in the
product design were limited. After uncovering problems with
using the technology, the project team performed extensive user
testing, which led to improvements in the application’s design.
However, additional issues were uncovered when implementa-
tion resumed. Engaging the target user population in the testing
may have yielded further improvements to design and increased
overall adoption and use.
In some cases, the technology was designed to be used in
a manner incompatible with health management tasks or
household arrangements. For example, one project found that
many people did not have phones in bedrooms. Rather, phones
were located in other areas of the house, such as the kitchen,
while most scales were kept in bathrooms. Device arrangements
may be unsafe, in particular for older or disabled populations, by
having cables crossing rooms. Alternatively, requiring people to
write down information and relay it to clinicians or electronic
systems may increase errors. In addition, people living in diverse
household arrangements (eg, transient or shared homes) influ-
enced how the consumer health ITapplication was implemented
and used.
The problems encountered by users suggest a need to assess
the functional tasks patients and/or their care givers needed to
perform as well as to test and observe the applications in the
context in which they will be used. However, each project
approached IT system development from a clinical study
perspective, focusing on an established healthcare need. Based on
guidelines of care or clinician recommendations, they found
a technology that could provide a solution. The needs identified
by the investigators centred primarily on content and IT func-
tionality. While it may be difficult to account for all possible
Table 2 Recommendations to improve design of consumer health IT and sample relevant human factors methods
Recommendations
The design process should consider the physical and cognitive needs and abilities of all types of end
users (eg, patients, care givers or clinicians)
The design process should be user-centred with the goal of understanding the ‘health work’ of the
users and the context in which the work occurs
The design process should be iterative, and evaluations should be conducted in environments in
which the intervention will be used. Subjects involved in testing should represent all possible users.
Appropriate training and IT support should be available to users
Technology, platforms and devices used should consider compatibility and user access (eg, by
considering using different platforms or testing using varying internet speeds)
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897
Original research
variations of households, assessments conducted prior to design
can better ensure compatibility of the health IT system with
a majority of the patient population and promote system use.6
The level of provider interaction with system information and
with patients appears to be critical to the success of most of
these projects, which is also consistent with findings from the
literature on consumer health IT.6 This suggests that careful
attention must be paid to ensure that providers see the value of,
and will be able to use, the IT application when needed. Addi-
tional research may be needed, however, to determine the
optimal level of provider involvement and what may be feasible
within the context of a clinician’s workload.
Two projects found differential use of the technology based on
access to the internet, especially broadband. Differential access
to internet services due to cost may prevent different user
populations from benefiting from consumer health IT applica-
tions. Testing the final application might have indicated down-
load ease by internet speed. Similarly, while project teams were
limited by the technology available in the market, in some
instances, they did not recognise these limitations until the
application’s implementation and use. The declining use of land
lines will be an important consideration in design of consumer
health IT applications, as well as the lack of newer technology
among some older patients. Furthermore, the compatibility
between devices and information systems also needs to be
tested. Persistent technical issues led to patients refusing to use
the system in some instances or abandoning it altogether, thus
affecting its impact on quality.6
Engaging patients through clinical settings was the primary
method used to enrol patients. However, installation experiences
varied among projects. For more complex ITapplications, project
teams were required to visit patients’ homes to install the
devices, as patients were unable to do so themselves. While all of
the projects provided support or customer service, feedback on
its effectiveness, from the patients’ or care givers’ perspective,
was unavailable for all but one project. Some of these applica-
tions were designed to be used as part of a home care visit, in
which the visiting nurses or home care workers could provide
additional training or support to the patients, but many did not
anticipate further help for the patient. Even though the poten-
tial risk of harm from misuse of consumer health IT applications
might be less than with complex medical devices, appropriate
Human factors methods
< Cognitive task analysis
< Function analysis
< Usability evaluations (eg, usability tests; heuristic evaluations;
cognitive walkthroughs; think-aloud protocols)
< Visual and hearing impairment simulators
< Cognitive work analysis
< Task analysis
< Function analysis
< Work system analysis
< Interviews
< Direct observations
< Focus groups
< Analysis of artefacts
< Usability evaluations
< Field testing
< Interviews
< Direct observations
< Usability evaluations
< Field testing and evaluation
< Pilot tests
< Surveys
< Review of services and infrastructure available in the community
i65
 Original research
training and support are still needed.25 In addition, system
purchasers may need to negotiate appropriate levels and avail-
ability of support for patients and training modules ahead of
time.
Table 2 includes a list of design recommendations based on the
human factors and ergonomics issues encountered by the five
projects along with sample methods to implement each
recommendation. The recommendations are congruent with
suggestions to improve clinical IT systems and proposed
methods, such as usability and pilot testing, which have been
shown to improve the ease of use and usefulness and improve
overall technology acceptance.26e30
CONCLUSIONS
The innovative work in these five projects has helped begin
a discussion around human factors and ergonomic considerations
for in-home use of consumer health IT. The projects illustrate the
diversity of applications that can be implemented for in-home use
and the breadth of human factors and ergonomic issues that can
arise during implementation and adoption.
Overall, the different projects and applications illustrate
important lessons regarding the complexities of supporting home
healthcare activities through health IT. First, the interaction
between technology, patients, care givers, the home environment
and the clinician must be considered in the design and imple-
mentation of the technology to produce a dynamic that will
improve care. Devices and applications that are difficult to use or
abandoned will likely fail to achieve the quality, safety and effi-
ciency goals of consumer health IT advocates and designers.
Second, a lack of attention to human factors and ergonomic issues
could lead to inadvertent threats to safety by creating hazards in
the home environment or facilitating user error.
Moving forward, consumer health IT application developers
should design in-home applications to function in the absence of
significant IT support. Human factors and ergonomics should be
incorporated early and iteratively into the design of consumer
health IT. If human factors and ergonomics considerations are
incorporated at the end of the development process, any rede-
sign work may be resource-intensive and significantly impair
technology acceptance.29 Therefore, developers should include
human factors professionals in the multidisciplinary design
team to adequately inform the concept, needs assessment and
development process. Consumer health IT developers can also
look to other industries that develop consumer products or
software to see how human factors and ergonomics are
embedded in their development processes. Appropriate inclusion
of these considerations in the design of consumer health IT
applications will enhance adoption and enable long-term
achievement of quality, safety and efficiency goals.
LIMITATIONS
The application of lessons described in this study to other
in-home consumer health IT projects is limited by the small
number of research projects examined. While the lessons here are
important, they may not be applicable in all settings and
contexts. Furthermore, none of the studies employed structured
methods to collect data on human factors and ergonomic issues
encountered when designing, implementing or using the
consumer health IT applications. Thus, other important issues
may exist that were not documented in project reports but are
nonetheless important to consider in the adoption, use, quality
or safety of in-home consumer health IT products. Future
research should explore these and other human factors and
ergonomic issues in more depth and in relation to primary
i66
research studies that examine in-home consumer health IT
application use.
Acknowledgements The authors would like to thank the investigators and staff
involved in the AHRQ health IT projects that contributed to this article. The authors
also thank J Marquard, Assistant Professor at the University Massachusetts
Amherst Department of Mechanical and Industrial Engineering, D Meyers, from the
Agency for Healthcare Research and Quality, C Tokarski from RTI International and
K Dixon for their valuable advice on this document. The authors of this article are
responsible for its content. Statements in the article should not be construed as
endorsement by the Agency for Healthcare Research and Quality or the US
Department of Health and Human Services.
Funding This work was supported in part by the AHRQ National Resource Center for
Health IT, contract no 290-04-0016.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Kaplan B, Brennan PF. Consumer informatics supporting patients as co-producers of
quality. J Am Med Inform Assoc 2001;8:309e16.
2. Corrigan JM, Donaldson MS, Kohn LT, eds. Crossing the Quality Chasm: a New
Health System for the 21st Century. Washington, DC: The National Academies Press,
2001:39e60.
3. Bakken S, Cimino JJ, Hripcsak G. Promoting patient safety and enabling evidence-
based practice through informatics. Med Care 2004;42(Suppl 2):II49e56.
4. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information
technology on quality, efficiency, and costs of medical care. Ann Intern Med
2006;144:742e52.
5. Keselman A, Logan R, Smith CA, et al. Developing informatics tools and strategies
for consumer-centered health communication. J Am Med Inform Assoc
2008;15:473e83.
6. Jimison H, Gorman P, Woods S, et al. Barriers and drivers of health information
technology use for the elderly, chronically ill, and underserved. Rockville, MD: Agency
for Healthcare Research and Quality, 2008. AHRQ Publication No. 09-E004.
7. Zayas-Cabán T, Marquard JL. A holistic human factors evaluation framework for the
design of consumer health informatics interventions. Human Factors and Ergonomics
Society Annual Meeting Proceedings 2009;53:1003e07.
8. Brennan DM, Holtz BE, Chumbler NL, et al. Visioning the technology for the future of
telehealth. Telemed J E Health 2008;14:982e85.
9. Chan M, Campo E, Estève D, et al. Smart homesdcurrent features and future
perspectives. Maturitas 2009;64:90e7.
10. Stead WW, Lin HS, eds. Computational technology for effective health care:
immediate steps and strategic directions. Washington, DC: The National Academies
Press, 2009.
11. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry
systems in facilitating medication errors. JAMA 2005;293:1197e203.
12. Unertl KM, Weinger MB, Johnson KB. Applying direct observation to model
workflow and assess adoption. AMIA Annu Symp Proc 2006:794e8.
13. Or CK, Karsh BT. A systematic review of patient acceptance of consumer health
information technology. J Am Med Inform Assoc 2009;16:550e60.
14. Beuscart-Zéphir MC, Pelayo S, Bernonville S. Example of a human factors
engineering approach to a medication administration work system: potential impact
on patient safety. Int J Med Inform 2010;79:e43ee57.
15. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss.
Ann Intern Med 2006;14:510e16.
16. Koppel R, Wetterneck T, Telles JL, et al. Workarounds to barcode medication
administration systems: their occurrences, causes, and threats to patient safety.
J Am Med Inform Assoc 2008;15:408e23.
17. Saleem JJ, Patterson ES, Militello L, et al. Exploring barriers and facilitators to the
use of computerized clinical reminders. J Am Med Inform Assoc 2005;12:438e47.
18. International Ergonomics Association. What is ergonomics, 2000.
http://www.iea.cc/browse.php?contID¼what_is_ergonomics (accessed Dec 2007).
19. Or CK, Valdez RS, Casper GR, et al. Human factors and ergonomics in home care:
current concerns and future considerations for health information technology.
Work 2009;33:201e9.
20. Kaufman DR, Pevzner J, Rodriguez M, et al. Understanding workflow in telehalth
video visits: observations from the IDEATel project. J Biomed Inform
2009;42:518e92.
21. US Department of Health and Human Services. Transforming Healthcare Quality
Through Information Technology (THQIT)dimplementation grants. Publication No.
RFA-HS-04e011, 2003. http://grants.nih.gov/grants/guide/rfa-files/RFA-HS-04-011.
html (accessed Mar 2008).
22. US Department of Health and Human Services. Limited Competition for AHRQ
Transforming Healthcare Quality through Information TechnologydImplementation
Grants. Publication No. RFA-HS-05e013, 2004. http://grants.nih.gov/grants/guide/
rfa-files/RFA-HS-05-013.html (accessed March 2008).
23. Zayas Cabán T. Application of human factors analysis in the home: a methodology
[dissertation]. Madison: University of Wisconsin-Madison, 2006.
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897

24. Ralston JD, Carrell D, Reid R, et al. Patient web services integrated with a shared
medical record: patient use and satisfaction. J Am Med Inform Assoc 2007;14:798e806.
25. Fex A, Ek AC, Söderhamn O. Self-care among persons using advanced medical
technology at home. J Clin Nurs 2009;18:2809e17.
26. Martı́nez-Ramos C, Cerdán MT, López RS. Mobile phone-based telemedicine
system for the home follow-up of patients undergoing ambulatory surgery.
Telemed J E Health 2009;15:531e7.
27. Trudel M, Cafazzo JA, Hamill M, et al. A mobile phone based remote patient
monitoring system for chronic disease management. Stud Health Technol Inform
2007;129:167e71.
Qual Saf Health Care 2010;19(Suppl 3):i61ei67. doi:10.1136/qshc.2010.041897
Original research
28. Kaufman DR, Pevzner J, Hilliman C, et al. Redesigning a telehealth diabetes
management program for a digital divide seniors population home. Home Health Care
Manag Pract 2006;18:223e34.
29. Karsh BT. Clinical practice improvement and redesign: how change in workflow can
be supported by clinical decision support. Rockville, MD: Agency for Healthcare
Research and Quality, 2009. AHRQ Publication No. 09-0054-EF. http://healthit.ahrq.
gov/images/jun09cdsworkflow/09_0054_ef.html.
30. Saleem JJ, Russ AL, Sanderson P, et al. Current challenges and opportunities for
better integration of human factors research with development of clinical information
systems. Yearb Med Inform 2009;(1):48e58.
i67
 Original research
1
UT - Memorial Hermann Center
for Healthcare Quality & Safety,
University of Texas School of
Biomedical Informatics at
Houston, Texas, USA
2
Houston VA Health Services
Research and Development
Center of Excellence and The
Center of Inquiry to Improve
Outpatient Safety Through
Effective Electronic
Communication, Michael E.
DeBakey Veterans Affairs
Medical Center and Section of
Health Services Research,
Department of Medicine, Baylor
College of Medicine, Houston,
Texas, USA
3
The Center of Inquiry to
Improve Outpatient Safety
Through Effective Electronic
Communication, Michael E
DeBakey Veterans Affairs
Medical Center and the Section
of Health Services Research,
Department of Medicine, Baylor
College of Medicine, Houston,
Texas, USA
Correspondence to
Professor Dean F Sittig,
UTdMemorial Hermann Center
for Healthcare Quality & Safety,
University of Texas School of
Biomedical Informatics at
Houston, 6410 Fannin St UTPB
1100.43, Houston, TX 77030,
USA; dean.f.sittig@uth.tmc.edu
Accepted 3 July 2010
i68
A new sociotechnical model for studying health
information technology in complex adaptive
healthcare systems
Dean F Sittig,1 Hardeep Singh2,3
ABSTRACT
Background Conceptual models have been developed to
address challenges inherent in studying health
information technology (HIT).
Method This manuscript introduces an eight-
dimensional model specifically designed to address the
sociotechnical challenges involved in design,
development, implementation, use and evaluation of HIT
within complex adaptive healthcare systems.
Discussion The eight dimensions are not independent,
sequential or hierarchical, but rather are interdependent
and inter-related concepts similar to compositions of
other complex adaptive systems. Hardware and software
computing infrastructure refers to equipment and
software used to power, support and operate clinical
applications and devices. Clinical content refers to textual
or numeric data and images that constitute the
‘language’ of clinical applications. The humanecomputer
interface includes all aspects of the computer that users
can see, touch or hear as they interact with it. People
refers to everyone who interacts in some way with the
system, from developer to end user, including potential
patient-users. Workflow and communication are the
processes or steps involved in ensuring that patient care
tasks are carried out effectively. Two additional
dimensions of the model are internal organisational
features (eg, policies, procedures and culture) and
external rules and regulations, both of which may
facilitate or constrain many aspects of the preceding
dimensions. The final dimension is measurement and
monitoring, which refers to the process of measuring and
evaluating both intended and unintended consequences
of HIT implementation and use. We illustrate how our
model has been successfully applied in real-world
complex adaptive settings to understand and improve
HIT applications at various stages of development and
implementation.
INTRODUCTION
An ongoing challenge to the design, development,
implementation and evaluation of health informa-
tion technology (HIT) interventions is to oper-
ationalise their use within the complex adaptive
healthcare system that consists of high-pressured,
fast-paced and distributed settings of care delivery.
Many conceptual models of user interaction,
acceptance and evaluation exist,1 2 but most are
relatively limited in scope. Given the dearth of
models that are specifically designed to address safe
and effective HIT development and use, we have
developed a comprehensive, sociotechnical model
that provides a multidimensional framework
within which any HIT innovation, intervention,
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085
application or device implemented within
a complex adaptive healthcare system can be
studied. This model builds upon and bridges
previous frameworks, and is further informed by
our own work to study the safe and effective
implementation and use of HIT interventions. In
this paper, we describe the conceptual foundations
of our model and provide several examples of its
utility for studying HIT interventions within real-
world clinical contexts.
BACKGROUND
Previous analyses of HIT interventions have been
limited by a lack of conceptual models that have
been specifically developed for this purpose.
Examples of models previously applied by HIT
investigators include Rogers’ diffusion of innova-
tions theory,3e5 Venkatesh’s unified theory of
acceptance and use of technology,6e9 Hutchins’
theory of distributed cognition,10e14 Reason’s
Swiss Cheese Model15e17 and Norman’s seven-step
humanecomputer interaction (HCI) model.18e20
Although all of these models account for one or
more important facets of technology implementa-
tion, we believe that the scope of each model limits
its utility to address the full range of factors that
should be considered in the design, development,
implementation, use and evaluation of HIT inter-
ventions. For example, these models were not
specifically designed to address the complex rela-
tionships between the HIT hardware, software,
information content and HCI. Furthermore, while
most of these models provide general guidance to
study the high-level aspects of HIT implementation
within a given clinical environment, none of
them includes a measurement and monitoring
infrastructure (eg, methods to routinely collect
data, create or review reports or conduct
surveillance of outcomes). Based on these limita-
tions, our aim was to develop a more comprehen-
sive model to integrate specific technological and
measurement dimensions of HIT with other soci-
otechnical dimensions (eg, people, workflow,
communication, organisational policies, external
rules and regulations).
Previous health information technology-focused
sociotechnical systems models
Four related sociotechnical models have been
particularly influential in providing the foundation
of our proposed model. First, Henriksen’s model
addresses (1) individual provider characteristics; (2)
the nature or complexity of the work or task
performed; (3) the physical environment where

care takes place; (4) the humanesystem interfaces involved; and
(5) various characteristics of the organisation (social, environ-
ment and management).21 Second, Vincent’s framework for
analysing risk and safety proposes a hierarchy of factors that can
potentially influence clinical practice.22 Third, Carayon’s
Systems Engineering Initiative for Patient Safety model23 iden-
tifies three domains: (1) characteristics of providers, their tools
and resources, and the physical/organisational setting; (2)
interpersonal and technical aspects of healthcare activities; and
(3) change in the patient’s health status or behaviour. Finally,
Harrison et al’s Interactive Sociotechnical Analysis framework
provides an excellent broad overview of the complex, emergent,
inter-relationships between the HIT, clinicians and workflows
within any healthcare system.24
While these sociotechnical models include a ‘technology’
component, none break down the ‘technology’ into its indi-
vidual components to enable researchers to dissect out the
causes of particular HIT implementation or use problems, or to
help identify specific solutions. We have found that many HIT
problems we are studying revolve around the interplay of
hardware, software, content (eg, clinical data and computer-
generated decision support) and user interfaces. Failing to
acknowledge these specific technology-specific elements or
attempting to treat them separately can hinder the overall
understanding of HIT-related challenges. For example, the
‘content’ dimension of our model accounts for much of what
informaticians do, that is, studying the intricacies of controlled
clinical vocabularies that provide the cognitive interface between
the inexact, subjective, highly variable world of biomedicine and
the highly structured, tightly controlled, digital world of
computers.25 A well-constructed, robust user interface vocabu-
lary can make all the difference in the world to a busy clinician
struggling to quickly and accurately enter a complex clinical
order for a critically ill patient,26 and it is important to distin-
guish this aspect of technology from others that may contribute
to additional challenges (eg, a user interface ie, difficult to
Figure 1 Illustration of the complex
inter-relationships between the eight
dimensions of the new sociotechnical
model.
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085
Original research
navigate, an order entry application, ie, slow to respond, or
computers that are only available at the main nursing station).
Failure to do so, for example, leads to general statements such as
‘clinicians struggled with the new technology’ or ‘it takes
clinicians longer to complete their tasks using the new tech-
nology’ without providing any insight into specific causes of the
problems or their solutions. In this example, without a multidi-
mensional understanding of the technological dimensions of the
failed IT application, the researcher may conclude incorrectly
that the hardware, application software or user was responsible,
when in fact a poorly designed or implemented clinical vocab-
ulary might have been the root of the problem.
Finally, the preceding models do not account for the special
monitoring processes and governance structures that must be
put in place while designing and developing, implementing or
using HITdfor example, identifying who will make the decision
on what, when and how clinical decision support (CDS) inter-
ventions will be added27; developing a process for monitoring
the effect of new CDS on the systems’ response time28; building
tools to track the CDS that is in place29; developing an approach
for testing CDS; defining approaches for identifying rules that
interact; developing robust processes for collecting feedback
from users and communicating new system fixes, features and
functions; and building tools for monitoring the CDS system
itself.30
Moving towards a new sociotechnical model for HIT
To overcome the limitations of previous models, we propose
a new sociotechnical model to study the design, development,
use, implementation and evaluation of HIT (figure 1). Our
comprehensive eight-dimensional model accounts for key factors
that influence the success of HIT interventions. A major
assumption of our model is that the eight dimensions cannot be
viewed as a series of independent, sequential steps. As with
other components of complex adaptive systems, these eight
interacting dimensions must be studied in relation to each other.
i69
 Original research
Clearly, several of our model’s components are more tightly
coupled than othersdfor example, the hardware, software,
content and user interface are all completely dependent on one
another. However, all the other social components also exert
strong influences on these technical components.
In our model, one cannot expect to gain an in-depth under-
standing of the intricacies of complex HIT interventions simply
by integrating the results of studies performed within any single
dimension of the model.31 Rather, HIT interventions must be
understood in the context of their simultaneous effects across
multiple dimensions of the model. For instance, a recent evalu-
ation of a national programme to develop and implement
centrally stored electronic summaries of patients’ medical
records in the UK revealed their benefits to be lower than
anticipated and cautioned that complex interdependencies
between many sociotechnical factors at the clinical encounter-,
organisational- and national level are to be expected in such
evaluations.32 These study findings are illustrative of how and
why our proposed model could be useful.
The eight dimensions are detailed in the following sections.
Hardware and software computing infrastructure
This dimension of the model focuses solely on the hardware and
software required to run the applications. The most visible part
of this dimension is the computer, including the monitor,
printer and other data-display devices along with the keyboard,
mouse and other data-entry devices used to access clinical
applications and medical or imaging devices. This dimension
also includes the centralised (network-attached) data-storage
devices and all of the networking equipment required to allow
applications or devices to retrieve and store patient data. Also
included in this dimension is software at both the operating
system and application levels. Finally, this dimension of the
model subsumes all the machines, devices and software required
to keep the computing infrastructure functioning such as the
high-capacity air-conditioning system, the batteries that form
the uninterruptable power supply that provides short-term
electrical power in the event of an electrical failure and the
diesel-powered backup generators that supply power during
longer outages.
In short, this dimension is purely technical; it is only
composed of the physical devices and the software required
keeping these devices running. One of the key aspects of this
dimension is that, for the most part, the user is not aware that
most of this infrastructure exists until it fails.33 For example, in
2002, the Beth Israel Deaconess Medical Center in Boston
experienced a 4-day computer outage due to old, out-of-date
computer equipment coupled with an outdated software
programme designed to direct traffic on a much less complex
network. Furthermore, their network diagnostic tools were
ineffective because they could be used only when the network
was functioning.34
Clinical content
This dimension includes everything on the data-information-
knowledge continuum that is stored in the system (ie, structured
and unstructured textual or numeric data and images that are
either captured directly from imaging devices or scanned from
paper-based sources).35 Clinical content elements can be used to
configure certain software requirements. Examples include
controlled vocabulary items that are selected from a list while
ordering a medication or a diagnostic test, and the logic required
to generate an alert for certain types of medication interactions.
These elements may also describe certain clinical aspects of the
i70
patients’ condition (eg, laboratory test results, discharge
summaries or radiographic images). Other clinical content, such
as demographic data and patient location, can be used to manage
administrative aspects of a patient’s care. These data can be
entered (or created), read, modified or deleted by authorised users
and stored either on the local computer or on a network. Certain
elements of the clinical content, such as that which informs CDS
interventions, must be managed on a regular basis.36
Humanecomputer interface
An interface enables unrelated entities to interact with the
system and includes aspects of the system that users can see,
touch or hear. The hardware and software ‘operationalise’ the
user interface; provided these are functioning as designed, any
problems with using the system are likely due to HCI issues.
The HCI is guided by a user interaction model created by the
software designer and developer.37 During early pilot testing of
the application in the target clinical environment, both the
user’s workflow and the interface are likely to need revisions.
This process of iterative refinement, wherein both the user and
user interface may need to change, must culminate in
a humanecomputer interaction model that matches the user’s
modified clinical workflow. For example, if a clinician wants to
change the dose of a medication, the software requires the
clinician to discontinue the old order and enter a new one, but
the user interface should hide this complexity. This dimension
also includes the ergonomic aspects of the interface.38 If users
are forced to use a computer mouse while standing, they may
have difficulty controlling the pointer on the screen because they
are moving the mouse using the large muscles of their shoulder
rather than the smaller muscles in the forearm. Finally, the lack
of a feature or function within the interface represents
a problem with both the interface and the software or hardware
that implements the interface.
People
This dimension represents the humans (eg, software developers,
system configuration and training personnel, clinicians and
patients) involved in all aspects of the design, development,
implementation and use of HIT. It also includes the ways that
systems help users think and make them feel.39 Although user
training is clearly an important component of the user portion
of the model, it may not by itself overcome all user-related
problems. Many ‘user’ problems actually result from poor
system design or errors in system development or configuration.
In addition to the users of these systems, this dimension
includes the people who design, develop, implement and eval-
uate these systems. For instance, these people must have the
proper knowledge, skills and training required to develop appli-
cations that are safe, effective and easy to use. This is the first
aspect of the model that is purely on the social end of the
sociotechnical spectrum.
In most cases, users will be clinicians or employees of the
health system. However, with recent advances in patient-
centred care and development of personal health record systems
and ‘home monitoring’ devices, patients are increasingly
becoming important users of HIT. Patients and/or their care
givers may not possess the knowledge or skills to manage new
health information technologies, and this is of specific concern
as more care shifts to the patient’s home.40
Workflow and communication
This is the first portion of the model that acknowledges that
people often need to work cohesively with others in the
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085

healthcare system to accomplish patient care. This collaboration
requires significant two-way communication. The workflow
dimension accounts for the steps needed to ensure that each
patient receives the care they need at the time they need it.
Often, the clinical information system does not initially match
the actual ‘clinical’workflow. In this case, either the workflow
must be modified to adapt to the HIT, or the HIT system must
change to match the various workflows identified.
Internal organisational policies, procedures and culture
The organisation’s internal structures, policies and procedures
affect every other dimension in our model. For example, the
organisation’s leadership allocates the capital budgets that
enable the purchase of hardware and software, and internal
policies influence whether and how offsite data backups are
accomplished. The organisational leaders and committees who
write and implement IT policies and procedures are responsible
for overseeing all aspects of HIT system procurement, imple-
mentation, use, monitoring and evaluation. A key aspect of any
HIT project is to ensure that the software accurately represents
and enforces, if applicable, organisational policies and proce-
dures. Likewise, it is also necessary to ensure that the actual
clinical workflow involved with operating these systems is
consistent with policies and procedures. Finally, internal rules
and regulations are often created in response to the external
rules and regulations that form the basis of the next dimension
of the model.
External rules, regulations and pressures
This dimension accounts for the external forces that facilitate or
place constraints on the design, development, implementation,
use and evaluation of HIT in the clinical setting. For example,
the recent passage of the American Recovery and Reinvestment
Act of 2009, which includes the Health Information Technology
for Economic and Clinical Health Act, makes available over $20
billion for healthcare practitioners who become ‘meaningful
users’ of health IT. Thus, the American Recovery and Rein-
vestment Act introduces the single largest financial incentive
ever to facilitate electronic health record (EHR) implementation.
Meanwhile, a host of federal, state and local regulations regulate
the use of HIT. Examples include the 1996 Health Insurance
Portability and Accountability Act, recent changes to the Stark
Laws and restrictions on secondary use of clinical data. Finally,
there are three recent national developments that have the
potential to affect the entire healthcare delivery system in
the context of HIT. These include: (1) the initiative to develop
the data and information exchange capacity to create a national
health information network41; (2) the initiative to enable
patients to access copies of the clinical data via personal health
records42; and (3) clinical and IT workforce shortages.43
System measurement and monitoring
This dimension has largely been unaccounted for in previous
models. We posit that the effects of HIT must be measured and
monitored on a regular basis. An effective system measurement
and monitoring programme must address four key issues related
to HIT features and functions.44 First is the issue of availabili-
tydthe extent to which features and functions are available and
ready for use. Measures of system availability include response
times and percentage uptime of the system. A second
measurement objective is to determine how the various features
and functions are being used by clinicians. For instance, one such
measure is the rate at which clinicians over-ride CDS warnings
and alerts. Third, the effectiveness of the system on healthcare
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085
Original research
delivery and patient health should be monitored to ensure that
anticipated outcomes are achieved. For example, the mean
HbA1c value for all patients with diabetes in a practice may be
measured before and after implementation of a system with
advanced CDS features. Finally, in addition to measuring the
expected outcomes of HIT implementation, it is also vital to
identify and document unintended consequences that manifest
themselves following use of these systems.45 For instance, it
may be worth while to track practitioner efficiency before and
after implementation of a new clinical charting application.46 In
addition to measuring the use and effectiveness of HIT at the
local level, we must develop the methods to measure and
monitor these systems and assess the quality of care resulting
from their use on a state, regional or even national level.47 48
Relationships and interactions between our model’s components
Our research and experience have led us, and others, to conclude
that HIT-enabled healthcare systems are best treated as complex
adaptive systems.49 The most important result of this conclu-
sion is that hierarchical decomposition (ie, breaking a complex
system, process or device down into its components, studying
them and then integrating the results in an attempt to under-
stand how the complete system functions) cannot be used to
study HIT.50 As illustrated by the evaluation of centrally stored
electronic summaries in the UK, complex interdependencies
between various sociotechnical dimensions are to be expected,
and our HIT model (had it existed at the time) might have
potentially predicted some of them and allowed them to address
them prior to go-live rather than in the evaluation stages of the
project. Therefore, one should not view or use our model as a set
of independent components which can be studied in isolation
and then synthesised to develop a realistic picture of how HIT is
used within the complex adaptive healthcare system. Rather, the
key to our model is how the eight dimensions interact and
depend on one another. They must be studied as multiple,
interacting components with non-linear, emergent, dynamic
behaviour (ie, small changes in one aspect of the system lead to
small changes in other parts of the system under some condi-
tions, but large changes at other times) that often appears
random or chaotic. This is typical of complex adaptive systems,
and our model reflects these interactions.
For example, a computer-based provider order entry (CPOE)
system that works successfully on an adult, surgical nursing unit
within a hospital may not work at all in the nearby paediatric
unit for any number of potential reasons, including: (1) hard-
ware/software (eg, fewer computers, older computers, poor
wireless reception, poor placement); (2) content (eg, no weight-
or age-based dosing, no customised order sets or documentation
templates); (3) user interface (eg, older workforce that has
trouble seeing the small font on the screen); or (4) personnel (eg,
no clinical champion within the medical staff). However, each of
these dimensions has a potential relationship with one or more
of the other dimensions. For instance, computers may have been
few or old because of some organisational limitations, there may
be no customised order sets because clinician-users did not agree
on how best to do it, and there was no clinical champion
because the organisation did not provide any incentive for the
additional time this role would entail. Other reasons could
include problems with the user interface, and the communica-
tion and workflow related to how nurses process new medica-
tion orders using the EHR and record administration of
medications. These issues, in turn, may have been due to
organisational policies and procedures. For example, the unit
governance committee may have decided not to approve
i71
 Original research

a request for mobile computers, with the result that nurses
spent more time away from patients and therefore had a slower
workflow related to processing new orders. The preceding
example illustrates the interaction of six dimensions of our
model: hardware/software, clinical content, user interface,
people, workflow and organisational policies. Additionally, some
form of monitoring could have detected these issues. In
summary, our model provides HIT researchers with several new
avenues of thinking about key technology components and how
these dimensions can be accounted for in future research.
New HIT model in action in real-world settings
The following sections illustrate how we have used the socio-
technical model of safe and effective HIT use within our
research. In an attempt to describe how the model can be
applied across the breadth of HIT research and development, and
to provide examples of different systems and interventions that
can be analysed within this new paradigm, we highlight key
elements of our model in the context of several recent projects.
HIT design and development
The design and development of CDS interventions within
clinicians’ workflow present several challenges. We conducted
several qualitative studies to gain insight into the eight dimen-
sions of our model during the development of a CDS tool within
a CPOE application. This CDS intervention was designed to
alert clinicians whenever they attempted to order a medication
that was contraindicated in elderly patients or one that had
known serious interactions with warfarin. We used several
methods, including focus groups, usability testing and educa-
tional sessions with clinician users,51 to identify issues related to
hardware/software, content, interface, people, measurement,
workflow/communication, and internal policies and procedures.
These efforts helped us, for example, to understand the need to
meet with the organisation’s Pharmacy and Therapeutics
(P & T) committee (ie, internal policy) to convince them to
modify the medication formulary as well as the information
technology professional (ie, people) who was responsible for
maintaining the textual content of the alerts (ie, font size,
contents and order of the messages) to fit within the constraints
of the alert notification window (ie, user interface) which elimi-
nated the need to train clinicians to use the horizontal scrolling
capability. This is just one simple example of how use of the
eight-dimensional model paid huge dividends during the devel-
opment and implementation stages of this highly successful
project.52 53
HIT implementation
In a recent article, we described lessons that could be learnt from
CPOE implementation at another site.54 One of the most

Table 1 Illustration of how the eight dimensions of our sociotechnical model have been used to analyse different health information technology-
related interventions and how other dimensions might need to be addressed for every dimension
Sociotechnical model dimension
Hardware and software
Clinical content
Humanecomputer interface
People
Workflow and communication
Organisational policies and procedures
External rules, regulations, and pressures
System measurement and monitoring
CPOE, computer-based provider order entry; EMR, Electronic Medical Record; IOM, Institute of Medicine; PCP, Primary Care Physician.
Lessons learnt from implementation of CPOE
The majority of computer terminals were linked to the hospital
computer system via wireless signal, and communication
bandwidth was often exceeded during peak operational
periods, which created additional delays between each click on
the computer mouse
No intensive-care-unit-specific order sets were available at the
time of CPOE implementation. The hurried implementation
timeline established by the leaders in the organisation
prohibited their development.
The process of entering orders often required an average of 10
clicks on the computer mouse per order, which translated to
1e2 min to enter a single order. Organisational leaders
eventually hired additional clinicians to ‘work the CPOE system’
while others cared for the patients.
Leaders at all levels of the institution made implementation
decisions (re: hardware placement, software configuration,
content development, user interface design, etc) that placed
patient care in jeopardy
Rapid implementation timeline did not allow time for clinicians
to adapt to their new routines and responsibilities. In addition,
poor hardware and software design and configuration decisions
complicated the workflow issues.
Order entry was not allowed until after the patient had
physically arrived at the hospital and been fully registered into
the clinical information system
Following the IOM’s report ‘To Err is Human: Building a Safer
Health System’ and subsequent congressional hearings, the
issue of patient safety has risen to a position of highest priority
among healthcare organisations
Monitoring identified a significant increase in patient mortality
following CPOE implementation
Follow-up of alerts related to abnormal diagnostic imaging
results
Alerts should be retracted when the patient dies, the radiologist
calls or the patient is admitted before the alert is
acknowledged. However, this can be done only through
a centralised organisational policy.
Interventions to reduce alert overload and improve the signal to
noise ratio should be explored. Unnecessary alerts should be
minimised. However, people (physicians) may not agree as to
which alerts are essential and which ones are not.55
Unacknowledged alerts must stay active on the EMR screen for
longer periods, perhaps even indefinitely, and should require
the provider’s signature and statement of action before they are
allowed to drop off the screen. However, providers might not
want to spend additional time stating their actions; who will
make this decision?
Many clinicians did not know how to use many of the EMR’s
advanced features that greatly facilitated the processing of
alerts so training should be revamped. However, providers are
only given 4 h of training time by the institution
Communicating alerts to two recipients, which occurred when
tests were ordered by a healthcare practitioner other than the
patient’s regular PCP, significantly increased the odds that the
alert would not be read and would not receive timely follow-up
action. No policy was available that stated who was
responsible for follow-up.
Every institution must develop and publicise a policy regarding
who is responsible (PCP vs the ordering provider, who may be
a consultant) for taking action on abnormal results. Also meets
External Joint commission requirements.
Poor reimbursement and heavy workload of patients put
productivity pressure on providers. The nature of high-risk
transitions between healthcare practitioners, settings and
systems of care makes timely and effective electronic
communication particularly challenging.
An audit and performance feedback system should be
established to give providers information on timely follow-up of
patients’ test results on a regular basis. However, providers
may not want feedback, or the institution might not have the
persons required to do so.

i72 Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085


important conclusions from this implementation was that
problems could, and often do, occur in all eight dimensions of
the model (see table 1).56
HIT use
Safe and effective use of an EHR-based notification system
involves many factors that are addressed by almost all dimen-
sions of our model.57 58 This CDS system generates automated
asynchronous ‘alerts’ to notify clinicians of important clinical
findings. We examined communication outcomes of over 2500
such alerts that were specifically related to abnormal test results.
We found that 18.1% of abnormal lab alerts and 10.2% of
abnormal imaging alerts were never acknowledged (ie, were
unread by the receiving provider). Additionally, 7e8% of these
alerts lacked timely follow-up, which was unrelated to
acknowledgement of the alert.
Despite a notification system that ensured transmission of
results, it was concerning that abnormal test results did not
always receive timely follow-up, even when acknowledged. This
study revealed complex interactions between users, the user
interface, software, content, workflow/communication and
organisational policies related to who was responsible for
abnormal test follow-up. Our findings thus highlighted the
multiple dimensions of our model that need to be addressed to
improve the safety of EHR-based notification systems and
perhaps other forms of CDS (see table 1).59e62 We are now
applying the sociotechnical model to study barriers, facilitators
and interventions for safe and effective test result notification
through EHRs.
HIT evaluation
Our model recently provided us with guidance in HIT evalua-
tion, reminding us that however technologically savvy we make
our patient care processes, we must also monitor their impact,
effectiveness and unintended consequences carefully. We
recently evaluated why, despite implementation of an auto-
mated notification system to enhance communication of fecal
occult blood test results, providers did not take follow-up
actions in almost 40% of cases.63 Again, our findings highlighted
multiple dimensions corresponding to our sociotechnical model.
For instance, we found that clinician non-response to automated
notifications was related to a software configuration error that
prevented transmission of a subset of test results but we also
found that if the institution was using certain types of work-
flows related to test performance and that if organisational
procedures for computerised order-entry of fecal occult blood
tests were different, the problem may not have occurred. Thus,
we found our multidimensional approach, which accounted for
interactions, to be useful for comprehensive evaluation of HIT
after implementation.
CONCLUSIONS
The eight dimensions of the safe and effective HIT use model
introduced in this manuscript establish a new paradigm for the
study of HIT. We have successfully applied this model to study
several HIT interventions at different levels of design, develop-
ment, implementation, use and evaluation. We anticipate that
additional study of the eight dimensions and their complex inter-
actions will yield further refinements to this model and, ultimately,
improvements in the quality and safety of the HIT applications
that translate to better health and welfare for our patients.
Acknowledgements We thank D Espadas and A Esquivel, for their help in creating
the graphical depiction of the model, the reviewers of this paper, for their
constructive criticism, and A Bradford, for editorial assistance.
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085
Original research
Funding This research was supported in part by the National Library of Medicine
R01-LM006942, NIH K23 career development award (K23CA125585), the VA National
Center of Patient Safety, Agency for Health Care Research and Quality and in part by
the Houston VA HSR&D Center of Excellence (HFP90-020).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Beuscart-Zéphir MC, Aarts J, Elkin P. Human factors engineering for healthcare IT
clinical applications. Int J Med Inform 2010;79:223e4.
2. Holden RJ, Karsh B. A theoretical model of health information technology usage
behaviour with implications for patient safety. Behav Inf Technol 2009;28:21e38.
3. Rogers EM. Diffusion of Innovations. 5th edn. Free Press, New York, NY. 2003.
4. Ash J. Organizational factors that influence information technology diffusion in
academic health sciences centers. J Am Med Inform Assoc 1997;4:102e11.
5. Gosling AS, Westbrook JI, Braithwaite J. Clinical team functioning and IT
innovation: a study of the diffusion of a point-of-care online evidence system. J Am
Med Inform Assoc 2003;10:244e51.
6. Venkatesh V, Morris MG, Davis FD, et al. User acceptance of information
technology: toward a unified view. MIS Quarterly 2003;27:425e78.
7. Holden RJ, Karsh BT. The technology acceptance model: its past and its future in
health care. J Biomed Inform 2010;43:159e72.
8. Duyck P, Pynoo B, Devolder P, et al. User acceptance of a picture archiving and
communication system. Applying the unified theory of acceptance and use of
technology in a radiological setting. Methods Inf Med 2008;47:149e56.
9. Kijsanayotin B, Pannarunothai S, Speedie SM. Factors influencing health
information technology adoption in Thailand’s community health centers: applying the
UTAUT model. Int J Med Inform 2009;78:404e16.
10. Hutchins E. Cognition in the Wild. Cambridge, MA: MIT Press, 1996:401.
11. Hazlehurst B, McMullen C, Gorman P, et al. How the ICU follows orders: care
delivery as a complex activity system. AMIA Annu Symp Proc 2003:284e8.
12. Cohen T, Blatter B, Almeida C, et al. A cognitive blueprint of collaboration in context:
distributed cognition in the psychiatric emergency department. Artif Intell Med
2006;37:73e83.
13. Hazlehurst B, McMullen CK, Gorman PN. Distributed cognition in the heart room:
how situation awareness arises from coordinated communications during cardiac
surgery. J Biomed Inform 2007;40:539e51.
14. Patel VL, Zhang J, Yoskowitz NA, et al. Translational cognition for decision support
in critical care environments: a review. J Biomed Inform 2008;41:413e31.
15. Reason J. Human error: models and management. BMJ 2000;320:768e70.
16. van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in
computerized physician order entry. J Am Med Inform Assoc 2006;13:138e47.
17. Lederman RM, Parkes C. Systems failure in hospitalseusing Reason’s model to
predict problems in a prescribing information system. J Med Syst 2005;29:33e43.
18. Norman D. The Psychology of Everyday Things. New York: Basic Books, 1988.
19. Malhotra S, Jordan D, Shortliffe E, et al. Workflow modeling in critical care: piecing
together your own puzzle. J Biomed Inform 2007;40:81e92.
20. Sheehan B, Kaufman D, Stetson P, et al. Cognitive analysis of decision support for
antibiotic prescribing at the point of ordering in a neonatal intensive care unit. AMIA
Annu Symp Proc 2009;2009:584e8.
21. Henriksen K, Kaye R, Morisseau D. Industrial ergonomic factors in the radiation
oncology therapy environment. In: Nielsen R, Jorgensen K, eds. Advances in
Industrial Ergonomics and Safety V. Washington, DC: Taylor & Francis, 1993:325e35.
22. Vincent C, Taylor-Adams S, Stanhope N. Framework for analysing risk and safety in
clinical medicine. BMJ 1998;316:1154e7.
23. Carayon P, Schoofs Hundt A, Karsh BT, et al. Work system design for patient
safety: the SEIPS model. Qual Saf Health Care 2006;15(Suppl 1):i50e8.
24. Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information
technologies in health careean interactive sociotechnical analysis. J Am Med Inform
Assoc 2007;14:542e9.
25. Rector AL. Clinical terminology: why is it so hard? Meth Inf Med 1999;38:239e52.
26. Rosenbloom ST, Miller RA, Johnson KB, et al. Interface terminologies: facilitating
direct entry of clinical data into electronic health record systems. J Am Med Inform
Assoc 2006;13:277e88.
27. Wright A, Sittig DF, Ash JS, et al. Governance for clinical decision support: case
studies and best practices of exemplary institutions. J Amer Med Inform Assoc 2010
(under review).
28. Sittig DF, Campbell EM, Guappone KP, et al. Recommendations for monitoring and
evaluation of in-patient computer-based provider order Entry systems: results of
a Delphi Survey. AMIA Annu Symp Proc 2007:671e5.
29. Sittig DF, Simonaitis L, Carpenter JD, et al. The state of the art in clinical knowledge
management: An inventory of tools and techniques. Int J Med Inform
2010;79:44e57.
30. Hripcsak G. Monitoring the monitor: automated statistical tracking of a clinical
event monitor. Comput Biomed Res 1993;26:449e66.
31. Rasmussen J. Risk management in a dynamic society: a modelling problem. Saf Sci
1997;27:183e213.
32. Greenhalgh T, Stramer K, Bratan T, et al. Adoption and non-adoption of a shared
electronic summary record in England: a mixed-method case study. BMJ
2010;340:c3111.
i73
 Original research
33. Leveson NG, Turner CS. An Investigation of the Therac-25 Accidents. IEEE Comput
1993;26:18e41. http://sunnyday.mit.edu/papers/therac.pdf.
34. Kilbridge P. Computer crashelessons from a system failure. N Engl J Med
2003;348:881e2.
35. Bernstam EV, Smith JW, Johnson TR. What is biomedical informatics? J Biomed
Inform 2010;43:104e10.
36. Sittig DF, Wright A, Simonaitis L, et al. The state of the art in clinical knowledge
management: an inventory of tools and techniques. Int J Med Inform
2010;79:44e57.
37. Shneiderman B, Plaisant C, Cohen M, et al. Designing the User Interface: Strategies
for Effective HumaneComputer Interaction. 5th edn. Pearson Educaiton, New Jersey.
2009:672 Pgs.
38. Svanæs D, Alsos OA, Dahl Y. Usability testing of mobile ICT for clinical
settings: Methodological and practical challenges. Int J Med Inform 2010;79:
e24e34.
39. Sittig DF, Krall M, Kaalaas-Sittig J, et al. Emotional aspects of computer-
based provider order entry: a qualitative study. J Am Med Inform Assoc
2005;12:561e7.
40. Henriksen K, Joseph A, Zayas-Caban T. The human factors of home health care:
a conceptual model for examining safety and quality concerns. J Patient Saf
2009;5:229e36.
41. American Recovery and Reinvestment Act of 2009, State Grants to Promote Health
Information Technology Planning and Implementation Projects. http://healthit.hhs.
gov/portal/server.pt/document/880499/foa_state_health_information_exchange_
cooperative_agreement_program_doc (accessed 20 August 2010).
42. Sittig DF. Personal health records on the internet: a snapshot of the pioneers at the
end of the 20th Century. Int J Med Inform 2002;65:1e6.
43. Detmer DE, Munger BS, Lehmann CU. Medical informatics board certification:
history, current status, and predicted impact on the medical informatics workforce.
Applied Clinical Informatics 2010;1:11e18. http://www.schattauer.de/nc/en/
magazine/subject-areas/journals-a-z/applied-clinical-informatics/issue/special/
manuscript/12624/download.html.
44. Leonard KJ, Sittig DF. Improving information technology adoption and
implementation through the identification of appropriate benefits: creating IMPROVE-
IT. J Med Internet Res 2007;9:e9.
45. Ash JS, Berg M, Coiera E. Some unintended consequences of information
technology in health care: the nature of patient care information system-related
errors. J Am Med Inform Assoc 2004;11:104e12.
46. Bradshaw KE, Sittig DF, Gardner RM, et al. Computer-based data entry for nurses in
the ICU. MD Comput 1989;6:274e80.
47. Sittig DF, Shiffman RN, Leonard K, et al. A draft framework for measuring progress
towards the development of a National Health Information Infrastructure. BMC Med
Inform Decis Mak 2005;5:14.
i74
48. Sittig DF, Classen DC. Safe electronic health record use requires a comprehensive
monitoring and evaluation framework. JAMA 2010;303:450e1.
49. Begun JW, Zimmerman B, Dooley K. Health care organizations as complex adaptive
systems. In: Mick SM, Wyttenbach M, eds. Advances in Health Care Organization
Theory. San Francisco: Jossey-Bass, 2003:253e88.
50. Rouse WB. Health care as a complex adaptive system: implications for design and
management. The Bridge, National Academy of Engineering, Washington, DC.
2008:17e25.
51. Feldstein A, Simon SR, Schneider J, et al. How to design computerized alerts to
safe prescribing practices. Jt Comm J Qual Saf 2004;30:602e13.
52. Feldstein AC, Smith DH, Perrin N, et al. Reducing warfarin medication interactions:
an interrupted time series evaluation. Arch Intern Med 2006;166:1009e15.
53. Smith DH, Perrin N, Feldstein A, et al. The impact of prescribing safety alerts for
elderly persons in an electronic medical record: an interrupted time series evaluation.
Arch Intern Med 2006;166:1098e104.
54. Sittig DF, Ash JS, Zhang J, et al. Lessons from ‘Unexpected increased mortality
after implementation of a commercially sold computerized physician order entry
system.’ Pediatrics 2006;118:797e801.
55. van der Sijs H, Aarts J, van Gelder T, et al. Turning off frequently overridden drug
alerts: limited opportunities for doing it safely. J Am Med Inform Assoc
2008;15:439e48.
56. Sittig DF, Ash JS. Clinical Information Systems: Overcoming Adverse Consequences.
Sudbury, MA: Jones & Bartlett, 2010.
57. Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging
test results in an outpatient setting: are electronic medical records achieving their
potential? Arch Intern Med 2009;169:1578e86.
58. Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal laboratory test results
in an electronic medical record: do any safety concerns remain? Am J Med
2010;123:238e44.
59. Hysong SJ, Sawhney MK, Wilson L, et al. Improving outpatient safety through
effective electronic communication: a study protocol. Implement Sci 2009;4:62.
PMID:19781075.
60. Hysong SJ, Sawhney MK, Wilson L, et al. Provider management strategies of
abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc
2010;17:71e7. PMID:20064805.
61. Singh H, Wilson L, Reis B, et al. Ten strategies to improve management of
abnormal test result alerts in the electronic health record. J Patient Saf
2010;6:121e3.
62. Singh H, Vij M. Eight recommendations for policies for communication of abnormal
test results. Jt Comm J Qual Patient Saf 2010;36:226e32.
63. Singh H, Wilson L, Petersen LA, et al. Improving Follow-up of Abnormal Cancer
Screens using Electronic Health Records: Trust but Verify Test Result Communication.
BMC Med Inform Decis Mak 2009;9:49.
Qual Saf Health Care 2010;19(Suppl 3):i68ei74. doi:10.1136/qshc.2010.042085

University of Stavanger,
Stavanger, Norway
Correspondence to
Professor Karina Aase,
Department of Health Studies,
University of Stavanger, Faculty
of Social Sciences, Stavanger
N-4036, Norway;
karina.aase@uis.no
Accepted 20 January 2010
Qual Saf Health Care 2010;19(Suppl 3):i75ei79. doi:10.1136/qshc.2009.036558
A comparative study of safety climate differences
in healthcare and the petroleum industry
Espen Olsen, Karina Aase
ABSTRACT
Aim The aim of this article is to compare safety climate in
healthcare and the petroleum industry by collecting
empirical evidence of differences between the two
sectors.
Methods The Hospital Survey on Patient Safety Culture
(HSOPSC) is used to measure the safety climate in two
organisations operating in the two different sectors: (1)
a large Norwegian university hospital offering a wide
range of hospital services and (2) a large Norwegian
petroleum company producing oil and gas worldwide.
Results and discussion Statistical analyses supported
the expected hypotheses that safety climate is positively
related to outcome measures and that the level on safety
climate and outcome measures are generally higher in
the petroleum sector. Empirical findings indicate that
healthcare should learn from the petroleum industry
regarding safety improvement efforts, and the
implication of this is discussed in the paper.
INTRODUCTION
Safety climate generally has the potential to vary
within1 as well as between organisations2 and
sectors.3 Safety climate features used in healthcare are
most often adapted from other industries. This
research trend indicates that cross-industrial overlaps
exist with regard to safety-related concepts and
challenges. Previous research has suggested that the
safety climate level is less ‘ideal’ in hospitals compared
with the aviation sector,3 4 but there is a lack of
studies assessing safety climate across sectors.
The aim of the present study is to investigate
differences in safety climate between employees in
healthcare and the petroleum industry. The
empirical data used in the study stem from two
parallel research projects: one patient safety study
of a large Norwegian university hospital and one
safety programme study of an international petro-
leum company with head offices in Norway. Safety
climate assessments were conducted as part of both
projects, making it possible to conduct comparisons
between the two sectors.
SAFETY CLIMATE DIFFERENCES
Even though today the petroleum industry is
generally considered to be safe,5 this has not always
been the case. Haukelid described the evolution of
safety through four stages in the petroleum
industry (Haukelid;6 see also Hudson7). The first
stage (1966e1980) was characterised by a macho
culture with severe accident rates; the second stage
(the 1980s) was characterised by a great change
through new technology, committed leadership and
employee participation; in the third stage (the
Original research
1990s) attention was directed towards safety-
management systems; while the fourth and
subsequent stage (2000epresent) has focused on
improving safety culture. In Norway, the Petroleum
Safety Authority in 2001 implemented a new
regulation stating that companies operating on the
Norwegian shelf should encourage and promote
a sound culture for health, safety and environment.
Today, it is common within many companies in the
Norwegian petroleum industry to implement
safety programmes with the aim of improving
safety culture among workers. This is based on
a common understanding that human behaviour is
crucial in achieving the goal of zero accidents.
The current petroleum company included in this
study has additionally invested extensively in
safety activities for a long period of time. The
overall aim of these activities has been to improve
safety behaviour and safety culture by introducing
different safety interventions.
Compared with the petroleum industry, health-
care has lacked the same attention towards safety
systems and safety culture. Several studies8e11
indicate unacceptable levels of adverse events
(2.9e16.6% of patient admissions), and several
governmental health agencies have expressed
concerns regarding patient safety.12 13 The concerns
have been agreed upon by researchers, stating that
for instance hospital systems are designed to rely
on the error-free performance of individuals.14 If
this is the case, the attitude is substantially
different from the petroleum industry where
human error is considered inevitable, and risks are
understood as combinations of active failures and
latent conditions.15 Two studies have compared
safety climate dimensions in healthcare and avia-
tion. Both studies found the safety climate levels to
be lower among healthcare workers than among
aviation workers. Sexton et al concluded that
medical staff found it hard to discuss errors and
that errors were not handled appropriately among
hospital workers.4 Gaba et al reported that
perceived safety problem areas among hospital
workers were up to 12 times the amount perceived
by aviation workers within certain areas.3 Based on
the above, it would be reasonable to expect that
workers in the petroleum industry perceive the
safety climate level to be higher than workers
within healthcare: employees’ perceptions of safety
climate are higher among workers within the
petroleum industry than among workers within
the healthcare sector (Hypothesis 1).
SAFETY CLIMATE OUTCOMES
The safety climate concept was traditionally
developed to explain additional variation in safety
i75
 Original research
related outcomes.16 Associations between safety climate and
safety performances have been documented both directly and
indirectly.
Griffin and Neal illustrated how workers’ knowledge, skills
and motivation mediated the effect of safety climate on safety
performance.17 Later studies have also illustrated the indirect
effects of safety climate.18 19 In the theoretical models developed
by Zohar1 and Flin,5 they proposed that the effect of safety
climate on safety outcomes is mediated through individual
workers’ motivation and behaviour. In other words; safety
motivation and behaviour, as well as lower accidents levels, can
be expected outcomes of a good safety climate. The link
between safety climate and self-reported safety behaviour has
been supported by DeJoy et al, and more recently Johnson
demonstrated that safety climate predicted accidents in
a manufacturing organisation.20 21 Based on the above, it would
be reasonable to expect a positive relation between workers’
perception of safety climate and desirable safety outcomes both
in healthcare and in the petroleum industry. Since perceived
safety climate is expected to be higher within the petroleum
industry than within healthcare (Hypothesis 1), it is further-
more reasonable to expect that the perceived level of safety
climate outcomes is higher in the petroleum industry than in
healthcare: safety climate dimensions positively influence
workers’ perception of safety outcomes both in petroleum and
in healthcare (Hypothesis 2).
METHOD
Instrument
A review of available safety climate instruments was conducted
to investigate the factorial structure and psychometric proper-
ties of instruments. The Hospital Survey on Patient Safety
Culture (HSOPSC) was selected mainly because the dimen-
sionality covered general topics revealed as important in
a broader patient safety study,22 and because studies show that
HSOPSC has met more psychometric criteria compared with
other instruments.23 The dimensionality of HSOPSC is
conventional, measuring several dimensions that are typical
within the safety theory of both healthcare and other indus-
tries.23 All items in HSOPSC are rated on Likert-type scales with
verbal anchors. ‘Number of events reported (last 12 months)’ is
measured on a scale from 1 to 6; all other concepts are measured
on scales from 1 to 5 (more details on HSOPSC at http://www.
ahrq.gov/qual/hospculture).
HSOPSC was validated on hospital data before distribution to
the petroleum sample.24 Some limitations of the instruments
were revealed: ‘Number of events reported (last 12 months)’ and
‘Frequency of event reporting’ did not function well as outcome
variables, and were therefore not included in the study.
The original version of HSOPSC was translated into Norwe-
gian before distribution to the hospital sample.24 Some adapta-
tions of the HSOPSC instrument were conducted before
distribution to the petroleum sample. The term ‘patient’ was
removed from items in the instrument. For example, the item
measuring perceived safety outcome, ‘Give a general judgement
about the patient safety in your department’ (Pasient safety
grade), was rephrased into ‘Give a general judgement about the
safety in your department’ (Safety Grade) in the petroleum
sample. As a result of meetings with safety experts in the
petroleum company, nine items were removed in order to trim
the original version of HSOPSC to increase the workers’
understanding of the questionnaire. This resulted in removing
the outcome measure ‘Overall perceptions of safety.’ A new
outcome measure concerning the likelihood of an employee
i76
stopping to work in dangerous situations was added. This
measure was already included in the instrument used in the
hospital sample. The dimension ‘Stop working in dangerous
situations’ consisted of three items (‘I ask my colleagues to stop
work, that is, dangerously accomplished,’ ‘I notify if I see
dangerous situations’ and ‘I stop working if I consider the
situations to be dangerous for me or my colleagues’). Finally,
a ‘do not know’ category was added to seven items measuring
safety climate at the organisational level and to the ‘Safety
grade’ measure.
After adapting the HSOPSC instrument to the petroleum
sample, a total of 37 items could be used to analyse cross-
sectional differences.
Questionnaire samples
The target group in the hospital included all patient-related
employee categories and other personnel employed in the same
working environment. A total of 1919 workers answered the
survey at the hospital, resulting in a response rate of 55%. Of
these respondents, 89% had direct patient contact, whereas 62%
worked between 20 and 37 h per week.
For the petroleum company, 1806 workers answered the
survey resulting in a response rate of 52%. The sample includes
296 contract workers of four different subcontractors working
for the petroleum company. In the petroleum sample, 45% had
a job of administrative character, and 44% were employed in
jobs offshore.
Printed questionnaires in Norwegian were distributed to the
hospital sample. In the petroleum company, HSOPSC was
distributed electronically via email in both English and Norwe-
gian. In both samples, the respondents answered anonymously.
Statistical procedures
Data were analysed using SPSS 13.0 (SPSS, Chicago, Illinois).
The level of scores in the ‘do not know’ category was low, and
was therefore treated as missing values before the remaining
analyses were conducted. As explained by Sorra and Nieva, mean
scores for the dimensions were created after reversing the coding
for reverse items.25 To determine if factor scales yielded accept-
able a coefficients and internal consistency, the Cronbach a was
estimated. Multiple analysis of variance (MANOVA) was used
to test whether or not there was an overall difference in
employee perceptions of safety climate and safety outcomes. t
Test statistics were estimated to test if the mean differences
were significant for each measurement concept. Pearson r and
regression analyses were estimated separately for each sector to
investigate if the safety climate dimensions were positively
correlated with the outcome variables.
RESULTS
Internal consistency
The Cronbach a was estimated for both samples (table 1). The
a scores ranged from 0.39 to 0.82 in the petroleum sample, and
from 0.38 to 0.78 in the hospital sample. In the petroleum
sample, the lowest a score was estimated for the dimension
measuring Staffing (0.39), and likewise Teamwork across units
(0.38) was the dimension with the lowest score in the hospital
sample. In contrast to a high a score, lower scores indicate that
the dimension measures a wider domain.26 Since a scores are
sensitive to the number of items in the dimensions, this may
explain the scores lower than 0.70, since these are estimated
based on dimensions measured with only two or three items.27
The aim of this study was to explore the safety climate differ-
ences between two sectors. Based on this, no threshold for
Qual Saf Health Care 2010;19(Suppl 3):i75ei79. doi:10.1136/qshc.2009.036558
 Original research

Table 1 Internal consistency of measures an overall difference in the two outcome variables used in the
a study: Wilks l of 0.920 (df¼2), p<0.001, effect size¼0.080 (h2).
No of items
in scale Petroleum Hospital Mean differences and t tests are shown in table 2. These
analyses provide additional information to MANOVA because
Outcome measures
the t tests estimate whether differences between sectors are
Stop working in dangerous situations 3 0.80 0.63
significant for each measurement concept.
Safety culture dimensionsdunit level
The t-test analyses indicate that safety climate is higher in the
Supervisor/manager expectations and 4 0.77 0.77
actions promoting safety petroleum industry, with the exception of two dimensions: the
Organisational learningdcontinuous 3 0.59 0.51 difference regarding ‘Communication openness’ was not signif-
improvement icant and contrary to what was expected, ‘Non-punitive
Teamwork within units 4 0.78 0.77 response to error’ has a higher score in the hospital sample. The
Communication openness 3 0.65 0.67 difference between the mean scores is highest for the dimension
Feedback and communication about error 3 0.79 0.70 ‘Organisational management support for (patient) safety’ (0.98),
Non-punitive response to error 3 0.66 0.64 followed by ‘Teamwork across units’ (0.43) and ‘Feedback and
Staffing 2 0.39 0.44 communication about errors’ (0.43).
Safety culture dimensionsdorganisational
It was also estimated whether workers in the petroleum
level
Organisational management support for 2 0.82 0.78
industry scored higher than workers in healthcare on two
(patient) safety measures used for measuring the safety level in general:
Teamwork across units 2 0.62 0.38 ‘Frequency of no harm reporting’ and ‘Number of events
Organisational handoffs and transitions 2 0.76 0.59 reported.’ On both measures, t test statistics revealed higher
scores in the petroleum industry than in healthcare.

Associations with outcome measures

a scores was defined for the dimensions, and all dimensions are
included in the remaining analyses.
Differences between samples
MANOVA and t test statistics were conducted to investigate the
differences in safety climate and safety outcomes between
workers in healthcare and the petroleum industry. MANOVA
revealed that there was an overall difference in the safety climate
between the two industries when the 10 dimensions were used
as dependent variables, and sector was defined as a dichotomised
(hospital vs petroleum) independent variable: Wilks l of 0.624
(df¼10), p<0.001, effect size¼0.376 (h2). Hence, the results
therefore generally support the hypothesis that safety climate
differs across the hospital and petroleum sample. MANOVA,
using the same dichotomised independent variable, also revealed
Regression analyses (table 3) and Pearson r were estimated to
investigate the association between the safety climate dimen-
sions and outcome measures. All correlations between the safety
climate dimensions and the two outcome measures were posi-
tively correlated (p<0.001, two-tailed), both in the total sample
and when the analyses were conducted separately for the two
sectors. In the hospital sample, the safety climate dimensions
were correlated between 0.11 and 0.32 with ‘Stop working in
dangerous situations,’ and between 0.22 and 0.45 with ‘Patient
safety grade.’ In the petroleum sample, the safety climate
correlations with ‘Stop working in dangerous situations’ varied
between 0.17 and 0.45 and similarly between 0.17 and 0.37 with
the ‘Safety grade’ dimension.
In the regression analyses, the ‘(Patient) safety grade’ and
‘Stop working in dangerous situations’ were defined as

Table 2 Descriptive statistics and mean differences between the sectors

Petroleum Hospital Petroleumehospital
Mean SD Mean SD Mean difference Significance
Outcome measures
Patient safety grade/safety grade 3.75 0.66 3.44 0.67 0.31 ***
Stop working in dangerous situations 4.14 0.55 3.90 0.57 0.24 ***
Safety culture dimensionsdunit level
Supervisor/manager expectations and 4.16 0.62 3.82 0.68 0.33 ***
actions promoting safety
Organisational learningdcontinuous 3.54 0.57 3.37 0.64 0.17 ***
improvement
Teamwork within units 3.90 0.57 3.84 0.60 0.06 **
Communication openness 3.72 0.59 3.71 0.65 0.01
Feedback and communication about 3.67 0.78 3.24 0.76 0.43 ***
errors
Non-punitive response to errors 3.64 0.75 3.81 0.65 -0.17 ***
Staffing 3.56 0.70 3.26 0.79 0.30 ***
Safety culture dimensionsdorganisational level
Organisational management support for 3.83 0.78 2.85 0.82 0.98 ***
(patient) safety
Teamwork across units 3.41 0.75 2.98 0.62 0.43 ***
Organisational handoffs and transitions 3.22 0.75 3.02 0.66 0.20 ***
Total 3.71 0.67 3.21 0.68
*p<0.05, **p<0.01, ***p<0.001.

Qual Saf Health Care 2010;19(Suppl 3):i75ei79. doi:10.1136/qshc.2009.036558 i77

 Original research
Table 3 Regression analyses with standardised b coefficients
Dependent variables
Stop working in dangerous situations
Hospital
Safety climate dimensionsdunit level
Supervisor/manager expectations and 0.01***
actions promoting safety
Organisational learningdcontinuous 0.10***
improvement
Teamwork within units 0.13***
Communication openness 0.11***
Feedback and communication about error 0.02
Non-punitive response to error 0.01
Staffing 0.03
Safety climate dimensionsdorganisational level
Organisational management support for 0.00
(patient) safety
Teamwork across units 0.02
Handoffs and transitions 0.06*
Explained variance (R2) 0.164
*p<0.05, **p<0.01, ***p<0.001.
dependent variables, and the safety climate dimensions as
independent variables. In the estimations, analyses were sepa-
rated for the two sectors. Except for one negative influence from
‘Non-punitive response to error,’ regression analyses generally
support the hypothesis that safety climate dimensions have
positive influences on workers perception of safety outcome
variables. The most consistent influences are in the hospital
sample where six out of 10 safety climate dimensions have
positive influences on ‘Patient safety grade,’ while five positive
influences were revealed in the petroleum sample on ‘Safety
grade.’ The safety climate dimensions are also generally posi-
tively related to ‘Stop working in dangerous situations’; five
significant positive relations in the hospital sample, and six
significant positive relations in the petroleum sample.
DISCUSSION
Generally, results support the usability of HSOPSC in the
petroleum sample after minor adjustments of the instrument.
With some exceptions, the Cronbach a scores are satisfactory
considering the low number of items for some of the measures.
Some precautions, however, should be made regarding
comparing dimensions with low a scores. Correlations in both
samples show a moderate association among the safety climate
dimensions, indicating a discriminant validity of measures.
As expected, results using MANOVA indicate an overall
difference in the level of safety climate and safety outcomes
between the two sectors. t Test statistics indicate a higher level
on all measures among petroleum workers (hypothesis 1), with
the exceptions of ‘Communication openness’ and ‘Non-punitive
response to errors.’ Generally, a low degree of missing responses
was not a problem in the study. Also, both samples are believed
to be representative for the target samples in the study. In sum,
the empirical data and statistical results show that data from
the two samples can be compared. With one significant excep-
tion, ‘Non-punitive response to errors,’ all the significant
b coefficients and all correlations are positively associated with
the outcome measures. This demonstrates that the safety
climate dimensions influence the outcome variables to a high
degree as expected (Hypothesis 2).
The study has shown that safety climate can be measured
using the same validated instrument (HSOPSC) across work
i78
Patient safety grade Safety grade
Petroleum Hospital Petroleum
0.05 0.16*** 0.21***
0.22*** 0.14*** 0.11***
0.23*** 0.07** 0.10***
0.11*** 0.08** 0.04
0.15*** 0.03 0.05
0.08** 0.01 0.01
0.03 0.20*** 0.07**
0.01 0.23*** 0.02
0.01 0.04 0.00
0.06* 0.01 0.07**
0314 0.389 0209
domains. So far, only a few studies have attempted to do so,3 4
even though the need for research to understand risk issues
across sectors has been raised.28 29 At the same time, certain
limitations of the study should be mentioned. First, only one
hospital represents the healthcare sector, while several compa-
nies represent the petroleum sector. Research has shown that
the safety climate may vary between organisations,2 and so the
samples in this study may not represent the two sectors in
general. Second, minor adjustments were conducted to make
HSOPSC suitable for the petroleum sample. Caution therefore
should be exercised when directly comparing the two sectors.
Third, the outcome measures used in the study are measured
with the same questionnaire as the safety climate dimensions,
limiting the criterion validity of the study. Finally, many reasons
for why ‘Non-punitive response to errors’ has a higher score in
healthcare than in the petroleum industry can be offered. One
explanation might be that a low safety focus in general influences
hospital workers’ perception of not feeling blamed. The inverse
relation estimated with regression analyses between ‘Non-puni-
tive response to errors’ and ‘Stop working in dangerous situations’
in the hospital sample gives an additional indication that caution
should be exercised in analysing the higher score in healthcare on
this dimension. Caution also should be taken when interpreting
dimensions with low scores on Cronbach a.
Our results have shown that the level of safety climate in the
hospital setting is less advanced than the petroleum industry.
The most substantial difference between the two sectors is
measured on the dimension ‘Organisational management
support for (patient) safety.’ Thus, top-level healthcare
managers could learn from petroleum managers how to put
patient safety on the organisational agenda. Other findings at
the organisational level document that the healthcare sector
could improve their capabilities on the dimensions ‘Teamwork
across units’ and ‘Organisational handoffs and transitions.’ As
a consequence, healthcare providers should consider introducing
improvement efforts at an overall organisational and strategic
level in addition to tactical and practical issues at the depart-
ment level, where department managers have a particular role in
the patient-safety improvement measures. Our results suggest
that healthcare could improve further on the following
dimensions: ‘Supervisor/manager expectations and actions
Qual Saf Health Care 2010;19(Suppl 3):i75ei79. doi:10.1136/qshc.2009.036558

promoting safety,’ ‘Feedback and communication about error,’
‘Organisational learningdcontinuous improvement’ and
‘Teamwork within units.’ Finally, compared with petroleum,
healthcare could improve the alignment between work tasks and
human resources. In addition to increasing patient safety, this
could potentially decrease stress and increase the general health
status of healthcare workers. Healthcare workers have better
scores on ‘Non-punitive response to error’ than petroleum
workers. Effort should be made to uphold this level when safety
improvements are introduced to improve the other safety
climate dimensions.
Even though the contextual settings and the types of risks are
highly different between hospitals and petroleum companies, it
is appropriate to suggest how the safety management systems in
the petroleum setting might possibly benefit healthcare. It is
important to note that the petroleum industry has a strong
regulatory regime that specifies very clearly the safety and risk
issues that should be addressed in the petroleum companies.
These regulatory requirements have influenced the petroleum
industry to take actions at different levels. An example is the
role of safety delegates in the petroleum industry of having
a substantial impact on work processes and giving direct input
to managers. Also included in the regulatory regime is the
inclusion of a safety culture formalisation in the Norwegian
Petroleum Safety Authority’s (PSA) regulations which stresses
the systematic development of a sound health, environment,
and safety culture in all petroleum activities (PSA 2002, The
Framework Regulations, Section 11). As indicated, this specific
demand in the regulatory framework has influenced petroleum
companies to take specific actions in order to address cultural
issues relating to health, environment and safety.28 One example
is the implementation of comprehensive safety programmes and
campaigns. In the petroleum company included in this study,
the safety programme involved 33 000 workers with a 3-year
implementation plan incorporating five specific strategies that
focused on: (1) correct prioritisation (taking the time needed to
work safely, ie, safety is more important than production), (2)
compliance (emphasises the importance of following procedures,
requirements, guidelines and decisions), (3) open dialogue
(enables employees to feel free to discuss safety issues with
colleagues and managers at any level, (4) continuous risk
assessment (taking the time to evaluate potential incidents and
accidents related to unexpected activities) and (5) caring about
colleagues (taking care of others and oneself when doing some-
thing that puts individuals at risk).30
The safety-management systems for preventing risk and
improving safety performance in the Norwegian petroleum
industry were developed over several decades. It is recognised
that a focus on patient safety is relatively new in healthcare.
Hence, we are encouraged by the opportunities, where appro-
priate, that exist for healthcare organisations to learn and to
develop truly pervasive safety cultures based on the highly
formalised programmes briefly described here.
Acknowledgements The authors would like to thank the employees in the two
sectors for participating in the survey.
Funding The study is partly funded by the Norwegian Research Council, the
participating hospital, and the petroleum company. The funding sources have had no
role in the development of this paper.
Competing interests None.
Qual Saf Health Care 2010;19(Suppl 3):i75ei79. doi:10.1136/qshc.2009.036558
View publication stats
Original research
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1. Zohar D. Safety climate: conceptual and measurement issues. In: Quick J, Tetrick
LE, eds. Handbook of occupational health psychology. Washington, DC: American
Psychological Association, 2003:123e42.
2. Singer SJ, Gaba DM, Geppert JJ, et al. The culture of safety: results of an
organization-wide survey in 15 California hospitals. Qual Saf Health Care
2003;12:112e18.
3. Gaba DM, Singer SJ, Sinaiko AD, et al. Differences in safety climate between
hospital personnel and naval aviators. Hum Factors 2003;45:173e85.
4. Sexton JB, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and
aviation: cross sectional surveys. BMJ 2000;320:745e9.
5. Flin R. Measuring safety culture in healthcare: a case for accurate diagnosis. Saf Sci
2007;45:653e67.
6. Haukelid K. Theories of (safety) culture revisiteddan anthropological approach. Saf
Sci 2008;46:413e26.
7. Hudson P. Implementing a safety culture in a major multi-national. Saf Sci
2007;45:697e722.
8. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence
in hospitalized patients: results of the Harvard Medical Practice Study. N Engl J Med
1991;324:370e7.
9. Mills D. Medical insurance feasibility study. A technical summary. West J Med
1978;128:360e5.
10. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events
and negligent care in Utah and Colorado. Med Care 2000;38:261e71.
11. Wilson RM, Runciman WB, Gibberd RW, et al. The quality in Australian health care
study. Med J Aust 1995;163:458e71.
12. Institute of Medicine. To err is humandbuilding a safer health system.
Washington, DC: National Academy Press, 1999.
13. Department of Health. Building a safer NHS for patients. London: HMSO, 2001.
14. Leape L, Wood D, Hatlie M, et al. Promoting patient safety by preventing medical
error. JAMA 1998;280:1444e7.
15. Reason J. Managing the risks of organizational accidents. Aldershot: Ashgate, 1997.
16. Hale AR, Hovden J. Management and culture: the third age of safety. A review of
approaches to organizational aspects of safety, health, and environment. In: Feyer
AM, Williamson A, eds. Occupational injury: risk, prevention and intervention. London:
Taylor & Francis, 1998:129e65.
17. Griffin MA, Neal A. Perceptions of safety at work: a framework for linking safety
climate to safety performance, knowledge, and motivation. J Occup Health Psychol
2000;5:347e58.
18. Neal A, Griffin MA. Safety climate and safety at work. In: Barling J, Frone M, eds.
The psychology of workplace safety. Washington, DC: American Psychological
Association, 2004.
19. Neal A, Griffin MA. A study of the lagged relationships among safety climate, safety
motivation, safety behavior, and accidents at the individual and group levels. J Appl
Psychol 2006;91:946e53.
20. DeJoy D, Murphy L, Gershon R. The influence of employee, job/task, and
organizational factors on adherence to universal precautions among nurses. Int J Ind
Ergon 1995;16:43e55.
21. Johnson SE. The predictive validity of safety climate. J Saf Res 2007;38:511e21.
22. Aase K, Høyland S, Olsen E, et al. Patient safety challenge in a case study
hospitaldof relevance for transfusion processes? Transfus Apher Sci
2008;39:167e72.
23. Flin R, Burns C, Mearns K, et al. Measuring safety climate in health care. Qual Saf
Health Care 2006;15:109e15.
24. Olsen E. Reliability and validity of the Hospital Survey on Patient Safety Culture at
a Norwegian hospital. In: Øvretveit J, Sousa P, eds. Quality and Safety
Improvement Research: methods and Research Practice from the International
Quality Improvement Research Network (QIRN). Lisbon: National School of Public
Health, 2008:173e86.
25. Sorra JS, Nieva VF. Hospital Survey on Patient Safety Culture. AHRQ Publication No.
04-0041, September. 2004. Rockville, MD, USA: Agency for Healthcare Research and
Quality, 2004. http://www.ahrq.gov/qual/patientsafetyculture/usergd.htm.
26. Netemeyer RG, Bearden WO, Sharma S. Scaling procedures: Issues and
application. London: SAGE Publications Ltd, 2003.
27. Schmidt N. Uses and abuses of coefficient alpha. Psychol Assess 1996;8:
350e3.
28. Olsen E. Safety culture and safety climate in health care and the petroleum industry:
psychometric quality, longitudinal change, and structural models. [PhD thesis]
Stavanger, Norway: University of Stavanger, No: 74.
29. Wiig S. Contributions to risk management in the public sector [PhD thesis].
Stavanger, Norway: University of Stavanger, No. 48.
30. Olsen E, Bjerkan AM, Nævestad TO. Modelling the effects of a large-scale safety
culture programme: a combined qualitative and quantitative approach. J Risk Res
2009;12:1e21.
i79