See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/248384747

A prospective, observational study of the effects of implementation strategy

on compliance with a surgical safety checklist

Article in BMJ quality & safety · July 2013

DOI: 10.1136/bmjqs-2012-001749 · Source: PubMed

CITATIONS READS

52 498

8 authors, including:

John A Windsor Kathleen Callaghan

University of Auckland University of Auckland
597 PUBLICATIONS 21,522 CITATIONS 20 PUBLICATIONS 155 CITATIONS

SEE PROFILE SEE PROFILE

Alan F Merry
University of Auckland
261 PUBLICATIONS 13,841 CITATIONS

SEE PROFILE

All content following this page was uploaded by John A Windsor on 01 July 2014.

The user has requested enhancement of the downloaded file.

 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
BMJ Quality & Safety Online First, published on 9 July 2013 as 10.1136/bmjqs-2012-001749
▸ Additional material is
published online only. To view
please visit the journal online
(http://dx.doi.org/10.1136/bmjqs-
2012-001749).
1
Department of Anaesthesiology,
Faculty of Medical and Health
Sciences, University of Auckland,
Auckland, New Zealand
2
Department of Surgery, Faculty
of Medical and Health Sciences,
University of Auckland,
Auckland, New Zealand
3
Auckland City Hospital,
Auckland, New Zealand
Correspondence to
Associate Professor Simon
Mitchell, Department of
Anaesthesiology, University of
Auckland, L12, Auckland City
Hospital, 2 Park Road, Grafton,
Auckland 1023, New Zealand;
sj.mitchell@auckland.ac.nz
Received 12 December 2012
Revised 12 May 2013
Accepted 15 June 2013
To cite: Hannam JA, Glass L,
Kwon J, et al. BMJ Qual Saf
Published Online First: [ please
include Day Month Year]
doi:10.1136/bmjqs-2012-
001749
Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
Copyright Article author (or their employer) 2013. Produced by BMJ Publishing Group Ltd under licence.
ORIGINAL RESEARCH
A prospective, observational study
of the effects of implementation
strategy on compliance with a
surgical safety checklist
J A Hannam,1 L Glass,1 J Kwon,2 J Windsor,2,3 F Stapelberg,1
K Callaghan,2 A F Merry,1,3 S J Mitchell1,3
ABSTRACT
Background The reported benefits of using the
WHO Surgical Safety Checklist (SSC) are likely to
depend on compliance with its correct use.
Compliance with SSC administration in centres
that have introduced the checklist under a
research protocol may differ from centres where
the SSC is introduced independently.
Objective To compare compliance with SSC
administration at an original WHO pilot study
centre (Hospital 1) with that at a similar
neighbouring hospital (Hospital 2) that
independently integrated the SSC with pre-
existing practice.
Methods This was a prospective, observational
study. One hundred operations were observed at
each hospital. We recorded: compliance with
administration of SSC domains (Sign In, Time
Out and Sign Out) and individual domain items;
timing of domain administration; and operating
room team engagement during administration.
Results Domain compliance at Hospital 1 and
Hospital 2, respectively, was: 96% and 31%
(p<0.0005) for Sign In; 99% and 48% (p<0.0005)
for Time Out and 22% and 9% (p=0.008) for
Sign Out. Engagement of two or more teams
during Sign In and Time Out occurred more
frequently at Hospital 2 than at Hospital 1.
Discussion Compliance with administration of
SSC domains was lower at Hospital 2 which
introduced the SSC outside the context of a strict
study protocol. This finding mandates caution in
extrapolation of benefits identified in SSC studies
to non-study hospitals. Staff engagement was
better at Hospital 2 where checklist administration
leadership is strategically shared among
anaesthetic, surgical and nursing team members
as compared with exclusive nursing leadership at
Hospital 1.
Study registry number Australian and New
Zealand Clinical Trials Registry: Ref:
ACTRN12612000135819, http://www.anzctr.org.
au/trial_view.aspx?ID=362007
INTRODUCTION
The WHO Surgical Safety Checklist
(SSC)1 is a set of prompts designed to
prevent errors or omissions in periopera-
tive care, and to promote communication
between operating room (OR) team
members. The SSC is comprised of three
domains, each consisting of a series of
items: ‘Sign In’ is administered when the
patient arrives in the OR; ‘Time Out’ is
administered just prior to first incision;
and ‘Sign Out’ is administered prior to the
patient leaving the OR. Administration of
domains is announced and designed to be
treated as a process to which all OR team
members give their undivided attention.
Studies investigating either the WHO
SSC1 or a suite of quality improvement
strategies containing a similar variant2
have demonstrated absolute reductions of
0.7%1 2 in mortality and of 4–5% in com-
plication rates.1 2
The potential benefits of using such a
tool are substantial given that over 230
million operations are performed globally
each year.3 It has been demonstrated that
at least some of the safety benefits of the
SSC are correlated to checklist compli-
ance.4 It is notable that data demonstrat-
ing checklist benefits1 2 were gathered in
highly supervised study environments in
which staff were trained to ensure levels
of compliance adequate for the evalu-
ation of checklist efficacy. However, the
quality of checklist administration at
these centres may not reflect that of hos-
pitals where the SSC was introduced
outside of a research setting. Maintaining
1
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Original research
checklist practice, even after an effective introduction,
is also an on-going challenge.5
Our institution was one of the original sites partici-
pating in the WHO SSC pilot study,1 and a modified
version of the original SSC has been used for 5 years.
Practices at this hospital may not be representative of
other centres that have since adopted the SSC but
were not involved in the WHO study. We assessed
compliance and engagement in SSC administration at
our institution, and compared this with another local
hospital of similar size and activity that introduced the
SSC independently. We hypothesised that if the cir-
cumstances under which the SSC is introduced to hos-
pital practice do not influence compliance and quality
in its administration, then there should be little or no
difference between these sites.
METHODS
This prospective, observational study was approved by
the Northern Y Regional Ethics Committee (ref:
NTY/10/EXP/077) and registered with the Australian
New Zealand Clinical Trials Registry (ref:
ACTRN12612000135819). Required consultative
processes were completed for institutional approval.
Data were collected concurrently at both sites between
November 2011 and January 2012 using methods
previously established by our group.6
Differences between sites and checklist modifications
The study hospitals were two large tertiary teaching
hospitals in Auckland City. Prior to roll-out of the
SSC, both hospitals used less comprehensive checks in
the perioperative period. They employ medical and
nursing staff from the same workforce pool with
common training and registration requirements.
Surgical and anaesthesia trainees rotate between the
institutions, and both hospitals receive medical stu-
dents from the same local university for training.
The first site (‘Hospital 1’) was a study centre for
the initial WHO SSC initiative in 2006.1 The SSC was
introduced to OR staff as part of participation
through a combination of iterative seminars and
written material. There was a sustained effort to estab-
lish and maintain compliance with the checklist, and
it was widely understood among OR staff that SSC
administration was being observed during the WHO
SSC study. SSC use became standard practice for all
surgical procedures following the study in 2008. The
second site (‘Hospital 2’) was not part of the initial
WHO SSC study, and use of the SSC was implemen-
ted independently in 2010 under the auspices of the
hospital’s surgical quality and safety committee. The
SSC was integrated into existing surgical time-out
practice and promoted through education seminars.
Compliance was not audited as part of a research
project as it was at Hospital 1.6
Both hospitals now use adaptations of the SSC
modified for their individual OR environments.
2 Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
Although largely comparable, some differences
between adaptations are notable. The adaptation used
at Hospital 2 allows Sign In administration in the pre-
anaesthetic room or the OR. It also specifies that Sign
In should be led by the anaesthetist, Time Out by the
surgeon and Sign Out by the circulating nurse. At
Hospital 1, all domains must be completed in the OR,
and the circulating nurse is responsible for initiating
and ensuring SSC completion. Full checklist adapta-
tions for Hospital 1 and Hospital 2, and a summary
of checklist differences, are given in online supple-
mentary appendices 1–3, respectively.
Data collection
Data were collected prospectively by direct observa-
tion of 100 surgical cases at each site (total 200
cases). Adult surgical cases requiring the presence of
all three OR teams (anaesthesia, nursing and surgery)
were eligible. Two medical students trained as obser-
vers rotated weekly between sites to avoid introducing
interobserver bias. They attended OR lists primarily as
medical students to minimise the impact of observa-
tion on staff behaviour. The observers (one at each
site) were allocated to an OR list at the beginning of
each day by the anaesthetic coordinator, and observed
the entirety of that list wherever possible. It is custom-
ary for students to attend the non-elective ORs at
Hospital 2, and no attempt was made to influence
this. However, it must be emphasised that the cases
observed were not emergency cases, and all patients
had been admitted to a ward and properly evaluated
prior to surgery. We recorded the procedure, surgical
specialty, the elective or non-elective nature of the
case, and which OR team led administration of each
SSC domain. No information identifying patients,
staff or the OR was collected.
We assessed compliance and engagement in SSC
administration using a previously reported checklist
Compliance Assessment Tool (CAT).6 We updated this
tool to align it with the current SSC versions in use at
both hospitals (see online supplementary appendices 1
and 2).
Compliance, engagement and timing
Compliance with administration of each domain item
was defined as verbal communication of that item by
the checklist administrator or other OR team member
during SSC administration. We did not interpret com-
munication of checklist items between team members
outside the context of formal SSC administration as
compliance with that item. However, since it is likely
that such communication is better than omission of
the item entirely, we did record and report instances
where this occurred.
Timing of domain administrations, and OR team
engagement, was recorded. Engagement was rated
according to the number of OR teams engaged. At
least one team member had to be engaged in SSC
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
administration for the team to be considered engaged,
and engagement was defined as listening or contribut-
ing to SSC administration with cessation of other
activities and conversations. The presence of each
team was recorded to ensure engagement was not
confounded by team absence (eg, the surgical team
was often not present during Sign In). Timing of
domain administration was rated as compliant (or
not) with the following recommended practices: Sign
In to be administered in the OR (Hospital 1) or prea-
naesthetic room (Hospital 2) prior to any drug admin-
istration; Time Out to be administered after surgical
site preparation and draping, but prior to the first sur-
gical incision; and Sign Out to be administered while
the surgical team was still present in the OR.
Data quality
Observers underwent training for 1 week prior to the
study, during which they attended OR lists together
with a senior investigator and completed CATs inde-
pendently on the same cases. Observations during
training were subsequently compared and discussed.
Both observers attended the same OR for 10 (5%)
cases during the study to allow evaluation of inter-
observer reliability. Data from completed CATs were
entered electronically. Ten (5%) cases were randomly
allocated for re-entry to test data entry accuracy.
Endpoints and analysis
Primary outcomes were compliance (by hospital) with
administration of SSC domains and individual domain
items, given as ‘domain compliance’, ‘domain comple-
tion’ and ‘item completion’. Domain compliance was
the percentage of cases in which the domain was
administered. Domain completion was the percentage
of all eligible items administered in domain-compliant
cases. Item completion was the percentage of domain-
compliant cases in which each individual item was
administered. Those items that appeared on one hos-
pital’s SSC adaptation but not the other, and those
items considered not applicable to the observed case,
were excluded from analysis. Secondary outcomes
were the percentage of cases in which the timing of
domain administration complied with checklist recom-
mendations, and the percentage of cases in which
one, two or three OR teams were engaged during
domain administration. The association between hos-
pital and domain completion was investigated for each
domain individually using logistic regression with
adjustment for potential confounding factors (elective
vs non-elective case status and surgical specialty). For
this purpose, domain completion included those cases
for which the domain was not completed (ie,
0=domain not completed, and 1=all applicable
domain items completed).
Domain compliance, domain completion and item
completion were compared between the two hospitals
using a χ2 test or a Fisher’s exact test. Compliance
Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
Original research
with prescribed SSC items within each domain was
compared between the two hospitals using an
unpaired t test. We did not correct our p values for
multiple testing. Results of logistic regression analyses
were expressed as ORs and 95% CIs. Analyses were
performed using SPSS Statistics V.19.
RESULTS
Cases
One hundred cases were observed at Hospital 1, and
104 at Hospital 2. The elective vs non-elective status
and surgical specialty of the cases is shown in table 1.
Primary outcomes
SSC domain compliance and mean domain comple-
tion are given in table 2. Surgical specialties were
grouped for logistic regression analysis as: general
surgery, orthopaedics and other ( plastics, urology, vas-
cular, neurosurgery and gynaecology). Hospital and
surgical specialty were significant predictors of
domain compliance and completion (where
0=domain not completed, and 1=all applicable
domain items completed) for all domains. The likeli-
hood of the Sign In domain being fully or partially
completed was greater at Hospital 1 (OR 7.463, 95%
CI 5.382 to 10.381, p<0.001) and for general
surgery cases (OR 1.255, 95% CI 1.020 to 1.541,
p=0.031). The likelihood of the Time Out domain
being fully or partially completed was greater at
Hospital 1 (OR 18.102, 95% CI 11.531 to 28.456,
p<0.001) and for general surgery cases (OR 1.852,
95% CI 1.398 to 2.452, p<0.001). The likelihood of
the Sign Out domain being fully or partially com-
pleted was greater at Hospital 1 (OR 9.895, 95% CI
4.595 to 21.285, p<0.001) and general surgery cases
(OR 2.442, 95% CI 1.639 to 3.640, p<0.001), but
less for orthopaedic cases (OR 0.096, 95% CI 0.028
to 0.325, p<0.001). Case acuity (elective vs
non-elective) was not a significant predictor for
domain completion for any domain (Sign In elective
cases OR 1.077, 95% CI 0.776 to 1.493, p=0.657;
Time Out elective cases OR 0.974, 95% CI 0.620 to
Table 1 The distribution of cases at the two hospitals classified
by acuity and surgical specialty
Hospital 1 Hospital 2
Non-elective 14 101
Elective 86 3
Surgical specialties
General surgery 38 57
Orthopaedics 21 19
Plastics 0 24
Urology 25 0
Vascular 11 0
Neurosurgery 5 0
Gynaecology 0 4
3
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Original research
Table 2 Domain compliance and mean (range) domain completion
Hospital 1
Domain compliance* (%)
Sign in 96
Time out 99
Sign out 22
Domain completion† (%)
Sign in 59 (3–100)
Time out 78 (15–100)
Sign out 74 (9–100)
Domain compliance is expressed as the percentage of cases in which the domain was administered. Domain completion is expressed as the percentage of
all eligible items administered in domain-compliant cases.
*Groups were compared using a χ2 test (χ2 refers to the observed test statistic).
†Groups were compared using a 2-tailed unpaired t test (T refers to the observed test statistic).
1.531, p=0.910; and Sign Out elective cases 0.604,
95% CI 0.298 to 1.228, p=0.164).
Item completion (the percentage of domain-
compliant cases in which each individual item was
administered) is given in table 3, which also shows the
percentage of domain-compliant cases in which the
item was not applicable, and non-compliance was,
therefore, expected (column labelled ‘NA’ (not applic-
able)). For example, the surgical team was often not
present during Sign In at either hospital, and so a
response from this team to the statement of allergies
was not expected. Finally, table 3 (column labelled
‘outside’) shows the percentage of the total cases in
which an item was not formally administered as part
of the SSC (either because the entire domain or the
particular item was omitted) but was discussed at
some point outside the context of SSC administration.
Note that the denominator here is different to that for
the item compliance or ‘NA’ columns, so the row
totals may exceed 100%.
Secondary outcomes
OR team engagement during administration of the
SSC is shown in table 4. No cases were observed in
which all members of all teams present were engaged
for the administration of a domain. Interpretation of
these data must take account of the frequent and
accepted absence of the surgical team at Sign In; a
member of the surgical team was present in just 11%
of cases at Hospital 1 and 19% of cases at Hospital
2. The compliance with recommendations around
timing of domain administration is also reported in
table 4.
Data quality
Simultaneous observations for evaluation of interob-
server reliability produced 266 assessable data points
at Hospital 1, of which 96% were concordant and
275 assessable data points at Hospital 2, of which
90% were concordant. Re-entry of 5% of cases to
4 Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
Hospital 2 p Value
T
31 92.80 <0.0005
48 67.13 <0.0005
9 7.05 0.008
χ2
69 (3–100) 2.19 0.031
59 (2–98) 9.55 <0.0005
44 (11–89) 4.98 <0.0005
evaluate data entry quality produced 99% accuracy
for the resultant 570 data points.
DISCUSSION
We recorded domain compliance of 96, 99 and 22%
for Sign In, Time Out and Sign Out, respectively at
Hospital 1. Domain compliance was considerably
lower at Hospital 2 (table 2), where Sign In was con-
ducted in under a third of cases and Time Out in less
than half. Domain completion was also lower at
Hospital 2, with the exception of Sign In items of
which 69% were completed on average compared
with 59% at Hospital 1. Failures to administer some
items during formal use of the checklist at Hospital 2
were partly mitigated by discussion of those items at
other times. For example, the item pertaining to
thrombo-prophylaxis requirements was administered
in 50% of observed Time Out domains and was dis-
cussed in 24% of cases at times not clearly linked to
checklist administration.
Compliance with SSC administration is important
because it appears to be associated with improved
patient outcomes.2 4 De Vries et al2 introduced a
patient safety system with a SSC as a crucial compo-
nent in a controlled study, and reported complication
rates of 7.1% when completion of checklist items was
greater than the median of 80% versus 18.8% when it
was less than the median. Van Klei et al4 reported a
significant reduction in perioperative mortality when
the WHO SSC was fully completed, but not when the
SSC was incomplete or unused. We did not evaluate
the effect of compliance on patient outcome.
However, we believe these previous studies provide
data which support the contention that compliance
impacts on potential safety benefits.
Our study hospitals have demonstrated different
levels of SSC compliance. One potential reason for
the difference may be the way in which the checklist
was introduced. Several studies have identified active
leadership, a clear rationale for checklist use, exem-
plars of ideal practice and an ongoing process of
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Original research

Table 3 SSC item completion

Hospital 1 Hospital 2

Completion NA Outside Completion NA Outside

Sign In item completion (%)
Patient’s identity stated and agreed 100 100 29
The patient’s wristband is checked 100 81 28
The patient’s procedure stated and agreed 99 94 28
Patient’s consent stated and verified 99 84 33
Communication with patient 65 3 100 31
Presence or absence of allergy stated 93 94 32
Anaesthetist responds 19 24 33
Surgeon responds 0 89 3 81 3
Complex airway query 47 2 6
Anaesthetist responds 89 100
Blood availability statement 82 50 9
Anaesthetist responds 24 100 11
Patient’s surgical site stated and agreed 97 – – –
The surgical site marking is checked if applicable* 47 17
The surgical site and side marking is checked if applicable – – – 63 28
Time Out item completion (%)
Team introduction
Name and role 3 16
Name only 33 6
Acknowledgment that team is known† 31 18
None 34 59
Patient’s identity stated and agreed 100 96 48
Patient’s procedure stated and agreed 100 1 98 48
Patient’s surgical site stated and agreed 83 17 1 – – –
Presence of correct imaging confirmed 15 67 2 70 4
Confirmation of antibiotic requirements 96 86 41
Confirmation of thrombo-prophylaxis requirements 90 50 24
Anticipated critical events are reviewed 94 32 17
Anaesthetist enumerates/denies anticipated critical events 83 63 61
Surgeon enumerates/denies anticipated critical events 87 69 33
Nurse enumerates/denies anticipated critical events 35 0 17
Sign Out item completion (%)
Procedure name is recorded 95 3 89 4
Correct count is verified 91 13 89 29
Confirmation that specimen is correctly labelled 41 55 7 33 56 11
Key concerns for handover are discussed 100 3 44 1
Anaesthetist responds 91 67 75 0
Surgeon responds 91 50 100
Nurse responds 9 0 0
Item completion is the percentage of domain-compliant cases in which each individual item was administered, ‘NA’ is the percentage of cases in which the
SSC domain was administered but the item was not applicable, ‘Outside’ is the percentage of the total cases in which the item was missed (either because
the entire domain or the item itself was omitted) but was discussed outside the context of correct administration of the SSC. See text for further
explanation of data.
*Surgical site is stated and agreed, and surgical site marking items are combined as one item for the Hospital 2 adaptation; these are retained as separate
items on the Hospital 1 adaptation.
†Note that the Hospital 1 adaptation requires confirmation that the team is known, whereas this is not mandated for each case at Hospital 2 (see online
supplementary appendices 1–3).
SSC, Surgical Safety Checklist.

discussion, training and feedback as important for WHO SSC pilot study.1 This involved extensive engage-
successful implementation of checklists in an OR ment with OR staff in an attempt to optimise SSC use
setting.7–9 Hospital 1 participated in the original during the study, including training seminars by the

Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749 5

 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Original research

Table 4 Team engagement during domain administration, and required for participation in the WHO study. This
timing of domain administration appears to have fostered an incomplete appreciation of
Hospital Hospital the SSC as a tool that facilitates communication and
1 2 χ2 p Value teamwork in addition to simply preventing items being
Sign in overlooked. We doubt this is unique to Hospital
Engagement
2. One recent study made the observation that SSC
All team members present (%) 0 0
Time Out appeared to be treated as ‘a double-checking
routine that someone should go through (as opposed
3 teams (%) 1 0
to a team effort)’.10 Some staff seem to believe that the
2 teams (%) 39 75
discussion of checklist items outside the context of
1 team (%) 60 25 <0.001†
formal SSC administration (table 3) is acceptable, and
Timing
this contributes to a perception of adequate compli-
Predrug intervention* (%) 89 31
ance with SSC use at Hospital 2.
Postdrug intervention (%) 7 0 Neither hospital exhibited a high rate of domain com-
Not done (%) 4 69 <0.0005† pliance for Sign Out. In fact, we found poor compliance
Time out with Sign Out at Hospital 1 (administration in only 2% of
Engagement cases) in 2010.6 This observation was fed back and dis-
All team members present (%) 0 0 cussed at an OR staff forum to identify a potential solu-
3 teams (%) 14 40 tion. There was consensus that, unlike the other domains,
2 teams (%) 47 58 Sign Out was not clearly linked to an easily identifiable
1 team (%) 38 2 26.23 <0.0005 OR event. The resulting ambiguity around when it should
Timing occur frequently resulted in it not occurring at all. This
Preincision* (%) 97 48 issue has also been noted by others,11 and an attempted
Postincision (%) 2 0 resolution at our centre involved linking Sign Out to com-
Not done (%) 1 52 <0.0005† pletion of the first swab and instrument count. The
Sign out process of feedback, discussion and identification of a
Engagement potential solution has resulted in an improvement in Sign
All team members present (%) 0 0 Out compliance from 2% in 2010 to 22% in 2011. This
3 teams (%) 9 0 minor improvement illustrates both the challenges of
2 teams (%) 36 33 achieving behavioural change and the requirement for
1 team (%) 55 67 1.0†
more comprehensive interventions when attempts are
Timing
made to establish and maintain new practice.
Ideally, all OR staff would be fully engaged in check-
With surgical team* (%) 21 8
list administration. We defined ‘acceptable’ engage-
Without surgical team (%) 1 1
ment as at least one member of each team participating
Not done (%) 78 95 0.01†
in checklist administration without other activity or
Engagement is given as the percentage of cases in which there was
engagement of all operating room team members present, three team
conversation. Even by this definition, engagement was
engagement, two team engagement, or one team engagement during generally poor at both hospitals. However, one
Surgical Safety Checklist (SSC) domain administration. Timing is given as unanticipated finding was better team engagement in
the percentage of cases in which the timing of domain administration Sign In and Time Out at Hospital 2. Sign In and Time
complied with SSC guidelines (the guideline-compliant option is indicated
by *). Out domains were administered more often at
The test statistic is reported for χ2 comparisons. Hospital 1, but when they were completed at Hospital
†Groups compared using a 2-tailed Fisher’s Exact test. 2 we observed that two and three team engagements
occurred more frequently (table 4). All domains at
Hospital 1 were led by a circulating nurse, whereas at
international study principals, and the wide dissemin- Hospital 2 a member of the anaesthetic and surgical
ation of written materials on the premise of the SSC and teams led Sign In and Time Out, respectively. This
its correct use. A study coinvestigator and a study nurse tactic ensured the involvement of at least one member
were present in the OR on a regular basis over a pro- of those teams most central to the processes occurring
tracted period to field questions and audit practice. at that time, and whose failure to properly engage
These implementation process factors almost certainly made ideal completion of many checklist items impos-
contributed to establishing a ‘checklist discipline and sible. It had the added advantage that several senior
culture’ at Hospital 1 that has persisted over time. members of the OR team lead checklist administration
Hospital 2, by contrast, was not part of the study by example. The improvement in engagement achieved
and was largely left to its own devices when rolling out by this approach is potentially very important. For
the SSC. A series of internally convened staff education example, engagement is crucial in preventing the com-
seminars were held, but Hospital 2 did not have the munication break-down described by Lingard et al12 as
benefit of the comprehensive SSC roll-out programme ‘audience failures’.

6 Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749

 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Strengths and weaknesses
We used trained observers to directly observe SSC
administration in real time, as opposed to retrospect-
ively reviewing SSC forms completed by OR staff.
Our data reflect not only how often SSC domains and
items are vocalised, but also when, by whom and to
whom. The observers attended roughly equal
numbers of cases at both sites, and inter-rater reliabil-
ity was high. We used multiple tests to investigate our
study hypothesis, and this is associated with an
increased risk of Type I error.
The balance of elective and non-elective cases was
substantially different between sites (Hospital 2 cases
were predominantly designated ‘non-elective’). This
arose because Hospital 2 runs several OR suites, and
cases at the suite attended by students are mostly
non-elective. However, no cases involved emergency
surgery per se where time pressure might affect com-
pliance with the SSC. As previously noted, all patients
had been admitted to a ward and evaluated before
surgery. It follows that the elective versus non-elective
status should not have influenced SSC use. Indeed,
domain compliance was similar in the elective and
non-elective cases observed at Hospital 1, and regres-
sion analysis did not identify case acuity as an inde-
pendent influence on domain completion for any of
the domains. Re-evaluation of data from our 2010
study at Hospital 1,6 which used identical outcome
measures, also showed no difference in domain com-
pliance between the 54 elective and 46 non-elective
cases observed at that time.
The SSC has been in use for 5 years at Hospital 1 as
against 2 years at Hospital 2, and so a maturation
effect may have confounded checklist use at the two
sites. However, our experience suggests that enthusi-
asm and diligence around checklist administration
declines rather than increases over time, and others
have reported that checklist use declines after active
encouragement from a research team or similar group
is withdrawn.9 11 Another potential confounder is
that a small number of staff probably have worked (or
do work) at both hospitals. However, this confounder
would also serve to reduce any differences between
the hospitals rather than magnify them. Finally, we
cannot definitively exclude the possibility that other
unapparent site contextual factors may have influ-
enced relative compliance at the two sites.
Summary
We investigated compliance and quality in administra-
tion of the WHO SSC at two tertiary teaching hospi-
tals. Domain compliance was higher at Hospital 1
where the SSC was introduced as part of a formal
study that brought comprehensive training, strong
leadership and early support for its use. This finding
suggests that checklist roll-out strategies can make
a difference, and Hospital 2 is likely to be more
representative of institutions implementing the SSC
Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
Original research
independent of a study protocol. As a consequence, it
cannot be assumed that outcome benefits of the SSC
demonstrated in formal studies will be achieved
simply by SSC adoption unless there is careful atten-
tion to maximising compliance.
Comprehensive education, motivation, feedback
and engagement of relevant leadership ‘champions’,
akin to that which took place as part of the original
WHO SSC study,1 is probably a model of what is
necessary to establish good checklist practice.
However, Hospital 1 has several deficiencies (includ-
ing poor compliance with Sign Out and suboptimal
team engagement) suggesting that, even with a well-
planned implementation, repetitive audit and feedback
is needed to maintain good practice. Optimisation of
checklist design to suit local circumstances may also
be important and provides an opportunity to involve
checklist users with its development. On the basis of
these findings, our group is in the process of develop-
ing and evaluating educational strategies that could be
adopted for use during roll out of the SSC at a new
site, or for reinvigoration of its use at established sites
where compliance is poor.
What is already known on this topic
▸ Surgical checklists can reduce perioperative complica-
tions and mortality, but the magnitude of such bene-
fits appears greater if compliance with checklist item
administration is optimised.
What this study adds
▸ Compliance with the WHO surgical safety checklist at
a centre where the checklist was introduced as part
of a comprehensive investigation (the original Safe
Surgery Saves Lives Study) differed from that at a
similar and colocated centre which adopted the
checklist independently of a study protocol.
Compliance at institutions involved in checklist
research does not always reflect that in the wider
hospital community.
▸ Successful integration of checklists into existing oper-
ating room systems requires a careful and compre-
hensive implementation strategy. Strategies used in
the inception of the Safe Surgery Saves Lives Study at
our institution, such as comprehensive staff educa-
tion, recruitment of influential checklist ‘champions’,
a clear definition of what constitutes correct checklist
compliance, and strategic involvement of key team
members, may help achieve better checklist
compliance.
7
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com
Original research
Acknowledgements The authors would like to acknowledge Dr
Matthew Pawley for his assistance with statistical analyses.
Contributors KC, JH, AM, SM, FS and JW were responsible for
study design. LG, JH and JK were responsible for data
collection. FS and SM provided clinical oversight of data
collection at the two study sites. SM, AM and FS provided
general oversight of the study processes. JH, SM and JW wrote
the initial draft of the manuscript. All authors provided critical
revision of manuscript.
Competing interests AFM was the anaesthesia lead in the WHO
Safe Surgery Saves Lives initiative and is Chair of the Board of
the Health Quality and Safety Commission New Zealand.
Ethics approval This study was approved by the Northern Y
Regional Ethics, New Zealand. Approval number NTY/10/
EXP/077.
Provenance and peer review Not commissioned; externally
peer reviewed.
REFERENCES
1 Haynes A, Weiser T, Berry W, et al. A surgical safety checklist
to reduce morbidity and mortality in a global population. N
Engl J Med 2009;360:491–9.
2 de Vries EN, Prins HA, Crolla RMPH, et al. Effect of a
comprehensive surgical safety system on patient outcomes. N
Engl J Med 2010;363:1928–37.
3 Weiser TG, Regenbogen SE, Thompson KD, et al. An
estimation of the global volume of surgery: a modelling
strategy based on available data. Lancet 2008;372:139–44.
8 Hannam JA, et al. BMJ Qual Saf 2013;0:1–8. doi:10.1136/bmjqs-2012-001749
4 van Klei WA, Hoff RG, van Aarnhem EE, et al. Effects of the
introduction of the WHO ‘Surgical Safety Checklist’ on
in-hospital mortality: a cohort study. Ann Surg 2012;
255:44–9.
5 Windsor JA, Petrov MS. Patient safety in medicine: are
surgeons ready for checklists? ANZ J Surg 2010;80:3–5.
6 Vogts N, Hannam JA, Merry AF, et al. Compliance and quality
in administration of a Surgical Safety Checklist in a tertiary
New Zealand hospital. N Z Med J 2011;124:48–58.
7 Conley DM, Singer SJ, Edmondson L, et al. Effective Surgical
Safety Checklist implementation. J Am Coll Surg
2011;212:873–9.
8 Edmondson AC, Bohmer RM, Pisano GP. Disrupted Routines:
Team Learning and New Technology Implementation in
Hospitals. Adm Sci Q 2001;46:685–716.
9 Fourcade A, Blache J-L, Grenier C, et al. Barriers to staff
adoption of a surgical safety checklist. BMJ Qual Saf
2012;21:191–7.
10 Rydenfält C, Johansson G, Odenrick P, et al. Compliance with
the WHO Surgical Safety Checklist: deviations and possible
improvements. Int J Qual Health Care 2013;25:182–7.
11 Vats A, Vincent CA, Nagpal K, et al. Practical challenges of
introducing WHO surgical checklist: UK pilot experience. Br
Med J 2010;340:b5433.
12 Lingard L, Espin S, Whyte S, et al. Communication failures in
the operating room: an observational classification of recurrent
types and effects. Qual Saf Health Care 2004;13:330–4.
 Downloaded from qualitysafety.bmj.com on June 30, 2014 - Published by group.bmj.com

A prospective, observational study of the

effects of implementation strategy on
compliance with a surgical safety checklist
J A Hannam, L Glass, J Kwon, et al.

BMJ Qual Saf published online July 9, 2013

doi: 10.1136/bmjqs-2012-001749

Updated information and services can be found at:

http://qualitysafety.bmj.com/content/early/2013/07/08/bmjqs-2012-001749.full.html

These include:
Data Supplement "Supplementary Data"
http://qualitysafety.bmj.com/content/suppl/2013/07/07/bmjqs-2012-001749.DC1.html

References This article cites 12 articles, 4 of which can be accessed free at:
http://qualitysafety.bmj.com/content/early/2013/07/08/bmjqs-2012-001749.full.html#ref-list-1

Article cited in:

http://qualitysafety.bmj.com/content/early/2013/07/08/bmjqs-2012-001749.full.html#related-urls

P<P Published online July 9, 2013 in advance of the print journal.

Email alerting Receive free email alerts when new articles cite this article. Sign up in
service the box at the top right corner of the online article.

Notes

Advance online articles have been peer reviewed, accepted for publication, edited and
typeset, but have not not yet appeared in the paper journal. Advance online articles are
citable and establish publication priority; they are indexed by PubMed from initial
publication. Citations to Advance online articles must include the digital object identifier
(DOIs) and date of initial publication.

To request permissions go to:

http://group.bmj.com/group/rights-licensing/permissions

To order reprints go to:

http://journals.bmj.com/cgi/reprintform

To subscribe to BMJ go to:

http://group.bmj.com/subscribe/

View publication stats