Original article 109
Development of key performance indicators for emergency
departments in Ireland using an electronic modified-Delphi
consensus approach
Abel Wakaia, Ronan O’Sullivanb,c, Paul Stauntond, Cathal Walshe,
Fergal Hickeyf and Patrick K. Plunkettg
Objective The objective of this study was to develop a
consensus among emergency medicine (EM) specialists
working in Ireland for emergency department (ED) key
performance indicators (KPIs).
Methods The method employed was a three-round
electronic modified-Delphi process. An online
questionnaire with 54 potential KPIs was set up for round
1 of the Delphi process. The Delphi panel consisted of
all registered EM specialists in Ireland. Each indicator on
the questionnaire was rated using a five-point Likert-type
rating scale. Agreement was defined as at least 70% of the
responders rating an indicator as ‘agree’ or ‘strongly agree’
on the rating scale. Data were analysed using standard
descriptive statistics. Data were also analysed as the mean
c 2013 Wolters
of the Likert rating with 95% confidence intervals (95%
CIs). Sensitivity of the ratings was examined for robustness
by bootstrapping the original sample. Statistical analyses
were carried out using SPSS version 16.0.
Results The response rates in rounds 1, 2 and 3 were 86,
88 and 88%, respectively. Ninety-seven potential indicators
reached agreement after the three rounds. In the context
of the Donabedian structure–process–outcome framework
of performance indicators, 41 (42%) of the agreed
indicators were structure indicators, 52 (54%) were
process indicators and four (4%) were outcome indicators.
Overall, the top-three highest rated indicators were:
presence of a dedicated ED clinical information system
(4.7; 95% CI 4.6–4.9), ED compliance with minimum design
Introduction
Performance monitoring is a continuous process that
involves collecting data to determine whether a service
meets desired standards or targets. It is dependent on
availability of good quality information on healthcare, which
can only be achieved by having a systematic process to
ensure that data are collected consistently, both within and
across organizations. One tool that is frequently used to
assist performance monitoring at an organizational level,
and which can subsequently contribute to improvement of
quality and safety, is key performance indicators (KPIs).
Supplemental digital content is available for this article. Direct URL citations
appear in the printed text and are provided in the HTML and PDF versions of this
article on the journal’s website (www.euro-emergencymed.com).
0969-9546
c 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
standards (4.7; 95% CI 4.5–4.8) and time from ED arrival
to first ECG in suspected cardiac chest pain (4.7; 95%
CI 4.5–4.9). The top-three highest rated indicators
specific to clinical care of children in EDs were: time to
administration of antibiotics in children with suspected
bacterial meningitis (4.6; 95% CI 4.5–4.8), separate area
available within EDs (seeing both adults and children)
to assess children (4.4; 95% CI 4.2–4.6) and time to
administration of analgesia in children with forearm
fractures (4.4; 95% CI 4.2–4.7).
Conclusion Employing a Delphi consensus process, it
was possible to reach a consensus among EM specialists
in Ireland on a suite of 97 KPIs for EDs. European Journal
of Emergency Medicine 20:109–114
Kluwer Health | Lippincott Williams & Wilkins.
European Journal of Emergency Medicine 2013, 20:109–114
Keywords: Delphi technique, emergency departments, emergency medicine,
key performance indicators, performance indicators
a
Emergency Care Research Unit (ECRU), Division of Population Health
Sciences, Royal College of Surgeons in Ireland (RCSI), Dublin, bEmergency
Department, Our Lady’s Children’s Hospital, Crumlin, cDepartment of Paediatrics,
University College Dublin (UCD), dDepartment of Emergency Medicine,
Beaumont Hospital, eDepartment of Statistics, Trinity College, Dublin,
f
Department of Emergency Medicine, Sligo General Hospital, Sligo and
g
Department of Emergency Medicine, St James’s Hospital/School of
Medicine, Faculty of Health Sciences, Trinity College, Dublin, Ireland
Correspondence to Abel Wakai, MD, FRCSI, FCEM, Royal College of Surgeons
in Ireland (RCSI), Dublin, Ireland
Tel: + 353 1 402 2304; fax: + 353 1 402 2764; e-mail: awakai@rcsi.ie
Received 6 September 2011 Accepted 27 January 2012
KPIs are performance indicators in key areas of a service.
They are specific and measurable elements of healthcare
that can be used to assess the quality of care. They are
measures of performance, based on standards determined
through evidence-based academic literature or through the
consensus of experts when evidence is unavailable [1].
They are used to identify areas in which performance is
good and meets desired standards and those in which it
requires improvement. In and of themselves, KPIs cannot
improve quality; however, they can effectively act as flags or
alerts to identify good practice and provide comparability
within and between similar services, identifying areas
having opportunities for improvement and those in which
more detailed investigation of standards is warranted.
Increasing numbers of countries are using performance
DOI: 10.1097/MEJ.0b013e328351e5d8
110 European Journal of Emergency Medicine 2013, Vol 20 No 2
indicators to evaluate the quality of clinical care, whereas
some link payment to achievement [2].
Traditionally, measurement of emergency department (ED)
performance has centred on time targets [3,4]. Although
standards based on time targets may reduce ED patient
stay, there are dangers in focusing too heavily on only time
spent in the ED. Time targets are a blunt instrument for
monitoring ED performance, as they do not differentiate
the time spent on administering active treatment to the
patient from the time spent by the patient in just waiting
for the next step in their care [5]. Furthermore, time targets
incentivize moving the patient rapidly through the ED,
without a counterbalance to ensure that the patient receives
the highest quality of care [5].
Modern EDs perform ever increasing numbers of
diagnostic investigations and therapeutic interventions
before patient disposition [5]. Consequently, such EDs
require a more balanced suite of KPIs to measure ED
quality of care. Suites of KPIs should ideally reflect not
only timeliness but also quality of care and clinical
outcomes. The development of a suite of KPIs suitable
for audit is also essential in defining the role of the ED
and monitoring the standard of care by emergency
medicine (EM) within the healthcare system. The aim
of this study was to develop a consensus among EM
specialists working in Ireland for a suite of KPIs aimed at
monitoring the performance of EDs in Ireland.
Methods
A systematic review of the academic literature was carried
out to establish the evidence base for performance
indicators directly referable to EM practice in Ireland.
The proposed study was approved in April 2009 by the
Joint Research Ethics Committee of St James’s Hospital
and the Adelaide and Meath Hospital incorporating the
National Children’s Hospital, Dublin. A three-round
electronic modified-Delphi study was conducted be-
tween September 2009 and October 2010.
Systematic literature review
We searched the following electronic databases to identify
studies on ED performance indicators directly referable to
EM practice in Ireland: HealthStar (1966 to September
2009), MEDLINE (1950 to October 2009), Embase (1980
to October 2009) and System for Information on Grey
Literature in Europe. Additional efforts were made to
locate eligible studies by cross-referencing from the
reference lists of major articles on the subject. No language
or other limitations were imposed.
Delphi panel recruitment and sample
Purposeful sampling was used to ensure that appropriate
experts were invited to participate [6,7]. A Delphi panel
of all registered EM specialists (Consultants in Emer-
gency Medicine and Fellows of the College of Emergency
Medicine awaiting Consultant appointment) working in
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Ireland were recruited by initially using personal contacts
and then employing the ‘snowballing’ technique [8]. One
study investigator (A.W.) made contact with all EM
specialists in Ireland (through electronic mail), inviting
them to participate in the Delphi process. Each specialist
was sent an introductory letter describing the study.
Delphi process
An online/web-based electronic questionnaire was used
for all rounds of this Delphi process. The first-round
questionnaire consisted of items (KPIs) derived from the
systematic review of the academic literature and con-
sidered by the study investigators as relevant to EM
practice in Ireland.
A pilot study was carried out using four eligible Delphi panel-
lists for each Delphi round to refine the study questionnaire.
At the pilot stage, the questionnaire items and the format of
the information were agreed upon. After the pilot study, the
weblink to the study questionnaire was sent to the panel-
lists and they were asked to rate each KPI electronically
(by clicking on the appropriate answer) using a 5-point
Likert rating scale that ranged from ‘strongly disagree’ (1) to
‘strongly agree’ (5). The panellists then submitted their
responses to the questionnaire electronically to a data collec-
tion office. The study investigators were blinded to the
source of the data submitted by the data collection officer,
who was not involved in the interpretation of the study.
At least 70% of Delphi panellists had to rate a KPI in the
high ‘agreement’ range of scores (score 4 or 5) for the
indicator to be selected for further inclusion and
development. In the questionnaire for each Delphi
round, panellists were given the opportunity to suggest
additional KPIs that they felt were appropriate to be
considered in the next round of the process.
Once the KPIs were rated by panellists, the next step
involved identifying the KPIs rated in the high ‘agree-
ment’ range of scores (score 4 or 5) and additional KPIs
suggested by the panellists. KPIs that did not reach
agreement (< 70% of the panellists rating the KPI in the
high ‘agreement’ range of scores) in rounds 1 and 2 were
included in the following Delphi round. KPIs that
reached agreement and statements that reached con-
sensus at the end of each round were not reiterated in
subsequent rounds. Panellists were asked to rerate each
KPI that did not reach agreement in the subsequent
round in the light of the results and the qualitative
comments of respondents in the previous round.
Data collection and analyses
The study involved three phases of data collection and
analyses.
Phase one
Fifty-four potential ED KPIs informed by the results of
the systematic review of the academic literature and

considered by the study investigators to be relevant
to EM practice in Ireland provided the starting point
for this Delphi process. The weblink to the online
first-round questionnaire containing a set of 54 po-
tential indicators was sent to all the EM specialists in
Ireland.
The study investigators were provided with responses
in a blind format by the data collection office. A summary
report of the first round results was prepared within a
month of completion of the first Delphi round and the
results were sent back to the panellists. This comprised
graphs showing the collated responses from the first-
round questionnaire and a synopsis of any qualitative
comments made by respondents to that round.
Phase two
The panellists were sent a second-round questionnaire
asking them to rate KPIs suggested by respondents to the
Delphi round-1 questionnaire and to rerate the KPIs that
had not reached agreement in the previous round.
The same 5-point Likert rating scale used in the first-
round questionnaire was used for the second-round
questionnaire. The KPIs that reached agreement and
the statements that reached consensus at the end of
round 2 were not reiterated in round 3. The same
predetermined decision rule of more than 70% of
panellists strongly agreeing (rating of 4 or 5) was used
to determine whether a KPI reached agreement.
Phase three
In the final round of the Delphi study, those KPIs that
had not reached consensus at the end of round 2,
together with a summary of the rest of the panel’s
findings, were presented. In this round, we used the same
5-point Likert rating scale and the predetermined
decision rule regarding agreement used in the previous
two rounds.
Data and statistical analysis
Data were collated online and analysed using SPSS
version 16.0 (SPSS Inc., Chicago, Illinois, USA) for
statistical analysis. Data were analysed using standard
descriptive statistics. The mean of the Likert rating and
95% confidence interval (95% CI) were also calculated for
the data from the questionnaire of each Delphi round.
The proportion of individuals in each Likert category for
each KPI was also reported in the summary report of each
Delphi round. The sensitivity of the ratings was examined
for robustness by bootstrapping the original sample.
Results
Systematic literature review
Of the 693 citations identified by the comprehen-
sive literature search, there was no publication on
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Development of ED KPIs in Ireland Wakai et al. 111
performance indicators directly referable to EM practice
in Ireland.
Response rate and number of potential key
performance indicator questionnaire items
In the first Delphi round, 41 of the 48 (86%) EM spe-
cialists invited to participate responded to the ques-
tionnaire. The first round questionnaire contained 54
potential KPIs, of which 36 (67%) indicators reached
agreement. In the second Delphi round, 50 of the 57
(88%) EM specialists invited to participate responded to
the questionnaire. The second-round questionnaire con-
tained 68 potential KPIs, of which 47 (69%) indicators
reached agreement. In the third Delphi round, 52 of the
59 EM specialists invited to participate (88%) responded
to the questionnaire, which contained 36 potential
indicators with 15 (42%) indicators reaching agreement.
The increased number of specialists from rounds 1–3,
who were eligible to be Delphi panellists, was the result
of newly qualified specialists and new Consultant
appointments in the specialty of EM in Ireland during
the study period.
Donabedian classification of key performance
indicators that reached agreement
After the three Delphi rounds, 97 of the potential KPIs
in the study questionnaires reached agreement (see
Supplementary Tables 1–3, Supplemental digital content 1,
http://links.lww.com/EJEM/A18). In terms of the Donabedian
structure–process–outcome framework, 41 (42%) of the indi-
cators that reached agreement were structure indicators, 52
(54%) were process indicators and four (4%) were outcome
indicators.
Key performance indicators that reached agreement
Overall, the top-three highest rated indicators (reported
as the mean values of Likert rating with 95% CIs) after all
three Delphi rounds were as follows:
(1) presence of a dedicated ED clinical information
system (4.7; 95% CI 4.5–4.9);
(2) ED compliance with minimum design standards
recommended by the Irish Association for Emergency
Medicine (IAEM)(4.7; 95% CI 4.5–4.8);
(3) time from ED arrival to first electrocardiogram in
patients with suspected cardiac chest pain (4.7; 95%
CI 4.5–4.9).
Seven potential KPIs specific to Paediatric Emergency
Medicine (PEM) reached agreement from all three
Delphi rounds. The first-round questionnaire contained
11 potential KPIs specific to PEM, three of which
reached agreement. The second-round questionnaire
contained two potential KPIs specific to PEM, both of
which reached agreement. The third-round questionnaire
contained nine potential indicators specific to PEM, two
of which reached agreement.
112 European Journal of Emergency Medicine 2013, Vol 20 No 2
The top-three highest rated indicators (reported as
means of Likert rating with 95% CIs) specific to PEM
were as follows:
(1) time to administration of antibiotics in children with
suspected bacterial meningitis (4.6; 95% CI 4.5–4.8);
(2) separate area to assess children in EDs treating both
adults and children (4.4; 95% CI 4.2–4.6);
(3) time to administration of analgesia in children with
forearm fractures (4.4; 95% CI 4.2–4.7).
Discussion
The aim of this study was to develop a consensus among
EM specialists working in Ireland for a suite of KPIs to
monitor the performance of EDs in Ireland. A Delphi
technique was used to reach consensus in this study as it
is a widely accepted method of achieving convergence of
opinion on real-world knowledge solicited from experts
within a topic [9–12].
We initially carried out a systematic literature review for
two reasons. First, to ensure that there were no previous
studies on KPIs directly referable to EM practice in
Ireland. Second, to inform our choice of indicators for the
Delphi questionnaires we used in this study. The
systematic review confirmed the existence of a knowl-
edge gap in relation to evidence-based KPIs for EDs in
Ireland. These findings are consistent with the previously
identified knowledge gap that exists in relation to a
robust evidence base for ED quality-of-care indicators
and performance measures [13–15].
There was a high participation rate by the Delphi
panellists (all EM specialists in Ireland) invited to
participate in this study. A typical Delphi panel size
ranges from 15 to 35, with the expectation that 35–75%
of invitees will actually participate [6]. In this study, the
participation rates in Delphi rounds 1, 2 and 3 were 86, 88
and 88%, respectively. This makes the findings of this
Delphi process particularly robust, diminishing the like-
lihood that the findings are compromised by nonresponse
error. Furthermore, when we investigated the location of
the nonrespondents, there was no geographical bias or
practice bias in relation to their EM practice (adult EM,
adult and paediatric EM and paediatric EM only). The
robustness of the findings of this study was further
confirmed by examining the sensitivity of the Likert
ratings in each Delphi round by bootstrapping (resam-
pling) the original samples.
In the context of the Donabedian structure–process–
outcome framework for quality of care indicators, the
findings of this study are consistent with the practical
reality of the majority of quality improvement initia-
tives [16]. The majority (54%) of the 97 KPIs that
reached agreement were process indicators. These are
events that occur while the patient is in the ED [17].
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Process indicators are accessible, practical and amenable
to change and are thus usually the focus of quality
improvement efforts. By contrast, only 4% of the 97 KPIs
that reached agreement were outcome indicators (those
events occurring after the patient leaves the ED).
Outcome indicators evaluate the effect of care and
typically include readmission, mortality, morbidity, pa-
tient satisfaction and quality of life [13,17]. Although
policymakers and patients have shown greater interest in
outcome measures than in either structure or process
measures, few outcome measures have currently been
validated for implementation in the emergency-care
setting [13,18,19]. This is because outcome measures
are challenging and difficult to validate in the ED setting.
Methods of risk adjustment in the ED setting are
imperfect, and attributing a patient’s clinical outcome
after ED care to whether or not a specific treatment
intervention was performed is generally difficult [20,21].
Furthermore, as admitted patients spend a relatively
small fraction of their overall hospital stay in the ED,
attribution of patient outcomes to ED care alone is
probably invalid.
Using ED patient satisfaction as an outcome indicator is
controversial because of poor response rates and negative
association with ED overcrowding, both of which are
often out of the control of the providers of ED care [22].
In this study, use of ED patient satisfaction was proposed
as a KPI by respondents in round 2. It was therefore
included for consideration by the Delphi panellists in the
round 3 questionnaire but failed to reach agreement; 67%
of the panellists in round 3 rated patient satisfaction as a
potential ED KPI in the high ‘agreement’ range of scores
(score 4 or 5).
The two highest rated indicators at the end of all three
Delphi rounds in this study were structure indicators
namely presence of a dedicated ED clinical informa-
tion system and ED compliance with the IAEM-recom-
mended minimum design standards. Structure indicators
relate to aspects that are present before the patient visits
the ED [17]. They are related to the characteristics of the
care environment and include the physical layout of the
department, the equipment (e.g. the presence of an
electronic health record), available laboratory facilities,
staff resources (e.g. the nurse-to-patient ratio), protocols,
clinical guidelines and the accreditation status of the
ED [17]. ED management often focuses much attention
on structural issues (building, equipment, protocols), and
structural issues may be relevant to quality improvement
initiatives but they are not the primary focus of most
quality improvement initiatives. Nevertheless, the find-
ings of this study suggest that EM specialists in Ireland
currently consider structural issues as an important factor
in ED quality improvement initiatives in Ireland. This view
is supported by numerous position statements published
by IAEM highlighting the importance of structural issues in

relation to the improvement of quality of ED patient care in
Ireland [23–25]. This may be taken as an indication of a
frustrated focus on poorly implemented recommendations.
Although in this consensus study the majority of potential
KPIs included in the questionnaire for each Delphi round
were applicable to both adult and paediatric ED practice
(e.g. time to administration of antibiotics in sepsis of any
cause, proportion of ED patients triaged within the ac-
cepted timeframe of the triage scale used), the question-
naire for each round also contained potential KPIs specific
to the ED care of children (e.g. time to administration of
analgesia in children with forearm fractures and time to
administration of antipyretics in children with a tempera-
ture greater than 38.51C if not given in the preceding 6 h).
Some of the KPIs relevant to both adult and paediatric EM
that reached agreement in this study have also been chosen
as clinical indicators of ED care in a previous consensus
study on ED care of children [14]. For example, in this
study the proportion of patients who make an unscheduled
return attendance to the ED within 7 days of discharge
reached agreement as an ED KPI.
Valid and reliable KPIs depend on the availability of high-
quality data. Consequently, both the WHO and, in
Ireland, the Health Information and Quality Authority
recommend performing a feasibility analysis before using
KPIs for performance monitoring in healthcare services.
This is because the feasibility of collecting the relevant
minimum data set (MDS) is always a limiting factor. MDS
is defined as core data identified as the minimum data
required to measure performance for a KPI [1,26]. However,
this study did not aim to measure criteria such as the
feasibility, reliability (interrater reliability and test–retest
reliability), sensitivity and specificity of KPIs; future studies
are required to measure these criteria as part of the process
of building a framework for measuring ED quality of care in
Ireland. In the meantime, using a scientifically valid
methodology, we have started to measure the availability,
reliability and internal consistency of MDS items relevant to
some of the KPIs that reached consensus in this study [27].
Study limitations
Although the Delphi technique is a widely used and
accepted consensus-building method for gathering data
from respondents within their expertise [9] and has been
used previously to develop quality of clinical care indicators
for EDs [10–12], this study has some limitations.
First, the focus of this study was on ED patient care, and
the Delphi panel consisted of medical specialists in EM
only, rather than focussing on the emergency care
‘system’ involving other emergency care providers and
settings (e.g. prehospital emergency care). Performance
indicators that are based on separate services may be less
patient-oriented than the measures based on a ‘systems’
approach to emergency and urgent care [12]. However,
we consider this as a necessary starting point, not the end
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Development of ED KPIs in Ireland Wakai et al. 113
point, for the development of performance indicators for
the emergency care system in Ireland.
Second, the external validity of the findings of this study
may be limited because many of the agreed KPIs in this
study probably reflect what the Delphi panellists consider
priorities in relation to ED quality improvement in Ireland.
This may not be directly transferable to EM practice in
other countries. Nevertheless, the methodological frame-
work for this study and some of the agreed KPIs may
inform the development of ED KPIs in other countries.
Third, some of the agreed KPIs in this study may be
considered to be outside the control of ED care providers
(e.g. the proportion of admitted patients transferred to an
inpatient ward within 6 h of ED arrival). Nevertheless, this
does not diminish the value of such KPIs to still effectively
act as flags or alerts to identify good practice, provide
comparability between EDs and hospitals and identify
areas of opportunities for improvement and those in which
a more detailed investigation of standards is warranted [1].
Fourth, as a suite, the KPIs are generic and ‘context-free’,
and their interpretation in the broader Irish healthcare
system may necessarily be limited.
Finally, the study investigators must assume that all the
Delphi panellists, as specialists in EM, had relevant
knowledge on the evidence base for all 97 KPIs that
reached consensus. However, it is conceivable that not all
panellists were aware of the evidence supporting all the
KPIs that reached agreement.
Conclusion
Improving the quality of ED care requires EM specialists
to first define performance indicators relevant to key
areas of ED patient care. In this study, EM specialists in
Ireland reached consensus on a suite of 97 KPIs to
monitor the performance of EDs in Ireland. This suite of
KPIs may be useful in starting to build a framework for
measuring ED quality of care in Ireland.
Acknowledgements
This study was supported by grants from the following:
Irish Association for Emergency Medicine; National
Children’s Research Centre (NCRC), Our Lady’s Chil-
dren’s Hospital, Crumlin, Dublin, Ireland; Department of
Emergency Medicine Research and Education Fund, St
James’s Hospital, Dublin, Ireland. The authors thank all
Emergency Medicine Specialists in Ireland for their
support and participation in this study.
Conflicts of interest
There are no conflicts of interest.
References
1 Health Information and Quality Authority (HIQA). Guidance on developing
key performance indicators and minimum data sets to monitor healthcare
quality, September 2010. Available at: http://www.hiqa.ie/publication/
114 European Journal of Emergency Medicine 2013, Vol 20 No 2
guidance-developing-key-performance-indicators-kpis-and-minimum-data-
sets-monitor-health. [Accessed 27 July 2011].
2 Lindenauer PK, Remus D, Roman S, Rothbeg MB, Benjamin EM, Ma A,
Bratzler DW. Public reporting and pay for performance in hospital quality
improvement. N Engl J Med 2007; 356:486–496.
3 Health Service Executive (HSE). Emergency department task force report,
2007. Available at: http://www.hse.ie/eng/services/Publications/services/
Hospitals/HSE_Publications/Emergency_Department_Task_Force_Report_.pdf.
[Accessed 24 January 2011].
4 NHS Executive. Quality and performance in the NHS: NHS performance
indicators. London: Department of Health; 2000.
5 Department of Health. A&E clinical quality indicators implementation
guidance. Available at: http://www.dh.gov.uk/en/Publicationsandstatistics/
Publications/PublicationsPolicyAndGuidance/DH_122868. [Accessed 24
January 2011].
6 Steele R, Bosma H, Johnston MF, Cadell S, Davies B, Siden H, Straatman L.
Research priorities in paediatric palliative care: a Delphi study. J Palliat Care
2008; 24:229–239.
7 Ospina MB, Bond K, Schull M, Innes G, Blitz S, Rowe BH. Key indicators of
overcrowding in Canadian emergency departments: a Delphi study. CJEM
2007; 9:378–379.
8 Mason J. Qualitative researching. Thousand Oak, CA, USA: Sage
Publications; 1996.
9 Hsu C, Sandford BA. The Delphi technique: making sense of consensus.
Pract Assess Res Educ 2007; 12:1–8.
10 Lindsay P, Schull M, Bronskill S, Anderson G. The development of indicators
to measure the quality of care in emergency departments following a
modified-delphi approach. Acad Emerg Med 2002; 9:1131–1139.
11 Beattie E, Mackway-Jones KA. Delphi study to identify performance
indicators for emergency medicine. Emerg Med J 2004; 21:47–50.
12 Coleman P, Nicholl J. Consensus methods to identify a set of potential
performance indicators for systems of emergency and urgent care. J Health
Serv Res Policy 2010; 15 (Suppl 2):12–18.
13 Pines JM, Fee C, Fermann GJ, Ferroggiaro AA, Irvin CB, Mazer M, et al. The
role of the Society for Academic Emergency Medicine in the development
of guidelines and performance measures. Acad Emerg Med 2010;
17:e130–e140.
14 Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development
of measures of the quality of emergency department care for children using
a structured panel process. Pediatrics 2006; 118:114–123.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
15 Welch S, Augustine J, Camargo CA Jr, Reese C. Emergency department
performance measures and benchmarking summit. Acad Emerg Med 2006;
13:1074–1080.
16 Donabedian A. Evaluating the quality of medical care. Milbank Q 1966;
44:166–200.
17 Graff L, Stevens C, Spaite D, Foody J. Measuring and improving quality in
emergency medicine. Acad Emerg Med 2002; 9:1091–1107.
18 Krumholz HM, Normand SL, Spertus JA, Shahian DM, Bradley EH.
Measuring performance for treating heart attacks and heart failure: the case
for outcome measurement. Health Aff 2007; 26:75–85.
19 National Quality Forum. NQF-endorsed standards. Available at: http://
www.qualityforum.org/Measures_List.aspx. [Accessed 27 July 2011].
20 Iezzoni LI. The risks of risk adjustment. JAMA 1997; 278:1600–1607.
21 Glance LG, Dick A, Mukamel DB, Li Y, Osler TM. Are high-quality cardiac
surgeons less likely to operate on high-risk patients compared to low-quality
surgeons? Evidence from New York State. Health Serv Res 2008; 43:300–312.
22 Pines JM, Iyer S, Disbot M, Hollander JE, Shofer FS, Datner EM. The effect
of emergency department crowding on patient satisfaction for admitted
patients. Acad Emerg Med 2008; 15:825–831.
23 Irish Association for Emergency Medicine (IAEM). Standards for emergency
department design and specification for Ireland 2007. Available at: http://
www.iaem.ie/images/stories/iaem/publications_position_statements/2007/
iaem_standards_for_ed_design__specification_for_ireland_300907.pdf.
[Accessed 27 July 2011].
24 Irish Association for Emergency Medicine (IAEM). Health and safety
standards for Irish emergency departments – providing an optimum
environment for patients and staff. Available at: http://www.iaem.ie/images/
stories/iaem/publications_position_statements/2007/iaem_health__
safety_standards_for_irish_eds_120607.pdf. [Accessed 27 July 2011].
25 Irish Association for Emergency Medicine (IAEM). Staffing needs for
emergency departments in Ireland 2006. Available at: http://www.iaem.ie/
images/stories/iaem/publications_position_statements/2006/iaem_
staffing_needs_for_eds_in_ireland_231106.pdf. [Accessed 27 July 2011].
26 World Health Organisation (WHO). Mental Health Information Systems
(Mental Health Policy and Service Guidance Package). Geneva,
Switzerland: WHO; 2005.
27 Wakai A, McCabe A, Cummins FH, McCoy S, Cronin J, Anagor C, et al. The
availability and reliability of minimum data set items for four Emergency
Department key performance indicators – a pilot study. Natl Inst Health Sci
Res Bull 2011; 6:65.