Quality

By Jeremiah D. Schuur, Renee Y. Hsia, Helen Burstin, Michael J. Schull, and Jesse M. Pines
doi: 10.1377/hlthaff.2013.0730

Quality Measurement In The

HEALTH AFFAIRS 32,
NO. 12 (2013): 2129–2138
©2013 Project HOPE—
The People-to-People Health

Emergency Department: Foundation, Inc.

Past And Future

Jeremiah D. Schuur is an
ABSTRACT As the United States seeks to improve the value of health care, attending physician; chief of
the Division of Health Policy
there is an urgent need to develop quality measurement for emergency Translation; and director of
departments (EDs). EDs provide 130 million patient visits per year and quality, patient safety, and
performance improvement, all
are involved in half of all hospital admissions. Efforts to measure ED in the Department of
quality are in their infancy, focusing on a small set of conditions and Emergency Medicine, Brigham
and Women’s Hospital, in
timeliness measures, such as waiting times and length-of-stay. We review Boston, Massachusetts. He is
the history of ED quality measurement, identify policy levers for also an assistant professor of
emergency medicine at
implementing performance measures, and propose a measurement Harvard Medical School.
agenda. Initial priorities include measures of effective care for serious
Renee Y. Hsia is an associate
conditions that are commonly seen in EDs, such as trauma; measures of professor in the Department
efficient use of resources, such as high-cost imaging and hospital of Emergency Medicine at the
University of California, San
admission; and measures of diagnostic accuracy. More research is needed Francisco.
to support the development of measures of care coordination and
Helen Burstin is senior vice
regionalization and the episode cost of ED care. Policy makers can president for performance
advance quality improvement in ED care by asking ED researchers and measures at the National
Quality Forum, in Washington,
organizations to accelerate the development of quality measures of ED D.C.
care and incorporating the measures into programs that publicly report
Michael J. Schull is the
on quality of care and incentive-based payment systems. president and CEO of the
Institute for Clinical
Evaluative Sciences in
Toronto, Ontario, and a
professor in the Division of
Emergency Medicine,

T
he care delivered in hospital-based
emergency departments (EDs) is
an important element of the US
struggle to improve access to and
quality of health care.1 EDs are the
critical staging area for severely ill patients, the
site of one in eleven ambulatory care visits.2 EDs
play a key role in half of hospital admissions. And
EDs are an important part of America’s safety
net, because they are required by law to evaluate
all patients, regardless of ability to pay.3 In 2010
there were 130 million ED visits in the United
States, and yearly increases in ED visits have
consistently outpaced population growth.4
Providers, payers, and the general public have
differing views of the quality of ED care. For
example, payers often classify the ED as over-
used and costly, while emergency medicine spe-
cialty societies consider ED care to be efficient
and safe. And the increasing use of the ED by
Department of Medicine, at
the public can be interpreted as indicating that the University of Toronto.
many people see the ED as a source of high-
quality care. Jesse M. Pines is director of
Part of the difficulty in knowing which view of the Office for Clinical Practice
Innovation, School of Medicine
ED care is correct is that ED quality measure-
and Health Sciences, and a
ment today is incomplete. Care for certain con- professor of emergency
ditions, such as acute myocardial infarction medicine and health policy at
(heart attack), is closely monitored, perfor- the George Washington
University, in Washington, D.C.
mance is publicly reported, and high-quality care
is rewarded—all of which creates a strong incen-
tive to improve. But the majority of care delivered
in EDs goes largely unmeasured, with the excep-
tion of metrics for length-of-stay and wait-
ing times.
Rigorous quality measurement in health care
is still a relatively new field. The focus to date has
been on developing measures for specific condi-
tions, such as congestive heart failure, instead of

December 2 013 3 2: 12 H ea lt h A f fai r s 2 129

Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
Quality
on developing a measurement strategy in a care
location like the ED to assess the quality of its
myriad functions, such as timely access, diagno-
sis, prognosis, treatment, and disposition.
To enable the measurement and improvement
of ED quality, the care delivered in EDs should
first be viewed in the context of the quality pri-
orities for the US health care system. In 2001 the
Institute of Medicine divided health care quality
into the following six domains: patient safety,
timeliness, effectiveness, efficiency (of resource
use), equity, and patient centeredness.5 The
framework was useful for measuring the quality
of care, but it did not prioritize specific areas of
quality improvement.
The National Strategy for Quality Improve-
ment in Health Care, released in 2011, prioritizes
areas for quality improvement according to the
following three aims: better care, healthy people
and communities, and affordable care.6 Six spe-
cific priorities accompany those aims: making
care safer, engaging people in their care, pro-
moting the coordination of care, promoting
the use of best practices to address leading
causes of mortality, working with communities
to implement those practices, and making high-
quality care more affordable (for the full text of
the six priorities, see the online Appendix).7
Although the national strategy does not spe-
cifically address the delivery of emergency care,
these priorities should guide the vision for high-
quality ED care and for measuring that care. The
strategy’s focus on population health serves as a
reminder that high-quality ED care should be
aligned with the needs of the community. Issues
such as access to timely care, emergency pre-
paredness, and cost should be measured and
improved, both in individual EDs and across
communities.
In this article we review the history of quality
measurement of ED care and lay out a vision for
the future of that measurement. We discuss how
measurement can drive meaningful improve-
ment in the quality of ED care, and we describe
lessons learned from successful and failed ED
quality measures, using historical and interna-
tional examples. Next, we identify policy levers
for implementing new measurement schemes
for ED care. Finally, we propose an agenda for
ED performance measurement research and pol-
icy, aiming to capture EDs’ essential function of
time-critical care, and we identify the areas with
the greatest room for improvement. Examples
include coordinating care and controlling costs.
The History Of ED Quality Measures
The current framework of ED quality measures
evolved from several different policy and aca-
2130 H e a lt h A f fai r s D e c em b e r 2 0 1 3 32:12
Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
demic initiatives, which led to the fragmented
state of ED care measurement today. Quality
measurement in US health care began with
health plans, later spread to hospitals, and final-
ly expanded to physician specialty societies.
The first quality measures of ED care emerged
in the early 2000s from the Hospital Quality
Alliance’s work to measure hospital care for com-
mon conditions such as acute myocardial infarc-
tion, pneumonia, congestive heart failure, and
stroke.8 Several of these hospital-based mea-
sures apply in part to the ED, but the majority
focus on other areas of hospital care. An example
of an ED measure is the percentage of patients
with acute myocardial infarction and specific
findings on the electrocardiogram (ST-segment
elevation or left bundle branch block) who re-
ceive primary angioplasty within ninety minutes
of arriving at the hospital.
In 2004 the Joint Commission and the Centers
for Medicare and Medicaid Services (CMS) col-
lected performance data on these measures and
publicly reported the results on their websites.
These were the first comparative data on ED
performance that were widely available to the
public and payers.8
In 2006, in preparation for CMS’s adoption of
physician-level metrics, the American Medical
Association’s Physician Consortium for Perfor-
mance Improvement developed a set of emergen-
cy medicine physician metrics.9 Six of these
measures—all related to chest pain, syncope
(fainting as a result of low blood pressure), or
community-acquired pneumonia—were adopted
into the Physician Quality Reporting System,10
CMS’s program to measure, report, and reward
physician quality.
Similarly, the American College of Emergency
Physicians developed a set of ten measurements.
Four—those addressing abdominal pain, ectopic
pregnancy, pulmonary embolus, and the use of
central lines—have been incorporated into the
Physician Quality Reporting System.
In addition, over the past fifteen years there
have been a number of independent efforts to
develop quality measures for emergency medi-
cine. Most of these efforts have been led by aca-
demic groups and have focused on specific clini-
cal conditions or populations. The measures
assess timeliness,11 geriatric care,12 pediatric
care,13,14 and general ED care.15 We are not aware
of any measures of ED cost and value to date,
although the American College of Emergency
Physicians has recently initiated a project to de-
velop such measures.
None of these independent ED-specific efforts
was directly linked to a payer, government, or
provider group; as a result, they have not been
widely adopted. In the period 2007–09 the Na-
tional Quality Forum convened two projects to
evaluate and endorse measures of emergency
care, including prehospital care and hospital-
based ED care. Ultimately the organization en-
dorsed twenty-two standards for emergency
care.9 Some of these standards have been incor-
porated into measurement programs, such as
those of CMS or private payers, but most have
not. Thus, ED care is covered by a patchwork of
measures that neither align with national quality
priorities nor reflect the full scope of ED care.
Lessons From Early ED Quality
Efforts
Quality measurement is intended to drive im-
provement by focusing on clinically important
areas where there is a gap between care that is
delivered and care that is supported by clinical
evidence. An example of this in ED care is acute
myocardial infarction.
Timely care—rapidly transferring patients
with a certain type of acute myocardial infarction
(ST-segment elevation or left bundle branch
block) to a cardiac catheterization laboratory
for intervention—is a primary goal. Patients
who receive early reperfusion therapy (that is,
a blocked artery is opened) have lower mortality
rates and fewer complications than patients who
do not receive this therapy.16 A hospital measure
of the time it takes the patient to get from arrival
at the ED to the catheterization laboratory
(known as “door-to-balloon time”) was devel-
oped to quantify this process. It was publicly
reported in 2004 and later used by CMS to deter-
mine a proportion of Medicare’s hospital
payment.
In response, hospitals and EDs have focused a
tremendous amount of resources on the early
identification of patients with acute myocardial
infarction by screening patients at triage with
electrocardiograms. They also have focused on
ensuring that teams of ED physicians and cardi-
ologists are immediately available to recognize
this condition and deploy a complex interven-
tion in a very short period of time (less than
ninety minutes). The measurement has resulted
in dramatic improvements in the quality of care
for acute myocardial infarction. Secondarily, it
has been an impetus for hospitals to create
multidisciplinary quality improvement com-
mittees.17
Not all ED quality measurement efforts have
been as effective. Some have failed because they
were not based on strong evidence. Other efforts
that were focused on improving the care for pa-
tients with a given condition had unintended
effects that worsened the care for other patients.
The initial Core Measure Set for pneumonia im-
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
plemented by the Joint Commission and CMS in
2002 and reported in 2004 included two mea-
sures—the timing of the administration of anti-
biotics and obtaining blood cultures before
administering antibiotics—that serve as caution-
ary tales.
The first measure was the percentage of pa-
tients admitted to a hospital with pneumonia
who received antibiotics within four hours of
arriving at the hospital (either the ED or the
admitting office). In response to the measure,
EDs worked to speed the diagnosis and treat-
ment for pneumonia patients. Some EDs also
responded by prescribing antibiotics any time
they identified a patient with respiratory symp-
toms, because it is difficult to rapidly determine
which patients have pneumonia. Although per-
formance on the measure improved, patient
outcomes did not. In addition, the resulting anti-
biotic overuse for uncomplicated upper respira-
tory infections promoted antibiotic resistance.
The implementation of the pneumonia mea-
sures illustrates that widespread adoption of
well-intended quality measures can have un-
intended consequences. The evidence for the
early use of antibiotics in pneumonia was not
as strong as that for the measure of acute myo-
cardial infarction care, and the initial four-hour
target was arbitrary.18,19 Recent studies have
failed to show a link between mortality and an-
tibiotic timing in pneumonia.20
Similarly, the measure of obtaining blood cul-
tures before administering antibiotics to admit-
ted patients with pneumonia was not based on
published evidence. The presence of this mea-
sure led to the widespread use of blood cultures,
which are seldom clinically useful in community-
acquired pneumonia and which have a signifi-
cant rate of false positive results. The false
positives, in turn, lead to unnecessary tests,
treatments, and costs.21,22
A pernicious effect of quality measurement is
to focus time, resources, and attention on areas
that are measured, to the detriment of other im-
portant areas. Pneumonia and acute myocardial
infarction are only a small fraction of the con-
ditions treated in EDs. But because the results
from quality measures are published on the In-
ternet and affect reimbursement, hospital and
ED leaders spend a great deal of time and
resources on data collection and analysis, and
on making changes to maintain or improve per-
formance for the measured conditions.
There is no direct incentive to focus similarly
on many other frequent, dangerous, and costly
conditions or problems. For example, diagnostic
errors are the leading source of errors in the
ED,23 yet there are no nationally endorsed or
implemented measures addressing diagnostic
D e c em b e r 2 0 1 3 32:12 Health A ffairs 2131
Downloaded from HealthAffairs.org on November 16, 2021.
Quality
accuracy in the ED. Thus, when one is consider-
ing an agenda for ED quality measurement, it is
important to give priority to measures that ad-
dress common serious disease processes and all
six domains of high-quality care.1
Challenges To ED Quality
Measurement
There are several challenges to accurately mea-
suring ED quality. First, the majority of ED care
is based on diagnosing and treating a patient’s
symptoms, instead of on the longitudinal treat-
ment of a specific disease. It is more difficult to
measure the care of patients with chest pain than
that of patients with acute myocardial infarction.
This is because there is no standardized classifi-
cation system in wide use for patients’ com-
plaints and symptoms such as chest pain, as
the International Classification of Diseases (ICD)
system is for diagnoses such as acute myocardial
infarction.
There are few measures based on the most
common complaints of patients in the ED, and
there has been little research into the accuracy of
current measures that are based on common
complaints. This is a major hurdle to measuring
diagnostic accuracy and the efficiency of diag-
nostic testing—two of the most important and
costly areas in ED care.
Second, quality measures would ideally reflect
changes in patient outcomes such as mortality,
instead of process measures such as whether or
not a medicine was administered. However, data
on outcome measures are difficult to obtain be-
cause the US health care system is not integrated,
and data on ED treatment and subsequent out-
comes cannot easily be linked in existing da-
ta sets.
Third, because ED care is team based and in-
volves numerous providers, from the prehospital
emergency medical services (EMS) team to in-
hospital providers, it is difficult to determine the
appropriate unit of attribution. In other words,
should measurement attempt to capture the per-
formance of the individual physician or that of
the ED as a whole? Attribution is even more
challenging when patient outcomes, such as
mortality, are involved, because ED care plays
a small role in the overall outcome. In general,
the most reasonable unit of attribution for emer-
gency care is the ED: Using that would encourage
systematic improvement in the entire ED, which
would have a greater impact than individual phy-
sician improvement.
Fourth is the challenge of using a measure to
assess how well the entire system is functioning
to improve population health, instead of to as-
sess a certain process in one ED. The patient with
2132 H e a lt h A f fai r s D e c em b e r 2 0 1 3 32:12
Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
acute myocardial infarction is a good example
again. EMS systems are being regionalized so
that the patient can receive the right care at
the right place at the right time.24 Thus, the ideal
quality measure would show how one coordinat-
ed system compares to another, instead of how
one patient fared after treatment. But such mea-
sures are not feasible in most of the United States
because acute care systems are fragmented.
In addition, when cost or resource use is being
measured, the attribution of both outpatient and
inpatient care (for example, the total cost of
medical care for an ED visit and during the fol-
lowing month) to an accountable care organiza-
tion would lead to a more coordinated effort to
deliver high-value care. Measures of cost that
include the costs of only the ED physician and
the facility risk encouraging patterns of care that
are of low cost in the ED, but also of low value to
the patient and the health care system overall.
For example, performing fewer detailed ED eval-
uations that would lead to outpatient care but
would result in higher rates of admission to the
hospital could appear to be less expensive if the
subsequent costs were not included.
National ED Quality Measurement
Programs
ED care in the United States is subject to external
quality measurement by the following four large
groups or organizations: CMS and other govern-
ment payers, the Joint Commission and other
hospital accrediting bodies, private payers, and
state regulators. Voluntary groups also assess
care, including organizations such as Health-
Grades25 and Press Ganey26 that give awards
and ratings to measure EDs.
EDs are required to report a significant num-
ber of measures to multiple agencies, each of
which may have its own list of measures using
different specifications; this creates a large mea-
surement burden. CMS has eleven quality mea-
surement programs, including three that have
ED-specific measures: the Hospital Inpatient
Quality Reporting Program, Hospital Outpatient
Quality Reporting Program, and Physician Qual-
ity Reporting System. The online Appendix
shows the measures from these programs that
affect EDs, grouped by clinical condition.7
Hospitals that fail to report measures in these
programs are penalized by a 1 percent reduction
in their annual payment update (an incentive
called “pay for reporting”); the reduction in-
creases to 2 percent in 2014. Additionally, a per-
centage of a hospital’s Medicare payments is at
risk based on the facility’s performance on pa-
tient experience surveys and a subset of the in-
patient quality measures, including several ED
measures (“pay for performance”). Physicians
who participate in the Physician Quality Report-
ing System are now eligible for an incentive pay-
ment. However, beginning in 2015 this incentive
will transition to a combination of a penalty for
nonparticipation and bonuses for low-cost,
high-quality care.
The Joint Commission implements quality
measures through its Core Measure Sets and ac-
creditation standards. The Core Measures are a
set of hospital quality measures that are in gen-
eral alignment with CMS’s measure set for acute
myocardial infarction, pneumonia, and ED
crowding. However, the Joint Commission has
an additional set of stroke measures that affect
ED care.
The Joint Commission’s accreditation stand-
ards are based on measures of structure and
process. A hospital can be cited for not meeting
the standards and might lose accreditation as a
result. Although top performers on some Joint
Commission standards are publicized, compara-
tive data on all institutions are not publicized,
nor is payment tied to performance.
For example, Patient Flow Standard
LD.04.03.11 specifies that all hospitals must
use data and metrics to manage patient flow
throughout the hospital; ensure safe care during
ED boarding—that is, the hours or days when
admitted patients wait in EDs or ED hallways
before reaching an inpatient room; and mitigate
the risks experienced by psychiatric boarders.
One part of the standard, which goes into effect
in 2014, requires hospitals to measure and set
goals for boarding, and it recommends “that
boarding time frames not exceed 4 hours in
the interest of patient safety and quality of
care.”27 Yet the Joint Commission does not col-
lect data on boarding or publicize hospitals’ per-
formance on the standard.
Lessons From ED Quality Measures
Abroad
Looking outside the United States for examples
of innovation in ED quality measurement reveals
a mixed picture. On the one hand, policy makers
in jurisdictions including England,28 Australia,29
and several Canadian provinces28 have enacted
benchmarks and targets for ED length-of-stay
(sometimes called “waiting time”) to address
overcrowding. EDs have been held accountable
for waiting time to varying degrees through pub-
lic reporting of their performance and through
hospital performance incentives, penalties, or
both.28–30
Although success at reducing crowding varies
across jurisdictions,27,28 the implementation of
these benchmarks and targets has resulted in
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
an unprecedented focus on the problem of over-
crowding by hospital and regional decision mak-
ers.30 This focus has led some US observers to
advocate that either the Joint Commission or
CMS should impose similar strict timing targets.
On the other hand, there has been less focus
abroad at the system level on implementing oth-
er measures of ED quality of care. When the gov-
ernment of Ontario, Canada, announced ED
length-of-stay benchmarks in 2007, it commit-
ted to implementing other ED quality mea-
sures—yet no other provincewide ED quality
measures are being publicly reported.31 Many
hospitals measure and report on (sometimes
publicly) a variety of self-selected measures of
quality of care. However, these are voluntary,
variable, and based on potentially noncompara-
ble definitions, data sources, or both.
England’s experience with implementing and
strictly enforcing waiting time measures holds
an important lesson for policy makers. In 2005
England’s National Health Service originated
the four-hour rule, requiring that the maximum
length-of-stay for 98 percent of patients in any
ED be four hours. The country’s EDs achieved
that performance target a few years later.28 Al-
though this led to many improvements in flow
and timeliness, the strict and arbitrary nature of
the rule also led to gaming, with a large number
of patients being admitted to another unit just
before the four-hour deadline arrived.32
In 2010 the new UK government relaxed the
target slightly, to 95 percent of patients from
98 percent, and promised a broader focus on
the quality of care.33 Since then, seven other mea-
sures of ED quality of care have been introduced.
Several of them focus on waiting times and time-
liness, such as time to triage, time before being
seen by a physician, and percentage of patients
who left without being seen. Additional mea-
sures address other aspects of ED care, including
patient experience and rates of unplanned re-
turn ED visits within seven days of discharge,
admission for deep venous thrombosis or cellu-
litis, and emergency medicine consultant physi-
cian sign-off for defined high-risk cases.34
These National Health Service measures are
relatively basic metrics of ED quality. Nonethe-
less, they are an impressive exercise in measur-
ing quality across a jurisdiction. A recent
high-profile report identified low-quality care
provided to patients admitted to the hospital
from the ED as a risk factor for higher mortality,
but it did not propose specific measures to ad-
dress this problem.35
Based on international examples, it does not
appear that public financing and accountability
of a health system are, in themselves, a guarantee
of broad ED quality measurement.
Dece mber 2013 32:12 Health A ffairs 2133
Downloaded from HealthAffairs.org on November 16, 2021.
Quality
Policy Levers For Influencing ED
Quality Measurement
US policy makers have several tools that they
could use to influence quality in EDs. First, ad-
ditional measures could be added to CMS’s qual-
ity measurement programs that affect ED care.
These could include new measures of ED care in
the hospital inpatient and outpatient payment
program and individual provider metrics in the
Physician Quality Reporting System. Adding a
measure to one of these programs results in pub-
lic reporting of performance on the measure on
the Hospital Compare website,36 which in turn
leads hospitals and physicians to focus on im-
proving performance. In addition, hospitals
have a small percentage (1 percent in 2013, rising
to 2 percent in 2017) of their CMS reimburse-
ment tied to their performance on a subset of the
hospital measures, which increases the mea-
sures’ importance.
Furthermore, the Affordable Care Act includes
a new Value-Based Payment Modifier that for the
first time in Medicare’s history incorporates the
value of care into a provider’s reimbursement.
The modifier is a payment adjustment that will
be based on measures of quality, efficiency, and
cost (dollars billed to CMS), as illustrated in the
Appendix.7 Providers will be rewarded in a zero-
sum fashion within a specialty. In other words,
high-value providers will get a bonus, while low-
value ones will be penalized.
CMS is statutorily required to implement the
value-based modifier for all providers for 2017
payments. CMS is designing the modifier for
primary care providers based on the average to-
tal cost per Medicare beneficiary.
It is not clear how the modifier will be calcu-
lated for emergency medicine providers. Al-
though no validated cost or utilization metrics
for ED care are available, CMS will need such
metrics by 2015 (for 2017 payment). This need
presents an opportunity for emergency medicine
researchers and organizations to propose a
method of comparing quality and costs. The
modifier should become an important lever for
reorganizing ED care to reward high-value pro-
viders.
Second, policy makers could use direct regula-
tion to address ED quality. For example, to high-
light the problem of boarding in the ED, CMS
could change the inpatient hospital regulations
so that the start time of an inpatient admission
would not begin until a patient is placed in a
regular hospital bed, instead of beginning while
the patient is waiting in the ED.
Third, policy makers could encourage volun-
tary nongovernmental organizations, such as
the Leapfrog Group, to adopt ED quality stand-
ards. Because it would be prestigious for hospi-
2134 H e a lt h A f fai r s D e c em b e r 2 0 1 3 32:12
Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
tals to meet such standards, many EDs would
work on improving their performance in the
areas related to the standards.
Of note, there is some debate about the effec-
tiveness of pay-for-performance programs on
improving care. Although a detailed review of
pay-for-performance is beyond the scope of this
article, several themes should shape the use of
quality measures in ED payment systems. First,
providers respond to financial incentives: There
is evidence that ED providers respond to the
current fee-for-service system by increasing uti-
lization and upcoding—that is, billing an in-
creasing proportion of visits as high intensity
over time—on the five-level coding system used
to value ED visits.37
Second, although there is little evidence that
Medicare’s pay-for-performance programs im-
prove quality, they affect less than 5 percent of
Medicare payments. That is just a fraction of the
average physician’s or hospital’s revenue stream.
It is possible that performance incentives that
account for a larger fraction of revenue would
be more effective.
Third, rewarding performance on poorly de-
signed measures risks unintended consequenc-
es, such as those described above that resulted
from the ED pneumonia measure. Physicians
and hospitals will continue to be paid in some
manner, and EDs are likely to be paid by fee-for-
service for the immediate future. Thus, we be-
lieve that a pay-for-performance program with a
broad array of measures covering different as-
pects of ED care would be useful for reorienting
the priorities of ED care and improving perfor-
mance on measured areas.
The Future Of ED Care Measurement
Quality measures that reflect the National Strat-
egy for Quality Improvement6 should be devel-
oped for ED care. Initial measurement efforts
should address effective care for common life-
threatening conditions for which there are not
currently measures, as well as patient safety, di-
agnostic error, costs of care, and resource use.
Exhibit 1 shows key measure domains, sample
measures, and their strengths and weaknesses.
The first priority for developing ED measures
should be to address variations in the use of
effective care for a wider range of conditions
and populations than is currently measured.
There are no nationally implemented ED quality
measures for certain high-priority conditions
that affect many patients and have a substantial
burden of disease, such as sepsis, trauma, chron-
ic obstructive pulmonary disease, and asthma. In
addition, special populations—including pediat-
ric and geriatric patients—have unique care con-
 Exhibit 1

High-Priority Areas For The Development Of Emergency Department (ED) Performance Measures
Area Sample measures Strengths Weaknesses
Making care safer
Reducing diagnostic errors, Rate of missed acute myocardial Diagnostic errors are a leading gap in The population at risk for a missed
preventing health care– infarction after ED visit (O), ED quality, and EDs play an condition is large and challenging
associated infections hospital rate of catheter important and underrealized role in to define, it is often not possible to
associated urinary tract infection health care–associated infections link ED visits and follow-up care in
(O) current data, and measuring
diagnostic errors will create
pressure to increase use of testing
Ensuring person- and family-centered care
Improving patient ED Consumer Assessment of ED patients frequently do not Traditional patient surveys have small
experience, engaging Healthcare Providers and Systems understand what was done during sample sizes and low response
patients in decision making (O)a the visit, providers’ conclusions at rates and thus are not reliable;
the end of the visit, or what steps focusing on patient experience
to take after the visit, leading to may lead to overuse, as use of
poor compliance and increased tests is associated with higher
costs survey scores
Promoting the coordination of care
Improving handoffs from the Percentage of patients discharged Transitions after ED care are often Measuring the structure or process
ED to inpatient and from the ED who have a return visit poorly coordinated, leading to of handoffs does not guarantee the
outpatient settings within a defined time, such as errors that affect patient safety quality of the handoff or following
7 days (O) and increase utilization; handoff up on transition actions; outcome
communication is a well- measures, such as return ED visits,
documented gap in ED care are influenced by patient
comorbidities and
sociodemographic characteristics
Preventing and treating the leading causes of mortality
Sepsis, congestive heart Sepsis measures: initiation of critical Sepsis measures would focus ED There are not clear gaps in quality of
failure, chronic obstructiveelements of early goal-directed improvement on leading causes of ED care for all of the leading
pulmonary disease, atrial therapy within timelines for ED morbidity and mortality causes of hospital mortality, and
fibrillation patients with sepsis (P), 30-day process measures can promote
mortality among patients with interventions that may not actually
sepsis (O) improve patient outcomes
Working with communities to implement best practices
Preventive public health Performance of brief motivational As part of the safety net, the ED is Nonacute services must be shown to
interventions interventions for substance abuse often the only point of contact with be effective before they become a
(for example, for intoxicated high-risk patients and the only quality standard in the ED, which is
patients with injuries) (P), presence opportunity to deliver preventive challenged to deliver multiple
of a trauma violence prevention care services in crowded conditions
program (S)
Making high-quality care more affordable
Efficient use of diagnostic Appropriate use of CT for minor head Overuse of costly tests, treatments, Measuring utilization or costs must
tests such as computed injury, hospitalization rate for and hospital admission is well account for patient severity, and
tomography (CT) and conditions that can be managed in documented and leads to patient unintended consequences of
hospital admission, cost of the outpatient setting (such as harm and substantial costs underuse will need to be
ED episode of care deep venous thrombosis) (P), monitored; in current delivery
median cost of care for ED patients models, ED providers have little
(including ED and post-ED services) control over post-ED utilization
(O) and costs

SOURCE Authors’ analysis. NOTES Sample measures are labeled according to Donabedian A. The quality of care. How can it be assessed? 1988. Arch Pathol Lab Med.
1997;121(11):1145–50. S is structural measure; P is process measure; O is outcome measure. aBeing developed by the RAND Corporation.

cerns that merit specific measures. To avoid rep-
licating the unintended consequences of prior
efforts, measures must be evidence-based and
show that time-critical approaches are associat-
ed with improved outcomes.
Efficient resource use should also be an initial
focus of quality measurement in the ED, because
finding a balance between overtesting and diag-
nostic error represents the major challenge to
and potential value of ED care in the United
States. ED care accounts for 5–10 percent of na-
tional health expenditures, and tests and treat-

Dece mber 2013 32:12 Health A ffairs 2135

Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
Quality
ments make up a large component of that cost.38
Whether to order advanced imaging, such as
computed tomography (CT) or magnetic reso-
nance imaging, or to hospitalize a patient are
common decisions with major cost implications
for EDs, and there is wide variation in decision
making at both the physician and ED levels.39,40
Furthermore, the overuse of medications (for
example, controlled substances and antibiotics)
and other treatments is a reasonable target for
utilization measures. Such measures should
form the initial value-based modifier for emer-
gency medicine.
Research is needed to develop measures of
episode costs for ED visits, such as the average
cost for a patient with chest pain.41 Episode costs
of ED care will become important as new pay-
ment models, including accountable care organ-
izations and bundled payments, make hospitals
and provider groups more responsible for high-
cost decisions. However, efficiency measures
have limitations, such as risk adjustment. This
is particularly true in the case of claims data,
which do not capture the underlying reasons—
such as exam findings or physiology—for re-
source use.
The safety and validity of efficiency measures
will also need to be tested, since rewarding re-
duced use may have the unintended conse-
quence of increasing diagnostic errors. The rap-
id implementation of untested utilization
measures can lead to provider pushback, as
was illustrated in CMS’s proposed imaging effi-
ciency measure for head CT in ED patients with
headache.42
Utilization measures should be carefully bal-
anced by measures of “misses”—that is, patients
discharged home who return later with a serious
diagnosis related to the initial complaint, such as
a headache patient who returns with atraumatic
subarachnoid hemorrhage. The Agency for
Healthcare Research and Quality is developing
ED patient safety indicators that may be useful in
capturing diagnostic errors in the ED.43
Patient experience will play an increasing role
in the future of ED quality measurement. Al-
though multiple patient experience surveys for
EDs exist, to standardize measurement, CMS has
contracted the RAND Corporation to develop an
ED Consumer Assessment of Healthcare Pro-
viders and Systems survey. The survey will be
integrated into the value-based purchasing pro-
gram for all hospitals participating in Medicare.
Thus, it is likely to become the dominant method
for measuring patient experience.
Of note, measuring patient experience in the
ED has been criticized because of the small sam-
ple sizes and historically low response rates (in
the 10 percent range) achieved by current survey
2136 Health Affairs D e c e m b e r 20 1 3 3 2: 1 2
Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
vendors, which limit reliability.44 In addition,
current patient experience surveys focus on pa-
tient satisfaction, which is often heavily influ-
enced by wait times and the use of advanced
technology.
Future surveys should ensure higher response
rates and larger sample sizes so that the conclu-
sions drawn will be reliable. Further research is
needed to measure patients’ engagement in their
care, including shared decision making and how
well they understand their medical care and dis-
charge instructions.
EDs face challenges to care coordination, in-
cluding the fact that many EDs and the physi-
cians who work in them are not affiliated with
larger health systems. As a result, there is little
direct incentive to spend time coordinating
care.45
ED quality measurement should focus on the
quality of care coordination. However, most cur-
rent measures focus on care delivered in a single
ED visit and rarely assess how that care is inte-
grated with a patient’s prior health care (for ex-
ample, do patients with benign headache have
repeated CT scans?) or their subsequent health
care (for example, do patients follow up with
primary care providers, specialists, or both after
an ED visit for a chronic condition?). The latter
seems particularly relevant given recent evi-
dence that the timeliness of follow-up after ED
discharge for congestive heart failure,46 chest
pain,47 and atrial fibrillation48 is associated with
mortality.
Developing meaningful measures of care coor-
dination is challenging and requires further re-
search. This is because measures of the process
of conducting coordination activities (for exam-
ple, contacting a patient’s personal physician,
arranging follow-up appointments, and provid-
ing comprehensive discharge instructions) do
not clearly predict the quality of the coordination
or outcomes. Developing measures of quality
that reflect health system integration is particu-
larly relevant now that new structures meant to
promote integration—including accountable
care organizations—are in various stages of im-
plementation.49
Conclusion
A relatively large number of quality metrics are in
use in the United States. However, only certain
aspects of ED care as practiced are measured in
formal quality improvement programs. ED lead-
ers should make it a high priority to develop
evidence-based quality measures that can assess
the extent to which the goal of high-value inte-
grated care is being achieved. Initial priorities
for ED quality measurement should be aligned
with the National Strategy for Quality Improve-
ment in Health Care.6 They should include new
measures of effective care for a broad group of
high-priority conditions; of efficient resource
use, including high-cost imaging and hospital
admissions; and of diagnostic accuracy and
errors.
Important topics that require further research
before new quality measures can be imple-
mented include care coordination, regionaliza-
tion, and the episode cost of ED care. To over-
come the challenges of developing outcome and
efficiency measures for ED care, emergency med-
icine organizations and health systems should
work together to create reliable coding systems
for patients’ symptoms and chief complaints and
data registries that link ED data with more com-
plete claims data and electronic health record
data. Such registries would make it possible to
measure patients’ outcomes and costs after they
leave the ED.
Policy makers have several tools at their dis-
posal to incorporate additional ED quality mea-
sures into public reporting and payment sys-
tems, the most important of which is
Medicare’s new Value-Based Payment Modifier.
If well planned, quality measurement can play an
important role in improving the quality and val-
ue of ED care. ▪

Jeremiah Schuur serves on the Primary

Care and Emergency Medicine Scientific
Advisory Board of UnitedHealthcare. The
authors acknowledge Stacie Jones for
assistance with the figure in the online
Appendix.

NOTES
1 Burstin H. “Crossing the Quality
Chasm” in emergency medicine.
Acad Emerg Med. 2002;9(11):
1074–7.
2 Pitts SR, Carrier ER, Rich EC,
Kellermann AL. Where Americans
get acute care: increasingly, it’s not
at their doctor’s office. Health Aff
(Millwood). 2010;29(9):1620–9.
3 Morganti KG, Bauhoff S, Blanchard
JC, Abir M, Iyer N, Smith AC, et al.
The evolving role of emergency de-
partments in the United States [In-
ternet]. Santa Monica (CA): RAND
Corporation; 2013 [cited 2013
Oct 28]. (Research Report No. RR-
280-ACEP). Available from: http://
www.rand.org/content/dam/rand/
pubs/research_reports/RR200/
RR280/RAND_RR280.pdf
4 National Center for Health Statistics.
National Hospital Ambulatory Med-
ical Care Survey: 2010 emergency
department summary tables [Inter-
net]. Hyattsville (MD): NCHS; [cited
2013 Oct 28]. Available from: http://
www.cdc.gov/nchs/data/ahcd/
nhamcs_emergency/2010_ed_
web_tables.pdf
5 Institute of Medicine. Crossing the
quality chasm: a new health system
for the 21st century. Washington
(DC): National Academies
Press; 2001.
6 Department of Health and Human
Services. 2011 report to Congress:
National Strategy for Quality Im-
provement in Health Care [Internet].
Washington (DC): HHS; 2011 Mar
[cited 2013 Nov 7]. Available from:
http://www.ahrq.gov/workingfor
quality/nqs/nqs2011annlrpt.htm
7 To access the Appendix, click on the
Appendix link in the box to the right
of the article online.
8 Jha AK, Li Z, Orav EJ, Epstein AM.
Care in U.S. hospitals—the Hospital
Quality Alliance program. N Engl J
Med. 2005;353(3):265–74.
9 National Quality Forum. National
voluntary consensus standards for
emergency care: a consensus report
[Internet]. Washington (DC): NQF;
2009 [cited 2013 Nov 7]. Available
from: http://www.qualityforum.org/
WorkArea/linkit.aspx?Link
Identifier=id&ItemID=22009
10 CMS.gov. Physician Quality Report-
ing System [home page on the In-
ternet]. Baltimore (MD): Centers for
Medicare and Medicaid Services;
[last modified 2013 Oct 31; cited
2013 Nov 7]. Available from: http://
www.cms.gov/Medicare/Quality-
Initiatives-Patient-Assessment-
Instruments/PQRS/index.html?
redirect=/PQRI/
11 Hwang U, McCarthy ML, Aronsky D,
Asplin B, Crane PW, Craven CK, et al.
Measures of crowding in the emer-
gency department: a systematic re-
view. Acad Emerg Med. 2011;18(5):
527–38.
12 Terrell KM, Hustey FM, Hwang U,
Gerson LW, Wenger NS, Miller DK,
et al. Quality indicators for geriatric
emergency care. Acad Emerg Med.
2009;16(5):441–9.
13 Guttmann A, Razzaq A, Lindsay P,
Zagorski B, Anderson GM. Devel-
opment of measures of the quality of
emergency department care for
children using a structured panel
process. Pediatrics. 2006;118(1):
114–23.
14 Emergency Medical Services for
Children National Resource Center.
Emergency department pediatric
performance measures [Internet].
Silver Spring (MD); EMSC; [cited
2013 Nov 7]. Available from: http://
www.childrensnational.org/emsc/
pubres/oldtoolboxpages/hospital-
based_performance_measures
.aspx#resources
15 Schull MJ, Guttmann A, Leaver CA,
Vermeulen M, Hatcher CM, Rowe
BH, et al. Prioritizing performance
measurement for emergency de-
partment care: consensus on evi-
dence-based quality of care indica-
tors. CJEM. 2011;13(5):300–9,
E28–43.
16 American College of Emergency
Physicians, Society for Cardiovascu-
lar Angiography and Interventions,
O’Gara PT, Kushner FG, Ascheim
DD, Casey DE Jr., et al. 2013 ACCF/
AHA guideline for the management
of ST-elevation myocardial infarc-
tion: executive summary: a report of
the American College of Cardiology
Foundation/American Heart Associ-
ation Task Force on Practice Guide-
lines. J Am Coll Cardiol. 2013;61(4):
485–510.
17 Krumholz HM, Herrin J, Miller LE,
Drye EE, Ling SM, Han LF, et al.
Improvements in door-to-balloon
time in the United States, 2005 to
2010. Circulation. 2011;124(9):
1038–45.
18 Walls RM, Resnick J. The Joint
Commission on Accreditation of
Healthcare Organizations and Cen-
ter for Medicare and Medicaid Ser-
vices community-acquired pneumo-
nia initiative: what went wrong? Ann
Emerg Med. 2005;46(5):409–11.

December 2013 32 : 1 2 Health Affairs 2137

Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
Quality
19 Wachter RM, Flanders SA, Fee C,
Pronovost PJ. Public reporting of
antibiotic timing in patients with
pneumonia: lessons from a flawed
performance measure. Ann Intern
Med. 2008;149(1):29–32.
20 Pines JM, Isserman JA, Hinfey PB.
The measurement of time to first
antibiotic dose for pneumonia in the
emergency department: a white pa-
per and position statement prepared
for the American Academy of Emer-
gency Medicine. J Emerg Med.
2009;37(3):335–40.
21 Afshar N, Tabas J, Afshar K,
Silbergleit R. Blood cultures for
community-acquired pneumonia:
are they worthy of two quality mea-
sures? A systematic review. J Hosp
Med. 2009;4(2):112–23.
22 Bates DW, Goldman L, Lee TH.
Contaminant blood cultures and re-
source utilization. The true
consequences of false-positive re-
sults. JAMA. 1991;265(3):365–9.
23 Fordyce J, Blank FS, Pekow P,
Smithline HA, Ritter G, Gehlbach S,
et al. Errors in a busy emergency
department. Ann Emerg Med.
2003;42(3):324–33.
24 Wheatley B. Regionalizing emer-
gency care: workshop summary.
Washington (DC): National Acade-
mies Press; 2010.
25 HealthGrades. HealthGrades 2012
emergency medicine in American
hospitals [Internet]. Golden (CO):
HealthGrades; 2012 [cited 2013
Nov 7]. Available from: http://hg-
article-center.s3-website-us-east-
1.amazonaws.com/05/2e/
0d51596645d4a25641d1520b7b03/
HealthGradesEmergencyMedicine
Report2012.pdf
26 Press Ganey [home page on the In-
ternet]. South Bend (IN): Press
Ganey; [cited 2013 Nov 12]. Avail-
able from: http://www.pressganey
.com
27 Joint Commission. Patient flow
through the emergency department.
R3 Report [serial on the Internet].
2012 Dec 19 [cited 2013 Nov 7].
Available from: http://www.joint
commission.org/assets/1/18/
R3_Report_Issue_4.pdf
28 Pines JM, Hilton JA, Weber EJ,
Alkemade AJ, Al Shabanah H,
Anderson PD, et al. International
perspectives on emergency depart-
ment crowding. Acad Emerg Med.
2011;18(12):1358–70.
29 Creswell A. Emergency wait times
tumble as four-hour target does the
trick. Australian [serial on the In-
ternet]. 2011 Sep 5 [cited 2013
Nov 7]. Available from: http://
www.theaustralian.com.au/
2138 H e a lt h A f fai r s D e c em b e r 2 0 1 3 32:12
Downloaded from HealthAffairs.org on November 16, 2021.
Copyright Project HOPE—The People-to-People Health Foundation, Inc.
For personal use only. All rights reserved. Reuse permissions at HealthAffairs.org.
national-affairs/emergency-wait-
times-tumble-as-four-hour-target-
does-the-trick/story-fn59niix-
1226129326986
30 Ovens H. ED overcrowding: the
Ontario approach. Acad Emerg Med.
2011;18(12):1242–5.
31 Ontario Ministry of Health and
Long-Term Care. Ontario wait times:
Ontario’s emergency room wait time
strategy [Internet]. Toronto (ON):
The Ministry; [last modified 2011
Jan 13; cited 2013 Nov 7]. Available
from: http://www.health.gov.on.ca/
en/pro/programs/waittimes/edrs/
strategy.aspx
32 Mason S, Weber EJ, Coster J,
Freeman J, Locker T. Time patients
spend in the emergency department:
England’s 4-hour rule—a case of
hitting the target but missing the
point? Ann Emerg Med. 2012;
59(5):341–9.
33 Topping A, Campbell D. Waiting
targets for accident and emergency
to be scrapped. Guardian [serial on
the Internet]. 2010 Jun 9 [cited 2013
Nov 7]. Available from: http://
www.guardian.co.uk/politics/2010/
jun/10/accident-and-emergency-
waiting-time-nhs
34 UK Department of Health. A&E
clinical quality indicators: data defi-
nitions [Internet]. London; 2010
Dec 17 [cited 2013 Nov 7]. Available
from: http://webarchive.national
archives.gov.uk/20130107105354/
http://www.dh.gov.uk/prod_
consum_dh/groups/dh_digital
assets/@dh/@en/@ps/documents/
digitalasset/dh_122892.pdf
35 Keogh B. Review into the quality of
care and treatment provided by 14
hospital trusts in England: overview
report [Internet]. London: National
Health Service; 2013 Jul 16[cited
2013 Nov 7]. Available from: http://
www.nhs.uk/NHSEngland/bruce-
keogh-review/Documents/
outcomes/keogh-review-final-
report.pdf
36 Medicare.gov. Hospital Compare
[home page on the Internet]. Balti-
more (MD): Centers for Medicare
and Medicaid Services; [cited 2013
Nov 7]. Available from: http://
www.medicare.gov/hospital
compare/search.html
37 Abelson R, Creswell J, Palmer G.
Medicare bills rise as records turn
electronic. New York Times. 2012
Sep 12.
38 Lee MH, Schuur JD, Zink BJ. Own-
ing the cost of emergency medicine:
beyond 2%. Ann Emerg Med.
2013;62(5):498–505.
39 Abualenain J, Frohna WJ, Shesser R,
Ding R, Smith M, Pines JM. Emer-
gency department physician-level
and hospital-level variation in ad-
mission rates. Ann Emerg Med.
2013;61(6):638–43.
40 Pines JM, Mutter RL, Zocchi MS.
Variation in emergency department
admission rates across the United
States. Med Care Res Rev. 2013;
70(2):218–31.
41 Wiler JL, Beck D, Asplin BR,
Granovsky M, Moorhead J, Pilgrim
R, et al. Episodes of care: is emer-
gency medicine ready? Ann Emerg
Med. 2012;59(5):351–7.
42 Schuur JD, Brown MD, Cheung DS,
Graff L 4th, Griffey RT, Hamedani
AG, et al. Assessment of Medicare’s
imaging efficiency measure for
emergency department patients with
atraumatic headache. Ann Emerg
Med. 2012;60(3):280–90.
43 Agency for Healthcare Research and
Quality. Development of emergency
department quality indicators (QI)
(text version) [Internet]. Rockville
(MD): AHRQ; 2009 Dec [cited 2013
Nov 7]. Available from: http://
www.ahrq.gov/news/events/
conference/2009/mcdonald2/
index.html
44 Glickman S, Mehrotra A, Vissers R.
Better methods, better numbers.
Emergency Physicians Monthly [se-
rial on the Internet]. 2013 Jun 7
[cited 2013 Nov 7]. Available from:
http://www.epmonthly.com/
features/current-features/better-
methods-better-numbers/
45 Katz EB, Carrier ER, Umscheid CA,
Pines JM. Comparative effectiveness
of care coordination interventions in
the emergency department: a sys-
tematic review. Ann Emerg Med.
2012;60(1):12–23.
46 Lee DS, Schull MJ, Alter DA, Austin
PC, Laupacis A, Chong A, et al. Early
deaths in patients with heart failure
discharged from the emergency de-
partment: a population-based anal-
ysis. Circ Heart Fail. 2010;3(2):
228–35.
47 Czarnecki A, Chong A, Lee DS,
Schull MJ, Tu JV, Lau C, et al. As-
sociation between physician follow-
up and outcomes of care after chest
pain assessment in high-risk pa-
tients. Circulation. 2013;127(13):
1386–94.
48 Atzema CL, Austin PC, Chong AS,
Dorian P. Factors associated with 90-
day death after emergency depart-
ment discharge for atrial fibrillation.
Ann Emerg Med. 2013;61(5):
539–48.
49 Fisher ES, Shortell SM. Accountable
care organizations: accountable for
what, to whom, and how. JAMA.
2010;304(15):1715–6.