Second Edition

QUALITY
HEALTH
CARE
Second Edition

QUALITY
HEALTH
CARE DEVELOPING
A GUIDE TO

AND USING
INDICATORS

Robert C. Lloyd, PhD

Vice President, Institute for
Healthcare Improvement
World Headquarters
Jones & Bartlett Learning
5 Wall Street
Burlington, MA 01803
978-443-5000
info@jblearning.com
www.jblearning.com

Jones & Bartlett Learning books and products are available through most bookstores and online booksellers. To contact Jones & Bartlett
Learning directly, call 800-832-0034, fax 978-443-8000, or visit our website, www.jblearning.com.

Substantial discounts on bulk quantities of Jones & Bartlett Learning publications are available to corporations, professional associations,
and other qualified organizations. For details and specific discount information, contact the special sales department at Jones & Bartlett
Learning via the above contact information or send an email to specialsales@jblearning.com.

Copyright © 2019 by Jones & Bartlett Learning, LLC, an Ascend Learning Company

All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical,
including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.

The content, statements, views, and opinions herein are the sole expression of the respective authors and not that of Jones & Bartlett Learning,
LLC. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not
constitute or imply its endorsement or recommendation by Jones & Bartlett Learning, LLC and such reference shall not be used for advertising
or product endorsement purposes. All trademarks displayed are the trademarks of the parties noted herein. Quality Health Care: A Guide to
Developing and Using Indicators, Second Edition is an independent publication and has not been authorized, sponsored, or otherwise approved
by the owners of the trademarks or service marks referenced in this product.

There may be images in this book that feature models; these models do not necessarily endorse, represent, or participate in the activities
represented in the images. Any screenshots in this product are for educational and instructive purposes only. Any individuals and scenarios
featured in the case studies throughout this product may be real or fictitious, but are used for instructional purposes only.

This publication is designed to provide accurate and authoritative information in regard to the Subject Matter covered. It is sold with the
understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert
assistance is required, the service of a competent professional person should be sought.

Production Credits
VP, Executive Publisher: David D. Cella Cover Design: Timothy Dziewit
Publisher: Michael Brown Director of Rights & Media: Joanna Gallant
Associate Editor: Danielle Bessette Rights & Media Specialist: Merideth Tumasz
Vendor Manager: Nora Menzi Media Development Editor: Shannon Sheehan
Senior Marketing Manager: Sophie Fleck Teague Cover Image: © Michal Steflovic/Shutterstock
Manufacturing and Inventory Control Supervisor: Amy Bacus Printing and Binding: Edwards Brothers Malloy
Composition and Project Management: S4Carlisle Publishing Cover Printing: Edwards Brothers Malloy
Services

Library of Congress Cataloging-in-Publication Data

Names: Lloyd, Robert C., author.
Title: Quality health care : a guide to developing and using indicators /
Robert Lloyd.
Description: Second edition. | Burlington, MA : Jones & Bartlett Learning,
[2019] | Includes bibliographical references and index.
Identifiers: LCCN 2017022526 | ISBN 9781284023077
Subjects: | MESH: Quality Indicators, Health Care--organization &
administration | Program Development | Quality of Health Care
Classification: LCC RA399.A1 | NLM W 84.41 | DDC 362.1068/5--dc23 LC record available at https://lccn.loc.gov/2017022526

6048

Printed in the United States of America

21 20 19 18 17 10 9 8 7 6 5 4 3 2 1
“To Gwenn and Devon…thanks for giving my life
balance, quality and love!”
Contents
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Introduction to the Second Edition. . . . . . . . . . . . . . xiv
About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Chapter 1 
Setting the Context
for Measurement . . . . . . . . . . . . 1
The Growing Demand for Transparency. . . . . . . . . . . 2
The Growing Focus on Patient-Centered Care
and Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
The Quality Funnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Chapter 2 Why Are You Measuring? . . . . 25
Connecting the Dots!. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Types of Studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Research for Efficacy, Efficiency, and
Effectiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
The Three Faces of Performance Measurement. . . 33
CASE STUDY #1: Being a Translator. . . . . . . . . . . . . . . 39
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Chapter 3 
Measuring the Voice
of the Customer . . . . . . . . . . . 45
It All Starts with Listening. . . . . . . . . . . . . . . . . . . . . . . . 45
Creating a Service Excellence Culture. . . . . . . . . . . . 46
Who Are Your Customers?. . . . . . . . . . . . . . . . . . . . . . . 55
Defining Key Quality Characteristics. . . . . . . . . . . . . . 57
Listening Three Times. . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Understanding VOC Tools . . . . . . . . . . . . . . . . . . . . . . . 61
Image Credit © Michal Steflovic/Shutterstock
vi
Focus Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64
Personal Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Leadership Walk-Rounds. . . . . . . . . . . . . . . . . . . . . . . . . 65
Unsolicited Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
High-Tech Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
The Experiential Shopper. . . . . . . . . . . . . . . . . . . . . . . . 66
Surveys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Chapter 4 
Milestones in the Quality
Measurement Journey. . . . . . 93
Developing a Measurement Philosophy. . . . . . . . . .93
Measurement Roadblocks. . . . . . . . . . . . . . . . . . . . . . . 95
Milestones in the Quality Measurement
Journey. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Selecting a Specific Indicator . . . . . . . . . . . . . . . . . . . 104
Developing Operational Definitions. . . . . . . . . . . . . 109
Developing Data Collection Plans. . . . . . . . . . . . . . . 113
Probability Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Nonprobability Sampling. . . . . . . . . . . . . . . . . . . . . . . 120
The Indicator Development Worksheet. . . . . . . . . 127
CASE STUDY #1: Transcription
Turnaround Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Chapter 5 
Organizing Indicators into
a Strategic Dashboard. . . . . 143
Evolution of the Strategic Dashboard. . . . . . . . . . . 144
Focusing on the Vital Few. . . . . . . . . . . . . . . . . . . . . . . 151
 Contents vii

CASE STUDY #1: East London National Key Questions About Shewhart Charts. . . . . . . . . . 214
Health Service (NHS) Foundation Trust’s Deciding Whether a Special Cause
Strategic Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . 152 Is Present . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
The Role of Benchmarking. . . . . . . . . . . . . . . . . . . . . . 155 Deciding Which Shewhart Chart
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Is Most Appropriate. . . . . . . . . . . . . . . . . . . . . . . . . . 225
Types of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
Chapter 6 
Tapping the Knowledge Types of Shewhart Charts. . . . . . . . . . . . . . . . . . . . . . . 229
That Hides in Data . . . . . . . . 159 Defining the Key Terms. . . . . . . . . . . . . . . . . . . . . . . . . 230
Data Versus Information. . . . . . . . . . . . . . . . . . . . . . . . 159 You Make the Call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Static Versus Dynamic Approaches to Additional Shewhart Charts. . . . . . . . . . . . . . . . . . . . . 247
Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Using Shewhart Charts Effectively. . . . . . . . . . . . . . . 249
CASE STUDY #1: The Monday Morning References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Dilemma. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Chapter 10 
Applying Quality
Measurement Principles. . 259
Chapter 7 
Overcoming Numerical
Illiteracy. . . . . . . . . . . . . . . . . 171 CASE STUDY #1: Predicting a
Cardiovascular Event. . . . . . . . . . . . . . . . . . . . . . . . . 259
Understanding Variation Conceptually. . . . . . . . . . 171 CASE STUDY #2: Sampling Central
Distinguishing Common from Special Line Infections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Causes of Variation. . . . . . . . . . . . . . . . . . . . . . . . . . . 179 CASE STUDY #3: Sampling Medicare
Making the Appropriate Responses to Insurance Audits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Common and Special Causes of Variation. . . . . 182 CASE STUDY #4: Tracking Patient Falls. . . . . . . . . . . 268
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 CASE STUDY #5: Pressure Ulcer
Prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Chapter 8 
Understanding Variation CASE STUDY #6: Evaluating Staffing
with Run Charts . . . . . . . . . . 187 Effectiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
What Is a Run Chart?. . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 CASE STUDY #7: To Flash or Not to
How Do I Construct a Run Chart?. . . . . . . . . . . . . . . 188 Flash—That Is the Question. . . . . . . . . . . . . . . . . . 283

How Do I Analyze a Run Chart?. . . . . . . . . . . . . . . . . 190 CASE STUDY #8: Clarifying the Operational
Definition of Readmission. . . . . . . . . . . . . . . . . . . . 286
CASE STUDY #1: Hand Hygiene
Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 CASE STUDY #9: Managing a Breast
Cancer Patient’s Clotting Levels . . . . . . . . . . . . . . 287
A Few Closing Thoughts on Using
Run Charts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 CASE STUDY #10: Group B Streptococcus
in Pregnant Women. . . . . . . . . . . . . . . . . . . . . . . . . . 290
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
CASE STUDY #11: Emergency Department
Fast Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Chapter 9 Understanding Variation
CASE STUDY #12: Tracking Patient
with Shewhart Charts. . . . . . . 211 Complaints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Run Charts Versus Shewhart Charts. . . . . . . . . . . . . 211 CASE STUDY #13: Reducing Ventilator-
What Is a Shewhart Chart?. . . . . . . . . . . . . . . . . . . . . . 211 Associated Pneumonia. . . . . . . . . . . . . . . . . . . . . . . 300
viii Contents

CASE STUDY #14: Pain Management for Linking Measurement to Improvement. . . . . . . . . 339
Hip and Knee Replacement Patients. . . . . . . . . . 306 Building Capacity and Capability for
CASE STUDY #15: Hospice/911 Paramedic Improvement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
System Partnership to Improve Care. . . . . . . . . . 315 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
CASE STUDY #16: Improving Access to
Community Services for Mental Health
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
and Community Health Patients. . . . . . . . . . . . . . 317
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Chapter 11 Connecting the Dots. . . . . . 331

Adopting Quality as a Business Strategy. . . . . . . . . 332
Developing a Learning System to Support
Improvement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
Acknowledgments
A
nyone who thinks that writing a book
is a solitary activity is sorely mistaken.
Sure, authors do squirrel themselves away
when developing the content, conducting research
for the book, and writing chapters. But writing a
book is more of a team sport than anything else. I
have had the pleasure of being part of the Jones &
Bartlett Learning team since the fall of 2000.
That was when Mike Brown, publisher at Jones &
Bartlett Learning, informed me that the group I
thought I was writing a book for decided to get
out of the healthcare publishing business. The
rights for the first edition of this book were then
transferred to Jones & Bartlett Learning and, as
the expression goes, this was the beginning of a
beautiful relationship.
Over the past 17 years, I have met many
talented and dedicated professionals at Jones &
Bartlett Learning. The one constant in this jour-
ney, however, has been Mike Brown. He has not
only been a wonderful sponsor of my work but
when you are collaborating with an individual
for 17 years you move beyond a writer–publisher
relationship and become friends. We have shared
many professional and personal stories. Mike
has had to gently and at times not so gently use
his diplomatic motivational skills (and “sharp
elbows” as he calls them) to get me to pick up
the pace. There have been times when I missed
deadlines because of international travel, speak-
ing engagements, and family commitments. But
Mike has always been able to maintain a positive
approach to our work and be very tolerant of my
rather hectic schedule. For your professionalism,
support, and friendship, Mike, I thank you.
Image Credit © Michal Steflovic/Shutterstock
The team that Mike assembled to support
this second edition has been very skilled and
extremely patient. In particular, I want to rec-
ognize and thank Danielle Bessette, associate
editor at Jones & Bartlett Learning, for her
leadership in coordinating all aspects of this
endeavor. Danielle organized a wonderful team
of individuals skilled in marketing, media and
design, production, rights clearance, and co-
pyediting. Special recognition and thanks are
extended to the Jones & Bartlett Learning team
including Shannon Sheehan, media development
editor; Sophie Fleck Teague, senior marketing
manager; Merideth Tumasz, rights specialist;
and Nora Menzi, vendor manager. In addition to
the Jones & Bartlett Learning team, I also want
to thank several independent contractors for
their significant contributions to the editorial
process. They include Palaniappan Meyyappan,
project manager at S4Carlisle Publishing Services;
Maria Leon Maimone, rights clearance editor;
and Sandra Kerka, copyeditor, who painstakingly
read and carefully edited the text. This team of
well-coordinated and courteous professionals
has allowed me to focus on the content and
messages of this book while they attended to
the technical and editorial details.
Other individuals who need to be acknowl-
edged are the many dedicated professionals I
have had the pleasure of working with over
the years in many countries around the world.
This group includes participants in the quality
improvement (QI) workshops I teach, improve-
ment teams that I have had the pleasure of
coaching, senior leaders and board members
ix
x Acknowledgments
who have shared their visions for what is possible
as well as the challenges they face in terms of
realizing these visions, and, most important,
those who work tirelessly each day to deliver
care and support to those in need. These are
the individuals who have been kind enough
to share their stories and experiences with me
and helped to set the context for this book. The
case studies presented in Chapter 10 all stem
from interactions with these individuals. Those
who have made substantive contributions to
the content or case studies are recognized in
various chapters and the footnotes.
I also want to thank Dr. Don Berwick for
kindly writing the foreword to this second
edition. I have known Don for almost 30 years.
In 1992, he invited me and a colleague, Dr. Ray
Cary, to teach a daylong program on the ap-
plication of statistical process control (SPC)
methods to healthcare situations at the Institute
for Healthcare Improvement’s (IHI) National
Quality Forum. This was one of the first expo-
sures many healthcare professionals had to SPC.
These initial workshops were so successful that
Don invited us back to the forum each year and
then encouraged us to write a book for healthcare
professionals focused on quality measurement
and in particular SPC applications. I have been
dedicated to the quality measurement journey
ever since. In 2005, Don invited me to join the
IHI as a full-time employee. We have traveled
many roads together over the years and I am
grateful for his support, guidance, and willingness
to contribute to this edition.
Finally, I want to thank my wife Gwenn
and daughter Devon for their moral support,
tolerance, and input to this process. Many
evenings and weekends I disappeared into my
office to work on “the book.” This happened
so often that they would actually just nod at
each other and say “the book” when I started
to head to my office. Unlike the first edition,
however, I was successful in making them
part of this process. They provided reviews of
various charts and graphs as well as played a
major role in selecting the cover design. Gwenn,
I especially appreciate all the time you spent
reading the draft chapters and offering edito-
rial comments and suggestions. Your outside
view and practical perspectives contributed
significantly to making the text flow and be
easy to understand. I always looked forward
to you saying, “Now this just does not make
sense.” This provided an opportunity for im-
provement. Thanks to both of you for making
my life a quality journey!
Foreword
I
n the now 30-year history of bringing modern
quality methods into the control, improve-
ment, and planning of health care, skeptics
sometimes comment on the “religious” tone
of that movement. Leaders and others in the
workforce who get the quality “bug” seem to
buzz with their enthusiasm. They adopt phrases
like “joy in work,” “pursuing perfection,” and a
“never-ending journey” and sprinkle their vocab-
ulary with unfamiliar technical expressions, like
“PDSA cycles,” “high reliability organizations,”
and “statistical process control.” And, they seem
to think they are right, lamenting together that
too many others do not see what they, at last, see.
So it does, indeed, seem to newcomers as if
a religion, or at least a cult, has arrived in town.
The “immune reaction” can be strong.
If you are of that mind, think again. Imagine,
maybe, that what these newfound enthusiasts
are evincing is not religiosity but intellectual
excitement. To overstate, what might Galileo have
felt when, for the first time in human history, he
saw those moons of Jupiter and realized that they
must be orbiting a sphere? Or, more mundane,
what did my 5-year-old grandson, Caleb, feel
last weekend, when all of a sudden at a Sunday
lunch he “got” the idea of letters spelling a word.
(It was the word “bark.”) Like Galileo, maybe,
he laughed out loud.
I do not know why for so many decades health
care called itself “modern,” which it technically
became in the era of bioscience, but remained
distinctly “unmodern” in its understanding of
its work as a system—complex, interdependent,
Image Credit © Michal Steflovic/Shutterstock
and improvable. My teachers in medicine taught
me to be a heroic individual problem solver. My
mentors in organizational management taught me
to use incentives, hierarchy, and accountability
to extract excellence. The language and tools of
improvement revealed the underlying theory that
“trying harder” was the route to success and that
metrics somehow—magically—led to results.
Ideas like that now seem to me to be a
pervasive form of system illiteracy. They are
not scientific. I simply did not know that for
much of my early career because it was, before
I studied systemic quality sciences, as evident
to me that effort is the root of results as it was
to most people before Newton that apples fell
because they just moved toward the center of
the universe.
It took breakthroughs in a number of sciences
to reach today’s level of understanding of how
things get better, or worse, in complex systems.
That understanding—call it “quality sciences”
if you need a name for it—came through even-
tually intersecting lines of progress in statistics,
general systems theory, cognitive and behavioral
psychology, epistemology, and more. It also
continues to be dynamic. Like all sciences,
quality sciences are in continual evolution and
increasingly powerful.
Happenstance introduced me to these sciences
in my mid-forties, and I have never looked back.
By understanding systems better, by relearning
how to interpret and learn from variation, by
realizing how informative very small-scale, local
tests of change can be, by rethinking my theories
xi
xii Foreword
of human motivation and communication, I
was able to see more clearly where defects were
coming from and how to find and change their
causes. Those subjects, mastered over time, gave
me lenses and tools far more persuasive and
helpful than the atheoretical approaches of the
first part of my career.
Maybe it was an epiphany of sorts. But there
was nothing at all “religious” about it. I just
learned things I had not previously known—new
guides to effective action. Someone showed me
Jupiter’s moons.
That’s not comfortable, at least at first. It is
not easy to let go of theories closely held, even
when shown logically to be wrong. Galileo
paid a huge price for that in a public that found
misconception less disruptive than changed
perception. And so, the jargon and excitement
of the quality sciences are easy prey to those
whose beliefs are time honored, though wrong.
To accept the change in understanding
depends in part on teachers—people with the
patience to meet learners where they are and
walk them down the path of new perception.
Like religion this takes empathy and compas-
sion. But, far from faith-based change, this job
also takes rigor and commitment to science.
I love the quotation from Albert Einstein at the
entrance to the Keck Building headquarters of
the National Academies of Sciences, Engineering,
and Medicine: “The right to search for truth
implies also a duty; one must not conceal any
part of what one has recognized to be true.” If
that be a religion, sign me up.
And that brings us to this book, Quality
Health Care: A Guide to Developing and Using
Indicators, and to its author, my longtime friend,
mentor, and colleague, Bob Lloyd. No topic is
more thoroughly a battleground between the
older, and unscientific, methods of improvement
and the newer, more theoretically grounded
methods that I call “modern” than is the topic
of measurement. And few topics generate more
controversy at first.
The prior, hegemonic, view of measurement
is that it causes improvement and is therefore
a powerful tool for implementing a theory of
exhortation, accountability, and incentive. Not
so. Get in touch with the workforce and ask
them how it is going with the metrics in their
work lives, and they will tell you how scrutinized
they feel and how demoralized, threatened, and
misunderstood they feel by that scrutiny. They
will equate measurement with waste and risk,
not growth and learning.
The scientific approach to improvement
also values measurement, but it is measurement
for learning, not measurement for judgment.
It knows, in the words of an African proverb,
that “weighing a pig does not make the pig
fatter.” But it also knows that careful, respectful
metrics, linked with sound interpretation of
variation, trust in the workforce, methods for
local trials and tests, celebration, and supports
for innovation, can be invaluable in continual
improvement. And that all of this matters in the
search for knowledge, put to use.
Bob Lloyd is the best teacher I have ever
met in those vineyards of measurement-for-­
improvement. He is stunning in the classroom. I
have teased him often about how relentlessly at
the top his ratings are in the many Institute for
Healthcare Improvement conferences where he
is an instructor. It’s very hard for the rest of us,
like watching a gymnast do what is for normal
people impossible. We watch him in awe as he
takes novices by the hand and in days shapes
them into expert interpreters of variation and,
therefore, far more helpful quality champions
in their home organizations.
This book is a resource for that change,
written by a master. Bob has been able to
skillfully blend the quantitative aspects of the
­science of improvement with the more qualitative
and strategic aspects that allow organizational
transformation to flourish. In the final chapter
he provides clear guidance on how to “connect
the dots” by linking measurement efforts to

improvement. As he points out, “Data without
a context for improvement are useless!” We have
a long way to go yet in grafting quality science
into the core of our healthcare systems, but those
who really want to do it, to help our patients,
their families, and our communities, have no
better place to turn for their development than
to this book and this teacher.
Foreword xiii
It’s not a religion. It’s intellectual progress,
personal and cultural. So, welcome it, and read on.
Donald M. Berwick, MD, MPP
President Emeritus and Senior Fellow
Institute for Healthcare Improvement
Cambridge, MA
Introduction to the
Second Edition
T
he previous edition of this book outlined
the foundation for applying statistical pro-
cess control (SPC) methods to healthcare
situations. As I have used this book to teach classes
and seminars on SPC, however, it soon became
clear to me that a number of measurement chal-
lenges were still facing healthcare professionals.
Specifically, I discovered that the key concepts
related to SPC, such as common and special
causes of variation, variables and attributes data,
and proper control chart selection, were quickly
being grasped and understood. What healthcare
professionals seemed to be struggling with most
often, however, was something I had taken for
granted. I had assumed that people knew what
they wanted to measure and how to organize
data collection efforts. It is difficult to apply
control chart knowledge properly if you have
not defined appropriate indicators and collected
data that accurately represent the process being
measured. SPC charts developed from incom-
plete operational definitions and poor data
collection strategies may look nice graphically
but they have limited utility for improving care
processes and building knowledge. Helping
healthcare professionals address issues related
to indicator development, data collection, and
statistical analysis, therefore, are the primary
objectives of this edition.
This edition is a completely revised version
of the first edition. The first edition contained
Image Credit © Michal Steflovic/Shutterstock
xiv
8 chapters whereas this one has been expanded
to 11 chapters. The context for this edition
has also been expanded. In the first edition,
the context was based on events occurring in
the U.S. health system. My work over the past
12 years has allowed me to expand my perspec-
tives by working in many different countries
and experiencing a variety of health and social
care delivery systems. In this current edition,
therefore, I have included examples of my work
and experiences in other countries.
Chapter 1 provides a context for understand-
ing quality measurement with a central focus
on four key issues: (1) the growing demand for
greater transparency of healthcare performance
indicators; (2) the increasing role that patients,
families, and caregivers are playing in making
healthcare decisions; (3) the increasing use of
quality improvement (QI) concepts, tools, and
methods; and (4) the definition of quality. This
chapter provides an overview of some of the
key historical as well as recent forces that have
shaped the transformation of the healthcare
industry throughout the world. It concludes
by discussing three key definitions of the term
quality that often lead organizations to send
mixed messages about the role of quality and
what they are trying to achieve.
In Chapter 2, a simple question is posed to
challenge the reader. Specifically, “Why are you
measuring?” There is constant and ever-growing

demand to measure healthcare performance across
the globe. But to what end? Is it to do academic
studies, pass judgment on performance, and hold
healthcare providers accountable for their actions
or to improve the efficiency, effectiveness, and
safety of healthcare delivery?
Measuring the voice of the customer (VOC)
is the topic of interest in Chapter 3. This chapter
is greatly expanded from what I covered in the
first edition. All quality journeys should begin
by listening to the VOC, the point at which you
can learn what your customers want, need, and
expect. The growing role of the customers and
methods for documenting and analyzing their
expectations are the central topics in this chap-
ter. Three key listening points in the customer
experience are identified (preservice, point-of-
service, and postservice), and a variety of tools
designed to capture data at these critical junctures
are reviewed. Because surveys are used so much
in health and social care settings I have added a
large section in this chapter to address the design,
development, and use of surveys. This chapter ends
with a brief example of how VOC measurement
needs to be connected with voice of the process
(VOP) measurement in order to make continuous
QI a practical reality.
Chapter 4 provides details on the milestones
in the quality measurement journey (QMJ). All
too often organizations take a serious detour in
their QMJ because they try to measure a broad
concept (e.g., patient harm) rather than identi-
fying specific indicators that could be used to
measure actual performance (e.g., medication
errors per 1,000 doses dispensed or the per-
centage of patients appropriately placed on bed
alarms). This chapter provides specific guidance
in selecting indicators, developing operational
definitions, and executing data collection strate-
gies (including stratification and sampling). An
Indicator Development Worksheet is offered to
assist the reader in working through all the issues
involved with selecting and building a few good
indicators. This chapter ends with a case study
Introduction to the Second Edition xv
on transcription turnaround time to demonstrate
how a successful QMJ can be completed.
After a number of specific indicators have
been developed, there is value in organizing
them into a cogent and parsimonious format
that can be shared with those who do not need
to know all the details involved with indicator
development. Chapter 5 provides guidance on
how to accomplish this by building strategic
dashboards. The discussion begins by character-
izing the similarities and differences between two
popular ways of organizing indicators—report
cards and dashboards. The argument is made that
the dashboard concept is much more relevant to
healthcare performance measurement than is the
notion of the report card. A case study provides a
practical example of how one healthcare organiza-
tion in London has built a very effective strategic
dashboard. Chapter 5 concludes with a review of
the role of benchmarking in healthcare settings.
After collecting data and organizing your
indicators, the next major milestone in the QMJ
is to tap the knowledge that hides in the data.
Chapter 6 addresses two fundamental issues related
to this challenge: (1) the differences between data
and information and (2) how static and dynamic
approaches to data analysis provide fundamentally
different perspectives to data analysis. A case
study on presenting data to a management team
is used to illustrate the importance of building a
dialogue on these issues.
With these distinctions established, Chapter 7
provides a way to start immunizing yourself
and others against numerical illiteracy. This
requires that everyone in the organization needs
to have a clear understanding of what Dr. Walter
­Shewhart established as a fundamental principle
for interpreting data aimed at QI. Specifically, his
distinction between common and special causes
of variation needs to be the primary way that
everyone in the organization approaches data
analysis and interpretation. Yet, the frequent
use of aggregated data and summary statistics
compounded by data presentations that rely on
xvi Introduction to the Second Edition
red/yellow/green displays of the data create major
challenges for most healthcare providers when it
comes to understanding variation. Options for
moving to a new way of thinking are offered in
this chapter.
Chapters 8 and 9 provide details on how
to understand variation statistically with SPC
methods. Specifically, Chapter 8 focuses on the
run chart and Chapter 9 dives into the details
of the Shewhart control charts. The statistical
and operational foundations for using these
practical statistical tools are discussed as well as
the basic elements of the charts, decision criteria
for selecting the appropriate chart, and rules for
detecting special causes of variation. These chapters
provide case studies as examples and exercises to
test the reader’s knowledge of selecting the most
appropriate Shewhart chart for various indicators.
Chapter 10 presents 16 case studies de-
signed to demonstrate how the various quality
measurement principles and tools can be applied
to healthcare situations. These examples span a
wide range of topics, including indicator selec-
tion and development, sampling applications,
and control chart analysis. The case studies
address clinical as well as operational aspects
of healthcare delivery. All of these case studies
are grounded in actual experiences I have had
in applying the QMJ principles. In some of the
cases the organization or individuals who were
kind enough to share their stories are identified.
In other case studies I have respected individual
wishes to remain anonymous. For all of those
involved with sharing their stories, I am grateful.
In the concluding chapter, Chapter 11, I move
away from the technical details of the QMJ and
address how an organization can s­ trategically
and operationally build quality thinking and
action into the very fabric of the daily life.
Four key strategies for accomplishing this are
discussed: (1) adopting quality as a business
strategy, (2) developing a learning system to
support improvement, (3) linking measurement
to improvement, and (4) building capacity and
capability for improvement.
My design for this book has been to start
rather broadly by establishing the context for
measurement. It then proceeds to become more
detailed and specific as I describe the milestones
in the QMJ and their technical components.
It then concludes by moving back to a broad
perspective describing how an organization can
establish the structures, processes, and culture
required to demonstrate that every day is truly
a quality journey. Some readers may want to
start at Chapter 1 and progress in a logical
sequence through the chapters. Others should
feel free, however, to jump around from chapter
to chapter depending on their interests. Each
chapter has been written to stand more or less
on its own. So, follow your own path and enjoy
the journey.
About the Author
Dr. Robert Lloyd is vice president at the Institute
for Healthcare Improvement (IHI). Dr. Lloyd
provides leadership in the areas of performance
improvement strategies, statistical process control
methods, development of strategic dashboards,
and building capability for quality improvement
(QI). He also serves as lead faculty for various
IHI initiatives and demonstration projects in
the United States, the United Kingdom, Sweden,
Denmark, Norway, New Zealand, Australia,
­India, Malaysia, Qatar, Dubai, and Africa. Before
joining the IHI, Dr. Lloyd served as the corporate
director of Quality Resource Services for Advocate
Health Care (Oak Brook, IL). He also served
as senior director of quality measurement for
Lutheran General Health System (Park Ridge,
IL), directed the American Hospital Association’s
Quality Measurement and Management Project
(QMMP), and served in various leadership roles
at the Hospital Association of Pennsylvania. The
Pennsylvania State University awarded all three
of Dr. Lloyd’s degrees. His undergraduate degree
is in sociology, his master’s degree is in regional
planning, and his doctorate is in rural sociology.
Dr. Lloyd has addressed over 2,500 national
and international meetings of professional
groups and associations. Over 200,000 par-
ticipants from the United States and abroad
have attended his classes and presentations on
QI. He has served as faculty for the Harvard
School of Public Health, the American College
of Healthcare Executives, the American Society
for Quality (ASQ), the University of Wisconsin’s
graduate program in administrative medicine,
Image Credit © Michal Steflovic/Shutterstock
the University of Chicago’s graduate program in
health administration, the Healthcare Forum, the
International Quality and Productivity Center,
the American Health Information Management
Association, the Joint Commission, the Group
Practice Improvement Network, the Ontario
Hospital Association, the Vancouver (British
Columbia) Quality Forum, the Medical Group
Management Association, the BMJ European
Quality Forum (London, Barcelona, Berlin,
Amsterdam, Goteborg, and Paris), and numer-
ous QI organizations around the United States.
Dr. Lloyd has also presented his seminars on
statistical thinking to physicians and adminis-
trators from the Federation of County Councils
in Stockholm and Jonkoping, Sweden, to leaders
of the National Health Service throughout the
United Kingdom and New Zealand, and to
patient safety leaders throughout Denmark,
South Africa, and Ghana.
He has published numerous articles, chapters,
and reports on a wide range of topics, including
QI theory and implementation, measurement
and statistical methods, clinical outcomes,
customer satisfaction, information systems, and
parish nursing.
Dr. Lloyd is coauthor of the internationally
acclaimed book Measuring Quality Improvement
in Healthcare: A Guide to Statistical Process C
­ ontrol
Applications (American Society for Quality Press,
2001, fifth printing). Dr. Lloyd’s second book,
Quality Health Care: A Guide to Developing and
Using Indicators was first published by Jones &
Bartlett Learning in 2004.
xvii
CHAPTER 1
Setting the Context
for Measurement
B
elieve it or not, there was a time when
data and measurement were not at the
center of the healthcare debate. For
example, when a patient went to see a doctor
back in the 1960s, 1970s, and 1980s, he or she
usually said, “Thank you very much, doctor. Do
I need to schedule a follow-up appointment?”
Today it is more likely that the patient will ask
questions such as: “Where did you get your
medical degree? How many times have you
done this procedure? What is your compli-
cation rate for this procedure? Do you have
data to show that this procedure will improve
my health? Have you had a malpractice claim
filed against you in the past 5 years?” Then,
if the patient is really actively involved in his
or her own healthcare delivery, he or she will
probably pull out a piece of paper and inform
the doctor, “This is what I found on the Internet
last night and here is what I think is wrong
with me. . . .”
Over the past two decades the healthcare
industry has moved away from being an industry
based on high levels of trust and partnership
between physicians and their patients to one
of gentle (and at times not so gentle) tension
© Michal Steflovic/Shutterstock
between those who provide care, those who
receive it, and those who pay for it. Whenever
society begins to lose confidence in an insti-
tution, however, there is typically a demand
for greater oversight of the institution and a
related push for more data on the products
or services it provides.1 This is precisely what
has happened and is happening within the
healthcare industry.2 As consumers, political
leaders, the media, and the public in general
have increasingly asked questions about the
efficiency, effectiveness, and reliability of
healthcare services, there has been a concom-
itant growth in the demand for more data on
healthcare providers and results.
For decades, providers have been collecting
what is classically referred to as administrative
data (e.g., in a hospital these data included the
number of admissions and discharges, percentage
of patients who die in the hospital, percentage of
patients discharged to home or nursing homes,
number of lab tests performed, visits to the
emergency department, number of setup and
staffed beds, resource usage, average length of
stay, and selected infection rates). These data
have routinely been used internally to make
1
2 Chapter 1 Setting the Context for Measurement
management decisions and then submitted
externally to various oversight or regulatory
bodies at the state, regional, or national levels
at designated points in time for aggregation.
Typically, these data are lagged anywhere from
a few months to a year or more.3 Analysis of
administrative data is usually performed using
summary statistics (e.g., the mean, median,
mode, minimum, maximum, range, and stan-
dard deviation) and comparisons by quarter
or year are often the preferred approach to
determine whether time period 1 is different
from time period 2.
Healthcare administrative data are still
collected on a regular basis around the world.
Yet over the past 15 years, three key develop-
ments within the healthcare industry have had
a profound impact on moving healthcare mea-
surement and data collection in new directions:
(1) the growing demand for greater transparency,
(2) the growing focus on patient-centered care and
concerns over poor service, and (3) the growing
role of quality improvement (QI) concepts, tools,
and methods.4
▸▸ The Growing Demand
for Transparency
Even though the discussions on transparency
in health care have increased significantly
over the past few years many people fail to
realize that this is not a new topic within
the healthcare profession. Three individuals
specifically played key roles in the earlier
debates on data disclosure in healthcare set-
tings: Florence Nightingale, Ernest Codman,
and Francis Peabody. Florence Nightingale
(1820–1910) is probably the best known of this
threesome. She was one of the first advocates
for data collection and release. Her work in
the Crimean War is legendary. She wrote
Notes on Matters Affecting Health, Efficiency
and Hospital Administration of the British
Army (1858), which detailed not only her
experiences in caring for the war’s wounded
but also provided detailed recommendations
for reforming the organization and delivery
of care within the British military hospitals.
She developed statistical graphs to summarize
the causes of mortality, known as Polar Area
Diagrams (FIGURE 1-1) and then used these
data to criticize the unsanitary conditions that
prevailed in military hospitals and decried the
incidence of preventable deaths. Although
she did have her critics, she was not vilified
nearly as much as the most ardent proponent
of transparency, Dr. Ernest A. Codman.
Dr. Ernest Codman (1869–1940) is best
known for his controversial “end results” sys-
tem. “The end result idea is simply that doctors
should follow up with all patients to assess the
results of their treatment and that the outcomes
actively be made public” (Swensen and Cortese,
2008, p. 233). As a prominent surgeon in Boston,
Codman was a firm believer in making data not
only more available to those who deliver care
but also that it should be provided openly and
freely to the public so they could make informed
decisions on which doctors and hospitals they
would use. His colleagues at Massachusetts
General Hospital and Harvard Medical School
(he graduated in 1895) favored many of his sug-
gestions and recommendations for improving
treatment procedures and outcomes as long
as the information and data stayed within the
medical profession. It was Codman’s insistence
that the public have access to these results
that created the major controversy. Codman
held fast in his beliefs but it was at a price. He
received stern criticism from his colleagues
at Massachusetts General Hospital when he
proposed the outlandish notion that data on
physician and hospital performance should be
released to the public:
I am called eccentric for saying this
in public; that hospitals, if they wish
to be sure of improvement, must find
out what their results are, must analyze
their results, to find out their strong
 The Growing Demand for Transparency 3

Death from wounds in battle

Death from other causes
Death from disease AU
April 1854 to March 1855 JULY

G
E

US
JUN

T
BULGARIA
Y

SE
A
M

PTE
APRIL

MBER
1854
CRIMEA

OCTOB
1855
RCH

ER
MA

NO
VE
M
BE
R
Y
AR
RU

B
FE
DEC
EMBE
R

855
NUARY 1
JA

FIGURE 1-1 Florence Nightingale’s Polar Area Diagram Showing Mortality in the Crimean War by
Cause (1855)
Nightingale, F., Notes on Matters Affecting the Health, Efficiency, and Hospital Administration of the British Army. Founded Chiefly on the Experience of the Late War. Presented by Request to the Secretary of State
for War. Privately printed for Miss Nightingale, Harrison and Sons, 1858.

and weak points, must compare their could further his practices of transparency and
results with those of other hospitals disclosure of patient results. In 1916, he pri-
and must care for what cases they can vately published a compendium of his hospital’s
care for well. Such opinions will not be results from 1911 through 1916 including the
eccentric a few years hence. (Codman, “clinical misadventures” that he and others had
1917, p. 183) made (Codman, 1916, 1924; Donabedian, 1989;
Neuhauser, 1990). Dr. Codman was clearly a
In 1914, his hospital refused his plan to man ahead of his time. He saw the medical
evaluate the skill and outcomes of surgeons. profession as a caring and compassionate
Eventually, he was denied privileges at Massa- discipline but one in which there was an obli-
chusetts General Hospital, which led Codman gation on the part of the providers of care to
to resign and establish his own hospital (inter- be open and forthright with the results of their
estingly enough called the Codman Hospital work. Unfortunately, this was a message that
and also the End Results Hospital) where he was not appreciated in the early 1900s by his
4 Chapter 1 Setting the Context for Measurement
colleagues. In 1914, Dr. Edward Martin wrote
to Codman the following:
Dear Codman:
God bless you! I suppose I should
hate you if I lived in the same town, but
my feeling, being remote, is quite other.
Indeed the very enemies who lurk in
second story windows with muffled
rifles are waiting your passing, are the
ones who take off their hats in deepest
respect as your cold, but beautiful, corpse
is carried away. (Mallon, 2000, 63)
Codman remained steadfast in his beliefs
and practices. In the end, however, the profes-
sion’s dissatisfaction with him and his insistence
on the end results system continued to grow.
Although no one actually took shots at him,
he did spend the last years of his life with few
patients, no referrals from his medical colleagues,
and little money. During his final year of life,
he wrote that in the future he hoped that he
would receive more favorable reviews than he
did while he was alive (Mallon, 2000). Berwick
in a Milbank Quarterly article (1989, p. 266)
offered a postscript on Codman’s life and work,
“Codman looked ahead. He looked, indeed,
beyond us. Seventy-eight years ago he began his
life’s work; forty-eight years ago he died. Are we
ready for him yet?”
The third individual who helped to set the
stage for a dialogue on transparency is Francis
Peabody (1881–1927). Although less well known
than Florence Nightingale and certainly less
controversial than Ernest Codman, Dr. Peabody
made major contributions to the healthcare
field as a clinician, teacher, and researcher.
He is best known for stressing transparency
with individual patients and involvement
of the patient in making care decisions. His
basic philosophy was captured in a series of
lectures to students at Harvard Medical School
in 1926 and subsequently published under the
straightforward title, “The Care of the Patient”
(Peabody, 1927). Several key quotations from
Dr. Peabody’s classic 1927 article provide not
only guidance but also a challenge for today’s
medical professionals:
The treatment of a disease must be
completely impersonal; the treatment of
a patient must be completely personal
(1927, p. 878).
The good physician knows his
patient through and through, and his
knowledge is bought dearly. Time,
sympathy and understanding must
be lavishly dispensed, but the reward
is to be found in that personal bond
which forms the greatest satisfaction
of the practice of medicine. One of
the essential qualities of the clinician
is interest in humanity, for the secret
of the care of the patient is in caring
for the patient (emphasis added by
author) (1927. p. 887).
The fact that Dr. Peabody stressed the need
for transparency between the physician and the
patient is probably one of the key reasons that
he was highly regarded by his colleagues and
Dr. Codman was not. Codman’s insistence on the
public disclosure of all hospital and individual
practitioner data challenged the existing status
quo. Peabody on the other hand was seen as a
compassionate and caring physician who was
devoted to the individual patient’s needs and
expectations. This was something that could be
addressed quietly in the privacy of the doctor’s
office and in one-to-one conversations with
the patient. Such behaviors did not threaten
the status quo of the medical profession as
did public postings and disclosure of results
proposed by Codman. Furthermore, Peabody
did not stress that the individual physicians
who did not engage in these open exchanges
with patients should be publicly identified. It
was, in his view, all a matter of personal taste
and preference, and the results should be
discussed only between the physician and the
patient. For additional detail on the life and
practice of Dr. Peabody, see Oglesby (1991)
and Hurst (2011).

If we fast forward from the transparency
pioneers of the early 1900s, we realize that many
of the barriers they campaigned against are still
with us today. Despite the growing push for
greater transparency in healthcare data, there is
still considerable confusion and lack of consensus
surrounding this topic. The classic definition of
transparency (Webster’s Dictionary, 1984) is that
it is something that is easily detected; obvious;
readily understandable; clear; without guile or
cover, candid; clearly perceived; lucid. This seems
to provide a reasonably straightforward set of
definitions and criteria. But in a healthcare set-
ting, the question of transparency has not been
quite so straightforward. A colleague of mine, for
example, told me he thought that transparency
is “A frequently used term in healthcare settings
that has no apparent consistent definition or
meaning.”
Transparency has been and will continue to
be a challenge conceptually, operationally, and
politically. A few examples of recent transparency
challenges are worth reviewing.
■■ An article in the Chicago Tribune (March 28,
2013) had the following headline, “Do Hos-
pital Ratings Bring Clarity or Confusion?”
The story led with the following conclusion:
“Patients left to judge credibility of rankings
groups and an array of data.” The article
focused on St. Mary Mercy Hospital in
Livonia, Michigan. The hospital received
rankings in terms of quality and safety by
four different organizations. The results
showed that the Leapfrog Group gave the
hospital an A rating. Healthgrades named
the hospital as one of the top 50 hospitals
in the United States. At the other end of the
spectrum, however, the Joint Commission
(JC) and U.S. News & World Report excluded
St. Mary Mercy hospital from their lists of
“best hospitals.” Finally Consumer Reports
gave the hospital an average rating of 47
out of a possible 100 points. The article also
describes how the Medicare program will
be reducing the hospital’s payments owing
to poor performance related to Medicare
The Growing Demand for Transparency 5
quality measures. If you were a patient
living in the Livonia, Michigan area and
trying to decide which hospital to select
for your knee replacement surgery, how
would you use these results? The data are
made public, so this is not so much an issue
about reporting data to the public but rather
a lack of consistency in developing public
reporting methods and systems, which is one
of the serious issues related to transparency
in the United States.
■■ The April/May 2013 issue of AARP: The
Magazine made its first foray into the
transparency arena when it published an
exclusive report titled “America’s Safest
Hospitals: Is Yours on the List?” When you
read the article, however, you discover that
AARP (formerly American Association of
Retired Persons) did not do the assessments
but partnered with the Leapfrog Group to
prepare the report using the Leapfrog Group’s
methodology and their 26 measures. A total
of 66 “safety superstar” hospitals are listed
in the AARP report. Although singling
out 66 of the 5,724 hospitals in the United
States may bring comfort to those who
live near these facilities and recognition
to the hospitals, it makes you wonder how
well these 66 institutions would fare if they
were assessed by the other organizations
that rated St. Mary Mercy Hospital (i.e.,
Healthgrades, the JC, U.S. News & World
Report, and Consumer Reports).
■■ Even though AARP The Magazine published
the safest hospital list in their April/May
2013 edition, the July/August edition of
another AARP publication, AARP Bulletin,
included an article titled “Lifting the Veil on
Hospital Rates.” This piece lauded the recent
releases by the government showing what
U.S. hospitals charge for various inpatient
procedures, noting that this “may be ushering
in a new era of transparency in healthcare
costs, which have long been closely guarded
by hospitals” (AARP Bulletin, July/August,
2013, p. 4). Although the release of such data
is a step in the right direction, the challenge
6 Chapter 1 Setting the Context for Measurement
is that what hospitals charge for a procedure
may have little to do with what it costs them
to produce this procedure or service and
may be a far cry from what Medicare (or
an insurance company) pays for the proce-
dure. As the article points out the data show
“wild disparity in billing rates for the same
procedure.” One example cited references
a Houston hospital that charged patients
$126,157 for a hip replacement whereas a
hospital in Appleton, Wisconsin charged
$26,787 for the same procedure. They point
out that Medicare pays a hospital about
$14,000 for a hip replacement procedure.
The article concludes with a quotation from
Stuart Guterman of the Commonwealth
Fund: “the data demonstrate that hospital
charges have nothing to do with the cost or
quality of care. It’s becoming less and less
clear what hospital charges are based on. The
long term goals should be to make hospitals
charges relate to what the actual cost of care
is” (2013, p. 47). A similar article in the
Chicago Tribune (“Hospital Sticker Shock:
Prices, It Turns Out, Can Vary Widely” May
18, 2013, 10) provides charges for different
procedures by hospital and arrives at the
same basic conclusion as the AARP Bulletin
article, that charges show wide variation for
the same procedure. Yet, they take a positive
stance on the benefits of such attempts at
transparency. They conclude that any data
on doctor and hospitals performance has
value. They write: “The more information
about doctor and hospital prices and quality
of care the better the choice. No people won’t
visit or shun a hospital only because of its
price list. But the pressure is on hospitals.
They need to explain why they’re charging
double or triple for the same procedure as
the hospital down the street.”5 So, even when
there is a spirit of transparency the data and
hence the conclusions being derived from
the released data may in fact be confusing,
especially for consumers.
■■ A more dramatic response to the release
of hospital ratings comes from Dr. Ezekiel
Emanuel, the University of Pennsylvania
bioethicist and oncologist who helped
the White House draft the healthcare
reform law. In the July 25, 2014 edition
of The Wall Street Journal Dr. Emanuel’s
sharp criticism of the U.S. News & World
Report’s “Best Hospitals” rankings starts
with the claim that they are “flawed to the
point of being useless.” He continues his
critique by pointing out that the rankings
are based on an overreliance on reputation
and a failure to take into account quality
indicators, such as hospital-acquired in-
fections and the incidence of preventable
falls and pressure ulcers. His conclusion
is that hospital rankings are good only for
“media hoopla and a few chest-thumping
press releases from hospitals at the top of
the list.” Reading this story caused me to
reflect on when I worked for the Hospital
Association of Pennsylvania during the
mid-1980s. The first hospital ratings re-
port issued by the Pennsylvania Health
Care Cost Containment Council (PHC4)
was received with mixed reactions. Yet, a
few of the hospitals that were ranked in
the top decile of about 300 hospitals took
it upon themselves to place ads in local
newspapers extoling their good outcomes
and high quality. Interestingly enough
the next report released by the PHC4 was
not so kind to some of these top-ranking
hospitals. In turn, there were a few ads in
local newspapers placed by competing
hospitals pointing out that those who were
previously ranked in the top decile had now
fallen from grace. The ranking sword can
cut both ways! The Healthcare Business
Blog from Modern Healthcare documents
this story at http://www.modernhealthcare
.com/article/20130725/BLOG/307259986.
■■ If we move from hospital transparency
to physician transparency the challenges
become even more pronounced. If Dr.
Codman was alive today he would prob-
ably feel that not much progress has been
made since he advocated for his end results

system over 110 years ago. Finding data
on physician volumes and results can be
a real challenge. Some of the state data
commissions (e.g., the PHC4) do offer data
on the volume of procedures for selected
surgeries for surgeons by name. For hip
and knee surgery, for example, the PHC4
lists individual surgeons by name and the
number of procedures done, but their results
on quality measures (i.e., infections, blood
clots, readmissions, and length of stay) are
not made public. Postings of doctor-specific
results have been received well by the public
but are still being resisted by many state
medical societies.
■■ A recent issue for transparency in the
physician arena is the manner in which
physician payments are determined. The
American Medical Association (AMA),
the chief lobbying group for physicians in
the United States, meets confidentially each
year to determine what values (time and
intensity of a procedure) should be assigned
to the services and procedures performed
by physicians. These value assessments are
then turned over to the Centers for Medi-
care and Medicaid (CMS) to establish the
actual payments being made to physicians
for treatment of eligible Medicare patients.
A recent investigation into this practice
(Chicago Tribune, July 26, 2013, section
2,10) revealed that the values and subse-
quently the payments to doctors are “so
exaggerated that many doctors ‘averaged’
more than 24 hours of work per day.” In a
study of 340 doctors at outpatient surgical
clinics in Florida, it was discovered that
the doctors performed at least 16 hours of
procedures each day, even though most of
the clinics were open for about 10 hours
each day. The study also discovered that
78 of the 340 doctors actually fit 24 or more
hours of work into a day. Twenty-one of the
doctors actually filed reports (and claims
for payment) that showed they fit over
30 hours of procedures into a 24-hour day.
As a result of this new transparency in
The Growing Demand for Transparency 7
the way physician payments are derived a
bipartisan group of legislators has drafted
a bill that would require the reshaping of
the way Medicare pays doctors.
■■ The issue of transparency as I mentioned
earlier, is not restricted just to the United
States. In England, for example, The Daily
Telegraph (June 13, 2013, no. 49, 157) had
the following headline on the front page:
“How Bad Doctors Can Hide Failings.” The
story details how the National Health Service
(NHS) developed “league tables” that would
show doctor-specific results for selected
surgeries and treatments. All this seemed
like a good idea designed to assist patients
in selecting “top performing doctors.” But
the newspaper’s reporters discovered that
the NHS has been contacting doctors and
asking whether “their” data can be released.
Those who refuse to have their results re-
leased will not have data relating to their
performance published and they will not be
identified for refusing to participate in the
release. A high-ranking government official
told the newspaper that this was a “farcical
situation.” Jeremy Hunt, the Health Secre-
tary for the English NHS, stated that “The
medical profession has closed ranks to stop
patients finding the truth.” He concluded
that “transparency and participation must
be the operating principles of the NHS.
They can lead to more effective health care,
better outcomes, greater accountability and
efficiency.” In response the Royal College
of Surgeons said that doctors are covered
by data protection laws and that therefore
they must give their consent before their
data can be released to the public. Yet Mr.
Hunt indicated that the NHS is proceeding
to release individual surgeon data within
several months on 10 different surgical spe-
cialties. They then expect to release data on
all doctors, including general practitioners,
over the next year and a half. Referencing
how the release of similar data on individual
surgeons in New York State put pressure on
poor performing doctors and led to improved
8 Chapter 1 Setting the Context for Measurement
treatment and outcomes, the NHS Depart-
ment of Health concluded: “Patients should
be able to see how individual senior doctors
are performing. If there are legal grounds for
individual doctors opting out, any patient
and their family would be entitled to ask
why and may prefer their operation to be
carried out by someone who was prepared
to be fully transparent” (Daily Telegraph,
June 13, 2013, no. 49, vol. 157, 2).
In the midst of all this controversy and the
challenges surrounding transparency, there are
some guiding lights. There does seem to be a
growing realization, even on the part of providers
of care, that greater transparency in health care is
not only a good thing but an inevitability. So, how
can leaders become better prepared to ­address
the transparency issue? Dr. Jim Reinertsen,
a senior fellow at the Institute for Healthcare
Improvement (IHI), developed guidance for
increasing an organization’s awareness of and
practice in transparency. He calls it “how to go
naked” (Reinertsen, 2012). Reinertsen identifies
four specific things leaders can do to motivate
greater transparency:
■■ Start undressing at the top. By this he means
that board reports are too often all about
good news, which leads the board members
to think that quality and safety are much
better than they really are. He encourages
senior leaders to clearly identify the good,
bad, and ugly events for the board without
rationalizations and language that “explains
away suboptimal performance.”
■■ Don’t hide behind your lawyer. Dr. R
­ einertsen
stresses that too many healthcare organi-
zations’ lawyers “require that safety data
discussions occur behind a thick veil in
apparent belief that they are protecting
the institution from legal risks” (p. 42). He
suggests having a dialogue on this stance is
critical to an organization’s orientation to
transparency. He also points out that as one
health system lawyer told him, “Patients don’t
learn they got hurt by seeing a PowerPoint
slide. They don’t sue us because of our data.
They sue us because of broken relationships.
How can we get on with the job of making
care better if we can’t talk openly about
what goes wrong?” (p. 42). Increasingly
healthcare providers are being more open
with their data despite the legal concerns
of some. It is not uncommon, for example,
to see data on the number of days since the
last pressure ulcer or fall posted on the wall
of a hospital unit. This is occurring more in
countries outside the United States where
malpractice claims are not as prevalent but
there are plenty of organizations that are not
hiding behind their lawyers.
■■ Keep it simple. This point is an extension of
the previous recommendation. The essence
of it is that you do not need to purchase and
install an expensive, complex, and automated
data repository in order to understand
the current status of quality and safety
within your organization. Plotting run or
control charts by hand and posting them
on the unit for staff and the public to see
provide the foundation for improvement.
Rather than wait for monthly or quarterly
data to be aggregated in the expensive,
complex, and automated data repository,
verified, and then distributed back to the
departments and units frequently weeks or
months after the fact, this recommenda-
tion stresses (1) establishing a baseline on
current performance, (2) posting data as
close to production as possible (e.g., daily
or weekly), and (3) making sure that even
suboptimal performance is posted when it
occurs rather than calculating averages that
meld the anomalies into large aggregations
and make them invisible.
■■ Shape up! The final recommendation on
how to go naked is based on a phrase from
Tapscott and Ticoll (2003), “If you’re going
to be naked, it’s good to be buff.” Basically
this refers to the fact that patients do not
seem to make choices about healthcare
providers (doctors or hospitals) based on
publicly reported data, even when these
data are available. But when staff sees data

that show their department or unit is not
performing well they do take notice and
start to ask how they can shape up. Because
staff and managers are frequently engaged
in the details of day-to-day work, they
frequently do not step back and ask, “How
are we performing over time?” Caregivers
struggle with the N = 1 challenge; that is,
they think of the individual patient, surgery,
or procedure and generally do not look at how
their systems and processes are performing
for groups of patients over time. This would
be similar to a person who wants to get in
shape and lose weight saying, “I ate only a
salad yesterday and some fruit, so why do
I weigh the same today?” The question of
getting into shape is not answered by the
detailed analysis of one meal or even one
exercise session or the average hand hygiene
compliance for the year. The answer lies in
being able to track performance over time,
not at a single point in time. Improvement is
more like running a marathon rather than
engaging in a sprint.
Some organizations have started to adopt
some of these suggestions for “going naked.”
For example, in an effort to promote price
transparency Steven Sonenreich, president and
chief executive officer of Mount Sinai Medical
Center in Miami Beach, Florida, made a promise
during an interview with a local radio station
to post the contract rates Mount Sinai Medical
Center pays private payers for diagnoses and
treatments. Sonenreich then went a step further
by challenging all other hospitals in the Miami
area to do the same. Sonenreich gained national
attention for his candid promise of transparency
and his challenge to the other hospitals (Gamble,
2013). This example illustrates what can be done
to change the status quo and move toward greater
transparency not only of pricing structures but
also results and processes. Hospitals that make
public promises, set measurable goals, and ex-
ecute strategies to share price information and
results can use this transparency as a competi-
tive advantage. If one hospital in a community
The Growing Demand for Transparency 9
demonstrates transparency with its data while
other hospitals hold back, this will undoubtedly
send a message to patients, political leaders, and
groups that commission healthcare services
that at least one provider is not hiding behind
its lawyers or being secretive. The transparent
hospitals will benefit from increased patient and
public trust, which is especially valuable in this
time of growing mistrust of healthcare providers.
Reinertsen’s recommendation on “how to
go naked” provides a starting point for dialogue
on where the organization stands with respect
to transparency. To dive even further into the
specifics of how transparency will play out in
your organization, I suggest that you address
three critical questions:
■■ Transparency of what?
■■ Transparency for whom?
■■ Transparency at what level?
I developed the Transparency Assessment
Tool (TABLE 1-1) as a way to help build a dialogue
around this topic. There is no right or wrong
answer to each of the 15 questions in this as-
sessment. Without such a dialogue, a provider of
healthcare services will be placed in a defensive
posture when asked to reveal the quality of the
care they offer or their results. The time to think
about transparency is not when the Channel 5
TV mobile unit parks outside your hospital or
clinic, cranks up their satellite dish, and asks to
“talk with someone in charge” about why your
data placed you in the bottom decile of a recent
rating and ranking of providers.
In order to gain a clear understanding of the
full range of opinions on transparency within
your organization, the following questions need
to be addressed:
■■ Do you know your data better than anyone
else?
■■ Do you use data that are made available
to the public to identify opportunities for
improvement?
■■ Or, do you look for ways to deny the public
released data and develop rationalizations
as to why you think you are actually better
than the reported outcomes?
 10

TABLE 1-1 Transparency Assessment Tool

Strongly Strongly
Level Frequency of Transparency Agree Agree Not Sure Disagree Disagree

1. Greater transparency is needed across all healthcare settings

and providers.

2. Patients should be able to compare hospitals as easily as they

do cars and other products.

3. Results on hospital outcomes (mortality, infections, falls,

med errors, etc.) should be made public once a year.

4. Results on hospital outcomes (mortality, infections, falls,

med errors, etc.) should be made public twice a year.
Chapter 1 Setting the Context for Measurement

5. Results on hospital outcomes (mortality, infections, falls,

med errors, etc.) should be made public four times a year.

6. Results on groups of doctors (surgeons, GPs, intensivists,

dentist, etc.) should be made public once a year.

7. Results on individual doctors should be made public

once a year.

8. All clinical outcomes on hospital performance should be

made available to the public.
TABLE 1-1 Transparency Assessment Tool (continued)

Strongly Strongly
Level Frequency of Transparency Agree Agree Not Sure Disagree Disagree

9. Operational outcomes on hospital performance (wait times,

referral times, access) should be made available to the public.

10. Patient satisfaction results for each hospital should be made

available to the public.

11. Financial results (including salaries) for each hospital should be

made available to the public.

12. Mortality rates for individual surgeons should be made

available to the public.

13. Infection rates for individual physicians should be made

available to the public.

14. Errors and harm rates for individual physicians should be made
available to the public.

15. Salaries of individual physicians should be made available to

the public.

©2017 R Lloyd All Rights Reserved. Reprinted by permission of Robert Lloyd.

The Growing Demand for Transparency
11
12 Chapter 1 Setting the Context for Measurement
■■ Do you share all your results openly with staff?
■■ Do you share all your results with patients,
family members, and caregivers?
■■ If not, why not?
The specific steps I recommend in using the
Transparency Assessment Tool include:
■■ Distribute the assessment tool to the organi-
zation’s leadership team. This could include
the governing board and senior and middle
management groups.
■■ Ask each person to complete the tool
without talking to other members of the
leadership team.
■■ Have the completed assessments tabulated
to show the spread of responses for each
question and distribute the summary back
to the participants for review and reflection.
A graphic summary rather than a tabular
or numeric summary is a preferable way to
report the results.
■■ Convene a leadership meeting to discuss
the results and the variation within and
among each of the questions.
■■ Finally, create a work group to begin drafting
an organizational position on transparency
and how the organization will respond to
the specific types and levels of transparency
referenced in the assessment tool.
In summary, the work that Nightingale,
Codman, and Peabody started years ago con-
tinues today. This issue will not be going away
anytime soon and will only grow in intensity over
the next 5 years. Organizations certainly have a
choice in building a philosophy and strategy for
dealing with the growing pressures for a more
transparent healthcare industry. Some will meet
this challenge and approach it from a proactive
perspective. They will build measurement sys-
tems that enable them to understand their data
better than anyone else and share it internally
and externally before they are forced to do so.
Others will sit back and “hope” that their data
will at least meet standard expectations or that
their performance is more or less “about average.”
They will wait till data are released on them by
external bodies and then circle the wagons to
justify or rationalize why they end up where
they do in the rating or ranking system. Life is
full of choices. The choices you make over the
coming year related to transparency of your
performance will have a major impact on your
organization’s future.
▸▸ The Growing Focus on
Patient-Centered Care
and Service
The second major development that has had
and will continue to have a dramatic impact on
the role of measurement in healthcare settings
is the growing concern by both consumers and
political leaders that healthcare professionals
not only do not listen very well but in many
instances act as if they just do not care about
those they serve. This is a stark criticism for an
industry that supposedly is designed to “care for
people” and a far cry from what Francis Peabody
was teaching back in the early 1900s about the
“care of the patient.”
I am always intrigued at the response I get
when I ask participants in one of my workshops,
“Why did you get into health care?” The most
frequent response is something along the lines
of, “Because I want to help people and make a
difference.” Although this is a fairly consistent
response, irrespective of the country in which
I ask it, it is interesting to follow up on such re-
sponses by unobtrusively walking the halls of a
hospital, clinic, or nursing home and observing
the interactions of staff and patients to gain a
sense of the quality of these interactions. Even
though most healthcare professionals will say
that the patient and family are at the center of
what we do, the casual observer walking around
the hospital will probably conclude that there
seems to be a conflict between what healthcare
professionals frequently tell you about their rea-
sons for entering the profession and the degree
to which this intention is demonstrated moment
to moment and patient to patient.
 The Growing Focus on Patient-Centered Care and Service
All too often, the patient is not really placed
at the center of the healthcare universe. Instead,
we frequently place ourselves in that center
and expect the patient and family to rotate
around the structures and processes that meet
our wants and needs. For example, we tell a
patient, “Oh, I’m sorry, we are open here at the
outpatient testing and therapy center only from
9 a.m. until 4 p.m. Monday through Friday. We
are closed from noon to 1 p.m. for lunch. No, I
am sorry, we are not open during the evenings
or on the weekends.” If we really placed the
patient at the center of our universe, we would
offer hours that accommodate patient needs
and schedules. Just imagine if restaurants,
movie theaters, or shopping malls took this
approach and were open only during times
that fit the workers’ schedules. They would be
out of business very quickly. Customers would
walk away from these businesses and look for
those that accommodated their requirements.
It is actually rather surprising that it has
taken the consumers of health care so long to
become concerned about the lack of listening in
health care and start to take action. Historically,
the healthcare profession has been characterized
as a profession that tells patients what they want,
need, or can expect rather than asking them for
their opinions or desires. The standard approach
historically has been that healthcare providers
do things to people or for people rather than with
people. Francis Peabody argued that engagement
with the patient was essential. But it has taken
decades to chip away at the old paradigm that
places the providers of care at the center of the
universe.
A variety of books, reports, and consumer
organizations supported by the growing push
for transparency and the availability of data and
information on the Internet, all have set the stage
for creating greater involvement of the patient
and family in making healthcare decisions. It
has also forced healthcare providers to rethink
how they measure the voice of the customer
(VOC). Patient advocacy groups, in particular,
have been very instrumental in driving the
emergence of a more participatory model of
13
healthcare delivery. One of the early leaders in
this area has been a rather controversial physician
by the name of Charles Inlander. Inlander, with
backing from Robert Rodale, board chairman
of Rodale Press and publisher of Prevention
magazine, became president of the People’s
Medical Society (PMS) in 1983. Inlander, Levin,
and Weiner then wrote Medicine on Trial in
1988. This book was not merely a criticism of
the medical status quo; it was a direct attack
on the very foundations of medical training,
practice, and the business of medicine. The
foreword to Medicine on Trial states:
It is a catalog of ineptitude, malfeasance,
gross neglect, indifference, and incredible
arrogance. The indictment is not ours.
It is an unabridged exposé of medical
mistakes taken directly from the annals
of medicine and public research. No
one can count on reform from within
the medical establishment. There are
simply too many vested interests in
the system trying to protect the status
quo. (p. 11–12)
The authors conclude, “It is amply clear
that the medical care system is not capable of
significant and sustained efforts to improve the
quality of its services. It simply has too much at
stake in preserving benefits to its own members.
Their proposal for improvement lies with the
people who are the recipients of care” (p. 15).
They write, “The people who should—and, indeed,
must—alter the practice of medicine are the
people it is practiced on” (p. 19). The remainder
of the book provides detailed examples of how
the healthcare system has failed those it serves.
Chapter 12 of this book has an intriguing
title: “What Did He Say? What Did I Hear?
What Did We Do?” In this chapter, the authors
cite case after case of how providers (especially
physicians) have demonstrated a total lack of
listening to patients. They cite a study at UCLA’s
Cancer Rehabilitation Project, for example, that
concluded, “9 out of 10 physicians had never
received any formal training in how to disclose
14 Chapter 1 Setting the Context for Measurement
a cancer diagnosis to an afflicted patient”
(p. 192). Although many of the clinical recom-
mendations of the PMS have been called into
question for promoting “unscientific methods”
(Barrett, 2011), Inlander et al. at the PMS did
contribute to rejuvenating the early messages
of Codman and Peabody about the role of the
patient in medical practice.
In 1988, a second book was released that
served as a complement to Medicine on Trial.
Taking Charge of Your Medical Fate (Horowitz,
1988) provided a roadmap to guide the patient
through the healthcare maze. This book actually
received more recognition than Medicine on
Trial especially when Senator Edward Kennedy
claimed that “the Horowitz method can save
your life—the way it saved my son’s.” The au-
thor, Lawrence Horowitz, MD, was director of
the U.S. Senate Subcommittee on Health and is
considered a very astute observer of the healthcare
industry. He meticulously laid out the steps for
patients to take control of their own healthcare
decisions and how to behave when interacting
with healthcare professionals. Despite Horowitz’s
efforts, however, consumers of the late 1980s
did not take charge of their own medical fate.
It was not until the early 1990s that the
consumer movement in health care actually
started to gain serious momentum. Television
talk shows, magazines (e.g., on January 22, 1996,
Time magazine had the following cover page:
“Special Investigation—What Your Doctor Can’t
Tell You”), investigative reports (e.g., the TV show
60 Minutes), and the Internet all contributed to
the growing concern that healthcare providers
really did not listen or pay much attention to
those they serve.
A third book that made a significant contri-
bution to the role of consumers and measurement
in health care is Demanding Medical Excellence:
Doctors and Accountability in the Information Age
by Michael Millenson (1999). This book picks up
where the works of Inlander et al. and Horowitz
left off. Millenson provides a detailed account
of the major historical and political events that
have shaped the U.S. debate over cost and qual-
ity. He then proceeds to offer a critical review of
the serious flaws and fallacies in contemporary
medicine. Through many examples, quotations,
and case studies, Millenson traces the growing
role of data and outcomes research in health care.
He concludes that “the largest barriers to system-
atically measuring and improving the quality of
American medicine are not technical but cultural”
(p. xvi). He argues that the healthcare profession
lacks the will to objectively measure and monitor
the quality of what it produces and that it is only
a matter of time before the consumers become
more engaged in changing the profession.
One of the more consistent advocates for
patient involvement over the past 20 years has
been Consumers Union and their related pub-
lication Consumer Reports. Starting in 1996, for
example, Consumer Reports began publishing
ratings and rankings on health maintenance
organizations (HMOs). In January 2003, Con-
sumer Reports added a new dimension to its
focus on health care by leading with the cover
page headline, “How Safe Is Your Hospital?” In
2013, Consumer Reports and AARP both started
publishing what they consider to be the “top”
hospitals in America.
The publications that probably have had the
most impact worldwide on patient involvement
as well as quality and safety have been those
released by the Institute of Medicine (IOM).
In 1999, the IOM released To Err Is Human:
Building a Safer Health System, which addressed
primarily the fact that health care is essentially
an unsafe enterprise. The chair of the committee
that produced the report, William Richardson,
states in his preface: “This report describes a
serious concern (i.e., medical error) in health-
care that, if discussed at all, is discussed only
behind closed doors” (p. vii). After outlining
the current state of medical error in health care,
offering estimates of the number of patients
experiencing needless death and harm while
under care, and explaining error theory, the
report provides a series of recommendations on
ways to improve the healthcare system, several
of which address measurement and transpar-
ency issues. In the second IOM report, Crossing
the Quality Chasm: A New Health System for the
 The Growing Focus on Patient-Centered Care and Service
21st Century (2001), the focus was on how the
healthcare system can be redesigned to promote
innovation and improve care. Redesign refers to
“a new perspective on the purpose and aims of
the healthcare system, how patients and their
physicians should relate, and how care processes
can be designed to optimize responsiveness to
patient needs” (p. ix). The report proposes six
aims for healthcare improvement. These aims
are aligned with key dimensions in which today’s
healthcare system “functions at far lower levels
than it can and should (p. 5). The six IOM aims
are that health care should be:
■■ Safe
■■ Effective
■■ Patient centered
■■ Timely
■■ Efficient
■■ Equitable
Although all six aims ultimately are directed
toward the patient’s healthcare experience and
outcomes, the third aim explicitly addresses
the patient-centered issue. The IOM defines
patient-centered care as “providing care that is
respectful of and responsive to individual patient
preferences, needs, and values and ensuring
that patient values guide all clinical decisions”
(2001, p. 6). Inclusion of this explicit aim on
patient-centered care provided a much needed
stimulus for rekindling the vision that Francis
Peabody had for the healthcare profession.6
The IOM reports raised the consciousness
of many healthcare leaders to realize that poor
quality, unsafe processes, and indifference to
patients and their situations were unacceptable.
The reports moved the dialogues about trans-
parency, service excellence, and measurement
to the forefront of healthcare professionals’
minds. They also provided the basis for the
third key development within the healthcare
industry that has had a profound impact on
moving healthcare measurement and data
collection in new directions, namely, making
QI concepts, tools, and methods part of the
operating strategy of many healthcare providers
around the world.
15
The Growing Role of Quality
Improvement Concepts, Tools,
and Methods
There is no doubt that the healthcare industry
is under tremendous pressure to demonstrate
that it can transform itself. We have responded
extremely well in many arenas. For example,
the technological advances in medicine have
been dramatic.7 The industry has also been very
creative in developing a variety of outpatient
clinical and support services (e.g., mobile dental
clinics, home care services for individuals with
special needs, and collaborative initiatives that
have brought together healthcare providers as
well as community groups to improve population
health). What has not occurred, in my opinion,
is the full-scale adoption and diffusion (Rogers,
2003) of the concepts, tools, and methods of
QI. The same approaches to QI that have made
many manufacturing companies very successful
and known throughout the world are not being
fully embraced by the health, social services,
and educational industries. There are numerous
health and social service organizations that
can be singled out for their adoption of QI
strategies, concepts, and applications. But the
widespread adherence to quality principles and
constancy of purpose for QI has been spotty at
best (Deming, 1992).
Over the years a number of national and
international organizations have worked tire-
lessly to spread the word about QI and how it
can contribute to long-term survival, increase
profits, and add value for customers. Probably,
the longest standing of these organizations is the
American Society for Quality (ASQ). Founded
in 1946, ASQ has been leading the application
of QI thinking throughout the world and across
industries. With 25 different topic and industry
divisions in over 140 counties and nearly 300,000
members worldwide ASQ can rightfully claim
that they serve as the “Global Voice of Quality.”
In the 1980s, ASQ began expanding beyond
manufacturing and working to get quality
thinking into education, service industries, and
16 Chapter 1 Setting the Context for Measurement
health care. Although the healthcare division is
one of the smaller subgroups within ASQ, it has
become a viable group of professionals who are
keen to learn from other industries.8
Within the healthcare field the IHI has been
a leading innovator in health and healthcare
­improvement worldwide. For more than 28 years,
the IHI has partnered with national and local
leaders as well as frontline practitioners to im-
prove the health of individuals and populations.
The IHI work is focused on five key areas to
advance its mission: (1) building capability for
improvement within the healthcare professions;
(2) enhancing person and family-centered care;
(3) making patient safety more reliable; (4) ad-
dressing issues of quality, cost, and value; and
(5) advancing the triple aim for populations
(i.e., simultaneously enhancing the health of
populations, reducing costs, and improving the
quality and service components of the hospital
experience).9
Another influential leader in promoting QI
within the healthcare field, especially through
quality measurement, has been the National
Quality Forum (NQF). The NQF was created
in 1999 in response to recommendations from
the President’s Advisory Commission on Con-
sumer Protection and Quality in the Health Care
Industry. The commission recommended that a
forum for healthcare quality measurement and
reporting should be established to (1) develop
a plan for implementing quality measurement,
data collection, and reporting standards through-
out the healthcare community; (2) establish
measurement priorities focused on national
aims for QI; (3) endorse quality measures and
standardized methods for measurement and
reporting; (4) ensure the public has access to
quality measurement and performance data;
and (5) support the development of health
information technology systems to advance
measurement efforts.10
Finally, it should be noted that the JC and
its affiliate Joint Commission International (JCI)
have a long history of establishing standards
for quality and performance that can be traced
back to the pioneering work of Dr. Codman.
The JC was officially established as a not-for-
profit organization in 1951. It emerged as a
result of a collaborative effort by the American
College of Physicians, the American Hospital
Association (AHA), the AMA and the Cana-
dian Medical Association who joined with the
American College of Surgeons as corporate
members to create the JC on Accreditation of
Hospitals (JCAH) based in Chicago, Illinois.
At the time of its creation, the JCAH’s primary
purpose was to provide voluntary accreditation
and standardization of practice grounded in
the “end results” thinking of Dr. Codman.
Over the years, the JCAH changed its name to
reflect a broadening of its scope and coverage.
Although one of many accrediting bodies,
the JC is a leading force in promoting quality
and safety across a very broad spectrum of
healthcare services. In the United States alone,
it accredits over 21,000 healthcare organiza-
tions and related healthcare services. JCI is
currently accrediting healthcare organizations
in Asia, Europe, the Middle East, Africa, and
South America. To learn more about the JC, its
history, and work around the world visit their
website at https://www.jointcommission.org
/about_us/history.aspx.
In addition to the numerous organizations
that focus on the concepts and methods of
QI, there are many professional organizations
that are oriented toward supporting specific
interest groups (e.g., the AMA, the AHA, the
Voluntary Hospital Association, the American
Nurses ­Association, the Veterans Hospital
­Association). If you drop down one more level,
you start to discover a myriad of organizations
and groups that advocate for various clinical
and medical specialties and subspecialties
(e.g., the Society for Thoracic Surgery, the
American Osteopathic Association, the Royal
College of Surgeons [United Kingdom], the
Royal College of Surgeons of Edinburgh, the
Pharmacists’ Association of Saskatchewan).
My point in providing this brief historical
review is to demonstrate that today there are
many groups and organizations supporting,
endorsing, and sponsoring QI meetings,
 The Growing Focus on Patient-Centered Care and Service 17

conferences, publications, and programs for
healthcare professionals. Quality has become
a very popular phrase from the boardroom
to the frontline staff. Who does not want to
support quality? In many ways, quality has
become a very popular buzzword that lacks a
consistent and universally accepted definition.
Part of the challenge in defining quality is that
everyone is essentially an expert in quality. We
have all experienced good, bad, and downright
ugly quality. We may not be able to define it or
articulate the criteria precisely, but the general
opinion is that “I know it when I see it or experi-
ence it.” When I teach introductory workshops in
QI, I frequently start the class by asking everyone
to take a sticky note and complete the following
statement: “Quality is . . .” and let them fill in the
blank. I then take all the notes and place them
on a flipchart so that the participants can review
them during a break. BOX 1-1 provides a sum-
mary of some of the definitions I have received
over the years. As you review these comments
you will notice that people participating in an
introductory class on quality have keen insights
into the nature of quality.
Although numerous formal definitions of
quality have been offered (see Schultz, 1994),
I favor the simple yet straightforward perspective

BOX 1-1 Definitions of quality

Quality is. . .
■■ a combination of value and outcome in the eyes of the consumer
■■ a process with minimal opportunities for improvement
■■ a product or service delivered with 100% satisfaction the first time, every time
■■ a product or service that provides an expected value
■■ a product that lasts, for the best price
■■ a very good product—one you would want again
■■ above-standard results
■■ accountability
■■ an excellent product delivered by professional, friendly, knowledgeable people in a timely manner
at the appropriate time
■■ an unending struggle for excellence
■■ anticipation and fulfillment of needs
■■ attention to detail, timeliness, competence
■■ compassion
■■ completing a job in an accurate, efficient, and timely manner
■■ customer-focused service at a reasonable price
■■ data driven
■■ difficult to define
■■ doing the job right the first time
■■ going above and beyond what is expected
■■ listening/responding
■■ listening to feedback and, it is hoped, making changes to meet customer needs
■■ exceeding expectations
■■ making others feel important
■■ meeting and exceeding customer needs
■■ meeting our customer/patient needs in a cost-effective manner
■■ providing a product/service marketed to your customer above and beyond his or her expectations
■■ providing the best we can to our customer through kind and understanding dealings

(continues)
18 Chapter 1 Setting the Context for Measurement

BOX 1-1 Definitions of quality (continued)

■■ striving for excellence

■■ superior performance
■■ taking care of customers’ needs
■■ the best possible service; the most durable goods; the presence of a program for customer-focused
education and service recovery when expectations fall short
■■ the high human motivation to perform their job at their best
■■ the highest form of any service with satisfaction reflected in outcome that is measurable (I don’t
agree with this strike out) timely, optimal service that provides the best outcome and satisfaction
for the receiver of the service
Reprinted by permission of Robert Lloyd.

that Dr. Deming presented. He basically said
that he refused to define quality in a few words
or a sentence. What he did say was that “Quality
begins with intent, which is fixed by management”
(Deming, 1992, p. 5). Dr. Deming based his entire
approach to QI on the assumption that quality
has no meaning without listening to the VOC. He
stated that “Quality can be defined only in terms
of the agent” (i.e., the customer or end user of a
process). “Who is the judge of quality?” he would
often ask in his 4-day seminars. He clarified his
position with the following statement:
The difficulty in defining quality is to
translate future needs of the user into
measurable characteristics, so that a
product can be designed and turned
out to give satisfaction at a price that
the user will pay. This is not easy, and as
soon as one feels fairly successful in the
endeavour, he finds that the needs of the
consumer have changed, competitors
have moved in, there are new materials
to work with, some better than the old
ones, some worse; some cheaper than
the old ones, some dearer. The quality
of any product or service has many
scales. (Deming, 1992, p. 169)
When you step back from all the theoretical
and philosophical underpinnings offered by the
recognized experts in quality (e.g., in Schultz,
1994), they all discuss, in one way or another,
three fundamental activities that form the
foundation of QI:
■■ Listening to the VOC
■■ Listening to the voice of the process (VOP)
■■ Using statistical process control (SPC)
methods (i.e., using data to make decisions)
Organizations that clearly demonstrate
quality and excellence (e.g., the Baldrige winners
or organizations that have won state or inter-
national quality awards) are able to skillfully
blend all three activities together. A singular
focus on one or even two of these activities
is not going to achieve QI. It is when all three
activities are combined simultaneously and
on a daily basis that quality, as envisioned by
Deming and his contemporaries, will be real-
ized. FIGURE 1-2 depicts the interconnectivity
of these three activities.
When considering the implementation of QI,
it must be remembered that it is a never-ending
process of continuous improvement. Deming
Using SPC methods
CQI
Listening to Listening to
the VOP the VOC
FIGURE 1-2 Three key activities of quality
improvement

made this point abundantly clear in his writings
and seminars. “If I have to define it,” Deming
would say in his 4-day seminars, “it would be
meeting and exceeding the customer’s needs and
expectations, and then continuing to improve”
(Shultz, 1994, p. 47). It is acknowledging that
past success is no guarantee of future success.
Quality does not happen by accident or because
you want it to, wish it to, or hope that things will
get better. Remember, hope is not a plan! Qual-
ity results from the deliberate and intentional
actions of individuals within an organization.
Quality is not a program or a single project,
nor the responsibility of one individual (e.g.,
the ­director of quality) or those assigned to the
quality department (Lloyd, 2016). In short, quality
is a way of thinking about work, approaching
its improvement, and getting everyone involved.
Quality is about achieving and sustaining
excellence—nothing less.
If quality is viewed as something that has
to be done, “in addition to everything else I
have to do,” then the organization will never
understand quality, be able to achieve it, or
demonstrate excellence over time.11 The essential
ingredients, therefore, that enable an organization
to achieve quality and excellence and sustain it
over time include:
■■ A commitment to quality starting with the
board and senior management
■■ A strategically defined role for quality
throughout the organization
■■ A model or framework upon which to build
QI strategies
■■ A plan for deploying quality thinking and
application throughout the entire organization
■■ QI education and professional development
at all levels of the organization including
the board, senior leaders, middle managers,
and frontline staff
■■ A measurement philosophy and the use of
SPC methods
■■ Strategies to determine when a process or
system needs to be improved (e.g., reduc-
ing variation) or redesigned because it is
fundamentally broken
The Quality Funnel 19
■■ Plans for testing, implementing, ­sustaining
and spreading improvement (Langley et al.,
2009)
These are critical aspects of an organiza-
tion’s quality journey. When they are in place
and sustained over time, QI will be part of
the very fabric of the organization. Anything
less will relegate quality to just a word in the
organization’s lexicon or a mere focal point of
a banner or poster.
▸▸ The Quality Funnel
One final point about the use of the word
“quality” especially as it relates to the delivery
of healthcare services. As I work with teams
and organizations to help them develop
capacity and capability for improvement I
usually ask, at an early stage of our work
together, “What aspect of quality do you
want to improve?” Quite often I get quizzical
looks and interesting responses. I point out
that quality is not a unidimensional concept
and that there are three basic modifiers that
historically have been associated with this
word: quality assurance (QA), quality control
(QC), and QI. After making this distinction,
most healthcare professionals acknowledge the
three approaches but they do not see how they
relate to each other. Some even point out that
they are separate and distinct and do not have
any connections. This is when I show them the
quality funnel (FIGURE 1-3). The characteristics
of the three approaches to defining quality are
summarized in this figure.
The three aspects are shown in a funnel
because they do have close connections. In
health care, there is often almost a singular
focus on QA. QA has also been a central ap-
proach to the delivery of healthcare services
for a longer period of time than either QC or
QI, which in my opinion has created many of
the challenges we face in our industry today.
The problem is that QA should be done only
when there is such a high level of QC that
20 Chapter 1 Setting the Context for Measurement
Quality Control is a process by
which procedures and methods are
established to review and
Control
standardize the reliability and
quality of all factors involved in the
production of products or services.
Improvement
Quality Improvement is the combined and unceasing
efforts of everyone (e.g., healthcare professionals,
patients and their families, researches, payers, planners,
and educators) to make the changes that will lead to
better patient outcomes (e.g., health), better system
performance (e.g., care), and better professional
development.
FIGURE 1-3 The quality funnel
you can afford to check or audit the process
or system performance periodically. What I
find is that QA, especially point prevalence
audits, are conducted on healthcare processes
and systems that have very low levels of QC
and, therefore, low levels of reliability. Then
when the results of the audit are not favorable
the organization’s management team becomes
very concerned and wants to know why there
is poor performance. Well, the answer is easy
and should be obvious, but it is frequently
not acknowledged or discussed. Namely, if
a process does not have high levels of QC,
defective products and services will naturally
be produced. The manufacturing industry
knows this principle very well. Manufactur-
ers of everything from cars to computers to
ballpoint pens or processed lunchmeat first
establish systems for QC before going ahead
with ongoing production of the item. When
they have a high degree of control of the pro-
duction of the product (e.g., they are confident
that 99% of the time the product is produced
exactly as it was designed or planned), then
they can periodically conduct audits on a
sample of the product to check for consistency
and reliability. QA is not the first thing they
do. This is done only after they have put in
place reliable QC standards and procedures. If
Assurance
Quality Assurance is any
systematic process of checking
or auditing periodically to see if
a product or service being
developed is meeting specified
requirements, targets, or goals.
Quality
they detect through a QA audit, however, that
the product is starting to drift away from the
production specifications or tolerances then
they initiate QI strategies and methods. This
approach was laid out very nicely by Joseph
Juran in his quality trilogy (Juran, 1992).
Juran’s quality trilogy has served as a standard
for decades and describes clearly the critical
roles of quality planning, QC, and QI.
In health care, we seem to have this se-
quence backwards. We engage heavily in QA
and apply it to processes or systems that have
very low levels of QC to begin with. Then we
become surprised when the QA audits reveal
that either internal or external expectations
or targets are not being met. The result is that
more QA is put into place when in fact what
is needed is better QC. Even after a QA audit
reveals poor performance, applying principles
and methods of QI to the process frequently
does not produce the desired results. This is
because the QI approach is being applied to
a process that is most likely not stable and
therefore not predictable. Trying to improve
a process that is not in a state of statistical
control will only make things worse. So, as
you begin your quality journey, make sure you
have agreement on what approach to quality
is guiding your journey.

Notes
1. Regardless of the industry in question, bad
outcomes, unethical behavior, or scandal
rapidly lead to increased scrutiny and the
demand for more data to discover “what
is going on.” A classic example in the
healthcare industry can be found in
the Mid Staffordshire NHS Foundation
Trust scandal (http://www.midstaffs
publicinquiry.com/home). From January
2005 through March 2009, hundreds
of patients were routinely neglected
and many have been classified as dying
needless while in the Trust’s care. One of
the conclusions in the Francis Report as
it is generally known is that the Trust’s
leaders were so preoccupied with cost
cutting, achieving targets, and qualifying
for incentives that they lost sight of Mid
Staffordshire’s fundamental responsibility
to provide safe care. One of the major
outcomes of the inquiry is the increased
demand for greater scrutiny of all the Trust
in England and the need for more data,
so that the scrutiny bodies can track and
assess performance of providers. Exam-
ples from other industries reveal similar
response patterns. For example, one only
has to look at the television evangelist
scandals of the 1970s, the damaging claims
(still disputed by the way) in 1978 that
the gas tanks of Ford Pintos were more
susceptible to fire and explosion than
the gas tanks of other cars, the Enron
financial scandal of 2001, and the ongoing
investigation of Catholic priests. All of
these examples have led to loss of trust
in the product, service, or institution
under the microscope, a demand for
greater transparency of what the group
does, and a push for “more data.”
2. It is important to note that this phenom-
enon is not limited to one country, state,
region, or province. The move away from
an absolute trust in the medical system
Notes 21
and its providers to a perspective that
demands data as proof of the quality,
safety, competency, and value represents
a worldwide concern. It does not matter
how healthcare services are structured,
organized, delivered, or financed (e.g., em-
ployer-based insurance plans as we have in
the United States or national health systems
as we find throughout a majority of the
rest of the world). All countries today are
concerned about the value being delivered
by its healthcare system and the money
being spent on that delivery of services in
light of the results being tracked. There is
no such thing as “free health care” despite
what some people think or say. Somebody
pays for healthcare services whether it is
through taxes to support a national health
system, companies paying for the healthcare
benefits of workers, or individuals paying
out of their own pockets for coverage.
Value, quality, safety, access, and service are
the driving factors challenging healthcare
providers today and they are concerns
throughout the world.
3. The AHA DataViewer is a classic example
of this type of administrative data: https://
www.ahadataviewer.com/. Other examples
include the annual reports produced by
state data commissions in the United
States (e.g., the PHC4, http://www.phc4
.org/) and the Quality and Efficiency in
Swedish Health Care reports published
annually (e.g., http://www.socialstyrelsen
.se/publikationer2011/2011-5-18).
4. In this text, I am using the term quality
improvement (QI) as the generic reference
to a broad set of terms that have been used
over the years, including total quality man-
agement (TQM) and continuous quality
improvement (CQI). Other terms that have
also been used to refer to improvement in
various forms are quality control (QC),
performance improvement, performance
management, quality assurance (QA),
quality management (QM), quality circles,
22 Chapter 1 Setting the Context for Measurement
quality of care, and clinical QI. If you study
the history of the quality movement, you
will find many terms used to describe and
define this notion. I would encourage the
reader to explore a bit of this history and
become knowledgeable about the different
paths that the discipline of quality has
taken. It is a fascinating journey that is
quite enlightening.
5. For more information on hospital charges
in the United States, visit the CMS and
Medicaid website: https://www.medicare
.gov/hospitalcompare/. Similar profiles
can be obtained for each U.S. state by
visiting the state’s healthcare report card
site. The actual titles of state organizations
responsible for tracking and reporting
healthcare services vary by state. A search
under the general title of “state healthcare
data commissions” or some variant of these
terms, however, will provide a good start-
ing point. In my home state of Illinois, for
example, the data may be found at http://
www.healthcarereportcard.illinois.gov/.
6. Since 2001 when Crossing the Quality
Chasm was released a myriad of initiatives
both public and private have advanced the
role of patients, families, and caregivers in
making healthcare decisions. Some groups
like AARP, National Quality Foundation,
the Agency for Healthcare Research and
Quality (AHRQ), and the CMS have
developed programs and initiatives
designed to increase the involvement of
patients and their caregivers in making
healthcare decisions. Individuals who have
experienced either personal harm or the
loss of someone close to them through
medical error have also been playing
major roles. The Josie King Foundation is
one of the early leaders in this area. Josie’s
Story, written by her mother Sorrel King
(2009), tells the painfully honest story
about how medical errors led to the pre-
mature death of a vibrant 18-month-old
little girl. It recaps the family’s struggles
to deal with their grief and how Sorrel
started a journey into a world she never
had thought about. Today, Sorrel and her
family are patient safety advocates working
tirelessly to improve quality and safety in
Josie’s memory. More about the Josie King
Foundation may be found at http://josiek-
ing.org/Home. At the IHI, we too have
been heavily engaged in patient-centered
work for over a decade. It is one of the
four IHI strategic aims: (1) Optimize
Healthcare Delivery Systems: Encourage,
empower, and enable healthcare delivery
systems to provide truly value-based
care that ensures the best healthcare
outcomes at the lowest costs; (2) Drive
the Triple Aim for Populations: Strive to
achieve the Triple Aim, simultaneously
improving the health of the population,
enhancing the experience and outcomes
of the patient, and reducing per capita
cost of care for the benefit of communi-
ties; (3) Build Improvement Capability:
Build improvement capability into every
organization, healthcare executive, and
professional, while driving innovation
to dramatically improve performance at
all levels of the healthcare system; and (4)
Realize Person—and Family—Centered
Care: Usher in a new era of partnerships
between clinicians and individuals where
the values, needs, and preferences of the
individual are honored; the best evi-
dence is applied; and the shared goal is
optimal functional health. Our focus on
person-centered care recognizes that one
of the reasons that the needs of individuals
are often overlooked when they enter the
healthcare system is because they are de-
fined as patients not as people. Webster’s
dictionary (1984, p. 862) defines a patient
as one under medical treatment or one
who suffers. It also offers the alternative
definition of patient as one who is capable
of bearing affliction calmly or one who
is capable of bearing delay. These too
seem like good definitions of what hap-
pens to individuals when they enter the

healthcare system. So IHI is striving to
get healthcare professionals to realize that
person-centered care should address all
the needs of the individual and do so in a
manner that respects their values, beliefs,
and culture. The individual seeking care
is the primary focus of this IHI aim but it
also is encompassing enough to include
the individual’s family or caregivers as
well as staff, who are often the individuals
“bearing affliction and delay” as well. There
is no doubt that progress has been made.
But we still a have long way to go before
healthcare systems and providers of care
emulate the words and actions of Francis
Peabody. A recent study (Lavizzo-Mourey,
2006) demonstrated that things have not
changed much since 1988. The study set
out to measure how well physicians are
practicing patient-centered care. Between
80% and 90% of physicians surveyed said
they favored patient-centered care, 83%
of the respondents said they supported
sharing medical records with patients,
and 87% responded that they support
team-based care. But upon observation, less
than a quarter of doctors actually practice
patient-centered care as defined by the
study’s authors. The authors concluded
that the physicians basically don’t “walk
the talk.” I am reminded of an episode
of the TV show House where the ever
so smart and ever so rude Dr. House is
telling a mother what he plans to do to
her sick daughter. The mother is politely
objecting and asking questions about
his plan of care. In his usual brusque
and arrogant manner, he cuts her off
and turns to leave. She tells him to stop
and delivers an absolutely marvelous
line. She says with a very stern face and
pointing a finger in his face, “Look, I
am the mother and you’re the doctor.
I outrank you. This is what I want for my
daughter.” QED!
7. For example, today a patient can have
laparoscopic gallbladder surgery at an
Notes 23
ambulatory surgery center and be back to
work in a day or two. In the 1970s, that same
patient would have stayed in the hospital for
at least a week and be off work for several
more. Cataract surgery is routinely done
on an outpatient basis. Many hospitals are
getting patients with total hip replacements
discharged in 2 days or 3 days max. I had
a total hip replacement several years ago
and was out of the hospital in 2 days after
surgery. In October 2016, my wife had a
total knee replacement procedure that
was done as outpatient surgery. At Rush
St. Luke’s Presbyterian Hospital in Chicago,
they are now performing same-day hip
replacement surgery. Another techno-
logical advancement has been in the area
of micro and robotic surgery. The use of
the da Vinci robot to repair mitral valves
in the heart, for example, is now common
practice at many hospitals. We have the
technology to look inside the patient
without penetrating the skin. In the past,
it was not uncommon to do “exploratory
surgery” to determine whether there was a
problem. Today, we rely on CT scans and
MRIs as primary noninvasive diagnostic
tools. The application of new technology
seems endless. It will continue to change
not only how we think about medicine
but also how it is taught and practiced.
8. More on the ASQ may be found on their
website: https://asq.org/about-asq/history.
9. A more detailed history of the IHI along
with its current programs, resources,
and initiatives may be found at http://
www.ihi.org/Pages/default.aspx.
10. More about the NQF may be found at
http://www.qualityforum.org/Home.aspx,
including a listing of the more than 700
measures they have been assembling on
healthcare topics, their programs, and
other resources.
11. I actually had a participant in one of my
workshops say this in class one day. I
was speaking about the issue of making
quality thinking and practice part of daily
24 Chapter 1 Setting the Context for Measurement
work. This individual was sending body
language messages that said “Boy this is
really a lame idea” or something similar.
So, I asked her if she would care to make
a comment on my points or if she had a
different perspective. She then came out
with this classic response, “In addition
to everything else I have to do I am now
expected to work on quality. Where do
they think I can find the time to do this?”
I will never forget it. But before I could
respond others in the class challenged her
response. The best comment was from a
quiet woman in the back of the class who
raised her hand and simply asked, “Well if
quality is not your job, why are you here?”
References
Barrett, S. “The Rise and Fall of the People’s Medical Society
and Charles Inlander.” June 17, 2011. Retrieved from
http://www.quackwatch.com/01QuackeryRelatedTopics
/pms.html.
Berwick, D. M. “E. A. Codman and the Rhetoric of Battle:
A Commentary.” Milbank Quarterly 67, no. 2 (1989):
262–267.
Codman, E. A. A Study in Hospital Efficiency (1917 original
private printing). Reprinted. Oak Brook Terrace, IL:
Joint Commission on Accreditation of Healthcare
Organizations, 1996.
Codman, E. A. “Committee for Standardization of Hospitals
[of the American College of Surgeons] Minimum Stan-
dards for Hospitals.” Bulletin of the American College of
Surgeons, 8 (1924): 4.
Deming, W. E. Out of the Crisis. Cambridge, MA: Massa-
chusetts Institute of Technology, Center for Advanced
Engineering Study, 1992.
Donabedian, A. “The End Results of Health Care: Ernest
Codman’s Contribution to Quality Assessment and
Beyond.” Milbank Quarterly 67, no. 2 (1989): 233–256.
Gamble, M. “10 Things the Most Progressive Hospitals
Do.” Becker’s Hospital Review July 8, 2013. Retrieved
from http://www.beckershospitalreview.com/hospital
-management-administration/10-things-the-most
-progressive-hospitals-do.html.
Horowitz, L. Taking Charge of Your Medical Fate. New York:
Random House, 1988.
Hurst, J. W. “Dr. Francis W. Peabody, We Need You.” Texas
Heart Institute Journal 38, no. 4 (2011): 327–329.
Inlander, C., L. Levin, and E. Weiner. Medicine on Trial.
New York: Prentice Hall, 1988.
Institute of Medicine. To Err Is Human: Building a Safer
Health System. Washington, DC: National Academy
Press, 1999.
Institute of Medicine. Crossing the Quality Chasm: A New
Health System for the 21st Century. Washington, DC:
National Academy Press, 2001.
Juran, J. Juran on Quality by Design. New York: Free Press, 1992.
King S. Josie’s Story. New York: Grove/Atlantic, 2009.
Langley, J., R. Moen, K. Nolan, T. Nolan, C. Norman, and
L. Provost. The Improvement Guide: A Practical Approach
to Enhancing Organizational Performance. San Francisco,
CA: Jossey-Bass, 2009.
Lavizzo-Mourey, R. Remaking American Medicine: Health
Care for the 21st Century. The Secret of Patient Care.
Adapted from The Malcolm Peterson Honor Lecture,
National Scientific Meeting, Society of General Internal
Medicine, Los Angeles, California, presented April
28, 2006.
Lloyd, R. “Improvement Tip: Quality Is Not a Department.”
Institute for Healthcare Improvement website, 2016.
http://www.ihi.org/resources/Pages/Improvement
Stories/ImprovementTipQualityIsNotaDepartment.aspx
Mallon, B. Ernest Amory Codman: The End Result of a Life
in Medicine. Philadelphia: WB Saunders, 2000.
Millenson, M. Demanding Medical Excellence: Doctors and
Accountability in the Information Age. Chicago: University
of Chicago Press, 1999.
Neuhauser, D. “Ernest Amory Codman, M.D., and End Results
of Medical Care.” International Journal of Technology
Assessment in Health Care 6 (1990): 307–325.
Nightingale, F. Notes on Matters Affecting Health, Efficiency
and Hospital Administration of the British Army. London:
Harrison and Sons, 1858.
Oglesby, P. The Caring Physician: The Life of Dr. Francis
W. Peabody. Cambridge, MA: Countway Library of
Medicine, distributed by Harvard University Press, 1991.
Peabody, F. “The Care of the Patient.” Journal of the American
Medical Association 88.8 (March 19, 1927): 877–882,
reprinted in JAMA 252, no. 6 (1984): 813–818.
Rabin, P. L. and D. Rabin. “The Care of the Patient: Francis
Peabody Revisited.” JAMA 252 (1984):819–820.
Reinertsen, J. “How to Go Naked.” Leadership (Healthcare
Financial Management Association) (Summer 2012): 42.
Rogers, E. M. Diffusion of Innovations. New York: Free
Press, 2003.
Schultz, L. Profiles in Quality. New York: Quality Resources,
1994.
Swensen, S. J., and D. A. Cortese. “Transparency and the
End Result Idea.” Chest 133, no. 1 (2008): 233–235. doi:
10.1378/chest.07-2101
Tapscott, D. and D. Ticoll. The Naked Corporation: How the
Age of Transparency Will Revolutionize Business. New
York: Free Press, 2003.
Webster’s II New Riverside University Dictionary. Boston:
Riverside Publishing Company, 1984.
CHAPTER 2
Why Are You Measuring?
does bear a resemblance to many meetings that
▸▸ Connecting the Dots! occur on a regular basis throughout healthcare
settings. A frequent expression that arises in such
We have all sat through meetings where a vari-
meetings is, “You know what we need to do here?
ety of tables, charts, and graphs were presented
We need to connect the dots!” This is a rather
for entirely more minutes than we thought we
popular expression that started me t­hinking.
could ever endure. The presenter drones on in
Why not give people a bunch of dots (data
a monotonic voice describing the results of this
points) and ask them to predict what the image
month compared to last month or this quarter
will be before we connect the dots? FIGURE 2-1A
compared to the same quarter a year ago. You
provides a series of dots. What do you predict
are bored but don’t have the nerve to get up and
will appear when the dots are connected? I have
walk out. So you endure. But you hold on to a
had many creative responses in class. The most
vague hope that some minor crisis will erupt
frequent response is that the image will reveal a
that will cause someone to call or send you a
person riding a motorcycle. Others have seen a
page with an urgent request for you to leave and
bear, a leaping animal, and even a rough image
attend to the pending crisis. Unfortunately, the
of their city, county, or country. Because no one
call never comes. So, you endure the meeting
has been able to accurately predict what the dots
till the end. As you are leaving the meeting
will produce when they are connected, I start
you turn to a colleague and ask, “Did you get
to give them hints by connecting a few dots at a
anything out of that meeting?” She looks at you
time. I show an image with dots 1–25 connected.
and mumbles something that you believe sounds
I ask, “Now what do you see?” After a few more
vaguely like “No, but that was certainly typical
interesting guesses someone usually shouts out
of how we attempt to connect the dots at our
“an arm.” Although this is correct, it does not give
monthly management meetings.” As you walk
much insight into predicting what the remaining
back to your office you realize that you and your
158 dots will produce when connected. By the
colleague have just survived another journey
time I reveal the connections for the next 25 dots
into the “Data Zone”!
people are realizing that the image they are trying
Okay, your Monday morning management
to predict is not a bear. At this point, however,
meeting might not be quite like this one but it

© Michal Steflovic/Shutterstock

25
26 Chapter 2 Why Are You Measuring?

not be correct. The central point is that the dots

36
37 35

46

53
47

52 50
48 33
34
have been connected for you by someone else
so that when you walk out of the meeting you
45
55 54 32
42
38 57 60 51 49
43 61 31
41 56 59
44 62 63
39 78 58 65 30

leave with an image and a conclusion. How

79 64
40 80 66 24 23
67
86 85 81
69 29
68
84 70 25 22
77 72 71 28
73

many times have you been in a meeting where

74 21
1 83 75
82 76 7
3 27
87 26
5 8

someone connected the data dots for you?

2
4
88 178 9 20
176 175
6 19
89 177 173 18
180 171
174 183 17

When this happens do you raise your hand

179 172 168
10 16
170 167
182 169
90
181 12 15 155
166 11

and ask whether there is another way to look

91 165 154
162 14
97 164
13 156
92 94 163
98 153

at the data? Do you ask whether the presenter

160
96 161
99 93 134
100
158
95 157 152

is merely describing what has happened in the

101 159
145
102 121 141
140

103 122 151

133 143

past as opposed to predicting what we might

135
105 146
139 142
107 120 144
106 150
119 123 136
127

expect the process to produce in the future? Do

124 132 137 138 147
108 149
118 130 148
131
125 126 128
109 129
104

110 111
117 you politely ask whether there are alternative
ways to analyze the data instead of showing
112

113

two bar graphs and calculating a percentage

116

114 115

FIGURE 2-1A Connecting the dots (What image do change from Time 1 compared to Time 2? In
you predict will emerge?) short, can you predict what the dots are trying
to tell you before they are connected? This is a
key question for anyone interested in quality
they have narrowed down the range of possible improvement (QI).
outcomes but they still cannot predict what the Data should be used to help you predict where
dots will yield when connected. As I continue to you will be going in the future. Customers are
connect more of the dots the class quickly comes not concerned about the average wait time last
to realize what the image is (see the End Notes of month or last quarter. They want to know why
this chapter for the answer if you have not figured they are not being served now. A mother waiting
it out by now).1 They have now connected the dots in the emergency area, for example, wants to know
and have produced an image of what the data when her febrile child will be seen by a doctor.
points were hiding. So, the next image I show She does not care one iota about the average
the class is Figure 2-1A again where we have all wait time in the emergency area last week, last
183 dots but no lines. Of course they all see the month, or last year. Similarly, a physician waiting
image produced by the dots even though none for a STAT (immediate) troponin result takes no
of the dots are connected by lines. I then point comfort in being told that the average turnaround
out that once the image has been implanted in time (TAT) last week was statistically significant
their brains it is hard not to see the final pattern from the average TAT last month and that the
created by the connected dots. standard deviation went from 10.3 minutes to
This simple example illustrates precisely 9.4 minutes. In this case, the physician would
what happens when people enter the “Data have every right to voice concern.
Zone.” Data are presented, and the individual The level of quality for a particular product
making the presentation explains what the or service is determined by understanding cur-
dots are showing. In doing so, he or she is rent performance and predicting the future not
connecting the dots for you and in this pro- by how the process or system performed in the
cess is imputing meaning to the dots that then past. If you need an everyday example of how
sticks in your brain. You walk away thinking, this works all you have to do is ask sports fans
“I guess the outpatient satisfaction scores are about the current performance of their favorite
getting better.” Or, “Wow I didn’t realize that team, especially if that team won a championship
there was an upward trend in the inpatient fall in the past. I live in Chicago. We have a variety
rate.” These impressions, however, may or may of professional sports teams including football,

basketball, baseball, ice hockey, and soccer. Several
of these teams have won a championship within
their respective sport. In 1985, the Chicago
Bears professional football team won their one
and only Super Bowl championship. Even today
fans remember and reflect on the Super Bowl
victory and talk with nostalgia about where they
were and with whom when the Bears won. But
that moment in 1985 does not help us predict
the performance of the Bears this season, which
was the worst in the history of the franchise. In
2016, a major sporting miracle occurred here in
Chicago. Yes, after 108 years of failed seasons the
Chicago Cubs won the World Series. The city
went wild with excitement. But now that the
celebration has ebbed some are already starting
to talk about a repeat of this year’s performance.
Although hope springs eternal, people who make
these types of statements have little understand-
ing of a basic principle of human performance:
past success is no guarantee of future success.
Just ask the Bears or the Bulls fans!
Frequently, statistical methods reinforce
this focus on past performance. The average (or
any other descriptive statistic for that matter)
for last month or last quarter provides an aggre-
gated summary statistic that merely describes
a single characteristic of past performance.2
Dr. Myron Tribus, retired director of the Center
for Advanced Engineering Study at Massachu-
setts Institute of Technology and a student of
Dr. W. Edwards Deming, had a classic line about
the problem with making comparisons based
on monthly numbers: “Managing a company
by means of the monthly report is like trying
to drive a car by watching the yellow line in
the rear-view window” (Wheeler, 1993, p. 4).
Historical data by month, quarter, or year can
be useful in helping you understand where
you have been historically but it provides no
basis for determining where you are right now
(i.e., your baseline) or, more important, how
you will perform in the future. Consider pilots
and how they analyze data. You are flying in a
Boeing 777 over the Atlantic from Chicago to
London. You are having dinner and watching a
movie on the tiny screen in the back of the seat
Connecting the Dots! 27
in front of you. You are completely unaware of
how much data the pilots are processing and
the hundreds of dials and gauges they need to
constantly monitor. They are processing data in
real time to make predictions about the future.
They are not sitting up front in the flight deck
having this conversation: “Gee, Bill, what was
our average fuel consumption per hour when we
flew this same route last month?” “I don’t know
about fuel consumption, Anne, but I do have the
average altitude and the standard deviation from
projected altitude by quarter for the last 2 years.”
Because the pilots are concerned about where
they are right now, the current flying conditions,
and their ability to predict how the rest of the
flight will go, the aggregated summary statistics
from the past have little or no value.3 From an
improvement perspective data are meant to
predict the future not describe past performance.
The future can be understood only by analyzing
repeated measures of the variables of interest
over time and under a wide range of conditions.
Data and subsequently the interpretation
of data require inquiry and dialogue. For ex-
ample, when you are sitting in a meeting that
feels like the one described at the beginning of
this chapter you should realize that even though
you are bored with the lengthy presentation of
figures and statistics, the presenter is actually
connecting the dots for you. When the presenter
says, “And you can see from this summary
­table of numbers that the average wait time for
registration, the average wait time to see the
doctor, and the standard deviations for each of
these measures have all dropped significantly
over the past two quarters,” your bored brain is
actually connecting the dots. You walk out of
the meeting with an image of what the presenter
chose to tell you about the dots. You have no
ability to predict what next quarter’s wait times
will be but you did make the connection that
all the measures “dropped significantly over
the past two quarters.” The presenter used data
that are aggregated and dated to plant a little
causal model in your head. The dots have been
connected. Lower wait times are better than
high wait times and this past quarter was lower
28 Chapter 2 Why Are You Measuring?
than the previous quarter, therefore, we have
improved. Right? The correct answer is that you
do not know whether things have improved or
not. All you have is two dots. If I would have
connected only the first two dots in Figure 2-1A
no one would ever have figured out the im-
age. What about the other 181? Deming had
a great line about two data points. In one of
his 4-day seminars that I had the privilege of
attending he said, “When you have two data
points it is very likely that one will be different
from the other.”
Being able to predict where a process or a
system will go in the future builds much more
knowledge than merely describing what has
happened in the past. In order to be successful at
connecting the dots and using the data to make
predictions about future performance, however,
two questions need to be addressed:
■■ What type of study have you setup and what
action(s) will result from the study?
■■ Why are you measuring performance?
These two questions are addressed in the
remainder of this chapter.
▸▸ Types of Studies
In 1938, Dr. Walter Shewhart delivered four
lectures to the Graduate School of the Depart-
ment of Agriculture in Washington, DC, on
statistical thinking and quality control (QC).
In 1939, he expanded on the content of these
lectures and wrote Statistical Method from the
Viewpoint of Quality Control. Shewhart outlined
three components of knowledge (Shewhart
1939, 85–86):
1. The data of experience in which
the process of knowing begins
2. The prediction in terms of data that
one would expect to get if he were
to perform certain experiments in
the future.
3. The degree of belief in the predic-
tion based on the original data or
some summary thereof as evidence.
For Shewhart prediction was the central concept
driving quality.
Dr. W. Edwards Deming (1942, 1950) built
on Shewhart’s initial thinking about prediction.
In the foreword to Quality Improvement Through
Planned Experimentation, Deming wrote this
about prediction:
Why does anyone make a comparison
of two methods, two treatments, two
processes, or two materials? Why does
anyone carry out a test or an experiment?
The answer is to predict—to predict
whether one of the methods or mate-
rials tested will in the future, under a
specified range of conditions, perform
better than the other one. The question
is, What do the data tells us? How do
they help us predict? (Moen, Nolan, &
Provost, 1991, p. xiii)
One’s ability to predict, however, depends
exclusively on the type of study designed and
the actions to be taken as a result of the study.
Deming provided a foundation for thinking about
the types of studies that could be designed. He
classified studies as being either enumerative or
analytic (Deming, 1975). According to Deming
(1975, p. 147), an enumerative study is one in
which “action will be taken on the material in
the frame being studied.” He defined the frame
as “an aggregate of identifiable tangible physical
units of some kind, any or all of which may be
selected and investigated” (p. 146). A classic
example of an enumerative study is a census
conducted on a country’s population. In this case,
the frame is the entire population of the country
and the objective is to find out how many people
live within the country’s geographic boundaries.
Following this definition, a frame in a healthcare
setting might be all inpatients, all hip and knee
replacement patients, a nursing unit, a long-term
care facility, or all attending physicians.
Three key points highlight enumerative
studies:
1. The aim of an enumerative study
is principally descriptive in nature

(Deming, 1975, p. 147; Wheeler, 1995,
p. 18). Enumerative studies basically
describe how many or how much and
are essentially based on historical data.
The focus of an enumerative study, for
example, is not on explaining why
there were more males than females
receiving a particular medication but
rather on merely quantifying how
many males and females received the
particular medication. Prediction of
the future is not possible with enu-
merative studies.
2. The actions to be taken or the decisions
to be made as a result of an enumerative
study will be (or should be) directed
only to the subjects in the frame.4
3. Random sampling methods based
on probability theory and related
techniques such as confidence inter-
vals and tests of significance usually
provide the statistical approach to
enumerative studies.
In contrast to the enumerative study is an
analytic study. Deming defined an analytic study as
one “in which action will be taken on the process
or causal system that produced the frame studied
with the aim being to improve practice in the
future” (1975, p. 147). A key distinction between
an enumerative and analytic study, therefore, is
that analytic studies focus on the future perfor-
mance of the system or frame being studied, in
other words prediction. A fundamental assump-
tion of analytic studies is that the conditions
that produce an observable outcome today will
be different tomorrow and the next day and the
next. Enumerative studies, on the other hand,
look at data that have occurred in a defined
period of time (e.g., last month, last quarter, or
even last year) that is fixed or static in nature.
Provost (2011) and Provost and Murray
(2011) provide a useful analogy for thinking about
this important distinction of the role of time in
study design. The analogy is to water quality in
a pond versus a moving stream or river. They
compare an enumerative study to a pond that
Types of Studies 29
is fixed and not moving. One or more random
samples drawn from the pond are sufficient to
determine the water quality of the entire pond.
They point out that traditional statistical meth-
ods such as hypothesis testing (i.e., rejecting or
not rejecting the null hypothesis), confidence
intervals, and tests of significance can be used to
make decisions about the quality of the water in
the pond based on random sampling techniques.
But note that the conclusions and subsequently
the decisions about the pond in question must be
restricted to only that pond. If there is a pond of
similar size 15 yards away from the tested pond,
no conclusions should be made about that pond
even though they are relatively close to each other.
The writers’ analogy for an analytic study is a
stream or river that is in constant motion. If you
are standing along a rapidly moving stream, for
example, the water quality and the properties in
the water right in front of you are gone in a second
and new water is now in front of you. Does the
water now in front of you have the same properties
and qualities as the water that just went past a few
seconds ago? Thus water quality determined by
a random sample of the water at a single point in
time will not be able to reveal anything about the
quality of the water that will pass in front of you
10 seconds later. The stream is in constant motion.
It is not fixed or static like the pond. The present
condition of the water in the stream will change
over time. For example, imagine that a chemical
plant discharges its waste into the stream at 2 a.m.
every other day. If you drew a single random
sample of water from the stream at 2 p.m. on the
day between when the chemical plant dumps its
waste and took this sample back to the laboratory
for testing you would most likely miss the chemical
waste that was dumped at an entirely different
point in time. As Wheeler (1995, p. 18) points out,
“there is no way to define a random sample of the
future.” Without repeated sampling over time it is
impossible to detect a change in the water quality.
So, whereas enumerative studies rely heavily on
random sampling methods, analytic studies use
primarily judgment sampling techniques, both
of which will be discussed in greater detail in
Chapter 4.
30 Chapter 2 Why Are You Measuring?
Similar analogies can be found throughout
the healthcare industry. Patients connected to
telemetry in the intensive care unit (ICU), for
example, are monitored in an analytic not an
enumerative fashion. We connect ICU patients
to a variety of electrical leads in the ICU that
allow us to track their heartbeats, respiration rate,
and blood pressure in real time and over time.
Why do we do this? Why don’t we just take one
random sample of their blood pressure at some
random point during their 11-day stay in the
ICU and use this one reading to make clinical
decisions about the patient’s progress? It would
save time for the nurses and most likely save the
ICU money. Or maybe we could get their average
of all blood pressure readings on the first day
of admission to the ICU then hold off till they
are discharged 10 days later and take the second
average of all blood pressure readings on this
last day. We could even go further and discover
whether the two average blood pressure readings
between admission and discharge are statistically
different by applying a test of significance to
determine whether the resulting difference be-
tween admission and discharge is significant at
the 0.05 or 0.01 level of significance. If we took
this enumerative approach to patient care we
would end up causing considerable harm to the
patients. Clinical decision making is essentially
grounded in analytic study designs. We track
patient conditions over time, especially when the
need to know necessitates moment-to-moment
monitoring as in the ICU. The medical model
is clearly more analytic in nature than enumer-
ative. Yet, the distinctions between these two
approaches are typically not typically taught in
medical, nursing, allied health professions, or
healthcare administration programs.
One final distinction between enumerative
and analytic studies needs to be reinforced
before moving on. Specifically, it is critical to
realize that each approach employs different
statistical methods to arrive at conclusions.
Enumerative studies rely on probability theory,
descriptive and comparative statistics, and tests
of significance. If you have ever had to take a
basic or even advanced course in statistics, these
are the methods and techniques you have been
exposed to. In some circles these area referred
to as “traditional statistics.” These include both
univariate (descriptive) as well as multivariate
statistical methods, which includes everything
from tabular analysis (i.e., crosstabs) to various
forms of regression analysis, factor analysis, clus-
ter analysis, discriminant analysis, and analysis
of variance. A major reference from the social
science field for these approaches can be found
in the works of Hubert Blalock (1979) and in
particular his classic textbook Social Statistics.
When conducting analytic studies graphical
methods of analysis are used most often. This
branch of applied statistics is generally referred
to as statistical process control (SPC) and was
developed by Dr. Walter Shewhart in the early
1920s (more about this in Chapter 9). SPC meth-
ods, most notably the run chart and the Shewhart
control chart, accommodate repeated sampling
over time that in turn allows the researcher to
understand the variation inherent in the process
and thus predict the future performance of the
process. More advanced statistical methods of
prediction are incorporated in analytic studies
by using multifactorial designs and planned ex-
perimentation (Moen Nolan, & Provost, 2012).
Because this book is concerned principally with
analytic studies these methods will be discussed
in detail in the remaining chapters.
▸▸ Research for Efficacy,
Efficiency, and
Effectiveness
Closely aligned with the type of study you
design is a very pragmatic question: Why are
you measuring? This seems like a simple and
straightforward question. Yet many healthcare
professionals do not think about why they are
actually measuring. You may hear managers
or frontline workers, for example, say, “Look,
we need to submit some data on our progress
related to how much time patients have to wait
 Research for Efficacy, Efficiency, and Effectiveness
before actually seeing the doctor in the family
practice clinic so find some recent wait time
numbers and send them in.” Frequently this
means the data submitted may not be the most
recent data, it may not be defined in the same
way it was defined when it was first submitted last
year, or it may not be presented in a format that
adequately answers the questions being posed.
Anyone engaged in performance measurement,
therefore, needs to be very clear about their
reasons for starting the quality measurement
journey (QMJ).
Brook, Kamberg, and McGlynn (1996)
provided guidance for helping healthcare
professionals think about why they are mea-
suring. They differentiated two basic research
paths that can help individuals determine the
purpose of their measurement efforts: research
for efficacy and research for efficiency and
effectiveness. The choice of which path is to be
followed should be based on the nature of the
questions the researcher is trying to answer.5
All scientific inquiry begins with questions
not with statistical tools.
The efficacy road takes the researcher down
the traditional enumerative path of scientific in-
quiry described previously. This path is grounded
in experimental and quasiexperimental designs
(Campbell and Stanley, 1966; Posavac and Carey,
1980; Weiss, 1972). In health care, efficacy studies
are frequently used to test the ability of a par-
ticular drug, treatment, procedure, or protocol
to improve medical conditions. For example, a
researcher might pose the following questions
about a new drug:
■■ Is this drug capable of producing the de-
sired effect?
■■ Will this drug act differently with different
types of patients (age, gender, race, etc.)?
■■ Does a 5-mg dose of the drug produce
different results than a 10-mg dose?
■■ What does the literature tell us about the
use of this drug?
■■ What does past research reveal about the
drug’s application in experimental or qua-
siexperimental trials?
31
Every time healthcare researchers conduct
a randomized clinical trial (RCT) to compare
the impact of a new drug or a protocol, they
are conducting research for efficacy purposes.
A typical study might involve testing the effect
of a new blood pressure drug on two groups
of patients. One group would receive the drug
and be labeled as the experimental group. The
other group would receive a placebo and would
be referred to as the control group. Baseline
blood pressure readings would be obtained on
each group, as well as demographic information
about their current physiological condition,
family history, and activities of daily living, etc.
After a period of time, the two groups would
have their blood pressure levels measured
again, and statistical tests would be performed
to see whether there is a “significant” difference
between the two groups. The demographic and
other patient characteristics would be used as
control variables (i.e., they are used in an effort
to “hold constant” confounding or exogenous
variables that might play a role in blood pressure
variation).
If we really wanted to increase the precision
of this type of study, we would establish matched
samples of patients (i.e., we would make every
possible effort to have the members of the
experimental and control groups be similar in
gender, age, race, socioeconomic status, and so
on). The participants would be randomly assigned
to the experimental and control groups, and
then steps would be taken to make the study a
double-blinded trial. The double-blinded com-
ponent means that neither the participants nor
the researchers know which group gets the drug
and which one gets the placebo. This is done in
order to minimize bias, increase the validity of
the results, and thereby enhance the researcher’s
ability to answer the efficacy question.
The study design I have just described is a
classic approach to conducting a research study.
In a design of this type, the goal is to test the null
hypothesis (Ho) that there is no difference between
the experimental and control groups with respect
to the blood pressure drug. Statistical analysis of
the before and after blood pressure readings for
32 Chapter 2 Why Are You Measuring?
the two groups allows the researchers to reject or
not reject the null hypothesis and test for statistical
significance (Babbie, 1979; Morrison & Henkel,
1970; Selltiz Jahoda, Deutsch, & Cook, 1959).
Although some traditional research studies
are based on time series analysis or interrupted
time-series analysis (McDowall McCleary,
Meidinger, & Hay, 1980; Ostrom, 1978), the
majority of the studies designed to test efficacy
are not designed to monitor moment-to-moment
fluctuations in an observed process or outcome.
Instead, they typically obtain rather large sample
sizes (e.g., 100 or more observations in two com-
parison groups), let weeks or months transpire
as the research trial is allowed to run its course,
and then see whether the two comparison groups
show a significant statistical difference. The
focus of most experimental and quasiexper-
imental designs, therefore, is on static com-
parisons (i.e., enumerative designs that make
comparison of datasets that are fixed at discrete
points in time).
In summary, the primary purpose of research
for efficacy purposes is to build knowledge by
testing theories against empirical evidence. The
research results may not have immediate or even
readily apparent practical outcomes, however.
For example, research done in the vacuum of
space by astronauts has produced amazingly
pure crystals, but scientists are not sure what
they can actually do with this new knowledge
on earth. This new knowledge will be added
to the rest of the body of knowledge about
crystalline formation, and someday someone
will figure out how to use this knowledge for
practical purposes. In the meantime, articles will
be written and new theories about the nature of
crystals will be explored.
When healthcare professionals receive training
in research designs and statistical methods, the
approach described here is usually the standard
frame of reference (i.e., research for efficacy).
In my mind, this is one of the primary reasons
why many healthcare professionals seem to
have a strong proclivity to focus on comparing
two numbers (e.g., this month compared to last
month or this quarter compared to the previous
quarter) and asking whether one time period is
statistically different from the other. Research
for efficacy purposes is designed to answer this
type of question.
Brook et al. (1996) refer to the second
research pathway as one that leads to insights
about efficiency and effectiveness. This is the road
that leads to QI research and is closely aligned
with analytic study designs described earlier
in this chapter. Research related to efficiency
and effectiveness is also consistent with what
Westfall, Mold, and Fagnan (2007) and Khoury
et al. (2007) call translational research.
Closely linked with the ideas of Shewhart
(1931), Deming (1992), and Juran (1988, 1992),
research for efficiency and effectiveness takes
a very practical approach that seeks to answer
questions about the variation in process or out-
come indicators over time instead of whether
one number is different from another (i.e.,
the efficacy question). The key characteristics
of research for efficiency and effectiveness
include:
■■ A focus on solving practical problems rather
than testing or building theoretical models.
■■ A dynamic perspective rather than a static or
aggregated perspective. This is the distinc-
tion discussed earlier about the difference
between enumerative and analytic studies.
■■ Use of small samples of data selected con-
tinuously over time. For example, sample
sizes could be as small as five patients a day
or 10 each week. The frequency of sampling
when conducting research for efficiency
and effectiveness depends on the unit of
time (i.e., the subgroup) to be used in the
study. More will be said about this point
in Chapter 4.
■■ More reliance on graphical displays of data
than on descriptive statistics and tests of
significance.
Since Brook et al. made the distinction
between research for efficacy and research for
efficiency and effectiveness in 1996, considerable
 The Three Faces of Performance Measurement
progress has been made to incorporate the latter
form of research into the healthcare field. Today,
what Brook called research for efficiency and
effectiveness is essentially known as the science
of improvement (SOI). The rich history of the
SOI has been summarized nicely by Moen and
Norman (2010). Perla, Provost, and Parry (2012)
provide additional depth in exploring what they
call the “seven propositions of the SOI.” Taylor
et al. (2013) provide a very nice review of how the
SOI has been applied to improvement efforts in
healthcare settings.6
The key messages Brook et al. (1996) helped
to reinforce were that all measurement is the same
and that the scientific method lies at the heart
of all good research, whether it is done on large
static comparison groups (i.e., efficacy research
or enumerative studies) or in real-time settings
(i.e., research for efficiency and effectiveness or
analytic studies). All research designs and methods
have utility. It is up to the individual researcher
to build a knowledge base that allows him or her
to know which design, method, and/or statistical
technique is most appropriate for the questions
that need to be answered. Consider an analogy
from the surgical field. The surgeon has many
tools and methods available for surgery. The chal-
lenge is not to use all of them just because they
are there or to state unequivocally that one tool
or method is superior to all the others. Instead,
the real challenge is to have knowledge of all
the tools and methods and know when to use the
right tool at the right time to solve the problem
at hand. It is the same way with research. The
question or problem being addressed should be
the primary driver for deciding which research
design or methods are appropriate for turning
data into information. Some research methods
and tools are best suited to answer questions
related to efficacy. Others are more appropriate
for questions related to effectiveness and efficiency.
The wise (and in my opinion humble) researcher
should know enough about each approach to
know when to use one approach and not the
other. The questions, not the statistical tools or
methods, should drive the research endeavor.
33
▸▸ The Three Faces
of Performance
Measurement
In 1997, Solberg, Mosser, and McDonald wrote
an excellent article that provided a very practical
compliment to the more conceptual framework
provided by Brook et al. (1996). Solberg et al.
identified what they called the three faces of
performance measurement: measurement for
improvement, measurement for accountability
(or what I refer to as judgment), and measure-
ment for research. They further argue that these
three approaches to measurement should not be
mixed: “We are increasingly realizing not only
how critical measurement is to the QI we seek
but also how counterproductive it can be to mix
measurement for accountability or research with
measurement for improvement” (p. 135).
The authors describe the characteristics of
each of the three faces and how each approach
to measurement is based on a different purpose
or aim and methods. TABLE 2-1 provides my
adaptation of the key points from this valuable
article. The three faces are listed as the column
headings and the rows identify the major
aspects that need to be addressed during the
measurement journey. The reader is encouraged
to spend a few minutes reviewing the details
in each cell of this. Even a quick perusal of
the table’s content will quickly reveal that the
three faces not only have very different aims
but also use different methods and strategies
to collect and analyze data. For each of the
three faces the following key characteristics
are briefly discussed:
■■ Measurement aim
■■ Testing methods and observability
■■ Data collection and sample size
■■ Determining whether the data demonstrate
an improvement or change (I don’t get why
these words were capped and the previous
lines were not? So I changed them to lower
case.)
34 Chapter 2 Why Are You Measuring?

TABLE 2-1 The three faces of performance measurement

Accountability
Aspect Improvement (Judgement) Research

Aim Improvement of Comparison, choice, New knowledge

care (efficiency and reassurance, motivation (efficacy)
effectiveness) for change

Methods: Test observable No test, evaluate current Test blinded or

■■ Test performance controlled
observability

■■ Bias Accept consistent Measure and adjust to Design to eliminate

bias reduce bias bias

■■ Data “Just enough” data, Obtain 100% of available “Just in case” data
small sequential relevant data
samples

■■ Flexibility of Flexible hypothesis, No hypothesis Fixed hypothesis (null

hypothesis changes as hypothesis)

■■ Testing strategy Sequential tests No tests One large test

■■ Determining Analytic statistics No change focus Enumerative statistics

whether (SPC); run & control (maybe computer a (t-test, F-test, chi
change is an charts percent age change or square, p-values)
improvement rank the order of results)

■■ Confidentiality Data used only by Data available for public Research subjects’
of the data those involved with consumption and identity protected
improvement review

Modified from Lief Solberg, Gordon Mosser and Sharon McDonald, Journal on Quality Improvement vol. 23, no. 3, (March 1997), 135-147.

As you review Table 2-1 notice that I have placed
dashed lines as the vertical dividers between the
three columns. This was done to help remind us
that the three approaches should not be separate
and isolated from each other. The learning from
each column should be permeable and influence
the other two columns. More will be said about
this notion later.
The improvement column is where many
healthcare professionals claim to be working.
Yet, I often find that although it is quite popular
to say you are committed to quality and safety
these days, many organizations do not follow
the measurement principles and practices that
Solberg et al. laid out a number of years ago.
How well do your QI measurement activities
align with the following characteristics?
■■ Measurement Aim. As mentioned previously,
the aim of measurement for improvement
 The Three Faces of Performance Measurement
is centered on enhancing the efficiency and
effectiveness of care processes and outcomes.
There is a direct connection between mea-
surement for improvement and measurement
for research. Improvement is basically the
extension of the research act. Traditional
research helps us determine “the what” (i.e.,
what might be a reasonable new idea to test?)
whereas improvement research allows us to
determine “the how.” How can we implement
an efficacious idea, technique, procedure, or
drug so that it performs 100% of the time
in an efficient and effective manner every
time it is administered or applied?
■■ Testing Methods and Observability. When a
team is working on testing a new idea that
they feel will improve a process or outcome
they are usually following what Campbell
and Stanley (1963, p. 37–43) would classify as
quasiexperimental time series design. This is
not as rigorous or complex a design as a true
experimental design but it is quite good for
addressing issues of internal validity, which
is what QI is all about (i.e., comparing your
current performance against a more desirable
level of performance over time). The other
characteristic of improvement measurement
related to methods is that the work of the
researchers (i.e., the team) is observable
not only to the team and management but
to anyone else who wants to see the team’s
work. It is not uncommon for example, to
observe team members posting the results
of their most recent tests in a location that
the patients or the public can review.7
■■ Data Collection and Sample Size. Data collection
for improvement is structured around small
samples collected sequentially within time
periods as close to the actual productions
of work as possible. Solberg et al. describe
this approach to data collection as “good
enough” data collection. They write, “Because
a high degree of precision is not necessary
for improvement purposes and because data
collection needs to be simple and repetitive,
small samples, for example, 10–20 cases per
sample are usually appropriate” (Solberg
35
et al., 1996, p. 142). This approach to data
collection is referred to as gathering “just
enough” data.
■■ Determining whether the data demonstrate
an improvement or change. (Again these
capitals) the primary branch of statistics used
to analyze improvement data is SPC. This
branch of analytical statistics is a combination
of graphical displays of data over time plus
statistical estimates of the variation within
the data display. This is achieved by analyzing
the data with run charts or Shewhart charts.
Note that statistical tests of significance (e.g.,
p-values) are not appropriate for improve-
ment research. On this point Deming (1992,
p. 312) wrote “Students are not warned
in class nor in the books that for analytic
purposes (such as to improve a process),
distributions and calculations of mean, mode,
standard deviation, chi-square, t-test, etc.
serve no useful purpose for improvement
of a process unless the data were produced
in a state of statistical control. The first step
in the examinations of data is accordingly to
question the state of statistical control that
produced the data.” Details on these statistical
methods are provided in Chapters 8 and 9.
The second column in Table 2-1 addresses
measurement for accountability or as I refer
to it, judgment. The role of measurement for
judgment within the healthcare industry has
increased dramatically over the past 10 years
throughout the world (see Chapter 1 for addi-
tional detail on this topic). Measurement for
accountability (judgment) should not always be
labeled as negative. We all need to be accountable
for our actions and the work we produce. It is
important for both internal as well as external
purposes. The problem from my perspective is
when individuals are judged, rated, or ranked
and such assessments are not linked to improve-
ment. The key characteristics of measurement
for accountability and judgment are:
■■ Measurement Aim. The primary purpose
of measurement for accountability or
judgment is to make comparisons, rate
36 Chapter 2 Why Are You Measuring?
and rank performance, pass judgment on
performance, help customers make choices
(as is done for example by Consumer Reports
or the Leapfrog Group in the United States
that rates and ranks healthcare providers),
decide on the distribution of bonuses and
incentives, and/or drive competition and
stimulate a desire for change. This form
of measurement is typically focused on
comparing groups or individuals and
asking a very simple question, “Is your
performance better now than it was the
last time we looked at you? Yes or no?” In
most instances, the answer to this question
is based on current performance compared
to past performance or performance against
targets or goals.
■■ Testing Methods and Observability. There is
no rigorous testing occurring within this
approach to performance measurement
let alone any application of experimental
or quasiexperimental designs. Because the
focus is on evaluating current performance
against past performance all that matters
is whether there is a difference between
time 1 and time 2. When the comparisons
are made it is not uncommon to use na-
tional, regional, state or provincial norms
as comparative reference markers (dare I
say benchmarks?) for external compar-
isons and targets and goals for internal
comparisons.
■■ Data Collection and Sample Size. The ob-
jective is to obtain 100% of available data
for the defined period of time, which is
usually by quarters or years and is usually
aggregated into summary statistics. The
major controversy that emerges from this
approach is that the data are lagged and
the aggregated results, therefore, may not
reflect current performance. For example,
hospital ratings and rankings at the state,
province, or national level may lag for as
much as a year or more. In these cases
when the results are released hospital
leaders are quick to point out that the
current conditions of operation and their
results are quite different than they were
a year ago when the comparative data
were collected. The final aspect of data
collection for judgment is that most of
the measures in this category are outcome
measures (e.g., overall hospital mortality,
infection rates, or patient responses to a
general survey question such as, “How
would you rate the overall quality of your
care?”). The problem created by a singular
focus on high-level outcome indicators is
that a focus on outcomes alone provides
little or no insight into the processes that
produce the outcomes or more important
what needs to be changed in order to move
the outcomes to a more preferred level
of performance. This occurs most often
when external agencies or organizations
pass judgment on healthcare providers.
When accountability or judgment is done
internally against target or goals, however,
there is usually more concern over hitting
internal targets or goals that can be aimed
at improving the processes that drive the
outcomes.
■■ Determining Whether the Data Demonstrate
an Improvement or Change. Statistical
analysis for accountability and judgment
is essentially a binomial question. Are you
better now than when we last looked at you?
Yes or no? Do you have fewer infections
now than when we posted your data a year
ago? Has your inpatient mortality dropped?
Yes or no? The answer to this question is
usually based on comparing raw numbers,
percentages, or rates at two points in time.
Some comparisons will be based on calcu-
lating a percentage change to see whether it
meets or exceeds a specified target or goal.
One of the more popular methods used to
determine whether a change has occurred
from a judgment perspective is to use what
is popularly known as the “traffic light”
scorecard. This approach establishes three
cut points against a target or goal and then
 The Three Faces of Performance Measurement
assigns red, yellow, or green colors to the
units of observation based on where each
unit falls against the targeted cut points. In
this case, green is usually at or above target,
yellow indicates that the unit of observation is
not getting better or worse, and red indicates
performance below target or goal. A variation
on this same methodology is to assign stars
rather than colors to indicate the level of
performance. The better the performance
the more stars you get. Such methods to
determine whether a change has occurred
are frequently tied with pay for performance
bonus and incentive programs. Although
the negative impacts of these approaches
have been well documented by Deming
(1992, 1994), Kohn (1986, 1993), Berwick
(1995), and Herzberg (2003), the traffic
light method of analyzing performance
is still widespread in healthcare settings.
Besides being antithetical to the basic tenets
of QI, the problem with the traffic light
approach is that it completely ignores the
underlying variation in the processes and
related outcomes being judged. Further
details on understanding variation will be
explored in Chapter 6.
The third face of performance measurement is
measurement for research purposes. This approach
serves a vital and extremely important function
within the healthcare industry. Interestingly
enough, however, although healthcare profes-
sionals frequently reference RCTs and research
findings there are actually very few healthcare
professionals who work full time in this area.8
■■ Measurement Aim. As has been stated the
primary aim of research (or efficacy to use
both Solberg’s and Brook’s term), is to de-
velop new theories, test existing theories,
and build knowledge.
■■ Testing Methods and Observability. We take
elaborate steps to help ensure that exogenous
variables that could confound or mess up
our study and thereby invalidate the results
are controlledor held constant. Experimental
37
and quasiexperimental designs (Campbell &
Stanley, 1963) are used to maximize the
validity and reliability of the results and
reduce threats to the study.9 Frequently,
blinded tests are used so that neither the
researchers nor the participants in the study
know which participants receive the actual
test intervention and which ones receive
the placebo.
■■ Data Collection and Sample Size. One of the
key points of difference between the three
faces of performance measurement relates to
data collection strategy. Solberg et al. point
out that when conducting research studies
we typically create large, complex datasets
that are based on data that have occurred in
the past (i.e., last quarter or more often last
year). The authors refer to this type of data
as “just in case” data. I believe this term is
used because when doing research we usually
collect more data than we may need “just
in case” reviewers of our research, journal
editors, or critics raise questions about our
results or methods. If this happens, we can
respond, “No problem. I collected additional
data ‘just in case’ you raised that question.”10
■■ Determining Whether the Data Demonstrate
an Improvement or Change. When we conduct
statistical analysis for research we typically
use methods that were mentioned previously
in the description of enumerative studies.
Usually these approaches compare groups
or identify which variables are believed to
have “significant” influence over the de-
pendent or outcome variable. In a research
context we usually employ statistical tests
of significance and most notably a p-value
to determine whether the null hypothesis
(fixed hypothesis) is rejected or not and
significant results are observed between the
experimental and control groups. Although
the statistical notion of “significance” has
played a major role in making clinical
decisions, there has been considerable
debate in the literature about the “signif-
icance test controversy” (Ioannidis, 2005;
38 Chapter 2 Why Are You Measuring?

As... As a...

h
earc
Res
or,
Improvement

Judgment

Research
Im
pr
ov t
em
gmen
en
t Jud

FIGURE 2-2 Viewing the three faces of performance measurement as silos and as a Rubik’s cube
© Georgi Nutsov/Shutterstock; © tinka’s/Shutterstock

Zwarenstein & Oxman, 2006; Morrison &

Henkel, 1970; Ziliak & McCloskey, 2011).
Readers interested in this topic should make
a special effort to read the works of Ziliak
and McCloskey. They provide a wonderful
historical review of the development of the
test of significance, especially the p-value, and
then provide a comprehensive review of the
debate surrounding the test of significance
controversy. In their overview of the three
faces of performance measurement Solberg
et al. conclude “how counterproductive it is
to mix measurement for accountability or
research with measurement for improve-
ment” (1997, p. 135). This may have been
true in 1997 but I do not think it is quite
true today or an appropriate approach for
current time and challenges. In my opinion,
healthcare organizations need leaders and
frontline individuals who are comfortable
and competent in being able to blend the
three faces of performance measurement.
Although I agree with the authors that it
can send confusing messages to say you are
doing research and then apply procedures and
methods that characterize measurement for
accountability (judgment) or improvement
or any other combination of the three faces
and the aspects of measurement, I do not
earc
h
Res
Im
pr
ov t
em
g men
en
t Jud
FIGURE 2-3 Viewing the three faces of performance
measurement as a Rubik’s cube
view the three approaches as separate and
distinct silos as shown in FIGURE 2-2. I also
do not believe that it is “counterproductive
to mix measurement for accountability or
research with measurement for improve-
ment.” Instead I believe that it is more
effective to view the three faces as a Rubik’s
cube. FIGURE 2-3 provides a graphic image
of this analogy.
Healthcare leaders will improve their
measurement capacity if they navigate the
edges and interfaces of the three faces of per-
formance measurement. Sometimes, a person
becomes strongly invested in only one type
of measurement. A person may talk about
 The Three Faces of Performance Measurement 39

measurement for research as the “only” valid
type of measurement for example. Similarly,
others may talk about aggregate and summary
data as being preferable because it allows them
to compare hospitals, clinics, and groups of
doctors, cities, or regions. In this case, they are
invested squarely in the center of the measure-
ment for judgment space of the cube. Others
may say that although the other approaches to
measurement must be addressed, the prefer-
able way to proceed with measurement is for
QI and all other approaches are of secondary
importance. Instead of creating more silos
within the healthcare industry we should be
investing time in creating what Konan (1981)
calls translation. According to Konan (p. 12),
“The term ‘translation’ is chosen because the
process is like translating prose from one lan-
guage to another. A good language translation is
not a literal rewriting of each sentence. Rather
it captures the essence of the original copy and
then uses the mentality of the second language
to present the material to a different audience.”
If we are to have translation between the three
faces of performance measurement, then we
need translators not silo builders. Konan picks
up on this notion by stating, “Being a good
foreign-language translator requires more than
the ability to read and write another language.
Similarity, the skills required for translating
research into everyday language are different
from those required for doing research. The
translator must be able to read and understand
scientific research and then synthesize and
interpret it in relation to policy perspectives”
(p. 13). Healthcare organizations need individ-
uals who can function as translators and be able
to talk to individuals who are leading each of
the three faces of performance improvement
and find the linkages between improvement,
judgment, and research. It is not that difficult but
it does require the development of specialized
skills as Konan points out.
For physician and nurse leaders especially,
the edges of the cube where the three faces are
connected are critical. They need to be able
to walk comfortably back and forth between
the three faces of the cube so as to understand
how valid and reliable measurement is struc-
tured and organized within each of the three
approaches. Translators are not stuck in any
one of the measurement silos but work to break
down barriers and then build on the strengths
of each approach. All three approaches must
be understood as a system. The problem is that
individuals identify with one of the approaches
and dismiss the value of the other two. This
leads to duplication of effort and redundancy
in data collection and places limitations on
learning. A short case study demonstrating
how a translator with the appropriate skills and
knowledge can flip the cube around to address
the aims of each of the three faces while using
the same data follows.

CASE STUDY #1: Being a Translator

Situation
The board of a 310-bed community hospital is concerned that the hospital did not fare too well in a
recent citywide report on inpatient infection rates. This report compared 16 hospitals and showed that
the hospital ranked 13th out of the 16 hospitals. The report was recently highlighted in a newspaper
story and the hospital was singled out as one of the worst performers. An improvement team was
established to improve these results.

(continues)
40 Chapter 2 Why Are You Measuring?

CASE STUDY #1: Being a Translator (continued)

The Data
The QI department worked with the 11 inpatient units and 2 critical care units to establish historical
baselines on the number of infections using run charts with the number of infections (e.g., central line
bloodstream infections, clostridium difficile, and urinary tract infections) plotted on the vertical axis and
week displayed as the unit of time on the horizontal axis. Each unit then initiated various improvement
strategies and continued to track the various infections on their run charts. Each time they tested a
new idea they annotated the run chart to identify when the test was conducted.

Discussion
As the teams tracked their improvement tests and plotted the weekly number of infections, they
noticed that a majority of the units demonstrated a gradual decline in the number of infections. The
three units that did not show declines in the number of infections visited the other units to determine
what they were doing that did produce better results. After several team meetings these two units
also started to notice declines in their numbers of infections. So now it was time to report back to
management and the board. The staff in the QI department, functioning as translators, knew that the
board had a very simple question for them: “Do we have fewer infections now than when we first saw
the ranking of the hospital in the public report?”
The QI staff had all the detailed improvement data for the 13 units plotted by week on the run
charts. They knew, however, that this was too much detail for the board. So, they flipped the Rubik’s
cube around to the accountability face and aggregated the detailed run charts into two numbers; the
median number of infections for the entire hospital in the baseline period and the median number of
infections over the last 12 weeks. The baseline median for the entire hospital was 29 infections whereas
the more recent median after changing the infection assessment protocol and management processes
was 17. Both the baseline and the current number of infections were below the number that had been
printed in the public report. This allowed the QI director to go to the board meeting and provide a
simple answer to the board’s question: “Yes, we are safer now than we were when the public report
was released and we have continued to improve since then.” The QI translators have been able to take
the detailed improvement data from the units, flip it around, aggregate it, and provide the board with
an answer to their accountability question.
The final step in translating these data would come about when the translators take the detailed
improvement data and the run charts from the units and meet with researchers to discuss and
possibly design a quasiexperimental research study to see whether a totally new intervention that
had not been tested previously could be conducted to determine whether the idea is efficacious.
In this way, the same dataset has been used to address questions relevant to all three faces of
performance measurement. The key, however, is having translators who are able to understand the
aims and methods of each of the three faces and then be able to develop linkages between the sides
of the Rubik’s cube.
We have covered a lot of ground in this chapter. We started by connecting the dots and thinking
about our ability to predict with data. This led us into a discussion on the types of studies with a focus on
the differences between enumerative and analytic studies. Next we thought about the roles of research
for efficacy as compared with research for efficiency and effectiveness. This led us finally to discussing the
three faces of performance measurement and the fundamental question “Why are you measuring?” With
these fundamental concepts in place we are now ready to start our own QMJ.
 Notes 41

Notes descriptive statistics consists of measures

that reveal the dispersion in a distribution
1. Figure 2-1B with all 183 dots connected. of data. The simplest of these measures
Did you predict that it would be a hurdler? is the maximum and minimum values
in a distribution. This is followed by the
36
37 35

52 50
48
46

53 33
47 34
range, which is the absolute difference
between minimum and maximum values.
45
54 55 32
42
38 60 51 49 57
43 61 31
41 56 59
44 62 63
39

Finally there are a number of descriptive

78 58 65 64 30
79 23
40 80 66 67 24
86 85 81
69 29
68
84 70 25 22
77 72 71 28
73

statistics that capture various aspects of

74 21
1 83 75
82 76 7
3 27
87 26
5 8

dispersion. These include the sum of the

2
4
88 178 9 20
176 175
6 19
89 177 173 18
180 171
174 183 17

deviations, the average deviation, the

179 172 168
10 16
170 167
182 169
90
181 12 15 155

sample or population variance, and the

166 11
91 165 154
162 14
97 164
13 156
92 94 163
98 153

standard deviation. Further details on

160
96 161
99 93 134
100
158
95 157 152

any of these descriptive statistics can be

101 159
145
102 121 141
140

103 122 151

found in any basic statistics text.

133 135 143
105 146
139 142
107 120 144
106 150
119 123 136

3. This is not to say that the historical data

127
124 132 137 138 147
108 149
118 130 148
131
125 126 128
109 129

have no value at all. They have little or no

104

110 111
117

value to the pilots making the flight you

112

113

114 115
116

are on right now. But the historical data are

FIGURE 2-1B All 183 dots connected. Did you of great value to the designers, engineers,
predict that it would be a hurdler? and technicians at Boeing who built and
maintain the 777s. The historical data on
2. You will remember from your basic past performance is also of value if there
statistics class (or “sadistics” as some is an accident and the Federal Aviation
might call them) that there are two basic Administration is trying to determine
groups of descriptive statistics. First, when a possible failure started to occur.
there are measures of central tendency The key point to be made in this example is
characterized by the mean, the median, that the question you are trying to answer
and the mode. The mean is simply the should guide your measurement journey.
arithmetic average of all the numbers in a The measures you choose to track, your
distribution. The median is the point data collection plan, statistical analysis
where half the data are above this point techniques, and the conclusions you can
and half are below it. Another name for make all start with the question(s) you
the median is the 50th percentile. The are trying to answer.
mode is the most frequently appearing 4. This is a sometimes a challenge for indi­
number in a distribution of numbers. viduals who conduct enumerative studies.
In the classic picture of the theoretical Specifically they do not confine themelves
normal curve these three numbers are to making inferences about the ­members
shown as the vertical line in the center or entities in the frame. As one of the
of the bell curve and depict the mean, professors in my doctoral program used to
median, and mode all at the same posi- say, “They frequently want to drive beyond
tion. Typically this happens only when a their headlights.” That is they define a frame
theoretical normal curve is being shown and then arrive at a conclusion about the
or when computer simulations are run. In members or entities in the frame but they
real life, however, it is rare for the mean, don’t stop there as they should. If, for ex-
median, and mode to all be represented ample, an enumerative study is conducted
by the same number. The second group of on all the hip replacement surgeries done
42 Chapter 2 Why Are You Measuring?
last year at Hospital A and the research-
ers report that 62% of the patients who
participated in pre-hab (i.e., engaging in
rehabilitation exercises prior to surgery
to strengthen their muscles) were able to
walk with the assistance of only a cane
on the second day after surgery then all
they can rightfully conclude is that this
is the outcome for Hospital A. If there
are two other hospitals in the system,
however, that also do hip replacement
surgery but were not part of the original
frame would the researchers be justified
in telling the other hospital orthopedic
directors that they should have all their
hip replacement patients engage in pre-
hab because Hospital A had 62% of its
patients walking with a cane on the second
day after surgery? If they did arrive at this
conclusion they would be “driving beyond
their headlights.” Their study merely
described a characteristic of a subset of
patients in the frame. This conclusion
provides no insights as to why 62% of the
patients were walking with assistance of a
cane on the second day. Who were these
patients? All male? A mix of males and
females? What ages were these patients?
Did they have any comorbidities? When
questions like these are being asked, the
study needs to move from being an enu-
merative to an analytic design.
5. Note that I am using the term “researcher”
in a fairly broad context in this text. For
me, you do not need an academic degree
or work for an organization that has the
word research in its name to be called a
researcher. I maintain an applied perspective
when it comes to defining who might be
a researcher. Anyone who is interested in
answering a question, finding a better way
to perform a task or activity, or generating
new knowledge for themselves or others
is in my mind a researcher.
6. Despite the rich history of the SOI and its
very direct links to the scientific method,
it is interesting that the SOI is frequently
challenged as not being a “legitimate
science.” When I hear this type of statement
I always wonder if the person making
the comment would ask sociologists,
psychologists, economists, or chemists to
prove that their disciplines are scientific
in nature.
7. Observability or transparency of improve-
ment tests and the related results are not
always widely embraced, especially in a
country as litigious as the United States.
Some healthcare organizations in this
country, for example, have been counseled
by their legal staff to not reveal, report, or
publicly post QI results. The concern is that
if a patient experienced an adverse event
or harm and data are posted publicly then
the family and their lawyers might use the
data to support their claim that the provider
did not have quality processes in place at
the time of the incident. This position has
been shown to have little credibility, how-
ever, and the transparency of QI initiatives
is increasing with fewer legal challenges
each year (see, e.g., Baily et al., 2006).
8. I have arrived at this conclusion by asking
participants in my workshops how many
of them spend a majority of their time
working on setting up and running ex-
perimental or quasiexperimental medical
research projects. Even in audiences of
several hundred I usually have either only
a few or no hands go up. Most healthcare
professionals do not spend a majority
of their time in conducting academic
research studies designed for efficacy
purposes. On the other hand, when I
ask audiences how many of them spend
most of their time being accountable for
results or trying to improve results nearly
all the participants have their hands up
in the air. The point is that although we
rely on research to move the healthcare
field forward, a majority of healthcare
professionals, those delivering care on
a day-to-day basis, are not engaged in
conducting research. These people are

being charged with being accountable
for and/or improving the efficiency and
effectiveness of care.
9. One of the best resources on this point
is Campbell and Stanley (1963). This is
a classic reference relevant to any field
of research and something that in my
opinion should be required reading for
any healthcare professional. The authors
discuss three preexperimental designs,
three true experimental designs, three
quasiexperimental designs, counterbal-
anced designs, and four separate-sample
pretest–posttest designs. They discuss each
of these designs in terms of 12 threats to
internal and external validity, which are
a challenge to be addressed in any study
design. Some of the designs handle these
threats better than others. For example
the weakest of all designs is the one-shot
case study. Campbell and Stanley point
out that this design is used frequently
in educational studies. I would add to
this that it is also used unfortunately all
too often in healthcare settings. You will
probably recognize this study design. It
carried out with a single study group over
two time periods. Time period 1 is the
pretest marker, sometimes referred to as
a baseline measure. The intervention is
administered to the study group and then a
posttest measurement of the variable of
interest is taken. For example, imagine
that your long-term care facility has been
ranked in the bottom decile of a statewide
study on falls in nursing homes. The board
of your facility has made it clear to the
management team that this outcome is
unacceptable and must be improved. You
have the pretest result showing that your
facility is in the bottom decile. Someone
comes up with the idea of placing posters
up throughout the facility admonishing
everyone to “Pay Attention! Only YOU
can prevent patient falls!” The posters
are in place for 1 week. Then the number
of falls is recorded each day for the next
Notes 43
week and the average is calculated. This
serves as the posttest value. Although your
pretest number of falls over the past year
was on the average 23 per month, there is
great excitement when the posttest average
(which remember is only for 1 week) re-
veals only 16 falls. What do you conclude
from these two numbers? The wrong
conclusion is that the posters “caused”
fewer falls. The correct conclusion is that
it reveals absolutely nothing. Campbell
and Stanley write that “such studies (i.e.,
one-shot case studies) have such a total
absence of control as to be of almost no
scientific value” (p. 6). Yet we see many
pretest–intervention–posttest designs in
health care in which completely incorrect
conclusions are offered. Most healthcare
study designs fall into the category of
preexperimental designs. This is why I
believe that all healthcare professionals,
especially managers and leaders, need
to spend a little time learning from Drs.
Campbell and Stanley.
10. An example of this “just in case” approach
to data collection comes from my own ex-
perience. When I was building the dataset
for my doctoral dissertation I spent the
better part of a year designing, organizing,
and executing the data collection plan. My
dissertation research was designed to iden-
tify factors that led manufacturing firms to
locate in rural and small communities in
Pennsylvania (Lloyd, 1983). My data plan
involved collecting data from Pennsylvania
state archives, the U.S. Census, Dun and
­Bradstreet’s Market Identifiers File, County
Business Patterns, the Pennsylvania Bureau
of Economic Analysis, local government
documents, and phone call flow data from
AT&T. This was clearly a “just in case” data-
set that enabled me to respond not only to
additional analytic questions my committee
posed to me but also to address editorial
questions when it came time to publish
the results of my research in a professional
journal (Lloyd & Wilkinson, 1985).
44 Chapter 2 Why Are You Measuring?
References
Babbie, E. R. The Practice of Social Research. Belmont, CA:
Wadsworth Publishing Company, 1979.
Baily, M., M. Bottrell, J. Lynn, and B. Jennings. “The Ethics
of Using QI Methods to Improve Quality and Safety in
Health Care.” Hastings Center Report 36, no. 4 (2006):
S1–S40.
Berwick, D. “The Toxicity of Pay for Performance.” Quality
Management in Health Care 4, no. 1 (1995): 27–33.
Blalock, H. Social Statistics, rev. 2nd ed. New York:
McGraw-Hill, 1979.
Brook, R., C. Kamberg, and E. McGlynn. “Health System
Reform and Quality.” JAMA 276, no. 6 (1996): 476–480.
Campbell, D. and J. Stanley. Experimental and Quasi-Experimental
Designs for Research. Boston, MA: Houghton Mifflin, 1966.
Deming, W. E. “On Classification of the Problems of Sta-
tistical Inference.” Journal of the American Statistical
Association 37 (1942): 173–185.
Deming, W. E. Some Theory of Sampling. New York: Wiley &
Sons, 1950, reprinted in 1960.
Deming, W. E. “On Probability as a Basis for Action.” ­American
Statistician 29, no. 4 (1975): 146–152.
Deming, W. E. Out of the Crisis. Cambridge, MA: M ­ assachusetts
Institute of Technology, 1992.
Deming, W. E. The New Economics. Cambridge, MA:
­Massachusetts Institute of Technology, 1994.
Herzberg, F. “One More Time: How Do You Motivate Employ-
ees?” Harvard Business Review 81, no. 1 (2003): 86–96.
Ioannidis, J. “Why Most Published Research Findings Are
False.” PLoS Medicine 2, no. 8 (2005): 124. doi:10.1371
/journal.pmed.0020124
Juran, J. Juran on Planning for Quality. New York: Free
Press, 1988.
Juran, J. Juan on Quality by Design. New York: Free Press, 1992.
Khoury. M. J., M. Gwinn, P. W. Yoon, N. Dowling, C. A.
Moore, and L. Bradley. “The Continuum of Translation
Research in Genomic Medicine: How Can We Accel-
erate the Appropriate Integration of Human Genome
Discoveries into Health Care and Disease Prevention?”
Genetics in Medicine 9, no. 10 (2007): 665–674.
Kohn, A. No Contest: The Case Against Competition. New
York: Houghton Mifflin Company, 1986.
Kohn, A. Punished by Rewards. New York: Houghton Mifflin
Company, 1993.
Konan, M. “Translation: A Neglected Stage on the Research
Process.” Rural Sociologist 1, no. 1 (1981): 11–18.
Lloyd, R. “Vertical and Horizontal Linkages on Manufacturing
Employment In Rural Communities of Pennsylvania.”PhD
diss., The Pennsylvania State University, 1983.
Lloyd, R., and K. Wilkinson. “Community Factors in Rural
Manufacturing Development.” Rural Sociology 50, no. 1
(1985): 27–37.
McDowall, D., R. McCleary, E. Meidinger, and R. Hay.
­Interrupted Time Series Analysis. Beverly Hills, CA:
Sage Publications, 1980.
Moen, R., T. Nolan, and L. Provost. Quality Improvement
Through Planned Experimentation, 3rd ed. New York,
McGraw Hill, 2012.
Moen, R., and C. Norman. “Circling Back: Clearing up
Myths about the Deming Cycle and Seeing How It Keeps
Evolving.” Quality Progress, November 2010, 22–28.
Morrison, D., and R. Henkel, eds. The Significance Test
Controversy: A Reader. Chicago: Aldine, 1970.
Ostrom, C. Time Series Analysis: Regression Techniques.
Beverly Hills, CA: Sage Publications, 1978.
Perla, R., L. Provost, and G. Parry. “Seven Propositions of
the Science of Improvement: Exploring Foundations.”
Quality Management in Health Care 22, no. 3 (2012):
170–186.
Posavac, E., and R. Carey. Program Evaluation: Methods and
Case Studies, 4th ed. Englewood Cliffs, NJ: Prentice
Hall, 1980.
Provost, L. “Analytic Studies: A Framework for Quality
Improvement Design and Analysis.” BMJ Quality &
Safety 20, supplement 1 (2011): i92–i96.
Provost, L., and S. Murray. The Health Care Data Guide.
San Francisco, CA: Jossey-Bass, 2011.
Selltiz, C., M. Jahoda, M. Deutsch, and S. Cook. Research
Methods in Social Relations. New York: Holt, Rinehart &
Winston, 1959.
Shewhart, W. Economic Control of Quality of Manufactured
Product. New York: D. Van Nostrand, 1931; reprinted
by the American Society for Quality, 1980.
Shewhart, W. Statistical Method from the Viewpoint of
Quality Control. Mineola, NY: Dover Publications, 1939.
Solberg, L., G. Mosser, and S. McDonald. “The Three Faces
of Performance Measurement: Improvement, Account-
ability and Research.” Journal on Quality Improvement
23, no. 3 (1997): 135–147.
Taylor, M., C. McNicholas, C. Nicolay, A. Darzi, D. Bell,
and J. E. Reed. “Systematic Review of the Application
of the Plan–Do–Study–Act Method to Improve Quality
in Healthcare,” BMJ Quality & Safety 23, no. 4 (2013):
1–9. doi:10.1136/bmjqs-2013-001862.
Weiss, C. Evaluation Research: Methods of Assessing
Program Effectiveness. Englewood Cliffs, NJ: Pren-
tice-Hall, 1972.
Westfall, J., J. Mold, and L. Fagnan. “Practice-Based
Research—Blue Highways on the NIH Roadmap.” JAMA
297, no. 4 (2007): 403–406.
Wheeler, D. Understanding Variation: The Key to Managing
Chaos. Knoxville, TN: SPC Press, 1993.
Wheeler, D. Advanced Topics in Statistical Process Control.
Knoxville, TN: SPC Press, 1995.
Ziliak, S., and D. McCloskey. The Cult of Statistical Signifi-
cance: How the Standard Error Costs Us Jobs, Justice, and
Lives. Ann Arbor, MI: University of Michigan Press, 2011.
Zwarenstein, M., and A. Oxman. “Why Are So Few
Randomized Trials Useful, and What Can We Do
About It?” Journal of Clinical Epidemiology 59 (2006):
1125–1126.
CHAPTER 3
Measuring the Voice
of the Customer
▸▸ It All Starts With
Listening
Good evening, my name is Pam and
I’m calling on behalf of Acme Storm
Window Company. Recently we installed
new windows in your home and I was
wondering if you would take a few
minutes to answer several questions
about our service? Yes, I realize that
it is dinnertime, but I have only eight
brief questions to ask you and it will
take less than 2 minutes. No, sir, I’m
not trying to sell you our service. As
I said, we installed new windows in
your home last week and I merely
want to find out if our staff met your
expectations. OK, thanks; I’ll be brief.
Here’s the first question.
Sound familiar? We receive one or more of
this type of customer feedback calls each day or
at least each week. In addition, you probably also
receive periodic surveys in the mail asking for
your opinions on everything from how you feel
© Michal Steflovic/Shutterstock
about your political representatives to whether
or not your community should raise taxes, so
more trees can be planted in the parkways. If you
are like most people, these interruptions to your
dinner hour, the added paper in your mailbox,
or unsolicited feedback requests in your email
are basically irritations. But if you look at these
“irritations” from the viewpoint of those making
the inquiry, they are usually sincere attempts to
listen to the voice of the customer (VOC).
Quality improvement (QI) begins with lis-
tening to the VOC (Deming, 1992; Hayes, 2008;
Langley, Nolan, Nolan, Norman, & Provost, 2009;
Scherkenbach, 1991). Customers define quality
and set the expectations for performance. So when
Pam from Acme Storm Window Company calls
asking for your opinions about the new windows
you recently had installed, she is doing it to see
whether you are delighted with their product and
service, moderately satisfied, or totally turned off.
Basically, she wants to listen to you, the VOC.
It is the first step on the road to improvement.
In health care, however, it is ironic that we
seem to be rather ambivalent about taking this
first step and really listening to those we serve.
Although we list the various things we offer to
45
46 Chapter 3 Measuring the Voice of the Customer
patients as “services” (e.g., outpatient services,
laboratory services, emergency services, labor
and delivery services), it is frequently brought
to our attention that our services often fall short
of even meeting expectations let alone exceed-
ing them. Webster’s II New Riverside University
Dictionary (1984, p. 1066) defines service as “an
act of assistance or benefit.” In order to provide
such benefit, however, it is essential that the
provider of a service understands the type of
assistance needed or the nature of the desired
benefit—all of which requires listening to those
who receive the output of our work.1
The very name of what we provide, healthcare
service, explicitly states that we intend to serve
those who come to us in need. This requires
listening, real listening to actually hear the
concerns and expectations of the patient and
their caregivers. Too often, however, we listen
to patients and families in a very passive and
disengaged manner rather than engaging in
reflective listening and dialogue (Bohm, 1990).
We hear what patients and caregivers say, but
we do not listen to what is behind the words.
So instead of asking “What is the matter with
you?” we should be asking, “What matters to
you?” (Barry and Edgman-Levitan, 2012). What
matters to you? Four simple little words that
totally change the tone of an exchange between
a provider of care and the recipient of that care.
If the conversation begins with the question
“What matters to you?” then rather than doing
things to or for the patient we would be more
inclined to do things with the patient.
I genuinely believe that the majority of
healthcare providers do care about patients and
their needs. When I ask the participants in my
workshops, for example, why they decided to
enter the healthcare profession there is usually
a resounding reply, “because I want to help
people.” I always find this fascinating. The intent
to listen and serve seems to be there, but we
frequently fall short in actually delivering care
that is characterized by listening. I continue by
asking workshop participants the follow-up
question, “Well, if you got into health care to
help people, do you think that we meet, exceed,
or fall short in meeting customer expectations?”
An equally quick and resounding response is
that at best we meet expectation, rarely exceed
expectation, and more often than not fall short
of customer expectation. When I then ask why
they responded this way, the responses do not
come as quickly. You can see the participants
thinking about their responses. Eventually, a few
brave souls offer comments. Some folks point
to “the complexity of the system,” “not enough
staff,” or “not having enough time to relate to
the patients.” Others say that they do not have
“accurate data on what customers want or need,”
and therefore, the patient satisfaction survey
results “do not reflect reality and are therefore
are not reliable.” You can see the growing conflict
in their eyes as the reality of not being able to
fulfill their desire to “help people” is frequently
compromised by the systems, processes, and
cultures in which they have to work.
▸▸ Creating a Service
Excellence Culture
The creation of a service excellence culture does
not happen by merely telling everyone to smile
and be nice to people. Health care is not like the
old television show Fantasy Island (1978–1984).
The host (Ricardo Montalban) would start each
episode by making sure the staff were all properly
aligned and polished. As the plane circled the
island and made its final turn for the landing
approach, Montalban’s character, Mr. Roarke,
would recite the opening line, “Smiles, everyone.”
A dazzling smile, however, will never replace
behaviors that demonstrate that the workers
really care about the guests on Fantasy Island or
the patients waiting in a clinic reception area.
What it takes is having structures and processes
in place that are reliable and user friendly and
exceed customer expectations.
FIGURE 3-1 depicts the organizational
com­ponents that drive the creation of values,
norms, and behaviors that in turn determine
the organization’s culture. Service excellence

• Recruitment
Human
• Training resource
• Development issues
• Attitude
Culture
(norms and behaviors)
• Performance
• Evaluations
• Rewards
Incentives
• Celebrations
• Compensation
FIGURE 3-1 Components that build an organization’s culture
does not happen by chance. It also does not
exist merely because your organization won a
quality award 2 years ago or has a great mission
statement. Service excellence results from a very
deliberate focus on the four basic components
shown in Figure 3-1. These factors will determine
the culture your internal and external customers
experience on a day-to-day basis.
Component #1: Human
Resource Issues
It all begins with the individuals we hire, how we
train them, and what we do to retain them for the
long run. If the employees are our most valuable
resource (which almost every organization says
at one point or another), then a considerable
amount of effort ought to be directed to this
component. Many healthcare organizations have
initiated programs to screen potential applicants
to see whether they are in alignment with the
organization’s mission, values, and philosophy.
The Disney organization and Southwest Air-
lines are two well-known companies that have
used initial screenings for years to determine
this degree of alignment. Southwest refers to
their process as targeted selection. Central
Creating a Service Excellence Culture 47
• Data for
Measurement assessment
and
information • Data for
action
• Common
tools and
methods
• Communication
• Education
• Information
Organizational • Support
design structures
• Leadership
to the Southwest selection process have been
the characteristics of humor and the extent to
which individuals seem capable of becoming
part of an extended family of people who work
hard and have fun at the same time (Freiberg
and Freiberg, 1996, p. 67). If applicants do not
meet certain criteria, they are told this up front
so that they do not waste their time or the com-
pany’s. Failure to achieve a particular score on
the profile exam is not an indication that the
applicant is a “bad” person. It merely reflects
the fact that the applicant’s perspectives and the
organization’s expectations are not aligned. It is
much better to find this out at the initial stages
of the recruitment process than 6 months or
a year after the person is hired. But this is not
usually done in a majority of healthcare settings.
As a result, we hear comments such as, “How
did she get her job?” “Who does he know?”, and
“He offends everyone in the department but no
one [in management] seems to care.”
Every organization has employees who are
notorious for being rude, domineering, and out of
sync with what the organization claims to value.
Yet they seem to remain, and sometimes they
even get promoted while “good” people leave.
How does this happen? I believe that several
key factors create human resource (people)
48 Chapter 3 Measuring the Voice of the Customer
challenges in healthcare organizations. First,
healthcare organizations often hire strictly for
technical skill and assume that the person’s people
skills will somehow develop automatically over
time. Herb Kelleher, the retired chief executive
officer (CEO) of Southwest Airlines, made it very
clear that he expected his People Department
(what most organizations refer to as the human
resources department) to “Hire for attitude,
train for skill.” Kelleher made his hiring policy
very clear: “We look for attitudes; people with a
sense of humor who don’t take themselves too
seriously. We’ll train you on whatever it is you
have to do, but the one thing Southwest cannot
change in people is inherent attitudes” (Freiberg
and Freiberg, 1996, p. 67). Although healthcare
professionals, especially those delivering direct
care to patients, must be able to demonstrate
technical knowledge, skill, and proficiency in their
area of specialization, healthcare organizations
frequently miss or ignore the “inherent attitudes”
component of the hiring process.
Second, individuals who have done well in
one role (e.g., as a bedside nurse, a family practice
physician, or a home healthcare worker) are often
assumed to be prime candidates for promotion
to managerial or administrative roles. Further-
more, if they have been with the organization
long enough, then it is a foregone conclusion that
they should be “moved up.” Faced with the offer
of more money, status, and responsibility most
individuals will say, “Thank you, I am ready for
a promotion!” Rarely will you hear, “No, thank
you, I am perfectly content to stay where I am.” So
they accept the new managerial or administrative
position. In many cases, however, this is a major
mistake for the individual and the organization.
The individual soon becomes the living embod-
iment of what Peter and Hull (1970) refer to as
The Peter Principle—that is, people will be raised
to their level of incompetence. Many extremely
competent physicians, for example, have been
placed in high-level administrative roles only
to discover that they are incompetent when it
comes to managing people, conflict, and dealing
with the administrative and political aspects
of the job. They might have been excellent as
practicing physicians, but they quickly become
another example of The Peter Principle in action.2
The third human resource consideration
relates to retention of good staff. As mentioned at
the beginning of this chapter, most organizations
have a statement that goes something like this:
“Our employees are our most valuable resource!”
Yet, when you look at what many organizations
actually do to retain, reward, and build a bond of
loyalty with the employees, you start to wonder
why anyone stays with the organization. Deming
put it this way: “The greatest waste in America
is failure to use the abilities of people” (1992,
p. 53). Why do some people love going to work
each day and others hate even to get out of bed?
Part of it relates to paying fair and competitive
wages. This is only part of the equation, how-
ever. The other part, which is a major factor in
the healthcare industry, is not related to money
and perquisites. It is related to how people are
treated, how they feel about their coworkers,
the respect they have for their managers, and
the opportunities they have for self-expression
and meaningful work. It relates to leadership
and how organizations create forces to motivate
their most “valuable resource.”
Component #2: Motivation
Any healthcare organization that does not give
serious consideration to the broad range of factors
that motivate their employees will never be able to
develop and sustain a service excellence culture.
Besides basic compensation and benefits, other
key factors that lead to employee motivation
include (1) the performance evaluation process;
(2) reward and recognition systems; (3) the ex-
tent to which micromanagement exists within
the organization; (4) professional freedom and
autonomy to make decisions; and (5) organi-
zational hypocrisy (i.e., the extent to which the
organization’s leaders behave in a manner that
is inconsistent with the organization’s mission,
values, and philosophy).
Deming was keenly interested in the topic
of motivation. In fact, one of his four areas of
profound knowledge (Deming, 1994; Langley
 Creating a Service Excellence Culture 49

et al., 2009; Scherkenbach, 1991; Schultz, 1994) by instituting extrinsic motivational factors. In
includes human behavior or psychology and The New Economics (1994, p. 121) he wrote: “We
the key roles they play in managing people. must throw overboard the idea that competition
Although Deming acknowledged readily that is a necessary way of life. In place of competition,
he was not a psychologist, he pointed out con- we need cooperation.” To demonstrate how
stantly that if leaders do not realize that people he believed that the present style of extrinsic
(their employees) have differing desires, hopes, rewards robs people of joy in work, he laid
aspirations, ambitions, expectations about the out what he called the forces of destruction.
role of work in their lives, and learning styles, An adaptation of Deming’s portrayal of these
the organization will never succeed. forces of destruction (1994, p. 122) is shown in
Central to his interest in this subject was FIGURE 3-2. In this figure, intrinsic motivation
the question of whether individuals are in- is shown to be high during the early years of
trinsically or extrinsically motivated. Deming life. Over time, however, Deming maintained
believed very strongly in the value of a person’s that competition driven by extrinsic motivation
intrinsic motivation. His writings and lectures gradually crushes an individual’s internal or
all reinforced the notion that people are born intrinsic motivational forces. Nelson (1994)
with a natural inclination to learn and be in- provides a creative discussion on ways to es-
novative. His concern was that organizations tablish reward systems that strengthen intrinsic
knock intrinsic motivation out of individuals motivation. I think Deming would have found
by making them compete with each other and great value in Nelson’s work.
si o .
vi iti le

ho ls

nt p,
ar s

ce .
an ay
st e

e
di et op
on n

et a

ce u
s.

er
d ad

nc
m l go
rm p

it ro
s
s, mp pe
ol r

ria
fo ive

of g
G fg

t a ca

pr ery
up o e

va
er nt
s. o

ro . C dg

ou eri
r p ce
ol ion

, a Ev
of
, g ts u

ith m
le slo . J

fo In

n
ho ut

on n.
w Nu

io
op o m
sc trib

si io
at
e

vi t
pe in st

di iza
an
In dis

n m sy

y
Pa

pl

y m
t
ed

ee he st

Ex

er pti
tw ut t Mo
rc

ev ob
Fo

Su
Be P

Life begins Life ends

Extrinsic motivation
• Gradually replaces intrinsic motivation
Level of Intrinsic Motivation

• Competition and recognition drives actions

• Focus on the Individual not the System
Intrinsic • Problems attributed to Individuals not System
motivation • Resignation to external pressure - demotivational
• Curiosity
• Cooperation
• Joy in learning Intrinsic motivational factors are high at the beginning
• Self-esteem, dignity of life but are gradually crushed by the forces of
destruction.

Time

FIGURE 3-2 Forces that destroy intrinsic motivation

Reprinted courtesy of The MIT Press from The New Economics for Industry, Government, Education by W. Edwards Deming.
50 Chapter 3 Measuring the Voice of the Customer
On this point Deming wrote (p. 121):
What they (forces of destruction) do
is to squeeze out from an individual,
over his lifetime, his innate intrinsic
motivation, self-esteem, dignity. They
build into him fear, self-defense,
extrinsic motivation. We have been
destroying our people, from toddlers
on through the university, and on the
job. We must preserve the power of
intrinsic motivation, dignity, cooper-
ation, curiosity, joy in learning that
people are born with.
The literature on this topic is legion. I will not
go into depth on these materials at this time,
but there are three important items that deserve
mentioning. Probably the best known historical
piece of research on motivation took place at
Western Electric’s Hawthorne Works in Cicero,
Illinois from 1924 to 1932. Most readers will be
familiar with the term the “Hawthorne effect”
(aka the observer effect), which emerged as a
result of these studies. It is interesting to note,
however, that the term (i.e., the Hawthorne effect)
actually did not emerge until 1950 when Henry
A. Landsberger (1958) began to reanalyze the
outcomes of the numerous studies conducted
at the Hawthorne Works. Essentially, the design
of these studies was based on the hypothesis
that if you made working conditions worse,
productivity would go down. Over the roughly
8 years that spanned the numerous investiga-
tions at the Hawthorne Works, however, the
researchers discovered that as they made both
the environmental and managerial conditions
worse productivity actually improved. Then
when the researchers stopped their observa-
tions and left the factory, productivity declined.
The researchers’ major conclusion was that
productivity increased, even under increasing
worse operating condition, because the workers
were being observed. But, not all the original
researchers or those who followed have all agreed
on the interpretation of the original results. Yet
the term “the Hawthorne effect” is widely used
today to reflect motivational aspects of human
performance.
A second critical piece of research on mo-
tivation was published in 1968 by the Harvard
Business Review. The article, “One More Time:
How Do You Motivate Employees?” by Freder-
ick Herzberg, provides an essential review of
social and psychological factors that motivate
and demotivate workers. Herzberg noted in
this article, “What has been unraveled [about
the psychology of motivation] with any degree
of assurance is small indeed” (1968, p. 86). He
was very clear in his writings, however, that
managers have only limited power to motivate
employees. He also maintained that extrinsic
motivators, like pay-for-performance (PFP) or
incentive programs, do not necessarily make
workers happy and productive. Like Deming,
Herzberg maintained that only interesting
and challenging work will result in motivating
the workers. Deming referred to this simple
principle as finding joy in work.
One of Herzberg’s key concepts is “moti-
vating with KITA (i.e., a kick in the a _ _!).” He
defines the components of both negative and
positive KITA. On the negative side of the ledger
he points out that historically (late 1800s and
early 1900s) many organizations felt that the
best way to motivate employees was through
“negative physical KITA,” which included
creating fear in the employees, intimidating
them, and in some instances causing actual
physical harm. He emphasizes that fortunately
in a more modern context, organizations have
moved away from negative physical KITA to
more subtle approaches characterized by what
he calls “negative psychological KITA.” KITA
according to Herzberg produces movement but
not motivation. His analogy is that if he kicks
his dog, the dog will move. If he wants the dog
to move again what does he do? Of course, he
kicks the dog. But is the dog really motivated?
Herzberg argues that by kicking his dog he
achieves movement but not motivation. “I can
charge a person’s battery and then recharge it,
and recharge it again. But it is only when one
has a generator of one’s own that we can talk

about motivation. One then needs no outside
stimulation. One wants to do it” (1968, p. 88).
Herzberg (1968) proceeds to review positive
KITA personnel practices that were developed
as attempts by organizations to “motivate”
employees. This is a very interesting list of
nine practices that range from reducing time at
work to increasing salaries and fringe benefits
to requiring human relations training and even
employee counseling. But in the long run, he
maintains that such efforts produce only short-
term movement not motivation. Instead of
employing any form of KITA, Herzberg argues
that it is essential to understand what he calls the
“motivation-hygiene theory of job attitudes.” This
theory hinges on a key principle that the opposite
of job dissatisfaction is not job satisfaction but
rather no job dissatisfaction. Elaborating on
this conclusion Herzberg writes: “The growth
or motivator factors that are intrinsic to the job
are: achievement, recognition for achievement,
the work itself, responsibility and growth or
advancement. The dissatisfaction-avoidance or
hygiene (KITA) factors that are extrinsic to the
job include company policy and administration,
supervision, interpersonal relationships, working
conditions, salary, status and security” (p. 91–92).
He cites 12 different studies that interviewed
a total of 1,685 employees from a variety of fields
as well as countries to document his conclusions.
In short, I think that anyone interested in this
topic of employee motivation or human moti-
vation in general needs to spend time reading
and digesting Herzberg’s classic work. For some
it may fly in the face of conventional thinking
and organizational strategies about motivation.
But if nothing else, Herzberg’s work provides a
wonderful provocation for setting up a dialogue
on a subject that many organizations avoid.
The final writer who deserves mention for
his work on motivation is Alfie Kohn. Kohn
has written two critically acclaimed books
on this subject. No Contest: The Case against
Competition (1986) provides a review of the
role that competition has played in human
development. Contrary to popular opinion,
Kohn demonstrates that our fascination with
Creating a Service Excellence Culture 51
being “number one” makes no one number one.
He reviews studies from around the world to
place the intrinsic–extrinsic debate into clear
(and frequently controversial) context. His
second book, Punished by Rewards: The Trouble
with Gold Stars, Incentive Plans, A’s, Praises and
Other Bribes (1993), continues his analysis of
intrinsic and extrinsic motivation but is less
academic and far more controversial context (as
you can probably surmise from the title). Kohn
pulls no punches and takes a strong stance by
claiming that “rewards simply do not work to
promote lasting behavior change or to enhance
performance.” He continues:
Gradually it began to dawn on me that
our society is caught in a whopping
paradox. We complain loudly about
such things as the sagging productivity
of workplaces, the crisis of our schools,
and the warped values of our children.
But the very strategy we use to solve
those problems—dangling rewards like
incentive plans and grades and candy
bars in front of people—is partly re-
sponsible for the fix we’re in (p. xii–xiii).
Despite all the work that has been done
indicating that extrinsic motivation does not
motivate employees, there is still an extremely
strong belief within many organizations and
governmental bodies that incentives, bonuses,
and rewards do in fact motivate employees. The
extrinsic motivation camp is firmly grounded in
the work of the behaviorists. Notable among this
group are Ivan Petrovich Pavlov (1849–1936),
Edward Lee Thorndike (1874–1949), John
Broadus Watson (1878–1958), and of course,
B. F. (Burrhus Frederic) Skinner (1904–1990).
Skinner’s influence in the field of psychology is
widely recognized. Yet, his theories about human
behavior and motivation have been quite con-
troversial. There is little middle ground when it
comes to discussing (debating?) Skinner’s (1974)
“radical behaviorism.” He was a firm believer
of the idea that human free will was actually
an illusion and any human action or reaction
was the result of the consequences of that same
52 Chapter 3 Measuring the Voice of the Customer
action. If the consequences were bad, there was
a high chance that the action would not be re-
peated. If the consequences were good, on the
other hand, he maintained that the actions that
led to the good outcome would be reinforced.
He called this the principle of reinforcement
(Schacter, 2011).
This fundamental belief of Skinner (that
free will was an illusion) supported the extrinsic
motivation theories that are still popular today.
The proponents of “behavioral engineering” as
Skinner called it maintains that people can and
should be controlled through the systematic allo-
cation of external rewards (Grenning, 1991). For
organizations today, this translates into offering
bonuses, incentives, and other inducements to
the workers in order to get them to perform in
ways that management wants them to. It assumes
that the workers will do anything for the reward
(just like Pavlov’s dog salivating even before the
treat was given).
I believe this debate over intrinsic and extrinsic
motivation is one of the more controversial and
critical dimensions of QI and one that is central
to an organization’s success. Yet most healthcare
leaders have never had a serious dialogue with
their management team about this topic. Too
often, healthcare leaders believe that extrinsic
motivators do in fact drive the workers. I have
seen, for example, many not-for-profit health-
care organizations create elaborate incentive
programs to dole out a few extra vacation days
or dollars each year to the workers. For many
employees (e.g., nurses) gaining an extra day
or two is not of great benefit, because they are
usually not able to use up the regular vacation
time they have accrued. A major challenge for
creating bonus/incentive programs in health
care is that the actual monetary payouts or
other rewards, such as extra vacation days, are
small and trivial relative to what they hear their
counterparts receive in the for-profit sectors. As
a result, many incentive programs in health care
do less to motivate and more to provide fodder
for employee humor and sarcasm.3
In contrast to extrinsic motivation methods,
some healthcare organizations have decided
to go the route of gain sharing (as opposed to
PFP) as a way to reward the entire organization.
Because operating margins are relatively small in
the healthcare industry, a gain sharing approach,
where everyone gets the same monetary share of
the overall gain or operating margin, can receive
favorable support. Gain sharing also has some
inherent appeal to employees in not-for-profit
industries, because true PFP systems, such as those
found in the sales industry, are more difficult to
tie directly to individual performance in healthcare
settings. In a service industry like health care, it is
often hard to decide which member of the team
has outperformed the others. For example, the
surgeon may have performed an excellent hip
replacement surgery with no complications.
Yet one of the floor nurses discovered that the
patient was having a negative reaction to a par-
ticular medication, which could have severely
compromised the patient’s recovery. Does the
nurse deserve a bonus for catching a potential
medication error? Or has the physical therapist,
who convinced the patient that it was actually
better to move the new hip than to stay in bed,
performed better than the others on the team?
In short, given the collaborative nature of the
healthcare profession, it is often very difficult to
determine which individuals have outperformed
the others. An organization’s culture begins with the
people it hires. The culture is cultivated, however,
by the way in which the organization designs its
motivational strategies (Berwick, 1995).
Component #3: Organizational
Design
“Every system is perfectly designed to get the
results it gets.” This phrase has been popular
at the Institute for Healthcare Improvement
(IHI) for many years. In fact, it is a quotation
we have on one of the walls in our Cambridge,
Massachusetts, office. Since it was first coined
by Tom Nolan, PhD, statistical consultant
to the IHI, this expression has been used by
thousands of quality professionals to reflect the
important contribution of organizational design.
An organization’s structure and design are the

result of a direct and purposive set of decisions
made by the leadership of the organization.
Central among these decisions are the ways in
which healthcare leaders design, integrate, adapt,
update, and deploy the following five functions:
1. Education of staff and management
2. Communication (including both
the actual flow of information and
the ways in which it is disseminated)
3. Information flow (including the
amount that is shared with the
employees as well as the types of
information shared)
4. Support structures for providing
patient care (e.g., the admitting and
scheduling processes and the physical
layout of the office space)
5. Leadership structures (including
the number of formal leaders, their
ranges of responsibility, and their
abilities to manage complex systems)
There is no doubt that health care is a
complex enterprise. Yet from an organizational
point of view there are several critical charac-
teristics of the healthcare industry that are not
as prevalent in other industries and that often
contribute to organizational stagnation when
it comes to being innovative and creative. First,
innovation and rapid change do not come easily
to healthcare organizations even when research
has demonstrated that a new method or technique
is efficacious. This is highlighted very clearly in
the classic article by Balas and Boren (2000).
They conclude that the healthcare industry has
been extremely slow on the transfer of research
findings to everyday practice. For example, they
refer to the work of Oliver Wendell Holmes who
in 1843 delivered one of his famous papers on
the “Contagiousness of Puerperal Fever” before
the Boston Society for Medical Improvement.
Holmes made a strong argument for doctors
to wash their hands prior to examining a
pregnant woman, which was quite novel at the
time. Balas and Boren point out, however, that
it took decades for Holmes’ recommendations
to become a universally accepted practice and
Creating a Service Excellence Culture 53
that even then change did not come without
considerable resistance (p. 65).4 The most fre-
quent and well-known quotation from the Balas
and Boren article is a single sentence: “Studies
suggest that it takes an average of 17 years for
research evidence to reach clinical practice”
(p. 66). It is not uncommon to be in a meeting
about adoption and diffusion of new ideas in
healthcare settings and someone will reference
this finding and the length of time it takes to get
new ideas accepted by healthcare professionals.5
The second characteristic that I believe
prevents healthcare organizations from being
nimble and quick to respond operationally or
managerially are the myriad of layers that we
have created in this industry. Hierarchies are
quite strong and play dominant roles in the
healthcare profession. There are hierarchies
within the ranks of physicians, nurses, and ad-
ministrators. In many cases, these hierarchical
layers are built on decades of tradition and reflect
fairly conservative tendencies to avoid change.
These hierarchies did not happen by accident
or pop up overnight. The leaders of the organi-
zation created them and they are perpetuated
because we allow them to continue. Yet nearly
all of the healthcare professionals I encounter
complain regularly about the lack of flexibility
and innovation in their organization’s design and
administrative structures. Test this when you are
next in a meeting to discuss new or innovative
ways to admit or discharge patients. Within
minutes of someone presenting the new idea
the “Yes Buts” arrive. These are the people who,
when presented with a new way of discharging
patients, respond, “Yes but, that’s not how we do
things around here!” The other variation on this
situation is when a recently hired person tenders
an idea that she had seen work at her previous
organization. After presenting the new idea she
will quickly hear the voice of Yes Buts loud and
clear: “Look, you’re new here. I’ve been here for
18 years and we tried that idea 5 years ago. It
didn’t work then and it won’t work now. So if you
plan to have a future at this organization don’t
go around offering ideas that we know won’t
work.” Although not explicitly stated, the goal
54 Chapter 3 Measuring the Voice of the Customer
of many healthcare organizations seems to be to
maintain the status quo rather than to challenge
existing ways of thinking and/or to redesign the
ways in which things are done.
Jack Welsh, the flamboyant and outspoken
former CEO of General Electric, provided a good
example of how the leader of an organization sets
the tone for organizational design and change. He
made it very clear that an employee who wanted
to be promoted within the GE ranks had better
take all their courses in quality and be able to
demonstrate the application of the concepts to
make things better. He personally taught many of
the classes on quality to the employees. Despite
what you may think of his management style
and methods, he was quite clear that the status
quo was not going to be the modus operandi at
GE. How many healthcare executives can even
say that they have attended quality classes, let
alone taught them? In an interview with the
Chicago Tribune (April 14, 2002), Welsh made the
following statement about the role of education
and learning within an organization:
The challenge in the organization is to
get the collective intellect raised every
single day. People say to me as I go
around the country, “Well, gee, this is
a terrible time to be spending on edu-
cation. We’ve got these problems.” My
God, this is the time to be spending.
You can’t stop educating employees.
Every system (organization) is perfectly designed
to get the results it gets. I think Jack Welsh would
agree with this statement. But as Deming (1992,
p. 53) pointed out, “Money and time spent for
training will be ineffective unless inhibitors to
good work are removed.”
It is very easy to overlook organizational
design. For some it seems too abstract, for others
too detailed and complicated. System thinking
lies at the very heart of QI and at the core of
organizational design. If the organization’s indi-
vidual parts and related processes are not aligned
toward a common aim, then the organization’s
long-term results will always be at odds with what
goes on within the organization on a day-to-day
basis and the VOC will surely not be heard. What
processes do you have in place for communicating
your organization’s aim, strategic objectives, and
performance? Can the employees articulate your
mission and strategic objectives? Do you place
your key indicators on an intranet or publish
them in employee newsletters so that everyone
can see how you are doing? I have frequently
seen managers tell the employees that they need
to improve the patient satisfaction scores, for
example, but they never share the detailed reports
with the staff. If an organization is to become
truly a “learning organization” (Garvin, 1993;
Senge, 1994; Watkins and Marsick, 1993; Wick
and Leaon, 1994), then it needs to have a plan
for continuously evaluating and adjusting the
five functions of organizational design.
Component #4: Measurement
and Information
The final component that builds an organization’s
culture (Figure 3-1) is its approach to measurement
and information. Organizational measurement
can be classified into two basic categories: (1) mea-
surement of the VOC and (2) measurement of
the voice of the process (VOP). Within each of
these major categories, management and staff
must decide how they will assess performance
and how they will use the data they collect to
make improvements. They must also make sure
that they are using the right tools and techniques
to really understand the variation that exists
in their processes. Good measurement begins
with having an overall philosophy and approach
toward measurement and monitoring. Doing
what has always been done and measuring what
has always been measured or what is convenient
to track will not contribute to the organization’s
long-term survival.
The role of measurement in health care will
only increase. Organizations that do not develop
strategies for continuously measuring the VOC
and the VOP will most likely find themselves on
the outside looking in. They will be very upset
when external organizations begin making judg-
ments about the care that they deliver. In the end,

they will not be sure of their way. In fact, they
will be like Alice in Wonderland when she asks
the Cheshire Cat how to get out of Wonderland.
The exchange goes as follows:
“Would you tell me, please, which way
I ought to go from here,” asked Alice?
“That depends a good deal on
where you want to get to,” said the Cat.
“I don’t much care where—” said
Alice.
“Then it doesn’t matter which way
you go,” said the Cat.6
Measurement without a roadmap and a context
for action is a fruitless journey. It creates activity
but not accomplishments. The remainder of
this book addresses the quality measurement
journey (QMJ), so I reserve further comments
on measurement for subsequent chapters.
These four components (human resource
issues, motivation, organizational design, and
measurement) form the key ingredients for
creating a service excellence culture. Although
people often think of organizations as physical
structures and tangible assets, the truth is that
people and culture they create form the very
essence of organizations. People form the basis
of all organizations. Individuals with hopes,
wishes, aspirations, and desires form the cul-
ture of an organization. How the leaders of an
organization choose to manage and build this
culture will determine the organization’s fate.
In the last 50 years, there have been wonder-
ful examples of how a variety of organizations
have built positive and healthy cultures aimed
at service excellence and examples of those that
seem to have forgotten some very simple and
basic principles. Service excellence does not
happen by chance.
▸▸ Who Are Your
Customers?
In the past, health care was fairly simple and
direct. There was a provider of care, typically a
Who Are Your Customers? 55
doctor or nurse, and the recipient of care (i.e.,
the patient). This created a fairly straightforward
supplier–customer relationship. Today, things are
much more complex. Providers of care have to
deal with many individuals and organizations
in order to practice their profession. Similarly,
administrators and managers must also deal
with a myriad of both internal and external
customers.7 For example, internal customers can
include the board, management, the providers
of care, and all those employees not involved
directly with patient care but essential to the
organization’s operation. External customers, on
the other hand, are probably the most rapidly
growing and demanding group of customers for
healthcare providers to accommodate.
This means that today everyone in the
healthcare profession must be deliberately aware
of the myriad of customers they serve if they
wish not only to survive but to flourish. We
depend on others to supply us with various types
of input. At this point, we are their customer.
After we perform the work assigned to us, we
pass it along to someone or some group that
is in turn our customer. A supplier–customer
flowchart, therefore, is a very useful tool to help
individuals and teams appreciate how they and
their customers form an interdependent chain
of events. (See my short whiteboard videos on
flowcharting methods for details on the supplier/
customer flowchart and many other flowchart-
ing methods at http://www.ihi.org/education
/ihiopenschool/resources/Pages/BobLloydWhite
board.aspx.) Two key components of everyone’s
job, therefore, are to (1) know who your internal
and external customers are and (2) take steps
to actively listen to what these customers want,
need, and expect (Lloyd, 2003).
The simple truth is that health care has
become customer driven whether the providers
of care like it or not. When I have discussed this
topic with healthcare providers, managers, and
support personnel, however, I have experienced
a variety of reactions. The traditionalists in the
group ask why all these “untrained patients and
laypeople” are trying to make decisions about a
specialized profession that they are not qualified
56 Chapter 3 Measuring the Voice of the Customer
to address. Those with a more forward-thinking
view of the world, on the other hand, are con-
stantly looking for opportunities to include their
customers in the analysis and delivery of care.8
In order to determine where you and your
organization stand with respect to the VOC,
consider your and your colleagues’ responses
to the following questions:
■■ Who are our internal and external customers?
■■ Which group of customers is our highest
priority?
■■ How deeply do we really listen to our
customers?
■■ How often do we seek their input?
■■ What do our customers want, need, and
expect from us?
■■ Do our customers really know what they
want, need, or expect?
■■ We serve patients—what’s all this stuff about
customers, anyway? Customers buy cars and
refrigerators, not health care.
■■ Is the customer always right? Some of these
people make unrealistic demands on us.
■■ The patients are not trained healthcare
professionals—how do they know what
they need?
I have had many interesting (and sometimes
quite enthusiastic) conversations with healthcare
professionals about these questions. There seem
to be three primary schools of thought about the
role of customers. The first group is the “You’ve
got to be kidding” school. Members of this group
basically cannot believe that they even have to
listen to what the consumers of health care want,
need, or expect. This group is primarily made up
of individuals who long for the “good old days”
when patients were seen but not heard. They
maintain beliefs that are essentially grounded
in the supposition that they are the center of
the universe and that all the other objects ought
to rotate around themselves. Those who come
to these individuals for care and advice are as-
sumed to have less medical knowledge and are,
therefore, not qualified to be equal participants
in the discussions about care. Fortunately, this
category is not as dominant as it once was.
The second group is the “This, too, shall
pass” camp. Here, we have a collection of folks
who have been around long enough to become
genuinely cynical about any new ways of think-
ing about customers. This group could also be
classified as the “flavor of the month club.” They
have seen management fads come and go and
believe that if you merely wait long enough, some
new jargon or acronym will replace last month’s
quick fix. From this group’s perspective, nothing
really changes except the new management fad
or slogan of the month. Although skeptical, this
group can become tremendous agents of change
if they are given evidence that the organization
has constancy of purpose (Scherkenbach, 1990,
p. 13) and is sincere about turning management
jargon into action. The third group is the “eternal
optimist group.” These people are true believers.
They will get behind new ideas and support them
because they fit with the organization’s mission
and values and their own personal belief struc-
tures. They want to help and be team players, and
they will go out of their way to be nice to their
coworkers and especially patients. Frequently,
this group irritates the first group because of
their egalitarian optimism and receives benign
indifference from the second group because
they are not cynics.
Patients basically believe that the caregivers
are competent and know the technical aspects of
their jobs. For example, most patients do not know
whether they should have stitches or staples in
their incisions. They do not know whether they
should have a pill for their pain, an IV drip, or an
injection. What they do know, however, is that
they want to be treated with dignity and respect.
They want their questions answered, and they
do not want to wait for hours on end before they
receive service or get answers to their questions.
In many ways, our patients and their families are
asking for basic human courtesy. This point has
been well documented over the years. In a study
of more than 6,000 parents of children cared for
in 38 hospitals across the country, Co, Ferris,
Marino, Homer, and Perrin (2003) discovered
that overall parental satisfaction ratings of the
care their children received was most closely

associated with (1) the communication the
parents received about their child’s condition
and (2) the involvement they had in making
decisions about their child’s care.
Another perspective on this issue comes
from Dr. John Stone, a cardiologist and poet from
Emery University School of Medicine. Dr. Stone’s
classic book, In the Country of Hearts (1990),
directly addresses the human side of the medical
profession. In the section titled “Listening to the
Patient,” he writes: “The inventor of the stetho-
scope, Laennec, knew that the medical history
given by a patient is just as important—sometimes
even more important—than the ‘lup-dup’ of the
patient’s heart.” Among his advice to physicians
is: “Listen! Listen to your patient! He is giving
you the diagnosis” (1990, p. 65). This is the same
advice given by Dr. Francis Peabody in 1927 in
the article “The Care of the Patient.”
Service excellence stems from a constant
and deliberate dedication to listening to the
VOC. But listening is not enough. Listening
provides the context for responding. If pro-
viders of healthcare services do not listen and
respond appropriately, their customers will find
alternatives. Establishing what Connellan and
Zemke call “knock your socks off service” (1993)
requires more than just listening. It requires
finding out what customers care most about and
then building the structures, process, and culture
that sustains service excellence as an ongoing
part of every employee’s mindset and behavior.
▸▸ Defining Key Quality
Characteristics
Having a customer service orientation or phi-
losophy is only the beginning. I have observed
numerous healthcare organizations that claim
to have a “fully integrated service excellence
program” but the program exists primarily on
paper, within the pages of their annual report
or in the methodology of the management
incentive bonus program. When you look past
the words and rhetoric, there is no substance in
Defining Key Quality Characteristics 57
these approaches to actually listen to the VOC.
A viable service excellence program, therefore,
requires the deliberate creation of structures
and process aimed at making the VOC part of
everyone’s job 24/7/365.
A critical step in this journey lies in turning
this verbal orientation to service excellence
into tactics and strategies for actions that guide
behaviors. It begins by identifying what matters
most to your customers. Classically, these have
been referred to as quality characteristics (QCs).
QCs represent a broad collection of things the
customers care most about. For example, my
wife has a floral design business. Her business
has been growing (blossoming, if you will), and
she decided it was time to buy a vehicle to carry
all the arrangements to her customers. So we set
out exploring our options. She quickly developed
a list of QCs. She knew right away that she did
not want a sport utility vehicle (too small and
boxy). She also eliminated a station wagon for
the same reasons. This quickly reduced the field
to a van. As we explored the various QCs of
vans, she soon got down to a list of key quality
characteristics (KQCs), the things that she cared
most about. Interestingly enough, she actually
got down to a single KQC—sliding doors on
both sides of the van. She really did not care
about too many other options. She knew that
she would need to be able to access the interior
of the van from both sides. Some vans had only
one door, which was unacceptable from her
point of view.
Another example of defining a KQC comes
from a friend who planned to buy a new car. Being
a car buff, I asked her what she was thinking of
buying, she said proudly, “Anything that is less
than 190 inches long.” I told her that I had never
heard of anyone buying a car strictly based on
length of the vehicle. She said that it was very
simple. Both she and her husband are great
Harley-Davidson fans. Between the two of them
they own three Harley-Davidson motorcycles,
which they keep in the front part of their garage.
This leaves roughly, yes, 190 inches remaining
in the garage for a car. KQCs are simple and
straightforward when you listen to the VOC.
58 Chapter 3 Measuring the Voice of the Customer
Defining KQCs enables you to start the
VOC process. KQCs are the things customers
care most about. They are the aspects of a process
that are foremost in the minds of the customers.
We all have KQCs irrespective of what the issue
or process is. If we are not asked, however, our
wants, needs, and expectations remain unknown.
KQCs have the following characteristics:
■■ They reflect quality as defined or judged
by the customer.
■■ They reflect aspects of the process that the
customer cares most about.
■■ They represent to the customer the key
measures of quality output.
■■ They will vary from customer to customer.
■■ They will create conflicts for you because
you cannot meet all of your customers’
needs simultaneously.
Whenever you start to listen to the VOC, you
should answer several basic questions:
■■ What is the process or system the customer
experiences?
■■ When does the process or system begin and
when does it end?
■■ Who is the primary customer? The second-
ary customer?
■■ Do we have consensus on the primary
customer and what matters most to them?
■■ Can we segment the customers into rea-
sonably homogeneous subgroups so that
we are clear on what the KQCs are for each
group?
■■ What aspects of the process (the KQCs) are
most important to the primary customer?
If you do not at least have a dialogue on
these questions, you will probably assume that
you know what the customer wants, needs, or
expects. In health care, we have a long history
of being quick to prescribe and slow to listen.
Historically, orders were given and the patients
were expected to follow them. Yet the instruc-
tions or orders, which seemed perfectly logical
to the physician, may not have been what the
patient wanted, needed, or desired. Because most
patients defer to the judgment of the physician,
however, orders will usually be followed even if
they are far from the patients’ KQCs.
We all have stories about our encounters
with the healthcare system. Here is one of mine.
In 2009, I had a total right hip replacement.
I interviewed five surgical groups in my area to
determine which one I wanted to surrender my
hip to. I used the World Health Organization
(WHO) surgical checklist as a starting point for
discussion with each surgeon and then asked
to see their data on operating time, revisions,
infection rates, type of procedure used, and
type of implant used. Yes, I know what you are
thinking; I was not the usual type of patient.
In fact, when I asked one of the surgeons all
these questions he stopped and asked me,
“Who are you and why are you asking me all
these question?” He did not get picked as my
surgeon. One of the other groups I eliminated
right away when the surgeon was “too busy” to
meet with me and sent a physician’s assistant
(PA) to interview me. When the surgeon did
not show up and the PA explained that he was
“tied up,” I politely thanked her for her time
and left. Now I was down to three groups. The
surgeon I finally selected (remember they had
to convince me that I should give them my hip)
started by asking me “What can I do to improve
the quality of your life?” This fellow had my
attention right away. He wanted to listen to the
KQCs that brought me to him. He focused on
me and instead of asking “what’s the matter”
took the VOC road and asked “what matters to
you?” The procedure and the outcomes exceeded
my expectations.
One of the most frequently overlooked aspects
of KQCs is that they will vary from customer
to customer. This requires that you once again
realize that you do not have just one customer
but many. Even within a group (e.g., hip or knee
replacement patients or breast cancer patients)
there will be variation. Treating hip replacement
patients as if they all have the same KQCs is
not only disrespectful to the patients but also a
good indicator that the organization does not
really listen to the VOC. The concept of mass
customization is critical to understanding the

KQCs within segments of those you serve. Mass
customization is defined by Tseng and Jiao (2001,
p. 685) as “producing goods and services to meet
individual customers’ needs with near mass pro-
duction efficiency.” Manufacturing companies
have used this notion of mass customization to
deliver exactly what the customer wants. I saw
this occur during a tour of the Harley-Davidson
motorcycle assembly plant in York, Pennsylvania.
As the bike frames come down the line, a laminated
barcode card is scanned and all the specific cus-
tomer KQCs are read and built into the assembly
process. This means that the fenders, type of seat,
size of engine, handle bars, and paint color are
customized on the spot. Even in the paint shop
each bike is painted the color preference of the
customer. This means that they do not paint all
black bikes on Monday and red bikes on Tuesday.
As the bar code is read, the robotic paint ma-
chines switch out the colors for each bike. I know
what you’re are thinking. “But we are health
care not Harley-Davidson!” True. Health care
is different, but we have many opportunities for
applying the principles of mass customization
to our patients.
I experienced mass customization when I
had my hip replacement surgery. The surgical
group I selected to do my procedure had a meeting
of all the hip and knee replacement patients and
their respective caregivers 2 weeks before the
procedure. In this “Joint Adventures” meeting
(seriously it was called Joint Adventures), they
explained how the procedure would be performed,
what would happen to the patient, the expectations
for both patient and caregiver involvement in
the recovery process, and rehabilitation plans.
They also had us fill out a brief survey asking
us about our exercise habits, expectations for
how this would improve the quality of our lives,
activities of daily living and current functional
status, and what we hoped to be able to do once
the procedure was over that we could not do
now. All of this information was then used by the
care management team and the nursing staff to
customize a recovery plan for each of us. At 63
I was the youngest patient in the group. Sitting
next to me was an 84-year-old woman who was
Defining Key Quality Characteristics 59
frail and had a very different set of KQCs than
those on my list. This was mass customization
applied to a healthcare procedure.
Ultimately, if you cannot agree on the primary
customer and their specific needs and expecta-
tions, then the KQC discussion will go round
and round. This can also occur if you decide to
change the primary customer midway through
an improvement initiative. If this happens, then
you should revisit the KQCs and see whether
they are still relevant to the new customer group.
It constantly amazes me how improvement
teams cannot reach consensus on the primary
customer. As an example, let me offer a story
from another service sector—education. I was
working with a local school district to assist them
in defining indicators. They were participating in
a national demonstration project that required
them to identify “quality indicators” that could
be tracked over time. They were ready to start
defining indicators and looking at the data.
I asked them to consider a simple question be-
fore we started to peer at numbers: “Who is the
primary customer of the educational system?”
The team was composed of administrative staff,
teachers, and community representatives. As
you can imagine, there were mixed responses to
my question. The majority of the responses had
something to do with the students and/or teachers
as the primary customers. I commented that if
the students are the primary customers and the
school really listened to their wants, needs, and
expectations, then school would probably begin
around noon, lunch would be the first period,
then free time would be scheduled for the next
2 hours before dismissal at 3 p.m. School sys-
tems are not designed to maximize the KQCs of
students. As we explored this question further,
some team members admitted honestly that the
teachers were the primary customers. Others
concluded that the community was the primary
customer. Still others thought the business sector
was the primary customer. Eventually, as the
discussion slowed the majority concluded that
society was the primary customer of the edu-
cational system. Personally, I am not sure they
ever really reached consensus on their primary
60 Chapter 3 Measuring the Voice of the Customer
customer of the educational system. Periodically
I run into the school superintendent and ask him
if they ever resolved this question. He usually
responds, “No, but we are still working on it.”
Now consider a healthcare example. Imagine
that you are responsible for the billing process.
If you define the patient as the customer, the
KQCs might be an accurate bill, a timely bill, or
a bill that is easily understood (i.e., no procedure
codes, cryptic Latin phrases, or other medical
jargon). If, on the other hand, the insurance
company is the primary customer, then detailed
procedure codes, insurance classification codes,
and medical terminology might be the desired
KQCs. My point is that if you change the primary
customer the KQCs will also change. But if you
cannot agree on who actually is the primary
customer, then the team’s discussion will have
a lot of rhetoric but little practical value.
▸▸ Listening Three Times
FIGURE 3-3 shows three key points at which
listening to customers should occur. Note that
these three listening points are not mutually
Preservice
Solicit
Point-of-service point-
of-service
feedback
Postservice Measure organizational performance
How often do you listen to the VOC?
FIGURE 3-3 Three listening points
exclusive. In fact, relying on only one or even
two of these approaches will place tremendous
limitations on understanding what customers
want, need, and expect. Establishing multiple
listening points should be the standard for anyone
who genuinely cares about customers. Multiple
listening points not only enhance the amount
of information received but also increase the
validity and reliability of the findings (Campbell
and Fiske, 1959). TABLE 3-1 provides a summary
of the objectives of the three listening points.
The first time to be listening to our cus-
tomers is before they experience our products
or services. This is what is typically known as
preservice assessment. For example, if a hospital is
thinking about creating a new service for women
and children, they should devise methods and
techniques to find out what potential patients,
families, and payers expect from this service before
it is designed and implemented. How often do
we decide to open a service or program before
we determine whether there is a market for the
service? Health care is not like the classic line
from the movie Field of Dreams—“If you build
it, they will come!” Asking people how they
would like something to work is the first step
Identify customer expectations
Design customer-friendly
systems
Manage
unsolicited
feedback
Identify opportunities for
further improvement
 Understanding VOC Tools 61

TABLE 3-1 Summary of the objectives of the three listening points

Preservice Point-of-Service Postservice

Identify the needs and
expectations of potential
customers before they
experience the actual
delivery of care or service.
This should be an essential
part of designing new
services.
Delivering and managing care
while patients and families are
experiencing care or service. This
must be done at the site of care,
for all levels of care and along all
points of the continuum. These
contact points become the
“moments of truth.”
Determining how well customer-
focused care and service were
delivered. Evaluation is after the
delivery of care and should focus
on how well we met customer
expectations and needs and/or
the experience of the customer.

to a successful design. This type of listening is
frequently overlooked in health care.
A hospital I worked with in the Chicago
area successfully applied preservice listening
when they redesigned their outpatient testing
and therapy area. Focus groups were used to
collect data from patients, families, physicians’
office staff, and insurance companies. They were
all asked to describe how and when they would
like the testing and therapy process to work.
Because each group had a little different perspec-
tive on the situation, the redesign team was able
to gather a variety of insights that were used to
design the new testing and therapy area. If they
had not asked the customers before they designed
the new service area, they would have built a
facility and supporting processes that made
sense to the staff, managers, and designers of
the service but not necessarily to the customers.
The second time to listen to those we serve is
while they are experiencing service or care. This
is known as point-of-service (POS) assessment.
Many healthcare providers are actively engaged
in this type of listening. This can include patient
interviews, leadership walk-rounds,9 focus groups,
interactive TV surveys, recorded interviews
(audio or video), and computer feedback kiosks.
POS feedback can also come from unsolicited
feedback and complaints.
The third and most frequently used method
of listening to the VOC is postservice assessment.
In this case, patients are asked to provide their
opinions after they have been discharged. Most
postservice assessments are obtained through
mailed, telephone, or computer-based (email)
surveys. Postservice assessments have three
major advantages over face-to-face interviews:
■■ They can be more representative, reliable,
and valid than asking patients for opinions
while they are undergoing care.
■■ They provide patients with an opportunity
to objectively reflect upon their experiences.
■■ They allow patients to respond according
to their own time frames.
■■ They allow you to gather data from a larger
segment of your total population of patients.
Because surveys are used so frequently in
healthcare settings, I provide more detail on
this method in the next section.
▸▸ Understanding VOC
Tools
A comprehensive VOC measurement system
should combine all three types of assessments.
The first step toward building such a system is
to become familiar with all the different tools
and methods that can provide VOC insights.
TABLE 3-2 summarizes the dominant methods
TABLE 3-2 VOC feedback methods and tools
62

Service Level
Tool or
Approach Pre Pos Post Cost Advantages Disadvantages

Focus Groups X X Moderate ■■ In-depth qualitative data ■■ Low generalizability

to high cost ■■ Do not require large samples ■■ Requires skilled facilitators
depending on ■■ High flexibility ■■ Not anonymous
the number ■■ Can identify new issues or concerns
conduced and while conducting a focus group
who conducts
them

Observation X Low cost ■■ Easy to do ■■ Low generalizability

■■ Highly flexible ■■ Low value for comparisons
■■ Limited to publicly
observable behavior
Chapter 3 Measuring the Voice of the Customer

■■ Requires considerable time

and effort

Personal X X X High cost ■■ Very detailed data ■■ Very labor intensive

Interviews ■■ Easy to probe for additional data ■■ Very time consuming
■■ Effective with all socio-economic ■■ Quality of data depends on
levels skill of interviewer
■■ Not anonymous

Leadership X Low cost ■■ Provides real time feedback from ■■ Scheduling can be a
Walk-rounds staff challenge for senior leaders
■■ Identifies issues related to quality, ■■ Focusing the walk-round
safety and customer service discussions on key quality
■■ Engages senior leaders and staff in characteristics requires focus
dialogue and commitment
■■ Taking action on the
identified issues requires
leadership commitment
TABLE 3-2 VOC feedback methods and tools (continued)

Service Level
Tool or
Approach Pre Pos Post Cost Advantages Disadvantages

Unsolicited X X Very low cost ■■ Identifies extreme dissatisfiers/ ■■ Virtually no opportunity to

Feedback satisfiers generalize findings

High-Tec Tools X Very high cost ■■ Provides real time feedback ■■ May not be appealing to
(TV & Touchpads) ■■ User-friendly certain groups of respondents
■■ Flexibility ■■ No control over sample of
respondents

Experiential X High cost ■■ Tremendous depth of data ■■ Low generalizability for

(The “Mystery ■■ Can cover all aspects of a customer’s defining a “typical experience”
Shopper”) experience ■■ Requires the mystery
shoppers to be trained and
articulate

Surveys X X X Moderate ■■ Generalizability ■■ Requires rigorous protocols,

to high cost ■■ Offers continuous monitoring valid and reliable instruments
depending ■■ Provides comparative reference data ■■ May require sampling
on the ■■ Versatility ■■ Not highly flexible
methodology ■■ Reasonably quick to implement ■■ Unwillingness of individuals
used to participate
■■ Inability of respondents to
recall
Understanding VOC Tools

Modified from Advocate Health Care. Used by permission of Robert Lloyd.

63
64 Chapter 3 Measuring the Voice of the Customer
and tools used most frequently to gather cus-
tomer input. This table indicates at which of the
three measurement points (pre, POS, and post)
the method or tool can be used, as well as some
of the advantages and disadvantages of each
approach. Note that some of these methods or
tools can be used at multiple listening points.
▸▸ Focus Groups
Focus groups are typically the method of choice
for gathering detailed information after the
patient is discharged. The major drawback of
focus groups, however, is that they are generally
done with small groups (10–15 people) and the
results cannot be generalized to larger popu-
lations (even though many people try to draw
conclusions about the population represented
by the focus group). However, focus groups do
provide a wonderfully rich and detailed listening
mechanism. Remember—focus groups provide
depth but very little breadth or generalizability.
Another pitfall with focus groups is that the
people running the focus group frequently think
that they do not have to be very rigorous in the
planning and implementation of a focus group
session. Many of the same threats that challenge
the validity and reliability of surveys also apply
to focus groups. If focus group facilitators are
not trained in how to form and manage a focus
group, ask questions, and then build on the re-
sponses of the participants, they will end up with
a rambling manuscript that provides little useful
information. Focus groups are not informal chats
with customers. Focus group feedback should
not be used in isolation, however. It should be
combined with the results from leadership walk-
rounds and staff rounding and then compared with
the preservice and postservice results to build a
more comprehensive understanding of the VOC.
▸▸ Observation
Observation is an effective method for gathering
data on the current experience of customers. This
is actually one of the most insightful methods
of obtaining POS feedback. This approach is
relatively inexpensive (compared to surveys
and focus groups) because the principal costs
are essentially related to staff time. The observer
merely selects a reasonably unobtrusive location
and notes what occurs during a defined period
of time. With a minimum level of training,
the observer can record a fairly rich volume of
information. The key to success with observa-
tional studies, however, is that there needs to be
a standardized method for observing and most
important a consistent method for recording and
documenting the observations. Unfortunately,
all too often observational studies are done as
ad hoc and periodic exercises designed to see
if there is “a problem.” As a result, the observa-
tional studies produce a few anecdotal insights
but little in the way of data or information that
can be used to make improvements. The major
drawbacks with data collected through obser-
vational studies are:
■■ The observer can be in only one place at a
time. If you want to cross-validate observa-
tional results you will need to assign multiple
observers assigned to multiple groups.
■■ The observer sees events from a limited
time perspective (e.g., the observer was in
the emergency department [ED] for 3 hours
on a Monday morning).
■■ The results, like those for focus groups,
have very low generalizability (i.e., can
the observations made on Monday morn-
ing in the ED be generalized to the rest of
the day, to the other days of the week, or
to the other shifts? The simple answer is
“No.”).
■■ The observers need to be trained in a
standardized methodology so that they
are all looking for the same types of events,
situations, or encounters. Observation fre-
quently suffers from not only respondent
bias but also situational bias (i.e., the very
nature of the situation causes the observer
to selectively focus on some events and
ignore others).

■■ Finally, observational studies require having
individuals skilled in conducting qualitative
data analysis (Crabtree and Miller, 1992;
Krippendorff, 1980; Patton, 1987; Webb,
Campbell, Schwartz, & Sechrest, 1969).
▸▸ Personal Interviews
Personal interviews can be used at all three VOC
listening points. Like observation, this method
can provide very rich and detailed information.
Interviews have the added advantage, however,
of being able to provide direct one-on-one inter-
action with the customer. Interviews have a great
deal of appeal because they seem so simple on
the surface—find someone and go talk to him
or her. The challenge comes in having skilled
interviewers who are able to probe without
guiding the respondents to the answers being
sought. Personal interviews are not investigative
reporting sessions. Interviews need to be well
thought out and conducted with consistently
applied methods of inquiry. Finally, if the re-
sults are to be reflective of a larger population,
then the interviews need to be of sufficient
volume to capture a representative sample. This
can be quite time consuming, expensive, and
more involved that just talking to a few people.
▸▸ Leadership Walk-Rounds
One of the more successful methods used to
gather POS feedback is leadership walk-rounds,
sometimes called leadership rounds (Budrevics
and O’Neill, 2003; Frankel et al., 2005; IHI,
n.d.; Thomas, Sexton, Neilands, Frankel, &
Helmreich, 2005).9
In this approach to listening, managers
and administrators review a very brief set of
structured questions with patients and/or staff.
The leadership walk-rounds process of gather-
ing real-time feedback allows management to
(1) get out of their offices and visit the places
where work occurs, (2) listen to patients and
staff to determine which topics or issues need
Unsolicited Feedback 65
to be improved, and (3) determine priorities. An
additional benefit of conducting leadership walk-
rounds is that the patients and family members
find it very comforting to have someone from the
management team spend a few minutes talking
with them about their care and the processes they
are experiencing. This form of feedback not only
provides an opportunity for immediate service
recovery but also the ability to start identifying
patterns and opportunities for QI teams.
Two cautions with this type of feedback. First,
it typically produces more positive results than
postservice evaluations do. If you rely strictly
on this form of feedback, therefore, you will
probably start to think that you are much better
than you really are. The reason for this positive
bias is simple—patients are hesitant to be totally
honest with you while they are still under your
care. They are concerned that you may hold
something back or retaliate against them if they
are honest and tell you what they really think of
the care and service they are receiving. As a result,
it is not uncommon to have a frequent response
from POS assessments be, “Oh, everything is just
fine.” Second, if follow-up action is not taken
on the identified issues or topics presented by
patients or staff management loses credibility.
This is why I usually set up two key measures
when a group wants to engage in leadership
walk-rounds: (1) the number of issues identified
during the walk-round and (2) the percentage of
identified issues that were addressed (or better
yet resolved) within 30 days of identification.
Listening without responding is in most cases
worse than not listing in the first place.
▸▸ Unsolicited Feedback
Unsolicited feedback is one of the most effective
methods of gathering information on customer
concerns. Regrettably it is frequently overlooked
in healthcare settings and in some instances
ignored. Ignoring unsolicited feedback occurs
most often when customers are so upset that they
say, “I want to speak to someone in charge and I
want to speak to them right now!” Most often this
66 Chapter 3 Measuring the Voice of the Customer
happens when customers feel the need to voice
dissatisfaction with the care, service, or outcomes
they are receiving. Most unsolicited feedback
is negative, which is why some organizations
choose to bury it in administrative channels or
ignore it. But unsolicited feedback, especially
negative feedback, does provide an opportunity
to identify potential areas for improvements. The
central challenge with this type of feedback is
not to overreact to single events. Ask yourself,
“How many times has this happened in the past
week or month?” If it is an isolated incident, en-
gage in root cause analysis and service recovery
strategies immediately. If the complaint happens
repeatedly, consider collecting more data on the
issue to see if there is a pattern and a need for
widespread improvement work.
▸▸ High-Tech Tools
Increasingly healthcare organizations are discover-
ing high-tech tools as a way to listen to the VOC.
These include interactive televisions in patients’
rooms, small handheld computers (personal dig-
ital assistants or PDAs) used by nurses to record
patients experiences, and interactive kiosks. This
approach to listening is the most expensive of all
the methods, and the returns are frequently of
marginal value. High-tech tools are quite flexible
because they can be used 24 hours a day, 7 days
a week, 365 days a year and they do not require
direct staff interaction as is required by personal
interviews or focus groups. They are also able to
provide rapid-cycle feedback because the devices
for recording the interaction can be tabulated
on demand. Besides being expensive, the major
disadvantage is that high-tech tools appeal to a
limited segment of the population. Elderly patients
usually are not eager to engage with interactive
TVs and computer kiosks. Typically, high-tech
devices pique the interest of younger cohorts. I
had one organization, for example, tell me that
they discovered that one 13-year-old boy had
completed a patient satisfaction survey on an
interactive kiosk in the lobby 23 times. He was
bored waiting for his mother. Related to this
issue of limited appeal is the fact that it is hard
to control who uses them, how many times the
same person might interact with the same touch
screen, or most important, if they provide a
reliable estimate of the perspectives of the total
population being served.
▸▸ The Experiential
Shopper
The use of the experiential shopper has increased
in popularity over the past 5 years. This is a classic
POS method that stems primarily from the retail
shopping industry (i.e., the “mystery shopper”).
Its adaptation to the healthcare field has required
a number of revisions, however. In healthcare
settings, the experiential shopper is usually a
hired external consultant or a staff person who
poses as a patient to gain firsthand knowledge
of what it is like to be a patient. Typically, the
experiential shopper will call to see how long it
takes to get an appointment or have an outpatient
procedure done. I know of one medical group,
for example, where each week the CEO spins a
wheel similar to a roulette wheel that contains
the names of each medical practice around the
edges of the wheel. When the wheel stops, she
calls the particular medical office indicated and
asks how long it will take to get a prescription
refilled, get an initial visit scheduled, or obtain
a follow-up appointment. She then uses the
responses she gets as input for the weekly prac-
tice manager meetings. A new wrinkle on this
approach has been that some consulting groups
will actually hire out their employees to schedule
and experience selected procedures (e.g., blood
draws, x-rays, CAT scans, and believe it or not
some invasive procedures like colonoscopies)
to see how well the administrative and clinical
processes go. One of the major problems with
the high-tech methods is that they suffer from
low generalizability. If staff pose as the mystery
shopper, one of the side benefits of this method
has been that they typically gain a new apprecia-
tion of how patients view healthcare encounters.

▸▸ Surveys
Surveys are essential tools for a majority of the VOC
approaches outlined in Table 3-2. Focus groups,
personal interviews, and interactive high-tech
kiosks, for example, all require a survey format
of some sort to lay out the questions presented
to the respondents. The methods of distributing
a survey may differ (e.g., structured interview,
mailed, telephone interviews, an Internet survey,
or an interactive high-tech device), but they
all rely on a survey of some sort to start with.
Surveys can be used at all three listening points
(preservice, POS, and postservice) and if they
are properly constructed, administered, and
analyzed surveys not only can provide valuable
insights about current customer experiences but
also can provide an opportunity to make com-
parisons over time. Carey (1999) provides good
guidance on how to choose a survey system.
So what is a survey? Basically, a survey is
a standardized measuring instrument that is
designed to take concepts, ideas, or behaviors
that may be vague in nature (e.g., satisfaction,
excellent care, conservative, liberal, or quality)
and attempt to quantify these concepts so that
decisions can be made (inferred) from the data.
Key features of a survey include that it:
■■ Is purposeful and is designed to build a
systematic information collection system.
■■ Is focused on a question, problem of interest,
or issue and is designed to be exploratory
(i.e., what’s wrong?) in nature rather or
confirmatory (i.e., did we fix it?).
■■ Involves question design, a data collection
plan (including stratification criteria and a
sampling strategy), analysis, and reporting.
■■ Should have a degree of validity and reli-
ability. Even if a survey is considered to be
valid and reliable, based on psychometric
analysis, it must be realized that valid and
reliable in one context or for one segment
of the population does not necessarily mean
that the survey will be valid and reliable in
other contexts or with other segments of
the population.
Surveys 67
■■ Creates direct and indirect interactions
between the survey instrument, modality of
administration (e.g., mail, telephone, struc-
tured interview, focus group, or computer
based), the person conducting the survey
(especially if it is a structured interview or
focus group survey), the survey format, and
the respondent!
■■ Consists of individual questions (sometimes
called items) and potentially subscales. Sub-
scales (or scales) are groupings of individual
questions, usually in the range of five to
eight questions that are thought to “hang
together” and capture some underlying
concept, function, or domain. Examples
of subscales frequently formed for inpa-
tient satisfaction surveys include nursing
care subscale, physician communication
subscale, pain management subscale, or
comfort and cleanliness subscale. Not all
surveys are constructed around subscales.
External surveys developed by vendors or
oversight bodies generally will group the
individual survey questions into specific
subscales, so that comparisons of the vari-
ous function related to patient care can be
compared. It is important to realized that
subscale development is not an intuitive
process (i.e., you do not create subscales
because you or your colleagues “feel” that
a particular number of questions more
or less hang together and reflect a similar
concept). Subscales are created through a
fairly elaborate statistical analysis process
that is based on factor and correlation
analysis (Child, 1973; Harman, 1976; Kim
and Mueller, 1978a, 1978b). Many people
erroneously say they have “scales” on their
surveys, but these were most often not de-
rived by applying psychometrics methods
based on factor and related analytic tech-
niques. Note that most commercial survey
products have been put through factor
analysis methods, but most ad hoc (locally
developed) surveys have not. So, if someone
uses the term subscale or even scale, you
are well within your rights to politely ask
68 Chapter 3 Measuring the Voice of the Customer
them what they mean by the term. In some
instances, people use the terms subscale
and scale synonymously. Again, subscales
are groupings of individual questions that
have been shown statistically to capture
some underlying concept or domain. The
term “scales,” on the other hand, is used in
survey research primarily to refer to the
type of measurement used in a response
format (i.e., nominal, ordinal or rank order,
interval, and ratio scales). More will be said
about response scale shortly.
The details on these key features of surveys
can be obtained from a myriad of books that have
been developed on this subject over the years.10
To begin building your knowledge and skill in
the design and use of surveys, I suggest reading
a few of the classics in the field of psychometrics
and then some of the more recent the works.
Classic Readings in Survey Design and Use:
■■ Interviewing: Its Forms and Functions by
Stephen Richardson, Barbara Dohrenwend,
and David Klein. New York: Basic Books,
1965.
■■ Readings in Attitude Theory and Measurement
edited by Martin Fishbein. New York: John
Wiley & Sons, 1967.
■■ Social Psychology by Muzafer Sherif and
Carolyn Sherif. New York: Harper & Row,
1969.
■■ Attitude Measurement edited by Gene Sum-
mers. Rand McNally & Co., 1970. (This is a
wonderful collection of many of the classic
articles on attitude measurement.)
More Recent Books on Survey Design and Use:
■■ Asking Questions: The Definitive Guide to
Questionnaire Design—For Market Re-
search, Political Polls, and Social and Health
Questionnaires by Norman M. Bradburn,
Seymour Sudman, and Brian Wansink. San
Francisco, CA: Jossey-Bass, 2004.
■■ Measuring Customer Satisfaction and
Loyalty: Survey Design, Use, and Statistical
Analysis Methods, 3rd ed., by Bob E. Hayes.
Milwaukee, WI: ASQ Press, 2008.
■■ Internet, Phone, Mail, and Mixed-Mode
Surveys: The Tailored Design Method, 3rd
ed., by Don Dillman, Jolene Smyth, and
Leah Melani Christian. New York: Wiley &
Sons, 2009.
The Bradburn et al. book on Asking Questions is
a very good place to start learning about what
constitutes a well-written survey question.
Hayes’s 2008 book not only describes how to
construct a good survey but also lays out very
clear steps for including surveys into a QI pro-
gram, including the use of Shewhart control
charts for analyzing the results. Dillman’s work
(2009) is considered by most experts in this field
to be essential reading. He not only provides
guidance on creating what he calls the “total
design method” but also discusses in a very
practical manner the historical developments
surrounding the use of surveys, questionnaire
formats, open and closed-ended questions,
implementation and administration of surveys,
and the use of mixed-mode surveys. The first
edition of Dillman’s work was published in
1978. It is no wonder that this book is now in
its fourth edition. If you are going to read one of
the books in this listing I’d start with Dillman’s
book. You won’t be disappointed. Finally, in
addition to reading you might also consider
sending one or more individuals on your team
to a class or workshop on survey design. Fre-
quently local colleges and universities provide
a good resource for classes in psychometrics.
These course are usually taught in sociology,
psychology, or marketing research programs.
Now that we have outlined the key features
of a survey, let’s apply this knowledge to a survey.
Spend a few minutes reviewing the survey shown
in EXHIBIT 3-1. What is your opinion of this sur-
vey? Is each question clear and unambiguous?
Are the response options appropriate for each
of the questions? Do you think patients would
complete this survey or find it bothersome and
therefore not even finish it before leaving the
clinic? Develop some of your ideas and impres-
sions before reading my observations following
the exhibit.
 Surveys 69

EXHIBIT 3-1 Survey assessment exercise

Happy Valley Medical Clinic

Please complete this survey and give it to the receptionist before leaving.
Date of visit: ___________ Time: ___________ Dr. ____________
1. Was your appointment on time?
Yes _____ No ____ Not Sure ____
2. Were you greeted in a friendly manner and did the receptionist answer your questions?
SA _____ A ____ Not Sure ____ D ____ SD
3. Did you have to wait longer than expected to see the doctor?
Not Too Long _____ Sort of Long ____ Very Long ____
4. Where the chairs comfortable?
Yes _____ No ____ Not Sure ____
5. Did we treat you with dignity and respect?
Definitely ____ A Little ____ Somewhat ____ Not at All ____
6. Did the doctor answer all your question in a thorough and friendly manner?
Yes _____ No ____ Not Sure ____
7. How long ago did you make this appointment?
Date: _____________
8. Did you use our valet parking service?
Yes _____ No ____ Not Sure ____
9. Will you recommend our services to friends and family?
Definitely ____ A Little ____ Somewhat ____ Not at All ____
Use the following space to tell us anything else that concerns you about your health or our services.
_______________________________________________________________________________

You’ve probably concluded that this survey
has a number of problems. How many of the
following issues did you identify?
■■ Why didn’t a staff person fill in the admin-
istrative items (Date of Visit, Time of Visit,
and Doctor to be seen)? This is probably just
going to annoy the respondent. They are
thinking, “Why didn’t they fill this in? They
knew I had an appointment.” Little things
like this can create a negative context for
respondents and can bias their responses.
■■ Question 1: Was your appointment on time?
Yes, No, or Not Sure. This is a problem on
several counts. First, what is it trying to
ascertain? Is this trying to capture whether
the patient showed up at the appointed
time? Or is it asking about whether the
doctor was on time for the appointment?
The question is confusing. Second, the in-
clusion of Not Sure as a possible response is
not appropriate. This question is basically an
event question that should have simply a Yes
or No response format. But again the entire
question is confusing. An alternative would
be to offer forced-choice response categories
relating to how many minutes the patient
had to wait in the waiting area, in the exam
room, or until the doctor actually saw the
patient. This could also be done by having
the patient write down the time when each
of these steps in the process occurred.
■■ Question #2: Were you greeted in a friendly
manner and did the receptionist answer your
questions? This is a classic example of what
is known as a double-barreled question. It
70 Chapter 3 Measuring the Voice of the Customer
contains two parts. Part 1 is, “Were you
greeted in a friendly manner?” Part 2 is, “Did
the receptionist answer your questions?”
If both of these aspects are important then
they should be separate questions. When
given double-barreled questions respondents
are frequently caught in a quandary. What
if I think they did greet me in a friendly
manner but did not answer my questions?
What response do I give? Most often if the
respondent agrees with part of the ques-
tion but not the other half they will either
not answer the question or pick a neutral
middle of the ground response. The final
problem with this question is the response
choices. The use of a Strongly Agree (SA)
to Strongly Disagree (SD) response format
is not appropriate for this item. Also note
that they did not spell out SA, A, D, and
SD. Will the respondent know what these
abbreviations mean? Typically a response
scale based on level of agreement is reserved
for attitudinal questions. The wording of
Question #2 is more of an event-based item
in which case a Yes or No response option
would be adequate. Again, you do not need
a Not Sure response.
■■ Question #3: Did you have to wait longer
than expected to see the doctor? When a
question is trying to capture some aspect
of time the best approach is not to provide
vague concepts like Not Too Long, Sort of
Long, and Very Long but rather either (1) ask
the respondent to select categories of time
(e.g., less than 5 minutes, 5 to 10 minutes,
11 to 15 minutes, or over 15 minutes). A
preferable way to capture the time dimen-
sion is to ask a staff person or the patient
to write down the actual times when key
steps in the process occurred. This applied
also to Question #1. With the actual times
recorded you can now analyze the variation
in the process which you cannot do with
response options such as Not Too Long,
Sort of Long, and Very Long.
■■ Question #4: Were the chairs comfortable?
The issue here is obvious. What chairs? In
the waiting area, the exam room, in the
lab area where blood samples are drawn?
The lack of specificity will make it difficult
for the respondent to provide an informed
opinion. The Yes/No response options are
appropriate. The Not Sure option is not
applicable for this item. It would also be
an option to have a response format that
assessed the level of comfort (e.g., very
comfortable, somewhat comfortable, not
very comfortable, or very uncomfortable).
As a general operating rule you should
offer response choices that address the key
concept being assessed. Respondents get
confused and possibly irritated when you
ask a question and then provide a response
format that makes no sense. For example, if
the question is, “Do you have any questions
about the medicine the doctor prescribed
for you today?” then a reasonable response
option would be Yes or No. An inappropriate
response option for this question would
be to offer opinion-based responses such
as the level of agreement. You may laugh
because this seems so obvious but I have
seen surveys that are asking event-based
Yes/No type questions and then offer a SA
to SD response option.
■■ Question #5: Did we treat you with dignity
and respect? This is another double-barreled
question. Break it into two questions if both
concepts are important. The response options
are reasonable for this item.
■■ Question #6: Did the doctor answer all your
questions in a thorough and friendly manner?
Yes, you spotted another double-barreled
question that needs to be broken apart.
The doctor may have been a very friendly
person but provided incomplete answers to
the patient. How do they respond to these
two items when they are combined? The
response options are appropriate and in this
case the Not Sure option does have value.
■■ Question #7: How long ago did you make
this appointment? This question is another
annoying one because the office should know
when the patient made the appointment. If

they do not know the patient will surely not
know the date she called for an appointment.
■■ Question #8: Did you use our valet park-
ing service? This may be of interest to
the practice manager of the clinic but the
patient will most likely find the question
annoying. The best way to gather this bit of
data is to ask patients when they check in at
the registration desk if they used the valet
parking service. Staff should be collecting
the response to this item.
■■ Question #9: Will you recommend our
services to your friends and family? This is a
popular question on many patient surveys.
The issue with this item is the word “ser-
vices.” How is the patient to know which
services are being referenced? Registration
service? Doctor services? Nurse services?
Phlebotomy services? Valet services? If
each of these services is critical then list
them separately and have the respondent
provide opinions on each type of service.
The problem then becomes the length of
the survey and the time commitment you
are requesting of the respondent. The other
issue with this item is that there may be a
recommendation patients might make to
a friend that could be different from that
which they would offer to a family member.
For example, if the patient is a 42-year-old
female she might recommend the practice
to her friends but not to her 80-year-old
mother. Again, this has the potential to be
a double-barreled question. For the res­
ponse options, I would ask what is the dif-
ference between “A Little” and “Somewhat”?
These seem to both be dealing with the
same or similar level of magnitude. Given
the wording of this question the responses
of Yes, Maybe, and No could be appropriate.
■■ The final invitation to “tell us anything
else that concerns you about your health
or our services” seems, on the surface, to
be a friendly, open-ended request. But the
limited space provided suggests that we
really don’t want you to write too much.
They should either make the survey card
Surveys 71
bigger or instruct the respondent to use
the reverse side of the card for additional
comments.
Now that you have had time to evaluate a
survey and to start thinking about some key
components of survey design it is time to dive
into five topics that greatly affect the success of
your surveying efforts:
■■ The growth of surveys in healthcare settings
■■ Writing survey questions
■■ Response formats for the questions
■■ The logistics of survey administration
■■ Linking survey results to improvement
strategies
These five topics are addressed next.
The Growth of Surveys
The use of surveys in healthcare settings has
grown faster than the use of any of the other VOC
approaches and tools already discussed. There
are so many internal and external surveys within
the healthcare industry that a participant in one
of my workshops on the design, development,
and use of surveys asked at the beginning of the
class, “How long will this survey madness go on?”
My response was that the use of surveys is most
likely to continue and probably increase. In the
United States particularly, we have been survey
crazy for decades. This has been the case not only
in the healthcare industry but also in all sectors
of our society (remember Pam from the Acme
Storm Window Company whom you met at the
beginning of this chapter?). Other countries are
not as fixated on the use of surveys but there is
a growing tendency in western European coun-
tries, especially in the United Kingdom, to use
surveys in healthcare settings as the dominant
approach to determining whether the service
users’ expectations are being properly addressed.11
The growth in surveys has been charac-
terized by two major types of surveys: formal
and informal. Formal surveys come from two
primary sources. First, there are surveys required
by external agencies, ministries, inspectorates,
departments of health, and other groups that
72 Chapter 3 Measuring the Voice of the Customer
have oversight responsibility for the provision
and delivery of healthcare services and outcomes.
The second type of formal surveys is those that
are internally sponsored by the healthcare or-
ganization itself. Most of the time these surveys
are designed, administered, and processed by a
commercial vendor who then sends the analysis
and interpretation of the results to the sponsoring
organization. These surveys typically are aimed
at gathering data on patient, employee, and/or
physician satisfaction. The vendor is retained
to provide or develop valid and reliable survey
instruments, design sampling strategies, handle
the distribution and collection of the survey
results (usually by mail or telephone methods),
process the completed surveys, and then produce
statistical and graphical reports for management.
Although the formal surveys usually have
a high degree of visibility and require a line in
the organization’s budget to cover the costs, it
is informal surveys that are creating the most
challenges within healthcare organizations. These
informal or ad hoc surveys are usually conducted
in response to a problem or a need to answer the
ever popular questions: “How are we doing with
(fill in the blank)?” Or, “We have a problem with
(fill in the blank) so we better develop a survey
and get some views on how extensive the problem
is.” Popular topics for these informal surveys
include handwashing compliance, handoff at
shift change, communication among and be-
tween staff, medication availability, discharge
processes, clinic wait times, educational needs,
feedback on in-service training programs, phy-
sician satisfaction with laboratory services, the
culture for reporting errors, and the list goes on!
Although the individuals who create these
informal surveys have good intentions, fre-
quently their surveys suffer from (1) validity
issues (i.e., the survey does not actually measure
what it purports to measure), (2) reliability
issues (i.e., the survey cannot reproduce the
same or similar results under different admin-
istrations), (3) questions that are generally not
constructed properly, (4) inappropriate response
formats for the items or questions on the survey,
(5) inappropriate sampling and data collection
plans, (6) being single point-in-time (one-shot)
surveys rather than being done in a continuous
manner, (7) inappropriate statistical analysis, and
(8) the inability to have the results generalized
to broader segments of the population being
served. Yet, the survey development and distri-
bution process probably made the people who
created the survey feel like they did something
worthwhile or they could then tell their manager
that they “did a survey.” Too often, these home-
grown surveys provide inadequate and at times
misleading findings that are then used to make
poor management decisions.12
So, how many surveys are being used
throughout your organization? Think about this
for a minute. How many formal surveys do you
have to complete? More important, how many
informal or ad hoc surveys exist? When I ask
participants in my workshops to answer this
question I get comments like, “I have no idea how
many surveys are floating around the hospital”
or simply “Too many!” Based on the feedback
I get from workshop participants it is not un-
usual for a hospital of 200 beds to be sponsoring
50 or more different surveys throughout the
course of a year. Ultimately the critical question is,
once a survey is conducted what do you do with
the results? Do the results get used by anyone?
Repeatedly I have seen surveys conducted on
wards or units that end up sitting on a manager
or supervisor’s shelf.
To help you get a handle on the number
of surveys floating around your organization
I have created a Survey Inventory Worksheet
(EXHIBIT 3-2). The seven columns in the worksheet
are as follows:
■■ Column 1: What is the short title of the
survey?
■■ Column 2: Who is the target group for
the survey (e.g., patients, family members,
physicians, staff, and managers)?
■■ Column 3: Is the survey a commercial survey
provided by a vendor or an internal (ad hoc)
survey developed by staff?
■■ Column 4: What is the frequency of dis-
tribution for the survey (e.g., is a one-time
EXHIBIT 3-2 Survey inventory worksheet
Facility Name: _______________________________________________ Date: 

Column 5
Column 3 Column 4 What method is
Is this survey a: Frequency of used to collect Column 6
■■ Commercial distribution the survey How many Column 7
Column 2 Survey (CS)? ■■ A one-time responses (e.g., individual How do you
Column 1 Target group ■■ Internal (ad survey (OTS) mailed, telephone, questions intend to use
Short Title of the for the hoc) Survey ■■ A repeat survey? handout, are on this the survey
Survey survey (IS)? (RS) interview) survey? results?

CS  IS OTS  RS

CS  IS OTS  RS

CS  IS OTS  RS

CS  IS OTS  RS

CS  IS OTS  RS

CS  IS OTS  RS

CS  IS OTS  RS
Surveys

Page _____ of _____

73
74 Chapter 3 Measuring the Voice of the Customer
event or is it repeated at specific periods of
time)? If it is repeated, indicate the frequency
of administration (e.g., weekly, monthly,
quarterly, or biannually?).
■■ Column 5: What method is used to collect
the survey responses (e.g., mailed, telephone,
handout, or interview)?
■■ Column 6: How many individual questions
are on this survey?
■■ Column 7: How do you intend to use the
survey results (e.g., for improvement, per-
formance management, incentive bonus
payouts, or external requirements)?
Once you complete the inventory, use it as
a basis for a dialogue about the purpose of the
survey and how useful or valuable are the results
to the organization. Which surveys are essential
and must be done? How many are interesting
but not essential to the organization’s strategic
objectives. What surveys could you eliminate?
Try to get the total number of surveys reduced to
the vital few. I think you will be surprised at how
many surveys, formal and informal, are being
conducted that you did not even know existed.
Writing Survey Questions
It seems like writing survey questions should
be a rather straightforward activity. You have
in your mind something you want to ask an-
other person and you just write it down. Right?
Although this may seem like a simple task, the
major issue in writing survey questions is that
you are writing the question but someone who
does not have your frame of reference and view
of the world will be responding to the question.
This is the problem with many ad hoc surveys.
People with good intentions sit down and
write questions for a survey but never get any
input from the people expected to respond to
the questions. Consider Dillman’s guidance on
writing survey questions:
One must think about many things at
once to write a good question and failure
to do so can have significant effects on
how the question performs. (Dillman,
Smyth, & Christian, 2009, p. 67)
As you look at either commercially developed
surveys or your local ad hoc ones consider the
following issues when assessing the individual
survey questions:
■■ Will the aim of the questions be to capture
attitudes, opinions, behaviors, or events?
Each of these objectives requires different
approaches to asking questions.
■■ What type(s) of question(s) will you be
using? Will you be developing a survey
with open-ended, forced-choice (sometimes
called closed-ended) questions or a com-
bination of both types of questions? Each
type has advantages and disadvantages that
are summarized in TABLE 3-3. Open-ended
questions, as the term implies, provide a
sentence or two inviting the respondent to
provide, in her own words, commentary
on the topic of interest. Frequently, open-
ended questions are used to explore new
ideas or as a provocation to the respondent
when there is little existing knowledge on
the topic. Rich insights can be gleaned
from open-ended questions but there are
several obvious issues with including them
in a survey: (1) many respondents do not
want to take the time to write a response
that requires them to formulate their an-
swer then write or type it out; (2) there is
a high potential for nonresponse bias in
completing open-ended question due to
age, literacy, or language issues; and (3) tab-
ulation of open-ended questions requires
considerable time and effort as well as skill
in using qualitative methods to identify the
underlying constructs or issues identified
by the respondents. A final challenge I have
found with the use of open-ended ques-
tions in healthcare surveys is that they are
generally not very probing. They typically
have a lead-in that says something like this:
“Please use the following space to provide
us with and additional details on your visit”
or “Please use the following space to tell
us what went well or could be improved.”
Most of the time these invitations go
unanswered.

TABLE 3-3 Advantages and disadvantages of open- and closed-ended questions
Format Type Advantages
Open ended ■■ Reduces surveyor bias
■■ Enables rich and detailed
responses (stories)
■■ Can provide insights the
surveyor never anticipated
Closed ■■ Provides standardized responses
ended (aka ■■ Reduces ambiguity
forced-choice) ■■ Enhances comparability
■■ Reduces response time
■■ Facilitates coding of the
responses and tabulation
The majority of surveys in and outside of
healthcare settings use the second approach
to writing survey questions—forced-choice or
closed-ended questions. These types of questions
present a statement that is then followed by a
limited set of response choices. Some of the
response choices are offered in the form of a rank-
ordered progression (e.g., SA, Agree, Undecided,
Disagree, or SD) whereas others provide more of
a multiple-choice format where the choices are
not necessarily in any rank order or progression.
Response options of the multiple-choice category
usually are used to classify the responses but do
not have any particular valance or directionality.
Demographic questions, gender or age categories,
educational attainment, location of residence
(e.g., urban, rural, or suburban), or preferences
(e.g., from the list below please select the type of
car you drive) are examples of this type of type
of question. Another approach to this response
format is to use what Dillman et al. (2009, p. 3)
call the “nominal closed-ended questions with
unordered response categories.” In this case, a
question is posed and response choices are offered
but they are all independent responses that have
no overall directionality. I imagine you have
Surveys 75
Disadvantages
■■ Creates challenges for summarizing
the responses
■■ Many respondents don’t provide
written comments
■■ Increases the length of the survey, time
to respond, and potentially the cost
■■ Success is closely linked to how well
the questions were written
■■ Increased variation in interpretation of
the question
■■ Regression to the mean in response
patterns
■■ Measurement scale confusion
actually responded to a number of surveys that
had questions structured this way. For example
consider the following question: Which of these
five statements best describes your manager?
___ Supportive of the department
___ Supportive of me and my work
___ A real people person
___ Focused on results
___ Singularly driven by the department
budget
In this example, there is no sense of order or
a progression of good to bad or favorable to
unfavorable. The responses are independent
descriptors of how the manager is perceived by
an employee. The respondent is merely being
forced to select among several alternatives that
have equal valance.
Alternatives to this type of approach could
be to (1) create response choices that reflect a
rank order, progression, or weighting schema;
(2) allow the respondent to select more than one
of the five response options; or (3) use an open-
ended question to let the respondent describe in
her own words what she thinks of her manager.
76 Chapter 3 Measuring the Voice of the Customer
In summary, it is critical that you give some
thought to the type(s) of questions you have
on your surveys. One of the most important
issues at this point is that you give serious con-
sideration to how the types of questions will be
received, viewed, and answered by the targeted
respondents. As I mentioned earlier, too many
survey questions are developed not from the
respondent’s perspective but rather from the ■■
perspective of the question writer.
■■ What is the question stem? The stem of a
question (Dillman et al., 2009, p. 77–78)
consists of a very few words that lie at the
core of what the question is attempting to
ascertain. Often questions are developed
with entirely too many words that end up
confusing the respondent. What is the
essence of the question you are trying
to ask? Dillman et al. (2009, p. 77) write,
“Crafting good survey questions requires
understanding how each component of the
question conveys meaning independently
to respondents as well as how all the
parts work together to convey meaning.”
For example, look at the questions in
TABLE 3-4. The stem for each question is
shown in the right column of the table.
TABLE 3-4 Determining the question stem
Original Question
1. Not counting the number of times you might have
flown through O’Hare Airport or transferred to another
airline, how many times have you visited Chicago?
2. When you think about a recent successful vacation that
was not only fun but also added considerable value to
your family, how much did you spend?
3. Excluding any recent visits to the ED or the outpatient
clinic, how satisfied were you with the information
the nurses on the inpatient unit provided on your
condition?
Notice that the original questions contain
excessive wording that make the questions
confusing. The question stem, therefore,
captures the topic of interest with the fewest
number of words. Anyone developing a
survey should conduct an exercise with
colleagues to see if they can determine the
question stem for each item.
Will the wording and terms you use be
clear to the respondents? A majority of the
books on designing surveys point out that
the level of words used in a survey should
be geared to a to sixth- or seventh-grade
reading level. What happens in healthcare
settings, however, is that survey questions
frequently use words and terms that make
sense to the people writing questions but not
to the respondent. Dillman et al. (2009) go
so far as to recommend that if a word in a
survey question exceeds six or seven letters
a shorter and more easily understood word
should be substituted. For example, instead
of asking “Do you feel exhausted most of
the time?” substitute the word “tired” for
exhausted. Health care has a lot of medical
terms and jargon that make no sense to the
patients or their caregivers who are asked
The Question Stem
How many times have you visited
Chicago?
How much did you spend on your
recent vacation?
How satisfied were you with the
information the nurses on the
inpatient unit provided on your
condition?

to complete surveys. So instead of asking,
“Did the phlebotomist draw your CBC
specimen without causing you excessive
pain?” consider a shorter and less technical
question such as “Did the person who drew
your blood sample cause you any pain?” A
related aspect of choosing your words is to
make sure you avoid the double-barreled
question. I addressed this issue at the be-
ginning of this section when we looked at
a test survey but it bears repeating. This
is not a technical issue but a logical one.
People can legitimately have different
views on different topics so why combine
multiple concepts in one question. BOX 3-1
provides a few classic examples of how we
often write double-barreled question into
healthcare surveys. You will improve your
surveys tremendously if you become aware
of double-barreled question and vigilant in
detecting them. If the multiple components
referenced in the question are all equally
important then break them into separate
questions. If after detecting a double-barreled
question, however, you discover that the
BOX 3-1 Double-barreled questions
■■ Did we treat you with courtesy and
respect?
■■ Did the nurses and the doctors respond to
your needs and answer your questions?
■■ When you came to the clinic for your visit
last week, did you think that the way you
were treated demonstrated our corporate
values of compassion and dignity?
■■ In the past 30 days, have you taken all
the medicines we prescribed at time
of discharge and had any unusual side
effects?
■■ When you were ready to leave the clinic,
did you receive full information about
your copay and next appointment?
■■ As an employee of XYZ Medical Center, do
you feel the benefits are reasonably priced
and comprehensive?
Surveys 77
multiple items or actions being referenced
are not all equally important then drop
one of the items and shorten the question.
Response Formats
What response format options are most appro-
priate for the questions on your survey? This
topic probably has had more books and articles
written about it than any of the other items in this
section. The easiest way to set the context for this
topic is to tell a brief story. I was working with an
improvement team at a family practice clinic who
wanted to develop a survey that could be handed
out to patients upon arrival. They wanted to have
them fill it out as they progressed through their
visit and then deposit it in a box before leaving.
They also had a variety of questions they thought
they would like to ask. Before I even got to start
asking them about the number and types of ques-
tions they planned on asking or their sampling
strategy, one of the staff raised her hand and
proudly announced, “And you know we are going
to use a Likert scale.” Now this was not the first
time I have had this happen. I am not quite sure
how this statement has become such a popular
refrain when talking about the use of surveys in
healthcare settings but it happens often enough that
I now anticipate it when working with healthcare
teams on surveys. I find this reference particularly
curious because most healthcare professionals
have not had formal courses in survey design
yet they seem to know Rensis Likert as if he was
one of their regular patients.
Before we get into the details of what Likert
contributed to survey methodology we need
to back up a little. Let’s start by considering
the nature of response formats. A response
format determines how individuals can provide
feedback to a specific question on a survey.
Response formats are used almost exclusively
with closed-ended (forced-choice) questions.
Over the years there have been many response
formats developed by a variety of researchers in
the fields of sociology and psychology. Each of
these numerous response formats are dependent
on the type of measurement scale being used.
78 Chapter 3 Measuring the Voice of the Customer

Measurement scales are the means by which
we assign a number to an object or entity (Hayes
2008, p. 183). Stevens (1951) defined four basic
types of measurement scales: nominal, ordinal,
interval, and ratio. Although there has been
considerable debate in the literature over Stevens’
typology, these four categories of measurement
have remained the primary ways to organize
scales of measurement. TABLE 3-5 provides a
summary of the basic characteristics of these
measurement scales. If you are not familiar with
these four categories of measurement you can
gain considerable insight by typing “measure-
ment scales” into an Internet search engine and
you will gain a great deal of information on this
important topic.
With a little background in measurement
scales it is now time to return to our friend
Likert (pronounced correctly by the way as LIK-
дrt and not LY-kд rt). Rensis Likert (August 5,

TABLE 3-5 Measurement Scales and Characteristics

Scale Type Characteristics of the Measurement Scale Examples

Nominal ■■ Naming or categorizing objects or ■■ Male = 1, Female = 2

characteristics ■■ Single = S, Married = M,
■■ The categories are mutually exclusive Divorced = D
■■ If numbering is used it is only for ■■ The numbers used to identify
identification purposes players on a sports team
■■ Cannot use mathematical operations on
this type of measurement

Ordinal ■■ Objects are ordered according to some ■■ Win, place, or show in a horse
defined characteristic race
■■ But we do not know exactly by how much ■■ Low, medium, or high
they differ satisfaction
■■ Rank order hospitals in a
city by overall percentile on
inpatient satisfaction

Interval ■■ Items are ordered and the difference ■■ Temperature in Fahrenheit or

between each pair of items is the same; Celsius
distance is meaningful ■■ A 1–10 scale where the
■■ Average is meaningful, ratios are not patient response is Strongly
■■ No true zero level Agree, Agree, Neutral, etc.
when question is properly
worded and tested

Ratio ■■ Same characteristics as an interval scale ■■ Temperature in Kelvin

■■ But has a true zero point on the scale (i.e., ■■ Length, weight, distance,
no numbers exist below zero) time, etc.
■■ Ratios are meaningful
■■ All parametric statistics permissible

Note: Some writers describe interval and ratio scales as being virtually the same because it is difficult to find a legitimate interval scale that is not also
a ratio scale (Blalock, 1960, p. 15).

1903–September 3, 1981) was a psychologist
who was keenly interested in figuring out how to
measure the extent of a person’s beliefs, attitudes,
or feelings toward some object or issue. In 1932,
Likert detailed a technique for the measurement
of attitudes that was new, empirical, and quite
practical. Since then, he and others have made
a variety of adjustments and adaptations to his
original technique. I do not intend to go into all
the subtle statistical and methodological aspects
of Likert’s approach However, I will clarify a key
point related to Likert’s work and its application
to healthcare survey work.
Remember the statement, “And you know
we are going to use a Likert scale”? Well, in most
instances when people say this they are confusing
two critical aspects of Likert’s work. First, Likert
developed a balanced response format using a
five-point rank-ordered rating scale (e.g., SA,
Agree, Undecided, Disagree, and SD) that is
applied to an individual item or question on
a survey. Second, he defined what has become
known as a Likert scale, which is created from
the aggregation of the individual Likert items
on a questionnaire that capture different aspects,
concepts, or phenomena being studied. Therefore,
the term Likert scale, properly used, is the sum
of responses to individual Likert items using
the 5five-point Likert response format (scale).
As you have probably figured out by now
the term “scale” in survey research can actually
be somewhat confusing because it is used in
two different ways especially when we are con-
sidering the work of Likert. When most people
refer to a “Likert scale,” they are referencing the
five-point balanced response format (i.e., two
positive responses, a neutral midpoint, and two
negative responses). Likert’s principal concern,
however, was with creating collections of indi-
vidual items (questions) that capture or reflect
the underlying phenomenon being studied.
A Likert “scale,” therefore, cannot consist of
one or even two items on a questionnaire. He
claimed that you needed to have at least eight
items (questions) on a survey that address the
same topic or issue to even begin thinking about
Surveys 79
having the beginnings of a scale. The individual
items would then need to be tested statistically to
see if they “hang together” and capture the same
concept or issue. This is usually accomplished,
as mentioned at the beginning of this section,
by submitting the individual survey questions
to factor and correlation analysis.
In healthcare settings, therefore, when most
people say, “And you know we are going to use a
Likert scale” what they actually mean is that they
are applying a five-point Likert response format
to a bunch of independent questions that are not
related to the same underlying concept and have
little or no hope of ever forming a true Likert
scale. What the person on the family practice
improvement team should have said was, “And
you know that we are going to use a five-point
Likert rating scale as our response format.”13
The Logistics of Survey
Administration
Another challenge with conducting surveys,
especially ad hoc or locally developed surveys,
is that the logistics and administrative aspects
of the survey process are either ignored or for-
gotten about. When this happens, biases and
errors will be inherent not only in the design
and administration of the survey but also in the
results. TABLE 3-6 provides a summary of some
of the more prevalent sources of bias and error
in survey research.
There are many things you can do to avoid
or at least minimize the impact of the biases and
errors listed in Table 3-6. Some of these things
relate to the logistics of the survey process whereas
others relate to the design and formatting of the
survey. TABLE 3-7 provides a checklist and guidance
on how you can address and resolve bias, survey
error, and logistical challenges. These are actually
rather easy issues to address. All it requires is
to (1) be aware of the potential impact of each
of the various biases and errors on your survey
and (2) take steps to build knowledge about how
to create the structures and processes needed to
address each issue.
 80

TABLE 3-6 Potential sources of bias and error in survey research

Potential Source of
Survey Bias and Error Description

Motivation to respond Individuals are motivated for different reasons to respond to a survey. Some individuals are intrinsically motivated whereas
others require some form of extrinsic motivation (e.g., money, a coupon for free coffee or lunch at the hospital cafeteria).
Some individuals respond to a survey because they are loyal customers and feel a sense of “belonging” to the organization.
Others respond if they find the survey to be easy to understand and well designed. The point is that if you do not
consider and discuss the potential motivational factors for respondent participation your survey will probably suffer from
nonrespondent bias.

Coverage error This error occurs when not all the members of a targeted population have an equal and known, nonzero chance of being
included in the sample and when those excluded from the sampling strategy are different from those who are included.
For example, if at your clinic 63% of the patients are females and 37% are males do you know the historical probabilities of
each group completing the surveys you send to them? If you get 87% completed surveys from females and only 18% of
the surveys completed by males then you have a high likelihood of a coverage error. This also will occur when you decide
to send only to respondents who you think will be more responsive (e.g., female patients usually are more responsive than
Chapter 3 Measuring the Voice of the Customer

males so if you send to proportionally more females than males then you will experience a coverage error).

Sampling error Sampling error occurs when the precision of the survey estimates is limited because not every person in the population
is sampled. For example, a random sample of about 100 members of the U.S. general public can produce margins of error
of ± 10% with a 95% confidence level (i.e., 95 out of 100 times the estimate will be within ± 10% of the true value). But if
you surveyed 2,000 people the margin of error is reduced to about ± 2%. Sampling error is a frequently quoted statistic
especially when doing polls related to elections or critical national issues. Sampling issues are discussed in greater detail in
the next chapter.

Nonresponse error When you do not get everyone in the sample to respond to the survey, you will experience nonresponse error. This is
especially a critical error when the survey also suffers from the coverage error problem and those who respond have
different characteristics from those who do not respond.

Measurement error Measurement error occurs when a respondent’s answer to a survey item is inaccurate or imprecise. Measurement error is
usually the result of poor question wording or survey design. This is especially a problem with mailed or Internet surveys
when there is no one from the surveying group around to explain the meaning of a question.
TABLE 3-6 Potential sources of bias and error in survey research (continued)

Potential Source of
Survey Bias and Error Description
Wording of questions The details related to these two topics have been addressed earlier in this section. The key issues related to bias and error with
and response formats the wording of the questions include (1) using a reading or language level for the questions that is not appropriate for the
respondents, and (2) creating questions that are not clear, double barreled, or too wordy. Bias and error are also introduced
when the selected response format does not correspond to the logic of the wording of the questions. These problems were
also addressed earlier in this section. They occur most often, however, when the question has a stem that asks the respondent
to assess one level (e.g., the level of importance of some aspect of care) and then provides a response format that is based on
a rank ordering of levels of agreement (e.g., SA, Agree, Undecided, Disagree, SD). In this case, the question sets up a bait and
switch situation. The respondent was asked to indicate how important a topic or aspect is to her but is then given a level of
agreement as a response format. This creates confusion in the respondent and leads to response bias and errors in responding.

Reliability errors Most ad hoc or local surveys are not evaluated adequately for reliability and validity. Reliability refers to consistency and
the lack of distortion over multiple applications of the survey. Issues of concern with reliability include (1) test–retest
reliability (i.e., do we get similar responses with multiple applications of the survey?), (2) the level of inter-rater reliability
(i.e., agreement among different reviewers that the concepts or questions do in fact lead the reviewers to the same
conclusions), which is usually assessed with the kappa statistic (Cohen, 1960), and (3) internal consistency among the
individual questions on a survey which is usually assessed with Cronbach’s alpha (Cronbach, 1951).

Validity errors If the survey you have developed has not been assessed for validity you run a good chance of having the survey instrument
not actually capture the key construct or idea you are trying to study. Validity simply refers to the fact that the concepts,
constructs, or issues being measured are actually captured by the questions in the survey. A classic example frequently
referenced and debated, is IQ testing (i.e., does an IQ test actually measure IQ?). From a healthcare perspective we can ask
whether the medical record is a valid measure of the quality of care the patient received. The medical record reflects certain
aspects of the care experience, namely clinical details, but it tells only part of the story. In 1966, the American Psychological
Association’s Standards for Educational and Psychological Tests and Manuals listed three major types of validity: content
validity, criterion-related validity, and construct validity. Frequently, face validity is referenced as a fourth type of validity. But
face validity provides only an initial quick and dirty assessment of validity and is the weakest form of validity. Face validity
is basically a common sense approach to determine if the connection between the measurement instrument and the
construct or concept under investigation is apparent or self-evident. But, as Selltiz Johada, Deutsch, and Cook (1959, p. 165)
Surveys

point out, “Whether such an assumption (of face validity) is justified in any given case is ultimately a matter of judgment.”
Finally, note that reliability and validity are usually considered together. A test or set of questions might be reliable (i.e., the
survey gets roughly the same results upon multiple administrations of the instrument) but it could also have no validity
81

because the survey instrument does not adequately capture or measure the constructs being investigated). Additional
references on both reliability and validity include Bauer (1996), Bohrnstedt (1970), Hayes (1998), and Selltiz et al. (1959).
 82

TABLE 3-7 Addressing survey logistics and formatting issues

Who Is Going to… Why It Matters

. . . lay out the Frequently not enough time and attention are devoted to the graphic design and layout of a survey, especially ad hoc
survey, size the surveys. If it is a mailed or handed-out survey will you present the survey on a landscape or portrait format? Will you make
answer spaces, and full-page single- or double-sided surveys or make a folded booklet? What point size will you use to print the survey? What
graphically design color paper will you use to print the survey? You may be saying at this point, who cares? Well, research has shown that the
the survey? layout, spacing, font, bold/not bold, use of italics, and point size all can make a big difference in terms of response rates
on a survey. For example, elderly patients respond more completely and accurately to surveys that use simple words and
are printed on light yellow paper (rather than white) with a 14 point Arial font. Do you even think about the font and point
size used in making a survey? The science of visual perception has provided considerable insights on how the visual and
graphic aspects of a survey can play a key role in motivating a respondent to complete the survey (Hoffman, 2004; Palmer,
1999; Tufte, 1983, 1990, 1997; Ware, 2004). Dillman et al. (2009, p. 89–106) provide very practical guidance on the visual
presentation of surveys. It is a topic that most healthcare professionals grossly underestimate when they are developing
surveys.

. . . write the All too often surveys are distributed to patients or staff with no or inadequate instructions for responding to the survey or
instructions for what they can expect with respect to the results. Every survey, whether it be a mailed, phone, or Internet survey or focus
Chapter 3 Measuring the Voice of the Customer

responding to the group structured interviews should provide information on (1) the purpose of the survey, (2) how to complete or respond to
questions and what the survey, (3) how the results will be used, and (4) whether a copy of the results will be made available to the respondents.
will happen with the Organizations that are focused on transparency will also have a strategy for distributing the survey results to the public, the
final report? media, and possibly to local political leaders.

. . . identify the actual There is a need to keep the target groups for receiving the survey clearly defined. Identifiers for each respondent need to be
target group(s), established so that completed surveys can be distinguished from those not returned. Location codes and maps may also be
service lines, needed in order to determine if there are clusters of nonrespondents. Stratification may also be required to establish target
physician groups, etc. groups that have similar characteristics.
for the survey?

. . . define the How soon after a patient engages with healthcare services should they receive a survey? Will you construct a structured
distribution and interview while they are in your care? Maybe you will give them a survey at time of discharge. Some service providers favor
collection process? using phone surveys whereas others prefer mailing the patient a survey within 7–10 days of discharge. Will you consider
using an Internet survey? Will your patients respond to one of these approaches more positively than another? Will age,
gender, or socioeconomic status influence the distribution and collection process? You also need to make sure that some
people don’t get the survey at beginning of their experience whereas others in the same target group receive it in the
middle or at the end of their experience.
TABLE 3-7 Addressing survey logistics and formatting issues (continued)

Who Is Going to… Why It Matters

…follow up with Without a high percentage of returned and completed surveys you should analyze the results and make conclusions with
nonrespondents caution. Yet many surveys are processed with insufficient response rates. Although there are no response rate figures that
to obtain a high everyone agrees with there are guidelines that commercial survey groups do provide and to use as references. Check with
response rate on the your survey vendor if you have not seen the response rates that they use. In conducting staff or employee surveys you
survey? should be getting a response rate of 80% or better. Because the employees should have a vested interest in the organization
they work for we expect higher response rates from this group than from patients. For response rates related to hospitals,
clinics, home care services, mental health services, outpatient labs, and other allied health services, recommendations on
response rates depend in large part on the size of the sample or target population you are surveying. This is why your data
collection strategy is so important. For example, if someone says “We got a 50% response rate to our survey” you should ask
them how many surveys they sent out to get this number. If they sent out 20 surveys and got 10 back the 50% response
rate does not carry the same importance as having sent out 100 surveys and receiving 50 back. So, what will be your
process be for sending out reminders to those who do not respond to the survey? Will you call them, send them a reminder
letter with another survey and self-addressed stamped envelope, or send them a text message?

…scan the results Structures and processes need to be established for ongoing data collection, retrieval, and analysis of the surveys. If your
and/or enter them survey is being conducted by a commercial vendor this will be taken care of for you. But how will you process the ad hoc
into a database for surveys? Most healthcare organizations do not have a survey processing department. So these functions are often left up to
analysis? the team or group sponsoring the survey. Without a plan for these activities surveys frequently go unanalyzed or experience
long delays in processing the results.

…do the quality Even if you do have structures and procedures in place to scan and process completed surveys do not overlook the need for
checks on the quality control checks on the returned surveys. Steps should be established to ensure that the responses can be processed
completed surveys? (e.g., decide what to do with missing values, wrong responses, weak check marks, or lost surveys).

…own the analysis, Data without a context for action are useless. How will the results be reported and to whom (e.g., to respondents,
Surveys

reporting, and use of management, or staff )? There is a need to close the loop especially with the survey respondents. Do you share survey results
the survey results? transparently with those who provided responses? Do you publicly post the results? Finally, do you use the results to drive
improvement? Or, are they used only for incentive and bonus payouts in a PFP system?
83

The points of bias and error listed in this table have been adapted from Dillman, D., Smyth, J. and Christian, M. Internet, Mail and Mixed Mode Surveys: The Tailored Design Method. John Wiley & Sons, Inc., 2009. The detailed
descriptions of the points, however, were created by this author.
84 Chapter 3 Measuring the Voice of the Customer

Linking Survey Results of experiencing real improvement is greatly

to Improvement Strategies
From a QI perspective, listening to the VOC
is only a start. Listening to those we have the
privilege of serving, including staff and cowork-
ers, provides us with a baseline upon which we
can build and improve. Once we have received
customer input, however, the next step is to iden-
tify the processes that influence the customers’
perspectives. This is referred to as listening to
the VOP. Customers can tell us what is wrong
with the processes they experience. It is up to
the owners of the processes to take action and
make things better. When a department or unit
combines the VOC with the VOP, the probability
increased (Scherkenbach, 1991).
FIGURE 3-4 shows how listening to the VOC and
the VOP creates the foundation for improvement.
The improvement journey begins by conducting
an enumerative study14 at Time 1 to evaluate how
the patients feel about their experiences in the
ED. The survey gathered levels of satisfaction with
various aspects of patient care, such as waiting
time, explanation of diagnosis and treatment,
comfort of the waiting area, and adequacy of
discharge instructions. Suppose that the results
of the survey showed that waiting time was the
issue with which patients were least satisfied (i.e.,
less than 25% were satisfied) and that a majority
wanted to be treated and discharged in 2 hours

Voice of Voice of
the customer Voice of the process the customer
% satisfied with Waiting time in % satisfied with
waiting time Minutes emergency room waiting time
100 240 100

210

75 180 75

150

50 120 50

90

25 60 25

30

0 0 0
Time 1 Time 2 Time 3 Time 4 Time 5
Enumerative Analytical studies Enumerative
study study

FIGURE 3-4 Relating the voice of the customer to the voice of the process.
Reprinted with permission of ASQ

or less. As a result of this finding, a QI team was
initiated and charged with improving ED wait
times. So where do they start? They should begin
by realizing that the enumerative study they con-
ducted at Time 1 captured the VOC at a single
point in time. It provides a baseline of not only
what the patients and their families want, need,
or expect but also about the actual experiences
of those going through the ED processes.
Figure 3-4 shows that during Time 2, the
team collected data on the actual waiting time
for 13 consecutive patients and plotted the data
on a control chart. Results show that the average
wait time was about 160 minutes (a little over
2.5 hours) and that the process showed wide
variation, with some waiting times both above
and below the control limits. Analyzing the
wait time data over time in order to understand
the inherent variation in the process is what
Gitlow, Gitlow, Oppenheim, and Oppenheim
(1989) and Deming (1942) referred to as an
analytic study.15 (Control charts are discussed
in detail in Chapter 9. The distinction between
enumerative and analytic studies is discussed
in detail in Chapter 2.)
Next, the QI team developed an improve-
ment strategy (the exact details of which are
unimportant for this illustration) and tested
one improvement idea. They then collected
data on 20 new patients during Time 3 to see if
their improvement idea had an impact. As you
can see in Figure 3-4 Time 3 their improvement
idea did have the desired impact. The average
wait time during Time 3 was reduced to a little
over 2 hours and the degree of variation was also
reduced. However, this improvement was still not
meeting customer expectations (i.e., to wait less
than 2 hours). Therefore, another improvement
idea was introduced, and data were collected
on 20 more patients during Time 4. Results for
this period demonstrated that the average wait
time was reduced to approximately 90 minutes
and the upper control limit (UCL) was now
at the 2-hour target voiced by patients in the
enumerative study conducted during Time 1.
Finally, during Time 5, the same patient
questionnaire used during Time 1 was repeated
Surveys 85
(i.e., another enumerative study was conducted).
Results showed that the percentage of patients
satisfied with the waiting time had increased
substantially to approximately 70%. The analytic
study used to measure the VOP showed that
there had been two shifts in the process from
Time 2 to Time 3 and from Time 3 to Time 4.
The enumerative study (another patient sur-
vey) at Time 5 showed the extent to which this
improvement had been perceived by patients.
Although this example is only for teaching
purposes, actual QI teams use this line of think-
ing every day. Listening to the VOC without a
plan for improving the process they experience
demonstrates listening without responding. On
the other hand, listening to the VOP without
connecting the process’s performance to the
expectations of the customers sets the stage for
responding without listening. QI requires both the
VOC and the VOP, but it all begins with listening.
In this chapter, I have explored the key concepts
and methods required to build a VOC listening
system. Although a majority of the discussion has
focused on obtaining quantitative feedback from
customers I want to stress that you need to have
a balance between quantitative and qualitative
feedback. Although the numbers do provide a
useful foundation for building improvement
strategies, some of the most telling responses
come from the written and verbal comments
of the patients, their caregivers, and staff. One
comment that I read years ago from a patient
satisfaction survey still sticks out very clearly in
my mind today. It was from an elderly female
patient. The handwriting was a little unstable, but
the words were very clear. When asked what we
could do to improve the care and service delivered,
she responded, “I wish someone would have just
touched me and told me everything would be
all right.” We frequently forget that some of the
best healing comes from a simple smile, a pat on
the hand, or a few words of encouragement. We
deliver services that are much more important
than installing storm windows, yet we seem
to miss the basic point that Pam understood
when she called us during dinner—if you ask
customers for their opinions they will usually
86 Chapter 3 Measuring the Voice of the Customer
provide feedback. But it begins with a serious
desire to listen. Once you listen (really listen)
to those you serve, you will be better positioned
to respond appropriately.
Notes
1. It is interesting to note that the U.S. gov-
ernment clearly views health care not as
a science or a technology but as a service.
Historically, the U.S. government has
used Standard Industrial Classification
(SIC) codes to organize all industries
in the country. The service industries
are covered by codes 7200–8900. The
healthcare industry is specified by codes
8000–8093. In the United States, Canada,
and Mexico, the SIC coding system is
being supplanted by the six-digit North
American Industry Classification System
(NAICS code), which was implemented
in 1997. Although this new classification
schema is now being used throughout
North America, a number of U.S. gov-
ernmental departments and agencies
(e.g., the U.S. Securities and Exchange
Commission) still use the traditional
SIC codes. So, although we often hear
healthcare professionals talk about the
“science” of medicine and the technology
in our industry, the provision of health
care is classified as a service. This should
not come as a surprise to any of us who
have worked in the healthcare industry
for any length of time. Consider the
starting point—the Hippocratic Oath.
The modern version of the Hippocratic
Oath states, “I will remember that there
is art to medicine as well as science, and
that warmth, sympathy, and understand-
ing may outweigh the surgeon’s knife
or the chemist’s drug.” Another classic
perspective on the central role of the
patient in the care process came from
Dr. Francis Peabody. On October 21,
1925, he titled his lecture to his Har-
vard medical students “The Care of the
Patient.” Dr. Peabody closed his Harvard
lecture on this day with the following
statement, “For the secret of the care of
the patient is in caring for the patient”
In the early 1930s, the fictional Dr. James
Kildare emerged from the pen of author
Frederick Schiller Faust (pen name Max
Brand). Dr. Kildare was featured in
books, radio, and movies throughout
the 1930s, 40s, and 50s. In the 1960s,
the actor Richard Chamberlain played
Dr. Kildare, portraying him as a caring
physician who tended to the emotional
as well as physical needs of his patients.
This series was followed by the Dr. Ben
Casey character (1961–1966) played by
Vince Edwards. This program depicted
a highly compassionate surgeon who
fought the establishment to make sure
his patients received the right care at the
right time and tailored to their needs.
Probably the most well known of the
doctor shows of this era, however, was
Marcus Welby, MD, played by Robert
Young. This show ran from 1969 through
1976. In his portrayal of a family practice
doctor, Marcus Welby was the epitome
of the caring physician who would do
anything to make sure his patients were
treated both physically and emotionally.
Nothing could stop Dr. Welby from ex-
ceeding the expectations of his patents.
My point in recapping this bit of history
is that the recent push for and interest
in patient-centered care are not a recent
development. Even before these TV
shows the focus on the patient has been
an essential part of medical practice for
centuries. We just seem to be finding this
critical aspect of medicine once again.
2. I knew a physician who was an excellent
emergency room doctor. He could handle
trauma cases with skill and a calming air
that made the rest of the staff less tense
in difficult situations. In fact, he was so
highly regarded by staff and management
that the CEO promoted him to chief

medical officer (CMO). His office was
next to mine in the administrative wing
of the hospital. Although he did a good
job of striking a balance between all the
challenging and oftentimes conflicting
forces that must be addressed by the
CMO, he was internalizing many of
the issues. All of this eventually caught
up with him and he had a heart attack.
After a period of recovery and cardiac
rehab he returned to his previous role
as head of the emergency room’s trauma
services. When I had a chance to sit and
talk with him about this experience he
was quite honest and reflective. The best
comment he made was that he was much
more relaxed now that he could deal with
trauma cases than trying to manage all
the personalities and administrative is-
sues that came with the CMO position.
I thought this was quite interesting. He
found it more relaxing to treat gunshot
wounds, stabbings, car accident victims,
and strokes than trying to manage the
social and psychological aspects of human
behavior. When I asked him if he would
ever consider an administrative position
again, he quickly responded, “NO!” I
knew with that answer that this man will
not fall prey to the Peter Principle again.
3. One particular case comes to mind related
to a medical group that I consulted with
for several years. The management team
for this group decided to institute a PFP
system for the doctors. They created a
fairly complicated formula that included
measures such as number of appointments
booked, number of patients seen every
hour, patient satisfaction survey results,
revenue generated, and coworker feed-
back. They took all of these measures and
then weighted them to create a composite
index. In order to reduce the extremes
they discovered in the variation of the
index numbers, they created normalized
z-scores and then rank ordered all the
physicians in the group. The doctors in
Notes 87
the top decile received the highest PFP
bonus with lesser amounts given to the
remaining ranked doctors down to the
80th percentile. Then, and this is where it
became very interesting, they funded the
bonus payout pool by reducing the salaries
of the doctors who fell into the bottom
20% of the ranking. Talk about ways to
demotivate a group! A very unexpected
outcome of this procedure was experienced
by two of the doctors who were married.
The husband fell into the bottom 20% of
the ranking whereas his wife was in the
top decile. A postscript to this story. After
all this calculating and normalizing of the
index, the amount of money for the doctors
in the top decile was a little less than
$5,000. The staff received no bonuses.
4. Today we still struggle with getting people
to wash their hands properly before and
after seeing a patient. I find it fascinating
that regardless of what country I am
working in the issue of hand hygiene
is still a major challenge. Although the
practice of handwashing as envisioned
by Oliver Wendell Holmes may be uni-
versally accepted as being efficacious, it
certainly is not a part of daily behavior
of many healthcare workers. A number
of years ago when I was preparing a pre-
sentation on hand hygiene for a workshop
at the Illinois Hospital Association my
daughter Devon who was 8 at the time
came into my office and asked what I was
doing. I told her I preparing a talk on why
healthcare workers need to wash their
hands. She thought for a moment and
then said, “da ah, we learned to do that
in kindergarten.” I changed the title of my
presentation from Building a Successful
Hand Hygiene Program to If a Kid Can,
Why Can’t We?
5. Another well-known historical example
of how long it takes to have a proven
idea that provides medical benefit fully
embraced, implemented, and spread is
the story of lemon juice and scurvy. In
88 Chapter 3 Measuring the Voice of the Customer
1601, British Captain Sir James Lancaster
ran an experiment that helped to confirm
that the administration of lemon juice
prevented scurvy in his crew. On one of
his four ships making a trading voyage
to South Africa, he administered daily
doses of lemon juice to the crew. The
men on the other three ships did not
receive the daily lemon juice ration and
within 4 months of leaving England were
suffering seriously from scurvy. Lancaster
had clear evidence of the link between
lemon juice and the virtual elimination
of scurvy. An interesting sidebar is that
eventually in the mid-1800s lime juice was
also being used in addition to the lemon
juice to prevent scurvy in British sailors.
As the story goes, lime juice was added
to the sailors’ daily grog ration (water
and rum). This eventually led to a British
sailor being referred to as a “limey.” Over
time the term lost its naval reference and
was used to refer to any individual from
Britain. When Lancaster presented his
findings to the British Admiralty, how-
ever, they rejected his findings. Finally, in
1795, 194 years after Lancaster’s discovery
and after many unnecessary deaths, the
admiralty finally mandated lemon juice
be distributed to the sailors each day.
6. Lewis Carroll, Alice in Wonderland (New
London: Brimax Books, 1990, p. 55),
(originally published in 1865 by Macmillan
Publishing in London).
7. When I use the term “customer” I am
referring to anyone who receives the
output of one’s efforts. In a sense, we are
all customers and suppliers during most
transactions. When I am teaching a class,
for example, I am the supplier of infor-
mation to the participants, but I am also
a customer, because I rely on others to
register participants, schedule rooms, and
prepare handouts. My 9-year-old daughter,
Devon, is also my customer. I want her
to eat dinner, but I am not comfortable
with her preparing a meal from scratch
(especially lighting the gas stove). So I ask
her what she would like, rice or noodles.
I started doing this when she was about
3 years old. Back then she would look
intently at the rice and noodle packages.
For whatever reason at the time, she would
pick one of the rice products and my wife
or I would prepare it. Then Devon would
proceed to eat it. Ever since we have always
involved her in the decisions about what
we will eat. She is a customer, and we
should listen to her voice.
8. At the IHI’s 2002 National Forum on
Quality Improvement in Healthcare, Don
Berwick, MD, CEO of the IHI, provided
a wonderful keynote address by having a
three-way conversation with himself. He
cleverly played three roles: he served as the
narrator attempting to manage a debate
between Dr. Oldway and Dr. Neway. Dr.
Oldway was the traditionalist who pined
for the way things used to be. He did not
understand why the healthcare industry
no longer was centered around him, why
nurses no longer seemed to respect him,
why patients questioned him, and why all
these outside groups were sticking their
noses into his business. Dr. Neway, on the
other hand, argued that by allowing patients
to become more involved with their care,
it actually made life easier for the doctor.
It was a very engaging and entertaining
address and still available on videotape
from the IHI for those who are interested.
9. This version of rounding is quite differ-
ent from “clinical rounding” or grand
rounds.” In fact, most organizations that
have initiated rounding programs have
had to develop specialized training to
help staff realize that patient rounding
for service excellence is very different
from conducting clinical rounds or grand
rounds. Clinical rounding usually involves
reasonably small groups of professionals
seeking to confirm, refute, or debate
clinical findings, processes, and theories.
Grand rounds have emerged to become

educational lectures for larger groups on
selected clinical and operational topics.
Frequently grand rounds will be delivered
by outside invited experts in a particular
field. Rounding for service excellence, on
the other hand, is an interaction designed
to establish a dialogue between those
delivering care and support and the indi-
vidual, family, or caregiver experiencing
the care or support. Similar terms but very
different intents and designs.
10. It should be noted that survey research is
actually a fairly young field of endeavor.
Dillman et al. (2009) provide a very nice
overview of the history of survey research.
They trace how the survey process for the
first two-thirds of the 20th century relied
exclusively on personal interviews. Then
they describe how the nascent field of mail
and telephone surveys got off to a rather
shaky start in the late 1930s. The rather
unsuccessful launch of mailed surveys
is linked to the classic story of the U.S.
presidential election of 1936 between
Franklin D. Roosevelt and Alf Landon.
The Literary Digest (a popular magazine
of that day) conducted a mailed survey
of over 10 million people sampled from
telephone directories and auto registries.
They received over 2 million completed
surveys and predicted from the results
that Alf Landon would win the presidency
by almost 15%. But as any student of U.S.
history knows, Roosevelt won the election
with over 60% of the vote. So what went
wrong? Well, it was a sampling problem.
The Literary Digest ended up with a
very biased sample because only 36% of
household had telephones and even less
(about 20%) had cars. The people who
had cars and/or telephones were typically
individuals of wealth and status who were
primarily Republicans who voted for
Alf Landon, the Republican candidate.
Dillman et al. point out that this “botched
prediction cast a shadow over the use
of both mail and telephone surveys for
Notes 89
many decades” (3). Dillman and his team
then go on to describe the development
of mail and telephone surveys from the
1940s to the present. It is very interesting
and worthwhile bit of history for anyone
serious about survey research.
11. The United States clearly leads in the
proliferation and use of surveys. Western
European countries are becoming more
enamored with surveys than are eastern
European countries, the Middle East,
Africa, and the Far East. The United
Kingdom has seen a marked increase in
the use of national surveys over the past 5
years and I do not expect to see a decline
in this pattern in the near term. A major
difference between the use of surveys in
the United States and Western Europe,
however, is that in European countries
surveys are not distributed and collected as
frequently as they are in the United States.
It is not uncommon, for example, that a
healthcare system in the United States
will have a continuous process in place to
distribute and collect patient satisfaction
surveys on a weekly or monthly basis.
In the United Kingdom (UK), on the
other hand, national surveys are usually
done once every year or two. Another
difference is that in the UK there are
only a few standardized national surveys
developed and sponsored by the National
Health Service (NHS). In the United
States, healthcare providers are able to
select one or more survey vendors that
suit their needs and budget. As a result
there is considerable variation in the
types of surveys, the response formats,
and the procedures for data collection.
This creates huge challenges for making
comparisons. The only national stan-
dardized healthcare experience survey
in the United States is one required by
the Centers for Medicare and Medicaid
(CMS) called HCAHPS Hospital Con-
sumer Assessment of Healthcare Providers
and Systems (HCAHPS). The survey is
90 Chapter 3 Measuring the Voice of the Customer
designed to measure quality of care as
perceived by patients and the results are
reported publicly so comparisons can be
made among and across providers.
12. Now, I know that this sounds rather
negative. I do not mean to be rude or
insulting. But based on many years of
teaching survey methods and helping
healthcare professionals design, develop,
and use surveys correctly, I have seen a lot
of surveys that are excellent examples of
how not to develop a survey. But, this is
not surprising. It is not uncommon, for
example, for a group of employees to sit
down in the break room and develop a
survey that is distributed to patients or
family members by the end of the day.
In this case, good intentions frequently
lead to low levels of knowledge and poor
decisions. A healthcare analogy might be
useful at this point. We would not think
of asking a person who has no knowledge
of or prior experience with drawing blood
to just start sticking patients with needles
to obtain samples. Why, therefore, would
we expect people who have no formal
training in or experience with the design,
development, and use of surveys to produce
valid, reliable, and useful surveys? We
set up the workers for failure. Although
we have all filled out many surveys, the
mere completion of surveys does not
make us experts in survey methodology.
I have had people tell me in class, for ex-
ample, that “I know a lot about surveys.
Matter of fact, I filled out another one
just last week for the hotel where I spent
my vacation.” Filling out surveys is not
substitute for knowledge about survey
design, administration, and use.
13. I know, some of you at this point are saying
to yourself, “So what? Who cares? You told
me all this but it is a very fine point that
only statistical geeks care about.” Well, I
agree with you . . . to a point. The point
being that Likert is only one of numerous
individuals who have developed response
formats and “scales.” Gene Summers
complied in 1970 one of the best compen-
diums of classic readings on the field of
attitude measurement. In Summers’ book,
you will find articles not only on Likert’s
approach to developing scales but also
other pieces on alternative approaches
that have played a dominant role in the
literature including Guttman’s scalogram
analysis, Coombs’ unfolding technique,
Thurstone’s method of equal-appearing
intervals, the Bogardus social distance
scale, and Heise’s semantic differential.
My purpose in referencing these methods
is to highlight the fact that there is a great
deal more to the methodology behind
survey research than just saying, “And you
know we are going to use a Likert scale.”
So, I would suggest that you, or others
within your organization, build on your
current level of knowledge about survey
research and explore the alternatives to
what Likert proposed for both response
formats and building scales.
14. An enumerative study (Carey and Lloyd,
2001; Deming, 1942; Gitlow et al., 1989)
is done on a static population at a fixed
period in time. Its purpose is merely to
describe some variable of interest. Enu-
merative studies can be compared to taking
a snapshot with a camera. It provides a
limited view of events because it does not
focus on how things may vary over time.
An enumerative study is based on questions
such as, “What was the average turnaround
time last month? What was the mortality
for coronary artery bypass graft (CABG)
patients last year?” Enumerative studies
would tell us nothing about the processes
of care the patients experienced or why
they received the treatment. In the most
general sense of the word, enumerative
studies are classic examples of data for
evaluation or judgment. When you look
at the results of an enumerative study, you
are left with only a few options: you can
ask, “Does this outcome number strike

me as being acceptable or not?” or “Is
this outcome number different from the
previous number?”
15. Analytic studies are performed to answer
questions about a dynamic process. They
are not restricted to a single point in time
and are used to predict the future rather
than describe past outcomes. In this respect,
an analytic study can be compared to a
video recording rather than a snapshot.
Analytic studies are done to determine why
the outcomes were observed and how the
process that produced the outcomes can
be improved. An analytic study answers
questions such as, “What can we predict
about the length of stay for the coming
year?” or “What were the causes of the
observed decrease in surgical inpatients
for the previous year?” A more extensive
overview of the differences between
enumerative and analytic studies can be
found in Gitlow et al. (1989).
References
Balas, E., and S. Boren. “Managing Clinical Knowledge for
Health Care Improvement.” In Yearbook of Medical
Informatics 200: Patient-Centered Systems, edited by
J. Bemmel and A. T. McCray, 65–70. Stuttgart, Germany:
Schattauer Verlagsgesellschaft mbH, 2000.
Barry, M. J., and S. Edgman-Levitan. “Shared Decision
Making: The Pinnacle of Patient-Centered Care.”
New England Journal of Medicine 366, no. 9 (2012):
780–782.
Bauer, J. Statistical Analysis for Decision Makers in Healthcare.
Chicago: Irwin Professional Publishing, 1996.
Berwick, D. “Toxicity of Pay for Performace” in Measuring
Clinical Care – A Guide for Physician Executives. Tampa,
Fla.: American College of Physician Executives, 1995
and in Quality Management in Health Care, 1995, 4(1),
27–33.
Blalock, H. Social Statistics. New York: McGraw-Hill, 1960.
Bohm, D. On Dialogue. Transcription of a seminar conducted
by Dr. Bohm in Ojai, California, November 6, 1990.
Bohrnstedt, G. “Reliability and Validity Assessment in
Attitude Measurement.” In Attitude Measurement,
edited by G. F. Summers, 80–99. Chicago: Rand
McNally, 1970.
Budrevics, G., and C. O’Neill, C. “Changing a Culture with
Patient Safety Walkrounds.” Healthcare Quarterly Special
Issue, October 2003.
References 91
Campbell, D., and D. W. Fiske. “Convergent and Discrimi-
nant Validation by the Multitrait-Multimethod Matrix.”
Psychological Bulletin 56, no. 2 (1959): 81–105.
Carey, R. “How to Choose a Patient Survey System.” Journal
on Quality Improvement 25, no. 1 (January 1999): 20–25.
Carey, R., and R. Lloyd. Measuring Quality Improvement
in Healthcare: A Guide to Statistical Process Control
Applications. Milwaukee: ASQ Press, 2001.
Child, D. The Essentials of Factor Analysis. New York: Holt,
Rinehart and Winston, 1973.
Co, J., T. Ferris, B. Marino, C. Homer, and J. Perrin. “Are
Hospital Characteristics Associated with Parental Views
of Pediatric Inpatient Care Quality?” Pediatrics 111,
no. 2 (2003): 308–314.
Cohen, J. “A Coefficient of Agreement for Nominal Scales.”
Educational and Psychological Measurement 20, no. 1
(1960): 37–46.
Connellan, T., and R. Zemke. Sustaining Knock Your Socks
Off Service. Chicago: American Management Associ-
ation, 1993.
Crabtree, B., and W. Miller, eds. Doing Qualitative Research.
Newbury Park, CA: Sage Publications, 1992.
Cronbach, L. “Coefficient Alpha and the Internal Structure
of Tests.” Psychometrika 16, no. 3 (1951): 297–334.
Deming, W. E. “On a Classification of the Problems of
Statistical Inference.” Journal of the American Statistical
Association 37, no. 218 (1942): 173–185.
Deming, W. E. Out of the Crisis. Cambridge, MA: Massa-
chusetts Institute of Technology, Center for Advanced
Engineering Study, 1992.
Deming, W. E. The New Economics. Cambridge, MA: MIT
Press, 1994.
Dillman, D. A., J. D. Smyth, and L. M. Christian. Internet,
Phone, Mail, and Mixed-Mode Surveys: The Tailored
Design Method, 3rd ed. New York: Wiley, 2009.
Freiberg, K., and J. Freiberg. NUTS! Southwest Airlines’
Crazy Recipe for Business and Personal Success. Austin:
Bard Press, 1996.
Frankel A., S. P. Grillo, E. G. Baker, C. N. Huber, S. Abookire,
M. Grenham, P. Console, M. O’Quinn, G. Thibault, and
T. K. Gandhi. “Patient Safety Leadership WalkRounds™
at Partners HealthCare: Learning from Implementation.”
Joint Commission Journal on Quality and Patient Safety
31, no. 8 (2005): 423–437.
Garvin, D. “Building a Learning Organization.” Harvard
Business Review, 71, no. 4 (1993): 78–91.
Gitlow, H., S. Gitlow, A. Oppenheim, and R. Oppenheim.
Tools and Methods for the Improvement of Quality.
Homewood, IL: Irwin Press, 1989.
Grenning, T. “B. F. Skinner: 1904–1990.” Journal of Humanistic
Psychology 31, no. 2 (1991): 112–113.
Harman, H. Modern Factor Analysis, 3rd ed. Chicago:
University of Chicago Press, 1976.
Hayes, B. Measuring Customer Satisfaction: Survey Design,
Use and Statistical Analysis Methods. Milwaukee: ASQ
Press, 1998.
92 Chapter 3 Measuring the Voice of the Customer
Hayes, B. E. Measuring Customer Satisfaction and Loyalty,
3rd ed. Milwaukee: ASQ Press, 2008.
Herzberg, F. “One More Time: How Do You Motivate
Employees?” Cambridge, MA: Harvard University,
Graduate School of Business Administration, 1968.
Hoffman, D. Visual Intelligence. New York: Norton Pub-
lishing, 2004.
Institute for Healthcare Improvement. Patient Safety
Leadership WalkRounds.™ Institute for Healthcare Im-
provement Idealized Design Group and Allan Frankel,
MD ­Cambridge, Massachusetts, n.d. http://www.ihi
.org/resources/Pages/Tools/PatientSafety­Leadership
WalkRounds.aspx
Kim, J.-O., and C. Mueller. Factor Analysis: Statistical Methods
and Practical Issues. Sage University Paper Number 07-14.
Beverly Hills: Sage Publications, 1978a.
Kim, J.-O., and C. Mueller. Introduction to Factor Analysis:
What It Is and How To Do It. Sage University Paper
Number 07-13. Beverly Hills: Sage Publications, 1978b.
Kohn, A. No Contest. Boston: Houghton Mifflin, 1986.
Kohn, A. Punished by Rewards: The Trouble with Gold Stars,
Incentive Plans, A’s, Praise, and Other Bribes. Boston:
Houghton Mifflin, 1993.
Krippendorff, K. Content Analysis: An Introduction to Its
Methodology. Beverly Hills: Sage Publications, 1980.
Landsberger, H. Hawthorne Revisited. Ithaca, NY: New York
State School of Industrial and Labor Relations, 1958.
Langley, G., K. Nolan, T. Nolan, C. Norman, and L. Provost.
The Improvement Guide. San Francisco: Jossey-Bass, 2009.
Likert, R. A. “Technique for the Measurement of Attitudes.”
Archives of Psychology, no. 14, 1932.
Lloyd, R. “Improving Ambulatory Care Through Better
Listening.” Journal of Ambulatory Care Management
26, no. 2 (April–June 2003): 100–109.
Nelson, B. 1001 Ways to Reward Employees. New York:
Workman, 1994.
Oglesby, P., The Caring Physician: The Life of Dr. Francis W.
Peabody. Cambridge, MA: Harvard University Press, 1991.
Palmer, S. Vision Science: Photons to Phenomenology. London:
Bradford Books, 1999.
Patton, M. How to Use Qualitative Methods in Evaluation.
Newbury Park, CA: Sage Publications, 1987;
Peabody, F. “The Care of the Patient.” Journal of the American
Medical Association 88 (March 19, 1927): 877–882.
Peter, L. and R. Hull. The Peter Principle: Why Things Always
Go Wrong. Toronto: Bantam Books, 1970.
Schacter, D. Psychology, 2nd ed. New York: Worth Publish-
ers, 2011.
Scherkenbach, W. The Deming Route to Quality and Produc-
tivity. Washington, DC: CEEPress Books, 1990.
Scherkenbach, W. Deming’s Road to Continual Improvement.
Knoxville, TN: SPC Press, 1991.
Schultz, L. Profiles in Quality. New York: Quality Resources,
1994.
Selltiz, C., M. Johada, M. Deutsch, and S. Cook. Research
Methods in Social Relations. New York: Holt, Rinehart
and Winston, 1959.
Senge, P. The Fifth Discipline: The Art and Practice of
the Learning Organization. New York: Currency/
Doubleday, 1994.
Skinner, B. F. About Behaviorism. New York: Vintage
Books, 1974.
Stevens, S. Handbook of Experimental Psychology. New York:
John Wiley and Sons, 1951.
Stone, J. In the Country of Hearts: Journeys in the Art of
Medicine. Baton Rouge: Louisiana State University
Press, 1990.
Summers, G. F., ed. Attitude Measurement. Chicago: Rand
McNally, 1970.
Thomas, E., J. B. Sexton, T. B. Neilands, A. Frankel, and
R. L. Helmreich. “The Effect of Walk Rounds on Nurse
Safety Climate Attitudes: A Randomized Trial of Clin-
ical Units.” BioMed Central Health Services Research
(April 11, 2005). doi: 10.1186/1472-6963-5-28
Tseng, M. M. and J. Jiao. “Mass Customization.” In Handbook
of Industrial Engineering, Technology and Operation
Management, edited by G. Salvendy, 3rd ed. New York:
Wiley, 2001.
Tufte, E. The Visual Display of Quantitative Information.
Cheshire, CT: Graphics Press, 1983.
Tufte, E. Envisioning Information. Cheshire, CT: Graphics
Press, 1990.
Tufte, E. Visual Explanations. Cheshire, CT: Graphics
Press, 1997.
Ware, C. Information Visualization: Perception for Design.
San Francisco, CA: Morgan Kaufmann, 2004.
Watkins, K., and V. Marsick. Sculpting the Learning Orga-
nization. San Francisco: Jossey-Bass, 1993.
Webb, E., D. Campbell, R. Schwartz, and L. Sechrest Un-
obtrusive Measures: Nonreactive Research in the Social
Sciences. Chicago: Rand McNally, 1969.
Webster’s II New Riverside University Dictionary. Boston:
Riverside Publishing Company, 1984.
Wick, C., and L. Leaon. The Learning Edge: How Smart
Managers and Smart Companies Stay Ahead. New York:
McGraw-Hill, 1994.
CHAPTER 4
Milestones in the Quality
Measurement Journey
L
istening to the voice of the customer (VOC)
provides the starting point. Once you un-
derstand the wants, needs, and expectations
of your internal and external customers, which
are usually expressed as concepts (e.g., “I want
better health service,” “Why don’t you have shorter
waiting time?,” or “Communication between
the staff needs to get better”), it is up to you to
translate these concepts into indicators that can
be measured and tracked to determine whether
your processes are capable of meeting the VOC
expectations. Unfortunately, in health care, it is
often the case that indicators are selected not
because the providers of a service actually took
time to listen to the VOC, but rather because
(1) they made a priori decisions that they know
what is best for the customers, (2) they have
been given measures by external oversight or
regulatory bodies that require certain measures
be submitted to them, or (3) they take the mea-
surement journey shortcut by selecting indicators
that they have “always collected” and assume
that these are good enough for the purposes at
hand. This chapter has been designed to provide
you with a roadmap for selecting and building
© Michal Steflovic/Shutterstock
indicators that will help you move from concepts
to quantifiable measures and connect the VOC
with the voice of the process (VOP).
▸▸ Developing a
Measurement
Philosophy
The search for a few good indicators begins by
having a clear understanding of why you are
engaged in measuring performance in the first
place. Historically, healthcare providers have
collected and analyzed data strictly for internal
purposes that were directed at improving clinical
and operational effectiveness and efficiency. Over
the years, however, the growing external demand
for data has shifted much of the focus away from
an internal need to understand the effectiveness
and efficiency of the organization’s processes to
one of addressing external demands that lead to
judgment. In other words, the business community,
regulatory bodies, government officials, the media,
93
94 Chapter 4 Milestones in the Quality Measurement Journey
and consumers are all interested in answering a
very simple question: “Which provider is the best?”
In an effort to answer this question, many
initiatives, projects, and pieces of legislation
have been developed over the years. The goal of
these efforts has been to develop “report cards”
or “score cards” on healthcare providers that can
be used by various groups and consumers to
make decisions about their healthcare choices.
Regardless of the country or the approach to
funding health care, however, there seems to be
no quick or easy answer to the simple question
of “Which provider is the best?”
What has been the typical response to this
question is that external groups voluntarily ask
for or mandate certain performance indicators
from providers. These numbers are then combined
with those from other providers, risk adjustments
may be applied to the data to account for severity
of the patient populations, and finally, reports
are released to the public. These releases usually
stimulate the following chain of events:
■■ Local and/or national media become interested.
■■ Investigative reporters are sent out to discover
why Your Hospital has a higher coronary
artery bypass graph (CABG) mortality
percentage than My Hospital and why both
are higher than the average for the county,
region, province, or country in which they
are located.
■■ The reporters present their findings in the
next day’s newspaper or on the 6 o’clock
news, which usually focuses on the providers
at the top and bottom of the list.
■■ Your Hospital and My Hospital convene internal
meetings to develop strategies (­ rationales)
for countering why their numbers are higher
or lower than the average.
■■ Consumers become confused and/or cynical
because the data do not necessarily reflect
their experiences (e.g., “My father went to
Your Hospital for his heart operation and
everything was fine” or “My father went to
Your Hospital and nearly died”).
Whatever your view on the public release of
data, it is quite obvious that the demand for data
on provider performance and greater transparency
will increase over the coming years. The simple
question is “Are you prepared for it?” Healthcare
organizations that have a measurement strategy
and a proactive plan for investigating their own
results will be in a much better position to deal
with external scrutiny than those that sit back
and hope that the local or national news service
does not show up outside their hospital. (Refer
to Chapter 2 for more on provider performance
and transparency.)
Even though there is a renewed interest in
the public release of provider data, the more
important reason for knowing your data better
than anyone else is that it is the right thing to
do and it makes business sense. The complexity
of today’s healthcare delivery system requires
that leaders have a clear understanding of their
processes and the related outcomes. In order to
meet operational and financial objectives, patient
safety goals, and customer service expectations,
healthcare providers should consider developing
what Caldwell (1995) refers to as a “strategic
measurement deployment matrix.” Such a matrix
combines strategic vision with tactical measures.
It allows an organization to determine if the things
they are working on are really connected to what
the organization is supposed to be achieving.
The first step, therefore, on the quality mea-
surement journey (QMJ) is achieved by having
some sense of why you are measuring and your
approach to measurement. Is measurement a part
of the organization’s day-to-day functioning? Or
is it something that is done periodically in order
to prepare reports for board meetings or external
oversight bodies? A good place to start is to develop
the organization’s measurement philosophy and
share it with staff, patients, and caregivers. This
does not need to be a long document. Something
as simple as this could serve as a starting point:
Responsible leadership demands that we
know our data better than anyone else. It
further requires that we have processes
in place to accurately and consistently
obtain a balanced set of measures that
monitor clinical outcomes, functional

status, customer satisfaction, process
effectiveness, and resource utilization.
Finally, we are committed to using these
data to develop improvement strategies
and then take ACTION to make these
strategies a reality.
An organization needs to have a serious
dialogue about its measurement philosophy
and why it is measuring (i.e., for improvement,
for judgment, or for research). Included in this
ongoing dialogue should be specific discussions
about the role of indicators and how they will
(and will not) be used. Without a measurement
philosophy, your efforts to identify key indicators
and collect and analyze data will be nothing
more than a random walk.
Ideally, indicators should be designed to
improve quality by:
■■ Moving us away from anecdotes and focusing
on objective data
■■ Enhancing our understanding of the vari-
ation that exists in a process
■■ Monitoring a process over time
■■ Seeing the effects of changes made to a process
■■ Providing a common frame of reference
■■ Providing a more accurate basis for prediction
Unfortunately, many organizations run into
serious roadblocks when they attempt to select
indicators and use them to improve quality.
▸▸ Measurement
Roadblocks
Many things impede good measurement practice.
Based on my 40 plus years of working in the
quality measurement arena, I believe there are five
major roadblocks that people usually encounter
in their QMJ:
Roadblock #1: Measurement
Is Threatening
This is probably the largest roadblock we face
with healthcare measurement. There are many
Measurement Roadblocks 95
examples of how data have been used both
internally and externally to (figuratively) “beat
people up.” We often hear coworkers say that
they did not want to take the monthly numbers
to the boss because he or she “won’t like these.”
Organizations have long memories when it
comes to the use of data. Seasoned employees
quickly tell new workers what happens when the
numbers do not meet management’s ­expectations.
Quickly, the new workers hear the story about
how Gwenn, nurse manager of 3 East, did not
get her patient satisfaction scores up by the end
of the year and now Gwenn is no longer with the
organization. What the new workers didn’t know,
however, is that Gwenn left because her husband
was transferred to another city. But her leaving
and the decline of her unit’s patient satisfaction
scores do provide the basis for a compelling yet
causally incorrect story. As time passes, this story
becomes legendary, gets embellished a little, and
becomes part of the organization’s folklore. “Re-
member what happened to Gwenn” becomes the
standard response whenever someone’s patient
satisfaction scores are below the expected targets.
What I find absolutely fascinating, however,
is the fact that people actually like to measure
things, including their own performance. There
seems to be a natural curiosity in human beings
about measurement. When my daughter Devon
was 9 years old, for example, she loved to measure
things. When I was in the garage one day doing a
project she came up to me and said, “Measure me,
Daddy.” I took my tape measure and proceeded
to measure her height. She acknowledged the
measurement and went about her business. Ten
minutes later, she returned and stated, “Measure
me, Daddy.” I said, “Devon, I don’t think you
have grown much in the last 10 minutes.” But
she insisted and seemed to find the actual act
of measurement not only enlightening but also
entertaining. The next time I observed her in
the garage, she was using the tape to measure her
bike, her doll, and the dog (or at least trying to
measure the dog). Even adults love to measure
what they do. I have a number of friends who
participate in triathlons. They are very meticulous
about measuring and monitoring their training
96 Chapter 4 Milestones in the Quality Measurement Journey
regimens. I have seen similar behavior from
people involved with bowling, cycling, and golf.
How do we drive what seems to be an almost
natural curiosity about measurement out of
people when they get into work situations? The
answer to me seems rather simple. Organizations
frequently use data to instill a sense of fear in
the employees. Once data are used for judgment
and fear then the data are not for learning and
improvement but rather for intimidation and
control. It is not surprising, therefore, that the
workers rapidly conclude, “Why should I partic-
ipate in a measurement system that will be used
against me?” Several years ago I experienced this
attitude when I was facilitating a team that was
attempting to reduce call button response time.
During a meeting that was intended to identify
a measurement plan, one team member blurted
out, “Why don’t you go measure 4 West? I know
they are worse they we are.” When measurement
becomes threatening, the workers will conclude
that measurement should be for someone else,
not for them. The truth of the matter is that the
primary audience for measurement is the man-
ager of the department or unit and the workers.
These are the people who own the process and
who should be responsible for its performance.
If the organization does not have a philosophy
of measurement and a set of related tactics for
deploying measurement throughout the organi-
zation, then measurement will generally become
a threat. A strategic focus on measurement as
described by Caldwell (1995) will do wonders
to overcome this roadblock.
Roadblock #2: The Desire
for Precision
Health care is not classified as a science. The
federal government actually classifies healthcare
jobs as service jobs, along with car repair, lawn
service, and beauty shops. Sure, we use science
and technology to accomplish what we do, but
by and large health care is considered a service.
It is interesting, therefore, that many people in
our profession use the illusion of precision as
a convenient excuse for not measuring. I have
heard, for example, the following responses many
times when I asked a team if they had finished
their measurement plan:
■■ “We think it will take a little longer to make
sure the survey is right.”
■■ “The log sheet does not seem to capture all
the elements we think we need to collect.”
■■ “Why don’t you check with us in a couple
of weeks? We might have a better plan in
place at that time.”
The key point is that quality measurement
does not have to be as precise as many people seem
to think. We are not conducting research to win
the Nobel Prize in physiology or medicine. We
are trying to understand the variation that lives
within our processes in order to make things more
effective and more efficient for those we serve.
Therefore, the concept of measurement that is
“good enough” needs to be our guiding principle.
The basic purpose of quality measurement is to
inform the team or organization about its general
direction and whether it is moving toward its
goals and objectives. You do not need p-values
at the 0.01 or 0.05 level of significance to tell you
this. As one chief executive officer (CEO) told me,
“If it passes the sniff test, that’s good enough for
me.” Furthermore, we are not trying to conduct
research that is designed around the randomized
control trial (RCT) approach. RCTs are essential
to test theories and build new knowledge. This
is how medical science has advanced. But when
we are engaged in quality improvement (QI) we
are designing analytic rather than enumerative
studies as was discussed in Chapter 2. Do not
make your measurement efforts so precise and
pure that you never proceed to the most important
question: “Are we making a difference?” In short,
if an organization spends its time developing
academically or scientifically precise measures,
it will probably never get started on its QMJ.
The desire for precision will be a convenient
detour in your QMJ and an excuse for avoiding
the measurement mandate.

This detour was demonstrated very nicely to
me by a group of physicians during an evening
meeting designed to discuss their hospital’s QI
plan. The manager of quality was doing a very
good job of presenting the plan and the related
indicators. Then she got to the project on deep
vein thrombosis (DVT). She described the
indicator (percentage of patients evaluated for
DVT risk) and then showed the historic baseline
and the results for the last 8 months. Instead
of discussing why the hospital’s performance
on this indicator was declining, the physicians
became embroiled in a debate over the num-
ber of charts being reviewed and whether the
sample of patient charts had sufficient “power”
to be statistically significant. The detour they
took was based on not understanding sampling
methods for QI projects. The sample pulled for
the improvement project (20 charts per month
through a stratified random process) was good
enough for the purposes at hand (i.e., deter-
mining how well the hospital was evaluating
the risk for a DVT). As I sat and watched this
discussion unfold, I realized that it was a perfect
example of Roadblock #2. They were questioning
the method and the data instead of discussing
the processes by which they evaluate a patient’s
potential for a DVT. Precision was creating a
roadblock for improvement.
Roadblock #3: Using Standards
as Performance Objectives
Standards basically set limits on performance. In
fact, standards are usually considered minimal
acceptable levels of performance. Excellence
is a very different concept. For example, when
you go to a restaurant you have certain stan-
dards you expect to have without paying. You
expect to have a table, a chair, eating utensils,
a napkin, salt and pepper, and a water glass.
What if, however, the waiter showed you to an
open area of the restaurant that had none of
these expected standard components and told
you that the table would cost $20, a chair $15,
Measurement Roadblocks 97
utensils $10, salt and pepper $2 each, a napkin
for $4, and a water glass will cost you $5? You
went to the restaurant with expectations that
certain minimal standards would be met before
you ordered your meal. Not finding them you’d
probably leave.
What are the minimal acceptable standards
in a hospital or medical setting? In the United
States, standards for healthcare organizations are
set by a variety of governmental and nongov-
ernmental bodies. The Joint Commission (JC),
which accredits hospitals and other healthcare
providers, is a dominant player in this field. The JC
sets standards and regularly through announced
and unannounced visits provides assessments on
whether or not the facility “met the standards.”
Once a standard is achieved, however, complacency
often sets in and people say, “What more do you
want from us? We met the standard.” I have heard
many healthcare professionals claim that they
did not have to get any better because they were
already at the JC standard. I guess this means they
believe that their performance is acceptable and
in need of no further improvement. If standards
serve as the goal for the quality journey, then it
will be a limited journey.
What worked to satisfy customers or meet
the prescribed standards today may not be
acceptable tomorrow. For example, assume
that you met the JC standards during your last
survey review. What are you going to do when
the Centers for Medicare and Medicaid Services
(CMS) starts releasing hospital data showing that
your facility is “significantly” above expected
mortality percentages for the treatment of heart
attack patients? Your insistence that you met the
JC standards will carry little weight at this point.
The concepts of baseline, target, and goal provide
a much better frame of reference than standards.
Compliance with standards and the desire to
perform only at this level, therefore, guarantee
that an organization is not really committed to
QI. Improvement is a never-ending pursuit of
excellence. Meeting standards is acceptance of
current performance and a willingness to say,
“We’re good enough.”
98 Chapter 4 Milestones in the Quality Measurement Journey
Roadblock #4: Limited
Knowledge of Statistical
Process Control
This roadblock relates to the use of statistical
techniques, such as Shewhart control charts, to
(1) understand the variation that lives in a process
and (2) determine whether interventions have
actually made a difference in the performance of
the process. Most healthcare professionals have
had at least one course in statistics at some point
in their careers. Yet exposure to basic statistics
is not sufficient for those who plan to manage,
coach, or lead improvement efforts.
Statistical process control (SPC) is a separate
and distinct body of knowledge from what many
refer to as “traditional” or enumerative statistical
methods. Individuals who attempt to apply statistical
notions (such as testing the null hypothesis and
using p-values to determine statistical significance)
to their QI efforts will quickly make the wrong
decisions and then become disillusioned.1 The
reason for this disillusionment is simple: they
are using statistical techniques and methods that
are designed to answer questions about efficacy
instead of techniques designed to answer ques-
tions about effectiveness and efficiency (Brooke,
Kamberg, & McGlynn, 1996).
I have been teaching SPC methods to
healthcare professionals for more than 30 years.
During this time, there has been an increase in
not only the level of knowledge that healthcare
professionals have about SPC but also its appli-
cation to healthcare issues. Organizations like
the American Society for Quality (ASQ) and
the Institute for Healthcare Improvement (IHI)
have made major contributions to spreading
statistical thinking and the use of SPC methods.
But we are still at the beginning stages of this
journey when compared to the use of SPC in
manufacturing and industry. Use of SPC in these
sectors can be traced back to the mid-1920s
when Dr. Walter Shewhart first formalized the
theories and methods behind the control chart.
In 1931, Shewhart published Economic Control of
Quality of Manufactured Product, which stands
even today as the landmark reference on SPC.
The good news is that healthcare professionals
are becoming more aware of what SPC can do
to assist them in their quality journey. The bad
news is that we still have a long way to go before
statistical thinking is commonplace throughout
the healthcare industry.
Roadblock #5: Numerical
Illiteracy
Having skills in the use of SPC is not enough
to produce world-class quality. SPC provides a
wonderful foundation, but the real test comes
in applying SPC knowledge to overcome the
fifth and final roadblock—numerical illiteracy.
Wheeler (1993, p. vi) describes numerical illit-
eracy as follows: “Numerical illiteracy is not a
failure with arithmetic, but it is instead a failure
to know how to use the basic tools of arithmetic
to understand data. Numerical illiteracy is not
addressed by traditional courses in primary
or secondary schools, nor is it addressed by
advanced courses in mathematics. This is why
even highly educated individuals can be nu-
merically illiterate.”
What is needed to overcome numerical
illiteracy is what the Statistics Division of ASQ
calls “statistical thinking.” The vision of the
Statistics Division is that statistical thinking
will be found in all aspects of organizational
behavior and performance. FIGURE 4-1 depicts
Systems thinking Statistical methods
Process Variation Data Improvement
Philosophy Analysis Action
Statistical thinking everywhere
FIGURE 4-1 Vision of the ASQ Statistics Division
Reprinted with permission of ASQ.
 Milestones in the Quality Measurement Journey
this vision. Statistical thinking encompasses five
key components:
■■ Systems thinking
■■ Statistical methods
■■ Philosophy (of measurement)
■■ Analysis (and interpretation)
■■ Action
As you can see from Figure 4-1, knowledge of
statistical methods is only one aspect of statis-
tical thinking. Statistical thinking is a much
broader notion that has the ability not only to
overcome the numerical illiteracy roadblock but
also to provide a clear roadmap for the entire
quality journey.
Deming’s views on the value of statistical
thinking are well known and have been clearly
detailed in his writings (1950, 1960, 1975, 1992,
1994). Mann (1989) provides an excellent over-
view of how Deming and his colleagues regarded
the role of statistical thinking with examples of
how Deming influenced statistical thinking in
this country and in Japan. In Chapter 3, titled
“Statistical Methods for Tapping into the Infor-
mation Flow Generated by a Process,” Mann uses
the following quotation from Deming to clarify
the difference between using common sense and
statistical thinking to make decisions: “There
are many hazards to the use of common sense.
Common sense cannot be measured. You have to
be able to define and measure what is significant.
Without statistical methods you don’t know what
the numbers mean” (Mann, 1989, p. 62).
Along this same line, Mann references the
following point made by William Conway, the
former CEO of the Nashua Corporation: “He
pointed out [during a panel discussion] that
one of the greatest handicaps of people who
are trying to improve productivity and quality
is that they attempt to deal with these matters
in generalities. The use of statistics is a way of
getting into specifics that will allow managers
and workers to make decisions based on facts
rather than speculation and hunches” (Mann,
1989, p. 62). In short, statistical thinking is a
99
way to approach all aspects of work. It is a way of
thinking about numbers and how they can be
used to make improvements. Statistical thinking
is the primary way to immunize yourself against
numerical illiteracy.
The five roadblocks described in this sec-
tion are not insurmountable. The first step in
overcoming them is merely to be aware that they
exist. Once they are acknowledged and under-
stood, then it is time to take steps to immunize
yourself against their proliferation. The rest of
this text is directed toward this goal.
▸▸ Milestones in the
Quality Measurement
Journey
Any successful journey begins with a plan, a good
roadmap and a clear understanding of the key
the milestones along the way. Developing good
indicators is not all that different from planning
a good road trip. The roadmap we use to guide
our QMJ is shown in FIGURE 4-2. A completed
QMJ is shown in FIGURE 4-3. The details on each
milestone are presented next.
Welcome to Conceptland
There are two major segments of the QMJ. In
the first segment of the journey, you will come
upon two of the milestones: Aim and Concepts.
Specifically, the team will need to develop an aim
for the improvement work and then identify the
relevant concepts that characterize or capture
this aim. But know full well that these milestones
take you to “Conceptland” not “Measurement-
land.” Many people seem to live permanently in
Conceptland. This is not a bad place to visit but
if you never leave this land your QMJ will come
to an abrupt and unproductive end.
The first milestone, therefore, is to establish
an aim for your improvement work. What does
100 Chapter 4 Milestones in the Quality Measurement Journey
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
FIGURE 4-2 Milestones in the quality measurement
journey
AIM - reduce patient falls by 37% by the end
of the year
Concept - reduce patient falls
Measures - Inpatient falls rate (falls per 1000
patient days)
Operational Definitions - # falls/inpatient days
Data Collection Plan - monthly; no sampling;
all IP units
Data Collection - unit collects the data
Analysis - control chart
(u-chart) ACTION
FIGURE 4-3 A completed quality measurement
journey
the improvement team want to accomplish? How
good do they want to be? By when do they plan
to accomplish this outcome?2 An aim statement
is like a compass, it sets the direction for the
QMJ and points you toward your destination:
Measurementland. But Aims and Concepts are
not indicators.
In Figure 4-3, we see that the team’s aim is
to reduce inpatient harm by 37% by the end of
the calendar year. Is this an indicator? No. It is
a desired end state or a vision of what could be.
So, we ask the team to be more specific in order
to measure the dimension of “patient harm.”
Then they respond, “OK, a key dimension of
inpatient harm is falls. So we need to reduce
inpatient falls.” Is reducing inpatient falls an
indicator? No. It is a concept that captures
one dimension of harm. We still do not have
specific quantifiable indicators that allow us to
measure inpatient falls. Stating an aim or even
the concepts that further define the components
or aspects of an aim causes teams to live in
Conceptland.3 For example, when I ask teams
or managers what they plan to measure they
say things like, “We need to improve patient
satisfaction,” “We need to reduce medication
errors,” or “We need be more efficient.” Again
these are visions of what might be. They are
noble and good but the statements are visions or
desired end states. The problem is that these are
not even aim statements because the concepts
they are referencing (i.e., patient satisfaction,
medication errors, or efficiency—concepts not
indicators) do not have a specific reference as
to how good they want performance to be and
by when they expect to achieve this result. They
are visions of what might be.
This does not mean that these two milestones
are not important. These types of statements are
essential in order to get a team pointed in the
right direction for the start of their QMJ but
such statements only provide a vague sense that
we need to go “that way.”4
In Figure 4-2 and 4-3, therefore, you can
actually draw a line after the second milestone
(Concept), which serves as a frontier or barrier
that separates the two milestones in Concept-
land from the remaining five milestones in, you
guessed it. . . “Measurementland.”
You Are Now Entering
Measurementland
Because many people (e.g., board members,
nonexecutives, senior leaders, the press, ­political
leaders, and even patients) either live in or
frequently visit Conceptland, an organization
needs to have individuals with skills that can
move teams and leaders beyond visions, aim
statements, and concepts to address the milestones
marking the road through Measurementland.
These individuals need to have skills in building
 Milestones in the Quality Measurement Journey
quantifiable indicators (e.g., a count, a percent-
age, a rate, a score, an index, days between an
event or cases between events) that are accepted
as reasonable ways to capture the concepts of
interest, build data collection plans, and have
knowledge of applying SPC methods to the
collected data.
Indicator Milestones
Before you actually start your measurement
journey and reach the individual milestones in
the QMJ, however, you need to make two brief
stops in order to prepare for your journey. The
first stop is where you need to decide upon the
types of indicators that appropriately capture
the team’s aim and related concepts that need
to be tracked. The second stop is where you
will need to select specific indicators within
the various types you have identified. Let’s start
by considering the various types of indicators
that could be used to capture a team’s aim and
related concepts.
Besides the seminal work of Florence
Nightingale and Ernest Codman (see Chapter
1 for details) Avedis Donabedian is another
physician leader who contributed significantly to
the field of indicator development. Donabedian
provided the first contemporary framework for
developing what I consider to be a balanced
set of indicator types related to the delivery of
medical care. In his classic two-volume work
Explorations in Quality Assessment and Mon-
itoring (1980, 1982), Donabedian described,
in considerable detail, three key points in the
delivery of medical services:
■■ Structures (the tools, resources, and orga-
nizational components)
■■ Processes (activities that connect patients,
physicians, and staff)
■■ Outcomes (results)
He then suggested that measures should be
developed to capture these three dimensions of
medical service. Even though Donabedian pro-
vided a simple model for organizing indicators,
like Codman, he, too, was a little ahead of his
101
time. Most healthcare professionals during the
early 1980s did not readily embrace Donabedi-
an’s model for evaluating medical quality or his
suggestions for building indicators that represent
structures, processes, and outcomes.
Kaplan and Norton (1992, 1993, 1996) made
major contributions in this area by describing
the components of what they call a “balanced
scorecard.” Even though their work has been
directed more toward for-profit companies,
the basic message they present is applicable
to the healthcare industry. Specifically, they
argue that “no single measure can provide a
clear performance target or focus attention on
the critical areas of the business” (1992, p. 71).
According to Kaplan and Norton, an organization
should monitor a set of “balanced” indicators
that represent the key strategic areas in the or-
ganization’s business plan. A well-selected and
organized set of indicators should also place
strategy and vision, not control, at the center of
the organization (1992, 79). The key word for me
when I first read the work of Kaplan and Norton
was “balanced.” Health care has a long and rich
history when it comes to tracking data. What
we have not done particularly well, however,
is to make sure that the data we do collect is
tied to our strategic objectives and represents,
therefore, a balanced set of measures that cut
across the full range and scope of the clinical,
operational, and customer-focused services
being delivered.
The Joint Commission, a U.S.-based and
international accreditation body for healthcare
providers, in 1993 identified nine dimensions
of clinical performance that could be used to
categorize indicators:
■■ Appropriateness
■■ Availability
■■ Continuity
■■ Effectiveness
■■ Efficacy
■■ Efficiency
■■ Respect and caring
■■ Safety
■■ Time lines
102 Chapter 4 Milestones in the Quality Measurement Journey
The Institute of Medicine’s (IOM) report
Crossing the Quality Chasm (2001) played a
major role in identifying six aims that many
organizations use to organize their indicators:
■■ Safe
■■ Effective
■■ Patient centered
■■ Timely
■■ Efficient
■■ Equitable
Another very useful way in which to think
about categories or types of indicators is the
value compass (Nelson, Batalden, & Godfrey
2007). The authors propose two forms of the
value compass: one for clinical systems and the
other for the patient. The clinical system value
compass proposes organizing indicators around
functional outcomes, clinical outcomes, customer
satisfaction, and costs or resource issues. On the
patient side the four dimensions are similar but
have slight modifications to accommodate the
VOC: functional status, expectations (of the pa-
tient), clinical status, and costs or resource issues.
At the IHI we teach teams to consider
three types of indicators: outcome, process, and
balancing. This is what we refer to as a family of
measures that capture three distinct and critical
aspects of any improvement effort:
■■ Outcome Indicators: These indicators
should reflect and capture the VOC. How
is the process or system performing in light
of the stated aim? What are the results?
How close are the observed outcomes to the
specified targets or goals? How satisfied are
the individuals who receive the outcome(s)
of the process?
■■ Process Indicators: These should reflect the
processes and their related indicators that
drive the outcomes. How much variation is
there in the process? Are the parts or steps in
the process or system performing as planned?
Are the process indicators you select causally
connected to the outcomes?
■■ Balancing Indicators: These indicators
help you look at a process or system from
different directions or dimensions. Balancing
indicators help you think about unanticipated
consequences or other factors that might
influence the outcome. Indicators of this
type will help you determine if (1) you have
improved one aspect of the system but made
something else worse or (2) witnessed an
improvement in the outcome that was not
causally related to anything that the team
actually did to change the process.
TABLE 4-1 provides examples of outcome, process,
and balancing indicators for a family practice
clinic.
In Table 4-1, the topic of interest is fo-
cused on the patient experience. Two concepts
are being addressed: (1) waiting time and
(2) patient satisfaction. Two outcome indicators
have been identified: (1) the total length of time
(in minutes) for a scheduled appointment at
the clinic (note that it is only for scheduled
appointments); and (2) the percentage of
patients marking Strongly Agree to the single
question “Would you recommend our clinic to
family and friends.”
In terms of process indicators, the team
decided to track four dimension of the care
process. Two of the indicators relate to different
components of waiting (i.e., check-in time to
being seen by the doctor and time spent wait-
ing for ancillary service). The third indicator
focuses on the discharge process and whether
the patient received appropriate discharge in-
structions related to the reason for the visit. The
fourth and final process indicator is qualitative
in nature (patient and staff comments on the
flow of the process).
The final column in Table 4-1 addresses
balancing indicators of which there are four.
Remember that in specifying balancing indicators
we are attempting to understand whether our
improvement efforts are creating any unintended
consequences. For example, consider the first
balancing indicator volume of patients. What if
the volume of patients coming to the clinic or
scheduling appointments for a particular month
declined? What impact might a declining number
 Milestones in the Quality Measurement Journey
TABLE 4-1 Outcome, process, and balancing indicators for a family practice clinic
Topic Outcome Measures
Improve waiting Total length of
time and patient stay (in minutes)
satisfaction in the for a scheduled
family practice clinic appointment at the
clinic
Percentage of
patients marking
Strongly Agree to the
question “Would you
recommend our clinic
to family and friends?”
of visits or scheduled appointments have on wait
times? We would most likely see a drop in wait
times to see the doctor because there are fewer
people in the pipeline and thus the backups and
delays would be reduced. So, when you show up
for your appointment you get to see the doctor in
less time than your previous visit simply because
the volume of patients coming to the clinic has
been reduced. Chances are that patients would
also be more satisfied with the process (one of
the outcome indicators) because they waited
less time to see the doctor.
Similarly, if the percentage of patients
leaving without being seen by the doctor (the
second balancing indicator) goes up, chances
are that those who do not leave will be seen
faster and therefore also have higher satisfac-
tion levels. In both these situations, the team
has done nothing to intentionally improve the
process. Other factors (i.e., reduced volume
and an increase in the percentage of patients
leaving without being seen) created a false
impression that things have gotten better.
Now consider staff satisfaction as a balancing
indicator. What if the improvement team did
103
Balancing
Process Measure Measures
Time from check-in Volume of patients
until seeing the Percentage of
doctor patients leaving
Patient/staff without being seen
comments on the by a doctor
flow Staff satisfaction
Percentage of Financials
patients receiving
discharge material
Wait time for
ancillary services (lab,
x-ray, ultrasound)
during a visit
make changes in the clinic’s process (the de-
tails of which are not important at this point)
that actually reduced wait time and improved
patient satisfaction? But when you assess staff
satisfaction you discover that it is going down.
When you talk to the staff you get comments
like this: “Sure the changes you made to the
process have improved things for the patients
but they have made operating conditions for the
staff more complicated. If this continues I know
of a couple staff members who are considering
leaving the clinic.” In this case, what have you
gained? You have improved the status of one
group (i.e., the patients) and compromised
another group (i.e., the staff). Deming referred
to this as suboptimizing the system. Balancing
indicators help prevent suboptimization and
make sure you are considering unintended
consequences of your efforts.
Irrespective of the various types of indicators
that could be identified, the key point is that
a balanced approach to the types of indicators
is far superior to a narrow focus. A singular
or narrow focus on one or even two types of
indicators will lead to shallow knowledge and
104 Chapter 4 Milestones in the Quality Measurement Journey
ultimately suboptimal performance of improve-
ment teams. A balanced approach to indicator
development does not mean, however, that you
have to measure 30 or 40 indicators. Focusing
on the vital few (with emphasis placed on the
word “few”) is preferable to assembling an
unmanageable array of indicators that require
a small army to collect, analyze, and interpret.
More will be said on this point in Chapter 5
when we look at the development of strategic
dashboards.
▸▸ Selecting a Specific
Indicator
Once you have decided which types of indicators
are most appropriate, the next step is to select the
specific indicator(s) that will be measured within
each type. Although this seems like a straight-
forward activity, I have found it surprising how
many teams struggle with this task. An indicator
is a specific quantifiable aspect of an outcome or a
process. Yet all too often teams confuse themselves
by wandering around in Conceptland and never
move on to the detailed markers indicating that
they have actually entered Measurementland. For
example, in Figure 4-3 the concept of interest is
to reduce inpatient falls. There are two critical
aspects of moving from this concept of reducing
inpatient falls to actually measuring whether
falls have been reduced. First, the team needs to
decide on the specific quantifiable indicator(s)
that will represent inpatient falls. Second, the
word “reduce” is not relevant to the selection
and development of the indicator(s) that will
be used to measure the concept of inpatient
falls. Whether the indicator demonstrates a
reduction or an increase or stays at the current
level is irrelevant to specifying the indicator.
We will find out if the indicator is moving in
the desired direction once we collect data and
move to the analysis milestone in the QMJ. Until
then, the targeted direction for the indicator
(i.e., an increase in indicator X or a decrease in
indicator Y) has no added value to the naming
and specification of the indicators. Too often,
however, teams focus almost exclusively on the
direction of change, the target, the expected goal,
or the desired end state and end up developing
confusing indicators.
In terms of our inpatient falls example from
Figure 4-3, the critical question is what specific
indicators do you propose to develop that capture
the concept of inpatient falls? The following
specific indicators could be used:
■■ The number of inpatient falls (e.g., a simple
count of the number of inpatient falls each
day or week)
■■ The percentage of inpatients who fell once
or more while they were in the hospital
■■ The falls rate, which includes multiple falls
by the same patient during their admission
and is defined as the number of falls per
1,000 inpatient days5
■■ Days between inpatient falls
Each of these indicators identifies a specific
way to look at the inpatient falls concept. Each
indicator has value and the team will have to
decide from the various ways to measure in-
patient falls which one or two indicators will
best serve as the outcome indicators. The team
will also need to identify a list of indicators for
processes related to the falls prevention process
and they should consider selecting one or two
balancing indicators to provide insights on the
issue of suboptimization.
TABLE 4-2 provides examples of concepts
and the specific indicators that could be used
to measure each concept. The decision as to
which indicator is selected (from this list or a
new list of indicators that a team might develop)
depends on the questions that a QI team is
trying to answer, the availability of data, and
ultimately the team’s aim. If you phrase the
question in terms of the absolute volume of an
activity, you might be interested in tracking a
simple count of the number of events (e.g., the
number of inpatient falls). If, on the other hand,
you were interested in a relative measure, then
you would be better off measuring falls as a
percentage or possibly as a rate. When it comes
 Selecting a Specific Indicator 105

TABLE 4-2 Moving from a concept to a specific indicator

Concept Potential Indicators for This Concept

Patient falls ■■ The number of patient falls

■■ The percentage of patient falls
■■ The patient falls rate
■■ The number of days between inpatient falls

Cesarean sections ■■ The number of cesarean sections

■■ The percentage of cesarean sections
■■ The cesarean section rate

Care of surgical ■■ The percentage of post-op deaths (sorted by American Society of

patients Anesthesiologists class)
■■ The number of days between the occurrence of post-op deaths
■■ The percentage of unexpected returns to surgery
■■ The number of successful cases before there was a return to surgery
within 24 hours

Care of coronary ■■ Intubation time post CABG

artery bypass graft ■■ The percentage of prolonged post-op CABG intubations
(CABG) patients ■■ The percentage of CABG patients with a hospital acquired infection
■■ The percentage of CABG patients returning to surgery within 24 hours

Patient scheduling ■■ The average number of days between a call for an appointment and the
actual appointment date
■■ The percentage of appointments made within 3 days of the call for an
appointment
■■ The number of appointments scheduled each day
■■ The number of days between a call for an appointment and the first
available appointment

Employee retention ■■ Total number of full-time equivalents (FTEs)

■■ Percentage of employee turnover
■■ Employee turnover rate
■■ Average number of years employed by the organization
■■ The percentage of new hires who leave during the first year

Employee evaluations ■■ The number of evaluations completed

■■ The percentage of evaluations completed on time
■■ Variance from due date of a completed evaluation

Care of emergency ■■ The number of unplanned returns to the emergency department (ED)
patients within 24 hours
■■ The percentage of ED patients admitted as inpatients
■■ The percentage of ED transfers to other facilities
■■ The patient wait time in the ED

(continues)
106 Chapter 4 Milestones in the Quality Measurement Journey

TABLE 4-2 Moving from a concept to a specific indicator (continued)

Concept Potential Indicators for This Concept

Implementation of a ■■ The number of patients who had restraints applied

restraint protocol ■■ The percentage of patients placed in restraints
■■ The restraint usage rate

Documentation of ■■ Transcription turnaround time

histories and physicals ■■ The time from patient admission to the physician-dictated H&P
(H&Ps) ■■ The percentage of incomplete H&Ps

Medication usage ■■ The total number of medication orders placed each day
■■ The number of medication orders that had one or more errors
■■ The time it takes to deliver a med order to the unit once the order is
received in the pharmacy
■■ The medication error rate
■■ The number of wasted IVs

Customer satisfaction ■■ The number of patient complaints

■■ The percentage of patients providing positive responses to a survey
■■ The percentage of patients who indicated that they would recommend
the facility to a family member or friend
■■ The percentile ranking for employee satisfaction in a national database
■■ The percentage of physicians indicating that your hospital is an
“excellent” facility

Home care visits ■■ The number of home care visits

■■ The average time spent during a home care visit
■■ The percentage of time spent traveling during each home care visit
■■ The number of visits each days for each home care nurse
■■ The number of bottles of home oxygen delivered

Pastoral care ■■ The number of patient encounters by the pastoral care staff
■■ The number of minutes spent during a patient encounter
■■ The percentage of inpatient admissions that have properly documented
the patient’s religious preference
■■ The number of requests from nursing units for assistance

Delivery of oncology ■■ The percentage of outpatient oncology patients who have to be admitted
services ■■ An individual patient’s platelet counts
■■ The total inpatient cost to treat a cancer patient
■■ Mood scale index scores for cancer patients

Successful quality ■■ The number of participants attending a QI class

improvement (QI) ■■ The percentage of cancellations
training ■■ The percentage of no-shows
■■ The information recall scores at 30, 60, and 90 days
 Selecting a Specific Indicator 107

TABLE 4-2 Moving from a concept to a specific indicator (continued)

Concept Potential Indicators for This Concept

Ventilator ■■ The number of patients on a ventilator

management ■■ The percentage of patients placed on a ventilator
■■ The number of days on a ventilator
■■ The ventilator-associated pneumonia rate

Electronic access to ■■ The percentage of med orders submitted via the computerized
information physician order entry (CPOE) system
■■ The minutes of system downtime
■■ The percentage of physicians who regularly use online protocols
■■ The number of visits (hits) to the organization’s website

Outpatient testing ■■ The total number of outpatient visits and therapy

■■ The wait time to have a blood draw (or any other procedure)
■■ The percentage of outpatient procedures with a complication
■■ The complication rate for outpatient procedures
■■ The time it takes to complete a colonoscopy procedure

Lab production ■■ Lab turnaround time

■■ The total number of lab orders
■■ The percentage of inaccurate lab orders
■■ The percentage of stat lab orders exceeding target
■■ The percentage of stat lab orders

to indicator selection, there are more options
than most people realize. It is also important
to realize that there are no universally accepted
“best” indicators of healthcare performance.
A concept may be the same (e.g., inpatient falls)
or even types of measures (e.g., outcome, process,
or balancing) across different systems, regions,
provinces, or even countries but the specific
indicators and the subsequent milestones that
mark the QMJ can be very different.
TABLE 4-3 provides a worksheet to help
you move from concepts to indicators. In the
left column of this worksheet, list the concepts
you are interested in measuring. The next
column should then list the specific quantifi-
able indicator(s) (e.g., count, percentage, rate,
score, index, days between, cases between) you
think will best capture each concept of interest.
Finally, indicate whether each listed measure
is an outcome, process, or balancing measure.
TABLE 4-4 provides an example of a completed
indicator worksheet. A key point related to
this worksheet is that you do not need to have
a lengthy summary of each of your indicators.
You can take this completed worksheet to a
management meeting and say, “Here are the
indicators for our improvement team.” It is
clear and yet specifies the key components of
how you have moved from a concept to specific
indicators plus identifying the types of indicators
you will be tracking.
Summary conclusions about moving from a
concept to a quantifiable indicator are provided
in BOX 4-1.
108 Chapter 4 Milestones in the Quality Measurement Journey

TABLE 4-3 Organizing your indicators worksheet

Topic for Improvement: __________________________________________________________________

Concept Potential Indicators Outcome Process Balancing

TABLE 4-4 Example of a completed organizing your indicators worksheet

Topic for Improvement: Inpatient Falls Process

Concept Potential Indicators Outcome Process Balancing

Patient harm Inpatient falls rate ✓

Patient harm Number of falls ✓

Compliance Percentage of inpatients ✓

assessed for falls

Staff education Percentage of staff fully trained ✓

in falls assessment protocol

Assessment The additional time it takes ✓

time to conduct a proper falls
assessment

BOX 4-1 Conclusions about moving
from a concept to an indicator
1. Moving from a concept to an indicator
requires focused work to create
agreement about adjectives, such as
recovery, major, timely, complete,
accurate, or excellent.
2. A concept may need more than
one indicator and, therefore, the
development of more than one
operational definition.
3. The transition from concept to an
indicator doesn’t just happen; it requires
both technical and clinical decision
making to be blended with pragmatism
and acceptance of the imperfections of
the measures.
4. There is no such thing as a fact!
(W. E. Deming)
▸▸ Developing
Operational
Definitions
The real work of indicator development begins
after you have selected and named a specific
indicator. Now it is time to develop an oper-
ational definition. I find the specification of
operational definitions to be one of the more
interesting and intriguing aspects of indicator
development. Every day we are challenged to
think about operational definitions. They are
not only essential to good measurement but
also critical to successful communication be-
tween individuals. For example, if you tell your
teenage son or daughter to be “home early”
from a party, you will quickly understand the
necessity of establishing a clear operational
definition.
An operational definition is a description, in
quantifiable terms, of what to measure and the
Developing Operational Definitions 109
specific steps needed to measure it consistently.
A good operational definition:
■■ Gives communicable meaning to a concept
or idea
■■ Is clear and unambiguous
■■ Specifies the measurement method, proce-
dures, and equipment (when appropriate)
■■ Provides decision-making criteria when
necessary
■■ Enables consistency in data collection
Some groups are better at developing op-
erational definitions than others. For example,
political leaders typically shy away from clear
and unambiguous operational definitions so they
can change their positions or follow a different
approach. But in this age of instant information,
social media, and the ability to record statements
easily and quickly politicians are starting to be
more concerned about the terms and definitions
they use. For example, consider the following
list of terms that are used frequently during
political campaigns:
■■ A “fair tax”
■■
■■
A “tax loophole”
We need to “jump start” the economy
■■ The “rich” need to give more to the “poor”
The “middle class” needs tax relief
■■
■■ We need to get this country “moving” again
■■ The “small farmer” needs economic support
All of these terms require clear operational
definitions if there is to be a consistent under-
standing of what they mean and how we would
measure them. In the political arena, however,
the desire is to frequently have a certain amount
of ambiguity surrounding concepts and terms
so that the person presenting the idea cannot be
held to a single position or definition.6
On a more personal note, I had a great
example of an operational definition when my
daughter Devon was 9 years old. Devon and
her friend Janine called up to me and asked,
“When are you going to take us for the ice cream
you promised?” My answer would have made
any politician proud. I responded confidently,
“Soon.” That appeased them for about 15 m ­ inutes.
110 Chapter 4 Milestones in the Quality Measurement Journey
Then they called up again, and this time when
I answered, “Soon,” they demanded to know
how many minutes made up “soon.” Even a
9-year-old child understands the need for a clear
operational definition.
One of the more interesting problems with
an operational definition involved the September
23, 1999 incident with the Mars space probe.
European scientists used metric measurements
and calibrations (newton-seconds) to guide the
spacecraft. The probe was built by Lockheed
Martin and their engineers used decimal ref-
erents (pounds/foot-seconds) to calibrate the
maneuvering of the probe. When the probe went
around the far side of Mars and was ordered to
go down toward the surface for a closer look and
flyby, it was essentially receiving two different
sets of operational definitions. It followed the
programming commands but because of the
differences in the operational definitions used
by the builders of the spacecraft and those
maneuvering it the probe took a trajectory that
took it entirely too close to the planet, causing
it to burn up in the Martian atmosphere. The
difference between metric and decimal units of
measurement created an inconsistent operational
definition of the term “unit of distance.” As a
result, a $125 million project became a NASA
embarrassment.
A more recent example of confusing oper-
ational definitions can be found in the ongoing
debate over what is a “healthy or natural” food.
The U.S. Food and Drug Administration (FDA)
has been debating these terms for years. Food
scientists maintain that a majority of our food
products are not natural and therefore not healthy
because they have been processed in one form or
another and that they are no longer a “product of
the earth.” A similar debate has been occurring
for over 15 years on the definition of an “organic
fish.” I thought all fish were organic because I
have never been served a mechanical fish but
I must be missing something in this debate. It turns
out that the debate hinges on what the fish in a
fish farm (not wild fish but farm-raised fish) are
fed. If they are fed pellets that are made of other
fish they are considered “organic.” But, if they
are fed pellets that consist of vitamin-enriched
corn, wheat, and other non-fish protein then
they are defined as “nonorganic.” Seriously, you
cannot make up stuff that is better than what
you discover in real life.
One final personal story about operational
definitions before returning to healthcare ex-
amples. This is absolutely one of my all-time
favorites. My wife Gwenn was a nationally ranked
official for women’s field hockey in the United
States. She was doing one of the final games for
the National Collegiate Athletic Association
(NCAA) championship in Iowa City, Iowa in
the middle of November. It was very cold. So
cold in fact, that they had to place the hockey
balls in a small warming device so they would
not crack or break when hit. The game was tied
when one of the forward players hit a wicked
shot that was headed toward the opponent’s goal
cage. The telltale sound of a hockey ball hitting
the metal backplate of the cage was heard by
all. The attacking team believed they had just
scored a winning goal. But the defending team
quickly pointed out that only half the ball was in
the cage. The other half of the ball was still out
on the playing pitch. So now what? Is it a goal
or not a goal? What is the operational definition
of a goal? Time out was called. The officials met
at the center of the field along with the timer,
the backup umpires, and an NCAA judge. This
is a very important game so a decision has to be
made. Goal or no goal? Gwenn, who is the lead
official, is asked to render an opinion. She honestly
says, “I have no idea. This has never happened
before.” So, they turn to the official rule book
to see if there is an operational definition of “a
goal.” After a couple of minutes that seemed to
last hours, Gwenn announces that there is an
answer in the rule book. It clearly states that
a goal occurs “when the entire ball passes the
plane of the goal line.” With half the ball still
lying on the pitch and the other half in the cage
the answer is easy. . .no goal. On the ride home
I was mulling over the operational definition
of a goal and asked Gwenn a question “What
if upon being hit the ball did break in half but
both halves went across the goal line and into

the cage? Would this now be considered a goal?
The entire ball was in fact in the cage.” There was
silence for a few seconds then she said, “I’m not
even going to address that question.” But I still
wonder whether it would be a goal.
Every day healthcare professionals must
deal with operational definitions. There are many
healthcare terms that beg for more precise oper-
ational definitions. How does your organization
define the following terms?
■■ A patient fall
■■ A restraint
■■ A good outcome for the patient
■■ A medication error
■■ A complete and thorough physical exam
■■ A good employee performance review
■■ Surgical start time
■■ An accurate patient bill
■■ A readmission
■■ A successful surgical outcome
■■ An organization that supports its workers
■■ A late food tray
■■ A clean patient room
■■ Healthcare disparities
■■ A quick admission
■■ A blameless culture for reporting errors
Consider one of these terms that has
intrigued me for years—a patient fall. One of
the first definitions I heard for a patient fall
was “a sudden and rapid movement from one
plane to another.” This sounds like something
you try to do at a busy airport rather than the
definition of a negative patient outcome. It is not
very precise and leaves a lot to the imagination.
I have frequently heard nurses talk about two
basic types of falls: partial falls and assisted
falls. Partial falls usually occur when the patient
attempts to get out of bed and discovers that
he or she does not have an adequate amount
of strength to permit ambulation. In this case,
the patient might stagger a little, slump back
onto the bed, try to stand again, and attempt
to make it to the chair by the window but ends
up collapsing to the floor. As I have explored
this scenario with nurses and asked them if this
constituted a partial fall, I get mixed responses.
Developing Operational Definitions 111
One meeting I especially remember produced
two very different views of a partial fall. After
describing the conditions of a partial fall, half
of the nurses indicated that they would classify
the situation as a partial fall because the patient
did bounce around a little before ending up on
the floor. Their reasoning was that the patient
bounced around a little, came in contact with
some furniture, and eventually ended up on the
floor. The other nurses in the group reserved
their opinion until they found out the answer
to one question: “Did the patient’s knee hit the
floor first?” If the answer was “Yes,” then they
agreed it would not be a partial fall. If the answer
to this question was “No,” however, this group
of nurses believed that this was a partial fall.
The knee touching the floor was the primary
determinant of a “partial fall.”
Assisted falls are even more interesting
than partial falls. When I first heard this term
I envisioned nurses getting so fed up with a
patient that they gave him a gentle nudge and
“assisted” him in falling. As I learned more about
this topic, however, I came to realize that an
assisted fall fortunately has nothing to do with
the nurses causing the fall. It does, however, have
a very distinctive operational definition. Here is
the scenario. A patient decides to go for a walk
tethered to his IV pole. The patient takes a few
steps then announces to the nurse that she does
not feel very well and that things are starting to
move in circles. As the patient begins to sway
the nurse moves into position, grabs the patient,
and assists her to the floor. But is this really a
fall? It seems to me it is more like a recline or
possibly a lay-down. Most nurses I have worked
with agree that being present when a patient is
starting to go down and intervening to help break
the patient’s fall constitutes an “assisted fall.” But
there is certainly not universal agreement on the
precise operational definition of an assisted fall.
You can see the problem that all this poses
for measurement. If you are part of a multihos-
pital system, a region, province, or a country or
plan on comparing hospital outcomes across
providers, then you should make sure that
each provider being compared is defining the
112 Chapter 4 Milestones in the Quality Measurement Journey
indicator of interest in the same way. Without
such consistency you will end up with apples
and oranges at best and more likely apples and
carburetors. The pieces will not be comparable,
which means that ultimately the conclusions
that are derived from the data are not accurate.
All good measurement begins and ends with
operational definitions.
An example of an operational definition for
the percentage of medication errors is summa-
rized as follows.
Indicator Name: Percentage of medication
errors
Numerator: Number of outpatient medi-
cation orders with one or more errors. An
error is defined as wrong med, wrong dose,
wrong route, or wrong patient
Denominator: Number of outpatient
medication orders received by the family
practice clinic pharmacy
Data Collection:
• This indicator applies to all patients
seen at the clinic
• The data will be stratified by type of
order (new versus refill) and patient age
• The data will be tracked daily and grouped
by week
• 100, with only 1 decimal place)
The data will be pulled from the phar-
macy computer and the computerized
physician order entry (CPOE) systems
• Initially, all medication orders will be
reviewed. A stratified proportional
random sample will be considered
once the variation in the process is
fully understood and the volume of
orders is analyzed.
A second example of an operational definition
provides the details for a perioperative nasal
swabbing indicator.
Indicator: Percentage of patients under-
going hip and knee replacement surgery
during the measurement period who have
had preoperative nasal swabs to screen for
Staphylococcus aureus carriage
Goal: 95%
Frequency of Data Collection: Monthly
Numerator Definition: Number of patients
undergoing hip or knee replacement surgery
who have had a nasal swab specimen pro-
cessed to screen for Staphylococcus aureus
carriage prior to surgery
Denominator Definition: Number of
patients undergoing elective hip or knee
replacement surgery
Numerator and Denominator Exclusions:
• Patients who are less than 18 years of age
• Patients who had a principal or admission
diagnosis suggestive of preoperative
infectious diseases
• Patients with physician-documented
infection prior to surgical procedures
• Patients undergoing nonelective hip
or knee replacement surgery
Definition of Terms: Hip or knee replace-
ment surgery includes operations involving
placement of a nonhuman-derived device
into the hip or knee joint space. ICD-9
Codes include 00.70-00.73, 00.85-00.87,
81.51-81.53, 00.80-00.84, 81.54, and 81.55.
Calculate as: (numerator/denominator *
Summary conclusions about developing
operational definitions are provided in BOX 4-2.
BOX 4-2 Conclusions about developing
operational definitions
1. Operational definitions are not universal
truths!
2. Operational definitions require
agreements on terms, measurement
methods, and decision criteria.
3. Operational definitions need to be
reviewed periodically to make sure
everyone is still using the same definitions
and that the conditions surrounding each
measure have not changed.

▸▸ Developing Data
Collection Plans
I have separated this milestone from the actual
collection of data because I do not believe that
as an industry we have devoted enough time to
thinking about the numerous factors that influ-
ence the success or failure of our data collection
efforts. Most people want to move directly from
“I have an indicator” to “Let’s go get some data”
without spending much time thinking about how
to actually collect the data. From my perspective,
planning for data collection should occupy about
80% of your data collection time and the actual
act of collecting the numbers should consume
about 20% of your time.
Data collection is not unlike other aspects
of life that require planning. Whether the ac-
tivity is painting a house, planting a garden, or
going on a major vacation, preparation is key.
If you do not spend enough time preparing a
wooden house, the paint will not last as long
as you would like it to. Similarly, if you do not
take time to properly prepare the soil in your
garden, the seeds and young plants will not get
off to a very good start. Finally, a major vaca-
tion (e.g., a cruise or a bed-and-breakfast tour
of Ireland) usually requires more time to plan
than the time you actually spend on the holiday
itself. The act of data collection is very similar.
Inadequately prepared data collection plans will
usually produce unacceptable results. The data
will be challenged, questioned, and/or seen as
being rather useless.
There are several important data collection
issues that require some elaboration, most no-
tably stratification and sampling. Stratification
is one of the best things a team can discuss
when building indicators, yet it is frequently
overlooked. Stratification is more of a logical
issue than a statistical one. It essentially consists
of the separation and classification of data into
categories or homogeneous buckets that reflect
common characteristics. The objective of stratifi-
cation is to create strata or categories within the
data that are mutually exclusive and allow you
Developing Data Collection Plans 113
to discover patterns that would not otherwise
be observed if the data were all aggregated. The
overall strategy is to minimize the variation within
a stratification category in order to compare the
variation between categories. By doing this you
can increase your knowledge about the possible
influence that the stratification levels might
have on the outcome indicator. Frequently used
stratification levels include:
■■ Age
■■ Gender
■■ Socioeconomic status
■■ Prior admission for the same diagnosis
■■ Day of the week
■■ Time of day
■■ Month of the year
■■ Shift (day, afternoon, night)
■■ Type of order (stat versus routine)
■■ Type of ambulatory procedure
■■ Type of surgery
■■ Machine (such as ventilators or lab equipment)
■■ Severity of the patients
■■ Tenure of the staff
If you do not think about the factors that
might influence the outcome of your data before
you collect the data, you run the risk of having to
try to tease out the stratification effect manually
after the data have been collected. At this point
not only is it too late to effectively address the
stratification question, but you will also have
to engage in rework and wasted time to even
attempt to untangle the stratification questions.
FIGURES 4-4 and 4-5 provide examples of
stratification problems. In the first example
­(Figure 4-4), the indicator of interest is turn-
around time (TAT) in the lab (the particular test
does not matter at this point). The data reveal
that the process displays extremes because
the team did not separate the TATs for the
day and evening shifts. They merely collected
data and combined the two shifts, which are
obviously different. In this case, the average
TAT will fall exactly in the middle of the two
extremes of data. The average and even the
standard deviation of the TAT are meaning-
less statistics for data like this. The mean and
114 Chapter 4 Milestones in the Quality Measurement Journey
Turnaround time (TAT)
Day shift TAT
Average TAT
P.M. shift TAT
Day of the week
FIGURE 4-4 A stratification problem with
turnaround time
Weekday revenue
Revenue
Weekend revenue
Day of the week
FIGURE 4-5 A stratification problem with tracking
revenue
standard deviation can be calculated of course
but they are mathematical artifacts that are the
result of two distributions of data—one high
and one low. These data should be separated
and two charts should be made—one chart for
the day shift TAT and another for the evening
shift. Stratification is an essential aspect of
data collection. If you do not spend some time
discussing the implications of stratification, you
will end up thinking that your data are worse
(or better) than they should be.
In Figure 4-5, the indicator of interest is
revenue by day of the week. There are several
data points in a row that are at relatively high
and roughly at the same level. Then there is
a sharp drop in the data for two data points.
This pattern demonstrates a clear problem with
stratification. In this case, the revenue generated
Monday through Friday (the higher data points)
is markedly different from that generated on
Saturday and Sunday (the lower data points).
This hospital is clearly not a 7-day-a-week
hospital. If you were to calculate the average
revenue generated per day for this hospital
you would get a misleading number. Although
the overall mean would be skewed toward the
weekday revenue side of the chart (because of
the higher volume generated during this time
as well as more days), it would not reflect the
average generated during the weekdays. For this
example, someone should have said before the
data were collected, “Because we do not generate
the same amount of revenue on the weekend
as we do on weekdays, we should stratify the
data into two categories—weekday revenue
and weekend revenue—and analyze the data
separately.”
Sampling is the second key component of
a data collection plan. Not every data effort will
require sampling. If a process does not generate a
lot of data, then you will probably analyze all the
occurrences. This happens most often when the
indicator is a percentage. For example, when
we compute the percentage of primary C-sections
for the month we typically do not use a sampling
plan. We usually take all the C-sections for the
month and divide this numerator by the total
number of deliveries (the denominator) for the
month. When a process generates considerable
data, however (e.g., lab TAT for blood tests or
all admissions during the month), a sampling
plan is usually appropriate. From my perspec-
tive, building knowledge of sampling methods
is one of the most important things you can do
to establish efficient and effective data collection
strategies.
Like stratification, sampling deals more
with logic than statistics. Individuals trained
in the social sciences are typically exposed to
extensive training in sampling principles and
concepts. Unfortunately, most healthcare pro-
fessionals are given only a cursory foundation
in this subject. The irony with this situation is
that sampling is actually quite easy. Healthcare

professionals would grasp sampling principles
quickly if they were exposed to them throughout
their formal training.
Try this simple test to demonstrate this point.
The next time you are with a group of healthcare
professionals, ask them, “Have any of you ever
drawn a random sample?”7 Rather quickly you
will receive a bunch of positive nods. When you
ask one of the people who was nodding rather
energetically how they actually drew the random
sample, they will usually announce rather proudly
that they “picked every 10th chart.” Selecting
every 10th chart is a form of random sampling
known as systematic sampling (described later),
but it can introduce considerable bias if the
steps involved in drawing a systematic sample
are not followed.
The purpose of sampling is to be able to
draw a limited number of observations and to
be reasonably comfortable that they represent
the larger population from which they were
drawn. If you had all the time and money in
the world you would never draw a sample. You
would always do a complete enumeration of
all cases. But time and resources are limited,
so we draw samples.8 Whenever you draw a
sample, however, the key question is, “How
much data do I need?” One of my professors
in graduate school, Dr. Bob Bealer, had a great
answer to this question. When asked by one of
my fellow doctoral students how much data we
should collect for our dissertation research, he
merely answered, “As much as you must and
as little as you dare.” At the time I thought this
was a clever and rather professorial response.
But after spending many years trying to help
healthcare professionals develop reasonable
sampling strategies, I have come to realize that
this was very practical advice. For example, if I
wanted to check your weight by weighing you
on only one day of the year, would you say this
is a representative sample of your true weight?
As an aside, I should tell you that the day I have
selected to weigh you is Thanksgiving Day (A
U.S. holiday celebrated on the third Thursday
of November) after you eat. Most people would
Developing Data Collection Plans 115
say, “No way do I want you to use my Thanks-
giving Day weight.” Their initial reaction would
probably be correct. On the average, for exam-
ple, adults consume upwards of 5,000 calories
on Thanksgiving Day. So most people would
probably say, “If you are going to weigh me once,
check me in the spring when I am trying to get
back into my shorts or bathing suit.” To be even
more reasonable (reliable), I might weigh you
every couple of weeks as they do in many weight
control programs. In this way, I would obtain a
more representative sample of your weight as
it fluctuates over time. Remember, as much as
you must and as little as you dare.9
What happens if you draw a sample and it is
not representative of the population from which
it was drawn? FIGURE 4-6 shows the relationship
between three samples and a population. The
larger curve represents the total population of
interest (e.g., all asthma patients returning to the
emergency department [ED] within 24 hours).
Curve A identifies a properly pulled sample of
patients. The shape and location of this sample
are very similar to the population. Curve C,
on the other hand, represents a sample that
was drawn with a negative bias. In this case,
you could get the false impression that your
results were much worse than they really were
just because you pulled a sample that came
from the negative end of the population curve.
Similarly, Curve B depicts a positive sampling
bias, which leads you to an overly optimistic
conclusion. A well-designed sampling plan will
not only produce data that are representative of
the population but also save time and money for
those collecting the data.
There are many ways to draw a sample.
The key question you have to ask yourself
whenever you want to draw a sample is, “How
representative and precise do I need to be with
this sample?” For example, if you have received
numerous calls and complaint letters about the
wait time in outpatient testing and therapy, you
basically have two sampling options: (1) develop
a statistically based sample that allows you to
generalize to your total outpatient population,
116 Chapter 4 Milestones in the Quality Measurement Journey

Population

A representative
sample
A negatively A positively
biased sample biased sample

C A B

Negative outcome Positive outcome

Ideally the sample will have the same shape and location as the total population but have fewer
observations (curve A). A sample improperly pulled could result in a positive sampling bias
(curve B) or a negative sampling bias (curve C).

FIGURE 4-6 The relationship between a sample and the population

or (2) go out on any given day, grab a convenient
handful of willing patients, and ask them how
they like your outpatient testing and therapy
services. If the level of precision you need to
answer this question is low, then option 2 is
appropriate. If, on the other hand, you need
to be very sure (statistically sure) that there is
a problem in outpatient testing and therapy,
then you need to formulate a more scientific
approach to sampling.
Ishikawa, in his classic work Guide to Quality
Control (1982), identifies four conditions for
developing a sampling plan:
■■ Accuracy
■■ Reliability
■■ Speed
■■ Economy
These four criteria should serve as a
fundamental checklist for building sampling
designs. Not every sample will maximize all four
criteria. There are times when accuracy will be
the primary objective of sampling (e.g., when
designing a randomized clinical trial). At other
times reliability will become more important
(e.g., when you are establishing a sampling
plan for patient satisfaction and you want to
be able to draw reliable samples each month
or quarter). Speed may be essential when you
have to sample a number of blood specimens
to determine whether there is a contamination
problem. Finally, the economics of sampling
will usually pose a challenge for everyone.
Each time you draw a sample, whether it is
a sample of medical records or a sample of
patients, there are economic factors involved
with the pull of data. Complicated sampling
plans require more time, effort, and money. In
the end, however, it all comes down to a fairly
simple question—how can you pull an accurate,
reliable, fast, and inexpensive sample? Obviously
it is difficult to obtain a sample that meets all
four criteria simultaneously. Sampling, there-
fore, really consists of a series of compromises.
It basically gets us back to Professor Bealer’s
words of wisdom, “As much as you must and
as little as you dare.”
Sampling methods are basically divided
into two major categories—probability and non-
probability. Any standard research methods or

statistics book will provide a review of sampling
methods. I would encourage you to obtain several
books on this topic to see how different writers
classify and describe the various methods. Do
not worry about the age of the book. Most of
my books on sampling, for example, are 20 to
30 years old. Even when I pick up a new book on
sampling, the terms remain virtually the same
as those I find in my older books.
The terms and approaches to sampling have
remained rather constant since the late 1930s.10
I do not intend to replicate in this text what
can be found in many good references (Babbie,
1979; Campbell, 1974; Daniel and Terrell, 1989;
Duncan, 1986; Gonick and Smith, 1993; Hess,
Riedel, & Fitzpatrick, 1975; Ishikawa, 1982;
Miller, 1964; Selltiz, Jahoda, Deutsch, & Cook,
1959; Weiss, 1968; Western Electric Co, 1985).
I provide, however, a brief review of the major
sampling methods and let the reader explore
the details.
▸▸ Probability Sampling
Probability sampling is designed to provide
the highest possible level of predictability and
confidence in the sampled data at the most eco-
nomical cost to the researcher. Although most
people have some notion of what a probability
or random sampling entails, many are unclear
on the specific aspects of actually designing
and selecting the sample. At the very founda-
tion of probability sampling is trust: trust in
statistical probability and the fact that when
you draw a random sample you do not throw
it away merely because it does not conform
to your personal belief about what the data
are supposed to tell you. I have drawn many
random samples for people over the years. On
numerous occasions I have been questioned
about the “accuracy” of the samples because
the individuals who requested the samples did
not like the results. In their minds, they thought
they should be allowed to pick and choose
what should be included (and excluded) in
the sample. If they were to do this it would
Probability Sampling 117
not make the sample a probability sample.
Whenever judgment, purposeful intent, or
convenience enter into the sampling plan,
you have moved from probability sampling to
nonprobability sampling, which is addressed
in the next section.
Campbell (1974, p. 143) identifies three
characteristics of probability sampling:
1. A specific statistical design is followed.
2. The selection of items from the popu-
lation is determined solely according
to known probabilities by means of
a random mechanism, usually using
a table of random digits.
3. The sampling error—that is, the differ-
ence between results obtained from a
sample survey and that which would
have been obtained from a census
of the entire population conducted
using the same procedures as in the
sample—can be estimated and, as a
result, the precision of the sample
result can be evaluated.
There are numerous ways to draw a proba-
bility sample. They are all essentially variations
on the simple random sample.
Simple Random Sampling
A random sample is one that is drawn in such
a way that it gives every element in the pop-
ulation an equal and independent chance of
being included in the sample. This is usually
accomplished by using a random number table
(usually found in the back of any good statistics
book) or a computer-based random number
generator (found in all statistical software
programs and in many spreadsheet packages).
Step-by-step procedures for drawing a random
sample can be found in Probability Sampling of
Hospitals and Patients (Hess et al., 1975) and
in Flaws and Fallacies in Statistical Thinking
(Campbell, 1974). Even though this method is
referred to as a “simple” random sample, the
term “simple” can be a little misleading. The
mechanics of drawing a random sample may
118 Chapter 4 Milestones in the Quality Measurement Journey
not feel simple to those who have to number
all the elements in the population and learn
how to apply a random number table or a
computerized random number generator.
As an alternative, you can simply write the
names or numbers of the population elements
on separate pieces of paper, place them in a bowl,
and draw out the sample. I did this to develop a
sampling plan for a medical group. They wanted
to sample the wait times of patients in one of
their clinics, but they did not have the resources
to sample every day of the week. They initially
said they would pull a sample of patients every
Monday. I advised them that this could produce
biased results, because Mondays are typically busier
than other days of the week. So I wrote the days of
the week (excluding the weekend) on five pieces
of paper (all of the same size), placed them in a
bowl, and then drew out a day of the week. The
first day I pulled was a Wednesday. This meant
that during the first week, a sample of patients
would be pulled on Wednesday and their wait
times to see the physician would be recorded. I
placed the slip of paper with Wednesday written
on it back into the bowl and drew another piece
of paper. The second piece of paper had Friday
written on it, which would be the sampling day
during week 2. I replaced the piece of paper and
repeated this process 23 more times to obtain a
total of 25 sample days. To pull the sample of
patients on a given day, I first asked the staff
to run a report showing the actual volume of
patient visits by day for the last 6 months. From
this report we determined the minimum and
maximum number of visits as well as the mean,
median, and standard deviation (to see if the data
approximate a normal distribution). We determined
that the average number of visits each day was 74
with a minimum of 63 and a maximum of 86. I
advised them to place the numbers 1 through 86
on pieces of paper (again all the same size) and
place them all in the bowl. We then proceeded to
draw a random sample of eight patients from the
bowl for each sampling day. The first day to be
sampled, for example, was a Wednesday. On this
day the following eight patients would have their
wait times tracked: 43rd, 15th, 63rd, 2nd, 47th,
23rd, 18th, and 4th. Because the clinic basically
knew how many patients they had scheduled for
that day, they could identify those patients from
the charts that had been pulled ahead of time.
This allowed the staff to be prepared to track the
various steps in the process for these patients
(i.e., the time from check-in to being called by
the nurse, time with the nurse, wait time to see
the doctor, time with the doctor, and finally
checkout time). In this example, two random
samples were selected, one for the day of the
week and another for the patients to be tracked
within a selected day. By using the pieces of paper
and a bowl, we were able to apply the principles
and precision of probability sampling and avoid
some of the complexity associated with using
random number tables or computer-generated
random samples.
Stratified Random Sampling
This method of sampling is not an alternative to
simple random sampling but rather a variation
on a theme. Simple random sampling assumes
that the composition of the total population is
unknown. A random selection process is seen,
therefore, as the best way to obtain a “representa-
tive” sample. The problem is that the very nature
of a random selection process could produce
a sample that is not truly representative of the
characteristics of the total population. This is
where stratification comes into the picture. By
stratifying the population into relatively homo-
geneous strata or categories before the sample
is selected, you increase the representativeness
of the sample and decrease the sampling error.
Once the stratification levels have been identified,
a random selection process is applied within
each stratum. For example, you might stratify
a hospital’s patients into medical and surgical
strata and then sample randomly within each
group. This would help to ensure that one group
was not over- or underrepresented in the sam-
ple. A key point to remember when setting up
a stratified random sample, however, is that it
requires the knowledge of people who actually
work in the process. As subject matter experts,

they can tell you what key stratification categories
are relevant. An external statistician who is very
skilled in sampling methods, for example, will not
have knowledge of the local characteristics that
affect the decisions about proper stratification.
Bring the subject matter experts together with
a skilled statistician and you will be able to set
up a good sampling strategy.
Stratified Proportional
Random Sampling
In this case, we are going to use the approach
outlined for stratified random sampling, but we
are going to add another twist. We are going to
determine the proportion that each stratum
represents in the population and then replicate
this proportion in the sample. For example, if we
knew that medical services represented 50% of the
hospital’s business, surgical services represented
30%, and emergency services represented 20%,
then we would draw 50% of the sample from
medical units, 30% from surgery, and 20% from
the ED. This would produce a sample that not
only was representative but also proportionally
representative. This would further increase the
precision of the sample and reduce the sampling
error. The stratified proportional random sample
is one of the more sophisticated sampling designs.
It does require knowledge of the population being
sampled, however, as well as having a sufficiently
large enough population that as you stratify the
population into a variety of categories you will
have sufficient numbers in each category to be
eligible for sampling. Note that a stratified pro-
portional random sample can be more costly in
terms of both money and time.
Systematic Sampling
Systematic sampling offers one of the easiest
ways to draw a sample. It consists of numbering
or ordering each element in the population and
then selecting every kth observation after you
have selected a random place to start, which
should be equal to or less than k but greater than
Probability Sampling 119
zero. For example, if you had a list of 500 medical
records and you wanted to pull a sample of 50,
you would pull every 10th record. To determine
the starting place for the sample, you would pick a
random number between 1 and 10. For argument’s
sake, imagine that when we do this we select the
number 6. So to start our systematic sample we
would go to the 6th medical record on our list,
pick it, and then proceed to select every 10th
record after this starting point. Technically, this is
known as a systematic sample with a random start
(Babbie, 1979, p. 178). The most frequent ways to
organize the elements are either alphabetically or
chronologically. There are two major advantages
of systematic sampling: it is simple and you have
to generate only the first random number. This
sampling method is what many healthcare profes-
sionals think of as a random sample. Although it
is a form of random sampling, it does have certain
limitations. The major problem with systematic
sampling is that you are eliminating chunks of
data that could provide knowledge about the
process. If, for example, you are selecting every
10th record, you have automatically eliminated
from further consideration records 1 through 9.
You pick the 10th record, then skip 11 through 19
and pick number 20. The records in between the
ones you select will never have a chance of being
included in your analysis. If there is something
that occurs regularly in the data or something that
causes your data to be organized into bunches of,
say, seven or eight, then these records would be
automatically eliminated from consideration. The
other problem I have observed with this form of
sampling in healthcare settings is that the people
drawing the sample do not base the start on a
random process. They merely pick a convenient
place to start and then start applying the sampling
interval they have selected. This introduces bias
and greatly increases the sampling error.
Cluster Sampling
In cluster sampling, the population is divided
into mutually exclusive and exhaustive clusters,
then a simple random sample is drawn within
each cluster. On the surface this approach does
120 Chapter 4 Milestones in the Quality Measurement Journey
not seem very different from stratification.
Cluster sampling differs from stratified random
sampling in that cluster sampling seeks to create
“bunches” within the population. Sampling in this
way is almost always less expensive than simple
random sampling (which is not as focused). The
other key distinctions between stratification and
cluster sampling include the following: (1) with
stratified sampling, a sample of elements is se-
lected from within each stratum or category; and
(2) with cluster sampling, a sample of stratum is
selected. Because the cluster sample is selecting
a sample of stratum or categories, it is desirable
to have each cluster be a small collection of the
population. Cluster samples, therefore, should
establish groupings that are as heterogeneous as
possible. Stratified samples, on the other hand,
attempt to create homogeneous categories (e.g.,
all medical and all surgical patients).
Another distinction with cluster sampling
is that it is typically done with fairly large
populations. This method could be applied,
for example, to a large system that has 15 to
20 hospitals. Each hospital could be considered
a cluster, or they could be grouped into regional
clusters. A cluster sample also could be drawn
in a large metropolitan area. Instead of looking
at individual hospitals or hospital systems,
you could divide the metropolitan area into
neighborhoods or regions (the clusters) and
then sample patients within these regions. In
Chicago where I live, for example, it would be
possible to divide the metropolitan area into
north, south, west, and urban core clusters
(east would not work because Lake Michigan
is located to the east of Chicago). If we did
this we would not be so concerned with the
individual hospitals and their organizational
affiliations but rather with bundling people
together into common geographic areas. With
large populations, therefore, cluster sampling
can be a very economical approach to sampling.
Cluster sampling would not apply to the unit
or department level because the population
of interest (e.g., all hip or knee replacement
patients) is not large enough to permit clusters
to be created.
▸▸ Nonprobability
Sampling
Nonprobability sampling is typically used when
the researcher is not worried about estimating
the reliability and precision of the sample or of
generalizing the results to a larger population.
This is not to say, however, that nonprobability
samples do not serve a useful purpose. More
specifically, nonprobability approaches to sam-
pling can be used when:
■■ Probability samples are either too expensive
to collect or too complicated for the question
being asked
■■ There is no need to draw inferences or
generalize to larger populations
■■ There is no need to estimate the probability
that each element has of being included in the
sample
■■ There is no need to have assurance that every
element (e.g., patient) had an equal oppor-
tunity to be included in the sample
■■ The objective is to conduct an exploratory
or descriptive study on an issue or process
that has not been studied in detail
■■ You are testing a potential improvement
strategy and want to run a quick pilot study
(i.e., sending up a trial balloon to see if it
has any hope of succeeding)
■■ Mechanical selection of the sample is not
required; personal judgment and subjective
choice are sufficient
The major forms of nonprobability sampling
are convenience sampling, quota sampling, and
judgment sampling. The basic objective with
all of these methods is to select a sample that
the researchers believe is “typical” of the larger
population. The problem is that there is no way
to actually measure how typical or representative
a nonprobability sample is with respect to the
population it supposedly is representing. In short,
nonprobability samples can be considered “good
enough samples” (i.e., they are good enough for
the people pulling the sample).

Convenience Sampling
As the name implies, convenience sampling is
designed to obtain a handful of observations
that are readily available and convenient to
gather. Convenience sampling is also referred
to as “chunk” sampling (Hess et al., 1975, p. 8)
or accidental sampling (Maddox, 1981, p. 3;
Selltiz et al., 1959, p. 516). A classic example of
convenience sampling is found in the “man on
the street” interview conducted by TV stations.
The local TV channel parks its action-cam van
along a busy downtown street at lunchtime.
The investigative reporter positions herself
strategically and begins to scan the people who
walk by. She knows that she needs to get at
least four good comments from local citizens
(her quota sample), so she eliminates anyone
from consideration who looks like they would
be (1) uncooperative, (2) argumentative, or
(3) too chatty without any substantive sound
bites. Then she sees a likely candidate and strikes:
“Hi, I’m from Channel 5 News and I’d like to
know how you feel about . . . (fill in the blank).”
Okay, one down and three more to go (to meet
the quota). So the search continues. There is no
science behind this type of sampling. It produces
a biased sample that is essentially a collection of
anecdotes that cannot be generalized to larger
populations. In technical terms, this is what is
referred to as a convenient quota sample (i.e.,
I need a quota of four people and I’m willing
to take anyone who is convenient and agrees
to talk). In the healthcare setting, convenience
sampling is used frequently, possibly too often. I
have seen it used to pull a convenient sample of
medical records, obtain patient satisfaction input
(go grab a few people waiting in the ED and ask
them how they feel about their wait time), or
select a “typical” day to study call button response
time. The primary question that someone should
ask when a convenience sample is drawn is,
“How important is it to know whether the sample
of elements we just selected are representative
of the larger population?” If the consequences of
being wrong do not matter, then the convenience
sample might be good enough.
Nonprobability Sampling 121
Quota Sampling
Quota sampling was developed in the late 1930s
and used extensively by the Gallup organization
to gain great recognition as well as ridicule. (See
Chapter 3 for additional details on how Gallup
benefited from quota sampling in 1936 and then
was criticized in 1948 for its failure to predict
accurately.) If you ask healthcare professionals to
describe quota sampling, they will probably tell
you that it is merely a simple way to determine
the total minimum number of elements needed
in a sample (e.g., we need a quota of 5% of the
medical records) or the total minimum amount
of data that the team can afford to gather. These
two factors, although part of quota sampling, are
only part of the picture. Babbie (1979, p. 196)
nicely describes the steps involved in developing
a quota sample:
1. Develop a matrix describing the
characteristics of the target popu-
lation. This may entail knowing the
proportion of males and females;
various age, racial, and ethnic pro-
portions; as well as the educational
and income levels of the population.
2. Once the matrix has been created
and a relative proportion assigned
to each cell in the matrix, you collect
data from persons having all the
characteristics of a given cell.
3. All persons in a given cell are then
assigned a weight appropriate to their
proportion of the total.
4. When all the sample elements are
so weighted, the overall data should
provide a reasonable representation
of the total population.
Theoretically, an accurate quota sampling
design should produce results that are reasonably
representative of the larger population. Quota sam-
pling has several inherent problems, however, that
are related primarily to how the cells in the quota
matrix are actually populated. If, for example, the
individuals collecting the quota samples are not
particularly vigilant and honest about filling their
122 Chapter 4 Milestones in the Quality Measurement Journey
quotas, the results will be biased. Remember, the
actual selection of the elements to fill the quota
is left up to the individual gathering the data, not
to random chance. If the data collectors are not
diligent and/or honest about their work, they will
end up obtaining their quotas in a manner that is
more like a convenience sample than a true quota
sample. This happens frequently when quota samples
are being collected in neighborhoods. The 2000
census in Chicago provided a good example of
this type of bias. The census workers were given
quotas to fill on the North Shore of the city. This
is a rather wealthy area where it is not uncommon
to find homes that are gated and monitored by
security. Many of the census workers were not
given access to these homes, even though they
were technically in the cell they were supposed
to obtain. Apparently pressured by the require-
ment to meet their quotas, the census workers
creatively began to substitute other residents for
the ones defined by the quota sample. As a result,
the cells in question (i.e., neighborhoods) were
underreported and not properly representative of
the area (Chicago Tribune, July 5, 2000, “Census
Shortcuts Alleged”). Another threat to the validity
of the quota sample is that the patient population
characteristics might be outdated and not reflect
the current patient population. The final threat
involves the process by which the data collectors
actually gather the data. For example, if a quota
sample was established to gather data in the ED
but only during the day shift, you would run
the risk of missing key data points during the
afternoon and evening shifts.
Judgment Sampling
I saved the discussion of judgment sampling until the
end because it can be viewed in two very different
lights. If you approach sampling from an academic
research perspective, then judgment sampling is
regarded as having a low level of precision and
statistical rigor. If, on the other hand, your objective
is not academic research but rather QI research,
then judgment sampling provides a useful approach
to sampling. The academic view that judgment
sampling (also referred to as purposive sampling)
has a low level of precision is based on the fact that
the sample is drawn on the basis of the knowledge
of the person(s) drawing the sample. No ­objective
mechanical means are used to select the sample.
The assumption is that experience, good judgment,
and appropriate strategy can select a sample that
is acceptable for the objectives of the researcher.
An example of judgment sampling is seen every
4 years when a handful of states and communi-
ties are selected to be “pulse checks” for the U.S.
presidential election. In this case, the assumption
is that the people in Iowa and New Hampshire
are “typical” of the rest of the nation and that the
responses of these citizens provide a snapshot of
how the average American views the presidential
candidates. Obviously the major challenge to
judgment sampling is related to the knowledge
and wisdom of the person making the judgment
call. If everyone believes that this person exhibits
good wisdom, then they will have confidence in the
sample that the person selects. If, on the other hand,
people doubt the person’s wisdom and knowledge,
then the sample will be discredited.
Now, consider the nonacademic use of
judgment sampling. Deming considered judg-
ment sampling to be the method of choice for
QI research. Langley, Nolan, Nolan, Norman,
and Provost, (1996, p. 111) maintain that “A
random selection of units is rarely preferred to
a selection made by a subject matter expert.” In
QI circles, this type of sampling is also known
as expert sampling or rational sampling. It
essentially consists of having those who have
expert knowledge of the process decide on how
to arrange the data into subgroups and pull the
sample. The subgroups can be elected either by
random or nonrandom procedures, which is a
major distinction between the QI perspective and
the academic view of judgment sampling. The
other important distinction about Deming’s view
of judgment sampling is that the samples should
be selected at regular intervals over time, not at
a single point in time. Most sampling designs,
whether they are probability or nonprobability,
are static in nature. The researcher decides on a
time frame then picks as much data as possible.
In contrast, Deming’s view of sampling was that
 Nonprobability Sampling 123

it should be done in small doses (rather than
large quantities) and pulled as a continuous
stream of data (Deming, 1950, 1960, 1975). The
primary criticism of judgment sampling is that
the “expert” may not fully understand all facets
of the population under investigation and may
therefore select a biased sample. The second
criticism is that the sampling error cannot be
measured. The final challenge is that the results
of a judgment sample cannot be generalized to
the larger population because the sample was
not selected by random methods.
A review of the probability and nonproba-
bility sampling methods is provided in TABLE 4-5.
Developing a working knowledge of these sam-
pling techniques will be one of the best ways to
reduce the time spent on collecting data. Done
correctly, sampling will also be a way to ensure
that the data you do collect are directly related
to your QI efforts.
Now that you are familiar with the princi-
pal approaches to stratification and sampling,
it is time to start applying these techniques to
your own set of indicators. TABLE 4-6 provides

TABLE 4-5 Advantages and disadvantages of various sampling methods

Sampling
Method Description Advantages Disadvantages

Probability Sampling

Simple A sample that is ■■ Requires minimum ■■ Does not take advantage

random drawn in such a way knowledge of the of the knowledge the
sample that every member population in advance researcher might have
of a population has ■■ Free of possible about the population
an equal chance classification errors ■■ There could be over- or
of being included. ■■ Easy to analyze the underrepresentation of
A random number data and compute subgroups within the
table or a random errors population
number generator ■■ Fairly inexpensive ■■ Typically produces larger
is typically used to sampling errors for the
actually pull the same sample than a
sample. stratified sample

Stratified The population ■■ Helps to reduce the ■■ Requires knowledge of

random is divided into chances of over/ the presence of various
sample relevant strata underrepresenting characteristics within the
before random subgroups within the population
sampling is applied population ■■ Sampling costs can
to each stratum. ■■ Allows you to segment increase if knowledge
the data into “buckets” of the population is
during the analysis shallow
phase ■■ If the strata are not highly
■■ Create more efficient homogeneous then
samples sampling error goes up
■■ Reduces sampling error and efficiency goes down

(continues)
124 Chapter 4 Milestones in the Quality Measurement Journey

TABLE 4-5 Advantages and disadvantages of various sampling methods (continued)

Sampling
Method Description Advantages Disadvantages

Proportional The proportion (or ■■ Adds even more ■■ Requires more

stratified percentage) of a precision than the
random particular stratum stratified random
sample is determined in sample
the population and ■■ Increases sample
then applied to the representativeness
random sample. ■■ Creates very efficient
samples
■■ Reduces sampling error
human and financial
resources than other
methods
■■ Requires even more
information about the
population than stratified
random methods

Systematic Select every kth ■■ Very easy to conduct ■■ Can produce bias due
sample observation from ■■ Has “intuitive” appeal to periodic ordering
the population after ■■ Inexpensive to conduct of observation, which
a random starting produces exclusion
point has been of segments of the
selected. population
■■ Increased probability of
sampling bias

Cluster Clusters or ■■ Can be low cost, ■■ Clusters need to be

sample “bunches” of the especially if geographic as heterogeneous as
population are clusters are used possible
identified, and then ■■ If properly done, each ■■ Typically has lower
random sampling cluster is a small model statistical efficiency
is applied to each of the population ■■ Large samples are
cluster. ■■ High level of often needed to ensure
practicality precision

Nonprobability Sampling

Convenience Observations are ■■ Ease of obtaining a ■■ Extremely low

sample selected based sample generalizability
on availability ■■ Relatively low cost ■■ No way to determine
and convenience. sampling bias or
Also known as sampling error
“accidental”
samples.
 Nonprobability Sampling 125

TABLE 4-5 Advantages and disadvantages of various sampling methods (continued)

Sampling
Method Description Advantages Disadvantages

Quota A population ■■ Stratification effect is ■■ The people assigned to

sample is divided into achieved if the strata
relevant strata. The are appropriately
desired proportion structured
of samples to be ■■ In theory, the quota
obtained from sample should
each stratum is be reasonably
determined, and representative of the
then a fixed quota population
within each stratum ■■ Human and financial ■■
is set. costs can be kept
to a minimum if the
strata from which the ■■
quotas are to be drawn
are grouped close ■■
together (reduced the
amount of travel the
data collectors have
to perform in order to
gather the data)
collect the quotas need
to be scrupulous and free
from selection bias and
follow the prescribed
sampling design
(otherwise this method
becomes a convenience
sample)
It is difficult to guarantee
that the quotas were
filled accurately
In-depth knowledge of
the population is required
Nonrandom selection
of the quotas can also
introduce bias

Judgment Subgroups are ■■ Samples in a ■■ Sampling bias and

sample drawn from a subgroup can be small sampling error cannot be
process over time (3–5) because many calculated
based on expect subgroups will be ■■ Expert knowledge of the
knowledge. The selected process or population is
subgroup samples ■■ Data collection costs required
can be drawn can be reduced ■■ Generalization of the
either by random ■■ Provides a dynamic judgment sample to
or nonrandom picture of the data larger populations cannot
procedures. and serves as the be done
basis for process ■■ Personal bias enters
improvement into the selection of the
■■ Minimum stratification sample
effect is achieved

a Data Collection Plan Worksheet designed to
help you clarify the data collection plan for your
improvement team. For each indicator that you
identified earlier in this section (see Table 4-3,
the Organizing Your Indicators Worksheet),
outline the decisions your team has made
related to stratification of the data, sampling
(if appropriate), and frequency and duration of
data collection. The frequency of data collection
addresses how often you plan to collect the data.
Will you, for example, collect the wait time of
every patient or develop a sampling strategy?
126 Chapter 4 Milestones in the Quality Measurement Journey

TABLE 4-6 Data collection plan worksheet

Team Name and Improvement Topic: __________________________________________________________

Will you use Duration of

Is stratification sampling? If data collection
appropriate? Yes, describe Frequency of (i.e., how long
If Yes, list the sampling data collection do you plan
Indicator the levels of method you (e.g., hourly, to collect the
Name stratification will use daily, weekly?) data?)

BOX 4-3 Conclusions about data collection

1. Sampling should produce representative and workable numbers for the unit of interest.
2. Customers providing feedback about their service or care they receive can be very susceptible to
sampling bias (sampling and recall biases).
3. Sampling bias can be introduced if you always use the same place or time and this is not
representative of the whole. This is a major problem when single point in time audits are relied
on as the sampling method.
4. When conducting surveys recall bias occurs if the questions are reliant on the individual’s
memory.
5. The worst case scenario occurs when you have no idea where the sample came from or how
representative it is of the population or organization overall.
6. Clear guidance on data collection methods, in particular sampling and stratification, are required
whether you are collecting data for improvement, judgment, or research.
7. Data on “why did this happen” are critical to improvement efforts!

Will you collect the wait time of all patients but
only on Mondays? These questions relate to the
frequency of data collection (i.e., how often do
you need to dip into the ongoing stream of data to
gain an adequate understanding of the variation
in the indicator?). The duration issue deals with
how long you plan to collect the data. Will you
do it for a week, a month, or several months?
If you do not spend time discussing the frequency
and duration questions, you will inevitably come
to a point when someone says, “How long do I
have to collect this stuff?”
Summary conclusions about key data col-
lection issues are provided in BOX 4-3.
 The Indicator Development Worksheet
▸▸ The Indicator
Development
Worksheet
Now that we have reviewed the individual mile-
stones in the QMJ it is time for you to organize
your indicators into a coherent roadmap. The
Indicator Development Worksheet is shown in
EXHIBIT 4-1. It provides a practical and convenient
way for a team to organize the details for one
specific indicator. If you can provide responses
to all of the items on this worksheet you will have
an indicator that, at least for the short term, will
enable you to proceed with data collection and
eventually analysis. The details related to each
section on the Indicator Development Worksheet
are provided next.
1. What is the overall AIM of this im-
provement initiative?
We addressed the particulars of
building an aim statement earlier in
this chapter. It is important, however,
to make sure that the overall aim
is stated when developing specific
indicators so that the team is able
to clearly see how the indicator is
linked to what the team is trying
to accomplish. How good do you
want to be? And by when do you
expect to achieve the outcome? These
two simple questions are essential
to a team’s journey. Again, it does
not have to be a long and detailed
statement. One or two sentences
should suffice.
2. What is the NAME of this SPECIFIC
INDICATOR?
Naming indicators is an important
component of indicator development
that is frequently taken for granted.
Some might ask, “What’s the big
deal? Just give it a name.” Indicator
names should be objective, and they
should reflect quantifiable nouns.
Often, however, teams feel the need
127
to include adjectives and adverbs
as well as targets and goals in the
indicator name (e.g., history and
physical transcription TAT will be
12 hours or less). This produces what
I call “thou shalts” (i.e., thou shalt
perform this task in 12 hours or less or
there will be consequences). Indicators
named in this fashion identify the
desired outcome. When you include
the desired level of performance in
the indicator name, you have basi-
cally built in a barrier or, worse yet,
a threat. It sends a message to the
workers that you had better do this
or else. If the desired outcome is an
unrealistic goal, the workers quickly
figure this out. The indicator then
becomes an unrealistic metric or a
joke. Consider this example: I was
working with a medical group on
their indicators and asked a team
what they intended to measure.
A member of the team said that
the indicator was, “No one should
have to wait more than 30 minutes
to see the doctor.” My comment to
the team was that this was not the
name of an indicator but rather a
threat. The indicator name should
have been “wait time to see the
doctor.” Although it seems like a
minor aspect of performance mea-
surement, I believe that the naming
of indicators sets the tone for the
rest of the measurement journey.
It is the point at which you leave
Conceptland and actually enter into
Measurementland.
3. What TYPE of INDICATOR is this?
It is important to be clear about
the types of indicators being developed.
Teams need to be careful to not develop
so many indicators that they become
over burdened with collecting data. Most
improvement teams will be tracking
one to three outcome measures, four or
128 Chapter 4 Milestones in the Quality Measurement Journey

EXHIBIT 4-1 Indicator Development Worksheet

Team name and topic of interest: 
Date: __________________ Contact person: ________________________ Email 
1. What is the overall AIM of this improvement initiative?
(How good do you want to be? By when do you expect to achieve the outcome?)
2. What is the NAME of this SPECIFIC INDICATOR? (e.g., the number of x-ray exams performed,
the percentage of x-ray reports that could not be found, the medication error rate or the days
between a patient fall).
3. What TYPE OF INDICATOR is this?
____ Outcome ____ Process ____ Balancing
4. What is the OPERATIONAL DEFINITION for this indicator?
Define the specific components of this indicator. Specify the numerator and denominator if it is
a percentage or a rate. If it is an average, identify the calculation for deriving the average. Include
any special equipment needed to capture the data. If it is a score (such as a patient satisfaction
score) describe how the score is derived. When an indicator reflects concepts such as accuracy,
complete, timely, or an error, describe the criteria to be used to determine “accuracy.”
5. What is your DATA COLLECTION PLAN?
•• How frequently will the data be collected?
____ Every Patient____ Hourly____ Daily ____Weekly ____Monthly ____Other (please
specify)
•• What are the data sources to be used for this indicator (be specific)?
•• What is to be included or excluded (e.g., only inpatients are to be included in this measure or
only stat lab requests should be tracked).
•• How will these data be collected?
____Manually ____From a logbook ____From an automated system
•• Who will be responsible for the actual collection of the data?
_________________________________________________
•• Will you use stratification? If “Yes” specify the stratification levels you will use.
•• Will you employ sampling? If “Yes” specify your sampling strategy you plan to use.
6. Do you have BASELINE DATA for this indicator? ____Yes ____No ____Unknown
•• What is the actual baseline number? ________________________________
•• What time period was used to collect the baseline? ____________________
7. Are there TARGET OR GOALS for this indicator?
Internal target(s) or goal(s)? ____Yes ____No ____Unknown
If “yes” please list the actual internal target or goal (e.g., the number, rate, or volume, etc., as well
as the source of the target/goal)
______________________________________________________________________
External target(s) or goal(s)? ____Yes ____No ____Unknown
If “yes” please list the actual internal target or goal (e.g., the number, rate, or volume, etc., as well
as the source of the target/goal)
______________________________________________________________________
 The Indicator Development Worksheet
five process measures, and one or two
balancing measures (if appropriate).
As mentioned earlier in the Types
of Indicators section of this chapter,
the objective should be to identify
a reasonably small set of indicators
(six to eight total) that capture the
critical aspects or dimensions of the
team’s aim (i.e., build a balanced set
of indicators).
4. What is the OPERATIONAL DEFI-
NITION for this indicator?
This is essentially the heart and
soul of the Indicator Development
Worksheet. In this section, you should
provide detail on the components of
the operational definition in very spe-
cific terms. If it involves a percentage,
then the numerator and denominator
should be described. Similarly, if the
measure is a rate, then the rate-based
statistic should be defined (i.e., falls
per 1,000 patient days). The easiest
way to do this is to take the indica-
tor name (e.g., inpatient fall rate)
and then say, “Inpatient fall rate is
defined as . . .(fill in the blank).” If it
is an average, identify the calculation
for deriving the average. Include any
special equipment needed to capture
the data. If it is a score (such as a
patient satisfaction score) describe
how the score is derived. When an
indicator reflects concepts such as
accuracy, complete, timely, or an
error, describe the criteria to be used
to determine “accuracy.” Remember
to describe what is to be included
(e.g., all inpatients, including pe-
diatrics, and geriatrics and falls in
the ED) and what is to be excluded
(e.g., visitor falls in and out of the
hospital, staff falls, and falls in the
rehabilitation unit). The litmus test
for a good operational definition is
really quite simple. Just ask yourself,
“How could someone get confused
129
with this definition and collect
wrong data?” If you have written a
clear and unambiguous operational
definition, then you will be able to
avoid confusion during the data
collection stage.
5. What is your DATA COLLECTION
PLAN?
The items listed in this sec-
tion require knowledge, skill, and
experience with data collection
practices for QI, not data collection
practices for conducting randomized
clinical trials (RCTs) or traditional
academic research but QI research.
A key difference between data for
QI and data for RCTs or more tra-
ditional academic research is that
QI data will typically be collected
more frequently and in smaller
doses than traditional research. The
questions related to data collection
are listed in the worksheet but a few
suggestions on data collection are
listed here.
• Collection Frequency and
Duration. Remember that
frequency deals with how often
you plan on collecting the data,
whereas duration addresses the
question of how long you will
continue to collect the data.
Will you collect the wait time
of every patient or develop a
sampling strategy? Will you col-
lect the wait time of all patients
but only on Mondays? Next
focus on the duration of data
collection question. How long
do you plan on collecting the
data? Will you do it for a week,
a month, or several months?
Frequency and duration are
critical issues for the team to
address and they need to be
clarified before you actually
start to collect the data.
130 Chapter 4 Milestones in the Quality Measurement Journey
• Data Sources. Where do you plan
on getting the data? Will they
be manually collected, or will
they come from an automated
system? If it is to be a manual
process, will they come from
existing log sheets or the medical
record, or do you have to create a
new data collection tool? If they
come from an automated system,
what segment of the automated
system will be used (e.g., is it the
registration system, the billing
system, or the patient satisfaction
system)? A related issue is by
what method do you propose
to actually gather the data? For
example, if you are tracking lab
TAT, will the recorded time
come from the watch of the
individual who is recording the
log-in time, the clock on the wall
by the door, or the automated
time stamp produced by the
computer system? If the data
are to be collected manually, do
you have a procedure outlining
how the person recording the
data is supposed to identify the
particular piece of data (e.g.,
the log-in time to the lab of a
specimen) and then enter it into
the logbook? For some of you
this probably seems like a very
left-brained, compulsive set of
questions. If someone does not
attend to the details, however, they
will be ignored. This is why you
need both left- and right-brained
people on QI teams. A team
with all left- or all right-brained
people usually does not achieve
as much as teams that have a mix
of perspectives. Sometimes you
need vision and creativity, and
sometimes you need structure
and attention to details.
• Person(s) Responsible for
Collecting the Data. It is not
uncommon to arrive at this step
in the process and realize that
you have not made any provi-
sions for actually collecting the
data. Everyone seems to assume
that someone else will do the
work of recording wait times
or extracting documentation
history from the medical re-
cords. Someone has to do the
data collection. Frequently,
however, there is a wonderful
chain reaction when it comes
to this task. Physicians assume
that the nurses will collect the
data. The nurses assume that the
unit secretaries will complete
this task. The unit secretaries
hope that several administrative
interns will be assigned to the
department for the summer
and this job can be pawned
off on them. If all else fails, the
volunteers can be asked to do
the data collection. Now think
about this. If you have spent
considerable time building
the indicator’s operational
definition and data collection
plan, why would you assume
that some undetermined per-
son will magically appear and
solve all your data collection
problems? In Greek dramas,
this solution was referred to
as the deus ex machina (the
god from the machine) or the
unexpected solution to a dif-
ficult problem.11 If this aspect
of the measurement, journey
is not determined prior to the
collection of data, I can guar-
antee you that (1) it will not
go smoothly and (2) the data
gathered will be questionable.
 The Indicator Development Worksheet
• Stratification and Sampling.
Considerable detail on stratifi-
cation and sampling has already
been provided in this chapter.
It is essential, however, that
the team devote time to dis-
cussing the relevance of these
two extremely important data
methods to their data collec-
tion plan. Both are extremely
important from a practical as
well as resource perspective.
As was mentioned, they are
less about statistical issues than
logical issues and they can be
properly decided upon only by
those who have subject matter
experience.
6. Do you have BASELINE DATA for
this indicator?
Remember that the baseline
is what the current process is pro-
ducing. It is not what you want it
to be or expect it to be. Baseline is
a fundamental concept in medicine.
We get a baseline on a patient before
we start to prescribe medications
or treatments. In a similar fashion,
we basically want to know how the
indicator we are tracking is per-
forming and what it is capable of
producing under current operating
conditions. Also note that targets
and goals should not be established
without having a clear understand-
ing of what the current baseline
is. Frequently, however, teams set
targets and goals without knowing
the current performance, which
typically leads to the establishment
of arbitrary and capricious targets
or goals that have little chance of
being reached.
7. Are there TARGETS or GOALS for
this indicator?
This is where you identify what
you want or expect the indicator’s
131
performance to be. Targets are usually
seen as short-term objectives (several
months to a year). Goals, on the other
hand, are usually designed for a little
longer period of time, say 2–3 years.
I have seen numerous examples,
however, of how organizations have
confused staff by not being clear on
whether the new number was a target
or a goal. Frequently, people use the
terms as if they were synonymous, but
note that targets are typically more
short term (e.g., weeks or months)
whereas goals are more long term
(e.g., a year or longer). The critical
points are that you (1) develop tar-
gets and/or goals that are reasonable
and achievable and (2) have a plan
for how the targets and/or goals are
to be achieved. As Deming pointed
out frequently in his seminars and
writings, “Goals are necessary for
you and me, but numerical goals for
other people, without a road map to
reach the goal, have effects opposite
to the effects sought” (Deming, 1992,
p. 69). The simpler version of Dem-
ing’s basic challenge was “By what
method?” That is, by what method
do you propose to achieve this target
or goal?
Another key aspect of targets
and goals is whether they are es-
tablished internally or externally.
Increasingly, healthcare targets and
goals are being established by exter-
nal bodies and given to providers.
In this case, however, the external
bodies may not fully understand the
capability of the existing processes
to achieve the targets or goals. This
can lead and has led to considerable
stress and challenge for healthcare
providers.
The final aspect of setting targets
and goals is one that has fascinated
me for years. My experience has
132 Chapter 4 Milestones in the Quality Measurement Journey

convinced me that nearly all targets innovation into a meeting propose

and goals are established as whole
numbers that are divisible by 5. The
next time you are in a meeting where
targets and goals are being discussed
test my theory. You will generally
find that people start a discussion
by saying “I think we need a 10%
reduction in X.” This will be followed
by someone else saying, “No, I think
we should shoot for a reduction of
15%.” If 15% is not accepted as the
target someone else will then raise the
ante to 20% or even 25% . . . and so
on. If you want to offer a disruptive
that the next target or goal be 11.58%
not 15 and see what happens. I did
this once in a management meeting
and a majority of the attendees looked
at me in a very strange way. One
of them even said, “Are you trying
to be a wise guy? Why would we
have a target of 11.58%?” I smiled
at the fellow and asked him why he
thought whole numbers divisible by
5 provided a realistic foundation for
the target. He hesitated for a moment
the responded, “Well, that is how we
always set targets.” QED!

CASE STUDY #1: Transcription Turnaround Time

The following case study is designed to demonstrate how the milestones addressed in this chapter can
be applied to an improvement challenge: reducing transcription turnaround time (TAT) for histories
and physicals (H&Ps).

Situation
Imagine that you are the director of QI at a medium-sized hospital. One morning you receive
a call from your friend Becky, the manager of medical transcription, who asks if she can meet with
you ASAP. You sense that she is bothered by something and tell her that you will come to her office in
an hour. You have known Becky for more than 8 years and are a little surprised when she dispenses
with the usual pleasantries and jumps right into her concern. She tells you that several physicians
have been complaining recently about transcription TAT for H&Ps. She even confides in you that she
actually had a rather “energetic” exchange with the head of surgery about this issue earlier in
the morning. Now Becky is asking for your help, as she explains, “to get the doctors to realize that TAT is
actually very good.” To “prove” her point she shows you FIGURE 4-7. She points out quickly that the goal
is to get the transcriptions completed in 12 hours or less. Becky feels that the graph demonstrates that
over the last 10 months the transcription team has been able to meet this goal 92–99% of the time.
She asks, “So what is the big deal with the physicians? Why are they complaining?”

Diagnosis of the Problem

As you look at the title of Figure 4-7, the first thing that strikes you as confusing is that the
indicator of interest is transcription TAT, yet Becky has presented the data as the percentage of
H&P transcriptions completed within 12 hours. You also point out that she has actually violated
one of the basic principles of naming an indicator—she created a “thou shalt” (H&P transcription
TAT will be 12 hours or less). Her response is, “We have to have a goal; otherwise, the doctors will

(continues)
 The Indicator Development Worksheet 133

CASE STUDY #1: Transcription Turnaround Time (continued)

100%

98%

96%
Percentage

94%

92%

90%

88%
Jan-00 Feb Mar Apr May Jun Jul Aug Sep Oct
Month
FIGURE 4-7 Percentage of history and physical (H&P) transcription turnaround times completed
within 12 hours

not take us seriously.” You respond by saying that goals are critical to improvement initiatives
but that they should not be built into the name of the indicator. Indicator names should be
objective statements about what is to be measured, not what you want it to be. You also point
out that by measuring TAT as a percentage, you are losing information about the true variation
in the process. For example, consider the month of August in Figure 4-7. During this month, 92%
of the transcriptions were completed in 12 hours or less. What was the variation in the TATs for
this month? What was the shortest TAT and what was the longest TAT in this month? You cannot
answer this question because the measure is whether the transcriptions were completed in more
than 12 hours or less than 12 hours. This is a yes or no type of indicator. The longest TAT might
have been 13 hours or 40 hours. But because of the way in which the indicator was developed,
you will never know the answer to this question about variation. You suggest that it would be
useful, therefore, to look at the actual time (in hours) to transcribe H&Ps. Becky is so desperate for
help that she agrees to do this even though she is not sure why.
You also point out that there is no necessary reason why this indicator is structured around
monthly data. Becky has shown you only 10 months of data. This is the minimum amount of data for
a run chart (i.e., 10 data points) but you are thinking that there is probably a sufficient amount of data
that it could be broken down into smaller subgroups that would allow the construction of a Shewhart
chart.12 So, you ask, “Why can’t you display this indicator by week or every 2 weeks?” Becky’s response
is one you have heard many times before: “Well, there are several reasons. This is how we have always

(continues)
134 Chapter 4 Milestones in the Quality Measurement Journey

CASE STUDY #1: Transcription Turnaround Time (continued)

TABLE 4-7 Summary data by month for H&P turnaround times

Month Number of H&P Reports Total Hours TAT* Average TAT Hours

January 500 5,140 10.3

February 487 5,734 11.8

March 498 4,948 9.9

April 521 6,024 11.6

May 517 5,882 11.4

June 508 5,913 11.6

July 489 5,756 11.8

August 501 6,031 12.0

September 520 5,850 11.3

Goal = 12 hours *Hours are rounded up to the nearest hour

collected this data, it is how the tracking system is set up and how management expects to see the
data reported.” You explain that by moving to more frequent subgroups such as by week or every
2 weeks, a more detailed picture of the true variation in the process will be obtained. Becky is not quite
sure what you are talking about but she is willing to bring you the detailed data.
The next day Becky comes to your office with TABLE 4-7 and FIGURE 4-8. Table 4-7 shows
the number of H&P reports completed each month, the total hours required to complete the
transcriptions, and the average hours. Figure 4-8 presents these data as a line graph with the 12-hour
goal as a reference line. Becky is quick to point out that this chart clearly “proves” her point. All of the
average TATs she says, are below the goal of 12 hours. So, again, why are the doctors upset? What
would be your next set of recommendations for these data?

Recommendations
As you look at Figure 4-8 you notice what you believe might be causing the physicians to complain
about TAT. You see from the title of this chart that the process starts with dictation and ends
when transcription is completed. You ask Becky a simple question: “What do the physicians care
most about?” She looks at you in a quizzical manner and says, “I don’t understand your question.”
You proceed to explain that the physicians probably do not care very much about the time
from dictation to transcription but rather the time from dictation to posting the results on the

(continues)
 The Indicator Development Worksheet 135

CASE STUDY #1: Transcription Turnaround Time (continued)

20

15

Goal < 12 hours

Hours

10

0
Jan-00 Feb Mar Apr May Jun Jul Aug Sep Oct
Month
FIGURE 4-8 Average H&P transcription turnaround times from dictation to transcription

computer or finding the results in the chart. Becky is very quick to point out that “now you’re being
unreasonable.”
She continues by explaining that they have never looked at the end piece of the process (i.e.,
transcription to posting of the results) because that part of the process has never been very good,
and it is something they do not feel that they have much control over. She concludes her comments
by stating, “We look at what we do and expect someone else to take care of the rest of the process
by posting the results in a timely manner. It is not my problem that the people who are supposed
to post the results don’t do it according to the doctor’s demands.” You politely point out that the
customers apparently do believe that the transcription department is responsible for making the
posting of results happen in a timely manner. With reluctance Becky agrees to look at the process
from dictation to charting, but she points out that there is no way that they can look at all the cases
for these new starting and ending points. At this point you are glad she has raised this issue because
it provides you with an opportunity to return to the topic of the monthly data collection. Your
guidance is as follows:
■■ Upon looking at Table 4-7 you explain that having roughly 455 transcriptions each month
provides ample reason to make smaller subgroups.
■■ With this many transcriptions each month it would be possible to make a subgroup for every
2 weeks of data (with roughly 226 transcriptions being received every 2 weeks). Or if week was

(continues)
136 Chapter 4 Milestones in the Quality Measurement Journey

CASE STUDY #1: Transcription Turnaround Time (continued)

selected as a subgroup then there would be about 114 transcriptions received each week.
Finally with this amount of data, day could be used as the subgroup, which would provide
approximately 15 transcriptions being received each day. The problem with looking at the data
by day, however, would be that this would produce over 300 individual data points on a chart
for the 10-month period, which is clearly not necessary.
■■ In discussing these options with Becky, you reach agreement that looking at the data by week
makes the most sense. This will produce 40 data points on a chart, which is not excessive. You
go a step further to point out to Becky that analyzing all the data in a week (approximately 114
transcriptions each week) is also not necessary. A smaller subset of transcriptions (approximately
15–20 each week) can be selected by developing a sampling strategy. Becky agrees to help you
in implementing a sampling strategy on future data but asks that you use the existing data for 10
months to help her think through her current challenge of addressing her customers’ dissatisfaction
with the TAT.
When Becky returns to your office on Monday, she seems to be somewhere between shock
and embarrassment. When you look at FIGURE 4-9 you start to understand why she is in such a state
of mind. It shows that the average TAT for the transcription process, dictation to posting on the
patient’s chart, is somewhere around 18 hours—much higher than the goal of 12 hours discussed
earlier. When you ask Becky about these results, she says that they are very surprising. Because you
know her well, however, you quickly figure out that she is not being totally honest with you. With a
little probing she admits that she knew this all along and that this is why she decided to focus only

20

15

Goal < 12 hours

Hours

10

0
Jan-00 Feb Mar Apr May Jun Jul Aug Sep Oct
Month

FIGURE 4-9 Average H&P transcription turnaround times from dictation to chart

(continues)
 The Indicator Development Worksheet 137

CASE STUDY #1: Transcription Turnaround Time (continued)

on the dictation-to-transcription part of the process (this is where the embarrassment starts to settle
in). So, you offer another quality measurement insight. Specifically, you ask if this process needs to
be stratified. Becky is not quite sure where you are going with this question. You explain that it might
be possible that the transcription process varies by day of the week, time of day when the dictation
is received, or possibly by type of procedure. As you explore this idea with her, she indicates that it is
possible that the TAT could vary by type of patient, namely, nonsurgical versus surgical H&Ps. When she
says this you notice a slight hesitancy in her voice, but you are not sure why.
To your surprise, Becky returns to your office later that same day. She now has two new charts.
FIGURE 4-10 shows the TAT for nonsurgical patients and FIGURE 4-11 shows the results for surgical
patients. As soon as you look at the charts you realize why Becky was looking a little embarrassed
and sounded hesitant earlier in the day. The answer to her problem is obvious. There is not one
transcription TAT process occurring here but two. There is one process for nonsurgical patients and
another for surgical patients. The nonsurgical patient TATs are turned around in roughly 21 hours
whereas those for the surgical patients have TATs of about 5 hours. This spread in the two processes
helps to explain why the average for all patients is about 18 hours (Figure 4-9): it reflects the average of
two very different processes.
So why the big discrepancy? It turns out that two of the surgeons were very upset with
Becky and her department about 8 months ago. They complained not only to the president of

25

20

15
Hours

Goal < 12 hours

10

0
Jan-00 Feb Mar Apr May Jun Jul Aug Sep Oct
Month

FIGURE 4-10 Average H&P transcription turnaround times for nonsurgical patients from dictation to
chart

(continues)
138 Chapter 4 Milestones in the Quality Measurement Journey

CASE STUDY #1: Transcription Turnaround Time (continued)

14

12 Goal < 12 hours

10

8
Hours

0
Jan-00 Feb Mar Apr May Jun Jul Aug Sep Oct
Month

FIGURE 4-11 Average H&P transcription turnaround times for surgical patients from dictation to chart

the medical staff but also to the CEO about the TAT. Becky said she still remembers vividly the
discussion she had with the CEO when she was called to her office. In order to avoid that situation
again, Becky made sure her staff gave preferential treatment to any H&P related to surgical
procedures. As a result, the nonsurgical H&Ps received less attention than the surgical patients, as
demonstrated in Figures 4-10 and 4-11.

The Improvement Plan

Once Becky was over her cathartic release of pent-up angst, she was able to acknowledge readily
that she and her department set themselves up for this outcome. Obviously, one objective will be
to maintain the performance of the surgical H&P TAT process. The challenge will be to work on the
nonsurgical process and take steps to reduce this time to meet the goal of 12 hours or less. You agree
to assist Becky by helping her (1) create an improvement team to work on the TAT process, (2) serve
as the improvement advisor for the team, (3) guide them through the next round of data collection
and analysis, (4) develop a stratified proportional random sample for a weekly subgroup of 15–20
transcription times, and (5) create a Shewhart chart that is appropriate for understanding the variation
in TAT (i.e., an X bar and S chart described in Chapter 9). Becky leaves your office not only feeling better
about revealing the entire story behind this issue but also motivated to work with her team to improve
the nonsurgical process.

Notes
1. An excellent book on this topic is The Cult
of Statistical Significance: How the Standard
Error Costs Us Jobs, Justice and Lives by
Stephen Ziliak and Deirdre McCloskey
(The University of Michigan Press, 2011).
2. This is actually the first question in IHI’s
approach to improvement. It is part of a
larger framework called the Model for
Improvement (MFI), which was developed
by Associates in Process Improvement
(API). The MFI is detailed in The Im-
provement Guide (Langley et al., 2009).
IHI has adopted the MFI as its overarching
approach to improvement. More will be
said about the MFI in Chapter 9.
3. The higher up individuals go in the food
chain the more they seem to live in Con-
ceptland. Board members, senior leaders,
political leaders, and even the media
frequently live in or have time-shares
in Conceptland. This is not necessarily
bad. These folks are supposed to set di-
rection and provide visions of what can
be achieved. The problem is that if an
organization does not have people skilled
in the science of improvement who can
move the organization out of Conceptland
and into Measurementland the visions
will never be achieved.
4. When teams get stuck in Conceptland and
have vague notions of what they want to
improve it reminds me of the famous state-
ment often credited to Horace Greeley (or
as the story goes possibly to John Babson
Lane Soule): “Go West young man.” A
general direction is offered but no specific
aim or milestones are given as to where in
the West you should end up. So the journey
stays essentially in Conceptland.
5. Knowing the difference between a pro-
portion, a percentage, and a rate is critical.
Often in healthcare settings people refer
to a rate when they are really referencing
a percentage. Additional detail on these
distinctions is provided in Chapter 6.
Notes 139
6. Increasingly, politicians are being asked to
clarify what they say and what they mean.
It is not uncommon, for example, for a
political candidate or an incumbent to have
a reporter repeat what the politician said
yesterday or last week about an issue and
then ask them why they are now saying
something different. The evening news is
notorious for playing a clip of what the
president said last month and then show-
ing that he said just the opposite today.
Think how easy our nation’s founders
had it in this regard. It was probably very
easy for them to take a position one day
and another position the next. No one
recorded their comments verbatim, and
there were no cameras, videotapes, or
recording devices. Operational definitions
in George Washington’s day could be very
loose. Today, however, there is increasing
scrutiny on the part of the public and a
desire to have the politicians be more
precise in their definitions of terms.
7. You will need to be clever in how you
actually fit this into a conversation or
a meeting. One possible opening is to
couch it in light of some vague reference
to the public release of healthcare data
and indicate that you read in the paper
that there would be a “random sample of
patients pulled from all admissions.” Now
you can innocently ask, “Have any of you
ever drawn a random sample?” and it will
seem to be consistent with your setup.
8. Although it seems counterintuitive, it is
possible to have too much data. There is
frequently a belief (although it is generally
a false belief) that if a little data is good,
then a lot of data must be better. This is
not always the case. For example, when
a national news service conducts one of
its “man on the street” surveys to test
the political climate or a national public
polling agency conducts a survey of
American opinions, how many people do
you think they include in their sample?
Typically they shoot for 1,000 to no more
140 Chapter 4 Milestones in the Quality Measurement Journey
than 2,000 people. We have more than 280
million people in this country and they
get only 1,500 respondents. Why don’t
they get more? After a certain point the
additional data do not add anything to the
statistical precision of a study. It merely
wastes resources and time. A general rule
of thumb is that 30–50 observations (data
points, survey respondents, or numbers)
will start to produce a distribution. If you
stratify your respondents by age, gender,
race, region of the country, urban/rural
status, education, income, and religious
preference, which is what the national
news polls do, then you need more than
30–50 observations in order to ensure that
each level of stratification has sufficient
data to enable the appropriate statistical
analysis. Telemarketers are experts at
sampling. They can pinpoint down to the
neighborhood area or census track level
how many people represent the categories
they need for their marketing study. A
stratified proportional random sampling
plan is put into place, the computer au-
tomatically dials the numbers, and your
dinner is interrupted because you fit the
sampling profile they need. But remember
they do not need much data to complete
their sampling plan.
9. While I was a doctoral student at Penn
State University in the Department of
Agricultural Economics and Rural So-
ciology, Dr. Bob Bealer used this phrase
frequently. He used it when a student
would ask, “How many pages do you
want for this paper?” or when one of us
would want to know how much data we
needed to produce a “significant” result.
Professor Bealer challenged us to think
by using few words. He knew that the
answers were rattling around somewhere
within our developing brains. His skill
was in providing a light for us to find the
path. As much as you must and as little
as you dare—it is a wonderfully simple
phrase that relates to many aspects of life.
10. There are two classic stories about sam-
pling that both the critics of sampling
and its proponents have referenced
for years. The first is the 1936 Literary
Digest poll that predicted the landslide
victory of Alf Landon over incumbent
president Franklin D. Roosevelt. Using
a mailed sample of more than 2 million
voters, the Literary Digest predicted that
Landon would win by almost 15%. The
mistake they made was in selecting the
list of individuals for the sample (this is
referred to as the sampling frame). The
sample was drawn from telephone direc-
tories and automobile registration lists.
These methods had worked in the past
elections quite nicely. What the Literary
Digest pollsters forgot was that in 1936
the nation was still feeling the negative
effects of the depression and the more
positive impacts of Roosevelt’s New Deal
program. The 1936 election witnessed an
unprecedented turnout of poor voters.
These people were not proportionately
represented in the telephone and car
registration lists because they could not
afford such luxuries. The other key issue
that the Literary Digest missed was that
the poor voters were primarily Demo-
crats whereas the more wealthy voters,
who could afford cars and telephones,
were primarily Republicans. In this same
year, however, George Gallup correctly
predicted that Roosevelt would be the
winner. Gallup’s approach was based on
using quota sampling, which ensured
that samples were drawn from various
segments of society (e.g., urban, rural,
rich, poor, Republicans, and Democrats).
As a result of this event, Gallup’s credi-
bility increased dramatically while that
of the Literary Digest plummeted. The
next major sampling fiasco occurred in
1948 when Gallup, and most other public
opinion polling organizations, predicted
that Thomas Dewey would be victorious
over Harry Truman. What they all missed

in this case was (1) that nearly all the
pollsters finished their polling too soon
and missed the late surge for Truman and
(2) the people who in earlier polls said
they were not sure who they would vote
for decided to vote predominantly for
Truman. The success that Gallup had in
1936 with quota sampling proved to be
disastrous 12 years later. It was after the
1948 election that academic statisticians
began a serious push for using probability
theory as a basis for drawing samples.
Today the use of probability sampling
methods remains the accepted standard
for drawing the least amount of data
with the highest level of predictability
and confidence.
11. The other alternative to the deus ex machina
is found in the following story:
The Facts of Life
The story that follows is about four
people named Everybody, Somebody,
Anybody, and Nobody. There was an
important job to be done and Every-
body was asked to do it. Anybody
could have done it, but Nobody did
it. Somebody got angry about that
because it was Everybody’s job. Ev-
erybody thought Anybody could do it,
but Nobody realized that Everybody
blamed Somebody when Nobody
accused Anybody.
I am not sure of the origin of this story.
My mother gave me a copy of it when I
first started college. At the time I accepted
it graciously and tucked it away, thinking
that it was one of those things mothers give
their children as they go off to college and
hope that it makes them think about how
their actions affect others. That was back in
1966. Today, I still have the original piece
of paper she gave me with this story typed
on it. Over the years it is funny how many
times I have pulled out this little piece of
paper or run across it in a cluttered desk
drawer and realized how relevant the
References 141
lines are to so many aspects of life. Many
of the challenges we face with data and
measurement stem from the fact that the
people who own the process do not take
ownership of their data and the results
produced by their processes. Inevitably,
when I am involved with assisting people in
developing their indicators, there will be a
moment when their discussion about data
collection makes me think of this story.
12. The subgroup is basically how you have
organized your data (e.g., daily, weekly or
monthly) and appears on a chart as the
label on the x or horizontal axis. More
detail on selecting and using subgroups
will be provided in Chapters 8 and 9.
References
Babbie, E. R. The Practice of Social Research. Belmont, CA:
Wadsworth, 1979.
Brooke, R., C. Kamberg, and E. McGlynn. “Health System
Reform and Quality.” Journal of the American Medical
Association 276, no. 6 (1996): 476–480.
Caldwell, C. Mentoring Strategic Change in Health Care.
Milwaukee: Quality Press, 1995.
Campbell, S. Flaws and Fallacies in Statistical Thinking.
Englewood Cliffs, NJ: Prentice-Hall, 1974.
Daniel, W., and J. Terrell. Business Statistics. Dallas: Houghton
Mifflin, 1989.
Deming, W. E. Some Theory of Sampling. New York: John
Wiley & Sons, 1950.
Deming, W. E. Sample Design in Business Research. New
York: John Wiley & Sons, 1960.
Deming, W. E. “On Probability as a Basis for Action.” Amer-
ican Statistician 29, no. 4 (1975): 146–152.
Deming, W. E. Out of the Crisis. Cambridge, MA: Massa-
chusetts Institute of Technology, Center for Advanced
Engineering Study, 1992.
Deming, W. E. The New Economics for Industry, Government,
Education. Cambridge, MA: MIT Press, 1994.
Donabedian, A. Explorations in Quality Assessment and
Monitoring. Vol. 1: The Definition of Quality and
Approaches to Its Assessment. Ann Arbor, MI: Health
Administration Press, 1980.
Donabedian, A. Explorations in Quality Assessment and
Monitoring. Vol. 2: The Criteria and Standards of Quality.
Ann Arbor, MI: Health Administration Press, 1982.
Duncan, A. Quality Control and Industrial Statistics, 5th ed.
Homewood, IL: Irwin Press, 1986.
Gonick, L., and W. Smith. The Cartoon Guide to Statistics.
New York: Harper Perennial, 1993.
142 Chapter 4 Milestones in the Quality Measurement Journey
Hess, I., D. Riedel, and T. Fitzpatrick. Probability Sampling
of Hospitals and Patients. Ann Arbor, MI: Health
­Administration Press, 1975.
Institute of Medicine. Crossing the Quality Chasm. Wash-
ington, DC: National Academy Press, 2001.
Ishikawa, K. Guide to Quality Control. White Plains, NY:
Quality Resources, 1982.
Joint Commission on Accreditation of Healthcare Orga-
nizations. The Measurement Mandate: On the Road to
Performance Measurement in Health Care. Oak Brook,
IL: JCAHO, 1993.
Kaplan, R., and P. Norton. “The Balanced Scorecard—­
Measures That Drive Performance.” Harvard Business
Review (January–February 1992): 71–79.
Kaplan, R., and P. Norton. “Putting the Balanced Scorecard
to Work.” Harvard Business Review (September–October
1993): 134–147.
Kaplan, R., and P. Norton. “Using the Balanced Scorecard
as a Strategic Management System.” Harvard Business
Review (January–February 1996): 75–85.
Langley, G., K. Nolan, T. Nolan, C. Norman, and L. Provost.
The Improvement Guide. San Francisco: Jossey-Bass, 1996.
Maddox, B. Sampling Concepts, Strategy and Techniques
(Technical Report 81-1). Harrisburg: Pennsylvania De-
partment of Health, State Health Data Center, July 1, 1981.
Mann, N. R. The Keys to Excellence: The Story of the Deming
Philosophy. London: Mercury Books, 1989.
Miller, D. Handbook of Research Design and Social Mea-
surement. New York: David McKay Company, 1964.
Nelson, E., P. Batalden, and M. Godfrey. Quality by De-
sign: A Clinical Microsystem Approach. San Francisco:
Jossey-Bass, 2007.
Selltiz, C., M. Jahoda, M. Deutsch, and S. Cook. Research
Methods in Social Relations. New York: Holt, Rinehart
and Winston, 1959.
Shewhart, W. Economic Control of Quality of Manufactured
Product. New York: D. Van Nostrand, 1931. Reprint,
Milwaukee: Quality Press, 1980.
Weiss, R. Statistics in Social Research. New York: John Wiley
& Sons, 1968.
Western Electric Co. Statistical Quality Control Handbook.
Indianapolis, IN: AT&T Technologies, 1985.
Wheeler, D. Understanding Variation: The Key to Managing
Chaos. Knoxville, TN: SPC Press, 1993.
CHAPTER 5
Organizing Indicators into a
Strategic Dashboard
M
ost quality improvement (QI) teams
will create more than one indicator.
If your organization has 10 to 15 im-
provement teams, this means that you could be
looking at managing somewhere between 50 and
120 indicators (assuming that each team creates
five to eight indicators, which is usually at the low
end of what teams want to track). So, how do you
go about organizing multiple indicators? First
of all you need a simple way to summarize and
organize the various indicators you have produced
so that they can be presented in meetings and
shared with others. TABLE 5-1 provides a template
for organizing your indicators. The details for
each indicator would be documented in the
Indicator Development Worksheet presented in
Chapter 4. You do not need (nor want) to give
all the details of the indicators to the Quality
Council or to senior management. They need
a 40,000-foot view. As owners of the process,
you and your team members need to have the
details on each indicator. Groups listening to
you explain your measurement plan, however,
want to know the big picture, not the details.
© Michal Steflovic/Shutterstock
Each row in Table 5-1 represents a single
indicator. The columns identify the major pieces
of information that need to be summarized about
each indicator. TABLE 5-2 provides an example
of a completed measurement plan for a team
working to improve the medicines management
process. Three key concepts provide the focus
for the team (i.e., volume, patient safety, and
efficiency). Each concept will be captured by:
■■ The count of the number of med orders
each day (volume)
■■ Percentage of medication orders with one
or more errors (patient safety)
■■ Turnaround time (TAT) to process a med
order (efficiency)
The details for each measure would be
specified in the Indicator Development Work-
sheet, which would not be presented to others
outside the improvement team. When it came
time to present to the Quality Council or to the
management team, all the improvement team
will need to do is present the summary table
(Table 5-2) to the group and briefly describe
143
144 Chapter 5 Organizing Indicators into a Strategic Dashboard

TABLE 5-1 Dashboard summary worksheet

Operational
Definition
(Define the
Indicator indicator in
Name very specific
(Be sure to terms.
indicate Provide the
whether it numerator Data
is a count, and the Collection Plan
percentage, denominator ■ Frequency
rate, days if a ■ Location
between, percentage ■ Stratification Short-Term Long-Term
Concept etc.) or rate.) ■ Sampling Target Goal

Concept #1 Indicator #1

Concept #2 Indicator #2

Concept #3 Indicator #3

key indicators the team is measuring. Once you
have started to organize your indicators into a
format that can be shared easily with others,
you are well on your way to building what is
known as a dashboard or instrument panel of
strategic indicators.
▸▸ Evolution of the
Strategic Dashboard
In the United States, the growing demand for
healthcare data can be traced back to the early
1980s when a variety of external groups began
pushing for the development of healthcare report
cards. Just like the report cards we received in
school, these summaries were intended to evaluate
providers on a number of key clinical and opera-
tional indicators. Report card sponsors (usually
external oversight or regulatory bodies) felt that
these assessments were needed in order to make
prudent decisions about which providers offered
the highest quality at the lowest cost. The use of
some sort of grading system to evaluate providers
of care was also motivated by the fact that (1) most
providers were not being totally transparent about
releasing their own results, and (2) patients and
their caregivers started complaining that there
was more information available on evaluating car
purchases than there was on where they should
go for their hip or knee replacement surgery.
As Nelson, Batalden, Plume, Mihevc, and
Swartz (1995) point out, three primary groups
in the United States have led the development
of healthcare report cards: (1) purchasers
of care, (2) health service researchers, and
(3) providers of care who were anxious to show
that they were concerned about cost, service, and
quality. Today I would add two more groups to
these initial three: politicians and the media.
Even in national health systems as we find in
most other countries outside the United States
where purchasers of care (e.g., manufacturing
companies or insurance groups) play little
or no role in the healthcare system, the push
TABLE 5-2 Completed dashboard summary worksheet for the medicines management team

Indicator
Name
(Be sure to
indicate
whether it Operational Definition
is a count, (Define the indicator Data Collection Plan
percentage, in very specific terms. ■ Frequency
rate, days Provide the numerator ■ Location
between, and the denominator if a ■ Stratification & Sampling
Concept etc.) percentage or rate.) ■ Data Source Short-Tem Target Long-Term Goal

Volume The ■■ This is a simple count ■■ The data will be tracked daily. Not Applicable Not Applicable
number of the individual med ■■ This measure applies to all Volumes will Volume analysis
of med orders. inpatient units. be analyzed to will be used to
orders ■■ A med order is ■■ The data will be stratified by shift determine the determine the
received defined as a count of and by type of order (stat versus ordering patterns by maximum order
in the the written or verbal routine). hour. loads the pharmacy
inpatient orders received in ■■ Initially all medication orders can handle.
pharmacy. the pharmacy from a will be reviewed. A stratified
qualified clinician. proportional random sample will
■■ A med order will be be considered once the variation
included only if it will in the process is fully understood
be administered to an and the volume of orders is
inpatient within the analyzed.
hospital. ■■ The data will be pulled from the
pharmacy computer and the
computerized physician order
entry (CPOE) systems.
Evolution of the Strategic Dashboard

(continues)
145
 146

TABLE 5-2 Completed dashboard summary worksheet for the medicines management team (continued)

Indicator
Concept Name Operational Definition Data Collection Plan Short-Tem Target Long-Term Goal

Patient Percentage Numerator: Number of ■■ The data will be tracked daily and Once the baseline The long-term
Safety of inpatient medication grouped by week. data are analyzed a goal is to have
inpatient orders with one or more ■■ This measure applies to all target will be set. 0 inpatient med
medication errors. An error is defined inpatient units. errors.
orders as: wrong med, wrong ■■ The data will be stratified by shift The probability of
with an dose, wrong route, or and by type of order (stat versus reaching 0 will be
error. wrong patient. routine). better understood
Denominator: Number ■■ Initially all medication orders once the baseline is
of inpatient medication will be reviewed. A stratified established.
orders received by the proportional random sample will
pharmacy. be considered once the variation
in the process is fully understood
and the volume of orders is
analyzed.
■■ The data will be pulled from the
pharmacy computer and the
Chapter 5 Organizing Indicators into a Strategic Dashboard

CPOE systems.

Efficiency Inpatient TAT is defined as the time ■■ TAT will be tracked for each ■■ Historically the The long-term goals
med order (in whole minutes) from inpatient med order and analyzed targets were: will be set once the
turnaround when the med order is daily. ■■ 60 minutes or baseline and current
time (TAT). received in the pharmacy ■■ The orders will be stratified by less for routine performance
until it is posted in the shift and type of order (stat versus orders in TAT is better
computerized reporting routine). ■■ 30 minutes or less understood.
system. ■■ A stratified proportional random for stat orders
sample of the orders will be ■■ These will be
pulled daily. reviewed after
■■ The TAT will be pulled from the the baseline is
pharmacy order entry system. obtained.

from researchers, consumers, commissioners,
political leaders, and the media for more and
more data on provider performance has also
grown in geometric proportions. This push for
more and more data on providers that can be
accessed openly and transparently provides clear
motivation for having a measurement philosophy
and knowing your own data better than anyone
else, especially external groups.
Irrespective of the country where care is
provided, the rapid growth of the Internet and
the related ability of anyone to obtain almost
instant access to information on a wide variety
of issues and topics also have been primary
drivers for the increasing demand to release
and publish healthcare process and outcome
indicators. The primary problem with thinking
that scorecards or report cards provide the
solution to the growing demand to evaluate
and rate providers, however, is that these ap-
proaches are basically designed to produce data
for judgment, not improvement. These forms of
reporting on healthcare provider performance
are usually derived from historical data that are
aggregated, static in nature, and lagged by as
much as a year or more.
Nelson et al. provide the following portrait
of the report card orientation (1995, p. 157):
The Report Card Image. . .
Who gets report cards? Students
Who gives them? Teachers
What is the focus? Past performance
Who wins? The As
Who loses? Everybody else
What’s learned? I’m above aver-
age, average, or
below average
Placed within the healthcare context, the
report card or scorecard approach is used by
both internal and external groups and organi-
zations. Those producing the report card may
involve the providers in the development and
design of the final report or they may totally
exclude the providers from the process. The
primary objective is to rate and rank (or rank
Evolution of the Strategic Dashboard 147
and yank as some refer to it) providers for their
performance on selected indicators.
The most popular display format for ratings
has become the ever popular assignment of red,
yellow (amber), or green assessments based on
performance to target or goal. Star ratings come
in a close second as a popular format. Providers
rated or ranked at or near the top (i.e., those
receiving green ratings or four or five stars) are
judged to be better performers than those further
down (i.e., those receiving yellow or red ratings
or one or two stars). It is also not uncommon for
percentiles to be used as the ranking mechanism
to determine performance. Percentile rankings
are frequently used in assessing patient or staff
satisfaction results.
All of these approaches and reporting
formats have the uncanny ability to produce
a wide variety of emotional reactions. If the
ranking process places your organization in the
best category (i.e., a green rating, the maximum
number of stars, or in the top decile), two reac-
tions typically occur: (1) unbridled joy emerges
and the marketing department includes the
results in the most recent board report or puts
advertisements in the local paper about how
high they ranked, or (2) there is complacency,
which is derived from the satisfaction that you
were not in the middle or bottom part of the
ranking schema. On the other hand, when the
rankings place your organization in the middle
of the distribution (e.g., between the 25th and
75th percentiles) some form of rationalization
is usually offered (e.g., the data are old, we have
sicker patients, we are a large academic medical
center that treats the most difficult cases, or the
demographics of the population we serve create
extreme challenges for us). Finally, when your
organization is in the bottom of the ranking with
a majority of red ratings, one star for a majority
of the indicators, or below the 25th percentile,
then fear, finger pointing, and comments such
as “someone needs to be held accountable”
are typical reactions. Deming’s cycle of fear
(Scherkenbach, 1990, p. 71) frequently can be
found when the rating approach being used to
148 Chapter 5 Organizing Indicators into a Strategic Dashboard
Increased Kill the
Fear Messenger
Filtered
Micro-
management Information
FIGURE 5-1 Deming’s cycle of fear
produce report cards or scorecards does not place
you organization in a particularly good light.
Deming’s cycle of fear is shown in FIGURE 5-1.
This cycle starts when there is growing fear
that the numbers given to management may
not reinforce what management wants to hear.
This then leads to a concern that the individual
delivering bad news (i.e., numbers that do not
fit someone’s view of reality) will be “killed.”
This fear then leads to filtering the data or the
interpretation of the data in order to report only
“good news.” The final step in this cycle of fear
is that there is increased micromanagement of
the data by the workers and of the workers by
management, which leads to increased fear and
then the cycle begins again.
Another major challenge with the report
card approach is that it does not provide any
understanding of the variation in the indica-
tors that produce the results. Without under-
standing the variation in the processes and
related outcomes it will be virtually impossible
to make improvements. More is said on this
point in Chapter 6. The final problem with the
report card approach, especially when done to
providers of care by external groups, is that the
groups providing the judgments do not say, “We
realize that you are in the bottom decile of our
most recent ranking but we will be sending 12
improvement specialists to your organization
to help you move up in the rankings.” Or, as
Nelson et al. put it (1995, 157), those passing
judgment on providers do not “help providers
know how to make improvements.” The ranking
occurs and people are merely told to “improve
your performance.” True improvement requires
not only a different approach to the collection
and analysis of data but also a different way of
thinking about what to do with data once you
have it.
The alternative to the static report card is the
development of a more dynamic performance
measurement system that enhances decision
making and encourages improvement strate-
gies. Two terms have emerged to describe this
alternative approach: the dashboard and the
instrument panel. Some writers have favored
the image of a car dashboard for this approach
whereas others have used the analogy of the
instrument panel of an airplane. Regardless
of the term, the concept that they are trying
to promote is the same—you need to look at
how you are performing right now and make
predictions about where you are headed in the
future. Report cards and scorecards are focused
on the past not the future. Don Wheeler (1993,
4), a well-known writer on statistical process
control (SPC) methods and understanding
variation, puts it this way: “Managing a process
on the basis of monthly (or quarterly) averages
is like trying to drive a car by looking in the
rear view mirror.” I think most of us would
agree that driving a car this way is a very unsafe
approach. It looks at where we have been not
where we are going. Similarly, trying to look
at last year’s or last quarter’s results, especially
with aggregated numbers or summary statis-
tics, provides no basis for predicting where the
­organization will go in the future. The dashboard
of your car, for example, is providing real-time
feedback on indicators important to the safe

performance of your car and your interaction
with the system. As you drive down the road,
you don’t say, “Gee, I wonder what my speed was
on this stretch of road when I drove on it last
month?” Or, “I wonder what my average miles
per gallon was on this highway when I drove
this way back in December?” These questions
are looking to the past and have no relevance
to your current performance. Knowledge of
your current speed or the amount of gasoline
remaining in your tank can be used to predict
if you run the risk of getting a speeding ticket
or if you will make it to your destination be-
fore running out of gas. Current knowledge
and prediction are hallmarks for the quality
measurement journey (QMJ).
According to Bader (1993), Henry ­Berman,
chief executive officer (CEO) of Group Health
Northwest (Spokane, WA), was the first to
popularize the dashboard analogy. Nelson
et al. favor the instrument panel analogy over
the dashboard: “Just like the cockpit crew of
a jet airplane need instrument panels to fly
safely, health care delivery system leaders need
instrument panels to manage wisely” (1995,
157). The characteristics of an instrument panel
are as follows”
Instrument Panel Image. . .
Who uses them? The cockpit crew
(pilot, copilot,
navigator)
Who interprets The cockpit crew
the results?
What is the focus? Present and future
performance
What is the utility? Real-time monitor-
ing, predicting the
future, and taking
action
Nelson et al. (1995, p. 158) conclude, “The
instrument panel or dashboard metaphor has
an entirely different aura from that of the report
card. It has vitality, timeliness, and a clear-cut
utility that is absent from report card thinking.
Evolution of the Strategic Dashboard 149
A key feature is providing critical, real-time
information to the user to prompt wise decisions
and—if need be—make rapid midcourse cor-
rections.” The decision to create an instrument
panel or dashboard rather than a report card or
scorecard is based on much more than seman-
tics. Although there are obvious differences in
the intent of each approach (judgment versus
improvement) there are also major differences in
the ways in which data are collected, tabulated,
and displayed.
Robert Kaplan and David Norton have
developed a third concept to describe the orga-
nization of key strategic indicators. They have
coined the term the “balanced scorecard” as their
organizing rubric. In 1992, they published the
first of a series of articles defining the balanced
scorecard, its components, and how they envision
its use. Subsequent articles provided examples
of companies applying their ideas (1993) and
how their framework can be used to create a
strategic management system (1996). Although
the terminology is different from that used by
Nelson et al., Kaplan and Norton also see the
balanced scorecard as “the dials and indicators
in an airplane cockpit.” They continue their
analogy by stating, “Reliance on one instru-
ment can be fatal. Similarly the complexity
of managing an organization today requires
that managers be able to view performance
in several areas simultaneously” (Kaplan and
Norton, 1992, p. 72)1
In their first article (1992), Kaplan and Norton
identified the following benefits of developing
a balanced scorecard:
■■ It brings together, in a single management
report, many of the seemingly disparate
elements of an organization’s strategic agenda.
■■ It helps to reduce information overload,
by focusing on the “vital few” indicators.
■■ It helps to guard against suboptimization
by forcing senior managers to consider
all the important measures together and
lets them see whether improvement in
150 Chapter 5 Organizing Indicators into a Strategic Dashboard
one area may be achieved at the expense
of another.
■■ It puts strategy and vision, rather than
control, at the center of an organization’s
effort.
■■ It is based on an understanding of interre-
lationships between functions, not on the
performance of individual functions or units.
■■ It provides an opportunity for organizational
learning at the executive level.
Nugent et al. (1994) was one of the first
clinicians to actually apply the instrument panel
notion to a healthcare situation. He and his col-
leagues describe the creation of an instrument
panel to monitor and improve coronary artery
bypass graft (CABG) surgery. In this classic
article, they walk the reader through the selec-
tion and definition of key indicators for CABG
surgery and then show how control charts can
be used to understand the variation that lives
within the process. Although the authors focus
on the details of how to create a dashboard, one
of the side benefits of the article is to show how
the instrument panel concept can be applied to
a very specific clinical procedure and its related
outcomes.
In 2000, the Health Care Advisory Board
published one of the first comprehensive
reports on healthcare dashboards. . This
document, CEO Dashboards: Performance
Metrics for the New Health Care Economy, has
emerged as a general guide for the develop-
ment of healthcare dashboards. It addresses
the following issues:
■■ The problem of inadequate performance
measurement
■■ Elements of an effective dashboard
■■ Recommendations for rapid dashboard
development
■■ Dashboards of leading hospitals and health
systems
Based on their research, the Health Care
Advisory Board identified four key elements of
an effective dashboard:
■■ Building a dashboard around a balanced
set of performance measures
■■ Selecting a fairly austere set of measures
(i.e., keeping it simple by selecting the vital
few measures, usually 15–30)
■■ Presenting data in graphic displays (rather
than tabular formats)
■■ Developing action triggers (i.e., setting targets
and goals that trigger the need for action)
They then proceeded to summarize the
leading categories for organizing healthcare
dashboards:
■■ Financial performance
■■ Operational effectiveness/efficiency
■■ Quality (clinical and service quality)
■■ Satisfaction (patient and family as well as
employee and physician satisfaction)
Examples of specific indicators that char-
acterize each of these four categories are also
provided in the Health Care Advisory Board
document. The report concludes with actual
examples of dashboards from health systems,
stand-alone hospitals and academic medical
centers.
Designed correctly, the dashboard (or in-
strument panel) can be used to meet not only
internal management needs but also those in-
creasing demands made on providers by external
groups interested in building report cards. If an
organization builds what Caldwell (1995) refers
to as a “strategic measurement deployment
matrix,” it will be able to measure key indicators
at various levels within the organization and
then have the capability to roll these measures
up into overall summary measures that also
satisfy external requirements. Dashboards and
instrument panels can be used to navigate cor-
rectly at the board and senior management level

(macrolevel), the departmental or meso level
and finally at the point where care is delivered
(microlevel). The choice in how you structure
your dashboard(s) is up to you.
▸▸ Focusing on the
Vital Few
One of the major challenges with creating a
dashboard is parsimony (what the Health Care
Advisory Board calls austerity). I have seen many
organizations create a dashboard that contains 50
to 80 indicators or more. A dashboard with this
many indicators is counter to what the concept
of the dashboard is trying to achieve. Again,
consider the dashboard of your car. There are
a few vital indicators that you rely on regularly,
most notably the speedometer and the fuel
gauge. After these two indicators, the rest of
the dashboard gauges may (or may not) be of
importance to you. You might pay some attention
to the temperature gauge but you probably do
not give much consideration to your electrical
system status indicators or the accumulating
mileage indicator (a.k.a., the odometer). Then
there is the tachometer. This is probably the most
useless indicator on your dashboard, especially if
you are driving an automatic transmission vehi-
cle. The point to be made here is that regardless
of the context, cars or healthcare, there are in-
dicators that we have access to or even review
regularly that frequently add little value to our
decision making processes or knowledge.
The Pareto principle provides a good
conceptual and organizational framework for
selecting dashboard indicators. The Pareto
principle (named after the Italian economist
Vilfredo Pareto, 1848–1923) basically states that
for any event (or problem) a small number of
factors will account for a majority of the reasons
why the event (or problem) occurred. Out of
the Pareto principle emerged the 80/20 concept
Focusing on the Vital Few 151
(i.e., 80% of a problem can be attributable to
20% of the causes). The Pareto diagram (Graham
and Cleary, 2000; Plsek and Onnias, 1989) is
used most often to prioritize opportunities for
QI teams (Carey and Lloyd, 2001). Its primary
goal is to identify the vital few and separate
these causal factors from the common or trivial
many (the things that do not matter).
The Pareto principle applies very well,
however, to the dashboard concept. Dashboards
should be populated with the vital few not the
trivial many. Yet all too often organizations de-
cide that nearly everything they measure needs
to be placed on the dashboard. In this case,
their dashboard looks more like the dashboard
(instrument panel) of a space shuttle or a 747
jumbo jet. Although you may not be able to
achieve the simplicity of a moped’s dashboard that
has about three indicators, you certainly do not
need the complexity of the space shuttle. From
my perspective, I believe that a parsimonious
dashboard should consist of a maximum of 10
to 15 indicators. Note, however, that depending
on whether this is a macro-, meso-, or microlevel
dashboard the 10 to 15 indicators being tracked
may not all be the same indicator at each level.
If you are interested in obtaining examples
of how many indicators can or should be on a
dashboard, I would suggest that you obtain a
copy of the Health Care Advisory Board’s CEO
Dashboard report (2000) and then contact
several of the organizations listed in the report.
These organizations can be used as a basis for
a healthy dialogue on both the contents of
your dashboard and the data display options
for you to consider. But, any external example
you use as a reference should not be accepted
automatically as the preferred template for a
dashboard. Each organization needs to develop
and design dashboards that not only meet their
short- and long-term strategic objectives but
also, and most important, fit with the organi-
zation’s culture.
152 Chapter 5 Organizing Indicators into a Strategic Dashboard

CASE STUDY #1: East London National Health Service

(NHS) Foundation Trust’s Strategic Dashboard 2
The East London NHS Foundation Trust (ELFT) is one of the largest providers of mental health and
social care services in England (https://www.elft.nhs.uk/). Originally formed in 2000, ELFT has long
been recognized as a center of excellence for mental health care, innovation, and improvement. The
Trust was first established as a mental health trust to cover East London but it has since broadened its
service area to include its core area of City of London, Hackney, Newham, and Tower Hamlets. In April
2015, the ELFT also encompassed the Bedfordshire and Luton service area.
East London’s quality journey began in earnest in 2014 when they became a strategic partner of
the Institute for Healthcare Improvement (IHI). We collaborated with the Trust to help them identify
strategic objectives for quality and safety, outline tactical plans for building capacity and capability
throughout the Trust, and develop key indicators to track progress toward its goals. One of the
objectives during the first year of our partnership was to develop a strategic dashboard that could be
shared with the board, the staff, and the public. Two overarching organizational aims were established:
(1) reducing harm and (2) right care, right place, right time. Their high-level causal models and the
factors that drive these two key outcomes are shown in FIGURES 5-2 and 5-3. Building upon these two
central aims, the ELFT quality team worked with senior leaders and the board to build their strategic
dashboard. Four types of indicators were identified as the major dimensions of the dashboard:
■■ Safety
■■ Clinical Effectiveness
■■ Patient Experience
■■ Staff Experience
The specific indicators for each of these dimensions are shown in FIGURES 5-4 through 5-7. At
this point, it is not important to delve into the details of each dimension and its related indicators. The
key point is that the ELFT leaders established aims and then developed a measurement strategy and
related indicators that captured and operationalized these aims. They track the performance of these
indicators each month and everyone from the board to the frontline staff get to review and comment
on the progress that is being made. I’d highly recommend that your organization should engage in a
dialogue that creates a similar environment that will allow you to embark on your own QMJ.

Falls Reliable Reducing

delivery of delays and
evidence-based inefficiencies in
care the system
Medication Pressure
errors ulcers
Improving Improved
patient and Right care, access to
carer right place, services at the
Physical Reducing experience right time right location
Restraints
violence Harm

FIGURE 5-2 East London NHS Foundation Trust FIGURE 5-3 East London NHS Foundation Trust
reducing harm diagram right care, right place, right time diagram
East London NHS Foundation Trust East London NHS Foundation Trust

(continues)
 0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
2
3
4
5
6
7
8
9
10
20
25
30
35
40
45
50
% Caseload Contacted In Month
% Bed Occupancy
Days
Apr 11 Apr 11 Apr 11

80%
82%
84%
46%
88%
90%
92%
94%
96%
May 11 May 11 May 11
Jun-13 Jun 11 Jun 11 Jun 11

24
29
34
39
44
49
60%
65%
70%
75%
80%
85%
90%
95%
100%
105%
Jul 11 Jul 11 Jul 11
Jul-13 Aug 11 Aug 11 Aug 11
Apr-14 May 12 Aug-13 Sep 11 Sep 11 Sep 11
Oct 11 Oct 11 Oct 11
Jun 12 Sep-13 Nov 11 Nov 11 Nov 11
May-14 Jul 12 Dec 11 Dec 11 Dec 11
Oct-13 Jan 12 Jan 12 Jan 12
Aug 12 Feb 12 Feb 12 Feb 12
Nov-13 Mar 12 Mar 12 Mar 12
Jun-14 Sep 12 Apr 12 Apr 12 Apr 12
Dec-13 May 12 May 12 May 12
Oct 12 Jan-14 Jun 12 Jun 12 Jun 12
Jul-14 Jul 12 Jul 12 Jul 12
Nov 12 Feb-14 Aug 12 Aug 12 Aug 12
Sep 12 Sep 12 Sep 12
Dec 12 Mar-14 Oct 12 Oct 12 Oct 12
Aug-14 Jan 13 Nov 12 Nov 12 Nov 12
Apr-14 Dec 12 Dec 12 Dec 12
Feb 13 Jan 13 Jan 13 Jan 13
Sep-14 May-14 Feb 13 Feb 13 Feb 13
Mar 13 Jun-14 Mar 13 Mar 13 Mar 13
Apr 13 Apr 13 Apr 13
Apr 13 Jul-14 May 13 May 13 May 13
Oct-14 Jun 13 Jun 13 Jun 13
May 13 Aug-14 Jul 13 Jul 13 Jul 13
Jun 13 Aug 13 Aug 13 Aug 13
Nov-14 Sep-14 Sep 13 Sep 13 Sep 13
Jul 13 Oct 13 Oct 13 Oct 13

East London NHS Foundation Trust

East London NHS Foundation Trust
Oct-14 Nov 13 Nov 13 Nov 13
Aug 13 Nov-14 Dec 13 Dec 13 Dec 13
Dec-14 Jan 14 Jan 14 Jan 14
Sep 13 Dec-14 Feb 14 Feb 14 Feb 14
Mar 14 Mar 14 Mar 14
Jan-15 Oct 13 Jan-15 Apr 14 Apr 14 Apr 14
May 14 May 14 May 14
Nov 13 Feb-15 Jun 14 Jun 14 Jun 14
Dec 13 Mar-15 Jul 14 Jul 14 Jul 14
Feb-15 Aug 14 Aug 14 Aug 14
Jan 14 Apr-15 Sep 14 Sep 14 Sep 14
Oct 14 Oct 14 Oct 14
Mar-15 Feb 14 May-15 Nov 14 Nov 14 Nov 14
Incidents Reported per 1,000 Bed Days u-Chart

Dec 14 Dec 14 Dec 14

Mar 14 Jun-15 Jan 15 Jan 15 Jan 15

Falls Reported per 1,000 Occupied Bed Days u-Chart

Feb 15 Feb 15 Feb 15
Apr-15 Apr 14 Jul-15 Mar 15 Mar 15 Mar 15
Apr 15 Apr 15 Apr 15

Episodes of Restraint per 1,000 Occupied Bed Days u-Chart

CPA Caseload Contacted In Month Percentage i-Chart

May 14 Aug-15 May 15
May 15 May 15
May-15 Jun 14 Sep-15 Jun 15 Jun 15 Jun 15
Jul 15 Jul 15 Jul 15
Jul 14 Oct-15 Aug 15 Aug 15 Aug 15
Sep 15 Sep 15 Sep 15
Jun-15 Aug 14 Nov-15 Oct 15 Oct 15 Oct 15
Dec-15 Nov 15 Nov 15 Nov 15
Sep 14 Dec 15 Dec 15 Dec 15
Jul-15 Jan-16 Jan 16 Jan 16 Jan 16
Oct 14 Feb 16 Feb 16 Feb 16
Nov 14 Feb-16 Mar 16 Mar 16 Mar 16
Aug-15 Dec 14 Mar-16

0.0
0.2
0.4
0.6
0.8
1.0
1.2
0
1
2
3
4
5
6
0.0
0.2
0.4
0.6
0.8
1.0
1.2

Jan 15
Sep-15 Feb 15 Apr 11 Apr 11
May 11 May 11 Oct 12

Adult Acute Mental Health Occupancy i-Chart

Oct-15 Mar 15 Jun 11 Jun 11 Nov 12
Apr 15 Jul 11 Jul 11 Dec 12

8.0%
9.0%
10.0%
11.0%
12.0%
13.0%
14.0%
Aug 11 Aug 11
May 15 Sep 11 Sep 11 Jan 13
Nov-15 Aug 12 Oct 11 Oct 11 Feb 13
Jun 15 Sep 12 Nov 11 Nov 11
Dec 11 Dec 11 Mar 13
Oct 12 Jan 12 Jan 12
Dec-15 Jul 15 Nov 12 Feb 12
Apr 13
Feb 12
Aug 15 Dec 12 Mar 12 Mar 12 May 13
Apr 12

Adult CMHTs Days Waited until First Face to Face Contact i-Chart
Jan 13 Apr 12 Jun 13
Jan-16 Sep 15 May 12 May 12
Feb 13 Jun 12 Jun 12 Jul 13
Oct 15 Mar 13 Jul 12 Jul 12 Aug 13
Apr 13 Aug 12 Aug 12 Sep 13
Feb-16 Nov 15 May 13 Sep 12 Sep 12
Oct 12 Oct 12 Oct 13
Dec 15 Jun 13 Nov 12 Nov 12 Nov 13
Jan 16 Jul 13 Dec 12 Dec 12
Mar-16 Aug 13 Jan 13 Jan 13 Dec 13
Feb 16 Feb 13 Feb 13 Jan 14
Sep 13 Mar 13 Mar 13
Oct 13 Apr 13 Apr 13 Feb 14
Mar 16 Nov 13 May 13 May 13 Mar 14
Dec 13 Jun 13 Jun 13
Jul 13 Jul 13 Apr 14
Jan 14 Aug 13 Aug 13 May 14
Feb 14 Sep 13 Sep 13 Jun 14
Mar 14 Oct 13 Oct 13
Nov 13 Nov 13 Jul 14
Days Apr 14 Dec 13 Dec 13
Days May 14 Jan 14 Aug 14
Jan 14
Jun 14 Feb 14 Feb 14 Sep 14
Jul 14 Mar 14 Mar 14 Oct 14

24
25
26
27
28
29
30
Apr 14 Apr 14

26
31
36
41
46
51
56
61
66
71
Aug 14 May 14 May 14 Nov 14

DNA Rates p-Chart

Jun 12 Sep 14 Jun 14 Jun 14 Dec 14
Apr-14 Oct 14 Jul 14 Jul 14
Aug 14 Aug 14 Jan 15
Jul 12 Nov 14 Sep 14 Sep 14 Feb 15
Dec 14 Oct 14 Oct 14
May-14 Aug 12 Nov 14 Nov 14 Mar 15
Jan 15
Dec 14 Dec 14 Apr 15
Falls Resulting in Harm / 1,000 Bed Days u-Chart

Sep 12 Feb 15 Jan 15

Jan 15 May 15
Mar 15 Feb 15 Feb 15
Jun-14 Oct 12 Apr 15 Mar 15 Mar 15 Jun 15
Serious Incident per 1,000 Occupied Bed Days u-Chart

Nov 12 May 15 Apr 15 Apr 15 Jul 15

May 15 May 15
Jun 15 Jun 15 Jun 15 Aug 15
Restraints in prone position per 1,000 Occupied Bed Days u-Chart

Jul-14 Dec 12 Jul 15 Jul 15 Jul 15 Sep 15

Jan 13 Aug 15 Aug 15 Aug 15
Sep 15 Sep 15 Oct 15
Sep 15 Oct 15
Aug-14 Feb 13 Oct 15 Nov 15
Oct 15 Nov 15 Nov 15
Nov 15 Dec 15 Dec 15 Dec 15
Mar 13 Dec 15 Jan 16 Jan 16 Jan 16
Sep-14 Feb 16 Feb 16 Feb 16
Apr 13 Jan 16 Mar 16 Mar 16
May 13 Feb 16 Mar 16
Mar 16

FIGURE 5-4 East London NHS Foundation Trust safety dashboard

Oct-14 Jun 13
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6

2
3
4
5
6
7
8
9
10
11
12

Jul 13
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9

Nov-14 Aug 13 Apr 11 Nov 12

Sep 13 Apr 12 May 11 Dec 12
Dec-14 N May 12 Jun 11

0
5
10
15
20
25
30
35
Oct 13 ov Jun 12 Jul 11 Jan 13
-1 Aug 11
4 Jul 12 Feb 13
Nov 13 Aug 12
Sep 11
Jan-15 D Oct 11 Mar 13
Dec 13 ec
-1 Sep 12 Nov 11 Apr 13
Jan 14 4 Oct 12 Dec 11
Jan 12 May 13
Feb-15 Nov 12
Feb 14 Ja
Dec 12 Feb 12 Jun 13
n-1 Mar 12
5 Jan 13 Apr 12 Jul 13
Mar-15 Mar 14 Feb 13 May 12 Aug 13
Apr 14 Fe Mar 13 Jun 12 Sep 13
b-1
Apr 13 Jul 12
May 14 5 Aug 12 Oct 13
Apr-15 May 13 Sep 12
Jun 14 M Jun 13 Oct 12 Nov 13
ar-
15 Jul 13 Nov 12 Dec 13
May-15 Jul 14 Aug 13 Dec 12 Jan 14
(NHS) Foundation Trust’s Strategic Dashboard

Sep 13 Jan 13
Aug 14 Ap Feb 13 Feb 14
r-1 Oct 13 Mar 13
Jun-15 Sep 14 5 Nov 13 Mar 14

FIGURE 5-5 East London NHS Foundation Trust clinical effectiveness dashboard
Apr 13
M Dec 13 May 13 Apr 14
Oct 14 ay Jan 14 Jun 13 May 14
-1 Jul 13
Jul-15 Nov 14 5 Feb 14 Jun 14
Aug 13
Dec 14 Ju Mar 14 Sep 13 Jul 14
n-1 Apr 14 Oct 13
Aug-15 Jan 15 5 May 14 Nov 13 Aug 14
Jun 14 Dec 13 Sep 14
Feb 15 Ju Jul 14 Jan 14 Oct 14
Sep-15 l-1 Feb 14
Mar 15 5 Aug 14 Mar 14 Nov 14
Sep 14 Apr 14
Occupied Bed Days u-Chart

Dec 14

Adult Acute Mental Health Length of Stay i-Chart

Apr 15 Au Oct 14 May 14
Oct-15 g-1
Nov 14 Jun 14 Jan 15
May 15 5 Jul 14 Feb 15
Dec 14 Aug 14
Jun 15 Se Jan 15 Sep 14 Mar 15

CAMHS Days Waited until First Face to Face Contact i-Chart

Nov-15 Jul 15 p-1 Feb 15 Oct 14 Apr 15
5
(Registered and Unregistered Staff) c-Chart

Mar 15 Nov 14
Aug 15 Dec 14 May 15
O Apr 15 Jan 15
Dec-15 ct- May 15 Jun 15
Sep 15 Feb 15
Grade 3 & 4 Pressure Ulcer Originating at ELFT per 1,000

15
Jun 15 Jul 15
Unexpected Deaths per 1,000 Occupied Bed Days u-Chart

Mar 15
Oct 15 N Jul 15 Apr 15 Aug 15
Jan-16 ov Aug 15 May 15
Nov 15 -1 Jun 15 Sep 15
5 Sep 15
Instances of Variation between Planned and Actual Staffing Levels

Jul 15
Focusing on the Vital Few

Oct 15
Oct 15
CASE STUDY #1: East London National Health Service

Feb-16 Dec 15 D Aug 15

Nov 15
ec Nov 15 Sep 15
Jan 16 -1 Dec 15 Oct 15 Dec 15
5
Reported incidents of Physical Violence per 1,000 Occupied Bed Days u-Chart

Feb 16 Jan 16 Nov 15 Jan 16

Mar-16 Ja Dec 15
n-1 Feb 16 Jan 16 Feb 16
Mar 16 6 Mar 16 Feb 16 Mar 16
Mar 16
(continued)
153

(continues)
 154

13
23
33
43
53
63
73
70%
75%
80%
85%
90%
95%
100%

0
1
2
3
4
5
6
7
8
9
2%
3%
4%
5%
6%
Nov 12 Apr 11
Nov 12 Jun-13
Dec 12 Dec 12 Jun 11
Jul-13
Jan 13 Jan 13 Aug 11 Aug-13
Feb 13 Feb 13
Oct 11 Sep-13
Mar 13 Mar 13
Apr 13 Dec 11 Oct-13
Apr 13
May 13 May 13 Feb 12 Nov-13
Jun 13 Jun 13 Apr 12 Dec-13
Jul 13 Jul 13 Jan-14
Aug 13 Jun 12
Aug 13 Feb-14
Sep 13 Aug 12
Sep 13 Mar-14
Oct 13 Oct 12
Nov 13 Oct 13 Apr-14
Dec 13 Nov 13 Dec 12 May-14
Jan 14 Dec 13 Feb 13 Jun-14
Feb 14 Jan 14 Apr 13 Jul-14
Mar 14 Feb 14 Aug-14
Jun 13
Apr 14 Mar 14 Sep-14
May 14 Aug 13
Apr 14 Oct-14
Jun 14 Oct 13
May 14

East London NHS Foundation Trust

East London NHS Foundation Trust
Jul 14 Nov-14
Jun 14 Dec 13

Period
Aug 14 Dec-14
Sep 14 Jul 14 Feb 14

Complaints c-Chart
Jan-15
Oct 14 Aug 14
Apr 14 Feb-15
Nov 14 Sep 14

Physical Attacks on Staff

Jun 14 Mar-15
Dec 14 Oct 14
Jan 15 Aug 14 Apr-15 understandable answer, involvement in care.)
Nov 14
Community Health Newham PREMs

Feb 15 Oct 14 May-15

Dec 14

Sickness and Absence Levels i-Chart

per 1,000 Occupied Bed Days u-Chart

Mar 15 Jan 15 Jun-15
Dec 14
Apr 15 Feb 15 Jul-15
May 15 Feb 15
Mar 15 Aug-15
Jun 15 Apr 15
Apr 15 Sep-15
Jul 15
May 15 Jun 15 Oct-15
Aug 15
Sep 15 Jun 15 Aug 15 Nov-15
An average across 5 responses (Patients giving a “Yes - definitely” answer to questions
covering confidence in the consultant, respect and dignity, understandable information,

Oct 15 Jul 15 Oct 15 Dec-15

Nov 15 Aug 15 Jan-16
Dec 15
Dec 15 Sep 15 Feb-16
Jan 16 Feb 16
Oct 15 Mar-16
Feb 16 Nov 15
Mar 16
Dec 15

0.0
1.0
2.0
3.0
4.0
5.0
6.0

0
5
10
15
20
25
Apr 11

0.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Apr 11

0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
May 11 May 11
Nov 12 Jun 11 Jun 11
Dec 12 Nov 12 Nov 12 Jul 11 Jul 11
Dec 12 Aug 11 Aug 11
Jan 13 Dec 12 Sep 11
Jan 13 Sep 11
Feb 13 Jan 13 Oct 11 Oct 11
Feb 13 Feb 13 Nov 11 Nov 11
Mar 13 Dec 11 Dec 11
Mar 13 Mar 13 Jan 12
Apr 13 Jan 12
Apr 13 Apr 13 Feb 12 Feb 12
May 13 May 13 May 13 Mar 12 Mar 12
Apr 12 Apr 12
Jun 13 Jun 13 Jun 13 May 12 May 12
Jul 13 Jul 13 Jul 13 Jun 12 Jun 12
Jul 12 Jul 12
Aug 13 Aug 13 Aug 13 Aug 12 Aug 12
Sep 13 Sep 13 Sep 13 Sep 12 Sep 12
Oct 13 Oct 12 Oct 12
Oct 13 Oct 13 Nov 12 Nov 12
Nov 13 Nov 13 Nov 13 Dec 12 Dec 12
Dec 13 Jan 13 Jan 13
Dec 13 Dec 13 Feb 13 Feb 13
Jan 14 Jan 14 Mar 13 Mar 13
Jan 14 Feb 14 Apr 13
Feb 14 Apr 13
Feb 14 Mar 14 May 13 May 13
Mar 14 Mar 14 Jun 13 Jun 13
Apr 14 Jul 13 Jul 13
Apr 14 May 14
Apr 14 Aug 13 Aug 13
Jun 14 May 14 Sep 13 Sep 13
May 14 Oct 13 Oct 13
Jul 14 Jun 14 Nov 13
Jun 14 Nov 13
Aug 14 Jul 14 Dec 13 Dec 13
Jul 14 Jan 14

Period
Sep 14 Aug 14 Jan 14
Aug 14 Feb 14 Feb 14
Compliments c-Chart

Oct 14 Sep 14 Mar 14 Mar 14

Sep 14 Oct 14 Apr 14 Apr 14
Nov 14 May 14
Oct 14 May 14

Vacancy Rates p-Chart

Dec 14 Nov 14 Jun 14 Jun 14

Staff Leaving Employment

Nov 14 Jan 15 Dec 14 Jul 14 Jul 14
Aug 14 Aug 14
Dec 14 Feb 15 Jan 15 Sep 14
Sep 14

per 1,000 Occupied Bed Days u-Chart

Jan 15 Mar 15 Feb 15 Oct 14 Oct 14
Nov 14

per 1,000 Occupied Bed Days u-Chart

Health and Safety Incidents Involving Staff

Feb 15 Apr 15 Mar 15 Nov 14 Dec 14
May 15 Dec 14 Jan 15
Mar 15 Apr 15 Jan 15
Jun 15 Feb 15
PALs Enquiries Per 1000 Occupied Bed Days u-Chart

Apr 15 May 15 Feb 15 Mar 15

Jul 15 Mar 15 Apr 15
May 15 Jun 15 Apr 15
Aug 15 May 15
Jun 15 Jul 15 May 15 Jun 15
Sep 15 Aug 15 Jun 15 Jul 15
Jul 15 Oct 15 Jul 15 Aug 15
Aug 15 Sep 15 Aug 15 Sep 15
Nov 15 Sep 15 Oct 15
Sep 15 Oct 15 Oct 15
Dec 15 Nov 15
Nov 15 Nov 15 Dec 15
Oct 15 Jan 16 Dec 15
Dec 15 Jan 16
Nov 15 Feb 16 Jan 16 Feb 16
Mar 16 Jan 16 Feb 16 Mar 16
Dec 15 Feb 16 Mar 16
Jan 16 Mar 16
Feb 16
Chapter 5 Organizing Indicators into a Strategic Dashboard

30%
40%
50%
60%
70%
80%
90%
100%

0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%

0.0
0.2
0.4
0.6
0.8
1.0
1.2

0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
May 12
Nov 12 Jun 12 Apr-15
Nov 12 Jul 12
Dec 12 Dec 12
Aug 12

FIGURE 5-7 East London NHS Foundation Trust staff experience dashboard
Jan 13 Jan 13
Sep 12
Feb 13 Feb 13 Oct 12 May-15
Mar 13 Mar 13 Nov 12
Apr 13 Apr 13 Dec 12
FIGURE 5-6 East London NHS Foundation Trust patient experience dashboard

May 13 May 13 Jan 13

Jun 13 Jun 13 Feb 13 Jun-15
Jul 13 Jul 13 Mar 13
Aug 13 Aug 13 Apr 13
Sep 13 Sep 13 May 13
Jul-15
(NHS) Foundation Trust’s Strategic Dashboard

Oct 13 Jun 13
Oct 13 Jul 13
Nov 13 Nov 13
Dec 13
Aug 13
Dec 13 Sep 13
Jan 14 Jan 14 Aug-15
Oct 13
Feb 14 Nov 13
Feb 14
Mar 14 Dec 13
Mar 14
Apr 14 Jan 14
May 14
Apr 14 Sep-15
May 14 Feb 14
Jun 14 Mar 14
Jul 14 Jun 14
Apr 14
Month

Aug 14 Jul 14 May 14

Sep 14 Aug 14 Jun 14 Oct-15
Sep 14
Friends and Family

Oct 14 Jul 14
Nov 14 Oct 14 Aug 14
Dec 14 Nov 14 Sep 14
Oct 14 Nov-15
Jan 15 Dec 14
Feb 15 Jan 15 Nov 14
Mar 15 Feb 15 Dec 14
Jan 15
per 1,000 Occupied Bed Days u-Chart

Apr 15
Complaints Closed Within 25 Days p-Chart

Mar 15 Feb 15 Dec-15

May 15 Apr 15
Staff Leaving Employment Within 12 months

Mar 15

Harm/Injury per 1,000 Occupied Bed Days u-Chart

Jun 15 May 15 Apr 15
Proportion of Extremely Likely Responses p-Chart

Jul 15 Health and Safety Incidents Involving Staff Resulting in Jun 15 May 15
CASE STUDY #1: East London National Health Service

Aug 15 Jun 15
Jan-16
Jul 15
Complaints still open, withdrawn and those with agreed extension excluded

Sep 15 Jul 15
Aug 15
Oct 15 Aug 15
Sep 15
Nov 15 Sep 15
Dec 15 Oct 15 Feb-16
Oct 15
Jan 16 Nov 15
Nov 15
Feb 16 Dec 15 Dec 15
Mar 16 Jan 16 Jan 16
Feb 16 Mar-16
Feb 16
Mar 16
(continued)

▸▸ The Role of
Benchmarking
Once you have defined a specific indicator,
collected data, and analyzed the variation in
the data you are ready to find out just how
good your performance really is. (See Chapters
6 to 9 for discussions on variation and how to
analyze it.) This can be accomplished by using
comparative reference data (e.g., a national or
regional norm) or by benchmarking. Unfor-
tunately there seems to be considerable mis-
understanding within the healthcare industry
on the nature and intent of benchmarking. The
simple message about benchmarking is that it
is not about numbers! It is about the search for
excellence. It is about the transformation of
the organization’s culture and the way it does
business. Benchmarking is about understanding
the relationships between structures, processes,
culture, and outcomes. Unfortunately, I have
seen too many healthcare professionals think
that benchmarking is strictly about meeting
a numeric target or goal (i.e., a benchmark).
Before the month is over, I bet that you will hear
someone say, “Here is the benchmark we need
to hit by the end of the next quarter.” When
you hear a statement like this you are basically
hearing confusion between the concepts of
a benchmark (a noun) and benchmarking (a
verb). In cases like this, the confusion goes even
deeper because it usually involves confusion
over the relationship of targets and goals to
benchmarks and the benchmarking process.
This confusion was highlighted back in 2003 in
a newsletter published by the Mihalik Group
(2003). In a brief article titled “When Is a Per-
formance Goal a Benchmarking?” they make
the following conclusions:
Occasionally there is some confusion
over the correct use of the term “bench-
mark.” A benchmark is a measure of
best performance against which an
organization’s performance is compared.
A benchmark, however, is never derived
The Role of Benchmarking 155
from average or aggregate performance.
Because it represents the best, a bench-
mark must refer to the performance of
only one organization. (p. 2)
Too often I have seen organizations use an
average or aggregated number as THE bench-
mark. It not only flies in the face of what good
benchmarking is supposed to achieve but it
also leads to confusion within the organization.
I believe there is a real need within the
healthcare industry to clarify these concepts.
Let us start our discussion by addressing the
terms target and goal. In a general sense, targets
are short-term markers of performance that are
usually designed to be achieved over the span
of several months to a year. Goals, on the other
hand, are more long term in nature, usually in
the range of 1 to 2 years.3 A target or a goal can be
based on a benchmark (the noun) if it is derived
from an organization that is considered the “best
of the best.” Otherwise targets and goals should
be established against the current performance
and capability of the system and its related
processes. Targets and goals can certainly be
part of a benchmarking process. But if you stop
at the targets or goals that may emerge from a
benchmarking initiative and then get fixated on
them you will never achieve the potential that
benchmarking offers.
Benchmarking as a verb, which is my focus in
this section, is a way to identify and understand
best practices that enable organizations to realize
new levels of performance. It is a journey not a
destination designed to establish highly reliable
structures and processes, create a new culture
that is focused on continuous improvement
and excellence, and create the conditions that
enable a learning organization (Senge, 1990) to
emerge. Confusion over these concepts leads an
organization to accept a number, either from
an internal source or an external consultant, as
“THE benchmark.” This orientation typically
leads to a fairly singular focus on the numbers
(outcomes) without giving due consideration
to the interplay of the structures and processes
that produce the numbers. Although you will
156 Chapter 5 Organizing Indicators into a Strategic Dashboard
hear organizations claim that, “We are bench-
marking” this statement usually means that they
are hoping that they hit the benchmark metric
someone gave them but have not developed a
strategy for achieving or sustaining this ethereal
number. The result is usually confusion within
the ranks and unrealistic expectations on the
part of management and the board (especially
if the organization has paid a large fee to an
external consulting group that produced the
“benchmark numbers”).
The most widely referenced work on
­benchmarking is Robert Camp’s classic book
Benchmarking: The Search for Industry Best
­Practices that Lead to Superior Performance (1989).
Camp traces the history of benchmarking, clearly
defines what it is (and is not), outlines the steps
in the benchmarking process, and then offers a
series of examples of how successful companies
have engaged in benchmarking. In the preface
of his book, Camp introduces the Japanese term
dantotsu, which loosely translates as striving to be
the “best of the best.” Camp (1989, p. 3) points out
that “we in America have no such word, perhaps
because we always assumed we were the best.”
Essentially when an organization is clas-
sified as a “benchmark” it is being recognized
for its superior performance against which all
other performances can be judged. Although
organizations as benchmarks have been iden-
tified in other industries (e.g., the Ritz Carlton
Hotels, Motorola, L.L. Bean, Ben & Jerry’s Ice
Cream, or Toyota), there is little consensus on
which organization is consistently THE best in
health care and the standard to follow. How an
organization becomes a “benchmark” is what
benchmarking is all about. Camp (1989, p. 10)
provides the following formal definition of
benchmarking: “Benchmarking is the continuous
process of measuring products, services, and
practices against the toughest competitors or
those companies recognized as industry leaders.”
He continues with a more direct working
definition of the concept: “Benchmarking is the
search for industry best practices that lead to
superior performance” (1989, p. 12). Accord-
ing to Camp, benchmarking is more about
understanding practices that lead to performance
than selecting a number to achieve. He puts this
way, “Benchmarking metrics are seen as a result
of understanding best practices, not something
that can be quantified first and understood
later” (13). Frequently in health care, however,
we do exactly what Camp cautions against. We
look at current performance as a single number
and then quickly pick a new target or goal that
is a desired end state. In many instances, the
decision process goes something like this; “We
are currently at 80% on completed histories and
physicals for outpatient surgery so next quarter
we expect it to be at 85%. No, maybe we should
pick 90%? Oh what the heck, let’s use 100% as
the target.”
A key question related to the selection of
any target or goal is, how were these numbers
decided upon in the first place? I am constantly
amazed at how we decided on new levels of
performance. Most often we seem to set targets
and goals as whole numbers divisible by 5. If
the performance last year was X, then X plus
5%, 10%, or 15% should do for next year. We do
this with budget cuts as well but in this case it is
usually reductions of 5%, 10%, or 15%. The next
time you are involved with setting new levels of
performance take a moment and listen to the
numbers that are bandied about. A majority of
the time they will be whole numbers divisible by
5. If you want to offer a provocation in a man-
agement meeting, however, instead of using the
traditional approach of whole numbers divisible by
5, propose the use of unconventional increments
for targets or goals (e.g., 4.75% or 13.837%). You
will most likely get some strange looks when you
offer numbers that are not whole and divisible by
5. I did this in a management meeting once when
there was a debate on the new targets for patient
satisfaction. The members of the team were ap-
proaching the number as if it were an item in an
auction. The exchange goes something like this:
■■ Person #1: “Let’s set the new target for the
‘Would you recommend this hospital to
your family and friends?’ question on the
inpatient survey at 80% Strongly Agree.”

■■ Person #2: “No, that seems too low to me.
Let’s go with 85%.”
■■ Person #3: “No you are both too low. Let’s
set a stretch target of 90%.”
As you can imagine, this little bidding war
continued until we had a target of, you guessed
it, 100% of the patients marking the Strongly
Agree response to the question. When the 100%
bid was hit I proposed that an alternative would
be to have a target of 73.4%. There was actually
silence for a moment then one of the members
of the senior management team asked if I was
joking. Another one said, “What are you, some
sort of wise guy?” At least I got their attention
and it got the management team thinking about
why they pick the numbers they do and if they
were achievable in the proposed time frame.
Note that the average percentage of respondents
selecting Strongly Agree for the previous 12
months was only 27%. I then proposed that an
alternative way to approach setting the target
would be to understand current performance
by placing the results from the past year on
a Shewhart (control) chart, understanding
the variation that lives in the data, evaluating
the spread of the control limits and process
capability, and finally establishing new targets
through objective statistical means. Whereas
some members of the team thought this violated
some sacred rule, such as the whole numbers
divisible by 5 rule, others thought it sounded
like a reasonable approach.
The basic problem with how many teams
and organizations approach setting targets or
goals is that little thought is generally given
to how the new level of performance will be
achieved. In most instances, this leads to what
Deming referred to as the creation of arbitrary
numeric targets and goals, which demoralize the
workers and mislead the organization. On this
point, Deming wrote, “Goals are necessary for
you and for me, but numerical goals set for other
people, without a road map to reach the goal,
have effects opposite to the effects sought” (1992,
p. 69). Benchmarking is one of the best ways to
develop a roadmap for developing appropriate
Notes 157
targets and goals as well as a strategy for achieving
new levels of performance.
Camp outlines five phases of benchmarking
and 10 related specific steps. The five phases
of benchmarking include: planning, analysis,
integration, action and maturity (Camp, 1989,
p. 17). The key to understanding these phases
and their related steps is that they have to be
seen as an ongoing process, not as something
that the organization does once and then moves
on to the next popular management solution.
The other important aspect of applying Camp’s
benchmarking framework is the recognition that
although benchmarking is a structured process
it is first and foremost a search for knowledge
and learning. Organizations looking for a silver
bullet or a desire to hit a numerical target or goal
will quickly abandon benchmarking as being too
detailed and involved. They will find it much
easier to merely issue a memo that says, “Next
year’s performance targets will be 5% higher
than this year’s! Good luck! Be productive!
Work hard!”
Notes
1. Although Kaplan and Norton continue
to make major contributions to strategic
planning and performance measurement,
the majority of their work has been di-
rected toward for-profit companies with
the objective of returning profits to stock-
holders. Their ideas are certainly relevant
to healthcare organizations (especially
their notion about creating a balanced
set of measures) but the term “scorecard”
is so similar to the concept of the report
card that it often leads to confusion about
the meaning and intent of the scorecard
concept. Is it designed for judgment or
improvement? Nelson et al. (1995) classify
the balanced scorecard as a “hybrid” of
report card and instrument panel thinking.
Personally, I believe the concept of the
dashboard or instrument panel is much
more appropriate for healthcare organiza-
tions. As stated earlier in this chapter, there
158 Chapter 5 Organizing Indicators into a Strategic Dashboard
are plenty of groups that want to develop
scorecards or report cards on healthcare
providers in order to pass judgment on
our performance. It is my belief, therefore,
that the provider community needs a more
robust set of terms and tools to guide our
improvement efforts. At the IHI, for exam-
ple, we have made a conscious decision to
use the term dashboard as the organizing
rubric for our various indicator sets. It is
much more dynamic in nature, supports
our commitment to QI, and helps man-
agement and staff feel less threatened by
performance measurement.
2. I want to express my sincere gratitude and
appreciation to Dr. Kevin Cleary, medical
director, and Dr. Amar Shah, associate
medical director, at the East London NHS
Foundation Trust for granting permission
to tell their story and share their strategic
dashboard data.
3. It is interesting to note that 10 to 15 years
ago both targets and goals in healthcare
settings were of longer durations. Targets
back then might be set for 6 months to
a couple years. Goals were set for 3–5
years. My theory is that this has happened
primarily as a result of external groups
placing greater demands on healthcare
providers to produce better outcomes
in shorter periods of time. Consumers
as well as political leaders want to see
healthcare change more quickly. So, instead
of addressing the complexity of changing
healthcare systems and process many
external groups have started to reduce the
expected time periods to achieve targets
and goals. Unfortunately the complexity
of the issues facing healthcare provid-
ers has not reduced. In many ways, the
complexity of health care has increased.
All that has been achieved by reducing
the mandated time frames for achieving
targets and goals is to frustrate both the
external reviewers and the providers.
Targets and goals need to be realistic not
demoralizing or arbitrary.
References
Bader, B. “CQI Progress Reports: The Dashboard Approach
Provides a Better Way to Keep Board Informed about
Quality.” Healthcare Executive (September-October
1993): 8–11.
Caldwell, C. Mentoring Strategic Change in Health Care.
Milwaukee, WI: ASQ Press, 1995.
Camp, R. Benchmarking: The Search for Industry Best Prac-
tices that Lead to Superior Performance. Milwaukee,
WI: ASQ Press, 1989.
Carey, R., and R. Lloyd. Measuring Quality Improvement
in Healthcare: A Guide to Statistical Process Control.
Milwaukee, WI: ASQ Press, 2001.
Deming, E. Out of the Crisis. Cambridge, MA: Massachusetts
Institute of Technology, 1992.
Graham, J., and M. Cleary, eds. Practical Tools for Con-
tinuous Improvement. Miamisburg, OH: PQ Systems,
Inc., 2000.
Health Care Advisory Board. CEO Dashboards: Performance
Metrics for the New Health Care Economy. Washington,
DC: Advisory Board Company, 2000.
Kaplan, R., and D. Norton. “The Balanced Scorecard-Measures
that Drive Performance” Harvard Business Review
(January-February 1992): 71–79.
Kaplan, R., and D. Norton. “Putting the Balanced Scorecard
to Work” Harvard Business Review (September-October
1993): 134–147.
Kaplan, R., and D. Norton. “Using the Balanced Scorecard
as a Strategic Management System” Harvard Business
Review (January-February 1996): 75–85.
Mihalik Group, LLC. “When Is a Performance Goal a
Benchmark?” The Mihalik Globe 7, no. 2 (Summer 2003).
Nelson, E., P. Batalden, S. Plume, N. Mihevc, and W. Swartz.
“Report Cards or Instrument Panels: Who Needs What?”
Journal on Quality Improvement 21, no. 4 (1995): 155–166.
Nugent, W., W. Schults, S. Plume, P. Batalden, and E. Nelson.
“Designing an Instrument Panel to Monitor and Improve
Coronary Artery Bypass Grafting.” Journal of Clinical
Outcomes Management 1 (1994): 57–64.
Plsek, P., and A. Onnias. Quality Improvement Tools. Wilton,
CT: Juran Institute, 1989.
Scherkenbach, W. The Deming Route to Quality and Produc-
tivity. Washington, DC: Ceep Press, 1990.
Senge, P. The Fifth Discipline: The Art and Practice of the Learn-
ing Organization. New York: Doubleday Currency, 1990.
Wheeler, D. Understanding Variation. Knoxville, TN: SPC
Press, 1993.
CHAPTER 6
Tapping the Knowledge
that Hides in Data
I
n the previous chapters, we focused on the
milestones in the quality measurement
journey (QMJ) that guided the identification
of indicators, the operational definitions for
these indicators, and the data collection plan(s)
needed to tap the knowledge that hides in the
data. Now we need to address the question of,
“What do you do with the data once I have it?”
The answer is really quite simple—if you are
genuinely interested in quality improvement (QI),
then you need to understand the variation that
lives within your the data. This is best achieved
by building a knowledge and skill base around
the following topics:
■■ Data versus information
■■ Static versus dynamic approaches to data
analysis
■■ Understanding variation conceptually
(common and special causes of variation)
■■ Making the appropriate management de-
cisions when presented with common and
special causes of variation
■■ Statistical process control (SPC) methods
(run charts and Shewhart charts)
■■ Linking your measurement efforts to
improvement
© Michal Steflovic/Shutterstock
This chapter addresses the first two topics.
Chapter 7 focuses on understanding variation
conceptually and making the appropriate man-
agement decision when presented with common
and special causes of variation. Chapter 8 provides
the details on run chart construction, analysis,
and interpretation, and Chapter 9 does the same
for Shewhart charts. Chapter 10 (Case Studies)
and Chapter 11 (Connecting the Dots) provide
guidance on how to link your measurement
efforts to your improvement strategies.
▸▸ Data Versus
Information
Although we live in an information age, it is
interesting to observe how many people really do
not clearly understand the difference between data
and information. I have been in many meetings
where someone says, “We need more data,” and
I am thinking, “No, you have enough data; what
you really need is more information.” Similarly,
I have heard people say, “We need more detailed
information.” What they really should have said,
159
160 Chapter 6 Tapping the Knowledge that Hides in Data
however, is “We need more detailed data.” This
issue was central to a discussion I remember
vividly on National Public Radio (NPR) back in
2003. The topic then was the creation of the new
Cabinet-level Department of Homeland Security.
The NPR announcer was interviewing various
representatives and senators on the merits of such
a department. Quickly the discussion turned to
the ability of the United States to obtain useful
“data and information” on terrorists groups. One
congressman stated that he thought we needed
more data on individuals thought to be threats
to this country. He proposed keeping all of these
data in a massive database the new department
would manage. According to one of the senators
interviewed, however, he thought that the problem
was not having enough information on various
terrorists groups. Finally, an official from one
of the bureaus that would fall under the new
department said that the central problem was
that they did not have enough people who knew
how to turn the massive amounts of data they
already have into useful information. The lack
of consensus on this data/information issue was
readily apparent even in the summary of the NPR
reporter. He kept using the two words as if they
were synonyms indicating to me that he really
did not understand the difference between the
two concepts even though this was the primary
theme of the report. Recent stories and examples
of this data/information confusion can be found
almost daily by listening to the evening news or
reading stories in the daily newspaper.
The distinction between data and information
is more than mere semantics. The two concepts
are quite different from each other. I have always
found Austin’s (1983, p. 24) distinction between
data and information to be extremely useful:
Data refers to the raw facts and figures
which are collected as part of the normal
functioning of the hospital. Informa-
tion, on the other hand, is defined as
data, which have been processed and
analyzed in a formal, intelligent way,
so that the results are directly useful
to those involved in the operation and
management of the hospital.
Data are the bits and bytes that we collect
in an effort to measure the performance of a
process. Data are not information. Information,
on the other hand, can only be produced by
submitting data to an intelligent inquiry process
that is grounded in deductive (general to the
specific) and inductive (specific to the general)
thinking. This classic approach to data-based
inquiry is referred to as the scientific method
(Lastrucci, 1967) and forms the foundation for
appreciative inquiry and improvement.
FIGURE 6-1 depicts the six key steps needed
to gather data and turn it into useful information
for decision making. The details related to each
step are highlighted here:
■■ Step 1: Theoretical Concepts. All scientific
inquiry begins with theoretical concepts
(ideas and hypotheses). These concepts
are either derived from abstract thinking
about the world and the way it works (e.g.,
theoretical physics) or they are generated
from data that enable the researchers to
propose theories based on the observed
patterns in the accumulated data. Some-
times the theories people pose are merely
variations on existing theories. For example,
the basic theories about germs and their
impacts on human populations have been
around thousands of years, yet each year
there are new and subtle modifications to
fundamental theories about germs and
their impacts on the human body (Block,
2001).1 Existing theories become modified
or refined as additional research is per-
formed, and data are placed into context of
the theoretical principles. At other times,
however, the theoretical concepts present
new thinking, which then creates the
foundation for debate. The human genome
project, for example, has spawned new
theories about human genetics that were
not considered 5 or 10 years ago. Whether
a theory has been around for a while or is
relatively new, the real test of any theory or
hypothesis lies with the empirical evidence
that can be assembled to test the validity
and reliability of the idea.

Step 1
Deductive
Theoretical
Concepts
Information
Step 5 for Decision
Making
Theory and
Prediction
Step 6 Interpretation
Data
Analysis and
Output
Step 4
FIGURE 6-1 Turing data into information
■■ Step 2: Select and Define Indicators. The
next step in turning data into information is
to select a key set of indicators that purport
to measure the theoretical concepts under
investigation and then reach consensus on
an operational definition of each indicator.2
Several key points are worth noting. First,
I believe that this is one of the major chal-
lenges healthcare professionals face when
attempting to measure what they do. Second,
not devoting sufficient time to clarifying
operational definitions usually leads to
(1) confusion over what the numbers really
mean, (2) challenges to the results, and/or
(3) a desire to “kill the messenger” because ■■
the data do not fit certain individuals’
views of reality (e.g., this is seen frequently
when a politician fires his public opinion
pollster because the results of the poll do
not match the politician’s view of herself).
Finally, improving operational definitions is
not that difficult. It is not a statistical issue
but rather one of logic and consensus. The
healthcare industry has some of the best
educated professionals in this country.
Reaching consensus on operational defini-
tions, therefore, should be relatively easy.
Data Versus Information 161
Operational
Step 2
Definitions
Measurement
and Data Step 3
Collection
Inductive
I believe the reasons we perform poorly at
this step include:
• The lack of formal training in the
­selection and specification of indicators
• Time constraints (i.e., we have to get
the data now so why waste time talking
about what the indicators mean)
• A fatal assumption that “Everyone knows
what a (fill in your favorite indicator
name) is, so why do we need to discuss
it further and waste time?”
If we do nothing more than improve our
efforts in this area, we will be making a
significant improvement in our QMJ.
Step 3: Measurement and Data Collection.
The basic principles of measurement and
data collection were reviewed in Chapter
4. Issues such as stratification, sampling,
the role of pilot tests, the duration and
frequency of data collection, respondent
and data collector bias, and data collection
methods are all critical to the success of this
step. These issues are often overlooked when
conducting both quantitative and quali-
tative studies.3 I have seen teams develop
well-defined operational definitions for their
indicators and then come to a screeching
162 Chapter 6 Tapping the Knowledge that Hides in Data
halt because they did not think through all
the details related to data collection. Data
do not magically collect themselves and
then conveniently populate a database on
their own. Physicians frequently assume
that the nurses will collect the data. The
nurses hope the unit secretaries or clerks
can collect it. The unit secretary hopes that
students will be brought on board to collect
the data. Finally, when there are no obvious
candidates available to complete this task,
someone will offer the following suggestion,
“Let’s have the hospital volunteers collect
the data.” Teams frequently underestimate
the importance of actually designating a
properly prepared person to collect the data.
Do not take this step too lightly.
■■ Step 4: Data Analysis and Output. Failure
to develop a well-thought-out data analysis
plan will lead a team to sit on their data and
hope that it will eventually hatch something
they can use. What often happens, however,
is that when the data are not subjected
to analysis, they become antiquated and
eventually obsolete. A well-designed and
executed analysis plan allows a team to move
to the next important step—­interpretation.
You should start to think about this step
early on in the QMJ and make decisions
about what will be done to analyze the
data and who will be responsible for doing
it. For example, you should have a dia-
logue about whether you have access to a
statistical package to tabulate and analyze
data and to produce graphical displays of
the results. Also, determine which type of
statistical analysis will be conducted. Will
you merely calculate the average, minimums
and maximums, and standard deviations
for the data (static approach), or will you
analyze the variation in the data using run
or control charts (dynamic approach)? If you
are focusing on data for improvement (not
judgment or research) then you will need
to use the dynamic approach. But who on
the team has knowledge of SPC methods?
■■ Step 5: Interpretation of the Results. This is
what the previous four steps were designed
to achieve. To use the analogy of a fine
dinner, the first three steps constitute the
appetizer, soup, and salad. Interpretation is
the main course. The first three steps relied
on a combination of machine power and
brainpower. Interpretation relies entirely
on the machine that sits on top of your
shoulders. Interpretation basically seeks the
answer to one simple question—why? This
is the point at which the data and the theory
are compared to each other. Do the analytic
results support the theories (hypotheses)
you initially developed? If not, are the data
correct and the theory wrong or vice versa?
This is also the point at which previous
research and data play key roles. Are the
results consistent with what others have
found? Are they consistent with what was
found when similar studies were conducted?
Interpretation discussions should involve
issues of reliability and validity (Blalock,
1971; Campbell and Stanley, 1963; Forcese
and Richer, 1970; Selltiz et al., 1959). This
is the point at which all your hard work
should pay off because it sets the stage for
the final piece of the puzzle.
■■ Step 6: Information for Decision Making.
If interpretation is the main course, then
this step represents dessert. This is what
you have been working toward (which
interestingly enough is also how many
people regard dessert). This final step is
crucial, and unfortunately many improve-
ment initiatives never reach this step. They
often end up with considerable data but no
information for decision making. I have
heard this referred to as being a DRIP (i.e.,
being data rich and information poor!). The
key to success in this final step is building a
dialogue about the data and what decisions
you will make with the results. Questions
such as those identified in Step 5 provide
the basis for a healthy dialogue designed
to build information. The dialogue should
 Static Versus Dynamic Approaches to Data Analysis
center on the data collected, the merit of the
proposed theories and concepts, the variation
found in the data, and the interpretation of
what the data mean. This dialogue should
prepare you to then develop action plans
for improvement. The Plan–Do–Study–Act
(PDSA) cycle (Deming, 1992; Gaucher
and Coffey, 1993; Langley Langley, Nolan,
Nolan, Norman, & Provost, 1996) as well
as Deming’s notion of profound knowledge
(Deming, 1992; Schultz, 1994, p. 18–27) all
identify taking action (decision making) as
the primary goal of QI. The previous four
steps are designed to bring you to this point.
QI has been labeled by some as taking too
long to demonstrate an improvement. My
experience has convinced me that when
this occurs it is usually due to the team’s
lack of knowledge of how to turn data
into information or due to a management
structure that merely wanted to have data
to confirm their view of reality. Just as we
would not proceed with knee replacement
surgery without a properly skilled and trained
surgical team, we should not proceed with
QI initiatives without a properly skilled
and trained measurement team. Making
appropriate decisions in the 21st century
requires data and information. The steps are
clear but sometimes the path is obstructed
with roadblocks.
Two final comments about Figure 6-1. First,
notice that the diagram circles around a center
block labeled theory and prediction. This block
lies at the very core of improvement science.
When I am working with a team I am constantly
asking them two simple questions: (1) What is
your theory? and (2) What is your prediction?
When a team proposes an improvement idea
these two questions should be posed. For ex-
ample, if you think putting up posters around
the hospital that promote hand hygiene my first
question would be: “What is your theory as to
why you think a poster telling people it is im-
portant to properly wash their hands will make
a difference? Of all the ideas for improvement
163
you could come up with for improving the hand
hygiene process, what theory do you have that a
poster will achieve this objective?” Then I’d ask
the follow-up question; “What is your prediction
about the success of putting up posters? Do you
predict that the poster campaign will increase
the percentage of staff who properly wash their
hands?” Now we can have a dialogue.
The second thing to note about Figure 6-1
is that it is divided into two main sections. The
lower right portion of the figure is essentially an
inductive learning process. Inductive reasoning
is the process of coming up with a conclusion
based on a series of events or data points that
yield similar findings or repeat a particular
pattern. Not to oversimplify this approach but it
can be thought of as moving from the specific to
the general (i.e., deriving general principles from
specific observed instances, facts, or data). The
other side of Figure 6-1 is deductive in nature.
Deductive learning arises from testing a theory
against a prediction (Langley et al., 1996, p. 82).
In a general sense, deductive learning goes from
general suppositions, theories, or conclusions and
seeks specific observations or data to confirm the
theory or initial premise. You can decide how much
you care to dive into the inductive-deductive area.
My point is that the scientific method (Lastrucci,
1967), which is essentially what Figure 6-1 is
depicting, is an inductive and deductive process
of thinking and learning. It not only provides the
foundation for QI and the science of improve-
ment (Perla, Provost, & Perry, 2013) but also is
essentially how the human brain approaches
problem solving situations. .
▸▸ Static Versus Dynamic
Approaches to Data
Analysis
A basic data challenge that we face in health care
is that we have historically relied on aggregated
data and summary statistics to understand the
164 Chapter 6 Tapping the Knowledge that Hides in Data
quality of what we do on a day-to day-basis.4 Such
an approach leads the researcher to a static way of
thinking about data analysis and interpretation.
For decades, we have been using methods and
tools that are best suited for efficacy research,
judgment, or accountability and trying to use
these tools to answer questions about quality and
safety. From my perspective, too many healthcare
professionals are using the wrong statistical tools
and thinking to analyze and interpret the data
they have collected.
Static approaches to data analysis focus on
using aggregated data and tend to compare the
most recent data point with a previous data point
(usually the last month or quarter). If the current
data point is regarded as being better than the
last data point, then all is right with the world.
If, on the other hand, the current data point is
determined to be worse than the previous result,
then management concludes that all is not right
with the world. If both data points are the same
number, what do you conclude?
The essential point to remember is that
aggregated data, presented in tabular formats
or with summary statistics, will not help you
measure the impact of process improvement or
redesign efforts. It is just that simple. Deming was
very clear on this point. He wrote (1992, p. 312):
Students are not warned in classes nor
in the books that for analytic purposes
(such as to improve a process – author’s
emphasis), distributions and calculations
of mean, mode, standard deviation,
chi-square, t-test, etc. serve no useful
purpose for improvement of a process
unless the data were produced in a state
of statistical control. Aggregated data,
therefore, can only lead to judgment
not to improvement.
The healthcare industry has a long and rich
history of using data for judgment. For example,
it is not uncommon for a hospital or a health
system to compare the current month’s results
on a particular indicator to those for the same
month a year ago. There is also a tendency to
compare the average for this year (e.g., length of
stay for a particular diagnosis) with the average
from last year. Although this may have some
utility for making determinations on incentive
or bonus payout programs it has very little to
do with quality. Quality is determined by the
moment-to-moment variation in a process. It
is determined by the workers who deliver care
at the bedside, at the registration desk, and at
the blood draw station. Administrators and
managers who rely on aggregated data to make
decisions, therefore, are not really focused on
quality. They are focused more on making judg-
ments about the difference between two numbers
rather than on how the numbers perform over
time. Because variation exists in all that we do,
chances are quite high that two data points will
differ. To paraphrase Deming, if you have two
data points, there is a very high probability that
one will be different from the other.5
If aggregated data and summary of statis-
tics are your primary frames of reference, then
two data points are all you will need to make a
conclusion about performance. If, for example,
the result at Time 2 is better than the result at
Time 1 then management will probably conclude
that things must be getting better. What people
fail to recognize when they do this, however, is
that two numbers do not determine quality, a
trend, or a “significant difference.” Again, if you
have two numbers it is very likely that one will
be different from the other. So, when healthcare
professionals look at the most recent data point
and determine success (or failure) in light of
the data point’s relative position to the previous
data point, they will celebrate to excess, punish
themselves beyond necessity, or take comfort in
the fact the numbers have not changed. Such an
approach to data leads organizations to reward
and punish workers for things over which they
may have no control. It leads an organization
to suffer from what I call the “good dog/bad
dog” syndrome. Caught up in this syndrome
an organization will establish incentives and
rewards that are not unlike Pavlov’s approach to
training his dogs. If the numbers are good and
 Static Versus Dynamic Approaches to Data Analysis
High
Which time period is better?
Low
Last year This year
High
Now which time period
looks better?
Low
Last year This year
FIGURE 6-2 A comparison of CABG surgery
mortality percentages
where I want to see them, I’ll reward you with a
bonus or an incentive (good dog, you salivated
on command, here have a biscuit). On the other
hand, if the numbers are not good (by whatever
criteria I use to judge good), then you do not
receive the reward (bad dog). This syndrome can
drive individuals crazy and lead organizations
down a path to ruin. It is a complex syndrome
that encompasses an organization’s views on
intrinsic and extrinsic motivation (Deming,
1992; Kohn, 1986, 1993), incentive programs,
performance reviews, compensation practices,
and data analysis.6
FIGURE 6-2 demonstrates the major differ-
ence between static and dynamic displays of
data. Imagine that your hospital is interested in
gaining a better understanding of its coronary
artery bypass graft (CABG) surgery outcomes.
Historically, you have evaluated your performance
in this area by comparing the annual percent
mortality. The top portion of Figure 6-2 depicts
how you have typically looked at these data. Note
that for the last 2 years you have had essentially
the same percent mortality. So, what do you
conclude from this comparison? The answer is
easy—you are the same this year as you were
last year. The overall average leads you to the
conclusion that nothing has changed.
165
Now look at the lower portion of Figure 6-2.
In this chart, we observe the CABG surgery
mortality percentages as they occurred over the
12 months of each year. What do we observe? We
note that last year the CABG surgery mortality
was on a constantly increasing trend upward
and that this year reveals that the percentage of
mortality has been on a downward trend. How
is it that you can have two averages that are the
essentially the same and two monthly patterns
that are fundamentally different? The answer is
simple. The top part of Figure 6-2 presents data
in a static fashion, whereas the data shown in the
lower part of the figure are depicted in a dynamic
fashion. Because quality is about efficiency and
effectiveness, we need to look at data in dynamic
rather than static fashion. This means, therefore,
that we need to plot data over time rather than
aggregate the data into summary statistics.
FIGURE 6-3 provides another example of why
static approaches to variation should not be used
for QI. Imagine that Figure 6-3 represents two
parts of the turnaround time (TAT) process for
a particular lab test. Process A (the top chart)
is the TAT from when the physician writes the
lab order until the specimen is received in the
lab and logged into the computer. Process B, on
the other hand, is the in-lab processing time. If
you showed only the aggregated distributions,
represented by the bell-shaped curves on the far
right side of the graphic, you would conclude
that the two processes are exactly the same. Both
curves have the same width and height and are
centered at the same point. In fact, these two
distributions could have the same mean, me-
dian, mode, and standard deviation. Applying
aggregated or static thinking to these data would
lead the researcher to conclude that Process A
and Process B are the same.
If we look at these data from a dynamic point
of view, however, we would arrange the lab test
in chronological order and observe the TATs as
they laid themselves out over time. These data
are shown in the main body of the chart as the
solid line that fluctuates up and down around
the centerline (average). When we look at the
166 Chapter 6 Tapping the Knowledge that Hides in Data

High

Process A

Average

Low
Static displays
of data that
have the same
Dynamic displays of data that show mean and
different underlying distributions standard
deviation
High

Average

Process B

Low

FIGURE 6-3 Static versus dynamic data displays

data in this manner we see that Process A has
shifted downward over time whereas Process
B has been migrating gradually upward. The
dashed lines imposed on each chart show these
directional patterns.
How is it that the two summary curves on
the far right side of the graphic are exactly the
same yet the data in the main body of the graphic
showing that TAT for each individual lab tests
present two very different patterns? The answer
is rather simple. The aggregated distributions
(the bell-shaped curves on the far right of the
charts) are static displays of data that merely
summarize where the data are centered (the
average) and the spread of the data (the spread
of the data would typically be reported as the
 Static Versus Dynamic Approaches to Data Analysis
minimum and maximum values, the range, and/
or the standard deviation). The main body of each
chart, however, represents dynamic displays of
data that essentially shows how process A and B
vary over time. The static displays, represented by
the bell-shaped curves, have basically aggregated
time out of each process.
A medical example will help to further clarify
these distinctions. Using dynamic displays of
data are equivalent to hooking patients up to
telemetry monitors and tracking their vital signs
moment by moment and minute by minute.
Consider, however, what patient monitoring
would look like from a static point of view. First,
we add up all of the patient’s heartbeats during
his/her length of stay. Next we would totally
ignore the telemetry strips and calculate the
average number of heartbeats and the standard
deviation for the patient. Finally, we would want
to know if the average number of heartbeats for
the current patient was statistically different,
using a p-value of 0.05 to determine significance,
from the average heartbeats of a patient who was
in the same bed last month or last quarter. This
aggregate or static approach to patient care might
save the nurses some charting time and could
be seen as a way to save money, but it would be
irresponsible. Medical practice is more concerned
with dynamic than static displays of data. Yet,
in healthcare management the determination
of quality has often been based on using static
displays of data to make conclusions.
The objective in QI research, therefore, is
to understand how the data present themselves
over time. This is how effectiveness and efficiency
are determined and how QIs are made. In short,
the use of aggregated data will not allow the
researcher to:
■■ Understand the variation that lives within
the data
■■ Understand the underlying distribution
that produced the observed average (i.e.,
you can obtain the exact same average from
two entirely different distributions as shown
in Figure 6-3)
167
■■ Evaluate the impact of process improvement/
redesign efforts
Although these issues might not be critical
to researchers conducting efficacy studies, they
are essential information to those interested
in efficiency and effectiveness of a process.
Another example will help to further highlight
this distinction. Suppose you were interested
in calculating the average height of group of
people. You measure everyone as they come into
the room and determine that the average height
is 5 feet 11 inches tall. You then have everyone
stand up and discover that half the people are
professional basketball players and the other half
are first-grade students. We quickly conclude
that there are no people in the room who are 5
feet 11 inches tall. What we have is a distribution
that has two extremes with an average that does
not match any of the individuals in the room.
The average height is a mathematically correct
statistic but it does not provide a clear picture
of the underlying nature of the distribution that
produced the average. This is the basic problem
of relying on aggregated data.
The up and down fluctuation in data
may be due to random variation, which is an
inherent part of any process. It may be due
to periodic special causes or to a true shift in
process performance. It may be up one month
and down the next. Unfortunately, organiza-
tions spend a considerable amount of energy
rewarding (and punishing) employees because
the numbers are up then down. Quality is
not about judging whether one data point is
different from another. It is about analyzing
data patterns as they occur over time and then
determining whether an intervention has made
a difference. Dynamic not static approaches to
data provide the road to knowledge about the
quality of healthcare outcomes, processes, and
services. Healthcare organizations can enhance
their QI efforts dramatically if they do nothing
more than make a deliberate effort to increase
their knowledge and use of dynamic displays
and analysis of data.
168 Chapter 6 Tapping the Knowledge that Hides in Data

CASE STUDY #1: The Monday Morning Dilemma

It’s Friday afternoon, and you have already started thinking about the relaxing weekend that lies
ahead. Then the phone rings. Within seconds of picking it up, you realize that this is the call you had
anticipated but wished would not come on a Friday afternoon. It’s your boss reminding you about the
management meeting on Monday. Typically, this would not be a major event. But this time, for you, it
will be. It seems that the numbers for your facility are “below expectations” for the past month, and you
have to explain why your facility is not meeting its targets. You thank your boss for the reminder, and
as you hang up, you remember what happened to your friend Amelia when she had to go through
this ordeal the previous month. At that time, your numbers looked fine. In retrospect, all you can really
remember about that day was how pale Amelia looked as she tried to explain why her numbers were
below average. You also remember thinking how this whole affair reminded you of an inquisition.
Now as you sit in a numb stupor looking out your window, you realize that your time has come.
Suddenly your assistant sticks his head in your office and brings you back to reality by saying, “I’m out
of here, have a nice weekend.” “Oh sure,” you think, “I’ll have a nice weekend. I’ll spend all my time trying
to develop a list of reasons why my numbers are not acceptable. I’ll spend all weekend worrying about
the inquisition on Monday!”
This is obviously a situation you do not want to be in. Yet, stories like this are legendary in
organizations that do not understand variation. So, what would you do? You basically have two
choices. First, you could develop a list of reasons why your numbers do not live up to expectations. This
usually consists of one or more of the following tactics: (1) developing a series of complex sentences
that try to divert attention from the numbers, (2) pointing fingers at other factors (some real and some
imaginary), (3) blaming individuals (who are usually not attending the meeting) for poor work habits
and low motivation, and (4) throwing yourself on the mercy of the inquisition court.
The second, and more preferable choice, is to not waste your time trying to explain why this
month’s number is different from last month’s number. The reason they are down this month is that
they were up last month. Variation occurs in all that we do. Therefore, you will drive yourself absolutely
crazy and mislead those you are presenting to if you try to explain (rationalize) why the two data points
are different. If the performance of the process over time is unacceptable, then change the process.
Focusing on aggregated numbers without understanding how they vary over time, is a futile exercise.
This type of thinking typically fosters competition between units and departments, builds barriers
between individuals, and, most important, undermines the delivery of quality care. It is reflective of
what Wheeler (1993, p. vi) refers to as “numerical illiteracy.” According to Wheeler:
Numerical illiteracy is not a failure with arithmetic, but it is instead a failure to know how to
use the basic tools of arithmetic to understand data. Numerical illiteracy is not addressed by
traditional courses in primary or secondary schools, nor is it addressed by advanced courses
in mathematics. This is why even highly educated individuals can be numerically illiterate. (vi)
Fortunately there is a cure for numerical illiteracy. It involves the following remedies:
■■ Understanding variation conceptually
■■ Understanding variation statistically
■■ Linking your measurement to improvement
If you inoculate yourself with these simple remedies (as opposed to spending your time trying to
justify/rationalize why this month’s number is lower than last month’s number) you will be in a much
better position to really explain what is occurring with the numbers on Monday morning. You will
also be able to enjoy your weekend! The details related to these three remedies are presented in the
remaining chapters.

Notes
1. There are documented accounts in the
Bible, for example, of directives to be
clean in body as well as in the prepara-
tion and storage of food. Disinfectants
are referenced by Aristotle, Homer, and
Hindu medical teachings. The “modern”
era in infection control can be traced to
the work of Swiss physician Paracelsus
(1493–1541) and most notably to Girol-
amo Fracastoro (1478–1553) who wrote a
three-volume set on the nature of infection
and its transference from person to per-
son. Block (2001) provides a wonderful
review of the key steps in the development
of this branch of medicine. Not only is
this account extremely interesting, but it
demonstrates the fundamental connection
between theory and experience.
2. I say, “purport to measure” because all
measurement is subject to bias and error.
Deming claimed that there is no such thing
as a “fact.” He wrote (1992, p. 292): “There
can be no operational definition of the true
value of anything. An observed numerical
value of anything depends on the definitions
and operations used. The definitions and
operations will be constructed differently by
different experts in the subject matter.” (See
Chapter 4 of this book for a more detailed
discussion of operational definitions.) It
is reasonable to conclude, therefore, that
all measurement is merely proxy for what
it intends to measure.
3. Although this text deals primarily with
topics related to conducting quantitative
studies, the field of qualitative analysis
should not be overlooked. Qualitative
methods should be an integral part of
any QI initiative. This can include vide-
ography and a variety of methods used in
social, psychological, and anthropological
research efforts. Methods discussed in
Chapter 3 can also be used.
4. The traditional summary statistics are
basically divided into two categories.
References 169
The first consists of measures of central
tendency, the mean, the median, and the
mode. The second category consists of
measures of dispersion, which usually
include the minimum value, the maximum
value, the range (the difference between
the maximum and the minimum values),
and the standard deviation. Any basic
statistics book will provide the details on
how to calculate these measures.
5. This statement was taken from the author’s
personal notes while attending a Deming
4-day seminar (Quality, Productivity, and
Competitive Position) in Indianapolis,
August 11–14, 1992.
6. In this chapter, I do not intend to elaborate
on the issues of intrinsic and extrinsic
motivation. I do believe that if an orga-
nization does not have a healthy debate
on how it intends to approach motivation
and rewards, it will be haunted by this
indecision and its culture will suffer. The
dominant writer on this topic is Kohn
(1986, 1993).
References
Austin, C. Information Systems for Hospital Administration.
Chicago: Health Administration Press, 1983.
Benneyan, J., R. Lloyd, and P. Plsek. “Statistical Process
Control as a Tool for Research and Health Care
­Improvement.” Quality and Safety in Healthcare 12,
no. 6 (2003): 458–464.
Blalock, H., ed. Causal Models in the Social Sciences. Chicago,
IL: Aldine Publishing Company, 1971.
Block, S. Disinfection, Sterilization, and Preservation. ­Philadelphia,
PA: Lippincott Williams & Wilkins, 2001.
Campbell, D., and J. Stanley. Experimental and Quasi-Experimental
Designs for Research. Boston, MA: Houghton Mifflin
Company, 1963.
Deming, W. E. Out of the Crisis. Cambridge, MA: M ­ assachusetts
Institute of Technology, 1992.
Forcese, D., and S. Richer, eds. Stages of Social Research:
Contemporary Perspectives. Englewood Cliffs, NJ:
Prentice-Hall, Inc., 1970.
Gaucher, E., and R. Coffey. Total Quality in Healthcare. San
Francisco, CA: Jossey-Bass, 1993.
Kohn, A. No Contest. Boston, MA: Houghton Mifflin, 1986.
Kohn, A. Punished by Rewards. Boston, MA: Houghton
Mifflin, 1993.
170 Chapter 6 Tapping the Knowledge that Hides in Data
Langley, G., K. Nolan, T. Nolan, C. Norman, and L. Provost. The
Improvement Guide. San Francisco, CA: Jossey-Bass, 1996.
Lastrucci, C. The Scientific Approach: Basic Principles of
the Scientific Method. Cambridge, MA: Schenkman
Publishing, 1967.
Namboodiri, N., L. Carter, and H. Blalock. Applied Multi-
variate Analysis and Experimental Designs. New York:
McGraw-Hill, 1975.
Perla, R., L. Provost, and G. Parry. “Seven Propositions of the
Science of Improvement: Exploring Foundations” Quality
Management in Health Care 22, no. 3 (2013): 170–186.
Selltiz, C., M. Jahoda, M. Deutsch, and S. Cook. Research
Methods in Social Relations. New York: Holt, Rinehart &
Winston, 1959.
Schultz, L. Profiles in Quality. New York: Quality Resources,
1994.
Shewhart, W. Statistical Method from the Viewpoint of
Quality. Washington, DC: Graduate School, Depart-
ment of Agriculture, 1939. (Reprinted by Dover, New
York, 1986).
Van de Geer, J. Introduction to Multivariate Analysis for
the Social Sciences. San Francisco: W. H. Freeman and
Company, 1971.
Wheeler, D. Understanding Variation. Knoxville, TN: SPC
Press, 1993.
Wheeler, D. Advanced Topics In Statistical Process Control.
Knoxville, TN: SPC Press, 1995.
CHAPTER 7
Overcoming Numerical
Illiteracy
Variation (var’-i-a’shun), n 1. Act or instance of varying; change in the form, position, state, or qualities
of a thing; modification, deviation, or an instance or example of such. 2. Extent to which a thing varies;
amount or rate of change.
—Webster’s Collegiate Dictionary, fifth edition, 1946
By permission. From Merriam-Webster’s Collegiate® Dictionary, 11th Edition ©2017 by Merriam-Webster, Inc. (www.Merriam-Webster.com).

I
n this chapter, three specific activities are dis- based on being able to write the first letter of
cussed that can help to immunize you against your first name three times with no variation in
numerical illiteracy. These activities include: the form, structure, or overall appearance of the
letter. If you are able to perform this simple task,
■■ Understanding variation conceptually
you will receive a 50% increase in your salary.
■■ Distinguishing common cause variation
Remember that there can be no variation in
from special cause variation
the letters. Give it a try. Now here is the second
■■ Making the appropriate responses to common
part of your performance evaluation. Place your
and special causes of variation
pen or pencil in your opposite hand and write
the same letter three times. To receive the 50%
increase in salary all six letters must be exactly
▸▸ Understanding the same with no variation. How many of you
Variation Conceptually passed the performance evaluation test? If you
are like me, your results look something like this:
Variation exists in all that we do, even in the
simplest of activities. For example, consider
writing your name. This is a simple activity
that you probably do each day. Imagine that
your annual performance review, however, was Bob’s left hand Bob’s right hand
© Michal Steflovic/Shutterstock

171
172 Chapter 7 Overcoming Numerical Illiteracy
The three letters on the right were done
with my dominant (right) hand. Although these
letters look similar, they are not identical. There
are subtle variations in the curves, where the
loops intersect another part of the letter, and
in the size of the letters (the middle letter B is a
little smaller than the other two). Now compare
these three letters with the three I wrote with
my left hand. The differences are obvious. There
is not only obvious variation between the two
sets of letters but also the variation within the
left-handed letters is greater than the variation
within the right-handed set. You might even
think that two different people wrote these two
sets of letters.
So it is very likely that none of us would
receive the 50% pay raise. Why? It is because
variation exists in all that we do. Sometimes we
are quick to recognize this fact, but at other times
we act as though there should be no variation.
Our society is full of great examples of how we
understand (and frequently do not understand)
variation. Consider the following list. How much
variation do you expect to see in each of these
items? A lot? A little? None?
■■ Snowflakes
■■ Twins
■■ Siblings (not twins)
■■ Your commute to work
■■ The stock market
■■ Your household expenses each month
■■ A patient’s blood pressure readings
■■ Bowling or golf scores
■■ A patient’s heartbeat
■■ Your weight throughout the year
■■ Patient satisfaction results
■■ A department’s monthly expenses compared
to budget
As you scan, this list you will probably
think that all of these items listed will exhibit
some form of variation. The snowflakes usu-
ally receive the most definitive response: “You
know, no two snowflakes are alike.” This popular
expression presents a view that there is clearly
considerable variation among snowflakes.1 We
have all heard and even said this snowflake
statement many times. But how does this clear
position on variation play out across the other
items? When I have asked these questions in
class, most participants tend to say that even
with identical twins there is some variation—
subtle variation, not unlike the variation found
when writing three letters with your dominant
hand. More variation usually comes into play as
people describe siblings. One of the classic lines
related to siblings is: “I can’t believe they came
from the same gene pool!” To me, this is type of
a statement is a clear signal that the respondent
has a view that there is more variation when it
comes to siblings than to twins.
The responses to variation associated with
the remaining items on the list typically depend,
to a large extent, on personal experiences (e.g.,
How long is your commute? What stocks have
you purchased recently? How skilled are you
at golf?). The historical experiences you have
had with these items will typically guide your
estimates of the amount of anticipated variation
in the process in the future. This is exactly what
Deming meant when he said, “The past is helpful
to us only if it helps in the future, if it predicts”
(quoted in Schultz, 1994, p. 23).
The last two items in the list (i.e., patient
satisfaction results and monthly expenses
compared to budget) deserve a few additional
comments. Even though patient satisfaction
scores are typically derived from sample data,
which requires an understanding of sampling
error, many people seem to think that patient
satisfaction scores will (or should) keep going
up. The essential nature of variation in patient
satisfaction scores, however, reveals little
fluctuation in the numbers. Sometimes the
numbers go up, sometimes they go down but
most of the time patient satisfaction scores
reflect random variation. The point is that
assuming, wishing, or hoping that the patient
satisfaction scores constantly go up is a clear
demonstration of the lack of understanding of
variation. In this case, the desired state (i.e.,
improved scores) is being confused with the
objective evaluation of the variation that lives
within the data.

When people do not understand variation
they will demonstrate a number of rather unique
behaviors. Most of these behaviors can be grouped
into one or more of the following behaviors:
1. Deny the data (It doesn’t fit my view
of reality!)
2. Distort the process that produced
the data
3. See trends where there are no trends
4. Try to explain natural variation as
special events
5. Blame and give credit to people for
things over which they have little or
no control
6. Find it very hard to understand past
performance, make predictions, and/
or make improvements.
Each of these behaviors is explored next.
Deny the Data (They Don’t Fit My
View of Reality)
Unfortunately, this reaction occurs much more
frequently than we would like. Individuals at all
levels of the organization frequently get into a
state of denial when the data do not fit their view
of reality. The major drivers of this orientation
are internal and external targets and goals that
are not linked to the current capability of the
processes being observed. Individuals expect
their arbitrary target and goals to be achieved
without having any clear understanding the
current performance of the process and what
the processes are capable of producing. Another
reaction when the data do not fit one’s view of
reality is that “This is not our data. It must be a
mistake and our data have been confused with
some other provider’s.” The final reaction to data
that do not fit one’s view of reality is “These are
old data and do not reflect our current perfor-
mance.” Irrespective of the response, the central
issue is what are the factors that are causing the
denial orientation? Does the organization not
really understand its actual performance and
results? Are the leaders of the organization not
familiar with their current levels of performance
Understanding Variation Conceptually 173
because they look only at quarterly comparisons?
Do they use aggregated numbers and summary
statistics to discuss their results? If these are the
behaviors of the organization’s leaders then it
is not surprising that denial is one of their key
strategies.
Distort the Process that Produced
the Data
A follow-up behavior to denying that the data are
either not relevant or not yours is to be critical
of the process (and people) that produced the
data in the first place. We do this frequently
in health care especially when the data are
assembled on us by outside groups and then
released to the media. The criticisms are levied
at the dates of data collection (e.g., these are old
data), the procedures used (e.g., the sampling
design, if observers were used to collect data in
a qualitative manner, poor stratification levels,
or no risk adjustments were applied to the data).
Politicians and those running for elected office
probably engage in this practice even more
than those of us in the healthcare industry. If,
for example, the results of a political poll on a
candidate’s performance or the degree of sup-
port the candidate is receiving from the public
is not acceptable to the candidate then there is
usually a fairly strong disclosure that the results
of the poll were biased or not representative of
the candidate’s true constituency. Many of these
reactions are key components of Deming’s cycle
of fear referenced in Chapter 5.
People Will See Trends Where
There Are No Trends
There are many popular views on what consti-
tutes a trend so that people end up confused
about the true nature of a trend. For example,
it is not uncommon for the media to describe a
“trend” in fashion, dining, the weather, or cars.
Most of these references have little to do with
statistical trends. At best they reflect opinions
about fads, marketing plans, or buying patterns
174 Chapter 7 Overcoming Numerical Illiteracy
of consumers. Every day the media reports about
an upward or downward trend in something.
More often than not, this simply refers to the
fact that the most current number is higher (an
upward trend) or lower (a downward trend) than
the previous number. The other perspective on
a trend was documented nicely in the Chicago
Tribune Magazine (August 16, 2015). The head-
line on the front page of the magazine read, “Fall
Fashion: The new trend is no trends!” The cover
picture accompanying this headline showed a
very fashionable young lady in a rather strange
mixture of clothing styles that did not look like
they actually went together. I suspect that the
picture was trying to capture the headline or vice
versa. The key point is that trends in fashion, food,
cars, or music are not the same as a statistical
trend. So what constitutes a statistical trend?
A true statistical trend is defined by an ever
increasing or decreasing series of consecutive
numbers (Carey, 2003; Carey and Lloyd, 2001;
Pyzdek, 1990; Western Electric Company, 1985).
Because many people do not engage in statistical
thinking they will start to see trends where there
are no trends. They look at two or three numbers
that appear to be going up or down and call it a
trend.2 I bet you will, in the near future, hear or
see in the paper or on TV someone proclaiming
that there is a trend in a particular aspect of
financial or social performance when in fact it
is merely (1) the comparison of two numbers or
(2) an individual’s opinion about a vague tendency
that something is moving in a new direction as
in a fashion trend. The specific statistical rules
for determining a trend are discussed later in
this chapter.
People Will Try to Explain Natural
Variation as Special Events
At an earlier point in time I thought that im-
properly defining a trend was the most prevalent
problem that we needed to address. Over the
years, however, I have changed my mind. I now
think that trying to explain natural variation as
special events is a much more pervasive problem.
This behavior can be found everywhere: in staff
meetings, board meetings, casual conversations,
and the media. We pride itself on being “action
oriented.” So when a group is presented with
data, group members will look for anomalies
or special events so that they feel justified in
taking action or arriving at a conclusion about
the data’s performance.
I had a physician in a class one day who
described this behavior very nicely. As I finished
explaining the differences between common
(random) variation and special cause or nonran-
dom variation, he blurted out, “I explain natural
variation as special events.” When I asked what
he meant, he explained that when his diabetic
patients come to see him for checkups, he is
prone to change their medications when he
notices the slightest up or down movement in
their blood glucose readings. He also noted that
his patients seem to have rather wide swings in
their readings—wider than the ranges his part-
ners reported for their patients. As we processed
these insights, he came to the conclusion that
his patients were probably experiencing wider
variation because he was, in fact, trying to
explain natural variation as special events. He
continued his own self-assessment by concluding
that he was overreacting to both high and low
blood glucose readings. In both cases, he was
attributing special cause status to readings that
were essentially common or random cause. In
doing so, he was essentially tampering with a
stable process that actually ends up increasing
variation. More on this point in a minute. The
result of treating natural variation as if it were
special cause is that you increase variation. When
I asked him why he thought he did overreact to
natural variation, he made two comments. First,
he said that he was basically taught to react to
each data point. Second, he thought his patients
would think he was not a “good” doctor unless
he took action when they came to see him. Over
the years, this physician and I have become good
friends. He is now a staunch supporter of what
Shewhart and Deming were trying to teach
people—that (1) variation exists and (2) you
need to be able to determine if it is common or
 Understanding Variation Conceptually 175

special cause variation before deciding upon a
course of action.
Every day you will find people who try to
explain natural variation as special events. The
inevitable outcome of such behavior is overreacting
to the numbers, which leads to tampering and
ultimately to increased variation in the process.
This principle has been best demonstrated by
Deming’s experiments with the funnel (Deming,
1992). The objective of the funnel demonstration
is to drop a series of marbles through a funnel and
see where they land on a target placed beneath
the funnel. The setup for this demonstration is
shown in FIGURE 7-1. Deming identified four rules
to guide the dropping of the marbles, all of which
produced very different patterns of variation. The
rule of most interest to us right now is Rule 4 of
the funnel. In this case, the funnel is adjusted to
drop the next marble over the spot where the
previous marble came to rest. Rule 4 basically
states that the variation in the system will “ex-
plode” (Deming’s term) if you keep changing the
position of the funnel on every subsequent drop
of a marble. This essentially amounts to tampering
by overreacting to individual data points without
understanding how they all fit together to form
a system of variation. Deming described Rule 4
as follows: “Rule 4 will yield a random walk. The
successive drops of the marble resemble a drunk
man, trying to reach home, who falls after each
step and has no idea which way is north. He steps
in any direction, with no memory. His efforts
eventually send him by faltering steps further
and further from his target” (1992, 329).

Marble #2
does not go to
Reposition the bull’s-eye—
funnel here and it lands here
drop marble #2

Target M2

M1

Marble #1 hits the bull’s-eye then bounces and rolls

to its final resting spot on the target (M1) Marble #1 lands here

Measure the distance from marble #1 (M1) to the bull’s-eye. Then swing the
funnel an equal distance in the opposite direction and drop marble #2 at this point.
The assumption is that the 2nd marble will follow the same course as the first one
and end up in the bull’s-eye. But marble #2 does not go where marble #1 went.
So measure the distance that marble #2 is from the bull’s-eye and swing the
funnel an equal distance in the opposite direction. Continue this procedure until
you have dropped at least 50 marbles.

FIGURE 7-1 The funnel demonstration

176 Chapter 7 Overcoming Numerical Illiteracy
If you dropped all the marbles without ever
adjusting the position of the funnel, you would
end up with the minimum amount of variation
and a stable process. The resultant pattern on
the target would essentially resemble a flatten
normal (bell-shaped) curve (i.e., the majority of
the marbles would be clustered around the center
of the target and as you proceeded out from the
bull’s-eye you’d find a gradual reduction in the
number of marbles similar to the tails of a normal
curve). In this case, the process variation is at
a minimum because natural variation was not
interpreted and reacted to as special events. In
a word, there was no tampering. The reader is
encouraged to spend time reading the details of
this demonstration and becoming familiar with
all four rules of the funnel. They will provide
critical knowledge about the subject of variation.
An example from health care will help to
anchor Rule 4 of the funnel. Imagine that you
are the manager of an outpatient clinic. You have
been tracking the average wait time by day for
several weeks. As you review the figures, you
notice that the average wait time yesterday was
higher than it was the day before. So you call
the staff together and tell them that this higher
wait time is unacceptable. Your response to this
increased wait time is to adjust the staffing patterns
to improve efficiency. The next time you look at
the data you observe that the average wait time
was lower than it had ever been during the past
month. So, you change the staffing patterns to
“lighten up” on the number of workers and their
respective workloads. What have you accom-
plished? By trying to force every day’s average
wait time to be like the previous day, you have
been demonstrating Rule 4 of the funnel. You
have overreacted to both the high and low aver-
age wait times and adjusted the process without
knowledge of the system’s performance. The
prudent manager would first plot the data on a
control chart and then determine if the process
exhibited common or special causes of variation.
This tendency to try to explain natural varia-
tion as special events is demonstrated in nearly all
walks of life. For example, consider the following
illustrations from the newspaper industry. The
examples are a little dated but they provide such
classic examples of not understanding variation
that they need to be highlighted. The first story
comes from front pages of two newspapers in
the same city on the same day. On September 1,
1998, the morning edition of the San Francisco
Chronicle had the following headline: “Panic
hammers market.” It was accompanied by a pic-
ture of a stock market analyst holding his head
in dismay at the plunging stock quotes. This was
accompanied by multicolored graphs of rapidly
dropping stock prices. The evening edition of
the San Francisco Examiner, however, told a very
different story. Its headline read, “Stocks bounce
back.” The article went on to explain how the
stock market “soared” late in the closing hours of
the day to overcome the panic that had occurred
that morning. This the San Francisco Examiner
concluded was an upward trend in the stock
market and a very positive sign for investors.
The morning drop and the afternoon rise were
merely natural fluctuations in the data. Neither
one was a special event and the movement of two
data points is definitely not a statistical trend.
The second example comes from the Wall
Street Journal (September 19, 1997).3 The headline
read, “Trade deficit surges 25% to 10.3 billion.”
The article goes into great depth describing how
the trade imbalance with Japan was the “largest
in two years” and that the “Chinese gap widens
to 9%.” There was very colorful language in the
article about how the trade deficit has “ballooned”
to $10.3 billion. The two-and-a-half-page article
does not paint a very positive picture for the U.S.
economy. They even showed a little line graph
depicting the deficit over time. The article based
its 25% “surge” conclusion on a comparison of
the most recent deficit with the previous time
period’s deficit. If one computed a percentage
change between these two time periods, it was
in fact about a 25% increase (from roughly
$8.2 billion to $10.3 billion). If they had placed
the data on a control chart, however (which
they did not do), they would have realized that
the deficit over the 25-month period (July 1995
through July 1997) was nothing more than
common cause variation. FIGURE 7-2 shows
 Understanding Variation Conceptually 177

14.0
UCL = 13.378
13.0
Was this the pinnacle of despair?
No, but
12.0 this was a
“surge”!
11.0
Billions of dollars

10.0

9.0 CL = 8.936

8.0

7.0

6.0
Was this the abyss of happiness?
5.0
LCL = 4.494
4.0
May96

May97
Nov95

Nov96
July95

Sept95

Jan96

Mar96

July96

Sept96

Jan97

Mar97

July97
Month

FIGURE 7-2 Deficit spending from July 1995 through July 1997

that the average deficit over the study period nation will continue to have an annual deficit of
was $8.9 billion with an upper control limit roughly $9 billion, then they should take steps
(UCL, to be described later in this chapter) of to (1) shift the average deficit to a lower level of
$13.4 billion and a lower control limit (LCL, performance, and/or (2) reduce the variation in
also to be described later) of $4.4 billion. The the entire process and thereby bring the UCL
average, along with the UCL and LCL, basically and LCL closer together.
describes what is known as the capability of the Regardless of the action plan, the key
process. Stated differently, the deficit process, learning point in both of these examples is that
on the average, would continue to be about the newspapers clearly saw natural variation
$8.9 billion per year. It could go as high as as a special event. In the second example, I am
$13.4 billion or as low as $4.4 billion, which actually a little surprised that they did not pick
represented the variation in the deficit process. up on earlier data patterns and give them top
The question is not “What are you going billing in this article. For example, the month of
to do about a 25% surge in the last two data January 1997 had the highest deficit during this
points?” but rather, “Are you satisfied with an entire study period. Why did they not label this
annual average deficit of $8.9 billion with a as the “pinnacle of despair”? They could have
potential swing of anywhere between $4.4 and said that prior to this point there was a “sudden
$13.4 billion?” If the government is willing to and rapid upward trend followed by a sudden
accept this process capability, then they should and rapid precipitous fall in the deficit.” If the
do nothing because the process will continue to month of January 1997 was the highest deficit,
perform in this manner into the future. If, on why did they not also highlight the period from
the other hand, they do not like the fact that this August through December 1995, when the deficit
178 Chapter 7 Overcoming Numerical Illiteracy
was below the average for 5 straight months? If
January 1997 was the pinnacle of despair, then
this 5-month period of relatively low deficit
spending must surely have been the “abyss of
happiness.” In short, the deficit will never im-
prove by seeing events as special when in fact
they represent nothing more than random or
common cause variation.4
People Will Blame and Give Credit
to Others for Things over Which
They Have Little or No Control
This is probably one of the most damaging aspects
of not understanding variation, because it has
a direct and immediate impact on the workers.
Consider Bill, the manager of the outpatient
testing and therapy department. You see him
in the cafeteria and he looks rather depressed.
You ask him why he seems down, and he tells
you that his boss just chewed him out because
the monthly patient satisfaction scores were
lower this month than they were last month.
The next month you also happen to run into
Bill, who is in a very good mood. You say, “Bill,
good to see that you are happier than the last
time I saw you. What’s going on to make you
this happy?” Bill responds, “Well, the boss likes
the patient satisfaction scores this month and he
just told me I could buy pizza for everyone in
the department to celebrate.” With an inquisitive
look on your face you ask, “Wow, what did you
do to increase the scores?” Bill responds with a
puzzled look, “Nothing. I guess we just lucked
out this month.” Unfortunately, I have been
the recipient of this type of behavior and have
observed many other individuals who have tried
to cope with this form of numerical illiteracy.
Behaviors surrounding budget variances
provide another classic example of this behavior.
Organizations will typically take a budget and
divided it into 12 equal segments. The average
monthly expected expense figure is then pre-
sented to the manager as the “target” for each
month’s actual expenses. Everyone knows that
the actual expenses will not be exactly the same
every month (i.e., no variation from target). Or-
ganizations that understand variation recognize
this fact. But some organizations act as though
there should be no variation from the expected
average monthly budget figure. When a given
month’s expenses are above the average expected
expenses for the month, the manager of that
area is usually blamed and required to write a
lengthy memo explaining why there is variance
above the expected average. For the manager,
this behavior creates confusion, because there
will frequently be praise and potentially rewards
when the department’s budget is below the target.
It gets even more confusing, however, when the
manager is blamed when the budget is above the
expected average but receives no credit when it
is below the target for each month. Why is there
no requirement for a memo when the budget is
below the target?5
The final and most negative aspect of this
last behavior is the “kill the messenger” perspec-
tive highlighted in Deming’s cycle of fear. This
happens when the boss wants to hear only good
news. If the numbers are down this month the
staff draw lots to see who will have to deliver
the bad news. Fortunately today people do not
literally get killed for delivering bad news to the
boss but organizations have come up with rather
clever ways to deal with receiving bad news and
then blaming the worker for being incompetent.
Dr. Laurence Peter’s classic book The Peter
Principle: Why Things Always Go Wrong (1970)
is one of the best reads on what happens when
processes fail to deliver the desired results. One of
Peter’s principles is that the messenger bringing
poor results to the boss can easily be “laterally
arabesqued.” In this case, the individual is not
outright fired but moved to a new role with a new
title and stuck in a remote part of the building.
Such a “pseudo-promotion” as Peter calls it will
generally demoralize the worker and from the
boss’s perspective, be enough to cause the person
to leave on their own. Actions that organizations
and individuals take in regard to blaming and
giving credit for things over which the workers
have little control create what Deming (1994)
referred to as the “heavy losses.”
 Distinguishing Common from Special Causes of Variation
People Will Find It Very Hard to
Understand Past Performance,
Make Predictions, and/or Make
Improvements
Quality improvement is based on understanding
these three activities. Yet if an organization ap-
proaches the analysis of data from a static rather
than a dynamic point of view, then there is a
very high probability that they will not be able to
understand past performance, make predictions,
and/or make improvements. Demonstrating skill
in these behaviors requires an understanding
of where measures have been, where they are
now, and where they will go in the future. The
answers to these questions cannot be found in
summary statistics and tests of significance. They
can be found only in knowledge of variation
and statistical process control (SPC) methods.
▸▸ Distinguishing
Common from Special
Causes of Variation
The previous examples should help to highlight
some of the inconsistencies we have when it
comes to understanding and interpreting vari-
ation. Once we begin to think about the nature
of variation and how it plays a major role in our
daily lives, however, we will be in a much better
position to actually start distinguishing between
two very distinctive types of variation, common
and special cause variation.
The origins of common and special causes of
variation can be traced back to the early 1920s,
a time when this country was struggling with
a fairly basic question: How can you increase
production and maintain quality? The man who
helped answer this question was Dr. Walter
Shewhart. Shewhart argued that all work could
be viewed as a series of interrelated processes.
Because customers are the recipients of process
output, it stands to reason that organizations
179
that focus on maintaining efficient and effec-
tive processes will meet and exceed customer
expectations. Shewhart’s recommendation for
creating efficient and effective processes was very
simple. He maintained that if you understand
the variation that occurs within a process, you
will be able to make appropriate management
decisions that will produce high-quality prod-
ucts and services. Shewhart distinguished two
types of variation: assignable and unassignable.
These terms were later revised by Deming to
the more popular terms common and special
causes of variation (Deming 1992; Schultz
1994; Shewhart 1931). Deming (1992, 314)
classified common cause variation as “faults of
the system” and special causes as “faults from
fleeting events.”
The genius of Dr. Shewhart was that he had
a unique ability to turn academic principles and
concepts into practical and easy-to-understand
concepts that the workers on the shop floor of the
Western Electric Company back in the 1920s could
easily grasp. The essence of what Dr. Shewhart
was conveying about variation is shown in
FIGURE 7-3. His basic point was that if you
look at a distribution of data as a normal bell
curve you will obtain a static view of the data
in which time has been removed. When you
summarize an array of data as a distribution
you have no idea which data point came first,
in the middle, or at the end. But, if you turn
this distribution on its side and then extract the
data from the static distribution you see how
the data array themselves in time order. This
will give you a very different view of the data.
In the static distribution, time is not relevant.
When you add the dimension of time back into
the distribution you will see the variation in the
indicator as it lays itself out in chronological
order (i.e., hour by hour, day by day, week by
week, or patient by patient). When you look at
data this way, Shewhart maintained that you
could then understand whether the data dis-
played characteristics of controlled (common)
or uncontrolled (special) causes of variation.
Common cause variation is random variation
that results from regular, natural, or ordinary
180 Chapter 7 Overcoming Numerical Illiteracy
Static view
St
at
ic
vi
ew
Every process displays variation:
• Controlled variation (random variation)
• Special cause variation (non-random variation)
Static view
FIGURE 7-3 Shewhart’s view of variation
forces. Common cause variation affects all
outcomes of the process and results from the
regular random rhythm of the process. It produces
processes that are stable or “in control.” One can
make predictions, within statistical limits, about
a process that has only common cause variation
(this is often referred to as process capability). In
a common cause state, there are no indications
of special cause, because the variation results
only from chance fluctuations in the data. Your
morning commute to work provides a good
practical example of understanding variation.
If you ask people, “How long does it take you
to get to work in the morning?” most respond
“Oh about (fill in the blank) minutes.” Each
day’s commute is not exactly the same number
of minutes (unless you live two blocks from
your place of work, in which case we would
not include you in this analysis). Let’s say that
someone responds “Oh, about 55 minutes.” The
word “about” is an indication that there is some
degree of variation around 55 minutes. Then she
might add, “It could take me as long as about
Dynamic View UCL
time
LCL
stable, consistent pattern of variation
“chance”, constant causes
“assignable”
pattern changes over time
85 minutes or as little as 40 minutes.” What has
she done here? She has described the variation
in her morning commute in a way that would
have pleased Dr. Shewhart. If all she said was
“Oh, about 55 minutes” all we would have is some
rough estimate of her average commute time.
But when she added the other two numbers (i.e.,
from about 85 to 40) we now know something
about the variation in her morning commute.
The key thing to remember with common
cause variation, however, is that it does not mean
that the performance of the process is good or
even acceptable. It means only that the process
is stable and therefore predictable. A process
can be predictably bad. For example, a patient
may have blood pressure readings that are stable
and very predictable but at an unacceptably
high level (e.g., a systolic pressure that averages
175 with a minimum at 165 and a maximum
at 185). It is stable, and therefore, predictable
but unacceptable clinically. The same could be
true for cholesterol, white blood cells counts
or blood glucose levels. In all these cases, we
 Distinguishing Common from Special Causes of Variation
would need to shift the various process outputs
to more acceptable levels of performance. This is
what (QI) is designed to do. Remember, though,
common cause means stable and predictable,
not necessarily acceptable.
Special cause variation, on the other hand,
results from irregular or unnatural causes that
are not inherent in the process. Special cause
variation affects some but not necessarily all
outcomes of a process. When special causes
are present, a process will be classified as “out
of control” and, therefore, unstable. Wheeler
(1993) refers to special causes as “signals” that a
process is in a state of chaos. The future perfor-
mance of a process that exhibits special causes
will be unpredictable. This is why improvement
strategies should not be applied to processes
exhibiting special cause variation. Because
they are unstable and unpredictable, attempts
to improve them will only lead to wasted time,
effort, and money as well as increase variation.
The response to special causes of variation should
be to investigate the origin of the special causes,
determine why they occurred (conduct a root
cause analysis), and then take steps to eliminate
the causes from the process. If you do not extri-
cate the factors that led to a special cause they
will rear their ugly heads again in the future. No
one can predict exactly when they will occur but
as James Reason points out in his classic work
Human Error (2003) the latent causes of error
(i.e., special causes) are inherent in the system
and under the right set of conditions they will
occur again if you do not take steps to remove
these causes from the system. This is classically
referred to as the Swiss cheese theory of errors
(Reason, 2000).
Several healthcare examples should help
to clarify these two types of variation. When a
patient is connected to telemetry, variation in
his or her vital signs will be observed. In fact,
there is great concern if variation is not observed
and the patient is considered to be a “flat line.”
If the patient’s heartbeat is observed at 59, then
61, then 60, and so on, there is no need for im-
mediate concern. Why? Because the patient is
181
exhibiting essentially common cause variation
with respect to a normal heart rhythm. A little
up, a little down but within expected performance
capability parameters for assessing heartbeats.
If, on the other hand, the patient’s heart rate
starts to climb and goes from 65 to 70 to 75,
continues past 100 beats per minute, and settles
at around 140 beats per minute, this would be
seen as a signal that the patient is exceeding
normal expectations and too much variation.
At some point the patient would move from a
common cause state to one that reflects special
cause variation, and action would be taken to
investigate why the heartbeats had progressed
to such high levels and then corrective strategies
would be put in place to correct the heart’s rhythm
and bring it back into a state of common cause
variation. Action would not be taken, however,
while the patient was demonstrating acceptable
common cause variation.
Now consider a trauma patient who is
brought into the emergency department (ED).
The patient has a severe head injury, is hardly
breathing, and has lost a considerable amount of
blood. As you are examining him you also notice
that he is wearing a medical bracelet indicating
that he is diabetic. What is the first thing on the
minds of the ED team? What would they do for
this patient? They would first and foremost focus
their efforts on “stabilizing the patient.” They
would work to get the patient breathing under his
own power, stop the bleeding, and deal with the
head trauma. What they would not do is gather
around a flipchart and brainstorm ideas about
how this patient might improve his diet in order
to have better control of his diabetes (maybe the
driver passed out because of poor management
of his diabetes). The patient came in with several
of the key indicators of life reflecting special
cause variation (unstable and unpredictable).
The trauma team would work to get the patient
into a common cause state of affairs. Once the
patient was stable and predictable relative to
breathing, bleeding, and head trauma the trauma
team might then discuss improvement strategies
related to diabetes management.
182 Chapter 7 Overcoming Numerical Illiteracy

▸▸ Making the The rows of TABLE 7-1 identify the type of

Appropriate Responses
to Common and Special
Causes of Variation
Being able to describe the differences between
common and special causes of variation starts
you on the road to statistical thinking. But,
do you and those around you use this type of
thinking to make management decisions? The
attributes of a leader who understands variation
include the following:
■■ Leaders understand the different ways that
variation can be viewed.
■■ They explain changes in terms of common
causes and special causes.
■■ They use time series graphical methods
to learn from data and expect others to
consider variation in their decisions and
actions.
■■ They understand the concept of stable and
unstable processes and the potential losses
due to tampering.
■■ They understand the capability of a process
or system before establishing targets or goals
and before changes are attempted.
variation, the right choice, the wrong choice,
and the consequences of making the wrong
choice. The columns are split in two based on
whether you observe a stable or unstable pro-
cess. If you have only common cause variation
the right choice is to change the process by
either working to reduce variation or move the
entire process through redesign to a totally new
level of performance. The wrong choice when
you have common cause variation is to treat
normal (random) variation as if somehow it
was special. This is what was described earlier
as tampering (i.e., over- or underreacting to
individual data points when they are part of
a common cause process) and demonstrated
by Rule 4 of Deming’s funnel demonstration.
The right-hand column starts off with a mix
of special and common cause variation. In
this case, the correct decision would be to
investigate the reason(s) why the special cause
is present and take steps to extricate these
causes from the process. The wrong choice
when you have special cause variation is to
change the process or system in reaction to
that one special cause without investigating
why it occurred. When this happens you end
up wasting resources including time, effort,
morale, and money.

TABLE 7-1 Appropriate Responses to Common and Special Causes of Variation

Is the process stable?

YES NO

Type of variation Only common Special + Common

Right choice Change the process Investigate the origin of the special cause(s)

Wrong choice Treat normal variation as a Change the process

special cause (tampering)

Consequences of making Increased variation! Wasted resources!

the wrong choice (time effort, morale, money)
 Making the Appropriate Responses to Common and Special Causes of Variation
Almost every day you will read a story in the
newspaper or hear a TV news report demonstrat-
ing how people regularly overreact to a special
cause, don’t investigate the origin of the special
cause, and proceed to change the system. This
happened in Chicago a number of years ago.
An elderly driver entering O’Hare Airport to
pick up a family member became confused in
the heavy traffic congestion. At some point he
either panicked or became confused and instead
of hitting the brake put his foot on the gas. The
car proceeded to jump the curb and drive down
the departure area sidewalk. Unfortunately there
was a group elementary school students on a
field trip on the sidewalk when the gentleman
lost control of his car. He drove into the children,
killing three of them and injuring many more.
This tragedy immediately was brought to the
attention of the mayor of Chicago and the city
council. Something must be done was the public
response. My question is was this a special cause
or a common cause? It had never happened be-
fore and has not happened since. But the initial
reaction of the Chicago City Council was that they
should erect three-foot-high barriers along the
curbside check-in area so cars could not jump
the curb. Another idea was to raise the height of
the curbs along the departure driveway to a height
of about 15 inches. Now think of this for a minute.
Placing large construction barriers all along the
curb or increasing the height of the curb at one
of the busiest airports in the world would prove
to be a traveler’s nightmare. Dragging suitcases
and other baggage up over a 15-inch high curb
or between construction barriers would totally
delay and irritate probably everyone. So I ask
again, do you think this was a special cause or
a common cause? When I tell this story in class
participants agree that it was a special cause. It
was a singular event that although tragic required
investigation not changing the system. I turns
out the gentleman who was 85 years old claimed
his gas pedal stuck. The car turned out to be
in proper working order. He had never driven
into O’Hare Airport before, became confused
and then unfortunately panicked and made the
wrong decision. So, the city council did nothing.
183
But their decision was based not on the findings
of why this special cause occurred but because
they did not want to spend the money to totally
redesign the departure area at the airport. What
someone should have said was, “This is a special
cause. It has not happened in the past but we
do have a pretty good understanding as to why
it happened.” The conclusion and follow-up
questions should not be about the curb height
at O’Hare Airport but rather about the process
the State of Illinois has in place for reviewing
the driving skills of elderly drivers. They almost
made the wrong decision.
Once you have developed a comfort level
with the concepts of common and special cause
variation and how to make the appropriate deci-
sion, it is time to apply this knowledge to daily
work. Engage in a dialogue with your colleagues
around these issues. Do this by:
■■ Selecting several measures your organization
tracks on a regular basis.
■■ Then determine whether you and the leaders
of your organization evaluate these measures
according to the criteria for common and
special causes of variation.
■■ If you do not use common and special causes
of variation to interpret your indicators, what
criteria do you use to determine whether
data are improving or getting worse?
In summary, the basic points that Shewhart
(1931) taught about these two types of variation
were as follows:
■■ Variation exists in all that we do
■■ Processes that exhibit common or chance
causes of variation are predictable within
statistical limits
■■ Special causes of variation can be identified
and eliminated
■■ Only processes that exhibit common cause
variation can be improved
■■ Attempting to improve processes that contain
special causes will increase variation and
waste resources.
With this as a backdrop it is now time to
dive into understanding variation statistically.
184 Chapter 7 Overcoming Numerical Illiteracy
Notes
1. The science of snowflakes is actually
quite interesting. The first person to
actually photograph a snowflake was
Wilson Bentley on January 5, 1885 in
Jericho, Vermont. He spent the majority
of his life photographing snowflakes
and guess what? He never found two
that were alike. Subsequent research
has essentially confirmed Mr. Bentley’s
supposition. If you would like to learn
more about this topic watch this very
interesting You Tube video and find out
more about snowflake science: https://
youtu.be/fUot7XSX8uA.
2. I have experienced this reaction with my
financial advisor. I got a call from him
recently telling me that my investments
have experienced “an upward trend.”
I then looked at my portfolio and see
that the most recent quarter’s results are
a little higher than the previous quarter’s
results. My financial advisor was quite
excited about this “trend” in one of
my investments. I am more sanguine.
So, I took the data on this particular
investment over the past 2 years and
made a control chart to see if there were
any statistical shifts or trends in the in-
vestment results. There were none. The
control chart revealed merely common
cause variation indicating that nothing
had really changed over the past year
for this particular investment. I had
not lost money but on the other hand,
there was certainly not an upward trend
in the data. Many professions including
financial, legal, the media, political, and
even health care love to see trends where
there are none. In most cases, people who
voice such conclusions are looking for
data that somehow confirm their view of
reality. It is up to all of us to help these
folks see that they lack statistical thinking.
By the way, the rest of my financial story
is that I have reoriented my financial
advisor and now twice a year we review
the control charts on my investments
to see if there is really a trend or shift
in any of my investments. He no longer
calls to tell me that this quarter’s returns
are higher than the previous quarter’s.
3. I want to acknowledge and thank my good
friend and colleague Dr. Ray Carey for
finding this example. We used this story
in our public seminars for many years but
never placed it in print. It is a wonderful
demonstration of the tendency to attribute
special cause characteristics to common
cause variation. I appreciate Dr. Carey’s
keen eye for observing examples of vari-
ation in everyday life.
4. It is interesting to note that the deficit did
demonstrate a special cause in the months
following the Wall Street Journal article of
September 19, 1997. Dr. Carey continued
to track the deficit and discovered that the
next 4 months after July 1997 continued
upward. When these four data points
were combined with the last two data
points shown in Figure 7-2, an upward
trend was identified (i.e., six data points
constantly going up). If the newspaper had
taken the time to understand variation
and track it properly, they would have
had a real story.
5. I worked for an organization a number
of years ago that perfected this behavior.
The project I was managing would be
above the expected budget one month
and below it the next. The individual
responsible for “leading” our area was
probably one of the most numerically
illiterate individuals I have ever met.
Not only did I have to write a detailed
justification for why the project was
over the expected monthly average but
I also had to sit through a ceremonial
tongue-lashing for allowing the expected
budget threshold to be breached. I would

be told I was a poor manager for allowing
this to happen and that it would have
a negative impact on my performance
review if it continued. I found it curious
that I was never praised for being a good
manager when the expenses were below
the expected monthly average. When this
did occur, which was about 6 months
out of 12, I was merely told that I was
doing what was expected of me. Well,
I got very tired of this numerical illiteracy
and decided to prepare six memos that
provided different (and very compelling)
reasons why the expenses were greater
than expected. Every time the breach
occurred, which again was about half the
year, I would select a memo that had not
been used in a while, date it, and send
it on to you-know-who. This cut down
significantly on the writing process but
did not help the tongue-lashing aspect.
That component, unfortunately, continued
and quickly motivated me to find an orga-
nization that had a better understanding
of variation.
References 185
References
Carey, R. Improving Healthcare with Control Charts. ­Milwaukee:
Quality Press, 2003.
Carey, R., and R. Lloyd. Measuring Quality Improvement
in Healthcare: A Guide to Statistical Process Control
Applications. Milwaukee: Quality Press, 2001.
Deming, W. E. Out of the Crisis. Cambridge, MA: M
­ assachusetts
Institute of Technology, Center for Advanced Engineer-
ing Study, 1992.
Deming, W. E. The New Economics, 2nd ed. Cambridge,
MA: Massachusetts Institute of Technology, Center for
Advanced Engineering Study, 1994.
Peter, L. The Peter Principle: Why Things Always Go Wrong.
New York: Bantam Books, 1970.
Pyzdek, T. Pyzdek’s Guide to SPC. Vol. 1: Fundamentals.
Milwaukee: Quality Press, 1990.
Reason, J. “Human Error: Models and Management.” British
Medical Journal 18, no. 320 (2000): 768–770.
Reason, J. Human Error. Cambridge, England: Cambridge
University Press, 2003.
Schultz, L. Profiles in Quality. New York: Quality Resources,
1994.
Shewhart, W. Economic Control of Quality of Manufactured
Product. New York: D. Van Nostrand, 1931. Reprint,
Milwaukee: Quality Press, 1980.
Western Electric Company. Statistical Quality Control
Handbook. Indianapolis: AT&T Technologies, Inc., 1985.
Wheeler, D. Understanding Variation: The Key to Managing
Chaos. Knoxville, TN: SPC Press, 1993.
CHAPTER 8
Understanding Variation
with Run Charts
U
nderstanding variation from a con- of run charts. Shewhart charts are discussed in
ceptual point of view provides a solid the next chapter.
foundation but it is only a start. If you
are truly interested in immunizing yourself and
those around you from numerical illiteracy, then
you have to understand variation statistically
▸▸ What Is a Run Chart?
as well as conceptually. This is where statistical The run chart is a practical and easy-to-use
process control (SPC) comes into the discussion. statistical tool that is used to understand the
Although there are a number of tools and methods inherent variation that lives in your data. Al-
that can be grouped under the SPC term, two though there are many different types of Shewhart
primary tools are essential for anyone interested charts there is basically only one way to make
in applying statistical thinking to daily work: the a proper run chart. It is a plot of data over time
run chart and the Shewhart chart (a.k.a. control with the unit of time (e.g., day, week, or month)
chart). These charts are used in order to: always laid out along the horizontal or x axis
and the indicator values (i.e., the data points)
■■ Make the process performance visible
always plotted against the vertical or y axis. Any
■■ Determine if a change statistically demon-
type of data (e.g., a count, a percentage, a rate,
strates an improvement
money, time, a score, an index, or days between
■■ Determine if once an improvement has been
events) can be plotted on a run chart. The data
observed it the improvement has been sus-
must be arranged in chronological order and
tained (Provost and Murray, 2011, p. 85–86)
the median is placed through the field of data
The remainder of this chapter addresses points as a reference line.1 FIGURE 8-1 shows the
the construction, analysis, and interpretation basic elements of a run chart.

© Michal Steflovic/Shutterstock

187
188 Chapter 8 Understanding Variation with Run Charts

6.00

5.75

5.50

5.25

5.00
Measure

4.75
X (CL)
4.50 Median
4.25

4.00

3.75

3.50

3.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920212223242526272829
Time Point Number

FIGURE 8-1 Elements of a run chart

▸▸ How Do I Construct a The median can be placed on the run chart

Run Chart?
Constructing a run chart is a very simple activity.
It can be constructed by hand or with software. I
have made many run charts on flipcharts during
team meetings. I have even made them on a
napkin or a piece of paper at a nursing station.
The following steps should be followed if you
wish to make a run chart by hand:
■■ Select the indicator of interest and make sure
the data have been organized in time order.
In terms of the number of data points you
should have before constructing a run chart,
there are mixed views. Purists would proba-
bly tell you that you should have 12–15 data
points before making a run chart. This may
pose a problem for a team, however, if they
are tracking a measure that has no historical
data. For healthcare applications I support the
perspectives of my colleagues Lloyd Provost
and Sandy Murray who write that “a run
chart should begin when data for the first
point to be plotted is available” (2011, 86).
after four or five data points have been col-
lected. You have to realize, however, that as
you obtain more data the median will move
a little. Typically it will not move as much
as the mean when there is a small amount
of data but it will fluctuate some. When you
apply the run chart rules (described in the
next section) you should have a minimum
of 10 data points. This issue of how much
data is needed to make a chart relates back
to the discussion in the previous chapter on
developing a data collection plan. Remember
that run and Shewhart charts are designed
to help you understand the variation in the
current operation of a process. So if you are
collecting data at the patient level (e.g., wait
time for a patient to see the doctor) you will
most likely have more than enough data to
make a run chart in a day. But as you extend the
time frame for your data collection frequency
and move to days, weeks, or months you will
have to wait a longer period of time to obtain
sufficient data to make a run or Shewhart
chart. Again, I would suggest steering away

from even thinking about using quarterly
data to make any of the charts.
■■ Lay out the horizontal (x axis) and vertical
axis (y axis) and label them clearly. It is rec-
ommended that you extend the horizontal
axis beyond the last data point you have
collected so that future data can be plotted.
On the y axis, which is the reference axis
for the actual data values, make sure you
extend this axis about ± 20% more than the
maximum or minimum data values in order
to make sure the chart has ample space for
future data values that may be greater than
any that you have already collected. This
will also provide enough extra space on the
chart to accommodate a target or goal line.
Also do not forget to make a clear title for
the run chart that includes the name of the
indicator, the unit, or facility involved and
the dates for the period of data collection.
■■ Plot the data in chronological order and make
sure each data point is marked by a symbol
of some sort (usually this would be a dot).
■■ It is customary to connect the data points
(i.e., the dots) with a line but note that there
are some situations in which you would not
connect the data points with a line.2
■■ The centerline on a run chart is usually the
median.3 Determining the median on a run
chart is quite simple. Here is what you do:
• First, you need to find the median position.
This is the location in the distribution of
data the median. This is analogous to the
median’s house number or where it lives
in the distribution of data points. To find
the median position use the formula
(n + 1)/2. If you had 25 data points,
for example, this would work out to be
(25 + 1) 2 = 13. In other words, the
median lives at the 13th data position.
Note that if you have an even number
of data points, say 24, you will not
get a whole number. If we put 24 into
the media position equation we get
(24 + 1)/2 = 12.5. In this case, the median
will reside between the 12th and 13th
data points. In this case, you will need
How Do I Construct a Run Chart? 189
to average the two values found at the
12th and 13th positions to determine
the median value.
• Next, you need to find the median value
(i.e., the number that actually resides at
the median position). To do this you take
a piece of paper or a ruler and place it at
the top of the chart. Slide the paper or
ruler down the chart and as you do you
will reveal the data points in descending
order. Because the y axis is calibrated
from low to high, as you slide the paper
down the page you are viewing the data
in descending rank order, which is a
requirement for calculating the me-
dian. As you slide the paper down the
chart and reveal each data point place
a checkmark next to it. Continue to
reveal the dots until you reach the 13th
data point (using our example where
the media position = 13).
• Now draw a horizontal line through
the 13th data point and see where it
intersects the y axis. The median value
is the point on the y axis where the hor-
izontal line ends. If we use the second
example where the median position was
12.5 all you would need to do is draw
the horizontal line equidistant between
the 12th and 13th data points and see
where that line intersects the y axis to
find the median value.
• To verify that you have done this cor-
rectly you can reverse the process by
sliding the paper up the chart and you
should end up at the same position as
you did when you slid the paper down
the chart and you should end up with
the same median value.
• A final point. The median position and
the median value will usually not be
the same. It is possible that this could
occur but extremely rare. The median
position is merely the location or place
where you will find the median and the
other is the actual numerical value of
the median.
190 Chapter 8 Understanding Variation with Run Charts

Sometimes explaining in words some of these
statistical procedures can be more confusing than
showing you how it is done. FIGURE 8-2 provides a
picture of how these steps are completed to discover
the median value of 4.6. Note that this is the same
chart as was shown in Figure 8-1 displaying the
elements of a run chart. I keep referring to this
same graph throughout this section so you can see
the progression of constructing and interpreting
a run chart. Once you are comfortable with this
method for finding the median (shown symbol-
ically as X with a ~ above it) you will actually be
able to compute the median faster by hand than
entering all the data into the computer and using
software to produce the median value.
▸▸ How Do I Analyze a
Run Chart?
Once you have laid out a run chart, the first
question to be answered is, “What is a run?”
A run is defined as one or more consecutive
data points on the same side of the median
(Lloyd, 2010). FIGURE 8-3 shows how runs are
determined. The first thing to do is to identify
the data points that fall exactly on the median.
These data points need to be identified but they
are ignored when determining the number of
runs. You can count the number of runs by
(1) drawing a circle around each run and
counting the number of circles you have
drawn or (2) count the number of times the
sequence of data points (the line on the run
chart) crosses the median and add 1 to this
count. If you have done this correctly, the
number of circles you have drawn around the
clusters of data points (i.e., the runs) should
match the number you get when you count
the number of times the data line crosses the
median plus 1.4
In Figure 8-3, there are two data points
that fall exactly on the median value of 4.6.
These two data points are highlighted by plac-
ing a square around each one to remind us to

Formula to find the Median Position (n + 1)/2

(29 + 1)/2 = 30/2 = 15 = Median Position
6.00

5.75 When you slide a piece of paper

down, you reveal the dots in
5.50 descending order. When you have
revealed the 15th data point you have
5.25 found where the median lives.
Measure

5.00
The Median
4.75 Lives here at
the15th data
4.50 point
4.25
But, the
Median
4.00
Value = 4.6
3.75

3.50

3.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920212223242526272829
Time Point Number

FIGURE 8-2 Locating the median on a run chart


6.00
5.75
5.50 14 runs
5.25
5.00
Measure
4.75
4.50
4.25
4.00
3.75
(don’t count these when counting
3.50
3.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1920212223242526272829
Point Number
FIGURE 8-3 Determining the number of runs on run chart
exclude them from the identifying the number
of runs. Circles have been drawn around the
runs in Figure 8-3. Fourteen runs are present
on this chart. If you count the number of times
the data line crosses the median you will end
up with 13 crosses. If you add 1 to this num-
ber you end up with 14, which is the same as
the number of circles highlighting the runs.
The first run contains two data points, then
the next run of two data points is below the
median line. Then the next run of one data
point goes back above the median. Remember
that a run is defined as one or more consecu-
tive points on the same side of the median, so
you can have a run with one data point in it.
The other technical point to remember when
counting runs is that because you are ignoring
data points that fall on the median, these data
points do not get included in the determination
of a run. You can see this in Figure 8-3. Notice
that the circles do not include the data points
on the median (the median data points are
identified by the two boxes).
How Do I Analyze a Run Chart? 191
Median=4.610
Points on the Median
the number of runs)
When people are just starting out with
run charts, they have a tendency to (1) include
the data point(s) on the median in the run, or
(2) count the data before the point on the me-
dian as one run and then count the data after
the point on the median as a second run. For
example look at FIGURE 8-4. The chart has six runs
shown by the circles. If you count the number of
times the data line crosses the median you get 5
plus 1 equals 6 runs. So either way you get the
same result. The first run in Figure 8-4 contains
six data points. It does not contain seven data
points, which would include the median data
point, and it does not reflect two separate runs
broken by the second data point, which falls
on the median. The same logic is the same for
the fourth run, which contains five data points.
Notice that there is a data point on the median
but it is not part of the run. This point of ignor-
ing points on the median when determining a
run is probably the only real challenge when
determining the number of runs but with a
little practice you will master this step quickly.
192 Chapter 8 Understanding Variation with Run Charts
50
45
40
Number of Patient Complaints
Don’t count
35
data points on
the median
30
25
20
15
Run of 6
10
5 Don’t count data points on the median
0
1 2 3 4
FIGURE 8-4 Determining the number of runs on a patient complaints run chart (6 runs)
Once we have identified the number of
runs on the chart, the next question that most
people ask is “So what? I’ve made this nice little
line graph and circled some clusters of data, but
why did I do it?” What can we learn from the
number of runs? The answer should be obvious.
The number of runs allows you to apply statis-
tical rules that will help you identify the type of
variation that lives within your data. A technical
caveat needs to be noted at this point. Earlier we
discussed the types of variation and pointed out
that variation can be classified as either common
cause or special cause. These terms technically
should be reserved for use with Shewhart charts.
Because run charts are not as precise as Shewhart
charts (think x-ray versus an MRI) the proper
terms for describing variation on a run chart
are random and nonrandom patterns rather
than common and special causes of variation.
Now, I know some of you are thinking, “Picky,
picky, picky.” You will hear many people using
the terms common and special to describe the
variation patterns on a run chart. You will have
to decide for yourself if it is worth pointing out
to these folks that the terms common and special
should be applied only to variation analysis on
Shewhart charts and not on run charts. But, at
least you are now aware of this technical point.
Run of 4
Run of 5 Run
of 1
Run of 1 Run of 1
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Week
It is important to realize that there are a
variety of run chart rules that different writers
have offered over the years to identify nonrandom
signals on a run chart. Some of these rules, although
having the same name, for example a “trend,” will
use different decision criteria to decide if a trend
exists or not. This is another important aspect of
run and Shewhart chart analysis. So, do not get
too concerned if when you present a chart and
start to explain how you have analyzed it some-
one says, “But that is not what I was taught was
a trend.” Life is full of options and even among
statisticians there can be considerable variation.5
At the Institute for Healthcare Improvement
(IHI), we have reached agreement on four run
chart rules that we have found to be most relevant
to healthcare applications:
■■ Rule 1: A shift in the data
■■ Rule 2: A trend in the data
■■ Rule 3: Too many or too few runs in the data
■■ Rule 4: An astronomical data point
Rule 1: A Shift in the Data
When do you know if a process has really moved
to a new level of performance? If you suffer from
numerical illiteracy, you will probably think that
one or two data points that are “better” (however

defined) constitute a shift to an improved level.
When looking at data on a run chart, however,
the key to defining a shift in the process is when
you have a run that contains six or more con-
secutive data points on one side of the median
or the other (Lloyd, 2010). According to Provost
and Murray (2011, 77) the probability of this
occurring by change alone when there has been
no real change made to the process is less than
5% chance. In Figure 8-3, a shift is observed in
the fourth run where six data points are in a
run that is below the median. The question is
what were the conditions affecting the process
that caused the data to stay below the median
for six points in a row? The run chart rule does
not provide an answer but merely points you to a
period when a nonrandom pattern was detected.
In Figure 8-4, the process starts out with a shift
of six data points in a row below the median.
Then a random pattern emerges for the rest of
the chart. Often when a run chart starts out with
a shift below the median it can be because the
team was still learning how to collect the data
and properly apply the operational definitions
for the indicators. After a period of learning
how to apply the operational definition and data
collection plan, however, the true performance of
the process is observed. Again, the detection of
a nonrandom rule does not provide the answers
but should lead the team to raise question.
Rule 2: A Trend in the Data
This is probably one of the more difficult rules
to apply because of all the popular notions about
what constitutes a “trend.” People love to find
a trend. This issue was discussed earlier in this
chapter and is not elaborated further in this
section. The key point to remember, however, is
that at the IHI we define a trend statistically as
five or more consecutive data points constantly
going up or constantly going down (Lloyd, 2010;
Provost and Murray, 2011). This choice, which
is at odds with other decision rules on trends
(see for example Pyzdek, 1990; Western Electric
Company, 1985) is based on our practical expe-
rience of applying run chart rules to healthcare
How Do I Analyze a Run Chart? 193
data and the fact that more stringent criteria for
a trend (e.g., six or seven constantly increasing
data points for a trend) require more data that
many healthcare teams do not have the luxury
of collecting, especially when they are just start-
ing their improvement efforts. So, the decision
to use five constantly increasing or decreasing
data points as a trend is grounded not only in
statistical theory (i.e., the probability of getting
five data points that constantly increase or
decrease is grounded is the same probability as
getting five heads in a row on a coin flip or .031
probability) but also practical experience. Two
final points to remember when deciding whether
a trend exists: (1) the median does not come
into consideration when determining whether
a trend exists so a data point that falls on the
median can be considered as part of a trend,
and (2) ignore points that repeat the previous
value. FIGURE 8-5A shows a run chart with 29
data points with 1 on the median (marked by a
square) and 13 runs. Do you see a trend in this
chart? Actually there are two downward trends
highlighted in FIGURE 8-5B. Notice how both of
the trends cross over the median. Remember
that a trend is not dependent on the median.
Rules 1 and 3, however, do require the median to
make a determination as to whether nonrandom
variation is present in the run chart.
Rule 3: Too Many or Too Few Runs
in the Data
Another nonrandom pattern is defined as too
much or too little variation in a dataset. Like Rule
1, Rule 3 is based on probability theory. When
applying this rule, the first thing that you need to
do is to determine the number of “useful obser-
vations” in your dataset. As stated earlier, this is
determined by subtracting the number of data
points on the median from the total number of
data points. Once you have determined the number
of useful observations in your dataset, you refer
to a table (TABLE 8-1) to identify the minimum
and maximum number of runs that should be
observed statistically for each respective amount
of data that is defined as useful observations.
194 Chapter 8 Understanding Variation with Run Charts

Run Chart Median

90

80

70

60

50
Visits

40

30

20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

FIGURE 8-5A Run chart with 13 runs

Run Chart Median

90

80

70

60

50
Visits

40

30

20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

FIGURE 8-5B Run chart with 13 runs and two downward trends

The table most commonly used is one developed
in 1943 by Swed and Eisenhart. If the number of
runs on your chart falls between this minimum
and maximum number of runs then you do not
have a nonrandom pattern. For example, if you
had 23 useful observations Table 8-1 indicates
that the minimum number of runs should be 7
and the maximum number of runs should be 17.
If your run chart had less than 7 (i.e., too few
runs) or more than 17 runs (i.e., too many runs)
this would be a statistical signal that your data
exhibit a nonrandom pattern.
To properly use Table 8-1 you:
■■ First calculating the number of “useful ob-
servations” in your dataset. This is done by
subtracting the number of data points on the
median from the total number of data points.
■■ Then, find this number in the first column.
The lower number of expected runs for this
 How Do I Analyze a Run Chart? 195

TABLE 8-1 Table to determine whether there are too many or too few runs on
a run chart

Number of Useful Lower Number of Upper Number of

Observations Expected Runs Expected Runs

10 3 9

11 3 10

12 3 11

13 4 11

14 4 12

15 5 12

16 5 13

17 5 13

18 6 14

19 6 15

20 6 16

21 7 16

22 7 17

23 7 17

24 8 18

25 8 18

26 9 19

27 10 19

28 10 20

(continues)
196 Chapter 8 Understanding Variation with Run Charts

TABLE 8-1 Table to determine whether there are too many or too few runs on
a run chart (continued)

Number of Useful Lower Number of Upper Number of

Observations Expected Runs Expected Runs

29 10 20

30 11 21

31 11 22

32 11 23

33 12 23

34 12 24

35 12 24

36 13 25

37 13 25

38 14 26

39 14 26

40 15 27

Excerpted from: Swed, F. and Eisenhart, C. (1943) “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.” Annals of Mathematical
Statistics. Vol. XIV, pp. 66–87, Tables II and III. This is a segment of a longer table that goes from 10 to 60 useful observation.

number of useful observations is found in
the second column.
■■ The upper number of expected runs can
be found in the third column. If the num-
ber of runs in your data falls below the
lower number of expected runs or above
the upper number of expected runs then
this is a statistical signal of a nonrandom
pattern.
FIGURE 8-6 shows a run chart with too many
runs. The chart has 25 data points with 1 data
point on the median so there are 24 useful data
points. Table 8-1 indicates that for 24 useful
data points we should find 8 to 18 runs if the
chart displays only random variation. Figure 8-6
has 21 runs (20 crossings of the median plus
1 = 21 runs), which exceeds the upper num-
ber of expected runs of 18 runs thus signaling
the presence of a nonrandom pattern. This is
a signal that the number of home care visits
reflects a nonrandom pattern that is unstable
and therefore unpredictable. The team needs to
investigate why the number of home care visits
keeps fluctuating so much.
 How Do I Analyze a Run Chart? 197

Number of Home care Visits

25

20

15
Number

Median = 11
10

5
Data point on the median

0
M T W Th F M T W Th F M T W Th F M T W Th F M T W Th F
Day of the week

FIGURE 8-6 An example of too many runs on a run chart

Run Chart for Inpatient Falls by Month

70

60

50
Number of Falls

40
Median = 39.5
30

20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Month

FIGURE 8-7 An example of too few runs on a run chart

The other aspect of Rule 3 (i.e., too few runs)
is shown in FIGURE 8-7. This run chart contains
24 data points with no data points on the median,
which gives us 24 useful observations. If we look
up 24 useful observations in Table 8-1 we see
that if the data exhibited just random variation
we should observe between 8 and 18 runs but
we detect only 2 runs. Did you also notice that
Rule 1 is present in Figure 8-7? There are two
shifts in the data. Both contain runs with 12 data
points. The occurrence of Rule 1 and Rule 3 will
often be observed together. The reason is that
if you have several runs on a chart that contain
six or more data points indicating shifts in the
data, then this will reduce the opportunity to
have more runs on the chart and increase the
probability of having too few runs. In the case
of Figure 8-7, half the data are hanging on one
side of the chart and the other half of the data
have shifted (for some unknown reason at this
point) above the median. The challenge for the
team is to determine why the number of inpatient
falls has shifted upward to a new (and probably
less desirable) level of performance. Did they
198 Chapter 8 Understanding Variation with Run Charts

institute a change between month 12 and 13? As have a high and a low data point they basically
an improvement advisor to this team I would ask balance each other out.6 We are looking for one
them to annotate the run chart and show where point that is egregiously different from the rest.
they made changes in the process so we could Now what do we do with this data point? You
see if in fact their intervention(s) have caused have several options:
the shift. If their actions do not coincide with 1. First, I would investigate whether
the shift then we would need to explore other
this data point is based on the same
possible cause for the shift. operational definition as the other
data points. Someone new to the team
Rule 4: An Astronomical Data who was collecting the data might
have and included (or excluded) ob-
Point servations that were not included (or
This is not a statistical rule but an observational excluded) in the rest of the data. It is
rule that serves as a filter to guide further in- also possible that the data collection
vestigation and dialogue. I refer to this as the process was different for data point 24.
interocular test of significance. An astronomical So when you see an extreme data
data point is one data point that is dramatically point before jumping to conclusions
different from the rest. Look at FIGURE 8-8A for check the operational definition and
an example of an astronomical data point. Data data collection plan that was used
point 24 on the far right of the run chart clearly during the week in question.
looks like it is very different from the field of 2. If you are satisfied that a different
data points lower on the chart. All you have operational definition and/or the data
to do is look at the chart and consensus would collection plan used during week 24
most likely be that data point 24 is different. The was not different from that used for
key to applying this rule is that it relates to only the rest of the data, then it is time
one data point. In every dataset, you will have a to place all 25 weeks of data on a
high data point and a low data point or possibly Shewhart chart and see whether the
several high and several low points. These are data for week 24 exceeded the upper
not astronomical data points. In fact, when you control limit (UCL). FIGURE 8-8B

25
What do you think about this data point?
Number of Cancelled Appointments

Is it astronomical?
20

15

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Week

FIGURE 8-8A An astronomical data point

 How Do I Analyze a Run Chart? 199

25
The Shewhart chart confirms that this is an

Number of Cancelled Appointments

astronomical data points since it exceeds
20 the upper control limit

15

UCL
10

LCL
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Week

FIGURE 8-8B Verifying an astronomical data point on a Shewhart chart

shows how this would be done.
Placing the data on a Shewhart chart
will provide clear confirmation as to
whether this data point was astro-
nomical or not. More will be said
on the use of control limits in the
next chapter.
3. Finally, the team needs to conduct
a root cause analysis and determine
what the conditions were during week
24 that led to a much higher number
of cancelled appointments. If they do
not investigate this data point and
take steps to extricate the factors that
caused this astronomical data point
to occur then they can expect to see
the point pop up at some point in the
future. No one can predict exactly
when this will happen again. But
because the conditions that caused
it initially would still be inherent in
the process or system (Reason, 2003)
you can count on it happening at
some point in the future.

CASE STUDY #1: Hand Hygiene Compliance

As improvement advisor to a team trying to improve hand hygiene compliance you told them that
at the next meeting you would show them how to make a run chart and determine if their recent
interventions have made a difference in the key indicator, percentage of compliance with proper hand
hygiene. The indicator is defined as the percentage of compliance with proper hand hygiene by week
where the numerator is the total number of properly completed hand washings and the denominator
is the total number of hand washing observations performed in the week.
The data the team collected each week were obtained through observations of a stratified
random sample of staff on three different units. The data along with the initial run chart you created on
your laptop in the meeting are shown in FIGURE 8-9.

(continues)
200 Chapter 8 Understanding Variation with Run Charts

CASE STUDY #1: Hand Hygiene Compliance (continued)

Percent Compliance
90

85
Percent Compliance

80

75

70
Median = 81

65
How many runs on this chart

60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Week

Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Percent
Compliance 79 82 86 84 85 79 77 86 82 74 85 74 78 83 81 81 74 84 78 75 74 68 81 84 70 85 77

FIGURE 8-9 Hand hygiene compliance data for 27 weeks and initial run chart

Even though you made the run chart on your laptop using SPC software, you provide all the team
members with a paper copy of the chart so you can:
■■ Walk the team through the calculation of the median and make sure they understand how it
was obtained using the formula to find the median position [(n = 1)/2]. In this case, 27 data
points plus 1 = 28 divided by 2 = 14 which is the median position or the location of the median
in the array of data.
■■ Then you have them all slide a piece of paper down the chart to reveal the data points in
descending order.
■■ Because the formula indicated that the median position is 14 you have them stop revealing the
data points when they arrive at the 14th point. They then discover that there are actually three
data points at this location.
■■ You then have them draw a line through the points at the 14th position and have it intersect the
vertical (y axis). This line points to the median value of 81. You have the team members confirm
this by referring to the data in Figure 8-1 where they find that there are in fact three data points
at the value of 81% compliance.
■■ The next question for the team is, “How many runs are there on this chart?”
■■ You ask them all to determine the number of runs and when they have a number to shout it
out. Someone says “14” then another says “15” and after a period of contemplation another team

(continues)
 How Do I Analyze a Run Chart? 201

CASE STUDY #1: Hand Hygiene Compliance (continued)

Percent Compliance
90
3 data points on the median

85
Percent Compliance

80

75

70
Median = 81

65
15 runs
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Week

FIGURE 8-10 Number of runs for the hand hygiene compliance run chart

member says “16.” So now the learning begins. You describe the definition of a run, which you
had covered at the last meeting, then show them the two methods for determining the number
of runs (i.e., draw circles around the clusters of dots on either side of the median or count the
number of times the data line crosses the median line and add 1). Both methods should produce
the same number of runs. After a little reflection the team arrives at consensus: 15 runs as shown
in FIGURE 8-10.
■■ Now that the team members know the number of runs on the chart the next step is to have the
team analyze the run chart and determine whether any of the run chart rules are detected. Note
that as the improvement advisor to the team your job is not to analyze the chart for them. The
data are their data not yours. Your job is to provide guidance not answers.
■■ You review the four run chart rules with the team (that you printed on the back of the chart
you gave them) and ask them to interpret the chart. FIGURE 8-11 shows the result of the
analysis. A downward trend was detected from weeks 18–22, then random variation was
observed after week 22. Even though the team had put in place a hand washing
campaign during weeks 16 and 17 they realized that this was prior to the start of summer
vacations and the rotation of the resident doctors. So although the results were not
what the team expected they did gain great insights about factors affecting their
improvement efforts. Now, the next question is, “What are you going to do to make sure
the process works reliably during the next vacation period and when residents come on
the scene?”
202 Chapter 8 Understanding Variation with Run Charts

CASE STUDY #1: Hand Hygiene Compliance (continued)

Percent Compliance
90
A downward trend

85
Percent Compliance

80

75

70

65 Apply the rules and interpret the chart.

Note: 27 data points with 3 on the median gives you 24 “useful observations.”
For 24 useful observations you expect between 8 and 18 runs.
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Week

FIGURE 8-11 Interpreting the hand hygiene run chart

▸▸ A Few Closing Thoughts start to plot it. At first this can just be a simple
line graph. When you get four or five data points
on Using Run Charts you can place an initial median on these data to
see how the data are centering around a common
As you gain more knowledge of and experience point but I find it useful to refer to these early
with run charts there are a few additional things medians as “trail medians” because the median
to think about. Specifically the following questions may change until you obtain sufficient data
are the ones I get asked most often: that you have a reasonably stable distribution.
■■ Is there a minimum and maximum amount This stability will start to occur with a median
of data I can put on a run chart? when 10 or more data points are on the chart
■■ When do I change the median on a run chart? (Provost and Murray, 2011). At the other end of
■■ Why do I need to annotate the run chart? the spectrum is the question of the maximum
amount of data that should be placed on a run
chart. You can easily place a lot of data on a run
Is There a Minimum and chart especially if you use a computer. In general,
Maximum Amount of Data I Can however, once you get upwards of 40–50 data
points you have exceeded the practical design
Put on a Run Chart? of what a run chart is trying to convey.
At the beginning of this section I mentioned that Two issues will influence the amount of
as soon as you start collecting data you should data you should consider placing on the chart.
 A Few Closing Thoughts on Using Run Charts
First is the issue of current versus old data.
Remember that the primary use of a run chart
(or Shewhart chart) is to help you understand
current performance. So when you place data
that go back 2, 3, 4, or more years on a chart
these are historical data not real-time data.
Wheeler and Chambers refer to charts like
this as report card charts. They point out that
although the chart may be valid and constructed
properly it is a “weak use of the charts” (1992,
p. 18). The second issue that will influence the
amount of data you place on a chart depends
on how you have organized the data (e.g., by
patient, day, week, or month). How you have
organized your data, referred to as the subgroup,
will provide different amounts of data for the
chart. For example, if you are tracking patient
wait time to see the doctor in a family practice
clinic that sees approximately 50 patients a day
you will end up with over 200 data points in a
week if you recorded every patient’s wait time
to see the doctor. This is way too much data. If
you make the subgroup day instead of every
patient, however, and select a sample of patients
each day (e.g., 5–10) then you will have a more
manageable amount of data to place on a chart.
When Do I Change the Median on
a Run Chart?
The answer to this question is essential in order
to decide whether the change a team has made
to a process or the system actually made a dif-
ference. Here are the steps to follow in order to
answer this question:
■■ Establish a baseline for the indicator of in-
terest. Ideally it would be nice to have a solid
baseline that established the performance
of the indicator prior to any intervention.
Again, the number of data points in this
baseline will depend on how you have or-
ganized your subgroups (i.e., the labels on
the horizontal axis of your run chart) and
your data collection plan. The following are
not strict recommendations based on hard
and fast statistical rules but rather guidelines
203
I have used with teams that have proven
to be practical yet useful. You will have to
discuss with your team what seems to be
reasonable for your work.
• If you have organized your data into
monthly subgroups it would be desirable
to have approximately 10–12 months as
the baseline. However, you may need
to make compromises. For example,
when I was setting up the measurement
system for our work with the Scottish
Patient Safety Program (SPSP), I made
a practical decision because many of
the indicators had not been collected
before at the 42 hospitals. So, we settled
on collecting the first 6 months of data
for newly developed indicators and
calling this the baseline. Not ideal but
practical.
• If you have sufficient data to organize
your data into weeks as the subgroup,
then having a baseline of 10–12 weeks
would provide an acceptable baseline
(e.g., the percentage of newly admitted
inpatients properly assessed for pressure
ulcers each week).
• If you have organized your data into
daily subgroups then 12–15 days of
data could serve as a baseline (e.g.,
the number of food trays produced
each day).
• If you have organized your data by
individual patients, tests, surgeries, or
some other discrete unit that is essen-
tially an n of 1, then I would build a
baseline with roughly 25–30 individual
occurrences (e.g., the wait time for each
patient to see the doctor at the family
practice clinic or the turnaround time
in minutes for a stat medication order).
■■ Once you have a baseline, calculate the median
for this block of data. Then you “freeze” the
median for the baseline and extend it into
the future as a reference marker. Ideally the
baseline period is the time when the process
was functioning without any improvement
work occurring. At the end of the baseline
204 Chapter 8 Understanding Variation with Run Charts
period you would draw a vertical line on
the run chart to show that this is the point
at which the team began to test new ideas
that they believed would improve the per-
formance of the indicator being tracked. The
vertical line on the chart provides a visual
demarcation point between the old way and
the improvement journey. The question is,
what will happen to the indicator after the
team tested new ideas for improvement. ■■
The new data will be plotted against the
baseline median. The run chart rules will
then be applied to the new data using the
baseline median as the reference point. This
is shown in FIGURE 8-12A.
■■ FIGURE 8-12B shows how the new data after
the intervention performed against the
baseline median that was extended. Not only
did six data points stay below the baseline
median indicating a shift in the data but
the remainder of the new data also stayed
below the baseline median. This not only
indicates that there has been a downward
shift in the wait times but also that this shift
has been sustained.
■■ FIGURE 8-12C shows the final step in calculat-
ing a second median. Once the nonrandom ■■
pattern (Rule 1: a shift) has been detected it
is now appropriate to create a new median
Wait Time to See the Doctor
30
Baseline
25
Wait Time (in minutes)
20
15
10
Intervention
5 run chart rule has been detected.
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Individual Patients
FIGURES 8-12A Freezing the median on a run chart
for the new data. In Figure 8-12c, we
see that not only has the median shifted
downward but also the variation in the wait
time to see the doctors has been reduced
(i.e., the dots are all closer together). This
actually indicates that two improvements
in the process have occurred: (1) a shift in
the median in the desired direction and
(2) a reduction in the variation.
If the team planned further improvements,
this sequence of freezing and extending the
second median, making the next intervention,
and then plotting the new data against the
second frozen median would be the strat-
egy. If after making the next improvement
intervention the data did not demonstrate a
shift or trend then we would conclude that
the intervention did not have the expected
impact on the process. The team would then
need to rethink its strategy. Was the idea the
wrong idea? Did we have the wrong theory
about how this idea would affect the process?
Did we collect the data in the same way we
did previously? All these questions would
be part of the team’s learning based on the
results they observed.
In summary, the sequence for creating a
new median is to establish a baseline, create
the initial baseline median, freeze it, and
Freeze the Median, extend it and
compare the new process performance
to this reference line to determine if a
 A Few Closing Thoughts on Using Run Charts 205

Wait Time to See the Doctor

30
Baseline
25
Wait Time (in minutes)

20

15

10
Intervention
5

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Individual Patients

A nonrandom pattern is deteced

A run of 6 or more data points below the
median is signal of a shift in the process.
Note that the rest of the data also is below the
median which shows the shift has been
sustained.

FIGURES 8-12B Plotting the new data against the baseline median

Wait Time to See the Doctor

30
Baseline
25
Wait Time (in minutes)

20 Calculate a new median on for the

process to show the improvement
15

10

Intervention
5

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Individual Patients

FIGURES 8-12C Creating a second median on new level of performance

then extend it into the future. Next, put in
place the new idea that the team believes
will make an improvement in performance.
Plot new data against the frozen median
to see if a run chart rule is detected. If it
is, then start the sequence all over. On the
other hand, if a run chart rule is not evident
then the team needs to evaluate why the
improvement idea it introduced did not
have the expected impact.
206 Chapter 8 Understanding Variation with Run Charts
Why Do I Need to Annotate the
Run Chart?
A challenge with any form of data analysis in
this day and age is that it is very easy to push
a button on a computer and get a variety of
numbers, charts, and graphs. Some of these
displays of data are very intricate and colorful.
Some are three dimensional and some you can
even rotate. Yet, I think we frequently rely on
the machine at our fingertips and not the one
that rests upon our shoulders. Pushing the
buttons on the computer to get statistical and
graphical results is in many ways the easy answer.
Computers produce data not information. The
goal is to turn data into information (Austin,
1983), which requires the machine on the top
of your shoulders to be functioning fully and
engaged.
Annotating your charts is a perfect way to
start moving from data to information. Items
that should be annotated on a chart include:
■■ The baseline period.
■■ The point at which the improvement team
began testing new ideas. For example,
annotating every Plan–Do–Study–Act
(PDSA) cycle the team ran provides a
well-documented journey.
■■ Annotating the detection of a run chart
rule (e.g., a shift or trend) on the chart is
very beneficial especially when present-
ing your work to a quality council or in a
management meeting where there might
be individuals who are not familiar with
run chart analysis and interpretation.
Moments like this provide a good op-
portunity for you to help educate others
in your organization about run charts.
But without annotations the key learning
points might be missed.
■■ Besides annotating the actual run chart
other useful annotations include:
•
Making sure you have a full and complete
title on the chart. This should include
a primary title identifying the facility
or unit where the improvement work
is occurring. This is usually in a little
larger font (e.g., a size 14 or 16).
• The secondary title for a chart should
indicate the indicator by name and the
dates for the data (e.g., Percentage of
Compliance with the Ventilator Bundle,
January 2017–March 2018).
• Try to leave space to insert the oper-
ational definition for the data. If the
chart takes up most of the page of
your handout you can always place
the operational definition and data
collection plan on the reverse side of
the page. If you are using slides, try
reducing the size of the chart a little
so you can place a small footnote (in
a text box) at the bottom of the slide
summarizing the operational definition
and data collection plan.
• If the indicator is a percentage or a
rate it is always a good idea to at least
provide a note (usually at the bottom of
the chart under the horizontal axis) that
reports the minimum and maximum
size of the denominators. You do not
necessarily have to include a full data
table, which often is entirely too small
and hard to read, but indicating the size
of the denominators (min to max) will
usually be enough to keep people from
asking about them.
• Finally, I find it useful to put the name
of the individual who made the chart
and contact information (in about a
size 10 font) somewhere on the chart
in case there are ­follow-up questions
about the chart or its interpretation.
This also comes in handy when ma-
terials handed out at a meeting are
sent around to people who did not
attend the meeting and they would like
to talk to someone about the chart.
FIGURE 8-13 provides an example of applying
these annotation suggestions to a run chart.
 Notes 207

Stay Healthy Family Practice Clinic

Wait Time to See the Doctor, June 6th through June 17th

Wait Time to See the Doctor

30
Baseline (week of June 6th)
25
PDSA#1
Wait Time (in minutes)

PDSA#3
June 13th PDSA#2
20 June 16th
June 15th

15

10
Team began PDSA testing here
5 June 13th

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Individual Patients

Operational Definition: Wait Time to See the Doctor is defined as the amount of time (measured in whole
minutes) from when the patient checks in at the reception desk until he or she is seen by the doctor in the
exam room.

Data Collection Plan: A systematic sample of 5 patients is pulled on 3 randomly selected days each week.
Data are pulled during regular office hours. Weekends and scheduled holidays are excluded.
This chart was prepared by R. Lloyd who can be reached at Extension 1234.
FIGURE 8-13 Example of an annotated run chart

Notes 2. The data points on a run or Shewhart

1. There is frequent confusion over the
difference between the mean and the
median. They are both measures of cen-
tral tendency (i.e., they identify where a
distribution of data is centered), but they
are different. The mean is the arithmetic
average (you add up all the numbers
and divide by the total number of data
points). The median, on the other hand,
is the midpoint of a distribution of data
points. This is the numerical value that
divides the dataset exactly in half, the
50th percentile. The run chart uses the
median as a centerline, but the control
chart uses the mean as the centerline.
chart are usually not connected when you
are interested in comparing the variation
among members of a group (e.g., indi-
vidual hospitals, clinics, or doctors) at a
fixed point in time. For example, imagine
that you are part of a nine-hospital system
and want to see how all nine hospitals
compare on the percentage of patients
being properly assessed for the risk of a
fall. Are all our hospitals performing as
a system or do we have some doing very
well and others not so well? In this case,
you would collect data on each of the nine
hospitals for 15 to 20 months and then
make a run chart where each hospital
208 Chapter 8 Understanding Variation with Run Charts
would be arranged along the horizontal
axis of the run chart. The hospitals could
be in alphabetical order, for example, or
by bed size. The order of the units being
compared would not matter for the pur-
poses of comparison if the indicator is a
percentage. Each hospital’s data would be
plotted and because this is not data over
time but in the aggregate we would not
connect the dots with a line. Typically,
connecting the dots with a line indicates
that this is a process that is flowing over
time. The objective is to understand the
variation amongst the dots irrespective
of whether you connect them or not.
An example of not connecting the dots
is shown in Chapter 10, Case Study 1.
3. Although it is standard practice to place the
median on a run chart as centerline (Carey
and Lloyd, 2001; Provost and Murray, 2011;
Pyzdek, 1990), some writers (e.g., Torki,
1992) advocate the use of the mean as the
centerline on a run chart. The median is
preferred by most writers because it is a
measure of central tendency (along with
the mean and mode) but more important
it is not sensitive to extreme values as is
the mean. All that matters with using the
median is whether a data point is on this
side of the median or the other side. If a
data point is a small amount above the
median or a very large distance above
the median the median value will not
be affected. This is because the median is
positional (i.e., the 50th percentile point
where half the data are above this point
and half the data are below it). For the
same reason, the median is not affected
by data that are not normally distributed
(i.e., suffering from skewness or kurtosis).
4. I frequently get asked in class, “Why do
we add 1 to the number of crossings?”
The answer is simple. Think of cutting a
loaf of bread. If you cut the loaf of bread
10 times you actually end up with 11 slices
of bread. The first cut actually produces
two pieces of bread; one is the end heel
cut and the rest of the loaf is actually the
second slice. Similarly when the data line
crosses the median the first time you have
two segments of day. Adding 1 to the final
count of the number of crossing accounts
for the two segments that were created with
the first crossing. You can verify this the
next time you cut a loaf of bread. Count
the number of times you have sliced the
loaf and you will discover that you have
one more slice of bread than the number
of cuts you made.
5. I think a lot of people who don’t have
formal training in statistics think that it
is a rather clear and definitive discipline.
It uses numbers, formulae, Greek letters,
and a variety of symbols that certainly
make statistics and math look precise.
We even refer to the precision of the
p-value and talk about the .05 or even
.01 level of significance. But there is a
lot of gray area and debate in the field
of statistics. A classic reference on this
topic appeared in 1970. It is called The
Test of Significance Controversy edited
by Denton Morrison and Ramon Henkel
(1970). If you are a student of statistical
thinking this collection of readings is a
must. A more contemporary and critical
exploration of this topic is The Cult of
Statistical Significance: How the Standard
Error Costs Us Jobs, Justice and Lives by
Stephen Ziliak and Deirdre McCloskey
(2011). I have been fascinated by this de-
bate for years. My family knows this and
at Christmas they bought me a sweatshirt
that captures the ongoing debate nicely. It
states in simple letters on the front of the
sweatshirt “Statistics mean never having
to be certain.”
6. When you have a high and a low data point
on a chart or even a couple high and several
low points they essentially form the tails
of a distribution. Think of the normally
distributed bell-shaped curve. As you go
out the tails of the normal distribution
you expect to find less and less data, but

you will find a high and low data point
residing in the tails of the distribution. But
if you had only one extreme data point in
a distribution it would not take the shape
of the normal bell curve. In this case, you
would have a skewed distribution with
one rather long tail. The astronomical
data point rule is similar to finding a
nonnormal curve with one very long tail.
References
Austin, C. Information Systems for Hospital Administration.
Chicago: Health Administration Press, 1983.
Lloyd, R. “Navigating in the Turbulent Sea of Data: The
Quality Measurement Journey.” Special edition on Quality
Improvement in Neonatology and Perinatal Medicine,
Clinics in Perinatology 37, no. 1 (March 2010): 101–122.
References 209
Morrison, D., and R. Henkel, eds. The Test of Significance
Controversy. Chicago: Aldine, 1970.
Provost, L., and S. Murray. The Health Care Data Guide. San
Francisco: Jossey-Bass, 2011.
Pyzdek, T. Pyzdek’s Guide to SPC. Vol. 1: Fundamentals.
Milwaukee: Quality Press, 1990.
Reason, J. Human Error. Cambridge: Cambridge University
Press, 2003.
Schultz, L. Profiles in Quality. New York: Quality Resources,
1994.
Swed, F., and C. Eisenhart. “Tables for Testing Randomness
of Grouping in a Sequence of Alternatives.” Annals of
Mathematical Statistics 14 (1943): 66–87.
Western Electric Company. Statistical Quality Control
Handbook. Indianapolis: AT&T Technologies, Inc., 1985.
Wheeler, D., and D. Chambers. Understanding Statistical
Process Control. Knoxville, TN: SPC Press, 1992.
Ziliak, S., and D. McCloskey. The Cult of Statistical Signifi-
cance: How the Standard Error Costs Us Jobs, Justice and
Lives. Ann Arbor: University of Michigan Press, 2011.
CHAPTER 9
Understanding Variation
with Shewhart Charts
▸▸ Run Charts versus
Shewhart Charts
For many teams just beginning their quality mea-
surement journey (QMJ) the run chart provides
an excellent starting point. It is easy to construct
with paper and pencil, it does not require a soft-
ware package in order to make one, and it can
be used with any type of data (i.e., time, money,
counts of errors, percentages, rates, scores, or days
between adverse events). Also, the four run chart
rules are easy to understand and apply. So, why
would I want to use a Shewhart chart instead of a
run chart?1 There are basically three reasons why
Shewhart charts are preferable over run charts:
1. Shewhart charts are more sensitive
than run charts.
2. Shewhart charts have the added
feature of control limits and zones,
which run charts do not have.
3. Shewhart charts allow us to more
accurately predict process behavior,
future performance, and process
capability than do run charts.
© Michal Steflovic/Shutterstock
The details related to these differences are dis-
cussed in the remaining sections of this chapter.
▸▸ What Is a Shewhart
Chart?
Like run charts, Shewhart charts are graphic
displays of process variation as it lays itself out
over time. FIGURE 9-1 shows the basic elements of
a Shewhart chart and one of the tests to identify
a special cause (i.e., a data point exceeded the
upper control limit [UCL], signaling too much
variation in the data, which, by the way, you
should recognize as an astronomical data point
on the run chart). A run chart and a Shewhart
chart look similar in that the indicator of interest
and its values are plotted on the vertical or y
axis and the chronological order of the data are
organized by what are called subgroups (e.g.,
by individual patients, by day, week, or month)
along the horizontal or x axis. The data points
are then connected by a line and the mean of the
data points is then plotted as the centerline (CL)
on the Shewhart chart. The presence of control
211
212 Chapter 9 Understanding Variation with Shewhart Charts

Signal of a Upper Control

special Limit

60.0

50.0
Number of Patient Complaints

UCL=46.910
40.0 Data are plotted
in time order
30.0
CL=23.381
20.0 Centerline
(the mean)
10.0

0.0 LCL=0.148
Lower Control
-10.0 Limit
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Week

The unit of time is plotted along the horizontal axis

FIGURE 9-1 Elements of a control

limits on Shewhart charts are major points that
separate it from a run chart.
Shewhart charts are more sensitive than
run charts because the run chart cannot detect
special causes that result from point-to-point
variation. This is because the CL on the run
chart is the median (i.e., the 50th percentile).
The run chart basically allows you to classify
the data points as being only above or below
the median. The actual distance a data point
is from the CL is not an issue on a run chart.
Therefore, if one data point is 2 units above
the median and another point is 22 units
above the median, they will both be treated
the same because they are both on the same
side of the median. The logic for this decision
is related to the definition of the median and
of a run (i.e., one or more data points on the
same side of the median). If these same two
data points (i.e., 2 and 22) were placed on a
Shewhart chart, however, you would notice a
discernable difference because the CL on the
control chart is the mean or average of all the
data point actual values. By using the mean we
are ensuring that the absolute value and the
distance of each data point from the CL will be
considered in determining the variation in the
indicator and if special cause variation exists.
Another reason why Shewhart charts are
more sensitive than a run chart is that Shewhart
charts have the added feature of control limits,
which run charts do not have. The control
limits are properly referred to as the UCL and
the lower control limit (LCL). They are also
referred to as sigma limits. You will probably
hear someone refer to control limits, however,
as confidence intervals, confidence limits, or
even standard deviation (SD) limits, which they
are not (Blalock, 1960; Carey, 2003; Daniel &
Terrell, 1989; Provost & Murray, 2011).
The UCL and LCL basically define the
boundaries of process variation around the
mean. The developer of the chart does not set or
define the UCL and LCL. These are determined
by mathematical formulae and the width of these
limits is dependent on the inherent variation

that lives within the data. The only thing the
developer of the chart can place on the Shewhart
chart is a target or goal and annotations as to
when improvements were introduced.
The control limits enable the Shewhart
charts to have increased precision over the run
chart. A run chart will miss certain nonrandom
patterns that would be detected on a Shewhart
chart as special causes. According to Perla,
Provost, and Murray (2011, p. 47), “The three
probability-based (run chart) rules are used to
objectively analyze a run chart for evidence of
non-random patterns in the data based on an
α error of p < 0.05.” This means that run charts
could miss a nonrandom pattern in the data
approximately 5% of the time. Shewhart chart
rules, on the other hand, will not miss detecting
a special cause. This is why it is recommended
that the terms special and common cause as well
as stable or unstable should be reserved for use
only with Shewhart charts and that the terms
random and nonrandom patterns be applied
to run charts.
Shewhart charts also allow us to more
accurately predict process behavior and future
performance than do the run charts. On a
run chart, if the variation is random the best
prediction of the future performance of an
indicator is the median value. For example,
if a team is trying to improve the wait time
to see a doctor and have plotted the data on
a run chart the median is the best estimate of
future performance. Let’s say that the median
wait time is 27 minutes. If you were present-
ing this data to a team or a committee all you
could say would be, “Ladies and gentlemen, the
median wait time is 27 minutes. The process
reflects only random variation. Therefore, if
we do nothing to change the current process
we can expect to have patients wait about 27
minutes to see the doctor.” On a Shewhart
chart, however, because we have the UCL,
LCL, and the mean as the CL, we have more
precision. In this case, when you present the
data to the team or a committee you would be
able to say, “Ladies and gentlemen, the average
wait time to see a doctor is 35 minutes, the
What Is a Shewhart Chart? 213
UCL is 47 minutes, the lower control limit is
23 minutes and there are no special causes
detected. This means that the process is a stable
and predictable. Therefore, if we do nothing
to change how this process works we can
predict that patients will wait on the average
35 minutes with the possibility that the wait
time could go up as high as 47 minutes or as
low as 23 minutes. In light of the target of
having all patients seen by their doctor within
20 minutes or less, however, you can see that
we have our work cut out for us!”
This scenario provides a summary of how
process capability for the wait time in a clinic
can be based on the parameters calculated for a
Shewhart chart (i.e., the UCL, LCL, and mean).
Classically, process capability is defined as, “The
calculated inherent variability of a characteristic
(indicator) of a product or service. It represents
the best performance of the process over a period
of stable operation” (ASQ, 2005, p. 78). Process
capability is essentially aimed at determining
whether under current operating conditions the
process can meet the predetermined specifications
or achieve the target or goal we have established
(Blank, 1998; Carey, 2003; Kume, 1985; Provost &
Murray, 2011; Western Electric, 1985; Wheeler &
Chambers, 1992).
Besides a verbal summary of the Shewhart
chart parameters using the UCL, LCL, and mean
as described previously, process capability can
also be defined statistically by “a single number
assessment of the ability of the process to meet
specification limits on the quality characteris-
tic(s) of interest (ASQ, 2005, p. 78). When you
move to the statistical indices that capture pro-
cess capability it is necessary to have an upper
specification limit (USL) and a lower specifi-
cation limit (LSL), which are then compared
to the performance of the process as defined
by the UCL, LCL, and the mean.2 Although
these indices have not been used extensively
in healthcare settings I believe that they have
great utility. We have many physiological tests
that have upper and lower preferred levels of
performance (i.e., ­specification limits). These
include such indicators as temperature, blood
214 Chapter 9 Understanding Variation with Shewhart Charts
pressure, hematocrits, neutrophils, white and
red blood cell counts, platelets, and clotting
factors.3
There are many useful books and articles on
the statistical theory behind Shewhart charts, how
to construct them, and how to interpret the results.
I have provided only a brief introduction to the
key principles behind the Shewhart charts. This is
a very rich field of study that has been developed
over the past 100 years. Readers interested in
the detailed aspects of statistical process control
(SPC) and in particular Shewhart charts should
consult the rich variety of books and articles on
this topic. Ones I have found particularly useful
include Benneyan (2001); Benneyan, Lloyd, and
Plsek (2003); Blank (1998); Carey (2003); Carey
and Lloyd (2001); Duncan (1986); Ishikawa
(1989); Mohammed, Worthington, and Woodall
(2008); Montgomery (1991); Provost and Murray
(2011); Pyzdek (1990); Western Electric (1985);
Wheeler (1993, 1995); Wheeler and Chambers
(1992); and Woodall (2006).
If you wish to build a firm foundation in
Shewhart charts and SPC in general, I would
recommend that you read widely on this topic
and read what different authors have written. If
one item you read seems too academic or math-
ematical, read another author’s description and
use of SPC. As you read more of the literature
and different authors at some point there will be
a moment when you say, “Okay, I get it.” Fur-
thermore, if you do not have a reasonably solid
working knowledge of the theory and mechanics
of Shewhart charts and how they are constructed,
it will be rather difficult to successfully apply
them to your improvement work. This becomes
even more problematic when people say “No
problem with the charts. We have software that
makes the charts for us.” This orientation creates
several problems. Although it is easy to push
a few buttons on your computer and “make a
chart” this does not necessarily mean it is the
most appropriate chart for the indicator you are
tracking. More important, the SPC software does
not help at all with interpreting what the chart
is trying to tell you. The chart can come from
the machine in front of you with a keyboard but
the interpretation of the chart and what can be
learned from it must come from the dialogue
that emerges when people with subject matter
knowledge interpret the chart. This requires
knowledge not a computer.
▸▸ Key Questions about
Shewhart Charts
There are three basic questions people typically
ask as we start them on the road to using She-
whart charts:
1. How many data points do I need to
make a Shewhart chart?
2. What is a sigma limit? And, why do
I need three of them?
3. Do I apply the run chart rules to
Shewhart charts?
Each of these is discussed next.
How Many Data Points Do I Need
to Make a Shewhart Chart?
As soon as the team begins to gather data they
should start plotting the data points (dots) on a
chart. At first this will simply be a line graph. A
run chart requires less data because the median
as the CL is not as sensitive to point-to-point
variation as is a Shewhart chart. Also the run
chart rules start to come into play with different
amounts of data. The trend rule can be detected
when you have five or six data points. As Provost
and Murray point out (2011, p. 87) “a trend
will remain a trend no matter the amount of
additional data added to the graph.” The run
chart rules related to a shift and too many or
too few runs, however, require more data to be
detected. The general rule is that a minimum of
10 data points is necessary to properly determine
whether a shift has occurred on the run chart or
whether too many or too few runs are present.
When we move to using Shewhart charts more
data are usually required because (1) the mean
is now used as the CL and the absolute value of

any data point enters into the calculations of the
UCL and LCL, and (2) the rules for detecting
special causes of variation are more rigorous
and precise than the four rules for run charts.
But, the simple answer to the question of how
much data I need to make a Shewhart chart
is . . . it depends. I know some readers will
be thinking, “What kind of a lame answer is
this? Just tell me how many data points I need
to make a Shewhart chart!” Because there are
many types of Shewhart charts, which are dis-
cussed in the next section, it must be realized
that the different charts can be produced with
differing amounts of data. The subgroup, that
is how you have organized your data along
the x or horizontal axis of the chart, is key
to determining how much data you need to
make a particular type of Shewhart chart. For
example, if you want to track the wait time of
each patient at a family practice clinic to see the
doctor then the subgroup is one patient and the
one bit of data for this patient will be her wait
time to see the doctor. If, on the other hand,
you decide that you want to track wait time by
day then the horizontal axis of your chart will
have Monday, Tuesday, Wednesday, etc. rather
than patient 1, patient 2, patient 3, etc. as the
subgroup. Selecting day as the subgroup for
a clinic could now provide upwards of 30–40
patients’ wait times as possible observations (bits
of data) within a single day. Having multiple
data points in a subgroup or only one will play
a major role in deciding which Shewhart chart
you can make. This is why it is very important
to make sure you have a well-thought-out data
collection plan. Again, more will be said about
these issues in the next section when I discuss
the types of Shewhart charts.
All this being said, I do know that many
people still want to have at least some general
guidelines for organizing their data, so here are
a few that I offer to improvement teams as we
begin to work on developing Shewhart charts:4
■■ It is usually recommended that you have
20–30 subgroups of data before construct-
ing a Shewhart chart. These data should be
Key Questions about Shewhart Charts 215
pulled from the process when it is stable
and predictable.
■■ If you have less than 20–30 subgroups of
data you can still create a Shewhart chart but
the UCL and LCL should be referred to as
“trial” control limits (Carey, 2003; Carey &
Lloyd, 2001; Provost & Murray, 2011).
The trial limits can be used for learning
but the use of the word “trial” is to remind
those using the chart that these limits may
change as more complete data are obtained
and make the limits more reliable and stable.
The issue here is that when you have less
than the recommended amount of data
(i.e., 12–15 data points) the control limits
and CL (i.e., the mean) can change rather
quickly and dramatically with the addition
of each new data point. With a fewer num-
ber of subgroups you also run the risk of
committing a type II error (i.e., concluding
that the chart indicates no special causes
when in fact one or more special causes do
exist). When you start to have more than
the recommended 20–30 subgroups of data,
say 40–50, you run the risk of committing
a type I error (i.e., finding special causes
by chance alone). Additional detail on the
theory and use of the type I and type II
error concepts can be found in Carey and
Lloyd (2001), Carey (2003), and Provost and
Murray (2011). In summary, the underlying
question here is how much data do you
need to create a reasonably stable distri-
bution? Different disciplines recommend
different amounts of data needed to form
a distribution (e.g., from only a few data
points to over 500) but generally speaking
a reasonably stable distribution of data for
improvement purposes occurs when you
have 20–30 subgroups of data.
■■ As a general rule I also recommend not
using quarterly data for your improvement
efforts. There is just too much variation
being aggregated in quarterly data to be
useful for improvement efforts. A quarter
consists of 3 months approximately 90 days
and over 2100 hours. During this time a
216 Chapter 9 Understanding Variation with Shewhart Charts
great deal of variation can occur. So when
someone begins making conclusions about
the quarterly average or SD you should ask
them to provide the actual variation that
produced these summary statistics by day,
week or at a minimum by month.
What Is a Sigma? And Why Do
I Need Three of Them?
These two questions probably pose the most
challenging technical aspects of Shewhart chart
construction. Some of you will really enjoy this
issue and want to learn more whereas others
of you will say, “I really don’t care about this
statistical distinction just make the chart that is
most appropriate for my indicator and tell me
what it means.” Either position is fine. I do not
intend to go into great detail on this topic but
I do want to frame it properly so that you can
decide if you want to learn more or accept the
fact that these statistical principles have been
discussed, debated, and written about extensively
for many decades.
Let’s start with the basics. Whenever you
have an array of data you need to consider three
characteristics of the distribution these data
create: the central tendency of the distribution,
the dispersion or spread of the distribution,
and the shape of the distribution. You were
acquainted with these characteristics when
you took your first statistics class, which was
probably a number of years ago. So, this should
all sound rather familiar even if you have not
used the concepts in a while. Measures of central
tendency include the mean (i.e., the arithmetic
average), the median (i.e., the midpoint of the
distribution or 50th percentile), and the mode
(i.e., the most frequently occurring number).
Measures of dispersion include the minimum
and maximum values, the range (i.e., the absolute
difference between the min and max values),
the sum of the deviations, the mean deviation,
the sample variance, and the SD. The shape of
a distribution is determined by skewness and
kurtosis. A distribution can be symmetrical
(i.e., the normal bell-shaped curve), skewed to
the left, or skewed to the right in which case the
distribution will have a tail that is longer on one
side than the other. A distribution with a long
tail skewed to the right will have a mean that is
greater than the median whereas a left-skewed
distribution will have the mean be less than the
median. Kurtosis, on the other hand, refers to
how spread out or peaked the distribution is.
For additional details on measures of central
tendency, dispersion, and distribution shape
you can consult any basic stat book. Some of
the books I have on this topic come from my
undergraduate days and are just as relevant as a
statistics book published last year (Blalock, 1960;
Daniel & Terrell, 1989; Gonick & Smith, 1993;
Levine & Stephan, 2005). These principles are
fundamental and have not changed over the years.
FIGURE 9-2 provides examples of d­ istributions
with different characteristics of center, dispersion,
and shape. Note that the normal (or Gaussian)
distribution, which is popularly referred to as the
bell curve, is typically not found in the real world
of data collection and analysis. In a t­ heoretical
normal distribution, the mean, median, and
mode are all at the same position and the data
are distributed randomly and symmetrically
about the mean. But it needs to be pointed out
that not all symmetrical bell-shaped curves are
normal (Blalock, 1960, p. 80). You can have,
for example, three normal curves that have the
same SDs but different means. Similarly, you
could have several curves that have the identical
means but very different SDs that in turn create
different shapes for the distributions.
It is important that you have a comfort level
with the characteristics of distributions so that
you can more fully understand the character-
istics of the data you have collected and their
potential limitations. Although the Shewhart
charts can accommodate both normally and
nonnormally distributed data (Wheeler, 1995)
having knowledge of the data you have gathered
is the first step toward creating and interpreting
Shewhart charts.
With a few of the basics about distributions
in hand it is now time to address in a little more

FIGURE 9-2 Examples of distributions with different centers, spreads and shapes
detail the topic of sigma limits. As was mentioned
earlier in this chapter, the UCL and LCL are prop-
erly referred to as sigma limits or alternatively,
estimates of the SD (Provost & Murray, 2011,
p. 115; Wheeler, 1993, 1995). Although some
writers will call the UCL and LCL SD limits
and not even reference the term sigma, I think
it is important to use the term sigma to refer to
the estimates of variation in a Shewhart chart
rather than SDs for several reasons:
■■ This is how Dr. Shewhart (1980, 1986)
originally described the limits on the charts
he developed.
■■ The SD of a distribution is calculated dif-
ferently than a sigma. The SD is a single
number that represents the average distance
any individual data point in a distribution
is from the mean. It cannot be a negative
number and it will go, theoretically, from
zero to a rather large positive value (but
generally speaking the SD usually does
not go much beyond double digits). A
sigma unit, on the other hand, is a “mea-
sure of scale for the data” (Wheeler and
Key Questions about Shewhart Charts 217
Chambers, 1992, p. 60). Wheeler also
points out that the common dispersion
statistic for a distribution (i.e., the SD)
needs to be converted into sigma units by
the use of specific formulas. He concludes:
“By shifting from measurement units (i.e.,
SD of a distribution) to sigma units, it is
possible to characterize how much of the
data will be within a given distance on
either side of the average. Thus, the sigma
units express the number of measurement
units which correspond to one standard
unit of dispersion” (p. 61)
A final point to acknowledge is that if you
­calculate the SD of a distribution using the
traditional formula that you will find in
many software packages, multiply this
number by 3, and then add and subtract this
value (i.e., 3 SDs) to the mean, you will get
the incorrect UCL and LCL for a Shewhart
chart. This becomes even more important
when you realize that each type of Shewhart
chart has its own formula to compute sigma
values and that none of these formulae use
218 Chapter 9 Understanding Variation with Shewhart Charts
the traditional SD formula for a sample based
on the following formula:
∑ (x − x )
n 2
i
Sx = i =1
n −1
n = The number of data points
–x = The mean of the xi
xi = Each of the values of the Data
Note that if you were calculating the SD for
a defined population the formula would
use N as the denominator (i.e., the total
number of observations in the population
being observed) rather than n - 1, which is
typically used when calculating a SD for a
sample.
If you do not use the appropriate formula
for computing a particular chart’s limits, you
will produce limits that are too wide or too
narrow. This will then lead you to make the
wrong decision about the variation in your
data (i.e., you will see special causes when
they do not exist and miss them when they
are actually present).
■■ The third reason I prefer using the terms
sigma units and sigma limits with Shewhart
charts rather than SD units or SD limits is
to avoid confusion. A majority of health-
care professionals have been exposed to
the concept of a SD but not to a sigma. We
have all been in meetings where someone
presenting data has proudly said, “We have
analyzed the data from last month for pro-
cedure X and discovered that the average
length of stay is 4.3 days and the SD is 2.6
days.” When this occurs most participants
in the meeting either nod their heads and
say nothing or mumble a few words to the
person sitting next to them about this is just
what we heard last month. The SD is a very
popular statistic presented in healthcare
management meetings. People hear the
number but most could not explain what
it is, how it was calculated, or how to inter-
pret it. But, it is a regular part of healthcare
management meetings. So, when the word
standard deviation is used to describe the
units of variation on a Shewhart chart or the
UCL and LCL in healthcare settings con-
fusion occurs conceptually and statistically.
I would strongly encourage you, therefore,
to use the proper terms when constructing
and explain Shewhart charts. 5
Now that we know a little more about what
a sigma is, the next question is, “Why do we need
three of them?” The answer to this question is
found partially in statistical theory and partially
in practicality. According to Wheeler (1995, p. 14)
Shewhart’s use of 3 sigma limits (i.e., three above
the mean and three below the mean for a total of
6 sigma units) as opposed to any other multiple
of sigma did not stem from any specific math-
ematical computation. Rather Shewhart said
that three “seems to be an acceptable economic
value,” and that the choice of 3.0 was justified by
“empirical evidence that it works” Provost and
Murray (2011) provide a succinct summary of
the rationale for using Shewhart’s 3 sigma limits:
■■ The limits have a basis in statistical theory.
■■ The limits have proven in practice to distin-
guish between special and common causes
of variation.
■■ In most cases, use of the limits will ap-
proximately minimize the total cost due to
overreaction and underreaction to variation
in the process.
■■ The limits protect the morale of the workers in
the process by defining the magnitude of the
variations that has been built into the process.
Provost and Murray’s point about overreaction
and underreaction to variation deserves a few more
comments. There are basically two mistakes or
errors that you need to avoid when interpreting
data. The first mistake (a type I error) is a risk
of concluding that a data point requires special
action when it is actually reflecting common cause
(random) variation. This leads to tampering with
a process that is in fact stable and predictable.
Tampering (i.e., reacting one way a data point
then reacting another way to the next data point
when they are part of a process that is stable and
predictable) leads to increases variation in a
 Deciding Whether a Special Cause Is Present 219

High

The combine total risk of a Type I and

a Type II Error is minimized when 3
sigma limits (SLs) are used.
Risk

Low

+/- 1SL +/- 2SL +/- 3SL +/- 4SL +/- 5SL +/- 6SL

FIGURE 9-3 Balancing the risk of a Type I and Type II error

nonlinear manner making things worse. (See the and Leavenworth (1988), Montgomery (1991),
reference to Rule 4 of the funnel demonstration Shewhart (1931), Wheeler (1995), and Wheeler
in Chapter 7 for more detail on tampering.) Type and Chambers (1992).
I errors happen most often when you decide to
use sigma limits on a Shewhart chart that are less
than three. On the other hand, the second mistake Do I Apply the Run Chart Rules
(a type II error) occurs when you basically do to Shewhart Charts?
the opposite of a type I error. In this case, you
The simple answer to this question is no. The
would conclude that a data point indicates no
four run chart rules (a shift in the data, a trend
need for action when it fact it reflects a special
in the data, too many or too few runs in the data,
cause. Type II errors lead to under controlling
an astronomical data point) should be applied
or what Provost and Murray call underreacting.
only to the run charts. Shewhart charts have
This happens most often when you decide to use
their own rules to determine whether special
sigma limits that are wider than plus and minus 3
causes are present. These rules ae explored next.
around the mean. As Carey and Lloyd (2001,
p. 67–68) point out, “The challenge, therefore, is
to balance the risk of tampering against the risk
of under controlling. In the first case, you will ▸▸ Deciding Whether
see special causes when they do not exist, and
in the second case, you will miss special causes
a Special Cause Is
when they are present. The combined total risk
of type I and type II errors is minimized when
Present
3 sigma limits are used.” FIGURE 9-3 provides a Much of the beauty of the Shewhart charts lies
visual of how the total combined risk of two types in their simplicity. They require just enough data
of error is minimized when the limits are set at (about 20 data points) to construct a reliable chart,
+/− 3 sigma. Those of you interested in exploring are easy to read, and allow you to determine
these issues further should refer to the works very quickly whether special cause variation is
of Blumenthal (1993), Deming (1994), Grant present in your data. Shewhart charts, according
220 Chapter 9 Understanding Variation with Shewhart Charts
to Pyzdek (1990, p. 90), “are an operational defi-
nition of a special cause,” which I think is a very
appropriate way to summarize the purpose of
the charts. Shewhart (1931, p. 6) also captured
the purpose of the charts nicely when he wrote,
“A phenomenon will be said to be controlled
when, through the use of past experience, we can
predict, at least within limits, how the phenome-
non may be expected to vary in the future.” This
acknowledges the fact that no one can predict
the exact value of the next data point. But, if you
understand the differences between a process
being in control (i.e., merely random variation)
and out of control (i.e., detecting special causes
in the data) then you will be well on your way to
understanding Shewhart’s notion of prediction
within limits. He basically argued that in order
to understand the variation in a process you
needed to move away from static and aggregated
displays of data and look at the process from a
more dynamic view by plotting the data over
time and understanding the inherent variability
in the process. Figure 7-3 (Chapter 7) depicts
what Shewhart was recommending.
For decades the Western Electric Statistical
Quality Control Handbook (1985) has served as
the standard reference for the special cause rules.
In fact, in many circles and even in several SPC
software packages the rules are frequently referred
to as the “Western Electric tests for detecting
special cause.” Although there are dozens of tests
or rules to detect special causes, most experts in
the field of SPC maintain that only a few of the
tests are essential for a basic understanding of
what the charts are trying to tell you. Wheeler
(1995) and Wheeler and Chambers (1992) provide
excellent summaries and critique of the Western
Electric rules and the variations that have been
proposed by leading SPC experts. Consider the
following passage from Wheeler (1995, p. 139)
on this issue:
■■ Shewhart used Detection Rule One.
■■ David Chambers remarked that “No data
set could stand up to the scrutiny of all of
the detection rules in the Western Electric
Handbook.”
■■ Irving Burr recommended using no more
than Detection Rules One and Four.
■■ Ellis Ott recommended the use of Detection
Rules One, Two, and Four.
■■ Lloyd Nelson recommends the routine use
of Detection Rules One and Four, along
with Test 3 (trends) and Test 4 (sawtooth).
The selection of the most appropriate rules,
however, should be linked to the subject matter
being analyzed, the types of data being collected,
and the ability of those who own the processes
that produce the outcomes to actually move
the relevant indicators in the desired direction.
The application of the rules for special
causes to a Shewhart chart begins by dividing
the chart into zones. The area between the cen-
terline (CL) (the CL or otherwise known as the
mean or average) and the UCL is divided into
three equal areas or zones. Because the control
limits are referred to as sigma limits, each zone
is the equivalent of 1 sigma. The area from the
CL to the LCL is divided in a similar manner.
These zones are labeled C, B, and A, respectively
and emanate outward from the CL. FIGURE 9-4
provides an example of how a Shewhart chart
is divided into six zones. The creation of zones
is a very simple process that can be achieved
easily with any reputable SPC software program.
A natural or random pattern of data will
bounce around across the zones, between the
UCL and LCL, and include the following three
characteristics:
Note: Each zone is equal to 1 sigma
UCL
Zone A +3 SL
Zone B +2 SL
Measure
Zone C +1 SL
X (CL)
Zone C -1 SL
Zone B -2 SL
Zone A -3 SL
LCL
Time
FIGURE 9-4 Dividing the Shewhart chart into zones
 Deciding Whether a Special Cause Is Present
■■ Most of the data points are near the CL
■■ A few of the data points spread out and
approach the UCL and LCL
■■ None of the data points (or at least only a
very rare and occasional point) exceeds the
control limits (Western Electric, 1985, p. 24)
A natural pattern or random distribution of data
will exhibit these three characteristics simulta-
neously. One of the first signals that a process
has special causes, therefore, is the absence of
any one of these characteristics.
Because these rules for detecting special
causes have grown primarily out of industrial
and manufacturing applications, however, we
need to evaluate them in light of which rules
are most appropriate in health, education, and
social services settings. We have done this at
the Institute for Healthcare Improvement (IHI)
with our colleagues from Associates in Process
Improvement (API)6 and decided that five of the
rules for detecting special causes on a Shewhart
chart are most appropriate for these disciplines.
The five IHI/API rules for detecting special causes
on a Shewhart chart are:
Rule 1: 1 point outside the +/− 3 sigma limits
Rule 2: 8 successive consecutive points above
(or below) the CL
Rule 3: 6 or more consecutive points steadily
increasing or decreasing
Rule 4: 2 out of 3 successive points in Zone
A or beyond
Rule 5: 15 consecutive points in Zone C on
either side of the CL
Each of these rules is discussed next.
Rule 1: 1 Point Outside the +/−
3 Sigma Limits (FIGURE 9-5)
This is usually referred to as a 3-sigma violation and
is classified as a signal of instability in a process.
It is also one of the most easily recognized of all
the tests because it is based on a single data point.
On the run chart this was a visual determination
of the “astronomical data point.” Now we have a
221
A 3 sigma violation
UCL
A
B
C
CL
C
B
A
LCL
A 3 sigma violation
FIGURE 9-5 Rule #1: A single data point that
exceeds the upper or lower control limit
statistical way to determine whether, in fact, it is
astronomical. This is the only test that Shewhart
used to identify special causes and the reason
why Wheeler (1995) stated that “Shewhart used
Detection Rule One.” Some texts refer to a single
data point that exceeds 3-sigma as a “freak” point
(Pyzdek, 1990). Irrespective of the term being
used, a 3-sigma violation is a clear signal that
the variation of this single point is very different
from the variation demonstrated by the rest of
the data points on the chart.
When you detect a 3-sigma special cause do
not overreact. The first thing you should do is
check the data to make sure that the data point
is legitimate. For example, if someone used a
different operational definition for this data point
it may in fact be a false positive. This data point
might also be due to a data collection procedure
that sampled the population differently than
the other data points. Finally, it could be due to
a stratification problem. In this case, data may
have been pulled from the afternoon shift when
the rest of the chart was based on data sampled
from the day shift. My point is that before you
see a 3-sigma violation as a true special cause,
investigate the methods used to gather that data
point. If the data point was based on the same
operational definition as the rest of the data and
there were no sampling or stratification issues
then you do in fact have a special cause that
222 Chapter 9 Understanding Variation with Shewhart Charts
requires investigation. Why is this data point
statistically different from the rest of the data?
The presence of a true special cause provides the
opportunity for learning.
Rule 2: 8 Successive Consecutive
Points Above (or Below) the
Centerline (FIGURE 9-6)
People generally find it easy to detect a 3-sigma
violation (Rule 1). But, as the Western Electric
Statistical Quality Control Handbook points out
(1985, p. 26) the data can reflect instability even
when all the data points fall between the UCL and
LCL. A shift in the process is one such indication of
instability. Most writers refer to this rule as “eight
consecutive data points on the same side of the
centerline.” When such a pattern is observed, it
signals that there has been a shift in the process.
Another way to think of this rule is that it reflects
a run of data that has lingered too long above or
below the mean, which indicates a nonrandom
pattern. This test is a variation on the run chart
shift rule but you will notice that it requires eight
data points in a run whereas the run chart rule
required six data points to determine a shift.
This test is one of the original Western Electric
four primary tests and it is a frequent signal on
healthcare charts. As teams work continuously
on improvement strategies, their work typically
produces results that are not immediate and dra-
matic in nature (i.e., the 3-sigma rule) but rather
UCL
A
B
C
CL
C
B
A Decreasing (FIGURE 9-7)
LCL
Too many data points in a row below the
centerline signals a downward shift in the process.
FIGURE 9-6 Rule #2: 8 successive consecutive points
above (or below) the centerline
are observed as a gradual movement of the data
over time, which is demonstrated as a shift in the
process. Ideally this shift would be in the desired
direction but the shift could also be in the oppo-
site direction. The data are neutral. They do not
know if they are in the direction of goodness or
away from it. This is why it is important that you
apply the statistical decision rules that allow you
to know when there is a true signal in the data of a
special cause and when it is just random variation.
Although the rule of 8 is a classic Western
Electric rule you will see other alternatives offered
(e.g., 7 in a row, 9 in a row or even one approach
that favors a spread of 8 to 10 in a row). Wheeler
(1995) lists all the various options defining a
shift that have appeared over the years and offers
commentary on which ones he has seen used most
often. My point in even mentioning these alterna-
tives is that you will hear a variety of opinions on
the number of data points used to define a shift
and also on what constitutes a trend (rule 3). The
challenge is if you define a shift with say seven data
points you may see special causes when they do
not exist (i.e., a type I error). If, on the other hand,
you choose to use 9 or 10 data points as a shift you
may fall prone to a type II error, which is missing
a special cause when it is present. The rule of 8 has
been regarded as a solid practical rule and it is the
one I and my colleagues at IHI have decided to
use. It is neither too lenient nor too conservative
for health and social service application. Unless
you like to get into rather heavy statistical theory
debates about which rule is “the best” I’d suggest
that you accept a set of rules that are practical and
appropriate for your work.
Rule 3: Six or More Consecutive
Points Steadily Increasing or
This rule detects a trend in the data that Provost and
Murray (2011, p. 117) define as “a small, consistent
drift in a process.” When deciding if a trend exists,
duplicate points (i.e., repeating values) should be
ignored. This rule engenders considerable debate.
First, there is the popular definition of a trend. We
 Deciding Whether a Special Cause Is Present
A
C
C
A
Downward trend
FIGURE 9-7 Rule #3: 6 or more consecutive points steadily increasing or decreasing
see a trend in fashion, a trend in food, a trend in
the stock market, which is usually referring to
the fact that the stock market closed higher than
it started the day. I regularly hear the weather
reporter on the Chicago TV stations referring to
a “trend in the temperature.” In this case, the trend
is usually a comparison of today’s temperature
to the average temperature for the past week or
month or the comparison of today’s temperature
to the temperature on the same day a year ago.
The point is that there are very popular usages
of the word trend and then there are statistical
definitions. As we analyze Shewhart charts we
definitely should be using a statistical definition.
But, I have been in many meetings where people
interpreting either static or dynamic displays of
data have devised their own definitions of a trend.
Over the next week make a mental note of how
often you hear your coworkers or people in the
media look at data and declare a trend is present.
People will conclude that there is a “trend” in the
data when in fact they are merely comparing two
data points. If the second data point is higher than
the first and in the direction of goodness then this
gets labeled as an upward trend. Deming (1992)
had a very good bit of guidance: when you have
two data points, “it is very likely that one will be
different from the other.”
The Western Electric handbook does not
specify how many data points are needed in order
to identify a trend. They merely indicate that a trend
223
UCL
CL
LCL
Upward trend
is detected when there is “a series of consecutive
data points without a change in direction.” At the
IHI we have decided to use the rule of 6 as initially
defined by Nelson (1985) and then by Pyzdek
(1990) as a common practical basis for detecting
an upward or downward trend in the data. The
final point I will make about this particular rule
is that like Rule 2 this rule engenders considerable
debate. Wheeler (1995, p. 137), for example, states
that “all of these tests (for a trend) are problem-
atic.” He offers a number of reasonable statistical
principles as to why he maintains this perspective.
Others will argue with you about a trend because
they are (1) wanting to see a trend, (2) are using
a popular definition of a trend, or (3) have some
other statistical reference that says their trend is
preferred over the one you propose. So, once again,
unless you are ready for these debates I would
suggest that you accept the rule of 6 as a trend and
see how well it fits with your analysis of the data.
Rule 4: Two out of Three
Successive Points in Zone A
or Beyond (FIGURE 9-8)
Another of the classic Western Electric rules for
instability is when two out of three consecutive
data points are more than 2 sigmas away from the
CL. In this particular case, the single data point
not in Zone A or beyond can be anywhere on the
224 Chapter 9 Understanding Variation with Shewhart Charts
Rule #2: Two out of three
consecutive data points that fall
in Zone A or beyond
A
C
C
A
consecutive data points that fall
FIGURE 9-8 Rule #4: 2 out of 3 successive points in Zone A or beyond
chart. The deciding criterion is whether two out of
the three successive data points are in Zone A or
beyond on the same side of the CL. This is one of
the rules that is more difficult to explain in words
than pictures. Observing Figure 9-8 will help in
understanding this rule. The primary question I
get with this rule, however, is “so what?” “Why
is it that two out of three data points in Zone A
constitutes a special cause?” First, envision the
static normal curve. Slightly over two thirds of
the data (68.26%) will fall within ± 1 SD of the
mean. When you go out to ± 2 SDs of the mean
you will find 95.46% of the data. This means
that by the time you are out to ±3 SDs from the
mean you should be observing 99.73% of all the
data in the distribution. But, because the normal
curve theoretically extends infinitely in either
direction you do not account for 100% of the
data. Now let’s get back to the two out of three
data points in Zone A of a Shewhart chart. As
you go out the tails of the normal distribution
you should expect to see less data. The two out
of three rule, therefore, is signaling that you
are observing too much data bunching in the
UCL
CL
LCL
Rule #2: Two out of three
in Zone A or beyond
tail(s) of the distribution when you should in
fact be observing less and less the further you
go out. There certainly are more complex sta-
tistical explanations of why this rule detects a
special cause. But as Wheeler points out (1995,
p. 135) “this rule provides a reasonable increase
in sensitivity without an undue increase in the
false alarm rate.”
Rule 5: 15 Consecutive Points
in Zone C on Either Side of the
Centerline (FIGURE 9-9)
This test is generally described as reflecting an
issue with stratification. Stratification usually
indicates that two or more different causal sys-
tems are present in every subgroup. This pattern
of stratification is also known as “hugging the
centerline” because there is a run of 15 or more
data points within 1 sigma of the CL (i.e., in Zone
C above or below the CL) and the variation is
relatively small for these data points compared
to the width of the UCL and LCL. Stratification
 Deciding Which Shewhart Chart Is Most Appropriate
A
C
C
A
FIGURE 9-9 Rule #5: 15 consecutive points in Zone C on either side of the centerline
occurs most often because the data collection
plan was flawed. For example, you will find a
stratification pattern when two separate distri-
butions of data have been collected (e.g., day shift
turnaround time was combined with night shift
turnaround time) or the sample of data points was
drawn from two different distributions of data.
Even though we have these rather specific
statistical rules for determining special cause
variation that are grounded in decades of testing in
manufacturing settings, I think it is important that
when we apply these rules to healthcare situations
we apply them with a serious dose of common
sense. For example, if we are trying to improve
food tray delivery time we may be willing to fully
accept six data points constantly increasing as a
trend. But, on the other hand, if we are dealing
with wrong site surgeries we may not want to
wait until we have six occurrences of wrong site
surgery to declare a trend and then take action. As
my colleague Dr. Ray Carey (2003, p. 19) wrote:
“When the well-being of patients is at risk, a case
can be made for using 2-sigma limits as ‘early
warning limits’ or for using 6 rather than 8 points
to detect a shift.” In these situations, clinicians
would still be looking for signals of special cause
so they do not over react to a single data point.
But, they would use the data not necessarily to
justify changing the system but rather as a basis
to investigate potential instability in their process
that could cause harm to patients. Wheeler and
Chambers (1992) refers to this as having a process
on the “brink of chaos.” Statistical decisions must
be moderated with and filtered through a healthy
dose of common sense and rational thinking.
225
UCL
CL
LCL
▸▸ Deciding Which
Shewhart Chart
Is Most Appropriate
Although there is only one way to make a run
chart, there are numerous ways to make a She-
whart chart. The basic design and look of any
Shewhart chart is essentially the same as shown
in Figure 9-1 (i.e., data plotted over time, the
mean of the indicator as the CL and the calcu-
lation of the UCL and LCL). Furthermore, the
charts are all grounded in established statistical
theory and are all interpreted in terms of the
fundamental ideas related to common and
special causes of variation. But, there are many
different types of Shewhart charts and the user
needs to know which one is most appropriate
for the indicator being studied. The variety of
Shewhart charts is summarized by Benneyan
et al. (2003, p. 16):
There are at least a dozen different
types of control charts in common use
in manufacturing and other industry,
with three or four new types being
developed each year. The various types
differ by the statistic plotted (e.g., av-
erages, percentages, counts, moving
averages, cumulative sums, etc.) and
the distribution assumed (e.g., normal,
binomial, Poisson, geometric, etc.). All
have different formulae for calculating
centerlines and control limits.
226 Chapter 9 Understanding Variation with Shewhart Charts
If there are all these different types of Shewhart
charts how do you decide which one is the most
appropriate for your data? The decision involves
two rather simple steps: (1) deciding on the
type of data you have collected and (2) deciding
which of the various Shewhart charts is the most
appropriate for this type of data.
▸▸ Types of Data
The first step in selecting a Shewhart chart is to
determine the type of data you are collecting.
There are basically two types of data: (1) v­ ariables
data (also known as continuous, interval, ratio,
or measurement data depending on your back-
ground and training) and (2) attributes data
(also known as classification or count data). The
term used to identify the type of data is a matter
of taste and preference. Most SPC books will
use the terms variables or continuous data and
attributes, classification, or count data. What is
more important than the terms you choose to
use are the concept the terms are capturing and
how you apply them to your data. In this text,
I use the terms variables and attributes data as
the primary categories. FIGURE 9-10 provides
examples of these two types of data.
Variables data can be measured along a
continuous scale. In Figure 9-10 this type of data is
depicted as money, time, weight, length, and tem-
perature. Consider the ruler as a form of variables
data. It has equal appearing intervals that can be
divided into as many subdivisions as your calibration
instruments will permit. With variable data you
can perform all the mathematical function. Data
measured this way can be either counts of whole
numbers or they can have decimals or fractional
parts. Examples of variables data include:
■■ Wait times in the emergency department (ED)
■■ Turnaround time for a lab test
■■ Blood pressure readings
■■ Newborn weight (measured in grams or
pounds and ounces)
■■ international normalized ratio (INR) and
prothrombin times (PTs)
■■ Blood glucose readings
■■ The number of procedures or tests performed
■■ The number of surgeries done each day
■■ Financial measures such as revenue, oper-
ating margin, or expenses
■■ Duration of a surgical procedure in minutes
or hours
Attribute data are essentially counts of events
that can be placed into discrete categories. Unlike
measuring a patient’s weight on a continuous scale,
attribute data are looking at characteristics that
can be classified and placed into categories or
“buckets.” For example, any time you are measuring
mortality you are using attribute date (the patient
is either alive or dead). Similarly, pregnancy is an
attribute indicator. There are only two outcomes:
the woman is either pregnant or not pregnant.
A woman does not proudly announce that she
is 53.9% pregnant. This is essentially a binomial
outcome. Attribute data can be further divided
into two subdivisions, defectives and defects.
Defectives (also known as nonconforming
units) require that you have a count of the total
number of items or events being observed or
produced and the number of items from this total
that were not acceptable. The unacceptable items or
events become the numerator and the denominator
is the total number of items or events observed.
When you know how many items out of the total
are unacceptable you can either plot the number
of defective items on your Shewhart chart or you
can compute the percent of defectives. When we
compute a percentage, therefore, we are basically
determining what proportion or percentage the
numerator is of the denominator. The standard
terminology used in most SPC books to define
defectives is that you know both the occurrence
of the defective product or service (the numer-
ators) and the nonoccurrences (defectives plus
nondefectives which when added together form
the denominator). Knowing these two pieces
allows you to calculate a percentage or proportion
of defectives. Keep in mind that when you use
percentages you are comparing the same types of
items, products, or services. If you are looking at
the percentage of food trays delivered late to the
 Types of Data 227

Variables Data

© Pedjami/Shutterstock

© Ultrashock/Shutterstock

© Butterfly Hunter/Shutterstock

© Paul Velgos/Shutterstock © Lipskiy/Shutterstock

Attributes Data

Defectives Defects
(occurrences only)
(occurrences plus
Nonconformities
non-occurrences)
Nonconforming Units

© HeinzTeh/Shutterstock

FIGURE 9-10 Examples of variables and attributes data

228 Chapter 9 Understanding Variation with Shewhart Charts
patient, for example, you will have the number of
late food trays as the numerator and total number
of food trays produced as the denominator. In
this case, you have trays divided by trays—like
divided by like. The only attribute that is different
for classification purposes is whether the food
tray was delivered late to the patient. This is an
important distinction to keep in mind because
as we move next to define defects, this condition
will not hold. In summary, data classified as de-
fective can be divided into one of two categories
(i.e., a binomial situation) when you know both
the occurrences and the non-occurrences of an
event. Examples of this form of classification in-
clude conforming/not conforming to standards,
harm/no harm, go/no-go, pass/fail, OK/not OK,
complete/­incomplete or present/absent.
Defects pose an interesting challenge. Defects
occur and can be counted. But, how do you count
all the nondefects? Stated differently, you know
when a defect occurs (the occurrence of an event)
but you do not know when the nondefects or
nonoccurrences happen. I know, at this point you
are thinking, “This makes no sense.” When I first
heard this statement it did not make a lot of sense
to me either. Examples should help to clarify this
concept. Look down at the rug in your office or
in your family room. How many spots, stains, or
snags do you find in the carpet? For argument’s
sake I’ll imagine that you found three dirt spots,
two coffee stains, and four snags on the carpet.
Now, count the number of nonstains on the carpet?
How did you do? You cannot count the nonstains
or blemishes on the carpet. This is an unknown.
Similarly, when the highway department records
traffic accidents they do the same thing. They can
count how many accidents occurred today on a
particular segment of the highway but they have
no idea how many nonaccidents there were today.
There are times, therefore, when we know the
occurrence of an event when the nonoccurrences
are unknown and unknowable.
In health care, we experience this situation with
patient falls, needle sticks, nosocomial infections,
medication errors, and liability cases. We know
only when the event happens. Think of needle
sticks. A staff member comes into your office to
report that she just stuck herself. After you try to
calm her down and explain the next steps you will
take, you do not say, “Oh by the way, how many
times didn’t you stick yourself today?” Similarly, if
a nurse asked a patient, “How many times didn’t
you fall today?” she would probably get a rather
confused look from the patient.
When you are dealing with defects you need
to remember that a count of the number of falls,
needle sticks, or medication errors gives you a
numerator but you do not have a denominator. So
you cannot compute a percentage. So, you either
just count the number of defects as whole numbers
(e.g., the total number of falls today was eight)
or you create a falls rate. A rate is a ratio (i.e., it
has a numerator and a denominator just like a
proportion or percentage) but the two numbers
you are using to form this ratio are not alike. For
example, when we compute an inpatient falls rate
by month we have the number of inpatient falls
(including multiple falls) for the month as the
numerator and the denominator is usually the
total number of inpatient days for the month. Now
we have falls divided by days, two unlike things.
The resulting number is reported as so many
falls per 1,000 patient days. Any time you report
that there are so many defects per 1,000, 10,000
or 100,000 units (e.g., inpatient days, medication
orders, lab tests, or surgeries) you have just created
a rate. Note that when you see the little word per
included with the name of a measure you know
that it is a rate and not a percentage. Most of the
patient safety indicators as well as epidemiology
indicators are constructed as rates (e.g., patient
fall rate, restraint rate, surgical site infection
rate, ventilator-associated pneumonia (VAP)
rate, needle stick rate, or medication error rate).
The other characteristic of a rate is that the
numerator of a rate can be larger than the denom-
inator. For example, if you had a 20-bed unit and
each patient fell two times you would have 40 falls
for 20 patients. What do you call this? Is it 200%
falls? No. If you wanted to use a percentage you
would have to make a different indicator, which
would be the percent of patients who fell once or
more while they were with us. In this case, we do
not care about the total number of falls, which

If Yes, then the car is
fit to be shipped out!
If No, then the car is classified as being “defective” but
we do not know why it is defective (not fit to be shipped)
FIGURE 9-11 Defectives versus Defects
Ed Aldridge/Shutterstock
includes duplicates. All we are concerned with is
at the time of discharge did this patient fall once
or more, yes or no? Basically if the indicator is the
percentage of patients who fell we do not care if
they fell more than once. But, because patients
can fall more than once, and we are concerned
about this problem, we generally do not use the
percentage of patients who fell as a binomial in-
dicator (i.e., fell/did not fall). If we are concerned
about the magnitude of the falls and severity we
typically track all falls, which means that we have
the possibility of having a numerator that is larger
than the denominator. When this can occur, we
normalize the total number of falls by creating a
rate (e.g., 3.2 falls per 1,000 inpatient days). A final
point about defects is that they usually occur less
often than indicators measured by a percentage.
FIGURE 9-11 provides an easy way to remember
the differences between defectives and defects. When
cars come off the assembly line they get inspected.
If the car is determined to be acceptable by the
inspectors it is fit to be shipped to a dealer. But, if
the inspectors find one or more things wrong with
the car it is not fit to be shipped. In this case, the car
would be classified as being defective. This fit to ship
determination is a binomial decision: the car is okay
or the car is not okay to be shipped. At the end of
the shift the inspectors take all the defective cars and
provide a summary of why each car was classified
as defective (i.e., as a nonconforming unit). This is
where the defects come into the picture. The first
defective car has a summary report pasted to the
windshield. It reads that this defective car has three
Types of Shewhart Charts 229
OK?
Not OK?
until we inspect it and count the number of specific
“defects” that make the car “not OK” or defective.
defects: one headlight does not come on (defect
1), the driver’s side door does not close flush with
the body (defect 2), and the driver’s seat moves
backwards but not forward (defect 3). The entire
car is classified as defective but three defects have
been discovered. The next car is also classified as
defective but it has only one defect (the oil pressure
warning light on the dashboard does not go out
after the specified period of time). In summary,
defects or nonconformities are the specific things
that make a product or service defective. Once you
understand the distinctions between defectives and
defects you will be well on your way to selecting
the most appropriate Shewhart chart. To help you
in building your skills in differentiating between
defectives and defects refer to EXERCISE 9-1. For each
indicator listed decide if it is describing a defective
or defect. The answers to this exercise can be found
at the end of this chapter.
▸▸ Types of Shewhart
Charts
After determining whether your data are variables
or attributes, the next step is to decide which She-
whart chart is most appropriate for the type of data
you have collected. Seven basic control charts are
regularly described in the literature and taught in
most classes and seminars on SPC. After working
with the charts for over 15 years, however, I have
found that five of the seven charts are the most
230 Chapter 9 Understanding Variation with Shewhart Charts

EXERCISE 9-1 Defective or defect? You make the call!

Defective Defect
Indicator (Classification) (Count)

1. Number of accidents per 1,000 employee days

2. Number of errors per 25 food trays

3. Percentage of acute myocardial infarction (AMI)

patients receiving aspirin within 24 hours of arrival
in ED

4. Percentage of inpatient deaths each month

5. Number of surgical complications per 1,000

surgeries performed

6. Proportion of hand hygiene observations done

incorrectly

7. Number of falls per 1,000 patient days

8. Number of medication errors per 10,000 doses

dispensed

relevant and frequently used with healthcare,
social services, and educational indicators. I focus
on these five Shewhart charts but encourage you
to explore the full range of charts as discussed in
the Western Electric Statistical Quality Control
Handbook (1985), Wheeler (1993, 1995), Wheeler
and Chambers (1992), Carey and Lloyd (2001),
Carey (2003), Duncan (1986), Pyzdek (1990),
Kume (1996), and Provost and Murray (2011).7
FIGURE 9-12 presents the Shewhart chart
decision tree with the five control charts that
have the most relevance to health care, social
service and educational indicators. Two of the
five charts are used with variables data (i.e., X-bar
and S chart and the XmR chart) and three of the
charts are used with attributes data (i.e., p-chart,
u-chart, and c-chart). Each of the five charts is
described next and examples of how to apply
the charts are offered. I would suggest, however,
that you read beyond what I summarize in this
chapter. As I mentioned earlier in this chapter,
by reading the explanations of different authors
describing Shewhart charts and their uses you
will build knowledge on how to use them with
your own improvement efforts.
▸▸ Defining the Key Terms
Before addressing the details related to each of
the reviewing the five basic Shewhart charts
shown in Figure 9-12, however, is it necessary
to review three key terms that play a critical
role in helping you work your way successfully
through the Shewhart chart decision tree shown
in Figure 9-12. These key terms are subgroup,
observation, and area of opportunity and are
summarized in FIGURE 9-13.
 Defining the Key Terms 231

Variables data Decide on the type of Attributes data

data

More than one

Occurrences and
observation per
Yes No nonoccurrences?
subgroup? No Yes

Is there an equal
Yes area of No
opportunity?

X-bar & S XmR c-chart u-chart p-chart

Average and Individual The number of The defect The proportion

Standard measurement defects rate or percentage of
deviation defectives

FIGURE 9-12 The Shewhart chart decision tree

Subgroup Observation Area of Opportunity

How you organize your data (e.g.,
by day, week, or month)
The label of your horizontal axis
Can be patients in chronological
order
Can be of equal or unequal sizes
Applies to all the charts
The actual value (data) you
collect
The label of your vertical
axis
May be single or multiple
points
Applies to all the charts
Applies to all attributes or
counts charts
Defines the area of frame in
which a defective or defect
can occur
Can be of equal or unequal
sizes

FIGURE 9-13 Defining Subgroup, Observation, and Area of Opportunity

Subgroup axis of the chart. The subgroups will be arranged in

The subgroup defines how you have organized your
data and usually captures some dimension of time
such as when patients show up for an appointment,
day of the week, week, or month. The subgroup
will be the label you place on the horizontal or x
chronological order of occurrence. When deciding
on a subgroup you should strive to select them so
that if special causes exist the chances for differences
between subgroups will be maximized, whereas the
chances for differences due to special causes within
232 Chapter 9 Understanding Variation with Shewhart Charts
a subgroup will be minimized (Duncan, 1986;
Montgomery, 1991). The traditional subgroups
for Shewhart charts have been:
■■ An individual patient as the subgroup in
which case you would order the patients
along the x axis of the chart in the order
that they presented themselves in the office.
Patient 1 arrived at 9:00 a.m., patient 2 at
9:25 a.m., patient 3 at 9:50 a.m. and so on.
■■ A day as your subgroup in which case you
would have Monday, Tuesday, Wednesday, etc.
across the x axis. Then each day you would
select either all of the patients or a sample of
them and record their wait times.
■■ A week the subgroup and you would label the
x axis as Week 1, Week 2, Week 3, etc. You
would then have to decide if you were going
to track the wait time for all the patients in a
week or just a sample. Usually when a week
or even a 2-week period is selected as the
subgroup and patient wait time is the indicator
of interest you would probably want to draw
a sample of the patients. A total enumeration
would probably provide more data than you
need and create data collection challenges.
■■ A month is a frequently used period of
time for a subgroup. But it is not always
the best block of time in which to think
about improvement or understanding
variation. Remember that the Shewhart
charts are designed to help you understand
the variation in a key process indicator as
close to the production of the indicator as
possible. In manufacturing, they evaluate
products and services on an hourly, shift,
or daily basis. Although they may aggregate
the key indicators for management reports
by month or quarter the ability to improve
quality and insure reliability does not come
by looking at monthly or quarterly averages.
It comes by looking at production almost
as it happens. In health care and many
social services, there is a strong tendency
to track indicators by month or even quar-
ter. This is what I refer to as the tyranny
of monthly data. Think of it this way: in a
month we have 30 or 31 days, upwards of
90 shifts in a hospital, and approximately
720 hours in which to deliver care. Why
would we want to aggregate all this activity
into a monthly average or monthly total?
Monthly data frequently lead people down
the path of judgment or accountability not
quality improvement (QI). In my view, a
primary reason we have so many health-
care indicators structured around monthly
subgroups is that this is how financial and
resource allocation systems are organized.
In health care, work is being produced
every minute of the day not in monthly
blocks. Patients are waiting to be seen,
have tests performed, or surgery started.
Their focus is on minutes or possibly hours
not months. Administrators and managers
think in terms of months but patients think
about the here and now not in monthly
aggregates.8 The other challenge with using
month as a subgroup is that the variation
in the indicator of interest is usually not
visible because the data is aggregated into
an average. Although administrators, man-
agers, and policy makers frequently rely on
aggregated data and summary statistics to
make decisions, no customer, patient, or
service user cares about the average. They
are concerned about why they or their loved
one are not getting service or treatment
now. A patient takes no comfort in being
told that the average wait time to see the
doctor last month was only 49 minutes. Or
a physician waiting for her stat lab result to
come back will rightfully be irritated if she
is told “We don’t have your result quite yet
but don’t worry, the average time to get a
result last month was only 63.5 minutes.”
My point is that although we have a ten-
dency to fall back on making month the
subgroup for many healthcare indicators
there is no reason to do so. I have made
many charts for teams that are collecting
monthly data. But each time this happens
I make sure we have a discussion about
what is the smallest unit of time that we

could gather data on. Using month as a
subgroup should be a fallback option not
the first choice.
■■ Finally, it is probably quite evident at this
point that it is not advisable to use quarters
as a subgroup choice because quarterly data
represent a very long time period and the
variation you are interested in understanding
has been aggregated and therefore severely
dampened. Quarterly data can lead only to
judgment not improvement.
Observation
As the term implies this is the actual piece of data
or the measurement that you record or observe
(e.g., turnaround time for a lab test or medication
order, blood glucose readings for a patient, or
time to administer beta blockers to heart attack
patients in the ED). The vertical axis label on the
chart defines the observation and the units of
measurement along the y axis show the potential
distribution of these values. An observation
can be classified as either variables or attribute
data (e.g., time, money, weight, a percentage of
defectives, a count of defects, or a defect rate).
For example, if your indicator is wait time to see
the doctor in a clinic your observation will be
the actual wait time in minutes that occurs from
when the patient checks in at the registration
desk until she is seen by the doctor. This amount
of time will be what gets plotted on the chart.
Therefore, the dot on the chart, or the “doink”
as I like to refer to it, represents the quantitative
aspect of the indicator you are observing during
the defined period of time (i.e., the subgroup).
Area of Opportunity
All Shewhart charts must have a subgroup and an
observation clearly defined or the chart cannot
be constructed. When we move to the right side
of the decision tree (Figure 9-12) and consider
the attributes charts, discussed in detail later, a
third term comes into play. Notice that the first
decision point when dealing with attributes
charts is determining whether you have both the
Defining the Key Terms 233
occurrence and nonoccurrences of the events
being studied. If you answer “yes” to this question
you will be able to calculate either the proportion
or percentage of defectives and proceed to make
a p-chart. If you respond that you do not have
the occurrences and nonoccurrences you are
left with having only the occurrence of an event
when the nonoccurrences are not know. As was
mentioned previously, this gives you a count of
the defects and you will make either a c-chart or
a u-chart. The decision to make a c- or u-chart is
based on your answer to the following question:
“Is there a relatively equal area of opportunity
for the defect to occur?” If you respond “yes” to
this question you will make a c-chart, which is
a plot of the number of defects occurring within
each subgroup (e.g., a count of the number of
falls occurring each day). If you respond “no” to
this question (i.e., there is not an equal area of
opportunity for a fall to occur) then you would
make a u-chart, which would be a plot of the
defect rate by subgroup (e.g., 3.2 falls per 1,000
patient days). So, it really does not matter if you
respond “yes” or “no” to the area of opportunity
question. Consider it essentially a filtering ques-
tion that will help you select the correct chart for
your indicator. As each chart type is explained
the use of these terms is demonstrated.
The terms used in the Shewhart decision tree
(Figure 9-12) and summarized in Figure 9-13 are
not only central to understanding SPC theory but
also from a practical perspective, understanding
the terms subgroup, observation, and area of
opportunity are essential in the operation of SPC
software packages. Many of the SPC software
packages I have used explicitly ask you to identify
the subgroup and the observation or some variant
of these terms. Although most software appli-
cation do not ask you the “area of opportunity”
question, understanding this concept is critical
to selecting the most appropriate chart for your
indicator. With these basic terms in mind, we
can start using the Shewhart chart decision tree
to understand the conditions that will lead us to
select each chart. We will start on the left side of
the decision tree and address the variables data
charts then move over to the attributes charts.
234 Chapter 9 Understanding Variation with Shewhart Charts

X-Bar and S Chart If the subgroups are of unequal size, however

The left side of the Shewhart decision tree
(Figure 9-12) follows a pathway to two charts.
The first one is referred to as the average (X-bar)
and SD (S) chart. It is the most powerful of the
five Shewhart charts because it has multiple
observations of continuous data that have been
organized into subgroups. In this case, the “doink”
on the chart (i.e., the plotted dot) has multiple
“doinkettes” (i.e., observations) going into it. For
an X-bar and S chart the subgroups can be of
equal size or unequal size. If the subgroups are
of equal size (e.g., a stratified random sample of
15 patients is selected each day and their wait
times to see the doctor are recorded) then the
UCL and LCL on the chart will be straight lines
as shown in FIGURE 9-14.
(e.g., on Monday, we sample 10 patient wait
times, on Tuesday, we had 15, and Wednesday,
we collected 20 wait times) the UCL and LCL
will not be straight lines. Instead, they will be
what are called “stair-step” control limits as
shown in FIGURE 9-15. With an unequal sub-
group size the amount of data varies within
each subgroup and so the dots on the chart
(i.e., the observations) each have their own
individual UCL and LCL calculated. With more
data the limits are tighter and with less data in
each subgroup the limits are wider as shown in
Figure 9-15. Day 4 in Figure 9-15, for example,
has tighter limits indicating that there is more
day being collected on this day. Day 9, on the
other hand, has wider limits due to less data
being collected on this day.

X-bar chart: patient wait time

60.0
UCL
50.0
Wait time in minutes

40.0

30.0 LCL
20.0

10.0

0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Week

S chart: patient wait time

18.0
16.0 UCL
14.0
Standard deviation

12.0
10.0
8.0
6.0
4.0
2.0
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Week

FIGURE 9-14 X-bar and S chart with straight control limits due to equal subgroup sizes
 Defining the Key Terms 235

140

137
UCL = 135.22
134
Average (mmHg)

131

128 CL = 127.333

124

121
LCL = 119.44
118

115

20
16
UCL = 12.071
Sigma

12
8
CL = 6.129
4
0 LCL = 0.186
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Day

FIGURE 9-15 X-bar and S chart with stair-step control limits due to unequal subgroup sizes

When you make an X-bar and S chart most
software programs will give you the option of
producing two charts as shown in Figures 9-14
and 9-15. The top chart is the X-bar chart or
­average chart and the bottom chart is the S chart
or SD chart. The X-bar chart is considered to
be the primary chart. Both charts have three
main components: (1) the CL or average, (2)
the UCL, and (3) the LCL. The X-bar chart
will show the average of the data within each
subgroup and the lower chart (the S chart)
shows you the SD for each subgroup (i.e., each
dot) plotted on the X-bar chart. In Figure 9-14,
for example, Week 1 on the X-bar chart has an
average wait time of 38 minutes and an SD of 5
minutes (seen on the bottom chart). On Week
2 the average wait time is 39.7 minutes and the
SD is 7.2 minutes. So for each week we can see
what the average wait time is and the amount
of spread around that average for this week as
measured by the SD.
In Figure 9-15 the indicator of interest is a
patient’s systolic blood pressure. The patient re-
corded several blood pressure readings each day
(a minimum of three and a maximum of five each
day). As a result the subgroups are of unequal sizes,
which produces the stair-step control limits. If the
patient had recorded exactly the same number
of blood pressure readings (observations) each
day (e.g., four) then the UCL and LCL would be
straight lines. As you will see in subsequent ex-
amples, stair-step control limits will also be found
on p- and u-charts. As was mentioned above, on
those days when more data were collected (e.g.,
Day 4 in Figure 9-15) the control limits will be
tighter. On days when fewer data were collected
(e.g., Day 9) the limits will be wider.
The upper chart in Figure 9-15 reveals
the average systolic blood pressures by day
and the overall average. The lower chart shows
the SD for each day as well as the average SD
across all 25 days.9 The average systolic blood
236 Chapter 9 Understanding Variation with Shewhart Charts
pressure shown in the top chart for the patient
in Figure 9-15 is 127. Note that the decimal
places on the chart can be ignored in this case
because this is entirely too finite a reading for
blood pressure results. The degree to which you
can control the decimal places on a chart will
depend on the software being used. The average
UCL is 135 whereas the average LCL is 119.
Because this chart reveals only common cause
variation, the way to describe the performance of
this patient’s systolic blood pressure this chart is
as follows: “On the average this patient’s systolic
blood pressure is 127. It could go up as high as
135 on any given day or down to 119 and that
is the natural rhythm of this patient’s systolic
blood pressure process.”
The lower chart is the S-chart. This chart
has two primary purposes. First, it helps you
to understand the variation that exists within
each subgroup (i.e., day). For example, the SD
for Day 4 is around 4. On Day 9 the standard
deviation is about 3 mmHg. As you look at
each day, therefore, you will see that there is a
different average and standard deviation which
reflects the variation in this patient's blood
pressure over time. The second purpose of the
S-chart (i.e., the bottom chart) is that the av-
erage SD (the CL) is used in the calculation of
the control limits for the average (upper) chart.
In this case, the average systolic blood pressure
(i.e., CL) is 127. What is important to realize is
that if the SD chart reveals wide variation, then
the average SD will likewise be large. Because
the average SD is used to compute the UCL and
LCL, a large SD will also contribute to making
the control limits of the top chart (the average
chart) wider. The relationship of the two charts
must be understood together.
I do not intend to elaborate on the statistical
formulae for the calculation of the control limits.
It is important to realize, however, that each
Shewhart chart has a different set of formulae to
calculate the chart’s UCL and LCL. For additional
details on calculating the statistical parameters
for the various Shewhart charts readers should
refer to Wheeler (1995), Wheeler and Chambers
(1992), Western Electric (1985), and Provost and
Murray (2011).
XmR chart or the I-chart
The XmR chart is also known properly as the
Individuals and Moving Range chart. But it
can also be referenced as the Individuals chart
or simply the I-chart. The key characteristic of
this chart is that each subgroup contains one
and only one individual observation or bit of data
(i.e., the “doink” on the chart has only 1 bit of
data and no “doinkettes” as we discovered in the
X-bar and S chart). In the Shewhart decision tree
(Figure 9-12), this decision point is identified
by the question “More than one observation
for each subgroup?” When the answer is “no”
then the chart of choice is the XmR chart. Like
the X-bar and S chart, you will typically get
two charts when you request this type of chart
from your SPC software. The X chart shows the
values for the individual data points as well as the
average for all the individual data points. The mR
chart documents the “moving range.” The XmR
chart is typically used when you are interested
in answering questions such as:
■■ How many surgeries do we do each week?
■■ What is the cost of each knee replacement
surgery?
■■ How long does each patient wait before
being seen by the doctor?
■■ How many home care visits do we conduct
each week?
■■ How many calories do I eat each day?
■■ What is the length of stay for each coronary
artery bypass graft (CABG) patient?
This chart is used frequently to address questions
related to volume, frequency of events, or financial
issues. Note that you are not interested in finding
out what percentage of surgeries started late (this
would be considering a late surgery start classified
as a defective, which would require a p-chart),
but rather you merely want to know how many
surgeries are done in the course of a day or a
week. In this case, the day or week becomes the
 Defining the Key Terms 237

subgroup (x axis label) and the total number of
surgeries completed each day or week becomes
the individual observation for that week (i.e., the
dot on the chart). In short, the XmR chart can
be used in many situations. Remember, however,
that the indicator being placed on the XmR chart
is not being classified as a defective or defect.
When you use the XmR chart you will usually
be asking a more neutral question such as “How
many of procedure X do we do?”
FIGURE 9-16 provides an example of an XmR
chart. In this particular example, the indicator of
interest is the total number of U.S. dollars saved
each month as a result of implementing a new
transcription system for radiology. Note that like
the X-bar and S chart there are two charts. The
top portion of the chart provides a plot of the
individual data points along with the average of
all the data points and the UCL and LCL. This
chart also has the zones identified. These are the
dashed lines that divide the chart into three areas
above and below the CL. As was discussed earlier
in this chapter, the zones are used to assist in
identification of special causes. Typically, and you
will see exceptions to this point, the zones are used
when you have a chart with equal subgroup sizes.
The bottom chart is referred to as the moving
range chart. The moving range is derived by calculating
the simple difference between two successive data
points on the Individuals chart and then plotting
this difference on the mR chart. This is also referred
to as creating an artificial subgroup of 2 since each
subgroup on the chart initially contains only 1 bit
of data. These steps are highlighted in Figure 9-16
by the circles drawn around each neighboring data
point on the top chart and the corresponding arrows
that point to the mR value between the coupled
data points on the lower chart. Notice that the first
three data points on the mR chart (Months 2–4)
are relatively close together. This is due to the fact

5750.0
5500.0 UCL = 5470.10
5250.0 A

5000.0
B
4750.0
Dollars saved

4500.0 C

4250.0 CL = 4360.90
C
4000.0
3750.0 B

3500.0 A
3250.0 LCL = 3251.70
3000.0

1500.0
UCL = 1362.79
Moving range

1000.0

500.0 CL = 17.20
0.0 LCL = 0.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Month

FIGURE 9-16 XmR chart for the total amount of dollars saved each month in radiology transcription
238 Chapter 9 Understanding Variation with Shewhart Charts

that there are small differences between the first
four data points that have been coupled together
on the Individuals chart. If you look at data points
for Months 15 and 16, however, you see a very
different picture. The difference (i.e., the range)
between Months 15 and 16 is much larger ($938
to be exact) than any of the ranges found when the
first four data points were compared. In short, the
individual values when coupled together produce
an artificial subgroup of two, which you must have
in order to calculate a range and subsequently the
moving ranges. One final thing to note about the
XmR chart. The mR chart will always have one less
data point on it than the Individuals chart. This
is due to the fact that the first data point (Month 1)
does not have a neighboring data point to be com-
pared with until the data for Month 2 is posted.
Therefore, there is no moving range for Month 1
on the mR chart, which is identified by the triangle
surrounding Month 1. This will always be the case
when you use an XmR chart.
EXERCISE 9-2 (You make the call: Is it an X-bar
and S chart or XmR chart?) will test your ability to
determine whether a particular indicator should
be placed on an X-bar and S chart or an XmR
chart. Answers to this exercise may be found at
the end of this chapter.
When we move to the Attributes side of the
Shewhart chart decision tree (Figure 9-12), we
need to address two questions:

EXERCISE 9-2 Is it an XmR (I) or X-bar and S? You make the call!

X-Bar and XmR

Indicator S Chart (I Chart)

Time to clean an inpatient room (in minutes)

Patient satisfaction scores for subgroups of 15 patients in the outpatient clinic

Average turnaround time for all STAT labs done each day

Cost for each normal delivery

A diabetic patient’s 3x a day blood sugar readings

Average length of stay for a subgroup of 20 intensive care unit (ICU)

patients

The distance (in feet) that a sample of 10 knee replacement patients can
walk in 15 seconds

■■ Do we have the occurrence and nonoccur-
rences of an event? If “yes” then we make a
p-chart (i.e., a percentage chart)
■■ If the answer is “no” meaning that we
have only the occurrence of an event (i.e.,
a defect when we do not know the non-
defects) then we need to ask, “Is there an
equal opportunity for a defect to occur?”
If we have an equal opportunity for a
defect to occur we make a c-chart. If not,
then we make a u-chart. The details are
explained next.
We start with the first question and address the
use of the p-chart.

P-Chart
The p-chart derives its name from the fact that
either a percentage or a proportion is what you
actually are plotting on the chart. Most of the
time the percentage will be the statistic of interest
rather than the proportion. When you make a
p-chart, or any other attributes chart, you will get
only one chart (unlike the variables charts which
gives you two charts). The p-chart is used to
monitor the proportion or percentage of defectives
when you know the occurrence of the defective
product, unit, event, or service (the numerator)
and the nonoccurrences (the denominator,
which is the total being observed). This chart is
used frequently in healthcare settings because
we track many indicators that look at accuracy,
completeness, errors, or the percentage of some-
thing done or not done (e.g., cesarean sections,
completed history and physical reports, proper
hand washing, or compliance with a standard
protocol). FIGURE 9-17 provides an example of
a p-chart with stair-step control limits. In this
case, the indicator is the percentage of hospital
readmissions for home healthcare patients. The
35.0
30.0
Percentage of readmissions
25.0
20.0
15.0
10.0
Denominator = 326
5.0 (widest limits)
0.0
1 2 3 4
FIGURE 9-17 p-chart on the percent of hospital readmissions for home healthcare patients
Defining the Key Terms 239
denominators for each dot on the chart are not
equal and spread from a minimum of 326 cases
(Month 3) to a maximum of 1,041 (Month 15).
Notice that the smallest denominator (326 for
Month 3) has the widest control limits whereas
the largest denominator (1,041 for Month 15) has
the tightest set of control limits. The numerators
go from a low of 75 readmissions (Month 3) to a
high of 249 in Month 16. If the distance or spread
between the stair-step limits is relatively small
this means that the denominators are relatively
close in size.
Finally, it should be noted that if the sub-
groups (the denominators) were of equal size, the
control limits on the p-chart would be straight.
But because most healthcare indicators that
are defined as percentages differ from one sub-
group (i.e., time period) to another (e.g., we do
not have the same number of deliveries each
month, produce the same number of food trays
each day, or have the same number of patients
visit at clinic each day) we usually do not have
equal subgroups when calculating percentages
or proportions. Therefore, most p-charts will
generally have stair-step control limits.
UCL = 24.59
CL = 20.64
LCL = 16.68
Denominator = 1041
(tightest limits)
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Month
240 Chapter 9 Understanding Variation with Shewhart Charts
C-chart
The c-chart and the u-chart are the Shewhart
charts of preference when you are tracking defects.
The c-chart is the appropriate chart when you
have an equal area of opportunity for a defect
to occur. As shown in Figure 9-13, an area of
opportunity has the following characteristics:
■■ It applies to all attributes or count charts
(c, u, and p)
■■ It defines the frame or area in which a
defective (i.e., a nonconforming unit) or
a defect (i.e., a nonconformity) can occur
■■ It can be of equal or unequal sizes.
A manufacturing example may help clarify this
concept. Imagine that you work on a paint line at
an automobile manufacturing plant. If you were
assigned to paint the hoods of a single model
of a car (e.g., a Ford Taurus) there would be an
equal area of opportunity for a paint blemish
because all Ford Taurus hoods are the same size.
In this case, we would make a c-chart and plot
the total number of paint blemishes (defects)
on each hood you paint. Because each hood has
the same number of square inches of surface
area there is a constant area of opportunity for
a paint blemish.
The challenge now becomes determining
when this equal area of opportunity condition
exists in healthcare settings. One of the more
frequently used examples of how this might occur
is with monitoring patient falls. If you conclude
that there is basically an equal opportunity for
a patient to fall each day of the week at your
hospital, rehabilitation facility, or long-term care
facility, then you would merely count the num-
ber of falls occurring each day, week, or month
and plot the number of falls on a c-chart. Other
indicators that could be placed on a c-chart if
the equal area of opportunity assumption was
met include the number of:
■■ Patient restraints
■■ Lawsuits
■■ Patient complaints
■■ Needle sticks
■■
■■
Medication errors
Central line infections
Figure 9-1 was used to show the elements of a
Shewhart chart. It also provides an example of
a c-chart. In this example, the customer service
manager of a large medical group is interested in
charting the total number of patient complaints
received each week at 17 sites of care. Because
each patient could file more than one complaint,
complaints are viewed as defects. The alternative,
considering the registration of a complaint as
a defective, is not selected because this would
preclude counting more than one complaint from
an individual patient. Remember that defectives
are based on the binomial distribution (i.e., the
patient complained or did not complain). If you
approached this indicator as a defective you
would not be concerned with the magnitude of
the complaint problem (i.e., the total number
of complaints) but rather with the fact that a
patient complained or did not complain and
you do not care if a patient complained more
than once. Measuring complaints as a defective
would produce a percentage of patients who
complained (a p-chart). As a defect, however,
we are concerned with the magnitude of the
problem so we count the total number of com-
plaints, including multiple complaints from the
same patient. The c-chart is selected because the
volume of patients seen at the 17 clinics Monday
through Friday remains fairly constant each week
and the number of sites included in the study
does not change. These two conditions allow the
manager to assume an equal area of opportunity
for a complaint to occur. She merely counts the
total number of complaints received each week
and plots this number on a c-chart. The chart
produces a CL (average number of complaints)
and a UCL and LCL.
A frequent challenge with using the c-chart
for healthcare applications is that the condition
of equal area of opportunity may not be met.
Frequently in healthcare settings there are few
indicators that have equal areas of opportunity.
The severity of a patient’s condition can change

quickly, the census can show fluctuations, the
clinic is not a 7 days a week operation, the volume
of orders may change rapidly, and the ED may
have to go on bypass because there are no more
inpatient beds available on a Friday or Saturday
night. So if the assumption of an equal area of
opportunity is violated, what do we do next? The
answer is simple—you make a u-chart.
U-Chart
This chart is used frequently in health care,
especially now that there has been a more con-
centrated effort to track patient safety indicators.
The u-chart, like the c-chart, is used to track
defects. The difference is that the u-chart is
selected when you conclude that there is not an
equal area of opportunity for the defect to occur.
Let us return to the paint line at the Ford plant
for a moment. Although you have in the past
painted one model of car at a time, today you
have been told that the line will have a mixture
of cars and a mixture of hood sizes. So, how do
you count the paint blemishes on the hoods
of a Ford Escort, a Taurus, a Mustang, and an
Expedition? Each hood has a different number
of square inches, takes a different volume of
paint to cover the surface of the hood, and has
a varying probability of experiencing a paint
blemish. The u-chart takes care of this problem
very quickly by computing a defect rate. The
number of paint blemishes is used as the nu-
merator and the number of square inches of the
hood’s surface is used as the denominator. The
resultant ratio provides the number of blemishes
per so many square inches of hood area. The
rate essentially normalizes the differences in
denominator size (i.e., the area of opportunity
for a blemish to occur).
One technical point about rates. Explaining
a rate-based statistic can be a little challenging. It
is much easier to say, “This past month we had
21 inpatient falls” than to say, “This past month
we experienced an inpatient falls rate of 4.4 falls
per 1,000 inpatient days.” Some in your audience
may struggle with what this rate-based statistic
Defining the Key Terms 241
actually means. An example should help clarify
how these are opposite sides of the same coin. If
we have 21 inpatient falls, this number becomes
the numerator of the rate-based statistic. When we
place this count of 21 falls over the total number
of patient days for the month (e.g., 4,775) we have
a ratio of two different numbers that produces
a result of 0.00439 (i.e., 21/4775 = 0.00439).
Because the number of inpatient days is in the
thousands we multiple the resultant value of
0.00439 by 1000 to produce the inpatient falls rate
of 4.4 falls per 1,000 inpatient days. The number
of spots you slip the decimal point on the resultant
ratio depends on how large your denominator is.
In this case, we had 4,775 inpatient days so we
slip the decimal point three places to the right
by multiplying the value of 0.00439 by 1000. If
you had patient days in the tens of thousands you
would slip the decimal point four places to the
right and have 43.9 falls per 10,000 inpatient days.
Or you could go out to 100k inpatient days and
say “I’m sorry but we had 439 inpatient falls per
100,000 inpatient days.” Or if you really wanted
to depress the senior management team or board
you could report 4,397 inpatient falls per 1 million
inpatient days. You can adjust the result of the ratio
of 21/4,775 very easily for any value you place in
the denominator position. The general rule for
rates, however, is that the denominator you use
should be based on the volume you are observing
on a regular basis. In the case of inpatient falls,
most hospitals are dealing with inpatient days
that are in the thousands so this is what should
be used to calculate the final rate-based statistic
of 4.4 falls per 1,000 inpatient days. If, on the
other hand, you were tracking medication errors
you would most likely be justified in making the
number of errors per 10,000 doses dispensed or
scripts written because an average-size hospital
will general dispense 10,000 or more doses each
month. Finally, if your measure was the neonatal
death rate for a state, province, or region then the
proper denominator size might be per 100,000
live births.
Because it is an extremely rare to have the
same number of medication orders each week,
242 Chapter 9 Understanding Variation with Shewhart Charts
the same patient census, or the same number
of central line days in the ICU, the u-chart is
used more often in healthcare settings than the
c-chart. Furthermore, because epidemiologists
frequently produce rate-based statistics (e.g.,
the neonatal death rate or the VAP rate) the
use of terms associated with the u-chart should
sound familiar to many healthcare professionals.
Examples of u-chart applications are provided
in the case study chapter (Chapter 10).
TABLE 9-1 provides an overview of the five
charts just described and offers examples of
indicators that could be placed on each type
of chart. Other useful tables that summarize
how charts should be set up and their various
uses can be found in Statistical Quality Control
Handbook (Western Electric, 1985) and Benneyan
(2001).10 Readers wishing to gain additional
insights about the selection of control charts
should consult Wheeler (1995), Montgomery
(1991), Pyzdek (1990), Ishikawa (1989), Duncan
(1986), Carey and Lloyd (2001), Carey (2003),
and Provost and Murray (2011).
▸▸ You Make the Call
Now that you are familiar with the basic ideas
behind the Shewhart charts, the next step is to
apply this knowledge to your own indicators.
The study questions in BOX 9-1 will serve as a
quick overview of some of the central issues
related to Shewhart chart development and as
a test of your current knowledge. If you struggle
with some of the questions you can review the
material presented in this chapter and then
explore some of the listed references for addi-
tional explanations. Another way to enhance
your knowledge base is to attend workshops
on SPC. The ASQ, for example, offers public
seminars on SPC. You may want to check with
your local ASQ chapter to see when such courses
will be offered. The IHI also offers workshops
on building effective measurement systems and
SPC. The various program offerings that I and
my colleagues teach throughout the year can be
reviewed on the IHI home page (www.ihi.org).
Finally, if you have the opportunity to attend a
local or national quality conference (e.g., the
IHI National Forum on Quality Improvement
in Healthcare or the IHI-BMJ International
Forum on Quality and Safety in Healthcare),
make sure that you sit in on sessions that are
discussing Shewhart charts and SPC. Hearing
about control charts from multiple sources will
be very beneficial.
You can also test your knowledge of the var-
ious charts by completing the You Make the Call
exercise found in EXERCISE 9-3. When I teach my
classes on Shewhart chart applications, I give the
participants this exercise at the end of the class
to provide a final test of their understanding of
the selection of appropriate Shewhart charts. It
gives them a chance “make the call!” and tests
their control chart knowledge. The indicators
listed in this exercise are taken from actual teams
I have had the opportunity to facilitate or coach.
Start the exercise by determining the subgroup.
Remember that the subgroup is the label for
the horizontal axis and reflects how you have
organized your data (e.g., by day or week). Next
decide if you have variables or attributes data.
Finally, list the chart you think is most appropriate
for this situation. You may want to refer to the
Shewhart Decision Tree shown in Figure 9-12 to
assist you in thinking through the chart options.
The answers to the You Make the Call exercise
can be found at the end of this chapter.

TABLE 9-1 Shewhart chart summary
Type of Control Chart
X-bar and S chart
This is known as the Average
(X-bar) and Standard Deviation
(S) chart. Most SPC software
programs will give you two
charts when you select this
chart: one for the X-bar portion
and one for the S portion. This
is considered to be the most
statistically powerful of all the
charts. The X-bar and S chart
can have straight or stair-step
control limits.
XmR chart
This chart is known as the
Individual values (X) and
moving range (mR) chart.
Sometimes it will be referred
to as the Individuals or I-chart.
It does not have the statistical
rigor or power of the X-bar
and S chart because each dot
on the chart is representing
only one observation. This
chart is used frequently to
answer questions related to
volume, for example, “How
many surgeries did we do this
week?” The XmR chart does
not address the question as to
whether these surgeries were
started on time (this would
require a p-chart). Instead,
the XmR chart is answering a
neutral question, “How many?”
or “How much?” The XmR
chart will always have straight
control limits.
Type of Data and Data Examples of Indicators
Collection Issues Used on This Type of Chart
Variables data ■■
The X-bar and S chart usually
involves drawing a sample of
observations (e.g., 3–10 per ■■
subgroup). Rational subgrouping
is frequently used with this ■■
chart. The statistical principles
behind this chart are based
on the assumptions of the ■■
normal (Gaussian) bell-shaped
distribution. ■■
Variables data ■■
The XmR chart is used when you
have a single observation for each
subgroup (i.e., n = 1). Sampling ■■
typically is not done but might be
if the process being monitored
has an extremely large volume. ■■
Because this chart frequently uses
aggregates as the plotted number
(e.g., days in accounts receivable ■■
this month), it is important to
make sure that the data are
consistently collected from one
time period to the next. This chart ■■
is used to evaluate questions
related to process outcomes ■■
(volumes), with no concern as
to whether the outcomes of the ■■
process are acceptable or not
acceptable. ■■
■■
■■
You Make the Call 243
Actual turnaround time
for five lab tests or three
pharmacy orders each day
Blood pressure readings
(e.g., three to five per day)
Diabetes monitoring (e.g.,
three fasting blood sugar
readings each day)
Anesthesia time for a
sample of cases each day
Patient satisfaction scores
Patient wait time to see
the physician or to be
seen in the ED
The number of days to
mail a patient bill after
discharge
The number of calls
coming into a clinic each
day
Average length of stay
by week for a particular
diagnosis-related group
(DRG)
The number of surgeries
done each week
Operating margin by
month
Pounds of laundry
each day
Average turnaround time
by day
The number of food trays
produced
Patient satisfaction score
(continues)
244 Chapter 9 Understanding Variation with Shewhart Charts
TABLE 9-1 Shewhart chart summary
Type of Control Chart
p-chart
The p-chart is used frequently
in health care to compute the
percentage (or proportion)
of defective products or
services. The p-chart requires
being able to count both
the numerator and the
denominator. The p-chart is
the weakest of the attributes
charts because it is based on
the binomial distribution (i.e.,
there are only two outcomes
such as yes/no, acceptable/
not acceptable, or complete/
not complete. The p-chart
can have straight or stair-step
control limits.
c-chart
The c-chart is used to count
the number of defects that
occur within an equal area
of opportunity when the
nondefects are unknown.
In this case, each observed
unit (e.g., a patient) can have
multiple defects (e.g., falls).
Generally speaking, defects
are the specific reasons why a
product or service is classified
as defective (i.e., a defective
product or service will suffer
from one or more defects).
Generally speaking, indicators
appropriate for a c-chart
should be considered “rare
events.” The c-chart will always
have straight control limits.
(continued)
Type of Data and Data Examples of Indicators
Collection Issues Used on This Type of Chart
Attributes data ■■ Percentage of cesarean
These data are classified as sections
defectives or nonconforming ■■ Percentage of late food trays
units because they reflect the ■■ Percentage of incomplete
percentage (or proportion) of charts
things or events that do not meet ■■ Percentage of late surgery
specifications or criteria (the starts
numerators). The denominators ■■ Percentage of bills that are
usually (but not always) are of inaccurate
varying sizes, which produce ■■ Percentage of mortality
stair-step control limits. Data of ■■ Percentage of staff
this type reflect the binomial turnover
distribution. The denominators ■■ Percentage of patients
need to be sufficiently large (e.g., responding “Very Good” to
usually greater than 12) to enable a survey question
a reasonable percentage to be ■■ Percentage of x-rays that
calculated yet not too large (e.g., had to be redone
over 5,000). ■■ Percentage of did not
attends (DNAs) at an
outpatient clinic
Attributes data ■■ The number of falls
The key to using a c-chart is that ■■ The number of restraints
there should be an equal area of ■■ The number of needle
opportunity for a defect to occur. sticks
This condition frequently makes it ■■ The number of lawsuits
difficult to use this chart in health filed
care because the conditions under ■■ The number of ventilator-
which we provide care do not associated pneumonias
always remain constant. One way ■■ The number of
to address this inequality in the nosocomial infections
area of opportunity is to apply ■■ The number of
stratification. For example, if the medication errors
conclusion is that there is not an ■■ The number of returns to
equal area of opportunity for an surgery
inpatient fall because the hospital ■■ The number of surgical
functions differently on weekends site infections
than weekdays then separating the ■■ The number of violent
data by weekdays versus weekends events in a mental health
may be sufficient to conclude that ward
there is a relatively equal area of ■■ The number of central line
opportunity for a fall during each of infections
these periods. The c-chart is based
on the Poisson distribution.
 You Make the Call 245

TABLE 9-1 Shewhart chart summary (continued)

Type of Data and Data Examples of Indicators

Type of Control Chart Collection Issues Used on This Type of Chart

u-chart Attributes data ■■ Medication errors per

The u-chart is used to track
defects when the area of
opportunity is not equal. For
this reason, the u-chart is
typically used more often in
health care than the c-chart.
This chart is based on rates
rather than simple counts. The
u-chart can have straight or
stair-step control limits.
The Poisson distribution is also
used as the frame of reference for ■■
this chart. The u-chart presents ■■
rates (e.g., so many falls per 1,000
patient days). Knowledge of how ■■
to collect data to form rates is
essential. ■■
10,000 doses dispensed
VAP per 1,000 vent days
Total falls per 1,000 patient
days
Total readmissions per
1,000 discharges
Bloodstream infections
per 1,000 line days

BOX 9-1 Shewhart charts study questions

■■ When is it appropriate to use Shewhart charts? Should I use them in place of descriptive statistics?
■■ What is the relationship between Shewhart charts and tests of significance?
■■ How many data points do I need to make a Shewhart chart? What do I do if I do not have
enough data?
■■ Which is better, attributes or variables data?
■■ What is a subgroup? Do I have to have one to make a Shewhart chart?
■■ Can I make a Shewhart chart with only single data points?
■■ Do my subgroups have to be of equal size when I make Shewhart charts?
■■ Much of the data I get does not have the date on it. So, does it really matter if the data points are
not in chronological order?
■■ I still don’t get this distinction between a SD and a sigma limit. Why aren’t they the same? Does it
really matter? My spreadsheet software will give me a SD. Why can’t I just multiple this number by 3
and then add and subtract this product from the mean to get the control limits?
■■ Why do I have to use 3 sigma control limits? Why can’t I use two or maybe 1.5 sigma limits?
■■ Do defects add up to make defectives or is the other way around?
■■ When I make a p-chart, does the size of the denominator make a difference? Can I have, for
example, 4 or 5 in my denominator?
■■ What is the difference between a proportion, a percentage, and a rate?
■■ Should I view common cause variation as “good” variation and special cause variation as “bad”
variation?
■■ Do I really have to investigate a special cause? Can’t I just remove the data point from the chart and
get on with making changes?
246 Chapter 9 Understanding Variation with Shewhart Charts

EXERCISE 9-3 You make the call! Selecting the right chart

Type of Type of
Situation Subgroup? Data? Chart?

1. Each day you record the number of films processed in V or A

the radiology department.

2. Each day you record the number of films requested V or A

and the number that cannot be found in the radiology
library.

3. The number of inpatient restraints each month is V or A

placed over the total inpatient days each month.

4. Each day you pull a stratified random sample of V or A

15 complete blood counts (CBCs) and record the
turnaround time (in minutes) for each CBC.

5. The number of minutes it takes to get a stat med V or A

order administered to the patient (order time to
administration time).

6. Every 2 weeks you pull a sample of 30 medication V or A

orders and count the total number of orders that have
one or more errors.

7. The wait time in the ED (door to discharge) is tracked V or A

for each patient.

8. The clinic receptionist notes the time of check-in for V or A

each patient. The physician notes the time when he/
she first sees the patient in the exam room. An analyst
compiles the data daily and reports the percentage of
patients who had to wait more than 30 minutes.

9. The director of surgery keeps track of the total number V or A

of surgical procedures performed each week.

10. The dietary department records the number of food V or A

trays that come back uneaten each day and the total
number of trays they produced for that day.

11. You are interested in the average time patients spend V or A

in your waiting area, so every day a student randomly
picks eight patients and measures their actual waiting
time in whole minutes.

(continues)
 Additional Shewhart Charts 247

EXERCISE 9-3 You make the call! selecting the right chart (continued)

12. The ICU nurses want to evaluate the ventilator- V or A

associated pneumonia (VAP) rate. So every 2 weeks
they record the total number of pneumonia episodes
and the total number of vent days.

13. Each week patient satisfaction scores for three units V or A

are compiled and an average is calculated for the three
units.

14. The finance department tracks the total number of V or A

business days it takes to process a vendor’s request
for payment. Process time starts when the request
for payment is received in the finance department
and ends when the payment is sent (electronically or
posted in the mail) to the vendor.

15. Every week each medication order is checked against V or A

five potential types of errors. The total number of errors
for the week is divided by the total number of orders
submitted that week.

16. You know the number of people who come to the ED V or A

complaining of chest pain and the number who are
actually diagnosed with an AMI or unstable angina.

▸▸ Additional Shewhart ■■
■■
Multivariate Shewhart-type Charts
P primed chart (p′-chart)
Charts ■■ U primed chart (u′-chart)
Provost and Murray (2011) do a very good job of
In addition to the five basic Shewhart charts not only describing these alternative charts and
described previously, there are many other charts provide examples of their use. I do not intend to
that have their roots in manufacturing but have go into depth about these various charts but I do
proven to be very useful in certain healthcare want to make a few comments about the t- and
situations. Some of these alternative Shewhart g-charts that are being used more and more in
charts include: healthcare improvement work.
■■ Median chart The t- and g-charts are designed to address
■■ t-chart the occurrence of rare events. I know, you are
■■ g-chart wondering, “What is the operational definition
■■ Moving average chart of a rare event?” When I was first learning
■■ Cumulative Sum chart (CUSUM) about these charts the instructor used a simple
■■ Exponentially weighted moving average example. He would ask, “What is the probability
chart (EWMA) of looking out the window and seeing a car go
■■ Standardized Shewhart Chart by?” Everyone would respond, “High.” Then he
248 Chapter 9 Understanding Variation with Shewhart Charts
would ask, “What is the probability of looking
out the window and seeing an accident occur?”
Everyone would respond, “Low.” He then would
proudly announce, “You now understand a rare
event.” Now this is a pretty casual explanation of
a rare event but I think it helps to set the context
for thinking about rare events. If you wish to get
very statistical about rare events you can study
what is called the “rare event rule for inferential
statistics.” Within this body of statistical theory
you will be reacquainted with probability theory
that you were exposed to relatively early in your
statistical training. Most of the time this is ex-
plained by using the probabilities associated with
rolling various combinations on dice or getting
a particular combinations of cards while playing
blackjack or poker or betting on a roulette wheel.
At the IHI we use a practical approach to
defining rare events that is grounded in statistical
theory but does not require detailed compu-
tations. Simply stated, if you have more than
25% of the data on a p-, c-, or u-chart at zero
(or conversely at 100%) you need to consider
moving to a t- or g-chart. With 25% or more of
the data points at zero the use of the traditional
rules for detecting special causes on a Shewhart
chart become questionable (Provost & Murray,
2011). It also is a practical issue. If you do not
have sufficient nonzero data for an attribute
chart the LCL may not exist, which makes the
interpretation of the chart difficult. In these
situations, you should consider moving to the
time between chart (t-chart) or the cases between
chart (g-chart). The t-chart (the t part of the
name refers to “time”) or time between chart
shows you how much time has gone by since the
last adverse event. Nelson (1994) provided the
details on how this chart is constructed. When
you use this chart you have to reorient the way
in which you explain the chart. For example, the
horizontal axis on the t-chart is a discontinuous
time sequence. If you start next Monday to begin
tracking patient falls but a fall does not occur
until Wednesday then you would place a dot
on the chart’s vertical axis at 2 (i.e., 2 days have
gone by before a fall occurred). When an event
happens (e.g., a fall, a pressure ulcer, a surgical
site infection) you basically reset the counter and
begin counting the number of days again until
the next fall occurred. This is the same approach
that factories use to track the number of days that
have gone by without an accident in the factory.
If you never had an event you would never have
a dot on the chart. Because you are counting the
number of days that have gone by since the last
event (i.e., a defective or a defect) the horizontal
axis will not have Monday, Tuesday, Wednesday
or January, February, March, etc. marked. When
an event occurs this is when you place the date
of the event on the horizontal axis, which will
not be occurring in equal periods of time. The
indication of improvement on a t-chart is when
you observe an ever-increasing run of days
without the occurrence of an event.
As healthcare providers have become more
focused on safety indicators and reducing harm
the t-chart has become increasingly popular.
But a word of caution is in order. The statistical
basis for properly calculating the limits on a
t-chart are a little involved. First, you need to
realize that a distribution of rare events does not
follow a normal Gaussian bell curve. A Poisson
distribution is a better referent for rare events.
The Poisson distribution is appropriate as a
referent for the c- and u-charts as well as the
time between chart. In the case of the t-chart,
however, the form of the Poisson distribution
is actually an exponential distribution, which
is in turn highly skewed. Second, the skewness
of the exponential distribution is not a major
problem and is addressed by transforming the
time between events (i.e., days gone by) into
a quasi-normal or symmetric distribution by
performing what is called a Weibull transfor-
mation. Third, once the data are more or less
approximating a normal distribution the UCL
and LCL can be calculated by using the formulae
for the XmR chart. Finally, after the limits and
CL are calculated they are transformed back
to their original state for plotting on the chart.
I know, this all sounds rather complicated.
The detailed steps for constructing the t-chart

are clearly discussed by Provost and Murray
(2011) and Nelson (1994). Also take heart
in knowing that any reasonably good SPC
software package will do all the calculations
for a t-chart quickly and easily once you cre-
ate the time between data file. Your biggest
challenge will be to explain how to interpret
the t-chart. But because its use is growing
in popularity in healthcare setting, it is well
worth your time to gain more knowledge of
the time between chart.
The g-chart (or geometric chart) is similar in
principle to the t-chart. It too is a chart for rare
tracking events except that instead of plotting
the amount of time (e.g., days) between a rare
event, the g-chart plots the number of cases
that are regarded as being successful against
cases considered to be failures. A failure in
this situation might be a surgical site infection,
patients experiencing a medication error, or
a return to surgery within 24 hours. Like the
t-chart success is determined by having a long
run of successful cases with no failures or ad-
verse events. Although the t-chart is modeled
after an exponential distribution the g-chart
referent is a geometric distribution. Again,
the steps for computing the limits and the CL
on a g-chart are nicely laid out in Provost and
Murray (2011). There is also a considerable
body of literature on both the exponential and
geometric distributions that can be found in the
ASQ's Journal of Quality Technology. The g-chart
is also becoming a standard offering in most
SPC software packages. Once again, however,
the challenge is making sure you have at least
a moderate foundation in being able to explain
the chart and how to interpret it.
▸▸ Using Shewhart Charts
Effectively
At some point after reading various books on
run and Shewhart charts and listening to others
explain control chart theory, you will all of a
Using Shewhart Charts Effectively 249
sudden proclaim, “I think I get it!” Once you
reach this point it is now time to start applying
this knowledge to actual improvement oppor-
tunities. But be careful. I have seen some people
become so enthusiastic about the various charts
that they start making graphs on any process that
produces data. It is at this stage that I remember
an old adage—if you give a child a hammer, the
whole world looks like a nail! The charts play a
valuable and central role in all QI efforts. It is
important to realize, however, what they can do
and what they cannot do.
First, appreciate the fact that the charts do
not answer the following questions:
■■ What is the reason for a special cause?
■■ Should a common cause process be improved?
■■ What should I do to improve the process?
The answers to these questions do not come
from the charts or statistics. They come from
the will, ideas, and ability of the team to execute
tests of change. I have seen too many teams feel
that once they have created a chart their work
is finished. I think that this occurs because the
chart is a tangible thing that can be pointed to
and shown to others. Improvement strategies,
on the other hand, are not as finite or discrete.
Developing improvement strategies is actually
much more difficult than mastering control
chart theory and construction because you are
dealing with people, behaviors, and culture
not numbers.
Second, after you make a chart and decide
whether the process exhibits common or special
cause variation, you then need to decide how
you are going to approach the variation you
have identified. Do you need to merely reduce
variation in the process or fundamentally redesign
the process and change the way in which work
is envisioned and delivered? All improvement
strategies emanate from an understanding of
variation. If the process exhibits special cause
variation the appropriate decision is to investigate
the special cause(s) and determine why they have
made the process unstable and unpredictable.
Just as we would investigate a patient safety event
250 Chapter 9 Understanding Variation with Shewhart Charts
(i.e., a sentinel event) by conducting a root
cause analysis, we also need to do the same
Notes
thing when a special cause is detected on a run
or Shewhart chart. Ignoring a special cause will
guarantee that it will rear its ugly head at some
point in the future. We cannot predict exactly
when a special cause will occur but you can
be sure that it will pop up again if you choose
to ignore it.11
The other aspect of a special cause is that not
every special cause is negative and undesirable.
Remember that special causes are not bad and
common causes are not good. The key point
is the special causes make a process unstable
and unpredictable. It is very likely that you will
observe a special cause that you want to emulate
(e.g., when lab turnaround time is much faster
than it has been or the past 15 days). In this
case, you want to investigate why the process
worked so well on those days and see whether
these conditions can be replicated. Common
causes on the other hand are not inherently
good. Common cause variation merely means
that the process is stable and predictable (i.e.,
predictable within the boundaries of the UCL
and LCL). Just as you can have a special cause
that you might want to emulate, you can also
have common cause variation that is unac-
ceptable (e.g., when a patient’s blood pressure
is running at a very high level and staying
there or when the wait time to see your family
physician is consistent and predictable but it
is at such a high level that it is predictably bad
and unacceptable).
QI starts with making the correct decision
about the variation that lives in your data. Walter
Shewhart introduced the control chart and the
notions of common and special causes of variation
in 1924 (Hare, 2003). Since then SPC has become
the foundation for all successful QI initiatives.
It is a key component of the Baldrige criteria, Six
Sigma, Lean, and International Organization for
Standardization (ISO). Without a clear under-
standing of variation and its causes, however,
individuals and organizations will continue to
suffer from numerical illiteracy.
1. Historically these charts have been known
as control charts. Shewhart himself even
referred to them as control charts as have
many writers since Shewhart’s time. But as
Blank (1998, p. 1) points out, “It is important
to understand that SPC does not control
processes. People control processes. SPC
is merely a tool that provides you with
information you need to reduce variation
and tell you whether or not your processes
can meet the customer’s expectations.” In
more recent times, the charts have been
referred to more and more as Shewhart
charts (Provost & Murray, 2011, p. 113)
to emphasize their use primarily in un-
derstanding variation and to facilitate
learning about process capability rather
conveying images of “control.” The term
Shewhart chart is also used to recognize the
significant contributions of Dr. Shewhart to
the field of SPC. A final note on the use of
the word “control.” The ASQ was originally
called the American Society for Quality
Control (ASQC). In 1997, the membership
voted to drop the word “control” from the
organization’s name. This was to recognize
that quality was becoming a broader con-
cept and used in many other fields besides
manufacturing where initially in the early
1900s control was used as a key operative
word. Shewhart’s book, Economic Control
of Quality of Manufactured Product (1931)
provides a classic reference to the initial
use of the term “control.” So for a variety
of reasons I use the term Shewhart chart (s)
in this text rather than control charts.
2. The USL and LSL are frequently referred
to in manufacturing as “tolerance limits”
and are also frequently referred to as the
voice of the customer (VOC, i.e., what the
customer wants, needs, or expects from
the product or service). There are many
different types of indices that have been
developed to capture statistically process

capability. The three basic process capability
indices are the process capability index
(Cp), the minimum process capability
index (Cpk), and the process capability
index to the mean (Cpm). The traditional
statistical use of process capability (Cp)
is to indicate whether or not the process
can meet the predetermined specifica-
tions (Blank, 1998). There are numerous
variations on the Cp statistic, all of which
are designed to help the quality control
(QC) researcher investigate special causes
and get the process to perform as closely
as possible to the expectations of the
customer (i.e., the specifications).
3. I have calculated a Cp and Cpk statistics
only once for a healthcare indicator. It
was when I was helping to set up an out-
patient clinic designed to manage patients
on anticlotting medication (i.e., warfarin
sodium). Several key indicators are used
in assessing clotting issues. The PT, along
with its derived measures of prothrombin
ratio (PR) and INR, are assays evaluating
the extrinsic pathway of coagulation.
This test is also called “ProTime INR”
and “PT/INR” (MedlinePlus medical
encyclopedia, https://medlineplus.gov/
ency/article/003652.htm). Because there
are defined therapeutic limits associated
with these measures they can be regarded
as USL and LSL. These values would be set
on the Shewhart charts as reference lines.
Then the patient’s actual results on the PT
and INR would be plotted on the chart,
and the UCL and LCL of the patient would
then be compared to the USL and LSL.
Because we had both an USL and a LSL and
control limits the capability statistics could
be calculated to determine how well the
patient was conforming to the therapeutic
limits (USL and LSL) of the drug. But in
most instances in healthcare settings, there
is only a single target or goal rather than
the USL and LSL. Healthcare simply does
not currently function at the same level of
Notes 251
precision for many of its indicators as the
manufacturing industry. So, when I use the
term process capability I am using it in a
general sense to describe the variation in
the process as defined by the mean (CL)
and the UCL and LCL. These numbers
define how well the process is performing
relative to the target or goal.
4. Some of my colleagues may disagree with
these guidelines. I have found over the years
that there are two general issues that need
to be balanced against each other: statisti-
cal purity and practicality. The science of
improvement (SOI) is as Shewhart referred
to it an “applied science.” Therefore, in my
work I have always tried to balance the
precision of statistical requirements with
a heavy dose of practicality. For example,
I have worked with wonderful people in
the National Health Service (NHS) of
Scotland for over 12 years. During this
time we have developed a variety of health
and social service measurement systems.
Most of the data are collected monthly and
many of the indicators were not collected
historically. So, we were starting out with
no data on selected indicators and had
to build charts as we went along. In this
case, trail control limits were essential.
We also made a very practical decision to
use the first 6 months of data as baseline
for indicators that had no history. Again
some would argue that this is not enough
data to establish a baseline but it was
sufficient to get us started on the road to
improvement.
5. In one of my measurement workshops,
a few years ago this confusion was high-
lighted very clearly. A young woman
near the front of the room raised her
hand after I was done explaining that a
sigma was not equivalent to a SD. She
had a bit of a wrinkled brow and looked
concerned. She said, “I was told that the
UCL and LCL were calculated as SD. Is
this not correct?” I drew the formula for
252 Chapter 9 Understanding Variation with Shewhart Charts
the SD on a flipchart and asked her if this
is what she used to calculate the three
sigma control limits. She said “Yes, that is
what I was told to use.” I then proceed to
politely tell her that the control limits on
her charts were wrong. The UCL and LCL
would either be too wide or too narrow
if she used the SD of the data. She got a
very strange look on her face, was quiet
for a moment, then burst out, “But this
means I have been giving the charts to
the senior management team and to the
board! What am I going to do?” I asked
her if anyone had ever noticed or com-
mented on the fact that the limits were not
properly calculated. She responded, “No.”
I suggested that she learn how to make
the charts correctly with SPC software
(she was merely using Excel with no SPC
add-on software that properly computes
sigma limits) and then submit the correct
charts to the senior management team
and board the next time around. She still
looked a little perplexed, however. She was
concerned that she would lose credibility
with the management team when they
found out the charts were wrong. I told her
that unless she tells them that her original
charts had the wrong limits it does not
sound like anyone on the board or the
senior management team had sufficient
grounding in Shewhart charts to actually
discern that the charts were different. I
told her to let me know how it went when
she showed them the correct charts. She
wrote back and said that no one asked
any questions.
6. API develops methods, works with leaders
and teams, and provides education and
training to help organizations improve their
products and services and to build their
capability for ongoing improvement. The
principals of API have worked in industrial,
educational, health, and social service
settings. They have worked extensively
with Dr. Deming and provided support
for his 4-day seminars. They have written
a number of key books on QI including
The Improvement Guide (Langley, Moen,
Nolan, Nolan, Norman, & Provost, 2009),
Quality Improvement Through Planned
Experimentation (Moen, Nolan, & Provost,
2012), and The Health Care Data Guide
(Provost & Murray, 2011).
7. When Dr. Ray Carey and I first started
teaching control chart applications to
healthcare professionals in 1992, we
taught the traditional list of seven control
charts. In 1995, we wrote a book that
described these seven charts and their
use (Carey & Lloyd, 2001). In December,
2002, Dr. Carey and I taught a minicourse
and two workshops on control charts
at the 14th National Forum on Quality
Improvement in Health Care sponsored
by the IHI. This was the first time in the
12 years that we had been teaching for
the IHI that we reduced the number
of charts we taught from seven to five.
The sessions were well received, and the
participants found the more simplified
approach to be appealing. The two charts
we dropped were the X-bar and R chart
and the np-chart. Our reasoning for doing
this was that the X-bar and S chart can
be used in any situation that calls for the
X-bar and R chart (when the subgroup
is greater than 2). The np-chart, which
is a count of the number of defectives,
requires equal subgroup sizes (i.e., the
denominators), which do not happen very
often in healthcare settings. The p-chart
can be used effectively, however, in any
situation where an np-chart could be used.
If there are equal subgroup sizes then the
p-chart will have straight control limits.
If, on the other hand, the measure has
unequal subgroups then the p-chart will
have what is known as “stair-step” control
limits. In this case, the control limits are
different for each data point. The closer
in size the denominators the smaller the

“steps” between each of the control limits.
If there are large differences between the
denominators the “steps” will be greater
between the individual data points.
8. I wrote a commentary in JAMA a few
years ago titled “A Matter of Time” (Lloyd
&G ­ oldmann, 2009) highlighting how
clinicians, researchers, patients, and
improvement specialists all have very
different views of time. To these four
categories I could add management time,
which focuses on monthly aggregates
of data.
9. The control chart examples presented
in this chapter have been developed to
demonstrate the five different charts. The
substantive importance of the various
charts is not the focus of this chapter. The
charts have been developed for heuristic
purposes, and the clinical or operational
impacts of the indicators presented on
the charts are not the primary objective
in this chapter. Analysis and interpreta-
tion of control charts are addressed in
Chapter 10.
10. The idea for creating this table came from
Dr. James Benneyan of Northeastern
University in Boston. In a paper titled
“Design, Use, and Preferences of Statistical
Control Charts for Clinical Process Im-
provement” (September 16, 2001), he used
a table to summarize the various charts.
After reading this paper, I realized that
the table was something I had not used
to summarize the control charts. I believe
a table format works nicely to augment
the utility of the decision tree shown
in Figure 9-12 and the textual details.
Dr. Benneyan has written extensively
on the topic of control charts in health
care and I would encourage readers to
review his work. He can be reached at the
­following address: MIME Department, 334
Snell Engineering Center, Northeastern
University, Boston, MA 02115; phone 617-
373-2975; email benneyan@coe.neu.edu.
Answers to the Chapter 9 Exercises 253
11. There are many good examples of how
people have ignored special causes when
they first occurred and then decided to
deal with them when they popped up
again. The terrorist attacks on our nation
on September 11, 2001 provide a classic
example. Several years prior to 9/11
the World Trade Center was bombed
by terrorists. Although this seemed to
draw the nation’s attention for a while,
interest in this special cause soon faded
into the “old news” category and steps
were not taken to extricate the factors
that led to the special cause. The condi-
tions for 9/11 were still existing within
our system. The September 11 special
cause, however, generated a completely
different reaction. Our nation mobilized
not only to investigate the special cause
but also take steps to literally try to
eliminate the origin of the special cause.
Every day there are stories in the news
that should prompt a discussion as to
whether the event is a special cause or
part of a common cause system. All too
often, however, we overreact to a special
cause and want to change the system
without fully investigating the reasons
why it occurred. Other times, however,
we ignore a special cause and “hope” that
it will not happen again. Hope is not a
plan. Knowing how to appropriately react
to common and special causes is a much
better approach than hoping a special
cause will not pop up again.
Answers to the Chapter 9
Exercises
This section provides the answers to the exer-
cises presented earlier in this chapter. The first
(EXERCISE 9-1) deals with differentiating defectives
from defects. EXERCISE 9-2 provides indicators
that could be placed on either an X-bar and
S chart or an XmR chart. The answers to these
two exercises are shown here.
254 Chapter 9 Understanding Variation with Shewhart Charts

EXERCISE 9-1 Defective or defect? You make the call! (Answers)

Defective Defect
Indicator (Classification) (Count)

1. Number of accidents per 1,000 employee days *

2. Number of errors per 25 food trays *

3. Percentage of AMI patients receiving aspirin within *

24 hours of arrival in the ED

4. Percentage of inpatient deaths each month *

5. Number of surgical complications per 1,000 surgeries *

performed

6. Proportion of hand hygiene observations done incorrectly *

7. Number of falls per 1,000 patient days *

8. Number of medication errors per 10,000 doses dispensed *

EXERCISE 9-2 You make the call: Is it an X-bar and S chart or XmR chart? (Answers)

X Bar and S XmR

Indicator Chart (I Chart)

Time to clean an inpatient room (in minutes) *

Patient satisfaction scores for subgroups of 15 patients in the *

outpatient clinic

Average turnaround time for all STAT labs done each day *

Cost for each normal delivery *

A diabetic patient’s 3x a day blood sugar readings *

Average length of stay for a subgroup of 20 ICU patients *

The distance (in feet) that a sample of 10 knee replacement *

patients can walk in 15 seconds

The final (EXERCISE 9-3) brings together the
key issues related to selecting the most appro-
priate Shewhart chart for different measurement
situations. In this exercise, the subgroup, type of
data, and type of chart all need to be specified.
Depending on how you interpret the word-
ing describing the situations in Exercise 9-3,
you might think that a type of chart other
than that I have listed could be selected. A
key leaning point for this exercise is that slight
changes to the wording of the situation could
lead you to selecting a different chart. For
example, take a close look at situations 8 and
11 in Exercise 9-3. The wording for situation
8 points you to select a p-chart because they
decided to focus on patients who had to wait
more than 30 minutes. Even though they had
variables data (i.e., time) they basically turned
it into attributes data because of the 30-minute
target. They have taken the more powerful
form of data (variables data) and relegated it
to a binomial condition, over 30 minutes and
under 30 minutes. They will never understand
the true variation in wait time. What is the
longest wait? You have no idea. All we know is
that a certain percentage of patients had to wait
more than 30 minutes. The longest wait could
be 31 minutes or 13,184 minutes. The more
appropriate approach is found in situation 11.
Here they are taking a sample of eight patients
each day and recording their actual wait times.
The chart of preference in this situation is
the X-bar and S chart. We will now have the
average wait time for a given day and the SD
from this average. We can lay a separate line
of the chart showing the target of 30 minutes.
This gives us much more information about
the process variation and how capable it is of
achieving the target, which cannot be deter-
mined by using the p-chart. The final aspect
Answers to the Chapter 9 Exercises 255
of this situation is that if the target is to have
all patients be seen in 30 minutes or less, the
30-minute target actually needs to be the UCL
of the X-bar and S chart not the average. If 30
minutes is the average on the chart you will
naturally have some patients waiting more
than 30 minutes and some waiting less. A
target is useful on a chart but it needs to be
understood in light of the actual variation in
the process and the capability of the current
process to achieve the target. The Shewhart
chart can help you determine the magnitude
of improvement needed to achieve the target
and but in the case of improving wait time, this
is best accomplished by not turning variables
data into attributes.
The most appropriate chart for each situation
described in Exercise 9-3 is shown here. Note
that situation 16 is a trick question. Did you
determine that a chart cannot be identified?
Why? Because there is no subgroup identified in
the situation description. Remember, a Shewhart
chart must have a subgroup and an observation
as minimum requirements. In this situation,
there is no subgroup. But if the situation had
been worded as follows then we would have
a subgroup: “You know the number of people
who come to the ED complaining of chest
pain EACH MONTH and the number who are
actually diagnosed with an AMI or unstable
angina.” Now you would be able to determine
which chart is most appropriate. In this situ-
ation, the Shewhart chart of choice would be
the p-chart because we know the denominator
(i.e., the number of people coming to the ED
complaining of chest pain) and the numerator
(i.e., the number who were actually diagnosed
with an AMI or unstable angina). Without a
subgroup, however, we cannot make a decision
about which chart is most appropriate.
256 Chapter 9 Understanding Variation with Shewhart Charts

EXERCISE 9-3 You make the call!: Selecting the right chart (Answers)

Type of Type of
Situation Subgroup? Data? Chart?

1. Each day you record the number of films processed in Day V XmR
the radiology department.

2. Each day you record the number of films requested Day A p-chart
and the number that cannot be found in the
radiology library.

3. The number of inpatient restraints each month is Month A u-chart

placed over the total inpatient days each month.

4. Each day you pull a stratified random sample of Day V X-bar & S
15 CBCs and record the turnaround time (in minutes)
for each CBC.

5. The number of minutes it takes to get a stat med Stat med V XmR
order administered to the patient (order time to order
administration time).

6. Every 2 weeks you pull a stratified sample of 30 Two weeks A p-chart

medication orders and count the total number of
orders that have one or more errors.

7. The wait time in the ED (door to discharge) is tracked Patient V XmR

for each patient.

8. The clinic receptionist notes the time of check-in for Day A p-chart
each patient. The physician notes the time when he/
she first sees the patient in the exam room. An analyst
compiles the data daily and reports the percentage
of patients who had to wait more than 30 minutes.

9. The director of surgery keeps track of the total Week V XmR

number of surgical procedures performed each week.

10. The dietary department records the number of food Day A p-chart
trays that come back uneaten each day and the total
number of trays they produced for that day.

11. You are interested in the average time patients spend Day V X-bar & S
in your waiting area, so every day a student randomly
picks eight patients and measures their actual waiting
time in whole minutes.
 References 257

EXERCISE 9-3 You make the call!: Selecting the right chart (Answers) (continued)

Type of Type of
Situation Subgroup? Data? Chart?

12. The ICU nurses want to evaluate the ventilator- Two weeks A u-chart
associated pneumonia (VAP) rate. So every 2 weeks
they record the total number of pneumonia episodes
and the total number of vent days.

13. Each week patient satisfaction scores for three Week V XmR
units are compiled and an average is calculated for
the three units.

14. The finance department tracks the total number of A request V XmR
business days it takes to process a vendor’s request for payment
for payment. Process time starts when the request
for payment is received in the finance department
and ends when the payment is sent (electronically or
posted in the mail) to the vendor.

15. Every week each medication order is checked against Week A u-chart
five potential types of errors. The total number of
errors for the week is divided by the total number of
orders submitted that week.

15. You know the number of people who come to the Unknown* A Unknown*
ED complaining of chest pain and the number who
are actually diagnosed with an AMI or unstable
angina.

*NOTE: Item 16 is a trick question. A subgroup is not specified. Without a subgroup you cannot make a decision about the most appropriate chart. If
this description indicated that “You know the number of people who come to the emergency department EACH MONTH . . .” you would have a subgroup.
The chart of choice would then be a p-chart.

Benneyan, J., R. Lloyd, and P. Plsek. “Statistical Process Control

References
ASQ Statistics Division. Glossary and Tables for Statistical
Quality Control, 4th ed. Milwaukee: ASQ Press, 2005
Benneyan, J. “Design, Use and Performance of Statistical
Control Charts for Clinical Process Improvement.”
Unpublished paper, Northeastern University, ­September
2001. For access to this paper contact Professor James
C. Benneyan, Ph.D., MIME Department, 334 Snell
Engineering Center, Northeastern University, Boston
MA 02115; tel: 617-373-2975; fax: 617-373-2921;
e-mail: benneyan@coe.neu.edu.
as a Tool for Research and Health Care Improvement.”
Journal of Quality and Safety in Healthcare 12, no. 6
(December, 2003): 458–464.
Blalock, H. Social Statistics. New York: McGraw-Hill, 1960.
Blank, R. The SPC Troubleshooting Guide. New York: Quality
Resources, 1998.
Blumenthal, D. “Total Quality Management and Physicians’
Clinical Decisions.” Journal of the American Medical
Association 269 (1993): 2775–2778.
Carey, R. Improving Healthcare with Control Charts. ­Milwaukee:
Quality Press, 2003.
258 Chapter 9 Understanding Variation with Shewhart Charts
Carey, R., and R. Lloyd. Measuring Quality Improvement
in Healthcare: A Guide to Statistical Process Control
Applications. Milwaukee: Quality Press, 2001.
Deming, W. E. The New Economics, 2nd ed. Cambridge,
MA: Massachusetts Institute of Technology, Center for
Advanced Studies, 1994.
Deming. “Quality, Productivity and Competitive Position.”
Indianapolis, Indiana, August 11–14, 1992.
Duncan, A. J. Quality Control and Industrial Statistics, 5th ed.
Homewood, IL: Irwin Press, 1986.
Gonick, L., and W. Smith. The Cartoon Guide to Statistics.
New York: Harper Perennial, 1993.
Grant, E., and R. Leavenworth. Statistical Quality Control.
New York: McGraw-Hill, 1988.
Hare, L. “SPC: From Chaos to Wiping the Floor.” Quality
Progress (July 2003): 58–63.
Ishikawa, K. Guide to Quality Control. White Plains, NY:
Quality Resources, 1989.
Kume, H. Statistical Methods for Quality Improvement. Tokyo:
Association for Overseas Technical Scholarship, 1996.
Langley, G. L., R. Moen, K. M. Nolan, T. W. Nolan, C. L.
Norman, and L. P. Provost. The Improvement Guide: A
Practical Approach to Enhancing Organizational Perfor-
mance, 2nd ed. San Francisco, CA: Jossey-Bass, 2009.
Levine, D., and D. Stephan. Even You Can Learn Statistics:
A Guide for Everyone Who Has Been Afraid of Statistics.
Upper Saddle River, NJ: Pearson-Prentice Hall, 2005.
Lloyd, R., and D. Goldmann. “A Matter of Time.” Journal of the
American Medical Association 302, no. 8 (2009): 894–895.
Moen, R., T. Nolan, and L. Provost. Quality Improvement
Through Planned Experimentation, 3rd ed. New York:
McGraw-Hill, 2012.
Mohammed, M., P. Worthington, and W. Woodall. “Plotting
Basic Control Charts: Tutorial Notes for Healthcare
Practitioners.” Quality and Safety in Health Care 17
(2008): 137–145.
Montgomery, D. C. Introduction to Statistical Quality Control,
2nd ed. New York: John Wiley & Sons, 1991.
Nelson, L. “Interpreting Shewhart Average Control Charts.”
Journal of Quality Technology 17 (1985): 114–116.
Nelson, L. “A Control Chart for Parts-Per-Million Noncon-
forming Items.” Journal of Quality Technology 26,
no. 3 (1994): 239–240.
Perla, R., L. Provost, and S. Murray. “The run chart: a simple
analytical tool for learning from variation in healthcare
processes” British Medical Journal Quality and Safety in
Healthcare 20, no. 1 (2011): 46–51.
Provost, L P., and S. Murray. The Health Care Data Guide.
San Francisco: Jossey-Bass, 2011.
Pyzdek, T. Pyzdek’s Guide to SPC. Vol. 1: Fundamentals.
Milwaukee: Quality Press, 1990.
Shewhart, W. Economic Control of Quality of Manufactured
Product. New York: D. Van Nostrand, 1931. Reprint,
Milwaukee: Quality Press, 1980.
Shewhart, W. Statistical Method from the Viewpoint of Quality
Control. New York: Dover Publications, 1986.
Western Electric Company. Statistical Quality Control
Handbook. Indianapolis: AT&T Technologies,
Inc., 1985.
Wheeler, D. Understanding Variation: The Key to Managing
Chaos. Knoxville, TN: SPC Press, 1993.
Wheeler, D. Advanced Topics in Statistical Process Control.
Knoxville, TN: SPC Press, 1995.
Wheeler, D., and D. Chambers. Understanding Statistical
Process Control. Knoxville, TN: SPC Press, 1992.
Woodall, W. “The Use of Control Charts in Health-Care
and Public-Health Surveillance.” Journal of Quality
Technology 38, no. 2 (2006): 89–104.
CHAPTER 10
Applying Quality
Measurement Principles
T
his chapter provides case studies that
demonstrate the quality measurement
concepts and principles and tools discussed
in previous chapters. The intent is to show brief
practical applications of these ideas to daily work.
Some of the case studies are short and address is-
sues related to indicator identification, operational
definitions, stratification, and sampling. All of the
examples discussed are based on real-life situations
that I have encountered with improvement teams.
Teams that have given me permission to show their
actual data or tell their story are noted. In those
cases, when a team asked to remain anonymous,
however, I have adjusted the story and their data
slightly to honor their request for anonymity.
Each case study is divided into two sections.
First, the quality measurement challenge is
described in the Situation section. This is fol-
lowed by a Discussion section in which options
for remedying the situation are presented. In
some cases, the discussion section is fairly brief
(e.g., if the challenge is to clarify an operational
definition). In other cases (e.g., the Shewhart
control chart case studies), the discussion
section goes into a little more depth, because
the example involves the actual presentation,
analysis, and interpretation of data. Finally, note
that there is no particular order or grouping of
the case studies relative to their content or the
issues being explored.

CASE STUDY #1: Predicting a Cardiovascular Event1

Situation
The medical director of a 165-member medical group decides to evaluate how well his physicians are at
predicting if a cardiac catheterization is necessary. Within the ranks of those who make these predictions
are two groupings: (1) nuclear medicine–trained cardiologists and (2) nuclear medicine–trained
radiologists. Inappropriate cardiac catheterization not only places patients at unnecessary levels of risk
but also wastes valuable resources and physician time that could be directed to more seriously ill patients.

(continues)
© Michal Steflovic/Shutterstock

259
260 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

The total number of cases referred for possible cardiovascular (CV) intervention constitutes the
patient population. From this population, a subset of patients is actually referred for a thallium treadmill
stress test in nuclear medicine. Once the films are read, a certain number of patients are then identified
as being at high risk for coronary artery disease. These high-risk patients are recommended to the
cardiologist for a cardiac catheterization procedure. Once the catheterization procedure is completed a
determination is finally made as to whether the procedure was appropriate or not necessary.
The medical director is grappling with two questions. First, was the catheterization procedure
warranted? This is actually a rather straightforward question based on the data obtained during the
procedure related to the percentage of blockage discovered in the cardiac arteries. The second question,
however, is a more delicate one. Specifically, “What is the accuracy of the nuclear-trained cardiologist
compared to the nuclear-trained radiologist in predicting a cardiac catheterization that shows significant
(greater than 50% occlusion) coronary artery disease?” The literature is not definitive on this issue, but
the general belief is that the predictive percentage (i.e., true positives) should be somewhere between
70% and 90%. Again, this question can be answered objectively by reviewing the predictive percentages
of the cardiologists and the radiologist. But, here is the medical director’s real dilemma. There are seven
physicians who read the thallium stress tests and make recommendations to proceed or not proceed
with a cardiac catheterization. Six of these physicians are cardiologists and one is a radiologist. Historically,
the cardiologists were the only ones who read the thallium stress tests results and determined whether
a catheterization was appropriate. When the radiologist was added to the mix last year the group
dynamics began to change. Several of the cardiologists took issue with allowing the radiologist to make
decisions about cardiac issues. The tension in the group was increased when the most senior cardiologist
presented data from a 2-week period that he claimed “proved” that the radiologist had a “significantly”
lower predictive percentage than all of the cardiologists. He even had a p-value to “prove” significance
of the differences at the 0.05 level. This situation became even more complicated because all of the
cardiologists were male and had been with the medical group for years. The new radiologist was a
female and had been with the group for only a year and a half. You can see that this situation moved
very quickly beyond the data and was going to require not only analytic skills but also a delicate dose of
social psychology. The medical director felt that the data analysis prepared by the cardiologist was not
providing an accurate picture of the results. This is where I entered the story to shed some light on the
analytic side of things. I left the social psychology challenge and group dynamics to the medical director.
I asked the medical director whether he could obtain data for the entire time period in which
all seven physicians were employed by the medical group. When we next met he provided data for
the past 18 months. The data are shown in TABLE 10-1. During the year and a half period, the seven
physicians read a total of 505 stress test results (column 2 in Table 10-1), of which 303 (60%) were done
in an outpatient setting and the remaining 202 (40%) were performed in an inpatient setting.

Discussion
■■ When looking at an outcome that is either correct or not correct, what type of data have we
collected—attributes or variables?
Indicators that are based on only two possible outcomes (e.g., correct or not correct) form a
binomial distribution, which is a form of attributes data. In this present case, the measure of
interest is whether the CATH procedure was appropriate (yes or no?).
■■ Do we have equal or unequal subgroups?
The subgroups are of unequal size because each physician reviewed a different number of stress
test results during the 18 months (i.e., between 58 and 85, column 2 in Table 10-1).

(continues)
 Applying Quality Measurement Principles 261

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

TABLE 10-1 Cardiac CATH data by Physician

Column 1 Column 2 Column 3 Column 4 Column 5 Column 6

Physician Total Number of Percentage Number Percentage
Number Number Stress Test of Stress of CATHs of CATHs
of Stress Results Test Results Determined Initially
Test Initially Initially to Be Thought to
Results Thought to Thought to Appropriate Be “High
Read Be “High Be “High After the Risk” that
by Each Risk” and Risk” Rec- Procedure Were
Physician Recom- ommended Actually
mended for a CATH Found to Be
for a CATH (Col 3/2) Appropriate
By Each After
Physician the CATH
Procedure
(Col 5/3)
1 77 50 64.9 30 60.0

2 69 59 85.5 44 74.6

3 85 49 57.6 30 61.2

4 58 35 60.3 28 80.0

5 63 56 88.9 23 41.1

6 81 57 70.4 32 56.1

7 72 41 56.9 34 82.9

Total 505 347 68.7 221 63.7

■■ What is the preferred chart for these data?

The preferred chart for this type of data is the p-chart. The key outcome indicator, the positive
predictive percentage, is a binomial condition (i.e., the prediction that a CATH procedure was
warranted is either “yes, it was appropriate” or “no, it was not appropriate”). A p-chart requires a
numerator and a denominator and produces either a percentage or proportion. There are two
key percentages that can be considered in this case:
1. The percentage of stress test results initially thought to be high risk and recommended
for a CATH (column 4 in Table 10-1). This measure reflects what each physician “thought”

(continues)
262 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

the results of the stress test were revealing. If the physician thought there was sufficient
blockage in one or more of the patient’s arteries then the patient was recommended for a
CATH procedure.
2. The second percentage is the one the medical director was most interested in. This is what they
referred to as the “predictive percentage.” It reflects how well each physician was able to predict
a true positive (i.e., the CATH was deemed appropriate after the procedure was completed).
Specifically, this measure, shown in column 6 of Table 10-1, is the percentage of CATHs initially
thought to be high risk that were actually found to be appropriate after the CATH procedure.
The numerator for this measure is the number of catheterizations that were determined to
be appropriate (i.e., the cardiologist confirmed greater than 50% stenosis). These numbers are
shown in column 5 of Table 10-1. The denominator is the number of catheterizations performed
(column 3 in Table 10-1). The objective is to be as close to 100% as possible, which means that
every catheterization that was done was required and determined to be appropriate by the
cardiologist after the procedure was completed. Every time a catheterization is performed and
it is discovered after the procedure that the patient did not need the catheterization, this is
considered to be an incorrect reading of the thallium stress test and therefore a wrong decision
(i.e., a defective decision), which again is the appropriate concept to be placed on a p-chart.
■■ Why are the dots on the p-charts not connected by a line?
The dots are not connected because the seven physicians do not represent a process
displayed over time. Most control chart examples show some unit of time (e.g., day, month,
or patients in chronological order) displayed along the horizontal axis. This is the traditional
format for a control chart. In this example, however, the horizontal axis displays the individual
physicians numbered 1–7. This p-chart is known, therefore, as a “comparative chart” because
it is comparing each physician’s percentage of correctly interpreted films at a fixed point in
time. Prior to making this comparative chart, a p-chart for each physician’s performance over
the last 18 months should be prepared. These physician-specific charts (not shown) should
all reflect common cause variation before proceeding to make the aggregated comparative
charts shown in FIGURES 10-1 and 10-2. If one or more of the physician-specific charts
contained special causes, then the comparative p-chart should not be constructed. The
reasoning behind this decision is straightforward: if the constituent parts (i.e., the individual
physicians) reflect special cause variation, then the aggregated comparative chart will
similarly reflect an unstable process. For additional applications of the comparative chart see
Carey (2003).
■■ Is there any value in knowing the numbers of films done on an outpatient and an inpatient basis?
How could these data be used to further our understanding of the accuracy of the reading process?
The data in Table 10-1 show the total number of stress tests read by each physician (column 2).
Once the overall pattern has been analyzed, the next step would be to see if there are any notable
differences between the positive predictive percentages for inpatient and outpatient settings.
Stopping any analysis after analyzing only the totals can be misleading. Stratification enables you
to drill down to the constituent parts of a process. In this case, there could be differences between
the inpatient and outpatient settings because of differences in the severity of the patients or
possibly the amount of time the physicians spend on evaluating inpatient and outpatient stress
tests. Various theories would have to be explored with the subject matter experts.
■■ What do you conclude about the seven physicians who read the films? Do they form a common cause
system or are there special causes present?

(continues)
 Applying Quality Measurement Principles 263

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

Percent of stress test results initially thought to be of “high risk” and

recommended for a CATH (p-chart)
100%
90%
UCL
80%
CL = 68.7%
70%
60%
Percent

LCL
50%
40%
30%
Special cause
20% (3 sigma violation)
10%
0%
1 2 3 4 5 6 7
Physician Number

FIGURE 10-1 Percentage of stress test results initially thought to be of “high risk” and recommended
for a CATH

Percent of CATHs initially thought to be of “high risk” that were actually found to
be appropriate after the CATH proceure (p-chart)
100%
90%
80% UCL

70%
CL = 63.7%
60%
Percent

50%
40% LCL

30%
20% Reference range
10% (70%-90%) Special cause (3 sigma violation)
0%
1 2 3 4 5 6 7
Physician Number

FIGURE 10-2 Percentage of CATHs initially thought to be of “high risk” that were actually found to be
appropriate after the CATH procedure (p-chart)

Figures 10-1 and 10-2 provide the p-charts for the performance of the seven physicians. We see
in Figure 10-1 that physician 5 is a special cause relative to the other six physicians. This physician
predicted that 88.9% of the thallium stress test results required a cardiac CATH. All of the other

(continues)
264 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

physicians’ initial conclusions fell between the upper control limit (UCL) and the lower control limit
(LCL). Note that physician 2’s value of 85.5% fell exactly on the UCL but did not exceed it. This is
not a special cause. So, physician 5 had the highest percentage of recommended CATHs. Now the
question becomes, how did each physician’s positive predictive percentage compare to what they
initially concluded about the necessity of a CATH? The answer to this question can be found in
Figure 10-2. This p-chart shows us the percentage of CATHs initially thought to be high risk that were
actually found to be appropriate after the CATH procedure. The positive predictive percentage for
physician 5 is below the LCL signaling a special cause. This physician initially thought that 88.9% of
the stress tests he or she reviewed required a CATH yet only 41.1% of them actually were appropriate
for a CATH after the procedure was completed. The remaining six physicians’ percentages all fell
between their UCL and LCLs. Note that because this is a p-chart, each dot (which represents
a physician) has its own control limits. The width of these limits is based on each physician’s
denominator (i.e., the total number of stress tests results reviewed that were initially thought to
be appropriate for a catheterization, column 3 in Table 10-1). The more stress test results read, the
tighter the control limits, whereas the smaller the denominator, the wider the limits. Physician 4, for
example, had the widest limits because he or she had only 35 films as a denominator. On the other
hand, physician 2 had the largest denominator with 59. Even though physician 5 falls below the LCL,
this does not mean that this physician’s performance is necessarily “bad” and that of the other six
physicians is “good.” Good and bad are not appropriate concepts for interpreting Shewhart charts.
The more appropriate terms are stable and predictable (common cause variation) versus unstable
and unpredictable (special cause variation). With one physician falling below the LCL, therefore, the
primary interpretation of this chart is that when it comes to the percentage of positive predictive
outcomes, this group of physicians does not exhibit common cause variation. Stated otherwise they
do not perform as a system. The secondary interpretation is that one physician’s performance (i.e.,
physician 5) is notably different from the rest (i.e., this physician is a special cause). Why he or she
is different or whether this difference is clinically meaningful remains to be seen. Remember that
control charts do not tell you why a process is unstable or not performing in an acceptable manner.
The control chart only tells you (1) whether the process is stable and predictable and (2) what
impact an intervention had on the performance of the process.
The two p-charts for these measures are shown in Figures 10-1 and 10-2
■■ Of what value is the reference range for positive predictive percentage accuracy (70–90%)?
The reference range provides an opportunity to see how well this process is performing
against an established norm or comparative reference data. These numbers also can be viewed
as the lower specification limit (LSL = 70%) and upper specification limit (USL = 90%) for
the process. In this case, if the UCL and LCL coincide with the USL and LSL, then the process
would be performing in line with the expectations for the process. When the control limits are
wider than the specification limits (SLs), then the process is not performing in line with the
expectations for the process. Conversely, if the process control limits are less than the SLs, then
the process is functioning better than expected. As you can see from Figure 10-2 , only three
physicians (2, 4, and 7) are performing within the reference range of 70–90%. The remaining
four are performing below the LSL of 70%, and one of these has fallen below the LCL. The
challenge in this case is deciding whether the SLs (i.e., 70–90%) are clinically appropriate for
the population of patients being served by these physicians. For example, this range allows for
10% error because 90% is the USL. Why would we not expect this process to function in the
range of 70–100%? Why was 70% selected as the minimum acceptable standard? And why

(continues)
 Applying Quality Measurement Principles 265

CASE STUDY #1: Predicting a Cardiovascular Event (continued)

are these both whole numbers that are divisible by 5? This seems too convenient. Why not a
spread of 73.4% to 91.9%? As I mentioned previously targets and goals are frequently set not in
a statistical manner but because people have a tendency to think in terms of whole numbers
that are divisible by 5. When a team starts to discuss these questions, then they have opened
the door for a dialogue as to whether the process is capable of achieving the targets and goals
being set relative to the current performance of the system. The Shewhart chart does not
necessarily answer all the questions, but it should certainly serve as a foundation for raising
questions. One final note to this story. Physician 5 was the senior cardiologist who claimed that
the female radiologist (physician 7) had a “significantly” lower predictive percentage than all
of the cardiologists. As can be seen from Figure 10-2, the radiologist actually has the highest
positive predictive percentage. When I presented the charts to the medical director he was
pleased with the analysis and the results. I told him that the charts were the easiest part. I said
that he now had the difficult challenge of explaining these results to the group of physicians.
He looked at me and started to smile. “Yes” he said, “you and I do have a challenging meeting
awaiting us.”

CASE STUDY #2: Sampling Central Line Infections

Situation
The director of infection control is interested in getting a clear understanding of the central line
infections occurring within her hospital. So each day she and her staff review all the charts that
have notes indicating that the patient has a central line. They look for any sign of infection noted in
the charts. If they detect any potential characteristics of an infection they request pathology to test
cultures from any suspicious lines to see if an infection is actually present. Each day a patient has a
central line in place it is counted as 1 central line day. Because keeping track of central line days is a
very time-consuming task, the staff have been considering doing a sample of the central lines but are
concerned that they might miss something.
After attending a national infection control conference, the director of infection control tells her
staff what she learned about sampling at one of the conference’s workshops. The presenter told the
group that the easiest method of sampling for central line infections or any infection for that matter is
to take a sample of 2 random days during the month and review all the central lines on these 2 days.
Then she said the presenter told them to take the total number of central line infections found in these
2 days and multiply this number by 15 to get the estimated number of central line infections for the
month (the reasoning being that 2 days × 15 = 30 days in the month). The total number of central line
days could be estimated in a similar manner. Does this seem like a reasonable approach to sampling?
Will such an approach produce data that properly represents the central line infection process? What
challenges do you see with this approach to sampling?

(continues)
266 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #2: Sampling Central Line Infections (continued)

Discussion
Although I do not disagree with the idea of possibly applying sampling to central line indicators,
several points need to be considered prior to devising a sampling strategy:
■■ First, I would want to have a discussion with the director of infection control to discover the
number of central lines being inserted and managed each day or week. By knowing the actual
volume we are dealing with we can decide if sampling is even necessary. For example, if only
five central lines are inserted each week, which would not be even one each day, then I would
recommend taking all occurrences and not engaging in sampling. When volume increases and
the clinical staff is inserting 10 or more new central lines a day and attempting to manage over
50 each week, then we can have a discussion about a sampling strategy.
■■ Related to the issue of volume is the subgroup. When data are pulled will they be organized as a
subgroup by day, week, every 2 weeks, or month?
■■ Another discussion should occur about the primary concern with central lines. This could have
an impact on a sampling strategy. For example, is the concern with the insertion of a central
line, the maintenance of the central line, or both issues? If the director of infection control wants
to stratify the entire central line population into insertion and maintenance categories then
the volume of patients in each category would have to be reviewed and decisions made as to
whether a total enumeration of all cases is considered or a sample.2
■■ Finally, if sampling is determined to be a viable path forward then a discussion about the
advantages and disadvantages of various sampling approaches needs to occur outlined
in Chapter 4. What are the sampling options for studying central line infections? The first
thing I would do is not to follow the recommendation of the individual who presented at
the infection control conference. Selecting 2 random days out of the month, reviewing all
the central line cases in the hospital on these days, and noting any with infections then
multiplying this number by 15 to get a monthly estimate of the number of central line
infections and central line days will produce misleading numbers. I believe the director of
infection control would be better off just picking a simple random sample of 5–10 patients
(possibly stratified by new central lines inserted versus those that are in a maintenance status)
each day or each week (depending on the volume of central lines) and then applying the
review criteria for central line infections to this sample of patients. Another approach could
be to select a random day each week and review all the central line cases on these selected
days. If the hospital had a large volume of central line patients it would then be feasible to
draw a random sample on the randomly selected day of the week. All of this could be figured
out ahead of time so that a schedule could be produced to minimize the burden on the staff
pulling the sample of patients and conducting the reviews. Now some readers might say that
this approach sounds similar to what the speaker at the infection control conference was
suggesting. It is in the sense that selecting a random set of days is not a bad idea. The issue I
have with the speaker’s suggestion is that selecting only 2 random days in a month is probably
too few days to get an accurate picture of the central line process. Selecting a random day
each week provides a more representative sampling approach. What I totally disagree with,
however, is the speaker’s suggestions to take the central line infections on the 2 randomly
selected days and merely multiplying the results for these 2 days by 15 to get an estimate of
what the month’s infection might be. Finally, given the seriousness of central line infections, it
might be advisable not to apply sampling to this indicator at all.
 Applying Quality Measurement Principles 267

CASE STUDY #3: Sampling Medicare Insurance Audits3

Situation
You are the director of performance improvement at a medium-sized hospital. You receive a call from
the manager of admissions and registration asking how many Medicare claims her staff needs to
pull in order to obtain a “significant” sample that “proves they are 95% accurate.” In a single day, there
are around 150 new Medicare claims that are eligible for review. This includes both inpatient and
outpatient activity. You think this seems like one of those questions that you really cannot answer with
a single number. You also feel there is a need to clarify a few questions about the ultimate objective
behind this request. What would you recommend to the woman in registration?

Discussion
The first thing the manager of performance improvement should do is to find out what the ultimate
purpose of this request is. Her initial request has some confusing and conflicting aspects. She said she
wants a “significant” sample that “proves they are 95% accurate.” People frequently confuse accuracy,
reliability, and significance. These are three very different concepts. What is the operational definition
the manager is applying to each of these concepts? For example, I am not sure what the manager
means when she says she wants a “significant” sample. Significance is not a relevant term when
discussing sampling options. The key concept for sampling is representativeness (i.e., is the sample
representative of the population?). Significance, at least from a statistical point of view, is used when
conducting tests of significance to determine if two numbers are statistically different from each
other. This is when the p-value comes into play and we talk about significance at the 0.05 or 0.01
level of significance. Sometimes people will use the term significance as a synonym for meaningful or
important or even accurate. Another aspect of sampling that sometimes comes into play is the notion
of “power.” Specifically researchers will conduct a power analysis to determine the sample size required
to detect an effect of a given size with a given degree of confidence. Statistical power is affected chiefly
by the size of the effect of an intervention on the test subject and the size of the sample used to detect
it. Large samples offer greater test sensitivity than small samples. But it is very unlikely that the manager
was thinking of these statistical references when she was using the word “significance.” Furthermore,
she has also stated that she wants to “prove that they are 95% accurate.” Accurate in terms of what?
Are they confusing accuracy with completeness? A Medicare claim could be 100% complete but also
100% inaccurate. Why 95% accurate? Shouldn’t all claims be 100% accurate? She may be thinking
something to the effect that if she pulled repeated samples of Medicare claims, these claims would
be “representative” of the entire population 95% of the time. If this is the case then accuracy is not the
concept of interest and representativeness is. I do not mean to confuse you with these various terms.
My point is that people frequently use terms that have rather precise definitions statistically but they
are referring to the terms in a broad and general sense. The challenge for the improvement specialist is
to explore what the requester means by the terms. Once the terms and objectives for collecting data
have been clearly defined then it is time to discuss the specifics of the data collection milestone.
If we look at just the numbers, however, we can make some recommendations on a sampling
strategy. Given that about 150 new Medicare claims are processed each day, there is clear justification
to consider a sampling approach when doing the claims reviews to determine “accuracy.” Here are
some key considerations when discussing sampling options:
■■ Depending on the discipline in which you are trained there are slightly different positions on
how much data you need before a stable distribution starts to form. Generally speaking, a
distribution will start to form and be reasonably stable with about 25–30 data points. By the

(continues)
268 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #3: Sampling Medicare Insurance Audits (continued)

time you get 50 data points most disciplines will agree that there are enough data to form a
reasonably stable distribution.
■■ The manager was correct in asking what is practical. Data collection requires time and effort. So it
is important to make sure that the data collection procedures are not creating a burden on staff.
■■ When it comes to answering the question “how much data do I need to collect?” there is no
simple number as an answer. Sampling should be structured around a very simple guideline—
pull as much as you must and as little as you dare. In the current situation with about 150 new
Medicare claims a day, it would be reasonable to first of all have a discussion about stratifying
these claims into a few relevant categories. Are all the charts the same? No, they are not all the
same. So, let us look at how to divide them into more relevant buckets (e.g., medical versus
surgical, male versus female, age splits, complications, readmissions versus first-time admits,
or total costs). With around 150 new occurrences every day there are clearly enough data to
stratify into relevant categories. The staff will know best what these categories are. When the
stratification categories are selected and applied to the data, the volume within each category
should be evaluated to determine the sample size that is feasible within each category.
■■ Once the stratification issues are addressed a random sample of the Medicare claims could be pulled
each day within each category. A random sample of three to five claims from each stratification
category each day would be sufficient as a starting point. As the samples are drawn, they should be
regularly reviewed and studied to see how well they match the characteristics of the stratification
category they are from and the total population. Sampling decisions are not a one and done
situation. Sampling plans need to be evaluated continuously throughout the life of an improvement
project to make sure the sample data are in fact representative of the total population.

CASE STUDY #4: Tracking Patient Falls

Situation
Eight months ago your facility introduced a new program to reduce the number of inpatient falls.
The key question being asked by the Quality Committee of the board is: “Did this new program have
the desired impact?” To answer this question, you decide to take advantage of the knowledge and
experience of the team that created the new program. They have been working on improving this
process since last year and have (1) developed a standardized operational definition of a patient fall
(which was probably a major accomplishment in and of itself ), (2) developed a falls assessment tool,
(3) established and implemented an ongoing data collection plan, (4) established a baseline, and
(4) prepared Shewhart charts to determine whether a change has occurred.

Discussion
Before deciding whether the falls prevention program made a difference we need to decide which
Shewhart chart is most appropriate for the falls data. The key points in selecting a chart for tracking
inpatient falls are as follows:
■■ Patient falls are not something that is desired. Therefore, a patient fall can be classified as
attributes data and viewed as being either a defective or a defect. If falls are viewed as a

(continues)
 Applying Quality Measurement Principles 269

CASE STUDY #4: Tracking Patient Falls (continued)

defective then the indicator would be the percentage of patients who fell once or more while
in the hospital. If a patient falls at any time during a hospital stay it is considered an undesirable
outcome (i.e., a defective). But if we take this approach we do not care about how many times
a patient falls because a percentage is a binomial opportunity (i.e., the patient fell, yes or no,
and we do not care if they fell multiple times). If we take this approach a p-chart would be the
chart of choice. Because the same patient can fall more than once, however, most improvement
teams want to know the total number of falls, including duplicates for the same patient. When
we take this approach each fall is considered to be a defect that can occur more than once for
the unit of analysis, which in this case is a patient. There are two options for analyzing defects,
the c-chart, which tracks the number of falls, and the u-chart, which tracks the falls rate (i.e., so
many falls per 1,000 patient days). The reader should refer to the Shewhart chart decision tree in
Chapter 9 to review the various charts and the conditions for selecting each one.
■■ The chart selected by the improvement team in this case study to track inpatient falls was the
c-chart. They did this because they decided that there was basically an equal opportunity for a
patient to fall any day of the week. If they had decided that there was not an equal opportunity
for a fall each day of the week then the chart of choice would have been the u-chart. Again, refer
to Chapter 9 for more details on the differences between the c- and u-charts.
■■ Each data point on the charts shown in FIGURES 10-3 through 10-6 represents the total number
of inpatient falls occurring within the facility each month. Figure 10-3 presents the baseline data
for months 1–24 before the components of the falls prevention program were tested. Notice that
the falls process reflected common cause variation with an average of about 53 falls a month. This
means that the falls process is stable and therefore predictable. The control limits indicate that if
nothing was done to try to reduce the number of falls they could predict falls to go as high as 75 a
month or as low as 31. The UCL, LCL, and the mean or centerline on the chart (CL), therefore, define
what this process is capable of producing given current operating conditions. The primary question
for the team and the Quality Committee of the board to discuss is a simple one, “Given current
performance of the falls process, as defined by the baseline ULC, LCL, and CL, is it acceptable?”
80
Baseline (months 1-24)
UCL UCL = 74.7
70

60
Number of Falls

CL = 52.9
50

40

30 LCL LCL = 31.1

20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Month

FIGURE 10-3 Number of falls by month, baseline months 1–24 (c-chart)

(continues)
270 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #4: Tracking Patient Falls (continued)

80
Baseline (months 1-24) UCL = 74.7
UCL
70

60
CL = 52.9
Number of Falls

50

40

30 LCL
LCL = 31.1
20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Month

FIGURE 10-4 Number of falls by month, baseline months 1–24 with “frozen” control limits and mean
extended (c-chart)

After the falls

Baseline (months 1-24) prevention program
80
UCL = 74.7
UCL
70
CL = 52.9
60
Number of Falls

50

40

30 LCL
LCL = 31.1
20 Note: The rules for detecting a special cause are
applied to the new data (months 25-32) with the A special cause: 8 data points below
10 “frozen” limits and CL as reference points. the CL indicating a shift in the process
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Month

FIGURE 10-5 Number of falls by month, baseline months 1–24 with “frozen” control limits, mean
extended and the number of falls after the falls prevention program, months 25–32 (c-chart)
■■ Everyone agreed that having about 53 falls a month is unacceptable. So, the falls prevention
team initiated its work. Because the baseline reflects only common cause variation they “froze”
the UCL, LCL, and CL of the baseline chart and extended them into the future as reference lines
against which the new data will be plotted and evaluate. They will use the five rules for special
causes to determine if a special cause is detected and their falls prevention program has been
able to reduce the number of falls. Figure 10-4 shows how the baseline parameters have been
extended into the future as reference lines. (See Chapter 9 for more on special cause rules.)

(continues)
 Applying Quality Measurement Principles 271

CASE STUDY #4: Tracking Patient Falls (continued)

100

90

80 After the falls

Baseline (months 1-24)
prevention
70 UCL = 74.7 program
UCL = 64.1
Number of falls

60

50 CL = 52.9
CL = 44.1
40

30
LCL = 31.1 LCL = 24.2
20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Month

FIGURE 10-6 Number of falls by month, baseline months 1–24 compared to the new control limits and
CL for the number of falls after the falls prevention program, months 25–32 (c-chart)

■■ Figure 10-5 shows the new data (months 25–32) compared to the baseline UCL, LCL, and CL.
The team observed that eight data points were below the CL, which is a signal that there has
been a shift in the process in the desired direction (i.e., fewer falls each month).
■■ The final chart (Figure 10-6) demonstrates how to phase the data into two separate charts.
The left side of the c-chart shows the baseline period (months 1–24) with its respective UCL,
LCL, and CL. The right side of Figure 10-6 shows the new falls process. Not only has the process
shifted to a lower level with a new average of roughly 44 falls a month down from 53, but
the UCL and LCL of the new process on the right side of the chart are a little closer together
indicating less variation in the process.
The information presented in Figure 10-6 does not tell the team whether or not it should continue
its efforts to reduce the number of inpatient falls. It merely tells the team that the initial efforts have
been successful and that the current process produces fewer falls each month than the baseline
process. Should the team decide to introduce another intervention to further reduce the number of
falls? The answer to this question lies with the team and the target or goal they have established as their
aim. As owners of the process, they need to decide if the process is capable of further improvement
and if the resources are available to support this work. If the facility is part of a system, for example,
they could obtain comparative reference data (norms) and see how their performance compares to
that of the other facilities in the system. If such data do exist, then there is an opportunity for internal
benchmarking. The team would also be well advised to continue monitoring the number of patient falls
over the coming months, even if they decided not to introduce any further improvement efforts. The
reason is that the team should be responsible for the performance of the process and confident that
the observed improvement in months 25–32 is being sustained during future months. The control chart
provides the conceptual and statistical foundation for doing this.
272 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #5: Pressure Ulcer Prevention

Situation
You are the director of care management at a 275-bed hospital and have become curious about
the incidence and prevalence of pressure ulcers (PUs) within the hospital. Pressure ulcers seem
to be one of those topics that everyone takes for granted. It is not unusual, for example, to
hear staff say, “They will occur and there is not a lot we can do about it.” You believe, however,
that there are some things that can be done to reduce the occurrence of PUs, improve clinical
outcomes, and save the hospital money. You begin by preparing a brief summary of the two
types of PUs. The first is nosocomial or hospital-acquired PUs (incidence). The second is known as
community-acquired PUs (prevalence). This summary is distributed to the inpatient nursing staff,
the physical therapists, and the staff at your skilled nursing facility to increase awareness and start
a dialogue on this topic. Next you develop a measurement plan. The primary outcome measure
is the percentage of patients that develop nosocomial PUs during their admission (the incidence
indicator).
The improvement team created to work on reducing hospital-acquired PUs decides that it will
introduce a new type of mattress with a pressure-relieving surface and a downward-slanted heel slope
as their first intervention. They also begin working on developing a new protocol for assessing patients
and preventing PUs before they develop. Based upon a national comparative norm of 7%, you set off
to establish a baseline for the incidence of nosocomial PUs. After establishing a baseline, they began
to test the two initial improvement strategies they developed and see if they made a difference in the
proportion of hospital-acquired PUs. FIGURES 10-7 through 10-9 show the baseline and the results after
the interventions were introduced. What conclusions do you make about the effectiveness of these
two interventions? Did one work better than the other?

0.7
Proportion of nosocomial pressure ulcers

0.6

0.5
UCL = 0.449
0.4

0.3
CL = 0.241
0.2

0.1

LCL = 0.033
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Week

FIGURE 10-7 Baseline proportion of patients with nosocomial PUs (p-chart)

(continues)
 Applying Quality Measurement Principles 273

CASE STUDY #5: Pressure Ulcer Prevention (continued)

Baseline
0.7
Proportion of nosocomial pressure ulcers

0.6
Changes in the
process began here
0.5
UCL = 0.471

0.4 A special cause – 8 or more data

points below the centerline
0.3
CL = 0.241
0.2

0.1
Baseline (weeks 1-17)
LCL = 0.010
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Week

FIGURE 10-8 Proportion of nosocomial PUs comparing the baseline period to the new process with the
average control limits and the CL (average) based on weeks 1–17 (p-chart)

Baseline (weeks 1-17) Results after changes

0.7
Proportion of nosocomial pressure ulcers

0.6

0.5

0.4
UCL = 0.373

0.3

0.2
CL = 0.170

0.1
National comparative norm of .07

LCL = 0.000
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Week

FIGURE 10-9 Proportion of nosocomial PUs comparing the baseline (weeks 1–17) period to the new
process (weeks 18–28) (p-chart)

(continues)
274 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #5: Pressure Ulcer Prevention (continued)

Discussion
Figure 10-7 shows the baseline data for the incidence of nosocomial PUs. Note that this is presented
as a proportion, not as a percentage.4 The baseline p-chart reflects common cause variation with a
process average proportion of 0.24 or 24%. The process could operate as high as the UCL 0.449 (45%)
or as low as 0.033 (3.3%). Because the process exhibits common cause variation, we can expect the
process to continue to perform within these parameters until (1) an unexpected special cause enters
the picture and makes the process unstable, or (2) the team plans an intervention that they believe will
move the process to a new level of performance.
The improvement team decided to choose the second option (i.e., the improvement route).
Now the question is whether the interventions they selected had a positive impact on nosocomial
PU prevention. Figure 10-8 provides the answer to this question. This chart displays the baseline data
shown in Figure 10-7 (i.e., weeks 1–17) on the left side of the chart and the proportion of nosocomial
PUs after the team introduced the interventions on the right side of the chart (weeks 18–28). The average
UCL and LCL as well as the average CL are all computed on the baseline period; they have been “frozen”
and then extended across the period when the changes were being tested (note that the averages for
the control limits and the CL in Figures 10-7 and 10-8 are the same). It is quite clear that the process has
shifted to a new level of performance. This is confirmed by having eight or more data points below the
CL from the baseline period.
Figure 10-9 displays the final step in presenting the data for this case study. Notice that this
chart has two sets of control limits. The left side of the chart is the same baseline we observed in the
previous charts. The right side of the chart (labeled Results after changes) reveals what happened to
the process after the interventions were tested. Not only did the control limits narrow but we also
see that the centerline or process average has dropped to 0.17 or 17%. Both sides of the chart display
common cause variation.
The team has just cause to celebrate its success. But when you compare the new process average
to the national comparative norm of 7%, you realize that there is still considerable work to be done. The
next steps would be to continue to educate staff on the new protocol for detecting and preventing
PUs. Because it is easier to change mattresses than it is to change the behaviors of individuals, it
might be that the impact of the protocol training will take longer to have an influence on the process
average. The control chart methodology provides, however, a very easy and simple way to track the
interventions as the team progresses.
The final question to be addressed is, Did one intervention work better than the other? The team
simultaneously tested the introduction of new mattresses as well as the use of a new protocol for
assessing patients for PUs and a positive response was observed (i.e., fewer PUs). But did one of these
interventions work better than the other? Was the effect seen in Figure 10-9 a result of the mattresses,
the protocol, or a combination of both? Because the team introduced both changes at the same time
it is difficult to tease apart the individual effect of each intervention and the combined interaction of
the two interventions with a single control chart. In order to decide whether one intervention had a
larger effect on reducing PUs than the other or whether it was a combination of the two factors being
introduced simultaneously, the team would need to setup and run a planned experiment (Moen,
Nolan, & Provost, 2012). A planned experiment would allow the team to determine statistically whether
one of the factors (e.g., the mattresses) had a larger effect on the reduction of PUs than the protocol.
The planned experiment would also allow the team to determine the combined effect of introducing
the mattresses and the protocol. At the Institute for Healthcare Improvement (IHI), we teach planned
experimentation (PE) in our Improvement Advisor Professional Development Program. It is not a topic I

(continues)
 Applying Quality Measurement Principles 275

CASE STUDY #5: Pressure Ulcer Prevention (continued)

cover in this text but if you expect to be a person with deep knowledge of the science of improvement
(SOI), building knowledge about the application of PE is essential.
A final technical point related to the use of a p-chart needs to be mentioned. Note that the right
side of Figure 10-9 (weeks 18–28) has no LCL. This is because when you are using a p-chart many
software programs will not allow the UCL or LCL to exceed the typical minimum (0.0) or maximum
(1.0 or 100%) values for a proportion or percentage. Remember that the UCL and LCL are calculated
to be symmetrical about the mean (CL). As the data approach either the minimum or maximum
theoretical limits for a proportion or percentage the calculated limits may exceed these boundaries.
So, to prohibit having a control limits that go either above 100% or below 0% the software will cap the
limits. You may have to explain this as you present your data.

CASE STUDY #6: Evaluating Staffing Effectiveness

Situation
A national regulatory and oversight body has recently raised concerns over the growing number
of nurse vacancies and the potential impact these vacancies are having on patient care outcomes.
There have even been recent articles in the paper and stories on the evening news that nurse
vacancies are leading to “increases in patient harm.” Although many of the stories and opinions
about poor quality are anecdotal there was a situation at one hospital last week where a reporter
claimed she uncovered evidence that “proved” that nurse vacancies lead to an increasing number of
medication errors.
Because of the growing concern over this issue the national regulatory body has proposed that
there be new staffing effectiveness requirements for all hospitals and nursing homes. Although still
be developed this new requirement is designed to test relationships between two sets of indicators:
human resources indicators and clinical/service indicators. The basic theories driving development of
this new requirement are that changes in the human resources indicators, especially nursing levels,
are negatively correlated with clinical outcomes, patient safety, and patient satisfaction. While these
theories have not been consistently substantiated in research studies they have gained increased
popularity with the public, the press and political leaders. The specifics and requirements related to
assessing staffing effectiveness have not been finalized but your organization has created a team
to be proactive about this issue and it has started to study your own data to see if any patterns or
correlations exist. What steps would you advise the team to take in order to determine whether there
is a relationship between staffing effectiveness and clinical outcomes?

Discussion
The first thing your team would most likely do is to gain as much information about the regulatory
body’s assessment strategy and plan as possible. Unfortunately the regulatory body has not specified
exactly how healthcare providers will have to demonstrate clinical effectiveness and quality outcomes.
In their preliminary news releases, however, they have indicated that each hospital or nursing home
will be expected to test the relationships between various human resources indicators and clinical
outcomes. They have also released a preliminary set of indicators they are planning to use to evaluate

(continues)
276 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

human resource effectiveness and clinical/service outcomes. The indicators by category they are
considering are as follows:
■■ Human resource indicators
•• Overtime hours
•• Staff vacancy rates
•• Staff satisfaction scores
•• Staff turnover rates
•• Nursing hours per patient day
•• Staff injuries
•• On-call per diem use
•• Sick time usage rates
•• Resources used to bring in agency nurses
■■ Clinical/service indicators
•• Patient complaints
•• Family/caregiver complaints
•• Patient satisfaction scores
•• Patient falls
•• Adverse drug events
•• Patient injuries
•• Skin breakdowns
•• Pneumonia rates
•• Urinary tract infections
•• Shock/cardiac arrest
•• Length of stay
One of the main challenges with these two lists is that very few are actually indicators. Some
are still vague concepts that require clear operational definitions. (See Chapter 4.) For example, how
will staff and patient satisfaction be measured? Will individual questions be used or a composite
score? (See Chapter 3.) What is the operational definition of an “adverse drug event”? Even indicators
that sound somewhat specific like “staff turnover rates” or “length of stay” require more detailed
operational definitions. A spokesperson for the regulatory body has indicated that they do not plan
to define precisely every concept or indicator. Each hospital, they have said, must decide which
indicators they wish to evaluate and then decide how they will define terms such as vacancy,
turnover, patient injuries, skin breakdowns, or staff satisfaction. Once the providers decide how they
will define the various indicators, they then must select one human resource indicator and one
clinical/service indicator and test the relationship between these two indicators. As you can imagine,
structuring an assessment program by having the organizations select the indicators upon which
they will be judged and then define these indicators creates a variety of challenges, especially if
the regulatory body wishes to make any comparisons between organizations. But, in light of these
challenges your team presses onward.
The second thing the team should discuss is what it means to “test the relationship between one
human resources indicator and one clinical outcome indicator.” Whenever two measures are matched
and the word “relationship” is used, the objective is to determine if there is a causal relationship
between the two measures. In this case, the human resources indicator (e.g., number of nursing
vacancies) is considered to be the cause (i.e., the independent variable), and the patient safety indicator

(continues)
 Applying Quality Measurement Principles 277

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

(e.g., the number of patient falls) is viewed as the effect or dependent variable. Stated differently, if you
decided to compare the number of nurse vacancies with the number of patient falls, one possible
theory that the regulatory body would expect you to explore is as follows: As the number of nursing
vacancies increases, we expect patient falls also to increase.
Depending on the indicators selected, the expected relationship between the two will have one
of the following outcomes:
■■ There is a positive relationship between the two indicators (i.e., as one indicator increases, the
other indicator also increases).
■■ There is a negative relationship between the two indicators (i.e., as one indicator increases, the
other indicator decreases).
■■ There is no relationship between the two indicators.
Once you have selected the indicators to be assessed, the following steps should be taken to test
the relationships between the two indicators:

Step 1: Organize the Data

In order to determine whether there is a relationship between the two indicators, you need to have
an adequate amount of data. Ideally, the indicators would be based, at a minimum, on monthly
data. Some indicators may even be available on a weekly basis. The least desirable time frame for any
indicator is quarterly. Quarterly data will produce only four data points for the entire year, which is too
little data to successfully test the relationship between the two indicators. Generally speaking, if you
are going to test the relationship between two indicators you should have a minimum of 15–20 data
points before testing the relationship between the two indicators. Also note that it is best practice to
have both indicators based in the same time frame. For example, you would not want to have patient
falls by week and nurse vacancies by month.

Step 2: Prepare Control Charts

A Shewhart chart should be prepared for each indicator. If an indicator exhibits special cause
variation, then the special causes need to be addressed before the indicator is matched up with the
second indicator. If both of the selected indicators exhibit special causes, then the conclusion about
the relationship between the two indicators will be suspect because both indicators are unstable
and therefore unpredictable. As a result you have little understanding of what the indicator is
capable of producing.
If special causes are present, the first thing to do is to review the data collection procedures.
Inconsistent data collection procedures can produce special causes that are a result of:
■■ Operational definitions that keep changing
■■ Changes in the patient population (e.g., a more severely ill patient population emerged midway
through the year)
■■ Improvement strategies implemented by a team during the data collection period
■■ New employees who did not understand the data collection procedures
Any or all of these factors can create special causes in your data.
As an alternative to preparing control charts, you can develop a histogram for each indicator to
see if the data are reasonably close to forming a normal distribution. Histograms can be made very
easily in most statistical software packages.

(continues)
278 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

Step 3: Develop Line Graphs

There are two basic approaches for testing the relationship between the paired indicators. The first
approach is to make a simple line graph with both indicators plotted on the same graph. The second
approach is to develop scatter diagrams, which are discussed in Step 4.
FIGURES 10-10 and 10-11 show two possible conditions for the relationship between the number
of nursing vacancies and the number of falls each month. Each of these graphs has two y axes: one
for the number of nurse vacancies on the left side of the graph and one for the number of falls on
the right side of the graph. It is possible to have only one y axis on the left side of the graph if the two
indicators being compared have a very similar value range.
Figure 10-10 shows a pattern in which both indicators follow a similar path. Notice that the two
lines have different amplitudes but basically follow the same path over time. As the number of nursing
vacancies goes up so do the number of falls. Similarly, as the nursing vacancies go down so do the
number of falls. This demonstrates that the two indicators are related and follow a similar course (i.e.,
as one goes up the other goes up, and when one goes down the other also goes down). This is the
simplest way to show a relationship between two indicators. In Figure 10-11, we see a different pattern
between the two indicators. As the number of nursing indicators goes down we observe that the
occurrence of falls goes up. For these two indicators, however, the hypothesis was stated as a positive
relationship: As the number of nursing vacancies increases, we expect patient falls also to increase. Simple
line graphs provide the first level of analysis, but they do not allow you to evaluate the strength of the
relationship between the two indicators. To do this you need to move to scatter diagrams.
# of Nurse Vacancies

# of Falls

Month

Number of nurse vacancies

Number of patient falls

FIGURE 10-10 A line graph showing two indicators that are positively related

(continues)
 Applying Quality Measurement Principles 279

# of Nurse Vacancies
CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

# of Falls
Month

Number of nurse vacancies

Number of patient falls

FIGURE 10-11 A line graph showing two indicators that are negatively related

Step 4: Develop Scatter Diagrams

A scatter diagram (also called a scatterplot, a scattergram, or an x,y plot) is a graph of data points based
on two variables, where the one variable defines the horizontal (or x) axis and the other defines the
vertical (or y) axis. These graphs enable you to test for possible cause and effect relationships. Note that
scatter diagrams do not prove that one variable causes the other. Instead they allow you to determine
whether a relationship exists and the possible strength of that relationship.
The data for scatter diagrams must be variables data that is measured along a continuous scale.
Attributes data (e.g., categories such tall versus short, cesarean section delivery versus vaginal delivery,
surgery started on time or not on time) cannot be placed on a scatter diagram because they are
categorical types of data. The values of the two indicators are then arranged as pairs of coordinates (i.e.,
the x axis is the independent variable and the y axis is the dependent variable or effect). These coordinates
are plotted as dots on the graph. If the dots are clustered close together, the relationship between the two
indicators is considered to be stronger than if the dots are spread farther apart. The more the clustering of
dots looks like a straight line, the stronger the relationship is between the two indicators.
FIGURE 10-12 shows a strong positive relationship (correlation) between the number of falls (the
y axis) and the number of registered nurse (RN) vacancies (the x axis). When the number of nurse
vacancies is low the number of falls is also low. FIGURE 10-13, on the other hand, shows a weak positive
relationship between falls and RN vacancies (the dots are more spread out and not clustered in a tight
pattern). Note that when the two indicators are positively related, the pattern of the dots will rise from
the lower left to the upper right. In this case, as one indicator increases the other one also increases.

(continues)
280 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

High High

Number of falls
Number of falls

Low Low
Low Number of RN vacancies High Low Number of RN vacancies High

FIGURE 10-12 A strong positive relationship FIGURE 10-14 A strong negative relationship
between the two variables between the two variables

High
High

Number of falls
Number of falls

Low
Low
Low Number of RN vacancies High
Low Number of RN vacancies High
FIGURE 10-15 A weak negative relationship
FIGURE 10-13 A weak positive relationship
between the two variables
between the two variables

High
Number of falls

Low
Low Number of RN vacancies High
FIGURE 10-16 No relationship between the two
variables
In contrast, negative relationships present patterns that are similar to those shown in FIGURES 10-14
and 10-15. In Figure 10-14, the pattern reveals that when the number of vacancies was high, the number
of falls was low. When the number of falls was high, however, the RN vacancies were low. If you detect
negative relationships, one indicator’s values will increase and the other indicator’s values will decrease.
Again, the stronger the relationship, the tighter the pattern of the dots.
The final outcome of a scatter diagram might be that you discover no relationship between the
indicators. This is shown in FIGURE 10-16. When this happens the dots are randomly scattered on the
graph with no discernable pattern emerging.

(continues)
 Applying Quality Measurement Principles 281

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

High High
Strong - r
# of Falls

# of Falls
Strong + r

Low Low
Low # of RN Vacancies High Low # of RN Vacancies High

High High
Weak - r
# of Falls

# of Falls
Weak + r
Low Low
Low # of RN Vacancies High Low # of RN Vacancies High

FIGURE 10-17 No relationship between the indicators

No correlation (r = ~0)

High
# of Falls

Low
Low # of RN Vacancies High

FIGURE 10-18 No relationship between

two indicators

Note that if you really want to maximize the use of a scatter diagram, you should produce a
correlation coefficient to accompany the scatterplot. This is a single value that tells you the actual
strength of the relationship between the two indicators. The correlation coefficient (usually designated
as r) will range from –1.0 to + 1.0. The other option, besides having a correlation coefficient calculated,
is to have the software plot the linear least-squares regression line through the scatterplot as shown in
FIGURES 10-17 and 10-18. If you do go the route of plotting regression lines through a scatterplot realize
that not all relationships are linear in nature as shown in FIGURE 10-19.
In summary, the regulatory agency is interested in knowing whether you can detect relationships
between pairs of indicators. This means that that team assigned to answer this question needs to:
1. Select pairs of indicators representing human resource and clinical/service activities.
2. Develop theories about the possible relationships between the indicator pairs that you have
selected. For example, if you select staff satisfaction as the human resource indicator and patient

(continues)
282 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

800
700 Y=0.0477x2 + 2.2108x + 2.3052

Stopping Distance
600
500
Y-axis

400
300
200
100
0
0 20 40 60 80 100 120
Speed
X-axis

Y=-0.25X2 + 2.73X - 0.1 20 r = -0.172

Fuel Used (liter/100km)

12
10
15
8
6
Y

4 10
2
0 5
0 2 4 6 8 10 12
0 50 100 150
X
Speed (km/h)

Mrs. Johnson's 2nd Grade at Smith

Elementary School
90
Performance

80
IA Reading Score

70
60
50
40
30
20
10
0
Low Moderate High 0 5 10 15
Average Number of Hours of Sleep per Night
Amount Anxiety

FIGURE 10-19 Nonlinear relationships for scatterplots

(continues)
 Applying Quality Measurement Principles 283

CASE STUDY #6: Evaluating Staffing Effectiveness (continued)

satisfaction as the clinical service indicator, what theories or hypotheses do you have about the
possible relationship between these two indicators? Do you think there will be a positive or
negative relationship?
3. Do a quick analysis of the data to see whether they have been collected with consistent
operational definitions and the time periods for the indicators all match.
4. Create a control chart for each indicator to determine whether the data represent common or
special causes of variation. If special causes are present, investigate them and try to determine
why they are present in your data. Indicators that reflect special cause variation will create
problems for you when you try to interpret the line graphs and/or scatter diagrams.
5. Do a preliminary analysis by placing both indicators on the same line graph and determining
whether the lines track in a similar or dissimilar fashion.
6. Create scatter diagrams to analyze the pattern of the dots and decide whether there is a positive
relationship, a negative relationship, or no relationship between the indicators.
7. Determine the strength of the relationship. You can do this by merely using common sense
(the tighter the pattern, the stronger the relationship), or if you are comfortable with correlation
coefficients and regression analysis, you can request these statistics from the software program
you used to create the scatter diagram.

CASE STUDY #7: To Flash or Not to Flash—That

Is the Question
Situation
You are the director of surgical services at a 180-bed hospital that also has an outpatient surgery
center. Recently during lunch with one of your friends, who just happens to be the hospital’s director
of infection control, you mention that you have noticed an increase in the hospital’s surgical infection
rate. Your friend suggests that one possible area that could be contributing to this increase is the use
of flash sterilization (FS), which is frequently overlooked as a potential cause of surgical infections.5
When you first started working at the hospital, FS was used only in emergency situations (e.g., when an
instrument was dropped during surgery). Now it seems to have become a rather routine practice.
You decide to begin by looking at the FS rate and how it has varied over time. You consult with
the hospital’s measurement expert and with her assistance FIGURES 10-20 and 10-21 were developed.

Discussion
The data are plotted weekly (the subgroup) on u-charts. The FS rates are calculated as follows:
Total number of FSs done each week
FS rate =
Total number of surgeries done each week
■■ Why do you think a u-chart was used instead of a c-chart or a p-chart?
The u-chart is the chart of preference because each surgery case could have more than one
FS. Each time an FS occurs it is viewed as a defect. A c-chart would not be used because the

(continues)
284 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #7: To Flash or Not to Flash—That

Is the Question (continued)

0.9

0.8
Flash sterilizations per 100 surgeries

0.7
UCL = 0.653
0.6

0.5
CL = 0.433
0.4

0.3

0.2 LCL = 0.212

0.1

0.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40
Week

FIGURE 10-20 Flash sterilization rate by week (u-chart)

Baseline New surgeons Inspection Post Inspection

1.1

1.0

0.9
Flash sterilizations per 100 surgeries

0.8 UCL = 0.823

0.7

0.6
CL = 0.570
0.5

0.4

0.3
LCL = 0.317
0.2

0.1

0.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

FIGURE 10-21 Flash sterilization rate by week with defined sets (u-chart)

(continues)
 Applying Quality Measurement Principles 285

CASE STUDY #7: To Flash or Not to Flash—That

Is the Question (continued)

c-chart requires that the subgroups for counting the defects (which are weeks in this case
study) have an equal opportunity for a defect to occur. Because a different number of surgeries
are performed each week, this constitutes an unequal subgroup, which requires the use of the
u-chart. A p-chart would be used only if the measurement question was as follows: “Was FS used
during this surgical case?” The answer to this question is simply yes or no. The p-chart does not
address the issue of multiple occurrences of a FS. If a surgical case had 1 FS or 100 FSs, it would
merely be recorded as a “Yes” on a p-chart because the frequency of the event does not matter.
The p-chart is used to measure the percentage or proportion of defectives, but the u- and
c-charts are used to count the actual number of defects.
■■ Do you detect any special causes in the charts? If so, what are they and what do they tell you?
Figure 10-20 reveals numerous special causes owing to shifts in the process. As you look at the
patterns in the data, you notice that there seems to be a cyclical nature to the data. The dots
form a cluster in the lower left side of the chart, then they jump up to a higher FS rate. After
running for a few weeks at a fairly high level, the data drop dramatically to a lower level. Finally,
in the last part of the u-chart the process shifts again to a higher level (actually the highest level
so far). What could be causing these cyclical shifts in the data? The statistician supporting this
initiative suggests that it might be useful to partition the data into sets. Not quite sure what
she is suggesting, you follow her lead and agree. After a few quick strokes on the keyboard, she
produces the chart shown in Figure 10-21. In this chart, all the special causes have disappeared,
and there are four distinct phases that form common cause sets of data. As you ponder
this chart, you notice the dates associated with the four segments. All of a sudden the light
bulb goes on and you identify the four phases as (1) the baseline—that is, when you started
tracking the FS rate; (2) when a new group of surgeons came on board; (3) when the hospital
was gearing up for an inspection by an external regulatory body; and (4) the period after the
regulatory body’s inspection.
This is a clear demonstration of how organizational conditions can have rather dramatic
impacts on the performance of a process. The addition of new surgeons created an entirely new
process. Why these new surgeons had such an impact is unknown at this point but certainly
merits further study. The impact of a pending inspection by an external regulatory body had
another dramatic impact on the process. Even the new surgeons seemed to have changed their
behaviors with respect to FS during this time. But after the inspection was over, the use of FS not
only increased but also moved to an all-time average high of 57 FSs per 100 surgeries.
■■ How will you know whether a planned change improves the current process?
If there is serious concern over the FS rate, then the last time period (weeks 30–40) should
serve as the new baseline. If the objective is to lower the FS rate, then the creation of a team
would be advisable. The team should continue to track the FS rate and begin to gain a better
understanding of why FS is used. In this case, the creation of Pareto diagrams documenting
the reasons for flashing an instrument and the types of surgical instruments being flashed
would help to identify the vital few areas for improvement. The team might also consider
finding out what the FS rates are at other hospitals in the area. If the hospital is part of a system,
then getting this same indicator from other hospitals in the system sets up the opportunity
for internal benchmarking. As the team develops improvement strategies they can freeze the
control limits and CL on the new baseline (i.e., Phase 4) and then use these reference lines as a
basis to decide if their interventions are able to reduce the FS rate.
286 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #8: Clarifying the Operational Definition

of Readmission
Situation
A home healthcare agency is interested in tracking the percentage of its patients that are readmitted
to the hospital. So, they set about collecting data only to find out that the numbers seem to vary rather
dramatically from what they expected. Do you (1) assume that the data are the true reality and that the
agency staff has been missing something, or (2) do you go back to square one and check the accuracy
and efficiency of their data collection procedures?

Discussion
Whenever data seem to be at odds with what you think they should be telling you, the first thing to
do is to check the data before you jump to any conclusions about the results. Three key questions
should be explored: (1) Is the operational definition clear? (2) Is everyone interpreting the operational
definition in the same way? and (3) Have the data collectors gathered the data in the same way?
Based on my experience in developing indicators, the most common problem is that the operational
definition is not clear, which sets the stage for the next two issues to occur.
A good operational definition should be clear and unambiguous; it should define the decision
criteria to be applied when collecting the data and the methods to follow when actually collecting
the data. Basically, if you have developed your operational definition correctly, all three questions
will be answered simultaneously. Next consider issues related to defining a readmission. The
major issue I have encountered is first determining if the readmission is for the same diagnosis
or any diagnosis. Most of the time the team’s interest is in determining whether the patient was
readmitted for the same diagnosis. But it is surprising how many times I have found teams in
conflict over this issue. Although the team members usually reach consensus on the fact that
readmission for the same diagnosis makes the most sense, the people pulling the data are often
not told to eliminate all readmissions for a different reason. For example, if a congestive heart
failure (CHF) patient was admitted to the hospital for a CHF-related problem and then readmitted
within 30 days for a broken arm, would this be considered a readmission? Most staff would
conclude that this is not a readmission because the patient was not admitted for a CHF-related
problem. I have seen teams debate this conclusion because some think that the CHF caused the
patient to stagger and then fall on his arm. Therefore, the patient’s CHF was the causal factor
leading to the readmission. The other issue that I have seen enter into discussions of readmission
is whether the team was considering a 30-, 60-, or 90-day readmission. It seems obvious, but I have
seen teams waste considerable time and effort because they did not specify the time frame for a
readmission. So, before assuming that the data do in fact reflect the current performance of the
process, always start by reviewing the operational definition and your data collection procedures.
If the operational definitions and data collection procedures prove to be consistent with expected
practice, then it is time to start understanding the variation inherent in the data both conceptually
and statistically.
 Applying Quality Measurement Principles 287

CASE STUDY #9: Managing a Breast Cancer Patient’s

Clotting Levels6
Situation
A breast cancer patient and her oncologist decide to have a mediport inserted into her chest to aid
in the delivery of chemotherapy drugs. The insertion of the port goes without incident, but about
a month after it is in place, the patient develops a deep vein thrombosis (DVT) in her left arm. She
is hospitalized for several days on heparin, sent home on a brief regimen of enoxaparin injections,
and then begins oral doses of warfarin sodium. During each visit to a local outpatient clinic (which
takes about 15 minutes), a small amount of blood is extracted via a finger stick and analyzed, and
adjustments are made on the spot. The patient assists the pharmacist by keeping track of her daily
intake of vitamin K (inconsistent intake of vitamin K can have a dramatic impact on the effect of
warfarin sodium). The challenge with managing a patient on this particular drug is that there is a very
narrow SL for the proper management of the effects of the drug (i.e., two to three on the international
normalized ratio [INR] scale). When the patient exceeds the USL of 3.0, there is an increased risk of
bleeding. Conversely, when the patient falls below the recommended therapeutic limit of 2.0, there is
an increased risk of clotting. The pharmacist creates control charts specific to each patient to assist in
managing the patient’s progress on warfarin sodium. FIGURES 10-22 and 10-23 show the patient’s INR
values during and after chemo and radiation.

5.0
4.5 UCL=4.14
UCL USL
4.0
3.5
INR Value

3.0
CL=2.38
2.5 CL
2.0
1.5
1.0 LCL LSL LCL=0.62
0.5
0.0
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233
Blood Draw

2.5
UCL 2.16
2.0
Moving Range

1.5
1.0
CL 0.66
0.5
0.0
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233
Blood Draw

FIGURE 10-22 INR values for a breast cancer patient on warfarin sodium (XmR chart)

(continues)
288 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #9: Managing a Breast Cancer Patient’s

Clotting Levels (continued)

6.0
During chemo and radiation
5.0 UCL 4.92
4.0 USL
INR Value

After chemo and radiation

3.0 CL 2.60
2.73
2.0
1.78
1.0 LSL
LCL 0.53 0.95
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
Blood Draw

3.0
UCL 2.70
2.5
Moving Range

2.0
1.5
1.01
1.0 CL 0.83
0.5 0.31
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
Blood Draw

FIGURE 10-23 INR values for a breast cancer patient on warfarin sodium before and after chemo and
radiation (XmR chart)

Discussion
■■ What chart did the pharmacist make for this patient? Why do you think she used this chart? Is there
another chart that she could have used?
Because the indicator used to measure the effect of warfarin sodium is a continuous measure,
the pharmacist has a choice of making the X-bar and S chart or the XmR chart. The chart of
preference in this case is the XmR chart (a.k.a. the Individuals or I chart) because there is only
one observation per subgroup (i.e., each time the patient visits the clinic, which is weekly, the
pharmacist takes one blood draw for analysis). If multiple blood draws were taken each week,
the pharmacist could make the X-bar and S chart. Because multiple blood draws each week are
not necessary, the appropriate chart is the XmR chart.
■■ Is this patient’s process in control or out of control?
The patient’s process is not in control. There are two special causes in Figure 10-22 which are
highlighted in black. One occurs at blood draw 16 (a 3 sigma violation) and the other one
occurs after chemo and radiation were terminated after blood draw 21 (i.e., the process stayed

(continues)
 Applying Quality Measurement Principles 289

CASE STUDY #9: Managing a Breast Cancer Patient’s

Clotting Levels (continued)

below the centerline for more than eight data points in a row [data points 22-30], signaling a
downward shift in the INR values).
■■ What do we learn by looking at the moving range (mR) chart?
The mR chart (the bottom chart of Figure 10-22) depicts the variation from one data point to
the next. If there are large swings from one data point to the next, then this chart will have wide
control limits. The mR chart exhibits wider variation during the period when the patient was
receiving chemo and radiation. After these treatments were ended at blood draw 21, the mRs
do not show as much fluctuation (blood draws 22–33). This is a subtle point but notice that the
mR chart (the bottom chart) has one less data point than the top chart (known as the X chart).
This is because in order to calculate a mR there must be two data points. The first INR data point
on the X chart is a single measure which has no variation. Variation requires at a minimum the
comparison of two data points, which does not happen until the first INR value is compared to
the second INR value on the X chart. So, the mR chart has one less data point because the mR
cannot be calculated on the first data point by itself. The first mR on the chart is zero because
blood draws 1 and 2 happen to be the same value (2.3) and there is no range between numbers
of the same value. Because blood draw 2 is different from blood draw 3 (2.3 compared to 2.8)
we now observe the first mR on the bottom chart, which is 0.5. This process of creating what is
called “artificial subgroups of two” by coupling each single data point with its neighbor is key to
understanding how the XmR chart is constructed.
■■ Note that the chart shows the USL and LSL. Of what value are these lines? How do they differ from the
UCL and LCL?
A number of clinical measures actually have SLs. The healthcare measures that typically have
SLs are those that relate to physiological measures (e.g., hemoglobin levels, white blood
counts, neutrophils, prothrombin time or platelet counts). Warfarin sodium levels are measured
in terms of clotting times. The standard measure is known as the INR value. The normal INR
for individuals not on warfarin therapy is typically 0.9 to about 1.1. On warfarin therapy,
the INR usually elevates to between 2.0 and 3.0 but most hospital pharmacies and clinical
hematology services will have specific INR goals documented in their treatment protocols.
These therapeutic values (i.e., 2.0–3.0) can be used as the USL and the LSL for the drug. When
the patient’s control limits do not coincide with these SLs, then the patient is at risk. The
principal difference between SLs and control limits is that the SLs represent the desired levels
of performance, sometimes referred to as the voice of the customer (VOC), whereas the control
limits capture the actual levels of variation in the data. SLs can be developed by those making
the chart whereas the control limits are derived from the inherent variation that lives in the
data. Stated otherwise, the maker of the chart does not decide what the UCL and LCL are but
can establish the USL and LSL.
■■ Figure 10-23 partitions the data points into two segments. The left side of the chart shows the
INR values for the patient while undergoing chemo and radiation (blood draws 1 through 21).
The second set of control limits shows the INR values for the patient after all chemo and radiation
treatments were completed (blood draws 22–33). Why do you think the pharmacist divided the
chart into two parts? Does it help understand the patient’s process? What could be causing the shift
in the INRs?

(continues)
290 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #9: Managing a Breast Cancer Patient’s

Clotting Levels (continued)

The pharmacist knew, from experience, that once a patient completes the chemo and radiation
treatments, the management of warfarin sodium usually moves into a new phase. Specifically,
without the influence of chemo and radiation, the patient’s body began to exhibit its true
response to warfarin sodium. So the pharmacist divided the control chart into a treatment
phase (blood draws 1–21) and an after-treatment phase (blood draws 22–33). Consistent with
previous patients, this patient also exhibited a fairly dramatic drop in the INRs after chemo and
radiation treatments were terminated. This was followed by a gradual increase to a more stable
period that coincided (for the first time) with the INRs being close to the specification limits.
The mR chart after chemo and radiation ended shows much greater stability than in the mR in
the baseline phase. Frequently the mR chart can be quite valuable when studying clinical issues
at the patient level. Stability in INRs, blood sugar, or blood pressure, for example, are in many
ways easier to address than constant swinging between extreme readings. So, when applying
the Shewhart charts, especially the XmR or X-bar and S charts, to physiological measures at the
patient level the mR chart can be just as valuable as the average (X) chart.
■■ What other patient conditions can you think of that could be monitored with Shewhart charts?
Shewhart charts have great utility for tracking blood pressure; blood glucose levels; the daily
intake of fat, sodium, and cholesterol; or the impact of physical therapy treatments. Take a few
minutes to think about how you could apply Shewhart charts to your discipline.

CASE STUDY #10: Group B Streptococcus in Pregnant

Women
Situation
A team of nurses at Advocate Good Samaritan Hospital (Downers Grove, Illinois) was interested in
studying the administration of antibiotics to pregnant women who present for delivery with a positive
culture for group B streptococcus (GBS).7 According to the Centers for Disease Control and Prevention,
about 20–30% of all women have GBS naturally in their bodies. Although GBS is benign to the woman,
the transference of GBS to the newborn during delivery can have extremely negative consequences
for the baby, including death (approximately 1 death per 1,000 deliveries). Women should be tested for
the presence of GBS during weeks 35–37 of pregnancy. If they test positive for the flora, they should
be receiving intrapartum antibiotic prophylaxis (IAP) for at least 4 hours prior to delivery in order
to minimize the potential of transferring the GBS to the newborn and thereby creating a very high
probability of sepsis and possibly death of the newborn.

Discussion
The nurse manager for labor and delivery was somewhat dismayed when she called me because she
thought the percentage of women receiving IAP was not improving. She said that she thought the
problem was the result of an increase in the number of women presenting with a positive GBS who

(continues)
 Applying Quality Measurement Principles 291

CASE STUDY #10: Group B Streptococcus in Pregnant

Women (continued)

were under the 4-hour threshold. Because you cannot tell a pregnant woman, “We’re going to wait 3
more hours to deliver your baby so we can administer antibiotics,” these women receive less than the
recommended dose of antibiotics. I asked the nurse manager if they were combining all cases (i.e.,
women presenting 4 hours or more before delivery and women presenting less than 4 hours prior to
delivery) in their data analysis. She said that they were. I asked if they could stratify the cases into these
two categories and she said that they could. The issue here is that the increased volume of women
presenting 4 hours before delivery is something the hospital cannot control. The application of the
antibiotic protocol should be applied only to the women who present at least 4 hours before they
deliver. These are the women who can experience positive results from the application of prophylactic
antibiotics. Those who present less than 4 hours prior to delivery should be placed in a separate
category (stratum) and dealt with separately.8
FIGURE 10-24 depicts the various levels of stratification developed with this team. Note that the
decision tree places patients into categories or buckets that are mutually exclusive. In this way, the
team could decide which categories it wishes to measure and which are not particularly relevant
to their intended objectives. Stratification is an aid, however, in making these decisions because
it can assist them in determining whether the data can be broken down into smaller and more
homogeneous comparison groups.
While developing the decision tree in Figure 10-24, the team also started to collect data on the
GBS process. They had only 3 months of data as a baseline. This is not ideal for a baseline but it is what
the team had so it had to suffice. After the decision tree was developed and the patient population
stratified properly they collected 5 more months of data. FIGURE 10-25 shows a p-chart with the data
organized into monthly subgroups, phased to show the baseline compared to the properly stratified
patients receiving IAP and the national average of 89%. The average for the first 3 months when the
two patient populations were combined is 79.9% whereas the average for the 5 months after the
patient population was properly stratified is 96.1%, which is the true percentage of compliance with
the IAP. This is a good example of how a team can be thinking that their performance is worse than it
really is simply because they had not properly stratified the data into relevant segments
The team’s next challenge will be to move this average of 96.1% to be as close to 100% as
possible. The team’s improvement strategies included:
■■ Revising their data collection instruments and procedures to make sure they are all using the
logic shown in Figure 10-22 and properly stratifying the patients into the relevant categories
■■ Evaluating the volume of cases they experience each month to see whether they can use week
or every 2 weeks as the subgroup rather than month
■■ Continuing to update the p-chart on the percentage of IAP compliance with GBS positive
patients
■■ Providing continuing education for the nursing staff and patients on the importance of
understanding that IAP should be initiated for women with GBS within 30 minutes of admission
■■ Enhancing collaboration between nursing and clinical management to make sure pitocin
inducement is delayed for women with GBS
■■ Increasing the number of nurses involved with the data collection efforts
■■ Adding GBS status to the mother/infant record and prenatal record

(continues)
292 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #10: Group B Streptococcus in Pregnant

Women (continued)

Total patients for the month

Number with Number with

complete incomplete
documentation documentation

GBS culture not done GBS culture done

Women with Women with GBS negative GBS positive

no risk factors risk factors culture culture

Women with
no risk factors

Women Women
presenting less presenting more
than or equal than 4 hours
to 4 hours prior to delivery

Women not Women

receiving the receiving the
appropriate appropriate
antibiotic antibiotic
regiment regiment

FIGURE 10-24 Stratification of pregnant women with group b streptococcus (GBS)

(continues)
 Applying Quality Measurement Principles 293

CASE STUDY #10: Group B Streptococcus in Pregnant

Women (continued)

CL=96.1%
100%
UCL
90%

80% National Average = 89%

70% CL=79.9%

LCL
60%

50%

40%
Jan Feb March April May June July Aug
Month

FIGURE 10-25 Percentage of IAP compliance for patients with group B streptococcus (p-chart)

CASE STUDY #11: Emergency Department Fast Track

Situation
An emergency department (ED) has proudly announced to the community that it now has a “fast
track” in place for emergency visits that are not life threatening. Brochures have been developed and
a banner has been placed across the ED proclaiming that “Fast Track is here!” The ED team set a goal of
75 minutes from ED registration to discharge from the fast track area. They did not use data to establish
this goal; it “just felt right.” Besides, one of the staff had read that a hospital in New York City was able to
get people in and out of its fast track service in 60 minutes. If an urban city hospital could achieve
60 minutes or less, then surely a suburban hospital should be able to do it in 75 minutes. Unfortunately,
after a month of operation the fast track process was receiving more complaints than compliments.
What would you advise the ED Fast Track team to do?

Discussion
The quality measurement expert helped the team develop two analytic tools. First they developed a
Pareto diagram to identify the vital few reasons people were presenting to the fast track. As you can
imagine the reasons were quite varied. Patients were presenting for minor sports injuries like badly
sprained ankles to more complex issues requiring stitches or even rule-outs for potential cardiac

(continues)
294 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #11: Emergency Department Fast Track (continued)

events. This analysis allowed the team to decide that two categories within the fast track area would be
established (i.e., visible blood versus no visible blood).
Next the QI measurement expert guided the team on the development of baseline data. She
cautioned the team that establishing a goal, in this case 75 minutes or less, without baseline data was
not a particularly good idea. But this suggestion seemed to fall on deaf ears because the team had
already publicized the 75 minute or less limit not only to management but to the public. The team was
at least willing to start building a baseline to see how close they were to 75 minutes.
The volume of people coming to the ED fast track process during the first 2 weeks of operation
varied from a minimum of 47 and a maximum of 89 each day. Based on these initial numbers, the QI
measurement expert advised the team that they did not need to record the wait time of every patient.
A systematic random or stratified systematic random sample would be sufficient to understand the
variation in the process. After she explains the differences between the two sampling approaches, the
team decided that it would be best to draw a stratified systematic sample each day. They also thought
that two stratification levels would be relevant to this work: (1) day versus afternoon shift (the fast track
process does not operate during the night shift), and (2) type of injury coming to the fast track process
(i.e., visible blood versus no visible blood). This means that the team will have five different charts to
evaluate:
■■ Wait time in the fast track day shift—visible blood patients
■■ Wait time in the fast track day shift—no visible blood patients
■■ Wait time in the fast track afternoon shift—visible blood patients
■■ Wait time in the fast track afternoon shift—no visible blood patients
■■ Total wait time in the fast track—both shifts and both injury types
A systematic sampling plan was laid out to accommodate the four stratification levels, log sheets
were created, and a date was selected to start collecting patient wait times. The day of the week was
selected as the subgroup for the Shewhart chart (i.e., the units placed on the x axis of the Shewhart
chart) including weekdays and weekends.
Each day the team selected a random number from 1 to 10 at the beginning of each shift.
This random number then became the starting point for pulling the systematic sample. If a random
starting point is not selected for a systematic sample then it cannot be considered a random sample.
In Chapter 4, the details of pulling a systematic sample are provided. The team agreed to track the wait
time for every fifth patient presenting in each of the four stratification levels. Each day at the start of
each shift a team member generated a random number between 1 and 10 and then selected every
fifth patient after that number and tracked the patient’s wait time in the fast track process.
Although a systematic sample of five patients in each of the four stratification levels was
the objective, some days they get more than five patients and other days they got less. The
team expressed concern that this variable subgroup size would affect the analysis of the data.
The QI measurement expert assured them, however, that this will not be a problem, because the
Shewhart chart she will be preparing for the team (the X-bar and S chart) can handle varying
subgroup sizes. They collected data for 20 days, all the while trying to get patients through the fast
track process in 75 minutes or less. The baseline data on the X-bar and S chart for the day shift and
patients with no observable blood is presented in FIGURE 10-26. What does this chart tell you about
the fast track process?
The first thing that you observe is that on the last Wednesday on the chart (fourth data point from the
right) a special cause occurred. This is a 3-sigma violation because the average for the day (84.4 minutes) fell

(continues)
 Applying Quality Measurement Principles 295

CASE STUDY #11: Emergency Department Fast Track (continued)

140.0

135.0

130.0
UCL = 127.235
125.0
Fast track wait time (in minutes)

120.0

115.0

110.0
CL = 107.958
105.0

100.0
Special cause
95.0 (3-sigma violation)

90.0
LCL = 88.680
85.0

80.0

40.0

30.0
UCL = 28.215
Sigma

20.0
CL = 13.506
10.0

0.0 LCL = 0.000

M W F M W F M W F
Day

FIGURE 10-26 Emergency department “fast track” wait time (day shift with minor injuries) X-bar
and S-chart
below the LCL for this data point. This is actually a special cause that the team would like to see repeated,
because it is the lowest wait time during the entire study period but still not near the goal of 75 minutes.
The team should investigate why the process worked well on this day. Were the patients presenting with
less critical problems? Was the regular ED slow so that staff came over to assist on the fast track side? Could
it be that there was a data-recording mistake on this day and someone transposed a number? Why was
this day different from the other 19 days? There are lessons to be learned with every special cause.
The second thing you should notice is that the control limits are not straight. These are called
stair-step control limits. They are found on any chart that does not have a consistent subgroup size.
Stair-step control limits are seen most of the time on the p- and u-charts, but you will find them on an
X-bar and S chart when the sample being pulled is not consistent over time. In the present example,
the subgroups ranged from three to eight each day.

(continues)
296 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #11: Emergency Department Fast Track (continued)

Finally, you do not need a degree in statistics to see that this process is not operating anywhere
near the desired goal of 75 minutes. In fact, the process is running so high that the chart does not
have room to show the goal of 75 minutes or less. Despite the fact that there is a special cause present,
which basically makes the process unstable and unpredictable, you can still apply the “intraocular test
of significance” to the data (i.e., the old eyeball test) and see that the average time to go through the
fast track process is 108 minutes. The average UCL is 127 minutes and the LCL estimate is 89 minutes.
The complaints of the patients seem to be justified—particularly if word got out that this process was
going to get people in and out in 75 minutes or less.
So what should the team do now? One of the staff suggests taking down the banner that
proclaims that “Fast Track is here!” This idea is considered but then discarded; the banner stays up. The
next idea is to see whether the baseline data for the afternoon shift differ from that of the day shift and
whether the presence of blood or no blood in the presenting patients has an impact on the process
wait time. They need to look at the other four X-bar and S charts to see what they can learn from the
variation in the entire process. The QI expert suggested that there also may be difference based on
the hour of the day and the volume of patients coming into the ED. In conjunction with the additional
data work, they also decided to develop a flowchart on the process and see whether they can identify
any bottlenecks and delays in the process.
The final thing the team did was to take time to engage in a dialogue about the nature of the
concept of the “fast track.” They begin to wonder if everyone has the same operational definition of this
concept. How fast can the ED fast track process be expected to go? What does the public think about
when they see a sign that tells them there is a fast track process in place? Part of this team’s problem is
simply managing the customers’ expectations. It is a classic example of creating an expectation in the
customer’s mind, publicizing the idea, and then implementing a process that has not been tested under
different conditions and is not capable of even meeting expectations, let alone exceeding them. Quality
does not happen merely because you have good intentions and a colorful banner across the entrance
to the ED. It happens when you understand the VOC and the voice of the process (VOP) and then take
action to make sure that the process is capable of at least meet minimum expectations. In this case
study, it is not surprising that the fast track team was not near their goal. They did not start out with an
understanding of the capability of the process. Their good intentions were detoured by poor planning.

CASE STUDY #12: Tracking Patient Complaints

Situation
A group practice clinic decides to change the processes for registering patients and scheduling
follow-up visits. Within the center are the following subspecialties: family practice, internal medicine,
and obstetrics/gynecology. In addition, there is an outpatient laboratory, a pharmacy, and a durable
medical equipment service. In the past, each of the medical subspecialties had their own areas for
registration, scheduling follow-up visits, and payment (billing/insurance questions, etc.). Driven
primarily by financial factors, the managers of the various subspecialties agreed to consolidate all the
administrative functions. As a result, they decide to have one area to handle the three functions of
registration, scheduling follow-up visits, and handling billing questions. They also decide to physically
rearrange the floor space and the way in which the patients move within the facility.

(continues)
 Applying Quality Measurement Principles 297

CASE STUDY #12: Tracking Patient Complaints (continued)

Discussion
■■ What type of control chart should be used to track the number of complaints?
Complaints are typically regarded as defects (i.e., attributes data). In addition, because one
patient or their representative can make more than one complaint we can eliminate further
consideration of this being a binomial measure (i.e., the patient either complained once or
more or did not complain). Given that we want to count all the complaints including multiple
ones by one patient, the charting options are the c-chart and the u-chart. The c-chart provides
a count of the number of defects (complaints) when there is an equal area of opportunity
for a complaint to occur. The u-chart, on the other hand, is reserved for creating a defect rate
when there is an unequal area of opportunity for the defect to occur. The u-chart produces
a rate-based statistic (i.e., so many complaints per 1,000 patient visits). The c-chart was used
because the daily number of visits to the various subspecialties showed very little fluctuation
and the types of patients visiting the clinic remained reasonably constant. If either or both of
these assumptions were to change, however, the chart of preference would be the u-chart.9
■■ What conclusions do you make about the administrative changes that were put in place? Did they
have a positive or negative impact?
FIGURE 10-27A shows the results of changing the administrative process, which occurred during
month 9. The upward shift in the number of complaints is obvious. But, there are actually
eight specific special causes on this chart. Can you identify all of them? To detect some of the
special causes you will need to refer to the light dashed lines on the chart showing the zones

55

50

45 UCL=45.74
A
40
Number of Complaints

B
35
C
30 CL=29.46
C
25
B
20
A
15
LCL=13.18
10

0
1 3 5 7 9 11 13 15 17 19 21 23 24

Month

FIGURE 10-27A Total number of patient complaints at a group practice clinic (c-chart)

(continues)
298 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #12: Tracking Patient Complaints (continued)

55

50

UCL = 45.74
45
A

40
B
Number of complaints

35
C
30
CL = 29.46
C
25

B
20
A
15
LCL = 13.18
10

0
1 3 5 7 9 11 13 15 17 19 21 23
Month

Numerous special causes are present

FIGURE 10-27B Total number of patient complaints at a group practice clinic with special causes
identified (c-chart)

(designated by the letters A, B and C). You also may need to refer back to Chapter 9 for a brief
refresher on the use of the zones and applying the special cause rules. All eight special causes
are shown in FIGURE 10-27B. If this chart were to be used in a presentation to the management
team the presenter would merely say, “Ladies and gentlemen, the chart reveals numerous
special causes, which makes the complaint process unstable and unpredictable. The first thing
we need to do is to investigate why we are observing these various special causes.”
In the overview of this case study, it was noted that the administrative changes were made in
1 month, which is month 9 on Figure 10-27b. After this point the number of complaints started
to increase. When eight data points stayed above the mean (CL) this was evidence that the
complaint process had shifted to a new level of performance. This is justification to partition or
phase the data into two segments as shown in FIGURE 10-28. Now it becomes very clear that
there are really two processes. One occurred under the old administrative process and another
with the new process. Under the old arrangement the process produced around 18 complaints

(continues)
 Applying Quality Measurement Principles 299

CASE STUDY #12: Tracking Patient Complaints (continued)

Old Process New Process

70

60

UCL = 54.08
50 A

B
Number of complaints

40 C
CL = 36.07
C
30
B

A
20
LCL = 18.05

10

0
1 3 5 7 9 11 13 15 17 19 21 23 24
Month

FIGURE 10-28 Total number of patient complaints at a group practice clinic with historical control
limits (c-chart)

each month. After the changes were made, however, the complaints jumped up to an average
of 36 each month with an UCL of 54 and a LCL of 18, which by coincidence coincides with the
process average of the baseline. You can see from Figure 10-28 that the original process and the
new process both reflect common cause variation which means they are stable and therefore
predictable. The problem is that the new process (months 10–24) is predictably bad and seems
to be moving to even higher levels of complaints.
■■ Given these results, what are the next steps?
Because there has clearly been a shift in the process, the management team needs to have
a serious discussion about the trade-offs between cost and quality. If the desire is to cut
costs by consolidating the administrative functions and cutting staff, then the customer
satisfaction consequences of these changes need to be evaluated. If the complaints continue
to increase, it is very likely that patients may start to look elsewhere for medical care. Although
patients typically have strong allegiances to their physicians, there comes a point when the
administrative functions serve as a catalyst to push patients away from their current providers.

(continues)
300 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #12: Tracking Patient Complaints (continued)

This clinic staff needs to conduct further data analysis and develop a Pareto diagram on the
reasons for the complaints. They also should develop a series of flowcharts to identify the steps
involved in completing all the administrative functions. This would also help the team focus on
some of the reasons patients are complaining. If they do not gain a better understanding of why
patients are complaining, then they will flounder around in the dark, cursing the ever growing
number of complaints but not understanding which ones are the major problems or how to
address them. A systems view of this process is clearly needed.

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia10
Situation
The medical/surgical intensive care units (ICUs) just celebrated the longest run (20 days) they had ever
experienced without a ventilator-associated pneumonia (VAP). But within 2 days of this celebration
they began to see the occurrence of VAPs. The occurrence was sporadic at first with one VAP then a
run of 2–3 days without a VAP then another one or two VAPs. They decided that it was time to revisit
their approach to VAP management. Maybe they had become too comfortable with approaching a
month without any VAPs. Maybe the complexity of patients had changed. Maybe the new staff were not
properly trained in the application of the VAP bundle. They began by reviewing the literature (see for
example the resources we have on the IHI website related to the VAP bundle and management of VAPs:
http://www.ihi.org/Topics/VAP/) and then developed an updated flowchart on how they assess and
treat patients placed on a ventilator. One area that came up repeatedly in the team’s discussions was
that oral care procedures could be a causal factor for the increasing occurrence of VAPs. Upon further
investigation they also discovered that their current policy and practices on oral care for VAP patients
were not particularly clear and not being followed consistently. Finally, they discovered that new oral
health products had come on the market and had received very positive results in both hospitals and
skilled nursing facilities. The team decided to look first at their current oral care process and products
and determine if poor oral care could be a key factor leading to the increased occurrence of VAPs.

Discussion
Two nurses on the QI team partnered to review and then update the hospital’s policy and procedures
for the performance of oral care for all adult mechanically ventilated patients in the medical/surgical
ICUs. Current literature was reviewed to identify recommendations for comprehensive oral care on
ventilated patients. The nurses identified the following best practices:
■■ A daily assessment should be performed to evaluate the level of oral dysfunction and provide
the most appropriate care to keep the patient comfortable and prevent complications.
■■ Brushing a patient’s teeth should occur at a frequency of every 2–4 hours and as needed to
prevent the formation of plaque, which can be a reservoir for respiratory pathogens.
■■ Alcohol-free, antiseptic oral rinse should be used to prevent bacterial colonization of the
oropharyngeal area.

(continues)
 Applying Quality Measurement Principles 301

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia (continued)

■■ Suctioning of oral secretions in both the oral cavity and the subglottic area (above the cuff )
should be performed to prevent the aspiration of microorganisms.
■■ Application of a water-based mouth moisturizer should be used to maintain the integrity of the
oral mucosa.
The nursing staff also reviewed new oral hygiene products on the market. After a thorough
assessment of several new products, one was selected and a plan was developed to test the new
products. Once the new oral care policy and procedure statement was finalized, copies were
distributed to ICU nursing staff. Group education on the rationale for the changes and instructions on
proper product usage were provided by the manufacturer. The manufacturer also provided posters,
evaluation forms, and follow-up to further clarify, troubleshoot, and educate the staff. The presence of a
“champion” for this process change (in this case a RN from the ICU) provided leadership, accountability,
and support to test the new process and products.
The specific outcome indicator selected for analysis was the VAP rate. This indicator had been
tracked at the hospital for many years, so there was a solid baseline of 24 months. The VAP rate was
operationally defined as the number of pneumonia occurrences per 1,000 ventilator days. The specific
components of the VAP rate are:
■■ Numerator—Total number of inpatient ICU occurrences of VAP
■■ Denominator—Total number of days ICU patients spent on a ventilator
This resultant value is then multiplied by 1,000, so the rate can be compared with state and national
data (i.e., so many VAPs per 1,000 vent days).
The key process measure was the percentage of compliance with using the new oral health
procedures. This was defined as:
■■ Numerator—the number of VAP oral health cleaning opportunities performed correctly (i.e.,
according to the new procedures)
■■ Denominator—the number of VAP oral health cleaning opportunities
The resultant value from this ratio is then multiplied by 100 to produce the percentage of correctly
performed oral cleanings. Data for this percentage were collected manually each day and tabulated
on a monthly basis to match how the VAP rate was tabulated. The chart selected for analysis of the VAP
rate calculation was the u-chart, because this is the chart of preference for the presentation of rate-
based statistics (e.g., fall rate, medication error rate, nosocomial infection rate, or neonatal death rate).
The process measure of percentage of compliance with the new oral hygiene procedures was placed
on a p-chart (i.e., oral cleaning was done either correctly or not correctly).
FIGURE 10-29 shows the baseline VAP rate for 24 months before the new oral hygiene process
was tested. Because the baseline reflects only common cause variation the process can be considered
stable and therefore predictable. The best quick prediction of future performance is the average or
centerline on the chart. If nothing is done to change the process, therefore, the team can expect
to have about 28 VAPs each month. Also because the baseline demonstrates only common cause
variation the baseline centerline can be “frozen” and extended into the future as a reference line for
future data. This is shown in FIGURE 10-30.
The team began testing the new oral hygiene process in month 25. They plotted the new data
against the frozen centerline of the baseline to determine whether they were able to create a special

(continues)
302 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia (continued)

Baseline Ventilator Associated Pneumonia Rate (u-chart)

90

80
Rate (VAPs per 1000 vent days)

UCL
70

60

50

40

30

20
CL = 28.3 VAPs per
10 1000 vent days
0 LCL
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Month

FIGURE 10-29 Baseline Ventilator Associated Pneumonia Rate (u-chart)

Baseline Ventilator Associated Pneumonia Rate With extened centerline (u-chart)

90
Rate (VAPs per 1000 vent days)

80
70 UCL

60
50 Baseline
New oral hygiene
40 process started here
30
20
Extended CL of 28.3
10 from the baseline period
0 LCL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Month

FIGURE 10-30 Baseline Ventilator Associated Pneumonia Rate with extended centerline (u-chart)

cause. They noticed that the VAP rates were lower than those in the baseline period but it was not until
they observed eight consecutive data points below the baseline centerline of 28.3 that the team could
declare that a downward shift in the process had actually occurred. FIGURE 10-31 shows the shift in the
process and the fact that the data for subsequent months also remained below the baseline centerline.
The team then phased the data and made two sets of control limits and two centerlines to show how
the new oral hygiene process affected the VAP rate. These chart revisions are shown in FIGURE 10-32.

(continues)
 Applying Quality Measurement Principles 303

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia (continued)

Baseline Ventilator Associated Pneumonia Rate before and after the

new oral hygiene procedures (u-chart)
90
Rate (VAPs per 1000 vent days)

80
New oral hygiene
70 UCL process started here
60
50 Baseline Extended CL from
the baseline
40
30
20
10
0 LCL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Month

Downward shift in the process (8 or

more data point below the “frozen” CL)

FIGURE 10-31 Baseline Ventilator Associated Pneumonia Rate with extended centerline and shift in the
process (u-chart)

Ventilator Associated Pneumonia Rate before and after the new oral
hygiene procedures with new control limits and centerline (u-chart)
90
Rate (VAPs per 1000 vent days)

80
New oral hygiene
70 UCL process started here
60
50 Baseline CL = 28.3

40 New CL = 15.3 VAPs

30 per 1000 vent days

20
10
0 LCL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Month

FIGURE 10-32 Baseline Ventilator Associated Pneumonia Rate compared to the VAP rate with the new
process (u-chart)

(continues)
304 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia (continued)

Special cause
(3 sigma violations)
90%
80%
UCL
70%
60%
Percent

50%
LCL
40%
30%
20%
10%
0%
25 26 27 28 29 30 31 32 33 34 35 36
Month

FIGURE 10-33 Percentage compliance with the new oral hygiene protocol (p-chart)

Note that in the period after the new oral hygiene procedure was being used that the control limits on
months 25–36 are tighter than those in the baseline period and the centerline has now shifted from
28.3 VAPs each month to 15.3.
The team was also tracking the percentage of compliance with the new oral hygiene protocol. As
can be seen from FIGURE 10-33 the percentage of compliance progressed in a positive upward manner
once the new method was introduced in month 25 culminating in a special cause (the last 3 months
of data exceeded the UCL). This special cause is reflective of a process in transition that frequently will
be observed during the introduction and testing of a new procedure and methods. The team will need
to continue tracking compliance to make sure all staff are properly trained in the new methods and
their application of the new approach is reliable. Because the last three data points exceed the UCL
and seem to be stabilizing around 79% the team decided to phase the results shown in Figure 10-33.
FIGURE 10-34 shows the chart with two phases. The team can use this phased chart to see if they can
now move the process closer to 100% compliance.
In summary, the team’s theory about the impact of improved oral hygiene for mechanically
ventilated patients seems to be substantiated. As compliance with using the new oral hygiene
protocol increased they began to see a drop in the number of VAPs. This relationship between the
process indicators and the outcome indicator is central to QI initiatives. The process indicators are
essentially the independent variables that are believed to influence or drive changes in the dependent
or outcome indicator.
The team’s next steps should include:
■■ Obtaining comparative reference data or norms on VAP rates within their local area, state, region,
or country.

(continues)
 Applying Quality Measurement Principles 305

CASE STUDY #13: Reducing Ventilator-Associated

Pneumonia (continued)

100%
90%
80%
70%
UCL
60%
Percent

50%
40%
LCL
30%
20%
10%
0%
25 26 27 28 29 30 31 32 33 34 35 36
Month

FIGURE 10-34 Percentage compliance with the new oral hygiene protocol with phased control limits
(p-chart)

■■ Continuing to track both the VAP rate and the percentage of compliance with the new
procedure. If the VAP rate starts migrating back toward the higher preprotocol average of 28.3,
then it would appear that (1) the new oral care procedures were not as effective as first thought
and/or that (2) staff behaviors did not change as much as needed in order to sustain the initial
improvements. It is also possible that other untested factors could be influencing the decline
observed in the postprotocol VAP rates. Finally, if the postprotocol VAP rates continue to stay
below the preprotocol average for several more months, then a shift in the process will have
been verified. The team should explore all of these possibilities, however, to make sure that they
fully understand the factors that are driving the hospital’s VAP rate.
■■ The team should also explore the feasibility of moving away from monthly data to a weekly
or every 2-week period for the subgroup. Waiting for monthly data will not only slow down
improvement efforts but for some topics, like infections, the consequences of waiting for
monthly results could have serious implications for the patients. The decision to go to a
weekly or biweekly time frame will hinge on the volume of vent days and the number of VAPs
occurring in the ICUs. In this case study, the count of VAPs each month was always in the single
digits. If the team were to move to weekly or even biweekly subgroups they may have too few
occurrences to make viable rates. Also as they get closer to zero VAPs the VAP rate calculation
becomes less meaningful. When they do start to have a run of 50 days or more without a VAP
they will need to move away from the VAP rate and start to plot the days between a VAP. This
requires using a t-chart. T-charts are discussed in Chapter 9.
306 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients11
Situation
A suburban hospital known for its high inpatient satisfaction survey results noted a less than desired
overall score with the following survey question: “How well was your pain controlled?” The nursing
staff from the orthopedic department investigated pain control on their units by drilling down further
into the data. Through data analysis and qualitative feedback form both patients and nurses they
determined that hip/knee replacement patients would be an ideal target group for improved pain
management.
Two nurses who worked regularly with these patients decided to investigate the possible causes
of dissatisfaction. They enlisted the assistance of an internal quality measurement consultant and
decided to start tracking the effectiveness of the pain management process. They designed a study to
determine whether pain was being controlled effectively after the postsurgical epidural and/or patient-
controlled analgesia (PCA) pump were removed. Their concern was that once these more automated
systems for the administration of pain medication were removed, gaps were occurring in the manual
administration of pain medications. They believed that these gaps led to (1) lengthy delays in the
administration of pain medications, (2) the undesirable “peak and trough” pain management cycle, and
(3) increased dissatisfaction on the part of patients and the nurses.

Discussion
First the team randomly pulled 50 hip and knee replacement charts from the past 2 months and
evaluated the pain medication administration process. For all 50 patients, the times that they
were administered pain medications were recorded, and a bar graph showing the number of
medications given by hour of the day was prepared (FIGURE 10-35). This bar graph reveals that
the most popular time for administering pain medications was between 9 and 10 a.m. Note that
there is a ramping-up effect from midnight to 9 a.m. then there is a decline in the frequency with
spikes at 2 p.m. and 10 p.m. The diagram reinforced the nurses’ beliefs that there was considerable
variation in the amount of time between the manual administration of medications and that some
patients were experiencing unacceptable delays in receiving pain medications. This suggested
that the unit took a patient-reactive approach to pain management rather than a more desired
nursing-proactive approach.
Next they developed an indicator to evaluate the effectiveness of the pain management
program. They used a modified version of the Indicator Development Worksheet described in Chapter
4 to specify the key outcome indicator and the related measurement plan (EXHIBIT 10-1). Using the
universal pain scale, the nurses asked the patients to evaluate their levels of pain two times during
the day and evening shifts and at least one time during the night shift. This provided a minimum
of five pain assessments for each patient each day. Even though the universal pain scale is based
on a 0–10 “scale” for the evaluation of pain, the data are basically attributes in nature, not variables.
This is because the pain scale is not a true measurement scale that has equal-appearing intervals
between each number on the scale. The pain scale is a very subjective scale that basically produces
a rank order distribution rather than a true interval scale such as on a ruler where there are equal
appearing intervals between each mark on the ruler. Time and weight are further examples of a true
measurement scale. On the pain “scale,” one patient’s pain level of 5 may be a 7 for another patient
or a 3 for yet another. Therefore, the appropriate way to analyze pain scale data is by computing

(continues)
 Applying Quality Measurement Principles 307

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

Total counts

27

25

20
20 19
18

16

15
Count

13
12 12

10 10
10
8 8 8
7
6 6
5 5 5 5
5
4 4

2
1 1
12:00.00 A.M.
1:00.00 A.M.
1:30.00 A.M.
2:00.00 A.M.
3:00.00 A.M.
4:00.00 A.M.
5:00.00 A.M.
6:00.00 A.M.
7:00.00 A.M.
8:00.00 A.M.
9:00.00 A.M.
10:00.00 A.M.
11:00.00 A.M.
12:00.00 P.M.
1:00.00 P.M.
2:00.00 P.M.
3:00.00 P.M.
4:00.00 P.M.
5:00.00 P.M.
6:00.00 P.M.
7:00.00 P.M.
8:00.00 P.M.
9:00.00 P.M.
10:00.00 P.M.
11:00.00 P.M.

FIGURE 10-35 Number of pain medications administered by hour of the day

a percentage of patients selecting a particular pain level, not by computing averages or standard
deviations. The only decision that needs to be made is where to divide the scale in order to compute
the percentages. The nurses reached consensus that an unacceptable pain level would be any
evaluation that was greater than a 4 on the universal pain scale.

(continues)
308 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

EXHIBIT 10-1 Indicator development worksheet (Ortho Pain Management Team)

1) What Key Quality Characteristic 2) What is the specific name of this
(KQC) does this indicator measure? (e.g., indicator?
turnaround time, answering telephones (e.g., Radiology Turnaround time,
within 3 rings) percentage is missing charts)
Prompt relief from pain for hip & knee patients. “Quality of Pain Control in post-op Patients“
Maintain low level of pain. Acknowledgement
of pain.
3) What is the rationale for this indicator? (Why has this indicator been selected? What is the
purpose of the indicator?)
Keeping pain levels low for hip and knee replacement patients is an essential part of the care
process. Low levels of pain will also increase the patient’s satisfaction with their treatment
and will be demonstrated by increased scores on the pain-related question on our patient
satisfaction survey. Observations have also indicated the lack of a standardized approach
to the dispensing of oral pain medications to post-op patients. Oftentimes patients have to
complaint of pain, then wait for medication, and finally wait for medication to take effect. This
often results in dramatic shifts of pain levels.
4) Data source collection method: Collection Frequency and Duration:
◽ Medical Record ◾ Patient Satisfaction Collected: ◾ Daily ◽  Weekly ◽  Monthly
Reports Reported: after baseline and after
◽  Data Logs   ◽  System Reports, intervention
please specify Duration: 07/08/02 to 15–20 patients
◾ Other: recording of pain level on laminated
form by nursing staff
5) Operational definition: Remember to include the full definition with all inclusions and
exclusions, specify all required data elements (e.g., patient types, financial class, data dictionary
elements, DRGs, codes, clinical specialty. Given a solid Op Def, 10 different people measuring the
same thing should arrive at identical results.)
Quality Pain Control is defined as: Using a population of total hip/knee replacement patients
concurrently, nursing will use the universal pain scale to record patients pain number two (2)
times per shift (standard shift is 7–3, 3–11) and at least one (1) time per night shift (11–7). RN
on each shift will record on laminated collection tool the actual time and pain number when
assessed utilizing the universal pain scale (0–10). At the end of night shift, start a new form each
day. Place completed form in the 2 North nursing office.
The lead ortho nurses (MS and SH) will be accountable for entering data from the laminated
sheets and compiling additional information.
BP, our improvement advisor (IA), will be responsible for generating control charts. Order on
control charts will be based on admit dates (date of surgery). P charts will be used along with a
Pareto chart.

(continues)
 Applying Quality Measurement Principles 309

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

6) Numerator Statement: (if applicable, 7) Denominator statement: (if applicable,

i.e. number of C-Sections) i.e. number of births)
number of pain measurements over 4 number of pain measurements obtained

8) What is this indicator 9) Baseline: 10) Goal:

measuring? *see attached charts a reduction in pain levels
◽ Rate ◽ Days ◽ Time responses over 4 as demonstrated
◾ Percentage by control chart mean and upper
◽ Other and LCL *see charts

11) This indicator will satisfy the following objective(s):

◽ Physician Partnership ◾ Customer Satisfaction ◽ Regulatory Requirement
◽ Culture Transformation ◾ Clinical Excellence ◽ Risk Management
◽ Value Enhancement ◽ Operational Excellence ◽ Patient Safety
◽ Market Development ◽ Cost Reduction

12) Which of our 5 Pillars of Organizational Excellence does this indicator effect?
◾ Service ◽ Financial ◽ Growth ◾ Quality ◽ People

The baseline data are shown in FIGURE 10-36. Fifteen patients were initially assessed using the pain
scale and any response that was greater than 4 was defined as a not acceptable level of pain. The data
used to create the p-chart are shown in the table at the top of the p-chart. Note that the row labeled
“inspected” represents the denominator for the percentage calculation and the row labeled “count”
contains a particular patient’s pain scale ratings that were greater than 4. For example, patient 1 was
asked 31 times to provide a pain scale rating. Out of 31 assessments 21 of these were greater than 4
producing a 67.74%. Three conclusions can be derived from Figure 10-36: (1) the baseline data contain
two special causes (patients 1 and 9) both of which exceeded their UCL; (2) there is a fair amount of
variation from patient to patient (i.e., the average control limits go from a LCL = 0.82% to the UCL =
50.69%); and, (3) if nothing is done to improve this process the nursing staff and physicians can expect
about 25% of their hip and knee surgery patients to be experiencing serious levels of pain. This process
is basically unstable and unpredictable because of the two special causes, which essentially confirmed
what the nurses thought was occurring with this process. The data also reinforced their belief that
improvements needed to be made. After evaluating the baseline data, the team:
■■ Conducted several inservice sessions for the nursing staff to reacquaint them with the importance
of pain management and the hospital’s commitment to managing it consistently and effectively.
■■ Introduced pain management concepts in their established presurgical education program for
hip and knee patients. This consisted of a presurgery orientation program as well as bedside
education after the surgery. The content of these sessions helped the patients understand the

(continues)
310 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

Inspected 31 21 22 23 31 26 17 26 31 29 25 24 26 25 24
Count 21 0 6 7 4 4 2 5 15 3 2 7 13 4 2
Percent 67.74 0.00 27.27 30.43 12.90 15.38 11.76 19.23 48.39 10.34 8.00 29.17 50.00 16.00 8.33

Beyond limits
70

60

UCL = 50.69

50
Percentage of ratings over 4

40

30

Mean = 24.93

20

10

0
LCL = 0.82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Baseline Patient number Baseline

FIGURE 10-36 Baseline percentage of hip and knee replacement patients indicating that pain was
greater than 4 on a scale of 0–10 (p-chart)

different methods for managing pain and the different routes for the delivery of medications
(e.g., epidural, PCA, oral medications, and injections).
■■ Created posters that were placed on the units informing the public of the hospital’s
commitment to effective pain management practices and outcomes.
■■ Instituted a Pain Interventions Pledge with the nurses (EXHIBIT 10-2). This was a direct effort not
only to get the nurses to think about the pain management process and its components but

(continues)
 Applying Quality Measurement Principles 311

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

EXHIBIT 10-2 Pain intervention pledge for hip and knee replacement patients
Pain control is very important to our patients and a quality goal that is very important to the 2
North Nursing Staff. Therefore, as an RN on the 2 North Team, I agree to change my practice in
relation to pain control, specifically for Total Joint Replacement Patients:
1.Pain Assessment will be done on my patients at least every 4 hours
2.Be PRO-ACTIVE regarding pain – asking about pain levels on hourly checks and offering
pain medications before the patient has to ask
3. Total joint patients will receive pain meds around the clock (at the minimum at bedtime
and at awaking in the morning) so that therapy and pain control can be obtained
4. PM staff will medicate for pain at the hour of sleep. PM staff will also script: “Would you
like to be awakened during the night when your next dose of pain medication is due?”
If patient does not receive pain meds during the night, pain med will be given when the
patient is awaked in the morning or at the very latest at the breakfast meal. This practice will
increase the ability for our patients to be able to participate More fully in Physical Therapy
sessions
5. Education regarding around the clock pain med administration will be done with my
patients
_____________________________________
Signature of RN
_____________________________________
Date

also emphasize the importance of making a personal commitment to “change my practice in

relation to pain control, specifically in the total joint patients.” Each nurse on the unit was asked
to sign the pledge as a demonstration of their endorsement of this initiative.
Once the interventions were in place and operational, additional data were collected to see
whether the percentage of patients indicating that their pain levels were over 4 had declined.
FIGURE 10-37 shows the baseline period (patients 1–15) and the patient assessments of pain after the
intervention (data points 16–41). The control limits for the baseline period were “frozen” and extended
across the postintervention period. This was done in order to see whether the interventions were able
to produce a special cause (e.g., a downward trend or possibly a shift in the process).12 FIGURE 10-38
does not reveal that the process has been improved. There are two special causes (3-sigma violations)
that needed to be addressed in the new data (i.e., patients 29 and 34). These two patients exceeded the
UCL and had much higher levels of self-assessed pain than the rest of the other patients. Whenever
there are data points that are markedly different from the rest of the data, it is useful to ask:
■■ Was there a data entry error?
■■ Is this a stratification problem (i.e., are these two patients different from the rest of the patients
in the study)?
The team did not uncover any data entry errors. What the nurses did discover, however, was that
these two patients were unlike the rest of the patients.13 These two patients, therefore, were removed

(continues)
312 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

Inspected 31 21 22 23 31 26 17 26 31 29 25 24 26 25 24 24 21 31 31 28 30 22 20 30 45 38 45 46 42 25 55 47 25 47 43 26 21 31 36 42 28

Count 21 0 6 7 4 4 2 5 15 3 2 7 13 4 2 11 5 9 12 4 12 2 3 9 19 4 8 20 24 2 4 8 6 33 7 5 1 1 7 9 6

Percent 67.74 0.00 27.27 30.43 12.90 15.38 11.76 19.23 48.39 10.34 8.00 29.17 50.00 16.00 8.33 45.83 23.81 29.03 38.71 14.29 40.00 9.09 15.00 30.00 42.22 10.53 17.78 43.48 57.14 8.00 7.27 17.02 24.00 70.21 16.28 19.23 4.76 3.23 19.44 21.43 21.43

Beyond limits
7 below centerline
70
Pain contract intervention after 15th patient

60
Percentage of ratings over 4

UCL = 50.69
50

40

30
Mean = 24.93

20

10

0
LCL = 0.82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Baseline Patient number Follow-up

FIGURE 10-37 Pain assessments for hip and knee patients before and after the intervention (control
limits are based on patients 1–15, the baseline period and extended) p-chart

from the analysis and the p-chart was recalculated. The result is shown in Figure 10-38. Notice that in
this figure there is a period after the intervention when the percentage of patients who indicated that
their pain was greater than 4 is very similar to the baseline (i.e., compare patients 1–15 with patients
16–28 and you will see similar patterns of variation). Then an interesting thing is observed. A special
cause (i.e., a downward shift in the data) is detected from patient 29 to 36 (i.e., eight consecutive data
points initially below the frozen baseline mean of 24.93%. Also notice that the next three patients
(37–39) had their percentages of pain ratings also stay below the baseline mean suggesting that the
improvement was being sustained. You would not expect to see that many data points below the
baseline centerline if this was merely a random process.
Why did it take the process so long to respond to the interventions? Is this potentially another
data entry problem, or could it be that the last 11 patients did not fit the profile of the previous
patients? As the team discussed these questions, they arrived at several conclusions:
■■ The data were clean, and patients 29–39 were very similar to those in the previous time periods.
■■ They had instituted four new interventions simultaneously. Typically you would introduce one
change at a time and evaluate the impact of that single improvement strategy. This is the nature

(continues)
 Applying Quality Measurement Principles 313

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

Inspected 31 21 22 23 31 26 17 26 31 29 25 24 26 25 24 24 21 31 31 28 30 22 20 30 45 38 45 46 25 55 47 25 43 26 21 31 36 42 28

Count 21 0 6 7 4 4 2 5 15 3 2 7 13 4 2 11 5 9 12 4 12 2 3 9 19 4 8 20 2 4 8 6 7 5 1 1 7 9 6

Percent 67.74 0.00 27.27 30.43 12.90 15.38 11.76 19.23 48.39 10.34 8.00 29.17 50.00 16.00 8.33 45,83 23.81 29.03 38.71 14.29 40.00 9.09 15.00 30.00 42.22 10.53 17.78 43.48 8.00 7.27 17.02 24.00 16.28 19.23 4.76 3.23 19.44 21.43 21.43

Beyond limits
7 below centerline
70
Pain contract intervention after 15th patient

60
Percentage of ratings over 4

UCL = 50.69
50

40

30
Mean = 24.93

20

10

0
LCL = 0.82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Baseline Patient number Follow-up

FIGURE 10-38 Pain assessments for hip and knee patients before and after the intervention and after
removing patients 29 and 34 (control limits are based on patients 1–15, the baseline period and
extended) p-chart

of the Plan–Do–Study–Act (PDSA) cycle. When you implement several changes simultaneously,
however, it is hard to determine which intervention or combination of interventions caused the
change in the process.14
■■ Several of the changes required behavioral adjustments on the part of the nurses and the
patients. Because behavioral changes do not happen instantaneously, there is probably a lag
effect when it comes to observing the impact of the interventions.
■■ Data points 16 through 28 can be regarded as a “learning curve” and a transition phase after the
intervention for both the patients and the nursing staff.
■■ Therefore, the chart should be partitioned into three segments to reflect the baseline, the
transition period, and the current process.
FIGURE 10-39 shows the process segmented into three defined segments or phases. The left side
of the chart (patients 1–15) shows the baseline with an average of about 25% of the patients reporting

(continues)
314 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #14: Pain Management for Hip and Knee

Replacement Patients (continued)

Inspected 31 21 22 23 31 26 17 26 31 29 25 24 26 25 24 24 21 31 31 28 30 22 20 30 45 38 45 46 25 55 47 25 43 26 21 31 36 42 28

Count 21 0 6 7 4 4 2 5 15 3 2 7 13 4 2 11 5 9 12 4 12 2 3 9 19 4 8 20 2 4 8 6 7 5 1 1 7 9 6

Percent 67.74 0.00 27.27 30.43 12.90 15.38 11.76 19.23 48.39 10.34 8.00 29.17 50.00 16.00 8.33 45.83 23.81 29.03 38.71 14.29 40.00 9.09 15.00 30.00 42.22 10.53 17.78 43.83 8.00 7.27 17.02 24.00 16.28 19.23 4.76 3.23 19.44 21.43 21.43

Beyond limits
70
Pain contract intervention after 15th patient

60

UCL = 52.85
Percentage of ratings over 4

UCL = 50.69
50

40
UCL = 32.81

30 Mean = 28.71

Mean = 24.93

20
Mean = 14.78

10

0
LCL = 0.82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Baseline Patient number Follow-up

FIGURE 10-39 Pain assessments for hip and knee patients before and after the intervention and after
removing patients 29 and 34 (baseline [patients 1-15], after Interventions [patients 16–28], and
follow-up [patients 29–39], all have separate control limits) p-chart

pain levels greater than 4. The middle section of the chart (patients 16–28) reflects the transition or
learning period. The control limits during this period are not very different from the baseline, but
the centerline is actually a little higher (29%) than the baseline. This is not uncommon for a period of
transition, because people are adapting to new ways of behaving, which frequently has a tendency to
create more variation rather than less. These conditions are compounded when multiple interventions
are implemented simultaneously. In this case, staff and patients are not sure of the steps in the new
process or the expected behaviors which can lead to increased variation and/or instability.
Finally, the right side of Figure 10-39 (patients 29–39) reveals how the process has shifted to a
new (and more acceptable) level of performance after the transition. During this phase the patients
and staff appear to be settling into the new process. The real test of improvement will be if the average
percentage of patients evaluating their pain levels greater than 4 stays at or below the new process
average of about 15%. Also, is it possible to get the percentage to be even lower? The team will have to
evaluate its aim and decide whether further improvement is possible.
 Applying Quality Measurement Principles 315

CASE STUDY #15: Hospice/911 Paramedic System

Partnership to Improve Care15
Situation
When people with a terminal illness are enrolled in hospice care the goal is to provide them with
comfort care and dignity in the final stages of life. Patients and their families are instructed during
the hospice intake process to call their hospice provider’s 24-hour phone number with any issues,
concerns, or emergencies. They are told not to call 911. This is so that the patient’s wishes for a “good
death” with minimal suffering and intervention can be honored. Despite clear instructions, 911 is
activated for hospice patients all over the United States often by visiting relatives or concerned friends,
who were not part of the hospice intake process.
Once 911 paramedics, who are trained and equipped for life-saving care, are involved
the patient is often transported to a hospital ED by ambulance. The movement involved with
transport often aggravates the patient’s levels of pain and nausea. Most hospital emergency
teams are focused on life saving and disease treatment. Therefore, the comfort care and minimal
intervention approaches of hospice workers are frequently in conflict with what the ED is
designed to accomplish. Admission to the hospital also causes problems with hospice insurance
benefits, resulting in additional suffering and stress for patients and their families.

Discussion
In the fall of 2014, hospice providers and the 911 paramedic system in Ventura County, California,
began a project to address this issue and improve care for hospice patients who had 911 activated
on their behalf. Baseline data indicated that between 250 and 300 hospice patients had 911
activated for them in Ventura County each year with 80% of these patients being transported to the
hospital ED.
The aim of the project was to decrease the percentage of 911 hospice patients transported by
50% within the first year. The primary theory for improvement was that if paramedics were integrated
more thoroughly into the hospice system of care that more appropriate decisions could be made
regarding the need for transport.
The first PDSA in the project involved having a few paramedics ride along with hospice nurses
as they made their rounds visiting patients, entered new patients into the hospice program, and
mediated conflicts between family members of various hospice patients. The prediction was that
paramedics could see themselves acting as part of the hospice team and this was confirmed as valid
by the paramedics involved.
The second PDSA cycle involved having paramedics attend the weekly case conferences of two
different hospice organizations. These conferences led by the hospice’s physician medical director
reviewed situations that came up with their patients the previous week through the lens of how a
well-trained paramedic might support the patient’s care and blend with the hospice team. The hospice
medical directors, case managers, and nurses who participated in these case conferences all agreed
that having paramedics function as members of their team would likely improve care for patients and
keep them within the hospice program.
For the third PDSA two of the largest hospice providers, Assisted Hospice and Livingston
Memorial Hospice, trained 14 paramedic supervisors from American Medical Response Ventura, Gold
Coast Ambulance, and LifeLine Medical Transportation using a curriculum for new hospice nurses.
In order to support this initiative the state of California Emergency Medical Services (EMS) Authority

(continues)
316 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #15: Hospice/911 Paramedic System

Partnership to Improve Care (continued)

100.00%

90.00%

80.00%
CL = 80.3%
Percent Transported by EMS

70.00%

60.00%

50.00%

40.00%
CL = 37.1%
30.00%

20.00%

10.00%

0.00%
Jan 15

Feb 15

Mar 15

Apr 15

May 15

Jun 15

July 15

Aug 15

Sep 15

Oct 15

Nov 15

Dec 15

Jan 16

Feb 16

Mar 16

Apr 16

May 16

Jun 16

Jul 16

Aug 16
Month

FIGURE 10-40 Percentage of Ventura 911 hospice patient calls transported to the emergency room (p-chart)

granted a temporary order to increase the state scope of paramedic practice and the local institutional
review board (IRB) approved the project.
For hospice patients this meant that medical control of paramedic practice shifted from the
county EMS medical director to the hospice physician for the patient’s hospice. When 911 was
activated for a hospice patient one of the specially trained paramedic supervisors would be dispatched
along with a paramedic ambulance and the first responders from the fire department. Rather than
taking their clinical directives from the county EMS medical director, they would follow the direction
of the patient’s hospice physician. The results of putting this new initiative in place are shown in
FIGURE 10-40.
The baseline data shown on the left side of Figure 10-40 show that a little over 80% of the 911
calls received were transported to the emergency room. After the Community Paramedic Project
described in the previous paragraph was tested, however, the new average dropped to 37%. This new
average represented a 54% reduction in the baseline average. The original aim was to reduce this
number by 50%. Although improvement in the process is noted, the challenge to the team is to work
to standardize the new procedures and not have the process drift back to the original baseline levels.
This concern is based on looking at the right side of Figure 10-40 where the data are starting to drift
upwards. Specifically, the last three data points are above the new average of 37.1%. If this pattern
continues and the next five data points remain above the CL then a shift in the wrong direction will be
detected. Questions the team discussed included:
 Applying Quality Measurement Principles 317

CASE STUDY #15: Hospice/911 Paramedic System

Partnership to Improve Care (continued)

■■ Do we have a reliable design for the new Community Paramedic Project?

■■ Has everyone been trained properly in the new way of addressing 911 calls for hospice patients?
■■ Has the severity of the patient population or the mix of conditions and diagnoses changed
recently?
As the team discussed these issues with their IA they arrived at the following findings:
■■ The upward drift seemed to be caused by a few of the trained community paramedics getting
promoted or moving on to take other positions, which caused a staffing issue, which in turn
caused some long response times.
■■ With the longer response times the first responders from the fire department and the paramedic
ambulance crew were more likely to initiate transport before the arrival of the community
paramedic.
■■ The state gave the participating paramedic organizations permission to train additional
community paramedics (initially they were limited to the initial group). Class will start later this
month.
With the help from their IA Mr. Mike Taigman, the team laid out an ongoing data tracking process. They
agreed that further testing and data analysis were needed before declaring a complete victory. Initial
results, however, do look promising.

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients16
Situation
Two key topics drive the strategic dashboard of the East London National Health Service (NHS)
Foundation Trust (ELFT): (1) reducing harm and (2) right care, right place, right time. Four aspects of
care delivery were identified as the major dimensions of these two key topics:
■■ Safety
■■ Clinical effectiveness
■■ Patient experience
■■ Staff experience
ELFT has over 160 teams working on improvement projects that fall into one of these four aspects of
care delivery. The team charged with improving access to community services for mental health and
community health patients is one of the first improvement teams that was chartered by the Trust. This
team has a fairly broad range of responsibilities including the delivery of community-based mental
health services as well as community health in general, which encompasses all aspects of primary

(continues)
318 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

care delivered in the community except general practice (e.g., it includes cardiac rehab, physiotherapy,
district nursing, and home health visiting).
This topic is central to ELFT’s work for several reasons:
■■ Long waits to be seen after being referred from primary care create backups in the system and
delays in providing essential care to the service users.
■■ Delayed access is a major source of poor patient experience, which increases complaints.
Comparative data from the NHS show ELFT as having long waits when compared to other
providers across England.
■■ This topic was almost too “wicked” of a problem to be talked about at ELFT until QI became the
Trust’s central business strategy.
The responses from many staff typically centered on two reasons why access to community services
could not be improved: (1) they were short staffed and their budgets had been reduced (i.e., they were
approaching the issue as a first-order change with the solution being give us more staff and more
money), and (2) delays in access are just part of the way the NHS works.
The team faced several initial challenges. First, they did not know the magnitude or extent of
delays in gaining access to the system. Second, they had to address the attitudinal and behavioral issues
that allowed delays in gaining access to community services to be part of the staff’s general mindset.
Finally, they needed to understand how this issue fit into the Trust’s overall strategy to reduce harm
and provide the right care, right place, and right time. Through ELFT’s QI strategy, however, the team
began to address these three challenges and realized that they were empowered to tackle this issue but
with no extra staff or increased budget. What they did have, however, was support from senior leaders
and the board to improve access and the skills and tools of quality improvement (QI) through a large
capability building program delivered by ELFT and IHI. The access team’s journey is described next.

Discussion
After assembling subject matter experts from the various service lines that dealt with the access issue,
the QI team coach met with the team to start laying out their journey. The first thing they did was to
develop an aim statement specifying how they would work on their theory of improvement and the
methods they would employ. The team’s aim statement is shown in BOX 10-1.
Because of the complexity of the access issue the team took the advice of their QI coach and
devised a “learning system” that would support their work. They established three key components of
this learning system: (1) Webex learning sets, (2) individual project support structures and, (3) measures.
The details of this learning system are shown in FIGURE 10-41. The time spent discussing how the
team would learn and approach its work was well spent. It allowed the team members and the QI
coach to engage in a rich dialogue about the nature of the challenges that lie ahead and their plan for
addressing them. Too many teams begin their quality journeys without a roadmap or plan. This leads to
the lack of direction, wasted time, and frequent conflicts among team members because they are not
clear about their purpose and how they plan to achieve their aim.
The Webex sessions were held every 6–8 weeks and provide a time-efficient way to bring team
members together virtually. This not only saved time and resources but also helped prepare the team
members for the face-to-face meetings. During the Webex sessions external case studies and best
practices from other NHS Trust and IHI’s work in other countries on the issue of access were reviewed,
current and future PDSA tests were reviewed and future tests planned. One of the new ideas that

(continues)
 Applying Quality Measurement Principles 319

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

BOX 10-1 Aim statement for the improving access team

Aim:
■■ Improving access to services for new patients by increasing uptake, reducing wait times or

reducing DNAs (did not attend) according to locally set targets by April 2017
Theory:
■■ These three areas affect the start of the care pathway and are interrelated. Improvement in

one area may adversely affect the others.

Contract:
■■ Project team appoints a project lead.
■■ Project team meets regularly as a team and invites QI lead.
■■ At least one member of the team attends collaborative learning sets.
■■ Copy QI team into updates to sponsor/coach
■■ Use of shared folder as primary location for project documentation
■■ Use of the three core measure for the collaborative as a minimum

Method:
■■ Project teams that are part of the learning system will have at least one of these measures as their

main outcome measure and the rest as a balancing measure in addition to their own measures
■■ Teams develop and test change ideas in their own areas and are encouraged to collaborate

with other project teams in the learning system

■■ A QI lead regularly meets with the project teams to help them apply the QI tools and

methodology to advance their work and to act as a link across similar projects
■■ Change ideas that are shown to be effective at reducing wait times and DNAs will be

evaluated by the learning system members and cascaded to appropriate services

■■ Change ideas that are effective within the learning system will be cascaded to the rest of the

organization

emerged from this sessions was the need for a monthly newsletter, posted on the ELFT’s intranet,
which provided:
■■ Information on what each team is testing or planning to test
■■ Project team interviews (internal case study)
■■ Information about a QI tool (e.g., flowcharting, data collection or a particular Shewhart chart)
and other internal and external resources that could be of assistance to the improvement teams
■■ Dates, times and locations of upcoming learning sets.
As the access team gained increased knowledge about the SOI they realized that the sequence
of improvement started with ideas that can be put into PDSA testing, which leads over time to
implementation of reliable changes that have been shown to work and finally to spreading the
improvement work to other locations. To share this piece of learning with other teams at ELFT the

(continues)
320 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

Improving Access Learning System

WebEx Learning Sets

12 Projects

6 Wait Times 6 Increasing Uptake

Projects Projects

• Sharing progress • Core measures & DD • Dashborad

• Case Studies • Change Ideas & Tests • Newsletter
Measures
Facilited by: Director of Ops, 2x QI Leads, 2x Senior IAS

Wait Times - Average days from

referral accepted to first face-to-face
Individual Project Support contact

% DNA - Do Not Attends (DNAs)

Coaching Testing Dashboard Alignment before first face-to-face contact /
total number of appointments
booked (excluding cancellations)

Documen- New Referrals - Total number of

Tools Research Strategy referrals received from external
tation
referrers (non-ELFT)

Supported by: 2x QI Leads Supported by: QI Data Analyst

FIGURE 10-41 ELFT Improving Access learning system

access team developed FIGURE 10-42 that laid out their thoughts on a scale-up and spread strategy for
improving access to care. Even though the team was still in a testing phase the discussion with their
QI coach about their ultimate destination provided a vision of the end game and helped the team
understand more fully the nature of their journey.
The access team chartered 12 project teams organized under two key headings: (1) reducing
waiting times and (2) increasing uptake. A summary of these teams and the current changes they are
each testing is shown in TABLE 10-2. Each of the 12 teams developed their own aim statement, created
flowcharts to detail the steps involved in each respective process, and then created measures and data
collection plans to understand the variation inherent in each of the respective processes they were
working to improve.
Three indicators formed the core of all 12 teams’ measurement strategies:
■■ Waiting Time—Average days from referral accepted to first face-to-face contact. This measure
looks at external referrals (i.e., non-ELFT referrals from GPs, primary care, or other Trusts)

(continues)
 Applying Quality Measurement Principles 321

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

Scale-Up and Spread

Change Change
Ideas Ideas
Designing and Testing
multiple change ideas by
Change teams within the
Ideas learning system

Degree of belief

Up-scale & Test Bundle

within the learning
system
Bundle

Degree of belief

Recruit & Spread

the bundle as a menu
of options across the
organization

FIGURE 10-42 ELFT scale-up and spread strategy for improving access to care

■■ Number of Referrals—Total number of referrals received from external referrers. This measure
looks at external referrals (i.e., non-ELFT referrals from GPs, primary care, or other Trusts)
■■ Percentage of DNAs—DNAs before first face-to-face contact divided by total number of
appointments booked (excluding cancellations). This measure looks at external referrals (i.e.,
non-ELFT referrals from GPs, primary care, or other Trusts)
Data on these three indicators were analyzed with Shewhart charts. Totals for the system as well as
individual teams and borough locations were developed. When appropriate the dashboard indicators

(continues)
322 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

TABLE 10-2 ELFT change ideas being tested by the Improving Access Team
Current Tests

Reducing Waiting Times Projects

Project Team Currently Testing

NH Psychological Therapies Services After several change ideas, running session in August to develop
new change ideas

CH Psychological Therapies Service Text message reminders and service user focus group

Memory Clinics in East London Currently making technical changes to have uniform processes
across East London memory clinics

Community CAMHS Tower Hamlets Offering two sessions to some new referrals before final decision

CH Adult Community Mental Health Telephone referral and screening; Text message reminders; Posting
Access Services appointment letters directly

TH Psychological Therapies Service DNA policy statement on therapy contract; New referral form;
Mailshot to service users and carers to attend SU engagement event

Increasing Uptake Projects

Project Team Currently Testing

NH Psychological Therapies Services Text message reminders using web platform

Community Sexual and Reproductive Testing manual text messages sent immediately after appointment
Health booked, confirmation texts rather than reminders.

Clozapine team and EPCL team Data cleaning of RiO in preparation for iPlato. Testing EE messages
for one clinic.

Health Visiting, Sickle Cell &
Thalassaemia and Children & Young
People service, NUHT
Testing an emergency clinic slot to ensure that clients who are at
20 weeks or over gestation are seen within 48 hours. Completing a
DNA audit and planning to contact a small sample of service users
to gather qualitative information about why service users DNA
appointments

MSK Physio iPlato text message reminder system for new referrals and
follow-up appointments

TH Adult Community Mental Health Designing leaflet specific for Isle of Dogs GP and community teams
Services

(continues)
 Applying Quality Measurement Principles 323

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

were displayed as small multiples to maximize the amount of information being displayed (Tufte, 1983,
1990). FIGURES 10-43, 10-44, and 10-45 show the Shewhart charts for these three indicators aggregated
across all sites and all teams. You will notice that the charts have phased control limits and extensive

Average waiting time

Novemver 2016 -Baseline data

Average Waiting Time/Days

Average waiting time from referral to 1st face to face appt (Collaborative, 10/12 teams) - X-bar Chart
70
65 UCL
60.66
60
55 53.17
LCL 50.74
50
45
40 44.51
35
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16
27/07 03/09 05/01 2 new teams 3 team leave
Testing begins

Learning Learning Learning join collaborative

Set 3 Set 4 Set 7 collaborative

28/03 01/10 16/02 10/05

Learning Learning Learning Learning
Set 2 Set 5 Set 8 Set 9

25/11
Learning
Set 6

Average waiting time from Referral to 1st face to face appointment - I Chart
Child and Adolescent Mental Health Service (Tower Hamlets) Community Mental Health Teams (City and Hackney & Tower Hamlets)
80.0 80.0
Initial baseline Initial baseline
60.0 60.0
Decrease identified Decrease identified
Decrease identified
40.0 40.0 Decrease identified

20.0 20.0

0.0 0.0
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Psychological Therapy Service (City and Hackney, Newham & Tower Hamlets) MHCOP Memory Service (City and Hackney Newham & Tower Hamlets)
140.0 80.0
Initial baseline
120.0 Initial baseline
Decrease identified
100.0 60.0

80.0
40.0
60.0
40.0 20.0
20.0
0.0 0.0
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

FIGURE 10-43 ELFT Dashboard: average waiting time from referral to 1st face-to-face appointment (I charts)

(continues)
324 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

Referrals

Novemver 2016 -Baseline data

No. of referrals received (Collaborative, 10/12 teams) - I Chart

1700
No. of Referrals

1500
UCL 1,274.14
1300
1100 1,021.71 1,213.13

900
LCL
700
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16
16/02 27/07 03/09 05/01 2 new teams 3 team leave
Learning Learning Learning Learning join collaborative
Set 1 Set 3 Set 4 Set 7 collaborative
Testing begins

28/03 01/10 16/02 10/05

Learning Learning Learning Learning
Set 2 Set 5 Set 8 Set 9

25/11
Learning
Set 6

No. of Referrals Received - I Chart

Child and Adolescent Mental Health Service (Tower Hamlets) Community Mental Health Teams (City and Hackney & Tower Hamlets)
200 1000
Initial baseline increase identified
increase identified
800 721.38
150 126.36 increase identified 646.60
600 Initial baseline 556.50
100 414.21
400
50
200

0 0
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Psychological Therapy Service (City and Hackney, Newham & Tower Hamlets) MHCOP Memory Service (City and Hackney, Newham & Tower Hamlets)
350 250
300 Initial baseline Initial baseline
200
250 145.82
150
200
150 211.79 100
100
50
50
0 0
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

FIGURE 10-44 ELFT dashboard: number of referrals received (I chart)

(continues)
 Applying Quality Measurement Principles 325

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

Did Not Attend (DNA)

Novemver 2016 -Baseline data

% of 1st face to face appts DNAs (Collaborative, 10/12 teams) - P Chart

38%
36% UCL
34% 32.21%
32%
DNA / %

30%
28% LCL 25.23%
26%
24% 26.39%
22%
20%
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16
16/02 27/07 03/09 05/01 2 new teams 3 team leave
Learning Learning Learning Learning join collaborative
Set 1 Set 3 Set 4 Set 7 collaborative

28/03 01/10 16/02 10/05

Testing begins

Learning Learning Learning Learning

Set 2 Set 5 Set 8 Set 9

25/11
Learning
Set 6

% of first appointment non-attendance - I Chart

Child and Adolescent Mental Health Service (Tower Hamlets) Community Mental Health Teams (City and Hackney & Tower Hamlets)
50.0% 60.0%
Initial baseline
Initial baseline
40.0% Decrease identified
40.0% Decrease identified
30.0%

20.0%
20.0%
10.0%

0.0% 0.0%
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Psychological Therapy Service (City and Hackney, Newham & Tower Hamlets) MHCOP Memory Service (City and Hackney, Newham & Tower Hamlets)
50.0% 40.0%
Initial baseline
Initial baseline
40.0%
30.0%
30.0%
20.0%
20.0%
10.0%
10.0%

0.0% 0.0%
Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

Jan-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Jan-16
Feb-16
Mar-16
Apr-16
May-16
Jun-16
Jul-16
Aug-16
Sep-16
Oct-16

FIGURE 10-45 ELFT Dashboard: % of first appointment non-attendance (I chart)

(continues)
326 Chapter 10 Applying Quality Measurement Principles

CASE STUDY #16: Improving Access to Community Services

for Mental Health and Community Health Patients (continued)

annotations to highlight the changes that were tested and their and the impacts of these changes.
Each figure shows the aggregated chart on the top and a series of small multiples underneath the
aggregated chart showing the various teams and their locations. When you study the charts you will
see that wait times have gone down, whereas the number of referrals have gone up. This has been
accompanied with a decline in the percentage of DNAs.
This case study demonstrates nearly all the principles that I have discussed in this book. It
started out with a concern for the VOC, specifically that staff observed and service users complained
about long wait and poor referral processes. It then moved to organizing the access team, building
an aim statement, understanding the learning system they were trying to build, defining indicators
that would capture the variation in the key outcomes, and organizing a data collection plan. Once
the data were collected, they proceeded to analyze the three outcome measures on appropriate
Shewhart charts (X-bar and S, XmR, and p-charts), annotated the charts to show when they ran
PDSA tests and then interpreted the charts for common and special causes of variation. Finally, they
tested their improvement strategies against a baseline to see whether their interventions made a
difference. At first, they were a little discouraged because the improvement did not happen right
away. But they soon realized that process and behavioral changes do not occur overnight. This
case study also helps to explain why ELFT has received numerous awards over the past year for its
application of QI to daily work. It is not an impossible journey but one that does require knowledge,
skills, the will to change, ideas on how to make improvements, the ability to execute improvement
strategies, and constancy of purpose.

Notes
1. I want to acknowledge Dr. Charles Derus,
of Advocate Dreyer Medical Clinic for
tutoring me in the clinical aspects of this
case study. His technical consultation is
very much appreciated.
2. For more information on issues related
to central line insertion and maintenance
readers can consult the IHI’s website where
we have extensive information on the
central line bundle and infection issues.
The link is http://www.ihi.org/sites/search
/pages/results.aspx?k=central+line+bundle.
3. Mr. William Peters, IHI IA, called one
day and posed this question on sampling
options. Our conversation prompted me
to develop this case study. I thank him
for the idea.
4. Most control charting software will provide
you with the option for presenting the
data as a proportion or as a percentage.
Remember, to convert a proportion to a
percentage you merely multiply the propor-
tion by 100. Thus the average proportion
of nosocomial PUs in this figure is 0.241,
which translates to 24.1% (0.241 × 100).
5. Like many aspects of health care, this is
one of those topics that most people do
not think much about on a regular basis.
It can have profound implications not
only for process efficiency and resource
allocation but also, and most important,
for patient safety. If you have not looked at
this issue recently I would encourage you
to at least investigate how often FS occurs.
6. This is a very personal case study because
it tells the tale of my wife’s breast cancer
journey. My objective in sharing this
story is to help highlight the opportunity
to apply statistical thinking to the care
of individual patients and not just to
larger groups or aggregations. My wife
was diagnosed in April 2001 and is still
cancer free today.
7. I want to thank Carol Burke, RN, Nancy
Longino, RN, and Julie Marre, RN, from
Notes 327
Advocate Good Samaritan Hospital in
Downers Grove, Illinois, for sharing
this case study with me and helping
to develop the measurement plan and
indicators. The potential seriousness
of this situation requires a passion and
commitment to break down barriers
and make improvements. These women
have met this challenge with great energy
and spirit. They have spent numerous
hours with me thinking through the
logic of how the levels of stratification
influence the various outcomes and
spent after-work hours assembling data,
analyzing control charts, and developing
improvement strategies. I thank them
for their dedication and willingness to
share their story.
8. I also suggested to the nursing team that
they should consider tracking a new
indicator—the percentage of pregnant
women with positive GBS who deliver
in less than 4 hours after arriving at the
hospital. This indicator will help them
understand if the volume of women
who do not meet the criteria is actually
increasing and potentially influencing
the overall performance of this process.
The last recommendation I had was that
they should develop a Pareto diagram to
document the reasons why the antibiot-
ics were not delivered according to the
protocol for those women arriving at the
hospital 4 or more hours prior to delivery.
The Pareto diagram will enable the team
to focus on the major reason(s) why the
protocol is not working for the targeted
population.
9. If the conditions creating an equal area of
opportunity for a complaint changed (i.e.,
there were wide swings in the daily volume
of patients and/or the severity and mix
of patients changed constantly), then the
chart of preference would be the u-chart.
The u-chart would provide a complaint
rate where we would have the number
of complaints as the numerator and the
328 Chapter 10 Applying Quality Measurement Principles
number of visits as the denominator. This
would produce a complaint rate (i.e., the
number of complaints per 100 or 1,000
patient visits).
10. I had the pleasure of working with ­Bonnie
Schleder, critical care clinical nurse specialist,
and Karen Stott, ICU registered nurse, on
this project. I also want to thank the ICU
staff for their support of this initiative,
Eileen Yaeger, clinical epidemiologist,
and Lori Pinzon, QI specialist, for data
analysis and technical support. The results
of this improvement effort were initially
published in Nursing Management (fall
2003) and served as a foundation for
other clinical groups to begin their own
initiates to reduce VAPs. The IHI website
provides up-to-date literature and ref-
erences on reducing VAPs (http://www
.ihi.org/Topics/VAP/Pages/default.aspx).
11. I want to thank Susan Herrmann, RN,
MS, clinical specialist, medical/surgical
unit, and Marjie Schoolfield, RN, BSN,
OCN, at Northwestern Medicine Delnor
Hospital (Geneva, Illinois) and William
Peters, IHI IA for sharing the context
and data for this case study. They have
been very vigilant in applying quality
measurement principles and tools to
this initiative. They have also been gen-
erous in devoting time to explain to me
the steps they have taken to track their
indicators, interpret the charts, and make
improvements. William Peters did a very
good job of constructing the charts and
partitioning the data to demonstrate the
points I wanted to make with this case
study. Congratulations to all of you for
a job well done.
12. Technically the centerline and control
limits should not be “frozen” and extended
if the process that produced these limits
contains special causes. Why? Because
the special causes make the process and
the observed variation in the process
unstable and, therefore, unpredictable.
But this team was eager to see how the
new process compared to the baseline,
so they acknowledged this issue but still
proceeded to freeze the limits and cen-
terline for comparison purposes. Life is
full of options!
13. Because these are individual patients, I
am not revealing the specific factors that
each patient possessed.
14. There are methods to test the simultaneous
effects of multiple variables on a process.
The most appropriate technique when
conducting QI research is known clas-
sically as DOE or design of experiments
and is frequently used in manufacturing
(Sloan, 1997). In healthcare settings, this
approach is referred to, however, as PE
and is detailed nicely in a book I regularly
teach from titled Quality Improvement
through Planned Experimentation (Moen
et al., 2012). Other approaches that might
be considered involve the development
of causal models (Blalock, 1971) and
multiple regression models (Kerlinger &
Pedhazur, 1973; Namboodiri, Carter, &
Blalock, 1975).
15. I am indebted to Mr. Mike Taigman,
improvement guide for FirstWatch
in Encinitas, California for initially
developing the idea for this case study.
Mr. Taigman is a graduate of the IHI
Improvement Advisor Professional
Development Program and has been a
champion of statistical thinking for years.
I appreciate his willingness to share this
important story with me and work with
him on this important topic.
16. I want to thank Dr. Amar Shah, associate
medical director and consultant forensic
psychiatrist at ELFT for permission to use
this case study. I have had the pleasure
of working with Dr. Shah and his team
at ELFT for over 3 years and it has been
a wonderful journey. They have made
great progress in a short period of time
to create a culture of quality and improve
the lives of those they serve. I am honored
to be part of their journey.

References
Blalock, H., ed. Causal Models in the Social Sciences. Chicago:
Aldine, 1971.
Carey, R. Improving Healthcare with Control Charts: Basic and
Advanced SPC Methods and Case Studies. Milwaukee:
ASQ Press, 2003.
Kerlinger, F., and E. Pedhazur. Multiple Regression in Behavioral
Research. New York: Holt, Rinehart and Winston, 1973.
Moen, R., T. Nolan, and L. Provost. Quality Improvement
through Planned Experimentation, 3rd ed. New York:
McGraw-Hill, 2012.
References 329
Namboodiri, N., L. Carter, and H. Blalock. Applied Multi-
variate Analysis and Experimental Designs. New York:
McGraw-Hill, 1975.
Sloan, D. Using Designed Experiments to Shrink Health Care
Costs. Milwaukee: Quality Press, 1997.
Tufte, E. The Visual Display of Quantitative Information.
Cheshire, CT: Graphic Press, 1983.
Tufte, E. Envisioning Information. Cheshire, CT: Graphic
Press, 1990.
CHAPTER 11
Connecting the Dots
“Transformation is required to move out of the present state, metamorphosis, not mere patchwork on
the present state of management.”
—W. Edwards Deming, The New Economics, 1994: 123
I
n Chapter 2, I introduced the importance of
connecting the dots in order to predict what
the collective distribution of data points (i.e.,
dots) is trying to tell you. Most of us remember
being engaged with the connect-the-dots activity
as children. It was a great way to help us see the
relationship of apparently disparate dots and
the image that can emerge if we connect the
dots correctly. As adults, it might not be a bad
idea to return to those early years and practice
connecting the dots once again. Frequently we
fail to connect all the dots and begin making
connections only between selected dots that
confirm our own view of reality. As a result, this
leads us to not only reinforcing our own theory
of knowledge, but also it provides an incomplete
view of the world and subsequently leads us to
make the wrong conclusions.
Just as we need to connect all the dots to
make sense of the variation in a set of data, we
also need to connect the dots at the organizational
level in order to build organizational excellence.
If leaders, managers, and staff all take time
to deliberately think about the various factors
© Michal Steflovic/Shutterstock
(the dots) that affect their individual and organi-
zational performance and then make the linkages
between these dots, they will not only be able to
adapt to the myriad of changes facing the health
and social services industries but they will also
be able to proactively harness these changes and
manage them to their benefit.1 We need to move
away from the old fragmented ways of thinking
and begin to have serious dialogue about the level
of transformation needed to achieve the new
state of management that Dr. Deming describes
in the New Economics (1994).
I believe there are four key activities that
will prove beneficial for leaders interested in
working on the transformation Deming discusses:
■■ Adopting quality as a business strategy
■■ Developing a learning system to support
improvement
■■ Linking measurement to improvement
■■ Building capacity and capability for
improvement
The remainder of this final chapter briefly
addresses each of these four activities.
331
332 Chapter 11 Connecting the Dots
▸▸ Adopting Quality as
a Business Strategy
Dr. Deming was well known for asking leaders
short yet very challenging questions:
■■ What business are you in?
■■ By what method will you improve your
business?
■■ Will best efforts bring improvement?
■■ What is management’s job?
■■ Who determines quality?
On this last point he was very clear: “Quality
is determined by the top management. It cannot
be delegated” (Deming, 1994, p. 17). What he
observed in many organizations, however, was
the lack of constancy of purpose, delegation of
quality to a department or a few individuals, and
management styles that were in his opinion “the
biggest producer of waste, causing huge losses
whose magnitudes cannot be evaluated, cannot
be measured” (Deming, 1994, p. 22). Dr. Deming’s
14 Points for Management (1992) were developed
as a strategy for management to avoid what he
called the “heavy losses” and stay in business,
protect jobs, and add value to society. In the early
1950s, while working in Japan with members of
the Union of Japanese Scientists and Engineers
(JUSE), he proposed what he called the “chain
reaction” to show the logical consequences of
transforming management and making quality
the organization’s overarching strategy for the
long run. I have adapted the wording he used in
his lectures and writings to explain his thinking
behind the chain reaction (see, e.g., Deming,
1992, p. 3) to make it more visual and relevant
to health and social service organizations. My
adaptation is shown in FIGURE 11-1. Notice that
this chain reaction begins with improving quality.
It does not begin with “hit the targets” or “meet
financial goals.” Deming’s logic was quite clear.
By improving quality there will be reductions in
costs because the organization is more efficient
and produces fewer errors and less waste. This
in turn leads to increased productivity, greater
staff engagement, and better morale among the
workforce. The result of this causal chain is that the
Improve quality
Cost decrease due to less
rework, fewer errors and
mistakes, better efficiency
Productivity and
morale improve
Attract more customers
Stay in business
Provide more jobs
Add value to society
FIGURE 11-1 An adaptation of Deming’s chain
reaction
organization will be more appealing to existing and
new customers, stay in business for the long run,
generate additional employment opportunities,
and most important, add value to society. Now
initially, Deming developed his chain reaction for
manufacturing industries. But, given the increasing
demands for health and social service industries
to add value for the money being spent on these
services, Deming’s chain reaction provides a logical
starting point to think about the role of quality
improvement (QI) within an organization.
Although the chain reaction notion provides
a good starting point to think about the logical
sequence of events required in order for an or-
ganization to stay in business and add value to
society, what is needed is a way to move beyond a
conceptual model and evaluate the specific activities
and behaviors that leaders need to support in order
to make quality their central business strategy.
In 1985, such a framework began to emerge.
Members of Associates in Process Improvement
(API)2 began to develop a template designed
to help leaders incorporate the philosophy and
 Adopting Quality as a Business Strategy 333

concepts taught by Dr. Deming into the way 2. Understanding the organization as
they managed their organizations. In 1987, they a system
formalized this framework and named it Quality 3. Designing and managing a system for
as a Business Strategy (QBS). The three basic gathering information for improvement
principles behind the development of QBS are 4. Conducting planning for improve-
that an organization needs to: ment and integrating it with business
planning
1. Establish a foundation of continuously
5. Managing and learning from a
matching products and services
portfolio of improvement initiatives
to a defined need of the organi-
zation and its customers through These five activities can be traced directly
design and redesign of processes, to Dr. Deming’s classic diagram of production
products, and services viewed as a system shown in FIGURE 11-2 and
2. Perform as a system to achieve this described by Dr. Deming in both Out of the
matching of products and services Crisis (1992) and The New Economics (1994). This
with the defined needs as the targets diagram was also used initially by Dr. Deming in
or goals of the organization 1950 at a conference in Japan. He developed it to
3. Maintain a set of methods to ensure depict how production of products or services
that changes result in real improve- needs to be viewed as an interrelated system
ments to the organization3 that requires generation of new ideas, product
design, production inputs (e.g., supplies, mate-
The details behind QBS can be found in the
rials, and equipment), creation and delivery of
API publication Quality as a Business Strategy
the product or service, and customer feedback
(API, 1998) and Norman (2007). In this books,
on whether the product or service met their
and their related publications (Langley et al., 1996,
expectations. Although the initial focus and
2009), they describe the five activities that leaders
intent of Figure 11-2 was on helping Japanese
need to carry out in order to make QBS a reality:4
leaders understand manufacturing production
1. Establishing constancy of purpose Dr. Deming was quick to point out, “In a service
in the organization (mission, vision, organization, the sources A, B, C, etc. could
and values) be sources of data, or work from proceeding

Stage 0:
Generation
of ideas

Design Consumer
and research
Suppliers of redesign
Consumers
materials
Receipt and
and equipment
test of
A materials Distribution

B Production, assembly, inspection

D Tests of processes,
machines, methods,
costs

FIGURE 11-2 Deming’s production viewed as a system

Deming, W. Edwards. The New Economics for Industry, Government, Education, second edition, figure, page 58, © 2000 Massachusetts Institute of Technology, by permission of The MIT Press.
334 Chapter 11 Connecting the Dots
operations, such as charges (as in a department
store), calculations of charges, deposits, with-
drawals, inventories in and out, transcriptions,
shipping orders and the like” (1994, p. 58). In
Out of the Crisis Dr. Deming directly addressed
quality and productivity in service organization
in detail. On pages 203 to 205 he offers “sugges-
tions on study of performance in a hospital.”5
As we have used Figure 11-2 with our vari-
ous strategic partners, we have clarified further
that “production” can easily be translated into
the diagnosis and treatment of patients, surgical
procedures, laboratory and pharmacy services,
home care services, rehabilitation services, mental
health care, and a variety of educational and other
social care services. In 2009, the API team created
an adaptation of Figure 11-2 to make the diagram
even more relevant to encompass service indus-
tries (Langley et al., 2009). All work, regardless of
the product or service produced, is a process or a
series of processes that create the system.
When we are invited to conduct site visits to
assess an organization’s commitment to making
quality a central operating strategy at some point
in the visit we will ask the leaders if they are
adopting and working on the five QBS activities.
Many leaders say they have never even heard of
QBS let alone adopting the five activities. The
ensuing dialogue is always interesting. This is
where the issues about defining quality start to
come out. A typical response from leaders will
often be this: “Oh sure, we are very committed
to quality. We have three people working in our
Quality Department. They handle everything
from Department of Health regulations to the
Joint Commission accreditation process as well as
tracking all of our improvement projects. We’ve
delegated the quality work to this department
and they are doing a very good job for us. Yes,
we are very committed to quality.”
After listening attentively to responses like
this we point out that quality is not a department
or something to delegate to a department or
selected individuals (Lloyd, 2016). We then ask
the senior leaders questions such as:
■■ How involved are you in making quality a
central strategy for the organization?
■■ Where do the topics of quality, safety, and
the patient/family experience fit into your
board and senior leader meeting agendas?
■■ Are quality, safety, and patient-centered
care the topics at the top of your agenda
or do they appear after the discussions
about finance, budget, market share, and
resourcing have occurred?
■■ Do you start each senior leader or board
meeting with a quality, safety, or customer
experience story?
■■ How much of senior leadership’s time is
devoted to discussions around quality,
safety, and the patient/family experience?
From the following list of questions the ones
that always cause a pause in the dialogue are the
ones that explore the priority of financial issues
against the priority placed on quality-related issues.
■■ How involved are you with the organiza-
tion’s finances?
■■ Do you delegate responsibility for the finan-
cial performance of the organization to a
department as you do quality and have no
further involvement with the financial issues?
■■ Do you spend at least equal time in board and
senior management meetings discussing
quality issues and financial issues?
■■ If not, which topics dominate your leader-
ship discussions?
As you can imagine, these questions create an
interesting exchange. To facilitate this exchange
we refer to the QBS five activities for leaders and
ask them where they stand with respect to each
one. Deming in the New Economics (1994) was
quite blunt about where he felt leadership of the
transformation stands. He did not feel that the
transformation of overall business strategy and
planning has penetrated the ranks of management.
He wrote: “Ninety-five percent of changes made by
management today make no improvement” (Dem-
ing, 1994, p. 38). He maintained that management
spends too much time focusing on what he called
“unique processes that produce figures” and not
enough time on the organization’s overall business
strategy or company-wide systems that influence
the organization’s future. He concluded that even
though management spends more time on the
 Adopting Quality as a Business Strategy 335

unique processes that produce figures, this rather
singular focus still would produce only about 3%
success in meeting organizational long-term goals.
To help leaders in establishing a dialogue
on QBS, I developed the brief QBS Assessment
Tool shown in EXHIBIT 11-1. In order to establish
the foundation for dialogue among your own
senior leaders about QBS, I would recommend
the following steps:
■■ Distribute the QBS self-assessment to
senior leaders.
■■ Ask them to complete the assessment and
send it back to an individual for tabulation,
preferably in a graphic format.
■■ Schedule a meeting of the senior leadership
team to discuss the results.
■■ Be sure to have a designated individual to
serve as facilitator.
■■ A key point when interpreting the re-
sults is to focus on the alignment of the
response patterns rather than the percent-
age of respondents selecting any given
response option. For example, if 50% of
the respondents select “Not started yet” for
activity 1 (establishing & communicating
the purpose of the organization) and the
remaining 50% selected “Completed,”
then there is poor alignment within the
leadership team on this particular item.
A discussion should occur as to why half the
group selected the one end of the response
spectrum and the other half selected the
other end. For leaders, alignment around a
particular activity is more important than
the absolute percentages. Remember that
the data should be used for learning and
dialogue not judgment.

EXHIBIT 11-1 Quality as a business strategy assessment tool

Instructions: For each of the Five Quality as a Business Strategy (QBS) Activities listed here select the
one response that best captures the current status of the activity within your organization.

Response Options
Quality as a Business Strategy
Five Activities for Leaders 1 2 3

1. Establishing constancy of purpose in the organization

(mission, vision, and values)

2. Viewing the organization as a system

3. Designing and managing a system for gathering

information for improvement

4. Conducting planning for improvement and integrating it

with business planning

5. Managing and learning from a portfolio of improvement

initiatives

Response Options
1= Not started yet 2 = In progress 3= Completed and firmly in place
336 Chapter 11 Connecting the Dots
In summary, QBS can (and many of us argue
should) serve as an overarching framework for
any organization. A caution is in order, however.
According to API (1998, p. 3), “The QBS template
cannot simply be installed or implemented like a
new computer system in an organization. There is
a need for knowledgeable leadership to carry out
the strategy and make it successful. Five activities to
be led by the top management of the organization
provide the structure to begin working on making
quality a business strategy. The five activities form
a system for the leaders of an organization to focus
their learning, planning, and actions.”
▸▸ Developing a Learning
System to Support
Improvement
Systems’ thinking is not only a core activity of
QBS but it also forms the foundation of what
Deming called a SoPK.6 He detailed four key
components that allow individuals to gain a clear
picture of the system they are trying to improve:7
1. Appreciation for a system
2. Understanding variation
Appreciation
of a system
Theory of
knowledge
Understanding
variation
FIGURE 11-3 Looking through the lens of profound knowledge
© Tom Wang/Shutterstock
3. Building knowledge
4. The human side of change
These points are detailed nicely by Deming
in his classic works (1992, 1994). Other good
summaries may be found in Langley et al. (2009),
Schultz (1994), and Scherkenbach (1991). A very
good short summary of profound knowledge
was written by Moen and Norman in 2016. Their
article, “Always Applicable: Deming’s System
of Profound Knowledge Remains Relevant for
Management and Quality Professionals Today”
not only provides an excellent overview of Dem-
ing’s system of profound knowledge but also
describes why it provides an “essential framework
for management and quality professionals to
understand the messiness of the business world
and develop productive paths forward” (p. 27).
Deming argued that a system cannot under-
stand itself. He said that in order to understand
the system you want to improve you need to
have an outside view. People involved with the
system are too close to it and involved to a level
of detail that makes it quite difficult for them to
objectively see all the interrelated and intercon-
nected parts of the system. He offered his system
of profound knowledge (SoPK) as a “lens” that
when looked through and applied can provide
Human
Behaviour
 Developing a Learning System to Support Improvement
all members of the organization with a more
clear and comprehensive understanding of the
system they are trying to transform. FIGURE 11-3
is my depiction of the lens Deming referenced.
Notice that the four components are all inter-
connected. Deming pointed out, “One need not
be eminent in any part nor in all four parts in
order to understand it and apply it” (1994, p. 93).
What he stressed as essential, however, was the
interrelated nature of the four components. A
brief overview of each of the four components
of profound knowledge is provided next.
Appreciation of a System
For Deming, this was the starting point for
developing deep insight about an organization.
His production viewed as a system (Figure 11-2)
served as the starting point. The literature on sys-
tems thinking is very rich and besides Deming’s
views on this topic (1992, 1994) readers would
do well to become familiar with the works of
Ackoff (1981); Argyris (1990); Churchman (1968);
Drucker (1973); Forrester (1971, 1986); Nelson,
Batalden, and Godfrey (2007); Senge (1990); Senge,
Roberts, Ross, Smith, and Kleiner (1994); and von
Bertalanffy (1968). The central theme that runs
throughout all these works is that a system is es-
sentially an interdependent series of functions and
activities that bring together people, equipment,
methods, processes, and procedures to achieve
a common purpose or aim. The key word that
characterizes all the writing about appreciation
of a system is interdependence of the parts. Think
about this from a medical perspective. When cli-
nicians are in training they learn about the body
as a system. There is the circulatory system, the
musculoskeletal system, the nervous system, the
respiratory system, and so on. Systems thinking is
a fundamental aspect of medical school training.
You would think, therefore, that it would be an
easy transition to apply system thinking to how
healthcare organizations function. But this is
frequently not the case. When only part of the
system is made to function efficiently or effectively
other parts of the system will be compromised.
Deming called this “suboptimizing the system.”
337
Understanding Variation
This has been a central theme of this book. But
suffice it to say that variation exists. Period! It is
critical, therefore, that if you are truly interested
in improvement you need to be able to:
■■ Distinguish enumerative from analytic
studies (Chapter 2)
■■ Identify common cause variation from
special cause variation (Chapter 7)
■■ Build skills in the construction and inter-
pretation of run charts (Chapter 8) and
Shewhart charts (Chapter 9)
Make the correct decision when you identify
■■
common and special causes of variation
(Chapter 7)
If a run or Shewhart chart exhibits common
cause variation then it is reflecting a process
that is stable and therefore predictable. But the
performance of the stable process must now be
placed against a second question: Is the process
capable of meeting the target or goal that the
improvement team has established? Quality
improvement is about understanding stability,
instability (special causes), and capability. This
can be achieved only by applying statistical
process control (SPC) methods.
Building Knowledge
Originally, Deming referred to this component
as Theory of Knowledge. Langley et al. (2009)
changed this to Building Knowledge, which I
believe is a more practical label. Essentially what
Deming was conveying with this component is
that we all have theories as to how and why things
occur. Theories lead to predictions, predictions
lead to a test, and the results of a test, whether
successful or a failed test, lead to new knowledge.
For example, I come up with a theory that if I
alter my route to work and take a different com-
bination of roads I will reduce my drive time. I
have a theory that leads to a prediction. I test the
new route and record the number of minutes this
alternative route takes to get me to work. Did it
take more time, less, or the same amount of time
338 Chapter 11 Connecting the Dots
as my traditional route? I had a theory, made a
prediction, and then tested this prediction against
the data I record. Deming captured this sequence
when he wrote, “Without theory, experience has
no meaning. Without theory one has no questions
to ask. Hence without theory there is no learning”
(Deming, 1994, p. 103). Building knowledge,
therefore, is a deliberate and iterative process
that involves inductive and deductive approaches
to learning (Langley et al., 2009). Do you and
the leaders within your organization regularly
explore the theories you have about how and
why things occur? Do you establish predictions
about your theories? Do you build knowledge or
make judgments?
The Human Side of Change
Deming originally labeled this component of the
lens of profound knowledge as psychology. The
API authors (Langley et al., 2009) modified this to
be the human side of change to give it a broader
meaning. The human side of change involves the
points about psychology that Deming emphasized,
but it also addresses issues of social psychology,
group dynamics, conflict resolution, intrinsic
and extrinsic motivation, and systems of hiring,
performance review, and rewards. Understanding
and embracing the human side of change will
help leaders realize more fully that individuals
and groups have a wide variety of needs and ex-
pectations as Herzberg detailed in his 2003 article
on intrinsic and extrinsic motivation. When we
teach our yearlong IA Professional Development
Program many of the participants struggle more
with this component of profound knowledge
than with any of the other three. Although they
become concerned when we start sessions on
Shewhart charts, statistical analysis, and planned
experimentation, they actually become nervous
when we ask them to have a difficult conversation
with an individual they are having a challenge
with at work. The human side of change is very
difficult and is frequently an area in which staff
and managers are not given adequate training and
development. We focus on filling out forms and
following correct technical procedures but we do
not spend an equivalent amount of time dealing
with the interaction and motivation of people.
I really find this interesting especially because
we are involved in an industry that is classified
as a service. We are in the people business but
all too often we pay little or no attention to the
people involved in delivering and receiving care.
Leaders and managers need to dedicate time to
finding out what factors are most important to
their individual staff members. Each person is
motivated by different things. Some individuals
are more extrinsically motivated whereas others
are more intrinsically motivated. If you have not
read Herzberg’s (2003) classic article “One More
Time: How Do You Motivate Employees?” you
should. He succinctly lays out the factors and
events on the job that lead to extreme dissat-
isfaction (hygiene factors) and those that lead
to extreme satisfaction (intrinsic motivators).
What do you think are the most satisfying as-
pects of a person’s job? Herzberg’s cross-industry
and cross-country investigations have clearly
documented that the top six factors leading
to job satisfaction include, in descending rank
order, achievement, recognition, the work itself,
responsibility, advancement, and growth. Six
factors that lead to extreme job dissatisfaction in
descending rank order, include, company policy
and administration, supervision, relationship
with supervisor, work conditions, salary, and
relationship with peers.8 In addition to the work
of Herzberg, readers interested in learning more
about intrinsic and extrinsic motivation should
become familiar with the works of Alfie Kohn.
Kohn’s first book, No Contest: The Case Against
Competition (1986) traces the history of com-
petition and argues that “healthy competition”
is a contradiction in terms. His second book,
Punished by Rewards: The Trouble with Gold
Stars, Incentive Plans, A’s, Praise and Other Bribes
(1993) speaks much more directly to negative
consequences of extrinsic motivation. Deming
praised Kohn for highlighting the destructive
forces of life (1994) and how they contribute to
the “heavy losses” within an organization. Kohn’s
work has not been without its critics. His work
is controversial but it offers provocations that

leaders and managers need to consider seriously.
In summary, the human side of change poses a
major challenge for many leaders and managers.
Listening, not just hearing but really listening to
employees in order to find out what motivates
and demotivates them, is essential not only for
good management but is also critical to the over-
all success of the organization. But this requires
that leaders and managers must have an open
and honest dialogue with each individual they
supervise to find out what really matters to each
person and how they as managers can contribute
to creating joy in work for each employee.
This brief review of the four components
provides a start. I strongly recommend building
a deeper understanding of Deming’s think-
ing on this subject if you are serious about
making quality a reality within your organization.
▸▸ Linking Measurement
to Improvement
Data without a context for action is waste of
time, effort, and money. Yet, considerable data
are collected within healthcare and social care
settings that have no direct connection to actions
aimed at improvement. Staff time and effort are
resources not to be wasted. Therefore, a deliberate
plan needs to be established that:
■■ Periodically evaluates the relevance and
utility of the indicators
■■ Regularly reviews the amount, frequency,
and duration of the data being collected on
the various indicators
■■ Assigns responsibility for turning the data
into useful information for decision mak-
ing, including understanding common and
special causes of variation
Linking your measurement efforts to
improvement begins with understanding the
milestones in the quality measurement journey
(QMJ) and then building skills throughout the
organization that allow these milestones to be
part of daily work. Although you may identify
Linking Measurement to Improvement 339
a small group of individuals who will be the
deep knowledge experts in the QMJ milestones,
you should be taking steps to build knowledge
in all employees about the nature of data for
improvement.
To assist you in determining the level of
knowledge and your ability to apply the mile-
stones in the QMJ, I developed the Measurement
Self-Assessment tool shown in EXHIBIT 11-2. This
self-assessment tool provides five rank-order
responses. Read the description of each response
option (A through E) and then select the one
response that best describes your current level
of knowledge and skill related to each item. You
should administer this assessment tool to indi-
viduals designated to serve as IAs, coaches, and
improvement team members. There are no right
or wrong answers, only an opportunity to gain
a better understanding of where you and your
colleagues personally stand with respect to the
milestones in the QMJ. For example, what would
your reaction be if you had to explain why it is
preferable to plot data over time rather than using
aggregated statistics and tests of significance?
Can you construct a run chart or help a team
decide which measure is more appropriate for
their project? You may not be asked to do all of
the things listed in this assessment today or even
next week, but, if you are facilitating a QI team or
expect to be able to demonstrate improvement,
sooner or later these questions will be posed. How
will you deal with them? The place to start is to be
honest with yourself and see how much you know
about concepts and methods related to the QMJ.
Once you have had this period of self-reflection,
you will be ready to develop a learning plan for
yourself and those involved with improvement
to close the gap between where you are currently
in your QMJ and where you would like to be.
As you build new knowledge of and
skills with the milestones in the QMJ, it is
critical to be cognizant of the fact that data
analysis and making run or Shewhart charts
can become a very captivating activity. I have
seen individuals become very involved in
the physical process of collecting data and
making charts. The tactile stimulation gained
EXHIBIT 11-2 Measurement self-assessment
340

Response Options
Measurement Topic or Skill A B C D E
Help people in my organization determine why they are measuring (improvement, judgment, or research)

Move teams from concepts to specific quantifiable measures

Building clear and unambiguous operational definitions for measures

Develop data collection plans (including stratification and sampling strategies)

Chapter 11 Connecting the Dots

Explain why plotting data over time (dynamic display) is preferable to using aggregated data and summary
statistics (static display)

Explain the differences between random and nonrandom variations

Construct run charts (including locating the median)

Explain the reasoning behind the run chart rules

Interpret run charts by applying the run chart rules

Explain the statistical theory behind Shewhart control charts (e.g., sigma limits, zones, special cause rules)

Describe the basic seven Shewhart charts and when to use each one

Help teams select the most appropriate Shewhart chart for their measures

Describe the rules for special cause variation on a Shewhart chart

Help teams link measurement to their improvement efforts

A. I’d definitely have to call in an outside expert to explain and apply this topic/method.
B. I’m not sure I could apply this appropriately to a project.
C. I am familiar with this topic but would have to study it further before applying it to a project.
D. I have knowledge about this topic, could apply it to a project, but would not want to be asked to teach it to others.
E. I consider myself an expert in this area, could apply it easily to a project, and could teach this topic/method to others.

by entering commands on a keyboard and
watching a multicolored charts print out can
give one a sense of accomplishment. But, the
production of charts or statistical analysis is
not the end objective. The charts do not tell
you (1) the reasons(s) for a special cause,
(2) whether or not a common cause process
should be improved (i.e., is the performance of
the process capable?), or (3) how the process
should actually be improved or redesigned. To
accomplish these aims you need a framework or
roadmap to guide your overall quality journey.
There are many frameworks that can provide
a roadmap for your improvement journey.9
At the Institute for Healthcare Improvement
(IHI) the framework we use to frame and drive
improvement is the Model for Improvement
(MFI) developed by Langley et al. (2009). My
adaptation of the MFI, shown in FIGURE 11-4,
consists of three questions combined with the
Plan–Do–Study–Act (PDSA) cycle. The three
fundamental questions are:
■■ What are we trying to accomplish? This is
the team’s aim statement. How good do
they want to be and by when do they plan
to achieve this result?
■■ How will we know that a change is an im-
provement? This is the measurement question
and the central theme of this text.
Aim
PDSA cycle
Ideas Measures
FIGURE 11-4 The model for improvement
Reprinted by permission of Robert Lloyd.
Linking Measurement to Improvement 341
■■ What changes can we make that will result in
improvement? This question challenges a team
to develop ideas that they believe will lead
to improvement and achieve the stated aim.
These questions guide a team’s journey
and should be reviewed and evaluated contin-
uously. It is also important to realize that the
three questions can be addressed in any order.
For example, an improvement project could be
initiated because several nurses while reviewing
their unit’s data discover that the number of
falls has been increasing over the past 2 weeks.
This could lead to developing an aim statement
to reduce the number of falls. Alternatively, a
staff member might have attended a conference
where she heard a presentation on a new idea
that was successful at improving medication
reconciliation at time of discharge. Now she
has an idea that leads her to explore her unit’s
current percentage of med reconciliation at
time of discharge, which in turn leads to writing
an aim statement for improvement. The three
questions, therefore, should not be viewed as a
linear progress (i.e., first, develop an aim, then
gather some data, and finally, develop ideas for
improvement). Instead, they should be viewed
as a dynamic set of interrelated questions.
The second part of the MFI is the PDSA cycle.
Whereas the three questions provide guidance
and checks to help a team maintain its direction
and stay on course, the PDSA cycle provides the
logical path for testing the ideas that the team
believes will lead to improvement. The PDSA
cycle has a rich history that has been described
and analyzed by many writers, including Langley
et al. (2009), Schultz (1994), and Scherkenbach
(1991). An excellent overview of the history of
the PDSA cycle was written by Moen and Nor-
man (2010). They not only trace the evolution
of the scientific method and how it provides
the foundation for the PDSA cycle but they
also provide a very easy to follow historical
journey from the creation of the Shewhart cycle
to the emergence of Deming wheel, the Japanese
development of the PDCA cycle (Plan, Do,
342 Chapter 11 Connecting the Dots

Check, Act), and the more contemporary term, what the letters PDSA stand for. What we have
the PDSA cycle. FIGURE 11-5 provides a summary not achieved so far, however, is getting a majority
of the key activities involved with each step in of healthcare professionals to embrace and then
the PDSA cycle. apply the PDSA cycle to daily work. There are
One of the things I find most interesting a number of very good examples of how the
about teaching the PDSA cycle is how many PDSA cycle has been successful in improving
people know what the letters PDSA stand for. quality, reducing cost, and enhancing the cus-
I have conducted workshops and seminars with tomer experience but there is still considerable
large audiences and say: “If you know what the work to be done to make this the way healthcare
letters PDSA mean please stand up.” Consistently, professionals approach improvement.11 Sending
nearly the entire audience will stand.10 Then I out more policy statements, edicts, arbitrary
ask people to remain standing if they have run targets, and goals or demanding that staff attend
a PDSA test in the last month. At this point an in-service training session will not achieve
about 75% of the audience sits down. Then, I the levels of excellence most organizations
ask the 25% still standing to remain standing and governments envision or desire. What is
if they have run a PDSA in the past 2 weeks. required is a focused approach that creates and
This now gives me about 10% of the audience maintains the will, ideas, and execution skills
standing. Finally, I ask this small group to remain needed to achieve excellence.
standing if they have run a PDSA in the week When working with teams I frequently
prior to coming to the workshop. Regardless of find that they think that if they “do” a PDSA
the size of the audience I typically end up with cycle once then they are done with testing. It
only 1–2% of the participants actually having is then time to implement the new idea and
run a PDSA test the previous week. My point then spread it to all other locations within the
is that in the health services field we have been system. This is a fundamental flaw in execution
reasonably successful in getting people to know as well as strategy. The sequence of improve-
ment is shown in FIGURE 11-6. Throughout
this sequence the PDSA cycle and data are
used to test a variety of ideas under different
What will happen
if we try something conditions and to gain new knowledge about
different? the improvement idea. All improvement begins
What's next?
ACT PLAN with an idea that should be initially tested
• Test again? • Objective on a small scale, preferably in a pilot unit
• Test a new idea? • Questions & or ward. Then as shown in Figure 11-6, this
• Compare theory predictions testing should be conducted under a variety
to predictions • Plan to carry it our: of conditions to determine how robust and
• What changes need Who? When?
to be made? Where? How? reliable the new idea is. This is where quality
control (QC) should start to come into the
STUDY DO sequence. The new idea for med reconciliation,
• Complete the data • Carry out the plan for example, might have worked well in the
analysis • Collect data pediatric unit but failed in the geriatric unit.
• Compare result to • Document
predictions problems
Why? What was learned about the conditions
• Summarize the • Start data in one unit versus those in the other? Did the
learning analysis new idea not work as well because the patients
are different, the physical layout of the units
Did it work? Let's try it!
is different, or maybe the staff members on
FIGURE 11-5 The components of the PDSA cycle each unit have different levels of knowledge
 Linking Measurement to Improvement 343

Make part of Sustaining improvements

routine and spreading changes to
operations other locations
Test under
a variety of Implementing
conditions a change Act Plan
Theory
and Testing a
change Act Study
Plan Do
prediction

Developing Act PlanStudy Do

a change
nce
Act Plan Study Do ue
eq
es
t th
ou
Act Plan Study Do
h
o ug
Study Do
d thr
use
Act Plan

are
Study Do

ta
Da

FIGURE 11-6 The sequence of improvement

East London NHS Foundation Trust

about the science of improvement (SOI)? the chart shows that there has been a shift
Testing under different conditions is where of the outcome indicator in the direction
ideas are refined, adapted, and made more of goodness and it has been sustained for
reliable. This is one of the most critical steps a period of time (e.g., four to five new data
in the improvement sequence. points staying in the general vicinity of the
Implementation differs from testing in shift) then you have a higher degree of belief
several ways. When you are testing a new that the improvement has been sustained and
idea you are (1) trying and adapting existing the conditions for implementation appear to
knowledge on a small scale and (2) learning be in place. On the other hand if you detect
what works in the process or system of inter- a shift in the data in the desired direction but
est and what does not work well. When you the next four or five data points start drifting
move to implementation you are ready to (1) back to the less desirable level of performance,
make the new idea a permanent part of the this is a reasonably strong indication that the
day-to-day operation of the process or system, improvement has not been sustained and that
(2) develop all the support and infrastructure you are not ready for implementation of the
required to maintain the change(s), (3) expect new idea. Implementation requires three ba-
to see sustained improvement with few or no sic conditions to be met: (1) you have a high
failures or back-sliding in performance, and (4) degree of belief that the new idea will in fact
deal with the potential of increased resistance lead to improvement; (2) the cost of failure in
from individuals who may have to adopt new terms of time, effort, morale, and resources is
ways of working or thinking. low; and (3) individuals are ready, willing, and
The data on your run or Shewhart chart eager to embrace the change. If any of these
should also be used to help determine whether three conditions are not present, the chances
you are ready to implement. If, for example, of successful implementation will be greatly
344 Chapter 11 Connecting the Dots

diminished. The Improvement Guide (Langley
et al., 2009) has a detailed chapter that describes
all the conditions under which implementation
can be successful. They provide an excellent
implementation checklist that should be used
to guide a team’s implementation journey.
The final step in the sequence of improvement
(Figure 11-6) is sustaining the improvements
and spreading changes to other locations. Like
all the previous steps in the sequence, data play
a central role in realizing a successful spread
effort. This stage also requires running PDSA
tests to see if the conditions are right to support
spread. The classic reference on the adoption
and diffusion (spread) of new ideas was written
by Everett Rogers (1995). At the IHI we have
focused extensively on the topic of spread in
healthcare settings and have produced a variety
of publications that can all be obtained from the
IHI website (www.ihi.org) including a framework
for spread (Barker, Reid, & Schall, 2016; Massoud
et al., 2006) and the important role of scale-up
in planning a spread initiative (McCannon,
Schall, & Perla, 2008). These publications will
help you clarify the nature of spread (i.e., having
individuals adopt the new ideas and changes)
and how scale-up (i.e., overcoming the process
and structural issues that arise during spread)
is an absolutely crucial component of every
successful spread initiative. Chapter 9 in the
Improvement Guide (Langley et al., 2009) also
provides excellent guidance on how to develop
a spread plan and guarantee its success. Finally,
you should be aware of the key pitfalls that
frequently arise when organizations engage in
premature spread work. We refer to these pitfalls
endearingly as the “seven spreadly sins.”12 The
seven spreadly sins include:
1. Don’t bother testing just start with
large pilots!
2. Find one person willing to do it all!
3. Rely solely on vigilance and hard work!
4. If a pilot worked then spread the pilot
unchanged! After all, if it worked well
once why won’t it work that way in
other locations?
5. Require the person and team who
drove the pilot to be responsible for
system-wide spread!
6. Look at process and outcome mea-
sures on a quarterly basis!
7. Early on expect marked improvement
in outcomes without attention to
process reliability.
The final aspect of linking your measurement
efforts to improvement work comes by assessing
three organizational drivers: will, ideas, and exe-
cution. These three organizational assets were first
proposed by Dr. Tom Nolan to Dr. Don Berwick
back in 1998 when Dr. Berwick was preparing
his plenary address for the 10th Annual National
Forum on Quality Improvement in Health Care
(Berwick, 2004). Like many of the other con-
cepts and components discussed in this chapter,
these three characteristics interact to create the
cultural and tactical milieu in which change can
either flourish or flounder. FIGURE 11-7 depicts
the interaction of these three drivers. For many
years, this triumvirate of will, ideas, and execution
has served as a fundamental departure point for
assessing the likelihood that an organization will
be able to achieve excellence. When conducting
diagnostic visits to an organization we regularly
ask leaders what level of will they have within
their organization to change. Where do they get
their ideas for improvement and how many new
ideas do they generate every week? Finally, are
they building within their staff the appropriate
skills to take the ideas they have generated and
execute change?
TABLE 11-1 provides a very brief assessment
tool related to the level of will, ideas, and execu-
tion skills within an organization. You can use
this assessment at the organizational level, at
the department level, or at the team level. If you
decide to use this assessment tool I would suggest
sending it to the designated respondents prior
to a meeting and instruct them to send it back
to you for tabulation. Then present the results
at the next meeting. The responses provide a
wonderful opportunity to create a dialogue on
why the responses were selected. I have used

Having the Will (desire) to change the current state to one that is better
Will
Developing Ideas that will
contribute to making processes and
Ideas
outcome better
FIGURE 11-7 The primary drivers of improvement
TABLE 11-1 The primary drivers of improvement
Key Components*
■■ Will (to change)
■■ Ideas
■■ Execution
*All three components MUST be viewed together. Focusing on one or even two of the components will guarantee suboptimized performance. Systems
thinking lies at the heart of QI!
this assessment in many countries and have
consistently discovered that the level of will to
change is usually overestimated. Responses usu-
ally are in the medium-high to high categories.
When we explore the reasons for this positive
response pattern I get comments like, “Oh yes,
we definitely have a high desire to change and
make things better” or “Our mission, vision, and
values all point us in the direction of changing
things to make them better.” Yet, as we discuss
how it is that their results are not near the targets
or goals they have established, I get responses
like, “Well we have not been able to convince
the (fill in the name of the appropriate group of
staff or employees) to actually change the way
they do things.” This is not surprising. Health
care is a fairly conservative, traditional, and very
hierarchical industry that is not known for rapid
change and the quick adoption of new ideas.
This was explored very thoroughly by Balas and
Linking Measurement to Improvement 345
QI Having the capacity to apply CQI
theories, tools and techniques that
Exucution
enable the Execution of the ideas
Self-Assessment
■■ Low Medium High
■■ Low Medium High
■■ Low Medium High
Boren in their classic article “Managing Clinical
Knowledge for Health Care Improvement” (2000).
The authors’ conclusion is a well-known fact
within medical circles: “Studies suggest that it
takes an average of 17 years for research evidence
to reach clinical practice” (p. 66). Although there
are healthcare organizations that have demon-
strated a reasonably high level of will to change,
my experience has led me to conclude that the
will to change in a majority of healthcare settings
is more accurately in the medium to low end of
the responses. When I was conducting this little
assessment in a workshop one day, I had a fellow
raise his hand when I asked what the level of will
was in their organizations. He wanted to know
if the response options included zero or even
negative numbers. Being realistic about the will
to change is a starting point. Because the desire
to improve the current state will most likely vary
within an organization I think it is important
346 Chapter 11 Connecting the Dots
to do this assessment with different groups of
staff and also to apply it across departments and
functions. Building a culture of quality is not a
simple challenge. Therefore, it is critical to begin
understanding the degree to which the will to
change varies within the organization.
When I ask what the level of ideas is within
the organization I usually get a medium-high
to high assessment. I agree with this response
and think it is pretty accurate. Meta studies
have shown that each month over 10,000
new bits of knowledge and information (e.g.,
books, articles, journals, and blogs and other
web-based content) are disseminated within
the healthcare industry. This is a huge amount
of material to digest. There is so much new
information that many physicians have trouble
keeping current on their discipline’s journals.
Many organizations actually have a knowledge
management staff that is charged with sorting
and organizing new information. So, it does
seem reasonable that access to new ideas gets
a high rating.
But, when I get to asking the question
about execution skills, I usually get a loud and
resounding “low” response. Having the ability
to blend the will to change with new ideas and
produce results requires a special set of execution
skills and knowledge. These skills are typically
not taught in schools of medicine or manage-
ment. So, it is not surprising that we usually
have quick consensus that organizations need
to develop and institute strategies and tactics
aimed at building capacity and capability for
improvement. This leads us to the final of the
four transformational activities.
▸▸ Building Capacity
and Capability for
Improvement
Building capacity and capability for improve-
ment is not accomplished by sending staff to
one-off “training” sessions. The capacity and
capability journey is not a singular event but
rather an ongoing strategic and tactical com-
mitment to prepare the organization for the
future. Organizations interested in building
capacity and capability need to start with an aim.
For example, a straightforward aim to achieve
organizational excellence might be: To build a
renewable infrastructure that produces highly
reliable quality and safety and customer service
by (fill in the date). This journey follows a path
that is shown in FIGURE 11-8.
Are Capacity and Capability
the Same Thing?
It is important to realize that the concepts of ca-
pacity and capability are not synonymous terms.
Capacity refers to the following characteristics:
■■ The ability to receive, hold, or absorb content
and new knowledge
■■ The maximum or optimum amount of
production or output that can be delivered
■■ The ability to learn or retain information
■■ A measure of volume; the maximum amount
of new knowledge that can be held
■■ The power, ability, or possibility of doing
something or performing
Essentially building capacity refers to filling
people with the knowledge, methods, and skills
associated with the SOI. It can be considered
an initial step in creating the potential for an
organization to improve but by itself building
Excellence
Sustainability
Capability
Capacity
FIGURE 11-8 Milestones in the journey to build
excellence
 Building Capacity and Capability for Improvement
capacity provides no assurance that improve-
ment thinking and applications will be part of
the very fabric of daily life within the organi-
zation. Many organizations make the mistake
of thinking that if they send staff to inservice
training sessions and “fill them up” they have
built capability. Wrong! All they have done is
to increase their capacity. Capability, on the
other hand, refers to:
■■ The power or ability to generate an outcome
or results
■■ The ability to execute a specified course
of action
■■ The sum of expertise and capacity
■■ Knowledge, skill, ability, or characteristics
associated with desirable performance on
a job, such as problem solving, analytical
thinking, or leadership
■■ Motivation, beliefs, and values about work
and the individual’s role in the organization
If capacity is focused on filling people with
the appropriate amount or dose of knowledge
(i.e., giving them potential) then capability is
releasing this potential energy and turning it
into kinetic energy. That is, allowing people with
knowledge and skills of QI to (1) have protected
time to apply the knowledge they have been given,
(2) have access to structures and processes that
support quality and safety initiatives, and (3) be
part of a learning organization that values ongo-
ing development and growth. In a 2010 article,
Bevan highlighted the critical role of building
capacity and capability: “A focus on building
capacity and capability for improvement is a key
strategy. Global analysis of healthcare systems
that deliver outstanding performance in cost and
quality shows their most common characteristic
is a systematic approach to capability building
for improvement” (p. 139).
Who Needs to Know What?
So, how does an organization begin to build
capacity and capability for improvement? A
key point in properly mastering this aspect of
your quality journey is to first realize that not
347
everyone in the organization needs to have the
same depth of knowledge about the SOI. To help
people understand this concept I have used the
analogy of how medicines are dosed. When a
medicine is prescribed, for example, a critical
aspect of the effectiveness of the particular
medication is the dose that is prescribed. Even
for patients receiving the same medication,
the dose will differ depending on the specific
condition and needs of the patient. This no-
tion of dosing also applies to an organization’s
need to build capacity and capability for QI.
Not everyone within an organization needs to
have the same “dose” of the SOI. Therefore, it
is incumbent on the leaders of an organization
to have a serious dialogue about the dose of the
SOI that needs to occur at various levels within
their organization. For example, the dose that
board members and senior leaders need will be
different than the dose that middle managers
and supervisors need. The dose of the SOI that
those delivering care at the bedside need to be
effective in their QI efforts will be different from
that which supervisors receive and both of these
doses of knowledge will be different from that
which is needed for those expected to serve as
team coaches and IAs.
The specific doses of the SOI are not de-
termined by a strict mathematical formula.
Instead each organization must be assessed in
terms of its current understanding of the SOI,
its demonstrated ability to actually improve
results, and the existing levels of will, ideas,
and execution skills within the organization.
Such diagnostic analysis allows us to develop
a dosing strategy that is tailor made to each
organization. Therefore, the dosing strategy for
each organization will be different. When I have
helped organizations develop a dosing strategy
we typically follow these steps:
■■ Conduct a diagnostic assessment to obtain
a clear picture of where the organization is
in its quality journey (e.g., just beginning,
evolving, or mature) and where it wants to
be in the next 1, 3, and 5 years. In order to
help organizations during this diagnostic
348 Chapter 11 Connecting the Dots

phase I have developed a number of as-
sessment tools that can be used at different
levels of the organization. The specific as-
sessment tool, the target audience for each
one and guidance for administering the
tool are summarized in TABLE 11-2. If you
are interested in using any of these tools
within your organization you can find out
how to access them by going to Jones and
Bartlett Learning’s Instructor Resources
Center which can be found at the following
link: (Danielle needs to complete this link).

TABLE 11-2 QI assessment tools

Assessment Target
Tool Name Audience Guidance for Administration of the Tool

From the Top Board Members, It is recommended that this survey be administered to the
Non-Execs and target audience(s) at a minimum of twice a year in order
Senior Leaders to gauge progress against a baseline on the 6 things all
boards and senior leaders should do to improve quality
and reduce harm: (1) setting aims to reduce harm this year,
(2) getting data and hearing stories about the impact of
harm, (3) establishing, monitoring and displaying system level
measures, (4) changing the environment, policies and culture,
(5) learning. . .starting with the board, and (6) establishing
executive accountability.

Issues and Board members, This is a shorter version of the first tool. With only 9 items
Priorities for Non-Execs and it will take less time than the first tool to administer but
Healthcare Senior Leaders will still identify a number of key issues that need to be
Leaders addressed if the organization is serious about making QI
part of daily work. This also should be administered at least
twice a year.

Improvement Senior Leaders, This tool has been designed to assess alignment of senior
Capability Directors, Middle and middle management leaders. It involves selecting
Managers and descriptive summaries related to 6 areas: (1) leadership for
Supervisors improvement, (2) results, (3) resources, (4) workforce and
human resources, (5) data infrastructure and management,
and (6) improvement knowledge and competence. For each
of these 6 areas, the tool provides a brief description of levels
of capability, ranging from just beginning, to developing,
to making progress, to significant impact, to exemplary.
Respondents are asked to select the one description that
best describes where they believe the organization is on each
dimension. The tool should be administered separately to
relevant target audiences, tabulated then brought to a joint
meeting to see how well aligned the various target audiences
are around the 6 key areas.

(continues)
 Building Capacity and Capability for Improvement 349

TABLE 11-2 QI assessment tools (continued)

Assessment Target
Tool Name Audience Guidance for Administration of the Tool

Science of All staff but This tool has been designed to help individuals gain a better
Improvement especially QI understanding of where they personally stand with respect
Self- team members to their knowledge of the basic principles, tools and methods
Assessment of the Science of Improvement (SOI). It can be administered
Tool at the beginning of a program or workshop designed to
build knowledge and skill sets related to the SOI and then
at the end of the program. It can also be administered
several months after a program offering to determine the
level of retention of SOI knowledge. Six skills to support
improvement are assessed: (1) supporting a change with
data, (2) developing a change, (3) testing a change, (4)
implementing a change, (5) spreading a change, and (6) the
human side of change.

Quality QI team This is the first of two tools designed to assess and
Measurement members, QI individual’s knowledge of and skill with measurement tools
Assessment coaches and and methods. The tool itself is shown in Exhibit 11-2 of this
Tool (short facilitators and QI chapter. It is short and can be administered as a pre-post
version) experts assessment when conducting workshops related to the
quality measurement journey. It can be administered to QI
team members and those expected to actually engage in
quality measurement activities.

Quality QI team This is a much more comprehensive measurement

Measurement members, QI assessment tool. It should not be administered frequently
Assessment coaches and or as often as the shorter version (Exhibit 11-2) and should
Tool (long facilitators and QI be aimed primarily at those expected to function as
version) experts measurement leads or experts for the organization.

■■ Determine the level of commitment that
the board and senior leadership have for
making quality their business strategy.
■■ Identify the total number of employees
and staff within the organization and then
stratify this total into appropriate categories
(e.g., board members, senior management,
physician leaders, nurses, allied health
professionals, and support staff). If the
organization is large, has multiple sites of
care, and/or covers a broad geographic area
we will use these factors to further stratify
the number of employees and staff into
relevant strata.
■■ Establish estimates of the amount of SOI
knowledge and skills the individuals in the
organization currently possess by applying
various assessment tools on the SOI.
Once the assessments are made and the
organizational demographics are determined
decisions can be made about the appropriate dose
of the SOI that needs to be administered to each
of the relevant groups. FIGURE 11-9 provides an
350 Chapter 11 Connecting the Dots

Science of improvement Board Sr. Sr. Nurse Admin QI team QI

topic mgmt clinicians mgrs. mgrs. ldrs. experts

History of QI
Profound knowledge

Quality as a business stratey

Model for improvement

PDSA testing
Understanding variation
Scale-up and spread
Construction of control charts

Legend

Minimal Moderate Maximum

dose dose dose
Note that the intensity of the color reflects the “does” of the science of improvement knowledge and skills
that would be administered to each respective group. The mechanisms for administering the allocated dose
would range from the IHI open school to the Improvement Advisor Professional Development Program.

FIGURE 11-9 Dosing the SOI to selected groups within an organization

example of how the dosing notion can be applied
to an organization. The column headings identify
the relevant groups of individuals within the
organization requiring varying does of the SOI.
The rows indicate selected content areas within
the SOI and the shades of color indicate the dose
or intensity (minimal, moderate, or maximum)
of the content being administered to each group.
Look at the last row in the matrix labeled “con-
trol chart construction.” Notice that the board,
senior leaders, and senior clinicians would all
receive a minimal dose of knowledge related to
the selection and construction of control charts.
Nurse managers and administrative managers
need to have a moderate dose on control chart
construction whereas the QI team leaders and
QI experts need to have a more intense dose of
control chart, construction knowledge.
After the initial decisions have been made
about who needs to know what, as shown in
Figure 11-9, then we typically proceed to deter-
mine how to deploy the dosing of knowledge
to the individuals within each group. It must be
realized that there is not a single set of numbers,
percentages, or a mathematical formula that can
be used to determine the appropriate doses of the
SOI for each group. As we have applied the dosing
approach to a variety of organizations, we have stressed
that the actual doses will vary depending on a
number of organizational characteristics, including:
■■ The number of individuals within each group
■■ The mix of services provided across the
organization
■■ The geographic region covered by the
organization
■■ The amount of development already pro-
vided on the SOI
■■ The resources (time as well as money) com-
mitted to learning and employee development
■■ The level of commitment that the board
and senior leadership are providing to make
quality the organization’s strategic focus
With these conditions in mind, the dosing
approach will produce different numbers for
different organizations.
 Building Capacity and Capability for Improvement 351

FIGURE 11-10 shows the initial 2016 dosing
strategy I helped the East London Foundation
Trust (ELFT) develop, and FIGURE 11-11 shows
their updated strategy for 2017. These figures
provide an update on the journey of each
identified group, estimates of the numbers
of individuals to be developed within each
group and a diagram (the triangle) that shows
the relative size of each group. Notice that in
Figure 11-10 the QI experts are around the edge
of all the other groups in the triangle. This was
done to show that the QI experts are supporting

Where are we?

Estimated number = 3,300

On track to train over
Requirement = introduction to
400 people through 5
quality improvement, identifying
six-month waves of
problems, change ideas,
learning between
testing and measuring
2014-16. First 3 waves
change
delivered with the IHI
Time frame = train 10-20% in 2 years

Experts

Estimated number = 250

On track. All Requirement = deeper Front line staff
senior staff understanding of improvement
being encouraged methodology, measurement and
to join QI training over using data, leading teams
next 2 years in QI
Time frame = train 30-50% in 2 years
Clinical leaders
Exp

er ts
Estimated number = 25-30
New need
er ts

Requirement = deeper

Exp
recognized. Developing Directorate
understanding of improvement improvement
Improvement
methodology, understanding leads
coaches program
variation, coaching teams
will train 30 QI
and individuals
coaches in 2015
Time frame = train 100% in 2 years Executives

Estimated number = 10
On track. Most
Requirement = setting direction
Executives will have
and big goals, executive
undertaken the ISIA,
leadership, oversight of
and all will have
improvement, being a champion,
received Board training
understanding variation to lead
with the non-Executives
Time frame = train 100% in 2 years

Currently have 3 Estimated number = 5

improvement advisors,
with 1.5wte
deployed to QI.
Will need to build more
capacity at this level.
Requirement = deep statistical
process control, deep improvement
methods, effective plans for
implementation & spread
Time frame = train 100% in 2 years
2016

FIGURE 11-10 East London Foundation Trust 2016 dosing strategy

East London NHS Foundation Trust
352 Chapter 11 Connecting the Dots
Psychology trainees-Pocket QI, embedded into QI project
teams with 4 bespoke learning sessions
Nursing students-intro to QI delivered within undergraduate
and postgrad syllabus, embedded into QI project
teams during student placements
363 completed Pocket
QI so far. All staff
receive intro to
QI at induction
690 graduated form
ISIA in 6 waves.
Wave 7 in 2017-18.
Refresher training for
ISAI grads.
47 QI coaches trained
so far, with 35 currently
active. Third cohort of
20 to be trained in 2017
58 current sponsors.
All completed ISIA.
Leadership, scale-up &
refresher QI
training in 2017
Currently have 6
improvement
advisors, with 3 further
QI leads in training
All Executives have
completed ISIA.
Annual Board session
with IHI & regular
Board development
Bespoke QI learning
sessions for service
users and careers.
Over 95 attended so
far. Build into recovery
college syllabus
FIGURE 11-11 East London Foundation Trust 2017 dosing strategy
East London NHS Foundation Trust
upstream
Working
Estimated number needed
to train = 4,000 Needs =
introduction to QI & systems
thinking, identifying problems,
Experts by experience
how to get involved
All staff
Estimated number needed
to train = 1,000 Needs =
Model for improvement, PDSA,
measurement and using Staff involved in or
data, leading teams leading QI projects
Exp
nce
er ts
Estimated number needed = 50
perie
QI coaches
Needs = deep
by e
x
understanding of method
by e
x
peri
& tools, understanding
er ts
variation, coaching teams
enc
Sponsors
Exp
e
Needs = Model for Internal
improvement, PDSA, experts
(QI leads)
measurement & variation,
scale-up and spread,
leadership for improvement Board
Estimated number needed
to train = 10 Needs = deep
statistical process control,
deep improvement methods,
effective plans
for implementation & spread
Needs = setting direction
and big goals, executive
leadership, oversight of
improvement,
understanding variation
Needs = introduction to
QI, how to get involved
in improving a service, practical
skills in confidence-building,
presentation, contributing ideas
2017
 Building Capacity and Capability for Improvement
all the other groups and helping them apply the
SOI. In Figure 11-11, however the QI experts
are now part of the triangle as a group unto
themselves. The group surrounding the triangle
is now called the “Experts by Experience.” This
group emerged from the various workshops
we conducted and stepped forward to be local
experts in training. They have deepened their
knowledge of the SOI by participating in teams,
leading improvement projects and working
with the ELFT IAs to obtain a deeper dose of
the SOI. These individuals have become part
of the self-sustaining infrastructure that ELFT
is building.
Finally, notice that ELFT has now developed
10 IAs. These individuals are graduates of our
yearlong IHI Improvement Advisor Professional
Development Program (IAPDP) and serve as
the full-time deep knowledge experts in the
SOI. They support improvement teams, the
improvement coaches, and managers and also
provide strategic guidance to senior leaders on
the SOI. In short, the ELFT has deployed the
dosing strategy in a very deliberate and effec-
tive way. They have viewed improvement as a
journey not a short trip. They have developed
a very focused yet flexible strategy to apply the
dosing approach as a primary way to build both
capacity and capability.
What Core Skills of
Improvement Get Dosed?
The first thing to realize when discussing the
content being delivered on the SOI is that the
topics do not differ for the various groups of indi-
viduals involved. The content of the SOI remains
pretty constant across all the groups within an
organization. What differs, however, is the dose
or intensity of the content being delivered. For
example, everyone in the organization needs to
understand how the aims or target conditions
for teams and the organization as a whole are
being developed and how they are phrased. At
the IHI, we build all team and organizational
aim statements around the first question in the
353
MFI (What are we trying to accomplish?). In
constructing an aim statement, therefore, we
expect it to identify:
■■ The boundaries of the system to be improved
(scope, patient population, processes to
address, providers, beginning and end, etc.)
■■ Specific numerical goals for the outcome
measure(s) that are ambitious but achievable
(How good do you want to be?)
■■ The time frame (By when do you pan to
achieve the numerical goal?)
■■ Any guidance on issues or circumstances
that may have an impact on the project’s
success or progress (e.g., potential changes
in sponsorship, resource issues, or opera-
tional issues such as a pending merger or
even construction)
Basic awareness of the organization’s structure
and format for an aim statement is a minimum
requirement (i.e., a minimal dose). This will al-
low everyone in the organization to understand
an aim statement when they read one or see it
presented in a meeting. But team coaches and
IAs, on the other hand, need to have a deeper
dose or exposure to building aims from scratch,
how to conduct a the dialogue around whether
the aim is reasonable or overly ambitious, and
how modify the aim in light of the baseline
data collected. The topic of aim statements does
not vary from one group to another. But the
dose does. This same principle applies to all
the content topics included in learning about
the SOI.
The essential skills required to make quality
thinking and practice the guideposts for how
an organization approaches work consist of the
following content categories:
QI Philosophy and Theory
■■ Understanding the differences between
quality assurance (QA), QC, and QI (see
Chapter 1)
■■ Having a firm grounding in the SOI theories
articulated by Dr. Water Shewhart, Dr. W.
Edwards Deming, and Dr. Joseph Juran.
354 Chapter 11 Connecting the Dots
■■ Selecting an approach to QI that can serve
as a roadmap for the organization’s quality
journey. Note that more important than the
decision as to which approach or model to
QI is selected is what Dr. Deming referred to
as “constancy of purpose.” (Refer to note 9 in
this chapter for more details on this point.)
■■ Organizations that follow a “flavor of the
month” approach to QI will not only lack
direction and focus but will also send very
mixed messages to the staff about where
the organization is headed.
QI Methods
■■ Applying systems thinking to all aspects of
the organization
■■ Viewing all work as an interconnected set
of processes
■■ Using analytic statistical thinking and
methods to understand the variation that
lives within the organization’s data. This
requires a fundamental shift away from the
use of enumerative statistical methods (e.g.,
comparing averages or using red, amber, and
green rating and ranking schemas to make
conclusions about performance).
QI Tools
There are a variety of QI tools that need to be
understood and used to help diagnose, analyze,
and drive improvement work. The various QI
tools are critical to the success of improvement
teams but they are not to be used in isolation
from improvement strategies and action. In-
dividuals and teams often become enamored
with the use of tools such as flowcharting or
cause and effect diagrams with little or no un-
derstanding of when the tools should be used
and, more important, how a particular tool
fits into the QI journey. There are many good
resources on a wide spectrum of qualitative
and quantitative tools used in QI work. A few
that are quite relevant to health and social care
services include publications by Graham and
Cleary (1992), Murray and Murray (1997), AIP
(2007), and Goal QPC (2008). In addition to
these publications, you may be interested in a
series of Whiteboard and On Demand videos
I have made that provide overviews of the SOI
and QI tools, methods, and concepts. These
videos are all free on the IHI website (www.ihi
.org) or by following these direct links:
■■ R. Lloyd’s Whiteboard videos on tools,
methods and concepts:
https://tinyurl.com/q4s4pe7
■■ R. Lloyd’s On Demand Videos on the SOI:
https://tinyurl.com/moxlct3https://tinyurl
.com/knt8cyz
https://tinyurl.com/ldg252k
At this point many of you are probably
using QI tools in your daily work. Tools such
as flowcharts, cause and effect diagrams, Pareto
charts, force field analysis, scatter plots, two-way
tables and SPC charts are some of the more
frequently used tools. The tools are usually
organized into categories that capture the pur-
pose of the various tools (e.g., understanding
the system, gathering data and information,
organizing information, analyzing variation, or
understanding relationships). Many of the QI
tools are aimed at making sense of or putting
order into quantitative issues facing a team.
Frequently overlooked, however, are a variety
of tools designed to help teams identify ideas
for improving a process, tools to assist a team
in deciding which improvement idea(s) they
would like to pursue, and tools for setting
priorities. These tools are aimed at generating
ideas (divergent thinking) and then helping a
team decide which improvement idea(s) they
wish to pursue (convergent thinking). The tools
related to divergent and convergent thinking
are summarized in FIGURE 11-12.
Many teams become overly focused on tools
during their QI journey. Like the run and Shewhart
charts, however, the tools are there to help a team
along their way not be an end in themselves.
Just like each surgical tool or instrument has a
specific purpose so too do the QI tools. Part of
the dosing strategy is to (1) make people aware
of the tools, (2) know how to construct and
 Building Capacity and Capability for Improvement 355

Affinity Diagram which is designed to provide general awareness

Nominal Group Co Multi-voting to large numbers through online modules that
Technique nv

g
my colleagues and I have designed. Many uni-

in
er

nk
Brainstorming ge Rank ordering
versities around the globe have Open School

i
th
nt
th Structured

nt
in discussion Chapters for students enrolled in the health
e
rg
ive kin
g sciences. It is a light dose designed to build
D

Few ideas or None Many Ideas Few Ideas
• Teams start with a few ideas for improvement
or none.
• They need to engage in divergent thinking to
open up their brains and generate ideas.
• Once a team generates many ideas, however,
they need to engage in convergent thinking to
reduce the many to the vital few that they
can test.
FIGURE 11-12 Divergent and convergent thinking
tools
interpret the tools, and, most important, (3) know
when and under what conditions a specific tool
should be used.
The final part of understanding the core
skills of improvement that need to be dosed
is to develop programs that deliver the right
dose of the SOI at the right time. FIGURE 11-13
provides an example of different SOI programs
we offer at the IHI but that all deliver a different
dose of the same content. For example, at the
top of Figure 11-13 is the IHI Open School,
awareness but one that can be applied easily
and broadly. Over a million modules in the
Open School have been accessed to date by
students and healthcare professionals. When
organizations request the next dose of the
SOI we offer face-to-face workshops. Two of
our flagship programs are in the middle of
Figure 11-13 and have been identified with a
deeper shade of grey indicating a slightly heavier
dose of the SOI. The Improvement Science
in Action (ISIA) Program is a 3-day offering
designed for a maximum of 200 participants.
It provides all the essential SOI concepts, tools,
and methods needed to initiate improvement
projects. Virtual sessions are incorporated with
this program so that participants can present
their project updates and receive feedback. The
Improvement Coach Professional Development
Program (ICPDP) receives the deepest shade
of color. This program is designed for 30–50
participants and provides a deeper dive into the
human side of change or as we at IHI refer to it
as, the “care and feeding of teams.” This program
is a 7-day offering, delivered as a 2–3–2 day or

IHI Open School Designed for the masses to build

awareness

• ISIA designed for 100-200

ISIA & ICPDP • ICP designed for 30-50
• Building team and project based skills

IA • Designed for 20-25 participants

• Building deep knowledge in the SOI

FIGURE 11-13 Dosing the content of IHI program offerings

356 Chapter 11 Connecting the Dots
a 3–4 day design. A virtual pre-call session is
required as well as additional calls between the
actual workshops. During the virtual sessions
we provide additional SOI content and the par-
ticipants provide updates on their experiences
with coaching a team. The ICPDP is intended
for individuals who have already attended the
ISIA program and have a working knowledge
of how to apply the SOI to projects and teams.
The deepest and most intense dive into
the SOI shown in Figure 11-13 is found in the
IAPDP. This 12-day workshop is spread out
over a year of learning with eight of IHI’s senior
faculty. Each participant must have a project that
they are expected to move to the results stage
of improvement. The program consists of three
4-day workshops with 11 virtual sessions spaced
between the workshops. Upon completion of
all the requirements participants are presented
with a certificate of completion and can add
the title of IHI IA to their resume.
Whether an organization has developed
its own SOI offerings or participates in those
provided by outside groups, the key point is
that you need to have access to a variety of
programs that (1) provide consistent messages
about the SOI, (2) are dosed appropriately for
the intended audience or group of employees,
(3) are well grounded and designed around adult
learning theory, (4) are focused on applying the
SOI knowledge to actual improvement work,
and (5) are flexible enough to accommodate
the organization’s strategic and operational
changes.
How Do We Create the Conditions
for a Successful Dosing Strategy?
Creating the conditions for change is probably
THE most important aspect of the quality jour-
ney. In agricultural circles there is a great bit
of guidance that applies here.13 The expression
is that farmers do not grow crops; they create
conditions under which crops can grow. For
farmers this is determined by finding out how
well they prepare the soil, which then determines
how the seeds they plant will take root and grow.
Healthcare leaders and managers are like farmers
in this respect. They are charged with creating
the conditions under which the delivery and
improvement of healthcare services can grow
and flourish.
A basic premise of the dosing approach is
the need to have an organization-wide plan that
allows for dispensing the appropriate “dose” of
the SOI to the appropriate individuals over a
prescribed period of time. But, the initial dose for
any group of individuals will not be sufficient to
transform the organization. For example, when
we administer a medication to a patient we do
not tell them, “Okay, take one pill twice a day
then stop after the first day.” The appropriate dose
needs to be administered for the proper period
of time; otherwise it is not effective. Similarly,
the dose of the SOI does not produce the desired
results after one application. Whether you are
receiving a light, moderate, or heavy dose of the
SOI it typically needs to be administered multiple
times over a defined period of time. The pitfalls
to avoid, therefore, when deciding to undertake
the dosing approach include (1) lack of constancy
of purpose; (2) lack of leadership commitment;
(3) lack of a strategy for deploying QI concepts,
methods, and tools throughout the organization;
(4) the belief that you can apply one dose of
the SOI and it will solve all the organization’s
problems; and (5) failure to connect the SOI
learning to daily work.
In this chapter I have outlined four key
activities that I believe will help leaders connect
the dots and greatly enhance their organization’s
chances of demonstrating excellence:
■■ Adopting quality as a business strategy
■■ Developing a learning system to support
improvement
■■ Linking measurement to improvement
■■ Building capacity and capability for
improvement
There is no doubt that all health and social
service providers are going to be under increas-
ing pressure to demonstrate greater value and
improved performance. By connecting the dots
with the suggestions discussed in this chapter,

I sincerely believe that we will be able not just to
cope with these pressures but flourish in spite of
them. As Dr. Deming wrote in Out of the Crisis,
“Who will survive? Companies that adopt con-
stancy of purpose for quality, productivity, and
service, and go about it with intelligence and
perseverance, have a chance to survive” (1992,
p. 155). But, in every one of his seminars I at-
tended he would always throw in an afterthought
and say, “Survival is not mandatory!” Our goal
should not be merely to survive but rather to
thrive! I hope the information presented in this
book has given you a framework to enhance
your journey toward excellence and thriving.
Notes
1. From my perspective the dominant
changes facing the healthcare industry
across the world over the next 5 years will
be centered on the following key topics:
• Declining trust in the medical
community
• More public scrutiny of the healthcare
profession by people who are not
trained as clinicians
• Increased demand on the part of
patients and families to become more
involved in making decisions about
their care and treatment
• Less autonomy and income for
physicians
• Increased pressure to create safe
environments for the delivery of
medical services
• Greater demand for transparency
and the release of data related to cost,
quality, service, access, and safety
• Increased focus on containing health-
care costs and adding value for the
money spent.
2. API develops methods, works with leaders
and teams, and provides education and
training to help organizations improve
their products and services and to build
their capability for ongoing improve-
ment. The initial principals of API, Jerry
Notes 357
Langley, Ron Moen, Tom Nolan, and
Lloyd Provost, worked with Dr. Deming
at his 4-day seminars throughout the
1980s. Cliff Norman and Kevin Nolan
joined the API team in 1988. API has
been a strategic partner with IHI for over
20 years building knowledge of the SOI
within IHI and within IHI’s customers.
Additional detail on API can be found at
http://www.apiweb.org/.
3. I have modified the language describing
these three principles slightly from how
they appear in the API book Quality as
a Business Strategy (1998). This was done
only for clarity and style that made the
original wording more compatible with
the style of this book. The essence of the
principles remains consistent with that
which the API authors initially wrote.
4. Readers interested in the details behind
each of these activities are encouraged to
read chapter 13 in the 2009 edition of the
Improvement Guide and pages 260–292
in the 1996 edition (Langley et al., 1996,
2009). Those interested in a deeper dive
into QBS should obtain a copy of Quality
as a Business Strategy (API. 1998), which
is available through Process Improvement
Products (www.pipproducts.com).
5. Two good references on the application
of QI to nonmanufacturing situations
can be found in Quality Is Personal by
Harry Roberts and Bernard Sergesketter
(1993) and Total Quality Ministry by
Walt Kallestad and Steve Schey (1994).
The Quality Is Personal book provides
an excellent discussion on how to apply
QI thinking and methods, especially
checklists, to your own personal growth
and development. As the name suggests
the Total Quality Ministry book applies
the total quality management (TQM)
ideas to the operations of churches and
ministerial practice. When I worked at
Lutheran General Health System in Park
Ridge, Illinois, I shared this book with
several of the chaplains I worked with.
358 Chapter 11 Connecting the Dots
After reading the book one of the chaplains
took the initiative to write a brief paper for
the pastoral care staff titled “Total Quality
Chaplaincy.” These are all good examples
of how quality thinking and methods can
be applied to any aspect of life.
6. I have had participants in class and even
some colleagues who take issue with
Deming’s term “profound knowledge.”
I have had comments such as “Well,
isn’t that profound?” Or, “Profound to
whom? It doesn’t seem profound to me!”
The term profound seems to stir in some
folk a feeling that Deming was being too
academic and even a bit intellectually
exclusive. But as Langley et al. (2009,
p. 75) write: “The word profound denotes
the deep insight that this knowledge offers
into how to make changes that will result
in improvement in a variety of settings.”
If I sense that people are either getting
confused with this term or starting to be
cynical about its descriptive qualities, I
will merely say that Deming identified
four key components that are essential
to making improvement a mainstay
of organizational performance. Then I
name the four components. So you will
have to decide how you wish to convey
this important point that Deming was
making. What is important is the use of
the four components. Don’t let the word
“profound” become a stumbling block to
learning and change.
7. Deming originally called these four
components (1) appreciation for a system,
(2) knowledge about variation, (3) theory
of knowledge, and (4) psychology. The
slight change in the wording of these
components was offered by Langley et al.
(2009) to try to make the theory more
accessible to others and more descriptive
of how the theories are to be applied.
Deming was still evolving and revising
his SoPK when he passed away in 1993.
Thinking and writing about the SoPK was
continued by the API team. Additional
details can be found in chapter 1 of the
API book Quality as a Business Strategy
(1998), which is available through Process
Improvement Products (www.pipproducts.
com) and in chapter 5 of Transforming
Health Care Leadership by Maccoby,
Norman, Norman, and Margolies (2013).
8. As you scan these two lists notice where
salary appears in the rankings. First, it
is a factor that appears on the extreme
dissatisfaction side of the picture. Sec-
ond, it falls in the fifth position not the
very first dissatisfying factor. So although
salary is always thought of as a major
motivator, Herzberg’s work demonstrates
that salary is not quite as important as
many think. I think this finding should
be of particular interest to healthcare
and social service organizations, which
are typically not-for-profit organizations.
Rarely are health and social service
workers even eligible for stock options,
incentives, or bonus plans. These individ-
uals are usually quite pleased to receive
an annual cost-of-living adjustment. So,
the role of money needs to be viewed in
light of a bigger systems view. According
to Deming (1994, p. 113–114), “An award
in the form of money for a job done for
the sheer pleasure of doing it is demor-
alizing, overjustification. Money, above
a certain level, is not enticement. Money
may entice someone that knows that he
is inferior. Certainly a boss should give
a pat on the back for a job well done.”
9. There are a number of models or ap-
proaches to guide an organization’s
improvement journey. Besides Deming’s
approach based on his notion of profound
knowledge and the API’s MFI, the other
leading approaches or models that have
been used in healthcare settings include:
• Juran’s Quality Trilogy and its related
methods
• The Toyota Production System (TPS)
or as it has become known in health-
care settings Lean Six Sigma and its

derivatives—Define, Measure, Analyze,
Improve and Control (DMAIC) and
Design for Six Sigma (DFSS)
• FOCUS PDSA
• Malcolm Baldrige National Quality
Award
• European Foundation for Quality
Management (EFQM), a.k.a. the
European Quality Award
• International Organization for Stan-
dardization (known as ISO)
Although all of these approaches have
been show to work effectively the key
for me is the issue that Deming kept
stressing—constancy of purpose. Too
many organizations lack constancy of
purpose and keep switching models or
approaches to improvement. They are
all too often looking for quick fixes to
solve immediate problems, which most
often are aligned with budget and finan-
cial maximization not quality, safety, or
customer experience. Organizations
cannot be successful in the long run if
they keep changing their approach to
achieving organizational excellence.
Although the various approaches or
models do have common elements,
trying to apply Juran’s quality trilogy
one month, then Lean thinking the
next month and Six Sigma the next not
only creates confusion among the staff
but also creates strategic inconsistency
for the organization. So, as your orga-
nization reviews and evaluates different
approaches or models for improvement
my recommendation is to pick one and
stick with it. Modify and adapt the ap-
proach or model you select to meet your
local conditions and strategic objectives
but don’t keep jumping from one model
to another. Only confusion and lack of
consistency will result.
10. I asked this question years ago at the
first learning session for the Scottish
Patient Safety Program collaborative we
were running. I had a resounding and
Notes 359
overwhelming positive response. This
was a wonderful response given that it
was in 2007 and we were just getting this
major countrywide collaboration started.
I looked at my colleagues from the IHI
and we all smiled at each other. One of
my friends even gave me the old “thumbs
up” gesture. All of us thought that this
was a very good sign that they all knew
what PDSA meant. At the coffee break we
found out what PDSA actually meant not
only in Scotland but also throughout the
United Kingdom and it was not PDSA.
During lunch, those of us from across
the pond were proudly presented by one
of our Scottish hosts with PDSA pins.
They were lovely little label pins that in
bold blue and white letters stated “PDSA.”
We thought this was great. Not only did
everyone know what PDSA stood for but
they already had pins to help reinforce the
ideas. Then the presenter of the PDSA
pins asked the audience to shout out what
PDSA meant: People’s Dispensary for Sick
Animals! The Yanks learned a good cul-
tural lesson that day. The rest of the story
is that I went to a People’s Dispensary for
Sick Animals location in Edinburgh and
made a donation in order to obtain all the
PDSA pins they could provide. So, today,
I proudly give out PDSA pins to those I
mentor and teach. Those from the United
Kingdom ask me why I am giving them a
pin from the People’s Dispensary for Sick
Animals. I simply say, “Now really, think
about it for a minute.”
11. A number of the IHI’s strategic partners
have demonstrated enhanced results by
applying the MFI. Examples of these
organizations include Kaiser Perma-
nente, Cincinnati Children’s Hospital
Medical Center, Virginia Mason, ELFT
(UK), NHS Scotland, Jonkoping County
Council (Sweden), and Hamad Medical
Corporation (Qatar).
12. The term “the seven spreadly sins” emerged
when we were running the Safer Patients
360 Chapter 11 Connecting the Dots
Initiative (SPI) in the United Kingdom. We
rotated meetings among England, Wales,
Scotland, and Northern Ireland. During
one of the sessions in Northern Ireland
we put slides up on the screen to discuss
spread initiatives, with a rather bland title
about the seven things you should not
do if you want to be successful at spread.
A nurse from Ulster hospital who was
looking pensively at our rather bland title
offered “the seven spreadly sins” as an
alternative. We immediately said it was a
brilliant revision. We changed the title on
the spot and it has been with us ever since.
13. Because my PhD is in agricultural eco­
nomics and rural sociology from the
Pennsylvania State University I frequently
provide a variety of ag stories and refer-
ences while teaching. The question I get
quite often is, “How did you end up in
health care when you come out of an ag
background?” Long story short, I have
done research on rural health issues and
community health. I also have worked
in the community development field
with a focus on social well-being and
measurement of the impact of social
policy. With no formal degrees in health
care or medicine I bring what Deming
called an “outside view.” My colleagues
from Associates in Process Improvement,
who taught with Dr. Deming and have
written several of the leading books
on the SOI, do not have any degrees in
health care or medicine either. My point
is that you do not need to have formal
degrees in the healthcare field, or any
other field for that matter, to be able to
help people in that particular discipline
make improvements. The SOI can be
applied to any discipline.
References
Ackoff, R. Creating the Corporate Future. New York: John
Wiley & Sons, 1981.
Associates in Process Improvement, Quality as a Business
Strategy. Austin, TX: API-Austin, 1998.
Associates in Process Improvement. The Improvement
Handbook: Model, Methods and Tools for Improve-
ment. Austin, TX: Associates in Process Improvement,
2007.
Argyris, C. Overcoming Organizational Defenses. New York:
Prentice Hall, 1990.
Balas, E., and S. Boren. “Managing Clinical Knowledge for
Health Care Improvement.” In Yearbook of Medical
Informatics 2000: Patient-Centered Systems, edited by
J. Bemmel and A. T. McCray, 65–70. Stuttgart, Germany:
Schattauer Verlagsgesellschaft mbH, 2000.
Barker, P., A. Reid, and M. Schall. “A Framework for Scaling
up Health Interventions: Lessons from Large-Scale
Improvement Initiatives in Africa.” Implementation Science
11 (2016): 12, Open Access. This article is distributed
under the terms of the Creative Commons Attribution
4.0 International License (http://creativecommons
.org/licenses/by/4.0/).
Berwick, D. Escape Fire: Designs for the Future of Health
Care. San Francisco: Jossey-Bass, 2004.
Bevan, H. “How Can We Build Skills to Transform the
Healthcare System?” Journal of Research in Nursing 15,
no. 2 (2010): 139–148.
Churchman, C. W. The Systems Approach. Philadelphia: Dell
Publishing Co., 1968.
Deming, W. E. Out of the Crisis. Cambridge, MA: MIT
Press, 1992.
Deming, W. E. The New Economics, 2nd ed. Cambridge,
MA: MIT Press, 1994.
Drucker, P. Management Tasks, Responsibilities, Practices.
New York: Harper & Row, 1973.
Forrester, J. Principles of Systems. Cambridge, MA: Produc-
tivity Press, 1986.
Forrester, J. “The Counterintuitive Behavior of Social Systems.”
Technology Review 73, no. 3 (1971): 52–68.
Goal QPC. The Memory Jogger II: Healthcare Edition. Salem,
NH: Goal QPC, 2008.
Graham, J., and M. Cleary. Practical Tools for Continuous
Improvement: Vols. 1 and 2. Dayton, OH: PQ Systems
Publishing, 1992.
Herzberg, F. “One More Time: How Do You Motivate
Employees?” Harvard Business Review 81, no.1 (2003):
86–96.
Kallestad, W., and S. Schey. Total Quality Ministry. Minne-
apolis: Augsburg Press, 1994.
Kohn, A. No Contest: The Case Against Competition. New
York: Houghton Mifflin, 1986.
Kohn, A. Punished by Rewards: The Trouble with Gold Stars,
Incentive Plans, A’s, Praise and Other Bribes. New York:
Houghton Mifflin, 1993.
Langley, G., R. Moen, K. Nolan, T. Nolan, C. Norman, and L.
Provost. The Improvement Guide: A Practical Approach
to Enhancing Organizational Performance, 2nd ed.
San Francisco: Jossey-Bass, 2009.
Lloyd, R. “Improvement Tip: Quality Is Not a Department.”
Institute for Healthcare Improvement website, 2016.

http://www.ihi.org/resources/Pages/Improvement
Stories/ImprovementTipQualityIsNotaDepartment.aspx
Maccoby, M., C. Norman, C. J. Norman, and R. Margolies.
Transforming Health Care Leadership. New York: Wiley
& Sons, 2013.
Massoud, M. R., G. A. Nielsen, K. Nolan, T. Nolan, M. W.
Schall, and C. Sevin. A Framework for Spread: From
Local Improvements to System-Wide Change. IHI
Innovation Series white paper. Cambridge, MA: Institute
for Healthcare Improvement; 2006. http://www.ihi.org
/resources/Pages/IHIWhitePapers/default.aspx
McCannon, J., M. Schall, and R. Perla. Planning for Scale: A
Guide for Designing Large-Scale Improvement Initiatives.
IHI Innovation Series white paper. Cambridge, MA:
Institute for Healthcare Improvement; 2008. http://www
.ihi.org/resources/Pages/IHIWhitePapers/default.aspx
Moen, R., and C. Norman. “Circling Back: Clearing Up
Myths About The Deming Cycle and Seeing How It Keeps
Evolving.” Quality Progress (November 2010): 22–28.
Moen, R., and C. Norman. “Always Applicable: Deming’s
System of Profound Knowledge Remains Relevant for
Management and Quality Professionals Today.” Quality
Progress (June 2016): 47–53.
References 361
Murray, S., and B. Murray. Practical Tools for Healthcare
Quality. Dayton, OH: PQ Systems Publishing, 1997.
Nelson, E., P. Batalden, and M. Godfrey. Quality by Design:
A Clinical Microsystems Approach. San Francisco: John
Wiley & Sons, 2007.
Norman, C. Quality as a Business Strategy: An Overview.
Austin, TX: Associates in Process Improvement, 2007.
Roberts, H., and B. Sergesketter. Quality Is Personal. New
York: Free Press, 1993.
Rogers, E. Diffusion of Innovations, 4th ed. New York: Free
Press, 1995.
Scherkenbach, W. Deming’s Road to Continual Improvement.
Knoxville: SPC Press, 1991.
Schultz, L. Profiles in Quality: Learning from the Masters.
White Plains, NY: Quality Resources, 1994.
Senge, P. The Fifth Discipline: The Art and Science of The
Learning Organization. New York: Doubleday, 1990.
Senge, P., C. Roberts, R. Ross, B. Smith, and A. Kleiner. The
Fifth Discipline Fieldbook: Strategies and Tools for Building
a Learning Organization. New York: Doubleday, 1994.
von Bertalanffy, L. General System Theory: Foundations,
Development, Applications, rev. ed. New York: Penguin
University Books, 1968.
Index
A case studies, quality measurement principles
breast cancer patient’s clotting levels, managing, 287–290
cardiovascular event, predicting, 259–265
administrative data, analysis, 2
emergency department fast track, 293–296
aggregate statistics. See data analysis
flash sterilization, 283–285
analytic studies, 29–33, 85, 91
group B streptococcus in pregnant women, 290–293
area of opportunity, Shewhart chart, 231, 233
hospice/911 paramedic system partnership to improve
ASQ Statistics Division, 98
care, 315–317
astronomical data point, 198–199
improving access to community services for mental
attributes data, 226
health and community health patients, 317–326
Austin, C., 160
operational definition of readmission, clarifying, 286
pain management for hip and knee replacement
patients, 306–314
pressure ulcer prevention, 272–275
B reducing ventilator-associated pneumonia, 300–305
sampling central line infections, 265–266
Babbie, E. R., 121 sampling Medicare insurance audits, 267–268
Bader, B., 149 staffing effectiveness, evaluating, 275–283
balanced scorecard, 101, 149 tracking patient complaints, 296–300
Baldrige Award, 18 tracking patient falls, 268–271
bar charts, 235 central tendency, measures of, 169, 207, 216
baseline data, 274, 309 Chambers, D., 203, 213–214, 219–220, 225, 230
benchmarking, 155–157 CL (center line), 211–212
Berman, H., 149 Cleary, M., 158, 354
bias, sampling, 115 cluster sampling, 119–120
binomial distribution, 240 Codman, E. A., 2–4, 6, 12, 14, 16
Blalock, H., 30 Coffey, R., 163
Blank, R., 213–214, 250 common cause variation, 176, 179–183. See also
Block, S., 169 variation
business strategy, adopting quality as, 332–336 comparative charts, 262
Conceptland, 99–100
continuous data, and control charts, 226
control chart decision tree, 230
C control charts, 229–230, 243–245
control limits
c-charts, Shewhart charts, 240–241 for control charts calculating, 214–215
Caldwell, C., 94, 96 freezing, 204
Campbell, D., 35, 43, 117 stair-step, 234
capability building, improvement, 346–357 trial, 215
capability of process, 180, 213, 251 vs. specification limits, 213
capacity building, improvement, 346–357 convenience sampling, 121

© Michal Steflovic/Shutterstock

363
364 Index

convenient quota sampling, 121

Conway, W., 99 F
count data. See attributes data
Fiske, D.W., 60
CQI, 21
Fitzpatrick, T., 117
culture of service excellence, creating, 46–55
focus groups, 64
customer, primary, 58–60
Forcese, D., 162
Freiberg, J., 47–48
Freiberg, K., 47–48

D funnel experiments, of Deming, 19–20

Daniel, W., 117, 212, 216

dashboards, 143–158
data G
gathering of, steps for, 160–163 g-chart, 247–249
data analysis and output, 162 Garvin, D., 54
information for decision making, 162–163 Gaucher, E., 163
interpretation of the results, 162 Gitlow, H., 85, 91
measurement and data collection, 161–162 Godfrey, B., 102, 337
select and define indicators, 161 Gonick, L., 117, 216
theoretical concepts, 160 Graham, J., 151, 334
vs. information, 159–163
data analysis, 35–38
static vs. dynamic approaches, 163–167
data collection, 35–37, 113–117, 126
data types, Shewhart charts, 226–229 H
defectives, 226–228 Hare, L., 250
defects, 228–229 Hayes, B. E., 45, 68
Deming, W. Edwards, 18–19, 27–29, 32, 35, 37, health care system
48–50, 54, 85, 90, 99, 103, 122–123, 131, accountability, 7
147–148, 157, 163–164, 172–175, 178–179, and consumer movement, 14
182, 185, 219, 223, 331–334, 336–339, 341, HMOs, 14
353–354 reform, 2, 6
design of experiments (DOE), 328 Henkel, R., 32, 38
Dillman, D. A., 68, 74–76, 89 Hess, I., 117
dispersion, measures of, 169, 216 high-tech tools, for data collection, 66
Donabedian, A., 3, 101 histograms, 277
dosing strategy, 356–357 Hull, R., 48
Duncan, A., 117, 214, 230, 242 hypotheses, developing, 162
dynamic approaches (data analysis), 163–167

E i-chart, Shewhart charts, 236–238
effectiveness, quality measurement and, 30–33
efficacy, quality measurement and, 30–33
efficiency, quality measurement and, 30–33
elements of control, 212
end results, description of, 2
enumerative study, quality measurement and, 28–29
experiential shopper, for data collection, 66
experimental designs, for research, 328
expert sampling, 122
extrinsic motivation, 49
I
improvement. See also quality improvement (QI)
and quality measurement, 339–346
capability building for, 346–357
capacity building for, 346–357
core skills and, 353–356
indicators, 101–108
benchmarking, role of, 155–157
development worksheets, 127–132
selection of, 104–109
strategic dashboard
 Index 365

evolution of, 144–151 measurement, quality. See quality measurement

organizing into, 143–158 measurement, self-assessment tool, 339
vital, focusing on, 151 Measurementland, 100–101
information vs. data, 159–163 median, statistical, 41
intrinsic motivation, 49 Miller, D., 117
Ishikawa, K., 116–117, 214 mode, statistical, 41
ISO registration, 268, 359 Montgomery, D. C., 214, 219, 232, 242
Morrison, D., 32, 38
Mosser, G., 33

J moving range chart, 236

mystery shoppers, 66

judgement sampling, 122–123, 125–126

K Namboodiri, N., 328
Kamberg, C., 31
Kerlinger, F., 328
Key Quality Characteristics (KQC), 57–60
KQC. See Key Quality Characteristics (KQC)
L quota sampling, 121–122
Langley, G., 45, 48, 122, 163, 336–337, 341, 358
Lastrucci, C., 160, 163
LCL. See lower control limit (LCL)
leadership rounds, VOC and, 65
Leaon, L., 54
learning system, development of, 336–339
Likert scale, 79
line graphs, 278
listening
and voice of the customer (VOC), 45–46
points, 60
Lloyd, R., 214–215, 219, 230, 242
lower control limit (LCL), 212–213, 215, 217–218,
220, 234
N
Nelson, B., 49
Nelson, E., 102, 144, 147–149, 157
Nightingale, Florence, 2, 4, 12
nonconforming units, 226
nonprobability sampling, 120–126
convenience sampling, 121
judgment sampling, 122–123, 125–126
norms, 46
Nugent, W., 150
numerical illiteracy, overcoming of, 171–185
variations
common vs. special causes of, 179–183
understanding of, conceptually, 171–179
O
observation, for data collection, 35–37
Onnias, A., 151
operational definitions, 109–112
organizational components, VOC and
human resource issues, 47–48
measurement and design, 54–55

M motivation, 48–52
organizational design, 52–54
organizational culture, 47
Maddox, B., 121
Ostrom, C., 32
Mann, N. R., 99
outcome measures, 36, 353
Marsick, V., 54
matched samples, 31
McDowall, D., 32
McGlynn, E., 31
mean, statistical, 41
measurement data, 226
P
measurement scales, 78 p-charts, 239
measurement, of performance. See performance Pareto diagram, 151, 327
measurement Pareto principle, 151
366 Index

patient-centered care, definition, 15 and improvement, 339–346

Patton, S., 65 data collection plans, developing, 103–117
PDSA cycle, 163, 341–342 effectiveness and, 30–33
Peabody, Francis, 2, 4, 12–15, 23 efficacy and, 30–33
Pedhazur, E., 328 efficiency and, 30–33
performance measurement, 33–39 importance of, 25–43
data analyzing, 35–38 indicator development worksheet, 127–132
data collection and sample size, 35–37 milestones in, 93–141, 101–108
measurement aim, 34–37 Conceptland and, 99–100
testing methods and observability, 35–37 indicator, 101–108
three faces of, 33–39 Measurementland and, 100–101
personal interviews, VOC and, 65 nonprobability sampling, 120–126
Peter, L., 48, 178 convenience sampling, 121
Plsek, P., 151, 214 judgment sampling, 122–123, 125–126
point-of-service (POS) assessment, 61 quota sampling, 121–122
Polar Area Diagrams, 2–3 operational definitions, 109–112
POS (point-of-service assessment). See point-of-service performance measurement and. See performance
(POS) assessment measurement
Posavac, E., 31 philosophy development, 93–95
postservice assessment, 61 principles, application of, 259–326
preservice assessment, 60 clarifying the operational definition of
probability sampling, 117–120 readmission, 286
cluster sampling, 119–120 emergency department fast track, 293–296
simple random sampling, 117–118 evaluating staffing effectiveness, 275–283
stratified proportional random sampling, 119 flash sterilization, 283–285
stratified random sampling, 118–119 group B streptococcus in pregnant women, 290–293
systematic sampling, 119 hospice/911 paramedic system partnership to
process capability, 157, 177, 180, 213, 251 improve care, 315–317
profound knowledge, 48, 163, 336–337 improving access to community services for mental
proportional stratified random sampling, 124 health and community health patients, 317–326
purposive sampling, 122 managing a breast cancer patient’s clotting levels,
Pyzdek, T., 214, 220, 230, 242 287–290
pain management for hip and knee replacement
patients, 306–314
predicting a cardiovascular event, 259–265
Q pressure ulcer prevention, 272–275
reducing ventilator-associated pneumonia, 300–305
QA. See quality assurance (QA) sampling central line infections, 265–266
QC. See quality control (QC) sampling Medicare insurance audits, 267–268
QI. See quality improvement (QI) tracking patient complaints, 296–300
quality tracking patient falls, 268–271
as business strategy, 332–336 probability sampling, 117–120
definition, 17–18 cluster sampling, 119–120
essential ingredients to achieve, 19 simple random sampling, 117–118
level of, 26 stratified proportional random sampling, 119
quality assurance (QA), 20 stratified random sampling, 118–119
quality control (QC), 20 systematic sampling, 119
quality funnel, 20 reason for, 25–43
quality improvement (QI), 26, 32–35, 37, 39 roadblocks in, 95–99
and voice of customer, 45 desire for precision, 96–97
assessment tools, 348–349 numerical illiteracy, 98–99
key activities of, 18 performance objectives, 97
methods, 354 statistical techniques, 98
philosophy and theory of, 353–354 threatening, 95–96
quality measurement, 19–20, 100 setting the context, 1–24
 Index 367

patient-centered care and service, 12–19 Sergesketter, B., 357

quality funnel, 19–20 service excellence, VOC, 46–55
transparency, growing demand, 2–12 Shewhart charts, 211–257
types of studies, 28–30 additional, 247–249
quota sampling, 121–122 appropriate, decision of, 225–226
data types, 226–229
defining the key terms, 230–242

R area of opportunity, 233

c-chart, 240–241
I-chart, 236–238
range, statistical, 216
observation, 233
rate, 228
p-chart, 239
rational sampling, 122
S chart, 234–236
reliability, as criteria for sampling plan, 120
subgroup, 231–233
Richer, S., 162
u-chart, 241–242
Riedel, H., 117
X-bar, 234–236
roadblocks in quality measurement, 95–99
XmR chart, 236–238
desire for precision, 96–97
description of, 211–214
numerical illiteracy, 98–99
effective use of, 249–250
performance objectives, 97
exercises, 242, 246
statistical techniques, 98
key questions about, 214–219
threatening, 95–96
application of run chart rules, 219
Roberts, H., 357
how many dots?, 214–216
Rubik’s cube, 38
sigma, importance of, 216–219
run charts, 187–209
special cause variation, presence of, 219–225
analyzing of, 190–199
study questions, 245
annotation and, 206–207
summary of, 243–245
construction of, 188–190
types of, 229–230
description of, 187
variation and, 180
elements of, 188
vs. run charts, 211
median, changing of, 203–205
shift in process, 167
minimum and maximum amount of data, 202–203
sigma limits, 212, 232
nonrandom pattern, 193
simple random sampling, 117–118
random pattern, 193
Six Sigma, 250, 359
rules, 192–199
Skinner, B., 51–52
astronomical data point, 198–199
Smith, W., 117, 234
shifting, 192–193
SPC (statistical process control). See statistical process
too much or too little variation, 193–198
control (SPC)
trending, 193
special cause variation, 176, 179–183
vs. Shewhart charts, 211
specification limits, 213
run, definition of, 190
stair-step control limits, 234
standard deviation (SD), 114
Stanley, J., 35, 43
S static approaches (data analysis), 163–167
statistical process control (SPC), 18, 30, 98, 148, 159,
s-chart, Shewhart charts, 234–236 179, 187, 214
sample size, 35–37 statistical thinking, 98–99
sampling methods, advantages and disadvantages of, statistics, 30
123–125. See also individual sampling methods strategic dashboard, indicators and
Scherkenbach, W., 336 evolution of, 144–151
Schultz, L., 336, 341 organizing into, 143–158
scientific inquiry, steps, 31, 160 stratification, 113
scientific method, 160 stratified proportional random sampling, 119
self-assessment tool, measurement, 339 stratified random sampling, 118–119
Selltiz, C., 32 structures, processes, and outcomes (SPO), 101
368 Index

subgroups, 231–233 voice of the customer (VOC), 45–91

surveys, VOC, 67–86 and listening, 45–46
administration, logistics of, 79, 82–83 key points, 60–61
growth of, 71–74 experiential shopper, 66
improvement strategies, linked with, 84–86 focus groups, 64
questions, 74–77 high-tech tools, 66
response formats, 77–79 identification of customers, 55–57
system thinking, 54 key quality characteristics, defining, 57–60
systematic sampling, 119 leadership rounds, 65
observation, 64–65
organizational components
human resource issues, 47–48
T measurement and design, 54–55
motivation, 48–52
t-chart, 247–249 organizational design, 52–54
targets, 20 personal interviews, 65
Terrell, J., 117, 212 relating with voice of the process, 84
testing methods, 35–37 service excellence, creation of, 46–55
time-series analysis, 32 surveys, 67–86
transparency administration, logistics of, 79, 82–83
and measurement, 2–12 growth of, 71–74
definition, 5 improvement strategies, linked with, 84–86
trends, statistical, 173–174 questions, 74–77
Trial control limits, 215 response formats, 77–79
turnaround time, 113–114, 165 tools, 61–64
unsolicited feedback, 65–66
voice of the process (VOP), 18, 54, 84, 93
VOP. See voice of the process (VOP)
U
u-charts, 241–242
UCL. See upper control limit (UCL)
unsolicited feedback, 65–66
W
upper control limit (UCL), 212, 220–221, 234 Watkins, K., 54
Weiss, R., 117
Wheeler, D., 29, 98, 148, 181, 203, 214, 217–225, 230, 236, 242
Wick, C., 54
V worksheets, indicators and, 127–132

variables data, 226

variations, 171–183
and behaviors, people’s, 173–179
run charts, understanding with, 187–209
X
Shewhart charts, understanding with, 180, 211–257 X-bar, 234–236
VOC. See voice of the customer (VOC) XmR chart (I chart), 236–238