Data Management 3

Analysis - Validation
Session Four
 What our sessions will be about !
1

Introduction to Continuous
Improvement Methodologies

Data Management 2
2
3
Common Approaches to
Performance Improvement
FOCUS PDCA Six Sigma Part II

5 7

4 Lean
9
6 8 Management -
Part II

Data Management 3 10
Six Sigma Part I
Lean Management -Part Text
I This is a sample text.
Insert your desired
Practical Application of text here.
CIM (The teams
presentations)
 Learning Objectives

Identify the importance and types of 02 Apply quality tools in continuous

quality tools improvement practices

03 Identify data validation methods

• What tools does a cook use?
• What are the quality professional’s skills and
tools?
• What is the difference between SQC and SPC?
Quality Tools
 Tools for Team Effectiveness
• Whiteboard
• Flipcharts
• Parking lot
• Action list
• Meeting minutes
 Tools for Planning

• Project Charter
 Tools for Root Cause Analysis
Defect Concentration Diagram
Example:
• A long-term care facility wanted to see whether patient falls
occurred randomly throughout the facility’s buildings and
outdoor area or were concentrated in some areas.
• The investigation team reviewed all reported falls during the
last year, and the majority of reports described the location
of the falls. Where the exact location was missing, in most
cases, the team could deduce the location from the
description of the fall. Based on these data, the team
produced a problem concentration diagram.
• The diagram was based on a rough sketch of the buildings
and grounds, indicating types of rooms. The recorded falls
were then plotted on this diagram, using stars to mark falls.
A clear pattern emerged, with bathrooms and bedrooms
seeing the most falls (and even more specifically, the
immediate surroundings of toilets and beds, respectively)

Source: Excerpted from Root Cause Analysis and Improvement in the Healthcare Sector: A Step-by-Step Guide, ASQ Quality Press
What are the 7 Basic Quality Tools ?and
the extra 3?

https://www.youtube.com/watch?v=r-3jveJ-uBA
 Flow
Chart

Cause and
Scatter
Effect
Diagram
Diagram

7
Quality
Pareto Tools Check
Chart Sheet

Control
Histogram
Chart
1. Flowcharts
Flowchart
-- IsUsing
a diagram of the steps in a process and their sequence
a flowchart has a variety of benefits:
- It helps to clarify complex processes.
- It identifies steps that do not add value to the internal or
external customer, including: delays, needless storage
and transportation, unnecessary work, duplication, and
added expense; breakdowns in communication.
- It helps team members gain a shared understanding of
the process and use this knowledge to collect data,
identify problems, focus discussions, and identify
resources.
- It serves as a basis for designing new processes.
 Flowchart Symbols
Standard Flowchart Symbols

Source: Adapted from: Process Analysis Tools

Institute for Healthcare Improvement Boston, Massachusetts, USA
 Flowchart Symbols
Detailed Flowchart Symbols

Source: Adapted from: Process Analysis Tools

Institute for Healthcare Improvement Boston, Massachusetts, USA
 Types of Flowcharts

High Level Flowchart

- Showing 6 to 12 steps, gives a panoramic
view of a process
- These flowcharts show clearly the major
blocks of activity, or the major system
components
- High-level flowcharts are especially
useful in the early phases of a project.

High Level Flowchart Example: Ischemic Heart Disease Patient

Flow

Source: Adapted from: Process Analysis Tools

Institute for Healthcare Improvement Boston, Massachusetts, USA
 Detailed Flowchart Example: Filling an order

Detailed Flowchart
-A close-up view of the process, typically
showing dozens of steps.
- These flowcharts make it easy to identify
rework loops and complexity in a process
- Detailed flowcharts are useful after
teams have pinpointed issues or when they
are making changes in the process

Source: Adapted from The Quality Toolbox, Second Edition, ASQ Quality Press
 Swim Lane/Cross Functional
Flow Chart
- Itacross
is useful when the process spans
multiple teams / departments /
units

- Itdoesprovides rich information on who

what and can also be expanded to
show times (e.g. when tasks are done
and how long they take)

Swim-lane diagram of breast center

Source: Dehghanimohammadabadi, Mohammad & Rezaeiahari, Mandana & Keyser, Thomas. (2017). Simheuristic of patient scheduling using a table-experiment approach -
Simio and MATLAB integration application. 10.1109/WSC.2017.8248015.
 SIPOC Diagram
- may
It helps define a complex project that
not be well scoped, identifying the
high-level picture of the components of
a process and clarifying the
relationships with key customers and
suppliers
- Suppliers: People and entities
providing inputs
- Inputs: Items used to generate outputs,
plus item requirements
- Process: Steps you perform (at a high
level)
- Outputs: Results of the process steps,
plus output requirements
- Customers: People and entities who SIPOC diagram showing the prescription dispensing process
receive and use the outputs

Source: Alkuwaiti, Ahmed. (2016). Application of Six Sigma Methodology to Reduce Medication Errors in the Outpatient Pharmacy Unit: A Case Study from the King Fahd
University Hospital, Saudi Arabia. International Journal for Quality Research. 10. 267-278. 10.18421/IJQR10.02-03.
 Activity
You will be divided into groups, each group
choose any hospital process and design a
flowchart for this process using one of the
flowcharts types:
• High level flowchart
• Detailed flowchart
• Swim Lane
• SIPOC
2. Check Sheet
• Check sheets are a simple but
effective way to gather data (count
defects/occurrences) during process
observations.
• Data is captured “real-time” as it is
observed and may provide insight
into potential root causes for defects,
errors, or variation.

What is the difference between checklist and check sheet?

3. Cause and Effect Diagram
 Cause and Effect Diagram

Source: Institute for Healthcare Improvement

4. Pareto Chart
• A very powerful tool for showing the
relative importance of problems
• The 80/20 Rule (also known as the Pareto
principle or the law of the vital few &
trivial many) states that, for many events,
roughly 80% of the effects come from
20% of the causes
• Using a Pareto diagram helps a team
concentrate its efforts on the factors that
have the greatest impact. It also helps a
team communicate the rationale for
focusing on certain areas
5. Scatter Diagram
• The scatter diagram graphs pairs of
numerical data, with one variable
on each axis, to look for a
relationship between them.
• If the variables are correlated, the
points will fall along a line or
curve. The better the correlation,
the tighter the points will hug the
line
6. Histogram
 • A frequency distribution shows how often each different value
in a set of data occurs.
• A histogram is the most commonly used graph to show
frequency distributions.
• Use Histogram when:
• The data are numerical
• You want to see the shape of the data’s distribution,
especially when determining whether the output of a
process is distributed approximately normally
• Analyzing whether a process can meet the customer’s
requirements
• Analyzing what the output from a supplier’s process looks
like
• Seeing whether a process change has occurred from one
time period to another
• Determining whether the outputs of two or more processes
are different
• You wish to communicate the distribution of data quickly
and easily to others
Source: Adapted from The Quality Toolbox, Second Edition, ASQ Quality Press
7. Control Charts
 Control Charts

• A graph used to study how a process

changes over time.
• Data are plotted in time order.
• A control chart always has a central line for
the average, an upper line for the upper
control limit, and a lower line for the lower
control limit.
• By comparing current data to these lines,
you can draw conclusions about whether the
process variation is consistent (in control) or
is unpredictable (out of control, affected by
special causes of variation).
Source: Excerpted from The Quality Toolbox, ASQ Quality Press
 Control Chart/Shewhart Charts or
Statistical Process Control Charts
(SPCC)
• There are five rules for identifying special cause in control charts to understand
whether improvement is occurring:
• A single point outside the control limits
• Eight or more consecutive points above or below the center line
• Six consecutive points increasing (trend up) or decreasing (trend down)
• Two out of three consecutive points near a control limit (outer one-third)
• Fifteen consecutive points close to the center line (inner one-third).

Source: Clinical Excellence Commission Academy-Quality Improvement Tools

Source: Clinical Excellence Commission Academy-Quality Improvement Tools
 Activity
You will be divided into groups, each group will
discuss the quality tools used in the “Budgetary
Bandage” case study, its importance and
relevance to the case, then you will share your
thoughts with the rest of the group.
 Data Validation

Why Data Validation?

• Data
quality is essential to ensure the usefulness of the data for quality
improvement

Which Data to Validate?

• All data involving quality / performance improvement metrics shall be
validated before reporting
• Each Metric should be validated at least once
 When to Validate Data?
• The data source has changed, such as when part of
Accreditation the patient record has been turned into an electronic
Recommendations format
for Validation
• A new measure is implemented

• The existing measure has changed, such as the data

collection tool
• The subject of the data collection has changed, such
as the patient demographics, or new practice
guidelines implemented
• The data of an existing measure have changed in an
unexplainable way

Source: JCI Accreditation Standards for Hospitals 6th Edition, QPS.6

 How to Validate Data?

Source: JCI Accreditation Standards for Hospitals 6th Edition, QPS.6

 What are the Validation Sampling
Requirements?

Number of cases reviewed by the Validation Sample (2nd

1st Abstractor Abstractor)

≥ 180 A minimum of 5% of the cases reviewed

by the 1st data abstractor, or 50 cases

< 180 A minimum of 9 cases

<9 All cases

Source: JCI Accreditation Standards for Hospitals 6th Edition, QPS.6

 Validation Inter-rater Reliability
Comparison
Option 1: Measure Category Assignment (MCA) Match Rate
Comparison
Focus: A check to ensure that the combined data element answers
collected result in a case correctly being assigned to the measure’s
numerator and denominator used to calculate the measure rate.
Process:
a) 2nd abstractor re-abstracts the originally abstracted cases
b) 2nd abstractor assigns the measure category letter result to each case
c) Compare the 2nd abstractors MCA letter to the 1st abstractor’s MCA
Letter
Expected Result: The 2nd abstractor’s assigned measure category letter (E,
D, B) should match the 1st abstractors measure category assignment letter

Source: Specification Manual for the Joint Commission International Library of Measures Version 2.0
 Validation Inter-rater Reliability
Comparison
Measure Category Assignment (MCA) letter value to determine if the MCA
values’ match

B Excluded from the

denominator
D Did not meet the numerator
criteria
E Met the numerator criteria

Source: Specification Manual for the Joint Commission International Library of Measures Version 2.0
 Validation Inter-rater Reliability
Comparison
Measure Category Assignment (MCA) Example:
For the I-PN-2 Pneumococcal Vaccination measure, if 8 cases were identified for
re-abstraction, then there would be a total of 8 possible MCA matches. If there
are 2 MCA mismatches then the MCA match reliability rate would be calculated
as follows:
• Validated sampled cases = 8 cases for the I-PN-2 measure (denominator)

• Cases with a MCA match = 6 cases for the measure (numerator)

• 6 cases with MCA matches/ 8 sampled cases = .75 x 100% = 75% reliability
rate

Source: Specification Manual for the Joint Commission International Library of Measures Version 2.0
 Validation Inter-rater Reliability
Comparison
Option 2: Data-Element Agreement Rate Comparison
Focus: a check to ensure that the 1st and 2nd data abstractors have the
same understanding how to collect the data element answer values used in
determining whether or not the case met the measure.
Process:
a) 2nd abstractor re-abstracts the originally abstracted cases
b) Compare the 2nd abstractor’s data element answers to the 1st
abstractor’s data element answers for each of the data elements in the
measure.
Expected Results: The 2nd abstractor’s data element answers should be in
agreement with the 1st abstractors data element answers

Source: Specification Manual for the Joint Commission International Library of Measures Version 2.0
 Validation Inter-rater Reliability
Comparison
Data-Element Example:
I-VTE-1 has 14 data elements question answer values. If 6 records were
identified for re-abstraction, then a total of 84 data elements (i.e., 14 data
elements x 6 cases = 84) will be re-abstracted. If 5 mismatches are noted during
the re-abstraction process, then the agreement rate is 84 total data elements – 5
mismatches = 79 agreements.
The agreement rate is calculated by dividing the total number of answer value
agreements (numerator) by the total number of data element question answer
values (denominator) and multiplying by 100 (i.e., 79/84 x 100 = 94%)

Source: Specification Manual for the Joint Commission International Library of Measures Version 2.0
Actions Taken For Invalid measures

• Identify discrepancies, and the reason behind them

• Take corrective actions as appropriate – For example:
• Train abstractors in the same way
• Clearly define data collection elements
• Specify additional parameters for data collection (source,
timeframe)
• Re-validate after all corrective actions have been
implemented
Final Project Follow Up…

• Join your group

• Present the progress of your project
Exit Slip
• Used to know
• Now Know
• Want to Know more
Thank you ☺