MSA for Attribute or Categorical Data

MSA Example:
Attribute or
Categorical Data

All Rights Reserved, Juran Institute, Inc. 1 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Definitions

Accuracy: Overall agreement of the measured value with the true value (which may be an
“expert” value). Bias plus precision.

Attribute Data: Discrete qualitative data.

Attribute Measurement System: Compares parts to a specific set of criteria and accepts the
item if the criteria are satisfied.

Bias: A systematic difference from the true value. Revealed in the differences in averages from
the true value.

Precision: Variation in the measurement process.

R&R: Repeatability and Reproducibility. Two elements of precision.

Repeatability: The variation observed when the same operator measures the same item
repeatedly with the same device.

Reproducibility: The variation observed when different operators measure the same parts using
the same device, sometimes it can be the same operator using different devices.

The list provides a quick reference for key terms used in Measurement System Analysis.

All Rights Reserved, Juran Institute, Inc. 2 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

The Fundamental MSA Question

“Is the variation (spread) of the measurement system

too large to study the current level of process
variation?”

Variation
Product Variability in the (Observed Variability)
Process Variability measurement Total Variability
process

+ =

MSA for Continuous Processes 3 .PPT All Rights Reserved, Juran Institute, Inc.

Quantifying Variation

Like all processes, the measurement process has CTQs. The graph above lists
some of the most common CTQs used for the measurement process. MSA
quantifies the amount of variation for:
ƒ Accuracy
ƒ Repeatability
ƒ Reproducibility
ƒ Stability (this is typically covered in the Black Belt workshop)
ƒ Linearity (covered in Black Belt workshops)

All Rights Reserved, Juran Institute, Inc. 3 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Bias

True
Value or Possible Causes of Bias
Standard
ƒ Sensor not properly
calibrated
Bias
ƒ Improper use of sensor

ƒ Unclear procedures
Observed
Average ƒ Human limitations

MSA for Continuous Processes 4 .PPT All Rights Reserved, Juran Institute, Inc.

Bias

Bias is the difference between the observed average of measurements and the true
average. Validating accuracy is the process of quantifying the amount of bias in the
measurement process. Experience has shown that bias and linearity are typically
not major sources of measurement error for continuous data, but they can be.
In service and transaction applications, evaluating bias most often involves testing
the judgment of people carrying out the measurements.

¾Example
A team wants to establish the accuracy of its process to measure defects in
invoices. First, they gather a “standard” group of invoices and have an “expert”
panel establish the type and number of defects in the group. Next, they have the
standard group of invoices measured by the “normal” measurement process.
Differences between averages the measurement process came up with, and what
the known defect level was from the expert panel represented the bias of the
measurement process.

All Rights Reserved, Juran Institute, Inc. 4 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Repeatability

Possible Causes of Poor

Repeatability

Equipment
ƒ Gage instrument needs
maintenance
ƒ The gage needs to be more
rigid

People
ƒ Environmental conditions
Repeatability (lighting, noise)
ƒ Physical conditions (eyesight)

MSA for Continuous Processes 5 .PPT All Rights Reserved, Juran Institute, Inc.

Repeatability

Repeatability is the variation in measurements obtained when one operator uses

the same measurement process for measuring the identical characteristics of the
same parts or items.

Repeatability is determined by taking one person, or one measurement device, and

measuring the same units or items repeatedly. Differences between the repeated
measurements represent the ability of the person or measurement device to be
consistent.

Possible causes of the lack of repeatability are listed on the slide.

All Rights Reserved, Juran Institute, Inc. 5 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Reproducibility

Mean of
the measurements Possible Causes of Poor
of Operator B Reproducibility
Mean of
the measurements ƒ Measurement procedure is not
of Operator A clear

ƒ Operator is not properly trained

in using and reading gage

ƒ Operational Definitions not

established
Reproducibility

MSA for Continuous Processes 6 .PPT All Rights Reserved, Juran Institute, Inc.

Reproducibility

Reproducibility is very similar to repeatability. The only difference is that instead of

looking at the consistency of one person, you are looking at the consistency
between people.

Reproducibility is the variation in the average of measurements made by different

operators using the same measurement process when measuring identical
characteristics of the same parts or items.

Possible causes of poor reproducibility include: measurement process is not clear,

operator not properly trained in using the measurement system, and operational
definitions are not clear nor well established.

All Rights Reserved, Juran Institute, Inc. 6 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Attribute Measurement Systems Study

ƒ Discrete qualitative data

ƒ Go/no-go basis; or limited data categories
ƒ Compares parts to specific criteria for accept/not accept or
to be placed in category
ƒ Must screen for effectiveness to discern good parts from
bad
ƒ At least two appraisers and two trials each
ƒ If available, have Quality Master rate parts first

MSA for Continuous Processes 7 .PPT All Rights Reserved, Juran Institute, Inc.

Listed here are the key highlights of conducting an MSA for attribute or categorical
data. The “parts” can be invoices, parts or reason codes for customer returns, for
example.

All Rights Reserved, Juran Institute, Inc. 7 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Attribute MSA Study

Attribute MSA Example

Appraiser A Appraiser B
Master 1 2 1 2
1 G G G G G
2 G G G G G
3 G NG G G G
4 NG NG NG NG NG
5 G G G G G G = Good
6 G G G NG G
NG = Not Good
7 NG NG NG G NG
8 NG NG NG G G
9 G G G G G
10 G G G G G
11 G G G G G
12 G G G G G
13 G G NG G G
14 G G G G G
15 NG G G G G
16 G G G G G
17 G G G G G
18 G G G G G
19 G G G G G
20 NG G G G G

MSA for Continuous Processes 8 .PPT All Rights Reserved, Juran Institute, Inc.

This shows the results of 2 rounds using 2 appraisers, assessing the same 20
items.

All Rights Reserved, Juran Institute, Inc. 8 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Challenges of Continuous Process MSA

ƒ MSA study is an experiment

ƒ Requires two or more trials for calculating Repeatability
ƒ Needs a way to present the inspection units to the appraiser
multiple times
ƒ Is not possible within the continuous process

MSA for Continuous Processes 9 .PPT All Rights Reserved, Juran Institute, Inc.

When conducting an MSA for a continuously running process, parts should be

taken off-line to conduct the MSA study.

All Rights Reserved, Juran Institute, Inc. 9 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Visual Inspection of Glass

Packers

Catwalk
Cutter

Inspector

Glass

MSA for Continuous Processes 10 .PPT All Rights Reserved, Juran Institute, Inc.

The example given here is that of visual inspection of glass.

All Rights Reserved, Juran Institute, Inc. 10 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Challenges to Overcome

ƒ Bias to the standard could be evaluated on-line.

ƒ Repeatability and Reproducibility (R & R) could not be
evaluated on-line.
ƒ A method had to be devised to allow the inspectors to view
the same pieces of glass repeatedly.
ƒ The solution was an off-line conveyor which simulated the
on-line condition as closely as possible.

MSA for Continuous Processes 11 .PPT All Rights Reserved, Juran Institute, Inc.

All Rights Reserved, Juran Institute, Inc. 11 #### XXXX.#### vx

 MSA for Attribute or Categorical Data
Case Example: Attribute MSA Method Employed

ƒ 20 pieces of glass from the process that included both good

and bad samples were selected.
ƒ A team of people well versed in the quality standard
classified each piece of glass as either “pass” or “fail.”
ƒ All regular inspectors independently evaluated each piece
twice (in random order).
ƒ The inspectors used a log sheet to record the data.
ƒ Minitab® was used to analyze the data.

MSA for Continuous Processes 12 .PPT All Rights Reserved, Juran Institute, Inc.

There were two outcomes in this inspection or measurement process: pass or fail.
Twenty pieces, a team of inspectors, and two rounds (or trials) were used in the
MSA.

All Rights Reserved, Juran Institute, Inc. 12 #### XXXX.#### vx

 MSA for Attribute or Categorical Data
Case Example: Attribute MSA Study Data

Excerpt of
full data
for 20
inspectors

MSA for Continuous Processes 13 .PPT All Rights Reserved, Juran Institute, Inc.

This slide shows the data in MINITAB®. The “Standard” column documents the
correct or expert answer for each piece of glass.

All Rights Reserved, Juran Institute, Inc. 13 #### XXXX.#### vx

 MSA for Attribute or Categorical Data
Case Example: Attribute MSA Study Results

Date of study : 9/13 and 9/19/07

Assessment Agreement
Reported by : Bill Sunderland and Bob Panatieri
Name of product: Solar
Misc: Mainline Mark ing

Within Appraisers Appraiser vs Standard

100 95.0% C I 100 95.0% C I
P ercent P ercent
90 90

80 80
Percent

Percent
70 70

60 60

50 50

40 40

l t l t
n y o lly Tim E a r e v e ni e oh n n ie lin rry n y o lly T im E a r e v e ni e oh n n ie lin rry
To M n
S t J ea J Ro C L
n a To M n
S t J ea J Ro C L
n a

Appraiser Appraiser

MSA for Continuous Processes 14 .PPT All Rights Reserved, Juran Institute, Inc.

The graph on the left shows the agreement (or repeatability) of each appraiser
between Trial 1 and Trial 2.
The graph on the right shows the agreement of each appraiser with the Standard.
As shown on both graphs, the blue dots show the percent agreement and the
redlines are the 95% confidence intervals.

All Rights Reserved, Juran Institute, Inc. 14 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Attribute MSA Study Results (continued)

MSA for Continuous Processes 15 .PPT All Rights Reserved, Juran Institute, Inc.

This slide shows the Within Appraiser agreement.

For example, Larry scored 100% - Trial 1 and Trial 2 are in full agreement.
On the other hand, Allen scored only 50% agreement – there is only a 50%
agreement between his Trial 1 and Trial 2 measurements. Or he disagrees with
himself half the time!

All Rights Reserved, Juran Institute, Inc. 15 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Attribute MSA Study Results (continued)

MSA for Continuous Processes 16 .PPT All Rights Reserved, Juran Institute, Inc.

This slide shows the agreement of each Appraiser (across both trials) with the
Standard.
For example, Larry has 89% agreement with the Standard but, Allen only has 39%
agreement with the Standard.

All Rights Reserved, Juran Institute, Inc. 16 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Attribute MSA Study Results (continued)

MSA for Continuous Processes 17 .PPT All Rights Reserved, Juran Institute, Inc.

This shows the level of agreement across all Appraisers. In this case, only 5.56%
agreement!

All Rights Reserved, Juran Institute, Inc. 17 #### XXXX.#### vx

 MSA for Attribute or Categorical Data

Case Example: Attribute MSA Study Conclusions

ƒ What could have caused the poor agreement?

ƒ What was done to improve consistency?

MSA for Continuous Processes 18 .PPT All Rights Reserved, Juran Institute, Inc.

Given the results of the MSA study, what could have caused the poor agreement?
And what should be done to improve the measurement system?

The measurement system must be improved and tested again (with another MSA
study) to reach at least 90% agreement before the data can be used for baselining
process performance or further analysis.

All Rights Reserved, Juran Institute, Inc. 18 #### XXXX.#### vx