Using Design of Experiments in Method Development and Validation

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/321975479
Using Design of Experiments in Method Development and Validation
Conference Paper · December 2017
CITATIONS READS
0 1,258
4 authors, including:
Ron Snee
Snee Associates, LLC
428 PUBLICATIONS 7,824 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Working to Bring Analytics to Construction Industry Safety Data View project
Quality by Design View project
All content following this page was uploaded by Ron Snee on 21 December 2017.
The user has requested enhancement of the downloaded file.

Using Design of Experiments in
Method Development and Validation
Ronald D. Snee, PhD
Newark, DE
IVT Laboratory Week
Philadelphia, PA
December 4-6, 2017
1
About the Speaker …..
He is also an Adjunct Professor in the
pharmaceutical programs at Temple and Rutgers
Universities. He worked at DuPont for 24 years prior
to starting his consulting career.
Ron received his BA from Washington and
Jefferson College and MS and PhD degrees from
Rutgers University. He is an academician in the
International Academy for Quality and Fellow of the
American Society of Quality, American Statistical
Ron Snee, PhD is Founder of Snee Associates,
Association, and American Association for the
LLC, a firm dedicated to the successful
Advancement of Science.
implementation of process and organizational
improvement initiatives. He provides guidance to Ron is an Honorary Member of ASQ and has been
pharmaceutical and biotech senior executives in awarded ASQ’s Shewhart, Grant and Distinguished
their pursuit of improved business performance Service Medals, and ASA’s Deming Lecture, Dixon
using Quality by Design, Lean Six Sigma and Consulting and Gerald J. Hahn Quality and
other improvement approaches that produce Productivity Achievement Awards.
bottom line results. He is a frequent speaker and has published 6 books
He has authored several articles on how to and more than 300 papers in the fields of quality,
successfully implement QbD, coauthored 3 books performance improvement, management, and
on QbD tools and speaks regularly at statistics. He recently received the Institute of
pharmaceutical and biotech conferences. Validation Technology’s Speaker of the Year Award.
2
Abstract
Pharmaceutical industry is experiencing a growing need to improve
performance driven by global competition and the increasing impact of
information technology. How to go about this improvement in a systematic,
focused and sustainable manner is the question. Fortunately a world-class
body of improvement technology exists known as Quality by Design (QbD); a
science and data-based approach that builds quality into products and
processes during development and validation as well as operations.
A characteristic of good science is good data. Quality data are arguably more
important today than ever before. Design of experiments, an important QbD
tool, is very effective in developing test methods that produce quality data. It
will be shown how DOE can be used to effectively design and improve
laboratory processes enabling prompt, successful method validation.
Methods discussed in this session include assessment of method
repeatability and reproducibility, improving method robustness and
measurement process control. Methods for assessing that amount of product
variation that can be attributed to the manufacturing process, sampling
procedures and test method are also presented. The concepts and methods
involved will be introduced and illustrated with pharmaceutical and biotech
case studies and examples.
3
Session Outline
I. A Look at Today’s Realities
- Importance and need to improve measurement quality
- Need to Reduce Risk in method validation
- The Promise of QbD – Building Quality into Test Methods
- Understanding measurement processes
II. Using Design of Experiments in Method Validation
- Test method development
- Measurement quality - Repeatability and reproducibility
- Measurement system capability
- Improving test method robustness
- Measurement process control – Continued process verification
III. How to Troubleshoot Measurement Test Methods
- Is the variation due to sampling method or test method?
- Using designed experiments (DOE) to quantify variation
- Using data to identify opportunities for improvement
- Graphical methods to visualize and understand variation

4
Lab Results
Tell Us How Our Processes Are Performing
 Test labs produce the data that are used to:
 Develop products and processes
 Control and improve processes
 Assess product quality
 Improving test methods improves the quality of the
data used to design, control and improve products and
processes:
 Risk of poor quality product getting to patients and
ineffective and inefficient process performance is
reduced in the process
Improved Test Methods Performance Reduces Risk

5
Improving Lab Performance
Reducing Risk in Test Method Measurements
 Developing test method design
 Improve the quality of measurements process:
 Gage R&R Study
 Improve the stability of the measurement process:
 Process control charting of control samples over time
 Reducing the sensitivity of the measurement process to
uncontrollable variables:
 Robustness studies using 2-level fractional-factorial
designs
 Improving flow through the lab
 Use lean operating principles for the flow of individual
methods as well as the overall lab flow
Quality by Design Addresses These Issues

6
Quality by Design
Generic Definition
“A Systematic Process for

Building Quality into a Product
from the Inception to Final Output”
“Product” of Analytical Laboratory Processes:
Test results that have the desired precision and
accuracy delivered in a cost-effective and timely
manner compliant with GLPs

7
Test Method QbD Building Blocks
Life Cycle Management and Continuous Improvement

Continued Process Verification
8
Use of QbD in Analytical Methods
 Method Design
 Meet method design performance criteria
 Method Evaluation
 Risk assessment using Failure Modes and Effects Analysis
(FMEA)
 Assess method repeatability and reproducibility (Gage R&R)
 Assess robustness of method design space (DOE)
 Assess sampling and testing variation
 Method Control and Continued Verification
 Control of method using control samples and statistical
process control techniques (SPC)
 Continued process verification using an integrated system
of method control and improvement techniques

9
Test Method and Its Variables
Controlled Variables
• Test Time and Temperature
• Mixing Speed and Time
• Reagent Concentrations
• Apparatus
Process Inputs Process Outputs
• Raw Materials Test • Quality Measurements
• Instruments • Test Results on Time
Method
• Personnel • Cost Effective Operations
Environmental Variables
• Ambient Temperature and Humidity
• Reagent Quality
• Analyst
• Day of Week
• Season of Year
• Shift 10
Test Method Design
Experimentation Strategy
Screening Optimization
Experiment Experiment
 Screening Experiment for 6 or more method input variables

 Select variables with largest effects
 Study selected variables in an optimization experiment
 Optimization experiment – Identify Method Design Space
 Design Space – Settings of the variables that will give
the best method performance

11
Example – Test Method Development
 Screening Experiment
 11 Design Variables evaluated – Each at 2 levels
 Experiment Design
 24 run Plackett-Burman design in 4 blocks
 6 runs in each of 4 weeks
 4 critical variables were identified
 Optimization Experiment
 4 variables from screening experiment were optimized
 Experiment Design
 28-Run subset of a 3x3x2x3 factorial design
 Runs identified by computer-aided selection procedure enabling
estimation of main effects and two-factor interactions

12
Example – Test Method Development
Results Summary
 Assay Method is robust to variation in the four principal
control parameters.
 Assay precision is 2- 4% depending on the number of samples
tested and the number of tests made on each sample.
 Reaction time and reaction temperature had the largest effects
 Increasing reaction temperature significantly decreases
method variation
 Method design space can be expressed as a function of the
four principal control parameters
 Next Step: Method raw material optimization in progress

13
Test Method Development
Experimental Strategy– Underlying Theory
 Study a large number of variables to ensure that no
important variables are missed
 Pareto principle applies
 Only a few variables (3-6) will have major effects
 For critical variables - linear (1st Order) effects are typically
much larger than interaction and curvature effects (2nd
Order)
 Studying a variable at high and low levels will identify
critical variables
 Studying the critical variables in optimization experiments
is an effect way to optimize a test method.

14
Reducing Test Method Risk
Improving Test Method
Repeatability and Reproducibility
Using
Gage R&R Studies

15
Sources of Variation
Total
Variation
Process or Measurement
Product System
Variation Variation
Accuracy Precision

16
Sources of Variation
Total
Variation
Gage R&R Studies
Process or Measurement Focus on
Product System Repeatability and
Variation Variation Reproducibility
Accuracy Precision
Repeatability Reproducibility

17
Gage Repeatability and Reproducibility (R&R)
Gage R&R Study:
 Procedure used to determine how much of the observed
total variation is due to the measurement system
precision variation
 “Gage” is any device used to obtain measurements
Why is Gage R&R Important?
 Pharmaceutical R&D and Operations are data driven
 Before data are collected, it is crucial that the gage is
adequate
 Gage R&R should also be used whenever a new
measurement device is released to manufacturing

18
Repeatability and Reproducibility
Gage Repeatability Gage Reproducibility
Operator B
Operator C
Repeatability
Operator A
Reproducibility
Gage Repeatability is the Gage Reproducibility is the

variation in measurements variation in the average of
obtained when one measurements made by
operator uses the same different operators using
gage several times for the same gage when
measuring the identical measuring identical
characteristics of the same characteristics of the same
sample or part. sample or part. Snee Associates, LLC
19
Gage R&R Study Design?
 Select N Samples that represent the full range of long-term
variation for your process
 Select K Analysts that customarily test the samples
 Each Analyst will measure each sample R Times
Analyst 1 Analyst 2 Analyst 3
Sample T1 T2 T1 T2 T1 T2
1
2
3 Example Design
4 • 10 Samples
• 3 Analysts
5
• 2 Tests by Each Analyst
6 on Each Sample
7 • 10x3x2=60 Test Results
8
9
10 Snee Associates, LLC
20
HPLC Gage R&R Study
Sampling Design
- 6 Samples
- 4 Analysts per Sample – Total of 24 sub-samples
- 4 Tests per sub-sample – Total of 96 tests
Sample Sample 1 …… Sample 6
Analyst A1 A2 A3 A4 …………..
A21 A22 A23 A24
Tests T1 T5 T9 T13 …………..

T81 T85 T89 T93
T2 T6 T10 T14 …………..
T82 T86 T90 T94
T3 T7 T11 T15 T83 T87 T91 T95
T4 T8 T12 T16 T84 T88 T92 T96
Adapted from Ahuga and Dong (2005) Handbook of Pharmaceutical Analysis by HPLC

21
HPLC Gage R&R Study
Partial Data List - Samples 1 and 6
Sample Analyst Test Assay% Sample Analyst Test Assay%
1 1 1 99.3 6 1 1 98.1
1 1 1 99.4 6 1 1 98.4
1 1 1 99.9 6 1 1 97.2
1 1 1 99.4 6 1 1 97.1
1 2 2 99.5 6 2 2 96.4
1 2 2 99.2 6 2 2 97.9
1 2 2 98.6 6 2 2 94.4
1 2 2 101.6 6 2 2 97.3
1 3 3 97.8 6 3 3 95.4
1 3 3 97.4 6 3 3 96.1
1 3 3 98.1 6 3 3 96.9
1 3 3 99.1 6 3 3 96.4
1 4 4 97.9 6 4 4 95.9
1 4 4 98.3 6 4 4 95.2
1 4 4 99.1 6 4 4 97.2
1 4 4 98.9 6 4 4 96.8
22
HPLC Gage R&R Study
Two-Way ANOVA Table
Source DF SS MS F P
Sample 5 187.501 37.5001 39.9815 0.000
Analyst 3 15.623 5.2076 5.5522 0.009
Sample*Analyst 15 14.069 0.9379 1.4339 0.155
Repeatability 72 47.097 0.6541
Total 95 264.290
P< .05 indicates significant effects

• Significant differences between
• Samples (as expected)
• Analysts
• Analyst differences are the same for all Samples
• Sample-Analyst interaction is not significant
23
HPLC Gage R&R Study
Variance Components
Source VarComp % of Total
Total Gage R&R 0.90299 28.32
Repeatability 0.65413 20.52
Reproducibility 0.24885 7.81
Analyst 0.17790 5.58
Analyst*Sample 0.07095 2.23
Part-To-Part 2.28514 71.68
Total Variation 3.18812 100.00
• Total Gage R&R should be <30%
• Majority of Gage Variation is due to Test-to-Test
Number of Distinct Categories = 2

• Number of categories should be >4)

24
80
% Study Var 100.0
Percent
HPLC Gage
40 R&R Study 97.5
Repeatability – Test-to-Test Variation 95.0
0
Gage R&R Repeat Reprod Part-to-Part 1
R Chart by Analyst
1 2 3 4
4 UCL=3.802
Sample Range 100.0
2 _ 97.5
R=1.667
95.0
0 LCL=0
Xbar Chart by Analyst

1 2 3 4
100.0
Control Chart for Range 100.0
• Repeatability
Sample Mean
is stable across Analysts

UCL=97.782
Average
97.5 _
_ 97.5
X=96.568
LCL=95.354 95.0
95.0

25
Number of Distinct Categories
 Reflects measurement resolution
 Ability to detect differences between samples
 Number of distinct categories
>4 distinct data

 partto part  categories is
  1.41
  measurement 
desired

26
Number of Distinct Categories
The number of distinct levels that can be measured by
the measuring device – also known as resolution
1 Distinct Category –
unacceptable for estimating
process parameters; only
indicates whether the process is
producing conforming or
nonconforming samples
2-4 Distinct Categories –

generally unacceptable for
estimating process parameter;
provides only coarse estimates
5 or more Distinct Categories –

acceptable for estimating
process parameters
27
HPLC Gage R&R Study
Measurement Variation vs Total Variation
Measurement
Variation
Measurement Variation Takes up 63% of the Total Variation

Method Not Useful for Distinguishing Between Different Samples

28
Thickness Measurement System
Number of Categories = 5 (Acceptable Resolution)
Measurement
Variation
Measurement Variation Takes up 23% of the Total Variation

Method Useful for Distinguishing Between Different Samples
29
Sample and Operator Differences
Study A
Operator 2
Test-to-Test
Variation Is
Higher than
Other Operators
Study B
Operators
Do Not Agree
on Length of
Sample 6

30
Improving Measurement Methods
 My measurement method is not adequate. The Gage R&R

statistics for repeatability and reproducibility are too large.
 Now what do I do?
 One possibility is to evaluate the measurement process /
procedure for ruggedness
 Effects of small variations in the how the method is used.
 There are other sources of variation in a measurement
method in addition to instruments and analysts which are
typically the subject of Gage R&R studies.

31
Measurement Method Ruggedness
 Measurement method is “rugged,” if it is immune to
modest (and inevitable) departures from the conditions
specified in the method (Youden 1961)
 Ruggedness/Robustness can be evaluated using two-
level fractional-factorial designs including Plackett-
Burman designs
Example – Dissolution Method Validation
 Objective: How robust (insensitive) is the dissolution
method to small changes in its execution.
 Procedure:
 Make a large batch of tablets to assure uniformity
 Vary 8 factors of the method using a 12-run Plackett-
Burman design
 Measure the dissolution time for each run Snee Associates, LLC
32
Dissolution Method Validation
Ruggedness Test Variables
Variable (X) Low Level (-) High Level (+)

X1=Acid Conc (x1000) 9 11
X2=Polysorb Conc (x100) 4 6
X3=Stirring Speed (rpm) 45 55
X4=Temperature (deg C) 35 39
X5=Degasssing No Yes
X6=Filter Position Low High
X7=Operator A B
X8=Apparatus V W
Mixture of Quantitative and Qualitative Variables

33
Ruggedness Test Design (Plackett- Burman)
Poly- De- Oper- Appa- Disso

Run Acid sorb Stir Temp gas Filter ator ratus D1 D2 D3 Time
1 11 6 45 39 Yes High A V -1 1 -1 20.5
2 9 6 55 35 Yes High B V -1 -1 1 16.5
3 11 4 55 39 No High B W -1 -1 -1 16.7
4 9 6 45 39 Yes Low B W 1 -1 -1 19.3
5 9 4 55 35 Yes High A W 1 1 -1 14.7
6 9 4 45 39 No High B V 1 1 1 18.1
7 11 4 45 35 Yes Low B W -1 1 1 15.5
8 11 6 45 35 No High A W 1 -1 1 17.9
9 11 6 55 35 No Low B V 1 1 -1 16.5
10 9 6 55 39 No Low A W -1 1 1 19.1
11 11 4 55 39 Yes Low A V 1 -1 1 16.7
12 9 4 45 35 No Low A V -1 -1 -1 20.5
Adapted from Lewis, Mathieu and Phan-Tan-Luu (1999)

34
Pareto Analysis of Method Variable Effects
Pareto Chart of the Effects

(response is Dissolution Time, Alpha = .05)
4.030
Stirring Speed
Temp
Polysorb Conc
Operator
Dummy 1
Term
Apparatus
Degassing
Acid Conc
Dummy 3
No Significant
Filter Position
Dummy 2
Effects
0 1 2 3 4
Effect
Lenth's PSE = 1.4

35
Analysis of Test Method Variable Effects
Low High
Variable (X) Level Level Effect %
Acid Conc 9 11 -0.73 -4.1
Polysorb Conc 4 6 1.27 7.2
Stirring Speed 45 55 -1.93 -10.9
Temperature 35 39 1.47 8.3
Degasssing No Yes -0.93 -5.3
Filter Position Low High -0.53 -3.0
Operator A B -1.13 -6.4
Apparatus V W -0.93 -5.3
Dummy 1 -1 1 -0.93 -5.3
Dummy 2 -1 1 -0.53 -3.0
Dummy 3 -1 1 -0.73 -4.1
Effect = Change in dissolution time when variable moves low to high

% = Effect relative to average dissolution time=17.7 min.
36
Conclusion
 Method is rugged (robust)
 None of the factor effects is significant
 Observed method dissolution time
 Standard deviation is 1.9 min
 2 standard deviation limits +/- 3.8 min
 Coefficient of variation 10.7%
 If variation is a concern consider tighter control on the
variables with the largest observed effects
Factor Low High Effect Pct
Polysorb Conc 4 6 1.27 7.2
Stirring Speed 45 55 -1.93 -10.9
Temperature 35 39 1.47 8.3

37
Method Robustness – Underlying Theory
 Departures from test method standard operating
procedures happen frequently
 Human variation is a significant cause
 Small variations in operating parameters have no
practical effect on performance
 Interactions are small or non-existent
 Additive main effect models are adequate
 Plackett-Burman and two-level fractional factorial
designs work well
 Small run size provides adequate sensitivity

38
Process Variation Is Too High
Where is the Source of the Variation?
Process?
Sampling Method?
Analytical Lab?
Some or All of the Above?

39
Process Variation Issues
Excessive variation in Source of Process Variation

process output results Category
Manufacturing
Sampling
may be due to: Test Method
 Manufacturing Process
 Sampling Method
 Measurement Method
 A combination of these
sources of variation
Need a Procedure for Identifying Major Sources of Variation

40
Chemical Analysis Variation
Runs were made by 3 operators to locate a source of

variation in a chemical analysis. The procedure
consisted of:
 Taking a specimen,
 Heat treating the specimen
 Performing a chemical analysis.
In the test, 3 operators each took 2 specimens and
made 3 combustion runs on each specimen and
titrated each run in duplicate.

41
42
• Large Operator Differences
• Operator 2 Has More
Run-to-Run Variation than
Other Operators

43
Pigment Paste Production
Box, Hunter & Hunter (2005)
 Batch Process:
 A sample is taken form each batch and analyzed in the
lab to determine the moisture content of the product
 It is desired to understand the sources of
variation for the moisture content measurement
so that it can be determined the nature and
magnitude of the needed improvements
 Three sources of variation are identified:
 Batch-to-batch
 Sample-to-sample within a batch
 Test-to-test within a sample

44
 Sampling Study:
 15 Batches
 2 Samples per batch
 2 Analytical tests per sample
 Total of 30 samples and 60 tests
Batch B1 B2 ……….. B15
Sample S1 S2 S3 S4 ………….. S29 S30
Test T1 T3 T5 T7 ……….. T57 T59

T2 T4 T6 T8 ……….. T58 T60

45
Individual Value Plot of MOISTURE CONTENT
40
35
MOISTURE CONTENT
30
25
20
15
10
SAMPLE 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2
BATCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

46
Analysis of Variance for MOISTURE
Source DF SS MS F P
BATCH 14 1210.9333 86.4952 1.492 0.226
SAMPLE 15 869.7500 57.9833 63.255 0.000
TEST 30 27.5000 0.9167
Total 59 2108.1833
Source of Process Variation
Test
Variance Components 3% Batch
19%
Source Var Comp. % of Total StDev
BATCH 7.128 19.49 2.670
SAMPLE 28.533 78.01 5.342
TEST 0.917 2.51 0.957 Sample
78%
Sampling Variation is Large and Statistically Significant

47
Interpretation of Results
 Largest variance component is due to Sampling

 Sampling procedure:
1. Randomly select 5 drums from a batch of 80
drums
2. Use a special sampling tube to sample
material from all levels of the drum
3. Thoroughly mix the 5 samples
4. Test a sample of the aggregate mixture

48
Interpretation of Results
 Investigation found:
 Procedures not followed and unknown
 Test samples were routinely taken from a single
drum
 When the proper sampling method was used the
“sample” variance component reduced dramatically
Component Before After
Batches 7.1 4.3
Samples 28.5 16.0
Tests 0.9 1.3
49
 Corrective Action:
 Operators were trained to use the proper sampling
methods
 Sampling process was more closely supervised
 Studies were initiated to further improve the
sampling procedure
 Alternative sampling schemes were investigated:
Guideline:
Sample those sources with the largest variance

50
Raw Material Supplier Selection Study
( Adapted from Gonzalez-de la Parra and Rodriguez-Loaiza, Quality Engg 2003)
 Purpose
 Identify and quantify the sources of variability in a critical
raw material used to manufacture an API
 Nested Sampling Design was used
 Material from 2 suppliers was analyzed
 3 lots were evaluated from each supplier
 4 containers were sampled from each lot
 3 measurements (assay %) were made on each container
 Total of 2x3x4x3 = 72 assay results

51
Analysis of Variance Results
Source DF SS MS F P
Supplier 1 1.8412 1.8412 1.494 0.289
Lot 4 4.9301 1.2325 1.570 0.225
Container 18 14.1317 0.7851 5.597 0.000
Assay 48 6.7330 0.1403
Significant
Total 71 27.6361 Container
Variance Components Variation
Source Var Comp. % of Total StDev
Supplier 0.017 4.13 0.130
Lot 0.037 9.11 0.193
Container 0.215 52.50 0.464
Assay 0.140 34.26 0.375
Total 0.409 0.640
Let’s Look at a Plot of the Data

52
I Chart of Result(%) by Supplier
1 2
1
102.5
Supplier 1 Supplier 2 UCL=102.231
102.0
101.5
Individual Value
101.0
_
100.5 X=100.523
100.0
99.5
99.0
LCL=98.814
1 8 15 22 29 36 43 50 57 64 71
Observation

53
Boxplot of Result(%)
102.5
102.0
101.5
Result(%)
101.0
100.5
100.0
99.5
99.0
1 2
Supplier

54
Supplier 1 Supplier 2
Variance % of Std Variance % of Std
Source Component Total Dev Component Total Dev
Batch 0.030 50.0 0.174 0.044 6.1 0.211
Container 0.005 7.6 0.068 0.425 58.7 0.652
Assay 0.026 42.4 0.160 0.255 35.2 0.505
Total 0.061 100.0 0.246 0.725 100.0 0.851
• Supplier 2
• Has Greater Variation than Supplier 1
• Variation is due to sampling (59%) and testing (35%)
• Next Steps
• Assess sampling and test methods used by Supplier 2
• Consider evaluating what Supplier 1 is doing differently

55
Method Control
Reducing Analytical Risk

Maintaining Stable Analytical Methods
Using
Shewhart Control Charts
Enables Continued Measurement

Process Verification

56
Use of Blind Controls (Reference Samples)
 Blind Control – Samples from a common source are
regularly submitted for analysis along with routine
production samples in a way that the analyst can not
determine the difference between the production samples
and the control samples
 “ The use of blind controls is rare in the pharmaceutical
industry. The roots of this probably lie in the compliance
aspects of the study. However there is no better way to
understand the true variability of the analytical method”
B. K Nunnally and J. S McConnell (2007)

Six Sigma in the Pharmaceutical Industry
Understanding, Reducing and Controlling Variation in Pharma and Biologics

57
Measurement of Potency (%) on a QC
Control Material
Potency (%) on a QC
Control Material
84 Rows of Data
42 QC Runs
2 Measurements / Run

VII-58
58
Measurement of Potency (%) on a QC
Control Material (Continued)
Variables Control Chart
XBar of Potency
96.0
95.5
Mean of Potency
95.0 UC L=95.101
94.5
94.0
Av g=94.178 Out-of-Control Runs
93.5 Detected. Process Is
LCL=93.254
93.0 Not Stable; Long-Term
92.5
Variation = 58%
12
15
18
21
24
27
30
33
36
39
42
3
6
9
Run
Note: T he s ig ma wa s cal c ul ated us in g th e ra nge .
R of Potency
UC L=1.604
1.5
Good Repeatability
Range of Potency
1.0
Within-Run Variation in
0.5 Av g=0.491 Control
0.0 LCL=0.000
12
15
18
21
24
27
30
33
36
39
42
3
6
9

Run VII-59
59
Assessing Measurement Process Stability
 Q: When Should I worry about measurement stability?
 A: When Long-Term Variation represents more than 20% of
the total variation
 Total Variation = Long-Term Variation (Reproducibility)
+ Short-Term Variation (Repeatability)
 This guideline is based on the assumption that
 Well-controlled process will detect a shift of 1.5 short-
term standard deviations
Measurement Process Stability Long-Term Variation
Not a Problem < 20%
May be A Problem 20 – 30%
Corrective Action May Needed? > 30%
Ref: Snee and Hoerl, Quality Progress, May 2012, 39-41
60
Assessing Measurement Process Stability
Example – Stable and Unstable Processes
• Stable Process
• Good Reproducibility
• Long-Term
Variance = 21%
• Un-Stable Process
• Poor Reproducibility
• Long-Term
Variance = 58%

61
Using QbD Techniques Improves
Test Method Performance
 Screening experiments followed by optimization studies
is an effective way to design effective test methods
 Measurement process can be controlled using Control
samples and Statistical Process Control techniques
 Measurement quality can be improved using Gage
Repeatability and Reproducibility studies
 Creating robust measurement systems using statistical
design of experiments
 Product variation: Be sure to separate sampling and
process variation from test method variation
Risk is Reduced when
Test Method Performance is
Stable, Capable, Efficient and Cost-Effective
62
References – Improving Measurement Systems
Borman, P., M., etal (2007) “Application of Quality by Design to Analytical
Methods”, Pharmaceutical Technology, October 2007, 142-152.
Box, G. E. P. , J. S. Hunter and W. G. Hunter (2005), Statistics for
Experimenters, 2nd Edition, John Wiley and Sons, New York, NY, 345-353
Mason, R. L., Gunst, R. F. and Hess, J. L. (2003), Statistical Design and
Analysis of Experiments, 2nd Edition, John Wiley , New York, NY, Ch13
Montgomery, D. C. (2009), Design and Analysis of Experiments,7th Edition,
John Wiley and Sons, New York, NY, Chapter 13.
Schweitzer, M., etal (2010) “Implications and Opportunities of Applying QbD
Principles to Analytical Measurements”, Pharma Tech, Feb 2010, 52-59.
Snee, R. D. (1983), “Graphical Analysis of Process Variation Studies”, J.
Quality Technology, 15, 76-88.
Snee, R. D. (2005) “Are We Making Decisions in a Fog? The Measurement
Process Must Be Continually Measured, Monitored and Improved”, Quality
Progress, December 2005, 75-77
Snee, R. D. (2006) “If You’re Not Keeping Score, It’s Just Practice. If you’re Not
Measuring, You’re Not Competing”, Quality Progress, May 2006, 72-74.
Snee, R. D. (2014) “QbD in Test Method Development and Validation”,
Pharmaceutical Processing, May 2014, 28-29.
Snee, R. D. & R. W. Hoerl (2012) “Going on Feel: Monitor and Improve Process
Stability to Make Customers Happy”, Quality Progress, May 2012, 39-41
63
For Further Information,
Please Contact:
Ronald D. Snee, PhD
Newark, DE
(610) 213-5595
Ron@SneeAssociates.com
Please visit our website at:

www.SneeAssociates.com

View publication stats 64

Using Design of Experiments in Method Development and Validation

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Using Design of Experiments in Method Development and Validation

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

Using Design of Experiments in Method Development and Validation

Conference Paper · December 2017

Working to Bring Analytics to Construction Industry Safety Data View project

Quality by Design View project

The user has requested enhancement of the downloaded file.

Snee Associates, LLC

Improved Test Methods Performance Reduces Risk

Snee Associates, LLC

Quality by Design Addresses These Issues

Snee Associates, LLC

“A Systematic Process for

Snee Associates, LLC

Life Cycle Management and Continuous Improvement

Snee Associates, LLC

 Screening Experiment for 6 or more method input variables

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Gage Repeatability is the Gage Reproducibility is the

Sample Sample 1 …… Sample 6

Tests T1 T5 T9 T13 …………..

Snee Associates, LLC

P< .05 indicates significant effects

Number of Distinct Categories = 2

Snee Associates, LLC

Repeatability – Test-to-Test Variation 95.0

Xbar Chart by Analyst

is stable across Analysts

Snee Associates, LLC

>4 distinct data

Snee Associates, LLC

2-4 Distinct Categories –

5 or more Distinct Categories –

Measurement Variation Takes up 63% of the Total Variation

Snee Associates, LLC

Measurement Variation Takes up 23% of the Total Variation

Snee Associates, LLC

 My measurement method is not adequate. The Gage R&R

Snee Associates, LLC

Variable (X) Low Level (-) High Level (+)

Mixture of Quantitative and Qualitative Variables

Snee Associates, LLC

Poly- De- Oper- Appa- Disso

Adapted from Lewis, Mathieu and Phan-Tan-Luu (1999)

Snee Associates, LLC

Pareto Chart of the Effects

Snee Associates, LLC

Effect = Change in dissolution time when variable moves low to high

Snee Associates, LLC

Snee Associates, LLC

Snee Associates, LLC

Excessive variation in Source of Process Variation

may be due to: Test Method

Need a Procedure for Identifying Major Sources of Variation

Snee Associates, LLC

Runs were made by 3 operators to locate a source of

Snee Associates, LLC