Consensus Standards for

Method Validation

1
Sources

◼ CLSI: Clinical and Laboratory

Standards Institute, Wayne, PA
– Area Committee for Evaluation Protocols
◼ ISO: International Organization for
Standardization, Geneva
– Technical Committee 69: Application of
Statistical Methods

2
Purpose of presentation

◼ Promote broader use of TC 69 and

CLSI EP consensus documents
◼ More active involvement by FDA and
IVD Industry Statisticians
– TC 69 US TAG
– EP AC Members, Observers, project
participants

3
My Background

◼ CLSI EP Area Committee

– Observer, Adviser, Member, since 1984
– Chairman, EP5, EP6, EP17
– Co-author EP7, EP9, EP10, EP15
◼ ISO TC 69
– Member, US TAG
– ILAC Liaison (International Laboratory
Accreditation Cooperation), since 2002
– “Expert” for ISO 13528, 11843-5
4
My Background
◼ Medical Laboratories
– Statistician, CAP, 1981-1995
– Expert Consultant, CDC DLS (2000-current)
– Adviser, CLSI AC on General Lab Practice (1998)
◼ International Standards
– ISO CASCO (Convener, WG 28 (PT))
– European Commission (EEE-PT, IMEP)
– Asia-Pacific (Member, APLAC PT Committee)
◼ Other fields of testing
– EPA (NELAC), FDA (CFSAN, CBER), AOAC
– A2LA Assessor for ISO 17025, Guides 43 & 34 5
Method Capability Statistics:
Consensus vs. Traditional
◼ Traditional: Publications, convention
– One person (or team)
– Specific field of application
– Peer review
– Expedited publication
◼ Consensus standards
– Many people, countries, areas of interest
– Exhaustive process
– Very general language, basic approach
– No “pride of authorship” 6
Consensus organizations

◼ Accredited SDO (in US, by ANSI)

– Impartiality
– All stakeholders represented
– All comments addressed
– Records
◼ Organizations
– ISO (100+ TCs, CASCO, REMCO)
– CLSI
– ASTM
7
Consensus process

◼ ISO
– Anyone participates, through TAG or
“mirror committee”
– One vote per country
– CD, DIS, FDIS, Approved stages
◼ CLSI
– Anyone participates
– Balance Industry, Profession, Government
– SC Draft, Proposed, Approved stages 8
TC 69 Scope

• Standardization in the application of

statistical methods, including
generation, collection (planning and
design), analysis, presentation and
interpretation of data.
• Advisor to all ISO technical
committees in matters concerning the
application of statistical methods in
standardization. 9
TC 69 Subcommittees

◼ TC: Basic Statistical Techniques

◼ SC 1: Terminology
◼ SC 4: Process Management
◼ SC 5: Acceptance Sampling
◼ SC 6: Measurement Methods and
Results
◼ SC 7 (New): Procedures for Six Sigma
10
TC 69 Basic Techniques

◼ Estimation & Tests for Mean & Variances

◼ Tests for departure from Normality
◼ Confidence & Tolerance Intervals
◼ Estimation of median

11
SC 1: Terminology

◼ ISO 3534 Series -- Vocabulary and

symbols
– Part 1: Probability and general statistical
terms
– Part 2: Statistical quality control
– Part 3: Design of experiments
◼ Mostly consistent with VIM, more frequently
updated
12
SC 4: Process
Management
◼ Control Charts
– Acceptance
– Shewhart
– CUSUM
◼ Process Capability
◼ Machine Performance
◼ Statistical Process Control

13
SC 5: Acceptance
Sampling
◼ Inspection by variables
◼ Inspection by attributes
◼ Assessment of declared quality levels
◼ Double sampling
◼ Sequential sampling
◼ Accept-zero sampling plans

14
SC 6: Measurement
Methods and Results
◼ ISO 5725: Accuracy (trueness and precision)
of measurement methods and results
– Part 1: General principles
– Part 2: Repeatability and reproducibility
– Part 3: Intermediate precision
– Part 4: Trueness
– Part 5: Alternative methods for precision
– Part 6: Use of accuracy values

15
ISO 5725 Procedures

◼ Part 2 (precision)
– Selection of laboratories
– Selection of materials (levels)
– Statistical analysis
◼ Outliers (labs, samples, replicates)
◼ Graphics (h and k statistics)

◼ 1-way ANOVA for each level

◼ Linear model for r and R vs. level

16
ISO 5725 Procedures

◼ Part 3 (intermediate precision)

– Similar to Part 2
– Consideration of sources
◼ Time
◼ Operator

◼ Equipment

◼ Conditions

– Possible use of nested ANOVA

17
ISO 5725 Procedures

◼ Part 4 (Trueness)
– Comparison to reference method
◼ By level
◼ Across levels

18
SC 6: Limit of Detection

◼ ISO 11843: Capability of Detection

– Part 1: Terms and definitions
– 2: Methodology in linear calibration case
– 3: When no calibration data exist
– 4: Comparing minimum detectable value
with a given value
– 5: Linear and nonlinear calibration (under
development)
19
SC 6: 11843 Procedures

◼ “Currie” model
– Considers Type 1 and Type 2 error
(consistent with new VIM)
– Assumes normal distribution
– Assumes repeatability conditions
– No “less than” values allowed

20
SC 6: Other Documents

◼ ISO 10576: Conformity with Specified

Requirements
◼ ISO 11095: Linear calibration using
reference materials
◼ Issues with Measurement Uncertainty
– Formal Liaison to JCGM WG1

21
CLSI Evaluation Protocols

◼ Protocols for evaluating clinical

method capability
◼ Consistent with ISO
– Language for clinical lab
– User friendly
◼ Guidance: NOT mandatory
◼ Strong preference by FDA CDRH
22
EP Accuracy Documents

◼ EP5: Precision of Quantitative Methods

◼ EP9: Method Comparison and Bias
Estimation
◼ EP12: Evaluation of Qualitative Test
Performance
◼ EP21: Estimation of Total Analytical Error
◼ GP10 (EP24): Accuracy Estimation with
Receiver Operating Characteristic Plots
23
EP 5 Procedures

◼ Nested 2 factor design

– Days (at least 20)
– Runs (usually 2/day)
– (revision: add device, calibration)
◼ Two or more levels (revision: 5 or
more)

24
EP 9 Procedures

◼ Comparison with reference method

◼ Use certified materials of patient
samples
◼ Test entire measuring interval

25
EP 21 Procedures

◼ Combines precision and bias

◼ Requires setting goals

26
EP Process Capability

◼ EP6: Linearity of Quantitative Methods

◼ EP17: Limits of Detection and
Quantification

27
EP 6 Procedures

◼ Straight line and polynomial regression

◼ Statistical significance of nonlinear
coefficients
◼ Difference between straight line and
polynomial, compared to goal, at
every point (not same as CAP LN)

28
EP 17 Procedures

◼ “Currie” method, Type I and Type II

errors
◼ Truncated at zero
◼ Nonparametric
◼ LOQ based on goal for error

29
EP Method Verification
◼ EP7: Interference Testing
◼ EP10: Preliminary Evaluation of Quantitative
Methods
◼ EP14: Evaluation of Matrix Effects
◼ EP15: User Verification of Precision and
Trueness
◼ EP18: Quality Management for Unit-Use
Testing

30
EP Risk Management

◼ EP22: Establishment of Manufacturer's

Recommendations for User Quality
Control (under development)
◼ EP23: User Verification of
Manufacturer's Recommendations for
Quality Control (about to start)
◼ Started as a result of CMS initiative on
“Equivalent Quality Control” (EQC)
31
Other CLSI Activities

◼ Other Area Committees

– Field specific (Chemistry, Hematology,
Microbiology, Molecular Methods, General
Lab Practice, etc.)
◼ Secretariat for ISO TC212: IVD
Devices
– Also for US TAG to TC 212

32
Contact Information

◼ General TC 69: www.iso.org

◼ US TAG: standards@asq.org
◼ CLSI: www.clsi.org
◼ CLSI EP: lschmidt@clsi.org
◼ Me: tholen.dan@gmail.com
1.231.929.1721

33
Thank you!
(Questions?)

34