r

Method Validation Studies

It’s a Jungle out There!
 r

Well Duh...
 r
Purposes of Method Validation
Studies:
• To quantifiably characterize system performance

• To assess potential for error

• To identify method-to-method differences

• To meet regulatory guidelines

 r

When to Do Method Validation

Studies:
• When considering purchasing a new system

• When placing a new system into service

• At regular intervals to assess on-going system performance

• When troubleshooting questionable system performance

 r
Basic Components of Method
Validation Studies:
Any or All of the Following...
• Precision Studies
• Accuracy Studies
• Range Validation/Verification Studies
• Reportable Range
• Critical Limits
• Analytical Measurement Range
• Clinically Reportable Range
• Calibration Verification / Linearity
• Reference Range Studies
 r
How to Do Method Validation
Studies:

Don’t wing it!!! Consult appropriate reference

guides.

• NCCLS evaluation document series

• Manufacturer’s package inserts, evaluation protocol manuals

and supplemental protocols.
 r
Precision Studies
• Characterize the reproducibility of a test system
• Performed by repetitive testing of the same sample
• Intra-run
• Inter-run
• Inter-instrument
• Statistical parameters
• Mean
• Standard Deviation (SD), mg/dL
• Coefficient of Variation (CV), %
• Acceptability Expectations…the lower the better
 r

Accuracy Studies

• Method-to-method studies comparing a POC instrument to a

properly calibrated primary Laboratory analyzer.
• For the purpose of assessing accuracy, the same samples
should be tested on both the Laboratory analyzer and the
Point-of-Care instrument.
Accuracy Studies, cont’d: r

• A typical Accuracy Study includes:

• all, or a representative sub-set, of Point-of-Care
instruments
• optimally at least 40 different samples (minimum of 20
for statistical validity)
• good distribution of sample values ranging from low to
high
• same samples tested on Point-of-Care instruments and
Laboratory analyzer. If capillary samples are compared
to venous glucose results, use only samples from fasting
patients (Capillary glucose levels can be 20-70 mg/dL
higher than venous levels at the same time in the non-
fasting state).
Accuracy Studies, cont’d r

• Statistical analysis of accuracy study results

typically includes:

• Regression analysis
• Bias analysis
Accuracy Studies, Cont’d r

Raw Data Table

Lab Accu-Chek Lab Accu-Chek

Glucose Glucose, Glucose Glucose,
mg/dL mg/dL mg/dL mg/dL
19 12 108 118
29 25 146 158
34 30 189 184
35 36 200 184
37 38 217 189
37 30 232 222
38 40 237 214
45 42 254 244
47 52 266 243
49 45 302 283
55 55 321 296
55 55 325 309
59 57 330 292
62 57 355 341
74 78 365 344
80 81 403 342
82 85 405 387
87 83 458 484
105 112 552 551
 Regression Analysis r
Plotting the Comparative Data...

Accu-Chek Vs Lab Regression Analysis

500

450

400
(Y - Axis) Accu-Chek Glucose, mg/dL

350

300

250

200

150

100

50

0
0 50 100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
 Regression Analysis r
Plotting the Comparative Data...

Accu-Chek Vs Lab Regression Analysis

500

450

400
(Y - Axis) Accu-Chek Glucose, mg/dL

350

300

250

200

150

100

50

0
0 50 100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
 Regression Analysis r
Plotting the Comparative Data...

Accu-Chek Vs Lab Regression Analysis

500

450

400
(Y - Axis) Accu-Chek Glucose, mg/dL

350

300

250

200

150

100

50

0
0 50 100 150 200 250 300 350 400 450 500
(X - Axis) Lab Glucose, mg/dL
Accuracy Studies, cont’d r

• Regression Statistics Review:

• Correlation Coefficient (r) - characterizes the dispersion of
results around the line of best fit.
• Slope - The “lean” of the line of best fit (proportional bias)
• Y-Intercept - the point at which the line of best fit
intersects the Y axis. (constant bias)

• Acceptability Criteria:
• Correlation Coefficient (r) - the closer to 1.0 the better
• Slope - The closer to 1.0 the better
• Y-Intercept - the closer to zero the better
 r
Accuracy Studies, Cont’d

The Equation of the Line of Best Fit:

y = mx + b
m = Slope
b = Y Intercept
 r
Accuracy Studies, Cont’d

The Equation of the Line of Best Fit

y = 0.991x + 8.4

Slope

Y-intercept
 r
Accuracy Studies, Cont’d

The Equation of the Line of Best Fit

y = 0.991x + 8.4

Slope
Y-intercept
Accuracy Studies, cont’d: r

• Bias Analysis:
• Calculate mg/dL or % difference between Point-of-Care
instrument and Laboratory analyzer for each set of paired
data-points.

• Acceptability Criteria (Glucose):

• 95% of meter results within +/- 15 mg/dL of Laboratory
analyzer results below 75 mg/dL or within +/-20% of
Laboratory analyzer results equal to or greater than 75
mg/dL (NCCLS document C30A-2)
 r
Bias Analysis, Cont’d

Bias Chart
Accu-Chek vs Lab Bias Chart

50
45
40
35
30
25
20
15
A ccu-C hek B ias*

10
5
0
-5 0 50 100 150 200 250 300 350 400 450 500
-10
-15
-20
-25
-30
-35
*Bias is calculated as mg/dL difference from lab results
-40
<75 mg/dL and as % difference from lab results >75 mg/dL.
-45
-50
Lab Glucose, mg/dL
 r
Range Validation Studies
• Definitions
• Reportable Range - The range of results for which a system
has been proven to yield numerically accurate results
• Analytical Measurement Range (AMR) - The range of
results through which a method yields numerical values
• Clinically reportable range - (CRR) An extension of the
AMR supplemented by dilution or concentration protocols
• Calibration Verification - The process of verifying that a
system is properly calibrated
• Critical Limits - Low and high result limits which, when
exceeded, require follow up action
• Reference Range - Normal range
 r
Range Validation Studies
• Definitions
• Analytical Measurement Range (AMR) - The range of
results through which a method yields numerical values
• Clinically reportable range - (CRR) An extension of
the AMR supplemented by dilution or concentration
protocols
• Calibration Verification - The process of verifying that a
system is properly calibrated
• Reportable Range - The range of results for which a system
has been proven to yield numerically accurate results
• Critical Limits - Low and high result limits which, when
exceeded, require follow up action
• Reference Range - Normal range
Range Validation Studies, Cont’d r

• Validation of Reportable Range, Analytical

Measurement Range, Reference Range,
Critical Limits and Calibration Verification
(CAP POC July 2003 Checklist, Page 34 Commentary)
• Can all be accomplished through the same protocol if the
range of testing spans the broadest range (AMR)
• Materials used must be “matrix appropriate” such as:
• Split blood samples
• linearity/calibration verification materials supplied
by manufacturer or other vendor
• However...
 r
CAP POC.08700 - If the laboratory has more than one
method-system for performing tests for a given analyte, are
they checked against each other at least twice a year for
correlation of patient results?

NOTE: This includes same or different instrument

makes/models. This comparison must include all instruments. If
multiple methods are used for a particular analyte, comparing a
subset of each method to one another is acceptable only after all
like instruments have been compared to one another to ensure
correlation and establish that a representative subset is used in
the method-to-method correlation.
 r
CAP POC.08700, Cont’d: Significant changes to
the commentary!!!

• “POCT glucose devices are a special case in which a large number of

devices may be in use in an institution. In this case alternate approaches
are acceptable for documenting correlation of patient results among the
devices and with the main laboratory. One approach is to document for a
lot of reagent strips the agreement between patient results for a laboratory
method and representative POCT device(s) and simultaneously collect
results for QC materials run on the POCT devices. “
• “Other correlation schemes may be suitable. The critical factor is to have
a mechanism that documents the agreement between patient results
performed by the POCT glucose devices and the main laboratory.”
 r

Where to Find the Method Validation

Regulations

CLIA
CAP
JCAHO
 r
Regulatory Guidelines for Method
Validation:

• CLIA 88 Final Rule - (January 24, 2003)

• CAP General Checklist (July 2003)
• CAP Point-of-Care Checklist (July 2003)
• JCAHO - WT and QC Section (2004 Pre-Publication with
Crosswalks from 2003 Guidelines)
• NCCLS - C30A-2 - (Item 6.3.2).
 r
CLIA Guidelines for Method Validation:

• CLIA 88 Final Rule - Subpart K (Quality Systems for Non-

waived Tests) 493.1253
• Revised January 24, 2003
• Outlines requirements for unmodified “non-waived”,
FDA cleared tests
• By definition, waived tests are not subject to items in
Subpart K
 r
CAP Guidelines for Method Validation:

• CAP Laboratory General Checklist

• Revised July 2003 (GEN.42020 - GEN.42163)
• Does not distinguish based on CLIA designated test
complexity
• Requires validation of precision (intra- and inter-run) and
accuracy
• Requires verification of analytic sensitivity (Low end)
• Requires assessment of analytical interference
• Requires verification of reportable range (AMR)
• Requires verification of reference range
 r
CAP Guidelines for Method Validation:

• CAP Laboratory Point-of-Care Checklist

• Revised July 2003 (POC.08200 - POC.08700)
• Does not distinguish based on CLIA designated test
complexity
• Requires calibration verification
• Requires validation of Analytical Measurement Range
• Requires method-to-method and instrument-to-instrument
correlation
 r
JCAHO Guidelines for Method Validation:

• WT Section (2004 Pre-Publication with Crosswalk from 2003

Guidelines)
• Item WT.1.40 - Non-specific reference to requiring
“Equipment performance evaluation”
• QC Section (2004 Pre-Publication with Crosswalk from 2003
Guidelines)
• Items QC.1.70 - Method Validation requirements for
precision, accuracy and reportable range apply to FDA
cleared, unmodified moderate complexity tests.
• Requires reference range verification for all new
methods
 r
JCAHO Guidelines for Method Validation:

• QC Section (2004 Pre-Publication with Crosswalk from 2003

Guidelines)
• Item QC.1.70
• Method Validation requirements for precision,
accuracy and reportable range apply to FDA cleared,
unmodified moderate complexity tests.
• Requires reference range verification for all new
methods
• Item QC.1.80
• Method-to-method and instrument-to-instrument
requirement specifies that waived tests are exempt.
 r
Informative Regulatory Links
• http://www.phppo.cdc.gov/clia/regs2/toc.asp (CLIA Regs
with 1/24/03 updates)
• http://www.aacc.org/meetings/qc/ (Oct. 29th Teleconference
• http://www.jcaho.org/accredited+organizations/
laboratory+services/standards/new+standards/
2004+standards.htm (JCAHO Pre-publication 2004 Standards
with Crosswalk)
• http://www.cap.org/html/ftpdirectory/checklistftp.html (CAP
Checklist download page)
• http://www.dgrhoads.com/ee5/ee5.sht (EP Evaluator Release
5)
 r
Links to Informative Articles

• http://www.cap.org/captoday/archive/2003/0303/
language_barrier.html (CAP Today on AMR/CRR and
Calibration Verification)
• http://www.westgard.com/cliafinalrule5.htm (CLIA 1/24/03
method validation changes by Sharon Ehrmyeyer, Ph.D)
• http://www.westgard.com/essay50.htm (CLIA Key 1/24/03
Changes by Dr Westgard)
• http://www.westgard.com/essay15.htm (Method Validations:
The Inner, Hidden, Deeper, Secret Meaning by Dr Westgard)
 r

Method Validation Studies

Questions and Discussion

r