Method validation

Yash Pal Agrawal

10/15/03
Further Reading: Basic method validation: Training in analytical
quality management for healthcare laboratories by James O
Westgard (copy available at the VA)
 Scenario
• As lab.director you are testing a new
chemistry analyzer. You have done all the
tests, statistics etc which the med.tech gives
you in a neat 100 page folder.
• Is the new analyzer (method) acceptable ?
 How to know that the new
method/analyzer is acceptable ?
• Answer: ERROR ASSESSMENT.
• Amount of error won’t affect interpretation
of the test result and compromise patient
care
 Correlation between 2 methods
• Cholesterol assays
R=0.999
• Intercept 15 mg/dL
• Information about the

New Method
size of the analytical
error is more useful

Old method
 ERROR ASSESSMENT
• What kind of analytical errors may occur?
• What expts. provide data about those errors
• How to perform those experiments ?
• How much data to collect to assess errors ?
• What statistics provides best estimates of
errors ?
• What size errors are allowable ?
 Method Validation Regulations
http://cms.hhs.gov/clia/default.asp
• Waived tests:Follow manufacturer’s instr.
• Unmodified mod-high complexity:
– 1. Replication -Æ imprecision (Random error)
– 2. Comparison of methods -Æ Bias (Systematic
Error)
– 3. Linearity -Æ reportable range
– 4. Reference range: own or text
 Method Validation Regulations
http://cms.hhs.gov/clia/default.asp
• Modified moderate-high complexity:
– Previous 4 requirements PLUS
– 5. Detection Limit: -Æ analytical sensitivity
– 6. Interference:Æ constant interferences
– 7. Recovery: Æ proportional interferences
 CLIA quality requirements
• Absolute concentration limits: target value
± 1 mg/dL for Ca++
• As a %: target value ± 10 % for albumin
• As a distribution of a survey group: Target
value ± 3SD for TSH
 Linearity/Reportable range
• Lowest and highest
test results that can be
reported
• Draw line manually or
by linear regression
• Check 4-5 levels in
triplicates
 Types of analytical errors
• 1. Random error or imprecision
• 2. Systematic error or inaccuracy
– Constant error
– Proportional error
• 3. Total error: Random + Systematic
 1. Random Error (imprecision)
Observed
• Replicate measurements mean
on single sample
• Error which is positive or
negative
• Direction/magnitude
cannot be predicted
• SD and CV of a set of
results express the random
error
235 265
2. Systematic error or inaccuracy
True value
• Systematic error
always in one
direction-Æ causes
test results to be high
or low
• How high or low =
Bias
235 265
• Two types: Constant
or proportional error Bias
 3. Total Error
True value
Observed value
• RE +SE=Total Error
• A single measurement
can be in error by the
expected bias + 2SD
(random error)
235 265

Bias RE

Total Error
 Errors
Do replicates
Interference (within/between run)
study
Proportional
Constant RE SE
SE
New Method

RE

Recovery
study
Old method
 Determination of Random error
• Test results on 20 samples of the same material
• Time factor: within run, within day, between days
• Sample Matrix (blood, urine, CSF): Standards,
control, pools
• Test at medical decision levels
• Short term: sw-run or sw-day < 0.25 Tea
• Long term: S tot < 0.33 Tea
• E.g. ALT target value ± 20 %
Determination of Bias/inaccuracy
• Comparison of methods experiment
• Minimum 40 patient samples that cover
reportable range
• Preferably duplicates, time period factors
• Sample stability
• Plot as “difference plot” if 1:1 agreement
expected or as “comparison plot” if not
Determination of Bias/inaccuracy
• Which statistics to use:
– Linear regression : when wide analytical range.
E.g. glucose
– T-test: narrow analytical range, e.g. K
Determination of Bias/inaccuracy
• Linear Regression:
– Yc = a + bXc, slope=b, intercept = a
– Y= 2.0 + 1.03X, thus Y value for a decision
value of Chol at 200 mg/dL is = 208
– The systematic error at level of 200 mg/dL is (2
+ 1.03 * 200)=208, minus 200 =8
 Difference Plot
Difference (Test-Comparative method)

Zero
line

Comparative method result

Determination of Bias/inaccuracy
• Paired t-test. Also provides SD of the
differences as a t-value (i.e. is there a
difference or bias between the 2 methods)
• E.g. n=40, Na = 141 mmol/L and 138.5
mmol/L by other method. Bias = 2.5
mmol/L
 Determination of Interference
Add interference
• To estimate constant Add water
systematic error
• Interferences such as
bilirubin, hemolysis,
lipemia
water
Analyte, e.g. glucose
6 glucose _ 6 glucose
Interference, e.g. lipemia
Zero bias
 Determination of Recovery
• Used less frequently Add analyte Add water
• To estimate
proportional SE
• Caused by a substance
in sample matrix that
reacts with analyte and
competes with
analytical reagents (10 G-6G/4G) * 100 = 100 %
recovery
Determination of Detection limit
Blank Spiked
• To estimate the lowest sample sample
conc. of analyte that
can be measured
• Blank solution= zero
conc. of analyte
• Spiked solution = low
conc. of analyte
 Types of detection limits
LLD BLD FS
• Lower limit of
detection, LLD
• Biologic limit of
detection, BLD
• Functional sensitivity,
FS
Measurement value
Zero or
blank Spiked samples
 Determination of LLD
LLD BLD FS
• Lower limit of
detection
• Mean of the blank +
2 SD
• LLD=mean blk + Zs blk
where Z= 2 Measurement value
Zero or
blank Spiked samples
 Determination of BLD
LLD BLD FS
• Biologic limit of
detection, BLD
• LLD + 2 SD of a
spiked sample whose
conc. Approximates
the detection limit
Measurement value
• Represents real world
Zero or
blank Spiked samples
 Determination of FS
LLD BLD FS
• Functional Sensitivity,
FS
• Mean conc. of spiked
sample whose cv is 20
%
• Many samples need to
Measurement value
be run
Zero or
blank Spiked samples
 Transference of reference
intervals
• IFCC gives formal protocols, n=120 for
each sub group
• From literature: Divine judgement
• Verified with 20 samples, if 2 or fewer
results fall outside the reported limits
• Verify with 40-60 samples: compare mean
and SD with reported ref. values
• Calculation from comparative method
 Method validation keys
• Linearity/reportable range
• Replication experiment
• Comparison of methods experiment
• Interference, recovery and detection limits
• Method performance compared with CLIA
recommendations
 Further reading
• Basic method validation: Training in
analytical quality management for
healthcare laboratories

By James O Westgard