Method Validation &

Verification
(Introduction to Method Validation)

Wilai Chalermchan
Senior Laboratory Advisor (Contractor)
DGHT/Global HIV/AIDS Program
Thailand MOPH – U.S. CDC Collaboration (TUC)
1
Contents
• Introduction
Standard requirements
• Validation/Verification Parameters, Experiments
and Process
• Samples of method validation/verification plan and
experiments
• Summary

2
Validation and Verification (MT Std)
Validation (การสอบทวน ) หมายถึงกิจกรรม หรือ
กระบวนการ เพื่อพิสจู น์วา่ ขันตอนการทดสอบ
้ กระบวนการ
ทดสอบ ระบบการทดสอบ เครื่ องมือ หรื อวิธีการทดสอบที่
นามาใช้ สามารถใช้ งานได้ ตามวัตถุประสงค์ที่กาหนดไว้

Verification (การทวนสอบ ) หมายถึงการยืนยัน

โดยการตรวจสอบและมีหลักฐานแสดงว่าเป็ นไปตาม
ข้ อกาหนด
3
Validation and Verification (ISO 15189)
VALIDATION: confirmation, through provision of objective evidence, that
the requirements for a specific intended use or application have been fulfilled
การจัดหาหลักฐานที่ยืนยันว่าวิธีทดสอบมีคณ
ุ สมบัติเป็ นไปตาม......
….ข้ อกาหนดต่ างๆในการนามาใช้ งานหรือตามความมุ่ง
หมายในการใช้ งานที่กาหนด

VERIFICATION: confirmation, through provision of objective evidence, that

specified requirements have been fulfilled
การจัดหาหลักฐานที่ยืนยันว่าวิธีทดสอบมีคณ
ุ สมบัติเป็ นไปตาม.....
….ความต้ องการเฉพาะ
4
 Standard Requirements (MT Std.)
5.4.1.1 ....หากใช้ วิธีวิเคราะห์ที่กาหนดขึ ้นเอง ..ไม่ใช่วิธีสากล หรื อ
ดัดแปลง ต้ องมีข้อมูลผ่านการสอบทวน (validation) เทียบเคียง
กับวิธีที่สากลยอมรับในขณะนัน้ บันทึกผลการสอบทวนและการ
ทวนสอบ คูม่ ือวิธีปฏิบตั ิการวิเคราะห์
• ข้ อมูลการสอบทวน (validation) ได้ จากผู้ผลิต หรื อผู้พฒ ั นาวิธี
วิเคราะห์ ข้ อมูลต้ องแสดง ถึงความเที่ยงตรง ถูกต้ อง แม่ นยา
สามารถระบุความคลาดเคลื่อน ความจาเพาะ ความไว ช่ วง
วิเคราะห์ และความสอดคล้ องกับอาการทางคลินิกได้
Standard Requirements (MT Std.)
How….
• ข้ อมูลการทวนสอบ (verification) ได้ จากข้ อมูลเชิงประจักษ์ เพื่อ
ยืนยันได้ ว่าเหมาะสมกับการใช้ งานในงานบริการ
verification parameter…..?
5.4.1.2 ทบทวนวิธีวิเคราะห์ อย่างน้ อยปี ละ 1 ครัง้ ... บันทึก
5.4.1.3 มีการทบทวนค่าอ้ างอิง อย่างน้ อยปี ละ 1 ครัง้
5.4.2 คุณลักษณะของแต่ ละวิธีวเิ คราะห์ ต้องเหมาะสม
สอดคล้ องตามวัตถุ ประสงค์ การใช้ งาน และตามความ
ต้ องการของผู้ใช้ บริการ
6
 Method Evaluation
Validation Verification
• Establishment of performance • Confirmation of known
characteristic by defining and performance parameters
characterizing the magnitude of published by the
the analytical error present developers of the method.
• Performed by the developers of • Performed by the
the method (e.g. laboratory to explore error
manufacturers) in system
• Requires extensive studies to • Requires studies to
reveal and assess the error confirm manufacturer's
present claims

Slide from Anna Murphy, SLAMTA program 7

Why method need to be
validated/verified?

8
Propose of Method
Validation/Verification
• Identification of Sources and Quantitation of Potential
errors
• Determination if Method is Acceptable for Intended Use
• Establish Proof that a Method Can be Used for Decision
Making
• Satisfy Regulatory Requirements

9
 Establishing a Laboratory Routine Test
Select, Evaluate
Diagnostic Test Process for
Establishing a
Select Method of Routine Test
Analysis

Validate/Verify Method
Method Performance Improvement
Define test SOP -
Select appropriate QC - - Maintain Method
Personnel training - Implement Method
- Prevent Problem

Acquire Check with Report

Perform Test Results
Specimen Statistical QC
A Routine Laboratory Testing Process
10
Why method need to be
validated/verified?

11
Validation
Parameters and Experiments

12
Common Validation Parameters for a
Quantitative Test
• Precision
• Accuracy
• Linearity
 Analytical Measurement range
 Clinical Reportable Range
• Sensitivity
• Limit of detection, Limit of quantification
• Specificity
• Reference Interval
13
Performance Characteristic required
by Thai FDA on HIV Test
• Analytical sensitivity
• Diagnostic sensitivity
• Diagnostic specificity
• Reproducibility Precision
• Inhibition
• Cross contamination
• Whole system failure rate
14
What should I do?

Performance Characteristics must be

relevant to the laboratory intend use.
5.4.2 คุณลักษณะของแต่ละวิธีวิเคราะห์ต้อง
เหมาะสมสอดคล้ องตามวัตถุ ประสงค์การใช้
งาน และตามความต้ องการของผู้ใช้ บริ การ
15
The Guideline
CLIA (Clinical Laboratory Improvement Amendment)
Non-waived test
• Application: after April 2003
• Unmodified, FDA-approved test
1. Performance specifications comparable to the
manufacturer
• Accuracy
• Precision
• Reportable range
2. Verify manufacturer’s reference interval
(normal range)
16
 Performance Characteristic to consider
Example:
Application Performance Characteristic
• diseases monitoring • Reproducibility
• For diagnosis of • Specificity, False positive
infectious diseases
• Blood safety • Sensitivity, LOD
• High workload • System failure rate

17
Performance Characteristic and Experiments
Performance
Experiments
Characteristic
Accuracy • Comparison methods to determine
systemic error (SE) or bias
Precision • Replication experiment
• Repeatability
• Intermediate precision
• Reproducibility
Reportable range • Linearity experiment

Reference internal • Study sample from normal/healthy

person
• Adopt from the publication or
reference lab 18
 Verification Process
• Establish General Validation/verification procedure;
• Define the application, purpose and scope of the method use;
• Method selection; (from manufacturer claim)
• Establish validation/verification plan/protocol;
• Define the validation/verification parameters and experiments;
• State of the acceptability;
• Perform pre-validation experiments:
• Adjust method parameters and/or acceptance criteria if
necessary:
• Perform full validation experiments:
• Record, report with statement of acceptance;
• Method approval 19
Define Laboratory Application
Requirements
Three types of test characteristic can be considered
•Application Characteristic: Cost, type an vol of spec,
TAT, workload, space, portability, personnel
•Methodology Characteristic: principle contribute to
the best performance, chemical reaction, rigor of
analytical procedure
•Performance Characteristic: reportable range,
precision, accuracy, detection limit
20
 Sample of defined POC Performance
Characteristic or specific requirements
Application Methodology Performance
• WB from finger stick or • Enzymatic • Specified
with anti coagulant reaction reportable range
• Ease of use for non lab- • Built in • Reproducibility
personnel calibration and less than 3%CV
• RT° storage traceable
• Minimal maintenance • TAT of 10 mins
and downtime or less
• Cost less than..
• Size, weight, portable

21
Considerations Prior to
Method Validation/Verification
Consider Suitability of
• Instrument
Status of Qualification and Calibration
• Materials
Status of Reference Standards, reference material,
Reagents lots
• Analyst
Status of Training and Qualification Records
• Documentation
approved protocol with pre-established acceptance
criteria and worksheet 22
Samples of Method Verification
Experiment

23
Sample 1: Method evaluation/verification at National level
AIM: Compare two new NAT methods for use in routine
infectious screening……….
STUDY DESIGN AND METHODS:
The sensitivity, specificity, and robustness were
determined by testing 486,676 seronegative blood donations
within one year.
Samples from each day of collection were divided into two
sets; the odd-numbered samples were tested individually on the TIGRIS and
the even-numbered samples were tested in pools of 6 on the cobas s 201.

The status of reactive samples was confirmed by…

- duplicate testing of samples from the plasma bag to
calculate the test specificity.
- tested on the alternate system and followed up.
Reference: Phikulsod et al. One-year experience of nucleic acid technology testing for human immunodeficiency virus Type 1, hepatitis
C virus, and hepatitis B virus in Thai blood donations. TRANSFUSION, Volume 49, P 1126-1135 24
Evaluation Parameters
Analytes: HIV-1, HBV and HCV nucleic acid
Analytical sensitivity
WHO International Standards for HCV (NIBSC 96/798), HBV (NIBSC
97/746), and HIV-1 (NIBSC 97/760).
The standards were diluted to the 95% detection limit for each
assay.
• 3, 10, and 50 IU/mL for HBV, HCV, and HIV-1 for the Roche MPX test.
• 3, 10 and 30 IU/mL for HBV, HCV, and HIV-1, for the Ultrio test IU/mL
for HBV, HCV, and HIV-1, respectively
• testing individually 24 replicates

25
Cross contamination
studied by including five samples containing high
titers of HBV (>108 IU/mL), HCV (105 IU/mL, NIBSC 96/798), and HIV-1 (280,000 copies/mL,
determined by the COBAS TaqMan HIV-1 test for use with the High Pure s ystem), which were
randomly distributed in the batches of routine blood
donor samples that were tested in 1 day.
Genotype detection (detectability)
Genotypes of the three viruses, isolated in Thailand,
were
- 41 HIV-1 isolates, 12 subtype B and 29 subtype A/E (CRF01_AE);
- 20 HBV isolates, 8 genotype B and 12 genotype C; and
- 20 HCV isolates, 4 genotype 1a, 14 genotype 3a, and 2 genotype 3b, were tested.

26
Clinical specificity and sensitivity
- The initial-reactive and repeat-reactive rates for each
test were calculated.
- Follow-up study of donors with reactive donations
All donors with confirmed reactive donations were
followed up until seroconversion or until a conclusion
about the donor status could be reached. (Sensitivity)
- 5000 donor samples, found nonreactive by each test,
were retested using the alternate test. The testing was
included during the routine testing (Specificity)

27
Whole system failure rate
The overall failure rate for the two systems was
monitored. Any failure that resulted in the inability to
release donor results was regarded as a system
failure.
In addition, the human errors and the invalid controls
(both run controls, internal controls, and in the case of
the TIGRIS, the calibrators) were monitored.

28
Sample 2: Verification of new Chemistry Analyzer
Validation Plan
1) Aim: The verification will be conducted on the X Chemistry Analyzer , serial
number …. for AST (Aspartate Amino Transferase) compare with in-service
method Y during dd/mm/yy…. to dd/mm/yy ……
2) Description …….background, method, scope of use and
specific requirements
3, Parameters, experiment methods and result record
a. Precision: Repeatability, Intermediate precision
- Description, detail procedure, material and acceptance
criteria…….
b. Accuracy/correlation ………. c. Linearity: ……….
d. Sensitivity (LOD and LOQ): e. Specificity/interferance: ……….
f. Reference Ranges: ……….
4. Study Conclusion and Method Approval: ……….
5. Reference: ……….
29
 Precision Experiment
Short Term (within-run)/Repeatability
Sample: Method
Manufacturer
XYZ Cheistry Analyzer
ABC
Experimental Analyte
Units of Measure
AST
U/L
Sample name/Description
worksheet Material 1 QC Multi 1; Lot# QC123; exp 30-Aug 20XX
Material 2 QC Multi 2; ; Lot# QC223; exp 30-Aug 20XX
Material 3
Analyst Miss USA
Date 3-Nov-15
Experiment Results Preliminary Estimate of Precision, Short Term
Run # Material 1 Material 2 Material 3 Material 1 Material 2 Material 3
1 35 120 Laboratory
2 34 119 Mean 35.0 U/L 120.0 U/L
3 35 120 SD 0.8 U/L 1.5 U/L
4 35 121 %CV 2.20% 1.25%
5 35 120
6 36 121 Manufacturer's Claim
7 35 120 Mean 36.6 U/L 128 U/L
8 34 118 SD 0.3 U/L 1.0 U/L
9 36 120 %CV 0.80% 0.40%
10 35 122
11 35 119 25% of CLIA Allowable error is 5%
12 35 120 Short term precision is acceptable
13 35 121 (acceptable/unacceptable)
14 35 117
15 34 119
16 35 118
17 35 120
18 37 122
19 36 123 30
20 34 120
What’s next?

31
On-going Verification
• Test performed by multiple systems required
comparison of test results (Comparability study)
• Monitoring of test precision through QC
• Monitoring of accuracy PT/ EQA results
No PT program- at least semi annual re-
validate/verified
• Kit lot verification

32
Re-validation

When
• Instrument repaired, relocate
• Method performance have been significant
changed
• Scope of application has been change

33
What are the evidence that an
assessor looking for?

Minimum
• General method validation/verification procedure
• Validation/Verification plan/protocol
• Records
• On-going verification measure(s) and records

34
Summary
Method verification
• Procedure, plan and records;
• Meet specific intend use and extensive;
• before conducting the study, qualify instrument and
ensure personnel competency;
• observe errors that may have in our laboratory
system;
• First step in establish quality control procedure
• on-going verification, re-validation

35
References

36
Questions & Comments

37