Validation protocol

What is Validation?

Validation is procedureused for checking that a system meets requirements and

specifications claimed, and fulfills its intended purpose.

Our validation depends on the recommendations from accreditation agencies like

CLIA and CAP
The CLIA minimum requirements is four procedures, while the CAP required of
nine different procedures, we will cover the CAP procedures in this manual.

Procedures are:

A-Quantitative test

1- Precision - CLSI -EP15

2- Accuracy - CLSI-EP09-A2
3- Linearity–CLSI- EP06
4- Comparison - CLSI-EP15-A2
5- Sensitivity - CLSI-EP17-A
6- Reference range - CLSI- C28
7- Carryover - CLSI- EP10
8- Reportable range -

B- Qualitative tests - CLSI-EP15-A2

1- Precision
2- Comparison (Concordance)
3- Cutoff verification
 Precision - CLSI -EP15
DEFINATION:Precision indicates how well the assay or system
Provides the same result when a given sample is
Tested repeatedly.
We have twomethods of precision
- Simple precision -within-run (Repeatability)
- Complex Precision - withinDay (reproducibility).
-
• Simple Precision Within-run (Repeatability)
 Within-run precision provides a "best case” estimate of the
expected performance.
 there is minimal opportunity for conditions to change
during the course of the run

• Complex precision - within days (Reproducibility)

 Is the most realistic assessment of the Performance.
 Because it includes variables such as different operators and
laboratory conditions.
Simple precision
• Purpose: To establish or verify within-run precision and to
Verify manufacturer’s claims only if the
Manufacturer makes within-run precision claims.

• Materials:need minimum of two or three samples of below

- Quality control materials.
- Calibrator
- Patient samples
The samples should cover the AMR (Low,Med,High)
• Experiment:
- Assay each specimen at least 20 times in the same
run.
- SD and CV: are to be calculated. They should be
compared with the manufacturer’s claims at similar
concentrations.
• Special requirements :
- No special requirements needed.
complex precision
• Purpose:To establish or verify a total precision and to
verify manufacturer’s claims only if the manufacturer
makes total and within-run precision claims.

• Materials: need minimum of two or three samples of below

- Quality control materials.
- Calibrator
- Patient samples
The samples should cover the AMR (Low,Med,High)
Experiment:
- 2 replicates per run, 2 run per day for at least 5
days.
SD and CV: are to be calculated. They should be compared
with the manufacturer’s claims at similar concentrations.
• Special requirements :
- No special requirements needed.
 Accuracy - CLSI-EP09-A2

Definition : Accuracy is the agreement between test result and “true” result.
CLSI Guidelines for Trueness (Measured as Bias).

Purpose: Is to check if your machine is accurate and capable to

Recover known material within allowable percentage.

Materials:need minimum of 3 to 5 samples Known concentrations of

Below Samples and it should cover the low, normal, high
(Measuringrange).
- Quality control materials (Lev 1, Lev 2, Lev 3)
- Calibrator (Different lot # from machine calibrated with)
- Patient samples.

Experiment: Each sample should be replicates 3 to 5 replicates.

Calculations:% Recovery= (Mean / Target) x100%
Acceptance Criteria:If the % recovery is within ±10 %,then the method
Is accurate.
 Linearity –CLSI- EP06
Purpose: is to assess the lowest and highest test results that are
Reliable and can be reported.
• Manufacturers make claims for the reportable range of their
methods by the stating the upper and lower limits of the range.
• CLIA recommends that laboratories verify the reportable range of
all moderate and high complexity tests.

Materials: need minimum of 5 samples prefer 7 samples which

Cover the whole AMR with known concentrations
Samples could be:
- Linearity materials e.g CAP, Audit Micro
- Calibrator(different lot # the machine calibrated with)
- Patient sample with high conc exceed AMR and a serial
dilution to get 5-7 different concentrations :
-
To perform serial dilution for (Cal F) or high pt. sample to get the
following:
 - Highest concentration is considered 100%- Specimen 5
- Lowest concentration is considered 0%- Specimen 1
- Mid concentration is considered 50%- 2 quarters of 5 + 2
quarters of 1 (50:50)
- 25% concentration- 1 quarter of 5 + 3 quarters of 1
- 75% concentration- 1 quarter of 1 + 3 quarters of 5
-
Or you can use the following calculation to get 7 samples:
- Highest concentration is considered 100%- Specimen 5
- 2nd highest concentration is considered 80% - 4 parts of 5 + 1
part of cal 1
- 3rd highest concentration is considered 60% - 3 parts of 5 + 2
part of cal 1
Continue Linearity –CLSI- EP06

- 4th highest concentration is considered 40% - 2 parts of 5 + 3

part of cal 1
- 5th highest concentration is considered 20% - 1 parts of 5 + 4
part of cal 1
- 6th highest concentration is considered 10% - 0.5 parts of 5 +
4.5 part of cal 1
- 7th concentration is considered 0% is cal 1

Experiment: Each sample should be run 3 to 5 replicates.

Acceptance Criteria:If the % recovery is within ±10 %,then the method

Is Linear and linearity verified.
 Comparison - CLSI-EP15-A2

Purpose: When converting to a new method or a new analyser, it

is important to determine if the new method or new analyser
provides essentially equivalent results.
Comparative method
Using Reference method
 A test method is compared to a reference method (Correct
method), any deviation from the reference method is related
to a test method.
Using routine method
  A relative method is two methods compared to each other one
a test method and the other is routine method (Current) , any
deviation , one of the two method is not correct.

Materials: preferred patient samples covering AMR .

Experiment:
• A minimum of 20 - 40 different patient specimens should be
tested by the two methods.
• These specimens should be selected to cover the entire
working range of the method and should represent the

Continue Comparison - CLSI-EP15-A2

Spectrum of diseases expected in routine application of the

method.
• The actual number of specimens tested is less important than
the quality of those specimens.

• There is two methods to follow:

o Run these samples 20-40 samples in the same day
(same run) in both analyzers.
o To minimize any systematic errors that might occur in a
single run, a minimum of 5 days is recommended4-8
samples per day or 2 samples per day for 20 days.
o Samples should be run in duplicate in both analyzers.
 • Specimens should generally be analyzed within Three hours of
each other by the test and comparative methods. Except NH3
and Lactateshould run on the same time.

Acceptance Criteria:

- Acceptability must take into account methodological

differences and values outside of the following ranges may
be acceptable:
- Correlation Coefficient (R) should be more than 0.98
- Slope should be 0.95 - 1.05
- Intercept should be less than 1% of the linear high value.
 Continue Comparison - Qualitative

Qualitative Method Comparison

The purpose of this experiment is to compare two methods that report

results as Positive/Negative. QMC is useful in the following
circumstances:
 to determine the ability of an instrument to detect a
Pathological condition.

In this case, the reference method is a definitive diagnosis, or Gold

Standard. The key statistics are Sensitivity and Specificity, and the
proportion of False Positives and False Negatives.

 To compare the performance of two laboratory instruments,

perhaps when a new instrument is introduced into the lab.
Here both the reference method and the test method are subject to
experimental error. There is no reason toassume the reference method is
correct, or even that itis better than the test method. In fact, when
thereference method is an old method and the test methodis a new
method, it is quite possible that the newmethod is the better
method.When comparing two laboratory methods, the keystatistic is
Agreement -- do the two methods give the
Same results?

The procedure is the same as the quantitative except the followings:

- 20-40 samples needed for comparison.

- The samples should have 50% positive and 50% negatives.
- If not possible to have appositive samples should be spikes
with high positive control for that test.
- Don’t pool all positive controls with sample.
- Only spike one control each time.
- Run the samples in both instruments the reference and the
test instruments.
- You can run samples on one day or in different days, but the
same samples should be run on the same day (8 samples run
on A today, same samples should be on B today also).
- Then we calculate the matching between results.
- False positive, false negative will appear if 2 instruments are
not comparable.

Agreement: The percent of total cases in which the twomethods give

The same result. Related statistics for
Qualitative tests:
 Positive Agreement is the percent of cases that match
When the reference method is positive: TP/(TP+FP).
 Negative Agreement is the percent of cases that match
When the reference method is negative: TN/(TN+FN).

Chart Interpretation
The Bubble Chart is the equivalent of a scatter plot fornon-quantitative data.

- Green circles on the central diagonal represent points of agreement.

The size (area) of the circle is proportional to number of specimens.
The ideal chart has only green circles.
- Yellow and red circles represent points of disagreement.
- For a qualitative comparison
 red indicates falsepositives,
 Yellow indicates false negatives.

- For a semi quantitative comparison,

 Yellow circlesrepresent cases where the reference and test are in
Adjacent levels.
Red circles are cases where the testand reference are two or more
levels apart.
 Sensitivity - CLSI-EP17-A

We have 3 different sensitivity tests:

- Limit of blank (LOB)
- Limit of detection (LOD)
- Limit of quantitation (LOQ)

Limit of blank (LOB):is the highest apparent analyte concentration expected to be

Found whenreplicates of a blank sample containing no
analyte are tested.
LoB = mean(blank) + 1.645(SD ) (blank)

Limit of detection (LOD): is the lowest analytes concentration likely to be reliably

Distinguished from the LoB and at which detection is
Feasible. LoD is determined by utilizing both the
Measured LoB and test replicates of a sample known to
Contain a low concentration of analytes.
LoD = LoB + 1.645(SDlow concentration sample)

Limit of quantitation (LOQ) : concentration at which quantitative results can be

Reported with a high degree of confidence and CV
Is less than or equal to 20%.
LOQ = Mean blank + 10 * SD blank

Procedures :

LOB :
Materials: Distilled water or normal saline

Experiment: Run the DW 10 times

Calculations: calculate the mean and the SD

LoB = mean(blank) + 1.645(SD ) (blank)

LOD :
Materials:
 - Distilled water or normal saline or CAL- A
- Control (the lowest level)
- Patient sample with low concentration.
- CAL B

Experiment: method(1)

- Run the DW or NS or CAL-A 10 times

- Run the control, CAL-B, Pt sample 4-5 times
- The Low linearity should change to zero
- The decimal points should change to 4 digits

Calculations: calculate the mean and the SD

LOD= 2(Cal A RLU SD) X Cal B concentration

(Cal B RLU mean – Cal A RLU mean)

We use EP Evaluator for processing the data

LOQ
Limit of Quantitation – LOQ

This method verifies a LoQ (Limit of Quantitation) claim based on the definition
From CLSI document EP17- A. The LoQ is defined as the lowest amount of
analyte in a sample that can be quantitatively determined with the stated
Acceptable precision and trueness.
This method verifies functional sensitivity where the sensitivity is the
Concentration at which imprecision is 20% CV or less.

Procedure
1. Prepare a suitable sample with a concentration at or below the functional
Sensitivity stated in the ARCHITECT i System assay-specific package insert.
2. Prefer to have 5 to 7 different concentration around the claim value , 3 above
Andthree lower than the LOQ claim.
3. Samples can be prepared as follows:
 (-60%,-40%,-20%, Target, +20%, +40%, +60%).
4. Run the samples a total of 20 runs across multiple days, for example 4
Replicates on each of 5 days for all samples.
5. Run the results with EP Evaluator or any other software to calculate the LOQ

Data evaluation
Calculate the mean, SD, and %CV for the 20 replicates of all samples.
The claim is verified if the calculated %CV is less than or equal to 20%.
A worksheet is provided:

Experiment Procedure
1- Define the Target CV (usually 20%).
2- Prepare several (5-7) specimens that cover the rangeof concentrations
Where you expect the Target CV tooccur.Starting with cutoff (LOQ)
And Make +-20% from target, +-40% from target5 samples as follows:
(-40%, -20%, Target, +20%, +40%)
3- Assay each specimen 20 times, or 4 runs per day for at least 5days for
Eachspecimensubstantially improves the quality of the result.
4- From these results, EP Evaluator calculates the CV ateach
concentration.
The LOQ is the lowest concentrationfor which the CV is less than the or
equal to 20%.

.
 Carryover - CLSI- EP10

Carryover: CLSI- EP10

CLSI Guideline for Carryover EP10: includes an assessment of sample

carryover along with other parameters.

NOTE: EP10 is intended to determine if a device has unacceptable performance.

It is recognized in the CAP Chemistry Checklist as an acceptable way to

measure carryover.

Materials: 2 samples one with 0 values could be:

- CAL-A , normal saline, Distilled water
Other sample should have high concentration could be :
- CAL-F, patient result very high conc.
 Make 10 aliquots of high sample
 Make 11 aliquots of low sample

Experiment:Run the samples in the following order:

L1 L2 L3 H1 H2
L4 H3 H4 L5 L6
L7 L8 H5 H6 L9
H7 H8 L10 H9 H10
L11
 Low-Low mean and standard deviation: Mean and SD of the
replicates L2, L3, L6, L7, and L8 (low sample following a low
sample.)
 High-Low mean and standard deviation: Mean and SD of the
replicates L4, L5, L9, L10, and L11 (low sample following a high
sample.)
 Carryover: Difference between the High-Low mean and the Low-
Low mean.
 Error limit: 3 X the Low-Low SD
Acceptance Criteria:
The test passes if carryover does not exceed the error limit
 Reference range - CLSI- C28

1. Reference Intervals : CLSI- C28

Interpretive information for laboratory test results that is frequently provided as

the central 95% interval of results for a group of well-defined reference
individuals.
 Purpose: To verify the normal range population.

Materials: Obtain 20 normal samples, i.e. donor specimens.

Experiment: Run the 20 specimens once and check for outliers outside the
normal range.

Acceptance Criteria:
- If outliers≤2 (10%), the test passed.
- If outliers≥3, repeat the test with another 20 samples without
excluding the first 20 (total is 40)
- If outliers≥3, repeat the test with another 20 samples without
excluding the first 40 (total is 60)

 If it fails again then you need to establish the normal range .

To establish a new normal range you need to have :

- 120 normal samples for each sex, age.
- Run in EP evaluator to establish the normal range.
- Any outliers should be removed from the experiment.
 Summary of the procedures
procedure No. of samples No. of replicates No. of days
Precision 2 levels 20 runs 5 days
Accuracy 3-5 levels 3-5 runs Same day
Comparison(quantitative) 20-40 samples duplicate 5 days
Cover AMR
carryover 2 levels 10 high,11 low Same day
Linearity 5-7 levels 3-5 replicates same day
Sensitivity(LOD) 2 levels 10 for low, 5 for high Same day

Sensitivity(LOQ) 5-7 levels 20 replicates same day

Reference interval-verify 20 samples once Same day
Reference interval-Estab. 120 samples once Same day
Comparison(qualitative) 20-40 samples duplicate Same day
20 pos,20 neg

With Complements of

Hisham Tharwat
Application Supervisor
BECKMAN COULTER
RIYADH -KSA