8/14/2020

`
External Quality Assurance /Assessment

Stanley Medical College

AMBI TN PG Crash Course
22nd January 2020
Dr V.K.Ramadesikan
Thanks to Dr Saravanan & AMBI TN PGCC team
for giving me this teaching opportunity

What will we learn in this module ?

1. What is EQA & What is PT ?

2. Why EQA ?
3. What does it identify ?
4. How to handle EQA samples ?
5. How to read EQA reports & identify problems ?
6. How to do Root Cause Analysis (RCA) of outliers in
EQA samples
7. How to take CA & PA ?

1
 8/14/2020

Purpose of Internal & External QC

Internal QC
• For CONTINUOUS & IMMEDIATE i.e DAILY
monitoring of results
• Whether the patient results are reliable enough to be released
to physicians (WHO, 1981)
• A measure of Repeatability or Reproducibility of the systems
& methods in use in the lab. - Measures Analytical
Imprecision &
• Measures Random Error
• Does not detect the accuracy of patient results over a ‘ longer
term ’

EQA or PT is External Quality

Control
• External Quality Assessment or Assurance /
Proficiency Testing
• is a program used to evaluate a laboratory’s
measurement procedure performance by comparing its
results with those of other laboratories for the same set
of sample(s)
• Whether the lab’s measurement procedure has a bias
from a true value.

2
 8/14/2020

What is EQA or PT ?
• Each laboratory measures the EQA/ PT samples in singlicate or
duplicate as if they were patient samples and submits results to
the results to the EQA/PT provider for evaluation.

• Provider assigns or obtains a target value for the EQA/ PT

samples and determines if the results of a laboratory are in close
enough agreement with the target value to be consistent with
acceptable measurement procedure performance.
• Instrument Mean
• Method Mean
• All Methods Mean

What does EQA or PT assess ?

External Quality Assessment or
Proficiency Testing is
PERIODIC AND RETROSPECTIVE
monitoring of lab results by an independent
external agency
• Assessment of ACCURACY ( or inaccuracy) in
the lab measurement procedure / methods
• A measure of SYSTEMATIC ERROR &
RANDOM ERROR (BIAS & IMPRECISON)
• Total Error in Measurement Procedure

3
 8/14/2020

Types of EQA Schemes

• Proficiency Testing (PT) – Involves interpretation of
results in addition to testing
• Sanctions are linked to inadequate performances. For
example the laboratory may be required to repeat the
testing or even be disaccredited from testing.
• •“Traditional” EQA
• No interpretation, Only testing.
• No sanctions for poor performance. Performances
may be scored but there are no consequences for
poor performance to the laboratory. Most EQA will
fall into this category.
• •“Educational” EQA
• Emphasis is on quality improvement. The scheme may
include long-term follow up and assessment of quality
improvement.

EQA schemes available for Indian labs

1. ACBI - CMC External Quality Assessment Scheme – Clinical
Biochemistry, Hormones tumor Markers, HbA1c
2. Bio-Rad EQAS
3. RIQAS – Randox International Quality Assurance Scheme
4. RCPA QAP - Royal College of Pathology Australasia Quality
Assurance Programme
5. CAP PT Programme
6. NCG EQAS - MPQAP –Tata Memorial Centre, Mumbai
7. EQAS Diagnostic Cytopathology –Tata Memorial Hospital,
Mumbai
8. ISHTM AIIMS –External Quality Assurance Programme New
Delhi
9. ISHTM-CMC EQAS haematology – for molecular genetics,
transfusion medicine & haemostasis
10. RML Quality Assurance Program, Lucknow
11. IAMM – Sankara Nethralaya Chennai – Microbiology & Serology

4
 8/14/2020

Why EQA is necessary ?

1. Measures the inaccuracy of lab’s results and calibration
stability of methods – Over a longer period of time in terms of
years
2. Identifies asymptomatic but potential problems related to
imprecision, human error & systematic error
3. Mandatory requirement of medical lab accreditation – for
both applicant & accredited labs - Clause 5.6.3
• Participation in EQA / PT / Alternative assessment a must for all
tests –
• Non participation in EQA or PT / repeated failures in an EQA or
PT Programme may result in not granting of / temporary
suspension or cancellation of accreditation for those non EQA
tests
4. Benchmarks the lab’s performance against others

Is IQC not enough ?

• SUPPLEMENTS Internal Quality Control

• NEVER a SUBSTITUTE for Internal QC

• Both measure 2 different aspects of quality of test

results –
• IQC measures IMPRECISION - Random Error
• EQC measures INACCURACY – Systematic Error

5
 8/14/2020

WHAT DOES EQA IDENTIFY ?

• Identifies following errors – An Early Warnings System of

potential problems
1. Reagent problems – reconstitution, storage, onboard stability,
lot to lot / pack to pack variations
2. Water Quality problems – contamination & high conductivity
3. Analyte calibration stability
4. Equipment performance – light source, tubing, pressure pumps,
temp maintenance
5. Environmental conditions
6. Technologist knowledge & adherence to Lab procedures
• Indicator of where to direct improvement efforts
• Identifies training needs of lab personnel

1. HOW SHOULD EQA SAMPLES BE

HANDLED ?

2. SHOULD THEY BE HANDLED WITH

EXTRA PRECAUTION ?

6
 8/14/2020

HOW TO HANDLE EQA SAMPLES

• EQA / PT samples should be handled, stored & tested
exactly as you would treat routine patient samples
• Follow the usual procedures used in your laboratory.
• Handle high, out-of range samples in accordance with
established laboratory procedures when a sample result
exceeds the linear range of the method.
• Technologists who would normally process patient
samples should test EQA samples
• Use the usual instrument & reagents used for testing
patient samples
• Use the same criteria used for a patient result to determine if
the EQA Program sample result is acceptable for reporting
• Only then the objective assessment of performance of lab
can be made so that fixing the problem becomes easy

NEVER DO THESE …
• Ask a specific tech to run the EQA / PT sample

• Process EQA sample using a new reagent unless scheduled

• Process PT samples using backup / secondary instrument

• Run EQA /PT sample immediately after running Internal QC

sample Always introduce EQA samples in between patient samples

• Do not do calibration on the day of reporting EQA / PT sample

unless it is a scheduled /required calibration (as per policy of lab)

• Repeat the EQA samples and give the mean of multiple values or
the best value
• Discuss EQA results with others &
• Use PT samples for other purposes before submission date

7
 8/14/2020

Reconstitution of EQA material

• Accurate reconstitution of freeze-dried material is critical

• “ Reconstitute exactly as per EQA / PT provider’s

instructions”

• Preferable to use single dispensing volume pipette, same

person reconstitutes EQA material

• Process immediately after reconstitution on the same day

after allowing for lyophilized material to dissolve and
homogenize as per EQA provider instructions

• Aliquots can be made & stored as per instructions

Commutable EQA material is

preferable
• Each laboratory measures the EQA/ PT samples in singlicate or
duplicate
• The EQA/ PT provider assigns or obtains a target value / mean
/ Median Value for the EQA/PT samples – Robust statistics as
per ISO 13528
• Outliers are not eliminated but compressed or brought in
• Range established based on group SD / Allowable % Deviation
(DCV)
• Grouped based on Instrument & reagent / Method / all
methods
• & determines if the results for an individual laboratory are in
close enough agreement with the target mean / value of
appropriate group

8
 8/14/2020

INTERPRETATION OF EQA REPORTS

Commonly used terms
1.SDI –Standard Deviation Index (or)
2.Z -Score – Classical & Robust
3.Bias % / % Deviation
4.VIS – Variance Index Score

SDI - STANDARD DEVIATION INDEX

• A Statistical tool –
• A score given to the lab by the EQAS / PT provider
on the performance of the lab for each analyte in a
EQAS cycle

• Sometimes called Z-Score

• A measure of relative inaccuracy among group

participants

9
 8/14/2020

What is normal or Gaussian distribution ?

Out of 100 results

• 68.2% of values fall
within ± 1SD

• 95.5% of values fall

within ± 2SD

• 99.7% of values fall

within ± 3SD

Interpretation of SDI & VIS

1. In all the three cycles, VIS is the same since VIS is not
related to SD; it is only related to DCV, which is constant
2. On the other hand, SDI is more than double in cycle # 2 as
compared to cycle # 1, and SDI is double in cycle # 3 as
compared to cycle # 2, although the lab value remains the same
in all the three cycles
3. It is clear that SDI is related to SD
4.Therefore, while interpreting SDI values, the laboratory
personnel should take into account participants’ SD value
5. SD indicates variation in participants values & its distribution
– greater the SD wider is variation of participants – not good
Courtesy Dr A.S.Kanagasabapathy

10
 8/14/2020

Table – 1

• EQA (Target) Mean 40 90 160 350

• Lab value 50 100 170 360
• Diff 10 10 10 10

Table – 2

• EQA (Target) Mean 40 90 160 350

• Lab value 35 75 120 250
• Diff 5 15 40 100

• Data of which Table indicates Constant Systematic Error?

• Data of which Table indicates Proportional Systematic
Error?
Courtesy Dr A.S.Kanagasabapathy

WHAT IS SDI ?
IS KNOWING SDI USEFUL ?
• “ The number of Standard Deviations your lab’s value lies
from the Peer Group Mean” – At what SD of peer group is
your lab value ?

• Since 95 % of all results in a normal population fall within ± 2

SD of the mean, ± 2 SD is considered an acceptable laboratory
value

• SD is always expressed in the units being measured, whereas SDI

has no units, being a ratio
• Simply said
‘Difference between your result and group mean in terms of
the number of standard deviations of the overall mean’

11
 8/14/2020

HOW SDI IS CALCULATED ?

YOUR LAB’S RESULT – PEER GROUP MEAN

_______________________________
SD OF THE PEER GROUP
SDI
< 0.5 - Excellent
0.5- 1.0 - Very good
1.0 -1.5 - Good
1.5 -2. 0 - Satisfactory
> 2.0 - Not satisfactory
3.0 - Poor
0 – ideal No difference between your lab’s value & peer
group mean

IS KNOWING SDI USEFUL ?

• Any SDI of 2.0 or greater in a EQAS cycle for any analyte
deserves special concern – indicates some for of systematic error

• Any test whose average SDI is 1.0 – 2.0 deserves some special
attention because your method shows a systematic difference from
the group. – potential to lead to unacceptable results in future

• SDI of < 1.0 – your performance is satisfactory – your result is

with 1 SD of the group – you are with in the 68.7 % of lab’s result
whose values are close to mean

12
 8/14/2020

IS KNOWING SDI USEFUL ?

if you observe SDIs such as -0.4, -0.2,
-0.5 - 0.5, - 0.5 and -1.0 (all negative)
for an analyte in successive cycles,
your method is generally running on
the low side and is negatively biased,
on average, by - 0.6 SDI
You are reporting precise pateint
values but lower than the true value
by 0.6 SD
So better
calibrate your instrument and analyte
or requires instrument maintenance

Calculation of VIS
Calculated from % bias / % variation of lab result from DV

Designated Value [DV] = 75.8 mg %

Participant's result = 71.8 mg%

% Bias % Variation [%V]=Participant's Result - Designated value

--------------------------------------- X 100
Designated value

75.8 – 71.8 X 100 = 4 X 100

120 75.8

= 5.3
Variance Index = %V X 100 = 5.3 X 100 = 76
DCV 7.0
VIS = 76

13
 8/14/2020

Desired CV of 25 common Analytes for Indian Labs - previously

called Chosen CV 0r CCV set by WHO for Indian labs

Glucose 7% Sodium 3%
Urea 10 % Potassium 4%
Creatinine 9% Chloride 5%
CK 7.3 AST 12.5 %
Bilirubin Total 15% ALT 15 %
Protein Total 6% ALP 15 %
Albumin 6% Amylase 15 %
Calcium 6% Magnesium 7.5 %
Uric acid 7.7 % Phosphorus 7.8 %
Cholesterol 7.5 % Bicarbonate 9.0 %
TGL 12% HDL- C 8%
HDL 7.6 Iron 15 %
CK 7.3%

VCRM –Value Corrected for

Reference Method
• Lyophilized samples prepared at CMC Vellore are sent for analysis by
International reference methods, to the WEQA Reference Laboratory
at Cardiff & Vale University Health Board UK, a certified lab for
certain parameters. Samples analysed by LC/GC-MS or AAS AES at
reference Lab
• The values provided by the Reference Labs are used for assigning the
VCRM.
• The same lot samples(those sent to the Reference Laboratory) are
analysed every month at CMC Vellore along with the month’s EQA
samples. The values of present month EQA sample values are then
corrected to the reference method values. This value VCRM -
Value Corrected to Reference Method.
• Values traceable to reference method are available at present for
glucose, creatinine, cholesterol, triglyceride sodium potassium.
• Participants can compare their lab results to their DV and also to the
value corrected to the internationally accepted reference method.

14
 8/14/2020

Bias % :
Inaccuracy in lab’s result
for the analyte expressed in
as a percentage
& calculated using DV

15
 8/14/2020

BIAS %
75.8-71.8 x 100 = 5.3%
75.8

% bias (% inaccuracy)
• Percentage measure of inaccuracy in your lab’s results
compared to the method group or DV
• Lower the % bias more accurate is your lab method
• Designated Value - 75.8
• Your Lab’s Value - 71.8
• % Bias - 100 x 4 = 5.3%
75.8

16
 8/14/2020

Interpretation of VIS

VIS Performance
< 50 Excellent
50 - 100 Very good
100 -150 Good
150 -200 Satisfactory room for improvement
200 Not acceptable
 400 & more Stop testing and set right the problem
before starting again

• If VIS of >200 on two or more occasions for the same analyte,

them check your standardization procedures & calibration

• Indicates an accuracy problem (systematic error / bias )

Why Analysis of EQA reports is important ?

• 1. True benefit of EQA /PT proficiency testing, lies in

carefully & critically evaluating your results and report
• 2. Proper analysis of EQA testing results can reveal
problems even before failure in EQA or even an adverse
patient result -
• 3. Potential problems can be recognized by recognizing
patterns from SDI or Z score graphs - trends & shifts
• 4. Review both the present cycle results as well as
performance from previous cycle for the same analyte &
results of other analytes in the same cycle
• 5. Observe Charts of Z score or SDI or VIS or %
deviation for trends - Otherwise trends will be missed

17
 8/14/2020

How to recognise problems ?

• Recognise whether due to Systematic Error or Random
Error
• Results are consistently be different from the target /
peer group mean - All results on one side of the mean
(may be close to mean or at variable distances from
mean) – Systematic Error
• Majority of results are close to the target value but
some show larger deviation s on one side or both side of
mean – Random Error
• Corrective action differs

How to recognise problems ?

• Random error is an error / mistake / inaccuracy that has no set
pattern. Its occurrence cannot be predicted

• Detected by sudden, undue % deviation / SDI / Z-Score – > 3.5

• Indicates lab’s imprecision / poor reproducibility

• Easily identified – values are far beyond the usual – e.g SDI from
1.5 in earlier cycles to an SDI of 4.0 to 6.0

18
 8/14/2020

How to recognise problems ?

• Systematic error – set pattern of error / mistake. Its occurrence
can be explained and corrected

• Constant difference between the participant lab’s value and

group mean All results lie on one side of peer group mean

• Indicates lab’s bias or inaccuracy but good precision

• A progressive increase / decrease in deviation on the same side –

‘ shift’ or stabilizes after a gradual increase ‘trends’

Sources of Random Errors

• Random errors are blunders

• Transcription errors – result not correctly transcribed from the

instrument to workbook or PT sheet or from the Workbook to the
system

• Misplaced decimal points e.g serum potassium 58.2 instead of 5.82

• Result was entered in the wrong instrument or method group on the

result form of PT provider.

• Lab might have changed the method or instrument recently but not
updated with PT provider

19
 8/14/2020

Sources of Random Errors

• Mislabeling errors, interchanging results of PT specimens,
misplacing specimens in analyser rack, inappropriate reagents and
standards

• Result of some other analyte entered in the PT form of the provider

in the system

• Wrong units. Result in one unit in the instrument but not converted
to unit of EQA provider e.g ug /mL instead of ng/L , mg/dL instead
of g/L

• Result found on evaluation report from PT provider not matching

with the result on the report form - mistake of PT provider

• Calculation errors (dilution factor correction error, conversion from

one unit to another)

Sources of Systematic Errors

• Improper reconstitution of QC materials, water quality, not
following manufacturer’s / PT provider’s instructions while
reconstituting

•Recalibration if not done earlier – esp if a new lot of reagent has

been used or if the open vial stability of the reagent is doubtful

• Instrument maintenance – major part needs servicing or

replacement (optics, alignment, incubation temperature failure,
Internal QC values having a bias

• Recent instrument malfunction, instrument failure soon after PT

specimen testing

• Assay settings (sample volume, reagent volume, delay time

incubation time no of readings to be taken etc. )
• improper reconstitution of reagents, not following manufacturer’s
instructions while reconstituting, storing or handling reagents

20
 8/14/2020

How to investigate unacceptable EQA

results
• Stepwise & systematic investigation

• 1st step - Gather & Review Data

• 2nd Step Classify the problem – RE or SE

• 3rd Step Evaluate patient results

• 4th Step – Take suitable CA & PA

How to investigate unacceptable EQA

results
• Gather & Review Data
 Review all documents (IQC results, machine maintenance
& breakdown log etc), machine printouts, worksheets

 Discuss with technical staff who processed, analyzed

sample & transcribed the results

 Check for clerical errors

 Check QC, calibration status instrument function logs

 Repeat analysis if available

 Review historical data for the analyte – previous cycles

21
 8/14/2020

Investigating unacceptable EQA results

Clerical Errors
 Misplaced decimal point,
 Incorrect units
 Incorrect Conversion factor
 Dilution factor
 Incorrect Instrument or Incorrect method
 Result not correctly transcribed from instrument or
reading
Method Problems
 Incorrect instrument calibration
 Expired analyte calibration
 Instrument startup function checks or maintenance not
performed
 Improper reconstitution of reagents, calibrators, QC
material, used beyond expiration dates, new reagent lots
 Data processing problem with instrument

Investigating unacceptable EQA results

Method Problems
 Carry over from previous specimen
 Incorrect incubation times
 Incorrect temperature settings / other parameter settings
 Water Quality problems
 Close to LoD or LoQ or beyond linearity of instrument
Technical Problems
 Improper reconstitution of PT material
 Delayed testing after reconstitution
 PT sample not placed in correct order/ Secondary tubes
improperly labeled
 QC data – unacceptable /showing a trend still results released
 Instrument Calculation error
 Non homogenous PT material – during lyophilisation

22
 8/14/2020

PT evaluation Problems
 Comparison with Inapt. Peer group
 Very narrow range due to small SD of peer mean
 Incorrect data entry by PT provider
No explanation
 Random error – single / occasional PT is unacceptable esp
when repeat analysis is OK

23