Average of Normals

Graham RD Jones
Graham RD Jones
Chemical Pathology, SydPath, St Vincent’s 
Hospital S dne A stralia
Hospital, Sydney, Australia.
 QC Overview
• Setting of quality standards 
g q y
• Selection of materials 
• Selection of concentrations 
• Setting (and re‐setting)
Setting (and re setting) of QC targets 
of QC targets
• Setting (and re‐setting) of QC limits 
• Selection of rules
• Frequency of running QC
Frequency of running QC 
• Response to out of range results 
• Other supporting QC activities (eg AoN) 
 Background ‐ Clinical
• The results from many patients tested at your
laboratory form a population.
• This population tends to be stable over time.
time
• This stability can be used for assessing assay
performance.
 Midpoint
• The
The midpoint is the most stable part of a 
midpoint is the most stable part of a
distribution
• Can be measured as:
b d
– Mean
– Median
– Mode
– Other 
( Bhattacharya Centre)
(eg Bh tt h C t )
• Other Centiles (5th, 20th, 99th) more variable
 Variables
Stability of data can often be improved:
y p
• Excluding outliers (especially for average)
• Limiting age ranges
– egg 18 – 50 for serum creatinine
• Limiting sources
– Eg
E GPs, all outpatients, exclude some clinics
GP ll t ti t l d li i
• Other: one result per patient; only patients 
with one result
• Depends on available demographics
Depends on available demographics
Creatinine ‐ Restrictions
 How can this data be used?

• Examples from SydPath (mostly)
 Monitoring Assay Over Time

Ad‐hoc query for specific concern

Report from Siemens Immulite 2000 users that 
IGF 1
IGF‐1 assay was drifting upwards
d if i d
 +50%

August
2010
2005 2007 2009 2011

IGF‐1 – Immulite 2000. QEII, Pathwest WA.

Monthly median of patient results. 2004 – July 2012
(with thanks to Dr Ee‐Mun Lim)
 Percent flagged Low

August 2010

Percent flagged High

IGF‐1 – Immulite 2000. Monthly Flagging Rates.
(with thanks to Dr Ee‐Mun Lim)
SydPath: IGF‐1 Immulite 1000
 Recent Publication (Aug 2013!)

IGF‐1 Failures
Mayo Clinic & University of Virginia – July 2013
Mayo Clinic & University of Virginia  July 2013
(received Feb 2013)
IGF‐1

Percent flagged Low

Percent flagged High
 Monitoring Over Time
• Planned quality activity
• eg review of In‐house IVD
• Serum Serotonin (annual)
Serum Serotonin (annual)
Assessing Reference Intervals
Spreadsheet Application
Application
• Standardised 
data entry
• List of result, sex, 
List of result sex
age, (date)
Application
• Additional 
data input
• Standardised
Standardised 
numerical 
output
Application
• Output graph
• Allows visual 
assessment
 Application
• Averages 
compared over 
g
age and sex
• Over time
Application

Instructions
Serum Bicarbonate
Application
• Additional 
data input
• Standardised
Standardised 
numerical 
output
 Validates against midpoint of reference interval
Sydpath data analysis for Common Reference Intervals
Test Sodium Potassium Chloride Bicarbonate Creatinine F Creatinine M
Harmonised Limits LRL 135 3.5 95 22 45 60
URL 145 5.2 110 32 90 110
Current Limits LRL 137 3.5 95 24 40 60
URL 146 5 110 31 90 120
Excluded data LexL 125 3 85 18 30 40
UexL 150 6 120 36 130 150
Age exclusions Age-L 18 18 18 18 18 18
Age H
Age-H 90 90 90 90 55 55
Sex exclusions Sex(M,F,B B B B B F M
Harmonised Limits Midpoint 140 4.35 102.5 27 67.5 85
Results Average 141 2
141.2 44
4.4 102 6
102.6 27 4
27.4 67 2
67.2 87 8
87.8
Median 141 4.4 103 27 66 87
%low 1.5% 0.5% 1.4% 1.2% 1.1% 1.3%
g
%high 1.7% 2.6% 0.2% 1.5% 2.8% 6.4%
2.5th 136 3.7 96 22 48 63
97.5th 145 5.3 108 32 92 121
n 9016 9101 9097 9084 2393 2078
start date 2/5/12 2/5/12 2/5/12 2/5/12 2/5/12 2/5/12
end date 20/9/12 22/9/12 22/9/12 22/9/12 22/9/12 22/9/12
Serum Calcium
Location effect
Location effect
Comparing with other laboratories

• Midpoint
Midpoint (and other levels) can be compared 
(and other levels) can be compared
with different laboratories

• Bias assessment which includes:
Bias assessment which includes:
– Population
– Pre‐analytical
– Analytical

• (AACB Harmonisation Project)
(AACB H i ti P j t)
Short Term Monitoring
 Patient Percentile Monitoring

• “Empower” Project
• Dietmar Stockl and Linda Thienpont
and Linda Thienpont
• Daily averages of common analytes
• Outpatients only
• Sent to Ghent via e‐mail
Sent to Ghent via e mail
Example Data – Total Protein
 But…

• Calcium v day of week: r2 = 0.62
GGT v day of week: r2=0.39
• GGT v day of week: r 0.39

• ALP v Age: r2 = 0.50

• Patient demographics change daily!
 Daily Data

Pitfalls:
• Day of the week
Day of the week
• Average age of patients
• Clinics
Weekly Data
Monthly Data
 Average of Normals
• Running average of “x” results
• Real‐time flagging
• Variables:
V i bl
– Number of samples
– Limits for flagging
– Exclude extreme results
Exclude extreme results
• Based on work of Brian Bull (Bull’s Algorithm)
• Can run in real time, added to standard QC
• Does it help?
Does it help?
 Average of Normals

Serum Creatinine – All sources, M+F, exclude >130, <40
20 pt and 75 pt moving averages
 Average of Normals

+20 umol/L shift

+20 umol/L shift
 Does it help?
• Performance depends on:
p
– Number of samples included in average
– Ratio of CVg
Ratio of CVg to CVa
to CVa
(if analytical variation small compared with 
population variation need large numbers)
population variation, need large numbers)
• Trade off:
– Smaller numbers  finds error more quickly
Large numbers  finds smaller error
– Large numbers  finds smaller error
• Need to work out power of error detection
Cembrowski
 AVERAGE OF DELTA – A NEW 
CONCEPT IN QUALITY CONTROL

GRD Jones
Department of Chemical Pathology, 
St Vincents Hospital Sydney Australia
St Vincents Hospital, Sydney, Australia
APCCB 2004 42
 Background
• The Average of Normals (AON) is an accepted QC 
process for clinical laboratories
process for clinical laboratories.
• AON is the average of a set number of patient results, 
usually within set limits (eg normal range).
usually within set limits (eg normal range)
• The AON rule “fires” when the function exceeds a pre‐
set limit (eg
set limit (eg 2.5 x analytical CV).
.5 x analytical CV).
• Delta checks are the comparison of a result with a 
p
previous result from the same patient.
p
• Delta checks are used to detect blunders or other 
errors
• I combine these concepts to produce the Average of 
Delta (AOD) a new QC tool for clinical laboratories.
APCCB 2004 43
 0.9
0.7 AOD(2)
AOD 2
Figure 2.
0.5
0.3 • AOD functions for
0.1
-0.1
NAOD of 2, 10 and 50.
-0.3
-0.5
14 36 • CVa = 0.1, CVwii = 0.2
1 11 21 31 41 51 61 • A fixed bias of 0.3 was

00.99 p 10.
introduced at sample
0.7
0.5 AOD
AOD(10)
10 • The red lines show the
0.3
0.1 2.5 x SDAOD. For AOD2
-0.1
-0.3
10 20 the value is 0.56; for
-0.5
1 11 21 31 41 51 61
AOD10, 0.24; for AOD
50, 0.11.
0.9
0.7 AOD(50) • Five example AOD
0.5 AOD 50
0.3 f ti
functions are shown
h iin
0.1
-0.1
each graph ( ).
18 30
-0 3
-0.3
-0.5
• Blue arrows show N90.
1 11 21 31 41 51 61 • Orange arrows show
APCCB 2004 44
Sample Number
average first rule firing.
 Discussion
• The limitation of AON is the ratio of group biological 
variation to analytical CV.
to analytical CV.
• AOD may outperform AON if:
– CVwi is small compared to between‐person biological 
p p g
variation (a low Index of Individuality).
– The frequency of samples with previous results is high.
q y p p g
– Clinics or weekends affect AON results. Note than AOD 
should not be affected by change in patient mix as it uses 
patients as their own control.
• AON may complement standard QC if it can:
– Detect smaller errors than standard QC
– Detect errors before standard QC
– Allow less frequent use of standard QC
APCCB 2004 45
 Same ideas…
• Alan Remaly – NIH
• Same concept, Poster 2005 AACC
Whew….
 Average of Normals

• Can be run in batches or in real time
C b i b h i l i
• Provides independent information
• Can provide a warning between standard QC runs
• “free”
free  in terms of QC material and assays
in terms of QC material and assays
• Not free in terms of lab time
– Setting up
Setting up
– Responding to non‐significant flags
• Need to validate benefits to lab
N dt lid t b fit t l b
• Need appropriate IT processes
 Survey?

Who performs:
• Real
Real‐time
time Average of Normals
Average of Normals (or Similar)
(or Similar)
• Regular review Average of Normals
• As needed Average of Normals
• Have not done Average of Normals
Have not done Average of Normals

• Other non‐standard QC practices