DOI 10.
1515/cclm-2013-1090      Clin Chem Lab Med 2014; 52(7): 973–980
Koen Hens, Mario Berth, Dave Armbruster* and Sten Westgard
Sigma metrics used to assess analytical quality
of clinical chemistry assays: importance of the
allowable total error (TEa) target
Abstract
Background: Six Sigma metrics were used to assess the
analytical quality of automated clinical chemistry and
immunoassay tests in a large Belgian clinical laboratory
and to explore the importance of the source used for esti-
mation of the allowable total error. Clinical laboratories
are continually challenged to maintain analytical quality.
However, it is difficult to measure assay quality objectively
and quantitatively.
Methods: The Sigma metric is a single number that esti-
mates quality based on the traditional parameters used in
the clinical laboratory: allowable total error (TEa), preci-
sion and bias. In this study, Sigma metrics were calculated
for 41 clinical chemistry assays for serum and urine on five
ARCHITECT c16000 chemistry analyzers. Controls at two
analyte concentrations were tested and Sigma metrics
were calculated using three different TEa targets (Ricos
biological variability, CLIA, and RiliBÄK).
Results: Sigma metrics varied with analyte concentration,
the TEa target, and between/among analyzers. Sigma
values identified those assays that are analytically robust
and require minimal quality control rules and those that
exhibit more variability and require more complex rules.
The analyzer to analyzer variability was assessed on the
basis of Sigma metrics.
Conclusions: Six Sigma is a more efficient way to control
quality, but the lack of TEa targets for many analytes and
the sometimes inconsistent TEa targets from different
sources are important variables for the interpretation and
the application of Sigma metrics in a routine clinical labo-
ratory. Sigma metrics are a valuable means of comparing
the analytical quality of two or more analyzers to ensure
the comparability of patient test results.
Keywords: allowable total error (TEa); quality; Sigma
metrics.
*Corresponding author: Dave Armbruster, Abbott Diagnostics
Division, Department 09AA, Building CP1-1S, Abbott Park, IL 60064,
USA, E-mail: David.Armbruster@abbott.com
Brought to you by | Universiteit Gent
Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
Koen Hens and Mario Berth: Algemeen Medisch Laboratorium
(AML), Antwerpen, Belgium
Sten Westgard: Westgard QC, Madison, WI, USA
Introduction
Analytical quality is a prerequisite for the clinical labora-
tory, but it can be difficult to assess it. Six Sigma metrics,
hereafter referred to as Sigma metrics, have been used to
measure quality in an objective and quantitative manner,
combining the three traditional elements used to evaluate
assay performance: the allowable total error (TEa), bias
and precision. Both IVD manufacturers and clinical labo-
ratories have implemented the Sigma metrics approach,
leading to reduced operational defects in the laboratory
and/or quantitation of test performance quality [1–4]. A
higher Sigma metric value means fewer analytical errors
and fewer questionable test results are accepted and
reported, and fewer acceptable test results are falsely
rejected and not reported [5]. A Six Sigma assay is one for
which 99.99966% of results are error free, corresponding
to 3.4 defects per million opportunities, in this case, assay
results. Sigma metrics can be translated into appropriate
and robust quality control rules thus improving overall
laboratory quality. As a result, the laboratory can maxi-
mize efficiency and reduce control costs through fewer
re-runs and work-arounds.
The objective of this study was to evaluate the perfor-
mance of 41 chemistry assays on five Architect c16000®
instruments in terms of Sigma metrics.
Materials and methods
Analyzers
Five ARCHITECT c16000 clinical chemistry analyzers (Abbott Diag-
nostics, Chicago, IL, USA) were used. The c16000 is a general pur-
pose, high volume instrument that uses spectrophotometry for a
variety of absorbance/colorimetric and immunoturbidimetric assays
and ion-specific electrodes (ISE) for electrolytes (Na, K, Cl). Three
974      Hens et al.: Sigma metrics

of the c16000s were used to test routine clinical chemistry analytes are based on replicate analysis performed by the manufacturer and/
and selected high volume-specific protein analytes. The other two or independent laboratories using Architect analyzers and reagents.
c16000s were used for low volume assays and special tests. The sys-
tems were maintained and used as per the manufacturer’s operations
manual. Testing was conducted in February, 2010.
Allowable total error
Tolerance limits, in the laboratory, are best expressed as an allow-
Assays able total error (TEa) specification. TEa is a model that combines both
imprecision and bias (trueness) of a method to calculate the impact
Only assays for which TEa goals from at least two independent
on a test result. The Sigma metrics were calculated using TEa goals
sources were available were included in this study. Thirty-three serum
from three sources in order to understand the effect of TEa on esti-
clinical chemistry tests were included: albumin BCG (bromocresol
mates of Sigma metrics: 1) CLIA (Clinical Laboratory Improvements
green method), alkaline phosphatase, alanine aminotransferase
Amendments) from the US; 2) RiliBÄK (German Medical Council for
(ALT, without pyridoxal-5-phosphate), amylase, aspartate ami-
the Quality Assessment of Quantitative Analyses in Medical Labora-
notransferase (AST, without pyridoxal-5-phosphate), bicarbonate,
tories, 2008 version; the inter-lab or “Ring Trials” values, in contrast
direct bilirubin, total bilirubin, calcium (Ca), chloride (Cl), choles-
to the intra-lab values); and 3) the Ricos biological variability data-
terol, creatinine (alkaline picrate Jaffé method), creatine kinase (CK),
base (desirable target values, in contrast to the minimal or optimal
CRP (Standard range and Vario 32), γ glutamyltransferase (GGT), glu-
target values) [7, 8]. These three sources are regularly updated and
cose, high density lipoprotein (direct HDL), immunoglobulin A (IgA),
can be freely accessed through http://www.westgard.com.
immunoglobulin G (IgG), immunoglobulin M (IgM), iron, lactic acid,
lactate dehydrogenase (LDH, IFCC reference method formulation),
lipase, magnesium (Mg), phosphorus, potassium (K), total protein,
sodium (Na), transferrin, triglycerides, urea and uric acid. Two thera-
peutic drugs in serum (carbamazepine and lithium) and six specific
Sigma metric calculation
urine assays were included: Ca, creatinine, phosphorus, protein, Sigma metrics were calculated using the standard equation:
urea, and uric acid. All reagents were obtained from Abbott and used
as per the manufacturer’s package inserts. Sigma metric = (TEa–bias)/precision [all values expressed as
percent (%)].

Three sets of Sigma metrics were calculated, each using different

Controls
Bio-Rad (Bio-Rad, Marnes-la-Coquette, France) controls used were:
1) Lyphochek Assayed Chemistry Control material (two levels, lyo-
philized); 2) Lyphochek Immunology Plus (two levels, lyophilized);
and 3) Liquichek Urine Chemistry Control (ready to use, stored at
2–8 °C, non-frozen). The manufacturer’s package inserts were fol-
lowed. Controls were mixed thoroughly prior to use. Open vial con-
trols were handled as per the manufacturer’s directions and repeat
freeze/thaw cycles were avoided. Controls were tested within 30 min
of being placed on the analyzers. Calibration curves were verified as
valid when concentrations of the two Bio-Rad Lyphochek Assayed
Chemistry Control levels were within the listed validity ranges. All
runs subsequent to the initial calibration included two control levels
to verify acceptability of the analysis. Data were not accepted if a con-
trol was out of the specified acceptance range.
Precision
Precision, expressed as coefficient of variation (% CV), was deter-
mined for all assays using a 20-day protocol performed according
to CLSI (formerly NCCLS) guideline EP5-A2 [6]. The controls for each
assay were tested twice a day, with a minimum of 2 h separating the
analytical events. One reagent lot was used for each assay.
Bias
Bias was estimated based on the difference of the average of observed
results for each analyte from the target values provided in the Bio-
Rad control package inserts. Target values for the Bio-Rad controls
TEa targets (providing that more than one set of TEa targets were
available for an analyte) and Sigma metrics were calculated for
both of the two control concentrations. For the more common (high
volume) analytes, three Sigma metrics were calculated for each
control concentration, one for each of the three Architect c16000
analyzers testing that analyte. For the less common (low volume)
analytes, two Sigma metrics were calculated for each control con-
centration, one for each of the two Architect c16000 analyzers test-
ing that analyte.
Results
Complete Sigma metrics for 41 assays are shown in
Table 1. Using RiliBÄK TEa targets, in general the Sigma
values are very acceptable and indicate robust, high
quality results, ranging from about 5 or 6 up to a high
value of about 30. The Sigma for Control 1 for Ca ranged
only from 0 to 1 due to the high bias, about 10% for a
target value of 2.02 mmol/L, but the Sigma values for
Control 2 ranged from 8 to 13. Lithium Sigma values
for both controls ranged from 1.7 to 3.1 due to very high
bias values. Control 2 creatinine Sigma metrics ranged
from only 0.2 to 0.9 due to very high biases at a target
value of 5.63 mg/dL. Potassium Control 2 (target value
5.67  mmol/L) Sigma values were only 2.6–2.9. By and
large, the Sigma metrics between two or among three

Brought to you by | Universiteit Gent

Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
 Table 1 Sigma metrics using three different sources for TEa target values (RiliBÄK, biological variability, CLIA).

Assay   Control 1   Control 2   Control 1 Sigma   Control 2 Sigma   Control 1 Sigma   Control 2 Sigma   Control 1 Sigma CLIA   Control 2 Sigma CLIA
target value target value RiliBÄK RiliBÄK biological variability biological variability
a a a
Albumin   39 g/L   25 g/L   27/27.5/19.5   31.7/32/32   //   1.3/a/a   8.5/9.0/6.2   12.8/12/12
Alk. Phosph.   115 U/L   470 U/L   3.9/4.2/5.3   9.2/12/12.9   1.7/1.9/2.6   4.8/6.5/7.1   6.2/6.4/7.9   13.3/17.4/18.5
ALT   34.6 U/L   101 U/L   7.8/7.8/7.4   8.1/11.1/10.8   12.2/12.3/11.2   13.0/21.6/17.6   7.4/7.4/7.0   7.6/13.0/10.2
Amylase   66.1 U/L   411 U/L       5.0/6.8/5.5   9.1/9.8/7.9   11.2/15.3/11.8   19.5/21.3/21.2
AST   35.9 U/L   190 U/L   8.8/9.6/9.5   14.4/19.8/19.9   6.2/6.6/6.8   7.6/4.9/13.0   8.4/9.1/9.1   13.5/18.7/18.7
a a a a a a a a a
Bicarbonate   34.4 mmol/L   16.3 mmol/L       //   //   1.1/1.7/a   //
Bilirubin, direct   0.436 mg/dL   1.93 mg/dL       15.0/21.4/22.2   25.2/31.2/28.9    
Bilirubin, total   0.838 mg/dL   4.29 mg/dL   9/7.7/6.4   8.8/9.7/10.3   12.9/10.9/9.1   12.9/14.2/15.1   8.2/7.0/5.8   7.8/8.7/9.3
a a a
Ca   2.02 mmol/L   3.03 mmol/L   0.3/0/1.0   12.3/13.2/8.1   //   0.6/0.9/0    
Ca (urine)   0.86 mmol/L   2.75 mmol/L   6.3/6   9.1/14   17.8/16.8   18.8/28.5    
Carbamazepine   3.47 mg/L   15.1 mg/L   6.1/3.8   6.94/5.4       8.3/5.4   8.9/7.1
Cholesterol   249 mg/dL   97.1 mg/dL   20.5/17.9/17.1   19/16.9/15.8   12.4/10.5/10.2   11.1/10.5/9.7   15.1/12.9/12.5   13.7/12.7/11.7
a a a
Cl   103 mmol/L   83.1 mmol/L   9.3/9.6/8   7/7.1/5.7   //   1.1/0.8/0.4   5.0/5.0/3.9   5.9/4.2/3.2
a a a a a a
Creat.   2.25 mg/dL   5.63 mg/dL   8.9/6.1/9   0.7/0.9/0.2   3.3/2.4/3.6   //   6.6/4.6/6.7   //
Creat. (urine)   83.4 mg/dL   243 mg/dL   9.9/6.3   7.6/5.2   18.0/8.9   10.9/7.5    
CK   148 U/L   502 U/L   12.4/15.6/15.4   17.6/8.4/14.6   19.8/24.7/23.9   26.8/38.9/36.1   19.6/24.5/23.7   26.6/38.5/35.8
CRP   8.3 mg/L   49.8 mg/L   10.4/9.9/10.2   7.7/10.1/12.6   37.7/38.2/31.5   25.2/34.2/41.4    
GGT   65.1 U/L   161 U/L   17.8/22.9/11.3   18.5/12.6/10.5   18.8/24.2/12.6   19.6/13.4/11.1    
Glucose   93.5 mg/dL   297 mg/dL   14.5/13.1/12.4/   10.5/13.8/14.2/   6.4/5.6/5.7/3.3/8.6   4.5/5.3/5.4/3.9/7.0   9.5/8.4/8.2/5/13   6.8/8.5/8.8/8.9/10.6
7.5/20.2 9.8/12.5
HDL   64.4 mg/dL   32.2 mg/dL       5.7/9.3/6.4   3.3/2.9/3.1   18.6/25.2/21.3   10.5/11.2/8.9
IgA   1.16 g/L   3.07 g/L   12.7/19.5   18.5/25.3   7.1/10.4   11.4/16.0    
IgG   8.18 g/L   24.5 g/L   17.9/16.2   11.3/6   6.9/6.8   4.9/1.3   25.6/22.8   15.8/9.3

Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
Brought to you by | Universiteit Gent
IgM   0.536 g/L   1.69 g/L   9.7/9.1   14.3/33.5   5.2/4.3   7.1/18.1    
Iron   302 µg/dL   56.4 µg/dL       29.5/29.5/28.2   23.9/23.7/35.7   18.8/18.8/18.1   8.4/15.2/13.6
Lactic acid   445 mg/L   N/A   4.2/6.2   N/A   8.8/13.4   N/A    
LDH   181 U/L   362 U/L   5.2/6.5/5.7   14.9/19.3/22.5   2.9/3.9/3.1   8.6/10.5/13.5   5.8/7.3/6.5   16.9/21.9/25.2
Lipase   29.4 U/L   69.3 U/L   12.7/10.2/8.2   8.4/7.6/9.5   14.8/12.3/9.7   10.3/9.4/11.8    
Lithium   0.69 mmol/L   2 mmol/L   2.7/1.7   1.7/3.1       5.8/4.1   4.6/6.1
Mg   0.8 mmol/L   1.88 mmol/L   9.8/7.2/5.8   7.8/7.9/6.7   2.9/2.1/1.7   2.5/2.4/2.1   16.7/12.2/16.1   13.1/13.2/11.3
Phosphorus   1.05 mmol/L   2.28 mmol/L   7.9/11.3/9.8   19/24/17.3   4.6/6.8/5.9   11.6/15.2/10.5    
Phosphorus   12.5 mmol/L   23.7 mmol/L   10.4/9.9   7.7/16.8   16.9/15.7   12.3/26.4    
(urine)
Potassium, K   3.77 mmol/L   5.67 mmol/L   6.2/6.5/3.9   2.6/2.7/2.9   1.9/3.9/2.4   0.9/1.2/1.4    
Protein   68 g/L   42 g/L   16.9/16.7/14.1   15/7/9.3   5.6/5.5/4.4   4.8/1.6/3.1   17.0/16.7/14.1   15.0/7.0/9.3
Protein (urine)   221 mg/L   648 mg/L   5.8/3.5   7.6/13.9   12.3/6.6   13.8/25.3    
a a a
Sodium, Na   146 mmol/L   125 mmol/L   4.9/7.6/7.9   3.2/11.1/6.1   0.1/1.0/0.7   //    
a a a a a a
Transferrin   2.49 g/L   1.65 g/L   6/9.3/9.3   10.5/14.7/2.6   //   //    
Triglycerides   185 mg/dL   89.6 mg/dL   19/17/24   5.9/13.6/15.8   33.3/30.1/42.5   10.6/25.5/29   29.8/26.9/38.0   9.5/22.6/25.7
Hens et al.: Sigma metrics      975
 976      Hens et al.: Sigma metrics

analyzers were comparable, although Control 2 Sigma

  Control 2 Sigma CLIA

4.5/4.9/5.4

24.5/21.6/18.9
values for Na showed ­considerable variability: 3.2, 6.1,
and 11.1. It is not surprising that the Sigma metric at one

Results are shown for three or five analyzers, at two different control levels. If no results are shown (blank field), no TEa is available. N/A Target value for control material not available.
control concentration may be very different from that at
another concentration because the precision and/or bias
can change considerably with analyte concentration.
Using biological variability (desirable) TEa targets,
 
 
 
 

most analytes again exhibited impressive Sigma metrics:

  Control 1 Sigma CLIA

0.9/1.3/1.5

17.4/13.6/13.9

5 or 6 and up to about 40. However, several analytes

produced low Sigma values: albumin, bicarbonate, Ca,
Mg, K, Na and transferrin (both controls); alkaline phos-
phatase, LDH, and HDL, total protein, and IgG (control 2).
It is notable that bicarbonate and the electrolytes exhib-
 
 
 
 

ited especially poor Sigma values, but the biological

biological variability
Control 2 Sigma

11.0/10.2/11.2
10.7/16.6
17.7/15.7/13.7
12.8/15.7

variability TEa goals are very demanding. Again, Sigma

metrics between two or among three analyzers were fairly
consistent.
Using CLIA TEa targets, once again Sigma metrics for
most assays were very impressive: 5 or 6 and up to about
35. Bicarbonate Sigma values ranged from negative to only
 

 
 
 
 

1.7, reflecting the bias for this difficult assay and the 8%
biological variability
Control 1 Sigma

5.1/4.7/6.5
18.3/19.0
12.3/9.5/9.8
7.5/10.1

TEa target. Surprisingly, urea Sigma metrics for control 1

ranged from 0.9 to 1.5 (target: 33.8 mg/dL) and for control
Bias exceeds the TEa target resulting in a negative value for the numerator in the Sigma metric equation.

2 from 4.5 to 5.4 (target: 97.9 mg/dL). The CLIA target for Cl
is 5% and Sigma values were 5, 5, and 3.9 (control 1) and
5.9, 4.2, and 3.2 (control 2). Creatinine Sigma values for
 

 
 
 
 

control 1 were respectable (6.6, 4.6, and 6.7) but negative

RiliBÄK
  Control 2 Sigma

15.3/13.5/15
11.2/12.6
  18.6/16.5/14.4
9.3/11.8

for control 2 because of the large bias at the higher con-

centration value. Sigma metrics between two or among
three analyzers were again consistent.
Although Table 1 is informative, displaying the same
data as normalized method decision charts, as pioneered
 
 

 
RiliBÄK

by Westgard, is more visually appealing and allows Sigma

Control 1 Sigma

7.8/6.9/9.7
13.7/14.5
12.9/10.1/10.3
5.7/7.8

metrics for a test at two analyte control concentrations

using three different TEa targets to be readily examined [9,
10]. Three graphs are included as examples, illustrating
especially the importance of the source of the TEa used
to estimate the Sigma metrics. Figure 1 displays Sigma
 

 
 
 
 

metrics for albumin. Using either the RiliBÄK or CLIA TEa

target value

97.9 mg/dL

9.14 mg/dL
24.8 mg/dL
760 mg/dL
Control 2

targets, assay quality is world class (Sigma metric   ≥  6) at

both low and high control concentrations, but using the
biological variability TEa, all Sigma metrics are unac-
ceptable. Figure 2 displays Sigma metrics for Cl. Using
 

 
 
 
 
target value

33.8 mg/dL

5.16 mg/dL
11.7 mg/dL
395 mg/dL

the RiliBÄK TEa, assay quality is excellent; using the

Control 1

CLIA TEa, assay quality ranges from marginal to good to

excellent; but using the biological variability TEa, assay
(Table 1 Continued)

quality is unacceptable. Figure 3 displays Sigma metrics

 

 
 
 
Uric acid (urine)  

for glucose. Using either the RiliBÄK or CLIA TEa targets,

Urea (urine)

assay quality is world class, and even using the biological

Uric acid

variability TEa, assay quality ranges from good to excel-

Assay

Urea

lent to world class.

a

Brought to you by | Universiteit Gent

Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM

Figure 1 Normalized method decision chart for albumin comparing
Sigma metrics calculated using biological variability ( ), CLIA ( )
and RiliBÄK ( ) TEa targets.
Data points represent the Sigma value for controls at two concentra-
tions and from three different analyzers.
Discussion
Calculating Sigma metrics offers the advantage of setting
a “quality base line”. Using the Sigma values, readily
calculated by, e.g., EZ Rules software (Westgard QC Inc.,
Madison, WI, USA), appropriate quality control (QC) rules
Figure 2 Normalized method decision chart for chloride comparing
Sigma metrics calculated using biological variability ( ), CLIA ( )
and RiliBÄK ( ) TEa targets.
Data points represent the Sigma value for controls at two concentra-
tions and from three different analyzers.
Brought to you by | Universiteit Gent
Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
Hens et al.: Sigma metrics      977
Figure 3 Normalized method decision chart for glucose comparing
Sigma metrics calculated using biological variability ( ), CLIA ( )
and RiliBÄK ( ) TEa targets.
Data points represent the Sigma value for controls at two concentra-
tions and from five different analyzers.
based on the inherent analytical quality of a test can be
formulated to ensure acceptable patient test results are
reported and false rejection of results is minimized. For
example, a Six Sigma assay can be controlled using a
simple 1:3s or 1:3.5s control rule, i.e., test results may
be accepted and reported if values for two controls fall
within 3–3.5 standard deviations (SD) of the mean value.
Assays with lesser Sigma values require more sophis-
ticated control rules and they can be chosen according
to the observed quality, being customized to match each
test. This is an improvement over the traditional but
simplistic “control ± 2 standard deviations” approach by
which all assays are assumed to be of equal quality and
the same rule applies to all analytes, despite the fact that
some assays are clearly more robust and some are typi-
cally more problematic. Gras et  al. reported calculating
Sigma metrics for 69 assays on the Roche Modular PPE
and Integra using TEa targets from biological variabil-
ity, bias assessed from peer group values, and precision
observed over 2 months [11]. A simple 4s 1:4s rule could be
used for assays with Sigma values  > 9 and a 1:3s rule for
assays with Sigma values  > 6. These authors found that
two thirds of assays could be controlled using control
rules based on biological variability and corresponding
Sigma metrics.
However, as clearly illustrated in this study, it must be
recognized that distinctly different Sigma metrics can be
obtained depending on the TEa values selected and using
the same bias and precision values in the Sigma equation.
978      Hens et al.: Sigma metrics
In this study, precision was based on a typical 20-day
study. An even more robust precision estimate can be
made from long-term QC data, e.g., over a 6-month or
1-year period, and it would be expected that this estimate
would result in a larger % CV, reflecting greater variability
over a longer time period, but also % CVs that are more
typical of long-term analytical performance variation.
Using a precision value calculated for a shorter period of
time could result in a more optimistic estimate and result
in a higher Sigma metric.
The Sigma metrics reported here reflect assay perfor-
mance at the time the data was collected and thus rep-
resent a “snapshot”. Naturally, performance can change
over time for a variety of reasons (e.g., reagent lot to lot
variation). Periodic calculation of Sigma metrics is appro-
priate to determine if assay quality has been maintained,
has decreased, or has improved. Sigma metrics thus rep-
resent another quality assurance tool to be monitored
­periodically to assess changes in assay quality.
Bias is more difficult to realistically estimate. In this
study, bias was estimated as the difference between the
target values (mean values) provided for the controls by
the control manufacturer and the observed mean values.
The control target values were not assigned by reference
method analysis and it’s uncertain how close they are to
“scientific truth” as determined by a gold standard method
as opposed to a routine “field method”. Also, the controls
used in this study may not be commutable, that is, their
analytical response with an assay may not be equivalent
to that of a fresh patient sample and observed bias may
be misleading, possibly being greater or smaller than
that observed with commutable materials and patient
specimens. Therefore the observed bias is only “relative”
instead of “absolute”. However, the reality is that labora-
tories routinely assess their bias by comparison to assayed
control target values and external quality assurance (EQA)
participant reports, which often use peer group or all par-
ticipant means instead of accuracy-based grading. Freid-
ecky and coworkers critically evaluated EQA acceptance
criteria and noted that EQA peer group evaluation was not
sufficient to determine analytical quality and may com-
promise desired patient care, although satisfying partici-
pant laboratories and manufacturers [12]. These authors
recommend that EQA results be based on comparison to
reference method target values, requiring metrological
traceability and assessment of absolute trueness instead
of relative bias (i.e., peer group comparison). The biases
used in our study reflect the “real world” conditions of
regular laboratory operations, even if they are not optimal.
A weakness of this study is the use of standard com-
mercially available controls. As noted in the previous
Brought to you by | Universiteit Gent
Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
paragraph, it is unknown whether these controls are com-
mutable. The metrological traceability of the controls is
also uncertain. The Bio-Rad materials used are typical
“precision” controls, as opposed to “trueness” controls,
and are intended to assess assay performance on the basis
of precision, i.e., % CV, and not accuracy, i.e., bias or true-
ness. Trueness controls, like calibrators, must be manu-
factured following a strict traceability chain, anchored by
established reference materials and/or reference method
[13]. The target values of trueness controls are established
by direct linkage to “gold standard” reference materials/
methods and accompanied by uncertainty estimates. This
is not the case with precision controls as is readily appar-
ent by examining the mean values listed in control package
inserts, which typically vary for an analyte depending on
the assay manufacturer. Realistic estimates of assay bias/
trueness require proper metrological standardization of
all field assays and analysis of trueness controls, of which
there are few, which may not be prepared using appropriate
reference materials, which often are not readily available,
and which may be prohibitively expensive for typical clini-
cal laboratories. In addition, the commutability of trueness
controls and reference materials must be established, and
commutability studies are challenging and rare.
Another weakness of this study is that the bias esti-
mates were made by comparing the observed results from
the analyzers to the target values listed in the control
manufacturer’s package insert. In retrospect, it would
have been preferable to estimate the bias as the difference
between the observed results and the Architect inter-lab-
oratory peer group means for each analyte. Bias estimates
based on the peer group means would have better reflected
the typical bias for the assays as performed in the testing
laboratory.
The choice of TEa is critical and has a major impact on
the Sigma metric, as clearly illustrated by this study. Three
common sources of TEa were chosen: biological variabil-
ity, CLIA, and RiliBÄK. There are several other sources for
TEa targets and a laboratory must decide which TEa goal
is most appropriate for it. One has to take into account that
a TEa goal is not available for every analyte. Also, when
using biological variability as the basis of TEa, it must be
noted that there are actually three possible TEa targets for
analytes: minimal, desirable, and optimal.
Biological variability TEa values are often suggested
as the most stringent, which is generally true, and appro-
priate, but their appropriateness is debatable. Taking
albumin as an example, the TEa targets for RiliBÄK, bio-
logical variability, and CLIA were 20%, 3.9%, and 10%,
respectively. The Sigma metrics under RiliBÄK ranged
from about 20 to about 32. With biological variability,

using exactly the same bias and precision values, the
Sigma values ranged from negative to 1.3. Using the CLIA
TEa target, Sigma metrics ranged from about 6 to 12. So,
depending on the TEa chosen, the same albumin assay
would be classified under CLIA as definitely of world class
quality (Sigma 6 or above) or even well above world class
using RiliBÄK, but unacceptable and not “fit for purpose”
using biological variability. But albumin is routinely
tested and apparently with acceptable clinical outcomes,
suggesting typical albumin assays are “fit for purpose.”
This suggests the biological variability TEa is perhaps
too demanding for analytical performance for some
typical field assays. Bicarbonate and the electrolytes also
exhibited poor Sigma values, undoubtedly reflecting the
demanding, and in our opinion unrealistic, TEa targets for
these analytes using biological variability.
Although it seems logical to use TEa targets consist-
ently from the same source, with experience a laboratory
may find it desirable to choose TEa values from different
sources, mixing and matching as seems appropriate for
individual assays. Some TEa targets are likely too liberal
and give a falsely optimistic estimate of quality, while
others, in particular those established using biological
variability, are conversely too demanding and yield overly
pessimistic estimates of quality. Currently it is usually up
to laboratory directors to use their best practical and pro-
fessional judgment to choose appropriate TEa goals.
Whether a laboratory chooses to set QC on an analyte-
specific basis using Sigma metrics, estimation of Sigma
metrics to compare analytical between two or among more
than two analyzers is a useful exercise. It is clear that the
patient test results from two or more analyzers if used to
analyze the same sample should be equivalent (within
defined variability limits). Comparison of quantitative
values for two analyzers can readily determine if they are
exhibiting similar performance. However, if results are not
equivalent, the question is which analyzer is “right” and
which is exhibiting aberrant performance. If Sigma metrics
are used for comparison analysis, the TEa value is the same
for both analyzers, and the Sigma metric difference is due
to either the precision or bias, or both. Comparison of the
bias and precision of both analyzers will easily establish
References
1. Lasky FD, Boser RB. Designing in quality through design control:
a manufacturer′s perspective. Clin Chem 1997;43:866–72.
2. Stankovic AK, Romeo P. The role of in vitro diagnostics
companies in reducing laboratory error. Clin Chem Lab Med
2007;45:781–8.
Brought to you by | Universiteit Gent
Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM
Hens et al.: Sigma metrics      979
which has the better precision and bias and for which the
bias and/or precision is less than desirable, allowing for
troubleshooting and correction of that analyzer.
Conclusions
In conclusion, the classical Westgard rules need not
always be used for every assay. Sigma metrics can be a
more efficient way to control quality by matching the QC
rules to the analytical quality of each individual assay.
However, the lack of TEa targets for many analytes and the
sometimes inconsistent TEa targets from different inde-
pendent sources are a major variable in the interpretation
and the application of Sigma metrics. Estimation of Sigma
metrics represents an alternative method of evaluating
comparability of analyzers.
Acknowledgments: The technical support by Ilse Van
Gysel and Patrick Rosiers was much appreciated. We also
thank Frank Heyvaert for his professional cooperation.
Conflict of interest statement
Authors’ conflict of interest disclosure: The authors
stated that there are no conflicts of interest regarding
the publication of this article. Employment and fees for
lecturing played no role in the study design; in the collec-
tion, a
­ nalysis, and interpretation of data; in the writing
of the report; or in the decision to submit the report for
publication.
Research funding: None declared.
Employment or leadership: Mario Berth: has received fees
from Abbott for lecturing. Dave Armbruster: is employed
by Abbott Diagnostics; receives salary from and holds
stock from Abbott. Sten Westgard: has received fees from
Abbott for lecturing and preparing educational materials.
Honorarium: None declared.
Received December 17, 2013; accepted February 17, 2014; previously
published online March 8, 2014
3. Gras JM, Philippe M. Application of the Six Sigma concept in
clinical laboratories: a review. Clin Chem Lab Med 2007;45:
789–96.
4. Llopis MA, Trujillo G, Llovet MI, Tarrés E, Ibarz M, Biosca C,
et al. Quality indicators and specifications for key analytical-
980      Hens et al.: Sigma metrics

extranalytical processes in the clinical laboratory. Five years’
experience using the Six Sigma concept. Clin Chem Lab Med
2011;49:463–70.
5. Kinns H, Pitkin S, Housley D, Freedman DB. Internal quality
control: best practices. J Clin Pathol 2013;66:1027–32.
6. CLSI EP5-A2. Evaluation of precision performance of quantitative
measurement methods; approved guideline – 2nd ed. Wayne, PA:
CLSI, 2004.
7. Fraser CG, Petersen PH, Ricos C, Haeckel R. Proposed quality
specifications for the imprecision and inaccuracy of analytical
systems for clinical chemistry. Eur J Clin Chem Clin Biochem
1992;30:311–7.
8. Ricós C, Alvarez V, Cava F, García-Lario JV, Hernández A,
Jiménez CV, et al. Current databases on biological variation: pros,
cons and progress. Scand J Clin Lab Invest 1999;59:491–500.
9. Westgard JO. Internal quality control: planning and
implementation strategies. Ann Clin Biochem 2003;40:593–611.
10. Westgard JO, Darcy T. The truth about quality: medical
usefulness and analytical reliability of laboratory tests. Clin
Chim Acta 2004;346:3–11.
11. Gras JM, Goffinet P, Bormans F. Practical internal QC protocol
based on Six Sigma and biological variation in a routine clinical
chemistry laboratory. Clin Chem 2009;55(Suppl):A32.
12. Friedecky B, Kratochiva J, Budina M. Why do different EQA
schemes have apparently different limits of acceptability? Clin
Chem Lab Med 2011;49:743–5.
13. Armbruster DA. Accuracy controls: assessing trueness (bias).
In: Westgard JO, Westgard S, editors. Clinics in laboratory
medicine, quality control in the age of risk management, 1st ed.
Vol 33. Philadelphia, PA: Elsevier, 2013:125–37.

Brought to you by | Universiteit Gent

Authenticated | 157.193.96.165
Download Date | 5/27/14 2:27 PM