Measurement uncertainty:

Trends and developments in the

medical laboratory
Paulo Pereira, Ph.D., Department of R&D

73º Congreso Argentino de Bioquímica

Buenos Aires, August 22, 2019
Contents
• What is happening?
• A brief introduction to the “Uncertainty Approach”
• What is the importance of measurement traceability?
• How to combine uncertainty components?
• Why are modeling models not suitable for the medical laboratory?
• Why are empirical models suitable for the medical laboratory?
• How to report measurement uncertainty?
• How to assess measurement uncertainty?
• How to select the best model to compute measurement
uncertainty?

p. 2
What is happening?
What is happening?
• The quality control skills in medical laboratory staff is
probably at the weakest level of past decades
• The medical laboratory quality control is limited in several
areas, opposing what happens in chemistry and physics
• ISO 15189 has not been successfully implemented
worldwide, and it presents several limitations regarding
standardize practices, and quality control is one of these
fields
• There is a need for harmonized practices

p. 4
What is happening?
• The “Uncertainty Approach” is not systematically used in
medical laboratories
• Medical laboratories are probably the only field on
chemistry (or physics) representing an unsuccessful case of
its implementation close to 23 years after the publication of
the Guide to the Expression of Uncertainty in Measurement
(GUM)
• The implementation of the “Uncertainty Approach” remains
a complex challenge
• A consistent and reliable discussion is needed to assure a
successful implementation of the “Uncertainty Approach”

p. 5
What is happening?
• Myths:
a) “Uncertainty Approach” does not considers the effect of bias
to the uncertainty of results since bias must be corrected if it
is significant or even neglected
‒ Absolutely misunderstood!
‒ Bias correction is a good lab practice and should be always
considered, independently of the approach!
‒ The contribution of bias to the uncertainty is referred to as
“bias uncertainty”
‒ If bias is significant and it is not corrected, bias uncertainty
will be a major contribution to the measurement uncertainty

p. 6
What is happening?
• Myths:
b) Total analytical error (TAE) = measurement uncertainty
‒ Only true if bias is zero or nonsignificant
‒ If bias is significant, the TAE is systematically higher, and
that’s a fact!
‒ It is due to the major difference between both mathematical
models, i.e, the way how bias is combined
‒ TAE sums to the precision multiplied by a coverage factor k
‒ Measurement uncertainty follows the standard variation
rules to combine uncertainty components (root of the sum of
the squared deviations)

p. 7
What is happening?
• Myths:
c) Measurement uncertainty cannot combine biological
variation components
‒ False!
‒ Biological variation components published on tables can be
interpreted as Type B sources
‒ When they are know, they should be included or the
measurement uncertainty will be misestimated!
d) Modeling models cannot be applied in med lab results, so
measurement uncertainty is not a reliable estimate
‒ Another misunderstanding!
‒ See what is happening in chemistry with empirical models!
p. 8
What is happening?
• Myths:
e) EQA/PT standard deviation is a contributor to a reliable and
consistent measurement uncertainty
‒ Absolutely false!
‒ EQA/PT is reliable uniquely to determine bias!
‒ Its standard deviation is a well-recognized cause of
overestimation/unrealistic/inapplicable
f) Measurement uncertainty is not suitable to evaluate the
performance of quantitative results in the med lab
‒ False!
‒ See what has been published!
p. 9
 Paradigm change in the
clinical laboratory by moving
from “Error Approach” to
“Uncertainty Approach”?
Paradigm change in the clinical laboratory by
moving from “Error Approach” to “Uncertainty
Approach”?
• The 2014 conference in Milan led to a consensus for the
review of quality requirements in the clinical laboratory
• Sandberg S, Fraser CG, Horvath AR, Jansen R, Jones G,
Oosterhuis W, Petersen PH, Schimmel H, Sikaris K,
Panteghini M. Defining analytical performance specifications:
Consensus statement from the 1st Strategic Conference of the
European Federation of Clinical Chemistry and Laboratory
Medicine. Clin Chem Lab Med 2015, 53(6): 833-835
• One of the most important issues related to measurement
uncertainty and model failures to calculate the total error

p. 11 de 56
Paradigm change in the clinical laboratory by
moving from “Error Approach” to “Uncertainty
Approach”?
• During the Milan conference it was recognized that many
issues, including those related to total analytical error, remain
unresolved and require further development
• A group was established to discuss the continuity of the total
analytical error by an EFLM Task and a Finnish group
• Oosterhuis EP, Bayat H, Armbruster D, Coskun A, Freeman
KP, Kallner A, Koch D, Mackenzie F, Migliarino G, Orth M,
Sandberg S, Sylte MS, Westgard S, Theodorsson E. The use of
error and uncertainty methods in the medical laboratory. Clin
Chem Lab Med 2018, 56(2):209-219
• A consistent and reliable discussion remains necessary
p. 12 de 56
A brief introduction to the
“Uncertainty Approach”
A brief introduction to the “Uncertainty
Approach”
• “Error Approach”
Reference value Mean
Frequency, f

b + k · sRW
Total analytical error

p. 14
A brief introduction to the “Uncertainty
Approach”
• “Uncertainty Approach”
Reference value Mean
Frequency, f

k  ( sRw
2
 ub2 )1/2
Expanded measurement
uncertainty

p. 15
A brief introduction to the “Uncertainty
Approach””
• Systematic error vs.
random error vs.
measurement

Systematic effect (trueness)

uncertainty
Precise and with significant bias Imprecise and with significant bias
b) d)

Precise and with no significant bias Imprecise and with no significant bias
a) c)
Random effect (precision)
p. 16
A brief introduction to the “Uncertainty
Approach”
• The result of the total analytical error can be interpreted as
the result of the combination of the difference between the
results and the difference between its mean and the
reference value

• The result of the measurement uncertainty can be

interpreted as the result of the combination of the difference
between the results, including the contribution to this
difference caused by the bias

p. 17
A brief introduction to the “Uncertainty
Approach”
• Measurement uncertainty (MU) is defined as “non-negative
parameter characterizing the dispersion of the quantity
values being attributed to a measurand, based on the
information used” (entry 2.26 of VIM)
• Measurement uncertainty expresses the combination of
sources of uncertainty
• GUM (“uncertainty bible”) is firstly published in 1993 and
two years later is corrected and reprinted.
• GUM is an open access document since it is republished
with minor correction by Bureau International des Poids et
Mesures (BIPM) in 2008
p. 18
A brief introduction to the “Uncertainty
Approach”
• The uncertainty in the result of a measurement generally
consists of several components which may be grouped into
two categories according to the way in which their
numerical value is estimated:
‒ Type A - Those which are evaluated by statistical methods,
e.g., within-laboratory reproducibility uncertainty and bias
uncertainty.
‒ Type B - Those which are evaluated by other means, e.g.,
biological variation

p. 19
A brief introduction to the “Uncertainty
Approach”
• MU characterizes “the quality of a result of a measurement”
expressed in uncertainty (quantitative indication)”
• MU is not intended to be applied to another quantity than
numerical, for what it cannot be a standard to the
determination of measurement uncertainty in all estimations
• Measurement uncertainty provides information on the level
of confidence on the measurement result
• It is the expression of multiplying the standard combined
uncertainty uc by a coverage factor k, referred to as the
“expanded uncertainty,” U

p. 20
A brief introduction to the “Uncertainty
Approach”
• Expanded uncertainty represents an interval which may be
expected to include a large fraction of the distribution of
values which could reasonably be attributed to the
measurand
• Typically k is assumed as 2 for a confidence interval of 95%
when the effective degrees of freedom veff are higher than
about six (on a Type A evaluation is equal to n - 1)
• An accurate k is dependent from n
• Simply use the Microsoft® Excel® function
=TINV(probability,deg_freedom)

p. 21
A brief introduction to the “Uncertainty
Approach”
• Commonly, the measurement uncertainty result is expressed
as value ± expanded uncertainty
• For example, in an alanine aminotransferase test: 60 ± 0.3
IU/L (expanded uncertainty k = 2) corresponds to the
interval 59.7-60.3 considering the clinical decision limit
equal to 60 IU/L
• Allows assessment of “fitness for purpose” of result

p. 22
 A brief introduction to the “Uncertainty
Approach” e.g., components in reproducibility
and repeatibility conditions, method
Describe
e.g., within- validation, IQC, and EQA/PT
measurand
laboratory measurements
reproducibility
uncertainty, Measurement
Identify uncertainty
bias uncertainty
components
uncertainty assessment

Convert uncertainty
Report measurement
components to standard
uncertainty
deviations, u(xi)

Determine combined Determine expanded

standard uncertainty, uc standard uncertainty, U

Using the rules for Determine k

combination of p. 23
uncertainty
What is the importance of
measurement traceability?
 Why is it important? It assures the
reliability of the measurements due to
What is the importance of measurement “the result can be related to a reference
traceability? through a documented unbroken chain
of calibrations, each contributing to the
SI measurement uncertainty”.
Units A measurement result is related to a
(a)
reference.
Metrology International 𝑢c 𝑦(𝑎, 𝑏) = 𝑢(𝑎)2 + 𝑢(𝑏)2
agencies Standards
laboratories
𝑏 𝑦(𝑎, 𝑏) = 𝑦 𝑏 − 𝜇(𝑎)
(b)
Pure chemical 𝑢c 𝑦(𝑏, 𝑐) = 𝑢(𝑏)2 + 𝑢(𝑐)2
reference materials 𝑏 𝑦(𝑏, 𝑐) = 𝑦 𝑐 − 𝜇(𝑏)
(c)

Metrology 𝑢c 𝑦(𝑐, 𝑑) = 𝑢(𝑐)2 + 𝑢(𝑑)2

Primary methods (d)
laboratories 𝑏 𝑦(𝑐, 𝑑) = 𝑦 𝑑 − 𝜇(𝑐)

𝑢c 𝑦(𝑑, 𝑒) = 𝑢(𝑑)2 + 𝑢(𝑒)2

Primary
reference materials (e) 𝑏 𝑦(𝑑, 𝑒) = 𝑦 𝑒 − 𝜇(𝑑)

Reference laboratories and 𝑢c 𝑦(𝑒, 𝑓) = 𝑢(𝑒)2 + 𝑢(𝑓)2

Secondary methods and
manufacturers of
reference materials (f) 𝑏 𝑦(𝑒, 𝑓) = 𝑦 𝑓 − 𝜇(𝑒)
reference materials

Working methods and 𝑢c 𝑦(𝑓, 𝑔) = 𝑢(𝑓)2 + 𝑢(𝑔)2

Medical laboratories
reference materials (g) 𝑏 𝑦(𝑓, 𝑔) = 𝑦 𝑔 − 𝜇(𝑓)

p. 25
What is the importance of measurement
traceability?
• However… there are several limitation in the medical
laboratory…
• It is not widely employed in most of medical laboratory
tests due to the unavailability of reference materials and
reference methods for most of the tests
• Also the “medical traceability” is hard to achieve due to the
“physicochemical complexity” of human samples caused
principally by the within-individual and inter-individual
biological variation

p. 26
How to combine uncertainty
components?
How to combine uncertainty components?
• Rule 1 for combination of uncertainty: Sum or difference
“For models involving only a sum or difference of
quantities, e.g., y = (p + q + r + …), the combined standard
uncertainty uc(y) is given by”:
𝑢c 𝑦 𝑝, 𝑞 … = 𝑢 𝑝 2 +𝑢 𝑞 2 + ⋯
• Rule 2 for combination of uncertainty: Product or quotient
“For models involving only a product or quotient, e.g.,
y = (p ∙ q ∙ r ∙ …) or 𝑦 = 𝑝/ 𝑞 ∙ 𝑟 ∙ ⋯ the combined
standard uncertainty uc(y) is given by”:
𝑢c 𝑦 = 𝑦 𝑢 𝑝 /𝑝 2 + 𝑢 𝑞 /𝑞 2 + ⋯

p. 28
Why are modeling models
not suitable for the medical
laboratory?
Why are modeling models not suitable for
the medical laboratory?
• A cause-and-effect diagram to the determination of
measurement uncertainty in a hypothetical medical
laboratory test
Pipetting Incubation Reagents

Batch
Accuracy Length
Expiry

Precision Uniformity Storage

Temperature Stability
Robustness
Measurement
uncertainty

Controls Linear range

Samples Count Inter-operator

Calibrators Within-operator
Blank count

Non significant
Materials Reading Operator
causes
p. 30
Why are modeling models not suitable for
the medical laboratory?
• A Pareto diagram to the determination of measurement
uncertainty in a hypothetical medical laboratory test
10 120%

9 100%
95% 100%
8
86%
7
75% 80%
6
58%
5 60%

4
39%
40%
3

2
20%
1

0 0%
Operator Materials Reagents Incubation Pipetting Reading
p. 31
Why are modeling models not suitable for
the medical laboratory?
• Simulation technique: Histograms from six simulations with data
normally distributed: A: 10; B: 102; C: 103; D: 104; E: 105; and F: 106
A B C
trials

D E F

p. 32
Why are modeling models not suitable for
the medical laboratory?
• Modeling models require a model equation with the
functional relationship y = f (x1, x2, ..., xN)
• The modeling model is based on the Law of Uncertainty
Propagation by the partial derivative method
• In the future edition of GUM, Monte Carlo simulation as an
alternative method to the Uncertainty Propagation Law will
be recommended as a preferred method
• Depth chemistry, mathematical and statistical skills are
required - not commonly available in medical laboratories

p. 33
Why are modeling models not suitable for
the medical laboratory?
• Compared to the empirical models, they misestimate
measurement uncertainty
• They are very complex to calculate and expensive
• They are adequate to the manufacturer of reagents, where the
most critical sources of uncertainty should be controlled to
assure more realistic results

p. 34
Why are empirical models
suitable for the medical
laboratory?
Why are empirical models suitable for the
medical laboratory?
• They do not require a model equation
• They do not require depth statistical skills
• They are focused on “total uncertainty” instead “uncertainty
components”
• They are based on empirical data readily available and well-
known by the medical laboratory staff
• They use data from method validation, including
interlaboratory comparisons, internal quality control (IQC),
and External Quality Assessment (EQA)/(Proficiency
Testing (PT))

p. 36
Why are empirical models suitable for the
medical laboratory?
• The computation of the within-laboratory reproducibility
standard deviation (combining repeatability standard
deviation) is similar to those used on TAE
• Bias uncertainty can also easily be computed
• Practicable in a common spreadsheet
• The empirical models are successfully used in chemistry for
close to 20 years - let learn with successful cases of
application!

p. 37
 Which empirical
models are suggested?
Which empirical models are suggested?
• Single laboratory validation (including quality control)
Two uncertainty sources are combined in measurement
uncertainty:
a) The within-laboratory reproducibility uncertainty sRw
It is calculated by pooling the repeatability standard deviation sr
arising from replicate measurements of samples, and the
intermediate standard deviation, sI from between runs:

𝑠Rw = 𝑠r2 + 𝑠I2

p. 39
Which empirical models are suggested?
• Single laboratory validation (including quality control)
Two uncertainty sources are combined in measurement
uncertainty:
b) Bias uncertainty ub
‒ Bias b is the result of the mean deviation of measurement
results of replicates from the corresponding reference value.
‒ sb is the bias standard deviation, 𝑢𝑐ref the reference value
standard uncertainty, and m the number of replicate
determinations:

𝑢𝑏 = 𝑏 2 + 𝑠𝑏 / 𝑚 2 + 𝑢𝑐2ref
p. 40
Which empirical models are suggested?
• Single laboratory validation (including quality control)
c) To obtain the combined standard uncertainty the uncertainty
due to precision and that due to bias are combined:

2
𝑢c 𝑦 = 𝑠Rw + 𝑢𝑏2
d) Expanded uncertainty is compute, as follows
U = k ∙ u(y)c

p. 41
Which empirical models are suggested?
• Single laboratory validation (including quality control)
Two sub-models are considered:
a) Short-term data
‒ Short-term data is required
‒ Method validation protocol intended for validating the
precision of numerical quantity tests is required
‒ Clinical Laboratory Standards Institute (CLSI) EP15-A3
protocol to evaluate the precision (and bias) (within-run and
between-run precision)
‒ A series of five analytical runs with three replicates per run
is suitable
p. 42
Which empirical models are suggested?
• Single laboratory validation (including quality control)
Two sub-models are considered:
b) Long-term data
‒ Long-term data is required
‒ CLSI EP5-A3 protocol or IQC data to compute the precision
‒ Using data from within-run, between-run, and between-day
precision

p. 43
Which empirical models are suggested?
• Interlaboratory comparisons
a) In the inter-laboratory validation approach, the principal
sources of variability can often be assessed by inter-laboratory
studies performed and evaluated according to ISO 5725
b) This approach to estimating uncertainty is fully described in
ISO 21748
c) The approach requires the determination of the between-
laboratories reproducibility standard deviation sR from the
results in an inter-laboratory trial according to ISO 5725
d) In a standardized method, these precision data are usually
given in an Appendix
p. 44
Which empirical models are suggested?
• External Quality Assessment (EQA)/(Proficiency Testing (PT))
a) EQA programs are proposed to verify periodically the
performance of a laboratory test based in data of a
laboratories’ group using proficiency tests
b) Medical laboratory could also use the comparison data to
determine the measurement uncertainty
c) EuroLab Technical Report 1/2007 presents an approach to
laboratories evaluate measurement uncertainty
d) The mathematical model could be the same than in the single
laboratory validation

p. 45
Which empirical models are suggested?
• External Quality Assessment (EQA)/(Proficiency Testing (PT))
e) Bias confidence could be several affected by the
heterogeneity of the results!
f) Use the EQA/PT data just to compute bias when a reference
material is not available; therefore, combine the standard
uncertainty as follows:

2
𝑢c 𝑦 = 𝑠Rw + 𝑢𝑏2
g) Expanded uncertainty is compute, as follows
U = k ∙ u(y)c

p. 46
How to report measurement
uncertainty?
How to report measurement uncertainty?
• Unless otherwise required, the result x should be stated together
with the expanded uncertainty U calculated using a coverage
factor k = 2 (….). The following form is recommended:
(Result): (x ± U) (units) [where] the reported uncertainty is [an
expanded uncertainty as defined in the International Vocabulary
of Metrology - Basic and General Concepts and Associated
Terms, 3rd ed., BIPM 2012, calculated using a coverage factor
of 2, [which gives a level of confidence of approximately 95%]
• Terms in parentheses [] may be omitted or abbreviated as
appropriate. The coverage factor should, of course, be adjusted
to show the value actually used
p. 48
How to assess measurement
uncertainty?
How to assess measurement uncertainty?
• Target measurement uncertainty is defined as “measurement
uncertainty specified as an upper limit and decided on the basis
of the intended use of measurement results”
• So, target uncertainty could be interpreted as the analytical
quality goal that sets a limit for measurement uncertainty that is
tolerable in a single measurement or single test result
• Laboratorians use published performance specifications based in
the “Error Approach” in method validation
• Considering that measurement uncertainty is ≤ total analytical
error

p. 50
How to assess measurement uncertainty?
• So, measurement uncertainty must be ≤ than the “allowable total
error” or “biological error”
• Well, that’s a condition!
• Therefore, an expanded uncertainty not significantly distant
from acceptance limits appears to support the conformance of
uncertainty

p. 51
How to assess measurement uncertainty?
• R. Bettencourt da Silva, A. Williams (Eds), Eurachem/CITAC
Guide: Setting and Using Target Uncertainty in Chemical
Measurement, (1st ed. 2015)
a)
b)
y U

Upper c)
limit
d)

p. 52
 How to select the best
model to compute
measurement uncertainty?
 How to select the best model to compute
measurement uncertainty?
Selection of a
• Flowchart to select a modelmethod
to the determination of measurement uncertainty
in a medical laboratory
R&D R&D or Medical
(modeling medical laboratory
approach) laboratory (empirical
approach)

Is Monte Carlo Intralaboratory

Simulation Is long-term comparisons -
Yes method method validation or Yes

Probability of realistic estimate

method feasible? “CLSI EP5-A3”
IQC data available? or IQC
No No

Partial Is partial Is short-term Intralaboratory

derivative Yes derivative method validation Yes comparisons -
method method data available? “CLSI EP15-A3”
feasible?

Risk of overestimate
No
No Is short-term Interlaboratory
inter-laboratory Yes comparisons -
studies data available? Reference lab
No

Not No Is EQA / PT Yes EQA / PT

determined data available? comparisons p. 54
Case
Case
• CASE: measurement uncertainty of serum ferritin (μg / L)
‒ Measurement uncertainty method: Single laboratory validation
using data from within-run, between-run, and between-day
variation
‒ Analyte: Serum ferritin μg / L (quantitative method)
‒ Mensurand: Serum ferritin tested by immunoturbidimetric
method. Human serum is stored 7 days at 2 to 8 ° C or 1 year at
-20 ° C. The sample may be thawed only once. Do not use
highly hemolyzed or lipemic samples. Fibrin specimen should
be centrifuged prior to testing.
‒ Clinical decision point of 140 μg / L
p. 56
Case
• CASE: measurement uncertainty of serum ferritin (μg / L)
‒ Previously bias is corrected using Passing-Bablok equation;
ubias compute the uncertainty arising from the remaining bias
‒ sRw = 1.7% - computation based on CLSI EP5-A3 combining
reproducibility sI and repeatability sr conditions over 20 days
2
‒ ubias = 𝑏2 + 𝑠𝑏 / 𝑚 2 + 𝑢𝑐2ref = 0.91% + 1,6%/ 6 + 0.5% =
1.1%

p. 57
Case
• CASE: measurement uncertainty of serum ferritin (μg / L)
2
‒ 𝑢c 𝑦 = 𝑠Rw + 𝑢𝑏2 = 1.7%2 + 1.1%2 = 2.1%
‒ U = k ∙ uc = 2 ∙ 2.1% = 4.2%
‒ Serum ferritin: (140 ± 4.2%) (μg / L)*

* The reported uncertainty is an expanded uncertainty calculated

using a coverage factor of 2 which gives a level of confidence of
approximately 95 %
‒ Compliance assessment: published limits from 15 to 30%
‒ U of 4.2% is significantly away from all acceptance limits and
therefore appears to support the conformance of uncertainty
p. 58
Case
• Note: free “workbook for the computation of measurement
uncertainty in medical laboratory tests based on empirical
approaches v1.3” is available at:
https://www.researchgate.net/publication/324798053_Workboo
k_for_the_computation_of_measurement_uncertainty_in_medi
cal_laboratory_tests_based_on_empirical_approaches_v13

p. 59
Paulo Pereira, Ph.D., Department of R&D,
Portuguese Institute of Blood and Transplantation

e-mail: paulo.pereira@ipst.min-saude.pt

ResearchGate:
http://www.researchgate.net/profile/Paulo_Pereira11
Measurement uncertainty:
Trends and developments in the
medical laboratory
Paulo Pereira, Ph.D., Department of R&D

73º Congreso Argentino de Bioquímica

Buenos Aires, August 22, 2019