Clinical Biochemistry 50 (2017) 550–554

Contents lists available at ScienceDirect

Clinical Biochemistry

journal homepage: www.elsevier.com/locate/clinbiochem

Review

Performance specifications for the extra-analytical phases of laboratory

testing: Why and how
Mario Plebani ⁎, on behalf of the, EFLM Task Force on Performance Specifications for the extra-analytical phases:
Department of Laboratory Medicine, University-Hospital, Padova, Italy

a r t i c l e i n f o a b s t r a c t

Article history: An important priority in the current healthcare scenario should be to address errors in laboratory testing, which
Received 16 January 2017 account for a significant proportion of diagnostic errors. Efforts made in laboratory medicine to enhance the di-
Received in revised form 1 February 2017 agnostic process have been directed toward improving technology, greater volumes and more accurate laborato-
Accepted 2 February 2017
ry tests being achieved, but data collected in the last few years highlight the need to re-evaluate the total testing
Available online 4 February 2017
process (TTP) as the unique framework for improving quality and patient safety. Valuable quality indicators (QIs)
Keywords:
and extra-analytical performance specifications are required for guidance in improving all TTP steps. Yet in liter-
Quality indicators ature no data are available on extra-analytical performance specifications based on outcomes, and nor is it pos-
Performance specifications sible to set any specification using calculations involving biological variability. The collection of data representing
Extra-analytical phases the state-of-the-art based on quality indicators is, therefore, underway. The adoption of a harmonized set of QIs, a
Total testing process common data collection and standardised reporting method is mandatory as it will not only allow the accredita-
Diagnostic errors tion of clinical laboratories according to the International Standard, but also assure guidance for promoting im-
Outcomes provement processes and guaranteeing quality care to patients.
© 2017 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
2. Patient safety and laboratory - associated errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
3. Vulnerability of extra-analytical phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
4. Outcome measures and added-value of laboratory information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
5. Extra-analytical performance specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
6. Quality indicators, measurement and diagnostic error reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
7. Quality indicators and performance specifications in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
8. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554

1. Introduction
One of the more famous aphorisms is Osler's maxim “Medicine is a
science of uncertainty and an art of probability” [1]. Laboratory informa-
tion is increasingly recognised as crucial to reducing diagnostic uncer-
tainty and enhancing quality care. Sound medical diagnoses and
effective treatments are dependent on the accurate and timely
reporting of laboratory test results, and the trend toward disease
⁎ Corresponding author at: Department of Laboratory Medicine, University-Hospital of
Padova, 35128 Padova, Italy.
E-mail address: mario.plebani@unipd.it.
prevention and personalized care calls for more complex and effective
tests and biomarkers. Today's clinical laboratory provides essential in-
formation for diagnosis, monitoring, screening, prevention, early diag-
nosis, tailored treatment and more effective monitoring of human
diseases [2]. The better understanding gained concerning the molecular
basis of human disease, the identification of risk factors for disease pre-
vention and biomarkers for an early diagnosis as well as the advent of
tailored treatment strategies has led to an upsurge in the demand for
more numerous and more reliable laboratory tests. The increasingly im-
portant role of laboratory information in medical decision making has,
however, led to the need for a greater focus on the quality and safety
of laboratory services.

http://dx.doi.org/10.1016/j.clinbiochem.2017.02.002
0009-9120/© 2017 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.
 M. Plebani / Clinical Biochemistry 50 (2017) 550–554
2. Patient safety and laboratory - associated errors
The core of patient safety, an important aspect of quality across, and
between, all settings of care, is prevention of errors associated with
healthcare and, failing that, the mitigation of their effects [3]. Thanks
to the landmark report from the Institute of Medicine, To Err is Human
[4], it became clear that avoidable patient harm was far more common
in health systems than previously realised. Yet the focus of most re-
search was on falls, medication errors and related adverse drug events,
wrong-site surgery, and nosocomial infections, diagnostic errors receiv-
ing little attention [5]. Although not all diagnostic errors, identified as an
important patient safety issue, translate into harm, a substantial num-
ber are associated with preventable morbidity and mortality [6]. Diag-
nostic errors, which affect inpatients and out-patients, have an impact
on each and every medical discipline, including laboratory medicine
[7]. Historically, efforts to improve the diagnostic process in laboratory
medicine have aimed to improve technology, achieving greater volumes
and more accurate laboratory tests, but data collected relatively recently
indicate the need to re-appraise the total testing process (TTP), the right
framework for improving quality and patient safety [8]. After decades of
focusing on improving indicators of analytical quality, such as analytical
performance specifications (in particular, bias and imprecision),
adopting valuable tools such as internal quality control (IQC) and exter-
nal quality assurance/Proficiency tests (EQA/PT), clinical laboratories
are now aware that the vulnerability of extra-analytical phases drives
need for change in the paradigm. The focal point of the concept of qual-
ity in laboratory medicine is therefore shifting from internal processes
(analytical quality) to the impact of laboratory information on patient
care and/or assuring a healthy status to any individual and/or the entire
population. As stated elsewhere “quality in laboratory medicine should
be defined as the guarantee that each and every step in the total testing
process is correctly performed, thus ensuring valuable decision making
and effective patient care” [9]. This means that the current perspective
on quality and errors in laboratory medicine focuses on a global view
of the testing process, on the issue of laboratory-associated errors and
a search for tools that minimize the risk of these errors in clinical prac-
tice. In addition to data available on errors in laboratory medicine, the
evidence that errors related to laboratory testing are common, and ac-
count for a significant fraction of diagnostic errors in medicine, has
heightened the awareness of the scientific community concerning the
need for an innovative approach to quality and safety in laboratory test-
ing [10]. There are thus two main drivers of the paradigmatic change of
the landscape of laboratory medicine: one, the evidence of the vulnera-
bility of the extra-analytical phases, and the other, the increased recog-
nition of the need for a focus on the added value of laboratory
information in improving the decision making process and clinical out-
comes [8].
3. Vulnerability of extra-analytical phases
Evidence collected in the last few decades demonstrates that
pre-analytical and post-analytical phases of the TTP are more
prone to error than the analytical phase [11–13]. Moreover, the
pre-pre-analytical (initial procedures performed for test request,
sample collection, handling and transportation) and the post-
post-analytical (final procedures performed after the notification
of laboratory results) phases are even more error prone, and vul-
nerable to errors compromising patient safety [14]. The diagnostic
testing process has therefore been divided into five phases: pre-
pre-analytic, pre-analytic, intra-analytic, post-analytic and post-
post-analytic [15]. Yet although the paradigm of quality in labora-
tory medicine is widely accepted, there is little clarity concerning
the inter-relationship between the different phases of the cycle
and the inter-dependence between quality in the pre-analytical
phase and analytical quality, and the role of post-analytical steps
in affecting ultimate laboratory information [16]. In pre-pre-
551
analytical steps the procedures for test requesting, sample collec-
tion, handling and transportation are still neglected even though
they greatly affect the quality of biological specimens and, in
turn, the accuracy of analytical results. Evidence collected on the
final steps of the loop, reveals poor/delayed acknowledgment of
laboratory reports, errors in interpretation and utilization of labo-
ratory information, and a direct correlation between missed,
wrong and delayed diagnoses and patient harm [17]. In the first
Strategic Conference of the European Federation of Clinical Chem-
istry and Laboratory Medicine, it was unanimously agreed that
“for patient care, optimizing the quality of the total (pre-analyti-
cal/analytical/post-analytical) examination process is the ultimate
goal and therefore it would be desirable to go beyond setting ana-
lytical performance specifications and to establish examination
performance specifications. In principle, the performance specifi-
cations for the pre- and post-analytical laboratory processes
should follow the same models as for analytical performance spec-
ifications. When components of these additional phases can be
expressed in numerical terms, they should be added in defining ex-
amination performance specifications. In other situations, pre- and
post-analytical performance specifications will be best represented
by separate quality indicators….” [18]. There are no data in the lit-
erature on extra-analytical performance specifications based on
outcomes, neither on clinicians' opinion, and collection of data on
the state-of-the-art based on quality indicators is ongoing.
While analytical quality indicators (QIs) based on a widely ac-
cepted hierarchy of performance criteria and well-known tools
(Internal quality control and external quality assurance/Proficien-
cy testing schemes) have been available for the last fifty years,
the development and utilization of reliable QIs for the extra-
analytical phase is still in its infancy. There is not only a need to
achieve consensus on a harmonized list of QIs, but also to collect
data and establish performance specifications for each and every
QI. According to the ISO 15189:2012, clinical laboratories should
identify critical activities and implement Quality Indicators (QIs)
in order to highlight and monitor errors when they occur [19].
QIs, managed as a part of laboratory improvement strategy, are a
suitable tool for monitoring and achieving improvement [20],
their end purpose being to keep the error risk to a minimal level,
thus curbing the likelihood of patient harm, given that no activity
is completely risk-free. However, data in the literature demon-
strate that this tool's effectiveness is closely linked to the list of
QIs chosen, and to: a) data collection method, b) data processing
procedure in use, c) appropriate analysis of results, and d) an un-
derstanding of the priorities for corrective actions according to
performance of the various QIs [21–22]. If individual laboratories
were to implement and monitor their own QIs for establishing “in-
ternal” improvement actions, only a consensually harmonized list
of QIs could assure reliable comparison between individual perfor-
mances, thus serving as a valuable benchmark and a realistic eval-
uation of quality [23]. If the final goal is “zero defect”, this level of
performance should be selected only for high-risk errors linked to
some QIs (e.g., patient and sample identification errors), while for
many other extra-analytical quality indicators a better definition
of “acceptable” performances is needed since “zero defect” is a
“mission impossible”. The project launched by the “Laboratory Er-
rors and Patient Safety” (LEPS) Working Group of the International
Federation of Clinical Chemistry and Laboratory Medicine (IFCC)
aimed to develop a Model of Quality Indicators (MQI) based on a
list of consensually defined QIs, a common reporting system and a
preliminary proposal for performance specifications representing
the state-of-the-art [24–26]. Most QIs included in the MQI are pro-
cess measures; papers on this project have already appeared in lit-
erature, and an update on it is available in this current special issue
of the Journal. Another initiative, promoted by the European Feder-
ation of Laboratory Medicine (EFLM), involves a Task Force on
552 M. Plebani / Clinical Biochemistry 50 (2017) 550–554
“Performance specifications for the extra-analytical phases” (TFG-
PSEP) whose aim is identify reliable performance specifications
for both pre- and post-analytical phases. The results of an interna-
tional survey on the state-of-the-art of QIs and related perfor-
mance specifications have recently been issued [27].
4. Outcome measures and added-value of laboratory information
An important achievement made in recent years is the raised
awareness that clinical decision making and patient outcomes are
affected not only by analytical results, but also by overall laborato-
ry information, which is closely related to the quality of biological
samples and to the requesting physician's reaction to laboratory
reports [28]. QIs, tools of crucial importance in improving the
total quality of laboratory services and patient safety, have been
defined as measurable items pertaining to processes and/or out-
comes allowing the measurement of, and improvement in, quality
of care and services [23]. Although process measures are prerequi-
site to evaluating all steps of the testing process and identifying the
procedures/processes at high risk of errors, including those incur-
ring a risk of patient harm, it has been emphasized that “process
measurement has had limited effect on value…….as patients are
interested in results” [29]. Systematic outcomes measurement is
therefore “a sine qua non of value improvement” also in laboratory
medicine. The need for an outcomes research agenda for clinical
laboratory testing was originally highlighted by George D.
Lundberg in his seminal paper published in 1998 [30]. However,
later on, in an article on appropriateness in test request/utilization
Lundberg wrote “Thirty-nine years later, these questions, plus the
big new one - what was the outcome that resulted from the test?
- still beg to be answered” [31]. The barriers to developing an
outcome-based agenda in clinical laboratories have been pin-
pointed, but there has also been a focus on the relatively recent
shift from an activity-based service to a service based on patient-
outcomes [32]. In addition, an outcome-based approach to classify-
ing and reducing testing-related diagnostic errors has been report-
ed by Epner and colleagues [33]. The proposed “five causes
taxonomy of testing-related diagnostic error” are a welcome step,
moving away from a laboratory-centered approach toward a
much more effective patient-centered focus on evaluating all
steps of the cycle and on measuring the ultimate contribution of
laboratory information in order to assure reliable diagnoses and
quality of care. This is why the LEPS WG incorporated some out-
comes measurements in the MQI. The adoption of these QIs calls
for standardisation of definition, and a common reporting system,
as already undertaken for process measures; this will provide per-
formance specifications that enable clinical laboratories to identify
and prioritize areas of improvement.
5. Extra-analytical performance specifications
The risk of errors of extra-analytical steps, the raised awareness
of a need to shift the focal point from analytical performances to a
more comprehensive evaluation of quality in the TTP, and the im-
portance of laboratory information in enabling clinicians to make
reliable diagnoses are key drivers in changing the current para-
digm's focus on analytical quality, and implementing quality indi-
cators to evaluate and improve upon the extra-analytical
processes. Setting extra-analytical quality specifications, however,
is no mean feat, as they cannot be based on outcomes (no data are
yet available), or on biological variability (as this affects
measurands but not pre- and post-analytical variables). Lab profes-
sionals should therefore follow the model based on the state-of-
the-art by collecting data on a harmonized list of QIs (MQI). Initial-
ly it was suggested that three levels of performance specifications
might be defined in accordance with the well-known model
developed by Callum Fraser for analytical specifications [34]. The
categorization into three performance levels (optimal, desirable
and minimal) is a sound rationale for analytical quality specifica-
tions, the level strongly affecting the total variability of laboratory
results. This, however, has not been demonstrated for extra-
analytical performances, which should have a desirable level
equal to zero defect. A small modification was recently introduced,
the three levels being defined as: 1) high, 25th percentile value,
representing the best performance; 2) medium, 50th percentile
value, representing the more frequent/common performance; and
3) low, 75th percentile representing the worst performance. Alter-
natively, the 10th percentile might be used to designate “first class
laboratories”, most laboratories being grouped between the 10 and
90th percentile, those with performance values outside the 90th
percentile being considered as having the “worst performance”.
In whatever case, clinical laboratory operators must clearly under-
stand that only values within the 25th or 90th percentile metrics
should be recommended. However, the main issue is to involve a
more representative number of laboratories in the MQI project,
and to collect more data to ensure that those already gathered are
representative of the state-of-the art. In future, a better definition
of the metrics to be used, as well as the level and name of categori-
zations, must be attained.
6. Quality indicators, measurement and diagnostic error reduction
In the last few years, growing evidence has been accumulated on:
a) epidemiology of diagnostic errors (frequency, types, and conse-
quences), b) causes of diagnostic errors (cognitive and system issues),
and c) error prevention strategies [35]. Diagnostic errors reflect the
complex interplay between system-related and cognitive factors, typi-
cally with multiple root causes identifiable in each single/case [36]. An
intriguing issue in laboratory-associated diagnostic errors is the inter-
play between cognitive and system-related factors, which interact
with each other both in pre-pre- and post-post-analytical phases. Cog-
nitive factors are, in fact, the leading cause of errors in test request, re-
sult interpretation and utilization, while system-related factors are
more likely to be associated with errors in sample collection, handling,
transportation and delayed acknowledgement of laboratory reports.
QIs should therefore be used to identify and document system-related
and cognitive errors, while root cause analysis should allow each clinical
laboratory to identify the nature of errors and to promote appropriate
corrective and preventive actions. System-related errors in clinical lab-
oratories spring from a poor design or poor compliance with the stan-
dard operating procedure (SOP), particularly for sample collection,
Table 1
System and cognitive diagnostic errors: recommended interventions.
System errors Cognitive errors
Failures in: Failures in:
- Technology - Information gathering
- Physical environment - Interpretation
- Organizational characteristics (quality - Data integration
system)
- Poor compliance with standard operating - Establishing a diagnosis
procedures
- Communicating explanation
to patient
Interventions: Interventions:
- Quality indicators - Education/training
- Accreditation according to ISO 15189 - Checklist
- Clinical outcomes studies
- Clinical audit
- Diagnostic conferences
- Postmortem examinations
- Decision support
- Second opinion
 M. Plebani / Clinical Biochemistry 50 (2017) 550–554
transportation, as well as delayed/missed acknowledgment and follow-
up of laboratory data. Corrective and preventive actions should there-
fore be based on improving quality systems, policies and procedures
and health information technologies, as shown in Table 1.
On the other hand, cognitive errors in laboratory medicine mainly
affect appropriateness in test request, result interpretation and utiliza-
tion. In this case, the error aetiology calls for different corrective/pre-
ventive actions based on efforts to increase knowledge and
experience, availability of decision support tools at the point of care
(e.g. incorporating decision support logic in computer-based order
entry systems) on ordering laboratory tests and interpreting results
[37], and, if necessary, recourse to “second opinions” [38] which play a
major in pathology, cytology, and other laboratory testing activities,
such as Gram stain, known to have substantial rates of inter-
laboratory variability [39].
7. Quality indicators and performance specifications in action
The purposes of measurement science, an evolving field in medicine
in particular, are to: a) establish the incidence and nature of errors, mis-
takes and defects; b) determine the causes and risks of errors;
c) evaluate interventions; d) provide education and training; and e)
guarantee accountability [8].
QIs are known to be valuable in identifying errors in health care
[40], and the International Standard for accreditation (ISO 15189)
includes their development, implementation and monitoring as a
requirement for laboratory medicine. However, not only are QIs es-
sential tools for individual laboratories for identifying procedures
and processes at an increased risk of error, but they also should
be used as a component of external quality assurance/proficiency
testing to document the level of quality in the total testing cycle
and, in particular, in extra-analytical phases. To assure inter-
laboratory comparability and a valuable benchmark, however, con-
sensus must be achieved on a harmonized list of QIs and a common
reporting system, as proposed in the MQI developed by IFCC LEPS
WG [23–24]. Only by using common QIs and a standardised
reporting system can we document the state-of-the-art and define
current performance specifications. It is time to close the gap
Fig. 1. Right utilization of quality indicators in clinical laboratories.
553
between the increased awareness of the importance of extra-
analytical performance specifications and the fact that only a few
clinical laboratories measure a comprehensive set of QIs.
However, it should always be borne in mind that data on extra-
analytical quality indicators, as already demonstrated for intra-
analytical indicators, do not per se improve quality, and nor are QIs a
magic wand. Only the correct interpretation and reaction to the data
can clarify the nature of errors, thus leading to valuable corrective and
preventive actions. The main “take-home” message is, as shown in Fig.
1, that commitment of laboratory professionals in adopting and using
QIs data is crucial to promoting improvement processes.
8. Conclusions
Timely and accurate diagnosis, the keystone to good clinical practice,
is of utmost importance in achieving optimal patient outcomes [41]. Er-
rors related to laboratory testing account for a significant proportion of
diagnostic errors and addressing these errors represents a priority in the
current healthcare system scenario [42]. Today, the perspective on qual-
ity and errors in laboratory medicine focuses on an overall view of the
testing process, on the issue of laboratory-associated errors and on the
search for tools minimizing the risk of these errors in clinical practice.
QIs and related performance specifications are valuable tools for
documenting, monitoring and improving quality in the total testing pro-
cess, in particular in extra-analytical phases. Laboratories attending the
College of American Pathologists Q-Tracks program have achieved a sig-
nificant decrease rates following their use of performance QIs, regular
collection of data, and issuing of reports [43]. The adoption of a harmo-
nized set of QIs, a standardised data collection system and reporting
method is essential to achieving reliable performance specifications
reflecting the state-of-the-art and guiding improvement initiatives.
The MQI project promoted by the IFCC LEPS WG and the recently orga-
nized Second Conference on the harmonization of quality indicators
promoted by the EFLM TFG-PSEP represent a valuable framework for
developing reliable performance specifications in the extra-analytical
phases of laboratory testing. Laboratory professionals, International
Federations and National Societies of Laboratory Medicine must in-
crease their efforts to ensure a more generalized use of these tools in
554 M. Plebani / Clinical Biochemistry 50 (2017) 550–554
order to improve quality and safety in diagnostic and therapeutic
pathways.
References
[1] R.B. Bean, W.B. Bean, Sir William Osler: Aphorisms from His Bedside Teachings and
Writings, Henry Schuman, New York, 1950.
[2] M. Plebani, Clinical laboratories: production industry or medical services? Clin.
Chem. Lab. Med. 53 (2015) 995–1004.
[3] Patient safetyAvailable from http://www.euro.who.int/en/health-topics/Health-sys-
tems/patient-safety 2016 (accessed December 18th, 2016).
[4] L.T. Kohn, J.M. Corrigan, M.S. Donaldson, To Err is Human: Building a Safer Health
System, National Academies Press (US), Washington (DC), 2000.
[5] H. Singh, M.L. Graber, T.P. Hofer, Measures to improve diagnostic safety in clinical
practice, J. Patient Saf. 20 (Oct 2016) (Epub ahead of print).
[6] T.K. Gandhi, A. Kachalia, E.J. Thomas, Missed and delayed diagnoses in the ambula-
tory setting: a study of closed malpractice claims, Ann. Intern. Med. 145 (2006)
488–496.
[7] H. Singh, M.L. Graber, S.M. Kissam, A.V. Sorensen, N.F. Lenfestey, E.M. Tant, et al.,
System-related interventions to reduce diagnostic errors: a narrative review, BMJ
Qual. Saf. 21 (2012) 160–170.
[8] M. Plebani, Quality in laboratory medicine: 50 years on, Clin. Biochem. (2016) (Epub
ahead of print).
[9] M. Plebani, Quality indicators to detect pre-analytical errors in laboratory testing,
Clin. Biochem. Rev. 33 (2012) 85–88.
[10] M.L. Graber, The physician and the laboratory, Am. J. Clin. Pathol. 124 (2005) s1–s4.
[11] M. Plebani, P. Carraro, Mistakes in a stat laboratory: types and frequency, Clin. Chem.
43 (8 Pt 1) (Aug 1997) 1348–1351.
[12] P. Carraro, M. Plebani, Errors in a stat laboratory: types and frequencies 10 years
later, Clin. Chem. 53 (2007) 1338–1342.
[13] M. Plebani, Errors in laboratory medicine and patient safety: the road ahead, Clin.
Chem. Lab. Med. 45 (2007) 700–707.
[14] P. Carraro, T. Zago, M. Plebani, Exploring the initial steps of the testing process: fre-
quency and nature of pre-preanalytic errors, Clin. Chem. 58 (2012) 638–642.
[15] M. Plebani, The detection and prevention of errors in laboratory medicine, Ann. Clin.
Biochem. 47 (2010) 101–110.
[16] M. Plebani, Towards a new paradigm in laboratory medicine: the five rights, Clin.
Chem. Lab. Med. 54 (2016) 1881–1891.
[17] A. Kachalia, T.K. Gandhi, A.L. Puopolo, C. Yoon, E.J. Thomas, R. Griffey, et al., Missed
and delayed diagnoses in the emergency department: a study of closed malpractice
claims from 4 liability insurers, Ann. Emerg. Med. 49 (2007) 196–205.
[18] S. Sandberg, C.G. Fraser, A.R. Horvath, R. Jansen, G. Jones, W. Oosterhuis, et al., Defin-
ing analytical performance specifications: consensus statement from the 1st Strate-
gic Conference of the European Federation of Clinical Chemistry and Laboratory
Medicine, Clin. Chem. Lab. Med. 53 (2015) 833–835.
[19] ISO 15189:2012, Medical Laboratories – Requirements for Quality and Competence,
International Organization for Standardization, Geneva, Switzerland, 2012.
[20] L. Sciacovelli, O. Sonntag, A. Padoan, C.F. Zambon, P. Carraro, M. Plebani, Monitoring
quality indicators in laboratory medicine does not automatically result in quality
improvement, Clin. Chem. Lab. Med. 50 (2011) 463–469.
[21] M.J. Kirchner, V.A. Funes, C.B. Adzet, M.V. Clar, M.I. Escuer, J.M. Girona, et al., Quality
indicators and specifications for key processes in clinical laboratories: a preliminary
experience, Clin. Chem. Lab. Med. 45 (2007) 672–677.
[22] W. Shcolnik, C.A. de Oliveira, A.S. de Sao Josè, C.A. de Oliveira Galoro, M. Plebani, D.
Burnett, Brazilian laboratory indicators program, Clin. Chem. Lab. Med. 50 (2012)
1923–1934.
[23] M. Plebani, M.L. Astion, J.H. Barth, W. Chen, C.A. de Oliveira Galoro, M.I. Escuer, et al.,
Harmonization of quality indicators in laboratory medicine. A preliminary consen-
sus, Clin. Chem. Lab. Med. 52 (2014) 951–958.
[24] L. Sciacovelli, M. Plebani, The IFCC Working Group on laboratory errors and patient
safety, Clin. Chim. Acta 404 (1) (2009) 79–85.
[25] L. Sciacovelli, M. O'Kane, Y.A. Skaik, P. Caciagli, C. Pellegrini, G. Da Rin, et al., IFCC
WG-LEPS. Quality indicators in laboratory medicine: from theory to practice. Pre-
liminary data from the IFCC Working Group Project “Laboratory Errors and Patient
Safety”, Clin. Chem. Lab. Med. 49 (2011) 835–844.
[26] L. Sciacovelli, G. Lippi, Z. Sumarac, J. West, I. Garcia Del Pino Castro, K. Furtado Vieira,
et al., Working Group “Laboratory Errors and Patient Safety” of International Feder-
ation of Clinical Chemistry and Laboratory Medicine (IFCC). Quality indicators in
laboratory medicine: the status of the progress of IFCC Working Group “Laboratory
Errors and Patient Safety” project, Clin. Chem. Lab. Med. (Dec 19 2016) pii:/j/
cclm.ahead-of-print/cclm-2016-0929/cclm-2016-0929.xml 10.1515/cclm-2016-
0929 (Epub ahead of print).
[27] M. Plebani, M. O'Kane, P. Vermeersch, J. Cadamuro, W. Oosterhuis, L. Sciacovelli,
EFLM Task Force on “Performance specifications for the extra-analytical phases”
(TFG-PSEP). The use of extra-analytical phase quality indicators by clinical laborato-
ries: the results of an international survey, Clin. Chem. Lab. Med. 54 (2016)
e315–e317.
[28] M. Plebani, The quality indicator paradox, Clin. Chem. Lab. Med. 54 (2016)
1119–1122.
[29] M.E. Porter, S. Larsson, T.H. Lee, Standardizing patient outcomes measurement, N.
Engl. J. Med. 374 (2016) 504–506.
[30] G.D. Lundberg, The need for an outcomes research agenda for clinical laboratory
testing, JAMA 280 (1998) 565–566.
[31] G.D. Lundberg, Adding outcome as the 10th step in the brain-to-brain laboratory
test loop, Am. J. Clin. Pathol. 141 (2014) 767–769.
[32] C.P. Price, A. St John, R. Christenson, V. Scharnhorst, M. Oellerich, P. Jones, et al.,
Leveraging the real value of laboratory medicine with the value proposition, Clin.
Chim. Acta 462 (2016) 183–186.
[33] P.L. Epner, J.E. Gans, M.L. Graber, When diagnostic testing leads to harm: a new
outcomes-based approach for laboratory medicine, BMJ Qual. Saf. 22 (Suppl. 2)
(Oct 2013) ii6–ii10.
[34] C.G. Fraser, E.K. Harris, Generation and application of data on biological variation in
clinical chemistry, Crit. Rev. Clin. Lab. Sci. 27 (1989) 409–437.
[35] L. Zwaan, G.D. Schiff, H. Singh, Advancing the research agenda for diagnostic error
reduction, BMJ Qual. Saf. 22 (2013) ii52–ii57.
[36] M.L. Graber, S. Kissam, V.L. Payne, A.N. Meyer, A. Sorensen, N. Lenfestey, et al., Cog-
nitive interventions to reduce diagnostic error: a narrative review, BMJ Qual. Saf. 21
(2012) 535–557.
[37] K. Kawamoto, D.F. Lobach, Clinical decision support provided within physician order
entry systems: a systematic review of features effective for changing clinician be-
havior, AMIA Annu. Symp. Proc. 361-5 (2003).
[38] S.S. Raab, C.H. Stone, C.S. Jensen, R.J. Zarbo, F.A. Meier, D.M. Grzybicki, C.M. Vrbin,
N.P. Ohori, L. Dahmoush, Double slide viewing as a cytology quality improvement
initiative, Am. J. Clin. Pathol. 125 (2006) 526–533.
[39] L. Samuel, M. Plebani, Targeting errors in microbiology: the case of the Gram stain,
Clin. Chem. Lab. Med. (Oct 8 2016) pii:/j/cclm.ahead-of-print/cclm-2016-0828/
cclm-2016-0828.xml 10.1515/cclm-2016-0828 (Epub ahead of print).
[40] J. Mainz, Defining and classifying clinical indicators for quality improvement, Int. J.
Qual. Health Care 15 (2003) 523–530.
[41] H. Singh, M.L. Graber, Improving diagnosis in health care – the next imperative for
patient safety, N. Engl. J. Med. 373 (2015) 2493–2495.
[42] D. Khullar, A.K. Jha, A.B. Jena, Reducing diagnostic errors - why now? N. Engl. J. Med.
373 (2015) 2491–2493.
[43] F.A. Meier, R.J. Souers, P.J. Howanitz, J.A. Tworek, P.L. Perrotta, R.E. Nakhleh, et al.,
Seven Q-Tracks monitors of laboratory quality drive general performance improve-
ment: experience from the College of American Pathologists Q-Tracks program
1999–2011, Arch. Pathol. Lab. Med. 139 (2015) 762–775.