Research Article

Annals of Clinical Biochemistry

2018, Vol. 55(2) 254–263
! The Author(s) 2017
Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/0004563217712291
journals.sagepub.com/home/acb

Designing and evaluating autoverification rules for thyroid

function profiles and sex hormone tests

Jiancheng Li1,*, Bizhen Cheng2,*, Huizhen Ouyang2, Tongtong Xiao2, Jing Hu2 and
Yingmu Cai2

Abstract
Purpose: Following the analytical phase, the current practice of many hospital laboratories involves the manual veri-
fication of all test results followed by the production of the report. However, manual verification is a time-consuming and
tedious process. In this paper, we provide a detailed description of how to design autoverification rules for thyroid
function test profiles and sex hormones.
Materials and methods: We used DM2 (Data manager 2) to construct the algorithm and build the database for
autoverification of thyroid function test profiles and sex hormones, with reference to Boolean logic, Auto 10-A and
CLSI’88. The rules consist of checking quality control, instrument error flags, critical values, the analytical measurement
range (AMR), the limit range, consistency check and delta check. Firstly, we established the rules in the DM2, collected
clinical specimens for validation, then tested the rules in a ‘live’ environment.
Results: Agreement was achieved between manual verification by two senior laboratory personnel and verification using
the autoverification rules in 99.78% of the cases. The total autoverification rate for all tests was 77.06%. Following
implementation of the rules, the laboratory turnaround time (TAT) was reduced by 54.55% and staffing numbers fell from
three to two whole time equivalents (WTE). Statistical analysis resulted in a kappa statistic of 0.99 (P < 0.001). Moreover,
after implementing the autoverification rules, the error rate fell to 0.04%, indicating that errors were almost completely
eliminated.
Conclusion: Implementing autoverification rules can reduce TAT, minimize the number of samples that require manual
verification and allow for a reduction in staffing numbers. It also allows laboratory staff to devote more time and effort to
the handling of problematic test results and contributing to improved patient care.

Keywords
Autoverification, laboratory automation, DM2, thyroid hormones, sex hormones
Accepted: 5th May 2017

1
Department of Clinical Laboratory, Central Hospital of Hengyang,
Hunan, People’s Republic of China
2
Department of Clinical Laboratory, The First Affiliated Hospital of
Introduction Shantou University Medical College, Guangdong, People’s Republic of
China
Currently, clinical laboratories are under continual
pressure to increase their productivity, through the *Contributed equally
handling of larger workloads with fewer qualified
Corresponding author:
staff. Following the parallel developments of increased Yingmu Cai, Department of Clinical Laboratory, The First Affiliated
laboratory automation and major developments in Hospital of Shantou University Medical College, Shantou 515041, China.
information technology (IT), many clinical laboratories Email: st_ymcai@163.com
Li et al.
in hospitals now utilize two-way communications
between laboratory information system (LIS) and ana-
lytical instruments.1,2 However, despite this, the speeds
by which some test results are released remain compro-
mised by the low efficiency of manual verification,
which is performed by one or more senior member of
staff, on a single analytical result or on a group of
results, in order to ensure that no incorrect results are
reported to the hospital information system (HIS) or to
clinicians. Autoverification, a process by which clinical
laboratory results are released without manual inter-
vention or review,3–9 can overcome this limitation.3,10,11
The Autoverification of Clinical Laboratory Test
Result Approved Guideline (AUTO 10-A) was issued
by the American Clinical Laboratory Standards
Institute (CLSI) in 2006.9 This guideline provides a
basic framework to allow each clinical laboratory to
design, implement and validate specific autoverification
rules for specific tests according to a laboratory tech-
nologist’s demands. The use of such autoverification
can enhance the efficiency of the laboratory, reduce
budgetary costs and decrease turnaround time (TAT),
as well as ensure quality12 and enable laboratory staff
to focus more on potentially problematic test results.4
Currently, many laboratories worldwide are exploring
and implementing autoverification in several areas,
including urine analysis,13,14 haematology,15,16 clinical
biochemistry,17–20 coagulation,21,22 and clinical immun-
ology.7 Although there is nearly a 20-year history of
autoverification systems that can verify and validate
clinical laboratory results and the Auto10-A guidelines
have been around for nearly 10 years, there remains a
lack of standardization, especially for the algorithms
and verification limits,18 and it remains unclear how
to build autoverification rules and parameters.12,22
Although commercial autoverification systems, such
as VALAB, are available, their processing algorithms
and verifying rules are considered proprietary; there-
fore, they cannot be modified by the users.10,23
Moreover, commercial software addresses only the
most basic levels of autoverification, e.g. the reference
interval, instrument alarm and internal quality control
(IQC), and none of the commercially available pro-
grams can handle complex clinical data, such as testing
logical relationships, clinical presentation and clinical
history. As a result, clinical laboratories have built
their own autoverification systems, such as
DNSevTM24 and LabRespond.10 This can also be
achieved by developing a within-laboratory autoverifi-
cation system that would work as a part of the LIS or
the middleware, which in our laboratory consists of
Data Management 2 (DM2), and which was developed
by Beckman Coulter.
Thyroid function and sex hormone tests are two
examples of common clinically complex tests that are
255
processed in the laboratory. Familiarity with the physi-
ology and pathophysiology of the thyroid gland and
the gonads is important to ensure appropriate request-
ing of the hormones they produce. A number of medi-
cations and diseases have been shown to influence the
circulating concentrations of these hormones.25,26
Circulating concentrations of sex hormones can also
vary with the menstrual cycle and in patients undergo-
ing treatment through assisted reproductive technology
(ART). In those women receiving ART, circulating sex
hormone concentrations differ from those found in
normal or pregnant women,27,28 presenting a challenge
for the setting of any autoverification rules.26,29–32 In
this study, we present a detailed description of how to
construct autoverification rules for thyroid function
and sex hormone tests using the DM2 middleware,
and describe their subsequent validation and evaluation
in the clinic.
Materials and methods
Instrumentation
All equipment was provided by Beckman Coulter Inc.,
CA, USA (Power Processor Automated Sample
Processing System, UniCel DxI 800 Immunoassay
System, Prelink, Centrifuge, Specimen Stockyard,
Outlet Rack System).
Methods
All tests were measured using chemiluminescent micro-
particle immunoassay (CMIA), on a Unicel DxI 800
immunoassay analyser (Beckman Coulter Inc., CA,
USA). Thyroid function tests consisted of thyroid-sti-
mulating hormone (TSH), total triiodothyronine (TT3),
total thyroxine (TT4), free triiodothyronine (FT3) and
free thyroxine (FT4).
Sex hormone tests consisted of: b-human chorionic
gonadotropin (b-HCG), progesterone, testosterone, oes-
tradiol, prolactin, follicle-stimulating hormone (FSH).
In addition, all internal quality control serum prod-
ucts were provided by Beckman Coulter Inc., CA, USA.
Software
The DM2 was obtained from Beckman Coulter Inc.,
CA, USA. The hospital information system (HIS) and
Laboratory Information System (LIS) were both
obtained from B-Soft Co., Ltd, Hangzhou, China.
Statistical analysis
Analysis of the data was performed using the SPSS
statistics software package, version 20.0 for Microsoft
256
Windows (SPSS Inc., Chicago, IL, USA). A P < 0.05
was considered significant.
A kappa test was used to evaluate the difference
between autoverification and manual verification. The
kappa coefficient represents the observed agreement
above and beyond that due to chance. The strengths
of any observed agreements were as follows: <0.20
bad; 0.20–0.40 common; 0.41–0.60 moderate; 0.61–
0.80 strong and 0.81–1.00 very strong.
Methods
This study was conducted in the Clinical Chemistry
Core Laboratory of the First Affiliated Hospital of
Shantou University Medical College in cooperation
with a medical expert from the Department of
Reproductive Medicine, the DM2 vendor (Beckman
Coulter, CA, USA) and the LIS vendor (B-Soft Co.,
Ltd, Hangzhou, China). Our hospital is a regional gen-
eral academic tertiary care referral centre and university
affiliated hospital with 67 clinical departments and 1806
beds. In this study, the DM2 was integral to the auto-
verification rules. We designed them by using Boolean
logic according to the CLSI Auto10-A guidelines.9 We
then simulated and validated their performance on the
DM2 using historical patient data stored in the data-
base. The details of the rules and the flowchart are
shown in Figure 1. The flowchart shows that the rule
processes consist of IQC, instrument error flags, critical
value, instrument analytical range, limit range, consist-
ency check and delta check.
Quality assurance
Our laboratory routinely uses IQC and takes part in
external quality assessment. Daily IQC results were
transmitted from the analyser to the DM2 and LIS
and were evaluated using Levey-Jennings charts33 and
Westgard quality control multirules.34,35 In addition,
Figure 1. Flowchart of autoverification rules for thyroid function profiles and sex hormone tests.
Annals of Clinical Biochemistry 55(2)
we also used the ‘moving average’ method as an add-
itional quality assurance (QA) method to help ensure
the quality of the results. This involves the collection of
patient results over a period of 20 consecutive days.
From these results, a mean result is determined, and a
warning limit is calculated, which is the mean 2SD,
together with an action limit of the mean  3SD.36 For
example, in the case of FT4, over one period of 20
consecutive days, the warning limits (mean  2SD)
were 10.72 and 13.96 pmol/L.
Instrument error flags
The instrument will give alerts when there are problems
with the reagents, barcode, samples or mechanical fail-
ure, e.g. in the event of reagent crystallization or sample
clots forming.
Critical value
The critical values or medical decision levels were deter-
mined locally and were based upon those described
by Statland et al.37 The critical values used in our hos-
pital were as follows: b-HCG > 180,000.00 IU/L, oestra-
diol > 9177.50 pmol/L, LH > 40.00 IU/L, FSH >
70.00 IU/L, testosterone < 1.04 nmol/L, prolactin >
2120.00 mIU/L, FT3 > 40.00 pmol/L, FT3 < 1.50 pmol/
L, FT4 > 22.00 pmol/L, FT4 < 3.50 pmol/L, TT3 >
10.00 nmol/L, TT3 < 0.50 nmol/L, TT4 > 380.00 nmol/
L, TT4 < 6.50 nmol/L, TSH > 80.00 mIU/L, and
TSH < 0.02 mIU/L. Results that were outside the
range of the critical value required verification by a tech-
nologist, and those within the range passed and contin-
ued to the limit check and delta check.
Instrument analytical range
The analytical measurement ranges were: b-HCG,
0.50–1000.00 IU/L (1000.00–200,000.00 IU/L following
Li et al.
dilution); progesterone, 0.32–127.20 nmol/L; testoster-
one, 0.35–55.50 nmol/L; oestradiol, 73.00–
17,621.00 pmol/L; prolactin, 5.30–4240.00 mIU/L;
FSH, 0.20–200.00 IU/L; LH, 0.20–250.00 IU/L; TSH,
0.01–100.00 mIU/L; T4, 6.40–386.00 nmol/L; FT4,
3.20–77.20 pmol/L; T3, 0.20–12.30 nmol/L; FT3, 1.40–
46.00 pmol/L. Results out with the analytical range
generated a warning flag and required sample dilution
prior to reanalysis.
Limit range
The limit range is the rule that screening test results
must not be outside the critical values. It serves as a
filter to verify whether the results are within the limit
range specified for the analyte in the DM2. Fraser
et al.38 reported that conventional reference intervals
are not usually ideal for autoverification strategies
due to factors relating to biological variation, making
conventional population-based reference intervals of
little value. We discussed the limit range with local
physicians, and the acceptable range for the limit
check for all 12 tests was based upon the 95% confi-
dence interval determined from the distribution of his-
torical patient results obtained between October 2013
and October 2014 (Table 1).
Consistency check
Given the nature of acute illness and the fact that clin-
ical test results fluctuate frequently, it was very difficult
to perform a consistency rule check. Only some tests
had a consistency check established based on clinical
and practical diagnostic criteria. The hypothalamic
pituitary-gonadal (HPG) and hypothalamic-pituitary
thyroid (HPT) axes are negative feedback control sys-
tems. Therefore, there are some necessary relationships
between the circulating concentrations of the hor-
mones, e.g. elevated T4 or T3 concentrations associated
with a decreased TSH concentration, in patients with
toxic diffuse goitre. It was necessary that any consist-
ency rule took account of the interrelationships of all of
the hormones, and consequently, if any results violated
Table 1. Limit ranges of all the 12 tests.
Test Limit range Test
TSH 0.01–12.15 FT4
TT3 0.54–3.91 Progesterone
TT4 46.35–222.17 HCG
FT3 2.66–10.86 LH
FT3: free triiodothyronine; TT3: total triiodothyronine; TT4: total thyroxine; TSH: thyroid-stimulating hormone; FSH: follicle-stimulating hormone; LH:
luteinizing hormone; FT4: free thyroxine; HCG: human chorionic gonadotropin.
257
the consistency check, test results were unable to be
sent to the HIS. For example, if the test result for
TSH was lower than 0.34 mIU/L, with a
FT3 < 6.00 pmol/L or a TT3 < 2.73 pmol/L and
FT4 < 14.40 pmol/L or TT4 < 157.40 pmol/L; or
TSH > 0.34 mIU/L and FT3 > 6.00 pmol/L or
TT3 > 2.73 pmol/L, the report was intercepted as a
manual verification (MV) report and could not be
sent to the HIS.
Delta check
The delta check compares current test results with pre-
vious results to identify those results that differ by more
than a defined amount.39,40 Using the delta check rule
enables the identification of changes in the test results
beyond their expected variation, e.g. due to changes in
a patient’s clinical status or due to preanalytical
errors.41,42 We therefore believe that all test results
that have previous test values must be verified by a
delta check.40 The parameters that have previously
been used include the delta difference, rate difference,
rate percent change, cumulative incremental weighted
index43–45 and delta percent change.46 In this study, we
chose the rate percent changes, which were calculated
as the current value subtracted from the previous value
and then divided by the previous value. We chose a
time interval as less than seven days based on discus-
sions of the biological and pathological functions of the
thyroid gland and gonads with physicians in the
Department of Reproductive Medicine. In addition,
we suggest that if the rate of change of the concentra-
tions of the hormones was within the set range in the
previous seven days, the result could pass the delta
check. The set ranges were: 50%/7 days for TSH,
TT3, FT3, TT4, progesterone, LH, testosterone, pro-
lactin and FSH; 70%/7 days for FT4 and oestradiol
and 80%/7 days for HCG.
Patient diagnostic information
All relevant data that could affect the interpretation of
the tests results were collected according to the CLSI
Limit range Test Limit range
6.43–34.15 Testosterone 0.35–15.34
0.41–110.25 Oestradiol 73.42–9704.66
0.50–171,512.00 Prolactin 77.59–1327.97
0.45–46.88 FSH 1.62–68.37
258 Annals of Clinical Biochemistry 55(2)

Auto-10 A guidelines.9 Patient-specific information and
preanalytical variables included: gender; age; preg-
nancy status; menstrual cycle status; clinical presenta-
tion; thyroid history; pituitary and gonad disorders;
drug history; presumptive diagnosis and other medical
conditions that may affect the concentrations of the
thyroid or gonad hormones, e.g. history of a vesicular
mole, choriocarcinoma. Having patient information
can be important in the interpretation of these test
results. Because of this and because patient information
cannot be sent to the DM2, we used both the LIS and
DM2 to enable autoverification to be implemented
safely. For example, if the primary diagnosis was
hyperthyroidism, but test results of thyroid function
were all within the reference intervals, the report
would be handled by technologists and would not be
sent to the HIS (Table 2).
The DM2 was installed in our Clinical Chemistry
Core Laboratory, so we could use it to enable autover-
ification of the test results. The rules were written in
computer language with reference to the literature;47
the results that passed the autoverification rules were
marked in green, and those that failed were marked in
red to achieve autoverification in the LIS or DM2. The
results that passed all the autoverification rules could
be sent directly to the HIS. In our laboratory, the
report for autoverified results contained ‘autoverifica-
tion’ in place of the verifier’s signature as well as the
programmed comment, if any. Cases that failed the
autoverification rules were red flagged on the computer
screen, indicating that the laboratory technologist
needed to manually verify the results. In such cases,
the verifier’s name was recorded on the reports and
saved within the LIS. The details of autoverification
rules and their actions are given in Table 3.
Validation methods
To validate whether the autoverification rules and their
settings in the DM2 were able to meet our requirements
and be implemented, we used electronically simulated
cases, special samples and historical data, and the entire
validation process was based on the recommendations
in the CLSI Auto10-A.9
Electronically simulated case validation
Electronically simulated cases were used to verify that
the programmed autoverification rules followed the
expected logic and achieved the anticipated outcome.
One case for each rule was previously programmed.
The total number of cases was 1063. A total of 538
(50.6%) cases passed autoverification and 525
(49.4%) failed. The electronically simulated cases that
passed the autoverification rules were reviewed in order
to ensure that they matched the expected outcome.

Table 2. Patient information associated with tests report results.

Number Key diagnosis words Tests results that stop autoverification

1 Hyperthyroidism
2 Hypothyroidism
3 Subacute thyroiditis
4 Autoimmune thyroiditis
5 Hyperprolactinemia
6 Polycystic ovarian syndrome
7 Premature ovarian failure
8 Precocious puberty
9 Hydatidiform mole
10 Ectopic pregnancy
11 Prolactinoma
12 Sheehan’s syndrome
13 Ovarian hyperstimulation
syndrome
FSH: follicle-stimulating hormone; LH: luteinizing hormone; TSH: thyroid-stimulating hormone; FT3: free triiodothyronine; TT3: total triiodothyronine;
FT4: free thyroxine; TT4: b-HCG: b-human chorionic gonadotropin.
TSH > 4.85 mIU/L or FT3 < 3.60 pmol/L or TT3 < 1.12 nmol/L or FT4 < 7.86 pmol/L or
TT4 < 91.90 nmol/L
TSH < 0.51 mIU/L or FT3 > 5.70 pmol/L or TT3 > 2.41 nmol/L or FT4 > 14.41 pmol/L
or TT4 > 167.80 nmol/L
TSH > 4.85 mIU/L or FT3 < 3.60 pmol/L or TT3 < 1.12 nmol/L or FT4 < 7.86 pmol/L or
TT4 < 91.90 nmol/L
TSH < 0.51 mIU/L or FT4 > 14.41 pmol/L or TT4 > 167.80 nmol/L
Prolactin < 530.00 mIU/L
FSH > 8 IU/L or LH < 10 IU/L or oestradiol < 99.12 pmol/L
FSH < 40 IU/L or LH < 40 IU/L or oestradiol > 100 pmol/L or progesterone > 2 nmol/L
LH < 5 IU/L or oestradiol < 275.33 pmol/L or testosterone < 0.52 nmol/L
b-HCG < 100,000 IU/L
Progesterone > 79.50 nmol/L or b-HCG < 5 IU/L
Prolactin < 173.50 mIU/L
FSH > 8 IU/L or LH > 10 IU/L or prolactin > 566.46 mIU/L or TSH > 4.85 IU/L or
FT3 > 5.70 pmol/L or TT3 > 2.41 nmol/L or FT4 > 14.41 pmol/L or TT4 > 167.80 nmol/L
Oestradiol < 3671.00 pmol/L
Li et al.
Table 3. Autoverification rules and actions.
Rules Action(s)
Quality control Stop, identify reasons, and correct if QC
unmet; conversely, pass
Instrument error Stop and investigate if indicated; conversely,
flags pass
Critical value Stop and manually verify or recheck if
exceeded, then call doctors; conversely,
pass
Result above If manual protocol, perform dilution and
AMR manually verify; certain analytes have
automated dilution protocol (Dil-HCG);
once completed, proceed to other rules
Result below Stop and analyse if indicated; conversely,
AMR pass
Limit range Stop and manually verify; conversely, pass
Consistency Stop and manually verify or recheck if vio-
check lated, then communicate with doctors;
conversely, pass
Delta check If violated, do not autoverify; conversely,
autoverify
AMR: analytical measurement range.
Special sample validation
Special samples included the abnormal proficiency test
samples and more than 50 patient samples, containing
low and high concentrations of each analyte. Most of
the test results fell outside the acceptable range of the
critical value, instrument analysis and limit check.
These tests results were chosen to verify the efficiency
of the autoverification rules and the reliability of the
reports. The validation results demonstrated that there
were no errors in the autoverification rule use, which
indicated that the reports generated using the rules were
accurate.
Validation using historical data
The DM2 went live in the Clinical Chemistry Core
Laboratory in 2013. All clinical chemistry test results
of patients, from October 2013 to the present, were
stored in the DM2 computer, which allowed us to val-
idate the rules by using this historical data. The total
number of cases, from October 2013 to December 2014,
with thyroid and gonad hormones analysis in the DM2
were 47,448 and 23,983, respectively. All cases had been
manually verified and revised by skilled laboratory spe-
cialists in the LIS. These cases were used to represent
the same distribution and type of cases that are received
by the laboratory, and they provided an estimate of the
quality of autoverification. This validation process
which uses many test results should uncover problems
259
with both the autoverification rules and electronic
equipment that occurred infrequently prior to autover-
ification. We had to ensure that those test results
reported by autoverification were consistent with the
clinical status of the patients. The use of historical
data indicated that the rate of autoverification was
57.56%. In addition, the system and rules ran well
and did not demonstrate any error flags.
Results
According to the CLSI Auto10-A Guideline,9 autover-
ification rules must be validated by using actual patient
results before go-live. To confirm whether the autover-
ification rules were reliable, we established the rules in
the DM2, and collected and assessed patient test
results, of which there were 77400, produced during
the calendar year 2015. The validation was performed
by requesting that two senior technologists, who spe-
cialized in thyroid and gonadal disease and who
worked in cooperation in the Clinical Chemistry Core
Laboratory, verify the results in triplicate.
This study considered all factors, including the
instrumentation, reagents, standard serum samples,
specimen status, laboratory temperature and correl-
ation of the hormones. Instrument flags appeared 45
times during validation of the rules. All IQC results
were within limits, as judged by the Levey-Jennings
quality control chart33 and Westgard multi-rules.34,35
The results of actual patient validation were as follows:
1. The autoverification rate for all thyroid function
profiles and sex hormone tests ranged from 65.35%
to 77.06% (Table 4 ). The autoverification rate for
thyroid function profiles and sex hormone tests
implemented in the Clinical Chemistry Core
Laboratory increased over the period of 12 months
from 65.57% to the current overall rate of 86.78%
and from 52.78% to 63.70%, respectively.
2. A high percentage of results with critical values were
autoverified. The average rates of results exceeding
the critical value, limit range and delta check were
7.21%, 25.20% and 10.97%, respectively.
3. The reported laboratory TAT was reduced by
54.55% (from 66 to 30 min), and the time interval
from completion of analysis to verification
was reduced by 61.80%. The staffing numbers
were reduced from three to two whole time
equivalents (WTE).
Evaluation of the benefits
We used the TAT (from the time of sample receipt by
the laboratory staff to the time of result verification on
260 Annals of Clinical Biochemistry 55(2)

Table 4. Passing rate of whole items.

Date Status Jan. 01 Mar. 01 May. 01 Jul. 01 Sep. 01 Nov. 01

The number of requisition sheets 210 220 209 232 208 242
Quality control Passa 12 12 12 12 12 12
Faila 0 0 0 0 0 0
Instrument error flags 4 6 5 4 3 3
Critical value Yes 29 26 23 18 15 17
No 177 188 181 210 190 222
Out of AMRs Yes 3 4 5 3 6 4
No 203 210 199 225 199 235
Limit range Yes 122 134 130 148 137 155
No 52 50 46 59 47 63
Consistence rule check Pass 113 127 124 142 129 148
Fail 9 7 6 6 8 7
Delta check Pass 92 113 115 129 120 138
Fail 21 14 9 13 9 10
No. of previous data 13 15 12 15 16 11
Number of autoverifications 95 116 117 130 122 142
Autoverification passing rate (%) 45.23 52.73 55.98 56.03 58.65 58.68
Number of manual interventions 115 104 92 102 86 100
Manual intervention rate (%) 54.77 47.27 44.02 43.97 41.35 41.32
Error rate (%) 0.48 0 0 0.43 0 0
Average autoverification passing rate (%) 77.06
Average rate of manual verification (%) 22.94
Average error rate (%) 0.04
AMRs: analytical measurement ranges.
a
QC results for thyroid function profiles and sex hormone tests.
Bold numerals represent the number of results that passed the autoverification rules.

the LIS) and risk of human verification error to evalu-
ate the advantages of implementing autoverification for
the calendar year 2015. Following the implementation
of autoverification, the number of samples that
required manual verification decreased by 77.06%
(from 77,400 to 59644, Table 4). Furthermore, the
TAT fell by 54.55% (from 66 min to 30 min), findings,
which are consistent with previous reports that
described the use of autoverification rules as a way of
reducing TAT. This approach also reduces the heavy
workload in clinical laboratories.4,20 In addition, our
study demonstrates the additional advantage of
improved consistency in the verification process over
that undertaken by different clinical medical technolo-
gists, since the rules were based on the same standar-
dized processes and all results were therefore treated in
the same consistent manner. Before implementation,
manually verified errors occurred at an average of
approximately four results or fewer per day. In con-
trast, after implementing the rules, errors were almost
eliminated in a ‘live’ clinical environment, using the
observations of two senior technologists as a reference
standard. The results of the kappa test for the inter-
observer degree of agreement, indicated that there
was a close agreement between results from profes-
sional reviewers and the results obtained following
autoverification (Table 5).
Discussion
Autoverification is a process of using computer-based
rules to undertake the initial validation of test results
without manual intervention. Data that fall outside the
set rules should be reviewed by the laboratory technolo-
gists and those that pass the set rules can be directly
released by the computer.6,7,9,20,48 Although the auto-
verification of clinical tests results is an essential tool
with which to increase the accuracy and efficiency in
clinical laboratories,6,48 there is relatively scant infor-
mation on its practical applications within a clinical
laboratory setting.18,20 In this paper, we described the
establishment and application of autoverification in a
busy clinical chemistry core laboratory.
Thyroid function and sex hormone tests were chosen
for the initial evaluation of the rules because they are
tests that have a complex, but defined relationship with
Li et al.
Table 5. Degree of agreement between the autoverification rules and two senior technicians.
Autoverification rules
Reviews Pass Fail
Two senior Pass 61484 123
technicians Fail 49 15744
a
Degree of agreement: <0.2 poor; 0.2–0.4 fair; 0.41–0.6 moderate; 0.61–0.8 good; 0.81–1.0 very good.
b
Highly significant difference, indicating the agreement between the autoverification rules and two senior technicians was not caused by accident.
other system functions. Furthermore, they have clearly
defined reference intervals and well-defined cut-off
values for certain clinical abnormalities, and they rep-
resent the main bulk (85%) of hormone cases in our
laboratory.
As shown in Figure 1, the autoverification rules util-
ize the following parameters: (1) IQC check; (2) initial
screening rules check (including instrument error flags,
critical value and out of analytical measurement range);
(3) limit range; (4) consistency check; (5) delta check.
The IQC check confirms whether each item falls within
the acceptable range before the autoverification rules
start; otherwise, the verification procedure will be
halted according to the preconditions for autoverifica-
tion.9 The IQC system should be integrated with the
autoverification rules, e.g. when IQC failure occurs,
the autoverification rules of the given test items will
be automatically stopped to enable a qualified person
to analyse the reasons for the failure and correct the
problems. Then, once IQC is within limits, the autover-
ification rules would be restarted.
The critical value is important for enabling phys-
icians to make a diagnosis. To the best of our know-
ledge, it is unusual to find published data on the critical
values for autoverification. Our process allows the
autoverification of critical value to proceed as long as
no other rules are violated. In addition, critical values
still need to be communicated to the clinical locations
from which the requests originated. However, the com-
munication of these results by the laboratory is facili-
tated by clinicians often having access to results, e.g.
following their display on ward-based terminals, that
have undergone autoverification, prior to any tele-
phone call being made.
Because test results fluctuate substantially during
acute illness, it was difficult to perform a consistency
check on each item, and only portions of the test items
were confirmed based on practical and clinical diagnos-
tic criteria. By using the consistency rule check, we
would avoid most of the base errors. However, the
delta check could help us quickly find changes in a
patient’s condition and identify any preanalytical errors.
We chose the total agreement between the two pro-
fessional reviewers as a benchmark to maximize patient
261
Kappa approximate
Agreement Disagreement Kappa value significance
99.78% 0.22% 0.99a P ¼ 0.00b
(77,230) (170)
safety. A similar method was applied in the VALAB
system.11 There was a significant degree of consistency
between our autoverification rules and the manual veri-
fication undertaken by two senior technicians (kappa
value ¼ 0.99, P < 0.001), and consistency was observed
in 99.78% of cases. There were only five cases when the
two reviewers decided to stop autoverification due to
any inconsistency. These cases involved thyroid func-
tion tests in which the related rules stated that if the
TSH is <0.01 mIU/L, TT4 is within limit range, and
TT3 is above the limit range, one should autoverify and
comment (possible T3 toxicosis). A high TSH alone
was found in 29 cases; 34 cases had a T4 assay within
normal limits and another 24 cases had both T3 and T4
within normal limits with a high TSH. In the previously
mentioned 87 cases, the TSH values were below a five-
fold increase in the upper reference limit (URL), and no
clinical data were provided to support the diagnosis.
According to previous research,29 we did not verify
those cases because the TSH value did not diagnose thy-
roid disease with certainty. However, they were verified
manually by the reviewers. Therefore, despite these dis-
crepancies, we did not believe it necessary to change any
area of the autoverification rules to achieve the max-
imum possible patient safety. The biggest risk involved
in using the rules is that some test results may be released
without proper review or editing. The interpretation of
sex hormone results is considered by some to be more
complicated than the interpretation of thyroid function
tests, and the hormone concentrations can change due to
various factors, e.g. the menstrual cycle, ART.27,28
Therefore, we believe that caution is needed in the inter-
pretation of test results for sex hormones.
Most publications describe autoverification rules as
an approach to shortening the TAT in reporting test
results, through reduction in manual intervention,
which in turn can facilitate staffing reductions in clinical
laboratories.4,20 In our laboratory, the implementation
of the rules resulted in the staffing WTE numbers in the
Clinical Chemistry Core Laboratory falling from 8 to 6.
Furthermore, because results that undergo autoverifica-
tion are released immediately, the TAT of patient
reports was dramatically reduced in comparison to
those obtained previously with manual verification.
262
Our experience has suggested that the TAT is the
parameter that is the most affected by the introduction
of autoverification rules. However, this improvement
was reduced when comparing the average TAT values
for the year. Our reduction in the TAT was 54.55%.
This fall in improvement was due to the simultaneous
increase in the average daily workload for hormone
analysis in the clinical chemistry core laboratory from
51,367 samples to 77,400 samples (a 150.68% increase).
However, we believe that the TAT will further improve
following the full utilization of all the established pro-
grammed rules. McFadden49 indicated that with the
application of autoverification rules, there could be
up to a 44% savings in time and labour capacity for
the laboratory staff. We found that the autoverification
rate obtained using historical patient data was 77.06%;
this is close to that of the VALAB system which has a
mean autoverification success of approximately 50% to
90%.11 However, VALAB was applied on a wider
range of tests. Our success rates are lower than that
of the DNSevTM system, which shows a verification
rate of approximately 80%.24 These differences could
be explained by differences in the studied group of tests,
patient presentations and laboratory equipment used.
In our study, the autoverification rates of sex hormone
single tests were lower. For example, the autoverifica-
tion rates of oestradiol and LH were 76.65% and
72.97%, respectively. We initially wanted to include
menstrual status and pregnancy in a limit range.
However, due to time constraints, this was not possible
and it is our aim to develop this area in the future when
time permits.
Because autoverification rules are relatively new to
laboratory test result reporting, the scientific literature
on the subject is limited. In this study, we found that the
greatest benefit following the implementation of the
rules is in the consistency of the test results, as previously
observed.4,7,20 In contrast, despite the advantages of
autoverification, potential disadvantages could include
an increase in false-negative results and the reporting of
erroneous results due to analytical interference, such as
when a partial or tiny clot is not detected by the instru-
ment. Mechanical errors of this kind, although few,
remain unavoidable. Our study also has some limita-
tions. The limited connectivity between the LIS and
HIS resulted in the disabling of any rules associated
with any patient clinical information or drug history,
which prevented us from taking full advantage of the
programmed rules. Although we could make a connec-
tion with the keywords of the diagnosis, this work-
around approach is insufficient. For the time being, we
can only overcome this drawback by registering the pre-
viously established comprehensive patient sheets.
In conclusion, we developed and implemented a
rule-based autoverification, according to Boolean
Annals of Clinical Biochemistry 55(2)
logic and the guidelines of ‘Design of Algorithms’ in
the Auto-10A document that utilizes clinical tests
data and other parameters. The autoverification rate
of the rules was comparable with that obtained using
commercially available software for other systems. The
rules that we designed can shorten the TAT, reduce
manual data entry, and decrease the probability of
errors associated with human review, as well as, min-
imize the number of samples requiring technologist
intervention. However, there was a small sample size
in the current study and the implementation time was
short (only three months). In the future, we will con-
tinue implementing autoverification rules for thyroid
function profiles and sex hormones, and add other
tests to construct a more complete expert system.
Acknowledgements
We thank Yongxin Qiu, from Beckman Coulter Inc., for computer and DM2
technical support and Linli Fang and Yingqiu Zu, the senior technicians in the
Clinical Chemistry Core Laboratory, for result verification. Additionally, we
thank Dr. Lin, a professor of Shantou University Medical College (SUMC), for
his review of the manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the
research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or
publication of this article.
Ethical approval
Not applicable.
Guarantor
JL and YC.
Contributorship
All authors reviewed and edited the manuscript and approved the final version
of the manuscript. Due care has been taken to ensure the integrity of the work.
References
1. Guidi GC and Lippi G. Laboratory medicine in the 2000s: programmed
death or rebirth? Clin Chem Lab Med 2006; 44: 913–917.
2. Panteghini M. The future of laboratory medicine understanding the new
pressures. Clin Biochem 2004; 25: 207–215.
3. Valdlgulie PM, Rogari E and Pilippe H. VALAB: expert system for valid-
ation of biochemical data. Clin Chem 1992; 38: 83–87.
4. Torke NBL, Nguyen T, Perri A, et al. Process improvement and operational
efficiency through test result autoverification. Clin Chem 2005; 51:
2406–2408.
5. Crolla LJ and Westgard JO. Evaluation of rule-based autoverification
protocols. Clin Leadersh Manag Rev 2003; 17: 268–272.
6. Guidi GC, Poli G, Bassi A, et al. Development and implementation of an
automatic system for verification, validation and delivery of laboratory test
results. Clin Chem Lab Med 2009; 47: 1355–1360.
7. Li J, Cheng B, Yang L, et al. Development and implementation of autover-
ification rules for ELISA results of HBV serological markers. J Lab Autom
2016; 21: 642–651.
Li et al.
8. College of American Pathologists. Laboratory Accreditation Program,
Laboratory General Checklist. Commission on Laboratory
Accreditation, College of American Pathologists, 2013.
9. CLSI. AUTO10-A. Wayne, PA: CLSI, 2006.
10. Oosterhuis WP, Ulenkate H and Goldschmidt H. Evaluation of
LabRespond, a new automated validation system for clinical laboratory
test results. Clin Chem 2000; 46: 1811–1817.
11. Prost L and Rogari E. How autoverification through the expert system
VALAB can make your laboratory more efficient. Accred Qual Assur
2002; 7: 480–487.
12. Jones JB. A strategic informatics approach to autoverification. Clin Lab
Med 2013; 33: 161–181.
13. GM D. Autoverification of macroscopic urinalysis. Lab Med 1999; 33:
56–60.
14. Stankovic AK and DiLauri E. Quality improvements in the preanalytical
phase: focus on urine specimen workflow. Clin Lab Med 2008; 28: 339–350.
15. Smith NRD and Watman R. Hematology autovalidation system. Lab Hem
1999; 5: 52–55.
16. Butch SH. Automation in the transfusion service. Immunohematology 2008;
24: 86–92.
17. Shih MC, Chang H, Tien N, et al. Building and validating an autoverifica-
tion system in the clinical chemistry laboratory. Lab Med 2011; 42:
668–673.
18. Gómez-Rioja R, Alvarez V, Ventura M, et al. Current status of verification
practices in clinical biochemistry in Spain. Clin Chem Lab Med 2013; 51:
1739–1746.
19. Sediq AM and Abdel-Azeez AG. Designing an autoverification system in
Zagazig University Hospitals Laboratories: preliminary evaluation on thy-
roid function profile. Ann Saudi Med 2014; 34: 427–432.
20. Krasowski MD, Davis SR, Drees D, et al. Autoverification in a core clin-
ical chemistry laboratory at an academic medical center. J Pathol Inform
2014; 5: 1–17.
21. Davis GM. Rules-based system for autoverification of routine coagulation
testing. Lab Hem 2008; 30: 58–147.
22. Zhao Y, Yang L, Zheng G, et al. Building and evaluating the autoverifica-
tion of coagulation items in the laboratory information system. Clin Lab
2014; 60: 143–150.
23. Fuentes-Arderiu X, Castineiras-Lacambra MJ and Panadero-Garcia MT.
Evaluation of the VALAB expert system. Eur J Clin Chem Clin Biochem
1997; 35: 711–714.
24. Dorizzi RM, Caruso B, Meneghelli S, et al. The DNSevTM expert system in
the auto-verification of tumour markers and hormones results. Accred Qual
Assur 2006; 11: 303–307.
25. Roef G, Taes Y, Toye K, et al. Heredity and lifestyle in the determination
of between-subject variation in thyroid hormone levels in euthyroid men.
Eur J Endocrinol 2013; 169: 835–844.
26. Yang YM and Choi EJ. Efficacy and safety of metformin or oral contra-
ceptives, or both in polycystic ovary syndrome. Ther Clin Risk Manage
2015; 11: 1345–1353.
27. Ortega-Hrepich C, Polyzps NP, Anckaert E, et al. The effect of ovarian
puncture on the endocrine profile of PCOS patients who undergo IVM.
Reprod Biol Endocrinol 2014; 12: 1–6.
28. El-Talatini MR, Taylor AH and Konje JC. Fluctuation in anandamide
levels from ovulation to early pregnancy in in-vitro fertilization-embryo
transfer women, and its hormonal regulation. Hum Reprod 2009; 24:
1989–1998.
263
29. Demers LM, S. C. Tietz textbook of clinical chemistry and molecular diag-
nostics, 4th ed. St Louis: Elsevier Saunders, 2006, pp.2053–2095.
30. Vadiveloo T, Donnan PT, Murphy MJ, et al. Age- and gender-specific
TSH reference intervals in people with no obvious thyroid disease in
Tayside, Scotland: the Thyroid Epidemiology, Audit, and Research
Study (TEARS). J Clin Endocrinol Metab 2013; 98: 1147–1153.
31. Zhang Y, Post WS, Cheng A, et al. Thyroid hormones and electrocardio-
graphic parameters: findings from the third national health and nutrition
examination survey. PloS One 2013; 8: e59489.
32. Melmed S, Casanueva FF, Hoffman AR, et al. Diagnosis and treatment of
hyperprolactinemia: an Endocrine Society clinical practice guideline. J Clin
Endocrinol Metab 2011; 96: 273–288.
33. Levey S and Jennings ER. The use of control charts in the clinical labora-
tory. Am J Clin Pathol 1950; 20: 1059–1066.
34. Westgard JO, Barry PL, Hunt MR, et al. A multi-rule Shewhart chart for
quality control in clinical chemistry. Clin Chem 1981; 27: 493–501.
35. Westgard JO. Internal quality control: planning and implementation stra-
tegies. Ann Clin Biochem 2003; 40: 593–611.
36. Hayashi S, Ichihara K, Kanakura Y, et al. A new quality control method
based on a moving average of ‘‘Latent Reference Values’’ selected from
patients, daily test results. Rinsho Byori 2004; 52: 204–211.
37. Statland BE. Clinical decision levels for lab tests. Ann Arbor, MI: Medical
Economics Books, 2008.
38. Fraser CG, Stevenson HP and Kennedy IMG. Biological variation data are
necessary prerequisites for objective autoverification of clinical laboratory
data. Accred Qual Assur 2002; 7: 455–460.
39. Garner AE, Lewington AJ and Barth JH. Detection of patients with acute
kidney injury by the clinical laboratory using rises in serum creatinine:
comparison of proposed definitions and a laboratory delta check. Ann
Clin Biochem 2012; 49: 59–62.
40. Ovens K and Naugler C. How useful are delta checks in the 21 century? A
stochastic-dynamic model of specimen mix-up and detection. J Pathol
Inform 2012; 3: 5.
41. Strathmann FG, Baird GS and Hoffman NG. Simulations of delta check
rule performance to detect specimen mislabeling using historical laboratory
data. Clin Chim Acta 2011; 412: 1973–1977.
42. Chakraborty S, Bhattacharya C, Das S, et al. Delta check: a must in the
management of hyponatremia. Clin Chem Lab Med 2013; 51: e161–e162.
43. Yamashita T, Ichihara K and Miyamoto A. A novel weighted cumulative
delta-check method for highly sensitive detection of specimen mix-up in the
clinical laboratory. Clin Chem Lab Med 2013; 51: 781–789.
44. Lok AS. Chronic hepatitis B. N Engl J Med 2002; 346: 1682–1683.
45. Torbenson M and Thomas DL. Occult hepatitis B. Lancet Infect Dis 2002;
2: 479–486.
46. Tran DV, Cembrowski GS, Lee T, et al. Application of 3-D Delta check
graphs to HbA1c quality control and HbA1c utilization. Am J Clin Pathol
2008; 130: 292–298.
47. Duco DJ. Autoverification in a laboratory information system. Lab Med
2002; 33: 21–25.
48. Torke N, Boral L, Nguyen T, et al. Process improvement and operational
efficiency through test result autoverification. Clin Chem 2005; 51:
2404–2406.
49. S, M., McFadden S. Evaluation of autoverification system which used in
our laboratory. Adv Admin Lab 2006; 15: 64.