HHS Public Access

Author manuscript
J Immunol Methods. Author manuscript; available in PMC 2021 November 16.
Author Manuscript

Published in final edited form as:

J Immunol Methods. 2017 September ; 448: 80–84. doi:10.1016/j.jim.2017.05.012.

A novel Python program for implementation of quality control in

ELISA
Hanna N. Wetzel, Cinder Cohen, Andrew B. Norman, Rose P. Webster
Department of Pharmacology and Cell Biophysics, University of Cincinnati College of Medicine,
Cincinnati, Ohio 45267-0575, USA
Author Manuscript

Abstract
The use of semi-quantitative assays such as the enzyme-linked immunosorbent assay (ELISA)
requires stringent quality control of the data. However, such quality control is often lacking in
academic settings due to unavailability of software and knowledge. Therefore, our aim was to
develop methods to easily implement Levey-Jennings quality control methods. For this purpose,
we created a program written in Python (a programming language with an open source license)
and tested it using a training set of ELISA standard curves quantifying the Fab fragment of an
anti-cocaine monoclonal antibody in mouse blood. A colorimetric ELISA was developed using
a goat anti-human anti-Fab capture method. Mouse blood samples spiked with the Fab fragment
were tested against a standard curve of known concentrations of Fab fragment in buffer over a
period of 133 days stored at 4°C to assess stability of the Fab fragment and to generate a test
dataset to assess the program. All standard curves were analyzed using our program to batch
Author Manuscript

process the data and to generate Levey-Jennings control charts and statistics regarding the datasets.
The program was able to identify values outside of two standard deviations, and this identification
of outliers was consistent with the results of a two-way ANOVA. This program is freely available,
which will help laboratories implement quality control methods, thus improving reproducibility
within and between labs. We report here successful testing of the program with our training set
and development of a method for quantification of the Fab fragment in mouse blood.

Keywords
ELISA; quality control; Python; Levey-Jennings control charts

1. Introduction
Author Manuscript

Implementation of appropriate and robust quality control processes is commonplace in

the clinical laboratory where a false positive or false negative result may have serious
consequences. The basic premise of all quality control methods is to verify that the assay is

Address correspondence to: Rose P. Webster, Department of Pharmacology and Cell Biophysics, University of Cincinnati, College of
Medicine, 231 Albert Sabin Way, Cincinnati, OH 45267-0575., websterp@ucmail.uc.edu, Tel: (513)558-2371.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our
customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review
of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be
discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
 Wetzel et al. Page 2

working within statistical control, where some degree of natural variability is unavoidable.
Author Manuscript

However, a good quality control process should be able to identify when an assay is out of
statistical control due to pipetting errors, a bad reagent, analyst error, etc. Commonly used
quality control tools include Shewhart type and Levey-Jennings control charts to monitor
most processes (Shewhart, 1931; Westgard et al., 1977). The fundamental idea behind these
control charts is that all control values will fall between specific control limits. Assuming a
Gaussian distribution, 68% of all observed values should be within one standard deviation of
the mean (1SD), 95% of all values should fall within plus or minus two standard deviations
of the mean (2SD) and 99.7% within plus or minus three standard deviations of the mean
(3SD) (Westgard et al., 1977). Extending this, it would be highly unlikely (0.3%) that a
value would fall outside 3SD of the mean. Therefore, if a value did fall outside of 3SD,
it likely indicates a methodological issue. The statistician, Walter A. Shewhart, made the
original recommendation using scientific methods to calculate the mean for a group and
Author Manuscript

use control charts to assess a dataset. This was later expanded by Levey- Jennings and
Westgard, who ultimately refined these methods for use in clinical laboratories. Control
charts graphically display if a particular measurement is in control or out of control. These
charts display the mean of a dataset of approximately 20 measurements of control samples
as the central line as well as upper and lower limit lines representing either 2SD or 3SD.

Based on multi-rule controls, deviations outside of 2SD or 3SD become actionable, resulting
either in run rejection or careful method inspection to identify systematic errors due to slow
reagent decay (Westgard and Barry, 1986). Westgard rules generally utilize a multi-level
control system wherein the 12S triggers a warning to carefully inspect the data. 12S
represents a situation when quality control value exceeds 2 SD. The next set of rules are
22s (reject when 2 consecutive control values exceed same mean +− 2SD), 13s (when a
control exceeds 3 SD -run rejection), R4s (when 1 control value in a group exceeds +2SD
Author Manuscript

and another exceeds −2SD, reject when within the same run), 4 1s (when 4 consecutive
readings exceed same mean +1s or mean −1s control limit). It is important to understand
that when deviation from 2SD, i.e. 12S alone is used to reject a run and to repeat it, this
could result in unnecessary rejections, creating a false alarm situation. Error detection is
maximized by a judicious selection of a combination of these rules when a warning is
triggered. This approach will enable detection of random and systematic errors, at the same
time avoiding expensive repeats of the assay. In the clinical laboratory, most instruments
have either inbuilt control checking or link to a laboratory information system that can
check for process performance. This is less common in academic laboratories where basic
research is conducted. However, appropriate quality control processes are important for the
characterization of therapeutic proteins discovered in translational research laboratories,
since these compounds could be ultimately used in human subjects. Therefore, freely
Author Manuscript

accessible open source tools and methods that allow for easy implementation of these
controls without excessive repeat testing or the purchase of additional expensive software
in an academic setting are desirable. The use of open source software over proprietary
programs also helps facilitate reproducibility between labs. Python represents an ideal
language to develop such a platform, as it is readily available, open-source, well-supported
with internal and external libraries, and runs on most operating systems (Bassi, 2007).

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 3

Herein, we seek to develop such a set of quality control tools. These tools are tested using
Author Manuscript

the development of an ELISA to detect a novel monoclonal anti-cocaine Fab fragment that
is in preclinical development for the treatment of cocaine overdose (Kirley and Norman,
2015). This provides an excellent test dataset, as it represents an example of a drug that has
been developed in an academic setting, but will eventually be translated into the clinic. We
have generated a dataset exploring the variability of this ELISA, as well as a test dataset
on the stability of the Fab fragment in mouse blood. Overall, we present a novel open
source program for generating control charts for ELISA data that will assist in developing
appropriate quality control methods.

2. Materials and Methods

2.1 Sample collection
A set of samples investigating the stability of the Fab fragment in mouse blood was
Author Manuscript

generated as a data set with which to test the quality control program. All animal studies
were conducted in accordance with the National Institutes of Health Guide for the Care and
Use of Laboratory Animals (Institute for Laboratory Animal Research, 2011) and under a
protocol approved by the Institutional Animal Care and Use Committee at the University
of Cincinnati. Adult Swiss-Webster mice (19–22 g) were purchased from Taconic Farms.
Mice (n=3) were housed individually on a 14/10- hour light/dark schedule with free access
to food and water. Blood (10 μl) was collected using heparinized capillary pipette tips by
making a small incision at the tip of the tail. Blood was first diluted in 90 μL of citrate buffer
(pH 4) (1:10) into a 0.5 % BSA-Tris buffer (0.5% BSA in 10 mM Tris, 140 mM NaCl, and
0.02% NaN3, pH 7.2) and spiked with 9.6 μg/mL of the anti-cocaine Fab antibody. Diluted
blood (final 1:100) spiked with the Fab fragment (final 1:2) was stored in the refrigerator at
4°C. Collection and dilution of blood, as well as the ELISA capture method will be used in
Author Manuscript

pharmacokinetic studies of the Fab fragment injected into mice in the future.

2.2 ELISA
A modification of the previously described ELISA (Paula et al., 2004) was used to generate
standard curves using known concentrations of Fab fragment. Goat anti-human Fab region­
specific IgG antibodies were obtained from Bethyl Laboratories, Durham, NC. These
antibodies (100 μl/well) at a concentration of 4 μg/ml in 1 mM Tris-EGTA, pH 7.4 were
adsorbed onto the 96-well microtiter plates and incubated for 1 hour. The plates were then
washed twice, and all wells were blocked for 15 min with BSA-buffer to block nonspecific
protein binding. Next, 100 μl/well over a concentration range of 0–0.4 μg/ml of either
purified Fab fragment in buffer or blood samples spiked with the Fab fragment was applied
and incubated for 1 hour. The plates were washed with BSA-PBS buffer (0.5% BSA, 10
Author Manuscript

mM sodium phosphate, 145 mM NaCl, 1.5 mM MgCl2, 0.05% Triton X-100, and 0.02%
NaN3, pH 7.2) twice, and 50 μl/well biotin-labeled goat anti-human IgG (Fab′)2 (Abcam
Laboratories, Cambridge MA) diluted 1:500 BSA-PBS buffer was added and incubated for
1 hour. After washing three times with the same solution, 50 μl/well streptavidin-alkaline
phosphatase conjugate (diluted 1:200 in BSA-PBS buffer), was added, incubated for 1
hour, and washed with BSA-PBS buffer. Then, 50 μl/well of the substrate para-nitro-phenyl­
phosphate (1 mg/ml) in substrate buffer (50 mM Na2CO3, 50 mM NaHCO3, and 1 mM

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 4

MgCl2, pH 9.8), was added. After 6 to 8 min, the reaction was stopped with 1 M sodium
Author Manuscript

hydroxide (50 μl/well). All steps were performed at room temperature. The optical density
(OD) was measured with a SpectraMax M3 Multi-Mode Microplate Reader (Molecular
Devices, CA) at a wavelength of 405 nm.

2.3 Novel Python program for implementation of quality control methods

A custom program was written in Python to generate Levey-Jennings control charts and
a table of statistics about the standard curves. The program uses Numpy’s genfromtxt
function to import the data from the user specified rows from any files that contain the
tag entered by the user (see Instruction Manual for full instructions, posted on Github at
https://github.com/hanna133/ELISA_QC). It then slices these arrays every three values to
account for the triplicates, and takes the mean of these triplicates. Next, it calculates the
mean and standard deviation for each concentration from all of the included standard curves
Author Manuscript

using Numpy’s mean and standard deviation functions. The coefficient of variance (CV) is
also calculated as (SD/mean)*100. The graphs containing Levey-Jennings plots are created
by the program, showing each concentration as a separate plot including a central line for
the mean, and lines above and below mean representing ±2 and ±3 standard deviations using
Matplotlib. A .csv file containing the statistics about the curves is written and saved using
the csv writer function. In the terminal, a key including what number on the x-axis of the
subplots corresponds to what file name, as well as a guide to which curves fell outside of
2SD and 3SD is displayed. The graphical user interface (GUI) was generated using Tkinter,
which is included in the standard Windows and Mac install. For this to function, files must
be saved as “.csv” (case-sensitive, and must be included in the file name). Each standard
curve must be in a separate file. Tags may be included at the end of the file name to indicate
which curves should be treated as controls, and which are considered tests. The source code,
Author Manuscript

along with a users guide with full instructions can be found on Github (https://github.com/
hanna133/ELISA_QC), or using the DOI 10.7945/C2DK67. The user guide was based on
that found in Naylor et al. (2017).

2.4 Positive control for methodological error identification

In order to test this program’s ability to identify methodological errors, a standard curve
was generated using two-fold less (1:1000 instead of 1:500) than the standard concentration
of biotinylated secondary antibody. This was used to mimic a reagent error that could
occur without investigator knowledge. This curve was included in the standard analysis, and
treated as a control curve to simulate a situation where a mistake was unknowingly made
and included in the dataset.
Author Manuscript

2.5 Generation of test data sets and test of stability of the Fab fragment in mouse blood
All data was initially analyzed with the Softmax Pro software (Molecular Devices, CA).
Standard curves were generated using a rectangular hyperbola non-linear regression model
over a range of 0 to 0.4 μg/ml of Fab fragment. Our aims from this test set of experiments
was to 1) examine the variability of the standard curve to demonstrate how to arrive at
a laboratory mean, 2) to evaluate the stability of the diluted Fab fragment in blood over
time, and 3) test the ability of our program to identify outliers. ELISAs quantifying Fab
fragment were run on days 1, 4, 10, 18, 24, 60, 88, and 133 days after blood collection,

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 5

for a total of 24 curves in mouse blood. Additionally, each day, two standard curves with
Author Manuscript

the Fab fragment in buffer were run by the same operator (total of 16 curves in buffer).
Several types of reagent blanks such as the antigen, BSA and EGTA alone were run on
each plate to both ensure no cross contamination and check for any possible cross reactivity
(data not shown). Mice (n=10) blood without the Fab fragment were also tested in separate
experiments to ensure absence of cross reactivity with any of the antibodies used in the
assay (data not shown). The standard curves in buffer were entered into the program as
control data, and the curves in mouse blood were entered as test data. The program created
Levey-Jennings control charts and identified any values outside of 2SD. The samples in
mouse blood normalized to their respective OD measured in buffer on the same plate were
also compared using a two-way ANOVA with repeated measures.

3. Results
Author Manuscript

3.1 Control dataset

Control charts were generated using the standard curves in buffer as the included control
group, and the standard curves in blood as the excluded test group. The program was able to
rapidly (under 1 second) generate charts for these data, which included a total of 40 files (16
included as controls and 24 excluded as test curves). In the control set, one value (#11, day
24) fell outside 2SD at the 0.02 μg/mL concentration, and a different value (#15, day 133)
fell outside 2SD in the blank (Fig. 1, black symbols). No values fell outside of 3SD. Based
on Westgard’s rules, it would be recommended that the investigator check these datasets and
possibly repeat them, though since the effect is not systematic (over many concentrations
along the curve or across many days), these curves may still be included if no errors were
found in the methods, since they did not fall outside of 3SD and only occurred once. They
are included here for demonstration purposes. The mean, SD, and CV at each concentration
Author Manuscript

for this dataset was also generated by the program, and is shown in Table 1. These could be
used to set laboratory means and define limits for standard deviations.

3.2 Positive control for methodological error identification

This method was able to successfully identify the standard curve that was generated using
a lower concentration of biotinylated secondary antibody (Fig 2). This is further positive
proof of principle, demonstrating the program’s ability to identify problems. This curve
was identified as falling outside of 2SDs at multiple points along the curve, particularly at
the higher concentrations. Since the higher concentrations have been observed to show less
variability (lower CVs) it is especially relevant when data falls outside of the 2SDs at these
points (Fig 2).
Author Manuscript

3.3 Test data sets and test of stability of the Fab fragment in mouse blood
In the test set (samples that had been stored in mouse blood at 4°C for varying amounts
of time), many values fell outside of the 2SD established by the control set. Specifically,
two values fell outside at the 0.4, 0.2, and 0.1 μg/mL concentrations, and 3 fell outside
at the 0.05, 0.02, 0.01, 0.005, and blank concentrations. Of these, at the 0.1, 0.05, and
blank concentrations, points fell outside of 3SD (Fig. 1, magenta points). These values were
identified by the Python program and can be easily seen on the control charts (Fig. 1).

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 6

All of these outliers came from three different days (24, 60, 133 days). Two of these days
Author Manuscript

correspond to the two control curves that also fell outside of 2SD. A two-way ANOVA
was applied to OD values obtained in blood normalized to their respective OD measured in
buffer on the same plate. The ANOVA was consistent with the 2SD method at identifying
values that differed. No difference was found between any of the days at the highest three
concentrations. At the lower concentrations on three days (the same that were identified
using the 2SD rule) values were identified as significantly different.

Discussion
A program written in Python was successfully tested on a dataset used to both develop
an ELISA to detect a humanized anti-cocaine Fab fragment in blood and to train a new
operator. These values could also be used to establish a laboratory mean and SD. This
method was able to successfully identify errors. A scenario was designed to mimic a
Author Manuscript

situation where the investigator accidentally made an error in the preparation of one of their
reagents. It is important that any quality control method be able to identify such problems to
allow for the exclusion and/or repetition. A standard curve where the concentration of one
of the reagents was incorrect was included in the analysis, and this curve was identified as
being the only one outside of 2SD at all concentrations tested when included in a set of 16
control datasets. If this were a real-life dataset, the investigator would have a quantitative
rationale for critically examining this data and would easily identify the error. Thus, this
program provides an easy to utilize application of the Levey-Jennings control chart method.

The stability of the Fab fragment in mouse blood was used to demonstrate the program’s
ability to analyze a test dataset against an established standard set. Here, the 16 standard
curves of Fab fragment in BSA were treated as a control set, while the Fab fragment spiked
Author Manuscript

into mouse blood and stored in the refrigerator was the test set. The control charts identified
three days that contained values outside of 2SD at various concentrations. These results
were consistent with an ANOVA, which is a traditional and trusted method for identifying
values that fall outside of a distribution. This further confirms the ability of this method to
identify outliers quickly and easily without the need for extensive data formatting, advanced
statistics, which require time, knowledge, and commercial software that can be difficult to
use. Based on this data, we can conclude that the Fab fragment is stable in mouse blood
for at least 18 days at 4°C, since the first significantly different point was observed after
18 days. However, it is likely that the fragment is stable even longer since on a later day
(84 days), no difference was detected by ANOVA and the values were within 2SD. This
highlights the ability of this program to provide insights into test datasets when compared
to established controls that are consistent with more advanced statistical tests. This method
Author Manuscript

has applications in many situations other than testing stability as demonstrated here. In
this use case, the traditional LJ chart used to apply multi-rule control was not used to test
quality control samples, although they were generated in the same way. Instead, this method
was used to compare a control dataset with a test dataset (blood spiked with antibody) to
examine changes with time during storage. For future pharmacokinetic studies, the stability
of the antibody in blood is an important issue, as immediate sampling may not be possible
after collection. It was also used to test the program’s ability to identify an intentional
method error. We were able to successfully evaluate stability of the test sample and also

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 7

to identify reagent failure in the simulated method error. It can be applied to training and
Author Manuscript

testing new operators, validating changes in methods, or looking at a new biological matrix
(blood, urine, tissue homogenates).

The program presented herein provides a tool for investigators to easily, yet quantitatively,
compare multiple concentrations between a test and control set to evaluate variability.
Since it is written in Python, it is easily accessible to anyone. Python is a community
developed, non-compiled, easy to learn language that runs on most operating systems.
It is a high-level language with readable syntax and many internal and external libraries
for scientific computing, making this program amenable to adaptation (Bassi, 2007). The
inherent features of the Python language make it an ideal platform for the development
of the program presented here and in the future a suite of laboratory control tools. It
provides a solution for the common practice of making data exclusion decisions based on
individual visual examination of the generated plots of OD vs. concentration, which could
Author Manuscript

lead to erroneous decisions while visually comparing multiple plots of ELISA data. The
program enables objective application of Levey-Jennings quality control methods easily,
freely and without the need for unnecessary repeat testing. It also provides each operator
the ability to evaluate their own datasets to independently identify issues. It represents a
time-saving mechanism for small labs where all the ELISA data would otherwise have to
be exported and analyzed curve by curve after the long process of doing the assay itself.
Batch processing of the files also eliminates errors that could occur due to excessive manual
data handling and reformatting. Investigators are able to, with minimal formatting, batch
process their ELISA standard curves to generate quality control charts and statistics. Since
this program is open source, most laboratories can use this as an add-on to the experiments
they already routinely perform. We believe that this provides a novel and useful tool in
helping laboratories easily develop and implement quality control methods and validate test
Author Manuscript

methods, which will facilitate the reproducibility of ELISA.

Acknowledgments
The authors would like to thank Fatima Saeed for technical assistance and helpful discussion and William J. Ball
for advice on the ELISA. This work was supported by the National Institutes of Health, National Institute on Drug
Abuse grant number U01DA039550.

Citations
Bassi S. 2007; A primer on python for life science researchers. PLoS computational biology. 3 :e199.
[PubMed: 18052533]
Institute for Laboratory Animal Research. Guide for the care and use of laboratory animals. National
Research Council of the National Academies; 2011.
Author Manuscript

Kirley TL, Norman AB. 2015; Characterization of a recombinant humanized anti-cocaine monoclonal
antibody and its Fab fragment. Human vaccines & immunotherapeutics. 11 :458–67. [PubMed:
25692880]
Naylor BC, Porter MT, Wilson E, Herring A, Lofthouse S, Hannemann A, Piccolo SR, Rockwood
AL, Price JC. 2017 DeuteRater: a Tool for Quantifying Peptide Isotope Precision and Kinetic
Proteomics. Bioinformatics.
Paula S, Tabet MR, Farr CD, Norman AB, Ball WJ Jr. 2004; Three-dimensional quantitative structure­
activity relationship modeling of cocaine binding by a novel human monoclonal antibody. Journal of
medicinal chemistry. 47 :133–42. [PubMed: 14695827]

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 8

Shewhart, W. Economic Control of Quality of Manufactured Products. Van Hostrand Company Inc;
1931.
Author Manuscript

Westgard, JO, Barry, PL. Cost-Effective Quality Control: Managing the Quality and Productivity of
Analytical Processes. CD: AACC Press; 1986.
Westgard JO, Groth T, Aronsson T, Falk H, de Verdier CH. 1977; Performance characteristics of rules
for internal quality control: probabilities for false rejection and error detection. Clinical chemistry.
23 :1857–67. [PubMed: 902411]
Author Manuscript
Author Manuscript
Author Manuscript

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 9

Highlights
Author Manuscript

• A python program was developed to rapidly screen ELISA data for quality
control

• This method was applied to the quantification of a Fab fragment of anti­

cocaine mAb in blood
Author Manuscript
Author Manuscript
Author Manuscript

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 10
Author Manuscript
Author Manuscript
Author Manuscript

Fig 1.
A Levey-Jennings control chart comparing standard curves generated from Fab fragment
in buffer (black) and curves generated in mouse blood (magenta) stored at 4°C for various
amounts of time. The top plot shows all optical densities from all concentrations in both
groups. The subplots below show each concentration (μg/mL) displayed separately, where
the x-axis represents the curve number (corresponding to a key that will print in the
terminal), and the y-axis represents the individual measured optical densities.
Author Manuscript

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 11
Author Manuscript
Author Manuscript
Author Manuscript

Fig. 2.
A Levey-Jennings control chart generated from 17 standard curves, where a standard curve
with 2x less biotinylated secondary antibody was generated. It was included in a trusted
data set of 16 other standard curves to mimic what would happen if an investigator
unintentionally used the wrong amount of some reagent. The values outside of 2SD are
circled in orange. It falls outside of 2 SD at the highest 4 concentrations, which are the most
reliable (lowest CV). This curve was the only one to fall outside 2SD at any concentration.
Author Manuscript

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.

 Wetzel et al. Page 12

Table 1

Statistics, generated by the Python program, from the control set of 16 standard curves, including the mean
Author Manuscript

optical density, the standard deviation, and the coefficient of variance for each concentration on the standard
curve. This output is automatically created and saved by the program.

Concentration (μg/mL) Mean OD SD CV

0.4 1.33 0.17 13.03

0.2 1.28 0.17 12.94

0.1 1.18 0.16 13.71

0.05 1.00 0.16 16.22

0.02 0.61 0.15 24.52

0.01 0.33 0.11 33.34

0.005 0.15 0.06 41.50

Blank −0.01 0.01 −201.48

Author Manuscript
Author Manuscript
Author Manuscript

J Immunol Methods. Author manuscript; available in PMC 2021 November 16.