1040
Method Development and Evaluation
Reviewed by Standard Methods Committee, 2014. Editorial revisions, 2021. Joint Task Group: L. Malcolm Baker.
1040 A. Introduction
Although standardized test methods are available from many
nationally recognized sources, there may be occasions when they
cannot be used or when no standard method exists for a particular
constituent or characteristic. Therefore, method development may
be required. Method development is the set of experimental pro-
cedures devised for measuring a known amount of a constituent
in various matrices, in the case of chemical analyses; or a known
characteristic (e.g., biological or toxicological) of various matrices.
1040  B. Method Validation
Whether an entirely new method is developed by accepted
research procedures or an existing method is modified to meet
special requirements, validation by a 3-step process is required:
determination of single-operator precision and bias, analysis of
independently prepared unknown samples, and determination of
method ruggedness. Universal acceptance criteria are difficult to
establish and may vary by test type (e.g., chemical, microbiolog-
ical, toxicological), matrix, intended use, and regulatory author-
ity. It is therefore necessary to establish these criteria during the
method-development stage and apply them in the design of vali-
dation efforts. The single-operator characteristics may be a good
place to begin evaluating and establishing these criteria.
1. Single-Operator Characteristics
This part of the validation procedure requires determining
detection and reporting levels as in Section 1020 B.4 and 1030 C,
applicable analytical range, the method’s bias (i.e., its systematic
error), the precision obtainable by a single operator (i.e., the ran-
dom error introduced in using the method), matrix effects, and
interferences. To make these determinations, analyze at least 7
but preferably 10 or more portions of a standard at each of several
concentrations in each matrix that may be used. Use one concen-
tration at, or slightly above, the detection limit and one relatively
high so the range of concentrations for which the method is appli-
cable can be specified.
The use of several concentrations to determine bias and preci-
sion will reveal the form of the relationship between these method
characteristics and the substance’s concentration, the substance’s
characteristic toxicity, or the biological factor of interest. This
relationship may be constant, linear, or curvilinear and is a sig-
nificant characteristic of the method that should be explained
clearly. Table 1040:1 shows the calculation of precision and bias
for a single concentration in a single matrix from 8 replicate anal-
yses of a standard with a known concentration of 1.30 mg/L.
https://doi.org/10.2105/SMWW.2882.007
The guidance provided in this section is generalized and slanted
toward chemical analyses in most cases. Bioassays, taxonomic
classification, and microbiology testing frequently demand addi-
tional or alternative considerations during development and val-
idation. In some cases, these may be spelled out in other parts of
this compendium (e.g. Parts 8000, 9000, 10000) or from other
publications or regulatory authorities.
2. Analysis of Unknown Samples
This step in the method-validation procedure requires the anal-
ysis of independently prepared standards whose value is unknown
to the analyst. Analyze each unknown in replicate by following the
standard operating procedure for the method. The mean amount
recovered must be within 3 standard deviations (s) of the stan-
dard’s mean value but preferably within 2 s.
Obtain the unknowns from other personnel in the analyst’s lab-
oratory using either purchased analytical-grade reagents or stan-
dards traceable to National Institute of Standards and Technology
(NIST). If available for the particular constituent, performance
evaluation samples from accredited proficiency test (PT) provid-
ers are also recommended.
3. Method Ruggedness
A test of the method’s ruggedness (i.e., stability of the result
produced when steps in the method are varied) is the final valida-
tion step. It is especially important to determine this characteristic
if the method will be proposed as a standard or reference method.
A properly conducted ruggedness test points out those procedural
steps in which rigor is critical and those in which some leeway is
permissible.
The Association of Official Analytical Chemists1 has suggested
a method for this test in which 8 separate analyses can be used to
determine the effect of varying 7 different steps in an analytical
procedure. To illustrate, suppose the effect of changing the factors
in Table 1040:2 is to be determined. To make the determination,
denote the nominal factors by capital letters A through G and the
variations by the corresponding lower-case letters. Then set up a
table of the factors (Table 1040:3).
If Combination 1 is analyzed, the result will be s. If Combina-
tion 2 is analyzed, the result will be t, and so on until all 8 com-
binations have been analyzed. To determine the effect of varying
1
 1040 METHOD DEVELOPMENT AND EVALUATION - B. Method Validation

Table 1040:1. Precision and Bias for a Single Concentration in a Single Table 1040:3. Factor Matrix for Method Ruggedness Determination
Matrix
Combinations
Result (mg/L) Difference (1.30 ) Squared Difference Factor value 1 2 3 4 5 6 7 8
1.23 –0.07 0.0049 A or a A A A A a a a a
1.21 –0.09 0.0081 B or b B B b b B B b b
1.30 0.0 0.0 C or c C C C c C c C c
1.59  0.29 0.0841 D or d D D d d d d D D
1.57  0.27 0.0729 E or e E E E e e E e E
1.21 –0.09 0.0081 F or f F F f F F f f F
1.53  0.23 0.0529 G or g G g g G g G G g
1.25 –0.05 0.0025
Result s t u v w x y z
Sum = 0.49 = 0.2335
n =8 Source: Youden WJ, Steiner EH. Statistical Manual of the AOAC. Washington
n DC: Association of Official Analytical Chemists; 1975.
Bias 0.49 mg /L
= ∑ (difference)i /n = = 0.06 mg /L
i =1
8
An explanation of each step, with additional techniques and
n
2 examples, has been published.4 Because the number of anal-
= ∑ (difference)i / (n −1) = 0.2335
(8 −1)
= 0.18 mg /L
yses can be very large, the calculations become complex and
Precisiona i =1
This is similar to the calculation for standard deviation.
a familiarity with basic statistics is necessary. A listing of stan-
dard, reference, and equivalent methods for water analysis is
available.5
a factor, find the 4 results where the factor was nominal (all caps)
and the 4 where it was varied (all lower case) and compare the References
averages of the 2 groups. For example, to compare the effect of
changing C to c, use results (s + u + w + y)/4 and (t + v + x + 1. Youden WJ, Steiner EH. Statistical Manual of AOAC. Washington
z)/4. Calculate all 7 pairs to get 7 differences, which can then be DC: Association of Official Analytical Chemists; 1975.
ranked to reveal those with a significant effect on the results. If 2. Williams LR. Harmonization of biological testing methodology:
there is no outstanding difference, calculate the average and stan- a performance based approach in aquatic toxicology and hazard
dard deviation of the 8 results s through z. The standard deviation assessment. 8th Symp. ASTM STP 891, Bahner RC, Hansen DJ, eds.
is a realistic estimate of the method’s precision. This design tests Philadelphia (PA): ASTM International, 1985.
main effects, not interactions. 3. Natrella MG. Experimental statistics. Washington DC: National
Bureau of Standards Handbook 91, 1963.
4. U.S. Environmental Protection Agency. Guidelines for establish-
4. Equivalency Testing ing method equivalency to standard methods; Rep. 600/X-83-037.
Las Vegas (NV): Environmental Monitoring Systems Laboratory;
After a new method has been validated by the procedures listed 1983.
above, it may be necessary to test the method for equivalency to 5. U.S. Environmental Protection Agency. Guidelines establishing test
standard methods, unless none exist. This requires analyzing at procedures for the analysis of pollutants under the Clean Water Act.
least 3 concentrations by both the alternate and standard methods. Final rule. 1994. 40 CFR Part 136; Fed Reg. 59:20:4504.
If the range of concentration is very broad, test more concentra-
tions. Once an initial set of analyses (5 or more) has been made at Bibliography
each chosen concentration, apply the following statistical steps:2
1.  Test the distribution of data for normality and transform Huber L. Validation of computerized analytical systems. Buffalo Grove
the data if necessary (Section 1010 B). (IL): Interpharm Press, Inc.; 1995.
2.  Select an appropriate sample size based on an estimate of Parkany M. Use of recovery factors in trace analysis. Cambridge (UK):
the standard deviation.3 Royal Society of Chemistry; 1996.
Huber L. Validation and qualification in analytical laboratories. Buffalo
3. Test the variances of the 2 methods using the F-ratio statistic.
Grove (IL): Interpharm Press, Inc.; 1999.
4.  Test the average values of the 2 methods using a Student Thompson M, Ellison SLR, Fajgelj A, Willetts P, Wood R. Harmonised
t statistic. guidelines for the use of recovery information in analytical measure-
ment. Pure Appl Chem. 1999;71(2):337–348.
Table 1040:2. Variations in Factors for Method Ruggedness Aboul-Enein HY, Stefan R, Baiulescu G. Quality and reliability in analyt-
Determination ical chemistry. Boca Raton (FL): CRC Press; 2000.
Fajgelj A, Ambrus A. Principles and practices of method validation.
Factor Nominal Variation
Cambridge (UK): Royal Society of Chemistry; 2000.
Mixing time 10 min 12 min Huber L. A Primer: Good laboratory practice and current good manufac-
Portion size 5g 10 g turing practice. Deutschland GmbH, Waldbronn (Germany): Agilent
Acid concentration 1M 1.1 M Technologies; 2000.
Heat to 100 °C 95 °C National Institute of Standards and Technology. The NIST Reference
Hold heat for 5 min 10 min on Constants, Units, and Uncertainty; 2002 [accessed 2020 June 15].
Stirring yes no http://physics.nist.gov/cuu/Uncertainty/index.html.
pH adjust 6.0 6.5

https://doi.org/10.2105/SMWW.2882.007 2
 1040 METHOD DEVELOPMENT AND EVALUATION - C. Collaborative Multilaboratory Testing

Thompson M, Ellison SLR, Wood R. Harmonized guidelines for sin-
gle-laboratory validation of methods of analysis (IUPAC technical
report). Pure Appl Chem. 2002;74(5):835–855.
Huber L. Validation of computerized analytical and networked systems.
Denver (CO): Interpharm Press, Inc.; 2002.
Barrentine L. Concepts for R & R Studies, 2nd ed. Milwaukee (WI): ASQ
Quality Press; 2003.
Cacuci DG. Sensitivity and uncertainty analysis - Theory, Vol 1. Boca
Raton (FL): CRC Press, 2003.
Egli H, Dassenakis M, Garelick H, van Grieken R, Peijnenburg WJGM,
Klasinc L, Kördel W, Priest N, Tavares T. Minimum requirements for
reporting analytical data for environmental samples. Pure Appl Chem.
2003;75(8):1097–1106.

1040 C. Collaborative Multilaboratory Testing

After a new or modified method has been developed and val- r > 1 (30 /P )
idated, it is appropriate to determine whether the method should
be made a standard method. The procedure to convert a method where:
into a standard method is the collaborative multilaboratory test.1
r = number of replicates and
In this test, multiple laboratories collaborate by using the same
P = the product of several variables.
standard operating procedure to analyze a select number of sam-
ples to determine the method’s bias and precision, as would occur The minimum number of replicates is 3. As an example, if 3
in normal practice. levels of a substance will be analyzed by single operators in 6
In planning for a collaborative multilaboratory test, consider laboratories on a single apparatus, then P is calculated as follows:
the following factors: a precisely written standard operating pro-
cedure, the number of variables to be tested, the number of lev- P = 3 × 1 × 6 × 1 = 18
els to be tested, and the number of replicates required. Because
method precision is estimated by the standard deviation, which
and the number of replicates is
itself is the result of many sources of variation, the variables that
affect standard deviation must be tested. These may include the
laboratory, operator, apparatus, and concentration range. r > 1 (30 /18) > 2.7 or r = 3.

1. Variables 3. Example of a Collaborative Multilaboratory Test

Test at least the following variables: Send each of 5 laboratories 4 concentrations of a compound
a. Laboratory: Involve at least 3 different laboratories, although (4.3, 11.6, 23.4, and 32.7 mg/L) with instructions to analyze in
more are desirable to provide a better estimate of the standard triplicate using the procedure provided. Tabulate results as shown
deviation. Note: Some standards organizations require at least in Table 1040:4 (the results for only one concentration are shown).
8 laboratories for collaborative testing. Each collaborating lab- Because there are no obviously aberrant values (use the method in
oratory and analyst should demonstrate proficiency with the new Section 1010 B.4 to reject outliers), use all the data.
method before interlaboratory data are generated in a multilabo-
ratory study.
b. Apparatus: Because model and manufacturer differences can
Table 1040:4. Sample Collaborative Multilaboratory Test Results
be sources of error, analyze at least 2 replicates of each concen-
tration per laboratory. Deviation
c. Operators: To determine overall precision, involve at least 6 Result Experimental From From Grand
analysts (no more than 2 from each laboratory). Laboratory (mg/L) x±s Known Average
d. Levels: If the method development has indicated that the
1 32.7 34.7 ± 1.8 2.0 1.7
relative standard deviation is constant, test 3 levels covering the
35.2
range of the method. If it is not constant, use more levels spread 36.3
uniformly over the operating range. If developing a new method 2 32.6 33.3 ± 0.6 0.6 0.3
to compare to an existing standard, use the number of levels equiv- 33.7
alent to the existing method’s calibration range and requirements. 33.6
If matrix effects are suspected, conduct the test evaluation in 3 30.6 31.2 ± 1.0 –1.5 –1.8
each medium for which the method was developed. If this is not 30.6
feasible, use appropriate grades of reagent water as long as this 32.4
is stipulated in the resulting statement of method characteristics. 4 32.6 33.0 ± 0.8 0.3 0
Results are reported as being matrix-specific for the types of 32.5
matrices to which the method is purportedly applicable. 33.9
5 32.4 32.9 ± 0.5 0.1 0.2–0.1
2. Number of Replicates 33.4
32.9
Calculate the number of replicates after the number of variables (Σx)/n = 33 S = 1.5 Σ = –0.1
to be tested has been determined by using the formula: s = 1.5

https://doi.org/10.2105/SMWW.2882.007 3
 1040 METHOD DEVELOPMENT AND EVALUATION - C. Collaborative Multilaboratory Testing

Table 1040:5. Method Precision and Bias As noted in Table 1040:4, the sum of the deviations from the
Known Amount CV (% Standard known value for the laboratories was 1.3, so the average deviation
Amount (mg/L) Detected (mg/L) Deviation) Bias (%) (bias) was 1.3/5 = 0.26, rounded to 0.3, which is the same as the
difference between the grand average and the known value.
 4.3  4.8 12.5 11.5
For all 4 unknowns in this test, the percentage results indicated
11.6 12.2 10.2  5.6
increasing bias and decreasing precision as the concentration
23.4 23.8  5.4  1.9
decreased. Therefore, to describe the method in a formal state-
32.7 33  4.5  0.9
ment, the precision would be given by a straight line with the
CV = coefficient of variation; CV% = (standard deviation/grand average) × 100 formula y = mx + b; where y is the relative standard deviation, m
is the slope of the line, x is the concentration, and b is the relative
Calculate the average and standard deviation for each labora- standard deviation at concentration 0. The values found from the
tory; use all 15 results to calculate a grand average and standard collaborative test are shown in Table 1040:5.
deviation. The difference between the average of each laboratory and These results indicate that the method is acceptable. However,
the grand average reveals any significant bias, such as that shown for concentrations of less than about 10 mg/L require greater care in
Laboratories 1 and 3. The difference between the grand average analysis.
and the known value is the method bias (e.g., 33.0 – 32.7 = 0.3 mg/L,
or 0.9%). The relative standard deviation of the grand average Reference
(1.5 mg/L) is 4.5%, which is the method precision, and the s for
each laboratory is the single-operator precision. 1. Youden WJ, Steiner EH. Statistical Manual of the AOAC. Washing-
ton DC: Association of Official Analytical Chemists; 1975.

Published Online: August 27, 2018

Revised: July 21, 2021
https://doi.org/10.2105/SMWW.2882.007 4