STIMULI TO THE REVISION PROCESS

Stimuli articles do not necessarily reflect the policies Pharmacopeial Forum

772 of the USPC or the USP Council of Experts Vol. 35(3) [May–June 2009]

Acceptable, Equivalent, or Better: Approaches for Alternatives to Official

Compendial Procedures
W.W. Hauck, PhD,a A.J. DeStefano, PhD, T.L. Cecil, PhD, D.R. Abernethy, MD, PhD, W.F. Koch, PhD, R.L. Williams, MD

ABSTRACT The General Notices in the US Pharmacopeia (USP) permit analysts to use acceptable (suitable) alternatives
to an official procedure. In a revision that will be effective May 1, 2009, the General Notices will require that the
alternative procedure be demonstrated to give results that are equivalent to or better than those obtained by the
official procedure. This Stimuli article discusses approaches for determining equivalent or better procedures. The
concepts and tests discussed in this paper may become one or more USP General Chapters. The preceding Stimuli
article speaks to approaches that determine an acceptable procedure without requiring a study to document
equivalent or better.
Stimuli to the Revision Process

INTRODUCTION
In its General Notices, the US Pharmacopeia (USP)
states: ‘‘Compliance may be determined also by the
use of alternative methods, chosen for advantages in ac-
curacy, sensitivity, precision, selectivity, or adaptability to
automation or computerized data reduction or in other
special circumstances. Such alternative or automated
procedures or methods shall be validated (1).b The Gen-
eral Notices requirement for validation means the analyst
using the alternative procedure has determined the
procedure is acceptable (suitable) for its intended use
in the specified monograph test. The wording of the
USP General Notices presumes that the alternative proce-
dure possesses some property for which there is an ad-
vantage, however defined—otherwise why use it? But
the General Notices statement does not require com-
parison to the compendial procedure on advantageous
properties. A priori, one would think that the alternative
procedure should be not be worse, however defined,
than the compendial procedure, but this is not stated
in the current General Notices. A revision to the General
Notices (3) that will be official May 1, 2009, calls for
the alternative procedure to ‘‘give equivalent or better
results’’ by comparison with the compendial procedure.
In addition, elsewhere the General Notices state: ‘‘Propor-
tionately larger or smaller quantities than the specified
weights and volumes of assay or test substances and
Reference Standards may be taken, provided the mea-
surement is made with at least equivalent accuracy. . .[em-
phasis added]‘‘ (1). General Chapter Validation of
Alternative Microbiological Methods h1223i states, ‘‘The
critical question is whether or not the alternat[iv]e meth-
a
Correspondence should be addressed to: W.W. Hauck, PhD, Senior
Scientific Fellow, US Pharmacopeia, 12601 Twinbrook Parkway, Rock-
ville, MD 20852-1790; tel. 301.816.8390; e-mail wh@usp.org.
b
Using terminology of the International Conference on Harmonization
(ICH), the US Pharmacopeial Convention (USPC) uses the term proce-
dure to describe the steps to be followed by the analyst in conducting
a test. This is in agreement with the International Vocabulary of Metrology
(VIM) (2), which describes a measurement procedure as a detailed de-
scription in contrast to a measurement method, which is a generic de-
scription (such as ‘‘indirect, a direct method, or an instrumental
method’’). This Stimuli article thus will use the terminology comparison
of procedures rather than comparison of methods.
od will yield results equivalent to, or better than, the results
generated by the conventional method [emphasis
added]‘‘ (1, p. 681). USP does not provide a definition
or approaches for alternative procedures, whether they
are deemed acceptable, equivalent, or better.
The Food and Drug Administration (FDA) and ICH em-
ploy similar language regarding alternative procedures.
The Code of Federal Regulations, in the section that cov-
ers requirements for Equivalent Methods and Processes
for biological products, requires ‘‘that the modification
will provide assurances of the safety, purity, potency,
and effectiveness of the biological product equal to or
greater than the assurances provided by the method or
process specified in the general standards or additional
standards for the biological product [emphasis added]’’
(4). FDA also uses a similar phrase in its draft guidance,
Analytical Procedures and Methods Validation: Chemistry,
Manufacturing, and Controls Documentation (5). In dis-
cussing alternative analytical procedures this guidance
notes, ‘‘A validated alternative analytical procedure
should be submitted only if it is shown to perform equal
to or better than the regulatory analytical procedure [em-
phasis added]’’ (5). ICH Q6A, 2.7 Alternative Procedures,
states, ‘‘Alternative procedures are those which may be
used to measure an attribute when such procedures con-
trol the quality of the drug substance or drug product to
an extent that is comparable or superior to the official
procedure [emphasis added]’’ (6). In MAPP 5310.7,
FDA’s Office of New Drug Quality Assessment states the
following policy for its reviewers: ‘‘If there is no USP/NF
monograph for an excipient, drug substance, or drug
product, and the applicant proposes to use an analytical
procedure from the BP, EP, or JP in a specification in lieu of
the corresponding analytical procedure in the General
Chapters of the USP/NF, the BP, EP, or JP procedure is
considered an alternative analytical procedure and may
be used provided the analytical procedure in the BP, EP,
or JP is equivalent to or better than the corresponding ana-
lytical procedure in the USP/NF [emphasis added].’’
‘‘Equivalent’’ is used in this paper rather than the ‘‘equal’’
of 21 CFR 610.9(c) in order to avoid the connotation of
equal as meaning identical (7).

# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
 STIMULI TO THE REVISION PROCESS
Pharmacopeial Forum Stimuli articles do not necessarily reflect the policies
Vol. 35(3) [May–June 2009] of the USPC or the USP Council of Experts
Based on this brief review, the various regulatory and
compendial statements appear to lack precision in defini-
tion. Following internal discussions, USPC has identified
four options (Table 1) to allow consideration of alterna-
tives to official procedures, even though the usual word-
ing allows only ‘‘equivalent or better.’’
 The first is for USP to set minimum performance re-
quirements for alternative procedures. Procedures
meeting these requirements are termed Acceptable
Procedures. For example, for chromatographic proce-
dures, manufacturers could establish minimum re-
quirements for repeatability and peak separation.
The concept then would be that any alternative
procedure that meets the applicable requirements
would be acceptable, i.e., suitable for its intended
use. In effect, two procedures that both meet the
performance requirements would be considered
‘‘equivalent’’ (and hence acceptable) under this op-
tion without any direct comparison to the official
procedure.

The remaining options directly compare the alterna-
tive procedure to the compendial procedure and re-
quire ‘‘method-performance studies,’’ (8) i.e.,
procedure comparison studies, thus dealing more
explicitly with the requirement to show that the al-
ternative is equivalent to or better than the official
procedure. This article proposes three questions that
drive the subsequent three options. For the second
option, termed performance equivalence, the ques-
tion is: Are the two procedures equivalent or better
with respect to validation characteristics? This is sim-
ilar to Option 1 because it considers validation char-
acteristics but features a direct comparison to the
official procedure. For the third and fourth options,
the question is: Do the two procedures agree? Agree-
ment can be assessed by considering the numerical
result (Option 3) of the procedure or a decision (Op-
tion 4) relative to some acceptance criterion that is
based on the numerical result. For the third and
fourth options, USPC proposes the terms results
equivalence and decision equivalence, respectively, to
delineate the two options. Each question leads to dif-
ferent statistical approaches to allow the com-
parison.
Following the metrological distinctions between proce-
dure and method in VIM, the discussion of this paper ap-
plies regardless of whether the comparison is between
Table 1. Comparison of Options
Option Name Demonstrating
1 Acceptable Procedures Acceptable
2 Performance Equivalence Equivalent or Better
3 Results Equivalence Equivalent
4 Decision Equivalence Equivalent
a
There need not be an official method.
# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
773
two procedures that correspond to different methods
or between two different procedures for the same meth-
od (2). The statistical and compendial issues are the same
in either case. This paper assumes that the two proce-
dures being compared are measuring the same attribute.
Comparison of two bioassays that measure different attri-
butes of an article, for example, is a topic not addressed
by this discussion. A similar problem is termed method
transfer (same procedure but different laboratories). That
is not a pharmacopeial issue but is statistically similar to
the topic considered here (see Appendix for some refer-
ences).
This paper first addresses each of the four options indi-
vidually with consideration of advantages and disadvan-
tages and a brief overview of statistical considerations. In
the discussion the paper summarizes choices and ways
forward. The purpose is to identify issues that should
be considered and candidate statistical approaches with
the intent of initiating a dialog that could lead to com-
pendial recommendations or requirements in one or
Stimuli to the Revision Process
more General Chapters. This Stimuli article includes an
Appendix that cites some publications that are relevant
to the discussion.
OPTION 1: ACCEPTABLE PROCEDURES
General Comments
The essence of Option 1 is to allow the analyst more
than one—perhaps even many—acceptable alternative
procedures without a requirement for comparison to an-
other procedure. The comparison, instead, is to a refer-
ence material with known properties. This option may
be particularly valuable in cases where the current com-
pendial procedure is especially accurate or precise. It
would be difficult to demonstrate that the alternative
procedure is equivalent to or better than the official
procedure in such cases. Implicitly, the judgment of ac-
ceptability of a procedure would acknowledge that it is
‘‘equivalent’’ in meeting compendial expectations. To
this end, USPC has proposed, in a related article in this
Pharmacopeial Forum, performance-based monographs
in which one and perhaps more than one optional ac-
ceptable procedure would be deemed acceptable for
use in a particular test (9). The key characteristic of a per-
formance-based procedure in a monograph is that it
would specify performance requirements for a procedure
rather than provide a required compendial procedure,
although one could be deemed preferred (official) for
compliance purposes.
Comparison Based on Number of
to Official Numerical Results Characteristics
Procedure? or Conclusion? Considered
Noa Results Many
Yes Results Many
Yes Results One
Yes Conclusions One
 STIMULI TO THE REVISION PROCESS
Stimuli articles do not necessarily reflect the policies
774 of the USPC or the USP Council of Experts
For Option 1, the criteria for an acceptable procedure
could be based on validation characteristics such as spe-
cificity, absence of bias, and precision. Criteria for accep-
tance could be based on the procedure’s operating
characteristics, specifically on whether articles of known
properties pass or fail appropriately when compared to
the relevant acceptance criteria. See (9) for an example
of this approach.
OPTION 2: PERFORMANCE EQUIVALENCE
General Comments
Performance equivalence means demonstrating
‘‘equivalent or better’’ results on some subset of valida-
tion properties. General Information Chapter h1225i lists
many candidate properties. By focusing on a subset, Op-
tion 2 is similar to verification in General Information
Chapter h1226i, where determination of which proper-
ties for the procedure’s intended use are important. In
Stimuli to the Revision Process
the context of alternative procedures, the most impor-
tant generally are specificity and accuracy. Although pre-
cision (variability) is important, analysts can sometimes
address precision in different ways, such as the choice
of the number of replicates (discussed below). Even with
focus on a subset of characteristics, a disadvantage of
Option 2 is the multiplicity of statistical tests correspond-
ing to the multiple validation properties for a given
procedure and the designated attribute of the measur-
and. One can easily envision an alternative procedure
that is better on some properties, equivalent on some,
and worse on some. What should be the final decision
regarding the alternative? Prioritizing the importance of
the validation properties seems to be a key. An additional
question is whether the new procedure actually is better
than the official procedure on some property. An affirma-
tive answer does allow that the alternative procedure
may be preferable based on considerations not captured
by validation characteristics (e.g., cost or time).
Equivalence Testing
Comparing two procedures to show equivalent or bet-
ter performance for one or more validation properties
leads to a body of statistical work referred to as equiva-
lence testing. The basic idea is to properly align the sta-
tistical hypotheses with the intent of the work. If the goal
is to positively demonstrate equivalent or better perfor-
mance, then ‘‘equivalent or better’’ needs to be the sta-
tistical alternative hypothesis. The statistical null
hypothesis is then ‘‘neither equivalent nor better.’’ This
has not been common practice, although the literature
in this field has evolved to equivalence testing. For com-
parison of means, for example, a common practice has
been to declare equivalence if the two means are not
found to be statistically different with a standard t-test.
With a standard t-test, the alternative hypothesis is differ-
ence, and the null hypothesis is no difference. The papers
by Limentani et al. (10) and Chambers et al. (11) contain
good examples of why the equivalence approach is pre-
ferred to the still commonly used standard t-test.
# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
Pharmacopeial Forum
Vol. 35(3) [May–June 2009]
When comparing two procedures to understand if
they are ‘‘equivalent or better,’’ the analyst must deter-
mine which of three outcomes (questions) are sought
in the comparison study: 1) the new procedure may be
importantly worse than the current procedure; 2) the
new procedure may be importantly better than the cur-
rent procedure; and 3) there may be no important differ-
ence, either better or worse, between the new and
current procedures. When outcomes 2 and 3 are found,
the results are ‘‘equivalent or better,’’ and only the first
outcome is excluded. In clinical trials this is referred to
as noninferiority. When outcome 3 is found in the com-
parison results, the procedures are referred to as ‘‘equiva-
lent.’’ Thus in a comparison study, the analyst must be
clear about the intent of the study in order to derive use-
ful conclusions.
Demonstrating Equivalent or Better—Means
A starting point for comparing two procedures is to
compare only means. Ideally, though, this would be a
comparison of biases (accuracy in ICH and USP terminol-
ogy). A noninferiority hypothesis would be that the bias
of the alternative procedure is equivalent to or better
(less) than that of the official procedure. This would re-
quire a comparative accuracy (trueness) assessment of
the two procedures. If bias cannot be assessed, a related
approach would be to suggest that the bias of the official
procedure is acceptable. This leads to a two-sided
equivalence hypothesis of equivalence of the two proce-
dures’ means for the alternative. Specifically, the alterna-
tive procedure must be similar on average to the official
procedure. Statistical procedures for one- and two-sided
equivalence hypotheses for means are well developed
(12, 13).
The problem with a simple comparison of means is
that the difference may vary with the quantity being
measured. Analysts may need to compare accuracy sepa-
rately at each of a range of values rather than perform a
single, composite accuracy comparison. Calibration (re-
gression) approaches have been developed and allow for
both systematic and proportional differences between
the results of the procedures; see the Appendix for refer-
ences. Most of the procedures in the literature determine
the regression for the relationship and then, inappropri-
ately, test the null hypotheses of intercept equal to 0 (no
systematic difference) and slope equal to 1 (no propor-
tional difference). If both are not rejected, equivalence
is claimed. This is the common error of wrong hypoth-
eses for an equivalence problem. Equivalence hypotheses
should specify a range for an intercept sufficiently close
to 0 and for a slope sufficiently close to 1. These are
two-sided equivalence hypotheses.
Demonstrating Equivalent or Better—Variances
Variance comparisons are clearly one-sided (noninfer-
iority); i.e., a procedure that results in an important in-
crease in variance only would not be considered
performance equivalence. The general approach is al-
ready covered in USP’s General Information Chapter Ana-
lytical Data—Interpretation and Treatment h1010i. The
larger question relates to which variance(s) to compare.
If the context is a single laboratory, reproducibility is not
 STIMULI TO THE REVISION PROCESS
Pharmacopeial Forum Stimuli articles do not necessarily reflect the policies
Vol. 35(3) [May–June 2009] of the USPC or the USP Council of Experts
relevant. For companies with multiple laboratories using
a procedure and for compendial statements about pro-
cedures that are ‘‘equivalent or better,’’ reproducibility
is relevant. Where comparisons are confined to a single
laboratory, analysts also must bear in mind that repeat-
ability is not a full characterization of the precision of a
procedure. The question then becomes, Which inter-
mediate precision components should be assessed?
The other issue with variances is that, in a certain
sense, the variance comparison sometimes can be made
less important, at least for repeatability. That is, variance
may be overcome with increased sample size. If the new
procedure appears to be more variable than the official
procedure, the sample size can be increased to make
the new procedure comparable. With respect to a proce-
dure being developed for use as an alternative proce-
dure, variance can be addressed during validation. The
‘‘sample’’ for determining what needs to be increased
will depend on the source(s) of variability and the var-
iance of interest. If repeatability is at issue, this may mean
simply more samples and/or replicates. If intermediate
precision or reproducibility is at issue, improving repeat-
ability is not sufficient and it may not be feasible to in-
crease, e.g., the number of analysts conducting the
procedure or the time over which the procedure is con-
ducted in order to reduce the contribution from those
components of intermediate precision.
OPTION 3: RESULTS EQUIVALENCE
General Comments
When results equivalence is sought, the hypothesis of
the comparison study is that the alternative and official
procedures obtain equivalent numerical results. Showing
that two procedures obtain similar results could be use-
ful, e.g., for a procedure applied in stability testing when
it is desirable to avoid a discontinuity in results with a
change of procedure. Option 3 is thus similar to Option
2 in that the focus is on the numerical results obtained by
the procedures. Options 2 and 3 differ in the nature of
the comparison. In Option 3, there is a direct comparison
of results. In Option 2, the comparison is indirect; valida-
tion characteristics are evaluated and compared.
Demonstrating Essentially the Same Numerical Result
There are at least two general approaches that could
be taken. For the first, a large body of literature examines
the general issue of inter-rater reliability. The general idea
is to assess the extent to which two raters get the same
result when using the same instrument. Many applica-
tions of inter-rater reliability involve psychometric studies
in which the instrument is a questionnaire, but applica-
tions are widespread. For continuous measures, the intra-
class correlation coefficient is the most common measure
of agreement, and generally accepted terms describe the
degree of agreement (10). This literature can be adapted
to comparison of compendial procedures if one inter-
prets ‘‘rater’’ as ‘‘procedure.’’ A more recent alternative
is to use the concordance correlation coefficient.
# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
775
Design of experiments for interprocedure agreement
should focus on ensuring that the two procedures obtain
similar results for each sample tested. If the testing is non-
destructive, this is straightforward. For destructive test-
ing, this involves, e.g., preparing replicate aliquots from
common preparations.
A difficulty with both intraclass and concordance cor-
relation approaches is the lack of a direct connection to
consequences with respect to producer risk (failing to
adopt an equivalent alternative procedure) and consu-
mer risk (adopting an inequivalent alternative proce-
dure). With this consideration in mind, the second
general approach considered here, the total error ap-
proach, was proposed for procedure transfer. The basic
idea is to use tolerance intervals to control how often
the two procedures give numerical results that differ by
an important amount. What constitutes an important
amount then depends on the acceptance criteria asso-
ciated with the procedure.
Stimuli to the Revision Process
OPTION 4: DECISION EQUIVALENCE
General Comments
Comments for decision equivalence are similar to
those for results equivalence. The difference between
Options 3 and 4 is whether agreement occurs with re-
spect to numerical results (Option 3) or a decision rela-
tive to some acceptance criteria (Option 4). The
distinction leads to different statistical procedures. Thus
the key characteristic of Option 4 is that it looks only at
the pass/fail results, not the actual numerical results that
are the basis of Options 1, 2, and 3.
Demonstrating Essentially the Same Pass/Fail Results
If the only concern is whether or not the two proce-
dures agree on meeting acceptance criteria—rather than
the actual numerical value—different statistical proce-
dures apply. A related application arises when a proce-
dure yields only an either/or result (meets/does not
meet), as is the case for limit or identity tests. The primary
statistical procedure here is the kappa coefficient. Other-
wise the issues are the same as for the intraclass and con-
cordance correlations. The kappa coefficient shares the
same disadvantage, namely no easy connection to con-
sequences. A possible alternative is to consider the full
operating characteristic curves for the two procedures.
The operating characteristic curves, by moving the com-
parison to the probability rather than correlation scale,
may make it easier to consider which differences be-
tween the procedures are important.
DISCUSSION
This paper has considered four options for what is
meant by ‘‘equivalent or better.’’ Although there is some
similarity across the four, they do differ. An alternative
procedure that might be judged equivalent or better
by one option need not be judged the same by the other
options. One important distinction between Option 2
and Options 3 and 4, is that Options 3 and 4 preclude
‘‘better’’ because the alternative procedure must be dif-
 STIMULI TO THE REVISION PROCESS
Stimuli articles do not necessarily reflect the policies
776 of the USPC or the USP Council of Experts
ferent to be better. Another distinction is that Option 1,
because it does not involve a formal procedure com-
parison study, yields no guarantee that any two accepta-
ble procedures will agree with respect to either numerical
results or decisions relative to some acceptance criterion,
though tight criteria about what constitutes ‘‘accepta-
ble’’ makes it likely two acceptable procedures will not
differ greatly with respect to results.
Discussion regarding alternative procedures tends to
return to the decision made based on results from the
procedure. The various options differ in how they con-
nect to the accept/reject decision. With Option 4, agree-
ment of procedures with respect to decision is a direct
connection. For Options 1, 2, and 3 the connection must
be indirect. A general issue, as for any equivalence assess-
ment, is: How close is close enough? A general approach
is to link the procedure comparison or the definition of
acceptable back to its use in support of accept/reject de-
cisions: What is the impact on consumer and producer
risks? The analytic error is just one component of uncer-
Stimuli to the Revision Process
Pharmacopeial Forum
Vol. 35(3) [May–June 2009]
made and justified by the laboratory. In the future the
choice may be specified in the monographs. Should ana-
lysts, however, wish to demonstrate ‘‘equivalent or bet-
ter,’’ they must posit the correct hypothesis and conduct
the appropriate statistical tests. In so doing, they can
readily determine through careful design of experiments
whether an alternative procedure is equivalent or better.
The general approach relies on good understanding of
the control space of the public monograph—the accep-
tance criteria for the specified tests.
REFERENCES
1. USPC. USP 31–NF 26, General Notices. Rockville, MD:
USPC; 2008:7.
2. Joint Committee for Guides in Metrology. International Vo-
cabulary of Metrology—Basic and General Concepts and As-
sociated Terms (VIM). 3rd ed. Geneva, Switzerland: ISO;
2007:2.6.
3. USPC. USP 32–NF 27, General Notices. Rockville, MD:

USPC; 2009:6.
tainty. Applied to a product that naturally varies from
4. 21 CFR 610.9(c).
batch to batch, a good analytic procedure is one whose
5. FDA. Analytical Procedures and Methods Validation: Chemis-
uncertainty is small relative to the acceptance interval so
try, Manufacturing, and Controls Documentation. Rockville,
that the procedure only minimally leads to wrong deci-
MD: FDA; 2000:3–4. Available at: www.fda.gov/cber/
sions.
gdlns/methval.htm; accessed December 17, 2008.
Options based on validation characteristics (Options 1
6. ICH. Q6A, Specifications for New Drug Substances and Pro-
and 2) share a problem of multiplicity, i.e., many charac-
ducts, 2.7 Alternative Procedures. Geneva, Switzerland:
teristics to be compared. It seems desirable to attempt to
ICH; 1999. Available at: www.ich.org/cache/compo/276-
reduce the number of comparisons. Partly, this is being
254-1.html, accessed December 17, 2008.
selective in the characteristics to compare. Further, one
7. Williams RL, Chen ML, Hauck WW. Equivalence ap-
can envision multivariate approaches at least for related
proaches. Clin Pharmacol Ther. 2002;72(3):229-237.
characteristics or a combination of some characteristics,
8. Horowitz W. Nomenclature of interlaboratory analytical
e.g., by use of mean squared error rather than bias and
studies. Pure Appl Chem. 199;66:1903–1911.
repeatability. A further problem is termed assay sensitiv-
9. Small Molecules Collaborative Group and USPC Staff. Per-
ity. That is, Is the procedure–equivalence study capable of
formance-based Monographs. Pharm Forum.
finding an important difference between the two proce-
2009;35(3):765–771.
dures if one exists?
10. Limentani GB, Ringo MC, Ye F, Bergquist ML, McSorley EO.
Regulatory and compendial statements have empha-
Beyond the t-test: statistical equivalence testing. Anal
sized the importance of ensuring that alternatives to pri-
Chem. 2005;77(11):221A–226A.
vate or public (compendial) procedures be ‘‘equivalent
11. Chambers D, Kelly G, Limentani G, Lister A, Lung KR, War-
or better.’’ As discussed in this article, USPC believes that
ner E. Analytical method equivalency: an acceptable
the requirement or recommendation should be framed
analytical practice. Pharm Technol. 2005;29(9):64–80.
somewhat differently. Option 1 simply states that within
12. Schuirmann DJ. A comparison of the two one-sided tests
the acceptance criteria delineated for a test, many proce-
procedure and the power approach for assessing equiva-
dures using the same or different methods may be accep-
lence of average bioavailability. J Pharmacokinet Biopharma-
table. No further studies are needed, and specifically it
ceutics. 1987;16:657–680.
isn’t necessarily sensible to require additional studies to
13. Wellek S. Testing Statistical Hypotheses of Equivalence. New
show equivalence (options 3 and 4) or better (option
York: Chapman & Hall/CRC; 2003.
2). For USPC this leads to ‘‘performance-based mono-
graphs.’’ Another Stimuli article in this issue of Pharmaco-
APPENDIX
peial Forum describes performance-based monographs,
some details regarding how acceptable might be defined, Following are procedure comparison references that
and USPC’s plans for implementation (9). are based on equivalence hypotheses and other useful
Although USPC is advancing consideration of a perfor- or interesting references. This does not claim complete-
mance-based monograph approach (an implementation ness; a large literature is available, and the authors wel-
of Option 1), the choice of option can be viewed as come suggested additions. More general statistical
monograph specific. For monographs for which the per- literature about equivalence testing is not included.
formance-based option does not apply, one of the other
options will be needed. At this writing, USPC is not spec- 1. Altman DG, Bland JM. Measurement in medicine: the ana-
ifying which option to use for implementing the General lysis of method comparison studies. Statistician.
Notices requirement to demonstrate equivalent or better 1983;32:307–317.
for a proposed alternative procedure. The choice is to be

# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
 STIMULI TO THE REVISION PROCESS
Pharmacopeial Forum Stimuli articles do not necessarily reflect the policies
Vol. 35(3) [May–June 2009] of the USPC or the USP Council of Experts
2. Bland JM, Altman DG. Statistical methods for assessing
agreement between two methods of clinical measure-
ment. Lancet. 1986;1(8476):307–310.
3. Bland JM, Altman DG. Measuring agreement in method
comparison studies. Stat Methods Med Res. 1999;8:135–
160.
4. Carstensen B. Comparing and predicting between several
methods of measurement. Biostatistics. 2004;5:399–413.
Note: assumes replicates.
5. Chambers D, Kelly G, Limentani G, Lister A, Lung KR, War-
ner E. Analytical method equivalency: an acceptable analy-
tical practice. Pharm Technol. 2005;29(9):64–80. Note:
Based on a 2003 PhRMA workshop.
6. Dewe W, Govaerts B, Boulanger B, Rozet E, Chiap P, Hubert
Ph. Using total error as decision criterion in analytical
method transfer. Chemometrics Intelligent Lab Syst.
2007;85:262–268.
7. Dewitte K, Fierens C, Stöckl D, Thienpont LM. Application
of the Bland–Altman plot for the interpretation of method-
comparison studies: a critical investigation of its practice
[letter]. Clin Chem. 2002;48:799–801. Note: Commenting
on use of Bland–Altman and other methods in Clin Chem.
8. Dunn G, Roberts C. Modeling method comparison data.
Stat Methods Med Res. 1999;8:161–179.
9. Gerbet D, Richardot P, Auget J-L, et al. New statistical ap-
proach in biochemical method-comparison studies by
using Westlake’s procedure, and its application to continu-
ous-flow, centrifugal analysis, and multilayer film analysis
techniques.Clin Chem. 1983;29(6):1131–1136. Note: In-
difference zone for bias and calibration but does not ad-
dress variance comparison.
10. Hoffman D, Kringle R. A total error approach for the valida-
tion of quantitative analytical methods. Pharm Res.
2007;24:1157–1164.
11. Kringle R, Khan-Malek R, Snikeris F, Munden P, Agut C,
Bauer M. A unified approach for design and analysis of
transfer studies for analytical methods. Drug Inf J.
2001;35:1271–1288.
12. Lao CS. Equivalence in test assay method comparisons for
the repeated-measure, matched-pair design in medical de-
vice studies: statistical considerations. J Biopharmaceutical
Stat. 2000;10:433–445. Note: Has hypotheses wrong but
does stress confidence intervals for equivalence.
13. Liao JJZ, Lewis JW. A note on concordance correlation coef-
ficient. PDA J Pharm Sci Technol. 2000;54:23–26.
14. Liao JJZ. An improved concordance correlation coefficient.
Pharm Stat. 2003;2:253–261.
15. Liao JJZ, Capen RC, Schofield TL. Assessing the concor-
dance of two measurement methods. Proc Biopharmaceuti-
cal Section, ASA. 2006; 660–667.
16. Liao JJZ, Capen RC, Schofield TL. Assessing the reproduci-
bility of an analytical method. J Chromatogr Sci.
2006;44:119–122. Note: Bridging study to evaluate repro-
ducibility.
17. Liao JJZ, Capen RC. Multiple evaluators. In: D’Agostino A,
Sullivan L, Massaro J, eds. Wiley Encyclopedia of Clinical
Trials. Hoboken, NJ: John Wiley & Sons; 2008.
18. Limentani GB, Ringo MC, Ye F, Bergquist ML, McSorley EO.
Beyond the t-test: statistical equivalence testing. Anal
Chem. 2005;77(11):221A–226A.
19. Lin LIK. A concordance correlation coefficient to evaluate
reproducibility. Biometrics. 1989;45:255–268.
# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.
777
20. Lung KR, Gorko MA, Llewelyn J, Wiggins N. Statistical
method for the determination of equivalence of auto-
mated test procedures. J Automated Methods Manage
Chem. 2005;25:123–127. Note: Considers only equiva-
lence of means.
21. Magari RT. Bias estimation in method comparison studies. J
Biopharmaceutical Stat. 2004;14:881–892.
22. Mantha S, Roizen MF, Fleisher LA, Thisted R, Foss J. Com-
paring methods of clinical measurement: reporting stan-
dards for Bland and Altman analysis. Anesth Analg.
2000;90:593–602. Note: Survey of the anesthesia litera-
ture.
23. Martin RF. General Deming regression for estimating sys-
tematic bias and its confidence interval in method-com-
parison studies. Clin Chem. 2000;46:100–104.
24. Martı́nez A, Riu J, Rius FX. Multiple analytical method com-
parison using maximum livelihood principal component
analysis and linear regression with errors in both axes. Ana-
lytica Chimica Acta. 2001;446:145–156. Note: Bivariate
least squares; good method of analysis but wrong hypoth-
Stimuli to the Revision Process
eses.
25. Nguyet ANM, van Nederkassel AM, Tallieu L, et al. Statisti-
cal method comparison: short- and long-column liquid
chromatography assays of ketoconazole and formalde-
hyde in shampoo. Analytica Chimica Acta. 2004;516:87–
106. Note: Interesting HPLC example with sophisticated
statistics and wrong hypotheses.
26. Nix ABJ, Dunstan FDJ. Maximum likelihood techniques ap-
p l ie d t o m e t h o d c o m p a r i s o n s t u d i e s . S t a t M e d .
1991;10:981–988 [Discussion: 988]. Note: For relative er-
ror of two methods; no gold standard required but need to
know response–error relationship for proper weighting.
27. Petersen PH, Stöckl D, Blaabjerg O, et al. Graphical inter-
pretation of analytical data from comparison of a field
method with a reference method by use of difference
plots. Clin Chem. 1997;43:2039–2046.
28. Quiroz, Jorge. A method for determining equivalence in in-
dustrial settings: defining and testing the equivalence of
two methods or two laboratories. Quality Eng.
2006;18:5–14. Note: Uses a population bioequivalence
approach to compare means and variances simulta-
neously.
29. Rousson V, Gasser T, Seifert B. Assessing intrarater, interra-
ter, and test–retest reliability of continuous measurements.
Stat Med. 2002;21:3431–3446.
30. Schepers U, Watzig H. Application of the equivalence test
according to a concept for analytical method transfers
from the International Society for Pharmaceutical Engi-
neering (ISPE). J Pharm Biomed Anal. 2005;39(1–2):310–
314.
31. Schepers U, Watzig H. Application of the equivalence test
for analytical method transfers: testing precision using the
United States Pharmacopoeia concept (1010). J Pharm
Biomed Anal. 2006;41(1):290–292.
32. Shao J, Zhong B. Assessing the agreement between two
quantitative assays with repeated measurements. J Biophar-
maceutical Stat. 2004;14:201–212. Note: Addresses re-
gression with errors in both methods; uses population
equivalence approach for hypotheses; no variance com-
parisons; assumes repeated measures on each item.
 STIMULI TO THE REVISION PROCESS
Stimuli articles do not necessarily reflect the policies Pharmacopeial Forum
778 of the USPC or the USP Council of Experts Vol. 35(3) [May–June 2009]
33. Stöckl D, Dewitte K, Thienpont LM. Validity of linear regres- 37. Walter SD, Eliasziw M, Donner A. Sample size and optimal
sion in method comparison studies: is it limited by the sta- designs for reliability studies. Stat Med. 1998;17:101–110.
tistical model or the quality of the analytical input data? Note: For test of intraclass correlation.
Clin Chem. 1998;44:2340–2346. 38. Zhong B, Shao J. Testing the agreement of two quantita-
34. Stöckl D, Cabaleiro DR, Van Uytfanghe K, Thienpont LM. tive assays in individual means. Commun Stat Theory Meth-
Interpreting method comparison studies by use of the ods. 2002;31:1283–1299. Notes: Sets a measure of
Bland–Altman plot: reflecting the importance of sample agreement; hypotheses wrong (fixed in their 2003 pa-
size by incorporating confidence limits and predefined er- per—see next entry); does not assume linearity and does
ror limits in the graphic [letter]. Clin Chem. 2004;50:2216– not require replication.
2218. 39. Zhong B, Shao J. Evaluating the agreement of two quanti-
35. Tan CY, Iglewicz B. Measurement-methods comparisons tative assays with repeated measurements. J Biopharmaceu-
and linear statistical relationship. Technometrics. tical Stat. 2003;13:75–86. Notes: Method of their 2002
1999;41:192–201. Keywords: Confidence interval; equiva- paper with equivalence hypotheses; requires replication.
lence test; errors in variables; measurement error.
36. Westgard JO. Points of care in using statistics in method
comparison studies [editorial]. Clin Chem. 1998;44:2240–
2242.
Stimuli to the Revision Process

# 2009 The United States Pharmacopeial Convention, Inc. All Rights Reserved.