See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/223735068

Least-squares analysis of data with uncertainty in x and y: A Monte Carlo

methods comparison

Article  in  Chemometrics and Intelligent Laboratory Systems · June 2010

DOI: 10.1016/j.chemolab.2010.07.003

CITATIONS READS

17 4,708

1 author:

Joel Tellinghuisen
Vanderbilt University
332 PUBLICATIONS   7,976 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

qPCR Data Analysis View project

The Art of Least Squares View project

All content following this page was uploaded by Joel Tellinghuisen on 04 February 2018.

The user has requested enhancement of the downloaded file.

This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
 Author's personal copy

Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169

Contents lists available at ScienceDirect

Chemometrics and Intelligent Laboratory Systems

j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / c h e m o l a b

Least-squares analysis of data with uncertainty in x and y: A Monte Carlo

methods comparison
Joel Tellinghuisen
Department of Chemistry, Vanderbilt University, Nashville, TN 37235, United States

a r t i c l e i n f o a b s t r a c t

Article history: The least-squares analysis of data with error in x and y is generally thought to yield best results when the
Received 10 May 2010 quantity minimized is the sum of the properly weighted squared residuals in x and in y. As an alternative to
Received in revised form 6 July 2010 this “total variance” (TV) method, “effective variance” (EV) methods convert the uncertainty in x into an
Accepted 10 July 2010
effective contribution to that in y, and though easier to use are considered to be less reliable. There are at
Available online 15 July 2010
least two EV methods, differing in how the weights are treated in the optimization. One of these is identical
Keywords:
to the TV method for fits to a straight line. The formal differences among these methods are clarified, and
Least-squares Monte Carlo simulations are used to examine the statistical properties of each on the widely used straight-
Total variance line model of York, a quadratic variation on this, Orear's hyperbolic model, a nonlinear binding (Langmuir)
Effective variance model, and Wentworth's kinetics model. The simulations confirm that the EV and TV methods are
Monte Carlo statistically equivalent in the limit of small data error, where they yield unbiased, normally distributed
Data analysis parameter estimates, with standard errors correctly predicted by the a priori covariance matrix. With
Uncertainty in x and y increasing data error, these properties fail to hold; and the TV method is not always statistically best.
Nonetheless, the method differences should seldom be of practical significance, since they are likely to be
small compared with uncertainties from incomplete information about the data error in x and y.
© 2010 Elsevier B.V. All rights reserved.

1. Introduction
Much has been written on the least-squares (LS) analysis of data
having uncertainty in more than one variable, as comprehensively
reviewed through 1991 by Macdonald and Thompson [1]. Most such
works have dealt with just two uncertain variables, x and y, usually
assumed to fit a straight line, with the model of York [2] being a
frequent target. This problem has been revisited recently by York et
al. [3], and is the subject of at least two reviews [4,5]. In the present
work I show that the “prevailing wisdom” on this subject is not
completely correct. In particular the accepted “best” method does
not always yield statistically best results. Most prior studies have
dealt just with specific problems and data sets. In the present work, I
use Monte Carlo methods to produce typically 105 simulated data
sets for a given model, thereby obtaining what I believe are the first
valid statistical comparisons of several approaches to this problem.
The comparisons are limited to the simplest case of random normal
(Gaussian) error added to true models.
It is generally thought that the best solution is obtained by
minimizing the quantity STV,
which was proposed long ago by Deming [6] and implemented in
iterative form as early as 1972 [7–12]. In Eq. (1) the δ′is are residuals in
the uncertain variables, here just x and y. A complicating factor in
specifying and minimizing STV is the need for “calculated” or
“adjusted” values (designated yi and xi) for both y and x, because
the residuals are Yi − yi and Xi − xi, where upper-case letters represent
the observed values. Minimum-variance estimates of the model
parameters are assumed to result from the use of weights inversely
proportional to variance—wxi = Cσ−2 −2
xi , wyi = Cσyi , etc. (C a propor-
tionality constant, the same for all variables)—as can be proven to
hold for linear LS, in which there is a single uncertain variable (y).
The subscript TV in Eq. (1) is short for “total variance,” to distinguish
it from similar quantities labeled “effective variance,” in which the
variance in x is converted into an effective contribution to the variance
in y = f(x) using error propagation, to yield weights [13–16]
−1 2 2 2
w ∝ varðyÞeff = σy + ðdy=dxÞ σx : ð2Þ
For y = a + bx, σy,eff 2 = σy2 + b2σx2, yielding wi ∝ (σy2 + b2σx2)−2. The
minimization target is

2
2 2
SEV = ∑wi δyi : ð3Þ
STV = ∑wxi δxi + wyi δyi + …; ð1Þ
This approach attempts to take uncertainty in x into account, but still
treats x as error-free; thus only the y residuals, δyi = Yi − yi, need be
E-mail address: joel.tellinghuisen@vanderbilt.edu. evaluated.

0169-7439/$ – see front matter © 2010 Elsevier B.V. All rights reserved.
doi:10.1016/j.chemolab.2010.07.003
 Author's personal copy
J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
The EV method is easy to use with standard programs for
weighted linear (LLS) and nonlinear LS (NLS), but it has been
criticized for some shortcomings [17,18]: (1) The minimum in SEV
is not the same as that in STV, and thus the solutions are not
“optimal.” (2) When the weights in x and y are constant (but not
necessarily the same), the solution to the straight-line fit becomes
just the unweighted or ordinary LS (OLS) solution for regression of
y upon x and converges in one iteration. This is true even when the
uncertainties in x greatly exceed those in y, making regression of x
upon y more sensible. (3) It follows that the results depend upon
the manner in which the fit relation is expressed. In contrast, the
TV method yields a single solution for all mathematically
equivalent ways of writing the fit relation [19].
With a slight change in the minimization procedures, we obtain
!2
δyi
SEV2 =∑
σyi;eff
which is formally identical to Eq. (3), but is written differently to
indicate that now the dependence on the adjustable parameters in
the weights is included in the optimization (see below). William-
son showed that minimizing SEV2 is equivalent to minimizing STV
for a straight-line fit [20]. Yet the computations can still be
accomplished with NLS routines designed for single uncertain
variables, because the weights are evaluated at the observed Xi and
Yi, again making δyi the only required residual [21]. The equivalence
of STV and SEV2 appears to hold only for the straight-line fit, but it is
tempting to imagine that the EV2 approach approximates the TV
method in general applications [18].
Does minimizing STV yield statistically best estimates of the
parameters in any fit model? The question is relevant, because for
NLS there is no guarantee that minimizing the variance will also
minimize the parameter uncertainties [22,23], in part because many
NLS parameters do not have finite variance. Even finite-variance
nonlinear estimators are usually biased and nonnormal [24]—
including the TV results for the straight line [25]. Further, inverse
regression—x upon y when x is error-free and y is uncertain—
actually yields better precision in calibration [26–30]. Thus neglect-
ing the uncertainty in x (as when the EV method degenerates to OLS
for constant σx and σy) might give more precise estimates in some
cases of error in both x and y; and in fact it does [31]. Such questions
cannot be addressed through calculations on single data sets for
single problems—the approach that has dominated the literature on
this problem; but they can be answered through Monte Carlo
simulations.
In subsequent sections, I first review the mathematical relations of
LLS in their compact matrix form. I then show how simple extensions
give the relations that can be used for iterative solution of NLS
problems, first for a single, explicit uncertain variable, then for the EV
and TV methods. The latter results are largely consistent with
Deming's early treatment. I further emphasize a point that has not
always been clear from previous studies: All of the results are exact for
the parameters and their standard errors (SEs) in the limit of small
data error, but become increasingly biased for NLS with increasing
data error. These properties are illustrated for York's linear model
through MC simulations employing a range of scaling factors for the
data error. The parameter SEs are obtained from the covariance
matrix, replacing algebraically tedious treatments [3,32] by a simple
computation that applies for any fit model, linear or nonlinear. The
comparisons are extended to other models, confirming that the EV2
and TV methods yield different but quite close results for models
other than the straight line. As hinted earlier, the TV method is not
always statistically better than the EV method; but the differences will
seldom be practically important.
161
2. Theory and computational methods
2.1. Linear least-squares
The mathematics and procedures of linear and nonlinear least-
squares are readily available from many sources [24,33–35]; here I
will repeat just enough to provide continuity to the modified
expressions needed for the EV and TV methods. A least-squares
problem is linear when there is a single uncertain variable (y), and the
relations among the measurements Y, the independent variables (x, t,
u,..., all error-free), the adjustable parameters β, and the fit residuals δ
can be expressed as
Y = Uβ + δ; ð5Þ
where Y and δ are column vectors containing n elements, β is a vector
containing the p parameters, and the design matrix U has n rows and p
ð4Þ columns, and depends only on the values of the independent variable
(s) and not on β or Y. Minimization of SLLS = Σwi δ2yi with respect to the
p parameters yields the equations,
T T
U WUβ ≡ Aβ = U WY; ð6Þ
where UT is the transpose of U, and the square weight matrix W is
diagonal, with n elements Wii = wi. The solution to Eq. (6) is
−1 T
β=A U W Y; ð7Þ
where A− 1 is the inverse of A. Knowledge of β permits calculation of
the residuals δ from Eq. (5) and then SLLS, which can be written
SLLS = δT W δ .
The parameter standard errors (SEs) are obtained from the
variance–covariance matrix, V ∝ A− 1, in which the parameter var-
iances appear on the diagonal and the covariances off-diagonal. If the
data errors σyi are known absolutely, we take wi = σ−2
yi and obtain
−1
Vprior = A : ð8Þ
If the scale of the σyi is not known, we use
SLLS −1
Vpost = A ; ð9Þ
ν
where ν = n − p is the number of statistical degrees of freedom. The
most common use of Vpost is in OLS, where the data are thought to be
homoscedastic (constant σy) but the scale is not known. Then wi = 1 is
used for all data, and the resulting value of (SLLS/ν) is an estimate of σ2y .
Vpost is also appropriate for weighted LS (WLS) fits where unequal
weighting results from a data transformation (e.g. logarithmic
conversion), and where the form of the data error is known (e.g.,
proportional to signal), but not its scale. The data errors must be
known within a scale factor, or the parameter SEs will be unreliable;
the common example of this fault is use of OLS on heteroscedastic
(nonconstant σy) data. The resulting error in Vpost depends on the
span of the weighting errors across the data set [36]. On the other
hand, the use of variance sampling estimates s2i to assign weights
wi = 1 / s2i , can actually be worse than complete neglect of weights,
when the number of replicates is small [37].
In MC simulations the analyst sets the data error, so Vprior is
appropriate, as it can also be in cases where σyi is well characterized
from accumulated data. Vprior is exact for LLS with known σyi (see
below), so it provides a way to check MC computational routines.
2.2. Properties of linear LS solutions
We assumed the measured Y have random, independent error
(correlated error would lead to off-diagonal elements in W). They also
 Author's personal copy
162 J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
must be of finite variance. Then if they are unbiased, the LLS parameter
estimates β will be unbiased, with variances given exactly by Vprior. If
further the data error is normal (Gaussian), the β estimates will
similarly follow the Gaussian distribution,
" #
2
ðu−μ Þ
PG ð μ; σ; uÞ = C exp − ; ð10Þ
2σ 2
with C a normalization constant and u representing an estimate of βi
having true value μ and SE σ. This distribution will hold even with
incorrect weighting, but unless wi ∝ 1 / σ2yi, the estimates of β will not
be minimum-variance. With properly weighted, normal data, LLS is
also a maximum-likelihood method. Under these conditions, SLLS will
be χ2 distributed (mean ν, variance 2ν) and SLLS/ν will follow the
reduced χ2 distribution (mean 1, variance 2/ν),
ðv–2Þ = 2
Prcs ðzÞ = C z expð–vz = 2Þ;
where z = χ2ν (=χ2 / ν) and C is a normalization constant. Similarly
Vpost-based variance estimates s2βi follow a scaled χ2 distribution. And
the quantities (βi − βi,true) / sβi follow the t-distribution for ν degrees
of freedom. In the large-ν limit, both the χ2 and the t distributions
become Gaussian.
From the properties of the χ2 distribution, sampling estimates of
variance have relative uncertainty (2/ν)1/2, and the SEs have relative
uncertainty (2ν)−1/2. This “uncertainty in the uncertainty” is
generally ignored in data analysis texts; for small ν, it can significantly
increase the effective parameter uncertainty—by 25% for a 10-point
model, for example. Formally, the uncertainty in the SE leads to the
need for the t-distribution to assess parameter confidence limits.
2.3. Nonlinear LS with a single uncertain variable
The equations must now be solved by successive approximation.
Let the fit relation be
Fðx; y; βÞ = y−f ðx; βÞ = 0;
such that exactly fitting data give F = 0. Eq. (12) is explicit in the
dependent variable, so correct weighting is again wi ∝ 1 / σ2yi. Starting
with an approximate solution β0, we obtain a new estimate,
β1 = β0 −Δβ
where, in what is sometimes known as the inverse Hessian method
[35], the corrections Δβ are obtained by solving equations formally
the same as Eq. (6),
T T
J W J Δ β = A Δ β = J W F0 :
F0 contains the n values from Eq. (12), and the elements of the
Jacobian matrix J are Jij = (∂Fi / ∂βj). All quantities are evaluated for the
ith point using the current parameter estimates β0. These equations
become the same as Eqs. (6) and (7) for linear functions f(x;β).
The relation between V and A is the same as for LLS, except that
here A (hence V) may include dependence on Y and β, thus may vary
for different data sets. We define the “exact” V as the Vprior for exactly
fitting data.
2.4. Nonlinear LS with implicit and multiple uncertain variables
The fit relation is still expressed as F(x,y;β) = 0, but one or more
uncertain variables do not appear explicitly. For each point i, wi ∝ 1 / σ2Fi,
where error propagation gives the effective variance for two uncertain
variables (x and y) as
2 2 2 2 2
σF = ð∂F =∂xÞ σx + ð∂F =∂yÞ σy ;
with all quantities evaluated at the observed points (Xi,Yi). Eq. (15) is
extendable in an obvious way for more than two uncertain variables.
This treatment assumes the x- and y-errors are independent from
each other and from those at all other points. With these wi, the
computations proceed as in Eqs. (13) and (14), except that through
Eq. (15), the weights generally depend upon the parameters and must
be adjusted in the iterations. This is the EV treatment, and it leads to
minimization of SEV in Eq. (3).
The weights can be handled differently: Include a factor of w1/2i =
1 / σFi with each element of J and F and replace W by the identity
matrix. With one further change, we obtain the EV2 method. Replace
the fit relation in Eq. (12) by
Gðx; y; βÞ = Fðx; y; βÞ = σF : ð16Þ
Then the elements of the J matrix—now the partial derivatives of G wrt
βj—include the dependence of the weights on the parameters. [These
ð11Þ
derivatives are commonly computed by finite difference, evaluating G
for the ith point at βj + ε and βj − ε and dividing the difference by 2ε;
here this also requires evaluating Eq. (15) at βj ± ε. Again, all
quantities are evaluated at (Xi,Yi).]
2.5. Nonlinear LS: total variance method
Most of the foregoing for NLS still applies, but with two changes:
We now also adjust all uncertain variables, and we evaluate all
derivatives on the adjusted curve. Deming [6] expressed the residuals
δx and δy in terms of Lagrangian multipliers λ, given by
λ = F0 −JΔβ ð17Þ
where F0 still designates evaluation with β0 at the observed points X
and Y, F0i = F(Xi,Yi;β0). (On the calculated curve, convergence should
achieve F = 0.) Then
λi ∂F
δxi = ð18Þ
wxi ∂xi
ð12Þ
and similarly for δyi (and all other uncertain variables). Numerical
tests show that the results from this procedure depend weakly on the
way F(x,y;β) is expressed. However, the modifications of Powell and
Macdonald [7], Britt and Luecke [8], and Jefferys [11,12] remove this
ð13Þ dependence. In place of F0 in Eqs. (14) and (17), we use F0′, having
elements
∂F ∂F
F0′ i = Fi + δ + δ +… ð19Þ
∂xi xi ∂yi yi
where all quantities on the rhs are evaluated on the calculated curve
ð14Þ
(giving Fi = 0 on convergence). STV (Eq. (1)) is also correctly given by
STV = FT0 W F0 (Deming) or F0′T W F0′. [Numerical checks show that
Deming's expressions yield F(x,y;β) = 0 on the calculated curve only
in certain cases, like the straight-line fit. Use of F0′ from Eq. (19) gives
this result in all cases.]
2.6. Properties of NLS solutions
Monte Carlo simulations have demonstrated that NLS solutions for
single uncertain variables behave like LLS in the limit of small data
error: unbiased, normally distributed parameter estimates with SEs
given by the exact Vprior, and χ2-distributed S values [24]. With
increasing data error these properties fail to hold. The parameter
estimates become nonnormal and biased, with confidence limits
exceeding those given by Vprior. A good guideline for the reliability of
the SEs is a “10% rule”: If the SE is less than 10% of the magnitude of the
parameter, it should yield confidence limits reliable within 10%. Most
ð15Þ previous efforts to check these properties for NLS with multiple
 Author's personal copy
J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
uncertain variables, including the TV method, have been flawed, as is
discussed below.
With nonlinear functions f(x;β) there may be multiple minima in
S, or none at all; and convergence on a minimum may require quite
close initial values β0. Use of the Marquardt algorithm can greatly
improve the convergence properties for NLS with a single uncertain,
explicit variable [38]. With it, the diagonal elements of A are increased
any time SNLS rises on iteration, enhancing a more robust gradient
mode of optimization [34,35]. However, with multiple uncertain
variables, there are problems using this method, because convergence
can occur from below (see below).
2.7. Monte Carlo simulations
Simulated data are generated by adding random normal error in x
and y to points on the true curve, using standard methods [24,35]. The
resulting data are then fitted to the model under study by the TV and
EV methods. This cycle is repeated for typically 105 simulated data
sets, with evaluation of means and variances for key quantities. The
MC precisions are governed by the estimated standard errors from the
MC statistics, e.g., sa/N1/2 for the parameter a when it and its standard
error are estimated from N analyzed data sets. The uncertainty in sa is
governed by the statistics of χ2, so it is uncertain by 1 part in (2N)1/2,
which is 0.224% for N = 105. Also, when the MC results are binned to
yield histograms and the latter are fitted, the bin counts are weighted
in accord with Poisson statistics [34], meaning they are assigned
uncertainties = count1/2.
2.8. Program tests
The MC computations were done with programs written in
Microsoft FORTRAN for PCs, with spot checking of results using the
KaleidaGraph program [39]. Under tight convergence, the TV program
yielded 10-figure agreement with “exact” parameter values for York's
model [4,5] and 12-figure agreement with STV [4,12] The standard
errors (SEs) agreed with those of Reed (Vprior-based, evaluated at
calculated points) [32], which are given to 7–8 significant figures in
his Ref. [15]. Agreement was obtained also for the examples given by
Powell and Macdonald [7] and Britt and Luecke [8]. And I have verified
that the TV code gives identical results for parameters and SEs in all
ways of expressing the fit relation among the variables and
parameters. Deming's equations for the TV method gave exact
agreement with Eq. (19) for straight-line fits, but gave very small
differences for other models (see below). Interestingly, they also gave
slightly smaller STV values in the cases checked so far (though they do
not satisfy F = 0 on the calculated curve).
In the present MC computations and others in the past, I have
observed that convergence on SEV and STV can occur from below,
evidently as a consequence of the iterative changes in the weights (EV
method) and in the residuals (TV). This behavior would seem to rule
out use of the Marquardt procedure. However, in the EV method it is
still possible to use this tool for a given set of estimated weights wi,
using Eq. (15) to reassess these only after Δβ ≈ 0 for the current
weights. A similar procedure should work to achieve Δβ ≈ 0 before
the weights and variables are readjusted in the TV method. I have not
observed convergence from below in the EV2 method.
Codes for carrying out TV computations have been available since
1972; in fact the program of Powell and Macdonald [7] is still
downloadable from Macdonald's web site. Their paper includes a
number of test problems, on which they first used “Deming's method”
(actually the EV method) to obtain starting estimates of parameters,
for subsequent TV computations. Those choosing to check their own
routines on these examples should be aware of a minor flaw: The Ref.
[7] parameter SEs were incorrect but were later fixed [9,10].
163
3. Results and discussion
3.1. The York model
York [2] devised strongly heteroscedastic data weights for the
variables of a 10-point straight-line model originally considered by
Pearson [40], yielding a model that has served as a touchstone for x-
and-y-error fitting methods for over 40 years [1,4,5]. York and
colleagues included this example among others in their more recent
study [3], where they used MC methods to check the reliability of
predicted uncertainties for the parameters. They also reported that
histograms of the parameters from all these tests “almost perfectly
matched Gaussian distributions.” My results contradict that claim and
support a different interpretation of the reliability of the parameter
SEs.
For simplicity here, the true model was taken to have the rounded
values, α = 5.5 and β = −0.48. York's weights were converted to data
σ values for x and y using, e.g., wxi = 1 / σ2xi, yielding the model
illustrated in Fig. 1. Results from MC simulations run as a function of a
scaling factor f for the data errors (Table 1) show that the parameter
biases increase as the square of the data error (hence as f2) and
become statistically significant beyond f = 0.25. However, at the
precision level of 105 simulated data sets, the sampling SEs follow
predictions until f = 1. Fits of histograms of the parameters (Fig. 2)
were Gaussian for f = 0.1 but clearly not for f = 0.25 and larger. (For
ν = 28, values of χ2ν as large as 2.0 occur less than 0.1% of the time
[34].) On the other hand, the reduced χ2 distribution (Eq. (11)) holds
for the STV values even for f = 2.3, but fails for the EV results at f = 1.
This behavior by the TV results—S being χ2 distributed for data errors
much larger than needed to give non-Gaussian parameter distribu-
tions—is similar to that observed previously for single uncertain
variable NLS [24].
The behavior of sa and sb in Table 1—increasing excess with
increasing f—is not the apparent behavior in Table II of York et al. [3], in
which some of the comparisons are positive, some negative. However,
these authors took the MC estimate as the reference, changing the signs
of the discrepancies from my convention. Thus, for the original York
model, the results for data set 3 in Table II of Ref. [3] mean that the MC
estimates of σa and σb were higher than predicted by 0.251% and
0.464%, respectively (with uncertainty 0.022% for N = 107). My excesses
for f = 1 in Table 1 (TV analysis) are 0.28% and 0.46% for σa, 0.66% and
0.60% for σb (±0.22% for N = 105)—in adequate agreement. There is not
enough information to check other results in Table II of Ref. [3].
However, the largest discrepancies exceed −1%, meaning the MC
results show excesses in these amounts; and these occur for a case
Fig. 1. Straight-line model of Pearson and York, as adapted for MC tests. Error bars are
1σ for scaling factor f = 1. Dashed lines show the 1 − σ errors on the fitted function.
 Author's personal copy

164 J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169

Table 1
Monte Carlo results (105 10-point data sets) for York model. a

b b
f a σa sa −10 b σb sb χ2v (v)c
b b
(MC) (MC) (MC) (MC)

0.10 5.5001 0.029708 0.02972 4.8002 0.0058287 0.005832 1.28 (28)

5.4993 0.02973 4.7987 0.005832 1.24 (29)
0.25 5.5004 0.074270 0.07435 4.8012 0.014572 0.01460 2.05 (28)
5.4957 0.07443 4.7916 0.01460 2.11 (29)
0.50 5.5019 0.14854 0.1488 4.8050 0.029143 0.02920 3.90 (28)
5.4831 0.1494 4.7668 0.02927 3.48 (27)
1.00d 5.5060 0.29708 0.2979 4.8156 0.058287 0.05867 19.7 (28)
5.4299 0.3031 4.6614 0.05919 17.4 (29)
1.00d 5.5053 0.29708 0.2985 4.8153 0.058287 0.05864 17.6 (28)
5.4290 0.3042 4.6608 0.05931 14.0 (30)
2.00 5.5241 0.59416 0.6024 4.8647 0.11657 0.11916 62.1 (28)
a
True intercept 5.5, true slope − 0.48, and data errors of York scaled by factor f. In
each pair of results lines, the first comes from analysis with the TV method, the second
with EV.
b
σ values are exact Vprior-predicted SEs, while s values are their MC sampling
estimates.
c
Reduced χ2 values obtained fitting histograms of the estimated slope values to 3-
parameter Gaussian functions, with bin counts weighted as the count− 1 (Poisson
approximation).
d
Results for two different random number seeds for f = 1.00.

where the parameter relative SEs (RSEs) exceed 10%. By contrast,

almost all positive discrepancies are within 2σ (0.045%) of statistical
agreement; and these occur in cases where the RSEs are much smaller
than 10%.
The results from EV analysis in Table 1 are inferior to those from TV
analysis, with parameter SEs slightly larger and biases substantially
larger. However, the magnitudes of these losses are still arguably
negligible up to f = 1, where the biases amount to about 1/4 of the
parameter SEs, making the rms errors 4% larger than for TV analysis
[rms error = (bias2 + SE2)1/2].
The results in Table 1 do not include EV estimates at f = 2, because
this method produced many nonconvergent data sets for f N 1.3. The
effort to understand this problem led to further insight into the
differences between the TV and EV methods, as follows. We first fix
the slope at a range of reasonable values b′ and compute the weights
from Eq. (2). When we then solve for the value of a that minimizes the
sum S for each b′, the resulting sums are STV(b′) = SEV2(b′). Minima in
these curves represent the TV solutions. Alternatively, if we solve for
both a and b, the minimized sums constitute SEV(b′). The EV solution Fig. 2. York model results from 105 MC data sets. (a) Binned estimates of the slope b for
occurs where the resulting value of b equals b′. These points are TV results obtained with two error scaling factors; (b) binned values of S/ν for both
illustrated for the original York data set in Fig. 3. Note that there is a methods. The weighted fits to the normal distribution (Eq. (10)) in (a) gave χ2 = 36.0
(f = 0.1) and 2276; fits to the reduced χ2 distribution (Eq. (11)) in (b) gave ν = 8.02 ±
second TV solution to the York problem at a much higher value of STV,
0.04 and χ2 = 39.3 for the illustrated curve, ν = 7.84 ± 0.03 and χ2 = 120.8 for the EV
but only a single EV solution. results. The smaller χ2 values in each case are within the 15% and 30% probability
Fig. 4 is a similar display for a simulated data set that failed to ranges, respectively.
converge for the EV method. There are again two solutions for the TV
method, with the preferred one near b ≈ −0.35. At slightly larger b′ biases are worst for the TV method, but at f = 1 they increase the
there is a near solution in the EV method, but the two curves fail to rms parameter uncertainties by b3%.
intersect. The apparent solution near b = 0.05 is unstable in the usual Table 2 includes information about another property of the
iterative procedure, as increases in b′ lead to decreases in b and vice computations that can be a practical concern: the divergence rate.
versa, producing an oscillatory divergence. [Readers interested in At f = 1 about 4% of the TV computations failed to converge, even with
pursuing this case further can find the data listed in Supplement use of damping formulas to limit the magnitude of the iterative
Table 1S.] changes in the parameters, which were often oscillatory. [No effort
has yet been made to include Marquardt-type algorithm modifica-
tions for this model.]
3.2. A quadratic model

In an extension of the York model, I have added a quadratic term
0.03 x2, for which the quadratic coefficient (c = 0.03) has large
relative uncertainty, σc = 0.017035, for f = 1. The TV and EV2
methods no longer yield identical results, though their solutions are
very close. Results are given for all three methods in Table 2.
Surprisingly, for this model the EV method outperforms TV and EV2
on both bias and precision. However, as before, these results are
more of fundamental than of practical concern. For example, the
3.3. Orear's model
Orear presented results for all three methods under study here for
a model having the fit function, y = ax − b / x [16]. His corrected results
for a specific 5-point data set showed the TV and EV2 solutions to be
very close, with identical S values. However, on pushing the precision
of the calculations, I find the S values to differ slightly—STV = 2.13377,
and SEV2 = 2.13318. The closeness persists when the σxi values are
 Author's personal copy
J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
Fig. 3. Graphical illustration of TV and EV solutions to the York problem. The S curves
(logarithmic scale to right) show the dependence on slope b′ used to set the weights wi
when minimizing just with respect to a (TV method) and with respect to both a and b
(EV). Also shown are the resulting values of b in the EV method (left scale). The EV
solution occurs where b = b′, given by the intersection of this curve with bEV (where
also the two S curves coincide). The TV solutions occur at the minima of STV(b′) and
include a previously unrecognized solution at a = 1.6326, b = 0.24879, and S = 231.100.
The more familiar solution occurs slightly to smaller b from the EV solution, as can be
seen from the inset.
made much more variable than Orear's (nearly homoscedastic)
values, so this property is not an accident of this aspect of his model.
A limited number of MC computations were conducted for this
model as a function of a scale factor applied to the data errors. Because
of the close agreement between the TV and EV2 results, MC results
were generated for just the EV and EV2 methods. The results (Table 3)
show the expected pattern of increasing bias in the parameters and
their SEs with increasing data error. But interestingly, the EV
parameter SEs are smaller than those for the EV2 method by enough
to make the EV rms errors smaller—a behavior similar to that
observed for a straight-line model with constant σx [31]. Again the
differences are unlikely to be of practical significance.
3.4. A nonlinear model: equilibrium binding
The Langmuir model is inherently nonlinear for any assumption
about the data error. In Ref. [19], the emphasis was the special
problem in which the “independent” variable—the free ligand or
Fig. 4. TV and EV behavior for a data set simulated with f = 1.4 that failed to converge
for the EV method. Note the logarithmic scale to the right for the S curves, linear scale
left for b. STV lies everywhere above SEV.
165
sorbate concentration—was the measured quantity, hence subject to
experimental uncertainty. In such cases the model is usually
expressible as
abx
“y” ≡ x0 −x = ð20Þ
1 + bx
where x0 and x are the prepared and measured equilibrium
concentrations, respectively. When “y” is treated as the dependent
variable here, the EV weights require attention to correlation [19].
However, any manner of expressing the relation in terms of its actual
variables, x0 and x, will yield correct results in the EV and TV methods
without any such attention to correlation. Accordingly, in discussing
results for this model below, I will replace x0 by y for notational
consistency.
Monte Carlo results were obtained for the 7-point data structure
from Table 4 of Ref. [19], with two different error structures for x and
y, in all four combinations. The two error structures were roughly 6%
proportional error for each and nearly constant error, as given in
Table 4. Results for combinations of these errors with various scale
factors are given in Table 5.
The statistical properties of this model are not radically different
from those for the earlier models examined in this work. In the limit of
low data error, the parameter estimates and their Vprior-based SEs
become unbiased for all three methods. The parameters are normally
distributed in this limit, just as for the earlier models (but not shown
in Table 5). With increasing data error there are no clear winners
among the three methods: Often one method shows smallest bias and
best precision for one of the two parameters but not for the other.
Also, the sampling estimates of the SEs and their rms values from
Vprior typically exceed the “exact” values progressively as the data
error increases, though there are interesting exceptions, like an
undershoot in sb and V1/2 22 for the EV method in the last two data
structures. I conclude that all three methods are statistically
comparable for this model, in which the parameter SEs are pursued
to magnitudes that exceed 10% of the parameter values. Further, the
V-based SEs (the pertinent estimates for an analyst working with a
single data set) adequately reflect the true precisions, even when both
exceed the “exact” values.
The results for the EV2 method—both parameters and SEs—often
fall between those for the TV and EV methods in Table 5, but are closer
to the former. This behavior holds also for the individual data sets. For
example, for the 4th set of results in the table, the TV–EV2 differences
for a showed a standard deviation of 0.013, while the other two
combinations yielded values of 0.063 (TV–EV) and 0.052. For b the
corresponding values were 0.007, 0.043, and 0.037. Thus, expectations
for close agreement between the TV and EV2 results are again borne
out in the results, though the differences for this model are greater
than for the Orear model.
The S distributions were examined for the largest data error
included in Table 5. The results were as observed for the York model:
The STV values satisfactorily followed the χ2 distribution, while the SEV
did not. The EV2 results (equal to TV for the York model) were not
satisfactorily fitted by the χ2 distribution, but were closer than the EV
results.
The Langmuir binding model has been the subject of much work,
as reviewed in Ref. [19]. These efforts have included some Monte Carlo
studies, but almost all dealing with just a single uncertain variable. In
one exception, Alper and Gelb used MC methods to examine a related
dimerization model having three parameters (K and two molar
absorptivities), and uncertainty in the measured absorbance and in
the total concentration [41]. Using the TV method, they found MC
confidence limits a factor of 3 smaller than predicted by Vprior. I have
examined their model and found complete consistency with the
behavior noted for all models in the present study—parameter biases
about 1% in magnitude and sampling estimated SEs only a few percent
 Author's personal copy

166 J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169

Table 2
Monte Carlo results (105 10-point data sets) for quadratic model.a

f a 10σa 10 sa − 10 b 10σb 10 sb 102 c 102 σc 102 sc χ2v (v)b Div.c

0.025 5.5000 0.12156 0.1217 4.7999 0.046305 0.04630 2.9998 0.042586 0.04257 1.11 (30) 0
5.5000 0.1216 4.8001 0.04630 3.0000 0.04256 1.14 (30) 0
5.5000 0.1217 4.7999 0.04630 2.9998 0.04257 1.10 (30) 0
0.05 5.4999 0.24313 0.2434 4.7993 0.092610 0.09265 2.9990 0.085172 0.08521 1.60 (30) 0
5.5000 0.2433 4.7999 0.09260 2.9997 0.08514 1.50 (30) 0
5.4999 0.2434 4.7993 0.09265 2.9990 0.08521 1.59 (29) 0
0.10 5.4992 0.48625 0.4878 4.7952 0.18522 0.1865 2.9940 0.17034 0.1720 3.94 (30) 0
5.4996 0.4872 4.7977 0.1860 2.9968 0.1714 3.42 (30) 0
5.4992 0.4877 4.7950 0.1864 2.9938 0.1719 3.82 (30) 0
0.25 5.4958 1.2156 1.2270 4.7737 0.46305 0.4722 2.9663 0.42586 0.4400 18.0 (32) 0
5.4992 1.2181 4.7938 0.4655 2.9888 0.4309 13.8 (32) 0
5.4958 1.2264 4.7737 0.4717 2.9666 0.4393 17.7 (32) 0
0.50 5.4816 2.4313 2.526 4.6844 0.92610 1.0106 2.8517 0.85172 0.9824 280
5.5003 2.463 4.7960 0.9604 2.9816 0.9089 32.4 (33) 2
5.4831 2.547 4.6929 1.0260 2.8617 1.0028 7
1.00 5.4386 4.8625 5.315 4.4047 1.8522 2.288 2.4764 1.7035 2.404 3832
5.5177 5.034 4.9053 2.051 3.0908 2.051 80.1 (42) 614
5.4685 5.498 4.5613 2.419 2.6530 2.580 58
a
True coefficients 5.5, − 0.48, and 0.03, and data error structure of York scaled by factor f. In each trio of results lines, the first is from analysis with the TV method, the second EV,
and the third EV2.
b
Reduced χ2 values for fitting histograms of the estimated c values to 3-parameter Gaussians.
c
Number of divergent data sets.

above predictions, which is well within the 10% guideline, since two of literature. It is explicit in t, so we can treat that as dependent, re-
the parameters have predicted RSEs N 0.17. It is possible that their expressing as
anomalously small MC confidence limits stemmed from the afore-
mentioned problem of convergence from below, for which their t = ½ð2P0 −PÞ
1−n
−P0
1−n
ðn−1Þ
−1 −1
k ; ð22Þ
Marquardt algorithm might have effectively frozen the parameters
near their starting values, preventing true optimization. [In private and computing the effective variance contributions from P to t as
correspondence, Alper has acknowledged this possibility and noted (dt/dP)2 σ2P. Alternatively, we can solve for P, obtaining
that later checks showed general agreement between MC and
predicted SEs.] −1 q
P = 2P0 −exp½q lnðP0 −qktÞ; ð23Þ

3.5. Wentworth's kinetics model

Wentworth [42] used Deming's formalism to illustrate the fitting
of kinetics data (7 points) in the form of pressure P as a function of
time t, to the relation
1−n 1−n
ð2P0 −PÞ −P0 = ðn−1Þk t;
in which the parameters are P0, n, and the rate constant k. He defined
F(x,y;β) as the rhs of this equation minus the lhs and took the
uncertainties in both variables to be 1.00 in their respective units.
Comparison of his results with present findings confirms that he did
not adjust the variables, thus meaning that these were EV results.
Eq. (21) is not explicit in the intuitive dependent variable P, so these
are not the results one would expect on the basis of most EV
Table 3
Monte Carlo results (105 5-point data sets) for Orear model.a
where q = 1 − n, and apply the EV treatment to P. Other implicit
versions of Eq. (21) can be obtained a number of ways, including by
taking logarithms, of which I consider
q
Fðt; P; βÞ = q lnð2−P = P0 Þ−lnð1−qkt = P0 Þ: ð24Þ
ð21Þ
Results for the EV and EV2 methods using all four fit relations are
given in Table 6, along with results for the TV method, as
implemented using Deming's equations (TVD) and using Eq. (19)
(identical to stated precision for all four). As anticipated, the EV
(Eq. (21)) results agree most closely with those provided by
Wentworth (and are most discordant of all), while all EV2 and TV
results display better mutual agreement. [The high precision in
Table 6 is intended to facilitate tests of algorithms, for which
numerical results often provide a better answer than formal
descriptions.]
Monte Carlo simulations were run on this model for a range of data
error scaling factors from 0.1 to 1.0. Results were mostly in line with

f 103 a (MC) 104 σa 104 sa (MC) 10−5 b (MC) 10−5 σb 10−5 sb (MC)

0.05 1.0699 0.10354 0.1036 6.2492 0.057967 0.05796

1.0701 0.1036 6.2505 0.05798 Table 4
0.10 1.0694 0.20708 0.2067 6.2465 0.11593 0.1158 Data error structures for the binding constant model.a
1.0703 0.2070 6.2516 0.1159
y σy1 σy2 σx1 σx2
0.25 1.0661 0.51771 0.5157 6.2281 0.28984 0.2887
1.0718 0.5197 6.2603 0.2910 0.1 0.006 0.10 0.003 0.05
0.50 1.0540 1.0354 1.025 6.1618 0.57967 0.5743 0.2 0.012 0.15 0.008 0.05
1.0770 1.056 6.2906 0.5916 0.5 0.03 0.15 0.015 0.05
1.00 1.0071 2.0708 1.993 5.9019 1.1593 1.117 1.0 0.06 0.10 0.035 0.10
1.1002 2.253 6.4240 1.264 2.0 0.12 0.10 0.09 0.09
5.0 0.3 0.15 0.35 0.10
a
True parameter values are 1.06×10−3 and 6.25×105; data error scaled by factor
10.0 0.6 0.10 0.8 0.05
f. In each pair of results lines, the first is from analysis with the EV method, the
second with EV2. a
True parameter values, a = 2.1 and b = 1.2.
 Author's personal copy

J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169 167

Table 5
Monte Carlo results (105 10-point data sets) for binding model.a

f (i, j)b a σa sa V1/2

11
c
b σb sb V1/2
22
c

0.1 (1,1) 2.1006 0.049472 0.04952 0.04956 1.2007 0.036955 0.03695 0.03701
2.1002 0.04950 0.04955 1.2009 0.03695 0.03703
2.1006 0.04952 0.04955 1.2007 0.03695 0.03702
0.2 (1,1) 2.1028 0.098945 0.09927 0.09937 1.2024 0.073910 0.07405 0.07421
2.1010 0.09913 0.09930 1.2032 0.07407 0.07431
2.1025 0.09924 0.09935 1.2025 0.07404 0.07421
0.4 (1,1) 2.1115 0.19789 0.2004 0.2009 1.2093 0.14782 0.1494 0.1499
2.1042 0.1994 0.2002 1.2124 0.1496 0.1507
2.1104 0.2002 0.2007 1.2097 0.1493 0.1499
1.0 (1,1) 2.1773 0.49472 0.5408 0.5450 1.2583 0.36955 0.3980 0.4017
2.1292 0.5227 0.5332 1.2783 0.4017 0.4156
2.1691 0.5363 0.5411 1.2604 0.3970 0.4019
0.4 (1,2) 2.1154 0.22051 0.2260 0.2255 1.2271 0.23530 0.2455 0.2466
2.1077 0.2245 0.2244 1.2338 0.2466 0.2489
2.1142 0.2257 0.2253 1.2276 0.2455 0.2468
0.8 (1,2) 2.1687 0.44102 0.4881 0.4979 1.3131 0.47061 0.5637 0.5731
2.1357 0.4737 0.4836 1.3421 0.5747 0.5947
2.1637 0.4853 0.4955 1.3151 0.5638 0.5740
0.4 (2,1) 2.1091 0.18328 0.1855 0.1854 1.2298 0.23145 0.2444 0.2468
2.1281 0.1873 0.1911 1.1856 0.2330 0.2346
2.1136 0.1853 0.1860 1.2190 0.2397 0.2422
0.8 (2,1) 2.1374 0.36656 0.3834 0.3837 1.3332 0.46290 0.5935 0.6233
2.2187 0.4019 0.4388 1.1408 0.4720 0.4875
2.1526 0.3829 0.3876 1.2828 0.5455 0.5600
0.6 (2,2) 2.1044 0.093202 0.09439 0.09435 1.2299 0.24000 0.2522 0.2536
2.1152 0.09525 0.09702 1.1797 0.2375 0.2392
2.1064 0.09431 0.09451 1.2192 0.2471 0.2488
1.2 (2,2) 2.1188 0.18640 0.1974 0.1972 1.3357 0.47999 0.6176 0.6614
2.1646 0.2061 0.2239 1.1191 0.4624 0.4715
2.1248 0.1974 0.1977 1.2886 0.5576 0.5579
a
In each trio of results lines, the values come respectively from analysis with the TV, EV, and EV2 methods.
b
Data error structures of Table 4 (σxi, σyj), scaled by factor f.
c
RMS Vprior estimates of the parameter SEs.

observations for earlier models—expected agreement between pre- 4. Conclusion

dicted and sampling estimates for small data error, with progressively
increasing bias in both parameters and SEs with increasing f. The TV
and EV2 results were very close for all f, while the EV results displayed
noticeably greater parameter biases, especially for the most uncertain
quantity (k), where the bias for f = 1 increased the rms error by 9%. The
EV method also yielded divergent results 0.7% of the time. Details are
given in Supplement Table 2S.
This model was also tested by MC simulations previously by Alper
and Gelb [41]. In this case their parameter confidence limits agreed
with predictions, in contrast with their observations for the
dimerization model.
Monte Carlo simulations are used to investigate the statistical
properties of least-squares methods for analyzing data having uncer-
tainty in both x and y. Several test models are used to compare the
presumed best method, called “total variance” here, with two different
effective variance methods. All methods are equivalent in the limit of
small data error, where the a priori covariance matrix correctly predicts
the parameter standard errors, and the parameter estimates are
normally distributed. With increasing data error, all models and
methods exhibit bias in the estimates of the parameters and their
standard errors, and non-Gaussian parameter distributions. The three

Table 6
Analysis of Wentworth kinetics data.a

Μethod P0 σP0 k × 106 σk × 106 n σn χ2

TV 363.947575 0.995396 7.4480743 1.089324 1.97631876 0.02514050 2.4077377654

TVD (21) 363.947834 0.995402 7.4488576 1.089275 1.97630369 0.02513717 2.3989617615
EV (21) 364.102962 0.996318 7.1276277 1.066934 1.98376054 0.02568551 2.5038107768
EV2 (21) 363.947556 0.994817 7.444115 1.092771 1.9764005 0.0252592 2.4165350
TVD (22) 363.947561 0.995402 7.4494311 1.089514 1.97629066 0.02514029 2.3989589433
EV (22) 363.945434 0.991909 7.4540869 1.091629 1.97617586 0.02514347 2.4167624302
EV2 (22) 363.947556 0.994817 7.4441151 1.092771 1.97640051 0.02525917 2.416534945
TVD (23) 363.947846 0.995395 7.4478937 1.089298 1.97632214 0.02514053 2.4073417128
EV (23) 363.950348 0.997253 7.4488825 1.088788 1.97628530 0.02512757 2.4081542036
EV2 (23) 363.946769 0.994927 7.448623 1.089353 1.97630841 0.02514072 2.408132663
TVD (24) 363.947704 0.995397 7.4487103 1.089409 1.97630524 0.02514028 2.4030544401
EV (24) 363.953071 0.992379 7.5013227 1.100073 1.97497227 0.02522477 2.4161788287
EV2 (24) 363.947195 0.994864 7.446331 1.09105 1.97635517 0.02519985 2.4124186
a
Displayed values are thought to be computationally reliable to several units in the last digit. EV results are listed by employed fit expression in text. TVD designates TV method
with Deming's equations.
 Author's personal copy
168 J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
methods perform comparably in all cases, with differences that would
seldom be important in practical work. Only in one statistical property
does the TV method always excel: The STV values closely follow the χ2
distribution for all data error scaling, while the corresponding EV sums
do not. The Deming equations for the TV method can be considered a
near statistical equivalent of the Powell–Macdonald–Britt–Luecke–
Jefferys equations, though Deming's yield neither F = 0 on the
calculated curve nor invariance with respect to different ways of
expressing the fit relation.
While this study had to employ specific models for the MC
comparisons, the models are representative ones. Thus I anticipate
that the behavior observed for the three methods will be found to
hold approximately in other analysis problems. In particular, it is
likely that the close similarity in the TV and EV2 results (rigorously
identical for the straight line) is general. It also appears that the EV2
iterations always converge from above. Together these properties
mean that the EV2 computations can be carried out with some
standard nonlinear programs that utilize the Marquardt algorithm
(like KaleidaGraph [39]).
A requirement for successful use of any of these methods is good
information about the data error in both variables. In most applications,
this need will likely limit the reliability of the analysis more than the
choice of method. The usual approaches for getting this information
entail analysis of the statistics of replicates or residuals [43–45].
Reassuringly, the data variance function does not need to be precisely
known to yield near-optimal results in the LS fit of the data to the model
[46].
The need for information about the data error in multiple variables
also means that reported SEs should normally be based on Vprior
rather than on Vpost. The latter (in which the prefactor S/ν is included)
is the default choice in much physical science work. It is appropriate in
situations where a single uncertain variable has constant unknown
uncertainty or some simple form of heteroscedasticity (e.g., error
proportional to signal or to signal1/2) of unknown scale, and this
extends to cases where weight changes are needed when such data
are transformed. However, with multiple uncertain variables, all need
to be known within a single scale factor (C after Eq. (1)) to obtain
minimum-variance results. This requirement will normally require
absolute information about the data error in all variables, making
Vprior the appropriate choice. A benefit of this choice is the availability
of the χ2 test as a simple indicator of statistical inadequacy.
About terminology: Physical scientists have often referred to
treatments of problems with uncertainty in x and y as “generalized
least-squares.” That label unfortunately means something different
among statisticians, namely the estimation of both the variance
function and the response function for a data set, with the former
usually obtained by fitting squared residuals or their logarithms [43–
45]. Statisticians have used “errors in variables (EIV) models” and
“Model II regression,” but both of these also seem inadequately
descriptive. “Total least-squares” (TLS) is also used, but seems
presumptuous in that in many applications of TLS (as well as EIV)
there is no attention given to the relative magnitudes of the
adjustments of x and y needed to minimize STV [47]. Sometimes the
approach has been to minimize “perpendicular” residuals (also called
“orthogonal regression”), whereupon results can vary with arbitrary
scale changes in the variables. In the TLS community there also appears
to be little recognition of Deming's early contributions on this problem.
At the same time “Deming regression” has incorrectly come to refer to
just the straight-line model with two uncertain variables, when in fact
Deming's treatment was not so limited. Suitable terminology can
remain a subject for further discussion.
Acknowledgments
I thank Carl Bolster and David Parrish for rekindling my interest in
this topic, Joe Alper for helpful correspondence, and Ross Macdonald
for his conscientious, constructive criticism of an earlier version of this
work, which led to important realizations about the interrelations
among the various methods that have been used to treat uncertainty
in multiple variables.
Appendix A. Supplementary data
Supplementary data associated with this article can be found, in
the online version, at doi:10.1016/j.chemolab.2010.07.003.
References
[1] J.R. Macdonald, W.J. Thompson, Least-squares fitting when both variables contain
errors: pitfalls and possibilities, Am. J. Phys. 60 (1992) 66–73.
[2] D. York, Least-squares fitting of a straight line, Can. J. Phys. 44 (1966) 1079–1086.
[3] D. York, N.M. Evenson, M.L. Martinez, J. DeB Delgado, Unified equations for the
slope, intercept, and standard errors of the best straight line, Am. J. Phys. 72
(2004) 367–375.
[4] J. Riu, F.X. Rius, Univariate regression models with errors in both axes, J.
Chemometr. 9 (1995) 343–362.
[5] C.A. Cantrell, Technical note: review of methods for linear least-squares fitting of
data and application to atmospheric chemistry problems, Atmos. Chem. Phys.
8 (2008) 5477–5487.
[6] W. E. Deming, Statistical Adjustment of Data, (Wiley, New York, 1938, 1943;
Dover, New York, 1964).
[7] D.R. Powell, J.R. Macdonald, A rapidly convergent iterative method for the solution
of the generalized nonlinear least squares problem, Comput. J. 15 (1972)
148–155.
[8] H.I. Britt, R.H. Luecke, The estimation of parameters in nonlinear, implicit models,
Technometrics 15 (1973) 233–247.
[9] W.H. Southwell, Fitting data to nonlinear functions with uncertainties in all
measurement variables, Comput. J. 19 (1976) 69–73.
[10] D.R. Powell, J.R. Macdonald, Comments on ‘Fitting data to nonlinear functions
with uncertainties in all measurement variables’, Comput. J. 20 (1977)
285–286.
[11] W.H. Jefferys, On the method of least squares, Astron. J. 85 (1980) 177–181.
[12] M. Lybanon, A better least-squares method when both variables have uncertain-
ties, Am. J. Phys. 52 (1984) 22–26.
[13] M. Clutton-Brock, Likelihood distributions for estimating functions when both
variables are subject to error, Technometrics 9 (1967) 261–269.
[14] D.R. Barker, L.M. Diana, Simple method for fitting data when both variables have
uncertainties, Am. J. Phys. 42 (1974) 224–227.
[15] D.S. Riggs, J.A. Guarnieri, S. Addelman, Fitting straight lines when both variables
are subject to error, Life Sci. 22 (1978) 1305–1360.
[16] J. Orear, Least squares when both variables have uncertainties, Am. J. Phys. 50
(1982) 912–916 erratum 52 (1984) 278.
[17] J.P. Chandler, On an iterative procedure for estimating functions when both
variables are subject to error, Technometrics 14 (1972) 71–76.
[18] M. Lybanon, Comment on ‘Least-squares when both variables have uncertainties’,
Am. J. Phys. 52 (1984) 276–278.
[19] J. Tellinghuisen, C.H. Bolster, Weighting formulas for the least-squares analysis of
binding phenomena data, J. Phys. Chem. B 113 (2009) 6151–6157.
[20] J.H. Williamson, Least-squares fitting of a straight line, Can. J. Phys. 46 (1968)
1845–1847. This treatment is essentially Exercise 3 on page 145 in Deming's book.
[21] S.D. Christian, E.E. Tucker, LINGEN—a general linear least squares program, J.
Chem. Educ. 61 (1984) 788.
[22] M.L. Johnson, L.M. Faunt, Parameter estimation by least-squares methods, Meth.
Enzymol. 210 (1992) 1–37.
[23] E. Di Cera, Use of weighting functions in data fitting, Meth. Enzymol. 210 (1992)
68–87.
[24] J. Tellinghuisen, A Monte Carlo study of precision, bias, inconsistency, and non-
Gaussian distributions in nonlinear least squares, J. Phys. Chem. A 104 (2000)
2834–2844.
[25] A.H. Kalantar, R.I. Gelb, J.S. Alper, Biases in summary statistics of slopes and
intercepts in linear regression with errors in both variables, Talanta 42 (1995)
597–603.
[26] R.G. Krutchkoff, Classical and inverse regression methods of calibration, Techno-
metrics 9 (1967) 425–439.
[27] R.G. Krutchkoff, Classical and inverse regression methods of calibration in
extrapolation, Technometrics 11 (1969) 605–608.
[28] J. Berkson, Estimation of a linear function for a calibration line; consideration of a
recent proposal, Technometrics 11 (1969) 649–660.
[29] G.K. Shukla, On the problem of calibration, Technometrics 14 (1972) 547–553.
[30] J. Tellinghuisen, Inverse vs. classical calibration for small data sets, Fresenius J.
Anal. Chem. 368 (2000) 585–588.
[31] J. Tellinghuisen, Least squares in calibration: dealing with uncertainty in x, Analyst
135 (2010) 1961–1969.
[32] B.C. Reed, Least-squares fits with errors in both coordinates. II: comments on
parameter variances, Am. J. Phys. 60 (1992) 59–62.
[33] W.C. Hamilton, Statistics in Physical Science: Estimation, Hypothesis Testing, and
Least Squares, The Ronald Press Co., New York, 1964.
 Author's personal copy
J. Tellinghuisen / Chemometrics and Intelligent Laboratory Systems 103 (2010) 160–169
[34] P.R. Bevington, Data Reduction and Error Analysis for the Physical Sciences,
McGraw-Hill, New York, 1969.
[35] W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vetterling, Numerical Recipes,
Cambridge Univ. Press, Cambridge, U. K., 1986.
[36] J. Tellinghuisen, Weighted least-squares in calibration: what difference does it
make? Analyst 132 (2007) 536–543.
[37] J. Tellinghuisen, Least squares with non-normal data: estimating experimental
variance functions, Analyst 155 (2008) 161–166.
[38] D.W. Marquardt, An algorithm for least-squares estimation of nonlinear
parameters, J. Soc. Ind. Appl. Math. 11 (1963) 431–441.
[39] J. Tellinghuisen, Nonlinear least squares using microcomputer data analysis
programs: KaleidaGraph in the physical chemistry teaching laboratory, J. Chem.
Educ. 77 (2000) 1233–1239.
[40] K. Pearson, On lines and planes of closest fit to systems of points in space, Philos.
Mag. 2 (1901) 559–572.
[41] J.S. Alper, R.I. Gelb, Standard errors and confidence intervals in nonlinear
regression: comparison of Monte Carlo and parametric statistics, J. Phys. Chem.
94 (1990) 4747–4751.
View publication stats
169
[42] W.E. Wentworth, Rigorous least squares adjustment: application to some non-
linear equations, J. Chem. Educ. 42 (1965) 96–103.
[43] M. Davidian, R.J. Carroll, Variance function estimation, J. Am. Stat. Assoc. 82 (1987)
1079–1091.
[44] R.J. Carroll, D. Ruppert, Transformation and Weighting in Regression, Chapman
and Hall, New York, 1988.
[45] M. Davidian, P.D. Haaland, Regression and calibration with nonconstant error
variance, Chemom. Intell. Lab. Syst. 9 (1990) 231–248.
[46] J. Tellinghuisen, Variance function estimation by replicate analysis and general-
ized least squares: a Monte Carlo comparison, Chemom. Intell. Lab. Syst. 99 (2009)
138–149.
[47] I. Markovsky, S. Van Huffel, Overview of total least squares methods, Signal
Process. 87 (2007) 2283–2302.