Chemometrics and Intelligent Laboratory Systems 64 (2002) 169 – 179

www.elsevier.com/locate/chemometrics

Uncertainty estimation for multivariate regression coefficients

Nicolaas (Klaas) M. Faber
Department of Production and Control Systems, ATO, PO Box 17, 6700 AA Wageningen, The Netherlands
Received 18 June 2002; received in revised form 22 August 2002; accepted 27 August 2002

Abstract

Five methods are compared for assessing the uncertainty in multivariate regression coefficients, namely, an approximate
variance expression and four resampling methods (jack-knife, bootstrapping objects, bootstrapping residuals, and noise
addition). The comparison is carried out for simulated as well as real near-infrared data. The calibration methods considered are
ordinary least squares (simulated data), partial least squares regression, and principal component regression (real data). The
results suggest that the approximate variance expression is a viable alternative to resampling.
D 2002 Elsevier Science B.V. All rights reserved.

Keywords: Multivariate calibration; Regression vector; Uncertainty estimation; Resampling; Jack-knife; Bootstrap; Monte Carlo simulation;
OLS; PLSR; PCR; NIR

1. Introduction component regression (PCR). By contrast, Faber

Typically, applications of multivariate models are
concerned with the prediction of a property of
interest. To achieve an acceptable predictive ability,
the uncertainty in the model parameters, i.e., the
regression coefficients, should not be too large. In
keeping with this principle, Centner et al. [1]
eliminated variables for which the regression coef-
ficients carry a relatively large uncertainty. They
used jack-knifing to estimate this uncertainty when
partial least squares regression (PLSR) is used for
calibration. In the chemometrics literature, two
alternatives to the jack-knife have been proposed
for assessing the uncertainty in multivariate regres-
sion coefficients. Wehrens and Van der Linden [2]
used the bootstrap in connection with principal
E-mail address: n.m.faber@ato.wag-ur.nl (N.M. Faber).
and Kowalski [3] derived approximate variance
expressions for PLSR and PCR (see their Sections
3.3.3, 3.3.4, and 3.3.5). These expressions account
for all sources of measurement error and accom-
modate for heteroskedastic as well as correlated
noise.
The jack-knife and bootstrap are resampling
methods [4]. Briefly, resampling amounts to gen-
erating new data sets from the available one by
introducing an artificial perturbation. The desired
uncertainty estimate follows from the spread in the
results obtained for the new data sets. This
approach essentially assumes that the artificial per-
turbation mimics the effect of the real perturbation
already present in the original data set. Another
resampling method, which has received less atten-
tion in the chemometrics literature, is the noise
addition method. Excellent discussions of this
method are available. Press et al. [5] treat con-

0169-7439/02/$ - see front matter D 2002 Elsevier Science B.V. All rights reserved.
PII: S 0 1 6 9 - 7 4 3 9 ( 0 2 ) 0 0 1 0 2 - 8
170 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
fidence limits on estimated model parameters,
whereas Carrol et al. [6] focus on bias estimation.
It was used by Derks et al. [7] to assess the
uncertainty in the output of artificial neural net-
works (ANNs), while Duewer et al. [8] and Dable
and Booksh [9] reported successful application to
pseudo-rank estimation. The versatility of noise
addition is further illustrated by the work of del
Rı́o et al. [10], who used the method to validate
expression-based prediction intervals in linear
regression with errors on both axes.
The purpose of this study is to investigate the
relative merits of various resampling methods and an
approximate variance expression. The resampling
methods under investigation are the jack-knife, boot-
strapping objects, bootstrapping residuals, and noise
addition. The different approaches are compared for
simulated as well as real near-infrared (NIR) data.
The simulated data are modeled using ordinary least
squares (OLS). This allows one to study the methods
under idealized circumstances. For example, the
approximate variance expression specializes to a
well-known exact one under these circumstances. It
is believed that these simulations yield insight that
can be used to better interpret the results obtained for
real NIR data modeled by PLSR or PCR.
2. Theory
2.1. Model assumptions
The multiple linear regression model is assumed,
i.e.,
y ¼ Xb þ e ð1Þ
where y (I  1) is the true predictand (property of
interest); X (I  J) is the true predictor matrix (e.g.,
spectra); b ( J  1) is the true regression vector; e
(I  1) is a vector of residuals; and I and J denote
the number of training samples and predictor var-
iables (e.g., wavelengths), respectively. The actual
modeling may be based on realizations of y and X
that are corrupted by non-negligible measurement
errors. However, the presence of measurement
errors is not indicated by additional notation to
simplify the presentation.
Underlying all resampling methods is the assump-
tion that the resampled entity is independently iden-
tically distributed (iid). The validity of this
assumption depends, among others, on the experi-
mental set-up. For a correct treatment of resampling
methods, it is, therefore, useful to distinguish between
two fundamentally different experimental set-ups,
namely random and controlled calibration [11]. In
the first case, the training set predictor variables (rows
of X) are randomly observed, whereas in the latter,
they are fixed by design. A design leads to exact
relationships in the training set data because measure-
ments are taken at special points. As a result, a certain
iid assumption will be violated. Clustering of the data,
which is often the case in quantitative structure
activity relationship (QSAR) work, may have similar
consequences. Score plots are convenient for visual-
izing relationships in the data.
2.2. Estimation of regression coefficients
The OLS estimate for b is:
bOLS ¼ ðXT XÞ1 XT y ð2Þ
where the superscripts ‘  1’ and ‘T’ denote matrix
inversion and transposition, respectively. For the OLS
solution to exist, X must be of full column rank.
The F-factor PLSR estimate for b can be
expressed under the SIMPLS formalism [12] as:
bPLSR ¼ ðRRT ÞXT y ð3Þ
where R ( J  F) is a matrix of weights; when
applied to the predictor variables, one obtains the
scores as T = XR. Eq. (3) includes Eq. (2) as a
special case because the full-factor PLSR model
reproduces the OLS solution. The PLSR estimate
also exists if X is rank-deficient (as long as the
number of factors does not exceed the rank of X)
or even if J >I, which is the ‘underdetermined’ case
often encountered in spectroscopy. Similar to Eq.
(3), the PCR estimate is given by:
bPCR ¼ ðVK1 VT ÞXT y ð4Þ
where V and K contain a subset of the eigenvectors
and eigenvalues of XTX, respectively. Eq. (4) can
be brought in the form of Eq. (3) by a simple
rescaling of the eigenvectors, i.e., R = VK 1/2.
 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
2.3. Uncertainty in regression coefficients
In the current study, the uncertainty in the coef-
ficient estimates is quantified by a standard error
(square root of a variance):
rðbj Þ ¼ VðbÞjj ;
1=2
j ¼ 1; . . . ; J
where V() symbolizes the covariance matrix of a
vector quantity and b stands for bOLS, bPLSR, or bPCR,
respectively. It is important to note that the standard
error fully accounts for the uncertainty only when
OLS is applied in connection with errorless predictors
(e.g., spectra). With measurement errors in X, the
OLS solution will be biased [13]. The relative impor-
tance of this bias depends on the size of the errors in
X. Often, the signal-to-noise ratio is rather high for
spectroscopic data (H10), so that we can safely
neglect this bias. The situation is further complicated
when using PLSR or PCR because these methods owe
much of their popularity to the bias –variance trade-
off. It is well known that the number of factors is
selected as a compromise between bias (too few
factors) and variance (too many factors). However, a
successful bias –variance trade-off implies the bias to
be relatively unimportant.
2.3.1. Approximate formula
An approximate covariance matrix of the OLS
regression coefficients is given by:
VðbOLS Þ ¼ MSEC  ðXT XÞ1
where MSEC denotes the mean squared error of
calibration estimated as:
X
I It is important to note that the validity of the
ðyi  yfit;i Þ2
i¼1
MSEC ¼
I  df
in which yi is the predictand for the ith training
sample; yfit,i is the corresponding fitted value; and df
denotes the degrees of freedom consumed by the
model parameters. For OLS, each parameter takes
away a degree of freedom from the data, likewise a
potential intercept. Eq. (6) is approximate [13], unless
X is without error.
171
De Jong [12] noticed that the analogy between
Eqs. (2) and (3) suggests that RRT is proportional
to an approximate covariance matrix for the PLSR
coefficients. Faber and Kowalski [3] further worked
out this observation for PLSR as well as PCR (see
their Sections 3.3.3, 3.3.4, and 3.3.5). Recalling that
ð5Þ the notation RRT also applies to OLS (full-factor
PLSR) and PCR (R = VK 1/2) yields:
VðbÞ ¼ MSEC  ðRRT Þ ð8Þ
where b stands for bOLS, bPLSR, or bPCR, and R is
estimated using the appropriate method.
Two comments seem to be in order. First, the
correct estimation of MSEC requires an adequate
number of degrees of freedom. As OLS, PCR con-
sumes a single degree of freedom for each factor
when the factors are chosen without reference to the
predictand vector (e.g., in the order of their corre-
sponding eigenvalues). By contrast, the appropriate
number of degrees of freedom for PLSR is not a
trivial matter, since the construction of the factors
includes the predictand vector. The rigorous study of
van der Voet [14] has clearly established that the
conventional number, i.e., a single degree of freedom
for each factor, is too small for the early factors and
too large for the latter ones. A sound alternative can
be calculated using the results of leave-one-out
cross-validation (see Eq. (26) in Ref. [14]). Second,
MSEC in Eq. (8) may contain a bias term (see
Denham [15] for more details). Thus, Eq. (8) yields
a mean squared error in the regression coefficients,
ð6Þ
rather than a variance. Because we have assumed
bias to be relatively small (successful bias – variance
trade-off), this distinction will not be made explicit
to simplify the presentation.
approximate variance formulas (6) and (8) does not
depend on the experimental set-up leading to the data
ð7Þ
(random or fixed calibration). Clearly, the variance in
the parameter estimates depends on the data only, not
on how they are obtained. The ‘design’ of the training
set is reflected in the matrix (XTX) 1 or RRT, which
determines the amount of error propagation. Obvi-
ously, error propagation is minimized by a proper
design, but in many applications, e.g., when dealing
with natural produce, it is impossible to construct a
design. The main assumption underlying Eqs. (6) and
172 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
(8) is that the noise in y and X is adequately accounted
for by MSEC.
2.3.2. Jack-knife
The jack-knife generates reduced data sets by
deleting objects, i.e., an element of y and the
corresponding row of X (cf. Fig. 1a and b for a
univariate model):
yi ¼ ðy1 ; . . . ; yi1 ; yiþ1 ; . . . ; yI ÞT ;
Xi ¼ ðxT1 ; . . . ; xTi1 ; xTiþ1 ; . . . ; xTI ÞT ;
For these reduced data sets, coefficient vectors
(bi) are estimated. Combining these estimates with
the estimate of the entire data set (b) yields so-
called pseudo-values [4]:
bipseudo ¼ Ib  ðI  1Þbi ; i ¼ 1; . . . ; I:
The desired covariance matrix follows from the
spread in the pseudo-values as:
1 XI
VðbÞ ¼ ðbi  bÞðbipseudo  bÞT
IðI  1Þ i¼1 pseudo
where b denotes the average of the pseudo-values.
When comparing Eq. (11) with the common expres-
sion for a covariance matrix (see Eq. (14) below), the
additional division by I is noteworthy. The reason for
this additional division is that the covariance matrix for
the mean of the pseudo-values estimates the desired
covariance matrix [4]. Martens and Martens [16]
introduced a modification of the jack-knife that yields
similar results.
It is seen that this procedure does not make any
assumption about the noise in y or X. The
resampled entities are ( yi,xi)-pairs, so they should
form a random sample from some multivariate
distribution. This implies that the data should not
be designed or— maybe even worse—clustered.
2.3.3. Bootstrapping objects
In the regression context, two modes of the
bootstrap exist, namely, bootstrapping objects (also
known as bootstrapping pairs or observations) and
bootstrapping residuals (see Chapter 9 in Ref. [4]).
The method working with residuals will be
explained in Section 2.3.4. Bootstrapping objects
proceeds as follows. New data sets are generated
by randomly drawing objects with replacement (cf.
Fig. 1a and c for a univariate model):
ðybi ; xbi Þ ¼ ðynbi ; xnbi Þ; i ¼ 1; . . . ; I;
i ¼ 1; . . . ; I
b ¼ 1; . . . ; B ð12Þ
i ¼ 1; . . . ; I:
ð9Þ where
nbi ¼ int½U ð0  1Þ  I þ 1; i ¼ 1; . . . ; I;
b ¼ 1; . . . ; B ð13Þ
in which int[] symbolizes the integer part of the
ð10Þ associated number and U(0– 1) is a random number
that is uniformly distributed between zero and unity.
The use of random numbers effectively makes boot-
strapping a Monte Carlo simulation technique. The
procedure is repeated B times, where B should be
selected large enough to yield precise estimates for
the desired standard error. Eq. (13) ensures that for
each draw, any of the ( yi,xi)-pairs is selected with
ð11Þ probability I -1. As a result, some of the objects will
be present more than once, whereas others are not
selected at all. The desired covariance matrix fol-
lows from the common formula of a covariance
matrix of independent vectors:
1 X B
VðbÞ ¼ ðbb  bÞðbb  bÞT ð14Þ
B  1 b¼1
where b denotes the average of the bootstrapped
values of b. Bootstrapping objects is similar to the
jack-knife in the sense that it does not make any
assumption about the noise in y or X, but the
( yi,xi)-pairs should form a random sample from
some multivariate distribution (random calibration).
2.3.4. Bootstrapping residuals
Bootstrapping residuals starts off from the
regression model for which one attempts to esti-
 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179 173

Fig. 1. Illustration of resampling methods for univariate straight-line fit of single x versus y. (a) Original data points (o), model (—), fitted points
.
( ), and residuals (: : : ). (b) Original data points (o) and model (—) for objects {1,2,4,5} selected by jack-knife (  ). (c) Original data points
.
(o) and model (—) for objects {4,4,4,5,2} selected by bootstrap (  ). (d) Fitted points for original data ( ) and model (—) for objects obtained
by adding bootstrap-selected residuals {3,2,3,1,4} to the fitted points (  ). Resampling objects works directly with the original data points (o),
.
whereas resampling residuals works with the fitted points ( ) and estimated residuals ( – – – ).

mate the uncertainty. First, residuals are calculated new residual vectors are generated by randomly
as: drawing residuals with replacement. Finally, new
data sets are constructed by adding these new
yi  yfit;i residual vectors to the fitted predictand vector (cf.
ei ¼ ; i ¼ 1; . . . ; I: ð15Þ
ð1  df =IÞ1=2 Fig. 1a and d for a univariate model):

The ‘raw’ residual in the numerator is corrected for ybi ¼ yfit;i þ enbi ; i ¼ 1; . . . ; I; b ¼ 1; . . . ; B ð16Þ
degrees of freedom because the difference between
observed and fitted data is consistently smaller than
the deviation from the expected values (E[ yi ]; where nib is as defined in Eq. (13). The procedure is
i = 1,. . .,I). Alternatively, one could adjust the ‘raw’ repeated B times and the desired covariance matrix
residuals by means of the associated leverage. Next, follows from Eq. (14). Unlike jack-knifing and boot-
174 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
strapping objects, resampling residuals alters the data
similarly as the original perturbation would (e.g.,
measurement noise). The procedure assumes that
the order in which the residuals are drawn is imma-
terial (exchangeability), which is the case when the
noise is iid. As discussed by Efron and Tibshirani
[4], bootstrapping residuals yields better results for
the classic linear regression model (full column rank
X) than bootstrapping objects when this condition is
met. However, since the results depend critically on
this iid condition [4], bootstrapping objects is the
preferred mode [17].
Finally, it is noted that the ( yi,xi)-pairs need not
form a random sample from some multivariate
distribution. The reason for this is that the simu-
lations are performed conditional on the model.
This conditioning effectively fixes the xi, so that
it is immaterial whether the data are designed (fixed
calibration) or not (random calibration).
2.3.5. Add noise to original data
Similar to Eq. (16), N new data sets are gen-
erated according to:
yni ¼ yi þ MSEC1=2  Fð0; 1Þ; i ¼ 1; . . . ; I;
n ¼ 1; . . . ; N ð17Þ
where F(0,1) symbolizes a random number gener-
ated from a distribution with mean zero and stand-
ard deviation unity. The covariance matrix follows
from the equivalent of Eq. (14).
Noise addition and bootstrapping residuals have
in common that the ( yi,xi)-pairs need not form a
random sample. However, the noise addition
method is more versatile because it can deal with
heteroskedastic and correlated noise.
3. Experimental
3.1. Simulated NIR data
Fearn [18] published a NIR data set that was
collected for the prediction of protein content in
ground wheat samples. Because wheat samples
cannot be designed, the experimental set-up con-
forms to random calibration. Hence, no resampling
methods can be excluded from the start. The
reference values were obtained using the Kjeldahl
method. The training set consists of 24 objects
(I = 24). The NIR spectra are digitized at six wave-
lengths in the range 1680– 2310 nm ( J = 6). This
data set has been used extensively in the chemo-
metrics literature for method testing (see Refs.
[3,15] and references therein). The simulated data
are generated by the following two steps:
1. The ‘true’ y, X, and b in Eq. (1) are the OLS fit of
the experimental y, the experimental X, and bOLS,
respectively.
2. ‘Experimental’ realizations of y and X are
constructed by artificially adding noise to y and
X. The noise added to y is iid with standard
deviation 0.2% (m/m), which is the estimated
uncertainty of the Kjeldahl method [19]. The noise
added to X is either iid or proportional. The
standard deviation of the iid noise takes the values
0%, 0.25%, 0.5%, 0.75%, and 1% of the maximum
value of X. Similarly, the proportional noise has
standard deviation 0%, 0.25%, 0.5%, 0.75%, and
1% of the associated value of X. It is believed that
the level of the noise in X is unrealistically high for
certain spectroscopies (e.g., NIR), but it may be
adequate for testing the validity of uncertainty
estimates.
A single ‘experimental’ realization suffices to
calculate estimates of standard error in the regres-
sion coefficients. However, these estimates contain
an uncertainty themselves. Clearly, an unavoidable
source of the uncertainty in error estimates is that
some ‘experimental’ realizations are noisier than
others are by chance alone. To quantify the total
uncertainty, the error estimates are calculated for
100 independent ‘experimental’ realizations. Boot-
strapping and noise addition are based on 1000
replicates constructed for a single ‘experimental’
realization (B = N = 1000). This large number is
chosen to ensure that the uncertainty of the error
estimates is mainly determined by the variability
among the ‘experimental’ realizations [20].
The ‘ideal’ estimate of standard error is obtained
from the spread in the regression vectors obtained for
1000 ‘experimental’ realizations. Obviously, one can-
not calculate this ‘ideal’ estimate in practice because
only few realizations are available—most often just a
 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179 175

single one. However, there is a way to approach the

‘truth’ in practice if the spectra of all constituents are
known. In that case, Monte Carlo simulations can be
performed to validate the various approaches in a
similar fashion. Keller et al. [21] have detailed how
to conduct these simulations for two industrial appli-
cations (NIR and visible spectra).

3.2. Real NIR data

This NIR data set was measured to predict the mass

fraction percent O delivered by oxygenates in gaso-
line. The prediction was carried out for the oxygenates
methyl tert-butyl ether (MTBE) and ethanol (EtOH),
as well as water (H2O), which is an unavoidable major
contaminant of EtOH. Full details concerning the
spectral data acquisition, peak identification, reference
method (gravimetry), and data pretreatment (multi-
plicative signal correction) are presented elsewhere
[22]. Briefly, training and test sets consist of 40
samples each (I = 40). NIR absorbances are taken at
391 wavenumbers evenly spaced in the region 6000–
9000 cm 1 ( J = 391). Cross-validation and external
prediction testing using the test set lead to the follow-
ing optimum dimensionalities for MTBE, EtOH, and
H2O, respectively: 5, 8, and 7 (calibration method is
PLSR) and 6, 12, and 10 (calibration method is PCR).
To support the validity of certain resampling meth-
ods, it is necessary to investigate whether relation-
ships exist among the training set data. It is noted that
the data set was designed with a number of practical
considerations in mind, in particular:
1. The target values of 2.0% O and 2.7% O (total
amount) should be well covered.
2. EtOH, which is typically added to a gasoline at the
point of delivery, is likely to be the minor
component of a mixed oxygenate gasoline.
3. The H2O concentration should not be unrealisti-
cally high.
The analyte concentrations initially proposed by
the design were adjusted to facilitate the preparation
of the samples. As a result, no clear relationships
exist among the training set data (Fig. 2). Thus, it
is safe to assume that the random calibration
assumption is not severely violated, although, in a
strict sense, the training set is designed.
Fig. 2. Distribution of samples in plane spanned by PC scores 1 and
2: training set (  ) and test set (o).
3.3. Calculations
All calculations are performed in MATLAB (The
Mathworks, Natick, MA).
4. Results and discussion
4.1. Simulated NIR data
Only results obtained when adding proportional
noise to X are presented because the iid results are
very similar. All results are divided by the ‘ideal’
results, which are obtained by 1000 realizations.
This normalization facilitates the interpretation
because the target value after normalization is unity.
The order of the calculation is as follows. First, the
mean variances are calculated. Then, these means
are divided by the optimum value. Finally, the
square root is taken. Recall that Eq. (6) is exact
with error-free X, not approximate. The means of
100 normalized uncertainty estimates obtained using
the exact formula, bootstrapping residuals, and add-
ing noise are close to the target value (Table 1). By
contrast, the jack-knife and bootstrapping objects
overestimate the true standard error by approxi-
mately 20%. Apart from overestimating the true
176 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179

standard error, the latter estimates are more variable Table 2

(Table 2). This result is consistent with Denham’s Standard deviation in normalized estimates of standard error in OLS
coefficients for simulated data set with proportional noise in X (100
observation that formula-based estimates of root runs). All standard error estimates are normalized with respect to the
mean squared error of prediction (RMSEP) are ideal results
more stable than the results obtained for certain Noise in Method Variable
resampling methods [15]. X (%)
1 2 3 4 5 6
The standard deviation in the formula-based
0 Exact formula 0.18 0.18 0.18 0.18 0.18 0.18
result is close to what is expected. This can be
Jack-knife 0.35 0.36 0.34 0.33 0.33 0.36
understood as follows. Without noise in X, the Bootstrap objects 0.26 0.24 0.24 0.25 0.26 0.25
square of the standard error estimate is distributed Bootstrap residuals 0.18 0.18 0.18 0.18 0.18 0.18
proportional to a v2 variable. For a v2 variable Noise addition 0.18 0.19 0.19 0.18 0.18 0.18
with m degrees of freedom, the mean is m while the 1 Approximate 0.25 0.24 0.25 0.21 0.23 0.26
formula
variance is 2m. The relative standard deviation pffiffiffiffiffiin Jack-knife 0.40 0.41 0.45 0.38 0.44 0.42
the estimate of standard error follows as 1= 2m . Bootstrap objects 0.32 0.34 0.34 0.30 0.36 0.35
This is an approximation (because the square root Bootstrap residuals 0.25 0.24 0.25 0.21 0.23 0.26
is a nonlinear function) that works quite well for Noise addition 0.25 0.24 0.25 0.21 0.23 0.26
large degrees of freedom. Inserting the degrees of
freedom 24  6 = 18 yields 0.17, which is close to
the corresponding numbers in Table 2. The stand-
ard deviations in the mean values presented in high by approximately 20%. It is remarkable that
Table 1 follow by multiplying
pffiffiffiffiffiffiffiffi the numbers in for all methods, the standard error estimate has
Table 2 by the factor 1= 100 ¼ 0:1. In this way, degraded for variable 5. Currently, there is no
it can be inferred that the results obtained by the explanation for this phenomenon. One of the
exact formula do not deviate significantly from reviewers noted that the degradation of variable 5
unity. is presumably connected with the correlation struc-
When adding 1% proportional noise, the approx- ture between the predictors. This correlation struc-
imate formula slightly overestimates the true stand- ture can be investigated using diagnostics such as
ard error, but remains consistent with bootstrapping variance inflation factors. However, these diagnos-
residuals and adding noise (Table 1). The alterna- tics did not reveal anything particular for variable
tive resampling methods yield results that are too 5. There is an overall increase in variability in the
estimates for standard error (Table 2), which is a
logical consequence of the increased variability in
the data. The results obtained for the intermediate
Table 1
Mean estimates of standard error in OLS coefficients for simulated noise levels in X (0.25%, 0.5%, and 0.75%) are
data set with proportional noise in X (100 runs). All results are similar to the ones presented for 1% (not shown).
normalized with respect to the ideal results The main assumption underlying Eq. (6),
Noise in Method Variable namely, that MSEC accounts for all noise in y
X (%) and X, is known to be valid for iid noise in X in
1 2 3 4 5 6
0 Exact formula 0.98 0.98 0.98 0.98 0.98 0.98
connection with OLS [13]. The generalization to
Jack-knife 1.24 1.21 1.21 1.25 1.23 1.26 proportional noise seems to work well.
Bootstrap objects 1.22 1.19 1.18 1.24 1.19 1.25
Bootstrap residuals 0.98 0.98 0.98 0.98 0.98 0.98 4.2. Real NIR data
Noise addition 0.98 0.98 0.98 0.98 0.98 0.98
1 Approximate 1.02 1.04 1.01 1.05 1.27 1.13
formula
Only PLSR results are shown because the PCR
Jack-knife 1.19 1.21 1.27 1.32 1.64 1.22 results are very similar. As expected from the
Bootstrap objects 1.15 1.17 1.19 1.26 1.54 1.22 simulations, the resampling methods fall apart in
Bootstrap residuals 1.02 1.04 1.01 1.04 1.26 1.13 two categories that yield comparable results: jack-
Noise addition 1.03 1.04 1.01 1.05 1.26 1.14 knife and bootstrapping objects versus bootstrap-
 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179 177

Fig. 3. Standard errors in PLSR coefficients for oxygenate data (  10 2): formula-based versus the ones obtained by bootstrapping objects
(top) and bootstrapping residuals (bottom).

Fig. 4. Reliabilities of PLSR coefficients for oxygenate data: bootstrapping objects (top) and bootstrapping residuals (bottom). The dotted line
(: : :) indicates the value corresponding to a 90% confidence interval including zero.
178 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
ping residuals and noise addition. Consequently, we
will restrict ourselves to the discussion of the two
modes of bootstrapping. Unlike bootstrapping
objects, bootstrapping residuals is consistent with
Eq. (8) (see Fig. 3). As noted by one of the
reviewers, this just indicates consistency between
the methods and not whether the approaches pro-
duce the correct estimates of standard error, which
is the important question. However, Eq. (8) is
consistent with a formula for sample-specific stand-
ard error of prediction that gave promising results
for NIR [22 – 24] as well as fluorescence data [25].
These all lead to a convenient error analysis for
multivariate calibration models. The standard errors
obtained when bootstrapping objects are believed to
be overly pessimistic. This conjecture is supported
by calculating the reliabilities of the regression
coefficients as [1]:
bj
cj ¼ ; j ¼ 1; . . . ; J :
rðbj Þ
Wehrens and Van der Linden [2] removed wave-
lengths for which the 90% confidence interval
includes zero. Assuming t-statistics, this condition
translates to demanding the reliability to exceed
1.69 for all three analytes. The largest difference
between the two bootstrap methods is observed for
the H2O calibration (Fig. 4). Bootstrapping objects
suggests 142 coefficients to be insignificant versus
only 28 for the residual-based method. However,
this large number of insignificant coefficients (142
Fig. 5. PLSR coefficients for H2O.
Fig. 6. PLSR coefficients for H2O between 7600 and 8200 cm 1 for
which the 90% confidence interval includes zero: bootstrapping
objects () and bootstrapping residuals (*).
of 391) contradicts the smoothness of the regres-
sion vector (Fig. 5). Moreover, on the basis of 142
insignificant coefficients, one would expect a much
ð18Þ
larger number of actual zero crossings. This is
further illustrated by plotting only the insignificant
coefficients for a limited region (Fig. 6). Similar
observations can be made for Figs. 5– 8 in Ref.
[2].
5. Conclusions
The results presented in this work suggest that the
best resampling methods for uncertainty estimation
explicitly work with noise, rather than objects.
Besides more closely resembling the probability
mechanism underlying the observed data [4], this
also leads to more stable uncertainty estimates. The
latter is an added bonus that concurs with results
presented by Denham [15]. Because the approximate
formula, i.e., Eq. (8), performed equally well to
bootstrapping residuals and noise addition, it seems
to offer a viable alternative to resampling.
It is important to note that this finding strongly
disagrees with the main conclusion of a study where
the uncertainty in second-order bilinear predictions
using the generalized rank annihilation method
(GRAM) was assessed [26]. In that specific context,
noise addition seems to be the most versatile method.
The reason for this discrepancy is that the approx-
imate variance expression for GRAM requires
 N.M. Faber / Chemometrics and Intelligent Laboratory Systems 64 (2002) 169–179
detailed knowledge about the measurement noise,
which is not always available. By contrast, Eq. (8)
requires only the MSEC as input.
Acknowledgements
The National Institute of Standards and Technol-
ogy (NIST) is thanked for making the oxygenate data
available for this study. The critical remarks by Frank
Schreutelkamp, Age Smilde, and two reviewers are
appreciated by the author.
References
[1] V. Centner, D.L. Massart, O.E. de Noord, S. de Jong, B.G.M.
Vandeginste, C. Sterna, Anal. Chem. 68 (1996) 3851.
[2] R. Wehrens, W.E. Van der Linden, J. Chemom. 11 (1997) 157.
[3] K. Faber, B.R. Kowalski, J. Chemom. 11 (1997) 181.
[4] B. Efron, R.J. Tibshirani, An Introduction to the Bootstrap,
Chapman and Hall, London, 1993.
[5] W.H. Press, B.P. Flannery, S.A. Teukolski, W.T. Vetterling,
Numerical Recipes. The Art of Scientific Computing, Cam-
bridge Univ. Press, Cambridge, 1988(Section 14.5).
[6] R.J. Carrol, D. Ruppert, L.A. Stefanski, Measurement Error in
Nonlinear Models, Chapman and Hall, London, 1995(Chap. 4).
179
[7] E.P.P.A. Derks, M.S. Sánchez Pastor, L.M.C. Buydens, Che-
mometr. Intell. Lab. Syst. 28 (1995) 49.
[8] D.L. Duewer, B.R. Kowalski, J.L. Fasching, Anal. Chem. 48
(1976) 2002.
[9] B.K. Dable, K.S. Booksh, J. Chemom. 15 (2001) 591.
[10] F.J. del Rı́o, J. Riu, F.X. Rius, J. Chemom. 15 (2001) 773.
[11] P.J. Brown, Measurement, Regression, and Calibration, Clar-
endon Press, Oxford, 1993(Chap. 5).
[12] S. de Jong, Chemometr. Intell. Lab. Syst. 18 (1993) 251.
[13] S.D. Hodges, P.G. Moore, Appl. Stat. 21 (1972) 185.
[14] H. van der Voet, J. Chemom. 13 (1999) 195.
[15] M.C. Denham, J. Chemom. 14 (2000) 351.
[16] H. Martens, M. Martens, Food Qual. Prefer. 11 (2000) 5.
[17] R. Wehrens, H. Putter, L.M.C. Buydens, Chemometr. Intell.
Lab. Syst. 54 (2000) 35.
[18] T. Fearn, Appl. Stat. 32 (1983) 73.
[19] H. Martens, T. Naes, Multivariate Calibration, Wiley, Chiches-
ter, 1989.
[20] J.S. Alper, R.I. Gelb, Talanta 40 (1993) 355.
[21] H.R. Keller, J. Röttele, H. Bartels, Anal. Chem. 66 (1994)
937.
[22] N.M. Faber, D.L. Duewer, S.J. Choquette, T.L. Green, S.N.
Chesler, Anal. Chem. 70 (1998) 2972.
[23] R. Boqué, M.S. Larrechi, F.X. Rius, Chemometr. Intell. Lab.
Syst. 45 (1999) 397.
[24] J.A. Fernández Pierna, L. Jin, F. Wahl, N.M. Faber, D.L.
Massart, Chemometr. Intell. Lab. Syst., accepted for pub-
lication.
[25] A.C. Olivieri, J. Chemom. 16 (2002) 207.
[26] N.M. Faber, Anal. Chim. Acta 439 (2001) 193.