You are on page 1of 18

Journal of the Franklin Institute 348 (2011) 15561573

www.elsevier.com/locate/jfranklin

Global derivative based sensitivity method


for parameter estimation
H. Sulieman, I. Kucuk
Department of Mathematics and Statistics, American University of Sharjah, P.O. Box 26666, Sharjah,
United Arab Emirates
Received 4 June 2009; received in revised form 14 May 2010; accepted 18 May 2010
Available online 2 June 2010

Abstract
In nonlinear parameter estimation local sensitivity assessment; conventionally measured by the
rst-order derivative of the predicted response with respect to a parameter of interest fails to provide
a representative picture of the prediction sensitivity in the presence of signicant parameter codependencies and/or nonlinearities. In this article we derive the prole-based sensitivity measure
developed by Sulieman et al. (2001, 2004) [1,2] in the context of model re-parameterization. In
particular, the so-called predicted response re-parameterization is shown to ultimately lead to the
prole-based sensitivity coefcient dened by the total derivative of the model predicted response
with respect to a parameter. Although inherently local, the prole-based measure is shown to handle
simultaneous perturbations in parameter values while accounting for their co-dependencies. Thus the
proposed measure possesses a central property of a global sensitivity measure and so it is considered
hybrid localglobal measure.
The global Fourier amplitude sensitivity test (FAST) is added to the analysis and compared with
both marginal and prole-based sensitivity methods. The Fourier sine amplitude is utilized here as a
rst-order sensitivity measure and shown to be directly linked to the local sensitivity coefcient
averaged over all ranges of parameter uncertainties and so it is also considered hybrid localglobal
measure. The comparisons are explained by three compelling model cases with different degrees of
parameter co-dependencies and nonlinearities.
& 2010 The Franklin Institute. Published by Elsevier Ltd. All rights reserved.

Corresponding author. Tel.: 971 65152928; fax: 971 65152950.

E-mail addresses: hsulieman@aus.edu (H. Sulieman), ikucuk@aus.edu (I. Kucuk).


0016-0032/$32.00 & 2010 The Franklin Institute. Published by Elsevier Ltd. All rights reserved.
doi:10.1016/j.jfranklin.2010.05.014

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1557

1. Introduction
Parametric sensitivity analysis in general describes the impact of perturbations in the
values of model input parameters on the model outputs. The objective of the sensitivity
assessment is to improve the quality of an existing model representing a physical system
perhaps by reducing complexity or by guiding further experiments to reduce uncertainty.
Through better understanding of the interplay between the model input parameters and
the relative importance they have on the model outputs, one can target specic parameters
for more detailed study to reduce model uncertainty.
This article investigates the sensitivity of the predicted responses from nonlinear
regression models to variations in parameter values. The numerical values of these
parameters are usually estimated using available experimental data. The resulting
uncertainties associated with these parameter estimates propagate into the model
predictions via sensitivities. Results of the sensitivity analysis applied to an existing model
of a physical system can be used to strengthen the knowledge base by guiding subsequent
research in order to increase the condence in the model and its predictions.
In the literature, there are two different schools of thought for parametric sensitivity
methods. Local methods are derivative-based methods and generate sensitivity information that is valid over small ranges of parameter uncertainties where linear approximation
to the model function is assumed adequate. The conventional measure of local parametric
sensitivity is dened by the rst-order partial derivatives of the model response function
with respect to the parameters.The resulting sensitivity coefcients measure the marginal
impact of the parameter of interest on model predictions since only the value of the
parameter of interest is perturbed while all other parameters are held xed at their nominal
values. Therefore, co-dependencies among parameter estimates are ignored by these
measures, Sulieman et al. [1] called them marginal sensitivity coefficients. Global sensitivity
methods assess the inuence of a parameter on the model response by exploring the entire
parameter space. Simultaneous changes in parameter values across their respective ranges
of uncertainty are incorporated in an overall measure of sensitivity. Majority of global
sensitivity techniques are variance-based measures that apportion the output uncertainty
to the uncertainty in the parameters. Some of these methods are model independent and
can be applied to nonlinear and non-monotonic models.
To surmount the drawbacks of the local sensitivity assessment, Sulieman et al. [1,2]
proposed an alternative assessment procedure in which simultaneous perturbations in the
values of all model parameters are achieved using the proling scheme introduced by Bates
and Watts [3] for nonlinearity assessment of regression models. The prole-based
sensitivity measure, dened by the total derivative of the model function with respect to
parameter of interest, was shown to account for both nonlinearity within the parameter
estimation problem and parameter estimate co-dependencies. Like any derivative measure,
prole-based sensitivity is inherently local, it provides, however, a more comprehensive
picture of the prediction sensitivity in the presence of parameter co-dependencies and
model nonlinearity. Detailed discussion of the prole-based sensitivity measures is
presented in Section 2.
Recently, Sulieman et al. [4] conducted comparative study of prole-based sensitivity
approach versus Fourier amplitude sensitivity test (FAST) for parameter estimation in
multi-response regression models. FAST was developed by Cukier et al. [57] and Schaibly
and Shuler [8] to assess sensitivity of model outputs to input parameters in complex model

1558

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

systems. It is one of the most widespread techniques in the sensitivity literature. FAST
main essence is the simultaneous perturbations of all parameter values using a single set of
computational runs. The convenience of FAST perturbation scheme manifests itself
because all the terms in the Fourier expansion of the model function are mutually
orthogonal. In contrast to prole-based approach, classical FAST is shown to ignore the
parameter estimates co-dependencies in the model while it accounts for the full model
nonlinearity. Description of FAST and its features are given in Section 3.
In this article, we present prole-based sensitivity procedure using the notion of model
re-parameterization in single-response nonlinear regression models. In particular, we
adopt predicted value re-parameterization in which the expression for the predicted
response at a selected design point is dened as one of the parameters in the new system.
Such a re-parameterization allows the proling-based sensitivity measure to be calculated
automatically with the calculations of the prole vector when tting the newly formulated
model, and the inferential results about the model prediction to be obtained in a more
straightforward and computationally efcient manner. We also explore comparisons
between the measures of the conventional local sensitivity, the prole-based sensitivity and
conventional FAST method. The comparisons are demonstrated by the implementation of
the three measures to three model cases with different model structures and different
degrees of parameter estimates nonlinear co-dependencies.
In Section 2 we describe the development of the prole-based sensitivity measure for the
nonlinear parameter estimation using the predicted response re-parameterization of the
model function. In Section 3 we review FAST, consider the most recent developments in
the application of this analysis technique and discuss its link to marginal and prole-based
sensitivity measures. Illustrative model cases are presented in Section 4, and conclusions
are summarized in Section 5.
2. Prole-based sensitivity analysis
Consider the general mathematical form of a single response nonlinear regression model
y fX; Y e

where y is an n-element vector of observed values of the response variable for particular
values of the regressor variables X={x1, x2,y,xn}, Y is a p-element vector of unknown
parameters, f is an n-element vector of predicted values of the response variable for given X
and Y, fX; Y ff x1 ; Y; f x2 ; Y; . . . f xn ; Yg, and e is an n-element vector of
independent random errors with a specied joint distribution. In most cases, including
the case here, e is assumed to have a spherical normal distribution, with Ee 0 and
0
vare Eee s2 I.
Sulieman et al. [1] developed a prole-based sensitivity measure to assess the sensitivity
of the predicted responses from model (1). The motivation for the prole-based approach
arises from the proling algorithm developed by Bates and Watts [3] for constructing
likelihood intervals for individual parameters in single response nonlinear regression
models. The proling algorithm was implemented to a reformulated model function using
special re-parameterization in which the predicted response is one of the parameters in the
new formulation. The slope of the prole trace of the predicted response parameter has led
to the denition of prole-based sensitivity coefcient. The resulted sensitivity coefcient
quanties the net change in the predicted response due to perturbations in a parameter of

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1559

interest, after the remaining parameters are updated to their estimates conditioned on the
perturbed value of the parameter.
The model in Eq. (1) can be re-expressed as
y gY e

2
0

where the elements of the n-dimensional vector gY Z1 Y; Z2 Y; . . . ; Zn Y are the


predicted responses dened in Eq. (1), i.e.,
Zj Y f xj ;Y;

j 1;2; . . . ;n

The notation Zj Y emphasizes the dependence of each predicted response on the


parameters Y. The predicted response parameterization is dened as follows:
f1 Z0 Y f Y;x x0
f2 y2
^
fp yp

where Z0 Y f Y; x x0 is the predicted response at a selected prediction point x0. It


should be mentioned that the above form (4) does not necessarily suggest that only f1
should be used to dene the predicted response at x0, or that only y1 should be used as its
inverse f1
1 . Any other two parameters from F and Y parameters can be chosen for these
purposes. For readers convenience, however, we will denote the predicted response
parameter by f1 throughout the paper, whereas the inverse transformation f1
1 will be
dened by one of the least nonlinear behaving parameters in gY.
The idea of using this particular class of transformations was rst suggested by Ross
[9,10]. Bates and Watts [11] showed that when the regression model has only one nonlinear
parameter, the transformation (4) will reduce parameter nonlinearity to zero for all Y.
Clarke [12] referred to this class of transformation as optimal parameter transformations since it produces zero or small parameter nonlinearities in the model and minimum
estimation bias. However, as shown by Clarke [12], the effectiveness of such
re-parameterizations depends substantially on the choice of the points x0 at which the
predicted responses are to be estimated.
In using this class of transformations, our purpose is not to reduce either the parameter
nonlinearities in the model but rather to characterize the parameter sensitivities of the
predicted response at a predetermined point x0. Therefore, the re-parameterization dened
in Eq. (4) is more functional here since it denes the identity transformation for (p1)
parameters, so that neither the nonlinearities of these parameters nor the biases in their
estimates are changed by the transformation. The only parameter for which the
nonlinearity or the bias may change is the one dened by the inverse transformation
f1
1 , say y1 , where y1 is chosen to be one of the most linearly behaving parameters in the
model.
Let gnew F be the new formulation of the model function in terms of F and suppose that
fi is the parameter of interest. In proling algorithm, the vector of parameters F is
partitioned as F fi ; Fi which is, in terms of Y, equivalent to Y yi ; Yi . fi yi is
then varied across its range of uncertainty and for each value of fi , let F~ i fi be the
conditional least squares estimates of Fi . The joint behavior of the predicted response
parameter and each of the remaining parameters in gnew can be understood through the
prole traces, and so sensitivity information can be extracted from the prole trace plots.

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1560

Using the studentized parameters, the prole trace of the predicted response parameter f1
~
~
versus fi , i=2,3,y,p, is the curve consisting of the points dfi ; df
1 , where df1 is the
studentized conditional estimate of f1 given a xed value of fi , i.e,
f~ 1 f^ 1
~
df
1
sef^ 1

and
dfi

fi f^ i
sef^

where the parameter estimates f^ 1 and f^ i and their associated standard errors sef^ 1 and
sef^ i are obtained from the unconditional least squares tting of gnew to the data.
~
The slope of df
1 with respect to dfi is equal to the prole-based sensitivity coefcient
[1]. The mathematical derivations of prole-based sensitivity coefcients are detailed in the
following subsection.
An argument against the use of the predicted value transformation is the difculty with
the inverse transformation from F to Y. In practice this need not be serious. Even when no
explicit inverse transformation exists, an iterative solution can be made almost trivial
because of the ready availability of good starting values.
Example 2.1. MichaelisMenten Model. MichaelisMenten model is commonly used in
enzymatic kinetics with well-known formulation:
f x;Y

y1 x
y2 x

where y is the measured initial velocity of an enzymatic reaction and x is the substrate
concentration. The unknown parameters y1 and y2 represent maximum conversion rate
and MichaelisMenten constant, respectively. Several types of linearization parameterization in which the model terms are re-arranged so that it becomes linear-in-parameters have
been proposed for the MichaelisMenten model [13]. The purpose of these linearization
parameterizations is to obtain global minimum when using least squares estimation
method. Alcazar and Ancheyta [14] argued that the nonlinear estimation techniques must
be used in order to avoid large estimation bias and signicant error distortion by the
linearization transformation. To overcome the drawbacks of the linearization process,
Doeswijk and Keesman [15] used a bias compensated total least squares method in which
bias compensation is incorporated into the calculation of the parameter estimate
covariance matrix in an ordinary least squares estimation strategy.
For the model formulation in Eq. (7), y1 appears linearly and is known to have
insignicant nonlinear behavior in the least squares tting of the model. Hence, y1 is used
for the inverse transformation f1
1 . The transformation (4) is applied to the two-parameter
MichaelisMenten model as follows:
f1

y1 x0
y2 x 0

and
f 2 y2

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1561

which leads to the following new formulation of the model:


gnew F

f1 xf2 x0
x0 f2 x

The data set used by Bates and Watts [3] for tting model (7) is used here and the
estimation results are reproduced in Table 1: The data set consists of six prediction (design)
points, each is replicated twice. Using the new formulation in Eq. (8), sensitivity of the
predicted response at each of the six prediction points is assesses with respect to f2 y2 .
The range of uncertainty for parameter y2 is considered to be the 95% condence interval,
i.e., within two standard errors of y^ 2 0:06411. Twenty values of y2 were generated in the
prescribed range and for each value, f1 is conditionally estimated using model (8). The
resulting six prole traces are depicted in Fig. 1.
In Fig. 1 the six prole traces intersect at the point (0,0) corresponding to
y^ 1 212:68; y^ 2 0:06411. Different sensitivity behaviors are clearly seen for y2 f2 at
the six prediction points. For the prediction at x0=0.02, 0.06, 0.11, y2 possesses negative
impact with sharpest slope at x0=0.02. The slope of the prole trace at x0=0.22 is
insignicant indicating small inuence of y2 at the predicted response at this design point.
At x0=0.56, 1.10 the prole traces have positive slopes indicating positive sensitivity
behavior of the corresponding predicted responses to y2 . Among the six prediction points,
the largest sensitivity is exhibited at x0=0.02 with the most nonlinear trend.
The sensitivity analysis procedure described in the above example along with the visual
summary in Fig. 1 has prompted the development of a prole-based sensitivity measure
dened by the slope of the prole trace of f1 versus a parameter of interest. The resulting
measure is named prole-based sensitivity coefcient (PSC).
2.1. Profile-based sensitivity coefficient
For given model formulation, parameterization and design space, the sensitivity of the
predicted response at x0 represented by f1 to a parameter of interest yi fi is measured by
the sensitivity coefcient:
PSC i x0

~
~
@df
sef^ i @f
1
1

@dfi sef^ 1 @fi

where df1 and dfi are the studentized parameters dened in Eqs. (5) and (6).
Expressing Eq. (9) in terms of Y parameters yields
PSC i x0

~ i
sey^ i @Z~ 0 Y sey^ i DZ0 yi ;Y

seZ^ 0 @yi
seZ^ 0
Dyi

10

Table 1
Summary of parameter estimates for the MichaelisMenten model.
Parameter

Estimate

St. error

y1
y2

212.68
0.06411

6.94
0.008

corry^ 1 ; y^ 2 0:77, s2=119.5 with 10 degrees of freedom

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1562

x0=0.02

(
1)

x0=0.06

x0=0.11
x0=0.22
x0=0.56
x0=1.10

8
2

1.5

0.5

0
(
2)

0.5

1.5

Fig. 1. Prole trace plots for the studentized predicted response parameter f1 at the six prediction points
x0=(0.02, 0.06, 0.11, 0.22, 0.56, 1.10) for the MichaelisMenten model.

~ i denotes the total derivative of Z0 yi ; Y


~ i with respect to yi
where the operator DZ0 yi ; Y
[1]. Expressing the total derivative in terms of partial derivatives gives the following result:

~ i
DZ0 yi ;Yi yi @Z0
@Z0  @Y

11

Dyi
@yi @Yi Y~ i @yi
~ i =@yi
Using least squares estimation criterion, Sulieman et al. [1] showed that the term @Y
is given by
(
)
1 2
~ i
@Y
@2 Syi ;Yi yi
@ Syi ; Yi yi 

12

0
~
@yi @Yi
@yi
@Yi @Yi
Y i

Pn

where Syi ; Yi yi i1 yi Zi yi ; Yi yi is the conditional least squares function.
Substituting Eqs. (11) and (12) into Eq. (10) yields
8
 9

1
sey^ i <@Z0
@Z0
@2 S
@2 S  =
13
PSC i x0


0
seZ^ 0 : @yi @Yi @Yi @Yi
@yi @Yi  ~ ;
Y i

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1563

Using rst and second order derivative information of the model function gY, Eq. (13)
can be shown to equal
PSC i x0

sey^ i
0
0
0
0
0
fv0 v V Vi e Vii 1 Vi vi D eg
seZ^ 0 i 0i i

14

where v0i is the ith component of the rst derivative vector v0 evaluated at x0; V i is an
n  p1 matrix consisting of rst derivative vectors of gY with respect to Yi ; v0i is a
(p1) dimensional vector consisting of the elements in the row of V i which corresponds
to x0; V i i is the n  (p1)  (p1) array of the second derivatives of gY with respect
to Yi ; D is the n  (p1) matrix of the second derivatives of gY with respect to Yi and
yi , and e is the n-element residuals vector. The quantities in Eq. (14) are evaluated at
~ i y^ i .
y^ i ; Y
The rst term in Eq. (14), sey^ i =seZ^ 0 v0i gives a scaled (studentized) measure of the
conventional marginal sensitivity coefcient. The second term in the equation,
0
0
0
0
0
sey^ i =seZ^ 0 fv0i Vi Vi e Vii 1 Vi vi D eg, represents an adjustment term
containing two components of information. The rst is the scaled marginal effects of Yi
on the predicted response at x0 through the derivative vector v0i . The second component of
information is about the co-dependency structure of the parameters Y measured by the
reaming part of the second term. It includes two sets of scaled co-dependencies: the pairwise
~ i via the term V 0 Vi e0 Vii 1 and correlations
correlations among the elements of Y
i
~ i via the term V 0 vi D0 e. Because second-order derivatives of gY are
between y^ i and Y
i

included in the co-dependency terms, model nonlinearity is accounted for by the prole-based
sensitivity coefcient, PSCi( x0), up to second-order derivatives.
The adjustment term in Eq. (14) incorporates the simultaneous changes in the parameter
values making PSCi(x0) global sensitivity measure while it is inherently local since it is
derivative-based. It is only when the parameter co-dependency structure and model
nonlinearity are insignicant that PSCi(x0) becomes equivalent measure to local sensitivity
coefcient. Under these facts, PSCi(x0) is called hybrid localglobal sensitivity measure.
Following the same philosophy, Sulieman et al. [2] extended the prole-based sensitivity
approach to the parameter estimation in multi-response regression models. A multiresponse regression model is expressed as
ykj fj xk ;Y zkj ;

k 1; . . . ;n j 1; . . . ;J

15

where now there are J response variables for the n experimental settings, zkj is the
disturbance term assumed to have a normal distribution with zero mean, for all k and j,
independent for different k but having a xed variancecovariance matrix R of dimension
J  J when k is common. Using the determinant criterion by Box and Draper [16],
Sulieman et al. [2] dened the following sensitivity measure:


1 2
0
0
@g0
@g0 @2 dyi ;Yi yi
@ dyi ;Yi yi 
PSCi x0

16

0
@yi @Yi  ~
@yi @Yi
@Yi @Yi
Y i

where dY jZ Zj is the determinant function, the vector @g0 =@yi has J elements of the
rst-order partial derivatives giving marginal sensitivity coefficients of the J model
0
functions gY with respect to yi evaluated at x0. @g0 =@Yi is an J  p1 matrix
consisting of the marginal sensitivity coefcients of the J model functions with respect to
0
Yi evaluated at x0. The matrix @2 d=@Yi @Yi is the (p1)  (p1) sub-matrix of the

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1564

Hessian matrix of dY, and @2 d=@yi @Yi is a ( p1)-element vector of the Hessian terms
corresponding to yi and Yi . These Hessian terms are computed using rst and second
order derivatives of model function with respect to parameters. The (r,s)th component is
given by
@2 d
0
jZ ZjtrUs trUr trUs Ur 
@ys @yr
0

trZ Z1 Zs Zr Zr Zs  trZ Z1 Z Zrs Zsr Z

17

where
0

Ur Z Z1 Z Zr Zr Z;
Zsr

@2 Z
@2 gY

;
@ys @yr
@ys @yr

Zr

@Z
@gY

@yr
@yr

r;s 1; . . . ;i1;i 1; . . . ;p

where the notation tr(A) represents the trace of a matrix A.


The (p1) elements of the vector @2 d=@yi @Yi in Eq. (16) are obtained by taking s= i
and r=1,y,i1, i1,y,p in Eq. (17). Because the Hessian matrix is symmetric, there are
only (p1)p/2 distinct Hessian terms in the (p1)  (p1) sub-matrix. This gives a total
number of Hessian terms to be evaluated as (p1)(p1)p/2=(p1)(p2)/2.
As in the single-response model case, Eq. (16) suggests that
PSCi x0 Marginal Sensitivity of yi adjustment term

18

The marginal sensitivity of yi , MSCi, in Eq. (18) is measured by the local sensitivity dened
by partial derivatives of model functions with respect to yi . The adjustment term,
measuring nonlinear co-dependencies among parameters up to second-order derivative
information, accounts for the overall impact of the simultaneous changes in the parameter
space. The derivative-based sensitivity measure, PSCi, stems its global property from the
adjustment term which in turns depend on the degree of parameter co-dependencies and/or
model nonlinearity.
3. Fourier amplitude sensitivity test (FAST)
FAST was rst formulated by Cukier et al. [5,6] and Schaibly and Shuler [8] to
investigate the sensitivity of the solutions of large scale chemical reaction systems to
uncertainties in rate coefcients. Cukier et al. [7] reformulated the procedure to facilitate its
applications to systems other than chemical reaction systems. The core feature of FAST
method is to reduce the dimension of the problem from p-dimensional parameter space to
one dimensional parameter space by suitably dening a search variable s using the
transformations:
yi gi ui ;

i 1; . . . ;p

19

where the variable ui serves to vary yi and is dened by


ui Gi sinwi s;

i 1; . . . ;p

20

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1565

P
wis are integer incommensurate frequencies, i.e., pi1 zi wi a0 for any integers zi . With
appropriate choice of G transformation, variation of s over the range prsrp generates a
curve which traverses the p-dimensional parameter space in a manner consistent with the
assumed probability distribution of Y. The set of frequencies, w=( w1, w2,y,wp) cannot be
truly incommensurate causing the Fourier coefcients associated with the ith parameter yi
to reect sensitivity of the model function not to yi only, but rather to possible
combination of yi and other parameters. To avoid or minimize this interference error, an
interference factor M is judiciously for a suitable nite set of integer frequencies. The
factor M postpones the interference to harmonics of order higher than M and therefore,
harmonics up to order M are included in the analysis. The order M is generally taken as
4 or higher in order to obtain sufcient accuracy of the calculations [17,18]. Cukier et al. [6]
tabulated frequency sets free of interferences to 4th order for up to 50 parameters.
The predicted response at x0, Z0 , becomes periodic in s and so it is expanded by a Fourier
series giving
Z0 s

1
X

Aq x0 sinqs Bq x0 cosqs

21

q1

where
Z

Arwi x0 1=2p

Z0 ssinrwi s ds

r 1;2; . . .

22

Z0 scosrwi s ds

r 0;1; . . .

23

p

Brwi x0 1=2p
p

Arwi x0 and Brwi x0 are the Fourier coefcients for the fundamental frequency wi and its
harmonics, r=1,2,yat x0.
Arwi x0 and Brwi x0 are computed numerically using nite summations. The number of
model evaluations Ns required for the analysis is at least 2 Mwmax1 [17].
The transformations ui in Eq. (20) are selected so that the two integrals (22) and (23) in
the s space are equal to the corresponding ones in Y space. Several transformations have
been proposed in the literature. The transformation proposed by Saltelli et al. [17] and used
by Haaker and Verheijen [24] for uniform distribution is applied here. It is given by
!
2 yui yli
ui Gi sinwi s
24
arcsinsinwi s
p yui yli
where yui and yli are the upper and lower bounds of the uniform parameter distribution.
The function gi in Eq. (19) is appropriately chosen to be
gi ui y^ i eui ;

1oui o1

25

y^ i is the unconditional estimate of the parameter yi using least squares criterion in singleresponse model or determinant criterion in multi-response model.
In the original formulation of FAST which is used here, uis are considered to be
independent random variables with respective probability distributions. Most recently Xu
and Gertner [19] proposed, using rank correlations, a procedure to extend the application
of FAST to models with correlated parameters.

1566

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

The authors of FAST centered their efforts on the rst-order Fourier sine coefcients,
Awi , corresponding to r=1 as sensitivity measures. They showed that
Awi x0

2
/@Z0 u=@ui S
bi

26

where bi is a distributional parameter for ui, and /@Z0 u=@ui S is the expected value of
@Z0 u=@ui over the u-space probability distribution. Cukier et al. [6] established a more
direct relation between the sensitivity measures Awi 0 , and the marginal sensitivity
coefcients @Z0 u=@ui . They showed, using moments representation of the probability
distribution and Taylor series expansion of model function, that
Awi x0

2
2
/@Z0 u=@ui S @Z0 u=@ui ju0 R
bi
bi

27

where @Z0 u=@ui ju0 is the value of the marginal sensitivity coefcient (MSC) when u=0
^ in Eq. (19)), and R is a remainder term containing the expected
(corresponding to Y Y
values of the higher-order terms in the Taylor series expansion of @Z0 u=@ui .
The remainder R is small only if the higher-order terms are essentially small, implying a
linear model in parameters. Thus Eq. (27) indicates that Awi are nonlinear sensitivity
measures that can be valid over broad ranges of parameter uncertainty.
Eq. (27) can be expressed as
Awi x0 Marginal Sensitivity of yi R

28

The parallel between Eqs. (28) and (18) is quite clear and indicates that the prole-based
sensitivity coefcient PSCi(x0) and Awi x0 are directly related to the marginal local
sensitivity coefcient while globally and simultaneously perturb parameter values and have
broader ranges of validity than the local analysis due to their nonlinear nature. Both are
hybrid localglobal measure that provide adjustment factors to the local sensitivity
assessment. The PSC measure adjusts the marginal sensitivity coefcient MSC for the
nonlinear co-dependencies among parameter estimates. Using second-order derivative
information of the model function, shown in Eqs. (14) and (17), PSC accounts for model
nonlinearity in two parts. The rst part lies in the second-order derivatives with respect to
the p1 parameters in the conditional estimation problem. The second part lies in the
second-order derivatives with respect to the parameter of interest and the remaining
parameters. The only second-order derivative information that is not incorporated in the
analysis is the one corresponding to the parameter of interest. This is because the PSC is a
rst-order sensitivity coefcient. On the other hand, the Fourier amplitude Awi adjusts the
marginal sensitivity coefcient for the full nonlinearity in the model function up to all
orders but not for the dependencies among parameters. It incorporates second-order
derivative information with respect to the parameter of interest which is not incorporated
by the PSC.
In examining the differences between the three sensitivity measures, the difference
between MSC and PSC values is directly attributed to the magnitude of the correction
term measuring the nonlinear dependencies among parameters. The difference between
MSC and Awi is attributed to the magnitude of R involving higher-order terms in
the Taylor series expansion without identication of particular terms that contribute the
most to the difference. On the other hand, the difference between PSC and Awi is not easily

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1567

attributed to particular terms. The difference can arise because of parameter


co-dependencies or because of model nonlinearity. PSC accounts for nonlinear
correlations among parameter estimates using second-order derivative information only.
It also does not account for second-order derivative information with respect to the
parameter of interest. Awi accounts for model nonlinearity using higher-order derivative
terms with respect to all parameters including interactions terms. It does not, however,
account for parameter estimate correlations. It should be pointed out that signicant
interaction terms in Awi are not necessarily associated with signicantly correlated
parameters. In addition, insignicant parameter correlations do not necessarily exhibit
insignicant interaction terms [20]. Separation of the effects of dependence and
nonlinearity can only be achieved if an assessment of the magnitudes of the secondorder terms in yi and higher orders in other parameters is carried out.
For sensible comparisons with MSC and PSC, Awi is scaled by the factor sey^ i =seZ^ 0 .
In the next section the three sensitivity measures, MSC, PSC and Aw are applied to
nonlinear parameter estimation of three model cases. Discussions and comparisons of the
sensitivity results obtained by the three measures are included.
4. Illustrative examples
Example 4.1. MichaelisMenten model. For each of the design points discussed in Example
2.1, the prole-based and marginal sensitivity coefcients for y1 and y2 in the Michaelis
Menten model in Eq. (7) calculated using Eq. (14). For the FAST numerical calculations, the
frequency set w=(11, 13) used by Cukier et al. [7] is utilized here. The order of interference for
w is M=6 giving a minimum number of model evaluation Ns to 157. We carried out 201
evaluations. The two Fourier sine coefcients, Aw1 and Aw2 , at each x0, are calculated using
201 model evaluations. The results are summarized in Fig. 2 where the dashed, solid and
dotted lines join, respectively, the values of Awi , MSCi and PSCi. For concise graphical
representation and convenience in comparing the three measure of sensitivities, the scaled
values of Awi are re-scaled by a factor of 101 for y1 and by 102 for y2 .
The values of MSCi (solid curves) and Awi (dashed curves) are very close to each other
and they are larger in magnitude than the PSCi values (dotted curves) of both parameters
for the majority of the data points. The nonlinear behaviors of y1 and y2 in this Michaelis
Menten model formulation is mild as demonstrated by Bates and Watts [3]. This mild
nonlinearity implies insignicant nonlinear effects in Awi causing FAST results to be very
close to the marginal analysis results. The parameter estimate correlations accounted for
by the PSCi exhibit substantial impact on the sensitivity behavior of the predicted
responses at low levels of concentration for y1 and at high levels of concentration for y2 .
According to the prole-based sensitivity curves, the largest value of PSC1 is observed at
the largest value of x (observation number 12), indicating that y1 is most inuential on the
predicted responses at large x values in the observed range. This result is coherent because
y1 is the horizontal asymptote for the MichaelisMenten equation (7), and therefore the
predicted responses at large values of the regressor variable depend largely on y1 . An
analogous result is obtained for the effect of y2 at small values of x, suggesting that y2 is
strongly inuential when predicting at small x values in the data range. y2 represents the
half-concentration, i.e., the value of x at which the velocity is one-half its ultimate value.
This usually occurs at low values of x.

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1568

1 sensitivity

1.5
1
0.5
0
0.5
1

6
7
Observation Number

10

11

12

6
7
Observation Number

10

11

12

2 sensitivity

0.5
0
0.5
1
1.5
2

Fig. 2. Parametric sensitivities for the MichaelisMenten model. The dashed, solid and dotted lines join the values
of Awi , MSCi and PSCi, respectively.
Table 2
Summary of parameter estimates for the double exponential model.
Parameter

Estimate

St. error

y1
y2
y3
y4
y5
s2 1:95  106 with 28 degrees of freedom

0.375
1.936
0.013
1.465
0.022

0.002
0.220
0.0004
0.221
0.0009

Obviously these relations between the parameters and the predicted responses are not
possible to uncover using either MSCi or Awi .
Example 4.2. Osborne [21] tted the model
Ey y1 y2 expy3 x y4 expy5 x

29

to data supplied by A.M. Sargeson of the Research School of Chemistry in the Australian
National University. The results of model tting using least squares estimation are given in
Table 2. A common feature of tting linear combination of exponentials is the pronounced
parameter estimate co-dependencies that are induced by model formulation. In this

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1569

example, parameter estimates correlations involving y2 , y3 , y4 and y5 approach 1. In


addition to severe parameter correlations, model (29) suffers from pronounced parameter
nonlinearities as shown by Sulieman et al. [1]. Using Eq. (14) prole-based and marginal
sensitivity coefcients for the ve parameters were calculated for each of the 33 cases
considered in the data.
The Fourier sine coefcients Awi x0 were calculated for each parameter at all design
points using the frequency set w=(11, 21, 27, 35, 39) reported in Schaibly and Shuler [8].
The order of interference is M=4 and the corresponding minimum number of model
evaluations is Ns=313. We used Ns=600. The resulting values of the three scaled measures
are displayed in Fig. 3.
While the MSC values suggest some strong sensitivity relations for all parameters at
most of x0, the PSC values (dotted curves) show no distinguishing strong effects of any of
the parameters on the predicted response at any x0 [23]. FAST results for the parameters
y1 , y3 and y5 shown in the plots of the rst column are substantially similar to the
corresponding PSC values. Aw1 , Aw3 and Aw5 are also nearly zero at all x0. This to say that
the nonlinearity effects of these three parameters on the prediction sensitivity are more
6
200

150

100
50

0
5

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

200

20

100
0

40
5

10

15

20

25

30

10

15

20

25

30

100

40
30
20
10
0

Fig. 3. Parameter sensitivities for the exponential model. The dashed, solid and dotted lines join the values of Awi ,
MSCi and PSCi, respectively. The plots in the rst column represent sensitivities with respect to y1 , y3 and y5 . The
second column represents sensitivities with respect to y2 and y4 .

1570

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

dominant than their co-dependencies accounted for by PSC analysis. On the other hand,
the values of Aw2 and Aw4 show some signicant sensitivities for small x0. According to Awi
and PSCi values, signicant relationships between the parameters and the predicted
responses are not easily determined due to the extreme levels of parameter nonlinearities
and/or co-dependencies. Signicant relationships, if any, can be made clearer when these
parameter nonlinearities and/or co-dependencies are reduced by using appropriate
parameter transformations.
Example 4.3. The model here is a multi-response regression model taken from Bates and
Watts [22]. The three expected responses are linear functions in two parameters,
f1 xk1 ;b b0 b1 xk1 ;

f2 xk2 ;b b0 b1 xk2 ; f3 xk3 ;b b0 b1 xk3

where xkj is the value of the jth regressor variable at the kth experimental settings,
k=1,2,y, n=8 and j=1,2,3. The interesting feature of this simple linear multi-response
model is that there is no nonlinearity associated with the parameters, however, the use of
the determinant criterion produces a nonlinear estimation problem. Using the data given
by Bates and Watts [11], the parameters were estimated by minimizing dY in Eq. (16).
The resulting values of the parameter estimates are b^ 0 ; b^ 1 0:408; 2:555 with an
optimum determinant value of 569 and corrb^ 0 ; b^ 1 0:968. The estimated variance
covariance matrix is
2
3
0
0:962 1:130 1:231
^ Z
^
Z
6
7
2:464 1:166 5
30
S^
4
8
2:660
The vector-valued prole-based and marginal sensitivity coefcients for the two
parameters were calculated for each of the eight cases using Eq. (17). The frequency set
used for the FAST analysis is the same as the one used in Example 4.1, namely, w=(11, 13)
with Ns=201. Aw2 was re-scaled by a factor of 102 for the graphical t. The results are
shown in Fig. 4. Because the model function is linear in b0 and b1 , the marginal sensitivity
calculations and FAST produce equal or nearly equal sensitivity value at all design points
for both parameters. The insignicant differences in the values of the two sensitivity
measure can be partly attributed to the error by the numerical integrations used in FAST
method. When the calculations are adjusted for the correlations between b^ 0 and b^ 1 , prolebased sensitivity analysis produces smaller values than corresponding Awi and MSCi at all
design points for both parameters. The PSC results also suggest different sensitivity
relations of the three predicted responses to the two parameters from those indicated by
FAST and marginal analysis. For example, according to the marginal sensitivity
coefcients and FAST, the predicted response Z^ 1 (top row in Fig. 4) exhibits high
sensitivity with respect to both parameters at large x0. For the same x0, the prole-based
sensitivity coefcients decline to almost zero indicating very minimal sensitivity effects.
Similar conclusions can be drawn for the sensitivities of Z^ 2 and Z^ 3 to both parameters.
5. Conclusions
The prole-based sensitivity coefcient introduced by Sulieman et al. [1] has been
derived in this paper using the concept of model re-parameterization. In particular, the
predicted response re-parameterization has been shown to directly lead to the formulation

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573


6

1 (x0)

2
0
2

0
0

2 (x0)

6
4

2
0
2

0
0

3 (x0)

1571

2
6
4

2
0
2

0
0

2
8
0
Observation Number

Fig. 4. Parametric sensitivities for the multi-response linear model. The dashed, solid and dotted lines join the
values of Awi , MSCi and PSCi, respectively. Plots in the rst column correspond to the sensitivities to b0 and in the
second column to the sensitivities to b1 .

of the prole-based sensitivity measure. The distinct feature of prole-based sensitivity


coefcient, dened by the total derivative of the predicted response with respect to a
parameter of interest, is the ability to handle simultaneous changes in parameter values
while accounting for parameter co-dependencies. The resulting derivative-based sensitivity
measure is considered hybrid localglobal measure because of the global perturbation
scheme used to vary the parameter values.
Comparisons between prole-based sensitivity, conventional marginal sensitivity and
classical Fourier amplitude sensitivity test (FAST) have also been conducted in this paper.
The measures provided by these three methods are rst-order sensitivity measures that
have different degrees of accuracy and validity. The marginal sensitivity coefcient is a
local measure that fails to account for model nonlinearity and parameter dependencies.
The classical FAST is a global method that accounts for model nonlinearity up to all
derivative orders but fails to account for parameter estimate dependencies. This article
focuses on the Fourier sine amplitude as a rst-order sensitivity measure. The results show
that the measure is directly linked to the local marginal sensitivity coefcient averaged over
all ranges of parameter uncertainties and so it can also be considered hybrid localglobal

1572

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

sensitivity measure. The prole-based sensitivity method has been shown to provide a
more reective picture of the sensitivity behavior because it accounts for both parameter
estimate correlations and model nonlinearity up to second-order derivative information.
Three illustrative model examples with different degrees of nonlinearity and parameter
co-dependencies have been used to demonstrate the comparisons.
The comparison exercise carried out in this work has shown that using an appropriate
analysis procedure, computing the sensitivity coefcients in the nonlinear parameter
estimation and plotting them against the design points provides a concise visual way to
investigate the sensitivity behavior of the parameter estimation problem.
Acknowledgment
The authors gratefully acknowledge the nancial support of the American University of
Sharjah, United Arab Emirates.
References
[1] H. Sulieman, P.J. McLellan, D.W. Bacon, A prole-based approach to parametric sensitivity analysis of
nonlinear regression models, Technometrics 43 (4) (2001) 425433.
[2] H. Sulieman, P.J. McLellan, D.W. Bacon, A prole-based approach to parametric sensitivity in
multiresponse regression models, Computational Statistics and Data Analysis 45 (2004) 721740.
[3] D.M. Bates, D.G. Watts, Nonlinear Regression Analysis and its Applications, Wiley, New York, 1988.
[4] H. Sulieman, I. Kucuk, J. McLellan, Parametric sensitivity: a case study comparison, Computational
Statistics and Data Analysis 53 (7) (2009) 26402652.
[5] R.I. Cukier, C.M. Fortuin, K.E. Shuler, A.G. Petschek, J.H. Schaibly, Study of the sensitivity of coupled
reaction system to uncertainties in rate coefcients. I, Theory, Journal of Chemical Physics 59 (8) (1973)
38733878.
[6] R.I. Cukier, J.H. Schaibly, K.E. Shuler, Analysis of the approximations, III, Journal of Chemical Physics 63
(3) (1975) 11401149.
[7] R.I. Cukier, H.B. Levine, K.E. Shuler, Nonlinear sensitivity analysis of multiparameter model systems:
review, Journal of Computational Physics 26 (1978) 142.
[8] J.H. Schaibly, K.E. Shuler, Study of the sensitivity of coupled reaction system to uncertainties in rate
coefcients. II. Applications, Journal of Chemical Physics 59 (8) (1973) 38793888.
[9] G.J.S. Ross, Exact and approximate condence regions for functions of parameters in nonlinear models, in:
COMPSTAT 1978, Physica-Verlag, Wien, 1978, pp. 110116.
[10] G.J.S. Ross, Uses of nonlinear transformations in nonlinear optimisation problems, in: COMPSTAT 1980,
Physica-Verlag, Wien, 1980, pp. 381388.
[11] D.M. Bates, D.G. Watts, Parameter transformation for improved approximate condence regions in
nonlinear least squares, Annals of Statistics 9 (1981) 11521167.
[12] G.P.Y. Clarke, Approximate condence limits for a parameter function in nonlinear regression, Journal of
the American Statistical Association 82 (397) (1987) 221230.
[13] P.M. Doran, Bioprocess Engineering Principles, Elsevier Academic Press, London, UK, 1995.
[14] L.A. Alcazar, J. Ancheyta, Sensitivity analysis based methodology to estimate the best set of parameters for
heterogeneous kinetic models, Chemical Engineering Journal 128 (2007) 8593.
[15] T.G. Doeswijk, K.J. Keesman, Linear parameter estimation of rational biokinetic functions, Water Research
43 (2009) 107116.
[16] G.E.P. Box, N. Draper, The Bayesian estimation of common parameters from several responses, Biometrika
52 (1965) 355365.
[17] A. Saltelli, S. Tarantola, K. Chan, A quantitative model-independent method for global sensitivity analysis
of model output, Technometrics 41 (1999) 3956.
[18] A. Saltelli, K. Chan, E.M. Scott, Sensitivity Analysis, Wiley, New York, USA, 2000.
[19] C. Xu, G. Gertner, Extending a global sensitivity analysis technique to models with correlated parameters,
Computational Statistics and Data Analysis 51 (12) (2007) 55795590.

H. Sulieman, I. Kucuk / Journal of the Franklin Institute 348 (2011) 15561573

1573

[20] A. Saltelli, M. Ratto, S. Tarantola, F. Campolongo, Sensitivity analysis practices: strategies for model-based
inference, Reliability Engineering and System Safety 91 (2006) 11091125.
[21] M.R. Osborne, Some Aspects of Non-linear Least Squares Calculations, Numerical Methods for Nonlinear
Optimization, Academic Press, London, 1972.
[22] D.M. Bates, D.G. Watts, A generalized GaussNewton procedure for multi-response parameter estimation,
SIAM Journal on Scientic and Statistical Computing 7 (1) (1987) 4955.
[23] H. Sulieman, Improved local sensitivity measures for regression models with correlated parameters, in:
Proceedings in Computational Statistics 18th Symposium, Porto, Portugal, 2008.
[24] M.P.R. Haaker, P.J. Verheijen, Local and global sensitivity analysis for a reactor design with parameter
uncertainty, Chemical Engineering Research and Design 82 (A5) (2004) 591598.