JOURNAL OF MULTI-CRITERIA DECISION ANALYSIS

J. Multi-Crit. Decis. Anal. 8: 162–180 (1999)

Response Surface Methodology as a Sensitivity Analysis Tool in

Decision Analysis
KENNETH W. BAUER Jr.a, GREGORY S. PARNELLb,c,* and DAVID A. MEYERSa
a
Department of Operational Sciences, Graduate School of Engineering, Air Force Institute of Technology,
2950 P Street, Wright – Patterson AFB, OH 45433 -7765, USA
b
Department of Systems Engineering, United States Military Academy, West Point, NY 10996 -1779, USA
c
Toffler Associates, 302 Harbor’s Point, 40 Beach Street, Manchester, MA 01944, USA

ABSTRACT
This paper proposes the use of Response Surface Methodology (RSM) as a sensitivity analysis tool for single
attribute and multi-attribute decision analysis (DA). It is shown that any single or multi-attribute decision analysis
value (or utility) function can be transformed to a response function of the uncertain variables. A sensitivity
analysis framework designed to facilitate simultaneous perturbation of a number of uncertain variables is
proposed. Specifically, RSM is used with influence diagrams, but the methodology can also be used with decision
trees. This approach is illustrated with the well-known Oil Wildcatter Problem. This new framework exceeds the
current sensitivity analysis capability of decision analysis software tools. The approach is shown to be more
efficient than current DA sensitivity analysis techniques and can provide improved insights for decision makers.
Copyright © 1999 John Wiley & Sons, Ltd.

KEY WORDS: decision analysis; Multi-attribute Utility Theory; Multi-attribute Preference Theory; influence
diagrams; multi-criteria decision analysis; response surface methodology; experimental design;
regression; statistics

1. INTRODUCTION gram solution (decision tree) cycles required for

The objective of this article is to show that the use
of Response Surface Methodology (RSM) can
facilitate effective and efficient sensitivity analysis
for decision analysis (DA) models. Effectiveness
(to within an acceptable tolerance of error) is
measured against current methods. Efficiency is
evaluated through resource utilization (analyst
time) as compared with current practice. Specifi-
cally, RSM is employed in the use of influence
diagram models. However, the approach is the
same for decision tree models.
The purpose of using RSM as a sensitivity
analysis tool is to improve the analyst’s under-
standing of the sensitivity between the output of
the model (response) and its associated input
parameters, and then use this information to im-
prove the decision model. There are two major
benefits of using RSM for sensitivity analysis on a
decision analysis problem. The first is a substan-
tial savings in the total number of influence dia-
* Correspondence to: Department of Systems Engineer-
ing, United States Military Academy, West Point, NY
10996-1779, USA. e-mail: FG7526@exmail.usma.edu
thorough sensitivity analysis. As the model in-
creases in complexity, the savings can be signifi-
cant. The second major benefit is the added
insights available to decision makers from the use
of RSM. These insights are gained through the
estimation of specific coefficients (effects) of sig-
nificant variables and the identification of the
interaction between variables. These insights are
invisible to one-way sensitivity analysis
approaches.
The preliminary results show that using RSM
for sensitivity analysis on decision analysis prob-
lems is a sensible approach. Further, RSM sub-
stantially increases the explanatory power of
sensitivity analysis by offering the decision analyst
an improved tool and presenting the decision
maker a more useful sensitivity analysis.
This paper is organized as follows. In Section 2,
influence diagrams and their subsequent sensitiv-
ity analysis are briefly described. An overview of
RSM and an introduction to the basic paradigm
are also provided. Then, it is shown that any
single or multi-attribute decision analysis value
(or utility function) can be transformed to a
response function of the uncertain variables.

CCC 1057–9214/99/030162-19$17.50 Recei6ed 8 October 1997

Copyright © 1999 John Wiley & Sons, Ltd. Accepted 6 April 1999
 RESPONSE SURFACE METHODOLOGY
Finally, the Oil Wildcatter Problem is introduced,
which will be used to illustrate the proposed
methodology. Next, the sensitivity analysis frame-
work is presented. Various screening experiments
are described and standard DA sensitivity analysis
procedures are applied to the problem. In Section
3, RSM is applied to generate an expanded sensi-
tivity analysis. Section 4 compares the DA sensi-
tivity analysis techniques with the present RSM
sensitivity analysis. Section 5 concludes this
paper.
2. BACKGROUND
In this section, we briefly discuss influence dia-
grams and standard DA sensitivity analysis proce-
dures. We also provide an overview of RSM and
its use within a general sensitivity analysis proce-
dure. The section ends with an expository prob-
lem description.
2.1. Influence diagrams
Howard provided an excellent summary of the
practice of decision analysis and identified the
central role of influence diagrams in the decision
analysis process (Howard, 1998). Howard and
Matheson (1983) developed the concept of an
influence diagram to provide a problem descrip-
tion that could be solved by a computer and yet
be understood by people. An influence diagram is
a graphical representation of the problem situa-
tion condensed to the decisions, the important
uncertain variables and the values (see Figure 1).
Decision nodes are depicted as rectangles, uncer-
Figure 1. Influence diagram
Copyright © 1999 John Wiley & Sons, Ltd.
163
tain event nodes as circles, and value nodes as
rounded rectangles. These three components are
pictorially represented as nodes connected with
arcs that describe their relationships. Decision
analysts use the influence diagram to communi-
cate with the decision maker and use an influence
diagram or decision tree algorithm to perform the
decision analysis.
Shachter (1986) describes the influence diagram
as ‘ . . . intuitive enough to communicate with de-
cision makers and experts and, at the same time,
precise enough for normative analysis.’ In the
same paper, Shachter proposed an algorithm for
evaluating influence diagrams. The capability to
solve the influence diagram was an important step
for decision analysis. The influence diagram pro-
vides analysts with a good communication tool in
a convenient structure for computer solution tech-
niques. There are several decision analysis soft-
ware packages on the market. One such software
package, DPL, solves the influence diagram and
its associated decision tree. The solution outputs
include optimal decision policies, cumulative risk
profiles and sensitivity analyses (ADA, 1995).
2.2. Standard sensitivity analysis in decision
analysis
Sensitivity analysis approaches in decision analy-
sis have been largely ad hoc. Rios Insua and
French (1991) review the sensitivity analysis litera-
ture, propose a framework for mulit-objective de-
cision making, and examine distance based tools
for sensitivity analysis. Lowell (1994) provides a
recent survey of the sensitivity analysis literature
and proposes a new sensitivity analysis, sensitivity
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
164 K.W. BAUER Jr. ET AL.
to relevance. Sensitivity to relevance shows the
effects of modelling relevance (or conditional de-
pendence) between the uncertain variables.
In DA sensitivity analysis, typically one vari-
able is varied at a time. DPL displays this type of
analysis via a device known as a rainbow dia-
gram, since the changes in optimal policy are
denoted by colour changes (ADA, 1995; pp. 468–
474). It is also common to plot several one-way
sensitivity analyses on one graph. This technique
is known as a tornado diagram since a ‘tornado’
shape is obtained by sorting the variables by their
variation in value function and plotting them top
to bottom by decreasing variation (Howard, 1998)
and (ADA, 1995; pp. 474 – 481). An analyst typi-
cally examines the diagram to determine the most
significant variables. Significance can be assessed
by considering the contribution to the response (a
large percentage of the response is due to the
variable), contribution to risk (the range of uncer-
tainty in the variable generates a large range is the
response), and contribution to indecision (the
variable impacts the choice of the best decision).
In our experience, selection of the significant vari-
ables is based on analyst and decision maker
judgement. As analysts, we may want to delete
variables that do effect the above three criteria.
However, our decision makers sometimes prefer
to keep variables in the model to respond to
stakeholder perceptions that the variables are
important.
Next, time permitting, two-way sensitivity
analyses are performed. Two-dimensional graphs
are plotted and the preferred strategy regions are
identified. These plots are called strategy region
graphs (Clemen, 1991; pp. 121 – 138). (A rainbow
diagram, tornado diagram and strategy region
graph are illustrated in the example in Section 3.)
McNamee and Celona (1990) describe joint sensi-
tivity analysis when the variables are probabilisti-
cally dependent. A general limitation of the
technique (and most DA software) is the lack of
an automated two-way (or higher) sensitivity
analysis.
Although performing sensitivity analysis on any
one variable can be an effective screening tool for
many problems, stopping there could miss a large
part of the potential information to be gleaned
from a model. If the problem contains many
variables, what is the effect of three or four being
slightly different than anticipated? If the answer
does not change the recommended action, it does
Copyright © 1999 John Wiley & Sons, Ltd.
not matter. If it does change the recommendation,
the decision maker needs to know. One approach
would be to exhaustively enumerate the possibili-
ties and run the model at all the possible combi-
nations of variable settings. This approach may
be reasonable for small problems with a few
variables, but quickly becomes impractical for
large problems.
2.3. Response Surface Methodology
RSM is a collection of mathematical and statisti-
cal tools used in investigative experimentation by
scientists and engineers. When applied to models,
such as influence diagrams or decision trees, it is
helpful to think of the model as being character-
ized by a feature of interest called an output or a
response. This response arises in reaction to sys-
tem inputs or independent variables. In this pa-
per, the response is the value used to solve the
influence diagram (decision tree) and the indepen-
dent variables will be the various probabilities and
outcomes of the uncertain variables which
parameterize the influence diagram. Generally,
there will be a different response function for each
alternative. Consider a situation in which a re-
sponse, y, can be approximated over a region of
operability, R, by either a first- or second-order
polynomial function of its inputs. If the input
variables are j1, j2, . . . , jk, then a typical region
of operability is defined by a set of inequalities;
Ll 5 jl 5 Ul, l= 1, . . . , k, where Ll and Ul are the
lower and upper bounds of the lth input or factor,
respectively. Further, it is often convenient to
standardize or code the variables as
jl − jl0
xl = 2
Ul − Ll
for l= 1, . . . , k, where jl0 is the midpoint be-
tween Ll and Ul. This coding maps the upper
(lower) bound to +1 ( − 1). There are advantages
in accomplishing this transformation. The vari-
ables are unitless, this aids in interpretation. Fur-
ther, it is easy to array experimental design
settings into an array called a design matrix (de-
scribed below) in such a way to produce orthogo-
nal columns that in turn lead to uncorrelated
estimates of model parameters (see Equation (1)),
which also greatly aids interpretation. Now, the
response is
k k
y= b0 + % bi xi + % bij xi xj + % bii x 2i (1)
i=1 iBj i=1
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY
where the bs are coefficients that must be esti-
mated from the data. Expression (1) can be
thought of as a Taylor’s series expansion (of
degree 2) of some true underlying function.
The purpose of using RSM as a sensitivity
analysis tool is to improve the analyst’s under-
standing of the sensitivity between the output of
the model (response) and its associated input vari-
ables and then use this information to improve
the decision model. There are two major benefits
of using RSM for sensitivity analysis on a deci-
sion analysis problem. The first is a substantial
reduction in the total number of influence dia-
gram solution (decision tree) cycles required for
thorough sensitivity analysis. As the model in-
creases in complexity, the reduction can be signif-
icant. The second major benefit is the added
insights available to decision makers from the use
of RSM. These insights are gained through the
estimation of specific coefficients (effects) of sig-
nificant variables and the identification of the
interaction between variables. These insights are
invisible to one-way sensitivity analysis
approaches.
The analyst must choose a numeric perturba-
tion range for each variable. These ranges define
the region of operability, R. Depending on the
decision model development, the perturbation
range may vary from differential perturbations
(small perturbations about the nominal values) to
large perturbations (due to our preliminary
knowledge of the variables). If the perturbations
are large, it is more likely that the first-order
model will be inadequate (as reflected by the
goodness-of-fit test and other diagnostics). This
means that for R, certain variables have a nonlin-
ear impact on the response — changes in one vari-
able are related to the levels of another. Later in
our model development, as variable ranges be-
come smaller, we could define a smaller R and
redo the RSM analysis. This may result in a
model with less interaction terms. In this paper,
Equation (1) is used to approximate the expected
value of alternatives in an influence diagram.
Often, Equation (1) is called a metamodel.
Box and Draper (1987) advocate the use of an
iterative approach to building models of the type
portrayed in (1). At each stage, the experimenter
needs to make decisions about which input vari-
ables to include, what levels to set the inputs to
and what order of model to estimate. For exam-
ple, if it is decided to observe n realizations of the
Copyright © 1999 John Wiley & Sons, Ltd.
165
underlying system or model with the express pur-
pose of estimating a function of the form (1) then
the settings of the factors can be arranged in a
design matrix
Æ1 x 11 · · · x 21kÇ
à Ã
1 x 21 · · · x 22k
X= Ã ·· Ã
— — —
à · Ã
È1 x n1 · · · x 2nkÉ
Each row represents a set of experimental condi-
tions as well as transformations of these condi-
tions. If the responses are arrayed in a vector
Y= (y1, . . . , yn )% then the coefficients b. =
(b0, b1, . . . , bk, . . . , bij, . . . , bkk )% can be esti-
mated by the well-known least-squares formula
b. = (X%X) − 1X%Y
The resolution of a design reflects the ability of an
experimental design to render clear estimates of
effects of differing orders. For example, a resolu-
tion IV design can render estimates of interactions
which are ‘clear’ of single-factor effect estimates.
However, in designs of this resolution, certain
interaction estimates may be ‘confounded’ with
other interaction estimates. This means that the
effect that is actually computed reflects both the
interactions in question. For example, in a resolu-
tion III design, it is impossible to distinguish
between the effects of first-order factors (xi ) and
certain two-way interactive effects (xixj ); however,
such a design does not confound first-order effects
with one another. Excellent RSM references are
Meyers (1976), Box and Draper (1987) and Khuri
and Cornell (1987).
RSM has been used extensively to perform
sensitivity analysis in computer simulation. Simu-
lation models typically have many input variables
and usually only a fraction of them are critical
over the design region. Meidt and Bauer (1992)
take advantage of this property and use RSM to
create metamodels for simulation. The flow of
sequential experimentation in this metamodelling
scheme is shown in Figure 2. The purpose of this
paper is to demonstrate the usefulness of this
general scheme when applied in the sensitivity
analysis of an influence diagram. Decision analy-
sis models are typically characterized by a large
variable (factor) set. Rather than initially attempt-
ing to set up extremely large experimental designs,
in which we vary all the variables, we employ a
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
166 K.W. BAUER Jr. ET AL.
Figure 2. RSM metamodelling paradigm
‘group screening’ strategy. We examine the struc-
ture of the model and arrive at groups based on
function. Different analysts may select different
initial groupings. As long as the regression results
are interpreted correctly, the final groupings of
the sensitive variables should be the same.
Group screening combines sets of factors that
are believed to exhibit like behaviour in the
model. Each group is treated as a single variable
(each variable in the group is changed by the
same amount) and regressed against the re-
sponses using a low resolution design. A Plack-
ett–Burman or similar resolution fractional
factorial design could be used to ensure effi-
ciency. This technique would be extremely benefi-
cial where the effects of uncertain variables are
closely correlated in a large model. It can drasti-
cally reduce the number of runs required while
identifying the significant variable groups. Small-
wood and Morris (1980) developed a grouping
technique along these lines, but outside the RSM
framework.
Copyright © 1999 John Wiley & Sons, Ltd.
Once the group screening narrows the field to
critical groups of input variables, another low
resolution design is run on the main factors re-
maining. From here, a first-order (linear) model
can be used to estimate the response surface over
a small area of interest. There may be a lack of
fit and a second-order model can be used to
better fit the surface. Using this methodology,
valuable insight is gained into the response func-
tion over a range of inputs.
Stone (1988) used RSM as an analysis tool in his
study the nonconformity of parts in the US Air
Force Supply System. He developed a decision
model for determining the best policy for sampling
parts in the inventory. The model was displayed as
an influence diagram and solved using INDIA
(INDIA, 1991). A post-solution analysis was per-
formed using RSM. For his decision problem,
Stone found that RSM explored the interaction
effects between the variables in a structured, con-
sistent and reliable way. His research provided the
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY
initial groundwork for applying RSM to decision
analysis problems. Bauer et al. (1995) com-
pared the use of RSM and current decision anal-
ysis sensitivity tools in a Department of Defence
force structure decision problem. In their app-
lication, RSM significantly reduced the analysis
man hours and provided a more insightful analy-
sis.
2.4. Converting single and multi-attribute decision
analysis problems to the Response Surface
Paradigm
To use the Response Surface Paradigm, we must
be able to identify a response function of
the form y= f(X) (see Equation (1)), where y is
the response and X is a vector of the system
inputs or independent variables. Using the fol-
lowing four cases, we demonstrate that we can
define a response function of the form y = f(X)
for all single and multi-attribute decision analysis
problems.
2.4.1. Single attribute
For single attribute decision analysis, y is the
single attribute. For example, y might be the net
present value of the alternatives. The vector X is
the vector of the input variables that we would
assess using one-way sensitivity analysis (e.g. a
tornado diagram). Therefore, using RSM we can
develop a response function for each alternative
of the form y =f(X). For single attribute, we
usually refer to f(X) as the value model.
2.4.2. Single attribute utility
When we use utility, we convert our single at-
tribute to a utility, u =u(y). From the above,
y = f(X), therefore u= u( f(X)) =g(X). Again, if
we let u be our response, we have a function of
the required form.
2.4.3. Multi-attribute Preference Theory
Without uncertainty, we have Multi-attribute
Preference Theory (Kirkwood, 1997; pp. 227–
270). We can write the value function of the
form:
6(z)= g(61(z1), 62(z2), . . . , 6n (zn ))
where 6(z) is the value function, g( ) is any gen-
eral function, 6i (zi ) are the single-dimensional
value functions, and z =(z1, z2, . . . , zn ) is the vec-
tor of scores for the n attributes. But, zi is a
function of the input variables X (see Section
2.4.1), therefore zi =hi (X). Substituting, we get
Copyright © 1999 John Wiley & Sons, Ltd.
167
y= 6(z)= g(61(h1(X)), 62(h2(X)), . . . , 6n (hn (X)))
= f(X)
Again, if we let y be our response (6(z) is a
scalar), we have a function of the required form.
2.4.4. Multi-attribute Utility Theory
With uncertainty, we have Multi-attribute Utility
Theory (Kirkwood, 1997; pp. 245–259). We can
write the value function of the form:
u(z)= g(u1(z1), u2(z2), . . . , un (zn ))
where u(z) is the multi-attribute utility function,
ui (zi ) are the single-dimensional utility functions,
and z= (z1, z2, . . . , zn ) is the vector of scores for
each attribute. But, z is a function of the input
variables X (see Section 2.4.1), therefore zi =
hi (X). Substituting, we get
y= u(z)= g(u1(h1(X)), u2(h2(X)), . . . , un (hn (X)))
= f(X)
Again, if we let y=u(z) be our response, we have
a function of the required form.
2.5. Illustrative problem
Since we have shown in the above section that we
can obtain a response function for all single and
multi-attribute decision analysis problems, for
simplicity, we use a single attribute decision anal-
ysis problem to illustrate the methodology. Raif-
fa’s Oil Wildcatter Problem is chosen for its
familiarity to decision analysts (Raiffa, 1968).
The problem consists of two decisions faced by
an oil wildcatter determining whether to make
preliminary tests on a particular drilling site and
whether to drill or not. In this problem, the test
decision is to be made between an experimental
seismic test costing $3000, a core sample test
costing $10000 and no testing. The two tests
provide information on the seismic structure
(and, therefore, the probability of various
amounts of oil being found) at the drill site. The
potential amount of oil is discretized into three
classifications: dry, wet and soaking. These three
amounts are outcomes with varying revenue; a
dry well will have no revenue, a wet well will
produce $120000, while a soaking well will give
$270000. There is a variable drilling cost, which
is discretized as $40000, $50000 or $70000. The
expected profit is the revenue (from the amount
of oil produced) minus the costs of drilling and/
or testing. The influence diagram of the Oil Wild-
catter Model is shown in Figure 1.
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
168 K.W. BAUER Jr. ET AL.
3. SENSITIVITY ANALYSIS
In the following sections we present a sensitivity
analysis of the Raiffa Oil Wildcatter Problem.
Our overall strategy is to accomplish this analy-
sis within the general framework of the RSM
metamodelling paradigm described in Section
2.3. First, we form two main groups of variables:
probability and value variables (described be-
low). Perturbing these variables as a group, we
form a standard tornado diagram to assess the
possibility of screening one (or perhaps both)
groups of variables from consideration in a sensi-
tivity context. If neither group perturbation pro-
duces a change in the optimal decision policy, we
could consider the optimal decision as being ro-
bust relative to the magnitudes of the perturba-
tions being examined. The next step in the
paradigm involves decomposing the remaining
groups into subgroupings and/or individual vari-
ables and performing the appropriate analysis.
Finally, we contrast the use of rainbow diagrams
versus RSM to generate two-way strategy re-
gions.
3.1. Grouping variables and initial perturbations
Initially, the variables were placed into two main
groups. The probability variable group included
the 24 conditional and marginal probabilities
associated with the possible outcomes of each
node. The value variable group included nine
variables, made up of the possible drilling and
testing costs and the amount of oil revenue.
Perturbations of these variables were limited by
the relationships within each group. For in-
stance, within the probability variable group,
each discrete distribution’s probabilities must
sum to 1. We array these probabilities into
a vector. This vector of probabilities was varied
by changing the largest probability (9 0.1),
and reducing the other probabilities pro-
portionally. The value perturbations were accom-
plished by using a change of 10% of the largest
outcome (e.g. the greatest amount of oil outcome
is $270000 and hence the perturbation was
9 $27000). For illustration, we have used 90.1
for probabilities and 9 10% for values. For deci-
sion analysis practice, other systematic schemes
have been proposed; for instance, one common
practice is to use the 10th and 90th percentile
values for each variable (McNamee and Celona,
1990).
Copyright © 1999 John Wiley & Sons, Ltd.
The initial solution shows that the optimal ex-
pected profit is realized through no testing and
deciding to drill. The expected profit was
$40000. A decision tree showing the optimal de-
cision policy is shown in Figure 3.
We use a tornado diagram to show the effects
of perturbing one main group at a time. This
one-way sensitivity analysis displays the effect of
the perturbations on the overall expected profit
and any changes in decision policy. The shaded
areas indicate that the optimal decision policy
has changed. DPL evaluates the model at three
points; the original base case and two endpoints
on either side of the base case. The shaded area
shows only that the change in policy occurred
somewhere between the base case and the end-
point evaluation. We note that in the application
of tornado diagrams, single variables are per-
turbed individually. Here, we refer to group
effects.
This initial group analysis gives some in-
sight into the effect of changing the prob-
ability or value parameters (see Figure 4). Per-
turbing any of these parameters can lead to
a significant change in expected profit and/or
decision policy. For example, we did not delete
the probability group because there was a sig-
nificant change in NPV—from − 25% to
+ 20%. Clearly, additional groups would be re-
quired.
3.2. Expanded groups
We decided on seven subgroups: four probabili-
ties and three value nodes. The seven new groups
are defined by using the initial nodes in the influ-
ence diagram. The probability group has one
variable for each chance node. These new vari-
ables are:
Oil revenue probability perturbation ORP
Seismic structure probability SSP
perturbation
Experimental seismic test probability ETP
perturbation
Drilling cost probability perturbation DCP
The value group has one variable for each node
with cost or profit values:
Drilling cost perturbation DC
Test cost perturbation ETC
Oil revenue perturbation OR
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY 169

Figure 3. Optimal decision policy of initial solution

Figure 4. Tornado diagram for initial groups

Copyright © 1999 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
170 K.W. BAUER Jr. ET AL.
Figure 5. Rainbow diagram for ORP
3.3. One-way analysis
These groups were perturbed in the same manner
as the initial groupings. The probability pertur-
bations were 9 0.1 from the significant element
in each vector. The value perturbations were
again 910% of the largest possible outcome
in each distribution. A rainbow diagram display-
ing the effects of ORP variation is shown in
Figure 5. The tornado diagram for this case is
shown in Figure 6. Of the seven new variables,
the following variables exhibited the greatest ef-
fect on expected profit for the preferred decision
policy:
Oil revenue probability perturbation ORP
Oil revenue perturbation OR
Drilling cost perturbation DC
3.4. Two-way analysis
Clemen (1991) discusses various methods for the
construction of strategy regions. The DPL rain-
bow diagram option was used to do two-way
Copyright © 1999 John Wiley & Sons, Ltd.
sensitivity analysis. There are alternative methods
of generating strategy regions using standard
software. A rainbow diagram displays the model
solutions across a range of input settings (up to
21, using DPL) for a single variable. If there is a
specified relationship between two variables, they
can be varied at the same time by defining one
variable in terms of the other, or by defining
both variables in terms of a third (ADA, 1995).
As a reasonable alternative, we employed a
method in which we produced a rainbow dia-
gram of one variable across the interval − 0.1 to
0.1 and generated this rainbow diagram ten sepa-
rate times (across the range of the second vari-
able). The second variable was incremented
between runs to establish the strategy region
graph seen in Figure 7. We display only one of
the three possible strategy region graphs using
the three remaining variables given above. The
extraction of the data from the rainbow dia-
grams after each run was a time consuming pro-
cess, but it did provide the means to gather
two-way sensitivity information.
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY 171

If the most likely values of the uncertain variables actual values of these variables. Notice that as the
are near the break point between the decision to test estimates approach the bottom left corner of Figure
or not to test using experimental seismic methods, 7, the level of confidence increases in experimental
the optimal decision would be highly sensitive to the seismic testing being the best decision.

Figure 6. Tornado diagram of seven factors perturbed one at a time

Figure 7. Strategy region based on perturbation of oil revenue (OR) vs. probability of oil revenue (ORP)

Copyright © 1999 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
172 K.W. BAUER Jr. ET AL.

Figure 8. Oil Wildcatter problem

Table I. Initial fractional screening design (resolution III)

RUN Seven input factors Expected profit of test alternative

1 2 3 4 5 6 7 No Core Exp
ORP ORPP DCP DCPP SSPP TCP ESTPP test sample seismic

1 −1 −1 −1 1 1 1 −1 50.8 44.775 46.8

2 1 −1 −1 −1 −1 1 1 82.2 71.2 78.2
3 −1 1 −1 −1 1 −1 1 16.2 20.8 25.3724
4 1 1 −1 1 −1 −1 −1 38.3 29.8 36.8
5 −1 −1 1 1 −1 −1 1 36.8 27.8 34.8
6 1 −1 1 −1 1 −1 −1 68.2 64.475 66.2
7 −1 1 1 −1 −1 1 −1 2.2 1.17 4.03963
8 1 1 1 1 1 1 1 24.8 30.07 33.4701
9 0 0 0 0 0 0 0 40 34.3 37

4. THE RSM APPROACH TO SENSITIVITY
ANALYSIS
To demonstrate the use of RSM, we begin by
screening the seven groups described previously.
First, a 27 – 4 (resolution III) fractional design was
used to screen these seven groups for significant
factors. Hence, the seven main factors, discussed
above, are not confounded with other main fac-
tors. This design requires eight runs of the model.
Additionally, a centre point was included in the
design to better evaluate whether a first-order
model would be sufficient. The nine runs were
made using DPL (Figure 8 is the modified influ-
ence diagram that facilitated this portion of the
analysis) and the design matrix was regressed
against the expected profit response for each alter-
native. The three decision alternatives are: (1) no
test, (2) a core sample test and (3) an experimental
seismic test. The design matrix and associated
responses are shown in Table I.
As will be described in this section, four of the
seven main factors were significant in terms of
approximating the response surface for the no test

Copyright © 1999 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY
option. The intercept estimate was the same as
our initial solution of 40 and indicated that a
first-order model might be sufficient for this op-
tion. The regression analysis of the remaining
options showed less promise relative to first-order
models. The resulting residual analysis suggested
that the first-order model was not sufficient to
approximate the core sample test and experimen-
tal seismic test options. The residuals for these
options indicated a large amount of error at the
centre point, which was unacceptable for our
needs.
In all three cases, OR, ORP and DC, were the
most significant factors in the estimated response
surface. Therefore, a 23 full factorial design was
run using these three variables as the three factors
plus all of their interactions. Based on residual
analysis, we chose to examine second-order mod-
els (with quadratic terms) for the two test options.
The search for a design that could minimize the
additional runs of the model led to a central
composite design. Central composite designs are
discussed by Box and Draper (1987). One prob-
lem with using a deterministic model is that repli-
cations of the centre point (base case) are
meaningless, since they would produce the same
response in all cases (the base case solution),
hence, attempting to produce designs with desir-
able qualities, such as uniform precision and ro-
tatability (see Meyers, 1976; p. 153) are not
Figure 9. Residual plot of No test alternative response surface approximation
Copyright © 1999 John Wiley & Sons, Ltd.
173
obtainable. However, a central composite design
would provide the orthogonality for our regres-
sion analysis, and take advantage of the full facto-
rial design points, which have already been run.
The idea is to augment the original factorial ar-
rangement with axial points. These points are
taken along each variables axis by setting the
other variables to zero. Six axial points were
added to our previous full factorial design. The
final design matrix is shown in Table II.
An example of the regression analysis is de-
picted in the ANOVA table and corresponding
residual analysis found in Table III and Figure 9.
In this application of regression it is important to
remember that there was no stochastic error;
hence, standard errors, t statistics and p values
cannot be interpreted in the usual fashion. These
measures are only useful in that they are reflective
of that amount of total variation explained by the
various tactors, e.g. large t statistics are typically
associated with factors that exhibit a significant
effect. In the end, a second-order approximation
for each of the three decision alternatives fit
the surfaces quite accurately. The t statistics in the
regression table are directly proportional to
the proportion of the total sums of squares
explained by each variable. In Table III, we see
that only the bold faced variables explain mean-
ingful fractions of the total sums of squares. That
is, the t values for these variables are only ones
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
Copyright © 1999 John Wiley & Sons, Ltd.

174
Table II. Central composite design with one centre point

Model I X1 X2 X3 X12 X13 X1 ‚ 2 X23 X2 ‚ 2 X123 X3 ‚ 2 Alternatives

run ORPP ORP DCP No test Core sample Exp seismic

1 1 −1 −1 −1 1 1 1 1 1 −1 1 50.3 40.37 47.3

K.W. BAUER Jr. ET AL.

2 1 1 −1 −1 −1 −1 1 1 1 1 1 16.7 15.43 18.088
3 1 −1 1 −1 −1 1 1 −1 1 1 1 82.7 72.7 79.7
4 1 1 1 −1 1 −1 1 −1 1 −1 1 38.3 32.17 35.3
5 1 −1 −1 1 1 −1 1 −1 1 1 1 36.3 31.55 33.3
6 1 1 −1 1 −1 1 1 −1 1 −1 1 2.7 7.73 11.004
7 1 −1 1 1 −1 −1 1 1 1 −1 1 68.7 58.7 65.7
8 1 1 1 1 1 1 1 1 1 1 24.3 24.47 25.692
9 1 0 0 0 0 0 0 0 0 0 0 40 34.3 37
18 1 1.68 0 0 0 0 2.83 0 0 0 0 7.2051 10.1663 13.2313
19 1 −1.68 0 0 0 0 2.83 0 0 0 0 72.7949 62.7949 69.7949
20 1 0 1.68 0 0 0 0 0 2.83 0 0 62.7042 52.7042 59.7042
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)

21 1 0 −1.68 0 0 0 0 0 2.83 0 0 17.2958 16.5907 19.0603

22 1 0 0 1.68 0 0 0 0 0 0 2.83 28.2275 27.3542 28.2681
23 1 0 0 −1.68 0 0 0 0 0 0 2.83 51.7725 41.7725 48.7725
Copyright © 1999 John Wiley & Sons, Ltd.
Table III. Regression analysis for No test option on central composite design

CCD–NO TEST
SUMMARY OUTPUT ANOVA

Regression statistics df SS MS F Signif F

Multiple R 1 Regression 10 8409.4758 840.94758 4.695E+11 1.089E−23

R square 1 Residual 4 7.164E−09 1.791E−09
Adjusted R square 1 Total 14 8409.4758
Standard error 4.232E−05
Observations 15

RESPONSE SURFACE METHODOLOGY

RESIDUAL OUTPUT

Coeff Standard error t statistic p value Observ Predict Actual Residuals

Intercept 40 4.21E−05 950 702 7.345E−24 1 50.299974 50.3 2.63E−05

ORP −19.5 1.15E−05 −1 702 755 7.137E−25 2 16.700003 16.7 −3.348E−06
OR 13.5 1.15E−05 1 178 831 3.107E−24 3 82.699972 82.7 2.789E−05
DC −6.99999 1.15E−05 −611 245 4.298E−23 4 38.300002 38.3 −1.766E−06
X12 −2.7 1.5E−05 −180 448 5.659E−21 5 36.299998 36.3 1.766E−06
X13 4.441E−16 1.496E−05 2.968E 1 6 2.7000279 2.7 −2.789E−05
− 11
X1 ‚ 2 0 1.72E−05 0 1 7 68.699997 68.7 3.348E−06
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)

X23 4.441E−16 1.49E−05 2.968E 1 8 24.300026 24.3 −2.63E−05

− 11
X2 ‚ 2 0 1.72E−05 0 1 9 40 40 0
X123 4.441E−16 1.496E−05 2.968E 1 10 7.2050647 7.2051 3.526E−05
−11
X3 ‚ 2 0 1.72E−05 0 1 11 72.794935 72.7949 −3.526E−05
12 62.704202 62.7042 −1.881E−06
13 17.295798 17.2958 1.881E−06
14 28.227471 28.2275 2.918E−05
15 51.772529 51.7725 −2.918E−05

175
176 K.W. BAUER Jr. ET AL.

Table IV. Response surface coefficients

Factors Parameter estimates Parameter estimates Parameter estimates

No test Core sample Seismic test

Constant 40.0 35.25 37.09

ORP −19.5 −15.53 −16.92
OR 13.5 11.25 12.09
DC −7.0 −4.57 −5.80
OR*ORP −2.7 −3.25 −4.11
ORP*DC 0.93 1.41
OR*DC
ORP*OR*DC
OR2
ORP2 1.04
DC2

Table V. One-way sensitivity analysis using RSM equations

Range of perturbation

ORP −0.1 −0.08 −0.06 −0.04 −0.02 0 0.02 0.04 0.06 0.08 0.1
No test 59.5 55.6 51.7 47.8 43.9 40 36.1 32.2 28.3 24.4 20.5
Core sample 50.8 47.7 44.6 41.5 38.4 35.3 32.1 29.0 35.9 22.8 19.7
Exp seismic 55.1 51.3 47.6 44.0 40.5 37.1 33.7 30.5 27.3 24.2 21.2
Max 59.5 55.6 51.7 47.8 43.9 40.0 36.1 32.2 28.3 24.4 21.2
OR −27 −21 −15 −9 −3 0 3 9 15 21 27
No test 26.5 29.5 32.5 35.5 38.5 40.0 41.5 44.5 47.5 50.5 53.5
Core sample 24. 26.5 29 31.5 34 35.25 36.5 39 41.5 44 46.5
Exp seismic 26.0 28.3 30.7 33.2 35.8 37.1 38.4 41.2 44.1 47.1 50.2
Max 26.5 29.5 32.5 35.5 38.5 40.0 41.5 44.5 47.5 50.5 53.5
DC −7 −5.6 −4.2 −2.8 −1.4 0 1.4 2.8 4.2 5.6 7
No test 47 45.6 44.2 42.8 41.4 40 38.6 37.2 35.8 34.4 33
Core sample 39.8 38.9 38.0 37.1 36.2 35.3 34.3 33.4 32.5 31.6 30.7
Exp seismic 43.9 42.4 40.9 39.6 38.3 37.1 36.0 34.9 34.0 33.1 32.3

significantly different from 0. Figure 9 shows that
the fit is good since the residuals are small and
there is no systematic pattern. The response sur-
face approximations are shown in Table IV. We
included those coefficients that were not approxi-
mately zero. These equations are the expected
profit response for each of the three options. They
are shown here in their coded form. One quick
accuracy check for the centre point can be accom-
plished by comparing the intercept with the initial
centre point (e.g. 40=40, 34.3: 35.25, 37:
37.09). These response surface equations can be
used to create standard sensitivity analysis tools.
Tornado diagrams are used to assess one-way
sensitivity, while strategy regions are used to ex-
amine two-way sensitivity. Both of these tools
provide useful pictorial representations of critical
variables but do not explicitly identify the interac-
tions between variables. The table of response
surface coefficients offers two benefits. First, the
table identifies the linear effects of each variable
and as well as the interactive effects. Second, the
table allows for direct comparisons of effects
across variables without requiring an overlay of
tornado diagrams.
4.1. One-way analysis
Table V shows the results of comparing the three
alternative RSM equations when the factors are
perturbed one at a time. The preferred alternative
is determined by evaluating:

Copyright © 1999 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY
Max(E(profit)No test , E(profit)Core ,
sample
E(profit)Exp seismic ) (5)
This calculation produces data that very closely
matches the rainbow diagram in Figure 5 and the
tornado diagram shown in Figure 6.
4.2. Using response surface approximations to
generate strategy regions
The strategy region graph introduced in Figure 7
is an excellent tool for presenting two-way analy-
sis. The new approach was to use the three RSM
equations to estimate each alternative’s expected
profit. Each of the equations was evaluated across
the region of interest. The alternatives were com-
pared to generate a strategy region graph like that
shown previously in Figure 7. The result shows
that there are only two optimal options in this
region (No test and Experimental seismic testing).
Therefore, the expected profit for the core sample
test is never the highest in this region. The line
separating the strategy regions for the RSM equa-
tions depicted in Figure 10 was constructed by
solving
Figure 10. Strategy region comparison for model vs. RSM
Copyright © 1999 John Wiley & Sons, Ltd.
177
E(profit)No test − E(profit)Exp seismic = 0.
5. ANALYSIS
5.1. Comparison of one-way sensitivity analyses
The objective of one-way sensitivity analysis is to
identify significant factors and their impact on the
optimal policy. The first technique applied to
generate this one-way analysis was the tornado
diagram. The tornado diagram was produced in
DPL, which varies one factor at a time and
evaluates the model at the base case and the
endpoints. This required 14 additional runs of the
model (the endpoints of seven factors). The result-
ing tornado diagram, shown in Figure 6, indicates
that changes in one of the seven factors would
affect the optimal policy.
ORP impacted the optimal decision and had
the greatest impact on the overall expected profit.
The other factors are not critical to the overall
decision in this range, when taken one at a time.
These variables are listed here in descending order
of impact on the overall expected profit: ORP,
OR, DC, DCP, SSP, ETC and ETP.
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
178 K.W. BAUER Jr. ET AL.
It is important to realize that tornado diagrams
show changes in decision policy (either decision),
not just the initial decision. The analyst should
run the model at specific points of interest to
clarify the information gained from the tornado
diagram. This will require some additional runs of
the model, but should ensure that the correct
conclusions are drawn. Also, since (in typical
software) the tornado diagram does not specify
where the decision policy changes, it is necessary
to systematically narrow the range to pinpoint
this crossover. Again this requires a number of
additional runs of the model (rainbow diagram),
depending on the accuracy required.
The top three variables in the tornado diagram
were ranked in the same order as suggested by
examination of the coefficients of the RSM equa-
tions, providing validation of our use of RSM.
The resulting equations are highly significant in a
least-squares (curve fit) sense. It is a simple exer-
cise to plug in the variables at the base case,
where all of our coded variables are zero. From
this, it is established that the maximum of the
three alternatives is the E(profit)No test. One-way
sensitivity analysis can then be accomplished by
varying one factor at a time and comparing the
three expected utilities. This point of view would
indicate that there are only three significant vari-
ables: ORP, OR and DC. Obviously, there is
some information lost at this point due to the
approximation. The question is how much. Upon
closer inspection of the tornado diagram, our
coefficients of regression have picked out the most
significant factors in predicting the optimal test
decision. Again, there is only one variable that
affects the initial test decision (ORP). With this in
mind, the information lost may not be as critical
as it initially appears.
Some indication of accuracy can be calculated
after some clarification of the tornado diagram. It
was found that the decision policy change-over
point predicted by the approximations agrees to
within 0.01 of the factor perturbation. This level
of accuracy is far beyond that required for the
purpose of sensitivity analysis in this model. The
cost, in terms of model runs, is the eight fractional
screening design runs plus the 23 full factorial
design runs plus the six axial points in the central
composite design. These 22 runs exceed the 14
runs needed for a basic tornado diagram, but the
tornado diagram analysis required an additional
ten or more runs to clarify the results. RSM’s
Copyright © 1999 John Wiley & Sons, Ltd.
frugal use of model runs has only begun to be
exploited, since no further runs are required to
develop additional strategy region graphs. The
RSM equations can be used in the current form to
predict a response to any number of changes in
the parameters across the evaluated design region.
The usefulness of the approximations formulated
using RSM is seen again in the two-way analysis
discussed in the next section.
5.2. Discussion of two-way sensitivity analysis
Standard sensitivity analysis leads to an examina-
tion of two-way perturbations. The obvious fac-
tors of interest are those that proved significant in
the one-way analysis; but other variables may
have significant two-way interactions. As dis-
cussed previously, DPL rainbow diagrams were
used to collect the data required for the two-way
sensitivity analysis. Each rainbow diagram re-
quired eleven runs of the model. The model was
evaluated at ten intervals of OR, ranging from
− 27 to 27. ORP was set across the range (from
− 0.1 to 0.1) for these runs. It would take 11 such
rainbow diagrams to complete a strategy region
as seen in Figure 7. The total cost, in terms of
model runs (influence diagram solution cycles), is
110 per strategy region. In addition, we generated
the strategy region by using the data off each of
the eleven rainbow diagrams. This would amount
to a significant amount of influence diagram eval-
uations and substantial analyst time if there were
a number of significant factors involved. Further,
in the case where there are five significant factors,
two-way sensitivity analysis would require:
5
= 10 strategy region graphs.
2
These ten strategy region graphs would demand
a total of 1100 model runs plus the time spent
pulling 1100 data points off the rainbow dia-
grams. Another disadvantage of using the rain-
bow diagram is that the optimal policy change is
shown, but the actual optimal policy is not specif-
ically identified. This requires model runs to de-
termine the optimal decision policy. At this point
in the analysis, the initial method required nearly
160 model runs as compared with RSM’s 22.
Obviously, RSM can lead to a significant savings
in time and model runs. However, the RSM equa-
tions are approximations; hence, some accuracy
has been forfeited.
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
 RESPONSE SURFACE METHODOLOGY
There were several strategy regions generated
with both methods for comparison. The results
were similar from an accuracy viewpoint. The
RSM method was accurate to within $1000 of
expected profit results and agreed with the rain-
bow diagram based strategy regions to 9 0.01 of
the decision crossover-point. A comparison of
strategy regions for the two methods is shown in
Figure 10. As before, the result shows that there
are only two optimal options in this region (No
test and Experimental seismic testing). Therefore,
the expected profit for the core sample test is
never the highest in this region. These strategy
regions would be used to locate areas were the
decision policy is sensitive to changes in two
variables. Therefore, this accuracy appears suffi-
cient for sensitivity analysis in decision analysis.
The 22 RSM runs gave response surface equa-
tions that could be used to produce sensitivity
analysis on any combination of the seven group
factors involved. This efficiency would be more
pronounced as the model increases in complexity.
Finally, there is one additional advantage of the
RSM equations for each decision alternative. If
there are no interaction terms, the analyst is as-
sured that no two-way sensitivity analysis is re-
quired. This is an important insight on the
problem structure and can result in savings of
analyst time.
6. CONCLUSION
This paper illustrated the use of RSM as a sensi-
tivity analysis tool for DA. It was shown that any
single or multi-attribute DA value (or utility)
function can be transformed to a response func-
tion of the uncertain variables. A sensitivity anal-
ysis framework designed to facilitate simultaneous
perturbation of a number of uncertain variables
was presented. RSM was used with influence dia-
gram models, but the methodology can also be
used with decision tree models. The present ap-
proach was illustrated with the well-known Oil
Wildcatter Problem.
RSM can be used efficiently and effectively in
the sensitivity analysis of decision analysis prob-
lems. The RSM approach is more efficient than
current DA sensitivity analysis techniques and the
RSM coefficients can provide improved insights
for decision makers. RSM offers the potential to
significantly reduce the time to perform the sensi-
Copyright © 1999 John Wiley & Sons, Ltd.
179
tivity analysis. For the illustration, the one-way
and two-way sensitivity analysis results were very
close. The minor loss of accuracy by using the
RSM approach for sensitivity analysis is offset by
the analyst time savings. In addition, RSM coeffi-
cients (which show the direct impact of the vari-
able on the alternatives value/utility) can provide
important insights for the decision makers.
REFERENCES
ADA Decision Systems, DPL™ Ad6anced Version User
Guide. ADA Decision Systems, 2710 Sand Hill Road,
Menlo Park, CA 94025, 1995.
Bauer, K.W., Parnell, G.S. and Meyers, D.A., ‘Identi-
fying key uncertainty relationships in decision analy-
sis via designed experiments’, Proceedings of the
NATO AC/243 (Panel 7) Symposium on ‘Coping with
Uncertainty in Defence Decision Making’, The Hague,
The Netherlands, 16 – 18 January 1995.
Box, G.E.P. and Draper, N.R., Empirical Model-Build-
ing and Response Surfaces, New York: Wiley, 1987.
Clemen, R.T., Making Hard Decisions: An Introduction
to Decision Analysis, Boston: PWS-Kent Publishing
Company, 1991.
Howard, R.A., ‘Decision analysis: practice and
promise’, Manage. Sci., 34, 679 – 695 (1998).
Howard, R.A. and Matheson, J.E., Readings on The
Principles and Applications of Decision Analysis, Vol.
I, General Collection, Strategic Decisions Group,
1983.
Khuri, A.I. and Cornell, J.A., Response Surfaces: De-
signs and Analyses, New York: Marcel Dekker, 1987.
Kirkwood, C.W., Strategic Decision Making: Multiob-
jecti6e Decision Analysis with Spreadsheets, Belmont,
CA: Wadsworth Publishing, 1997.
Lowell, D.G., ‘Sensitivity to relevance in decision anal-
ysis’, Ph.D. Dissertation, Department of Engineering-
Economic Systems, Stanford University, 1994.
McNamee, P. and Celona, J., Decision Analysis with
Supertree, 2nd edn. South San Francisco, CA: The
Scientific Press, 1990, p. 155.
Meyers, R.H., Response Surface Methodology, Virginia
Polytechnic Institute and State University, 1976.
Meidt, G.J. and Bauer, K.W. Jr., ‘PCRSM: A decision
support system for simulation metamodel construc-
tion’, Sim, 59, 183 – 191 (1992).
Raiffa, H., Decision Analysis: Introductory Lectures on
Choices under Uncertainty, Menlo Park, CA: Ad-
dison – Wesley, 1968.
Rios Insua, D. and French S., ‘A framework for sensi-
tivity analysis in discrete multi-objective decision
making’, Eur. J. Oper. Res., 54, 176 – 190 (1991).
Shachter, R.D. ‘Evaluating influence diagrams’, Oper.
Res., 34, 871 – 882 (1986).
J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)
180 K.W. BAUER Jr. ET AL.

Smallwood, R.D. and Morris, P.A., ‘A task force deci-
sion analysis’, Oper. Res., 28, 61–80 (1980).
Stone, M.A., ‘Decreasing nonconformance of parts in
the air force supply system’, MS Thesis, AFIT/GOR/
MA/88D-6, School of Engineering, Air Force Insti-
tute of Technology (AU), Wright – Patterson AFB
OH, December 1988 (AD-A206162).
INDIA™, User’s Guide, Version 2.0, Student guide,
Decision Focus Incorporated, Boston: PWS-Kent
Publishing Company, 1991.

Copyright © 1999 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 8: 162 – 180 (1999)