Comput Geosci (2013) 17:167–188
DOI 10.1007/s10596-012-9323-1
ORIGINAL PAPER
A simultaneous perturbation stochastic approximation
algorithm for coupled well placement and control
optimization under geologic uncertainty
Lianlin Li · Behnam Jafarpour ·
M. Reza Mohammad-Khaninezhad
Received: 30 September 2010 / Accepted: 19 September 2012 / Published online: 1 November 2012
© Springer Science+Business Media Dordrecht 2012
Abstract Development of subsurface energy and en-
vironmental resources can be improved by tuning im-
portant decision variables such as well locations and
operating rates to optimize a desired performance met-
ric. Optimal well locations in a discretized reservoir
model are typically identified by solving an integer
programming problem while identification of optimal
well settings (controls) is formulated as a continuous
optimization problem. In general, however, the deci-
sion variables in field development optimization can
include many design parameters such as the number,
type, location, short-term and long-term operational
settings (controls), and drilling schedule of the wells. In
addition to the large number of decision variables, field
optimization problems are further complicated by the
existing technical and physical constraints as well as the
uncertainty in describing heterogeneous properties of
geologic formations. In this paper, we consider simulta-
neous optimization of well locations and dynamic rate
allocations under geologic uncertainty using a variant
of the simultaneous perturbation and stochastic ap-
proximation (SPSA). In addition, by taking advantage
of the robustness of SPSA against errors in calculat-
ing the cost function, we develop an efficient field
development optimization under geologic uncertainty,
where an ensemble of models are used to describe
important flow and transport reservoir properties (e.g.,
L. Li · M. R. Mohammad-Khaninezhad
Texas A&M University, College Station, TX 77843, USA
B. Jafarpour (B)
University of Southern California, 925 Bloom Walk,
HED 313, Los Angeles, CA 90089, USA
e-mail: behnam.jafarpour@usc.edu
permeability and porosity). We use several numerical
experiments, including a channel layer of the SPE10
model and the three-dimensional PUNQ-S3 reservoir,
to illustrate the performance improvement that can be
achieved by solving a combined well placement and
control optimization using the SPSA algorithm under
known and uncertain reservoir model assumptions.
Keywords Well placement · Production optimization ·
Simultaneous perturbation stochastic approximation
(SPSA) · Geologic uncertainty · Field development
1 Introduction
A significant capital cost in developing subsurface
reservoirs for energy (e.g., hydrocarbon, geothermal),
hydrological (groundwater), and environmental (nu-
clear and fossil fuel wastes) resource development is
related to drilling of injection/production wells. Poorly
selected well locations can lead to inefficient field
development and inadequate reservoir performance.
Optimal well placement can significantly improve the
performance of subsurface reservoirs and reduce the
overall cost of field development. In optimizing the
overall reservoir life-cycle performance, in addition to
identifying optimal well locations, one needs to specify
the best operational setting (working fluid rates or
pressures) for each well. Furthermore, the number of
wells to be drilled and the drilling schedule (in se-
quential drilling) are among the decision variables that
can play important roles in maximizing the net present
value (NPV) of the project and improving the sweep
efficiency and productivity of the system. To date, re-
search in this area has mainly focused on addressing
168
only one of the above optimization problems at a time.
In particular, well placement optimization has been
carried out by assuming fixed injection/production well
controls [2, 3, 9, 15, 26] and production optimization is
performed to find optimal well controls for fixed well
locations [4, 18]. When solved separately, each of these
problems can only provide a suboptimal solution to the
overall field performance optimization. A more general
formulation is to solve the coupled well placement and
control optimization problem simultaneously. To com-
plicate the problem even further, one needs to consider
the unavoidable uncertainty and the risk implications
due to lack of complete and perfect knowledge about
heterogeneous reservoir flow properties. A generalized
problem formulation, however, can be too complex and
have too many degrees of freedom to solve for. In this
paper, as a step toward this generalization, we only
combine the well placement and production optimiza-
tion problems.
The current industry practice in finding optimal well
locations is based primarily on semi-manual meth-
ods where a team of geoscientists and reservoir en-
gineers combine engineering and geologic expertise
and judgment with available information and simulated
response of the reservoir to identify reasonable well
locations. Numerical simulation is typically employed
to facilitate the process by quantifying the performance
of candidate well configurations. When combined with
dependable geologic information and reservoir engi-
neering expertise, manual techniques may be reliable
for small reservoirs with simple geologic descriptions
where very few well location candidates may exist; how-
ever, manual placement of multiple wells in large reser-
voirs with complex geologic structures is less likely to
provide near-optimal well locations. Several optimiza-
tion algorithms have been developed in the literature
to provide a systematic way to explore a broader set of
scenarios that aim at finding improved well location and
configurations.
Automatic well placement techniques are generally
classified as (heuristic) stochastic methods such as ge-
netic algorithm and simulated annealing techniques
[2, 3, 15] that are hoped to search for the global so-
lution, and gradient-based deterministic methods such
as adjoint-based algorithms that can only find local
solutions [9, 18, 25]. Hybrid techniques that exploit
the advantages of the two approaches have also been
developed. Although stochastic approaches are easy
to implement and aim at finding the global solution,
they do not guarantee a monotonic improvement of
the objective function with successive iterations and
usually do not scale well for application to large prob-
Comput Geosci (2013) 17:167–188
lems. Gradient-based techniques, on the other hand,
guarantee a monotonic improvement in the objective
function with each iteration; however, they can only
find local solutions. Regardless of the method used,
the computational bottleneck in well placement opti-
mization is associated with the complexity of the re-
peated objective function evaluations under different
well configuration scenarios that require many full
numerical flow simulations. Therefore, automatic ap-
proaches, whether stochastic or deterministic, have
been hampered by the overwhelming computational
cost of running hundreds or thousands of full-blown
simulations before finding a solution. The computa-
tional cost can become prohibitive when robust opti-
mization is considered in which the objective function
is selected to optimize the performance of the reser-
voir under a large number of model descriptions that
account for the uncertainty in the knowledge about the
reservoir properties [24].
Gradient methods provide a computationally more
efficient alternative to stochastic search methods. A
key component of gradient-based search methods is
the calculation of the objective function gradient with
respect to well locations. In recent years, development
of efficient adjoint techniques for calculating the gra-
dients of well production responses to well control
variables has led to a number of creative attempts to
approximate the gradients with respect to well locations
using pseudo wells and adjoint methods [9, 18, 25]. In
this approach, pseudo wells are placed in the imme-
diate neighboring grid blocks to each major well in
the reservoir domain. The pseudo wells are wells of
the same type as the major target wells in the sim-
ulation (injection or production) with sufficiently low
flow rates to minimize their effect on the simulation
results. The standard adjoint method is then used to
compute the gradient of the objective function with re-
spect to pseudo-well rates. At each iteration, among the
pseudo wells surrounding each major well, the one with
maximum gradient is used to approximate an ascent
direction for well placement optimization (that is, the
major well is moved to the location of the neighboring
pseudo well with the largest gradient) [9].
A correct interpretation of the adjoint-based gra-
dient information in the pseudo-well approach is in
fact different from the gradients needed for well place-
ment. The interpretation of gradients calculated by the
pseudo-well approach is as follows: “assuming a major
well with large injection/production quantities is at the
center of the neighboring pseudo wells, and given the
small injection/production rates of the pseudo wells, a
small (local) increase in the rate of the pseudo well with
Comput Geosci (2013) 17:167–188
the largest gradient leads to a locally steepest improve-
ment in the objective function.” In this approach, the
∂f
derivatives ∂q i
(where f is the objective function and q
is the rate for the pseudo well i) of the pseudo wells are
∂f
used to approximate the true gradients ∂u i
, where ui is
the position index. In addition, gradient calculation is
carried out when a well with high rate is present and
surrounded by the pseudo wells (instead of moving the
major well to neighboring locations and computing the
corresponding gradient using numerical perturbations).
To improve the gradient definition in [9], Sarma
and Chen [18] proposed that the Dirac delta function
representing the well (sink/source) terms in reservoir
simulation be approximated by a multivariate Gaussian
function (describing the spatial distribution and rates
of many pseudo wells around the major target wells).
After adopting this modification, the adjoint method
is borrowed to first find the gradient of the objective
function f (◦) with respect to pseudo-well rates and
then convert them to the gradients with respect to well
locations using the chain rule of differentiation and the
functional relation between pseudo-well locations and
rates. Wang et al. [25] also used a gradient-based well
placement optimization where they started the opti-
mization by placing one well with constant rate in all
grid blocks and iteratively removed the wells based on
the gradient information to improve the objective func-
tion that accounted for drilling costs. In this approach,
the discrete variables (i.e., well locations) are replaced
with continuous decision variables (i.e., well rates) and
whether a grid block is a good candidate for drilling
a well is inferred from the rates. The numerical ex-
periments presented in [9, 18, 25] suggest that, despite
the approximations involved, adjoint-based gradients
provide reasonable ascent directions for well placement
optimization. However, the approximations involved
in computing the gradients in this method can lead to
occasional iterations at which the NPV maximization
objective function is decreased.
Within the reservoir engineering context, gradient-
based methods can only be efficient if efficient ad-
joint models are used to calculate the gradients. How-
ever, constructing an adjoint model is a relatively
involved process that requires additional development
in the forward modeling source codes. An alterna-
tive approach is the stochastic optimization method.
In particular, the simultaneous perturbation stochastic
approximation (SPSA) has been developed for use with
large dimensional multivariate optimization problems
in several applications where the gradient information
is not available and when the objective function can
be noisy. The theoretical convergence of the SPSA
169
algorithm along with several variations of it, includ-
ing discrete SPSA [6, 7, 11], adaptive (second-order)
SPSA [21], and global search SPSA [14], has been
reported in the literature. In [16], a coordinate descent
approach was developed for large-scale optimization
with SPSA algorithm. The SPSA algorithm has also
been compared with other stochastic search methods
[23]. An important result is that [19, 22, 23] for a p-
dimensional problem under reasonably general condi-
tions, the SPSA algorithm reaches the same level of
accuracy as the finite-difference stochastic approxima-
tion method for a given number of iterations; however,
SPSA uses p times fewer function evaluations at each
iteration. The efficiency of SPSA becomes particularly
significant when the number of decision variables is
large. This implies that, overall, a properly generated
random perturbation of all p variables is as effective as
approximating the p partial derivatives one at a time.
In [22, 23], SPSA is compared with several state-of-the-
art optimization methods such as simulated annealing
and genetic algorithm, where it is shown to be quite
competitive (and possibly more efficient) in terms of
computational time for the same solution accuracy.
As an alternative to gradient-based approaches, a
variant of SPSA (integer SPSA) has been introduced
to solve the well placement problem by Bangerth et al.
[2]. In [2], the authors use the standard SPSA algo-
rithm in a set of numerical waterflooding experiments
and reported nearly optimal well locations with a high
probability. As implied by the name, the SPSA was
introduced as a stochastic method, and its convergence
analysis is carried out within the framework of the
stochastic theory. A randomized coordinate descent
(RCD) algorithm has also been recently proposed as
an alternative stochastic approximation method [16]. In
this approach, at each iteration, only one component
of the gradient vector is approximated and used to
determine a descent direction. Our preliminary evalua-
tion of this method for field development applications
has indicated a comparable performance to the SPSA
algorithm.
In this paper, we investigate a variant of the SPSA al-
gorithm for solving of the combined well placement and
production optimization. We maximize the objective
function f (xi , yi )i=1,2,...,N that often denotes the NPV
of the reservoir asset, by placing N wells in a reservoir
and simultaneously optimizing their life-cycle rate al-
locations. Here, xi , yi are the position of the ith well
in x- and y-coordinates, respectively. Although f (◦) is
a continuous function of xi and yi , the discretization
of the model and direct-delta form of the source/sink
functions in the mathematical formulation of the fluid
170
flow equations complicate the gradient calculation [18].
We, therefore, use the stochastic perturbation approach
to approximate ascent directions for improving the ob-
jective function. In addition to solving the combined
optimization problem, we evaluate the performance of
the SPSA algorithm under geologic uncertainty. To
reflect the uncertainty in reservoir model description,
we use an ensemble of 100 model realizations with
distinct flow property distributions, where each model
has a different production response under an identical
injection/production scenario. In this case, we maximize
the expected value of the NPV function over the entire
ensemble of models. This approach uses the expected
value of NPV to hedge the risk of field development
under geologic uncertainty so that the resulting well
locations and rates provide reasonable NPV values for
the majority of the models considered. In this case, we
exploit the stochastic nature of the SPSA algorithm and
its robustness to noisy objective functions to develop an
efficient solution algorithm.
In the remainder of the paper, we present the pro-
posed methodology and discuss its application to si-
multaneous well placement and control optimization in
experiments involving waterflooding of oil reservoirs
under geologic uncertainty. In Section 2, we briefly
present the optimal field development problem formu-
lation followed by the description of the SPSA algo-
rithm for combined well placement and dynamic rate
allocation optimization in Section 3, where we also dis-
cuss an efficient and robust field development approach
by exploiting an important property of the SPSA al-
gorithm. In Section 4, we present several numerical
experiments to demonstrate the performance of the
proposed method under reservoir model uncertainty.
We conclude the paper by providing a summary of
our findings and a general discussion of the challenges
and opportunities in formulating and solving the gen-
eralized field development optimization problems in
Section 5.
2 Problem statement
In this section, we discuss the proposed simultane-
ous well placement and rate allocation optimization
problem formulations, starting with the governing flow
equations.
2.1 Governing flow equations
We consider a two-phase incompressible and immisci-
ble flow system (oil/water) in a three-dimensional (3D)
Comput Geosci (2013) 17:167–188
space. The flow equations after ignoring the capillary
pressure (i.e., po = pw = p) can be compactly written
as [1]
 
krm
∇ · K (x, y, z) ∇ p (x, y, z, t)
μm
∂ Sm (x, y, z, t)
= φ (x, y, z)
∂t
qm (x, y, z, t)
− , m = o, w (1)
ρm (x, y, z, t)

So + Sw = 1,  := {(x, y) 0 ≤ x ≤ Lx , 0 ≤ y ≤ L y ,
0 ≤ z ≤ Lz }.
where Sm (x, y, z, t) denotes the saturation for phase m
and p(x, y, z, t) refers to the pressure field; the absolute
permeability and porosity of the rock are represented
by K(x, y, z) and φ(x, y, z), respectively. In our exam-
ples, we have assumed that porosity φ(x, y, z), is ho-
mogeneous with a constant value of 0.2 throughout the
domain. The fluid properties are defined by viscosities
μw,o (assumed constant) and saturation-dependent rel-
ative permeability krw,o (Sw ), which are described with
the following quadratic functions:
(s∗ )2 (1 − s∗ )2 ∗ sw − swc
λw (sw ) = , λo (sw ) = , s =
μw μo 1 − sor − swc
(2)
where sor is the irreducible oil saturation and swc is
the connate water saturation. We have also assumed
that μo = μw = 1.0 and so = swc = 0. The source and
sink terms are denoted by qm with a negative value
for production wells and positive value for injection
wells (and are nonzero only if the grid cell (x, y, z) is
intersected by a well). For the simulations in this paper,
we have assumed no flow boundary conditions, i.e.,
K(∂, t) ≡ 0, where the boundary of the domain  is
denoted by ∂. The initial conditions for the pressure
and saturation are given by
p (x, y, z, t = 0) = p0 (x, y, z) (3)
Sw (x, y, z, t = 0) = Sw0 (x, y, z) . (4)
Following [1], the governing flow equations are solved
using the finite elements discretization of the domain
into Ngb grid blocks to obtain the spatiotemporal distri-
bution of saturation and pressure (the dynamic states of
the system). For compactness, we combine the well grid
block indices in a vector u of length Nw = Ninj + Nprod
Comput Geosci (2013) 17:167–188
where each component ui represents the location (lin-
ear index) of the ith well and Ninj and Nprod denote the
number of injection and production wells, respectively.
In addition, we assume that all wells are fully completed
in the vertical direction to eliminate the need to include
the vertical index in the well locations search space.
We note that a more challenging problem is finding the
entire well trajectory in directional completions, which
we do not consider in this paper.
2.2 Optimization problem formulation
Optimization of well locations and operating controls
(rates in this paper) can be formulated using the fol-
lowing problem statement:
 
q̂, û = arg min f q (u, t) , u
q∈R,u∈Z
 
s.t. g q (u, t) , u, m = 0 (5)
q (u, t) ∈ q and u ∈ u
Here, f (◦) is a specified objective function (dis-
cussed next), q(u, t) contains continuous decision vari-
ables representing the well control setting (injec-
tion/production rates in this paper) with the feasible set
q , the set of discrete decision variables u has the fea-
sible set u , and g(q(u, t), u, m) denotes the discretized
multiphase flow equations with m as the model input
parameters (e.g., petrophysical and fluid properties).
We note that in this formulation both the simulation
time and number and type of wells (Ninj and Nprod ) are
explicitly specified by choosing a predetermined simu-
lation time and assuming a fixed number of injection
and production wells. In general, however, the number
and type of wells, the drilling schedule, and the reser-
voir lifetime may be considered as additional decision
variables. In this paper, we implement a generalized
SPSA algorithm to solve the above mixed integer pro-
gramming problem. First, we apply the original SPSA
algorithm to solve the integer programming subprob-
lem in which the rates q(u, t) are known and fixed but
the well locations are unknown; that is, we seek only the
optimal well locations u for a predetermined number
of wells with specified controls by solving the following
optimization problem (in this case, the decision variable
q is dropped):
ûwp = arg min f (u)
u∈Z
  (6)
s.t. g u, q, m = 0 and u ∈ u
where ûwp denotes the solution of the well placement
problem. We then modify the SPSA algorithm for
171
application to simultaneous well placement and rate
allocation optimization problem in Eq. 5 and evalu-
ate its performance under two different assumptions:
(1) deterministic case in which the reservoir model is
known and (2) stochastic case, where multiple realiza-
tions of the reservoir model are considered to repre-
sent geologic uncertainty. For field optimization under
geologic uncertainty, we take advantage of the SPSA’s
robustness against noisy objective functions and intro-
duce an efficient implementation that, at each iteration,
only uses a small subset of randomly selected model
realizations to compute the expected value of the cost
function (instead of using the full ensemble of prior
models).
2.3 Objective function
For a single deterministic reservoir model, we consider
the minimization objective function f (◦) to be the neg-
ative of the reservoir NPV. In general, the objective
function can be any user-specified performance mea-
sure. In hydrocarbon production problems, the NPV
function contains the revenues from oil production and
the costs of water injection and water disposal/reuse
at production wells, along with other operation costs.
The objective function in our examples does not include
the capital cost associated with drilling wells since we
assume a predetermined number of wells with a fixed
cost of drilling. In addition, since we fix the amount
of injected water, we only consider the cost of water
production. While the objective function can, in gen-
eral, be more complex than what has been considered
in this paper, the choice of the objective function is
not likely to affect the solution approach. For a known
reservoir model, we write the objective function in the
form
  
K−1 N
prod
−rw qkw,n (u) + ro qko,n (u)
f u, q = − tk (7)
(1 + b )a
n=1
k=0
where the superscript k is the time index (with K as
the total number of time steps in the simulation), n is
the production well index, and ro and rw are the oil
price and cost of water production (per unit volume),
respectively. In this formulation, the NPV objective
function depends on the oil production rate qo and
water production rate qw . In Eq. 7, the time step size
is denoted by tk , the discount rate by b (expressed as
annual interest rate) and the number of years passed
since the start of production by a.
172
In practice, the exact distribution of reservoir prop-
erties (such as permeability) can be highly uncertain.
as (1) choosing a
A practical approach to account for geologic uncer-
tainty in predicting the reservoir flow behavior and
response is to use several models. Under ensemble-
level reservoir modeling and production forecasting,
one can define the corresponding optimization objec-
tive function in terms of the point statistics of the NPV
function [24]. A simple choice is to use the expected
value of the NPV over the existing models, that is
1 
Nens
      
fRobust u,q = E f u,q = f j u,q (8)
Nens j=1
where E[·] denotes the statistical expectation operator
and Nens denotes the number of model realizations
(ensemble size) used to calculate the objective function.
The choice of an ensemble-based objective function can
vary depending on the adopted risk attitude. In general,
one could include other statistics of the NPV function
such as the variance or percentiles in describing the
objective function, which is not considered in here. In
the next section, using the robustness of the SPSA al-
gorithm to noise in the objective function computation,
we propose a computationally efficient implementation
for the case when the objective function is defined over
an ensemble of reservoir models. In this approach, at
each iteration, instead of using an ensemble of size Nens ,
we include only Napprox  Nens “randomly selected”
model realizations in computing the cost function. This
results in a significant reduction in the computational
cost when robust optimization is considered. We fur-
ther discuss this approach in the methodology section
and show promising results in a number of numerical
experiments.
In the discussion that follows, for notation clarity,
we express the optimization problem in the following
compact form:
x∗ = arg minx∈X f (x) (9)
where the decision variable x is defined over
 a set of
T
feasible well locations and rates, i.e., x = u q . We
first consider the case where the decision variables are
the locations of the wells, x = u. Since the model is
discretized, the search space x is parameterized by the
integer lattice of the cell midpoints. The optimization
is performed by finding a descent direction at each
iteration to improve the value of the cost function. The
SPSA algorithm that was first introduced by Spall [19]
and later extended to several variants [5, 10, 21, 22]
presents a good candidate to solve the above discrete
optimization problem [11].
Comput Geosci (2013) 17:167–188
The basic idea of the discrete SPSA algorithm for
well placement can be outlined
“proper” random direction k = k1 , k2 , . . . , kp =
{ki }i=1: p from a distribution with bounded inverse ex-
pectation [11, 19, 23] (for example, a Bernoulli random
number) in the search space at each iteration and (2)
finding an ascent direction (using an approximate gra-
dient estimate) by computing a two-sided simultaneous
perturbation using the selected random direction.
Given the random choice of direction k , it is natural
to expect that a step in the selected direction does not
necessarily result in an improvement of the objective
function; meaningful computational saving over the
finite-difference perturbation would only be achieved
if very few random directions (relative to number
of decision variables) are needed to yield marked
improvements in the objective function. Interestingly,
under reasonably general conditions, it is shown in
[19, 23] that SPSA achieves a similar objective function
solution accuracy to finite-difference gradient approx-
imation, but with 1/ p times function evaluations.
In the original algorithm, a few (gain) parameters
that control the step lengths and stability of the algo-
rithm have to be carefully chosen [19]. In its basic form,
outlined above, the SPSA algorithm can only operate
on unbounded continuous sets and is not suited for
optimization on our bounded integer lattice x. The dis-
crete SPSA algorithm has been proposed and analyzed
in the literature [6, 7, 11]. In a related context, Bangerth
et al. [2] applied the SPSA algorithm to the discrete well
placement problem and reported promising results.
3 Methodology
We discuss the generalized SPSA optimization algo-
rithm in this section. We first present an overview of the
original SPSA algorithm and its important properties
prior to discussing the formulation we have used to
solve the joint field development optimization problem.
3.1 The SPSA algorithm
The SPSA algorithm derives its efficiency from its
approximate gradient estimation property, which is
based on two function calls. This property makes the
algorithm suitable for large-dimensional optimization
problems where gradient information is not available
or easy to compute [23]. Moreover, the approximate
nature of gradient estimation provides robustness to
noisy measurement of the cost function, which is an-
other key property of the SPSA algorithm [23]. Hence,
Comput Geosci (2013) 17:167–188
the algorithm provides an attractive option for solv-
ing large-scale problems under stochastic cost func-
tions and where gradient information is unavailable or
difficult to compute. It has been shown that under fairly
general conditions, the method can be superior to other
gradient-free optimization methods [23].
Denoting noisy measurement of the objective func-
tion f (x) and its gradients with respect to the vector of
decision variables x ∈ R N as y (x) = f (x) + ε and ∇ f (x),
respectively, the SPSA algorithm finds the solution x∗

such that g (x∗ ) = ∂ f∂x(x)  ∗ = 0. The minimizer is found
x=x
using the following standard recursion algorithm:
xk+1 = xk − ak ĝk (xk ) (10)
where ĝk (xk ) is a stochastic approximation of ∇ f (xk )
at the kth iteration and ak is the gain sequence (up-
date step size). The iterative form in Eq. 10 represents
the steepest descent algorithm if ĝk (xk ) is replaced by
∇ f (xk ). A central finite-difference approximation of
the gradients can be written as
y (xk + ck ei ) − y (xk − ck ei )
ĝki (xk ) = (11)
2ck
where ei denotes a vector with a one in its ith place
and zeros elsewhere and ck is some “small” positive
number. The finite-difference approximation loses its
applicability as the problem dimension p grows. The
SPSA algorithm provides an efficient alternative by
adopting a stochastic gradient approximation ĝk (xk )
that is defined as
y (xk + ck k ) − y (xk − ck k )
ĝk (xk ) =
2ck
173
approximation of the gradient. Under reasonably gen-
eral conditions, using the asymptotic normality of the
iterates for large k values, Spall [19, 23] shows that a
proper selection of the perturbation vector can realize
a p times computational savings relative to the finite-
difference approximation.
Using a differential equation approach, Spall [19]
presents the sufficient conditions under which the re-
cursion in Eq. 11 leads to the stochastic convergence
(x∗ )
to the solution ∂ f∂x = 0. The conditions for conver-
gence proof require that the gain sequence goes to
0 at rates neither too slow nor too fast (specifically,

∞
ak , ck > 0 ∀k; ak → 0, ck → 0 as k → ∞ and ak =
k=0

∞
∞, ak
ck
< ∞), that the objective function f (x) is
k=0
sufficiently smooth near the solution x∗ and that, as
stated earlier, the perturbations ki are independent
and symmetrically distributed
about 0 with finite in-
verse moments E |ki |−1 . Under these conditions, the
bias in ĝk (·) as an estimator of gk (·) vanishes as k →
∞ and x converges to x∗ in an “almost sure” sense
[19]. In addition, Spall [19] shows that the asymptotic
distribution of the iterate is normal, or more specifically
 
kβ /2 xk − x∗ → N (μ, ) as k → ∞ (13)
where β > 0 depends on the form of gain sequences,
μ is a function of the Hessian matrix and the third
derivative of f (x) at x∗ , and  is determined by the
Hessian of f (x) at x∗ and variance of the noise in the
objective function [19, 23]. The latter result has been
used as a theoretical framework in [23] for comparing
the performance of the SPSA algorithm with other

T established stochastic search methods. Finally, Spall
× −1 k1 −1
k2 . . . −1
kp (12)
in which the user-generated p-dimensional random
 T
perturbation vector k = k1 , k2 , . . . , kp con-
tains independent and symmetrically distributed (about
0)  entries with finite inverse moment expectation
E |ki |−1 for all k and i values. A common distrib-
ution that fulfills these requirements is the symmetric
Bernoulli (±1) distributions (note that the uniform and
normal distributions do not satisfy these conditions).
The appeal for the SPSA algorithm originates from the
computational advantage in approximating the gradi-
ent (i.e., the SPSA algorithm requires 1/ p times the
computation needed in a regular central difference
gradient calculation). This computational saving can,
however, be offset by the work per iteration if the
approximation in Eq. 11 does not provide a reasonable
[19, 23] provides useful guidelines on the selection of
the gain sequence parameters that can be used for
proper implementation of the SPSA algorithm. The
steps for implementing the original SPSA algorithm are
as follows:
1. Generate a vector of random perturbations k =
 T
k1 , k2 , . . . , kp that satisfies finite inverse ex-

pectation, i.e., E −1 ki , condition. The Bernoulli
random values taking ±1 fulfill this requirement
while Gaussian and uniform random vectors do
not.
2. Use the guidelines provided by Spall [19, 20] for
selecting ck and ak to take a real positive step size ck
+
and compute the objective functions
−
 + at xk = xk−+

ck k and xk = xk − ck k , i.e., y xk and y xk ,
respectively.
174 Comput Geosci (2013) 17:167–188

3. Using Eq. 11, compute the approximate gradient improved field development. Recalling the general op-

T
the ĝk (xk ) = ( k )2ck ( k ) −1
y x+ −y x− −1 −1
k1 k2 . . . kp .
4. With an appropriate value of the gain sequence ak
(see [19, 20]), find the new vector of variables from
Eq. 10, i.e., xk+1 = xk − ak ĝk (xk ).
While the original SPSA and its convergence proofs
are developed for continuous problems, extension of
the method to discrete optimization problem has been
considered in [6, 7, 11]. The extension of this approach
to a second-order SPSA optimization algorithm has
also been presented in [21]. SPSA has also been further
extended to a global random search algorithm (the the-
oretical results for convergence to the global solution
may be found in [14]).We begin by implementing and
evaluating the SPSA algorithm for the well placement
problem first. We note that application of the SPSA
algorithm to well placement problems was first intro-
duced by Bangerth et al. [2]. Our goal in this paper is
to extend the application of the SPSA to the mixed-
integer problem of combined well placement and well
control optimization.
3.2 Well placement with SPSA
We now consider a special case of the implementation
for generating k ∈ R N in the discrete well placement
problem. Note that the feasible set of each component
of  is finite and limited to the indices of the grid cells
in the domain. For any given
p-dimensional, real unit 
vector  p ∈  with u =  j ∈ R Nw , j = 1, 2, ..., J ,
where J is the total number of feasible values for each
entry of  p , the SPSA procedure summarized above
can be applied to solve the well placement optimization
problem. In an alternative stochastic approximation
method, RCD [16], at each iteration, one randomly
selects a single element of k and solves a one-
dimensional optimization problem in a manner analo-
gous to the coordinate-descent approach [16]. The main
difference is that in the coordinate-descent method, a
complete deterministic gradient is computed and the
optimization is performed with respect to the coordi-
nate or component with the largest gradient. For the
well placement problem, our investigation shows that
the two methods, SPSA and RCD, provide comparable
results (not shown). In this paper, we only present the
results related to the SPSA algorithm.
3.3 Simultaneous well placement and control
with SPSA
Next, we discuss the simultaneous problem of finding
well locations and dynamic well rate allocations for
timization objective function in Eq. 5, the correspond-
ing objective function for the combined well placement
and rate allocation optimization for a single model
and an ensemble of models is expressed in the same
way as in Eqs. 7 and 8, respectively. Note that in this
case, the NPV objective function f (u, q) depends on
both u and q as decision variables. To facilitate the
discussion that follows, we introduce a concise form of
the optimization problem as
 T   T 
x̂ = û q̂ = arg minx∈ f x = u q . (14)
We reiterate that the resulting optimization is a mixed-
integer programming problem where the feasible set
for u is discrete and bounded by the number of grid
blocks in the model domain while q belongs to a con-
strained (by operating conditions) continuous set. In
the SPSA algorithm used to solve the joint problem,
we proceed by first defining the feasible set for up-
date directions. During the course of the optimization
iterations, the update directions are randomly picked
from the feasible set. As in the well placement problem,
the set of vectors used to update the well locations are
specified as
u =  j ∈ R Nw , j = 1, 2, ..., J . (15)
For optimizing the well rates, we define a feasible set
of update directions by accounting for the constraints
on well rates. In our implementation, we impose a
total injection and production rate constraint (one pore
volume per year). Mathematically, the constraint is
expressed as
   
q j∈PROD  = q j∈INJ  = 1PV. (16)
1 1
Since the initial solution for well rates satisfies this con-
straint, we define the perturbation sets for updating

the
well control settings as prod := x ∈ R Nprod
, xi = 0
 i

for producers and inj := x ∈ R , yi = 0 for in-
Ninj
i
jectors to ensure balanced source and sink terms. The
feasible set for rate controls is defined as q = prod ⊕
inj . Therefore, the feasible update directions can read-
ily be defined as the combination of the feasible sets for
the well placement and rate allocation problems
 = u ⊕ q = u ⊕ prod ⊕ inj . (17)
We note that the general SPSA for combined well
placement and control optimization can be imple-
mented by either sequentially updating the two types
of decision variables (well locations and rates) or con-
currently updating them at each iteration. For the
Comput Geosci (2013) 17:167–188 175

placement problem in this paper, we limit the step
size of the optimization line search to a single grid
block. In the next section, we evaluate the performance
of the proposed algorithm for optimizing a suite of
waterflooding experiments.
3.4 Efficient implementation under geologic
uncertainty
In robust optimization where the expected value of the
NPV objective function is maximized, a main practical
limitation is the computational cost. In general, two
types of robust optimization approaches have been
used to address the problem, namely, stochastic ap-
proximation (SA) and sample average approximation
(SAA) [17]. The idea behind SAA is rather simple:
generate Nens samples to approximate the expectation
of the objective function. On the other hand, the work-
ing principal behind SA is to generate N samples and
use only one sample at each iteration to approximate
the objective function. It has been shown that the SA
approach can be competitive and in some cases may
outperform the SAA method [12].
Since the SPSA algorithm is originally developed for
application with noisy objective functions, the method
is suitable for problems in which the objective function
computation is inexact. That is, if the SPSA algorithm
computes y (x) = f (x) + ε instead of f (x) as the objec-
tive function, where ε represents the noise in measuring
the objective function, stochastic convergence can be
established in terms of the noise statistics, among other
conditions [19]. We exploit this important property
to propose a computationally efficient algorithm for
solving robust field development problems. Namely,
instead of computing the exact ensemble-based ob-
jective function in Eq. 8 by using Nens of realization,
we approximate this function by using only Napprox 
Nens randomly selected model realizations, that is we
compute
    
fRobust u, q + ε = E f u, q + E[ε]

Napprox
1  
= f j u, q (18)
Napprox j=1
as a noisy version of the objective function. As a re-
sult, the cost of calculating this objective function is
significantly reduced from Nens to Napprox full reservoir
simulations. We emphasize that at each iteration the
Napprox model realizations are selected randomly to
ensure that the entire ensemble of Nens are included
in computing the objective function throughout the
optimization procedure. When Napprox is too small, the
objective function can become too noisy and exhibit os-
cillatory behavior. As Napprox increases, this oscillatory
behavior is reduced and a more stable convergence is
expected. We illustrate the application of this approach
in significantly reducing the computational cost of ro-
bust optimization in the Section 3.
4 Numerical experiments
We present several numerical examples to evaluate
the performance of the proposed SPSA algorithm for
simultaneous well placement and dynamic rate allo-
cation optimization. The reservoirs used in our exam-
ples include Reservoir A, a two-dimensional (2D) two
phase (oil–water) model discretized into 45 × 45 × 1 =
2,025 grid blocks; Reservoir B, a 60 × 220 × 1 = 13,200
two-dimensional model borrowed from the channel-
ized section of the SPE10 benchmark model; and
Reservoir C, a 28 × 19 × 5 = 2,660 three-dimensional
PUNQ-S3 model. Reservoir parameters and impor-
tant simulation inputs are summarized in Table 1. We

Table 1 General simulation information for the examples used in the paper
Parameter Test case Reservoir A Reservoir B Reservoir C
Phases Two-phase (o/w) Two-phase (o/w) Two-phase(o/w) Two-phase (o/w)
Simulation time 5 × 50 days 5 × 50 days 10 × 30 days 10 × 25 days
Rock porosity 0.20 0.20 0.20 Heterogeneous
Initial water saturation 0.0 0.0 0.0 0.0
Injection volume 1PV 1PV 1PV 1PV
Cell dimensions 45 × 45 45 × 45 60 × 220 28 × 19 × 5
Reservoir dimensions 450 × 450 m 450 × 450 m 600 × 2,200 m 280 × 190 × 50 m
Injection well constraint Water rate Rate Rate Rate
Production well constraints Total fluid rate Total fluid rate Total fluid rate Total fluid rate
Number of injection wells 3 (fixed) 3 5 5
Number of production wells 1 2 10 3
176
assume the price of oil is $80/barrel and the cost of pro-
ducing water is $20/barrel. The annual rate of interest
is assumed zeros (b = 0).
Testing the original SPSA algorithm We first test the
original SPSA algorithm (as a proof of concept) by
considering a waterflooding experiment in which three
water injection wells with fixed locations and rates are
present in the field (Fig. 1), and the goal is to find the
best location for the production well. This is an overly
simplified example to examine the performance of the
SPSA algorithm in finding the known correct solution.
Note that, in this paper, we are interested in the more
complex problem in which the locations or rates of
all the wells are unknown. For the general problem,
the response surface cannot be plotted for verification.
Hence, here we only consider an overly simplified
case to check the performance of the original SPSA
in finding the solution. Since the SPSA algorithm is a
local search method, we start the optimization problem
from several initial guess locations to better explore
the solution search space. To this end, we divide the
solution space into s × s subdomains (with s = 3 in this
paper) as shown in Fig. 1a. We use the center gridblock
in each region to initialize the optimization search.
For this simple problem where there is only a sin-
gle decision variable, we can compute the NPV for
all possible well locations and plot the NPV response
map. The resulting NPV response surface for placing
a single producer is shown in Fig. 1b. A number of
remarks are in order at this point. First, the NPV map
appears to be smooth with high values in the center.
This is because the injection well locations are pre-
specified and configured such that the central regions
of the domain, far from the injectors, are more likely
Fig. 1 Log-permeability
model and corresponding (a)
NPV response map; reference
log-permeability and NPV
response maps. 7 8
a Log-permeability
distribution for the reference
model with nine subdomain
specifications used to
initialize the optimization 4 5
algorithm. b NPV response
surface for all possible well
locations in Example 1. In
each plot, x- and y- axes show 1 2
the cell numbers in the
respective direction
Comput Geosci (2013) 17:167–188
candidates for well locations. Therefore, with growing
distance from the injection wells, the NPV function
is expected to increase. Second, as can be seen from
Fig. 1b, two local solutions (marked with blue crosses)
are identified. This behavior is, however, only specific
to this example setup where a single decision variable
is assumed. The generalized mix-integer multivariate
optimization problem that we consider in this paper is
expected to have very complex objective functions with
multiple local solutions.
Figure 2a shows the reduction in the objective func-
tion for six different initial solutions as the SPSA iter-
ations proceeds. In this test case, the proposed SPSA
algorithm converges to the solution in less than 30
iterations irrespective of the initial well location. In
general, the final solution and convergence rate depend
on the initial location of the well, which necessitates
initialization of the algorithm from multiple locations.
Figure 2a shows that one out of the six trials shown
converges to the local solution with a lower NPV
value. The first four columns in Fig. 2b display example
initial well locations (top) and the corresponding oil
saturation distributions at the end of the simulation
(bottom) while the last column shows the same plots
for the final solution (identical for all the cases shown).
This simplified test case was only used to examine
the convergence of the SPSA algorithm for the case
with one single decision variable for which the exact
solution is known. In the general problems considered
in this paper, the goal is to simultaneously identify the
optimal location and control rates for several wells,
for which many local solutions usually exist. We now
consider scenarios with increasing level of complex-
ity where the objective function surface can be very
complicated.
(b)
9
6
3
Comput Geosci (2013) 17:167–188
Obj. fun. (-NPV)
Iteration
Injection well (fixed)
Well locations
y- index
x-index x-index
Oil saturation
y- index
x-index x-index
Fig. 2 Placement of a single production well for which the op-
timal solution is known from the NPV response map in Fig. 1b.
a Objective function (-NPV) value versus optimization iterations
for different initial solutions assuming a known geologic model
in Example 1. b Optimization results for Example 1: the f irst
4.1 Optimization for a known reservoir
Example 1a: placing two producers and three injectors
As our second example, we consider the simultaneous
placement of three injectors and two producers, i.e.,
without fixing the positions of any of the wells. The
search space for this problem has, taking into account
symmetries due to identical results when exchanging
2,025!
wells of the same type, 5!2,020! possible configurations,
which is clearly far too many for an exhaustive search.
For this example, initialization of the algorithm is car-
ried out as follows: first, the reservoir is divided into s ×
s subdomains; then, the centers of randomly selected
five subdomains are chosen as injection and production
well locations.
Figure 3a plots the objective function against the
iterations for four different initial well configurations.
The results indicate that all runs finally reach a function
177
(a)
(b)
Production well (variable)
x-index x-index x-index
x-index x-index x-index
through fourth columns contain the initial well locations (top)
and the corresponding final oil saturation profiles (bottom); the
optimized well configuration (top) and final oil saturation profile
are shown on the f ifth column
value of −2.2e + 9 or lower, respectively, within the
first 200 function evaluations. Figures 3b, c shows four
initial and optimized distributions of oil saturation at
the end of the simulation time. Well locations for the
two producers and three injectors have been marked
with crosses and circles, respectively. The final NPV
solutions are also reported which show clear improve-
ment over the initial configurations. From Fig. 3, it
is clear that while the relative distance between in-
jectors and producers in the solutions seems reason-
able, in some cases, two injectors or two producers
are clustered close to each other, suggesting that one
of them can be eliminated and be compensated for
by increasing the rate at the other well (not per-
formed here). Other investigators have also observed
a similar issue in well placement problems [2, 9]. One
way to address this well clustering issue is to incor-
porate a well distance constraint [13]. Examination
178 Comput Geosci (2013) 17:167–188

(a) Iteration of the objective function (–NPV)

-NPV(×109)
Iteration

(b) Final oil saturation maps with initial well locations

NPV(×109) 1.42 1.38 1.39 1.58
y-index

x-index x-index x-index x-index

(c) Final oil saturation profiles

NPV(×109) 2.21 2.22 2.20 2.19
y-index

x-index x-index x-index x-index

Fig. 3 a Objective function (-NPV) value versus optimization
iterations for four different initial solutions using the known
geologic model in Example 1 to find the best locations for two
producers and three injectors. b The distributions of oil satu-
ration at the end of simulation time for four different initial
well configurations: the two injectors and three producers are
indicated with cross and circle markers, respectively. c Optimized
distributions of oil saturation at the end of simulation time after
well placement

of the oil saturation profiles indicates that in some
cases, however, each production well is producing oil
from different sections of the reservoir. The clustering
of similar wells in a region may be attributed to the
number of wells and the rate of injection and produc-
tion a priori. In a more general problem formulation
where the number of wells and the injection/production
rates are variable, this issue is less likely to be present
if the cost of drilling wells is incorporated into the
objective function. We also note that the final well
locations tend to be close to the boundaries of the
reservoir domain. While this may not be an issue if the
reservoir boundaries are known with confidence, it can
be questionable when the boundaries of the reservoir
domain are uncertain. In our problem formulation, we
have not considered such uncertainties.
Example 1b: simultaneous well placement and control
optimization In this section, we study the simultane-
ous optimization of well locations and their rate trajec-
tories in time for three injectors and two producers. The
initial well locations are selected in the same way as
in Example 1a. The dynamic injection and production
rates are constrained to inject and produce a total of
one pore volume (1PV) during the entire simulation
time. The control time intervals are set at 10 days and
the simulation time is 250 days, resulting in 25 control
time steps.
Figure 4a shows the objective function behavior dur-
ing optimization iterations using the proposed algo-
rithm for four different initial well configurations. As
can be verified from Fig. 4a, despite the variability
in the solutions, almost all four solutions reach the
Comput Geosci (2013) 17:167–188 179

(a) Iteration of the objective function (–NPV)

-NPV(×109)
Iteration

(b) Final oil saturation profiles with optimized well locations

9
NPV(×10 ) 2.41 2.48 2.50 2.49
y index

x-index x-index x-index x-index

(c) Optimized well rate trajectories

Fig. 4 Joint optimization of the well locations and time-
dependent rate controls for three injectors and two producers.
a The objective function (-NPV) versus iterations for Example 3
with known geologic model and by initializing the solutions from
different regions. Here, the location and time-dependent rates of
two producers and three injectors are sought. b Optimized distri-
butions of oil saturation at the end of simulation time after well
placement; the two injectors and three producers are indicated
with cross and circle markers, respectively. c Optimized time-
dependent injection (f irst row) and production rates (second
row) trajectories for four different initial well configurations
corresponding to b

function values of −2.5e + 9, which is a significant im-
provement over the NPV values obtained from the ini-
tial well configurations (randomly picked). This value
represents about 10 % improvement over the NPV
obtained from the well placement problem alone, which
highlights the value of solving the joint optimization.
In all cases, the solution is obtained in less than 800
iterations. Figure 4b shows the final oil saturation
distributions and well locations solution. In addition,
the optimized injection (top) and production (bottom)
rate trajectories are shown in Fig. 4c. Although the
solution well configurations are different (due to the
presence of several local solutions), the final NPV val-
ues and the amount of bypassed oil are quite similar.
In most cases, the well configurations are reasonable in
terms of the distance between injector/producer pairs.
180 Comput Geosci (2013) 17:167–188

However, in a few cases, wells may be clustered in a in this example, we demonstrate that even if the result
particular region. Next, we apply the method to a layer of optimal control is used as the best case scenario
of the SPE10 model to gain additional insight. for the initial reactive control, a combined well place-
ment and rate allocation optimization would lead to
significant improvement over a reactive control with
Example 2: channel layer of SPE10 model In this sec-
fixed (non-optimized) well locations.
tion, we apply the proposed method to a more realistic
We performed optimization for well placement, rate
reservoir model borrowed from the SPE10 benchmark
controls, and joint optimization of well locations and
model. We optimize the locations and operating rates
rate controls using the proposed SPSA method. The
for a set of five injectors and ten producers in a chan-
corresponding distributions of oil saturation at the end
nel formation that is discretized into 60 × 220 × 1 grid
of the simulation time are shown in Fig. 5c–e, respec-
blocks. The log-permeability distribution for this model
tively. The optimized NPV values for each case are also
is shown in Fig. 5a (left). The significant contrast be-
reported on top of the saturation plots in Fig. 5b–e. In
tween the extreme permeability values in this example
comparison with the initial solution, the NPV shows
leads to well-defined streaks of water saturation in the
67, 82, and 110.42 % improvements, respectively, for
reservoir after waterflooding. Other reservoir parame-
well placement, rate control, and joint optimizations.
ters and simulation inputs are the same as in previous
This suggests that solving the joint optimization results
examples. Initially, each injector performs under a uni-
in approximately 25 % and 15 % NPV improvement
form rate to inject 0.2 PV of water into the reservoir
over optimizing, respectively, the well locations and
(total injection of 1PV) during the entire simulation
rate controls separately.
time. Each production well produces a total of 0.1 PV
The results from these experiments suggest that the
with a constant rate throughout the reservoir life. The
proposed SPSA algorithm is an effective, easy to im-
corresponding oil saturation at the end of the simula-
plement, and flexible approach for joint optimization
tion time, with the initial configurations, is shown in
of well locations and rates and that random selection
Fig. 5a (right). The injectors and producers are marked
of the descent direction may be advantageous in cer-
with circles and crosses, respectively. The NPV corre-
tain type of problems where gradients are not readily
sponding to this initial well configuration and operating
available.
rates is $0.758 × 109 . A more realistic scenario for
specifying the controls in the initial case would be using
the results of a reactive control scenario. This approach Example 3: 3D PUNQ-S3 model As a final exam-
would make the real improvements achieved through ple, we applied the SPSA algorithm to the three-
well control optimization more meaningful. However, dimensional PUNQ model with a complex geologic

Fig. 5 Application to a (a) (b) (c) (d) (e)

channelized layer of the
NPV(×109): 0.758 1.23 1.39 1.59
SPE10 benchmark model.
a True log-permeability map
and oil saturation distribution
after waterflooding with
b uniform injection and
production rates at fixed well
locations, c well placement
optimization with SPSA
algorithm, d well rate
optimization using SPSA
algorithm, and e combined
well placement and rate
optimization using the
generalized SPSA algorithm 1

8 0.8
6
0.6
4
0.4
2

0 0.2

-2 0
Comput Geosci (2013) 17:167–188 181

structure. In this problem, we consider five injection
wells and three production wells. The first two rows in
Fig. 6 show the porosity and permeability distributions
of the PUNQ-S3 model. As can be seen, different layers
of the model show distinct permeability and poros-
ity distributions. Application of the proposed SPSA
method to this example resulted in the final saturation
plots shown in Fig. 6c–f. Figure 6c shows the initial sat-
uration profile for each layer. Figures 6d–f display the
final saturation plots for the well placement (WP) only,
well placement followed by well control optimization
(WP&C), and simultaneous well placement and control
(SWPC), respectively. The corresponding NPV values
in each case are reported in the caption of Fig. 6 and
show an increase of 16 % for the SWPC case over the
best case obtained without simultaneous optimization

Fig. 6 Optimization results

with the PUNQ model
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5
(a) Log-perm

(known reservoir): a true

log-permeability, b true
porosity, final saturation
profiles corresponding to the
c initial (NPV = 1.864 × 107 ),
d well placement
optimization (WP,
NPV = 2.489 × 107 ), e well
placement followed by well
control optimization (WP&C,
(b) Porosity

NPV = 2.510 × 107 ), and

f simultaneous well
placement and control
optimization (SWPC,
NPV = 2.898 × 107 ). In c–f,
locations of the injection and
production wells are
identified with circles and
crosses, respectively
(c) Initial
(d) WP
(e) WP&C
(f) SWPC
182 Comput Geosci (2013) 17:167–188

(WP&C), that is, optimizing the well locations with fix
rates followed by optimizing the controls (rates) for
the best well locations obtained. The improvements are
also visible by comparing the saturation plots. Next,
we consider the optimization problem under geologic
uncertainty.
4.2 Field development optimization under geologic
uncertainty
Example 4: simple 2D problem with uncertainty In
this section, we consider the well placement optimiza-
tion problem under geologic uncertainty by assuming
multiple plausible permeability models. We use Nens =
100 permeability models to represent the uncertainty
in the geologic model. Figure 7a shows five sample
log-permeability models that exhibit distinct spatial
variability. The results of optimizing well rates and
locations for three injectors and two producers when
the true model is not known are shown in Fig. 7b, c.
The goal is to evaluate the performance of the SPSA
algorithm when robust optimization is implemented to
assess the validity of the proposed approximation in
computing the SPSA cost function using a limited num-
ber of random model realizations (at each iteration) for
computational efficiency.
Figure 7a displays the objective function behavior
during the optimization iterations for different initial
well configurations and using Napprox = 1 (black), 5

Fig. 7 Optimization results (a)

for the 2D synthetic model Sample 1 Sample 2 Sample 3 Sample 4 Sample 5
with uncertain reservoir
permeability model.
a Five-sample
log-permeability realizations.
b Expected NPV values over
the complete ensemble of
models with iteration for the (b) WP WP&C SWPC
well placement only (left),
E[NPV}(×109)

well placement followed by

control (center), and
simultaneous well placement
and control. The results are
shown for the case where 1
(black curve), 5 (blue line),
and 100 (red curve) models Iteration Iteration Iteration
are randomly selected to
approximate the expected (c) WP WP&C SWPC
value of the objective
function. c Optimized
expected value of saturation
y-index

profiles for well placement

(WP, left), well placement
followed by well control
optimization (WP&C, center),
and simultaneous well
placement and control
optimization (SWPC, right);
the results are shown for the
y-index

case where 1 (top row), 5

(middle row), and 100
(bottom row) models are
randomly selected at each
iteration to approximate the
NPV objective function. In c,
locations of the injection and
y-index

production wells are

identified with circles and
crosses, respectively

x-index x-index x-index

Comput Geosci (2013) 17:167–188 183

Table 2 Mean an variance of the NPV values for Example 4

each iteration, only one realization is randomly selected
Napprox WP WP&C SWPC to compute the objective function (the black curve
(μ, σ ) (μ, σ ) (μ, σ ) in Fig. 7b). This behavior is absent when the objec-
1 (1.8247, 0.0254) (1.8512, 0.0857) (2.2039, 0.1835) tive function is approximated using a modest value
5 (1.8250, 0.0220) (1.8561, 0.0542) (2.2252, 0.0942)
of Napprox = 5 randomly selected realizations at each
100 (1.8331, 0.0105) (1.8673, 0.0432) (2.2373, 0.0878)
iteration. The numerical NPV results for this case are
summarized in Table 2. This table reports the mean
and variance of the NPV values that are obtained by
(blue), and 100 (red). Interestingly, the final NPV val- applying the solutions to all 100 realizations in each
ues are very similar even though the objective functions optimization case. It is important to note that in all op-
and final well configurations are different. Also, notice timization scenarios, the final NPV solutions obtained
the noisy behavior of the objective function when, at for different Napprox are quite similar. Furthermore, the

Fig. 8 a Five-sample (out of (a) Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

100) realizations of
permeability for the
PUNQ-S3 model for
Sample 1

optimization under geologic

uncertainty. b Five-sample
(out of 100) realizations of
porosity for the PUNQ-S3
model for optimization under
geologic uncertainty
Sample 2
Sample 3
Sample 4
Sample 5
184
Fig. 8 (continued) (b) Layer 1 Layer 2
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
proposed simultaneous optimization approach achieves
a significant increase in the NPV value over the decou-
pled optimization cases (well placement and produc-
tion optimization). The improvement is quite evident
from the saturation plots.
These results suggest that the proposed approxi-
mation in calculating the cost function of the robust
optimization can be quite effective and provide sig-
nificant computational saving with a minimal impact
on the solution quality. A more realistic scenario is
the case in which the synthetic true model is available
(but unknown) and the optimization solution from the
uncertain models is evaluated by applying them to the
Comput Geosci (2013) 17:167–188
Layer 3 Layer 4 Layer 5
true model. We consider this case using the PUNQ-S3
model next.
Example 5: 3D PUNQ-S3 model with uncertainty Us-
ing the reservoir configuration in the PUNQ-S3 model,
we conducted a similar experiment to investigate the
effect of geologic uncertainty and evaluate the perfor-
mance of the proposed cost function approximation
method in a more realistic scenario. We considered
100 model realizations each with distinct permeabil-
ity and porosity maps (five sample permeability and
porosity models are shown in Fig. 8a, b, respectively).
The true permeability and porosity models (shown in
Comput Geosci (2013) 17:167–188 185

Fig. 9 a Optimization results (a) Convergence of the expected NPV value with iterations
for the PUNQ-S3 model with
uncertain reservoir Well placement Production optimization Combined optimization
permeability models; the
plots show the expected NPV

E[NPV}(×109)
values with iteration for the
well placement only (left),
well placement followed by
control (center), and
simultaneous well placement
and control. The results are
shown for the case where 1 Iteration Iteration Iteration
(black curve), 5 (blue line),
and 100 (red curve) models (b) Realized final saturation profiles with uncertain geology
are randomly selected to
approximate the expected Layer 1 Layer 2 Layer 3 Layer 4 Layer 5
value of the objective
function. b Optimized
expected value of saturation
WP

profiles when all 100 models

are used to approximate the
objective function at each
iteration; the results are
shown for well placement
(WP, top row), well
placement followed by well
control optimization (WP&C,
middle row), and
WP&C

simultaneous well placement

and control optimization
(SWPC, bottom row).
Locations of the injection and
production wells are
identified with circles and
crosses, respectively
SWPC

Fig. 6a, b, respectively) are not included in the 100
realizations used in this experiment. Figure 9a shows
the behavior of the objective function in different
optimization cases for this example. The results are
shown for Napprox = 1 (black), 5 (blue), and 100 (red)
and for WP alone (left), WP&C (middle), and SWPC
(right). The final saturation plots for Napprox = 100
are shown in Fig. 9b. These results are consistent
with the previous case in that the SWPC shows sig-
nificantly higher NPV solutions than the two optimiza-
tion methods where simultaneous optimization is not
used. From Fig. 9a, it is evident that the final NPV
solutions for different values of Napprox are compara-
ble, except that the case with Napprox = 1 exhibits a
noisy behavior, which is expected. Figure 10a, b shows
the final saturation plots for the case with Napprox = 1
and Napprox = 5, respectively. The improvement in the
saturation results is consistent with the obtained NPV
values.
The numerical NPV results for this case are sum-
marized in Table 3, which reports the mean and vari-
ance of the NPV values that are obtained by applying
the solutions to all 100 realizations in each case. Also
reported is the NPV value obtained by applying the
final solution for each method to the true PUNQ-S3
reservoir. These latter NPV values are the real values
that will be realized in the field after applying the so-
lutions as development strategies to the true reservoir.
The results clearly show pronounced improvement in
the field development NPV values despite the fact the
true model is unknown and only represented through
plausible but distinct realizations.
186 Comput Geosci (2013) 17:167–188

Fig. 10 Optimized expected

value of saturation profiles
when 1 (a) and 5 (b)
randomly selected models are
used to approximate the
objective function at each
iteration; the results are
shown for well placement
(WP, top row), well
placement followed by well
control optimization (WP&C,
middle row), and
simultaneous well placement
and control optimization
(SWPC, bottom row).
Locations of the injection and
production wells are
identified with circles and
crosses, respectively

Table 3 Mean an variance Napprox WP WP&C SWPC

of the NPV values for
Example 5 (μ, σ ) True (μ, σ ) True (μ, σ ) True
1 (2.4773, 0.0843) 2.4802 (2.5524, 0.0857) 2.5500 (2.8947, 0.1024) 2.8712
5 (2.4857, 0.0831) 2.4813 (2.5532, 0.0842) 2.5612 (2.9064, 0.0938) 2.8846
100 (2.4871, 0.0721) 2.4806 (2.5639, 0.0871) 2.5592 (2.9231, 0.0778) 2.8734
Comput Geosci (2013) 17:167–188
5 Conclusion
We presented a generalized SPSA algorithm for si-
multaneously finding optimal well locations and the
dynamic control trajectories in a reservoir. Recog-
nizing that with fixed well controls (i.e., flow rates)
the solution to well placement optimization problem
is suboptimal and that with fixed well locations only
suboptimal well controls can be found, we propose a
generalized version of the SPSA algorithm for solving
the coupled well placement and dynamic rate allocation
optimization. Furthermore, exploiting the robustness
of the SPSA to noisy cost functions, we introduced
an efficient approximate solution to the robust field
development optimization under geologic uncertainty
where instead of calculating the expected value of
NPV from a large number of plausible models, we
approximate this cost function value using a limited
number of randomly selected models at each iteration.
This step introduces noise (error) in calculating the
objective function, which can be tolerated in SPSA. The
reduction in the sample size may result in an overall
significant computational efficiency, as was the case in
the examples considered in this paper. We used sev-
eral numerical experiments, including examples with
the SPE10 and PUNQ-S3 models, to study the perfor-
mance of the proposed methods for well placement as
well as the combined well placement and rate alloca-
tion optimization under known and uncertain geologic
scenarios.
These results show that solving the well placement
and control optimization problem simultaneously can
significantly improve the field development strategies.
We also studied the performance of the method under
geologic uncertainty, which significantly complicates
field development activities. Our results suggest that,
even under geologic uncertainty, the joint optimization
of well locations and rate allocations is likely to
improve the expected NPV values (over the entire
ensemble of models) obtained from the solutions of
well placement or well control optimizations alone.
Moreover, our initial results suggest that the proposed
approximation for robust optimization with the SPSA
can result in marked improvement in computational
efficiency without compromising the quality of the so-
lution. This is an important and interesting observation
that requires further investigation and verification.
We consider our approach as an initial attempt to
formulate and solve a more general field develop-
ment optimization problem by combing well place-
ment and control optimizations. In a more general-
ized formulation, the number of wells, their location,
187
drilling schedule, and total simulation time could also
be considered as unknown decision variables. As is
the case with any realistic subsurface investigation,
the problem is further complicated by the presence
of uncertainty in physical property descriptions. While
solving the field development optimization problem
in its most general form (with realistic assumptions
and constraints) can be quite challenging and is likely
to lead to several alternative solutions with similar
performance, it is important to recognize the sub-
optimality introduced by fixing some of the decision
variables a priori. Clearly, generalization of the field
development formulation leads to complex optimiza-
tion problems that require more sophisticated solution
algorithms to tackle. However, given the significance
and complexity of this problem, further research is
needed to rigorously formulate and solve more gener-
alized and challenging field development optimization
problems.
References
1. Aziz, K., Settari, A.: Petroleum Reservoir Simulation. Ap-
plied Science, London (1979)
2. Bangerth, W., Klie, H., Wheeler, M.F.: On optimization algo-
rithms for the reservoir oil well placement problem. Comput.
Geosci. 10, 303–319 (2006)
3. Becker, B.L., Song, X.: Field development planning using
simulated annealing-optimal economic well scheduling and
placement. In: SPE Annual Technical Conference and Exhi-
bition, SPE 30650, Dallas, Texas (1995)
4. Brouwer, D.R., Jansen, J.D.: Dynamic optimization of water
flooding with smart wells using optimal control theory. SPE
J. 9(4), 391–402 (2004)
5. Chen, H.F., Duncan, T.E., Pasik-Duncan, B.: A Kiefer–
Wolfowitz algorithm with randomized differences. IEEE
Trans. Automat. Control 44(3), 442–453 (1999)
6. Gerencsér, L., Hill, S.D., Vágó, Z.: Optimization over dis-
crete sets via SPSA. In: Proceedings of the 38th Confer-
ence on Decision and Control, Phoenix, AZ, pp. 1791–1795
(1999)
7. Gerencsér, L., Hill, S.D., Vágó, Z.: Discrete optimization via
SPSA. In: Proceedings of the American Control Conference,
Arlington, VA, pp. 1503–1504 (2001)
8. Guyaguler, B., Horne, R.N.: Uncertainty assessment of
well placement optimization. In: SPE Annual Technical
Conference and Exhibition, SPE 71625, New Orleans, LA
(2001)
9. Handels, M., Zandvliet, M.J., Brouwer, D.R., Jansen, J.D.:
Adjoint-based well placement optimization under produc-
tion constraints. In: SPE Paper 105797 Presented at the
SPE Reservoir Simulation Symposium, The Woodlands, TX
(2007)
10. He, Y., Fu, M.C., Steven, I.M.: Convergence of simultaneous
perturbation stochastic approximation for nondifferentiable
optimization. IEEE Trans. Automat. Control 48(8), 1459–
1463 (2003)
188
11. Hill, S.D.: Discrete stochastic approximation with application
to resource allocation. Johns Hopkins APL Technical Digest
26, 15–21 (2005)
12. Juditsky, A., Lan, G., Nemirovski, A., Shapiro, A.: Robust
stochastic approximation approach to stochastic program-
ming. SIAM J. Optim. 19, 1574–1609 (2009)
13. Li, L., Jafarpour, B.: A variable-control well placement op-
timization for improved reservoir development. Comput.
Geosci. 16(4), 871–889 (2012)
14. Maryak, J.L., Chin, D.C.: Global random optimization by
simultaneous perturbation stochastic approximation. IEEE
Trans. Automat. Control 53, 780–783 (2008)
15. Montes, G., Bartolome, P.: The use of genetic algorithm
in well placement optimization. In: SPE Paper 69439 Pre-
sented in the SPE Latin American and Caribbean Petro-
leum Engineering Conference, Buenos Aires, Argentina
(2001)
16. Nesterov, Y.: Efficiency of coordinate descent methods on
huge-scale optimization problems. CORE Discussion Papers
2010002, Université catholique de Louvain, Center for Oper-
ations Research and Econometrics (CORE) (2010)
17. Nocedal, J., Wright S.: Numerical Optimization, 2nd edn.
Springer, New York (2006)
18. Sarma, P., Chen, W.H.: Efficient well placement optimization
with gradient-based algorithm and adjoint models. In: Pro-
ceedings of the 2008 SPE Intelligent Energy Conference and
Exhibition, SPE–112257 (2008)
Comput Geosci (2013) 17:167–188
19. Spall, J.C.: Multivariate stochastic approximation using a
simultaneous perturbation gradient approximation. IEEE
Trans. Autom. Control 37, 332–341 (1992)
20. Spall, J.C.: Implementation of the simultaneous perturbation
algorithm for stochastic optimization. IEEE Trans. Aerosp.
Electron. Syst. 34, 817–823 (1998)
21. Spall, J.C.: Adaptive stochastic approximation by the simul-
taneous perturbation method. IEEE Trans. Autom. Control
45, 1839–853 (2000)
22. Spall, J.C.: Introduction to Stochastic Search and Optimiza-
tion: Estimation, Simulation and Control. Wiley, Hoboken
(2003)
23. Spall, J.C., Hill, S.D., Stark, D.R.: Theoretical framework
for comparing several stochastic optimization approaches.
In: Calafiore, G., Dabbene, F. (eds.) Probabilistic and Ran-
domized Methods for Design Under Uncertainty, chap. 3.
Springer, Berlin (2006)
24. van Essen, G.M., Zandvliet, M.J., Van den Hof, P.M.J.,
Bosgra, O.H., Jansen, J.D.: Robust waterflooding optimiza-
tion of multiple geological scenarios, SPE-102913-PA. SPE J.
14(1), 202–210 (2009)
25. Wang, C. Li, G., Reynolds, A.C.: Optimal well placement for
production optimization. In: SPE Paper 111154 Presented at
the SPE Eastern Regional Meeting, Lexington, KY (2007)
26. Yeten, B., Durlofsky, L.J., Aziz, K.: Optimization of noncon-
ventional well type, location, and trajectory, SPE 86880. SPE
J. 8(3), 200–210 (2003)