TYPE Methods
PUBLISHED 16 November 2022
OPEN ACCESS
EDITED BY
Luis Puigjaner,
Universitat Politecnica de Catalunya,
Spain
REVIEWED BY
Bicheng Yan,
King Abdullah University of Science and
Technology, Saudi Arabia
Hari Ganesh,
Indian Institute of Technology
Gandhinagar, India
Hui Zhao,
Yangtze University, China
*CORRESPONDENCE
Shuoliang Wang,
wangshuoliang@cugb.edu.cn
SPECIALTY SECTION
This article was submitted to Process
and Energy Systems Engineering,
a section of the journal
Frontiers in Energy Research
RECEIVED 28 July 2022
ACCEPTED 27 October 2022
PUBLISHED 16 November 2022
CITATION
Li C, Fang C, Huang Y, Zuo H, Zhang Z
and Wang S (2022), Infill well placement
optimization for secondary
development of waterflooding oilfields
with SPSA algorithm.
Front. Energy Res. 10:1005749.
doi: 10.3389/fenrg.2022.1005749
COPYRIGHT
© 2022 Li, Fang, Huang, Zuo, Zhang and
Wang. This is an open-access article
distributed under the terms of the
Creative Commons Attribution License
(CC BY). The use, distribution or
reproduction in other forums is
permitted, provided the original
author(s) and the copyright owner(s) are
credited and that the original
publication in this journal is cited, in
accordance with accepted academic
practice. No use, distribution or
reproduction is permitted which does
not comply with these terms.
Frontiers in Energy Research
DOI 10.3389/fenrg.2022.1005749
Infill well placement optimization
for secondary development of
waterflooding oilfields with SPSA
algorithm
Congcong Li 1, Chaoqiang Fang 2, Yougen Huang 3, Hailong Zuo 3,
Zhang Zhang 1 and Shuoliang Wang 1*
1
School of Energy, China University of Geosciences, Beijing, China, 2CNPC Logging Company Limited
Geological Research Institute, Beijing, China, 3CNPC Exploration and Development Research Institute
of Changqing Oilfield Branch, Xi’an, China
A significant amount of bypassed oil resources often remain in a mature
waterflooding reservoir because of non-uniform sweep caused by natural
complexities of a subsurface reservoir and improper management of the
reservoir. Infill drilling is one of the most attractive options for increasing oil
recovery in consequence of its operational simplicity, low risk and promising
results. Determining optimal infill well placements in heterogeneous mature
reservoirs is a critical and challenging task that has a significant impact on the
recovery performance and economic revenue of subsurface remaining oil resources.
An integrated framework is constructed to attain best-obtained optimal location and
completion of infill wells in multi-layer mature oil reservoirs. The placement of an infill
vertical well is parameterized in terms of two sets of variables that define the location
and completion respectively. A variant of SPSA algorithm is used to solve the defined
optimization problem. The performance of the proposed algorithm is first tested for
the joint optimization of well location and completion of an injection well using a
synthetic model. The results show that the algorithm with average SPSA gradients
outperforms the single SPSA gradient method both in solution and convergence rate.
Besides, there are two plateaus on the performance curve of all algorithms: on the
first plateau, each algorithm is approaching to its optimal well location with relatively
little change on the completion parameters, while on the second plateau, each
algorithm obtains the corresponding optimal completions. A complex
heterogeneous reservoir model is then constructed by using the data of a mature
oil reservoir in Shengli Oilfield in China to design an optimal 10 years’ infill drilling
program. Four vertical production wells are placed in the oil-rich regions and both
simultaneous and sequential algorithms are tried to obtain their optimal locations and
completions. The performances of simultaneous joint optimization and sequential
joint optimization are compared and as a result it is recommended to use sequential
joint optimization as the optimization algorithm in the integrated framework.
KEYWORDS
mature oilfields, secondary development, infill wells, well placement optimization,
simultaneous perturbation stochastic approximation (SPSA)
01 frontiersin.org
Li et al.
1 Introduction
About 92% of oil reservoirs in China are characterized by
continental clastic sediments with complex vertical and lateral
heterogeneities and high levels of crude oil viscosity (Han, 2010).
Waterflooding with uniformly spaced well pattern has always
been the main development approach for Chinese oilfields. After
decades of producing, most of the major oilfields have entered the
late stage of waterflooding life cycle featured by high water-cut
(greater than 80%) and high recovery of recoverable reserves
(greater than 70%). Although most of mature oilfields are
encountering such terrible situations, they still contribute to a
great proportion of crude oil supply in China. According to
official statistics, more than 70% of the total national oil
production are from mature oilfields at present (Han, 2010).
Production practice shows that there still is a great potential to
improve the recovery of high water-cut oilfields. The decreasing
trend in discovering new conventional hydrocarbon resources is
necessitating optimal recovery from existing reserves in a more
cost-effective manner.
Due to strong reservoir heterogeneities, the remaining oil
distribution generally presents a feature of “highly scattered on
the whole and relatively abundant in local areas” when the
comprehensive water-cut in oilfields is beyond 80% (Han,
2007). The two major methods to improve water sweep
efficiency that were once applied effectively at lower water-cut
stage, including subdivision of development intervals and
uniform infilling wells, began to show little or even no effects.
This is mainly because the achievements of the above methods at
lower water-cut stage of development are owed to the large-area
distributions of remaining oil in relatively low-to-moderate
permeability layers at that time, but the current condition
becomes very different in that the overall remaining oil is
highly scattered even in low-to-moderate permeability layers.
In this case, uniform infilling wells are not as efficient as before.
The initial water-cut in most newly drilled infilling wells can be as
high as 80% and few even higher than 90% (Han, 2010). In order
to further improve the recovery of remaining oil reserves, it needs
to drill individualized infill wells rather than uniform infill wells.
Determining appropriate well placements of producers and
injectors is an important aspect of overall development strategy
of any field. Well placement optimization becomes particularly
noteworthy in mature oilfields where an enormous amount of
pore volumes are occupied by water and both reservoir geological
characteristic and remaining oil distribution are extremely
complex. Thus, new wells have to be drilled based on a
comprehensive understanding of reservoir description. In
Chinese mature oilfields, each wellbore may penetrate many
prospective reservoir layers that can range, depending on the
field size and number of reservoirs, from a few into the hundreds.
However, technical and reservoir management considerations
usually make every wellbore capable of producing from only a
limited completion interval. The region that an infill well to be
Frontiers in Energy Research
10.3389/fenrg.2022.1005749
located in, the reservoir layer to be penetrated and the length of
the perforated or open section of a well will together influence the
development performance greatly. Hence, the main objective of
infill well placement optimization should be optimizing not only
the areal drainage pattern but also the vertical scheme.
Optimizing both well location and perforation along
formation intervals could help maximize the control reserve of
a single well, homogenize the inflow-velocity profile and prevent
early water breakthrough and rapid water-cut increase in high-
permeability completion intervals, which are all critically
important for enhancing oil recovery and improving
economic revenue of mature oilfields. However, there could be
a large number of possible candidate locations for a new well. To
search through and evaluate all the possible locations is not
practically feasible, particularly for high resolution geologic
models consisting of multimillion cells. For large scale field
applications, a practical method is needed to mitigate the
computational burden associated with the possibly large
number of simulation runs.
Traditionally, optimization of well placement has been done
by using quality maps that indicate which regions of the reservoir
have not been adequately swept by the injected water (Badru,
2003; Kharghoria et al., 2003; da Cruz et al., 2004; Guimaraes
et al., 2005; Taware et al., 2012). The quality maps are typically
based on static properties such as permeability, porosity,
structure and net thickness, and dynamic properties such as
remaining oil, pressure, well productivity and cumulative oil
production. Although this method is convenient for application,
it cannot properly place wells in locations that take advantage of
long-term high oil saturation or respond to the dynamics of
reservoir fluid flow over a long period of time. Simply placing
wells based on saturation or quality maps may not guarantee long
term profitability of the project. In recent years, various
simulation-based optimization techniques have also been
introduced to automate the process of attaining the optimal
well locations. There are many challenges in the field of
automatic well placement optimization, such as variable
design, objective function, constraints formulation,
optimization algorithm, geological uncertainty, computational
cost and so on. We will just discuss a few of them for brevity in
this paper.
One important aspect of automatic well placement
approaches is the description of well location and/or well
trajectory. As for well placement problem, the actual physical
problem of determining well location and trajectory is a
continuous problem, while the completions of a well in the
formation are commonly introduced into the numerical
simulation model as integral indices of the connected
gridblocks. Thus, to parameterize the location and, in general
the trajectory of the wells, both discrete variables (Yeten et al.,
2003; Bangerth et al., 2006; Ozdogan and Horne, 2006; Emerick
et al., 2009; Bellout et al., 2012; Li et al., 2013) and continuous
real-valued variables (Onwunalu and Durlofsky, 2010;
02 frontiersin.org
Li et al.
Bouzarkouna et al., 2012; Forouzanfar et al., 2012; Forouzanfar
and Reynolds, 2013; Nwankwor et al., 2013; Forouzanfar et al.,
2016) are used. For a discrete optimization problem, decision
variables are well connection gridblock indices. The center point
of the well is moved from one gridblock to another at each
iteration of whatever optimization algorithm is used. For a
continuous optimization problem, the well trajectory is
generally parameterized in a more real but complex way,
which could make the method uneasy to implement.
Another important aspect of automatic well placement
approaches is the optimization algorithm. To solve the
optimal well placement problem, most researchers have
utilized heuristic global optimization methods such as genetic
algorithm (GA) (Montes et al., 2001; Yeten et al., 2003; Ozdogan
and Horne, 2006; Emerick et al., 2009; Salmachi et al., 2013),
differential evolution (DE) (Afshari et al., 2015; Awotunde,
2016), simulated annealing (Sa) (Beckner and Song, 1995),
particle swarm optimization (PSO) (Onwunalu and Durlofsky,
2010; Bouzarkouna et al., 2013; Afshari et al., 2014; Forouzanfar
et al., 2016), the covariance matrix adaptation-evolution strategy
(CMA-ES) algorithm (Bouzarkouna et al., 2012; Jesmani et al.,
2016a; Forouzanfar et al., 2016) and some hybrid algorithms
(Nwankwor et al., 2013). Although these methods have the ability
to avoid local solutions, their convergence to the global solution
is heuristic in natural and typically require a very large number of
reservoir simulation runs in an optimization loop. Moreover, to
account for model uncertainty, it is often need to evaluate the
performance of well placements over multiple geological
realizations, which makes them too computationally expensive
and thus, may not be well-suited for large scale field applications.
Some local search optimization algorithms were also applied to
the well placement optimization problem such as gradient-based
method algorithms with adjoint gradient (Wang et al., 2007;
Sarma and Chen, 2008; Zandvliet et al., 2008; Vlemmix et al.,
2009; Li and Jafarpour, 2012; Forouzanfar and Reynolds, 2014),
methods on the basis of the simultaneous perturbation stochastic
approximation (SPSA) (Bangerth et al., 2006; Li et al., 2013;
Jesmani et al., 2016b), derivative-free methods (Forouzanfar
et al., 2012; Forouzanfar and Reynolds, 2013; Isebor et al.,
2014a; Isebor et al., 2014b) and generalized pattern search
(GPS) (Bellout et al., 2012). Among these local search
algorithms, gradient-based method has the superior
computational efficiency. However, one must calculate the
gradient of the objective function and the constraints with
respect to the optimization variables to apply the efficient
gradient-based method. The adjoint for computing these
gradients is not commonly provided in commercial reservoir
simulators, which makes the adjoint methods are difficult to
implement and require access to the simulator source code.
An alternative approach to gradient-based method is the
stochastic optimization method. In particular, the simultaneous
perturbation stochastic approximation (SPSA) has been
developed for solving large dimensional multivariate
Frontiers in Energy Research
10.3389/fenrg.2022.1005749
optimization problems where the gradient information is not
available (Spall, 1992; Spall, 1994; Spall, 1998; Spall, 2000).
Various kinds of SPSA algorithms have been successfully
applied in several engineering fields. Early applications of
SPSA algorithm in petroleum engineering field were
performed by Bangerth et al. (Bangerth et al., 2006) and
Gao et al. (Gao et al., 2007). In Bangerth et al. (Bangerth
et al., 2006), a variant of SPSA (integer SPSA) is introduced
to solve the well placement problem. The authors compared
and analyzed the efficiency, effectiveness, and reliability of
several optimization algorithms, including the simultaneous
perturbation stochastic approximation (SPSA), finite
difference gradient (FDG), very fast simulated annealing
(VFSA) and genetic algorithm (GA), for the well
placement problem by a set of numerical waterflooding
experiments and found that none of these algorithms
guarantees to find the optimal solution, but both SPSA
and VFSA are very efficient in finding nearly optimal well
locations with a high probability. In Gao et al. (Gao et al.,
2007), both SPSA and adaptive SPSA are applied to the
history matching problem. The authors presented
modifications for improvement in the convergence
behavior of the SPSA algorithm for history matching and
compared its performance to the steepest descent, gradual
deformation and LBFGS algorithm and concluded that
although the convergence properties of the SPSA
algorithm are not nearly as good as the LBFGS methods,
the SPSA algorithm is not simulator specific and it can be
coupled easily with any commercial reservoir simulator to do
automatic history matching. Since the two primary works,
the SPSA algorithm has been widely used to solve various
optimization problems in petroleum engineering field
(Wang et al., 2009; Li and Reynolds, 2011; Do et al., 2012;
Do and Reynolds, 2013; Li et al., 2013; Jesmani et al., 2016b;
Pouladi et al., 2020).
For simulation-based well placement optimization
approaches, the reliability of the solution depends
seriously on the main parameters defining the geological
model, including formation structure, porosity and
permeability distributions, and fluid contacts. Due to
limited reservoir description knowledge, these parameters
are all uncertain and generally described by some probability
density functions. Management of the effect of geological
uncertainty on well placement optimization is another
research area that is gaining more and more consideration
from researchers (Ozdogan and Horne, 2006; Li and
Reynolds, 2011; Li et al., 2013; Chang et al., 2015; Jesmani
et al., 2016b; Temizel et al., 2018). A common practical
approach for taking into account the uncertainty in well
placement problem is to approximate the probability
distributions with multiple equally probable geological
realizations, and then optimize the expected value of the
objective function over an ensemble of the model
03 frontiersin.org
Li et al. 10.3389/fenrg.2022.1005749

realizations, namely ensemble-based optimization. To

reduce the computation time associated with the
optimization process, a variety of techniques have been
developed to select a relatively small ensemble of model
realizations, as the representatives of all possible
realizations. A fit-for-purpose sampling selection
procedure is preferred, as it in general can guarantee to
capture the underlying uncertainty by using as few samples
as possible.
Although there is a significant advancement on the subject of
well placement optimization in the last decades, few have
been done to consider the effect of perforation scheme in
multiply layers with different reservoir properties and oil
distributions. This work presents a methodology for joint
optimization of infill well location and completion using a
variant of SPSA algorithm. Our optimization algorithm is an
extension of the work by Bangerth et al. (Bangerth et al.,
2006). We follow Bangerth et al. (Bangerth et al., 2006) to
apply a bounding operator and a rounding operator to
handle inequality constraints and the integer problem of
well location variables, respectively. However, additional
treatments to the particular problem presented in this
FIGURE 1
study are also conducted. A parameter scaling handling Schematic of perforations of a vertical well in multiple layers.
process is implemented to eliminate the impact of
different ranges of optimizing variables since we solve a
joint optimization problem. Besides, we use average
the performance of the reservoir under different well
stochastic gradient calculated by multiply random
configurations.
sampling of perturbation vectors to improve the
estimation of the search direction. We also introduce a
simple line search procedure and a blocking step to
guarantee the convergence and stability of the algorithm.
2.1 Optimization problem formulation
The paper is organized as follows: Section 2 presents the basic
2.1.1 Well placement parameterization
components that are used for the solution of the well placement
We consider the placement of a vertical well in a
problem, including the formulations describing the optimization
heterogeneous multi-layer reservoir. The wellbore is assumed
problems and the objective function; Section 3 illustrates the
to have one or several completion/perforation intervals. Figure 1
methodology for solving the combined well location and
is the typical schematic of the location and completions of a
completion optimization problem based on SPSA algorithm
vertical well as a function of well placement variables in multiple
and the framework for coupling the reservoir flow simulator
layers. The placement of well w with nw,c completion intervals
and the optimization algorithm; Section 4 presents the validation
can be parameterized in terms of 2 + 2nw,c continuous variables
of the proposed method by using a synthetic model and its
that are given by
application on optimization of an infilling plan for a mature
T
oilfield in China; Section 5 summarizes the results of the Pw 
xw , yw , zw,1 , lw,1 , zw,2 , lw,2 , . . . , zw,nw,c , lw,nw,c  (1)
presented work and discusses possible directions of further
research. where nw,c is the number of completion intervals of well w;
(xw , yw ) represents the spatial coordinate of the well
location; zw,i and lw,i are the center point and the length
2 Problem description of the ith completion interval. It is noted that there are two
variables representing the well location and 2nw,c variables
The problem considered in this work is to determine the representing the nw,c well completion intervals for a
optimal location and completion of infill wells to maximize oil vertical well.
recovery and profits for mature waterflooding reservoirs in the With regard to numerical simulation, in most commercial
future. Numerical simulations are conducted using a commercial reservoir simulators, the set of well placement variables are often
reservoir simulator (Eclipse E100 (GeoQuest, 2018)) to evaluate treated as integers since the simulators require wells to be

Frontiers in Energy Research 04 frontiersin.org

Li et al.
assigned to discrete grid blocks in the model, namely “connecting
gridblocks (I, J, K)”. This implies that for a given number of infill
wells to be optimized, Ninf ill , the dimension of the set of well
location variables, nw , is determined, that is, nw  2Ninf ill .
However, the dimension of the set of well completion
variables,  2nw,c , is related not only to the number of infill
wells but also to the perforation intervals being optimized of each
well. This makes the optimization problem more complicated to
define and to solve, because the number, the position and the
length of the well completion in multi layers are all the
optimization variables. In order to simply the optimization
problem, we use the “well productivity index (WPI)
multiplier” instead of “connecting gridblocks” in reservoir
simulators (Eclipse E100) as the optimization variables,
because the inflow performance between the wellbore and the
connecting gridblock is calculated by WPI and WPI multiplier
can take into consideration the partially-penetrated length of a
connecting gridblock. We define WPI multiplier as η  Δl/Δz in
this paper, where Δl is the connecting length of a single gridblock
with the well and Δz is the thickness of that gridblock. It is worth
to mention that, to our knowledge, the concept of WPI multiplier
has been applied by Forouzanfar et al. (Forouzanfar et al., 2012)
for a different purpose. In Forouzanfar et al. (Forouzanfar et al.,
2012), the placement of a vertical and horizontal well was
described as four continuous parameters, (xw , yw , zw , lw ) and
WPI multiplier was used to modify well productivity indices in
the reservoir simulator to account for the location of the
centerline of a well that does not have to be at the center of a
gridblock when the optimal well placement is formulated as a
continuous optimization problem. However, the authors did not
consider multi-segment perforated intervals which are
commonly demanded for the redevelopment of mature oilfield
in China.
The infill well is initially assumed to connect with each layer
of the reservoir and then a guess WPI multiplier η is assigned to
each connection, where η  1 means the layer is fully penetrated
and η  0 means the layer is fully shut. By using the WPI
multiplier concept as a replacement of directly using
connecting grid blocks, the placement of well w can be re-
parameterized in terms of 2 + Nz variables that are given by
T
Pw 
Iw , Jw , ηw,1 , ηw,2 , . . . , ηw,Nz  (2)
Here, Nz is the dimension of the reservoir model in
z-direction; (Iw , Jw ) represents the integer grid coordinate of
the well location; ηw,i is the WPI multiplier of the ith connecting
grid block. Thus, the placement of a vertical well is parameterized
with a set of integer and continuous variables. In order to simplify
notation, we define the well location feasible set w 
{w ∈ ZNw ; w l ≤ w ≤ w u } and the well completion variable set c 
{c ∈ RNc ; cl ≤ c ≤ cu }.
Frontiers in Energy Research
10.3389/fenrg.2022.1005749
2.1.2 Joint well placement optimization problem
Based on the parameterization method for well placement,
the problem considered here can be given as the following
formulation:
^ ^c  arg
w, min f(s(m, w, c, t), m, w, c) (3)
w∈ZNw ,c∈RNc
g(s(m, w, c, t), m, w, c)  0 (4)
wl ≤ w ≤ wu (5)
cl ≤ c ≤ cu (6)
where f(·) is a specified objective function; s is the state of the
reservoir at time t; w denotes the discrete well location variables
with w u represents its upper bound and wl the lower bound; c is
the continuous completion variables with cu represents its upper
bound and cl the lower bound; g(s(m, w, c, t), m, w, c) denotes
the discretized multiphase flow equation with m represents the
reservoir model input parameters (e.g. petrophysical and fluid
properties).
2.2 Objective function
In the hydrocarbon production problems, the objective
function can be any user-specified performance measure, such
as cumulative oil production or recovery factor and net present
value (NPV). This paper considers the negative of the reservoir
NPV increment due to infilling wells as the minimization
objective function f(·). In general, the NPV is described as
the summation of present values in a time interval when present
values are calculated based on incoming and outflows cash,
which is calculated by below equation:
Nt
CFn
NPV   tn /365
(7)
n1 (1 + d)
CFn is the cash flow in the nth time interval; Nt is the number of
time intervals for evaluation of the infill project; d the annual
discount rate, tn represents the total production time in days at
the end of the nth time interval.
As for infill project, the outflows of cash include not only
the operational costs of existing wells but also the capital
expenditures associated with new wells drilled in the
reservoir. The capital expenditures consist of the initial
investment for drilling new wells and the need for
additional surface facilities at the start of the infill project.
The operational costs include costs of water treatment and
disposal for production wells and water injection for injection
wells in each time interval. The incoming cash flow is the
profit of oil production from production wells. Thus, the net
present value of the infill project can be further defined as the
following equation:
05 frontiersin.org
Li et al.
Nt Nprod Ninj
Δtn
NPV  ⎡⎢⎣  ro qno,i − rwp qnwp,i −  rwi qnwi,j ⎤⎥⎦ tn /365
n1 i1 j1 (1 + d)
− Niw ciw
(8)
where, ro , rwp and rwi are the oil price, cost of water production
disposal and cost of water injection per unit volume respectively;
qno,i and qnwp,i are the average oil and water production rates of the
ith producer over the nth time interval; qnwi,j is the average water
injection rate of the jth injector over the nth time interval; ciw is
the cost of drilling per well; Nprod and Ninj , respectively, denote
the total number of both existing and infill production wells and
injection wells; Niw is the number of infill wells.
In this formulation, the NPV increment objective function
depends on the oil production rate qo , the water production rate
qwp , the water injection rate qwi and the number of infill wells.
The reservoir simulation model can be used to forecast the
production performance for an infill project with fixed
number and placement of infill wells, and then the revenue of
the project can be estimated using the economic objective
function. Increment of net present value compared with the
value evaluated without infill wells means drilling additional
wells adds value to the current production from mature oil
reservoir and project should be economically acceptable.
3 Optimization methodology
The objective function is minimized using an approximate
gradient method, SPSA, which has been used successfully for
both well placement and well control optimization as discussed
in the Introduction. This section first presents an overview of the
general SPSA algorithm and its important properties. Then,
based on the general SPSA algorithm, some particular
procedures with respect to the problem of well placement
optimization are illustrated. The last of this section presents
an integrated framework that developed to couple the reservoir
simulator with the SPSA algorithm to form an automatic
optimization tool for infill well project in oil reservoir.
3.1 Basic SPSA algorithm
Assume that noisy measurement y(x) of the loss function
f(x) and its gradients are available at any value of x ∈ Rp , which
are respectively denoted as y(x)  f(x) + ε and ∇f(x) with ε
representing the noise term. Considering f(x) as the objective
function and x as the p-dimensional vector of decision variables,
the goal of an optimization algorithm is to find the minimum
solution x* such that g(x*)  zf(x)
zx |xx*  0. By using the method
of stochastic approximation (SA), the optimization variables are
found through the following standard recursive form:
Frontiers in Energy Research
10.3389/fenrg.2022.1005749
xk+1  xk − ak g^k (xk ) (9)
where g^k (xk ) is a stochastic approximation of ∇f(xk ) at the kth
iteration and ak is a nonnegative scalar gain coefficient. The
iterative form in Eq. 9 represents the steepest descent algorithm if
g^k (xk ) is replaced by ∇f(xk ).
The central finite difference method is usually used to
approximate the gradient by perturbing the elements of the
decision variable vector one at a time and evaluating the
objective function for each perturbation, which can be written as
y(xk + ck ei ) − y(xk − ck ei )
g^ki (xk )  (10)
2ck
where ei denotes a unit vector with its ith entry equal to unity and
all other entries equal to zero; ck is a small positive scalar referred
to as the perturbation size.
It is easy to see that 2p function evaluations are required to
approximate the gradient using the finite-difference method,
which makes the method increasingly inefficient for
application as the problem dimension p grows. The SPSA
algorithm provides an efficient alternative by perturbing all
vector elements simultaneously to obtain a stochastic gradient
approximation g^k (xk ) that is defined as
y(xk + ck Δk ) − y(xk − ck Δk ) −1 T
g^k (xk )  ×
Δ−1 −1
k1 Δk2 . . . Δkp 
2ck
(11)
where Δk is a p-dimensional random perturbation vector,
Δk  [Δk1 Δk2 . . . Δkp ]T , which contains independent and
symmetrically distributed entries with finite inverse moment
expectation E[|Δki |−1 ]. A common and simple distribution
that fulfills this condition is the symmetric Bernoulli (±1)
distribution. It is worth mentioning that symmetric uniform
and normal distributions do not have finite inverse moment
expectation, and thus they cannot be used with SPSA.
The basic step of SPSA algorithm is that: 1) in any given
iteration, generate a random perturbation vector Δk for the
decision variables in their search space; 2) run simulation and
evaluate objective functions at the two points, y(xk + ck Δk ) and
y(xk − ck Δk ); 3) compute the approximate gradient using Eq.
23; 4) take a step in the descent direction with an appropriate
value of the gain sequence ak to find the new vector of variables
using Eq. 9. The main attraction of SPSA algorithm lies in its
efficiency on approximate gradient estimation, which is based
on only two function evaluations. This property makes the
algorithm suitable for large-dimensional optimization
problems where gradient information is not available or easy
to compute. Moreover, the approximate nature of gradient
estimation provides robustness to noisy measurement of the
cost function, which is another key property of the SPSA
algorithm. It has been shown that under fairly general
conditions, the method can be superior to other gradient
free optimization methods.
06 frontiersin.org
Li et al.
Although the gradient approximation in Eq. 11 requires less
function evaluations, it is important to consider the number of
iterations required for effective convergence of the algorithm.
The SPSA algorithm converges to a local optimum point under
some conditions on the SPSA parameters (gain sequences),
smoothness of the objective function close to the optimum, and
when the properties regarding the perturbation distribution Δk
discussed above are fulfilled (Jesmani et al., 2016b). The
conditions for the gain sequences are as follows:
∞ ∞
ak > 0, ck > 0, ak → 0, ck → 0, ak  ∞, a2k c2k < ∞
k0 k1
To satisfy the above conditions, Spall (Gao et al., 2007)
provided some useful guidelines to select the perturbation size
ck and gain step length ak for SPSA, which are defined
respectively as follows:
a handling procedure. In this work, a bounding operator and a
ak 
(A + k + 1)β
c 2006) are applied to handle inequality constraints and the integer
ck 
(k + 1)γ
where a, A, c, β, and γ are positive real numbers which should
satisfy the conditions of A ≥ 0, β − 2γ > 0 and 3γ − (β/2) > 0. The
choice of these parameters can have some influence on the
performance of the SPSA algorithm. In Spall (Gao et al.,
2007), the commonly recommended values for β and γ are
0.602 and 0.101, respectively. Moreover, it is suggested to set
A at or around 10% of the maximum number of expected or
allowed iterations k max , that is, A  0.1k max . After specifying A
ag^0 (x0 )
and β, parameter a can be chosen such that a0 g^0 (x0 )  (A+1)
approximately equal to the minimum desired changes of x in the
early iterations. Finally, a suggested rule-of-thumb is to set c at a
level approximately equal to the standard deviation of the noise
in computing the objective function. However, it is inconvenient
to evaluate the magnitude of the standard deviation of the noise.
Do (Do and Reynolds, 2013) presented a simple and feasible
method to estimate the value of c. In his work, he first set the
maximum number of allowable iterations k max and let the
minimum allowable change in perturbation size be denoted by
c min , then calculated c from c min  c/(k max + 1)γ . In the two
examples presented later, we applied this recommendation in our
study to estimate the stochastic gradient.
3.2 Well placement with SPSA
In its basic form as outlined above, SPSA can only operate on
unbounded continuous sets, and is thus unsuited for
optimization on our bounded problem. Therefore, in order to
enhance the applicability of the SPSA algorithm to solve the
problem of well placement optimization, the basic SPSA was
modified in this study. Additional treatments include a
Frontiers in Energy Research
10.3389/fenrg.2022.1005749
parameter scaling and constraint handling process, a simple
line search procedure and a blocking step.
Firstly, because the well placement and completion
optimization formulation is a constrained optimization
problem and the elements of optimization variable set x has
very different magnitudes, the variables are scaled using a linear
transformation with their upper and lower bounds. For the ith
variable xi , we define the transformed new variable ui such that
xi − xm,i
ui  f ori  1, 2, L, Nx (15)
xr,i
(12)
where xm,i  xu,i2+xl,i and xr,i  xu,i2−xl,i with xl,i and xu,i denoting
the lower and upper bounds for the ith control variables. When
xi → xu,i , ui → 1 and when xi → xl,i , ui → − 1. With this
transformation, the optimization algorithm may simply works
within [−1, 1]. However, since SPSA is an unconstrained
optimization algorithm, it needs to incorporate a constraint
(13)
rounding operator suggested by Bangerth et al. (Bangerth et al.,
(14)
problem of well location variables, respectively.
In addition, as the gradients are stochastic and approximated,
the descent direction might not have sufficient accuracy and the
magnitudes of the gradient may change significantly for each new
generation. To avoid this problem and guarantee an appropriate
step size that minimizes the objective function, we also
implement the averaged stochastic gradient as a search
direction, i.e.
1 g
N
gk (uk )  g^ (uk ) (16)
β is Ng n1 kn
where each g^kn (uk ) is obtained from Eq. 11 using a different
sample of Δk . Averaged stochastic gradient has been tried by
Wang, et al. (Wang et al., 2009) to use to solve production
optimization problem in closed-loop reservoir management, and
the results were proven to be more reasonable than single
stochastic gradient. We adopt averaged stochastic gradient in
this study to see if it works as well.
When the optimization is start, a trial step size ak is used to
update the control vector uk+1 . In order to have a better idea of
how to choose an initial value of ak so that we can potentially vary
the components of uk+1 over its expected range, we normalized
the search direction using the infinity norm of the gradient. With
such a normalization, the recurrence formula is given by
g (uk )
uk+1  uk − ak  k  (17)
gk (uk )∞
Here,  · ∞ denotes the infinity norm. If the objective function
does not decrease with the trial step size, ak is cut by half until
f(uk+1 ) is less than f(uk ). If the objective function at the
intended value uk+1 does not show reduction relative to the
value uk after several line search steps, we simply reject this step,
07 frontiersin.org
Li et al. 10.3389/fenrg.2022.1005749

TABLE 1 Reservoir model parameters used in the synthetic model.

Reservoir parameters for simulation

Phases Three-phase (o/g/w)

Reservoir top depth 9030 ft
Reservoir dimensions 1250 ft × 1250 ft × 40 ft
Cell dimensions 25 × 25×4
Horizontal permeability 50, 200, 50, 50 mD
Vertical permeability 0.5 mD
Porosity 0.2
Reference depth 9035 ft
Pressure at reference depth 3600psi
Depth of water-oil contact 9950 ft
Numerical of production wells 4
Numerical of injection wells 1

3.3 Integration of reservoir simulation with

the SPSA algorithm

The optimization algorithm presented above is integrated

with the reservoir numerical simulation software Eclipse
E100 using Matlab. Figure 2 is the flow chart demonstrating
the required steps needed to be taken to run the optimization. For
fix number and type of infill wells, well locations are chosen
initially using quality map such as reservoir permeability, well
productivity, remaining oil, etc. by engineers. Then oil and water
flow are calculated using the reservoir flow simulator for the
period of the infill program. Total oil and water production and
water injection is simulated monthly and imported into Eq. 8 to
calculate the net present value for the infill program. This value
and the corresponding infill well locations are stored in a data file
at each iteration step.
The next infill well locations are selected by the SPSA
algorithm and assigned to new grids in the reservoir. Oil and
water production and water injection are again calculated using
the flow simulator and another net present value is calculated for
new placements. This continuous procedure is an automatic-
intelligent approach. The coupling of reservoir simulator and
Matlab enables us to repeat this procedure automatically for
numerous wells arrangements across the reservoir while the
FIGURE 2 program intelligently produces new allocations.
Flowchart of the optimization framework.

4 Results and discussion

generate a new set of perturbation vectors Δk , and repeat the
above process, which is referred to as a blocking step. We specify
a maximum allowable number of iterations (or function
evaluations) and a maximum allowable number of blocking
steps after a successful iteration as the stopping criteria. The
implementation of the modified SPSA is shown in Algorithm 1.
The integrated framework was first validated with the
standard five spot well placements in a synthetic reservoir
with lateral homogeneous permeability. Then computationally
expensive simulations were performed to attain the best
distributions for infill wells intended to enhance development
of a mature oil reservoir in China.

Frontiers in Energy Research 08 frontiersin.org