Production Optimization in Closed-Loop
Reservoir Management
Chunhong Wang, SPE, PetroTel; and Gaoming Li, SPE, and Albert C. Reynolds, SPE, University of Tulsa
Summary
In closed-loop reservoir management, one periodically updates
reservoir models by integrating production data and then solves
an optimal control problem to determine optimum operating con-
ditions to maximize hydrocarbon production or net present value
(NPV) for the remaining expected life of the reservoir. The cycle of
model updating and production optimization is repeated at speci-
fied times. Here, to account for geological uncertainty, we suggest
using the ensemble Kalman filter for reservoir model updating and
consider three different algorithms for production optimization.
Two simple but representative examples indicate that the steepest
ascent algorithm is the best of the optimization methods. If the
required adjoint software for calculating the gradient of NPV with
respect to the controls is not available, we show that iteration using
an easily computed stochastic gradient can yield a good estimate of
the optimal NPV if properly implemented. For the problem consid-
ered, it is shown that NPV is a nonlinear function of the controls,
but the final controls from the cases with both known true geology
and uncertain geology show “bang-bang” behavior.
Introduction
In recent years, the concept of closed-loop reservoir management
has attracted intensive research interest (Brouwer et al. 2004;
Jansen et al. 2005; Sarma et al. 2006). This approach enables one
to adjust the reservoir production control parameters to optimize
the reservoir production performance with geological uncertainty
while assimilating dynamic production data in real-time. There
are two optimization steps in the approach: The first step is the
dynamic data assimilation (history matching), and the second step
is to optimize the reservoir performance by adjusting the well
controls based on the history-matched reservoir models. Studies
in the literature have been focusing on one of the steps (Zakirov
et al. 1996; Brouwer and Jansen 2004; Sarma et al. 2005, 2006)
and only a few researchers have investigated the conjunction of
the two (Brouwer et al. 2004; Sarma et al. 2006).
For data integration problems of interest in reservoir modeling
and characterization, Bayesian statistics provides a convenient
framework for characterizing and evaluating uncertainty. The
method introduced into reservoir characterization by Oliver et al.
(1996) and also considered briefly by Kitanidis (1995), which is
now most commonly referred to as the randomized maximum
likelihood (RML) method, has frequently been used to generate an
approximate sampling of the probability density function (pdf) for
a reservoir model conditional to production or seismic data (Zhang
and Reynolds 2002; Zhang et al. 2005; Dong and Oliver 2005).
However, this method often takes the computational equivalent of
50 to 100 reservoir simulation runs to generate a single plausible
reservoir model, and its implementation requires efficient adjoint
code. The implementation of the adjoint method is not a trivial task,
and it appears that an optimal implementation of the adjoint can
only be done with detailed knowledge of the reservoir simulator.
In contrast, the ensemble Kalman filter (EnKF) method requires
only one reservoir simulation run per ensemble member. The EnKF
was proposed by Evensen (1994) in the context of ocean dynamics
Copyright © 2009 Society of Petroleum Engineers
This paper (SPE 109805) was accepted for presentation at the SPE Annual Technical
Conference and Exhibition, Anaheim, California, 11–14 November 2007, and revised for
publication. Original manuscript received for review 2 August 2007. Revised manuscript
received for review 16 May 2008. Paper peer approved 19 May 2008.
506
literature as a Monte Carlo approximation of the Kalman filter.
Since its introduction into the petroleum engineering literature
(Nævdal et al. 2002, 2005), it has been used by many researchers
for assimilating production and seismic data to update reservoir
variables, including gridblock rock properties (Gao et al. 2006;
Zafari and Reynolds 2007), boundaries between facies (Zhao et al.
2008), and initial fluid contacts (Thulin et al. 2007).
Although production optimization can be applied to any time
of the reservoir life, most of the studies in this area focus on opti-
mizing the reservoir performance under waterflooding (Brouwer
and Jansen 2004; Jansen et al. 2005; Sarma et al. 2005, 2006;
Alhuthal et al. 2006). One of the reasons for this trend is because
waterflooding is by far the most commonly used method to enhance
oil recovery after primary depletion. The efficiency of waterflood-
ing relies on sweep efficiency, which is determined by the flow
of water along streamlines. With fixed operating conditions, the
streamlines are effectively determined by reservoir heterogeneities
and, in particular, are highly influenced by the connectivity of high
and low permeabilities, which can provide conduits or barriers to
flow, respectively. Flow lines, however, also depend on the operat-
ing conditions at wells and Brouwer and Jansen (2004) provide a
smart well 2D example with one horizontal water injection well and
one horizontal producer where NPV is optimized with injection and
production by keeping one of two high-permeability connections
between injector and producer open to injection and production
throughout the assumed reservoir life cycle. Asheim (1998) inves-
tigated the optimization of the NPV of waterflooding with multiple
vertical injectors and a vertical producer by rate allocation on the
basis of permeability-thickness product. Brouwer et al. (2001)
studied static waterflooding optimization, in which they kept the
inflow control valves constant during the displacement process until
water breakthrough. Later, Brouwer and Jansen (2004) explored
the dynamic waterflooding optimization. The gradient calculated
with the adjoint method was used to dynamically optimize the
production performance with optimal control theory in a horizontal
injector/producer system. In the paper, they consider the simple
constraint where the total field injection is equal to the total field
production. Sarma et al. (2006) studied production optimization
with the adjoint gradient under nonlinear constraints. This paper
compares different existing methods for nonlinear path constraints
and focuses on the approximate feasible direction algorithm, which
lumps all the nonlinear path constraints into one equation and,
hence, requires only one adjoint for the constraint part. The imple-
mentation of the adjoint method requires detailed knowledge of the
reservoir simulator. To overcome this disadvantage, Lorentzen et al.
(2006) proposed to use the EnKF as an optimization algorithm.
Nwaozo (2006) extended the concept and used an average gradient
from the ensemble of realizations. However, they did not consider
constraints in this optimization process.
Sudaryanto and Yortsos (2000, 2001) suggest that the optimal
solution of waterflooding problems is a bang-bang control (i.e.,
each component of the control vector takes either its minimum or
maximum allowed values). Zandvliet et al. (2006, 2007) investi-
gated why and under what conditions waterflooding problems have
optimal solutions at bang-bang control. They derived the sufficient
and necessary conditions for bang-bang control optimal solutions
and concluded that the waterflooding problems with simple upper
and lower bound constraints where valve settings are the controls
sometimes have bang-bang optimal solutions, while problems with
other general inequality or equality constraints where rates are the
controls will have a smooth optimal solution.
September 2009 SPE Journal
 This work focuses on production optimization under bound
constraints on the controls while the geological description is
uncertain. The EnKF is implemented to update the reservoir
geological models as data become available during production.
This model updating step will typically reduce the geological
uncertainty. On the basis of the updated geological model with
reduced uncertainty, the well controls are optimized to maximize
NPV. Three different optimization algorithms—EnKF, simultane-
ous perturbation stochastic approximation (SPSA), and steepest
ascent method—are considered for this purpose. In this study, we
consider bottomhole pressure (BHP) as well controls with only
upper and lower bound constraints.
Problem Formulation
For production optimization problems, NPV is generally used as an
objective (cost) function. The well controls are adjusted to maxi-
mize this objective function (NPV). Following Brouwer and Jansen
(2004) and Sarma et al. (2005, 2006), the NPV is formulated as:
N ⎡ prod ⎛ ⎤ n
ro qon, j − rw qwn , j ⎞ inj
N N
J = ∑⎢ ∑ ⎜ n ⎟ − ∑rw ,injqinj
n
,l ⎥ t , . . . . . . . . . . . (1)
⎣ j =1 ⎝ (1 + b)
n =1 ⎢ ⎠ l =1 ⎥⎦
t
where, N is the total number of simulation timesteps; Nprod is the
total number of producers; Ninj is the total number of injectors; ro
is oil revenue (USD/STB); rw is water production cost (USD/STB);
rw,inj is water injection cost (USD/STB); qon, j and qwn , j are average
oil and water production rates of the jth producer (STB/D) over
the nth timestep, respectively; qinjn
,l is the average injection rate of
injector l (STB/D) over the nth timestep; b is annual interest rate
(%); tn is the cumulative time up to the nth timestep (year); tn
is the time interval of the nth time step (day). For the production
optimization problem considered here, we put zero cost on the
water injection (i.e., rw,inj = 0) so the second term in the brackets
of Eq. 1 is neglected.
Because oil and water production rates in the NPV of Eq. 1 are
functions of the dynamic state vector x, which includes pressures
and saturations, as well as the well control vector u, the NPV of
Eq. 1 can be written as a function x and u,
J = J [ x , u]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (2)
For the production optimization problem, we wish to maximize
the net present value J defined in Eq. 1 by adjusting the control vec-
tor u subject to the constraints shown in the following equations:
g[ x , u] = 0, . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (3)
Au ≤ b,
u low ≤ u ≤ u up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (4)
Eq. 3 represents the dynamic system (i.e., the reservoir simula-
tion equation). Eq. 4 gives the constraints on the well control vector
u, where ulow and uup are lower and upper bounds of the control
vector u, respectively. In this study, we only consider upper and
lower bound constraints. One way to deal with the upper and lower
bounds of the control variables is to use a log-transformation as in
Gao and Reynolds (2006). For the ith control variable ui, we define
the transformed new variable si such that
⎛ u − u low ⎞
si = ln ⎜ i up i ⎟ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (5)
⎝ ui − ui ⎠
As ui approaches its lower bound uilow , the transformed vari-
able si approaches −, and, as ui approaches its upper bound
uiup , si approaches +. If the upper and lower bounds are the only
constraints on the control variables, the constrained optimization
problem can be transformed to an unconstrained optimization
September 2009 SPE Journal
problem using this log-transformation, as is the case for the
examples shown in the paper. When the log-transformation is
applied during optimization, all the operations are done in the
transformed domain and the actual control variables are obtained
using the inverse log-transformation:
exp(si )uiup + uilow
ui =
1 + exp(si )
uiup + uilow exp(− si )
= . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (6)
1 + exp(− si )
Control Optimization Algorithms
In the production optimization literature, two categories of gradient-
based algorithms have been developed. The first category is based
on the gradient derived from the adjoint method (Brouwer and
Jansen 2004; Sarma et al. 2006). Another category uses an average
gradient derived from the ensemble using statistics (Lorentzen et al.
2006; Nwaozo 2006). In this work, we will explore both categories.
Because we have not implemented the adjoint method to obtain the
gradient of NPV with respect to the well controls, we temporarily
use the finite-difference method to calculate the gradient. The SPSA
algorithm of Spall (1998) was introduced into petroleum engineer-
ing by Gao et al. (2007) for history-matching problems. However,
SPSA did not prove to be as robust and efficient as L-BFGS.
Similar to the finite-difference method, SPSA calculates the gradi-
ent using perturbation. However, this method, as its name implies,
simultaneously perturbs all the variables to calculate a stochastic
gradient compared to the finite-difference method, which perturbs
one variable at a time. The expectation of the stochastic gradient
from SPSA is the true gradient. Bangerth et al. (2006) applied a
modified version of the SPSA algorithm (integer SPSA) to the well
placement optimization. As well locations correspond to lattice
points in a grid system, in their implementation, the perturbation
and the update solution are rounded to be integers. The efficiency of
the integer SPSA is comparable to the very fast simulated annealing
method. To the best of our knowledge, SPSA has not been applied
to the optimal control problems.
In the paper, we compare these three different optimization
algorithms.
Steepest Ascent Method. The steepest ascent method has been
used for production optimization problems by Brouwer and Jansen
(2004) and Sarma et al. (2005, 2006). We use the finite-difference
method (perturbation method) to calculate the gradient. To obtain
the gradient of the NPV J with respect to control variable ui, we
use the one-sided perturbation
dJ J (u) |ui +ui − J (u) |ui
≈ , i = 1,…, N u, . . . . . . . . . . . . . . . (7)
dui ui
where Nu is the total number of control variables and ui is the
perturbation size. Because only one control variable is perturbed
at a time to get the gradient and each perturbation requires one
simulation run to evaluate J (u) |u +u , the method is not applicable
i i
when there is a large number of controls to adjust, although it is
easy to use the procedure with any reservoir simulator. Here, our
focus is not to find an efficient way for gradient calculation but
to compare the steepest ascent method to other optimization algo-
rithms assuming one can obtain the gradient efficiently.
After the gradient of the net present value to all the control
variables are calculated using Eq. 7, we maximize the objective
function (J) with the steepest ascent method:
u k +1 = u k +  k +1∇ k J , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (8)
u
where k+1 is the step size for the (k+1)st iteration. In Eq. 8, ∇ k J
u
is the gradient of the net present value J with respect to the actual
k
control vector u evaluated at u , i.e.
507
 T T
⎡ dJ dJ ⎤  k = ⎡  k ,1 ,  k ,2 ,…,  k ,N ⎤ , . . . . . . . . . . . . . . . . . . . . . . . . (13)
∇ kJ = ⎢ ,…, ⎥ . . . . . . . . . . . . . . . . . . . . . . . . . . . . (9)
u
⎢⎣ du1 duN
u
⎥⎦u k
When there are upper and lower bounds on the control vari-
ables, we use log-transformation (Eq. 5) to eliminate this linear
constraint. The gradient of the net present value to the transformed
control variable si is calculated similar to Eq. 7, i.e.,
dJ J (s) |si +si − J (s) |si
= , i = 1,…, N s , . . . . . . . . . . . . . . . . (10)
dsi  si
where Ns = Nu and si is the perturbation size on the transformed
control variable. Because the log-transformation is a nonlinear
transformation, attention needs to be paid to the choice of the per-
turbation size of the transformed control variables, especially when
the actual control is close to its boundary. A small perturbation of
the transformed control variable when it is close to the boundary
will give a negligible change on the actual control, which results
in no change on the NPV and zero gradient. For this reason, the
perturbation size on the transformed variable is calculated using a
fixed perturbation size on the actual control. On the basis of experi-
ments, we choose 0.1 psi for the perturbation of the BHP controls
in the example of the paper. This ensures that each perturbation on
the transformed variable influences the net present value.
The transformed control vector is updated with the steepest
ascent formulation:
s k +1 = s k +  k +1∇ k J . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (11)
s
After the transformed control vector is updated, it is trans-
formed back to the actual control before it is input into the simula-
tor for NPV evaluation.
When the optimization is done in the transformed space, a trial
step size is used in Eq. 11 to update the control vector s, followed
by a quadratic fit to determine k+1. In the paper, we use the inverse
of the infinity norm of the gradient as the trial step size, which is
equivalent to a unit step after the search direction is normalized by
its infinity norm. If the objective function does not increase with
this step size, k+1 is cut by half until J(sk+1) is greater than J(sk).
The stoping criterion of the steepest ascent algorithm is based on
the relative change of NPV during iteration. When the relative change
of NPV is smaller than a specified small value, iteration stops.
SPSA. In the previous section, we have approximated the gradient
using a perturbation method because we do not have the adjoint
code coupled with the simulator used. As we mentioned, this
method is very time-consuming because each perturbation requires
at least one simulation run. SPSA is a compromised solution to the
finite-difference method, in which all the parameters are perturbed
at one time stochastically (Spall 1998; Bangerth et al. 2006; Gao et
al. 2007). As a result of this stochastic perturbation, the calculated
gradient is also stochastic; however, its expectation is the true gra-
dient (Gao et al. 2007; Spall 1998). Because all the parameters are
perturbed together, SPSA only requires two simulation runs for the
one-sided perturbation and three for the central difference, which
greatly saves the number of simulation runs compared to the finite-
difference gradient calculation of the previous section. Because
all the problems we are dealing with here have upper and lower
bounds, we optimize on the log-transformed control vector s.
The stochastic gradient is calculated using a central differ-
ence based on simultaneous perturbation of the control vector s
as follows,
J (s k + ck  k ) − J (s k − ck  k ) −1
ĝk (s k ) =  k , . . . . . . . . . . . . . . (12)
2ck
where k is an Nu-dimensional random column vector,
508
⎣ u⎦
and −1
k is defined as
T
−k 1 = ⎡ −k 1,1 , −k 1,2 ,…, −k 1,N ⎤ , . . . . . . . . . . . . . . . . . . . . . . . . (14)
⎣ u⎦
and  k ,i , i = 1,2 N u represents independent samples from the
symmetric ±1 Bernoulli distribution. This means that k,i can only
take either +1 or −1 and the probability of taking each value is 0.5,
so the expectation of k,i is zero (E[ k ,i ] = 0). Note that because k,i
can only take either +1 or −1, −1 k =  k . In Eq. 12, ck is a positive
coefficient, which controls the size of perturbation and is chosen
in the same way as the perturbation size in the finite-difference
method of the last section.
As in the steepest ascent method, the control vector is updated
at the (k+1)st iteration using the following equation:
s k +1 = s k +  k +1gˆ k (s k ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (15)
Note that ĝk (s k ) is a random vector because k is a random
vector; therefore, Eq. 15 seems to be a random search direction.
However, this random direction is always uphill from Eq. 12 for
sufficiently small ck. If −1 k =  k is an uphill direction, then the
J ( s k + ck  k ) − J ( s k − ck  k )
scalar is positive and ĝk and −1
k have
2ck
−1
the same direction, the uphill direction. If  k =  k is a downhill
J ( s k + ck  k ) − J ( s k − ck  k )
direction, then the scalar is negative and
2ck
ĝk and −1
k are in the opposite directions (i.e., ĝk has the uphill direc-
tion). Provided the step size k+1 is reduced in an appropriate way, the
method should at least converge to a local maximum (Spall 1998).
Because the expectation of the stochastic gradient ĝk (s k ) is the
true gradient, we also tried to use the average stochastic gradient
as a search direction, i.e.
s k +1 = s k +  k +1 gˆ k (s k ), . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (16)
where the average stochastic gradient ĝk (s k ) is defined as,
1 M
gˆ k (s k ) = ∑gˆi (s k ), . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (17)
M i =1
with each ĝi (s k ) obtained from Eq. 12 using a different sample
of k.
When the control vector is updated using a single stochastic
gradient (Eq. 15), k+1 is determined by a quadratic fit along the
stochastic gradient direction with three points J (s k − ck  k ), J (s k )
and J (s k + ck  k ). If the NPV does not increase with the fitted step
size, we simply reject this step, generate a new k, and repeat the
above process. If five trials do not result in an improvement in
NPV, we stop the algorithm. We also set a maximum number of
iterations as a complementary stoping criterion. When the aver-
age stochastic gradient is used to update the control vector s with
Eq. 16, the same line search for the steepest ascent method is
applied. The same stopping criteria as in the single SPSA gradient
algorithm are applied in the average SPSA gradient method.
EnKF. Zafari and Reynolds (2007) have shown that the EnKF
method is similar to one iteration of the Gauss-Newton method
using an average sensitivity of the production data to the model
parameters. The average sensitivity is an approximation to the sen-
sitivity evaluated at the ensemble average. Similarly, the EnKF can
be used as an optimization method instead of a data assimilation
method (Nwaozo 2006).
September 2009 SPE Journal
 Let u be a random vector for the well control. Assuming the Using Eqs. 21 and 23 in Eq. 24 gives
random vector follows a Gaussian distribution, its mean at the
(k+1)st iteration can be approximated by Ne realizations 1 e N

C
uk J
≈ ∑ ⎡(u kj − u k )(u kj − u k )T ⎤⎦ ∇J . . . . . . . . . . . . . . (25)
N e − 1 j =1 ⎣
N
1 e k
k
u = ∑u j , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (18)
N e j =1 Substitution of Eq. 20 into Eq. 25 yields

and its covariance is given by C ≈ C k ∇J . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (26)

uk J u

C k = E ⎡⎣(u k − u k )(u k − u k )T ⎤⎦ , . . . . . . . . . . . . . . . . . . . . . . (19)

u The average gradient can be obtained by
which can be approximated using the Ne samples of the control
vector u by ∇J ≈ C †k C , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (27)
u uk J

Ne
where C k is approximated using Eq. 20 and C k is approximated
1 u u J
Ck ≈
u
∑ ⎡(u kj − u k )(u kj − u k )T ⎤⎦ . . . . . . . . . . . . . . . . . . (20)
N e − 1 j =1 ⎣
using Eq. 24. Because C k is rank deficient, its pseudoinverse (C †k )
u
is calculated using singular value decomposition (Li and Reynolds
u

2007). For production optimization, we use this approximate aver-

Note that the subscript j represents the jth realization of the
control vector instead of the jth element of the control vector. With
each realization of the control vector u kj , we evaluate the net present
value J (u kj ). From Taylor’s series following Zafari and Reynolds
(2007), J (u kj ) is approximated by,
T
J (u kj ) ≈ J (u k ) + ∇J (u kj − u k ), j = 1,2,…, N e, . . . . . . . . . . . (21)
where ∇J is the gradient of the NPV with respect to the control
vector evaluated at u k. The mean of the NPV is estimated by
N
1 e
Jk ≈ ∑J (u kj ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (22)
N e j =1
As in Zafari and Reynolds (2007), we use the following
approximation
J k ≈ J (u k ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (23)
The covariance between the control vector u and the net pres-
ent value J at the (k+1)st iteration can be approximated using the
Ne samples of the control vector and the corresponding NPVs as
follows:
C = cov(u k , J (u k ))
uk J
age gradient for the control vector update. The well control vector
can be updated by
u k +1 = u k +  k +1 ∇J , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (28)
where the step size k+1 is determined in the same way as in the
steepest ascent method.
As the EnKF method does not always generate an uphill direc-
tion, it does not seem to be easy to set the stopping criteria. Usually
a maximum number of iterations is set as the stoping criterion.
Example 1
Reservoir Model Description. In this example, we deal with
production optimization with geological uncertainty (closed-loop
reservoir management) for a small synthetic reservoir. The closed-
loop reservoir management involves two steps: production optimi-
zation with geological uncertainty and data assimilation (history-
matching) to reduce the geological uncertainty as data become
available. This is an iterative process with production optimization
and data assimilation alternating throughout the lifetime of the
reservoir. The reservoir in this example has a uniform grid system
of 11×11 gridblocks with x = y = 200 ft. The thickness of the
reservoir is 10 ft. The true log-permeability and porosity fields are
shown in Fig. 1. The reservoir is under five-spot waterflooding,
with one injector located at the center of the reservoir (6, 6) and
four producers placed at the four corners as shown in the perme-
ability distribution map of Fig. 1a. Two high-permeability channels
are in the reservoir: One runs from the lower left corner (P1) to
the top right corner (P4), and one is a short channel at the top left

1
Ne corner. The high-permeability channel connecting the injector to
≈ ∑ ⎡(u kj − u k )( J (u kj ) − J k )T ⎤⎦. . . . . . . . . . . . . . (24)
N e − 1 j =1 ⎣
the two producers P1 and P4 will cause early water breakthrough
in these two wells. A low-permeability barrier is at the lower right

−3.0 0
10 10
0.13 0.08

8 3.3 8 0.16

6.4 0.24
6 6
9.5 0.32

4 4

2 2

2 4 6 8 10 2 4 6 8 10
(a) lnk (b) Porosity

Fig. 1—True property fields, Example 1.

September 2009 SPE Journal 509

 1.6×107 1.60×107

1.5×107 1.50×107

Steepest ascent method

NPV, USD
1.4×107 1.40×107 Average SPSA gradient
NPV, USD

Single SPSA gradient

EnKF

1.3×107 1.30×107
Steepest ascent method
Average SPSA gradient 1.20×107
1.2×107
EnKF
Single SPSA gradient
1.10×107
1.1×107 0 200 400 600 800 1,000
0 20 40 60 80 Numbers of Simulation Runs
Iteration Number
Fig. 3—NPV as a function of the number of simulation runs,
Fig. 2—NPV as a function of iteration number, Example 1. Example 1.

corner between the injector and the producer P2, which retards
the flow of water toward P2. In the time period considered, there
is no water breakthrough at P2. The water breakthrough time in
P3 is after that of P1 and P4. Similar features are shown in the
porosity distribution because we use a correlation of 0.8 between
porosity and log-permeability to generate these two fields. During
the waterflooding project, we keep the water injection rate constant
at 1,000 STB/D. The anticipated waterflooding project life is 960
days, and we set the control step for the producers to 120 days so
there are eight control steps and the maximum number of controls
for production optimization is 32. All the producers are at BHP
control with an upper bound of 6,000 psi and a lower bound of
400 psi. Only two phases are in the reservoir: water and oil. In
the example, the following parameters are used: ro = 50 USD/STB,
rw = 15 USD/STB, and b = 20%.
Production Optimization With True Geology. We first compare
three different optimization algorithms (steepest ascent, SPSA,
and EnKF), assuming the true geology is known. For the steepest
ascent and SPSA, we set the initial BHP of four producers all equal
to 1,000 psi. For the EnKF method, similar to Nwaozo (2006),
we generate Ne = 40 realizations of the initial BHP for producers
using the following steps: (i) The mean BHP for each realization
of each well is independently sampled from a uniform distribution
between 1,000 psi and 5,500 psi. (ii) With each realization of the
mean of each well from Step 1, the BHP distribution as a function
of time is generated by sampling a Gaussian distribution with the
prescribed mean and the following covariance function:
NPV of USD 1.58 × 107 in 80 iterations with approximately 1,000
simulation runs, an NPV value only slightly lower than the value
obtained from the steepest ascent method and the average SPSA
gradient method. The performance from the EnKF is shown by the
curve with triangles. Because there are Ne realizations of BHP, the
NPV plotted is the highest among the Ne realizations. Note that,
in this method, the average gradient calculation (Eq. 27) requires
Ne simulation runs (one for each realization). Therefore, the num-
ber of simulation runs for each iteration is equal to Ne plus the
number of trials for the line search. With an initial highest NPV
of USD 1.14 × 107, the EnKF method increases the NPV to USD
1.54 × 107, the lowest value attained for any method. Moreover,
the EnKF procedure requires 70 iterations for convergence, which
is equivalent to more than 2,500 reservoir simulation runs. We can
see that the steepest ascent algorithm will be sufficiently efficient
if the adjoint gradient is available. In the following production
optimization with data assimilation, we will focus on using the
steepest ascent method. We have tested the production optimiza-
tion procedure with different initial guesses for the steepest ascent
method and all of them ended up with the same final controls and
the same NPV. Fig. 4 shows the NPV vs. iteration number of dif-
ferent initial guesses for the steepest ascent method. All the cases
converge to the same final NPV within five iterations.
Fig. 5 shows the final controls obtained from the steepest
ascent method. The results show a pure bang-bang solution to
this problem, which is consistent with the results from Zandvliet
et al. (2007) because the only constraints on the BHP control are

⎡ −3 | i − j | ⎤
Ci , j =  2 exp ⎢ ⎥⎦ , . . . . . . . . . . . . . . . . . . . . . . . . . . (29)
⎣ a 1.6×107

where  is the standard deviation (200 psi for this case); a = 5 is

the correlation range; i, j are the control step indices. 1.4×107
Fig. 2 shows the NPV change with iteration number for differ-
NPV, USD

ent optimization algorithms, and Fig. 3 shows the corresponding 1.2×107

NPV vs. the number of simulation runs. The curve with squares
shows the performance from the steepest ascent method. The
algorithm converges in five iterations. Note that, in Fig. 3, we 1.0×107 500 psi Initial BHP
assume the gradient used in the steepest ascent method is obtained 1,000 psi Initial BHP
from the adjoint method and each adjoint calculation is counted
8.0×106 2,000 psi Initial BHP
as one simulation run. The steepest ascent method took approxi- 3,000 psi Initial BHP
mately 20 equivalent simulation runs for convergence. The NPV
increases from USD 1.34 × 107 to USD 1.59 × 107. The curve 6.0×106
with circles is the performance using the average of 10 SPSA 0 1 2 3 4 5
gradients. Although, the convergence is slower than the steepest Iteration Number
ascent algorithm, this method converges to the same NPV as the
steepest ascent method using the same initial guess. The algorithm Fig. 4—Optimization comparison of different initial BHP for
with a single SPSA gradient (curve with crosses) converges to an steepest ascent algorithm, Example 1.

510 September 2009 SPE Journal

 6,000
5,000
BHP of P1, psi
4,000
3,000
2,000
1,000
0 0
0 240 480
Time, days
6,000
5,000
BHP of P3, psi
4,000
3,000
2,000
1,000
0 0
0 240 480
Time, days
Fig. 5—Final BHP controls from steepest ascent and SPSA with average gradient, Example 1.
the upper and lower bound constraints. Producers P1 and P4 stay
at the lower bound of the BHP specified at early times and then
are equal to the upper bound of the BHP specified at very late
times. This is because these two producers are connected to the
injector by a high-permeability channel. The increase in BHP at
late times corresponds to the increase in watercut. As the BHPs
increase to the upper bound, the wells are effectively shut in to stop
water production. Producer P2 remains at the lower bound of the
BHP specified for the whole reservoir life because this producer
is separated from the injector by a low-permeability region, which
acts as a barrier for the water movement. Although there is water
breakthrough in Well P3, the BHP remains at its lower BHP bound
for the whole reservoir life of 960 days, which may seem unusual.
However, a close check on the gradient shows that the elements
in the gradient corresponding to the controls at the lower bound
are negative and the elements in the gradient corresponding to
the controls at the upper bound are positive. As any change in the
control will either violate the constraints or reduce the NPV, any
points in the feasible neighborhood of the final control will yield
a lower NPV. Therefore, the final control is a local maximum fol-
lowing the reasoning in Nocedal and Wright (1999) for constrained
optimization. Note that the well is shut-in automatically whenever
the BHP is higher than the gridblock pressure in which it resides.
When the BHP reaches its upper bound, we shut in the well even
if this upper bound is still lower than the gridblock pressure. Note
that, in Fig. 5 and the following similar figures, we plot the BHP at
its upper bound as long as the well is shut-in. The actual BHP from
the production optimization might be lower than the upper bound.
At the first iteration of the steepest ascent method, all the compo-
nents of the gradient are negative except the ones corresponding
to the last control step (between Day 840 and Day 960) of P1 and
P4, which have water breakthrough times much earlier than 840
days, are positive. Early iterations will drive these controls of the
last control step to the upper bound and all the others to the lower
bound. Once the controls of the last control step get to the upper
bound, the ones next to them will be driven to the upper bound.
This continues until a local maximum is found. This explains
why the steepest ascent method with different initial guesses
gave the same NPV at convergence: All the cases with different
initial guesses have the same controls after the first iteration. The
September 2009 SPE Journal
6,000
5,000
BHP of P2, psi
4,000
3,000
2,000
1,000
720 960 0 240 480 720 960
Time, days
6,000
5,000
BHP of P4, psi
4,000
3,000
2,000
1,000
720 960 0 240 480 720 960
Time, days
observation seems to coincide with switching time optimization
(Zandvliet et al. 2007) (i.e., optimizing on the time to switch from
the lower bound (“on” status in valve setting) to the upper bound
(“off” status in valve setting) or vice versa. For this example, we
have one switching time per producer and the switching time starts
from the end of the reservoir life and then moves backward in time
during optimization.
The final BHP controls obtained from the algorithm using an
average of 10 SPSA gradients are the same as the ones from the
steepest ascent method shown in Fig. 5. This suggests that SPSA
with an average gradient might be promising for production opti-
mization in the case that the true gradient cannot be readily cal-
culated. Fig. 6 shows the final BHP obtained from a single SPSA
gradient. Although the NPV increased to a value close to that from
the steepest ascent method as shown in Fig. 2, the final BHP does
not seem realistic with the nonsmooth behavior for Well P2. This
is mainly because, with a single SPSA gradient, SPSA is similar
to a random search algorithm. Other wells show earlier well shut-
in (BHP controls at the upper bound) than that from the steepest
ascent method. Fig. 7 shows the BHP with the highest NPV from
the N realizations of the EnKF method.
Production Optimization and Data Assimilation. In this case,
we assume that we do not know the true geology. With sequential
gaussian cosimulation, we generated 90 ensemble members of the
porosity and log-permeability fields from the prior geological infor-
mation. As an ensemble of geological models are history matched
with the EnKF, it seems natural to use the robust optimization
method proposed by van Essen et al. (2006). However, the robust
optimization requires NPV and gradient evaluation for each ensem-
ble member at every iteration, which is very computationally costly
and impractical for us as we use finite-difference method to approxi-
mate the gradient. Instead, the production optimization in the paper
is done on the central model, which is the updated model obtained
by assimilating measurements without perturbation using the prior
mean as its initial realization. For the linear case, the central model
is equivalent to the maximum a posteriori (MAP) estimate (Zafari
and Reynolds 2007). The basic procedure follows:
1. Optimize on the control with the prior mean model (central
model) for the whole reservoir life.
511
 6,000 6,000
5,000 5,000

BHP of P1, psi

BHP of P2, psi

4,000 4,000
3,000 3,000
2,000 2,000
1,000 1,000
0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

6,000
6,000
5,000
5,000
BHP of P3, psi

BHP of P4, psi

4,000
4,000
3,000
3,000
2,000
2,000
1,000
1,000
0
0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

Fig. 6—Final BHP controls from a single SPSA gradient, Example 1.

2. Generate true data using the final optimum control to the time
there are measurements. Note that we have measurements every
30 days and the measurements include oil and water production
rates from producer and the BHP from the injector. The synthetic
data are generated by adding noise to the true data. The standard
deviation for the measurement error in oil and water rates is 5
STB/D and in BHP of the injector is 10 psi.
3. Assimilate data with EnKF up to a point tn that production
optimization is requested, which occurs every 120 days.
4. Optimize on the controls based on the updated central model
from tn to the end of the reservoir life.
5. Repeat Steps 2, 3, and 4 until the end of the reservoir life.
Fig. 8 shows the evolution of the average log-permeability
after data assimilation at 60, 120, 240, and 480 days. After data
assimilation up to Day 60, the long high-permeability channel
connecting Wells P1 and P4 is recognizable. After Day 120,
the short high-permeability channel between the injector and
the producer P3 becomes evident. After Day 240, it seems that

6,000 6,000
5,000 5,000
BHP of P1, psi

BHP of P2, psi

4,000 4,000
3,000 3,000
2,000 2,000
1,000 1,000
0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

6,000
6,000
5,000
5,000
BHP of P3, psi

BHP of P4, psi

4,000
4,000
3,000
3,000
2,000
2,000
1,000
1,000
0
0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

Fig. 7—Final BHP controls from EnKF, Example 1.

512 September 2009 SPE Journal

 −3.0 −3.0
10 10
0.13 0.13
8 3.3 8 3.3

6.4 6.4
6 6
9.5 9.5
4 4

2 2

2 4 6 8 10 2 4 6 8 10
(a) 60 days (b) 120 days

−3.0
−3.0
10 10
0.13
0.13
8 3.3
8 3.3
6.4
6.4 6
6
9.5
9.5
4
4

2
2

2 4 6 8 10 2 4 6 8 10

(c) 240 days (d) 480 days

Fig. 8—Evolution of the average log-horizontal permeability field during data assimilation, Example 1.

both high-permeability channels become wider than in the true
geology (Fig 1a).
Fig. 9 shows the average log-permeability and average poros-
ity distribution after data assimilation at 960 days. Comparing
Fig. 9a to the true permeability distribution in Fig. 1a, we see that
the average log-permeability distribution after data assimilation
with the EnKF captures the main geological features, especially
the long high-permeability channel connecting the injector with
the two producers (P1 and P4), although the channel in the EnKF
result is a little wider than the truth. The short high-permeability
channel close to producer P3 in the EnKF result is much wider
than the truth. The high permeability area around producer P2 is
more or less shown in the average log permeability distribution,
which is similar to the true case. However, the low-permeability
barrier between the injector and the producer P2 is shifted toward
the injector. The average porosity distribution in Fig. 9b after
data assimilation at 960 days roughly captures the true geological
features, but the estimate is poorer than that of the permeability
distribution.
Fig. 10 shows the ensemble predictions of the oil production
rate during data assimilation compared to the truth. In all the
similar figures, red curves represent the true case, blue curves
are the central model, and gray curves are the ensemble predic-
tions from each step of data assimilation. It can be seen that the
ensemble predictions give large uncertainty during early-time data
assimilation and the uncertainty band becomes very small as more

−3.0 0
10 10
0.13 0.08

8 3.3 8 0.16

6.4 0.24
6 6
9.5 0.32

4 4

2 2

2 4 6 8 10 2 4 6 8 10
(a) lnk (b) Porosity

Fig. 9—Average property fields after data assimilation to 960 days, Example 1.

September 2009 SPE Journal 513

 2,000 1000
Ensemble Ensemble
Central model

WOPR of P2, STB/D

WOPR of P1, STB/D
1,500 800 Central model
True True
600
1,000
400
500
200

0
0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days
1,500 1,500
Ensemble Ensemble
WOPR of P3, STB/D

Central model Central model

WOPR of P4, STB/D

True True
1,000 1,000

500 500

0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

Fig. 10—Ensemble oil production rate compared to the truth during data assimilation, Example 1.

data are assimilated. During the course of data assimilation, the
true case is always within the uncertainty band, so the results do
not appear to be biased. Fig. 11 shows the ensemble predictions
of the water production rate during data assimilation compared to
the truth. Similar behavior to the oil production rate is observed.
Fig. 12a show the ensemble prediction of the BHP compared to the
truth. As in the water and oil production rate predictions, there is
a large uncertainty at early times and some of the ensemble mem-
bers even reach the maximum BHP specified (10,000 psi). After
approximately 240 days, the uncertainty band of the ensemble

400 50
Ensemble Ensemble
40 Central model
WOPR of P1, STB/D

Central model
WOPR of P2, STB/D

300 True
True
30
200
20

100
10

0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days
500
600

500 Ensemble 400

WOPR of P4, STB/D
WOPR of P3, STB/D

Central model
400 True
300
300
200 Ensemble
200 Central model
100 True
100

0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

Fig. 11—Ensemble water production rate compared to the truth during data assimilation, Example 1.

514 September 2009 SPE Journal


10,000
Ensemble
8,000 Central model
WBHP of INJ, psi
True
6,000
4,000
2,000
0 0
0 240 480 720
Time, days
(a) WBHP inj
6×105
Cum. Oil Production, STB
4×105
2×105
Ensemble
Central model
True
0 0
0 240 480 720
Time, days
(c) FOPT
Fig. 12—Ensemble prediction compared to the truth during data assimilation, Example 1.
predictions is so narrow that we see the truth essentially coincides
with the ensemble predictions. The average reservoir pressure in
Fig. 12b shows the typical “saw tooth” behavior of sequential data
assimilation: The uncertainty band increases during prediction and
then gets reduced after data assimilation. Figs. 12c and 12d present
the cumulative oil production and cumulative water production for
the ensemble obtained during data assimilation compared to the
truth. Again the average prediction is close to the truth and the pre-
dictions are unbiased. The relatively large uncertainty band for the
cumulative oil production arises from the fact that the uncertainty
in the oil rates (Fig. 10) is larger than the uncertainty in the water
rates (Fig. 11) at times before 240 days.
As stated in the procedure for data assimilation and production
optimization, these two steps alternate. The final controls from
production optimization based on the central model are exactly
the same as the ones obtained based on the true geology of Fig 5.
Although not shown here, the updated permeability and porosity
distribution for the central model after 960 days is very much like
the average permeability and porosity distribution shown in Fig. 9.
In fact, all the models are fairly close to each other after data
assimilation. It should be noted that every time we do production
optimization, we use an initial guess of 500 psi instead of the final
control from the last step of production optimization. The reason
for doing this is because, once the control goes higher than the
grid block pressure at the first step of production optimization,
the producer will be shut-in because the BHP is higher than the
gridblock pressure. In this case, NPV is never sensitive to the well
control and cannot be adjusted during the optimization.
Nonlinearity. For the closed-loop reservoir management procedure,
we also tried to use different initial guesses for the BHP during pro-
duction optimization to test its stability. Other than 500 psi, we have
tried to use initial BHP of 400 psi, which is the lower bound, 1,000 psi,
2,000 psi, and 3,000 psi. With initial guesses of 400 psi and 500 psi,
we obtained the same results as knowing the true geology. However,
with initial guesses of 1,000 psi, 2,000 psi, and 3,000 psi, we obtained
September 2009 SPE Journal
Avergage Reservoir Pressure, psi
4×103
Ensemble
3×103 Central model
True
2×103
1×103
960 0 240 480 720 960
Time, days
(b) FPR
Cum. Water Production, STB
6×105
Ensemble
Central model
True
4×105
2×105
960 0 240 480 720 960
Time, days
(d) FWPT
slightly different final controls. Fig. 13 shows the final controls when
the initial BHP is 1,000 psi with geological uncertainty. Compared to
the final control obtained with an initial guess of 500 psi, which is the
same as that with true geology shown in Fig. 5, the only difference is
that Well P1 was shut in one control step earlier when the initial guess
for BHP is 1,000 psi. All the other wells have the same final control in
these two cases. As mentioned earlier, the control shown in Fig. 5 is
at least a local maximum because all the controls at the lower bound
have a negative component in the gradient and all the controls at the
upper bound have either zero (well shut-in) or a positive component in
the gradient. When we use the final controls from the case with initial
BHP of 1,000 psi and run the simulator with the true geology, we find
that the NPV is even higher than the maximum obtained with controls
shown in Fig. 5. This confirms that the maximum obtained from the
steepest ascent method with true geology is only a local maximum.
Because there is only one control that is different between these two
cases (compare Figs. 5 and 13), we plot the NPV as a function of
that control from P1 between the lower bound and upper bound with
true geology. The result is shown in Fig. 14. Note that the NPV is a
nonlinear function of the control variable. When the control is at its
lower bound, it has a negative derivative, so it tends to increase NPV
by lowering its BHP. As the BHP increases, this derivative decreases
and reaches zero at approximately 1,350 psi and then become positive
as BHP further increases. The well is shut-in when the BHP reaches
approximately 3,000 psi, so the NPV becomes flat. A check on the
gradients of the control indicates that both solutions are local maxima,
but setting this control to the highest value (shut-in) gives higher NPV
as indicated by the results of Fig. 14.
As mentioned earlier, there is one switching time per producer
and during optimization, the switching time (defined as the time
the BHP control switches from lower bound to upper bound) moves
backward as a function of iteration for the problem considered in
the paper. Here, we explore the behavior of the NPV vs. the switch-
ing time. Fig. 15 shows the NPV vs. switching time for all the
producers in the vicinity of the final controls obtained with known
true geology (i.e., change the switching time of one producer while
515
 6,000 6,000

5,000 5,000

BHP of P2, psi

BHP of P1, psi
4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

6,000 6,000
5,000 5,000
BHP of P3, psi

BHP of P4, psi

4,000 4,000
3,000 3,000
2,000 2,000
1,000 1,000
0 0
0 240 480 720 960 0 240 480 720 960
Time, days Time, days

Fig. 13—Final controls with initial BHP = 1,000 psi for closed-loop reservoir management, Example 1.

keeping all the controls of other wells as in Fig. 5). The figures
show that the highest NPV is obtained at the switching time shown
in Fig. 5 for P2 (Day 960), P3 (Day 960), and P4 (Day 720), but
not for P1. Note that the switching time at Day 960 refers to keep-
ing the BHP at its lower bound for the whole reservoir life. The
highest for P1 is obtained at Day 720 instead of Day 840 (Fig. 5).
The final controls in Fig. 5 are at a local maximum according to
the earlier discussion as we optimize based on the BHP controls.
However, Fig. 15 shows that, if we optimize on the bais of the
switching time, the solution in Fig. 5 will not even be a local
maximum. Fig. 16 shows the NPV vs. the switching time for all
the producers in the vicinity of the final controls obtained from the
closed-loop reservoir management procedure (Fig. 13). The high-
est NPV is obtained at the switching times shown in Fig. 13 for P1
(Day 720), P2 (Day 960), and P4 (Day 720), but not for P3. The
highest for P3 is obtained at Day 840 instead of Day 960. Fig. 16
also shows that the final controls in Fig. 13 can only be a local
maximum when we optimize on the basis of BHP and they will not
be a local maximum if we optimize on the switching time.
15,892,000
15,890,000
15,888,000
NPV, USD
Figs. 15 and 16 show that the NPV vs. the switching time has
a concave up shape, while the NPV vs. a BHP control shown in
Fig. 14 has a concave down shape with maximum at the upper and
lower bounds. This may be an indication that optimization based
on the switching time might be easier than that using the BHP
controls but only if the assumption that there is one switching
time per well is true. Although the consideration of switching time
provides an enhanced understanding of the problem, we have not
implemented a switching time optimization procedure because its
reliable application requires that we have a priori certain knowl-
edge that the optimal control solution is accurately described by
bang-bang behavior.
Example 2
Reservoir Model Description. This example pertains to a reservoir
with 25 × 25 gridblocks and x = y = 118 ft. The thickness of the
reservoir is 50 ft. The true porosity is set to be homogenous with a
value of 0.25, and the true horizontal log-permeability field is shown
in Fig. 17. The reservoir is under five-spot waterflooding, with wells
shown in the permeability distribution map of Fig. 17. Several high-
permeability channels are in the reservoir at 45°. The injector is
drilled through a high-permeability channel. During the waterflood-
ing project, we keep the water injection rate constant at 5,000 STB/D.
The anticipated waterflooding project life is 6 years (2,190 days), and
we set the control step for the producers to half year, so there are 12
control steps and 48 maximum number of controls for production
optimization. During production optimization, we adjust the BHP

15,886,000 controls subject to the same upper and lower bound constraints as in
Example 1. Other parameters are also the same as in Example 1.
15,884,000
Production Optimization With True Geology. In Example 1, we
15,882,000 compared three different algorithms, EnKF, SPSA, and steepest
ascent, and found that EnKF and a single SPSA gradient generate
15,880,000 unrealistic final controls. However, an average of 10 SPSA gradi-
ents gives the same final controls as the steepest ascent method.
15,878,000 In this example, we will focus on comparing the efficiency of
0 2,000 4,000 6,000 the algorithm using the average SPSA gradient to that of steepest
BHP, psi ascent method.
Fig. 18 shows the increase of NPV as a function of iteration
Fig. 14—NVP as a function of one BHP control for P1, Example 1. number. The steepest ascent method converges in six iterations

516 September 2009 SPE Journal

 1.6×107
1.6×107

1.5×107

NPV, USD

NPV, USD
1.4×107 1.5×107

1.3×107

1.2×107 1.4×107
0 240 480 720 960 0 240 480 720 960
Switching Time of P1, days Switching Time of P2, days
1.6×107 1.6×107

1.5×107 1.5×107

1.4×107 1.4×107
NPV, USD

NPV, USD
1.3×107 1.3×107

1.2×107 1.2×107

1.1×107 1.1×107

1.0×107 1.0×107
0 240 480 720 960 0 240 480 720 960
Switching Time of P3, days Switching Time of P4, days

Fig. 15—NPV vs. switching time in the vicinity of final controls with true geology, Example 1.

and gives a final NPV of USD 1.7985 × 108. The final controls
for the four producers using the steepest ascent method are shown
in Fig. 19. As in Example 1, final controls for this problem show
bang-bang behavior. Producer P1 stays at the lower bound of 400
psi for the study period as this producer is not really connected
to the high-permeability channel around the well and, as shown
later, this well has the lowest water production rate. The other three
producers (P2, P3, and P4) were produced at lowest allowable BHP
(400 psi) during early times and then were shut in at 1,278 days,
1,461 days, and 1,278 days, respectively. Using an average of 10
SPSA gradients, the SPSA algorithm increases the NPV from USD
1.68 × 108 to USD 1.78 × 108 in 30 iterations. The final controls
obtained from this algorithm are shown in Fig. 20. Compared to
results from the steepest ascent method, producers P1 and P2 have
the same controls while producers P3 and P4 were shut in one
control step later. However, P4 was turned back on between 1,825
and 2,008 days with the control equal to its lower bound. This may
not be realistic. This is mainly because SPSA gives only a stochas-

1.6×107 1.6×107

1.5×107
NPV, USD
NPV, USD

1.4×107 1.5×107

1.3×107

1.4×107
1.2×107
0 240 480 720 960 0 240 480 720 960
Switching Time of P1, days Switching Time of P2, days

1.6×107 1.6×107

1.5×107 1.5×107

1.4×107 1.4×107
NPV, USD

NPV, USD

1.3×107 1.3×107

1.2×107 1.2×107

1.1×107 1.1×107

1.0×107 1.0×107
0 240 480 720 960 0 240 480 720 960
Switching Time of P3, days
Switching Time of P4, days

Fig. 16—NPV vs. switching time in the vicinity of final controls of closed-loop reservoir management, Example 1.

September 2009 SPE Journal 517

25 0 1.82×108

1.80×108
1.5
20 1.78×108

3.0 1.76×108

NPV, USD
15 1.74×108
4.5
1.72×108
6.0 1.70×108 Steepest ascent
10
Average 10 SPSA gradient
1.68×108 Average 20 SPSA gradient

1.66×108
5 0 10 20 30 40
Iteration Number

Fig. 18—NPV vs. iteration number, Example 2.

5 10 15 20 25

Fig. 17—True horizontal log-permeability distribution, Example 2.

tic gradient, an average of 10 stochastic gradient does not totally
eliminate its stochastic behavior. As we increase the number of
SPSA gradients to 20 to calculate the average gradient, this behav-
ior in the final controls was eliminated as shown in Fig. 21. The
controls for producers P1, P2, and P3 are the same as that obtained
from the steepest ascent method, but P4 was shut in at 1,460 days
compared to 1,278 days in the steepest ascent method (Fig. 19).
The final NPV (USD 1.8 × 108) obtained is slightly higher than the
“optimized” NPV obtained with the steepest ascent method (USD
1.7985 × 108). It took approximately 40 iterations to converge for
the algorithm with an average of 20 SPSA gradients. Compared
to Example 1, a larger number of SPSA gradients is required to
obtain reasonable controls as the problem size (the number of
controls) increases. If we assume the steepest ascent method uses
adjoint gradient and one adjoint gradient calculation is counted as
one simulation, the steepest ascent method takes approximately 30
simulation runs (one base simulation run, one adjoint calculation,
and one for line search) with nine iterations. The SPSA algorithms
with 10 stochastic gradients and 20 stochastic gradients requires
approximately 500 and 1,000 simulation runs, respectively.
Production Optimization and Data Assimilation. As in Example
1, we have generated 90 ensemble members of the log-permeabil-
ity fields from the prior geological information for data assimila-
tion with the EnKF. The production optimization is done on the
central model. In this example, the data assimilation is done for
every half year and control optimization is implemented after each
data assimilation for the rest of the reservoir life using the steepest
ascent method.
Fig. 22 shows the average of the initial ensemble of the ln(k)
field and the evolution of this average field as data are assimilated.
The ensemble average of the updated ln(k) fields is shown at the
data assimilation times, 730, 1,460, and 2,190 days. The high-
permeability channel through the injection well appears after only

6,000 6,000
5,000 5,000
BHP of P1, psi

BHP of P2, psi

4,000 4,000
3,000 3,000
2,000 2,000
1,000 1,000
0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days Time, days

6,000 6,000

5,000 5,000
BHP of P4, psi
BHP of P3, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days Time, days

Fig. 19—Final BHP controls from steepest ascent, Example 2.

518 September 2009 SPE Journal

 6,000 6,000

5,000 5,000

BHP of P1, psi

BHP of P2, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days Time, days

6,000 6,000

5,000 5,000

BHP of P4, psi

BHP of P3, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190
0 365 730 1,095 1,460 1,825 2,190
Time, days
Time, days

Fig. 20—Final BHP controls from average of 10 SPSA gradients, Example 2.

approximately six data assimilation steps and is readily apparent
in the estimate of the average ln(k) field obtained at 730 days.
The estimate of the average field at 2,190 days bears geological
resemblance to the truth and results in good matches of produc-
tion data as shown later but is far from the true ln(k) field (Fig.
17) in many aspects. For example, the estimated field results in
an overlap of the two high-permeability streaks running from the
lower left to the upper right in the true model. Also, the estimated
field does not show the two short disconnected high permeability
streaks that appear toward the upper left in the true model. This,
however, is not very surprising as we expect that flow from the
injector to producer P3 will be largely controlled by the large low-
permeability region that connects the injection well to P3. As can
be seen in Figs. 23 and 24, all ensemble members result in good
but similar data matches because all ensemble members give a
ln(k) field fairly similar to the average field.

6,000 6,000

5,000 5,000
BHP of P1, psi

BHP of P2, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days Time, days

6,000 6,000

5,000 5,000
BHP of P4, psi
BHP of P3, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190
0 365 730 1,095 1,460 1,825 2,190
Time, days
Time, days

Fig. 21—Final BHP controls from average of 20 SPSA gradients, Example 2.

September 2009 SPE Journal 519

 25 25 0
0

1.5 1.5
20 20
3.0 3.0

15 4.5 15 4.5

6.0 6.0
10 10

5 5

5 10 15 20 25 5 10 15 20 25
(a) 0 days (b) 730 days

25 25 0
0

1.5 1.5
20 20
3.0 3.0

15 4.5 15 4.5

6.0 6.0
10 10

5 5

5 10 15 20 25 5 10 15 20 25

(c) 1,460 days (d) 1,290 days

Fig. 22—Average lnk during data assimilation, Example 2.

5,000 5,000
Ensemble
4,000 Central model 4,000
WOPR1, STB/D

WOPR2, STB/D

True Ensemble
3,000 3,000 Central model
True
2,000 2,000

1,000 1,000

0 0
0 365 760 1,095 1,460 1,825 2,190 0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days
5,000 5,000

4,000 Ensemble 4,000 Ensemble

WOPR3, STB/D

WOPR4, STB/D

Central model Central model

True True
3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 760 1,095 1,460 1,825 2,190 0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days

Fig. 23—Ensemble oil production rate compared to the truth during data assimilation, Example 2.

520 September 2009 SPE Journal

 800 800
Ensemble
Central model

WWPR1, STB/D

WWPR2, STB/D
600 True 600

400 400

200 200 Ensemble

Central model
True
0 0
0 365 760 1,095 1,460 1,825 2,190 0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days
800
800
WWPR3, STB/D

WWPR4, STB/D
600
600

400
400

200 Ensemble
200 Ensemble
Central model
Central model
True
0 True
0 365 760 1,095 1,460 1,825 2,190 0
0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days

Fig. 24—Ensemble water production rate compared to the truth during data assimilation, Example 2.

Figs. 23 and 24 show the ensemble predictions of the oil and truth. Figs. 25c and 25d present the field total oil production rate
water production rate during data assimilation compared to the and field total water production rate for the ensemble obtained
truth. Figs. 25a and 25b show the ensemble prediction of the BHP during data assimilation compared to the truth. All the ensemble
of the injector and the average reservoir pressure compared to the predictions during data assimilation show the typical saw-tooth

10,000 10,000
WBHP of INJ, psi

8,000 8,000
FPR, psi

6,000 6,000

4,000 Ensemble 4,000

Central model Ensemble
2,000 True 2,000 Central model
True
0 0
0 365 760 1,095 1,460 1,825 2,190 0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days
(a) WBHP inj (b) FPR

10,000 10,000

8,000 Ensemble 8,000

Central model
FOPR, STB/D

FWPR, STB/D

True
6,000 6,000

4,000 4,000
Ensemble
Central model
2,000 2,000
True

0 0
0 365 760 1,095 1,460 1,825 2,190 0 365 760 1,095 1,460 1,825 2,190
Time, days Time, days
(c) FOPR (d) FWPR

Fig. 25—Ensemble prediction compared to the truth during data assimilation, Example 2.

September 2009 SPE Journal 521

 6,000 6,000

5,000 5,000

BHP of P1, psi

BHP of P2, psi

4,000 4,000

3,000 3,000

2,000 2,000

1,000 1,000

0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days Time, days

6,000 6,000
5,000 5,000
BHP of P3, psi

BHP of P4, psi

4,000 4,000
3,000 3,000
2,000 2,000
1,000 1,000
0 0
0 365 730 1,095 1,460 1,825 2,190 0 365 730 1,095 1,460 1,825 2,190
Time, days
Time, days

Fig. 26—Final BHP controls of closed-loop reservoir management, Example 2.

behavior of sequential data assimilation with decreasing uncer- p = dynamic state variables
tainty as more data are assimilated. The true prediction always qo = oil production rate
falls within the uncertainty band of the ensemble, even though the qw = water production rate
band is quite small. The final controls from production optimiza- qinj = water injection rate
tion based on the central model are shown in Fig. 26. Note that ro = oil revenue
the final controls for producers P1 and P3 are the same as obtained
rw,inj = water injection cost
using the true geology but are different for producers P2 and P4.
In the closed-loop scenario with uncertain geology, wells P2 and rw = water production cost
P4 are both shut in one control step earlier than in the case where s = transformed control vector
optimization of the controls is based on the true known geology. u = control vector
y = state vector
Conclusions  = step size
• Comparison of different optimization algorithms for closed-loop
reservoir management shows that the steepest ascent algorithm Subscripts
is the most efficient one, and it gives reasonable results. The n = timestep index
EnKF, when it is treated as an optimization algorithm, requires obs = observed
significantly more time and yields poor estimates of the optimal uc = unconditional realization
controls. SPSA, with an average stochastic gradient, gives rea-
sonable final controls with slower convergence. The final control Superscripts
from SPSA using a single stochastic gradient is not realistic. k = iteration idex
• Closed-loop reservoir management with the EnKF for data low = lower bound
assimilation and the steepest ascent for production optimization n = timestep index
based on the central model gives reasonable results for the test
p = prediction
examples in the paper. The updated permeability and porosity
fields capture the geological features of the true fields. The final T = matrix transpose
control is similar to that obtained assuming known geology. up = upper bound
• Production optimization is a nonlinear problem that presents † = pseudoinverse
multiple maxima, at least for the cases considered here. Local
maxima are obtained when the controls take their upper and Acknowledgments
lower bounds. The support of the member companies of the University of Tulsa
Petroleum Reservoir Exploitation Projects (TUPREP) is very
Nomenclature gratefully acknowledged.
b = annual interest rate
ck = SPSA perturbation References
C = covariance Alhuthali, A., Oyerinde, D., and Datta-Gupta, A. 2006. Optimal Waterflood
d = data Management Using Rate Control. SPEREE 10 (5): 539–551. SPE-
102478-PA. doi: 10.2118/102478-PA.
g = gradient
Asheim, H. 1998. Maximization of water sweep efficiency by controlling
J = net present value production and injection rates. Paper SPE 18365 presented at the European
Kn = Kalman gain matrix Petroleum Conference, London, 16–19 October. doi: 10.2118/18365-MS.

522 September 2009 SPE Journal

Bangerth, W., Klie, H., Wheeler, M.F., Stoffa, P.L., and Sen, M.K. 2006. On
optimization algorithms for the reservoir oil well placement problem.
Computational Geosciences 10 (3): 303–319. doi: 10.1007/s10596-
006-9025-7.
Brouwer, D.R. and Jansen, J.-D. 2004. Dynamic Optmization of Water
Flooding With Smart Wells Using Optimial Control Theory. SPEJ
9 (4): 391–402. SPE-78278-PA. doi: 10.2118/78278-PA.
Brouwer, D.R., Jansen, J.-D., van der Starre, S., van Kruijsdijk, C.P.J.W.,
and Berentsen, C.W.J. 2001. Recovery Increase Through Water Flood-
ing With Smart Well Technology. Paper 68979 presented at the SPE
European Formation Damage Conference, The Hague, 21–22 May.
doi: 10.2118/68979-MS.
Brouwer, D.R., Nævdal, G., Jansen, J.-D., Vefring, E.H., and van Kruijsdijk,
C.P.J.W. 2004. Improved Reservoir Management Through Optimal
Control and Continuous Model Updating. Paper SPE 90149 presented
at the SPE Annual Technical Conference and Exhibition, Houston,
26–29 September. doi: 10.2118/90149-MS.
Dong, Y. and Oliver, D.S. 2005. Quantitative Use of 4D Seismic Data
for Reservoir Description. SPEJ 10 (1): 91–99. SPE-84571-PA. doi:
10.2118/84571-PA.
Evensen, G. 1994. Sequential data assimilation with a nonlinear quasi-
geostrophic model using Monte Carlo methods to forecast error statis-
tics. J. of Geophysical Research 99 (C5): 10143–10162.
Evensen, G. 2007. Data Assimilation: The Ensemble Kalman Filter. Berlin:
Springer Verlag.
Gao, G. and Reynolds A.C. 2006. An Improved Implementation of the
LBFGS Algorithm for Automatic History Matching. SPEJ 11 (1), 5–17.
SPE-90058-PA. doi: 10.2118/90058-PA.
Gao, G., Li, G., and Reynolds, A.C. 2007. A Stochastic Algorithm for
Automatic History Matching. SPEJ 12 (2): 196–208. SPE-90065-PA.
doi: 10.2118/90065-PA.
Gao, G., Zafari, M., and Reynolds, A.C. 2006. Quantifying Uncertainties
for the PUNQ-S3 Problem in a Bayesian Setting With RML and EnKF.
SPEJ 11 (4): 506–515. SPE-93324-PA. doi: 10.2118/93324-PA.
Jansen, J.-D., Brouwer, D.R., Nævdal, G., and van Kruijsdijk, C.P.J.W. 2005.
Closed-loop reservoir management. First Break 23 (January): 43–48.
Kitanidis, P.K. 1995. Quasi-linear geostatistical theory for inversing. Water
Resources Research 31 (10): 2411–2419. doi: 10.1029/95WR01945.
Li, G. and Reynolds, A.C. 2007. An Iterative Ensemble Kalman Filter for
Data Assimilation. Paper SPE 109808 presented at the SPE Annual
Technical Conference and Exhibition, Anaheim, California, USA,
11–14 November. doi: 10.2118/109808-MS.
Lorentzen, R.J., Berg, A.M., Nævdal, G., and Vefring, E.H. 2006. A new
approach for dynamic optimization of waterflooding problems. Paper
SPE 99690 presented at the Intelligent Energy Conference and Exhibi-
tion, Amsterdam, 11–13 April. doi: 10.2118/99690-MS.
Nævdal, G., Johnsen, L.M., Aanonsen, S.I., and Vefring, E.H. 2005. Reservoir
Monitoring and Continuous Model Updating Using Ensemble Kalman
Filter. SPEJ 10 (1): 66–74. SPE-84372-PA. doi: 10.2118/84372-PA.
Nævdal, G., Mannseth, T., and Vefring, E.H. 2002. Near-Well Reservoir
Monitoring Through Ensemble Kalman Filter. Paper SPE 75235 pre-
sented at the SPE/DOE Improved Oil Recovery Symposium, Tulsa,
13–17 April. doi: 10.2118/75235-MS.
Nocedal, J. and Wright, S.J. 1999. Numerical Optimization. New York:
Springer.
Nwaozo, J. 2006. Dynamic optimization of a water flood reservoir. MS
thesis, University of Oklahoma, Norman, Oklahoma.
Oliver, D.S., He, N., and Reynolds, A.C. 1996. Conditioning perme-
ability fields to pressure data. Proc., 5th European Conference on
the Mathematics of Oil Recovery (ECMOR V), Leoben, Austria, 3–6
September, 1–11.
Sarma, P., Aziz, K., and Durlofsky, L.J. 2005. Implementation of Adjoint
Solution for Optimal Control of Smart Wells. Paper SPE 92864 pre-
sented at the SPE Reservoir Simulation Symposium, The Woodlands,
Texas, USA, 31 January–2 February. doi: 10.2118/92864-MS.
Sarma, P., Chen, W.H., Durlofsky, L.J., and Aziz, K. 2006. Production
Optimization With Adjoint Models Under Nonlinear Control-State Path
Inequality Constraints. SPEREE 11 (2): 326–339. SPE-99959-PA. doi:
10.2118/99959-PA.
September 2009 SPE Journal
Spall, J.C. 1998. Implementation of the simultaneous perturbation algo-
rithm for stochastic optimization. IEEE Transactions on Aerospace and
Electronic Systems 34 (3): 817–823. doi: 10.1109/7.705889.
Sudaryanto, B. and Yortsos, Y.C. 2000. Optimization of fluid front dynam-
ics in porous media using rate control. I. Equal mobility fluids. Physics
of Fluids 12 (7): 1656–1670. doi: 10.1063/1.870417.
Sudaryanto, B. and Yortsos, Y.C. 2001. Optimization of displacement in
porous media using rate control. Paper 71509 presented at the SPE
Annual Technical Conference and Exhibition, New Orleans, 30 Sep-
tember–3 October. doi: 10.2118/71509-MS.
Thulin, K., Li, G., Aanonsen, S.I., and Reynolds, A.C. 2007. Estimation of
Initial Fluid Contacts by Assimilation of Production Data With EnKF.
Paper SPE 109975 presented at the SPE Annual Technical Conference
and Exhibition, Anaheim, California, USA, 11–14 November. doi:
10.2118/109975-MS.
van Essen, G.M., Zandvliet, M.J., van den Hof, P.M.J., Bosgra, O.H., and
Jansen, J.-D. 2006. Robust Waterflooding Optimization of Multiple
Geological Scenarios. Paper SPE 84571 presented at the SPE Annual
Technical Conference and Exhibition, San Antonio, Texas, USA, 24–27
September. doi: 10.2118/102913-MS.
Zafari, M. and Reynolds, A.C. 2007. Assessing the Uncertainty in Reservoir
Description and Performance Predictions With the Ensemble Kalman
Filter. SPEJ 12 (3): 382–391. SPE-95750-PA. doi: 10.2118/95750-PA.
Zakirov, I.S., Aanonsen, S.I., Zakirov, E.S., and Palatnik, B.M. 1996.
Optimizing reservoir performance by automatic allocation of well rates.
Proc., 5th European Conference on the Mathematics of Oil Recovery
(ECMOR V), Leoben, Austria, 3–6 September.
Zandvliet, M.J., Bosgra, O.H., Jansen, J.-D., van den Hof, P.M.J., and
Kraaijevanger, J.F.B.M. 2006. Bang-bang control in reservoir flood-
ing. Paper A040 presented at the 10th European Conference on the
Mathematical Oil Recovery, Amsterdam, 4–7 September.
Zandvliet, M.J., Bosgra, O.H., Jasen, J.-D, van den Hof, P.M.J., and
Kraaijevvanger, J.F.B.M. 2007. Bang-bang control and singular arcs in
reservoir flooding. J. Pet. Sci. Eng. 58 (1–2): 186–200. doi: 10.1016/
j.petrol.2006.12.008.
Zhang, F. and Reynolds, A.C. 2002. Optimization algorithms for automatic
history matching of production data. Proc., 8th European Conference
on the Mathematics of Oil Recovery (ECMOR VIII), Frieiberg, Ger-
many, 3–6 September, 1–10.
Zhang, F., Skjervheim, J.A., Reynolds, A.C., and Oliver, D.S. 2005. Auto-
matic History Matching in a Bayesian Framework: ExampleApplications.
SPEREE 8 (3): 214–223. SPE-84461-PA. doi: 10.2118/84461-PA.
Zhao, Y., Reynolds, A.C., and Li, G. 2008. Generating Facies Maps by
Assimilating Production Data and Seismic Data With the Ensem-
ble Kalman Filter. Paper SPE 113990 presented at the SPE/DOE
Symposium on Improved Oil Recovery, Tulsa, 20–23 April. doi:
10.2118/113990-MS.
Chunhong Wang is a reservoir engineer at PetroTel in Plano,
Texas. She holds a BS degree from University of Petroleum
(East China) and MS degree from University of Tulsa, both in
petroleum engineering. Her current research interests include
reservoir characterization, history matching, and enhanced
oil recovery. Gaoming Li is an associate professor in petroleum
engineering at the University of Tulsa. He also serves as associate
director of TUPREP, an industry research consortium focused on
research in reservoir characterization, reservoir simulation, and
well testing. He holds a BS degree in geophysical well logging
and an MS degree in applied geophysics, both from the
University of Petroleum, China, and a PhD degree in petroleum
and natural gas engineering from Pennsylvania State University.
His current research interests include reservoir characterization
and uncertainty assessment, automatic history matching, and
enhanced oil recovery. Albert C. Reynolds holds the McMan
Chair in Petroleum Engineering at the University of Tulsa, where
he has been a faculty member since 1970. He also serves as
director of TUPREP. He has authored or coauthored more than
100 technical papers and one book. He holds a BA degree
from the University of New Hampshire, an MS degree from Case
Institute of Technology, and a PhD degree from Case Western
Reserve University, all in mathematics.
523