WATER RESOURCES RESEARCH, VOL. 48, W05522, doi:10.

1029/2011WR011778, 2012

Efficient methods for large-scale linear inversion using

a geostatistical approach
Arvind K. Saibaba1 and Peter K. Kitanidis1,2
Received 20 December 2011; revised 16 March 2012; accepted 23 March 2012; published 16 May 2012.
[1] In geophysical inverse problems, such as estimating the unknown parameter field
from noisy observations of dependent quantities, e.g., hydraulic conductivity from head
observations, stochastic Bayesian and geostatistical approaches are frequently used. To
obtain best estimates and conditional realizations it is required to perform several matrix-
matrix computations involving the covariance matrix of the discretized field of the
parameters. In realistic three-dimensional fields that are finely discretized, these operations
as performed in conventional algorithms become extremely expensive and even prohibitive
in terms of memory and computational requirements. Using Hierarchical Matrices, we show
how to reduce the complexity of forming approximate matrix-vector products involving the
Covariance matrices in log linear complexity for an arbitrary distribution of points and a
wide variety of generalized covariance functions. The resulting system of equations is
solved iteratively using a matrix-free Krylov subspace approach. Furthermore, we show
how to generate unconditional realizations using an approximation to the square root of the
covariance matrix using Chebyshev matrix polynomials and use the above to generate
conditional realizations. We demonstrate the efficiency of our method on a few standard test
problems, such as interpolation from noisy observations and contaminant source
identification.
Citation: Saibaba, A. K., and P. K. Kitanidis (2012), Efficient methods for large-scale linear inversion using a geostatistical approach,
Water Resour. Res., 48, W05522, doi:10.1029/2011WR011778.

1. Introduction attainable data involves algebraically underdetermined

[2] The characterization or imaging of the subsurface is
required for the efficient discovery and utilization of natu-
ral resources, the removal of contaminants, and the moni-
toring and management of engineered subsurface systems
such as geothermal plants, nuclear-waste repositories, and
CO2 sequestration sites. In particular, in hydrogeologic and
environmental applications, we are interested in imaging,
among other properties, the hydraulic conductivity and po-
rosity of geologic formations, the soil moisture content in
partially saturated zones, the concentration of dissolved
chemicals, and the location of nonaqueous liquid phases. In
practice, subsurface imaging remains a problem fraught
with difficulties. Some measurement techniques, such as
wells tests or well logging [Batu, 1998], are expensive to
obtain and provide information with limited area coverage.
Imaging between wells introduces interpolation error.
Other measurements such as surface or cross-well geophy-
sics [Rubin and Hubbard, 2005] and hydraulic tomography
[Cardiff et al., 2009] provide good areal coverage but have
low resolution, i.e., cannot identify small-scale features.
Thus, the imaging of the subsurface from practically
1
Institute for Computational and Mathematical Engineering, Jen-Hsun
Huang Engineering Center, Stanford University, Stanford, California, USA.
2
Department of Civil and Environmental Engineering, Stanford Univer-
sity, Stanford, California, USA.
Copyright 2012 by the American Geophysical Union
0043-1397/12/2011WR011778
inverse problems.
[3] A general method for solving such problems is
known as geostatistical, see for example [Kitanidis, 1995].
The approach is based on the idea of combining data with
information about the structure of the function that needs to
be estimated. In Bayesian and geostatistical approaches
(for example, see discussion in the work of Kitanidis
[2007, 2010]), the structure of the function is represented
through the prior probability density function, which in
practical applications is often parameterized through vario-
grams and generalized covariance functions. The method
has found several applications because it is generally prac-
tical and quantifies uncertainty. The method can generate
best estimates, which can be determined in a Bayesian
framework as a posteriori mean values or most probable
values, measures of uncertainty as posterior variances or
credibility intervals, and conditional realizations, which are
sample functions from the ensemble of the posterior proba-
bility distribution.
[4] The major challenges in the large-scale implementa-
tion of the geostatistical approaches are as follows:
[5] 1. The number of unknowns increase, the storage and
computational costs involving the dense covariance matrix
are overwhelming, especially on unstructured grids.
[6] 2. Certain problems with large number of measure-
ments, the measurement operator cannot only be dense,
forming it explicitly would require the repeated solution of
(possibly) time-dependent partial differential equations. In
such cases, the prevailing approach to form matrix-matrix

W05522 1 of 13

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
products with the covariance matrix (even using fast sum-
mation schemes described below) proves to be extremely
expensive.
[7] 3. Uncertainty quantification by generating condi-
tional realizations, an important step is generating uncondi-
tional realizations. General procedures such as Cholesky
factorization are extremely expensive and special treatment
is required.
[8] In this paper, we will deal sequentially with these
issues. The covariance matrices although dense, have spe-
cial structure which can be exploited. They are similar to
dense matrices that arise from the discretization of integral
equations. Various fast summation schemes have been
devised to provide matrix-vector products in OðN log  N Þ
for such problems, where N is the number of unknowns and
  0 is some integer, depending on the chosen approach.
They broadly fall under the following categories: (1) Tree
Codes like Barnes-Hut algorithm [Barnes and Hut, 1986],
(2) Fast Multipole Methods [Greengard and Rokhlin,
1987; Ying et al., 2004] and (3) Hierarchical matrices and
Adaptive Cross Approximation [Bebendorf, 2000; Börm
et al., 2003; Grasedyck and Hackbusch, 2003; Rjasanow
and Steinbach, 2007].
[9] Previously, for regular equispaced grids, the Toeplitz
or block-Toeplitz structure of the covariance matrix has
been exploited to accelerate matrix-vector products to
OðN log NÞ, where N is the number of unknowns. In partic-
ular, Nowak and Cirpka [2006] utilized this method to esti-
mate hydraulic conductivity and dispersivities in a large-
scale problem. However, they cannot easily be extended to
handle nonuniform grids. In the work of Fritz et al. [2009],
they extended the FFT based algorithm to deal with irregu-
larly spaced measurements but they did not show how
to extend their algorithm for general measurement opera-
tors or the case when the underlying unknowns are not on a
regular grid. An algorithm to deal with irregular grids was
developed in the work of Li and Cirpka [2006], that
exploited the Karhunen-Loève expansion to represent the
spatial random field corresponding to a stationary covari-
ance function. They concluded that when the covariance
function is smooth, with a large correlation length, the
number of terms required in the expansion will be small, so
this method would be efficient compared to the FFT based
approach on regular grids. However, they still suffer from
the same defect as the FFT based methods, namely, when
the measurement operator is dense and the number of
measurements are high, forming matrix-matrix products
with the covariance matrices are expensive.
[10] Recently [e.g., Rjasanow and Steinbach, 2007;
Bebendorf, 2008], Fast Multipole Methods and Hierarchi-
cal matrices have gained a lot of popularity in computing
fast matrix-vector products for certain class of dense matri-
ces. Hierarchical matrices appear ideally suited to our cur-
rent situation because they are applicable to a wide variety
of kernels, without much reimplementation, since they han-
dle most operations algebraically. We choose to use the
Hierarchical matrix approach over the kernel independent
versions of the Fast Multipole Method because they are
relatively easier to implement. They can also be combined
effectively with iterative solvers like GMRES [Saad and
Schultz, 1986], to solve linear systems involving these
dense matrices.
2 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
[11] Our major contributions are as follows. This paper
describes methods for the efficient computational imple-
mentation of the geostatistical approach when measure-
ments are collected in both space and time. We focus on
the linear inversion case, and for cases for which the prior
can be described as Gaussian. We will demonstrate that, for
large-dimensional problems, these methods can reduce the
computational burden by orders of magnitude compared to
direct or ‘‘naive’’ implementations that are common in
applications. Our algorithm heavily uses the Hierarchical
matrix approach and, thus, is not limited to regular equi-
spaced grids unlike FFT based algorithms. We use a matrix-
free iterative solver to compute the maximum-a-posteriori
estimate and in order to keep the number of iterations small,
we have designed an efficient preconditioner that relies on a
low-rank representation of the covariance matrix, whose
eigenvalues are known to decay rapidly. An important aspect
of the geostatistical approach is to be able to quantify the
uncertainty of the unknown field and we perform this by
generating conditional realization from the posterior distri-
bution of the unknowns. We apply our method to interpola-
tion from noisy observations and the problem of determining
the initial contaminant distribution from the time-history of
measurements from spatially distributed sensors. This prob-
lem has been discussed extensively in the following referen-
ces [Akcelik et al., 2003, 2005; Flath et al., 2011; Michalak
and Kitanidis, 2003].
[12] The rest of this article is organized as follows.
Section 2 describes the Geostatistical approach to linear
inverse problem. In section 3.1 we describe in detail the
Hierarchical matrix approach to compute fast matrix-vector
products involving the dense Covariance matrix in a black-
box fashion and section 3 describes utilizes these fast
matrix-vector products along with a Krylov subspace solver
to solve the linear inverse problem. We also discuss the
implementation details regarding the preconditioner and a
summary of the operation cost of our algorithm. Further-
more, in section 4.1, an algorithm for computing approxi-
mate conditional realizations based on Chebyshev matrix
polynomials utilizing the fast matrix-vector products
described in section 3.1. Finally, we demonstrate the per-
formance of our algorithm in section 5.
2. Problem Formulation
[13] For the sake of completeness, we will review here
the problem formulation and the prevalent solution
approach. Consider a function sðxÞ, such as log conductiv-
ity, to be estimated. Focusing on a prevalent approach, the
function is represented as follows
X
p
sðxÞ ¼ fk ðxÞk þ ðxÞ; (1)
k¼1
where typically, fk 2 dp , the space of pth degree polynomials
in d variables,  are unknown coefficients and the second
term is a random function with zero mean and character-
ized by a covariance function. After discretization (e.g.,
using finite differences, or finite element models), sðxÞ is
represented by the vector s 2 Rm , from a set of noisy meas-
urements y 2 Rn
y ¼ Hs þ v v  N ð0; RÞ; (2)

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
where H 2 Rnm is called the measurement operator and v
is an random n-vector corresponding to measurement error,
with mean zero and covariance matrix R. We also have,
E½s ¼ X E½ðs  XÞðs  XÞT  ¼ Q; (3)
where X is m  p drift matrix, whose columns are com-
puted as Xij ¼ fi ðxj Þ and  are p unknown drift coefficients.
The entries of the covariance matrix Q, are given by
Qij ¼ Kðxi ; xj Þ, where Kð; Þ is a generalized covariance
function which must be conditionally positive definite. For
a more detailed discussion on covariance kernels that per-
missible, we refer the reader to the following references
[Christakos, 1984; Matheron, 1973]. Here R, X, and Q are
considered known and are part of a modeling choice.
[14] Some possible choices for the Covariance Kernel
Kð; Þ arise from the Matérn family of covariance kernels
[Stein, 1999], corresponding to isotropic, stationary stochas-
tic process. They are defined as K; ðx; yÞ ¼ C; ðrÞ; r ¼
kx  yk with
 We also denote by  ¼ HQHT þ R,  ¼ HX and we also
C; ðrÞ ¼ 1
ðrÞ K ðrÞ;  > 0;  > 0;  > 0; (4)
2 CðÞ
where K is the modified Bessel function of second kind of
order  and C is the gamma function. Equation (4) takes
special forms for certain parameters . For example, when
 ¼ 0:5, C; corresponds to the exponential covariance
function,  ¼ 0:5 þ n where n is an integer, C; is the
product of an exponential covariance and a polynomial of
order n. In the limit as  ! 1, and for appropriate scaling
of , C; converges to the Gaussian covariance.
[15] Since the prior probability distribution of the
unknowns s and the probability density of the error are
both Gaussian, the prior and noise pdfs can be written as
follows
 
1 n matrix-vector products involving Q. Constructing the ele-
pðsjÞ / exp  ðs  XÞT Q1 ðs  XÞ
2 ments of QHT in a direct fashion, requires significant CPU
 : (5)
1 time [Nowak et al., 2003]. Next, we will discuss a method
pðyjsÞ / exp  ðy  HsÞT R1 ðy  HsÞ
2
Using Bayes’ theorem, assuming a uniform prior for , i.e.,
pðÞ / 1, we have pðs; Þ, the prior probability distribution
of the unknowns and the probability distribution of the
error can be combined to give the posterior probability dis-
tribution of the unknown parameters, s and 
pðyjs; Þpðs; Þ
pðs; jyÞ ¼
pðyÞ : (6)
/ pðyjs; Þpðs; Þ
Plugging in the expression for the respective terms, and
using an alternative convenient notation, we find that
 
1 1
pðs; jyÞ / exp  ks  XkQ1  ky  HskR1 ; (7)
2 2
where kxkM ¼ xT Mx is a norm, when M is a positive defi-
nite matrix. The resulting posterior distribution is Gaussian,
3 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
as well. The posterior mean values, ^s and ^ is given by the
maximum a posteriori estimate, which is equivalent to solv-
ing a weighted least squares optimization problems
 
1 1
arg mins; ks  XkQ1 þ ky  HskR1 : (8)
2 2
[16] For n < m, it is more convenient to compute the so-
lution to this optimization problem (8) by first obtaining
the solution of the following linear system of equations,
! ! !
HQHT þ R HX ^ y
¼ ; (9)
ðHXÞT 0 ^ 0
and then computing the resulting unknown field from the
solution of the system of equations (9) by the following
transformation
^s ¼ X^ þ QHT ^ : (10)
def def
define the matrix A as
!
 
A¼ : (11)
T 0
[17] A straightforward implementation to solve the max-
imum a posteriori (MAP) problem can be done by con-
structing the system of equations in Oðm2 n þ mn2 þ mnpÞ
operations and solving the dense linear system using a
direct solver such as Gaussian Elimination in Oðn þ pÞ3 .
The storage costs are dominated by the dense Q matrix,
which is of size Oðm2 Þ. This cost can be significant in a
problem of size m  106 or higher, which can easily occur
in practice. However, this difficulty is usually circumvented
by computing QHT without constructing Q. This involves
for solving these systems, that rely on the fast multiplica-
tion of Q with a vector.
3. Solving the System
3.1. Hierarchical Matrices
[18] Hierarchical matrices [Börm et al., 2003; Grasedyck
and Hackbusch, 2003; Rjasanow and Steinbach, 2007;
Bebendorf, 2008] (or H-matrices, for short) are efficient
data-sparse representations of certain densely populated
matrices. The main idea that is used repeatedly in these kind
of techniques, is to split a given matrix into a hierarchy of
rectangular blocks and approximate each of the blocks by a
low-rank matrix. Hierarchical matrices have been used suc-
cessfully in data-sparse representation of matrices arising in
the Boundary Element method or for the approximation of
the inverse of a Finite Element discretization of an elliptic
partial differential operator. Fast algorithms have been
developed for this class of matrices, including matrix-vector
products, matrix addition, multiplication and factorization
in almost linear complexity [Börm et al., 2003].

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
[19] The covariance matrix Q has entries Qij ¼ Kðxi ; xj Þ
for a set of points fxi gNi¼1 for a certain class of generalized
covariance functions Kð; Þ. Using the Hierarchical matrix
approach, it can be shown that the complexity of storing Q
and multiplying Q and QT by an appropriately sized vector
has complexity Oðkmlog mÞ, where m is the number of
unknowns, and the numerical cost for approximating the
whole matrix is Oðk 2 mlog mÞ, where k is the block-wise rank
which is chosen such that the relative Frobenius norm of each
subblock is . For a rigorous analysis, the reader is referred to
[Bebendorf, 2000; Bebendorf and Rjasanow, 2003]. Thus,
the approximation for the H-matrix is of the form
kQ  QH kF  "kQkF :
Our implementation closely follows the methods described
in the following references [Rjasanow and Steinbach,
. For example, if H is dense then
¼ mn and
2007; Bebendorf, 2005]: the construction of the cluster
¼ nnz, where nnz is the number of
tree and block-cluster tree are based on the bisection
method described in the work of Rjasanow and Steinbach
[2007], whereas the low-rank approximation to the sub-
blocks is computed by using the Partially Pivoted Adaptive
Cross Approximation algorithm [Bebendorf and Rjasanow,
p þ k 2 mlog mÞ, where k is
2003; Bebendorf, 2000, 2008].
3.2. Iterative Solver
[20] The system of equations (9), is solved iteratively
using a Krylov subspace method. The key advantage of
using Krylov subspace methods, is that they never require
the explicit entries of the matrix but only rely on matrix-
vector products involving the matrix or its transpose. We
review the basic properties of Krylov subspace methods that
help us understand the convergence of the algorithms. These
methods when applied to the linear system of equations
Ax ¼ b, satisfy the following relation for the residual at the
þ np þ nÞ.
ith iteration
ri 2 spanfr0 ; Ar0 ; A2 r0 ; . . . ; Ai1 r0 g ¼ ðAÞr0 ;  2 P i ; (13)
where ri ¼ b  Axi is the residual at the ith iteration and
P i is the set of polynomials of degree at most i with value 1
at the origin. Minimal residual methods such as MINRES
[Paige and Saunders, 1975] or GMRES [Saad and Schultz,
1986] try to compute a polynomial  such that
kri k ¼ min 2P i kðAÞr0 k:
Further, if A is diagonalizable with A ¼ VV1 , we then
have the following tight bound [Benzi et al., 2005]
kri k  kr0 kcondðVÞmin 2P i max j jð j Þj:
A bound of the type in equation (15) provides intuition of
how the eigenvalue distribution influences the worst case
behavior. For example, if all the eigenvalues are clustered
tightly around a single point, that is far away from the ori-
gin, we can expect rapid convergence. Wide-spread eigen-
values, however, would lead to slow convergence. In order
to keep the number of iterations reasonable, sometimes
efficient preconditioners are necessary, which transform the
linear system into another linear system that has more favor-
able properties. For more details regarding the convergence
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
of the Krylov subspace methods, the interested reader is
referred to the following references [Benzi et al., 2005;
Golub and Van Loan, 1996]. Generally speaking, a good
preconditioner is a reasonable approximation to the inverse
of a matrix, should be cheap to construct and to apply, and
its application either clusters the eigenvalues or reduces the
condition number of the matrix, or both. We observed that in
certain instances, the number of iterations required for con-
vergence of our system were quite large and thus, we devise
a preconditioner to reduce the number of iterations. We will
show that the choice of this particular preconditioner clusters
several of the eigenvalues around 1, resulting in fewer
iterations.
(12) [21] We make the following assumptions while deter-
mining the cost of our algorithm: the matrix R is diagonal
and the cost of matrix-vector product involving H and HT
is given by
if H is sparse then
nonzeros that H has.
[22] The matrix A is never explicitly constructed, since Q
is never constructed explicitly. The cost for setting up A is
eliminated but the cost of setting up Q, using the H-matrix
approach and forming U, is Oð
the block rank used in the H-matrix approach. The action of
the matrix  on a vector x of size n, written as q ¼ x
can be computed in three steps: compute the adjoint of the
measurement operator z HT x, computing the matrix-
vector product w Qz approximately using the H-matrix
approach described in section 3.1 and finally compute,
q Hw þ Rx. The cost of computing one matrix-vector
product of matrix A involves the following operations: (1)
multiplication with Q, (2) multiplications with H, HT and R
and finally, (3) multiplication with  and T on appropriate
sized vectors. The cost per matrix-vector product of A is
Oðkmlog m þ 2
3.3. Preconditioner
[23] In several applications, the covariance matrix Q has
several eigenvalues that are practically zero. For such
cases, the covariance matrix Q can be approximated for the
purposes of computing a useful preconditioner, by a low-
rank decomposition, by retaining only the r largest eigen-
values and corresponding eigenvectors
Q Vr r VTr ; (16)
(14)
where the columns of Vr contain the r eigenvectors corre-
sponding to the largest eigenvalues that form the diagonals
of the matrix r . Of course, if we compute nearly all the
eigenvalues and eigenvectors of Q, then we can use the
(15) eigenvalue decomposition as a surrogate for the matrix Q
itself. However, this is extremely expensive in practice. To
compute a preconditioner, we only need an approximation
to Q which need not necessarily be very accurate. We have
found that the number of eigenvalues and eigenvectors that
need to be retained to compute such a low-rank representa-
tion is often on the order of 100 and is nearly independent
of the dimension of the problem. The relative error in con-
structing such an approximation is
def kQ  Vr r VTr k2 rþ1
r ¼ ¼ : (17)
kQk2 1
4 of 13

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
We use a matrix-free eigenvalue solver, for example the
Krylov-Schur method to compute the dominant eigenvalues
and eigenvectors of the covariance matrix Q, with matrix-
vector products accelerated in the usual fashion using Hier-
archical matrices, described in section 3.1.
[24] For such matrices, we can use the above observation
to construct an approximate inverse for the matrix , in the
steps listed below. As before, we make the following
assumptions : the matrix R is diagonal and the cost of
matrix-vector product involving H and HT is given by
. 2005], which is a related package to PETSc. We use the
[25] 1. Compute the low-rank representation of the ma-
trix Q to obtain Q Vr r VTr . The number of iterations
required to compute the dominant portion of the spectrum
using the Krylov-Schur method is independent of the size
of the system and the work required for computing this
low-rank representation is Oðrkmlog mÞ, where k is the
block rank of the covariance matrix and r is the number of
eigenvalues and eigenvectors of Q that we compute.
[26] 2. Form the matrix
M ¼ R1=2 HVr r1=2 :
The cost of this step is Oð
r þ rÞ. solver such as Gaussian elimination. However, it should be
[27] 3. Compute the singular value decomposition
(SVD) of the matrix M as defined above
M ¼ UVT ;
which can be computed in Oðnr2 Þ assuming that n > r.
[28] 4. Use Sherman-Morrison-Woodbury update [Hager,
1989] to compute the inverse of
ðMMT þ IÞ1 ¼ I  UDr UT ;
 
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
the choice of the Krylov subspace method. The matrix A,
as defined in equation (11), is symmetric but indefinite.
This precludes the use of Conjugate Gradient algorithm.
Thus, we use a restarted GMRES (50) solver for the linear
system of equations (9). We note that other possible
choices are either using MINRES [Paige and Saunders,
1975] or Transpose Free QMR [Freund, 1993]. The low-
rank representation of the matrix Q, which is required in
section 3.3 is computed using SLEPc [Hernandez et al.,
default Krylov-Schur option in SLEPc to compute the dom-
inant eigenvalues of Q.
3.5. Other Approaches
[34] For any iterative solver applied to an arbitrary linear
system of equations, it is difficult to guarantee convergence
to a desired tolerance in a reasonable number of iterations.
We briefly discuss an alternative approach to solving the sys-
tem of equations (9). Instead of using an iterative solver, we
can compute QHT using fast matrix-vector products using
the H-matrix approach. Subsequently, we form the dense
matrix  and solve the system of equations (9) using a direct
noted that this process can be expensive for a large number of
measurements n because forming QHT requires Oðmnlog mÞ
operations. Moreover, solving the system of equations has a
complexity of Oðn þ pÞ3 which can be rather high both in
terms of storage and computation time for large n. By com-
parison, the iterative solver when it converges in a few itera-
tions, requires far fewer matrix-vector products involving Q
(in practice, much smaller than the number of measurements)
than the direct solver approach. Therefore, we recommend
using this approach only when the number of measurements
is small or if the iterative solver fails to converge to a desired
2i tolerance.
where Dr ¼ diag , where i ; i ¼ 1; . . . ; min fn; rg
1 þ 2i
are the singular values of M. The only cost is forming Dr 4. Uncertainty Quantification and Conditional
which is OðrÞ. Realizations
[29] 5. Compute the approximate inverse of , which we [35] A commonly used strategy [Kitanidis, 1995; Zanini
denote by ^ 1 as
et al., 2009] to quantify the uncertainty associated with the
^ 1 ¼ R1  R1=2 UDr UT R1=2 :
 estimate of the solution, is via computing conditional real-
izations. This method avoids the computation of the poste-
[30] This can be computed in OðnrÞ. rior covariance matrix, which is expensive for large-scale
problems.
[31] The cost of applying  ^ 1 to a vector x is simply
OðnrÞ and is thus, independent on the number of 4.1. Conditional Realizations
unknowns m. [36] We first generate unconditional realizations vu 
[32] Finally, we show how to use  ^ 1 to construct a pre- N ð0; RÞ and su  N ð0; QÞ, which are realizations corre-
conditioner for our system. We use the block diagonal form sponding to the pdf of the noise and the prior probability of
[Benzi et al., 2005], which is commonly used in precondi- the unknown parameters s. The conditional realizations
tioning matrices arising from saddle point problems. Thus, [Kitanidis, 1995; Zanini et al., 2009] are generated by first
the preconditioner to the matrix defined in 11 is solving the linear inversion system with a modified right
" # hand side,
^ 1

P 1
¼ ^ 1 ¼ T 
S ^ 1 : (18) ! ! !
1
^
S HQHT þ R HX ^ y  Hsu þ vu
¼ ; (19)
3.4. Implementation Details ðHXÞT 0 ^ 0

[33] We have implemented the above algorithm in Cþþ,

and then, the conditional realization can be computed by
using PETSc [Balay et al., 1997, 2008, S. Balay et al.,
the following equation
PETSc, available at http://www.mcs.anl.gov/petsc, 2009]
which is a suite of data structures and routines for scalable
solutions in scientific applications. Finally, we comment on ^s c ¼ su þ X^ þ QHT ^ : (20)

5 of 13

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
Note that the system of equations (19) have the same left
hand side as the original system of equations (9), with only
modification in the right hand side. Thus, the implementa-
tion of conditional simulations can be done without much
recoding.
4.2. Generating Unconditional Realizations
[37] We now address the issue of generating uncondi-
tional realizations vu  N ð0; RÞ and su  N ð0; QÞ. In most
applications, we assume the noise to be uncorrelated, that is,
we assume that R is a diagonal matrix. Thus, we can easily
generate the unconditional realization vu ¼ R1=2 x, where
x  N ð0; IÞ. However, generating the unconditional realiza-
tion su is much more expensive because Q is dense. A proce-
dure that is often mentioned in literature, is to form the
Cholesky decomposition of Q ¼ FT F, where F is an upper
triangular matrix. We can then compute su ¼ Fx, where
x  N ð0; IÞ. But since Q is dense, Cholesky Factorization,
which scales as Oðm3 Þ, where m is number of unknown pa-
rameters, becomes very expensive to compute. Moreover,
Cholesky Factorization requires explicitly storage of Q,
which is not feasible.
[38] We will describe a more efficient approach sample,
which relies on a polynomial approximation to the square
root and fast matrix-vector products. The only difference
I, with
> 0, so that the matrix
between the two approaches is that the manner in which the
; max þ
 and x is
matrix-vector products are computed: our approach uses the
H-matrix approach, whereas their approach uses FFT based
. The use
methods. To sample from N ð0; QÞ, we need to compute
su ¼ Q1=2 x x  N ð0; IÞ:
This meets our requirements because
E½su  ¼ 0 E½su sTu  ¼ Q1=2 E½xxT Q1=2 ¼ Q:
We adopt a procedure similar to that of Dietrich and
Newsam [1995], that exploits the ability to form fast matrix-
vector products involving the matrix Q. The method tries to
approximate the square root of a matrix, using Chebyshev
matrix polynomials that can be generated using a three term
recurrence series. The explicit computation of Q1=2 is not
necessary. The quantity of interest is su ¼ Q1=2 x, where x 
N ð0; IÞ and whose polynomial approximation only requires
matrix-vector products involving Q and can be computed as
X
L
su ai xi ;
i¼0
where
x0 ¼ x x1 ¼ ðda Q þ db IÞx xnþ1 ¼ 2ðda Q þ db IÞxn  xn1:
The coefficients ai are the representation of the square root
function in Chebyshev polynomial basis in the domain
½ min ; max  [Mason and Handscomb, 2003]. The coeffi-
cients da and db are given by
2 max þ min
da ¼ db ¼  ;
max þ min max  min
6 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
where, max and min are the largest and smallest eigenval-
ues of the matrix Q. Thus, in order to generate uncondi-
tional realizations su  N ð0; QÞ, we need access to only the
upper and lower eigenvalues and fast matrix-vector prod-
ucts involving the covariance matrix Q, which we compute
using the H-matrix approach detailed in section 3.1. The
extreme eigenvalues can be computed using a matrix-free
Krylov-Schur algorithm, which we access via SLEPc pack-
age. In the appendix A, we extend the analysis of Dietrich
and Newsam [1995] to derive an error bound for the approx-
imation of the square root of the matrix using Chebyshev
polynomials to include approximation errors due to the
Hierarchical matrix approach.
[39] In practice, the number of terms in the Chebyshev
series necessary to obtain the desired tolerance of approxi-
mation, grows proportional to the square root of the largest
eigenvalue [Dietrich and Newsam, 1995]. Furthermore,
whenever the smallest eigenvalue of Q tends to zero, the
convergence of the Chebyshev series will be slow, which is
characteristic of any polynomial function approximation to
the square root, and is because pthe
ffiffiffi origin is the singularity
point of the first derivative of x. A workaround for this
problem is to use the so-called ‘‘nugget effect,’’ which
involves adding a diagonal term to the covariance matrix
and has the form Q þ
pffiffihas
ffi
a spectrum that lies in the interval ½
approximated in the very same interval. This is the
ffi same as
pffiffiffiffiffiffiffiffiffiffi
computing the Chebyshev expansion for x þ
of the nugget effect makes the number of terms in the Che-
byshev expansion small, which makes the algorithm more
efficient but adds some small-scale variability that is often
negligible, in practice.
5. Numerical Experiments
[40] We now demonstrate the performance of our algo-
rithm on a few applications. We apply it to two applica-
tions: interpolation involving noisy measurements on a
standard test function and contaminant source identifica-
tion. In the first application, the measurement operator is
extremely sparse and in the second application, the mea-
surement operator is not constructed explicitly but we only
have access to matrix-vector products of H and HT . The
above method was implemented in Cþþ using PETSc
libraries for the linear solvers. All test cases were run on a
workstation with Intel Xeon E7540 2GHz processor with
128 GB RAM, running Ubuntu 11.04.
5.1. Application: Interpolation
[41] To test our algorithm, we use a function introduced
by Franke [Franke, 1982] which is frequently used as a test
case for testing interpolation algorithms.
3 2 2 3 2 2
Fðx1 ; x2 Þ ¼ eðð9x1 2Þ þð9x2 2Þ Þ=4 þ eð9x1 þ1Þ =49ð9x2 þ1Þ =10
4 4
1 ðð9x1 7Þ2 þð9x2 3Þ2 Þ=4 1 ð9x1 4Þ2 ð9x2 7Þ2 :
þ; e  e
2 5
(21)
Figure 1 shows the plot of Franke’s function, which we
consider as a true field for this problem (see also Figure 2).
The setup for our problem is as follows. We pick m points

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
Figure 1. Plot of Franke’s function defined in equation
(21) in the domain ½0; 12 .
uniform randomly distributed in the domain ½0; 12 . Of
these m points, we pick at random n points and evaluate the
function Fð; Þ as defined in equation (21) and to which,
we add 0:01% noise. Thus, in this case both the measure-
ments and the evaluation points are irregularly distributed
which precludes the use of FFT based methods. The mea-
surement operator H for the interpolation problem is just
composed of entries 0 and 1, and the nonzero entries occur
when the measurement location coincides with the location
of the unknown. Thus, H is extremely sparse and only con-
tains n nonzero entries. We solve the system of equations
(9) and then evaluate the function at the m points. We make
the following modeling choices. We pick X ¼ ½1; . . . ; 1T ,
R ¼ 104 I and the covariance matrix Q corresponding to
the following covariance kernel
KðrÞ ¼ ð1 þ r þ ðrÞ2 =3Þexp ðrÞ; (22)
which is a member of the Matérn covariance family. The
reason for this choice will be explained shortly. We pick
Figure 2. (left) krFk for Franke’s function and (right) reconstruction of the derivative kr^sk from
geostatistical approach for m ¼ 3600 and n ¼ 200. The L2 relative error is 0.1965.
7 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
1 ¼ 0:5  L, for most of our calculations, where L is the
maximum domain size. For the construction of the H-matrix
QH , we pick the error to be " ¼ 109 .
[42] Table 1 summarizes the performance of our iterative
algorithm. For a given number of points m, we pick n to be
1%, 5% and 10% of m. With the increase in the number of
measurements, the error decreases. This is what we should
expect because we have more information about our sys-
tem. The number of iterations seems to increase linearly
with the number of measurements. While the number of iter-
ations without the preconditioner are not too large, with
increasing large number of measurements, the number of
iterations could be large in practice. To alleviate this prob-
lem, we use the preconditioner that we proposed in section
3.3, with the number of eigenvalues and eigenvectors r we
use for the preconditioner to be 50. Refer to Table 2 for the
results of the interpolation problem using the preconditioner.
[43] Now, we study the convergence of the iterative
scheme for various covariance kernels. We choose a few
different covariance kernels, the first four from the Matérn
covariance family and two others which are frequently
used in radial basis function interpolation. The parameters
were chosen such that they produced the neither the error,
as compared to the true solution, nor the number of itera-
tions were too high. We pick n ¼ 1000 and m ¼ 100;000
and tabulate the number of iterations for each covariance
kernel along with the L2 relative error. The results are
shown in Table 3. We observe that the best convergence
among the covariance kernels in the Matérn family is the
one with the second-order polynomial prefactor, for a toler-
ance of L2 relative error less than 10% and so we use it
henceforth. Even though the radial basis function covari-
ance kernel seem to perform better, they are not positive
definite but conditionally positive definite and we need spe-
cial care to treat them, especially for generating uncondi-
tional realizations.
[44] The geostatistical approach, can also be used to esti-
mate the derivative of the function from noisy measure-
ments of the function. To see this, we consider the best
approximation ^sðÞ to the to the underlying function that we
are trying to estimate sðÞ. After computing ^ and ^ as the
 19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH W05522

Table 1. Performance of the Iterative Scheme for the Interpola- Table 3. Performance of the Iterative Scheme for Various Covari-
tion Problem Without Preconditionera ance Kernelsa
Unknowns Measurements Iterations Relative Error Kernel K(r) Iterations Relative Error

10,000 100 33 0.0920 exp(r) 84 0.00447

10,000 500 43 0.0345 (1 þ r) exp(r) 153 0.0011
10,000 1000 49 0.0242 (1 þ r þ (r)2/3) exp(r) 48 0.022
50,000 500 44 0.0344 exp(r2) 35 0.1122
50,000 2500 81 0.0149 log r 23 0.0756
50,000 5000 95 0.0087 r2 log r 179 0.0012
100,000 1000 49 0.0225 a
100,000 5000 95 0.0087 Here r ¼ kx  yk and  ¼ 2.
100,000 10,000 95 0.0088
500,000 5000 95 0.0225
500,000 10,000 133 0.0052
be 100. For this problem we choose m ¼ 90;000 and
500,000 50,000 239 0.0015 n ¼ 1000. We used the same measurements and measure-
ment locations for generating all three conditional realiza-
a
A relative tolerance of krk k=kr0 k < 106 was used as termination cri-
teria for the iterative solver.
tions. The results are presented in Figure 3.
5.2. Application: Contaminant Source Identification
solution of the system of equations (9), we can compute [46] We now introduce a large-scale linear inverse prob-
^sðÞ as lem that is governed by the transient advection-diffusion
transport model. The inverse problem in this case is then to
X
p X
m determine the initial concentration of the contaminant given
^sðÞ ¼ fk ðÞk þ Kðk  xi kÞðHT ^ Þi : (23) noisy measurements taken in time from sensors distributed
k¼1 i¼1
spatially, assuming a suitable prior covariance matrix and
the PDE model that we use to simulate the transport.
Therefore, we can compute the approximation to the rsðÞ,
the gradient of the underlying function by computing r^sðÞ 5.2.1. Problem Setup
using the following formula, [47] Consider the transient advection diffusion problem,
that describes the evolution of the contaminant
X
p Xm
x  xi
r^sðxÞ ¼ rfk ðxÞk þ K 0 ðkx  xi kÞ ðHT ^ Þi : @u
k¼1 i¼1
kx  xi k  Du þ v:ru ¼ 0;   ½0; T 
@t
(24)
u ¼ u0 ;   ft ¼ 0g
; (25)
It is easy to verify that this generating function is asymptoti- u¼0 @D  ½0; T 
cally smooth as well. Therefore, it can be approximated @u
using a block wise low-rank matrix using the procedure out- ¼0 @N  ½0; T 
@n
lined in section 3.1.
[45] We conclude this application by presenting results where u is the contaminant concentration, u0 is the initial
for the conditional realizations as outlined in section 4. We contamination field that we would like to determine, D is
first generate unconditional realizations ^s u and v using the the diffusion coefficient, v the known velocity field and T is
procedure outlined in section 4.2. We compute the extreme the final time.
eigenvalues using the Krylov-Schur algorithm accessed via [48] The forward problem is discretized using linear fi-
the SLEPc package [Hernandez et al., 2005]. We take the nite elements. The time integration is carried out by Crank-
number of Chebyshev polynomials in the approximation to Nicholson method. At each time step, the resulting system
of equations was solved using transpose free QMR
(TFQMR) with ILU preconditioner.
Table 2. Performance of the Iterative Scheme for the Interpola- [49] We work with dimensionless quantities throughout.
tion Problem Preconditionera We assume that the domain of the problem is ½0; L2 and
Unknowns Measurements Iterations Relative Error the velocity field v is chosen to be a Poiseuille flow over
this domain, that is vx ¼ v0 ðL=2 þ yÞðL=2  yÞ=ðL=2Þ2 ;
10,000 100 6 0.085 vy ¼ 0, so that the Peclet number based on the centerline
10,000 500 8 0.040
10,000 1000 9 0.023
velocity is Pe ¼ Lv0 =D. The molecular diffusion D is cho-
50,000 500 8 0.030 sen to be 1. Observations are collected at distinct locations
50,000 2500 13 0.0145 xj ; j ¼ 1; . . . ; nm and are uniformly distributed in the do-
50,000 5000 17 0.009 main and n t measurements are computed evenly in ½0; T ,
100,000 1000 9 0.0249 so that in all we have n ¼ nm  nt measurements. Further-
100,000 5000 18 0.0085
100,000 10,000 23 0.0055 more, in the rest of the section, we pick L ¼ T ¼ 1, the
500,000 5000 17 0.0093 centerline fluid velocity to be v0 ¼ 1 and the time step for
500,000 10,000 22 0.0057 the transient advection diffusion problem to be t ¼ 0:05.
500,000 50,000 31 0.0016 Figure 4 shows the reconstruction of the true field which
a
For the preconditioner, we choose r ¼ 50. A relative tolerance of we assume to be 2exp ð30 kx  xckÞ which is centered at
krk k=kr0 k < 106 was used as termination criteria for the iterative solver. xc ¼ ð0:25; 0:5Þ. The unknowns are discretized on a

8 of 13

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
Figure 3. Conditional realizations for the interpolation
problem with m ¼ 90;000 and n ¼ 1000.
100  100 grid in space and 20 time steps, in the domain
with L ¼ T ¼ 1. We pick v0 ¼ 1. The measurements are
collected in a 10  10  10 grid, i.e., nm ¼ 100; nt ¼ 10.
The relative error in the reconstruction was 0.063.
9 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
[50] We pick an irregular grid to demonstrate the utility
of our algorithm. Suppose if we had reason to believe that
our source is located between ½0:125; 0:375  ½0:375;
0:625 and we want to resolve the location of our source
better. Instead of refining the grid everywhere in the do-
main, we could just refine the grid by a factor of 2, in the
desired region of interest. This is the principal advantage
that we have over FFT based methods that rely on regular
equispaced grids. an example of the grid that we used is
given in Figure 5.
[51] The advection diffusion problem described in equa-
tions (25), after discretization can be written as the linear
system of equations Ku ¼ Tu0 . Here K is the discretized
time-dependent convection diffusion operator and T is the
operator that maps the initial conditions u0 into space time.
The measurement operator, H can be constructed as H ¼
BK1 T, where B is the restriction operator that incorpo-
rates the sensor measurements. Of course, it may not be
feasible to explicitly construct this matrix. Instead, we can
compute matrix-vector product Hx in the following steps:
converting the initial conditions to space time z Tx,
solving the time dependent advection diffusion equation
w K1 z and finally computing the measurements
q Bw. The transpose of the measurement operator times
a vector can be computed in an analogous fashion. As
before, we pick X ¼ ½1; . . . ; 1T , R ¼ 104 I and the covar-
iance matrix Q corresponding to the following covariance
kernel in equation (22) with 1 ¼ 0:1  L, for all of our
calculations. For the construction of the H-matrix QH , we
pick the error to be  ¼ 109 .
5.2.2. Performance of the Iterative Method
[52] In this subsection, we study the convergence of the
iterative method for different grid sizes, number of meas-
urements and different covariance kernels. In all the cases
that we will discuss, perfect reconstruction will not be pos-
sible because of a nonzero regularization term R and a
fixed mesh size. In practice, the relative L2 error in our
reconstruction is usually less than 10%. Without using the
preconditioner, the number of iterations, even for moder-
ately large problem sizes, well exceed 300 iterations. This
number is unacceptably large in practice because the num-
ber of times the time-dependent advection problem is
solved is more than double the number of iterations, i.e.,
greater than 600. Thus, we will present results only for the
preconditioned case.
[53] First, we study the convergence of the iterative
scheme with increase in the number of sensors. With an
increase in the number of the sensors, the problem becomes
more well-posed because of new information provided by
the additional sensors. Thus, one might expect a quicker
inversion. However, this does not seem to be the case. The
iterative solver needs to perform more work to incorporate
additional information.
[54] In Table 4, we summarize the performance of the
iterative scheme. For the preconditioner, we use r ¼ 100
eigenvalues and the relative error in the low-rank approxi-
mation to Q, (see equation (17)) is r  105 . We immedi-
ately observe that the accuracy of the inversion increases
with the number of measurements. This is to be expected
since with increase in number of measurements, the amount
of information entering the calculations increases. The
 19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH W05522

Figure 4. Reconstruction of a Gaussian initial condition 2exp ð30kx  xc kÞ which is centered at

xc ¼ ð0:25; 0:5Þ. The unknowns are discretized on a 100  100 grid in space and 20 time steps, in the do-
main with L ¼ T ¼ 1. We pick v0 ¼ 1. The measurements are collected in a 10  10  10 grid, i.e.,
nm ¼ 100; nt ¼ 10. (left) Reconstructed field and (right) true field. The relative error in the reconstruc-
tion was 0.063.

number of iterations required to converge to the desired tol-
erance remains bounded for moderate number of measure-
ments 1000 but the iterations number of diverge when the
number of sensors were increased dramatically. Under such
circumstances, we increase the number of dominant eigen-
values and eigenvectors, r we use to approximate Q in the
preconditioner. We also observe that for a given number of
measurements, the number of iterations it took to converge
does not depend on the number of unknowns, and in this
sense, the convergence is essentially grid independent. This
is a very desirable property because the algorithm scales to
arbitrarily large number of unknowns. In all, the maximum
problem size that we solved for 9  104 inversion parame-
ters and 1.8 million space time unknowns.
[55] We now consider the problem with a large number
of measurements. We pick the number of sensors to be
25  25 and since we take nt ¼ 10 measurements in time,
altogether we have 6; 250 measurements. We increase r,
the number of eigenvalues and eigenvectors of Q that we
compute and study the convergence of the iterative method
on this problem. With increasing r, the approximation of
the eigendecomposition to the matrix Q improves and this
results in a better preconditioner because it approximates
the matrix A better. The downside is that our setup costs of
the preconditioner also increases significantly because not
only do we have to compute more eigenvalues and eigen-
vectors of Q but also, we need to multiply with H, r num-
ber of times(see step 2 in section 3.3), which is expensive
because each matrix-vector product with H involves solv-
ing the transient advection-diffusion equation. We set the
maximum number of iterations to be 300. In Table 5 we list
the number of iterations it took for a given r and the result-
ing L2 relative error of the computation. We observe that

Table 4. Performance of the Iterative Scheme for the Contami-

nant Source Identification Problema
Sensors Unknowns Iterations Relative Error

88 100  100 30 0.0941

88 200  20 32 0.0949
88 300  300 33 0.0953
10  10 100  100 38 0.0669
10  10 200  200 39 0.0675
10  10 300  300 41 0.0679
12  12 100  100 136 0.0495
Figure 5. An example of the grid that is used for the lin- 12  12 200  200 196 0.0503
ear inversion. The number of grid points are 10;000 and the 12  12 300  300 200 0.0500
grid is refined in the region x ¼ ½0:125; 0:375 and a
In each case the number of time measurements were nt ¼ 10, with L ¼
y ¼ ½0:375; 0:625. T ¼ 1 and Dt ¼ 0.05. For the preconditioner, we used r ¼ 100.

10 of 13
 19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH W05522

Table 5. Performance of Iterative Scheme With Increasing r for
Grid Size 100  100 and Number of Sensors 25  25, so That
Number of Measurements are 6250
r Iterations r Relative Error
a 5
129 300 2.45  10 0.0148
201 196 4.76  106 0.0146
278 96 1.75  106 0.0145
355 41 7.09  107 0.0144
a
Reached the maximum number of iterations, 300, without converging
to the desired solver tolerance.
with increasing r the number of iterations that it takes to con-
verge to the desired tolerance d, in this case we use the same
condition that we do for interpolation krk k=kr0 k< 106 ,
decreases. Therefore, we can view this scenario as a tradeoff
between setup time and computation time. If we have to
solve the linear inverse problem for multiple right-hand sides,
i.e., for multiple measurements, then it is worthwhile to com-
pute the preconditioner with larger r that will result in faster
convergence.
5.2.3. Spectrum of the Preconditioned Matrix
[56] To understand the working of the preconditioner,
we plot the spectrum of the preconditioned operator. Since
it is extremely expensive to compute the full spectrum of
the operator, we restrict ourselves to the case when the
number of sensors are 10  10 and the number of spatially
varying unknowns is at most 100  100. In setting up the
preconditioner, we explicitly compute 100 dominant eigen-
values and their corresponding eigenvectors. Even though
we request only a 100 eigenvalues from the SLEPc pack-
age, in practice a few more eigenvalues converge (135) in
the Krylov-Schur iterations and so we use these in our
preconditioner as well. In Figure 6, we plot the absolute
eigenvalues of the preconditioned operator. As can be seen
in Figure 6, most of the eigenvalues seem to cluster around
1 and the spectrum is more or less independent of the
number of unknowns. This explains the grid-independent
convergence that we observe, for the problem sizes that we
considered. We observed a similar trend for different num-
ber of sensors as well.
5.2.4. Comparison With FFT Based Methods on
Regular Grids
[57] We conclude this application with the comparison
of our algorithm with a comparison of our algorithm with
an FFT based approach on regular grids. For the setup of
the problem we pick 10  10 sensors with nt ¼ 10 and
t ¼ 0:05. The regular grid is constructed for a problem
size m ¼ 300  300. Everything else remains the same
from section 5.2. For comparison purposes we pick the fol-
lowing three strategies.
[58] 1. Strategy 1 is that the matrix-vector products are
computed using the FFT based method. We form QHT
first, then form the matrix A and solve the system of equa-
tions using a direct solver.
[59] 2. Strategy 2 is that the matrix-vector products are
computed using the FFT based method. The rest of the
algorithm is the same as the one described in section 3.
[60] 3. Strategy 3 is that the authors use the algorithm
described in section 3, with matrix vector products using
H-matrix approach.
[61] The first strategy is the direct implementation of FFT
based methods that is prevalent in the literature. We note
that it can be expensive because it involves constructing H,
which we have already discussed can be extremely expen-
sive. Thus, we propose a second alternative which is to use
our original algorithm but modify it such that the matrix-
vector products are computed using FFT based approaches
as opposed to the H-matrix approach. These two strategies
were implemented using MATLAB. This might not be a fair
comparison because we are comparing two different pro-
gramming languages and programming styles.
[62] Table 6 compares the time taken to setup and solve
the system of equations for the three different strategies. As
expected, the bulk of the computational time in Strategy 1

Figure 6. (left) Eigenvalues of the preconditioned operator. (right) Plot of #f i : j i  1j > xg against
x for all eigenvalues. For these plots, we assumed the number of observations to be 10  10  10, the
number of unknowns varied from 20  20 to 100  100. The covariance kernel is defined in equation (22).

11 of 13

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
Table 6. Comparison of the Three Strategies on a Regular Grid
300  300a
Strategy Setup Q Setup  Setup Preconditioner Solve
1 0.6 10559.64 – 12.298
2 0.6 – 1360.62 811.61
3 77.61 – 2917.95 3265.5
a
Time measured in seconds
is spent in setting up the system, whereas the time required
for solving the system is very small because of very effi-
cient direct methods and indeed it is the slowest of the three
strategies. We expect Strategy 2 to outperform, because it
not only leverages the fact that there is no need to form H
but also FFT based matrix vector products are much faster
than H-matrix matrix-vector products. In fact, it is nearly
twice as fast, which can be significant savings in time for
much larger problem sizes. FFT based methods have pro-
ven to be extremely fast, in no small part due to the avail-
ability of finely tuned libraries to perform FFT operations.
Even though our algorithm is slower, with a sufficiently
good implementation, it would be competitive with FFT
based methods even on regular grids. Of course, since
FFT based methods are not easily applicable on irregular
grids, and thus, the default choice should be to use our
algorithm.
6. Conclusion
[63] We have described an efficient numerical method to
compute the best estimate for a linear under determined set
of equations using the Stochastic Bayesian approach for
geostatistical applications. We emphasize here, the general-
ity of our formulation, in the sense that it is capable of han-
dling a wide variety of generalized covariance functions
Kð; Þ with a minimum amount of recoding and the fact
that this is applicable to situations in which the unknowns
can be distributed on an irregularly spaced grid. Thus,
unlike FFT based methods, we are not restricted to regular,
equispaced grids. However, FFT based methods are hard to
beat for regular grids because of the availability of finely
tuned software. Therefore, if it is known a priori that the
grid is regular, we recommend using FFT to accelerate the
matrix-vector products instead of the H-matrices and use
the rest of the algorithms described in section 3.
[64] The key idea was exploiting the Hierarchical matrix
formulation, which allows to compute fast matrix-vector
products involving the covariance matrix Q approximately.
Our method scales well up to measurements of size 5  104
and unknowns up to size 5  105 on irregular grids. Under
some circumstance, the number of iterations required to
converge to the desired tolerance become unacceptably
large and we need to use the preconditioner that we pro-
posed in section 3.3. The setup cost can be quite large for
large problems because it involves computing the dominant
spectrum of the covariance matrix Q. Further, exploiting a
result from polynomial approximation theory, we approxi-
mated the square root of the covariance matrix Q using Che-
byshev polynomials to generate unconditional realizations,
which were then used to generate conditional realizations
via a slight modification in the system of equations (9). The
extension to 3-D is also straightforward.
12 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
Appendix A: Approximation Error in
Chebyshev Matrix Polynomial Approach to
Computing Square Root of an H-Matrix
[65] In section 4.2, we described an algorithm for approx-
imating the squareroot of a matrix Q, using Chebyshev ma-
trix polynomials. We further, assumed that instead of using
the matrix, we replace it via its approximation QH . We
now compute the result of approximating the square root of
a matrix using Chebyshev matrix polynomials and approxi-
mating the matrix itself using the H-matrix approach.
[66] We assume that the matrix Q is symmetric positive
definite, with Q ¼ VVT , where V is the matrix of the
eigenvectors, whereas D is the matrix whose diagonal
entries contain the eigenvectors of the matrix Q, arranged
in a decreasing order. Further, we know that Qk ¼ Vk VT
and pnc ðQÞ ¼ Vpnc ðÞVT
[67] The result of n c terms of the Chebyshev matrix poly-
nomial, we can approximate the function as follows
pffiffiffi
x ¼ pnc ðxÞ þ ðxÞ;
where pnc ðxÞ is a polynomial term corresponding to the
Chebyshev series and ðxÞ is the remainder. Therefore,
Q1=2 ¼ Vðpnc ðÞ þ DÞVT ¼ pnc ðQÞ þ VDVT
0 1
ð 1 Þ
B C
B ð 2 Þ C
B C
B C
D¼B C:
B .. C
B . C
@ A
ð m Þ
As a result of using QH instead of Q, i.e., by introducing
the Hierarchical matrix approximation, we are computing
pnc ðQH Þ instead of pnc ðQÞ. The resulting error in this
approximation is given by
Q1=2  pnc ðQH Þ ¼ pnc ðQÞ  pnc ðQH Þ þ VDVT :
Therefore,
kQ1=2  pnc ðQH ÞkF  kpnc ðQÞ  pnc ðQH ÞkF þ kVDVT kF
Xm 1=2
 LkQH  QkF þ ð j Þ2
j¼1
pffiffiffiffi
 "LkQkF þ m max 1jm ð j Þ
pffiffiffiffi pffiffiffiffi
 "L m max þ mðnc ; max Þ
(A1)
In the third step, we have used the inequality from equation
(12) which gives us the relative error in the Frobenius norm
of the true covariance matrix Q by the H-matrix QH . Here
L is the Lipschitz constant (see for example, Wihler [2009])
of the function pnc ðÞ and ðnc ; max Þ is the maximum error
in the polynomial approximation due to nc terms and the
maximum eigenvalue is max , which is computed in the
work of Dietrich and Newsam [1995]. Thus, in order that
the error due to approximation by Hierarchical matrices is

W05522 SAIBABA AND KITANIDIS: EFFICIENT GEOSTATISTICAL APPROACH
consistent with the error due to truncation of the Chebyshev
polynomial approximation to the square root of the matrix,
the criterion for choosing  is
ðnc ; max Þ
 : (A2)
L max
In practice,  ¼ 109 easily satisfies this criteria even for
small values of ðnc ; max Þ.
[68] Acknowledgment. The authors were supported by NSF Award
0934596, Subsurface Imaging and Uncertainty Quantification.
References
Akcelik, V., G. Biros, O. Ghattas, K. R. Long, and B. B. Waanders (2003),
A variational finite element method for source inversion for convective-
diffusive transport, Finite Elem. Anal. Des., 39(8), 683–705.
Akcelik, V., G. Biros, A. Draganescu, J. Hill, O. Ghattas, and B. V. B.
Waanders (2005), Dynamic data-driven inversion for terascale simula-
tions: Real-time identification of airborne contaminants, in Proceedings
of the 2005 ACM/IEEE Conference on Supercomputing, p. 43, IEEE
Comput. Soc. Press, Washington, D.C.
Balay, S., W. D. Gropp, L. Curfman McInnes, and B. F. Smith (1997),
Efficient management of parallelism in object oriented numerical soft-
ware libraries, in Modern Software Tools in Scientific Computing, edited
by E. Arge, A. M. Bruaset, and H. P. Langtangen, pp. 163–202,
Birkhäuser Press, Boston, MA.
Balay, S., K. Buschelman, V. Eijkhout, W. D. Gropp, D. Kaushik, M. G.
Knepley, L. Curfman McInnes, B. F. Smith, and H. Zhang (2008), PETSc
Users Manual, Revision 3.0.0, Tech. Rep. ANL-95/11, Argonne Natl. Lab.,
Lemont, IL.
Barnes, J., and P. Hut (1986), A hierarchical O(N log N) force-calculation
algorithm, Nature, 324, 4.
Batu, V. (1998), Aquifer Hydraulics: A Comprehensive Guide to Hydro-
geologic Data Analysis, John Wiley, New York.
Bebendorf, M. (2000), Approximation of boundary element matrices,
Numer. Math., 86(4), 565–589.
Bebendorf, M. (2005), Hierarchical LU decomposition-based precondi-
tioners for BEM, Computing, 74(3), 225–247.
Bebendorf, M. (2008), Hierarchical Matrices: A Means to Efficiently Solve
Elliptic Boundary Value Problems, vol. 63, Lecture Notes Comput. Sci.
Eng., Springer, New York.
Bebendorf, M., and S. Rjasanow (2003), Adaptive low-rank approximation
of collocation matrices, Computing, 70(1), 1–24.
Benzi, M., G. H. Golub, and J. Liesen (2005), Numerical solution of saddle
point problems, Acta Numer., 14, 1–137.
Börm, S., L. Grasedyck, and W. Hackbusch (2003), Introduction to hier-
archical matrices with applications, Eng. Anal. Boundary Elem., 27(5),
405–422.
Cardiff, M., W. Barrash, P. K. Kitanidis, B. Malama, A. Revil, S. Straface,
and E. Rizzo (2009), A potential-based inversion of unconfined steady-
state hydraulic tomography, Ground Water, 47(2), 259–270.
Christakos, G. (1984), On the problem of permissible covariance and vario-
gram models, Water Resour. Res., 20(2), 251–265.
Dietrich, C. R., and G. N. Newsam (1995), Efficient generation of conditional
simulations by chebyshev matrix polynomial approximations to the sym-
metric square root of the covariance matrix, Math. Geol., 27(2), 207–228.
Flath, H. P., L. C. Wilcox, V. Akc
elik, J. Hill, B. van Bloemen Waanders, and
O. Ghattas (2011), Fast algorithms for bayesian uncertainty quantification
in large-scale linear inverse problems based on low-rank partial hessian
approximations, SIAM J. Sci. Comput., 33(1), 407–432.
13 of 13
19447973, 2012, 5, Downloaded from https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011WR011778 by Wuhan University, Wiley Online Library on [04/06/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
W05522
Franke, R. (1982), Scattered data interpolation: Tests of some method,
Math. Comput., 38(157), 181–200.
Freund, R. W. (1993), A transpose-free quasi-minimal residual algorithm
for non-hermitian linear systems, SIAM J. Sci. Comput., 14, 470.
Fritz, J., I. Neuweiler, and W. Nowak (2009), Application of FFT-based
algorithms for large-scale universal kriging problems, Math. Geosci.,
41(5), 509–533.
Golub, G. H., and C. F. Van Loan (1996), Matrix Computations, vol. 3,
Johns Hopkins Univ., Baltimore, Md.
Grasedyck, L., and W. Hackbusch (2003), Construction and arithmetics of
h-matrices, Computing, 70, 2003.
Greengard, L., and V. Rokhlin (1987), A fast algorithm for particle simula-
tions, J. Comput. Phys., 73(2), 325–348.
Hager, W. W. (1989), Updating the inverse of a matrix, SIAM Rev., 31(2),
221–239.
Hernandez, V., J. E. Roman, and V. Vidal (2005), Slepc: A scalable and
flexible toolkit for the solution of eigenvalue problems, ACM Trans.
Math. Software, 31(3), 351–362.
Kitanidis, P. K. (1995), Quasilinear geostatistical theory for inversing,
Water Resour. Res., 31(10), 2411–2419.
Kitanidis, P. K. (2007), Bayesian and Geostatistical Approaches to Inverse
Problems, in On Stochastic Inverse Modeling, Geophys. Monogr., vol.
171, ed. by L. Biegler, et al., pp. 19–30, AGU, Washington, D. C.
Kitanidis, P. K. (2010), Bayesian and Geostatistical Approaches to Inverse
Problems, pp. 71–85. John Wiley, New York.
Li, W., and O. A. Cirpka (2006), Efficient geostatistical inverse methods
for structured and unstructured grids, Water Resour. Res., 42, W06402,
doi:10.1029/2005WR004668.
Mason, J. C., and D. C. Handscomb (2003), Chebyshev Polynomials, CRC
Press, Boca Raton, FL.
Matheron, G. (1973), The intrinsic random functions and their applications,
Adv. Appl. Prob., 5(3), 439–468.
Michalak, A. M., and P. K. Kitanidis (2003), A method for enforcing pa-
rameter nonnegativity in bayesian inverse problems with an application
to contaminant source identification, Water Resour. Res., 39(2), 1033,
doi:10.1029/2002WR001480.
Nowak, W., and O. A. Cirpka (2006), Geostatistical inference of hydraulic
conductivity and dispersivities from hydraulic heads and tracer data,
Water Resour. Res., 42, W08416, doi:10.1029/2005WR004832.
Nowak, W., S. Tenkleve, and O. A. Cirpka (2003), Efficient computation of
linearized cross-covariance and auto-covariance matrices of interdepend-
ent quantities, Math. Geol., 35(1), 53–66.
Paige, C. C., and M. A. Saunders (1975), Solution of sparse indefinite sys-
tems of linear equations, SIAM J. Numer. Anal., 12(4), 617–629.
Rjasanow, S., and O. Steinbach (2007), The Fast Solution of Boundary Inte-
gral Equations. Mathematical and Analytical Techniques with Applica-
tions to Engineering, Springer, New York.
Rubin, Y., and S. S. Hubbard (2005), Hydrogeophysics, Springer, New York.
Saad, Y., and M. H. Schultz (1986), GMRES: A generalized minimal resid-
ual method for solving nonsymmetric linear systems, SIAM J. Sci. Stat.
Comput., 7(3), 856–869.
Stein, M. L. (1999), Interpolation of Spatial Data: Some Theory for Kriging,
Springer, New York.
Wihler, T. P. (2009), On the hölder continuity of matrix functions for nor-
mal matrices, J. Inequalities Pure Appl. Math., 10, 1–5.
Ying, L., G. Biros, and D. Zorin (2004), A kernel-independent adaptive fast
multipole algorithm in two and three dimensions, J. Comput. Phys.,
196(2), 591–626.
Zanini, A., and P. K. Kitanidis (2009), Geostatistical inversing for large-
contrast transmissivity fields, Stochastic Environ. Res. Risk Assess.,
23(5), 565–577.
P. K. Kitanidis and A. K. Saibaba, Institute for Computational and Math-
ematical Engineering, Jen-Hsun Huang Engineering Center, Stanford
University, 475 Via Ortega, Stanford, CA 94305-4121, USA. (arvindks@
stanford.edu)