STATUS OF AUTOMATIC CALIBRATION FOR HYDROLOGIC MODELS:

COMPARISON WITH MULTILEVEL EXPERT CALIBRATION

By Hoshin Vijai Gupta,1 Soroosh Sorooshian,2 and Patrice Ogou Yapo3

ABSTRACT: The usefulness of a hydrologic model depends on how well the model is calibrated. Therefore,
the calibration procedure must be conducted carefully to maximize the reliability of the model. In general,
manual procedures for calibration can be extremely time-consuming and frustrating, and this has been a major
factor inhibiting the widespread use of the more sophisticated and complex hydrologic models. A global opti-
mization algorithm entitled shuffled complex evolution recently was developed that has proved to be consistent,
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.

effective, and efficient in locating the globally optimal model parameters of a hydrologic model. In this paper,
the capability of the shuffled complex evolution automatic procedure is compared with the interactive multilevel
calibration multistage semiautomated method developed for calibration of the Sacramento soil moisture account-
ing streamflow forecasting model of the U.S. National Weather Service. The results suggest that the state of the
art in automatic calibration now can be expected to perform with a level of skill approaching that of a well-
trained hydrologist. This enables the hydrologist to take advantage of the power of automated methods to obtain
good parameter estimates that are consistent with the historical data and to then use personal judgment to refine
these estimates and account for other factors and knowledge not incorporated easily into the automated proce-
dure. The analysis also suggests that simple split-sample testing of model performance is not capable of reliably
indicating the existence of model divergence and that more robust performance evaluation criteria are needed.

INTRODUCTION AND SCOPE most hydrologic models were calibrated exclusively in a

Computer-based hydrologic models are becoming increas-
ingly sophisticated and are used routinely for tasks ranging
from real-time prediction of impending flood events to the
design of policies and structures for mitigating the effects of
extreme hydrologic events (e.g., floods and droughts). Because
many of these hydrologic models are intended to be applicable
to different watersheds, they typically have parameters that
must be adjusted to make the model behavior match the be-
havior of the watershed of interest. The process of adjusting
these parameters is called parameter estimation or model cal-
ibration.
The usefulness of hydrologic models for the purpose of op-
erational predictions depends on how well the model is cali-
brated. No matter how sophisticated the model structure is, if
the parameters are poorly specified, the model-simulated fluxes
(e.g., streamflows) can be quite different from those actually
observed. Therefore, the calibration procedure must be con-
ducted carefully to maximize the reliability of the model. In
many models, the values of certain parameters often can be
estimated by direct measurement of physically observable
characteristics of the watershed (e.g., area of the watershed,
fraction of watershed that is impervious, etc.). However, it is
typical for a model to have several parameters that are not
directly observable and therefore that must be estimated using
the indirect technique of matching the model to historical data.
The technique involves obtaining historical records of the im-
portant hydrologic inputs and outputs for the watershed (e.g.,
mean areal precipitation, streamflow, estimated evapotranspir-
ation, etc.), processing the input data through the model and
then adjusting the unknown parameters to obtain model-sim-
ulated outputs that match the observed watershed outputs as
closely as possible.
Before the widespread availability of high-speed computing,
1
Dept. of Hydro. and Water Resour., Univ. of Arizona, Tucson, AZ
85721. E-mail: hoshin@hwr.arizona.edu
2
Dept. of Hydro. and Water Resour., Univ. of Arizona, Tucson, AZ.
3
Dept. of Sys. and Industrial Engrg., Univ. of Arizona, Tucson, AZ.
Note. Discussion open until September 1, 1999. To extend the closing
date one month, a written request must be filed with the ASCE Manager
of Journals. The manuscript for this paper was submitted for review and
possible publication on June 23, 1997. This paper is part of the Journal
of Hydrologic Engineering, Vol. 4, No. 2, April, 1999. 䉷ASCE, ISSN
1084-0699/99/0002-0135 – 0143/$8.00 ⫹ $.50 per page. Paper No.
16054.
‘‘manual’’ fashion; a hydrologist having knowledge of the wa-
tershed and experience with the model would use a trial-and-
error procedure to adjust the parameters while visually com-
paring the observed and simulated outputs using graphical
plots. The feasibility of the manual method relies on the fact
that many hydrologic models are designed so the parameters
have some conceptual relationship to characteristics of the wa-
tershed. Therefore, the hydrologist expects that adjustment of
certain parameters in certain directions will have predictable
effects on the way in which the simulated outputs will change.
However, a large number of interacting parameters can result
in unpredictable effects when multiple parameters are adjusted.
For example, the Sacramento soil moisture accounting (SAC-
SMA) model used by the U.S. National Weather Service
(NWS) has 16 or more adjustable parameters in addition to
those associated with channel routing (Brazil and Hudlow
1981). Because the model structures are highly nonlinear, it is
difficult to predict just how sensitive the different portions of
the simulated outputs will be to parameter adjustments. In gen-
eral, the manual calibration procedure can be extremely time-
consuming and frustrating, and this has been a major factor
inhibiting the more widespread use of the more sophisticated
and complex models. Nonetheless, with considerable experi-
ence, many hydrologists have become very skilled at manual
calibration of their hydrologic models, especially because
modern high-speed computing enables the effects of parameter
adjustments to be observed very quickly. By adding semiau-
tomated calibration tools, the degree of training required be-
fore a hydrologist can perform an acceptable manual calibra-
tion also has been relaxed. For example, Brazil (1988)
developed a sophisticated and powerful interactive multilevel
calibration (MLC) method to assist in manual calibration of
the SAC-SMA model.
Because of the time-consuming and difficult nature of man-
ual calibration, researchers in the 1960s and early 1970s began
to explore methods by which the calibration process could be
speeded up and made more objective through automation.
Early attempts were reported by Dawdy and O’Donnell
(1965), Chapman (1970), Ibbitt (1970), Nash and Sutcliffe
(1970), Ibbitt and O’Donnell (1971), Monro (1971), and John-
ston and Pilgrim (1976), among others. The basic concept un-
derlying the following ‘‘automatic’’ calibration procedure has
changed little since its inception:
JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999 / 135

J. Hydrol. Eng., 1999, 4(2): 135-143

 1. A period of calibration data is selected.
2. An initial guess is made as to the probable values (or
range of values) for the parameters.
3. The model is run using these values for the parameters.
4. The ‘‘distance’’ between the model output and the ob-
served data is measured using a mathematical equation
called an objective function or estimation criterion.
5. An automatic optimization procedure (called a search al-
gorithm) is used to search for the parameter values that
optimize the value of the objective function.
The foregoing automatic procedure requires the selection of
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.
an objective function, a search algorithm, and a criterion by
which to terminate the search. The early studies took guidance
from what was then the state of the art in statistics and systems
theory in the selection of these items. However, it soom be-
came apparent that the automatic methods were not providing
very good results. One problem was that the parameter values
obtained by the automatic procedure often were unrealistic
conceptually. Another problem was that, although the solution
parameters might provide a good match to the calibration data,
the model performance on different periods of data often was
quite poor. A third problem was that the solution provided by
the automatic calibration procedure varied considerably with
choice of the calibration data, initial parameter guess, objective
function, and search procedure. For example, Johnston and
Pilgrim (1976) reported that, in more than 2 years of calibra-
tion research, they were unable to find a unique ‘‘best’’ set of
parameters for the Boughton model (Boughton 1965), and that
this seemed largely due to a highly irregular form of the ‘‘re-
sponse surface’’ of the objective function in the parameter
space. Ibbitt (1970), Ibbitt and O’Donnell (1971), and several
other researchers reported similar problems.
During the past two decades, a great deal of research has
been devoted to improving the automatic procedure. That re-
search has focused primarily on the following four issues: (1)
Development of specialized techniques for handling the kinds
of errors present in the measured data; (2) search for an op-
timization strategy that can reliably solve the parameter esti-
mation problem; (3) determination of the appropriate quantity
and most informative kind of data; and (4) efficient represen-
tation of the uncertainty of the calibrated model (structure and
parameters) and translation of that uncertainty into uncertainty
in the model response. Research into techniques for accounting
for data error has led to the development of maximum likeli-
hood functions for measuring the ‘‘closeness’’ of the model
and the data (Sorooshian and Dracup 1980; Sorooshian et al.
1981; James and Burges 1982; Sefe and Boughton 1982; Ku-
czera 1983a,b; Lemmer and Rao 1983; Sorooshian and Gupta
1983; Ibbitt and Hutchinson 1984). Research into optimization
methods has led to the use of population-evolution-based
search strategies (Brazil and Krajewski 1987; Brazil 1988;
Wang 1991; Duan et al. 1992; Duan et al. 1993; Sorooshian
et al. 1993). In this regard, the shuffled complex evolution
(SCE-UA) global optimization algorithm developed at the
University of Arizona has proved to be consistent, effective,
and efficient in locating the globally optimal model parameters
of a hydrologic model (Duan et al. 1992; Luce and Cundy
1994; Tanakamaru 1995; Bonan 1996; Gan and Biftu 1996;
Tanakamaru and Burges 1996; Kuczera 1997; Mroczkowski et
al. 1997; J. G. Ndiritu and T. M. Daniell unpublished work
1997). Research into data requirements has led to the under-
standing that the informativeness of the data is far more im-
portant than the amount used for model calibration (Kuczera
1982; Sorooshian et al. 1983; Gupta and Sorooshian 1985;
Yapo et al. 1996). Research into representation of model un-
certainty has led to practical procedures for rigorous statistical
analysis of model parameter uncertainty (Spear and Hornber-
136 / JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999
J. Hydrol. Eng., 1999, 4(2): 135-143
ger 1980; Jones 1983; Kuczera 1988). Finally, research into
methods for estimating model prediction uncertainty has led
to the development of the generalized likelihood uncertainty
estimation method (Beven and Binley 1992; Freer et al. 1996),
the Monte Carlo set membership procedure (Keesman 1990;
van Straten and Keesman 1991), and the prediction uncertainty
method (Klepper et al. 1991).
In this paper, we report on a study in which the capability
of the SCE-UA automatic procedure was compared with the
interactive MCL multistage semiautomated (manual) method
developed by Brazil (1988) for calibration of the SAC-SMA
model. The results indicate that the state of the art of automatic
calibration methods now has been elevated to the point where
such methods now can be considered seriously as a viable
alternative to the manual approach.
SCE-UA AUTOMATIC CALIBRATION METHOD
The SCE-UA method is a general purpose global optimi-
zation strategy designed to handle the various response surface
problems encountered in the calibration of nonlinear simula-
tion models, particularly the multilevel or ‘‘nested’’ optima
problem encountered with conceptual hydrologic models
(Duan et al. 1992). Detailed explanations of the method appear
in Duan et al. (1992, 1993); and Duan et al. (1994). In the
past 6 years, the method has been tested by numerous re-
searchers on a variety of models with overwhelmingly positive
results (Duan et al. 1992, 1993; Sorooshian et al. 1993; Luce
and Cundy 1994; Tanakamaru 1995; Gan and Biftu 1996; Tan-
akamaru and Burges 1996; Kuczera 1997; Mroczkowski et al.
1997; J. G. Ndiritu and T. M. Daniell unpublished work). The
algorithm typically is able to find consistently and efficiently
the global optimum of the problem, whereas other optimiza-
tion methods either fail or provide inconsistent results.
In brief, the SCE-UA method involves the initial selection
of a ‘‘population’’ of points distributed randomly throughout
the feasible parameter space. The population is partitioned into
several ‘‘complexes,’’ each consisting of 2n ⫹ 1 points, where
n is the number of parameters to be optimized. Each complex
‘‘evolves’’ independently in a manner based on the downhill
simplex algorithm (Nelder and Mead 1965; Press et al. 1992).
The population is ‘‘shuffled’’ periodically, and new complexes
formed so that the information gained by the previous com-
plexes is shared. The evolution and shuffling steps repeat until
prescribed convergence criteria are satisfied. In general, the
method is very easy to implement, having only one user-spec-
ified coefficient — the number of complexes — that conforms
to the difficulty of the optimization problem. Recommenda-
tions on the use of the SCE-UA appear in Duan et al. (1994);
the computer code can be obtained by contacting the first
writer. In each of the studies reported in this paper involving
calibration of the SAC-SMA model, the number of parameters
optimized was 13 and the number of complexes used was 15
[this gave a population size of 15(2 * 13 ⫹ 1) = 405 points];
each optimization run was terminated after 20 shuffling loops.
Implementation of the SCE-UA method requires the selec-
tion of an objective function to be optimized with respect to
the model parameters. In this study, we examined the perfor-
mance of the following two objective functions: the daily root-
mean square (DRMS) estimation criterion (DRMS = {SLS/
N}1/2, where SLS is the well-known simple least squares value
and N is the number of data points) and the heteroscadastic
maximum likelihood estimator (HMLE) criterion (Sorooshian
and Dracup 1980). The DRMS criterion implicitly assumes
that the error variance is constant (homogenous), whereas in
HMLE the error variance is assumed to vary with the level of
the output (magnitude of the flows) in a manner believed to
be common in streamflow data. The HMLE assumes that a
 transformation of the streamflows can be used to stabilize the interactive initial parameter estimator based on the use of com-

再
residual variance across flow ranges by using the relationship puter-generated graphics along with interactive input is used
to lead the user through some of the steps of initial data quality
q ␭t ⫺ 1
, ␭≠0 control and estimation of the feasible parameter region. In
Qt = ␭ (1) level II, a random search technique is used to obtain an im-
log(qt), ␭=0 proved estimate of the region of the best parameter values
using as many as 10 different statistical goodness-of-fit crite-
ria. In level III, a recursive parameter estimation procedure
where ␭ = an unknown transformation parameter estimated based on the Kalman filter is used to refine the parameter
from the data. The HMLE has the form estimates. Although the implementation of this procedure re-

冘
N
1 quires a fair degree of familiarity with the hydrologic model,
wt ε 2t Brazil (1988) indicated that it can reduce significantly the
N

冋写 册
t=1
amount of previous calibration experience required to obtain
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.

min HMLE = N 1/N (2)

␪,␭
a reasonable result. For specific details of these procedures,
wt
t=1
please refer to Brazil (1988).

where ε t = q obs
t ⫺ q tsim = model residual at time t; q obs
t and
q sim
t = observed and simulated flows, respectively; and wt =
weight assigned to time t, computed as
wt = f 2(␭⫺1)
t (3)
where ft = q true = expected true flow at time t, approximated
t
using q obs
t . Details of the recommended implementation pro-
cedure appear in Sorooshian et al. (1993).
INTERACTIVE MULTILEVEL CALIBRATION METHOD
The Interactive MLC strategy consists of a systematic three-
stage methodology for estimating the parameters of conceptual
hydrologic simulation models; it is designed to reduce the
larger problem to a number of subproblems that can be solved
using different optimization techniques. The procedure uses a
number of computer-based tools to assist the user in a hybrid
manual/automatic calibration of the model. In level I, a guided
MODEL, HYDROLOGIC DATA USED, AND
PARAMETERS OPTIMIZED
The SAC-SMA model (Fig. 1) is one of the options used
by the river forecast centers of the NWS to perform real-time
river and flood forecasts as well as extended streamflow pre-
dictions. In this study, a research version of the model main-
tained by the Department of Hydrology and Water Resources,
University of Arizona, was used. The inputs are mean areal
precipitation (mm/6 h) and potential evapotranspiration (mm/
day). The outputs are estimated evapotranspiration (mm/day)
and channel inflow; the latter is converted into streamflow (m3/
s) by means of a unit hydrograph.
The Leaf River basin (area 1944 km2) has been investigated
extensively (Brazil and Hudlow 1981; Sorooshian et al. 1983;
Brazil 1988). A reliable 40-water-year data set (October 1,
1948 – September 30, 1988), which represents a variety of hy-
drological conditions, was obtained from the NWS (see statis-

FIG. 1. Schematic Description of SAC-SMA Model [from Brazil (1988)]

JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999 / 137

J. Hydrol. Eng., 1999, 4(2): 135-143

 TABLE 1. Precipitation and Flow Statistics for Leaf River Ba- ‘‘IMLC estimates’’ in Table 2. To ensure a fair comparison,
sin the SCE-UA method was used to perform a single calibration
Daily run with the DRMS objective function (hereafter called the
measurements SCE-DRMS procedure) and a single calibration run with the
Flow precipitation Precipitationa HMLE objective function (hereafter called the SCE-HMLE
Statistic (cms) (mm) (mm) procedure). In each case, the same 13 parameters estimated by
(1) (2) (3) (4) Brazil were calibrated under slightly more difficult conditions,
Mean 30.90 3.92 8.69 created by employing a much larger initial parameter uncer-
Standard deviation 65.46 10.14 13.66 tainty on most of the parameters (Table 2).
Minimum 1.58 0 0.0001
Maximum 1,444.17 221.52 221.52 Parameter Estimates
Coefficient variance (%) 221.85 258.76 157.22
Coefficient skew 1.00 1.00 1.00 Fig. 3 shows the final parameter estimates obtained using
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.

2 the SCE-DRMS (thin solid line) and the SCE-HMLE (dashed

Note: Area = 1,944 km .
a
Nonzero precipitation.
tics in Table 1). The mean annual precipitation for this period
is 1,432 mm, and the mean annual runoff is 502 mm (30.90
cm). The hyetograph and hydrograph for the water-year with
the largest mean flow are shown in Fig. 2.
The model has 16 parameters (Table 2), an evapotranspir-
ation adjustment curve, and a unit hydrograph to be deter-
mined by the user. Following Brazil (1988), the three param-
eters SIDE, RIVA, and RSERV were fixed at prespecified
values (RSERV = 0.3, RIVA = 0.0, and SIDE = 0.0); the
evapotranspiration adjustment curve and the ordinates for the
unit hydrograph were assumed to have been estimated previ-
ously.
CASE STUDY
Methodology
In this study, we compared the performance of the single-
stage SCE-UA procedure and the three-stage interactive MLC
procedure with regard to the calibration of the SAC-SMA
model to the Leaf River basin. In previous work, Brazil (1988)
applied the interactive MLC procedure to this 13-parameter
calibration problem using 11 water-years (WY 1952 – 1962) of
data for the calibration and assuming the initial parameter un-
certainty estimates listed in Table 2. Based on numerous runs,
Brazil arrived at the best set of parameter estimates listed as
line) calibration runs in comparison with the best interactive
MLC parameter estimates (heavy solid line). Also shown as
the grey shaded region is the size of the initial parameter un-
certainty region used by Brazil (1998) for the interactive MLC
procedure in comparison with the larger one (the entire box)
used for the SCE-UA procedure. Note that in the interactive
MLC procedure, the estimates of ADIMP and REXP actually
drifted outside of the initial uncertainty region during the third
(unconstrained) optimization level of the search. The most im-
portant thing to notice here is that all three procedures (SCE-
DRMS, SCE-HMLE, and interactive MLC) give quite similar
parameter estimates for some of the parameters (PCTIM, AD-
IMP, ZPERC, LZTWM, LZFPM, and LZFSM) while differing
on the others. However, in general the SCE-DRMS parameter
estimates tend to be similar to the interactive MLC estimates,
which probably reflects the fact that Brazil used the DRMS
criterion as the primary statistic guiding his interactive param-
eter search.
Performance Comparison
To compare the performance of the parameter estimates ob-
tained by the SCE-DRMS, SCE-HMLE, and interactive MLC
methods, the three parameter estimates were used to simulate
the streamflow hydrograph for the entire 40-year period for
which data are available. Then, for each of the 40 years, the
four error statistics DRMS, PBIAS (percent bias), NSE (Nash-
Sutcliffe efficiency), and PME (persistence model efficiency)
were computed, where

FIG. 2. Leaf River Basin for Water-Year with Largest Daily Mean Flow (WY 1979): (a) Observed Daily Hyetograph; (b) Observed Daily
Hydrograph

138 / JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999

J. Hydrol. Eng., 1999, 4(2): 135-143

 TABLE 2. Parameters and State Variables of SAC-SMA Model
Study bounds Brazil’s initial bounds Interactive MLC
Parameters
optimized Description Lower bound Upper bound Lower bound Upper bound Estimates
(1) (2) (3) (4) (5) (6) (7)
(a) Maximum Capacity Thresholds
UZTWM Upper zone tension water maximum storage 1.0 150.0 10.0 150.0 9.0
(mm)
UZFWM Upper zone free water maximum storage 1.0 150.0 10.0 75.0 39.8
(mm)
LZTWM Lower zone tension water maximum storage 1.0 500.0 75.0 400.0 240.0
(mm)
LZFPM Lower zone free water primary maximum 1.0 1,000.0 120.0 140.0 120.0
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.

storage (mm)
LZFSM Lower zone free water supplemental maxi- 1.0 1,000.0 40.0 60.0 40.0
mum storage (mm)
ADIMP Additional impervious area (decimal frac- 0.0 0.4 0.0 0.2 0.25
tion)
(b) Recession Parameters
UZK Upper zone free water lateral depletion rate 0.1 0.5 0.2 0.4 0.200
(day⫺1)
LZPK Lower zone primary free water depletion 0.0001 0.025 0.006 0.01 0.006
rate (day⫺1)
LZSK Lower zone supplemental free water deple- 0.01 0.25 0.15 0.2 0.15
tion rate (day⫺1)
(c) Percolation and Other Parameters
ZPERC Maximum percolation rate (dimensionless) 1.0 250.0 5.0 250.0 250.0
REXP Exponent of the percolation equation (di- 1.0 5.0 1.1 4.0 4.27
mensionless)
PCTIM Impervious fraction of the watershed area 0.0 0.1 0.0 0.02 0.003
(decimal fraction)
PFREE Fraction of water percolating from upper 0.0 0.6 0.0 0.6 0.027
zone directly to lower zone free water
storage (decimal fraction)
(d ) Parameters Not Optimized
RIVA Riparian vegetation area (decimal fraction) 0.0
SIDE Ratio of deep recharge to channel base flow 0.0
(dimensionless)
RSERV Fraction of lower zone free water not trans- 0.3
ferrable to lower zone tension water (dec-
imal fraction)
(e) State Variables
UZTWC Upper zone tension water storage content
(mm)
UZFWC Upper zone free water storage content (mm)
LZTWC Lower zone tension water storage content
(mm)
LZFPC Lower zone free primary water storage con-
tent (mm)
LZFSC Lower zone free secondary water storage
content (mm)
ADIMC Additional impervious area content (mm)

DRMS = 冑冘 1
N
N

t=1
(q sim
t ⫺ q tobs)2 (4)
error; a smaller value indicates a better model performance.
The second statistic PBIAS (units m3/s) measures the average
tendency of the simulated flows to be larger or smaller than
their observed counterparts; the optimal value is 0.0; positive

冘 冒冘
N N
values indicate a model bias toward underestimation, whereas
PBIAS = (q obs
t ⫺ q sim
t ) q tobs ⫻ 100% (5) negative values indicate a bias toward overestimation. Both
t=1 t=1
NSE and PME provide normalized indicators of model per-

冘 冒冘
N N
formance in relation to benchmarks. The NSE (unitless) mea-
sures the relative magnitude of the residual variance (‘‘noise’’)
NSE = 1 ⫺ (q tsim ⫺ q obs
t )
2
(q obs
t ⫺ q mean)2 (6)
t=1 t=1
to the variance of the flows (‘‘information’’); the optimal value
is 1.0, and values should be larger than 0.0 to indicate ‘‘min-

冘 冒冘
N N imally acceptable’’ performance — a value equal to 0.0 indi-
PME = 1 ⫺ (q tsim ⫺ q tobs)2 (q tobs ⫺ q t⫺1
obs 2
) (7) cates that the mean observed flow is a better predictor than
t=1 t=1 the model. The PME measures the relative magnitude of the
residual variance (noise) to the variance of the errors obtained
These four statistics provide interesting perspectives on by the use of a simple persistence model; the optimal value is
model performance. The first statistic DRMS (units m3/s) sim- 1.0, and values should be larger than 0.0 to indicate minimally
ply computes the standard deviation of the model prediction acceptable performance. The simple persistence model repre-
JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999 / 139

J. Hydrol. Eng., 1999, 4(2): 135-143

Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.

FIG. 3. Comparison of Normalized Final Parameter Estimates Obtained Using SCE-UA Method with DRMS Criterion (Thin Solid Line)
and HMLE Criterion (Dashed Line) to Interactive MLC Parameter Estimates [Shaded Region Denotes Initial Parameter Uncertainty Re-
gion Used by Brazil (1988)]

FIG. 4. Evaluation of Calibrated Model Performance on Annual Basis for 40 Years of Record

sents a ‘‘minimal-information’’ situation in which we assume
that the best estimate of streamflow at the next time step is
given by the observed flow at the current time step. Clearly,
if a rainfall-runoff model has a one-step-ahead forecast error
that is larger than that provided by the simple persistence
model (i.e., PME < 0), its suitability for streamflow forecasting
should be questioned. The results of the comparison (Fig. 4)
may be summarized as follows:
1. Fig. 4(a) indicates that the interactive MLC and SCE-
DRMS methods generally provide similar DRMS statis-
tic performance for the individual years in the 40-year
period, with the SCE-DRMS method being slightly su-
perior. The plot also indicates that the model perfor-
mance varies from year to year (depending on the wet-
ness of the year — more on this later). The worst
SCE-HMLE performance on the DRMS statistic is, of
course, expected, given that the SCE-HMLE parameter
estimates tend to provide model forecasts with noncon-
stant error variance.
2. Fig. 4(b) shows that the SCE-DRMS and interactive
MLC methods also provide similar PBIAS statistic per-
formance, with a definite tendency to underestimation.
140 / JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999
However, the SCE-HMLE method tends to provide fore-
casts that are relatively unbiased.
3. Fig. 4(c) indicates that the calibrated SAC-SMA model
tends to be quite efficient with NSE statistic values larger
than 0.8 for most of the 40 years, but that it performs
poorly on some of the years with NSE less than 0.5.
Also, it is interesting to note that the NSE statistic for
four (WY 1956 – 1959) of the 11 years in the calibration
period tends to be quite poor (NSE < 0.7), suggesting
that it might have been better to select a different set of
water-years for model calibration.
4. Fig. 4(d) reveals that the PME statistic is a more pow-
erful test of model performance than the NSE statistic.
The negative PME values on the graph indicate that, for
some years, the SAC-SMA model is unable to provide
(on a daily average basis) a forecast that is superior to
that provided by a simple persistence model. For the
years for which the PME statistic is positive, the SAC-
SMA model performance is on the average only 25 – 50%
better than the simple persistence model.
The foregoing results indicate that there is a great deal of
variability in model performance from year to year as mea-

J. Hydrol. Eng., 1999, 4(2): 135-143

Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.
FIG. 5. Evaluation of Calibrated Model Performance on Annual Basis for 40 Years of Record Ordered by Annual Mean Flow
sured by the four statistics. Some of the following reasons for
this variability become apparent when the results are plotted
against the annual mean daily flow (cm) for the corresponding
year (Fig. 5):
1. Fig. 5(a) now clearly shows that the model error variance
(measured by DRMS) increases with wetness of the year,
indicating that the forecast error variance is larger for
higher flows. This observation supports the assumptions
behind the use of the HMLE objective function for pa-
rameter estimation. It also indicates that the model is
better able to match lower flows (on an absolute basis)
than higher flows. This fact should be considered when
attempting to do split-sample performance evaluation of
the model; if it is found that the performance evaluation
period DRMS is quite a bit larger than the calibration
period DRMS, this does not automatically mean that the
calibration is poor; more careful evaluation of the results
is required — we recommend a comparative evaluation of
the model fit by computing the statistics separately for
each of the years in the calibration and evaluation peri-
ods and plotting them against an indicator of wetness
such as annual mean daily flow.
2. Fig. 5(b) indicates that the PBIAS performance for the
three methods tends to be more different for dry water-
years but is quite similar for the wetter years.
3. Both of Figs. 5(c and d) show that the model NSE and
PME performance tends to be better for wetter years than
for dry years. This indicates that the SCA-SMA model
performs relatively well (PME > 0) for its intended func-
tion of flood forecasting (high flows), where the ‘‘per-
sistence model’’ performance must be necessarily poor,
but that the low flow performance could stand improve-
ment.
The results of the foregoing paragraph indicate that the per-
formance of the model tends to vary with flow level. This fact
is seen more clearly when, for each water-year, the data are
separated into two portions — ‘‘below-mean’’ flows and
‘‘above-mean’’ flows — and the four statistics are computed
again for each portion of the data. The following results are
presented in Fig. 6 with the above-mean and below-mean sta-
tistics shown side by side:
1. Figs. 6(a and b) show clearly that the SCE-HMLE pa-
J. Hydrol. Eng., 1999, 4(2): 135-143
rameter estimates provide the best below-mean DRMS
performance, whereas the SCE-DRMS parameter esti-
mates provide the best above-mean performance.
2. Figs. 6(c and d) indicate that the SCE-HMLE parameter
estimates provide the best below-mean PBIAS perfor-
mance, whereas above-mean performance tends to be
similar for the three estimators.
3. Fig. 6(e) indicates that the model error variance for be-
low-mean flows tends to be much larger than the vari-
ance of the flows themselves (NSE < 0), whereas Fig.
6(f) shows that the above-mean forecasts performance
tends to be very good for wet years and not so good for
drier years.
4. Figs. 6(g and h) clearly show that the persistence model
provides superior performance (PME < 0) for below-
mean flows and comparable performance (PME ⬃ 0) for
dry-year above-mean flows. However, for wet-year
above-mean flows the SAC-SMA model performance is
quite good (PME > 0).
CONCLUSIONS
In this paper, the capability of the SCE-UA automatic pro-
cedure was shown to compare favorably with the interactive
MLC multistage semiautomated method developed by Brazil
(1988) for calibration of the SAC-SMA streamflow forecasting
model of the U.S. NWS. The results of this study suggest that
the state of the art in automatic calibration of hydrologic mod-
els may have reached the point where automated methods can
be expected to perform with a level of skill approaching that
of a well-trained hydrologist. However, we do not mean to
imply that the skill of the hydrologist is becoming unneces-
sary, rather that the trained hydrologist may now be able to
place more confidence in the use of automated tools to assist
in the model calibration process. In particular, the hydrologist
can take advantage of the power of automated methods to
obtain good parameter estimates that are consistent with the
historical data and to then refine these estimates to account for
other factors and knowledge not easily incorporated into the
automated procedure. The major advantage of this is the tre-
mendous savings in time and energy that can be realized. For
example, to calibrate the SAC-SMA model by manual methods
to the more than 2,000 watersheds for which flood forecasting
must be performed by the U.S. NWS is practically infeasible;
the availability of reliable automated methods such as the
JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999 / 141
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.
FIG. 6. Evaluation of Calibrated Model Performance on Annual Basis for 40 Years of Record Ordered by Annual Mean Flow and Par-
titioned into Below-Mean and Above-Mean Flow Categories
SCE-DRMS and SCE-HMLE now makes this task much more
achievable.
The study also illuminates the fact that model performance
evaluation (sometimes incorrectly called verification) can be a
tricky process. It is clear that a simple split-sample test in
which the overall error variance for the calibration and eval-
uation periods are compared is incapable of reliably indicating
the existence of model divergence. Instead, it is necessary to
conduct a careful and detailed analysis of the properties of the
model residuals for both periods; this issue is discussed in
more detail in a separate paper (Yapo et al. 1996). Another
issue is the need to use performance evaluation criteria that
have the power to clearly indicate poor model performance.
In this study, the DRMS measure had the least power and the
NSE measure was only marginally informative, whereas the
PBIAS and PME measures were most useful. The power of
the PME measure comes from the fact that the model perfor-
mance is being compared with that of a simple persistence
forecast model. Clearly, much stricter tests could be con-
structed by comparing model performance to less simple mod-
els such as trend forecasts or linear time series forecasts. Cur-
142 / JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999
J. Hydrol. Eng., 1999, 4(2): 135-143
rently, we are investigating these issues and will report on our
findings in due course.
We welcome dialog on issues related to the calibration of
various kinds of hydrologic models.
ACKNOWLEDGMENTS
Partial financial support for this research was provided by the National
Science Foundation (Grants EAR-9415347 and EAR-9418147), the Hy-
drologic Research Laboratory of the NWS (Grants NA47WH0408 and
NA57WH0575), and by the National Aeronautics and Space Administra-
tion (NASA-EOS Grant NAGW2425). Financial assistance provided to
Dr. Yapo by the University of Arizona Graduate College is gratefully
acknowledged. The computer codes and data used in this study are avail-
able on request from the first author at hoshin@hwr.arizona.edu.
APPENDIX. REFERENCES
Beven, K. J., and Binley, A. M. (1992). ‘‘The future of distributed models:
Model calibration and uncertainty prediction.’’ Hydrological Processes,
Sussex, U.K., 6, 279 – 298.
Bonan, G. B. (1996). ‘‘Sensitivity of a GCM simulation to subgrid infil-
tration and surface runoff.’’ Climate Dyn., 12, 279 – 285.
 Boughton, W. C. (1965). ‘‘A new estimation technique for estimation of
catchment yield.’’ Rep. No. 78, Water Resources Laboratory, University
of New South Wales, Manly Vale.
Brazil, L. E. (1988). ‘‘Multilevel calibration strategy for complex hydro-
logic simulation models,’’ PhD dissertation, Colorado State University,
Fort Collins, Colo.
Brazil, L. E., and Hudlow, M. D. (1981). Calibration procedures used
with the National Weather Service river forecast system, in water and
related land resource systems. New York, 457 – 466.
Brazil, L. E., and Krajewski, W. F. (1987). ‘‘Optimization of complex
hydrologic models using random search methods.’’ Conf. on Engrg.
Hydro., Hydraulics Division, ASCE, New York, 726 – 731.
Chapman, T. G. (1970). ‘‘Optimization of a rainfall-runoff model for an
arid zone catchment.’’ UNESCO Publication 96, United Nations Ed-
ucational, Scientific, and Cultural Organization, Wallington, New Zea-
Downloaded from ascelibrary.org by University of Technology, Sydney on 09/27/22. Copyright ASCE. For personal use only; all rights reserved.
land.
Dawdy, D. R., and O’Donnell, T. (1965). ‘‘Mathematical models of catch-
ment behavior.’’ J. Hydr. Div., ASCE, 91(4), 113 – 137.
Duan, Q. Y., Gupta, V. K., and Sorooshian, S. (1992). ‘‘Effective and
efficient global optimization for conceptual rainfall-runoff models.’’
Water Resour. Res., 28(4), 1015 – 1031.
Duan, Q. Y., Gupta, V. K., and Sorooshian, S. (1993). ‘‘A shuffled com-
plex evolution approach for effective and efficient global minimiza-
tion.’’ J. Optimization Theory and Applications, 76(3), 501 – 521.
Duan, Q. Y., Sorooshian, S., and Gupta, V. K. (1994). ‘‘Optimal use of
the SCE-UA global optimization method for calibrating watershed
models.’’ J. Hydro., Amsterdam, 158, 265 – 284.
Freer, J., Beven, K. J., and Ambroise, B. (1996). ‘‘Bayesian estimation
of uncertainty in runoff prediction and the value of data: An application
of the GLUE approach.’’ Water Resour. Res., 32(7), 2161 – 2173.
Gan, T. Y., and Biftu, G. F. (1996). ‘‘Automatic calibration of conceptual
rainfall-runoff models: Optimization algorithms, catchment conditions
and model structure.’’ Water Resour. Res., 32(12), 3513 – 3524.
Gupta, V. K., and Sorooshian, S. (1985). ‘‘The relationship between data
and the precision of parameter estimates of hydrologic models.’’ J.
Hydro., Amsterdam, 81, 57 – 77.
Ibbitt, R. P. (1970). ‘‘Systematic parameter fitting for conceptual models
of catchment hydrology,’’ PhD dissertation, University of London,
London.
Ibbitt, R. P., and Hutchinson, P. D. (1984). ‘‘Model parameter consistency
and fitting criteria.’’ Proc., IFAC 9th Triennial World Congr., Interna-
tional Federation of Automatic Control, 153 – 157.
Ibbitt, R. P., and O’Donnell, T. (1971). ‘‘Fitting methods for conceptual
catchment models.’’ J. Hydr. Div., ASCE, 97(9), 1331 – 1342.
James, L. D., and Burges, S. J. (1982). ‘‘Selection, calibration, and testing
of hydrologic models.’’ Hydrologic testing of small watersheds, C. T.
Haan, H. P. Johnson, and D. L. Brakensiek, eds., American Society of
Agricultural Engineers, St. Joseph, Mich., 435 – 472.
Johnston, P. R., and Pilgrim, D. (1976). ‘‘Parameter optimization for wa-
tershed models.’’ Water Resour. Res., 12(3), 477 – 486.
Jones, D. A. (1983). ‘‘Statistical analysis of empirical models fitted by
optimization.’’ Biometrika, 70(1), 67 – 88.
Keesman, K. J. (1990). ‘‘Set-theoretic parameter estimation using random
scanning and principal component analysis.’’ Math. Computation and
Simulation, 32, 535 – 543.
Klepper, O., Scholten, H., and Kamer, J. P. G. v. d. (1991). ‘‘Prediction
uncertainty in an ecological model of the Oosterschelde Estuary.’’ J.
Forecasting, 10, 191 – 209.
Kuczera, G. (1982). ‘‘On the relationship of the reliability of parameter
estimates and hydrologic time series data used in calibration.’’ Water
Resour. Res., 18(1), 146 – 154.
Kuczera, G. (1983a). ‘‘Improved parameter inference in catchment mod-
els: 1. Evaluating parameter uncertainty.’’ Water Resour. Res., 19(5),
1151 – 1162.
J. Hydrol. Eng., 1999, 4(2): 135-143
Kuczera, G. (1983b). ‘‘Improved parameter inference in catchment mod-
els: 2. Combining different kinds of hydrologic data and testing their
compatibility.’’ Water Resour. Res., 19(5), 1163 – 1172.
Kuczera, G. (1988). ‘‘On validity of first-order prediction limits for con-
ceptual hydrological models.’’ J. Hydro., Amsterdam, 103, 229 – 247.
Kuczera, G. (1997). ‘‘Efficient subspace probabilistic parameter optimi-
zation for catchment models.’’ Water Resour. Res., 33(1), 177 – 185.
Lemmer, H. R., and Rao, A. R. (1983). ‘‘Critical duration analysis and
parameter estimation in ILLUDAS.’’ Tech. Rep. No. 153, Water Re-
sources Research Center, Purdue University, West Lafayette, Ind.
Luce, C. H., and Cundy, T. W. (1994). ‘‘Parameter identification for a
runoff model for forest roads.’’ Water Resour. Res., 30(4), 1057 – 1069.
Monro, J. C. (1971). ‘‘Direct search optimization in mathematical mod-
eling and a watershed model application.’’ NOAA Techn. Memo No.
NWS HYDRO-12, U.S. Department of Commerce, Silver Spring, Md.
Mroczkowski, M., Raper, G. P., and Kuczera, G. (1997). ‘‘The quest for
more powerful validation of conceptual catchment models.’’ Water Re-
sour. Res., 33(10), 2325 – 2336.
Nash, J. E., and Sutcliffe, J. V. (1970). ‘‘River flow forecasting through
conceptual models, Part 1 — a discussion of principles.’’ J. Hydro.,
Amsterdam, 10(3), 282 – 290.
Nelder, J. A., and Mead, R. (1965). ‘‘A simplex method for function
minimization.’’ Computer J., 7(4), 308 – 313.
Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P.
(1992). Numerical recipes in C: The art of scientific computing. Cam-
bridge University Press, Cambridge, London.
Sefe, F. T., and Boughton, W. C. (1982). ‘‘Variation of model parameter
values and sensitivity with type of objective function.’’ J. Hydro. (New
Zealand), 21(1), 117 – 132.
Sorooshian, S., and Dracup, J. A. (1980). ‘‘Stochastic parameter estima-
tion procedures for hydrologic rainfall-runoff models: Correlated and
heteroscedastic error cases.’’ Water Resour. Res., 16(2), 430 – 442.
Sorooshian, S., Duan, Q. Y., and Gupta, V. K. (1993). ‘‘Calibration of
rainfall-runoff models: Application of global optimization to the Sac-
ramento soil moisture accounting model.’’ Water Resour. Res., 29(4),
1185 – 1194.
Sorooshian, S., and Gupta, V. K. (1983). ‘‘Automatic calibration of con-
ceptual rainfall-runoff models: The question of parameter observability
and uniqueness.’’ Water Resour. Res., 19(1), 260 – 268.
Sorooshian, S., Gupta, V. K., and Fulton, J. L. (1981). ‘‘Parameter esti-
mation of conceptual rainfall-runoff models assuming autocorrelated
data errors — a case study.’’ Int. Symp. on Rainfall-Runoff Modeling,
Mississippi State, Miss., 491 – 504.
Sorooshian, S., Gupta, V. K., and Fulton, J. L. (1983). ‘‘Evaluation of
maximum likelihood parameter estimation techniques for conceptual
rainfall-runoff models: Influence of calibration data variability and
length on model credibility.’’ Water Resour. Res., 19(1), 251 – 259.
Spear, R. C., and Hornberger, G. M. (1980). ‘‘Eutrophication in Peel
Inlet — II. Identification of critical uncertainties via generalized sensi-
tivity analysis.’’ Water Resour. Res., 14, 43 – 49.
Tanakamaru, H. (1995). ‘‘Parameter estimation for the tank model using
global optimization.’’ Trans. JSIDRE, 178, 103 – 112.
Tanakamaru, H., and Burges, S. J. (1996). ‘‘Application of global opti-
mization to parameter estimation of the tank model.’’ Proc., Int. Conf.
on Water Resour. and Envir. Res.: Towards the 21st Century, 39 – 46.
van Straten, G., and Keesman, K. J. (1991). ‘‘Uncertainty propagation
and speculation in projective forecasts of environmental change: A
lake-eutrophication example.’’ J. Forecasting, 10, 163 – 190.
Wang, Q. J. (1991). ‘‘The genetic algorithm and its application to cali-
brating conceptual rainfall-runoff models.’’ Water Resour. Res., 27(9),
2467 – 2471.
Yapo, P. O., Gupta, H. V., and Sorooshian, S. (1996). ‘‘Automatic cali-
bration of conceptual rainfall-runoff models: Sensitivity to calibration
data.’’ J. Hydro., 181, 23 – 48.
JOURNAL OF HYDROLOGIC ENGINEERING / APRIL 1999 / 143