Environmental Modelling & Software 48 (2013) 129e140

Contents lists available at SciVerse ScienceDirect

Environmental Modelling & Software

journal homepage: www.elsevier.com/locate/envsoft

The spatial framework for weight sensitivity analysis in AHP-based

multi-criteria decision making
Yun Chen a, b, Jia Yu b, c, *, Shahbaz Khan d
a
CSIRO Land and Water, GPO Box 1666, Canberra, ACT 2601, Australia
b
Department of Geography, Shanghai Normal University, Shanghai 200234, China
c
Key Laboratory of Geographic Information Science, Ministry of Education, East China Normal University, Shanghai 200062, China
d
UNESCO Regional Science Bureau for Asia and the Pacific, Jakarta, Indonesia

a r t i c l e i n f o a b s t r a c t

Article history: Criteria weights determined from pairwise comparisons are often the greatest contributor to the un-
Received 28 February 2013 certainties in the AHP-based multi-criteria decision making (MCDM). During an MCDM process, the
Received in revised form weights can be changed directly by adjusting the output from a pairwise comparison matrix, or indirectly
16 June 2013
by recalculating the matrix after varying its input. Corresponding weight sensitivity on multi-criteria
Accepted 19 June 2013
Available online 18 July 2013
evaluation results is generally difficult to be quantitatively assessed and spatially visualized. This
study developed a unique methodology which extends the AHP-SA model proposed by Chen et al. (2010)
to a more comprehensive framework to analyze weight sensitivity caused by both direct and indirect
Keywords:
Pairwise comparison matrix
weight changes using the one-at-a-time (OAT) technique. With increased efficiency, improved flexibility
Uncertainty and enhanced visualization capability, the spatial framework was developed as AHP-SA2 within a GIS
OAT platform. A case study with in-depth discussion is provided to demonstrate the new toolset. It assists
MCDM stakeholders and researchers with better understanding of weight sensitivity for characterising,
ArcGIS reporting and minimising uncertainty in the AHP-based spatial MCDM.

2013 Elsevier Ltd. All rights reserved.

1. Introduction
Spatial multi-criteria decision making (MCDM1) and a wide
range of related methodologies aim to solve “real-world” GIS-based
planning and management problems. They offer a variety of tech-
niques and practices to incorporate knowledge from various disci-
plines and to integrate decision makers’ preferences. Since last
decade, researches on GIS-based MCDM have been extensively
covered in the literature (e.g. Jiang and Eastman, 2000; Store and
Kangas, 2001; Svoray et al., 2005; Grayson et al., 2008; Chen et al.,
2009, 2010; Rahman et al., 2012). They have unique, flexible capa-
bilities for automating and analysing spatial decision making issues
with large sets of feasible alternatives and multiple conflicting and
incommensurate evaluation criteria from varying sources to aid
policy analysis and implementation. However, because of concep-
tual difficulties involved in formulating and solving spatial decision
problems, it is often the case that input criteria uncertainties are
* Corresponding author. Department of Geography, Shanghai Normal University,
100 Guilin Road, Shanghai 200234, China.
E-mail addresses: yujiashnu@126.com (J. Yu), s.khan@unesco.org (S. Khan).
1
MCDM: multi-criteria decision making.
poorly understood, while taking large samples to reduce input un-
certainty is often impossible. Therefore, sensitivity analysis (SA)
provides a way for modellers and decision-makers to determine the
inputs that the outputs are most dependent on. It is recognized as
being a key component of all model development (Costa and Carlos,
1988; Wolf et al.,1996; Butler et al.,1997; Mendicino,1999; Doos and
Shaw, 1999; Saltelli et al., 2000; Chen et al., 2010; Allaire and
Willcox, 2012), particularly useful in model calibration to assist
with model adjustment to better fit the available data through
identifying unexpected dependencies on criteria. Furthermore,
rapid increases in model size and complexity, such as the integration
of numerous criteria to determine outcomes in line with an increase
in their non-linearities, MCDM outputs may also not be intuitive,
creating a new challenge for reliable, effective and informative
sensitivity analysis. Thus an urgent need to integrate sensitivity and
uncertainty analysis into spatial MCDM has been highlighted by
many recent studies (Walker et al., 2003; Refsgaard et al., 2007;
Chen et al., 2010; Pisoni et al., 2010; Chen et al., 2011).
There are well established techniques for sensitivity analysis,
ranging from basic differential analysis (Hwang et al., 1997) to
famous Monte Carlo simulation (Hornberger and Spear, 1981;
Dimov et al., 2012; Flores-Alsina et al., 2012; Zio and Pedroni, 2012),
from screening designs (Morris, 1991; Zhang et al., 2013) to scatter

1364-8152/$ e see front matter
2013 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.envsoft.2013.06.010
130 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140
plots (Frey and Patil, 2002), from measures of importance (Saltelli,
2002a; Cui et al., 2012) to sensitivity indices (Saltelli, 2002b; Allaire
and Willcox, 2012; Mara and Tarantola, 2012), from regression or
correlation (Manache and Melching, 2008) to Bayesian (Oakley and
O’Hagan, 2004) approach, and from variance-based Fourier
Amplitude Sensitivity Testing (Cukier et al., 1978) to Sobol’ method
(Sobol’,1993, Nossent et al., 2011). A thorough review of many SA
methods can be found in Kleijnen (1997), Saltelli et al. (2000) and
Campolongo et al. (2007). Special interests have been paid to
several of the more practical methods for conducting MCDM
sensitivity studies in literature, in particular one-at-a-time (OAT;
Morris, 1991) sensitivity measures (Archer et al., 1997; Crosetto
et al., 2000; Crosetto and Tarantola, 2001; Hyde et al., 2005; Hill
et al., 2005; Hyde and Maier, 2006; Ravalico et al., 2010; Chen
et al., 2010, 2011; Drouet et al., 2011). OAT is one of the typical
screening experiments, in which the impact of changing the values
of each of the chosen factors is evaluated in turn. It is methodo-
logically simple, computationally cheap, and easy to develop.
The analytic hierarchy process (AHP) has been widely applied to
solve a broad range of multi-criteria decision-making problems
since its inception more than 30 years ago (Saaty, 1977, 1980; Saaty
and Vargas, 1991; Wu, 1998; Ohta et al., 2007; Chen et al., 2009).
The pairwise comparison is the core of the AHP for understanding
the theory and application of the AHP. In an MCDM process, the
AHP derives the weights for criteria with the help of a preference
matrix where all identified relevant criteria are compared against
each other with a scale of relative importance (Saaty and Vargas,
1991). Through the pairwise comparison matrix (PCM2), the AHP
calculates the weight value for each criterion. The AHP has been
highly popular because of the ease in obtaining the criterion
weights and the capacity to integrate heterogeneous data. It is
especially useful in situations where it is impractical, or even
impossible, to specify the exact relationships between a large
number of evaluation criteria. However, one of the major draw-
backs of the AHP is its inability to address the uncertainty and
imprecision of the decision maker’s perceptions. In particular, it
tends to provide qualitative sensitivity measures, i.e. they rank the
input factors in order of importance, but do not quantify how much
a given factor is more important than another. Although the un-
certainty can come from many different sources, such as original
data, data processing, criteria selection and their thresholds,
criteria weights are often the greatest contributor to controversy
and uncertainty. This could be because decision makers are not
absolutely aware of their preferences regarding the criteria, and
may be because the nature and scale of the criteria is not known. Or,
especially when multiple decision makers are involved, it is often
not possible to derive only one set of weights, but ranges of criteria
weights leading to more than one set of results. Therefore, sensi-
tivity analysis on the weights of input criteria is crucial to reducing
uncertainty and increasing stability of the outputs.
There are two most common ways to analyze criteria weight
sensitivity (Chen et al., 2010). The first one is directly changing
absolute values of criteria weights output from a PCM; and the
other is changing relative importance scales of criteria in a PCM,
which results in a recalculation of criteria weights. Delgado and
Sendra (2004) conducted a review on how SA has been applied to
GIS-based MCDM models. It indicated that little attention had been
paid to the evaluation of the final results from these model simu-
lations. In addition, the SA method most frequently used is based on
the variation of the weights of criteria implied in the process to test
whether it significantly modifies the results obtained. Perhaps the
most critical shortcoming of SA procedures found in limited GIS-
2 3
PCM: pairwise comparison matrix.
MCDM applications is the lack of insight they provide into the
spatial aspects of weight sensitivity. It is recommended that SA
procedures should permit weight sensitivity to be visualized
geographically and to facilitate the spatial analysis of sensitivity,
where appropriate (Feick and Hall, 2004). Therefore, an ideal
criteria weight sensitivity analysis framework in support of AHP-
based MCDM should be capable of taking these uncertainties and
issues into account. A good example of such a tool, called AHP-SA,
was proposed by Chen et al. (2010). The AHP-SA is a spatial
framework on SA of multi-criteria weights, which evaluates im-
pacts on outputs by varying input criteria weights directly using the
OAT approach.
Building on the previous AHP-SA research, this study aims to
develop a methodology to systematically and quantitatively
analyze criteria weight sensitivity in AHP-based MCDM using the
spatial OAT method. It incorporates the capability of determining
output variation or uncertainty resulting from changing important
scales of input criteria in a PCM into the previous spatial AHP-SA
framework (Chen et al., 2010). The new framework was extended
as a comprehensive toolset. It can be used to conduct compre-
hensive SA on criteria weights and to visualize SA results spatially
and graphically. This enables consideration of criteria weight
sensitivity to be operationalized within the decision-making pro-
cess without assuming any prior knowledge of SA on the part of the
decision-maker. An application to the Macintyre-Brook catchment
of Australia is used to demonstrate the effectiveness of this toolset.
The main findings of the case study are presented and discussed.
2. Criteria weight sensitivity
2.1. Methods
The fundamental function of the original AHP-SA framework
(Chen et al., 2010) is to study criteria weight sensitivity through
altering criterion weight values calculated by the AHP in percent
increments and decrements using the OAT method. Both AHP and
OAT methods have been described in details by Chen et al. (2010).
We assume the reader is familiar with the concepts and theory of
the AHP and with the class of multi-criteria decision problems to
which it can be applied. The AHP-SA is an easy-to-use and
straightforward approach for decision makers to explore the
sensitivity of criteria weights output from the AHP analysis on the
MCDM results. But it does not provide the insight on how much the
change of input into an AHP PCM will affect the final weight deri-
vation. The interests of this study lie into four aspects: (1) exam-
ining the stability of an evaluation by changing element values in
the base PCM according to a predefined scale; (2) identifying
criteria that are practically sensitive to weight changes caused by
the modification of a relevant element value in the PCM; (3)
quantifying changes relative to criteria rankings and the evaluation
output; and (4) visualizing the spatial changes of analysis results.
An extended framework is presented in Fig. 1 as a workflow
comprising a number of steps which are ordered logically. It as-
sesses criteria weight sensitivity in relation to changing relative
important scales of criteria in an AHP PCM by altering the value of
one element at a time without changing the others using the OAT
technique. The value of each element in an AHP PCM is expressed
using a scale of intensity of importance (IOI3) representing relative
importance between each pair of criteria. Therefore, changes in
input IOI and resultant consequences in criteria weights and MCDM
IOI: intensity of importance.
 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140
Fig. 1. Flowchart of methodology for AHP matrix sensitivity analysis. *A user-defined
acceptable CR value.
output are the keys of the sensitivity evaluation. The analysis begins
from an existing AHP PCM defined as a Base PCM:
Step 1. Selecting an element (Base_EV4) as the candidate to be
examined from the Base PCM. The element is picked by specifying
the criteria on row (CoR5) and on column (CoC6), for example Cri-
terion 3 as CoR and Criterion 2 as CoC from a matrix with four
criteria in Fig. 2. The base IOI value of the Base_EV, which is based
on the well-known scale (Table 1) proposed by Saaty and Vargas
(1991), equals to 1/5 (Fig. 2).
Step 2. Assigning an IOI changing range (CREV(Base_EV, R))7 for
the Base_EV based on its base IOI value. The CREV(Base_EV, R) is a
set of inputs to be used to adjust the Base PCM. It is defined by two
parameters. The first one is a set of all possible indexed IOI values
listed in Table 2 which was developed in this study. The other is a
user-defined changing radius (R) which confines a threshold that
4
Base_EV: base element value.
5
CoR: criteria on row.
6 8
CoC: criteria on column.
7 9
CREV: changing range of element values.
131
Fig. 2. Selection of changing element from the Base PCM by specifying one criterion on
Row (CoR) and the other on column (CoC).
determines how many IOIs among all possible IOIs in Table 2 will be
examined for a selected Base_EV. The rule to determine a CREV(-
Base_EV, R) is given in Equation (1):
CREVðBase EV; RÞ ¼ fIOIðiÞjIndexB  R  i  IndexB
(1)
þ R and 0  i  16g
where IOI(i) is the IOI for index i in Table 2. IndexB is the index of the
IOI in Table 2 whose value is equal to the base IOI value of the
selected Base_EV. The meaning of the equation is if the decision
makers are not quite confident about the value of the Base_EV they
entered into the Base PCM, then they can change the value within a
certain range (CREV(Base_EV, R)) for sensitivity analysis through a
series simulations (depending on level of uncertainty). R is an
adjustable buffer threshold defined by the upper and lower bounds
of IOIs to reduce unnecessary runs according to the degree of
confidence of the decision makers. The value of R ranges from 0 to
16. The greater the R is, the higher the uncertainty level is.
Step 3. The current element value (CEV8) is an internal variable
which is used to store the changing value of the Base_EV for each
simulation. It changes according to IOI(i) within the range of
CREV(Base_EV, R). For each new CEV, the weights of all criteria will
be re-calculated and the resultant CR (consistency ratio of the
PCM) will be re-assessed to make sure the current CR value is
within the range of threshold defined by users. The methods to
obtain criteria weights and CR values can be retrieved by Equation
(6) and Equation (10) in the paper of Chen et al. (2010), respec-
tively. If the CR check is passed, an MCDM evaluation map for the
run will be created accordingly and results for matrix sensitivity
analysis will be recorded. Otherwise, the current PCM will be
marked to be unqualified. As a result, no sensitivity analysis can be
performed for that particular run. The process continues with next
CEV in the loop and stops when a CEV is out of the range of the
CREV(Base_EV, R).
It has to be noted that, because the PCM has the characteristic of
reciprocity, if a selected relative element value (EV9) in the matrix is
changed, the diagonally symmetric EV needs to be replaced with
the reciprocal of the revised EV. For example, in Fig. 2, if the
element value “1/5” is changed to “1/4”, the element value “5”
which is marked in red circle needs to be changed to “4”. From this
rule, it could be detected that changing element value for Criterion
3 (CoR) and Criterion 2 (CoC) has the same effect to the changing for
Criterion 2 (CoR) and Criterion 3 (CoC). Therefore, for each pair of
criteria, the number of runs for a specified Base_EV and R is:
Runs ¼ CountðIOIÞ; IOI˛CREVðBase EV; RÞ (2)
CEV: current element value.
EV: element value.
132 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140

Table 1 visualize the SA results. MATLAB COM-Compliant components

Scale of Intensity of Importance (IOI) (Saaty and Vargas, 1991). were also embedded as a DLL (Dynamic Link Library) file (Phan,
IOI Description 2004) to realize matrix sensitivity calculation of the AHP. Sum-
1 Equal importance
mary tables of simulation runs for matrix sensitivity analysis were
3 Moderate importance stored in a Microsoft Access database. The integration of these
5 Strong or essential importance programming environments ensures the enhanced performance of
7 Very strong or demonstrated importance spatial analysis in GIS.
9 Extreme importance
Fig. 3 shows the new AHP-SA2 framework. It consists of four
2, 4, 6, 8 Intermediate values
Reciprocals Values for inverse comparison modules with following functions:

where Count(IOI) is the number of IOIs within a changing range
CREV(Base_EV, R) according to different values of Base_EV and R,
respectively.
The total number of runs for sensitivity analysis of a given ma-
trix is:
X
n statistics extracted from those maps.
TotalRuns ¼ Runsi ; n ¼ C2N
i¼1
where n is the number of Base_EVs for matrix sensitivity analysis in
a decision problem, and Runsi which defined in Equation (2) is the
number of runs for each selected Base_EV. N is total number of
criteria. n equals to C2N , which represents the number of ways of
choosing two unordered outcomes from N possibilities.
Step 4. Once all runs for a selected Base_EV are completed, a
sequence of resultant MCDM maps and an associated tabular
summary statistics will be automatically generated. While the
maps provide spatial distribution of evaluation results for each
valid run based on the CR check, the table summarizes these sim-
ulations for the selected element (in terms of weights values, cell
numbers at different ranking levels, and changes of cells between
different ranking levels). These model outputs provide quantitative
information for users to analyze and assess the sensitivity of
element modification in a PCM on weight adjustment and changes
corresponding to MCDM evaluation results.
2.2. Tool development
A new module called “Matrix Sensitivity” was developed and
added to the AHP-SA framework (Chen et al., 2010). C# .Net com-
puter language was used to construct the framework of the soft-
ware. ArcGIS Engine 10 (ESRI, 2012) platform was utilised to
Table 2
Index of scale of intensity of importance
(IOI) between each two criteria.
Index IOI
0 1/9
1 1/8
2 1/7
3 1/6
4 1/5
5 1/4
6 1/3
7 1/2
8 1
9 2
10 3
11 4 in the AHP-SA2 tool is demonstrated using spatial data from a study
12 5 in which multi-criteria land suitability assessment at a catchment
13 6 scale was conducted to identify the potential of expanding irrigated
14 7 cropping landuse. The study area is Macintyre Brook catchment of
15 8
16 9
(1) Main Interface: loading and displaying inputs and outputs, and
configuration of the framework. This window allows users to
select relevant criteria map layers as inputs for the decision
problem. All criteria layers have to be pre-processed as ESRI
integer grids with the same coordinate system, extent and
number of cells. There are two types of outputs. One is the
MCDM maps and the other is the summary tables showing
(3) (2) AHP Calculation Window: conducting standard AHP analysis to
generate the Base PCM. The hierarchical structure and the PCM
are constructed on the basis of selected criteria. The elements
in the Base PCM (Base_EVs) are assigned values from 1/9 to 9
(Table 1). The weights of all criteria are generated in this
window if the CR check is passed. A base evaluation map is
then automatically displayed in the Main Interface.
(3) Weight Sensitivity Window: undertaking weight sensitivity
analysis by directly changing the values of criteria weights
calculated from a PCM as was discussed in Chen et al. (2010).
(4) Matrix Sensitivity Window: performing weight sensitivity
analysis by changing the Base_EVs. Users need to input three
parameters: 1) selecting a Base_EV to be changed according to
CoC and CoR in the Base PCM, 2) defining a changing radius (R)
based on the base IOI value of the Base_EV, and 3) specifying
the path of the store folder and the name of outputs from the
analysis. Because the model can automatically detect the base
IOI value of the selected Base_EV and its equivalent value for
IndexB so as to determine the change range (CREV(Base_EV,
R) for the selected Base_EV using Equation (1), a list of rec-
ommended R values will be selectable to help users with
assigning a reasonable R. The greater the R is, the more un-
certain the users are. By clicking the Calculate button in this
window, designated resultant MCDM map layers from different
runs based on a sequence of modified matrixes will be gener-
ated and visualized in the map frame of Main Interface win-
dow. A table which summarises relevant statistics on these
runs will also be simultaneously generated.
By repeating the operations in function module (4) through
reselecting another Base_EV from the AHP matrix generated from
function module (2), and reconfigure the SA parameters in the
Matrix Sensitivity window, users can change as many elements as
they want in the Base PCM, and run a series of decision scenarios
simulations. The total numbers of iterative runs for the matrix
sensitivity analysis of a given decision problem can be computed
using Equation (3).
3. Case study
The application of the new Matrix Sensitivity function module
Queensland in Australia. It is located at southern Queensland near
the state border with New South Wales (Fig. 4). The catchment
 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140
Fig. 3. Steps of using Matrix Sensitivity function in AHP-SA2 tool.
covers an area of 4200 km2. It is relatively flat in the western area,
with undulations becoming steeper towards east and northeast.
The Macintyre Brook flows from east to west, and its tributaries are
the main source of surface water for the region. The region is not
well endowed with groundwater. Coolmunda Dam supplies irri-
gation water to Macintyre Brook along which the main irrigation
areas of the catchment are located. Daily temperatures vary from
18  C to 32  C in summer and 4  C to 18  C in winter. The monthly
average evaporation ranges from 50 mm to 200 mm, while the
average annual rainfall is 640 mm. Most of the rainfall falls between
October and March. The catchment is characterised by extremely
diverse soil types and topography, making it suitable for a wide
variety of land uses and rural production. Currently about 1.5% of
the catchment area is devoted to irrigated cropping and perennial
horticulture, as well as sown pastures. The remainder is dominated
by dryland cropping (3%), native pasture grazing country (80%) and
State Forest Reserves (15%). The main crops include fodder
(lucerne), maize, sorghum, peas, and orchard such as peach, plum
and apricot (Chen et al., 2009).
3.1. Base run
Selection of evaluation criteria in this study was based on project
objective, spatial scale, and in particular, data availability. Five
criteria were available: hydraulic conductivity of soil (Ks), percent
slope (S), soil texture (ST), depth to water-table (DTW), and electrical
conductivity of groundwater (ECw). They were classified into
different suitability levels. The suitability classes used in this study
were adapted from the FAO (Food and Agriculture Organization of
the United Nations) system. They are stated as: highly suitable (S1),
133
moderately suitable (S2), marginally suitable (S3) and unsuitable
(N). Table 3 gives the threshold values of evaluation criteria for each
of the four suitability classes which were determined based on
literature survey and expert opinion (Chen et al., 2009).
Spatial data were converted into raster layers and processed in
ArcGIS. Slope was generated from a 25 m resolution DEM, which
was resampled to match the other four datasets at 100 m cell size.
The five criteria datasets were then classified into four classes as
integer rasters representing different suitability levels based on
assigned threshold values in Table 3. They were loaded into the map
frame of the Main Interface (Fig. 5a) which has a layer control panel
in the left and a map frame in the right side to provide visualized
operation of reading and displaying criterion layers and handling
MCDM mapping. All criteria were compared against each other in
the AHP window (Fig. 5b) which has the capability to list criteria
and to build the Base PCM (Fig. 2). Each element in the Base PCM is
a Base_EV with a base IOI from Table 1, which is a measure to ex-
press the relative importance among the criteria. The weighting
coefficients of criteria were automatically derived based on the
constructed Base PCM (Fig. 5b). The weighted criterion maps in
Fig. 4 were then aggregated, according to the derived base criteria
weights, to produce a final suitability map displayed in Main
Interface (Fig. 5c) if users are confident with the IOIs in the Base
PCM in Fig. 5b. The resultant map shows spatial pattern and dis-
tribution of the land suitability classes. The most suitable locations
are shown in dark green, and the unsuitable lands are shown in
dark brown. However, it is often the case that users are not entirely
sure about their input IOI value for one, or more, Base_EV in the
Base PCM. A sensitivity analysis will help them to make decisions.
Particularly in this case study, the decision makers were assumed
134 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140

Fig. 4. Location of the study area and criterion maps used for the evaluation of the irrigated croplands. The suitability levels are classified based on the threshold values in Table 3.

uncertain about 1) if the Base_EV defined by ECw (CoR4) and DTW
(CoC3) should be given a value of 2 which means ECw is slightly
important than DTW, because it is hard to decide which one is more
important than the other, and if so by how much, in practise; 2) if
the IOI of the Base_EV defined by ST (CoR2) and ECw (CoC4) should
equals to 4, instead of 1, or 2, or 3, or even 0.5. Similarly, the inputs
between soil properties (ST and Ks) and groundwater characteris-
tics (DTW and ECw) should also be examined in order to gain
confidence on decision making. These can be addressed using the
two sensitivity analysis functions in the AHP-SA2 framework. The
first one is Weight Sensitivity module (Fig. 5d) discussed in Chen
et al. (2010); and the second one is the Matrix Sensitivity tool
(Fig. 5e) focused in this study.
3.2. Matrix sensitivity analysis
Matrix sensitivity analysis should be conducted after at least one
base run which calculates base criteria weights and produces a
suitability map as a base line for sensitivity analysis. Fig. 6 dem-
onstrates an example of matrix sensitivity analysis using the Matrix
Sensitivity tool. The process was undertaken on the basis of
element selection and user-defined parameters:
 Accessing to the Matrix Sensitivity Window from the Main
Interface (Fig. 5a).
 Choosing CoR, e.g. ST, and CoC, e.g. ECw (Fig. 6a).
 The Base_EVs of the selected element pair were automatically
picked from the Base PCM and displayed (Fig. 6b).
 Defining a changing radius (R). It determines IOIs in the CREV
(Equation (1)), or number of simulation runs, for the selected
pair of criteria (Fig. 6b).
 A series of IOIs in the CREV were generated according to
Equation (1)(Fig. 6b). IOIs which are fractions in Table 2 are
represented by decimals, e.g. 0.125 representing 1/8.
 Specifying the output path and the prefix name of the resultant
maps (Fig. 6c).
 Clicking “Calculate” button to perform iterative runs (Fig. 6d).
According to Equation (2), the number of iterative runs is the
same as the number of IOIs in the CREV. Theoretically we
should have 11 runs in total, including one base run and 10
simulation runs, due to R ¼ 5. However, a CR value of the
updated PCM was checked for each run. If CR value is accept-
able, a resultant suitability map will be generated. Otherwise,
no map will be created for that run and the process will
automatically jump to the next run.

Table 3
Criteria for suitability assessment of irrigated croplands.

Highly suitable Moderately suitable Marginally suitable Unsuitable

S (%) 0e2 2e4 4e8 >8

ST Fine to medium Heavy clay Coarse or poorly drained Very coarse or shallow depth
DTW (m) >4 3e4 2e3 <2
ECw (dS/m) 0e0.5 0.5e2 2e5 >5 (if depth < 4m)
Ks (m/d) 0.3e1 0.05e0.3 or 1e2 2e2.5 < 0.05 or > 2.5
 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140 135

Fig. 5. AHP-SA2 tool interface. (a) Main Interface: Map Control, AHP, Weight Sensitivity and Matrix Sensitivity; (b) AHP window: Base_EVs 0.2, 0.25, 0.3333, 0.5 in the Base PCM
representing 1/5, 1/4, 1/3, 1/2, respectively; (c) Layer Control Window in Main Interface: displaying results from the base run; and (d) Weight Sensitivity Window: directly changing
criteria weights; and (e) Matrix Sensitivity Window: changing criteria weights through the Base PCM.

 Producing a stack of resultant suitability evaluation maps for The results in Table 4 are displayed in Fig. 8a. Plots in Fig. 8 present
sensitivity assessment in the Main Interface (Fig. 6e). These all possible changes for each CoR/CoC combination with a fixed R
maps are relative to a series of seven executed/valid simulation value of 5.
runs (Fig. 7).

A table of summary statistics calculated from each selected CoR/
CoC will be exported from the Matrix Sensitivity tool. Results of the
seven simulation runs for the selected ST/ECw were summarized in
Table 4 and mapped in Fig. 7. Each row in the table is an iterative
run. Detailed statistics information, including IOI, weight values,
number of cells in each ranking class (S1 to N), and changes in
number of cells between different ranking classes from base run are
listed. If CR is unacceptable for the updated PCM of a simulation, e.g.
those IOI ¼ 1/3, 1/2, 7, 8 and 9, that row will be marked as “CR
Unaccepted” without resultant maps and summary information.
3.3. Discussion
The simulated evaluation maps (Fig. 7) exported from the AHP
tool can demonstrate the added value of map-based visualization
to GIS-MCDM problem solving in general. They provide insights
into the spatial patterns of modelling results and where the most
significant land suitability classification changes take place cor-
responding to each simulation run. It can be easily observed from
Fig. 7 in combination with Fig. 8a that S1 and S2 are relatively
stable despite a certain degree of variations. The ranks of most
136 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140

Fig. 6. The screen shots illustrating matrix sensitivity analysis process step by step.

n  
cells in these two suitable levels remained the same, or slightly
X 
changed. There are no significant changes in S3 and N when Overall change rate ¼ CellNumi  CellNum_BaseRuni 
 CellNum_BaseRun 
decreasing the value of the Base_EV (Fig. 7a and b). Among the i¼1 i
seven valid runs, the most significant suitability class modification (4)
from S1 to S2 and S3 to N occur when the IOI is less or equal to 1
(Fig. 7a) and greater or equal to 5 (Fig. 7c and d), respectively. A
considerable cell exchange between S2 and S3 is noted when IOI where i is the sequence number of different suitability level,
changes from 4 (base run) to 5 (Fig. 7c). Fig. 8 shows the overall CellNumi is the cell number in the map relative to each suitability
degree of change in different simulation runs which was estimated level of each simulation run, CellNum_BaseRuni is the cell number
using an index defined as: relative to each suitability level of the base run. The value of the
 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140 137

Fig. 7. Irrigated cropland suitability maps of seven simulation runs for the selected element of ST/ECw (CoR/CoC).

overall change rate is in the range of 0 and 4. A greater value
indicates a higher degree of change in comparison with the base
run.
With the aid of the summary table generated from the SA
simulation runs (Table 4) and the chart presentations of Matrix SA
results (Fig. 8), a general summary of all simulation runs can be
rapidly derived:
 For most elements defined by CoR/CoC in the matrix, there are
no cells that either increased or decreased more than one
suitability level/class from their original rank of the base run.
 The CoR/CoC defined by S/Ks has the highest sensitivity. This is
because Ks and S are the two criteria with highest weight
values in the base run (Fig. 4).
 The S/DTW and DTW/ECw appear least sensitive to changes.
There are almost no cell shifts between classes. The fact that
the perturbation of the decision weights, in particular DTW and
ECw, has a small impact on the ranking of the most cells in all
suitability classes infers that the degree of domination of these
cells is almost independent of changes in the decision weights
associated with these selected criteria.
 ST/DTW and ST/ECw have a relatively similar sensitivity
pattern; DTW/Ks and ECw/Ks are almost the same. These
reveal DTW and ECw have played the same roles in the MCDM
process.
 S/ECw and S/Ks have similar degree of sensitivity with the or-
der of S/Ks greater than S/ECw. This follows the order of criteria
weights (Fig. 4) in the base run.

Table 4
Example of the summary table generated from the 11 SA simulation runs for the element of ST (CoR) to ECw (CoC), showing the 10 SA simulation runs plus the base run (bold).

I OI Weights values Cells in resultant map Changes in resultant map

ST/Ecw S ST DTW Ecw Ks S1 S2 S3 N S1 to S2 S2 to S1 S2 to S3 S3 to S2 S3 to N N to S3

1/3 CR unaccepted
1/2 0.2504 0.1552 0.0593 0.1447 0.3903 9568 158,447 198,646 48,071 15,463 268 373 54 0 645
1 0. 2513 0.1734 0. 0598 0. 1189 0. 3965 9568 158,440 198,653 48,071 15,463 268 373 47 0 645
2 0.2500 0.1937 0.0592 0.1015 0.3957 24,763 143,480 197,773 48,716 0 0 1 37 46 0 0
3 0.2482 0.2085 0.0581 0.0934 0.3918 24,763 143,434 197,819 48,716 0 0 1 37 0 0 0
4 0.2463 0.2210 0.0571 0.0881 0.3874 24,763 143,571 197,682 48,716 0 0 0 0 0 0
5 0.2445 0.2322 0.0561 0.0841 0.3831 24,763 139,291 135,102 115,576 0 0 4280 0 66,860 0
6 0.2428 0.2423 0.0552 0.0809 0.3788 24,763 135,227 139,166 115,576 0 0 8344 0 66,860 0
7 CR unaccepted
8 CR unaccepted
9 CR unaccepted
138 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140

 The biggest variation of evaluation classification happens in S3

and N. There is a dramatic change in the number of cells in
these two classes. Most big shifts between S3 and N take place
within R ¼ 1.
 S/ST is a special case in terms of the low tolerance for any IOI
changes greater than the base IOI. When both CoR and CoC are
more important, CoR/CoC will have higher sensitivity. There-
fore, the thresholds and IOIs for important criteria must be

Fig. 8. Summary charts of the total number of runs for sensitivity analysis of the case study (R ¼ 5). Each CoR/CoC has 11 simulation runs centered with a base run in brackets.
very carefully defined due to their extreme sensitivity to
perturbations.

The above quantitative analysis has provided stakeholders with

crucial inputs in reducing uncertainty in decision making by
increasing stability of the output of the land suitability assessment
in the catchment. The evaluation results derived from the base run
could be used to analyze potential expansion of irrigated cropland
because, when water is available, the highest possibility of such
expansion will only occur on S1 (highly suitable) and S2 (moder-
ately suitable) lands which are relatively stable from the matrix
sensitivity investigation in this study. The simulation results of the
base run have shown that there is about 7% of total catchment area
being classified as highly suitable (S1), unsuitable land (N) covers
about 32%, and moderately and marginally suitable classes (S2 and
S3) represent 25% and 36% of land area, respectively. The S1 is found
mainly on the flood plain of the Macintyre Brook and other big
creeks, as well as alluvial fans and flats of smaller streams, where
varying areas of land with better drained soils are suitable for
cultivation. These potentially irrigable lands are made up of four
soil types distinguished by texture which include alluvial sandy
loam, alluvial silty loam, clay loam and sandy loam. Generally, most
unsuitable areas are located in the east-southeast part of the
catchment where the surface is undulating, soil texture is poor and
soil hydraulic conductivity is very low. A large proportion of this
land is under grazing pasture landuse, only a small portion of it is
used for agriculture forestry.

4. Concluding remarks

This study has developed a new methodology to examine criteria

weight sensitivity in the AHP based MCDM. A new “Matrix Sensi-
tivity” module has been added to its predecessor AHP-SA model
(Chen et al., 2010) to form an extended spatial framework AHP-SA2
for a comprehensive weight sensitivity analysis that brings the
multi-criteria evaluation process into the decision-makers’ realm.
The key functionality of the AHP-SA2 is demonstrated using a case
study. With unique, flexible capabilities to quantitatively examine
weight sensitivity, this new tool is even more transparent and
efficient than its previously published version. Its transparency is
improved by the feature of directly changing element values in the
pairwise comparison matrix, which also makes the corresponding
CR checking and MCDM results immediately visible. Its efficiency is
increased by evaluating the CR value from its re-calculation after
each adjustment of the PCM. If a CR value exceeds the threshold, the
PCM will be marked as unqualified. This will remove unreasonable
situation so as to reduce the redundancy of weight sensitivity
analysis. In addition, the changing radius R can limit the range of
user-defined simulations and eliminates unnecessary runs. There-
fore, the new AHP-SA2 model is more reliable and effective, which
can further reveal potential relationships between criteria and their
weight sensitivities in MCDM problems with more informative re-
sults and less evaluation runs.
This study utilizes the OAT as the SA method. Although the OAT
has drawbacks as indicated in some literature (Czitrom, 1999; Wu
and Hamada, 2000; Haaker and Verheijen, 2004; Saltelli and
Annoni, 2010; Sun et al., 2012), it is feasible in a wide range of
 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140
applications. The reason to employ the OAT in this study lies into
three major aspects: 1) the OAT is easy to use. Because most
stakeholders involved in the MCDM process are not expert in
modelling world, the OAT method is suitable for improving stake-
holder engagement through facilitating a better interaction with
them. 2) The OAT changes element values in an AHP matrix one at a
time. Each element in the matrix represents a pair-wise compari-
son between two criteria which are independent to other criteria.
This characteristic of the AHP, in turn, satisfies the independence
requirement of the OAT. Saltelli and Annoni (2010) introduced
several suggested approaches, such as “elementary effects
method”, which vary several factors simultaneously. But these
methods are not applicable in this study since elements in the AHP
matrix are independently evaluated. Simultaneously changing
values of more than one element is not meaningful. 3) The results
generated from the OAT are easy to understand. Because of the
complexity in the spatial distributions of land suitability, the
exploration of the relationships between criteria and results re-
quires the OAT which is suitable to analyze complex problems with
the focus on that all the changes in spatial distribution are effected
by only one specified criterion under a certain condition (Saltelli
and Annoni, 2010).
The AHP-SA2 is a powerful, easy-to-use spatial decision support
tool designed to help easier access spatial models, to handle with
complex structures, visualize and combine mapped information in
an intuitive, flexible, interactive way. It delivers direct feedback to
evaluators or modellers, in particular non-experts, for an easier
understanding of how changes in the AHP pairwise comparison
matrix affect MCDM outcomes spatially and quantitatively. Origi-
nally developed for land suitability evaluation, its flexibility and
interdisciplinary nature does not limit its use: AHP-SA2 can be
efficiently applied in many other application domains. It can be
used to assist decision makers with making justifiable decisions for
solving complex issues in natural resources management and
environmental studies based on informed, systematic and trans-
parent analysis of diverse environmental, social and economic in-
formation along with value judgements and policy and
management goals. The software tool has been released for free
download and will be maintained in the future. The limitation of
the AHP-SA2 is that the data of criteria layers need to be inde-
pendently prepared before input into the tool. The pre-processing
can be conducted using ArcGIS software, which will be integrated
into the tool in the future.
It has to be emphasised that SA needs to be regarded as a basic
step in an MCDM process. The AHP-SA2 model developed in this
study forms an integrated platform which can be utilized to make
sensitivity analysis in the AHP method in a spatial context. It is
particularly useful to the evaluation of impacts of defining or
changing weights of a large number of input criteria on MCDM
outputs. The two GIS-based modules in the AHP-SA2, Weight
Sensitivity and Matrix Sensitivity, provide end users an alternative
way in SA practices. The choice of using either or both modules
depends on if it is fit-for-purpose. It is expected that further
development in SA analysis on changing criteria thresholds will
enrich the SA toolset and construct a more practical framework to
solve SA problems for supporting decision-making and policy
development in natural resources management, sustainable agri-
culture and environmental studies.
Acknowledgements
The study was funded by National Natural Science Foundation of
China (41201548), Open Research Funding Program of KLGIS
(KLGIS2011A05) and supported by Innovation Program of Shanghai
Municipal Education Commission (12YZ086). We would like to
139
thank the Department of Environment and Resource Management
of the Queensland Government, Australia, for providing data of the
case study. Thanks also to Sue Cuddy and two anonymous internal
reviewers at CSIRO Land and Water for reviewing and providing
comments on the manuscript.
References
Allaire, D.L., Willcox, K.E., 2012. A variance-based sensitivity index function for
factor prioritization. Reliability Engineering and System Safety 107, 107e114.
Archer, G.E.B., Saltelli, A., Sobol, I.M., 1997. Sensitivity measures, ANOVA-like tech-
niques and the use of bootstrap. Journal of Statistical Computing and Simulation
58, 99e120.
Butler, J., Jia, J.M., Dyer, J., 1997. Simulation techniques for the sensitivity analysis of
multi-criteria decision models. European Journal of Operational Research 103
(3), 531e546.
Campolongo, F., Cariboni, J., Saltelli, A., 2007. An effective screening design for
sensitivity analysis of large models. Environmental Modelling and Software 22,
1509e1518.
Chen, H., Wood, M.D., Linstead, C., Maltby, E., 2011. Uncertainty analysis in a GIS-
based multi-criteria analysis tool for river catchment management. Environ-
mental Modelling and Software 26, 395e405.
Chen, Y., Khan, S., Paydar, Z., 2009. To retire or expand? A fuzzy GIS-based spatial
multi-criteria evaluation framework for irrigated agriculture. Irrigation and
Drainage 59 (2), 174e188.
Chen, Y., Yu, J., Khan, S., 2010. Spatial sensitivity analysis of multi-criteria weights in
GIS-based land suitability evaluation. Environmental Modelling & Software 25,
1582e1591.
Costa, B.E., Carlos, A., 1988. A methodology for sensitivity analysis in three-criteria
problems: a case study in municipal management. European Journal of Oper-
ational Research 33 (2), 159e173.
Crosetto, M., Tarantola, S., 2001. Uncertainty and sensitivity analysis: tools for GIS
based model implementation. International Journal of Geographical Informa-
tion Science 15 (5), 415e437.
Crosetto, M., Tarantola, S., Saltelli, A., 2000. Sensitivity and uncertainty analysis
in spatial modeling based on GIS. Agriculture Ecosystems & Environment 81,
71e79.
Cui, L., Lu, Z., Wang, Q., 2012. Parametric sensitivity analysis of the importance
measure. Mechanical Systems and Signal Processing 28, 482e491.
Cukier, R.I., Levine, H.B., Shuler, K.E., 1978. Nonlinear sensitivity analysis of multi-
parameter model systems. Journal of Computational Physics 26, 1e42.
Czitrom, V., 1999. One-Factor-at-a-Time versus designed experiments. The Amer-
ican Statistician 53 (2), 126e131.
Delgado, M.G., Sendra, J.B., 2004. Sensitivity analysis in multicriteria spatial
decision-making: a review. Human and Ecological Risk Assessment 10, 1173e
1187.
Dimov, I., Georgeva, R., Ostromsky, T., 2012. Monte Carlo sensitivity analysis of an
Eulerian large-scale air pollution model. Reliability Engineering & System Safety
107, 23e28.
Doos, B.R., Shaw, R., 1999. Can we predict the future food production? A sensitivity
analysis. Global Environmental Change-Human and Policy Dimensions 9 (4),
261e283.
Drouet, J.L., Capian, N., Fiorelli, J.L., Blanfort, V., Capitaine, M., Duretz, S., Gabrielle, B.,
Martin, R., Lardy, R., Cellier, P., Soussana, J.F., 2011. Sensitivity analysis for
models of greenhouse gas emissions at farm level. Case study of N2O emissions
simulated by the CERES-EGC model. Environmental Pollution 159, 3156e3161.
ESRI, 2012. ArcGIS for Developers e an Introduction. http://www.esri.com.
Feick, R., Hall, G.B., 2004. A method for examining the spatial dimension of mul-
ticriteria weight sensitivity. International Journal of Geographical Information
Science 18 (8), 815e840.
Flores-Alsina, X., Corominas, L., Neumann, M.B., Vanrolleghem, P.A., 2012. Assessing
the use of activated sludge process design guidelines in wastewater treatment
plant projects: a methodology based on global sensitivity analysis. Environ-
mental Modelling & Software 38, 50e58.
Frey, H.C., Patil, R., 2002. Identification and review of sensitivity analysis methods.
Risk Analysis 22 (3), 553e577.
Grayson, R., Kay, P., Foulger, M., 2008. The use of GIS and multi-criteria evaluation
(MCE) to identify agricultural land management practices which cause surface
water pollution in drinking water supply catchments. Water Science and
Technology 58 (9), 1797e1802.
Haaker, M.P.R., Verheijen, P.J.T., 2004. Local and global sensitivity analysis for a
reactor design with parameter uncertainty. Chemical Engineering Research and
Design 82 (5), 591e598.
Hill, M.J., Braaten, R., Veitch, S.M., Lees, B.G., Sharma, S., 2005. Multi-criteria deci-
sion analysis in spatial decision support: the ASSESS analytic hierarchy process
and the role of quantitative methods and spatially explicit analysis. Environ-
mental Modelling & Software 20, 955e976.
Hornberger, G., Spear, R., 1981. An approach to the preliminary analysis of envi-
ronmental systems. Journal of Environmental Management 7, 7e18.
Hwang, D., Byun, D.W., Odman, M.T., 1997. An automatic differentiation technique
for sensitivity analysis of numerical advection schemes in air quality models.
Atmospheric Environment 31 (6), 879e888.
140 Y. Chen et al. / Environmental Modelling & Software 48 (2013) 129e140
Hyde, K.M., Maier, H.R., 2006. Distance-based and stochastic uncertainty analysis
for multi-criteria decision analysis in Excel using Visual Basic for applications.
Environmental Modelling & Software 21, 1695e1710.
Hyde, K.M., Maier, H.R., Colby, C.B., 2005. A distance-based uncertainty analysis
approach to multi-criteria decision analysis for water resource decision making.
Journal of Environmental Management 77, 278e290.
Jiang, H., Eastman, J.R., 2000. Application of fuzzy measures in multi-criteria eval-
uation in GIS. International Journal of Geographical Information Science 14 (2),
173e184.
Kleijnen, J.P.C., 1997. Sensitivity analysis and related analyses: a review of some
statistical techniques. Journal of Statistical Computation and Simulation 57,
111e142.
Manache, G., Melching, C.S., 2008. Identification of reliable regression- and
correlation-based sensitivity measures for importance ranking of water-quality
model parameters. Environmental Modelling & Software 23, 549e562.
Mara, T., Tarantola, S., 2012. Variance-based sensitivity indices for models with
dependent inputs. Reliability Engineering & System Safety 107, 115e121.
Mendicino, G., 1999. Sensitivity analysis on GIS procedures for the estimate of soil
erosion risk. Natural Hazards 20 (2e3), 231e253.
Morris, M.D., 1991. Factorial sampling plants for preliminary computational ex-
periments. Technometrics 33, 161e174.
Nossent, J., Elsen, P., Bauwens, W., 2011. Sobol’ sensitivity analysis of a complex
environmental model. Environmental Modelling & Software 26, 1515e1525.
Oakley, J., O’Hagan, A., 2004. Probabilistic sensitivity analysis of complex models: a
Bayesian approach. Journal of the Royal Statistical Society: Series B 66, 751e769.
Ohta, K., Kobashi, G., Takato, S., Kagaya, S., Yamada, H., Minakami, H., Yamamura, E.,
2007. Analysis of the geographical accessibility of neurosurgical emergency
hospitals in Sapporo city using GIS and AHP. International Journal of
Geographical Information Science 21 (6), 687e698.
Phan, J., 2004. MATLAB C# Book. LePhan Publishing.
Pisoni, E., Carnevale, C., Volta, M., 2010. Sensitivity to spatial resolution of modeling
systems designing air quality control policies. Environmental Modelling &
Software 25, 66e73.
Rahman, M.A., Rusteberg, B., Gogu, R.C., Lobo Ferreira, J.P., Sauter, M., 2012. A new
spatial multi-criteria decision support tool for site selection for implementation
of managed aquifer recharge. Journal of Environmental Management 99, 61e75.
Ravalico, J.K., Dandy, G.C., Maier, H.R., 2010. Management Option Rank Equivalence
(MORE) e a new method of sensitivity analysis for decision-making. Environ-
mental Modelling & Software 25, 171e181.
Refsgaard, J.C., van der Sluijs, J.P., Højberg, A.J., Vanrolleghem, P.A., 2007. Uncer-
tainty in the environmental modelling process e a framework and guidance.
Environmental Modelling & Software 22, 1543e1556.
Saaty, T.L., 1977. A scaling method for priorities in hierarchical structures. Journal of
Mathematical Psychology 15, 231e281.
Saaty, T.L., 1980. The Analytical Hierarchy Process. McGraw Hill, New York.
Saaty, T.L., Vargas, L.G., 1991. Prediction, Projection and Forecasting. Kluwer Aca-
demic Publisher, Dordrecht, p. 251.
Saltelli, A., Chan, K., Scott, M., 2000. Sensitivity Analysis, Probability and Statistics
Series. John Wiley & Sons, New York.
Saltelli, A., 2002a. Sensitivity analysis for importance assessment. Risk Analysis 22
(3), 1e12.
Saltelli, A., 2002b. Making best use of model evaluations to compute sensitivity
indices. Computer Physics Communication 145 (2), 280e297.
Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Envi-
ronmental Modelling & Software 25, 1508e1517.
Sobol’, I.M., 1993. Sensitivity estimates for nonlinear mathematical models. Math-
ematical Modeling and Computation 1 (4), 407e414.
Store, R., Kangas, J., 2001. Integrating spatial multi-criteria evaluation and expert
knowledge for GIS-based habitat suitability modelling. Landscaping and Urban
Planning 55 (2), 79e93.
Svoray, T., Bar, P., Bannet, T., 2005. Urban land-use allocation in a Mediterranean
ecotone: habitat Heterogeneity Model incorporated in a GIS using a multi-
criteria mechanism. Landscape and Urban Planning 72 (4), 337e351.
Sun, X.Y., Newham, L.T.H., Croke, B.F.W., Norton, J.P., 2012. Three complementary
methods for sensitivity analysis of a water quality model. Environmental
Modelling & Software 37, 19e29.
Walker, W.E., Harremoes, P., Rotmans, J., van der Sluijs, J.P., van Asselt, M.B.A.,
Janssen, P., Krayer von Krauss, M.P., 2003. Defining uncertainty: a conceptual
basis for uncertainty management in model-based decision support. Integrated
Assessment 4, 5e17.
Wolf, J., Evans, L.G., Semenov, M.A., Eckersten, H., Iglesias, A., 1996. Comparison of
wheat simulation models under climate change .1. Model calibration and
sensitivity analyses. Climate Research 7 (3), 253e270.
Wu, C.F.J., Hamada, M., 2000. Experiments: Planning, Design, and Parameter Opti-
mization. Wiley, New York.
Wu, F., 1998. SimLand: a prototype to simulate land conversion through the inte-
grated GIS and CA with AHP-derived transition rules. Geographical Information
Science 12 (1), 63e82.
Zhang, C., Song, X., Xia, J., Tong, C., 2013. An efficient integrated approach for global
sensitivity analysis of hydrological model parameters. Environmental Modelling
& Software 41, 39e52.
Zio, E., Pedroni, N., 2012. Monte Carlo simulation-based sensitivity analysis of the
model of a thermal-hydraulic passive system. Reliability Engineering & System
Safety 107, 90e106.