Computers & Geosciences 81 (2015) 1–11

Contents lists available at ScienceDirect

Computers & Geosciences

journal homepage: www.elsevier.com/locate/cageo

Evaluating machine learning and statistical prediction techniques for

landslide susceptibility modeling
J.N. Goetz a,b,d,n, A. Brenning d,b, H. Petschko c, P. Leopold a
a
Health and Environment Department, AIT-Austrian Institute of Technology GmbH, Konrad-Lorenz-Straße 24, 3430 Tulln, Austria
b
Department of Geography and Environmental Management, University of Waterloo, 200 University Avenue West, Waterloo, Ontario, Canada N2L 3G1
c
Department of Geography and Regional Research, University of Vienna, Universitätsstraße 7, A-1010 Vienna, Austria
d
Department of Geography, Friedrich Schiller University Jena, Loebdergraben 32, 07743 Jena, Germany

art ic l e i nf o a b s t r a c t

Article history: Statistical and now machine learning prediction methods have been gaining popularity in the field of
Received 6 June 2014 landslide susceptibility modeling. Particularly, these data driven approaches show promise when tack-
Received in revised form ling the challenge of mapping landslide prone areas for large regions, which may not have sufficient
14 January 2015
geotechnical data to conduct physically-based methods. Currently, there is no best method for empirical
Accepted 16 April 2015
Available online 20 April 2015
susceptibility modeling. Therefore, this study presents a comparison of traditional statistical and novel
machine learning models applied for regional scale landslide susceptibility modeling. These methods
Keywords: were evaluated by spatial k-fold cross-validation estimation of the predictive performance, assessment of
Statistical and machine learning techniques variable importance for gaining insights into model behavior and by the appearance of the prediction (i.e.
Landslide susceptibility modeling
susceptibility) map. The modeling techniques applied were logistic regression (GLM), generalized ad-
Spatial cross-validation
ditive models (GAM), weights of evidence (WOE), the support vector machine (SVM), random forest
Variable importance
classification (RF), and bootstrap aggregated classification trees (bundling) with penalized discriminant
analysis (BPLDA). These modeling methods were tested for three areas in the province of Lower Austria,
Austria. The areas are characterized by different geological and morphological settings.
Random forest and bundling classification techniques had the overall best predictive performances.
However, the performances of all modeling techniques were for the majority not significantly different
from each other; depending on the areas of interest, the overall median estimated area under the re-
ceiver operating characteristic curve (AUROC) differences ranged from 2.9 to 8.9 percentage points. The
overall median estimated true positive rate (TPR) measured at a 10% false positive rate (FPR) differences
ranged from 11 to 15pp. The relative importance of each predictor was generally different between the
modeling methods. However, slope angle, surface roughness and plan curvature were consistently highly
ranked variables. The prediction methods that create splits in the predictors (RF, BPLDA and WOE) re-
sulted in heterogeneous prediction maps full of spatial artifacts. In contrast, the GAM, GLM and SVM
produced smooth prediction surfaces. Overall, it is suggested that the framework of this model evalua-
tion approach can be applied to assist in selection of a suitable landslide susceptibility modeling tech-
nique.
& 2015 Elsevier Ltd. All rights reserved.

1. Introduction selection of quantitative methods applied for spatial modeling and

Mitigating the impacts of landslides remains a great challenge
for land-use planners and policy makers. Landslide susceptibility
models, which are used to derive maps of locations prone to
landslides, can support and enhance spatial planning decisions
focused on reducing landslides hazards. Currently there is a vast
n
Corresponding author at: Department of Geography, Friedrich Schiller Uni-
versity Jena, Loebdergraben 32, 07743 Jena, Germany.
E-mail address: jason.goetz@uni-jena.de (J.N. Goetz).
predicting landslide susceptibility (Chung and Fabbri, 1999; Guz-
zetti et al., 1999; Dai et al., 2002; van Westen et al., 2003; Bren-
ning, 2005; Goetz et al., 2011; Pradhan, 2013). Quantitative
methods for modeling landslide susceptibility can be generalized
into physically-based and statistical approaches (Soeters and van
Westen, 1996; van Westen et al., 1997). This study focuses on
statistical and machine learning techniques, which have become
common approaches for modeling landslide susceptibility over
large regions (van Westen et al., 1997; Brenning, 2005; Petschko
et al., 2014, Micheletti et al., 2014).
The basic assumption of the empirical approach is that future

http://dx.doi.org/10.1016/j.cageo.2015.04.007
0098-3004/& 2015 Elsevier Ltd. All rights reserved.
2 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
landslides are likely to occur in similar conditions of the past
(Varnes, 1984). Typically, a range of predictors (i.e., independent
variables) is used to represent landslide preparatory conditions
(van Westen et al., 2008). The exact relationship of the predictors
to the response (i.e., landslide presence/absence) is not always
well known a priori. In most cases, the predictors are proxies for
conditions and processes that are difficult to measure across large
regions (Pachauri and Pant, 1992; Guzzetti et al., 1999; Goetz et al.,
2011). The susceptibility model output is a prediction surface or
map that spatially represents the distribution of predicted values,
usually as probabilities distributed across grid cells.
The freedom of choice to decide which modeling method is
most suitable for a particular application is challenging. Numerous
comparisons of susceptibility modeling methods have been con-
ducted; yet no single best method has been identified (Brenning,
2005; Yesilnacar and Topal, 2005; Lee et al., 2007; Yilmaz, 2009,
2010; Yalcin et al., 2011; Goetz et al., 2011; Pradhan, 2013). The
search for the optimal susceptibility modeling method is a com-
plicated one and should not only consider model accuracy. Ro-
bustness to sampling variation and adequacy to describe processes
associated with landslides are also crucial model properties
(Frattini et al., 2010).
The simplest approach to select an optimal model for predic-
tion is to compare the error rates estimated from cross-validation,
where the modeling method with the lowest error estimate is
determined as the best one to use. This assessment on the pre-
diction performance is also viewed as essential for a model to have
any practical or scientific significance (Chung and Fabbri, 2003;
Guzzetti et al., 2006). There are a variety of measures to assess the
accuracy of landslide susceptibility models. Common ones are
derived from success rate curves, prediction rate curves (Chung
and Fabbri, 2003) or receiver operating characteristic (ROC) curves
(Brenning, 2005; Beguería, 2006; Gorsevski et al., 2006; Frattini
et al., 2010). It is necessary to carefully select a suitable perfor-
mance measure. Ideally, this measure should communicate per-
formance in the context of the model application (Brenning,
2012a). Performance should also be assessed using test sets that
are independent from the training set used to build the prediction
model, resampling-based estimation methods such as cross-vali-
dation being the state of the art (Brenning, 2012a): cross-valida-
tion utilizes the entire dataset for training and testing the model.
The ability to communicate model behavior is a desirable
quality for landslide susceptibility models (Brenning, 2012a).
Generally speaking, users feel more comfortable in the practical
application of a model if they understand how the model works.
The ability of a model to adequately describe the system behavior
can be assessed by determining how well the predictors represent
the processes associated with landslides (Frattini et al., 2010). In
statistical methods, this is relatively straight forward compared to
machine learning models. The model coefficients from generalized
linear models can be used to evaluate the relative importance of
landslide predictors (Dai and Lee, 2002; Ayalew and Yamagishi,
2005). Variable importance has also been estimated for regression
models by observing the relative frequencies of variable selection
when an automatic stepwise variable selection method has been
applied and tested with cross-validation (Brenning, 2009; Goetz
et al., 2011; Petschko et al., 2014). In contrast, the internal me-
chanisms defining the representation of response by the pre-
dictors are difficult to interpret in machine learning models be-
cause of their ‘black box’ nature. Micheletti et al. (2014) demon-
strated how some feature selection properties of different machine
learning techniques can be implemented to assess the relative
importance of variables for landslide susceptibility modeling.
However, since their approach applied features selection methods
only relevant to the corresponding machine learning technique,
making comparisons of variable importance with other modeling
techniques can be challenging. A standardized approach for com-
paring the relative variable importance of different modeling sta-
tistical and machine learning techniques for geospatial problems
was demonstrated by Brenning et al. (2012b). They assessed
variable importance using internal estimates of changes in error
rates by randomly permuting predictors in out-of-bag samples
(Breiman, 2001; Strobl et al., 2007).
There are many criteria that can be considered for model se-
lection in the context of landslide susceptibility (Brenning, 2012a).
This study focuses on one particular aspect, which is the predictive
performance. Therefore, a rigorous assessment of prediction per-
formance is performed on various statistical and machine learning
techniques in an attempt to determine the ‘best’ predictive model.
The modeling techniques include logistic regression (GLM), gen-
eralized additive models (GAM), weights of evidence (WOE), the
support vector machine (SVM), random forest classification (RF),
and bootstrap aggregated classification trees (bundling) with pe-
nalized discriminant analysis (BPLDA). The importance of pre-
dictor variables in each model is also analyzed to demonstrate
how a standard measure of variable importance can be applied to
communicate and compare model behavior, even when a model is
considered ‘black box’. The main objective of this paper is to de-
monstrate an approach to make a rigorous comparison of landslide
susceptibility models for the purpose of spatial prediction.
2. Materials and methods
2.1. Study area
Multiple areas of interest (AOIs) were selected to observe
model behavior under different landslide conditions. The model-
ing techniques were tested on AOIs that were each 50 km2 and
within the province of Lower Austria (Fig. 1). The Molasse AOI
(Fig. 1a) is located in a relatively low lying basin (o 300 m a.s.l). It
mainly consists of sand and clay sediments, sandstones, clays-
tones, and marls. These bedrock materials can be covered by
Quaternary gravels and eolian sediments (loess). Deep-seated and
shallow landslides occur in this area. The Austroalpine AOI
(Fig. 1b), which includes the Upper Austroalpine lithology units, is
made up of predominately steep terrain, and has the highest ele-
vations in Lower Austria (1000–2000 m a.s.l). The lithology is
dominated by limestone and dolomite rock, with some inter-
bedded strata of claystone and marl. Landslides in the Aus-
troalpine area are typically shallow. Generally, the Flysch AOI
(Fig. 1c) is very susceptible to landslide activity (Petschko et al.,
2014). This low mountain region has exceptionally undulating
terrain, and consists of sedimentary rocks that are made up of
layers of sandstones, marls and claystones. The main triggers for
landslides in Lower Austria are intense rainfall and rapid snow-
melt events (Schwenk, 1992; Schweigl and Hervás, 2009). For
more details on the lithology and geology of Lower Austria, please
refer to Gottschling (2006) and Wessely (2006).
2.2. Landslide inventory
The landslides in this analysis are a subset of an inventory for
Lower Austria that has been previously published by Petschko
et al. (2014), which consists of mapped initiation areas for deep-
seated and shallow landslides. These landslides were mapped in a
geographic information system (GIS) using topographic derivatives
(e.g. hillshade and slope angle maps) of an airborne laser scanner
(ALS) digital terrain model (DTM) with a 1 m
1 m spatial re-
solution, which was acquired between 2006 and 2009. The general
procedure for mapping these landslides was similar to Schulz
(2004, 2007). This inventory consists of points (single grid cells)
 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
Fig. 1. A lithological map of Lower Austria indicating (black rectangles) AOIs and landslide inventories.
that mark the landslide scarp. Only one point was mapped per
landslide to provide equal treatment of large and small landslide
occurrences, to reduce the effect of spatial autocorrelation be-
tween observations, to increase mapping effectiveness and to
avoid uncertainties in mapping landslide boundaries (Petschko
et al., 2014). Further details on the landslide inventory and its
quality can be found in Petschko et al. (2013, 2014). The numbers
of landslides for the three AOIs used in this study are 261 land-
slides in the Molasse area, 193 in the Austroalpine area and 285 in
the Flysch area.
2.3. Predictor variables
Terrain analysis commonly forms the basis of quantitative
landslide modeling (Dai and Lee, 2002; Gorsevski et al., 2006; Van
den Eeckhaut et al., 2006; Muenchow et al., 2012; Petschko et al.,
2014). Terrain attributes derived from elevation models function as
surrogates for surface processes and geophysical site conditions to
simplify complex meaningful geomorphological relationships
(Pachauri and Pant, 1992; Moore et al., 1991; Lineback Gritzner
et al., 2001). Goetz et al. (2011) demonstrated that empirical
models based on terrain attributes alone could outperform tradi-
tional physically based models such as SHALSTAB and the factor of
safety. The terrain attributes slope angle, elevation, profile curva-
ture, plan curvature, catchment area (C. area), catchment height (C.
height), convergence index (Conv. Ind.), topographic wetness
3
index (TWI; Beven and Kirkby, 1979), slope aspect and surface
roughness (SDS) were used as predictors in this study. These were
derived from a 10 m
10 m spatial resolution ALS-DTM, which is
also the resolution the modeling in the analysis was completed in.
All except surface roughness were processed from the DTM with
SAGA GIS (Conrad, 2006). Catchment area, which was extremely
skewed, was logarithmically transformed. Slope aspect was sepa-
rated into the ‘east-exposedness’ (sine transformation) and ‘north-
exposedness’ (cosine transformation; Brenning and Trombotto,
2006). Surface roughness can be used as variable that may dis-
tinguish between types of landslide activity (Glenn et al., 2006).
We define surface roughness as the variation of slope angle for a
given scale for this study (Atkinson and Massari, 1998). It was
calculated as the standard deviation of slope (SDS) in a 3
3
moving window. The predictor variables are summarized in Ta-
ble 1. Predictor variables sensitive to temporal changes, e.g. land-
use and rainfall, were not included in the analysis because the
landslide inventory does not have information on when the
landslides were triggered (Petschko et al., 2014). Therefore, a
meaningful empirical relationship to such temporally sensitive
predictors cannot be made.
In addition to the terrain attributes, lithological units (map
scale 1:200,000; Schnabel, 2002) were included as indicator
variables. The strong generalization provided by this map scale is
likely to result in a reduced contribution of lithology to the pre-
dictive performance.
4 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
Table 1
Descriptive statistics (median and interquartile range, in parentheses) of the predictor variables in each AOI and the sample size n for landslide and non-landslide points.
Variable Median (IQR) for landslide/non-landslide points
Flysch (n¼ 285)
Slope angle (deg) 21 (9)/15 (8)
Elevation (m) 507 (95)/541 (124)
Profile curv. (10  3 m  1)  4.8 (21.4)/0.5 (6.8)
Plan curv. (10  3 m  1)  6.1 (19.5)/0.4 (6.0)
C. area (log10 m2) 3.4 (0.4)/3.2 (0.5)
C. height (m) 32 (24)/21 (22)
Conv. Ind.  5.1 (18.7)/1.3 (25.4)
TWI 8.5 (1.3)/8.6 (1.5)
South( þ1)–north(  1)  0.45 (1.24)/  0.11 (1.50)
East( þ1)–west( 1) 0.19 (1.14)/  0.03 (1.22)
SDS 2.5 (1.9)/1.5 (1.5)
2.4. Susceptibility modeling techniques
Six statistical and machine learning techniques are compared
in this study: a generalized linear model (GLM) with stepwise
variable selection (Hosmer and Lemeshow, 2000); the generalized
additive model with stepwise variable selection (GAM; Hastie and
Tibshirani, 1990); weights of evidence (WOE; Bonham-Carter,
1994); the support vector machine (SVM; Moguerza and Muñoz,
2006); random forest (RF; Breiman, 2001); and bootstrap ag-
gregated classification trees (bundling) with penalized linear dis-
criminant analysis (BPLDA; Hastie et al., 1995; Hothorn and Lau-
sen, 2005). While each of these methods could potentially be used
with a variety of settings and procedures for model selection, we
choose configurations that we consider typical for most of their
applications.
The GLM with a logistic link function, or logistic regression, is
the most common statistical technique for prediction of landslide
susceptibility (Brenning, 2005). It was first applied to landslide
susceptibility modeling by Atkinson and Massari (1998) and
Guzzetti et al. (1999). In contrast, generalized additive models
(GAM) have recently been applied for landslide susceptibility
mapping (Goetz et al., 2011; Muenchow et al., 2012). A GAM is a
semi-parametric nonlinear extension of a GLM (Hastie and Tib-
shirani, 1990). In our application of the GLM and GAM we use
forward–backward stepwise variable selection based on the
Akaike Information Criterion (AIC; Akaike, 1974).
Weights of evidence (WOE) is a non-linear statistical technique
based on the log-linear form of the Bayesian probability model
(Bonham-Carter, 1994). Early application of WOE for landslide
susceptibility modeling was completed by Lee et al. (2002). The
weights, which represent the statistical relationship of the pre-
dictor variable to the response, are defined as the combination of
the natural logarithm of relative risk-the ratio of conditional
probabilities for the presence to absence of a response to a set of
discrete, categorical variables. The conditional probabilities are
calculated using Bayes' theorem. This technique requires con-
tinuous variables to be classified into a set of categories. We use an
automatic approach that ‘bins’ the data into quartiles, which is
believed to provide a reasonable trade-off between model flex-
ibility (more bins) and sufficient data availability for estimating
the weights. The WOE method assumes predictor variables are
conditionally independent (Bonham-Carter, 1994). Instead of sta-
tistically testing these assumptions (Bonham-Carter, 1994; Agter-
berg and Cheng, 2002), which is rarely done in practical applica-
tions and would require additional questionable assumptions such
as spatial independence, we use this technique as a purely pre-
dictive method and interpret predicted ‘probabilities’ only as a
relative index of susceptibility (Neuhäuser and Terhorst, 2007).
Molasse (n¼261) Austroalpine (n¼ 193)
15 (7)/7 (5) 28 (8)/27 (12)
291 (28)/310 (47) 537 (167)/666 (185)
0.4 (18.5)/0.0 (2.8)  9.7 (23.7)/  0.6 (9.1)
 2.1 (12.5)/0.2 (3.2)  11.6(24.6)/0.7 (9.1)
3.0 (0.4)/3.0 (0.5) 3.3 (0.5)/3.0 (0.5)
9 (8)/8 (10) 45 (37)/30 (40)
 3.2 (23.1)/2.3 (28.5)  5.5 (18.2)/4.2 (25.2)
8.5 (1.5)/9.4 (2.3) 8.2 (1.1)/7.7 (1.4)
0.65 (1.33)/0.40 (1.49)  0.25 (1.3)/0.23 (1.6)
 0.15 (1.02)/  0.01 (1.21) 0.50 (0.95)/0.09 (1.27)
2.8 (2.6)/0.86 (1.4) 2.4 (2.1)/1.6 (1.9)
The support vector machine (SVM) is a machine learning
technique that is based on discrimination of classes in a high-di-
mensional feature space that is generated through nonlinear
transformations of the predictors (Vapnik, 1998). In this high-di-
mensional space, a decision hyperplane is computed to separate
prediction classes. Brenning (2005) demonstrated the potential
use of SVM for susceptibility modeling. Our implementation of
SVM used the default parameter settings of the R package e1071;
the regularization parameter was C ¼1, and the kernel bandwidth
was γ ¼p  1, where p is the number of predictors. SVM parameter
tuning was not performed since this process does not necessarily
improve model performance and may result in poorly defined, or
highly variable optimal parameters when comparing different
cross-validation repetitions (Brenning et al., 2009, 2012).
Classification trees are a nonlinear technique for predicting a
response using a set of binary decision rules that determine class
assignment based on the predictors (Breiman et al., 1984). Random
forest (RF) is an ensemble technique that utilizes many classifi-
cation trees (a ‘forest’) to stabilize the model predictions (Breiman,
2001). These trees are fitted to resamples of the observations that
are selected randomly with replacement (bootstrap resampling).
Each decision of the tree is furthermore based on randomly se-
lected predictors. The prediction of class assignment is determined
by the majority voting among all trees, and the proportion of trees
in the ensemble that predict landslide presence can be used as an
index of landslide susceptibility. RF was recently applied for
landslide susceptibility modeling by Ließ et al. (2011).
Bundling is another ensemble classification tree technique
(Hothorn and Lausen, 2005). Like RF, bundling uses a bootstrap-
aggregation approach; however, an ancillary classifier is trained on
the part of the training set that is not included in the bootstrap
resample. This ancillary classifier is then used as an additional
predictor in constructing a tree. In this study we use penalized
linear discriminant analysis (PLDA) as the ancillary classifier with
the bundling approach (BPLDA). PLDA is a discriminant analysis
technique designed for high-dimensional data and correlated
predictors. It avoids overfitting by applying smoothing constraints
on coefficients of the predictor variables (Hastie et al., 1995).
Bundling with PLDA has not yet been applied for landslide sus-
ceptibility modeling; however, Brenning (2009) utilized it for an-
other geomorphological classification context, and Bundling with
other ancillary classifiers was included in a model comparison by
Brenning (2005).
2.4.1. Assessing prediction performance
The performances of the susceptibility models were estimated
with a repeated k-fold spatial cross-validation approach (Brenning
et al., 2012). This approach is similar to k-fold cross-validation,
 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
where the data is randomly partitioned into k disjoint sets, and
one set at a time is used for model testing while the combined
remaining k  1 sets are used for model training (e.g., James et al.,
2013). However, instead of partitioning the dataset into k random
subsets, spatial cross-validation splits the data into spatially dis-
joint sub-areas. In this study, these partitions were formed with
the k-means clustering algorithm (Ruß and Brenning, 2010). The
estimation of model performance was repeated 20 times using
k ¼5 spatial cross-validation folds. Each model training and testing
was based on the commonly applied 1:1 sampling strategy of
presence to absence of landslide initiation (Heckmann et al., 2014).
Other sampling strategies have been investigated by Heckmann
et al. (2014) and Regmi et al. (2014) for landslide susceptibility
modeling using logistic regression; however, they both conclude
that the sampling ratio of presence to absence of landslides does
not significantly influence the prediction accuracies.
The performance measure estimated with spatial cross-vali-
dation was the area under the receiver operating characteristic
(ROC) curve (AUROC). The ROC curve plots all possible true posi-
tive rates (TPR; sensitivity) to the corresponding false positive
rates (FPR, 1 – specificity). AUROC values close to 50% indicate no
discrimination, while an AUROC close to 100% indicates perfect
discrimination between binary prediction classes. The AUROC is
independent of a specific decision threshold (Zweig and Campbell,
1993; Beguería, 2006). In addition, to assess the ability of the
models to predict the occurrence of landslide initiation while
correctly classifying most non-landslide locations as stable, the
TPR at a low FPR of 10% (or a high specificity of 90%) was measured
(Brenning, 2012a).
The differences in model performance were tested for statis-
tical significance to show these differences may not only be caused
by random variability. The statistical comparison of AUROC and
TPR at 10% FPR performances for different susceptibility models
was based on the non-parametric Kruskal–Wallis test for sys-
tematic differences among a set of variables. The Wilcoxon–Mann–
Whitney rank sum test was then applied to individually detect
differences in model performances. An adjustment for multiple
comparisons was applied using the Benjamini–Hochberg proce-
dure to control the false discovery rate, i.e. the expected propor-
tion of falsely rejected null hypotheses among all rejected null
hypotheses (FDR; Benjamini and Hochberg, 1995; Brenning, 2009).
The modeling and statistical analysis was conducted entirely in
R, a free software environment for statistical computing (version
3.0.0; R Development Core Team, 2013), with the contributed
packages ‘sperrorest’ (Brenning, 2012b), ‘e1071’ (Dimitriadou et al.,
2007; Chang and Lin, 2011), ‘gam’ (Hastie, 2009), ‘ipred’ (Peters
and Hothorn, 2009), ‘mda’ (Hastie and Tibshirani, 2009), ‘ran-
domForest’ (Breiman and Cutler, 2012), ‘raster’ (Hijmans and van
Etten, 2013), and ‘ROCR’ (Sing et al., 2009).
2.4.2. Estimating which predictors are important
A permutation-based variable accuracy importance approach,
which computes how much a performance measure deteriorates
when an individual variable is randomly permuted, was used to
assess variable importance across all model types (i.e., ‘messed
up’; Strobl et al., 2007). As applied in a geomorphological classi-
fication study by Brenning et al. (2012), we measured variable
importance in a spatial cross-validation context. This permutation-
based spatial variable importance (SVI) measure was calculated for
the change in median AUROC values estimated by spatial cross-
validation. In this approach, one predictor variable was permuted
ten times for each test partition for a total of 1000 permutations
per predictor, and the AUROC of the prediction for each permu-
tation was measured and compared to the unperturbed predictive
performance. The SVI was standardized for each model by dividing
it by the highest value obtained for an individual predictor.
5
Therefore, the SVI values range from 0 to 1, 0 being low relative
importance, and 1 being high relative importance.
2.4.3. Mapping landslide susceptibility
Maps of the model predictions were produced for each AOI and
modeling technique to facilitate a visual comparison of model
outputs. Since the models were fitted using a 1:1 sampling ratio of
landslide to non-landslide points, the predicted unadjusted prob-
abilities should be interpreted as relative scores (see Petschko
et al., 2014 for possible adjustments). Since these models are ty-
pically classified into levels of susceptibility, the probabilities were
classified into five classes representing the relative potential for
landslide initiation. An approach based on an equal percentage of
overall area for each class was applied in this study to facilitate the
visual comparison of the output predictions (Chung and Fabbri,
2003). The probabilities were classified based on the 50th, 75th,
90th and 95th percentiles of the prediction values.
3. Results
3.1. The predictive performances
The variation in median AUROC model performances between
the models was relatively low for the Flysch and Molasse AOIs. In
the Flysch and Molasse AOIs, differences in median AUROC be-
tween models were only up to 2.9 and 3.6 percentage points (pp;
Table 2). In contrast, differences in median AUROC between
models in the Austroalpine AOI was up to 8.9pp. The tree-based
ensemble techniques, RF and BPLDA had the highest model per-
formance based on median AUROC values for all three AOIs
(Figs. 2 and 3, Table 2). RF was overall the highest ranked model
based on median AUROC and median TPR at 10% FPR. It achieved
the highest median AUROC for the Flysch (86.3%) and Molasse
(93.0%) areas, and lowest IQR of AUROC for AOIs. RF also had the
highest median TPR at 10% FPR estimation for all AOIs: Flysch
64.3%, Molasse 79.1% and Austroalpine 56.5%. The model perfor-
mance of BPLDA was similar to RF and always achieved a top three
estimation of median AUROC and median TPR at 10% FPR for all
AOIs, usually just behind RF. The rank in performances of WOE,
SVM, and GAM was much more variable than RF and BLPDA. GAM
had the second highest median TPR at 10% FPR in the Flysch area
(61.5%); however, it had the second lowest median TPR at 10% FPR
for the remaining AOIs. WOE usually performed in the middle of
the pack just behind SVM. The lowest model performance was
consistently achieved by the GLM, which includes having the
highest estimated AUROC and IQR values of all models in the AOIs.
Overall, the differences in AUROC performance were not only
small, but in many cases they were insignificant (Table 2). By es-
timating significance of AUROC difference, with a pairwise com-
parison, it was observed in all areas that there were no significant
systematic differences in AUROC values for RF and BPLDA, and
SVM and WOE. Also, it was observed that the GLM AUROC per-
formance was always significantly lower than the other models in
all AOIs.
3.2. A ranking of predictor importance
The ranking of relative variable importance was substantially
different for all modeling techniques and AOIs. However, there was
some consistency in the set of highest ranked predictors for each
area (Table 3). Slope angle, surface roughness (SDS) and plan
curvature were the only variables that were ranked in the top five
based on maximum SVI in all of the AOIs (SVI40.28). The highest-
ranked predictor based on maximum SVI was always slope angle
(SVI¼1.00). The most consistently lower-ranked variables
6 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11

Table 2
Model performance estimated with 20-repeated 5-fold spatial cross-validation. The median summarizes the central tendency of the estimated performance and the in-
terquartile range (IQR) its spread. The systematic difference (Δ), by percentage points (pp), and the associated significance of model performance (AUROC and TPR at 10%
FPR) were measured for pairwise comparisons. The significance of systematic differences was based on the adjusted p-values (to control the FDR) corresponding to the
Wilcoxon–Mann–Whitney tests.

Model AUROC Model TPR at 10% FPR

Median (IQR) Δ AUROC p-Values Median (IQR) Δ TPR p-Values

FLYSCH
GLM 83.4 (6.4) GLM 53.2 (25.8)
SVM 84.6 (6.0) þ 1.3 p ¼ 0.030n WOE 54.4 (22.5) þ 1.2 p ¼ 0.025n
WOE 84.9 (6.0) þ 0.3 p ¼ 0.468 SVM 58.9 (18.8) þ 4.6 p ¼ 0.012n
GAM 85.1 (5.4) þ 0.2 p ¼ 0.155 BPLDA 59.2 (12.5) þ 0.3 p ¼ 0.840
BPLDA 85.3 (5.2) þ 0.1 p ¼ 0.471 GAM 61.5 (25.3) þ 2.2 p ¼ 0.980
RF 86.3 (4.1) þ 1.0 p ¼ 0.094  RF 64.3 (18.7) þ 2.8 p ¼ 0.034n

MOLASSE
GLM 89.4 (7.7) GLM 64.1 (28.3)
GAM 91.9 (7.3) þ 2.5 p o 0.001nn GAM 75.2 (29.9) þ 11.1 p ¼ 0.012n
WOE 92.1 (7.0) þ 0.2 p ¼ 0.358 SVM 76.5 (33.1) þ 1.3 p ¼ 0.911
SVM 92.3 (6.9) þ 0.3 p ¼ 0.900 BPLDA 76.7 (25.3) þ 0.2 p ¼ 0.747
BPLDA 92.5 (5.7) þ 0.1 p ¼ 0.809 WOE 77.8 (35.7) þ 1.1 p ¼ 0.911
RF 93.0 (4.9) þ 0.5 p ¼ 0.707 RF 79.1 (19.8) þ 1.3 p ¼ 0.747

AUSTROALPINE
GLM 74.7 (12.1) GLM 43.9 (26.1)
GAM 76.6 (9.9) þ 2.0 p ¼ 0.092  GAM 46.6 (22.0) þ 2.7 p ¼ 0.499
WOE 80.1 (9.0) þ 3.5 p ¼ 0.015n WOE 50.0 (20.7) þ 3.4 p ¼ 0.103
SVM 80.4 (9.2) þ 0.3 p ¼ 0.878 SVM 50.0 (23.5) þ 0.0 p ¼ 0.464
RF 83.5 (5.5) þ 3.2 p o 0.001nn BPLDA 55.9 (18.7) þ 5.9 p ¼ 0.008nnn
BPLDA 83.6 (6.8) þ 0.0 p ¼ 0.977 RF 56.5 (16.2) þ 0.6 p ¼ 0.926

Significance codes for adjusted p-values: p o 0.001 “nnn”, p o 0.01 “nn”, p o0.05 “n”, p o0.1 “  ”, p 40.1 “ ”.

(SVIr0.15) were the convergence index (Conv. Ind), south-north
slope aspect and catchment height (C. height). Slope angle (max.
SVI ¼1.00), catchment area (C. area; 1.00), and plan curvature
(0.83) were the highest ranked variables in the Flysch Area. In the
Molasse area, slope (1.00), surface roughness (0.75) and elevation
(0.60) were the highest ranked variables; and in the Austroalpine
area they were slope (1.00), profile curvature and surface rough-
ness (1.00).
There was no pattern related to how SVI was distributed for
each modeling technique across the three AOIs. However, SVI was
distributed more uniformly for the AOIs with lower AUROC values.
The Austroalpine area, which had the lowest AUROC performances
(the mean median-AUROC¼79.8), had the largest number of
variables with an SVI40.15 (mean¼6). The Flysch area had a
mean median-AUROC of 84.9 and a mean of 5 variables with an
SVI4 0.15. The highest AUROC performances (mean median-
AUROC ¼91.9) in the Molasse area had generally the lowest spread
of variables with SVI 40.15 (mean ¼4).
Correlations between predictor variables were examined with
Spearman rank coefficient (ρSp). In each AOI, catchment area was
strongly correlated with catchment height, convergence index and
TWI (0.59 r|ρSp| r0.85); plan and profile curvature were moder-
ately to strongly correlated (0.59 r|ρSp| r0.85); and catchment
height, convergence index and topographic wetness were mod-
erately correlated (0.41 r|ρSp| r0.68). Slope angle was only
moderately correlated with other predictors, TWI (ρSp ¼  0.65)
and SDS (ρSp ¼0.66), in the Molasse AOI.

Fig. 2. Box-and-whisker plot of area under the receiver operating characteristic curve (AUROC %) estimated for each prediction technique applied to landslide susceptibility
modeling in different AOIs.
 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11 7

Fig. 3. Box-and-whisker plot of true positive rates (TPR %) estimated for each prediction technique at a 10% false positive rate (FPR %) applied to landslide susceptibility
modeling in different AOIs.

3.3. Comparing susceptibility map appearances
Modeling techniques whose predictions are a continuous function
of the predictors (GLM, GAM, SVM) had much smoother prediction
surfaces in the landslide susceptibility maps (Fig. 4). WOE, RF and
BPLDA, in contrast, produced more heterogeneous prediction surfaces.
In particular, the RF and BPLDA models had more spatial artifacts than
the other techniques, which make the prediction surface appear rather
noisy. Abrupt changes in prediction surface related to actual catego-
rical predictors (i.e., lithology) were not present in the maps.

Table 3
Spatial variable importance (SVI) based on median AUROC values from spatial cross-validation. The values are standardized relative to the most important predictor variable
for each model.

Variable Rank Max. SVI GAM GLM WOE SVM RF BPLDA

FLYSCH
Slope angle 1 1.00 0.74 0.68 1.00 1.00 1.00 1.00
C. area 2 1.00 1.00 1.00 0.11 0.46 0.19 0.52
Plan curv. 3 0.83 0.28 0.33 0.56 0.40 0.83 0.67
SDS 4 0.70 0.52 0.25 0.61 0.70 0.69 0.51
TWI 5 0.61 0.40 0.61 0.04 0.08 0.12 0.12
Profile curv. 6 0.31 0.02 0.02 0.28 0.07 0.31 0.21
East–west 7 0.17 0.00  0.01 0.00 0.15 0.14 0.17
Elevation 8 0.10 0.08 0.10 0.05 0.08 0.03 0.09
South–north 9 0.09 0.09 0.08 0.05  0.02  0.01 0.00
Conv. Ind. 10 0.06 0.02 0.00  0.07 0.01 0.06 0.05
C. height 11 0.04 0.03 0.00 0.04  0.02 0.04  0.01
Lithology 12 0.03 0.00 0.00 0.03 0.00 0.00  0.01

MOLASSE
Slope angle 1 1.00 1.00 1.00 1.00 1.00 1.00 1.00
SDS 2 0.75 0.05 0.02 0.75 0.17 0.17 0.12
Elevation 3 0.60 0.25 0.17 0.60 0.25 0.26 0.31
TWI 4 0.49 0.13 0.42 0.49 0.29 0.11 0.14
Plan curv. 5 0.28 0.01 0.02 0.28 0.05 0.07 0.03
Profile curv. 6 0.26 0.01 0.00 0.26 0.02 0.03 0.08
Lithology 7 0.20 0.04 0.06 0.20 0.05 0.02 0.04
C. height 8 0.06 0.00 0.00 0.06 0.02 0.01 0.02
Conv. Ind. 9 0.04 0.00 0.02 0.02 0.04 0.01 0.00
South–north 10 0.03 0.00 0.00 0.03 0.03 0.01 0.02
C. area 11 0.02 0.02 0.02 0.00 0.00 0.01 0.01
East–west 12 0.00 0.00 0.00  0.01  0.01 0.00 0.00

AUSTROAPLINE
Slope angle 1 1.00 1.00 1.00 0.43 0.79 0.39 0.48
Profile curv. 2 1.00 0.03 0.00 1.00 0.49 1.00 1.00
SDS 3 1.00 0.38 0.15 0.44 1.00 0.45 0.26
Plan curv. 4 0.86 0.78 0.86 0.78 0.77 0.80 0.61
Elevation 5 0.53 0.39 0.53 0.14 0.51 0.22 0.20
C. area 6 0.48 0.41 0.43  0.02 0.48 0.10 0.17
TWI 7 0.29 0.22 0.00  0.05 0.29 0.11 0.07
East–west 8 0.18 0.11 0.13 0.18 0.04  0.01 0.00
Conv. Ind. 9 0.15 0.04 0.01 0.04 0.12 0.15 0.08
South–north 10 0.09 0.02 0.09  0.04 0.04 0.03 0.06
C. height 11 0.01 0.00 0.01  0.05  0.16  0.10  0.07
Lithology 12 0.00  0.04  0.01  0.03  0.04 0.00  0.01

The three highest SVI values for each model area are printed in boldface.
8 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11

Fig. 4. An example of classified landslide susceptibility maps for each prediction technique in the Austroalpine AOI.

4. Discussion configurations may lead to different results, and it is beyond the

4.1. Evaluating prediction performance
With all of the available prediction techniques and methods for
model selection, it is always critical to remember that in reality no
one model is correct in a suite of competing models (Elith et al.,
2002). Therefore, we can only base our criteria and decision to
select a model on the specific scientific goals of the study (Elith
and Leathwick, 2009). In this paper, the spatial prediction ability of
empirically-based landslides susceptibility models was in-
vestigated while using typical model configurations. Clearly, as is
the case for any complex algorithm, different model or data
scope of this study to evaluate these possible differences. Among
the factors that may influence model performances are, in parti-
cular, feature selection procedures (Nguyen and de la Torre, 2010),
model setup (e.g., SVM kernel choice: Moguerza and Muñoz,
2006), sampling design (Mathur and Foody, 2008) or preproces-
sing of predictors (Xu et al., 2014).
Apart from the GLM, or logistic regression, the AUROC perfor-
mances of the prediction techniques appear to be very similar.
Previous comparison of prediction techniques also found no or
only small differences that were significant in prediction of land-
slide susceptibility (Yilmaz, 2009, 2010; Goetz et al., 2011). Con-
sequently, we cannot declare which model is best solely on the
 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
performance measures unless they had significant and practical
differences. In general, the ability of many machine learning pre-
diction techniques to represent complex nonlinear relationships
and higher-order interactions are similar. This can result in sta-
tistically insignificant results when comparing their performances
(Han and Kamber, 2006).
The small differences in AUROC performance can have some
practical significance depending on the prediction requirements
for a particular application (Beguería, 2006). In the case of land-
slide risk management, a higher detection rate of landslides in
areas where the detection of the absence of landslides is already
high can translate to an improved model prediction for high risk
areas (Goetz et al., 2011; Brenning, 2012a). The TPR at 10% FPR is
an example of an AUROC cutoff measure that can be used to assess
such a specific prediction requirement (Goetz et al., 2011). For
example, the difference between the best and worst median
AUROC performances was 2.9–9pp, depending on the AOI (Ta-
ble 2). These differences in median AUROC resulted in an increase
of 11–15pp in the TPR at 10% FPR. That is, 21–29% more landslides
were being detected in potential high risk areas.
The TPR at 10% FPR is only one such measure that was decided
to present in this paper. The ROC curve can be utilized to weigh the
assessment on either high accuracy of predicting the presence of
landslides or of identifying the most stable areas (Gorsevski et al.,
2006). More balanced decisions regarding class determination can
be obtained utilizing the ROC curve since the specific accuracy of
various cut-off combinations can be evaluated with this curve.
Therefore, the predictive costs of the susceptibility classes can be
clearly defined (Beguería, 2006; Gorsevski et al., 2006).
In addition to evaluating the prediction performance with a
specific accuracy measure, Guzzetti et al. (2006) suggested that
the model that is least sensitive to variation in performance given
different sampling conditions should be considered. We have ac-
counted for variation in model performance by reporting the in-
terquartile range (IQR) of the cross-validation results. Lower IQR
values indicated more robust model performances. Therefore,
when comparing two of the highest performing prediction tech-
niques, random forest (RF) and bundling with penalized linear
discriminant analysis (BPLDA), we can suggest that random forest
was the better technique for landslide initiation prediction based
on a lower IQR.
4.2. Considering other model criteria
Additional criteria can also be used to support decisions in model
selection. This study focused on assessing the performance of models
for spatial prediction. However, in addition to spatial prediction
performance, the ability to interpret the model for statistical in-
ference (i.e. spatial analysis) to gain insight into landslide distribution
characteristics may also be important (Brenning, 2012a). Therefore, a
statistically valid model is critical; good predictive properties are less
important in this situation (Brenning, 2005, 2012a). Interpretation of
models is more complex for machine learning algorithms, which are
generally considered to be ‘black-box’ models (Elith and Leathwick,
2009). The GLM and GAM, and with some limitations also WOE, are
prediction techniques that provide easily interpretable results that
can shed light on landslide conditioning factors (Lee et al., 2002;
Brenning, 2005; Regmi et al., 2010; Goetz et al., 2011). Especially the
integration of physically motivated predictors or model components
can facilitate the identification of possible causal mechanisms (Goetz
et al., 2011). The machine learning techniques, such as SVM, RF and
BPLDA, have been developed especially for prediction. They have the
advantage of automatically detecting interactions between pre-
dictors; thus, the prediction accuracy typically exceeds more con-
ventional techniques when complex interactions are present (Elith
et al., 2006).
9
In addition to careful validation of model performance, there
are also qualitative features that could be of importance to the end
user. For example, models with similar prediction performance do
not necessarily have similar prediction surfaces (Sterlacchini et al.,
2011). The geographic representation of susceptibility levels may
affect the way the mapped results are interpreted. For example,
isolated grid cells corresponding to a heterogeneous prediction
surface, which are predicted as being possibly unstable, may affect
planning decisions (also for surrounding stable terrain). It is gen-
erally easier to clearly define hazardous zones for application in
planning when the prediction surface is smooth. In practice, the
appearance of the prediction surface can also influence the end
users perception of the method; including their trust in the model.
Heterogeneous prediction surfaces are sometimes misinterpreted
by end users as being associated with a poor prediction of land-
slide susceptibility; which is not necessarily true. In this study, the
prediction performances of the RF, which does not produce a
smooth prediction surface and is more prone to spatial artifacts
(Brenning, 2005, 2012a), was for the majority better at prediction
than models that produced much smoother prediction surfaces
such as the GLM, GAM, and SVM. It is also important to mention
that abrupt changes in the prediction surface can also be caused in
any of the methods investigated in this paper when a categorical
variable, such as lithology, has a strong effect in the model.
4.3. Comparing the importance of predictor variables
A comparison of model behavior was roughly gauged on a
standardized approach for comparing relative importance of pre-
dictors for each prediction technique. In this paper, it was ob-
served with the SVI measure that the importance of predictors,
which in this case are based primarily on geomorphic conditions,
differed for each AOI. This relationship should be expected because
each AOI is associated with generally unique geomorphological
conditions. Petschko et al. (2014) observed a similar finding when
analyzing variable-selection frequencies of different GAMs applied
individually for lithology units in Lower Austria; the importance of
variables was different for each lithology unit. It has been well
established that the local site conditions have an important role in
the prediction of landslide susceptibility (van Westen et al., 2003;
Sidle and Ochiai, 2006; Lee et al., 2007; Blahut et al., 2010). The
dissimilar rank of predictor variables between each site provided
additional quantitative evidence to support this relationship.
The ranking of variable importance was irregular when com-
paring prediction techniques. Yet, the highest ranked variables
were generally consistent. Slope angle, surface roughness, and
plan curvature were the most common variables ranking high in
terms of relative importance in all AOIs (Table 3). The availability
of geologic data was only important for the prediction using WOE
in the Molasse area, which had the most heterogeneity in lithol-
ogy. The lack of importance of geology may be attributed to re-
gression dilution bias related to the thematic coarseness of the
lithology (Carrol et al. 2012); more detailed, in terms of scale,
geologic data may be required for the size of the AOIs. Regmi et al.
(2010) suggested only a handful of variables may be sufficient for
prediction of landslide susceptibility. The uneven distribution of
the SVI values observed in this study may indicate just that.
The different ranking of variables between prediction techni-
ques should be expected. Yesilnacar and Topal (2005) had made a
similar observation when comparing a GLM to artificial neural
networks (ANN), a machine learning technique, for landslide
susceptibility modeling. Although the prediction techniques had
similar performances, they are unique in their individual ap-
proaches to model construction and establishing the relevant re-
lationships between the predictor variables and landslide initia-
tion. Understanding of these differences is essential to select a
10 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
suitable prediction technique for a specific study goal (Brenning,
2009, 2012a).
5. Conclusions
Our study demonstrates that there was generally little differ-
entiation in prediction performance between statistical and ma-
chine learning landslide susceptibility modeling techniques ap-
plied to the AOIs in Lower Austria. This result underlines that even
when conducting model comparisons with a clear objective, such
as prediction performance, understanding the abilities and lim-
itation remains critical for model selection. In terms of pure pre-
diction performance, the RF and BPLDA modeling techniques were
the best. The most interpretable method and visually appealing
(i.e. had a smooth prediction surface) was the GAM, which sig-
nificantly performed better than the GLM. The SVM also had a
smooth prediction surface, but is generally difficult to interpret.
However, the SVM, RF and BPLDA may be particularly useful for
high-dimensional prediction problems where a large number of
highly correlated predictor variables are present.
Overall, it is recommended that model evaluation should be
tied closely to the goals of the study. The framework of this paper
is designed to assist in the evaluation of susceptibility modeling
techniques to enhance a user's decision on which method is most
suitable for a particular application.
Acknowledgements
The data for this research was provided by the Provincial
Government of Lower Austria from the MoNOE project – Method
development for landslide susceptibility maps for Lower Austria.
The authors are grateful for contributions of students from the
Department of Geography and Regional Research, University of
Vienna, under the supervision of Dr. Rainer Bell and Dr. Thomas
Glade, and colleagues at the Austrian Institute of Technology
GmbH, (AIT) in the Health and Environment department, for
construction of the landslide inventory. R. Cabrera implemented
the WOE method during a stay at the AIT that was partly sup-
ported by a Natural Sciences and Engineering Research Council of
Canada (NSERC) Discovery Grant (No. 355764-2013) awarded to A.
Brenning. We would also like to express our gratitude to the
anonymous reviewers for their constructive comments that helped
to improve the paper.
References
Agterberg, F.P., Cheng, Q., 2002. Conditional independence test for weights-of-
evidence modeling. Nat. Resour. Res. 11 (4), 249–255.
Akaike, H., 1974. A new look at the statistical model identification. IEEE Trans.
Autom. Control 19, 716–723.
Atkinson, P.M., Massari, R., 1998. Generalised linear modelling of susceptibility to
landsliding in the central apennines, Italy. Comput. Geosci. 24 (4), 373–385.
Ayalew, L., Yamagishi, H., 2005. The application of GIS-based logistic regression for
landslide susceptibility mapping in the Kakuda-Yahiko mountains Central Ja-
pan. Geomorphology 65 (1), 15–31.
Beguería, S., 2006. Validation and evaluation of predictive models in hazard as-
sessment and risk management. Nat. Hazards 37 (3), 315–329.
Benjamini, Y., Hochberg, Y., 1995. Controlling the false discovery rate: a practical
and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B (Methodol.),
289–300.
Beven, K.J., Kirkby, M.J., 1979. A physically based, variable contributing area model
of basin hydrology. Hydrol. Sci. Bull., 24; , pp. 43–69.
Blahut, J., van Westen, C.J., Sterlacchini, S., 2010. Analysis of landslide inventories for
accurate prediction of debris-flow source areas. Geomorphology 119 (1), 36–51.
Bonham-Carter, G., 1994. Geographic information systems for geoscientists; mod-
elling with GIS. Computer Methods in Geosciences. Vol. 13. Pergamon Press, p.
398.
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J., 1984. Classification and Re-
gression Trees. CRC Press, Wadsworth.
Breiman, L., 2001. Random forests. Mach. Learn. 45 (1), 5–32.
Breiman, L., Cutler, C., 2012. Breiman and Cutler's random forests for classification
and regression (randomForest). R package version 4.6-7, R port by A. Liaw & M.
Wiener.
Brenning, A., 2005. Spatial prediction models for landslide hazards: review, com-
parison and evaluation. Nat. Hazards Earth Syst. Sci. 5, 853–862.
Brenning, A., Trombotto, D., 2006. Logistic regression modeling of rock glacier and
glacier distribution: topographic and climatic controls in the semi-arid Andes.
Geomorphology 81 (1), 141–154.
Brenning, A., 2009. Benchmarking classifiers to optimally integrate terrain analysis
and multispectral remote sensing in automatic rock glacier detection. Remote
Sens. Environ. 113, 239–247.
Brenning, A., 2012a. Improved spatial analysis and prediction of landslide sus-
ceptibility: practical recommendations. In: Eberhardt E. Froese, C., Turner A.K.,
Leroueil S. (Eds.). Landslides and Engineered Slopes: Protecting Society through
Improved Understanding. Proceedings of the 11th International and 2nd North
American Symposium on Landslides and Engineered Slopes, vol. 1., Banff, Ca-
nada, 3–8 June 2012, CRC Press/Balkema Leiden, the Netherlands, pp. 789–794.
Brenning, A., 2012b. Spatial cross-validation and bootstrap for the assessment of
prediction rules in remote sensing: the R package ‘sperrorest’ 2012. IEEE In-
ternational Geoscience and Remote Sensing Symposium (IGARSS), 23–27 July
2012, pp. 5372–5375.
Brenning, A., Long, S., Fieguth, P., 2012. Detecting rock glacier flow structures using
Gabor filters and IKONOS imagery. Remote Sens. Environ. 125, 227–237.
Carrara, A., Guzzetti, F., Cardinali, M., Reichenbach, P., 1999. Use of GIS technology in
the prediction and monitoring of landslide hazard. Nat. Hazards 20 (2–3),
117–135.
Carrol, R.J., Rupert, D., Stefanski, L.A., Crainiceanu, C.M., 2012. Measurement Error in
Nonlinear Models: A Modern Perspective, 2nd edn. CRC Press, New York, p. 488.
Chang, C.C., Lin, C.J., 2011. LIBSVM: a library for support vector machines. ACM
Trans. Intell. Syst. Technol. (TIST) 2 (3), 1–27.
Chung, C.J.F., Fabbri, A.G., 1999. Probabilistic prediction models for landslide hazard
mapping. Photogramm. Eng. Remote Sens. 65 (12), 1389–1399.
Chung, C.J.F., Fabbri, A.G., 2003. Validation of spatial prediction models for landslide
hazard mapping. Nat. Hazards 30 (3), 451–472.
Conrad, O., 2006. SAGA – Program structure and current state implementation. In:
Böhner, J., McCloy, K.R., Strobl, J. (Eds.), SAGA — Analysis and Modelling Ap-
plications, vol. 115. Göttinger Geographische Abhandlungen, pp. 39–52.
Dai, F.C., Lee, C.F., 2002. Landslide characteristics and slope instability modeling
using GIS Lantau Island, Hong Kong. Geomorphology 42, 213–228.
Dai, F.C., Lee, C.F., Ngai, Y.Y., 2002. Landslide risk assessment and management: an
overview. Eng. Geol. 64 (1), 65–87.
Dimitriadou, E., Hornik, K., Leisch, F., Meyer, D., Weingessel, A., 2007. e1071: Misc
Functions of the Department of Statistics (e1071) TU Wien. R package version
1.6-1.
Elith, J., Burgman, M.A., Regan, H., 2002. Mapping epistemic uncertainties and va-
gue concepts in predictions of species distribution. Ecol. Modell. 157 (2),
313–329.
Elith, J., et al., 2006. Novel methods improve prediction of species' distributions
from occurrence data. Ecography 29, 129–151.
Elith, J., Leathwick, J.R., 2009. Species distribution models: ecological explanation
and prediction across space and time. Annu. Rev. Ecol., Evol. Syst. 40, 677–697.
Frattini, P., Crosta, G., Carrara, A., 2010. Techniques for evaluating the performance
of landslide susceptibility models. Eng. Geol. 111, 62–72.
Glenn, N.F., Streutker, D.R., Chadwick, D.J., Thackray, G.D., Dorsch, S.J., 2006. Ana-
lysis of LiDAR-derived topographic information for characterizing and differ-
entiating landslide morphology and activity. Geomorphology 73 (1), 131–148.
Goetz, J.N., Guthrie, R.H., Brenning, A., 2011. Integrating physical and empirical
landslide susceptibility models using generalized additive models. Geomor-
phology 129 (3), 376–386.
Gottschling, P., 2006. Massenbewegungen. Geologie der Bundesländer-Niederö-
sterreich. vol. 2. Geologische Bundesanstalt, Wien, pp. 335–340.
Gorsevski, P.V., Gessler, P.E., Foltz, R.B., Elliot, W.J., 2006. Spatial prediction of
landslide hazard using logistic regression and ROC analysis. Trans. GIS 10 (3),
395–415.
Guzzetti, F., Reichenbach, P., Ardizzone, F., Cardinali, M., Galli, M., 2006. Estimating
the quality of landslide susceptibility models. Geomorphology 81 (1), 166–184.
Han, J., Kamber, M., 2006. Data Mining: Concepts and Techniques, 2nd edn. Morgan
Kaufmann Publishers, San Francisco, p. 743.
Hastie, T.J., Buja, A., Tibshirani, R., 1995. Penalized discriminant analysis. Ann. Stat.
23 (1), 73–102.
Hastie, T.J., Tibshirani, R., 1990. Generalized Additive Models.. Chapman & Hall,
London, p. 352.
Hastie, T., 2009. GAM: Generalized Additive Models R package version 1.08.
Hastie, T.J., Tibshirani, R., 2009. MDA: Mixture and Flexible Discriminant Analysis. R
package version 0.4-2, R port by F. Leisch, K. Hornik & B. D. Ripley. 〈http://cran.
r-project.org/package¼mda〉.
Heckmann, T., Gregg, K., Gregg, A., Becht, M., 2014. Sample size matters: in-
vestigating the effect of sample size on a logistic regression susceptibility
model for debris flows. Nat. Hazards Earth Syst. Sci. 14, 259–278.
Hijmans, R.J., van Etten, J., 2013. Raster: Geographic data analysis and modeling
(raster). R package version 2.1-25.
Hosmer, D.W., Lemeshow, S., 2000. Applied Logistic Regression, 2nd edition. John
Wiley & Sons, New York, p. 373.
 J.N. Goetz et al. / Computers & Geosciences 81 (2015) 1–11
Hothorn, T., Lausen, B., 2005. Bundling classifiers by bagging trees. Comput. Stat.
Data Anal. 49 (4), 1068–1078.
James, G., Witten, D., Hastie, T., Tibshirani, R., 2013. An Introduction to Statistical
Learning. Springer, New York, p. 441.
Lee, S., Choi, J., Min, K., 2002. Landslide susceptibility analysis and verification using
the Bayesian probability model. Environ. Geol. 43 (1–2), 120–131.
Lee, S., Ryu, J.H., Kim, I.S., 2007. Landslide susceptibility analysis and its verification
using likelihood ratio, logistic regression, and artificial neural network models:
case study of Youngin Korea. Landslides 4 (4), 327–338.
Ließ, M., Glaser, B., Huwe, B., 2011. Functional soil-landscape modelling to estimate
slope stability in a steep Andean mountain forest region. Geomorphology 132
(3), 287–299.
Lineback Gritzner, M., Marcus, W.A., Aspinall, R., Custer, S.G., 2001. Assessing
landslide potential using GIS, soil wetness modeling and topographic attributes
Payette River, Idaho. Geomorphology 37 (1), 149–165.
Mathur, A., Foody, G.M., 2008. Crop classification by a support vector machine with
intelligently selected training data for an operational application. Int. J. Remote
Sens. 29 (8), 2227–2240.
Micheletti, N., Foresti, L., Robert, S., Leuenberger, M., Pedrazzini, A., Jaboyedoff,
Kanevski, M., 2014. Machine learning feature selectoin methods for landslide
susceptiblity mapping. Math. Geosci. 46, 33–57. http://dx.doi.org/10.1007/
s11004-013-9511-0.
Moguerza, J.M., Muñoz, A., 2006. Support vector machines with applications. Stat.
Sci. 21 (3), 322–336. http://dx.doi.org/10.1214/088342306000000493.
Moore, I.D., Grayson, R.B., Ladson, A.R., 1991. Digital terrain modelling: a review of
hydrological, geomorphological, and biological applications. Hydrol. Process. 5
(1), 3–30.
Muenchow, J., Brenning, A., Richter, M., 2012. Geomorphic process rates of land-
slides along a humidity gradient in the tropical Andes. Geomorphology 139-
140, 271–284.
Neuhäuser, B., Terhorst, B., 2007. Landslide susceptibility assessment using
“weights-of-evidence” applied to a study area at the Jurassic escarpment (SW-
Germany). Geomorphology 86 (1), 12–24.
Nguyen, M.H., de la Torre, F., 2010. Optimal feature selection for support vector
machines. Pattern Recognit. 43, 584–591.
Pachauri, A.K., Pant, M., 1992. Landslide hazard mapping based on geological at-
tributes. Eng. Geol. 32 (1), 81–100.
Peters, A., Hothorn, T., 2009. Ipred: Improved predictors. R package version 0.9-1.
Petschko, H., Bell, R., Leopold, P., Heiss, G., Glade, T., 2013. Landslide inventories for
reliable susceptibility maps. In: Margottini, C., Canuti, P., Sassa, K. (Eds.),
Landslide Science and Practice, vol. 1: Landslide Inventory and Susceptibility
and Hazard Zoning. Springer.
Petschko, H., Brenning, A., Bell, R., Goetz, J., Glade, T., 2014. Assessing the quality of
landslide susceptibility maps – case study Lower Austria. Nat. Hazards Earth
Syst. Sci. 14, 95–118. http://dx.doi.org/10.5194/nhess-14-95-2014.
Pradhan, B., 2013. A comparative study on the predictive ability of the decision tree,
support vector machine and neuro-fuzzy models in landslide susceptibility
mapping using GIS. Comput. Geosci. 51, 350–365.
R Development Core Team R, 2003. A language and environment for statistical
computing. R Foundation for Statistical Computing Vienna, Austria.
Regmi, N.R., Giardino, J.R., Vitek, J.D., 2010. Modeling susceptibility to landslides
using the weight of evidence approach: Western Colorado, USA. Geomorphol-
ogy 115 (1), 172–187.
Regmi, N.R., Giardino, J.R., McDonald, E., Vitek, J.D., 2014. A comparison of logistic
regression-based models of susceptibility to landslides in western Colorado,
USA. Landslides 11, 247–262.
Ruß, G., Brenning, A., 2010. Data mining in precision agriculture: Management of
spatial information. Lect. Notes Comput. Sci. 6178, 350–359. http://dx.doi.org/
10.1007/978-3-642-14049-5_36.
Schnabel, W., 2002. Geologische Karte von Niederösterreich 1:200000. Wien,
Austria (in German).
Schweigl, J., Hervás, J., 2009. Landslide Mapping in Austria, JRC Scientific and
Technical Reports, European Commission Joint Research Centre, Institute for
11
Environment and Sustainability, Italy. Available at: 〈http://eusoils.jrc.ec.europa.
eu/ESDB_Archive/eusoils_docs/other/EUR23785EN.pdf〉 (last access: 01.0311).
Schwenk, H., 1992. Massenbewegungen in Niederösterreich 1953-1990. Jahrbuch
der Geologischen Bundesanstalt. vol. 135. Geologische Bundesanstalt, Wien, pp.
597–660.
Schulz, W.H., 2004. Landslides mapped using LIDAR imagery, Seattle, Washington.
U.S. Geological Survey Open-File Report 2004-1396. 11 pp., 1 plate.
Schulz, W.H., 2007. Landslide susceptibility revealed by LIDAR imagery and his-
torical records, Seattle, Washington. Eng. Geol. 89, 67–87.
Sidle, R.C., Ochiai, H., 2006. Landslides: Processes, Prediction, and Land Use.
American Geophysical Union, Washington DC, p. 312.
Sing, T., Sander, O., Beerenwinkel, N., Lengauer, T., 2009. ROCR: Visualizing the
Performance of Scoring Classifiers. R package version 1.0-4. http://cran.r-pro
ject.org/package¼ ROCR.
Soeters, R., van Westen, C. J., 1996. Landslides: Investigation and mitigation.
Chapter 8-Slope instability recognition, analysis, and zonation. Transportation
Research Board Special Report 247.
Sterlacchini, S., Ballabio, C., Blahut, J., Masetti, M., Sorichetta, A., 2011. Spatial
agreement of predicted patterns in landslide susceptibility maps. Geomor-
phology 125 (1), 51–61.
Strobl, C., Boulesteix, A.-L., Zeileis, A., Horthon, T., 2007. Bias in random forest
variable importance measures: Illustrations sources, and a solution. BMC Bioinf.
8, 25. http://dx.doi.org/10.1186/1471-2105-8-25.
Van den Eeckhaut, M., Vanwalleghem, T., Poesen, J., Govers, G., Verstraeten, G.,
Vandekerckhove, L., 2006. Prediction of landslide susceptibility using rare
events logistic regression: a case-study in the Flemish Ardennes (Belgium).
Geomorphology 76, 392–410.
van Westen, C.J., Rengers, N., Terlien, M.T.J., Soeters, R., 1997. Prediction of the oc-
currence of slope instability phenomenal through GIS-based hazard zonation.
Geol. Rundsch. 86 (2), 404–414.
van Westen, C.J., Rengers, N., Soeters, R., 2003. Use of geomorphological informa-
tion in indirect landslide susceptibility assessment. Nat. Hazards 30 (3),
399–419.
van Westen, C.J., Castellanos, E., Kuriakose, S.L., 2008. Spatial data for landslide
susceptibility, hazard, and vulnerability assessment: an overview. Eng. Geol.
102 (3), 112–131.
Vapnik, V., 1998. Statistical Learning Theory. John Wiley & Sons Inc., New York, p.
736.
Varnes, D.J., 1984. Landslide Hazard Zonation: A Review of Principles and Practice,
Natural Hazards No. 3, IAEG Commission on Landslides and other Mass-
Movements, UNESCO, Paris.
Wessely, G., 2006. Geologie der Österreichischen Bundesländer: Niederösterreich.
Geologische Bundesanstalt, Vienna, p. 416.
Xu, L., Li, J., Brenning, A., 2014. A comparative study of different classification
techniques for marine-oil spill identification using RADARSAT-1 imagery. Re-
mote Sens. Environ. 141, 14–23.
Yalcin, A., Reis, S., Aydinoglu, A.C., Yomralioglu, T., 2011. A GIS-based comparative
study of frequency ratio, analytical hierarchy process, bivariate statistics and
logistic regression methods for landslide susceptibility mapping in Trabzon, NE
Turkey. Catena 85, 2011.
Yesilnacar, E., Topal, T., 2005. Landslide susceptibility mapping: a comparison of
logistic regression and neural networks methods in a medium scale study,
Hendek region (Turkey). Eng. Geol. 79 (3), 251–266.
Yilmaz, I., 2009. Landslide susceptibility mapping using frequency ration, logistic
regression, artificial neural networks and their comparison: a case study from
Kat landslides (Tokat-Turkey). Comput. Geosci. 35, 1125–1138.
Yilmaz, I., 2010. Comparison of landslide susceptibility mapping methodologies for
Koyulhisar, Turkey: conditional probability, logistic regression, artificial neural
networks, and support vector machine. Environ. Earth Sci. 61 (4), 821–836.
Zweig, M.H., Campbell, G., 1993. Receiver-operating characteristic (ROC) plots. Clin.
Chem. 39, 561–577.