Int J Comput Vis (2008) 80: 92–103
DOI 10.1007/s11263-008-0130-z
Performance Modeling and Algorithm Characterization
for Robust Image Segmentation
Robust Image Segmentation
Shishir K. Shah
Received: 13 July 2007 / Accepted: 4 March 2008 / Published online: 20 March 2008
© Springer Science+Business Media, LLC 2008
Abstract This paper presents a probabilistic framework
based on Bayesian theory for the performance prediction
and selection of an optimal segmentation algorithm. The
framework models the optimal algorithm selection process
as one that accounts for the information content of an in-
put image as well as the behavioral properties of a particular
candidate segmentation algorithm. The input image infor-
mation content is measured in terms of image features while
the candidate segmentation algorithm’s behavioral charac-
teristics are captured through the use of segmentation quality
features. Gaussian probability distribution models are used
to learn the required relationships between the extracted im-
age and algorithm features and the framework tested on the
Berkeley Segmentation Dataset using four candidate seg-
mentation algorithms.
Keywords Image segmentation · Performance modeling ·
Algorithm characterization · Algorithm selection
1 Introduction
In any computer vision and image analysis system, segmen-
tation is an integral part and is broadly defined as the par-
titioning of an image into separate regions. Each resulting
S.K. Shah
Quantitative Imaging Laboratory, University of Houston,
Houston, TX 77204-3010, USA
S.K. Shah ()
Department of Computer Science, University of Houston,
Houston, TX 77204-3010, USA
e-mail: shah@cs.uh.edu
region corresponds to a different object or area of inter-
est. Typically, the partitioning is derived from specific con-
straints and the segmentation process uses these constraints
to construct homogenous regions and smooth boundaries.
Over the past 40 years, a multitude of methods and algo-
rithms have been proposed in literature (Pal and Pal 1993).
Each of them attempt to solve the segmentation problem
based on image properties, constraints derived from the ap-
plication domain, or a combination. Various image prop-
erties have been investigated including gray-level intensity,
color, texture, etc. Higher level constructs such as geometric
models of objects of interest, or Gestalt principles of percep-
tual organization have also been used. The fundamental limit
in solving the segmentation problem is the fact that seg-
mentation is a problem of psycho-physical perception and
therefore not susceptible to a purely analytical solution (Pal
and Pal 1993; Pavlidis 1977; Fu and Mui 1981). Techniques
and algorithms developed to get satisfactory results range
from simple statistical models, to adaptive filters, inten-
sity and texture analyses, wavelets, entropy, fuzzy sets, and
neural networks (Pal and Pal 1993; Nair and Aggarwal 1996;
Spann and Nieminen 1988; Spann and Grace 1994). How-
ever, most approaches are developed for a specific applica-
tion and cannot be generalized for all images. In fact, no
single algorithm can be considered good for all images, nor
are all algorithms good for a particular image (Pal and Pal
1993). Each algorithm’s utility is limited by its specific char-
acteristics that makes it applicable for particular kind of im-
ages. The fundamental challenge in image segmentation is
then to provide a generalized framework that is capable of
choosing a suitable algorithm from many candidates given a
particular image (Haralick and Shapiro 1992).
This paper presents a probabilistic framework that allows
for the selection of an appropriate image segmentation al-
gorithm based on the characteristic properties of the im-
Int J Comput Vis (2008) 80: 92–103
age to be segmented and the algorithm’s behavioral proper-
ties. Since a large number of image segmentation algorithms
have already been developed, the present work focuses on
evaluating the ability of a candidate algorithm from a list
of possible algorithms to perform segmentation on a given
image. Within the developed framework, the ability to per-
form this evaluation is learned using a learning set of im-
ages. Based on this knowledge, the evaluation or prediction
of each candidate algorithm’s capability of segmenting the
input image is done without actually running any of the algo-
rithms. Segmentation is performed using only the algorithm
predicted to achieve the best outcome.
Performance analysis for tuning parameters of various
segmentation algorithms for the purpose of optimizing re-
sults within the context of an application domain has re-
ceived much attention in the computer vision community in
recent years (Konishi and Yuille 2000). A survey of major
research in the area of segmentation algorithm evaluation is
presented in Zhang (1996). Analytical examination is by far
the most common approach to evaluate the performance of
a segmentation algorithm. This is normally done by model-
ing the image to be processed, the noise that degrades the
original image with respect to the model, and analyzing the
objective of the segmentation task (Chalmond et al. 2001).
Another way of assessing performances has been through
examination of outputs of the candidate algorithms for an
input image. Methods that combine both analytical examina-
tion and systematic optimization from model images for the
derivation of performance have also been developed (Zhang
and Haralick 1993; Cho et al. 1997). In the context of opti-
mizing algorithm performance based on parameter tuning,
recent approaches have exploited machine learning tech-
niques and learning-based system where the parameters are
obtained from training data (Reynolds and Rolnick 1995;
Perner 1999). Predicting the probability of incorrect seg-
mentation for each candidate algorithm based on informa-
tion theoretic approaches has also been proposed (Konishi
and Yuille 2000). In a restricted sense, algorithm evalu-
ation is concerned with the study of the algorithm’s be-
havior under different inputs. That is, performance evalua-
tion is a process to characterize the capability of an algo-
rithm, independent of the application domain. This requires
a methodology to understand the input-output relationship
of the algorithm. In many cases, performance evaluation has
been coupled with methods for fair comparisons of algo-
rithms (Randen and Husoy 1999; Shufelt 1999). Algorithm
comparison then leads to ranking of different algorithms for
a given input image and the application of interest. How-
ever, instead of understanding the capability of a particular
algorithm in analyzing different images, it emphasizes the
relative merit of one algorithm over others, and hence, fails
to offer any predictive information.
93
The proposed framework in this paper is different from
the previous methods. The emphasis is on evaluating the ca-
pability of an algorithm to provide optimal results for a given
input image. We are not interested in optimizing an algo-
rithm for a given task. It is well understood that segmenta-
tion algorithms are difficult to generalize and, depending on
the context, a given algorithm will perform well and lead to
dependable results, while for another image, still dependent
on the context, the same algorithm will perform poorly and
provide unreliable results. In this paper, we introduce a gen-
eral probabilistic framework and a prototype system that are
designed for optimal selection of image segmentation algo-
rithms. This system can be considered an intelligent system
that combines algorithm evaluation with knowledge-based
prediction for segmentation algorithm selection. Evaluation
is used to obtain knowledge about an algorithm’s capability
and the intelligent system design is used to apply that knowl-
edge. Knowledge within the framework is learned from a
limited sample of images representing the context variety
and a measure of candidate algorithms’ performance.
Rest of the paper is organized as follows. Section 2 de-
scribes the proposed approach and provides the definitions
and the methodology of probabilistic performance predic-
tor. Section 3 presents the framework based on probability
models and a maximum a posteriori estimator. Segmenta-
tion algorithms, image features, and performance evaluation
measures are described in Sect. 4. Experimental results and
evaluation of the proposed approach on a large database of
segmented images with ground truth are presented in Sect. 5.
Finally, Sect. 6 concludes this work and discusses future ex-
tensions.
2 Performance Prediction Framework
Selection of an appropriate segmentation algorithm given an
input image is generally difficult for a computer, but rather
easy for an experienced person. Humans can learn an algo-
rithm’s characteristics by observing the algorithm’s perfor-
mance over a range of inputs. As a result, humans are able
to predict the performance of each algorithm on a given im-
age, and more over able to choose an optimal one. Simulat-
ing this process can lead to computerized automation in pre-
dicting the performance of various algorithms and hence the
eventual selection of an optimal one given an input image.
Performance analysis for any algorithm can include a range
of measures such as processing quality, stability, time and
memory complexity, etc. Each of these measures are rele-
vant and can weigh on the selection of an optimal algorithm.
For simplicity, we will restrict our discussion in this paper
to only the quality of segmentation.
A general flow chart of the proposed performance pre-
diction framework is shown in Fig. 1. The framework com-
prises of three main modules: feature extractor, performance
94
Fig. 1 Framework of the
proposed knowledge gathering
optimal image segmentation
system
evaluator, and a predictor. The feature extractor is meant to
identify the context of the image. This means that the fea-
ture extractor should be general yet powerful enough to ex-
tract image features that can provide sufficient information
about the characteristics of the image that can be associated
with the behavioral properties of a segmentation algorithm.
The performance evaluator simulates the human observer,
wherein the main objective of the performance evaluator is
to capture the behavioral properties of each algorithm. This
is based on a ground-truth reference and accurate image fea-
tures that can be measured to provide distinction among the
candidate algorithms and the one chosen as being closest
to the ground-truth. Finally, the predictor provides a conve-
nient mechanism to automatically acquire, store, and utilize
human knowledge in an implicit way. Specifically, the pre-
dictor combines information from the feature extractor and
the performance evaluator, which forms the basis of the im-
age context and the knowledge about each algorithm’s per-
formance characteristics. The predictor uses this informa-
tion to estimate the outcome of each candidate algorithm.
The one predicted to generate the most desirable outcome is
chosen. In this paper, we provide a statistical learning model
for the predictor based on Bayesian decision theory (Duda
and Hart 1973), relying on features extracted from the input
image and the measure of algorithm performance extracted
from a learning data set of images.
The framework is prototyped through the implementa-
tion of four image segmentation techniques as detailed in
Sect. 4. During training, the input image is processed with
each algorithm and the result compared to ground truth data.
A range of measures are computed for each input image and
associated with the obtained results of segmentation. Both
the image features and the measure of segmentation quality
Int J Comput Vis (2008) 80: 92–103
are used to design a predictor. This allows the representation
and capture of knowledge needed for algorithm selection.
The predictor itself is formulated such that confidence levels
can be associated with the knowledge captured. When this
knowledge is put to use, features extracted from each new
input image are used by the predictor and the performance
of all algorithms on that image is predicted. The algorithm
corresponding to the best performance is selected as optimal
and applied to that image.
3 Probabilistic Learning and Prediction
In presenting the probabilistic prediction framework, con-
sider the problem of segmenting an image into different re-
gions, each exhibiting an area of interest. Let I = {Is , s ∈ S}
be an image which belongs to V S where V is the set of val-
ues taken by Is , typically V = {0, . . . , 255}. Assume that
a set of local characteristics, or image features Y = {Ys ,
s ∈ S}, extracted from I , is available. Let us also denote a
set of segmentation algorithms A = {a1 , . . . , ak }, each re-
sulting in a corresponding segmentation R = {Ra , a ∈ A}.
Let us further denote Ra to be a random variable indicating
the segmentation for an algorithm a. We further define the
ground truth as being represented as R g . The ground truth
represents the exact state of nature with respect to the re-
gions. Our objective is clearly to obtain a segmentation as
close as possible to R g . In probabilistic terms, the quality of
segmentation obtained can be measured by the probability
of error, α = P (Ra = R g |R g ), calculated over all regions.
The goal of the predictor then is to minimize the probability
of error.
Generalizing the above point of view, the proposed prob-
abilistic learning and prediction approach can be considered
Int J Comput Vis (2008) 80: 92–103
Fig. 2 A probabilistic learning
and prediction approach for
hypothesis based segmentation
algorithm selection
a matching problem (Ettinger et al. 1996; Chiang and Moses
1999; Chiang et al. 2000) as summarized in Fig. 2. Given a
bank of segmentation algorithms available, let us consider
an index stage that provides a list of candidate hypothesis,
Hk , k = 1, . . . , K, based on a partitioning of the hypothesis
space. A feature extraction stage serves to measure the im-
age context, where such context is estimated from the image
and used as low-dimensional surrogates for sufficient sta-
tistics. This is modeled as a density function, p(Y |I ) and
reflects the sensitivity of the image features Y , given the im-
age I . Evaluation of the candidate hypotheses proceeds us-
ing a probabilistic interpretation for the observations. A fea-
ture prediction stage is used to compute a predicted feature
set for each segmentation algorithm by combining the image
context model from the feature extraction stage and a “good-
ness” of segmentation representation of a hypothesis Hk .
The predicted feature set, X k , is modeled by a distribution
function p(X|Hk ) acknowledging error in the model used
to compute the predicted feature set and variation among
segmentations in any algorithm. A Bayes Match is used to
compute the posterior probability of a candidate hypothe-
sis, ∧(Hk ), by combining the extracted and predicted feature
sets. A rank order of the top hypothesis and the associated
likelihoods provide the optimal selection of the segmenta-
tion algorithm in this proposed system.
The Bayes matching problem can be formulated as fol-
low. Consider n features
Y = [Y1 , Y2 , . . . , Yn ]T (1)
extracted from the input image, I , as one of the inputs to the
Bayes match stage. Each feature, Yi , can be a vector valued
function of the contextual attributes. Known also is a set of
K candidate hypotheses, given as H = Hk , k ∈ [1, . . . , K],
each having a prior probability P (Hk ). The set H is given
by the index stage and can be considered to be a list of the
candidate algorithms and their hypothesized ability to seg-
ment the input image. Further, consider a set of m predicted
features given as
X k = [X1k , X2k , . . . , Xm ] ,
k T
(2)
95
which form the second input to the Bayes match stage. We
assume available a feature prediction function, learned off-
line, which maps a hypothesis Hk to the set of predicted
features. We also assume a known probability model for the
attributes of any predicted or extracted feature, as well as for
the feature attributes if Yi provides no positive association to
a predicted feature (i.e., Yi has a negative relationship to a
predicted feature). The objective of the Bayes match stage is
to compute the posterior likelihood of the observed features,
Y , under each of the K hypotheses in the set H, given by
∧k = P (Hk |Y ), Hk ∈ H. (3)
The most likely hypotheses and their corresponding likeli-
hoods provide the optimal choice of the segmentation algo-
rithm.
For simplicity of presentation, we drop the subscript of
the hypothesis Hk considering the solution for H ∈ H and
the superscript k on X k . Then, the Bayes rule applied to
compute the posterior likelihood in (3) for any H ∈ H is
given by
p(Y |H )P (H )
P (H |Y ) = . (4)
p(Y )
Since the denominator of (4) does not depend on the hypoth-
esis H , the maximum a posteriori (MAP) estimate is found
by maximizing p(Y |H )P (H ) over H ∈ H. The prior P (H )
is computed during the model learning phase where the prior
for each hypothesis is given as a ratio of the number of times
a particular hypothesis was true given an input image over
total size of the learning image set.
To determine the conditional estimate of Y given hypoth-
esis H , we have

p(Y |H ) = p(Y |X, H )p(X|H )dX (5)
where, p(X|H ) models the feature prediction and
p(Y |X, H ) models the feature extraction. This is the case
since X is completely determined from the hypothesis H .
96
The computation of p(Y |X, H ) requires that there be a rela-
tionship between the elements of Y and X, which eludes to
the fact that the “goodness” of segmentation features will be
implicitly related to the image data since the characteristics
of the segmentation algorithms are defined by the character-
istics of the image data.
Let us define a mapping function λ that relates the ex-
tracted and predicted features, Y and X, respectively. The
mapping function itself is a nuisance parameter that arises
because the relationship between the predicted and extracted
features is not explicitly known and hence not amenable to
direct estimation. If we denote L to be the set of all admis-
sible mapping functions, the probabilistic correspondence
model for the Bayes likelihood estimate is given by
 of algorithm selection, and
p(Y |H ) = p(Y |λ, H )P (λ|H ) (6)
λ∈L
where, similar to (5),

p(Y |λ, H ) = p(Y |X, λ, H )p(X|λ, H )dX. (7)
To solve (6), we require a probability model for
p(Y |X, λ, H ). Let us assume that the uncertainties of Xi
are conditionally independent given H , and that the uncer-
tainties of Yj are conditionally independent given H and X.
The independence assumption allows the use of the product
rule to compute the likelihood as:

n
p(Y |λ, H ) = p(Yj |λ, H ). (8)
j =1
Each term in the product of (8) is a feature likelihood con-
ditioned on a mapping function λ and hypothesis H . Each
extracted feature Yj either provides a positive association to
a predicted feature or has no relationship. If a particular Yj
provides no positive association, the mapping can be consid-
ered to be a null function and the feature attribute modeled
as a random vector with a known probability density func-
tion
p(Yj |λj = 0, H ) = pNA (Yj ). (9)
On the other hand, for positive associations between Yj and
a predicted feature Xi , the mapping can be written as λj = i
and the likelihood of match computed according to (7). Fur-
ther, for all i = null, (7) can be simplified to
 proposed framework.
p(Yj |λj = i, H ) = p(Yj |Xi , H )p(Xi |H )dXi . (10)
If we assume the extracted and predicted feature attributes
have Gaussian distributed uncertainties, then (10) can be fur-
ther simplified to
p(Yj |λj = i, H ) = p(Yj |Xi , H ) ∼ N (Xi , ), (11)
Int J Comput Vis (2008) 80: 92–103
where  represents the error in the model that estimates the
predicted feature Xi .
The remaining difficulty in solving (6) is knowledge of
the priors P (λ|H ). In general, the mapping λ provides a re-
lationship between features Yj and a specific predicted fea-
ture Xi by the function λj = i. The mapping itself can be
considered independent of the image features, if the same
features are extracted from every image irrespective of the
segmentation algorithm. In this case, the prior estimates
P (λ|H ) would be simplified to Pi (H ), where Pi (H ) is the
probability of picking hypothesis H given output Xi .
It follows that if N is the probability that the extracted
feature has no positive association to a predicted feature, that
is, the extracted feature does not contribute to the problem
Pi (H )
Li (H ) = (1 − N ) m (12)
k=1 Pk (H )
is the probability that the selected algorithm comes from
the list of hypotheses, the likelihood score can be computed
as (Ettinger et al. 1996)

n
p(Y |H ) = p(Y |X) = p(Yj |X, H )
j =1
 

n 
m
= NpNA (Yj ) + Li (H )p(Yj |Xi , H ) (13)
j =1 i=1
where, p(Yj |Xi , H ) is the likelihood that image features Yj
correspond to predicted segmentation features Xi under seg-
mentation hypothesis H .
The selected hypothesis is the one that maximizes the
likelihood score.
4 Features, Algorithms, and Segmentation Quality
To assess the feasibility of the proposed framework of
optimal algorithm selection, we have created a bank of
four segmentation algorithms. Each of these algorithms use
color and texture features computed from the input image.
A dataset of human segmentations (Martin et al. 2001) is
used as ground truth to learn the segmentation quality model
and compute the segmentation quality features used in the
4.1 Image Features
Low-level features of the image are computed as character-
istics defining the image context. Specifically, color and tex-
ture features are computed from the color image and the con-
verted gray-level image, respectively. For the color features,
Int J Comput Vis (2008) 80: 92–103 97

we use 1976 CIE L∗ a∗ b∗ color space separated into lumi-
nance and chrominance channels. Color distribution is mod-
eled with histograms constructed with kernel density esti-
mates. Histograms are compared with the χ 2 histogram dif-
ference operator (Puzicha et al. 1999) to obtain a feature for
each pixel. A brightness cue is also computed based on the
use of L∗ histogram for each pixel. Once again, the actual
feature for each pixel is the χ 2 histogram difference. Texture
features used are computed from the gray-level image based
on gray-level co-occurrence matrices (GLCM), fractal mea-
sures, and Gabor filters (Fogel and Sagi 1989; Laws 1980;
Tourassi et al. 2000; Tuceryan and Jain 1993).
4.2 Segmentation Algorithms
The segmentation algorithms used in this study were im-
plementations of Graph-based segmentation proposed in
Felzenszwalb and Huttenlocher (2004), Level set based seg-
mentation proposed in Brox and Weickert (2004), Normal-
ized Cut approach (Shi and Malik 2000), and the EDISON
segmentor (Comaniciu and Meer 2002). Each implementa-
tion used the same feature set extracted from the input image
that represents the color and texture content of the image.
Further, the parameters for each algorithm were optimized
to obtain the best segmentation performance across the en-
tire learning set of images. These specific algorithms were
chosen in this study as they exploit local features of the in-
put image and aim to realize a global partitioning. The algo-
rithms used only differ from each other in the way optimal
partitioning of the image is achieved.
4.3 Segmentation Quality
One way to characterize a segmentation algorithm is based
on a measure of the quality of segmentation achieved. The
answer to “What is a good segmentation?” is not an easy
one to quantify. Many approaches have been proposed to
address this issue and both analytical and empirical meth-
ods have been developed (Zhang 1996). We use a set of fea-
proximity, similarity, good continuation, closure as well as
symmetry and parallelism are computed for each segmented
regions, Si , resulting from the ith algorithm, and the corre-
sponding ground-truth region, S g . Specifically, the features
used are the ones defined in Ren and Malik (2003) and enu-
merated below:
1. inter-region texture similarity
2. intra-region texture similarity
3. inter-region brightness similarity
4. intra-region brightness similarity
5. inter-region contour energy
6. intra-region contour energy
7. curvilinear continuity.
We further, combine the inter- and intra-region features for
texture, brightness, and contour, respectively to generate a
feature vector for each segmented region in the image. The
overall quality measure is given by the evaluation function,
originally proposed in Liu and Yang (1994):
√  N ej2
X(I ) = N  (14)
Pj
j =1
where, N is the number of segments, Pj is the number of
pixels in segment j , and ej2 is the squared error of region j
as computed from each of the above features, independently.
The error for each region is given by the χ 2 histogram dif-
ference of the features of an algorithm’s region and the cor-
responding ground-truth region. This metric results in four
measures of segmentation quality based on texture, bright-
ness, contour, and continuity. The measure allows for char-
acterization of pixel to pixel match between the achieved
segmentation and the ground truth, but tends to favor under-
segmentation.
To avoid over- and under-segmentation, we also use two
measures that evaluate the segmentation quality based on the
number of regions (Borsotti et al. 1998; Zhang et al. 2006).
These are computed as:
√ N
 ej2
 
tures computed from the regions in the image that have been N N (Sj ) 2
X(I ) = + (15)
segmented as an output of each segmentation algorithm to 1000 × SI 1 + log Sj Sj
j =1
characterize the quality of segmentation. The Berkeley Seg-
mentation Dataset (Martin et al. 2001) provides ground-truth and
segmentation data. This dataset contains 12,000 manual seg-
mentations of 1,000 images by 30 human subjects. Half of 
N
Sj Sj
X(I ) = − g log g (16)
the images were presented to subjects in grayscale, and half S Sj
j =1 j
in color. We use the color segmentations for 300 images,
divided into training and test sets of 100 and 200 each, re- where Sj is the true segmentation of region j in image I
g
spectively. Each image has been segmented by at least 5 sub- and Sj is a ground-truth segmentation for region j . The first
jects, so the ground-truth, S g , for each region is defined by of these two measures penalizes very large number of seg-
a set of human segmentations. We consider a true segmen- ments while emphasizing the regional errors, thus achieving
tation region, S g , such that every pixel in that region is clas- a balance between under- and over-segmentation. The sec-
sified as belonging to that region by all the subjects. Fea- ond measure is information theoretic and reduces bias to-
tures based on Gestalt principles of grouping that include wards over-segmentation.
98
5 Experimental Results
To evaluate the proposed framework, we performed a per-
formance analysis on a subset of the Berkeley Segmentation
Dataset (Martin et al. 2001). The experiment was based on
a two-part approach, learning the required models based on
a learning dataset of images, and testing the learned mod-
els on a dataset of test images. We divided the dataset of
300 images into 100 images for training and the remaining
200 for testing. All the images in the training set were sub-
jected to segmentation based on each of the four algorithms.
Image features Y and the segmentation quality features Xi
dependent on each of the i algorithms were computed. In
addition, given the four segmentations for each image, the
optimal segmentation was chosen as the one that minimized
the overall error between the segmentation output and the
ground-truth. A loss function was introduced such that a loss
of 1 was assigned for each pixel in a segmented region that
did not belong to that regions ground-truth segmentation.
Hence, the total error was computed as the sum of all losses
divided by the total number of pixels in the image.
During the training phase, Pi (H ) had to be estimated giv-
ing a probability for picking a segmentation algorithm given
the segmentation quality measure Xi . To do so, Pi (H ) was
computed as a ratio of the number of times a minimum mea-
sure of Xi across all segmentation outputs resulted in select-
ing the correct algorithm by the total number of images. In
addition, the likelihood distributions for p(Yj |Xi , H ) was
assumed to be Gaussian distributed. This captured the re-
lationship between the image features and the likelihood of
obtaining the best segmentation with quality measure Xi un-
der hypothesis H . The parameters of this distribution were
estimated by computing the mean and covariance of the fea-
tures Xi for each segmentation algorithm. In addition, a
likelihood model for negative association pNA (Yj ) based on
image features was also estimated by computing the mean
and covariance of the quality features Xi belonging to the
segmentation algorithm with the highest squared error mea-
sured between the segmentation output and the ground truth.
Fig. 3 Precision-recall curves for the developed performance predic-
tion system on training and test images
Int J Comput Vis (2008) 80: 92–103
Having learned the required priors and likelihoods, the
probability for selecting each of the segmentation algo-
rithms based on image features was computed for each
image in the test dataset. In addition, the precision-recall
framework (Ren and Malik 2003; van Rijsbergen 1979) was
used to evaluate the performance of the proposed approach.
Precision is the fraction of detections which are true posi-
tives while recall is the fraction of true positives which are
detected. Figure 3 shows the precision-recall curve for the
training and test datasets. Both curves are almost the same
indicating that the performance predictor generalizes well
and that the likelihood models estimated do not overfit the
data. On the other hand, the curves could possibly be im-
proved by employing more complex models for the like-
lihood functions. More sophisticated models may provide
better performance than the simple models used here.
To evaluate the sufficiency of extracted features in de-
scribing the image context, the effect of color and texture
features on the performance of the predictor was analyzed,
independently. Figure 4 presents the performance of the im-
age features during training phase. It is clear that both color
and texture features contain independent information and
provide useful knowledge towards the prediction of the opti-
mal segmentation algorithm. Between color and texture fea-
tures, texture has more information relevant to the algorithm
selection process. In addition, the analysis also shows that
other features could potentially add more information to bet-
ter represent the image context.
Figure 5 shows an example of segmentation selection un-
der the developed model. Figures 5a and b show the input
image and the ground-truth segmentation, respectively. Fig-
ures 5c–f show the segmentation result obtained by each of
the candidate segmentation methods. In this case, the devel-
oped model chose the mean-shift segmentation algorithm,
which is the correct choice as that results in the minimum
error between the resulting segmentation and ground-truth
segmentation. The errors for each of the resulting segmen-
Fig. 4 Precision-recall curves for prediction performance based on
color, texture, and combined features as evaluated on the training set
of images
Int J Comput Vis (2008) 80: 92–103
Fig. 5 Resulting segmentation
of the input image (a) based on
each of the four candidate
algorithms (c)–(f). The
mean-shift algorithm was
selected as the optimal choice
tations, Figs. 5c–f, were 0.101, 0.053, 0.061, and 0.025, re-
spectively.
Figure 6 presents another example of the ability of the
developed framework to select the optimal segmentation al-
gorithm. Once again, Figs. 6a and b show the input im-
age and the ground-truth segmentation, respectively. Fig-
ures 6c–f show the segmentation result obtained by each of
the candidate segmentation methods. In this case, the de-
veloped model chose the level set approach, and the errors
associated with each of the algorithms c–f were computed
to be 0.155, 0.048, 0.355, and 0.148, respectively.
Figure 7 presents an example where the optimal segmen-
tation algorithm chosen by the developed model is in fact the
second best segmentation. Figures 7a and b show the input
image and the ground-truth segmentation, respectively. Fig-
ures 7c–f show the segmentation result obtained by each of
the candidate segmentation methods. The developed model
chose the graph-based segmentation algorithm, when in fact
the error associated with the level set approach in lower than
that of graph-based algorithm. The errors associated with
each of the algorithms c–f were 0.121, 0.120, 0.253, and
0.334, respectively. Based on visual inspection, it is clear
that both algorithms achieve acceptable segmentations and
99
have similar visual qualities. This result reinforces the fact
that the problem of segmentation cannot be solved based
on a purely analytical solution and requires incorporation of
psycho-physical perception. It is also safe to conclude that
the developed framework, while did not select the algorithm
with the lowest error, did select the algorithm that resulted
in one of the better segmentations.
Table 1 presents the prediction rate of the developed sys-
tem. To get a reliable estimate, we used a 5-fold validation
scheme to evaluate the prediction rate. All images were di-
vided into 5 groups, each with 60 images. In every test, only
one group of images is used as training samples and the
other 240 images are used as testing samples. After 5 it-
erations, all images have already been used for training and
1200 images for testing. The chosen algorithm by the perfor-
mance predictor is considered to be correct if the respective
segmentation is the one with the lowest error as compared
to ground-truth segmentation. Accordingly, the predictor’s
performance in selecting the best algorithm is computed to
be 86.4%. Each of the algorithms were ranked based on esti-
mated probabilities and the table below shows that the algo-
rithm having the second, third, and fourth largest probability
100 Int J Comput Vis (2008) 80: 92–103
Fig. 6 Resulting segmentation
of the input image (a) based on
each of the four candidate
algorithms (c)–(f). The level set
approach was selected as the
optimal choice

Table 1 Accuracy rates of the predictor in selecting the algorithm Table 2 Accuracy rates in achieving segmentation with the lowest er-
resulting in segmentation with the lowest computed deviation from ror against the ground-truth segmentation for each of the candidate al-
ground-truth segmentation. The table also shows how often other gorithms as evaluated against the entire image set using a 5-fold vali-
choices of the predictor were ones with lowest error dation scheme

Probabilistic predictor ranking Prediction rate (%) Segmentation algorithm Best segmentation rate (%)

First (max probability) 86.40 Graph-based 27.4

Second 11.70 Level set 26.8
Third 1.90 Normalized cut 22.3
Fourth 0.10 Mean-shift 23.5

was the one with lowest error in 11.7%, 1.8%, and 0.1% of
the cases, respectively.
Further, 5-fold validation was also performed for each
segmentation algorithm independently to compare the ben-
efit of the performance predictor. Table 2 gives the accu-
racy rates for each algorithm, where the rate was determined
based on the number of times the particular algorithm re-
sulted in segmentation with the lowest error. As seen, all al-
gorithms have similar accuracy rates indicating that no sin-
gle algorithm consistently provides a result with the mini-
mum error as compared to the ground-truth. In contrast, the
predictor has an accuracy of 86.4%, clearly indicating its
ability to select an appropriate algorithm based on the given
image.
To additionally evaluate the performance of the predictor,
we employ four quantitative performance measures intro-
duced in Freixenet et al. (2002), Martin et al. (2001), Meila
(2005), Yang et al. (2006), namely, Probabilistic Rand Index
(PRI), Variation of Information (VoI), Global Consistency
Error (GCE), and Boundary Displacement Error (BDE). PRI
indicates the fraction of pixel pairs whose labels are con-
sistent between the obtained segmentation and the ground
Int J Comput Vis (2008) 80: 92–103 101
Fig. 7 Resulting segmentation
of the input image (a) based on
each of the four candidate
algorithms (c)–(f). The
graph-based approach was
selected as the optimal choice,
while the error associate with
the level set approach was lower
than that of the graph-based
segmentation result

truth. VoI captures the amount of randomness in the ob- Table 3 Average performance on the subset of the Berkeley Database
chosen as the testing set for the performance predictor. PRI ranges be-
tained segmentation that cannot be explained by the ground tween [0, 1], with 1 being the best. VoI ranges between [0, ∞), which 0
truth. This is expressed in terms of the average conditional being the best. GCE ranges between [0, 1], with 0 being the best. BDE
entropy of the obtained segmentation. GCE measures the ex- ranges between [0, ∞), with 0 being the best
tent of dependency between the obtained segmentation and PRI VoI GCE BDE
the ground truth providing a measure of consistency be-
tween the two. Finally, BDE measures the offset error in Algorithm chosen by predictor 0.8445 2.2743 0.1754 7.6821
the boundary pixels of the obtained segmentation over the Graph-based 0.7821 2.4271 0.1965 8.8932
ground truth. Table 3 gives the comparison of the segmenta- Level set 0.7667 2.2053 0.2374 8.5473
tion obtained by the predictor against each of the algorithms Normalized cut 0.7564 2.7541 0.2353 8.3678
as computed over the testing data set. Mean-shift 0.7675 2.4572 0.2265 8.6031
As can be seen, the predictor outperforms the individ-
ual algorithms in terms of PRI, GCE, and BDE, each of
which measure the segmentation in terms of under- and not the minimal, but closer to the average of the VoI of the
over-segmentation. This can be considered indicative of the chosen algorithm for each image in the testing data set.
fact that the models used by the predictor in estimating seg-
mentation quality are represented accurately since the seg-
mentation quality metrics outlined in Sect. 4 clearly are the 6 Conclusions
ones that penalize over- and under-segmentation along with
pixel-level differences in each of the segmented regions. On This paper has proposed a probabilistic framework based on
the other hand, VoI, which is more indicative of the random- Bayesian theory for the performance prediction and selec-
ness between the obtained segmentation and ground truth, is tion of an optimal segmentation algorithm. The framework
102
models the optimal algorithm selection process as one that
accounts for the information content of an input image as
well as the behavioral properties of a particular candidate
segmentation algorithm. The input image information con-
tent is measured in terms of image features while the candi-
date segmentation algorithm’s behavioral characteristics are
captured through the use of segmentation quality features.
Gaussian probability distribution models are used to learn
the required relationships between the extracted image and
algorithm features and the framework tested on the Berke-
ley Segmentation Dataset using four candidate segmentation
algorithms. The results show that in 86.4% of the test im-
ages, whose characteristics vary greatly, developed approach
picks the optimal algorithm. The optimality of the algorithm
is characterized by the segmentation quality metrics used to
model the predictor. A quantitative performance analysis is
also performed on the segmentation results using four in-
dependent measures, the results of which are indicative of
the predictor’s ability to model the segmentation process en-
tirely based on the segmentation quality and hence capturing
the ability to perform segmentation independent of any one
segmentation algorithm’s intrinsic properties.
Future extensions of this work will focus on evaluating
different image features as well as measures of segmentation
quality. In addition, more realistic models for learning the
relationship between the image features and segmentation
quality measures need to be explored. The precision-recall
framework can be used to evaluate the quality of the chosen
model.
References
Borsotti, M., Campadelli, P., & Schettini, R. (1998). Quantitative eval-
uation of color image segmentation results. Pattern Recognition
Letters, 19, 741–747.
Brox, T., & Weickert, J. (2004). Level set based image segmentation
with multiple regions. In LNCS: Vol. 3175. Pattern recognition
(pp. 415–423). Berlin: Springer.
Chalmond, B., Graffigne, C., Prenat, M., & Roux, M. (2001). Con-
textual performance prediction for low-level image analysis algo-
rithms. IEEE Transactions on Image Processing, 10, 1039–1046.
Chiang, H. C., & Moses, R. L. (1999). ATR performance predic-
tion using attributed scattering features. In Proceedings of SPIE
(Vol. 3721, pp. 785–796). Bellingham: SPIE.
Chiang, H. C., Moses, R. L., & Potter, L. C. (2000). Classification per-
formance prediction using parametric scattering feature models.
In Proceedings of SPIE (Vol. 4053, pp. 7546–557). Bellingham:
SPIE.
Cho, K., Meer, P., & Cabrera, J. (1997). Performance assessment
through bootstrap. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 19, 1185–1198.
Comaniciu, D., & Meer, P. (2002). Mean shift: A robust approach to-
ward feature space analysis. IEEE Transactions on Pattern Analy-
sis and Machine Intelligence, 24, 603–619.
Duda, R. O., & Hart, P. E. (1973). Pattern classification and scene
analysis. New York: Wiley-Interscience.
Int J Comput Vis (2008) 80: 92–103
Ettinger, G. J., Klanderman, G. A., Wells, W. M., & Grimson, W. E. L.
(1996). Probabilistic optimization approach to SAR feature
matching. In Proceedings of SPIE (Vol. 2757, pp. 318–329).
Bellingham: SPIE.
Felzenszwalb, P., & Huttenlocher, D. (2004). Efficient graph-based im-
age segmentation. International Journal of Computer Vision, 59,
167–181.
Fogel, I., & Sagi, D. (1989). Gabor filters as texture discriminator. Bi-
ological Cybernetics, 61, 103–113.
Freixenet, J., Munoz, X., Raba, D., Marti, J., & Cufi, X. (2002). Yet
another survey on image segmentation: Region and boundary in-
formation integration. In ECCV’02: Proceedings of the 7th Eu-
ropean conference on computer vision—Part III (pp. 408–422).
London: Springer.
Fu, K., & Mui, J. (1981). A survey on image segmentation. Pattern
Recognition, 13, 3–16.
Haralick, R. M., & Shapiro, L. G. (1992). In Computer and robot vision
(Vol. 1, pp. 303–370). Reading: Addison-Wesley.
Konishi, S., & Yuille, A. L. (2000). Statistical cues for domain specific
image segmentation with performance analysis. In IEEE com-
puter vision and pattern recognition or CVPR I (pp. 125–132).
Laws, K. I. (1980). Textured image segmentation. Ph.D. thesis.
Liu, J., & Yang, Y. H. (1994). Multiresolution color image segmenta-
tion. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, 16, 689–700.
Martin, D. R., Fowlkes, C. C., Tal, D., & Malik, J. (2001). A database
of human segmented natural images and its application to evaluat-
ing segmentation algorithms and measuring ecological statistics.
In International conference on computer vision II (pp. 416–423).
Meila, M. (2005). Comparing clusterings: an axiomatic view. In
ICML’05: Proceedings of the 22nd international conference on
machine learning (pp. 577–584). New York: Assoc. Comput.
Mach.
Nair, D., & Aggarwal, J. K. (1996). A focused target segmentation par-
adigm. In Fourth European conference on computer vision (Vol. 1,
pp. 579–588). Berlin: Springer.
Pal, N. R., & Pal, S. K. (1993). A review on image segmentation tech-
niques. Pattern Recognition, 26, 1277–1294.
Pavlidis, T. (1977). Structural pattern recognition. Berlin: Springer.
Perner, P. (1999). An architecture for a CBR image segmentation sys-
tem. In ICCBR’99: Proceedings of the third international confer-
ence on case-based reasoning and development (pp. 525–534).
Berlin: Springer.
Puzicha, J., Hofmann, T., & Buhmann, J. M. (1999). Histogram clus-
tering for unsupervised segmentation and image retrieval. Pattern
Recognition Letters, 20, 899–909.
Randen, T., & Husoy, J. H. (1999). Filtering for texture classification:
A comparative study. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 21, 291–310.
Ren, X., & Malik, J. (2003). Learning a classification model for
segmentation. In International conference on computer vision
(pp. 10–17).
Reynolds, R. G., & Rolnick, S. R. (1995). Learning the parameters
for a gradient-based approach to image segmentation using cul-
tural algorithms. In Proceedings of the first international sympo-
sium on intelligence in neural and biological systems (INBS’95)
(Vol. 240). Los Alamitos: IEEE Comput. Soc.
Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
22, 888–905.
Shufelt, J. A. (1999). Performance evaluation and analysis of monocu-
lar building extraction from aerial imagery. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 21, 311–326.
Spann, M., & Nieminen, A. (1988). Adaptive Gaussian weighted filter-
ing for image segmentation. Pattern Recognition Letters, 8, 251–
255.
Int J Comput Vis (2008) 80: 92–103
Spann, M., & Grace, A. (1994). Adaptive segmentation of noisy and
textured images. Pattern Recognition, 27, 1717–1733.
Tourassi, G. D., Frederick, E. D., Vittitoe, N. F., & Coleman, R. E.
(2000). Fractal texture analysis of perfusion lung scans. Comput-
ers in Biomedical Research, 33, 161–171.
Tuceryan, M., & Jain, A. K. (1993). Texture analysis. Handbook of
Pattern Recognition and Computer Vision, 235–276.
van Rijsbergen, C. J. (1979). Information retrieval (2nd ed.). London:
Butterworths.
Yang, A. Y., Wright, J., Sastry, S. S., & Ma, Y. (2006). Unsupervised
segmentation of natural images via lossy data compression (Tech-
103
nical Report UCB/EECS-2006-195). EECS Department, Univer-
sity of California, Berkeley.
Zhang, Y. J. (1996). A survey on evaluation methods for image seg-
mentation. Pattern Recognition, 29, 1335–1346.
Zhang, X., & Haralick, R. M. (1993). Bayesian corner detection. In
British machine vision conference.
Zhang, H., Cholleti, S., Goldman, S. A., & Fritts, J. E. (2006). Meta-
evaluation of image segmentation using machine learning. In
IEEE conference on computer vision and pattern recognition I
(pp. 1138–1145).