Reliable Object Recognition using SIFT Features
Florin Alexandru Pavel #1, Zhiyong Wang #*2, David Dagan Feng #*3
#
School of Information Technologies, University of Sydney, NSW 2006 Australia
*
Department of Electronic and Information Engineering, Hong Kong Polytechnic University, Hong Kong, China
alexandruyo@yahoo.com
zhiyong@it.usyd.edu.au feng@it.usyd.edu.au
Abstract— SIFT (Scale Invariant Feature Transform) features
have been one of the most efficient descriptors for object
recognition. However, the excessive number of key points and
high dimensionality has limited its capacity in object recognition.
In this paper we present a novel method based on SIFT features
for reliable object recognition. At first, a matching tree is
constructed to eliminate non-essential key points. In order to
achieve viewpoint independence, a 3D model is constructed for
each object in the filtered SIFT feature space. Experimental
results on both Caltech 101 and COIL 100 datasets indicate the
effectiveness of our proposed algorithm.
I. INTRODUCTION
SIFT [1] is a local feature detection and description
technique. Local image features extracted using this technique
are invariant to rotation and scale changes and are robust to
other transformation that can appear in an image as well as
minor viewpoint changes. SIFT has been proven to be one of
the most reliable techniques to extract invariant features from
images [5], a number of object recognition systems adopting it
with great success [7, 8]. Also a number of improvements to
the technique have been proposed. Most of the research in
enhancing the effectiveness of SIFT [2, 3, 4, 6] has revolved
around better acquisition and description of low level image
features on a per image basis. Other research has combined
low level image features with high level information in order
to achieve better performance [9]. Local features obtained
using SIFT are not always useful for characterizing an object
for recognition purposes. In real word images features may be
generated from background responses or other not in focus
objects which included in the object model will increase
computational times and degrade recognition performance.
Therefore, there is a stringent need to identify and eliminate
features that can negatively influence the performance of the
recognition task.
In this paper we propose a new technique to reduce the
number of key points so as to improve the matching efficiency
and recognition performance. In addition, in order to further
achieve viewpoint independence for reliable recognition, 3D
object models are constructed to accurately capture spatial
relationships between features. The filtering step identifies
object features that present a higher level of reliably and
consistency throughout the training dataset and also
significantly reduces the overall object feature space which
decreases the size of the problem, making the system more
computationally efficient.
Capturing strong spatial relationships between features
extracted from different images and constructing a 3D object
MMSP’09, October 5-7, 2009, Rio de Janeiro, Brazil.
978-1-4244-4464-9/09/$25.00 ©2009 IEEE.
model improves recognition precision by creating the
possibility to differentiate between a false positive match and
a partial occlusion problem. The measurement of density of a
matching region of features as well as viewpoint
approximation gives, with a certain degree of confidence, a
solution to the problem of partial occlusion vs. false positive.
Experimental results show that using the technique described
in this paper an increase in object recognition precision as
well as a decrease in processing time can be achieved.
II. SIFT FEATURE FILTERING
A. SIFT (Scale Invariant Feature Transform)
SIFT – Scale Invariant Feature Transform [1] is one of
several computer vision algorithms aimed at extracting
distinctive and invariant features from images. Features
extracted using the SIFT algorithm are invariant to image
scale, rotation, and partially robust to changing viewpoints
and changes in illumination. SIFT algorithm applies a four
stage approach in extracting features from an image.
SIFT applies a four stage approach to in order o extract
local features. The first two stages are focus on extracting
feature location information and more importantly ensuring
repeatable and reliable feature locations. The last two stages
are focused on creating a feature description. First an
orientation is assigned, making the feature invariant to
rotation, and then the descriptor is created. A set of 16
histograms with 8 orientation bins is used in calculating the
feature descriptor resulting in a feature vector 128 elements
long.
The invariance and robustness of the features extracted
using this algorithm makes it an extremely good candidate for
object recognition and it is an often used in detection /
description schemes, achieving one of the best performance
figures from all current feature extraction techniques [5].
However, due to the amount of features extracted and
variations present in images building a reliable object
recognition system based on SIFT features is still a
challenging task. SIFT approach transforms an image or an
image dataset into a large collection of local feature vectors.
Each feature vector possesses invariance to scale, translation,
orientation and noise. Usually between 1000 to 2000 features
are extracted from an average size image giving the possibility
to recognize an objects with substantial levels of occlusion
[5,1] but this also creates a set of problems over large datasets
of images due to the amount of features and false positive
matches that can be generated.
B. Feature filtering by longest matching trees
Real-world images, in their vast majority, always include
content that is not related to, or has only a weak relationship
with the objects of interest present in the image. Feature
extraction techniques are indiscriminative and have no
knowledge of image content therefore background responses
are identified as features and consequently used in the creation
of the model. For small datasets of images this proves not to
be a major problem but when larger datasets are used,
background response can cause performance degradation as
the dimension of the datasets grows.
Using SIFT the amount of features generated from an
image is in the order of thousands. Furthermore, a large part
of the features generated are due to background clutter and not
all of the features can be considered highly descriptive and
consistent for the object contained in the image over a large
image dataset. In order to address the problems generated by
the amount and quality of extracted features we are proposing
a feature filtering technique called Longest Matching Trees,
aimed at identifying features that have a higher degree of
consistency and distinctiveness throughout the object space.
Figure 1. Visual representation of matching trees
This technique identifies features that are only part of the
object space and can describe that object consistently and
distinctly throughout the training set. This approach measures
the propagation of a set of features throughout the object
space in order to select features that have a certain degree of
repeatability and consistency. In Figure 1. a visual
representation of matching trees in an object training dataset is
shown. We can observe that different sets of features are part
of longer or shorter matching trees, the aim of this technique
being to construct and select matching trees that have the
longest length. The length of a tree is given by the number of
object representations that the features contained in the tree
are part of, this giving a very good measurement of the
repeatability and descriptive power of a set of object features.
This step eliminates less descriptive features and features
generated by background clutter which heavily impact
recognition performance.
Experimental results show that only a small percentage of
features can in fact be used in describing an object
consistently and reliably. Considering that SIFT can generate
a few thousand features per image, depending on image size
and content, over a large image dataset the elimination of
background clutter and unrepresentative features is of great
concern as it significantly affects both recognition precision as
well as computational efficiency.
The creation of the object feature space from a large image
dataset with cluttered background poses its own challenges
when aiming to construct an object feature space with
minimal interference from object surroundings.
The method proposed requires analysing the entire image
dataset and has the following steps:
 Feature extraction. For each image SIFT features are
extracted and each set of features is annotated with the
image id consequently building a raw feature space.
 Matching tree construction for every feature. The matching
tree for a feature is constructed from the raw feature space,
extracted in the previous step, by performing a top-down
and than a bottom-up similarity matching. Each new feature
set from a new object representation is matched with the
current features contained in the tree and features that match
are introduced in the tree. A bi-directional approach is used
so that all features that share the same level of similarity are
included in the tree with the smallest number of iterations.
 Filtering. The longest matching trees are selected, where the
length of a tree is given by the number of object
representations that the features in the tree are part of, and
the resulting collection of features is then used in the
creation of the object model.
III. OBJECT MODEL CONSTRUCTION
Constructing a 3D model of an object presents its very own
sets of advantages given by capturing strong spatial relations
between features and more importantly capturing spatial
relations between features extracted from different object
views. A 3D object model enables the creation of feature
clusters or feature regions that were captured from different
object views at different scales, noise levels, orientations and
possible deformations. We created such a model in order to
capture strong spatial relations between features extracted
from different representations of the object. Such a construct
requires a relative object-to-object scale, orientation and
position estimate in order for extracted features to be located
as close to their actual position on the real object as possible.
Features are accurately mapped to the object model which
further improves the detection of partially occluded vs. false
positive matches by calculating the density of a matching
region of features when computing the similarity between the
model and a new image. This is of great importance because
only measuring the number of matching features between the
object model and a new image does not allow differentiating
between a partial occlusion problem and a false positive
match. Part of the novelty of the technique is given by not
only being discriminative in selecting features but actually
using the viewpoint to achieve viewpoint independence.
A. Pre-processing
The creation of the object model requires the following pre-
processing steps:
 Single scale approximation. The creation of a three
dimensional object model requires a scale approximation
between object representations in the training set in order
to accurately map object features on the object model. In
our approach scale is computed relative to an object
representation in the training set which is selected and
given a scale value of 1. The object representation that will
give the overall scale of the object model is chosen using
the information provided by the filtering step. In order to
approximate scale between two object representations, we
compute the Euclidian distance between all the matching
pair of features, generating a high enough number of
values on which outlier detection can be performed
successfully. The technique consists in calculating the
ratio between the distances of every two features from the
same image that match another two features from another
image. On the values obtained outlier detection is
performed and the relative scale between two object
representations is extracted as a percentage.
 Orientation approximation. In order to accurately create a
3D model all object representations have to poses the same
orientation. This is particularly important in order to assure
that features describing the same region of the real object
also describe a similar positioned region on the object
model. Approximating a relative orientation between two
object representations is calculated by approximating the
angle offset between two segments that are described by
matching pair of features. We divide the space for each
object representation into 4 regions (0, π/2, π and 3π/2)
and using the centroids of the two feature sets and
matching feature locations we calculate a third point. The
3 points calculated describe a right triangle from where an
angle can be extracted. The angle offset between every
matching pair of features from the two feature sets is
calculated and outlier detection is performed. The average
of the remaining values describes the relative rotation
between two feature sets and following this approach we
can compute the relative angle between two object
representations.
 Position approximation. Position approximation on X and
Y axes is calculated between two object representations by
calculating the centriods for each one and then calculating
the offset between centroids. Following the iterative
approach used in scale and orientation approximation we
ensure that the most similar representations and their
feature sets are used in approximating the position.
Position on the Z axis is calculated by first computing
bounds, height and width of the object. This is possible
because of the filtering step which ensures, with a certain
degree of confidence, that features are part of the object.
First features on the boundary are indentified which are
used in the calculation of the polygonal shape of the object.
Having an object shape allows the estimation of object
dimensions. Using the angle information given by the
dataset and the objects dimension a relative rotation of the
Z axis is computed considering that the object space has
spherical properties.
In order to propagate the scale, orientation and position to
all image representations and have a certain confidence on the
values extracted an integrative approach is used where the
decision to approximate scale, orientation and position
between two images is given by computing a similarity
distance between every two images of the training set . The
similarity distance between two images is extracted from the
data provided by the feature filtering technique as the
information contained in the longest matching trees of features
is the best indicator on how many features two images share.
B. Object model
Computing scale, orientation and position information for
every object representation the feature locations on the object
model can be calculated to match the location on the real
object. The model construction allows positioning features
based on their positioning on the real object, therefore creating
strong spatial relationships between features. In real-world
images an object‟s viewpoint has an important role due to the
fact that object description can change completely if the
viewpoint change is large enough.
Figure 2. . Object model representation - dataset images
In Figure 2. visual representations of two object models from
the dataset used to validate the systems are presented as well
as part of the training dataset with a viewpoint change of 90
degrees; red dots represent the location of features. The
resulting model is a spheroid used to preserve features
location, consequently preserving the viewpoint.
IV. OBJECT RECOGNITION
For matching between the object model and a new object
representation The Best-Bin-First (BBF) [1] algorithm is used
which is a more computationally efficient approach to the
nearest neighbor problem. Spatial relationships between
features, captured in the object model, are used to further
improve the reliability of this task by first eliminating false
positive matches based on viewpoint and then approximating
the density of a matching region of features on the object
model. Accurately positioning features of an object in a 3D
space can improve recognition performance, considering that
most deformations of objects are plastic in nature and that
properties such as proximity are kept.
A. Viewpoint false positives filtering
The 3D model allows for an „a posteriori‟ feature filtering
based on viewpoint when matching features between the
model and a new object representation. This is based on the
fact than no more than one hemisphere of the object model
can be visible from a single viewpoint; therefore all matching
features have to be included in one hemisphere.
This viewpoint feature filtering technique is based on
approximating the viewpoint of a new object by
approximating the centroids of the matching set features on
the model space and verifying if the matching features are part
of the described hemisphere.
The elimination of false positive matching features is of great
concern in order to reliably perform the task of object
recognition. The false positive feature elimination proposed in
this paper is part of the recognition process and is based on the
assumption that object features are viewpoint dependent. As
the description of the object can totally change with the
viewpoint this has proved a valid assumptions and
experimental results have validated this assumption.
B. Deformable patches of features
Positioning features on a 3D model with a high degree of
accuracy creates the possibility of identifying patches on the
object models generated by matching a new object
representation to the object model. The density of the patch is
used in the decision if matching features are part of a partially
occluded object or are false positives. This is done be
computing the density of a patch in regards to the object
model. The density is given by approximating the area of the
polygon described by the location of matching features on the
object model and dividing it by the number of features.
Lower densities of features in a region on the object model
can be mainly attributed to false positives whereas high
density patches are most likely to be representing a partially
occluded object.
Figure 3. Partial occlusion problem. High density patch of matching features
on the object model. Red dots represent the position of matching features
In Figure 3. an example of a high density patch on the object
model is shown, generated by a partially occluded object. The
density of a region of matching features has proven,
throughout many experiments, to be a very good solution in
differentiating between a false positive set of features and a
partial occlusion problem with eliminating any true positive
results. Although deformations of an object in an image are
common, especially in the case of more plastic objects like
animals, they tend to be topological in nature, preserving the
proximity or locality.
V. EXPERIMENTS AND RESULTS
A. Experimental results on Caltech
Experimental results using the Caltech 101 [11] have
shown a reduction in the global feature space by
approximately 40% to 90 %, depending on image content, the
amount of background clutter and similarity distance between
object views. Feature space filtering is of great concern as
matching between the features contained in the object model
and the features extracted from a new object representation is
computationally heavy. By identifying consistent features that
are part of the actual object space the precision of the
recognition task is increased and more importantly it remains
constant over large datasets.
a) b)
Figure 4. Unfiltered and filtered features.
a) Unfiltered features extract from the image (in blue).
b) Filtered features based on longest matching tree method (in yellow)
In Figure 4, two representations of the same object are shown,
where the left image has depicts all the features extracted
using SIFT and the right image depicts all the features that
remain after the filtering step. We can observe that after the
filtration method is applied the amount of features is reduced
considerably and most of the features are concentrated around
the object in focus (in this case stop sign and face).
In order for an object recognition system to be used in real
world applications it has to provide a way to compensate for a
wide variety of transformations that can occur in an image,
but more importantly it has to be able to learn an object model
from low quality data. From the object recognition task point
of view low quality data is represented by sets of images that
present a wide array of transformations and where the object
in focus does not represent the dominant region or part of an
image. Learning and object model over this type of datasets is
very challenging and significantly affects recognition
precision as features generated by background content account
for a large number of false positive matches. The filtering
technique proposed in this paper is aimed at addressing this
problem by identifying features that are only part of the object
space and have a certain degree of consistency. Processing
time, one of the most influenced measures by a variety of
factors, also decreases linearly with the decrease in the
number of features used in the object recognition
process.
Figure 5. Unfiltered vs. filtered features
In order to better examine the performance results of the
feature filtering technique (longest matching trees) we choose
not to average the results as this will eliminate spikes in the
number of unfiltered and filtered features, which are of great
importance when analyzing the results. In Figure 5 the
performance results of the filtering technique over 51 and 21
images are shown from the Caltech 101 image dataset.
Because most of the images have an average size and present
a high amount of background clutter the number of raw or
unfiltered features has a very high variance from image to
image. This is not an indicator on how many features are
extracted from an image for an object but a clear indicator of
the influence of background clutter. The number of filtered
features presents less variance as the number of features of an
object per image tends to remain constant, this being
dependent on object scale and the viewpoint from which it
was captured.
On the two object categories in Figure 5 we achieved a
reduction in the object feature space of approximately 80%,
removing most of the features generated by background
clutter. Identifying only reliable features of an object can
significantly increase the computational efficiency of the
system and make the object recognition task more reliable.
B. Experimental results on COIL-100
The COIL-100 dataset [10] consists of 7200 images from
100 objects which have been placed perpendicularly in front
of the camera resulting in a total of 72 representations of an
object with a 5 degree variation between them.
In Figure 6 a representative image for each object in the
dataset is shown. The dataset presents a diverse range of
objects that have also a wide range of textures, shapes and
colors.
Figure 6. COIL dataset – 100 images from each of the 100 object categories
For every set of images representing an object 25 % are
chosen as training data and 75% are chosen as testing data. An
object having 72 representations results 18 object
representations being used for training and 54 used to perform
test.
In Table 1. recognition results on the COIL-100 dataset are
shown. We achieve a precision of 99% with a recall of 90%
resulting in an F-measurement of 94%. These values show
that the object recognition system proposed in this paper can
be reliably and successfully used for the task of object
recognition over large datasets.
Table 1. Recognition accuracy
Method Precision
Snow[13] 92%
G-SVM[13] 97%
KL-SVM[13] 98%
Proposed approach 99%
In Figure 7. the performance results of the filtering
technique, longest matching trees, are shown which resulted in
a reduction of approximately 40% of the object feature space
significantly increasing computational efficiency of the
system. The filtering technique apart from allowing an
accurate construction of the object model by providing the
similarity distance between object representations also
improved the performance of the proposed system as only
features that have a higher degree of repeatability and
consistency are included in the object model.
Figure 7. Unfiltered vs. filtered number of features – COIL-100 dataset
 VI. DISCUSSIONS
The analysis of pairs of objects from different categories that
contributed mostly to the drop in the Recall value resulted in
the identification of interesting characteristics of the model.
In Figure 8 a comparison between two object categories is
shown, these categories registering between them almost 60 %
false positive identifications. Because the problem of partial
occlusion in the model is addressed as an identification of
matching patches of features, this results in cataloguing
images from these categories as a partial occlusion problem.
Figure 8. Comparison between two object categories that generated the
most number of false positives
In Figure 9 a comparison between two objects from
different categories is presented and we can observe that
partially occluding the objects results in completely
eliminating the possibility of distinguishing between them,
even from the human recognition capabilities perspective. The
generation of false positives on these objects categories is not
necessarily affecting the future performance measurements on
other datasets as it indicates good generalization capabilities.
This perspective is has been included in future work as it
indicates the possibility of learning an object model from two
or more categories and creating an inheritance tree of sub-
models to distinguish between objects that share a set of
features but are from different categories.
Figure 9. Comparison between two object categories that generated the most
number of false positives (only matching regions are shown).
VII. CONCLUSIONS
We presented a novel approach for reliable object
recognition by filtering local features to reduce irrelevant
features and to reduce computational complexity, and utilizing
3D models created from filtered SIFT feature space to achieve
viewpoint independence. Feature filtering allows for the
detection of features that can reliably describe that object
throughout the overall feature space and can eliminate much
of the background clutter. Features generated by background
clutter or have low quality features can generate, over large
datasets, false positives and reduce the certainty of a match.
Experimental results show that the proposed filtering strategy
is very effective in identifying reliable features of an object
and further improving the performance of the object
recognition system as well as computational efficiency.
Constructing a 3D space for the object model where features
are mapped accurately onto it introduces the possibility of
creating a region based approximation of a matching set of
features between the model and a new image by
approximating the density of those features on the model‟s
surface. This also allows the filtering of matches as an outlier
detection problem. This is done by approximating the
viewpoint and building on the fact that matching features must
be included in the hemisphere described by that particular
viewpoint. The experimental results on two widely used
datasets have shown that the proposed object recognition
method can improve both recognition precision and
computational efficiency by focusing the recognition process
on relevant object features and using feature location, in a 3D
space, as a decisional factor.
ACKNOWLEDGEMENT
The work presented in this paper is partially supported by
ARC and Hong Kong Polytechnic University grants.
REFERENCES
[1] D. G. Lowe, “Distinctive image features from scale-invariant
keypoints,” Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, 2004.
[2] A. Suga, K. Fukuda, T. Takiguchi, and Y. Ariki, “Object recognition
and segmentation using sift and graph cuts,” in 19th International
Conference on Pattern Recognition. Dec. 2008, pp. 1–4.
[3] F. Liu and M. Gleicher, “Region enhanced scale-invariant saliency
detection,” IEEE International Conference on Multimedia and Expo, vol.
0, pp. 1477–1480, 2006.
[4] Y. Ke and R. Sukthankar, “Pca-sift: a more distinctive representation for
local image descriptors,” in 2004 IEEE Computer Society Conference
on Computer Vision and Pattern Recognition, vol. 2, 2004, pp. 506–513.
[5] K. Mikolajczyk and C. Schmid, “A performance evaluation of local
descriptors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp.
1615–1630, October 2005.
[6] Y. Cui, N. Hasler, T. Thormählen, and H.-P. Seidel, “Scale invariant
feature transform with irregular orientation histogram binning,” in
ICIAR ’09: Proceedings of the International Conference on Image
Analysis and Recognition. Springer, 2009.
[7] Q. Fan, K. Barnard, A. Amir, A. Efrat, and M. Lin, Matching slides to
presentation videos using sift and scene background matching,” in
MIR ’06: Proceedings of the 8th ACM international workshop on
Multimedia information retrieval. New York, NY, USA: ACM, 2006,
pp. 239–248.
[8] J. Chen and Y.-S. Moon, “Using sift features in palmprint
authentication,” 19th International Conference on Pattern Recognition.
ICPR 2008, Dec. 2008, pp. 1–4.
[9] H.-H. Jeon, A. Basso, and P. Driessen, “A global correspondence for
scale invariant matching using mutual information and the graph
search,” in IEEE International Conference on Multimedia and Expo,
July 2006, pp. 1745–1748.
[10] S. Nene, S. K. Nayar, and H. Murase, “Columbia object image library
(coil-100),” Tech. Rep., 1996.
[11] F.-F. Li, F. Rod, and P. Pietro, “Learning generative visual models from
few training examples: An incremental bayesian approach tested on 101
object categories,” in CVPR ’04: Proceedings of the Computer Vision
and Pattern Recognition Workshop, 2004, p. 178.
[12] Libsift - scale-invariant feature transform implementation. [Online].
Available: http://user.cs.tu-berlin.de/ nowozin/libsift/
[13] T. Pham and A. Smeulders, “Sparse representation for coarse and fine
object recognition,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 28, no. 4, pp. 555–567, April 2006