Journal of Intelligent & Fuzzy Systems 31 (2016) 2543–2551 2543

DOI:10.3233/JIFS-169095
IOS Press

A hierarchical model to learn object

proposals and its applications
Liang Peng∗ and Xiaojun Qi
Department of Computer Science, Utah State University, Logan, UT, USA

PY
Abstract. Generating class-agnostic object proposals followed by classification has recently become a common paradigm

CO
for object detection. Current state-of-the-art approaches typically generate generic objects, which serve as candidates for
object classification. Since these object proposals are generic whereas the categories for classification are domain specific,
there is a gap between the generation of object proposals and the classification of object proposals. In this paper, by taking
advantages of the intrinsic structure and the complexity of each category of objects, we propose a novel tree-based hierarchical
model to learn object proposals, from top proposals produced by the existing object proposals generation methods. First,
we develop a tree-structured representation for each object to capture its hierarchical structure feature. Second, we propose
a 23D compact feature vector to represent objects’ visual features. Third, we formulate a learning schema which evaluates
OR
the objectness of each proposal. Experiments demonstrate the significant improvement of the proposed approach over the
state-of-the-art method in terms of object detection rate. An application is proposed based on this approach to help children
learn and recognize objects by their visual appearances and their sub-parts structures.

Keywords: Hierarchical tree model, object proposals, object detection, learning

TH

1. Introduction the number of potential candidate windows in each

image is extremely large; Second, to detect objects
AU

Object detection, as the task of locating and rec-
ognizing object categories in images and videos, is a
major research field in computer vision. Multi-class
object detection is challenging since it needs to not
only find the locations of the objects but also spec-
ify the category of the object at each location. A lot
of research work has been done in object detection
and recognition and achieved a lot of progress in
recent two decades. In earlier work, the problem was
treated as a binary classification problem for each
object category versus non-object category on a huge
number of windows (i.e., sliding windows) [7, 19]
in different sizes and aspect ratios scanned over each
image. This approach has two major drawbacks: First,
∗ Corresponding author. Liang Peng, Department of Com-
puter Science, Utah State University, Logan, UT, USA. Tel.:
+(785)236 1972; Fax: +(435)797 3265; E-mail: liang.peng@
aggiemail.usu.edu.
from n categories, n detectors need to be ran sepa-
rately on all windows of each image, which makes the
computational cost grow linearly with the number of
categories. Haar-cascades with boosting techniques
(e.g., AdaBoost) [10, 18] significantly boosted the
speed with integral images and quickly eliminated
a large number of non-object windows. These tech-
niques have achieved successful detection accuracy
on certain types of objects such as cars and human
faces. Segmentation-based [9, 16] and saliency-
based [1, 11] techniques were proposed to reduce the
search space for possible windows. These “filtered”
windows are also called proposals. In recent years,
the class-agnostic approaches [3, 17] were introduced
to generate proposals which are likely to contain
objects of interest. A (n + 1)-way classification with
n object categories and 1 background category is then
followed to classify each object. The ever-growing

1064-1246/16/$35.00 © 2016 – IOS Press and the authors. All rights reserved
2544 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
Fig. 1. An example of 20 generic object proposals generated by
EdgeBox.
CO
labeled data size and computing power (e.g., GPU
computing) made the aforementioned approaches
work with the deep neural network-based architec-
tures [5, 6, 8] (e.g., Convolution Neural Network
(CNN)). However, this type of approaches rely on
the deep architecture in both steps. The class-agnostic
generation of proposals is expensive and slow.
OR
Hence, object proposals generation remains a
bottleneck in object detection. Due to the rapid devel-
opment of mobile computing and cloud computing,
some applications of object detection needs partial
computing to be done on mobile devices [13–15].
TH
Quickly and accurately generating object propos-
als has been of a great topic of interest. A new
BING-feature-based objectiveness measure [2] was
proposed to quickly generate object proposals (at
300 frames per second) with a high detection rate
AU
(i.e., 90% with 50 proposals and 96.2% with 1000
proposals on the VOC 07 data set [4]). Another
approach, named EdgeBox [20], used edges as a
sparse yet informative representation, and the number
of contours that are wholly contained in a bound-
ing box as a measure to compute the objectness. It
achieved the state-of-the-art object recall rate (e.g.,
75% recall at Intersection over Union (IoU) of 0.7
using 1000 proposals on the VOC 07 data set [4]) with
the speed of approximate 0.25 seconds per image.
Existing methods for object proposals (e.g., BING
[2] and EdgeBox [20]) typically use measures of
likelihood of a bounding box containing objects to
retrieve top n bounding boxes as candidate proposals.
Due to the class-agnostic nature of these methods, all
generated object proposals are generic objects. On the
other hand, during the classification stage, the object
categories are usually bounded in a certain domain
due to the limited labeled training data sets. For exam-
ple, the VOC 07 data set [4] has 20 object categories
for training and the number of object categories in
generated object proposals is significantly more than
20. This difference means that the object propos-
als usually contain additional categories of objects,
which are not in the labeled categories for classifi-
cation. Figure 1 shows 20 generic object proposals
generated by an existing object proposals generation
method. These generic object proposals include the
aeroplane itself, a “propeller” part, and other parts of
the aeroplane. However, the labeled categories in the
PY
training data contain the “aeroplane” category instead
of the “propeller” category and other parts categories.
Similarly, a “bicycle wheel” might be detected as a
proposal whereas the classification labels only con-
tain the “bicycle” category; A “train window” might
be detected as a proposal but the classification labels
only contain the “train” category. The false posi-
tives caused by these gaps have significant negative
impact on object detection in terms of both recall and
precision. Consequently, it will further affect object
recognition accuracy in the object proposals classi-
fication stage. By reducing these false positives, we
could improve precision and retrieve more appropri-
ate candidate proposals to improve recall as well.
In the classification stage, all generated object pro-
posals, which are not in the labeled categories, are
treated as the “background” category. However, some
of these “background” objects are in out-of-domain
categories (i.e, object categories are not in the set of
object categories in the domain of problems being
addressed) and are therefore not truly “background”.
Hence, reducing these false positives in the object
proposals generation stage will also further improve
the classification accuracy.
We observe each object category differs from oth-
ers not only by its visual appearance as a whole
part, but also by its intrinsic structure such as their
sub-parts and the complexity of its structures. For
example, a bicycle differs from a train when being
looked as a whole part, which counts for visual
appearance as a whole. In addition, a bicycle con-
tains two “wheels” and one “frame” whereas a train
contains “multiple rectangular windows”. A bound-
ing box of the blue “sky” object may not contain any
sub-parts as an object since it’s relatively pure and
it’s true background. Hence, in additional to visual
appearance, sub-part structures and the complexity
of an object could be employed to construct richer
objectness to better represent the objects to reduce
the aforementioned two types of false positives and
 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
improve the recall rate for any domain specific object
detection task.
Driven by this intuition, we develop a model which
takes an object’s visual appearance as a whole and its
parts-based hierarchical structure into account. There
are four major novelties for the proposed work. First,
we develop a compact 23D feature vector as the visual
appearance representation of an object; Second, we
develop an algorithm to map each object to a hier-
archical tree structure representation, which captures
the relationships among sub-parts and the complexity
of the object structure. Third, we formalize a learn-
ing schema to measure each proposal’s objectness by
comparing visual features and tree structures using
the nearest neighbour method. Last, we introduce a
practical application utilizing our approach to help
children learn and recognize objects via automati-
CO
cally generated graph patches.
The paper is organized as follows: Section 2
introduces the proposed method in detail. Section 3
presents the experimental settings together with the
results. Section 4 describes a real-word application
utilizing our approach to help children learn and rec-
ognize objects. Section 5 concludes the work with
OR
summaries, discussions, and future work.
2. The proposed method
TH
The proposed method aims to select a few number
of proposals to achieve the same recall as obtained
by significantly more proposals produced by existing
object proposal generation method. As a high-level
view, we first employ the existing object proposal
AU
generator on each image to produce top n object pro-
posals. Since many of these proposals include generic
objects and backgrounds, and only a small portion
of them include the objects that belong to domain-
specific categories in the labeled training data, we
develop a learning schema to compute a confidence
score (i.e., objectness) for each object and re-rank top
n objects in each image. This learning schema aims
to rank category-specific objects towards the top to
improve the recall rate for different levels of n. In
other words, we will select a few number of proposals
from the top n object proposals to achieve the same
recall as obtained by n proposals produced by the
state-of-the-art methods. The subsections below are
organized as follows: Section 2.1 introduces the pro-
posed compact features for object’s appearance as a
whole. Section 2.2 describes the methodology to map
each object to a hierarchical structure represented by
2545
a tree. Section 2.3 formalizes the proposed learn-
ing schema to compute a confidence score of each
object using the compact features and the tree-based
hierarchical model.
2.1. Compact feature representation
Even though approaches based on deep neural net-
works could learn feature representations with high
discriminative power, it is quite costly in terms of
time. For object proposals generation, we want to
develop a fast learning schema to eventually rank
PY
object proposals from the category-specific dataset
higher than object proposals from other categories and
background. Hence, the constructed compact feature
should have high discriminative power to distinguish
objects from non-objects, and moderate discrimina-
tive power across different categories of objects.
We observe that object and non-object propos-
als differ a lot in terms of their properties. First,
non-object proposals usually contain background and
object proposals usually have one dominate object
present within the bounding box. Second, non-object
proposals are generally more evenly distributed and
have less variations in terms of their color pixels. In
contrast, object proposals are generally more skewly
distributed and have more variations due to the few
dominate colors in the main object and a small back-
ground region outside of the object boundary. Third,
from the spatial point of view, background proposals
are usually more evenly distributed among different
color pixels inside the bounding box while object pro-
posals would have a few dominant colors occupying
the majority region of the bounding box. Hence, the
variation-related and spatial-related properties could
be used to distinguish object proposals from non-
object proposals.
To this end, we define the first category of fea-
tures to measure the color variations by computing
variances in red, green, and blue channels (three val-
ues) and entropy of the luminance-based histograms
within the proposal (one value). We then define the
second category of features to measure the spatial
evenness of the color distribution. Specifically, we
convert the proposals into a histogram in red, green,
and blue color channels. We empirically choose
the top five bins with the highest frequencies and
compute their relative normalized frequencies. This
would produce 15 features (five relative normalized
frequencies for each of the three channels).
Finally, we observe that the positions of the object
proposals are not evenly distributed within an image.
2546 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
For example, the probability for object proposals near
the four corners or the edges of the image is far
less than the probability for object proposals near
the center of an image. Some aspect ratios of the
object proposals may occur much more frequent than
other aspect ratios due to the limited shapes of objects
in the entire set of object categories. Following this
intuition, we define the third category of features by
incorporating the distribution of the relative locations
and aspect ratios of proposals into extracted features
to better represent each proposal. Since the image
dimension (i.e., the width and the height) may vary
among different images, we use the relative positions
and dimensions of the proposals within an image
to define four additional features: the relative width
(ratio of the width of the proposal to the width of the
image), the relative height (ratio of the height of the
proposal to the height of the image), the relative hori-
zontal coordinate, and the relative vertical coordinate,
which are the coordinates of the upper-left corner of
the proposal divided by the width and the height of
the image, respectively. These four features would
capture the properties regarding the relative position
and the aspect ratio of a proposal. In all, the compact
OR
feature of the length of 23 is extracted for a proposal
and will be used to classify object and non-object
proposals later on.
To clearly explain the proposed features, we pro-
vide the mathematical definition for some of these
TH
features. For a given image I with width W and height
H and a proposal B in the image with x and y repre-
senting the horizontal and vertical coordinates of the
upper-left corner of B, respectively, let w and h repre-
sent the width and height of B and Pi,j,c represent the
AU
pixel value at the horizontal and vertical coordinates
(i, j) of I in the channel c of the RGB color space,
where c ∈ {R, G, B}.
For the first category of the proposed features,
the variance of B in a red color channel could be
expressed as:
y+h x+w
y+h x+w Pi,j,c=R
j=y i=x (Pi,j,c − j=y i=x
w×h
(w × h − 1)
The variances of B in green and blue channels are
similarly computed by substituting c with G and c
with B in equation (1), respectively.
To compute the entropy, we compute the histogram
based on the pixel counts for each of 768 bins (i.e.,
256 possible values times 3 channels), and apply the
standard entropy formula on values stored in 768 bins.
For the second category of the proposed features,
we use 64-bin histogram on each of the RGB channels
to compute the five highest normalized frequencies.
Firstly, we compute the histogram based on the pixel
counts of 64 bins on each of RGB channels. Secondly,
for each channel, we normalize the frequency of each
bin by dividing the total pixel counts in the channel.
Lastly, we rank relative frequencies and select top 5
relative frequencies in each channel to produce a total
of 15 values.
For the third category of the proposed features, we
compute the proposed relative horizontal coordinate
PY
(Xr), the relative vertical coordinate (Yr), the relative
width (Wr), and the relative height (Hr) as follows:
x
Xr = (2)
W
y
CO
Yr = (3)
H
w
Wr = (4)
W
h
Hr = (5)
H
2.2. Hierarchical tree structure
A total of n object proposal bounding boxes may
be retrieved by the proposal generator for an image.
Since each object or non-object bounding box could
include smaller bounding boxes corresponding to
sub-parts of an object, we could convert all bounding
boxes on each image to several trees in the follow-
ing way: each node represents a bounding box; If
a bounding box Ba is mostly included in another
bounding box Bb (defined by the ratio of the overlap
Bounding boxes on images Converted trees
)2
(1)
Fig. 2. Examples of bounding boxes and their corresponding trees.
 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
between Ba and Bb to Ba ), the corresponding tree
node Ta should be a child node of bounding box Bb ’s
tree node Tb . This condition will be applied to all
bounding boxes in an image. Figure 2 shows two
examples of this type of mapping.
Algorithm 1 Convert multiple bounding boxes in an
image to multiple trees
input: bbs: multiple bounding boxes; thresh: a pre-
defined threshold for overlap
output: trees: multiple trees built based on bound-
ing boxes
function name: convertBbsToTrees(bbs, thresh)
sortedBbsDescend ← sortBySizes(bbs) /*sort bbs
by sizes in descending order*/
trees ← new List /*initialize a new list to place all
resultant trees*/
CO
for bb in sortedBbsDescend do
insertSuccess ← False
for tree in trees do
if the ratio of the overlap between tree.rootBb
and bb to bb > thresh then
insert(tree, bb, thresh) /*see Algorithm 2
OR
for details of this function*/
insertSuccess ← True
break
end if
end for
if insertSuccess is True then
TH
continue /* process next bouding box */
else
newTree ← createNewTreeNode(bb)
trees.append(newTree)
end if
AU
end for
return trees
We then propose a method to build multiple trees
by considering all bounding boxes in an image. First,
we sort all bounding boxes by their areas (i.e., width
* height) in the descending order. Second, we use
the first bounding box (i.e., the biggest bounding
box) to create a root node as the initial tree and
sequentially process the remaining sorted bounding
boxes one-by-one. For each bounding box, we check
if it can be inserted in any of the existing trees
by calculating the ratio of the overlap between the
bounding box of the root of the tree and the candidate
bounding box to the candidate bounding box. If the
computed ratio is larger than a pre-set threshold, we
insert the bounding box into the currently examined
2547
tree. If the computed ratio is smaller than a pre-set
threshold for all existing trees, the bounding box
cannot be inserted into any of the existing trees, and
we use this bounding box to create a new tree by
setting it as the root node of the new tree. The detail
of the algorithm is shown in Algorithm 1.
Algorithm 2 Insert a bounding box into an existing
tree
input: bb: a bounding box; treeNode: the root node
of a tree; thresh: a pre-defined threshold for over-
lap
PY
output: altered tree after insertion
pre-condition: the ratio of the overlap between
treeNode.rootBb and bb to bb > thresh
function name: insert(treeNode, bb, thresh)
if treeNode.children is NULL /*base case*/ then
newNode ← createNewTreeNode(bb)
treeNode.addChild(newNode)
return
end if
for each child c in treeNode do
if the ratio of the overlap between c.rootBb and
bb to bb > thresh then
insert(c, bb, thresh) /*recursive call*/
end if
end for
newNode ← createNewTreeNode(bb) /* all chil-
dren of treeNode do not have sufficient overlap
with bb */
treeNode.addChild(newNode) /* Add newNode
as a child of treeNode */
return
The detail of the tree insertion function is summa-
rized in Algorithm 2, where the inputs are a bounding
box and a tree (i.e., treeNode) and the output is the
altered tree after the insertion. It should be noted
that the input bounding box is mostly included in
the bounding box corresponding to the root of the
input tree based on the pre-condition in Algorithm 1.
Therefore, calling this function will insert a bounding
box at the right location of the tree, where its parent
corresponds to the smallest bounding box which has
the sufficient overlap with the inserted bounding box.
We use recursion to implement this function as sum-
marized in Algorithm 2. Let’s denote the function by
insert(treeNode, bb, thresh). The based case is that
the treeNode has no children. In this case, we simply
create a new node for bb and add this new node as a
child of treeNode. If the treeNode has children, we
2548 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
will check if bb is mostly included in the bounding
box corresponding to any of the children. If yes, we
make a recursive call by passing this child, bb and
thresh as parameters. If bb is not mostly included in
the bounding box correspondings to any of the chil-
dren, we simply create a new node using bb and add
this new node as the child of treeNode. The details
of this algorithm are shown in Algorithm 2.
2.3. Learning schema to compute confidence
scores
Based on Sections 2.1 and 2.2, each object proposal
bounding box has a tree associated with it. If the root
node of a tree Ti corresponds to a bounding box Bi ,
all children of Ti correspond to all bounding boxes
inside of Bi where i ranges from 1 to the total num-
ber of trees. We compute feature Fi from box Bi as
the root node feature. Hence, for each bounding box
Bi,j within Bi , Fi,j represents its visual feature and
tree Ti,j represents its hierarchical structure, where j
ranges from 1 to the total number of nodes in the tree.
Since each bounding box maps to a tree, we can
compare two bounding boxes by comparing their cor-
OR
responding trees and their visual features. We define
the distance between two bounding boxes as:
Dist(Bi , Bj )
= w × FeatureDist(Fi , Fj )
TH
+(1 − w) × TreeDist(Ti , Tj )
where Bx , Fx , and Tx represent xth Bounding Box,
Feature, and Tree, respectively. FeatureDist(Fi , Fj )
is a distance measure between two feature vectors and
AU
TreeDist(Ti , Tj ) is a distance measure between two
general trees.
Here, we use the Euclidean distance to compare
two visual features and use the edit distance to com-
pare two trees.
For each object Bi , we compute its distance to each
Bk in the training data using Dist(Bi , Bk ) and find the
n closest Bk s. Among n Bk s, there may be p objects
and q non-objects (i.e., p + q = n). So, we define a
confidence score as:
p
ConfB = (7)
n boxes as the input to produce the training and the
The higher the confidence score, the more likely
the proposal being an object. Last, on each image, we
order all object proposals by their confidence scores
in the descending order. Suppose there are M bound-
ing boxes in the training set and N bounding boxes
in the testing set, similar to the nearest neighbour
algorithm, the computation complexity of the pro-
posed algorithm to compute the confidence scores of
all testing bounding boxes is O(MN).
3. Experimental results
The proposed method has been applied to the
Youtube-Object Dataset V2.2 [12] since we want
to test the proposed method on the video dataset.
The dataset contains a total of 720,000 frames with
PY
6,975 annotated bounding boxes from 10 categories
of objects. The annotated frames are divided into
training and testing sets. Specifically, one object was
annotated per image in the training set and multiple
objects were annotated per image in the testing set.
CO
There are 4306 annotated frames with 4306 bounding
box annotations in the training set. There are 1781
annotated frames with 2669 bounding box annota-
tions in the testing set.
Due to the nature of the video dataset, the frame
sequence within the same shot changes gradually and
the differences between frames within the same shot
are small. We select the first labeled image within
each shot as the representative image from that shot.
Some shots do not have any labeled image, we sim-
ply skip them. As a result, all images we selected are
from different shots. We call these labeled images
as shot representative images. Among these shot
(6) representative images, 870 images were from the
training set and 334 images were from the testing
set.
We apply EdgeBox [20] on every shot represen-
tative image and generate top n bounding boxes
per image. Since this is the video dataset and the
sizes of objects is relatively large in each image,
top 50 bounding boxes per image from EdgeBox
would detect nearly 70% of true objects. We use top
n = 50 in our experiment. We empirically set the
parameters for EdgeBox as follows: the step size
of sliding window search is 0.8; the non-maximum
suppression threshold is set to be 0.55, and min score
of boxes to detect is 0.01. As a result of this step,
we generate up to 50 bounding boxes for each shot
representative images. We will use these bounding
testing sets for the proposed method.
To produce the training set for the proposed
method, we compare the generated bounding boxes
with the ground truth annotations from the shot
representative images in the training set. We label
 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications 2549

Table 1
Comparison of the recall rates of the proposed method and EdgeBox for top n proposals when n = 1, 2, ..., 50
n 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
EdgeBox .10 .18 .22 .27 .29 .31 .33 .35 .36 .37 .39 .41 .42 .43 .43 .44
Proposed .11 .17 .22 .26 .29 .32 .37 .41 .44 .46 .47 .49 .51 .53 .54 .54
n 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
EdgeBox .46 .48 .49 .50 .51 .51 .52 .53 .55 .55 .57 .58 .58 .59 .59 .60
Proposed .54 .55 .55 .55 .57 .58 .58 .59 .59 .60 .60 .60 .61 .62 .62 .63
n 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
EdgeBox .60 .61 .61 .62 .62 .62 .63 .63 .64 .64 .64 .65 .65 .65 .65 .66
Proposed .63 .64 .64 .64 .64 .64 .65 .66 .66 .66 .66 .66 .67 .67 .67 .67
n 49 50
EdgeBox .66 .67

PY
Proposed .67 .67
The bold values indicates the higher recall rate for two methods.

0.5 and 0.7, we simply ignore them. We then apply

the proposed method described in Section 2 on shot

CO
representative images in the testing set to compute a
new confidence score for every proposal and re-order
them in the descending order. Same labeling process
has been applied to compute the recall.
To evaluate the performance of the proposed
method, we compute the recall rate of top n propos-
als ranked by EdgeBox and the proposed method.
OR
Table 1 shows the comparison of the recall rates
between EdgeBox and the proposed method with n
ranging from 1 to 50. Figure 3 shows the comparison
in curves. We clearly see that the proposed method
outperforms EdgeBox for most values of n. The con-
TH

vergence occurs at n = 50 is due to the fact we use

Fig. 3. Comparison of the recall curves of the proposed method
and EdgeBox for top n proposals when n = 1, 2,...,50.
a bounding boxes as positive (i.e., contains object)
AU
all 50 boxes from EdgeBox to re-order proposals.
It shows that the proposed method could use much
fewer top proposals (e.g., 40 proposals) to achieve the
similar recall as EdgeBox, using more top proposals

if Intersection over Union (IoU) is greater than 0.7 (e.g., 50 proposals). Figure 4 shows the qualitative
and as negative (i.e., does not contain object) if IoU results of the object proposals generated by the pro-
is less than 0.5. For boxes with IoU values between posed method.

Fig. 4. Qualitative results of the object proposals generated by the proposed method.
2550 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
4. One potential application
Using the method we described, we can design an
application to help children learn different categories
of objects. Given a set of images and a training set,
a number of object proposals can be automatically
generated by applying the proposed method on each
image using a training set. Since the proposed method
uses the hierarchical model to represent object pro-
posals as trees, each object proposal bounding box
may also contain a hierarchy of other bounding boxes.
We could crop out several bounding boxes from each
image. For each cropped out bounding box, we can
further crop out several bounding boxes which nat-
urally corresponds to different parts of the objects.
We can display these parts at random locations,
and promote children to use the parts of objects to
CO
fill corresponding object proposal holes by correctly
matching the object parts to the right positions. This
practice could enable children to gradually get famil-
iar with the concept of each category of “object” and
have a basic notion of how different parts form a
certain category of objects.
OR
5. Summary and discussions
In this paper, we propose a tree-based hierarchical
model to learn object proposals and design an appli-
TH
cation to teach children the concepts of objects. The
proposed work has the following contributions:
– Developing a compact feature vector to repre-
sent the object’s visual appearance
– Proposing a tree-based hierarchical model to
AU
capture the object’s internal structural features
– Formalizing a new objectness measure by
incorporating both visual and tree feature rep-
resentations
– Designing a practical application to teach chil-
dren to learn the concepts of objects using the
proposed method
We demonstrate the proposed method outperforms
the state-of-the-art method (i.e., EdgeBox) in terms
of the detection rate. Specifically, we use fewer object
proposals to achieve comparable recall rate achieved
by more object proposals produced by other methods.
Since object proposals generation usually serves
first step before classification for object detection
task, it will be interesting to use the proposed method
to generate the object proposals, followed by clas-
sification methods based on deep neural network to
test the possible improvement of final object detec-
tion rate. This is one direction of our future work.
Also, since the proposed method is more suited for
video datasets where the objects appear relatively
larger in each frame, we will combine the proposed
method with the tracking method to extend the object
detection from images to videos.
There are also some limitations of the proposed
method. Since the proposed method needs to build
the hierarchical structure based on the geometric rela-
tionships of object proposal bounding boxes, it does
not perform well when too many object proposal
PY
bounding boxes are generated per image and too
much overlap among all bounding boxes. To address
these limitations, we can fine tune parameters of exist-
ing object proposals generation methods to detect
a reasonable number of bounding boxes with good
overlap. In this way, the proposed method might cap-
ture better hierarchical structure of each object and
become more generally applicable to other domains.
References
[1] A. Borji, D.N. Sihite and L. Itti, Salient object detection:
A benchmark. In Computer Vision–ECCV 2012, Springer,
2012, pp. 414–429.
[2] M.-M. Cheng, Z. Zhang, W.-Y. Lin and P. Torr, Bing: Bina-
rized normed gradients for objectness estimation at 300fps.
In Computer Vision and Pattern Recognition (CVPR), 2014
IEEE Conference on, IEEE, 2014, pp. 3286–3293.
[3] D. Erhan, C. Szegedy, A. Toshev and D. Anguelov, Scalable
object detection using deep neural networks. In Computer
Vision and Pattern Recognition (CVPR), 2014 IEEE Confer-
ence on, IEEE, 2014, pp. 2155–2162.
[4] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn and
A. Zisserman, The PASCAL Visual Object Classes Chal-
lenge 2007 (VOC2007) Results. http://www.pascalnetwork.
org/challenges/VOC/voc2007/workshop/index.html
[5] P.F. Felzenszwalb, R.B. Girshick, D. McAllester and D.
Ramanan, Object detection with discriminatively trained
part-based models, Pattern Analysis and Machine Intelli-
gence, IEEE Transactions on 32(9) (2010), 1627–1645.
[6] R. Girshick, J. Donahue, T. Darrell and J. Malik, Rich fea-
ture hierarchies for accurate object detection and semantic
segmentation. In Computer Vision and Pattern Recogni-
tion (CVPR), 2014 IEEE Conference on, IEEE, 2014,
pp. 580–587.
[7] C.H. Lampert, M.B. Blaschko and T. Hofmann, Beyond slid-
ing windows: Object localization by efficient subwindow
search. In Computer Vision and Pattern Recognition, 2008
CVPR 2008 IEEE Conference on, IEEE, 2008, pp. 1–8.
[8] H. Lee, R. Grosse, R. Ranganath and A.Y. Ng, Convolutional
deep belief networks for scalable unsupervised learning
of hierarchical representations. In Proceedings of the 26th
Annual International Conference on Machine Learning,
ACM, 2009, pp. 609–616.
[9] B. Leibe, A. Leonardis and B. Schiele, Robust object
detection with interleaved categorization and segmentation,
 L. Peng and X. Qi / A hierarchical model to learn object proposals and its applications
International Journal of Computer Vision 77(1-3) (2008),
259–289.
[10] R. Lienhart and J. Maydt, An extended set of haarlike fea-
tures for rapid object detection. In Image Processing. 2002.
Proceedings. 2002 International Conference on, volume 1,
IEEE, 2002, pp. I–900.
[11] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang and
H.-Y. Shum, Learning to detect a salient object, Pattern Anal-
ysis and Machine Intelligence, IEEE Transactions on 33(2)
(2011), 353–367.
[12] A. Prest, C. Leistner, J. Civera, C. Schmid and V. Ferrari,
Learning object class detectors from weakly annotated video.
In Computer Vision and Pattern Recognition (CVPR), 2012
IEEE Conference on, IEEE, 2012, pp. 3282–3289.
[13] H. Qian and D. Andresen, Jade: An efficient energyaware
computation offloading system with heterogeneous network
interface bonding for ad-hoc networked mobile devices.
In Software Engineering, Artificial Intelligence, Network-
ing and Parallel/Distributed Computing (SNPD), 2014
15th IEEE/ACIS International Conference on, IEEE, 2014,
pp. 1–8.
[14] H. Qian and D. Andresen, Emerald: Enhance scientific
CO
workflow performance with computation offloading to the
cloud. In Computer and Information Science (ICIS), 2015
OR
TH
AU
2551
IEEE/ACIS 14th International Conference on, IEEE, 2015,
pp. 443–448.
[15] H. Qian and D. Andresen, An energy-saving task scheduler
for mobile devices. In Computer and Information Science
(ICIS), 2015 IEEE/ACIS 14th International Conference on,
IEEE, 2015, pp. 423–430.
[16] J. Shotton, A. Blake and R. Cipolla, Contourbased learning
for object detection. In Computer Vision, 2005 ICCV 2005
Tenth IEEE International Conference on, volume 1, IEEE,
2005, pp. 503–510.
[17] C. Szegedy, A. Toshev and D. Erhan, Deep neural networks
for object detection. In Advances in Neural Information Pro-
cessing Systems, 2013, pp. 2553–2561.
[18] P. Viola and M. Jones, Rapid object detection using a boosted
cascade of simple features. In Computer Vision and Pattern
PY
Recognition, 2001 CVPR 2001 Proceedings of the 2001 IEEE
Computer Society Conference on, volume 1, IEEE, 2001,
pp. I–511.
[19] C. Wojek, G. Dorkó, A. Schulz and B. Schiele, Sliding-
windows for rapid object class localization: A parallel
technique. In Pattern Recognition, Springer, 2008, pp. 71–81.
[20] C.L. Zitnick and P. Dollár, Edge boxes: Locating object
proposals from edges. In Computer Vision–ECCV 2014,
Springer, 2014, pp. 391–405.