See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221110794

Who Blocks Who: Simultaneous clothing segmentation for grouping images

Conference Paper in Proceedings / IEEE International Conference on Computer Vision. IEEE International Conference on Computer Vision · November 2011
DOI: 10.1109/ICCV.2011.6126412 · Source: DBLP

CITATIONS READS
75 639

2 authors, including:

Haizhou Ai
Tsinghua University
131 PUBLICATIONS 4,492 CITATIONS

SEE PROFILE

All content following this page was uploaded by Haizhou Ai on 18 November 2014.

The user has requested enhancement of the downloaded file.

 Who Blocks Who: Simultaneous Clothing Segmentation for Grouping Images

Nan Wang and Haizhou Ai

Computer Science and Technology Department
Tsinghua University, Beijing, China
wang-n04@mails.tsinghua.edu.cn, ahz@mail.tsinghua.edu.cn

Abstract 5
1 2 3 6
0 4
Clothing is one of the most informative cues of human
appearance. In this paper, we propose a novel multi-person
clothing segmentation algorithm for highly occluded im-
ages. The key idea is combining blocking models to address
the person-wise occlusions. In contrary to the traditional
layered model that tries to solve the full layer ranking prob-
lem, the proposed blocking model partitions the problem
into a series of pair-wise ones and then determines the local
blocking relationship based on individual and contextual in-
formation. Thus, it is capable of dealing with cases with a
large number of people. Additionally, we propose a layout
model formulated as Markov Network which incorporates
the blocking relationship to pursue an approximately opti-
mal clothing layout for group people. Experiments demon-
strated on a group images dataset show the effectiveness of Traditional Method Our Method
our algorithm. Figure 1. Clothing segmentation for highly occluded group images
will benefit from blocking information between neighbors. Ar-
rows in the upper-right image indicate the blocking relationship.
1. Introduction Figures in this paper are best viewed in color.

Clothing is one of the most informative cues of human
appearance. It has been proven that clothing segmentation
will benefit human detection [18] and recognition [9], pose
estimation [16] and human sketches [5].
This paper addresses the problem of multi-person cloth-
ing segmentation with high occlusions, since numerous
group pictures are captured in family gatherings and social
activities. To analyze these images, inter-person occlusions
must be taken into account.
Existing clothing segmentation algorithms [9][11] for-
mulate the problem in the MRF framework with differ-
ent clothing modeling approaches. In [11], clothing re-
gion (foreground) is initialized by torso detection based on
dominant colors determination, while in [9], clothing mask
is learned by mutual information analysis based on human
identities. In addition, Gallagher et al. [9] show that cloth-
ing segmentation and human recognition can benefit each
other. However, neither of the algorithms consider person-
wise occlusions explicitly. Note that segmentation results
in [9] seem to show its ability to address occlusions, but it
is achieved by co-segmenting multiple images of the same
person with the same clothes.
There have been a number of approaches to segment ob-
jects with partial occlusion. Some innovative papers use
part-based models. Winn et al. [21] and Hoiem et al.[10]
handle the object occlusions by using asymmetric pair-wise
potentials that pursue consistent layout output. Kumar et al.
[15] allow for self occlusion by building a Layered Picto-
rial Structure for specific class. Clothing shapes differ from
each other dramatically due to severe occlusion (Fig. 1). It
will be more appropriate to choose global masks for each
person, since local parts are difficult to learn when using
such inconsistent data.
A similar work to ours is the layered model introduced
by Yang et al. [22], where occlusions are handled by ob-
ject depth estimation. The layer order is searched in a brute
force way based on detector confidence and bottom edge lo-
 Data Preparing
1 0
0.66
Blocking Prob.
(Random Forest)
Clothing Shapes
(Random Forest) 𝑦0
𝐺 𝑥, ℰ
Input Multi-Person Clothing Model
Figure 2. Graphical diagram of our algorithm. Input data are the image [8] and face detection results [12]. Our algorithm outputs not only
the multi-person clothing segmentation results but also the blocking relationship between neighbors.
cations. Group images can contain dozens of people, which
makes the brute force search intractable sometimes. More-
over, it is not necessary to determine the occlusion relation-
ship between any pair of persons.
Our work is also inspired by the work of Eichner et al.
[7], where inter-person occlusions are modeled based on
relative face location and image border to facilitate multi-
person pose estimation. However, relative face location can
hardly help the occlusion determination for people abreast.
We argue that clothing self-similarity is more helpful in this
case and integrate the two cues in the form of Random For-
est.
In this paper, we propose to estimate the clothing shapes
for group people with a global view of the scene. We treat
each person as a node and model their interaction (block-
ing relationship) as edges. The reason why we use such a
person-graph (Fig. 1) is that the variation of clothing shapes
in group images can be distinguished by their individual and
context information (Note that in this scenario, human poses
do not change a lot, so clothing shapes are more affected by
inter-person occlusions). Pixel-level segmentation can ben-
efit from the top-down guide.
The key aspect of our method is the use of Blocking
Model for occlusion reasoning. As mentioned before, it is
intractable to search for the full layer order in a brute force
way for group images. However, it is not necessary to re-
cover the full order, since local blocking relationship is in-
formative enough for clothing shape decision. Thus, in con-
trary to the traditional layered model used in [22], we parti-
tion the full layer ranking problem into a series of pair-wise
ones. A Blocking Model combining individual and context
information is built to infer the pair-wise layer order.
Taken the inferred blocking results as prior information,
we propose a Layout Model formulated as Markov Net-
work to pursue the approximately optimal clothing layout
Bayesian Model
1 0
0.34
Blocking Model
0
1
𝑏𝑖𝑗 𝑃 𝑏𝑖𝑗
Layout Model
𝑦1 (Markov Network)
Output
for group people. Therefore, the ambiguities in overlapping
region are reduced to guide the bottom-up segmentation al-
gorithm more accurately.
Our main contributions are: (1) an effective Blocking
Model to estimate the inter-person occlusion, which inte-
grates individual and context information and is capable of
scaling to large number of people; (2) a Layout Model to
generate approximately optimal clothing layout for group
people conditioned on the inferred blocking relationship.
The rest of the paper is organized as follows: we de-
scribe our probabilistic model in Sec. 2 including the Lay-
out Model and the Blocking Model, and introduce the de-
tailed learning and inference algorithm in Sec. 3 and Sec. 4
respectively. Experiments are demonstrated in Sec. 5.
2. Multi-Person Clothing Model
Inter-person occlusions in group images are usually se-
vere and dense. An extreme case is shown in Fig. 1, where
some people’s bodies are completely blocked. In such sce-
narios, the largely-blocked people can only located through
face detection (rectangles in Fig. 1) rather than human de-
tection.
Given an image and its N faces being detected, our al-
gorithm is aimed to infer the appropriate clothing shape for
each person considering the occlusion between neighbors.
The whole algorithm is illustrated in Fig. 2. Let xi de-
note the features extracted for each person, including face
location and size, superpixel image [6] and RGB features in
clothing region. Suppose M candidate clothing shapes are
generated for each person (as described in Sec 3, these can-
didate shapes are generated by Random Forest), and then
shape label yi ∈ {1, · · · , M } represents the clothing shape
selected for each person.
The person pairs possibly occluded by each other are
denoted as the edge set E. For any pair of persons, bi-
nary blocking indicator bij denotes the blocking relation-
ship. When person i blocks person j, bij = 1; otherwise,
bij = 0. And bij = 1 − bji .
Let x = {x1 , · · · , xN }, y = {y1 , · · · , yN } and b =
{bij | (i, j) ∈ E}. The multi-person clothing model max-
imizes the conditional probability P (y, b|x) for clothing
shape y and blocking relationship b. The probability can
be factored based on Bayes’ theorem:
P (y, b|x) = P (y|x, b) P (b|x)
P (y|x, b) is the probability of each clothing layout y
given the blocking information b. We call it Layout Model.
P (b|x) is the probability of blocking relationships for cur-
rent image. We call it Blocking Model.
2.1. Layout Model
We assume that the person-graph is a Markov Network,
so the Layout Model can be factored as:
1 Y β Heuristically, we can say the person with a lower face
P (y|x, b) = ϕi (yi |xi ) u
Z i
βp
Y
ϕij (yi , yj |xi , xj , bij ) ij
(i,j)∈E
where Z is the partition function to make sure that
P (y|x, b) is a distribution. E denotes the graph structure,
which will be decided in runtime based on face locations
and sizes.
The unary potentials ϕi measure the consistency be-
tween the clothing shape and the actual edges. We segment
the image into superpixels [6]. The region boundaries are
used as the actual edges of the image. We perform a naive
inference in pixel and superpixel level respectively based
on the clothing shape. Suppose the inferred foregrounds
in pixel and superpixel level are Lpx and Lrg respectively,
then ϕi is defined as their consistency:
|Lpx ∩ Lrg |
ϕi =
|Lpx ∪ Lrg |
A uniformed unary parameter βu (Eq. 2) is assumed for all
unary potentials.
The pairwise potentials ϕij penalize the conflict fore-
ground part of the neighbor shapes and encourage larger
size of the unoccluded one. Denoting the overlapping re-
gion as A, ϕij is calculated as:
ϕij (yi , yj |bij ) = exp −ϕcij − ϕbij


where
X

ϕcij = yik yjk / |A|
k
and
yjk bij + k yik (1 − bij )
P P
k
ϕbij = P k P k (6)
k yi + k yj
p
βij is the weight parameter of pairwise potentials. In our
p
method, βij represent the confidence of the blocking rela-
p
tionship. So we set βij = P (bij ), where P (bij ) is the
blocking probability calculated as Eq. 10. yik is the binary
indicator of shape of pixel in A, the same to yjk ; |A| is the
(1) size of the overlapping region. In Eq. 5, the numerator ac-
cumulates the number of pixels where occluded and unoc-
cluded shapes are foreground simultaneously. This conflict
of foreground brings the ambiguity for bottom-up segmen-
tation. In Eq. 6, we encourage a bigger foreground size
of the unblocked person in the overlapping region by cal-
culating their foreground size proportions. For both of the
potentials, smaller values are preferred.
2.2. Blocking Model
location is more likely to block that with a higher one in a
(2) group image. However, estimating the occlusion relation-
ship by relative face location is not as reliable as human
body locations [22]. For example, the 0-th and 2-th face
locations in Fig. 1 are lower than their neighbors, but they
actually stand behind them. It is not a rare case, since in
group images, people prefer to stand in rows. A helpful ob-
servation is that unoccluded clothing usually is symmetric,
or contains more repetitive patterns. The two cues are used
in our model to estimate the blocking probability.
We assume that the blocking model for all pairs of neigh-
bors are independent to each other. so P (b|x) can be fac-
tored as:
Y
P (b|x) = P (bij |xi , xj ) (7)
(i,j)∈E
The blocking probability between pairs of persons takes the
(3) form of random forests as shown in Fig. 3.
3. Model Learning
We learn the model using a supervised algorithm based
on a manually labeled data set. Different persons are la-
beled respectively. The blocking relationship is generated
from the segmentation labeling results. For persons nearby
whose clothing regions (estimated by face location and size)
(4) overlap the other, we accumulate the foreground size of
them in the overlapping region and determine the blocking
relationship bgij and confidence P bgij based on the fore-
(5)
ground proportion:
 1
θ 
d −
2 H̃ = arg min d c+
H H1
H2
Weak Features
Decision Trees
(a) (b)
Figure 3. (a) Weak features and Decision Tree for clothing shape
sampling (b) Weak features and Decision Tree for blocking distri-
bution modeling.
X X
bgij = yik > yjk
k k
P k
P k

max k yi , y
P bgij P kkj


= P k with the one used in Sec. 3.1. For neighbors, H1 and H2
k yi + k yj
3.1. Clothing Shape
As stated in Sec. 2, M clothing shapes should be sam-
pled for each individual person. We learned the model using
Random Forest, since it is very efficient and is straightfor-
ward to implement [21]. Weak feature H is designed based
on the self-similarity properties of clothing (Fig. 3(a)).
Specifically, rectangle patches are densely sampled in the
clothing regions. The weak feature is defined as the his-
togram similarity (X 2 distance) between any pair of the lo-
cal patches. Since histogram similarity is continuous, the
training data in each non-terminal is partitioned by compar-
ing the current weak feature with predefined thresholds.
Each decision tree is a binary tree (illustrated in Fig.
3(a)), where terminals (solid nodes in Fig. 3) record the
clothing shape fallen in it. In contrary to a conventional
decision tree, the information gain is not available for a
high-dimensional clothing shape. Alternatively, we use the
hamming distance between shapes to calculate the decision
tree metrics. The hamming distance P between any two bi-
nary shapes is defined as d (ci , cj ) = m cm m
i 6= cj , where
ci and cj are the binary shape images (all clothing shape
are normalized with the same size based on face size). Let
c = {c1 , · · · , cn } be the shape set, then the distance (dis-
similarity) is defined on the set d (c) = maxi,j d (ci , cj ).
The feature in each non-terminal is selected as the one
whose partition is the most consistent, i.e.:
  
H,τ + d c H,τ (9)
H,τ
−
where c+ H,τ and cH,τ are the subsets produced by comparing
the response of H with threshold τ . The learning process
is halted when the remaining clothing shapes are consistent
enough (d (c) < 0.1) or there are no available weak fea-
tures.
A set of decision trees is learned using a random subset
[4] of the weak features for each node. Each tree is trained
on a different bootstrapped dataset. Since the features used
in the Random Forest imply the self-similarity of clothing,
the output clothing shapes can address the occlusion mod-
erately (referring to Fig. 6).
3.2. Blocking Distribution
We learn the blocking distribution as Random Forest too.
Since it is basically a binary classification problem, we use
traditional decision tree as its elemental model.
For each decision tree, a blocking distribution between
neighbors is learned based on relative weak features, as
shown in Fig. 3(b). Here we set the right-then-upper per-
(8) son as the original one. The first kind of features is related
are calculated for each one respectively, and then the simi-
larity difference H1 − H2 is used as the weak feature. The
second kind of features is the face angle θ and face dis-
tance d. Still, in the non-terminal nodes, these continuous
features are used by comparing with predefined thresholds
and the one with the maximum information gain is selected.
The terminal nodes maintain the probability that person A
blocks person B using non-parametric models, i.e. we count
the blocking and unblocking frequency from training sam-
ples fallen into them.
A set of decision trees is learned using a random sub-
set [4] of the weak features for each node, and each tree is
trained on a different bootstrapped dataset. The final block-
ing distribution is obtained by averaging outputs of all trees:
Nbt
1 X
P (bij |xi , xj ) = Pk (bij |xi , xj ) (10)
Nbt
k=1
where Nbt is the number of blocking decision tree and
Pk (bij |xi , xj ) is the blocking distribution obtained by a
single decision tree.
4. Model Inference
Given an image and its face detection results, we are aim-
ing at inferring the clothing layout and also the blocking
relationship for group people.
 1 First, as suggested by Kumar et al. [15], when we use the
accuracy
confidence
0.9
0.8
0.7
0.6
0.5
0.1 0.2 0.3 0.4 0.5
Shape Proportion Representative Images
Figure 4. To get a detailed view of the performance of our blocking
model, we partition the testing samples into different groups based
on the proportion of foreground sizes of neighbors in the overlap-
ping region ( min (A, B) / (A + B), referring to Eq. 6). The
smaller the proportion is, the more likely they block each other.
Based on the plot, we find that our algorithm performs well for
severe blocking cases (88%). For the inaccurate proportion, we
observe that the blocking relationship is not so obvious (see the
representative images in the figure), so we depress the inaccurate
effect by incorporating the confidence into the pairwise potential
functions ( in Eq. 4), since it (red curve) is highly correlated with
the accuracy (blue curve).
4.1. Data Preparing
Based on face locations, we estimate the possible cloth-
ing regions as rectangles. The edges E for Layout Model are
determined for the neighbor rectangles with large enough
overlapping regions (10% of the minimum rectangle).
For each person, we obtain the candidate clothing shapes
using our clothing shape random forest learned in Sec. 3.1.
For each pair of neighbors, we obtain the blocking distribu-
tion using Eq. 10.
4.2. Blocking Relationship
The optimal blocking relationship b̃ can be obtained by
maximizing the individual local blocking distribution, i.e.
Y between neighbors are downloaded from Flickr by different
b̃ = arg max P (bij |xi , xj ) (11)
bij
(i,j)∈E
Yang et al. [22] search the optimal ordering from all
possible permutations. In our method, we just use the MAP
solution, since we observe that the unsure blocking relation-
ship usually occurs when neighbors stand close but do not
block each other (as shown in Fig. 4). It means that “who
blocks who” is not critical in these cases. We incorporate
the confidence P (bij |xi , xj ) to Layout Model in Eq. 4 by
the pairwise weight, which decreases the effect of neighbor
constraints when the relationship is not so confident.
4.3. Sampling Layout
Given blocking configuration , we can approximate the
MAP solution of Eq. 2 by Loopy Belief Propagation. How-
ever, in our method, we propose to sample the layout instead
of a single MAP solution. The two reasons are as follows:
top-down model as latent variables to guide the bottom-up
image segmentation, the expectation of the log likelihood of
an MRF can be efficiently optimized by a single graph cut
optimization. Importantly, they argue that multiple samples
are necessary for some difficult cases in which RGB distri-
bution of background is similar to that of the object.
The other is from an insight of our model. In Sec 3.1,
a clothing shape random forest is learned for each person
individually. Random forest increases its generalization ac-
curacy nearly monotonic with respect to the number of de-
cision tree [4]. The final voting procedure of outputs will
make the model more robust to noise. However, the MAP
inference selecting only one result from all tree outputs will
decrease the generalization ability. From this point of view,
sampling from the Layout Model, where interaction of the
neighbors is taken into account, is a novel and effective vot-
ing method to improve the individual random forest perfor-
mance.
For convenience of sampling, we use a sum-product al-
gorithm for MAP inference [14] and then sample multi-
solutions around the local (global for tree-graph) minimum
by Gibbs sampling. Here we approximate the posterior dis-
tribution of layout solution as [15]:
Q
ij∈E Bij (yi , yj )
p (y) = Q qi −1 (12)
i Bi (yi )
where Bi and Bij are the unary and pairwise beliefs cal-
culated by the sum-product algorithm. qi is the neighbor
number of node i.
5. Experiments
We evaluate our method on a public available data set
of group images [8]. All the images with high occlusions
key words: wedding, family and group images. We manu-
ally label 281 images with 1051 persons. We partition the
ground truth dataset into two halves randomly and train our
model using one half and test it with the other. In our ex-
periments, clothing is labeled as any covering of the torso,
including naked shoulder and arms.
Implementation: We detect face location and scale us-
ing face detector [12] and normalize the images to make
each face 24 × 24. β u , the weight of unary potentials in Eq.
2 is set as 0.4 empirically by cross-validation.
Random Forests for clothing shape and blocking rela-
tionship consist of 50 decision
p trees. For each node of
them, we randomly select Nf [2] weak clothing self-
similarity features (relative face location features are used
for all nodes), where Nf is the weak feature number (around
300,000). The size of rectangles used in clothing self-
similarity features is 4 × 4, and they are densely sampled

Algorithm 1 Layout optimization for group images
Input: Image data and face detection results
Output: Probabilistic clothing mask
[Data Preparing]
1: Determine the inter-person relationship based on face
locations and construct the graph;
2: Generate shape candidates {yi } for each person based
on clothing model in Sec. 3.1;
[Model Inference]
1: Estimate the blocking relationship {bij } and its confi-
dence {P (bij )};
2: Optimize Eq. 2 by Loopy Sum-Product Belief Propaga-
tion to get the MAP inference result, the unary beliefs
{Bi } and pairwise beliefs {Bij };
3: Generate multiple solutions around the MAP one by
Gibbs sampling (Eq. 12);
4: Average the multiple clothing shapes for each person to
generate the probabilistic clothing mask;
with step 2. We calculate the upper and lower bound of fea-
ture values for each node and partition the interval into 10
pieces and then select the optimal threshold from them. No
pruning is made for decision trees.
A sketch of our layout optimization algorithm is shown
in Algorithm 1. Only original image data and face detection
results are required. The algorithm outputs a probabilistic
mask for each person providing top-down guide for further
pixelwise segmentation.
To benefit the bottom-up segmentation, we incorporate
the top-down cues as a unary term in a CRF based frame-
work [20]. Given the clothing shapes for each person by
sampling Eq. 12, we obtain the multi-person clothing mask
by a Bayesian fusion as:
M
Y blocking relationship between persons, while face informa-
p (li = 0) = p (li = 0|ym ) (13)
m=1
p (li = 1|ym )
p (li = m) = P · (1 − p (li = 0)) (14)
k p (li = 1|yk )
where li ∈ {0, 1, · · · , M } is the label of pixel i, in which
0 is the background and {1, · · · , M } are different persons.
Intuitively, li = 0 if and only if in each person’s clothing
mask, pixel i is background.
Clothing and background color models (histograms in
RGB space) are estimated based on the clothing masks.
Shape and color cues are integrated with superpixel-level
constraints [6]. The final segmentation result is obtained by
optimizing a Dual-Level Conditional Random Field [20].
Blocking Modeling: The blocking performance is eval-
uated for the ones whose confidence P bgij > 0.7, since


103
0.9 0.8
Full Model
0.8
Blocking accuracy
0.6 Face Infor
Forest Size
Cloth Infor
0.7
0.4
0.6 Full Model
Face Infor
0.5
0.2
Cloth Infor
0.4 0
1 5 9 13 17 21 25 29 1 2 3 4 5 6 7 8
tree number tree number
Figure 5. Detailed comparison of blocking model. Left: blocking
accuracy with respect to the number of trees. Right: forest size
with respect to the number of trees. The performance is evaluated
using only face information, only clothing information and both of
them (full model).
the blocking relationship is not so critical for the others, as
show in Fig. 4.
Experiments are performed with different configura-
tions: only face information, only clothing information and
both of them (our full model). Based on the curve in Fig.
5, we find that face information achieves baseline accuracy,
since it portrays the basic structure of group people. How-
ever, it cannot handle the occlusion among several people
abreast, which limits its capability for blocking relation-
ship prediction. Moreover, its performance will not increase
with respect to the number of trees, since actually there are
not so many available features for our random selection.
Both clothing information and our full model can
achieve better accuracies with a large enough amount of
trees. However, the experiments show that to achieve the
same performance, the full model needs fewer trees. In ad-
dition, as shown in the right curves of Fig. 5, the size of
random forest based on both kinds of information is much
smaller than that based on only clothing information. So the
conclusion is that the clothing information can capture the
tion can enhance it by reducing the required number of trees
and nodes. As a comparison, we implement the method
in [7]. However, its MAP result only achieves 56.2% accu-
racy. The reason may be that in our dataset, people stand
closer, which reduces the power of Face Relative Size and
moreover no self-similarity features are explored.
Layout Modeling: We investigate the performance of
Layout Model by segmentation accuracy based on the man-
ually labeled dataset.
Tab. 1 reports pixelwise occlusion error (the proportion
of pixels assigned to wrong persons) and segmentation ac-
curacy. We apply three different methods to segment the
clothing, which gives a detailed view of the performance
of our model. As a baseline method, we average over all
clothing shapes to get a global mask, and just scale the prob-
abilistic mask for each person with respect to the face size.
In this case, no blocking information is used. The perfor-
 Occlusion Error Pixel Accuracy
Single Mask 6.8% 88.2%
CF 5.9% 90.7%
CF + BF 4.6% 92.8%

Table 1. Segmentation performance comparison. Single mask: av-

eraging over all clothing shapes, which uses none occlusion cues.
CF: only use clothing shape forest, which addresses blocking in-
formation implicitly. CF + BF: Clothing Forest + Blocking Forest
(integrated by Layout Model), which addresses blocking relation-
ship explicitly.

mance can be improved by using our Clothing Forest model,

which is constructed based on clothing self-similarity fea-
tures. As stated before, unoccluded clothing usually con-
tains more repeatable patterns, so in fact Clothing Forest in-
corporates blocking information implicitly. Occlusion be-
tween persons can be handled moderately. An example is
shown in Fig. 6, which is marked with an orange single-
boundary rectangle. However, when assigning a putative
clothing shape to each person, Clothing Forest still works
in an independent way. Our Layout Model enhances this
procedure by optimizing the layout in a global view of the
entire people group. Blocking information is captured by
Blocking Forest and embedded as pairwise constraints.
86.5% 88.0% 90.3%
Figure 7. Comparison experiment with GrabCut. The initial rect-
angle (red one) for GrabCut is generated by our algorithm.
racy can produce remarkable qualitative segmentation im-
provement, which is also stated in [19][13]. We also com-
pare with GrabCut [17] (implemented by OpenCV [2]) in
Fig. 7. The initial rectangle is generated based on global
clothing template. Without blocking information, GrabCut
might fail for blocked people.
We show additional segmentation results from the data
set [8] in Fig. 8. In addition, as stated before, we parti-
tion the full layer sorting problem into pairwise ones, so
our algorithm is capable of dealing with cases with a large
number of persons. Fig. 8 also shows such examples. Our
algorithm produces nice segmentation results even in such
a crowded scene. Note that inaccurate segmentations occur
in the arm parts, since the color of the skin is different from
that of clothing while we do not filter that out especially.

88.5% 91.0% 93.2%
(a) (b) (c) (d)
Figure 6. Comparative segmentation results. (a) input image and
the face detections (b) results using single clothing mask (c) results
using clothing shape random forest directly (d) results by incorpo-
rating the blocking relationship.
Some of the comparison segmentation results produced
by the three methods are shown in Fig. 6. With the help
of blocking information, our segmentation algorithm can
produce much better object-level boundaries even when the
clothing of different persons is very similar (the occlusion
parts are marked by double-boundary rectangles). Also, we
report the pixelwise segmentation accuracy under each seg-
mentation result. We can see that although the quantitative
improvement in reported accuracy seems to be marginal, for
segmentation problem, a small increment in pixelwise accu-
6. Conclusion
In this paper, we have proposed a novel approach for
multi-person clothing segmentation problem. Blocking
information is incorporated in a principle way to bene-
fit a clothing shape sampling procedure. Experiments on
severely crowded group images with arbitrary number of
people show that our approach outperforms conventional
method and yield expressive segmentation results.
Currently, the model assumes that there are not serious
pose variations. Thus, there is clearly room for improve-
ment, particularly in combining with human pose estima-
tion (e.g. multiple-person pose estimation [7], simultaneous
pose estimation and human segmentation [3][1]).
7. Acknowledgements
This work is supported in part by National Sci-
ence Foundation of China under grant No.61075026, Na-
tional Basic Research Program of China under Grant
No.2011CB302203, and it is also supported by a grant from
HP Corporation.
Figure 8. Some additional segmentations. For each pair of images, the first one is the input image with detected face, and the other is the
segmentation result produced by our algorithm. The last two rows show expressive segmentation results for crowded scene.
References
[1] L. Bourdev, S. Maji, T. Brox, and J. Malik. Detecting peo-
ple using mutually consistent poselet activations. In Proc.
ECCV, 2010.
[2] G. Bradski. The opencv library. Dr. Dobb’s Journal of Soft-
ware Tools, 2000.
[3] M. Bray, P. Kohli, and P. H. S. Torr. Posecut: Simultane-
ous segmentation and 3d pose estimation of humans using
dynamic graph-cuts. In Proc. ECCV, 2006.
[4] L. Breiman. Random forests. Machine Learning, 2001.
[5] H. Chen, Z. J. Xu, Z. Q. Liu, and S. C. Zhu. Composite
templates for cloth modeling and sketching. In Proc. CVPR,
2006.
[6] Y. Deng and B. S. Manjunath. Unsupervised segmentation
of color-texture regions in images and videos. PAMI, 2001.
[7] M. Eichner and V. Ferrari. We are family: Joint pose estima-
tion of multiple persons. In Proc. ECCV, 2010.
[8] A. Gallagher and T. Chen. Understanding groups of images
of people. In Proc. CVPR, 2009.
[9] A. C. Gallagher and T. Chen. Clothing cosegmentation for
recognizing people. In Proc. CVPR, 2008.
[10] D. Hoiem, C. Rother, and J. Winn. 3d layoutcrf for multi-
view object class recognition and segmentation. In Proc.
CVPR, 2007.
[11] Z. Hu, H. Yan, and X. Lin. Clothing segmentation using fore-
ground and background estimation based on the constrained
delaunay triangulation. PR, 2007.
View publication stats
[12] C. Huang, H. Ai, Y. Li, and S. Lao. High-performance rota-
tion invariant multiview face detection. PAMI, 2007.
[13] P. Kohli, L. Ladicky, and P. H. S. Torr. Robust higher order
potentials for enforcing label consistenty. In Proc. CVPR,
2008.
[14] D. Koller and N. Friedman. Probabilistic Graphical Models:
Principles and Techniques. MIT Press, 2009.
[15] M. P. Kumar, P. H. S. Torr, and A. Zisserman. Obj cut:
Efficient segmentation using top-down and bottom-up cues.
PAMI, 2010.
[16] M. W. Lee and I. Cohen. A model-based approach for esti-
mating human 3d poses in static images. PAMI, 2006.
[17] C. Rother, V. Kolmogorov, and A. Blake. “grabcut” ł inter-
active foreground extraction using iterated graph cuts. Sig-
graph, 2004.
[18] J. Sivic, C. L. Zitnick, and R. Szeliski. Finding people in
repeated shots of the same scene. In Proc. BMVC, 2006.
[19] M. Szummer, P. Kohli, and D. Hoiem. Learning crfs using
graph cuts. In Proc. ECCV, 2008.
[20] N. Wang and H. Ai. A compositional exemplar-based model
for hair segmentation. In Proc. ACCV, 2010.
[21] J. Winn and J. Shotton. The layout consistent random field
for recognizing and segmenting partially occluded objects.
In Proc. CVPR, 2006.
[22] Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes. Lay-
ered object detection for multi-class segmentation. In Proc.
CVPR, 2010.