Multiple Human Object Tracking using Background

Subtraction and Shadow Removal Techniques

S. Saravanakumar#1, A.Vadivel#2, C.G Saneem Ahmed*3
#
Department of Computer Applications, National Institute of Technology
*Department of Electronics and Communication, National Institute of Technology
Tiruchirappalli, Tamilnadu, India.
1
somansk@yahoo.com
2
vadi@nitt.edu
3
saneem89@gmail.com

Abstract ² The main objective of this paper is to develop
multiple human object tracking approach based on motion
estimation and detection, background subtraction, shadow
removal and occlusion detection. A reference frame is initially
used and considered as background information. While a new
object enters into the frame, the foreground information and
background information are identified using the reference frame
as background model. Most of the times, the shadow of the
background information is merged with the foreground object
and makes the tracking process a complex one. In the approach,
morphological operations are used for identifying and removed
the shadow. The occlusion is one of the most common events in
object tracking and object centroid of each object is used for
detecting the occlusion and identifying each object separately.
Video sequences have been captured in the laboratory and tested
with the proposed algorithm. The algorithm works efficiently in
the event of occlusion in the video sequences.
Keywords²Background modeling and subtraction, human
motion detection, shadow removal and occlusion detection.
I. INTRODUCTION
In Computer Vision, object tracking is considered as one of
the most important task. Various methods have been proposed
and reported both in academia and industry for large number
of real-time applications. The object tracking methods may
broadly be categorized as template-based, probabilistic and
pixel-wise. While the template-based method represents the
object in a suitable way for tracking, the probabilistic method
uses intelligent searching strategy for tracking the target
object. Similarly, the similarity matching techniques are used
for tracking the target object in pixel-based methods.
However, among all the above said approaches, the template-
based approach is found to be suitable for many real-time
applications [1, 2]. In this category of tracking methods,
similarity of the predefined target is being calculated with the
object translation. However, for object transformations such
as translation, rotation and scaling this method often fails.
This is due to the fact that the procedures of selection of target
object as constant size templates. For handling this difficult
issue, varying templates are used. The inclusion of
background pixels into the template introduces the problem of
positioning error and the positioning error continuously
getting added while updating the template.
In template-based approach category, mean-shift method
[3] and Kernel-based tracking method [4] have been proposed,
where the color histograms of the target object is constructed
using a Kernel density estimation function. Since, the color
histogram is invariant feature for rotation, scaling and
translation, it is considered as one of the suitable feature for
handling the problem of change in the scale, rotation and
translation of target object. The object tracking is carried out
by comparing the color histogram of the template and the
target object. However, mean-shift method is not suitable for
3-D target object and monochromatic object. In case of
monochromatic target object, even small variation in
illumination, produces narrow histogram pattern and tracking
often fails.
In object tracking problem, the object representation is the
difficult aspect. Various ways of representing or describing
target objects have been proposed such as object appearance
[1, 2], image features [5, 6], target contour [7, 8] and color
histogram [4]. In both appearance-based and color histogram
based approaches, the region of the object has to be defined
for describing the target. Thus, if some of the background
pixels are mixed with the defined region, the tracking may
fail.
While tracking non-rigid objects, the probabilistic based
tracking methods have given better performance. Some of the
approach in this category can be found in [13, 14, 15, 19]. In
one of the probabilistic method [13], the factors such as
motion detector, region tracker, head detector and active shape
tracker have been combined for tracking the pedestrian. The
assumption made in this method is that there are no people
moving in the background. Since, this method uses contour as
one of the feature, initial contour definition is difficult for the
complicated contour target object.
Object tracking is also performed by predicting the object
position from the past information and the predicted current
position. These types of methods combine both statistical
computation and the parameter vector [11, 12, 16, 17].
However, for real-time object tracking systems, it is found to
be difficult for constructing the proper feature vectors. This
method has been extended by Khan, et al. [11], for dealing
with the problem of interacting targets. The Markov Random

978-1-4244-8594-9/10/$26.00 2010
c IEEE 79
Field (MRF) has been used for modelling the interactions.
This has been achieved by adding an interaction weighted
factor. However, in this method the tracking fails while there
is an overlap between targets.
In contrast to model-based tracking methods, the pixel-wise
tracking methods are data-driven methods. In pixel-wise
tracking method, prior model of the target is not required. A
parallel K-means clustering algorithm [18] has been used by
Heisele, et al. [9, 10] for segmenting the color image sequence
and moving region is identified as target. However, the
method is computationally expensive due to large number of
clusters. Similarly, another K-means based autoregressive
model has been proposed and the clustering is performed only
WR WKH SRVLWLYH VDPSOHV 7KXV WKH WUDFNLQJ IDLOXUH FDQ¶W EH
detected and the failure recovery may not be possible. For
tracking, the image pixels are divided as target and non-target
pixels and K-means clustering algorithm is applied on these
pixels [20@ +RZHYHU WKLV PHWKRG FDQ¶W GHDO ZLWK WKH
appearance changes of the target object such as size, pose, etc.
In addition, the computational cost is proportional to the
number of non-target points.
It is understood from the above discussion that pixel-based
methods are robust against the background interfusion
methods. In this kind of method, the failure detection and
automatic failure recovery can be carried out effectively.
A very fundamental and critical task in computer vision is
the detection and tracking of moving objects in video
sequences. Possible applications are as follows (i) Visual
surveillance: A human action recognition system process
image sequences captured by video cameras monitoring
sensitive areas such as bank, departmental stores, parking lots
and country border to determine whether one or more humans
engaged are suspicious or under criminal activity. (ii) Content
based video retrieval: A human behavior understanding
system scan an input video, and an action or event specified in
high-level language as output. This application will be very
much useful for sportscasters to retrieve quickly important
events in particular games. (iii) Precise analysis of athletic
performance: Video analysis of athlete action is becoming an
important tool for sports training, since it has no intervention
to the athletic.
In all these applications fixed cameras are used with respect
to static background (e.g. stationary surveillance camera) and a
common approach of background subtraction is used to obtain
an initial estimate of moving objects. First perform background
modeling to yield reference model. This reference model is
used in background subtraction in which each video sequence
is compared against the reference model to determine possible
variation. The variations between current video frames to that
of the reference frame in terms of pixels signify existence of
moving objects. The variation which also represents the
foreground pixels are further processed for object localization
and tracking. Ideally, background subtraction should detect
real moving objects with high accuracy and limiting false
negatives (not detected) as much as possible. At the same time,
it should extract pixels of moving objects with maximum
possible pixels, avoiding shadows, static objects and noise.
80 2010 International Conference on Signal and Image Processing
In the detection of shadows the foreground objects are very
common, producing undesirable consequences. For example,
shadows connect different people walking in a group,
generating a single object (typically called blob) as output of
background subtraction. In such case, it is more difficult to
isolate and track each person in the group. There are several
techniques for shadow detection in video sequences [21].
The main objective of this paper is to develop an algorithm
that can detect human motion at certain distance for object
tracking applications. We carry out various tasks such as
motion detection, background modeling and subtraction,
foreground detection, shadow detection and removal,
morphological operations and identifying occlusion.
The paper organized as follows. Section 2.0 the object
segmentation of the video frames from the HSV color space is
presented. The proposed method is explained in Section 3.0.
In section 4, we present the experimental results and we
conclude the paper in the last section of the paper.
II. BACKGROUND SUBTRACTION
Human motion analysis and detection are the foremost task
in computer vision based problems. Human detection aims at
segmenting regions corresponding to people from the entire
image. It is a significant issue in human motion analysis
system since the subsequent processes such as tracking and
action recognition follows the motion detection. The motion
detection and foreground object extraction algorithm consists
of several sequential processes. The process algorithm is
described in a flow chart and shown in Fig.1.
In general, the Sum of Absolute Difference (SAD)
algorithm is used for background modelling, which is based
on the frame differencing techniques. It is mathematically
represented as
1
D (t ) I (t i ) I (t j ) (1)
N
Where, N is the number of pixels in the frame and also
I (t i ) and I (t j ) are the frames at time
used as scaling factor,
i and j respectively. D(i ) is the normalized sum of
absolute difference for that time instance. In an ideal case,
when there is no motion, the following condition holds good.
I (t i ) I (t j ) and D(t ) 0 (2)
A. Background Subtraction
Background subtraction is a popular technique to segment
out the interested objects in a frame. This technique involves
subtracting an image that contains the object, with the
previous background image that has no foreground objects of
interest. The area of the image plane where there is a
significant difference within these images indicates the pixel
location of the moving objects [22]. These objects, which are
represented by groups of pixel, are then separated from the
background image by using threshold technique.
 where I is the current pixel intensity value, B is the
Sequence of video background intensity value and T is the foreground threshold.
frames

No Motion
detected?
(a) (b)
End
Yes

Background modelling

Background subtraction/
Foreground object (c) (d)
extraction Fig.2. Background subtraction and moving object identification (a) Video
Frame, (b) Background, (c) Background Subtracked Image and (d) B/W
frames Showing Objects

Shadow detection and
removal
Morphology process
Occlusions detection
Fig.2 shows the video frames used for the background
subtraction and moving object identification. In Fig. 2(b), the
background frame is shown and that has been used for
constructing the background model. The foreground
information and background information are identified and
finally subtracted for identifying the objects present in the
foreground frame, which is shown in Fig. 2(c). In Fig. 2(d),
the final outputs are shown where the objects present in the
frame is converted as black and white pixels for effective
idenfication of the object.

B. Shadow Detection and Removal

Draw bounding box and
human object tracking
Fig.1. Flow chart for the human object tracking
The mode model was chosen to perform the background
modeling, which provides better results. If the absolute
difference between the current pixel and the mode modelled
background pixel is larger than a threshold, then that pixel is
considered as foreground object [23, 24]. RGB values of
current frames pixels subtracted with that of background
modeling frame. The mean of absolute difference of red value,
green value and blue value are found. If the absolute
difference greater than threshold, indicates the foreground
pixels else background pixels. Foreground pixels are detected
by calculating the Euclidean norm,
1, if I ( x, y ) B( x, y ) T
D ( x, y ) (3)
0, otherwise
Once the foreground object identified, each foreground
pixels are checked whether they are part of a shadow or the
object. This process is necessary, since, shadow of the some
of the background object may get combined with the
foreground object. This causes the object tracking task as a
complicated task. For pixel ( x, y ) in a shadowed region, the
Normalized Cross Covariance (NCC) in a neighboring region
B ( x, y ) is found and the shadow can be detected using
equation given below,
NCC ( x, y) Lnccc (4)
Where L ncc is fixed threshold. If L ncc is low, several
foreground pixels corresponding to moving objects may be
misclassified as shadows. On the other hand, selecting a larger
value for L ncc results in less false positives, but pixels related
to actual shadows may not be detected [25].
NCC for a pixel at position (i , j ) is given by

2010 International Conference on Signal and Image Processing 81

 ER (i, j ) shows the entry of new objects into the frame or separation of
NCC (i, j ) (5) occluded objects. Consequently if there is a sudden reduction
E B (i , j ) ETij
of number of objects present in the frame indicates the process
where of occlusion of two or more objects or the exit of the objects
N N
from the frame to the outside area. This situation is
ER(i, j ) B(i n, j m)Tij (n, m) (6) experimented and is depicted in Fig. 4(a)-(d).
n Nm N
N N
E B (i, j ) B(i n, j m) 2 (7)
n Nm N
and
N N
ETij Tij (n, m) 2 (8)
n Nm N

B (i, j ) is the background image and Tij (n, m) is the (a) (b)
template on current image.

(c) (d)
(a) (b) Fig.4. Occlusion detection and background subtraction (a) Video Frame, (b)
Background Subtraction, (c) BW Image Showing Objects and (d) Occlusion
Detection

As Fig.4(c), shows the two objects whitch are being combined

and treated as single object.

III. EXPERIMENTAL RESULTS

The experimental results are presented to show that the

(c)
Fig.3. Background subtraction and shadow removal (a)Video Frame, (b)
Background Subtraction, (c) BW Image Showing Objects and (d)Shadow
Detection
Fig.3 shows video frames with shadows and the
identification of the same. The foreground and background
objects are identified and are shown in Fig. 3(b) and (c). In
Fig. 3(c), Eqs.5-8, are applied and shadows are identified. The
results are shown in Fig 3(d). In this approach, two dilation
and one erosion operation processes were carried out. For
objects having area less than 0.5% of total image area has
removed.
C. Occlusion Detection
While two moving objects coming closer to each other, the
background subtracted frame shows it as a single object. This
situation is called as occlusion and will create problem while
tracking two objects. In this approach, an algorithm is
proposed for detecting the occlusion. This approach will
inform the frame number where the occlusion has taken place.
In the number of object in the frame is increased suddenly
(d) proposed methods can achieve promising performance in
background subtraction and foreground object extraction. This
system detects and tracks the moving objects exactly. In this
approach, the background scene is modeled using a set of
background image frames, which basically consists of 5-30
consecutive frames. The object pixels are segmented out from
its background followed by post- morphological process such
as dilation and erosion to eliminate noisy pixels thus
producing better results.
Fig. 5(a)-(f), shows the object tracking process. This video
is a homemade video in the Laboratory and multiple objects
were tracked. In Fig. 5(a) the original Video frame is shown.
The background frame is presented in Fig. 5(d), which will be
used as background and latter has been used for background
modeling. Using 5(a) and (b), the foreground and background
information are extracted and subtracted for getting the target
object, which is shown in Fig. 5(c). The tracking process of
the target objects are shown in Fig. 5(d)-(f). It is observed that
the target objects are bound by a rectangular boundary and the
proposed work detects the object effectively.

82 2010 International Conference on Signal and Image Processing

 (a) (b) (c)

(d) (e) (f)

Fig.5.Object tracking process (a) Video Sequences, (b) Background Model, (c) Background Subtraction, (d) Multiple Object
Tracking with Bounding Box, (e) & (f) Occlusion Detection

objects are not tracked due to the static behavior of

A. Comparative results
Kalman filter approach [26], algorithm is used for
detecting and tracking the human objects in the video
sequences. This algorithm is compared with the
proposed approach for performance evaluation. Fig. 6(a)
shows the tracking strategy of Kalman filter algorithm
and (b) shows the tracking strategy of the proposed
technique.
While comparing the performance of Kalman filters
and proposed approach, the following issues are
observed. (i) In most of the frames, the moving human
objects in the consecutive frames by Kalman filter
algorithm. In the proposed approach, the background is
PRGHOOHG IURP ILUVW IUDPH WR VRPH ³Q´ QXPEHU RI
frames. Thus, the background model helps the proposed
approach to track the static human objects without
failure.
(ii) The computation time for tracking the human
objects in the video sequences using proposed approach
is very low compared to Kalman filter algorithm.

(a)

(b)
Fig.6. Performance comparison D .DOPDQILOWHU¶VUHVXOWV E 3URSRVHG%DFNJRUXQGVXEWUDFWLRQDQGVKDGRZUHPRYDO¶VUHVXOWV

2010 International Conference on Signal and Image Processing 83


IV. CONCLUSION
In this paper, an approach capable of detecting motion
and extracting object information which involves human as
object has been described. The algorithm involves
modeling of the desired background as a reference model
for later used in background subtraction to produce
foreground pixels which is the deviation of the current
frame from the reference frame. The deviation which
represents the moving object within the analyzed frame is
further processed to localized and extract the information.
The occlusion has also been dealt effectively.
ACKNOWLEDGMENT
The work done by Dr. A.Vadivel is supported by the
research grant from the Department of Science and
Technology India, Under Grant SR/FTP/ETA ± 46/07
dated 25th October, 2007 and DST/TSG/ICT/2009/27.
REFERENCES
[1] CranH +' DQG 6WHHOH &0     ³7UDQVODWLRQ-tolerant Mask
0DWFKLQJ XVLQJ 1RQFRKHUHQW 5HIOHFWLYH 2SWLFV´ Pattern
Recognition, Vol.1, No.2, pp.129-136.
[2] *UDVVO & =LQVVHU 7 DQG 1LHPDQU +    ³,OOXPLQDWLRQ
Insensitive Template Matching with HyperSODQHV´ in Proc. 25th
Pattern Recognition Symposium (DAGM '03), vol. 2781 of Lecture
Notes in Computer Science, pp. 273-280, Springer-Verlag,
Magdeburg, Germany.
[3] &RPDQLFLX ' 5DPHVK 9 DQG 0HHU 3    ³5HDO-time
Tracking of Non-rigid Objects usinJ0HDQ6KLIW´IEEE Conference
on Computer Vision and Pattern Recognition (CVPR'00), Vol.2,
pp.142-149.
[4] &RPDQLFLX ' 5DPHVK 9 DQG 0HHU 3    ³.HUQHO-Based
2EMHFW 7UDFNLQJ´ ,((( 7UDQVDFWLRQV RQ 3DWWHUQ $QDO\VLV DQG
Machine Intelligence, vol.25, No. 5, pp.564-577.
[5] &ROOLQV5DQG/LX<  ³2QOLQH6HOHFWLRQRI'LVFULPLQDWLYH
7UDFNLQJ )HDWXUH´  ,((( 7UDQVDFWLRQV RQ 3DWWHUQ $QDO\VLV DQG
Machine Intelligence, vol.27, No. 10, pp.1631-1643.
[6] 1JX\HQ+7DQG6HPHXOGHUV$  ³7UDFNLQJDVpects of the
IRUHJURXQG DJDLQVW WKH EDFNJURXQG´  th European Conference on
Computer Vision (ECCV), Proceedings, vol. 2, pp.446-456.
[7] .DVV 0 :LWNLQ $ DQG 7HU]RSRXORV '    ³6QDNHV DFWLYH
FRQWRXU PRGHOV´  International Journal of Computer Vision, vol.1,
No. 4, pp.321-331.
[8] ,VDUG 0 DQG %ODNH $   ³&RQWRXU WUDFNLQJ E\ VWRFKDVWLF
SURSDJDWLRQ RI FRQGLWLRQDO GHQVLW\´  th European Conference on
Computer Vision, Proceedings, vol.1, pp.343-356.
[9] +HLVHOH%  ³0RWLRQ-based Object Detection and Tracking in
&RORU,PDJH6HTXHQFH´th Asian Conference on Computer Vision.
[10] +HLVHOH%.UHVVHO8DQG5LWWHU:  ³7UDFNLQJ1RQ-Rigid
0RYLQJ 2EMHFWV %DVHG RQ &RORU &OXVWHU )ORZ´ Conference on
Computer Vision and Pattern Recognition, Proceeding, pp.253-257.
[11] .KDQ = %DOFK 7 DQG 'HOODHUW )   ³$Q 0&0&-based
3DUWLFOH )LOWHU IRU 7UDFNLQJ 0XOWLSOH ,QWHUDFWLQJ 7DUJHWV´ 8th
European Conference on Computer Vision (ECCV), Proceedings,
vol.4, pp.279-290.
84 2010 International Conference on Signal and Image Processing
[12] 7DR =KDR 5DP 1HYDWLD   ³7UDcking Multiple Humans in
&URZGHG (QYLURQPHQW´  ,((( &RPSXWHU 6RFLHW\ &RQIHUHQFH RQ
Computer Vision and Pattern Recognition (CVPR'04) vol. 2, pp.406-
413.
[13] 6LHEHO17DQG0D\EDQN6  ³)XVLRQRI0XOWLSOH7UDFNLQJ
Algorithms for Robust People TrackinJ´ 7th European Conference
on Computer Vision (ECCV), Proceedings, vol. IV, pp.373-387.
[14] <LQJ :X *DQJ +XD 7LQJ <X   ³6ZLWFKLQJ 2EVHUYDWLRQ
0RGHOV IRU &RQWRXU 7UDFNLQJ LQ &OXWWHU´ ,((( &RPSXWHU 6RFLHW\
Conference on Computer Vision and Pattern Recognition (CVPR
'03), vol. 1, pp.295-302.
[15] /L 3 DQG =KDQJ 7   ³9LVXDO &RQWRXU 7UDFNLQJ %DVHG RQ
Particle Filter, ,PDJHDQG9LVLRQ&RPSXWLQJ´YROSS-123.
[16] 9HUPDDN73HUH]3*DQJQHW0DQG%ODNH$  ³7RZDUGV
Improved Observation Models for Visual Tracking: Selective
$GDSWDWLRQ´7th European Conference on Computer Vision (ECCV),
vol.1, pp.645-660.
[17] 6LGHQEODGK + DQG %ODFN 0-   ³/HDUQLQJ ,PDJH 6WDWLVWLFV
IRU %D\HVLDQ 7UDFNLQJ´ IEEE International Conference on
Computer Vision (ICCV), Vol.2, pp.709-716.
[18] +DUWLJDQ - DQG :RQJ0   ³$OJRULWKP $6 $ .-Means
&OXVWHULQJ $OJRULWKP´ Journal of the Royal Statistical Society.
Series C (Applied Statistics), vol. 28, No.1. pp. 100-108.
[19] <LOPD] $ /L ; DQG 6KDK 0   ³2Eject Contour Tracking
8VLQJ/HYHO6HWV´Asian Conference on Computer Vision (ACCV),
Korea.
[20] +XD & :X + &KHQ 4 DQG :DGD 7   ³.- means
7UDFNHU $ *HQHUDO $OJRULWKP IRU 7UDFNLQJ 3HRSOH´ -RXUQDO RI
Multimedia, Vol.1, No. 4. Pp.46-53.
[21] R.CucchiDUD &*UDQD 03LFFDUGL DQG $3UDWL ³'HWHFWLQJ PRYLQJ
REMHFWVJKRVWVDQGVKDGRZVLQYLGHRVWUHDPV´ IEEE Transactions
on Pattern Analysis and Machine Intelligence, 25(10):1337-1342,
October 2003.
[22] , +DULWDRJOX ' +DUZRRG DQG /6 'DYLV ³: 5HDO-time
VXUYHLOODQFH RI SHRSOH DQG WKHLU DFWLYLWLHV´ IEEE Trans. Pattern
Analysis and Machine Intelligence .,2000, pp. 809-830.
[23] *XLVKHQJ<LQ<DQER/LDQG-LQJ=KDQJ³7KH5HVHDUFKRI9LGHR
7UDFNLQJ6\VWHP%DVHGRQ9LUWXDO5HDOLW\´Internation Conference
on Internet Computing in Science and Engineering, 2008.
[24] 6ZDQWMH-RKQVHQDQG$VKOH\7HZV³5HDO-Time Object Tracking and
&ODVVLILFDWLRQ 8VLQJ D 6WDWLF &DPHUD´ Proceedings of the IEEE
ICAR, Workshop on People Detection and Tracking, Kobe, Japan,
May 2009.
[25] Julio Cezar Silveira Jacques Jr, Claudio Rosito Jung, and Soraia
5DXSS 0XVVH ³Background Subtraction and Shadow Detection in
*UD\VFDOH 9LGHR 6HTXHQFHV´ IEEE Proceeding of the XVIII
Brazilian Symposium on Computer Graphics and Image Processing
(SIBGRAPI¶  pp: 530-1834/05, 2005.
[26] Shiuh-Ku Weng, Chung Ming Kuo and Shu-.DQJ 7X ³ 9LGHR
object tracking using adaptive Kalman filter´ Journal of Visual
Communication and Image Representation, Volume 17 , Issue 6,
pp.1190-1208, 2006.