2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)
Object Tracking using Joint Histogram of Color
and Local Rhombus Pattern
Manisha Verma
Department of Mathematics
Indian Institute of Technology Roorkee
Uttarakhand, India
Email: manisha.verma.in@ieee.org
Abstract—Object tracking is a challenging real world prob-
lem for traffic, crime scenes, sports, etc. A feature extraction
method, named as the local rhombus pattern (LRP) is proposed
in this work, and it is different from the conventional local bi-
nary pattern as it extracts the local relationship of neighboring
pixels itself instead of local relationship with the center pixel.
The proposed method is combined with HSV (hue, saturation
and value) quantized histogram, and is applied to object
tracking using mean shift tracking algorithm. Experiments
are carried out for road traffic and sports video, using joint
histogram of LRP and HSV color space, and compared to
two state-of-art approaches. The experimental results show the
effectiveness of the proposed method over existing methods.
Index Terms—Local binary pattern; local rhombus pattern;
mean shift tracking; object tracking.
I. I NTRODUCTION
Object tracking is a crucial issue in the field of pattern
recognition and computer vision. It mainly finds applications
in the areas of vehicle navigation, traffic monitoring, face
tracking, etc. Object tracking has two major tasks, first is
feature extraction of the target object in the video sequence,
and second is tracking of the target object in the video
sequence, using features. The proposed work targets feature
extraction. Previously, numerous feature extraction methods
based on color and texture have been proposed. An extensive
survey on object tracking has been presented for different
categories in [1].
Object tracking in a moving camera for non-rigid objects
has been performed with mean shift tracking algorithm and
dissimilarity has been measured with a distance measure
derived from Bhattacharya coefficient [2]. For better ob-
ject tracking, shadow detection and suppression have been
carried out using the HSV color information of moving
objects [3]. A kernel based object tracking was employed
for non-rigid objects using histogram as a feature space [4].
Shape features were also utilized with HSV color histogram
using edge histogram in different directions and applied
to object tracking [5]. An interest point based tracking
algorithm was proposed in [6]. Texture recognition has been
applied in the temporal domain for a dynamic sequence
using local binary pattern in three orthonormal planes [7]. A
modified LBP illumination variation was proposed in [8] and
applied to detect moving objects in a video sequence. Takala
et al. used color histogram, color correlogram and local
binary pattern for color and texture features. Motion features
were extracted using trajectories and applied for object
tracking in indoor and outdoor videos [9]. Object tracking in
978-1-4799-8996-6/15/$31.00 2015
c IEEE
Balasubramanian Raman
Department of Computer Science and Engineering
Indian Institute of Technology Roorkee
Uttarakhand, India
Email: balaiitr@ieee.org
illumination, occlusion and object/camera motion conditions
has been proposed using local features [10]. A two layer
feature learning module has been proposed using neural
network and pre-learned features has been been adopted in
tracking mode in video sequence [11]. Joint color texture
histogram created by LBP and RGB color channel, is used
to extract feature, and mean shift algorithm is applied for
object tracking [12]. A novel method called, spatial extended
center symmetric local binary pattern was proposed for
background subtraction from the image sequence [13]. Local
maxima edge binary pattern (LMEBP) has been proposed,
and rotation invariant and uniform LMEBP has been applied
for object tracking using mean shift tracking algorithm [14].
Dash et al. proposed a method based on local binary patterns
and Ohta color features instead of RGB, and employed it for
object tracking [15]. Multiple object tracking in a long sports
video was proposed by Liu et al. using short-term activity
of each player in the game [16].
A brief survey related to the proposed feature extraction
method is also given in this work. Ojala et al. proposed local
binary patterns that extract the local information of each
pixel based on neighboring pixels [17]. It has been used for
many pattern recognition applications, e.g., face recognition
[18], facial expression recognition [19], object tracking [12],
[15], etc. Further, local binary patterns were converted into
uniform and rotation invariant local binary pattern [20]. LBP
considered all patterns, but in dominant LBP [21], only
dominant patterns were considered because other patterns
do not contribute much in the feature description. LBP
combined with magnitude pattern and global mean of image,
is known as completed local binary patterns [22]. Marko
et al. proposed center symmetric local binary pattern [23]
which considers only center symmetric pixels and extracts
the pattern based on their relationship. The direction based
local pattern that extracts local information based on pixels
in four directions was proposed in [24]. To overcome the
problem of noise sensitive in local binary pattern, a local
ternary pattern was proposed which creates ternary patterns
based on threshold interval [25]. After LBP and LTP, local
tetra pattern that considers horizontal and vertical pixels of
the center pixel and generates a pattern, was proposed [26].
The proposed feature descriptor is motivated by the con-
ventional local binary pattern. LBP is very strong feature
descriptor but it has a high dimension for joint histogram
purpose. The modified rotation invariant uniform patterns
[20] have less feature vector length, however it loses in-
77
 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)
formation with feature reduction. The proposed method,
named local rhombus pattern (LRP) is based on mutual
relationship among neighboring pixels rather than the center
pixel and neighboring pixel relationship, and originally it
has less feature vector length, and no further reduction is
required. Hence, it is not losing extra information in reduc-
tion of feature vector length. RGB color histogram works
uniformly for all three channels. To over come this issue,
HSV color space has been adopted as it differentiates color,
brightness and intensity components. Each channel represent
different information with different values. Joint histogram
of HSV color and local rhombus pattern is used for feature
extraction. The feature vector has been employed for object
tracking experiment on two different video sequences.
This paper is systemized as follows: Section 1 presents a
brief introduction of the problem and summarize the related
work. Section 2 describes the local binary pattern and local
rhombus pattern. The framework of the proposed method is
presented in section 3. The experimental results have been
demonstrated in section 4. Finally, section 5 concludes the
paper.
II. L OCAL PATTERNS
A. Local Binary Pattern
The local binary pattern (LBP) has been proposed by
Ojala et al. for local information of pixels in an image [17].
In LBP, every pixel of the image is treated as the center
pixel, one at a time, and local information is acquired for
each pixel that depends on neighboring pixels. Each center
pixel is subtracted from all neighboring pixels and a binary
number is assigned to each neighboring pixel. These binary
numbers construct the local binary pattern for the center
pixel. Further, local binary patterns are multiplied by some
weights and summed up to a pattern value which is known
as local binary pattern value for a center pixel. For each
pixel in the image, local binary pattern value is calculated.
For a center pixel Ic and neighboring pixel In , LBP can be
obtained as follows:
p−1
LBPp,r = ∑ 2n × S(In − Ic ) (1)
n=0

1 x≥0
S(x) =
0 else
where p and r are the number of neighboring pixels and ra-
dius respectively. The rotation invariant LBP can be obtained
as follows:
⎧ p−1
⎨ S(I − I ) if U(LBP ) ≤ 2
riu2 ∑ c n p,r
LBPp,r = n=0 (2)
⎩ to reduce the complexity of the algorithm. Hue, saturation
p+1 if else
 
U(LBPp,r ) = S(I p−1 − Ic ) − S(I0 − Ic ) +
p−1 (3)
∑ |S(In − Ic ) − S(In−1 − Ic )|
n=1
A sample window example of LBP pattern is shown in
figure 1.
78
B. Local Rhombus Pattern
In the proposed method, features are derived using neigh-
boring pixels that evaluate the mutual relationship among
neighboring pixels instead of the center pixel. Four neigh-
borhood pixels, two each in the vertical and horizontal
directions have been used for pattern formation. For each
of the four pixels, two neighboring pixels are considered,
and relationships based on their comparison are extracted.
As shown in figure 2(a), I1 , I3 , I5 and I7 are considered
for pattern formation. A sample window of the image is
given in figure 2(b). In figure 2(c-f), the steps involved
in pattern creation process are demonstrated. Figure 2(g-
i) shows how the pattern values are obtained from local
rhombus pattern. In figure 2(c), the pixel I1 is subtracted
from pixel I2 and I8 , and based on both difference values,
a pattern is assigned to I1 . If both difference values are
of different signs, i.e., positive and negative, then ‘0’ is
assigned, and if both difference values are of same sign,
i.e., both positive or both negative then ‘1’ is assigned to
that pixel. Hence, in this example, 0, 1, 1 and 1 values are
assigned to I1 , I3 , I5 and I7 , respectively. These values are
further multiplied with weights as mentioned in the figure
2(h) and summed up to a single pattern value as shown in
figure 2(i). The four pixels which form a rhombus around
the center pixel, are used for pattern creation. Hence, this
method is named as the local rhombus pattern.
For a pixel (x, y), LRP is formulated as follows:
T1n = In−1 − In , T2n = In+1 − In , ∀ n = 3, 5, 7. (4)
T1n = I8 − In , T2n = In+1 − In , f or n=1 (5)
⎧
⎪
⎪0 i f T1n ≥ 0 & T2n < 0
⎨
0 i f T1n < 0 & T2n ≥ 0
F(T1n , T2n ) = (6)
⎪
⎪1 i f T1n ≥ 0 & T2n ≥ 0
⎩
1 i f T1n < 0 & T2n < 0
3
LRP(x, y) = ∑ 2i × F(T12i+1 , T22i+1 ) (7)
i=0
III. F RAMEWORK OF PROPOSED ALGORITHM
A. Target object representation
The proposed method is inspired by the local binary pat-
tern that extracts the local information based on neighboring
pixels [17] and center pixel. Ning et al. used joint histogram
of LBP and RGB color channels for object tracking [12].
In the proposed work, the HSV color space is used
for color information of the target object. It separates the
color component (hue), brightness (saturation) and intensity
(value) such that individual information regarding hue, satu-
ration and value can be extracted. Hue, saturation and value
components of the HSV color space are quantized in order
and value components are quantized into 18, 3 and 3 bins
respectively. Texture information of the object is created by
LRP and joint histogram of LRP, and then hue, saturation
and value components are generated. The local rhombus
pattern has a total of 16 features. Hue, saturation and value
have 18, 3 and 3 bins respectively. Hence, the total length
of histogram is 16×18×3×3. The target object is tracked in
the next frames using mean shift tracking algorithm [4]. The
2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)

Fig. 1: Local binary pattern example

Fig. 2: Local rhombus pattern sample window example

algorithm for the proposed system is given in the following
sequence:
B. Algorithm
Input: Video sequence with location of the target object
in the first frame.
Output: Tracked object in full video.
1) Upload the video and select the target object in the first
frame for tracking.
2) Compute LRP of the target object in the first and next
frame.
3) Convert the current and next frame from RGB to HSV
color space, and quantize hue, saturation and value bins
to 18, 3 and 3 respectively.
4) Create joint histogram of hue, saturation, value and
LRP for the target object in the current and next frame.
5) Track the target object in the next frame using mean
shift tracking algorithm with joint HSV and LRP his-
togram.
6) Repeat the process from step 2 to 5 till the end frame.
IV. E XPERIMENTAL RESULTS AND DISCUSSIONS
In this work, two experiments on different videos have
been conducted and the proposed algorithm is compared
with following two algorithms.
LBPriu2 RGB : Rotation invariant uniform local binary
pattern + RGB color histogram
LEP RGB : Local extrema pattern + RGB color histogram
The proposed method is abbreviated as LRP HSV. In
each experiment, the target object is selected manually and
marked as a red box. Our algorithm first extracts features,
and then tracks the required object in the next frames.
In the first experiment, a video sequence of nearly similar
moving cars is used. The video sequence comprises 201
frames of size 640× 480. A car is selected as the target
object for tracking, and marked with red color, as shown
in figure 4. In next frames, the target object is tracked, and
shown in red color. Results of LBPriu2 RGB, LEP RGB and
LRP HSV are shown in figure 4(a), (b) and (c) respectively.
It has been observed that up to frame 63, LBPriu2 RGB and
LEP RGB tracked the correct object, whereas they lost the
track of object near after frame 63. The failure in tracking
can be attributed to the fact that, near frame 63 another car
went through the target object, and both methods could not
identify the correct car between the two. On the contrary, the
proposed method handled this issue, and tracked the correct
object till the end as shown in figure 4(c).
In the second experiment, a video of football game
is employed for the purpose of tracking one player. The
tracking results of both methods have been demonstrated in

79
 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)

Fig. 3: Results from player tracking in football video of (a) LBPriu2 RGB (b) LEP RGB and (c) LRP HSV

TABLE I: Feature vector length of proposed method and previous methods

figure 3. At the beginning of the video, all three algorithms
worked equally well and tracked the correct object. This is Method Feature vector length Time taken
because the target object was almost in isolation, and no LBPriu2 RGB 10 × 8 × 8 × 8 = 5120 1:08
LEP RGB 16 × 8 × 8 × 8 = 8192 1:15
disturbing objects were present nearby. Whereas, towards LRP HSV 16 × 18 × 3 × 3 = 2592 1:02
the end frames, LBPriu2 RGB and LEP RGB missed the
target object, and started tracking other spurious objects. The
reason for incorrect tracking is that towards the end video
HSV color space is used for color features. Next, a joint
frames, other players with similar dress(white) have come
histogram is constructed for color-texture features. The pro-
close to the target object, and both methods have failed to
posed method is applied to object tracking application. Ob-
distinguish the target object from other objects. Whereas, in
ject tracking is performed on two video sequences of traffic
the proposed method, the target object is tracked correctly
and sports using mean shift tracking algorithm. Experimental
till the end of the video, and are shown in figure 3(c).
results prove that the proposed method is significantly better
Feature vector length of all three methods are given below than LBPriu2 RGB and LEP RGB algorithms.
in the table I. Feature vector length of the proposed method
is considerably less than other two approaches.
R EFERENCES
V. C ONCLUSION [1] A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” Acm
A novel algorithm in the field of object tracking is computing surveys (CSUR), vol. 38(4), 13, 2006.
[2] D. Comaniciu, V. Ramesh, and P. Meer, “Real-time tracking of
proposed. The proposed LRP for texture features extracts non-rigid objects using mean shift,” Computer Vision and Pattern
the local relationship among neighboring pixels, and the Recognition, IEEE Conference on, vol. 2, pp. 142–149, 2000.

80
2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)
Fig. 4: Object tracking in road traffic video (a) Results of LBPriu2 RGB (b) LEP RGB and, (c) LRP HSV
[3] R. Cucchiara, C. Grana, M. Piccardi, A. Prati, and S. Sirotti, “Im-
proving shadow suppression in moving object detection with hsv
color information,” in Intelligent Transportation Systems, Proceedings
IEEE, pp. 334–339, 2001.
[4] D. Comaniciu, V. Ramesh, and P. Meer, “Kernel-based object track-
ing,” Pattern Analysis and Machine Intelligence, IEEE Transactions
on, vol. 25(5), pp. 564–577, 2003.
[5] K. She, G. Bebis, H. Gu, and R. Miller, “Vehicle tracking using on-
line fusion of color and shape features,” Intelligent Transportation
Systems, 7th International IEEE Conference on, pp. 731–736, 2004.
[6] R. Babu, R. Venkatesh, and P. Parate, “Robust tracking with
interest points: A sparse representation approach.” Image and Vision
Computing, vol. 33, pp. 44-56, 2015.
[7] G. Zhao and M. Pietikainen, “Local binary pattern descriptors
for dynamic texture recognition,” Pattern Recognition, 18th IEEE
International Conference on, vol. 2, pp. 211–214, 2006.
[8] M. Heikkila and M. Pietikainen, “A texture-based method for
modeling the background and detecting moving objects,” Pattern
Analysis and Machine Intelligence, IEEE Transactions on, vol. 28(4),
pp. 657–662, 2006.
81
[9] V. Takala and M. Pietikainen, “Multi-object tracking using color,
texture and motion,” Computer Vision and Pattern Recognition, IEEE
Conference on, pp. 1–7, 2007.
[10] F. Pernici and A.D. Bimbo, “Object tracking by oversampling
local features,” Pattern Analysis and Machine Intelligence, IEEE
Transactions on, vol. 36(12), pp. 2538–2551, 2014.
[11] L. Wang, T. Liu, G. Wang, G., K.L. Chan and Q. Yang, “Video
tracking using learned hierarchical features,” Image Processing, IEEE
Transactions on, vol. 24(4), pp. 1424–1435, 2015.
[12] J. Ning, L. Zhang, D. Zhang, and C. Wu, “Robust object tracking
using joint color-texture histogram,” International Journal of Pattern
Recognition and Artificial Intelligence, vol. 23(07), pp. 1245–1263,
2009.
[13] G. Xue, J. Sun, and L. Song, “Dynamic background subtraction
based on spatial extended center-symmetric local binary pattern,”
Multimedia and Expo, IEEE International Conference on, pp. 1050–
1054, 2010.
[14] S. Murala, R.P. Maheshwari, and R. Balasubramanian, “Local
maximum edge binary patterns: a new descriptor for image retrieval
and object tracking,” Signal Processing, vol. 92(6), pp. 1467–1479,
 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)

2012.
[15] P.P. Dash, D. Patra, and S.K. Mishra, “Local binary pattern as a
texture feature descriptor in object tracking algorithm,” in Intelligent
Computing, Networking, and Informatics, pp. 541–548, Springer,
2014.
[16] J. Liu, P. Carr, R.T. Collins, and Y. Liu, “Tracking sports players
with context-conditioned motion models,” inComputer Vision and
Pattern Recognition (CVPR) IEEE Conference on, pp. 1830-1837,
IEEE, 2013.
[17] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of
texture measures with classification based on featured distributions,”
Pattern recognition, vol. 29(1), pp. 51–59, 1996.
[18] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition with
local binary patterns,” European Conference on Computer vision, pp.
469–481, Springer, 2004.
[19] C. Shan, S. Gong, and P.W. McOwan, “Robust facial expression
recognition using local binary patterns,” Image Processing, IEEE
International Conference on, vol. 2, pp. II:914–917, 2005.
[20] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale
and rotation invariant texture classification with local binary patterns,”
Pattern Analysis and Machine Intelligence, IEEE Transactions on,
vol. 24(7), pp. 971–987, 2002.
[21] S. Liao, M.W. Law, and A.C. Chung, “Dominant local binary patterns
for texture classification,” Image Processing, IEEE Transactions on,
vol. 18(5), pp. 1107–1118, 2009.
[22] Z. Guo and D. Zhang, “A completed modeling of local binary
pattern operator for texture classification,” Image Processing, IEEE
Transactions on, vol. 19(6), pp. 1657–1663, 2010.
[23] M. Heikkilä, M. Pietikäinen, and C. Schmid, “Description of interest
regions with local binary patterns,” Pattern recognition, vol. 42(3),
pp. 425–436, 2009.
[24] S. Murala, R.P. Maheshwari, and R. Balasubramanian, “Directional
local extrema patterns: a new descriptor for content based image
retrieval,” International Journal of Multimedia Information Retrieval,
vol. 1(3), pp. 191–203, 2012.
[25] X. Tan and B. Triggs, “Enhanced local texture feature sets for
face recognition under difficult lighting conditions,” in Analysis and
Modeling of Faces and Gestures, pp. 168–182. Springer, 2007.
[26] S. Murala, R.P. Maheshwari, and R. Balasubramanian. “Local tetra
patterns: a new feature descriptor for content-based image retrieval,”
Image Processing, IEEE Transactions on, vol. 21(5), pp. 2874–2886,
2012.

82