Multiple object tracking using motion vectors from
compressed video
Weisheng Li
School of Computer Science, Engineering and Mathematics
Flinders University
Adelaide, Australia
li0655@flinders.edu.au
Abstract— Motion vectors extracted from a compressed video
file can be used to track objects in the video and it could be efficient
as motion vectors provide trajectory information of the objects.
However, tracking objects represented by the motion vectors can
be inaccuracy because of camera movement, small size sets of
motion vectors acting as noise, unmoving of the object and
occlusion. These are conditions in most real world video
application. The system in this paper uses the statistical and
distributional information of motion vectors to overcome the
problems with three stages. 1) Frame preprocessing uses a Mode
reduction technique to remove unwanted motion vectors created
from camera movements. 2) Intra-frame processing: k-means is
used to segment and cluster moving objects. Statistical standard
deviation is used to extract objects’ torso and remove small size
sets of motion vectors. 3) Inter-frame processing: By comparing
the positional information between successive frames, tracking
object in successive frames is assigned a same label. A copying rule
is used to represent the stopping of the tracking object. The
direction and velocity information of motion vector is used for the
occlusion problems. Overall, an experiment on tracking multiple
basketball players demonstrates a good result of the system.
Keywords—motion vectors; multiple object tracking;
clustering;compressed video;
I. INTRODUCTION
Object tracking in video has many practical applications
such as video surveillance, human activity recognition and
robotic. While the availability of video is prevalent, compressed
domain methods are more suitable than pixel domain methods
as the lower computational requirement [1]. For video formats
like MPEG and H.264, motion vectors are used in compression
to reduce the transmission and storage requirement [2]. They
record the positional information which is indicating a pixel’s
source coordinates and destination coordinates between two
consecutive frames. It is also knew as optical flow. These
coordinates are useful for object tracking as they provide
trajectory information of moving objects in the scene and the
generated attributes of the objects such as the direction of
travelling and the velocity. There are other advantages of using
motion vectors. In contrast to the popular object tacking methods
[3] which use a target shape model, there is no need to predefine
any target shape template as an object is represented by a set of
motion vectors. Comparing to the feature based methods which
segment object by edge detection on pixel domain in every
978-1-5386-2839-3/17/$31.00 ©2017 IEEE
David Powers
School of Computer Science, Engineering and Mathematics
Flinders University
Adelaide, Australia
david.powers@flinders.edu.au
frame, motion vector approach requires much lower
computational cost. More importantly, the trajectory
information of motion vectors are valuable for tracking objects
which the above methods don’t possessed.
Motion vectors are determined during the motion estimation
step when encoding video by matching similar blocks between
successive frames. After encoding, motion vectors are used to
store the video. Extracting the motion vectors without decoding
the video and use them for object tracking is the current research
trend. A few articles discuss the use of motion vector to track
multiple objects. The approaches mainly contains two steps.
Object detection [4-6] which extracts the sets of motion vectors
representing objects and object correspondence [7-9] which
associates same object between consecutive frames. You et al.
[10] uses probabilistic spatio-temporal MB filtering (PSMF) to
detect objects and track the multiple objects with the color
information. Kas et al. [11] uses an outlier filter to smooth the
boundary of object and remove noise. For the object
correspondence, object size and moving direction is used to
associate the same object. In the system of this paper, different
techniques are used and the techniques are based on applying
statistical methods. These methods are designed to overcome the
problems including: First, camera movements, when video is
recording, generate homogenous background motion vectors
which are presented in the data after the extraction from
compressed video file. These background motion vectors need
to be removed for motion segmentation. Second, the number of
moving objects in a frame is unknown. This information is
needed to group motion vectors to represent an object as a set of
motion vectors. This system provides a method to apply k-means
with knowing approximated number of clusters. Third, when
tracking a person often the movement of the torso is interested.
But other movement of the other parts such as hands will distract
the tracking region. The distributional information of the
motion. Lastly, object occlusion is a changeling problem for
object tracking task and this system provides a method to
differentiate the front object and the back object by the direction
and the velocity of motion vectors to make sure correct labelling.
The system is designed for multiple object tracking under
those problems. Three stages are performed to track multiple
target objects. The first stage is frame preprocessing to remove
unwanted motion vectors generated by camera movements with
a Mode reduction technique. The second stage is to use k-means
to cluster each frame individually. All moving object for each
frame is detected and statistical standard deviation is used to
extract objects' torus. The third stage associates the same target
between successive frames with a same label by using the
positional information. Also, a copying rule is defined to
represent the stopping of the tracking object. Finally, the
occlusion problem between two objects is solved by using the
direction and the velocity of motion vectors.
II. FRAME PREPROCESSING
After extracting the motion vectors from a compressed video
using FFMPEG [12], they are imported into MATLAB. Each
frame is stored as a cell array and they contain motion vectors.
So, each cell array contains a 4 column matrix:
(1)
⋮ ⋮ ⋮ ⋮
Where and are the coordinates of a pixel in previous
frame and is for source. Where and are the coordinates
of a pixel in current frame and is for destination. A row
containing these four attributes contributes a motion vector. The
number of rows is the number of pixels that have a motion vector
in a frame. Figure 1 shows the motion vectors where the tail of
the arrow is the coordinate form previous frame and the head of
the arrow belongs to current frame. So they are the points that
have been moved. For visualizing purpose, the magnitudes of all
motion vector are set to be the same. Only the direction of each
motion vector is shown on Figure 1.
Fig. 1. Motion vectors with same magnitudes. (A YouTube video [14])
It can be seen that in Figure 1 there are a large number of
motion vectors that are similar in direction which is calculated
as (2) and in velocity as (3). On the other hand, the number
of motion vectors on the players is small comparing to these
background motion vectors. They are created by the camera
movement. Camera movement may create one or more such
group of background motion vectors. A Mode reduction
technique is designed to remove these groups.
= arctan ( ) (2)
= ( − ) + ( − ) (3)
In statistic, a Mode is the value that occurs most often in a
dataset within a dimension. By iteratively removing the motion
vectors of the Mode in the dimension of (2) until reaching a
threshold, the background motion vectors are removed. The
threshold can be estimated as the number of motion vectors on
the players. When removing the background motion vectors in
each iteration, some motion vectors on the players maybe have
same direction and different velocity to the background motion
vectors. To avoid removing these motion vectors, the Mode of
the background motion vectors in the dimension of (3) is also
considered.
For each frame the Frame Preprocessing algorithm is summaries
as:
1. Find the Mode of the motion vectors based on the
direction: In the dimension of , the most occurring
value is found as the Mode of direction. The motion
vectors having this Mode are selected.
2. Among the selected motion vectors, in the dimension of
, the most occurring value is found as the Mode of
velocity. The motion vectors having this Mode are
selected.
3. Remove the selected motion vectors.
4. Repeat steps 1 to 3 until the number of motion vectors
in the frame is the number of motion vectors on all the
objects.
Fig. 2. After removing background motion vectors.
 III. INTRA FRAME PROCESSING
The aim of this process is to detect all moving objects in each
individual frame with the objects’ torso represented by motion
vectors. The first step is to use k-means [13] to cluster the data
and of each frame. It clusters all the current position data
of all moving objects so that each cluster represents an object.
However, the difficulty of using k-means is the unknown of the
number of objects in the frame. The approach is to use an
approximated number of objects estimated by user and remove
unwanted motion vectors in any cluster after performing k-
means. Figure 3 shows a result after using k-means with
approximated number of clusters. It can be seen that some bad
clusters have a void inside. The small sparse groups around the
void are created by small parts of the object such as hand
movement or the object’s shadow. They are unwanted for object
tracking. They also appears in the compact clusters sitting
around the cluster. So, the unwanted motion vectors are all
around all clusters. The approach to remove them is to use
statistical standard deviation to detect them.
Fig. 3. k-means clustering with appromixated number of clusters.
Standard deviation is widely used in statistic to remove
outliers. It measures the distance between the mean and the tail
of the normal distribution where within this distance the
percentages of data points are distributed according to the
standard deviation, , which is equal to the square root of the
variance. For example, within ± there are around 68% data
points and within ±2 there are around 96% data points. Most
data points are within ± and less data points are in the outer.
This rule can be used to remove the unwanted motion vectors as
the unwanted motion vectors are properly located around
the ±2 . The determination of the number in front of ±
depends on the resolution of the video. In the system of this
paper, 1.2 is selected. So for each cluster, a motion vector is
removed if it is outside:
± 1.2 or ± 1.2 (4)
Where is the mean of the coordinates of all the
motion vectors in the cluster and is the standard deviation of
. is the mean of the coordinates of all the motion
vectors in the cluster and is the standard deviation of .
Final step for intra frame processing is to remove clusters
consisting small number of motion vectors. These motion
vectors are from the bad clusters as discussed and the outliers
when performing the k-means.
For each frame the Intra Frame Processing algorithm is
summaries as:
1. Perform k-means with approximated number of
clusters.
2. For each cluster, remove motion vectors that sitting
outside the 1.2 standard deviation from the center of the
cluster.
3. Remove clusters that have small number of motion
vectors.
Figure 4 shows the resulting clusters that their sizes are
reduced by removing the unwanted motion vectors and the
moving objects are now represented by the motion vectors on
the torso. They are also represented by ellipses with 4 as the
dimeter.
Fig. 4. k-means with motion vectors removing using 1.2 standard deviation.
IV. INTER FRAME PROCESSING
After every individual frame has been clustered, all moving
objects are labeled. However, the same object in successive
frames gets different labels. To correspond the labels between
successive frames, the system performs a comparison algorithm
between each tracking object in current frame and all the clusters
in next frame. The aim is to predict the tracking object’s position
in next frame. If there is a cluster of next frame is close to the
tracking object of current frame, it is a good indicator that the
cluster is the displacement of the tracking object.
 In the first step, each tracking object in current frame is
compared with all the clusters in next frame. The closeness of
the cluster to the tracking object is defined by their overlapping
area. This overlapping area is indicated by the number of motion
vectors from the cluster residing within the tracking object’s
ellipse defined by the standard deviations as shown in Figure 5.
The radius of the x and y axes are:
±2 (5)
±2 (6)
Where and are the mean and the standard deviation of
the coordinates of all the motion vectors on the tracking
object. and are the mean and the standard deviation of the
coordinates of all the motion vectors on the tracking object.
Motion vectors from a cluster of next frame and residing in
the tracking object’s ellipse are considered. If the number of
them is high enough, the cluster is considered as the possible
position of the tracking object in next frame. After comparing all
the clusters in next frame, the cluster having the highest number
is selected as the position of the tracking object.
V. EXPERIMENT
The experiment is the application of the system to track
basketball players. Two kinds of situation are included. The first
kind is that all the players are running from one side of the court
to the other and three players are selected at the first frame. The
second kind is that a player with the ball runs and passes the
defenders. The player with the ball and the player in occlusion
with him are selected at the first frame.
A video downloaded from YouTube [14] contains the
situations in different match. 5 video cuts for each situation are
performed the test. Each video is around 3 to 4 seconds and is
about 100 frames. The experiment tries to find out how many
players are loss track by the system. Figure 6 and Figure 7 are
examples for the two situations.

First frame

Fig. 5. Overlapping area.

The second step is to detect the stopping objects. In the case Occlusion frame
that there is no clusters having high enough number of
overlapping motion vectors, it indicates there is no cluster in
next frame is overlapped with the tracking object. In other
words, the tracking object is stop moving. Then the system copy
the coordinates and of the tracking object into next frame.
These coordinates represent the tracking object in next frame.
The last step is to deal with the occlusion problem. If a
cluster, , is assigning with two labels, it indicates there are two
tracking objects are in occlusion. is labelled with one of the
two labels of the two objects. This is determined by the
similarity between and the tracking objects. The object having
higher similarity in term of direction and velocity will be the
front object and is assigned with the same label as the object. Last frame
On the other hand, the back object’s label is recoded and it will
be assigned to a cluster when there is a less similar cluster close
Fig. 6. Tracking players with occulsion.
to . This approach make sure the front object is always having
correct label.
 In the tracking of two players when occlusion occurs, there
are 2 cases fail. In one case, both players are loss. In another, one
player is loss. There are total 10 players are tracked. So the
accurate rate is 70%. In the tracking of three players, 4 out of 15
players are loss. So the accurate rate is 73.33%. From both the
situation, the main cause for the trajectory loss is the occlusion.
First frame
Last frame
Fig. 7. Tracking multiple players.
VI. CONCLUSION
In this paper, it is demonstrated that tracking multiple objects
can be efficient by using motion vectors extracted from
compressed video. The system applies statistical and clustering
techniques on motion vectors to track multiple objects in real
video. This approach allows effective use of the positional,
distributional and directional information of the motion vectors.
To overcome the problems of using motion vector for
tracking multiple objects, the system performs three stages on
the data. The preprocessing stage uses a Mode reduction
technique to remove background motion vectors created by
camera movements. The intra frame processing uses k-means to
cluster the motion vectors for the representation of the moving
objects. To extract object’s torus as the tracking area, statistical
standard deviation is used to remove unwanted motion vectors.
The inter frame processing uses the positional information to
associate the same target between successive frames with a same
label. The occlusion problem is solved by using the direction and
velocity information. In the experiment of tracking multiple
basketball player, the system performs well.
Future work will be developing techniques for the more
difficult occlusion situations such as more than two objects are
in occlusion. Use of color information may improve the result.
Using k-means with approximated number of clusters may be
updated with a more advance method from statistics to
automatically determine the number of clusters. This makes the
application easier to use. The quality of the motion vectors
extracted from compression video is affected by the
compression formats such as MPEG and H.264. The parameters
such as the size of motion blocks, compression rates and
compression speed are to be considered in the future work.
REFERENCES
[1] Babu, R., Tom, V., & Wadekar, M. (2016). A survey on compressed
domain video analysis techniques. Multimedia Tools and Applications,
75(2), 1043-1078.
[2] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, "Overview of
the H.264|AVC Video Coding Standard," IEEE Trans. Circuits Syst.
Video Technol., vol. 13, No. 7, pp. 560–576, July 2003.
[3] Trucco, E., & Plakas, K. (2006). Video Tracking: A Concise Survey.
Oceanic Engineering, IEEE Journal of, 31(2), 520-529.
[4] Babu, R., Ramakrishnan, K., & Srinivasan, S. (2004). Video object
segmentation: A compressed domain approach. Circuits and Systems for
Video Technology, IEEE Transactions on, 14(4), 462-474.
[5] Xiao-Dong Yu, Ling-Yu Duan, & Qi Tian. (2003). Robust moving video
object segmentation in the MPEG compressed domain. Image Processing,
2003. ICIP 2003. Proceedings. 2003 International Conference on, 3, III-
933.
[6] Fei, W., & Zhu, S. (2010). Mean shift clustering-based moving object
segmentation in the h.264 compressed domain. IET Image Processing,
4(1), 11-18.
[7] Favalli, L., Mecocci, A., & Moschetti, F. (2000). Object tracking for
retrieval applications in MPEG-2. Circuits and Systems for Video
Technology, IEEE Transactions on, 10(3), 427-432.
[8] Yokoyama, T., Iwasaki, T., & Watanabe, T. (2009). Motion Vector Based
Moving Object Detection and Tracking in the MPEG Compressed
Domain. Content-Based Multimedia Indexing, 2009. CBMI '09. Seventh
International Workshop on, 201-206.
[9] Lan Dong, Zoghlami, & Schwartz. (2006). Object Tracking in
Compressed Video with Confidence Measures. Multimedia and Expo,
2006 IEEE International Conference on, 753-756.
[10] You, W., Sabirin, M., & Kim, M. (2012). Real-time detection and tracking
of multiple objects with partial decoding in H.264/AVC bitstream
domain. Proceedings of SPIE 2009, Volume: 7244, Publisher: SPIE,
Pages: 72440D-72440D-12.
[11] Käs, C., & Nicolas, H. (2009). An approach to trajectory estimation of
moving objects in the H.264 compressed domain. Lecture Notes in
Computer Science (including Subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics), 5414, 318-329.
[12] FFmpeg: Documentation. Retrieved from
https://www.ffmpeg.org/documentation.html
[13] MacQueen, J. B. (1967). Some Methods for classification and Analysis of
Multivariate Observations. Proceedings of 5th Berkeley Symposium on
Mathematical Statistics and Probability. University of California Press.
pp. 281–297.
[14] [NBA]. (2016). Best Highlights From the 2017 NBA Playoffs | 1st Round.
Retrieved from https://www.youtube.com/watch?v=yGnKHxNVibw