Int J Multimed Info Retr (2014) 3:15–28
DOI 10.1007/s13735-013-0042-8
REGULAR PAPER
Multivariate time series modeling of geometric features
of spatio-temporal volumes for content based video retrieval
Chiranjoy Chattopadhyay · Amit Kumar Maurya
Received: 30 July 2013 / Revised: 15 August 2013 / Accepted: 16 August 2013 / Published online: 3 September 2013
© Springer-Verlag London 2013
Abstract In this paper, we address the problem of Content
Based Video Retrieval using a multivariate time series mod-
eling of features. We particularly focus on representing the
dynamics of geometric features on the Spatio-Temporal Vol-
ume (STV) created from a real world video shot. The STV
intrinsically holds the video content by capturing the dynam-
ics of the appearance of the foreground object over time, and
hence can be considered as a dynamical system. We have cap-
tured the geometric property of the parameterized STV using
the Gaussian curvature computed at each point on its surface.
The change of Gaussian curvature over time is then modeled
as a Linear Dynamical System (LDS). Due to its capability
to efficiently model the dynamics of a multivariate signal,
Auto Regressive Moving Average (ARMA) model is used
to represent the time series data. Parameters of the ARMA
model are then used for video content representation. To dis-
criminate between a pair of video shots (time series), we
have used the subspace angle between a pair of feature vec-
tors formed using ARMA model parameters. Experiments
are done on four publicly available benchmark datasets, shot
using a static camera. We present both qualitative and quan-
titative analysis of our proposed framework. Comparative
results with three recent works on video retrieval also show
the efficiency of our proposed framework.
Keywords Content Based Video Retrieval ·
Spatio temporal volume · Gaussian curvature ·
Time series · ARMA
C. Chattopadhyay (B) · A. K. Maurya
Indian Institute of Technology Madras, Chennai 600036, India
e-mail: cchatto@cse.iitm.ac.in
A. K. Maurya
e-mail: peaceamit@gmail.com
1 Introduction
The process of Content Based Video Retrieval (CBVR),
involves retrieving similar video in response to users’ search
queries, using the inherent properties extracted from the
video itself. However, the quality of the retrieval result of
any CBVR system depends on the quality of content rep-
resentation in the feature database. On the other hand, the
quality of features also depends on the video segments (i.e.,
video objects, video shots, scenes, etc.). Hence, represent-
ing video based on its content has emerged as an important
research area in the field of multimedia retrieval. It has a large
number of applications in fields such as surveillance, secu-
rity, entertainment, education, etc. One major need of these
applications is to store large collections of video data. The
challenge is to represent these large video databases in terms
of their contents.
In many of these applications, with respect to a video data-
base when a query video is given, the following two questions
need to be addressed:
– In which category does the video shot belong to?
– In the given category, what are the “most similar” video
shots in the database, to the given query?
The former is known as video categorization or classifica-
tion, and the later is known as the retrieval task. Modeling the
intra- and inter-class variations among different categories of
video shots in a supervised (classification) or unsupervised
(clustering) way, will lead to answering the first question.
On the other hand, identification of content in a video shot,
use of proper feature and discriminating criteria and a proper
ranking function are required to answer the second ques-
tion. In our current work, we are particularly interested in
answering the second question by modeling the dynamics of
123
16
geometric features of the Space Time Volume (STV), created
from static camera video shots, as a LDS. Parameters of the
Auto Regressive Moving Average (ARMA) model are used
to represent the time series data, and video shots are retrieved
from the database based on the match-cost between the
parameters.
1.1 Related work
Spatio-temporal features and time series modeling has been
the area of interest to many researchers in recent past. In
literature, works related to our proposed framework can
be broadly grouped into two categories: (i) use of spatio-
temporal features as content descriptor, (ii) application of
time series modeling technique for various computer vision
tasks on video data. In the rest of this section of related work
we will discuss in details some of those key contributions.
Spatio-temporal features have been exploited extensively
by the researchers to represent and retrieve videos, both from
the pixel and compressed domains [3,19,25]. During recent
years, use of motion trajectory and object shapes as fea-
tures have gained importance for representing video shots.
Motion trajectory based features have been presented as con-
tent descriptors and used for retrieval tasks in [5,9,13,23].
Statistical features have been reported as a spatio-temporal
content descriptor in [27]. Existing video retrieval systems
[15,16,23] extract low-level features from video shots and
has reported promising results. A global motion trajectory
representation based video retrieval technique has been pro-
posed in [18]. The video object based retrieval technique
proposed in [16] segments video objects and represent the
color, shape, texture and motion features of video objects. In
one of our earlier work [10], a joint spatio-temporal repre-
sentation of a video object has been analyzed, where shape
and trajectory features are combined to generate an STV. A
Multi-spectral approach was followed to represent the video
content as peaks and ridge lines of the EMST-CSS surface. A
space time salient object approach has been reported in [21]
for video content representation and retrieval. It proposed a
fusion method for motion and spatial saliency integration to
detect spatio-temporal salient objects, based on both atten-
tion analysis and interest point trajectory tracking. MPEG-7
visual descriptors are used as video content descriptors in
[25] for CBVR task. Various approaches exist in literature
that use processing on the STV for human action recognition
[7,22,32]. Differential geometry and geometric optimization
techniques have been used to design robust vision based algo-
rithm [35,36,39].
Features extracted from video shot which are modeled
as time series have been utilized in certain computer vision
problems. Examples include, dynamic textures [17], face
recognition [1], gait recognition [6], image-video based
recognition [37] and activity recognition [8]. Technique for
123
Int J Multimed Info Retr (2014) 3:15–28
similarity search in multi-dimensional data sequence has
been reported in [24]. An algorithm to extract time series
from video to characterize type of motion is discussed in
[20]. A framework for finding similar time series with natural
relations is proposed in [14]. In [2], a similarity measure for
multivariate time series has been proposed using the Euclid-
ean distance based on Vector Autoregressive (VAR) models,
for human action classification. Full-featured wavelet trans-
forms for similarity search of time series data was proposed
in [31]. Bag-of-pattern approach [26], statistical modeling
[11], and kernel based methods [40] were also explored by
researchers in the past for modeling time series for different
applications.
Literature review suggests that most existing CBVR sys-
tems focus on video analysis, visual feature extraction, and
supporting query by example. Few CBVR systems focus
upon exploiting the spatio-temporal features available in
video shots for unique representation and their efficient
retrieval. Dearth of a compact and unified representation
of spatio-temporal information of the content present in a
video shot in most CBVR systems makes them less efficient
in terms of computing and effective in measuring similarity
between a pair of video shots.
Contribution In this work we particularly focus on repre-
senting video content as a multi-variate time series. We first
generate a smooth, parameterized STV from a given video
shot by combining the shape and trajectory information of
the moving foreground blob. Then we model the dynam-
ics of the surface curvature at each point on the STV as a
Linear Dynamical System (LDS) using ARMA model. Para-
meters of the ARMA model have been used for representing
the video content. We have also proposed a framework for
CBVR using this content representation technique.
Organization of the paper In Sect. 2, we briefly discuss
the overall framework of our proposed method. Section 3
describes the method of smooth, parameterized STV forma-
tion from a given video shot. Section 4 discusses our proposed
method of modeling STV as time series and matching a pair
of video shot. Experimental setup with details of the datasets
used for experimental analysis is given in Sect. 5. In Sect. 6,
we demonstrate the application of the proposed content rep-
resentation technique for CBVR task, using four benchmark
datasets. Finally, concluding remarks are presented in Sect. 7.
2 Brief description of the proposed method
Spatio-temporal features constitute important cues to the
human perception system so as to extract salient information
from a video shot. The main focus of our proposed CBVR
framework is to combine the spatial and temporal features
present in a video shot and use it for video retrieval based on
their content.
Int J Multimed Info Retr (2014) 3:15–28
Fig. 1 Overall framework of the proposed method of Content Based Video Retrieval
Assumption Our proposed algorithm works under the
following assumptions:
– all the videos are shot using a static camera
– videos are pre-segmented into shots of short duration
(approx. 5–10 s.)
– each video shot contains a single, predominantly large
moving object.
Figure 1 depicts the overall framework of our proposed
CBVR technique. Our entire framework is divided into two
stages, (i) Database population and (ii) Retrieval.
In stage (i), given a input video shot we apply change
detection algorithm reported in [4]. We apply post-processing
on the resultant output to detect the prominent foreground
block that exists for the entire duration of the video shot.
After that, we extract the silhouette of the blob for a given
frame by tracing the boundary of the blob. This process is
repeated for all the blobs to get the silhouettes from all the
video frames. Centroid of the foreground blob is tracked
across all the frames to get the motion trajectory of the mov-
ing object. The rationale behind extracting the silhouette and
the trajectory is that the former captures the change in object
appearance while the later captures the motion kinematics.
Together, these two features represent the overall low-level
content in a video shot. Next, we generate a smooth, para-
meterized STV (u along the spatial and v along the temporal
direction) using all the contours and the trajectory. Gaussian
curvature is then extracted at each (u, v) location on the STV
surface and a feature matrix, which resembles the time series,
is created. This time series data is then given as an input to the
ARMA model to compute the parameters. ARMA parame-
17
ters corresponding to each STV (of a video shot) are stored
as a feature set in the feature database. Figure 2 shows a
schematic illustration of the method proposed in this paper.
During matching (stage (ii)), for a given query video shot
we extract ARMA parameters from the query STV. Similarity
between two given videos is calculated using the technique
proposed in [28]. Retrieved results are rank ordered based
on this match cost. Precision, Recall and F-measure are used
as metrics to quantitatively measure the performance of the
proposed algorithm. Next section describes the process of
STV formation in details.
3 Parameterized spatio-temporal volume generation
Formation of STV from a video shot, using the shape con-
tours, involves extraction of the foreground moving object
from a given video shot and then generation of a 3-D sur-
face. Then the STV surface is parameterized and refined,
depending upon the requirement of the underlying applica-
tion to obtain a smooth and regular surface. The STV for-
mation process consists of two stages: (i) extraction of shape
contour and trajectory from the segmented foreground blobs,
and (ii) generation of STV surface from these shape contours.
We will discuss these processes in details in the following
sub-sections.
3.1 Contour and trajectory extraction from foreground
blobs
Extracting the foreground object from an input video is the
initial step for STV generation. In the present work, we
123
18
Fig. 2 Different stages of processing in our proposed framework, n number of points on the contour, t total number of frames in the video shot.
u parameter represents arc length, v parameter represents time
consider static camera videos for experimental purpose. We
use “ViBe” [4], as the algorithm for background subtraction,
which works best with static camera video shots, for extract-
ing foreground moving objects. The result of foreground seg-
mentation process is represented as a set of foreground pixels
within a 2D-frame ( f ) and often contains a lot of noise. The
pixels marked as foreground need to be combined together
to generate a foreground blob. Each frame f may have either
produce, single blob or multiple blobs or null (in the absence
of any significant motion). Our aim is to select one signifi-
cant blob from each frame that represents the predominant
foreground moving object in the video. Since ViBe produces
noisy blobs for most frames, the choice of considering the
largest blob from each frame does not give the desired result.
We assume that consecutive positions of the video object in
successive frames has sufficient overlap. Given the output of
ViBe, Algorithm 1 outlines the steps involved in determining
the prominent foreground blob and thereafter extracting the
contours (silhouette) of each blob.
A systematic illustration of the prominent foreground blob
extraction process is explained in Fig. 3. For multiple non-
occluding foreground objects present in the video, separate
temporal paths ( pi s) are first obtained for each foreground
object. We individually compute the sum of the size of the
blobs belonging to a particular pi (step 6 of Algorithm 1).
Then we select the path with maximum net sum of blob sizes
and discard all the other paths (step 7 of Algorithm 1). After
the blobs of the prominent foreground objects are extracted,
the smooth outer contour of the foreground object is obtained
as follows.
– Extract the outer contour of the blobs by tracing the
boundary of the blobs. The contours are re-sampled to
generate n = 200 (empirically determined) points per
contour in each frame.
123
Int J Multimed Info Retr (2014) 3:15–28
Algorithm 1: Prominent foreground blob detection and
contour extraction.
Input: A set of n segmented frames of a video shot (output of
ViBe [4]).
Output: A Set (S) of silhouettes of the prominent foreground
blob extracted from each frame.
1 Suppress foreground pixels that do not have any other foreground
pixel in a N 8 neighborhood;
2 Apply morphological operation (erosion and dilation) on
remaining foreground pixels to get foreground blobs;
3 Obtain forward and backward correspondences between blobs in
successive frames based on euclidean distance between centroids
of the blobs;
4 Take intersection of forward and backward correspondences
between blobs in successive frames to obtain final
correspondences;
5 Identify set of path P = { pi : 1 ≤ i ≤ N p } across all frames by
connecting correspondences obtained in step 4, where N p =
total number of paths detected;
N i i
6 Compute, ∀ pi ∈ P; V (i) = k=1 bk , where N i = number of
th i
blobs on i path, and bk is the k blob on the i th path;
th
7 Select B = {bil : 1 ≤ i ≤ N l } on the l th path such that
l = arg maxi V (i), as the set of prominent foreground blobs;
8 Construct S = {sil : 1 ≤ i ≤ N l } by the tracing the exterior
contours ∀bil ∈ B;
9 return S;
Fig. 3 Extracting the prominent foreground blobs (a, b). Forward and
backward correspondence between blobs (c). Intersection of forward
and backward correspondences (d). Final set of selected foreground
blobs
Int J Multimed Info Retr (2014) 3:15–28
Fig. 4 Different stages of smooth parameterized STV formation
– The contours are smoothed using a Gaussian function to
remove minor artifacts present in the contour.
To get the trajectory of the moving foreground object, we
track the centroid of the extracted prominent foreground
blob (as given in step 7 of Algorithm 1) across the frames
of the video. These extracted contours represent the change
of the object shape in the given video shot over time and
the trajectory captures the change in the motion kinemat-
ics. The process of generating smooth, parameterized STV
using the contour and trajectory points is discussed in the
next section.
3.2 Parameterized STV formation
Process of STV generation finds resemblance to the prob-
lem of 3-D surface reconstruction, in the field of computer
graphics, where a 3-D surface needs to be reconstructed
from a set of data points. Surface reconstruction from shape
contours takes advantage of the approximated shape of the
object, but has several challenging issues (for details please
refer to [29]). The prime issue is to detect correspondence
between points on a pair of shape contours in successive
video frames and fit surface patches while ensuring smooth-
ness constraint. Moreover, while finding point correspon-
dence between the contours, order of points must also be
preserved. We define two variables for parameterization: u,
for the contour boundary and v along the trajectory (time
axis). Given a set of n contours, the process of STV forma-
tion involves three intermediate steps: (i) approximate STV
formation, (ii) parameterization, and (iii) smoothing. Figure
4 illustrates a schematic example of the proposed parameter-
ization process. Stage 1 shows three contour Ci s (mapped as
straight lines where the arc length between a pair of succes-
sive points is preserved) with all the points on the contours
indexed.
19
Fig. 5 Results of point correspondence between successive contours
of (a). Human running and (b) moving car using the technique proposed
in [34]. Only a few correspondences are shown for visual clarity
3.2.1 Approximate STV formation
First we find the correspondence between a pair of contours
Ci and Ci+1 using the technique proposed in [34]. Output of
the point correspondence is shown in Fig. 5. Using these ini-
tial correspondence we stack the contours along the trajectory
points to generate the approximate STV. It can be observed
from Fig. 4b that we may get three types of mappings: (i)
one-to-one, (ii) one-to-many, and (iii) many-to-one. To rep-
resent the geometric properties of the STV surface as time
series we need a special case of one-to-one mapping where
the order of the contour points are also preserved. The tech-
nique proposed in [34] ensures that the order of mapping is
preserved but it does not always ensure one-to-one mapping.
To achieve that we propose a parameterization technique.
3.2.2 STV parameterization
This step ensures that a unique one-to-one correspondence
is established between the points on the successive contours,
and all others are removed. Point correspondences on the
contours generated in the previous step (Sect. 3.2.1) gives us
the axial paths, all of which originate at the first contour (C1 )
and terminates at the last contour (Cn ). These axial paths and
the radial contours are used together for the parameterization
123
20
of the STV surface. At first, for each successive contour
pair Ci and Ci+1 , one to many point correspondences are
removed by keeping only the connections with minimum dis-
tance, leaving one-to-one and many-to-one mapping between
points (Fig. 4c). Next, all possible axial paths pi s are traced
from the first contour C1 to the last contour Cn and lengths of
the paths are computed. Many-to-one correspondences will
cause some paths to merge mid-way before they reach Cn .
For those paths, we retain only the paths with smallest length
and discard others. If, for some paths, the starting point does
not lie on C1 , then we remove those paths also. Thus some
of the axial paths will be filtered and the remaining paths
will run along the STV from C1 to Cn . This will create gaps
between contour points (Fig. 4d). We interpolate to create
new paths in the gap between the existing paths.
Let pi and pi+1 be two such adjacent paths with m points
between their end points on contour Cn . We generate m paths
between pi and pi+1 , by retracing from contour Cn to C1 . For
example, in Fig. 4d, we can observe that between p5 and p7 ,
we need to trace a path from contour C3 to C1 . We achieve
that by introducing a point equidistant from p5 and p6 on C2
and C1 . Then we traced the path from C3 to C1 by joining
the p6 ’s on each contour (shown in dotted lines in Fig. 4e).
By this process we will get a unique one-to-one mapping
between all the points on the successive contours. However,
the axial paths may not be properly distributed (congested at
some places and far apart in other) on the STV surface. To
uniformly distribute the axial lines on the STV surface, we
propose a smoothing technique in the next section.
3.2.3 STV smoothing
In the context of 3D surface reconstruction, smoothing has
received a considerable amount of interest and numerous
approaches have been proposed for generating smooth sur-
faces which are suitable for computing the differential sur-
face properties. In our case, to evenly distribute the axial
paths, each point with a parameter value u p is shifted along
the contour (u) such that the transformed point satisfies the
constraint: u p = wu −1 + (1 − w)u +1 , where the parameters
u −1 and u +1 represent the adjacent points on either the side
of u p (and w = 0.4). Figure 4f illustrates the effect of our
smoothing algorithm. It can be observed that as compared to
stage 5 (Fig. 4e), axial paths are uniformly distributed, and
thus generates a smooth, parameterized STV.
Figure 6 (I − I I I ) shows results of our experiment of
the STV generation for three different video shots taken
from different benchmark datasets. Figure 6a shows the
segmented foreground blobs (for a few uniformly spaced
frames), stacked along the trajectory (indicated by a dashed
line). The initial and the final STVs are shown in Fig. 6b
and c, respectively. Enlarged views of small corresponding
patches on the STV surfaces highlight the importance of the
123
Int J Multimed Info Retr (2014) 3:15–28
path re-positioning and smoothing process. For example, in
Fig. 6 (I ), it can be seen that axial paths are non-uniformly
spread over the STV surface. But after smoothing they are
re-positioned in an uniformed manner (in part c). This shows
that the simple process of detecting point correspondences
and stacking of shape contours of the foreground blob is not
sufficient to generate a proper STV. This smooth, parame-
terized STV surface is processed further for video content
representation.
3.3 Properties of parameterized STV
In our case, the parameterized STV has been generated using
the set of contours extracted from the segmented foreground
blobs. We consider the STV as a manifold M(x, y, t), which
can be approximated by computing the surface equations for
each small surface patch. We can therefore represent the STV
using a 2-D parametric representation. If we consider the arc
length of the contour as u and the time as v, then we can write
M using this (u, v) parameterization as:
M = S(u, v) = [x(u, v), y(u, v), v] (1)
In this representation, arc-length encodes the object shape
(contour) and time encodes the motion trajectory. With this
parametric representation, fixing the u parameter generates
2D motion trajectories of any point on the object boundary.
Similarly, fixing the v parameter generates the object con-
tours at time v. Parameterized surface (STV) of this kind
motivates us to model the time-varying changes of geomet-
ric properties of surface points, as a time series. In the next
section, we discuss our proposed feature extraction and time
series modeling process.
4 Modeling STV as a time-series
We have modeled video content as a LDS, which represents a
class of parametric models for the time series. This assumes
that the video frames are output of a dynamical system, which
is driven by an IID process. We compute the parameters of
an ARMA model and use them for matching a pair of video
shots.
4.1 Feature extraction from STV
In our approach, the differential properties of STV captures
the video content in terms of “what is moving?” and “how
is it moving?”. These properties are implicitly encoded on
the STV surface and can be analyzed by the surface geom-
etry. These geometric features encode the shape and motion
changes simultaneously.
According to the famous Gauss’s Theorema Egregium
[30], Gaussian curvature is an intrinsic invariant of a surface.
Int J Multimed Info Retr (2014) 3:15–28 21

Fig. 6 Steps of STV creation for three different video shots, having lin- line). The enlarged views of small corresponding patches, given in
ear and non-linear motion trajectories. a Figure shows the segmented between (b) and (c), highlight the result of smoothing and parame-
foreground blob and the motion trajectory (superimposed as dashed terization process for STV generation

The sign of the curvature defines the surface type and the Since, Su · N = 0, partial differentiation with respect to u
value defines the surface sharpness. Let S(u, v) be our para- will yield,
metric STV, and let Su and Sv denote the partial derivatives
∂(Su · N )
of S with respect to u and v. The normal vector to the STV ⇒ Suu · N + Su ·
∂u
is perpendicular to the tangent vectors Su and Sv . Therefore,
Nu = 0 ⇒ Suu · N = −Su · Nu
the unit normal is given by,
The other terms in Eq. (4) can be established in a similar
Su × Sv manner. Therefore, Eq. (4) can be re-written as:
N = N (u, v) = (2)
|Su × Sv |  
−Su · Nu Su · Nv
II = (5)
Gaussian curvature for a parametric surface is defined in Sv · Nu −Sv · Nv
terms of the first fundamental form [30]:
The Gaussian curvature at a point (u, v) on the smooth para-
  meterized STV is obtained by,
Su · Su Su · Sv
I= (3)
Sv · Su Sv · Sv det(II)
KG = (6)
det(I)
and the second fundamental form (also known as the shape
tensor): Figure 7 depicts the Gaussian curvature computed for differ-
ent STVs created using our proposed algorithm. In Fig. 7, the
 
Suu · N Suv · N higher value of curvature is coded with ’red’ and relatively
II = (4)
Svu · N Svv · N smoother (flat) regions are coded in ‘green’. It can be seen

123
22
Fig. 7 Gaussian curvature plotted on the STV generated for four different actions (bend, running, walking and waving two hands) performed by
four different subjects (a–d). Different color represents the value of the curvature (red positive, blue negative). Videos were taken from [38] and
[22] (color figure online)
from the figure that similar classes of content depicts similar
pattern in terms of the structure of the STV. Moreover, results
also reveals that different action sequence has a distinct pat-
tern in the dynamics of Gaussian curvature over the STV
surface. Intra-class similarity and inter-class dissimilarity in
the change of Gaussian curvature over time is present for
different categories of videos. For example, in case of STV
generated from videos where a person is waving both hands,
we can observe that the undulation on the STV surface has a
specific pattern of Gaussian curvature across time, which is
unique to this particular class. At any time instance it repre-
sents the action state of the video object. This motivates us
to see the change of Gaussian curvature over time as a time
series and represent that using parameters of ARMA model.
4.2 Time-series representation of features
As discussed earlier the parametric representation of the STV
surface in an (u, v) space helps in encoding the shape and
motion of the moving object. With such a representation we
can visualize the time varying change of surface curvature
as a time series data. The Gaussian curvature of the points
123
Int J Multimed Info Retr (2014) 3:15–28
on a radial line, i.e., the shape contour (a sequence of u’s),
represents the state of the system as a particular time instance
(v). Therefore, at each point on a given axial line on the STV
surface we measure the change in curvature over time. Thus,
an STV surface can be considered as a dynamical system
and the Gaussian curvature extracted at each point (u, v) on
the STV surface can be written as a function of the current
state as:
statev+1 = φ(statev ), out putv = ψ(statev ) (7)
Here, φ and ψ are linear functions. In this study, observation
of the geometric features on the STV surface is modeled
using the ARMA representation, which is given by:

p 
q
xv = − ai xv−i + bi ev−i (8)
i=1 i=o
The response of the system at sample index v, as denoted
by xv , is expressed as a function of p previous observations
and q residual error terms. p and q are often referred to as
the model orders of the AR and MA processes, respectively.
The weights of the previous observations and residual error
Int J Multimed Info Retr (2014) 3:15–28
terms, ai and bi , are called the AR and MA coefficients,
respectively. It is assumed that the residual error of the
ARMA model, en , is influenced by the unknown excitation
to the structure and is modeled by a white-noise process with
variance 2.
In this work, we have extracted Gaussian curvature at each
point on the parameterized STV surface (see Sect. 4.1). Fea-
tures computed for each radial axis is combined row-wise to
form the feature matrix. Let, F = {f1 , f2 , ..., ft } be the fea-
ture matrix obtained using the sequences of features extracted
from each radial axis. The bold face of the elements of the
matrix F signifies that they are row vectors. Dimension of
each such feature vector is 200 (equal to the number of points
on the shape contour). To model the extracted features, the
time series model parameters need to be identified. We have
adapted the closed form sub-optimal solution discussed by
[17] to extract these model parameter values and hence use
the variant of ARMA model. For each video shot in the data-
base, these model parameters are estimated and stored in the
feature database. The query video undergoes the same fea-
ture extraction process and then those features are matched
with features of all the videos in the database. This matching
is performed using a time series based matching technique,
which is discussed in the next section.
4.3 Time series based STV matching
In [37], it was shown that the parameters of linear dynamic
models are finite-dimensional linear subspaces of appropriate
dimensions. Subspace angle has been defined by [28] as a
metric between a pair of ARMA models. Given two matrices
H1 and H2 , the principle angles between subspaces spanned
by H1 and H2 are denoted by an n-tuple, where n is the order
of the model:
H 1 ∧ H 2 = (θ 1 , θ 2 , ..., θ n ), θ i ≥ θi+1 ≥ 0 (9)
Here, θi is the subspace angles between H1 and H2 . The sub-
space angle based distance (d) between two ARMA models
H1 and H2 of order n, is defined [28] as:
 The UT-tower dataset [12] consists of 108 low-resolution
2
dH 1 ,H2
= −ln cos2 (θi ) (10)
The distance is based on the largest principal angle between
the two models. The value of d is used to match a pair of video
shots. Based upon this match-cost the retrieved videos are
rank ordered and results are displayed. Details of experiments
are discussed next.
5 Experimental setup
We have verified the performance of our proposed CBVR
framework by extensive experimental study, using four video
datasets of real world, static camera video shots. Performance
23
of state-of-the-art algorithms of foreground segmentation on
moving (and unconstrained) camera video shots is far from
satisfactory, while algorithms based on static camera shots
are stable and accurate. This motivates us to experiment using
the video shots only with static cameras. Use of moving
camera video shots must be resolved with more research,
and is beyond the scope of the current work. In this section,
we first describe the datasets on which we performed our
experiments.
5.1 Weizman dataset
We tested our proposed framework on Weizman action
dataset [22]. The dataset contains 93 low-resolution video
shots using a static camera at a frame rate of 50 fps. It has
10 natural actions namely: (i) running (run), (ii) walking
(walk), (iii) skip, (iv) jumping-jack (jack), (v) jump forward
on two legs (jump), (vi) jump in place on two legs (pjump),
(vii) gallop side ways (side), (viii) wave two hands (or
wave2), (ix) wave one hand (wave1), and (x) bend forward
(bend). Each action is performed by nine different subjects.
To increase the number of data samples, we extend the dataset
by augmenting it with the horizontally flipped version of each
sequence.
5.2 KTH dataset
The KTH dataset [33] has six types of human actions,
namely: (i) walking, (ii) jogging, (iii) running, (iv) boxing,
(v) hand waving and (vi) hand clapping. In a given video
shot one action is repeatedly performed by a subject. The
videos were shot in four different scenarios having variation
in scene (indoor, outdoor), scale, and appearance (with dif-
ferent clothes). In total the dataset contains 2,391 sequences,
shot with a static camera with 25 fps frame rate. The average
duration of each video shot is approximately 4 s.
5.3 UT-tower dataset
(360 × 240) video sequences with a frame rate of 10 fps.
It has nine types of actions, (i) pointing to a particular
direction (pointing), (ii) standing, (iii) digging, (iv) walking,
(v) carrying an object (carrying), (vi) running, (vii) wave one
hand (wave1), (viii) wave two hands (wave2), and (ix) jump
in place on two legs (jumping), performed 12 times by six
subjects. All the videos were shot in two different environ-
mental settings: (i) concrete square and (ii) lawn. The pri-
mary challenge is to recognize the content in low-resolution
videos with blurry visual cues. For our experimental purpose
we have used the foreground masks provided along with this
dataset for each video. We performed morphological opera-
tion (erosion and dilation) on the binary masks to get the
123
24 Int J Multimed Info Retr (2014) 3:15–28

Fig. 8 Performance of our proposed algorithm on Weizman dataset (median frames) for two different classes of query, b running, c waving
[22]. a Confusion matrix showing pairwise subspace distance between one hand (wave 1)
different classes of videos, retrieval results showing top eight videos

Fig. 9 Performance of our proposed algorithm on KTH dataset [33]. a Confusion matrix showing pairwise subspace distance between different
classes of videos, retrieval results showing top eight videos (median frames) for two different classes of query, b jogging, c hand clapping

Fig. 10 Performance of our proposed algorithm on UT-tower dataset [12]. a Confusion matrix showing pairwise subspace distance between
different classes of videos, retrieval results showing top eight videos (median frames) for two different queries, b running, c pointing

Fig. 11 Performance of our proposed algorithm on VP Lab dataset [38]. a Confusion matrix showing pairwise subspace distance between different
classes of videos, retrieval results showing top eight videos (median frames) for two different queries, b car, c aerial

foreground blob. We have extracted the contours and the
trajectory using the given foreground masks and then fol-
lowed the parameterization process.
5.4 VPLab dataset
We have also recorded real-world video shots consisting
of different outdoor locations, using a still hand-held Sony
camcorder and have also downloaded videos from internet.
The dataset in [38] consists of five different categories of
video shots, they are: (i) videos shot from the top floor of
the department building (aerial), (ii) bicycle and motorbike
(bike), (iii) moving car (car), (iv) human walking (walk), and
(v) cartoon. All the videos were hand labeled by the authors.
As previously discussed, the assignment was done based on
the dominant content present in the video. For simplicity, in
all the videos used for experimental purposes, there is only
one content in a shot (or clip).

123
Int J Multimed Info Retr (2014) 3:15–28
Fig. 12 Comparative result showing median frames of top six retrieved
video shots for three different queries using. a Liang et. al [25], b Gao
and Yang [21], c EMST-CSS [10], and d our proposed method. The
6 Experimental results
In this section we evaluate, both qualitatively and quantita-
tively, the performance of our proposed framework of CBVR
using four datasets (as described above) and compare it with
three state-of-the-art methods.
6.1 Retrieval performance
We have performed CBVR on all the above mentioned
datasets individually. Figures 8–11 depicts the retrieval per-
formance of our proposed CBVR framework. In all the
four figures, part (a) shows the confusion matrix depicting
the results of pairwise distance between videos of differ-
ent class. Remember here that the distance refers to sub-
space angle based distance between the ARMA parame-
25
three query video shots belong to: Weizman [22], VP Lab [38], KTH
[33], respectively
ters computed from the STVs. Parts (b) and (c) depict the
top eight retrieved videos (only median frame is shown)
for two different queries. Video shots which are wrongly
retrieved are highlighted using a “red” bounding box. It is evi-
dent from the confusion matrices that our proposed method
is able to discriminate (diagonally dominant similarity and
larger off-diagonal terms) among different classes of video
shots.
6.2 Comparative study
We have compared our approach with three baseline meth-
ods of CBVR [10,21,25], using spatio-temporal approach.
Retrieval results are shown for visual comparison in Fig. 12,
for three query video shots from the datasets [12,22,33].
Median frames of the top six retrieved results are shown row-
123
26
Fig. 13 Comparison of retrieval performance using Precision-Recall metric, for four different datasets: a Weizman [22], b KTH [33], c UT-tower
[12], d VP Lab [38]
Fig. 14 Comparison of retrieval performance using F-score metric, for four different datasets
wise in all cases. Wrongly retrieved results are indicated by
a red bounding box. Since ARMA models allows for general
dynamic relationships among variables in a system, it is more
adequate to represent the dynamics of the STV and thus pro-
vides more accurate retrieval results than [10,21,25]. How-
ever, in cases where the action classes are non-orthogonal,
i.e., inter-class similarity is high, ranking of our method is
also prone to error. For example, consider query 3 in Fig. 12
(d), where clapping sequence is misclassified as waving two
hands. This fact has been further emphasized by the quanti-
tative analysis done using the precision-recall and F-measure
metrics.
6.3 Quantitative analysis
For a retrieval system the most useful performance measures
include precision, recall, and F-measure. We computed the
precision-recall, and F-measure values for all three state-of-
123
Int J Multimed Info Retr (2014) 3:15–28
the-art methods [10,25,21] and compared with our proposed
method. Figure 13 depicts precision-recall graphs for exper-
iments on four different benchmark datasets, comparing our
results with [21,25] and [10]. For example, in case of the
videos from [22] (Fig. 13a), our approach (black line with ×
marker) clearly outperforms the other three approaches by
showing higher precision and recall values. Similarly, the
precision curves in Fig. 13b–d, clearly exhibit the superior
performance of our proposed multivariate time series based
approach for CBVR. Since our criteria for better performance
is the quality of ranking, it should be noted that the curves for
the time series based approach show more relevant videos in
the top ranked results.
Figure 14 shows that our approach provides higher
F-measure and outperforms the other three approaches. This
supports the claim that the proposed approach is able to effi-
ciently represent the content and produces superior results of
video matching.
Int J Multimed Info Retr (2014) 3:15–28
7 Conclusion and discussion
We draw upon insights from differential geometry and time
series modeling to propose a framework for CBVR task.
We construct a smooth, parameterized STV from a given
video, shot using static camera, and compute geometric fea-
ture (Gaussian curvature). Then these geometric features are
represented as parameters of an LDS, using an ARMA model.
Similarity between two videos is computed by measuring the
subspace angle based distance between the ARMA parame-
ters. Utilizing the proposed video matching framework, we
have achieved very promising and superior performance.
In future, we plan to explore the possibilities of extending
this framework for moving camera video shots. Foreground
segmentation is an intrinsic part of our proposed framework
and performance of the subsequent steps rely on efficient
foreground segmentation from videos shot using a moving
camera. Further, we plan to explore use of interest point
descriptors based features for video content representation.
Acknowledgments We are thankful to Prof. Sukhendu Das and the
members of Visualization and Perception Lab for their support and
comments. Additional thanks to Debarun Kar and Prahllad Deb for
their insightful suggestions.
References
1. Aggarwal G, Chowdhury A, Chellappa R (2004) A system iden-
tification approach for video-based face recognition. In: ICPR,
pp 175–178
2. Auguste R, El Ghini A, Bilasco M, Ihaddadene N, Djeraba C (2010)
Motion similarity measure between video sequences using multi-
variate time series modeling. In: ICMWI, pp 292–296
3. Babu RV, Ramakrishnan KR (2007) Compressed domain video
retrieval using object and global motion descriptors. Multiméd
Tools Appl 32(1):93–113
4. Barnich O, Van Droogenbroeck M (2011) ViBe: a universal back-
ground subtraction algorithm for video sequences. IEEE Trans
Image Process 20(6):1709–1724
5. Bashir FI, Member S, Khokhar AA, Member S, Schonfeld D,
Member S (2007) Real-time motion trajectory-based indexing and
retrieval of video sequences. IEEE Trans Multiméd 9:58–65
6. Bissacco A, Chiuso A, Ma Y, Soatto S (2001) Recognition of human
gaits. In: CVPR, vol 2, pp 52–57
7. Bobick AF, Davis JW (2001) The recognition of human move-
ment using temporal templates. Trans Pattern Anal Mach Intell
23(3):257–267
8. Brendel W, Todorovic S (2010) Activities as time series of human
postures. In: ECCV, pp 721–734
9. Chattopadhyay C, Das S (2012) A novel hyperstring based descrip-
tor for an improved representation of motion trajectory and retrieval
of similar video shots with static camera. In: Emerging Area in
Information Technology (EAIT)
10. Chattopadhyay C, Das S (2012) Enhancing the MST-CSS represen-
tation using robust geometric features, for efficient content based
video retrieval (CBVR). In: ISM
11. Chellappa R, Sankaranarayanan AC, Veeraraghavan A, Turaga P
(2010) Statistical methods and models for video-based tracking,
modeling, and recognition. Found Trends Signal Process 3:1–151
27
12. Chen CC, Ryoo MS, Aggarwal JK (2010) UT-tower dataset: aerial
view activity classification challenge. http://cvrc.ece.utexas.edu/
SDHA2010/Aerial_View_Activity.html
13. Chen PY, Chen ALP (2003) Video retrieval based on video motion
tracks of moving objects. In: Proceedings of SPIE, vol 5307,
pp 550–558
14. Cui B, Zhao Z, Tok WH (2012) A framework for similarity search
of time series cliques with natural relations. IEEE Trans Knowl
Data Eng 24(3):385–398
15. Das S, Chattopadhyay C, Dyana A (2011) Vidlookup: a web-based
online CBVR system for query video shots. Demo at ICCV
16. Deng Y, Manjunath BS (1998) NeTra-V: toward an object-based
video representation. IEEE Trans Circuits Syst Video Technol
8:616–627
17. Doretto G, Chiuso A, Wu YN, Soatto S (2003) Dynamic textures.
Int J Compu Vis 51:91–109
18. Dyana A, Das S (2009) Trajectory representation using Gabor fea-
tures for motion-based video retrieval. Pattern Recogn Lett 30:
877–892
19. Erol B, Kossentini F (2005) Shape-based retrieval of video objects.
IEEE Trans Multiméd 7:179–182
20. Florez OU, Lim S (2009) Discovery of time series in video data
through distribution of spatiotemporal gradients. ACM symposium
on applied computing
21. Gao HP, Yang ZQ (2010) Content based video retrieval using
spatiotemporal salient objects. In: Intelligence Information
Processing and Trusted Computing (IPTC)
22. Gorelick L, Blank M, Shechtman E, Irani M, Basri R (2007)
Actions as space-time shapes. IEEE Trans Pattern Anal Machine
Intell 29(12):2247–2253
23. Hsieh JW, Yu SL, Chen YS (2006) Motion-based video retrieval
by trajectory matching. IEEE Trans Circuits Syst Video Technol
16:396–409
24. Lee SL, Chun SJ, Kim DH, Lee JH, Chung CW (2000) Similar-
ity search for multidimensional data sequences. In: Proceeding of
ICDE
25. Liang B, Xiao W, Liu X (2012) Design of video retrieval system
using MPEG-7 descriptors. Procedia Eng 29:2578–2582
26. Lin J, Li Y (2009) Finding structural similarity in time series data
using bag-of-patterns representation. In: SSDBM, 2009, pp 461–
477
27. Ma Y, Zhang H (2002) Motion texture: a new motion based video
representation. In: International conference of pattern recognition
28. Martin R (2000) A metric for ARMA processes. IEEE Trans Signal
Process 48(4):1164–1170
29. Meyers D, Skinner S, Sloan K (1992) Surfaces from Contours.
ACM Trans Graphics 11(3):228–258
30. O’Neill B (1997) Elementary differential geometry, 2nd edn.
Academic Press, New York. http://www.apnet.com
31. Popivanov I, Miller RJ (2002) Similarity search over time series
data using wavelets. Proceedings of the 18th ICDE, pp 212–221
32. Rodriguez MD, Ahmed J, Shah M (2008) Action MACH: a spatio-
temporal maximum average correlation height filter for action
recognition. In: Proceedings of CVPR
33. Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions:
a local SVM approach. ICPR
34. Scott C, Nowak R (2006) Robust contour matching via the
order-preserving assignment problem. IEEE Trans Image Process
15(7):1831–1838
35. Srivastava A, Turaga P, Kurtek S (2012) On advances in
differential-geometric approaches for 2D and 3D shape analyses
and activity recognition. IVC 30:398–416
36. Turaga P, Veeraraghavan A, Srivastava A, Chellappa R (2011)
Statistical computations on Grassmann and Stiefel Manifolds for
image and video-based recognition. IEEE Trans Pattern Anal Mach
Intell 33(11):2273–2286
123
28 Int J Multimed Info Retr (2014) 3:15–28

37. Turaga PK, Veeraraghavan A, Srivastava A, Chellappa R (2011) 40. Zhang D, Zuo W, Zhang D, Zhang H (2010) Time series classifi-
Statistical computations on Grassmann and Stiefel manifolds for cation using support vector machine with gaussian elastic metric
image and video-based recognition. IEEE Trans Pattern Anal Mach kernel. In: Proceeding of ICPR, pp 29–32
Intell 33(11):2273–2286
38. VPLab-VID: http://www.cse.iitm.ac.in/vplab/videos.html
39. Yilmaz A, Shah M (2008) A differential geometric approach
to representing the human actions. Comput Vis Image Underst
109(3):335–351

123