Vis Comput
DOI 10.1007/s00371-012-0752-6
O R I G I N A L A RT I C L E
A survey on activity recognition and behavior understanding
in video surveillance
Sarvesh Vishwakarma · Anupam Agrawal
© Springer-Verlag Berlin Heidelberg 2012
Abstract This paper provides a comprehensive survey for
activity recognition in video surveillance. It starts with a de-
scription of simple and complex human activity, and var-
ious applications. The applications of activity recognition
are manifold, ranging from visual surveillance through con-
tent based retrieval to human computer interaction. The or-
ganization of this paper covers all aspects of the general
framework of human activity recognition. Then it summa-
rizes and categorizes recent-published research progresses
under a general framework. Finally, this paper also provides
an overview of benchmark databases for activity recogni-
tion, the market analysis of video surveillance, and future
directions to work on for this application.
Keywords Image processing · Automated surveillance ·
Video tracking · Human activity
1 Introduction
In a traditional surveillance system, it is not possible to de-
tect and prevent a suspicious activity at public places and im-
portant building, etc. The human operator could not prevent
a dangerous situation by keeping close eyes on the mon-
itor screen due to physical fitness, lack of CCTV cameras,
and security forces. Police guards may not keep surveillance
S. Vishwakarma () · A. Agrawal
Indian Institute of Information Technology, Allahabad, India
e-mail: rs51@iiita.ac.in
A. Agrawal
e-mail: anupam@iiita.ac.in
Present address:
S. Vishwakarma
N-297 Third Phase, Shivalik Nagar, BHEL, Haridwar, India
e-mail: sarvesh.vish@gmail.com
continuously. The question arises whether it is impossible to
make public places safe from any type of terrorist activities,
suspicious events. Many scientists and researchers are doing
great research of finding the solution to this question. This
paper discusses the methods and techniques implemented by
researchers to perform the automatic surveillance. In auto-
matic surveillance, a network of computers process the input
video continuously and thoroughly analyze the image frame
to sense any unusual findings, which they can report to the
supervisor for their alertness. The procedure is performed in
three levels: low-, mid-, and high-level image data process-
ing. In modern civilization, the threats of theft, accidents,
terrorist attacks, and riots are ever increasing. Due to high
amount of useful information that can be extracted from a
video sequence, video surveillance has come up as an effec-
tive tool to forestall these security problems. An automated
surveillance system has been involved in experimenting with
video surveillance data to improve the image processing task
by generating more accurate and robust algorithms in object
detection and tracking and human activity recognition.
1.1 Previous survey
This paper identifies a possible research direction in vision
based human action recognition and provides an overview of
framework of visual surveillance in dynamic scene address-
ing following issues, e.g., environment modeling, motion
detection, classification of objects, occlusion handling, fea-
tures extraction, unusual activity, a combination of two- and
three-dimensional tracking, a combination of motion analy-
sis and biometrics, anomaly detection and behavior predic-
tion, content-based retrieval of surveillance videos, behav-
ior understanding and natural language description, fusion
of information from multiple sensors, remote surveillance,
etc. The role of automated surveillance system is to sense

Table 1 Related literature survey summary
First author Yr Topic
Aggarwal 11 Human activity analysis: A review
Daniel 11 A survey of vision-based methods for
action representation, segmentation and
recognition
Ronald 10 A survey on vision-based human action
recognition
Zhan 08 Crowd analysis
Kang 07 Intelligent visual surveillance
Forsyth 05 Computational studies of human motion
Yilmaz 06 Object tracking: A survey
Valera 05 Intelligent distributed surveillance
Weiming 04 Motion and tracking for surveillance
Moeslund 01 Human motion capture
Aggarwal 99 Motion analysis of human body
and monitor unusual activity and alert the human operator
to take action against it.
The objective of this article is to focus on techniques
and methodology used to recognize human activity. Previ-
ous survey in related areas are identified (see Table 1).
This paper provides a comprehensive survey of the recent
developments in human activity analysis. It covers the latest
research paper ranging mainly from 2008 to 2012. It thus
contains many new references not found in previous surveys.
The organization of this paper covers all aspects of the gen-
eral framework of the human activity recognition such as
low-level tasks, intermediate- and high-level tasks and pro-
vides a comparison among few existing datasets. This pa-
per focuses on three main issues of vision related tasks. In
contrast to past surveys, we provide a detailed discussion
on object tracking and their activity related taxonomy. We
also provide more detailed discussions on open issues and
research challenges involved with human activity analysis
than any other earlier reviews.
To discuss the issues related to automated surveillance
more conveniently, we will focus on a more general analysis
system shown in Fig. 1. The remainder of this paper is or-
ganized as follows. Section 2 reviews the work on low-level
task including segmentation and moving object classifica-
tion. Section 3 covers human tracking, which is divided into
six categories of methods: model-based, region-based, ac-
tive contour-based, feature-based, hybrid-based, and optical
flow-based. The paper then extends the discussion to the hu-
man activity recognition in image sequences in Sect. 4. Sec-
tion 5 discuss machine learning methods to represent high-
level activity. In Sect. 6, we discuss various types of datasets
used for the testing of activity recognition methods. Some
surveys of the surveillance market are shown in Sect. 7. Fi-
nally, conclusions are drawn in Sect. 9 followed by open
issues and challenges in Sect. 8.
S. Vishwakarma, A. Agrawal
Ref.
[2]
[204]
[142]
[224]
[76]
[46]
[216]
[187]
[59]
[108]
[1] Fig. 1 A general framework of human motion analysis
1.2 Simple and complex human activity
High-level processing deals with reasoning and perception
to recognize the human activities (suspicious/unusual ac-
tions), and it gets more and more difficult as the complex-
ity level of human activity increases. Figure 2 illustrates an
overview of various types of human activities. Human activ-
ity (according to its complexity) is divided into four levels:
(i) Gesture: Elementary movement of the human body part
defines the gesture. This action is performed in a very
short span of time and its complexity remains low.
Waving a hand, stretching an arm and bending are used
to define the meaning of full motion of the person.
(ii) Action: Actions are the single-person activity where
multiple gestures (atomic actions) are temporarily or-
ganized in the time domain. Running, walking, and
jumping are good examples of action.
(iii) Interaction: For two or more humans and/or objects do-
ing activity, it is called interaction. Human-human in-
teraction, and human-object interaction also come in
this category. Carried/abandon bag, a person stealing a
bag from another, and pointing a gun are examples of
it. These types of human activity are lengthly.
(iv) Group activity: Group activities are the activities per-
formed by two groups composed of multiple objects.
A group of soldiers marching, groups of two parties
misbehaving in parliament, groups of people protest-
ing/fighting are examples of it.
1.3 Applications
In this section, we present a few application areas that will
highlight the potential impact of human activity recognition
systems.
A survey on activity recognition & behavior understanding in video surveillance
Fig. 2 Types of Human
Activities
1.3.1 Behavioral biometrics
Biometrics involves study of approaches and algorithms for
uniquely recognizing humans based on physical or behav-
ioral cues. Traditional approaches are based on fingerprints,
face, or iris, and can be classified as physiological biomet-
rics, i.e., they rely on physical attributes for recognition.
These methods require cooperation from the subject for col-
lection of the biometric. Recently, “Behavioral Biometrics”
have been gaining popularity, where the premise is that
behavior is as useful a cue to recognize humans as their
physical attributes. The advantage of this approach is that
subject-cooperation is not necessary and it can proceed
without interrupting or interfering with the subject’s activ-
ity. Since observing behavior implies longer-term observa-
tion of the subject, approaches for action-recognition extend
naturally to this task. Currently, the most promising example
of behavioral biometric is the human gait [161].
1.3.2 Content based video analysis
Video has become a part of our everyday life. With video
sharing websites experiencing relentless growth, it has be-
come necessary to develop efficient indexing and storage
schemes to improve the user experience. This requires the
learning of patterns from raw video and summarizing a
video based on its content. Content-based video summa-
rization has been gaining renewed interest with correspond-
ing advances in content-based image retrieval (CBIR) [151].
Summarization and retrieval of consumer content such as
sports videos is one of the most commercially viable appli-
cations of this technology [17].
1.3.3 Security and surveillance
Security and surveillance systems have traditionally relied
on a network of video cameras monitored by a human opera-
tor who needs to be aware of the activity in the camera’s field
of view. With recent growth in the number of cameras and
deployments, the efficiency and accuracy of human opera-
tors has been stretched. Hence, security agencies are seeking
vision-based solutions to these tasks, which can replace or
assist a human operator. Automatic recognition of anoma-
lies in a camera’s field of view is one such problem that has
attracted attention from vision researchers [190, 229]. A re-
lated application involves searching for an activity of inter-
est in a large database by learning patterns of activity from
long videos [60, 180].
1.3.4 Interactive applications and environments
Understanding the interaction between a computer and a hu-
man remains one of the enduring challenges in designing
human-computer interfaces. Visual cues are the most impor-
tant mode of nonverbal communication. Effective utilization
of this mode such as gestures and activity holds the promise
of helping in creating computers that can better interact with
humans. Similarly, interactive environments such as smart
rooms [137] that can react to a user’s gestures can benefit
from vision based methods. However, such technologies are
still not mature enough to stand the “Turing test,” and thus
continue to attract research interest.
1.3.5 Animation and synthesis
The gaming and animation industry rely on synthesizing re-
alistic humans and human motion. Motion synthesis finds
wide use in the gaming industry where the requirement is to
produce a large variety of motions with some compromise
on the quality. The movie industry on the other hand has tra-
ditionally relied more on human animators to provide high-
quality animation. However, this trend is fast changing [46].
 S. Vishwakarma, A. Agrawal

Fig. 3 Taxonomy for object Object Detection

detection approaches and the
lists of selected publications    
corresponding to each category
Background Statistical Temporal Optical flow
Subtraction Methods Differencing
Shibata[174]
Bayona[5]
Ince[65]
Haritaoglu[56] Liao[92]
Denman[38]
Kim[79] Collins[29]
Camplani[14] Ishii[67]
Allili[4] Lipton[96]
Sheikh[170] Chen[18]
Elgammal[42] Kameda[75]
Cheng[21] Liu[97]
Lin[94]
Reddy[148] Chen[20]
Wang[199]
Varcheie[188] Qi[144]
Tavakkoli[181]
Kim[81]
Paruchuri[136]
Monnet[111]
Oliver[127]
Tsai[185]
Yamazaki[214]
McHugh[104]
Bucak[11]
Migdal[106]
Li[91]
Paragios[131]
Pilet[139]
Porikli[143]
Vosters[196]
Parameswaran[133]
Zhao[228]

With improvements in algorithms and hardware, much more
realistic motion-synthesis is now possible. A related appli-
cation is learning in simulated environments. Examples of
this include training of military soldiers, fire-fighters, and
other rescue personnel in hazardous situations with simu-
lated subjects.
It attempts to detect, recognize, and track objects of in-
terest from video obtained by camera which may be either
fixed or moving [82]. In aerial surveillance, target sizes are
small and they must be acquired and tracked through chang-
ing terrain, evolving appearance, and frequent occlusion. In
order to extract any useful information about the moving ob-
ject, we need to detect and track them over long durations.
It is usually performed in the context of higher-level appli-
cations that require the location and/or shape of the object
in every frame. The proliferation of high-powered comput-
ers, the availability of high quality and inexpensive video
cameras, and the increasing need for automated video anal-
ysis has generated a great deal of interest in object tracking
algorithms.
2 Motion detection
2.1 Motion segmentation
Motion segmentation in video images is known to be a diffi-
cult problem. The objective of this section is to discuss var-
ious types of approaches involvement to detect the movable
region corresponding to moving objects. The taxonomy of
motion detection in Fig. 3 show the list of selected pub-
lications under each approach. Table 2 presents a detailed
comparison of characteristics and brief description for four
approaches and state-of-the-art methods.
2.1.1 Background subtraction
The background subtraction method detects the target in sta-
tionary scenes, but is sensitive to dynamic scenes, which
keep on changing due to illumination and extra events. There
is need to apply morphological operation after performing
raw subtraction and thresholding.
2.1.2 Statistical
The statistical method is a process of building a more ad-
vanced background subtraction by applying a statistical ap-
proach on individual pixels or a group of pixels. This is pop-
ular due to the robustness of noise, shadow, changes in light-
ing conditions, etc.

Nearly every human activity recognition system starts with
human detection. The task of human detection is to segment
regions corresponding to the persons from the rest of an im-
age. It is a significant issue as subsequent processes such as
tracking and action recognition stages are dependent on it.
This process usually involves segmentation and object clas-
sification. There are six conventional approaches to moving
object detection: background subtraction, statistical meth-
ods, temporal differencing, and optical flow.
2.1.3 Temporal differencing
To identify the moving target across a sequence of image
frames, the current image frame is subtracted either by the
previous frame or the next frame of the image sequences.
This is termed as temporal differencing. Thresholds are ap-
plied to the difference images to eliminate pixel changes
due to camera noise, small illumination changes, etc. This
method is very adaptive to the dynamic scene, but not fine
for extracting all relevant feature pixels.
A survey on activity recognition & behavior understanding in video surveillance

Table 2 Performance details of some object detection techniques

Ref. Model Scene Spectral Camera Detect Remarks

outdoor(o) color(c) single(s) single(s)
indoor(i) grayscale(g) stereo(t) multiple(m)
multiple(m) group(g)

[56] blob-based BGS o g s m handle occlusion

[14] mixture of Gaussians i,o g,c m s,m tolerate varying background/gradual
illumination changes
[170] pixel-wise o c s s –
labelling/trajectory removal
[21] discrete wavelet transform i,o g s m not suited for clutter environment
[148] patch by patch texture i,o c s s,m deal with noise, illumination, dynamic
descriptor background
[188] feature histogram i c s s varying illumination
[81] salient motion detection o g s m accuracy matter
[111] autoregression i,o g s m handle position change in light source and
local/global illumination changes and fail to
detect moving object in same direction of
wind flow
[185] independent component i g s s computational fast, significant illumination
analysis changes
[104] foreground adaptive o g s s,m significant to illumination changes
[106] Markov random fields o c s s global illumination changes, complex motion
[131] topology free MDL-splitting i g s s computationally efficient, handle
HMM sudden/gradual global illumination
[143] dual foreground i,o c s s robust to illumination changes, tolerate
occlusion, noise, difficulty to adjust parameter
[79] single general Gaussian i,o c s m low/static background
[4] mixture of general Gaussian i c s g sudden/or shadow
[42] kernel density estimation i g – s little/background
[94] support vector machine i,o g s s –
[199] support vector regression o c s m tracking-by-detection and slow computation
[181] support vector data i g s s low contrast and quasistation background
description
[136] support vector regression i c s s change in position of illumination source and
sudden change in illumination
[127] principle component analysis o g s s varying illumination, weather
[214] independent component i c s s partial illumination changes
analysis
[11] independent non-negative o g t m sudden illumination changes
matrix factorization
[91] incremental rank o c,g s s,g illumination changes, shadow movements
(R1 , R2 , R3 ) tensor
[139] mixture of Gaussians i,o c,g s s,m local/global illumination changes
[196] eigenbackground based i c s s sudden illumination changes
statistical illumination
[133] illumination transfer o g s g local/global illumination changes and low
function computation
[228] candid covariance o g s g extreme illumination changes (in night)
incremental (PCA) capture tiny appearance
[5] sub-sample frame i g m m support partial/total occlusion, detect
differencing stationary foreground region
[92] mask sampling + hough i c s g extreme illumination changes (in night)
transform capture tiny appearance
 S. Vishwakarma, A. Agrawal

Table 2 (Continued)

Ref. Model Scene Spectral Camera Detect Remarks

outdoor(o) color(c) single(s) single(s)
indoor(i) grayscale(g) stereo(t) multiple(m)
multiple(m) group(g)

[29] adaptive BGS + three frame o c t s accommodate slow lighting changes, noise in
differencing imagery
[96] temporal differencing + o g s s partial occlusion, cluttered background,
image template matching ambiguous pose
[75] double difference image o g s s computationally slow, less accurate
[174] optical flow distortion i c t s static/moving object detection
[65] extrapolation of two images i c s m detect occlusion
[38] optical flow on pixel o c s s improved speed/accuracy, detect occlusion
resolution between objects
[67] gradient-based + FPGA i c s s investigation of top’s spinning and a human’s
pitching motion
[18] frame straddling + FPGA i,o c s s less computation cost, high speed, less delay
involve
[97] coarse-to-fine + space o c s s detect texture among similar structures
feature
[20] optical flow context i,o c s m,g suited for fight scene
histogram
[144] Horn–Schunck method o g s s suited for infrared images

2.1.4 Optical flow
The optical flow method is used to detect moving human
being or objects independently based on their individual ve-
locities from the camera installed on movable platform. This
method is computationally data intensive and took more
time to segment the foreground objects from the scene and
that is why there is a need to use specialized hardware (such
as FPGA, GPU cards etc.) to beat the slow computation. The
optical flow method is generally used as a feature for both
object detection and object tracking. This feature is known
as the optical flow vectors. Optical flow vectors are used to
represent apparent velocities of movement of brightness pat-
terns in an image. Optical flow can give important informa-
tion about the spatial arrangement of the objects viewed and
rate of change of this arrangement. Discontinuities in the op-
tical flow can help in the segmentation of the image frame
into the object region. As the object is detected by their fea-
tures and tracked by matching the same features in subse-
quent image frames. The optical flow method is suitable for
object tracking that we will discuss in more detail in Sect. 3.
2.2 Object classification
Object classification refers to the task of an automatically
distinguished moving target of interest from other moving
objects such as vehicles, flying birds, clouds, etc. across suc-
cessive frames in an image sequence. To classify the object
is an essential process to further track and analyze their be-
haviors for high-level tasks. The small size of the object
and low resolution imagery occurs for the far field cam-
era view. Several classification methods have been proposed
in the literature. The main categories that we identify are:
shape-based classification and motion-based classification
or a combination of both. Shape-based classification meth-
ods classify moving objects by identifying a shape informa-
tion such as representation of point, box, blob, and silhou-
ettes of motion regions in the image. Motion-based methods
use periodic property of moving entities to categorize both
rigid and nonrigid articulated motion. In this section, we
have provided an overview of some of the selected papers
on object classification techniques. Table 3 shows the per-
formance details of some classification techniques for shape
and motion-based methods.
3 Techniques used for object tracking
Object detection and tracking certainly are important in the
captured video frames/images. These captured videos con-
tain a strong parallax. Actually, parallaxes contain rich in-
formation about the 3D nature of the scene as well as mov-
ing ground objects. But still it creates a challenge to extract
meaningful information for scene understanding and object
tracking. A geo-referenced depth estimation algorithm [210]
is used on input aerial video sequences to compute the
depth from the video frame and the whole video/images are
registered with respect to one global reference and gener-
ate corresponding aligned images. Then based on the es-
timated depth, nonground regions are segmented and then
A survey on activity recognition & behavior understanding in video surveillance

Table 3 Performance details of some object classification techniques

Study Method Classification Classifier tools Classification metrics Remarks

Collins et al. [29] shape single human, vehicles, neural network image blob area, dispersedness, classifier results kept in
human groups, clutter aspect ratio of blob bounding box histogram
Lipton et al. [96] shape human, vehicles, clutter neural network image blob area, dispersedness temporal consistency was
used for preciseness
Kuno et al. [83] shape human, vehicles, silhouette/shape shape parameter of human mean/standard deviation of
butterflies silhouettes pattern shape parameters
Cutler & Davis [32] motion human self-similarity, periodicity of moving objects –
time-frequency
analysis
Lipton [95] motion human, vehicles optical flow residual flow of rigidity, –
periodicity of moving objects
Stauffer [179] motion human, vehicles time color, velocity –
cooccurrence
matrix
Mohan et al. [110] motion component of human haar wavelets – –
body(head, legs, left arm,
right arm)
Zhange et al. [227] shape person, car, van, people multi-block local visual features work well for small size
groups, truck, bug binary pattern objects or low resolution
video
Javed & Shah [71] motion pedestrian, vehicles Recurrent size, compactness, aspect ratio, work well for restricted
Motion Images simple descriptor of shape or settings
motion
Bose & Grimson [9] motion human, vehicles scene-invariant image position & direction of fine for low resolution
classification motion of objects imagery & projective
image distortion
Brown [10] motion human, vehicles two-phase size, speed, color, texture just recognize human from
classification vehicles
Ma & Grimson [103] motion types of vehicles edge point sift repeatable/discriminative not recognize vehicle
descriptor features identity, not fine underview
changes, occlusion
Tsuchiya & Fujiyoshi shape single human, vehicles, adaboost shape, texture, speed computationally
[186] motion human groups, bike inexpensive, invariant to
lighting condition or
viewpoint

the Planar fitting plus depth extension approach is applied
to extract the structure of buildings, trees’ shape, moving
car/vehicles, pedestrians, water tanks, etc. Due to varying
image quality, the estimated depth map is highly variable.
To obtain a consistent depth map over the video frame se-
quences, the bilateral depth fusion techniques are used to
refine the depth map by fusing low quality depth informa-
tion [114]. There are great variety of object tracking ap-
proaches found in published literature. The best known and
most successful algorithms include model-based tracking
using geometric models, sum of square differences of opti-
cal flow, Bayesian random-sampling techniques [115], par-
ticle [226] and Kalman filtering [43], adaptive background
subtraction method [101], multiple hypothesis tracking ap-
proaches, affine image registration and local motion estima-
tion, and mean shift tracking algorithms [8]. To achieve a
real-time performance of Motion tracking, Jing Huang et al.
have described a technique based on GPU-accelerated com-
putation using the CUDA framework [61]. In the majority
of tracking applications, the measurements from the scene
of interest are acquired at a discrete time referred to as
scans [15]. The measurements are then used to deduce the
information about the objects being tracked, i.e., the targets.
The measurements may originate from targets, other ob-
jects, or just false detections in noisy signals. All unwanted
measurements are usually referred to as clutter. The task of
tracking objects as they move in substantial clutter, and to
do it at, or close to, the video frame-rate is challenging as
in most of the severe cases; the background may consist of
objects similar to the foreground object(s), e.g., when a per-
son is moving past a person, a group of people, or a crowd.
Figure 4 depicts the taxonomy for object tracking. In [114],
tracking of the moving object method can be divided into
six major categories:
 S. Vishwakarma, A. Agrawal

Object Tracking

     
Region-based Contour-based Feature-based Model-based Hybrid Optical flow-based
Tracking Tracking Tracking Tracking Tracking Tracking
Ladikos [84]
Wunsch [208] lalos [85] Schunck [166][165]
Meyer [105] Techmer [182] Coifman [28]
Davis [48] Girisha [51]
Fazli [43] Yokoyama [220] Chai [16]
Zhu [231] Denman [36][37]
Schmaltz [162] Yilmaz [217] Jang [70]
Ong [129] Ince [65]
Salembier [160] Chiverton [23][24] Zhou [230]
Vlasic [194] Sakaino [159]
Kim [80] Cai [12] Comaniciu [30]
Cheung [22] Senst [168]
Sclaroff [167] Qiang Chen [19] Ren [149]
Schmaltz [163] Paragios [130] Wen [205]
Niethammer [121] Lin [93]

Fig. 4 Taxonomy for Object Tracking approaches and the lists of selected publications corresponding to each category

a. Region-based tracking
b. Contour-based tracking
c. Feature-based tracking
d. Model-based tracking
e. Hybrid tracking
f. Optical flow-based tracking
3.1 Region-based tracking
These algorithms track objects according to the variation of
image regions corresponding to the moving objects. Meyer
and Bouthemy [105] had tried to track shape and teh position
of the object in successive frames using the recursive algo-
rithm in complex outdoor scenes. This algorithm was based
on the first-order motion model. In [43], the motion regions
are usually detected by subtracting the background from the
current images. Schmaltz et al. [162] focused on the separa-
tion of the object from its background using a localized mix-
ture model by partitioning the foreground and background
into several subregions. Salembier and Marques [160] have
discussed various types of algorithms that were used to
convert pixel-based representations into region-based repre-
sentations for multimedia applications. Jong Kim and Joon
Kim [80] pursued another route to segment a moving ob-
ject in a traffic scene. They first detected the position of the
moving object in a scene by applying adaptive threshold-
ing. Then the segmentation of the moving pixel was ob-
tained using the k-mean clustering algorithm. In Sclaroff
and Isidoro’s work [167], both the appearance and shape
of the object were integrated and processed in a color tex-
ture map and triangular mesh, respectively, to track nonrigid
motion. The recent work by Schmaltz et al. [163] focused on
occlusions and the self-occlusion problem found in multiob-
ject tracking. They attempted to handle occlusions and self-
occlusions using the probability density function by tracking
multiple objects and object parts simultaneously.
3.2 Contour-based tracking
In this method, instead of tracking the whole set of pixels
comprising an object, the algorithm tracks only the contour
of the object [182]. An interesting edge-based feature ap-
proach for detecting and tracking non-rigid objects is pro-
posed by Yokoyama and Poggio [220]. It assumes that the
image intensity of time t and t + dt would remain same.
They extracted the contour of the moving object in four steps
(line restoration, line-based background subtraction, cluster-
ing, and active contour). It works in real-time. A contour-
based nonrigid object tracking method via the contour en-
ergy function was proposed by Alper Yilmaz et al. [217]
where they tracked the complete region of the nonrigid ob-
jects and recovered the occluded objects parts. Chiverton
et al. [23] used shape-based level set active contour frame-
work to segment and track objects, which undergo frequent
shape deformation by defining the outline of the object in
first frame. In another paper, the same authors proposed the
concept of finite-sized shape memory to continuously re-
member relevant shape information and automatically elim-
inate unnecessary shape information [24]. The object was
tracked through the sequences under low contrast scenes by
integrating both region and boundary features, using the al-
gorithm proposed by Ling Cai et al. [12]. Based on kernel
and active contour, Qiang Chen et al. [19] located the object
in complex situations, and precisely tracked the object con-
tour. Paragios and Deriche [130] applied a geodesic active
contours to detect and track moving objects in a sequence
of images. They determined detection and tracking bound-
aries using probabilistic edge detectors. Level sets theory
was applied to reduce the computational cost. Niethammer
et al. [121] used the geodesic active contour originated by
Paragios and Deriche [130], in which the state information
(normal velocity) with every particle on a contour is repre-
sented by means of a level set functions.
3.3 Feature-based tracking
In this section, we discussed feature-based object tracking
involving both static features and dynamic features. The
static features are extracted from the appearance of the ob-
ject, such as color, geometry, texture, and image template
while the dynamic features are derived from the mathemat-
ical tools such as particle filter, Kalman filter, and Bayesian
A survey on activity recognition & behavior understanding in video surveillance
Fig. 5 Feature-based tracking decision conceptual diagram
network, etc. The study of motion and behavior are the two
important elements of dynamic features. Figure 5 shows
a conceptual diagram of a feature based tracking model.
Coifmann et al. [28] worked on a video image processing
system to detect and track passengers’ cars, motor cycles,
buses, and trucks in real-time. They took corner points of
vehicles as a relevant feature to track it using the Kalman
filter. This makes the system less sensitive to partial oc-
clusion. YoungJoon Chai et al. [16] carried out research
tracking objects using the particle filter based on an in-
tegral histogram to the kinematics chain model. Jang and
Choi [70] applied the greedy algorithm to minimize the
energy function to monitored shape, size, and color fea-
ture. They tracked the human in different directions while
preventing it to get mixed-up with other persons. Shaohua
Kevin Zhou et al. [230] proposed a particle filter based adap-
tive state transition paradigm for tracking with the following
properties: handle appearance changes between frames and
between frame and gallery images, occlusion analysis is em-
bedded in the particle filter, and it stabilizes the tracker by
embedding linear prediction into stochastic diffusion. Dorin
Comaniciu et al. [30] used a Kalman filter framework based
on the color histogram in conjunction with a kernel-based
target localization method for tracking human beings in the
subway. Ren and Chua [149] employed spatial and color
information of the target object into the expectation max-
imization (EM) framework to tackle illumination changes
and the reflection of shadow problems effectively. Zhiqiang
Wen and Zixing Cai [205] used the “weighted color kernel
histogram” approach to eliminate background pixels from
the object model. Then the mean shift algorithm was ap-
plied to track the object robustly against the occlusion in an
outdoor environment. Feng Lin et al. [93] developed an ap-
proach for the target tracking system using a set of static and
dynamic features. The system starts to track the target with
its color and shape and after that it associates motion fea-
tures and controls their values using the discriminate func-
tion for robust target tracking in a complex environment.
3.4 Model-based tracking
Models are usually constructed of-line with manual mea-
surements or computer vision techniques. As models have
more complex structures, the more expensive will be the
computation. Strictly speaking, model-based tracking is an
example of feature-based tracking. The reason why it is in-
dependently described is due to the requirement of group-
ing, reasoning, and rendering, which may defer it from the
feature based tracking. In addition, prior knowledge about
the investigated models is normally required. For example,
in the case of multiple objects tracking the binary repre-
sentation (models) of the targets must be obtained a priori.
This may be followed by applying a stage of model recog-
nition. The Hough transform was utilized to achieve a sim-
ilar idea in [208]. Model-based tracking schemes share the
same challenges as the feature-based trackers do. For exam-
ple, occlusion is a significant cause of instabilities, result-
ing in poor tracking performance. To track human motion
in activities such as walking, running, and jumping, Gavrila
and Davis [48] matched edges in the image with those of
an appearance model using distance transforms. A decom-
position approach and a best-first technique were used to
search through the high dimensional pose parameter space.
A robust variant of Chamfer matching was used as a fast
similarity measure between synthesized and real edge im-
ages. Youding Zhu et al. [231] reconstructed the 3D pose
for humans with the help of time-of-flight data to track them
reliably in a complex environment. Ong and Gong [129]
adopted the condensation algorithm to track the 3D skele-
ton of the human body in their dynamical linear framework.
Daniel Vlasic [194] estimated the skeleton pose to obtain the
closest mesh point on contour and captured the details geo-
metric information about the skeleton and shape. They have
achieved high quality tracking by providing full correspon-
dence and correct topology in an animated mesh structure.
Kong-man Cheung et al. [22] used a marker and sensor on
the human body to capture their skeleton motion. They ex-
perimented over skin tight cloth and manual adjustment so
that markers rigidly do movement corresponding to limbs.
3.5 Hybrid tracking
In Alexander Ladikos et al. [84], the authors used both
the template and feature-based approaches to build a real-
time tracking model. It tracked small patches on the mov-
ing object in order to cope with illumination changes and
partial occlusions. Constantinos Lalos et al. [85] designed
a hybrid approach by combining region and feature-based
techniques. They tracked moving objects using the Rao-
Blackwellized particle filter in order to provide an improved
description and extract initial information for undergoing
events. HOG parameters were studied for a moving object
by a set of two thousand extracted patches.
3.6 Optical flow-based tracking
Optical flow is the vector field which describes how the
image changes with time. The two-dimensional projec-
 S. Vishwakarma, A. Agrawal

tion from the three-dimensional velocity field observed by
the camera needs to be estimated. However, this has been
proved to be extremely difficult to achieve due to prob-
lems such as the aperture effect. To find the optical flow
in the image sequence, people attempt to use feature-based,
gradient-based, or correlation-based approaches. Most these
approaches are of intensive computation, and hence com-
putational optimization is demanding. Optical flow meth-
ods are normally used for generating dense flow fields by
computing the flow vector of each pixel under the bright-
ness constancy constraint [166]. This computation is often
carried out in the neighborhood of the pixel either alge-
braically [100] or geometrically [165]. Extending optical
flow methods to compute the translation of a rectangular
region is somewhat achievable. One of the examples was
reported in [172], where Shi and Tomasi proposed the well-
known ShiTomasiKanade (STK) tracker which iteratively
computes the translation of a region centered on an interest
point. Once the new location of the interest point is obtained,
the STK tracker evaluates the quality of the tracked patch by
computing the affine transformation. This scheme works ef-
fectively and is fast in most circumstances. Further work is a
need of reducing the incorrect point correspondence. Girisha
et al. [51] calculated optical flow from the silhouettes region
of a moving human using the Two Way ANOVA method. Si-
mon Denman et al. [36] proposed a system in which optical
flow calculation is done in adaptive background segmen-
tation using the YCbCr image format unlike the gray scale
image. They analyzed only those pixels of images, which are
in motion rather than the whole images. This helps to avoid
unnecessary computation of CPU time and capable of do-
ing horizontal and vertical movement in real-time. In [37],
the same authors provided a feedback mechanism to add
additional information into the existing object tracking al-
gorithm. Serdar Ince et al. [65] worked on the problem of
spatial discontinuities by detecting occlusions and extrapo-
lating optical flow in an occluded area. They used more than
two frames rather than two consecutive frames to exploit
the temporal correlation. Hidetomo Sakaino [159] focused
to solve a large displacement of semitransparent objects be-
tween frames using a two- step optimization of the optical
flow estimation model. A Gaussian mixture motion model
based on optical flow method is presented in [168]. The au-
thors implemented shot time and long time speed and di-
rection independent motion descriptor to detect and track
carried baggage in changing light conditions.
4 Techniques used for action recognition
Human activity recognition was classified by Agrawal and
Ryoo [2] into two categories: Nonhierarchical and hierar-
chical approach based on the direct or indirect recogni-
tion of human activities from the input video. Based on
the interpretation, the nonhierarchical approach is divided
into two classes: space-time approach and sequential ap-
proach and each of them further classified into three and two
classes, respectively. Similarly, the hierarchical approach
can be classified into three categories: statistical, syntactic,
and description-based approaches. Figure 6 shows a detailed
taxonomy used for human activity recognition approaches
covered in this section, together with a number of publica-
tions corresponding to each category.
4.1 Nonhierarchical approaches
It covers the simple and short activities such as primitive ac-
tion and periodic activities (running, jumping, waving, etc.).

Human Activity Recognition

Non-hierarchical Hierarchical

  
Space-time Sequential Statistical Syntactic Description-based

    
Ivanov [68]
Nguyen [119] Joo [74]
Pinhanez [140]
Space-time Trajectories Space-time Exemplar State-based Oliver [128] Moore [112]
Gupta [55]
Shi [173]
Volume Features Yu [221]
Minnen [107]
Intille [66]
Darrell [35] Vu [197]
Yamato [213] Damen [34]
Bobick [7] Campbell [13] Gavrilla [48] Ghanem [49]
Zelnik-Manor [223] Starner [178] Cupillard [31]
Shechtman [169] Rao [147] Yacoob [212] Ryoo [152]
Laptev [87] Vogler [195] Gong [52]
Rodriguez [150] Yilmaz [218] Efros [41] Ryoo [155]
Gilbert [50] Bobick [8] Zhang [225]
Ke [77] Sheikh [171] Lublinerman [99] Siskind [175]
Schuldt [164] Oliver [127] Dai [33]
Gorelick [53] Khan [78] Veeraraghavan [191] Nevatia [118]
Dollar [39] Park [135]
Oikonomopoulos [125] Jiang [73] Nevatia [117]
Laptev [89] Natarajan [116]
Vaswani [189] Ryoo [153]
Ryoo [156] Moore [113]
Peursum [138] Ryoo [154]
Chomat [25]
Niebles [120] Gupta [54]
Wong [207]
vishwakarma [193][192]

Fig. 6 Taxonomy for human activity recognition approaches and the lists of selected publications corresponding to each category
A survey on activity recognition & behavior understanding in video surveillance
Fig. 7 Space-time approaches
It recognizes the human activity from unknown image se-
quences by running the matching algorithm on it with ref-
erence to the training set containing the predefined activity
class.
4.1.1 Space-time
The space-time approach represents activity in volume, tra-
jectories, and set of features. The algorithm is used to cor-
rectly match input with their representative model to recog-
nize the activity class. As shown in Fig. 7, it is clear that
activity recognition can performed by matching similarity
in shape and appearance through any of the three types of
models.
Space-time volume Bobick and Davis [7] focused to rec-
ognize activity in real-time by using two components of
vector images: MEI and MHI. For unknown video frames,
it first constructs a vector image and then it is matched
against a stored representation of known movements. It
works successfully in static background where the motion
of object movement can be separated easily. Instead of re-
constructing a three-dimensional model of a person, it de-
scribes the motion spatially by using motion-energy image
and then investigate its motion using a motion-history im-
age. Prior foreground/background segmentation is required
in this approach. It represents and recognizes simple action
like sitting, arm waving, and crouching, etc. Shechtman and
Irani [169] avoid explicit flow computation by employing
a rank-based constraint directly on the intensity information
of spatio-temporal cuboids to enforce consistency between
a template and a target. It is used to detect “behaviors of in-
terest” in a video database and checked on the Weizmann
dataset. It detects similar behaviors and activities in video
sequences despite differences in appearance due to differ-
ent clothing, different backgrounds, and different illumina-
tion, etc. No prior modeling or learning activities are needed.
It can handle video sequences of complex dynamic scenes
such as a dynamic event that contains an unstructured ob-
ject like running water, flickering fire, and complex ballet
movements, etc. It is not restricted to a small set of prede-
fined activities and can bear small template deformations.
Their system was able to detect different types of multi-
ple activities that occur simultaneously in a camera view
field in the presence of cluttered dynamic backgrounds. Ro-
driguez et al. [150] devised a method in which the MACH
filter is operated on spatiotemporal volume and vector val-
ued data. It lowers the computational cost by analyzing the
frequency response of this filter. Yan Ke et al. [77] fused
shape and flow based techniques for the matching interest of
video. It does not require a background model. Gorelick and
Blank et al. [53] represented actions as space-time shapes,
which contained both spatial and dynamic information. This
method was fast and did not require prior video alignment.
A reliable performance can be achieved in action recogni-
tion.
Trajectories In 1995, Campbell and Bobick published a
paper [13] in which they recognize nine atomic movements
of a ballet dancer by tracking trajectories of joint positions
in a 3-D XY T plane. Rao and Shah [147] proposed a view-
invariant trajectory matching method to learn human actions
without any training model and to avoid ambiguity of action
and their trajectories. In [218], Yilmaz and Shah constructed
spatiotemporal volume by stacking a contour of object re-
gions in consecutive frames. The check points of the contour
were tracked in 4-D XY ZT plane. Sheikh et al. [171] pro-
posed a matching algorithm that worked on a set of 13 joint
trajectories in a 4-D XY ZT space. This technique has rec-
ognized simple actions (i.e., sitting, standing, and dancing)
despite changes in viewpoint, anthropometry, and execution
rate, while Khan et al. [78] targeted to recognize complex
behavior in group activities (e.g., people parades) through a
3D polygon. Every individual of the group was represented
by corner points of the 3D polygon. Each individual entity
was tracked in consecutive frames and recorded to get corner
point trajectories in a 4-D XY ZT space. Oikonomopoulous
et al. [125] introduced a representation of human action as a
collection of short trajectories that are extracted by a parti-
cle filtering tracking scheme in space and time. They used a
longest common subsequence algorithm to compare differ-
ent sets of trajectories corresponding to different actions.
Space-time features Space-time feature is defined as lo-
cal features of 3D volume on a space-time scale. Local
features often serves as a good approximation to represent
and recognize human activities, as the methods based on
space-time features assume that the 3-D space time vol-
ume is a rigid 3-D object, and it describes each action 3D
volume by performing an object matching procedure. The
space time feature provides s concise description of hu-
man action’s 3D volume by solving the object-matching

issues. Zelnik-Manor and Irani [223] proposed an event-
based distance measurement approach in which they uti-
lized local features at multiple temporal scales. It does not
require any prior knowledge about the event model and
background segmentation. Due to intensity gradient oper-
ation on multiple temporal scaled video volumes, this sys-
tem shows inferior accuracy while increasing the complex-
ity of the video. It is not useful for recognition of multi-
activities in video. Laptev and Lindeberg [87] proposed a
generalization of Harris and Forstner interest point detec-
tor to localize the compact representation of the event. Sim-
ilarly, Gilbert et al. [50] used a 2D Harris corner detec-
tor and data mining approach to localize multiple action in
scale-invariant real-time. But this required a lot of training
samples. Furthermore, Schuldt et al. [164] classified multi-
ple actions by applying SVMs to [87] and illustrated their
impact on activity recognition. A new database called the
“KTH actions dataset” containing action videos (e.g., jog-
ging and hand waving) was introduced, and has been widely
adopted. Dollar et al. [39] proposed a new method “spatio-
temporal feature” as the cuboids prototype for the recog-
nition of human (and animal) actions. It does not support
view-invariancy and multi-activities. Chomat et al. [25] pro-
posed a method for recognizing human actions directly from
image measurement. They modeled a segment of video as
a (x, y, t) spatio-temporal volume and computed local ap-
pearance models at each pixel using Gabor filter at vari-
ous orientation, spatial scale, and temporal scale. Niebles
et al. [120] also based on the interest point feature extrac-
tion approach and recognized multiple activities in a single
video sequence, robust to scale changes, and their localiza-
tion. But it does not support view invariancy as it models an
action class by clustering cuboids into a set of video code
words. As the recognition of a simple action does not re-
quire a spatial-temporal configuration in the local features
extracted from 3D space-time volume. Wong et al. [207]
captured the global information (spatiotemporal location)
and sparse set of interest points by constructing a pLSA
with an implicit shape model (pLSA-ISM). In contrast to
the pLSA used by [120], their pLSA-ISM captures the rel-
ative spatio-temporal location information of the features
from the activity center, successfully recognizing and lo-
calizing activities in the KTH dataset. They reliably recog-
nized actions and gestures, but it is not suited for multiple
action recognition. In a slightly different approach, Laptev
et al. [89] considered a 3D XY T space as a sequence of
grids, where each grid consists of the portion found in the
spatiotemporal histogram. This approach recognizes natural
human action, but has a less tolerance toward noise. It has
been checked on realistic videos (e.g., movie scenes) similar
to [150]. Ryoo and Aggarwal [156] presented an algorithm
to estimate structural correspondence between two contin-
uous videos. Their algorithm is modeled by the spatiotem-
S. Vishwakarma, A. Agrawal
poral relationship match. It can recognize complicated hu-
man activities such as interaction between human-human,
human-object, and human-object-human. In [193], the au-
thors have considered multiclass activities fused in a three-
dimensional (spatial and time) coordinate activity recogni-
tion system to achieve maximum accuracy. In a recent work
of Vishwakarma and Agrawal [192], they quantized feature
vectors of interest points utilizing a histogram. Their ap-
proach delivered the same performance with a less number
of features. It worked well in semantically varying events
and was robust to scale and view changes.
4.1.2 Sequential
Sequential approaches recognize human activities by ana-
lyzing the sequences of features extracted from input video
frames. These sequence of features are called observation
sequences for a particular class of activity in videos. The
idea here is to identify similarity measures associated with
extracted sequences observation in an input video.
Exemplar Recognition based on exemplar approaches aim
at directly extracting the sequences of feature vectors from
input video, and then compare it with template sequences
using an algorithm (e.g., dynamic time warping).
Darrell and Pentland [35] proposed a DTW algorithm
and used a view model to recognize gesture actions and ef-
fectively handle a variation in the execution of actions. In
their work, “hello,” “good-bye,” and “come closer” gestures
were recognized successfully. Gavrila and Davis [48] pro-
posed a 3D joint angle model and associated it with the
DTW algorithm to recognize human movements at a ges-
ture level. (e.g., waving hand for hello gesture, indication
for come to near, etc.) Yacoob and Black [212] have consid-
ered a model in which recognition of different actions were
treated by different type of features (set of eigenvectors).
They used PCA-based modeling and singular value decom-
position technique to represent activities. Efros et al. [41]
presented a methodology (motion descriptor along with op-
tical flow) to track people’s activities at public places where
each person’s height was around 30 pixels. Lublinerman
et al. [99] detected a simple action using a linear time vari-
ant system. Their system was sensitive to noise and per-
formance, which depend on the correctness of background
subtraction. Veeraraghavan et al. [191] developed a model
that was the extension of the DTW matching algorithm and
included a time function parameter to monitor execution
of action speed over a time. This model helped to repre-
sent pickup, throwing, pushing, and waving actions while
maintaining variation in intra and interperson speed. Jiang
et al. [73] used a geometrical model of local human part to
characterize an action as a sequence of postures and then ap-
plied a sequence matching method for action recognition. In
[189], Vaswani et al. uses non rigid shapes and a dynamic
model to represent activity trajectories.
A survey on activity recognition & behavior understanding in video surveillance
State-based State based approaches are the sequential ap-
proaches, which represent a human activity as a model. The
model is trained statistically corresponding to feature vec-
tors extracted from a particular class of activity. Yamato
et al. [213] adopted a hidden Markov model (HMM) to rep-
resent and recognize the activities. Originally, the HMMs
have been widely used for speech recognition. Starner and
Pentland [178] also used standard HMMs, and further ex-
tended its application in gesture level action recognition.
In order to recognize American sign language (ASL), they
have modeled each word of it as HMMs, and generated
a sequence of features. This sequence of features actually
described location and the shape of a hand. However, in
this HMM, it was not designed to handle a large num-
ber of combinations to represent American sign language.
Vogler and Metaxas [195] employed a concept of parallel
HMMs (PHMMs) to lower the number of combinations used
in recognition of American sign language (ASL). Bobick
and Wilson [8] also recognized gestures using state mod-
els. Here, gesture was represented as a 2-D XY trajectory
describing the location changes of a hand. Each curve is de-
composed into sequential vectors, which can be interpreted
as a sequence of states computed from a training exam-
ple. Oliver et al. [127] proposed a new concept of varia-
tion in HMM and introduced a couple of hidden Markov
models for modeling the complex individual activities. The
traditional HMM could not able to recognize the interac-
tion between more than two people due to large size of
the parameter set. Experimental results demonstrated that
CHMMs are better than HMMs from the learning point of
view. Park and Aggarwal [135] estimated human body ges-
tures using Bayesian networks and modeled the evolution of
two person interactions (such as a turning head in a left-
right direction or a gesture used in greeting each other)
by dynamic Bayesian networks (DBN). In Natarajan and
Nevatia’s work [116], coupled hidden semi-Markov mod-
els (CHSMMs) were proposed for modeling and recogniz-
ing the duration of subevents and representing the interac-
tion between multiple people. The major limitation with this
model is complexity. Gupta and Davis [54] proposed a prob-
abilistic model that exploit contextual information for visual
action analysis to improve object recognition as well as ac-
tivity recognition. Hidden Markov models and Bayesian re-
lations are used by Moore et al. [113], to classify objects
and identify human motion. But their method is limited to
an overhead view and only track hands. Peursum et al. [138]
suggest the use of human actions to infer object class based
on location and identity of objects.
4.2 Hierarchical approaches
It defined recognition methodologies for complex human ac-
tivities such as human-object interactions and group activi-
ties.
4.2.1 Statistical
Statistical approaches use statistical state-based models to
recognize activities. In the case of hierarchical statistical
approaches, multiple layers of state-based models (usually
two layers) such as HMMs and DBNs are used to recognize
activities with sequential structures. At the bottom layer,
atomic actions are recognized from sequences of feature
vectors, just as in single-layered sequential approaches. As
a result, a sequence of feature vectors are converted to a
sequence of atomic actions. The second-level models treat
this sequence of atomic actions as observations generated
by the second-level models. For each model, a probability
of the model generating a sequence of observations (i.e.,
atomic-level actions) is calculated to measure the likelihood
between the activity and the input image sequence. Oliver
et al. [128] developed layered hidden Markov models (LH-
MMs), to model a real-time office activities. LHMMs can
be regarded as a cascade of HMMs, and both upper layer’s
HMM and bottom layer’s HMM are connected via inferen-
tial results. Zhang et al. [225] used layered HMMs to rec-
ognize group actions from recognized individual actions.
Nguyen et al. [119] used hierarchical HMMs (HHMMs) to
recognize behavior correctly but not suited well for com-
plex behaviors. Similar to [128], Shi et al. [173] proposed a
hierarchical approach using a propagation network (P-net).
Their work was focused on a common task for elderly peo-
ple who develop late stage diabetes. Yu and Aggarwal [221]
used a block-based discrete hidden Markov model to rec-
ognize and analyze observation sequences containing a set
of multiple actions. However, it failed to recognize multiple
concurrent actions. Cupillard et al. carry out event recogni-
tion by mean of graphs combination mechanism, in [31] an
approach for recognizing group of people behaviors using
multiple cameras is presented. Dai et al. [33] indicated that
multilevel dynamic Bayesian network (DBN) model speci-
fies the detailed online analysis to detect multilevel events.
Damen and Hogg [34] constructed Bayesian networks using
a AND-OR grammars to encode pairwise event constraints.
Gong and Xiang [52] proposed a dynamically multi-linked
Hidden Markov model with the structure determined by
Bayesian Information Criterion (BIC) to automatically sum-
maries or categories activities within video. It avoids the dif-
ficulties associated with tracking under occlusion in noisy
scenes.
4.2.2 Syntactic
Syntactic approaches model human activities as a string of
symbols, where each symbol corresponds to an atomic-level
action. Ivanov and Bobick [68] suggested using stochastic
context-free grammars (SCFGs) to model visual activities

and introduced it on an upper layer to compute the prob-
ability of temporally consistent sequences of primitive ac-
tions. It worked well for temporally extended behaviors and
interactions between multiple objects. However, modeling
of temporally explicit behavior among interactive objects
(more than two) was a difficult task for them. Joo and Chel-
lappa [74] designed an attribute grammar for recognition,
which is an extension of the SCFG. Their grammar attaches
semantic tags and conditions to the production rules of the
SCFG, enabling the recognition of more descriptive activi-
ties. That is, their grammar is able to describe feature con-
straints as well as temporal constraints of atomic actions.
Moore and Essa [112] further extended the work of [68] by
incorporating error detection and recovery mechanism. They
have focused on multitask activities. One of the major draw-
backs was the inability to compensate the failure of compli-
cated human activities. However, they are not equally suited
to many other visual modeling tasks, which involve essen-
tially non-sequential data, for example the spatial relation-
ships in a single visual image. Minnen et al. [107] present a
system that uses human specified grammars (i.e., SCFGs) to
recognize a person performing the wood workshop activity
from a video sequence by analyzing object interaction be-
haviors separately within the structure of specified grammar.
For example, humans using a drill machine is a whole activ-
ity, which is now divided into three subevents (e.g., “switch,”
“drill,” and “switch off”). All three subevents are analyzed
separately in detail to yield corresponding subactivities, and
then clubbing them sequentially to recognize a whole activ-
ity in a better way.
4.2.3 Description-based
A description-based approach is a recognition approach
that explicitly maintains spatiotemporal structures for hu-
man activities. They represent a high-level human activity
in terms of simpler activities composing the activity (i.e.,
subevents), describing their temporal, spatial, and logical
relationships. Based on the IA-network concept, Pinhanez
and Bobick [140] have proposed the PNF-network. They
have shown that their method is computationally tractable
even though one of the atomic actions was not provided.
But the scope of their methodology was limited to tempo-
ral information. One of the drawbacks is that a subnetwork
has to be specified redundantly against occurrences of their
subevents. Intille and Bobick [66] have designed a more
complex Bayesian network model to identify actions of a
football player in a crowded scene. They described the tem-
poral structure in programming a language format instead
of a network form. It can recognize highly structured or
uncertain visual perception (e.g., players are continuously
interacting with each other and simultaneously changing
their behavior) by representing them in three level of hierar-
chy (atomic-level activities, individual-level activities, and
S. Vishwakarma, A. Agrawal
team-level activities) to deal with uncertain and incomplete
nature of real world application. Siskind [175] also proposed
a hierarchical description-based approach for human activ-
ity recognition. Notably, it decomposed high-level activities
into primitive events and applies force dynamics on it to
recognize simple actions. The event logic is used to com-
bine primitive events of participating humans/objects to rec-
ognize complicated semantic-level complex events. How-
ever, there remains a challenging problem in computational
models of language grounding. The representation language
called Video Event Representation Language (VERL) [117]
for composite events was presented by Nevatia et al. [118].
They have not only constructed formal description by clas-
sifying human activities into three level of hierarchy (e.g.,
primitive events, single-thread composite events, and multi-
thread composite events), but also illustrated a heuristic al-
gorithm to detect ongoing human activities from input im-
ages. One of the main drawbacks is their inability to describe
complex composition of activities due to its single/multiple
thread terminology. Significant progress has been made by
Vu et al. [197] to recognize long-term events. They have
adopted temporal constraints propagation techniques. The
spatial-temporal knowledge has been exploited to tackle
the activity recognition problem as a constraint satisfaction
problem.
Ghanem et al. [49] applied the Petri nets to model com-
plicated activities and interaction in traffic domain. In [55],
work on activity classification has focused on recognition
of atomic level action, using 2D video. Gupta et al. used a
context-free (AND-OR) grammar in their approach. How-
ever, the use of 2D video leads to relatively low accuracy
even in the absence of clutter. Ryoo and Agrawal [152, 154]
presented a framework for human action modeling by using
Bayesian Networks and Context-free grammars to recognize
the composite activities (such as, poses of body parts). Ryoo
and Agrawal [155] describe a method for recognizing com-
plex human activities using a context-free grammar (CFG)
based representation scheme, derived from [152]. In [153]
object context is used for recognition of human-object inter-
action.
5 Human behavior understanding
Behavior understanding involves analysis and recognition of
motion patterns, and the production of high-level description
of actions and interactions between or among objects. It is
necessary to analyze the behaviors of people and determine
whether their behaviors are normal or abnormal [82]. An
automated visual surveillance system which can understand
and learn behavior from observing activities in a video se-
quence requires a reliable combination of image processing
techniques and artificial intelligence techniques [69, 209].
A survey on activity recognition & behavior understanding in video surveillance
Detection of suspicious human behavior involves modeling
and classification of human activities with certain rules [26].
The idea is to partition the observed human movement into
some discrete states and then classify them appropriately.
Partitioning of the observed movements is very application
specific and overall hard to predict what will constitute sus-
picious or endangering behavior [26, 69].
The action recognition is governed by movement of gaits
and these movements can be performed at several levels of
abstraction. Different taxonomies were proposed here and
we consider the hierarchy used by Moeslund et al. [109],
i.e., action primitive, action, and activity. An action primi-
tive is an atomic movement. At limb level, action primitive
can be described. An action consists of action primitives and
describes a whole body movement. At last, activities contain
a number of subsequent actions, and yield an interpretation
of the movement that is being performed. For instance “rais-
ing the left hand” is an action primitive, where as “waving
the hand” is an action. “Punching someone” is an activity
that contains raising, waving, and punching actions.
Efros and Berg et al. [41] introduced a novel motion de-
scriptor based on optical flow measurements in a spatiotem-
poral volume for each stabilized human figure and demon-
strated the use of this descriptor in a nearest-neighbor query-
ing framework.
Yilmaz and Shah [218] proposed the spatiotemporal vol-
umes (STV) to solve the point correspondence problem be-
tween consecutive frames, and then recognized the actions
based on the descriptors, which were computed by analyzing
the differential geometric properties of STV. Gorelick and
Blank et al. [53] represented actions as space-time shapes
which contained both spatial and dynamic information. This
method was fast and did not require prior video alignment.
A reliable performance can be achieved in action recogni-
tion.
Laptev and Lindeberg [86] extended a 2D Harris cor-
ner detector to a 3D Harris detector, which detected regions
having significant local variations in both spatial and tem-
poral dimension. The representation has been successfully
applied to human action recognition combined with a SVM
classifier [164]. Oikonomopoulos et al. [126] extended the
concept of saliency from spatial to the spatiotemporal do-
main by using a sparse set of spatiotemporal features. Dol-
lar et al. [39] improved the 3D Harris detector based on
a convolution with a Gabor filter in time. Combined with
some global information, Wong and Cipolla [206] presented
a method to extract interest points using global informa-
tion, such as the organization of pixels in a whole video.
Nowozin et al. [122] used a sequential representation, which
retained the temporal order. A set of partial motion models
was combined into a global model of human motion for mo-
tion recognition by Filipovych et al. [45].
A probabilistic context-free grammar (PCFG) was
adopted by Ogale et al. [123] for view-invariant action
recognition. PCFG was constructed after background re-
moval, silhouette extraction, and keyframe extraction. Para-
meswaran [132] developed a high-level representation and
an efficient recognition algorithm for human action that
was resistant to variation in viewpoints, and used pro-
jective invariants of landmark points on a human body.
But this method needed the 3D motion capture for cre-
ating the action model database. Motion History Volumes
(MHV) [203] [201] was introduced as a free-viewpoint rep-
resentation for human actions in the case of multiple cali-
brated, background-subtracted, and video cameras. This rep-
resentation was used to recognize basic human actions, in-
dependently of gender, body size, and viewpoint. Weinland
et al. [202] proposed the 3D exemplar based hidden Markov
model (HMM). The exemplars were represented in 3D as
visual hulls that had been computed using a system of 5 cal-
ibrated cameras. The method can recognize a single video
or multiple videos. Lv and Nevatia [102] modeled actions as
a graph model (Action Net) and used an enhanced Pyramid
Match Kernel algorithm for fast matching between two sim-
ilar feature sets. After successfully tracking the movement
of the human subject from one frame to another in an image
sequence, the problem of understanding human behaviors
from the perspective of image sequences becomes appar-
ent. Behavior understanding involves action recognition and
description and may be simply considered a classification
problem of time varying feature data, i.e., matching an un-
known test sequence with a group of labelled reference se-
quences representing typical human actions. However, when
there is a need for a more complete description, other ap-
proaches are convenient: dynamic time warping [72], hidden
Markov models [6] or neural networks [47], followed by an
action recognition step and semantic description phase. Ex-
isting techniques can be grouped into the following types
based on the nature of the algorithms used: Naive Bayes
probabilistic models [40], hidden Markov models [6], fuzzy
logic: K-NN (K-Nearest Neighbors), NN (KNearest Neigh-
bors) [3], dynamic time warping [72], Sequential Minimal
Optimization (SMO) [141], trees and decision rules CART
(Classification and Regression Trees) [219] programs for
machine learning [145] RIPPER [27], neural networks [47].
5.1 Supervised model
The abnormal behavior is detected in the video based on
the predefined activity and their class. Only those abnormal
and normal behavior could be identified, which were already
existing in the database. However, in reality, abnormal be-
havior is difficult to define for certain domains of activity
clearly.
5.2 Unsupervised model
More recently, a number of techniques have been pro-
posed for unsupervised learning of behavior models. They
 S. Vishwakarma, A. Agrawal

can be further categorized into two different types accord-
ing to whether an explicit model is built. Approaches that
do not model behavior explicitly either perform cluster-
ing on observed patterns and label those forming small
clusters as being abnormal. Learning from unlabeled data
still remains one of the most challenging problems in the
fields of computer vision and machine learning. The chal-
lenge mainly results from the fact that unlabeled images
contain various objects, which change in pose, scale, and
degree of occlusion. Significant progress has been made
in [44, 57, 200], which proposes several unsupervised tech-
niques for learning object models as constellations of fea-
tures. Weber et al. [200] represent an object as a constel-
lation of parts. Fergus et al. [44] extend the model to ac-
count for variability in appearance. Their results are encour-
aging but achieved on a prepared set of images with the ob-
ject of interest being always in the foreground and cover-
ing a large part of the images. The cooccurrences of fea-
tures are introduced to separate a different object class, as
in [58, 90, 134, 146, 176]. In [134, 176]. The authors define
neighbors of features to represents objects. Hoiem et al. [58]
and Rabinovich et al. [146] model relationships for robust
object detection or image labeling, but require manually lo-
calizing the objects in training images. Leordeanu et al. [90]
link temporal dependent entities by observing their cooc-
currence, therefore, it is more appropriate for object track-
ing. In [215], authors propose a maximum likelihood al-
gorithm for unsupervised shared structure learning, where
shared structures are represented as the strongly connected
clusters of consistent pairwise spatial relationships.
5.3 Semisupervised model
A recent work by Xu et al. [211] focused on semisuper-
vised learning techniques to recognize both simple and com-
plex activities. Compared to supervised techniques, the ma-
jor contribution of [211] is to alleviate the requirement of
a large training data set. Hue Thi et al. [183] described a
framework called the weakly supervised approach, which
attempts to recognize human action by integrating the lo-
cal features both spatially and temporally. They tested their
technique on KHT, HoHA1, and the TRECVid dataset, and
compared results with existing supervised techniques.
6 Description of dataset
In this section, we discuss some of the datasets, which are
currently used by many of the action recognition approaches
as a benchmark (see Table 4).
6.1 Dataset defined for controlled environment
Several public datasets have been introduced in the past
10 years [158], encouraging researchers to explore vari-
ous action recognition directions. The KTH dataset [164]
and the Weizmann dataset [53] are the typical examples of
these dataset. These two single-camera datasets have been
designed for research purposes, providing a standard for
researchers to compare their action classification perfor-
mances. The datasets are composed of videos of relatively
simple periodic actions, such as walking, jogging, and run-
ning. The videos are segmented temporally so that each clip
contains no more than one action of a single person. They

Table 4 Comparison of characteristics of datasets

Specification KTH Weizmann HOHA1 TRECVID VIRAT PETS i-Lids UT-Int.a

# of Event types 6 10 8 10 23 3 4 6
Avg. # of samples per class 100 9 85 31670 101500 N/A N/A 8
Max. resolution 160 × 6 180 × 144 540 × 240 720 × 576 1920 × 1080 768 × 576 720 × 576 720 × 480
Human height in pixels 80 100 60 70 100 1200 20 200 20 180 20 N/A 200
Human to video height ratio 65–85 % 42–50 % 50–500 % 4–36 % 2–20 % N/A N/A N/A
# Scenes N/A N/A many 5 16 8 7 20
Viewpoint type side side varying 5/varying varying varying side side
Natural background clutter No No Yes Yes Yes Yes No Yes
Incidental objects/activities No No Yes, varying Yes Yes Yes Yes Yes
End-to-End activities No No Yes, varying Yes Yes Yes N/A N/A
Tight bounding boxes cropped cropped No No Yes No N/A N/A
Multiple annotations on movers No No No No Yes Yes N/A Yes
Camera motion No No varying No varying varying N/A No

a UT-Interaction
A survey on activity recognition & behavior understanding in video surveillance
Fig. 8 Example KTH dataset corresponding to different types of ac-
tions and scenarios (based on Schuldt’s work [164])
were taken in a controlled environment; their backgrounds
and lighting conditions are mostly uniform. In general, they
have a good image resolution and little camera jitters.
6.1.1 KTH dataset
The KTH dataset contains six actions. They are performed
by 25 actors under four different scenarios of illumina-
tion, appearance, and scale changes. In total, it contains 598
video sequences. The KTH video database containing six
types of human actions (walking, jogging, running, box-
ing, hand waving, and hand clapping) performed several
times by 25 subjects in four different scenarios: outdoors
s1, outdoors with scale variation s2, outdoors with different
clothes s3 and indoors s4. as illustrated in Fig. 8. There are
25 × 6 × 4 = 600 video files for each combination of 25 sub-
jects, 6 actions, and 4 scenarios. All videos were taken over
homogeneous backgrounds with a static camera with 25 fps
frame rate. The videos were down-sampled to the spatial res-
olution of 160 × 120 pixels and have a 4 seconds duration in
average.
6.1.2 Weizmann
This a dataset contains 90 low-resolution 180 × 144 50 fps
video sequences showing nine different people, each per-
forming 10 natural actions such as “run,” “walk,” “skip,”
“jumping-jack,” (or shortly “jack”) “jump-forward-on-two-
legs,” (or “jump”) “jump-in-place-on-two-legs,” (or
“pjump”) “gallop sideways,” (or “side”) “wave-two-hands,”
(or “wave2”) “waveone-hand,” (or “wave1”) or “bend.” Fig-
ure 9 shows examples of the Weizmann dataset.
6.2 Dataset defined for realistic environment
Recently, more challenging datasets were constructed by
collecting realistic videos from movies [77, 88, 150]. These
movie scenes are taken from varying view points with
complex backgrounds, in contrast of the previous public
Fig. 9 Example of WEIZMANN dataset (derived from Gorelick’s
work [53])
Fig. 10 Eight example scenes in HOHA1 Dataset [89]
datasets [53, 164]. These datasets encourage the develop-
ment of recognition systems that are reliable under noise
and view point changes. However, even though these videos
were taken in more realistic environments, the complexity
of the actions themselves were similar to [53, 164]: The
datasets contain simple instantaneous actions such as kiss-
ing and hitting. They were not designed to test recognition
of high-level human activities from continuous sequences.
6.2.1 HOHA1
The Hollywood Human Action HoHA1 dataset contains 8
action classes, Lying, StandUp, HugPerson, Kiss, GetOut-
Car, HandShake, Sitdown, and SitUp. The action classes
are distributed in 430 training and 448 testing videos. The
background of this dataset is highly complex in nature and
contain cluttered scenarios. Figure 10 represents the HoHA1
dataset Laptev et al. [89].
6.2.2 TRECVid
The TREC Video Retrieval Evaluation (TRECVid) [177]
is an international benchmarking activity to encourage re-
search in video information retrieval by providing a large
test collection, uniform scoring procedures, and a forum
for organizations interested in comparing their results. Fig-
ure 11 shows snapshot of this dataset.
 S. Vishwakarma, A. Agrawal

Fig. 11 Snapshots from the Event Detection track TRECVid

dataset [184]

6.2.3 VIRAT

A new large-scale video dataset [124] designed to assess the Fig. 12 Six example scenes in VIRAT Video Dataset (total 16 scenes)
performance of diverse visual event recognition algorithms is from [124]
with a focus on continuous visual event recognition (CVER)
in outdoor areas with wide coverage. Previous datasets for
action recognition are unrealistic for real-world surveillance
because they consist of short clips showing one action by
one individual [53, 164]. Datasets have been developed for
movies [88] and sports [98], but these actions and scene con-
ditions do not apply effectively to surveillance videos. This
dataset consists of many outdoor scenes with actions occur-
ring naturally by nonactors in continuously captured videos
of the real world. The dataset includes large numbers of in-
stances for 23 event types distributed throughout 29 hours
of video. This data is accompanied by detailed annotations,
which include both moving object tracks and event exam-
ples, which will provide a solid basis for large-scale evalu- Fig. 13 Example of PETS 2009 dataset containing crowd scenarios
ation. An example of this can be seen in Fig. 12 extracted with an increasing scene complexity (taken from PETS [62]) First
Row: left image concerns person count and density estimation and right
from the VIRAT dataset. image addresses people tracking. Second Row: left and right images in-
volve event recognition and flow analysis, respectively
6.2.4 PETS

The performance evaluation of tracking and surveillance

(PETS) [62] program organizes regular workshops and pro-
vides a benchmark dataset (see Fig. 13). This dataset ad-
dresses the problem of group activities such as crowd image
analysis (i.e., crowd count and density estimation, tracking
of individual(s) within a crowd, and detection of separate
flows and specific crowd events) within a public space. The
PETs 2009 provide researchers to evaluate new or existing
detection techniques on dataset captured in a real-world en-
vironment. The dataset scenarios were filmed from multiple
cameras and involve multiple actors. Fig. 14 Snapshot of i-Lids dataset (similar to CPNI’s work [63])

6.2.5 i-Lids
Analytics (VA) systems developed in partnership with the
The image library for intelligent detection systems (i-LIDS) Center for the Protection of National Infrastructure (CPNI).
(see Fig. 14, [63]) is the government’s benchmark for Video There are currently five scenarios within i-LIDS, that is, ster-
A survey on activity recognition & behavior understanding in video surveillance
ile zone monitoring, parked vehicle detection, abandoned
baggage detection, doorway surveillance, and multiple cam-
era tracking scenario. Each of these scenarios are made up
of three datasets. The datasets for the event detection scenar-
ios each contain approximately 24 hours of footage. Each of
these datasets are filmed to represent all weather, time of
day, and scene densities expected within the scenario. The
multiple camera tracking scenario datasets each contain ap-
proximately 50 hours of real world footage.
Each dataset consists of two or three camera views re-
ferred to as stages, and is further segmented into shorter
video clips of 30 to 60 minutes. The training dataset is fur-
ther split into individual events. Each dataset is supplied
with a user guide detailing the library structure, user inter-
face, and the procedure used to evaluate the systems against
the relevant scenario.
6.2.6 UT-Interaction
The UT-Interaction dataset is designed to encourage de-
tection of high-level human activities (e.g., hand shaking)
which are more complex than the previous simple actions. In
addition, it encourages localization of the multiple activities
involving multiple persons and objects [156, 198, 222] from
continuous video streams spatially and temporally. The UT-
Interaction dataset [157] contains videos of continuous exe-
cutions of 6 classes of human-human interactions: hand-
shake, point, hug, push, kick, and punch. Figure 15 shows
example snapshots of these multiperson activities. Ground
truth labels for all interactions in the dataset videos are pro-
vided, including time intervals and bounding boxes. There
are a total of 20 video sequences whose lengths are around 1
minute. Each video contains at least one execution per inter-
action, providing us about 8 executions of human activities
per video on average. Several actors with more than 15 dif-
ferent clothing conditions appear in the videos. The videos
are taken with the resolution of 720 × 480, 30 fps, and the
height of a person in the video is about 200 pixels.
Fig. 15 Example of UT-interaction dataset (based on Ryoo’s
work [157])
7 Economical impact on video surveillance market
The projection of the compound annual growth rate of
video surveillance market in the United States of America
is $67.07 million and in Europe is about $188.3 million (see
Fig. 16).
In the face of global recession, the security and video
surveillance industries continue to remain unaffected and
even growing. According to IMS Research, the demand
for CCTV and video surveillance equipment should re-
main strong throughout 2012, with potential growth for the
surveillance equipment market predicted to exceed 25 %. In
2010, analog sales were notably low, but leaps and bounds
were made in network video surveillance technology caused
by the market that year to increase by more than 300 %. Two
probable causes for this are:
– The demand for security and video surveillance continues
to be very relevant if not increasing in relevancy. The un-
deniable return-on-investment and government stimulus
spending have spurred video surveillance industry sales,
which continued to grow throughout 2010 and 2011.
Video surveillance has proven itself in decreasing crime
as well as compiling valuable evidence in the aftermath.
– The recent economic downturn did not affect all coun-
tries, worldwide equally. Technological transitions from
lower value video surveillance equipment to more ad-
vanced networked based surveillance continued to remain
strong and even bolstered the video surveillance market as
a whole.
As shown in Fig. 17, it is clear that activity recognition is
an active research topic. In fact, there have been three times
as many publications in the last 5 years than the number of
all publications found before 2005. As we enter 2012 amidst
a sea of predictions, forecasts and top ten trends for the se-
curity surveillance industry, one thing that remains constant
is the incredible technological innovations that we have seen
in the past year, and which will evolve in 2012.
Fig. 16 Projection of the compound annual growth rate of the video
surveillance market (Courtesy IMS Research [64])

Fig. 17 Increasing interest on
human activity recognition
research is shown through a
comparison of a number of
papers published up to 2011
8 Practical issues in real world scenario
Despite much advancement in the field of automated surveil-
lance, there are still challenges to realize it in teh practical
real world conditions due to following issues.
8.1 Robustness
Real world scenarios are characterized by sudden or gradual
changes in the input statistics. A major challenge for real
world object detection and tracking is the dynamic nature of
real world conditions with respect to illumination, motion,
visibility, weather changes, etc. Achieving robust algorithms
is a challenge especially (a) under illumination variation due
to weather conditions or lighting changes, for example, in an
outdoor scene, due to movement of clouds in sky and in an
indoor scene, due to opening of doors or windows; (b) under
view changes; (c) in the case of multiple objects with partial
or complete occlusion or deformation; (d) in the presence of
articulated or nonrigid objects; (e) in the case of a shadow,
reflections, and clutter; and (f) with video noise (e.g., Gaus-
sian white). Other scenarios with (a) low light and illumi-
nation variation (b) boat among moving waves, and (c) car
object with moving background of vegetation. Significant
research and advancement in solving these difficulties have
been achieved, but still the problem is unsolved in generic
situations with dynamically varying environmental condi-
tions and there is lack of generic multimodal framework to
achieve system robustness by data fusion.
8.2 Intelligent
With the advances in sensor technology, surveillance cam-
eras, and sound recording systems are already available in
banks, hotels, stores and highways, shopping centers, and
the captured video data are monitored by security guards and
stored in archives for forensic evaluation. In a typical sys-
tem, a security guard watches 16 video channels at the same
time and may miss many important events. There is a need
S. Vishwakarma, A. Agrawal
of intelligent, (semi) automated video analysis paradigms to
assist the operators in scene analysis and event classifica-
tion. Event detection is a key component to provide timely
warnings to alert security personnel. It deals with mapping
motion patterns to semantics (e.g., benign and suspicious
events). However, detecting semantic events from low-level
video features is major challenge in real world situations
due to unlimited possibilities of motion patterns and be-
haviors leading to well-known semantic gap issues. Further-
more, suspicious motion events in surveillance videos hap-
pen rather infrequently and the limited amount of training
data poses additional difficulties in detecting these so-called
rare events.
8.3 Real timeliness
A useful processing algorithm for surveillance systems
should be real time, i.e., output information, such as events,
as they occur in the real scene. Requirement of accuracy and
robustness result in computational intensive and complex
design of algorithms which makes real time implementation
of a system a difficult task.
8.4 Cost effective
For feasible deployment in a wide variety of real world
surveillance applications, ranging from indoor intrusion de-
tection to outdoor surveillance of important buildings, etc.,
a cost effective framework is required.
9 Conclusion
An automated detection of ongoing activities and behavior
analysis has become an active research area in the com-
puter vision field. It is strongly driven by many promising
applications such as smart surveillance, intelligent robots,
virtual reality, action-based human-computer interfaces, un-
usual activities and prevent terrorism, personal safety of
A survey on activity recognition & behavior understanding in video surveillance
passengers and public, etc. Technical development in this
field demonstrated how it deals with challenges that involve
complex human movements. It is exciting to see many re-
searchers gradually spreading their achievements into more
intelligent practical applications.
We have presented a general processing framework of hu-
man activity recognition systems, and discuss the recent de-
velopment explored for various stages of the system. The
state-of-the-art of existing methods in each key issue is de-
scribed and the focus is on three major tasks: detection,
tracking, and activity recognition or behavior understand-
ing. We have also discussed the publicly available datasets
built for the uniform testing of methodology proposed by au-
thors. We have provided a brief description of datasets and
characteristics comparison among them.
In this survey, a brief overview of all preprocessing (i.e.,
detection, classification, and tracking) steps has been in-
cluded. There are many limitations and open challenges,
which we have highlighted by providing the comparison.
Motion detection in dynamic scenes is a difficult task in
the presence of illumination and weather changes, detection
of a shadow, self-occlusion, and complete occlusion. Fast
and accurate methods are still needed for segmentation tech-
niques to affect the performance of latter stages.
We have discussed the various object tracking techniques
for human, groups of people, moving vehicles, etc. The
following six approaches are studied intensively in past
works: region-based, contour-based, feature-based, model-
based, hybrid, and optical flow. It represents a class of prob-
lems that are both computational and data intensive. During
the study, we found that the hybrid approach may be a bet-
ter choice to overcome occlusions of tracked entities, clut-
ter in background, and losing the object due to rapid move-
ments. More researchers are needed to track the human in
a crowded scene and robust improvement is required to im-
plement a method to redirect the tracking human or group of
humans in multiple perspectives. Tracking multiple persons
or a group of people are difficult due to a crowded environ-
ment, poor illumination, noisy images, and camera move-
ments. We have explored various methods for the recogni-
tion of activities of a single person, activities of crowd as
a whole (or as subgroup), and found that space time ap-
proaches have proven to yield good results for periodic ac-
tions and gestures. They usually do not require background
subtraction. However, their applicability is limited to slow
speed motion variation. Moreover, they cannot deal with oc-
clusions. The space-time volume representation is restricted
to recognize actions when multiple people appeared in the
scene. For accurate localization of actions, they are com-
putationally very expensive as they do not support view-
invariancy. Space-time trajectory does better work to repre-
sent and recognize human movements from different view-
ing angles, but still 3-D body modeling is in the research
pipeline, and most of the reported work was restricted to
fixed and known video points. The multiple view depen-
dent action required more training complexity. The space-
time features approach are reliable under noise, illumination
changes, and recognize multiple activities, but not suited to
model complex, nonperiodic, and viewing invariant repre-
sentations of activities. While several works (e.g., [50]) have
addressed the recognition of complex activities, but still it
remains a challenge.
The sequential approaches are reliable to detect the com-
plex and nonperiodic activities by using sequential features.
The exemplar-based methods are more flexible and require
less training data in comparison to the methods discussed
under the state-based. The problem with state-based ap-
proaches are that if the complexity level of activities get in-
creased, then they need a greater number of training data.
Description based approaches are doing well to recog-
nize high level activities whose subevents are organized con-
currently and occurring in a sequential manner in compari-
son to the statistical or syntactic approaches. The statistical
and syntactic approaches can effectively handle the activity
video polluted with noise.
References
1. Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Com-
put. Vis. Image Underst. 73(3), 428–440 (1999)
2. Aggarwal, J.K., Ryoo, M.S.: Human activity analysis: a review.
ACM Comput. Surv. 43(3), 1–43 (2011)
3. Aha, D.W., Kibler, D., Albert, M.K.: Instance-based learning al-
gorithms. Mach. Learn. 06, 37–66 (1991)
4. Allili, M.S., Bouguila, N., Ziou, D.: A robust video foreground
segmentation by using generalized Gaussian mixture modeling.
In: 4th Canadian Conf. on Computer and Robot Vision, pp. 503–
509 (2007)
5. Bayona, A., SanMiguel, J.C., Martínez, J.M.: Stationary fore-
ground detection using background subtraction and temporal dif-
ference in video surveillance. In: IEEE 17th Int. Conf. on Image
Processing, pp. 1–4 (2010)
6. Blunsom, P.: Hidden Markov models. Tech. rep, Human Lan-
guage Technology University of Melbourne, Victoria, Aus-
tralia (2004). http://www.cs.mu.oz.au/460/2004/materials/hmm-
tutorial.pdf
7. Bobick, A.F., Davis, J.W.: The recognition of human movement
using temporal templates. IEEE Trans. Pattern Anal. Mach. In-
tell. 23(3), 257–267 (2001)
8. Bobick, A.F., Wilson, A.D.: A state-based approach to the repre-
sentation and recognition of gesture. IEEE Trans. Pattern Anal.
Mach. Intell. 19(12), 1325–1337 (1997)
9. Bose, B., Grimson, E.: Improving object classification in far-field
video. In: Proc. of the Int. Conf. on Computer Vision and Pattern
Recognition, pp. 181–188. IEEE Computer Society, Washington
(2004)
10. Brown, L.M.: View independent vehicle/person classification. In:
Proc. of the ACM 2nd Int. Workshop on Video Surveillance &
Sensor Networks, pp. 114–123. ACM Press, New York (2004)
11. Bucak, S.S., Gunsel, B., Gursoy, O.: Incremental nonnegative
matrix factorization for background modeling in surveillance
video. In: IEEE 15th Signal Processing and Communications Ap-
plications (SIU), pp. 1–4 (2007)

12. Cai, L., He, L., Yamashita, T., Xu, Y., Zhao, Y., Yang, X.: Robust
contour tracking by combining region and boundary information.
IEEE Trans. Circuits Syst. Video Technol. 21(12), 1784–1794
(2011)
13. Campbell, L., Bobick, A.: Recognition of human body motion
using phase space constraints. In: ICCV, pp. 624–630 (1995)
14. Camplani, M., Salgado, L.: Adaptive background modeling in
multicamera system for real-time object detection. Opt. Eng.
50(12), 1–17 (2011)
15. Cavallaro, A., Steiger, O., Ebrahimi, T.: Tracking video objects in
cluttered background. IEEE Trans. Circuits Syst. Video Technol.
15(4), 575–584 (2005)
16. Chai, Y., Shin, S., Chang, K., Kim, T.: Real-time user interface
using particle filter with integral histogram. IEEE Trans. Con-
sum. Electron. 56(2), 510–515 (2010)
17. Chang, S.F.: The holy grail of content-based media analysis.
IEEE Multimed. 9(2), 6–10 (2002)
18. Chen, L., Yang, H., Takaki, T., Ishii, I.: Real-time frame-
straddling-based optical flow detection. In: Proc. of IEEE Int.
Conf. on Robotics and Biomimetics, pp. 2447–2452 (2011)
19. Chen, Q., Sun, Q.S., Heng, P.A., Xia, D.S.: Two-stage object
tracking method based on kernel and active contour. IEEE Trans.
Circuits Syst. Video Technol. 20(4), 605–609 (2010)
20. Chen, Y., Zhang, L., Lin, B., Xu, Y., Ren, X.: Fighting detection
based on optical flow context histogram. In: Proc. of IEEE 2nd
Int. Conf. on Innovations in Bio-inspired Computing and Appli-
cations, pp. 95–98 (2011)
21. Cheng, F.H., Chen, Y.L.: Real time multiple objects tracking and
identification based on discretewavelet transform. Pattern Recog-
nit. 39, 1126–1139 (2006)
22. Cheung, K., Baker, S., Kanade, T.: Shape-from-silhouette across
time part II: applications to human modeling and markerless mo-
tion tracking. Int. J. Comput. Vis. 63(3), 225–245 (2005)
23. Chiverton, J., Mirmehdi, M., Xie, X.: On-line learning of shape
information for object segmentation and tracking. In: Proc. of
British Machine Vision Conference, pp. 1–11 (2009)
24. Chiverton, J., Xie, X., Mirmehdi, M.: Automatic bootstrapping
and tracking of object contours. IEEE Trans. Image Process.
21(3), 1231–1245 (2012)
25. Chomat, O., Crowley, J.L.: Probabilistic recognition of activity
using local appearance. In: IEEE Computer Society Conf. on
Computer Vision and Pattern Recognition, vol. 2, pp. 637–663
(1999)
26. Cohen, C.J., Morelli, F., Scott, K.A.: A surveillance system for
recognition of intent within individuals and crowds. In: Conf. on
Technologies for Homeland Security, Waltham, MA, pp. 559–
565. IEEE Press, New York (2008)
27. Cohen, W.W.: Fast effective rule induction. In: Proc. of 12th Int.
Conf. on Machine Learning, pp. 115–123. Morgan Kaufmann,
San Mateo (1995)
28. Coifman, B., Beymer, D., McLauchlan, P., Malik, J.: A real-
time computer vision system for vehicle tracking and traffic
surveillance. Transp. Res., Part C, Emerg. Technol. 6(4), 271–
288 (1998)
29. Collins, R.T., Lipton, A.J., Kanade, T., Fujiyoshi, H., Duggins,
D., Tsin, Y., Tolliver, D., Enomoto, N., Hasegawa, O., Burt,
P., Wixson, L.: A system for video surveillance and monitor-
ing. Tech. rep, Robotics Institute at Carnegie Mellon University
(2000)
30. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object track-
ing. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 564–577
(2003)
31. Cupillard, F., Bremond, F., Thonnat, M.: Group behavior recog-
nition with multiple cameras. In: Proc. 6th IEEE Workshop on
Applications of Computer Vision, pp. 177–183 (2002)
S. Vishwakarma, A. Agrawal
32. Cutler, R., Davis, L.S.: Robust real-time periodic motion detec-
tion, analysis, and applications. IEEE Trans. Pattern Anal. Mach.
Intell. 22(8), 781–796 (2000)
33. Dai, P., Di, H., Dong, L., Tao, L., Xu, G.: Group interaction anal-
ysis in dynamic context. IEEE Trans. Syst. Man Cybern. 38(1),
275–282 (2008)
34. Damen, D., Hogg, D.: Recognizing linked events: searching the
space of feasible explanations. In: IEEE Conf. on Computer Vi-
sion and Pattern Recognition, pp. 927–934 (2009)
35. Darrell, T., Pentland, A.: Space-time gestures. In: Proc. IEEE
Computer Society Conf. on Computer Vision and Pattern Recog-
nition, pp. 335–340 (1993)
36. Denman, S., Chandran, V., Sridharan, S.: Adaptive optical flow
for person tracking. In: Proc. of the Digital Imaging Computing:
Techniques and Applications, DICTA ’05, pp. 1–7 (2005)
37. Denman, S., Chandran, V., Sridharan, S.: An adaptive optical
flow technique for person tracking systems. Pattern Recognit.
Lett. 28(10), 1232–1239 (2007)
38. Denman, S., Fookes, C., Sridharan, S.: Improved simultaneous
computation of motion detection and optical flow for object
tracking. In: IEEE Digital Image Computing: Techniques and
Applications, pp. 175–182 (2009)
39. Dollar, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recog-
nition via sparse spatio-temporal features. In: Int. Conf. on Com-
puter Communications and Networks, vol. 14, pp. 65–72. IEEE
Press, New York (2005)
40. Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analy-
sis. Wiley, Stanford Research Institute, Menlo Park (1973)
41. Efros, A.A., Berg, A.C., Mori, G., Malik, J.: Recognizing action
at a distance. In: Proc. 9th IEEE Int. Conf. on Computer Vision,
vol. 2, pp. 726–733 (2003)
42. Elgammal, A., Harwood, D., Davis, L.: Non-parametric model
for background subtraction. In: Frame-Rate Workshop, pp. 751–
767. IEEE Press, New York (2000)
43. Fazli, S., Pour, H.M., Bouzari, H.: Multiple object tracking us-
ing improved GMM based motion segmentation. In: IEEE ECTI-
CON, vol. 2, pp. 1130–1133 (2009)
44. Fergus, R., Perona, P., Zisserman, A.: Object class recogni-
tion by unsupervised scale-invariant learning. In: Proc. of IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition, vol. 2, pp. 264–271 (2003)
45. Filipovych, R., Ribeiro, E.: Combining models of pose and
dynamics for human motion recognition. In: 3rd International
Springer Symposium on Advances in Visual Computing, Ab-
erdeen, Scotland, pp. 21–32 (2007)
46. Forsyth, D.A., Arikan, O., Ikemoto, L., O’Brien, J., Ramanan,
D.: Computational studies of human motion: part 1, tracking and
motion synthesis. Found. Trends Comput. Graph. Vis. 1(02/03),
77–254 (2005)
47. Gallagher, M., Downs, T.: Visualization of learning in multilayer
perceptron networks using principal component analysis. IEEE
Trans. Syst. Man Cybern. 33, 28–34 (2003)
48. Gavrilla, D., Davis, L.: 3D Model-based tracking of humans in
action: a multi-view approach. In: Int. Proc. of the Computer Vi-
sion and Pattern Recognition, pp. 73–80 (1996)
49. Ghanem, N., DeMenthon, D., Doermann, D., Davis, L.: Repre-
sentation and recognition of events in surveillance video using
Petri nets. In: Conf. on Computer Vision and Pattern Recogni-
tion Workshop, pp. 112–121 (2004)
50. Gilbert, A., Illingworth, J., Bowden, R.: Fast realistic multi-
action recognition using mined dense spatio-temporal features.
In: IEEE 12th Int. Conf. on Computer Vision, pp. 925–931
(2009)
51. Girisha, R., Murali, S.: Tracking humans using novel optical flow
algorithm for surveillance videos. In: Proceedings of the 4th An-
nual ACM Bangalore Conf., COMPUTE ’11, pp. 1–8 (2011)
A survey on activity recognition & behavior understanding in video surveillance
52. Gong, S., Xiang, T.: Recognition of group activities using dy-
namic probabilistic networks. In: Proc. 9th IEEE Int. Conf. on
Computer Vision, vol. 2, pp. 742–749 (2003)
53. Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Ac-
tions as space-time shapes. IEEE Trans. Pattern Anal. Mach. In-
tell. 29(12), 2247–2253 (2007)
54. Gupta, A., Davis, L.S.: Objects in action: an approach for com-
bining action understanding and object perception. In: IEEE
Conf. on Computer Vision and Pattern Recognition, pp. 1–8
(2007)
55. Gupta, A., Srinivasan, P., Shi, J., Davis, L.S.: Understanding
videos, constructing plots learning a visually grounded storyline
model from annotated videos. In: IEEE Conf. on Computer Vi-
sion and Pattern Recognition, pp. 2012–2019 (2009)
56. Haritaoglu, I., Harwood, D., Davis, L.S.: W 4 : real-time surveil-
lance of people and their activities. IEEE Trans. Pattern Anal.
Mach. Intell. 22(8), 309–330 (2000)
57. Heisele, B., Ho, P., Wu, J., Poggio, T.: Face recognition:
component-based versus global approaches. Comput. Vis. Image
Underst. 91, 6–21 (2003)
58. Hoiem, D., Efros, A.A., Hebert, M.: Putting objects in perspec-
tive. Int. J. Comput. Vis. 80, 3–15 (2008)
59. Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual
surveillance of object motion and behaviors. IEEE Trans. Syst.
Man Cybern., Part C, Appl. Rev. 34(3), 334–352 (2004)
60. Hu, W., Xie, D., Tan, T., Maybank, S.: Learning activity patterns
using fuzzy self-organizing neural network. IEEE Trans. Syst.
Man Cybern. 34(3), 1618–1626 (2004)
61. Huang, J., et al.: GPU-accelerated computation for robust motion
tracking using the CUDA framework. In: Int. Conf. on Visual
Information Engineering, vol. 5, pp. 437–442 (2008)
62. 11th IEEE Int. Workshop on Performance Evaluation of Tracking
and Surveillance (2009). http://www.cvg.rdg.ac.uk/PETS2009/
authors.html
63. Imagery Library for Intelligent Detection Systems (2010).
http://www.ilids.co.uk
64. IMS Research. http://www.imsresearch.com/
65. Ince, S., Konrad, J.: Occlusion-aware optical flow estimation.
IEEE Trans. Image Process. 17(8), 1443–1451 (2008)
66. Intille, S.S., Bobick, A.F.: A framework for recognizing multi-
agent action from visual evidence. In: AAAI-99, pp. 518–525.
AAAI Press, Menlo Park (1999)
67. Ishii, I., Taniguchi, T., Yamamoto, K., Takaki, T.: 1000 fps
real-time optical flow detection system. Proc. SPIE 7538, 1–11
(2010)
68. Ivanov, Y.A., Bobick, A.F.: Recognition of visual activities and
interactions by stochastic parsing. IEEE Trans. Pattern Anal.
Mach. Intell. 22(8), 852–872 (2000)
69. Jan, T.: Neural network based threat assessment for automated
visual surveillance. In: Int. Joint Conf. on Neural Networks,
vol. 2, pp. 1309–1312. IEEE Press, New York (2004)
70. Jang, D.S., Choi, H.I.: Active models for tracking moving ob-
jects. Pattern Recognit. 33(7), 1135–1146 (2000)
71. Javed, O., Shah, M.: Tracking and object classification for auto-
mated surveillance. In: Proc. of the 7th European Conference on
Computer Vision, pp. 343–357. Springer, London (2002)
72. Jeong, Y.S., Jeong, M.K., Omitaomu, O.A.: Weighted dynamic
time warping for time series classification. Pattern Recognit. 44,
2231–2240 (2011)
73. Jiang, H., Drew, M.S., Li, Z.N.: Successive convex matching for
action detection. In: IEEE Computer Society Conf. on Computer
Vision and Pattern Recognition, vol. 2, pp. 1646–1653 (2006)
74. Joo, S.W., Chellappa, R.: Attribute grammar-based event recog-
nition and anomaly detection. In: Conference on Computer Vi-
sion and Pattern Recognition Workshop, CVPRW ’06, pp. 107–
114 (2006)
75. Kameda, Y., Minoh, M.: A human motion estimation method us-
ing 3-successive video frames. In: Proc. of Int. Conf. on Virtual
Systems, pp. 135–140 (1996)
76. Kang, W., Deng, F.: Research on intelligent visual surveillance
for public security. In: 6th Int. Conf. Comput. and Inf. Sci,
pp. 824–829. IEEE/ACIS, Melbourne (2007)
77. Ke, Y., Sukthankar, R., Hebert, M.: Spatio-temporal shape and
flow correlation for action recognition. In: IEEE Conf. on Com-
puter Vision and Pattern Recognition, pp. 1–8 (2007)
78. Khan, S.M., Shah, M.: Detecting group activities using rigidity
of formation. In: Proc. of the 13th Annual ACM Int. Conf. on
Multimedia, pp. 403–406 (2005)
79. Kim, H., Sakamoto, R., Kitahara, I., Toriyama, T., Kogure, K.:
Robust silhouette extraction technique using background sub-
traction. In: 10th Meeting on Image Recognition and Understand
(MIRU), Hiroshima, Japan, pp. 1–6 (2007)
80. Kim, J.B., Kim, H.J.: Efficient region-based motion segmen-
tation for a video monitoring system. Pattern Recognit. Lett.
24(1/3), 113–128 (2003)
81. Kim, T.K., Im, J.H., Paik, J.K.: Video object segmentation and its
salient motion detection using adaptive background generation.
IEEE Power Electron. Lett. 45(11), 542–543 (2009)
82. Ko, T.: A survey on behavior analysis in video surveillance for
homeland security applications. In: AIPR, pp. 1–8. IEEE Press,
New York (2008)
83. Kuno, Y., Watanabe, T., Shimosakoda, Y., Nakagawa, S.: Au-
tomated detection of human for visual surveillance system.
In: Proc. of the Int. Conf. on Pattern Recognition, ICPR ’96,
pp. 865–869. IEEE Computer Society, Washington (1996)
84. Ladikos, A., Benhimane, S., Navab, N.: A realtime tracking sys-
tem combining template-based and feature-based approaches. In:
VISAPP (2007)
85. Lalos, C., Anagnostopoulos, V.: Hybrid tracking approach for as-
sistive environments. In: In Int. Conf. Proc. Series, 05, vol. 39/64.
ACM Press, New York (2009)
86. Laptev, I.: On space-time interest points. Int. J. Comput. Vis.
64(2–3), 107–123 (2005)
87. Laptev, I., Lindeberg, T.: Space-time interest points. In: Proc. 9th
IEEE Int. Conf. on Computer Vision, pp. 432–439 (2003)
88. Laptev, I., Perez, P.: Retrieving actions in movies. In: Proc. of the
11th IEEE Int. Conf. on Computer Vision, pp. 1–8 (2007)
89. Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning
realistic human actions from movies. In: IEEE Conf. on Com-
puter Vision and Pattern Recognition, pp. 1–8 (2008)
90. Leordeanu, M., Collins, R.: Unsupervised learning of object fea-
tures from video sequences. In: Proc. of IEEE Computer Society
Conf. on Computer Vision and Pattern Recognition, Washington,
DC, USA, vol. 1, pp. 1142–1149 (2005)
91. Li, X., Hu, W., Zhang, Z., Zhang, X.: Robust foreground seg-
mentation based on two effective background models. In: Proc.
of the 1st ACM Int. Conf. on Multimedia Information Retrieval,
MIR ’08, pp. 223–228. ACM Press, New York (2008)
92. Liao, H.H., Chang, J.Y., Chen, L.G.: A localized approach to
abandoned luggage detection with foreground-mask sampling.
In: Proc. of the IEEE 5th Int. Conf. on Advanced Video and Sig-
nal Based Surveillance, AVSS’08, pp. 132–139. IEEE Computer
Society, Washington (2008)
93. Lin, F., Chen, B.M., Lee, T.H.: Robust vision-based target track-
ing control system for an unmanned helicopter using feature fu-
sion. In: 9th IAPR Int. Conf. on Machine Vision Applications,
vol. 13, pp. 398–401 (2009)
94. Lin, H.H., Liu, T.L., Chuang, J.H.: A probabilistic svm approach
for background scene initialization. In: Int. Conf. on Image Pro-
cessing, vol. 3, pp. 893–896 (2002)
95. Lipton, A.J.: Local application of optic flow to analyse rigid
versus non-rigid motion. http://www.eecs.lehigh.edu/FRAME/
Lipton/ieevframe.html

96. Lipton, A.J., Fujiyoshi, H., Patil, R.S.: Moving target classifica-
tion and tracking from real-time video. In: Proc. of the 4th IEEE
Workshop on Applications of Computer Vision, pp. 8–14. IEEE
Computer Society, Washington (1998)
97. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: Sift flow:
dense correspondence across different scenes. In: Proc. of the
10th European Conference on Computer Vision: Part III, pp. 28–
42. Springer, Berlin, Heidelberg (2008)
98. Liu, J., Luo, J., Shah, M.: Recognizing realistic actions from
videos in the wild. In: IEEE Int. Conf. on Computer Vision and
Pattern Recognition, pp. 1–8 (2009)
99. Lublinerman, R., Ozay, N., Zarpalas, D., Camps, O.: Activ-
ity recognition from silhouettes using linear systems and model
(in)validation techniques. In: 18th Int. Conf. on Pattern Recogni-
tion, vol. 1, pp. 347–350 (2006)
100. Lucas, B., Kanade, T.: An iterative image registration technique
with an application to stereo vision. In: Int. Joint Conf. on Artifi-
cial Intelligence, pp. 674–679. AAAI Press, Menlo Park (1981)
101. Luo, R., Li, L., Gu, I.Y.: Efficient adaptive background subtrac-
tion based on multi-resolution background modeling and updat-
ing. In: Springer-PCM, pp. 118–127. Springer, Berlin (2007)
102. Lv, F., Nevatia, R.: Single view human action recognition us-
ing key pose matching and Viterbi path searching. In: CVPR,
Minneapolis, Minnesota, USA, pp. 1–7. IEEE Computer Soci-
ety, Washington (2007)
103. Ma, X., Grimson, W.E.L.: Edge-based rich representation for ve-
hicle classification. In: Proceedings of the Tenth IEEE Interna-
tional Conference on Computer Vision, vol. 2, pp. 1185–1192.
IEEE Computer Society, Washington (2005)
104. McHugh, J.M., Konrad, J., Saligrama, V., Jodoin, P.M.:
Foreground-adaptive background subtraction. IEEE Signal Pro-
cess. Lett. 16(5), 390–393 (2009)
105. Meyer, F., Bouthemy, P.: Region-based tracking using affine mo-
tion models in long image sequences. CVGIP, Image Underst.
60(2), 119–140 (1994)
106. Migdal, J., Grimson, W.E.L.: Background subtraction using
Markov thresholds. In: Proc. of the IEEE Workshop on Motion
and Video Computing (WACV/MOTION’05), WACV-MOTION
’05, vol. 2, pp. 58–65. IEEE Computer Society, Washington
(2005)
107. Minnen, D., Essa, I., Starner, T.: Expectation grammars: leverag-
ing high-level expectations for activity recognition. In: Proceed-
ings IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, 2003, vol. 2, pp. 626–632 (2003)
108. Moeslund, T.B., Granum, E.: A survey of computer vision-based
human motion capture. Comput. Vis. Image Underst. 81(03),
231–268 (2001)
109. Moeslund, T.B., Hilton, A., kruger, V.: A survey of advances in
vision-based human motion capture and analysis. Comput. Vis.
Image Underst. 104(2–3), 90–126 (2006)
110. Mohan, A., Papageorgiou, C., Poggio, T.: Example-based object
detection in images by components. IEEE Trans. Pattern Anal.
Mach. Intell. 23(4), 349–361 (2001)
111. Monnet, A., Mittal, A., Paragios, N., Ramesh, V.: Background
modeling and subtraction of dynamic scenes. In: Proc. 9th IEEE
Int. Conf. on Computer Vision, vol. 2, pp. 1305–1312 (2003)
112. Moore, D., Essa, I.: Recognizing multitasked activities from
video using stochastic context-free grammar. In: Proc. AAAI
National Conf. on AI, pp. 770–776. AAAI Press, Menlo Park
(2002)
113. Moore, D.J., Essa, I.A., Hayes, M.H.: Exploiting human actions
and object context for recognition tasks. In: Proc. of 7th IEEE
Int. Conf. on Computer Vision, vol. 1, pp. 80–86 (1999)
114. Morris, B.T., Trivedi, M.M.: A survey of vision-based trajectory
learning and analysis for surveillance. IEEE Trans. Circuits Syst.
Video Technol. 18(08), 1114–1127 (2008)
S. Vishwakarma, A. Agrawal
115. Narayana, M., Haverkamp, D.: A Bayesian algorithm for track-
ing multiple moving objects in outdoor surveillance video. In:
CVPR, pp. 1–8. IEEE Press, New York (2007)
116. Natarajan, P., Nevatia, R.: Coupled hidden semi Markov mod-
els for activity recognition. In: IEEE Workshop on Motion and
Video Computing, pp. 1–8 (2007)
117. Nevatia, R., Hobbs, J., Bolles, B.: An ontology for video event
representation. In: IEEE Conf. on Computer Vision and Pattern
Recognition Workshop, pp. 119–128 (2004)
118. Nevatia, R., Zhao, T., Hongeng, S.: Hierarchical language-based
representation of events in video streams. In: Conf. on Com-
puter Vision and Pattern Recognition Workshop, vol. 4, pp. 39–
47 (2003)
119. Nguyen, N.T., Phung, D.Q., Venkatesh, S., Bui, H.: Learning and
detecting activities from movement trajectories using the hierar-
chical hidden Markov model. In: IEEE Computer Society Conf.
on Computer Vision and Pattern Recognition, vol. 2, pp. 955–960
(2005)
120. Niebles, J.C., Wang, H., Fei-fei, L.: Unsupervised learning of
human action categories using spatial-temporal words. In: Proc.
British Machine Vision Conference (BMVC) (2006)
121. Niethammer, M., Tannenbaum, A., Angenent, S.: Dynamic ac-
tive contours for visual tracking. IEEE Trans. Autom. Control
51(4), 562–579 (2006)
122. Nowozin, G.S., Bakir, G., Tsuda, K.: Discriminative subse-
quence mining for action classification. In: ICCV, vol. 11, pp. 1–
8. IEEE Press, New York (2007)
123. Ogale, A.S., Karapurkar, A., Aloimonos, Y.: View-invariant
modeling and recognition of human actions using grammars. In:
10th Conf. on Category Curve of Long Video, vol. 10, pp. 115–
126, Beijing, China. IEEE Press, New York (2005)
124. Oh, S., Hoogs, A., et al.: A large-scale benchmark dataset for
event recognition in surveillance video. In: Proc. of IEEE Int.
Conf. on Computer Vision and Pattern Recognition, pp. 3153–
3160 (2011)
125. Oikonomopoulos, A., Patras, I., Pantic, M., Paragios, N.:
Trajectory-based representation of human actions. In: Artificial
Intelligence for Human Computing, vol. 4451, pp. 133–154.
Springer, Berlin (2007)
126. Oikonomopoulos, A., Patras, I., Pantici, M.: Spatiotemporal
salient points for visual recognition of human actions. IEEE
Trans. Syst. Man Cybern. 36(3), 710–719 (2006)
127. Oliver, N.M., Rosario, B., Pentland, A.P.: A Bayesian computer
vision system for modeling human interactions. IEEE Trans. Pat-
tern Anal. Mach. Intell. 22(8), 831–843 (2000)
128. Oliver, N., Horvitz, E., Garg, A.: Layered representations for hu-
man activity recognition. In: Proc. 4th IEEE Int. Conf. on Multi-
modal Interfaces, pp. 3–8 (2002)
129. Ong, E.J., Gong, S.: The dynamics of linear combinations: track-
ing 3d skeletons of human subjects. Image Vis. Comput. 20(5/6),
397–414 (2002)
130. Paragios, N., Deriche, R.: Geodesic active contours and level sets
for the detection and tracking of moving objects. IEEE Trans.
Pattern Anal. Mach. Intell. 22(3), 266–280 (2000)
131. Paragios, R., Stenger, B., Ramesh, V., Paragios, N., Buhmann,
F.C.J.: Topology free hidden Markov models: application to
background modeling. In: IEEE Int. Conf. on Computer Vision,
pp. 294–301 (2001)
132. Parameswaran, V., Chellappa, R.: View invariance for human ac-
tion recognition. Int. J. Comput. Vis. 66(1), 83–101 (2006)
133. Parameswaran, V., Singh, M., Ramesh, V.: Illumination compen-
sation based change detection using order consistency. In: IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
pp. 1982–1989 (2010)
134. Parikh, D., Zitnick, C.L., Chen, T.: Unsupervised learning of hi-
erarchical spatial structures in images. In: Proc. of IEEE Conf.
on Computer Vision and Pattern Recognition, pp. 1–8 (2009)
A survey on activity recognition & behavior understanding in video surveillance
135. Park, S., Aggarwal, J.K.: A hierarchical Bayesian network for
event recognition of human actions and interactions. Assoc.
Comput. Mach. Multimedia Syst. J., 164–179 (2004)
136. Paruchuri, J.K., Sathiyamoorthy, E.P., Ching, S., Cheung, S.,
Chen, C.H.: Spatially adaptive illumination modeling for back-
ground subtraction. In: IEEE Int. Conf. on Computer Vision
Workshops (ICCV Workshops), pp. 1745–1752 (2011)
137. Pentland, A.: Smart rooms, smart clothes. In: Proc. Fourteenth
Int. Conf. on Pattern Recognition, vol. 2, pp. 949–953 (1998)
138. Peursum, P., West, G., Venkatesh, S.: Combining image regions
and human activity for indirect object recognition in indoor wide-
angle views. In: 10th IEEE Int. Conf. on Computer Vision, vol. 1,
pp. 82–89 (2005)
139. Pilet, J., Strecha, C., Fua, P.: Making background subtraction ro-
bust to sudden illumination changes. In: Proc. European Conf. on
Computer Vision, pp. 1–14 (2008)
140. Pinhanez, C.S., Bobick, A.F.: Human action detection using pnf
propagation of temporal constraints. In: Proc. IEEE Computer
Society Conf. on Computer Vision and Pattern Recognition,
pp. 898–904 (1998)
141. Platt, J.C.: Fast training of support vector machines using sequen-
tial minimal optimization. In: Proceedings of Advances in Ker-
nel Methods—Support Vector Learning, pp. 185–208. Microsoft,
Redmond (1998)
142. Poppe, R.: A survey on vision-based human action recognition.
Image Vis. Comput. 28, 976–990 (2010)
143. Porikli, F., Ivanov, Y., Haga, T.: Robust abandoned object detec-
tion using dual foregrounds. EURASIP J. Adv. Signal Process.
08, 197875 (2008)
144. Qi, Y., An, G.: Infrared moving targets detection based on op-
tical flow estimation. In: Proc. of IEEE Int. Conf. on Computer
Science and Network Technology, pp. 2452–2455 (2011)
145. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan
Kaufmann, San Francisco (1999)
146. Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Be-
longie, S.: Objects in context. In: Proc. of the IEEE 11th Int.
Conf. on Computer Vision, pp. 1–8 (2007)
147. Rao, C., Shah, M.: View-invariance in action recognition. In:
Proc. of IEEE Computer Society Conf. on Computer Vision and
Pattern Recognition, vol. 2, pp. 316–322 (2001)
148. Reddy, V., Sanderson, C., Sanin, A., Lovell, B.C.: Adaptive
patch-based background modelling for improved foreground ob-
ject segmentation and tracking. In: 7th IEEE Int. Conf. on Ad-
vanced Video and Signal Based Surveillance (AVSS), pp. 172–
179 (2010)
149. Ren, Y., Chua, C.S.: Bilateral learning for color-based tracking.
Image Vis. Comput. 26(11), 1530–1539 (2008)
150. Rodriguez, M.D., Ahmed, J., Shah, M.: Action MACH: a spatio-
temporal maximum average correlation height filter for action
recognition. In: CVPR. IEEE Press, New York (2008)
151. Rui, Y., Huang, T.S.: Image retrieval: current techniques, promis-
ing directions and open issues. J. Vis. Commun. Image Repre-
sent. 10, 39–62 (1999)
152. Ryoo, M.S., Aggarwal, J.K.: Recognition of composite human
activities through context-free grammar based representation. In:
IEEE Computer Society Conf. on Computer Vision and Pattern
Recognition, vol. 2, pp. 1709–1718 (2006)
153. Ryoo, M.S., Aggarwal, J.K.: Hierarchical recognition of human
activities interacting with objects. In: IEEE Conf. on Computer
Vision and Pattern Recognition, pp. 1–8 (2007)
154. Ryoo, M.S., Aggarwal, J.K.: Recognition of high-level group
activities based on activities of individual members. In: IEEE
Workshop on Motion and Video Computing, pp. 1–8 (2008)
155. Ryoo, M.S., Aggarwal, J.K.: Semantic representation and recog-
nition of continued and recursive human activities. Int. J. Com-
put. Vis. 82, 1–24 (2009)
156. Ryoo, M.S., Aggarwal, J.K.: Spatio-temporal relationship match:
video structure comparison for recognition of complex hu-
man activities. In: IEEE 12th Int. Conf. on Computer Vision,
pp. 1593–1600 (2009)
157. Ryoo, M.S., Aggarwal, J.K.: UT-Interaction Dataset, ICPR
contest on Semantic Description of Human Activities (SDHA).
http://cvrc.ece.utexas.edu/SDHA2010/Human_Interaction.html
(2010)
158. Ryoo, M.S., Chen, C.C., Aggarwal, J.K., Roy-Chowdhury, A.:
An overview of contest on semantic description of human ac-
tivities 2010. In: Proc. Int. Conf. Pattern Recognition Contests,
pp. 1–16 (2010)
159. Sakaino, H.: A semitransparency-based optical-flow method with
a point trajectory model for particle-like video. IEEE Trans. Im-
age Process. 21(2), 441–450 (2012)
160. Salembier, P., Marques, F.: Region-based representations of im-
age and video: segmentation tools for multimedia services. IEEE
Trans. Circuits Syst. Video Technol. 9(8), 1147–1169 (1999)
161. Sarkar, S., Phillips, P.J., Liu, Z., Vega, I.R., Grother, P., Bowyer,
K.W.: The humanoid gait challenge problem: data sets, perfor-
mance, and analysis. IEEE Trans. Pattern Anal. Mach. Intell.
27(2), 162–177 (2005)
162. Schmaltz, C., Rosenhahn, B., Brox, T., Weickert, J.: Localised
mixture models in region-based tracking. In: Proc. of the 31st
DAGM Symposium on Pattern Recognition, pp. 21–30. Springer,
Berlin (2009)
163. Schmaltz, C., Rosenhahn, B., Brox, T., Weickert, J.: Region-
based pose tracking with occlusions using 3D models. Mach. Vis.
Appl. 23(3), 557–577 (2012)
164. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions:
a local SVM approach. In: Proc. IEEE Computer Society Pattern
Recognition, vol. 3, pp. 32–36. IEEE Computer Society Press,
Los Alamitos (2004)
165. Schunck, B.: The image flow constraint equation. Comput. Vis.
Graph. Image Process. 35(1), 20–46 (1986)
166. Schunck, B., Horni, B.: Determining optical flow. In: DARPA81,
pp. 144–156 (1981)
167. Sclaroff, S., Isidoro, J.: Active blobs: region-based, deformable
appearance models. Comput. Vis. Image Underst. 89(2/3), 197–
225 (2003)
168. Senst, T., Evangelio, R.H., Sikora, T.: Detecting people carrying
objects based on an optical flow motion model. In: IEEE Work-
shop on Applications of Computer Vision, pp. 301–306 (2011)
169. Shechtman, E., Irani, M.: Space-time behavior based correlation.
In: IEEE Computer Society Conf. on Computer Vision and Pat-
tern Recognition, vol. 1, pp. 405–412 (2005)
170. Sheikh, Y., Javed, O., Kanade, T.: Background subtraction for
freely moving cameras. In: IEEE 12th Int. Conf. on Computer
Vision, pp. 1219–1225 (2009)
171. Sheikh, Y., Sheikh, M., Shah, M.: Exploring the space of a human
action. In: Tenth IEEE Int. Conf. on Computer Vision, vol. 1,
pp. 144–149 (2005)
172. Shi, J., Tomasi, C.: Good features to track. In: CVPR, pp. 593–
600. IEEE Computer Society, Washington (1994)
173. Shi, Y., Huang, Y., Minnen, D., Bobick, A., Essa, I.: Propagation
networks for recognition of partially ordered sequential action.
In: Proc. of IEEE Computer Society Conf. on Computer Vision
and Pattern Recognition, vol. 2, pp. 862–869 (2004)
174. Shibata, M., Yasuda, Y., Ito, M.: Moving object detection for ac-
tive camera based on optical flow distortion. In: Proc. of the 17th
World Congress the International Federation of Automatic Con-
trol, Seoul, Korea, pp. 14,720–14,725 (2008)
175. Siskind, J.M.: Grounding the lexical semantics of verbs in visual
perception using force dynamics and event logic. J. Artif. Intell.
Res. 15, 31–90 (2001)
176. Sivic, J., Zisserman, A.: Video data mining using configura-
tions of viewpoint invariant regions. In: Proc. of the IEEE Conf.

on Computer Vision and Pattern Recognition, Washington, DC,
pp. 1–8 (2004)
177. Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and
TRECVid. In: Proc. of the 8th ACM Int. Workshop on Multi-
media Information Retrieval, Santa Barbara, California, USA,
pp. 321–330 (2006)
178. Starner, T., Pentland, A.: Real-time American sign language
recognition from video using hidden Markov models. In:
Proceedings International Symposium on Computer Vision,
pp. 265–270 (1995)
179. Stauffer, C.: Automatic hierarchical classification using time-
based co-occurrences. In: IEEE Int. Conf. on Computer Vision
and Pattern Recognition, vol. 2, pp. 333–339 (1999)
180. Stauffer, C., Grimson, W.E.L.: Learning patterns of activity using
real-time tracking. IEEE Trans. Pattern Anal. Mach. Intell. 22(8),
747–757 (2000)
181. Tavakkoli, A., Nicolescu, M., Bebis, G.: A novelty detection
approach for foreground region detection in videos with quasi-
stationary backgrounds. In: Proc. of the 2nd Int. Symposium
on Visual Computing, pp. 40–49. Springer, Berlin, Heidelberg
(2006)
182. Techmer, A.: Contour-based motion estimation & object tracking
for real-time applications. In: IEEE Image Processing Proceed-
ings, vol. 3, pp. 648–651 (2001)
183. Thi, T.H., Zhang, J., Cheng, L., Wang, L., Satoh, S.: Semi-
supervised human action recognition and localization us-
ing spatially and temporally integrated local features (2009).
http://huetuan.net/semiaction.html
184. Trec Video Retrieval Evaluation Official Website. http://huetuan.
net/semiaction.html
185. Tsai, D.M., Lai, S.C.: Independent component analysis-based
background subtraction for indoor surveillance. IEEE Trans. Im-
age Process. 18(1), 158–167 (2009)
186. Tsuchiya, M., Fujiyoshi, H.: Evaluating feature importance for
object classification in visual surveillance. In: Proc. of the 18th
Int. Conf. on Pattern Recognition, vol. 2, pp. 978–981. IEEE
Computer Society, Washington (2006)
187. Valera, M., Velastin, S.A.: Intelligent distributed surveillance
systems: a review. IEE Proc., Vis. Image Signal Process. 152(2),
192–204 (2005)
188. Varcheie, P.D.Z., Sills-Lavoie, M., Bilodeau, G.A.: A multiscale
region-based motion detection and background subtraction algo-
rithm. Sensors 10, 1041–1061 (2010)
189. Vaswani, N., Chowdhury, A.R., Chellappa, R.: Activity recogni-
tion using the dynamics of the configuration of interacting ob-
jects. In: Proc. IEEE Computer Society Conf. on Computer Vi-
sion and Pattern Recognition, vol. 2, pp. 633–640 (2003)
190. Vaswani, N., Chowdhury, A.R., Chellappa, R.: Shape activity:
a continuous state HMM for moving/deforming shapes with ap-
plication to abnormal activity detection. IEEE Trans. Image Pro-
cess. 14(10), 1603–1616 (2005)
191. Veeraraghavan, A., Chellappa, R., Roy-Chowdhury, A.K.: The
function space of an activity. In: IEEE Computer Society Conf.
on Computer Vision and Pattern Recognition, vol. 1, pp. 959–968
(2006)
192. Vishwakarma, S., Agrawal, A.: A novel approach for feature
quantization using one-dimensional histogram. In: Annual IEEE
India Conference (INDICON), pp. 1–4 (2011)
193. Vishwakarma, S., Sapre, A., Agrawal, A.: Action recognition
using cuboids of interest points. In: IEEE Int. Conf. on Signal
Processing, Communications and Computing (ICSPCC), pp. 1–
6 (2011)
194. Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated
mesh animation from multi-view silhouettes. ACM Trans. Graph.
27(3), 97:1–97:9 (2008)
S. Vishwakarma, A. Agrawal
195. Vogler, C., Metaxas, D.: Parallel hidden Markov models for
American sign language recognition. In: IEEE Int. Conf. on
Computer Vision, vol. 1, pp. 224–228 (1999)
196. Vosters, L., Shan, C., Gritti, T.: Background subtraction under
sudden illumination changes. In: 7th IEEE Int. Conf. on Ad-
vanced Video and Signal Based Surveillance (AVSS), pp. 384–
391 (2010)
197. Vu, V.-T., Bremond, F., Thonnat, M.: Automatic video interpre-
tation: a novel algorithm for temporal scenario recognition. In:
Proc. 8th Int. Joint Conf. Artif. Intell, pp. 9–15 (2003)
198. Waltisberg, D., Yao, A., Gall, J., Gool, L.V.: Variations of a
hough-voting action recognition system. In: Proc. of Int. Conf.
on Pattern Recognition, pp. 1–7 (2010)
199. Wang, J., Bebis, G., Miller, R.: Robust video-based surveillance
by integrating target detection with tracking. In: Proc. Conf. on
Computer Vision and Pattern Recognition Workshop, CVPRW
’06, pp. 137–144. IEEE Computer Society, Washington (2006)
200. Weber, M.: Unsupervised learning of models for object recogni-
tion. Ph.D. thesis, California Institute of Technology, Pasadena,
California (2000)
201. Weinland, D., Boyer, E., Ronfard, R.: Action recognition from
arbitrary views using 3D exemplars. In: ICCV, Rio de Janeiro,
Brazil, vol. 11, pp. 1–7. IEEE Computer Society Press, Los
Alamitos (2007)
202. Weinland, D., Ronfard, R., Boyer, E.: Automatic discovery
of action taxonomies from multiple views. In: CVPR, vol. 2,
pp. 1639–1645. IEEE Computer Society, Washington (2006)
203. Weinland, D., Ronfard, R., Boyer, E.: Free viewpoint action
recognition using motion history volumes. Comput. Vis. Image
Underst. 104(02), 249–257 (2006)
204. Weinland, D., Ronfard, R., Boyer, E.: A survey of vision-based
methods for action representation, segmentation and recognition.
Comput. Vis. Image Underst. 115, 224–241 (2011)
205. Wen, Z., Cai, Z.: A robust object tracking approach using mean
shift. In: 3rd IEEE Int. Conf. on Natural Computation, vol. 2,
pp. 170–174 (2007)
206. Wong, S.F., Cipolla, R.: Extracting spatiotemporal interest points
using global information. In: ICCV, vol. 11, pp. 1–8. IEEE Press,
New York (2007)
207. Wong, S.F., Kim, T.K., Cipolla, R.: Learning motion categories
using both semantic and structural information. In: IEEE Conf.
on Computer Vision and Pattern Recognition, pp. 1–6 (2007)
208. Wunsch, P., Hirzinger, G.: Real-time visual tracking of 3D ob-
jects with dynamic handling of occlusion. In: Int. Conf. on
Robotics and Automation, 97, Albuquerque, New Mexico, USA,
vol. 4, pp. 2868–2879 (1997)
209. Xiang, T.: Video behavior profiling for anomaly detection. IEEE
Trans. Pattern Anal. Mach. Intell. 30(5), 893–908 (2008)
210. Xiao, J., Cheng, H., Han, F., Sawhney, H.: Geo-spatial aerial
video processing for scene understanding and object tracking. In:
CVPR, pp. 1–8. IEEE Press, New York (2008)
211. Xu, M., Zuo, L., Iyengar, S., Goldfain, A., DelloStritto, J.:
A semi-supervised hidden Markov model-based activity moni-
toring system. In: 33rd Annual Int. Conf. of the IEEE Engineer-
ing in Medicine and Biology Society (EMBC), Boston, Mas-
sachusetts USA, pp. 1794–1797 (2011)
212. Yacoob, Y., Black, M.J.: Parameterized modeling and recogni-
tion of activities. In: 6th Int. Conf. on Computer Vision, pp. 120–
127 (1998)
213. Yamato, J., Ohya, J., Ishii, K.: Recognizing human action in
time-sequential images using hidden Markov model. In: Proc.
IEEE Computer Society Conf. on Computer Vision and Pattern
Recognition, pp. 379–385 (1992)
214. Yamazaki, M., Xu, G., Chen, Y.W.: Detection of moving ob-
jects by independent component analysis. In: Proc. of the 7th
Asian Conf. on Computer Vision, ACCV’06, vol. 2, pp. 467–
478. Springer, Berlin, Heidelberg (2006)
A survey on activity recognition & behavior understanding in video surveillance

215. Yang, F., Li, B.: Unsupervised learning of spatial structures 231. Zhu, Y., Dariush, B., Fujimura, K.: Kinematic self retargeting: a
shared among images. Vis. Comput. 28(2), 175–180 (2011) framework for human pose estimation. Comput. Vis. Image Un-
216. Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM derst. 114(12), 1362–1375 (2010)
Comput. Surv. 38(4), 1–45 (2006)
217. Yilmaz, A., Li, X., Shah, M.: Contour-based object tracking Sarvesh Vishwakarma received
with occlusion handling in video acquired using mobile cam- the B.Tech. degree in electron-
eras. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1531–1536 ics and communication engineering
(2004) from the University Institute of En-
218. Yilmaz, A., Shah, M.: Actions sketch: a novel action represen- gineering and Technology, Kanpur,
tation. In: CVPR, vol. 1, pp. 984–989. IEEE Computer Society, India, in 2001, and the M.Tech. de-
Washington (2005) gree in computer science and en-
219. Yohannes, Y., Hoddinott, J.: Classification and regression trees. gineering from the Indian Institute
Tech. rep., International Food Policy Research Institute, Wash- of Technology, Roorkee, India, in
ington, DC, USA (1999) 2003. He is currently working to-
220. Yokoyama, M., Poggio, T.: A contour-based moving object de- ward the Ph.D. degree in informa-
tection and tracking. In: 2nd Joint IEEE Int. Workshop on Vi- tion technology with the Indian In-
sual Surveillance and Performance Evaluation of Tracking and stitute of Information Technology,
Surveillance, pp. 271–276 (2005) Allahabad, India. His research in-
221. Yu, E., Aggarwal, J.K.: Detection of fence climbing from monoc- terests include computer vision, im-
ular video. In: 18th Int. Conf. on Pattern Recognition, vol. 1, age processing, pattern recognition, and artificial intelligence applied
pp. 375–378 (2006) to unusual activity analysis and surveillance applications. He is a mem-
222. Yu, T.H., Kim, T.K., Cipolla, R.: Real-time action recognition ber of the IEEE Computer Society.
by spatiotemporal semantic and structural forests. In: Proc. of
British Machine Vision Conference, pp. 1–7 (2010)
223. Zelnik-Manor, L., Irani, M.: Event-based analysis of video. In: Anupam Agrawal received his
Proc. of IEEE Computer Society Conf. on Computer Vision and M.Sc. degree in computer science
Pattern Recognition, vol. 2, pp. 123–130 (2001) from the J.K. Institute of Applied
224. Zhan, B., Monekosso, D.N., Remagnino, P., Velastin, S.A., Xu, Physics, Allahabad University in
L.Q.: Crowd analysis: a survey. Mach. Vis. Appl. 19(5–6), 345– 1988, his M.Tech. degree in com-
357 (2008) puter science and engineering from
225. Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I.: Model- Indian Institute of Technology
ing individual and group actions in meetings with layered hmms. Madras, Chennai in 1995, and his
IEEE Trans. Multimed. 8(3), 509–520 (2006) Ph.D. degree in information tech-
226. Zhang, J., Tian, Y., Yang, Y.: Adaptive dynamic model particle nology from the Indian Institute
filter for visual object tracking. In: ISECS International Collo- of Information Technology Alla-
quium, vol. 1, pp. 333–336. IEEE Press, New York (2009) habad in 2006. He was a post-
227. Zhang, L., Li, S.Z., Yuan, X., Xiang, S.: Real-time object classi- doctoral researcher at the depart-
fication in video surveillance based on appearance learning. In: ment of computing and information
IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1– system, University of Bedfordshire
8 (2007) (UK). Earlier, he was working as a scientist “D” at DEAL, DRDO,
228. Zhao, Y., Gong, H., Lin, L., Jia, Y.: Spatio-temporal patches for Govt. of India, Dehradun. Presently, he is a Professor at the department
night background modeling by subspace learning. In: 19th Int. of information technology at Indian Institute of Information Technol-
Conf. on Pattern Recognition, pp. 1–4 (2008) ogy, Allahabad in India. His research interests include computer vision,
229. Zhong, H., Shi, J., Visontai, M.: Detecting unusual activity in image processing, medical image processing, multimedia, and graph-
video. In: Proc. of IEEE Computer Society Conf. on Computer ics. He has more than 75 publications on these areas in international
Vision and Pattern Recognition, vol. 2, pp. 819–826 (2004) journals and conference proceedings, and has authored one book. He is
230. Zhou, S.K., Chellappa, R., Moghaddam, B.: Visual tracking and a senior member of IEEE and a fellow of the Institution of Electronics
recognition using appearance-adaptive models in particle filters. and Telecommunication Engineers. He is serving as a Chairman of the
IEEE Trans. Image Process. 13(11), 1491–1506 (2004) ACM Chapter, IIIT-A.