Adaptive algorithm to identify anomalies in moving objects
using computer vision
Paola A. Mateus
Pontifical Xaverian University
mateusp@javeriana.edu.co
Abstract - In this document, an adaptive algorithm is
proposed to identify anomalies in moving objects, such as
pedestrians, cars, motorcyclists and cyclists. The anomalies
detected by this algorithm are: occlusions, an object par-
tially entering or exiting a frame, alterations in the object
velocities or collisions between two or more objects in the
frame. The classification between frames with anomaly and
no anomaly is achieved by finding an adaptive threshold
that depends on the video sequences.
I. I NTRODUCTION
Nowadays, one of the main problems of video surveil-
lance is the detection of anomalies while monitoring mul-
tiple cameras at the same time. To solves this problems
this project proposes and develops an adaptive algorithm
that works in uncontrolled scenes and automatically
detects a variety of anomalies, such as occlusions,drastic
changes in the velocities of the objects or at least two
objects colliding in the same frame.
The remainder of this work is organized as follows:
In Section II, related works are described. In Section III,
a summary of the diferente techniques in the algorithm
is given. Some of these techniques are background sub-
straction, apparent mass and kinetic energy estimation,
recursive least squares and an unsupervised classifier.
Finally, the experimental results on different video clips
are presented in Section IV.
II. R ELATED W ORK
Different works have presented solutions to ensure
anomaly detection in surveillance videos. Some of these
solutions are based on detection, segmentation and track-
ing [1-6]. Other projects [7-9] used temporal features,
such as, local velocity and local movement. Finally,
in projects [10] and [11] two kinds of anomalies are
detected through kinetic energy when the people are
exiting the boundary of the frame or when the frame
has crowded scenes.
All the aforementioned method use complex opera-
tions such as object tracking with a high numbers of tra-
jectories and people counting through the video, to name
978-1-5090-2532-9/16/$31.00 c 2016 IEEE
Cesar L. Nino
Pontifical Xaverian University
cesar.nino@javeriana.edu.co
just a few, which only offer a partial limited solution to
the problem of detect anomaly in the video. Also, many
of these projects make tests in controlled settings, where
a scene is arranged and labelled as anomalous, and then,
the algorithm detects this behavior when a similar scene
takes place.
III. S YSTEM OVERVIEW
A. Segmentation of Moving Objects
The correct segmentation of the moving objects in
each frame is very important to obtain the relevant
kinetic energy and area throughout the video. These two
features allow the detection of the different changes and
anomalies within that frame.
The segmentation of the foreground extraction was
developed as follows:
The first step is the use of three binarization tech-
niques: Gaussian Mixture Modelling [12] (GMM) (see
Figure 1b), Morphological Reconstruction [13] (see Fig-
ure 1d) and Motion Detection (see Figure 1b).
The second step involves an AND operation between
each pair of possible combinations of the three images
obtained from the previous step (see Figures 1e-g).
In the third and final step, the OR operation is pro-
posed in order to get a unique binary image I between
all the images obtained previously (see Figure 1h).
B. Feature Extraction
In order to identify the anomalies in the video, two
features are proposed: the apparent mass and the kinetic
energy.
The apparent mass is proposed knowing that the
volume of the object is a physical and solid characteristic
that should not have changed throughout the video. The el objeto solo cambia
cuando hay oclusiones
drastic changes in this feature can only occur when there o una entrada o salida
are occlusions or the partially entry or exit of an object parcial de un objeto en
el frame
in the frame.

Figure 1. (a) Original image. Foreground detection with: (b) Gaussian
Mixture Modeling, (c) Motion detection, (d) Morphological reconstruc-
tion. The resulting images from the AND operation between: (e) GMM
and motion detection, (f) GMM and morphological reconstruction and
(g) Motion detection and morphological reconstruction. (h) Foreground
detection resulting from the combination of the three methods.
Drastic changes in the kinetic energy allow detecting
alterations in the velocities of the objects or collisions
between two or more objects in the frame.
To calculate the first feature (apparent mass) is nec-
essary to obtain the volume estimation with the area
correction of each moving object through the vanishing
point. The vanishing point is found by the detection
of two parallel lines in the original RGB image. Then,
the distance di is calculated from the centroid of each
moving object i to the vanishing point.
With this distance, the apparent mass is attained as:
n
X the total kinetic energy, g is the vector of the buffered
M (u, v) = di Ii (u, v), (1)
i=1
where, M (u, v) represents the mass apparent matrix
where the background is equal to zero and the foreground
has positive values different of zero. The values of the
foreground correspond to the distances di of each moving
object i, n is the total number of moving objects in the
frame, Ii is the (U, V ) matrix of the binary image with
only the i-th moving object, U and V are the height and
width values of the real image respectively.
In order to estimate the second feature (kinetic en-
ergy), a pixel-by-pixel moving window P (i, j) is pro-
posed to make an horizontal sweep of the frame. This
window is created by selecting a neighbourhood of IxJ
pixels from the matrix of apparent mass M (u, v). This
neighbourhood is centered around a pixel of interest,
being (i, j) the pixel coordinates in the matrix.
The theory [11] denotes that the foreground entropy
H(u, v) is the dispersion of the foreground on the
horizontal and vertical directions. Therefore, the new
mass apparent matrix that use these foreground entropy
is defined as Crowd Dispersion Index and it is calculated
by the following:
Pu+ U2 P U2
( i=u− U
j=v− U
P (i, j))2
2 2
CDI(u, v) = , (2)
H(u, v)3
Finally, the kinetic energy of each frame is defined as
follows:
u+ U
2 2
U
X X
2
Ek(u, v) = CDI(u, v) ∗ vij , (3)
i=u− U
2 j=v− U
2
where vij is the velocity obtained through a Thomas
Brox [14] optical flow algorithm.
C. Recursive Least Square Filters
Two recursive least squares filters are proposed. One
to estimate the total apparent mass and the other one to
estimate the total kinetic energy in the frame k.
The RLS filter is calculated as:
ŷ(k) = wT (k − 1)g(k) (4)
where, ŷ could be the estimation of total apparent
mass M̂ t or the estimation of total kinetic energy Ekt ˆ
depending on the needed filter, w(k−1) are the respective
filter coefficients: 4 for the total apparent mass and 8 for
input samples at the frame k being g = M t for the total
apparent mass or g = Ekt for the total kinetic energy.
Two instantaneous error are calculated to know if there
are drastic changes in the total apparent mass and the
total kinetic energy respectively.
Taking into account that the proposed RLS filters track
the real signal of the total apparent mass and the total
kinetic energy throughout the video, it is supposed that
the absolute error is lognormal. Therefore, the calculated
histogram by the log error has a normal distribution when
there are not anomalies.
 Resolution Frames
Two frame-by-frame moving windows are used to Video 1 240x352 30/s
detect a classification threshold that adapts to different Video 2 360x640 30/s
scenes in the video. One window is linked to the total Video 3 360x528 30/s
Video 4 252x320 30/s
apparent mass instantaneous error and the other one to Table I
the total kinetic energy instantaneous error. This classifi- E ACH VIDEO SPECIFICATIONS USED BY THE ALGORITHM .
cation method is unsupervised since there is no previous
training.
These moving windows select the data from each error
and then, a log-normal histogram with an absolute error Figure 2 highlights drastic changes in the total kinetic
distribution for each of them is created. energy or total apparent mass with a red color and minor
A probability density function (PDF) that fits each changes in these features with a yellow color.
histogram has to be found. The PDFs that fits best the
different distribution of the histograms are: First, when
the scene has no anomalies, the proposed (RLS) filter
has soft changes making the histogram a lognormal dis-
tribution one. Therefore, this distribution is modelled as
a Gaussian PDF. Second, when the scene has anomalies
the histogram changes its distribution (not a lognormal
one) making the absolute error value to be modelled as
a Gaussian Mixture PDF.
A Gaussian mixture distributions, if the algorithm
detects a dataset which contains changes in the scenes
(anomaly or non-anomaly in the frame) and a Gaussian
distribution, if the dataset has only samples without any
anomaly in the apparent mass or the kinetic energy. The
term anomaly makes reference to a drastic change in the
total apparent mass or in the total kinetic energy.
In order to know wich of Gaussian PDF or Gaussian Figure 2. Processed images of Video 1 showing drastic changes in
mixture PDF most fit the histogram a Kullback–Leibler total kinetic energy or total apparent mass.
divergence is calculated for both of them:
Finally, if the Gaussian Mixture PDF is the one that The samples behave throughout all the Video 1 as
fit the histogram, the threshold thr is calculated as: shown in Figure 3, where the blue samples are the frames
that have normal behavior, the green ones are the frames
√ µ1 σ22 + µ2 σ12 that have changed in the total kinetic energy and the
thr = γ+ (5)
σ12 + σ22 red ones are the frames that have a change in the total
where, apparent mass.
In Table 2 the results obtained with the unsupervised
γ = σ12 σ22 (ln(a1)(σ12 σ22 ) − (µ1 − µ2 )2 classifier for each video are shown.
+ (σ12 + σ22 )ln(a2)) (6) Non-anomaly Anomaly VPR SPC ACC
Video 1 945 235 81.66% 71.66% 73.19%
Video 2 621 249 80.77% 78.26% 78.66%
Video 3 1466 307 77.78% 90.76% 88.36%
Video 4 947 268 84.00% 81.82% 82.17%
where µ1 and µ2 are the means of the first and second Table II
Gaussian distribution respectively, σ1 and σ2 are the S ENSIBILITY, SPECIFICITY AND PRECISION FOR THE FOUR VIDEOS .

standard deviation of each distribution.

IV. E XPERIMENTAL R ESULTS In Figure 4, the Receiver Operating Characteristic

The adaptive algorithm was evaluated in four videos
in outdoor areas. Using surveillance cameras, different
videos with occlusions, shadows and lightning conditions
that vary throughout the day were obtained. Each of them
contains two features (anomaly and non-anomaly) with
the specifications in Table 1.
curve was drawn from the sensibility and specificity.
V. C ONCLUSIONS
The proposed combination of the three methods deliv-
ers a better binary segmentation than the ones obtained
from each independent method. Drastic changes in the
total apparent mass occur when there are occlusions or an
object partially enters or exits a frame. Since the volume
of an object is a physical and solid characteristic, it
should not change throughout the video. Drastic changes
in the kinetic energy allow to detect alterations in the
object velocities or collisions between two or more
objects in the frame. The pixel-by-pixel windows used
in the kinetic energy allow to identify the specific pixels
that change in the frame. The two PDFs proposed in this
work (Gaussian Mixture PDF and Gaussian PDF) are the
best fit for the histograms created by the instantaneous
log error of the total apparent mass and the total kinetic
energy. The algorithm was tested in a ground truth that
contains four videos with 3079 frames with no anomaly
and 1059 frames with anomaly.
Figure 3. Blue samples are the frames that have not anomaly, the
green ones are the frames that have drastic changed in the total kinetic
energy and the red ones are the frames that have drastic change in the
total apparent mass.
Figure 4. Receiver Operating Characteristic curve that indicates the
average of ten iterations at the four parking lots.
R EFERENCES
[1] H. Dee, D. Hogg. Detecting Inexplicable Behavior, In BMVC,
2004.
[2] N. Robertson and I. Reid. Behaviour understanding in video: a
combined method. In Computer Vision, 2005. ICCV 2005. Tenth
IEEE International Conference on, pages 808-815 Vol. 1, Oct.
2005.
[3] A. Basharat and A. Gritai and M. Shah. Learning object motion
patterns for anomaly detection and improved object detection.
In Computer Vision and Pattern Recognition, 2008. CVPR 2008.
IEEE Conference on, pages 1-8, June 2008.
[4] M. P. Kumar and P. H. S. Torr and A. Zisserman. Learning layered
motion segmentations of video. In Computer Vision, 2005. ICCV
2005. Tenth IEEE International Conference on, pages 33-40 Vol.
1, Oct. 2005.
[5] R. Mehran and A. Oyama and M. Shah. Abnormal crowd behavior
detection using social force model. In Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on, pages 935-
942, June 2009.
[6] B. Antić and B. Ommer. Video parsing for abnormality detection.
In Computer Vision (ICCV), 2011 IEEE International Conference
on, pages 2415-2422, Nov 2011.
[7] J. Zhao and Y. Xu and X. Yang and Q. Yan. Crowd instability
analysis using velocity-field based social force model. In Visual
Communications and Image Processing (VCIP), 2011 IEEE, pages
1-4, Nov 2011.
[8] R. Raghavendra and A. Del Bue and M. Cristani and V. Murino.
Optimizing interaction force for global anomaly detection in
crowded scenes. In Computer Vision Workshops (ICCV Work-
shops), 2011 IEEE International Conference on, pages 136-143,
Nov 2011.
[9] D. Tran and J. Yuan. Optimal spatio-temporal path discovery for
video event detection. In Computer Vision and Pattern Recognition
(CVPR), 2011 IEEE Conference on, pages 3321-3328, June 2011.
[10] G. Xiong and X. Wu and Y. L. Chen and Y. Ou. Abnormal crowd
behavior detection based on the energy model. In Information and
Automation (ICIA), 2011 IEEE International Conference on, pages
495-500, June 2011.
[11] T. Cao and X. Wu and J. Guo and S. Yu and Y. Xu. Abnormal
crowd motion analysis. In Robotics and Biomimetics (ROBIO),
2009 IEEE International Conference on, pages 1709-1714, Dec.
2009.
[12] T. Huang and J. Qiu and T. Ikenaga. A Foreground Extraction
Algorithm Based on Adaptively Adjusted Gaussian Mixture Mod-
els. In INC, IMS and IDC, 2009. NCM ’09. Fifth International
Joint Conference on, pages 1662-1667, Aug. 2009.
[13] J. J. Chen and C. R. Su. Volume Image Segmentaton by
Dual Multi-Scale Morphological Reconstructions. In Intelligent
Information Hiding and Multimedia Signal Processing, 2009. IIH-
MSP ’09. Fifth International Conference on,pages 511-514, Sept.
2009.
[14] Thomas Brox and Jitendra Malik. Large Displacement Optical
Flow: Descriptor Matching in Variational Motion Estimation. In
IEEE Transactions on Pattern Analysis and Machine Intelligence,
33(3):500-513, 2011.
[15] W. Li and V. Mahadevan and N. Vasconcelos. Anomaly Detec-
tion and Localization in Crowded Scenes. In IEEE Transactions
on Pattern Analysis and Machine Intelligence, pages 18-32, Jan.
2014.
[16] H. Yang and Y. Cao and S. Wu and W. Lin and S. Zheng
and Z. Yu. Abnormal crowd behavior detection based on local
pressure model. In Signal Information Processing Association
Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific,
pages 1-4, Dec. 2012.
[17] C. Xie and L. Shang. Anomaly detection in crowded scenes
using genetic programming. In Evolutionary Computation (CEC),
2014 IEEE Congress on, pages 1832-1839, July 2014.