2022 14th International Conference on Mathematics, Actuarial Science, Computer Science and Statistics (MACS)

CNN-LSTM Based Smart Real-time Video

Surveillance System
Waqas Iqrar Malik ZainUl Abidien Waqas Hameed
Department of Electrical and Computer Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering Engineering
COMSATS University Islamabad, COMSATS University Islamabad, COMSATS University Islamabad,
Abbottabad Campus Abbottabad Campus Abbottabad Campus
Abbottabad, Pakistan Abbottabad, Pakistan Abbottabad, Pakistan
waqasiqrar99@gmail.com malikzn1234@gmail.com waqash1998@gmail.com

Dr. Aamir Shahzad

Department of Electrical and Computer
Engineering
COMSATS University Islamabad,
Abbottabad Campus
Abbottabad, Pakistan
ashahzad@cuiatd.edu.pk

Abstract— Modern surveillance technologies are intended to
be precise, cost-effective, and run without the need for human
interaction. In this regard, few topologies were employed in the
previous decade which treated human movement as traditional
object detection. Because of this approach, suspicious behavior
was also tackled as an object, which made the existing human
activity recognition (HAR) systems slow and inefficient when it
came to classification for real-time application. In these recent
years, machine learning algorithms have made significant
advancements in the time-dependent classification of events.
This research presents an HAR system that employs a
Convolution Neural Network (CNN) to extract spatial
information along with a Long Short-Term Memory (LSTM)
approach for the rapid and precise sequential tracking of an
identified object. This CNN-LSTM technique not only lowers
the model's complexity but also improves its accuracy which
allows it to be executed in real-time. Therefore, the proposed
CNN-LSTM approach can detect suspicious activities in real-
time at 10-13 FPS and obtain the best tracking performance in
any circumstance while implemented on Raspberry Pi which
works as a standalone system.
Keywords— CNN-LSTM, deep learning, fight detection,
human activity recognition, intelligent surveillance system,
sequence-based activity detection.
I. INTRODUCTION
Throughout human history, surveillance has been an
integral component of society. Humans have been attempting
to develop surveillance measures for a safer way of life since
ancient times. Due to current technologies such as cameras,
computers, GPUs, etc., surveillance has been vastly improved
in recent decades. Previously, metal detection, x-ray
scanning, and manual monitoring of security cameras were
utilized to conduct surveillance. As the primary function of
surveillance is to detect and monitor actions, surveillance
systems in the era of artificial intelligence can be extremely
precise and operate without human intervention.
One of the most significant developments in the Internet
of Things (IoT) is object detection, which can process objects
This project has been awarded sponsorship from the Federal Ministry of
Information Technology in Pakistan under IGNITE program
(FYP-Code: NGIRI-2022-12025).
in still images and moving videos. The object detection
process utilizes two distinct techniques; the first involves
image classification, while the second is object localization.
At the algorithm level, not only the object is detected but also
a tagged bounding box is drawn to point to the object’s
position. Consequently, object detection can detect several
objects in a single image or video frame. With the progress in
object detection algorithms, its applications broadened to
multiple fields including medical treatment, civil
engineering, defense, agriculture, etc. [1]. However, human
activity recognition (HAR) remained a challenge because it
involves sophisticated classification of motion and gestures.
In this research, using closed-circuit television (CCTV)
cameras on a real-time operating surveillance system, we
further explore HAR to detect criminal activities and fighting
sequences. A smart surveillance system is presented for
detecting and locating an unscrupulous act within a specific
time frame. If the proposed HAR system detects suspicious
activity, it will notify the authorities. To be more specific,
criminal conduct will be detected if individuals are fighting
or if one is trying to rob the other. It will distinguish the action
and identify the two individuals as objects engaging in it.
There is significant existing research related to
surveillance systems. In the earliest days of object detection,
background subtraction, the statistics method, or the frame
difference method were used. Point tracking was typically
employed for detecting [2]. Later motion detection was done
with the help of a histogram of oriented gradients (HOG) and
support vector machine (SVM) algorithms [3]. HOG is an
advancement on the scale-invariant feature transform and
shape contexts that were available at the time it was
developed. HOGs were most known for their application in
the detection of pedestrians, and it was the first multi-object
detection method. The deformable part-based model (DPM)
technique was the following one that was introduced later [4].
Instead of using the HOG rule for detection, it employs the
"Divide and Conquer" rule. The fact that they were not
focusing on sequence base detection is nonetheless their most
significant shortcoming. The primary issue with the many
other methods was that they lacked a reference to the

978-1-6654-6071-2/22/$31.00 ©2022 IEEE

movement of time, which means that they are unable to
meaningfully differentiate between scenarios such as
moving, fighting, or hugging. Consequently, there is room for
advancement in several aspects of these procedures.
For this paper, we trained the model using an HAR
technique that combines CNN and LSTM. Even though both
architectures (CNN and LSTM) have been the subject of a
significant amount of research and a lot of research has been
done on them, the authors of this paper have implemented
both of these methods in order to achieve the best possible
outcome for HAR. It is pertinent to mention that to the best
of the author’s knowledge, this combined method is not
employed for detecting the real-time HAR. Therefore, the
main contribution of this paper is to provide a smart real-time
surveillance solution using CNN and LSTM.
The remaining of this paper is organized as follows:
Section II contains a review of some previously published
relevant research studies. In section III, the proposed strategy
to provide a smart real-time surveillance system is presented
in detail. The experimental findings as a result of the
implementation of the proposed work are given in Section IV.
Finally, section V concludes this paper.
II. RELATED WORK
Due to advancements in the field of video recognition,
HAR research is extensively subdivided into numerous types.
A two-stream network is the most prevalent type of action
recognition in deep learning. This technique deals with two
CNN, the first one is used for spatial feature extraction which
deals with a single frame of action and tries to learn from it
[5] whereas the other CNN uses a sequence of frames to
extract features using optical flow vectors technique.
Another HAR system has been created employing sensors
that require very little power and may be worn on the body.
This architecture generates features by fast Fourier transform
as well as discrete wavelet transform by using data obtained
from the inertial measurement unit (IMU) sensor. A neural
network is trained on obtained data and as an experimental
result of this approach, six activities can be detected with
12.5mW power consumption [6]. This technology provides a
good approach to HAR, but it also comes with the expense of
wearable sensors and energy consumption, both of which are
not optimal when applied on a larger scale.
A different approach is discussed in an article [7], which
referred to the time series approach of HAR using multi-
channel CNN as a framework. This article presents us with
the methodology to convert data obtained from a 3D
accelerometer into an image, which is fed into CNN with 3
convolutional layers, and one fully connected layer.
III. METHODOLOGY
In this part of the paper, a discussion on the different
approaches to the extraction of features and categorization of
video segments for violence detection is carried out. Fig. 1
shows the architecture of a CNN and LSTM that is used for
this purpose.
A. Preprocessing
The videos in the dataset are represented by their number,
number of frames per clip, width, and height. These videos
are transformed into frames and resized to 128 by 128 by 3.
Using two additional approaches, the data is divided into two
sections. The first is for training, while the second is for
validation. 75% of the data is chosen for training, while the
remaining 25% is used for validation.
B. Feature Extraction
Various types of CNN architecture are tested for Feature
extraction like VGG16 [8] and MobileNetV2 [9]. Since we
have a smaller dataset, so we use transfer learning, therefor
our model is trained on the MobileNet V2 of CNN
architecture. For the implementation of this idea, this model
was best for real-time applications. MobileNet V2 usually
support any input size image greater than 32 x 32 dimension.
For this experiment, we took the input image shape (128 x
128). In April 2017, MobileNet v2 was released and available
to the public. The bottleneck layers and shortcut links were
updated from the previous edition [10].
Video segments are uniformed before sending for feature
extraction, 20 frames from every video are uniformly
selected. Then these frames are cropped to match the input
size of the architecture.
Fig. 1. Architecture of proposed methodology.
C. Classification
For classification, LSTM is used for activity detection as
it can learn on time sequence and thus can learn patterns
based on this [11]. As the name of LSTM suggests, it is a
sequential learning system that uses time as a reference to
discover patterns. LSTM networks were created primarily to
address the issue of long-term dependency. LSTMs have
feedback connections that distinguish them from
conventional feedforward neural networks. This trait enables
LSTMs to process whole sequences of data without treating
each point in the series individually. Instead, LSTMs retain
important information about prior data in the sequence to
assist in the processing of new data points. Consequently,
LSTMs excel at processing sequences of data such as text,
voice, and time series in general. LSTMs utilize a number of
"gates" that regulate how information in a data sequence
enters, is stored, and exits the network. In a standard LSTM
forget gate, there are three gates: input gate, output gate, and
forget gate.
The first gate in the series is called the forget gate [12].
At this stage, it determines which bits of the cell state, which
serve as the long-term memory of the network, are valuable
in light of both the previous hidden state and the newly
received input data. The name of the second gate in the
network is the "input gate" [13]. In this stage, it is decided
what new information needs to be added to the long-term
memory (cell state) of the network, taking into account the
previously hidden state as well as the incoming input data.
The name "output gate" refers to this particular gate. This
stage, which determines the new hidden state, takes place
once the update in the long-term memory has been finished
and is ready to be used.
Fig. 2 presented above provides an overview of the HAR
approach that we have proposed. In the first phase, a video is
broken down into frames, and these frames are then run
through a CNN Neural Network, which can identify people
in the video obtained directly from a Pi-Cam and processed
through a Raspberry Pi. After that, this result is sent on to
LSTM so that it can be compared with results for other blocks
of CNN for the detection of a fight scene. In the end, training
losses and validation losses are calculated and plotted.
1) Validation losses
On the other hand, validation loss is a metric that's used
to evaluate how well a deep learning model performs on the
validation set. A section of the dataset known as the
validation set has been set aside specifically for the purpose
of verifying the accuracy of the model. The validation loss is
comparable to the training loss in that it is determined by
adding up all of the mistakes made in each example that is
included in the validation set.
In addition, the validation loss is evaluated after every
epoch that passes. This tells us whether the model requires
further tuning or adjustments or whether it is already perfect.
Typically, this is accomplished by drawing a learning curve
to represent the validation loss.
2) Training losses
When evaluating how well a deep learning model fits the
training data, one metric that is utilized is called the training Fig. 2. Overview diagram of Proposed Method.
loss. In other words, it determines how inaccurate the model
3) Real Life Dataset: This is the largest collection,
is based on the data from the training set. Take note that the
containing over 2000 video clips of actual life situations
training set is only a subset of the dataset that is utilized to
train the model in the beginning. Computation, the training including physical conflict. The fact that this dataset is the
loss is determined by adding up all of the different kinds of best example of data augmentation is the primary benefit of
mistakes that were made during the training process. utilizing it. Within this dataset, there is a total of 1000
It is also essential that the training loss is calculated after examples accessible for each class. Previously, a VGG16 and
every batch that is processed. In most cases, this is shown an LSTM model that achieved an accuracy of 88.2% were
graphically by displaying a curve of the training loss. trained on this dataset [15].

IV. EXPERIMENTAL RESULTS AND DISCUSSION The proposed architecture is applied to the above datasets.
The accuracy of this architecture, with respect to various
This section of the paper will discuss the experiments and datasets, is provided below in Table 1.
their results. First, the datasets are explained in detail, then
the graphical results are obtained and analyzed to assess the Table 1: Comparison of Proposed model with different approaches.
performance of the proposed methodology. Approach Dataset Accuracy
A. Datasets VGG16 + LSTM Hockey Dataset 87.05% [16]
This subsection gives the details of the used dataset for
STIP + MoSIFT Hockey Dataset 90 % [14]
this approach and experimental setup. Later experimental
results will be discussed. There is some background in all VGG16 + LSTM Real-Life Dataset 88.2% [15]
clips but still, there is a lot of background motion. STIP + MoSIFT Peliculas 90% [14]
1) Hockey Dataset: This dataset includes short video CNN+LSTM Real-Life Dataset 91% (this research)
clips taken from the hockey dataset that depict either fighting
CNN+LSTM Peliculas 98% (this research)
or non-fighting scenes. This dataset includes a total of one
thousand movies of both scenes. After training the base model on the real life dataset and
2) Peliculas Dataset: This dataset includes 200 videos the Peliculas dataset we get accuracy of 91% and 98%
total, with 100 movies relating to violent acts and 100 videos respectively.
relating to non-violent acts. An accuracy of 90% was It is to mention that, while training only 50% (500 videos
achieved by the authors of this dataset when they used it to of each) of the dataset is used due to COLAB memory issues
train models on STIP and MoSIFT [14]. It is mentioned that therefore the performance is somewhat compromised. Fig. 3
each video of this dataset has about two to four seconds of shows the obtained total accuracy and total validation
runtime. accuracy trained on real life dataset which is 91%.
 Fig. 3. Achieved total accuracy and total validation accuracy on real life
dataset
Its total loss vs total validation loss can be observed in
Fig. 4. Model did perform well, but not as well as planned
because fewer datasets were used. But still has achieved a
total accuracy of 91% which is better than the other models,
previously presented in the literature.
Fig. 4. Achieved total loss vs validation loss on a real-life dataset
Another training of the model was performed on the
Peliculas dataset using the same architecture. Following the
training, the model was able to successfully reach an accuracy
of 98%, which is the greatest it has been able to achieve on
this dataset so far.
Fig. 5. Achieved total loss vs total validation loss on Peliculas dataset
Fig. 6. Achieved total accuracy vs total validation accuracy on the Peliculas
dataset
On this dataset, the model performs best with an accuracy
of 98%. The total loss vs validation loss while training is
presented in Fig. 5. Fig. 6 shows the graphical representation
of attained accuracy of the model on the Peliculas dataset.
It is noteworthy that the presented methodology resulted
in an accuracy of 91% that is significantly improved from
previous research. Moreover, the trained model has achieved
this much correctness with a varying background that causes
different illumination levels and adds more objects in the
video. Consequently, owing to discussed advantages the
presented CNN-LSTM-based HAR system can be
implemented as a real-time smart surveillance system.
V. CONCLUSION
The field of application that deals with identifying fights
and suspicious actions on video are expanding. This
capability may prove to be of great use in situations involving
video monitoring, such as in correctional institutions, mental
hospitals, and senior care facilities. Techniques of HAR that
have traditionally focused primarily on individual actors and
fundamental events can be utilized within the context of this
particular application. The fundamental objective of this
research is to present a standalone HAR system that can
identify potentially harmful video behavior speedily and
accurately. The presented HAR approach combines CNN and
LSTM to deliver optimal precision and speed, which is
beneficial for real-time applications. The accuracy of the
presented HAR technique is up to 98% for the Peliculas
dataset and reduces to 91% for real life datasets with varying
backgrounds. However, it is notable that his accuracy of 91%
is improved compared to its previous counterparts. To say in
nutshell, this paper introduces a real-time standalone HAR
solution that not only detects but also classifies unscrupulous
activities. In the future, a camera will be mounted on a
moving automated platform such that the overall system may
operate in the closed quarters and provide real-time HAR
feedback to the control room.
ACKNOWLEDGMENTS
The Project Lab in the Department of Electrical and
Computer Engineering at CUI Abbottabad is where the
majority of this work is accomplished. The author wishes to
express gratitude to both the department's employees and
management for their unwavering support during the course
of this research.
REFERENCES
[1] G. Swapna, S. Kp, and R. Vinayakumar, "Automated detection
of diabetes using CNN and CNN-LSTM network and heart rate
signals," Procedia computer science, vol. 132, pp. 1253-1262,
2018.
[2] P. K. Mishra and G. Saroha, "A study on video surveillance
system for object detection and tracking," in 2016 3rd
International Conference on Computing for Sustainable Global
Development (INDIACom), 2016, pp. 221-226: IEEE.
[3] P. K. Roy and H. Om, "Suspicious and violent activity detection
of humans using HOG features and SVM classifier in
surveillance videos," in Advances in Soft Computing and
Machine Learning in Image Processing: Springer, 2018, pp. 277-
294.
[4] T. Cucliciu, C.-Y. Lin, and K. Muchtar, "A DPM based object
detector using HOG-LBP features," in 2017 IEEE International
Conference on Consumer Electronics-Taiwan (ICCE-TW), 2017,
pp. 315-316: IEEE.
[5] M. Jogin, M. Madhulika, G. Divya, R. Meghana, and S. Apoorva,
"Feature extraction using convolution neural networks (CNN)
and deep learning," in 2018 3rd IEEE international conference
on recent trends in electronics, information & communication
technology (RTEICT), 2018, pp. 2319-2323: IEEE.
[6] G. Bhat, R. Deb, V. V. Chaurasia, H. Shill, and U. Y. Ogras,
"Online human activity recognition using low-power wearable
devices," in 2018 IEEE/ACM International Conference on
Computer-Aided Design (ICCAD), 2018, pp. 1-8: IEEE.
[7] Y. Zheng, Q. Liu, E. Chen, Y. Ge, and J. L. Zhao, "Time series
classification using multi-channels deep convolutional neural
networks," in International conference on web-age information
management, 2014, pp. 298-310: Springer.
[8] K. Simonyan and A. Zisserman, "Very deep convolutional
networks for large-scale image recognition," arXiv preprint
arXiv:1409.1556, 2014.
[9] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen,
"Mobilenetv2: Inverted residuals and linear bottlenecks," in
Proceedings of the IEEE conference on computer vision and
pattern recognition, 2018, pp. 4510-4520.
[10] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling, "Spherical
cnns," arXiv preprint arXiv:1801.10130, 2018.
[11] S. Hochreiter and J. Schmidhuber, "Long short-term memory,"
Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[12] A. Kulshrestha, L. Chang, and A. Stein, "Use of LSTM for
Sinkhole-Related Anomaly Detection and Classification of
InSAR Deformation Time Series," IEEE Journal of Selected
Topics in Applied Earth Observations and Remote Sensing, vol.
15, pp. 4559-4570, 2022.
[13] H. Abbasimehr and R. Paki, "Improving time series forecasting
using LSTM and attention models," Journal of Ambient
Intelligence and Humanized Computing, vol. 13, no. 1, pp. 673-
691, 2022.
[14] E. Bermejo Nievas, O. Deniz Suarez, G. Bueno García, and R.
Sukthankar, "Violence detection in video using computer vision
techniques," in International conference on Computer analysis of
images and patterns, 2011, pp. 332-339: Springer.
[15] M. M. Soliman, M. H. Kamal, M. A. E.-M. Nashed, Y. M.
Mostafa, B. S. Chawky, and D. Khattab, "Violence recognition
from videos using deep learning techniques," in 2019 Ninth
International Conference on Intelligent Computing and
Information Systems (ICICIS), 2019, pp. 80-85: IEEE.
[16] Ş. Aktı, G. A. Tataroğlu, and H. K. Ekenel, "Vision-based fight
detection from surveillance cameras," in 2019 Ninth
International Conference on Image Processing Theory, Tools
and Applications (IPTA), 2019, pp. 1-6: IEEE.