See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/331844217

Intelligent Multi-Camera Video Surveillance System for Smart City Applications

Conference Paper · March 2019

DOI: 10.1109/CCWC.2019.8666579

CITATIONS READS
20 4,272

2 authors:

Abdullah Alshammari Danda B Rawat

NEOM Howard University
6 PUBLICATIONS 89 CITATIONS 399 PUBLICATIONS 6,421 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Method to securely use Brain Machine Interfaces (BMI) View project

smart city View project

All content following this page was uploaded by Abdullah Alshammari on 17 April 2019.

The user has requested enhancement of the downloaded file.

 Intelligent Multi-Camera Video Surveillance
System for Smart City Applications
Abdullah Alshammari Danda B. Rawat
Electrical Engineering and Computer Science Electrical Engineering and Computer Science
Howard University, Washington, DC Howard University, Wasgington, DC
abdullah.alshammari@bison.howard.edu danda.rawat@Howard.edu

Abstract— One of the challenges that many security systems II. MOTIVATION AND VIDEO SURVEILLANCE SYSTEMS
face is the inability to allow the system to perform certain
functions without an operator monitoring the progress of the The outline of moving object location framework is vital for
system, automatically. People cannot operate twenty-four hours a observing a range and recognizing questions or focuses of
day. Human operators have a limit to operate during the time they hobby. Fig. 1 shows overall video surveillance design and this
are awake. People will always require sleep and even when they paper shows each subsystem in this intillegent multi camera
are awake they cannot sit in one place for a long time without any video surveillance system. Multi camera video reconnaissance
distraction. That is why there are unique features in the multi- is vital for analyzing territories of daily life. Video
camera video system that can make significant impacts in the reconnaissance is relied upon to arrange, perceive, and track
security industry. The technical aspect of the technology is different objects and scenarios according. In spite of the fact
exceptional and it can assist many people to counter security that, camcorders are now introduced in numerous ranges, for
challenges they face in their daily lives. It is important to consider example, the checking of indoor and outside scenes of air
these aspects when selecting which type of security systems are terminals, train stations, interstates, parking garages, stores, and
needed to adopt. Our approach could detect and recognize a not withstanding shopping centers. Video information as of now
human target from videos taken from cameras mounted on the utilized are generally to analyze. Much of the time a human
wall to cover a target area. The proposed approach consists of
administrator needs to watch the video to distinguish anomalous
detection and tracking of any targets that might be a threat. The
conduct which is extremely tedious and monotonous. Having an
decision-making process then decides whether or not the person is
a threat or not.
ongoing robotized video reconnaissance can be the response to
a hefty portion of the issues. The outline of moving item location
Keywords—smart city, surveillance, intelligent, security, IoT, for brilliant video observation will predominantly centered
communication. around ongoing programming advancement. The branch of
knowledge of picture preparing is to a great extent used to create
I. INTRODUCTION moving target framework. The configuration of programming
Video cameras are becoming a common feature of modern based observation framework can be exceptionally financially
life where they are useful for crime prevention, terrorist savvy than equipment based framework.
detection, and forensic evidence. Today, companies deploy many
surveillance cameras to watch broader areas 24 hours a day 7
days a week. Since video data is increasing and only the
security monitor staffs are able to watch the limited amount of
video data, people can quickly lose their ability to concentrate
on the monitoring task, and suspicious behavior on the screen
maybe missed easily. In this project, our goal is to remedy these
weaknesses and develop a system which provides an automatic
interpretation of the scenes by using motion detection and
recognition, trajectory analysis, pattern matching and the
understanding the actions and interactions of the observed
objects based on the information acquired by sensors. This
system uses three algorithms. Algoritms are object detection,
template matching and decision making. Our system will
consist of multiple surveillance cameras to gather video images,
a server to store and analyze that information, and broadband
communication networks to connect with the internet.
Fig. 1. Overall video surveillance design.

978-1-7281-0554-3/19/$31.00©2019 IEEE
 In fig. 3 shows our scenarios to calculate the delay time
overall implement intelligent multi-camera video surveillance
system in edge computing enabled resilient wireless network
virtualization. The advantages of edge computing include (i)
near-real- time reactions by using distributed processing of big
data which could take longer time by individual devices [1]; and
(ii) caching at the edge for popular contents for video or data
communications so that when the content is requested it can be
served from cache (which further reduces back-haul
communication cost) [2], [3].
Fig. 4. Evaluation, Analyses and Comparing the complexity view between
Camera, Edge Computing and Cloud.

III. PROBLEM STATMENT

Nowadays, there is a huge amount of video data being
recorded, but not processed and analyzed. Therefore, as a result,
events and activities are missed, and suspicious behavior is not
being detected in time to prevent potential serious incidents. The
goal of this project is to employ multiple cam- eras mounted on
the wall to automatically detect and track objects of interest, by
understanding and analyzing their activities. Since video
surveillance systems automatically capture and record images to
drastically reduce workload for the staff. This system will
incorporate multiple cameras in order to conduct on the analysis
of the videos that is captured. Video surveillance has a wide
variety of applications both in public and private environments,
such as the monitoring of indoor and outdoor scenes of airports,
train stations, highways, parking lots, stores, and even shopping
malls. Our paper intends to develop an intelligence video
Fig. 3. Overall implement intelligent multi-camera video surveillance system surveillance system that will use multiple cameras to track a
in edge computing enabled resilient wireless network virtualization design. person’s movement and analysis that information to determine
if their movement is deemed threating or not detect the abnormal
When we calculate the tasks of total delay Tdelay, after design behavior of that person. By implementing this system, we can
making subsystem send it to edge computing system and the tighten the security for important facilities such as hospitals,
delay time can be written as: power plants and children’s play grounds. We will utilize video
cameras that will record the movements of suspicious behavior
and send this data to a computer network for processing. This
Tdelay = TRecDelay + TxCamToEdgeDelay + TEdgeCompDelay +
network of cameras will also use a trajectory analysis algorithm
TxEdgeCompToCloudDelay + TCloudDelay, (1) to tracking the movement of the objects that they are recording.
Combining the data gathered from the behavior recognition
Tdelay = T1 + T2 + T3 + T4 + T5, (2) software and images the server will analyze the information and
if the user appears to be acting irregular then, the robot will act
where: to investigate the activity.
T1 is present camera record delay time.
T2 is present spending delay time between camera to edge IV. RELATED WORK
computing. This section will provide some terminologies and summarize
T3 is present edge computing delay. some of the existing work related to our project..
T4 is present spending delay time between edge computing to
A. Background Subtraction:
cloud.
T5 is presents clouding delay. The basic of background subtraction is to subtract the image
α = 1, when edge used. from a reference image that models the background scene. The
basic steps of this technique are to construct a reference image
α = 0, when cloud is not used.
representing the background. Then determine the values used in
α = 1, when cloud is used. the subtraction operation to obtain a desired detection rate. Our
project will perform background subtraction by using an
In terms of complexity and delay times in Fig. 4, we algorithm which we will create to allow the software will learn
compare and analyze the complexity view between camera, on its own, using the data and detect abnormal behaviors that
edge computing, and cloud. It is clear the complexity is high on appear.
the cloud.
B. Frame Differencing (Modeling Method):
Fig. 5 Frame Differencing is a strategy where the computer
checks the distinction between two video outlines. On the off
chance that the pixels have changed when there was something
changing in the picture this removed position is then used to
separate a rectangular picture layout from that area. The
grouping of layouts is created as item changes its position. The
thought is to take the total contrast between a references put
away foundation and another casing which is snatched from the
camera. On the off chance that no frame is entered, the outcome
pixel will be zero quality as a result. In the event that an
objective exists, the subsequent pixels will give a quality not
quite the same as zero.
Fig. 5. Frame Differences Example
C. Template matching (Modeling Method):
Fig. 6. shows template matching is a strategy for discovering
little parts of a picture which coordinate a format picture. It is a
clear process in this strategy format pictures for distinctive
articles are put away. At the point when a picture is given as
data to the framework, it is coordinated with the put away
format pictures to decide the item in the info picture. Formats
are as often as possible utilized for acknowledgement of
characters, numbers, and questions. It can be performed on
either shading or dark level pictures, it can either be pixel to
pixel coordinating or highlight based. All things considered the
elements of format picture is contrasted with components of
subpictures of the given data picture to figure out whether the
layout article is available in the info picture.
Fig. 6. Motion Detection Methods Examples
D. Foreground Detection (Modeling Methods):
Fig. 7 shows this technique comprises of contrasting the
foundation picture and the present picture to name pixels as
foundation or forefront pixels. In this way, forefront articles can
be identified by checking the contrast between the testing
casing and the foundation model constructed already. There has
been various works that have been done on foundation
demonstrating including building a legitimate representation of
the foundation scene. Since models catch the association
between video outlines, they can deal with the varieties out of
sight, for example, enlightenment change and element
compositions. Pixels amid such interims are viewed as
foundation, and the foundation scene is evaluated from these
interims. The legitimacy of this methodology depends on the
suspicion of static foundation. Subsequently, it is constrained
when handling element foundation or recordings caught by a
moving camera.
Fig. 7. Background and Foreground Detection Methods
E. Motion Detection:
Recordings are balanced out by following an unmistakable
element that is basic to every one of the edges and utilizing it
as a grapple point to counteract all unsettling influences in
respect to it. Be that as it may, to execute such a strategy, we
are required to know the position of the conspicuous component
in the first edge and utilize this system for point highlight
coordinating to naturally hunting down the foundation plane in
a video succession and utilizing its watched twisting to redress
for camera movement.
F. The part-based approach (Active and passive object
detection):
It uses passive scanning to determine interest points in an
image. This approach calculates an interest value for local
samples at all points of an evenly spaced grid. At the interest
points, the approach extracts new local samples that are
evaluated as belonging to the object or the background.
G. Color based recognition:
Color offers potent information for recognizing objects. An
easy and effective object detection scheme is to depict and
match images on the foundation of color histograms [6]. The
color attributes as a clear color representation for object
detection. Color information is expounded in two existing
techniques object detection, the part-based detection structure
and the efficient sub window search approach. The criteria that
must be considered when selecting an approach to combining
color to object detection are feature combination and
compatibility.
H. The window-sliding approach (Active and passive object
detection):
It uses passive scanning to check if the object is present or
not at all locations of an evenly spaced grid. This approach
extracts a local sample at each grid point and classifies it either
as an object or as a part of the background.
V. ANALYSIS
A. Trajectory Analysis:
Analyzing trajectory is one of the principle methods for
conducting moving articles it can be extricated by examining
the direction of moving questions and utilize this data to group,
recognize and comprehend conduct. By utilizing this data, you
can identify strange occasions in shrewd video reconnaissance
frameworks. With direction demonstrating, we can break down
the displaying prerequisites, propose a multi-layer direction Fig. 9. Behavior Recognition Architecture Example [1].
show, and compress direction on articles that we have
composed beforehand. VI. BUILDING BLOCKS OF VIDEO SURVEILLANCE SYSTEM
The concept used to develop multi-camera video system is
B. Pattern Matching:
highly complex and requires relevant background knowledge in
Fig. 8. shows in computer science; example coordinating is computer systems. The methodology applied is the basis for the
the demonstration of checking a given arrangement of tokens functionality of the systems and reduces the difficulty for the
for the vicinity of the constituents of some examples. As end user of the system to operate the equipment. The
opposed to example acknowledgment, the match as a rule must architectural design of the system is accurate enough to provide
be correct. The examples for the most part have the type of the user detailed information in different levels of the operation
either groupings or tree structures. The framework will utilize including motion detection capabilities, tracking system,
this technic to recognize the items. analysis system and decision making processing system.
A. System Architecture
The multi-camera video system is advanced equipment that
provides the user with various options to help in the execution
of intelligent decisions. The challenges facing many security
system users today is the inability to allow the system to
perform automatically certain functions without an operator
monitoring the progress of the system [4]. Human beings can
operate twenty-four hours a day, we have a limit to how we
operate during the time we are awake. It is impossible for
human beings to sit behind a computer and monitor a camera
constantly without interruptions or distraction. People always
require sleeping and even when we are awake, we cannot sit in
Fig. 8. Pattern Matching Example one place for a long time without any distraction [4]. It is,
C. Behavior Recognition: therefore, important to develop a device that can ensure
continuous system monitoring and communication during
Fig. 9. shows recognizing and characterizing unusual
unusual activities. This is achievable for machines because they
conduct from video is a standout amongst the most intriguing
do not require rest, only maintenance. This is when people
difficulties confronting computer vision specialists. Detecting
should consider adopting the current technologically advance
and classifying human behavior from video is among the most
system that can meet all this weakness of human feature. The
interesting issues facing computer vision studies. The analysis
multi-camera system contains complex features that function in
of behavior entails creating a vision front end that identifies,
such way that can meet security expectations required today.
segments, groups and tracks bodies. It is also composed of
The system design contains in-built complex algorithm
conducting a crisp description behavior incorporating
formulas that coordinate different subsystem components in the
distinguishing between short and simple movements, medium
system. The integration of the subsystem in different levels of
scale actions. Studying behavior in human beings brings forth
operations work surprisingly well with proper coordination
an opportunity for making developments on modeling and
methodology that any human can comprehend. The experts
grouping behavior, particularly social behavior that is hard to
behind this technology have done proper research and have
study. Group tracking together with event recognition in
come up with different designs and architecture that can operate
surveillance is an essential area of study since it helps expose
under any climatic conditions. Also, the physical appearance
criminal behaviors. The best approach to use is one that
can be altered to avoid detection by the threat that can lead to
recognizes activities in long videos and deals with several
damage. With proper installation guidelines, the intruder or
actions at a go.
threat cannot be able to locate the system.
B. Multi-Camera Video System Capability
There are unique features in the multi-camera video system
that can make significant impacts in the security industry. The
technical aspect of the technology is exceptional and can assist
many people to counter security challenges they face in their
daily lives. It is important to consider these aspects when
selecting which type of security systems we need to adopt. The
multi-camera video system can provide the visual user data in
seconds. The concept behind these features is to enable the
system operator to get accessed to valuable data on time for
faster and intelligent decision-making process. The cameras are
strategically designed to show multiple angles surveillance to
allow the operator to get a clear view point. This means the
operator will not miss any valuable angles that a threat might
be and provide enough evidence to act on the threat. Most of
the available camera systems are not designed to withstand
harsh climatic conditions. This has caused a lot of damage to
the equipment and forced many people to chip in a lot of money
on repairs and replacement. People are now focused on getting
camera systems that can operate under extreme conditions
without default. Our multi-camera video system will be one the
few security systems that can function under high-density
conditions and at the same time provide accurate information.
The system is also developed to operate without internet
connection for a specific period. The concept behind this
feature is that it records events and sends them to a storage
system that can be accessible by the operator on the event of
threat detection. The system functionality is widely considered,
and any form of malfunction (which is rare) will be sent to the
operator immediately by signaling using the modern alert
systems. Also, this methodology is used in the event of a
security breach that is common with hackers. Upon detection of
a security breach, the alert system alarm will respond
communicating with the system operator in real time.
Therefore, the system will always communicate with the
security department in any situation. The operator is always
guaranteed smooth operations and easy use of the system since
the algorithm used provides information that the operator can
understand and respond accordingly. Many people are usually
concerned about adopting a camera system due to privacy
reasons [5]. This is why most hotels and guest houses do not
have installed security cameras, because they consider some
areas as private. Even in public places such as toilets and
offices, do not allow the existence of security cameras. These
places also can be used by terrorists and other criminal
offenders as the base of their operations. This poses a danger to
human life and also the peace in that location. The multi-camera
systems are capable of protecting the rights of the public and
allow room for privacy. The system achieves this by deleting
files that infringe the privacy of the public after determining the
threat in that specific area.
C. Description of Four Major Subsystems
The multi-camera video system is made of four advanced
subsystems that operate in coordination with one another. The
subsystems play an important role in achieving the functionality
of the whole multi-camera system we built and are crucial in
improving the security situation. Therefore, people should
understand the capability of each subsystem one can
acknowledge the value of the multi-camera video system in the
security industry.
1) Object Detection System:
One of the three algorithms are used in this paper is object
detection. Fig.10 shows the subsystem. In this subsystem, we
have included a camera that enables to record videos and audios
on real time events that the operator receives. This will enable
the operator to hear and see as the events unfold. These features
in the subsystem can help identify potential threats through
listening and watching suspicious threats to the security of the
area. We also included a white blob detection capability that is
able to process the data provided by the high definition camera.
Camera
White Blob Detection
Mask Image
White Blob Filtering and Merging
Output
Fig. 10. Object Detection Architecture.
The white blob detection software will enable to detect
objects in areas with different properties as compared to the
natural surroundings. This means that the operator is able to see
through extreme lights that prevent proper view. Once the data
is processed and properly adjusted by the white blob detection
method, the data is the transferred to the mask image which
manipulates values for comparison and calculations of the data
provided. The data will then be transferred to the white blob
filtering and merging where the information transferred will be
properly adjusted and connected to produce a proper data that
be able to be used by the operator.
2) Tracking Subsystem
In fig. 11 shows we have made it possible to develop a
tracking subsystem that is sophisticated and provides a service
that is able to serve the security industry in different ways. The
tracking subsystem is now capable of analyzing images
according to information that user will be able to visualize. We
have encompassed a template matching system that is able to
process images and compare them with available data image in
Template (Object Image)
Template Matching
Update Template Position / Image
Output
Fig. 11. Tracking Design.
the security system. This means that the security operator is
able to match the capture image with the existing data image in
their system. This can help in the identification of the threat
much easily. Once the object image is processed through the
matching system, we have designed an update template position
that is able to analyze the image according to the requirements
of the security system updated. The update template will be able
to inform the user the threat analyzed by sending the data in the
security system.
3) Analysis subsystem and architecture:
In the analysis subsystem in fig. 12 we wanted to ensure
that the information the user gets from the multi- camera video
system is accurate and also in line with the command of the
operator. Therefore, we counted in trajectory software that is
able to move images to a specific direction or to get specific
information based on the command of the user. The image is
then transferred to the pattern matching section where the
analysis takes place. The image information undergoes a
thorough assessment by using a trajectory dictionary. The
dictionary will look for the information required about the
image and forward the data into the recognition result section.
The recognition result section is capable of providing accurate
data about the image provided it has the necessary references.
For this multi-camera video system, we have included common
reference sections extracted online that will serve as a
demonstration during the integration process.
alert the operator on the malfunction or the security breach of
the multi-camera video system by a cyber-attack. The decision
making functionality of the system will detect an object from
different camera angles when that target is successfully
detected. In order to use this process certain methods must be
done. In Fig. 14,15 and 16 shows location of the object will be
shown in the location variables these are supposed to be
between a certain value so that the X and Y coordinate can tell
if the person is near a certain object. These numbers are the X
and Y coordinates of an area that is step up as a targeted area in
a replicated scenario. If there are any coordinates between the
given numbers, then this would suggest that there would be an
object inside the target area. Moving on there is a location value
being kept inside the if statement now this will keep track of the
number of the locations in the X or Y area when the person
(object) is inside the designated area. Now the number of
frames will vary according to how accurate that detection is
inside of the area.

Fig. 14. The System Labels Moving Object (red).

Fig. 12. Analysis Design.

4) Decision-making subsystem:
This algorithm used to make final decision in this system. Fig. 15. The System Labels them as a Normal Zone (blue).
In fig. 13 shows the final process of security system operations,
and it is the subsystem that alerts the operator on the threat. The
decision-making subsystem is equipped with an alert system
that can alert the operator on the information provided by
another subsystem. Based on the analysis provided, the
subsystem will determine the threat level and immediately
relays the information to the security department, alerting them
of the intrusion by an unknown threat. The subsystem only acts
as the decision maker for the entire system and chooses who the
Fig. 16. The System Labels them as Threat Zone (purple).
threat is, by providing relevant data. The subsystem can also
VII. CONCLOSON
In this paper, video surveillance system has been discussed
in detail, and the attempt has been made to solve the problem
of object detection from a video stream. Specially, in this paper,
we have presented a multi-camera video surveillance system to
automate the video monitoring and to minimize the human
involvement for monitoring which can make significant
impacts in the security industry. The proposed approach can
Fig. 13. Design Making subsystem.
detect and recognize a human target from videos taken from
cameras mounted on the wall.
REFERENCES
[1]. D. B. Rawat, C. Bajracharya, and S. Grant, “nroar: Near real-time
opportunistic spectrum access and management in cloud-based
database-driven cognitive radio networks,” IEEE Transactions on
Network and Service Management, vol. 14, no. 3, pp. 745 – 755,
September 2017.
[2]. D. B. Rawat, M. S. Parwez and A. Alshammari, "Edge
Computing Enabled Resilient Wireless Network Virtualization for
Internet of Things," 2017 IEEE 3rd International Conference on
Collaboration and Internet Computing (CIC), San Jose, CA,
2017, pp. 155-162.
[3]. W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing:
Vision and challenges,” IEEE Internet of Things Journal, vol. 3,
no. 5, pp. 637– 646, 2016.
[4]. Azriel R. and Avinash K. Digital Picture Processing. New York:
Academic Press.1982.
[5]. Berthold K. and Paul H. Robot-camera surveillance Vision.
Austria: MIT Press.1986.
[6]. Richard H. and Andrew Z. Multiple View camera security
Vision.London: Cambridge University Press. 2003.
[7]. Grard M. and Sing B. Emerging issues in security. New York:
Prentice Hall.2004.
[8]. Chaple, Manisha. Vehicle Detection and Tracking from Video
Frame Sequence. IJSER, 10 Jan. 2013. Web. 9 Oct. 2015.
[9]. Renesas Solutions for HMI Applications Part 2: Applying CMOS
Cameras in Embedded Devices”. Renesas Electronics
Corporation, 8 Oct. 2010. Web. 9 Oct. 2015.
[10]. Danda B. Rawat and Kayhan Zrar Ghafoor, "Smart Cities
Cybersecurity and Privacy", Elsevier Press, ISBN:
9780128150320, December 2018.
[11]. C. Stauffer, W. Grimson (August 1999). Adaptive background
mixture models for real-time tracking (PDF). IEEE Computer
Society Conference on ComputerVision and Pattern Recognition.
pp. 246252. doi:10.1109/ CVPR 1999 .784637.
[12]. T. Bouwmans; F. El Baf; B. Vachon (November 2008).
Background Modeling using Mixture of Gaussians for
Foreground Detection - A Survey. Recent Patents on Computer
Science 1:219237.

View publication stats