See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/297676413

Heterogeneous Information Fusion and Visualization for a Large-Scale

Intelligent Video Surveillance System

Article  in  IEEE Transactions on Systems, Man, and Cybernetics: Systems · January 2016

DOI: 10.1109/TSMC.2016.2531671

CITATIONS READS

60 1,132

3 authors, including:

Ching-Tang Fan Yuan-Kai Wang

Fu Jen Catholic University Fu Jen Catholic University
12 PUBLICATIONS   242 CITATIONS    66 PUBLICATIONS   995 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Video Surveillance View project

Deep Learning for End-To-End Person Detection, Tracking and Re-identification across Cameras View project

All content following this page was uploaded by Yuan-Kai Wang on 15 May 2018.

The user has requested enhancement of the downloaded file.

Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
Heterogeneous Information Fusion and
Visualization for a Large-Scale Intelligent Video
Surveillance System
Ching-Tang Fan, Yuan-Kai Wang, Member, IEEE, and Cai-Ren Huang
Abstract—Wide-area monitoring for a smart community can be
challenging in systems engineering because of its large scale and
heterogeneity at the sensor, algorithm and visualization levels. A
smart interface to visualize high-level information fused from a
diversity of low-level surveillance data, and to facilitate rapid
response of events, is critical for the design of the system. This
paper presents an event-driven visualization mechanism fusing
multimodal information for a large-scale intelligent video
surveillance system. The mechanism proactively helps security
personnel intuitively beware events by the close cooperation
among the visualization, data fusion and sensor tasking. The
visualization not only displays 2D, 3D and geographical
information within a condensed form of interface, but also
automatically shows the only important video streams
corresponding to spontaneous alerts and events by a decision
process called display switching arbitration. The display
switching arbitration decides the importance of cameras by score
ranking that considers event urgency and semantic object features.
This system has been successfully deployed in a campus to
demonstrate its usability and efficiency for an installation with
two camera clusters that include dozens of cameras, and with a lot
of video analytics to detect alerts and events. A further simulation
comparing the display switching arbitration with similar camera
selection methods shows that our method improves the
visualization by selecting better representative camera views and
reducing redundant switchover among multiview videos.
Index Terms—Display switching arbitration, information
fusion, third-generation surveillance system (3GSS), visualization,
visual surveillance, visualizability.
I. INTRODUCTION
T HERE has been considerable emphasis recently on
intelligent video surveillance research for achieving
automatic interpretation of scenes and understanding the
behaviors of objects. Safety and security surveillance, which
targets crime prevention and forensic analyses, is one of the
most critical applications of video analytics and intelligence,
Copyright (c) 2015 IEEE. Personal use of this material is permitted.
However, permission to use this material for any other purposes must be
obtained from the IEEE by sending a request to pubs-permissions@ieee.org.
C.-T. Fan is with Graduate Institute of Applied Science and Engineering, Fu
Jen Catholic University, New Taipei 24205, Taiwan (e-mail: bevis@islab.tw).
Y.-K. Wang is with the Department of Electrical Engineering, Fu Jen
Catholic University, New Taipei 24205, Taiwan (e-mail: ykwang@ieee.org).
C.-R. Huang is with Graduate Institute of Applied Science and Engineering,
Fu Jen Catholic University, New Taipei 24205, Taiwan.
1
especially for smart communities [1]. Recently, automated
visual surveillance has been advanced to the third-generation
surveillance system (3GSS) [2] that installs a high number of
cameras in geographically diverse locations by a distributed
manner for establishing a multimodal camera network. The
3GSS is the culmination of research efforts from various
disciplines ranging from signal acquisition, data fusion,
communication models, video analytics, to software
architecture.
A technical evolution of surveillance system from first
generation to third generation is briefly explained in Table I.
The first generation of surveillance system (1GSS) is a
complete analogue system. Analogue closed-circuit television
cameras capture the observed scene and transmit the video
signals over analogue communication links to central back-end
systems, which present and archive the video data in analogue
form. Less intelligent techniques can be applied to improve the
1GSS. The second generation of surveillance system (2GSS)
uses digital back-end components but analogue front-end
equipment. However, digital video storage enables automated
event detection and alarms by robust computer vision
techniques such as abandoned objects detection and behavior
analysis. Proactive detection of alarming events substantially
improves the quality of the surveillance system and lessen the
burden of investigation tasks from manually watching all
recorded videos into semi-automatically collecting and
organizing detected events. 3GSS is evolved towards large
distributed and heterogeneous (with fixed, PTZ, and active
cameras) surveillance systems for wide-area surveillance.
Integrating huge volume of information from a diversity of
cameras enables a single human operator to monitor behaviors
over wide areas and investigate the inferred meaning from
compound events.
Ideally, an effective 3GSS should be capable of
automatically issuing alerts, events of interests to operators in a
control room environment. It avoids manual monitoring of
cameras and prevents the operators from suffering information
overload and a short attention span [3]. The 3GSS has several
capabilities: automatic event understanding for alerting
operators, computer-based monitoring of a target, and tracking
of targets among cameras [4]. Some systems in 3GSS focus
more on offering intelligent functions, such as intrusion
detection and parking space counting, based on predefined
objectives for each sensor. However, heterogeneity of diverse
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 2

TABLE I. SUMMARY OF THE THREE GENERATIONS OF INTELLIGENT SURVEILLANCE SYSTEMS.

Characteristics
Analogue system with
1GSS
centralized architecture
Partially digital system with
2GSS
centralized architecture
Fully digital system with
3GSS large-scale, distributed and
heterogeneous architecture
Surveillance Tasks
Video recording for manual
investigation after events
Video analysis for semi-automated
investigation after events
Video understanding for
semi-automated investigation
during event time
Research Issues
Video broadcasting and storage
Real-time computer vision for robust detection, tracking
and behavior analysis
Multisensor data fusion, space-time signal
correspondence, task coordination and distribution,
video communication and intelligent visualization etc.

communication and cooperation among the three main

Fig. 1. The proposed system with a visualization mechanism on top of a
classical 3GSS system.
information, including event messages, keyframes, and live and
recorded videos, complicates the design of the systems [5]. In
particular, their user interfaces are sundry in a mess due to the
need to display a lot of diverse events and information.
Information overload problem still exists in 3GSSs [6] because
messy information and events incurred by the large scale of the
system have to be digested and processed by well-trained
security personnel. Developing new visualization mechanisms
to improve interfaces by aggregating and fusing diverse
surveillance information is imperative to foster and accelerate
the progress of 3GSSs.
This study proposes a novel visualization mechanism for a
large-scale intelligent video surveillance system of 3GSSs. The
bottom of Fig. 1 represents a 3GSS with heterogeneous
cameras (analog, digital, embedded, and pan-tilt-zoom) in
distributed fashion. Intelligent tasks, such as the detection and
recognition of objects, events and behaviors, analyze videos
from several cameras and provide information to be shown in
user interfaces, that can be messy when a visualization scheme
fusing and condensing information is absent. Scene
visualization, data fusion and sensor tasking are the main
components of our visualization mechanism (middle of Fig. 1).
Events, keyframes, videos and 3D scene are automatically
fused by the visualization mechanism. A compact and
event-driven interface shows only critical information that is
filtered and fused by the visualization mechanism. Finally, a
scalable message-push architecture is adopted in the
communication management component to facilitate the
components.
The mission of scene visualization component is
event-driven 2D and 3D visualization, incorporating
information of events, keyframes, streaming videos, and 2D/3D
geographic context. In addition to the fusion of heterogeneous
information, the event-driven visualization includes a novel
design to adaptively display live video streams by selecting
only important camera views of interest from a high number of
heterogeneous cameras. A visualizability measure is devised to
evaluate the importance of the cameras and a display switching
arbitration is developed to decide the priority of camera views
for visualization. The sensor tasking component receives
commands from interface and dispatches coordinated tasks to
video analytics. Two missions, focus-of-attention and
collaborative surveillance, are included. The focus-of-attention
mission represents the task management of general video
analytics detecting events for a surveillance scene through the
use of stationary cameras, and the collaborative surveillance
mission indicates the task management of special video
analytics employing active cameras to assist stationary cameras
for capturing crucial details such as close-ups of faces. When an
event is triggered, the data fusion component activates sparse
geolocation mission to bind the event with geographical
information, and summarization mission to reduce redundant
information and extract keyframes, event clips and synopsis of
the events.
Our contributions are threefold. First, the event-driven
visualization scheme in Fig. 1 is first proposed by this paper to
promptly display only critical information in a compact user
interface. A visualizability measure concerning semantics of
visual features and the importance of cameras, views and
events are proposed. Critical information for visualization is
decided by display switching arbitration that optimizes the
visualizability measure. Second, the data fusion mentioned in
Fig. 1 is novel. It integrates a diversity of information from
camera sensors and geography is devised to present the
compact user interface for human operator. Video
summarization and sparse geolocation binding are two critical
techniques developed in this paper to achieve this. Third, a full
and improved 3GSS system was developed to demonstrate the
usability and necessity of intelligent visualization. The
communication management, sensor tasking, several video
analytics tasks and multimodal camera networks implemented
in this system shown in Fig. 1 are necessary to support our
intelligent visualization schemes. However, details of classical
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
topics in 3GSS, such as video analytics tasks are not proposed
in this paper and are referred to [7-16]. This paper gives more
attention to the proposed event-driven visualization and data
fusion.
The remainder of this paper is organized as follows. Section
II discusses background information and reviews related work
on large-scale intelligent surveillance systems. Section III
introduces the data fusion and event-driven visualization design.
In Section IV, the details of the remaining system components
are presented, including sensor tasking and communication
management. Section V describes a system testbed operated in
a campus, and a simulated experiment for the display switching
arbitration. Finally, conclusion and suggestions for future work
are provided in Section VI.
II. STATE-OF-THE-ART
The study of large-scale systems for wide-area video
monitoring is briefly reviewed and discussed. A substantial
number of studies have been devoted to video analytics,
cooperative surveillance, and software architecture in the
context of multi-camera surveillance systems; however, a
modicum of the systems explored the fusion and visualization
problem.
Intelligent video surveillance, which applies computer vision
algorithms to detect and recognize objects and events for
specific prevention and forensic tasks [17], aims at active
monitoring. A visualization scheme that effectively displays
videos and events can simplify human operation and reduce
manual tasks. Matrix arrangement of video displays to show
objects and events corresponding to each single camera has
been widely used as a traditional visualization scheme.
However, the one-camera-one-monitor methodology is not
feasible for a large-scale system with a high number of cameras
[18]. An alternate approach employs a large screen to show all
camera views in turn or randomly; however, lack of semantic
information such as spatial and temporal relationship among
objects and events, and importance of camera views, incurs
heavy overload for the management of the systems. Displaying
tremendous information within a limited visual space cannot be
a trivial task.
Object- and event-oriented interface designs are efficient
ways for information visualization of large-scale systems. In
[19] and [20], only information of specific objects such as
locations and tracks of objects are visualized for sustaining
attention. They offer only one interface by locating the object
information across multiple cameras on floor plan map for
assisting navigation. [21] provides predictive selection of
camera views by object tracking to simplify operator's tasks.
Nevertheless, methods [19]-[21] could be applied to
small-scale systems because they do not employ behavior
analysis and event detection for extraction of high-level
information. Event-oriented interface provides less but
important information for large-scale systems, and facilitates
responsive interface for critical threats. IBM S3 [22] integrates
diverse event detectors with one-detector-one-monitor
3
methodology into in a large-scale deployable system. The
number of user interfaces increases linearly and the workload
of the operator increases dramatically when the number of
detectors increases. Listing all events in an event inbox [23] is
beneficial, but text information only is deficient for responsive
decision and nonverbal information such as keyframes and
video clips is important.
Video summarization such as keyframe and skim provides a
compact representation by eliminating redundancy of videos
and preserving only crucial frames for better visualization of
stored videos. Keyframes are more widely used than video
skimming because of the ease of browsing and navigating [24].
A fundamental approach of keyframe extraction considers only
the changes of pixels and/or features in a frame [25]–[27]. A
more compact keyframe representation achieved by
incorporating object information is proposed by Erol and
Kossentini [28] and Kim and Hwang [29]. Studies [7] and [30]–
[31] further demonstrate that the object information with the
high-level features based on human visual perception improves
the optimization of keyframe extraction. In addition to these
methods for summarization of single videos, [32] and [33] are
more appropriate for large-scale systems because they extract
keyframes from multiple videos acquired from multiple
cameras with different viewpoints. Highly compact
representation is achieved by removing more redundant frames.
However, pixel and frame changes cannot be applied for the
multicamera keyframe extraction, but object and semantic
features become more critical for extracting meaningful
keyframes from multiple videos. Both centralized [32][33] and
decentralized [34][35] networking approaches have been
proposed. A global optimization mechanism not only adapting
to networking approaches but also incorporating semantic
features is important for video summarization in large-scale
systems.
Live video display within a limited visual space for a high
number of cameras is also challenging. A hand-off approach for
finding the next most meaningful camera is indispensable for
focusing visual attention on a specific object. Some studies [23],
[36]–[38] extract high-level semantic features for selecting a
dominant camera. The dynamic camera switching method
proposed in [23] identifies whichever camera exhibiting the
greatest image difference as the main sensor view. Huang and
Kim et al. [37] propose a probabilistic camera hand-off (PCH)
method for continuous object tracking. The ratio of foreground
blocks and the ratio of angle distance between the camera and
the object are calculated for each camera to obtain the
proximity probability. The camera with the highest proximity
probability is identified as the dominant camera. The dominant
camera can be identified using the tracking result of the
aforementioned hand-off approach, and its object trajectory is
represented in a map by using a homograph. Goshorn et al. [38]
propose a cluster-based multicamera surveillance network that
generates a camera selection manager (CSM) as a cluster
network. The CSM for the optimal view depends on weighted
importance and four semantic features.
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 4

The literature review shows that security personnel has to

assess events expeditiously by an effective visualization
mechanism. From the viewpoint of visualization, fulfilling two
requirements is necessary for enhancing the practicability of
surveillance cameras and monitors. The first requirement is a
universal user interface involving heterogeneous sensor tasking
that can effectively convey events, and the second requirement
is automatic display switching arbitration that selects
meaningful camera views. From the perspective of the system,
a relevant software and hardware architecture corresponding to Fig. 2. Four-tier scheme for the proposed visualization mechanism.
the new visualization mechanism is essential.

III. HETEROGENEOUS DATA FUSION AND EVENT-DRIVEN

VISUALIZATION
An event-driven user interface to guide the visual attention of
operators to appropriate situations is proposed. A four-tier
scheme shown in Fig. 2 is used to illustrate the idea. Each tier
digests and transforms diverse information into high-level
messages to be the input of next tier. The first tier acquires and
analyzes streaming videos to generate events. Tier 2 performs
sensor tasking for camera collaboration when events occur.
Geoinformation fusion for geolocation and visual fusion for
summarization are performed in Tier 3, which corresponds to
the data fusion component in Fig. 1. For data fusion, Tier 3
acquires public Web data from public-record databases. A
geomodel and an event model are used for supporting both
Tiers 2 and 3 for constructing a smart monitoring system. Tier 4
aggregates all related data, referred to the event-driven
visualization component in Fig. 1.
Tiers 1 and 2 are fundamental components been well
discussed in classical 3GSS systems. Our implementations of
these two tiers are described in next section or referred to
published papers. Most importantly, the tiers 3 and 4 are critical
components constituting the intelligent visualization scheme
proposed in this paper, and are explained in this section. The
data fusion subsection explains a compact interface for the
visualization of only one event with time-space correspondence.
The event-driven visualization subsection proposes an
arbitration mechanism for the visualization of multiple
spontaneous events.
A. Data Fusion for Compact Interface
A compact interface achieved by two techniques: time-space
correspondence and video summarization, is designed to fuse
multimodal information of surveillance data. Traditional data
fusion methods assume homogeneous signals are to be fused at
data level, feature level or class level by weighting schemes
which may be achieved by probabilistic approach. However,
heterogeneous information including symbolic, streaming
signal and geographic data cannot be fused by a homogeneous
weighting scheme, but only be fused into an effective and
compact interface by aggregating and binding the multimodal
information with spatial and timing constraints.
The configuration of the interface is shown in Fig. 3(a),
which consists of three areas: two-dimensional geographic map,
video streaming box and dynamic text message bar. The 2D
map shows a geographic layout of a monitoring area with
(a) (b)
Fig. 3. Interface configuration for multimodal information fusion. (a) Three
areas to display event’s information of geography, image, video and text. (b)
A set of master-slave video pairs to show streaming videos of clusters.
cameras placed in corresponding locations. An event box may
pop up alert messages and key frames for a camera when the
video analytics responsible for the camera detect events. Text
messages that can be read only by trained security personnel,
such as abbreviated event code names, camera identification
numbers, and event sketches, are shown in the dynamic text
message bar. Video streaming box is responsible for the play of
recorded videos of events and live videos of cameras.
As shown in Fig. 3(b), two streaming videos as a
master-slave pair is assigned to a camera cluster. A camera
cluster is a set cameras monitoring a physical area within a
neighborhood. There could be N clusters for a wide-area
surveillance system, and each cluster has two boxes to play
streaming videos. However, the number of cameras for each
cluster should be more than two, which means only two boxes
is not enough to show all the videos of a cluster. Therefore the
two cameras of a cluster chosen to be played in video streaming
box will be marked with specific colors corresponding to the
box’s colors. This color correspondence helps the
understanding of the synchronized and dynamic behavior
between 2D map area and video box area. In addition, an
automated scheme to dynamically select two camera views to
play recorded event videos is devised with a visualizability
model and a display switching arbitration method explained in
next subsection, while a manual operation by clicking a camera
mark in the 2D map can also force the interface to show event
clips and live videos in the master box of the cluster.
Two approaches of video summarization, event clips and key
frame, are provided in this compact interface for each detected
event. Both static and dynamic summaries improve the
efficiency of prompt event filtering and browsing, but have
different implications and effectiveness [39]. An object-based
key frame extraction method described in our previous paper [7]
is applied to this system. When an event is triggered, the
corresponding video analytic task records a clip of the event
and synchronizes the time and space of the event with our
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 5

visualization agent, which is described in Section IV. The agent camera with the highest visualizability score, 𝐶𝑣 =
analyzes the synchronized event clip and extracts key frames in
arg max 𝑉 𝐶𝑖 , where 𝐶𝑖 is the ith camera.
real-time to promptly show the key frame in the event box when 𝐶𝑖
an event occurs. The synchronized event clip will be displayed The event term is modeled by a weighted indicator function
in the video streaming box accompanying with the event box, as
manually or automatically. 𝑉+ 𝐶A = 𝑃+ ∙ ∆ 𝐶A (2)
Space-time correspondence of multimodal information for a
single event is therefore effectively fused in this compact where ∆ . is 1 if an event is triggered in camera 𝐶𝑖 . 𝑃𝑒 is
interface. Time stamps of events and GPS coordinates of defined as the priority of the event.
cameras are two fundamental data for space-time The object term represents the score of a weighted sum of 𝐽
synchronization. Spatio-temporal hypergraph [40] could be object features, and is defined as
applied to organize and synchronize the multimodal H

information. 𝑉/01 𝐶A = 𝑤1 ×𝑃1 𝐶A (3)

With more than two spontaneous events, this compact 1I%
interface still lacks of space to display the information of all
where 𝑃𝑗 . and 𝑤𝑗 denote presentation probability and
events. A strategy called display switching arbitration is
necessary to dynamically select the most important events for importance weight aggregated for the jth∈ 1,2, … 𝐽 feature,
𝐽
visualization. respectively, with 𝑗=1 𝑤𝑗 = 1. The presentation probability is
calculated by
B. Event-driven Visualization by Display Switching LM NO
𝑃1 𝐶A = P L N . (4)
Arbitration OQR M O

Traditional way to display multiple events or all camera
videos is infeasible for 3GSS due to limited space of interface.
A smart display switching method is required to assist a single
human operator to operate a multicamera surveillance system at
a relatively high level of abstraction. An event-driven
visualization strategy with dynamic arbitration of display
switch is devised here. The display of our visualization
interface can automatically switch to the most meaningful
camera views by using a visualizability model.
Display switching arbitration aims at providing a dynamic
view of video playing. The display switching arbitration
approach consists of a local optimized process and a
decentralized optimized process. Local visualizability is
signified by the most representative frame of an object in the
local video. In multiview videos, clustered visualizability is
represented for the most representative frame that provides the
optimal view. After assessing semantic significances of
visualizabilities, we design two criterion functions by applying
the visualizabilities to achieve the integration of meaningful
representations.
Let us first consider a cluster of cameras 𝐂 = 𝐶% , 𝐶' , … , 𝐶) .
For visualizability modeling, we express a scoring vector as a
linear combination of two individual terms:
𝑉 = 𝑉+ + 𝛾 ∙ 𝑉/01 . (1)
The event term 𝑉𝑒 encodes the contribution of events for
visualizability, and the object term 𝑉𝑜𝑏𝑗 the contribution of
object features for visualizability. A penalty 𝛾, limited to the
range [0,1), is added to adjust the importance of object features
with respect to the behaviors in the monitoring area. The
dominant camera 𝐶𝑣 is obtained by calculating the
visualizability of each camera, 𝑉 𝐶𝑖 , and then choosing the
The parameter 𝑋𝑗 𝐶𝑖 is the jth element in the feature vector
𝑋 𝐶𝑖 , which is calculated by applying the local optimized
process to camera 𝐶𝑖 . We perform keyframe extraction in a
specific period as the local optimized process. The process is
expressed as
𝑋 𝐶A = 𝑋T 𝐶A |𝜖 = argmax 𝐹Y 𝑋W 𝐶A , 𝑤 , 𝑡 − 𝛿
W
(5)
<𝑡 <𝑡+𝛿
where the criterion function 𝐹𝐿 𝑋𝑡 𝐶𝑖 , 𝑤 =
𝐽
𝑗=1 𝑤𝑗 ×𝑋𝑗 𝐶𝑖 can be considered as local object
visualizability. The weighted sum of 𝑋𝑗 . is the representative
score of the local camera.
The proposed method considers three object features, such as
the region of object size, the region containing the object’s skin,
and the face of the object, to calculate the presentation
probability for selecting a camera. We employ a Kalman filter
to optimize all features to reduce the measurement noises in
these object features.
IV. THE PROPOSED LARGE-SCALE INTELLIGENT VIDEO
SURVEILLANCE SYSTEM
The proposed large-scale IVS system has a complex
software and hardware architecture for use in a real
environment; the detailed architecture is shown in Fig. 4. The
main constituents of this system are four subsystems:
intelligent visualization, sensor tasking, communication, and
video streaming and storage.
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
Fig. 4. Detailed architecture of the large-scale intelligent video surveillance
system.
A brief description of interactions among the four
subsystems is given first. Events triggered by the focus of
attention in sensor tasking subsystem are transmitted to the
event agent in intelligent visualization subsystem. The event
agent then manages and stores diverse information on the event
into several databases; in addition, the event agent sends the
event to data fusion and the visualization. The visualization
agent immediately presents all related information on multiple
interfaces. The visualization agent also requests the cooperative
surveillance to invoke related video analytics and control PTZ
cameras. A queue manager is responsible for data bridging and
buffering among subsystems. Details of the four subsystems are
given below.
A. Intelligent Visualization Subsystem
Our intelligent visualization includes multiple user interfaces
implemented for the central control room of our system. A
Web-based interface, referred to herein as the main interface
(See Fig. 6(a) for an example), uses the data fusion and smart
display switching described in previous section to seamlessly
present a lot of prompt event results. It also offers a live mode
for real-time event alerting and a playback mode for event
retrieval.
Two other interfaces are also provided to supplement
abundant information of events. A 3D interface (See Fig. 7 for
an example) blending Google earth and live videos by
homography projection is provided to be an auxiliary display
for aiding operators in determining the location of tracked
objects [8]. The 3D interface provides an immersive
surveillance experience to simulate peripheral vision. Video
synopsis [9-10] as a high-efficiency video object retrieval
technique is also used in this system to effectively remove
redundancies from a video and condense long videos into clips.
It provides object-based video retrieval to supplement the event
information in the main interface. An example of video
synopsis is given in the middle one of the top three screens in
Fig. 6(c).
6
The interaction among the three interfaces is all trigged by
the main interface, because the main interface gives the
personnel an abstract but summarized information and the
other two interfaces give the personnel further detailed
information. The 3D interface is controlled from the main
interface by selecting a corresponding area on the 2D map.
More details of the area will then be monitored in-depth by the
security personnel with three-dimensional views. To see
object’s behavior in the video synopsis interface, the personnel
chooses an object of an event displayed in the main interface
and the visualization agent then controls the synopsis task to
play videos.
B. Sensor Tasking Subsystem
The sensor tasking subsystem includes a lot of video
analytics tasks that work independently or collaboratively.
Focus of attention represents a group of tasks to detect
interested events independently. Cooperative surveillance
represents collaborative tasks to monitoring a wider area of
interest. Focus-of-attention tasks are called attention tasks for
brevity, and collaboration tasks are the shortened form of
collaborative surveillance tasks.
Attention tasks in the focus of attention group are subdivided
into three stages according to the necessity of continually
manual inspection. Each stage is influenced by the consequence
of the foregoing stage, and the tasks in higher stage need more
cameras for analysis. The first stage involves preprocessing
tasks. The second stage involves detecting and tracking routine
information about objects such as moving pedestrians and
empty parking spaces. Complex situations are interpreted on
the basis of object recognition, and evidence reasoning is
performed for event inference. Processes at the third stage tend
to be highly situation specific, and behavior and activity
analysis tasks depend highly on the segmented regions
corresponding to the objects of interest in an image. The final
results of attention tasks are event-focused.
There are four attention tasks. To confirm that the image
quality and the field of view of surveillance videos are
favorable, a camera anomaly detection [11] task detects
abnormal camera events such as defocusing, translating, and
covering by statistically calculating accumulated variations of
features in the temporal domain. Robust detection of vacant
space in outdoor parking lots by using a multicamera
monitoring method was practiced; the task was named parking
space counting [12]. The task automatically counts the number
of vacant spaces in a large parking lot, demonstrating its
potential use for parking assistance. Vehicle anomaly, a
concern regarding illegal parking, is detected by performing an
illegal parking detection [13] task. This detection method was
designed without background modeling. A texture-based
method, which can overcome environmental challenges, is used
for target tracking. Intrusion detection for a climbing [14] task
is a constructive constituent for monitoring people anomalies.
The human object in videos is segmented into body parts using
deformable triangulation, and postures are analyzed for
verifying climbing events in a restricted area.
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
Each collaborative task in the cooperative surveillance group
is a single task with more than two cameras and more than two
video analytics for an event. It is beneficial for a monitored area
wider than one camera view. A PTZ camera is always utilized
to complement a static camera. Video analytics of a
collaborative task are decentralized and performed on
embedded systems in PTZ cameras. Collaborative tasks are
reactive rather than proactive.
Two collaborative tasks are implemented. In a loitering
detection task, a moving target in a wide open area is monitored
through the automatic control of an embedded PTZ camera [15],
combined with an intrusion detection algorithm with a static
camera for continuous tracking. An illegal parking task
combines a car parking detection algorithm with a human face
detection [16] for a dynamic scene. The car parking event
detected in an illegal position triggers the analytics in a PTZ
camera to dynamically track a high-resolution human face,
which is useful for forensics.
C. Communication Subsystem
A publisher-subscriber communication pattern was
implemented to facilitate message delivery and enhance
cooperation among the event agent, video acquisition and
streaming agent, visualization agent, attention tasks, and Web
clients. The sender is called a topic publisher, and the queue
manager according to the registered publisher-subscriber table,
who requires information from the appointed publisher, sends
the information to all subscribers. Because the information may
be too abundant to transmitting, many queues are used for
publishers by the queue manager. Three topics and five queues
are set for various publishers, and a topic can map to several
queues.
When an event is triggered, the visualization agent publishes
information fused to the topic of visualization, which is handled
by the alert queue and the short message service (SMS) queue.
The alert queue delivers push notifications to Web browsers,
and the SMS queue sends an SMS notification. The keyframe
extracted during the mission of summarization consists of
simple text messages and links that can be used to send SMS
notifications to mobile clients; such links can be effective for
driving users to a Web site or download link. Compared with
push notifications that are limited to smartphones, SMS support
is ubiquitous and available in all phones. Furthermore, a video
streaming server publishes on the topic of video streaming. The
video queue obtains videos and then embeds them into the main
interface.
D. Video Streaming and Storage Subsystem
The proposed system provides the output on the Web-based
user interface and receives streaming videos from multimodal
cameras. The Web interface facilitates systems used on
portable devices such as smartphones and smartpads. The user
interface is melded from static and dynamic data, where the
HTTP protocol is used for accessing static data such as
interface layout. The dynamic data achieve bidirectional
communication with a server through an AJAX or WebSocket
protocol, facilitating live content transmission and the
7
identification of real-time events. For streaming live video from
existing IP cameras and streaming servers, a VLC plugin with
AJAX is used. With the rapid development of HTML5
technology, the HTML5 elements are effective for accessing
static video clips and map services.
Cameras in the system are not restricted to networked and
analog cameras, where the analog signal is digitized through a
video server that provides real-time real time streaming
protocol (RTSP) streaming video to the back-end. To balance
the network traffic and monitoring needs, each stationary
camera owns at least two real-time streaming videos
simultaneously. The two streaming videos may have different
settings. One is a high-quality video encoded in the H.264
format with resolution as high as possible for providing
forensic evidence in the storage subsystem, which is mounted
to a large-scale storage area network with cloud storage.
Another video has the QVGA resolution which is enough for
video analysis. In addition to containing historical footage, in
the storage subsystem, a relational database management
subsystem built on SQLite is served as event database, and
video clips of marked events and video synopses corresponding
to events are recorded in the historical footage. If clients query
events, the native video clips can be cut from the event database
and streamed from the video streaming server.
V. TEST RUN AND CASE STUDY
The proposed system has been developed and installed in a
campus with the Vision Based Intelligent Environment (VBIE)
project [41] in Taiwan. More than 20 cameras classified into
two camera clusters were setup and distributed in two disjoint
regions of the campus, as shown in Fig. 5. Each cluster has a
cooperative task with one collaborative PTZ camera.
Configuration of the camera networks cannot be trivial and a
lot of approaches were applied in the proposed system for
different situations and constraints. The first thing to mention is
that both overlapping and nonoverlapping field-of-views (FoVs)
of cameras are utilized in this system. The four cameras for
parking space counting [12] is configured to be overlapping
FoV, and the cameras for other video analytics are configured
to be nonoverlapping FoV. Camera calibration for the parking
space counting utilizes epipolar-plane constraint among
cameras and feature points of the scene to obtain intrinsic and
extrinsic parameters of cameras. Exact object positions in
three-dimensional space can then be estimated for the accurate
counting of vacant spaces. A pair of PTZ and static cameras in
the cooperation surveillance tasks are collaborative to track the
same object for an event, which also needs to be coordinated
and calibrated. Object location is estimated first in the static
view and the PTZ camera is then controlled to capture a high
resolution object image or to track the behavior of the object.
An assumption of depth information is applied in advance, so
the static camera sends only x and y coordinates to the PTZ
camera to adjust pan and tilt parameters. Details of the camera
calibration and configuration for parking space counting and
cooperative surveillance tasks are referred to published papers.
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 8

(a)

Fig. 5. Sketch of the two camera clusters of the system in a campus map. The
two red rectangles in the top map represent the two disjoint clusters, and red
dots are cameras. 3D views of the two corresponding camera clusters are
shown at the bottom.

The remaining cameras have disjoint FoVs and an intuitive

method is applied to ease the calibration loading of the whole
system. A sparse geolocation method by calibrating GPS
coordinates of cameras with the coordinates in 2D map is used.
The sparse geolocation refers to the geographic location of a
(b)
camera where an event happens, which is used in our
event-based representation in the visualization mechanism.
The Web-based main interface offers live and playback
modes, as shown in Fig. 6(a) and (b). The live mode presents
real-time information, and playback mode is useful for
event-based retrieval. The illegal parking event shown in this
example is triggered and the event-box with summarized text
attributes and keyframe is popped up to indicate the alerted
camera. The corresponding camera views of a cooperative task
is live-streamed to the second line of the video streaming box,
with the master box for a collaborative zoomed view and the
slave box for alerted camera view. A playback mode example
shown in Fig. 6(b) illustrate the play of a recorded event clip of
an intrusion detection by illegal fence climbing. Fig. 6(c) shows
(c)
a real picture of the multiscreen interface deployed at our Fig. 6. Web-based main interface of (a) live mode and (b) playback mode.
central control room. The two large interfaces at the bottom are (c) Multiscreen interface.
the main interface in the right and the 3D interface in the left. the accuracy and stability of the display switching arbitration.
Three other screens at the top area display results of three
selected tasks from right to left: the parking space detection A. Overall System Performance of Real Test Runs
task, video synopsis, and live camera view for any selected one. The system was operated and tested for more than six months,
Six examples of the 3D interface for the road surveillance and its reliability and robustness were verified from 33 formal
around a building are shown in Fig. 7. The 2D map in the test runs. The test runs were conducted on normal and abnormal
middle is part of the main interface. A click in the 2D map behaviors of humans and vehicles under different weather
commands the 3D interface to show the view of Google earth at conditions in a cluttered outdoor environment. Few false alarms
the location, and a live video is blended with the 3D view to were triggered, and many illegal events were successfully
give a live 3D monitoring. detected. Detailed performance analysis of each single task is
The following give experimental results of overall referred to [7-16]. The system failed with three times, that is
performances in pragmatic test runs, and simulation results of 91% accuracy, when improper key frames and video clips were
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 9

(a)

Fig. 7. Six examples of 3D interface interacted with the 2D map in the center.
The darker areas in six 3D examples are 3D information from Google Earth,
and the lighter areas in 3D examples are blended live videos by homographic
transformation. (b)

shown in the visualization interface. Concurrency control and

conflicts among tasks induced the problem and a delay and
queuing mechanism with database concurrency could solve the
issue.
Fig. 8 shows examples of successful event detections under
challenging conditions. An illegal parking event occurred in a
rainy day is shown in Fig. 8(a); dynamic face detection was (c)
triggered to capture the driver’s face, which was partly
obscured by an umbrella. Fig. 8(b) and (c) show the events in a
sunny day with the presence of complex and dynamic shadows.
A successful case of intrusion detection is shown in Fig. 8(b); a
PTZ camera continuously tracked the person who climbed over
a wall, as shown in Fig. 9. When the camera was sabotaged by a
bag, the camera anomaly detection task issued an alert and
provided a pair of images before and after the anomaly event, as (d)
depicted in Fig. 8(c). Fig. 8(d) presents an example of object Fig. 8. Examples of successfully detected illegal events under different
weather conditions.
retrieval. The left image shows a frame of a video synopsis, and
the source video of the object in the yellow box is shown on the
right side.
Figs. 8(a) and (b) show two collaborative cases. The alert is
first triggered by the attention task working on stationary
cameras, and subsequently, visualization agent assumes the
responsibility of capturing a close-up image of human face by
controlling a collaborative PTZ camera.
When an event is triggered, the keyframe is fused with the
map and text messages. Fig. 10 shows three examples of
actively triggered events. Figs. 10(a) and (b) show a climbing
event and a camera anomaly event, respectively. A set of
collaborative events involving an illegal parking event and Fig. 9. An example of embedded PTZ camera control for moving-object
dynamic detection of the driver’s face is shown in Fig. 10(c). In tracking.
this case, after an alert was displayed in the 2D map for the different event and object visualizabilities of the cameras. The
illegal parking event, the collaborative dynamic face detection event visualizability is equal to watchful priority in this
task found the face close to the vehicle’s door. The face image situation because all cameras have events.
was captured if dynamic face detection was successful. There Table II shows three cases of different watchful priorities at
are only three cameras in the camera cluster and each camera three events, where the climbing event is occurring at Camera 1
simultaneously records a video for the same triggered event. and the camera anomaly event and illegal parking event are
Different display switching results are produced because of occurring at Cameras 2 and 3, respectively. The object
visualizability is given by 𝑉𝑜𝑏𝑗 = (0.2,0,0.8) because Camera
2 does not have any object and Camera 3 records a clear face
that is larger than the object of Camera 1. The penalty 𝛾 is set to
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems 10

80 183 240 340 400 600

C1

C2

C3

(a) (b)
C4

Top
view

(a)
77 128 189 237 249

340 382 417 572 624

(c)
Fig. 10. Examples of actively triggered events.

TABLE II. EXAMPLES OF DISPLAY SWITCHING ARBITRATION FOR EVENTS (b)

OCCURRING SIMULTANEOUSLY AND γ=0.8.
Fig. 11. Result obtained using the proposed approach in the case of Smart Home
Climbing Camera anomaly Illegal parking sequences of the ICDSC 2009 data set. (a) Original images of sequences and a
Event @ Camera 𝐶A top view. (b) Cluster keyframes of the decentralized process.
@𝐶% @𝐶' @𝐶c
d⃑/01 (𝐶A )
𝑉 0.2 0 0.8
d⃑+ (𝐶A )
𝑉 2 2 1 in Fig. 11(a). Three of the cameras have overlapped FoVs. The
Case 1
𝑉d⃑(𝐶A ) 2.16 2 1.64 resolution of all sequences was 320 × 240. The frame rate of the
Case 2
d⃑+ (𝐶A )
𝑉 2 2 2 video was approximately 10 frames/s, and the videos were
𝑉d⃑(𝐶A ) 2.16 2 2.64 produced using the XVID codec. We apply background
d⃑+ (𝐶A )
𝑉 2 3 2
Case 3
𝑉d⃑(𝐶A ) 2.16 3 2.64
subtraction to extract human objects in the videos.
Fig. 11(a) shows a person walking through the entrance of
the living room, proceeding to sit on the sofa for watching TV,
0.8. In Case 1, 𝑉𝑒 = (2,2,1), and the final visualizability is
and then picking up a table magazine, then changing to the
given by 𝑉 = (2.16,2,1.64), where Camera 1 is selected as the other seat during reading the magazine. The segment that we
dominant camera. In Case 2, when the three events have the sampled in sequences was from Frame 0 to Frame 450; there
same watchful priority of 2, the visualizability of Camera 3 was no overlap in the FOVs of Cameras 1 and 2 for this
increases to 2.64, resulting in this camera being the dominant sequence. The top view shows that the object remained in each
camera. When the priority of the camera anomaly is increased place. Three cameras in the living room detected the object in
to 3 in Case 3, Camera 2 becomes the dominant camera, and it Frame 183. The person who moves to another place is not
has a visualizability of 3. detected by Camera 3 in Frame 394. Finally, he returns to the
B. Evaluation of Display Switching Arbitration at Ordinary sofa and sit in Frame 470.
Times The selected dominant camera views of the proposed
clustered object visualizability are shown in Fig. 11(b). It
The accuracy and stability of display switching arbitration
demonstrates the capability of the proposed approach to select
are verified by a public data set, and compared with PCH [37]
the FOV with the representative object. When a person is
and CSM [38] algorithms. The weighted vector of the proposed
% % % monitored by two cameras, the camera that captures the face or
method was 𝑤 = ( , , ). We use the ICDSC 2009 data set [42] a larger extent of the skin is selected as the dominant camera.
c c c
that contains videos of routine activities of people in an indoor The accuracies, defined as the ratio of the number of image
and common home environment recorded using eight cameras. frames with the correct dominant camera selection over the
The multiview of this data set includes dense and sparse views. total number of image frames, of the proposed camera selection,
The data set includes complex activities rather than only PCH [37] and CSM [38] are 95.87%, 56.72%, and 56.73%,
walking. We selected an activity performed in the living room. respectively. When we replace the features of the PCH and the
The video sequences of four cameras of the activity are shown CSM with our proposed features, the accuracies of PCH and
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
(a)
(b)
(c)
Fig. 12. Results of camera selection by (a) PCH method [37], (b) CSM method
[38], and (c) the proposed method; the X-axis represents the frame number and
the Y-axis represents the camera number.
CSM are increased to 75.82% and 78.77%. The results show
that the proposed semantic features are very effective, and the
optimization algorithm with the visualizability criterion is an
effective design than the two compared algorithms. These two
components constitute the high accuracy of the proposed
method.
Fig. 12 shows the stability of camera selection for the test
scenario. Stability refers to the ability of an algorithm to stably
generate a cluster keyframe without false detection. Figs. 12(a)
and (b) show the results obtained using the PCH and CSM
algorithms, respectively. The PCH algorithm detected a
dominant camera only from Frame 160, which has false
negatives for frames from 70 to 150. Both algorithms were
highly unstable from Frame 160, where the object appeared in
both cameras 3 and 4. Fig. 12(c) shows a relatively smooth
result that is good for display switching. The use of Kalman
filtering, semantic features and the optimization of the
visualization model constitute the successful and stable results
of the proposed method.
11
VI. CONCLUSION
This paper presents a large-scale scalable system extended
from 3GSSs. The system integrates a lot of computer vision
tasks, such as object detection, camera anomaly detection,
keyframe extraction, and mobile surveillance, with the
knowledge acquired from over 10 years' experiences of
cooperation among academia, industry, and the government.
By integrating algorithmic tasks through the proposed
event-driven visualization and systems engineering techniques,
an efficient system for wide-area visual surveillance by single
security personnel is presented. The multitier scheme with a
novel visualization mechanism is proposed for centralizing all
heterogeneous surveillance information into a universal
Web-based user interface. A new camera selection method not
only considering objects but also events is developed with the
display switching arbitration. The method enables meaningful
FOV selection and smooth handoff from multiple cameras for
both normal and event-alerting and situations. The fuse of
multimodal information is advocated for event-driven
visualization. A collaborative scheme involving sensor tasking
at static and dynamic cameras is proposed for object-oriented
visualization for aiding continuous visual tracking.
Although the visualization mechanism and new system
concepts to complement 3GSSs have been presented in this
paper, future work remains. In addition to accuracy and
stability, the evaluation of system performances could apply
other measures such as time to receive events, ergonomics and
comfortability. A subjective evaluation of the degree of
assistance of the proposed approach to security personnel can
be quantitatively assessed with respect to user interface issues
such as ergonomics and comfortability. Robust multiobject
tracking across multiple cameras [43] can be helpful to display
switching arbitration. For example, if a person is to be
monitored across cameras, an efficient human tracking
technique can help the system determine the dominant camera
for display switching with more accuracy.
ACKNOWLEDGEMENTS
The authors would like to thank the professors in the Vision
Based Intelligent Environment project: Yi-Ping Hung,
Sheng-Wen Shih, Yong-Sheng Chen, Jun-Wei Hsieh,
Chi-Hung Chuang, Chin-Teng Lin, Cheng-Chang Lian,
Chin-Chuan Han, Hsi-Jian Lee, Sheng-Jyh Wang, Daw-Tung
Lin, Kuo-Chin Fan, Wen-Thong Chang and Wen-Hsiang Tsai,
for their contribution to the success of this system. The authors
also thank the graduate students of these professors for their
tireless efforts and supports.
REFERENCES
[1] X. Li, R. Lu, X. Liang, X. Shen, J. Chen, and X. Lin, "Smart
community: an internet of things application," IEEE Communications
Magazine, vol.49, no.11, pp.68-75, Nov. 2011.
[2] T. D. Raty, "Survey on contemporary remote surveillance systems for
public safety," IEEE Tran. on Systems, Man, and Cybernetics, Part C:
Applications and Reviews, vol.99, pp.1-23, Mar. 2010.
[3] G. Smith, "Behind the screens: Examining constructions of deviance
and informal practices among CCTV control room operators," Surveill.
Soc., vol. 2, no. 2–3, 2002.
Accepted by IEEE Transactions on Systems, Man, and Cybernetics: Systems
[4] X. Wang, "Intelligent multi-camera video surveillance: A review,"
Pattern Recognition Letters, vol. 34, no.1, pp.3-19, 2013.
[5] L. Yu and T. E. Boult, "System issues in distributed multi-modal
surveillance," IEEE Conference on Computer Vision and Pattern
Recognition, Minnesota, USA, Jul. 2007, pp.1-2.
[6] F. Porikli, F. Bremond, at al., "Video Surveillance: Past, Present, and
Now the Future", IEEE Signal Processing Magazine, vol. 30, no. 3, pp.
190-198, 2013.
[7] Y. K. Wang, L. Y. Wang, Y. C. Huang, and C. T. Fan, "An online
object-based key frame extraction method for the abstraction of
surveillance videos," in Proc. National Computer Symposium, Taiwan,
pp. 241-249, November 2009.
[8] K. W. Chen, C. W. Lin, T. H. Chiu, M. Y. Chen, and Y. P. Hung,
"Multi-resolution design for large-scale and high-resolution
monitoring," IEEE Transactions on Multimedia, vol.13, no.6,
pp.1256-1268, Dec. 2011.
[9] D. T. Lin and L. Y. Liu, "Method of detecting moving object." U.S.
Patent 121268,603. (Pending)
[10] Y. Pritch, A. Rav-Acha, and S. Peleg, "Nonchronological video
synopsis and indexing," IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol.30 no.11, pp.1971-1984, Nov. 2008.
[11] Y.K. Wang, C.T. Fan, K.Y. Cheng, and P.S. Deng, "Real-time camera
anomaly detection for real-world video surveillance," International
Conference on Machine Learning and Cybernetics, vol.4 , China, Jul.
2011, pp.1520-1525.
[12] C. C. Huang, S. J. Wang, Y. J. Chang, and T. Chen. "A hierarchical
Bayesian generation framework for vacant parking space detection,"
IEEE Transactions on Circuits and Systems for Video Technology, vol.
20, no. 12, pp.1770-1785, Dec. 2010.
[13] C. C. Lien, Y. T. Tsai, M. H. Tsai, and L. G. Jang, "Vehicle counting
without background modeling," International Conference on Advances
in Multimedia Modeling, vol.Part I, Taipei, Taiwan, Jan. 2011.
[14] J. W. Hsieh, C. H. Chuang, S. Y. Chen, C. C. Chen, and K. C. Fan,
"Segmentation of human body parts using deformable triangulation,"
IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems
and Humans, vol.40, no.3, pp.596-610, May 2010.
[15] C. T. Lin, Linda Siana, Y. W. Shou, and T. K. Shen, "A conditional
entropy-based independent component analysis for applications in
human detection and tracking," EURASIP Journal on Advances in
Signal Processing, vol.2010, Apr. 2010.
[16] R. Khemmar, J. Y. Ertaud, and X. Savatier, "Face detection and
recognition based on fusion of omnidirectional and PTZ vision sensors
and heteregenous database," International Journal of Computer
Applications, vol.61, no. 21, 2013.
[17] N. Haering, P. L. Venetianer, and A. Lipton, "The evolution of video
surveillance: an overview," Machine Vision and Applications, vol.19,
no.5-6, pp.279-290, Sep. 2008.
[18] J. Ferenbok and A. Clement, (2011). Hidden changes: from CCTV to
‘smart’ video surveillance. In A. Doyle, R. Lipert, and D. Lyon (Eds.),
Eyes Everywhere: The Global Growth of Camera Surveillance (pp.
218-234). New York: Routledge.
[19] P. M. Roth, V. Settgast, P. Widhalm, M. Lancelle, J. Birchbauer, N.
Brandle, S. Havemann, and H. Bischof, "Next-generation 3D
visualization for visual surveillance," IEEE Conference on Advanced
Video and Signal Based Surveillance, Sabta Fe, USA, Sep. 2011,
pp.343-348.
[20] A. Girgensohn, D. Kimber, J. Vaughan, T. Yang, F. Shipman, T. Turner,
E. Rieffel, L. Wilcox, F. Chen, and T. Dunnigan, "DOTS: support for
effective video surveillance," International Conference on Multimedia,
Augsburg, Germany, Sep. 2007, pp.423-432.
[21] N. Martinel, C. Micheloni, C. Piciarelli, and G. L. Foresti,"Camera
selection for adaptive human-computer interface," IEEE Tran. on
Systems, Man, and Cybernetics: Systems, vol.44, no.5, May 2014.
[22] Y. L. Tian, L. Brown, A. Hampapur, M. Lu, A. Senior, and C. F. Shu,
"IBM smart surveillance system (S3): event based video surveillance
system with an open and extensible framework," Machine Vision and
Applications, vol.19 no.5-6, p.315-327, Sep. 2008.
[23] D. Kieran, J. Weir, and W. Q. Yan, "A framework for an event driven
video surveillance system," Journal of Multimedia, vol.6, no.1, Feb.
2011.
[24] G. C. Chao, Y. P. Tsai, and S. K. Jeng, "Augmented keyframe," J. Vis.
Commun. Image R., vol. 21, pp. 682-692, 2010.
[25] A. M. Ferman, A. M. Tekalp, and R. Mehrotra, "Robust color histogram
descriptors for video segment retrieval and identification," IEEE Trans.
on Image Proc., vol. 11, no. 5, pp. 497-508, May 2002.
View publication stats
12
[26] H. C. Lee and S. D. Kim, "Rate-driven key frame selection using
temporal variation of visual content," Electron. Lett., vol. 38, no. 5, pp.
217-218, February 2002.
[27] K. Sze, K. Lam, and G. Qiu, "A new key frame representation for video
segment retrieval," IEEE Trans. on Circuits Syst. Video Technol., vol.
15, no. 9, pp. 1148-1155, 2005.
[28] B. Erol and F. Kossentini, "Automatic key video object plane selection
using the shape information in the MPEG-4 compressed domain," IEEE
Trans. on Multimedia, vol. 2, pp. 129-138, June 2000.
[29] C. Kim and J. N. Hwang, "Object-based video abstraction for video
surveillance systems," IEEE Trans. on Circuits and Systems for Video
Technology, vol. 12, pp. 1128-1138, 2002.
[30] E. Spyrou and Y. Avrithis, "A region thesaurus approach for high-level
concept detection in the natural disaster domain," Lecture Notes in
Computer Science, LNCS 4816, pp. 74-77, 2007.
[31] Z. Ji, Y. Su, R. Qian, and J. Ma, "Surveillance video summarization
based on moving object detection and trajectory extraction," Int. Conf.
on Signal Processing Systems, Yantai, China, pp. 250-253, July 2010.
[32] J. Yoder, H. Medeiros, J. Park, and A. Kak, "Cluster-based distributed
face tracking in camera networks," IEEE Trans. on Image Proc., vol. 19,
no. 10, pp. 2551-2563, October 2010.
[33] T. Mat and N. Ukita, "Real-time multitarget tracking by a cooperative
distributed vision system," Proc. IEEE, vol. 90, no. 7, pp. 1136-1150,
July 2002.
[34] H. Medeiros, J. Park, and A. C. Kak, "Distributed object tracking using
a cluster-based Kalman filter in wireless camera networks," IEEE
Journal of Selected Topics in Signal Proc., vol. 2, no. 4, pp. 448-463,
August 2008.
[35] I. F. Akyildiz, T. Melodia, and K. R. Chowdhury, "A survey on wireless
multimedia sensor networks," Computer Network, vol. 51, pp. 921-960,
2007.
[36] N. Martinel, C. Micheloni, C. Piciarelli, and G. L. Foresti, "Camera
Selection for Adaptive Human-Computer Interface," IEEE
Transactions on Systems, Man, and Cybernetics: Systems, vol. 44, no.
5, pp. 653-664, 2014.
[37] J. Kim and D. Kim, "Probabilistic camera hand-off for visual
surveillance," in Proc. Int. Conf. on Digital Smart Cameras, Stanford,
USA, pp. 1-8, September 2008.
[38] R. Goshorn, J. Goshorn, D. Goshorn, and H. Aghajan. "Architecture for
cluster-based automated surveillance network for detecting and
tracking multiple persons," in Proc.ACM/IEEE Int. Conf. on
Distributed Smart Cameras, Vienna, Austria, pp. 219-226, September
2007.
[39] A. Bakhtari , M. D. Naish , M. Eskandari , E. A. Cloft, and B. Benhabib
"Active-vision-based multisensor surveillance—An implementation",
IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 36, no. 5, pp. 668
-680, 2006.
[40] Y. Fu, Y. Guo, Y. Zhu, F. Liu, C. Song, and Z. H. Zhou, “Multi-view
video summarization.” IEEE Transactions on Multimedia, vol. 12, no.
7, pp. 717-729, 2010.
[41] Vision-based Intelligent Environment project (accessed Oct. 2015).
[Online]. Available: http://cvrc.nctu.edu.tw/~TT/home.php?&lang=en.
[42] ICDSC Challenge - Smart Homes Data set. (accessed 2009) [Online].
Available: http://wsnl2.stanford.edu/icdsc09challenge/.
[43] C. M. Huang and L. C. Fu, “Multitarget Visual Tracking Based
Effective Surveillance With Cooperation of Multiple Active Cameras,”
IEEE Trans. Syst., Man, Cybern. B, vol. 41, no. 1, pp. 234-247, 2011.