Appl Intell
DOI 10.1007/s10489-017-1033-x
Multi-label semantic concept detection in videos
using fusion of asymmetrically trained deep convolutional
neural networks and foreground driven concept
co-occurrence matrix
Nitin J. Janwe1 · Kishor K. Bhoyar1
© Springer Science+Business Media, LLC 2017
Abstract Describing visual contents in videos by seman-
tic concepts is an effective and realistic approach that can
be used in video applications such as annotation, index-
ing, retrieval and ranking. In these applications, video data
needs to be labelled with some known set of labels or
concepts. Assigning semantic concepts manually is not fea-
sible due to the large volume of ever-growing video data.
Hence, automatic semantic concept detection of videos is
a hot research area. Recently Deep Convolutional Neu-
ral Networks (CNNs) used in computer vision tasks are
showing remarkable performance. In this paper, we present
a novel approach for automatic semantic video concept
detection using deep CNN and foreground driven con-
cept co-occurrence matrix (FDCCM) which keeps fore-
ground to background concept co-occurrence values, built
by exploiting concept co-occurrence relationship in pre-
labelled TRECVID video dataset and from a collection
of random images extracted from Google Images. To deal
with the dataset imbalance problem, we have extended this
approach by making a fusion of two asymmetrically trained
deep CNNs and used FDCCM to further improve con-
cept detection. The performance of the proposed approach
is compared with state-of-the-art approaches for the video
concept detection over the widely used TRECVID data set
and is found to be superior to existing approaches.
 Nitin J. Janwe
nitinj janwe@yahoo.com
Kishor K. Bhoyar
kkbhoyar@yahoo.com
1 Department of Information Technology, YCCE, Nagpur, India
Keywords Semantic video concept detection · Foreground
driven concept co-occurrence matrix · Convolutional neural
network · Deep learning · Multi-label classification ·
Asymmetric training
Abbreviations
CCM concept co-occurrence matrix
FDCCM foreground driven concept co-occurrence
matrix
CNN convolutional neural network
1 Introduction
Due to the advancement of networking and multimedia
technologies, use of multimedia data increased exponen-
tially, and video indexing and retrieval is seen as one of
the most challenging issues in the area. Users always desire
to retrieve videos on the basis of some semantic objects
like car, road, airplane etc. Therefore, it is more realistic
and effective for the end user to express query in terms
of semantic objects or concepts. Such queries are mean-
ingless unless the video data has been labelled with some
known set of labels or concepts. It is not feasible to man-
ually label semantic concepts considering the huge volume
of video collections. Therefore, detecting semantic concepts
automatically for video samples is a promising research area
and is a key step in many video based applications such as
annotation, indexing and retrieval. There is a growing need
in automatically detecting concepts from low-level visual
properties by learning the correspondence from loosely
labeled data. Semantic gap [15] is the most challenging
problem in concept based video retrieval, which is a gap

between the low-level representation of video and higher
level semantic concepts which a user associates.
In implementing a system for semantic video concept
detection, the problems reported in the literature are: 1)
Most current methods based on statistical learning tech-
niques [16, 17], use low-level features and struggle in
bridging the semantic gap. While the methods based on
deep convolutional neural network did not exploit the fusion
and the concept of asymmetric training of CNN classi-
fiers for dealing imbalanced dataset problem and to improve
concept detection. 2) Many methods used context relation-
ship of concepts for improving performance but did not
focus on improving performance exploiting the nature of the
concept.
There are two main contributions of this paper. Firstly,
we have used a fusion of deep Convolutional Neural
Networks (CNN) for building a classifier and used an
idea of asymmetric training to deal with dataset imbal-
ance problem and to improve concept detection per-
formance. Secondly, we have built a novel foreground
driven concept co-occurrence matrix (FDCCM) by exploit-
ing the foreground nature of the concepts to predict
background concepts. It is built using local visual co-
occurrence measure from local training dataset and global
visual co-occurrence measure from a collection of random
images retrieved from Google Images. FDCCM is used
to refine concept prediction scores to improve concept
detection.
The reminder of the paper is organized as follows:
Section 2 summarizes the related work. Section 3 describes
various steps of semantic video concept detection. Section 4
presents the details of proposed method. Section 5 gives
experimental results and performance evaluation and lastly,
Section 6 concludes the paper.
2 Related work
A considerable amount of research has been reported in the
literature to address video processing applications such as
semantic video concept detection and video retrieval. This
section provides a brief review of the related work in the
area of semantic video concept detection.
2.1 Temporal-events and object actions based models
In order to detect temporal events and object actions, a
group of methods have been proposed to count temporal
information from video sequences. These approaches can
be roughly classified into those classifying global video
sequences into generic semantic categories (such as “wed-
ding”, “rally”) and other recognizing object-level action
(e.g. “people walking”, “train running”).
N. J. Janwe and K. K. Bhoyar
2.1.1 Object-level action recognition
Robust action detection and recognition of object-level
actions is very useful for video content description, since it
provides unique object-level dynamic information. Bobick
and Davis [18, 19] derived the temporal template repre-
sentation from background subtracted images. A variety of
choreographed actions across different subjects and view
can be handled. Zelnik and Irani [20] use marginal his-
tograms of spatio-temporal gradients at several temporal
scales to cluster and recognize video events.
2.1.2 Generic concept detection by classifying video
sequences
To recognize generic concepts in general domains, such
as broadcast news videos, consumer videos etc., is a more
challenging problem. In such cases, there are often fast mov-
ing small objects, large camera motion, articulate motions,
significant clutter and object occlusion, and the event of
interest may involve high-level semantics. Dong and Chang
[21] use kernel-based discriminative classification to detect
a large variety of generic events in news videos. Zhou et
al. [22] propose a SIFT-Bag based framework where each
video clip is encoded as a bag of SIFT feature vectors
[43], and the distribution of each video clip is modeled by
Gaussian Mixture Models (GMMs).
2.2 Static key-frame based
In the second approach, video streams are segmented into
shots and key-frames of a shot are identified which best rep-
resent it. Statistical learning models such as support vector
machine and neural network are used for training classi-
fiers. Low-level features are extracted from key-frames and
classifiers are trained. There are three approaches for train-
ing, 1) Supervised training: where the concepts are fixed
or known 2) Unsupervised training: if concepts are not
known and 3) Semi supervised training: if some concepts
are known. Zha et al. [35] presents video concept detec-
tion system using support vector machine as a supervised
machine learning tool. Janwe and Bhoyar [7] implemented
video concept detection using supervised neural network
model. Memar and Affendey [13] proposed an unsupervised
concept-based video retrieval system based on the integra-
tion of knowledge-based and corpus-based semantic word
similarity measures, in order to retrieve video shots for
concepts whose annotations are not available for the system.
2.2.1 Low-level feature-based classifier
Videos comprise of information from both visual and
audio modalities. Each modality brings some information
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...
complementary with the other and their joint processing can
help uncover relationships that are otherwise unavailable.
Besides the use of some conventional low-level visual fea-
tures from color, texture, shape and edge categories, some
advanced features such as scale invariant feature transform
SIFT and bag of feature BOF [43] can also be used in video
concept detection. All of the methods discussed above are
low-level features based.
2.2.2 Deep feature-based classifier
In recent years, because of tremendous increases in com-
puting power, more powerful statistical learning models
like deep convolutional neural networks have come up.
These models have drastically improved the robustness of
computer vision systems. CNNs have shown better scal-
ing properties over conventional machine learning methods
such as SVM, principal component analysis and linear
discriminant analysis. The study states that, the generic
descriptors extracted from the deep CNNs have proven to
be very powerful [11, 12, 14]. CNN has replaced tradi-
tional image representations using well designed features
and yield deep hierarchies of features. Recently, researchers
are moving more and more towards deep CNNs [23, 24]
as (1) they support large volumes of training data and (2)
due to the availability of high end computing devices such
as large array of CPU cores [25] and GPUs. Krizhevsky
et al. [23] showed that excellent recognition accuracy can
be achieved training a large CNN [24] on a large dataset
by standard back-propagation [26] algorithm. Chien-Hao et
al. [2] implemented high-level image features learning by
deep convolutional neural network in image retrieval sys-
tem. Similarly, Podlesnaya and Podlesnyy [3] built video
indexing and retrieval system based on features extracted
by convolutional neural network. McCormac et al. [4] dis-
cussed the use of CNN to implement intuitive user inter-
action by combining CNN and a state of the art dense
Simultaneous Localization and Mapping (SLAM) system.
2.2.3 Combining low-level and deep features
Kikuchi et al. [5] presented a new feature extraction method
where they used a combination of low-level features (e.g.,
SIFT and HOG) and convolutional neural network (CNN)
derived features to represent the meaningful objects that
contributes to the determination of semantic concepts to
achieve accurate and robust semantic indexing of videos.
In the literature of semantic video concept detection,
indexing and retrieval systems, TRECVID [47] contributes
a lot in the ongoing research in the field. Awad et al. [6]
presents a detailed review of the work presented and run at
TRECVID for semantic indexing task from 2010 to 2015 for
content based access to video documents and collections.
2.3 Multi-label based models
The approaches in the literature on video concept detection are
categorized into single-label and multi-label methods. In the
first type, a sample in the dataset is assigned a label (concept)
whereas in multi-label or multi-concept methods, multiple
concepts can be associated with a data sample. In multi-label
methods, a classifier is trained using either a binary classi-
fication approach or multi-class classification. Wang et al.
[34], discussed a transductive multi-label learning approach
for video concept detection. Li [10] discusses implementing
multi-label image classification with a probabilistic label
enhancement model by constructing auxiliary labels.
2.4 Visual co-occurrence based models
Visual co-occurrence assists in detecting concepts unlike
other conceptual and perceptual models [27] such as the
WordNet distance [28]. It is shown [29] that concept co-
occurrence strengthens the appearance of concepts in video.
The co-occurrence models in the literature are mainly
image based. The approaches used for concept detection
in these models have gained an increasing popularity [30–
32]. Feng and Bhanu [33] measured the relatedness of
semantic and visual concepts in the image and built con-
cept co-occurrence patterns. In [28, 29], pairwise concept
co-occurrence has been integrated into the concept catego-
rization framework by using a co-occurrence matrix. Modiri
et al. [9] proposed a contextual approach to complex video
classification based on generalized maximum clique prob-
lem which uses the co-occurrence of concepts as the context
model. In [1], Feng et al. used the concept co-occurrence
patterns and concept signatures for image annotation and
retrieval application. These approaches have several advan-
tages over standard concept inference techniques, for exam-
ple, incorporating semantic context compensates the ambi-
guity of concept visual appearance. However, the matrix of
the co-occurrence has an inevitable pairwise constraint on
the relationship.
Several recent works explore multi-concept learn-
ing/detection techniques for automated image annotation
that aim to model the co-occurrence information among
concepts/annotations. In very few works, a space has been
given on the possible coding of the concept tags or concepts
in concept detection methods. Chen et al. [40] proposed a
method about the coding of tags which are based on binary
numbers and transformed by Chinese characters.
3 Semantic video concept detection
The goal of semantic video concept detection is to detect
semantic concepts of a video segment based on its visual

appearance. The pipeline of a complete process of seman-
tic video concept detection system is shown in Fig. 1 and
consists of four steps, namely: (1) shot boundary detection
or shot segmentation, (2) key-frame extraction, (1) classifier
training, and 4) concept detection.
The input to the system is a video stream. The details of
the above four steps are described below.
3.1 Shot segmentation
In order to detect semantic concepts precisely from a video,
it needs to be segmented into video shots. Shot exhibits
strong content correlations between frames; hence shots
are considered to be the basic units in concept detection.
Generally shot boundaries are of two types, a cut, where
the transition between two consecutive shots is abrupt, and
gradual transitions, where the boundary is stretched over
multiple frames. The examples are dissolve, fade-in and
fade-out etc. The shot boundary detection methods usually
extract visual features from each frame and the similarities are
measured and detect shot boundaries between frames that
are dissimilar.
3.2 Key-frame extraction
In video processing applications, the shots are often repre-
sented by a frame, called a key-frame, which is supposed
to be a representative frame for a shot. There are great sim-
ilarities among the frames from the same shot; therefore
certain frames that best reflect the shot contents are chosen
as key-frames. Mostly, the middle frame of a shot is taken
as a key-frame, assuming that middle segment contains key
contents, but many other techniques do exist. It is not nec-
essary that a shot is always represented by a single frame; in
some cases of more visual complexity multiple key-frames
are required. For such cases, approaches like unsupervised
clustering can be used, where frames in a shot are clustered
depending on the variation in shot content and then choose
the frame closest to the cluster center as a key-frame. Each
cluster is represented by a unique key-frame. So a single
shot can have multiple key-frames. The choice of a key-
frame may also depend on the object or the event, one is
interested in. The frame that best represents the object or the
event can be chosen as a key-frame.
3.3 Training a CNN classifier
Automatic multi-label semantic concept detection for
videos is a supervised machine learning problem, where
Fig. 1 Pipeline of a semantic
video concept detection system
N. J. Janwe and K. K. Bhoyar
we provide a set of extracted key-frames of video shots
as input and the corresponding concepts as output. The
multi-label video concept detection task is posed into binary
classification problem. CNN is used as a baseline.
3.3.1 Overview of CNN
A neural network is a function g mapping data x, for exam-
ple an image, to an output vector y, for example an image
label. The function is the composition of
the sequence of simpler functions fl which are called com-
putational blocks or layers. Let x1 , x2 , . . . .xL be the output
of each layer in the network. And let x0 = x denote the
network input. Each intermediate output xl = fl (xl−1 ; wl )
is computed from the previous output xl−1 by applying the
function fl with parameters wl
In CNN, the data has a spatial structure: each xl RHl Wl Cl
is a 3D array or tensor where the first two dimensions
Hl (height) and Wl (width) are interpreted as spatial dimen-
sions. The third dimension Cl is interpreted as the number
of feature channels. Hence, the tensor xl represents Hl × Wl
field of Cl dimensional feature vectors, one for each spa-
tial location. A fourth dimension Nl in the tensor spans
multiple data samples packed in a single batch for effi-
ciency parallel processing. The number of data samples
Nl in a batch is called the batch cardinality. The network
is called convolutional because the functions fl are local
and translation invariant operators (i.e. non-linear filters)
like linear convolution. CNNs are often used as classi-
fiers or regressors. The output of CNN is ŷ = f (x) is
a vector of probabilities, one for each of an N possi-
ble image labels (dog, cat, trilobite, etc.). If y is the true
label of image x, we can measure the CNN performance
by a loss function ly ŷ ∈ R which assigns a penalty
to classification errors. The CNN parameters can then be
tuned or learned to minimize this loss averaged over a
large dataset of labelled example images. Learning a CNN
requires computing the derivative of the loss with respect
to the network parameters. Derivatives are computed using
an algorithm called back-propagation. CNNs are inherently
translation invariant. Their basic components (convolution,
pooling, and activation functions) operate on local input
regions, and depend only on relative spatial coordinates.
The details of the most important CNN operations is as
follows:
(1) Convolution: The convolutional block is implemented
by the function y =convolution(x, f, b), computes the
convolution of the input map x with a bank of K
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...

multi-dimensional filters f and biases b. Here, x ∈ channels. Note that input x and output y have the
     
RH ×W ×D , f ∈ RH ×W ×D×D , y ∈ RH ×W ×D . same dimensions.
The output of convolving a signal for 1D slice, is • Spatial Normalization: The spatial normalization
given by following (1), operator acts on different feature channels inde-
 
pendently and rescales each input feature by the

H 
W 
D
energy of the features in a local neighborhood.
yi  j  d  = bd  + fi  j  d × xi  +i  −1,j  +j  −1,d  ,d  .
First, the energy of the features in a neighborhood
i  =1 j  =1 d  =1
W  × H  is evaluated by (8)
(1)
(2) Padding and Stride: Convolution function allows to spec- 1 
n2i  j  d = x 2
ify top-bottom-left-right paddings (Ph− , Ph+ , Pw− , Pw+ ) W H  i +i  −1− H  −1
,j  +j  −1− W  −1
,d.
1≤i  ≤H  ,1≤j  ≤W  2 2
of the input array and subsampling strides (Sh ,Sw ) of
(8)
the output array:
(6) Softmax: Computes the softmax operator:
 

H 
W 
D
yi  j  d  = bd  + fi  j  d × xSh (i  −1)+i  −P − ,Sw (j  −1)+j  −Pw− ,d  ,d  . exij k
i  =1 j  =1 d  =1
h yij k = D
(9)
xij t
(2) t=1 e
Note that the operator is applied across feature chan-
(3) Spatial Pooling: It is of two types; max and sum nels and in a convolutional manner at all spatial loca-
pooling tions. Softmax can be seen as the combination of an
activation function (exponential) and a normalization
• Max Pooling: Computes the maximum response
operator.
of each feature channel in a H  × W  patch by (3)

yi  j  d = max1≤i  ≤H  ,1≤j  ≤W  Xi  +i  −1,j  +j  −1,d . 3.4 Concept detection

(3)
In CNN learning, given a key-frame Ki of size h × w × c
 
resulting in an output of size y ∈ RH ×W ×D of a video shot or a representative key-frame i, the aim is to
• Sum Pooling: Computes the average of the values obtain a measure, which indicates whether a key-frame Ki
by following (4) (for shot i) belongs to a set of semantic concepts C or not
 where C is a subset of concept vocabulary. In CNN learning,
1
yi  j  d = xi  +i  −1,j  +j  −1,d. in the training phase, it has to be trained by supplying a set
W H 
1≤i  ≤H  ,1≤j  ≤W  of key-frames, and in the test phase, the classifier assigns
(4) a probability p(Cj |yi ) to each semantic concept of concept
vocabulary. Hence, by ranking the probability scores of each
(4) Activation Functions: concept, the algorithm finds the presence of concepts in a
• ReLU: Computes the Rectified Linear Unit. test key-frame.
Once the CNN classifier is trained using the training
yij d = max{0, xij d }. (5) dataset of key-frames, the classifier outputs prediction prob-
•
Sigmoid: Computes sigmoid. ability scores for concepts in the vocabulary. These scores
  1 determine the probability of presence of a respective con-
yij d = σ xij d = (6) cept in a test key-frame. The scores are ranked in descending
1 + e−x ij d
order and the top-k concepts are considered as inferred
(5) Normalization: concepts.
• Local Response Normalization (LRN): This is In CNN, the probability estimation of concepts is done
applied independently at each spatial location and by the output of a last layer i.e. Softmax layer. In the simi-
to groups of feature channel and is computed by lar systems in the literature, Hidden Markov Model (HMM)
(7) for probability estimations [44] or confidence-estimation
⎛ ⎞−β methods [45] can be used for probability estimations. Some
 conventional probability estimator systems built estimators
yij k = xij k ⎝k + α xij2 t ⎠ (7) from a given training dataset and then evaluated on a holdout
t∈G(k)
testing set from the same dataset to display its effective-
where, for each output channel k, G(k) ⊂ ness. An estimator built on a specific dataset usually cannot
{1, 2, ..., D} is a corresponding subset of input be directly applied to make evaluations on other datasets
 N. J. Janwe and K. K. Bhoyar

because these are heterogeneous. Tian et al. [41] developed
a correlation component manifold space learning (CCMSL)
approach to first learn a common feature space by captur-
ing the correlations between the heterogeneous databases,
and then in the resulting space establish a single age esti-
mator across such heterogeneous datasets through correla-
tion representation learning (CRL). As a result, probability
distribution-incompleteness of individual datasets are com-
pensated, but also the discriminating ability of the estimator
is reinforced.
4 Proposed method
The framework of the proposed system is given in Fig. 2.
The figure also highlights the contributions of the paper.
The first contribution is made at the classifier level, where
a fusion of asymmetrically trained CNNs is proposed to
build a classifier to tackle dataset imbalance problem. The
second contribution is the use of foreground driven con-
cept co-occurrence matrix (FDCCM) after the classification
stage, to refine concept prediction scores to further improve
detection performance.
4.1 Fusion of asymmetrically trained deep CNNs
The asymmetric training approach is very useful in datasets
where the number of training samples of each class are con-
siderably different. Such a dataset is known as imbalance
dataset. The rationale behind the approach is the observation
that classes with sufficiently large sample size (strong
classes) tend to learn quickly and give better detection rate
in the early stage of the classifier training than the classes
with smaller sample size (weak classes). Therefore, in order
to achieve better detection rate for weak classes a classi-
fier needs to be trained for many more epochs. But, if the
classifier is trained further, we may lose the detection per-
formance of strong concepts because of overfitting problem.
To tackle this difficulty, we suggest to divide the dataset
classes into two groups by applying the Global Thresh-
olding method given in Algorithm 1. The first group will
contain the concepts from cluster-1 having smaller popu-
lation size and second group will have the concepts from
cluster-2 with larger sample population size. The proposed
method uses two different CNNs for two groups of classes.
These are trained independently with the same dataset so
that the detection rate of the concepts it is trained for will be
the highest. After this asymmetric training, the concept pre-
diction scores of both CNNs are fused to get final scores as
shown in Fig. 3.
This approach can be extended further for more numbers
of groups, depending on the variation in the sample size.
But as we go on increasing the groups, in worst case, we
would have a class (concept) per group and would result
in a dedicated concept detector. Another drawback is the
increased time complexity, as it requires more CNNs to be
used to build a classifier.
In the proposed method, a classifier is suggested to be
built using a fusion of asymmetrically trained deep CNNs.
The CNNs are built using the network shown in Fig. 4.

Concept Classifier
Test key-frames Contribution #1
(Fusing asymmetrically trained CNNs)
dataset Detected concepts
CNN-1 Face
Outdoor
+
Person
CNN-2 Sky
Asymmetrically trained CNNs Vegetation

Ranked
concept
Training prediction
key-frames scores
dataset
Concept Probability Concept Probability
Face 0.523 Face 0.523
Crowd 0.428 Sky 0.476
Key-frame extraction
Person 0.254 Vegetation 0.345
……… …….. ……… ……..

Refined concepts
Contribution #2
Truck Road Sky Studio Snow C1 C2 C3 C4 C5 C6 - C36
Studio Car Tree Bus Sports C1 1 0 0 0.5 0.6 0.9 - 0
Training Video C2 0 1 0.5 0 0 0 - 0.5
Sports Tree Person Road Studio
C3 - - 1 - - - - -
Video segmentation Car Sports Road Bus Tree
- - - - - - - - -
Dataset Road Person Studio Car Sports C36 - - - - - - - 1

Concept set FDCCM

Fig. 2 Flowchart of the proposed semantic video concept detection method using convolutional neural network (CNN) and foreground driven
concept co-occurrence matrix (FDCCM)
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...
4.1.1 Architecture used for convolutional neural network
The architecture of the CNN is comprised of 7 layers. A key-
frame of size 352 × 288 (with three color channels), first
resampled to 88 × 72 and then converted to gray-scale (sin-
gle channel) is presented as input. This is convolved with 40
different first layer square filters, each of size 6 × 6 using a
stride of 2 in both x and y. The resulting feature maps are
then: (i) pooled (max within 2 × 2 regions, using stride 2),
(ii) contrast normalized across feature maps and (iii) passed
through a rectified linear function, to give 40 different 17
× 21 element feature maps. Similar operations are repeated
in layer 2. The last two layers are fully connected, taking
features from the top convolutional layer as input in vector
form. C-way Softmax is the last layer where C being the
number of classes which is 36 in our case.
4.2 Concept co-occurrence matrix (CCM)
This section presents the introduction of CCM and discusses
the detailed algorithm to construct it. Following subsection
explains what FDCCM is and how it is derived from CCM.
Visual co-occurrence assists in providing semantic hints
in detecting concepts. It has been shown that the appear-
ance of each concept is consolidated by their co-occurrence
in a video shot. Existing approaches consider co-occurrence
 N. J. Janwe and K. K. Bhoyar

Fig. 3 A fusion of
asymmetrically trained CNN
classifiers

of pairs of concepts. If “Road” & “Outdoor” is a co-
occurrence pair, then the probability of presence of “Out-
door” can be strengthened by a strong confidence of
“Road”. Since we are dealing with multi-label data in our
approach, we have extended the pairwise co-occurrence
concept to multi-concept co-occurrence. For example, for
a concept “Car”, if the confidence of co-occurrence pairs,
“Car-Road”, “Car-Outdoor”, “Car-Vegetation” and “Car-
Sky” is significant in the training data, then the strong
confidence of concept “Car” in the test sample could con-
solidate the presence of multiple concepts “Road”, “Out-
door”, “Vegetation” and “Sky” by appropriate proportions
depending on co-occurrence values. We need to construct
CCM to keep co-occurrence data.
CCM is a matrix of size nx n, where n are the number of
concepts in the vocabulary set. It keeps the information of
the co-occurrence pairs and their co-occurrence values. The
structure of CCM is shown in Table 1 and is generated using
Algorithm 2. The highlighted row in the CCM, indicates that
the presence of a concept “Airplane” also predicts the prob-
ability of presence of concepts “Bus”, “Face”, “Outdoor”
and “Sports” by 0.5, 0.6, 0.9 and 0.5 respectively. It also
means the presence of concept “Airplane” consolidates the
presence of concepts “Bus” by a factor of 0.5, “Face” by
0.6, “Outdoor” by 0.9 and “Sports” by 0.5. The proba-
bility of these concepts will be strengthened by respective
co-occurrence values.
In our work, CCM is implemented using two data col-
lections. In this section, we discuss the construction of
the CCM using a training video dataset. The video shot
segmentation and key-frame extraction steps, transforms
a training video dataset into key-frame dataset. The key-
frames dataset is pre-annotated. We have used TRECVID
[47] development key-frame dataset supplied by NIST [46]
as a ground-truth for training dataset. We built a concept
co-occurrence matrix from the annotated key-frames of a
training video dataset. The key-frames in the ground-truth
dataset are multi-concept key-frames i.e. a single key-
frames has got multiple concepts. Let  = {c1 , c2 , . . . .., cn }
be the concept vocabulary in the training key-frames dataset,
where n is total number of unique concepts annotated to the
key-frames that the system is attempting to detect. Hence
the size of the concept co-occurrence matrix is n × n. Let
T = {kf1 , kf2 , . . . . . . , kfm ) denote the training key-frame

Convolution-1 Convolution-2 Convolution-3 Convolution-4 Convolution-5 Convolution-6 (Concept Detection)

88 x 72 42 x 34 10 x 8 8x6 7x5 3x3 1 x 36

filter size

6 300 36
60 100 500
40 class
2x2 2x2
Stride 2 2x2 Stride 2 Stride 1 Stride 1 softmax
max pool avg. pool
max pool
21
9 Contrast Stride 1
1 Contrast
Norm. Norm. 6 1
3 2 4
17 2 1
7
2 conv. 1 conv. 1 conv.
1 conv.
100 1 conv.
40 60 300 500
ReLU ReLU ReLU

Layer-1 Layer-2 Layer-3 Layer-4 Layer-5 Layer-6 Softmax Layer

Fig. 4 The architecture of CNN used in proposed method

Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...
dataset with size m. The CCM is constructed by count-
ing the total number of co-occurrences of each concept
ci for each individual concept cj for the m key-frames
in the training dataset. There are n rows in a matrix. A
row r1 represents the concept c1 and consists of total n
The CCM used is a combination of two CCMs computed
at two levels, namely, local visual co-occurrence level and
global visual co-occurrence level. CCM is evaluated at local
visual co-occurrence level from the local pre-labelled train-
ing dataset and at global visual co-occurrence level from a
collection of random images retrieved from Google Images,
and are maintained in two separate CCMs. The final CCM is
numbers of values each gives the co-occurrence count for
the concepts c1 to cn . Table 2 shows the concepts and their
range for the training dataset used to construct CCM. Figure 5
shows the CCM constructed from TRECVID dataset using
Algorithm 2.
obtained by combining two matrices by taking their average
as given by (10).
CCM Z [I, J ] = Avg (CCM X [I, J ] , CCM Y [I, J ]) (10)
Where CCM X and CCM Y are constructed at local level
and global level respectively and CCM Z is final resultant
 N. J. Janwe and K. K. Bhoyar

Table 1 CCM along with co-occurrence values

Airplane Animal Car Bus Face Outdoor Sports – – C36

Airplane 1 0 0 0.5 0.6 0.9 0.5 0

Animal 0 1 0.5 0 0 0 0 0.5
Car 0.12 0.07 1 0.4 0.65 1 0.23 0.19
Bus 0.07 0.01 0.2 1 0.3 0.4 1 0.32
– – – – – – – – 1 – –
C36 1

co-occurrence matrix. I and J are rows and columns of the
matrix.
4.2.1 Foreground driven concept co-occurrence matrix
(FDCCM)
The CCM we prepared gives the co-occurrence values of
every pair of concepts in concept vocabulary and is used to
improve concept detection. If we observe a list of concepts,
we find that it consists mainly of two types, 1) Foreground
or active concepts: the concepts or objects in a key-frame of
foreground nature (e.g. Airplane, Car or Bus) and 2) Back-
ground or passive concepts: these concepts are always a
part of background scenery (e.g. Sky and Vegetation). Here,
we propose a special type of CCM which considers above
division of concepts, we can divide co-occurrence pairs
into four types 1) Foreground-Background concept pair 2)
Background-Background pair 3) Background-Foreground
pair and 4) Foreground-Foreground pair. We do observe that
it is always possible to predict background concept if fore-
ground concept is available, e.g. if an image consists of
concept Airplane, then we can easily predict the concept
Sky in the background, whereas the Airplane concept can-
not always be predicted with the Sky in the background.
Hence, out of two co-occurrence pairs Airplane-Sky and
Sky-Airplane, first pair is meaningful and is of type Fore-
ground driven.
Inspired by this observation, we have proposed a novel
CCM, named Foreground driven Concept Co-Occurrence
Matrix (FDCCM) which consists of rows of type foreground
only. FDCCM is used for refining concept probabilities
only when foreground concepts are found in a list of top
ranked scores. FDCCM has the advantage that the negative
impact of predicting foreground concepts on the basis of
background concept is reduced.
To generate FDCCM, we have identified twelve concepts
out of 36 of foreground nature from the concept vocabulary.
Table 3 shows a list of identified foreground concepts.

Table 2 The training dataset with a range of start.to end key-frame no. for each concept

SN Concept From To SN Concept From To

1 Airplane 1 24 19 Natural-Disaster 6553 6588

2 Animal 25 714 20 Office 6589 7278
3 Boat Ship 715 967 21 Outdoor 7279 8816
4 Building 968 1828 22 People-Marching 8817 9053
5 Bus 1829 1863 23 Person 9054 10487
6 car 1864 2284 24 Police Security 10488 10609
7 Charts 2285 2346 25 Prisoner 10610 10624
8 Computer TV-screen 2347 2688 26 Road 10625 11284
9 Court 2689 2811 27 Sky 11285 12592
10 Crowd 2812 3984 28 Snow 12593 12709
11 Desert 3985 4045 29 Sports 12710 12955
12 Explosion Fire 4046 4083 30 Studio 12956 13086
13 Face 4084 5454 31 Truck 13087 13187
14 Flag-US 5455 5461 32 Urban 13188 14064
15 Maps 5462 5567 33 Vegetation 14065 15366
16 Meeting 5568 6090 34 Walking Running 15367 16422
17 Military 6091 6438 35 Waterscape Waterfront 16423 17086
18 Mountain 6439 6552 36 Weather 17087 17114
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...

Fig. 5 The CCM generated from TRECVID dataset using Algorithm 2

presents FDCCM created with above approach. An alternate

With above approach, CCM is converted to FDCCM clustering method of self-adaptive approach based on global
by considering only concepts given in Table 3. Figure 6 best [42] can also be used to cluster concepts in foreground
and background categories.

4.3 Refining concept prediction scores

Table 3 A list of foreground concepts
The procedure of refining concept prediction scores is given
Sr. No. Concept No. Concept Name
in Fig. 7. It shows that in testing phase the classifier outputs
1 1 Airplane concept prediction scores for all the concepts in concept
2 2 Animal vocabulary. These scores are then rearranged in descending
3 3 Boat Ship order. Top value in a ranked list indicates highest prob-
4 5 Bus ability of presence of a concept and so on. Initially, we
5 6 Car considered top n scores from a ranked list where n is
6 14 Flag-US the number of concepts in the ground-truth for a given
7 22 People-Marching key-frame. Our hypothesis is that the n concepts of the
8 23 Person ground-truth data will lie in the top n scores of a ranked
9 24 Police Security list.
10 25 Prisoner To improve concept detection rate, we make use
11 31 Truck of co-occurrence information; i.e. concept co-occurrence
12 34 Walking Running matrix to refine concept prediction scores. In the refining
 N. J. Janwe and K. K. Bhoyar

Fig. 6 The FDCCM derived from CCM using above approach

Fig. 7 The procedure of refining concept prediction scores using FDCCM (keeping n = 6)
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...

Table 4 Partition details of TRECVID development dataset 5 Experimental results

Dataset Partitions # of videos # of key-frames
In this section, the experimental setup i.e. the training
TRECVid Dev. Training dataset 90 17114 and testing dataset required to implement proposed method
Dataset Testing dataset 20 9352 and the performance measures to evaluate the results are
explained in detail.

5.1 Video datasets

procedure, a top score concept is picked-up from a ranked
list of concepts and using FDCCM, concept prediction
scores are updated according to the (11). This process is
repeated for all the top k concepts of the list and the updated
(refined) values are again re-ranked. This re-ranked list of
scores of concepts are the final detected concepts.
In our experimentation, the dataset used for training and
testing the CNN classifier is TRECVID. Once CNN is
trained, its performance can be tested using testing dataset.
The performance measures used are Average Precision (AP)
and Mean Average Precision (MAP). The details of the
dataset and performance measures are as follows:

4.4 Computing refined concept prediction scores 5.1.1 TRECVID datasets and ground-truth data

The problem of refining concept prediction scores obtained
in the output of CNN classifier fits as a problem of finding
a union of fuzzy sets using an equation for algebraic sum
for S-Norm. CCM is considered as a first fuzzy set of pos-
sibility distribution for concept co-occurrence and the CNN
output, a set of concept prediction scores to be refined is
considered as the second fuzzy set. The refined scores are
obtained by computing a fuzzy set union by an equation of
algebraic sum for S-norm given by (11),
The National Institute of Standards and Technology (NIST)
[46] is responsible for the annual Text Retrieval Confer-
ence (TREC) Video Retrieval Evaluation (TRECVID) [47]
since 2001. Every year, it provides a test collection of
video datasets along with a task list. It focuses its efforts
to promote progress in video analysis and retrieval. It
also provides ground-truth for genuine researchers. Many
researchers and research teams participate and contribute in
yearly organized conferences and workshops.

5.1.2 Dataset selection

Scoreref ined = U (Scoreold ,Wi )
= Scoreold +Wi − (Scoreold ×Wi )
Where Scoreref ined is the refined score, Scoreold is the
non-refined concept prediction score and Wi is the co-
occurrence value in the FDCCM.
We have used TRECVID 2007 development dataset in
(11)
our experimentation. It is composed of 110 video clips.
The videos are partitioned into 19,140 shots and 6,64,850
key-frames. As shown in Table 4, the Dev. dataset is par-
titioned into two parts, Training and Test dataset. Training
dataset consists of 17,114 randomly chosen positive key-
frames from first 90 videos to perform classifier training

Table 5 Concept list in the dataset

Id Concept Id Concept Id Concept Id Concept

1 Airplane 10 Crowd 19 Natural-Disaster 28 Snow

2 Animal 11 Desert 20 Office 29 Sports
3 Boat ship 12 Explosion Fire 21 Outdoor 30 Studio
4 Building 13 Face 22 People-Marching 31 Truck
5 Bus 14 Flag-US 23 Person 32 Urban
6 Car 15 Maps 24 Police-Security 33 Vegetation
7 Charts 16 Meeting 25 Prisoner 34 Walking-Running
8 Computer TV-screen 17 Military 26 Road 35 Waterscape-Waterfront
9 Court 18 Mountain 27 Sky 36 Weather

and Test dataset consists of 9,352 randomly chosen pos-
itive key-frames from next 20 videos and is used to test
classifier performance. There are 36 defined concepts in
the dataset. The concept list is given in Table 5. NIST
provides ground-truth data for the above dataset which has
manually annotated 36 concepts over these key-frames. The
ground-truth consists of video shots and their representative
key-frame/s. It is noted that a shot in a video clip may have
one or more positive and/or negative key-frames. For a con-
cept, positive key-frame is defined as a frame containing a
said concept as a visual content.
The ground-truth dataset consists of both positive as well
as negative examples. For the experimentation, the ground-
truth data for the development dataset is used. Figure 8
illustrates the distribution of positive examples for each
individual 36 concepts in the Training dataset and Table 6
presents concept defining key-frames from the TRECVID
training dataset.
5.2 Evaluation measures
The performance evaluation of the proposed concept detec-
tion method is done by the measure Mean Average Precision
(MAP) of Top-d detected concepts from the concept vocab-
ulary list.
5.2.1 Computing Top-D MAP
The ground-truth key-frames consists of multi-label data.
We preprocess the ground-truth test dataset as follows: the
concept set (label set), Yi, and concept count or label den-
sity, Ni , for each test sample, xi , are computed. Let D be
a multi-label test dataset, consisting of |D| multi-label test
examples (xi , Yi ), i = 1 . . . |D|, Yi ⊆ L, where L is a con-
cept vocabulary set for a dataset. When concept prediction
scores for all the L concepts are obtained, the following
procedure is adopted to compute Top-d MAP:
1. Rank the prediction scores and concepts in descending
order for a test sample xi .
Fig. 8 Number of positive frames in TRECVID training dataset
N. J. Janwe and K. K. Bhoyar
2. Pick-up top di scores from a ranked list, let Pi is
concept list from top d concepts. The intersection Mi
between Yi & Pi are the detected concepts.
Let Ni is the label density for xi , and intersection
between Yi and Pi is Mi , then Mi are correctly predicted
concepts by a classifier.
The average precision (AP) for a test sample is computed
by (12),
|Yi Pi | Mi
APi = = (12)
|Pi | Ni
And the Top-d MAP for a classifier, H , on dataset D, is
obtained by computing the mean of APs by (13),
1 |D| |Yi Pi |
MAP (H, D) = (13)
|D| i=1 |Pi |
5.3 MatConvNet
The experimentation on CNN is performed in Matlab16a
and using open source library: MatConvNet [8] version 1.0
beta 20. MatConvNet is a novel framework for experiment-
ing with deep CNN in Matlab. It allows fast prototyping
of new CNN architectures; at the same time, it supports
efficient computation on CPUs and GPUs allowing to train
complex models on large datasets. It is widely used for CNN
implementation.
5.4 Experimental setup & result analysis
The proposed method has two stages. In the first stage,
CNN-based concept detector is implemented and in the sec-
ond stage, concept prediction scores are refined using Fore-
ground driven Concept Co-occurrence Matrix for exploiting
context relationship.
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...

Table 6 Concept defining key-frames from the TRECVID development dataset

Concept Concept Definition Examples

Airplane Segment contains a shot of an airplane

Boat Ship Segment contains a shot of a boat or ship

Building Segment contains a shot of an exterior of a building

Car Segment contains a shot of a car

Crowd Segment contains a shot depicting a crowd

Face Segment contains a shot depicting a face

Road Segment contains a shot depicting a road

Sports Segment contains a shot depicting any sport in action

Snow Segment contains a shot depicting snow

Walking Running Segment contains a shot depicting a person walking or running

 N. J. Janwe and K. K. Bhoyar

Table 7 Number of positive samples per concept in the training dataset. The Group-1 and Group-2 concepts are marked by green and orange
color respectively

Id Concept # Id Concept # Id Concept #

1 Airplane 24 13 Face 1371 25 Prisoner 15
2 Animal 690 14 Flag-US 7 26 Road 660
3 Boat ship 253 15 Maps 106 27 Sky 1308
4 Building 861 16 Meeting 523 28 Snow 117
5 Bus 35 17 Military 348 29 Sports 246
6 Car 421 18 Mountain 114 30 Studio 131
7 Charts 62 19 Natural-Disaster 36 31 Truck 101
8 Computer TV-screen 342 20 Office 690 32 Urban 877
9 Court 123 21 Outdoor 1538 33 Vegetation 1302
10 Crowd 1173 22 People-Marching 237 34 Walking-Running 1056
11 Desert 61 23 Person 1434 35 Waterscape-Waterfront 664
12 Explosion Fire 38 24 Police-Security 122 36 Weather 28

Table 8 MAP with Top-d performance measures for random three CNN2 classifier models trained with different epochs

Methods # of epochs for CNN1 = 4 # of epochs for CNN1 = 4 Classifier for proposed method
# of epochs for CNN2 = 36 # of epochs for CNN2 = 76 # of epochs for CNN1 = 4
# of epochs for CNN2 = 187

MAP MAP MAP

Top-4 Top-5 Top-7 Top-n Top-4 Top-5 Top-7 Top-n Top-4 Top-5 Top-7 Top-n

Concept detection using individual CNNs

Scnn4 0.48 0.43 0.36 0.52 0.48 0.43 0.36 0.52 0.48 0.43 0.36 0.52
Scnn187 0.40 0.37 0.33 0.45 0.36 0.33 0.30 0.39 0.31 0.30 0.27 0.32
Concept detection using fusion of CNNs
Fcnn 0.48 0.42 0.36 0.56 0.47 0.42 0.36 0.55 0.51 0.45 0.37 0.55
Concept detection using individual CNN and CCM
Sccm 0.47 0.43 0.38 0.47 0.44 0.41 0.36 0.43 0.43 0.40 0.35 0.43
Concept detection using fusion of CNNs and CCM
Fccm 0.48 0.44 0.37 0.47 0.47 0.43 0.37 0.48 0.45 0.42 0.36 0.49
Concept detection using proposed method i.e. using fusion of CNNs and FDCCM
Proposed FFDCCM 0.49 0.44 0.37 0.57 0.50 0.44 0.37 0.61 0.50 0.45 0.38 0.65

Values in bold indicate the results of the proposed method for the classifier used

0.7

0.6

0.5

0.7 0.4
MAP

Top-4

0.6 0.3 Top-5

Top-7
0.5 Top-N
0.2
0.4
MAP

Proposed 0.1
0.3
method
performance
0.2 0
Scnn4 Scnn187 Sccm Fcnn Fccm Proposed
0.1 Ffdccm
CONCEPT DETECTION MEHODS
0
1 2 3 4 5 6 7 8 9 10 15 20 n
'n' Fig. 10 Video concept detection performance of the approaches used
on the TRECVID dataset and measured by Top-d MAP on the CNN1
Fig. 9 Selection of best value of ‘n’ for proposed method = 4 epochs & CNN2 = 187 epochs
Table 9 Comparison of results in terms of the predicted concepts on five test key-frames by proposed and other methods

Key-frame Id 1524 2846 4600 7434 7787

Ground-truth Building, Face, Outdoor, Building, Face, Outdoor, Boat Ship, Face, Outdoor, Building, Face, Outdoor,
concepts Person, Sky, Urban, Vegetation, Person, Urban, Outdoor, Sports, Person, Road, Person, Sky, Urban,
Walking Running Walking Running Waterscape Waterfront Urban, Walking Running Walking Running

# (N) 8 6 4 6 7
Scnn4 Face, Outdoor, Person, Sky, Face, Outdoor, Person, Outdoor Face, Outdoor, Person, Face, Outdoor, Person,
Urban Urban Urban Sky, Urban

AP 0.63 0.67 0.25 0.67 0.71

Scnn187 Face, Outdoor, Person, Sky Building, Outdoor, Boat Ship, Outdoor, Urban Building, Outdoor, Sky,
Walking Running Waterscape Waterfront Walking Running

AP 0.50 0.50 0.50 0.33 0.57

Sccm Building, Face, Outdoor, Building, Outdoor, Person, Boat Ship, Outdoor Face, Outdoor, Urban Face, Outdoor, Person, Sky,
Person, Sky, Urban, Vegetation Urban, Vegetation Walking Running

AP 0.88 0.83 0.50 0.50 0.71

Fcnn Face, Outdoor, Person, Face, Outdoor, Person Boat Ship, Outdoor, Face, Outdoor, Person Face, Outdoor, Person, Sky
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...

Sky, Urban, Vegetation, Waterscape Waterfront

Walking Running
AP 0.88 0.50 0.75 0.50 0.57
Fccm Building, Face, Outdoor, Building, Face, Outdoor, Waterscape Waterfront Face, Outdoor, Person, Face, Outdoor, Person, Sky
Person, Sky, Urban, Vegetation, Outdoor, Person, Walking Running

AP 0.88 0.67 0.50 0.67 0.57

ProposedFFDCCM Building, Face, Outdoor, Person, Building, Face, Outdoor, Boat Ship, Outdoor, Face, Outdoor, Person, Building, Face, Outdoor, Person,
Sky, Urban, Vegetation, Person, Urban, Walking Running Waterscape Waterfront Urban, Walking Running Sky, Urban, Walking Running
Walking Running
AP 1.00 1.00 0.75 0.83 1.00
 N. J. Janwe and K. K. Bhoyar

Fig. 11 Performance 1.2

comparison of proposed and
other methods used on sample 1

AVERAGE PRECISION
test key-frames
0.8
Scnn4
Scnn187
0.6
Sccm
Fcnn
0.4
Fccm
Proposed Ffdccm
0.2

0
KF-1524 KF-2846 KF-4600 KF-7434 KF-7787

TEST KEY-FRAMES

5.4.1 CNN-based concept detectors
We performed two experiments to build CNN-based con-
cept detector. In the first, a method Scnn4 is implemented
using a classifier C1 of type deep CNN. C1 is trained for
4 epochs with key-frames training dataset and its detection
performance is measured over testing dataset. The output of
the classifier C1 , as the last layer of the network shows, is a
set of concept prediction scores; and gives us the probability
of presence of individual concepts. The concept prediction
scores need to be ranked in descending order to measure the
performance and is evaluated using MAP of Top-d measure
for different values of d.
In the second experimentation, a classifier C2 is created
with the same network structure used for C1 . It is trained
for sufficiently large number of epochs so that the concepts
with smaller data size should be recognized. C2 is trained
for 187 epochs with the same training dataset. Its detection
performance is measured separately as with C1 . We name
this method Scnn187 .
In the third experimentation, in an attempt to further
improve detection rate, we implemented a method Fcnn187
where C1 and C2 are fused together. C1 is used to detect
concepts from Group-1 and C2 is used to detect con-
cepts from Group-2 as shown in Table 7. In the classifier
fusion process, the prediction scores of Group-1 concepts
from C1 and scores of Group-2 concepts from C2 are
combined to yield the final prediction scores. These final
merged scores are used to predict the concepts. The per-
formance of the fused classifier is evaluated over the test
dataset using TOP-d MAP values.
5.4.2 Refining concept scores using FDCCM
In this experimentation, to exploit context relationship
among concepts, CCM and later FDCCM is derived from
training dataset and random image dataset from Google
images using Algorithm 2. FDCCM is used to refine
concept probabilities. In the first part of the experimen-
tation, a method Sccm is implemented to combine CNN
and FDCCM and performance is tested. In the second
part, a proposed method Ff dccm is implemented where we
investigated combined application of classifier fusion and
FDCCM. The performance of these methods are tested and
measured on all the three sample CNN classifiers trained
with 36, 76 and 187 epochs and is given collectively in
Table 8. The results show that the MAP for all the Top-
d measures for all of the methods for the CNN classifier
trained with 187 epochs is better than rest of the two
classifiers. Hence we have chosen second CNN2 classifier
with 187 epochs whereas the first CNN1 is trained for 4
epochs.

Table 10 Performance comparison of proposed method with state-of-the-art other methods, showing superiority of our approach

Method Publication year Database used Classifier MAP (Top-N)

Proposed Method FFDCCM – TRECVID2007 CNNs 0.65

TagBook [36] 2016 CVV No trained classifier 0.52
NN based Concept detector [7] 2016 TRECVID2007 NN 0.49
HCI-IP 2016 LabelMe CNN 0.40
Cascading CNN & other local descriptors [37] 2015 TRECVID 2013 CNN & Linear SVM 0.28
DMTL LC [38] 2016 TRECVID 2013 Deep multi-task learning 0.25
13 M A NTT DUT 4 4 [39] 2016 TRECVID 2013 Deep learning 0.05
Multi-label semantic concept detection in videos using fusion of asymmetrically trained deep...
In our further experimentation, we thoroughly investi-
gated the optimal length of top values to be considered
from ranked list for refinement of concepts where classifier
performance is optimum. Interestingly, the optimal value of
the ranked list of scores to be considered is found it to be 6.
The detailed result is given in Fig. 9. In all our experimen-
tation, the MAP values are computed keeping this length to
top 6 scores.
From Fig. 10, it is observed that, the proposed method
outperforms all other methods.
Table 9 presents the comparison of results for the pro-
posed method and other experimented methods with respect
to concepts detected and the average precision (AP) for the
five sample test key-frames from test dataset. Figure 11
shows the superiority of proposed method.
We have also compared the performance of the pro-
posed method with other existing state-of-the-art methods.
Table 10 shows that the proposed method exhibits outstand-
ing performance over the other state-of-the-art methods in
the area.
The work in this paper can be extended to heterogeneous
training datasets where a concept classifier is trained using
two different training datasets to compensate the concept
estimation incompleteness of individual datasets and to rein-
force discriminating ability of the estimator [41]. It can also
be extended to support sharable environment such as cloud
infrastructure [44].
6 Conclusion
In this paper, a novel contribution in the area of multi-label
semantic video concept detection has been presented. It
introduced a framework for multi-label video concept detec-
tion using state-of-the-art asymmetrically trained fusion of
convolutional neural networks and foreground driven con-
cept co-occurrence matrix, FDCCM. FDCCM keeps the
co-occurrence data for each foreground concept with other
concepts that has been built from training dataset at local
level and from a collection of random images retrieved by
Google Images at global level. The approach is evaluated
for video concept detection task over the TRECVID dataset
and compared with state-of-the-art methods, using mean
average precision. The experimental results show that the
introduction of a novel classifier using a fusion of asymmet-
rically trained deep CNNs to deal with dataset imbalance
problem improves video concept detection rate, as it is evi-
dent from improved MAP scores for various Top-d values.
This work also proves that, the use of fusion of asymmet-
rically trained CNNs and FDCCM together has improved
overall concept detection. The results summarized in Table 10
shows that the performance of the proposed approach is
far better than the other existing techniques reported in the
literature.
References
1. Feng L, Bhanu B (2016) Semantic concept co-occurrence patterns
for image annotation and retrieval. IEEE Trans Pattern Anal Mach
Intell 38(2):785–799
2. Kuo CH, Chou YH, Chang PC (2016) Using deep convolu-
tional neural networks for image retrieval. Soc Imag Sci Technol.
https://doi.org/10.2352/ISSN.2470-1173.2016.2.VIPC-231
3. Podlesnaya A, Podlesnyy S (2016) Deep learning based semantic
video indexing and retrieval. arXiv:1601.07754 [cs.IR]
4. McCormac J, Handa A, Davison A, Leutenegger S (2016) Seman-
ticFusion: dense 3D semantic mapping with convolutional neural
networks. arXiv:1609.05130v2 [cs.CV]
5. Kikuchi K, Ueki K, Ogawa T, Kobayashi T (2016) Video seman-
tic indexing using object detection-derived features. In: Proc. 24th
European signal processing conference (EUSIPCO). Budapest, pp
1288–1292
6. Awad G, Snoek CGM, Smeaton AF, Quénot G (2016) TRECVid
semantic indexing of video: a 6-year retrospective. ITE Trans Med
Technol Appl (MTA) 4(1):187–208
7. Janwe NJ, Bhoyar KK (2016) Neural network based multi-label
semantic video concept detection using novel mixed-hybrid-
fusion approach. In: Proceedings of the 2nd international confer-
ence on communication and information processing, ICCIP 2016.
ACM, Singapore, pp 129–133
8. Vedaldi A, Lenc K (2015) MatConvNet: convolutional neural net-
works for MATLAB. In: Proc. of the int. conf. on multimedia.
ACM, pp 689-692. https://doi.org/10.1145/2733373.2807412
9. Modiri S, Amir A, Zamir R, Shah M (2014) Video
classification using semantic concept co-occurrences.
https://doi.org/10.1109/CVPR.2014.324
10. Li X, Zhao F, Guo Y (2014) Multi-label image classification with
a probabilistic label enhancement model. In: UAI’14 Proceedings
of the thirtieth conference on uncertainty in artificial intelligence,
pp 430-439
11. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E,
Darrell T (2014) Decaf: a deep convolutional activation feature
for generic visual recognition. In: Proceedings of the interna-
tional conference on machine learning, ICML. Beijing, pp 647–
655
12. Zeiler MD, Fergus R (2013) Visualizing and understanding con-
volutional networks. arXiv:1311.2901 [cs.CV]
13. Memar S, Suriani AL (2013) An integrated semantic-based
approach in concept based video retrieval. Multimed Tools Appl
64:77–95. 10.1007/s11042-011-0848-4
14. Oquab M, Bottou L, Laptev I, Sivic J (2013) Learning and trans-
ferring mid-level image representations using convolutional neural
networks. Technical Report HAL-00911179, INRIA
15. Ma H, Zhu J, Lyu MRT, King I (2010) Bridging the seman-
tic gap between image contents and tags. IEEE Trans Multimed
12(5):462–473
16. Jia D, Berg A, Fei-Fei L (2011) Hierarchical semantic indexing for
large scale image retrieval. In: Proceedings of the 2011 IEEE con-
ference on computer vision and pattern recognition, CVPR 2011.
Colorado Springs, pp 785–792
17. Farhadi A, Endres I, Hoiem D, Forsyth D (2009) Describing
objects by their attributes. In: 2009 IEEE Computer society con-
ference on computer vision and pattern recognition workshops,
CVPR Workshops. Miami, pp 1778–1785
 N. J. Janwe and K. K. Bhoyar

18. Bobick A, Davis J (2001) The recognition of human movement 40. Chen X, Chen S, Wu Y (2017) Coverless information hiding
using temporal templates. IEEE Trans Pattern Anal Mach Intell method based on the Chinese character encoding. J Int Technol
23(1):257–267 18(2):91–98. https://doi.org/10.6138/JIT.2017.18.2.20160815
19. Davis JW, Bobick AF (1997) The representation and recognition 41. Tian Q, Chen S (2017) Cross-heterogeneous-database age estima-
of action using temporal templates. In: Proc. IEEE International tion through correlation representation learning. J Neurocomput
conference on computer vision and pattern recognition, pp 928– 238:286–295
934 42. Xue Y, Jiang J, Zhao B, Ma T (2017) A self-adaptive artificial
20. Zelnik ML, Irani M (2006) Statistical analysis of dynamic actions. bee colony algorithm based on global best for global optimization.
IEEE Trans Pattern Anal Mach Intell 28(9):1530–1535 Soft Comput 1–18. https://doi.org/10.1007/s00500-017-2547-1
21. Dong X, Chang SF (2007) Visual event recognition in news 43. Yuan C, Xia Z, Sun X (2017) Coverless image steganog-
video using kernel methods with multi-level temporal alignment. raphy based on SIFT and BOF. J Int Technol 18(2):209–
In: Proc. IEEE international conference on computer vision and 216
pattern recognition. Minneapolis 44. Wei W, Fan X, Song H, Fan X, Yang J (2016) Imperfect infor-
22. Zhou X, Zhuang X, Yan S, Chang SF, Hasegawa-Johnson M, mation dynamic stackelberg game based resource allocation using
Huang TS (2008) Sift-bag kernel for video event analysis. In: hidden Markov for cloud computing. IEEE Trans Services Com-
Proc. ACM international conference on multimedia. Vancouver, put (99) https://doi.org/10.1109/TSC.2016.2528246
pp 229–238 45. Chen Y, Hao C, Wu W, Wu E (2016) Robust dense reconstruction
23. Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classifi- by range merging based on confidence estimation. Sci Chin Inf
cation with deep convolutional neural networks. In: ANIPS, pp 1–8 Sci 59(9):1–11. https://doi.org/10.1007/s11432-015-0957-4
24. LeCun L, Bottou Y, Bengio, Haffner P (1998) Gradient based 46. NIST: http://www.nist.gov
learning applied to document recognition. Proc IEEE 86(5):2278– 47. TRECVID: http://www-nlpir.nist.go
2324
25. Dean G, Corrado R, Monga K, Chen M, Devin Q, Le M, Mao M,
Ranzato A, Senior P, Tucker K, Yang, Ng A (2012) Large scale
distributed deep networks. In: NIPS, pp 1–9
26. Rumelhart D, Hinton G, Williams R (1986) Learning representa-
tions by back-propagating errors. Nature 323(6088):533–536 Nitin J. Janwe has completed
27. Torralba A, Murphy KP, Freeman WT (2004) Contextual models his B.E. from S.G.G.S. Insti-
for object detection using boosted random fields. In: Proc. Adv. tute of Technology, Nanded,
neural inf. process. syst., pp 1401–1408 Maharashtra, India. He did
28. Rabinovich A, Vedaldi A, Galleguillos C, Wiewiora E, Belongie S his M.Tech.(CSE) from RTM
(2007) Objects in context. In: Proc. 11th IEEE int. conf. comput. Nagpur University, Nagpur,
vis., pp 1–8 India. Currently he is a PhD
29. Galleguillos C, Rabinovich A, Belongie S (2008) Object catego- student at Department of In-
rization using co-occurrence, location and appearance. In: Proc. formation Technology, Yesh-
IEEE Conf. comput. vis. pattern recog., pp 1–8 wantrao Chavan College of
30. Hwang S, Grauman K (2010) Reading between the lines: object Engineering, Nagpur, Maha-
localization using implicit cues from image tags. In: Proc. IEEE rashtra, India. He is a member
Conf. comput. vis. pattern recog., pp 1145–1158 of CSI & IEEE. His area of
31. Torralba A (2003) Contextual priming for object detection. Int J interest is Image & Video Pro-
Comput Vis 53(2):169–191 cessing and Computer Vision
32. Divvala S, Hoiem D, Hays J, Efros A, Hebert M (2009) An empir- & Machine Learning. He is
ical study of context in object detection. In: Proc. IEEE Conf. working as Associate Professor in Department of Computer Technol-
comput. vis. pattern recog., pp 1271–1278 ogy, Rajiv Gandhi College of Engineering, Research & Technology,
33. Feng L, Bhanu B (2012) Semantic-visual concept relatedness and Chandrapur, Maharashtra, India.
co-occurrences for image retrieval. In: ICIP, pp 2429–2432
34. Wang J, Zhao Y, Wu X, Hua XS (2011) A transductive multi-label
learning approach for video concept detection. Pattern Recogn
44:2274–2286
35. Zha ZJ, Liu Y, Mei T, Hua XS (2007) Video concept detec-
tion using support vector machines - trecvid 2007 evaluations. Kishor K. Bhoyar is cur-
Technical report Microsoft Research Lab – Asia rently working as a Professor
36. Mazloom M, Li X, Snoek CGM (2016) TagBook: a semantic in the Department of Informa-
video representation without supervision for event detection. IEEE tion Technology, Yeshwantrao
Trans Multimed 18(7):1378–1388 Chavan College of Engineer-
37. Markatopoulou F, Mezaris V, Patras I (2015) Cascade of classi- ing, Nagpur, India. He has
fiers based on binary, non-binary and deep convolutional network over 27 years of experience.
descriptors for video concept detection. In: Proc. IEEE Int. conf. He has been awarded PhD
on image processing. Quebec City, pp 1786–1790 degree in Computer Science
38. Markatopoulou F, Mezaris V, Patras I (2016) Deep multi-task and Engineering by Vishwesh-
learning with label correlation constraint for video concept detec- warayya National Institute of
tion. In: Proc. of the ACM multimedia conference. Amsterdam, pp Technology, Nagpur. He is
501–505 a member of ACM,CSI, and
39. Sun Y, Sudo K, Taniguchi Y (2014) TRECVid 2013 semantic IACSIT. His areas of interests
video concept detection by NTT-MD-DUT. In: Proc. of Trecvid are Image & Video Processing
2014 and Machine Learning.