2020 International Conference on Power, Instrumentation, Control and Computing (PICC)
Transformer Network for video to text translation
*
2020 International Conference on Power, Instrumentation, Control and Computing (PICC) | 978-1-7281-7590-4/20/$31.00 ©2020 IEEE | DOI: 10.1109/PICC51425.2020.9362374
Mubashira N
Department of Computer
Science and Engineering
Government Engineering College Thrissur
Kerala,India
mubashiranechikkat@gmail.com
Abstract—Recently generation of natural language descriptions
for videos has created a lot of focus in computer vision and nat-
ural language processing research. Video understanding involves
detecting scene’s visual and temporal elements and reasoning
it for description generation.Several real world implementations
such as video indexing and retrieval,video to sign language
translation etc, are based on this.Because of the complicated
nature and diversified content, the captioning problem becomes
more challenging.This is a machine translation problem which
uses encoder decoder architecture of GRU or LSTM to deal
with this kind of problem.But here the decoding starts with
the final hidden state of encoder as input.This can’t be a good
summary of the input sequence,because all the intermediate
states of encoder are ignored.This paper proposes a transformer
network with deep attention based encoder and decoder to
generate the natural language description for video sequence
data. This network processes the sequences as a whole and learns
relationship between each elements in the sequence by providing
attention.
Index Terms—Video captioning,CNN,Transformer
network,Attention Mechanism,LSTM,RNN
I. I NTRODUCTION
Understanding the contents of visual data is a complex
task.Nowadays, Machine learning techniques allows us to
train the context for a dataset,so that an algorithm can
understand what are the content in a video.Problems re-
lated describing visual contents, computer vision and natu-
ral language processing is taking an increasingly complex
challenges and it seeing the accuracy comparable to hu-
man observations.Studying and analysing the content of a
video is the important research area of multimedia.The task
of generating contextual description of visual contents is a
challenging and essential task in computer vision.This gen-
erally includes feature extraction and generating descriptions
based on the feature vectors extracted.Semantic concepts like
scenes,objects,actions,interactions between objects and order-
ing of the events in temporal dimension should consider to
design an efficient solution architecture for the captioning
problem.Moreover, this requires translation of the extracted
visual information into grammatically correct natural language
by preserving the semantic concepts.Content based recom-
mendation and retrieval, human-robot interaction,autonomous
978-1-7281-7590-4/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.
Dr.Ajay James
Assistant Professor,Department of
Computer Science and Engineering
Government Engineering College Thrissur
Kerala, India
ajay@gectcr.ac.in
driving,video subtitling, procedure generation for instructional
videos, video surveillance software for visually impaired
people and understanding sign language are the real world
applications.
Convolutional neural networks(CNN) provides more sophis-
ticated feature representations by doing series of convolution
operations over images or videos(ie, series of frames).These
convolutions are for comparing the visual data frames against
some specific patterns(filters) that the network looking for.As
the network perform more convolutions, it could identify spe-
cific object.This is done by large amount of labelled data.It is
clear that CNN can only identify the spatial features or visual
data of an image,but can’t handle temporal features, that is
how a frame related to one before it.Temporal sensitive model
can be vector-to-sequence,sequence to-vector,or sequence-to-
sequence.Here is the importance of temporal sensitive models
like Recurrent neural networks(RNN),Long short term mem-
ory(LSTM),Auto encoders and Transformer networks which
takes the output of CNN to get the output which might be a
vector or sequence based on the model.Attention mechanism
helps to which part of input should focus to yield more
accurate outcome.Video captioning is kind of sequence to
sequence modeling,it takes series of frames in, to generate
textual description.
This process of collecting the interactions between ob-
jects,identifying the fine changes of video contents in temporal
dimension, prioritizing the activities captured in the video
makes the captioning problem more challenging.
In this paper Section II discuss the Related works of differ-
ent methodology proposed for video captioning.A transformer
network proposed for video captioning described at section
III. Different datasets used for training the models presents in
section IV and Section V explains about evaluation metrics.
II. R ELATED WORKS
The methodologies can be categorized into two methods
that are template-based methods, sequence learning methods.
A. Template based method
Template based method[1] are based on set of specific
grammar rules.First the sentences are divided in to three
 2020 International Conference on Power, Instrumentation, Control and Computing (PICC)
types of fragments:subject, verb and object by following the
grammar.visual contents detects word into objects, actions
and attributes categories.Then and each fragment is associated
with detected words.Then generated fragments are composed
to a sentence with predefined language template.this method
focused mainly on the detection of predefined entities and
events separately.Describing open domain videos using this
method found unrealistic or too expensive because of its
computational complexity.
B. Sequence learning method
Deep RNN based model[2] to translate videos to natural
language is a naive approach.The frame features are extracted
using CNN.Applying mean pooling along the features ex-
tracted across the entire video to get a single video descrip-
tor,which is then fed into the LSTM network to generate tex-
tual description.But this mean pooling causes loss of temporal
information.
Another work[3] of video to text discusses end-to-end
sequence to sequence-video to text(S2VT) model to generate
captions for videos.this method is similar to machine trans-
lation between natural languages.A two stack LSTM first
encodes the feature vectors generated by CNN for RGB
images or optical flow images.Then the decoder generates
sentences.Decoding starts only when all the features are en-
coded.So the decoder gets only the previous output and hidden
states as input,which works well for short sequences but can’t
memorize long term dependencies.
An attention based LSTM and semantic consistency used in
the work of[4] for video captioning.This framework integrates
attention mechanism with LSTM to capture salient structures
of the video and also explores correlation between multimodal
representations(ie text and visual data) to generate sentences
with rich semantic content.Inception v3[5] CNN used here
to extract more meaningful spatial features.This architecture
allows to do the convolution along with pooling operation
in a single layer of CNN and stack up the feature maps to
get a single volume output.The extracted features fed into an
attention based long short term memory encoder-decoder.
A dual stream recurrent neural network architecture based
on the work on video captioning[6] discusses about both
visual and semantic stream.This architecture includes a visual
descriptor and a semantic descriptor which encodes both visual
and semantic features respectively.Visual descriptor encodes
frame representations of video and the semantic descriptor
encodes each video frame with a high level representation
of semantic concept like objects,actions, and interactions.This
visual descriptor and semantic descriptor are different modal-
ities.Dual stram RNNs are used because of this two asyn-
chronous modalities to flexibly exploit the hidden states of
each stream.Finally it integrates the hidden state representa-
tions of visual and semantic descriptor.Then a dual stream
decoder is deployed to perform dual stream hidden state fusion
for sentence generation.An attentive multi grained encoder
module to enhance the local feature learning with global
semantics feature for each modality.
Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.
Another study based on a reconstruction network for video
captioning[7] is based on dual learning approach[8].It is a
learning framework that leverages the (primal dual)structure of
AI task to obtain effective feedback or regularization signals
to enhance the learning or inference process.Regularization
techniques is to avoid overfitting.This happens when the model
trying to capture the noise in the training data.The architecture
consist of three modules:a CNN based encoder which extract
the semantic representations of the video frames.A LSTM
based decoder which generates natural language for visual
content description.Then the reconstructor which exploits the
backward flow from caption to visual contents to reproduce the
frame representations.From this reconstructed representation it
provides a constrain for decoder to embed more information
in to the input video representations.
Fused GRU with semantic-temporal attention for video cap-
tioning[9],this methodology provides two types of attention,
semantic and temporal.Temporal attention involves directing
attention to specific instant of time while decoding the video
representation in to textual description.Semantic attention is
the ability to provide the representations of semantically
important objects that are needed when they are needed.The
different modules of this architecture are a CNN based video
encoder for feature extraction,semantic concept prediction
network and hierarchical semantic decoder network.
Video Captioning via Hierarchical Reinforcement Learn-
ing[10] is the work of incorporating reinforcement learning
techniques into video captioning.A high-level Manager module
learns to design sub-goals and a low-level Worker module
recognizes the primitive actions to fulfill the sub-goal.
Transformer network is a major breakthrough towards more
sophisticated sequence learning models.Transformer[11] is a
deep learning model introduced,which are designed to solve
the problem of sequence transduction, such as machine trans-
lation and text summarizing.This is better than all other
architecture because they totally avoid recursion by processing
sequences as a whole and by learning relationship between
input elements by providing attention based encoder and
decoder. Attention mechanism which provides the advantage
to look entire input and the target sequence generated so far
given as input to the encoder and to the decoder respec-
tively.The softmax induces the probability distribution among
output.Transformers not only provides the attention but also
make parallel the work of processing sequences.All these
properties of transformers benefited a lot in the studies of
image and video captioning.
III. P ROPOSED SYSTEM
The proposed system based on the transformer network.A
CNN module designed to extract features form video frames.A
deep self attention based encoder in transformer converts the
feature vectors from CNN in to set of vectors with context and
order information included.The decoder predicts the next word
by using the output from encoder and the predicted words so
far.Figure 1 shows the block diagram of the whole framework.
 2020 International Conference on Power, Instrumentation, Control and Computing (PICC)

Fig. 1. Proposed transformer network for video captioning

B. Attention based deep sequence encoder

The output from the inception network will be a feature
vector of some dimension ’d’.The extracted visual feature does
not represent the context and order information of the frames.
1) Attention mechanism: This is to take context of
each vector associated with all other vectors in the se-
quence.Consider the input embedding X= x1 , x2 , ..., xN .Then
follow the process below for all the frames from 1,...,N.
• Keys(K), Values(V) and queries(Q): Create three vectors
from each of the input feature vectors.These vectors are
created by multiplying the embedding by three weight
matrices Wk , Wv and Wq which will trained during the
training process.
Fig. 2. An inception module • k1 , v1 , q1 are the keys,queries and values generated
from (x1 .Wk ),(x1 .Wv ) and (x1 .Wq ).Then K
= {k1 , k2 , .., kN },V={v1 , v2 , .., vN } and Q =
{q1 , q2 , .., qN }.dk is the dimension of k,v and q.
A. Feature extraction using Inception Network • Inner product:Inner product of qi and kj , (qi .kj ) for every
j= 1,2,...N. is to quantify how similar all inputs to the the
vector xi
A single specific operation is defined between layers while • Softmax function Normalizes to get relative similarity
designing a CNN architecture.The inception network is an between Query qi and key kj vector, ri→j
exception, which is modelled so that it could perform convo-
lution along with pooling in a single layer.The output feature eqi .kj
ri→j = PN (1)
map is then stacked up into a single volume.Filters with qi .kl
l=1 e
different dimension can be used,but the output dimension will
not change in terms of height and width because it performs This measures how similar the ith input to the j th input,
’same convolution’ and padding in pooling.A bottleneck layer relative to all other N input in the sequence. ri→j always
PN
is introduced to reduce the computational cost without hurting positive,between 0 and 1 and j=1 ri→j = 1
the performance.Figure 2 shows an inception module of the • Multiply each relative relationship with corresponding
network, Value vector, (eg: ri→1 ) with v1 .Add all these together

Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.
 2020 International Conference on Power, Instrumentation, Control and Computing (PICC)

to get refined code zi for xi ,Then Z = z1 , z2 , ..., zN C. Deep Self and cross-attention based decoder
N
X Input of the decoder is the words are generated thus far
zi = ri→l .vl (2) marked as output sequence in the decoder network in figure
l=1
4.Left most word is the most recently predicted word.Every
The above process is only provides the context factor time a new word predicted ,input sequence on the bottom
irrespective of the frame ordering.But the ordering does mat- shifts to the right by one position.The positional embedding
ters.So positional embedding is for include the ordering. added to get the order.In cross attention(Encoder-Decoder
2) Positional Embedding(PE): attention) the keys K and values V are generated from the
• Constitute a ’d’ dimensional positional embedding each output of the encoder and query Q is the generated from
embedding dimension having a sinusoidal function. the output wordvector predicted thus far.Multihead attention
• The frequency of sine wave directly proportional to allows the attention mechanism to attend different aspects
embedding dimension. of characteristics of sequence.There is a feed forward neural
• Associate a number to each of ’d’ dimension in the PE network same as in the encoder.This operations performed J
connected to the sine wave at that dimension with respect times to form Deep Self and cross-attention based decoder A
to the the position.
The simple block diagram of deep sequence encoder shown
in the figure 3.The process performed by the encoder are:
• Positional embedding added to this feature vectors to
provide the order information.
• Then the attention network takes in to account the context
of the video frames.Every feature vector corresponding to
frames plays the roll of keys K,values V and queries Q.
• A skip connection provided for not to loose the original
feature vectors and add it into output of attention network.
• This added and normalized output then fed in to a
feed forward neural network to provide regularization
or structure on this network, and the hyperbolic tanh
function restricts the output of the network to be between
-1 and 1.
• Repeat the above process for K times for deep sequence
encoder.

Fig. 4. Deep self and cross attention based decoder

softmax on the top of this network is to predict the next word

in the sequence.This will be the next left most input to the
decoder network and previously generated sequence shifted to
right.

IV. DATASETS

The roll of datasets are crucial for structuring a machine

Fig. 3. Deep self attention encoder
Self attention is for modifying the vector representation of
each frames to take in to account the characteristics of
surrounding frames.
learning model.A dataset which contains enormous training
instances and discussing a wide range of contexts would con-
tribute for the effective learning process.This section describes
the datasets that are commonly used for image and video
captioning.

Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.
 2020 International Conference on Power, Instrumentation, Control and Computing (PICC)
A. Image Datasets
1) Microsoft COCO[12]: Short for common objects in
contexts is a large image classification/recognition, object
detection, segmentation, and captioning dataset. It contain a
total of about 165k training images, 81k of validation data,
and 81k test images with 91 categories with 5 captions per
image.
2) Flickr30K dataset[13]: This dataset itself is a bench-
mark for sentence based image description, which augments
the 158k captions from Flickr30k, contains 244k coreference
chains and 276k manually annotated bounding boxes for each
of the 31k images and 5 English captions for each image in
the original dataset.coreference chains means, each image in
the dataset has a txt file in the ”Sentences” folder. Every line
of this file contains a caption with annotated phrases blocked
off with brackets.
B. Video Dataset
1) Montreal Video Annotation Dataset (M-VAD)[14]: It
consist of 49k video clips taken from DVD movies. 39k video
clips from the dataset used for training the captioning model.
49k are allocated for validation and 5,000 video clips are used
for testing.A single sentence description provided even though
it is difficult and challenging to describe a movie in single
sentence.This video dataset follows face tracks to the semi-
automatic annotation process.Tracking face only because of it
directly depends on the body movements.
2) MPII Movie Description Corpus (MPII-MD)[15]: As
name indicates it is a movie description dataset.The video
snippets are taken from 94 HD hollywood movies and over
68k sentence descriptions in parellal.37k video clips are taken
from 55 movies.Audio descriptions also included along with
31k video clips are taken from the 49 movies.
3) Microsoft Research Video Description Corpus
(MSVD)[16]: This dataset consist of 1970 video clips each of
length about 10 seconds created by Amazon Mechanical Turk
(AMT).This dataset splits in to training,validation and test
dataset each having 1200,100,670 video clips respectively.
Every video snippet are annotated in multiple description
in different language with single sentence. Each video clips
annotated by 40 different sentences in English.
4) MSR Video to Text (MSR-VTT)[17]: new large-scale
video benchmark for describing video contents in texts.It
contains 10000 web based video clips and 200k single sen-
tence description.A single video may have multiple descrip-
tions.Here 20 captions included per video.The dataset splits in
to 6513 training,2990 testing,497 validating samples.
V. E VALUATION M ETRICS
There could be multiple equally good descriptions for a sinle
video clip.The following methods are commonly used machine
translation systems to calculate the accuracy. A machine
translation system can have more than one good answers.
Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.
A. BLEU Score(Bilingual evaluation Understudy)[18]
BLEU is the most commonly used algorithm to evaluate
the quality of the text description generated to the reference
sentences.The evaluation approach is by counting the matching
n-grams in the candidate translation to n-grams in the reference
text.The unigram would be each token bigram comparison
would be each word pair.All this comparisons are made
regardless of order.It is a modified precision measure in which
each word credit only up to the maximum number of times it
appears in the reference sentences.
B. METEOR[19]
Metric for Evaluation of Translation with Explicit Ordering,
is calculated as an average mean of precision and recall
by performing stemming and looking up for synonyms in
wordNet[20].It relies on finding an optimal visual to text
alignment. This metric evaluates machine generated output by
aligning it to the reference translations to find sentence level
similarity scores.The words are considered as matched if the
surface forms of words are identical(exact),stems are identical.
Match phrases if they are listed as paraphrases in a language
appropriate paraphrase table.
C. ROUGE[21]
This is a Recall-Oriented Understudy for Gisting Evalua-
tion.It is an intrincic metric for evaluating summaries based
on BLEU scores.Given the automatic description generated by
machine and set of reference descriptions.ROUGE-N is to find
that what percentage of N-grams from these human generated
references occur in machine generated description.
VI. C ONCLUSION
The purpose of this study is to design a transformer network
model for translating video clips in to natural language sen-
tence.A deep self attention based encoder encodes the frame
using keys,values, and queries.Self and cross attention based
deep decoder generates sentences.This network is different
from LSTM and other sequential learning models,because de-
pendency every vectors in sequence with surrounding vectors
can be perform parallel.
R EFERENCES
[1] H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Doll´ar,
J.Gao, X. He, M. Mitchell, J. C. Platt, et al. “From captions to
visual concepts and back,” in Proc. Conf. Computer Vision and Pattern
Recognition, pp. 1473–1482, 2015.
[2] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, K.
Saenko, Translating Videos to Natural Language Using Deep Recurrent
Neural Networks, Conference of the North American Chapter of the
Association for Computational Linguistics (NAACL), 2015, pp. 1494-
1504.
[3] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, K.
Saenko, Sequence to sequence-video to text, Proceedings of the IEEE
International Conference on Computer Vision, 2015, pp. 4534-4542.
[4] L. Gao, Z. Guo, H. Zhang, X. Xu, and H. T. Shen, “Video captioning
with attention-based LSTM and semantic consistency,” IEEE Trans.
Multimedia, vol. 19, no. 9, pp. 2045–2055, Sep. 2017.
[5] C. Szegedy,W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In
CVPR, 2015.
 2020 International Conference on Power, Instrumentation, Control and Computing (PICC)

[6] N. Xu, A.-A. Liu, Y. Wong, Y. Zhang, W. Nie, Y. Su, and M.

Kankanhalli,“Dual-stream recurrent neural network for video caption-
ing,” IEEE Transactions on Circuits and Systems for Video Technology
(TCSVT), 2018.
[7] Wang, B., Ma, L., Zhang, W., Liu, W.: ”Reconstruction network for
video captioning”.In: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. (2018) 7622–7631
[8] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T. Liu, and W.-Y. Ma. Dual
learning for machine translation. In NIPS, pages 820–828, 2016.
[9] L. Gao, X. Wang, J. Song, Y. Liu, Fused GRU with semantic-
temporal attention for video captioning, Neurocomputing (2019)
https://doi.org/10.1016/j.neucom. 2018.06.096 .
[10] X. Wang, W. Chen, J. Wu, Y.-F. Wang, and W. Y. Wang. Video caption-
ing via hierarchical reinforcement learning. In The IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2018.
[11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion
Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. ”At-
tention is all you need”. In Advances in Neural Information Processing
Systems, pages 6000–6010.
[12] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Doll´ar, and C. L. Zitnick. Microsoft COCO: Common objects in
context. In ECCV. 2014.
[13] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image
descriptions to visual denotations: New similarity metrics for semantic
inference over event descriptions,” ACL, vol. 2, pp. 67–78, 2014.
[14] A. Torabi, C. Pal, H. Larochelle, and A. Courville, “Using descriptive
video services to create a large data source for video annotation
research,” arXiv preprint arXiv:1503.01070, 2015.
[15] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele, “A dataset for
movie description,” in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2015, pp. 3202–3212.
[16] D. L. Chen and W. B. Dolan, “Collecting highly parallel data for
paraphrase evaluation,” in ACL: Human Language Technologies-Vol.
1. Association for Computational Linguistics, 2011, pp.190–200
[17] J. Xu, T. Mei, T. Yao, and Y. Rui. MSR-VTT: A large video description
dataset for bridging video and language. In CVPR, 2016.
[18] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: a method for
automatic evaluation of machine translation”, in Proceedings of the 40th
Annual Meeting on Association for Computational Linguistics (ACL
’02). Association for Computational Linguistics, Stroudsburg, PA, USA,
311-318, 2002.
[19] D. Elliott and F. Keller, “ Image description using visual depedency
representations,” in Proc. Empirical Methods Natural Lang. Process.
2013, vol. 13, pp. 1292-1302.
[20] Miller, G., Beckwith, R, Fellbaum, C., Gross, D., and Miller, K. 1990.
Introduction to WordNet: An on-line lexical database. International
Journal of Lexicography (special issue), 3(4):235-312.
[21] Lin CY, “ROUGE: a package for automatic evaluation of summaries”,
in Proceedings of the workshop on text summarization branches out,
Barcelona, Spain, (WAS2004) 2004

Authorized licensed use limited to: VIT University. Downloaded on March 30,2023 at 10:18:46 UTC from IEEE Xplore. Restrictions apply.