The 7th IEEE International Conference on E-Health and Bioengineering - EHB 2019

Grigore T. Popa University of Medicine and Pharmacy, Iasi, Romania, November 21-23, 2019

Enhancing the Accessibility of Hearing Impaired

to Video Content through Fully Automatic
Dynamic Captioning
AuthoBogdan Mocanu1,2, Ruxandra Tapu1,2, Titus Zaharia1
1 ARTEMIS Department, Institute Mines - Télécom/Télécom SudParis, UMR CNRS 5157
SAMOVAR Evry, France
2 Department of Telecommunications, Faculty of ETTI, University “Politehnica” of Bucharest

Bucharest, Romania

Abstract—In this paper we introduce an automatic subtitle
positioning approach designed to enhance the video accessibility
of deaf and hearing impaired people to multimedia documents.
By using a dynamic subtitle and captioning approach, which
exploits various computer vision techniques, including face
detection, tracking and recognition, video temporal segmentation
into shots and scenes and active speaker recognition, we are able
to position each video subtitle segment in the near vicinity of the
active speaker. The experimental evaluation performed on 30
video elements validates our approach with average F1-scores
superior to 92%.
Keywords—dynamic subtitle, active speaker recognition, deaf
and hearing impaired people.
I. INTRODUCTION
The proliferation of audio-visual production has led to the
development of efficient and effective techniques designed to
understand and organize video content to facilitate the access to
information. However, the millions of people that are suffering
from hearing impairment (HI) are facing important difficulties
in the comprehension of video content [1]. The incorporation
of subtitles/close captions in the multimedia content is today
the most largely used solution to this problem [2].
Conventionally, the subtitle is positioned in a fixed
location; at the bottom part of the TV screen. Although the
hearing impaired audience can retrieve certain information
from the associated script, the existent positioning techniques
are far from satisfactory in assisting the deaf or HI users. They
notably suffer from the following limitations: (1) Confusion
between various actors – for scenes with multiple active
characters the HI users need to judge to which one the script is
associated to; (2) Tracking the subtitle displayed on the
screen: for characters speaking rapidly the deaf and HI people
can miss parts of the sentence. Moreover, for various active
speakers the subtitle’s display time can vary over a wide
range.
In order to deal with such limitations, in this paper we
introduce a novel, completely automatic, dynamic subtitle
positioning methodology. The proposed approach is designed
to assist the HI user to comfortable follow the video content
and the associated subtitles simultaneously, while enhancing
the video consumption experience.
The rest of the paper is organized as follows. In Section II
we briefly review the state of the art in the field. Section III
introduces the proposed architecture, and details the video and
audio analyzers considered. The experimental evaluation is
presented in Section IV. Finally, Section V concludes the paper
and opens some perspectives of further work.
II. RELATED WORK
Efforts on accommodating the hearing impaired people in
accessing video content can be traced from early 1970’s where
the US National Television has demonstrated the efficiency of
transmitting close captions together with the video stream [3].
However, the issue of dynamic subtitling has been
addressed only recently by the scientific community. Hong et
al. [4] are one of the first to propose a technique for adjusting
the position of each text segment relatively to the video
content. Their system receives as input a video stream, its
associated script and subtitle files. A mapping between the
character’s face and its corresponding script is performed with
the help of face detection and recognition techniques. Then, a
visual saliency approach is exploited in order to determine a
non-intrusive region, around the speaker face, that is further
used to present the script. The method is extended in [5], where
a novel lip motion analysis algorithm for active speaker
detection is introduced. The position of the subtitles is
determined with the help of an optimization procedure that
jointly takes into account the presence/absence of the speaker
within the scene, the cross-frame coherence and the screen
layout. Even though the usability studies performed have
shown an increase in terms of quality of experience and content
comprehension, the method still presents several limitations
that need to be overcome: (1) The system functions solely for
active speakers facing the video camera (frontal faces). For
profile faces or, more generally, for characters with
unfavorable poses, no information regarding the identity of the
speaker can be determined.(2) The strategy of placing the video
subtitle in the near surrounding of the active speaker mouth is
not optimal in all cases. Thus, for characters moving within the

978-1-7281-2603-6/19/$31.00 ©2019 IEEE

scene, the subtitle position fluctuates around the speaker
mouth, which makes it difficult/tiresome to follow. In addition,
in some cases, the subtitle may occlude some relevant parts of
the video content (e.g., faces of the other characters in the
scene or useful, incrusted text elements that may be present in
the background). Moreover, such a strategy is convenient for
relatively short text segments. For long phrases covering two
lines of text, the subtitle fluctuation becomes very disturbing.
Other approaches such as those introduced in [6] and [7]
concern the so-called gaze-based subtitle positioning
techniques. The method introduced in [6] aims at reducing the
viewer's eye movement, without interfering with the target
region of interest (ROI). A ROI is defined as a salient region in
the video scene that should not be occluded by subtitles. For
each video frame, the ROI is determined using the eye-tracking
data extracted from multiple viewers. The method is further
extended in [7] with an active speaker recognition framework
based on audio and video information. However, even though
the subjective evaluation performed shows promising results,
both methods assume that the active speaker eyes and mouth
are visible in the scene, which is not always true. Furthermore,
the gaze information is difficult to obtain, since it requires the
set-up of a complex acquisition protocol that involves multiple
users.
In [8], the authors introduce a study regarding the user
experience when watching videos together with the associated
subtitle dynamically positioned on the screen. First, the eye
data is analyzed in order to determine the difference between
gaze patterns obtained for both subtitled and non-subtitled
videos. Then, the users were asked to view videos with
dynamic subtitles and to express their attitude towards it. As a
general conclusion, it can be highlighted that dynamic subtitle
approaches create gaze patterns that are closer to the baseline
than those induced by regular subtitles. The work is extended
in [9], where a comparative evaluation study of users watching
videos, with and without subtitles, is presented. The focus is
put on tracking the user’s eyes. The study pointed out that
static subtitles make it easier to look around, but more difficult
to understand the video content.
The analysis of the literature shows that the existent
approaches are still far from being satisfactory in assisting the
hearing impaired users in video content understating. Most
systems fail in identifying the speakers in the case where the
active character is not facing the video camera or not visible in
the scene. In addition, all methods are sensitive to visual
appearances of relatively poor quality. In this paper we
introduce a novel framework designed to overcoming such
limitations and detailed in the following section.
III. PROPOSED APPROACH
A. Face detection, tracking and recognition
The face detection is performed using the Faster R-CNN
architecture [10], extended with the region proposal networks
[11]. The faces are detected in every frame of the video
stream. Then, for each face its associated track is determined
with the help of the ATLAS algorithm [12], extended here to
work on multiple moving instances.
The face recognition process is based on the VGG16 CNN
architecture [13] that is extended with a learning-based weight
optimization scheme [14] that determines the relevance of
each face instance to the final global descriptor associated to a
face track. The set of weights is adaptively determined
depending on the frame’s degree of noise and motion, face
poses or viewing angles.
B. Video temporal segmentation into shots and scene
Initially, the video stream is divided into shots using the
graph partition strategy presented in [15]. Then, the shots are
used to create video scenes that satisfy a certain homogeneity
with respect to a semantic criterion (i.e., the presence of the
set of faces). Using the video shots and the recognized
characters, we construct a connected graph, developed at the
global level of the video sequence. The cut-edges of the graph
are used in order to form the set of scene boundaries. The
output of the module is a list of characters associated to each
video scene.
C. Active speaker recognition
The active speaker recognition process involves different
stages, starting with a segmentation of the audio signal into
segment correlated with the textual subtitles.
1. Identification of subtitle-related audio segments.
Using the synchronized subtitle file associated to each video
document, we extract the timestamps during which a specific
text segment is displayed. Let us note that the extracted
timestamps are not perfectly correlated with the audio stream,
because: (1) In order to facilitate the user reading, the subtitle
is always displayed longer time intervals than the actual
speaking moments. (2) The subtitle may contain one or two
lines of text that can correspond to multiple active speakers.
Initially, we divide the audio stream into smaller segments,
denoted by audio chunks, using the timestamps extracted from
the subtitle file. Then, based on the observation that a subtitle
segment belonging to two different speakers contains two
dialog lines, we propose to divide, in such cases, the
corresponding audio chunk into two sub-segments. The begin
time ( and ) and the end time ( and ) of each
audio sub-segment can be computed using the initial subtitle
segment timestamps and its total number of letters ( ), as
described in the following equations:
−
= , (1)
= ; = + ∗ , (2)
= ; = , (3)
where and are the beginning and respectively, the
end time of the current audio chunk, while denotes the
total number of letters from the first line of the text.
Finally, as recommended in [16], we apply traditional
speech processing techniques including voice activity
detection, silence and unvoiced speech removal, in order to
filter out the non-relevant segments. The extracted audio
segments are further used for speech recognition purposes, as
described in the following.
2. Speaker recognition process.
Fig.1 presents the synoptic scheme of the speaker
recognition framework.
Fig. 1. The proposed active speaker recognition framework
The previously determined audio segments are first
converted into a single-channel 16bits stream at a sampling
rate of 16 kHz. We treat the speaker recognition problem as a
multi-category classification task.
For each audio chunk, we generate spectrograms that are
represented as vectors of size (257 x T x 1), where 257 denotes
the number of spectral components used in order to computed
the short-time Fourier transform, T is the temporal length of
each audio segment (expressed in seconds) and 1 represents
the number of channels used to represent the spectrogram.
Mean and variance normalization are performed on each
frequency bin of the spectrum.
From each spectrogram, a set of features is extracted using
the residual-network (ResNet-34) CNNs architecture [17]. We
have modified the ResNet-34 in a fully convolutional way in
order to adapt it to 2D spectrogram inputs [18].
In order to obtain a compact (i.e., low-dimensional) output
descriptor at the end of the ResNet-34 a NetVLAD layer for
feature aggregation along the temporal axis has been added.
So, the ResNet-34 architecture maps the input spectrograms to
frame level descriptors. The NetVLAD layer takes dense
descriptors as input and produces a single matrix V of size K x
D, where K refers to the total number of chosen clusters and D
refers to the dimensionality of each cluster.
The final output of the network is obtained by performing
the L2 normalization and concatenation. In addition, in order
to reduce the processing burden a dimensionality reduction is
performed also in the fully connected layer where we impose a
maximum size of 512.
Since the active speaker recognition task is treated as an
image classification problem, the output of the last layer of the
CNN is fed into a softmax operator in order to produce a
distribution over all the considered classes. The top-3 results
for each audio chunk are validated, in descending order of
probability, using the list of characters available for each
scene. We finally retain as a correct prediction the active
speaker with the highest probability score. The final stage
concerns the set-up of a subtitle positioning strategy.
3. Subtitle placement.
Based on the information regarding the identity of the
active speaker, the objective is to position its associated
subtitle segment in the subject’s near vicinity.
In contrast with other state of the art techniques as [4], [5]
that propose using speech bubbles to place video subtitles, we
define 15 potential candidate regions for placing each speech
segment. The proposed locations are defined by considering 3
levels of subdivision along the horizontal axis and 5 levels on
the vertical one. In the vertical direction the subtitle can admit
an offset relative to the top part of the screen between 60%
and 80% with a step size of 5%. In the horizontal direction we
admit an offset relative to the left part of the screen of 0%,
20% and 40%.
The proposed approach has the advantage of minimizing the
subtitle fluctuation when the subject is moving. In our case,
for each subtitle segment its horizontal position is determined
using the active speaker’s face centroid, based on a majority
location voting scheme, applied over the whole set of frames
where the subtitle is present. The vertical location is computed
so that no overlap with other faces that are present in the scene
occurs. The default position considered has an 80% vertical
offset location and we continue the analysis in ascending
order.
IV. EXPERIMENTAL EVALUATION
The experimental evaluation has been conducted on a data
set of 30 video streams with an average duration of 20
minutes. Twenty videos have been selected from the France
Télévisions TV series “Un si grand soleil” and ten from the
movies “Friends” and “The Big Bang Theory”.
For the CNN training process involved in the active speaker
recognition process, we have considered a database of 110
known characters. In order to extract the audio segments
required for training, we have used the first 100 broadcasted
episodes of the “Un si grand soleil” series while for the US
series we have used all episodes from season I. First, we
applied the face tracking and recognition, in order to identify
the potential active speakers. Then, we have performed a
speech to mouth verification using the SyncNet [19] approach
in order to verify if there is a correlation between the audio
track and the mouth motion in videos. In this way, we have
developed a training dataset with 13000 appearances of the
110 characters involved. The lowest length of any speech
segment in the database is 1.25ms.
Fig. 2. Experimental results of the proposed subtitle positioning system
For the testing dataset, 15208 utterances have been
extracted. Let us underline that more than 3500 speech
segments belong to unknown individuals.
For evaluation purposes, we have retained traditional
performance metrics, including accuracy (A), recognition rate
(R) and F1 norm. In addition, we have compared our results
with state of the art techniques [4] and [5]. Some visual
examples of the results obtained are illustrated in Fig. 2. The
global objective performance scores are summarized in Table I.
TABLE I. OBJECTIVE PERFORMANCE SCORES
Method
Accuracy Recognition rate
Hong [4] 81.25% 78.47%
Hu [5] 88.14% 81.84%
Ours 98.57% 87.79%
As it can be observed, the proposed approach returns gains
of more than 8% in both precision and recall scores. This
behavior is explained by the robustness of the proposed
technique to face pose variation, camera/background motion,
and to various types of noise such as background chatter,
laughter or overlapping speech.
V. CONCLUSIONS
In this paper we have introduced a novel automatic system
designed to enhance video viewing experience by dynamically
place the video subtitle in the vicinity of the active speaker.
The proposed framework is based on a multimodal fusion of
information obtained from multiple media channels involved in
a video document: visual, audio and text. The experimental
evaluation performed on 30 multimedia documents validates the
proposed methodology that returns an average F1 score superior
to 92% and gains compared with state of the art techniques of
more than 8%. For further work, we envisage performing a
large usability study on deaf and hearing impaired people.
ACKNOWLEDGMENT
Part of this work has been supported by a mobility project
of the Romanian Ministry of Research and Innovation, CNS-
UEFISCDI, project number PN-III-P1-1.1-MCD-2019-0157,
within PNCDI III” and by French PIA (Plan d’Investissement
d’Avenir) SUBTIL project.
REFERENCES
[1] B. Safadi, M. Sahuguet, and B. Huet, “When textual and visual
information join forces for multimedia retrieval”. Proceedings of
International Conference on Multimedia Retrieval, pp. 265-272, 2014
[2] A. Tamayo, F. Chaume, “Subtitling for d/Deaf and hard-of-hearing
children: current practices and new possibilities to enhance language
development”. Brain Science; 7(7): 75, 2017.
[3] https://signlanguageco.com/a-brief-history-of-closed-captioning/ -
Accessed on 2 September 2019.
[4] R. Hong, M. Wang, M. Xu, S. Yan, T.-S. Chua, "Dynamic captioning:
Video accessibility enhancement for hearing impairment", Proc. ACM
Conf. Multimedia, pp. 421-430, 2010.
[5] Y. Hu, J. Kautz, Y. Yu, W. Wang, "Speaker-following video subtitles",
ACM Transactions on Multimedia Computing, Communications, and
F1 norm
Applications, vol. 11, no. 2, pp. 1-17, 2014.
79.83% [6] W. Akahori, T. Hirai, S. Kawamura, and S. Morishima, “Region-of-
84.87% interest-based subtitle placementusing eye-tracking data of multiple
viewers”. Proc. ACM International Conference on Interactive
92.85%
Experiences for TV and Online Video, pp.123–128, 2016.
[7] W. Akahori, T. Hirai and S. Morishima, “Dynamic Subtitle Placement
Considering the Region of Interest and Speaker Location”, Proc. of the
12th International Joint Conference on Computer Vision, Imaging and
Computer Graphics Theory and Applications, pp. 102-109, 2017.
[8] A. Brown, R. Jones, M. Crabb, J. Sandford, M. Brooks, M. Armstrong,
C. Jay, "Dynamic subtitles: the user experience", Proceedings of the
ACM International Conference on Interactive Experiences for TV and
Online Video, pp. 103-112, 2015.
[9] K. Kurzhals, E. Cetinkaya, Y. Hu, W. Wang, and D. Weiskopf “Close to
the Action: Eye-Tracking Evaluation of Speaker-Following Subtitles”
Proceedings of CHI Conference on Human Factors in Computing
Systems. ACM, USA, 6559–6568, 2017.
[10] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real- time
object detection with region proposal networks,” in NIPS, 2015.
[11] H. Jiang and E. G. Learned-Miller, “Face detection with the faster R-
CNN,” 12th IEEE International Conference on Automatic Face &
Gesture Recognition, pp. 650-657, 2017.
[12] B. Mocanu, R. Tapu and T. Zaharia, "Single object tracking using
offline trained deep regression networks," in IPTA, pp. 1-6, 2017.
[13] O. M. Parkhi, A. Vedaldi, and A. Zisserman. “Deep face recognition”. In
British Machine Vision Conference, vol. 1, pp. 6, 2015.
[14] B. Mocanu, R. Tapu and T. Zaharia, "DEEP-SEE FACE: A Mobile Face
Recognition System Dedicated to Visually Impaired People," in IEEE
Access, vol. 6, pp. 51975-51985, 2018.
[15] R. Tapu and T. Zaharia, "A complete framework for temporal video
segmentation," 2011 IEEE International Conference on Consumer
Electronics, Berlin, pp. 156-160, 2011.
[16] R. Yin, H. Bredin, and C. Barras, “Speaker change detection in
broadcast TV using bidirectional long short-term memory net-works,” in
Proceedings of Interspeech, pp. 3827–3831, 2017.
[17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” arXiv preprint arXiv:1512.03385, 2015.
[18] W. Xie, A. Nagrani, J. S. Chung, and A. Zisserman, “Utterance-level
Aggregation For Speaker Recognition In The Wild,” in Proceedings of
ICASSP, 2019.
[19] J. S. Chung, A. Zisserman, "Out of time: Automated lip sync in the
wild", in Proceedings of ACCV, pp. 251-263, 2016.