Received 8 May 2023, accepted 20 May 2023, date of publication 24 May 2023, date of current version 1 June 2023.

Digital Object Identifier 10.1109/ACCESS.2023.3279816

Step by Step: A Gradual Approach for Dense

Video Captioning
WANGYU CHOI 1, JIASI CHEN 2, (Member, IEEE), AND JONGWON YOON 1, (Member, IEEE)
1 Department of Computer Science and Engineering (Major in Bio Artificial Intelligence), Hanyang University, Ansan 15588, South Korea
2 Department of Computer Science and Engineering, University of California at Riverside, Riverside, CA 92521, USA

Corresponding author: Jongwon Yoon (jongwon@hanyang.ac.kr)

This research was supported in part by the Basic Science Research Program through the National Research Foundation of South Korea
(NRF) funded by the Ministry of Education under Grant NRF-2022R1A2C1008743; and in part by the MSIT (Ministry of Science and
ICT), Korea, under the Grand Information Technology Research Center support program (IITP-2023-2020-0-01741) supervised by the
IITP (Institute for Information & communications Technology Planning & Evaluation).

ABSTRACT Dense video captioning aims to localize and describe events for storytelling in untrimmed
videos. It is a conceptually very challenging task that requires concise, relevant, and coherent captioning
based on high-quality event localization. Unlike simple temporal action localization tasks without overlap-
ping events, dense video captioning requires detecting multiple/overlapping regions in order to branch out
the video story. Most existing methods generate numerous candidate event proposals and then eliminate
duplicate ones using a event proposal selection algorithm (e.g., non-maximum suppression) or generate
event proposals directly through box prediction and binary classification mechanisms, similar to object
detection tasks. Despite these efforts, the aforementioned approaches tend to fail to localize overlapping
events into different stories, hindering high-quality captioning. In this paper, we propose SBS, a dense video
captioning framework with a gradual approach that addresses the challenge of localizing overlapping events
and eventually constructs high-quality captioning. SBS accurately estimates the number of explicit events for
each video snippet and then detects the boundaries context/activities, which are the details for generating the
event proposals. Based on both the number of events and boundaries, SBS generates the event proposals. SBS
encodes the context of the event sequence and finally generates sentences describing the event proposals.
Our framework is fairly effective in localizing multiple/overlapping events, thus experimental results show
the state-of-the-art performance compared to the existing methods.

INDEX TERMS Dense video captioning, event captioning, event localization, event proposal generation,
video captioning.

I. INTRODUCTION shortcomings, dense video captioning has recently emerged

The advent of large-scale video activity datasets [1], [2],
[3] has made significant progress in video captioning tasks,
which aims to generate sentences describing activity in short
videos [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14],
[15], [16]. However, real-world videos such as video surveil-
lance are generally untrimmed and contain multiple stories
that overlap in time, therefore describing the video in a
single sentence is insufficient and does not provide all the
necessary information. To overcome the above-mentioned
The associate editor coordinating the review of this manuscript and
approving it for publication was Khursheed Aurangzeb.
to describe the video in complexity and detail.
The dense video captioning task is conceptually very
challenging because it is required to detect multiple salient
temporal regions (i.e., events) that overlap in untrimmed
video. Most of the existing work adopts an approach that
divides captioning task into two sub-tasks, event localiza-
tion and event captioning. It then sequentially constructs
the two sub-tasks from the encoded video, either bottom-
up [17], [18], [19], [20], [21] or top-down [22]. Despite
these efforts, there are still limitations to accurately localizing
overlapping events in video. In particular, the bottom-up
approach (i.e., localize-then-describe) generates numerous

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

VOLUME 11, 2023 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 51949
 W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning

FIGURE 1. Comparison between the existing methods and ours for dense video captioning. Similar to human interpretation, the proposed
algorithm first scans a given video and explicitly determines the number of events. It then generates events by specifying the start and end
timestamps of the salient region based on the number of events. Finally, it describes a specific sentence for each event region. On the other hand,
existing methods generate a large number of event proposals and then remove duplicates with a han-crafted algorithm such as NMS. This makes it
difficult to detect different events (i.e., actions) in the same time period.

event proposals [23] from the extracted temporal video fea-
tures and eliminates them with a hand-crafted algorithm such
as non-maximum suppression (NMS) algorithm. A simple
temporal feature is insufficient to detect overlapping events,
and NMS algorithm tends to eliminate overlapping events
despite having different activities. In contrast, the top-down
approach, that localizes after generating a paragraph from a
video, is usually difficult to generate multiple sentences for
the same temporal region. The undetectability of overlapping
events hinders the multi-story video description.
Figure 1 shows an example of the difference between the
existing methods and our method. Existing methods are an
example of a bottom-up approach that has been widely used.
They generate a number of proposals from the video and then
consider their start and end timestamps to remove duplicates
regardless of the video’s content. In other words, when there
are multiple actions in the same temporal region, they cannot
distinguish them and leave only one. On the other hand, our
method is an approach considering video content. We first
scan the video and determine the number of actions in each
temporal region. We then estimate the exact boundary for
each event and generate final event proposals. Consequently,
ours is a flexible framework for overlapping proposals.
We observed several key factors in dense video captioning
task that affect both the event localization and event caption-
ing. Unlike simple temporal action localization [24], [25],
[26], [27], [28] without event overlapping, video frame in
dense video captioning contains multiple overlapping events,
so the event localization module needs to detect an explicit
event count. In addition, algorithms used for event proposal
selection, such as NMS, use only start and end timestamps
without considering the content, and hence they unnecessar-
ily remove information. In captioning subtask, understanding
the context between events at a high level is essential to ensure
concise, relevant, and coherent sentence generation in event
captioning.
Inspired by the observations mentioned above, we propose
a framework for dense video captioning, SBS, which is a
gradual approach from abstract to detail. As depicted in
Figure 1, procedures in SBS consists of five steps as follows.
First, it estimates the number of events in the temporal domain
from the encoded features. Unlike binary classification such
as positive or negative for actionness, the explicit number
of events is effective for localizing overlapping events as
it provides the maximum number of overlapping events in
the corresponding temporal region. Second, SBS detects the
boundaries of events in the temporal domain for additional
information in order to generate event proposals. This step
detects changes in the video scene (e.g., appearance of an
object/start and end of an action) to accurately determine the
start and end of an event. Third, SBS generates a set of event
proposals contained in the video using the number of events
and their boundaries by introducing an actionness-weighted
tIoU algorithm without any proposal selection algorithms.

51950 VOLUME 11, 2023

W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
Fourth, SBS encodes the context of generated event proposals
to ensure coherence between sentences before generating a
sentence for events. Finally, it constructs sentences using the
encoded context for the event proposals. In other words, SBS’
captioning network consists of two-level RNNs (i.e., context
encoder and sentence decoder), which guarantees sentence
fluency and coherence between sentences in the video.
The rest of the paper is organized as follows. We first
discuss related work in Section II. The proposed method and
its training methodology are described in Sections III and IV,
respectively. We evaluate our framework in Section V, and
conclude the paper in Section VI.
II. RELATED WORK
A. VIDEO CAPTIONING TASKS
1) VIDEO CAPTIONING
Traditional video captioning task aims to generate a single
sentence that describe most important event in a short video
(i.e., trimmed video). Video captioning frameworks [16],
[4] have been proposed, inspired by sequence-to-sequence
tasks [29], [30]. Their architecture consists of two parts:
an input video encoder and a sentence generator. To this
end, they employ Convolutional Neural Networks (CNNs)
and Recurrent Neural Networks (RNNs) such as Long
Short-Term Memory (LSTM), and Gated Recurrent Unit
(GRU). The video encoder first extracts visual features from
input frames and then uses RNNs to identify the temporal
dependencies. The sentence generator uses RNNs to generate
words one by one. For example, methods [4], [16] extract
visual features from a video using CNNs, encode the video
features, and then use LSTMs to decode them into natural lan-
guage. To further improve the captioning quality, subsequent
work extends the encoder-decoder structure by consolidating
temporal attention mechanism [5], hierarchical RNNs [6],
[8], LSTMs with visual-semantic embedding [7], semantic
decoder [9], [10], [11], spatial-hard attention [31], recon-
struction network [12], and reinforced adaptive attention [32].
Despite these efforts, they are limited by generating a single
sentence from a trimmed video. To address this limitation,
several works [33], [34], [35] have emerged to describe the
video in one detailed paragraph.
B. PROS AND CONS OF EXISTING METHODS
We establish several classification criteria for existing
dense video captioning frameworks to clearly identify their
strengths and weaknesses.
1) END-TO-END TRAINABLE
End-to-end trainable models [20], [36] are typically preferred
because they are simple and highly reproducible. They have
a low inductive bias, allowing the loss for the final output to
propagate back to the input. This results in high performance
with sufficient support from a large dataset. However, for
the dense video captioning task, where the dataset is lim-
ited compared to the output’s complexity, a higher inductive
VOLUME 11, 2023
bias may be a better choice for achieving high performance.
As such, most models [17], [18], [19], [21], [37] are not
designed to be end-to-end trainable for dense video cap-
tioning. Furthermore, end-to-end trainable models are not
naturally interpretable.
2) ENCODING METHOD FOR TEMPORAL DEPENDENCY
To encode sequential data, like video, the encoder must
capture temporal dependency. This is particularly important
in dense video captioning, which deals with long videos.
Many frameworks adopt LSTMs [17], [18], [19], [21], [37]
or Transformers [20], [22], [36] based on their proven effec-
tiveness in the NLP field. Prior works have shown that Trans-
formers have advantages over LSTMs. First, Transformers
capture long-term dependencies through self-attention mech-
anisms, whereas LSTMs can lose important information
in long sequences. Second, Transformers can parallelize
processing for input sequences, leading to faster process-
ing during training and inference, whereas LSTMs increase
computation time proportionally to the length of the input
sequence. However, Transformers generally require more
memory for the same performance compared to LSTMs.
3) DESIGN PARADIGM
There are various design paradigms for dense video caption-
ing, including bottom-up, top-down, and parallel decoding.
Bottom-up and top-down approaches divide dense video cap-
tioning into two subtasks: event localization and captioning,
and process them sequentially. Bottom-up approaches [17],
[18], [19], [21], [37] describe the corresponding event after
event localization, while top-down approaches [22] perform
event localization based on multiple sentences generated
about the video. Bottom-up and top-down have the limitation
that the final result is influenced by the previous module
because they process subtasks sequentially. In addition, the
bottom-up approach requires an event proposal selection
module, such as NMS. To address this, the parallel decoding
approach performs event localization and captioning simul-
taneously. However, this approach has the disadvantage that
the number of events needs to be predefined by an additional
event counter module, which has a high dependency.
C. DESIGN CHOICE
Considering the observations mentioned earlier, we adopt a
bottom-up approach for high performance within a limited
dataset while overcoming its challenges. Firstly, we focus
more on event localization in the subtask to aim for more
accurate event localization. As a result, this minimizes error
propagation to subsequent modules and eliminates the need
for an event proposal selection algorithm. We adopt a trans-
former as an encoder, considering long-term dependencies.
Our method is superior to the existing methods in three
aspects. First, SBS explicitly evaluates the number of actions,
not binary, for actionness. Most methods of adopting action-
ness for event localization evaluate to probability (i.e., binary)
51951
 W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning

FIGURE 2. Summary of procedures in SBS. (1) Video encoder: Given a video, SBS first extracts the spatiotemporal video features using the C3D networks.
The transformer encoder encodes the C3D features with long-range temporal dependency. (2) Temporal event counter: The first step for event localization
is to estimate the explicit number of events by taking the encoded video as input. (3) Temporal boundary classifier: To accurately localize the start and
end of the event, the next step is to classify the boundaries of the events in the timeline. (4) Event proposal generation: The algorithm generates event
proposals from the acquired event count and boundaries without an event selection algorithm. (5) Context-level event encoder: A context-level event
encoder composed of a single LSTM takes a sequence of event proposals as input and outputs the context-encoded hidden state. (6) Sequential event
captioner: Finally, the sequential event captioner outputs each word together with the context feature.

such as in [24]. This makes it difficult to detect multi-
ple events in the same temporal region. Second, SBS does
not rely on hand-crafted algorithms such as NMS. Since
methods [7], [17], [19], [20], [21], [38] that employ NMS
algorithms ignore the content of the video, necessary events
may be eliminated, which may omit an important descrip-
tion of the video. Finally, a two-step caption generation net-
work allows the generation of fluent and coherent sentences.
In dense video captioning, one event is not only related to
one sentence, but also sentences in a video are related to
each other in context. Thus, the sentence generation module
adopting a two-level structure following this concept makes
sentences fluent and coherent.
III. METHOD
A. OVERVIEW
An overview of SBS is depicted in Figure 2. SBS consists of
six modules: (1) video encoder, (2) temporal event counter,
(3) temporal boundary classifier, (4) event proposal genera-
tor, (5) context-level event encoder, and (6) sequential event
captioner. Given a video V , it first composes a sequence
of snippets with the video encoder. It then uses a temporal
event counter network to estimate the number of overlapping
events in the timestamp corresponding to each snippet. For
detailed localization of video segments, it detects the start and
end boundaries for each event and generates event proposals
along with the number of events. The next step is to generate
a sentence for each event proposal. To improve coherence
between sentences, we construct the captioning networks
in two steps. It converts the video segments corresponding
to each proposal into context-level representations. Finally,
it generates sentences using the context-level representations.
B. VIDEO ENCODER
For a given video V , we divide the video into non-
overlapping, fixed-length (i.e., 8 frames) snippets V = {vn }.
The goal of the video encoder is to extract the hidden states
H v for all snippets. Our video encoder consists of a CNN
backbone and a sequential data encoder(e.g., RNN). Specif-
ically, we adopt the C3D network [39] and the transformer

51952 VOLUME 11, 2023

W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
encoder [40] with multi-head attention (MA) in considera-
tion of their efficiency and performance. C3D network is a
widely used pre-trained video feature extractor and the trans-
former encoder allows encoding of long-range dependencies
(The maximum length of video included in the dataset is
2, 827 × 8 seconds. Details are described in Section V-A).
In SBS, C3D networks take each snippet as input and extract
features Fv ∈ RT ×df , where T is the number of snippets and
df is the dimension of features. We apply up/down sampling
with the nearest-neighbor interpolation algorithm to feed the
video feature Fv of dimension df to the transformer encoder
of dimension dm :
H 0v = Inearest (Fv )
Then, starting with the input H 0v , it iteratively injects the
output of the layers as many as the number of encoder layers
(multi-head attention) into the input.
   
H l+1
v = FFN 9 H l
v + MA H l
v , H l
v , H l
v (2)
FFN(x) = max (0, xW 1 + b1 ) W 2 + b2 ,
where 9 (·) represents the layer normalization function.
Finally, the output of video encoder is the output of the last
layer H v .
C. TEMPORAL EVENT COUNTER
The goal of the temporal event counter is to accurately
estimate the number of events contained in given video
snippets. Toward this, we employ multiple temporal 1D
convolutional layers to capture the dependency of tem-
poral range. Temporal convolutional layer is denoted as
Conv(cn , ck , cs ), where cn , ck , and cs are the number of
filters, kernel size, and stride of temporal convolutional layer,
respectively. As shown in Figure 2, we define three types
of temporal convolutional layers to consider different ranges
of snippets: (i) Conv(512, 2, 1), (ii) Conv(512, 10, 5) and
(iii), Conv(512, 20, 10). One snippet has a duration of about
0.27 seconds, so the kernel sizes of temporal convolutional
layers are 0.53, 2.67, and 5.33 seconds. We stack the outputs
of each layer in temporal alignment at the channel level.
The average pooling layer outputs the probability for the
number of events that correspond to the snippet. We call this
probability an event counter table. The event counter table
−1
can be denoted as Pc = {pcl,n }L,N l=1,n=0 , where L and N are
the length of input snippets and the maximum number of
overlapping events, respectively. The whole process can be
represented:
Fnarrow = Conv(512,2,1) (H v )
Fmid = Conv(512,10,5) (H v )
Fwide = Conv(512,20,10) (H v )
Fall = Stack(Fnarrow , Fmid , Fwide )
Pc = AvgPool(Fall , N ),
where Stack(·) is a function that concatenates vectors with
temporal alignment at the channel level. AvgPool(·) is an
VOLUME 11, 2023
average pooling function with input vectors and output size.
We set N to 15 by adding the probability of no event proba-
bility (i.e., background snippet) to the maximum number of
overlapping events of 14.
Our temporal event counter has clear advantages in localiz-
ing the multiple overlapping events. Specifically, it eliminates
the need for the non-maximum suppression (NMS) algorithm
by explicitly detecting the number of events (i.e., salient
region or interval) taking into account the spatiotemporal
context of the video. The NMS algorithm enforces a tradeoff
between recall and precision because it is fully hand-crafted
and operates with a single, fixed threshold [41]. Our temporal
event counter helps to determine the boundaries of events
(1)
later.
D. TEMPORAL BOUNDARY CLASSIFIER
In the previous step, we detected snippets containing salient
actions and estimated their number with the temporal event
counter, however the number of events is not sufficient to
(3) generate the complete events (i.e., there are multiple cases).
To converge these into one case, we introduce an additional
module, temporal boundary classifier. The goal of the tem-
poral boundary classifier is to detect the boundary (i.e., start,
end or both) of an event given a snippet. To this end, we adopt
a two-stream architecture (shown in Figure 2) where the
first one reuses convolutional layers with fixed parameters
of the temporal event counter, and the second one employs
multiple dense (i.e., small size of kernel and stride) temporal
1D convolutional layers. Our intuition is that the temporal
boundary classifier extracts actionness and boundary features
from a sequence of snippets to localize the exact start and end
timestamps of an event. The subsequent process is the same
as the temporal event counter, but for each snippet, it outputs
the start and end probability Pb = {(pbl,s , pbl,e )}Ll=1 . We refer
to Pb as an event boundary table.
E. EVENT PROPOSAL GENERATION
We have obtained the event counter table and event boundary
table in Sections III-C and III-D. Next, SBS uses these two
tables to generate event proposals.
Our event proposal generation starts from the event counter
table Pc and event boundary table Pb obtained in sections III
and IV. We recall them as follows:
−1
Pc = {pcl,n }L,N
l=1,n=1
Pb = {(pbl,s , pbl,e )}Ll=1
(4) Then, we create a 1D actionness table Ta from Pc . We can
compute the score value for actionness as the sum of the
(5)
product of the probability of the corresponding event count
(6) and the event count itself in each snippet slot using the
(7) following formula:
(8) "N −1 #L
X
Ta = n · pcl,n (9)
n=0 l=1
51953

Next, we generate candidate event proposals by constructing
all possible start-end pairs from the event boundary table Pb .
Ê = {(ιs , ιe ) | ιs < ιe , pbιs ,s ≥ 0.5, pbιe ,e ≥ 0.5}
where, ιs and ιe are the start and end index of the event pro-
posal, respectively, meaning all possible pairs consisting of
start and end. We then select meaningful proposals from the
candidate proposals Ê. To do this, we utilize the previously
computed actionness score Ta . Specifically, we choose the
final event proposal whose average score for all slots corre-
sponding to the proposal is greater than 1. We calculate the
actionness score for candidate proposals using the following
equation:
Pιe
i=ιs Ta [i]
S=
ιe − ιs
Our event proposal generation algorithm has several
advantages over the traditional method of generating a large
number of proposals and then removing duplicates. First,
it selects event proposals considering the content, ensuring
meaningful events are maintained. In contrast, algorithms that
ignore content, such as NMS, simply determine duplicates
based on tIoU, potentially removing important proposals.
Second, it is computationally efficient as it does not generate
a large number of event proposals. Furthermore, it generates
event proposals with just a single forward pass.
F. SENTENCE GENERATION
Our sentence generation module consists of two steps: encod-
ing the context and captioning the sequential events. Before
generating words/sentences directly from the event proposals,
we encode the context of event proposals for concise and
coherent sentences with the context-level event encoder. The
sequential event captioner then generates sentences with the
encoded context. As shown in Figure 2, we employ two
LSTMs, which are denoted as LSTMc and LSTMs for the
context-level event encoder and the sequential event cap-
tioner, respectively. The context-level event takes the video
feature Fv ∈ RT ×df corresponding to the event proposals
and encodes the context for a sequence of events, where T is
the duration of the event proposal. The sequential event cap-
tioner generates word by word while maintaining the context
event for other sentences in the same video by leveraging the
encoded context. Given t-th event proposal et , this process is
formulated as follows:
rt = LSTMc (Fv (et ), gt−1 , rt−1 )
gt = LSTMs (Fv (et ), rt , gt−1 ),
where rt is the encoded context feature of t-th event proposal,
and Fv (·) is a function that outputs the video features corre-
sponding to the input event proposal.
IV. TRAINING
The training process for our model involves three stages.
First, we jointly train the video encoder and the temporal
51954
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
event counter to estimate the number of actions for each snip-
pet in the given video. We then jointly train the video encoder
and the temporal boundary classifier to estimate whether each
(10) snippet is the background, the start, or both of an event.
Finally, we use the estimated number of actions for each
snippet and the estimated start and end of the event to generate
event proposals, and jointly train the context-level encoder
and the sequential event captioner to generate sentences for
each event using the event proposals.
A. TEMPORAL EVENT COUNTER
To train the temporal event counter, we first need to construct
the target of the event counter table from ground-truth events.
A ground-truth event is denoted as e∗ = [ts∗ , te∗ ], where ts∗ and
te∗ are the start and end timestamps, respectively. To rescale
the timestamps to the range of snippets for all event proposals,
we use the formula e∗ = [m · ts /d, m · te /d], where m and d
are the number of snippets in the video and the duration of
the video, respectively. Then, we can obtain the ground-truth
label:
O∗ = [o1 , o2 , . . . , oL ]
for the counter table by counting the number of overlapping
events for all snippets, where ol is the number of overlapping
events at the l-th snippet. To handle imbalanced samples in
the event counter table (where most samples are 1 to 3),
we adopt focal loss [42]. Finally, we train the temporal event
counter by minimizing the negative log-likelihood loss for the
ground-truth label O∗ :
L
X
LTEC = − log pcl,ol


(13)
l=1
B. TEMPORAL BOUNDARY CLASSIFIER
To train the temporal boundary classifier, we create an event
boundary table similar to the label of the event counter
table. We obtain all ground-truth events E ∗ = {e∗ k}K k=1 =
{(t ∗ ks , t ∗ ke )}K
k=1 of the video. Then, we create a ground-truth
label for the temporal boundary classifier by indicating
whether the start and end points of the event are included in
the i-th snippet with 0 or 1. Finally, we obtain the following
ground-truth labels from the start and end timestamps for the
events contained in the dataset:
(11)
B∗ = [(c∗1,s , c∗1,e ), . . . , (c∗L,s , c∗L,e )]
(12)
where c∗s is 1 if the l-th snippet contains the start of the event
and 0 otherwise, and likewise for c∗e .
We use binary cross-entropy loss (i.e., multi-label classi-
fication) because overlapping events can cause the snippet
to contain the start and end of the event together. Although
it is similar to the temporal event counter, we employ an
asymmetric loss to handle imbalanced labels (i.e., most labels
VOLUME 11, 2023
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
are negative samples). The loss function is defined as follows:
L
1X + ∗
LTBC =− [(α cl,s log pbl,s + α − (1−c∗l,s ) log(1−pbl,s ))
2 multi-head attention to 512, the number of attention heads
l=1
+ (α + c∗l,e log pbl,e + α − (1 − c∗l,e ) log(1 − pbl,e ))]
(14)
where α + and α − are hyper-parameters for positive and
negative samples, respectively.
C. EVENT ENCODER AND EVENT CAPTIONER
We use teacher forcing [43] scheme to train the event encoder
and event captioner. Given the ground-truth sentence S ∗ , the
loss function is to minimize the negative log-likelihood of the
ground-truth words w∗t,j for t-th event as follows:
X Jt
T X  
LC = − log p w∗t,j | w∗t,<j , Fv (et ) (15)
t=1 j=1
where j and w∗t,<j are the current position in the sequence
for prediction and the preceding ground-truth words, respec-
tively. Jt is the length of ground-truth words for the t-th event.
V. EXPERIMENT
A. DATASET
We evaluated the performance of SBS on the ActivityNet
Captions dataset [17], which contains 19,994 YouTube
videos. The dataset is divided into three subsets for train-
ing, validation, and testing, consisting of 10,009, 4,917, and
4,885 videos, respectively. The videos range in length from
short to long, with minimum, average, and maximum lengths
of 1.58, 117.60, and 755.11, respectively. The number of
events per video ranges from 2 to 27, with minimum, average,
and maximum values of 2, 3.66, and 27. The lengths of the
sentences in the dataset range from 17 to 409 words, with an
average of 67.7 words.
B. METRICS
To evaluate the performance of SBS event localization,
we compare the temporal intersection over union (tIoU)
between ground-truth events and predicted events. We mea-
sure recall and precision for thresholds of 0.3, 0.5, 0.7, and
0.9. Specifically, we consider the sample to be true if the tIoU
between the two is above each threshold. For captioning per-
formance evaluation, we use three metrics: METEOR [44],
CIDEr [45], and BLEU [46]. We use publicly available evalu-
ation code1 provided by the ActivityNet Captions Challenge.
Given an event and sentence pair, we calculate a captioning
score by comparing the corresponding ground-truth sentence
if the tIoU between the predicted event and any ground-truth
event is greater than the threshold. Otherwise, the score is set
to 0.
1 https://github.com/ranjaykrishna/densevid_eval
VOLUME 11, 2023
C. IMPLEMENTATION DETAILS
The transformer of SBS’s video encoder follows the orig-
inal paper [40]. Specifically, we set the hidden size dm of
to 8, and the number of layers in the encoder to 6. The
feed-forward network has 2,048 nodes. The dropout rate for
residual blocks and attention is set to 0.1. Following previ-
ous works [21], [36], we set the hidden size of the event
context encoder and sequential captioner in the captioning
network to a single layer of 512. To prevent overfitting and
improve generalization, we use PRELU [47] and GELU [48]
as activation functions for the CNNs and fully-connected
layers for the temporal event counter and temporal boundary
classifier, which experimentally outperforms other options.
We set the epochs for each stage of SBS to 20, 20, and 30,
and train with an adamW optimizer [49] and a batch size
of 1. The focal loss hyperparameter, γ , is set to 2.0 (more
information on γ settings is in Section V-F). Several recently
proposed methods [21], [22], [38] use reinforcement learning
(RL) to further improve the captioning module. To ensure fair
comparison, we also fine-tune the context-level event encoder
and sequential event captioner using RL (based on [50]) with
the reward function METEOR.
D. PERFORMANCE COMPARISON
1) EVENT LOCALIZATION
We compared several state-of-the-art dense video caption-
ing methods to evaluate the performance of SBS. First,
we present the performance of event localization in Table 1.
Above all, SBS shows the best F1 score performance among
other methods. MFT and SDVC generate candidate event pro-
posals, then use an event selection network, such as ESGN,
to remove less significant or overlapping ones. On the other
hand, PDVC and PPVC generate proposals directly in par-
allel from a localization head composed of box prediction
and classification. In contrast, SBS has a gradual approach
to explicitly estimate the number of events, detect event
boundaries, and then generate event proposals. This approach
is more effective in localizing overlapping or short events.
Moreover, SBS can detect multiple context events in a video,
enabling detailed video descriptions (details in Section V-E).
SBS outperforms MFT by a large margin and is superior
to both SDVC and PDVC in terms of overall localization
performance and F1 score.
2) DENSE CAPTIONING
Table 2 shows the performance of state-of-the-art methods in
dense video captioning. When using ground-truth event pro-
posals, SBS achieves a remarkable improvement compared
to other methods in terms of the METEOR score, which
is a commonly used evaluation metric in the ActivityNet
Captions Challenge. When using predicted proposals, SBS
achieves comparable performance to state-of-the-art algo-
rithms on CIDEr and METEOR. Specifically, it exceeds SGR
by 5.8 on CIDEr and is only 0.02 below SGR on METEOR.
51955

TABLE 1. Performance comparison of the event localization with respect to the 4 temporal intersection of unions (@tIoU) thresholds on the Activity
captions validation set.
TABLE 2. A summary of the performance comparison using BLEU, CIDEr and METEOR on the ActivityNet validation set. We present the performances
obtained from both learned and ground-truth events. Asterisk (*) indicates methods evaluated on incomplete dataset (e.g., 80%) due to download issues.
CE and RL stand for cross-entropy and reinforcement learning, respectively.
In particular, SBS outperforms MV-GPT or Vid2Seq with
pre-trained models. These results prove that both the
context-level event encoder and sequential captioner, which
consist of two LSTMs, provide excellent captioning quality.
3) INFERENCE TIME
We compare the inference time of SBS with that of other
methods in Table 5. The inference time of SBS is 1.13 sec-
onds, faster than TDA-CG and MT, but slower than PDVC.
This can be explained by the fact that SBS does not gen-
erate unnecessary event proposals unlike previous works.
Secondly, SBS generates event proposals with only one
feed-forward without repetitive inferences for event proposal
generation. However, compared to PDVC, which decodes
captioning and localization in parallel as, SBS is inherently
inferior.
E. QUALITATIVE RESULTS
We visualize the results for two videos included in the Activ-
ityNet Captions validation set in Figure 3 to examine the
results of SBS in more detail through qualitative evaluation.
It indicates two main features that can clarify SBS. First,
SBS is highly effective at detecting events that occur at the
same time. For instance, in the first video, the red and blue
events have a temporal Intersection over Union (tIoU) of over
0.8, but SBS recognizes them as distinct events and retains
both. Algorithms that don’t take content into account, like
NMS, would remove one of the events. In the second video,
51956
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
TABLE 3. Ablation results on the ActivityNet captions validation set. TEC,
TBC, NMS stand for the temporal event counter, the temporal boundary
classifier, and non-maximum suppression algorithm, respectively. R, P,
M stand for recall, precision, and F1 score, respectively.
SBS creates temporally overlapped events and provides dif-
ferent captions, demonstrating its ability to caption multiple
stories that unfold simultaneously in the video. Second, the
1d actionness table approximately represents the number of
scene transitions and duplicate events in the video story.
In the first video, it is evident that the black event interval
(where three events overlap) has the highest actionness. In the
second video, we observe a pattern where actionness rapidly
decreases whenever the scene changes and remains consistent
when the scene is maintained.
F. ABLATION STUDY
We conduct several ablation studies on the ActivityNet Cap-
tions validation set to verify the effectiveness of SBS’s each
modules. We compare three models of modules in differ-
ent combinations as follows: (i) SBS without the tempo-
ral boundary classifier, (ii) SBS without the temporal event
counter, (iii) Full model. Specifically, model (i) generates
a large number of valid proposals from the event counter
VOLUME 11, 2023
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
FIGURE 3. Examples of qualitative result on the ActivityNet captions dataset. Sentences corresponding to the event are matched with the same color.
We also show the 1D actionness table obtained from the event counter table.
map and removes duplicates with a non-maximum suppres-
sion algorithm because information around the boundary
is ambiguous. Model (ii) generates event proposals from a
binary actionness map without inferring an explicit number
of events. Model (iii) is a full SBS using all modules. The
results are summarized in Table 3. Model (i) detects fewer
events per video due to the elimination of unnecessary event
proposals, which eventually leads to low recall. Furthermore,
ambiguous event boundaries hinder high-quality localization.
Since model (ii) generates event proposals using actionness
maps composed of positive or negative elements (threshold
is 0.5), it is difficult to detect events in overlapping or different
VOLUME 11, 2023
contexts. The full model successfully localizes events and
leads to improved performance in terms of METEOR. These
results demonstrate that both the temporal event counter and
temporal boundary classifier, which are modules for localiza-
tion of SBS, are effective in detecting multiple overlapping
events of dense video captioning.
We vary the focal loss hyperparameter, γ ∈ [0.5, 1.0,
2.0, 5.0] used in [42]. We report the F1 score of the event
localization for four trained model. The results are presented
in Table 4. With an F1 difference of up to 2.6 over a wide
range of γ , SBS shows stable results. Based on these results,
we set γ to 2.0 for the best performance.
51957

TABLE 4. Ablation study on focal loss weight γ . We report F1 scores for
tIoU thresholds of 0.3, 0.5, 0.7, and 0.9.
TABLE 5. Comparison of SBS’s inference time with existing methods.
We measure the average inference time per video on the ActivityNet
Captions dataset with a single RTX 3090 GPU.
VI. CONCLUSION
We propose SBS, a framework for dense video captioning that
takes a gradual approach, mimicking human interpretation
of video captioning from abstract to detail. SBS is designed
based on several key observations in the dense video caption-
ing task that highly affect both event localization and event
captioning. Our event localization is unique in that it uses
explicit event counters and temporal boundaries, maintaining
all necessary information even in highly overlapping events.
Unlike existing methods, SBS does not require any proposal
selection algorithms. As a result, SBS identifies events close
to the ground truth in terms of the average number of events
per video. This results in 7% higher recall than state-of-the-art
methods. SBS also uses the encoded context to construct sen-
tences while ensuring coherence between them. Experimental
results show that SBS improves METEOR by 35% (w/GT)
while providing the same performance for captioning with the
ActivityNet Captions dataset. Furthermore, qualitative results
show that SBS can detect highly overlapping events (with
more than 0.9 tIoU) with different stories, leading to a rich
description.
REFERENCES
[1] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei, ‘‘Large-scale video classification with convolutional neural
networks,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014,
pp. 1725–1732.
[2] F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles, ‘‘ActivityNet:
A large-scale video benchmark for human activity understanding,’’ in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015,
pp. 961–970.
[3] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier,
S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman,
and A. Zisserman, ‘‘The kinetics human action video dataset,’’ 2017,
arXiv:1705.06950.
[4] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and
K. Saenko, ‘‘Translating videos to natural language using deep recurrent
neural networks,’’ in Proc. Conf. North Amer. Chapter Assoc. Comput.
Linguistics, Hum. Lang. Technol., 2015, pp. 1494–1504.
[5] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and
A. Courville, ‘‘Describing videos by exploiting temporal structure,’’ in
Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4507–4515.
[6] P. Pan, Z. Xu, Y. Yang, F. Wu, and Y. Zhuang, ‘‘Hierarchical recurrent
neural encoder for video representation with application to captioning,’’
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016,
pp. 1029–1038.
51958
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
[7] Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui, ‘‘Jointly modeling embedding and
translation to bridge video and language,’’ in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4594–4602.
[8] L. Baraldi, C. Grana, and R. Cucchiara, ‘‘Hierarchical boundary-aware
neural encoder for video captioning,’’ in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jul. 2017, pp. 1657–1666.
[9] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng,
‘‘Semantic compositional networks for visual captioning,’’ in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 5630–5639.
[10] Y. Pan, T. Yao, H. Li, and T. Mei, ‘‘Video captioning with transferred
semantic attributes,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jul. 2017, pp. 6504–6512.
[11] Y. Yu, H. Ko, J. Choi, and G. Kim, ‘‘End-to-end concept word detection for
video captioning, retrieval, and question answering,’’ in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3165–3173.
[12] B. Wang, L. Ma, W. Zhang, and W. Liu, ‘‘Reconstruction network for video
captioning,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
Jun. 2018, pp. 7622–7631.
[13] F. Zhu, J. Hwang, Z. Ma, G. Chen, and J. Guo, ‘‘Understanding objects
in video: Object-oriented video captioning via structured trajectory and
adversarial learning,’’ IEEE Access, vol. 8, pp. 169146–169159, 2020.
[14] R. S. Bhooshan and K. Suresh, ‘‘A multimodal framework for video caption
generation,’’ IEEE Access, vol. 10, pp. 92166–92176, 2022.
[15] S. Li, B. Yang, and Y. Zou, ‘‘Adaptive curriculum learning for video
captioning,’’ IEEE Access, vol. 10, pp. 31751–31759, 2022.
[16] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and
K. Saenko, ‘‘Sequence to sequence—Video to text,’’ in Proc. IEEE Int.
Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4534–4542.
[17] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, ‘‘Dense-
captioning events in videos,’’ in Proc. IEEE Int. Conf. Comput. Vis. (ICCV),
Oct. 2017, pp. 706–715.
[18] Y. Li, T. Yao, Y. Pan, H. Chao, and T. Mei, ‘‘Jointly localizing and
describing events for dense video captioning,’’ in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7492–7500.
[19] J. Wang, W. Jiang, L. Ma, W. Liu, and Y. Xu, ‘‘Bidirectional attentive fusion
with context gating for dense video captioning,’’ in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7190–7198.
[20] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong, ‘‘End-to-end dense
video captioning with masked transformer,’’ in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., Jun. 2018, pp. 8739–8748.
[21] J. Mun, L. Yang, Z. Ren, N. Xu, and B. Han, ‘‘Streamlined dense video
captioning,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2019, pp. 6588–6597.
[22] C. Deng, S. Chen, D. Chen, Y. He, and Q. Wu, ‘‘Sketch, ground, and refine:
Top-down dense video captioning,’’ in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jun. 2021, pp. 234–243.
[23] S. Fujita, T. Hirao, H. Kamigaito, M. Okumura, and M. Nagata, ‘‘SODA:
Story oriented dense video captioning evaluation framework,’’ in Proc. Eur.
Conf. Comput. Vis., 2020, pp. 517–531.
[24] T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang, ‘‘BSN: Boundary sensitive
network for temporal action proposal generation,’’ in Proc. Eur. Conf.
Comput. Vis., 2018, pp. 3–19.
[25] Z. Yuan, J. C. Stroud, T. Lu, and J. Deng, ‘‘Temporal action localization
by structured maximal sums,’’ in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jul. 2017, pp. 3684–3692.
[26] F. Long, T. Yao, Z. Qiu, X. Tian, J. Luo, and T. Mei, ‘‘Gaussian temporal
awareness networks for action localization,’’ in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 344–353.
[27] T. Lin, X. Liu, X. Li, E. Ding, and S. Wen, ‘‘BMN: Boundary-matching
network for temporal action proposal generation,’’ in Proc. IEEE/CVF Int.
Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 3889–3898.
[28] P. Zhao, L. Xie, C. Ju, Y. Zhang, Y. Wang, and Q. Tian, ‘‘Bottom-up
temporal action localization with mutual regularization,’’ in Proc. Eur.
Conf. Comput. Vis., 2020, pp. 539–555.
[29] I. Sutskever, O. Vinyals, and Q. V. Le, ‘‘Sequence to sequence learning
with neural networks,’’ in Proc. Adv. Neural Inf. Process. Syst., 2014,
pp. 3104–3112.
[30] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, ‘‘Learning phrase representations using RNN
encoder–decoder for statistical machine translation,’’ in Proc. Conf. Empir-
ical Methods Natural Lang. Process. (EMNLP), 2014, pp. 1724–1734.
VOLUME 11, 2023
W. Choi et al.: Step by Step: A Gradual Approach for Dense Video Captioning
[31] A. Liu, Y. Qiu, Y. Wong, Y. Su, and M. Kankanhalli, ‘‘A fine-grained
spatial–temporal attention model for video captioning,’’ IEEE Access,
vol. 6, pp. 68463–68471, 2018.
[32] H. Xiao and J. Shi, ‘‘Video captioning with adaptive attention and mixed
loss optimization,’’ IEEE Access, vol. 7, pp. 135757–135769, 2019.
[33] H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu, ‘‘Video paragraph cap-
tioning using hierarchical recurrent neural networks,’’ in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4584–4593.
[34] Y. Xiong, B. Dai, and D. Lin, ‘‘Move forward and tell: A progressive
generator of video descriptions,’’ in Proc. Eur. Conf. Comput. Vis., 2018,
pp. 468–483.
[35] J. Lei, L. Wang, Y. Shen, D. Yu, T. Berg, and M. Bansal, ‘‘MART:
Memory-augmented recurrent transformer for coherent video paragraph
captioning,’’ in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics,
2020, pp. 2603–2614.
[36] T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo, ‘‘End-to-end
dense video captioning with parallel decoding,’’ in Proc. IEEE/CVF Int.
Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 6847–6857.
[37] V. Iashin and E. Rahtu, ‘‘A Better use of audio-visual cues: Dense video
captioning with bi-modal transformer,’’ in Proc. 31st Brit. Mach. Vis.
Virtual Conf., 2020, pp. 1–22.
[38] T. Wang, H. Zheng, M. Yu, Q. Tian, and H. Hu, ‘‘Event-centric hierarchical
representation for dense video captioning,’’ IEEE Trans. Circuits Syst.
Video Technol., vol. 31, no. 5, pp. 1890–1900, May 2021.
[39] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, ‘‘Learning
spatiotemporal features with 3D convolutional networks,’’ in Proc. IEEE
Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4489–4497.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez,
L. Kaiser, and I. Polosukhin, ‘‘Attention is all you need,’’ in Proc. 31st
Int. Conf. Neural Inf. Process. Syst., 2017, pp. 6000–6010.
[41] J. Hosang, R. Benenson, and B. Schiele, ‘‘Learning non-maximum sup-
pression,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jul. 2017, pp. 4507–4515.
[42] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, ‘‘Focal loss for dense
object detection,’’ in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017,
pp. 2980–2988.
[43] R. J. Williams and D. Zipser, ‘‘A learning algorithm for continually
running fully recurrent neural networks,’’ Neural Comput., vol. 1, no. 2,
pp. 270–280, Jun. 1989.
[44] S. Banerjee and A. Lavie, ‘‘METEOR: An automatic metric for MT
evaluation with improved correlation with human judgments,’’ in Proc.
ACL Workshop Intrinsic Extrinsic Eval. Measures Mach. Transl. Summa-
rization, 2005, pp. 65–72.
[45] R. Vedantam, C. L. Zitnick, and D. Parikh, ‘‘CIDEr: Consensus-based
image description evaluation,’’ in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2015, pp. 4566–4575.
[46] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ‘‘BLEU: A method for
automatic evaluation of machine translation,’’ in Proc. 40th Annu. Meeting
Assoc. Comput. Linguistics, 2001, pp. 311–318.
[47] K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Delving deep into rectifiers:
Surpassing human-level performance on ImageNet classification,’’ in Proc.
IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 1026–1034.
[48] D. Hendrycks and K. Gimpel, ‘‘Gaussian error linear units (GELUs),’’
2016, arXiv:1606.08415.
[49] I. Loshchilov and F. Hutter, ‘‘Decoupled weight decay regularization,’’
2017, arXiv:1711.05101.
[50] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, ‘‘Self-critical
sequence training for image captioning,’’ in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jul. 2017, pp. 7008–7024.
[51] W. Choi, J. Chen, and J. Yoon, ‘‘Parallel pathway dense video captioning
with deformable transformer,’’ IEEE Access, vol. 10, pp. 129899–129910,
2022.
VOLUME 11, 2023
[52] A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic,
and C. Schmid, ‘‘Vid2Seq: Large-scale pretraining of a visual language
model for dense video captioning,’’ 2023, arXiv:2302.14115.
[53] M. Suin and A. Rajagopalan, ‘‘An efficient framework for dense video
captioning,’’ in Proc. AAAI Conf. Artif. Intell., 2020, pp. 12039–12046.
[54] S. Chen and Y.-G. Jiang, ‘‘Towards bridging event captioner and sen-
tence localizer for weakly supervised dense event captioning,’’ in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021,
pp. 8425–8435.
[55] P. H. Seo, A. Nagrani, A. Arnab, and C. Schmid, ‘‘End-to-end generative
pretraining for multimodal video captioning,’’ in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 17959–17968.
WANGYU CHOI received the B.S. degree in com-
puter science and engineering from the Tech Uni-
versity of Korea, in 2017. He is currently pursuing
the Ph.D. degree with the Department of Com-
puter Science and Engineering, Hanyang Univer-
sity, South Korea. His research interests include
video streaming, mobile networking, and com-
puter vision.
JIASI CHEN (Member, IEEE) received the B.S.
degree from Columbia University, New York, NY,
USA, with internships at AT&T Labs Research,
Florham Park, NJ, USA, and NEC Labs America
Inc., Princeton, NJ, USA, and the Ph.D. degree
from Princeton University, Princeton. She is cur-
rently an Associate Professor with the Department
of Computer Science and Engineering, University
of California at Riverside, Riverside, CA, USA.
Her current research interests include edge com-
puting, wireless and mobile systems, and multimedia networking, with a
recent focus on machine learning at the network edge to aid augmented
reality (AR)/virtual reality (VR) applications. She was a recipient of the
Hellman Fellowship and the UCR Regents Faculty Fellowship.
JONGWON YOON (Member, IEEE) received the
B.S. degree in computer science from Korea Uni-
versity, in 2007, and the M.S. and Ph.D. degrees
from the University of Wisconsin–Madison, in
2012 and 2014, respectively. He is currently
an Associate Professor with the Department of
Computer Science and Engineering, Hanyang
University, South Korea. He is also the Founding
Director of the Intelligent Machines Laboratory
which broadly focuses on research in machine
learning, artificial intelligence, mobile systems, and HCI. He was a recipient
of the Lawrence Landweber Fellowship during his Ph.D. study.
51959