Neurocomputing 405 (2020) 247–258

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

Reducing human efforts in video segmentation annotation with

reinforcement learning
} rincz a,⇑
Viktor Varga a, András Lo
a
Faculty of Informatics, Eötvös Loránd University, Pázmány Péter sétány 1/A, 1117 Budapest, Hungary

a r t i c l e i n f o a b s t r a c t

Article history: Manual annotation of video segmentation datasets requires an immense amount of human effort, thus,
Received 26 September 2019 reduction of human annotation costs is an active topic of research. While many papers deal with the
Revised 18 December 2019 propagation of masks through frames of a video, only a few results attempt to optimize annotation task
Accepted 22 February 2020
selection. In this paper we present a deep learning based solution to the latter problem and train it using
Available online 8 May 2020
Reinforcement Learning. Our approach utilizes a modified version of the Dueling Deep Q-Network shar-
Communicated by Junwei Han
ing weight parameters across the temporal axis of the video. This technique enables the trained agent to
select annotation tasks from the whole video. We evaluate our annotation task selection method by
Keywords:
Video segmentation
means of a hierarchical supervoxel segmentation based mask propagation algorithm.
Interactive annotation Ó 2020 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://
Reinforcement learning creativecommons.org/licenses/by/4.0/).

1. Introduction In this paper, we attempt to learn annotation task selection

Over the last few years, many large scale computer vision data-
sets [1,2] were released catalyzing the evolution of research areas,
like image classification and object localization in images. How-
ever, video datasets with dense, pixelwise segmentation annota-
tion are still rare and most of them are small scale or lacking
variations. The recently published Youtube-VOS [3] dataset is the
first large-scale video segmentation annotation dataset, but the
majority of clips included in it do not exceed a few seconds in
length. The Densely Annotated VIdeo Segmentation (DAVIS) [4,5]
dataset is a very popular database containing close to a hundred
short clips. Many previous papers evaluate different models on
the SegTrack dataset [6]. SegTrack consists of six very short
sequences.
A key factor in the production of annotated video datasets is the
budget for human annotation costs. Although numerous video
propagation techniques automatically generate annotation for
the whole sequence from a few annotated images only [7,8], in
case of dense, pixelwise annotation, automatically propagated
masks degrade quickly, thus human supervision is essential. Also,
the cost of human annotation for drawing bounding boxes or
points in an image are orders of magnitude lower than the costs
of providing exact pixelwise labeling. Consequently, reducing the
need for human effort in video segmentation annotation is highly
relevant.
⇑ Corresponding author.
}rincz).
E-mail address: lorincz@inf.elte.hu (A. Lo
policies for human annotators to reach optimal segmentation qual-
ity from a given budget of human effort in dense video segmenta-
tion labeling. To this end, we present a reinforcement learning
based solution. We evaluate our method on the DAVIS and Seg-
Track datasets and compare it to other approaches in a quantitative
manner. Our contribution is a modification of the Deep-Q-Network
architecture which learns superior annotation task selection policy
compared to the state-of-the-art. The software code and the
trained model will be released1.
2. Related work
2.1. Video object segmentation
Approaches in video segmentation can be categorized into
either (semi-) supervised or unsupervised methods. Supervised
and semi-supervised methods are usually trained on part of the
target dataset or pretrained on closely related datasets in terms
of content and image features. A majority of recent results utilize
deep learning as well [7,9–11]. However, depending on the imple-
mentation, deep learning based image segmentation models may
easily become semantically biased and might exhibit significant
performance loss when evaluated on datasets of different nature.
Unsupervised methods include the partitioning of images or videos
to superpixels, or supervoxels respectively [12].
1
https://github.com/vvarga90/rlseg

https://doi.org/10.1016/j.neucom.2020.02.127
0925-2312/Ó 2020 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
248 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo
Stutz et al. [13] present a comprehensive analysis and a set of
benchmarks for the superpixel literature. Chang et al. [14] solve
temporal propagation of superpixels by modeling flow vectors of
regions with bilateral Gaussian processes. Grundmann et al. [15]
utilize a minimum spanning tree algorithm to build a tree of voxels
to obtain a hierarchical segmentation. Although these classical
superpixel/supervoxel methods are surpassed by deep learning
based solutions in terms of performance, still, they are less suscep-
tible to overfit to high-level semantic content of the training data.
We note that hierarchical superpixel/supervoxel algorithms are
ideal to form the base of interactive segmentation annotation,
since they provide image/video partitioning at an arbitrary level
of granularity.
2.2. Interactive segmentation
Numerous papers discuss the potential of hybrid annotation
systems where human and machine work together to speed up
the annotation process. A well known example is the GrabCut algo-
rithm [16] which performs image segmentation by means of
energy minimization and iterative corrections of the user. Several
works aim to exploit temporal context in image sequence annota-
tion. Nagaraja et al. [17] compute temporal point trajectories from
optical flow and assign partial labeling to trajectories intersecting
user inputs in the form of scribbles. They estimate dense segmen-
tation labels using energy minimization.
Some results deal with the selection of the most rewarding
annotation task. Fathi et al. [18] compute a superpixel-graph based
representation of videos and estimates a complete labeling from
partial annotations by finding the harmonic solution. They define
an uncertainty measure to select the next annotation task for the
user, iteratively. Vijayanarasimhan and Grauman [19] analyze sev-
eral algorithms based on optical flow and Markov Random Fields to
propagate partial annotations, estimate propagation errors
between frames and select the frame with the most expected
improvement to be the next annotation task. Luo et al. [20] gener-
ate segmentation proposals and estimate the label change caused
by correctly annotating them. They search for an optimal query
strategy to annotate proposals using energy minimization.
2.3. Reinforcement learning
Reinforcement learning (RL) has been gaining strength in recent
years due to the successful joining of deep networks and RL by
Mnih et al. [21]. Beyond the popular RL benchmark Atari game
environment, numerous computer vision problems have been
tackled by RL. These include, but are not limited to, video content
summarization [22], object localization [23], tracking [8,24] and
video object segmentation [25].
Several papers deal with the optimization of human-computer
collaboration via the interactive annotation of images or videos
using RL. Russakovsky et al. [26] selects tasks in a crowdsourced
image annotation setting. Song et al. [27] improve the GrabCut
algorithm by generating some of the seed points with a convolu-
tional net based Deep-Q-Network (DQN) agent. Acuna et al. [28]
evaluate their image segmentation network in a human-in-the-
loop setup, too. Rudovic et al. [29] integrate reinforcement learning
into an active learning framework. Their agent decides whether
labeling of a data point is requested from the user or not. Similarly,
our approach utilizes reinforcement learning to select annotation
tasks, but our agent is not restricted to proceed through each frame
in a sequential order. On the contrary, the modified Deep-Q-
Network presented in this paper, estimates state-action values
for each frame at once. Therefore, the selection of each annotation
task is carried out with a single inference phase of our network. For
details, see Section 3.4.
3. Method
In our work, we utilize deep learning to reveal potential errors
in segmentation and label predictions based on image features,
optical flow estimation and changes in occlusion. We formulate
the learning in the Markov Decision Process (MDP) framework, in
order to find cost effective strategies in annotation task selection
for the human annotator. We employ a hierarchical supervoxel
based segmentation label prediction algorithm to predict a com-
plete annotation of the video from partial user input and use it
with our task selection agent to achieve a significant reduction of
human effort in the dense annotation of videos.
Fig. 1 depicts our novel scheme that reduces human effort dur-
ing interactive video segmentation annotation. Below, we outline
our proposed procedure. We carry out the three steps repeatedly.
1. Annotation prediction: Initially, ‘background’ labels are
assigned to all pixels. After correcting annotation is added by the
user, the label propagation algorithm re-estimates its label predic-
tions for the whole video. Details of this step are elaborated in
Section 3.1.
2. Agent selects the next annotation task: State representation
of the MDP is made up of features extracted from the video and the
updated annotation prediction. Given the state, the RL agent
selects the next annotation task to be given to the human annota-
tor: the index of the next frame to annotate (f t ) and a hierarchical
level (ht ) corresponding to the underlying hierarchical supervoxel
segmentation.
3. Annotation correction by user: The user is presented the
level ht superpixel segmentation of frame f t . They are asked to
make corrections of the predicted superpixel labels in the given
frame: the user may change the label of each superpixel by clicking
on it. The user continues to make corrections in the frame until
they consider the superpixel labeling correct. The annotation pro-
tocol is detailed in Section 3.2. Examples of label correction are
shown in Fig. 2.
In the process described above, the annotation prediction algo-
rithm is responsible for the estimation of a complete dense anno-
tation of the whole video by propagating the partial annotations
and corrections provided by the user. The RL agent is trained to
select optimal annotation tasks for the human at any time, by esti-
mating the scale of potential discrepancies between the predicted
annotation and image-derived features.
3.1. Hierarchical supervoxel based segmentation propagation
We formulate superpixel and supervoxel hierarchies. To start
with, it is important to clarify that we refer to two independent con-
cepts of time in the following sections. We differentiate between
the temporal axis of a video (corresponding symbols include f ; F,
etc.) and the iterations of the RL framework (with corresponding
symbols t; T, etc.). A video can be represented as X ¼ ðP1 ; . . . ; PF Þ
– a 3-dimensional grid, where Pf stands for the representation of
frame f, a single image – a 2-dimensional grid of pixels.
We define the hierarchical supervoxel segmentation as
n o
V ¼ V 0 ; V 1 ; . . . ; V H , where V h is the level h hierarchical segmen-
tation of the video. The supervoxel segmentation of a video in a
single hierarchical level can be then formulated as V h ¼
n o
v h1 ; v h2 ; . . . ; v hNh , a partition of the set of voxels in the video. We
also define superpixel hierarchies in the following way:
U f ;h ¼ V h \ Pf denotes the level h superpixel segmentation of
frame with index f. Similarly to supervoxels, we define a superpixel
segmentation as a partition of the set of pixels in frame
n o
f : U f ;h ¼ u1f ;h ; uf2;h ; . . . ; ufM;hf ;h .
 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo 249

Fig. 1. Overview of our method. 1. A complete labeling of the video is predicted from the partial labeling provided by the user in past iterations. 2. RL agent selects the next
annotation task for the user. 3. User executes the assigned task by correcting mislabeled segments of the label predictions.

Fig. 2. Label propagation exploiting hierarchical supervoxel structure. The two rows show label propagation to different frames of the video after user input was given at a
high (top row) and at a lower hierarchical level (bottom row) in frame 20 of the soapbox DAVIS sequence. The first column contains the actual click points of the simulated
user (zoomed in). Note, that the number of click points form the cost in our procedure. Human work may be spared by allowing annotation with scribbles, as shown in the
second column. The remaining two columns to the right display the effect of propagating the user annotation to frames 22 and 80. It is clearly visible that user input in higher
hierarchical levels comes with significantly lower cost and is propagated further in time. However, lower hierarchical level inputs increase granularity of the propagation
locally. Best viewed in color.

The hierarchical nature of the superpixel and supervoxel seg- tively merges adjacent spatio-temporal regions using a sorted glo-
0
mentations is formulated in the following way: let h; h be bal edge weight list. The edge weights are a function of color and
0 optical flow histogram distances. The result of GBH is a minimum
indices of hierarchical levels, with h P h. Then,
F F
9k1 ; . . . ; km : v hj ¼ v hki , where denotes the disjoint union spanning tree constructed from the region adjacency graph. Opti-
0

i¼1;...;m cal flow is estimated with Flownet 2 [31].

of the family of its sets. Similarly
0 F
9k1 ; . . . ; kn : ufj ;h ¼ ufk;h
i
for any frame f. We will refer to pixel
i¼1;...;n
level segmentation as segmentation level 0. The distinction
between pixels and voxels is important, since our algorithm is
based on supervoxels, whereas user actions concern superpixels
of a frame chosen by the RL agent. However, to be able to use
superpixels and supervoxels in the same framework, we handle
superpixels as 3D regions in X with an extent of 1 frames along
the temporal axis. Therefore, a supervoxel or superpixel is an
ancestor of another supervoxel or superpixel if the former one
contains the latter one as a subset.
In our approach, we use Contour-Constrained Superpixels (CCS)
by Lee et al. [30] to acquire a base level supervoxel segmentation of
the video. We build a hierarchical segmentation of the video by
applying the method proposed in Efficient Hierarchical Graph-
Based Video Segmentation (GBH) by Grundmann et al. [15] on the
output of the CCS segmentation method. The GBH algorithm itera-
for superpixels: Since GBH was designed to handle pixel-level adjacency graphs,
an obvious choice would be to obtain the base level segmentation
of the video with this algorithm as well. However, we found that
the low-level segmentations computed by CCS are much better
in terms of boundary accuracy at similar supervoxel counts. Never-
theless, the exact implementation of the hierarchical supervoxel
method utilized is indifferent to the main thesis of our paper:
the configuration presented here may be replaced by other hierar-
chical supervoxel segmentation techniques.
CCS is parametrized by the approximate number of superpixels
in a single frame. We set this parameter to 2048, for which super-
voxel counts in the resulting video segmentation are between 3000
and 20,000 depending on the length and the amount of changes in
the video. In the following sections of the paper, we will refer to
the voxel level segmentation of the video (where each voxel forms
an individual segment) as hierarchical level 0. We define hierarchi-
cal level 1 as the base level segmentation produced by CCS. Higher
hierarchical segmentation levels are created from level 1 segmen-
250 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo
tation by the iterative merging procedure with GBH. We refer to
hierarchical level h P 2 as the segmentation given by the iterative
merging algorithm stopped at segment count jV h j ¼ bjV h1 j=cc. The
number of supervoxels at hierarchical level i is denoted by jV i j. We
found c ¼ 3:7 to be ideal for the DAVIS and SegTrack datasets. Note
that the ideal value of this constant is highly dependent on the
temporal extent of supervoxels present in the base level
segmentation.
Next, we formulate the labeling of a video. We follow the label-
ing conventions of the DAVIS dataset: our label set contains a sin-
gle background label LBG and multiple foreground labels. Let
 
L ¼ LBG ; L1 ; . . . ; LjLj1 be the set of labels. Our task is to give a
complete labeling of the video, by assigning a label to each pixel:
l : V 0 ! L.
Annotations already given by the user in an episode can be for-
n o
malized as a set of triplets: K ¼ f t ; ht ; lf t ;ht : U f t ;ht ! L . As
described in the previous section, the user is assigned the annota-
tion task ðf t ; ht Þ by the RL agent, for which the user provides the
labeling of all superpixels in level ht of frame f t : the lf t ;ht mapping.
The annotation propagation algorithm can be given in the follow-
ing way:
Let ufj ;0 be a pixel in frame f. If no supervoxel ancestor of the
given pixel exist that contains user-labeled superpixels as a subset,
 
then l ufj ;0 ¼ LBG . Otherwise, let v hk be the lowest level ancestor of
ufj ;0 which contains a user-labeled superpixel as a subset. In this
case, we choose the temporally nearest of such labelled superpix-
0 0
   0 0
els. Let this superpixel be ufi ;h . Therefore l ufj ;0 ¼ l ufi ;h , in other
words we assign the label of the selected superpixel to the pixel
ufj ;0 being discussed. The effects of propagating user input given
in different hierarchical levels are shown in Fig. 2.
3.2. Human annotation protocol
The human annotator is asked to make corrections to the pre-
dicted annotation at a given frame f t , in a given hierarchical level
ht . The annotator may carry out the label corrections by modifying
the assigned labels to each superpixel at level ht if they are
believed to be mislabeled. We decided to simulate the role of the
human annotator during the training and evaluation of our
approach. The protocol of our simulated human annotator can be
specified in the following way: each superpixel should carry the
Fig. 3. Architecture of the modified Dueling Deep Q-Network. F designates the length of the temporal dimension of the video, i.e. the number of frames: the action-advantage
prediction branch of the network shares its weights along this axis. C; C 0 ; C V ; C A denote the number of image channels. H stands for the number of actions available in each
frame. The index of time step (t) in the MDP episode is fed into both branches of the network.
label of the most frequent pixel label found in the given superpixel
(i.e. plurality voting). In case this does not hold for a superpixel, the
simulated human annotator corrects the label. We utilize the
ground truth labels in order to simulate user input. However,
real-life human annotators cannot be expected to perform label
selection based on plurality voting of pixel labels with absolutely
zero error. For this reason, we also carry out experiments simulat-
ing the noisy selection of superpixel labels in Section 4.4 and show
that our approach is robust to minor human errors.
In several works dealing with human segmentation annota-
tions, other ways of human input are investigated, such as drawing
scribbles to mark coherent regions in one action [17,32]. While
results show that in some cases segmentation annotation using
scribbles requires less effort, we chose to stick with simple mouse
clicks instead, which may serve quantitative comparisons in the
future.
3.3. Markov decision process
In our work, an annotation episode is modelled with a Markov
Decision Process (MDP). The MDP is defined by states st 2 S,
actions at 2 A, the state transition function P : S  A  S ! R
and the reward function R : S  A ! R. In each step, the agent
receives the current state, which stores downsampled pixelwise
video features, the current state of annotation and the time index
in the episode, t. The agent then takes an action, selecting the sub-
sequent annotation task to be queried from the user. The action is
represented by ðf t ; ht Þ where f t is the frame index of the video and
ht is the hierarchical level in which the user carries out the labeling
task. In other words, the algorithm may ask for annotations at
specific levels in the hierarchy. During the training process, user
labeling is computed from the ground truth segmentation labels
in accordance with the human annotation protocol, detailed in
the previous subsection. Finally, the complete labeling of the video
is estimated from the partial annotations already present. The pre-
dicted annotation is compared to the ground truth labeling using
the mean Jaccard-index (a.k.a. intersection-over-union or IoU).
3.4. Model architecture
To predict state-action values for each frame and each hierar-
chical level, we use a somewhat modified version of the Dueling
Double Deep Q-Network (DDDQN) [33]. Since we intend to learn
to approximate the same Q-values independently from the actual
 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo 251

temporal position in the video, we utilize layers with translation
invariance along the temporal axis in the action-advantage estima-
tion branch of the network. While the state value prediction
branch does not share weights along the temporal axis, we only
use this branch during the training of the network, as having the
action-advantage estimation Aðs; aÞ is enough to select the optimal
action. This procedure enables us to use the weights of the trained
model to initialize differently shaped models to work with video
clips of arbitrary length. The architecture of our modified Deep
Q-network can be seen in Fig. 3. The input of our model is the state
representation, which contains multiple channels of 3D pixel
(voxel) data. Consequently, at the front part of our model, multiple
3D convolutional and max-pooling layers are placed alternately.
The pooling layers of the shared, frontal part of the network only
reduce the dimensionality along the spatial axes and leave the
temporal axis untouched. The pooling layers in the action-
advantage prediction branch are of similar nature. The output of
the last pooling layer in this branch is reshaped to two axes, keep-
ing the temporal dimension unchanged. The following layer is a
bidirectional LSTM layer having recurrent connections along the
temporal axis of the video. This layer provides long-term informa-
tion propagation in time. The outputs of the bidirectional recurrent
layer are concatenated for each frame independently. The final
layer is fully connected within individual frames of the video and
learns the same transformations with tied weights along the tem-
poral axis. ReLU activations are placed after each convolutional
layer. The output of the action-advantage prediction branch is
shaped F  H, where F denote the number of frames in the video,
while H stands for the number of actions per frame, i.e. for the
number of hierarchical levels. H is chosen to be 4 in our experi-
ments. The structure of our action-advantage prediction branch
bears a resemblance to the Deep Recurrent Q-Network described
in [34], but in our case all weights that take part in the estimation
of Aðs; aÞ are shared along the frames of the video. see Fig. 4.
The state value estimation branch of the network is similar to
the original DDDQN, featuring two 3D max-pool and one 3D con-
volutional layer. The following two layers are ordinary fully con-
nected layers with ReLU activations.
Additionally, both branches of the network receive the index of
the time step in the episode as input.
The predictions of the two branches: Aðs; aÞ and V ðsÞ are merged
following [33]:
!
1 X
Q ðs; aÞ ¼ V ðsÞ þ Aðs; aÞ  Aðs; a0 Þ ð1Þ
jAj a0 2A
3.5. State representation
The state consists of image features and information about the
progression of the annotation process. The state comprises the
index of the current time step in the MDP episode and features
derived from the video in the shape of F  X  Y  C, where F
denotes the number of frames in the video, X; Y denote the size
of the downscaled frame, while C designates the number of image
channels. For subsequent episodes, F may vary, since all weights of

Fig. 4. Qualitative results on the DAVIS breakdance and soapbox sequences showing error in label prediction and correcting user input after 1, 2, 3, 4, 5 and 10 iterations in
frames selected for user annotation. Colored dots represent user clicks. Error rates for both sequences were above the dataset average. Note, that the number of click points
form the cost in our procedure. Human work may be spared by allowing annotation with scribbles, as shown in Fig. 2. Progression of labeling in consecutive frames of the
sequences are shown in Figs. 5 and 6. Best viewed in color.
252 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo
our neural network are shared along this dimension. X and Y are
chosen to be 32 and 56 respectively. During our experiments C
was 9. The details of the individual Ic image channels are the
following:
I0 ; I1 ; I2 ; I3 store the downscaled optical flow images: dx; dy com-
ponents for forward and backward flow, respectively. We obtained
the optical flow predictions from Flownet v2. I4 ; I5 contain the bin-
ary images of the estimated changes in occlusion computed from
the optical flow predictions. A pixel is considered occluded/disoc-
cluded when the forward/backward optical flow have no vectors
pointing to that pixel location or near it. I6 contains the boundaries
between similarly labeled regions. A pixel is considered a boundary
pixel, if it shares a side with at least one differently labeled pixel.
The image channels specified above, may give the neural network
information about the scale of potential deterioration during anno-
tation propagation, even when annotation input is not yet present
in the temporal vicinity of a certain frame.
In case the video lacks fast movements and quick changes in
occlusion, the annotation propagation may go on successfully for
several seconds. Although the temporal recurrent connections in
the action-advantage prediction branch of the network enable
long-term temporal information propagation, still, the introduc-
tion of features that encode the distance of user annotation
improved our results. Hence, we specify channels I7 and I8 as
follows.
I7 encodes the hierarchical distance dhier of a pixel from user
 
annotations. Let dhier ufj ;0 be the level of the lowest level ancestor
supervoxel of ufj ;0 which contains user labelled superpixels as a
subset.
Finally, I8 encodes the level 1 temporal distance dtemp from user
 
annotations. Let dtemp ufj ;0 be the temporal distance of the tempo-
rally closest labelled level 1 superpixel from the ufj ;0 pixel which
share a level 1 supervoxel ancestor.
3.6. Training the agent with deep reinforcement learning
To train an agent, we utilize the modified DDDQN described in
Section 3.4. Our network learns to estimate the discounted sum of
the remaining rewards in the episode, i.e. the Q-values, for each
possible action. The action space can be factorized into two inde-
pendent dimensions: index of frame and the hierarchical level of
the annotation task. The training of the network is executed in a
similar fashion to ordinary Deep Q-Networks with the Double
b targetNN ðs; aÞ and Q
b onlineNN ðs; aÞ denote
DQN technique [35]. Let Q
the Q values estimated by the target network and the network
being trained. The Q-learning target and the DQN loss function
can be expressed with s; a; s0 as:
 
b targetNN s0 ; arg min a0 Q
Q ðs; aÞ ¼ r ðs; aÞ þ c Q b onlineNN ðs0 ; a0 Þ
 2 
b ðs; aÞ
J ðhÞ ¼ E Q ðs; aÞ  Q
where h denotes the parameters of the networks.
The gradient backpropagation is analogous to that of regular
Deep Q-Networks: gradients are propagated backwards only from
the output corresponding to selected action a. The network esti-
mates a state-action value for each action, from which an e-
greedy policy selects the action to be taken. To avoid training our
network with correlated batches, we use an experience replay buf-
fer to store past experiences of the agent: sampling is prioritized by
the magnitude of the TD-error [36].
The reward function we use was inspired by the exponential
reward function suggested by Song et al. [27]. We aim to balance
between rewarding the improvement of segmentation accuracy
(improvement in Rexp ) and penalizing annotation cost (Costt ).
(
1; if frame index f t has already been selected by the agent
R¼ 
werr  Rexp ðMt ; GÞ  Rexp ðM t1 ; GÞ  wac  Cost
j ;
t
otherwise;
ð4Þ
where
expxq  1 Jmean ðM; GÞ
Rexp ðM; GÞ ¼ ; x¼  : ð5Þ
expq  1 Jmean G0 ; G
In the definition of the reward function, Jmean stands for the
mean Jaccard-index (also known as the mean intersection-over-
union for pairs of masks for each label present). M t ; G denote the
predicted mask for the whole video in time step t of the MDP epi-
sode and the ground truth mask respectively. Costt stands for the
annotation cost of the selected annotation task in time step t.
The reward for a given improvement in segmentation accuracy
increases exponentially as accuracy gets close to the upper limit.
With this, we model the exponential increase in human efforts,
as the labeling quality improves during the annotation process.
The exponential dependence can be modulated by parameter q.
In our studies, typical q values are between 3 and 10. Depending
on the border accuracy and the average temporal extent of the
supervoxel segmentation, the maximum achievable Jmean and
the annotation costs may greatly vary between videos. The expo-
nential nature of the reward function further amplifies the vari-
ance in expected rewards which impacts the stability of the
training. To fight off the instability, we introduce G0 and j which
may differ from video to video as detailed below. G0 stands for
the predicted mask when all frames of the video are annotated at

the base hierarchical level (level 1). Thus, Jmean G0 ; G determines
the minimum achievable annotation error for a specific video.
The role of x is to normalize Rexp to 1 when the minimum achiev-
able error is reached for a specific video. Similarly, j designates the
number of supervoxels in the base level segmentation (level 1). As
mentioned before, the number of supervoxels in CCS generated
segmentations vary significantly depending on the magnitude of
movement in the videos and j compensates for this.
4. Experiments and evaluations
In this section, first, we present details about the training of our
network. Later, we train and evaluate our method on the DAVIS
datasets and compare it to the semi-supervised state-of-the-art.
Additionally, we evaluate our trained model on the unseen Seg-
Track dataset and compare its accuracy to the state-of-the-art in
interactive video segmentation as well. Furthermore, we compare
the policy learned by the reinforcement learning agent to baseline
ð2Þ policies to validate our RL based approach. Finally, we simulate
potential mistakes made by human annotators and show that
our approach is robust to minor errors in following the annotation
ð3Þ protocol appropriately. We use the mean Jaccard-index to measure
our prediction for a given annotation budget.
4.1. Details of the training and the prediction process
We train our network on the training set of the DAVIS 2017
dataset. We split the training set into two parts, containing 40
and 20 videos respectively. We use the second part as a validation
set while training on the first part of the dataset. The subset of 20
videos of the DAVIS training set we use for validation should not be
confused with the DAVIS validation set, which we use only for final
evaluation purposes. We initialize a network with an input video
length F ¼ 40 and use random slices of videos for each episode
 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo 253

as the network input. The training process starts with an experi- Table 1
ence collection phase of 20,000 steps using a random policy with- Comparison of our method with state-of-the-art semi-supervised methods on the
DAVIS validation sets. Binary and multiclass segmentation results were reached on
out training the network. Following this, the network is trained the DAVIS 2016 and 2017 validation sets respectively. Mean Jaccard-index was used
while generating experiences. The policy is parametrized with an as metric. Note, that all cited methods receive the first frame ground truth annotation
e value gradually decreasing from 0:5 to a minimal value of 0:2. as input, while our method continuously improve the prediction using approximate
The discout rate c is set to 0:9. Episode length is fixed to 20 anno- annotations from user input. The budget for our method is given in number of clicks
per total frame count of all videos.
tation task selection iterations. The network is trained using an
RMSprop [37] optimizer with a learning rate set to 5  104 . The Method Binary Multiclass
segmentation segmentation
weights of the target network are updated with a soft update rate
(mean J) (mean J)
of 104 .
PReMVOS [38] 0.849 0.739
The number of hierarchical levels used (H) was set to 4. The fol- OSVOS-S [39] 0.856 0.647
lowing reward parameters were found to be optimal: OnAVOS [40] 0.861 0.616
werr ¼ 10; wac ¼ 70; q ¼ 8. Further analysis of these parameter val- CINM [41] 0.834 0.672
ues can be found in Section 4.3. OSVOS [7] 0.798 0.566

While the training process is carried out using random slices of
videos with fixed length, the prediction phase does not have such
restriction on the input. During inference, the branch of the net-
work responsible for the state-value estimation can be removed.
The remaining part of the network shares all its weights along
the temporal axis of the video, therefore the temporal extent of
the input is only limited by hardware capacities.
We measured the execution time of our method using a GTX
1080 graphics card and a Xeon E5-2620 v4 CPU. While the prepro-
cessing steps to generate the hierarchical segmentation and to
compute the optical flow may take several seconds per frame,
these computations can be done in advance. During the actual
interactive annotation procedure, the human annotator have to
wait after completing the labeling of a frame, before being assigned
the consecutive task. The waiting time in this case is on the order
of a few hundred milliseconds. The graphics card could load video
clips of more than 3000 frames at once. Longer videos may be sim-
ply cut to slices of appropriate length.
4.2. Interactive segmentation results
4.2.1. Comparison with the state-of-the-art
First, we compare our method quantitatively to the state-of-
the-art on the semi-supervised benchmarks of the DAVIS dataset
in Table 1. We cannot offer a straightforward comparison between
our method and semi-supervised methods, due to the underlying
methodological differences: the semi-supervised protocol of the
DAVIS benchmarks reveal the ground truth labeling of the first
frame to the evaluated methods, while our method iteratively
receives approximate corrections of the predicted labels, without
ever obtaining the exact ground truth labeling of a full frame. Still,
some comparisons seem worth as they give insights about the use-
fulness of our method.
In Table 1, we show our results in two benchmarks. The values
in the binary segmentation task correspond to the mean Jaccard-
index of the predicted binary masks over all frames of the DAVIS
2016 validation set. The multiclass segmentation task refers to
the evaluation of the models on the DAVIS 2017 validation set
using the mean Jaccard-index over all frames of the dataset and
with all the different labels present in each frame. Our results
improve as the budget for user annotation actions increases.
Clearly, state-of-the-art, deep convolutional network based semi-
supervised methods learn quickly to segment a single moving
object in front of a more or less static background. Nonetheless,
their performance plummets as multiple moving objects appear
with individual labels. On the contrary, our method presents an
approximately similar accuracy both for binary and multilabel seg-
mentation tasks. We attribute this to the main difference in our
training scheme compared to other approaches. In our case, we
use a trained model to estimate the label prediction error, while
the label prediction itself is computed by a black box algorithm.
Ours, 1 clicks per fr. budget 0.661 0.594
Ours, 2 clicks per fr. budget 0.710 0.685
Ours, 5 clicks per fr. budget 0.793 0.781
Ours, 10 clicks per fr. budget 0.852 0.837
to speed up the annotation process. Although, labeling multiple
adjacent regions by a single mouse gesture is indeed time-
efficient, the lack of a formal definition of a scribble makes compar-
isons hard. A few approaches try to quantify annotation costs by
measuring the number of frames annotated in a video. However,
in practice, the accurate pixelwise labeling of a frame could take
several hours, while for the majority of applications, reaching
100% IoU is needless. Therefore, annotation costs of labeling a
frame may vary several magnitudes depending on the required
level of accuracy.
To facilitate the comparison of our approach with future results,
we chose an annotation technique popular in image segmentation
since the introduction of the GrabCut algorithm [16]: we provide
hints (or seeds) to the segmentation algorithm by means of sug-
gested mouse clicks in images. The number of mouse clicks needed
to reach certain accuracy levels is an unambigous measure of
human effort in annotation tasks.
We note that none of the papers available on interactive seg-
mentation feature the metric described above, we compare our
results to the state-of-the-art measuring annotation costs by
counting the number of annotated frames. We evaluated our
method on the SegTrack dataset in order to study database depen-
dence when training on one set and testing on both. Results can be
seen in Table 2. Videos in the SegTrack dataset have relatively low
resolution and some sequences, e.g. cheetah, girl and monkeydog
contain a significant amount of compression related artifacts,
which may give rise to erroneous object boundary prediction for
some algorithms and may spoil performance.
Qualitative results of interactive annotation episodes for binary
and multilabel annotation tasks are shown in Fig. 5 and Fig. 6
respectively. Ground truth masks are shown in the leftmost col-
umns. Columns to the right show the progression of the improve-
ment of predicted labels throughout multiple frames of the video.
In all illustrations presented in Figs. 5 and 6 prediction error rate
was above the dataset average.
4.2.2. Evaluation of learned policy
In the previous section, we compared our method to other
approaches in terms of the quality of the obtained segmentation
annotation. However, utilizing a random task selection policy to
assign annotation tasks to the human participant also leads to ade-
quate annotation quality after a few iterations. In view of this, we
compare the policy learnt by the RL agent to baseline policies on
the DAVIS 2017 validation set.
254 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo

Table 2
Results on the unseen SegTrack dataset after a specific number of frames were annotated by the user. Note, that in contrast to [20], our method is optimized on the cost of
annotating each frame as well. Annotation costs are given in number of clicks per total frame count of all videos.

SegTrack sequence [20] (5 fr.) Ours (5 fr.) Ours (10 fr.) Ours (20 fr.)
birdfall 0.593 0.711 0.729 0.747
cheetah 0.502 0.348 0.448 0.711
girl 0.739 0.463 0.537 0.671
monkeydog 0.454 0.631 0.729 0.774
parachute 0.821 0.897 0.898 0.901
penguin – 0.941 0.946 0.947
Mean IoU w/o penguin 0.604 0.651 0.711 0.782
Mean IoU – 0.701 0.751 0.810
Annot. cost per frame – 2.028 4.282 8.545

Fig. 5. Qualitative results on videos with above average error from the DAVIS 2016 dataset. Rows represent frames number 0, 40 and 80 of the breakdance sequence and
frames number 0, 15, 30 of the motocrossjump sequence in this order. First column contains the ground truth masks in the specific frames, while following columns contain
the predicted masks after 1, 2, 5 and 10 iterations of user annotation inputs. Best viewed in color.

The results of the comparison are shown in Fig. 7. Annotation
costs are given in number of clicks per total number of frames in
the dataset to reach a specific level of predicted label quality.
Random policy selects ðf t ; ht Þ annotation tasks randomly, but
avoids redundant queries: in case a certain frame was already
annotated in hierarchical level h, queries targeting this frame with
0
hierarchical level h P h are ignored. In Fig. 7, the average results
for 10 such random policies are shown.
Best constant policy selects frame f t to be the most distant
from all previously queried frames, the beginning and the
end of the video. We experimented with other frame selection
strategies. We found those to be inferior, including sequential
equidistant sampling and a keyframe based frame selection
policy where keyframes were chosen according to the Keyframe
policy in [19]. To obtain a strategy to select hierarchical level
ht , we evaluated all possible monotonically decreasing policies
up to an episode length of 20 steps on a subset of the DAVIS
2017 training set combined with the frame selection policy
specified above. We evaluated those performing the best, again,
on the full DAVIS 2017 training set to acquire the final policy.
The hierarchical level selection policy giving the best results
was combined with the frame selection strategy mentioned
before produced Best constant policy. Results are depicted in
Fig. 7.
MRF baseline policy is generated by the Markov Random Field
(MRF) and is driven by an active learning approach. Our baseline
measure is derived from the paper of Luo et al. [20]. We made
the cited method compatible with our hierarchical supervoxel
framework by modifying their approach at a few points. We con-
tinue to use the notation defined in Section 3.
 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo 255

Fig. 6. Qualitative results on videos with above average error from the DAVIS 2017 dataset. Rows represent frames number 0, 15 and 30 of the shooting sequence and frames
number 0, 40, 80 of the soapbox sequence in this order. First column contains the ground truth masks in the specific frames, while following columns contain the predicted
masks after 1, 2, 5 and 10 iterations of user annotation inputs. Best viewed in color.

Fig. 7. Comparison of the learned policy with baseline policies on the DAVIS 2017 validation set. The figure shows annotation costs required to reach certain Jmean levels with
the specific task selection policy. Annotation costs are given in number of clicks per total number of frames in all videos of the dataset. Lower values are better.

0
To predict the expected effect of the user labeling of a certain receiving label l when annotated by the user. ELC ð f ; hÞ is the
frame at a certain hierarchical level, we utilize the Expected Label expected label change by annotating frame f at hierarchical level
Change defined in Eq. 3. in Ref. [20] as follows. h. We compute ELC ð f ; hÞ over k superpixels sampled randomly
X       from the segmentation. In our baseline we also include the esti-
0 0
ELC ð f ; hÞ ¼ P l ufj ;h ¼ l  C l ufj ;h l ð6Þ mation of the cost of annotating a frame at a specific hierarchical
f ;h
n  j;o
j21...jU level. This cost is given by the sum of the estimated probabilities
f ;h
l0 2Ln l uj that the user will change the labels of the sampled superpixels.
We normalize the cost with the sum of the areas of the sampled
  
0 superpixels. Eventually, we maximize the expected label change
Here, C l ufj ;h l denotes the total pixelwise label change in
0
divided by the estimated cost of the task to get the most efficient
the video in case superpixel ufj ;h was assigned label l . labeling task. Algorithm 1 summarizes the task selection
  
0
P l ufj ;h ¼ l denotes the estimated probability of the superpixel procedure.
256
V. Varga, A. Lo
Algorithm 1:MRF baseline policy: Annotation task selection
We construct a graph from the superpixel segmentation: each
node represents a level 1 superpixel. Superpixels of consecutive
frames belonging to the same level 1 supervoxel are connected
by edges. Neighboring superpixels in the same frame are also con-
nected. In the definition of the energy function, we follow [20] (see
Eqs. 5, 7 and 8 in [20]): unary potentials are derived from predicted
probabilities of Gaussian Mixture Model (GMM) based classifiers.
Individual GMMs are trained to estimate distribution of CIELAB
color space histograms in each user-labelled frame. GMMs for
unlabeled frames are computed by interpolating between GMM
models of the nearest annotated frames (see Eq. 5. in [20]). We also
use the smoothing term as given by Eq. 8 in Ref. [20]. We minimize
the energy function with the help of graph cuts. The baseline algo-
rithm was implemented with the help of the gco library [42–44].
Fig. 8 shows the estimated action-advantage values by our
model at different time steps during an annotation episode on
the unseen DAVIS soapbox sequence. The figure demonstrates that
the network learnt to extract meaningful information from the
structure of the video. At t ¼ 0 the estimated maximum advantage
corresponds to the frame where the soapbox cart passes in front of
the camera in the video. At this point the apparent size of the cart
is the largest and a high level, approximate labeling can be propa-
gated in both directions across the entire sequence effectively. In
the rightmost plot, it can be seen that seven of the first nine
selected tasks were located in the second part of the video, where
the intense movement of the camera spoiled the quality of the pre-
dicted border quickly.
4.3. Ablation studies
We performed an ablation study to analyze the relevance of
some of the components of the system.
Fig. 8. Action-advantage values (A-values) predicted by our model on the unseen soapbox sequence of the DAVIS validation set after t iterations of simulated user inputs. First
ten actions selected with greedy policy were as follows: ð67; 4Þ; ð97; 4Þ; ð8; 4Þ; ð88; 2Þ; ð60; 2Þ; ð83; 1Þ; ð53; 1Þ; ð46; 1Þ; ð76; 1Þ; ð33; 1Þ; format: ðf t ; ht Þ.
}rincz / Neurocomputing 405 (2020) 247–258
First, we experimented with modifications to the reinforcement
learning setup. Setting either of the x or j parameters of the
reward function to 1 had a significant effect on the results: the high
variances of the total expected sum of rewards from video to video
destabilized the training process and the model failed to converge.
Removing the explicit penalty of 1 for repetitive actions had a
major effect on the length of the training: although the training
converged, but during evaluations, the learned model often fell
into a cycle of selecting repetitive actions after 5 to 10 steps of
the evaluation episode.
Next, we tried to use different configurations of image channels
in our MDP state representation. As expected, removing any of the
image channels encoding the state of the current annotation, i.e.
removing I7 or I8 , prevents the learned policy to surpass the perfor-
mance of the MRF baseline policy detailed in Section 4.2.2. Remov-
ing time index t from the state representation decreased
performance on a similar scale. The removal of optical flow and
occlusion data from the state representation increased the annota-
tion costs needed to reach a mean Jaccard-index of 0:8 by 17% and
5% respectively, while removing both resulted in a more signifi-
cant increase of 25% in the annotation costs.
Besides these ablation studies, we conducted experiments to
measure the effect of the addition of color information to the state
representation. Downscaled color information was added to the
state representation in RGB and CIELAB color formats, but the
results did not improve. Replacing optical flow or information
about the occlusion with color data caused minor deterioration.
A state representation with a higher, 112  64 spatial resolution
was also tested. While the results were approximately similar to
the smaller representation of 56  32 spatial resolution, the size
of the experience replay buffer was now limited by our hardware
capacities and made experiments with the larger model impracti-
cal. An increased granularity of the base level supervoxel segmen-
tation is expected to demonstrate the advantage of the high
resolution representation.
We found that a higher number of hierarchical levels did not
improve the results. For H > 5, the agent tended to completely
omit the selection of specific hierarchical levels. Modifying the
ratio of reward parameters werr ; wac will lead to different priorities
in the learned policy. A higher werr pushes the agent towards
selecting lower level, more laborous tasks in the beginning.
4.4. Simulating human experiments
As mentioned earlier in the paper, we trained and evaluated the
network with simulated human input. Details of the human anno-
tation protocol can be found in Section 3.2. Although, this protocol
is quite straightforward, we expect the human annotator may
make mistakes. To test the robustness of our approach to human
error, we conducted experiments with noisy input as well. Results
are presented in Table 3. We evaluated our model in the following
 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo
Table 3
Effects of noisy simulated human input on predicted label quality. The table shows the mean Jaccard-index achieved from a 10 clicks per frame annotation budget when specific
levels of noise were added to the simulated user input. Binary and multiclass segmentation results were reached on the DAVIS 2016 and 2017 validation sets respectively.
Dataset r¼0 r ¼ 0:05
DAVIS 2016 (binary) 0.852 0.845
DAVIS 2017 (multilabel) 0.837 0.823
way: let pi be the relative frequency of label Li among the pixel
labels of a superpixel ufj ;h . Then, we select the label assigned to
 
ufj ;h in this way: l ufj ;h ¼ Lk , where k ¼ argmax ðpi þ ni Þ;
i¼1::jLj ^ pi >0
ni  Nð0; rÞ. Setting r to 0 eliminates the error. We found that
even at high noise levels (r  0:3), the method still gives accept-
able results. As an example, in the case of r ¼ 0:3, the error of
the annotation model means that the less frequent label is selected
with 12% probability when two pixel labels are present with rela-
tive frequencies of 1=4 and 3=4. Human annotators are not
expected to have such high error rates.
4.5. Limits of our implementation
In this paper we modeled clicking as the annotation method
and left aside other options, such as drawing polygons or scribbles
and alike. They would certainly affect annotation time. In addition,
the method to be chosen may depend on the annotation task that
can be subject of usability studies. Our work emphasizes the
advantages of the RL framework and not the specific choice of
the annotation procedure itself.
In our implementation, the finest segmentation granularity was
limited by the superpixel size of the CCS algorithm. In our setting –
detailed in Section 3.1 – a maximum of 91:4% mean Jaccard-index
can be reached on the DAVIS 2017 validation set. However, several
hierarchical superpixel/supervoxel methods are able to go down to
pixel/voxel levels (e.g. GBH) and full precision may be reached by
choosing a different method. That would also require a higher res-
olution state representation.
5. Conclusion and future work
In this paper, we have introduced a novel approach to reduce
human effort in interactive video segmentation annotation with
reinforcement learning. We presented a modified version of the
Dueling DDQN architecture, that shares its parameters across the
temporal axis of its video input. This way, our network leverages
the power of deep learning to select optimal annotation tasks dur-
ing the labeling of longer video clips in a single inference phase.
Potential future directions include the improvement of the label
prediction step. More sophisticated algorithms and trained models
seem promising to us. In addition, deep reinforcement learning has
been extended to multi-objective optimization (e.g. [45]) offering
further options for improving performance. The choice of clicking,
drawing polygons, or scribbles depending on the situation may
also improve quality, not mentioning usability.
CRediT authorship contribution statement
Viktor Varga: Conceptualization, Methodology, Software, Writ-
}rincz: Supervision,
ing - original draft, Visualization. András Lo
Writing - review & editing.
257
r ¼ 0:1 r ¼ 0:2 r ¼ 0:3
0.831 0.808 0.760
0.804 0.762 0.715
Declaration of Competing Interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Acknowledgments
The authors would like to thank Ádám Fodor for his cooperation
in the setup of the supervoxel algorithms utilized in the paper and
Márton Véges for his helpful comments. V.V. is supported by the
European Union and co-financed by the European Social Fund
(EFOP-3.6.3-16-2017-00002). Project No. ED-18-1-2019-0030
(Application-specific highly reliable IT solutions) of the National
Research, Development and Innovation Fund of Hungary, financed
under the Thematic Excellence Programme funding scheme sup-
ported the work of A.L.
References
[1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: a large-scale
hierarchical image database, in: CVPR09, 2009..
[2] M. Everingham, L. Gool, C.K. Williams, J. Winn, A. Zisserman, The pascal visual
object classes (VOC) challenge, Int. J. Comput. Vision 88 (2) (2010) 303–338,
https://doi.org/10.1007/s11263-009-0275-4.
[3] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, T. Huang, YouTube-VOS: A large-
scale video object segmentation benchmark, arXiv preprint arXiv:1809.03327..
[4] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, A. Sorkine-
Hornung, A benchmark dataset and evaluation methodology for video object
segmentation, Comput. Vis. Pattern Recogn. (2016).
[5] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, L. Van Gool,
The 2017 DAVIS Challenge on video object segmentation, arXiv:1704.00675..
[6] D. Tsai, M. Flagg, A. Nakazawa, J.M. Rehg, Motion coherent tracking using
multi-label mrf optimization, Int. J. Comput. Vis. 100 (2) (2012) 190–202.
[7] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, L. Van Gool,
One-shot video object segmentation, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2017, pp. 221–230.
[8] S. Yun, J. Choi, Y. Yoo, K. Yun, J. Young Choi, Action-decision networks for visual
tracking with deep reinforcement learning, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2017, pp. 2711–
2720.
[9] W.-D. Jang, C.-S. Kim, Online video object segmentation via convolutional
trident network, in: Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2017, pp. 5849–5858.
[10] L. Yang, J. Han, D. Zhang, N. Liu, D. Zhang, Segmentation in weakly labeled
videos via a semantic ranking and optical warping network, IEEE Trans. Image
Process. 27 (8) (2018) 4025–4037.
[11] D. Zhang, J. Han, L. Yang, D. Xu, Spftn: a joint learning framework for localizing
and segmenting objects in weakly labeled videos, IEEE Trans. Pattern Anal.
Mach. Intelligence..
[12] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, Slic superpixels,
Tech. rep. (2010)..
[13] D. Stutz, A. Hermans, B. Leibe, Superpixels: an evaluation of the state-of-the-
art, Comput. Vis. Image Underst. 166 (2018) 1–27.
[14] J. Chang, D. Wei, J.W. Fisher, A video representation using temporal
superpixels, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2013, pp. 2051–2058.
[15] M. Grundmann, V. Kwatra, M. Han, I. Essa, Efficient hierarchical graph-based
video segmentation, in ieee computer society conference on computer vision
and pattern recognition, IEEE 2010 (2010) 2141–2148.
[16] C. Rother, V. Kolmogorov, A. Blake, GrabCut: interactive foreground extraction
using iterated graph cuts, in: ACM transactions on graphics (TOG), vol. 23,
ACM, 2004, pp. 309–314..
258 }rincz / Neurocomputing 405 (2020) 247–258
V. Varga, A. Lo
[17] N. Shankar Nagaraja, F.R. Schmidt, T. Brox, Video segmentation with just a few
strokes, in: Proceedings of the IEEE International Conference on Computer
Vision, 2015, pp. 3235–3243.
[18] A. Fathi, M.F. Balcan, X. Ren, J.M. Rehg, Combining self training and active
learning for video segmentation, in: BMVC, Georgia Institute of Technology,
2011..
[19] S. Vijayanarasimhan, K. Grauman, Active frame selection for label propagation
in videos, in: European Conference on Computer Vision, Springer, 2012, pp.
496–509.
[20] B. Luo, H. Li, F. Meng, Q. Wu, C. Huang, Video object segmentation via global
consistency aware query strategy, IEEE Trans. Multimedia 19 (7) (2017) 1482–
1493.
[21] V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A.
Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, et al., Human-level control
through deep reinforcement learning, Nature 518 (7540) (2015) 529.
[22] K. Zhou, Y. Qiao, T. Xiang, Deep reinforcement learning for unsupervised video
summarization with diversity-representativeness reward, in: Thirty-Second
AAAI Conference on Artificial Intelligence, 2018.
[23] J.C. Caicedo, S. Lazebnik, Active object localization with deep reinforcement
learning, in: Proceedings of the IEEE International Conference on Computer
Vision, 2015, pp. 2488–2496.
[24] Y. Xiang, A. Alahi, S. Savarese, Learning to track: online multi-object tracking
by decision making, in: Proceedings of the IEEE International Conference on
Computer Vision, 2015, pp. 4705–4713.
[25] J. Han, L. Yang, D. Zhang, X. Chang, X. Liang, Reinforcement cutting-agent
learning for video object segmentation, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2018, pp. 9080–9089.
[26] O. Russakovsky, L.-J. Li, L. Fei-Fei, Best of both worlds: human-machine
collaboration for object annotation, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2015, pp. 2121–2131.
[27] G. Song, H. Myeong, K. Mu Lee, Seednet: automatic seed generation with deep
reinforcement learning for robust interactive segmentation, in: Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp.
1760–1768.
[28] D. Acuna, H. Ling, A. Kar, S. Fidler, Efficient interactive annotation of
segmentation datasets with polygon-rnn++, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2018, pp. 859–868.
[29] O. Rudovic, M. Zhang, B. Schuller, R.W. Picard, Multi-modal active learning
from human data: a deep reinforcement learning approach, arXiv preprint
arXiv:1906.03098..
[30] S.-H. Lee, W.-D. Jang, C.-S. Kim, Contour-constrained superpixels for image and
video processing, in: Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2017, pp. 2443–2451.
[31] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, T. Brox, Flownet 2.0:
evolution of optical flow estimation with deep networks, in: Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.
2462–2470.
[32] A. Benard, M. Gygli, Interactive video object segmentation in the wild, arXiv
preprint arXiv:1801.00269..
[33] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, N. De Freitas, Dueling
network architectures for deep reinforcement learning, in: International
Conference on Machine Learning, 2016.
[34] M. Hausknecht, P. Stone, Deep recurrent q-learning for partially observable
mdps, in: 2015 AAAI Fall Symposium Series, 2015..
[35] H. Van Hasselt, A. Guez, D. Silver, Deep reinforcement learning with double q-
learning, in: Thirtieth AAAI Conference on Artificial Intelligence, 2016.
[36] T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized experience replay, arXiv
preprint arXiv:1511.05952..
[37] G. Hinton, N. Srivastava, K. Swersky, Neural networks for machine learning
lecture 6a overview of mini-batch gradient descent, Cited on 14..
[38] J. Luiten, P. Voigtlaender, B. Leibe, Premvos: proposal-generation, refinement
and merging for the DAVIS challenge on video object segmentation 2018, in:
The 2018 DAVIS Challenge on Video Object Segmentation-CVPR Workshops,
2018..
[39] K.-K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, L. Van
Gool, Video object segmentation without temporal information, IEEE
transactions on pattern analysis and machine intelligence..
[40] P. Voigtlaender, B. Leibe, Online adaptation of convolutional neural networks
for the 2017 DAVIS challenge on video object segmentation, in: The 2017
DAVIS Challenge on Video Object Segmentation-CVPR Workshops, vol. 5,
2017..
[41] L. Bao, B. Wu, W. Liu, CNN in MRF: video object segmentation via inference in a
CNN-based higher-order spatio-temporal mrf, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2018, pp. 5977–
5986.
[42] Y. Boykov, O. Veksler, R. Zabih, Fast approximate energy minimization via
graph cuts, IEEE Trans. Pattern Anal. Mach. Intell. 23 (11) (2001) 1222–1239.
[43] Y. Boykov, V. Kolmogorov, An experimental comparison of min-cut/max-flow
algorithms for energy minimization in vision, IEEE Trans. Pattern Anal. Mach.
Intell. 9 (2004) 1124–1137.
[44] V. Kolmogorov, R. Zabih, What energy functions can be minimizedvia graph
cuts?, IEEE Trans. Pattern Anal. Mach. Intell. 2 (2004) 147–159.
[45] H. Mossalam, Y.M. Assael, D.M. Roijers, S. Whiteson, Multi-objective deep
reinforcement learning, arXiv preprint arXiv:1610.02707..
András Lörincz graduated from the Eötvös Loránd
University. His background is in physics. He habilitated
in laser physics and in computer science, too. His
research interest lies at the crossroad of artificial gen-
eral intelligence, cognition, and neuroscience, with
principal areas including human-computer interaction
and applications. He has different research projects
supported by the government and by companies. He
supervises more than 10 Ph.D. and M.Sc. students and
has published over 250 peer reviewed publications.
Viktor Varga received his B.Sc. and M.Sc. degrees in
Computer Science from Eötvös Loránd University of
Budapest in 2014 and 2017. He is currently a Ph.D.
student working in the Neural Information Processing
Group at the university. His research interests include
reinforcement learning and computer vision.