JID: IS

ARTICLE IN PRESS [m5G;April 18, 2018;12:52]

Information Systems 0 0 0 (2018) 1–11

Contents lists available at ScienceDirect

Information Systems
journal homepage: www.elsevier.com/locate/is

Searching for variable-speed motions in long sequences of motion

capture data
Jan Sedmidubsky∗, Petr Elias, Pavel Zezula
Masaryk University, Botanicka 68a, Brno, 602 00, Czechia

a r t i c l e i n f o a b s t r a c t

Article history: Motion capture data digitally represent human movements by sequences of body configurations in time.
Received 10 March 2017 Subsequence searching in long sequences of such spatio-temporal data is difficult as query-relevant mo-
Revised 26 October 2017
tions can vary in execution speeds and styles and can occur anywhere in a very long data sequence. To
Accepted 11 April 2018
deal with these problems, we employ a fast and effective similarity measure that is elastic. The prop-
Available online xxx
erty of elasticity enables matching of two overlapping but slightly misaligned subsequences with a high
Keywords: confidence. Based on the elasticity, the long data sequence is partitioned into overlapping segments that
Content-based retrieval are organized in multiple levels. The number of levels and sizes of overlaps are optimized to generate a
Motion capture data modest number of segments while being able to trace an arbitrary query. In a retrieval phase, a query is
Subsequence matching always represented as a single segment and fast matched against segments within a relevant level with-
Speed-invariant retrieval out any costly post-processing. Moreover, visiting adjacent levels makes possible subsequence searching
Similarity measure
of time-warped (i.e., faster or slower executed) queries. To efficiently search on a large scale, segment fea-
Hierarchical segmentation
Indexing
tures can be binarized and segmentation levels independently indexed. We experimentally demonstrate
effectiveness and efficiency of the proposed approach for subsequence searching on a real-life dataset.
© 2018 Elsevier Ltd. All rights reserved.

1. Introduction example, find occurrences of perfect backflip landings within hun-

Current motion capturing technologies can accurately record
a human motion at high spatial and temporal resolutions. The
recorded motion is represented as an ordered sequence of poses
that describe skeleton configurations in corresponding video
frames. The skeleton configuration is represented by a set of 3D
coordinates determining positions of the captured body joints in
space. The recorded motion sequences are used in a variety of ap-
plications, e.g., in healthcare to recognize movement disorders, in
sports to analyze performances of top athletes, or in computer ani-
mation to browse large databases of human motions for production
of realistically looking games or movies. These applications require
effective and efficient search operations to increase reusability and
findability of expensively recorded data in the past.
A search operation is primarily specified by a query object that
can be either selected as an existing example [1], or modeled
by special interfaces such as hand-drawn sketches [2] or puppet
models [3]. We especially focus on the query-by-example subse-
quence matching operation: Given a short query sequence and a
very long data sequence, search the data sequence and locate its
subsequences that are the most similar to the query sequence. For
∗ promise between search performance and accuracy. It is also dy-
Corresponding author.
E-mail address: xsedmid@fi.muni.cz (J. Sedmidubsky).
dreds of hours of exercise recordings. Locating query-relevant sub-
sequences constitutes a hard task since their lengths and positions
(i.e., beginnings and endings) are unknown. Moreover, the query
can not be anticipated in advance and need not correspond to any
semantic or known action class, so textual-annotation-based re-
trieval can not be applied. To deal with these problems, a fine seg-
mentation technique along with an effective similarity measure are
needed.
The contribution of this paper is an efficient subsequence
matching approach that is schematically illustrated in Fig. 1. The
retrieval process is based on a multi-level segmentation structure
that produces a minimum number of segments with respect to
an elasticity property of a similarity measure. The elasticity al-
lows segments to be shifted much more than of a single frame,
which increases overall search performance. We further binarize
segment features and employ a disk-based index structure to in-
dex individual segmentation levels independently and thus access
them in parallel. This enables real-time searching in long mo-
tion sequences, taking even dozens of days. A proposed speed-
invariant retrieval algorithm additionally supports searching for
query-relevant motions that are executed faster or slower. The
whole multi-level segmentation structure is customizable to com-
namic to support the addition of a new content.

https://doi.org/10.1016/j.is.2018.04.002
0306-4379/© 2018 Elsevier Ltd. All rights reserved.

Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
JID: IS
ARTICLE IN PRESS
2 J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
Fig. 1. Flowchart diagram presenting a general overview on the proposed subsequence matching approach in motion capture data.
2. Related work
Subsequence matching methods for motion capture data gen-
erally require a (1) segmentation technique to partition a data se-
quence into meaningfully-long data segments, (2) similarity mea-
sure to compare query and data segments, and (3) retrieval algo-
rithm to efficiently localize query-similar subsequences by group-
ing the most relevant data segments.
2.1. Segmenting motions
A segmentation technique partitions the long data sequence
into short segments that are directly comparable with segment(s)
of the query sequence. For specific applications, data segments
can be identified with respect to observed changes in repeat-
ing movement patterns [4] or changes in pose distribution [5].
The alternative is a semantic segmentation which localizes non-
overlapping segments corresponding to some of the predefined
classes (e.g., walking, kicking and jumping) [1,6,7]. Semantic and
domain-specific segmentations are suitable for discovering such
parts that comply with the pre-defined semantics. On the other
hand, they are rather inconvenient for subsequence retrieval in
general because of their poor ability to locate queries that do not
belong to any predefined semantic classes, such as transitional
motions in between two consecutive actions. Consequently, the
semantic segmentations are better applicable in search scenarios
where query categories are anticipated, for example in event de-
tection or stream annotation.
This problem is traditionally overcome by a fine-grained seg-
mentation that partitions both query and data sequence into short
segments of a fixed-size. The query can be partitioned into over-
lapping segments using the sliding-window principle and the data
sequence into disjoint (i.e., non-overlapping) segments to reduce
the data replication, or vice versa [8,9]. In both cases, however,
query segments need to be matched with data segments respect-
ing their temporal order. The temporal alignment of candidate
matches can be very computationally demanding as search com-
plexity raises with the query length. We avoid this problem by
considering a query as a single segment. Such query segment is
evaluated against a multi-level overlapping segmentation to en-
sure that an arbitrary query-relevant data subsequence (bounded
in length by a user) highly overlaps with at least one data seg-
ment of a similar length. For more details regarding existing on-
line, semi-online and offline segmentation techniques we refer to
a study in [10].
2.2. Comparing similarity
A variety of similarity measures exists to determine pair-
wise proximity of motions. The first group of similarity mea-
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
sures directly exploits the time-series representation of motion
data. Multi-dimensional time series of body joints (positions or
angles) are compared by the temporal alignment techniques, such
as Dynamic Time Warping (DTW) [11,12] and its variants [13–15],
or Longest Common Subsequence (LCS) [16]. Although such tech-
niques deal well with temporal discrepancies, such as faster and
slower movements of otherwise same actions, their comparison
has quadratic time complexity. Multiple time series can be repre-
sented by shapelets that have a great descriptive power and prove
to be very effective and efficient in binary classification [17]. Multi-
variate time series can further be transformed into univariate time
series, such as in [18], where motion primitives are obtained by
a hierarchical clustering. A comprehensive review on time-series
mining can be found in [19].
The feature-based comparison constitutes the second family of
approaches. Feature extraction is applied to discover the key mo-
tion characteristics in lower-dimensional vector spaces. The string-
matching Smith–Waterman distance [20,21] is applied to match
motions based on their string representation assembled from a
vocabulary of motion features. Features can be either carefully
chosen manually [22] or acquired by supervised learning using
Neural Networks [23,24] or Support Vector Machines [25]. Deep
[26,27] and recurrent [23] neural networks constitute the best-
performing methods in recognizing actions. Despite being very ef-
fective, they are inherently designed for classifying actions and
hardly employed to quantify a pair-wise similarity, which is re-
quired for the task of subsequence matching. Fortunately, some
exceptions exist where discriminative feature vectors are extracted
and used for a similarity comparison. For example in [24], the 160-
bit features extracted from a deep auto-encoder are fast-compared
by the Hamming distance. Other metric functions are commonly
used to compare feature vectors, such as the weighted Euclidean
distance in [28].
In this work we employ a convolutional neural network trained
on the motion data domain to achieve recognition rates compa-
rable with state-of-the-art classifiers as demonstrated in [29]. Im-
portantly, we benefit from a fixed-size 4096-dimensional highly-
descriptive feature representation that is extracted from the last
hidden layer of the network. The proposed similarity measure is
thus a fundamental building block for the design of our subse-
quence matching framework. This is particularly reflected in the
segmentation structure that is tailored to maximally utilize the ex-
pedient properties of the employed similarity measure.
2.3. Indexing and retrieving motions
By conveniently combining a segmentation technique and a
similarity measure, a subsequence retrieval algorithm is used to
locate query-relevant parts within a very long, but already seg-
JID: IS
ARTICLE IN PRESS
J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
mented, data sequence. Sequential search is commonly used, such
as the A-LTK method [30] or string-matching-based Knuth–Morris–
Pratt search algorithm [31]. However, sequential scan does not
provide much scalability for searching in large volumes of mo-
tion data. To significantly speed-up similarity searching, multi-
dimensional or metric-based index structures can be utilized [32].
To efficiently access data segment features, a self-organizing map
is used in [33] or trie-based structure in [34]. The M-Index struc-
ture in [35] is used to efficiently search for the key poses that are
the most similar to selected poses of a query sequence. The ob-
tained ranked sets of candidate poses are post-processed in tem-
poral order to identify query-relevant subsequences. In this paper,
we also focus on the large-scale subsequence searching. By utiliz-
ing the disk-based PPP-Codes index [36], we are able to search a
several-day long sequence within a single second, which outper-
forms the best-performing approaches. Both effectiveness and effi-
ciency of our retrieval algorithms are evaluated and confronted to
related work in more detail in the experimental part.
2.4. Our contributions
We extend our recent idea [37] by proposing a new speed-
invariant retrieval algorithm for subsequence matching in mo-
tion capture data. Speed invariance reflects the ability to search
for motions executed faster or slower but otherwise similar to a
query. The algorithm employs a segmentation technique that or-
ganizes data segments within a multi-level structure. This multi-
level structure ensures a traceability of an arbitrary query-relevant
data subsequence, which is bounded in its size. In particular, each
level of this structure keeps overlapping data segments of a given
size. The number of levels and segments within each level are
constructed to be minimum with respect to the query-length lim-
its and the ability of a similarity measure to compare slightly
cropped/extended/shifted motions.
We employ a similarity measure which is tolerant to
slower/faster and slightly changed motions and also generates
fixed-size feature vectors for motions of variable lengths. As the
fixed-size features are compared by the Euclidean distance, the
segments in each level can be efficiently indexed. Since we primar-
ily focus on search performance, we introduce a new transforma-
tion of feature vectors into a bit representation. The bit represen-
tation further decreases space occupancy and search costs by order
of magnitude with a negligible impact on search effectiveness. We
also make a study on large-scale subsequence matching to enable
searching potentially in a 48-day long motion sequence within one
second.
3. Similarity of motion data
To compare any pair of rather short motion sequences, we
adopt a similarity measure proposed in [29]. It uses very effective
and fixed-size feature vectors extracted from motions of variable
lengths and compares them by the efficient Euclidean distance. The
feature extraction is based on visualizing the normalized joint tra-
jectories into a 2D image, fine-tuning a deep convolutional neu-
ral network by the generated images, and extracting the 4096D
feature vector of a high descriptive power from the last hidden
network layer. The whole feature extraction process takes about
25 ms.
The employed similarity measure attributes very convenient
properties for segmentation-based subsequence matching because
it (1) extracts very effective and fixed-size 4096-dimensional fea-
ture vectors for motions of variable lengths, (2) compares these
vectors by the efficient Euclidean distance that can be indexed, (3)
tolerates a non-trivial degree of segmentation error when compar-
ing similar motions, and (4) tolerates faster and slower executions
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
3
of the same motions. The important properties are described in the
following paragraphs.
• Effectiveness. A high descriptive power of feature vectors can
be further increased by fine-tuning the neural network for the
specific application purpose, e.g., by feeding the network with
training samples of categorized motion images. Such fine-tuned
network generates feature vectors that not only better distin-
guish motions that are present during the training phase, but
also clusters similar samples on which the network has never
been explicitly trained. This is especially true when training
samples come from a diverse dataset of actions.
• Efficiency and indexability. The fixed size of features for
variably-long motions enables a fast similarity calculation by
the Euclidean distance, without the need of using any expen-
sive warping technique. The fixed size of features also enables
utilizing disk/main-memory-based metric index structures [32].
The float-number values of feature vectors can be further bina-
rized to occupy less memory space and to provide much faster
comparison by the Hamming distance.
• Elasticity. This property denotes the ability of the similarity
measure to tolerate “imprecisely” segmented motion sequences.
It means that the similarity measure returns a near-zero dis-
tance between a given action (e.g., jump) and the same ac-
tion with some additional/cropped frames. Fig. 2a/b shows how
these modifications influence the accuracy on the action recog-
nition scenario.
• Speed-invariance. The same action, such as running, can be
performed faster or slower. Fig. 2c shows that the proposed
similarity measure flawlesly recognizes actions which are per-
formed faster or slower. Even a 10-times faster/slower execu-
tion of the same action can be recognized with a negligible er-
ror.
Elasticity, as the ability to recognize imprecisely segmented mo-
tions, is a very synergistic property that is highly leveraged in the
proposed segmentation method. It allows for much bigger shifts
between consecutive overlapping segments compared to the con-
ventional methods [1,8]. Hand-in-hand with the elasticity of the
similarity measure, the amount of the shift strongly negotiates the
trade-off between the search accuracy and performance. To achieve
the maximum versatility, a user-defined covering-factor parameter
is introduced in the following section. This parameter strongly in-
fluences the segmentation policy as it denotes strict requirements
on the allowed spacing of segments. The lower the spacing, the
higher the data traceability, but also the higher space and perfor-
mance demands.
Additionally, the speed-invariance property brings the advan-
tage of searching for faster and slower motion executions. The
degree of tolerated speed-up is also controlled by a user-defined
stiffness parameter. This parameter determines how the segmenta-
tion structure is traversed to find desirably faster or slower subse-
quences with respect to a query.
4. Multi-level segmentation structure of data sequences
To search for query-relevant subsequences, the data sequence
has to be partitioned into segments. Traditional methods [8] sug-
gest partitioning the data sequence into disjoint (non-overlapping)
segments, while the query sequence into overlapping segments us-
ing the sliding window principle (or vise-versa). Such partitioning
facilitates locating relevant data segments that are similar to some
query segments. Although the data segments can be indexed and
efficiently retrieved, this concept has the following disadvantages:
• Longer queries are partitioned into a higher number of query
segments. For each query segment an independent search (i.e.,
JID: IS
ARTICLE IN PRESS
4 J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
Fig. 2. Tolerance of the similarity measure on the action recognition scenario evaluated by 1-NN queries on a database of 1464 HDM05 motions divided into 15 categories.
Each annotated query action is compared against all the remaining actions that are modified by (a) removing/adding both beginning and ending frames, (b) shifting actions
to remove/add a preceding/following frame content and (c) accelerating/decelerating actions (i.e., by removing/adding frames uniformly within actions). In particular, the
database actions are (a) cropped/expanded by 10–90%, (b) shifted by 10–90%, and (c) 2–10-times accelerated/decelerated with respect to their original frame content.
sub-query) has to be executed to retrieve the most similar data
segments;
• The retrieved data segments of all sub-queries have to be in-
telligently merged respecting chronological order of query and
retrieved segments to construct a set of relevant subsequences
as the query result.
To overcome these problems, we propose to consider the query
as a single segment. It means that only a single search is executed
without the need of any other merging procedure. However, this
would require sliding data segments for every potential query size,
which results in a huge number of data segments. Such number
can be dramatically reduced when the used similarity function can
deal with a certain versatility in segmentation – sliding data seg-
ments can be then shifted much more than of a single frame only
and can be constructed just for the specific sizes of queries.
4.1. Problem formalization
We partition the data sequence into segments in a way that an
arbitrary data subsequence (bounded in length) overlaps with at
least one segment in the majority of frames. Consequently, hav-
ing a query as a single segment, each query-relevant data subse-
quence highly overlaps with at least one data segment. The high
overlap ensures that relevant subsequences are always findable just
by searching for similar segments. To quantify the high overlap be-
tween the specific subsequence and segment, we define covering
factor cf ∈ [0, 1) which determines the maximum ratio between the
number of their non-overlapping frames and the segment length.
In other words, the covering factor quantifies the tolerance of the
similarity function towards cropped/added content of two similar
motions. The following definition defines the covering factor for-
mally.
Definition 1. Given data sequence m and covering factor cf ∈ [0, 1):
We say that any subsequence m[i : j ] is cf-covered by segment m[i:
 +| j  − j |
j] if and only if |i −i|j−i ≤ cf.
Our objective is to partition the data sequence into segments
having optimal sizes and minimum possible overlaps with respect
to the covering factor. To ensure that an arbitrary subsequence is
covered by at least one segment, we need to restrict the subse-
quence length by the minimum l min ∈ N and maximum l max ∈ N
value (the maximum length is supposed to be much shorter than
the length of the data sequence). According to these limits, we par-
tition the data sequence according to the following objective.
Objective 1. Given data sequence m and minimum lmin and max-
imum lmax subsequence length: Partition sequence m into a mini-
mum number of segments so that an arbitrary subsequence m[i: j]
(bounded in length l min ≤ j − i ≤ l max ) is cf-covered by at least one
segment.
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
4.2. Multi-level segmentation
A query sequence mQ is also restricted to a limited length in
[lmin , lmax ]
and always considered as a single segment. To partition
the data sequence m according to Objective 1, we need to cover all
potential query-relevant subsequences. Since positions (beginning
and ending frames) of relevant hits are not known in advance, all
possible data subsequences of the restricted length have to be cov-
ered.
Our idea is to define segmentation levels responsible for groups
of queries in certain length intervals. Each level has its own size
of segments that overlap by a fixed-size margin to cf-cover an ar-
bitrary data subsequence. We naturally require to minimize the
number of such levels as well as the size of overlaps between seg-
ments with respect to the predefined covering factor cf. These ob-
servations imply the following important lemma.
Lemma 1. A single segmentation level with segments of fixed-size l
can cf-cover the subsequences having their lengths in range [l · (1 −
c f ) , l · ( 1 + c f )] .
Proof. According to Lemma 1, a single level with segments of
fixed-size l can cover only the subsequences which are maximally
l · (1 + c f ) long. Suppose that this statement is not true, then some
segment m[i, j] can also cover subsequence m[i : j ] which is longer
than l · (1 + c f ) = ( j − i ) · (1 + c f ) frames:
j  − i > ( j − i ) · ( 1 + c f )
j  − i > j − i + j · c f − i · c f
j  − i > j − i + c f · ( j − i )
j  − i − j + i > c f · ( j − i )
j  − i − j + i
> cf
j−i
|i − i | + | j − j  |
> c f,
j−i
which is in contradiction with Definition 1. Similarly, a segment of
l frames can cover the subsequences which have minimally l · (1 −
c f ) frames. Due to the analogy with the previous case, the proof is
omitted. 
4.2.1. Lengths of segments
To minimize the total number of segmentation levels (i.e., also
the total number of segments), the individual levels have to cover
subsequences of the possibly largest length interval. At the same
time, the first level with segments of length l1 needs to cover the
shortest possible subsequences of length lmin :
l min
l min = l 1 · (1 − c f ) ⇔ l1 =
1 − cf
Consequently, this level also covers the subsequences which are
up to l 1 · (1 + c f ) frames long (based on Lemma 1). The second
JID: IS
ARTICLE IN PRESS [m5G;April 18, 2018;12:52]

J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11 5

level then covers the subsequences of at least l 1 · (1 + c f ) frames, 2. It can be shown that an arbitrary subsequence of length in
so l 2 = l 1 · (1 + c f )/(1 − c f ). Similarly as the first level, the second range [l r · (1 − c f ), l r · (1 + c f )] is covered by at least one seg-
one covers maximally the subsequences of l 2 · (1 + c f ) frames. This ment. This can be proven via an induction step. 
continues until the nth segmentation level covers the longest pos-
Fig. 3 illustrates the multi-level segmentation structure along
sible subsequences of lmax frames:
with segments generated at first two levels.
l max
l n−1 · (1 + c f ) < l max ≤ l n · (1 + c f ) ⇔ ln ≥ .
1 + cf 4.2.4. Number of segments
Respecting these properties, the segment length is determined by When we partition the data sequence m of |m| frames, the
constructing the individual levels. The fixed length lr of segments number nr of segments at the rth level is determined as:
at the rth level (r ∈ [1, n]) can be recursively defined as:
 
|m| − l r
nr = 1 + . (4)
l min 1 + cf lr · c f
l1 = l r = l r−1 · . (1)
1 − cf 1 − cf
4.2.5. Replication factor
4.2.2. Number of segmentation levels The covering factor cf has the most important influence on the
The number n of segmentation levels can be calculated as n = number of segmentation levels as well as the total number of gen-
x + 1, where x denotes the power parameter needed to skip to erated segments. To get an idea about “global” overlaps, we define
another level and is computed as: the replication factor rf that indicates how many times the same
 x frame is repeated in segments. Supposing that the length of data
l min 1 + cf l max sequence is much longer than lengths of segments, we express the
· =
1 − cf 1 − cf 1 + cf replication factor rf as:
 x
1 + cf l max · (1 − c f ) rf 
n
, (5)
=
1 − cf l min · (1 + c f ) cf
  where n stands for the number of segmentation levels. For exam-
1 + cf l max · (1 − c f )
x · log = log ple, having the covering factor c f = 0.2 and four segmentation lev-
1 − cf l min · (1 + c f )
  els, each frame of the original data sequence is involved twenty
l max · (1 − c f ) times in the specific segments.
x = log 1+c f
1−c f l min · (1 + c f )
   4.3. Index construction
l max · (1 − c f )
⇒n= log 1+c f + 1. (2)
1−c f l min · (1 + c f ) The data sequence is preprocessed to be partitioned into the
multi-level segmentation structure. Having specified covering fac-
4.2.3. Overlaps of segments tor cf and minimum lmin and maximum lmax query length, the
The size of overlap among segments in each level is selected number n of segmentation levels is calculated according to Eq. (2).
to minimize the number of segments while they all together cf- In each rth level (r ∈ [1, n]), the data sequence is partitioned into nr
cover all possible subsequences bounded in length [lmin , lmax ]. The segments of a fixed length of lr frames by applying Eqs. (1) and (3).
lowest possible overlap we can afford corresponds exactly to the As the data sequence is partitioned, each segment is independently
100 · (1 − c f ) % frames with respect to the segment length. The ini- processed to extract the 4096-dimensional feature vector using the
tial position sij of the jth segment at the rth level is then recur- deep convolutional neural network – see [29] for more detailed in-
sively defined as: formation.
The feature vectors within each level can be also independently
sr1 = 1 srj = srj−1 + l r · c f. (3) indexed to speedup the retrieval process. As the feature vectors are
Lemma 2. The segments at the specific rth level have to be maximally compared by the Euclidean distance, any metric-based index struc-
shifted by lr · cf frames to cf-cover any subsequence of length in [l r · ture can be utilized. We confront the naive sequential scan with
( 1 − c f ) , l r · ( 1 + c f )] . the usage of indexing structure in the experimental evaluation in
Section 6. In addition, the whole multi-level structure is dynamic
Proof. Given the specific rth segmentation level with segments of because it simply enables adding the feature vectors of segments
fixed length lr and an arbitrary subsequence m[i, j] belonging to of new data sequences into each level.
this level, then:

1. We show that the shift between segments about lr · cf frames is 5. Subsequence retrieval in multi-level segmentation structure
of a maximum possible size. Assume that the shift is higher,
i.e., l r · c f + 1 frames, and there exist two consecutive seg- Within the preprocessing phase a user specifies three compul-
ments m [i : i + l r ] and m [i + l r · c f + 1 : i + l r · c f + 1 + l r ] be- sory parameters – covering factor cf, minimum lmin and maximum
r r
tween which subsequence m[i + l ·c2f +1 , i + l ·c2f +1 + l r ] of the lmax length of query – so that the multi-level segmentation struc-
r
same length l is located. Then this subsequence has the same ture could be constructed for a long data sequence. The objective
number of non-overlapping frames with both the segments m of the retrieval phase is to search the long data sequence and lo-
and m  . Considering m and according to Definition 1, then: cate its subsequences that are the most similar to a short query se-
    quence, which is bounded in length [lmin , lmax ]. We approximate lo-
i + l r ·c f +1
− i + i + l r ·c f +1
+ l r − ( i + l r ) calization of such subsequences by traversing the multi-level struc-
2 2
r
≤ cf ⇒ ture and identifying data segments that are the most similar to the
 l r ·c f +1   l r ·c f +1  l
 +  lr · c f + 1 1
query. Although the identified data segments need not be perfectly
2 2 aligned with query-relevant subsequences, they should overlap in
≤ cf ⇒ ≤ c f ⇒ c f + r ≤ c f,
lr lr l the majority of frames, i.e., relevant subsequences are cf-covered
which is not valid for any (positive) length lr of segment.  by identified data segments according to Lemma 1.

Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
JID: IS
ARTICLE IN PRESS
6 J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
Fig. 3. Graphical illustration of segmentation: Based on covering factor c f = 0.2 and query length limits l min = 100 and l max = 500, the four segmentation levels are computed
(left) and used to partition the data sequence (right), where only first- and second-level segments are visualized. E.g., the first-level segments can 0.2-cover any data
subsequence of length in [100, 150].
The retrieval phase evaluates a k-nearest-neighbor (k-NN) query
that is specified by the number k of the most similar data seg-
ments to be returned and sequence mQ as the query object.
The query sequence is firstly preprocessed to extract its 4096-
dimensional feature vector which can be then compared to the fea-
ture vectors of data segments within the multi-level segmentation
structure.
We propose two variants of retrieval algorithms. We firstly in-
troduce a speed-dependent algorithm that returns data segments of
similar length as the query sequence. Then, we present a speed-
invariant algorithm capable of searching for subsequences that can
be executed slower or faster and thus the lengths of corresponding
data segments can differ much more.
5.1. Speed-dependent retrieval algorithm
The speed-dependent algorithm searches for relevant data seg-
ments having a similar length as the query sequence. In particular,
we firstly localize a responsible level containing the segments of a
similar length as the query and then search for query-similar data
segments within such responsible level.
Based on Lemma 1, a segmentation level is responsible for

queries of length within interval [(1 − c f ) · l r , (1 + c f ) · l r ], where
lr is the length of segments within the rth level. In other words,
all possibly-relevant data subsequences of length within such in-
terval are cf-covered by segments in the responsible level. Con-
sequently, the query-responsible level contains the data segments
whose length differs maximally about c f · 100%, e.g., the length of
retrieved data segments is maximally about 20% shorter or longer
with respect to the query length when c f = 0.2. We determine the
index of such responsible segmentation level for the query of |mQ |
frames by the function l evel (|mQ | ) ∈ N as:

1
[m5G;April 18, 2018;12:52]
spect to the query. For example, the same movement can be per-
formed slower or faster and thus their lengths differ significantly.
To solve this problem, we propose to search for relevant data seg-
ments not only in the responsible segmentation level but also in
surrounding levels. The number of surrounding levels to be ac-
cessed is determined by the stiffness parameter sf ∈ (0, 1].
The stiffness parameter quantifies how much shorter and longer
data segments can be retrieved with respect to the original query
length |mQ |. In other words, we want to search for those data seg-
ments that are relevant not only to the original query but also
to its up to s1f -times slower or faster variants. For example, if
s f = 1, retrieval is the same as of the speed-dependent algorithm;
if s f = 0.5, the algorithm searches for motion subsequences that
can be executed up to two times slower or faster.
To guarantee retrieval of s1f -times slower/faster subsequences,
we search for data segments in all levels that are responsible for at
least one query length within interval [|mQ | · s f, |mQ | · s1f ], where
|mQ | stands for the actual query length. The indexes of the respon-
sible levels can be determined as interval [rmin , rmax ] of integers,
where rmin and rmax values are computed on the basis of function
level() as:
1
rmin = l evel |mQ | · s f rmax = l evel |mQ | · . (7)
sf
Considering the example in Fig. 3, the four levels (i.e., rmin = 1 and
rmax = 4) are accessed by the retrieval algorithm for the query of
200 frames and stiffness parameter s f = 0.5.
5.2.1. Query length restrictions
Since the speed-invariant algorithm accesses multiple levels
based on the query length and user-defined stiffness parameter
sf, we need to either (1) create additional segmentation levels for
queries approaching minimum/maximum lmin /lmax bounds, or (2)

 |mQ | ≤ l min · 1+c f
1−c f restrict the admissible query-length interval, without the impact
l evel |m Q
| = |m |·[1−c f ]
Q
1 + log 1+c f l min ·[1+c f ]
otherwise. on the segmentation structure. We focus on the second option by
1−c f
restricting the query length to be within interval:
(6)  1 max

Considering the example in Fig. 3, the second level (i.e., l evel () = l min · ,l · sf . (8)
sf
2) is responsible for all queries ranging from 150 to 224 frames.
For instance, having the original admissible query-length interval
The advantage of this algorithm is that it accesses data seg-
[10 0, 50 0], we limit this interval into a new one [200, 250] by the
ments within only a single segmentation level, which gives an up-
possibility to retrieve up to two times slower/faster subsequences
per bound on search efficiency – search complexity depends on the
(i.e., s f = 0.5).
first level containing the highest number of segments. On the other
hand, if relevant data subsequences are performed slower or faster
5.3. Summary
than the query, they occur in surrounding levels of the accessed re-
sponsible level and thus this algorithm is not able to localize them.
The advantage of both retrieval algorithms against traditional
subsequence search methods is their high efficiency which is en-
5.2. Speed-invariant retrieval algorithm
sured by:
The speed-invariant algorithm solves the problem of finding 1. comparing variable-length segments by fixed-size feature vec-
data segments that are relevant but have different lengths with re- tors,

Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
JID: IS
ARTICLE IN PRESS [m5G;April 18, 2018;12:52]

J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11 7

2. having only one query segment evaluated against data seg- Table 1
Table of symbols.
ments,
3. presenting the results of a k-NN query directly without the Symbol Description
need of any further post-processing, such as matching retrieved m data sequence
and query segments with respect to their temporal order [35]. |m| length of data sequence in number of frames
m[i: j] subsequence of data sequence m starting at the ith frame
From the effectiveness point of view, the great advantage is the (inclusive) and ending at the jth frame (exclusive), i.e.,
possibility to retrieve segments of motions that are performed up |m[i : j]| = j − i
to s1f -times slower/faster or are not perfectly aligned with relevant mQ query sequence
n number of segmentation levels
subsequences up to 100 · cf % with respect to the query length. High nr number of segments within the rth segmentation level
effectiveness is ensured by the used similarity measure that is de- lr length of segments at the rth segmentation level
signed to tolerate imprecise segmentation and accelerated/slowed srj starting frame of the jth segment at the rth segmentation
motions. level
rf replication factor
cf, sf covering factor, stiffness – used-defined parameters
6. Experimental evaluation lmin , lmax minimum/maximum query length – used-defined
parameters
We experimentally evaluate both effectiveness and efficiency
of proposed subsequence retrieval algorithms that combine the Table 2
Analysis of the preprocessing phase of the 68-min subset of the HDM05 dataset
multi-level segmentation structure with the elastic similarity mea-
for the minimum l min = 41 and maximum l max = 2063 query length and different
sure to evaluate query-to-segment similarity. We also compare the settings of covering factor cf. The “rf” column denotes the replication factor, i.e.,
results against existing subsequence matching approaches. how many times each motion frame is part of some data segment.

cf # of # of segments rf Feature
6.1. Dataset
levels total 1st level ext. [min]

Effectiveness and efficiency of both retrieval algorithms are
evaluated on the motion capture dataset HDM05 [38]. This dataset
contains 324 sequences performed by 5 different actors (with sam-
pling frequency of 120 Hz). Similarly as in [1,35], we use a subset of
102 motion sequences (68 min in total) for which a ground truth is
provided. This ground truth describes 1464 actions (subsequences
within the 102 sequences) by 15 non-uniformly populated motion
categories. The shortest action takes only 0.34 s (41 frames) while
the longest one has 17.2 s (2063 frames).
6.2. Methodology
We concatenate 102 sequences into a single 68-min data se-
quence and set the minimum l min = 41 and maximum l max = 2063
query length according to the shortest and longest ground-truth
actions. By considering five settings of covering factor cf ∈ {0.1, 0.2,
0.3, 0.4, 0.5}, five different multi-level segmentation structures are
built.
A segmentation structure is evaluated by k-nearest-neighbor (k-
NN) queries for k ∈ {1, 5}. A k-NN query is constructed for each
of 1464 ground-truth subsequences, that are used as query ob-
jects. Using the speed-dependent or speed-invariant algorithm, the
k most similar data segments are retrieved, excluding the segments
that overlap with the query-object subsequence (i.e., exact match).
We also exclude less-relevant segments that overlap with more rel-
evant ones to finally obtain the k non-overlapping segments as the
query result.
6.3. Analysis of the preprocessing phase
The computationally demanding preprocessing phase primar-
ily involves the extraction of 4096-dimensional feature vectors for
all data segments. The total number of segments depends on the
choice of covering factor cf. Its selection is always a trade-off be-
tween search performance and accuracy and depends on the re-
quirements of a particular application. For instance, lower cf con-
stitutes a higher overlap between consecutive segments and also
a higher number of segmentation levels, implying a higher search
accuracy. On the other hand, the data replication increases since
the number of data segments grows rapidly. Using a GPU imple-
mentation, the extraction of feature vectors takes 25 ms on average
for a single segment. Total extraction times for the 68-min dataset
0.1 18 631,746 111,774 180.0 263.2
0.2 9 150,971 51,230 45.0 62.9
0.3 6 66,972 31,526 20.0 27.9
0.4 5 37,345 21,955 12.5 15.6
0.5 4 23,669 16,393 8.0 9.9
are depicted in the last column of Table 2. The analysis shows that
settings cf ≥ 0.2 can be used for real time processing.
6.4. Analysis of effectiveness
Search effectiveness is measured for each k-NN query by pre-
cision as a ratio between retrieved true-positive segments and all
k retrieved segments. The segment is considered as true positive
if it overlaps with some ground-truth subsequence that is labeled
with the same category as a query object. As the sparsest category
in the dataset has only 6 motion instances, we analyze results for
k = 1 and k = 5. The global precision is then averaged over all 1464
queries. We analyze both speed-dependent and speed-invariant re-
trieval algorithms.
6.4.1. Speed-dependent retrieval
A query is evaluated against data segments from a single level
only to retrieve motions of a similar length as the query. The re-
trieved motions can be in an extreme case at most (1 + c f )-times
longer or (1 − c f )-times shorter. As can be seen in Fig. 4, the pre-
cision @5 ranges from 82% to 87% and increases with a lower set-
ting of covering factor cf. A “finer-grained” segmentation linearly
improves the search accuracy, however, the number of generated
segments grows exponentially, as depicted on the right axis.
We further try to estimate how much error is introduced by the
retrieval algorithm which is primarily influenced by the segmenta-
tion granularity and the quality of similarity measure. To quantify
an error of the similarity measure, we evaluate its accuracy on the
action recognition scenario. The 94.13% precision (i.e., the error of
5.87%) is achieved when the similarity measure is used to eval-
uate 1-NN queries on the ground-truth database of 1464 actions,
without the necessity of any subsequence matching. Compared to
the highest achieved subsequence retrieval precision of 90.12% (for
c f = 0.1 and k = 1), the decrease of 4.01% percentage points in ac-
curacy is introduced by the segmentation. Such error is caused not
only by (1) an imprecise alignment of data segments with respect

Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
JID: IS
ARTICLE IN PRESS
8 J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
Fig. 4. Effectiveness of subsequence retrieval using the speed-dependent algorithm
on the 68-min data sequence with different partitionings (i.e., various settings of
covering factor cf).
to the ground truth but also by (2) the increase in data volume
from 1464 actions to 111,774 segments.
In case of the speed-dependent retrieval algorithm, the covering
factor of c f = 0.2 is recommended because it produces a reason-
able number of data segments (4-times fewer segments) in balance
with a high search accuracy (only 0.5% less) compared to c f = 0.1.
6.4.2. Speed-invariant retrieval
A query is evaluated against data segments originating from
multiple levels. This allows us to retrieve query-relevant subse-
quences that are performed faster or slower. Since the used 1464
ground-truth queries differ within a single category by about 25%
in their lengths on average, we set the stiffness parameter as
s f = 0.8. Such setting improves the search precision @5 about only
0.25% percentage points with respect to the speed-dependent al-
gorithm.
Artificially accelerated/decelerated queries. As the used ground truth
contains only a small number of intra-category motions that
differ at least about 50% in their length, we artificially create
faster/slower queries. A faster query is simulated by omitting some
of its original frames, while a slower one is obtained by adding
some artificial frames into the original motion by interpolating
joint coordinates of surrounding frames. We firstly filter out 745
(out of 1464) queries that violate the query-length restriction
in Eq. (8) for l min = 41, l max = 2063 and fixed stiffness s f = 0.5.
The remaining 719 queries are modified in lengths by acceler-
ating/decelerating them randomly to be up to 2-times faster or
slower.
When the modified queries are evaluated using the speed-
dependent algorithm – which corresponds to the speed-invariant
algorithm with setting s f = 1.0 –, the achieved precision is 85.16%
for k = 5. By using the speed-invariant algorithm, the precision
starts increasing as the stiffness parameter decreases – see Fig. 5.
The highest accuracies around 88.55% are obtained for sf ∈ [0.6,
0.7], which corresponds to an average size of query-length modifi-
cation. Such significant gain in accuracy requires to visit only 2–3
segmentation levels in parallel on average. On the other hand, the
precision decreases for stiffness sf ≤ 0.5 since additionally visited
levels do not already contain relevant results and only increase a
chance for false matches.
6.5. Analysis of retrieval efficiency
Efficiency of the retrieval phase primarily depends on the to-
tal number of generated data segments, which is influenced by the
setting of covering factor cf. In case of the speed-invariant retrieval
algorithm, search times are additionally influenced by user-defined
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
Fig. 5. Effectiveness of subsequence retrieval using the speed-invariant algorithm
on the 68-min data sequence by evaluating artificially modified queries. Orig-
inal queries are artificially modified to be up to 2x faster/slower. This speed-
up/deceleration corresponds to stiffness sf ∈ [0.5, 1].
stiffness parameter sf that determines the number of segmentation
levels to be accessed with respect to a query. Nevertheless, seg-
mentation levels can be accessed in parallel and then search times
of both retrieval algorithms can be, for simplicity, considered as
the same.
We analyze retrieval efficiency in the following three scenar-
ios. First, the feature vectors of all data segments are stored in
main memory and evaluated by the sequential scan approach. Sec-
ond, original feature vectors are transformed into bit vectors to sig-
nificantly reduce space occupancy and speed-up retrieval in main
memory. Third, a disk-based index structure is utilized to scale to
very large databases.
6.5.1. Sequential scan in main memory
The most-populated first segmentation level gives the upper
bound on search performance. Without any index structure, the
feature vectors of data segments of the used 68-min dataset can
be stored in main memory and accessed sequentially. By chang-
ing the covering factor from 0.5 to 0.1, the retrieval process needs
from 66 to 447 ms to evaluate a single query by browsing from
16 k to 112 k data segments (see Table 2 for the number of gener-
ated data segments). Actual search times for particular settings of
covering factor are also presented in Fig. 6 by the “Search time
on original vectors” curve. These search times are measured on
the speed-dependent algorithm by using a single CPU (i7 960 at
3.2 GHz), that can perform approximately 250,0 0 0 distance com-
putations per second. For such small dataset having hundreds of
thousands of data segments, the sequential scan approach is suffi-
cient.
6.5.2. Sequential scan in main memory using the bit-vector
representation
To further significantly speed-up the retrieval process, we trans-
form original 4096-dimensional feature vectors of real numbers
into 4096-dimensional vectors of bits. In particular, each dimen-
sion of a vector is transformed either into bit 1 in case the origi-
nal dimension is a non-zero value, or into bit 0 when the original
value is zero. The transformed 4096-dimensional bit vectors are
then compared for similarity by the Hamming distance. By trans-
forming the original features into bit vectors, we measure the im-
pact on both search effectiveness and efficiency.
Even though our bit-vector implementation is not optimal (each
distance computation creates a new BitSet object in Java), the per-
formance gain is significant. Fig. 6 demonstrates about 20-times
more efficient search of the bit-vector approach in comparison
with the original vectors – a single CPU (i7 960 at 3.2 GHz) can
compute in one second about 5.53 M Hamming distances on bit
JID: IS
ARTICLE IN PRESS [m5G;April 18, 2018;12:52]

J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11 9

Fig. 6. Trade-off between precision (left y axis) and search times (right y axis) for different settings of cf evaluated using the speed-dependent algorithm and 5-NN queries
on original and transformed bit vectors.

vectors compared to 0.25 M Euclidean distances on original vec- time (i.e., up to 1 second) to obtain the most similar subsequence
tors. On the other hand, precision of the bit-vector representation with respect to a query motion.
is only about 2 percentage points worse on average with respect
the original vectors. 6.6. Comparison with state-of-the-art approaches
The bit-vector transformation also significantly decreases the
space occupancy – a single original vector occupies about 16 kB (4 To the best of our knowledge, there is a limited number of ap-
bytes per dimension), while a transformed bit vector takes only proaches able to provide subsequence search in motion data on
0.5 kB (1 bit per dimension), which is 32-times less. This allows us a large scale. Only the following approaches [30,31,33,35,39] fairly
to possibly read a database of 8 M motions (i.e., bit vectors) into evaluate effectiveness and efficiency purely in the scenario of sub-
4GB-main memory and evaluate a query by the sequential scan up sequence matching in motion data. We compare our approach with
to 1.5 s. these papers from both points of view and also with recent well-
performing classifiers in effectiveness and annotation-based meth-
ods that could potentially scale to large volumes of motion data in
6.5.3. Efficiency study on an indexed disk-oriented approach efficiency.
Our objective is to efficiently search motion datasets having a
total length in order of days or even months. However, such large
6.6.1. Effectiveness
motion datasets are still not available, e.g., the whole CMU1 and
As there is no established benchmark for measuring effective-
HDM05 datasets take together only about 12 h of motion data. To
ness of subsequence matching in motion data, it is hard to pro-
estimate search efficiency on large motion data, we adopt the ex-
vide a fair quantitative comparison as dozens of different datasets
perimental evaluation presented by Novak et al. [36]. They intro-
(e.g., CMU,2 HDM05 [38], NTU [40]) are used. Moreover, some
duce an efficient disk-oriented approximate index structure, called
approaches [39] even use their own collection. We evaluate our
the PPP-Codes, and evaluate it on very similar feature vectors. In
approach on the same subset of 15 categories of the HDM05
particular, they use the same reference model of neural network to
dataset as the approach in [35], which achieves the precision of
extract 4096-dimensional feature vectors for common photographs.
83.7%. We outperform this result by reaching the precision of
Even if the domain of photographs is different than of our motion
89.65% using the speed-dependent algorithm (with setting c f =
images, the extracted feature vectors exhibit similar characteristics.
0.2). Even though other related works are evaluated on different
The experiment in [36] indexes 20 million image feature vectors
datasets, they achieve qualitatively comparable or worse results
that are stored on an SSD disk. In the retrieval phase, a single 1-
than our approach. For example, the A-LTK 2.0 [30] reaches the
NN query is evaluated in 800 ms on average, while achieving a 96%
top-match accuracy of up to 80% but on a very small dataset tak-
recall – recall is the percentage of the same vectors retrieved by
ing approximately 10 min in total. The probabilistic PCA-based ap-
the PPP-Codes with respect to the sequential scan approach. Such
proach [31] achieves about the 80% accuracy on average over 5
high recall value is reached by accessing only about 10,0 0 0 vectors
pre-selected categories. The index-based approach [33] uses self-
out of 20 M. The level of PPP-Codes search approximation can be
organizing maps and presents the accuracy of 77% on 13 different
further controlled by a user to find appropriate trade-off between
types of queries from the CMU dataset. The task of subsequence
search effectiveness and efficiency. In general, the PPP-Codes can
searching is performed using a user-controlled GUI in [41] whose
access two orders of magnitude fewer vectors than the sequential
effectiveness is hardly comparable as it is only supported by a field
scan, while achieving a very high recall. More detailed information
study in which domain experts analyze the quality of browsing ex-
about the experiment and the PPP-Codes structure is available in
perience by using different motion perspectives.
[36].
The majority of related work constitutes action-recognition
We could possibly employ the PPP-Codes to index each seg-
methods that are commonly represented by purposely-trained clas-
mentation level independently. By considering the same settings as
sifiers [23,26,27,42]. We show that our approach is competitive to
in Fig. 3 (i.e., covering factor c f = 0.2 and minimum query length
such classifiers. In particular, we employ our similarity measure
l min = 100), the most-populated first level with 20 million data
and assign labels to already segmented queries using a simple 1-
segments of length 125 frames corresponds to about a 48-day long
NN classifier, based on the labels of known training samples. We
motion. Such very long sequence could be then searched in real
evaluate the results on challenging subsets of the HDM05 dataset

1 2
http://mocap.cs.cmu.edu/. http://mocap.cs.cmu.edu/.

Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
JID: IS
ARTICLE IN PRESS
10 J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
Table 3
Comparison in effectiveness with the state-of-the-art classifiers on the HDM05
dataset with 65 and 130 categories.
Method Amount of Accuracy (%)
training data HDM05-65 HDM05-130
Du et al. [23] 90% 96.92 N/A
Zhu et al. [27] 90% 97.25 N/A
Huang et al. [26] 50% N/A 75.78
Laraba et al. [42] 50% N/A 83.33
Our approach 50% 93.52 88.15
containing 65 and 130 categories [29]. The results presented in
Table 3 show that we clearly outperform classifiers in [26,42] by
utilizing the same amount of 50% of training data. On a simpler
subset of HDM05 dataset, which comprises of only 65 categories,
classifiers in [23,27] achieve the classification accuracy of 97.25%
and 96.9% using 90% of training data. We achieve a competitive ac-
curacy of 93.5% using only 50% of training data. Such high-quality
results prove that the used similarity measure is very convenient
to be employed for processing motion capture data, e.g., for search-
ing, subsequence matching, and recognizing actions.
6.6.2. Efficiency
The method presented in [30] does not support indexing and
requires dozens of seconds to evaluate a single query. Similarly
in [39], where more than 1s is needed to evaluate a single query
within the dataset of only 1k motion sequences, which is several
orders of magnitude less efficient than our approach. The indexed
subsequence search in [33] requires 240 ms to evaluate a sin-
gle query within the CMU subset comprising roughly 177k frames
of training data. On the same dataset size, our approach requires
75 ms without indexing per query evaluation, even when using
the original 4096-dimensional vectors. The algorithm proposed in
[31] is much more efficient, taking 72 ms to search a much larger
part of the CMU dataset of roughly 1.7M frames. If we apply the
bit-vector representation on the same-size dataset, we need only
37 ms per query and still in a sequential way. Seemingly the fastest
motion subsequence matching is provided in [34]. They index each
feature (out of 43) within an independent trie-based structure that
can be traversed within 6.5 ms on average. Depending on the num-
ber of user-defined query features, 25–40 ms are needed to search
a 35-hour dataset. However, these times do not include additional
post-processing costs such as accessing and merging candidate re-
sults. On the comparable dataset-size, our PPP-Codes based struc-
ture needs only 3 ms to identify such results. Moreover, the ap-
proach in [34] lacks any evaluation on the search accuracy.
Additionally, our PPP-Codes index based approach has the
bounded time complexity log (|m|/cf) disregarding the query length
|mQ |, where |m| denotes the length of data sequence. This is not
true for approaches whose time complexity increases with the
length of query, such as |m| · |mQ | in [35] or log (|m|/cf) · |mQ | in
[43].
In summary, the synergy of the similarity measure along with
the segmentation technique and bit-vector representation beats
the existing approaches even in a sequential way. By applying
the index structure, our approach can reach much higher effi-
ciency yet and scale to very large datasets. Depending on the ap-
plication scope and hardware requirements, our approach can be
parametrized (using the cf parameter) to achieve an appropriate
trade-off between speed and accuracy. Furthermore, it is designed
to work in main memory or on disk in case of large volumes of
motion recordings.
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
7. Conclusions
We propose a new speed-invariant subsequence matching algo-
rithm that uses a synergy of elastic similarity measure and multi-
level segmentation. The search space comprises overlapping seg-
ments of various sizes that ensure the bounded coverage of arbi-
trary parts within very long motion sequences. The size of overlaps
and the total number of generated segments are bounded to be
formally minimum with respect to the covering factor parameter,
which reflects the versatility and effectiveness of the used similar-
ity measure. Due to the efficient comparison of 4096-dimensional
segment features by the Euclidean distance, the retrieval process is
also very efficient. Even on a standard PC (i7 960 at 3.2 GHz, 4GB
RAM), about a 2-h (for c f = 0.2) motion sequence can be searched
in real time ( < 1 s) for any query ranging from l min = 41 frames to
l max = 2063 frames (17 s). Binarized features in combination with
the Hamming distance enable in-memory real-time search in a
much longer sequence of more than a day. The disk-based PPP-
codes index even increases the searchable length into small dozens
of days. However, sequences in length of months could be searched
up to one second when query-length limits are reduced, a higher cf
value is set, or parallel access of segmentation levels is considered.
Besides high efficiency, we also achieve a high 1-NN search preci-
sion of 90.12% on a 68-min annotated sequence. In contrast to the
existing solutions, our approach is able to localize query-relevant
results that are executed faster or slower.
Acknowledgement
This research was supported by the Czech Science Foundation
grant no. GA16-18889S.
References
[1] M. Müller, A. Baak, H.-P. Seidel, Efficient and robust annotation of motion cap-
ture data, in: ACM Symposium on Computer Animation (SCA 2009), ACM Press,
2009, p. 10.
[2] M.W. Chao, C.H. Lin, J. Assa, T.Y. Lee, Human motion retrieval from hand-drawn
sketch, IEEE Trans. Vis. Comput. Graph 18 (5) (2012) 729–740.
[3] N. Numaguchi, A. Nakazawa, T. Shiratori, J.K. Hodgins, A puppet interface for
retrieval of motion capture data, in: ACM SIGGRAPH/Eurographics Sympo-
sium on Computer Animation, in: SCA 2011, ACM, New York, NY, USA, 2011,
pp. 157–166.
[4] A. Vögele, B. Krüger, R. Klein, Efficient unsupervised temporal segmenta-
tion of human motion, in: ACM Symposium on Computer Animation, 2014,
pp. 167–176.
[5] J. Barbič, A. Safonova, J.-Y. Pan, C. Faloutsos, J.K. Hodgins, N.S. Pollard, Segment-
ing motion capture data into distinct behaviors, in: Graphics Interface, Cana-
dian Human-Computer Communications Society, 2004, pp. 185–194.
[6] D. Bouchard, N. Badler, 7th International Conference on Intelligent Virtual
Agents (IVA 2007), Springer Berlin Heidelberg, pp. 37–44.
[7] R. Lan, H. Sun, Automated human motion segmentation via motion regulari-
ties, Vis. Comput. 31 (1) (2015) 35–53.
[8] C. Faloutsos, M. Ranganathan, Y. Manolopoulos, Fast subsequence matching in
time-series databases, SIGMOD Rec. 23 (2) (1994) 419–429.
[9] Y.-S. Moon, K.-Y. Whang, W.-K. Loh, Efficient time-series subsequence matching
using duality in constructing windows, Inf. Syst. 26 (4) (2001) 279–293.
[10] J.F.S. Lin, M. Karg, D. Kulic, Movement primitive segmentation for human mo-
tion modeling: a framework for analysis, IEEE Trans. Hum. Mach. Syst. 46 (3)
(2016) 325–339.
[11] C. Beecks, M. Hassani, J. Hinnell, D. Schüller, B. Brenger, I. Mittelberg, T. Seidl,
Spatiotemporal similarity search in 3d motion capture gesture streams, in:
14th International Symposium on Advances in Spatial and Temporal Databases
(SSTD 2015), Springer International Publishing, 2015, pp. 355–372.
[12] T.-Y. Kwak, Y.-J. Lee, A filtering method for searching similar multidimensional
sequences under the time-warping distance, Inf. Syst. 28 (7) (2003) 791–813.
[13] B. Krüger, J. Tautges, A. Weber, A. Zinke, Fast local and global similarity
searches in large motion capture databases, in: ACM Symposium on Computer
Animation, in: SCA 2010, Eurographics Association, 2010, pp. 1–10.
[14] J. Valcik, J. Sedmidubsky, P. Zezula, Assessing similarity models for human–
motion retrieval applications, Comput. Animat. Virtual Worlds 27 (5) (2016)
484–500.
[15] C. Beecks, M. Hassani, F. Obeloer, T. Seidl, Efficient query processing in 3D mo-
tion capture databases via lower bound approximation of the gesture matching
distance, in: 2015 IEEE International Symposium on Multimedia (ISM 2015),
2015, pp. 148–153.
JID: IS
ARTICLE IN PRESS
J. Sedmidubsky et al. / Information Systems 000 (2018) 1–11
[16] C. Ren, X. Lei, G. Zhang, Motion data retrieval from very large motion
databases, in: International Conference on Virtual Reality and Visualization
(ICVRV 2011), 2011, pp. 70–77.
[17] L. Ye, E. Keogh, Time series shapelets: a novel technique that allows accurate,
interpretable and fast classification, Data Min. Knowl. Discov. 22 (1) (2011)
149–182.
[18] F. Zhou, F.D. l. Torre, J.K. Hodgins, Hierarchical aligned cluster analysis for tem-
poral clustering of human motion, IEEE Trans. Pattern Anal. Mach. Intell. 35 (3)
(2013) 582–596.
[19] T. chung Fu, A review on time series data mining, Eng. Appl. Artif. Intell. 24
(1) (2011) 164–181.
[20] S. Wu, Z. Wang, S. Xia, Indexing and retrieval of human motion data by a hier-
archical tree, in: 16th ACM Symposium on Virtual Reality Software and Tech-
nology (VRST 2009), ACM Press, New York, NY, USA, 2009, pp. 207–214.
[21] J.-Y. Wang, H.-M. Lee, Recognition of human actions using motion capture
data and support vector machine, in: World Congress on Software Engineer-
ing (WCSE 2009), 1, 2009, pp. 234–238.
[22] M. Müller, T. Röder, M. Clausen, Efficient content-based retrieval of motion
capture data, in: ACM SIGGRAPH, ACM, 2005, pp. 677–685.
[23] Y. Du, W. Wang, L. Wang, Hierarchical recurrent neural network for skeleton
based action recognition, in: Int. Conference on Computer Vision and Pattern
Recognition (CVPR 2015), 2015, pp. 1110–1118.
[24] Y. Wang, M. Neff, Deep signatures for indexing and retrieval in large motion
databases, in: 8th ACM Conference on Motion in Games, ACM, 2015, pp. 37–45.
[25] H. Kadu, C.-C. Kuo, Automatic human mocap data classification, IEEE Trans.
Multimedia 16 (8) (2014) 2191–2202.
[26] Z. Huang, C. Wan, T. Probst, L. Van Gool, Deep learning on lie groups for
skeleton-based action recognition, arXiv:1612.05877 (2016) 1–10.
[27] W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, X. Xie, Co-occurrence fea-
ture learning for skeleton based action recognition using regularized deep
LSTM networks, in: Thirtieth AAAI Conference on Artificial Intelligence, 2016,
pp. 3697–3703.
[28] P. Elias, J. Sedmidubsky, P. Zezula, Motion images: an effective representa-
tion of motion capture data for similarity search, in: 8th International Con-
ference on Similarity Search and Applications (SISAP 2015), Springer, 2015,
pp. 250–255.
[29] J. Sedmidubsky, P. Elias, P. Zezula, Effective and efficient similarity searching in
motion capture data, Multimed. Tools Appl. (2017) 1–22.
[30] K. Sugano, Y. Fang, K. Oku, K. Kawagoe, A coarse-to-fine method for subse-
quence matching of human behavior using multi-dimensional time-series ap-
proximation, in: 17th International Conference on Information Integration and
Web-based Applications & Services, in: iiWAS 2015, ACM, New York, NY, USA,
2015. 34:1–34:9
Please cite this article as: J. Sedmidubsky et al., Searching for variable-speed motions in long sequences of motion capture data, Infor-
mation Systems (2018), https://doi.org/10.1016/j.is.2018.04.002
[m5G;April 18, 2018;12:52]
11
[31] Z. Deng, Q. Gu, Q. Li, Perceptually consistent example-based human motion
retrieval, in: Symposium on Interactive 3D Graphics and Games, in: I3D 2009,
ACM, 2009, pp. 191–198.
[32] P. Zezula, G. Amato, V. Dohnal, M. Batko, Similarity Search: The Metric Space
Approach, Advances in Database Systems, 32, Springer-Verlag, 2006.
[33] S. Wu, S. Xia, Z. Wang, C. Li, Efficient motion data indexing and retrieval
with local similarity measure of motion strings, Vis. Comput. 25 (5) (2009)
499–508.
[34] M. Kapadia, I.-k. Chiang, T. Thomas, N.I. Badler, J.T. Kider Jr., Efficient motion
retrieval in large motion databases, in: ACM SIGGRAPH Symposium on Inter-
active 3D Graphics and Games (I3D 2013), ACM, 2013, pp. 19–28.
[35] J. Sedmidubsky, J. Valcik, P. Zezula, A key-pose similarity algorithm for mo-
tion data retrieval, in: Advanced Concepts for Intelligent Vision Systems (ACIVS
2013), in: LNCS, 8192, Springer, 2013, pp. 669–681.
[36] D. Novak, J. Cech, P. Zezula, 8th International Conference on Similarity Search
and Applications (SISAP 2015), Springer, pp. 237–243.
[37] J. Sedmidubsky, P. Elias, P. Zezula, Similarity searching in long sequences of
motion capture data, in: 9th International Conference on Similarity Search and
Applications (SISAP 2016), Springer, 2016, pp. 271–285.
[38] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, A. Weber, Documenta-
tion Mocap Database HDM05, Technical Report, CG-2007-2, Universität Bonn,
2007.
[39] J.K. Tang, H. Leung, Retrieval of logically relevant 3d human motions by adap-
tive feature selection with graded relevance feedback, Pattern Recognit. Lett.
33 (4) (2012) 420–430. Intelligent Multimedia Interactivity.
[40] A. Shahroudy, J. Liu, T.-T. Ng, G. Wang, Ntu rgb+d: a large scale dataset for
3d human activity analysis, in: The IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2016, pp. 1010–1019.
[41] J. Bernard, N. Wilhelm, B. Krüger, T. May, T. Schreck, J. Kohlhammer, Motionex-
plorer: exploratory search in human motion capture data based on hierarchical
aggregation, IEEE Trans. Vis. Comput. Graph 19 (12) (2013) 2257–2266.
[42] S. Laraba, M. Brahimi, J. Tilmanne, T. Dutoit, 3D skeleton-based action recog-
nition by representing motion capture sequences as 2d-rgb images, Comput.
Animat. Virtual Worlds 28 (2017) 1–11.
[43] J. Sedmidubsky, P. Zezula, J. Svec, Fast subsequence matching in motion cap-
ture data, in: Advances in Databases and Information Systems, Springer, 2017,
pp. 59–72.