Body Motion/Mobile
If Words Could Dance: Moving from Body to Data through
Kinesthetic Evaluation
Shannon Cuykendall, Ethan Soutar-Rau, Thecla Schiphorst, Steve DiPaola
School of Interactive Arts and Technology, Simon Fraser University, Canada
{scuykend, esoutarr, thecla, sdipaola}@sfu.ca
ABSTRACT
The value of integrating the body and experience into tech-
nology has become an important goal in third wave HCI. One
challenge in using experiential design methodologies is un-
derstanding how to transmit and apply the fullness of an ex-
perience into a computational system. We explore the rela-
tionship and meaning that can be created between kinesthetic
experiences, words, and sensor data through the design of an
interactive system that generates poetry from movement data.
POEME (Portrayal of Ephemeral Movement Experiences)
combines words into insightful, whimsical, and witty poetry
that interprets one’s movement. We refined our design of PO-
EME through kinesthetically evaluating sensor data with the
aid of visual analytic tools and confirmed the utility of these
results through incorporating them into an effective machine
learning system. Through this combination of visual analytics
and kinesthetic evaluation, we contribute to the exploration
of innovative design practices that both embody and evaluate
movement interaction design strategies.
ACM Classification Keywords
H.5.2 User Interfaces: User-Centered Design, Prototyping;
H.5.3 Evaluation/Methodology: ,Web-based Interaction
Author Keywords
design; kinesthetic experience; expressive movement
interaction; movement classification; visual analytics;
INTRODUCTION
As motion sensors become more ubiquitous, there is a growing
need to understand how to design for full-body movement
interactions that extend beyond basic gestural interfaces. This
has given traction to the idea that, in HCI, designers must
experientially understand movement in order to design for it.
As stated in Hummels et al (2007), “One has to be or become
an expert in movement, not just theoretically, by imagination
or on paper, but by doing and experiencing while designing”
[7].
While experiential methodologies support designers in creat-
ing movement-based interactions [15, 6, 11], challenges still
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
DIS 2016, June 04–08, 2016, Brisbane, QLD, Australia
Copyright is held by the owner/author(s). Publication rights
licensed to ACM.
ACM 978-1-4503-4031-1/16/06. . . $15.00
DOI: http://dx.doi.org/10.1145/2901790.2901822
234
DIS 2016, June 4–8, 2016, Brisbane, Australia
remain in understanding how to transmit or incorporate bod-
ily knowledge into design. Kristina Höök points out, “...the
experiences I am trying to describe are wordless, and putting
detailed descriptions of them still fails to cover the complex-
ities and the uniqueness of my embodied experience” [6].
Helena Mentis and Georges Khut have also described a similar
challenge of transferring felt bodily experiences to the design
of technology [2].
Movement classification systems such as Labanotation or La-
ban Movement Analysis (LMA) are often used as intermediate
steps in translating and making sense of bodily experiences.
However, even Rudolf Laban (1879-1968), whose theories
led to the development of these classification systems, states,
“Many dance creations have perished because it was impossi-
ble to describe them in words and we hardly know how the
dances of former epochs were presented. Had descriptions
been attempted, as in fact they have, they could only have been
superficial and inadequate” [9]. While it may be difficult to
translate the nuances of a bodily experience into words, we
propose that the perceived gulf between words and movement
is not as wide as commonly believed.
We explore the relationship between words and move-
ment through our design of POEME (Portrayal of
Ephemeral Movement Experiences). POEME is a website
(https://poeme.iat.sfu.ca) that generates poetry in response to a
user’s movement. POEME records a user’s movement through
an IMU (Inertial Measurement Unit) that integrates an ac-
celerometer, a gyroscope, and a compass. This type of sensor
is included in many mobile devices and transparently used in
everyday interactions.
To create an embodied system that integrates our experience
of movement, we developed a novel methodology to kines-
thetically interpret sensor data with the aid of visual analytics
(VA) tools. VA tools provide a way for humans to more easily
analyze and interpret “big data.” One short movement phrase
or gesture can produce an overwhelming large amount of data
that is challenging to interpret in relation to experience. VA
tools can provide a way for designers to understand how their
experiences are being represented through data. Using VA
tools we compared different aspects of a kinesthetic experi-
ence when using POEME including: our kinesthetic memories,
first-person written reflections of the movement, numerical
data from the IMU, and the poetic verse that initially inspired
the movement interaction.
A strong current in recent VA research is how interaction with
visual representations of data can lead to knowledge generation
[14]. In POEME, knowledge is accumulated by documenting
Body Motion/Mobile
the results of kinesthetic evaluation and translation of these
insights into computational models. We call this novel method
of analysis an embodied VA approach to design. We propose
this methodology can help designers more seamlessly move
between body and data in the design process.
BACKGROUND
The overarching motivation behind creating POEME is to
further explore the complex interaction between words and
movement. Words, like movements, have layers of meaning
and can be recombined to create a multitude of sentences
or sequences that all vary in their underlying meaning. As
philosopher Mark Johnson states, “It is true that when we read,
we read words. But words have meanings, and meanings go
far beyond words” [8]. The same could also be said for bodily
movement.
POEME is heavily inspired by the Japanese form of poetry
known as haikai no renga (comic, collaborative poetry) which
involves the social creation of poems through turn taking [3].
Creating these linked poems requires participants to respond
to previous stanzas with original verse. In POEME, movement
and poetry take turns in what we call a “movement poem.”
Each verse generated in POEME is a Parts of Speech Poem
(PoSP). The PoSP template allows for very simple production
of verses and ensures that each movement poem describes a
subject, actions and qualities of these actions. While sophisti-
cated automatic poem generators exist, the point of POEME
is not to create human-like poetry, but instead to create poetry
that inspires and responds to a large range of movement expe-
riences. Each poem begins with a noun which is followed by
two adjectives, two verbs, and an adverb (See Figure 1). This
structure may not be able to adequately describe all kinesthetic
experiences; however, the PoSP template allows us to examine
word and movement relationships in a variety of contexts and
results in cohesive poems even with unusual word pairings.
Figure 1. Examples of POSP generated by POEME. ©Shannon Cuyk-
endall and Ethan Soutar-Rau 2015
POEME is meant to crowdsource a large corpus of movement
by engaging people to move in response to its poetry. This
lead us to design it as a mobile website which trades fidelity
of data for ubiquity of platform. Still, prior research has
found success in recognizing movement qualities from just
an accelerometer [16]. We approached the design of POEME
235
DIS 2016, June 4–8, 2016, Brisbane, Australia
with the understanding that we would need to compensate for
the limitations of our data source with our own insight.
KINESTETHIC EVALUATION
We used an auto-ethnographic methodology to support and
refine our design of POEME. This approach was chosen so
that we could understand how our kinesthetic experiences
are linked to both IMU data and poetic verse. Many HCI
researchers have stressed the importance of incorporating bod-
ily experience into design, suggesting that movement cannot
be fully understood from afar; therefore, auto-ethnographic
approaches are commonly used to design for movement (e.g.
[6, 13]). One major critique of this methodology is that the
results are not representative of a diverse population. In our
study we mitigate this challenge through having the first two
authors (SBC and ESR respectively) both engage in the auto-
ethnographic process. SBC and ESR have very different move-
ment backgrounds. SBC has a classical dance background;
while, ESR’s movement background is based primarily in
athletic activities, such as biking and soccer.
In the following sections we describe our auto-ethnographic
approach as a series of two studies. The outcome of the first
study was the development of our novel kinesthetic evaluation
technique that we refer to as an embodied VA approach to
design. In the second study we evaluate this approach by
incorporating our findings from the first study into a machine-
learning system.
Study 1: Development of Embodied VA Approach
We used VA tools to find features in the IMU data and com-
pared them with our kinesthetic memory, written descriptions
of the movement, and poetic verse. We describe this method-
ology as a series of three steps.
Step 1: Move in Response to Poetic Verse
SBC and ESR responded through movement to fifty PoSP,
created both by the POEME system and also human authors.
We chose to blend human authored and generated verse to
see if we preferred moving in response to one over the other.
As stated previously, the intent of POEME is not to create
human-like poetry, but to create poetry that could effectively
respond to and inspire a broad range of movement. We found
that both generated and human-authored poetry inspired rich
kinesthetic experiences.
We recorded our movement responses with the IMU in a mo-
bile phone. We explored with placing the sensor in various
places on the body including: placing the phone in our pants’
pockets, holding it in our hands and changing between right
and left sides. For the purposes of this study, we decided to
strap the the mobile device to the right arm for consistency.
While it would be very interesting to compare how our inter-
pretations of the data differ based on the location of the sensor,
the point of this initial study was to first establish how IMU
data and words could be linked to our kinesthetic experiences.
Step 2: Reflect on our Experience of Moving
Immediately following each movement recording, both au-
thors reflected on their kinesthetic experience by describing
their experience in words. Our movement descriptions were
Body Motion/Mobile DIS 2016, June 4–8, 2016, Brisbane, Australia

informed with the Language of Dance (LOD) movement clas-

sification system [5], which is based on Laban’s movement
theories and further developed by Anne Hutchinson Guest. In
this framework, Guest correlates parts-of-movement to parts-
of-speech. Our descriptions of movement included movement
nouns, verbs and adverbs in order to form a comprehensive
and easily comparable account of the kinesthetic experience.
For an example of these descriptions see Figures 2, 3, 4, and
5.
Step 3: Kinesthetically Interpret Data
We used the VA tool JMP [1] to create multiple graphical
Figure 2. Two similar kinesthetic experiences depicted through range of
representations of the movement data. This included scatter acceleration
plots, line graphs, bar graphs and histograms of measurements
of acceleration, rotation and orientation. We compared these
graphical representations side by side with the movement
descriptions and the poetic verse.
After comparing these descriptions with the data, we selected
the features that best aligned with our experiences of the move-
ment. Our criteria for selection was that the movement that
felt most similar should also have similar visualizations. This
is a key feature to our embodied VA approach that allowed us
to select movement features that could be both felt and seen.
Through our embodied VA approach both SBC and ESR found
that the range of acceleration, depicted on three axes through a Figure 3. Two different kinesthetic experiences depicted through range
bar graph, most closely related to our kinesthetic experiences. of acceleration
It is important to note that SBC and ESR rarely responded
to poetic verse in a similar manner; however, both of them
putational theory about the relationship between a statistical
agreed that the range of acceleration best correlated to their
summary of a movement and a poem that would express the
experience. Although bar graphs may appear counter-intuitive
same movement. Creating such a theory from scratch would
to represent dynamic experiences, they are simple representa-
be a daunting task, but the results of Study 1 suggest that
tions that are highly perceptually effective [17, 10]. In contrast
the ranges of acceleration across the three axes measured by
to prior work where designers have sought to model individual
the IMU could potentially be a robust method of statistically
qualities or actions of movement that make up an experience;
summarizing a kinesthetic experience. Study 2 evaluates the
in POEME we seek to characterize whole kinesthetic experi-
predictive efficacy of a theory based on these ranges versus
ences. The bar graph allowed us to more easily see the kines-
theories based on other summary statistics.
thetic experience as a whole, rather than separated moments
in time. Step 1: Create Legible Interactions
In Figure 2 we show similar kinesthetic experiences, as felt To understand how the accuracy of POEME could be evaluated
by SBC, juxtaposed with the visual depiction of the range of we needed to create more legible interactions. To do this
her acceleration and the poetic verse. In Figure 3 we show we curated two wordlists that depicted different themes of
different kinesthetic experiences, as felt by SBC, juxtaposed movement. The first wordlist, which we call the “stillness list”
with the visual depiction of the range of her acceleration and includes words such as, frozen, sustained, still, and serene.
the poetic verse. In these graphical representations the dif- These words could potentially describe little or no noticeable
ferent experiences are clearly distinguishable and the similar movement. The second wordlist, which we refer to as the
experiences are not as distinct. “motion list” includes words related to the physics of motion
such as, buoyant, centripetal, accelerated, forced, and collision.
In Figures 4 and 5 we demonstrate how the graphical repre- The stillness list contains 75 words and the motion list contains
sentation of mean acceleration related to SBC’s kinesthetic 250 words. Both lists include words from all parts of speech.
experiences. SBC did not feel these graphical representations We chose to use descriptive words in this study so that we
as strongly correlated to her experience because different kines- could more easily evaluate if the poetry responded effectively
thetic experiences are not as distinct and similar kinesthetic to our movement.
experiences are not as analogous.
Step 2: Train POEME using a Naive Bayes Algorithm
Study 2: Applying our Findings to a Machine Learning To train POEME the first two authors interpreted POEME’s
generated verse into movement, created from either the still-
System
ness or motion list. We used a simple Gaussian Naive Bayes
If a poem can serve as a method of expressing a kinesthetic algorithm [12] to classify movement. We will refer to this
experience, then it should be possible to construct a com- construct of language theory, training procedure, and verse

236
Body Motion/Mobile
Figure 4. Two similar kinesthetic experiences depicted through mean
acceleration
Figure 5. Two different kinesthetic experiences depicted through mean
acceleration
authoring process as the naive agent. It considers each word
placement in the PoSP template independently and can be
configured to map a variety of measurements to poetic verse.
Step 3: Evaluate Accuracy of Predictions
First, the naive agent interpreted a corpus of movement phrases
using the range, mean, and median of acceleration. Second,
the agent classified which wordlist a movement phrase corre-
sponded to (See Table 1). Third, the agent created a sample
of likely text of the poem (See Table 2). The classifications
were scored for accuracy and logistic loss. The accuracy of
the predicted text in each word placement was also measured.
Accuracy Log Loss
Range 0.97 0.28
Mean 0.81 0.37
Median 0.77 0.46
Table 1. Accuracy metrics are shown for the Naive Bayes algorithm pre-
dicting which word list that movement corresponds to when using vari-
ous summary statistics. Higher accuracy and lower log loss values are
better.
DISCUSSION
Effective quantitative evaluation of POEME is challenging
because an ideal measurement would require a more sophisti-
cated model of the relationship between movement and poetry
than we currently possess. These two studies illustrate how the
embodied VA approach can be used for feature selection tasks
in the absence of satisfactory objective measures. In Study
1, kinesthetic experience served as the basis for selecting the
range of acceleration as a feature to summarize movement.
237
DIS 2016, June 4–8, 2016, Brisbane, Australia
In Study 2, incorporating this insight allowed us to create a
successful machine learning system.
Unexpectedly, the naive agent’s predictions were much more
accurate for certain word positions in the PoSP template than
others. The range of acceleration, which the authors found to
be most congruent with their kinesthetic experiences in study
1, performed the best on the first word placement (See Table
2). We take this to mean that word placement affects how the
words are interpreted in the overall meaning of the verse. We
also speculate that the paired adjectives and paired verbs are
much harder to accurately predict because the intervening con-
junction influences how the words are interpreted in relation to
one another. Our Naive Bayes model is especially sensitive to
disruption from these logical dependencies because it models
all of the word placements independently. Anecdotally, the
naive agent seems able to compose verses in a graduated way
between still and more energetic movement, and the agent
also appears to make distinctions within each wordlist. For
example, words such as springlike and weightless are com-
monly displayed in response to light, jumping movements. It
may not be a coincidence that these are literally descriptors of
acceleration characteristics.
0.74 (noun)
0.60 (adj.) 0.23 (conj.) 0.65 (adj.)
0.47 (verb) 0.15 (conj.) 0.47 (verb)
0.51 (adverb)
Table 2. Accuracy measurements are shown for the naive agent predict-
ing each word of a verse using range acceleration values to summarize a
movement experience.
While these findings are intriguing, it is also important to
keep in mind that the original reason for embarking on the
embodied VA approach was to avoid a hasty literal mapping
of words to movement and premature conceptions of “right”
and “wrong” responses. We suggest that other methods of
evaluation may better serve the POEME system since our
intent is to use POEME as a probe to further explore the
relationship between movement and words. Boehner et al
(2008) suggests rethinking evaluation techniques in HCI to
include aesthetic experience [4]. Similarly, the embodied VA
approach should be considered as an alternate way to design
interactive systems which include kinesthetic experience.
CONCLUSION
Articulating the fullness of the movement experience in a way
that effectively transmits the richness of the experience is an
ongoing challenge to designers of interactive systems. We
contribute to the field of HCI by developing a novel design
approach that incorporates kinesthetic evaluation into the pro-
cess of selecting movement features that can be both felt and
seen in visualizations of data. The features chosen through
our embodied VA approach proved to be effective measures
when implemented into a machine-learning system.
Body Motion/Mobile
REFERENCES
1. JMP®, Version 10. SAS Institute Inc, Cary, NC.
2. Alaoui, Sarah Fdili and Schiphorst, Thecla and
Cuykendall, Shannon and Carlson, Kristin and Studd,
Karen and Bradley, Karen. 2015. Strategies for Embodied
Design: The Value and Challenges of Observing
Movement. In Proceedings of the 2015 ACM SIGCHI
Conference on Creativity and Cognition. ACM, 121–130.
3. Basho, Matsuo. 1998. Basho’s Narrow Road: Spring and
Autumn Passages. Stone Bridge Press, Inc.
4. Boehner, Kirsten and Sengers, Phoebe and Warner,
Simeon. 2008. Interfaces with the ineffable: Meeting
aesthetic experience on its own terms. ACM Transactions
on Computer-Human Interaction 15, 3 (Nov. 2008), 1–29.
DOI:http://dx.doi.org/10.1145/1453152.1453155
5. Guest, Ann Hutchinson. 2005. Labanotation: the system
of analyzing and recording movement (4th ed., rev ed.).
Routledge, New York.
6. Höök, Kristina. 2010. Transferring Qualities from
Horseback Riding to Design. In Proceedings of the 6th
Nordic Conference on Human-Computer Interaction:
Extending Boundaries (NordiCHI ’10). ACM, New York,
NY, USA, 226–235. DOI:
http://dx.doi.org/10.1145/1868914.1868943
7. Caroline Hummels, Kees CJ Overbeeke, and Sietske
Klooster. 2007. Move to get moved: a search for methods,
tools and knowledge to design for expressive and rich
movement-based interaction. Personal and Ubiquitous
Computing 11, 8 (2007), 677–690.
8. Johnson, Mark. 2008. The meaning of the body:
Aesthetics of human understanding. University of
Chicago Press.
9. Laban, Rudolf. 1980. The Mastery of Movement (fourth
edition ed.). Macdonald and Evans, Plymouth.
10. Mackinlay, Jock. 1986. Automating the Design of
Graphical Presentations of Relational Information. ACM
Trans. Graph. 5, 2 (April 1986), 110–141. DOI:
http://dx.doi.org/10.1145/22949.22950
238
DIS 2016, June 4–8, 2016, Brisbane, Australia
11. Moen, Jin. 2007. From Hand-held to Body-worn:
Embodied Experiences of the Design and Use of a
Wearable Movement-based Interaction Concept. In
Proceedings of the 1st International Conference on
Tangible and Embedded Interaction (TEI ’07). ACM,
New York, NY, USA, 251–258. DOI:
http://dx.doi.org/10.1145/1226969.1227021
12. Pedregosa, F. and Varoquaux, G. and Gramfort, A. and
Michel, V. and Thirion, B. and Grisel, O. and Blondel, M.
and Prettenhofer, P. and Weiss, R. and Dubourg, V. and
Vanderplas, J. and Passos, A. and Cournapeau, D. and
Brucher, M. and Perrot, M. and Duchesnay, E. 2011.
Scikit-learn: Machine Learning in Python. Journal of
Machine Learning Research 12 (2011), 2825–2830.
13. Pijnappel, Sebastiaan and Mueller, Florian. 2013. 4
Design Themes for Skateboarding. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI ’13). ACM, New York, NY, USA,
1271–1274. DOI:
http://dx.doi.org/10.1145/2470654.2466165
14. Sacha, D. and Stoffel, A. and Stoffel, F. and Bum Chul
Kwon and Ellis, G. and Keim, D.A. 2014. Knowledge
Generation Model for Visual Analytics. Visualization and
Computer Graphics, IEEE Transactions on 20, 12 (Dec
2014), 1604–1613. DOI:
http://dx.doi.org/10.1109/TVCG.2014.2346481
15. Schiphorst, Thecla and Andersen, Kristina. 2004.
Between Bodies: Using Experience Modeling to Create
Gestural Protocols for Physiological Data Transfer. In
Proc. CHI 2004 Fringe. ACM Press.
16. Diego Silang Maranan, Sarah Fdili Alaoui, Thecla
Schiphorst, Philippe Pasquier, Pattarawut Subyen, and
Lyn Bartram. 2014. Designing for Movement: Evaluating
Computational Models Using LMA Effort Qualities. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ’14). ACM, New
York, NY, USA, 991–1000. DOI:
http://dx.doi.org/10.1145/2556288.2557251
17. Tufte, Edward R and Graves-Morris, PR. 1983. The
visual display of quantitative information. Vol. 2.
Graphics press Cheshire, CT.