Poster Session I
Dealing(with(Long(Utterances:(How(to(Interrupt(the(User(in(
a(Socially(Acceptable(Manner?(
Katharina Cyra
University of Duisburg-Essen
Essen, Germany
katharina.cyra@uni-due.de
ABSTRACT(
Based on two contrasting case analyses we describe prac-
tices of handling long utterances, i.e. turn increments pro-
duced by users. The strategies to handle long utterances are
executed by a human wizard in the context of a speech-
based assistive system – and reveal specific characteristics
of human-human-interaction like precise timing that are not
easily implemented into a technical system. The prelimi-
nary analysis shows two basic strategies of handling long
utterances: 1) to interrupt and 2) to wait-and-see. But the
fine-grained analysis of the participants' perspective shows
yet another dimension when evaluating forms of handling
turn increments: depending on whether user input was taken
up and so, the initiated action is continued, even interrup-
tions (at uncommon points in interaction) are socially ac-
cepted by users. In contrast, the socially 'safe' strategy to
wait-and-see might even cause trouble, when the jointly
worked on task is not continued by the system.
Author(Keywords(
Human-Agent-Interaction; Assistive systems; Social robot-
ics; Natural Language.
ACM(Classification(Keywords(
H.1.2 User/Machine Systems: Human factors; H.5.2 User
Interfaces: Natural language; H.5.2 User Interfaces: User-
centered design.
INTRODUCTION(
Dialogue systems both in human-robot- and human-agent-
interaction (HRI/HAI) often have to cope with the fact that
users tend to produce long utterances when talking naturally
to the system. In comparison to short utterances, or longer
utterances which are ‘well-packaged’ into discrete units [8],
long 'un-structured' utterances are hard to deal with by au-
tonomous systems. Following the idea of exploring a tech-
nical system's possibilities to pro-actively shape the details
of user conduct, [16], [10] and [2] have reconstructed some
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than the author(s) must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specif-
ic permission and/or a fee. Request permissions from Permis-
sions@acm.org.
HAI '17, October 17–20, 2017, Bielefeld, Germany
© 2017 Copyright is held by the owner/author(s). Publication rights
licensed to ACM.
ACM ISBN 978-1-4503-5113-3/17/10…$15.00
https://doi.org/10.1145/3125739.3132586
341
HAI 2017, October 17–20, 2017, Bielefeld, Germany
Karola Pitsch
University of Duisburg-Essen
Essen, Germany
karola.pitsch@uni-due.de
of the circumstances under which such long utterances
emerge, and how these might lead to problems of the dia-
logue component, eventually turning already correctly de-
tected information into false interpretations. Hence, this
paper suggests an initial investigation of the subsequent
practical problem: How could a technical system adequate-
ly deal with long utterances produced by the user?
Image 1: Image of the Embodied Conversational Agent
BILLIE and exemplary appointments entered in the calendar
Research from interactional linguistics investigates natural
language in Human-Human-Interaction (HHI) and studies
show that utterances are not long from the outset but
emerge successively step by step [4]. Due to this additive
processual production, the units of speech are called (turn)
increments. Research shows that turn increments also occur
in HAI within specific interactional contexts. Moreover, it
could be shown that and by which means the system added
to turn production, i.e. when no uptake by the system is
perceivable, users tend to produce turn increments [10, 2].
As shown by Cyra and Pitsch [2] for HAI, these turn incre-
ments stand for different forms of actions performed by the
users such as continued action (i.e. adding more infor-
mation to an initial increment), parallel action (i.e. stating
information for e.g. a successive appointment even though a
previous is not completed) and separate action (i.e. actions
that lie beyond the system's task domain like commenting
or assessing prior actions). To provide conditions under
which a technical system could work best it might be good
to avoid certain of these actions that cause problems for
subsequent actions of the system like separate actions. But
once users successively produce emerging long utterances,
an autonomous system needs strategies to handle them.
In some studies, technical or autonomous systems handle
turn increments by the strategy of 'barging in' [6]. i.e. they
Poster Session I
interrupt user speech. This strategy might be perceived as
socially problematic or impolite and have a negative impact
on (social) acceptance [15]. In contrast, research from HHI
suggests that interruptions are quite common under certain
conditions: Jefferson [7] shows that in the context of turn
taking and overlapping speech of two participants, interrup-
tions occur systematically in specific parts of a turn and are
seldom handled as troublesome by participants. Following
this, typical points for overlaps and resulting interruptions
in HHI are at the beginning or end of an interlocutor's turn
[7] and are precisely timed by human interlocutors. Thus,
when a technical system is expected to function in a social-
ly acceptable manner, e.g. in collaborative actions, such as
to jointly enter appointments with the user, we might want
to carefully design system strategies for turn taking and
handling users’ long utterances. This is the groundwork for
the micro-analysis of two cases of contrary strategies ap-
plied by the system, i.e. interrupting to handle a long turn
(which might be perceived as socially problematic as stated
by [15]) in contrast to a case in which the system (at first)
does not interrupt but 'waits' until a long 'turn' is finished.
DATA(&(METHOD(
The analysis is based on a data corpus of interaction be-
tween human participants and an assistive system with the
task to enter appointments into a virtual calendar presented
on a screen with an Embodied Conversational Agent (see
Image 1). Overall system development aims at developing
an assistive system that might support users with mild cog-
nitive impairments like e.g. mild forms of dementia in time
management and time orientation [17, 1]. The corpus com-
prises audiovisual data, eye-tracker data, and system log
files that report system status changes. Interaction data of
52 participants was gathered (divided evenly in groups of
seniors (SEN), persons with mild cognitive impairments
(CIM) and a student control group (CTL); for a discussion
on possible interactional specificities of the special user
groups see [9]). Data processing comprises annotation and
transcription of speech following GAT conventions [13]
and, where necessary the annotation of relevant multimodal
interactional resources like participants' gaze or posture.
The system was operated by a human wizard, unnoticed by
participants. With the Wizard-of-Oz (WOZ) setup [11] the
study's underlying aim with the focus on interactional prac-
tices of the user and the system could be explored without
having to deal with misunderstandings caused by the Au-
tomatic Speech Recognition (ASR). Furthermore, a WOZ
setup allows to inform future system design by analyzing
interactional practices and strategies performed by the hu-
man wizard [5]. In the given system users had to enter
about 10 to 13 appointments. The system was built with
different appointment conditions that differed regarding the
appointment content (user-initiated appointment entries vs.
system-initiated appointment proposals) and anticipated
system errors (like e.g. misunderstandings, interruptions or
repetitions). To operate the system and for uptake of user
342
HAI 2017, October 17–20, 2017, Bielefeld, Germany
input the human wizard could choose from various verbal
resources, like verbal rephrasing realized by the ECA par-
tially linked to visual representations in the calendar (for
details see [2]). Except where interruption conditions apply,
wizards could operate the system flexibly regarding timing
or overlaps and were instructed to spontaneously decide on
uptake and verbal resources provided via the WOZ GUI.
A detailed micro-analysis of interaction allows to precisely
describe interactional practices of participants while apply-
ing contrary strategies to deal with long utterances. The
analytical method is informed by Conversation Analysis
(CA) and is based on a multimodal perspective on interac-
tion [14]. By following this approach, we focus on the se-
quentiality of interaction and the member’s perspective: We
describe how interaction evolves step by step and recon-
struct how participants subsequently react upon each oth-
er’s multimodal utterances. Likewise, this approach allows
to carve out what contributes to a user's social acceptance as
displayed by participants themselves (in contrast to re-
searcher's external perspective or pre-defined categories).
The following cases are taken from interactions of two
senior participants (SEN) with two contrastive system strat-
egies to handle user utterances. Case 1 shows a strategy of
'wait-and-see' and interruption, case 2 illustrates an 'inter-
ruptions-only' strategy.
CASE( 1:( INTERRUPTION( AND( UPTAKE( OF( RELEVANT(
USER(INFORMATION(AFTER('WAITNANDNSEE'(
The first case analysis (Transcript 1) shows how the tech-
nical system, i.e. the wizard operating the Embodied Con-
versational Agent (E), first contributes to the emergence of
a long user utterance through its own actions by applying a
‘wait-and-see’ strategy. Subsequently, the system needs to
deal with this situation and applies a strategy of interrupting
the participant (P) and makes use of an uptake of input
produced previously by the user. In this case the wizard
could spontaneously manage interaction without special
conditions regarding interruption or uptake.
01 E: WHAT is the next appointment (.)
02 that i should enTER for you- (.)
03 mister LASTname,
04 (0.228)
05 P: the next appointment would be
06 tuesday,
07 (1.552)
08 P: oh tuesday was already right?
09 (1.552)
10 P: no (rather wen-) (.) WEDNESday,
11 (1.561)
12 P: wednesday [DEN-]
13 E: [then] WEDNESday;
14 P: wednesday; (0.448) dentist from eight
15 to nine o'clock;
Transcript 2: WOZ1-SEN-021 / T5 "Dentist - interrupt"
with system (E) interruption in lines 12/13
(a) System contributes to the emergence of a long user
utterance by 'wait-and-see': E initiates the appointment
entry (l. 01-03). After a short pause P tells a parameter for
Poster Session I
[DAY] (l. 05/06: "tuesday") that is followed by a turn
vacant pause [12] in which no uptake (indicated by the
length of the pause in seconds) by E becomes noticeable (l.
07). The system does not show any uptake for another
1.552 seconds even though the user has stated task-relevant
information, so this practice might be interpreted as a 'wait-
and-see' practice. Reacting to the system's (missing) con-
duct P continues his turn with an increment indirectly ad-
dressing E and stating that appointments for Tuesday were
finished (l. 08: "oh tuesday was already right?"),
which is a separate action that might become problematic
as it deals with the assessment of prior user activities [2, 4].
Again in the emerging pause, the system shows no uptake
(l. 09), so P continues his turn with a concurrent parameter
for [DAY] (l. 10 "WEDnesday") followed by a pause (l. 11)
and missing uptake by E. Once again, P produces a turn
increment repeating the [DAY] and initiating the appoint-
ment [TOPIC] (l. 12). This extract shows how the strategy
of wait-and-see and missing uptake by the system (l. 07-11)
contributes to the production of a turn increment with a
length of about 9.641 seconds and concurrent information
on the parameter of [DAY] as shown by [2].
(b) System interrupts user through uptake of user input:
While P begins to state [TOPIC] as another turn increment,
E phrases the corrected parameter for [DAY] previously
named in l. 10 and 12 in overlap with P. In contrast to find-
ings in HHI [7] this overlap is not found at a point that is
common for overlaps like turn beginning or turn end. But
from the perspective of handling turn increments the prac-
tice is effective as P immediately stops producing the next
increment. E's rephrasing contains information that was
stated before by P, which we call an uptake of relevant user
input. This extract points at the wizard's ability to pick the
right, i.e. corrected, information for [DAY] which in the
Transcript 1: WOZ-SEN-022 / T10: "Thursday – interrupt": System repeatedly interrupts user (ll. 04, 06, 09) without taking up
information earlier requested and given by the user. Color blocks in transcript indicate gaze areas: pink = gaze at calendar; green
= gaze at ECA; blue = gaze at table; Brackets around transcript blocks indicate P's posture and show a major posture shift in l.09
343
HAI 2017, October 17–20, 2017, Bielefeld, Germany
context of autonomous systems might lead to difficulties.
(c) User treats interruption as non-problematic: After
E's interruption in overlap, P confirms [DAY] by repeating
the [DAY] (l. 14) and after a short pause incrementally adds
information on [TOPIC] and [TIME] (l. 14/15) without
further reference to the interruption on a verbal or multi-
modal level. This extract shows how the interruption at a
non-typical point in turn production is accepted by P: After
being interrupted there is no hint in the observable conduct
of the participant that the interruption might be a problem.
So, the system's interruption might not be featured with a
precise timing as found in HHI, but the system respectively
the wizard picks up relevant information that help to work
further on the joint task of appointment entry.
CASE( 2:( IMMEDIATE( INTERRUPTION( AND(
NEGLECTION(OF(RELEVANT(USER(INFO(
The second case (Transcript 2) shows how the technical
system, i.e. the wizard operating it, immediately – and pre-
cisely timed at turn-beginning – interrupts the participant's
turn when she starts answering the request for entering an
appointment. In this case we see the condition in which the
wizard had to interrupt user speech at turn beginning with a
repeated request. In contrast to the first case, the user's input
is not taken up by the system. This suggests to the user, that
her input does not seem to be relevant for the system. Inter-
estingly, this appears to result in interactional trouble.
(a) System interrupts user for the first time: The second
case also starts with an appointment initiation by E (l.
01/02). P answers after a short pause with information on
[DAY] (l. 04 "thursday") and, after a micro-pause goes
on to specify [TIME] of the appointment (l. 04). In the
exact same moment E starts to repeat the initial request in
overlap with P's phrasing of the time (l. 05) neglecting the
user input and simultaneous talk. Following Jefferson [7],
Poster Session I
overlaps at turn beginning are common in HHI. This results
in P interrupting her turn production. This practice of 'inter-
ruption' stands in contrast to the 'wait-and-see' practice
described in Case 1: The user's further turn increment pro-
duction is avoided by interrupting here. Besides, we find a
specific gaze pattern described by [2] that can be found
when trouble occurs in the task of appointment entry: when
planning and stating information about the appointment
during E's initial request, P's gaze is directed at the calendar
(CAL). Shortly after E's interruption P's gaze switches from
CAL to E, which was described by [2] as a practice fre-
quently found when interactional trouble occurs.
(b) System repeatedly interrupts user neglecting user
input: After being interrupted and asked for an appoint-
ment entry a second time, P again states information for
[DAY] (l. 07). Again her turn emerges in overlap with E's
continuation of the request (l. 08), which from a timing
perspective is placed precisely at turn beginning again. P
only states information for [DAY] and then breaks her turn
off while E finishes the programmed sentence. This extract
first shows how P orients her turn production toward pauses
for turn-taking (l. 06). It also shows that E's interruption
practice is effective regarding turn length as P even produc-
es a reduced utterance. But when taking into account the
multimodal perspective it becomes clear that P's appoint-
ment planning process (which is observable by P's gaze
directed at CAL), is again interrupted by E's repeated re-
quest. The specific gaze pattern (switching between CAL
and E) again highlights emerging interactional trouble.
When inspecting E's contribution in this context, the repeat-
ed request makes it apparent that the user's input is not
treated as relevant for the system.
(c) User treats interruptions as problematic: In the next
fragment E repeats the appointment request (l. 09) while P
starts to lean backwards, and withdraws her gaze from the
screen to the table. After a micro-pause P starts another
attempt for turn production (l. 10) gazing at CAL, but
breaks off as E continues its turn in overlap with P. While
observing E audio-visually, P projects E's turn end [3] and
produces another turn increment in overlap with E, but
breaks off. In l. 13, after E finishes its turn, P eventually
states information for [DAY] without interruption by E. In
the following turn vacant pause of 3.567 seconds P's gaze
switches between CAL and E while there is no noticeable
uptake by the system. This missing uptake by E eventually
contributes to the production of P's turn increment (l.15).
This extract highlights how interactional trouble occurs. E
interrupts P repeatedly in a very short period of time but at
a common point for overlaps in HHI, and neglects P's con-
tributions for appointment entry as well as her attempts for
turn-taking. On the other hand, the user has to deal with a
non-interruptible system, that produces options for turn-
taking via micro-pauses, but does not stop its own turn.
This points to the key element that causes interactional
trouble: The system repeatedly misses to take up relevant
information that it requested earlier. Instead of taking up
344
HAI 2017, October 17–20, 2017, Bielefeld, Germany
user input as described in Case 1, here the system repeated-
ly goes on asking the initial request. This practice is treated
as troublesome by the participant which becomes observa-
ble by a major posture shift and a gaze pattern that was
described as indicating trouble in earlier research [2]. So,
trouble becomes apparent in different modalities of P's
conduct: a specific gaze pattern indicating trouble, her no-
ticeable change of posture – from leaning forward and gaze
directed at CAL (which is P's predominant position and
orientation during the 10-minute interaction with the sys-
tem) to leaning backwards and withdrawing gaze – and
repeated attempts for turn taking at specific points typical
for HHI. Hence, in contrast to the first case analysis, that
showed that interruptions are acceptable even at uncommon
points of turn production, this case analysis highlights the
necessity to recognize and acknowledge the interlocutor's
contributions, i.e. making relevant user's input.
CONCLUSION(
The initial question was to explore how a system could
handle long utterances. Based on research from HRI and
HHI the detailed analysis of two cases of interruptions
revealed the circumstances and conditions under which two
basic strategies of handling long utterances, i.e. turn incre-
ments, were marked as (non-)acceptable by the users. First,
the analysis of the strategy of wait-and-see which is over-
whelmingly taken as socially acceptable, shows that to wait
and see has the risk of contributing to the production of turn
increments. As 'waiting' could be perceived by users as
missing uptake, this could eventually become problematic
for autonomous systems when users produce turn incre-
ments to handle missing uptake. The analysis of the strategy
of interrupting user speech which in contrast is taken as
socially problematic, points at diverse conditions and inter-
actional contexts in which interruptions might take place.
Analysis shows that interrupting speech per se or precise
timing cannot be taken as the criterion for social ac-
ceptance, as displayed by the human participants. Interest-
ingly, it could be shown that the key element for acceptance
of interruptions to handle turn increments is the proper
uptake and subsequent making relevant of user input and
not a specific interruption strategy and timing.
After this exploratory analysis the next steps are to analyze
whether the findings regarding strategies to handle turn
increments and their acceptancy by users can also be found
in the other user groups. Then a corpus analysis of man-
agement of turn increments with focus on interruptions and
uptake, and a quantification of the described categories to
make predictions on the frequency of these strategies will
follow. Eventually this could be the basis for implementing
the described strategies into an autonomous system to han-
dle turn increments with specific strategies that bypass
timing issues and focus on uptake of (relevant) information.
ACKNOWLEDGEMENTS(
This research was supported by the German Feder-
al Ministry of Education and Research (BMBF) in the pro-
ject KOMPASS. We would like to thank the participants.
Poster Session I
REFERENCES(
1.! Katharina Cyra, Antje Amrhein and Karola Pitsch.
2016. Fallstudien zur Alltagsrelevanz von Zeit- und
Kalenderkonzepten. In Mensch und Computer (MuC)
2016. 1-5.
2.! Katharina Cyra and Karola Pitsch. 2017. Dealing with
'Long Turns' Produced by Users of an Assistive Sys-
tem: How Missing Uptake and Recipiency Lead to
Turn Increments. Accepted for RO-MAN 2017.
3.! Jan-Peter De Ruiter, Holger Mitterer and Nick J. En-
field. 2006. Projecting the end of a speaker's turn: A
cognitive cornerstone of conversaton. In Langu-
age, 82(3), 515-535.
4.! Cecilia Ford, Barbara Fox and Sandra. A. Thompson.
2002. Constituency and the grammar of turn incre-
ments. In The language of turn and sequence, Cecilia
Ford, Barbara Fox and Sandra A. Thompson (eds.).
Oxford University Press, 14-38.
5.! Raphaela Gehle, Karola Pitsch, Timo Dankert and
Sebastian Wrede. 2017. How to Open an Interaction
Between Robot and Museum Visitor?: Strategies to Es-
tablish a Focused Encounter in HRI. In Proceedings of
the 2017 ACM/IEEE International Conference on Hu-
man-Robot Interaction, 187-195.
6.! Fabrizio Ghigi, Maxine Eskenazi, M. Ines Torres and
Sungjin Lee. 2014. Incremental dialog processing in a
task-oriented dialog. In Fifteenth Annual Conference of
the International Speech Communication Association,
308-312.
7.! Gail Jefferson. 1984. Notes on some orderlinesses of
overlap onset. Discourse analysis and natural rheto-
ric, 500: 11-38.
8.! Manja Lohse, Britta Wrede and Lars Schillingmann.
2013. Enabling robots to make use of the structure of
human actions-a user study employing Acoustic Pack-
aging. In RO-MAN, 2013 IEEE. 490-495.
9.! Christiane Opfermann, Karola Pitsch, Ramin
Yaghoubzadeh and Stefan Kopp. 2017. The Communi-
cative Activity of 'Making Suggestions' as an Interacti-
onal Process: Towards a Dialog Model for HAI. Sub-
mitted for HAI 2017. 1-10.
345
HAI 2017, October 17–20, 2017, Bielefeld, Germany
10.! Karola Pitsch, Ramin Yaghoubzadeh and Stefan Kopp.
2015. Entering Appointments: Flexibility and the Need
for Structure. In: Proceedings of the International Con-
ference of the German Society for Computational Lin-
guistics and Language Technology GSCL 2015. 140-
141.
11.! Laurel D. Riek. 2012. Wizard of oz studies in hri: a
systematic review and new reporting guide-
lines. Journal of Human-Robot Interaction, 1(1): 119-
136.
12.! Reinhold Schmitt, (2004). Die Gesprächspause: Verba-
le" Auszeiten" aus multi-modaler Perspektive. In Deut-
sche Sprache 32(1): pp. 56-84.
13.! Margret Selting, Peter Auer, Dagmar Barth-
Weingarten, Jörg Bergmann, Pia Bergmann, Karin
Birkner, Elizabeth Couper-Kuhlen, Arnulf Depper-
mann, Peter Gilles, Susanne Günthner et al. 2011. A
system for transcribing talk-in-interaction: GAT 2.
Gesprächsforschung - Online-Zeitschrift zur verbalen
Interaktion (12): 1-51.
14.! Jack Sidnell and Tanya Stivers (eds.). 2012. The hand-
book of conversation analysis (Vol. 121). John Wiley
& Sons.
15.! Mark Ter Maat, Khiet P. Truong and Dirk Heylen.
2011. How agents' turn-taking strategies influence im-
pressions and response behaviors. In Presence: Tele-
operators and Virtual Environments, 20(5): 412-430.
16.! Anna-Lisa Vollmer, Manuel Mühlig, Jochen J. Steil,
Karola Pitsch, Jannik Fritsch, Katharina J. Rohlfing
and Britta Wrede. 2014. Robots show us how to teach
them: Feedback from robots shapes tutoring behavior
during action learning. PloS one, 9(3): e91349.
17.! Ramin Yaghoubzadeh and Stefan Kopp. 2016. Towa-
rds graceful turn management in human-agent interac-
tion for people with cognitive impairments. In SLPAT
2016. 26-31.