CHI 2008 Proceedings · Special Interest Groups April 5-10, 2008 · Florence, Italy

Vocal Interaction
Sri H Kurniawan Abstract
Department of Computer Engineering Vocal interaction research has slowly been gaining
University of California Santa Cruz popularity in the mainstream HCI, assistive technology,
SOE-3, 1156 High Street arts and game development communities. One main
Santa Cruz CA 95064, USA reason for the uptake of this interaction style is the
srikur@soe.ucsc.edu potential of exploiting one of the most natural means of
expression: human vocalizations, speech, and vocal
Adam J Sporka gestures. This SIG meeting has three purposes: to
Dipartamento di Ingegneria e Scienza communicate the results of the CHI 2007 workshop on
dell’Informazione vocal interaction to interested CHI attendees; to sketch
Univerisità degli Studi di Trento a research agenda on the topic of the emerging
Via Sommarive 14 interaction styles in the context of vocal interaction and
38100 Povo (TN), Italy its implications for the design of interactive systems;
adam.sporka@disi.unitn.it and to bring together the communities of researchers
and practitioners who are addressing this topic.

ACM Classification Keywords

H.1.2 User/Machine Systems - Human factors, Software
psychology, H.5.2 User Interfaces - Theory and
methods, Voice I/O.

General Terms
Human factors, Theory, Design

Copyright is held by the author/owner(s).
CHI 2008, April 5 – April 10, 2008, Florence, Italy
ACM 978-1-60558-012-8/08/04.
Keywords
Speech interaction, vocal interaction, interaction style,
human voice, TTS, ASR
Introduction
Increasingly, the quality of communication between
interactive systems and their users is determined by

2407
CHI 2008 Proceedings · Special Interest Groups
the effectiveness of exchange of information between
these two parties. Faster, more natural and more
convenient means for users and interactive systems to
exchange information are needed. On the user’s side,
interactive system technology is constrained by the
nature of human’s functional abilities. The challenge is
to design devices and types of dialogues that better fit
and exploit the communication-relevant characteristics
of humans [7], some of which have been discussed
extensively in multimedia and pervasive computing
literature, i.e. voice-, gesture- and gaze-based
interactions [9]. This SIG meeting will be looking
specifically at voice-based interaction.
In the past, voice-based interaction was synonymous to
speech-based interaction, with a significant body of
literature reporting the success (with varying degrees)
of speech synthesizers and recognizers. In the late
1980s, two methods of non-verbal outputs emerged:
earcons and auditory icons [12]. More recently, work
on spatial sound has also been reported in various
domains [3]. The use of non-verbal vocalizations as
input was less common in the past, and has only
recently received more attention. Research of vocal
interaction is currently gaining popularity in domains as
diverse as entertainment [5], assistive technology [2,
10, 11], and visual or performing arts [1, 13]. Whilst
vocal interaction is a promising interaction style, there
are many issues that need to be discussed before it can
be successfully integrated into usable and acceptable
interactive styles.
Issues
This SIG meeting intends to investigate vocal
interaction as an emerging interaction style from
several perspectives and with regard to different
April 5-10, 2008 · Florence, Italy
contexts of use and users. The topic is likely to be of
interest to a range of research streams within HCI,
including games, assistive technologies, ubiquitous
computing, ambient intelligence, multimedia,
virtual/augmented reality, multimodal interaction, etc.
The purpose of this interdisciplinary SIG meeting is to
bring together researchers, designers, developers, and
user groups with experience, interest and knowledge in
vocal interaction. We expect to generate a discussion
on how vocal interaction is positioned among other
more established interaction styles and the implications
of this interaction style for the interactive system
design, and more generally experience design.
The larger questions and issues we hope to address
during the SIG meeting are the following:
• What is the state of the art in the research of vocal
interaction?
• In what contexts of use is vocal interaction
appropriate or less appropriate?
• What are the issues that need to be considered
when designing, evaluating, using, and learning
vocal interaction systems?
In particular, we intend to focus on the following topics:
Advantages and challenges of voice versus other
modalities. We will discuss the extent to which the
vocal interaction is beneficial in case of physiological or
situation-induced disability, such as hands-busy
situations [12] etc. It is necessary to assess the
problem of social acceptability and possible presence of
the ambient noise.
2408
CHI 2008 Proceedings · Special Interest Groups
Application domains. Vocal interaction has been used
to represent a virtual input device usable in numerous
applications, including assistive systems (emulation of
mouse [2, 10] and keyboard [11]), data entry,
including non-verbal dictation [8], game control [5],
music training [8, 13], multimodal interaction
enhancement, speech and language therapy, voice
care, and speech disability compensation techniques. It
will also be interesting to discuss the use of vocal
interaction in art where it may be used both as a tool
for artistic expression [1] or a means of control of
interactive installations.
Application platforms. We may discuss technical
requirements to support vocal interaction and the use
of the vocal interaction on different platforms, such as
desktop PC, PDA, or telephony services. Vocal
interaction also has potential in new interaction
paradigms such as ubiquitous computing and ambient
intelligence. However, several issues need to be
considered, such as whether the current mobile
infrastructure can support vocal interaction; or the
current image capturing equipment and processing
good enough to collect and interpret vocal and
multimodal input?
Human factors. It is necessary to find both
physiological and psychological limitations of vocal
interaction, especially the fatigue and precision of input
and the privacy aspect as output. It is also important to
understand cultural and language differences in vocal
interaction and identify possible problems with cultural
interpretation of the use of various vocal features.
April 5-10, 2008 · Florence, Italy
Pre and Post SIG Meeting Activities
Summaries of SIG discussions will be made available at
http://vocal-input.org/, the website we used for the
CHI 2007 workshop. SIG discussions will begin before
and continue after the SIG meeting via a wiki devoted
to the topic of vocal interaction available at
http://vocal-input.org/wiki.
SIG Meeting Format
• Introduction of the SIG goals and the participants.
(10 minutes).
• A short presentation of discussion results of the
CHI 2007 workshop titled “Striking a C[h]ord:
Vocal Interaction in Assistive Technologies, Games,
and More” that we organized and an overview of
the upcoming special issue of the Universal Access
in the Information Society (UAIS) journal titled
“Vocal Interaction: Beyond Traditional Automatic
Speech Recognition”, due to be published in 2008.
(10 minutes).
• Small group discussions, grouped either based on
application domains (e.g. assistive technology,
entertainment, etc) or backgrounds (e.g. more
technically-oriented and more socially-oriented).
Throughout the discussion sessions, participants
will be invited to take notes and record comments,
ideas, and issues on post-its. These will be
attached to A0-sized cardboards placed on the
walls of SIG meeting room, and will later be posted
on the website. (40 minutes).
• Reporting results of small group discussions to the
SIG meeting and further discussion if necessary.
(20 minutes).
2409
CHI 2008 Proceedings · Special Interest Groups
• Discussion on future plans. As the SIG meeting
aims at sketching a research agenda on the topic
vocal interaction, the follow-up activities will be
geared toward building a research community in
this area. We had started this initiative through
building a mailing list of the CHI 2007 workshop
participants and the contributors of the special
issue of UAIS journal, but will be asking whether
there is a need to facilitate full access to the
working papers, tools and deliverables of the
related projects in the website. (10 minutes).
Acknowledgment
We would like to express our thanks to Constantine
Stephanidis for shepherding the submission process of
the special issue of the Universal Access in the
Information Society journal and Susumu Harada for co-
organizing the CHI 2007 workshop on vocal interaction.
Adam Sporka’s research is currently supported by
Marie-Curie excellence grant, within the project
ADAMACH (contract. No. 022593).
References
[1] Al-Hashimi, S. Blowtter: A Voice-Controlled
Plotter. To appear in Proc. HCI 2006 Engage.
[2] Bilmes, J., Malkin, J., Li, X., Harada, S., Kilanski,
K., Kirchhoff, K., Wright, R., Subramanya, A., Landay,
J., Dowden, P., Chizeck, H.: The Vocal Joystick. In Proc.
IEEE Intl Conf Audio, Speech and Sig Proc. Toulouse,
France, IEEE (2006).
[3] Deutscher, M., Hoskinson, R., Takashashi, S., Fels,
S. Echology: an interactive spatial sound and video
artwork. In Proc. MULTIMEDIA 2005. ACM Press
(2005), 937-945.
April 5-10, 2008 · Florence, Italy
[4] Goto, M., Itou, K., Kobayashi, T. Speech interface
exploiting intentionally-controlled nonverbal speech
information. In Proc. UIST 2005, ACM Press (2005), 35-
36.
[5] Hamalainen, P., Maki-Patola, T., Pulkki, V., Airas,
M. Musical computer games played by singing. In Proc
7th Intl Conf on Digital Audio Effects, Naples, Italy,
2004, 367–371.
[6] Igarashi, T., Hughes, J.F. Voice as sound: using
non-verbal voice input for interactive control. In Proc
UIST '01, ACM Press (2001), 155-156.
[7] Jacob, R.J.K., Leggett, J.J., Myers, B.A. and
Pausch, R. Interaction styles and input/output devices.
Behaviour & Information Technology 12, 2 (1993), 69-
79.
[8] Nakano, T., Goto, M., Ogata, J., Hiraga, Y. Voice
drummer: a music notation interface of drum sounds
using voice percussion input. In Proc. UIST 2000, ACM
Press (2005), 49-50.
[9] Maglio, P., Matlock, T., Campbell, C., Zhai, S.,
Smith, B.A. Gaze and speech in attentive user
interfaces. In Proc. ICMI 2000, ACM Press (2000), 1–7.
[10] Sporka, A.J., Kurniawan, S.H., Slavik, P. Whistling
user interface (U3I). In Proc. UI4All 2004, LNCS 3196,
Springer-Verlag (2004), 472-478.
[11] Sporka, A.J., Kurniawan, S.H., Slavik, P.: Non-
speech Operated Emulation of Keyboard. In Clarkson J.,
Langdon, P., Robinson P. (eds) Designing Accessible
Technology, Springer-Verlag (2006), 145–154.
[12] Vilimek, R., Hempel, T. Effects of Speech and Non-
Speech Sounds on Short-Term Memory and Possible
Implications for In-Vehicle Use. In Proc. ICAD’05.
[13] Welch, G.F., Howard, D.M., Rush, C. Real-time
visual feedback in the development of vocal pitch
accuracy in singing. Psychology of Music, 17(2), 146-
157.
2410