See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261397584

Novel musical notation for emotional sound expression of interactive robot

Conference Paper · November 2012

DOI: 10.1109/URAI.2012.6462939

CITATIONS READS

0 67

3 authors, including:

Kyungsu Chun
Korea Advanced Institute of Science and Technology
4 PUBLICATIONS   1 CITATION   

SEE PROFILE

All content following this page was uploaded by Kyungsu Chun on 10 December 2015.

The user has requested enhancement of the downloaded file.

2012 9th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI)
The 9th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2012)
Daejeon,
Nov. 26-28,Korea / November
2012 in 26-29, 2012
Daejeon Convention Center(DCC), Daejeon, Korea
‫ٻ‬
Novel Musical Notation for
Emotional Sound Expression of Interactive Robot
Kyoung Soo Chun1, Eun-Sook Jee2 and Dong-Soo Kwon3
1
Graduate School of Culture Technology, KAIST, Daejeon, 305-701, Korea
(Tel : +82-42-350-3082; E-mail: chunks@robot.kaist.ac.kr)
2
Department of Mechanical Engineering, KAIST, Daejeon, 305-701, Korea
(Tel : +82-42-350-5251; E-mail: esjee@kaist.ac.kr)
3
Department of Mechanical Engineering, KAIST, Daejeon, 305-701, Korea
(Tel : +82-42-350-3042; E-mail: kwonds@kaist.ac.kr)

Abstract - Through the voice, people can understand each Modern music notation is not suitable to describe the
other’s emotional states. Since the auditory component has continuous change of pitch and tone in detail.
an older communicative history than the visual, the For example, [4] analyzed the sound of robots, R2-D2
expression using sound is more direct primarily. Therefore, and Wall-E, from two popular movies, Star Wars and
sound expression should be one of important part of Wall-E, respectively. [4] used modern music notation to
human-robot interaction research. In this paper, we depict robotic voices. However, the voice of the robots is
propose a musical notation based on spectrogram. With hard to describe using the classic notation, especially
this notation, we designed 30 patterns of sound for continuous change of pitch and timbre.
expression of the English teacher robot. Each note of classic notation shows only discrete
change of pitch. Although the terms glissando and
Keywords - Musical Notation, Emotional Sound, portamento describe a glide from one pitch to another,
Interactive Robot, HRI they only indicate start and end point. That is, they don’t
indicate how to glide, linearly or non-linearly.
1. Introduction
Sound is a part of expression; it can be used to express
different information and to communicate with humans.
It’s easy to control the impact and the user’s feeling using
sound easier than it is using gesture. Many applications
can use sound to increase the impact of different
information, including emotional information [1]. ‫ٻ‬
‫ٻ‬
Through the voice, people can understand each other’s
Fig. 1. Example of classic notation. top: La sonnambula
emotional states. Since the auditory component has an
(1831, V. Bellini), bottom: Duke Bluebeard's
older communicative history than the visual, the
Castle (1912, B. Bartok)
expression using sound is more direct primarily [2].
Therefore, sound expression should be one of important
2.2. Waveform
part of human-robot interaction research. In this paper, we
propose a musical notation based on spectrogram. With
Waveform, intensity versus time graph visualizes the
this notation, we designed 30 patterns of sound for
continuous signal. However, it is not intuitive for sound
expression of the English teacher robot.
design when the signal has more than one pitch(i.e.,
polyphony) makes harmony. Moreover, if a monophonic
2. Background signal changes the pitch by time, we cannot easily analyze
the signal.
Previous studies of robot mostly consider the sound
expression in view of paralanguage which refers to the
non-verbal elements of communication used to modify
meaning and convey emotion [3-4].
Previous studies mostly use modern music notation
originated in European classical music or waveform graph
[3-4]. We will compare and discuss existing notations and
graphs for sound expression. ‫ٻ‬
‫ٻ‬
2.1. Modern Music Notation Fig. 2. Waveform of Eq. (1)

‫ٻ‬978-1-4673-3112-8/12/$31.00 ©2012 IEEE

88
 ‫ٻ‬

For instance, the wave in Fig. 2. consists of two sine Since spectrogram only shows change of pitch for a
signal and one chirp signal with same amplitude shown as single non-harmonic and monophonic signal, we added
Eq. (1) more parameters which affect timbre of the basic sound
signal (e.g., sine, square, and sawtooth waveform.). It
y = sin(2π (230))
t + sin(2π (300))
t + sin(2π (f(t)))
t , (1) includes cutoff frequency of lowpass / highpass / bandpass
0 ≤t≤ 2 filter, oscillation frequency of low frequency oscillator
(LFO), modulation frequency of amplitude/frequency
where modulator, and level of delay.
Because many effect units have two or more parameters,
f(t) = 75(tri(2π (0.5)(t − 0.5))+ 1) (2) each parameter is represented as different color in one
graph shown as Fig. 4. Users can define which parameter
is used in one graph. In this way, they can read and write
( ) is a triangle wave function with period 2π ‫ٻ‬
and trit
sound in many points of view and have a lot of freedom.
and varying between -1 and 1.
For sound designers, it is difficult to get features only
from the waveform graph. 4. Expression for English Teacher Robot

2.3. Spectrogram We asked a non-classically-trained designer to make 30

Spectrogram is a time-varying representation that
shows the spectral density of a signal varies with time. It is
useful to analyze change of frequency by time. The graph
in Fig. 3. shows the same signal as Fig. 2. in more intuitive
way because the total signal consists of three sinusoidal
components.
Fig. 3. Spectrogram of Eq. (1)
Still, however, if each basic monophonic signal has
harmonics such as sawtooth or square wave, spectrogram
will be more complex.
Therefore, we propose a notation based on spectrogram
which is a time-varying representation that shows the
spectral density of a signal varies with time.
3. Novel Musical Notation
sound expressions for English teacher robot using our
proposed notation. After 30-minute instruction, the
designer made whole 30 expressions within 3 hours
without any problem.
5. Discussion and Further works
We took advice from professional musicians and sound
designers about prototype of our notation.
They commented it is intuitive and leaves complete
freedom. Also, they commented if they have to design
sound and can’t read music, they will use our notation
instead of classic notation.
In future, we will complete our notation system and run
a usability test for professional and non-professional
designers. We will use this notation for sound generation
from internal state of robot for our next study.
6. Acknowledgement
This paper is financially supported by the INNOPOLIS
Foundation through the research of “The Promotion of
English Education Market with Service Robot”.
We would like to thank Jamie Glaser, Jukka-Pekka
Kervinen, and Peter Thörn for advice. We would like to
thank Seri Sun for sound design of our English teacher
robot.

References
‫ٻ‬
[1] B. Bramas, Y.M. Kim, D.S. Kwon, “Design of a Sound
System to Increase Emotional Expression in
Human-Robot Interaction”, International Conference
on Control, Automation and Systems, 2008.
[2] N. D. Cook, Tone of Voice and Mind, John Benjamins
Publishing Company, Amsterdam/Philadelphia, 2002.
[3] G. Johannsen, “Auditory Display of Directions and
States for Mobile Systems”, International Conference
on Auditory Display, 2002.
‫ٻ‬ [4] E.S. Jee, Y.J. Jeong, C.H. Kim, H. Kobayashi, “Sound
‫ٻ‬ design for emotion and intention expression of socially
Fig. 4. Example of a notation proposed in this paper. interactive robots”, Intelligent Service Robotics, 2010.

‫ٻ‬
89

View publication stats