10

M eaning in M usical Gest ure

Fernando Iazzetta
Universidade de São Paulo
Departamento de M úsica - ECA
Rua da Antiga Reitoria, S/ N 433
05508-900 - São Paulo - SP - Brazil
iazzetta@usp.br

How does it come t hat w hen someone volunt arily list ens t o a song w it h ears and mind, he is also
involunt arily t urned t ow ard it in such a w ay t hat his body responds w it h mot ions somehow similar t o t he
song heard? (Boet hius, 1989: 8).

Introduction

Technique and t echnology are t w o cult ural aspect s t hat have been deeply involved w it h music, not only in
relat ion t o it s product ion, but also in relat ion t o t he development of it s t heory and t o t he est ablishment of
it s cult ural role. Since t he beginning of t he t w ent iet h cent ury t he relat ion bet w een music and t echnology
became more int ense due t o a series of reasons, among t hem, t he increasing know ledge about sound
physics and sound cognit ion; t he access t o low cost elect ricit y; and t he use of elect ronic and digit al
t echnology t o art ifi cially generat e and manipulat e sounds. Bef ore t hat , musical sounds w ere produced only
by mechanical means. Alt hough musical inst rument s, such as t he violin, t he piano or even t he human voice,
represent ed a w ide variet y of f orms and mechanisms, all of t hem w ere based on t he same principle of sound
product ion, t hat is, t he mechanic vibrat ion of an elast ic body. How ever, t he appearance of elect rical
t echnologies and t he use of elect romagnet ic signals brought t he possibilit y of generat ing sounds w it hout
using mechanical devices.
Alt hough t he sound w aves coming eit her f rom a clarinet or f rom t he elect ronic oscillat ors inside a
synt hesizer have t he same nat ure, t heir processes of product ion are quit e dif f erent . On one side t here is t he
concret e, visible and mechanic universe of t he t radit ional inst rument s w here t he body of t he inst rument and
 11

t he body and movement s of w ho is playing t hat inst rument are int rinsically relat ed t o t he qualit ies of t he
sound t hey are producing. On anot her side, in t he era of elect ricit y and elect ronics, w e st art list ening more
and more t o t he sounds of invisible bodies cont ained in t he elect ronic component s of synt hesizers, samplers,
and comput ers.
For many cent uries, people learned t o list en t o sounds t hat had a st rict relat ion t o t he bodies t hat produced
t hem. Suddenly, all t his list ening experience accumulat ed during t he long process of musical evolut ion w as
t ransf ormed by t he appearance of elect ronic and recorded sounds. When one list ens t o art ifi cially generat ed
sounds he or she cannot be aw are of t he same t ype of concret e and mechanic relat ions provided by
t radit ional acoust ic inst rument s since t hese art ifi cial sounds are generat ed by processes t hat are invisible t o
our percept ion. These new sounds are ext remely rich, but at t he same t ime t hey are ambiguous f or t hey do
not maint ain any defi nit e connect ion w it h bodies or gest ures.
Unt il t he development of sound recording syst ems early in t his cent ury, t he cont act w it h music occurred only
t rough t he perf ormance. The list ener, even if not involved in t he sound product ion, part icipat ed in t he music
realizat ion by ment ally reconst ruct ing t he connect ions bet w een t he sounds and t he physical and cult ural
cont ext w here t he music w as t aking place. When recording t echnologies became socially ef f ect ive, t hey
brought about t w o dif f erent alt erat ions.

Listening

The fi rst alt erat ion ref ers t o a shif t f rom t he prominence of music product ion processes (composit ion and
int erpret at ion) t o t he prominence of list ening act ivit ies as cult ural experience (M ow it t 1987). Here one can
easily not e t hat now t he number of people w ho are list ening t o music is much larger t han t he number of
people w ho are doing music. This f act show s more t han a st at ist ical aspect . It also refl ect s a cult ural
t ransf ormat ion: music is primarily produced t o be list ened t o and not t o be played or sung and t he process
of composit ion and perf ormance becomes t he means t o propit iat e it .
This project ion t ow ards t he list ener is reinf orced by t he processes of reproduct ion t hat imposed new models
of recept ion. When t he phonograph w as invent ed about one hundred years ago, music recept ion w as based
on t he list ening model of t hat t ime, t hat is, t he list ening of a live perf ormance. The main goal of t hose
recording syst ems w as guided by t he t erm fi delit y, w hich in t hat case w ould mean t hat t he bet t er a
recording could reproduce t he sound qualit ies of a live perf ormance t he bet t er it w ould be considered
(Thompson 1995). But w hat w e experience t oday is a dif f erent sit uat ion. The live perf ormance cannot be a
list ening model anymore since, f or most of us, list ening t o music means list ening t o t he music reproduced by
a device such as radio, or CD. Gradually, t his new cont ext based on recording and reproduct ion becomes t he
 12

model f or music recept ion and music product ion. The t erm fi delit y, w hich presupposes some relat ion of
similarit y, is st ill in use by phonographic and audio indust ries, but it s meaning became refl exive and
inappropriat e: t he fi delit y of a recorded sound is not based upon t he original sound it self but it is
est ablished as a f unct ion of t he available recording t echnology. This leads t o a paradoxical sit uat ion in w hich
more and more musicians t ry t o reproduce in t heir live perf ormances t he same sound qualit ies of t heir
records, specially in pop music. In a st imulat ing essay, John M ow it t observes t hat :
If recording organizes t he experience of recept ion by condit ioning it s present scale and est ablishing
it s qualit at ive norms f or musicians and list eners alike, t hen t he condit ions of recept ion act ually pre-
cede t he moment of product ion. [T]he social analysis of musical experience has t o t ake account of t he
radical priorit y of recept ion (M ow it t 1987: 176-77).

Gesture

The second alt erat ion is relat ed t o musical gest ure. Alt hough music has alw ays been st rict ly relat ed t o
gest ure, only in t he past f ew decades t his issue has deserved some at t ent ion by musicologist s. This concern
seems t o emerge w hen it became possible t o record and t o reproduce music and t he role of perf ormance
w as replaced by a list ening sit uat ion mediat ed by new t echnologies such as t he radio, magnet ic t apes, or
CDs. As Roland Bart hes says, t here is one music one plays and one music one list ens t o. The music one plays is
a muscular music, in w hich t he part t aken by t he sense of hearing is one only of rat ifi cat ion as t hough
t he body w ere hearing [...] a music w hich is not played 'by heart ': seat ed at t he keyboard or t he music
st and, t he body cont rols, conduct s, co-ordinat es, having it self t o t ranscribe w hat it reads, making
sound and meaning, t he body as an inscriber and not just t ransmit t er, simple receiver (Bart hes, 1977:
149).
This musica pract ica has been replaced by a passive and recept ive music. Elect ronic t echnology brings t his
sit uat ion a st ep f urt her by eliminat ing t he t radit ional role of t he perf ormer f rom t he chain of music
product ion. How ever, it seems t hat at t he same t ime t hat elect ronic t echnology st resses t his shif t f rom t he
idea of a music t o be done t o t he idea of a music t o be list ened t o (Chanan, 1994; M ow it t , 1987), it also
point s t o t he confl ict bet w een t he physicalit y t radit ionally involved in music product ion and t he absence of
concret e object s in t he virt ual environment of t oday's digit al st udios: " It is precisely t his physical absence
t hat is nat urally compensat ed f or in music by reint roducing a concern f or body, gest ure and space" (Chabot ,
1990: 15).
Indeed, t he int erest in gest ure's relat ed subject s has been developed by dif f erent fi elds relat ed t o human
communicat ion and cognit ion due t o dif f erent reasons: t he concern w it h t he body in cognit ive sciences as an
agent of know ledge (Johnson, 1987; M insky, 1986; Varela, Thompson, & Rosch, 1991); t he grow ing int erest
 13

in non-verbal f orms of communicat ion in semiot ic and psychology st udies (Kendon, 1981; Nöt h, 1990;
Poyat os, 1983); t he development of st udies in sign languages of t he deaf , especially American Sign
Language (Kendon, 1981: 31); t he st udies in Human-Comput er Int eract ion promot ed by t he large expansion
of digit al t echnology t hat brought t he possibilit y of simulat ing physical phenomena in " virt ual"
environment s (Barfi eld & Furners III, 1995; Carr & England, 1995; Laurel, 1990).
Gest ure is t aken here in a broad sense. It does not mean only movement , but a movement t hat can express
somet hing. Theref ore, it is a movement t hat embodies a special meaning. It is more t han a change in space,
or a body act ion, or a mechanic act ivit y: gest ure is an expressive movement t hat becomes act ual t hrough
t emporal and spat ial changes. Act ions such as t urning knobs or pushing levers, are current in t he use of
t oday's t echnology, but t hey cannot be considered gest ures. Also, t o t ype a f ew w ords in a comput er's
keyboard has not hing t o do w it h gest ure since t he movement of pressing each key does not convey any
special meaning. It does not mat t er w ho or w hat perf ormed t hat act ion, neit her in w hich w ay it w as
perf ormed: t he result is alw ays t he same. How ever, t he sit uat ion is complet ely dif f erent w hen a musician
plays somet hing on a piano keyboard: t he result , t hat is, t he musical perf ormance, depends in many
dif f erent w ays on t he player's gest ure. Pressing a key or sliding a bow during a perf ormance are movement s
t hat hold a meaning in t hemselves: t hey est ablish how a sound w ill be produced, t hey det ermine some
charact erist ics of t hat sound, t hey set up connect ions w it h previous sonic event s, and, at t he same t ime, t hey
f urnish an art iculat ory pat h t o f urt her sounds. Obviously t hey are more t han simple movement s, t hey are
meaningf ul gest ures.
Gest ure in music perf orms a f undament al role in t he generat ion of meaning (Henrot t e 1992; Lidov 1987). In
a cert ain w ay, w e have learned t o underst and musical sounds w it h t he aid of t he gest ures t hat produce and
represent t hese sounds. As G. Kurt enbach and E. Hult een have not ed, t he f unct ion of gest ure in music is
proport ional t o it s pow er t o express somet hing:
Gest ures increase f unct ion by virt ue of t heir expressiveness. That is, a gest ure may cont rol mult iple
paramet ers at t he same t ime, t hus allow ing a user t o manipulat e dat a in a manner not possible by
modif ying each paramet er individually. For example, a conduct or simult aneously cont rols bot h
t empo and volume of t he music gest ure. The rhyt hm of t he gest ure cont rols t empo and t he size of
t he gest ure cont rols volume. This allow s an effi cient communicat ion not possible by adjust ing t he
t empo and volume independent ly (Kurt enbach & Hult een 1990: 311-12).
Gest ure acquires it s signifi cat ion t hrough our experience w it h t he signs t hat surround us. This signifi cat ion is
const ruct ed via t he int eract ion bet w een t he brut e phenomena w e experience during our lives and t he w ay
our senses apprehend t hese phenomena. Thus, t he use of gest ure in communicat ing t hings can be a pre-
linguist ic act . Bef ore a child can speak or even cont rol his movement s he is able t o est ablish relat ions
 14

bet w een t he event s t hat occur around him and t he consequences brought by t hose event s. From t his, t he
child is able t o st art building up it s ow n gest ural vocabulary.
There is a st rong t radit ion in t he lit erat ure making an approximat ion bet w een gest ure and verbal language.
Linguist ics is of t en t aken as a basis t o analyze any kind of communicat ion and linguist ic t heories based on
verbal codes have infl uenced all disciplines involved w it h communicat ion and language, f rom inf ormat ion
t heory t o semiot ics. Inspiring w orks by dif f erent aut hors have st ressed t he connect ion bet w een gest ure and
verbal language and speech (Ef ron, 1972; Ekman & Friesen, 1969; M cNeill, 1992) even w hen t hey t ry t o
defi ne specifi c t rait s of gest ure (Kendon, 1981; Poyat os, 1983). The classifi cat ion developed by Ekman &
Friesen (1969) st art ed f rom Ef ron's t heory of gest ure and point s t o fi ve cat egories of gest ure: Emblems,
Illust rat ors, Af f ect Displays, Regulat ors, and Body M anipulat ors. In t his classifi cat ion, only Af f ect Displays
and Body M anipulat ion are not direct relat ed t o speech. M cNeill (1992) creat es his ow n classifi cat ion based
on t he ideas of dif f erent aut hors. He classifi es gest ure as f ollow : iconics, met aphorics, deict ics and beat s. In
t he same direct ion Kendon (1988) point s t o a cont inuum in order t o build a classifi cat ion of gest ure
occurrence -- Gest iculat ion, Language-Like gest ures, Pant omimes, Emblems, Sign-Languages -- in w hich one
can not e a decrease of t he presence of verbal component s going f rom gest iculat ion t o sign-languages
(M ulder, 1996).
Besides depart ing f rom a " linguist ic" point of view, most of t hese cat egories are specially relat ed t o empt y-
hand movement s t hat occur during ut t erance. In a narrow sense, gest ure is t aken as " bodily communicat ion
by means of hands and arms and t o a lesser degree by t he head" (Nöt h, 1990). This narrow sense excludes
t he possibilit y of considering ot her t ypes of body act ion, such as post ure, body movement , gaze and f acial
expression as genuine t ypes of gest ure.
Alt hough t his approach can be ef f ect ive f or most HCI applicat ions, w e believe t hat t o st udy t he role and
f unct ionalit y of gest ure in int eract ive music syst ems, one should consider it in a broader sense in order t o be
able t o t ake some music peculiarit ies int o account . For t his reason w e t ake gest ure as any t ype of " bodily
movement [...] made consciously or unconsciously t o communicat e eit her w it h one's self or w it h anot her"
(Hayes, 1966, apud Nöt h, 1990). This w ould include not only emblemat ic hand gest ures, but also any ot her
bodily movement capable of conveying meaning. This w ould also include act ions of t ouching, grasping or
manipulat ing physical object s in order t o cont rol music or sound paramet ers.
Besides t his concept ion based on t he physicalit y of gest ure, w e may also consider a parallel meaning of t he
t erm w hich has a met aphorical sense and has been w idely used in music analysis. M any aut hors have
ref erred t o music gest ure w hen describing some dynamic charact erist ics of musical ut t erance. Coker (1972)
has made one of t he fi rst ext ensive at t empt s t o relat e music meaning t o gest ure in a semiot ic cont ext . For
t he aut hor musical gest ure " comprises a recognizable f ormal unit and consist s of a select ion and
 15

organizat ion of sonic and rhyt hmic propert ies in sonorous mot ion, w hich signifi es ot her purely musical
object s or non-musical object s, event s, and act ions" (Coker, 1972:18). Anot her classical approach w as
developed by M anf red Clynes (1977) w ho used an elect ric device called sent ograph t o monit or and record
fi nger pressure. In his experiment s, subject s w ere required t o press t he sent ograph in an expressive manner
in response t o dif f erent inst ruct ions. The result s of t hose experiment s indicat ed a relat ion bet w een some
gest ural pat t erns and specifi c emot ional st at es, such as love, hat e, anger and joy. The relevance of a
" met aphorical" use of gest ure in music has also been f ormulat ed t o elucidat e ot her aspect s of musical
meaning such as in Iw anka St oïanow a's st udy on music-graph-gest ure (St oïanow a, 1973) and in David Lidov's
t ext on kinest het ic relat ions in music discourse (Lidov, 1987).
Alt hough t hese approaches do not direct ly assume t he t erm gest ure in a sense of physical bodily movement ,
t here is a st rong connect ion bet w een t his met aphorical sense and t he more physical use of t he t erm in
music. François Delalande (1988) indicat es t hree gest ural levels, ranging f rom t he f unct ional t o t he symbolic:
t he gest e ef f ect eur, " necessary t o mechanically produce sound" ; t he gest e accompagnat eur, w hich " engages
t he w hole body" of t he player, but may not be direct ly involved in t he sound product ion; and t he gest e
fi guré, " an imaginary ballet w hich const it ut es a t hird dimension of t he movement , and w hich could be
relat ed t o t he met aphorical use of musical gest ure w e ref erred above. These t hree levels are cert ainly
int errelat ed and t hey can even overlap in t heir f unct ion. While analyt ical and aest het ic approaches are more
inclined t o t he invest igat ion of t he gest e fi guré, fi elds relat ed t o t he development of int eract ive music
inst rument s and environment s rest upon t he st udy of t he gest e ef f ect eur. The gest e accompagnat eur,
besides playing an int ermediary role bet w een t he f ormer t w o levels, has a more defi ned place in
perf ormance st udies.
The idea of a gest e fi guré has a parallel in w hat Bernadet e Zagonel defi nes as ment al gest ure (Zagonel
1992). While physical gest ure (in t he sense of gest e ef f ect eur and accompagnat eur ) ref ers t o t he product ion
of sound as a physical phenomenon, ret aining an object ive relat ion bet w een t he gest ure and t he sounds
t hat are produced, ment al gest ures occurs as an idea or an image t hat ref ers t o physical gest ure. M ent al
gest ures are closely relat ed t o t he processes of composit ion and list ening. Thus, t he composer of t en st art s
f rom an idea or a ment al image of sound gest ure t o compose a vocal or inst rument al gest ure. The ment al
gest ure is learned t hrough t he experience and st ored int o t he memory t o be used as a model in composit ion.
At t he same t ime t hat a ment al gest ure ref ers t o t he corporal act ions of a perf ormer or t o t he behavior of a
musical inst rument , it can also ref er t o a part icular sound st ruct ure. For example, an arpeggio can be
underst ood as a gest ural movement f rom one point t o anot her in t he pit ch space and t he movement of a
conduct or's hand can be seen as a gest ure t hat unifi es t he t emporal and art iculat ory behavior of t he
 16

orchest ra. What w e should emphasize is t hat t he ment al gest ure alw ays ref ers t o a physical gest ure t hat has
been previously learned.
Concerning music, one can say t hat physical gest ure is direct ly relat ed t o music int erpret at ion w hile
composit ion is much closer t o ment al gest ure: As Bernadet e Zagonel has said, " if t he composer goes f rom
gest ure t o t he composit ion, t he perf ormer goes t he opposit e w ay, t hat means, he goes f rom t he score t o t he
gest ure" (Zagonel 1992: 17-18). To t his st at ement w e can add t hat t he list ener complet es t his chain by
ment ally recreat ing t he perf ormer's physical gest ures w hile list ening t o music.

Body

The idea of gest ure leads immediat ely t o t he body. The body is t he inst rument t hrough w hich t he gest ure
becomes act ual; it is t he int erf ace t hat t ranslat es ideas int o physical signs and t his can occur in t hree
dif f erent w ays, depending on how t his gest ure acquires it s signifi cat ion: by similarit y, by causalit y, and by
convent ion . The det erminat ion of t hese t hree t ypes of gest ural cat egories w as inspired by t he t riadic
cat egories draw n by t he American semiot ician Charles Sanders Peirce (1955) and can be t aken as an analogy
t o his sign concept s of icon, index and symbol. Inst ead of been root ed in linguist ics, Peirce's t heory of f ers a
more general scope t o invest igat e t he w ay any t ype of signs can represent -- or mean -- somet hing since he
uses logic as t he point of depart ure t o creat e his semiot ic t heory.
Here, w hat is import ant t o not e is t hat , according t o Peirce, his t riadic cat egories involve a hierarchy
according t o w hich t he t hird element , t he symbol, comprises t he second, t he index, w hich by it s t urn
comprises t he fi rst element , t he icon. Theref ore, a sign rarely w ould be represent ed by only one of t hese
cat egories. Follow ing Peirce's scheme, if w e st at e t hat a specifi c gest ure belongs t o a cert ain t ype, it does not
invalidat e it s ef fi cacy in t he domain of anot her t ype. To say t hat t he signifi cat ion of a gest ure operat es by
means of similarit y, causalit y or convent ion means t hat t he gest ure is more charact erist ic of one of t hese
t hree t ypes but st ill can be relat ed t o t he ot her t w o t ypes, specially w hen t he same gest ure is present in a
dif f erent cont ext .
One example of gest ure t hat operat es by similarit y is t he corporal gest ure t rying t o imit at e or emulat e t he
behavior of processes and object s in t he w orld. For example, t he f ast movement of a conduct or's hand t o
indicat e a f ast t empo and a sax player bending up and dow n during t he perf ormance as if his/her body w ere
f ollow ing t he melodic shape of a solo. These gest ures operat e by creat ing some t emporal and/or spat ial
relat ion of similarit y w it h ot her event s: t he speed of t he movement in t he fi rst case; t he melodic cont our in
t he second.
 17

On anot her hand, causal gest ures are connect ed t o an event t hrough a cause-and-ef f ect relat ion. Usually
t hese are gest ures t hat operat e as a response t o an act ual circumst ance. Bringing t he hands in f ront of t he
f ace w hen w e predict a dangerous sit uat ion is a direct react ion t o t his sit uat ion. In t he same sense one can
of t en relat e someone's at t it ude t o a personal charact erist ic such as in t he case of a shy person w ho has a
shrunk post ure during w alk. In t his case t he gest ure (post ure) has a necessary relat ion w it h it s cause
(shyness) and t he exist ence of t he f ormer depends on t he lat t er. We may not e t hat , alt hough some aut hors
t end t o t ake t he st at ic charact er of post ure in opposit ion t o t he dynamics of gest ure (M ulder, 1996), w e
consider post ure as an ef f ect ive and expressive bodily act ion and t heref ore as a t ype of gest ure w hich is
ext remely import ant t o music perf ormance since it est ablishes a close relat ion bet w een t he behavior of t he
perf ormer's body and his/her musical expressiveness.
As w e enlarge our gest ural vocabulary w e also t end t o apply arbit rary signifi cat ion t o specifi c gest ures.
These convent ional gest ures do not necessarily keep any relat ion of similarit y nor causalit y but t hey are
const ruct ed, shaped by ext ernal f act ors such as cult ure and language. They acquire signifi cat ion by an
abst ract and f unct ional process and must be learned t o be shared by a specifi c group. For example, t he
charact erist ic gest ure made by Nort h-Americans w it h t he fi ngers t o mean t hat somet hing is okay or posit ive,
in Brazil is t aken as an of f ense. The relat ion bet w een t his t ype of gest ure and it s signifi cat ion is t ot ally
convent ional and can easily change f rom one sit uat ion t o anot her.
These gest ural t ypes are generic and can ref er t o more specifi c levels of gest ural classifi cat ion. For example,
one could associat e t he cat egories of iconic and met aphoric gest ures draw n by M cNeill (1992) as examples of
gest ure t hat operat e by similarit y, w hile t he deict ic gest ures w ould be clearly relat ed t o causal gest ures. In
t he same w ay, emblemat ic gest ure w ould f all in t he convent ional gest ure t ype. Anot her relat ion can be
est ablished bet w een Delalande's considerat ions about gest e ef f ect eur, gest e accompagnat eur and gest e
fi guré (Delalande, 1988) and t he causal, similar, and convent ional gest ure t ypes, alt hough generally t hey
could be relat ed t o more t han one t ype at once.

Performance

In a book called " The Sight of Sound: M usic, Represent at ion, and t he Hist ory of t he Body" , Richard Leppert
calls t he at t ent ion t o t he import ance of t he gest ural images t hat are produced during a musical
perf ormance:
Precisely because musical sound is abst ract , int angible, and et hereal [...] t he visual experience of it s
product ion is crucial t o bot h musicians and audience alike f or locat ing and communicat ing t he place
of music and musical sound w it hin societ y and cult ure. [...] M usic, despit e it s phenomenological
sonoric et herealit y, is an embodied pract ice, like dance and t heat er (Leppert 1993: xx-xxi).
 18

The lack of musicological st udies t aking body gest ure int o account is in part due t o t he f act t hat t he body
has never been considered as a support f or musical expression. When elect ric and elect ronic means of
musical product ion and dif f usion eliminat ed t he presence of t he musician and his inst rument w e had t o
change t he w ay w e experience music. " Tradit ionally, t o at t end a music perf ormance is t o apprehend t hrough
t he sight t he int ent ion w hich is loaded in t he inst rument alist 's gest ure. In t he mediat ion of t he
t echnological w ork, t his predict ion does not w ork all t he t ime" (Bat t ier, 1992: 69). The symbiot ic relat ion
bet w een t he player's body and his inst rument plays a special role in t he comprehension of t he musical
discourse. For example, a violent gest ure produced by t he player reinf orces t he ef f ect of a sudden sound
at t ack in t he same w ay t hat t he body expression of a singer can lead t o a richer phrase art iculat ion.
The musical background and t he daily experience w it h t he object s t hat surround us help t o est ablish
connect ions bet w een physical mat erials and t he t ypes of gest ures t hat w ill produce specifi c sounds w it h
t hose mat erials. Through our experience w e can also est ablish pat t erns and links bet w een sonic event s and
t he mat erials t hat could produce t hese sounds. Of course, t he percept ion of t hese relat ions is not precise in
specifi c t erms (f or example, it is dif fi cult t o det ermine how loud is t he sound of t his book f alling on t he
fl oor), but t his percept ion is ext remely sophist icat ed in relat ive t erms (f or example, one can easily say t hat
rolling a pencil over a w ood t able produces a sof t er sound t han hit t ing t he t able w it h t hat pencil).
M oreover, part of music's expressive pow er is given by t his proport ional scale t hrough w hich one can
ment ally build relat ions among mat erials, gest ures and sounds. It is at t his point t hat lies part of t he
at t ract ion music exert s on us. Indeed, among t he object s t hat surround us w e don't know of any w hich
w ould allow t he product ion of such amazing sounds as w e can get f rom a musical inst rument . Our
percept ion t ells us t hat an ordinary piece of w ood, about 50 cent imet ers long, w ould never produce a sound
w it h t he same qualit y, pow er, bright ness and regularit y one could hear f rom on a violin, f or example. Also,
ordinary object s usually don't allow a precise cont rol of t he sound produced by a specifi c gest ure. Thus,
musical inst rument s subvert our expect at ions about t he sounds t hat such devices w ould produce. What a
musical inst rument allow s t o do in t erms of sound product ion goes f ar beyond w hat our common sense
w ould expect f rom any ot her ordinary object . It est ablishes an almost magical relat ion bet w een gest ure and
sound and, in t his w ay, t he musical inst rument is not only t he medium f or t he musical idea, but also part of
t his idea.
Our experience provides t he fi rst st ep t o creat e percept ual models t hat correlat e classes of gest ure,
inst rument al propert ies and sounds. The const ruct ion of a gest ural vocabulary st art s f rom t hese int uit ive
models and grow s up as w e enlarge our musical experience. What is import ant t o ret ain is t hat inst rument al
gest ures, t hat is, t he physical gest ures t he perf ormer applies on an inst rument t o produce or t ransf orm
sounds, are ext ensively det ermined by t he inst rument propert ies. Tradit ional inst rument s are subject t o
mechanic law s and w hen someone plays a piano or a t rumpet his/her gest ures are const rained by t hese law s.
 19

Cert ain t ypes of musical phrases can be more charact erist ic of a part icular inst rument just because it s design
makes it easy t o produce t hat chain of not es. All t hese charact erist ics are at t ached t o t he inst rument and are
due t o a long process of evolut ion and adapt at ion. Thus one can int uit ively draw st able relat ions bet w een
classes of inst rument s and classes of gest ures. For example, one could say t hat bow ed inst rument s w ould
f avor some kind of smoot h and cont inuous gest ure w hile percussion inst rument s w ould lead t o more
discret e and defi nit e movement s.
Obviously, each inst rument of f ers a dif f erent level of cont rol and int eract ion. For example, t he mechanism
of a pipe organ w orks almost aut omat ically in a w ay t hat t he perf ormer's gest ures have very lit t le infl uence
over t he process of sound generat ion. On anot her hand, some acoust ic inst rument f amilies, such as bow ed
st rings, reed or blow n inst rument s, and percussion, generally w ill allow t he cont rol of very subt le sound
charact erist ics t hrough t he int eract ion bet w een t he player's gest ures and t he st ruct ure of t he inst rument . As
Richard M oore point s out ,
Wit h such inst rument s t he microgest ural movement s of t he perf ormer’s body are t ranslat ed int o
sound in w ays t hat allow t he perf ormer t o evoke a w ide range of aff ect ive qualit y in t he musical
sound. That is simult aneously w hat makes such devices good musical inst rument s, w hat make t hem
ext remely diffi cult t o play w ell, and w hat makes overcoming t hat diffi cult y w ell w ort hw hile t o bot h
perf ormer and list ener (M oore 1987: 258).

M apping

Due t o t heir mechanic nat ure, t he operat ion of t radit ional inst rument s during perf ormance est ablishes a
close connect ion w it h t he mechanic of t he player's body. This connect ion w orks as a clue f or t he f ormulat ion
of inst rument al models of music percept ion. M ost of t ime, sounds derived f rom t he t radit ional orchest ra are
very clear in relat ion t o w hich t ype of inst rument t hey come f rom and t o t he qualit y of gest ure t hat have
produced t hem. But w hen w e consider sounds produced in elect roacoust ic music t hese relat ions are not so
explicit .
First ly because elect ronic inst rument s can be seen as black-boxes t hat produce dif f erent sounds as response
t o ext ernal input s. The process of producing sounds is invisible t o musicians and list eners alike and t here is
no necessary relat ion bet w een t he physical input and t he result ing sound. For every new inst rument it is
necessary t o learn how a specifi c gest ure act s on it . The mechanic law s t hat regulat e t he component s of
elect ronic inst rument s do not det ermine how t he musician's act ions are mapped int o sounds. It is a mat t er of
design: t his mapping is designed in t he same manner t hat elect ronic circuit s or cont rolling int erf aces are.
Secondly, because a new elect ronic inst rument may not be relat ed t o any previously know n inst rument al
t ype. M usic hist ory is f ull of inst rument s t hat did not survive because t hey could not be adapt ed t o a very
 20

w ell est ablished t radit ion. Exot ic or over-complicat ed inst rument s did not survive because t hey w ere not
general enough t o sat isf y our musical demands. This has changed because t oday's elect ronic inst rument s can
be designed w it hout any compromise on an evolut ionary process in w hich each st ep represent s an
improvement on t he previous one (Lansky 1990). The lif e of an elect ronic inst rument can be as short as t he
durat ion of a composit ion.
If t radit ional inst rument s are const rained t o t he mechanic behavior of t heir mechanisms and int erf aces,
elect ronic inst rument s of f er no such const raint s: t hey are art ifi cially creat ed by an inst rument builder. The
relat ions bet w een t he sounds and t he gest ures t hat w ill produce t hem are also part of t he inst rument 's
project . Here w e approach some of t he most prominent quest ions in t he design of t oday's elect ronic
inst rument s: How can w e design t he art ifi cial relat ions t hat w ill cont rol t he sound product ion in an ef f ect ive
and meaningf ul w ay? How can w e map specifi c gest ures t o cont rol specifi c sound paramet ers in a nat ural
and meaningf ul w ay? If w e agree t hat gest ure is a meaningf ul part of music, paying at t ent ion t o t hese
quest ions regarding t he design of elect ronic inst rument s is ext remely import ant t o propit iat e syst ems t hat
w ill allow perf ormers t o convey t heir expressiveness w it hout reducing t heir role t o simply act ivat e and
deact ivat e sound event s.
Alt hough t he sound generat ion in elect ronic music inst rument s is based on t he fl ow of elect rical signals,
usually t heir cont rol st ill relay on some t ype of physical act ivit y carried out by t he perf ormer's body. While in
t radit ional acoust ic inst rument s t he ef f ect s of t he perf ormer's physical act ivit y on an inst rument are already
est ablished by t he physical propert ies of t he inst rument , in elect ronic inst rument s t his relat ion must be
previously designed. M apping t his relat ion can be crit ical f or t he ef f ect iveness of an elect ronic inst rument .
Three dif f erent mapping st rat egies have been proposed in (Rovan, Wanderley, Dubnov, & Depalle, 1997) t o
provide a connect ion bet w een gest ure and t he cont rol of sound paramet ers: One-t o-One, Divergent and
Convergent . The simplest mapping scheme, one-t o-one, maps each independent gest ure t o one musical
paramet er. In t he divergent st rat egy each gest ure is mapped t o more t han one musical paramet er, w hile in
t he convergent mapping many gest ures can be mapped t o one musical paramet er. As t he aut hors have
observed, expressivit y increases f rom t he one-t o-one t o t he convergent st rat egy and w e may also not e t hat
t he dif fi cult ies t o implement t hese st rat egies increase in t he same proport ion. Thus, in t he design of a new
inst rument or musical environment t here may be a compromise bet w een t he expressivit y allow ed by t he
syst em and t he obst acles imposed by it s implement at ion. We w ould add t hat t radit ional acoust ic
inst rument s usually operat e according t o a f ort h layer -- w hich is a combinat ion of t he f ormer t hree -- in
w hich many dif f erent gest ures cont rol dif f erent paramet ers.
Of course, t here is a count erpart t o t his scheme. When expressivit y diminishes, as in t he case of t he one-t o-
one st rat egy, it becomes easier t o get a fi ner cont rol of specifi c paramet ers, and t his is one of t he keys f or
 21

t he success of most commercial elect ronic music syst ems, f orm t he M IDI prot ocol t o t he most recent
commercial keyboard synt hesizers.

Conclusion

Elect roacoust ic music f orces t he appearance of new list ening models, w hich are more ext ensive and
unst eady t han t he ones t hat support t radit ional music. As Dennis Smalley point s out , unt il recent ly t he
list ener expect at ions during a concert w ere limit ed by f amiliar models w hich had been det ermined by music
t radit ion and repert oire:
Prior t o t he elect roacoust ic era [...] t he list ener could aut omat ically assume bef ore even list ening t o a
piece of music t hat it w ould be root ed eit her in t he inst rument al gest ure model, or human ut t erance,
or bot h [...] So, t radit ionally, t he indicat ive f ramew ork and boundaries of a musical w ork w ere not
only pre-det ermined but , as f ar as cult ure is concerned, permanent (Smalley 1992: 544).
On anot her hand, in pure elect roacoust ic music t here is no such limit at ion t o inst rument al or vocal models.
The composer is st imulat ed t o creat e new sonic fi elds and t he list ener t o develop new list ening st rat egies.
The absence of perf ormers, inst rument s, and visual and gest ural ref erences provides a radically expanded
experience: " everyt hing remains t o be revealed by t he composer and discovered by t he list ener" (Smalley
1992: 545). Since t he fi rst w orks of elect roacoust ic music w orks deviat ed t he f ocus f rom inst rument al gest ure
t o acousmat ic list ening, music has of t en lost t he expressive and dramat ic pow er conveyed by t he gest ural
realizat ion of t he perf ormer. Since t hen, t o reconst it ut e t hese charact erist ics became a challenge in pure
elect roacoust ic music composit ion.
During t he 80's, w hen t echnology made it possible t o reint roduce t he part icipat ion of t he perf ormer in t he
elect roacoust ic pract ice t he debat e around t he role of gest ure in music emerged under a dif f erent
perspect ive. The perf ormer's act ion in elect ronic int eract ive music cont ribut e as a visual and physical
ref erence during t he perf ormance (Iazzet t a 1996). How ever, many real-t ime syst ems f or sound cont rolling
and processing usually do not provide any explicit relat ion bet w een t he gest ures perf ormed by t he musician
and t he result ing sounds. Rat her, comput ers and ot her digit al devices disorganize t he causal relat ions
bet w een gest ure and sound. A digit al music syst em can calculat e very precisely t he t raject ory of a sound
paramet er and use it t o recreat e t hat sound, but " it lacks t he ant icipat ion of t he ef f ect , because t he result s
can hardly recover t he gest ure" (Bat t ier, 1992:69).
Emmerson (1994) suggest s t hat in live perf ormances of elect ronic music t his could be at t enuat ed w it h a
balance bet w een t w o syst ems: on one side, w hat t he inst rument alist can cont rol and can be perceived as a
relat ion of a human perf ormer act ion and sound product ion (w hich he ref ers t o as local cont rol); on t he
ot her side, t he cont rol of an environment w here t he result s are relat ed t o t he cont ext in w hich t he
 22

perf ormance t akes place (w hich he calls fi eld cont rol). Emmerson ref ers t o local/fi eld cont rols as t w o
complement ary syst ems t hat might be t aken in t he research agenda of musicians w orking w it h elect ronic
syst ems. The balance bet w een local and fi eld cont rols could help in " reassembling some of t he 'cause/ef f ect '
chains w hich have been broken by recording and comput er t echnology" (Emmerson, 1994: 31).
Indeed, one of t he most pow erf ul f eat ures of an int eract ive comput er music syst em is t he abilit y t o
const ant ly reorganize it s confi gurat ion. While it opens a large spect rum of musical possibilit ies f or sound
product ion, it also avoids t he const it ut ion of st able connect ions bet w een perf ormance act ions and result s.
The absence of a unique gest ural mapping prevent s t he perf ormer f rom deeply exploring t he syst em's
cont rolling mechanisms at t he same t ime t hat it prevent s t he list ener f rom connect ing visual input and
music. While t he acousmat ic ef f ect in pure elect roacoust ic music t ends t o put t he sound in evidence by
breaking (or at least by w eakening) it s connect ion w it h it s source, int eract ive music renders t his connect ion
very unst able. As composer Dennis Smalley says:
[...] w e can arrive at a sit uat ion w here t he sounding spect ro-morphologies do not correspond w it h
perceived physical gest ure: t he list ener is not adequat ely armed w it h a know ledge of t he pract icali-
t ies of new 'inst rument al' capabilit ies and limit at ions, and art iculat ory subt let y is not recognized and
may even be reduced compared w it h t he t radit ional inst rument (Smalley 1992: 548).
Since t he 90's, inst rument designers and musicians have t o f ace t he part icularit ies of new elect ronic devices
f or music product ion in order t o creat e ef f ect ive int erf aces t hat w ould allow t he expressive cont rol of sound.
M apping gest ure int o cont rolling paramet ers is not a simple t ask. The manipulat ion of t ape recorders,
synt hesizers, and comput ers eliminat es t he mat erialit y of sound. Dif f erent ly f rom t he sounds produced by
mechanic inst rument s, elect ronic sounds do not embody any kind of gest ural relat ion t o t he devices t hat
produce t hem. It represent s a great aut onomy and f reedom in t erms of composit ional st rat egies, but at t he
same t ime it means a loss in t he symbolic and meaningf ul dimensions t hat can be present in a musical w ork.
In t he last 50 years elect ronic and digit al t echnology expanded t he possibilit ies f or creat ing music in a very
signifi cant w ay, but t his expansion also brought t he risk of rest rict ing t he semiot ic connect ions conceived
during t he perf ormance and list ening. Today's musical act ivit ies based on t he product ion of pre-recorded or
elect ronic generat ed music have deviat ed f rom t he embodied pract ice of perf ormance t o art ifi cial processes
of composit ion and dif f usion. In t hese processes t he loudspeaker replaces t he perf ormer, t hus eliminat ing
visual and gest ural ref erences t hat t radit ionally composed a signifi cant dimension of musical language. In
w hich w ay cont emporary musical pract ices such as elect ronic int eract ive music w ill rest ore t his dimension is
somet hing t hat st ill has t o be elaborat ed and can be seen as a challenge t hat w ill ent ert ain, f or some t ime,
composers, perf ormers and list eners alike.
 23

References

Barfi eld, W., & Furners III, T. A. (Eds.). (1995). Virt ual Environment and Advanced Int erf ace Design. Oxf ord:
Oxf ord Universit y Press.

Bart hes, R. (1977). " M usica Prat ica." In Image - M usic - Text (pp. 149-154). New York: Hill & Wang.

Bat t ier, M . (1992). " Sculpt er la Transparence." Les Cahiers de L'Ircam - Recherche et M usique, 1, 57-75.

Boet hius, A. M . S. (1989) [ c. 500]. Fundament als of M usic (Calvin M . Bow er, Trans.). New Haven & London: Yale
Universit y Press .

Carr, K., & England, R. (Eds.). (1995). Simulat ed and Virt ual Realit ies: element s of percept ion. London: Taylor &
Francis.

Chabot , X. (1990). " Gest ure Int erf aces and a Sof t w are Toolkit f or Perf ormance w it h Elect ronics." Comput er
M usic Journal , 14(2), 15-27.

Chanan, M . (1994). M usica Pract ica: The Social Pract ice of West ern M usic f rom Gregorien Chant t o Post modern-
ism . London / New York: Verso.

Clynes, M . (1977). Sent ics, t he Touch of Emot ions. New York: Anchor.

Coker, W. (1972). M usic & M eaning: A Theoret ical Int roduct ions t o M usical Aest het ics. New York: The Free
Press.

Delalande, F. (1988). " Le Gest e, out il d'analyse : quelques enseignement s d'une recherche sur la gest ique de
Glenn Gould." Analyse M usicale, 1er t imest re, 1988.

Ef ron, D. (1972). Gest ure, Race and Cult ure. Hague/Paris: M out on (1941).

Ekman, P., & Friesen, W. (1969). " The repert oire of nonverbal behavior: Cat egories, origins, usage and coding."
Semiot ica, 1, 49-98.
 24

Emmerson, S. (1994). " Local/Field': t ow ards a t ypology of live elect roacoust ic music." In Proceedings of t he
1994 Int ernat ional Comput er M usic Conf erence. San Francisco: Int ernat ional Comput er M usic Associat ion,
pp. 31-34.

Henrot t e, G. A. (1992). " M usic and Gest ure: A Semiot ic Inquiry." The American Journal of Semiot ics, 9 (4), 103-
114.

Iazzet t a, F. (1996). " Formalizat ion of Comput er M usic Int eract ion t hrough a Semiot ic Approach." Journal of
New M usic Research , 25(3), 212-230.

Jaff e, D. A., and Schloss, W. A. (1994). " The Comput er-Ext ended Ensemble." Comput er M usic Journal , 18(2), 78-
86.

Johnson, M . (1987). The Body in t he M ind: The Bodily Basis of M eaning, Imaginat ion, and Reason. Chicago,
London: The Universit y of Chicago Press.

Kendon, A. (Ed.). (1981). Nonverbal Communicat ion, Int eract ion, and Gest ure. Paris/New York: M out on Pub-
lishers.

Kurt enbach, G., & Hult een, E. A. (1990). " Gest ures in Human-Comput er Int eract ion." In B. Laurel (Ed.), The Art
of Human-Comput er Int erf ace Design (pp. 309-317). M assachuset t s, Calif ornia, New York, et al.: Addison-
Wesley Publishing Company, Inc.

Lansky, P. (1990). " A View f rom t he Bus: When M achines M ake M usic." Perspect ives of New M usic, 28(2), 102-
110.

Laurel, B. (Ed.). (1990). The Art of Human-Comput er Int erf ace Design. Reading, M ass.: Addison-Wesley.

Leppert , R. (1993). The Sight of Sound: M usic, Represent at ion, and t he Hist ory of t he Body. Berkeley: Universit y
of Calif ornia Press.

Lidov, D. (1987). " M ind and body in music" . Semiot ica, 66(1/3), 68-97.

M cNeill, D. (1992). Hand and M ind: w hat gest ures reveal about t hought . Chicago: Universit y of Chicago Press.

M insky, M . (1986). The Societ y of M ind . New York: Simon and Schust er.
 25

M oore, F. R. (1987). The Dysf unct ions of M IDI. In Proceedings of t he 1987 Int ernat ional Comput er M usic Con-
f erence. San Francisco: Int ernat ional Comput er M usic Associat ion, pp. 256-263

M ow it t , J. (1987). The Sound of M usic in t he Era of it s Elect ronic Reproduct ion. In R. L. &. S. M cClary (Ed.), M usic
and Societ y: The polit ics of composit ion, perf ormance and recept ion (pp. 173-197). Cambridge: Cambridge
Universit y Press.

M ulder, A. (1996). Hand Gest ures f or HCI (Technical Report No. 96-1). School of Kinesiology, Simon Fraser Uni-
versit y.

Nöt h, W. (1990). Nonverbal Communicat ion. In W. Nöt h (Ed.), Handbook of Semiot ics (pp. 387-418). Blooming-
t on & Indianapolis: Indiana Universit y Press.

Peirce, C. S. (1955). Logic as Semiot ic: t he t heory of signs. In J. Buchler (Ed.), Philosophical Writ ings of Peirce
(pp. 98-119). New York: Dover Publicat ions Inc.

Poyat os, F. (1983). New Perspect ives in Nonverbal Communicat ion. Oxf ord: Pergamon.

Rovan, J. B., Wanderley, M . M ., Dubnov, S., & Depalle, P. (1997). ." Inst rument al Gest ural M apping St rat egies as
Expressivit y Det erminant s in Comput er M usic Perf ormance" . In Proceedings of t he KANSEI The Technology
of Emot ion Workshop , Ant onio Camurri (ed), Genoa, It aly, pp 68--73.

Smalley, D. (1992). " The List ening Imaginat ion: List ening in t he Elect roacoust ic Era." In T. H. John Paynt er Rich-
ard Ort on, Pet er Seymor (Ed.), Companion t o Cont emporary M usical Thought (pp. 514-554). London / New
York: Rout ledge.

St oïanow a, I. (1973). " M usique, graphie, gest e, musique, graphie..." M usique en jeu , 13, 105-114.

Thompson, E. (1995). " M achines, M usic, and t he Quest f or Fidelit y: M arket ing t he Edison Phonograph in Amer-
ica, 1877-1925." The M usical Quart erly, 79(1): 31-171.

Varela, F. J., Thompson, E., & Rosch, E. (1991). The Embodied M ind . M assachuset t s: The M IT Press.

Zagonel, B. (1992). O Que É Gest o M usical . São Paulo: Brasiliense.