Professional Documents
Culture Documents
Conceptual Metaphor in The Practice of Computer Music
Conceptual Metaphor in The Practice of Computer Music
Thesis
by
Approved by:
Reading Committee
_________________________
Chris Brown
Director of Thesis
_________________________
James Fei
Reader of Thesis
_________________________
Chris Brown
Head of the Music Department
_________________________
Dr. Sandra C. Greer
Provost and Dean of the Faculty
CONTENTS
1. Introduction 4
2. Background: 5
3. Interface strategies 21
5. Conclusion 53
7. Bibliography 56
1. Introduction
study of very narrow aspects of its use; by small changes in key places in the
chain of causation, it can be made into nearly any kind of instrument. How
then, within a nearly infinite realm of possibility with regard both to generable
sounds and to input mechanisms, can one decide what kind of instrument to
build into it? After some background on the “mapping question” as it pertains
cognition, I will examine a crude controller prototype which will illustrate the
direction for future work which maintains consistency with this methodology.
4
The ideas which I will bring up in this paper are incredibly simple.
However, their simplicity belies a subtlety which should not be discounted; the
important distinctions.1
2. Background
There are two areas with which the reader will need to be familiar before
most general, it is little more than connecting what goes into the black box with
1 Michael J. Reddy, “The conduit metaphor: A case of frame conflict in our language about
language,” in Metaphor and Thought, 2nd ed., ed. Andrew Ortony (Cambridge: Cambridge
University Press, 1993).
2 Jon Drummond, “Understanding Interaction in Contemporary Digital Music: from
instruments to behavioral objects,” Organised Sound 14/2 (2009): 131.
3 Drummond's paper goes into much further detail about the different ways to conceive of
this metaphor, with varying degrees of complexity. This simpler conception will provide
better clarity here.
5
relationship is straightforward: A performer's physical gestures, breath and
the instrument itself, which may or may not be possible, but will always have
correlation between the processing and the response; the resonance of the
instrument’s body is, in a real sense, both these things. In addition, as Bown, et
incorporate any modifications in the next iteration. Thus the mapping is largely
4 Oliver Bown, Alice Eldridge, and Jon McCormack, “Understanding Interactive Systems,”
Organised Sound, 14/2 (2009): 191.
6
processing is, in contrast to acoustic instruments, something very open, its
especially with the current arsenal of human interface devices (HIDs),5 the
kinds of gestures that can be captured as input to the processing are even more
gesture to a single datum for processing. Several strategies have surfaced to deal
in particular with how to map the input half of the black box metaphor: one-to-
the most transparent, but as a proliferation of such mappings can affect either
differently scaled value from the control. To reduce output mappings while
This could be useful if, as one example, several performers each have separate
controls for the same parameter. Many-to-many combines the two in any
5 Commonly found on laptops at the time of writing are joysticks, trackballs, trackpads (many
multi-touch), keyboards, cameras, accelerometers, photosensors, fingerprint readers, infrared
sensors, bluetooth modems, wireless ethernet, and microphones, to name a few. This listing
excludes any attachable peripherals, which only increase the possibilities.
6 Drummond, “Understanding Interaction,” 131.
7
number of ways, and is probably the most commonly used in practice.7 Notably,
musician towards the instrument is then broken down, and one need not wait
for the next iteration of the instrument or for the builder’s whims to also allow
electronic musicians have been known to approach the design of systems and
computers, for Gordon Mumma, his “designing and building of circuits is really
7 Ibid.
8 A programmer for these purposes can be anyone who causes a change in a computer's
behavior through intentional manipulation of that behavior. This manipulation can be
accomplished through writing original software or by manipulating pre-written software.
9 Bown et al., “Understanding Interactive Systems.”
10 Gordon Mumma, “Creative Aspects of Live-Performance Electronic Music Technology,”
Papers of 33rd National Convention (1967): 1.
11 Among proponents are David Tudor (reported in John D.S. Adams, “Giant Oscillators,”
Musicworks 69 (1996)), Chris Brown, and John Bischoff (Chris Brown and John Bischoff,
Indigenous to the Net: Early Network Music Bands in the San Francisco Bay Area, (2002)
<http:crossfade.walkerart.org/brownbischoff/IndigenoustotheNetPrint.html> (15 April
2010)).
12 Bown et al., “Understanding Interactive Systems.”
8
music world. Interface systems and other components of the sound production
and are being shared as code snippets and modular patches. These objects
(Bown, et al. use the term in both its material and its programmatic sense) can
take a number of different forms, and have varying degrees of utility. They may
be nearly whole programs that could almost be considered pieces in their own
right, or modules that require some manipulation to be usable at all. That these
objects “can be shared, modified and repurposed and are the currency and
computer musician feels compelled both to build most of the instrument she or
he will use, from new or modified components, and to then use the instrument
take Michel Waisvisz’s The Hands instrument. Every sensor of the complex
interface is mapped to a specific parameter in the software-based sound engine.
A change in the engine will result in a new (or altered) instrument. Although
13 Ibid., 195.
14 “Live coding” as practiced by a small community would be a notable counter-example where
the building of the instrument is done in real time. However, this type of interface is far from
intuitive.
9
the interface has not been altered by a change in the mapping algorithm, the
instrument behaves differently. For Waisvisz, changing the algorithms that
constitute the sound engine means learning a new instrument, which involves
the re-incorporation of the conceptual understanding of the engine’s
functionality into bodily memory.15
interface can, for the purposes of motor memory, effectively become a new
the level of processing itself. This example brings forth yet another level of
mapping involved in the overall question, though one not specific to the
signs,17 and in the act of living we develop a mapping schema between these
symbols and our perceptions of the outside world.18 When one learns an
10
to produce that result.19 (see Figure 2)
side of a feedback loop that the cognitive symbolic map now completes (Figure
3). Sensibly, a break in one part of that loop destroys it, and a new loop of
relationships must be built. Given this tenuous hold on connectivity and the
technique and materials (it is a young field), and given also the rapid changes in
the hardware itself, it makes sense that one could fail to settle on a methodical
issue of how to actually capture the gestures themselves. Most HIDs that come
with computers, though they can be used as such, are not intended to be
musical interfaces. The devices beyond the usual keyboard, monitor, and
19 Gerald Edelman refers to this pattern as a re-entrant activation in what he calls a global
mapping, implying that the separation portrayed here is false. The motor response in a global
mapping is co-occurrent with the perceptual response (and thus is part of the same
mechanism). (see §2.2 below)
20 For an interesting discussion in this vein with a comparison of classically trained acoustic
instrumentalists to live-coders, see: Nick Collins, “Live Coding Practice,” (paper presented at
the International Conference on New Interfaces for Musical Expression, New York, USA,
June 6–10, 2007).
11
pointing device, though increasingly common on most laptops, are still not as
detached controllers, or, like Waisvisz, designing and building their own custom
holds in his or her lap and bows much as an orchestral instrument would be
played. Trueman has also designed multi-directional speakers and has used
various corded peripheral interfaces (e.g., tablets and drum pads) for use with
laptops in the Princeton Laptop Orchestra (PLOrk). His aim here was to pull
the laptop performer away from the disengaging stance of integrated controller
12
sound21 with spatial localization via the speaker clusters.22 These solutions show
level to translate gesture data into processable data for sonification, a similar
computer musician/performer often needs to find her or his own solution to the
dilemma of gesture capture within a realm of near infinite possibility. This area
would, however, be one where thoughtful design could improve the ability to
capture musical gestures. With these factors in mind, perhaps the computer,
instrument brings to light the strange fact that when it comes to the ability of
questions that have traditionally had more to do with instrument building and
design than with performance. Because of its relative youth in the performance
world, furthermore, there exist few pre–designed systems that would allow
question then becomes: how can one pare down the realm of possibility into a
13
2.2. Conceptual metaphor
However, it is one on which research in the last thirty years in cognitive science
and linguistics has shed some light. Of enormous explanatory potential and
builds upon its foundation in the theory of embodied cognition. This discipline
has given rise to a conception of cognitive organization that has direct relevance
computer–interface (HCI).
understood in evolutionary biology, and that the units of selection are groups of
neurons which activate together when they receive a particular input stimulus.
interconnected groups24 which Edelman calls maps. Key to music is the fact that
14
these maps are also activated together with non-mapped parts of the brain25 and
with the motor behavior of the animal in question, in this case, the musician.26
organism–environment interaction are the basis for our ability to survive and
The patterns of our ongoing interactions...define the contours of our world and
make it possible for us to make sense of, reason about, and act reliably within
this world. Thousands of times each day we see, manipulate and move into and
out of containers, so containment is one of the most fundamental patterns of
our experience. Because we have two legs and stand up within a gravitational
field, we experience verticality and up–down orientation. Because the qualities
(e.g., redness, softness, coolness, agitation, sharpness) of our experience vary
continuously in intensity, there is a scalar vector in our world.29
The experiences they mention are by no means arbitrary; they are some of the
in the conceptual system.”30 The plasticity of the human mind comes from an
25 i.e., specialized brain structures whose function is not mainly for cognition
26 Edelman, Bright Air, Brilliant Fire, 83–93. (cf. global mapping)
27 Hereafter, I will assume the reader understands that by “metaphor” I mean “conceptual
metaphor” as defined by Lakoff and Johnson (1980), and not “metaphorical linguistic
expression.”
28 Mark Johnson and Tim Rohrer, “We are living creatures: Embodiment, American
Pragmatism and the cognitive organism,” in Cognitive Linguistics Research, 35.1: Body,
Language, and Mind, Volume 1: Embodiment, eds. Tom Ziemke, Jordan Zlatev, Roslyn M. Frank
(Berlin: Mouton de Gruyter, 2008): 32.
29 Ibid.
30 George Lakoff, “The contemporary theory of metaphor,” in Metaphor and Thought, 2nd ed., ed.
15
ability to extend inferences one can make through anticipation of and
language, consider a part of speech: the preposition. These are familiar words
we use everyday, and which are such a vexing problem to native speakers of
other languages learning English: words like in and into. However, even native
speakers are hard pressed to define them. The reason for this difficulty is that
have already introduced us to the container image schema, which is the basis of
the preposition in. At its most basic, the word can only be taken to represent the
most salient part of our idea of a container, which is the location of its
containment: a simple enough idea. With into, the situation is slightly more
characteristics which can map to attributes of more than one schema. The word
16
is an extension of the PATH schema. We understand into as superimposing the
schema.32 The interesting part, however, arises when people say things like,
“Falling in love is just getting yourself into trouble.” When interpreting this
reasoning about states of being, and about transitioning between states of being
certain states, and so we speak of being in and out of love, despite the fact that
there is no such clearly defined line between these states that corresponds to
Such abstract experiences are of extreme importance to art, and their time–
17
inferences about the course of actions, a means through which we generate
expectation about those actions.34 As Johnson and Rohrer note, “We crave the
emotional satisfaction that comes from pattern completion, and witnessing even
just a portion of the pattern is enough to set our affect contours in motion.”35
cadence, and other such musical ideas work. In fact, Candace Brower brings
many of the metaphors discussed thus far into her analysis of Edgard Varèse’s
Density 21.5.36 She analyses the first seventeen bars in a series of phrases, in
each of which the melody, seen as an agent whose will is a driving force toward
pitches in the given phrase. Each phrase builds tension by slowly expanding the
container’s boundaries as the agent battles both against those boundaries and
analysis differs from others’, but, by means of the metaphors she applies, a
metaphor can also serve to obscure reasoning in subtle ways, as I briefly alluded
34 Johnson and Rohrer give the example of a child quieting down as soon as it sees its parent
begin to reach for the bottle. (Ibid., 34.)
35 Johnson and Rohrer, “We are live creatures,” 34.
36 Candace Brower, “Pathway, Blockage, and Containment in Density 21.5,” Theory and Practice
22–23 (1997-98): 35–54.
18
to above. Michael Reddy was an early pioneer of this discipline, and the parable
he invents in his classic paper “The Conduit Metaphor” serves to illustrate one
there are no possible means of communication except through the hub, and no
information can be gained in any other way about each neighbor’s space. Each
inhabitant can only pass notes through the hub, and when one invents a new
tool with which she improves her own life, she passes a note to the others with
there being a drastic difference in the environment and resources in each cell,
the tools are always manifested differently by each toolmaker, unless one or
more of them engage in a dialogue to figure out more about what was intended
to be built versus what was actually built. Reddy presents this paradigm as a
between speaker and listener, or else as only a shadow of the speaker’s intent.
19
However, the conduit metaphor, which is the operative metaphor in common
containers for ideas which are given, packaged and ready, to the listener,
obscures this cooperative effort, implying that the lion’s share of the effort in
In light of this simple but incredibly subtle distinction, Reddy portrays the
magician” who flies over the toolmaker’s world and modifies the hub such that
the toolmakers believe they are receiving the tools themselves instead of
wreak conceptual havoc fresh in the reader’s mind, consider another of Reddy’s
well–worded cautions:
20
pattern of organization present in the first system. Marks or sounds are not
transmuted into electronic pulses. Nor are thoughts and emotions magically
metamorphosed into words. ...Signals do something. They cannot contain
anything.”39
3. Interface strategies
metaphor can account simply and elegantly for a number of structures upon
which we call to manifest our language and our music. Given this frame of
clarity and simplicity, the controller will be one that can be varied over a single
21
metaphorically coherent interface design. A look at existing interfaces and
can be realized in physical objects through which the metaphors they manifest
decreases but its volume increases. These may be just two ways of thinking about
the same physical result, but it’s important to distinguish that the physical
irrelevant, as the important aspect is that when the temperature gets hotter, the
substance behaves in a predictable way which can be factored into the design of
22
the thermometer such that it expresses this change by means of the MORE–IS–
UP metaphor. Such objects “exhibit a correlation between MORE and UP and are
much easier to read and understand than if they contradicted the metaphor.”41
It is key, though, that one realize that in this case one has no control over what
is taken here as the crux of the mapping question, because this mapping is
determined by physical laws. Though one could modify the scale on the output
to indicate Fahrenheit, Celsius, or even mood or DEFCON level for that matter,
current standard has been the one to persevere; we map loud to more in our
41 Ibid.
23
thinking about amplitude in English, so one side of the fader, usually the
extreme that is farthest from the body of the actuating agent, is considered the
top of the fader’s throw.42 Thus, just like the thermometer example, a fader maps
computer output, the mapping is not limited to this one application. What if it
hypothetical scoff. The reader is invited to scoff away, but to know that in fact,
though they are less common, English also exhibits other metaphors for pitch;
low pitch can be deep and high pitch can be shrill, for example.43 Cross–
distinguish wóo su kéte (“voice with a large inside”) from wóo su kuro têi
(“voice with a small inside”). … These concepts of large and small apply to
singing voices, instrumental sounds, and speaking voices, and the idea
incorporates both pitch and resonance attributes. A large voice is both lower in
42 I am indebted to James Fei (personal communication) for bringing the BBC–style fader to
my attention.
43 The words used are not always nicely paired antonyms.
24
pitch and more resonant than a smaller voice.44
Farsi, Turkish, and Zapotec, three unrelated languages.45 Further, Eitan and
conventionally use them. They found that when asked to describe pitch,
subjects would consistently map pairs of antonyms to the pitch vector in the
same way as would native speakers of languages which did conventionally use
English between the top of a slider being both louder and higher pitch
different vectors. Small sounds (our high sounds) are quieter in languages that
use the big/small metaphor, while big sounds (our low sounds) are louder.47
25
However, following from the above discussion about metaphor and its
finding that these ideas can be readily understood in novel situations is not
image schematic structure which is, if not universal, then at least quickly
which the relevant principles are taken into account. The first step in designing
Were we to choose, for example, the Shona pitch metaphor crocodile/those who
26
be harder to implement than others. Furthermore, whether or not its physical
form would afford49 thinning and thickening would be debatable. For these
reasons, the current attempt will involve the big/small pitch metaphor used in
ball.51 Such a shape and material choice would encourage squeezing, which
Figure 6: prototype control for big/small pitch metaphor relaxed state would correspond
to lower pitch.52 However, rather than the ideal sphere, the prototype for the
controller (see Figure 6) is a cube the size of a small handful, composed of anti-
49 On the notion of affordance, see: Orit Shaer and Eva Hornecker, “Tangible User Interfaces:
Past, Present, and Future Directions,” Foundations and Trends in Human–Computer Interaction
3/1–2 (2009): 62–63. For incisive clarification, see: Shaleph O’Neill, Interactive Media: The
Semiotics of Embodied Interaction, (London: Springer–Verlag, 2008): 49–65.
50 Zibkowski cites the use of this metaphor in Bali and Java in: Lawrence M. Zibkowski,
“Metaphor and Music Theory: Reflections from Cognitive Science,” Music Theory Online 4/1
(1998): note 12.
51 A nod must go to Andrew Mead who, unbeknownst to me until revision of this paper,
proposed a very similar thought experiment in a footnote of his article: Andrew Mead,
“Bodily Hearing: physiological metaphors and musical understanding,” Journal of Music
Theory 43/1 (1999): 17, note 13.
52 An added benefit to such a design is that the controller would also be coherent with a
tense/relaxed opposition for other sonic parameters, although that application will not be
explored here.
27
static foam rectangles found in the packaging of integrated circuits, collected
together with two wired electrodes on opposing sides of the cube.53 This whole
assembly is a resistor
which is attached to a
simple inverter
integrated circuit,54 as
half of a voltage
Figure 7: circuit diagram for the squeeze ball big/small pitch controller. R1
and C1 can be varied to change the frequency of the oscillator divider which feeds
(approximate values suggested here). A 2 kΩ resistor was added on the
oscillator output to reduce the extremely hot signal. the audio input to a
computer. (see Figure 7) In this way one can control the amplitude of the analog
oscillator circuit with the squeezable cube. Importantly, the effect of the
distance between the electrodes yields less resistance. The computer tracks the
amplitude of the incoming signal, using that value to vary the frequency of
28
SuperCollider:
Ndef(\hiamp_to_hipitch, {
var in = SoundIn.ar(0), amptrack;
amptrack = Amplitude.kr(in, 0.01, 0.01, 1200, 400);
SinOsc.ar(amptrack ! 2, 0, 0.5);
}).play;
Since, in this case, the amplitude increases as the cube is squeezed, the
amplitude tracker’s output can simply be plugged into the frequency argument
causes a sound to move to a higher pitch when the controller gets smaller.
both for other instances of simple metaphors and also for more complicated
mappings and metaphors. (see §3.3.2 below) If one were to build on this
55 Though the processing in this example is fairly transparent (tracking amplitude and
assigning to pitch) it certainly still counts.
29
What is most important to keep stock of is twofold. First, the
correspondence between the physical form of the controller and the intended
output in step two does not exist until the successful completion of step four,
and only then if step four is approached in keeping with the aim of
Second, as is completely obvious when laid out this way, steps one through four
are not one step. These points are stressed to lay out the areas where conduit
interfaces and mapping. Though the simplicity of this example makes it easy to
grasp, one should remember that the signal is not contained within the controller.
Only once we built both the control interface and its chain of causation up to
step four could we generate a signal to transmit, and only when a listener
possible to place the variable resistor, the squeezable foam, on the other side of
the voltage divider, thus causing a drop in amplitude from the output of the
analogue circuit when the cube is squeezed. Had we chosen to do it this way,
and then rewrote the computer code such that a drop in amplitude caused a
30
higher frequency output, then there would be no perceptible difference
between the two versions of the controller. However, had we instead chosen to
correlate the bigger, relaxed state to a higher frequency, by moving the resistor
and leaving the code the same, the controller would no longer be
to vary the amplitude of the output, creating a foam–ball theremin of sorts, the
loudness. Squeezing the resistor to create a drop in input amplitude could map
destroy the coherence, but not necessarily the usability. The choice in this case
usability when they cohere with our conceptual expectations of their behavior.
Jacob et al. have demonstrated this empirically, noting that a controller which
task that requires the matching of three conceptually integral attributes, while
31
one that can only vary two at a time is faster for matching conceptually
means to match two shapes in X–Y position and size, while matching X–Y
position and color works better with a mouse; their experiment allows
positioning the shape with an unmodified mouse movement and changing the
color through mouse movement along one axis in conjunction with a button
depressed. They posit that this match between controller and task comes about
have also applied these ideas specifically to musical tasks, and Antle, Courness
mapping in gesture capture that is of concern in this paper. However, all these
studies exhibit a particular focus; as Wanderley and Orio ask, “[w]hat is part of
the composition, and what is part of the technology? How can we rate the
usability of an input device if the only available tests were done by few–possibly
testing the validity of these ideas, and applying them to the design of interfaces
56 Robert Jacob et al., “Integrality and Separability of Input Devices,” ACM Transactions on
Computer–Human Interaction 1/1 (1994): 3–26.
57 Marcelo Wanderley and Nicola Orio, “Evaluation of Input Devices for Musical Expression:
Borrowing Tools from HCI,” Computer Music Journal 26/3 (2002): 62–76.
58 Alissa Antle, et al., “Human–computer–intuition? Exploring the cognitive basis for intuition
in embodied interaction,” International Journal of Arts and Technology, 2/3 (2009): 235–254.
59 Ibid., 62.
32
for general musical use. However, on the strength of their work, as well as on
in computer music today, interface design and mapping choice, along with the
more traditional elements of structure, method, material, form and aesthetic are
It has been said that a good mapping should be intuitive, in the sense that you
should immediately understand the internals of the system. But this is not true
for most acoustic instruments. Many musicians do not know their instrument
from a physics point of view. Some phenomena are extremely complex, e.g.,
multiphonics in wind instruments, but instrumentalists learn to master them.61
degree of randomness, and thus raises this objection in order to defend his
60 David Wessel and Matthew Wright, “Problems and Prospects for Intimate Musical Control
of Computers,” Computer Music Journal 26/3 (2002): 11–22.
61 Palle Dahlstedt, “Dynamic Mapping Strategies for Expressive Synthesis Performance and
Improvisation,” Computer Music Modeling and Retrieval: Genesis of Meaning in Sound and Music.
5th International Symposium, CMMR Revised Papers (2008): 237.
33
an expressive performance instrument. Along similar lines are concerns raised
by Ian Whalley on the idea of software-agents, stating that one “should then
Machine agency can then lead or follow in the interactive process with human
These are completely valid points whose approaches and results I would
be sorry to see gone from the world of computer music. I would venture one
“intuitive” interfaces. The system of mapping proposed above does not suppose
output, as we saw with our input amplitude controlling the output pitch, it is
irrelevant to the user of such an interface what the internals of the system
actually are. It is precisely what is limited to the externals of the system that is
of concern for the use of such a controller. Indeed, as advocated by the IUUI
62 Ian Whalley, “Software Agents in Music and Sound Art Research/Creative Work: current
state and possible direction,” Organised Sound, 14/2 (2009): 165.
34
research group (Intuitive Use of User Interfaces), the very notion of intuitive use
precludes any conscious understanding on the part of the user; they define
builder of the interface and the mapper of the stimulus to the response is also
the composer and the performer, and so that person likely does “immediately
experience, … are learnt early in life, shared by most people and processed
reaction times and error rates.”64 As a young person in the 21st century
feeling the pressure to build and learn to play a new instrument for each piece,
as is often the case, I welcome the possibility of “decreasing reaction times and
error rates” so that I can focus on the music itself. That said, however, the music
itself, these days as like no earlier time in history, is so incredibly varied that all
35
Bearing these ideas in mind, let us now turn to examination of some
existing interfaces: the tangible user interface object in general, and then a
all uses of computers. Shaer and Hornecker cite three basic types of TUIs:
constraints.65
One of the most powerful ideas with regard to TUIs is the notion of
65 Ibid., 49–50.
66 Ibid., 47.
36
devices.”67 User feedback is collocated with the input device, thus giving the
seductive illusion that the user is directly touching the digital information. The
closely the input focus is tied to the output focus in a TUI application, or in
other words, to what extent does the user think of the state of computation as
“embodiment” index is high, the objects can help to extend memory during
computing and comes in a large part from product design, an arena where
mappings are predetermined by the designer and usually not alterable by the
end user, the coupling between TUIs, which generate an input, and their output
Wensveen discuss action to function coupling in a way that implies that the
design process obviates the basic concern in this essay. In their paradigm,
although they cite six parameters that need to coincide for a “natural coupling,”
67 Ibid., 48.
68 Ibid., 52–53.
69 Ibid., 67.
37
all the parameters map directly to the output; the designer chooses and fixes the
entirety of the black box.70 Thus, much of the discussion about what is
essentially mapping conflates the first four steps above (in §3.1.2), and the
discourse is replete with conduit metaphor phrases implying that the TUI
objects contain the information read from the outputs with which they are
coupled.
metaphorical coherence despite any discussion about them. For a brief example,
Figure 8: a fiducial marker are objects in this bounded space, and are therefore
(76)
beholden to the “physical” laws that define that space. In the commercial
be, when inputs and outputs between objects are connected dynamically and
70 Kees Overbeek and Stephan Wensveen, “From perception to experience, from affordances to
irresistibles,” Proceedings of DPPI03 (Designing Pleasurable Products and Interfaces (New York:
ACM, 2003): 95–6.
38
automatically based on physical collocation. That these markers are objects in a
bounded space implies that they are also destinations on an indeterminate path
through this space,71 which lends such interfaces to exploration via the journey
Beyond the interactive surface category, constructive assemblies are also open
one which we also occupy. Objects in this category are often manifested as
interfaces are only intuitive so long as their chosen mappings pan out with
form a separate category, they can be seen with respect to the present
With these ideas as background, I sought with the piece discussed below
39
to incorporate aspects of this type of interface in a simple and metaphorically
coherent controller.
performance piece that I composed and performed in 2010. With many of the
ideas discussed thus far floating nebulously apart from the verbal level of my
mind, I attempted to build a new interface which would yield a more human–
controlled sound than I had been able to achieve to that point with other live
electronic pieces. I had been able to develop sounds which were expressively
modulated by the rotation and position data from fiducial markers, recognized
idea was to place them on large objects with which one or more performers
would interact in a rule–based game piece. However, as most of the time was
spent with sound design, and the concert was impending, that idea was
ball, and placed the fiducial markers in various locations on its surface. Some
73 After the interstellar trade conglomerate in Frank Herbert’s Dune series. (The piece was
written for a science fiction–themed concert.)
40
were repeated, and they were grouped in clusters such that different areas on
the ball would have distinct sonic characters. The compositional process was
then the choice of sounds, parameters of variance, and the arrangement of the
interface allows tactile control over rather high–level aspects of the musical
connection.
A shortcoming of this interface was the use of the built–in camera on the
image, as the metaphorical agent should have been navigating the surface from
my point of view rather than from the computer’s. Though similar visual
experimentally with habituation,74 the point with this interface was to be usable
74 George Stratton, “Vision without inversion of the retinal image,” Psychological Review 4/4
41
without any training. A vast improvement would be to use an external camera
mounted somewhere on my head such that the computer’s input image would
be from the same point of view as mine. Exploration of the surface between
Along the lines of the ideas briefly skirted in “CHOAM,” my future work
sound in conjunction with the use of TUI objects. This mapping, however, will
cheapest such imager is the Xbox Kinect controller, but open–source versions
a known image onto a surface along with stereo imaging, which can calculate
(1897): 341–360.
42
in the known image and of triangulation between the two cameras. This
incoherently, with the metaphors that are co–expressed verbally in the time–
multidimensional spaces and to sonify dance, along similar lines as Antle et al.77
to these objects and to realtime exploration of musical form. From this point of
covered with velcro on which movable and recombinable fiducial badges and/or
barcodes can be placed. The new interface would use barcodes or smaller
fiducials to “zoom into” the fiducial destinations beside them, thus redefining
75 Eve Sweetser, “Looking at space to study mental spaces: Co–speech gesture as a crucial data
source in cognitive linguistics,” in Methods in Cognitive Linguistics, eds. Monica Gonzalez–
Marquez, Irene Mittelberg, and Seana Coulson (Amsterdam: John Benjamins Publishing,
2007).
76 That is, “metaphorically by means of physical movement.”
77 Antle, et al., “Human–computer–intuition?” 242.
43
the explorable space bounded by the object in hand. “Out–zooming” codes
would bring it back “up” a level, or physical zooming would shift focus to the
larger fiducials. One ball could thus effect multiple levels of control. The
Along the lines of the TUI constructive assembly model, marking physical
planning small sculptures which bear barcodes that can be scanned to produce
sound, effectively making the sculptural object the score, though it may only be
destinations, where a branching path will be defined along the sheet; sounds
will be both signified by and encoded in the barcodes along this path. The
Aside from these mainly technical directions, I plan also to look deeper
into the mapping question. While one can usefully think of mapping in
44
Drummond’s more mechanical terms, taking a step back and re–acknowledging
the end goal can also be of help. Wanderley and Orio do so by positing different
control.78 However, rather than adopt their terminology here, I prefer to use the
more generic “low–level” versus “high–level” control, as these terms don’t invite
several basic image schemas beyond the few discussed above, some of which
schemas follows:
1. containment
2. balance
3. blockage
4. diversion79
the pitch ball, and higher–level mappings like the “CHOAM” controller. As an
78 Wanderley and Orio, “Evaluation of Input Devices for Musical Expression,” 69.
79 See Brower, “Pathway, Blockage, Containment,” 36.
45
writing, both in laptops and in mobile phones, the balance schema seems ripe
using a balance scale as an interface. Tokens could be placed on the pans of the
scale, and then recognized by the computer as triggers for sound sources, either
could offset the scale’s balance, and thus the composition of the musical
could be factored in by tracking the rotation of the torque arm and pans,
executing appropriate changes when reversal (on the X–Y plane) or deflection
(on the Z plane) of direction occurs. Force and agency in such an interface
serve to illustrate the kind of possibilities that pursuit of this line of thought
can open up. However, the mechanics of generating computer music is not the
80 Along vaguely similar lines, as mentioned above, the pitch ball example above could also be
made to conform to a tense/relaxed concept by mapping to higher–level musical structures
in a similar way.
46
4. Informing the conceptual sphere
Metaphor is also a rich conceptual domain that can be drawn upon for
as an example of a work informed by this area of inquiry, both in its musical and
collaboration with Rebecca Gilbert, a Bay Area modern dancer and the main
choreographer.81 In the piece, six dancers first bring plain, empty boxes, on
code, to the musician, in the role of the checker, to be scanned for them,
they take it upon themselves to scan codes on different parts of these boxes,
and textural. The first six or so minutes, the pitched atmosphere, consists of
more homogenous group movement, from which some individual dancers show
a desire to break away, while the second half, about six minutes of the chaotic
81 Five other dancers, Kate Knuttel, Sergio Lobito, Mica Miro, Jeanne Platt, and Natalie Rael,
also contributed to the development of the movement in the piece.
47
atmosphere, is characterized by highly social interactions between pairs of
dancers and socially motivated actions by individual dancers. The course of the
depending on the randomly selected box. The pitch material comes from a scale
single chord structure gradually to different center pitches within the octave.
The musical structure is complicated by the fact that the melodic line, derived
from a path moving from pitch to pitch within the chord (see Figure 11 below),
is separated into three parts, with rests holding places for pitches that appear in
another part. All of these parts played together with an identical rhythmic
structure would yield the underlying melody, but each part in practice has a
systematically varying pitch contour when played together. Each box contains
codes for parts of the pattern around a particular tonal center, and so each time
which happens in three stages by virtue of its split into three parts. The chaotic
din of shopping carts rattling through a large warehouse and by the kernel of
48
The program notes were presented as follows:
elements seek to manipulate the majority into believing they haven’t the right to
this participation, that only those elements’ own idea of a proper course should
be followed. The ritual of which the dancers are part in the first half, “the
purchase,” is a familiar one to us, and is also one that is a powerful form of
conduit metaphor.
82 Fredric Rzewski, “Music and Political Ideals,” in Nonsequiturs: writings and lectures on
improvisation, composition and interpretation, eds.Gisela Gronemeyer and Reinhard
Oehlschlägel, (Köln: MusikTexte. 2007): 188–200.
83 Progress in the sense of movement somewhere, not necessarily betterment. In other words,
in our evolution, in its correctly construed sense.
49
The present discussion about just–intonation must necessarily assume
some background knowledge.84 The piece explores pitch in terms of the seven–
Figure 9: chord shapes pictured in a 7–limit pitch space: dimensional space entails a
dominant 9th (top) and the "4th/6th" chord (bottom)
number of correspondences which can carry over from that understanding and
apply to reasoning about pitch relationships and movement within that space.
Thus, chords can be represented as shapes within that space (see Figure 9),
transposition within that space (see Figure 10), purely by virtue of the system of
its organization. With regard to this piece, the melody was built thinking of
these chord shapes as path descriptors, with the pitch sequence manifesting the
course of travel along that path (see Figure 11). Thus, the compositional
84 For the reader who would like more background, the author would suggest: David B. Doty,
The Just Intonation Primer: an introduction to the theory and practice of just intonation, 3rd ed. (San
Francisco: Other Music, Inc., 2002–6).
50
extension of the SOURCE–PATH–GOAL schema.
assumption that there is an indefinable something that can be received that can
fill these lacunae, and the unspoken further assumption about these things is
that they will be received, unpacked, and taken in without effort, as that effort
has already been expended by the packager, the giver as he would be led to
51
believe. In the piece, the checker/musician represents the mouthpiece of the
forces who bear this message, and the computer, the sound source, is the hub in
Reddy’s toolmaker’s paradigm. The music is the set of instructions which are to
passed between the performers and the audience, rather than between each
other. In the first half, the signal, which is the sound itself, open for
participate. They present these empty boxes hoping for the solutions they are
purported to hold, and leave confused when the boxes are immediately
discarded. Once they realize that the information they seek is not contained
within these shells, they are freed from the group bondage and interpersonal
growth and relationships can ensue. Thus, the composition can be interpreted
It should be noted that this piece was originally conceived of long before
earnest. However, the ideas in Reddy’s article had been simmering below the
52
with life experience that this piece on which I had expended so much thought
illustration of how it, and the theory of conceptual metaphor Reddy’s article
5. Conclusion
into account this fact of our status as animals in a physical world, one that is
situation, can open the door onto a world of expressiveness that might
53
metamorphose into something that, possibly by virtue of our metaphorical
great benefit.
54
6. Appendix: contents of accompanying media
video partition:
2. Choices
data partition:
55
7. Bibliography
Brown, Chris and John Bischoff. “Indigenous to the Net: Early Network Music
Bands in the San Francisco Bay Area.” 2002.
<http://crossfade.walkerart.org/brownbischoff/IndigenoustotheNetPrint.h
tml> (15 April 2010).
Collins, Nicolas. Handmade Electronic Music: the art of hardware hacking. 2nd ed.,
New York: Routledge, 2009.
Doty, David B. The Just Intonation Primer: an introduction to the theory and practice
of just intonation. 3rd ed. San Francisco: Other Music, Inc., 2002–6.
56
Eitan, Zohar and Renee Timmers. “Beethoven’s last piano sonata and those who
follow crocodiles: Cross–domain mappings of auditory pitch in a musical
context.”Cognition 114 (2010): 405–422.
Edelman, Gerald. Bright Air, Brilliant Fire: on the Matter of the Mind. New York:
Basic Books, 1992.
Johnson, Mark and Tim Rohrer. “We are living creatures: Embodiment,
American Pragmatism and the cognitive organism,” in Cognitive
Linguistics Research, 35.1: Body, Language, and Mind, Volume 1: Embodiment.
Edited by Tom Ziemke, Jordan Zlatev, Roslyn M. Frank. Berlin: Mouton
de Gruyter, 2008.
Lakoff, George and Mark Johnson. Metaphors We Live By. Chicago: University of
Chicago Press, 1980.
57
Mumma, Gordon. “Creative Aspects of Live-Performance Electronic Music
Technology.” Papers of 33rd National Convention. New York: Audio
Engineering Society, 1967.
Naumann, Anja, Jörn Hurtienne, Johann Habakuk Israel, Carsten Mohs, Martin
Christof Kindsmüller, Herbert A. Meyer, Steffi Hußlein, and the IUUI
research group. Engineering Psychology and Cognitive Ergonomics: Lecture
Notes in Computer Science. 4562 (2007): 128–136.
Shayan, Shakila, Ozge Ozturk, and Mark A. Sicoli. “The Thicknes of Pitch:
Crossmodal Metaphors in Farsi, Turkish, and Zapotec.” Senses & Society
6/1 (2011): 96–105.
58
Sweetser, Eve. “Looking at space to study mental spaces: Co–speech gesture as a
crucial data source in cognitive linguistics.” in Methods in Cognitive
Linguistics, Edited by Monica Gonzalez–Marquez, Irene Mittelberg, and
Seana Coulson. Amsterdam: John Benjamins Publishing, 2007.
Trueman, Dan. “Why a laptop orchestra?” Organised Sound 12/2 (2007): 171-179.
Wanderley, Marcelo and Nicola Orio. “Evaluation of Input Devices for Musical
Expression: Borrowing Tools from HCI.” Computer Music Journal 26/3
(2002): 62–76.
Wessel, David and Matthew Wright. “Problems and Prospects for Intimate
Musical Control of Computers.” Computer Music Journal 26/3 (2002): 11–22.
59