Lines and Sounds: Hermeneutics of Multimedia Improvisation

In this paper, I propose to outline a mode of interpretively approaching multimedia

improvisation from an audience perspective; a methodology for subjectively interpreting
productions that combine two or more media However, there is no reason for this
methodology not to e!tend into more formalised appreciations, in the form of written
analyses, as well as inform one"s attitude in performance #hough I will be focusing on
audiovisual productions, the ideas e!tend easily into other $inds of multimedia, such as wor$
with te!t or movement

%s &icholas 'oo$ writes in Analysing Musical Multimedia, different media can find
themselves in three $inds of relationships: conformance, complementation and contest
'onformance signifies two media being entirely (the same); directed at the same goal;
accomplishing the same function 'omplementation appears when two media wor$ together;
fill each other in 'ontest is, predictably, two media somehow placed in opposition

%part from the fact that it is hard to imagine two media wor$ing in true conformance *
indeed, the e!amples that 'oo$ gives are of synaesthesic relationships * it is apparent that it is
complementarity and contest that will form the basis of an interpretation that foregrounds the
relationships between media

#he way these relationships are created is through metaphorical mapping: some +uality
found in one media is mapped onto the other, thus changing the resultant form ,or e!ample,
the image is made up of relatively static points, oscillating a little bit #he musical component
begins +uite calmly, but gradually turns into chaos; this development in the relationship will
change our perception of the final product as well as both constituents

-f course, every sound and every sight has an enormous number of factors and
parameters that could be mapped in this way and each subjective interpretation will
foreground a different one &ot only is this an inherent +uality of interpretation, but it also
serves to e!pose what we can consider to be the idea of clarity in multimedia improvisation
#he clearer the foregrounding and the clearer the relationship and its development, the more
.sense" a larger amount of listeners will be able to ma$e out of a given improvisation

% traditional reading would presume an author, capable of prioritising certain aspects,
and tying in all of the developments in the relationship between media to a general concept
However, there is no single author in improvisation, and so this model becomes unhelpful

If we consider /ean0/ac+ues &attie1 tripartite model, by which he e!plains the
functioning of semiotic systems, we will be able to see more clearly the ways in which a
traditional te!t0based mode of interpretation might prove unhelpful for improvisation

&attie1 divides analysis 2which, for our purposes, I ta$e to be analogous with
interpretation3 into three levels: the poietic, the esthesic, and the immanent or neutral level
#his divide indicates the three ways of loo$ing at a te!t or piece of music: the poietic is from
the creative side; the thin$ing and intention of the composer as he wor$ed on the piece #he
esthesic is the analysis of the reception by an audience or interpreter, and the problematic
neutral level is the study of the te!t (in itself); as it e!ists as a material trace in the world

,irstly, since what we are trying to address improvisation and the interpretation of
improvisation as an entirely contingent, momentary and present0tense phenomenon, there is
no point of spea$ing of the neutral level in improvisation, which leaves us only the poietic and
esthesic levels #he first and most obvious fact is that there is a multitude of poietic levels to
study; that of each particular improviser 4ach audience member tries to interpret not only
how each input sounds to him, but * if he is following the relationships between media * what
each musician is trying to achieve in relation to the whole

More interestingly, it is important that each improviser is not just thin$ing about how
his input will influence his relation to the whole 2ie the poietic3 but also how it might be
received by a listener 2ie the esthesic3 #herefore, everyone involved is constantly negotiating
both sides of the e+uation * the semiosis isn"t a clear line of creation of meaning as would
usually be projected, but rather a comple!, instable web of constantly shifting, permeable
relationships

#hese are of course not restricted to the relationships between the media, but also
between the improvisers active in one medium, but it generally seems to be true that
demarcating the interactivity between media also fosters or incites a greater awareness of the
relationships within the individual media themselves However, as soon as there is no single
authorial identity at wor$ 2which 'oo$ generally presupposes3, we need to consider a much
wider array of relationships

It is also important to note a willful sleight of hand that I have made in the last few
paragraphs In disregarding the neutral level as an impossibility in live improvisation, I have
also made the interpretation of the production dependent solely on intention, in both senses of
the word

&ot only is the intention of the creator in the eyes of the listener important, but the
intention of the listener is also important in the eyes of the creator; he plays based on his best
prediction 2hopefully3 of how an audience member, just as present at the moment of creation
as he is, can understand his input

%nd similarly, by prioriti1ing the relational and semiotic aspects of the production, I
have suggested a view of the production as an intentional object rather than a material reality
#he piece or wor$ of art is in no way ontologically e!istent in a material trace; the aspects that
give it meaning are in the phenomenal space of the producers and consumers; in the
relationships connected by each individual subjectivity #his then allows the entire
performance to be viewed not as a fi!ed object, but as a wide array of interpretations or
memories of events and processes, which could be widely different for every participant, and
most importantly, which is created just as much by the audience as by the performers

5eturning to the idea of clarity, it is apparent that the clearest improvisations will leave
a narrower set of interpretations than an e!tremely unclear one I don"t thin$ either of these
options says anything about the potential impact or +uality of the production, but I thin$ there
is a certain worth in e!ploring the theoretical basis that can inform both a performer"s and an
audience member"s view of multimedia improvisation, and to build this approach (from the
bottom up) * by beginning an improvisation with the most basic and clear relationship, there
is an implicit suggestion in how to view the production, which can then be e!tended as far as
is desired

It is another contention of semiotics that meaning arises from difference; when
something has a negative value against something else Meaning in music wor$s by much
wider, different methods, as attested by &attie1 and his teacher /ean Molino, but in this case
two things are central: the essential ontological difference between the various media, wherein
one wor$s by sound and one by sight, and the difference in any process or series of events that
develop, over time

If we return the idea of a multi0poietic analysis that we came across earlier, than we will
come to a concept that I"ll call temporal semiosis 4very action is defined in its relation to the
actions of the other improvisers and the previous actions of that improviser himself
4verything that continues, in a process, for some period of time, can be considered the
temporary norm, and every deviation from it the brea$ in which meaning can arise, until a
new temporary norm is established, and so on

#his e!planation gives off the impression that these relationships are fi!ed, but of
course, they only e!ist to the e!tent that someone mentally creates them #here is also nothing
said about the length of time to re+uire for us to be able to consider something a new norm,
and it is the negotiation of all of these contingencies, in a conte!t in which a lot is usually
happening at once, that ma$es this basic and banal idea so e!citing

In order to further +ualify the $inds of relationships that I have been discussing, I will
now list several $inds of metaphorical relationships which I have observed during the
rehearsal process #his list is in no way e!haustive and the typology is intended more to
suggest than to delineate

#he most obvious choice are spatial metaphors, which are intricately bound up with
music as we culturally perceive it 6e tal$ of high and low notes, full and deep sounds, the
volumes being low and high and so on #hese are the metaphors that are easiest to transfer
into an improvised production, seeing as they are already conventionalised before the music
starts However, there can also be arbitrarily chosen and defined spatialisations, to which I
will return in a few minutes

%n aspect that connects the two media and that defines a large amount of the potential
relationships is temporality; the fact that they both happen over time #his is a necessary
condition not only for processual relationships, of which there are so many I will not waste
time defining them, but can also inform and e!tend into more abstruse territory

#he most basic form of temporal metaphor is presence or absence; when there is a thing
you can see, there is a thing you can hear 'ontinuing from this, there might begin to develop
a tendency to focus on saturation, intensity, as well as speed, which +uic$ly becomes rhythm,
and so on

-f course, there is no reason to leave out some of the most abstract meanings attributed
to music: emotion #hese can be presented both through conventionalised symbols, such as
sad music or a smiley face, as well as through more suggestive, subliminal development of
mood and atmosphere In effect, emotional signification will probably ta$e place through
spatial and temporal means any way: we attach agitation and movement to certain states of
mind, calmness to others #his is an e!ample of how more comple!, abstract and yet
comprehensible relationships can be built out of very basic means

Having mentioned convention, there is no alternative but to point towards the rabbit
hole that is referentiality %s soon as symbols, signs or icons come into play, whether in the
visual, musical or te!tual spheres, they bring with them all of the associations that have the
potential to create enormously comple! relationships, as well as interpretively isolating a
certain portion of the audience or improvisers It brings into play a huge array of possibilities
of misfiring utterances, as /L %ustin calls them * instances of communication that somehow
misses its mar$; signifies something other than was intended to some of the addressees #his
of course brings a welcome contrast to the stripping down of communication and meaning
into basic relationships

#his leads me to the conclusion, in which I would li$e to suggest what I see as the worth
of the approach that I have outlined 7y distilling communication into an e!tremely primeval
form, that of direct relationships between clearly identifiable objects, we offer a way for
improvisation, and the art that results, to have its meaning, its interest and its attraction
located in none of the constituent media or in a combination of their material traces, but in the
phenomenal or mental space in which the conceptual blending of the metaphorical
relationships ta$es place

#his is the ultimate ideal of the 8esamt$unstwer$; that the effects of all the individual
arts mi! li$e a potion in the strong0willed consciousness of the perceiver * all that without the
problematic figure of the author, but rather a sense of shared co0authorship with the audience,
including the responsibility that comes with it

-f course, all art e!ists to a certain e!tent in the space that I spea$ of; anything that we
interpret intellectually has its place there to a certain e!tent, but the involvement and
participation necessary for perceiving two or more media at once allows us, as listeners, to
remain in this space for a concentrated and prolonged period of time It allows us the same
level of access as a trained analyst listening to his favourite 7eethoven sonata, but without
learning any 7eethoven sonatas #he aesthetic or sensorial +ualities present in the production
are mostly welcome contingencies Marginalised by the emphasis on relationships, they serve
no major purpose, e!cept for the potential e!tension of access, based on the idea that things
that loo$ and sound good will be more inviting to interpret in a relational manner

#his is a first attempt at formulating in writing an approach that I believe could be
e+ually well shared by doing, showing and performing However, I also see the credit in
attempting to describe this approach in as clear terms as possible, seeing as it does seem a
relatively new approach in which to approach improvising; one where the (aesthetic) effect is
not the main goal, but in which each player is constantly striving for, and usually balancing on
a thin string the relationships, developments and processes