Professional Documents
Culture Documents
Gesture
Gesture
Gesture
users.wowway.com/~brian_rotman/gesture.html 2/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 3/26
23/05/13 gesture
they semiotic or experiential? -- the answer will depend on the agent, on the 1. See chapter 2 of The Time of the Tribes: the
viewpoint. "There are two musics", Roland Barthes, tells us, "the music one Decline of Individualism in Mass Society
listens to, the music one plays. ... Two different arts, each with its own (translated Don Smith). London: Sage
history, its own sociology, its own aesthetics, its own erotic". In just the Publications, 1996; and The Shadow of
same way, dance as spectacle, as an art form of bodies choreographed for an Dionysus: a Contribution to the Sociology of
audience and designed to express, signify, and encapsulate meanings and the Orgy (translated Cindy Linse and Mary
affects for observers, can hardly escape the semiotic; whereas dance for the Palmquist). Albany: State University of New
dancer immersed in these same movements and gestures and participating in York Press, 1993.
their production is within the experiential; and likewise, the same doubleness
pervades the gestures of all theatrical performance. In what follows, my
concern here will be entirely within the semiotic -- the body as meaning,
representation, and significance. The topics of the participating body
immersed in the construction of experience, and the instrumental body
captured within a machinic circuit, I leave for another occasion.
users.wowway.com/~brian_rotman/gesture.html 4/26
23/05/13 gesture
argued for gesture as the origin of speech (an idea currently revived as the
claim that gesture contains the origins of syntax), and a figure like John
Bulwer (inventor of the first fingerspelling alphabet) could attempt in the
17th century to construct an entire gesturology, one that
taxonomized/anatomized significant body movement in terms of a close
analysis -- a "dissection" -- of the body's muscles and argued for the
expressive adequacy (if not superiority) of gesture to speech. From a
different direction (and at a level prior to the tripartite division of gestures
embraced here), haptic gestures, those involving touch, seem especially
important; as if the possibility of contact with the body of the self or the
other (kissing, hugging, clapping, slapping the back, shaking hands,
punching, grabbing and stroking oneself, squeezing the shoulder, wringing 2. Roland Barthes Image-Music-Text (translated
one�s hands, and so on) gives such gestures a particular intensity or Stephen Heath). London: Fontana/Collins,
potency; as if the creation of a circuit of reciprocity, a doubling or folding 1977: p. 149
over to form a new inside/outside, functioned as the site of origination not
available to non-haptic gestures. Let me briefly cite two examples.
According to more than one neurological account of the mind, a form of
internal touching, a virtual auto-hapticity, is a prefiguring or rehearsal or
condition for the possibility of self-consciousness. The idea rests on the fact
that most of the brain is connected not to sensors measuring the world but to
other parts of the brain -- including parts which contain a map of the body
on the surface of neo-cortex. This makes a form of self-touching or
monitoring inside the brain possible, in which the brain, by
exciting/inhibiting regions of itself, achieves a kind of phantom or virtual
proprioception, a primitive template of the self becoming (auto)aware. And a
quite different example of haptic gesture, touching an external social other
rather than an inner neurological self, comes from the work of psychologist
Robin Dunbar, who hypothesizes that a particular form of prolonged haptic
gesture -- chimpanzee grooming -- gave rise to speech. The idea being that
when primate bands got too big, too disseminated, too complex as a group
for grooming, a surrogate for it -- gossip -- which served the social purpose
of grooming but more efficiently, was made available through the
development of spoken language. But speculations on the origins of
consciousness and the evolution of spoken language are not my concern
here. I'm interested in the relation between speech and gesture in their
present day conception, specifically in the hierarchy of speech over gesture;
users.wowway.com/~brian_rotman/gesture.html 5/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 6/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 7/26
23/05/13 gesture
themselves as deaf [small 'd'] and re-signate themselves as Deaf [capital 'd']; Hears: A History of the Deaf New York:
as, in other words, a linguistic group who use Sign (no differently from any Random House, 1984), p. 391
other group defined/constituted by their language) and not as people who are
speech/hearing deficient. Experientially, those born deaf can by definition
have no voice-in-the-head so familiar to the hearing in its role as monitor,
author, vehicle and evidence of consciousness; instead all these functions are 9. Memoir Upon the Formation of a Deaf
discharged gesturally: those who Sign have internal kinesis, ghost Variety of the Human Race (New Haven:
movements not ghost voices, sights not sounds in the head. This raises the National Academy of Science, 1883)
intriguing possibility of a parallelist self, an internally 'seen/felt' creation of a
psyche different in certain phenomenological and cognitive directions, from
the internally 'heard/voiced' linear self-consciousness of the hearing familiar
to most of us. (A possibility that offers a new, not easily accommodated slant
on the Lacanian thesis of the unconscious being structured like a language;
insofar as that thesis assumes 'language' is speech). Finally, Sign -- a gestural
alternative to speech whose structure, genesis and practices are independent
of spoken languages and which unlike them such has no accepted written
form -- poses an interesting roadblock to Derrida's totalization of 'writing',
his conception of archewriting. For insofar as archewriting includes gesture,
as much of his talk of spatializing would indicate, then it suffers the same
metaphysical iniquities he attributes to speech, since Sign -- for the point in
question-- is speech.Put differently, ASL separates (in fact opposes) the
effect of presence-to-oneself from phonocentrism, whereas of course they
are conjoined and theoretically inseparable in Derrida's oeuvre.
users.wowway.com/~brian_rotman/gesture.html 9/26
23/05/13 gesture
Emblems, then, are social, experiential and interpersonal; put there to make
something happen, to impinge on the behaviour of the self and others;
they're not really interested in making statements, asking questions, or
conveying facts and propositions. They are in short performatives and,
according to McNeill, discharge the functions carried out by speech acts.
But unlike speech they do not combine via a syntax as part of a language.
And they differ from words in that their meanings are neither explicitly
defined nor (outside instruction in rhetoric or acting) are they intentionally
learned or studied, but rather picked up, absorbed, and (perhaps for this
reason) remain stable in form and import over long periods of time despite
linguistic changes in the communities of their users. These features suggest
that emblems might operate according to a different dynamic and logic, and
might accomplish different ends from that of speech. Calling them 'speech'
acts, assumes them to be individually interchangeable with or translatable
into speech, as well as assuming that they operate, as a system, in the same
ways and for the same purposes as speech. But is this so? Are emblems
always translatable into spoken language? What, for example, is the speech
equivalent of a wink? Or, for that matter, a shrug? a slap on the back?
folding one's arms? hands clasped in prayer? And does their mode of
operation resemble speech? If so, why do we bother with them?
The great range, robustness, and persistent use of emblems, their way of
refusing and displacing speech, calls for an explanation. McNeill offers one
in relation to the concept of 'word magic'. "Spoken words are special and
carry with them the responsibility for being articulated. However, conveying
users.wowway.com/~brian_rotman/gesture.html 10/26
23/05/13 gesture
the same meaning in gesture form avoids the articulatory act and, thanks to
word magic, this lessened responsibility for speaking transfers to the speech
act itself." (65) Doubtless, there is truth in the idea that gesturing rather than
talking removes one from the net of justifications, arguments, questions,
deceptions, and interpretive qualifications and recriminations that speech
immediately introduces. But how many gestures admit of a translation into
speech? What, absent any such reified abstraction does "having the 'same'
meaning" mean? How convincingly can speech render an emblem? Giving
the finger, for example, certainly carries a different charge from saying "up
yours" or "fuck you" or the very different anatomical procedure invoked by
"go screw/fuck yourself", and so on (the plurality of inequivalent
verbalizations suggests emblems generate meanings by their very exclusion
of speech). But, in any event, is the difference between gesturing and
voicing the 'same' meanings one of lessened responsibility?
users.wowway.com/~brian_rotman/gesture.html 11/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 12/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 13/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 14/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 15/26
23/05/13 gesture
they pre-interpret speech by being part of the sound which constitutes it, see note 4 of my �Exuberant Materiality: De-
determining thereby how it is to be taken, the spin or angle its words carry, Minding the Store�, Configurations 2, 257-
the degree of belief, seriousness, irony, kind of intention, agenda or 274, 1994
motivation they have, and so on. Descriptive linguistics parses the division
between the syntactic-phonemic and the prosodic as the difference between
what is said and how what is said is to be taken (15); a difference that, in
terms of the contribution of prosody to speech's social, affective and
interactional meanings, is crucial: in the absence of tone, for example, the
differences between speech being gentle, withering, questioning, 13. See the various works cited on pp. 8 - 11 of
threatening, seductive, flattering, menacing, pleading, sardonic, gleeful, sad, Gesture and the Nature of Language
encouraging, and countless other affects mediated by connections to our
limbic brains, disappear. A disappearance that as we shall see is central to
the metaphysical work accomplished by alphabetic writing.
Writing speech
Though alphabetic writing is not confined to the writing down of speech, it
is in this that writing's horizon, its nemesis even, is to be found, since it is the 14. The Symbolic Species: the Co-evolution of
body, through its gestures, that haunts writing. Confronted by the semiotic
Language and the Brain (New York: Norton,
effects of the body -- the prosodic gestures lodged intimately inside speech,
1997), 359.
the gesticulatory ones tightly accompanying it, the emblem gestures that
displace speech whose functions range from explicit signifying to the
creation of implicit silences -- writing offers notations, inscriptions, traces, an
organized system of marks constituting an algorithm; a code which the
reader of these marks must decode in order to retrieve or reconstruct the
original body effects.
which the bulk of prosodic effects occur, and for which there are (outside
musical notation) no accepted devices. This should be qualified: it is not true
to say there are no devices for writing to convey prosodic effects. In the
interests of brevity and directness, I am talking about writing insofar as it is
'printed', 'naked', and 'pure': printed, and hence omitting the ways
handwriting can make certain prosodically mediated meanings and effects
available; naked, in that the text has not been priorly processed and
annotated up by one of the various mark-up languages or style sheets now
being developed precisely in order to facilitate the reading of texts aloud by
machines (though their ability to code prosodic effects is at present very
limited); pure, in that the text is printed in a 'neutral' font, thus omitting the
effects, most of which are prosodic, achievable through the use of different
type-faces; a resource powerfully used in advertising and comic art. But
these restrictions and exceptions do not disable the general point here,
namely, that alphabetic writing deals with prosody -- insofar as it does --
through the addition of words, through description that leads to textual
augmentation. Faced with prosodic effects, writing is obliged to deal with
the information supplied by prosody by introducing more words --
descriptions, parses, glosses, amplifying locutions -- that function like stage
directions.Thus, in addition to alphabetically notating (necessarily de-
prosodized) words, writing has to supply a further coding of the missing
gestures -- the manner of the words' saying -- by the addition of words about
these words detailing how they are to be taken. Each time this happens, each
time prosody is projected onto syntax, the text becomes longer, more wordy,
more open to further augmenation, than the speech it is coding, since what
was before a co-occurrence, a simultaneity of words-with-prosody, becomes
sequential and linearized.(16) And so it goes on, text without end; always
the edge of writing is occupied by an incomplete coding of prosody and
always the recovery of some full -- prosodically complete -- meaning, affect,
significance or purpose is deferred; a potential infinity of deferrence. In
theory. In practice, one stops at some point of sufficient prosodic
recuperation long before death by excessive length and unreadability
intervene. Derrida's message of endless deferral of meaning (which, by his
practice, is confined to alphabetic writing and which, by his insistence, is
necessarily indifferent philosophically to any such extra-textual intervention
or external source of closure) is thus an effect of alphabetic writing, built into
users.wowway.com/~brian_rotman/gesture.html 17/26
23/05/13 gesture
the alphabet's asymptotic relation to the capture of prosody, an inevitable 15. I take this formulation from David Olson's
consequence of its inability to code the gestures inside de-gestured words The World on Paper (Cambridge: Cambridge
except by the addition of more (de-gestured) words. University Press, 1994) where it is explicated in
terms of the theory of speech acts put forward
This failure, or rather the disjunction of words from their prosody that by J. L. Austin, according to which, what is
prompts it, which allowed a separation of 'speech' (i.e. speech virtualized as communicated by prosody is the "illocutionary
alphabetic inscription) from its embodied production, was of course writing's force" of an utterance. Much of this valuable,
spectacular achievement. It allowed the alphabet to cut the word loose from but I do not use that description here since I
the place, time, context, circumstances, voice, gestures, presence, and don't think the term 'communication'
mortality of the one who utters it. But this separation has a neurological (understood as a process external to what is
correlate, namely, that such virtualized speech issues from -- gives rise to, communicated ) underlying the theory of
allows to come into being, gives autonomy to -- a dis-embodied cortex; for it speech acts is ultimately a helpful way to think
is precisely the cortex's connection to the midbrain and so-called limbic about the relation between speech and gesture
regions that produces the words-plus-prosody amalgam; a loop which the
autonomous presence of the de-prosodized alphabetically written word cuts.
More sharply, one can locate the connection from the limbic to the cortical
brain as routed through the prefrontal lobes and understand its severance as
an orthographic equivalent of a pre-frontal lobotomy: certainly, descriptions
of the speech of recipients of this procedure -- "In their words or attitudes,
no traces of affection could be detected." -- resonate in an uncanny way with
how we perceive words emptied of prosody. (17) Taken in the reverse
direction, the correspondence allows this writing -- in its relation to speech --
to enact a transcendental escape from the body: the de-prosodized word
issuing from a dis-embodied cortex forming one side of an equation the
other side of which is a marginalized/occluded body whose horizon, its
'highest' achievement, is that of mere limbic gesturing. (18) Alphabetic
writing thus contributes to the ontology of an abstract, invisible God, not by
inventing it as an object of thought, but by operating as a vehicle or machine
for the dissemination of an invisible, transcendental difference without
which no such God can be understood to exist. The alphabet in other words
furnishes the presence of a divine absence, not by representing or alluding to
it as subject matter (though it does that endlessly) or invoking it within itself
(though it does that too: "... The word was with God, the word was God"),
but through direct performance: constantly transcending the midbrain which,
in the terms of the metaphor here, functions as nothing less than the neo-
cortex's body.(19)
users.wowway.com/~brian_rotman/gesture.html 18/26
23/05/13 gesture
Writing gestures
Alphabetic writing's relation to prosody is symptomatic of a general problem
inherent in notating a performed action -- spoken word, musical phrase,
dance step, body movement, gesture -- with a predetermined signifier of
some system, since the system projects its own shape onto its subject-matter.
In other words, the action notated never escapes from a mirroring of the
separation into discrete, quasi-algebraic signifiers or 'notes' notating it.
Certainly, within the algebra of alphabetic inscription, the effects of prosody
-- geometric, topological, continuous, spread over a phrase or clause --
cannot, as we've seen, be represented as characteristics of separate words.
The attempt to recoup these effects producing, instead, the familiar
proliferation of ever-different interpretations, readings, and deferred
meanings set in train by a recuperation with no pre-assignable boundary.
Outside the notation of speech, the effects are more various. There is a
plethora of different systems that have been invented (each with its own
adherents, successful transcriptions and limitations) for notating dance -- a
multiplicity that in itself suggests certain intrinsic difficulties at work -- but
no single one works for all dances in the way that the alphabet does for all
spoken languages. Likewise, efforts to notate gesture run up against limits to
the feasability of taxonomizing and writing down the body's movements that
seem to reflect an intrinsic limitation of 'writing', of the medium of paper
itself, rather than any failure of method or imagination; and explanations for
the absence of an accepted writing system for ASL do not challenge this.
(20)
But writing's success was to sever (the words of) speech from its production,
to create a form of virtual speech. And speech, as we've seen, is made from
gestures; if writing could virtualize the products of these gestures, might not
something similar be possible on the visual/kinetic products of gesture in
general? What if the whole operation could in some way be repeated; if
three millennia after spoken words were virtualized, it became possible to do
something akin for other signifying productions of the body? What if
gestures could be brought into consciousness, made into discrete objects of
awareness, examined; made as identifiable, repeatable, portable, studiable, as
free-standing, iterable and quotable as words? Would it possible, is my
users.wowway.com/~brian_rotman/gesture.html 19/26
23/05/13 gesture
question, for there to be a medium that did for gestures what writing marks
on paper did for the words of speech?
Notating speech rescues it from oblivion, captures it in a form that allows the
original utterance to be (partially) reconstructed. Contemporary digital
technology offers a technique -- appropriately named motion capture -- that
promises to do precisely this for gesture and indeed any kind of physical
movement, meaningful of otherwise. Before describing it, it as well to place
it in the context of other and older meanings evoked by the idea of capturing
gesture and movement; meanings corresponding to different interpretations
of the term 'to capture', namely to mimic, to transduce, to record, to
incorporate, and to sample.
Mimicry is the most primitive, least mediated form of gesture capture; as old
as acting and dancing, mime uses the performer's body itself as the
instrument for capturing the movement of another body. Transduction
corresponds to the instrumental mode of gesture identified earlier, in which a
human movement is captured within a machinic circuit where it is converted
into other movements or regimes of action; winding up a clockwork
mechanism or turning the handle of a pulley wheel are traditional examples. 16. See Ad Infinitum for more on this in
Recording movement is what film and video are normally credited with relation to the potential infinity built into the
achieving; they capture movement (in relation to photography's freezing of definition of a Turing machine as its
it) not by way of a direct connection to its kinesis or production as a physical specification of an infinite tape. Without this,
process, but as an appearence, in the highly mediated form of a two- Turing machines would not be universal; not
dimensional visual representation. Incorporating movement occurs in able to reproduce the input/output workings of
calligraphy or various kinds of art production, for example, playing a violin any purely mechanical device -- since, like the
or guitar, the hand gestures responsible for a line drawing, or the brushstokes doomed attempt of alphabetic writing to
that constitute a painting, or their digital versions. In such cases, what is incorporate prosodic effects, the machine needs
captured enters directly and significantly into the created object. James unlimited (i.e. unbounded) sequential space to
Elkins talks of oil painting as making a cast of a painter's movements, of incorporate effects not given originally in
functioning to "preserve the memory of the tired bodies that made them, the sequential form.
quick jabs, the exhausted truces, the careful nourishing gestures." (x) What
Painting Is, p.5. Finally, sampling movement is what takes place within the
digital technology of motion capture. One attaches sensors (these can be
responsive to visual, magnetic, aural, or inertial tracking technologies) to
chosen points on the body (of an animal, machine, human) and takes
periodic readings, i.e. digitized samples, of where in space these sensors are
users.wowway.com/~brian_rotman/gesture.html 20/26
23/05/13 gesture
as the body moves. The resulting data-set contains the information needed to
reproduce the original motion of the chosen aspects of the body in a
potentially unlimited series of contexts. Unlike a film or video, the readings
are raw data not a representation, they effect a de-territorialization of the
original motion, cutting it free from the place, time, context, circumstances,
physical form, and presence of its performance; in which form it can be re-
territorialized able, for example, to drive an animation -- become the motion 17. Julio Rocha do Amaral and Jorge Martins
of an automaton, puppet, robot, cartoon figure, a virtual reality avatar. de Oliveira Limbic System: The Center of
Captured gestures are already being used in art objects, computer games, Emotions.
virtual choreography, animated films, chat rooms and virtual places, different
kinds of electronic installations, and various attempts at virtual theatre. Like
any digitized object, captured gestures can be stored, instantly replicated,
18. Mathematical ideograms are a form of
posted on the internet, and multiply processed. Capture technology, then,
writing that constitute the most extreme
offers gestures the same kinds of mobility, dislocation and freedom from the development of cortical autonomy. The fact that
contexts of their production as the notational system of alphabetic writing mathematical languages exclude gesture, are
allowed speech. devoid of any reference to an I, or to an other
Observe, from what has been said, that the question here is no longer one of or to a listener, addressee, or signee, and are at
marks on paper, of notation understood as inscription readable by the the same time the ur-site of an abstract,
unaided eye. Digital technology has already changed the terms here, transcendental ontology -- platonism -- should
not, therefore, be surprising; at least not from a
building into 'notation' the digital means by which what is notated can be
semiotic point of view.
read and produced; notation, in other words, embraces simulation. In light of
this, one can juxtapose motion capture and notation. The alphabet notates
sounds on paper, digital technology captures movement as stored data;
notation is algorithmic (the letter sequence a-l-p-h-a-b-e-t is an instruction to 19. For a different neurological take on the
a reader for reproducing a certain sound) and is limited by the pre-set deleterious effects of alphabetic writing within
identifications built into its symbols; capture is documentary and is limited Western culture in terms of its left-brained
by sensor resolution and sampling rate. Notation is metaphoric and requires exclusion/repression of the right-brained,
interpretation, capture is metonymic and delivers fidelity. In capture the life feminine-biased visual image, see Leonard
of a movement, in our case the significant dynamic of a gesture, is sealed on Shlain The Alphabet v The Goddess (London:
the inside at creation; in notation the life is applied, with all the ambiguity, Allen Lane, 1999).
creative augmentation and loss this entails, to the skeleton by the interpreter.
bringing (a newly digitized and objectified) gesture out from under the
shadow of the spoken word install a new order of body signification?
Without claiming (which would be absurd) that gesture could rival speech, it
is undeniable that it is on the ascendancy, that in some sense (which I've
tried to outline), certain kinds of silence and the saying of nothing,
achievements of a newly valorized but once marginalized and despised
body, are poised to come into prominence, have already arrived.
To say more about saying nothing, is to leap out of the semiotic mode
governing the present account and consider gesture in its a-signifying
instrumental and experiential modes. Particularly, the experiential, the mode
of immersion, participation. Nowhere, within contemporary culture, is the
experiential gesture more vital, uncompromising and powerful than in the
planet-wide phenomenon of techno music.
collective and its numerous realizations -- crowd, herd, pack, group, swarm,
audience, mass, chorus, mob, throng -- articulated with such obsessive
comprehension and brilliance by Elias Canetti.(21) Techno combines the a-
signifying groove of music with dance, oceanic immersion and ecstatic
participation with a silence in the face of words that is deafening, effacing
logos, nullifying all attempts at reason, ideation, speech or language. Not to
be understood as music for listening, techno is digital software for moving,
dancing, gesturing, experiencing, for getting the body inside and outside the
groove of the digital machine. The rave is the contemporary site of
Dionysus. But that's another story.
users.wowway.com/~brian_rotman/gesture.html 23/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 24/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 25/26
23/05/13 gesture
users.wowway.com/~brian_rotman/gesture.html 26/26