Members 

Work AGI Open Publications Search Members area (/members)

(http://a­g­ (http://a­g­ (http://a­g­
i.org/member­ i.org/agi­open) i.org/publications)
work) News (http://a­g­ AGI History
i.org/news) (http://a­g­
i.org/agi­history)
Flickr (http://a­g­ Current (http://a­ Archive (http://a­
i.org/flickr) g­i.org/current) g­i.org/archive)
Contact (http://a­
g­i.org/contact)

Current
Vortex. Visual Aural Textual. One Language.
Posted: 02 April 2011

Vortex. Visual Aural Textual. One Language.Robert Appleton interview by Majid Abbasi
(published in Farsi, in Neshan Magazine, Tehran)"Sound text and image integrate together no
matter what random order they're placed in. Our minds synchronize the experience. We invent the
meaning. And this forms the basis of human interaction."

M_ What is 'Vortex' - as it relates to the title of your recent performance?

R_ Vortex is my development of a single language of the visual, aural and textual. In our
interview today for example, the camera you are using captures the sounds, we speak to each
other in three languages - the visual, aural and textual. The full title of my work is 'Vortex.
Visual Aural Textual. One Language.' The term Vortex is represented by the sounds which form
it in English: V (for visual) + or (for aural) + tex (for textual) = Vortex. It also exists as an 80
page thesis which earned my Masters degree in Interdisciplinary Art, Media and Design at
Ontario College of Art and Design University in Toronto last October and as the outline of my
book which will have a foreword by Noam Chomsky - probably our greatest living linguist.

Music Gallery Toronto, 2010 with Bob Brough saxophone and Pieter Coussement, programming and electronics

M_ Why "one language"? Where does it come from? What is the background?

R_ Perhaps the oldest form in history to express the singularity of sound text and image is the
ancient OM chant - which is at once a word, a picture and a sound. For centuries however, we have
preferred to develop three separate languages - the visual (image), the aural (sound) and the
textual (word), for which we have created individual aesthetics, vocabularies and histories as well
as thousands of dialects, alphabets and sounds. Language has become too closely associated with
only one vocabulary – the word. Changes in philosophy, society and technology have made new
communications concepts essential. Vortex is the result of my investigation into expanding our
idea of language. Improvising the visual, aural and textual simultaneously in real-time uncovers
new meanings not expressed by any one medium alone. When Gyorgy Kepes published his
Language of Vision in 1938, he said "As the eye is the agent of conveying all impressions to the
mind, the achieving of visual communication requires a fundamental knowledge of the means of
visual expression. Development of this knowledge will generate a genuine 'language of the eye,'
whose 'sentences' are created images and whose elements are the basic signs, line, plane, halftone
gradation, colour, etc." (Gyorgy Kepes, 1938 in Bauhaus, Hans M. Wingler, MIT Press, 1969, p
197)And 50 years later in 1989, Armin Hoffman wrote "Among semioticians, information
theorists and media experts it has been commonly held for some years that reality can no longer
be captured and described with linguistic means alone. It is changing too quickly and growing too
complex. Clearly language as our most important medium of communication has reached
animpasse... A language of pictures, drawings, diagrams and photographs is in the process of
supplanting language, or at least extending and enriching its scope." (Notes on the Language of
Signs and Symbols, Birkhauser)With Vortex, I have set out to create a single language of all
three components – words, pictures and sounds. I have written several earlier publications on the
subject: A Word is a Picture became a design class in which I did ethnographic research with
students in China, Turkey, the US and Canada. This was followed by A Picture is a Sound which Event 2, Text 1

became another class. About Time: A Theory of Design, Music and the Play Instinct was
published in 2009 (in Tempus Fugit, Index Books, p 11). And Vortex is the latest in the series.

M_ My feeling on your work when I was at your performance at The Music Gallery is that it is
closer to art than design? Is this correct?

R_ I practice design as an art form – in the way I think all good designers do – attempting to
reach its ultimate level, as the art of design. I've also been called an artist by some designers and
a designer by some artists. In New York City in the early 90's when the debate over whether
someone was a graphic designer, an artist or a commercial artist was at it's height, new media
people would say "I don't care what you call it as long as it's interesting". And now that I'm also
working with music and language, I feel that I've taken this full circle. It's not design, it's not art,
it's not music and it's not literature. It's all of this. And for me that's the measure of the work. It's
by not attaching to any one subject that new things begin to emerge. I see this project going
beyond my personal work – it becomes public, a new medium if you will, which many can
contribute to.

M_ You mean that it can be communicated to an audience?

R_ It can be communicated. In the Music Gallery, the audience was essential since they
interpreted and contributed to everything we did. I made this announcement at the beginning of
my piece "Tonight we will all make this event. Everything you do, will be projected into the
performance, influencing sound, image and text." When the audience applauded, their sound
made the images and text shimmer and the performance began. My Music Gallery videos are here
https://www.youtube.com/watch?v=zv1UeD1XVDs&list=PLF940F177A6E47826&index=1

Event 2, Screen 1
M_ This is the spirit of design. So it's a kind of visual communication, and it's between art and
design?

R_ Yes. And you know, the first piece that I played at the performance on Monday is influenced by
Leo Lionni's Little Blue and Little Yellow, about the relationship between two colors, yellow and
blue, which represent the mixing of cultures or the connecting of ethnicities. My piece (Event 1) is
dedicated to Leo. It's about a mixture of the three 'colors' of image, text and sound into a new way of
speaking. We became friends after I presented him for the AIGA/NY American Masters Series at
Cooper Union. An informal video interview I made of him is in Roger Remington's collection in the
Vignelli Center for Design Studies at RIT. My artistic references are very broad. They include:
designers Moholy-Nagy, Jan Tschichold, Gyorgy Kepes, Armin Hoffman, Karl Gerstner, Paul
Rand, Leo Lionni, Wolfgang Weingart and Muriel Cooper; artists Marinetti, Kandinsky, Klee,
Robert Irwin, James Turrell and Christian Marclay; musicians Arnold Schoenberg, John Cage,
George Russell, Thelonious Monk, John Coltrane, Miles Davis, Iannis Xenakis, David Tudor and
Robert Ashley; film makers Oskar Fischinger, Norman McLaren, Michael Snow, John Whitney and
Stanley Kubrick; philosophers Walter Benjamin, Jacques Lacan, Fredric Jameson, Slavoj Zizek,
Nicolas Bourriaud and linguists Ferdinand de Saussure and Noam Chomsky.

M_ You mentioned that your work is at the "intersection of improvised sound, text and image." How
does this happen?

R_ That's a broad question, so I'll try to answer it in two parts: Physically it happens in the
computer because I use the software Max/MSP/Jitter, which translates sound text and image into
mathematics. And the mathematics is what makes everything work together. So I physically create
the responses between sound text and image by speaking and playing my keyboards. And there's
also a response from other performers and the audience. Everything which takes place during a
performance influences the totality of the performance and what happens. The question is also
interesting as philosophy – what are the relationships between sound text and image? In Event 5, I
say "Sound, text and image integrate together no matter what random order they're placed in. Our
minds synchronize the experience..." Different meanings are created by the juxtaposition of sound
text and image during improvised performance. And with Vortex, invention in the moment using
three vocabularies can be made by one single person or by as many people as desired.

Event 2, Book 1, 2009

M_ So maybe it's important that I ask this question: How does information help art and design, and
what is its role?

R_ Information informs. I believe we have not yet identified all the information we possess.

M_ How important is technology in your work? And also in art & design generally?

R_ Technology is essential to my work and appears in most of what I do. When I produce a perfor-
mance, or a movie, then publish it on the internet or in print, it becomes mediated by technology,
whether it began life in that form or not. I still love analog media however – and I always will.
Digital music, design and art are relatively new, as photography and film were in the early 20th
century. Their significance is their currency, the new forms which are created with them, and Immersion diagram
the intermingling of media which use mathematics as a base (almost everything today). What has
also grown with technology is language. New vocabulary has been developed to describe the
effects of technology in art, design and music. 'Crossfading' is one example. In film and music it
means moving between or mixing two sources such as video tracks in a movie. In art and
design it can mean mixing two images, or two subjects, or two typographic treatments. 'Pitch
Control' allows us to adjust the speed of a recording while the pitch (or scale) can remain constant.
In Douglas Gordon's art piece 24 Hour Psycho, Hitchcock's film is pitch controlled – or slowed
down to run for 24 hours. In design typography can be 'Pitch Controlled' by adjusting its kerning
while maintaining its scale. Nicolas Bourriaud has written extensively of these influences
in art (Postproduction, Lukas & Sternberg, 2002).

M_ You mentioned an experiment in China when you and DouDou were improvising together.
What was the relationship between music made by this parakeet and your playing on a keyboard?
Were both of you doing your own thing?

R_ Just before arriving in Toronto I spent a year teaching at CAFA in China. I had an electric
piano in my apartment and was looking after our landlady's parakeet called DouDou. One morning
I was playing piano as usual and my wife was making breakfast. Suddenly I realized that
DouDou and I were improvising together – she was singing and chirping and we were following each
other musically as improvising musicians. It lasted for over an hour. Cathe set up a video camera
and made a recording. DouDouBob can be seen here https://www.youtube.com/watch?v=Tbzrh1-XI0o
Interspecies communication is very interesting. When making music with other humans, we can
practice our vocabulary and become better at it; we can compare this with other vocabularies and
decide how we might change it to make it better than it was. And we can present that. When
communicating in an unknown environment however –with a creature which is not using these
human traits – the rules are gone. There is no way to 'practice' playing with a bird, except by doing
it. The bird doesn't get better, and we don't get better. We just do or don't do. And what's interesting
in this situation is the coincidences that occur. And the subconscious communication between one
mind and another. One very small creature and one much larger, listening to and responding to one
another. Proof that language beyond words is very powerful (note: I made a presentation of this
work to the Product Design Team at the Lenovo Research Center in Beijing in 2007 and they were
also interested in the idea).

M_ You explained a little about the software program you used. Please tell me more.

R_ In the simplest terms Max/MSP/Jitter acts like a series of electronic diagrams on music which is
translated into mathematics by Midi. It operates like a switch which allows a course of action
and is turned on or off with something called a 'bang.' The operator can also modify this course of
action with additional 'switches' - controlling speed, scale, volume, etc., with infinite flexibility
(using any known scale of measurement). It was written by Miller Puquette at IRCAM in Paris,
which was founded by the composer Pierre Boulez. Puquette recreated an electronic music
environment in software. It's quite well designed and others have added functionality so that as
Max/MSP/Jitter, it now controls audio, video and image. And it can operate objects outside
the computer using wireless technology and sensors like Arduino.In my use of Max, we have tailored
the software to work with ideas about immersive language and space. We created a test-environ-
ment, a 360 degree cube in which almost anything can happen – and be measured as research.
Projection can take place on multi-surfaces – in front, above, below, to the side, in a state of
immersion. Some people make anatomically correct models of the human skull – on which they
perform measurements. This is an anatomically correct model of the human imagination –
for use in pure science, installation or live performance.
M_ How can you discover new meaning in vocabulary?

R_ We are used to viewing things through particular lenses. The lens through which we view liter-
ature for example is a literary lens. And actually there are many more things going on at the same
time as we are reading. In John Cage's idea that everything we hear is music – there is no silence and
we can simply be quiet to hear everything that is. What I'm saying is that within this same 'silence'
there's a lot more than sound. There are a lot of pieces of information we haven't yet identified, and
we may not be aware of them until they're pointed out to us. In my work, information and meaning
can come from anywhere within the perceivable vicinity of an event. It is not restricted to the body of
the event itself. For example, a sound that happens across the road, outside my vision and within my
hearing is included. A person not involved in an event, though moving through or past the event is
also included. There are multiple layers of information which we tune in and out of. All of these
layers have meaning.

M_ And also tell me please about the installation on the performance stage. So many things were
there – a balloon, a three-projector system, with only one screen, a computer, three keyboards, three Max/MSP software patch (above)
performers with no apparent conducting by you...

R_ At The Music Gallery there were three performers, myself on electronics and voice projected on to
3 screens, Bob Brough on saxophone and Pieter Coussement on electronics and vibrator with an
inflated balloon. I was conducting invisibly and composing in the moment. My computer controlled
the electronic imagery and my sound. Bob and Pieter were playing independently. I was composing
in the moment. They were composing in the moment. My conducting was created with the score http://
www.robertappleton.com/VortexPerformanceScore.pdf which everyone played from and then with all
the imagery sound and text we created collectively – performers and audience. The space was also
important –one in which the full range of human interactions could occur. My ideal performance
space has projection above, below, behind and to the side. All our senses are represented, and its as if
you're inside your own head.

M_ What is your feeling about new concepts in design? Or to put this more clearly, how is design
changing in the 21st century?

R_ Graphic Design (to be specific) is no longer limited to two or three dimensions. It's not limited to
printing or manufacturing. It's not just words and pictures. It's not restricted by anything other than
our imaginations and the willingness of society to accept it. The design subjects are becoming
broader for me, rather than more specialized. The principles of quality, creativity and communi-
cation remain, even as they are modified by individual artistic and philosophical points of view.
M_ Do you believe that we have reached the end of desktop publishing? And old media have been
replaced with new ones?

R_ Not the end, the beginning. My father-in-law reached the age of 90 last week and purchased a new
13" MacBook Pro. I'm very familiar with Macs so I helped him set it up – and I was shocked at how
unnecessarily difficult this has become. Hopefully, new computers will take into account our
different cognitive abilities. Then our lives will become infinitely better. We have just begun with
computing. New media and environments such as Vortex go beyond the 'desktop.' The best is yet to
come if we can embrace human need.

M_ How does information enhance art and design? What is its role? What differences are there in
design process between now and the past?

R_ Information is knowledge. It informs design as much as it informs art or any other subject. It's
role is to allow access to knowledge and to allow new knowledge to be developed. The difference
between process in the past and now is the change in tools and perception caused by new conditions
in contemporary life. The pace of change on the surface seems faster all the time. Yet it's only when
we stop, and study the present, that we begin to understand what has changed. My perceptions
change each time I enter the academic world or change direction in my career. Then I learn, and
new knowledge, passing through me, is returned to my work and its audience.

M_ If Vortex is a language, what is it's grammar? And what are the relationships between sound
text and image?
Performance ScoreMusic Gallery, 2010
R_ In English grammar, adjectives modify nouns. With a statement like "water is hot" 'water' is a
noun, and 'hot' is an adjective that modifies it. The adjective tells us what kind of water this is. In
Vortex grammar an adjective can be a visual, aural or textual segment (not necessarily a word)
with the power to modify whatever is around it. In Sketch for Performance (which can be viewed
here https://www.youtube.com/watch?v=GcxJ4Vk2kuk my animated typography from Fredric
Jameson's essay on postmodernism loops 3 times. The text is edited together with a randomly
chosen soundtrack from Mickey Mouse, Karnival Kid, 1929. The audio segment by chance is
precisely the same length as the entire animation. It is played once and each time the animation
repeats, its meaning is subtly changed by the sound. In the opening scene the title appears with a
fanfare (a comic though real suggestion of significance). Initial screens which present Jameson's
well-accepted argument play over an opening sigh followed by the hushing sound associated with the
announcement of important remarks in a speech or an opening at the Opera. Rising tension is
maintained by the sound of a needle crackling on a scratched LP or 78 disk. A brief explosion after
the appearance of 'premonitions of the future,' another hush at 'catastrophic or redemptive' and 'the
end of this or that' is followed by a dog bark suggesting happy agreement with the premise? A creaky
spring, a munching sound and a whistle introduce 'post,' followed by 'modernism' which fades to a
strumming guitar and the beginning of the love song Sweet Adeline which introduces the second
playing of the animation. Now the song's lyrics are converted into approximations of words as their
meaning becomes forgotten by the singers. When 'catastrophic or redemptive' appears for the second
time it begins with a whistle and cork-pop (a celebration?). The forgotten lyrics continue briefly
until at 'the end of this or that' we hear a snore and whistle. The final rendition of the text begins out
of sequence, as the music begins to speed to a rousing finish, and the animated image immediately
zooms out ultimately disappearing in reference to a shrinking spotlight – as the words 'post' and
'modernism' now a fraction of their former size – vanish forever and the piece ends. Through our
interpretation of Jameson's text about the state of postmodernism and the synchronicity of visual,
aural and textual matter, this piece demonstrates how much sound with image and text can change
meaning. By playing this game of improvising three languages as one
(Vortex), we discover new vocabulary, where three senses contribute equally. 1 1
http://www.britannica.com/EBchecked/topic/534831/human-sensory-reception

Recent workshops and performancesAGI BeijingGrafist IstanbulCAFA Beijing Lenovo

BeijingAGI IstanbulAXA BarcelonaFuturePlaces PortoMusic Gallery Toronto
 AGI Istanbul, 2009 with Yahya Dai, saxophone

AXA Barcelona, 2010 with JP Carrascal guitarand Sergi Felipe Fernandez saxophone

M_ Any final thoughts?

R_ Anyone with an interest in communicating with 'One Language' should feel free to
communicate with me. There have been several titles for this current work. In New York "A Word
is a Picture" was a course at Parsons School of Design and a small book. Then came "A Picture is
a Sound" and "A Word is a Picture is a Sound". This is my life's work. It is performance,

Design, image, concept © Robert Appleton 2009­Contact bob@robertappleton.com. All Rights reserved © AGI 2012