Professional Documents
Culture Documents
Gestures: A Computational Drawing Machine
Gestures: A Computational Drawing Machine
Gestures: A Computational Drawing Machine
Lark Spartin
5 April 2020
Microsoft Xbox Kinect to track human movement, resulting in a computational drawing machine.
The project was inspired by a desire to explore movement as an art form, where the human body
functions as an expressive brush in a digital form. Gestures explores the intersections between the
physical and virtual body, with influences from live performance and early abstract animation,
attempting to bring the spectator into the artwork as a performer and creator. In Jim Campbell’s
‘Delusions of Dialogue: Control and Choice in Interactive Art’, he states that because the viewer’s
side of the interaction is not written, the viewer can “respond in a more open way” (133). This
shift of thinking of the computer not just as a recall system where a user enters commands, but
more as a collaborator in design process, is what helped to create Gestures, beginning by taking
Live performance and interactive media are both ephemeral in nature and exist mostly in
memory. David Saltz, in his paper, The Art of Interaction: Interactivity, Performativity, and
Computers, draws a connection between musicians, playwrights, and digital media artists, stating
“they do not directly produce tangible stimuli that any given audience will experience, but instead
produce what might loosely be described as a blueprint for performances” (118). Interactive media
can produce a unique experience, like a specific performance of a play or musical composition.
Spartin 2
However, instead of using a human performer to interpret the work, the viewer is controlling the
performance directly, meaning they gain the improvisation that comes along with live performance
but are given a deeper connection because they are creating the piece, versus watching it on a
stage. Gestures is similarly performative in the sense that participation and interaction become
what the piece consists of. In Studio Azzurro’s manifesto, Confidential Report on an Interactive
Experience by Paolo Rosa, it is stated viewers are participating in more than just a predefined
narrative, but can “explore and realize in time, in space, in matter, the potential of a work/event”
(6). In Live Media: Interactive Technology and Theatre, Davis Saltz speaks about the character of
Ariel in a contemporary version of William Shakespeare’s The Tempest, where the actress had
sensors attached to her body, including her head, waist and arms. These sensors transmitted
information about her movements to a computer and projected animations of her. Voice
recognition software was also used to sync the animation’s lips to the actress’s voice. Other
examples of technology integrated with theatre include Nayatt School by The Wooster Group in
1978, which incorporated film into a live performance and played back a previous version of the
work, and To You, The Birdie! (Phedre), a stage play featuring Willem Dafoe as Theseus. Theseus
is live on stage, and simultaneously, there is a pre-recorded video of him that is shown through a
monitor. This splitting between stage and screen time and playing with the idea of being physically
present somewhere helped me build a foundation for playing with a physical and virtual version
of the body. Learning about how performance intersects with interactive media and understanding
how a physical body could move within the space was an important precursor for the abstracted
After researching the similarities between performance and interactive art, I became
interested in the varieties of ways that performance could be depicted. It was clear to me that
Spartin 3
computers have the potential to do more than regurgitate raw data and can interpret input of many
different data types, so the next logical step was to learn how to display that input. Dichotomies
exist between the languages of expressive and gestural painting and drawing, and the world of
computational technology and discrete logic. Digital media is usually not thought to be as
expressive as physical media. However, Scott Snibbe and Golan Levin, in their article, Interactive
Dynamic Abstraction, contemplate possible limits of physical media, stating there “is one essential
element missing from these static paintings that is intrinsic to music: time” (2). Although I was not
necessarily exploring links between visual art and music, I noted that interactive technology is also
time-based media. Gestures was an important experiment to see if a greater sense of intrigue could
be expressed by displaying human movement over time. Furthermore, I wanted to emulate the
expressivity of physical media in the visual display on screen. Snibbe and Levin also speak on this
point of infusing computer programs with temporal painting expressivity, saying they wanted to
“design visual instruments that could possess these qualities of simplicity, intuitiveness and
possibility” (6). They began by expl`oring pioneers like Pythagoras and Kandinsky. Pythagoras
looked at similarities of tone and light, and explored ‘perceptual phenomena’ and aesthetics.
Kandinsky used his synesthesia to explore links between sound and color. Snibbe and Levin had
a large focus on the basic principles of drawing, quoting Paul Klee’s ideas of ‘a line is a dot that
went for a walk’, and ‘drawing is taking a line for a walk.’ Abstracting these primitive shapes like
points, lines, ellipses and rectangles was a good first step in building Gestures. Motion Sketch, a
digital animation that uses mouse and keyboard movements to display shapes and color on screen,
and Motion Phone, a program that allowed many machines to collaborate through network
connection to draw on one canvas, were early experiments by Snibbe and Levin. The authors did
concede however that at the time, “most computer programs were focused on the mathematical
Spartin 4
processes rather than the aesthetic (4), and they attempted to fix this in numerous ways. They used
pre-Cartesian geometry, applying direction, inertia, velocity, orientation, curvature, and coded
particle systems to emulate nature in computer graphics. They would also utilize processes like
taking the numerical derivative of successive screen positions, and used squash and stretch
animation techniques to distort shapes in the direction of motion. Using a program as an instrument
to inform movement, rather than just a machine to carry out simple tasks, helped me to view
computation in a new way. Snibbe and Levin finally took the approach of creating
showcase the subjective nature of experience, the knowing of oneself, and how perception is linked
to consciousness. These studies branch off of phenomenology, a field of study by Edmund Husserl.
Perception, as well as style and personality of an individual is so varied, and I was interested in
how something as intangible as perception or emotion could be linked to interactive media, and
Abstract animation is another early field that was important in my study. Len Lye drew
and scratched on celluloid film to create the 1958 animation Free Radicals, which imitated the
structure of music, making his process of creation as telling to his inner thoughts as the product
itself (Snibbe 2). However, early abstract animations had no live performer, making diverse
expressions difficult. The color organ was invented, a device that incorporated light with a live
performer. The first color organ, The Ocular Harpsichord, was made by Father Loise-Bertrand
Castel in 1734. A harpsichord was placed behind colored glass windows, using candles as its light
source. Pulleys from the keys of the keyboard created small panes of color as a musical
composition was played. In the 20th century, technology enhanced the pure light performances, as
well as using mirrors, lenses, and colored filters for projection. Thomas Wilfred made his version
Spartin 5
of a color organ, Clavilux, in 1930, and Oskar Fischinger, one of the first to create video that
accompanied music, also made the Lumigraph in 1952, consisting of a cloth sheet that intersected
beams of light controlled by foot pedals. These color organs were functioning equipment that
carried out a specific task, all while still mastering an ethereal look. This pairing of technological
capability with the potential to still show character in an abstract output, was exactly the challenge
I was willing to explore in Gestures. I planned to provide a natural quality in the installation by
employing the math and physics available in computer code. I decided to use Xbox Kinect as my
computer vision input, and knew there were many platforms I could utilize in tandem to abstract
this data. Usually, digital representations leave little room for the intuitive abstraction I have
spoken about thus far, so it was important to find a coding platform that had a focus on using code
in a poetic fashion. Zachary Lieberman, a founder of creative C++ library openFrameworks and
collaborator with Golan Levin, speaks about computational poetics in his talk, What could the
Creative Career of the Future Look Like?, at Adobe’s 99U Conference. Lieberman found his
beginnings in printmaking and painting, and soon after started to use code as a visual language to
evoke emotion and feeling, using code to write his version of poems. Processing was my platform
of choice because it was created to use code within the context of visual arts, it has very descriptive
documentation, and uses Java syntax, making it easy to learn. The fact that coding languages have
shifted from linguistic encoding with no display like TEI encoding, to languages created for visual
art like Processing or Max MSP, invites users to rethink the meaning of code.
Golan Levin’s manifesto, Computer Vision for Artists and Designers: Pedagogic Tools
and Techniques for Novice Programmers, reflected on other projects that incorporated a more
technical introspection into computer vision, such as Myron Krueger's Videoplace (1965-1975).
According to Levin, Krueger believed “that the entire human body ought to have a role in our
Spartin 6
interactions with computers” (2). Videoplace digitized participants figures, analyzing their shape
and motions, and in turn, drew animations on screen related to their actions. Users could also
choose to paint with their fingers or bodies. Messa di Voce is another example, created by Levin
and Lieberman. It was a projection-based, augmented reality project combined with speech
analysis, where a vision algorithm tracks the performers heads, analyzes audio coming from the
microphone, and displays visualizations in reaction to what is being heard and seen.
Gestures also took influences from Camille Utterback’s piece Entangled, where she
projected a variety of human controlled brush textures onto three suspended scrims. Participants
can interact with the work on both sides of the scrims. Utterback states that Entangled “reminds
us of our embodied relationships, requiring participants to literally view each other through a
screen” (1). Although Gestures is not doubled-sided, Entangled did provide ideas of turning
computer input into something physical, visual and artistic, and bring viewers back into connecting
with the digital medium: a medium that has been characterized as being somewhat disconnecting,
separating relationships and removing people from the environment around them. Gestures was
an experiment in allowing viewers to be present with themselves, using their own movement to
tangibly trying to express Gestures. The three techniques I found to be most useful in creating my
program were frame differencing, which checks the difference between two frames to detect
change in movement, background subtraction, which calculates the difference of pixels according
to their difference from a known background scene; and brightness thresholding, which uses
differences in luminosity between foreground and background. From here, I began to explore
computer vision more specifically within the context of Processing documentation, starting with
Spartin 7
simple object tracking, and moving into representing point clouds, raw depth data and the depth
image, color tracking, average point hand tracking, closest or highest point tracking, and
thresholds.
I started by displaying raw depth data and testing the data against a condition to see if it
fell within a certain distance threshold. If movement fell within this threshold, the Kinect would
pick up movement. A threshold was used to distinguish human movement from background
movement or noise. Within that threshold, the program calculates the highest and lowest point and
tracks those, updating with every frame. Another conditional loop checked if movement occurred
within a minimum threshold number, or in other words, a pixel was recognized as being the closest
point to the Kinect, the total number of pixels on screen are counted, as well as a sum of the X-
axis and Y-axis pixels. The sum of the X-axis and Y-axis pixels is then divided by the total, in
order to get an average center point that tracks the closest point on the screen. From these points
of movement, particle systems were displayed, consisting of different shape properties. The color
of each of the particle pixels was mapped to the color depth data. The color at that X/Y position
was sampled and would change as the participant moved across the screen, depending on the
distance. There were four particle systems displayed, each with different characteristics. I used
Object Oriented Programming to create particle constructors and used the principles of
Polymorphism and Inheritance to extend different attributes to each constructor. Using the Super()
function allowed me to extend the main characteristics of the original Particle class I created, and
create unique features for following particles, so I could create different shapes on screen. This
was more efficient, making it so I didn’t have to create each Particle class from scratch. Each
particle used the same Particle System class functionality. The Particle System class stays the same
for each different particle, and keeps track of the particles lifespan, removing it after a certain
Spartin 8
amount of time. Each particle is updated with an acceleration, velocity, and position, to give the
effect of streaming down the screen, and to play off what I learned from Snibbe and Levin about
emulating real forces into code to give it the feel of a temporal painting. In another iteration of this
project, I would be interested in exploring Blob Detection to obtain regions of motion, such as
both hands, the head, and maybe the middle of the torso. Using this would be a more seamless way
to detect specific regions of activity, and give better recognition of these areas, as you can create
code to detect areas of a certain color or distance. I would have also liked to explore the Skeleton
video data to try to accomplish this and map the different particle shapes to specific joints.
Reflecting on the installation process, testing of equipment and experimentation with materials
before setting up and in real time became essential, and I will carry these lessons into my next
installations. Another further iteration of Gestures could include audio, to create an interactive
visual music piece using movement to trigger sound in tandem with visuals.
The physical portion of Gestures was just as important in bringing together the concepts
of expression spoken about so far. As humans, we are intrinsically linked to space, allowing it to
inform feeling and memory. I wanted to free the computer-generated image I created from the
constraints of a computer monitor. I was trying to create textural materiality in the code itself, so
the physical installation needed to serve as an extension of this. I used tulle to help create an
ephemeral, invisible effect, which helps to represent an intertwining of the physical and virtual
body and experience. Toni Dove, author of Theater without Actors-Immersion and Response in
Installation, talks about scrims being used to break out of the stereotypical screen mould, saying
“projection-scrim sculpture creates the illusion of 3D characters from slides and video…projecting
moving animations onto structured scrims to create a holograph-like illusion” (4). Dove says the
viewer can become more of a participant this way, as compared to film, where a passive viewer
Spartin 9
just looks at flat space. The tulle used in Gestures helps give the movement a reverse holographic
effect. Paolo Rosa in Studio Azzurro’s Confidential Report on an Interactive Experience agrees
that using the right materials in installation is important when they say that projecting images onto
heavily textured materials “exalts the materiality which is so characteristic of the art of painting;
the warmth of this medium and the pleasure associated with a technique that has played such a
large role in the history of art is in some way recovered (8)”. Being able to emulate characteristics
of painting was important when looking at the structure of the ‘paintbrush’ in Gestures, but also,
the canvas. Rosa also meditated on the importance of natural interface, talking about how
relationships between the computer and participant are very tied to technological items such as the
mouse and keyboard and it’s better “if our sensitive environments are without even so much as the
shadow of an electric wire. This allows us to get a clearer view, not of technology, but of its effects,
and allows for a more effective relationship between the immaterial world of images and sounds
and the material world of the objects and spaces with which the work is completed” (7). Being
able to touch something physically and interact in three-dimensional space helps involve multiple
senses and brings technology into reality. In creating Gestures, I wanted to incorporate an element
of play, helping others to learn through an imaginary adventure, and encouraging freedom and
exploration by minimizing technology and wires, and maximizing being present with the work
(Azzurro 9). Rosa explains that “we are at the end of the era initiated by Duchamp and his
readymade works”, and no longer will everyday objects be taken out of the real world and put in
art galleries, but actually the opposite is happening (11). Gestures brings the work out of the art
Interactive media helps to build a symbolic system, that encourages a new point of view
from which to “observe reality…that [participants] have helped to change” (Rosa, 11). Toni Dove
Spartin 10
describes interactive media narrative as “not bound by the traditional ideas of plot or story, but
nonetheless has an engine that moves it through time” (2). Gestures is a multi-layered project,
taking cues from live performance, expressive painting and abstract animation, that helped me see
how perspective and human engagement can be used as an instrument. It’s interactive medium
attempted to infuse expression into a computational interface, to give its viewers the role of
creators, and to allow them to be able to inform design choices themselves while exploring and
creating in real time. This immersive installation functioned as a learning opportunity and
experiment, trying to create affect in its spectators and give them an experience that they could not
otherwise get through traditional art. The manifestation of code and physical installation in
Gestures is proof that interactive digital media can be immersive, and is an expressive realm that
Works Cited
Campbell, Jim. “Delusions of Dialogue: Control and Choice in Interactive Art.” Leonardo, vol.
2020.
Saltz, David Z. “The Art of Interaction: Interactivity, Performativity, and Computers.” The Journal
of Aesthetics and Art Criticism, vol. 55, no. 2, 1997, pp. 117–127. JSTOR,
Kaye, Nick. “Paolo Rose (Studio Azzurro), Confidential Report on an Interactive Experience.”
Routledge, 2007.
Saltz, David Z. “Live Media: Interactive Technology and Theatre.” Theatre Topics, vol. 11, no. 2,
Snibbe, Scott Sona, and Golan Levin. “Interactive Dynamic Abstraction.” Proceedings of the First
2000, doi:10.1145/340916.340919.
Behance, Inc. “Zach Lieberman: What Could the Creative Career of the Future Look Like?” Adobe
creative-career-of-the-future-look-like.
Levin, Golan. “Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for
Novice Programmers.” Ai & Society, vol. 20, no. 4, 2006, pp. 462-482.
Dove, Toni. “Theater without Actors: Immersion and Response in Installation.” Leonardo, vol.
2020.
“Free Radicals, 1958 (Revised 1979).” The Len Lye Foundation - Free Radicals, 1958 (Revised
1979), www.lenlyefoundation.com/films/free-radicals/33/.
“Getting Started with Kinect and Processing.” Getting Started with Kinect and Processing |
Klees Pedagogical Sketchbook.” International Journal of Art & Design Education, vol.