Annotation: Sketching On Sketches

M a k i n g a p l a n and sticking to it guarantees a sub-optimal solution.

-- Andrew Fitzgibbon

Imagine that you are a traditional designer working on your drawing board. The
concepts that you are developing catch the eye of a passing designer and this evolves
into an animated conversation. At some stage, your colleague wants to express an
idea that can be effectively communicated only by drawing on your work.
Alternatively, imagine that you want to augment a finished sketch with some
notes and/or graphic material that explain things like the colour scheme or some
other detail, yet you don't want to alter the original.
These are just two examples of situations where you may want to draw on or
otherwise mark up and annotate a rendering, but you don't want to change the
original. That way your work remains intact and can be viewed with or without the
annotations.
There are lots of other examples that crop up in the design process, but these two
should be sufficient to make it clear why this capability is worth discussing. With tra-
ditional media, the standard way that this is done is by placing a layer of velum over
the original, and making the additional marks on it. Since velum is much like the
tracing paper we used as children, the original drawing can still be seen, and what is
drawn or written on the velum appears as a layer on top of the underlying rendering.
An example of this is illustrated in Figure 64.
This is fine for traditional media, but what do you do if what you want to mark up
or annotate is not a static image on a piece of paper, but a video or animation? This
is a very relevant question given that the materials we use in interaction design are
often cinematic and involve time, for example. If annotation and layered mark-up
are important in traditional media, then the assumption should be that it will also be
so when sketching with other media as well.
One answer to the question of annotating dynamic media comes from the film
industry itself. It lies in h o w - - i n the predigital era--film editors used to indicate
when and where transitions such as dissolves, fades, and wipes were to occur in a
movie. To do so they would draw directly on the relevant portion of the film using a
grease pencil, commonly called a China Marker (Chandler 2004). This is illustrated
in Figure 65.
Although the editor's grease pencil marks were made directly on the film rather
than on a separate layer, the basic idea can be adapted to the digital domain in audio
and video editing, where one would draw over some form of timeline.

175
 lO000000000000OOO

// // // // //
)DDDDDDDD DDDDDDDDDDDDI I0000000000000000
~, ......
12 Frame Fade-in

0000000000000000 0000000000000000000000001

m~~~ //miNim
// // // iml
// i/
0000000000000000000000000000000000000000
000000000000000000000000000000000000000000000
ii
12 FromFade-out

O000000000000000000000000000000000000000~~~~~~~~~,~~~~~~j~~~,~~~~ll
0000000000000000000000000000000000000000,,
~_~0~0I
oooooooooooooooooooooooooooooooooooooooo oooooooooooooooooooooooooooooooooooooooooo0oo00ol

12 Frame Dissolve

Figure 65: Grease Pencil Annotation on

35 mm Film
With conventional film, one drew directly
on the film to indicate when and where ef-
fects such as dissolves, cross fades, etc.
were to occur. In the first two examples
the red marks indicate a twelve frame
fade-in from black and fade-out to black,
respectively. The third example indicates
a twelve frame dissolve, and the "C" in-
dicates the centre point between the two
shots where each is equally visible.
Images: Based on Chandler (2004) Ex-
cerpt from Cut by Cut: EditingYour Film or
Video provided courtesy of Michael Wiese
Productions, www.mwp.com.

Figure 66: Winky Dink and You

Drawing with grease pencils on acetate over
the TV screen in order to help the characters
in the episode solve problems.
Photo: CBS

176
 Annotation: Sketching On Sketches

There are other ways to think about drawing on top of video, such as drawing
directly on frames that are presented sequentially, like during video playback, rather
than spread out along a timeline. To clarify what I mean by this, compare the type of
annotation illustrated in Figure 65 with that seen in Figure 66.
The latter is drawn from a children's TV show called, Winky Dink and You. Pro-
duced by CBS, this 1953 program is probably the first example of free-hand drawing
on video. In this case, children drew with grease pencils on acetate overlays on the
TV screen in order to help the characters in the episode solve problems.
Of course what the children drew had no actual impact on what happened to
Winky Dink, except in their minds. But that is not the point. What matters from our
perspective is that the basic approach is valid and can be applied when implemented
using modern digital tools rather than a classic black-and-white television from 50
years ago.
Examples of this taken from both animation and videogame production can be
seen in Figure 67. There are two things in particular that I think are worth noticing
in these images. First, notice the ease with which you can distinguish between figure
and ground; that is, the foreground annotation layer and the background image that
is being annotated. Second, despite both images being still frames (albeit from a
dynamic animation), notice the way in which the use of line type and arrows convey
movement, energy, and the nature of the intended experience.
So far we have seen how annotation can be used on static and dynamic media.
However, in the examples discussed so far, the annotations themselves have been
static. They may indicate motion, but they themselves did not change over time.
There are examples from both practice and research that illustrate that the annota-
tion itself can be cinematic or dynamic.
We don't have to go very far to find an example. Simply listen to the director's
commentary while watching one of your favourite DVDs. That secondary voice
track is absolutely an example of annotation, and like the original use of velum over
a sketch, it is a separate layer (in this case audio) over the top of the original--and a
layer that can be turned on and off.
Likewise, most of us have seen a combination of voice and graphic annotation in
action while watching sportscasts on television. In this case, we have a commenta-
tor drawing over the video while giving a verbal analysis of some football or hockey
play.
The drawings may be crude, but that is not the point. Like sketching, it is the
timeliness of the commentary and its quality and relevance that is important. As it is
in sports and film making, so should it be in interaction design.

177
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
 9

Figure 68: Video Annotation

It is now c o m m o n to see sports broadcasts where the c o m m e n t a t o r draws

on top of the video to explain a play, or some matter of technique - as

illustrated here. One of the things that we need are simple and readily

available tools to do the same thing with the videos t h a t we use in design.
Source: Pat Morrow

180
 Annotation: Sketching On Sketches

My firm belief is that this kind of capability should be part of the standard rep-
ertoire of technology and techniques in any studio involved in interaction design,
for production, education, or research. Furthermore, the barrier to entry in terms
of using such tools must be very low. They must enable fluent and immediate com-
mentary by intermittent and regular users alike.
Yet, despite such systems being commercially available, this is not the case. One
of my primary motivations here is to change this.
I want to be clear that in making this point, I am not suggesting that one can go
out and just purchase the perfect tool for the job. Yes, some tools exist today, and
they are better than nothing, so there is no excuse not to have something workable
in place. But that does not mean that there is not room for substantial improvement.
Hence, my second agenda is to point out the importance of this type of tool to the
design process, and the value of investing in the development of better and more
appropriate solutions.
I conclude this section with a brief description of one such effort to do so. This is
to illustrate the kind of things that I think are needed and can be done. The underly-
ing point is that we need to be as conscientious in the design of our own tools as we
are in the design of the products that we make with them.
Around 2000, the group that I was involved with became interested in how de-
signers could add voice and marking annotations to 3D digital models. How could
they do so with the same type of fluency that they had previously exercised with
traditional media (Tsang, Fitzmaurice, Kurtenbach, Khan & Buxton 2002).
For example, how could the studio head walk around a digital model of a car
and visually indicate things while verbally commenting on them. W h e n appropri-
ate, how could this be accompanied by the ability to manually sketch on the surface
of the model? Could one do so with the same fluency and using the same skills as
traditionally used on velum? Could this be done in the studio without having to wear
cumbersome gear like head-mounted displays, special gloves, and such?
The simple answer is Yes to all these questions. The prototype system that we
built to demonstrate this was a kind of virtual video camera called Boom Chameleon.
It is illustrated in Figure 69.
The console of the device consisted of a microphone-equipped 17" touch screen
that was mounted on a counter-balanced articulated boom. All the articulated joints
were instrumented with shaft encoders that could be read by the computer in real
time. Hence, as you held the screen and walked around the central pedestal sup-
porting the boom, the position and orientation of the screen could be sensed by
the computer controlling what appears on the screen. Hence, the view of the model

181
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
 A n n o t a t i o n : S k e t c h i n g On Sketches

Figure 69: Boom Chameleon

Avirtual window that permits the user to view and annotate a 3D digital
model graphically using a finger on the touch screen, and simultane-
ously by voice. One can walk around the object to see it from all sides,

much as if the screen were a camcorder.

Source: T s a n g e t a l . ( 2 0 0 2 )

could dynamically change in a manner analogous to the way it would if you were
walking around it holding a video camera. Just like with the camera, you could move
up and down, or toward or away from the pedestal, which would result in being
presented a upper or lower view of the model, or a close up or distant view, re-
spectively. Furthermore, at any time, one could use the touch screen to highlight a
particular aspect of the model using a virtual flashlight. Likewise, one could draw on
the model's surface. This is illustrated in Figure 63, and can be seen in more detail in
the associated video.
As well as capturing what you pointed at and the marks that you made, the sys-
tem was capable of recording the movements of the display as well as your verbal
comments while doing so. In this sense, it was like a virtual video camera with point-
ing and marking capabilities.
Making and recording annotations is only half of the equation, however. Others
need to be able to know that tl~ey exist, where to find them, and how to view them.
The key to this is illustrated in Figure 70.
Annotations were indicated as virtual snapshots suspended above the 3D model.
Selecting any one of these initiated full-screen play-back of the annotation. So that
things didn't get too cluttered, the annotation indicators could be collapsed into just
a simple horizontal bar, or made invisible for unobstructed viewing of the model.
Is this the perfect annotation tool? Of course not. There is no such thing. This
is just an example of one approach to developing new tools to support the design
process.
Ultimately, there are two major take-away messages from this section.
First, in attacking new classes of design problems and adopting new technolo-
gies, we need to be vigilant that we don't inadvertently drop important and proven
aspects of the design process along the way, just because they are not well supported
by the tools that exist today.
Second, as I have already stated, we need to apply the same skills to the design
of our tools, environment, and process as we do to the products that all these are
intended to serve.
Fluent, simple, and spontaneous annotation is not the only example where this
applied. It is simply the one that I have chosen to make the point.

183
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
Figure 70: Annotate, and Record
During a design session, the users can move around the virtual model,
highlighting areas of interests, annotate using their fingers. The entire
session can be recorded, and key frames can be saved for reference.
Source: Tsang et al. (2002)

185
On Hunters, Gatherers, and Doodlers
The preceding discussion has made it pretty clear that designers are both doodlers, by
virtue of their sketches, as well as hunters and gatherers. The latter is made evident by
the types of paraphernalia that they collect for inspiration and reference material.
It is important to point out that not all of what is sketched or collected is for public consump-
tion. Some is for one's self, and is often private, like a journal, diary or scrapbook.
To give you a feel for the richness behind that simple term, I want to show you some
excerpts from the personal sketchbook of my colleague Richard Banks of Cambridge.
In so doing, I am showing you a heritage that predates Richard, by a long shot. His
notebook itself is from a company named Moleskine, "the legendary notebook of Van
Gogh, Chatwin, Hemingway, Matisse and C~line."
When my book designer Hong-Yiu Cheung was laying out this section, he insisted
that we reproduce the pages full size. His comment was"

... the hardcover sketchbook is one piece of original technology that is designed "for the
wild." If you think about it, the sketchbook is the artist/writer version of the mobile studio ....
the hardcover sketchbook was designed precisely to be portable, casual, and treated roughly,
for field work .... a place to deposit notions, ideas, intents, and half-formed thoughts.
We are often very self conscious about proposing ideas because they are not thought
through and hence embarrassing ourselves in front of others. The sketchbook is an environ-
ment where it can be safe for us to articulate these thoughts to ourselves.
The value ... does not come from the degree of finish, but the authenticity of the tracking
of a person's ideas. A sketchbook used only in the studio environment will probably not be
as rich as a sketchbook that followed a person that has travelled out in the field, in a variety
of contexts. By carrying it around with you all the time, it also allows you to reference back
in time in a moment's notice, to connect previous thoughts with the present situation. This
I think is also the power of the sketchbook. A kind of long-term development of criss-cross-
ing streams of consciousness.

Such is the designer's sketchbook. And such is the reason that we give this one such
prominence.
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
 ~.~
-~.,~ ~ z,~,./.. ~ :~,~ . ~ ~.,,~~.'-r~,:,~. ~~,~.

- ~,~.l; h

- ~. ~~ ~:h~, ~.L.o,~.
,. ~,.,.~..~,,, ,...~.

~,~-,~ _ ~ . C.,.c,,.r

I,, ~ 1,,~,-k. . L,'~,~I.,r c~~l.

I.,. - r ~,,,..
3 ~o,,. C , ~ ~ ~ . ' ~ I ~ .
"-" IZol~ .... L . . . . 9

.........................................................................................j
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
 ~.~
-~.,~ ~ z,~,./.. ~ :~,~ . ~ ~.,,~~.'-r~,:,~. ~~,~.

- ~,~.l; h

- ~. ~~ ~:h~, ~.L.o,~.
,. ~,.,.~..~,,, ,...~.

~,~-,~ _ ~ . C.,.c,,.r

I,, ~ 1,,~,-k. . L,'~,~I.,r c~~l.

I.,. - r ~,,,..
3 ~o,,. C , ~ ~ ~ . ' ~ I ~ .
"-" IZol~ .... L . . . . 9

.........................................................................................j
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-
,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
~i~:~!;z,~!,~!~:!~:~ii~i! ii~!;,~:::~-

,,:,.:,,:
~,~;.:i.:~
 -:'!,:-'?::::::::
. . . . . . . . . . . .

Figure 71: The Design Critique - Group Think or Firing Squad?

There are two extremes of design critique. One is the design jury,
generally preceded by a sleepless night and followed by jangled nerves
on the part of the person whose work is being reviewed. It probably
has its roots in the Spanish inquisition! The other is a respectful and
constructive commentary and review of the material under consid-
eration. It can be with peers, subordinates, supervisors or all three.
The objective is to find the best idea or understanding, not to score
points or show how clever you are. Individuals don't win, the design
does. And not being honest about problems or concerns is not an act
of friendship. The critique of this sort is the lifeblood of design, and
shows design to be one of the best types of team sport.
Credit: Harry Mahler