1

InterState: Interaction-Oriented Language

Primitives for Expressing GUI Behavior
Stephen Oney, Brad Myers
Carnegie Mellon University
Joel Brandt
Adobe Research


ABSTRACT
The event-callback programming model used by most GUI
development frameworks frequently leads to disorganized
and hard-to-maintain code. We believe there is significant
value in designing language primitives that expressly ad-
dress the challenges of writing and re-using GUI code. To
explore this, we created InterState, a language and pro-
gramming environment for expressing interactive behaviors
in graphical applications. With InterState, programmers
express behaviors declaratively as combinations of states
and constraints. InterState introduces new primitives for
defining and re-using behaviors, a visual notation for these
primitives, and a live editor. We conducted a laboratory
study to evaluate InterStates usability and found that par-
ticipants were faster at understanding and modifying GUI
components written in InterState than in event-callback
code. Additionally, to better understand InterStates scope
and expressiveness, we used it to implement a series of full-
featured interfaces.
INTRODUCTION
Nearly all widely-deployed user interface (UI) frame-
workse.g., Cocoa, QT, Java Swing, and JavaS-
cript/HTMLrely on an event-callback programming mod-
el [18]. By requiring programmers to express interactive
behavior as a series of imperative callbacks, this program-
ming model tends to produce error-prone spaghetti code
because it is necessary to split the implementation of a sin-
gle behavior across many locations in the source [21,28].
To address this issue, researchers and practitioners have
created libraries that augment existing languages and UI
frameworks with new programming models, including con-
straints (relationships that are maintained automatically)
[17,21,24] and state machines [1,28]. However, when pro-
ducing a UI is the programmers primary goal, many as-
pects of the underlying languagewhich evolved around
computation-oriented softwarecan be hindrances [16].
Our new development system, named InterState, is based
on the insight that by redesigning many language and
runtime featuresincluding the event-callback framework,
inheritance, the syntax, and the edit-compile-run evaluation
cyclewe may be able to create a development tools more
suitable for UI development. InterState is intended to be a
language for specifying UI behavior and appearance, rather
than a general-purpose programming language. It includes a
mechanism for communicating with back-end code written
in other languages, allowing developers to connect a front-
end written in InterState with a back-end written in another
language.
Overview of the InterState Approach
Two of the most difficult aspects of programming interac-
tive behaviors are dealing with states and expressing con-
straints [16,27,28]. ConstraintJS [24] addressed these diffi-
culties by allowing developers to express interactive behav-
iors in JavaScript and HTML/CSS as constraints that are
enforced only in particular states. InterState extends the
ideas behind ConstraintJS in two primary ways. First, new
language primitives for inheritance and templating increase
the expressiveness of constraints and states. Second, a visu-
al notation and a live editor remove the edit-compile-run
cycle, enabling immediate evaluation and allowing devel-
opers to see objects state.
An important innovation in InterState is a new form of in-
heritance. Developers often want to re-use, combine, and
inherit behaviors. However, the traditional notion of inher-
itance in languages like Java or JavaScript only allows
properties and methods to be inherited. InterState introduc-
es a style of inheritance that extends traditional prototype-
instance inheritance mechanisms to allow behaviors to be
inherited. It does this by allowing objects to inherit not only
properties and their constraints but also their prototypes
state machines. InterState also introduces a mechanism for
templating that allows interactive components to be dynam-
ically created and updated to reflect changes in an underly-
ing data model.
It is also important that users can understand their pro-
grams state. Thus, we introduce a visual notation for our

Figure 1. A screenshot of a basic InterState object, named rect.
Properties, which control rects display, are represented as rows
(e.g. x, y, and fill). States and transitions are represented as
columns (e.g. idle and dragging). An entry in a propertys row
for a particular state specifies a constraint that controls that prop-
ertys value in that state. Here, while rect is in the dragging
state, x and y will be constrained to mouse.x and mouse.y re-
spectively, meaning rect will follow the mouse while dragging.

Submitted to CHI 2014

2
language. In most languages, understanding what user
events affect a particular property or, conversely, what
properties are affected by a particular user event, can be
difficult because the code is often spread throughout multi-
ple locations. InterState displays object properties as rows
and states as columns, as illustrated in Figure 1. This layout
allows developers to see which events affect a property by
scanning across the propertys row and which properties an
event affects by looking at the events column. It also re-
moves much of the boilerplate required to express con-
straints in other systems, allowing users to express con-
straints with simple equationslike those in spreadsheets
rather than requiring inline functions [17,24].
Because the primary goal of most user interfaces is to be
usable, its also important that developers can immediately
use and evaluate their application as they create it, also
called reflection-in-action [29]. To enable reflection-in-
action, we implemented a live visual editor, meaning that
changes in the editor are immediately reflected in the run-
ning application and changes in the runtime state are imme-
diately visible in the editor
Contributions
This paper makes four primary contributions:
Language primitives for expressing UI behavior that
build on the idea of combining states and constraints,
a specification for inheritance and templating that al-
lows these primitives to be re-used across objects,
a visual notation and live editor that allow developers
to see the state of their program, and
an evaluation of these features through a laboratory
study and a series of example applications.
RELATED WORK
InterState is influenced by work in multiple domains.
The Difficulties of Developing Interactive Applications
Previous research has investigated why developing interac-
tive applications is particularly challenging. One factor is
that interaction-oriented development tools must support
different state and reuse mechanisms than general purpose
development tools [16]. InterState addresses these issues by
treating state and behavior reuse as primitives. Two moti-
vating studies found that designers think about relationships
between graphical objects with state, constraint, and event-
based concepts [27] and that reflection-in-action is im-
portant for designers creativity [25]. Thus, InterStates
basic primitives were built around dealing with states and
supporting constraints, and we implemented InterState as a
live editor to support reflection-in-action.
Spreadsheet Programming
Spreadsheets are considered by many researchers to be the
most popular form of programming [18]. Part of their
appeal lies in their beginner friendliness: the user always
has a working program and errors can be localized.
There has been research in ways to use spreadsheet-like
ideas to create graphical interfaces. One such system is
Forms/3 [4], which demonstrated that procedural and data
abstractions and graphical output were viable with spread-
sheets. However, Forms/3 does not offer explicit support
for dealing with state. By including state as a language
primitive, InterState becomes more expressive and better
suited to writing interactive behaviors, which are often
state-oriented.
Constraint Libraries for Imperative Languages
A number of libraries intended to simplify the development
of interactive behaviors have been written for imperative
languages. Of these, the most relevant to InterState is Con-
straintJS, a JavaScript library on which InterState is built
[24]. Like InterState, ConstraintJS combines states and con-
straints to define interactive behaviors. However, Con-
straintJS was built specifically to fit into the context of im-
perative programming with JavaScript and HTML/CSS,
whereas InterState uses a visual notation to allow users to
express interactive applications with little or no imperative
code. Achieving this goal went beyond simply adding a
visual notation on top of ConstraintJS; it also required de-
signing several new language primitives.
There are several other data-binding and constraint librar-
ies, including Knockout (knockoutjs.com), Ember (em-
berjs.com), Amulet [20], Kaleidoscope [6], and D
3
[3]. Ad-
ditionally, FlapJax [17] provides both a constraint mecha-
nism and event model. While all of these libraries can be
effective in allowing skilled developers to write clearer
code for interactive applications, none of them include
primitives for dealing with state or a visual notation.
Finite State Machine Libraries for Imperative Languages
Since interactive applications are highly state oriented, Fi-
nite State Machine (FSM) libraries seem particularly suita-
ble for GUI development tools [23]. TKZink, IntuiKit [15],
and HsmTk [2] use state machines to allow interface de-
signers to specify different application appearances in dif-
ferent states but do not allow developers to specify custom
interactive behaviors. Similarly, SwingStates [1] integrates
finite state machines with the Java Swing toolkit but does
not provide primitives for interaction or user event handling
as InterState does.
Interactive and Live Editors
Many programming environments provide non-textual ele-
mentssee [31] for a survey. Proton [13] also introduced a
declarative syntax and interactive editor for describing se-
quences of events for touch-based devices. LiveWorld [30]
and several GUI builders allow users to set object properties
using property sheets, which list settable properties and
allow users to change them, sometimes updating the inter-
face to reflect their current values. These property sheets
can specify the look (colors, fonts, positions, etc.) of an
application but InterState incorporates states and constraints
to allow developers to also specify how an application be-
haves.
3
Event Languages and Models
Finally, many commercial and research systems have used
and augmented the event-action framework. Early event
models, like Sassafras [10] and the University of Alberta
User Interface Management System [7] inspired the fea-
tures of future commercial systems [18]. One interesting
extension of the standard event model is the elements,
events, & transitions model, which allows developers to
more concisely express how user interfaces should respond
to user events [5]. Although such approaches have the po-
tential to make event-action driven user interfaces more
concise, they still split interactive behaviors code across
many locations.
DEVELOPING WITH INTERSTATE
InterState is implemented as a browser-based application.
Runtime and Editor
InterState separates its runtime (which maintains con-
straints, translates InterState objects into DOM objects in
the browser, and coordinates execution of the users pro-
gram) and editor. This allows developers to create applica-
tions using the editor and deploy them using only the
runtime. When the editor is open, it is displayed as a sepa-
rate window that communicates with the runtime window.
Because InterState is specifically tailored to implementing
GUI behavior, it is likely that programmers will want to
implement some non-UI portions of their applications out-
side of InterState. To support this, code running in Inter-
State can communicate with JavaScript in three ways: First,
InterState objects can call JavaScript functions on transi-
tions. Second, InterState objects can reference JavaScript
variables in exactly the same way they reference InterState
variables. Finally, JavaScript code can emit events to which
InterState objects react.
Live Development
One of the important aspects of our development environ-
ment is that it is live changes made to the programs
source code are instantly reflected in the running applica-
tion. We decided that it was important to have a live devel-
opment environment for three reasons.
First, liveness can make the environment more approacha-
ble and beginner-friendly by helping to bridge the gulf of
evaluation [22], a significant barrier for new developers
[14]. Another important aspect of most live development
environments is that the developer always has a running
program. One great aspect of spreadsheet programming, for
instance, is that when the user makes a mistake in a particu-
lar cells formula, the entire spreadsheet does not stop
working [4,18]. Similarly, InterState allows errors to be
localized: cells with errors only prevent the parts of the
program from running that depend on those cells.
Second, liveness can enable the user to quickly evaluate the
design. Although syntactic errors can sometimes be made
immediately visible in edit-compile-run environments, live
programming allows both syntactic and semantic errors to
become immediately apparent by enabling developers to
immediately test their code. This is particularly important
because reflection-in-actionstepping back and evaluating
their design as designers are in the process of creating itis
a crucial part of the design process [29]. While sketches and
drawing applications allow designers to quickly evaluate
the look of their application during the design process, In-
terState is designed to be one of the first tools to allow them
to quickly evaluate the feel of their application.
Finally, liveness can enable quick experimentation and pa-
rameter tweaking. Experimentation is a crucial part of the
design process and one that is not well supported by todays
development environments [8]. Again, it is relatively easy
to experiment with different application looks with sketch-
es, drawing programs, etc. However, it is more difficult to
change or experiment with the feel of the application.
INTERSTATE PRIMITIVES AND FEATURES
InterState expresses interactive behaviors by specifying
constraints that apply in particular states. There are three
basic entities in InterState: objects, properties, and cells. In
this section, we will explain these three basic entities and
their features in detail.
Objects
Objects may contain any number of named fields
1
. A field
may be a property or another object. A field with a property
always has a value. Fields containing objects form a con-
tainment hierarchy, with a top-level object that is always
called sketch. Every object also has at least one associated
state machine, where all state machines start with a single
state by default. The InterState editor displays every ob-
jects state machine horizontally at the top of the object (see
Figure 1). As we will describe in more detail later, an object
may also inherit from other objects, in which case it will
inherit the other objects fields and a copy of their state
machines and constraints. In the constraint and event ex-
pressions, the current object is called this.
Properties
A property maps every state and transition of an objects
state machine to a value. That value is either empty (repre-
sented as a grey circle) or a cell (represented as a rectangle
with the cells value). When a state is active or a transition
fires, the propertys value is computed from its value in that
state or transition. If that value is empty, then the last value
remains in use. Cells set on states are evaluated as con-
straints, whereas cells set on transitions are evaluated once,
to allow values to be stored. For example, in Figure 1, the
x field is a property, and in the dragging state, there is a
cell that constraints its value to the expression mouse.x. In
the transition from dragging to idle, its last x position is
stored. However, in the idle state, the property is empty,
so its last value continues to be used.

1
InterState does not distinguish between methods and fields, as Java
does. Instead, functions are first-class and can be used as a fields value.
4
Cells
Cells are meant to hold an individual value. That value can
be a constant (e.g. 50 or 'red') or it can be a constraint
(e.g. x+1 or sqrt(y)) that recalculates its value based on a
formula (see Referencing below). InterState uses Con-
straintJS [24] as its underlying constraint solver, and sup-
ports one-way constraints with indirection (the target object
can be itself be calculated by a constraint), as in:
this.object_at_left.color
In Figure 1, in the dragging state, the x and y properties
of rect are constrained to mouse.x and mouse.y respec-
tively.

State Machines
Behaviors often combine multiple state machines; an object
might, for instance, be draggable and selectable. In order to
avoid the state explosion problem where developers would
have to create combinatorial number of states (e.g.
dragging_and_selected, idle_and_selected, etc.),
InterState extends the standard finite-state machine model
in two ways. First, InterState allows state machines to be
concurrent, with multiple states active simultaneously
[9,12]. When multiple states are active, InterState uses left-
to-right precedence to choose which active value properties
should use, if there are conflicts. Second, it allows states to
be nested a state can hold any number of substates, al-
lowing some behaviors to be expressed more concisely [9].
In the current design of the interactive editor, states are
added by clicking the + button on the far right (see Figure
1). Transitions are added using a context menu that appears
using a right click. Every transition has an associated event,
which can be viewed and edited in the text box above the
transitions arrow. The trapezoidal shape of InterStates
states is designed to allocate a column every transition, hor-
izontally centered where the transitions arrow begins.
Transition Events
Most transition events are specified using the format:
on(<event_type>, <params>), where <event_type>
is a user event to listen for (e.g. 'mousedown') and
<params> represents any number of parameters for the
event (e.g. the target object for mouse events or the delay
for timer events). Guards may be added to events to further
specify conditions when their transition should fire:
on('mousedown', this).when(this.is_active).
In addition to user events, developers often need to express
events referring to changes in fields valuesfor instance,
to express that a draggable object should be highlighted
when it overlaps with its drop target. In imperative lan-
guages, this would require that the developer change that
property through a setter, which would then emit a custom
event. InterState simplifies this by introducing constraint
eventsBoolean expressions like (this.x > 500)that
fire any time the value of the expression switches from
false to true.
Higher-level Transition Events
Sometimes, we want to give semantic meaning to an event.
Thus, InterState includes a way to emit a higher-level event
when any transition fires by adding extra expressions after
the transition events expression. For example, the state
machine shown in Figure 3 can emit a triple-click event on
this object by setting the second transitions event to:
on('click', this); emit('tplclick', this)
These expressions can also call JavaScript functions:
on('click', play_button); play(curr_song)
Inheritance
It is often useful to inherit properties. For example, suppose
we have an object that represents a square. Every square
should have properties for width and height whose val-
ues are equal. InterState enables such inheritance. Objects
inherit from another by specifying what objects they should
inherit from in their prototypes field, as shown in Figure
4, where my_square inherits from square. Inherited
properties (e.g. my_square.height) are greyed out.
InterState inheritance is modeled after traditional prototype-
instance inheritance, but with several important differences.
First, rather than inheriting property values InterState inher-

Figure 2. Syntax and runtime errors are highlighted in the edi-
tor but do not prevent the program from running.


Figure 4. Objects can inherit properties from other objects by
placing them in their prototypes field. Here, my_square in-
herits from square. Because my_square has defined a proper-
ty for width but not height, it uses its own definition of width
but inherits squares definition for height, as indicated by
the greyed out color. Note that my_square inherits the defini-
tion of height, not the value. Thus, the variable width evalu-
ates to a different value (20) than it does in square (15).

Figure 3. A state machine representing triple clicking.
5
its property definitions. This is illustrated in Figure 4, where
my_square inherits the definition of height, rather than
its value, since my_square.height should equal
my_square.width, not square.width. By inheriting the
definition rather than the value, InterState allows prototypes
to define behaviors that use the state and property values of
the objects that inherit from them.
Second, unlike most prototype-instance inheritance models,
InterState allows multiple inheritance; objects may inherit
from any number of other objects. This is expressed by en-
tering an array of objects into the prototypes field. If an
InterState object inherits from multiple objects with con-
flicting values for the same field name, InterState uses left-
to-right precedence for conflict resolution. Object property
values are combined across inherited values; if an objects
property is not defined for a state but it is in one of the ob-
jects prototypes, it inherits that prototypes definition for
the state. This allows multiple behaviors to control the same
property simultaneously. An example of this can be seen in
Figure 5, where my_selectable_draggable inherits
from both selectable and draggable. Because both
prototypes define color, the left-most value will be used;
if my_selectable_draggable is in the selected state,
it will be 'blue'; otherwise, it will be 'black' or 'red',
depending on the dragging state.
Third, when one InterState object inherits from another, it
also inherits a copy of that objects state machine. For ex-
ample, in Figure 5, my_selectable_draggable gets a
copy of the state machines for both selectable and
draggable. The fact that a copy of the state machine is
inherited, rather than the state machine itself, is important;
we usually do not want all of the objects that inherit from a
particular object to be in the same state. For example, we do
not want every object that inherits from draggable to en-
ter the dragging state when one of them does
2
.

2
If desired, having them all change state together could be implemented
by changing target of the on('mousedown', this) event so that it is
We often want the same behavior to apply across multiple
unrelated objects. In order to enable this, other toolkits have
used the strategy of creating separate interactor objects that
describe behavior which are attached to graphical objects
[11,20]. However, InterStates standard notion of inher-
itance includes behavior inheritance.
Finally, the prototypes field, like every other property,
can have different values in different states, and can even
be computed by constraints, allowing the prototypes of any
given object to depend on its current state and other fields.
For instance, an object might inherit from draggable only
when it is enabled.
Field References & Scoping
To avoid naming conflicts, InterState scopes field name
references by object. Whenever a cell references a field
name as part of a constraint, InterState searches up its con-
tainment hierarchy to find that variable. If InterState cant
find a variable in the containment hierarchy, it then search-
es for JavaScript variables with the same name. This search
is done live, so that if properties change values or names,
then their reference also changes immediately. To illustrate,
suppose we have the following containment hierarchy:
x: 'far'
OBJ:
x: 'near'
prop_a: x
Then prop_a gets the value 'near'. If we delete the x
property of OBJ, then prop_a changes to 'far'. InterState
includes three reserved keywords:
this: the current object.
parent: the current objects parent in the containment
hierarchy.
sketch: the top of the containment hierarchy.

something like on('mousedown', obj1). That way, every draggable
instance would use the object obj1 as the event target.

Figure 5. An object that inherits from both draggable and selectable selectable behaviors. Note that the values from the color
field are inherited from both draggable ('red') and selectable ('blue').
6
For instance, in the above snippet, if we wanted to specify
that OBJ.prop_a refers to the x property of OBJ, we could
use either this.x or OBJ.x.
Copies and Templating
Often, developers need a list of similar items to display and
do not want to declare a display for every object in that list,
either because it is too tedious or because that list of items
will be computed at runtime. In imperative languages, this
functionality has been implemented as list views in data-
binding libraries or maps (e.g. Amulet [20]) that allow pro-
grammers to specify a template display and to specify the
number of instances they want.
InterState handles this by adding an optional copies field
to ordinary objects. When copies is set to an array or a
number, its parent object then creates a set of items rather
than a single item. The editor visually signals this by dis-
playing a stack under the objects display (see Figure 6).
For every item, InterState sets two properties: my_copy,
which carries the value for a particular item (e.g. 'item1',
'item2') and copy_num, which carries the index for a
particular item (e.g. 0, 1). When the value of copies
changes (dynamically or by user edits), the list is updated
with respect to added, moved, and removed items instead of
recreating the entire list.
For example, suppose we have a color palette that shows a
tiny swatch for a set of colors that a user has set as favor-
ites. The list of favorite colors is stored in the favorites
variable as hex values (e.g. ['0x900', '0x333']). When
users click add favorite, a new element is pushed onto
that list and when they click remove favorite, the selected
element is removed. The developer wants to specify only
once how to display every swatch, by using the
color_disp prototype, and by expressing that a copy of it
should be created for every element in the favorites list.
They can set copies to a constraint to favorites and the
InterState runtime environment creates a copy of
color_disp for every element in the favorites array
(updated automatically). color_disp can then constrain
its fill property to be my_copy, so that every instance has
the appropriate color.
Note that the copies prototypes fields can be computed
by a constraint that depends on each copys my_copy field.
For instance, in a directory viewer application, we could set
copies to the contents of the directory. Then, an item
could inherit from folder_view if my_copy is a folder or
from file_view if my_copy is a file.
Queries
It is often useful to have a powerful way to express opera-
tions on groups of objects to, for example, find every
checkbox that is in the selected state. InterState includes
a built-in function called find for making such queries
with a chaining syntax inspired by other query languages
and libraries like XQuery, jQuery, EET [5], and HANDS
[26]. For example, the aforementioned query could be rep-
resented with the expression:
find(chkboxes).in_state('selected')
Query expressions can be used in cells, for example, to
keep track of every selected checkbox, or they can be used
in transition events. For example, to create an event that
fires when no checkboxes are selected, we could write:
find(chkboxes).in_state('selected').len()==0
Performance was a prime concern in our implementation of
queries. One way we optimized performance was by using a
chaining syntax that does not require re-evaluating expres-
sions further up the chain. Thus, in the above example, if
one chkboxes switches to or from the state 'selected',
we would not re-evaluate the expression find(chkboxes)
and constraints that depend on non-changed items in the list
returned by the query would not be re-evaluated.
Manipulating Visual Objects
Finally, all of these components must be connected to an
output mechanism. We have implemented two primary out-
put mechanisms: one for creating HTML DOM objects and
another for creating Scalable Vector Graphics (SVG) ob-
jects. We have also experimented with WebGL as an output
mechanism for creating 3D interfaces. We will limit our
discussion to SVG objects, because three of the four exam-
ples in our Applications section use SVG objects (the
alternate InterState editor uses DOM output). The DOM and
WebGL output mechanisms are analogous.
To make an SVG object appear on screen, developers can
make that object inherit from one of seven types of SVG
objects, all children of the sketch.shape object: circle,
ellipse, image, rect, text, group, and path (which
includes lines, arcs and other custom shapes). All of these
prototypes declare default values for properties that control
how they are displayed (e.g. rect has a width attribute
and image has an src attribute). Other attributes include:
animated_properties: an array of property names
that should smoothly animate between values (if it is set
to true, every propertys changes will be animated)

Figure 6. An object (obj) with multiple copies. copies is set to
['item1', 'item2', 'item3']. Every copy has two prop-
erties: my_copy, which is set to that copy's item and
copy_num, which is set to that copy's index. Here, we are look-
ing at the first copy (as indicated by the [0, length 3]).
7
animation_duration and animation_easing to
control the animations length and easing
show to determine if the object is visible or hidden
Sometimes, we want to create SVG objects that should not
be shown on screen. For example, we might create a
button prototype object that specifies how a button should
behave but we do not want button itself to appear. There-
fore, InterState only places objects in the runtime window if
they are children of the top-level screen object.
USER STUDY
To evaluate the understandability of InterStates visual no-
tation and the usability of its editor, we conducted a user
study. We recruited 20 experienced developers between the
ages of 19 and 41. Participants were mostly professional
developers or students and were geographically distributed
across the United States; half of our participants participat-
ed remotely using screen-sharing software.
Method
Participants were sequentially given two interactive behav-
iors; one implemented in JavaScript using the RaphaelJS
drawing and event-handler library (called JS) and another
implemented with InterState (called IST). For each behav-
ior, participants were asked to make modifications to evalu-
ate their ability to understand the implemented behavior and
express a new behavior.
For the first behavior (called B1) participants were given
code for a standard drag and drop behavior. Participants
were asked to implement drag lockan accessibility fea-
ture that allows users to double click an object to drag it
until they double clicked again. The second behavior
(called B2) was an image carousel that displayed a large
featured image and series of thumbnails that changed the
featured image when clicked or auto-advanced automatical-
ly after a timeout. We asked participants to change display
features of the thumbnails, the auto-advance interval, and to
add an indicator below the featured thumbnail to count
down with the auto-advance interval. The relative number
of lines of code and objects is detailed in Table 1.
Both of these behaviors were implemented in JavaScript
and in Interstate. Participants were given the same task de-
scription regardless of implementation. We counterbalanced
the order of the tasks, creating a total of four participant
groups (B1
JS
/B2
IST
; B1
IST
/B2
JS
; B2
JS
/B1
IST
; and B2
IST
/B1
JS
).
To make our comparison as fair as possible, we started with
third-party code for the JavaScript implementations and
simplified it by reducing boilerplate and adding descriptive
variable names that were consistent with those used in the
InterState implementations. We also used a live JavaS-
cript editor (JSBin) that immediately re-evaluates JavaScript
snippets when their source changes. Finally, participants
were given tutorials in both JavaScript (with RaphaelJS and
JSBin) and InterState. They were also given comparable
reference sheets in for RaphaelJS JavaScript and InterState.
Results
Behavior 1 (Drag Lock)
Users were able to implement the drag lock task significant-
ly faster with InterState (two-tailed heteroscedastic Stu-
dents t-test, p < 0.05). However, there was a large variance
in time needed between participants in the JavaScript condi-
tion. Although relatively few lines of code were required,
reasoning about callbacks timing proved challenging for
many users. Many participants used the strategy of console
logging to help track their interfaces state.
Even after completing the task, many JavaScript partici-
pants were not confident in their implementations correct-
ness. As one participant in the B2
IST
/B1
JS
condition (who
completed the task relatively quickly) noted afterwards, I
dont know about all of the event combinations but I think I
got it right; I was more sure in InterState.
Behavior 2 (Image Carousel)
Behavior 2 was split into three sub-tasks. The first two sub-
tasks required users to find the variables that controlled the
thumbnail width and timing of auto-advances. Interestingly,
although participants were able to complete the overall task
significantly faster with the InterState editor (p < 0.05),
they were able to complete the variable modification sub-
tasks significantly faster in JavaScript. This appeared to be
largely because of the different approaches required by edi-
tor. Whereas JavaScript participants were able to scan their
code for expected identifiers or search for certain keywords,
InterState users had to navigate to the relevant object first.
The last sub-task required users to add a timer indicator.
Participants in both implementations used two strategies for
this: either creating an indicator for each thumbnail or cre-
ating one indicator that follows the featured thumbnail.

Figure 7. The relative times (in minutes) across 20 participants
to complete tasks in JavaScript (JS) and InterState (IST).

"#: urag Lock "$: lmg. Carousel
%&
Llnes of Code 33 (+ 12) 60 (+ 17)
# Callbacks 3 (+ 1) 2 (+ 0)
'&(
# Cells 11 (+ 2) 33 (+4)
# Cb[ecLs 1 (+ 0) 2 (+ 1)
# roperLles 7 (+ 0) 22 (+4)
# SLaLes 2 (+ 1) 3 (+ 0)
# 1ranslLlons 2 (+ 2) 6 (+ 0)
Table 1. The relative sizes of the user studys two behaviors
and the minimum size of modifications required for the tasks.
8
Post-Survey and Interview
Most participants felt comfortable with InterStates visual
notation, calling it intuitive and clean. Nearly every
user cited InterStates ability to display the current applica-
tion state and property values live as its most useful fea-
ture. This helped many users quickly debug and deduce the
meaning and roles of some properties. For example, in B2,
both implementations had a property that tracked the num-
ber of milliseconds before the featured image auto-
advanced. Most JavaScript (B2
JS
) participants missed this
variable while most InterState participants found it, appar-
ently by observing how its value changed over time. Partic-
ipants also cited the ability to look at any object to find its
attributes as another useful feature of the InterState editor.
On the other hand, some participants expressed skepticism
that they would ever use a visual language in practice; even
one participant who agreed that InterState was a clear rep-
resentation of behaviors could not imagine programming
using any visual language. The most cited reason for this
was that participants still felt more comfortable with
standard imperative code. This may be largely due to their
relatively long exposure to standard code or the dearth of
commercially available visual languages.
Discussion
Our user studies pointed to a number of ways to improve
future versions of the InterState editor. First, as a conse-
quence of how InterState lays out code, InterState develop-
ers often need to search to find objects. This is because
InterState requires property modifications to be written on
the propertys row. This is in contrast with JavaScript and
most imperative languages, where property modifications
can happen anywhere. This search could be textual or could
be in the form of an inspector that allows users to click on
objects in the runtime window to immediately open their
representation in the editor window (currently, InterState
only supports the opposite direction: highlighting runtime
objects when the user is editing their representation).
Additionally, when editing, participants often needed to
reference properties that were outside of the editors visual
scope, pointing to the need to increase the amount of infor-
mation available in the editor. We plan on exploring ways
to do this without increasing visual clutter.
One unexpected advantage of InterState over JavaScript
was that while performing JavaScript tasks, nearly every
participant broke (wrote code that did not compile) their
code at some point. This was largely because adding new
behavior to event-callback code often requires modifying
old callbacks in addition to adding new callbacks. By con-
trast, adding new behavior to InterState examples largely
only required adding states, transitions, and cells.
APPLICATIONS
To evaluate InterStates expressiveness, we implemented a
number of example applications. In this paper, we will de-
scribe four of these applications. Note that all of these ex-
amples were built entirely with InterState without any pre-
built widgets (as in no primitives for buttons, etc.).
Music Player & Playlist Editor
We implemented a music player and playlist editor shown
in Figure 8. This example illustrates how InterState can be
used to build front-ends that call code written in another
language; the actual audio output is controlled using JavaS-
cript functions that call the HTML5 audio API. It re-creates
many of the features of a desktop music player and playlist
manager, allowing users to add or remove songs from
playlists, create new playlists, and to play songs and control
the volume and playback position.
This example also demonstrates how InterState can be used
to build complex real-world interfaces. It leverages Inter-
States use of constraints to make dealing with changing
model data (playlists and songs being added and removed)
trivial. It also uses behavior inheritance for its buttons and
sliders. Overall, this example was built using 11 InterState
objects, and 90 cells across 16 states with 21 transitions.
Breakout
We also implemented a version of the Breakout game. In
this example, the position of the ball is updated by a transi-
tion that fires once per frame. Ball bounce events are im-
plemented as constraint events (when x<0 or
hit_test(ball, blocks) fires, we change the direction
of the ball). Here, hit_test is a simple hit testing function
written in InterState. In an alternate implementation, we
could write constraints that depend on the current time to
compute the balls position and when it would hit a particu-
lar block. We could also integrate third-party physics librar-
ies, an idea we have already experimented with.
The behavior for every block is implemented in a prototype
with multiple copies. By default they are laid out in a grid
but this can vary by level. In Breakout, the player reaches
the next level by destroying all of the blocks in the current
level. Thus, we have a level object that has an event to
detect if the user has finished a level (i.e. there are no more
blocks in the alive state):
find(blocks).in_state('alive').is_empty()
When this happens, every block transitions from dead back
to alive and the level variable increments (by setting it

Figure 8. A music player and playlist editor implemented in
InterState. In this application, the user can create, remove, and
edit playlists and play, pause, and control the playback position
& volume of songs. This example shows how InterState can
deal with user data that is created on the fly.
9
to level+1 in a transition). In this example, we increase
the speed and the number of rows every level. Overall, this
example required five InterState objects with 51 cells across
10 states and 21 transitions.
For comparison, we found an implmentation of breakout
written for the Web in JavaScript
3
. This example was
written by an expert as a tutorial for other developers and
was thus written as cleanly as possible with around 1,000
lines of code. Despite being relatively cleanly written, it is
difficult to answer some basic questions about how the code
works just by looking at it. For example, if we wanted to
understand what might affect the balls position, we would
need to search across multiple (about seven in this case)
event handlers, which are in turn initialized in various non-
local parts of the code.
This kind of scatterring of functionality across multiple
locations in code is inherent to the event-callback style
required by most languages; either the code that affects a
particular property must be grouped (e.g., all of the event
listeners affecting the ball are part of the balls class) or all
of the properties that an event affects must be grouped (e.g.,
everything that a keypress may affect is placed in one
keypress event handler).
Shape Editor
We implemented a shape editor that allows users to select
and manipulate properties of retained objects, including its
position, size (from all eight possible edges and corners),
rotation, and color. It also enables optional gridding and
keyboard modifiers to change the resize mode (e.g. resize
from center by holding CTRL). This example illustrates
InterStates ability to deal with applications that have a
large number of possible states; manipulable objects have
about 12 columns and its representation still fits on a single
15-inch laptop screen without scrolling.
InterState Editor
Finally, weve built an alternative InterState editor using its
own primitives. This editor displays InterState objects as a
tree. Its implementation looks at every InterState object and
decides which view to inherit from based on that objects
type (an object will inherit from object_view, a cell from
cell_view, etc). The object_view prototype has one
child_views property whose copies is set to that
objects list of properties. This allows the list of that
objects properties to change automatically with the
underlying data model. This example uses 76 cells across
34 objects and calls JavaScript code to manipulate the
InterState objects. Although implemented as a proof-of-
concept, we are exploring the idea of writing future
versions of the InterState editor with its own primitives.

3
github.com/jakesgordon/javascript-breakout/
IMPLEMENTATION
InterState is built in HTML and JavaScript using the Con-
straintJS constraint solver [24]. InterState also uses the es-
prima.org ECMAScript parser to generate constraints from
expressions written in cells. Communication between the
InterState editor and runtime windows is done through a
wrapper layer using the HTML channel messaging API. The
InterState editor uses asynchronous constraints to track the
variable states and values in the runtime window. The edi-
tor sends edit commands to the runtime window through the
same wrapper layer. InterState objects can also be serialized
and use the HTML local storage API to save and load Inter-
State programs across sessions or as files.
FUTURE WORK
Syntax
The InterState constraint syntax is currently based on Ja-
vaScript expressions (enabling, but not requiring any of the
control structuresif, while, etc.). However, we plan on
investigating ways of making the syntax more beginner-
friendly, guided by commonalities in how users naturally
describe behaviors [27].
Widgets and Modules
Widgets and example code can help beginners to get started
with any new language or development environment. Thus,
we want to create a set of white box widgetspre-built
components that can be used as-is, or whose source can be
modified or overridden, as necessary.
Other Features
Participants in our user study suggested a number of minor
improvements to our interface, including auto-complete,
automatically resizing state displays, and better debugging
support. We will investigate how to include these features.
CONCLUSION
In this paper, we presented the design and implementation
of InterState, a programming framework and development
environment for creating interactive applications. We also
evaluated InterStates usability with a laboratory study and
its expressiveness with a series of example applications. We
plan on making the full source freely available in the near
future. We also plan on adding more features and creating
tutorials and documentation.
ACKNOWLEDGEMENTS

(Blank for anonymity)

REFERENCES
1. Appert, C. and Beaudouin-Lafon, M. SwingStates:
Adding state machines to Java and the Swing toolkit.
Software: Practice and Experience 38, 11 (2008), 1149
1182.
10
2. Blanch, R., Beaudouin-lafon, M., and Futurs, I.
Programming Rich Interactions using the Hierarchical
State Machine Toolkit. AVI, (2006), 5158.
3. Bostock, M., Ogievetsky, V., and Heer, J. D
3
: Data-
Driven Documents. Visualization and Computer
Graphics 17, 12 (2011), 23012309.
4. Burnett, M., Atwood, J., Djang, R.W., Gottfried, H.,
Reichwein, J., and Yang, S. Forms / 3!: A First-Order
Visual Language to Explore the Boundaries of the
Spreadsheet Paradigm. Functional Programming 11, 2
(2001), 155206.
5. Frank, M.R. Model-Based User Interface Design By
Demonstration and By Interview. 1995.
6. Freeman-Benson, B. Kaleidoscope: Mixing Objects,
Constraints, and Imperative Programming. OOPSLA,
(1990), 7788.
7. Green, M. A Survey of Three Dialogue Models. ACM
Transactions on Graphics 5, 3 (1987), 244275.
8. Grigoreanu, V., Fernandez, R., Inkpen, K., and
Robertson, G. What designers want: Needs of
interactive application designers. VL/HCC, (2009), 139
146.
9. Harel, D. Statecharts: A Visual Formalism for Complex
Systems. Science of Computer Programming 8, 3
(1987), 231274.
10. Hill, R.D. Supporting Concurrency, Communication,
and Synchronization in Human-Computer Sassafras
UIMS. ACM Transactions on Graphics 5, 3 (1987),
179210.
11. Hudson, S.E. and Mankoff, J. Extensible Input Handling
in the subArctic Toolkit. CHI, (2005), 381390.
12. Jacob, R.J.K. A State Transition Diagram Language for
Visual Programming. Computer 18, 8 (1985), 5159.
13. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M.
Proton: Multitouch Gestures as Regular Expressions.
CHI, (2012), 28852894.
14. Ko, A.J., Myers, B.A., and Aung, H.H. Six Learning
Barriers in End-User Programming. VL/HCC, (2004),
199206.
15. Lecoanet, P., Lemort, A., Mertz, C., et al. Revisiting
Visual Interface Programming: Creating GUI Tools for
Designers and Programmers. UIST, (2004), 267276.
16. Letondal, C., Chatty, S., Phillips, W.G., and Andr, F.
Usability requirements for interaction-oriented
development tools. PPIG, (2010), 1226.
17. Meyerovich, L., Guha, A., and Baskin, J. Flapjax: A
Programming Language for Ajax Applications.
OOPSLA, (2009), 120.
18. Myers, B., Hudson, S.E., and Pausch, R. Past, Present,
and Future of User Interface Software Tools. TOCHI 7,
1 (2000), 328.
19. Myers, B., Park, S.Y., Nakano, Y., Mueller, G., and Ko,
A. How Designers Design and Program Interactive
Behaviors. VL/HCC, (2008), 177184.
20. Myers, B.A., Mcdaniel, R., Miller, R., et al. The Amulet
Environment!: New Models for Effective User Interface
Software Development. IEEE Transactions on Software
Engineering 23, 6 (1997), 347365.
21. Myers, B.A. Separating Application Code from
Toolkits: Eliminating the Spaghetti of Callbacks. UIST,
(1991), 211220.
22. Norman, D. The Design of Everyday Things. Doubleday,
New York, New York, USA, 1988.
23. Olsen, D.R. User Interface Management Systems:
Models and Algorithms. Morgan Kaufmann, San Mateo,
CA, 1992.
24. Oney, S., Myers, B., and Brandt, J. ConstraintJS:
Programming Interactive Behaviors for the Web by
Integrating Constraints and States. UIST, (2012), 229
238.
25. Ozenc, F.K., Kim, M., Zimmerman, J., Oney, S., and
Myers, B. How to support designers in getting hold of
the immaterial material of software. CHI, ACM Press
(2010), 25132522.
26. Pane, J.F., Myers, B.A., and Miller, L.B. Using HCI
Techniques to Design a More Usable Programming
System. HCC, (2002), 198.
27. Park, S.Y., Myers, B., and Ko, A.J. Designers Natural
Descriptions of Interactive Behaviors. VL/HCC, (2008),
185188.
28. Samek, M. Who Moved My State? Dr. Dobbs Journal,
2003.
29. Schn. The Reflective Practitioner. Temple Smith,
London, England, 1983.
30. Travers, M. Recursive Interfaces for Reactive Objects.
CHI, (1986), 379385.
31. Zhang, K. Visual Languages and Applications. Springer,
2007.