Journal of Building Performance Simulation

ISSN: 1940-1493 (Print) 1940-1507 (Online) Journal homepage: https://www.tandfonline.com/loi/tbps20

Effects of real-time simulation feedback on design

for visual comfort

Nathaniel L. Jones & Christoph F. Reinhart

To cite this article: Nathaniel L. Jones & Christoph F. Reinhart (2019) Effects of real-time
simulation feedback on design for visual comfort, Journal of Building Performance Simulation, 12:3,
343-361, DOI: 10.1080/19401493.2018.1449889

To link to this article: https://doi.org/10.1080/19401493.2018.1449889

Published online: 13 Mar 2018.

Submit your article to this journal

Article views: 290

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tbps20
Journal of Building Performance Simulation, 2019
Vol. 12, No. 3, 343–361, https://doi.org/10.1080/19401493.2018.1449889

Effects of real-time simulation feedback on design for visual comfort

∗
Nathaniel L. Jones and Christoph F. Reinhart
Department of Architecture, Massachusetts Institute of Technology, Cambridge, MA, USA
(Received 14 October 2017; accepted 3 March 2018 )

Building performance simulations and models of human visual comfort allow us to predict daylight-caused glare using
digital building models and climate data. Unfortunately, the simulation tools currently available cannot produce results fast
enough for interactive use during design ideation. We developed software with the ability to predict visual discomfort in real
time. However, we know little about how users react to simulation feedback presented in real time. In our study, 40 subjects
with backgrounds in building design and technology completed two shading design exercises to balance glare reduction
and annual daylight availability in two open office arrangements using two simulation tools with differing system response
times. Subjects with access to real-time simulation feedback tested more design options, reported higher confidence in design
performance and increased satisfaction with the design task, and produced better-performing final designs with respect to
spatial daylight autonomy and enhanced simplified daylight glare probability.
Keywords: daylighting; visual comfort; glare; simulation; real time; user behaviour; human–computer interaction; design

Introduction If building performance simulation can be made faster

Design firms increasingly seek to justify architectural deci-
sions with data. However, the simulations that produce
these data often run days or weeks after the design is
finalized. Most building performance simulations validate
completed architectural designs, and designers rarely run
the simulations themselves or see the direct output. In
the common use case, the user-friendliness and speed of
simulation tools are of secondary importance. How must
simulation tools themselves change in order to produce
results during the design process, rather than afterward?
What effect does the choice of simulation tool have on
design outcomes? To answer these questions, we investi-
gate how the design of simulation tools affects their use
and their users’ decisions.
Two factors that limit the expanded use of building per-
formance simulation tools are cumbersome interfaces and
feedback latency. Many simulation tools in common use
lack user interfaces that are intuitive or easy to learn. Pro-
fessional users expect a trade-off between simulation speed
and accuracy, although we have shown previously that
this need not be the case when new algorithms and paral-
lelization techniques are used (Jones and Reinhart 2017a).
These factors helped to create professional roles for build-
ing physics and lighting experts separate from the role of
designers. Unfortunately, this professional separation fur-
ther limits the ability of performance simulation to guide
early development of designs.
and more intuitive without losing accuracy, it may be pos-
sible to train a new generation of building designers to
use simulation as a design aid. In this paper, we investi-
gate the effect of providing simulation results in real time.
We compare two simulation tools for visual comfort based
on their effects on user behaviour, design performance,
and user satisfaction. We presented 40 subjects with two
design problems, asking them to reduce the annual like-
lihood of glare in office environments while maintaining
acceptable daylit illuminance values on the work plane.
Each subject analysed one space with DIVA-for-Rhino 4.0
(Jakubiec and Reinhart 2011), a commercial lighting anal-
ysis plugin for the Rhinoceros CAD environment (Robert
McNeel & Associates 2016), and one space with Acceler-
adRT, an experimental tool for physically based real-time
rendering. Both tools provided subjects with false colour
luminance maps representing occupant views and corre-
sponding daylight glare probabilities. However, DIVA-
for-Rhino required subjects to wait for feedback from
each viewpoint, while AcceleradRT provided continuous
feedback as subjects navigated the space.
From this study, we demonstrate a link between simu-
lation feedback latency and the behaviour, design choices,
and satisfaction of simulation users. To enable this anal-
ysis, we make several other contributions to the field of
daylighting and building performance simulation. First, we
present AcceleradRT, a novel real-time lighting simulation

*Corresponding author. Email: nljones@mit.edu

© 2018 International Building Performance Simulation Association (IBPSA).

344 N.L. Jones and C.F. Reinhart
tool developed as a design aid. Second, we present a
method for comparative usability testing of building per-
formance simulation tools. Third, we present a modified
visual comfort metric that accounts for annual, spatial,
and directional variation in glare. Finally, we examine the
role of user experience and discuss implications for tool
development and visual comfort metrics.
Background
Software tools have changed the nature of design think-
ing significantly in the last half century. Digitally based
design emphasizes specificity and detail over abstract rep-
resentation (Oxman 2006), and designers exhibit different
design strategies using CAD tools than when sketching,
according to think-aloud design studies (Salman, Laing,
and Conniff 2014). This may be in part because CAD (and
simulation tool) development has been driven more by
production needs rather than an effort to foster creativity
(Jonson 2005).
System response time
For simulation results to inform design decisions, they
must be available before those decisions are made. We are
referring to decisions made during active designing, not
those made in the boardroom after the fact, so informed
decisions require interactivity. System response time (SRT)
is the time a user waits after entering input until the system
begins to present results to the user (Doherty and Kelisky
1979). Studies conducted at IBM show that SRT affects
productivity, but not simply by adding SRT to the time
required for task completion. Instead, longer SRT seems
to “disrupt the thought process” and cause a longer user
response time as well (Doherty and Kelisky 1979). Reduc-
ing SRT increases productivity, job satisfaction, design
quality, and perceived power and decreases anxiety lev-
els (Doherty and Thadani 1982; Barber and Lucas 1983;
Guynes 1988; Hoxmeier and DiCesare 2000). From these
observations, Brady (1986) developed roll theory, which
states that given immediate access to organized data and
with concentration unbroken by distractions, “ideas and
solutions will suggest more ideas and solutions to succes-
sive steps of the creative process, in a rapid and orderly
flow.” When “on a roll,” an average user can exhibit higher
productivity than an expert user faced with high SRT
(Doherty and Thadani 1982).
Brady’s theory seems intrinsically related to Csíkszent-
mihályi’s concept of flow (1996). Flow is a focused mental
state in which tasks become automatic and effortless, often
referred to as being “in the zone.” Flow describes the cre-
ative mindset generally and is not restricted to digital work,
but its applicability to designers using digital media is
clear. To integrate simulation into the flow of the design
process, simulation results must be clearly organized and
immediately available.
High SRT affects user behaviour, but the latency that
users will accept depends on the speed of the user’s
though process. Types of thought and their time scales
can be classified as deliberate acts that carry expected
responses ( ∼ 100 ms), cognitive operations that prompt
unexpected responses ( ∼ 1 s), or unit tasks that involve
planning and strategy ( ∼ 10 s) (Newell 1994). Users notice
short delays in response to deliberate acts such as input
device manipulation. In one study, touchpad users detected
latency differences between 2.38 and 11.36 ms (Ng et al.
2012). However, users in another study often failed to
detect touchpad response latencies below 40 ms (Jota et al.
2013). In a first-person shooter game, introducing 75–
100 ms delays led to 50% fewer successful shots; players
found 100 ms latencies noticeable and 200 ms latencies
annoying (Beigbeder et al. 2004). Strategy games employ
a slower type of thinking based in cognitive operations
and unit tasks. Internet latencies up to several seconds
did not significantly affect performance in real-time strat-
egy games (Sheldon et al. 2003). However, even if high
SRT goes unnoticed, it can alter thought processes. In
a tile-moving game, subjects faced with higher latency
worked harder to develop strategies and moved fewer tiles
(O’Hara and Payne 1998, 1999). Similarly, increasing the
SRT of web searches causes users to submit fewer queries
(Brutlag 2009). Liu and Heer (2014) observe that reduced
SRT correlates to greater numbers of mouse movements
and application events, affecting both deliberate acts and
cognitive operations of users. Their study suggests that
SRT must be below 500 ms to support interactive user
behaviour.
Visual comfort
At its most basic, visual comfort can be defined as pro-
viding enough illumination without providing too much.
Visual discomfort occurs when light levels are too dim to
support a given task or when intense light sources and con-
trast cause glare. Concern for sustainability and connection
to the outdoors leads designers to meet as much of lighting
needs as possible through natural lighting, but sunlight is
also the most common cause of glare. Performance met-
rics for daylighting must therefore consider both daylight
availability and risk of glare.
Daylight availability metrics use illuminance values
measured or simulated over a grid of points. Annual sim-
ulations using climate data can give daylight illuminance
values for a grid of points for all hours of the year. Spatial
daylight autonomy refers to the fraction of occupied space
receiving at least 300 lux for at least 50% of occupied hours
and is abbreviated sDA300,50% (IESNA 2012). The LEED
(USGBC 2013) and WELL (International WELL Build-
ing Institute 2016) green building standards recognize
sDA300,50% as an indicator of good daylighting.
Some grid-based metrics also use illuminance lev-
els as a proxy for glare potential. Annual sun exposure,
 Journal of Building Performance Simulation
abbreviated ASE1000,250 , refers to the fraction of occupied
space receiving at least 1000 lux for at least 250 occupied
hours (IESNA 2012) and is similarly used by the LEED
and WELL standards. Useful Daylight Illuminance (UDI)
sets both lower and upper thresholds (100 and 2000 lux,
respectively) to encompass both daylight availability and
glare potential in a single metric (Nabil and Mardaljevic
2006). There are drawbacks to using illuminance as a proxy
for glare, however. Glare potential is not a measured like-
lihood of glare; rather, it is intuition that glare could occur,
and there is little evidence to support the upper thresholds
proposed by ASE1000,250 or UDI. Furthermore, it is diffi-
cult for designs to perform well in both sDA300,50% and
ASE1000,250 .
Glare metrics use luminance rather than illuminance
and consider the location, size, and brightness of individ-
ual glare sources. Discomfort glare occurs when bright
light sources in the field of view cause visual irrita-
tion or eyestrain. Increasing the source’s size or lumi-
nance can lead to loss of contrast in the retinal image,
resulting in disability glare (Hopkinson 1972). Daylight
glare probability (DGP) describes the likelihood that an
occupant will report the presence of glare in a given
view (Wienold and Christoffersen 2006). We calculate
DGP as:
DGP = 5.87 × 10−5 Ev + 0.0918
 
 n
L2s,i ωs,i
× log10 1 + + 0.16, (1)
i=1
Ev1.87 Pi2
where Ev is the scene vertical eye illuminance in lux,
Ls,i and ωs,i are the luminance and solid angle of the ith
source, and Pi is the Guth position index representing the
eye’s sensitivity to the source direction. In human subject
studies, values greater than 45% correspond to intolerable
glare, while those under 35% predict imperceptible glare
(Wienold 2009a).
DGP prediction requires the rendering of an image,
usually by physically based ray tracing, for every anal-
ysed time, position, and view direction, leading to very
high simulation SRT. However, glare sources (typically
defined as regions of the image with luminance at least five
times the average scene luminance) tend to represent direct
views to light sources, so the luminance terms of Equation
1 can be calculated through a faster zero-ambient-bounce
luminance simulation. The enhanced simplified DGP met-
ric (eDGPs) combines a zero-ambient-bounce luminance
simulation with Ev calculated by lower-resolution illumi-
nance simulation to approximate DGP with high accuracy
(relative mean bias error under 2% and relative root-mean-
square error under 5%) (Wienold 2009b).
Although DGP and eDGPs evaluate glare at a spe-
cific time, location, and viewing direction, they may be
extended to consider multiple times and locations. Wienold
(2009b) proposes that situations exceeding certain DGP
345
thresholds be limited to 5% of occupied hours at any
point. A recent European standard codifies this requirement
(CEN 2016). However, Wienold does not specify what por-
tion of the floor area needs to be evaluated, leaving it up to
the designer to choose critical locations and view direc-
tions for analysis. Kleindienst and Andersen’s (2012) glare
avoidance extent represents glare occurrence spatially and
can be displayed graphically as a temporal map. To reduce
computational complexity, they divide the year into a sta-
tistically representative sample of hours for simulation and
approximate DGP using illuminance. DIVA-for-Rhino cre-
ates temporal maps of eDGPs but provides no quantitative
metric to compare these visualizations (Jakubiec and Rein-
hart 2011). None of these methods quantify the annual,
spatial, and directional glare occurrence together in a single
metric.
Contrast on a computer screen or paper also plays a role
in visual comfort. For legibility, we describe the minimum
luminance difference between text and its background by
the contrast ratio CRmin :
CRmin = 2.2 + 4.84 × LL −0.65 , (2)
where LL is the lesser of the two luminance values (ISO
2008). Within a typical luminance range, this ratio is usu-
ally around 3:1 (ISO 1992). Veiling glare occurs when
reflected light, such as sunlight falling on a computer
screen, obscures text or images seen through the reflective
surface. Based on our approximation, a typical monitor that
emits 100 cd/m2 will exhibit veiling glare when it reflects
at least 50 cd/m2 .
Various standards and rating systems that address
visual discomfort emphasize adequate daylighting (IESNA
2012), views to the outdoors (USGBC 2013), circadian
entrainment (International WELL Building Institute 2016;
Konis 2017), visual interest (Amundadottir et al. 2017),
or glare (Hopkinson 1972; Einhorn 1979; CIE 1995; Naz-
zal 2001; Harrold 2003; Wienold and Christoffersen 2006).
To complicate matters, these phenomena are ephemeral
and occupant-centric, so lighting designers must consider
how each aspect of visual comfort will change with time,
occupant position, and viewing direction (Jakubiec and
Reinhart 2012; Amundadottir et al. 2017; Konis 2017;
Jones and Reinhart 2017a). Design for visual comfort is an
open-ended problem that in many cases requires the use of
simulation. The choice of simulation tool may produce dif-
ferent performance results (Bellia, Pedace, and Fragliasso
2015) and elicit different user behaviour (Liu and Heer
2014). It therefore plays a critical role in the success of
design outcomes.
Tools
The SRT of CAD tools has decreased substantially since
the early days of computer graphics and is now short
enough to avoid disrupting creative flow. However, many
346 N.L. Jones and C.F. Reinhart
building performance simulation tools have long SRTs of
minutes or hours. Excluding expert intuition and scaled
physical models, all methods for predicting visual dis-
comfort of which we are aware use physically based ray-
tracing simulations (Whitted 1980), which generally have
a high SRT.
For the past 20 years, RADIANCE has stood out as the
most validated simulation software for physically based
ray traced rendering and irradiance calculation in building
design (Larson and Shakespeare 1998). Previous studies
have validated its ability to predict daylit sensor readings
in test settings (Ubbelohde and Humann 1998; Reinhart
and Herkel 2000; Reinhart and Walkenhorst 2001; Rein-
hart and Andersen 2006; Reinhart and Breton 2009; Du
and Sharples 2011; Yoon, Moon, and Kim 2016) and
building atria (Grynberg 1989; Ng et al. 2001; Galasiu
and Atif 2002). Our own analysis shows that RADIANCE
accurately predicts periods of visual discomfort based
on DGP and contrast ratio (Jones and Reinhart 2017a).
DIVA-for-Rhino is a plugin for Rhinoceros CAD software
that enables many types of RADIANCE analysis, includ-
ing – relevant to this study – luminance maps and DGP
calculations.
Accelerad is a RADIANCE derivative created by the
authors that achieves faster computational performance
by running simulations in parallel on the graphics pro-
cessing unit (GPU) instead of using a single central
processing unit (CPU) thread (Jones and Reinhart 2014;
Jones and Reinhart 2015; Jones and Reinhart 2017a).
This strategy allows hours-long RADIANCE simulations
to run in minutes through the OptiX™ ray-tracing engine
(Parker et al. 2010). AcceleradRT modifies RADIANCE
still further by using progressive path tracing (Lafor-
tune and Willems 1993) instead of light-backwards dis-
tribution ray tracing (Cook, Porter, and Carpenter 1984).
Progressive path tracing makes images at intermediate
stages of rendering immediately available, reducing SRT
to fractions of a second. We have previously demon-
strated that progressive path tracing produces coherence
in contrast ratio and DGP calculations over time, and
that these results may be used with some degree of cer-
tainty starting early in the rendering process (Jones and
Reinhart 2016).
Experiment setup
We recruited 40 subjects aged 21–37 to take part in our
study. Twenty-five subjects reported previous professional
experience in architecture, ranging up to 13 years with
a median of 4 years’ experience. The remaining subjects
were engaged in graduate-level architecture coursework or
design research. Seventeen subjects had educational back-
grounds in architecture, 12 in building technology, 7 in
both, and 4 in related engineering disciplines. All but two
reported normal colour vision.
Design task
We gave each subject two design problems in which they
had to modify a space to mitigate glare issues while main-
taining acceptable daylight levels. Each problem involved
a Rhinoceros model of a 19 m × 10.9 m open-plan cor-
ner office space with floor-to-ceiling windows on two
sides (Figure 1(a,b)). Subjects had 20 minutes to com-
plete each problem. For each space, we asked subjects to
identify regions where occupants could experience glare
by marking on a printed floor plan, and then to choose a
combination of shading devices stored in predefined lay-
ers of the Rhinoceros model that would best mitigate glare
(Figure 1(c,d)). The two models had different climates
(Minneapolis and Albuquerque), different orientations, dif-
ferent workspace layouts, and different shading devices, so
subjects could not use the same strategy for both models.
To assist their analysis and decision-making, each partici-
pant had access to DIVA-for-Rhino for one design problem
and to AcceleradRT for the other. We randomly chose
which model and which tool each subject experienced first
to ensure an equal number of subjects in each condition.
Additionally, we sat subjects at computers of two dif-
ferent configurations. Half of the subjects used machines
with faster 2.60 GHz Intel® Xeon® E5-2604 processors
with 3.4 GHz overclocking, and half used machines with
slower 2.40 GHz Intel® Xeon® E5620 processors with
2.66 GHz overclocking. Each machine had an NVIDIA®
Tesla® K40 graphics accelerator with 2880 CUDA® cores
for AcceleradRT’s GPU-based computations.
Prior to starting each design problem, we gave each
subject simple training in glare analysis and use of the
tools. We instructed subjects to consider DGP, veiling
glare, and work plane illuminance in their designs. As
simple rules of thumb, we asked subjects to keep DGP
below 35%, reflections on monitors below 50 cd/m2 , and
reflected luminance on work surfaces above 48 cd/m2 ,
which roughly corresponds to 300 lux falling on a 50%
reflective Lambertian surface. Subjects could test the lat-
ter two criteria only through the false colour rendering
provided by each simulation tool (Figures 2 and 3).
After each design problem, subjects completed a sur-
vey. The survey assessed subjects’ confidence and satis-
faction while performing the task. It also asked subjects to
describe their approach to the design problems and their
impressions of the tools. We also asked each subject to
record the combination of shading devices they chose to
mitigate glare issues.
Tool design
We created a customized workflow for each tool that gave
subjects access to four input parameters: layer state, date
and time, view, and simulation initiation. Subjects added
or removed shading devices from the Rhinoceros model
using the layer panel within the main Rhinoceros window
 Journal of Building Performance Simulation 347

Figure 1. Floor plans of the two models used for the (a) Minneapolis and (b) Albuquerque climates and (c,d) the shading device options
for each.

(Figure 4). We provided month, date, and hour controls the Rhinoceros model to serve as an avatar to choose posi-
within a Grasshopper visual programming interface for tion and viewing direction. Subjects could move and rotate
both tools. For DIVA-for-Rhino, we provided an object in the avatar using Rhinoceros’ gumball navigation (shown in
348 N.L. Jones and C.F. Reinhart
Figure 2. The AcceleradRT user interface includes a luminance map image that supports mouse navigation and a DGP dial widget.
Subjects control date and time through Grasshopper.
Figure 4). We chose this method because preliminary trials
showed that subjects had difficulty placing the virtual cam-
era if they could not see its location. Because AcceleradRT
features an animated progressive rendering rather than a
static image, users can navigate the scene with mouse
gestures over the image. We used Grasshopper to place
an avatar in the Rhinoceros viewport to show the current
view, but subjects controlled the position and view direc-
tion through AcceleradRT rather than Rhinoceros. In both
instances, the avatars held the viewpoint at 1.3 m above the
floor, corresponding to a typical seated eye level. Finally,
we provided a Grasshopper button to start the simulation
process in DIVA-for-Rhino. After setting the desired layer
state, time, and view, subjects could click the button to ini-
tiate a simulation using DIVA-for-Rhino’s lowest quality
setting. AcceleradRT performs simulations continuously,
so its interface offered no equivalent feature.
Both tools provide graphical output. We set up both
tools to display the avatar’s current view in 180° equian-
gular fisheye projection using a logarithmic false colour
scale from 10 to 10,000 cd/m2 . This display allowed sub-
jects to estimate the brightness of spots such as monitors
and work surfaces in the model. In DIVA-for-Rhino, we
displayed the DGP value in a prominent text box. Acceler-
adRT displays DGP using a dial widget with green and red
colouring to represent acceptable and unacceptable values,
respectively (shown in Figure 2).
We used a custom Grasshopper component to log each
subject’s layer state, time, view, and simulation interac-
tions for both tools. A timestamp associated with each
interaction allowed us to record the length of time the spent
in each combination of layer state, time, and view, and for
DIVA-for-Rhino the elapsed time during simulation runs.
Results
Forty subjects collectively give us 800 minutes of recorded
user interaction for each tool. We analyse these data in
three ways. First, we examine the user behaviours, pro-
cesses, and strategies that subjects employed to see how
they differed depending on the tool used. Second, we exam-
ine the proposed designs from each subject to see if the
choice of tool affected design performance. Finally, we
examine the subjects’ survey responses to see if subjects
developed preferences for the tools.
User behaviours and design strategies
We are interested in the amount of information that sub-
jects accumulated while making their design decisions, and
 Journal of Building Performance Simulation
Figure 3. The DIVA-for-Rhino user interface includes a static luminance map image and a text box reporting DGP within Grasshopper.
Viewpoint navigation happens in Rhinoceros.
therefore we want to know how many views, times, and
layers states subjects explored (Table 1). However, setting
the desired configuration of view, time, or layer state might
involve multiple interactions to set the month, date, hour,
x and y positions, view rotation angle, or visibility states
of multiple layers. Rather than counting the total number
of interactions, we count only those that preceded suffi-
ciently long pauses to indicate that the subject might have
thought about the result. Using DIVA-for-Rhino, simula-
tions took on average 20 s using the faster computers and
29 s using the slower computers, so we can assume that
a series of interactions within a shorter timeframe form a
unit task such as setting up a time and viewpoint for the
next simulation. Using AcceleradRT, where preliminary
results are available in real time, subjects might reject an
idea after rendering only a few frames if the results did not
look promising. Each frame takes approximately 200 ms
to render, so a series of interactions over a period of sev-
eral seconds might represent multiple cognitive operations
as the user queries more information about the scene. In
2 s, AcceleradRT renders about 10 frames, by which point
global metrics such as DGP tend to stabilize in the ani-
mation (Jones and Reinhart 2016). We choose 2 s as the
minimum duration below which the subjects were unlikely
to gain information about a design.
349
Figure 5 shows the fraction of states (combination of
view, time, and layer state) that subjects viewed for dura-
tions up to one minute. Using AcceleradRT, half of those
states were active for no more than 2 s, and only 3.5%
were active for at least 20 s. Using DIVA-for-Rhino, 80%
of the states that subjects observed remained active for
at least 2 s, and 40% remained active for at least 20 s.
This indicates a change in the cognitive state of subjects
when using DIVA-for-Rhino. One possible explanation is
that the slower tool induced slower thought processes, so
that actions that would have been fast cognitive operations
became slower unit tasks. Another possibility is that sub-
jects relied more on strategy to cope with the increased
SRT and therefore performed fewer cognitive operations.
Different user strategies are apparent when plotting the
views visited by each subject. In Figure 6, the thickness
of a vector is proportional to the length of time the sub-
ject spent in that view, and colours indicate DGP values
for views where the subject ran a simulation in DIVA-for-
Rhino. Some common strategies were to concentrate views
near windows (Figure 6(a,b)) or in the brightest corner
(Figure 6(c,d)), visit all workstations in the space (Figure
6(e,f)), or concentrate on a small number of views in the
centre of the room (Figure 6(g,h)). These patterns are often
clearer in the DIVA-for-Rhino sessions, where the subjects
350 N.L. Jones and C.F. Reinhart

Figure 4. The Rhinoceros interface used with both tools includes the layer controls (right) and avatar, shown here with the gumball
navigation control used in DIVA-for-Rhino.

Table 1. Average number of interactions followed by 2-s

delays by type.

Interaction DIVA-for-
type AcceleradRT Rhino Mean ratio

Time 17.8 11.2 1.73

Viewpoint 93.6 26.9 6.06
Layer state 45.6 17.1 7.98
Simulation N/A 23.3 N/A

had more precise control over placement of their avatars.

Subjects were able to visit more of the space using Accel-
eradRT and visited on average six times as many views
for at least 2 s (Table 1). Aggregating the views visited
by all of the subjects shows that subjects collectively cov-
ered a greater portion of the space with their analysis using
AcceleradRT than using DIVA-for-Rhino (Figure 7).
Subjects using AcceleradRT also explored a greater
variety of times during the year than those using DIVA-
for-Rhino. Figure 8 shows the hours of the year studied
by subjects, with darkness indicating the cumulative time
that subjects spent examining each hour. Subjects tended
to focus on the first day of each month or on solstice and
equinox days. Using AcceleradRT, subjects were likely to
examine times earlier or later in the day than when using
DIVA-for-Rhino. The average subject studied 1.7 times the
Figure 5. Subjects spent less time on average examining each
combination of view, time, and layer state in AcceleradRT than
in DIVA-for-Rhino.
number of times for at least two seconds using Acceler-
adRT than they did using DIVA-for-Rhino (Table 1).
The greatest effect occurred for the number of layer
states that subjects investigated with the two tools. On
average, subjects viewed eight times as many layer states
for at least 2 s using AcceleradRT than using DIVA-for-
Rhino. Our method for counting layer states considers
 Journal of Building Performance Simulation 351

Figure 6. Sample traces show different strategies that subjects used to explore the space using AcceleradRT (left column) and DIVA–
for-Rhino (right column). Colours indicate DIVA-for-Rhino simulations with imperceptible glare (green), perceptible or disturbing glare
(yellow), and intolerable glare (red).

specifically the number of state changes, so some subjects
may have alternated between a smaller number of layer
states more often in order to compare them using Accel-
eradRT. The order in which subjects experienced the two
tools played a factor. Those who used DIVA-for-Rhino first
tried on average thirteen times as many later states when
they later used AcceleradRT, while those who used Accel-
eradRT first tried on average only three times more layer
states than they did later with DIVA-for-Rhino (Table 1).
We reason that early exposure to the faster tool conditioned
352 N.L. Jones and C.F. Reinhart

Figure 7. Subjects explored a greater portion of the spaces using AcceleradRT. The darkness of view symbols corresponds to the
cumulative time subjects spent analysing the view in that region and direction.

subjects to explore the design space more thoroughly. We
also note that some subjects spent the majority of their time
analysing the base case; for both tools, 10% ran out of time
before modifying any layer states.
A temporal view of user behaviour offers another win-
dow into the strategies that subjects employed. In Figure 9,
coloured dots represent different types of subject interac-
tions. Most subjects alternated between spatial exploration
and changing the layer state, with infrequent changes to
the time. This may be because the time controls required
keyboard interaction and therefore seemed to involve more
effort. Using DIVA-for-Rhino, most subjects adopted a
pattern of changing either view or layer state followed
by running a simulation, and the frequency of interaction
stayed relatively constant for each subject. Using Accel-
eradRT, subjects displayed a much higher frequency of
interaction, with many views and layer states observed for
only a matter of seconds. A quarter of subjects using Accel-
eradRT entered a prolonged period of inactivity ranging
from one to five minutes, usually towards the end of the 20-
minute session. This inactivity may indicate the need for a
break, attention turning to the printed floor plans instead
of the screen, or early completion of the task. We observe
periods of inactivity using DIVA-for-Rhino as well, but
they are generally shorter and occur earlier in the sessions.
This may indicate hesitance to start the task due to the less
intuitive interface.
With DIVA-for-Rhino only, we can examine the effect
of processor speed on user behaviour. Subjects using faster
computers examined 50% more views and 30% more layer
states, and ran 37% more simulations than subjects using
the slower computers. Subjects using faster computers also
spent less time reviewing results or changing the model
between simulations; there was almost no difference in
the fraction of time taken up by simulations between the
two groups (45% for subjects with fast computers versus
48% for subjects with slow computers). This may indi-
cate that subjects who had to wait longer for simulation
results became distracted more often, valued the time in
which they could interact with the models more, or were
otherwise less inclined to run simulations.
Design performance
We define high-performing designs as those that maximize
daylight availability while minimizing the risk of glare. In
general, a trade-off exists between these two goals, and
no single design will simultaneously provide the greatest
 Journal of Building Performance Simulation 353

Figure 8. Subjects tended to view the spaces on the first of the month or on solstice and equinox days. The darkness of each hour on the
temporal maps corresponds to the cumulative time subjects spent analysing that hour.

daylit area and the least glare risk. Instead, we define opti-
mal designs as those that lie on the Pareto frontier of these
two criteria; a design is optimal if and only if no other
design performs better in one respect without performing
worse in the other. Based on the instructions we gave to
subjects, we expected them to seek out Pareto optimal
designs.
To judge the performance of the designs that subjects
proposed, we simulated each model with each possible
combination of shading devices and calculated daylight
availability and glare risk. For each metric, we divided the
space according to a 0.5-m square grid of 819 sensors. To
assess daylight availability, we computed spatial daylight
autonomy (sDA300,50% ) at a work plane elevation of 78 cm.
354 N.L. Jones and C.F. Reinhart

Figure 9. Timelines of subject interactions with the simulation tools show greater interaction with AcceleradRT and more regularly
spaced interaction with DIVA-for-Rhino.

To assess the risk of glare, we computed enhanced sim-
plified daylight glare probability (eDGPs) in eight equally
spaced directions around each sensor at a seated head
height of 130 cm for all occupied hours using climate-
based skies. We report the fraction eDGPs values across
all occupied hours, positions, and directions that exceeded
35%. We refer to this fraction as spatial annual eDGPs
(saeDGPs).
Figure 10 shows the sDA300,50% and saeDGPs for each
unique combination of shading devices (320 for Min-
neapolis and 144 for Albuquerque). Both metrics tend to
increase as the unobstructed view to the sky increases,
resulting in a Pareto frontier of designs that maintain a
high sDA300,50% while minimizing saeDGPs. We highlight
the final design solutions chosen by subjects from among
all other available design choices: shape indicates the tool
used by the subject choosing that design, and colour indi-
cates the number of subjects who chose the design using
that tool. Subjects using AcceleradRT were more likely
to choose final designs with Pareto optimal combinations
of sDA300,50% and saeDGPs than subjects using DIVA-
for-Rhino; of subjects who selected final designs, 61%
 Journal of Building Performance Simulation 355

Figure 10. Shading designs chosen using AcceleradRT were more likely to lie on the Pareto frontier (high spatial daylight autonomy
(sDA300,50% ) and low spatial annual enhanced simplified daylight glare probability (saeDGPs)) than those chosen using DIVA-for-Rhino.

Figure 11. The most popular design for Minneapolis, chosen by 11 subjects, used slanted louvres to shade the top two-thirds of the
glazing.

of AcceleradRT users chose optimal designs, while only
39% of DIVA-for-Rhino users did. We did not instruct
subjects on how to value the trade-off between sufficient
work plane illuminance and glare, and the responses show
preferences at both ends of the spectrum and in the mid-
dle. In the west-facing Albuquerque model, subjects using
AcceleradRT preferred lower saeDGPs than those using
DIVA-for-Rhino, but the same trend was not apparent in
the south-facing Minneapolis model. One design option for
Minneapolis proved particularly popular and was chosen
by over a quarter of subjects, including seven using Accel-
eradRT and four using DIVA-for-Rhino (Figure 11). For
comparison, only three subjects using AcceleradRT and
one using DIVA-for-Rhino chose the most popular design
option for Albuquerque (Figure 12). Both of these popular
designs are Pareto optimal.
User evaluations
Clear trends are evident in the subjects’ own evaluations
of their work using the two tools. After using each tool
for 20 minutes, we asked subjects to evaluate their experi-
ence by answering 12 questions, each on a 7-point Likert
scale. These questions dealt with subjects’ confidence in
their own work and psychological state during the task
(Figure 13).
Subjects were more confident when they used Accel-
eradRT than when they used DIVA-for-Rhino. More than
half of subjects rated their confidence higher when it came
both to their assessment of the space prior to their interven-
tion and to the performance of their final design. This was
the case even though 55% of subjects ranked their famil-
iarity with DIVA-for-Rhino higher, and a plurality of 38%
trusted the two tools to predict glare equally well. Subjects
who used AcceleradRT first were likely to rate themselves
more confident when using AcceleradRT by a higher mar-
gin. Subjects who used the slower computers were likely
to put more trust in AcceleradRT by a higher margin.
Most subjects found the task using AcceleradRT to be
more enjoyable, more relaxing, less difficult, less frustrat-
ing, and less hurried than the task using DIVA-for-Rhino.
More than half of the subjects also reported learning more
356 N.L. Jones and C.F. Reinhart
Figure 12. The most popular design for Albuquerque, chosen by four subjects, used a light shelf and slanted louvre shading on the
middle potion of the glazing.
Figure 13. Subject responses to survey.
from the task using AcceleradRT. The order in which sub-
jects used the tools mattered. Subjects who had already
completed the task using AcceleradRT before they tried
DIVA-for-Rhino were likely to rate AcceleradRT even
more enjoyable while rating DIVA-for-Rhino more diffi-
cult, frustrating, and hurried by comparison. Relaxation
was not significantly affected by tool order, but subjects
who used the slower computers did tend to rate Acceler-
adRT more relaxing by a wider margin than those who used
the faster machines.
Finally, we asked subjects which tools they preferred
overall. In all, 90% of subjects preferred AcceleradRT for
glare prediction, while 10% preferred DIVA-for-Rhino.
As reasons for preferring AcceleradRT, subjects cited its
ease of use, interactivity, and smooth workflow. Subjects
appreciated the ability to see immediately when a design
would not work, to examine more views in order test
their designs, and to make changes “on the fly.” Some
felt that their time spent using DIVA-for-Rhino was less
productive because they had to wait for each simula-
tion to complete. Regarding the interface, several subjects
found navigation difficult in AcceleradRT, but they found
the dial widget display of DGP easier to read. The four
subjects who preferred DIVA-for-Rhino tended to cite
AcceleradRT’s progressive rendering as a reason. Some
felt that it was distracting, less precise, or inefficient com-
pared to static renderings. One subject, apparently unaware
that AcceleradRT was not in public release at the time,
cited DIVA-for-Rhino’s larger market share as a reason to
prefer it.
Discussion
In this study, we found that that changes in SRT cor-
relate with differences in user behaviour, user confi-
dence, user satisfaction, and optimality of design choices.
In this section, we look for a mechanism to explain
these correlations. Based on our analysis, we then make
 Journal of Building Performance Simulation 357

recommendations for the development of building perfor-

mance simulation tools and metrics.

Explaining the connection

We would like to explain why subjects produced opti-
mal solutions more often when they had access to a tool
with low SRT. An obvious hypothesis is that the faster
tool allowed them to explore more options in the allotted
time and therefore gather more knowledge about the design
space. Could increased exploration be a predictor of opti-
mal final designs? This turns out not to be the case. In fact,
within the tests of each tool, users who chose non-optimal
designs viewed just as many (or more) options as users who Figure 15. Subjects with at least three years of professional
chose designs on the Pareto frontier (Figure 14). architectural experience were much more likely to choose opti-
If exploration does not account for differences in design mal designs using AcceleradRT but showed no advantage using
DIVA-for-Rhino.
performance, then perhaps some subjects’ previous expe-
rience helped them in the task. Our second hypothesis
is that varying levels of architectural experience among
the participants affected their ability to choose optimal
designs. To test this, we divided subjects into four roughly
equal bins according to their self-reported length of pro-
fessional experience. Experience aided users in low SRT
situations. Subjects with three or more years’ professional
architectural experience chose optimal final designs using
AcceleradRT twice as frequently as their less experi-
enced counterparts (Figure 15). Subjects with backgrounds
in architecture were also more likely to choose optimal
designs using AcceleradRT, regardless of their level of
experience (Figure 16). However, when faced with high
SRT, subjects with professional experience or architec- Figure 16. Subjects with educational backgrounds in architec-
tural training did no better that inexperienced subjects at ture were more likely to choose optimal designs using Acceler-
adRT but showed no advantage using DIVA-for-Rhino.
choosing optimal final designs. It is worth noting that
inexperienced subjects still reported feeling more confi-
dent when using AcceleradRT, even though low SRT did We cannot attribute the performance of designs cho-
not make them significantly more likely to select optimal sen by the subjects solely to exploration or experience,
final designs. so instead we examine how experience related to the tool
used. Our third hypothesis is that subjects were more
likely to choose optimal final designs when their combina-
tion of experience and activity resulted in flow. Based on
interviews with recognized experts in creative fields, Csik-
szentmihalyi (1996) characterizes flow with nine traits. We
examine these traits in the context of our study:

1. Clear goals at every step. Subjects received clear

Figure 14. The average number of states (combination of view,
time, and layer state) viewed by subjects for at least 2 s has little
relationship to whether subjects chose optimal final designs.
instruction prior to completing the tasks. How-
ever, the design of graphic user interfaces could
affect subjects’ ease of recalling individual steps.
In particular, the AcceleradRT window contained
only the animated rendering, colour scale bar,
and DGP dial widget. This design emphasized
a process and a goal – view the rendering and
manipulate it to reduce DGP to an acceptable
value. This streamlined process is not evident in
DIVA-for-Rhino’s Grasshopper interface, where
358 N.L. Jones and C.F. Reinhart
the relationship between components is repre-
sented abstractly, the rendered view does not exist
until the first simulation finishes and then does not
update immediately to reflect user interactions, and
DGP is displayed as text.
2. Immediate feedback to one’s actions. AcceleradRT
produced renderings at five frames per second
and converged DGP results within 2 s, giving the
impression of immediate feedback. In contrast,
subjects using DIVA-for-Rhino had to wait 20–29 s
for results from each simulation.
3. Balance between challenges and skills. Csíkszent-
mihályi observes flow to occur when both the chal-
lenge of the task and skill level of the subject are
high. In our study, the task was challenging. Our
observation that subjects with past professional
experience in architecture chose optimal designs
more frequently suggests that their skill level was
better matched to the problem as presented in
AcceleradRT. On the other hand, DIVA-for-Rhino
made the task more difficult, such that these sub-
jects’ skill levels were no longer well matched
to it.
4. Actions and awareness are merged. This merger
describes deliberate acts – the type of thought asso-
ciated with gestures and mouse interaction charac-
terized by speeds on the order of 100 ms (Newell
1994). In our study, navigation through the mod-
elled offices using the mouse is the primary deliber-
ate act. AcceleradRT’s progressive rendering tied
subjects’ awareness of daylighting performance to
the act of navigating the space.
5. Distractions are excluded from consciousness.
More than half of subjects reported feeling more
distracted when using DIVA-for-Rhino than when
using AcceleradRT. However, the feeling of dis-
traction was heavily swayed by tool order; subjects
tended to report feeling more distracted during the
second half of the experiment. In their comments,
some subjects explained that distraction resulted
from finishing the task early.
6. No worry of failure. More than half of subjects
reported feeling more confident and more relaxed
when using AcceleradRT than when using DIVA-
for-Rhino.
7. Self-consciousness disappears. Although this is
similar to five above, we do not have a way to test
it directly.
8. Sense of time becomes distorted. A plurality (48%)
of subjects felt that time passed more quickly while
using DIVA-for-Rhino. However, like distraction,
many attributed this to finishing early with the
other tool. It is therefore possible that responses to
this question do not accurately represent the feeling
of flow while engaged in the task itself.
9. Activity becomes autotelic. Csíkszentmihályi pro-
poses that individuals who experience flow seek it
out for its own sake. We cannot test this directly as
it would require participants to continue the exper-
iment task of their own volition beyond the allot-
ted time. However, a majority (63%) of subjects
expressed finding the activity more enjoyable with
AcceleradRT than with DIVA-for-Rhino, so we
expect they would choose AcceleradRT if given
reason to continue the task.
AcceleradRT aligns more closely with the closely with
the characteristics of flow than DIVA-for-Rhino, and in
general, low SRT tools seem better positioned to encourage
flow. The balance between challenge and skill best explains
the link we observe between experience and choosing opti-
mal designs. We conclude that experienced subjects were
better equipped to use the information they got through
design exploration and thus entered a creative state of flow
more easily.
Recommendations for tools
We wish to emphasize one take-away from our discus-
sion of flow: the low SRT and simplified user interface of
AcceleradRT appeared to reduce the skill level necessary
to balance the challenge of the task. In effect, flow lowered
the barrier to entry for performing the task. To promote
competent use of building performance simulation tools,
developers should incorporate principles of flow into their
products. Tools need interactive speeds and intuitive user
interfaces.
To promote design exploration through cognitive oper-
ations rather than less interactive unit tasks, preliminary
simulation feedback should be available within 500 ms
(Liu and Heer 2014). This is far less than the 20–29 s
that DIVA-for-Rhino takes to render at its lowest quality
settings. AcceleradRT provides preliminary results after
rendering its first frame in approximately 200 ms, so it
meets our definition of an interactive tool, and it provides
converged DGP results within 2 s, which is approximately
the SRT necessary for feedback to cognitive operations.
However, our study method is not fine grained enough to
reveal what type of thinking subjects engaged in between
interactions or at what point they began to wait. Some sub-
jects who preferred AcceleradRT mentioned the ability to
dismiss a bad design idea based on an incomplete simu-
lation result as a benefit, while a subject who preferred
DIVA-for-Rhino called having to wait for convergence in
the progressive rendering a drawback. These comments
suggest that even a 2-s SRT can interrupt flow. Future stud-
ies should employ more fine-grained approaches such as
think-aloud methods to learn how simulation tools affect
users’ thought processes (Liu and Heer 2014; Salman,
Laing, and Conniff 2014).
 Journal of Building Performance Simulation
Building performance simulation tools need intuitive
user interfaces. AcceleradRT was successful in this regard;
while none of the subjects had any previous experience
with AcceleradRT, 28% rated themselves somewhat or
very familiar with it after only 20 minutes of exposure.
The ease with which subjects became familiar with the new
tool may be a result of its simplified interface in which visi-
ble components corresponded directly to the task they were
designed for.
Recommendations for metrics
We found that experienced subjects tended to achieve
designs optimized for minimal annual glare risk with max-
imal daylight autonomy, even though the subjects had no
access to gridded readings or annual simulation results.
Daylighting metrics must account for spatial and tempo-
ral variance of visual discomfort. Proxy metrics for glare
based on illumination only, such as ASE1000,250 , UDI, or
even simplified DGPs (Wienold 2009b), do not adequately
measure occupant experience of glare. The final designs
chosen by subjects in our study did not tend to fall on
Pareto frontiers involving any of these metrics, suggesting
that these metrics do not correspond well to the occupant-
centric views seen by our subjects (Figure 17). Indeed,
subjects were no more likely to choose designs on the
Pareto frontiers with respect to ASE1000,250 or UDI than if
they had chosen designs at random (indicated by horizon-
tal lines in Figure 17). Furthermore, we found that designs
with spatial UDI averages above the recommended 80%
(Mardaljevic 2015) often had high risk for glare.
Comparisons across a large design space require that
we condense visual comfort data. For this study, we devel-
oped a climate-based daylighting metric using eDGPs. For
each position and view direction, we calculated eDGPs at
all occupied hours and report the mean fraction of times
when eDGPs exceeded 35%. Our method is similar to
that proposed by Wienold (2009b) and adopted in the new
European standard (CEN 2016), except that it considers
Figure 17. Subjects chose designs with optimal combinations of
sDA300,50% and saeDGPs more often than optimal combinations
of sDA300,50% and any other proxy metric for glare.
359
multiple positions and view directions and does not impose
a 5% limit on time exceeding the eDGPs threshold. We did
not instruct subjects to consider an annual limit, nor did
using point-in-time simulations seem to prompt them to do
so (Figure 17). Furthermore, we are not aware of any sci-
entific basis for the 5% limit. Ultimately, glare mappings
that represent glare probability experienced by occupants
across space and time in a single graphic representation
may provide the most intuitive understanding of visual
discomfort. However, such simulations remain outside the
realm of practicality for large parametric spaces. Even
using the dctimestep programme with input from Accel-
erad’s GPU-accelerated rcontrib programme (Jones and
Reinhart 2017b) to calculate eDGPs, predicting the glare
probabilities in all 464 design variants of the two models
took days and generated over a terabyte of data. Computing
results at this level of detail as part of an iterative design
process would surely interrupt flow.
Conclusion
Forty subjects completed two design problems involving
the visual comfort of office spaces using two simulation
tools. The two tools differed in SRT; AcceleradRT pro-
vided immediate feedback with progressive refinement,
while DIVA-for-Rhino required 20–29 s to produce a
result. Subjects demonstrated differences in user behaviour,
confidence, satisfaction, and design performance in terms
of daylight autonomy and glare depending on which tool
they used. Low SRT correlated with more exploration
of the space, higher confidence in design performance,
increased satisfaction with the design task, and more final
designs that fell on the design space’s Pareto frontier with
respect to sDA300,50% and saeDGPs.
These results should provide guidance for creators
of performance simulation tools. Tools that present clear
goals and can be embedded in existing iterative design pro-
cesses have a lower barrier to entry than those that interrupt
flow and require the user to wait for results. Designers can
better mediate conflicting goals, such as maximizing day-
light availability and eliminating glare, when they are able
to view a scene from an occupant’s vantage point in real
time. Human subjects testing is an effective way to deter-
mine how well different simulation tools can guide their
users. To measure the performance of design options pro-
posed by the study’s subjects, we developed a metric that
considers eDGPs across all occupied hours, viewpoints,
and directions. Considering annual spatial data for a para-
metric design space quickly becomes a big data problem.
Future tools and metrics must therefore be sound in their
representation of human perception and efficient in their
use of computational resources.
We cannot understate the importance of human inter-
action with building performance simulation tools. The
choice of simulation tool affects the choices made by its
user, even when different simulation engines purport to
360 N.L. Jones and C.F. Reinhart
show the same output. Currently, building performance
simulation is primarily used to validate completed designs,
when any problem it uncovers will be expensive to correct.
However, far-reaching changes can be implemented at lit-
tle cost when data are available early in the design process.
If performance simulation is to inform design decisions as
they are made, it must offer interactive results through an
intuitive interface so that it does not interrupt flow.
Acknowledgements
The Tesla K40 accelerators used for this research were donated by
the NVIDIA Corporation. Jon Sargent developed the Grasshop-
per components that allowed interactivity between Rhinoceros
and AcceleradRT. The authors thank Philip Thompson for his
technical support of the user study and Les Norford for providing
space.
ORCID
Nathaniel Jones http://orcid.org/0000-0002-0041-1593
Christoph F. Reinhart http://orcid.org/0000-0001-6311-0416
References
Amundadottir, M. L., S. Rockcastle, M. S. Khanie, and M. Ander-
sen. 2017. “A Human-centric Approach to Assess Daylight
in Buildings for Non-visual Health Potential, Visual Interest
and Gaze Behavior.” Building and Environment 113: 5–21.
Barber, R. E., and H. C. Lucas. 1983. “System Response Time
Operator Productivity, and Job Satisfaction.” Communica-
tions of the ACM 26 (11): 972–986.
Beigbeder, T., R. Coughlan, C. Lusher, J. Plunkett, E. Agu, and
M. Claypool. 2004. “The Effects of Loss and Latency on
User Performance in Unreal Tournament 2003® .” Proceed-
ings of 3rd ACM SIGCOMM Workshop on Network and
System Support for Games, 144–151.
Bellia, L., A. Pedace, and F. Fragliasso. 2015. “The Impact of
the Software’s Choice on Dynamic Daylight Simulations’
Results: A Comparison Between Daysim and 3DS Max
Design® .” Solar Energy 122: 249–263.
Brady, J. T. 1986. “A Theory of Productivity in the Creative Pro-
cess.” IEEE Computer Graphics and Applications Magazine
6: 25–34.
Brutlag, J. 2009. Speed Matters for Google Web Search. Moun-
tain View: Google.
CEN Technical Committee 169. 2016. prEN 17037:2016 –
Daylight of Buildings. Brussels: European Committee for
Standardization.
CIE Technical Committee 3–13. 1995. CIE 117–1995 Discomfort
Glare in Interior Lighting. Vienna: Commission Interna-
tionale de L’Eclairage.
Cook, R. L., T. Porter, and L. Carpenter. 1984. “Distributed Ray
Tracing.” SIGGRAPH Computer Graphics 18 (3): 137–145.
Csikszentmihalyi, M. 1996. Creativity: Flow and the Psychol-
ogy of Discovery and Invention. 1st ed.New York: Harper-
Collins.
Doherty, W. J., and R. P. Kelisky. 1979. “Managing VM/CMS
Systems for User Effectiveness.” IBM Systems Journal 18
(1): 143–163.
Doherty, W. J., and A. J. Thadani. 1982. The Economic Value of
Rapid Response Time. White Plains, NY: IBM.
Du, J., and S. Sharples. 2011. “The Assessment of Vertical Day-
light Factors Across the Walls of Atrium Buildings, Part 1:
Square Atria.” Lighting Research and Technology 44 (2):
109–123.
Einhorn, H. 1979. “Discomfort Glare: A Formula to Bridge
Differences.” Lighting Research and Technology 11 (2):
90–94.
Galasiu, A. D., and M. R. Atif. 2002. “Applicability of Daylight-
ing Computer Modeling in Real Case Studies: Comparison
Between Measured and Simulated Daylight Availability and
Lighting Consumption.” Building and Environment 37 (4):
363–377.
Grynberg, A. 1989. Validation of Radiance. Berkeley, CA:
Lawrence Berkeley Laboratories. Document ID 1575.
Guynes, J. L. 1988. “Impact of System Response Time on
State Anxiety.” Communications of the ACM 31 (3): 342–
347.
Harrold, R. 2003. IESNA Lighting Ready Reference: A Com-
pendium of Materials from the IESNA Lighting Handbook.
9th ed.New York: Illuminating Engineering Society of North
America.
Hopkinson, R. 1972. “Glare from Daylighting in Buildings.”
Applied Ergonomics 3 (4): 206–215.
Hoxmeier, J. A., and C. DiCesare. 2000. “System Response Time
and User Satisfaction: An Experimental Study of Browser-
based Applications.” AMCIS 2000 Proceedings, 140–145.
IESNA Daylighting Metrics Committee. 2012. Lighting Mea-
surement #83, Spatial Daylight Autonomy (sDA) and Annual
Sunlight Exposure (ASE). New York: IESNA Lighting Mea-
surement.
International WELL Building Institute. 2016. The WELL Building
Standard® v1. New York: Delos Living LLC.
ISO 9241–303. 2008. Ergonomics of Human–system Interaction
– Part 303: Requirements for Electronic Visual Displays.
Geneva: International Organization for Standardization.
ISO 9241–3. 1992. Ergonomic Requirements for Office Work
with Visual Display Terminals (VDTs) – Part 3: Visual Dis-
play Requirements. Geneva: International Organization for
Standardization.
Jakubiec, J. A., and C. F. Reinhart. 2011. “DIVA 2.0: Integrat-
ing Daylight and Thermal Simulations Using Rhinoceros
3D and EnergyPlus.” Proceedings of Building Simulation
2011: 12th Conference of International Building Perfor-
mance Simulation Association, Sydney, 14–16 November,
2202–2209.
Jakubiec, J. A., and C. F. Reinhart. 2012. “The ‘Adaptive Zone’
– A Concept for Assessing Discomfort Glare Throughout
Daylit Spaces.” Lighting Research and Technology 44 (2):
149–170.
Jones, N. L., and C. F. Reinhart. 2014. “Physically Based
Global Illumination Calculation Using Graphics Hardware.”
Proceedings of eSim 2014: The Canadian Conference on
Building Simulation, 474–487.
Jones, N. L., and C. F. Reinhart. 2015. “Validation of GPU Light-
ing Simulation in Naturally and Artificially Lit Spaces.”
Proceedings of BS2015: 14th Conference of International
Building Performance Simulation Association, Hyderabad,
India, December 7–9, 1229–1236.
Jones, N. L., and C. F. Reinhart. 2016. “Real-time Visual Comfort
Feedback for Architectural Design.” PLEA 2016 Los Ange-
les – 32nd International Conference on Passive and Low
Energy Architecture.
Jones, N. L., and C. F. Reinhart. 2017a. “Experimental Vali-
dation of Ray Tracing as a Means of Image-based Visual
Discomfort Prediction.” Building and Environment 113:
131–150.
 Journal of Building Performance Simulation
Jones, N. L., and C. F. Reinhart. 2017b. “Speedup Poten-
tial of Climate-based Daylight Modelling on GPUs.” 15th
Conference of International Building Performance Simula-
tion Association, 1438–1447.
Jonson, B. 2005. “Design Ideation: The Conceptual Sketch in the
Digital Age.” Design Studies 26 (6): 613–624.
Jota, R., A. Ng, P. Dietz, and D. Wigdor. 2013. “How Fast Is Fast
Enough? A Study of the Effects of Latency in Direct-touch
Pointing Tasks.” Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, 2291–2300.
Kleindienst, S., and M. Andersen. 2012. “Comprehensive Annual
Daylight Design Through a Goal-based Approach.” Building
Research & Information 40 (2): 154–173.
Konis, K. 2017. “A Novel Circadian Daylight Metric for Build-
ing Design and Evaluation.” Building and Environment 113:
22–38.
Lafortune, E. P., and Y. D. Willems. 1993. “Bi-directional Path
Tracing.” Proceedings of CompuGraphics 93: 145–153.
Larson, G. W., and R. Shakespeare. 1998. Rendering with Radi-
ance: The Art and Science of Lighting Visualization. San
Francisco, CA: Morgan Kaufmann.
Liu, Z., and J. Heer. 2014. “The Effects of Interactive Latency
on Exploratory Visual Analysis.” IEEE Transactions on
Visualization and Computer Graphics 20 (12): 2122–2131.
Mardaljevic, J. 2015. “Climate-based Daylight Modelling and Its
Discontents.” CIBSE Technical Symposium, London, April
16–17.
Nabil, A., and J. Mardaljevic. 2006. “Useful Daylight Illumi-
nances: A Replacement for Daylight Factors.” Energy and
Buildings 38 (7): 905–913.
Nazzal, A. A. 2001. “A New Daylight Glare Evaluation Method:
Introduction of the Monitoring Protocol and Calculation
Method.” Energy and Buildings 33 (3): 257–265.
Newell, A. 1994. Unified Theories of Cognition. Cambridge, MA:
Harvard University Press.
Ng, A., et al. 2012. “Designing for Low-latency Direct-touch
Input.” Proceedings of the 25th Annual ACM Symposium
on User Interface Software and Technology, 453–464.
Ng, E. Y.-Y., L. K. Poh, W. Wei, and T. Nagakura. 2001.
“Advanced Lighting Simulation in Architectural Design in
the Tropics.” Automation in Construction 10 (3): 365–379.
O’Hara, K. P., and S. J. Payne. 1998. “The Effects of Operator
Implementation Cost on Planfulness of Problem Solving and
Learning.” Cognitive Psychology 35 (1): 34–70.
O’Hara, K. P., and S. J. Payne. 1999. “Planning and the User
Interface: The Effects of Lockout Time and Error Recovery
Cost.” International Journal of Human–Computer Studies
50 (1): 41–59.
Oxman, R. 2006. “Theory and Design in the First Digital Age.”
Design Studies 27 (3): 229–265.
361
Parker, S. G., et al. 2010. “OptiX: A General Purpose Ray Tracing
Engine.” ACM Transactions on Graphics – Proceedings of
ACM SIGGRAPH 29 (4).
Reinhart, C. F., and M. Andersen. 2006. “Development and Vali-
dation of a Radiance Model for a Translucent Panel.” Energy
and Buildings 38 (7): 890–904.
Reinhart, C., and P.-F. Breton. 2009. “Experimental Validation
of Autodesk® 3ds Max® Design 2009 and Daysim 3.0.”
LEUKOS: The Journal of the Illuminating Engineering
Society of North America 6 (1): 7–35.
Reinhart, C. F., and S. Herkel. 2000. “The Simulation of Annual
Daylight Illuminance Distributions – A State-of-the-art
Comparison of Six RADIANCE-based Methods.” Energy
and Buildings 32 (2): 167–187.
Reinhart, C. F., and O. Walkenhorst. 2001. “Validation of
Dynamic RADIANCE-based Daylight Simulations for a
Test Office with External Blinds.” Energy and Buildings 33
(7): 683–697.
Robert McNeel & Associates. 2016. Rhinoceros 5.
Salman, H. S., R. Laing, and A. Conniff. 2014. “The Impact of
Computer Aided Architectural Design Programs on Concep-
tual Design in an Educational Context.” Design Studies 35
(4): 412–439.
Sheldon, N., et al. 2003. “The Effect of Latency on User Perfor-
mance in Warcraft III® .” Proceedings of the 2nd Workshop
on Network and System Support for Games, 3–14.
Ubbelohde, M. S., and C. Humann. 1998. “Comparative
Evaluation of Four Daylighting Software Programs.”
In 1998 ACEEE Summer Study on Energy Efficiency
in Buildings, Pacific Grove, CA, 3, 325–3.340. Wash-
ington, DC: American Council for an Energy-Efficient
Economy.
US Green Building Council (USGBC). 2013. LEED Reference
Guide for Building Design and Construction, LEED V4.
Washington, DC: USGBC.
Whitted, T. 1980. “An Improved Illumination Model for Shaded
Display.” Communications of the ACM 23 (6): 343–349.
Wienold, J. 2009a. “Daylight Glare in Offices.” PhD Thesis.
University Karlsruhe.
Wienold, J. 2009b. “Dynamic Daylight Glare Evaluation.”
Eleventh International IBPSA Conference, Glasgow, Scot-
land, 944–951.
Wienold, J., and J. Christoffersen. 2006. “Evaluation Methods and
Development of a New Glare Prediction Model for Daylight
Environments with the Use of CCD Cameras.” Energy and
Buildings 38 (7): 743–757.
Yoon, Y., J. W. Moon, and S. Kim. 2016. “Development of
Annual Daylight Simulation Algorithms for Prediction of
Indoor Daylight Illuminance.” Energy and Buildings 118:
1–17.