Exploration & Validation

Making sense of generated data in large option sets

Rachel Tan1 , Trevor Patt2 , Seow Jin Koh3 , Edmund Chen4

1,3,4
Sembcorp Architects & Engineers 2 Singapore University of Technology and
Design
1,2,3
{rae.twx94|trpatt|kohseowjin}@gmail.com
4
chen_mingyie@yahoo.com

The project is a real-world case study where we advised our client in the selection
of a viable and well-performing design from a set of computationally generated
options. This process was undertaken while validating the algorithmic generative
process and user-defined evaluation criteria through scrutinizing the other
alternative options to ensure ample variability was considered. Optimisation
algorithms were not ideal as low performing options were not visible to validate
variability. We established variability by extracting the different groups of
options, proving to the client that various operational behaviours were present
and accounted for. In order to sieve through the noise and derive meaningful
results, we employed methods to filter through thousands of options, including:
k-means clustering, archetypal labelling and analysis, pareto front analysis and
visualisation overlays. We present a sense-making and decision-making process
that utilizes principles of genetic algorithms and analysis of multi-dimensional
user-derived evaluation scores. To enable the client's confidence in the
computational model, we proved the effectiveness of the generative model
through communicating and visualizing the impact of different criterias. This
ensured that operational needs were considered. The visualization methods we
employed, including pareto front extraction and analysis eventually helped our
clients to arrive at a decision.

Keywords: generative design, validation, multi-objective optimisation, k-means,

pareto front, decision-making

INTRODUCTION Generating a large amount of information invites a

One of the main benefits of computational design is fair amount of noise in resultant data. Managing this
the ability to generate large option sets, the upper data and controlling the limits of the algorithm is ex-
bound only limited by available computing power. tremely important when faced with clients that are

D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1 - eCAADe 38 | 653
 not design-trained or data-inclined. When dealing
with domain-specific, operational or performance-
driven concerns, helping the client to understand
and validate the logic of a computational model is a
necessary step.
Clients and stakeholders are also very likely to
provide a long list of domain-specific requirements
and criteria, not fully comprehending the impact of
these metrics on a computational model and unable
to confidently establish priorities or weightage. This
eventually translates into a multi-objective criteria,
and its complexity and noise increases with every di-
mension added.
Multi-objective optimization is currently very
popular in computational design (Ashour and Ko-
larevic 2015). As objectives compete with each
other, designers study the tradeoffs through opti-
mizing and visualizing the set of solutions along the
pareto front (Evins 2013). A 2-dimensional pareto
front curve has optimised solutions that move from
high score in one criteria to high score in another to
determine appropriate tradeoffs and compromises
(Gero and Balachandran 1986) In genetic optimisa-
tion, according to the principle of “survival of the
fittest”, the low-performing solutions or options not
in the Pareto front are usually weeded out behind the
scenes (Belem and Leitao 2019). Utilization of this
approach usually only visualises a small number of
top performing solutions and is not as effective in a
noisy dataset or a generated dataset still undergo-
ing validation. Furthermore, when the pareto-front
is in more than 2 or 3 dimensions, the visualisation
of trade-offs becomes problematic to analyse. In our
654 | eCAADe 38 - D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1
case, the client was keen to establish variability and
make sure that all possible options were evaluated.
Options needed to be visible and optimisation algo-
rithms were not ideal as the whole spectrum of op-
tions were not shown.
At the same time, traditionally, clients rely on
professional consultants to present a handful of de-
sign recommendations and are not used to mak-
ing decisions when presented with large amounts
of data. Urban planners and designers have begun
to use computer simulation and models to influence
decision-making by presenting the impacts of each
recommended option in the form of evaluation met-
rics (Sevtsuk and Mekonnen 2012, Wilson et al 2019,
Hemmersam et al 2015, Al-Douri 2018). When pre-
sented with generative design and the potential of
limitless options (Figure 1), our clients were inter-
ested in understanding the rationale behind the al-
gorithmic generation of options before agreeing on
the appropriate evaluation metrics and how to apply
them. This became a precarious participatory pro-
cess to define cardinal requirements, variables and
acceptable compromises upfront instead of the typi-
cal iterative design process with pen and paper. Due
to the multi-objective complexity of the project, the
clients were not only interested in the “best” op-
tions, but also the reasons why certain options were
deemed low-performing by the algorithm.
OVERVIEW OF CASE STUDY
Computational design need not revolve around “ar-
chitecture”, but can expand into realms of urban plan-
Figure 1
Examples of
generated options
 ning and design. Our project pushes into uncom- configurations needed to be reflected and generated
mon territory in a case of site-planning, focusing on by the model (Figure 3). This is in contrast with a more
a small port facility, comprising 60% sea and 40% parametric approach where the land or basin is deter-
land. At this early stage, the clients were less con- mined by setting out dimensions and positions.
cerned with what the facility looked like, the building
Figure 2 massing, or even area distribution. Instead they were
Adapted process more focused on space planning: where critical op-
from Akin and erations were located and the overall impact on out-
Ozkaya, 2002 put, connectivity and flexibility. To gather initial de-
sign concepts and possible land configurations, we
engaged the users and maritime experts in partici-
patory design workshops. At later stages, we worked
with a smaller group to provide updates and clarify
queries. The overall computational process was in-
herently performance driven and the contrast with
conventional design process can be exemplified in
Figure 3 the Figure 2. The conventional design process en-
Examples of gages in archetype-centred specification, where dif-
configurations ferent concepts (archetypes) are developed and iter-
generated by ated through design reviews (Akin and Ozkaya 2002).
grid-based To manage the client’s comfort level with the compu-
algorithm tational process, we re-introduced archetypal analy-
sis on top of generated data to tie in with the initial
design concepts.
We engaged the users through interviews and work-
GENERATION OF OPTION SETS ing sessions to derive the different levels of infor-
The algorithm for option generation was built in C mation needed and different evaluation metrics at
sharp programming language for Rhinoceros and each level (Figure 4). Many of these criteria were
Grasshopper. We approached the problem with a specific to the client’s operations and could not be
grid-based approach, taking the perspective of a benchmarked against architectural standards, hence
basin-centric facility. The general geometrical opera- even the logic of calculating these metrics had to
tions are as follows: be carefully validated. The users observed that raw
data from the land profile configurations was insuf-
• Starting with a blank area of “water” within site ficient for decision-making thus they proposed addi-
boundary tional information for a more comprehensive evalua-
• Inserting a few required spaces as “obstacles” tion and selection process. Additional levels of infor-
(areas such as berths and zones) mation were generated on top of the 1st level (Land):
• Determining a feasible, manoeuvrable path the 2nd level being Piers, Wharves and Vessels, and
through the obstacles the 3rd level being Land-side Activities. Each level
• Forming and joining up resultant land pixels of information was evaluated by a set of metrics and
from site boundary and berth areas compressed into a one-dimensional score through
We used this grid-based approach because it could normalization and weighting.
handle fewer constraints, as different types of land

D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1 - eCAADe 38 | 655
 Figure 4
Evaluation metrics
for different levels
of information and
corresponding R2
correlation

656 | eCAADe 38 - D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1
 SENSE-MAKING
The objective of sense-making was not to arrive at
a single decision from the output but, rather, to
make sure that the eventual solution is not a myopic
one. As the options were completely generated from
scratch and evaluation metrics were all user-defined,
there was no benchmarking data readily available for
this small scale of port facility to validate the output.
The evaluation criteria were expected to play a big
part in decision-making, therefore the clients wanted
to be sure of the geometry and measurements going
into it. It was a challenge to judge options simply by
a reduced one-dimensional fitness score, for the rea-
son that it did not lend much confidence in decision-
making despite it being weighted according to user-
defined priorities.
Our key struggle was dissecting the relation-
ship between evaluation metrics. Within all 3 levels
of evaluation criteria, the R2 correlation (Figure 4),
which quantifies the correlation between two pairs
of axes, was found to be relatively low. The different
evaluation criteria did not appear to achieve more
than 0.6 hence we were unable to make confident
statements about the impact of one user-defined cri-
teria on another.
Figure 5
Visualising k-means
cluster centroids &
neighbours with
2-dimensional PCA
D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1 - eCAADe 38 | 657
K-means clustering
To further analyse the individual criteria, we ap-
proached the issue from the angle of variability. We
identified methods to study variability at different
stages, using data generated from multi-dimensional
evaluation metrics. The first method at high-level
was simple unsupervised machine learning in the
form of k-means clustering. K-means clustering is
a centroid-based method to group data points into
“k” groups or clusters according to their “distance”
to each other (Everitt and Hothorn 2011, Han et al
2011, Chen et al 2015). The silhouette score, rang-
ing from -1 to +1, is used to measure the data point’s
similarity to its cluster in contrast with other clusters
(Rousseeuw 1987).
We applied k-means clustering for various k val-
ues. To validate, we visualised the data after re-
duction to 2-dimensions using Principal Component
Analysis (PCA) and retrieved the cluster centroids and
their nearest neighbours to try and visually analyse
the similarities (Figure 5). However, as with the cor-
relation data, the overall average silhouette scores
fared poorly and failed to exceed 0.2. Moving for-
ward, we recognized that this k-means was not as ef-
fective due to high noise and imbalance within the
dataset. Simply clustering them with existing unfil-
 tered and unlabelled data may not have been the
best approach.
Given the amount of noise overall and chal-
lenges faced in clustering data numerically, we be-
lieve that there are still a large amount of tacit infor-
mation that were not captured in the 1st, 2nd and 3rd
levels for meaningful k-means clustering and analysis
to show distinct results and recommendations.
Archetypal analysis
The second method was to manually introduce more
information through imposing archetypes and fea-
tures based on visually similar characteristics. This
was derived based on known preferences in discus-
sion with the clients and combined with several ob-
servations of patterns that we made. We manually
tagged a sample set of 1000 options according to 4
characteristics to identify trends in evaluation met-
rics. By comparing and overlaying the similarities
across options with the same unique combinations of
4 characteristics, we were able to identify the trends
in the location of certain spaces and configuration
of the basin. This effectively compressed 1000 indi-
vidual options into 89 number of unique combina-
tions. Upon further analysis of this additional data,
we found that the Archetypes and “H” visual charac-
teristics were shown to be rather distinct on a PCA
plot from evaluation data (Figure 6). This supports
the idea that the visual characteristics were in fact, re-
lated to the output from evaluation criteria. Further-
more, plotting the same PCA data labelled with the
unique combinations from 4 characteristics (stylized
in Figure 6 as w | x | y | z, where the letters each corre-
spond to a characteristic) demonstrated that the clus-
tering of visually similar options (derived from unique
combinations) were also reflected in evaluation data.
This method proved to be more useful in com-
municating the outcome of the generative algorithm
with the clients, as this divided the options into a
categorical grouping and provided a more sensible
yardstick for comparison besides the genetic fitness
score.
658 | eCAADe 38 - D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1
Pareto front analysis
At a lower level between individual metrics obtained
from all the three levels of information, we pro-
ceeded to visualize 2-dimensional pareto fronts and
continued to make observations based on perfor-
mance and imposed archetypes. Selection of axes
for each 2-D pareto front were scoped around ques-
tions. For example, for a question such as, “How do
the archetypes fare for flexibility and connectivity?”,
we selected metrics from Level 1 (Land) that corre-
sponded to flexibility and connectivity. For a ques-
tion such as “How do the archetypes fare in provid-
ing tucked in wharfage and compact activities?”, we
selected metrics from Level 2 (Pier) and Level 3 (Ac-
tivities) for analysis. The data points were labeled
with corresponding archetypes, and the visual rep-
resentation of each option was pulled up (Figure 6).
Through this process, we had hoped to persuade the
client that the perfect option did not exist and could
only be a combination of characteristics. The obser-
vations at sense-making stage were later developed
to recommended design principles.
Visualisation of operations
The evaluation criteria at the 3rd level of information
measured the movement between 28 land-side ac-
tivities. However, it also lacked the clients’ in-depth
prioritsation due to lack of time to develop elabo-
rate logic based on the large number of options al-
ready generated. As such, we proposed to group the
movement paths by user groups to try obtaining di-
rect feedback from the users (Figure 7).
While initially to designers the impact was not as
clear, the relevant users involved were able to quickly
identify possible trade-offs, such as accepting long
distances between logistics nodes with the possible
investment into autonomous vehicles. As informa-
tion in the 2nd and 3rd levels were closer to the users’
expertise, they were able to observe comment on re-
lationships between the 2. For example, they swiftly
drew a relationship between having east-west pier
configuration (2nd level: Pier) and the movement of
personnel (3rd level: Activities), where options with
Figure 6
Archetypal analysis
(top), the
effectiveness of
categorical
grouping (center),
pareto front
visualisation
(bottom)

D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1 - eCAADe 38 | 659

east-west piers tended to force the activities to the
east border and west border. Even though noise in
the 3rd level (travel distances) and limitations in cat-
egorical data in the 2nd level (pier configuration) pre-
vented this relationship from becoming apparent in
data analysis, we believe that feedback is equally use-
ful. Numerical data may not appear to show patterns
but visualizing them using different angles can result
in new insights.
DECISION MAKING
Through the various steps of sense-making, the
clients were assured that adequate variation and
“bad options” existed in the back-end of the algo-
rithm. Thus we were able to proceed to engage them
in the selection of the final design. As we recognize
that visually the clients were unable to process large
number of options, we proposed filters to reduce the
large data set down to a small number of viable op-
tions for selection and analysis.
660 | eCAADe 38 - D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1
Figure 7
Visualisation of
movement
between activities
according to user
groups
We ran the grid based algorithm at the 1st level
(Land) through a genetic algorithm. The 300 top per-
forming options from 3 different optimization runs
(with different initial seeds) were selected to con-
tinue with the generation of additional levels of in-
formation After the 2nd level (Pier, Wharves and Ves-
sels), a total of 772 pier configurations were gener-
ated. After encountering many options with inade-
quate berthing space, the clients decided to intro-
duce a cardinal requirement: that the design must be
able to fit in all the specified vessels. This drastically
reduced the option set by 90% to 72.
Based on these 72 options, we performed a ba-
sic pareto front visualization to understand what op-
tions were determined “best-performing” by the al-
gorithm (Figure 8). The axes were a three-way com-
parison between the weighted score of the three lev-
els (Land, Pier, Wharves and Vessels, and Activities)
This further reduced the number of options to exam-
ine visually to 18 options. Based on these 18 options,
we were able to recombine design concepts and prin-
ciples identified and make recommendations to the
clients.
Figure 8
Three-way pareto
front comparison
between scores
from 3 levels
CHALLENGES
Through this project, we acknowledge that obtain-
ing consensus from a multi-user group was indeed a
challenge. Prioritisation scoring was extremely hard
to lock in. Algorithmic logic was also heavy scru-
tinized to ensure all user concerns were accounted
for. At times, due to preconceived notions and ideas,
there was a degree of disbelief in the outcome as ex-
pectations were not reflected in top 10 options gen-
erated via genetic optimization. Hence, we saw the
need for making sense of the options generated.
We also recognized the value of employing vi-
sual options to a non-expert user group. Despite be-
ing output and performance-driven, we found that
the most useful method to obtain useful comments
from the users was to overlay operational informa-
tion (generated from data) on top of design options.
FUTURE DEVELOPMENTS
This paper presented a preliminary approach to en-
gage the clients in computational design process and
outlined the challenges and methods used to con-
vince them that the generative algorithm encom-
passed all their needs and requirements. This pro-
cess will continue to be refined in later parts of this
project, where we will explore other means of com-
D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1 - eCAADe 38 | 661
municating with user groups.
For the analysis of data, a future development is
the possible use of pre-trained convolutional neural
networks as a feature extraction tool, to make up for
tacit information that existed in the visual represen-
tation but was not apparent in numerical data. This
process may also validate the visual patterns we ob-
served. As generated datasets only get larger, we
must explore more complex ways to analyse data.
ACKNOWLEDGEMENTS
The authors would like to express gratitude to Semb-
corp Architects & Engineers and their clients for sup-
porting this project. Any opinions, findings, and con-
clusions or recommendations expressed in this mate-
rial are those of the authors and do not necessarily re-
flect the views of the aforementioned organizations.
REFERENCES
Akins, O and Özkaya, I 2002 ’Models of Design Require-
ment’, Sixth Design and Decision Support Systems in
Architecture and Urban Planning
Al-Douri, F 2018 ’The Employment of Digital Simulation
in the Planning Departments in US Cities’, Proceed-
ings of eCAADe 2018
Ashour, Y and Kolarevic, B 2015 ’Heuristic Optimization
in Design’, Proceedings of ACADIA 2015
 Belém, C and Leitão, A 2019 ’Conflicting Goals In Archi-
tecture: A study on Multi-Objective Optimisation’,
Proceedings of CAADRIA 2019
Chen, KW, Janssen, P and Schlueter, A 2015 ’Analysing
Populations of Design Variants Using Clustering and
Archetypal Analysis’, Proceedings of eCAADe 2015
Evins, R 2015, ’A review of computational optimisation
methods applied to sustainable building design’, Re-
newable and Sustainable Energy Reviews, 22, pp. 230-
245
Gero, JS and Balachandran, MB 1986 ’Knowledge and
Design Processes’, Applications of Artificial Intelli-
gence in Engineering Problems
Han, JW, Kamber, MB and Pei, J 2011, Data mining: con-
cepts and techniques, Elsevier
Hemmersam, P, Martin, N, Westvang, E, Aspen, J and
Morrison, A 2015, ’Exploring Urban Data Visualisa-
tion and Public Participation in Planning’, Journal of
Urban Technology, 22, pp. 45-64
Rousseeuw, P 1987, ’Silhouettes: a Graphical Aid to
the Interpretation and Validation of Cluster Analysis’,
Computational and Applied Mathematics, 20, pp. 53-
65
Sevtsuk, A and Mekonnen, M 2012, ’Urban Network
Analysis Toolbox’, International Journal of Geomatics
and Spatial Analysis, 22, p. 287–305
Wilson, L, Danforth, J, Davila, CC and Harvey, D 2019
’How to Generate a Thousand Master Plans: A
Framework for Computational Urban Design’, Pro-
ceedings of SimAud 2019

662 | eCAADe 38 - D1.T4.S2. MAKING THROUGH CODE –BUILT BY DATA AND THE ARCHITECTURAL ILLUSTRATION - Volume 1