2018 17th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/ 12th

IEEE International Conference On Big Data Science And Engineering

A Full-Scale Security Visualization Effectiveness

Measurement and Presentation Approach
Jeffery Garae
Cyber Security Lab
Department of Computer Science
University of Waikato
Hamilton, New Zealand 3240
Email: jg147@students.waikato.ac.nz
Ryan K. L. Ko
Cyber Security Lab
Department of Computer Science
University of Waikato
Hamilton, New Zealand 3240
Email: ryan@waikato.ac.nz
Mark Apperley
Department of Computer Science
University of Waikato
Hamilton, New Zealand 3240
Email: mapperle@waikato.ac.nz

Abstract—What makes a security visualization effective? How
do we measure visualization effectiveness in the context of
investigating, analyzing, understanding and reporting cyber secu-
rity incidents? Identifying and understanding cyber-attacks are
critical for decision making – not just at the technical level, but
also the management and policy-making levels. Our research
studied both questions, and extends our Security Visualization
Effectiveness Measurement (SvEm) framework by providing a
full-scale effectiveness approach for both theoretical and user-
centric visualization techniques. Our framework facilitates ef-
fectiveness through interactive three-dimensional visualization to
enhance both single and multi-user collaboration. We investigated
effectiveness metrics including (1) visual clarity, (2) visibility, (3)
distortion rates and (4) user response (viewing) times.
The SvEm framework key components are: (1) mobile display
dimension and resolution factor, (2) security incident entities, (3)
user cognition activators and alerts, (4) threat scoring system,
(5) working memory load and (6) color usage management.
To evaluate our full-scale security visualization effectiveness
framework, we developed VisualProgger - a real-time security
visualization application (web and mobile) visualizing data prove-
nance changes in SvEm use cases.
Finally, the SvEm visualizations aims to gain the users’ atten-
tion span by ensuring a consistency in the viewer’s cognitive load,
while increasing the viewer’s working memory load. In return,
users have high potential to gain security insights in security
visualization. Our evaluation shows that viewers perform better
with prior knowledge (working memory load) of security events
and that circular visualization designs attract and maintain
the viewer’s attention span. These discoveries revealed research
directions for future work relating to measurement of security
visualization effectiveness.
Index Terms - Security visualization; Effectiveness measure-
ment; Cyber-attacks; Cyber security; Mobile security; Network
security; Web application security; Cognitive load; Attention
span.
I. I NTRODUCTION
Security visualizations are useful tools for gaining insights
into security events but how effective are they? Do they im-
prove cognition and decision making in time-critical security
events, or do they slow the analysts down? This research
presents a framework for measuring the effectiveness and
presentation of security visualizations. Our primary focus is
improving the SvEm framework [11] by providing a full-
scale security visualization effectiveness measurement ap-
proach across an entire visualization experience. We assume
that users are already competent when interacting and collabo-
rating with security visualizations. We address data processing
performance, visual clarity and user interactive enhancement
features for a better user experience. To measure effectiveness
of security visualizations, we require extensive assessments
of both Web and mobile platforms, and their respective user
response reaction times.
The moment a user comes into contact with a given security
visualization, we are interested in understanding how the
visualizations captured the user’s attention within minimal
time, and measure their ‘attention span’. As such, we focus on
observing the user’s cognitive load and working memory load
efficiency. Visualization effectiveness is optimized when cog-
nitive load decreases while working memory load increases.
II. BACKGROUND AND S COPE
In most scientific, framework and user interface research,
increasing ‘effectiveness’ translates to the goal of improving
productivity by reducing time spent to gain tangible results.
In this paper, our framework defines ‘effectiveness’ in security
visualization as an entire full-scale security approach which
maximize user interaction and efficiently deliver tangible in-
sights (information) to the targeted audience with the use of
visualization.
In this paper, we measure effectiveness across the entire
security visualization process, including both graphical repre-
sentation and user interaction. We also take the view that se-
curity visualization supplements data analytics and automated
processes in transforming raw security/‘cyber-attack’ event
data into useful analysis. Security visualizations connect and
present security incidents to users by providing interactive [14]
visual experiences. Interactivity captures the user’s interest
to take further necessary steps into collaborating and under-
standing cyber-attack landscapes. Visualization also facilitates
the ability to process large data volumes and present trends
and patterns visually. However, presentation and performance
challenges exist. Hence, the core purpose of this paper which is
‘effectiveness measurement’ in security visualization. Our re-
search provides a link between user cognitive knowledge with
the security visualization effectiveness measurement frame-
work (SvEm) [11]. Our security visualization effectiveness ap-

2324-9013/18/31.00 ©2018 IEEE 639

DOI 10.1109/TrustCom/BigDataSE.2018.00095
proach covers the planing, design, implementation, evaluation,
validation and user (targeted audiences) interaction assessment
stages.
From the following section, we will review existing visual-
ization effectiveness measurement research work. The SvEm
framework design and results are later discussed with an eval-
uation section to analyze our framework. Finally we provide
a conclusion along with proposed future work.
III. R ELATED W ORK
While users normally choose visualization techniques based
on individual preferences or needs, there is a current lack of
addressing effectiveness measurement and assessment tech-
niques. Current techniques [11],[12],[17],[40] assess user per-
formance, clarity, image distortion rate, image quality/ranking,
cognitive & perception assessment, visualization correlation
measurement, brain activity measurement and how well vi-
sualization is presented. Hence we review past and existing
literatures to address these concerns.
A. Information Visualization Measurement Techniques
Most techniques address technical aspects such as clarity
and visual recognition abilities from users. However, prefer-
able visualizations are those that captures viewers and are
able to self-tell a story without the need of a person to
translate or explain what the visualization means. This is
the ideal expected outcome for most artists and visualization
experts. Common grounds between both the visualization and
users motivate viewers by activating cognitive capabilities to
trigger effectiveness mechanisms in users with interests in
the given visualization. Therefore, effectiveness requires some
assessment methods to allow improvement in visualization.
B. Measurement Technique: Ranking Visualization of Corre-
lations
The approach of leveraging on ‘training sessions and
datasets are commonly used in scientific measurement for per-
formance, transparency and integrity. Harrison et al. [12] out-
lined the use of Weber’s law [4],[15] and perception law [15]
in ranking visualization of correlation [20]. Other methods
involves the use of training dataset against best-case datasets
which are plotted using scatterplots and parallel coordinates to
show correlations in visualization [33]. Recent psychological
and cognitive science research [15] have shown that perception
laws [20] can be applied to model how humans perceive
certain data properties in a given visualization. Rensink et al.
demonstrated with the use of weber’s law [33] with creation
of Weber model Fit [12]. It states that there are connectivity
between humans and the data presented. Human perception
has the ability to differentiate correlation and the objective
differences in data correlation. This has a linear relationship:
dS
dp = k
S ing human cognition, perception, attention span and working
where dp indicates the differential change in perception, k is
the experimental obtained weber fraction, dS is the differential
increase in the data correlation.
640
Several statistical analyses [12] were conducted in aiding
ranking: (1) a Kruskal-Wallis [26] test to asses interactions
between visualization and correlations, (2) a Mann-Whitney-
Wilcoxon [16],[29] test for comparing visualization pairs, and
(3) finally, a Bonferonni correction [36] to address multiple
comparison problems and reduce false-positive results. While
this ranking visualizations of correlation has proofed to be an
effective technique to compare visualizations, it differs from
the goal and objectives of this paper.
C. Graphical Presentation Techniques
Fig. 1: The E 3 framework design
Most hardware over the Internet have logging capabilities
which contributes to the large rapid rate of data collected.
Therefore, presenting information from datasets require certain
techniques. For example, Leung K. Y and Apperley D. Mark’s
E 3 framework [24], a graphical presentation framework for
large data sets. The analytical E 3 framework provides an
avenue for comparison of different presentation techniques
with the volume of data presented. The core features Expres-
siveness, Efficiency and Effectiveness contributes to ranking of
presentation accuracy and perceptual tasks. Figure. 1 shows
the E 3 design indicating various key components with the
notations used, their relationships and the stages involved in
presentation system designing for large datasets.
D. Error Sensitivity in Visualizations
Another common visualization measurement technique in-
volve measuring and identify error rate calculation in image
quality or distortion measurement. Figure. 2 shows the design
framework comprising of pre-processing, filtering, channel
decomposition and error pooling stages. In this model, image
quality are assessed by a measure of visual clarity and dis-
tortion in the presented image. Executing pre-processing and
filtering enhances the outcome of quality/distortion measure-
ment which translate overall effectively.
E. Cognition, Perception and Insight Contribution in Visual-
izations
The psychological contribution to effectiveness measure-
ment in visualization are identified and measured from assess-
memory load. The rational between a user’s cognitive capabil-
ity and working memory load can be measured using his/her
mental effort rating as shown in Figure. 3. For example, an

Fig. 2: Error Sensitivity Design
ideal user rating would fall in ‘Region A’ (Figure. 3) [30]
whereby performance reading is high while mental effort is
low. This also means the user’s working memory load is
high. User studies have provided means for cognitive load
measurement [17] particularly mental effort and performance
assessment techniques which address visualization efficiency.
Fig. 3: Mental Effort Efficiency Reading [30]
Insight-based evaluation [40],[34],[35] by InfoVis have el-
evated the use of insights as an evaluation measure for tech-
nologies. ‘Insight’ [40] is defined as gaining accurate and deep
understanding of something i.e. a unit of discovery. It is often
not achieved by predefined tasks or procedure but with higher
probability a by-product of exploring without an initial goal
or destination. Moreover, ‘sensemaking’ [32] plays a major
role in gaining insights although the model (Information -¿
Scheme -¿ Insight -¿ Product) includes insight as a component.
It enhances the entire experience of gaining and understanding
insight.
Summarizing all related research, we see that effectiveness
measurement in visualization covers not only technologies
but in users as well. With all key areas mentioned in this
section, we now have a clear understanding on existing work
around effective measurement methodologies in visualization.
However, our framework covers specifically on security visu-
alization with the aim of assessing effectiveness measurement
with regards to the urgency of security incident information
641
presented.
IV. S V E M F RAMEWORK D ESIGN
In order to implement a framework whereby security visual-
ization effectiveness can be assessed, measured and evaluated,
framework designs are a very critical stage of the whole
development cycle. Therefore, in this section we present our
framework design. Our effectiveness measurement (SvEm)
components in security visualization are: (1) mobile display
dimensions, (2) security incident entities, (3) user cognition
activators and alerts, (4) threat scoring system, (5) working
memory load, and (6) color usage management. These com-
ponents are discussed in following subsections.
A. System Backend Design Architecture
The SvEm security visualization framework server-side
(backend) infrastructure is designed to accommodate both
static and real-time visualization scenarios. It handles all data
analytic processes occurring within the database, collectors
and parser environment. Our system architecture includes
the following components: Windows Progger (Logging Tool),
Redis, MongoDB, Nodejs and WebGL. Windows progger (a
windows version of linux progger [21]) is an internal sys-
tem/kernel level provenance logging tool currently being de-
veloped with emphasis of providing security within computer
and cloud systems. Redis 1 [2] facilitates our cache/database
link between Windows progger and mongoDB. All data are
stored permanently in mongoDB [3], while nodejs [39] and
webgl [5],[31] facilitates the client-side frontend security vi-
sualization framework.
The server-side architecture is designed to handle data
processes while managing data storage. Preprocessed data are
engineered based on the visualization scenario. For exam-
ple, a real-time logging of a computer’s kernel system for
provenance purposes is visualized to show and keep track
of file creation, modification and deletion. In addition, data
are standardised to meet the effectiveness assessment of the
security visualizations in web and mobile platforms. Both web
and mobile requirements enabled the need to provide efficient
data querying, processing, parsing, rendering and scaling tasks
for a security visualization. Figure. 4 shows the basic tools
and libraries required to host the SvEm security visualization
server-side backend.
While there are many application features to consider when
designing a framework, other key features include security,
data processing performance and visualization presentation
clarity. These are the main concerns for our framework.
B. Security Visualization Technical Aspects
Mobile dimensions contribute to visualization designs. For
example the limitations in an IPhone 6s Plus - 1920 x 1080
pixels display, with a hight of 122mm and width of 68mm
as shown in Figure. 5 indicates that our visualization designs
have to set design controls. These controls include the amount
1 Redis is an open source (BSD Licensed) in-memory data structure store,
used as a database, cache and message broker
    !   "

  design, data collected in real case scenarios are not able to
 
 
  
# $ 
  
 
Fig. 4: SvEm Backend Design and Architecture
Fig. 5: Mobile Dimensions
of data processed, how visualization should appear and what
types of visualization would best fit this screen dimensions.
A clear understanding of these limitations allow security
visualization developers to consider multidimensional and/or
circular representations. Such designs accommodate multiple
data attributes of the security incident.
1) Attribution Visualization Design: Attribution 2 [38] in
traditional security sense seeks to trace back to the source of a
cyber-attack. In security visualization, representing attribution
is complicated. It requires datasets and a clear understanding
of the attribution process. Our attribution design targets the
attribution process, connects dots between the source and
destination of an attack. The emphasis is on identifying and
2 Attribution is defined as determining the identity or location of an attacker
or an attacker’s intermediary
642
providing the ‘source’ of cyber-attack since most destination
are considered the victim. Despite our attribution visualization
show a full-scale attribution visualization. We therefore pro-
vide predictive analytics with patterns that are able to connect
dots between key identifiers of the attack for attribution at an
abstract level through visualization.
2) Provenance Visualization Design: Another core design
feature in this framework is effectively representing prove-
nance from large volume of data collected. Large volume of
data are transformed into visualization for mobile platforms
posing scalability and display space limitation. This frame-
work utilizes attribution and provenance design with abstracts
of data to alert and notify users of security events. Provenance
is crucial for security experts and end-users to be aware of.
Figure. 6 provides the provenance visualization design with
‘time’, ‘attack type’ and ‘source of attack’ information. This
circular design targets the user’s attention span with prime
focus on information presented and reduces the number of
clicks/tabs to acquire further information.
Fig. 6: Provenance Mobile Visualization Design
3) Types of Visualization Designs:: Another important as-
pect of effectiveness in visualization is giving users (viewers)
multiple visual design options to view required data rep-
resentation. This provides an avenue for a wider range of
audience. As part of the real-time security visualization, our
framework provides the following visual design options‘[6]:
‘Helix (Spiral)’, ‘Sphere’ and ‘Grid’.
As shown in Figure. 7, the helix visualization leverages
on Gestalt’s Principle/Law of Continuity [37] which displays
file and process execution order, i.e. first-in, first displayed
approach. This means when a user has established his/her
 Fig. 7: Helix (Spiral) Visualization Design
attention on a particular file, pattern, or group of interest
with common behavior/colors, cognitively he/she perceives
and builds a visual content that he/she could easily understand.
The sphere visualization design leverages on the Gestalt’s
law of Closure, where everything is perceived as being part of
a whole, therefore a closed complete visualization approach.
Figure. 8 shows the content of a system visualized and
also highlights the technique of providing a clear simple
visualization that can be scaled to fit any mobile platform
displays. Regardless of how many files or process required
to be visualized, the sphere approach provides visualization
where all parts are the sum of a whole.
Our grid visualization design provides a layered visual
approach where new files are either visually displayed on
top or in front of the grid visualization view. This design
attracts the viewers attention to the new files/process of interest
presented. Maintaining the viewer’s attention keeps them focus
while providing other alert mechanisms to relate information
across to the viewer. Figure. 9 shows a sample grid design
visualization with multiple layers of files and processes.
Additionally, we provide a ‘circular-layered’ design as
shown in Figure. 10 which addresses multiple attributes and
categorization of different files and processes. Effectiveness in
this case, is shown by traversing through the layers allowing
viewers to see and understand how different file systems func-
tion. The use of a layered visual approach connects various
levels of information hierarchy, and connecting information
relationships together.
4) Threat Scoring Components: Another effective com-
ponent of our SvEm framework is identifying threats and
visualizing them. Our threat scoring design (Figure. 11) covers
anomaly, malware and customized detection designed mecha-
nisms. Datasets are filtered through test/training & ground-
truth data [10] and known threat signature databases. The
643
anomaly detection utilizes existing algorithm which will be
discussed later in our evaluation and validation section. Our
ground-truth dataset consists of known pre-identified threats,
user-input logs and known threat patterns & behavior schema.
This provides a better controlling and monitoring environment.
C. Security Incident Entities, Relationships and Landscapes
1) Entities:: Entities (En), Relationships (EnR) and Secu-
rity Landscapes (SL) are core effective components of our
framework. Entities refers to the following: threat actors,
malicious payloads, Infected IP address and more. These
entities are the point of interest for how the SvEm effectiveness
measurement theory works. Identifying these entities through
visualization within the minimal time required affects the
performance of our framework.
2) Entity Relationships: Entity Relationships also known
as links are vital for our framework. The EnR functions
are to connect entities together. These links also activate
user-cognitive functionalities which allows a user to perceive
hidden information and potential security insights.
3) Security Incident Landscapes: Security Landscapes (SL)
provide the incident scope and environment for users (viewers)
to control their imaginations around a certain security incident.
A familiar SL enhances a user to establish a conceptual
boundary which enables him/her to confront a visualization
with confidence.
D. Security Visualization Color Standard
It is critical to standardize the use of colors in security vi-
sualization. Large volumes of dataset with potential interested
entities require simplified security visualization to enhance
rapid information processing. For example, using the color
‘red’ and ‘orange’ in the same visual space automatically
creates confusion to users therefore adding complication to
the entire visualization experience. Our standardized selection
 Fig. 8: Sphere Visualization Design

Fig. 9: Grid Visualization Design

of colors as shown in Figure. 12 are: ‘red, yellow, green,
blue, purple and orange’. These colors are categorized into
two categories, primary and secondary groups. Our primary
color choices for security visualization are red, yellow, green,
and blue. The secondary group are purple and orange. These
additional colors are specifically for law enforcement security
visualization with concepts matching the Interpol’s color-
coded Notice system [1],[18]. For example, orange is only
used to show illegal trafficking contents and it is regarded as
an independent visualization type.
The color standard addresses simplicity, familiarity and the
establishment of comfort environment with prior knowledge.
From a developer’s point of view, understanding colors and
matching them to security event attributes in visualization is
important. This triggers the need for color management to
avoid the issue of colors overlapping in representing security
incidents which can contribute to visualization misinterpreta-
tion.
E. Cognitive Requirements for Security Visualization
Information processing is a natural human role where tech-
nology can not control. However, there are methods applied
as forms of control while processing information in a given
security visualization. The establishment of these controls
aim at minimizing cognitive bias [13] which often leads
to perception distortion, inaccurate judgement and illogical
interpretations.

644

Fig. 10: Locky Ransomware Mobile Visualization Design
Training / Ground-Truth
Dataset
Threat Scoring Anomaly Detect-
System ion Mechanism
Signature
Database
Malware Detect-
ion Mechanism
Customized
Detection
Fig. 11: Threat Scoring Mechanism Design
Therefore a critical design component of our SvEm frame-
work is the establishment of cognitive psychological features
in security visualization. This allows setting up requirements
and tasks that users (viewers) need in order for the whole
visualization experience to be effective. The requirements are:
(1) cognitive load, (2) working memory load and (3) user
cognition activators. In addition, psychological tasks involves:
(1) attention process, (2) pre-attentive (pre-attention) process,
and (3) mental (memory) effort.
From the security visualization application stand-point, cog-
nitive ‘activators’ and ‘alert’ features are designed to create
Malicious Content: payload, event (file,process, etc.)
Suspicious Content: payload, event (file,process, etc.)
Normal Content: data traffic
Intelligence Content: tracking files, tagged files, etc.
Trafficking Content: drug trafficking, etc.
Fraud Content: currencies, account details, etc.
Fig. 12: Security Visualization Color Standard
645
the link between viewers and the visualization presented.
We refer to these activators as ‘Semi-permanent Hold’ and
‘Permanent Hold.’ Section V and the Evaluation (Section VI)
section further discuss how cognition plays an important role
in this framework.
V. R ESULTS :S ECURITY V ISUALIZATION F RAMEWORK
A. SvEm Theory
We begin with expanding our SvEm algorithm and explain-
ing it in details. The SvEm algorithm V-A has the following
components and attributes:
(SvEm) Theory for Distortion (dsvem ) Assessment
(w ∗ h)/Svf ∗ dn
SvEm = > 50%(Distortion)
Cl ∗ tme ∗ nclicks
(1)
(SvEm) Theory for Time (tsvem ) Assessment
(Cl ∗ tme )
SvEm = ≤ 0.25sec(s)(T ime)
nclicks ∗ Svf /dn
(2)
Where:
w * h : Mobile Display Area (dimensions)
Svf : Security Visual Nodes (e.g. Infected-IP, Times-
tamps, etc.)
dn : n-dimensional view of security visualization
Cl : Cognitive Load (Identifiable Attributes (Quantity)
- Prior Knowledge)
tme : Memory efficiency (Effort based on Working
memory - Time-base)
nclicks : Number of clicks on Visualization
Our SvEm theory is derived with respect to ‘distortion rate’
and ‘time’. Although the distortion pivot rate is 50%, our over-
all assessment are measured against a ‘high’ or ‘low’ rating
to make our assessment more realistic. Factors affecting our
SvEm-distortion rate are: (1) phone dimensions and resolution,
(2) user knowledge and (3) the number of clicks users execute.
SvEm-time is measured against a constant: 0.25 sec-
onds [27] - known in science and psychology research as the
least minimal cognitive time required for a human to process
and understand information through human perception. Thus,
our overall assessment are calculated as an average and rated
against many other samples.
Application performance, data processing and management
techniques are implemented to enhance the final distortion and
time outputs. Data representation in our application visual
space are managed in order to balance hardware processing
capabilities and visualization complexities.
 Cognitive load (Cl) and working memory load (tme ) are
calculated using past theorems and user studies. Our SvEm
algorithm inherits existing methods and utilizes them.
B. Data Process Flow
A contributor to making our SvEm framework effective is
the ability to process and manage data from our database
right through to our WebGL front-end visualization. Figure. 13
presents the data flow diagram with the core component
of the back-end infrastructure. Utilizing Progger, Redis, and
MongoDB enables a proper data flow management. High
volume of security data are pushed to the security visualization
front-end with high consideration of the mobile platform
used, i.e. accounting for the hardware processing power,
display dimensions and resolution capabilities. This allows our
‘analysis-scripts’ to scale data accordingly for better and clear
visualization with less complex appearance.
Fig. 13: Data Process flow Results
Based on the understanding of the data analyzed, we provide
several use cases as part of our implementation and testing of
our framework. A video demo of the use cases are compiled
and can be viewed in the link provided: SvEm Security
Visualization Use case demo.
C. Use-case 1: A Collaborative Real-time Security Visualiza-
tion Application
A provenance log visualizer, VisualProgger, was built by our
team to visualize provenance logs generated by Progger [21].
VisualProgger is a real-time user-centric security visualiza-
tion application which facilitates visualization effectiveness
through clarity, performance and the use of cognitive activa-
tors. Important security information were identified through
data analytic processes with visual animations and relevant
alert methods.
1) VisualProgger Features: VisualProgger provides distinc-
tive effective security features for both real-time and static
visualization as observed in a red-blue (attack - defend) team
cyber challenge [10]. In a real-time visualization scenario, we
created an alert system technique to capture our targeted users
(viewers) with a goal to increase their attention span. We refer
to this technique as ‘SvEm:cognitive-activators’:
1) ‘Semi-permanent Hold’ activator: An animated feature
(shown in Figure. 14) allowing a critical (suspicious)
file of interest being pushed out from the normal visual
pattern and behavior for at least 3 seconds to capture
the viewer’s attention.
2) ‘Permanent Hold’ activator: A permanent colored file
indicator, marking out a malicious (suspicious) file. Red
or yellow is used, depending on how critical the file is.
646
3) ‘Critical-File Detected’ activator: This is an alert iden-
tifier to gain the viewer’s attention.
4) ‘Sound alert’ activator: An additional alert identifier to
gain the viewer’s attention, particularly for color-blind
people.
Fig. 14: Semi-permanent Hold Activator Feature
These SvEm:cognitive-activators are primarily used to iden-
tify and display critical security files and malicious attributes.
Files and attributes within the data visualized are transformed
into several security visualization representation types to pro-
vide knowledge, awareness and aid decision making.
D. Use-case 2: Locky Ransomware Visualization
The Locky Ransomware security visualization seen in Fig-
ure. 15 uses the ‘circular-layered’ design purposely to contain
and maintain the user’s concentration entirely to focus on
the visualization provided. Our circular design allows mobile
platforms to accommodate the data used by building visual
layers on top of each other to represent classifications of
libraries, processes and files within the infected system/kernel.
The ability to visually see how Locky Ransomware traverse
through the system in a reconnaissance step of attack to iden-
tify files (.docx, .png, .jpeg, .xlsx, etc.) before encrypting them
gives the user a clear understanding of how the Ransomware
works. Encrypted files are then highlighted in red to indicated
that the file has been encrypted.
Encrypted files (critical files) are marked red which allow
viewers to select (mouse-over, click, etc) them for further anal-
ysis (See Figure. 16). With these visualization effectiveness
features, users are attracted and motivate to investigate while
interacting with the visualization for a longer time span.
E. Use-case 3: Effective Interactions with Augmented Reality
Visualization
An Augmented Reality (AR) experience for users in security
visualization activates a whole new realm of experience for
users. It increases the viewer’s attention span to security

Fig. 15: A Visualization of Locky Ransomware Encrypting
Files
Fig. 16: Security Visualization of Encrypted Files by Locky
Ransomware
incidents through the use of visualization. Providing view-
ers with a 3-dimensional visualization experience with color
interpretations contribute to higher performance rating with
a lower mental effort usage [28]. In return, viewers are
drawn to exploring security details to acquire relevant security
knowledge.
Our augmented reality visualization (Figure. 17) empowers
mobile users to utilize personal mobile platforms to visualize
a red-blue team (attack/defend) cyber challenge event. Estab-
lishing an augmented reality experience increases the viewer’s
perception capability [41]. Thus, with the right information
filtered through AR visualization, viewers were able to pro-
cess large amounts of data (information) and draw effective
decisions leading to the next potential security step.
647
Fig. 17: Augmented Reality Visualization Frontend Design
In Figure. 17 various colored spheres representing different
attack attributes are utilized to show and observe how cyber-
attacks are executed in a real-time simulated cyber challenge
environment. The interactive AR visualization enable users to
interact and understand type of attacks which are executed by
the red teams.
F. Visualization scaling for Mobile Display Dimensions
Due to the large volume of data collected regularly, com-
fortably visualizing cyber-attacks in mobile platforms verses
mobile platform limitations creates the need for smarter visual-
ization ideas and types. Parallel coordinates [9],[19] provided
us the opportunity to create multidimensional visualization
designs which can accommodate reasonable volume of data for
visualization. Figure. 18 shows multiple security data traffic
between the application, system and network layer.
VI. E VALUATION AND VALIDATION OF S V E M
F RAMEWORK
A. SvEm Conceptional Model
Based on an intensive research across computing science
and psychology domain, we have constructed our SvEm model
to translate a full-scale effectiveness measurement approach
across various vital entities: the user, human cognition and
security visualization. This led us to develop our conceptual
model which incorporated the user’s entire visualization expe-
rience with the security incident presented. Figure. 19 reveals
the SvEm model with all effectiveness entities: (1) user, (2)
visualization and the connecting factor (3) user cognition.
Incorporating these components creates linking relationships
and ultimately, aim of retrieving security insights from visu-
alization.
The inner SvEm model mechanism whereby perception
emerges as a result of incorporating user cognition and
the visualization together. As a result of user’s observation,
‘pre-attentive processing’ queries occur during the event of

Fig. 18: Multidimensional Security Visualization with Parallel
Coordinates
Fig. 19: SvEm Model Illustrating all Components linked
together
perceptual process. In the event of confronting a security
visualization, ‘mental effort’ resulting from the user (viewer),
leverages on his/her cognitive capabilities which involves the
thinking process. This entire process builds the user’s working
memory load towards the particular security visualization pre-
sented. Taking into consideration the known link relationship
between users -¿ user-cognition -¿ visualization, our effective-
visualization techniques creates the final relationship link
between the user and visualization presented. As a result,
‘Distortion and/or Time’ is identified and measured. Obtaining
SvEm insights are triggered when all SvEm components are
harmonized whereby mental effort reading is low, therefore
the right security information are transformed and translated
for the user to process.
B. SvEm Performance Testing
We executed a performance testing for our SvEm framework
for the following areas: (1) Visualization clarity and represen-
648
tation, (2)Back-end to Front-end data transfer performance,
and (3) SvEm:cognitive-activators presentation assessment.
Visualization clarity and representation testing were executed
during the application development stages. These includes
designing predefined data nodes based on the security visual-
ization (SCeeVis) standard. For example, The use of WebGL
with a 3-dimensional visualization presentation enables a new
interactive visual view for the user’s experience to process
more information. The application design has allowed large
volumes of data to be processed and presented in our security
visualization front-end.
Fig. 20: VisualProgger Application Performance Assessment
Figure. 20 delivers the data transfer performance assess-
ment. In this example, different executable (.exe) data transfer
time (ms) performance have been captured with an average
time. Additionally, we have also used various data types and
sizes for the performance assessment.
C. SvEm User Assessment
The assessment concluded that for a visualization to be
effective, working memory load is crucial for a higher per-
formance reading. User response were used to measure SvEm
effectiveness in our framework. Color standards contributed
to the user performance by enabling users to process patterns
and behaviors faster through classification and tracking/tracing
of links. When users understand the color standards (guides),
approaching the visualization enabled them to process and
connect the dots faster when known security attributes pre-
sented in the given visualization.
The introduction of SvEm:cognitive-activators into our vi-
sualization framework created the user-centric perspective
allowing users to be alert and watchful for security events. This
automatically activates the viewer’s cognition thus motivating
him/her to interact further with the visualization presented.
D. Cognitive Load Assessment expected results. This enabled us to verify our threat detection
Past research in psychology have heavily invested in user scoring system performance. A normal action will be scored
studies and theoretical proofs [7],[8] helped understand cog- 10-80, where an anomaly behavior will be scored with a
nitive loads in users. From a less psychological approach, negative value. Likewise, suspicious files in the systems are
we performed a cognitive and working memory load method also scanned against a stored signature-base Database. Fig-
with the concept of linking the user’s perception with their ure. 22 visually illustrate normal versus abnormal behaviors
cognition process to understand relationships between per- and malicious records. In addition, scanning files within the
ception, cognition and the SvEm framework. These are ex- system and having a pre-configured log history helps identifies
ecuted when the user’s mind has the ability to perceive known file paths. Therefore if a known or suspected file
and apprehend objects (e.g. security visual nodes) through appears in another location, automatically this file is flagged
interactive security visualization. Thus, the process allow users with a yellow or red color. We evaluated our threat scoring
to think of key words relevant to the security visual nodes system performance against the ground truth datasets and the
presented thus enhance their perception process which relates several anomaly algorithms and malicious signatures are used
back to past/previous visualization experiences. This process as filters. Our scoring system utilizes Progger (Logging mech-
is executed as a result of having a high memory working anism) to flag out anomalies and malicious files. Hence,‘files
load/capacity in the security incident presented. of interest’ are visually represented in the following color
codes: malicious (red in color), suspicious (yellow in color),
intelligence tracking (blue in color) and a normal legitimate
data (green in color).
^ǀŵͲtD>KďƐĞƌǀĂƚŝŽŶ^Ğƚ͘ϭ
ϭϰϬ

LJсϴ͘ϳϬϰϲdžнϮϮ͘ϰϮϵ
ZϸсϬ͘ϳϰϲϮ
ϭϮϬ

LJсϳ͘ϱϲϲϴdžнϭϯ͘Ϯϯϱ
ϭϬϬ ZϸсϬ͘ϵϬϵϵ
dŝŵĞŝŶ^ĞĐŽŶĚƐ

ϴϬ

ϲϬ

ϰϬ

ϮϬ

Ϭ
Ϭ Ϯ ϰ ϲ ϴ ϭϬ ϭϮ ϭϰ
^ĞĐƵƌŝƚLJsŝƐƵĂůEŽĚĞƐ/ĚĞŶƚŝĨŝĞƌ

DĞŵŽƌLJĨĨŝĐŝĞŶĐLJ;ƚͺŵĞͿ ŽŐŶŝƚŝǀĞ>ŽĂĚ;ůͿ
>ŝŶĞĂƌ;DĞŵŽƌLJĨĨŝĐŝĞŶĐLJ;ƚͺŵĞͿͿ >ŝŶĞĂƌ;ŽŐŶŝƚŝǀĞ>ŽĂĚ;ůͿͿ

Fig. 21: A Comparison Assessment of Cognitive and Working

Memory Load in Viewers Fig. 22: Anomaly Detection System Results

Hence we assessed various simulated user experiment and

got the following results shown in Figure. 21. This experiment VII. C ONCLUSION
has shown that there consistency in the viewers working This paper presented a full-scale security visualization ef-
memory and cognitive load performance, i.e. as working mem- fectiveness measurement framework with techniques ranging
ory load increases, cognitive load also increased as well but from technical to user-centric aspects. A conceptual model
maintains a limit. In Figure. 21 the lines of best-fit shows that was presented to illustrate how all entities interact with each
both performance are linear and cognitive load has a consistent other to obtain and improve effectiveness measurement in se-
load (capacity) limit, hence does not overlap working memory curity visualization. Implementing SvEm:cognitive activators
load performance in a user. This is the ideal situation a user captures the viewers attention and increases the user’s attention
(viewer) would be in when observing and analyzing security span. We evaluated our SvEm framework against existing
visualizations. frameworks and ground-truth datasets. Finally, we conclude
that users perform well with a higher security working memory
E. Threat Scoring Detection System load and effectively interact with the given visualization to
A selection of known signature-base anomaly and malicious acquire relevant security insights.
detection algorithms have been incorporated to filter collected
Progger data. Based on ground truth datasets, we initially A. Future Work
selected Local Outlier Factor (LOF) [22], DBscan [23] and In future, we would like to further evaluate our framework
K nearest neighbor (KNN) [25] to see which satisfied our by conducting effectiveness measurements for users across

649
different domains (health, finance education, etc.) - carrying
out further analysis on how users interact and respond.
ACKNOWLEDGMENT
The authors wish to thank Mark A. Will, Cameron Brown,
Meena Mungro, the members of Cyber Security Researchers
of Waikato (CROW Lab) and the contributions of our in-
terns [Isaiah Wong, Jia Cheng Yip, Wen Liang Goh, Xin
Li Yuan] from Nanyang Polytechnic, Singapore. This project
is supported by STRATUS (Security Technologies Returning
Accountability, Trust and User-Centric Services in the Cloud)
(https://stratus.org.nz), a science investment project funded
by the New Zealand Ministry of Business, Innovation and
Employment (MBIE)). This work was also supported in part
by the New Zealand and Pacific Scholarship Programme
(NZAid).
R EFERENCES
[1] M. Anderson. Policing the world: Interpol and the politics of interna-
tional police co-operation. Clarendon Press Oxford, 1989.
[2] J. L. Carlson. Redis in Action. Manning Publications Co., Greenwich,
CT, USA, 2013.
[3] K. Chodorow. MongoDB: The Definitive Guide: Powerful and Scalable
Data Storage. ” O’Reilly Media, Inc.”, 2013.
[4] H. Choo and S. Franconeri. Enumeration of small collections violates
webers law. Psychonomic bulletin & review, 21(1):93–99, 2014.
[5] J. Congote, A. Segura, L. Kabongo, A. Moreno, J. Posada, and O. Ruiz.
Interactive visualization of volumetric data with webgl in real-time. In
Proceedings of the 16th International Conference on 3D Web Technol-
ogy, pages 137–146. ACM, 2011.
[6] EOOD and M. Angelov. 20 Impressive Examples for Learning WebGL
with Three.js, Nov. 2017.
[7] C. Firestone and B. J. Scholl. Enhanced visual awareness for morality
and pajamas? perception vs. memory in top-downeffects. Cognition,
136:409–416, 2015.
[8] C. Firestone and B. J. Scholl. Cognition does not affect perception:
Evaluating the evidence for” top-down” effects. Behavioral and brain
sciences, 39, 2016.
[9] Y.-H. Fua, M. O. Ward, and E. A. Rundensteiner. Hierarchical parallel
coordinates for exploration of large datasets. In Proceedings of the
conference on Visualization’99: celebrating ten years, pages 43–50.
IEEE Computer Society Press, 1999.
[10] J. Garae, R. K. Ko, J. Kho, S. Suwadi, M. A. Will, and M. Apperley. Vi-
sualizing the new zealand cyber security challenge for attack behaviors.
In Trustcom/BigDataSE/ICESS, 2017 IEEE, pages 1123–1130. IEEE,
2017.
[11] J. Garae and R. K. L. Ko. Visualization and Data Provenance
Trends in Decision Support for Cybersecurity, pages 243–270. Springer
International Publishing, Cham, 2017.
[12] L. Harrison, F. Yang, S. Franconeri, and R. Chang. Ranking visu-
alizations of correlation using weber’s law. IEEE transactions on
visualization and computer graphics, 20(12):1943–1952, 2014.
[13] M. G. Haselton, D. Nettle, and D. R. Murray. The evolution of cognitive
bias. The handbook of evolutionary psychology, 2005.
[14] J. Heer, F. B. Viégas, and M. Wattenberg. Voyagers and voyeurs:
supporting asynchronous collaborative information visualization. In
Proceedings of the SIGCHI conference on Human factors in computing
systems, pages 1029–1038. ACM, 2007.
[15] V. A. C. Henmon. The time of perception as a measure of differences
in sensations. Number 8. Science Press, 1906.
[16] R. V. Hogg and A. T. Craig. Introduction to mathematical statistics.(5””
edition). Upper Saddle River, New Jersey: Prentice Hall, 1995.
[17] W. Huang, P. Eades, and S.-H. Hong. Measuring effectiveness of graph
visualizations: A cognitive load perspective. Information Visualization,
8(3):139–152, 2009.
[18] J. J. Imhoff and S. P. Cutler. Interpol: Extending law enforcement’s
reach around the world. FBI L. Enforcement Bull., 67:10, 1998.
650
[19] A. Inselberg and B. Dimsdale. Parallel coordinates for visualizing
multi-dimensional geometry. In Computer Graphics 1987, pages 25–
44. Springer, 1987.
[20] M. Kay and J. Heer. Beyond weber’s law: A second look at ranking
visualizations of correlation. IEEE transactions on visualization and
computer graphics, 22(1):469–478, 2016.
[21] R. K. Ko and M. A. Will. Progger: an efficient, tamper-evident kernel-
space logger for cloud data provenance tracking. In Cloud Computing
(CLOUD), 2014 IEEE 7th International Conference on, pages 881–889.
IEEE, 2014.
[22] A. Lazarevic, L. Ertoz, V. Kumar, A. Ozgur, and J. Srivastava. A
comparative study of anomaly detection schemes in network intrusion
detection. In Proceedings of the 2003 SIAM International Conference
on Data Mining, pages 25–36. SIAM, 2003.
[23] K. Leung and C. Leckie. Unsupervised anomaly detection in network
intrusion detection using clusters. In Proceedings of the Twenty-eighth
Australasian conference on Computer Science-Volume 38, pages 333–
342. Australian Computer Society, Inc., 2005.
[24] Y. K. Leung and M. D. Apperley. E3: Towards the metrication of
graphical presentation techniques for large data sets. In International
Conference on Human-Computer Interaction, pages 125–140. Springer,
1993.
[25] Y. Liao and V. R. Vemuri. Use of k-nearest neighbor classifier for
intrusion detection. Computers & security, 21(5):439–448, 2002.
[26] P. E. McKight and J. Najab. Kruskal-wallis test. Corsini Encyclopedia
of Psychology, 2010.
[27] T. Okoshi, J. Ramos, H. Nozaki, J. Nakazawa, A. K. Dey, and H. Tokuda.
Attelia: Reducing user’s cognitive load due to interruptive notifications
on smart phones. In Pervasive Computing and Communications (Per-
Com), 2015 IEEE International Conference on, pages 96–104. IEEE,
2015.
[28] T. Olsson, E. Lagerstam, T. Kärkkäinen, and K. Väänänen-Vainio-
Mattila. Expected user experience of mobile augmented reality services:
a user study in the context of shopping centres. Personal and ubiquitous
computing, 17(2):287–304, 2013.
[29] Ö. Öztürk and D. A. Wolfe. An improved ranked set two-sample mann-
whitney-wilcoxon test. Canadian Journal of Statistics, 28(1):123–135,
2000.
[30] F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. Van Gerven. Cognitive
load measurement as a means to advance cognitive load theory. Educa-
tional psychologist, 38(1):63–71, 2003.
[31] T. Parisi. WebGL: up and running. ” O’Reilly Media, Inc.”, 2012.
[32] P. Pirolli and S. Card. The sensemaking process and leverage points
for analyst technology as identified through cognitive task analysis.
In Proceedings of international conference on intelligence analysis,
volume 5, pages 2–4, 2005.
[33] R. A. Rensink and G. Baldridge. The perception of correlation in
scatterplots. In Computer Graphics Forum, volume 29, pages 1203–
1210. Wiley Online Library, 2010.
[34] P. Saraiya, C. North, and K. Duca. An insight-based methodology
for evaluating bioinformatics visualizations. IEEE transactions on
visualization and computer graphics, 11(4):443–456, 2005.
[35] P. Saraiya, C. North, V. Lam, and K. A. Duca. An insight-based lon-
gitudinal study of visual analytics. IEEE Transactions on Visualization
and Computer Graphics, 12(6):1511–1522, 2006.
[36] E. W. Weisstein. Bonferroni correction. 2004.
[37] M. Wertheimer. A brief introduction to gestalt, identifying key theories
and principles. Psychol Forsch, 4:301–350, 1923.
[38] D. A. Wheeler and G. N. Larsen. Techniques for cyber attack attribution.
Technical report, INSTITUTE FOR DEFENSE ANALYSES ALEXAN-
DRIA VA, 2003.
[39] J. R. Wilson and J. Carter. Node. js the right way: Practical, server-side
javascript that scales. Pragmatic Bookshelf, 2013.
[40] J. S. Yi, Y.-a. Kang, J. T. Stasko, and J. A. Jacko. Understanding and
characterizing insights: how do people gain insights using information
visualization? In Proceedings of the 2008 Workshop on BEyond time and
errors: novel evaLuation methods for Information Visualization, page 4.
ACM, 2008.
[41] F. Zhou, H. B.-L. Duh, and M. Billinghurst. Trends in augmented reality
tracking, interaction and display: A review of ten years of ismar. In
Proceedings of the 7th IEEE/ACM International Symposium on Mixed
and Augmented Reality, pages 193–202. IEEE Computer Society, 2008.