Four Levels of Validation

Chapter 5
Analysis: Four Level for Validation
Vis/Visual Analytics, Chap 5 Validation 1 CGGM Lab., CS Dept., NCTU Jung Hong Chuang
Contents
• Why Validate? • Validation Approaches
• Why is Validation – Domain
difficult? – Abstraction
• Four Levels of Design – Idiom
– Algorithm
– Domain Situation
– Mismatches
– Task & Data Abstraction
– Visual Encoding & • Examples
Interaction – Genealogical Graphs
– Algorithm – MatrixExplorer
• Angles of Attack – … and 4 more
• Threats to Validity
– Different for what, why,
how
Why Validate?
• Why?
– The vis design space is huge, and most
designs are ineffective
– Think about how you might validate your

choices from the very beginning of the
design space, rather than leaving it at the
end
Four Nested Levels of Design
• Why four nested
levels?
– Splitting the
complex vis design
into four cascading
levels provides an
analysis framework
that lets you to
address different
concerns separately
• Four nested levels
– Consider the details
of a particular
application domain
– The what-why
abstraction
• Map domain-
specification
problems and data
into forms that are
independent of the
domain
– How level: design of
idioms that specify
the visual encoding
and interaction
– Design of algorithms
to instantiate those
idioms
computationally
– The output from an
upstream level is
input to the
downstream below
– Choosing a poor
choice at an
upstream level
inevitably cascades
to all downstream
levels
• Domain situation
– A domain situation
includes
• a group of target users
– e.g. computational biologists
• domain of interest
– comparative genomics
• Questions
– e.g. genetic source of
adaptivity in a species
• Data
– e.g. genomic sequence data
– Methods to identify
domain
• Interviews
• observations
• careful research
User introspection is
insufficient!
– Output
• A detailed set of
questions asked or
actions carried out by
the target uses, about
a possibly
heterogeneous
collection of data
Two of the questions asked by the computational biologist working

on the comparative genomics:
1. What’s the difference between individual nucleotides (核甘酸) of feature pairs
2. Where are the gaps across a chromosome (染色體)?
Other:
What is the genetic basis of a disease? (X, not specific enough)
• Data/Task Abstraction
– Domain-specific into
generic representation
• Identify abstract tasks
– Ex. Browsing, comparing,
summarizing
• Design abstract data
forms
– Are designed, a creative
design step
» Often choose to
transform the original
data to something
quite different
• Data/Task Abstraction
– Explicitly considering the
choices made in
abstracting tasks and
data can be very useful in
design process.
– Bad alternative: to do
this implicitly and w/o
justification
» Solving the “lost in
hyperspace” problem
done by showing the
searcher a website
hyperlink connectivity
graph?
» Wrong, too much
cognitive load.
• Visual encoding and
interaction idiom
– Decide on the specific
way to create and
manipulate the visual
rep. of the abstract
data block, guided by
the abstract tasks
– Idiom: each distinct
possible approach
• Visual encoding
– How to represent data
visually (what users see)
– Idiom
• Visual encoding
– How to represent data
visually (what users see)
• Interaction
– How to manipulate that
representation
dynamically (how users
change what they see)
• Possible to analyze
encoding and interaction
as separate decision. In
some cases, need to be
considered as a combined
idiom
– Idiom blocks are
designed
• A big design space…
• Abstracting data and task
can be used to rule out
many bad options
• Should make decisions
about good and bad
matches based on
understanding human
abilities, especially
– Visual perception
– memory
Word Tree combines the visual encoding idiom of a hierarchical tree of

keywords laid out horizontally and the interaction idiom of navigation
based on keyword selection.
• Algorithm
– A detailed procedure
that allows a computer
to automatically carry
out the desired goal
• To efficiently handle visual
encoding and interaction
– Are designed
• Could have many
algorithms for the same
idiom. Ex. Many algorithms
for direct volume rendering Visual/interaction:
– Primary concerns: primary concerns are human
computational issues perceptual issues
Four Levels of Design
• Dependency:
– Wrong block upstream
cascades downstream
choices
• poor task, perfect
idiom => X
• Iterative process:
– Consider each level
separately
– A better
understanding of one
block will refine other
levels
Angles of Attack for Vis Design
• Two angles of attack for vis design
– Top down (problem-driven) or bottom up
(technique driven)
• Problem-driven (Top-down)
– Start with the problems of real-world user and
attempt to design solution that that helps them
work more effectively
– Often the problem can be solved using existing
visual coding and interaction idioms
• Much of the challenge lies at the abstraction level
– Sometimes the problem motivates the design of
new idioms, if no existing ones will adequately
solve the abstracted design problem
• Problem-driven (Top-down)
– Considering the four levels of nested model
explicitly can help you avoid the pitfall of
skipping important steps in problem-driven
approach
• Some designers skip over the domain situation level
completely, short-circuit the abstraction level by
assuming that the first abstraction is right and jump
immediately into the third level
• THE ABSTRACTION LEVEL IS OFTEN THE HARDEST TO
GET RIGHT!!
• The design process for problem-driven work involves
iterative refinement at all levels.
• Technique driven (Bottom-Up)
– Start with idiom or algorithm design
• Goal is to invent new idioms that better support
existing abstractions, or new algorithms that better
support existing idioms
– Considering the four nested model can help you
articulate your assumptions at the level just
above your focus
• Articulate the abstraction requirement for new idiom,
or articulate the idiom requirement for new algorithm
Treats to Validity
• Validating the effectiveness of a vis design
is difficult because there are so many
possible questions
– Considering the validity questions at each level
separately helps
– Each level has a different set of treats to validity
• Different fundamental reasons why you might have
made wrong choices
Threats to Validity
Validation Approaches
• Different treats require very different
approaches
• Difference between immediate and
downstream validation
– Most kinds of validation for the outer levels are
not immediate because they require results
from the level nested within them
– A poor showing of a test may misdirect
attention upstream, when in fact the problem
results from a poor choice at the current level
– Downstream validation is necessary!!
– The immediate validation only offer partial
evidence of success
• Domain Validation
– The primary treat is that the problem is
mischaracterized
• The target users do not in fact have these problems
– An immediate validation is to interview and
observe the target users to verify the
characterization
• Field study
– Observe how people act in real-world scenarios
– Ask questions when clarification is needed
– Interview people about their needs
– Downstream validation: Report adoption rates
– What the target users do of their own accord
• Abstraction Validation (downstream)
– The treat: the identified task abstraction and
the designed data abstraction do not solve the
characterized problems
– Key validation: the system must be tested by
target users doing their own work, rather than
doing an abstracted task specified by the
designer
• Collect anecdotal evidence that the tool is in fact useful
– insights found or hypothesis confirmed
– A more rigorous validation: to conduct a field
study to observe and document how the target
user uses the deployed system
• Idiom Validation
– Treat: the chosen idioms are not effective at
communicating the desired abstraction to the
users
– One immediate validation:
carefully justify the idiom w.r.t. known
perceptual and cognitive principles
• Heuristic evaluation
• Expert review
– A downstream validation
• Lab study
– A controlled experiment in a lab setting
– Tease out the impact of specific idiom choice by measuring
human performance on abstract tasks
– Quantitative measurement
» Time spent, errors made
» Performance
» Logging actions such as mouse moves and clicks
» Tracking eye movements
– Qualitative measurement
» Reflect about the strategies through questionnaires
» The number of participants needs to be sufficient for
statistical significance
– Another downstream validation
• Presentation of and qualitative discussion of results in
the form of still images or video
• Sometimes occur as usage scenarios, supporting that
the tool is useful for a particular task-data abstraction
– Third downstream validation
• The quantitative measures of result images by using
quality metrics
– e.g. Measure # of edge crossings and edge bends for
graphs
– Informal usability study
• Lead to better and more usable systems, but neither
offer validation nor provide evidence of the superiority
of an approach for a particular context
• Algorithm Validation
– Threat: the algorithm is suboptimal
• Either to a theoretical minimum or in comparison with
previous methods
• Is the algorithm correct?
Does it have good performance?
– Immediate validation
• Analyze complexity and memory
– Downstream validation
• Measure wall-clock time & memory usage
• Primary consideration: scalability, how data size affect
the speed
• One trickier question: what data used?
– Use standard benchmarks
• Algorithm Validation
– Another threat: incorrectness at the algorithm
• Poor algorithm design, or
• The implementation
• Mismatches
– Mismatch between the level at which the
benefit is claimed and the validation
methodology
– Example
• The benefit of a visual encoding cannot be validated by
wall-clock timings of the algorithm In practice, not all
validation tests are adopted
– The nested model explicitly separates the
design problem into levels in order to guide
validation according to the unique threats at
each level
Validation Examples
• Genealogical (宗譜) Graphs
– Proposed
• Multiple new visual encoding
– Classical dual-tree, classicial left-to-right, new indented
graph
• Interaction idioms
– Automatic camera framing, animated transitions, and a
new widget for ballistically dragging out subtrees to
arbitrary depth
– Explicitly cover all four levels
– Domain is genealogy, discuss the need and current tools
Validation Examples
• Abstraction level
– Point out the term fam ily tree is highly misleading because
the data type is a more general graph with specialized
constraints on its structure
– Discuss conditions for which the data type is a tree, a
multi-tree, or a directed acyclic graph
– Map the domain problem of recognizing nuclear family
structure into an abstract task of determining subgraph
structure
• Visual encoding and interactions
– Discuss the strengths and weakness of several visual
encoding design choices
» Connection, containment, adjacency and alignment,
indentation
– Address interaction idiom design
• Algorithm design
– Dual-tree layout
Validation Examples
Genealogical Graphs
Genealogical graphs. (a) Three layouts for the dual-tree:

classical node–link top-to-bottom at the top, classical left-to-right on the left,
and the new indented outline algorithm on the right.
(b) Widget for subtree collapsing and expanding with ballistic drags.
Validation Examples
– Validation
• Abstract level
– Downstream informal testing of a system prototype with a
target user to collect anecdotal evidence
• Visual encoding and interaction design
– Immediate justification of Established principles
– Downstream method of a qualitative discussion of result
images and video
Validation Examples
• MatrixExplorer for social network analysis
– Proposed
• Matrix representation to minimize clutter for large and
dense graph
• More intuitive node-link graph for smaller network
– Explicitly cover all four levels
– Explicit characterization of social network analysis domain
– Validated with qualitative interviews and exploratory
study with social scientists
• Abstraction level
– Include a detail list of the requirements in abstract form
Validation Examples
• MatrixExplorer for social network analysis
• Visual encoding and interaction
– A thorough discussion of the primary encoding idiom
– Discussion of basic and more complex interaction
» Interactive reordering and clustering
– Validation
» Use the immediate validation
» An extensive downstream validation using qualitative
discussion of result images
• Algorithm design
– Focus on reordering algorithm
– Validation
» Downstream benchmark timing
Validation Examples
(MatrixExplorer)
MatrixExplorer features both node–link and matrix representations in an

interface designed for sociologists and historians to explore social
Networks (Munzner 84)
Validation Examples
(MatrixExplorer)
MatrixExplorer validation methods
Validation Examples (Flow Maps)
Flow maps showing migration patterns from 1995 to 2000 US Census

data. (a) Migration from California. (b) The top ten states that sent
migrants to California shown in green, and to New York in blue
(Munzner 86)
Validation Examples (Flow Maps)
Validation Examples (LiveRAC)
LiveRAC supports exploration of system management time-series data with

a reorderable matrix and semantic zooming. (a) The first several dozen rows
have been stretched out to show sparklines for the devices. (b) The top three
rows have been enlarged more, so the charts appear in full detail.
(Munzner 87)
Validation Examples (LiveRAC)
Validation Examples (LinLog)
Validation Examples (Sizing the
Horizon)

Four Levels of Validation

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Four Levels of Validation

Uploaded by

Copyright:

Available Formats

Chapter 5

Analysis: Four Level for Validation

– Think about how you might validate your

Two of the questions asked by the computational biologist working

Word Tree combines the visual encoding idiom of a hierarchical tree of

Genealogical graphs. (a) Three layouts for the dual-tree:

MatrixExplorer features both node–link and matrix representations in an

MatrixExplorer validation methods

Flow maps showing migration patterns from 1995 to 2000 US Census

LiveRAC supports exploration of system management time-series data with

You might also like