Building Confidence in the Model

Verification and Validation

Ruang L301-302
Gedung Laboratorium
Departemen Teknik Industri
Fakultas Teknik
Universitas Indonesia
systems.ie.ui.ac.id
Agenda

• Principles of Verification and Validation

• DEM Validation
• SD Validation
All Models are Wrong
..but some are useful

• So the “real” Validation and Verification are impossible to achieve

• No Model has ever been or ever will be thoroughly validated ..
• “useful”, “illuminating”, “convincing”, or “inspiring confidence” are more apt
descriptors applying to models than “valid” Greenberger, Crenson and Crissey
(1976)
• Falsity (Falsifications)
• Hypothesis: “All swans are white, because we never seen black swan”. It is true
until we find one swan black (false evidence)
• Only theory that have not yet been refuted should be accepted. However, that
acceptance is always conditional
• Therefore, if we wait for the ultimate false-proof theory, we never have a theory
then?

“No model can claim absolute objectivity, for every model carries in it the modeler’s worldview.
Models are not true or false, but lie in a continuum of usefulness”
(from Barlas & Carpenter, 1990)
The model will not tell you what to do, it only tells you what will
happen if you do it
• Rationale of Model result
Challenges of Verification and Validation

• There is no such thing as general validity

• There may be no real world to compare against (Which real world? Each
persons sees the world differently)
• Often the real-world data are inaccurate
• There is not enough time to verify and validate everything
• Confidence not validity, but modeler confidence can help model “subjective”
validity
Both should be conducted simultaneously

Validation Definition: The assurance that a

product, service, or system meets the needs
of the customer and other identified Are you building the right things?
stakeholders. It often involves acceptance
and suitability with external customers.

Verification Definition: The evaluation of

whether or not a product, service, or system
complies with evaluation criteria (a Are you building things right?
regulation, requirement, specification, or
imposed condition) of a good simulation
models

Real
World
validation validation

Simulation verification Conceptual

Models Model
The complete picture of verification and validation interaction in
models
Data Validation
• Model-Data Comparison with Real-Data
Validating the Existing System
• In this approach, we use real-world inputs of
the model to compare its output with that of the
real-world inputs of the real system. This
process of validation is straightforward,
however, it may present some difficulties when
carried out, such as if the output is to be
compared to average length, waiting time, idle
time, etc. it can be compared using statistical
tests and hypothesis testing. Some of the
statistical tests are chi-square test,
Kolmogorov-Smirnov test, Cramer-von Mises
test, and the Moments test.
Validating the “First Time” Model
• Consider we have to describe a proposed
system which doesn’t exist at the present nor
has existed in the past. Therefore, there is no
historical data available to compare its
performance with. Hence, we have to use a
hypothetical system based on assumptions.
• Subsystem Validity − A model itself may not
have any existing system to compare it with, but
it may consist of a known subsystem. Each of
that validity can be tested separately.
• Internal Validity − A model with high degree of
internal variance will be rejected as a stochastic
system with high variance due to its internal
processes will hide the changes in the output
due to input changes.
• Sensitivity Analysis − It provides the
information about the sensitive parameter in the
system to which we need to pay higher attention.
• Face Validity − When the model performs on
opposite logics, then it should be rejected even if
it behaves like the real system.
Model Testing in Practice could be divided into 2 major groups…
Protective - explains
• Prove a point
• Keep assumptions hidden
• Use data selectively
• Support preconceptions and buttress
preselected answers
• ..and cover up the preselection
• Promote the authority of the modeler
Predictive Modeling
Reflective - explore
• Promote Inquiry
• Expose hidden assumptions
• Motivate widest range of empirical test
• Challenge preconceptions and support multiple
viewpoints
• …and involve the widest community
• Promote the empowerment of the clients
Exploratory Modeling
Purpose, Suitability, and Boundary

• What is the purpose of the model?

• What is the boundary of the model?
• Is the level of aggregation consistent with the purpose?
• Are the issues important to the purpose treated endogenously?
• What important variables and issues are exogenous, or excluded?
• Are important variables excluded because there are no numerical data to
quantify them?
• What is the time horizon relevant to the problem?
• Does the model include the factors that may change significantly over the time
horizon as endogenous elements?
Physical and Decision-Making Structure

• Does the model conform to basic physical laws?

• Are equations dimensionally consistent without fudging?
• Is the stock-flow structure explicit/consistent with purpose?
• Does the model represent disequilibrium dynamics or does it assume the
system is in or near equilibrium all the time?
• Are time delays, constraints, and bottlenecks included?

Robustness and Sensitivity to Alternative Assumptions

• Is the model robust in the face of extreme variations in input conditions or
policies?
• Are the policy recommendations robust/sensitive to plausible variations in
assumptions, including assumptions
• about parameters, aggregation, and model boundary
Policy Models: Pragmatics and Politics of Model Use

• Is the model documented? Is the
documentation publicly available?
• Can you run the model on your own
computer?
• What types of data were used to
develop and test the model?
• (e.g., aggregate statistics collected by
third parties, primary data sources,
observation and field-based qualitative
data, archival materials, interviews)?
• Are processes used to test and build
confidence described?
• Did independent critics and third parties
review the model?
• Are the results of the model
reproducible?
• Are the results fudged by the modeler?
• How much does it cost to run the
model? Does the budget permit
adequate sensitivity testing?
• How long does it take to revise and
update the model?
• Is the model being operated by its
designers or third parties?
• What are the biases, ideologies and
political agendas of the modelers and
clients?
• How might these biases affect the
results, both deliberately and
inadvertently?
Policy Models: Model Test Categorization
• Since there is so many technique and approach on verification and validation you can group
them into 3 generic categoru

Model Tests

Model Process
Questions
Structure
Purpose
Suitability Model
Consistency Output
Utility and
Behavior
Effectiveness
Policy Impacts
Implications
Measuring Model Success

• Quality of the Content

• the extent to which the technical work within
the modelling process conforms to the
requirements of the study.
• Quality of the Process:
• the extent to which the process of the
delivery of the work conforms to the clients’
expectations.
• Quality of the Outcome:
• the extent to which the simulation study is
useful within the wider context for which it is
intended.
Example Table for Validation

Category Technique Used Justification for Reference to Validation

Technique Use Supporting Results/
Report/Paper Conclusions

Theories Accepted Approach

Theoretical
Deviation

Assumptions Derived of Empirical

Data

Model Face Validity

Representation

Historical Data
The Importance of Model Documentation

• Create a standards of model documentation so that different people can

understand each other notes
• Document so other people can replicate the same experiment
• Write the Reports of any problems and model results that is not what you/client
wanted
• Build a single repository so all relevant people can access and comments
Agenda

• Principles of Verification and Validation

• DEM Validation
• SD Validation
Verifikasi Logic Model

• Tujuan :
• Untuk menghindari terjadinya kesalahan logik yang mungkin timbul
• Verifikasi model komputer  Proses debugging
• Correct model coding
• Equation per equation
• Isolate model subsections & test with controlled inputs
• Dimensional analysis
• Go beyond automated testing as in softwares
• Numerical errors
• ‘Numerical Method’ dependent errors
• Appropriate numerical method? App. step size?
• ‘Model’ dependent errors
Proses Verifikasi Logic Model

• Pelajari kembali bahasa • Trace Command

pemrogramannya
• Tujuannya : memeriksa kesalahan dan
ketidakkonsisten suatu perilaku
• Memeriksa output / keluaran yang logis
• Bisa dengan membuat data input
dummy yang outputnya telah
diverifikasi secara matematis (excel,
mathlab dsb)
• Mengamati animasi terhadap tingkah
laku yang benar
• Menggunakan fasilitas ‘trace and
debug’ yang disediakan pada software
• Verifikasi Statis atau Dinamis
• Statis = Trace, walkthrough of logic
• Dynamic = Run with changes
Verifikasi Variabel

• Model mungkin saja mencangkup variabel-variabel yang kurang penting

• Model mungkin saja tidak melibatkan suatu variabel yang signifikan
• Model bisa saja gagal menunjukkan ketelitian hubungan antara variabel-
variabel dan ukuran ketidakefektifan yang digunakan
• Nilai-nilai numerik yang digunakan bisa saja tidak benar dan karenanya dapat
mengurangi kemampuan model dalam mencerminkan perilaku sistem
Mengatasi kendala ketidaktersediaan data untuk verifikasi

• Dilakukan ‘running’ pada model yang telah dikomputerisasi untuk periode

waktu yang pendek sehingga hasilnya dapat dibandingkan dengan kalkulator
tangan
• Dilakukan ‘running’ pada masing-masing bagian dari model yang kompleks
sehingga hasilnya dapat diverifikasi
• Mengganti distribusi kemungkinan yang kompleks dengan distribusi yang lebih
mudah sehingga hasilnya lebih mudah diverifikasi
• Jika memungkinkan dapat dibuat situasi percobaan yang mudah dan dicoba
beberapa kombinasi dari keadaan-keadaan dalam model
Validasi Model Diskrit
• Validasi penampakan (face validity)
• Meninjau kebenaran diagram alir model
atau mekanisme logika model
• Validasi penelusuran (trace validity)
• Menelusuri kebenaran logika model dan
program komputer secara manual
(debugging)
• Validasi rasional (rational validity)
• Menguji kebenaran asumsi-asumsi
yang digunakan pada struktur model
• Animation Validity
• Event Validity
• Kejadian dalam model sama dengan
dunia nyata
• Comparisons to Other Model
• Simulasi di komparasi dengan Analytic
models
• Ke model lain yang juga telah divalidasi
• Extreme Condition Test
• Historical Data Validation
• Internal Validity
• Uji stochastic dengan menjalankan
replikasi berkali-kali. Variabilitas yang
terlalu tinggi bisa meragukan model
• Operational Graphics (Results)
• Parameter Variability (Sensitivity
Analysis)
• Predictive Validation
• Validate with the current accepted
forecast or prediction of system
behavior
• Turing Test
• Can the people with knowledgeable
about operations of a system are asked
if they can discriminate the outputs of
the model with the real system
Agenda

• Principles of Verification and Validation

• DEM Validation
• SD Validation
Main validation steps in Continuous Event Modeling

Structural validity tests:

• Is the structure of the model adequate representation of the real structure?
• Comparison of model equations;
• With real system relationships (empirical structural validation)
• With available theory (theoretical structural validation)

Behavior validity tests:

• Is the model capable of producing an acceptable output behavior?
• Comparison of model generated behavior pattern with the real system
behavior
• Model generated patterns are close enough to real ones?
Test for Assesment of Dynamic Models

Behavior Oriented Structure Oriented

• Boundary Adequacy
• Structure Assessment
• Dimensional Consistency
• Parameter Assesment
• Extreme Conditions
• Integration Error
• Behavior Reproduction
• Behavior Anomaly
• Family Member
• Surprise Behavior
• Sensitivity Analysis
• System Improvement
Boundary Adequacy Test
What
• Are the important concepts addressing the
problem endogenous to the model?
• Does the behavior of the model change
significantly when boundary assumptions
relaxed?
• Do the policy recommendations change when
the model boundary is extended?
How
• Use model boundary charts, subsystem
diagrams, causal diagrams, stock and flow
maps, and direct inspections of the model
equations
• Use interviews, workshops to solicit expert
opinion, archival materials, review of literature,
direct inspection/participation in system
processes, etc.
• Modify model to include plausible additional
structure; make constants and exogenous
variables endogenous, then repeat sensitivity
and policy analysis
Showing the model assumption and boundaries in front of
everything shall increase trust in the modeling effort
• Bull’s Eye Diagram
Structure Assessment Test
What
• Is the model structure consistent with relevant
descriptive knowledge of the system?
• Is the level of aggregation appropriate?
• Does the model conform to basic physical laws
such as conservation laws?
• Do the decision rules capture the behavior of
the actors in the system?
How
• Use policy structure diagram, causal diagrams,
stock and flow maps and direct inspection of
model equations
• Use interviews, workshops to solicit expert
opinion, archival materials, review of literature,
direct inspection/participation in system
processes, etc.
• Conduct partial model tests of the intended
rationality of decision rules
• Conduct lab. experiments to elicit mental
models and decision rules of system
participants
• Develop aggregate sub-models and compare
behavior to aggregate formulations
• Dis-aggregate suspect structures, then repeat
sensitivity and policy analysis
Dimensional Consistency Test
• How each variable has the right dimension, so the final functions is in correct dimension

What
• Is each equations dimensionally consistent
without the use of parameters having no real
world meaning
How
• Use dimensional analysis software or in
PowerSIM this is Built-In
• Inspect model equations for suspects
parameters
Parameter Assesment Test
What
• Are the parameter values consistent with the
relevant descriptive and numerical knowledge
of the system?
• Do all parameters have real world
counterparts?
How
• Use statistical methods to estimate parameters
(wide range of methods available)
• Use partial model tests to calibrate
subsystems
• Use judgmental methods based on interviews,
expert opinion, focus groups, archival material,
direct experience, etc.
• Develop disaggregate submodels to estimate
relationships for use in more aggregate models
Extreme ConditionsTest
What
• Does each equations make sense when its
inputs take on extreme values?
• Does the model respond plausibly when
subjected to extreme policies, shocks and
parameters?
How
• Inspect each equation
• Test response to extreme values of each input,
alone and incombination
• Subject model to large shocks and extreme
conditions. Implement test that examine
conformance to basic laws (eg. no inventory,
no shipments, no labor, no production)
Integration Error Test

What How
• Are the results sensitive to the choice of time • Cut the time step in half and test for changes in
step or numerical integration method? behavior. Use different integration methods
and test for changes in behavior
Behavior Reproduction Test
What
• Does the model reproduce the behavior of
interest in the system (qualitatively and
quantitatively)?
• Does it endogenously generate the symptoms
of difficulty motivating the study?
• Does the model generate the various modes of
behavior observed in the real system?
• Do the frequencies and phase relationships
among the variables match the data?
How
• Compute statistical measures of the
correspondence between model and data:
descriptive statistics (R2, MAE), time domain
methods (autocorrelation functions), frequency
domain methods (spectral analysis) and many
others
• Compare model output and data qualitatively
including modes of behavior, shape of
variables, asymmetries, relative amplitudes
and phasing, unusual events
• Examine response of model to test inputs,
shocks, and noise
Behavior Anomaly Test

What How
• Do anomalous behavior results when • Zero out key effects (loop knockout analysis)
assumptions of the model are changed or
deleted • Replace equilibrium assumptions with
disequilibrium structures
Family Member Test

What How
• Can the model generate the behavior observed • Calibrate the model to the widest possible
in other instances of the same systems range of related systems
Surprise Behavior Test
What
• Does the model generate previously
unobserved or unrecognized behavior?
• Does the model successfully anticipate the
response of the system to novel conditions
How
• Keep accurate, complete and dated records of
model simulation. Use model to simulate likely
future behavior of system
• Resolve all discrepancies between model
behavior and your understanding of the real
system
• Document participation and client mental
model prior to the start of the modeling effort
Sensitivity Analysis Test
What
• Numerical Sensitivity: do the numerical values
change significantly..
• Behavior Sensitivity: do the modes of behavior
generated by the model change significantly..
• Policy Sensitivity: do the policy implications
change significantly..
• ..when ssumptions about parameters,
boundary and aggregation are varied over the
plausible range of uncertainty
How
• Perform univariate and multivariate sensitiviy
analysis
• Use analytic methods (linearization, local and
global stability analysis, etc.)
• Conduct model boundary and aggregation test
above
• Use optimization methods to find parameters
combinations that generate implausible results
or reverse policy outcomes
System Improvement Test

What How
• Did the modeling process help to change the • Design instruments in advance to assess the
system for the better? impact of the modeling process on mental
models, behavior, and outcomes
• Design controlled experiments with treatment
and control groups, random assignment, pre-
intervention and post-intervention assessment