NWC15 JMS Presentation

Foundations of Verification and Validation
A Logical Derivation from the Scientific Theory of Truth
Jonathan Smith
Compusis
NAFEMS World Congress 2015 | 21-24 June | San Diego | California | USA
• Simulation governance
– simulation process flow diagram (ISO 9001, NAFEMS QSS)
• framework and taxonomy of process components
• verification and validation processes
AIAA (1998) ASME (2006) NAFEMS QSS Primer (2014)
• Scientific method
– inductively infer an idealised model from real-world observations
𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇓ ⇓ ⇓ ⇑
𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑖𝑑𝑒𝑎𝑙𝑖𝑠𝑎𝑡𝑖𝑜𝑛 𝑝𝑟𝑒𝑑𝑖𝑐𝑖𝑜𝑛
𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥= 2 ⇒ 𝑦= 2
𝑑𝑒𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑎𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛
𝑚𝑜𝑑𝑒𝑙 𝑑𝑎𝑡𝑎
• Syllogistic model of a problem
major premise minor premise conclusion
𝑝→𝑞 ∧ 𝑞→𝑟 ⟹ 𝑝→𝑟
– engineering example : linear spring system 𝐾𝑢 = 𝐹
mathematical model parametric data solution
𝑢 = 𝐹/𝐾 ∧ 𝐹/𝐾 = 𝑓/𝑘 ⟹ 𝑢 = 𝑓/𝑘
mathematical model ∧ data ⟹ solution
• Example : Determine sources of error
model data solution
𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2
Scientific tests for truth
• Coherence and correspondence tests for truth
– well-established tests for truth from the Philosophy of Science:
• correspondence theory of truth

– external conformance to observed facts
– “A judgment is said to be true when it conforms to the external reality”,
Thomas Aquinas, c.1250
• coherence theory of truth

– internal logical consistency
– “Does it make sense?”
Ref : Russell, B., The Problems of Philosophy, 1912
Sources of error
• Correspondence test – do the premises conform to the observed facts?
model data solution

𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2
– modelling error - incorrect assumptions, etc.

𝑦 = log 𝑥 + 1 ∧ log 𝑥 + 1 = log 2 + 1 ⟹ 𝑦 = log 2 + 1
– data error - incorrect coefficients values, etc.
𝑦= 𝑥 ∧ 𝑥= 9 ⟹ 𝑦=3
2
log(2)+1
3
x log(x)+1
y=f(x)
Sources of error
• Coherence test – is the model logically self-consistent?
model data solution
𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦= 2
– approximation error - discretisation, truncation, etc.
𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⟹ 𝑦 → 2
𝑛→∞
– calculation error - mistakes in arithmetic, bugs in software, etc.
𝑦= 𝑥 ∧ 𝑥= 2 ⟹ 𝑦=5
Sources of error
• Correspondence test • Coherence test

– idealisation error – computation error
• modelling error • approximation error
– incorrect assumptions, etc. – discretisation, truncation, etc.
• data error • calculation error

– incorrect loads, coefficient values, etc – iteration, arithmetic, etc.
–
correspondence

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒
𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

𝑐𝑜ℎ𝑒𝑟𝑒𝑛𝑐𝑒 𝑛→∞
𝑛𝑢𝑚𝑒𝑟𝑖𝑐𝑎𝑙 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑚𝑜𝑑𝑒𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 𝑑𝑎𝑡𝑎 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛
coherence
Sources of error
• Verification and validation definitions
– validation*
• the process of determining the degree to which the:
idealised model is an accurate representation of the real-world
engineering application from the perspective of the intended uses of
the model
• are we solving the correct model?
• test of external conformance to observed facts (correspondence)
– verification*
• the process of determining the degree to which the:
computational model is an accurate representation of the underlying
mathematical model and its solution
• are we solving the model correctly?
• test of internal logical consistency (coherence)
Sources* : What is Verification and Validation?, ASME/NAFEMS, 2010

NAFEMS QSS:2008, NAFEMS
Verification and Validation ≡ Coherence and Correspondence
• Coherence and correspondence

»
correspondence

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑐𝑒

𝑐𝑜ℎ𝑒𝑟𝑒𝑛𝑐𝑒 𝑛→∞
coherence
• Verification and validation

»
correspondence

𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭𝐬 𝑖𝑛𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝐝𝐚𝐭𝐚 𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
𝐛𝐞𝐡𝐚𝐯𝐢𝐨𝐮𝐫
⇑ ⇑ ⇑ ⇑
𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏 𝒗𝒂𝒍𝒊𝒅𝒂𝒕𝒊𝒐𝒏

𝒗𝒆𝒓𝒊𝒇𝒊𝒄𝒂𝒕𝒊𝒐𝒏 𝑛→∞
coherence
Verification and Validation
• Validation is a correspondence test

– idealisation error is reduced by the validation process
• compares model predictions against experimental measurements
• error sources tested
– mathematical model error
– parametric data error
• Verification is a coherence test

– computation error is reduced by the verification process
• compares numerical model against analytical mathematical model solution
error sources tested
– approximation error
– calculation error
Simulation process
• Application example
verification challenges the numerical accuracy of the computation

validation challenges the behavioural accuracy of the idealisation
Source : NAFEMS QSS Primer
Simulation process
•
engineered
1. Simulation application object
experimental data
– idealisation idealisation idealisation
• idealise model from engineered object validation
– mathematical model
idealised model
– parametric data values mathematical
model and data
– approximation
• discretise the model approximation approximation
verification
– numerical model
– discrete data values
computational
model computation
– calculation numerical model

and discrete data
verification
• compute the solution

calculation
calculation
–
verification
verification
• approximation verification
– discretisation, truncation solution
• calculation verification
– Iteration, round-off simulation process
interpretation
– validation V&V process
• idealisation validation behavioural

– mathematical model prediction
– parametric data values
Simulation process
• 2. Software testing application
– computational model
• compute solutions using increasingly refined models
• plot mesh convergence curves for DOFs of interest
– theoretical model
• compute theoretical order of discretisation convergence
• plot mesh convergence curves for DOFs of interest
– order of convergence
• theoretical vs computational
• identify sources of numerical error
– algorithm errors, code bugs
Simulation process
• Conclusions
– Require a simulation process flow diagram for simulation control

• crucial to use an appropriate process flow diagram engineered
object
experimental data
– Base simulation process diagram advantages idealisation idealisation

validation
• logically derived using a syllogistic model idealised model

mathematical
• structure based on logically derived sources of error model and data
• can be used as a base to derive higher level diagrams approximation approximation

verification
computational
– Verification and validation

model computation
numerical model verification
and discrete data
• shown to be correspondence and coherence tests calculation

calculation
verification
solution
• Reference
simulation process
– Smith, J. M., Quality Management in Engineering Simulation, NAFEMS 2014 interpretation
V&V process
behavioural
prediction

NWC15 JMS Presentation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NWC15 JMS Presentation

Uploaded by

Copyright:

Available Formats

Foundations of Verification and Validation

A Logical Derivation from the Scientific Theory of Truth

AIAA (1998) ASME (2006) NAFEMS QSS Primer (2014)

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

• Syllogistic model of a problem

major premise minor premise conclusion

𝑝→𝑞 ∧ 𝑞→𝑟 ⟹ 𝑝→𝑟

– engineering example : linear spring system 𝐾𝑢 = 𝐹

mathematical model parametric data solution

𝑢 = 𝐹/𝐾 ∧ 𝐹/𝐾 = 𝑓/𝑘 ⟹ 𝑢 = 𝑓/𝑘

mathematical model ∧ data ⟹ solution

• Example : Determine sources of error

model data solution

• Coherence and correspondence tests for truth

– well-established tests for truth from the Philosophy of Science:

• correspondence theory of truth

• coherence theory of truth

Ref : Russell, B., The Problems of Philosophy, 1912

• Correspondence test – do the premises conform to the observed facts?

model data solution

– modelling error - incorrect assumptions, etc.

– data error - incorrect coefficients values, etc.

• Coherence test – is the model logically self-consistent?

model data solution

– approximation error - discretisation, truncation, etc.

– calculation error - mistakes in arithmetic, bugs in software, etc.

• Correspondence test • Coherence test

• data error • calculation error

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

• Verification and validation definitions

Sources* : What is Verification and Validation?, ASME/NAFEMS, 2010

• Coherence and correspondence

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

• Verification and validation

𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐥 : 𝐠𝐞𝐧𝐞𝐫𝐚𝐥 ⇐ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫 ∧ 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫

𝐢𝐝𝐞𝐚𝐥𝐢𝐬𝐞𝐝 𝐦𝐨𝐝𝐞𝐥 : 𝑦= 𝑥 ∧ 𝑥 = 1.41 … 𝑛 ⇒ 𝑦 → 2

• Validation is a correspondence test

• Verification is a coherence test

verification challenges the numerical accuracy of the computation

– idealisation idealisation idealisation

• idealise model from engineered object validation

– calculation numerical model

• compute the solution

– validation V&V process

• idealisation validation behavioural

– parametric data values

• 2. Software testing application

– Require a simulation process flow diagram for simulation control

– Base simulation process diagram advantages idealisation idealisation

• logically derived using a syllogistic model idealised model

• structure based on logically derived sources of error model and data

• can be used as a base to derive higher level diagrams approximation approximation

– Verification and validation

• shown to be correspondence and coherence tests calculation

You might also like