Concepts and Commentary

Exploring Validation and Verification

How they Different and What They Mean to Healthcare Simulation

John Jacob Barnes, III, MD;
Mojca Remskar Konia, MD, PhD,
MACM
Downloaded from https://journals.lww.com/simulationinhealthcare by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3wX04VDhDA65JyIrQxIAmw8OudmFC7vF5gSn6ug+Sx+k= on 02/09/2020
Summary Statement: The healthcare simulation (HCS) community recognizes the im-
portance of quality management because many novel simulation devices and techniques
include some sort of description of how they tested and assured their simulation’s quality.
Verification and validation play a key role in quality management; however, literature

published on HCS has many different interpretations of what these terms mean and how
to accomplish them. The varied use of these terms leads to varied interpretations of how
verification process is different from validation process. We set out to explore the concepts
of verification and validation in this article by reviewing current psychometric science de-
scription of the concepts and exploring how other communities relevant to HCS, such as
medical device manufacturing, aviation simulation, and the fields of software and engi-
neering, which are building blocks of technology-enhanced HCS, use the terms, with the
focus of trying to clarify the process of verification. We also review current literature avail-
able on verification, as compared with validation in HCS and, finally, offer a working defi-
nition and concept for each of these terms with hopes to facilitate improved communication
within, and with colleagues outside, the HCS community.
(Sim Healthcare 13:356–362, 2018)

Key Words: Verification, validation, healthcare simulation, medical education, quality assurance.

F or the last two decades, healthcare simulation (HCS) has

grown exponentially and expanded its role as a valuable tool
for training medical professionals. There has been a 10-fold in-
crease in HCS literature from 1990s to 2000s,1,2 and the search
term “healthcare simulation training” in PubMed resulted in
the exponential growth of 909 peer-reviewed articles in 2016
compared with 300 in 2006 and 57 in 1996. Albeit this pro-
duces exciting possibilities, it also comes with some growing
pains. Efforts have been made to establish consensus defini-
tions and concepts in HCS, notably by The Society for Simula-
tion in Healthcare and the textbook Designing Excellence in
Simulation Programs by Palaganas et al3,4; however, the varied
use of terms and concepts remains a challenge in the field and
its subsequent research publications. One of these concepts,
which needs further defining, is how to ensure that an HCS
is built according to set specifications and fulfills the intended
training needs. Psychometric science will serve as the founda-
tion of our understanding of verification and validation. Then,
we will explore varied use of these terms in distinct industries.
Simulation itself is no longer in its infancy, even if its use and
application in medical training are. The aviation industry is a
good example of how simulation has grown and evolved for
nearly one century and recognized the importance of standards
for validation resulting in established, very specific guidelines
From the Department of Anesthesiology, University of Minnesota, Minneapolis, MN.
Reprints: John Jacob Barnes III, MD, Anesthesiology, B515 Mayo Bldg, 420 Delaware St SE,
Minneapolis, MN 55455 (e‐mail: BarnesJ@umn.edu).
The authors declare no conflict of interest.
This study is attributed to Department of Anesthesiology, University of Minnesota School
of Medicine.
Copyright © 2018 Society for Simulation in Healthcare
DOI: 10.1097/SIH.0000000000000298
through the Federal Aviation Administration (FAA). Software
and engineering, both of which heavily rely on verification and
validation to ensure their products work and function in their
intended use, are the foundational building blocks of technology-
enhanced HCS. Medicine itself has stressed the importance of
verification and validation in medical devices and pharmaceu-
tical industries with guidance documents from the Food and
Drug Administration (FDA). It is time for HCS training to
learn from its abutting industries and establish clear defini-
tions for verification and validation.
The exercise of exploring verification and validation in
psychometric science, software, engineering, aviation simula-
tion, and medicine offers insight into and understanding of
the concepts, but it is important to note that one must be cau-
tious when drawing conclusions from comparing verification
and validation of these distinct fields to HCS, because these
concepts are heavily dependent on context.5 The decision to
discuss certain fields of study in this article is meant to high-
light the depth and breadth of use of verification and valida-
tion in arenas that would be both familiar and relevant to
the audience interested in HCS and medical education using
simulation. The disciplines included are not meant to be all-
inclusive and were not selected in a scientific manner.
Validation in Psychometric Science
For the past 60 years, psychometric science has made sig-
nificant advances in the understanding and use of validation,
particularly in educational assessment. In 1990, Messick6 wrote
about viewing validation as a unified framework with varying
degrees (not all-or-nothing) and with constituent evidence (not
different types of validation). This view of validation was adopted
by the American Educational Research Association, Psycho-
logical Association, and National Council on Measurement

356 Exploring Validation and Verification Simulation in Healthcare

Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
in Education in 1999 with the publication of Standards for Ed-
ucational and Psychological Testing and continues to be present
in the 2014 edition.7 Before this time, validity was considered
to have different types, such as face validity, construct validity,
etc. More recent work in psychometric science by Kane and
Service8 further develops the idea of validation as an argument-
based approach to support claims that go beyond observed
performances. In the more modern approach to validation,
there are five sources of evidence that support the argument
for validation of assessments, which are content, response
process, internal structure, relationship to other variables,
and consequences.
Standards for Education and Psychological Testing does not
include a dedicated chapter to verification, like it does for val-
idation. The chapter on validation is arranged in the first part
of the book under “Foundation,” which also includes a chapter
titled “Reliability/Precision and Errors of Measurement.”
Publications with discussion of HCS verification efforts
use both reliability and errors of measurement concepts as parts
of verification.9,10
How Nonmedical Fields Use These Terms
Because it is difficult to find or establish consensus defini-
tions and techniques for verification and validation in HCS, it
is useful to look at how the foundational sciences behind
technology-enhanced HCS, software and engineering, use the
terms and how the aviation industry has developed established
guidelines for these concepts. Before delving into these three
fields and their use and implementation of verification and
validation, it is worth noting scientific or theoretical knowl-
edge and understanding on a technical level plays a fundamen-
tal role in these fields and their verification and validation
efforts. Oberkampf et al11 provide an excellent review of veri-
fication and validation in scientific fields, discussing specific
and technical aspects that likely are not applicable or relevant
to HCS development and quality control. The utility of explor-
ing these fields lies in growing and developing a better under-
standing of what verification and validation mean, how they
are used in other sectors, and how this applies to HCS.
In software, the notion of verification and validation has
been around since the 1960s.12 Wallace and Fujii12 summa-
rized verification and validation as a way to “comprehensively
analyze and test software to determine that it performs its
intended functions correctly, to ensure it performs no unin-
tended functions, and to measure its quality and reliability”
in their overview of the concepts. Boehm13 differentiated the
terms in the 1980s by defining verification as answering the
question, “Are we building the product right?” and validation
as “Are we building the right product?” With regard to soft-
ware testing, designers focus on making sure that the software
can be verified by being functional, without bugs, and that the
product is smooth, built according to specifications, and has
consistent output. Software validation can be less objective, be-
cause it often relies on opinions when answering the question
of “Are we building the right product?” To answer this ques-
tion, one must test the product to see whether it, indeed, does
serve its intended purpose. This can be achieved with beta test-
ing, where subjects evaluate usability and determine whether
the software or product achieves the intended goal, or through
Vol. 13, Number 5, October 2018
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
a more structured analysis of the computer-human interaction
of software through usability engineering, which focuses on
the ease of learning, ease of use, and user satisfaction. Carefully
planning validation that can be integrated into the software de-
velopment process, before actual development occurs, can ad-
dress some of the challenges of subjectivity and facilitate more
effective validation.14
Verification and validation are also important in the field
of engineering. The Institute of Electrical and Electronic Engi-
neers (IEEE) publishes standards for and definitions of verifi-
cation and validation in their guidebook for project managers:
“Verification. The evaluation of whether or not a product,
service, or system complies with a regulation, requirement,
specification, or imposed condition. It is often an internal
process. Contrast with validation.”15
“Validation. The assurance that a product, service, or sys-
tem meets the needs of the customer and other identified
stakeholders. It often involves acceptance and suitability
with external customers. Contrast with verification.”15
The IEEE definition partially overlaps with the software
industry's definitions of verification and validation, but there
is also a hardware or physical component that requires testing,
verification, and validation. As the IEEE oversees multiple in-
dustries, the definitions they provide are broader and further
reaching than those offered by the software industry.
Looking at how the fundamental sciences behind HCS de-
fine verification and validation is helpful as is looking at how
simulation in other fields, such as aviation, use the terms. It
is common to compare HCS with simulation in aviation and
one review article by Aebersold16 focuses on this compari-
son.17 Aebersold16 discusses the Link trainer's origins in the
1920s, often cited as the first flight simulator, and how 50 years
later, in the 1970s, most pilot training was conducted in simu-
lators. The author highlights that simulation use in healthcare,
specifically in the nursing field, is being used to develop tech-
nical skills and “crew resource management” skills in a similar
manner to the aviation industry but has failed to meet the
milestone set in the 1970s of having most training done in
the safety of the simulation umbrella.16 As aviation simulation
has matured as a field, so have its methods of validating the
quality of the experience. In the early stages of aircraft flight
simulators, there was a reliance on an experienced pilot to val-
idate a simulator based on its fidelity to actual performance
and handling of an aircraft.18 Although logical, these evalua-
tions are prone to subjectivity and fraught with vulnerability
to human error. Because of these susceptibilities, the industry
began using more quantitative validation of simulators in ad-
dition to the opinions of experienced pilots. The FAA has de-
veloped many requirements to validate and maintain certification
of flight simulators used to qualify commercial aircraft pilots
in their guidance documents under the title of Flight Simula-
tor Training Qualification. Because of these regulations, the
time and cost of validating flight simulator training devices
are tremendous, with estimates of US $15 million dollars in
monetary cost and enormous data sets required to be collected
and analyzed for validation. One advantage the aviation
© 2018 Society for Simulation in Healthcare 357
industry has is the ability to compare objective information
from mathematical models and data collected by the electron-
ics of actual aircraft flight with the output data produced by
simulators to form complex validation techniques.19
Validation tests are used to compare objectively flight train-
ing device data and airplane data (or other approved reference
data) to assure that they agree within a specified tolerance.
Functions tests provide a basis for evaluating flight training
device capability to perform for a typical training period
and to verify correct operation of the controls, instruments,
and systems.18
A common technique for validation of aviation simula-
tion is to test individual components in a quantitative way in
four levels, first individual modules individually, second inter-
related modules together, third models tested for dynamic re-
sponse, and finally a pilot using the model.18
It is important to note that aviation, based on physics
principles, lends itself nicely to modeling and simulation.
The principles of aviation are more fundamentally established
and have extensive and accurate mathematical models, which
have been vetted through experimental research over the years.
Mathematical models based on physics laws and principles can
accurately predict how an aircraft will behave under variable
circumstances and actions. This same luxury is not afforded
to HCS, because human physiology is highly variable within
and between subjects20 and running experimental research
on human studies is limited by ethical considerations. It is
conceivable that HCS and modeling focusing on specific tasks,
such as employing an endovascular stent, could approach the
precision found in aviation simulation; however, HCS attempting
to emulate a “whole patient,” as in the use of technology-
enhanced mannequins, is never likely to achieve the pin-
point accuracy of modeling used in aviation simulation based
on physics. The differences between simulation in aviation and
healthcare are important when discussing verification and
validation as the concepts are highly context dependent.
For example, the verification and validation activities for a
highly mathematical model of aviation physics would rightly
look much different than an HCS mannequin used as a
whole-patient trainer.
Verification and Validation in Medicine
Verification and validation are not foreign concepts to
medicine or healthcare. As healthcare is an industry with tight
regulations and restrictions, overseeing organizations, such as
the United States FDA, have defined the terms verification
and validation and how these concepts apply to determining
the safety of products used. Although HCS for training pur-
poses is not currently under the umbrella of an overseeing or-
ganization, such as the FDA, the industry can learn from the
use of these terms in other areas of medicine.
The FDA publishes guidelines for verification and validation
of medical devices, now including computational modeling.21–23
The FDA relies on making clear definitions and identifying
broad concepts that can be applied to each unique medical de-
vice.21 The FDA specifically makes note of how it is common
to group verification and validation together as “V&V” but
specifies it to consider them separate and distinct terms in
their guidelines for software development in medical devices.
358 Exploring Validation and Verification
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
It defines software verification as “objective evidence that de-
sign outputs…meet all of the specified requirements.”21 It
suggests software testing, static and dynamic analysis, and code
inspections as ways of completing this task. The FDA defines
software validation in bold text as:
“Validation is the confirmation by examination and pro-
vision of objective evidence that software specifications
conform to user needs and intended uses, and that the
particular requirements implemented through software
can be consistently fulfilled.”22
It also recognizes that validation depends on verification
tasks and recommends testing the functionality of the software
as a prototype in a simulated use environment and later in lim-
ited patient use as key components to ensure validity. The FDA
recommends using verification and validation throughout the
design process.
The pharmaceutical industry, also regulated by the FDA,
puts a strong emphasis on process validation defined as “the
collection and evaluation of data, from the process design stage
through commercial production, which establishes scientific
evidence that a process is capable of consistently delivering
quality product.”24 The FDA details this process in their docu-
ment “Process Validations: General Principles and Practices”
accompanied by their current recommendations. Process val-
idation breaks down into three stages, with continued process
verification as the final stage, in which “ongoing assurance is
gained during routine production that the process remains
in a state of control (the validated state).”24 Manufacturers of
pharmaceutical goods go through a stringent validation pro-
cess proving they “consistently produce drug products…relat-
ing to identity, strength, quality, purity, and potency”24 and
then must continue to verify that the process does not depart
from its approved and validated state by collecting and analyz-
ing product and process data. As the pharmaceutical industry
markets and intends their products for human consumption,
the process is scrupulous and overseen by the FDA. Although
HCS for training does not report to the FDA, the broad con-
cepts of validation through scrupulously checking for quality
and reproducibility at various points in production as well as
continued verification are important and practical for applica-
tion to HCS.
The act of validating and verifying biomechanical prod-
ucts for use in medicine incorporates several similar concepts
used by the nonbiological engineering fields but has its own
unique challenges because of their design for and use by the
medical community and patients. Hicks et al25 and Stanford
University colleagues published recommendations for validat-
ing and verifying biomechanical products. They recommend
verification of computational models and underlying algo-
rithms of simulated physiological or physical movement by
comparing with known standards, which they state is more
feasible if done individually on components and then on aggre-
gates of components, and validation through comparison of
whole model simulation data with patient data from indepen-
dent experiments or other published model data sets. Hicks
et al25 mention the limited criterion standard data sets due to
difficulty measuring desired output in humans and how there
Simulation in Healthcare
is inherent interpatient variability, making both verification
and validation more challenging than in nonmedical fields.
Fields of study, or areas, within medicine address verifica-
tion and validation in a similar, but diverse way, as it evident
by the examples provided. The FDA makes a point to differen-
tiate between verification and validation and the medical de-
vice, pharmaceutical, and biomechanical industries perform
different tasks to accomplish these two, distinct tasks. For ver-
ification, the medical device industry uses tools such as static
and dynamic analysis of software and code inspection, the
pharmaceutical industry uses ongoing data collection and anal-
ysis of product to ensure the approved manufacturing process
has not altered course, and the biomechanical field compares
individual components of a model to established principles.
For validation, the medical device industry tests devices in
simulated environments and limited patient sets, the pharma-
ceutical industry analyzes product at various stages of produc-
tion for quality, and the biomechanical field compares novel
devices to other published models or experimental data on pa-
tients. Healthcare simulation shares many of the challenges in
successfully performing verification and validation and reviewing
how these fields face those challenges may help develop these
concepts in HCS.
Healthcare Simulation Validation
There is much more information published on validation
of simulation use in healthcare compared with verification.
Validation has many different definitions, and it is important
to ask, “What are we really validating?” Validation of tools
for educational assessment has made significant advancements
in the last 60 years, and Downing26 and Cook et al have made
efforts to bring these ideas into medical education and HCS,
respectively.27,28 Downing provides a review of the contempo-
rary use of validity, being a unitary concept with multiple facets
of evidence, and discusses this in the context of assessments
(standardized exams) in medical education. In a systematic re-
view of validation evidence in 417 simulation studies used for
healthcare professionals, Cook et al27 discovered only 3% used
the unified view of validation discussed here and accepted by
Standards of Educational and Psychological Measurement, 35%
used the older framework focusing on “types” of validation,
and 24% used no validation framework at all. It seems that val-
idation is underused and not well understood in medical edu-
cation and HCS research.
The European Association of Endoscopic Surgeons (EAES)
saw simulation as a valuable tool for training and noted the lack
of consensus guidelines or uniform assessment of available sim-
ulators. In 2005, it formed a subgroup called the Work Group
for Evaluation and Implementation of Simulators and Skills
Training Programmes at their annual meeting to help define
validation of surgical simulators and to evaluate current prod-
ucts on the market based on their definition of validation and
available research. It defined validity as “the extent to which an
assessment instrument measures what it was designed to mea-
sure.” It argued that to do this, it was necessary to look at va-
lidity; however, it used the former understanding of validity
focusing on validity types rather than using the unified validity
framework. The subgroup noted “face validity” as the degree
of resemblance between the simulator and the real activity
Vol. 13, Number 5, October 2018
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
and considered this as one facet of validation. It also examined
“content validity” or how well the simulation covers the subject
matter of the real activity, “construct validity” or “contrast valid-
ity” as the ability for the simulation to differentiate between abil-
ity or experience levels, and “predictive validity” as the ability to
correlate performance on the simulator with established assess-
ment of a skill or attribute in patient care. It argued that the most
powerful evidence for validation found or the most important
aspect of validation for their group is the predictive validity.9
One of the most common validation questions addressed
is “Does a scoring of performance in this simulator correlate
with a measure of experience or ability?” The grouping of “nov-
ice” users like medical students or residents and comparing
their performance to “experts” such as attending physicians at
teaching hospitals or principal investigators is a common way
of validating that the simulation in question is able to test con-
struct validity or differentiate users based on experience and
skill levels.29–31 A similar technique is to have the novice group
broken down into a control group and a group, which receives
training on a simulator and then comparing their skills to those
of experts. This is a way to validate a simulator in teaching a skill
that translates into patient use.32 In the updated unified frame-
work of validation, these techniques are considered “relation-
ship with other variables” evidence and Cook et al27 note that
this was by far the most commonly used evidence for valida-
tion in their systematic analysis, with 73% of the studies using
this method and 33% using this as their only evidence to sup-
port validity. Expert-novice comparison does demonstrate a
simulation's sensitivity, or the ability to discriminate between
two groups, but Cook et al27 describe this as a “relatively weak”
form of validation evidence and that overreliance on expert-
novice comparison for validation is unfortunate.
Other publications focus on the face validity of an HCS.
The idea of face validity is to analyze a simulation and decide
how closely it resembles or feels like the actual activity. This
can be more closely examined and compartmentalized into
categories such as tissue or anatomic quality. For example,
one could examine the anatomy of an airway simulation trainer
and compare their experience with typical human anatomy.
Classifying or “proving” face validity often relies on the subjective
ratings of subject matter experts or those with some predefined
experience in the simulated subject matter.1,10,33
These techniques struggle with some form of subjectivity,
which is part of the intrinsic nature of validation. If you are an-
swering the question, “Did I build the right product?” or “Are
the user needs and intended uses met?” a subjective answer is
required. Users may have different ideas of what the intended
use is and may have different opinions on whether this need
was met. One method to make validation more objective is
to incorporate validation into the developmental cycle so that
there are agreed-upon ways to establish validity before the de-
sign process ever begins (Fig. 1). This can prevent bias from
coming in after a design is complete or a product is ready for
testing. This is common in medical device manufacturing as
part of the entire process termed “design control activities.”21
Healthcare Simulation Verification
Publications in HCS focus less on verification. This may
result from the “bundling” of the individual concepts of
© 2018 Society for Simulation in Healthcare 359

FIGURE 1. Adapted from FDA figure of Design Control Guidance
for Medical Device Manufacturers. A pictorial representation of how
verification and validation can be incorporated into the design
control process (developmental cycle) of HCS as it is in medical
device manufacturing.
verification and validation into one concept and lack of clear
separation in the individual activities around verification and
validation. Some studies use the term verification but do not
specify what the word means or how it was determined, whereas
other use different terms such as reliability or repeatability
when discussing quality management in HCS.9,10,34 The previ-
ously mentioned EAES subgroup on evaluating surgical simu-
lators also noted the importance of “reliability” or the “ability
to provide consistent results with minimal errors in measure-
ment” and named test-test reproducibility and internal consis-
tency as commonly used methods to estimate this.9 Software
and engineering fields perform these tests as a form of verifica-
tion.12,15 In the EAES's work on establishing consensus guide-
lines for rating simulation equipment available at the time, it
noted the paucity of data available for verification and decided
to focus their attention solely on validation. Because different
authors in different fields use the words reliability and verifica-
tion to describe similar concepts and tasks, it can be confusing.
It seems that verification is the preferred word in software, en-
gineering, and medical device industries for testing and dem-
onstrating test-test reproducibility, internal consistency,
and accuracy, whereas some HCS publications address sim-
ilar tasks and discuss them under the terms repeatability and
reliability.10,29,30,34,35
Gallagher and colleagues10 also discuss how reliability re-
lates to the medical field. Their article focused on creating val-
idation systems for evaluating surgical training and assessment
devices. They argued that reliability is a crucial part of a total
validation system (ie, that it is mathematically consistent, re-
producible, and dependable) rather than using the term verifi-
cation. Specifically, the group explored split-half method and
test-retest methods to establish reliability in assessment tools.10
In contrast, Hicks et al25 set out to perform verification in the
field of biomechanical devices by determining the accuracy
of the musculoskeletal models and simulations of movements.
They defined verification as the determining of agreement
between more complex novel computational models of
360 Exploring Validation and Verification
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
musculoskeletal movement and the underlying established mathe-
matical model of physical movement.25 In this case, the two au-
thors are using different words to describe the similar
mathematical concepts to discuss quality management within
HCS. Gallagher and colleagues,10 who use the word reliability
in their article, caution that readers may misconstrue the word
reliable, because it carries favorable connotations and is syn-
onymous with “good” or “worthwhile” in common speech,
when in scientific writing, reliable means consistent or how a
tool yields the same results when used repeatedly under similar
conditions. Further confounding matters, Cumin et al34 stud-
ied the consistency of physiologic response in various HCSs
and used the term repeatability. It seems that reliability and re-
peatability are not equivocal to verification, rather it may be
more appropriate to consider them individual components
of the larger verification effort. Furthermore, Lampotang36
notes, in an article discussing HCS repeatability, patient
variability constitutes an inherent challenge in HCS but also
argues that variability within a design framework of HCS
“fortuitously relaxes modeling requirements” as repeatabil-
ity of a simulation should only need to fall within a range of
plausible patient variation that has been predetermined. With
inconsistent use of terms, it is important for authors to state and
describe with clarity what they aim to accomplish and how this
comprises a component of quality control or assurance.
Sweet et al37 have proposed integration of validation and
verification into the developmental and evaluation process of
simulation tools. In the article, they describe the technique as
the Center for Research in Education and Simulation Technol-
ogies (CREST) process, which is based on the earlier “back-
ward design principles” of Wiggins and McTighe38 in their
book on curriculum development. Using backward design
principles, Sweet et al37 used verification and validation
techniques throughout the development process and ap-
plied these to early prototypes rather than the final product.
The authors noted that using this method helped refine
their models, made verification and validation easier to
complete, and improved their final product. Focus on verifica-
tion and validation throughout the developmental process,
rather than focusing validation and verification efforts after
design, is an interesting way to confirm that results are consis-
tent (verification) and the individual parts perform their spec-
ified function (validation).
Summary
Looking at the use of verification and validation in HCS
provides a glimpse into how these techniques are used and
how broadly defined they are as well as establishes them as
unique concepts with interdependence and overlapping of con-
stituent activities (Fig. 2). As pointed out in articles by
Gallagher and colleagues10 and Sweet,35 the educated reader
of scientific publications in this field must be aware of the
broad implications and use of the techniques and be careful
to draw their own conclusion of what has been “verified” or
“validated.” It is again worth noting the importance of using
caution when comparing highly context-dependent concepts
such as verification and validation in the distinct fields covered
in this review. Many of the techniques for verification and val-
idation in highly scientific or technical fields would not be
Simulation in Healthcare
FIGURE 2. The relationship between verification and validation in HCS showing tasks unique to each and overlapping tasks.
practical or have any utility in the evaluation of the applied sci-
ence of HCS, but the exploration of these topics helps develop
and understand purpose of verification and validation in qual-
ity assurance. Some fields rely on regulatory agencies to pro-
vide strict definitions of and laws governing verification and
validation such as aviation (FAA) and medicine (FDA). We
would suggest that this is too stringent for the current state
of HCS and the field would rather benefit from further re-
finement of commonly used terms and guidelines for their
activities such as verification, validation, and their constituent
elements in a comparable manner to the fundamental sciences
of software and engineering. Efforts have been made to do so
as evident by the 2016 publication of a dictionary by Simula-
tion in Healthcare, which included the terms simulation
reliability and simulation validity.
Borrowing from those fields, a hybrid definition for veri-
fication and validation in HCS could be the following:
Verification: the act of confirming the simulator was built
according to its specifications
Validation: the act of confirming the HCS accurately por-
trays what it is modeling and produces the desired outcome
These definitions differentiate verification and validation
and allow for a foundation on which authors can precisely
communicate their findings and readers can better understand
the concepts and more effectively draw their own conclusions.
We agree that verification and validation should be integral
parts of the design process rather than tasks completed after
the completions of process, as has been previously suggested.
The industries explored, in this article, all stress the impor-
tance of integration of verification and validation in the design
process, as do Sweet and colleagues37 for use in HCS educa-
tion. Formulating a way to answer the questions from early
work by Boehm13 with software “Did we build the right prod-
uct?” and “Did we build the product right?” is much easier and
may more appropriately reflect true validation and verifica-
tion, before the design process is initiated rather than after
the fact. In the end, the burden of proving a product or tech-
nique as valid or verified falls on the author and the burden
of drawing those conclusions falls on the reader. Having a clear
definition of these separate but overlapping concepts ensures
Vol. 13, Number 5, October 2018
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
that we are effectively communicating important ideas for
quality management in the rapidly emerging and exciting
realm of HCS.
REFERENCES
1. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ.
Features and uses of high-fidelity medical simulations that lead to effective
learning: a BEME systematic review. Med Teach 2005;27(1):10–28.
2. Rosen KR. The history of medical simulation. J Crit Care 2008;23(2):
157–166.
3. Loreiato J, Downing D, Gammon W, et al. The Terminology & Concepts
Working Group. Healthcare Simulation Dictionary. Rockville, MD: Agency
for Healthcare Research and Quality; 2016.
4. Palaganas JC, Maxworthy J, Epps CA, Mancini ME. Defining Excellence in
Simulation Programs. Philadelphia, PA: Lippincott Williams &
Wilkins; 2014.
5. Spiegel M, Reynolds PF Jr, Brogan DC. A case study of model context for
simulation composability and reusability. Paper presented at: Proceedings
of the 37th Conference on Winter simulation; December 4, 2005.
6. Messick S. Validity of test interpretations and use. ETS Research Report
Series 1990;1:1487–1495.
7. American Educational Research Association APA, National Council on
Measurement in Education, Joint Committee on Standards for
Educational and Psychological Testing. Standards for Educational and
Psychological Testing. Washington, DC: American Educational Research
Association; 2014.
8. Kane MT, Service ET. Validating the interpretations and uses of test scores.
J Educ Meas 2017;50(1):1–73.
9. Carter FJ, Schijven MP, Aggarwal R, et al. Consensus guidelines for
validation of virtual reality surgical simulators. Surg Endosc 2005;
19(12):1523–1532.
10. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of
validation, and reliability: rigorous science for the assessment of surgical
education and training. Surg Endosc 2003;17(10):1525–1529.
11. Oberkampf WL, Trucano TG, Hirsch C. Verification, validation, and
predictive capability in computational engineering and physics. Appl Mech
Rev 2004;57(5):345–384.
12. Wallace DR, Fujii RU. Software verification and validation: an overview.
IEEE Softw 1989;6(3):10–17.
13. Boehm B. Software risk management: principles and practices. IEE Softw
1991;8(1):32–41.
14. Rosson MB, Carroll JM. Usability Engineering: Scenario-Based Development
of Human-Computer Interaction. San Francisco, CA: Morgan
Kaufmann; 2002.
© 2018 Society for Simulation in Healthcare 361
15. IEEE Draft Guide: Adoption of the Project Management Institute (PMI)
Standard: A Guide to the Project Management Body of Knowledge (PMBOK
Guide)-2008. 4th ed. IEEE; 2011.
16. Aebersold M. The history of simulation and its impact on the future.
AACN Adv Crit Care 2016;27(1):56–61.
17. Abrahamson S, Denson JS, Wolf RM. Effectiveness of a simulator in
training anesthesiology residents. J Med Educ 1969;44(6):515–519.
18. National Research Council. Appendix F: Validation of Aircraft Flight
Simulators. In: Webster WC, ed. Shiphandling Simulation: Application to
Waterway Design. Washington, DC: National Academies Press; 1992.
19. Dillard AE. Validation of advanced flight simulators for human-factors
operational evaluation and training programs. Foundations for Verification
and Validation Workshop. John Hopkins University Applied Physics
Laboratory; 2002.
20. Lipsitz LA, Goldberger AL. Loss of 'complexity' and aging. Potential
applications of fractals and chaos theory to senescence. JAMA 1992;
267(13):1806–1809.
21. US Food and Drug Administration. Design Control Guidance for Medical
Device Manufacturers. Rockville, MD: Center for Devices and Radiological
Health; 1997.
22. US Food and Drug Administration. General principles of software
validation. In: Final Guidance for Industry and FDA Staff. Rockville, MD:
Center for Devices and Radiological Health; 2002:6.
23. US Food and Drug Administration. Reporting of computational modeling
studies in medical device submissions. In: Guidance for Industry and Food
and Drug Administration Staff. Rockville, MD: Center for Devices and
Radiological Health; 2016.
24. US Food and Drug Administration. Process validation: general principles
and practices. In: Guidance for Industry. Rockville, MD: Center for Drug
Evaluation and Research; 2011.
25. Hicks JL, Uchida TK, Seth A, Rajagopal A, Delp SL. Is my model good
enough? Best practices for verification and validation of musculoskeletal
models and simulations of movement. J Biomech Eng 2015;137(2):020905.
26. Downing SM. Validity: on meaningful interpretation of assessment data.
Med Educ 2003;37(9):830–837.
362 Exploring Validation and Verification
Copyright © 2018 by the Society for Simulation in Healthcare. Unauthorized reproduction of this article is prohibited.
27. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R.
Technology-enhanced simulation to assess health professionals: a
systematic review of validity evidence, research methods, and reporting
quality. Acad Med 2013;88(6):872–883.
28. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation
for health professions education: a systematic review and meta-analysis.
JAMA 2011;306(9):978–988.
29. Devitt JH, Kurrek MM, Cohen MM, et al. Testing the raters: inter-rater
reliability of standardized anaesthesia simulator performance. Can J
Anaesth 1997;44(9):924–928.
30. Devitt JH, Kurrek MM, Cohen MM, et al. Testing internal consistency and
construct validity during evaluation of performance in a patient simulator.
Anesth Analg 1998;86(6):1160–1164.
31. Ullah W, Hunter RJ, Finlay M, et al. Validation of a high-fidelity
electrophysiology simulator and development of a proficiency-based
simulator training program. Simul Healthc 2017;12(1):41–46.
32. Morris E, Kesser BW, Peirce-Cottler S, Keeley M. Development and
validation of a novel ear simulator to teach pneumatic otoscopy. Simul
Healthc 2012;7(1):22–26.
33. Berwick RJ, Mercer SJ, Groom P. Evaluating the fidelity of a novel
part-task trainer for emergency front of neck access training. BMJ Simul
Technology Enhanced Learn 2017. doi: 10.1136/bmjstel-2017-000208.
34. Cumin D, Chen C, Merry AF. Measuring the repeatability of
simulated physiology in simulators. Simul Healthc 2015;10(6):
336–344.
35. Sweet RM. The CREST simulation development process: training the next
generation. J Endourol 2017;31(S1):S69–S75.
36. Lampotang S. Unlike history, should a simulator not repeat itself? Simul
Healthc 2015;10(6):331–335.
37. Sweet RM, Hananel D, Lawrenz F. A unified approach to validation,
reliability, and education study design for surgical technical skills training.
Arch Surg 2010;145(2):197–201.
38. Wiggins G, McTighe J. Understanding by Design. 2nd ed. Alexandria, VA:
ACSD; 2005.
Simulation in Healthcare