See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/263134734

Safety Evaluation of Automotive Electronics Using Virtual Prototypes: State of

the Art and Research Challenges

Conference Paper · June 2014

DOI: 10.1145/2593069.2602976

CITATIONS READS

35 420

25 authors, including:

Nico Bannow Oliver Bringmann

Bosch University of Tuebingen
18 PUBLICATIONS   74 CITATIONS    242 PUBLICATIONS   1,648 CITATIONS   

SEE PROFILE SEE PROFILE

Andreas Burger Moomen Chaari

ABB Infineon Technologies
28 PUBLICATIONS   126 CITATIONS    12 PUBLICATIONS   61 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Matrix-Shaped mLSI View project

autoSWIFT View project

All content following this page was uploaded by Andreas Burger on 10 March 2018.

The user has requested enhancement of the downloaded file.

 Safety Evaluation of Automotive Electronics Using Virtual
Prototypes: State of the Art and Research Challenges

J.-H. Oetjens1 , N. Bannow1 , M. Becker2 , O. Bringmann3,4 , A. Burger4 , M. Chaari7 ,

S. Chakraborty5 , R. Drechsler6 , W. Ecker7 , K. Grüttner8 , Th. Kruse7 , C. Kuznik2 ,
H. M. Le6 , A. Mauderer1 , W. Müller2 , D. Müller-Gritschneder5 , F. Poppen8 , H. Post1 ,
S. Reiter4 , W. Rosenstiel3,4 , S. Roth1 , U. Schlichtmann5 , A. von Schwerin9 ,
B.-A. Tabacaru7 , A. Viehl4
1
Robert Bosch GmbH, 2 Universität Paderborn, 3 Universität Tübingen,
4
FZI Forschungszentrum Informatik, 5 Technische Universität München,6 University of Bremen,
7
Infineon Technologies AG, 8 OFFIS – Institute for Information Technology, 9 Siemens AG
Paper contact: Jan-Hendrik.Oetjens@de.bosch.com, wolfgang.ecker@infineon.com

ABSTRACT
Intelligent automotive electronics significantly improved driv-
ing safety in the last decades. With the increasing complexity
of automotive systems, dependability of the electronic com-
ponents themselves and of their interaction must be assured
to avoid any risk to driving safety due to unexpected failures
caused by internal or external faults.
Additionally, Virtual Prototypes (VPs) have been accepted
in many areas of system development processes in the automo-
tive industry as platforms for SW development, verification,
and design space exploration. We believe that VPs will sig-
nificantly contribute to the analysis of safety conditions for
automotive electronics. This paper shows the advantages
of such a methodology based on today’s industrial needs,
presents the current state of the art in this field, and outlines
upcoming research challenges that need to be addressed to
make this vision a reality.

Figure 1: Combined Active and Passive Safety (CAPS)

Categories and Subject Descriptors
B.8 [Performance and Reliability]: Reliability, Testing,
and Fault Tolerance; C.4 [Performance of Systems]: Re-
liability, availability and serviceability
General Terms
Verification, Reliability
Keywords
Safety, Reliability, Virtual Prototypes, Embedded Systems,
Automotive, Robustness
1. INTRODUCTION
With the increasing number and acceptance of complex
electronic-based embedded systems in daily life, e.g. in
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from Permissions@acm.org.
DAC ’14, June 01 - 05 2014, San Francisco, CA, USA
Copyright 2014 ACM 978-1-4503-2730-5/14/06 ...$15.00
http://dx.doi.org/10.1145/2593069.2602976.
Photo: Bosch
medical aids, industrial production and cars, their risk of
failure increases, which may threaten human lives or physical
conditions. Therefore, the evaluation of safety is a crucial
topic in several areas, especially for automotive electronics.
Of course, several functions could be implemented without
electronic components however they help to increase energy
efficiency (e.g. highly sophisticated motor control), com-
fort (e.g. adaptive cruise control), and safety (e.g. airbags).
So, year after year, more and more sophisticated electronic
components based on semiconductors make their way into
cars making them even more efficient and safer. In so doing,
new functions are often realized by the interaction of several
electronic components, e.g., in systems such as Combined
Active and Passive Safety (CAPS), which is illustrated in
Fig. 1. CAPS links the data from environment sensors with
the airbag control to better protect the passengers by esti-
mating direction and strength of a collision. As such it must
be, for instance, absolutely guaranteed that the failure of
any system component does not trigger the airbag in normal
operation.
In recent years, virtual prototypes (VPs) have been ac-
cepted in many areas of system development processes in
the automotive industry and other industrial sectors. In the
automotive industry, VPs are most frequently applied as plat-
forms for software development, in design space exploration
and system verification. Meanwhile, we can find dedicated
frameworks for Virtual ECU (Electronic Control Unit) design
and testing based on the automotive standard AUTOSAR
(AUTomotive Open System ARchitecture) [1]. Moreover,
we are expecting that VPs will become more important for
safety analysis and safety assurance of future automotive
systems as a result of the wide spectrum of dependability
challenges [2].
In this paper, we present an overview on how the correct
and safe functionality of complex systems can already be
examined in an early design phase using Virtual Prototypes.
A major challenge is to ensure the consistency between the
VP and the real system. Here, the worst case for an incon-
sistency between both is that a fault is not modeled in the
VP, but impacts the real system.
On the other hand, there are many scenarios that can
be hard to apply to real physical systems. Erroneous data
in arbitrary components, such as registers or memory cells,
and errors in hardware components, such as stuck-at errors
or disconnected wires between two subcomponents of an
ASIC are difficult to insert into a real hardware prototype.
Also, insertion of faults into physical HW may destroy the
prototype or lead to a system state that may cause safety
issues for the operator. Additionally, in a VP it is much easier
to observe the impact of the error on the system and track
the error propagation. Even more importantly, VPs can help
find unexpected error effects early in the design process. So,
measures to keep the overall system safe can be included
at an early phase therefore avoiding costly redesigns. As
a consequence, VPs have many advantages when analyzing
safety. The current state of the art in this field is outlined
in Section 2. A large research effort is required to make
the vision of early safety assurance based on VPs a reality.
The related, unresolved research challenges for applying the
methodology on future complex automotive electronics are
discussed in Section 3, before we close the paper with a
conclusion.
2. STATE OF THE ART
Safety and reliability of automotive and embedded systems
in general have been the focus of many research activities.
This section outlines some important work in these and
related areas.
2.1 System-Level Safety Evaluation
There are several established methods for evaluating sys-
tem safety. Fault Tree Analysis (FTA) and Failure Mode
Effects & Diagnostic Analysis (FMEDA) are two examples
for well known dependability analysis methods. Tradition-
ally, these methods have to be executed by system experts,
as the identification of failures, their logical interaction and
the estimation of failure rates are based on the engineer’s
experience and knowledge. To reduce the complexity of these
evaluations as well as the strong reliance on the engineer’s
system knowledge, different approaches were developed to
automate various steps [3–6]. Pinello et al. [3] propose an
approach based on Fault Tolerant Data Flow (FTDF) graphs
to create Fault Trees (FT). The Fault Propagation and Trans-
formation Calculus (FPTC) [4] allows the determination of
the system failure behavior based on information about the
failure behavior of components and their interconnections.
All approaches require that the engineer has to specify an
intermediate model that contains crucial information, e.g.
the dependency between system components, such as data
flow or control flow dependency.
Another very challenging task for system experts, which
is necessary for the intermediate format, is the estimation of
the failure propagation of system components, especially in
combination with failure tolerance.
In the area of system verification, simulation runs have been
established for the validation of functional and non-functional
requirements. System-level simulations are used to verify
functional, timing or power requirements like [7]. Executable
models of the system under test promise a huge potential
for safety evaluations. They cover inherently the internal
component dependencies and the provided error tolerance.
When using system simulation runs to overcome the chal-
lenges of traditional safety evaluation methods, new chal-
lenges emerge as well. In [8] an approach to implicitly support
the FTA with an error effect simulation is shown. To provide
a generalized and comprehensive methodology, additional
challenges have to be solved, which are beyond the scope of
the presented framework. Failure behavioral patterns, which
are currently mostly informally specified, have to be trans-
ferred into an executable format that can be used during
simulation to stimulate errors. Both the simulation and the
failure specification have to fit in system models at different
levels of abstraction, particularly with the abstract system
level. Additionally, methods for creating FTs from the simu-
lation results or for guiding the error simulation to handle
analysis work have to be developed.
2.2 RTL and Gate-Level Analysis
Intensive research was conducted to analyze the reliability
of integrated circuits (ICs). Errors can be injected as bit value
flips in memory cells or registers during logic simulation at the
gate or register transfer level (RTL) [9]. Monte-Carlo-based
techniques can be used to model different fault scenarios.
Based on the fault model, statistically varying error locations
and occurrence times - as well as stimulus [10–12] - are
generated.
These RTL and gate level analysis methods require fin-
ished RTL implementation of the investigated IC, which
may result in expensive redesigns if weaknesses are found at
this late design phase. Additionally, simulation at the gate
and RTL is usually too slow, so that acceleration techniques
are required. A prototype of the IC can be emulated on
an FPGA, for instance. The code can be instrumented by
error injection capabilities [9, 11, 13] or the bit-stream can
be adapted [14, 15] to inject errors in the IC during FPGA
emulation. Simulation-based error injection can accelerate
the evaluation by using higher levels of abstraction, e.g.,
micro-architectural level [12].
2.3 System-Level Testbenches
As stated in Section 2.1, simulations promise a huge po-
tential for safety evaluations. A complete simulation envi-
ronment consists of an executable model of the DUT and a
tesbench for stimulation. Design reuse and interoperability
became mandatory in system design to keep design produc-
tivity in pace with Moore’s law. Consequently, reuse and
interoperability had to become available for system-level
testbenches alike and resulted into concepts such as TLM
(Transaction Level Modeling) for modeling communications
[16] and UVM (Universal Verification Methodology) for test-
bench infrastructures [17]. TLM imitates communication
apart from specific hardware implementations across a va-
riety of abstraction levels: cycle-accurate, approximately
timed, loosely-timed, untimed, and token-level. The commu-
nication architecture is kept undefined and remains open for
design space exploration experiments, usually an application
area of VPs. Moreover, the different communication abstrac-
tion levels allow significant speed-up for system-level models
simulation, a crucial advantage on early safety assurance of
large VPs. Meanwhile, TLM-2.0 is widely accepted as an
industry standard and part of the SystemC standard IEEE
1666-2011 [18]. Nevertheless, the concept of TLM is generic
and can be found as part of the UVM, a methodology and
library for system level testbenches based on SystemVerilog.
UVM separates IP design from IP verification, and makes
testbenches and test cases reusable by employing standard-
ized components, such as agents, sequencers, drivers, and
monitors. In fact, UVM components utilize TLM interfaces
for communication and make use of UVM agents to interact
with the DUT by translating sequences of TLM messages
to wire toggles by means of a translating driver, which acts
as a transactor. UVM monitors are responsible for the op-
posite direction of communication, as they interpret wire
toggles to create TLM messages. Together with the UVM
environment and a concept called UVM factory, these give
UVM testbenches high reconfiguration and reuse potential
for system-level safety evaluation.
2.4 Testbench Qualification
Testbenches and virtual experimentation environments are
fundamental tools for early safety evaluations of automo-
tive systems. For instance, virtual ECU frameworks are
frequently applied to provide early integration and testing of
AUTOSAR software stacks according to automotive software
design processes, such as the V-Model flow. However, the
significance of the results obtained depends on the pertinence
of test cases being stimulated by the verification environ-
ment. Mutation analysis utilizes fault injection to qualify a
testbench with respect to its ability to reveal faults in terms
of fault activation, error propagation, and failure detection.
For this, artificial faults, referred to as mutations, are deliber-
ately seeded into the DUT (Design Under Test), i.e., models
of HW/SW components. A mutant is generated by applying
one mutation operator to a single place of the DUT.
Such mutation operators were originally1 introduced to
represent typical programmer failures [19]. These failures
are modeled as simple syntactic program faults due to the
Competent Programmer Hypothesis [19], which assumes that
programmers are competent and write programs that are
nearly correct. Further, another hypothesis called Coupling
Effect [19] attempts to establish the effectiveness of mutation
testing. It states that if a set of test data is able to kill (i.e.,
to monitor/detect) most of the mutants that are generated
with simple fault seeding, then they will also be capable
of revealing the real, more complex bugs in the program.
Thus, the mutation score, i.e. the fraction of killed mutants
with respect to the overall number of mutants, provides an
advanced metric to assess a testbench’s quality compared
with coverage based metrics. Mutation testing results can
be applied for automatic test pattern generation in case the
mutation score turns out to be insufficient [20].
The application of mutation analysis for testbench qualifi-
cation has been investigated for various domains, abstractions
and languages [21], such as software (Fortran, C/C++, Java,
bytecode, assembler/binary [22], Simulink [23]) and hardware
(VHDL [24], SystemC/TLM [25], IP-XACT [26]). As such,
various fault models are considered that cover faults related
to specification/integration, programmer failures, and design
tool failures. In [27] the authors consider fault injection for
1 Therefore, Mission Profiles are a promising approach for rec-
Mutation analysis originally addresses testbench qualifica-
tion with respect to design faults. However, testbenches can
be also qualified in order to reveal software errors/failures be-
ing caused by runtime faults in the digital/analog hardware
and the physical environment. As such, mutation analysis
can be also investigated for qualifying stress test environ-
ments.
dependability assessment of AUTOSAR [1] models. How-
ever, they do not investigate fault injection for testbench
qualification.
As the number of mutants to cover can be very large,
current research mainly addresses techniques to improve
mutation-based testing efficiency in order to keep up with
the ever-increasing system complexity. To achieve this, sev-
eral approaches have been recently investigated, such as
mutation schema [21], model abstraction [28], hardware ac-
celeration [29], software emulation [30], and directed test
generation [20, 22].
3. RESEARCH CHALLENGES
3.1 Overview
While using Virtual Prototypes for early safety evaluation
is a very promising approach, major challenges still must be
resolved to make it a reality. The first obstacle is to bridge
the big gap of abstraction levels between errors injected in
VPs and real-world faults, which are mostly of a physical
nature. The next obstacle is the complexity of simulating
stress tests on VPs to analyze the impact of faults on sys-
tem safety. Functional verification using VPs is relatively
well-understood today and has become a mature technology.
Still, managing the complexity of functional verification for
increasingly complex systems is a formidable task. On the
other hand, safety evaluation requires comparing the system
functionality in the presence of faults to the correct one.
Given the seemingly endless possibilities of fault manifesta-
tion, safety evaluation clearly adds another dimension to the
complexity problem.
In the following, we envision a potential safety evaluation
approach to tackle these challenges. The approach combines
two well-established methods with a novel approach: (i)
Mission Profiles from the automotive domain, (ii) UVM-
based testbenches from the hardware domain and (iii) error
effect simulation. The following sections discuss each of these
three pillars in more detail and outline concrete research
questions.
3.2 Mission Profile-Compliant Verification
Early system verification and validation is a mandatory
necessity in order to efficiently guarantee reliability and
robustness of a system. Especially in the automotive industry,
a high degree of reliability and robustness has to be ensured.
Therefore, novel methods have to be employed to recognize
as early as possible any malfunctions of the system and its
components due to faults. Consequently, these methods must
use parts of the system specification or design experiences
in order to define and simulate errors efficiently at an early
stage of development. As such, a novel approach to specify
requirements and scenarios, known as Mission Profiles [31,32],
has been developed in the automotive sector.
A Mission Profile (MP) defines the application-specific
context refined for a system or a component to ensure robust-
ness and reliability. The Mission Profile requirements are
expressed as a set of relevant environmental stresses, func-
tional loads and operating conditions regarding a component.
After a formalization process, the Mission Profiles are passed
down from the OEM to the semiconductor manufacturer.
ognizing malfunction of a system or its components as they
consider all system components along the entire supply chain
and are available in an early stage of the design flow.
The different requirements and relevant environmental
stresses within the Mission Profiles can be used to derive
targeted functional fault/error descriptions. Additionally,

Figure 2: System Validation with Mission Profiles
the Mission Profile specifies the normal operating states of
the system as well as operating states that describe a possible
malfunction or a special use case, for instance the high load
for the servo motor when steering against a curbstone. Nev-
ertheless the derivation of functional fault/error descriptions
from Mission Profiles is a very challenging task and currently
not yet solved.
Hence, it is necessary to thoroughly analyze the different
environmental stresses and functional loads to derive feasible
application- and system-specific fault/error descriptions. An
environmental stress, e.g., could describe vibration loads
for components according to their specific mounting point.
Based on this vibration load, a probability of errors due
to wiring, such as open load or short to ground, should be
derived.
Based on this functional fault/error description, a stressor
should be generated, which stimulates fault/errors in the
error effect simulation. The error effect simulation is pre-
sented in Section 2.1 more closely. How the stressor can be
integrated in the testbench is outlined in the next section.
As shown in Fig. 2 Mission Profiles are early available
in the design flow. Therefore, the derivation of functional
fault/error descriptions and the resulting stressors can be
done at every system level - from the OEM down to the
semiconductor manufacturer.
3.3 UVM Testbenches for Fault Analysis
UVM, as it was introduced in Section 2.3, is a well estab-
lished standard for testbenches in electronic systems design.
However, not all of the requirements are addressed:
Currently, standard UVM is implemented in SystemVer-
ilog. However, many other languages are applied for elec-
tronic systems design such as MATLAB/Simulink, VHDL
and SystemC, to name just a few. Ideally, when choosing one
language best suited for an individual application, it is prefer-
able to always adhere to the same verification methodology
and testbench library, for instance with C-based implementa-
tions. As such, the works still under development in [33, 34]
introduce a SystemC UVM implementation which is an initial
language reference definition and associated proof-of-concept
implementation to the Accellera Systems Initiative. Alterna-
tively, SVM (System Verification Methodology) [35, 36] has
been introduced as a SystemC implementation of a significant
UVM subset. Creating a unique verification methodology for
all design languages is not sufficient to be able to integrate ver-
ification environments from a mix of components. Therefore,
the Accellera Multi-Language (ML) working group is heading
for testbench and test case reuse across multi-language veri-
fication platforms, e.g. a UVM-MATLAB/Simulink environ-
ment hosting a UVM-SystemVerilog testbench, instantiating
a UVM-SystemC agent to stimulate a VHDL DUT. System
emulation platforms, like Zebu, Palladium, and Veloce, can
be seen as a special case for this. Like simulated components,
emulated system parts should fit under the umbrella of multi-
language based design and verification reuse. It should be
of no relevance for analysis if a part of a DUT is executed
by simulation or emulation. Zebu, e.g., offers transactors for
simulation to emulate TLM communication. Digital based
methodologies have to be extended towards AMS (Analogue
Mixed Signal) designs. Li et al. [37] target this by including
SystemC-AMS in their work.
Correctness analysis (verification, validation) focuses on
bug detection in specifications and designs of systems. How-
ever, there is a need for extending the methodologies for
fault/error analysis of automotive and other systems. Veri-
fied by design, a deployed system in the field is, nevertheless,
vulnerable to external and internal defects. Fault models for
ASIC fabrication tests are available (stuck-at, short, open,
...), but comparable fault/error models are missing at higher
levels of abstraction.
In Section 3.2 it was described, how mission profiles can
be used to derive formal fault/error descriptions. These
fault/error descriptions are included into the stressor, that
will become an additional component of the testbench for
fault/error evaluation.
One of the main challenges is how to inject the faults/errors.
For external faults, the stimuli, which are provided to the
DUT by the testbench, need to be modified. For internal
faults, errors need to be injected into the DUT, but the
design should not be changed. For this, we propose to add
injectors into the DUT and testbench. These provide an
interface to change the stimuli in the testbench or modify
the state or state transitions at different positions in the
DUT. The stressor uses these injectors to inject faults/errors
according to its formal fault/error description. Additional
to this, methodologies for fault/error classification and fault-
error-failure analysis are required at the monitoring side of
the testbench.
3.4 Stress Tests and Error Effect Simulation
Analyzing the impact of errors on system safety in the
automotive domain poses major challenges, which can be
addressed by simulating stress tests using VPs. VPs cover
the system with ECUs with the integrated software, digital
and analog hardware components and the interconnection
network. Stress tests based on VPs allow error effects to
already be investigated in an early design phase, by enabling
what-if analysis of the system when errors are present. Re-
peated stress tests enable a quantitative evaluation, e.g. to
determine the safety integrity level. A stress test runs the
error effect simulation in a closed loop, as shown in Fig. 3.
The stressor introduces different errors according to its error
scenarios via the injectors for each simulation as was outlined
in Sec. 3.3.
Specific characteristics of automotive systems pose some
major challenges that must be resolved when applying such
stress tests on VPs to assure automotive safety. Mission
Profiles in the automotive domain may require investigating
many seconds of real time. This directly translates to the
simulation of a vast amount of instructions of the embed-
ded cores to emulate the application software. Additionally,
automotive applications typically require the execution of
several concurrent tasks that exhibit hard and soft real-time
constraints. To evaluate the effect of errors, the paradigm
”The right value at the wrong time can still be an error.”
holds. Thus, it must not only be investigated whether an

Figure 3: Error Effect Simulation applying Stress Tests
error may lead to a wrong value, but also if the error may
induce additional delay, e.g., by applying some error correc-
tion that may cause deadline violations. The verification of
real-time requirements requires simulating time and concur-
rency. Usually, current reliability-aware emulators such as
VarEMU [38, 39] only take time in number of instructions
into account. In contrast, SystemC includes a simulation
kernel for event-driven simulation. It enables the evaluation
of the system under development, which becomes the DUT,
along the design process using abstract models in early de-
sign phases to evaluate and to verify coarse system aspects.
The model is refined as the design process progresses, get-
ting more accurate in mirroring the behavior of the physical
system. When software prototypes become available, they
can be integrated and tested in combination with the simu-
lated system parts. With this simulation platform, the error
effect simulation can easily evaluate a huge set of system
alternatives, allowing a quantitative safety assessment. Still,
synchronization poses an extreme overhead in SystemC and
approaches are required that increase simulation performance
of the simulation of concurrent processes, e.g., by temporal
decoupling or by raising the abstraction level with the guar-
antee that the error effect is simulated correctly in terms of
functionality and time.
Raising the abstraction level creates another major chal-
lenge, as faults that lead to possible errors are usually low
level technology-based effects. In [40], it was shown that error
injection at high level of abstraction may result in different
results than injecting errors at the gate level, but simulation
at low level is too time consuming, especially if many error
scenarios are to be investigated. Information on the fault
must be propagated to higher levels of abstraction, requiring
cross-layer analysis. The purpose of such analysis is to derive
the fault models for the high-level stressors, which should
ideally capture the effects resulting from low-level faults to
the full extent, or at least provide adequate approximations.
Furthermore, efficient error effect simulation requires avoid-
ing the error-prone process of manual error injection. Thus,
these fault models should be available in a formalized form
to enable automatic configuration/generation of the error
injectors inside the UVM-based testbenches as mentioned
in Section 3.3. Considering that errors affect various differ-
ent domains, e.g., digital hardware, analog hardware and
software, deriving a unified formalism is a very challeng-
ing task. One of the promising directions is to adopt UML
and the Object Constraint Language (OCL) for error model
specification.
While approaches such as temporal decoupling reduce the
overhead of single simulation runs, the number of simulation
runs has to be optimized as well. Given the huge number of
possible locations of errors, system states and alternatives of
a complex automotive system, it is not feasible to simulate all
the possibilities in the available time. Therefore, intelligent
coverage models are required to measure the completeness
of the error effect simulation as shown in Fig. 3. It has to
be investigated how coverage models can be systematically
derived from safety requirements and Mission Profiles. Then,
the strategy of error injection and stimuli generation should
be geared towards coverage closure. Maximal reuse of exist-
ing functional verification tests should be one of the main
targets in the design process. This is very challenging in
automotive systems design. Stress tests need to be executed
for highly protected systems, which feature a range of failsafe
measures and redundancy at several levels to assure driving
safety. Standard Monte-Carlo techniques may fail to identify
the critical error effects leading to system failure because
failure probabilities are extremely low. Thus, the probability
of exposing a weakness of the system not covered by the
safety and protection measures in a ”lucky guess scenario” is
extremely low, or may require a vast amount of simulation.
Thus, a systematic approach is required that stresses the sys-
tem at its possible weak spots to find error scenarios that lead
to safety-critical failures. Identifying the weak spots has to
be conducted by analysis of error propagation, error masking,
and error recovery by protection mechanisms in all system
domains (e.g. digital HW, analog HW, SW) with respect to
the system structure. For errors that are hard to propagate,
formal approaches such as symbolic execution [41, 42] might
be necessary to generate stimuli to bypass the protection
mechanisms.
4. CONCLUSION AND OUTLOOK
Progress in the application of Virtual Prototypes for safety
assurance in the automotive and other domains with safety-
critical applications is mainly driven by the needs of industry.
In combination with other standards, such as SystemC and
UVM, VPs can provide an ideal platform for error effect
simulation, for the identification of weak spots of a system in
an early design phase. In this paper, the state of the art and
most recent research challenges, which need to be resolved
to efficiently apply this methodology on automotive systems,
have been outlined. In the upcoming years, we expect that
the research community wil invest a huge amount of effort
to overcome these challenges. Brought to the industrial
level, this methodology will certainly increase the safety of
ever increasingly complex automotive systems. This level of
security is certainly required to address future verification
challenges for automotive systems designs such as those
required for autonomous driving.
ACKNOWLEDGMENT
This work has been partially supported by the German Fed-
eral Ministry of Education and Research (BMBF) in the
project EffektiV under grant 01IS13022.
5. REFERENCES
[1] AUtomotive Open System ARchitecture (AUTOSAR)
Development Partnership Website. http://www.autosar.org/.
[2] Georg Georgakos, Ulf Schlichtmann, Reinhard Schneider, and
Samarjit Chakraborty. Reliability challenges for electric
vehicles: from devices to architecture and systems software. In
50th Annual Design Automation Conference (DAC), page 98,
2013.
 [3] C. Pinello, L.P. Carloni, and A.L. Sangiovanni-Vincentelli.
Fault-tolerant deployment of embedded software for
cost-sensitive real-time feedback-control applications. In
Design, Automation and Test in Europe Conference (DATE),
pages 1164–1169 Vol.2, 2004.
[4] Malcolm Wallace. Modular architectural representation and
analysis of fault propagation and transformation. Electron.
Notes Theor. Comput. Sci., 141(3):53–71, December 2005.
[5] Xiaocheng Ge, Richard F. Paige, and John A. Mcdermid.
Probabilistic failure propagation and transformation analysis.
In 28th International Conference on Computer Safety,
Reliability, and Security (SAFECOMP), pages 215–228, 2009.
[6] Bernhard Kaiser, Peter Liggesmeyer, and Oliver Mäckel. A new
component concept for fault trees. In 8th Australian Workshop
on Safety Critical Systems and Software (SCS) - Volume 33,
pages 37–46, 2003.
[7] J. Zimmermann, S. Stattelmann, A. Viehl, O. Bringmann, and
W. Rosenstiel. Model-driven virtual prototyping for real-time
simulation of distributed embedded systems. In 7th IEEE
International Symposium on Industrial Embedded Systems
(SIES), pages 201–210, 2012.
[8] S. Reiter, M. Pressler, A. Viehl, O. Bringmann, and
W. Rosenstiel. Reliability assessment of safety-relevant
automotive systems in a model-based design flow. In 18th Asia
and South Pacific Design Automation Conference
(ASP-DAC), pages 417–422, 2013.
[9] Ningfang Song, Jiaomei Qin, Xiong Pan, and Yan Deng. Fault
injection methodology and tools. In International Conference
on Electronics and Optoelectronics (ICEOE), volume 1, pages
V1–47–V1–50, 2011.
[10] Giacinto P. Saggese, Nicholas J. Wang, Zbigniew T.
Kalbarczyk, Sanjay J. Patel, and Ravishankar K. Iyer. An
experimental study of soft errors in microprocessors. IEEE
Micro, 25(6):30–39, November 2005.
[11] M. Rebaudengo, M. Sonza Reorda, and M. Violante. An
accurate analysis of the effects of soft errors in the instruction
and data caches of a pipelined microprocessor. In Conference
on Design, Automation and Test in Europe (DATE), 2003.
[12] M.L. Li, P. Ramachandran, U.R. Karpuzcu, S. Hari, and S.V.
Adve. Accurate microarchitecture-level fault modeling for
studying hardware faults. In IEEE 15th International
Symposium on High Performance Computer Architecture
(HPCA), pages 105–116, 2009.
[13] D. May and W. Stechele. An fpga-based probability-aware fault
simulator. In International Conference on Embedded
Computer Systems (SAMOS), pages 302–309, 2012.
[14] T. Schweizer, D. Peterson, J.M. Kuhn, T. Kuhn, and
W. Rosenstiel. A fast and accurate fpga-based fault injection
system. In IEEE 21st Annual International Symposium on
Field-Programmable Custom Computing Machines (FCCM),
pages 236–236, 2013.
[15] D. Peterson, O. Bringmann, T Schweizer, and W. Rosenstiel.
Stml: Bridging the gap between fpga design and hdl circuit
description. In International Conference on
Field-Programmable Technology (ICFPT), 2013.
[16] Lukai Cai and Daniel Gajski. Transaction level modeling: An
overview. In 1st IEEE/ACM/IFIP International Conference
on Hardware/Software Codesign and System Synthesisv
(CODES+ISSS), pages 19–24, 2003.
[17] Accellera Systems Initiative. Universal Verification
Methodology (UVM), May 2012.
[18] IEEE Computer Society. IEEE 1666-2011 Standard SystemC
Language Reference Manual, 2011.
[19] R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test
data selection: Help for the practicing programmer. Computer,
11:34–41, April 1978.
[20] Richard A. DeMillo and A. Jefferson Offutt. Constraint-based
automatic test data generation. IEEE Transactions on
Software Engineering, 17(9):900–910, September 1991.
[21] Yue Jia and Mark Harman. An analysis and survey of the
development of mutation testing. IEEE Transactions on
Software Engineering, 2010.
[22] Markus Becker, Daniel Baldin, Christoph Kuznik, Mabel Mary
Joy, Tao Xie, and Wolfgang Mueller. Xemu: An efficient qemu
based binary mutation testing framework for embedded
software. In Tenth ACM International Conference on
Embedded Software (EMSOFT), pages 33–42, 2012.
[23] C. Berger R. Rana, M. Staron and F. Törner J. Hansson,
M. Nilsson. Increasing efficiency of iso 26262 verification and
validation by combining fault injection and mutation testing
with model based development. 8th International Joint
Conference on Software Technologies (ICSOFT-EA), July
2013.
[24] Synopsys. CERTITUDE Functional Qualification System.
[25] V. Guarnieri, N. Bombieri, G. Pravadelli, F. Fummi,
View publication stats
H. Hantson, J. Raik, M. Jenihhin, and R. Ubar. Mutation
analysis for systemc designs at tlm. In 12th Latin American
Test Workshop (LATW), pages 1 –6, 2011.
[26] Tao Xie, W. Mueller, and F. Letombe. IP-XACT based system
level mutation testing. In IEEE International High Level
Design Validation and Test Workshop (HLDVT), nov. 2011.
[27] Thorsten Piper, Stefan Winter, Paul Manns, and Neeraj Suri.
Instrumenting autosar for dependability assessment: A
guidance framework. 43rd Annual IEEE/IFIP International
Conference on Dependable Systems and Networks (DSN),
0:1–12, 2012.
[28] Nicola Bombieri, Franco Fummi, and Valerio Guarnieri.
Accelerating RTL Fault Simulation through RTL-to-TLM
Abstraction. In European Test Symposium, pages 117–122,
2011.
[29] Nicola Bombieri, Franco Fummi, and Valerio Guarnieri.
Fast-gp: An rtl functional verification framework based on fault
simulation on gp-gpus. In Conference on Design, Automation
and Test in Europe (DATE), pages 562–565, 2012.
[30] Markus Becker, Christoph Kuznik, Mabel Mary Joy, Tao Xie,
and Wolfgang Mueller. Binary mutation testing through
dynamic translation. In 42nd Annual IEEE/IFIP
International Conference on Dependable Systems and
Networks (DSN), pages 1–12, 2012.
[31] T. Nirmaier, A. Burger, M. Harrant, A. Viehl, O. Bringmann,
W. Rosenstiel, and G. Pelz. Mission profile aware robustness
assessment of automotive power devices. In Conference on
Design, Automation and Test in Europe (DATE), 2014.
[32] ZVEI. Handbook for Robustness Validation of Automotive
Electrical/Electronic Modules. ZVEI - Zentralverband
Elektrotechnik- und Elektronikindustrie e. V., June 2013.
[33] Y. Li, M.-M. Louërat, F. Pêcheux, R. Iskander, P. Cuenot, M.
Barnasconi, T. Vörtler, K. Einwich. Virtual Prototyping,
Verification and Validation Framework for Automotive Using
SystemC, SystemC-AMS and SystemC-UVM. Embedded Real
Time Software and Systems (ERTS2), 2014.
[34] M. Barnasconi, F. Pêcheux, T. Vörtler, K. Einwich. Advancing
system-level verification using UVM in SystemC. Design and
Verification Conference (DVCon), 2014.
[35] Marcio F.S. Oliveira, Christoph Kuznik, Hoang M. Le, Daniel
Große, Finn Haedicke, Wolfgang Mueller, Rolf Drechsler,
Wolfgang Ecker, and Volkan Esen. The System Verification
Methodology for Advanced TLM Verification. In International
Conference on Hardware/Software Codesign and System
Synthesis (CODES+ISSS), 2012.
[36] Marcio F. S. Oliveira, Christoph Kuznik, Wolfgang Mueller,
Wolfgang Ecker, and Volkan Esen. A SystemC Library for
Advanced TLM Verification. In Design and Verification
Conference (DVCON), 2012.
[37] Yao Li, Ramy Iskander, Farakh Javid, and Marie-Minerve
Louërat. A Design and Verification Methodology for
Mixed-Signal Systems Using SystemC-AMS. In Jan Haase,
editor, Models, Methods, and Tools for Complex Chip Design,
volume 265 of Lecture Notes in Electrical Engineering, pages
89–108. Springer International Publishing, 2014.
[38] Ankur Sharma, Joseph Sloan, Lucas F Wanner, Salma H
Elmalaki, Mani B Srivastava, and Puneet Gupta. Towards
analyzing and improving robustness of software applications to
intermittent and permanent faults in hardware. In IEEE 31st
International Conference on Computer Design (ICCD), pages
435–438, 2013.
[39] Lucas Wanner, Salma Elmalaki, Liangzhen Lai, Puneet Gupta,
and Mani Srivastava. Varemu: An emulation testbed for
variability-aware software. In IEEE International Conference
on Hardware/Software Codesign and System Synthesis
(CODES+ ISSS), pages 1–10, 2013.
[40] Hyungmin Cho, Shahrzad Mirkhani, Chen-Yong Cher, Jacob A
Abraham, and Subhasish Mitra. Quantitative evaluation of soft
error injection techniques for robust system design. In 50th
Annual Design Automation Conference (DAC), page 101,
2013.
[41] Cristian Cadar, Daniel Dunbar, and Dawson Engler. Klee:
Unassisted and automatic generation of high-coverage tests for
complex systems programs. In 8th USENIX Conference on
Operating Systems Design and Implementation (OSDI), pages
209–224, 2008.
[42] H. M. Le, D. Große, V. Herdt, and R. Drechsler. Verifying
SystemC using an intermediate verification language and
symbolic simulation. In Design Automation Conference
(DAC), page 116, 2013.