See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/324509526

SIL2 assessment of an Active/Standby COTS-based Safety-Related system

Article  in  Reliability Engineering [?] System Safety · April 2018

DOI: 10.1016/j.ress.2018.04.009

CITATIONS READS

12 2,418

5 authors, including:

Giovanni Mazzeo L. Coppolino

Parthenope University of Naples Parthenope University of Naples
34 PUBLICATIONS   293 CITATIONS    100 PUBLICATIONS   971 CITATIONS   

SEE PROFILE SEE PROFILE

Salvatore D'Antonio Claudio Mazzariello

Parthenope University of Naples Hitachi Rail STS
96 PUBLICATIONS   897 CITATIONS    28 PUBLICATIONS   400 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

KONFIDO View project

KONFIDO - Secure and Trusted Paradigm for Interoperable eHealth Services View project

All content following this page was uploaded by Giovanni Mazzeo on 28 November 2018.

The user has requested enhancement of the downloaded file.

 SIL2 Assessment of an Active/Standby
COTS-Based Safety-related System

Giovanni Mazzeoa,∗, Luigi Coppolinoa , Salvatore D’Antonioa , Claudio Mazzariellob , Luigi Romanoa
a University of Naples ’Parthenope’ - Centro Direzionale, Isola C4, 80133 Napoli, IT
b Hitachi Ansaldo STS - Via Argine, 425, 80147 Napoli, IT

Abstract
The need of reducing costs and shortening development time is resulting in a more and more pervasive use of Commercial-Off-The-
Shelf components also for the development of Safety-Related systems, which traditionally relied on ad-hoc design. This technology
trend exacerbates the inherent difficulty of satisfying – and certifying – the challenging safety requirements imposed by safety
certification standards, since the complexity of individual components (and consequently of the overall system) has increased by
orders of magnitude. To bridge this gap, this paper proposes an approach to safety certification that is rigorous while also practical.
The approach is hybrid, meaning that it effectively combines analytical modelling and field measurements. The techniques are
presented and the results validated with respect to an Active/Standby COTS-Based industrial system, namely the Train Management
System of Hitachi-Ansaldo STS, which has to satisfy Safety Integrity Level 2 requirements. A modelling phase is first used to
identify COTS safety bottlenecks. For these components, a mitigation strategy is proposed, and then validated in an experimental
phase that is conducted on the real system. The study demonstrates that with a relatively little effort we are able to configure the
target system in such a way that it achieves SIL2.
Keywords: Dependability, Reliability, Safety-Related, COTS, ICS, SIL.

1. Rationale and Contribution
The key building blocks of IT infrastructures responsible
for the management, control, and regulation of industrial oper-
ations are generally referred to as Industrial Control Systems
(ICS). Among these, a wide variety of critical systems exists,
notably Safety-Critical Systems (SCS) and Safety-Related Sys-
tems (SRS). A SCS has full responsibility for controlling haz-
ards and consequently its failure or malfunction may result in
catastrophic outcomes, such as death or serious injury to peo-
ple, loss or severe damage to equipment/property, or environ-
mental harm. SRS support SCS, since they include the hard-
ware and software that carries out one or more safety functions.
Thus, failure of an SRS increases the risk for the safety of peo-
ple and/or of the environment (EN50129 says: “SRS carries
responsibility for safety”). The focus of this paper is on SRS.
Due to their importance, SRS must be proven to be reliable,
through rigorous and internationally accepted methodologies.
Standards (e.g. EN50129) exist, classifying quantitatively the
likelihood of a failure through the concept of Safety Integrity
Level (SIL). Specifications include four SIL levels, where SIL0
indicates that there are no safety requirements and SIL4 is typ-
ically reserved to SCS. Table 1 shows the classification of the
∗ Corresponding
author
Email addresses: giovanni.mazzeo@uniparthenope.it (Giovanni
Mazzeo), luigi.coppolino@uniparthenope.it (Luigi Coppolino),
salvatore.dantonio@uniparthenope.it (Salvatore D’Antonio),
claudio.mazzariello@ansaldo-sts.com (Claudio Mazzariello),
luigi.romano@uniparthenope.it (Luigi Romano)
Preprint submitted to Reliability Engineering and System Safety
different SIL levels. For each of them, we reported the asso-
ciated Tolerable Hazard Rate (THR) bounds, the failure mode,
the consequent hazard, and the related system typology (i.e.,
SCS or SRS). Making a system compliant to a specific SIL
means providing evidence of the achievement of THR thresh-
olds, which – for a complex system – is by no means a trivial
task.
The assessment process is even more difficult when Commercial-
Off-The-Shelf (COTS) components – whose internals are par-
tially or totally unknown – come into play. COTS are being
increasingly used by the industry to reduce costs and to shorten
development (and possibly deployment) time. However, since
COTS are general-purpose components that have not been de-
signed and developed for robust operation, obtaining a predictable
operation profile (for individual components and – even more
– for the resulting system) is a challenging endeavour. The re-
search community has proposed techniques – mostly guidelines
– to help reliability engineers carry out the assessment of safety
properties [1][2][3][4][5]. However, the aforementioned stud-
ies are mainly qualitative. It is also worth emphasizing that
the existing literature – with a few exceptions (e.g. [1]) – does
not refer to real industry applications, which limits the applica-
bility of the proposed techniques to commercial setups. Con-
versely, we contribute a methodological framework that can be
used in practice as a reference for certifying a wide class of
emerging critical systems, virtually any system for which: i)
the general architecture has already been designed, ii) business
constraints impose that (radical) changes to the architecture be
avoided, and iii) the main COTS components that must be inte-
November 28, 2018
 Table 1: Classification of SIL levels with associated THR and system typology
THR Failure Mode
The system cannot be
≥ 10−9 to < 10−8
recovered by the operator
Only an experienced operator
≥ 10−8 to < 10−7
can recover the system
An operator can recover
≥ 10−7 to < 10−6
the system
Minor availability issues.
≥ 10−6 to < 10−5
Always recoverable
There is no safety requirement
on the system
grated have already been chosen. We demonstrate, with respect
to a real industrial system, that it is possible to achieve SIL2
via proper configuration of system parameters and set-up of re-
juvenation procedures. The paper in fact addresses SIL2 as-
sessment of a real COTS-based SRS, specifically the two-nodes
cluster server hosting the Train Management System (TMS) ap-
plication of Hitachi Ansaldo STS (ASTS). The study enabled
ASTS to identify the conditions under which the architecture of
the Active/Standby cluster – incorporating COTS software and
hardware – can be certified as SIL2 compliant, as specified in
EN50129.
In order to achieve this goal, the system is assessed by means
of a hybrid approach, i.e., a combination of analytical mod-
elling and experimental evaluation. Analytical modelling is
done using PRISM (the well-known and widely used tool for
formal model checking), while experimental evaluation relies
on direct measurements on the real system. The hybrid ap-
proach we present consists of several phases, which can be
grouped in five main iterative stages, i.e.: 1) Identifying the
THR and safety bottlenecks of the system in its current config-
uration through formal models; 2) Defining possible corrective
actions; 3) Weighing their effectiveness, i.e. evaluating the po-
tential impact on the THR cluster model; 4) Validating the re-
sults of previous phases through an experimental campaign on
the real system. 5) Using experimental estimates within models
to calculate the final THR.
In the specific case of the ASTS TMS system, the safety bot-
tlenecks turned out to be COTS Operating System (OS) and
Cluster Resource Manager (CRM). Extended versions of the
cluster model were drawn to evaluate the effectiveness of mit-
igation actions. The less costly one (in terms of resources and
effort), which also provided results of THR in SIL2, was soft-
ware rejuvenation[21]. A Quantitative Accelerated Life Test
(QALT) and an Accelerated Degradation Test (ADT) were used
to estimate the experimental Mean Time To Failure (MTTF) and
Time To Aging-Related Failure (TTARF) of a single server node
in the short-term and in stressed execution, respectively. Such
measurements were used as input to the rejuvenation-extended
cluster model, and the final value of THR was found.
This paper makes three important contributions:
• It presents a comprehensive methodology, providing a
practical path to SIL2 certification of COTS-based sys-
tems.
2
System
SIL Hazard
Typology
Catastrophic and Fatal
4 SCS
Outcome
Critical and Fatal
3 SCS
Outcome
Marginal, Injuries may
2 SRS
occur
Negligible, Minor injuries
1 SRS
may occur
Nuisance, Dissatisfying to
0
the user
• It provides quantitative measurements of failure rates in a
real setup, that can be reused in other industrial contexts.
• It quantifies the impact of some best practices that can be
implemented in a wide class of systems, to increase their
dependability.
The remainder of this work is organized as follows. Sec-
tion 2 gives an insight into previous work. Section 3 describes
the ASTS use case. Section 4 presents the approach adopted.
Sections 5-6 describe the cluster modelling and the mitigation
strategies proposed, respectively. These are followed by Sec-
tion 7, that describes the experimental validation phase. Finally,
section 8 concludes the document with some final remarks.
2. Related Work
Goal of this section is to show related work focused on
methods useful to assess the reliability of clustered COTS-based
systems.
Connelly et al. [1] propose an approach to delineate safety
assurance for the use of COTS OS in safety-related applications,
which must fall in SIL2. Their proposed solution is focused
on developing encapsulation mechanisms able to isolate the in-
fluence of a COTS failure. They analyzed OS failure modes,
which may affect safe functionalities, and then proposed miti-
gation techniques whose impact is not reported. Unlike [1], our
work provides an entire methodology with tangible and quanti-
fied results that can be actually useful for other use. Qualitative
estimations are definitely needed but not enough. Our contri-
bution is a rigorous analysis of models and experimental results
of the clustered system that gives deep evidence on the system
reliability.
Jones et al. [5] survey available practical methods useful to
assess the safe integrity of COTS-based systems with standard
IEC61508. Authors propose the adoption of testing techniques
like stress, interface and statistical test (black-box methods) or
tools available to check software, analyze data flows, and in-
ject faults (white-box methods). Limitations of both methods
are presented, i.e., the lack of failure data confidential to the
supplier, the difficulty when computing results of tests with-
out automated mechanisms, the problem of optimizing time-
consuming tests, the difficulty of covering a wide range of soft-
ware faults. Our paper does more than that: it defines an assess-
ment methodology flow and shows an implementation on a real order to regulate trains movements and prevent fatal accidents.
use case with also on-field evaluations. The overall architecture of the RTCS (figure 1) is composed by
Park et al. [4] evaluate the effect of rejuvenation actions on
the availability of an Active/Standby cluster. Authors defined a Human Machine Interface (HMI)
generic Markov model through which estimate how the system Train Management System Application Server
availability varies by changing repair time and period of reju-
ASTS TMS Application ASTS TMS Application
venation. Main weaknesses of this paper that we contribute to
address are: i) a purely theoretical estimation without evidence Pacemaker & CoroSync

or proves from real world; ii) an estimation of the best rejuve-
nation period without considering that the aging of the system
depends on aging factors and so needs empirical evidence; iii)
the underestimation of the possibility that the cluster resource
manager fails, which is something that can certainly occur.
Skramstad et al. [2] perform a study of the possible solu-
tions proposed by the academic and industrial community to
address the certification of critical systems composed of COTS
to a specific SIL level. They got to three different considera-
tions: one could be to supervise the memory through memory
mapped storage or by calculating checksums regularly; another
one could be to test the system even though this is an unfeasi-
ble solution when the COTS is represented by a whole operating
system; and finally, to diversify the adoption of COTS compo-
nents to avoid common failure modes. This study, unlike ours,
is fairly superficial, authors report few approaches available in
literature without providing a comprehensive analysis.
Finally, Pierce et al. [3] present a detailed report on how to
assess Linux for SRS. This document provides guidelines that
should set more guarantees on the OS reliability. Many OS fea-
tures are identified as possible sources of failure and for this
reason they should be disabled, i.e., the developer should create
a monolithic kernel with a minimum number of functionalities.
The conclusion of the study is that Linux, properly tuned, would
be suitable for use in many safety related applications with SIL1
and SIL2 requirements. Such a work is indeed of value for the
amount of information provided, but at the same time the ef-
fective impact of mitigation techniques on the reliability is not
predicted.
What emerges from literature is the lack of quantitative es-
timations and practical examples of certification assessment in
industrial applications. Several techniques or approaches are
presented but none of them can be actually of support for the
certification. This work wants to bridge the gap by providing
an entire verified approach to face the SIL2 certification of al-
ready existent systems set up.
3. Overall System Architecture
In this section we provide a description of the system un-
der study, and its surrounding environment, on an “as is” basis.
The architecture presented in the following is the one currently
adopted on-field by ASTS (e.g., at Rome train station).
The Train Management System (TMS) – being investigated in
this work – is part of a larger and more complex set of subsys-
tems needed for the train monitoring that goes under the name
of Railway Traffic Control System (RTCS). The RTCS, used by
ASTS, is a hierarchically structured system designed to monitor
railway traffic, control railroad signals, and track switches in
3
Linux Suse OS Linux Suse OS
SIL2 NIC NIC
Fan HDD Fan HDD
Pwr RAM CPU Pwr RAM CPU
Motherboard Motherboard
Node 1 - Active Node 2 - Standby
SIL4 Interlocking System (IXL) RBC
ERTMS Euroradio
Sensors Actuators
Figure 1: The ASTS Railway Traffic Control System
the following layers:
Sensors/Actuators - Multiple sensors and actuators are de-
ployed on rail tracks, trains, traffic lights, and the surrounding
environment to receive monitoring information and send con-
trolling signals.
European Rail Traffic Management System (ERTMS) Euro-
radio - A safety communication protocol used to transfer data
between sensors and the Radio Block Center.
Radio-Block Center (RBC) - A component responsible of
collecting signals coming from or directed to field sensors through
a GSM-R radio transmission (at 900MHz).
Interlocking (IXL) - A Safety-Critical System made of mul-
tiple signal apparatus able to prevent trains from conflicting
movements by only allowing trains to receive authority to pro-
ceed, when routes have been set, lock and detected in safe com-
binations. The IXL is a vital subsystem with hard real-time con-
straints. A missed deadline of the IXL may cause catastrophic
failures. For this reason, ASTS designed the IXL with HW&SW
dedicated to hard real-time systems, which were certified with
a SIL4 level.
Train Management System (TMS) - The TMS provides non-
vital functions that oversee and automatize trains movements
and support the dispatcher in operations of train traffic control
and management in a wide area of a railways network. The
TMS communicates with the distributed IXL signalling system
to monitor the railways area and send control signals. Data re-
ceived from the IXL is, e.g., stored in databases or sent to multi-
ple Human Machine Interfaces (HMIs) workstations. The TMS
is installed in a central control centre where operators take de-
cisions.
The TMS is considered failed if either the monitoring appli-
cation provides to the operator wrong information that do not
reflect the real railways status (functional failure), or if the ser-
vice completely stops being provided (non-functional failure).
The architecture currently adopted by ASTS, therefore, aims at
achieve a high grade of availability through redundancy. How-
ever, the only usage of redundant HW subsystem units is not
satisfactory for the SIL2 certification, especially if the system
brings into it COTS components.
Unlike the IXL, the TMS provides non-vital functions. A failure
of the TMS could lead, as an example, to the planning of col-
liding routes. In such a case the underlying SIL4 fail-safe IXL
would prevent loss of life avoiding the actual impact, e.g. by
stopping the trains. However, the shutdown of trains would re-
sult in degradation of QoS and reduction of service availability,
which turn in loss of money and reputation for the train com-
pany. For this reason, the TMS is classified as a Safety-Related
System.
The TMS is based on a client-server architecture. It is a col-
lection of commercial redundant equipment (dual application
servers, dual data base servers and at least dual workstations)
connected to a high speed LAN. Core of the TMS subsystem
is the Application Server which runs most important SW mod-
ules. The TMS Application Server – for simplicity, from now
on only TMS – is composed by two server machines clustered
in a Active/Standby configuration to provide uninterrupted ser-
vice. To further enhance reliability, redundancy is not only ap-
plied in terms of replicated machines but also in terms of sub-
system units (e.g. RAM, Disks, Fans, Power Supply) in order
to eliminate Single Points of Failure (SPF). Units replicas are
equal, that is, ASTS does not enforce a diversity fault-tolerant
mechanism which involves the usage of subsystems of different
technologies. Each server node, connected to its motherboard,
has: 2 hot-swap power supplies, 2 hot-swap fans, 2 RAMs, 1
CPU, 2 HDDs in RAID1 configuration, 1 Network Interface
Card (NIC).
On top of the hardware, ASTS employs Linux SUSE with High-
Availability (HA) Extension: a COTS OS provided with two
clustering software (Pacemaker Cluster Resource Manager (CRM)
and Corosync) responsible for resources orchestration, failure
diagnosis, nodes coordination, and fail-over management. The
Pacemaker CRM allows for monitoring the health and status
of node resources, managing dependencies, and automatically
stopping and starting services. It relies on Corosync messaging
layer, in charge of reliable messaging communications, mem-
bership and quorum information needed by the cluster for node
orchestration. For simplicity, in the rest of this paper, Pace-
maker and Corosync are indicated as a single entity under the
CRM notation.
The topmost layer of the system is the ASTS proprietary TMS
application. This consists of several SW modules that commu-
nicate with the IXL through a CORBA message broker. Since
the ASTS proprietary application software is beyond the scope
of our work, the reliability of the broker itself is not addressed
in our study (except for aspects related to its interaction with
other software modules). The decision of using a CORBA bro-
ker was taken by the ASTS proprietary application software
development team. In this study, we assume that the ASTS
proprietary application software (which includes the broker),
is the result of a rigorous development process, and of a thor-
4
ough reliability testing activity. The ASTS proprietary applica-
tion software uses FT-CORBA, which embeds fault tolerance
mechanisms that help to ensure a resilient and highly-available
message broker service. FT-CORBA was used, e.g., for an Air
Traffic Control System, which is indeed a safety-critical system
[12].
The focus of this work is to perform a thorough reliability eval-
uation on software layers below the ASTS proprietary applica-
tion.
4. Concepts and Approach
The Safety Integrity Level (SIL) is a way to indicate the tol-
erable failure rate – interchangeably defined as Tolerable Haz-
ard Rate (THR) – of a particular Safety Instrumented Function
(SIF). The TMS implements one main SIF function, i.e., the
continuous monitoring and controlling of a particular railways
area, which must be kept in a safe state, with respect to the func-
tional or non-functional failures defined in 3.
The correct provision of the SIF does not depend on the time
in which it is performed, i.e., the TMS has not stringent real-
time requirements. This work assumes that functional or non-
functional failures can be generated by (Figure 3): 1) Hard-
ware Faults caused by a hard or soft malfunction within the
electronic circuits or electromechanical components, which re-
quire a repair intervention or a reboot, respectively; 2) Software
Faults caused by errors influenced by the total time the system
has been running, i.e., Aging-Related Mandelbugs, or any other
SW fault not caused by aging errors, i.e., Non-Aging-Related
Mandelbugs [16].
Software aging is often cause of Linux failures at both Kernel
and User level, as Cotroneo et al. [21] demonstrate. Aging-
related faults – caused by exhaustion of OS resources (e.g. mem-
ory leak) – can reduce performance or culminate in a system
hang/crash.
If a fail-stop failure of a node occurs, that is, the server node
exhibits a crash or a hang, than, the CRM detects it, unless
the CRM itself is not failed. The TMS cluster stops providing
services (it is down) when: the active node fails and the CRM
fail-over operation is in action, or the active node and the CRM
fail, or the active and standby nodes fail together.
The ratio between the number of manifested TMS failures and
the operating time is the failure rate, which is the inverse of the
Mean Time Between Failures (MTBF): the mean operating time
(uptime) between failures of the cluster.
The overall goal of the assessment process is to provide
evidence able to demonstrate that the TMS failure rate falls
within SIL2 bounds: 10−7 ≤ T HRS IL2 < 10−6 . The assess-
ment would be facilitated if it had been possible to build the
system from scratch using hardware and software components
dedicated for mission-critical systems (i.e. VxWorks, PikeOS,
QNX). However, this would have meant for ASTS a drastic in-
crease in terms of costs, which cannot be justified for a system
without stringent real-time requirements. For this reason, the
fulfillment of SIL2 is pursued within the context of the already
defined TMS architecture, and using the COTS components that
have already been selected. The paper shows how it is possible
 TMS Modeling and Analysis

Model
Preliminary Units Nodes Cluster Reliability
Inputs
Study Modeling Modeling Modeling Evaluation
Definition

Enforce No
Mitigation THRTMS < THRSIL2
Strategies
Experimental Validation
No
Yes
QALT
Yes Analysis
End THR*TMS < THRSIL2 Reliability Accelerated
Evaluation Tests
ADT
Analysis

Figure 2: Assessment Methodology Flow Diagram

TMS to alternate the Active node. The latter is chosen as it is a good

Failure compromise between costs and impact on the overall THR.
Experimental Validation Experiments are really important
Software Hardware
in the certification process. Standards (e.g. EN50129) clearly
Fault Fault state that ”Fail-safe behaviour of component under adverse con-
ditions shall be demonstrated”, and it is desirable to obtain ”Ev-
idences that the failure mode will not occur as a result of com-
Non-Aging- Aging- ponent ratings being exceeded”.
Soft Hard
related related
Malfunction Malfunction The experimental phase, in this paper, aims at: 1) Demonstrate
Mandelbugs Mandelbugs
the goodness of TMS node failure rate estimation; 2) Define the
Figure 3: Typologies of faults that could cause TMS failure TMS TTARF in order to determine a proper period of rejuvena-
tion.
The system is subjected to a stress loading scheme through a
to configure the existing design so to achieve the desired level
workload generator. Then, failure and degradation data sets are
of reliability.
used for a QALT/ADT analysis. QALT and ADT are proven as
To do that, in this paper an hybrid approach is pursued (fig-
the best solutions to measure reliability metrics and, at the same
ure 2) composed by the following three main phases:
time, understand and quantify the effects of stress. QALT/ADT
System Analysis and Modeling - In this work, a three-level are usually leveraged for HW components. However these have
hierarchical modeling – composed by combinatorial and state- been demonstrated feasible to observe the behavior of SW suf-
space models – is used: 1) First, failure rates (λU ) of TMS fering from software aging [20][22].
servers subsystem units are estimated through continuous Markov
chains; 2) Then, the single server node failure rate (λN ) is pro-
duced through a Fault Tree, which OR results of previous phase; 5. Modeling and Analysis
3) Finally, at the top, the overall cluster is modelled. Such
The three-level hierarchical modelling of the TMS cluster is
a model is specified and analysed using a formal verification
here deepened. In the remainder of this section different rates
method. In particular, in this paper, the probabilistic model
– having an exponential distribution – are used. While some,
checking tool, namely PRISM [18], has been used. This allows
like failure rates, depend on the unit/system under study, others
an automatic verification of specific properties of the proba-
are fixed. The MTTR, e.g., is based on ASTS service agree-
bilistic model defined, useful to determine the compliance of
ments, which guarantee units replacement within 18Hrs → µ =
the cluster to SIL2 requirements.
1/64800s. The Time to Switch – equal to 30s – is evaluated
Mitigation Strategies Enforcement Most relevant outcomes through on-field tests of Active-Standby switches → θ = 1/30s.
coming from TMS modelling are leveraged to propose possi- In the same manner, the Time to Reboot has been calculated and
ble ways useful to enhance the final cluster THR. In this work, is equal to 302s → γ = 1/302s.
mitigation strategies are proposed to address both software and
hardware failures. Possible identified solutions were: i) increas- 5.1. Subsystem Units
ing the number M of Active nodes and enforce a M-to-1 config-
Estimating subsystem units failure rates is fundamental for
uration; ii) reducing Single Point of Failure (SPF); iii) enforcing
the final THR definition. In this regard, dedicated Markov chains
a software rejuvenation of server nodes to reduce the impact of
– needed to model the replication of a specific unit – have been
aging-related failures, and take advantage of the system reboot
5
used. Reporting a diagram for each element would be too much
as, in most cases, these are trivial. What is important, instead,
is the definition of failure rate values used in input to models
since these are particularly difficult to define.
Unfortunately, getting faithful estimates is non-trivial as it is
difficult to be sure of the accuracy. Nevertheless, field studies,
which are also strongly recommended by standards, represent a
good chance. The standard CEI-EN-50129, e.g., specification
says: ”Random HW failure rates, or probabilities of component
failure, should be based on field data if possible”.
5.1.1. Memory
TMS memories are replicated and equipped with IBM Chip-
kill technology that protects memory from any single chip fail-
ure and from multi-bit errors. Therefore, the Markov model
related to this subsystem should model the ChipKill behaviour.
However, Sridharan et al. [13] in their memory failure estima-
tion already bring in the ChipKill mechanism. For this reason,
the RAM Markov chain can take into account the only effect
of replication. Researchers estimated through an experimental
campaign that the Failure Rate – in accordance with [8] – is
λ = 1 × 10−5 . Using this estimate, the memory Markov model
yields λRAM = 2.6 × 10−8 .
5.1.2. Disks
Disks’ model needs to take into account the RAID1 mirrored
replication as Eckartet al. [19]. Schroeder et al. [7] performed
an exhaustive 5-year test campaign on 100’000 different types
of disks in high-performance computing sites. Authors got em-
pirical conclusions on field replacement rates of drives with less
than five years old, and on failure rates trends during the years.
They obtained a failure rate λ = 2 × 10−4 . Such a value, in input
to a RAID1-aware model, produces λHDD = 3.1 × 10−8 .
5.1.3. CPU
The failure rate estimate for this unit has been extracted
from Nightingale et al. [9] work who – in their field study –
made a thorough analysis of hardware failure rates for con-
sumer PCs investigating failures coming from CPU, DRAM,
and disks over 8 months of tests. They found that CPU and
DRAM failures are strongly dependent. Authors state that nu-
merical reliability results obtained are consistent with [7][8]. In
the worst case, the estimated Failure Rate of CPUs is λCPU =
5.8 × 10−5 [9]. This is the final value used as the CPU has no
replicas.
5.1.4. Fans
Jin et al. [14] conducted a deep study on the reliability of
fan systems. They report their fans’ reliability using different
parameter values in Weibull distribution, acceleration factors,
and reliability tests. Authors conducted accelerated tests with
higher level of stress in order to estimate the Failure Rate, ob-
taining 4 × 10−4 . This – given in input to a trivial Markov chain
needed to model the unit replication – yields λFan = 3.2 × 10−6 .
6
Table 2: Reliability Orders of Magnitude
ASTS
RAM CPU HDD Fans M.Board NIC PWR OS Node
Cluster
λ 10−8 10−5 10−8 10−6 10−5 10−6 10−8 10−5 10−5 10−6
5.1.5. Motherboard
Even if a rigorous statistical study on motherboards is not
available, an estimation of the motherboard failure rate is needed.
In this work a rough measure of the motherboard failure rate has
been carried out: starting from [11], rates of motherboard com-
ponents failed in one year is equal to 0.7%. Knowing that the
number of produced ASUS (one of the market leader) mother-
boards in one year is 17M, the number of failed units is around
119K. Hence, the evaluated Failure Rate, supposing a mean
usage time of 8Hrs per day, is λ Mboard = 1.6 × 10−5 .
5.1.6. NIC and Power Supply
Smith et al. [6] report a credible range of vendor-confidential
failure rates for different hardware units based on IBM predic-
tions. The upper bounds, representing the worst measure, may
be adopted in this case. Hence, failure rates of NIC and Power
Supply are, respectively: λNIC = 5.6×10−6 and λPWR = 6×10−4 .
The latter is replicated and therefore its final value – produced
through a Markov Model – is λPWR = 8.7 × 10−8 .
5.1.7. Operating System and CRM
Modelling the OS and the CRM is not feasible as their in-
trinsic ”stateful” characteristic make such a task impractica-
ble. However, their contribution to the overall reliability of the
server node must be taken into account as failures in the OS
or the CRM can certainly lead to an outage of the TMS. For
this reason, in this paper, the worst estimates of λOS and λCRM
that can be found in literature are considered for the modelling
phase. An estimate of OS Failure Rate is obtainable in [15]. In
this work, different servers operating systems were tested in one
year. Authors argue that Linux SUSE, the one on ASTS TMS
severs, had 13mins of outages. This means that in one year the
failure rate is λOS = 2.4 × 10−5 . Regarding the CRM, instead,
the estimation of failure rate is extracted from Mendiratta et
al. [17] work, which statistically determine λCRM = 2.2 × 10−4 .
5.2. Nodes
The evaluation of node’s failure rate is realized through a
Fault Tree which merges previous results. It is assumed that
a single node fails in case at least one of the replicated/non-
replicated units analysed before goes down. Hence, units are in
a OR relation between each other. This means that the resulting
failure rate is strongly influenced by the worst unit.
It is worth noting that the server fault tree brings in also the
COTS OS contribution. This, in fact, cannot be excluded from
the final evaluation as certainly its effects impact the overall
reliability. The CRM failure rate, instead, is not taken into ac-
count in the node estimation since it is part of the cluster model
shown in next subsection.
The estimated node failure rate is λNode = 6.7 × 10−5
5.3. Cluster before the repair is accomplished the CRM or the standby node
A formal modelling and analysis of the overall cluster is fail, then the cluster becomes down in state 3 or 5. The rest of
performed through the probabilistic model checker PRISM [18], the model is quite the same and does not require a further de-
developed by the University of Birmingham. PRISM provides scription. The TMS behaviour is then checked through a set of
automatic verification of Continuous Stochastic Logic (CSL)
λcrm
properties for Continuous-Time Markov Chains (CTMCs) prop-
µcrm
erly defined in a state-based language. It allows constructing a λa λs
6 θ
mathematical model able to capture the system’s behaviour, and 7 8 µs 5
then use it to analyse formally-specified quantitative properties.
PRISM parses one or more temporal logic CSL properties – λs µa
µs µa µcrm
properly defined by the user – and performs model checking, λcrm λs
λa 3
determining whether the model satisfies each property. 0
µcrm
The model reflecting the TMS cluster (available on github1 ) λa µs
λcrm 4
comprises three components that need to be evaluated: active 1
µa
and standby nodes, and the CRM. As already stated, these can λcrm
µs
manifest software or hardware faults leading to a crash. The λs
µa
TMS is considered down whether the active fails and the CRM is µcrm

performing the failover operation, or both the active and standby 2 λa

fail, or the active and the CRM fail. Failure rates of active and
standby nodes are assumed the same since the standby is in
Hot configuration, i.e., it is running and continuously updat-
ing the application state on the active node. The TMS analy-
sis in PRISM focuses upon a CTMC with transition rates ex-
ponentially distributed. To accurately delineate the different
states in which the cluster can be, four PRISM modules are de-
fined: active nodes, standby nodes, CRM, and cluster. The num-
ber of states (or nodes) belonging to active and standby mod-
ules are kept generic (NA , NS ) in order to subsequently verify
how results vary by changing the amount of active and standby
nodes (S A = {sa 0 , .., sa NA }; S S = {s s 0 , .., s s NS }). A transition
si −→ si−1 represents a node failure with rate λ, while the tran-
sition si −→ si+1 stands for a node repair with rate µ.
The CRM module, instead, is represented by only two states
(S C = sc 0 , sc 1 ) needed to indicate which of the two possible ex-
ecution condition the CRM is in, i.e. whether it is working, or
it is failed due to a software fault. The failure occurs with rate
λCRM , while the recovery, i.e. a node restart, is realized with
rate µCRM .
The cluster module, instead, is used to define the overall
state of the TMS in each instant of time. In PRISM jargon, it
is synchronized on the state of the other modules transitions:
when a module changes its state, the cluster module reacts ac-
cordingly. When the failure of the last active node occurs the
[active failure] action triggers the cluster module associated
states.
Figure 4 reports the Markov chain representative of the clus-
ter module. Gray nodes mean that the cluster is temporarily
down. The TMS starts in state 7 where both nodes and the CRM
are working correctly. The cluster may experience a failure of
the active node (with rate λa ) moving the system in an inter-
mediate state in which the CRM is performing the failover (in
time 1/θ). After that, the cluster goes in state 8 where it is up
but with only one node available. A repair intervention (with
rate µa = 1/MT T R) could bring the system back to state 7. If
1 https://github.com/dzobbe/cluster-reliability-model
Figure 4: TMS Cluster Markov Model
CSL properties defined in PRISM syntax. A set of rewards is
also assigned to states. Through these a wider range of quantita-
tive measures relating to model behaviour can be observed. For
the TMS case study, three rewards have been defined to evaluate
three cluster conditions: up, danger, down.
The following properties are model checked for the TMS:
(1) The expected TMS failure rate until time T.
(2) The expected time the TMS cluster is in state down, dan-
ger, up until T.
(3) The probability of any failure occurring within T days.
(4) The probability that Active/Standby/CRM cause the fail-
ure of the TMS.
5.4. Results
Figure 5 reports main outcomes of the TMS model checking
activity. A first result of interest comes from the evaluation of
property 1 that relates to the time spent by the system in con-
ditions of up (state 7), down (states 6,5,4,3), or danger (states
0,1,2,8). Figure 5a shows the evaluation in a time window of
365 days. The TMS is down 4.38 × 10−4 days, it is in dan-
ger 1.75 × 10−2 days, while it is up 364.999 days. This means
that the system Availability = U ptime/(U ptime+Downtime) =
0.99999.
It is interesting to emphasize another remarkable outcome: the
inverse proportionality relation between the number of active
nodes and the probability of TMS failure. The result – an esti-
mate of property 2 – is plotted in figure 5b in a time window of
30 days. The increased level of redundancy, in fact, leads to a
reduced downtime. When one active fails, the TMS experiences
any downtime caused by the active-standby switch.
Then, property 3 is evaluated. Table 5d reports the number of
failures occurred – and the related failure rate – as the time in-
creases. Numbers tell that, with the current configuration, the

7
 (a)
(c)
Figure 5: TMS Formal Model Results
cluster has a failure rate (2.75 × 10−6 in 8760Hrs) larger than
SIL2 thresholds.
Finally, by evaluating property 4, it is possible to notice how
much the COTS CRM impacts on the overall reliability. Figure
5c, in fact, shows the probability of each failure type (the CRM,
the active or the standby node) occurring first. It is evident that
the CRM has the highest effect on the overall reliability. At the
same time, the standby node has the lowest impact due to its
non-running condition.
6. Mitigation Strategies
Results coming from TMS formal model verification proved
that the current system configuration is not compliant with SIL2
bounds. Hence, mitigation strategies need to be defined and
applied in order to reach the desired level of reliability.
approach consists in using different cluster configurations. In
this sense, two possibilities may be pursued, i.e.:
• Enforce a M-to-1 structure, that is keeping one standby
node and increasing the number M of active nodes. In
fact, as mentioned in 5.4, the probability of failure could
drastically decrease by only adding one node. In that
case, the PRISM modeling tool produces a failure rate
of 3.41 × 10−8 , which could be enough to satisfy SIL2 re-
quirements. However, such a solution may be expensive
for ASTS in terms of costs of equipment purchasing and
on-field server maintenance.
• Replace the Active/Standby configuration with Active/Ac-
tive. In this way, both nodes actively run the TMS appli-
cation and the workload is balanced between them. This
8
(b)
Num Failure
Time(Hrs) Failures Rate
1080 0.003 3.43E-7
2880 0.008 9.17E-7
5040 0.014 1.59E-6
6840 0.019 2.17E-6
8760 0.024 2.75E-6
(d)
reduces the load on each server with less stressful situa-
tions for the HW components that get equally worn-out.
However, this solution would entail that both nodes ac-
cumulate software errors – e.g., due to memory leaks –
which could remain dormant or not. A different solution
that also ensures a uniform wear of HW units is explained
later.
An additional approach aims at mitigate the failure probability
of most critical components – i.e., the COTS OS and its CRM
– which were proven to be the weak points of the cluster (fig-
ure 5c). Regarding the OS, measures can be taken for example
at kernel level as suggested by Pierce et al. [3] that provide
thorough guidelines in this sense. The idea is to configure the
kernel to serve only the critical application disabling unused
modules, driver peripherals, the graphical interface X window
system, unused user processes. Other actions can be enforced
on Linux SUSE and, in particular, on Pacemaker/Corosync. In
fact, there are CRM settings that could affect the response of
the system in case of failure conditions and that define policies
on the management of the critical service. All of these, how-
ever, can provide a little improvement, which is also difficult to
quantify.
Hence, what is proposed here is a technique highly used in
reliability engineering: Software Rejuvenation [21][4], a tech-
nique of proactive fault tolerance in which the system is period-
ically reboot to clean the memory. In fact, it is well known that
most critical SW failures are transient. These may be caused
by error conditions due to software aging phenomenon, that is
the issue that SW can exhibit data corruption or unlimited re-
source consumption during its execution time. Although faults
left by developers still remain, the periodic rejuvenation can
help to remove or at least minimize transient failures thus re-
ducing possible outages.
In order to evaluate the rejuvenation impact, an extension
to the PRISM CTMC model seen in 5.3 is provided. Figure 6
shows the new states added to the cluster model. It is assumed
that each node after a certain time interval (α) moves to the
failure-prone state 9. From that moment, unexpected shutdown
of the Active node (with rate λa ) may be experienced and thus
requiring the switch-over to the Standby node (S i ) that will be
completed after Time to Switch (θ). Only a rejuvenation action
– occurring with rate β – can bring the cluster back from 9 to
the state of higher robustness.
The process of rejuvenation is leveraged as an opportunity to
periodically switch the Active node running the TMS applica-
tion, thus avoiding a situation where one node always runs, ac-
cumulates SW errors, and stresses the same HW units. Hence,
when the rejuvenation begins the cluster enforces the switch to
the other node within time θ. Therefore, the cluster will have
only one server available in the time interval required to imple-
ment the hardware reboot (rate γ) of the node under rejuvena-
tion. This is acceptable since the rejuvenation action may be
triggered in time windows of low workload. Rates α and β have
λs
β
10 9 at some stress levels to the system lifetime distribution in its
λa
θ
α
11
γ 6 θ
7 to estimate the time to failure is the Inverse Power Law (IPL):L(s) =
µa
Figure 6: Rejuvenation Extension
been tuned through a PRISM formal checking. Results, in line
with [10], demonstrate that optimal values are α = 336Hrs and
β = 168Hrs.
The rejuvenation yields a better failure rate result: 2.28 × 10−7 .
Such estimate – compliant with SIL2 requirements – needs now
an on-field verification.
7. Experimental Validation
This section illustrates the experimental validation conducted
on a TMS cluster test bed having the same architecture seen in
3. In such a test bed, the TMS receives messages on railway
status from a simulated IXL actually used by ASTS for test pur-
poses. Experiments aim at proving the validity of model results
and the proposed rejuvenation strategy. The analysis of exper-
iments moves in two directions. The idea is to test the TMS
under stressful conditions by looking at:
(1) Functional and crash/hang failures caused by aging/non-
aging related mandelbugs to evaluate the on-field Mean
Time To Fail (MTTF) of single server nodes.
(2) Aging degradation factors to estimate the Time To Aging
Related Failure (TTARF) and so being able to define an
empirical rejuvenation period.
9
In both cases, two widely accepted techniques [20][22] have
been used to obtain valid estimates and, at the same time, min-
imise the test duration.
A Quantitative Accelerated Life Test (QALT) is the tech-
nique adopted to evaluate (1). QALT is designed to provide
reliability information on a component or system through fail-
ure data obtained from an accelerated test. It is usually adopted
for hardware component testing but it is also demonstrated by
Matias et al. [20] as an accurate solution to observe – in short-
term and stressed executions – the behavior of systems suffer-
ing from software aging. QALT allows a quantification of the
MTTF by applying controlled stresses useful to reduce the test
period. Then, QALT uses the lifetime data obtained under stress
conditions to estimate the lifetime distribution of the system in
its normal use condition.
An Accelerated Degradation Test (ADT) is the method used
to analyse (2). ADT extends the QALT technique. Rather than
looking at failure times, ADT uses degradation measures to pro-
duce a time to failure. ADT fits well for aging-related failures
as these are particularly difficult to empirically observe. How-
ever, ADT is designed for physical systems. Hence, this paper
makes use of Matias et al. [22] approach who made ADT ap-
plicable for software aging studies.
Both QALT/ADT make use of a life-stress relationship to
model the connection between failure/degradation times observed
normal use conditions.
Following [20][22], the relationship model adopted in this work
1/ksw Where L is the SUT life characteristic (e.g. the mean
time to failure), s represents the stress level, while k and w are
model-related parameters to be defined from the observed ex-
periments. The particularity of IPL is that scaling s by con-
stant k causes the proportionate scaling of L. A linear relation-
ship exists when both L and s are on a log-log scale: ln(L) =
−ln(k) − wln(s).
Clearly, experiments results have a certain variability which
needs to be considered. This means that a standard IPL is not
enough. Therefore, the IPL is usually integrated with a specific
pdf that allows a definition of confidence intervals. The idea
is to make the pdf mean time to failure parameters dependant
on the stress variable. In this work, a IPL-Weibull distribution
is used. This choice seems reasonable as Weibull, unlike Expo-
nential distribution, has a non-constant Hazard function, which
is appropriate in case of experimental evaluations as systems
get old. The two-parameters Weibull is used, having the follow-
t β t β
ing pdf and CDF: f (t) = βη ( ηt )β−1 e−( η ) ; F(t) = 1 − e−( η ) ,
where β is the shape parameter, i.e., the slope the Weibull CDF;
and η is the scale parameter (or characteristic life). In terms
of reliability, f (t) represents the failure rate function, while
R(t) = 1 − F(t) represents the system availability. The ratio
f (t)
λ(t) = 1−F(t) is the Hazard function, i.e., the instantaneous fail-
ure rate as function of age. β governs the trend of λ(t) which
can increase (β > 1) or decrease (0 < β < 1).
A common approach is to evaluate β and η parameters using a
Maximum-Likelihood Estimation (MLE), which generates those
numbers from the observed failure data set. The scale param- Table 3: ASTS Statistical Data
n
eter, e.g., is calculated for n observations as η = [ T iβ /r]1/β ,
P
Naples Genoa Rome Florence
i=1
where T is the observation time and r represents the number of T rains/day 1150 650 1100 440
Extension(Km) 137 205 250 350
failures.
The IPL-Weibull is derived by setting η = L(s) in the Weibull
pdf defined before, yielding: of messages sent by the IXL to the TMS within a specific time
w β−1 −(ksw t)β window T ; ii) the TCPReplay tool to shoot the recorded traffic
f (t, s) = ks β(ks t)
w
e (1)
against the TMS at S 1, S 2, S 3 rates defined before.
So, using (1), the mean life of the systems in its use condition
can be evaluated: MT T F(s) = 1/ksw Γ(1/β + 1). 7.2. Results Analysis
Once defined the approach and the pdf distribution to use, the 7.2.1. MTTF Evaluation
workload and the stress schemes need to be characterized. Before showing the results obtained it is important to re-
mind that the MTTF analysis wants to observe failures of a
7.1. Workload and Stress Loading Scheme Characterization node caused by aging/non-aging related failures. In particular,
The enforcement of QALT/ADT requires the definition of the analysis here focuses on both functional and non-functional
the accelerating test variable, workload and the stress loading failures. That is, those causing wrong responses from the SUT,
scheme (i.e. stress levels, constant/varying stress). It is im- and those causing a crash or hang of the node.
portant to determine a proper stress variable that could really To automatically verify the functional correctness of TMS op-
expose the system to: functional and crash/hang failures caused erations, an Oracle is used. As already said, the TMS receives
by non-aging-related mandelbugs and those failures triggered UMs from the IXL and then provides the railway status to a
by aging-related errors. Human Machine Interface (HMI). It is here that the functional
The TMS continuously receives Update Messages (UMs) on correctness is checked, by verifying that the HMI has received
the railways occupation status from the underlying IXL. The Responses (R) from the TMS coherent with the IXL UMs. To
data obtained is then sent to other systems (e.g. workstations, this end – during normal execution – a dictionary of correct
databases) through a CORBA message broker. Each UM causes R in a one-to-one relation with corresponding UM is filled by
the highest and diversified number of libc syscall made by the dumping UM and intercepting correct R to the HMI. Then, dur-
TMS monitoring application to the Linux kernel. Furthermore, ing tests, the output of TMS application is compared with the
the higher is the content of UMs (i.e. the number of trains), specific entry in the dictionary in order to establish the func-
the more is the amount of TMS application states visited. This tional correct behavior.
inevitably yields more syscalls and a stronger use of resources The verification of crash or hang failures instead is made through
(e.g. memory). a log analysis conducted on system messages and on CRM logs.
In normal operating regime – following statistical measurements When the Active/Standby nodes do not reply to the heartbeat
provided by ASTS and reported in table 3 – at most 1150trains/day messages, the CRM reports the event on its logs and also records
are in the interested area. Assuming zero trains in night hours the switch operation enforced.
(1-5am), this means a number of 0.95trains/min. The TMS has been subjected to accelerated tests at the iden-
Therefore, the most appropriate choice for the accelerating test tified levels for a period T = 1080Hrs. At the end, any func-
variable (or stress variable) is the rate of UMs sent by the IXL tional failure of the TMS monitoring application has been de-
containing at least 0.95 trains/min. tected. Contrariwise, a total of five hang failures have been
The selection of stress levels represent also a key aspect since found: two for S 1 and S 3, one for S 2. Table 4 shows the es-
using levels out of TMS design limits may generate failures not timates for the IPL-Weibull model parameters realized through
present in the system normal operation. In normal usage, the a Maximum-Likelihood (ML) estimation using the failure data
TMS receives 2msg/s. Based on this, three levels of constant obtained.
stress are applied to the system: S 1 = 35msg/s, S 2 = 25msg/s, Parameter ML Estimate CI-Lower CI-Upper
S 3 = 20msg/s. In fact, it is observed that when the rate of U M k 5.8626E-7 1.0833E-7 3.7533E-6
exceeds 20msg/s, system resources start being considerably w 2.496 1.9776 3.0143
used. In particular, the CPU usage – measured in relation to β 5.3171 2.8095 10.8062
the power of one core with the top tool – exceeds 600%, which
means that more than 6 out of 8 CPU cores are used. Con- Table 4: IPL-Weibull Parameters
temporarily, the overall system memory utilization goes beyond
50%. The identified rate at which the TMS seems to wrongly Using those parameters, the SUT mean life in its use condi-
behave is 45msg/s. For this reason the stress levels chosen are tion (2msg/s) is estimated through the IPL-Weibull. Figure 7a,
enough below the critical level. reports the Life-Stress graph obtained. The green line is gener-
In order to implement automatic tests and reduce testing time, a ated by linearizing the IPL-Weibull, and the blue lines represent
workload generator is needed. This is realized by using: i) the its confidence intervals. The point at which 2msg/s, on the x-
well-known TCPDump tool to capture TCP packets in the flow axis, intersects the green line is the node MTTF estimated for
10

(a) (b)
Figure 7: IPL-Weibull Life-Stress Graphs
the normal condition level. The node failure rate can be now
produced using the MTTF lower bound of the 90%-Confidence
interval as this seems the most conservative choice. In that
case, the result obtained is λl90%
Node = 1/MT T F = 1/74738Hrs =
1.3 × 10−5 .
The empirical failure rate estimate of the TMS server node is
better than the one obtained during the modelling phase (λNode =
6.7 × 10−5 ). Although this might seem strange, it was quite ex-
pected as during the modelling phase pessimistic choices were
made in order to be as conservative as possible. The empirical
node failure rate, given in input to the cluster model (sec. 5.3) –
with no rejuvenation enforced – yields: T HRTMS = 8.73 × 10−7 .
Such a value is below SIL2 upper bound. However, this out-
come lies on the edge of the boundary and therefore needs to
be further improved to obtain a more robust evaluation. Hence,
the rejuvenation strategy is leveraged and the period of rejuve-
nation is empirically estimated.
7.2.2. TTARF Evaluation
Unlike QALT, where failures definition is provided, the ADT
execution requires the description of degradation factors, i.e.,
the system indicator that better represents the possible trend to-
ward aging-related failures (aging indicators). The detection of
aging phenomena is usually performed by looking at the pro-
cess Memory Consumption (MC) [21][22]. This has been mon-
itored through the /proc/meminfo and the valgrind tool wrapped
around the TMS application.
Since the manifestation of software aging phenomena in short
period of time is pretty difficult, this paper followed the ap-
proach of Matias et al. [22]. That is, tuning the memory page
size to influence more the aging intensity. In [22], in fact, a
thorough Design Of Experiment (DOE) activity arrived to the
conclusion that using a page size of ∼ 200KB can impact the
aging trend manifestation.
Afterwards, an additional important choice is the definition of
a threshold (D f ) for the MC, needed for the ADT execution to
establish that an aging degradation has occurred. This highly
depends on the application running. Therefore, an average MC
has been delineated by testing the TMS application. In normal
usage MC ≈ 1500MB. It has been observed that when MC
reaches 2500MB the syscall latency starts growing monotoni-
cally. Hence, setting D f to 2500MB seems a reasonable value.
During the log analysis the MC has been inspected every
60Hrs. Any degradation has been encountered for stress level
11
S 1. On the contrary, using levels S 2-S 3, five aging trends have
been detected in the testing period T = 1080Hrs. Using the ob-
tained degradation data in input to the ADT it is possible to esti-
mate the TTARF. The values for the IPL-Weibull model param-
eters realized through the ML estimation are: k = 0.00005; w =
1.1916; β = 9.2835.
Using those estimates, the TMS time to aging in its use con-
dition*) (2msg/s) is evaluated. Figure 7b, reports then the Life-
Stress graph useful to evaluate the TTARF. The ADT says that
the TMS server node may encounter an aging-related failure af-
ter 8170Hrs of execution. Since the lower bound of the 90%-
Confidence interval is 1710Hrs, the period for the rejuvenation
action to be enforced is set to 1500Hrs which is 3× bigger than
the one estimated during the modelling phase.
This empirical rejuvenation period can be now used in conjunc-
tion with the empirical node failure rate into the PRISM cluster
model. Results reveal that the TMS system reaches a THR value
that is highly SIL2-compliant:
T HRT MS = 1.45 × 10−7 < T HRS IL2 (2)
8. Conclusions
This paper presented a SIL2 assessment strategy conducted
on a commercial SRS. Differently from most of the existing
literature (which does not refer to real industry applications),
we contribute a methodological framework that can be used in
practice as a reference for certifying a wide class of emerging
critical systems, virtually any system for which: i) the general
architecture has already been designed, ii) business constraints
impose that (radical) changes to the architecture be avoided,
and iii) the main COTS components that must be integrated
have already been chosen. We demonstrate, with respect to a
real industrial system (namely: the train management system
by Hitachi Ansaldo STS), that it is possible to achieve SIL2 via
proper configuration of system parameters and set-up of rejuve-
nation procedures.The system is designed as an Active/Standby
cluster based on COTS components. A hybrid approach – i.e.
based on a combination of modeling and experiments – was
taken to estimate the reliability figures of interest. In a first
phase, by means formal model checking, it was proven that the
original configuration of the cluster, used on-field by Ansaldo,
was not compliant to SIL2 requirements. Then, based on the re-
sults of the modeling analysis, a set of mitigation strategies was
proposed, to improve the safety of the system and ultimately
make it SIL2-compliant. The strategies mainly rely on software
rejuvenation, which was proven to provide the best compromise
in terms of trade-offs between costs and reliability improve-
ment. Finally, the effectiveness of the proposed rejuvenation
strategy – as well as of other modelling predictions – was val-
idated by means of an experimental campaign, performed on a
real ASTS test-bed.
References
[1] Simon Connelly, Holger Becht. ”Developing a Methodology for the use
of COTS Operating Systems with Safety-Related software”; Queensland:
Australian System Safety Conference, 2011.
[2] Skramstad, Torbjorn. ”Assessment of Safety Critical Systems with COTS COTS Commercial-Off-The-Shelf
Software and Software of Uncertain Pedigree”; Trondheim.
[3] Pierce, RH. Preliminary Assessment of Linux for Safety Related Systems CRM Cluster Resource Manager
Health & Safety Executive; 2002.
[4] Kiejin Park, Sungsoo Kim; ”Availability analysis and improvement of Ac- CSL Continous Stochastic Logic
tive/Standby cluster systems using software rejuvenation” Journal of Sys-
tems and Software, Volume 61, Issue 2, 15 March 2002, Pages 121-128, CTMC Continous-Time Markov Chains
ISSN 0164-1212, http://dx.doi.org/10.1016/S0164-1212(01)00107-8.
[5] C. Jones, R.E. Bloomfield, P.K.D. Froome, P.G. Bishop; ”Methods for as- DOE Design Of Experiments
sessing the safety integrity of safety-related software of uncertain pedigree
(SOUP)”; Report No: CRR337 HSE Books 2001
[6] W.E. Smith, K.S. Trivedi, L. A. Tomek, J. Ackaret; Availability Analysis
HMI Human Machine Interface
of Blade Server Systems; IBM Systems Journal, 2008.
[7] Bianca Schroeder, Garth A. Gibson. ”Disk Failures in the Real World: ICS Industrial Control System
What Does an MTTF of 1,000,000 hours mean to you” In Proceedings
of the 5th USENIX conference on File and Storage Technologies. IPL Inverse Power Law
[8] Bianca Schroeder, Eduardo Pinheiro, Wolf-Dietrich Weber; ”DRAM Er-
rors in the Wild: a Large-Scale Field Study”; Seattle: SIGMETRIC/Per- IXL Interlocking
formance, 2009.
[9] Edmund B. Nightingale, John R Douceur, Vince Orgovan. ”Cycles, Cells MC Memory Consumption
and Platters: An Empirical Analysis of Hardware Failures on a Million
Consumer PCs”; Eurosys, 2011 ML Maximum-Likelihood
[10] Hoang Pham; ”Handbook of Reliability Engineering”; 2003, Springer;
doi: 10.1007/b97414 MTBF Mean Time Between Failures
[11] Matt Bach; Rates of retired components.
http://www.hardware.fr/articles/944-1/taux-retour-composants-13.html; MTTF Mean Time To Fail
Puget Systems
[12] Luca Montanari; “Online Failure Prediction in Air Traffic Control MTTR Mean Time To Repair
Systems” https://www.dis.uniroma1.it/~dottoratoii/media/
students/documents/tesi_143.pdf
[13] Vilas Sridharan and Dean Liberty. 2012. ”A study of DRAM failures in
NIC Network Interface Card
the field”; In Proceedings of the International Conference on High Per-
formance Computing, Networking, Storage and Analysis (SC ’12). IEEE QALT Quantitative Accelerated Life Tests
Computer Society Press, Los Alamitos, CA, USA, Article 76 , 11 pages.
[14] Xiaohang Jin, E. W. M. Ma, T. W. S. Chow and M. Pecht. ”An in- RTCS Railway Traffic Control System
vestigation into fan reliability” Prognostics and System Health Manage-
ment (PHM), 2012 IEEE Conference on, Beijing, 2012, pp. 1-7. doi: SCS Safety Critical Systems
10.1109/PHM.2012.6228836
[15] ”ITIC Global Server Hardware, Server OS reliability report”. 2014 SIF Safety Instrumented Function
[16] M. Grottke, A. Nikora, K. Trivedi, ”An Empirical Investigation of Fault
Types in Space Mission System Software” in Proc. IEEE/IFIP, Conf. on SIL Safety Integrity Level
Dependable Systems and Networks, 2010, pp. 447-456
[17] Veena B. Mendiratta; ”Reliability analysis of clustered computing sys- SPF Single Point of Failure
tems”; Software Reliability Engineering, 1998. Proceedings. The Ninth
International Symposium on SRS Safety Related Systems
[18] Kwiatkowska M., Norman G., Parker D. (2002) PRISM: Probabilistic
Symbolic Model Checker. In: Field T., Harrison P.G., Bradley J., Harder THR Tolerable Hazard Rate
U. (eds) Computer Performance Evaluation: Modelling Techniques and
Tools. TOOLS 2002. Lecture Notes in Computer Science, vol 2324. TMS Train Management System
[19] B. Eckart, X. Chen, X. He and S. L. Scott, ”Failure Prediction Models
for Proactive Fault Tolerance within Storage Systems,” 2008 IEEE Inter-
national Symposium on Modeling, Analysis and Simulation of Comput-
TTARF Time To Aging Related Failures
ers and Telecommunication Systems, Baltimore, MD, 2008, pp. 1-8. doi:
10.1109/MASCOT.2008.4770560 UM Updates Messages
[20] R. Matias Jr., K. S. Trivedi and P. R. M. Maciel, ”Using Accelerated Life
Tests to Estimate Time to Software Aging Failure,” 2010 IEEE 21st Inter-
national Symposium on Software Reliability Engineering, San Jose, CA,
2010, pp. 211-219. doi: 10.1109/ISSRE.2010.42
[21] Domenico Cotroneo, Roberto Natella, Roberto Pietrantuono, and Ste-
fano Russo. 2014. A survey of software aging and rejuvenation studies. J.
Emerg. Technol. Comput. Syst. 10, 1, Article 8 (January 2014), 34 pages.
DOI=http://dx.doi.org/10.1145/2539117
[22] R. Matias, P. A. Barbetta, K. S. Trivedi and P. J. F. Filho, ”Acceler-
ated Degradation Tests Applied to Software Aging Experiments,” in IEEE
Transactions on Reliability, vol. 59, no. 1, pp. 102-114, March 2010. doi:
10.1109/TR.2009.2034292

Appendix A. Acronyms
ADT Accelerated Degradation Test
ASTS Ansaldo STS
12

View publication stats