Deliverable D6.1 Current Test Condition and Benchmarking Report

X2Rail-1
Action Full Title: Start-up activities for Advanced

Signalling and Automation Systems
Starting date: 01/09/2016
Duration in months: 36
Call identifier: H2020-S2RJU-CFM-2015-01-1
Grant agreement no: 730640
Deliverable D6.1
Current test condition and Benchmarking report
Due date of deliverable Month 06

Actual submission date 31-03-2017
Organization name of lead contractor for this 8 – DB
deliverable
Dissemination level PU
Revision DB-001-30-R1
Deliverable D6.1
Authors
Author(s) Deutsche Bahn AG (DB)

Dr. Fabian Schober
Bernd Eberts
Asociacion Centro Tecnologio CEIT-IK4 (CEIT)
Iñigo Adin
Jaizki Mendizabal
Deutsches Zentrum für Luft- und Raumfahrt eV (DLR)
Oliver Röwer
Mer Mec SPA (MM)
Giulio Mongelli
Siemens AG (SIE)
Birgit Jeschka
Société Nationale des Chemins de fer Français (SNCF-R)
François Fleuret (Systra)
Catherine Maton (Systra)
Trafikverket (TRV)
Laura Mayer
Andreas Westerberg
Contributor(s) Alstom Transport S.A. (ALS)
Fernando Mejia
Ansaldo STS S.p.A. (ASTS)
Luigi Velardi
AŽD Praha SRO (AZD)
Dušan Vokoun
Bombardier Transportation Sweden (BTSE)
Paolo Girolami
Kapsch Carriercom AG (KCC)
Benoît Faup
Network Rail (NR)
Edwin Morton
Société Nationale des Chemins de fer Français (SNCF-R)
Juliette Fournier
Kawtar Hadjadj
Thales Transportation Systems GmbH (TTS)
Lisa-Marleen Scheile
Bettina Dötsch
Deliverable D6.1
1 Executive Summary
The present document constitutes the first issue of the Deliverable D6.1 “Current test
condition and benchmarking report” in the framework of the project titled “Start-up
activities for Advanced Signalling and Automation Systems” (Project Acronym: X2Rail-1;
Grant Agreement No 730640).
The key objective of zero on-site testing, is to perform functional and non-functional
tests (component test, integration test and system test) in laboratory, instead of testing
on-site, in order to save time and costs without compromising on safety.
A status-quo analysis within the railway sector has been performed, by means of a
questionnaire which has been sent to suppliers, infrastructure managers and research
institutes within the railway sector in order to get an overview over today’s testing
practice. Due to the focus on signalling systems, no Railway Undertakings participated
in our questionnaire. In this analysis, areas of improvement regarding testing have been
identified, even though there is no harmonised list of tests that should be shifted from
on-site to laboratory, as the system is too complex. Component tests are mainly done in
laboratories, where the required test environment is comparably easy to build up,
whereas system tests are done on-site as the real environmental conditions are
important, e.g. for acceptance tests, including human behaviour.
Overall, laboratory testing can enhance the quality of the product, improve bug fixing
and speed up development, as it enables parallelisation of different activities and
increases trust on the safety level.
Moreover, a benchmarking with safety-critical industries outside the railway sector has
been performed, in order to compare against lessons learnt for railway system
applications and give them a wider context. Therefore, another questionnaire has been
elaborated and sent out via the project partners. It has been discovered, that there are
big differences between sectors, where testing is a relevant input to standardisation of
approval processes. For various reasons, nearly all sectors perform tests both in
laboratory environment and on-site. Building up a laboratory environment with realistic
input data for the simulations, is quite time and cost consuming, but it allows the test of
worst case scenarios, which may not be possible in a real trackside environment
(without safety impacts) and which is necessary to fully understand the products and
services to be developed. Therefore, quite good test coverage is achieved.
GA 730640 Page 3 of 77
Deliverable D6.1
Table of Contents
1 EXECUTIVE SUMMARY ............................................................................................................................ 3
2 ABBREVIATIONS AND ACRONYMS ....................................................................................................... 6
3 BACKGROUND ........................................................................................................................................... 8
4 OBJECTIVES AND REPORT STRUCTURE ............................................................................................ 9
5 TEST ACTIVITIES WITHIN RAILWAY SECTOR ................................................................................... 10
5.1 CLUSTER OF ANSWERING COMPANIES .................................................................................................................. 10

5.2 DESCRIPTION OF THE FORM DISTRIBUTED ............................................................................................................. 11
5.3 EXPERIENCES OF SHIFTING TESTS TO LABORATORY .................................................................................................. 13
5.4 TEST SCOPE .................................................................................................................................................... 14
5.5 TEST STRATEGY ............................................................................................................................................... 16
5.5.1 In-house vs. external tests ................................................................................................................ 17
5.5.2 Reasons for performing laboratory tests ......................................................................................... 17
5.5.3 Executing on-site tests ..................................................................................................................... 18
5.5.4 Test automation ............................................................................................................................... 18
5.5.5 Effort of tests ................................................................................................................................... 19
5.5.6 Creation of Tests .............................................................................................................................. 19
5.6 REASONS FOR LABORATORY TESTS ....................................................................................................................... 22
5.7 GAIN TRUST AND QUALITY OF TEST ...................................................................................................................... 24
5.8 USE OF FORMAL METHOD VERIFICATION ............................................................................................................... 26
5.9 SHIFT TEST FROM ON-SITE TO LABORATORY ........................................................................................................... 27
5.9.1 Benefits executing laboratory test and limits for shifting ................................................................ 27
5.9.2 Plans to shift test into laboratory .................................................................................................... 28
5.10 IMPLEMENTATION OF AN IMPARTIAL LABORATORY ................................................................................................. 28
5.11 MAIN CONCLUSIONS OF RAILWAY SECTOR ANALYSIS................................................................................................ 29
6 TEST ACTIVITIES OUTSIDE RAILWAY SECTOR (BENCHMARKING) ............................................ 30
6.1 DESCRIPTION OF THE ANALYSIS DONE................................................................................................................... 30

6.1.1 Aim of the benchmarking ................................................................................................................. 30
6.1.2 Description of the form distributed .................................................................................................. 31
6.1.3 Distribution of company types ......................................................................................................... 32
6.1.4 Cluster of the safety-critical industries addressed ........................................................................... 33
6.2 RESULTS OF THE ANALYSIS ................................................................................................................................. 34
6.2.1 Types of tests currently performed .................................................................................................. 34
6.2.2 Tests in laboratory vs. tests on-site .................................................................................................. 36
6.2.3 Laboratory testing............................................................................................................................ 39
6.2.4 Impact of shifting tests from on-site to laboratory .......................................................................... 41
6.2.5 Explanation of the approval process & need for harmonisation by sector ...................................... 42
6.3 MAIN CONCLUSIONS OF THE BENCHMARKING ANALYSIS ........................................................................................... 44
6.4 RECOMMENDATIONS........................................................................................................................................ 45
GA 730640 Page 4 of 77
Deliverable D6.1
7 HARMONISATION AND AUTHORISATION ACTIVITIES .................................................................... 48
7.1 EXISTING HARMONISATION ACTIVITIES ................................................................................................................. 48

7.2 DIFFERENT APPROVAL PROCESSES IN DIFFERENT COUNTRIES ..................................................................................... 51
7.3 CHALLENGES OF EUROPEAN APPROVAL PROCESSES ................................................................................................. 52
7.4 MAIN CONCLUSION OF HARMONISATION AND AUTHORISATION ANALYSIS.................................................................... 55
8 CONCLUSION ........................................................................................................................................... 57
9 REFERENCES ........................................................................................................................................... 59
10 GLOSSARY ............................................................................................................................................... 60
11 APPENDICES ............................................................................................................................................ 64
11.1 QUESTIONNAIRE PART A – STATUS QUO ANALYSES IN RAILWAY SECTOR ...................................................................... 64

11.2 QUESTIONNAIRE PART B – BENCHMARKING WITH SAFETY-CRITICAL INDUSTRIES ........................................................... 73
GA 730640 Page 5 of 77
Deliverable D6.1
2 Abbreviations and Acronyms

Abbreviations / Description
Acronyms
3GPP 3rd Generation Partnership Project, collaboration between
groups of telecommunications associations
4thRP 4th Railway Package
APIS Authorisation for Placing Into Service
AsBo Assessment Bodies
CTC Centralized Traffic Control
CEF Connecting Europe Facility
CENELEC Comitée Européen de Normalisation Électrotechnique
(European Committee for Electrotechnical Standardization)
CER Community of European Railway and Infrastructure
Companies
CSM Common Safety Methods
CCS Control Command and Signalling
DeBo Designated Bodies
DMI Driver Machine Interface
EASA European Aviation Safety Agency
EIRENE European Integrated Railway Radio Enhanced Network
EMC Electromagnetic Compatibility
ERTMS European Rail Traffic Management System
ESA European Space Agency
ETCS European Train Control System
EU European Union
EUAR / ERA European Union Agency for Railways, formerly known as
European Railway Agency
EUG ERTMS Users Group
EURATOM European Atomic Energy Community
EVC European Vital Computer
FAA Federal Aviation Administration (US)
FAT Factory Acceptance Test
GPRS General Packet Radio Service
GSM-R Global System for Mobile Communication Rail
HHT Handheld Terminal
IAEA International Atomic Energy Agency
IEC International Electrotechnical Commission
IEEE Institute of Electrical and Electronics Engineers
IM Infrastructure Manager
IOP Interoperability
IOT Interoperability testing
IP Innovation Programme
IXL Interlocking
KPI Key Performance Indicator
GA 730640 Page 6 of 77
Deliverable D6.1
NoBo Notified Bodies

LEU Lineside Electronic Unit
NSA National Safety Authorities
NVIOT Network Vendors Interoperability Testing
OBU On Board Unit
RBC Radio Block Centre
RU Railway Undertaking
S2R Shift2Rail
SERA Single European Railway Area
SIT System- and Integration Test
STM Specific Transmission Module
SUT System under test
TEN-T Trans-European Transport Network
TMS Traffic Management System
TSI Technical Specifications for Interoperability
UIC Union internationale des chemins de fer (International Union
of Railways)
UNISIG Union Industry of Signalling
WENRA Western European Nuclear Regulators Association
GA 730640 Page 7 of 77
Deliverable D6.1
3 Background
The present document constitutes the first issue of Deliverable D6.1 “Current test
condition and benchmarking report” in the framework of the project titled “Start-up
activities for Advanced Signalling and Automation Systems” (Project Acronym: X2Rail-1;
Grant Agreement No 730640).
Shift2Rail (S2R) is the first joint European rail technology initiative, to seek focused
research and innovation (R&I) and market-driven solutions. This can be achieved by
accelerating the integration of new and advanced technologies into innovative rail
product solutions. Shift2Rail will promote the competitiveness of the European Rail
industry and will meet the changing EU transport needs. The R&I activities are carried
out under the Horizon 2020 initiative and will develop the necessary technology to
complete the Single European Railway Area (SERA). Further information can be found
on http://shift2rail.org/.
The X2Rail-1 project aims to research and develop six selected key technologies to
foster innovations in the field of railway signalling and automation systems. The project
is part of a longer term Shift2Rail IP2 strategy towards a flexible, real-time, intelligent
traffic management and decision support system.
In particular, Work Package 6 (WP6) “Zero on-site Testing” focuses on testing activities
to be standardised within SERA. System- and Integration Test (SIT) is a fundamental
method of system verification across a wide range of industrial sectors. Various
experiences show that the cost and the time consumption for the SIT is from 30 % up to
50% of the project costs and time. Due to the complexity of signalling systems and the
differences between sites, a large number of tests must be carried out on-site, which
takes about 5 to 10 times the effort of similar laboratory tests. Reduction of on-site tests
for signalling systems is hence a reasonable approach to reducing testing costs. WP6
as part of the X2Rail-1 project will make further improvements and the results will be
disseminated on a European level.
This report is the deliverable of Task 6.2 (Assessment of status quo in field testing and
benchmarking) and therefore, the first deliverable of WP6.
GA 730640 Page 8 of 77
Deliverable D6.1
4 Objectives and report structure

Within Task 6.2 “Assessment of status quo in field test and benchmarking”, the current
field test activities have been assessed, in order to identify work packages that could be
shifted to laboratory testing. The expectation, on one hand, has been to identify areas
where lead time and cost of field testing can be reduced, and on the other hand, to
improve further on the quality of delivered solutions. A benchmarking of rail signalling
and telecom activities with other safety critical industries like avionics, medical and
automotive was an important part of this task.
Data about today’s requirements for tests and verification / validation procedures has
been collected taking into account specifications in the European railway networks.
Moreover, harmonisation activities and approval processes of different countries have
been analysed. Testing experiences by assessing the telecom sector has been gained.
Furthermore, a benchmarking with safety-critical industries (automotive, aviation, space,
medical, nuclear) has been performed. Therefore, two different questionnaires have
been answered by WP6 members and by benchmarked industries. The results and
conclusions can be found in this report.
In chapter 5, the test activities within the railway sector are described with focus on test
strategy and reasons for laboratory tests and on-site tests. Moreover, there is a section
concerning shifting tests from on-site to laboratory environment.
Test activities outside the railway sector have been investigated and the results can be
found in chapter 6. Furthermore, some recommendations for improving test activities are
presented.
Harmonisation and authorisation activities within the railway sector are summarized in
chapter 7.
GA 730640 Page 9 of 77
Deliverable D6.1
5 Test activities within railway sector

This chapter consists of a status quo analysis of testing activities within the railway
sector in order to get an overview of common practises and lessons learned in the field
of testing. Therefore, a questionnaire has been provided and distributed among the
members of WP6. The questionnaire can be found in the Appendix 11.1.
Special focus is set on the test strategies and on the possibility to shift tests from on-site
to laboratory in order to reduce lead time and costs for testing and therefore, to enable
an accelerated approval process.
5.1 Cluster of answering companies

18 companies have answered the questionnaire concerning the status quo analysis
within the railway sector. Figure 5.1 shows the distribution of the companies participating
in the questionnaire.
A split of the outcomes between customers and suppliers was made. The focus of the
following analysis is the Control Command and Signalling (CCS) system. Therefore,
customer replies came from Infrastructure Manager (IM) only, and not from Railway
Undertakings (RU).
Validator
6%
Customer
33%
Supplier
50% Impartial Test
Laboratory
5%
Research
center
6%
Figure 5.1 – Split of replies by company type
From section 5.6 on, companies which have identified themselves as validator, have
been classified in the customer category, while companies which identified themselves
GA 730640 Page 10 of 77
Deliverable D6.1
as a research centre or as an impartial test laboratory are included in the supplier

category.
5.2 Description of the form distributed

An electronic benchmarking questionnaire was created and distributed via the X2RAIL-1
WP6 partners. Answers were given anonymously and all questionnaires have been
collected and evaluated confidentially.
The questionnaire starts with general questions about the company and personal
information of the replier:
 Type of your company (e.g. supplier, infrastructure manager, independent
assessor or validator)
 Technical area of your company
 Your personal area of expertise (e.g. product manager, system engineer, test
manager)
The questionnaire continued with topic related questions, starting with ETCS:
1. ETCS test cases have already been specified on a European level (ERA subset-
076 [3]).
1.1. Do you think these test sequences and test cases are sufficient regarding to
interoperability, safety, operational rules etc.?
1.2. If not, where does the need for further test cases come from?
1.3. What are the risks?
1.4. Which problems occur during operation? Which subsystems were affected?
Please explain the unexpected effects on the system.
2. What products/services do you test beside the ones already mentioned in question
1 (subset- 076 [3] ETCS)? Please list the most important tests.
3. Which type of tests do you perform in order to evaluate your product/service?
3.1. Do you perform laboratory environment or on-site/ on real infrastructure / real
environment tests?
3.2. If laboratory environment: Are you performing the tests in an in-house
laboratory or at an external company/institute? If external, please give the
name of the organization.
3.3. If laboratory environment: Why do you perform tests in the laboratory?
3.4. If on-site: real track, test track or temporary track?
3.5. Which of the following test procedure are automated in your company?
3.6. What is the effort of these types of test?
3.7. How are the types of test created?
4. Why do you perform tests in the laboratory, e.g. legal regulations, time saving, cost
saving and please specify the effects?
GA 730640 Page 11 of 77
Deliverable D6.1
5. Do you use on-site testing as fall back, if laboratory testing was not finished in
time?
6. Do you see on-site testing as complementary to laboratory testing, with a specific
interest?
7. If your product is updated, do you repeat the full tests? Do you have a strategy to
avoid that?
8. What are you doing to gain trust in the tests executed in the laboratory?
9. Can you think of any ways to increase the quality of your tests? Which ones?
10. Do you think formal verification techniques can replace testing?
11. If yes, which formal verification techniques may replace testing?
12. Do you recommend shifting any further tests from on-site to laboratory? What is
stopping you from doing these tests in the laboratory today?
13. Which tests do you plan to shift? And what are the necessary test system limits
and borders for your shifted test.
14. Which tests do you think can be replaced by formal verifications?
15. How can you make sure to perform only meaningful laboratory tests, which can
serve as a confirmation of product/service quality and safety level?
16. How do you make sure that at least all necessary meaningful tests are performed
in laboratory?
17. What are the tests, which cannot be shifted and will have to be performed on-site?
The next questions are about the experience and lessons learnt concerning shifting
tests to laboratories:
18. Please provide your experience about shifting tests to laboratory environment.
18.1. What have been the pains or the challenges you had to deal with before
shifting tests to the laboratory environment?
18.2. What system boundaries have to be taken into account?
18.3. What has helped you solve these challenges? What is the effort?
19. How does shifting tests from on-site to laboratory influence company-specific
processes and the organisation of your company (e.g. in the fields of product
management, development process, regulation)?
20. Would you like to participate in the implementation of an independent laboratory
responsible for CCS testing during Shift2Rail (collaboration between different
suppliers in a single laboratory, in remote laboratories, remote testing in distributed
laboratories)?
20.1. If yes: What competences and capacities do you like to yield in such a
laboratory?
At the end, a few questions concerning harmonisation and approval activities have been
asked. This was not the main focus of our status quo analysis, but should give an idea
about the challenges of European approval processes.
GA 730640 Page 12 of 77
Deliverable D6.1
21. Do you participate in any working groups regarding testing in the past or today?
Which one?
22. Is there a further need to harmonise test strategies?
23. Explain the approval process in your home country? Please specify the approval
process.
24. Is there a need for a standardised European approval process in the future? Why?
25. What will be the main advantages of a European approval process?
All the questions are intended to get a deeper understanding of the way testing is
addressed in the different companies, focusing mainly on their evolution from field
testing to laboratory testing.
5.3 Experiences of shifting tests to laboratory

Experiences in the railway industry of shifting any tests to laboratories have been
evaluated by using the answers of the questionnaires. In this section, general
conclusions are provided for shifting various types of tests to laboratory environment.
Shifting tests to laboratory induced a partial update of the test processes (incl. a
guidance for testing and the adaption of general rules on different projects), as well as
the test tools. The testability of the products and the necessary tools have to be
considered during the product development. Moreover, it seemed to be useful to
automate testing in order to enable time and cost savings compared to on-site tests.
The organisation requires a team to manage the test environment, not only to perform
the tests.
Shifting tests to laboratory environment can enhance the quality of the product, improve
the bug fixing and speed up the development, as it enables parallelisation of different
activities and increases trust on the safety level (also known as confidence testing).
While shifting tests to laboratory, the assessment processes have to be created and
established. These are the proofs to regulators, i.e. NoBo (notified bodies), DeBo
(designated bodies) and AsBo (assessment bodies). In this context, more and easier
laboratory certifications are needed. These roles may have to be reconsidered during
the test process development.
The tests, which cannot be shifted from on-site to laboratory, are those depending on
special equipment only available on-site (e.g. an existing interlocking), or tests regarding
the connectivity and the interactions to other equipment. These are directly related to
geographical or environmental factors which are currently impractical to reproduce in a
laboratory environment. On-site tests are the ones related with real time, timing and
dynamic behaviour and interaction, e.g. some GSM-R (Global System for Mobile
Communication Rail) test relying on signal strength, odometry or inside locomotive
GA 730640 Page 13 of 77
Deliverable D6.1
communication. Some of the operational tests are needed for on-site acceptance by the
customer, e.g. EMC tests.
The availability and the costs of appropriate laboratory test equipment are crucial as well.
Taking these factors into consideration, it is important to review the test strategy at the
start of any new project, in order to select a very small set of tests that shall be tested
on-site and plan to shift the remaining ones into the laboratory.
5.4 Test Scope

The survey shows that tests in the rail sector are carried out for different reasons and for
various products. The reasons are always to avoid risks (both safety and
operational/functional). To put system into successful operation, compatibility has to be
guaranteed. There is also the risk, that the tests are not sufficient, because some
operational scenarios are not covered by the generic tests.
Besides testing the conformity of the ETCS on-board equipment (subset 076 [3]), the
main products and services to be tested are shown in the Figure 5.2.
4
number of replies
0
STM
LEU
DMI
IXL
Balises
Field device
GSM-R
RBC
HHT
Train Interface
TMS/CTC
HW redundancy
Positioning system
Figure 5.2 – Products tested beside subset 076 [3]
Some completed and harmonized test specifications are already available. For example,
in ETCS, the so called subset 076 [3], deals with the test specification for compliance
within the systems requirements specification (subset 026 [1]), and thus with the
question of technical interoperability.
The requirements specifications form a good basis for creating test specifications, but
there is a need to specify further tests. Thus, a need for testing also results from the
GA 730640 Page 14 of 77
Deliverable D6.1
requirements of the respective project, for example in the case of operational

procedures. Although railway undertakings (RU) and infrastructure managers (IM) are
working on the harmonisation of operational procedures, there is still the need to define
tests adapted to specific applications, e.g. to project and country-specific operational
procedures.
According to the replies from IM, common mistakes and failures which occurred during
operation are as follows:
 Faults affecting safety or availability are possible during operation.
 Interaction with radio block centres (RBCs) or a different wayside design causes
service brake or failure.
 Incorrect or irrelevant information displayed in the driver machine interface (DMI)
 Loss of communication between equipment
 Problems with timing issues
 RBC roll over (protection)
 Interoperability problems
 Misinterpretation of standards / specifications leading to incorrect implementations
 Issues related to the border of systems
 Problems caused by external influence (e.g. electromagnetic compatibility (EMC)
of trackside devices and trains)
Tests performed follow a variety of test specifications. A large portion is based on
product specific system requirements specifications. For example, in ETCS, the Subset-
076 [3] defines test cases and test sequences to check the compliance of the ETCS on-
board equipment with the System Requirements Specification (subset 026 [1]), and thus
deals with the question of technical interoperability. Moreover, subset 085 (Eurobalise,
[4]), subset 092 (Euroradio, [5]), subset 093 (GSM-R, [6]), subset 110, [7] (in conjunction
with subset 111 [8] and subset 112 [9]) have to be regarded as well.
In addition, there are customer, supplier or project specific tests:
 Operational rules (including human behaviour) and operational scenarios
 Compatibility tests between train and track, besides ETCS subset 076 [3]
 Project specific system requirements specifications
 Diagnostic systems
There are several types of tests (defined in the glossary in chapter 10) which are
executed by most of the partners.
 Functional Tests
 System Tests
 Performance Tests
GA 730640 Page 15 of 77
Deliverable D6.1
 Stability Tests
 Product Tests
 Integration Tests
 Additional testing of the above mentioned in the field of confidence testing
5.5 Test Strategy

The partners were asked to list the different types of tests they perform, when evaluating
their own product or service. This way, 23 different types of tests were identified. Each
type has its own percentage of sharing between laboratory and on-site activities.
Four types, namely acceptance, net access, validation and principle tests, are only
executed on-site or in a real environment. Another five types of test are executed
exclusively in a laboratory environment. They are: component, ETCS subset 076 [3],
factory acceptance, IOT and environmental tests. The remaining types are tested in both
environments, with slight advantage for the laboratory.
Due to the fact that there are certain answers which can be understood to address the
same type of testing (e.g. interoperability, IOT, IOP), a more precise definition of the
different test types will be provided during the test process definition task. Therefore only
the concrete answers from the questionnaire are shown in Figure 5.3, even if they may
address the same topic.
Lab On-Site
24
22
20
18
16
number of replies
14
12
10
0
Product
System
Acceptance
Integration
IOT
Interoperability
Safety
Software
Compatibility
Data
EMC
Functional
Net Access
Component
Environmental
Validation
Hardware
IOP (Subset-110)
Principle
System Interface
Factory Acceptance
Site Acceptance
ETCS Subset-076
Figure 5.3 – Split of laboratory and on-site test execution
For detailed description of the terms and definition, see the glossary in chapter 10.
GA 730640 Page 16 of 77
Deliverable D6.1
5.5.1 In-house vs. external tests

Figure 5.4 shows where laboratory tests are executed. Of those tests executed in the
laboratory, the three types (1) compatibility tests, (2) ETCS subset 076 [3] and (3)
environmental tests, are fully executed at external companies or institutes. In contrast,
component test, data test, functional test, integration test, IOP (subset 110 [7]), IOT,
safety and system tests are fully tested in an in-house laboratory. The remaining types
are mainly tested in-house and only occasionally given to external partners.
Internal External
20
18
16
14
number of replies
12
10
0
Component
Functional
EMC
Principle
Compatibility
Integration
Safety
IOT
Data
Net Access
Software
Product
System Interface
Environmental
ETCS Subset-076
Factory Acceptance
Interoperability
System
Acceptance
Hardware
IOP (Subset-110)
Site Acceptance
Validation
Figure 5.4 – Whether laboratory test are executed in-house or external
5.5.2 Reasons for performing laboratory tests

Regarding the question of why tests are performed in a laboratory environment, the
participants of the survey replied as shown in Figure 5.5. The numbers in the bubbles
indicate the number of answers given for that specific question. This is done in Figure
5.6 up to Figure 5.8, too.
Contractual agreements are the main reason for performing factory acceptance,
hardware tests, interoperability tests, IOP (subset 110 [7]), tests related to safety, site
acceptance tests and software tests in the laboratory. ETCS subset 076 [3], hardware,
IOT, software and type tests are required by the NoBo for certification, while
compatibility, interoperability and IOP (subset 110 [7]) laboratory tests are of course
necessary for inter-system interoperability. Product related activities have a decisive
influence on the laboratory execution of component, hardware, product, safety and
software tests. Last but not least, system-related activities are the main reason for
performing data, functional, integration, IOP (subset 110 [7]) and system interface
laboratory tests.
GA 730640 Page 17 of 77
Deliverable D6.1
Contractually Defined 4 2 4 2 2 1 4 1 2 5 2
Required by Notified Body for

1 1 1 3 4 3 1 1 3 5 2 1
Certification
Necessary for Inter-System

2 1 2 1 5 2 1 2
Interoperability
Necessary for Product-Related

1 5 4 2 6 1 5 1
Activities
Necessary for System-Related

2 9 3 7 1 1 1 2 9
Activities
Factory Acceptance
ETCS Subset-076
Net Access
Acceptance
Principle
IOP (Subset-110)
System Interface
Interoperability
Validation
Functional
Data
Site Acceptance
Hardware
EMC
Product
System
Compatibility
Software
Environmental
Component
IOT
Integration
Safety
number of
replies x
Figure 5.5 – Tests performed in the laboratory environment
5.5.3 Executing on-site tests

Taking a look at the on-site tests, most of them are performed on a real track (see
Figure 5.6). This is especially true for functional, integration, site acceptance and system
tests. There are also some test types, which are executed on dedicated test tracks.
They are exclusively built to perform any kind of on-site tests with real equipment, but
they are not used as operational tracks for railway operation. The test track is less used
than a real track. The third kind of tracks is the temporary one, which is dedicated for
specific tests. They are rarely used, and mostly for functional and integration tests.
Real Track 1 1 1 1 7 1 4 2 1 1 1 5 1 6 1
Test Track 1 1 1 1 3 3 2 1 1 1 1
Temporary Track 1 1
Factory Acceptance
ETCS Subset-076
IOP (Subset-110)
Interoperability
Site Acceptance
System Interface
Validation
Net Access
Integration
Principle
System
Functional
Software
Product
Safety
IOT
Hardware
Environmental
EMC
Component
Acceptance
Data
Compatibility
number of
replies x
Figure 5.6 – Tests performed on-site
5.5.4 Test automation

A main goal of this project is to shift tests from on-site to laboratory environments in
order to simplify and speed up the whole testing process. An effective way of achieving
this goal is through the automation of the test procedures. Currently, the most
automated part of testing among the different types, is the execution phase, as shown in
GA 730640 Page 18 of 77
Deliverable D6.1
Figure 5.7, whereas some test types feature the possibility of automated test case
creation and analysis, the majority has to be handled manually.
Test Case Creation 1 1 1 1 2
Selection 1 1 2
Execution 1 2 7 1 4 1 1 1 2 5 4
Analysis 1 1 1 1 1 2 1
Evaluation 1
Factory Acceptance
ETCS Subset-076
System Interface
IOP (Subset-110)
Site Acceptance
Interoperability
Acceptance
Environmental
Compatibility
Software
IOT
Component
Principle
Net Access
Integration
Safety
Functional
System
Validation
Hardware
EMC
Data
Product
number of
replies x
Figure 5.7 – Some test procedures are automated.
5.5.5 Effort of tests

According to the answers in the questionnaire, the greatest test effort in resources such
as time and money is needed for acceptance, compatibility, data, ETCS subset 076 [3],
IOP (subset 110 [7]), IOT, system and validation tests. The least effort is required by
EMC and principle tests. Figure 5.8 displays the grade of effort needed for different
types of tests, clustered into a scale from 1 “less effort” up to 5 “huge effort”.
Resources for Performing Tests 3 3,5 3 2,3 1 4 2,5 3,1 2,5 3,1 2,8 4 4 4 1 2,9 4 3,4 2,8 3,4 2 3 4
Time Needed for the Tests

4 3 3 4,3 2 5 2,5 3 2,3 3,1 2,4 4 4 3 1 2,6 2 3,2 2,9 3,7 2 2 4
Test Preparation and Execution

3 3 2 4,6 2,5 4 2 3,6 3,3 3,3 2,8 3 4 2 1 3,3 3,5 3 3,5 4 3 3 5
Test Relevance for Service Operation 5 5 2 5 1,5 3 3 3,2 2,5 3,2 3,3 5 4 4 1 2,1 1,5 3,6 3,2 3,8 5 3 4
Factory Acceptance
ETCS Subset-076
System Interface
IOP (Subset-110)
Site Acceptance
Interoperability
Net Access
Hardware
IOT
Integration
Validation
Compatibility
Environmental
Software
Product
System
Principle
Functional
Component
Safety
Acceptance
Data
EMC
number of
replies x
Figure 5.8 – Effort of types of test (1: low effort, 5: huge effort)
5.5.6 Creation of Tests

Asking how the types of tests are created and on which sources they are based on, the
survey participants gave a wide range of answers. But there are some keywords which
GA 730640 Page 19 of 77
Deliverable D6.1
appear repeatedly. The main sources of test creation are requirements specifications of
all types, especially ETCS subset 026 [1], the 3GPP & EIRENE (European Integrated
Railway Radio Enhanced Network) standards, the EMC standard and other national
standards. If already available, a test specification is used, e.g. ETCS subset 076 [3].
Further significant items are operational scenarios, test specifications of the customer
and project specifications. It is also worth mentioning that hands-on experience can be
an important source, too. This can be either tester experience, information about training
modules or even knowledge about already occurred hazardous situations. Functional,
product and system tests are also created using a model based approach, which can
significantly speed up the whole testing process. Table 5.1 lists the sources for the
creation of specific tests. For detailed description of the terms and definition, see the
glossary in chapter 10.
GA 730640 Page 20 of 77
Deliverable D6.1
The creation of tests for each type is based on...

Acceptance Requirements
Compatibility Test specification of the customer
Component Requirements
Data Requirements specifications, track layout
EMC EMC standards
ETCS ETCS subset 076 [3]
Factory Operational scenarios, test specifications, requirements, agreed
Acceptance test plan
Functional Requirements specifications, SSRS, 3GPP & EIRENE standards,
subset 093 [6], Model, agreed test plan, subset 026 [1], project
specifications
Hardware Internal technical specifications, subset 076 [3], Requirements
specifications, System validation
Integration Requirements specifications, interface specifications, operational
procedures and scenarios, requirements, operational rules,
training modules, hazards
Interoperability 3GPP & EIRENE standards, subset 076 [3], trackside supplier,
customer defined test cases
IOP (subset 110) Operational procedures, subset 110 [7], subset 026 [1]
IOT Agreed test plan
Net Access Customer catalogue
Principle National standards
Product Subset 026 [1], requirements specifications, model based
approach, internal technical specification, customer catalogue,
tester experience
Safety National safety requirements, requirements specifications
Site Acceptance System test specifications, EIRENE standards, subset 093 [6],
operational procedures, operational scenarios
Software Requirements, operational scenarios, subset 076 [3], tester
experience, software requirement standards (SWRS), software
accomplishment summary (SWAS), software desing description
(SWDD)
System Subset 026 [1], requirements specifications, internal technical
specification, 3GPP & EIRENE standards, tester experience,
model based approach, project specification
System System requirements specification (subset 026 [1]), subset 076 [3]
Interface
Environmental Sources
Validation Requirements specification, operational test scenarios
Table 5.1 – Sources of tests
GA 730640 Page 21 of 77
Deliverable D6.1
Although, there are lots of sources where tests can be derived from, as shown in Table
5.1, it seems to be challenging to clearly define exit criteria for the whole testing process.
Today they are mainly based on personal judgement or contractual agreements which
make use of lessons learnt from previous test and project experiences.
5.6 Reasons for laboratory tests

It has been found from the replies that more than half of the partners, perform laboratory
tests, the main reason being to save time. Secondly, with 54 %, they want to save costs.
About a third of the partners perform their tests in the laboratory to be able to test
specific situations. Several other reasons were mentioned sporadically by the partners.
Most of the partners (about 80 %) see on-site testing as complementary to laboratory
testing.
In Table 5.2, the reasons for performing test in laboratory are listed. Therefore, answers
from customers and suppliers have been evaluated separately.
Customer Supplier Total
Time saving 56% 67% 63%
Cost saving 44% 60% 54%
Enable specific situations 22% 47% 38%
Early detection of failures 0% 27% 16%
Safety reasons 22% 27% 25%
Flexibility of test environment 0% 20% 13%
Availability of experts and time/effort 0% 20% 13%
to fix errors
Quality reasons 0% 20% 13%
Contractual or legal regulation 22% 13% 17%
Availability of real environment 11% 13% 13%
Proof of fulfilling all 0% 13% 8%
requirements/Assessment
Risk reduction 11% 0% 4%
Integration of different Suppliers 11% 0% 4%
Subsystems
Demonstration of Interoperability 11% 0% 4%
Integration of Hardware and Software 11% 0% 4%
Data Validation 11% 0% 4%
Time flexibility 11% 0% 4%
Table 5.2 – Reasons for testing in laboratory
GA 730640 Page 22 of 77
Deliverable D6.1
Suppliers mentioned laboratory tests more often than customers, see Figure 5.9. At the
same time, the suppliers mainly enjoy the advantages of high availability, safety and
flexibility in the laboratory environment. The customers, on the other hand, use the
laboratory tests to ensure integration of subsystems and interoperability.
Do you see on-site as a complementary to

laboratory testing?
Customer
Supplier
0% 20% 40% 60% 80% 100%
yes no n/a
Figure 5.9 – On-site as a complementary to laboratory testing?
The on-site fall back is only required as a last resort, e.g. if further laboratory tests were
too expensive, but only if the tests do not avoid the execution of already planned on-site
tests. The customers rather tend to on-site fall backs than the suppliers, see Figure 5.10.
Do you use on-site testing as fall back?
Customer
Supplier
0% 20% 40% 60% 80% 100%
yes no n/a
Figure 5.10 – On-site testing as a fall back?
GA 730640 Page 23 of 77
Deliverable D6.1
If a product is updated and a full test has already been made with the previous version,
most of the partners would not repeat the full tests. The detailed answers (in percentage)
are shown in Figure 5.11. Instead, they rely on impact analysis to select the tests to be
repeated. According to five replies, this is done for changes with minor impact, by
choosing a limited subset of the tests to execute, while for a major update complete
regression tests are performed.
If your product is updated, do you repeat the

full tests?
Customer
Supplier
0% 20% 40% 60% 80% 100%
yes no depends on
Figure 5.11 – Handling of updated products, regression tests
A high level of automation of the test environment can impact the decision, by lowering
testing costs and increasing the number of selected tests. On the other hand, increasing
experience and confidence in the testing process, may reduce the amount of selected
tests.
5.7 Gain trust and quality of test

The answers given in the questionnaire show that suppliers check a part of the
customers’ requirements, through tests executed in the laboratory and the success of
the test execution can be obtained only with a high level of trust and quality in the test
activities.
Robust documentation is the basic requirement to increase quality, which means
requirements documents with high level of quality are needed and/or the early
involvement of the supplier in writing requirements is important. Traceability between
requirements (or any other input to define tests) and tests must be well documented and
must be maintained up to date.
GA 730640 Page 24 of 77
Deliverable D6.1
Automatic and configurable simulators are another important aspect, because they can
integrate the highest possible number of real or virtualised elements, such as test
environments (RBC, EVC, etc.). Configurable simulators should be equipped with
additional tools that allow performing tests according to a configurable sequence of
automated commands, resulting in different test scenarios. In this way it is possible to
obtain many benefits, like test reproducibility and multi-session based testing, so that
some kind of faults is found earlier. Simulators increase the ability to compare the result
obtained with the requirements and on the other hand the possibility to perform tests in
degraded conditions.
The use of external laboratories offers an independent view on the system and the
laboratories can provide a standardisation in test procedures and databases.
Laboratories can also make use of and share their knowledge of operational rules used
in different countries, different engineering rules, as well as cross-industry sharing of test
results.
Finally, to really gain trust in test environments, test equipment and its capabilities are a
key goal. The verification of methods, procedures and equipment by audits is very
helpful.
In order to perform only meaningful laboratory tests, good quality requirements and
specifications is needed, as well as using requirements tracing. Furthermore, the
adequacy and the completeness of the tests have to be verified as well.
The implementation of a well-defined test strategy and test plan, including the methods
used is very important. The quality of the deliverables has to be continuously evaluated,
followed by improvement actions. Also the experiences with previous tests have to be
considered.
The evaluation of previous projects improves the quality for following projects, also the
experience with (and the relation to) previous tests (for example issues identified in on-
site tests) should be considered here. FAT or agreements with the stakeholder can also
be used. This gives rise to continuous improvement of the tools and processes.
To make sure that all necessary meaningful tests are performed in laboratory, the tests
are checked against the requirements to cover the specification completely. This is
defined / planned in the test strategy / test plan as well. This gives the required ‘holistic
approach’.
The process, as well as the used methods and the documentation of the different
configurations, including a definition of specific test scenarios for each configuration (i.e.
application specific) is very important.
GA 730640 Page 25 of 77
Deliverable D6.1
5.8 Use of formal method verification

Based on the answers in the questionnaires, formal verification techniques are not
considered as a way to replace testing in railway industries. They can be used just as a
complement of test activity, simplifying some test specifications and reducing test effort
by reducing the number of tests in the laboratory, see Figure 5.12.
Do you think formal verification techniques

can replace testing?
Customer
Supplier
0% 20% 40% 60% 80% 100%
yes partially no n/a
Figure 5.12 – Utility of formal verification
Indeed, even though there are very few companies (both customers and suppliers)
claiming that it is possible to replace tests with formal verification, their real feeling is
focused to reduce total test effort or the number of tests in the laboratory, that is not a
real replacing operation.
Testing and formal verification are complementary activities and formal verification will
not be able to completely replace testing, because some portions of complex signalling
systems can be tested by formal verification; fit for safety validation, not performance.
There is no unified opinion that formal verification is able to cover degraded mode or if it
can only be tested by operation tests on site.
Some replies show that formal verifications can be useful for the first steps of software
test, where generic low level problems must be identified and corrected, such as
memory usage, overflow, divide by zero, out-of-bound array, state flow machine
verification and functional requirements verification. But, from the experiences of recent
projects, formal methods can also be applied on a system level.
GA 730640 Page 26 of 77
Deliverable D6.1
5.9 Shift test from on-site to laboratory

In the railway sector, tests have historically been performed on-site and therefore, on
real tracks. Over the last decades, many of these tests have successfully been shifted to
laboratories. Today, the question arises, if one can recommend shifting any further tests
from on-site to laboratory environment. As one can see from Figure 5.13, it is not clear,
if it would be useful to shift any further tests to laboratories for various reasons which will
be discussed in the next sections.
Do you recommend shifting any further tests

from on-site to lab?
Customer
Supplier
0% 20% 40% 60% 80% 100%
yes no n/a
Figure 5.13 – Recommendation of shifting further tests to laboratory environment
5.9.1 Benefits executing laboratory test and limits for shifting

The opportunity to shift test(s) in the laboratory put more focus on the current
laboratories to increase their capability and become able to reproduce scenarios as
realistically as possible. One option to do this, is to set up impartial laboratories, able to
perform compatibility / integration tests with different on-board units and track-side
equipment that ensure neutral view on the results. Interface between laboratories should
be fully specified, in order to allow easy connection of external equipment (located in the
supplier facilities).
Some replies stated that the impartial laboratory should make agreements with supplier
and IMs or RUs in order to have access to railway data and equipment, and become
able to reproduce operational scenario with specific railway rules.
Current limitations to shift tests to laboratories are related to the environment simulation,
which must be as accurate as possible in order to avoid a large volume of re-testing
when the system is installed in the real environment.
GA 730640 Page 27 of 77
Deliverable D6.1
Some companies provided answers about having additional problems related to the lack
of confidence in ETCS rules and data in different countries (i.e. balises, RBC data and
behaviour). So from their point of view, it is much better to perform the test on-site
instead of trying to reproduce the infrastructure and installation in the laboratory.
Further investigations on the limits and the benefits for shifting to laboratory will be
performed in the test process definition.
5.9.2 Plans to shift test into laboratory

Plans to shift tests in laboratory are mainly focused on specific test activities:
 Integration tests: In terms of integration between different subsystems (provided by
same or different suppliers), protocols, GPS, preliminary system tests
 Functional tests: that means functional and operational test cases to prove the
complete compatibility for railway operation
 Negative tests: In terms of abnormal end-to end delay, errors on protocol level, fault
injections
 GSM-R or future technology for signals characterisation: In order to reproduce
on-site signals plus connection error ratio, connection loss, radio holes, network
registration delay
Note: GSM-R end of life is planned to be in 2030, new technologies (4G/5G RAT)
introduction will start in 2022. Consideration on future test platform should take into
consideration that evolution.
In any case, the choice of laboratory or site should be taken by the project depending on
several aspects including availability and cost of the laboratory and site.
Site acceptance tests can also be reduced with:
 Standardized interfaces
 Railway line data availability
 Agreements between different supplier aiming to prove compatibility between their

equipment
5.10 Implementation of an impartial laboratory

10 Suppliers and 4 customers are interested in implementation of an impartial laboratory
responsible for Control Command & Signalling (CCS) testing during Shift2Rail. They
bring in a lot of competences like test architecture and laboratory design, experience
with test tools and processes, simulation software development, experience in
GA 730640 Page 28 of 77
Deliverable D6.1
automated testing, IOP testing, GSM/GSM-R and GPRS testing, railway operation and
operational planning, staff as laboratory and field tester, track designer, wired and
wireless saboteurs for positioning and communication systems.
5.11 Main conclusions of railway sector analysis

In general, the greatest challenge while shifting tests to laboratory is the interaction of
the different products, to create an environment close to the reality. The complexity of
the system is especially important.
Rework or update of the test environment may be necessary.
Building the laboratory test environment is very expensive and time consuming. Much
effort has to be spent for creating a strategy and for buying or developing a test
environment with various tools.
Another challenge is to add as much trust into the laboratory as possible (e.g. by testing
in impartial laboratories), and therefore in the laboratory tests. It seems to be
challenging to clearly define exit criteria for testing.
System boundaries, especially the (external) interfaces and hardware as well as the
GSM-R and network communication have to be considered thoroughly. Moreover, the
human factor is very important in a dynamic and complex environment.
Mixed teams with different experts together with a tool harmonisation (new software
tools, analysis tools, development of a test bench) are beneficial while shifting tests to
the laboratory. An often used procedure is “record and replay” of real on-site scenarios
and data.
Nevertheless also tests according to subset 111 [8] and the rent of test capacity from the
suppliers are helpful.
The analysis raises the question of where accountabilities and liabilities will reside
during the transfer of testing to the laboratory. This will be discussed and confirmed
during the development of the test process.
GA 730640 Page 29 of 77
Deliverable D6.1
6 Test activities outside railway sector (benchmarking)

In this chapter, our benchmarking process with industries providing solutions for safety-
critical products or services will be described. Some of them have already shifted tests
to laboratory environment and may provide useful lessons learnt to our railway sector.
Moreover, approval processes may differ significantly with respect to harmonisation
across Europe (or the world).
Therefore, a questionnaire has been prepared and distributed among different industries
outside the railway sector. The questionnaire can be found in Appendix 11.2. Most of the
questions asked to aviation, space, nuclear, medical and other sectors are quite the
same as asked for in the railway sector part. The answers are discussed below and
grouped into different fields.
6.1 Description of the analysis done

This section includes the context in which the benchmarking activities have been carried
out. First, the aim of the benchmarking is explained, then, the questionnaire distributed
is described, after that, the timeframe and number of answers received is presented and
finally a clustering of the safety critical industries is shown.
6.1.1 Aim of the benchmarking

The concept of shifting tests from on-site into laboratories using simulation is not railway
specific. This process has already been started in several industries and may be
extended furthermore. Since ideas from other safety critical industries might be applied
to the railway sector, first of all as much information as possible about the current field
test and laboratory test activities in other industries have to be gathered with the aim of
identifying areas which can help to the goal of the zero on-site testing.
The major objectives of this benchmarking are:
• Identifying the types of test performed in laboratory and on-site
• Getting information about the reasons to perform the tests in laboratory and on-site
• Gathering experiences made by shifting tests
As stated, the focus has been set on safety critical industries which are comparable to
the railway sector and especially to the command and control systems within the railway
sector. Knowing that information about these internal processes is difficult to collect, the
broad variety of X2Rail-1 partners has been used. Every partner used his own network,
to get in contact with the relevant persons in other industries. The major advantage of
increasing the return rate by using this way of distributing the questionnaire came along
GA 730640 Page 30 of 77
Deliverable D6.1
with the disadvantage of not having a complete list of companies addressed and replied
to the questionnaire, due to confidentiality. Therefore, getting in contact with the people
who answered to the questionnaire is not possible easily.
6.1.2 Description of the form distributed

An electronic benchmarking questionnaire was created and distributed via the X2RAIL-1
WP6 partners. Answers were given anonymous and all questionnaires have been
collected and evaluated confidentially.
The questionnaire starts with general questions about company and personal
information of the replier:
 Type of your company (e.g. supplier, infrastructure manager, independent
assessor or validator)
 Technical area of your company
 Your personal area of expertise (e.g. product manager, system engineer, test
manager)
The questionnaire proceeds by topic related questions:

1. What tests do you perform in the laboratory and what tests in the real
environment?
2. Why do you perform tests in the laboratory, e.g. legal regulations, time saving, cost
saving and please specify the effects?
3. Please provide your experience about shifting tests from on-site to laboratory
environment.
3.1. What have been the pains or the challenges you had to deal with before
shifting tests to the laboratory environment?
3.2. What system boundaries have to be taken into account?
3.3. What has helped you solve these challenges? What is the effort?
4. How does shifting tests from on-site to laboratory influence company-specific
processes and the organisation of your company (e.g. in the fields of product
management, development process, regulation)?
5. How has or will shifting tests from on-site to laboratory influence certification and/or
approval processes and documents?
6. How can you make sure to perform only meaningful laboratory tests, which can
serve as a confirmation of product/service quality and safety level?
7. What are the tests, which cannot be shifted and will have to be performed on-site?
Why?
8. What are you doing to gain trust in the tests executed in the laboratory (in contrast
to on-site tests)?
GA 730640 Page 31 of 77
Deliverable D6.1
All the questions intend to achieve a deeper understanding of the way of testing of the
companies addressed, focusing mainly in their evolution from field testing to laboratory
testing.
6.1.3 Distribution of company types

The questionnaire was distributed in the end of November 2016. Deadline for receiving
answers was the in the middle of January 2017.
We have received 22 replies from 15 different companies. Some companies have sent
answers from different internal departments or sectors. Most of the replies came from
countries inside the European Union. However, there was also a reply from North
America.
Figure 6.1 shows the distribution of answers related to the company types. A broad
spectrum of industries into this type of companies can be identified and the replies are
randomly distributed among these industries:
• Supplier:
• Regulator
• Data & Service Provide
• Consultancy
• Research Institute
• Manufacturer
Manufacturer Supplier
18% 27%
Research Regulator
institute 5%
27%
Consultancy Data & service

14% provider
9%
Figure 6.1 – Split of replies divided into company types
GA 730640 Page 32 of 77
Deliverable D6.1
6.1.4 Cluster of the safety-critical industries addressed

The analysis of the replies in terms of areas of industry shows that the number of replies
from medical or defence sectors is very small, see Figure 6.2. Although the
questionnaire has been distributed to a large number of companies from these areas,
the feedback was quite small. One reason for that is the fact that benchmarking and the
questionnaire is dealing with highly critical issues for these industries and therefore,
these sectors have by nature high restrictions to reply to such questionnaire and to give
information to the public or other industries or companies.
Medical devices Defense

5% 5%
Automotive &
truck
manufacturing
9% Space
Wireless 27%
communication
& navigation
9%
Multimodal
traffic Aviation
management 18%
9%
System Nuclear
engineering & 9%
validation
9%
Figure 6.2 – Area of safety critical industries
Note: The answers coming from the communication industry are analysed in part A, as
they are part of the railway related industry.
Due to the distribution procedure and the fact that more or less all members of X2RAIL-
1 WP6 are from the transportation sector, the number of replies from that sector is very
high, as can be seen in Figure 6.3. Other sectors like the nuclear, the medical devices,
defence or consultants for system engineering also participated, but only in a very
limited way.
GA 730640 Page 33 of 77
Deliverable D6.1
Medical Defense
Devices 5%
System 4%
Engineering
9%
Nuclear
9%
Transpor-
tation
73%
Figure 6.3 – Sector of industries
6.2 Results of the analysis

All the information collected has been analysed with the aim of obtaining conclusions
towards the objective of the zero on-site testing. The analysis shows that in some topics,
there is broad agreement among different sectors, and not such an agreement in some
other topics.
The information is arranged as follows:
• Types of tests currently performed
• Tests in laboratory vs. tests on-site
• Laboratory testing
• Impact of shifting tests from on-site to laboratory
• Explanation of the approval process & need for harmonisation by sector
• Main conclusions of the analysis
6.2.1 Types of tests currently performed

The analysis of the data collected shows that all the companies are performing
laboratory tests. However, not all of them are performing tests in a real environment.
This is due to the fact that sometimes, there is no accessibility to the real environment,
e.g. components deployed in the space or in the satellite communications industry.
6.2.1.1 Types of tests performed in laboratory

The laboratory tests can be split into three different types of test: subsystem tests,
integration test and system validation tests.
GA 730640 Page 34 of 77
Deliverable D6.1
These types of tests are inter-dependent and follow a specific sequence. First the
subsystem tests are performed. Once the subsystems have been validated satisfactorily,
integration test(s) can be performed. Finally, system validation tests are carried out
when integration(s) test have succeeded. Therefore, laboratory tests are not limited to
subsystem tests, but also integration and/or even system tests may be performed in
laboratory.
Beside functional validation tests, laboratory tests are also used to perform non-
functional tests like safety, security and qualification tests (such as electromagnetic
compatibility, temperature tests…).
Figure 6.4 shows that most tests performed in laboratory are subsystem tests, which are
needed before the integration tests. Therefore, the number of integration tests is greater
than the system tests, which require the completion of the other types of tests.
System
20%
Subsystem
50%
Integration
30%
Figure 6.4 – Split of types of test performed in the laboratory
6.2.1.2 Types of tests performed on-site

In contrast to laboratory testing where three types of tests could be identified, on-site
tests cover also other types of test, but the focus is mainly put on system validation test.
Moreover, no subsystem tests are performed on site as it can be seen in the Figure 6.5.
Beside functional systems validation tests and integration tests, non-functional tests are
also performed on-site.
Moreover, the vast majority of the answers state that on-site tests are carried out for
activities that could not be reproduced in laboratory tests. This does not include
acceptance tests which can be considered as additional tests that might be done on-site
due to the customer request.
GA 730640 Page 35 of 77
Deliverable D6.1
Acceptance Subsystem
5% 0%
Non-functional
10% Integration
9%
System
76%
Figure 6.5 – Split of types of test performed on-site
6.2.2 Tests in laboratory vs. tests on-site

Laboratory tests have a number of benefits. First, laboratory testing is safer for human,
material and environment than the equivalent on-site testing. Laboratory testing can be
performed in early stages of the product lifecycle, when failures and features can be
identified very early and therefore, costs and time can be saved. Since the laboratory is
a controlled environment in opposite to on-site environment, it is possible to test worst-
case scenarios and stress-test scenarios. Moreover, the repeatability of the test is
guaranteed. Furthermore, tests can be corrected and modified more easily (e.g. for
regression tests) and various scenarios can be performed within a short time, both
leading to shorter overall testing time. Finally, the reduction of testing time also comes
from the higher availability of laboratory for testing compared to on-site areas, i.e. on-
site test areas might be restricted or not usable due to logistics or environmental
conditions.
Considering all these advantages, there is a number of reasons that makes the
companies asked perform laboratory test as shown in Figure 6.6. These reasons can be
classified into three groups. The main reason is the cost and time savings with a 36 % of
the answers. 27 % of the answers expose that the laboratory tests are contractually
defined or required by the authorities. Finally, the functional reasons such as
interoperability, product related and worst-case scenarios (safety and security,
reproducibility) cover 37%.
GA 730640 Page 36 of 77
Deliverable D6.1
Product
related tests Contractually
10% defined
11%
Time savings Required by

17% notified body
16%
Safety and
security
Cost savings 5%
Reproducibility
19% 9%
Interoperability
13%
Figure 6.6 – Distribution of the reasons for performing laboratory tests
6.2.2.1 Challenges of shifting tests from on-site to laboratory

Shifting from on-site tests to laboratory tests has a number of challenges to be fulfilled.
These challenges can be grouped into three main topics: creating a laboratory
simulation/test environment, technical issues and gaining trust in laboratory tests.
A number of difficulties are found at the time of creating a laboratory simulation/test
environment. Usually they require an expensive and long-term development of
simulation systems and laboratory test setup environment. Sometimes, it is required to
add specific operational modes to the simulation environment with regard to the
company or country needs. In addition, sometimes, final test cannot even be performed
in the laboratory (i.e. medical tests).
Technical issues might not allow laboratory testing due to technical limits of the
laboratory itself. In case these limits can be overcome, there is the need to gain
experience about the real environment conditions first, before transferring it to a
laboratory environment. For performing the laboratory tests, creating standardised
testing interfaces is very useful for the target sector where different suppliers and
different end-users can employ the same tests and test setup, and external impartial
laboratories can also take part in the process.
GA 730640 Page 37 of 77
Deliverable D6.1
Finally, in order to gain trust in laboratory tests, a “change in mindset” is required to rely
on laboratory tests and their outcomes Moreover, there is a need for the acceptance of
laboratory test results for safety and security.
6.2.2.2 What has helped to solve those challenges?

With the aim of solving these challenges, a look at the experience and lessons learnt
can be taken. From there, a number of actions to be applied have been identified. For
example use simulation tools, make the information available internally and from other
actors, and define the benefits of a laboratory testing.
Currently there are a number of new simulation tools available and easy to adapt to
many needs. Therefore, it is challenging to find the right tool for a specific application.
For this, a review of current industry practices and the definition of the test plan together
with the customer are really helpful. The contribution of the customer is the key, mainly
due to two aspects: the synergy created between the supplier and the customer and the
need of the on-site test data, which the customer has access to, with the aim of
comparison with the laboratory test data. Moreover, not only the involvement of the
customer is needed but also the involvement of employees with relevant experience that
will lead to the definition of the test plans and procedures. This should include the
demonstration of the coverage of as much hazards as possible showing, if needed,
complete safety and security tests. Finally, external information coming from the
exchange with regulators and experts (e.g. universities) to obtain good models, and the
standardisation of tests is also a good practice towards overcoming the limitation of the
laboratory testing.
Among all these actions, the validation of the test site should be highlighted. For that, it
is necessary to compare laboratory data with on-site data to ensure that the
environment reproduced in laboratory matches with the on-site and the behaviour of the
system under test (SUT) in laboratory matches with the behaviour on-site.
6.2.2.3 Boundaries of shifting to laboratory

In the process of shifting on-site tests to laboratory test a number of boundaries are
found. These boundaries have to be overcome to improve laboratory testing with the
aim of reducing on-site testing.
It has been stated that the real environment is the main boundary, since reproducing
that environment in the laboratory is not obvious. The data about real environment is
difficult to collect and it is not foreseen that it will ever be completely available.
Additionally, costs, technology and organisational limits, where different responsibilities
of factory and customer and environments (on-site) are available makes it difficult to
GA 730640 Page 38 of 77
Deliverable D6.1
develop the laboratory testing site. Finally, human behaviour affecting the system needs
to be simulated as well; therefore, this effect should be also included in the laboratory
test site.
6.2.3 Laboratory testing

With the aim of learning how to identify the laboratory test to be done, this subsection
describes the way other sector works to ensure only the use of meaningful tests in
laboratory, the identification of tests that cannot be shifted to the laboratory and the way
to gain trust on those laboratory tests.
6.2.3.1 How to ensure to perform only meaningful tests in the

laboratory
With the aim of overcoming the drawback of performing non-useful test in the laboratory,
the replies of the companies that answered the questionnaire varied. However, there
was one common answer related to rely on the experience and lessons learnt during the
years in the sectors the companies are targeting.
In order to complement the experience or to fill the lack of experience, a number of
specific actions were also mentioned. In order to find the right balance between
requirements and over specifications a clean traceability is proposed. Moreover,
definition of the test and the test coverage is also a key together with the essential
preparation work. Having a product risk analysis is also useful for performing only
meaningful tests. When possible, standardised tests, which have already been validated
by the regulator, should be employed. Additionally, to ensure that meaningful test are
performed, only standard tests, detailed in a specific European subset and performed in
external and qualified laboratories can be performed. Finally, adding realistic data to the
laboratory test, which were recorded under real life conditions, also contributes to the
quality of the laboratory tests.
6.2.3.2 Which test cannot be shifted into laboratory

There are many challenges and boundaries when shifting from on-site testing to
laboratory testing. Even though challenges can be fulfilled, there are boundaries that
cannot be overcome and therefore some tests cannot be shifted to the laboratory.
First, the tests are split into seven different types:
• Non-reproducible tests: Tests with no possibility to measure, simulate or reproduce
real environment data (e.g. tests with real mobility effects, …)
• Specific performance tests: Some specific performance tests have a significant
meaning if performed on-site only.
GA 730640 Page 39 of 77
Deliverable D6.1
• Installation tests: Installation test can only be performed once the installation is on-
site.
• Field acceptance tests: Some tests, which have already been performed in the
laboratory, have to be repeated on-site.
• Area specific tests: These kind of tests apply to the space and nuclear sector and
are related for example to interaction of rocket and start base, interaction of cold &
warm reactor, etc.
• Specific on-site tests: Tests required by authorities, standards and/or customers
(e.g. in the avionics and medical areas: “no one will accept an aircraft which is not
flight-tested”).
• Human behaviour tests: Tests including human behaviours are difficult to perform
in the laboratory properly. This is usually the case in the automotive industry.
Second, these seven types of tests are arranged into two categories of reasons for
laboratory testing: functional and requested. These reasons are linked to the main
boundaries for laboratory testing, namely creating a simulation environment and gaining
trust in tests, see Table 6.1.
GA 730640 Page 40 of 77
Deliverable D6.1
Boundaries for laboratory testing

Creating a simulation Gaining trust in laboratory
environment (technical issues): tests:
Non-reproducible tests
Specific performance tests Specific performance tests
Installation tests
Reasons for laboratory testing
Field acceptance tests

Functional
Area specific tests Area specific tests
Specific on-site tests
Human behaviour tests
Specific performance tests Specific performance tests
Field acceptance tests Field acceptance tests

Requested Area specific tests Area specific tests
Specific on-site tests Specific on-site tests
Table 6.1 – Reasons and boundaries for laboratory testing.
6.2.3.3 How to gain trusts in tests executed in laboratory?

A number of different activities performed by different industries have been collected
with the aim of gaining trust in laboratory testing. All these help to demonstrate that the
proposed laboratory tests are reliable and significant in comparison to the on-site tests.
Quantitative KPIs for each test are employed, and when possible, those KPIs agreed
upon with the customer. Customers and/or authorities should also participate in certain
tests. External audits resulting in certification are also performed. Moreover, test tools
which are qualified against applicable codes and standards are employed. The
laboratory test is validated performing the same test on-site and comparing the results.
Besides, systematic and transparent execution of tests and documentation including
traceability is considered. Finally, the reproduction is a key factor of the laboratory tests.
6.2.4 Impact of shifting tests from on-site to laboratory

This subsection describes the impact that laboratory testing had in the company-internal
process and on the certification or approval of the different sector analysed.
GA 730640 Page 41 of 77
Deliverable D6.1
6.2.4.1 Impacts on company processes

Shifting on-site tests to laboratory tests can affect the company processes. In this
context, the experience of the companies that replied to the questionnaire has been
collected. The most important topics addressed reflect the benefits of the laboratory
testing.
One of the most important points is that internal processes need to reflect importance of
testing and the processes have to be completed around laboratory testing, including
criteria, people, etc. Moreover, benefits of including the laboratory testing in the internal
processes reflect that investments in upgraded test equipment should be done.
The reduction or increase of the complexity of the internal processes due to the
introduction of laboratory testing differs significantly across the responses. Some
companies mentioned that technical choices are simplified, while others, especially
automotive, inform that laboratory testing increases the complexity of the processes and
requires additional competencies of the staff. Nevertheless, there is an agreement on
the reduction of logistics, management and communication processes due to laboratory
testing.
6.2.4.2 Impacts on certification or approval

Nowadays, laboratory testing is part of the process for certification or approval. Usually it
precedes the next process of the certification or approval, as in the case of the aviation
sector, where laboratory testing is a prerequisite for flight-testing permissions.
Even though most approval processes are based on laboratory tests today, the trend
might change. In the case of the automotive sector, the trend goes rather in the opposite
direction by shifting from laboratory tests to on-site tests for certification (e.g.
consequences of diesel emission affair).
It also happens that customers and regulators assess the whole development process
(planning, requirements, basic design, detail design, manufacturing, and all testing
levels) and certification is planned together with authorities (ARP4754 – Aerospace
Recommended Practice) as in the case of nuclear and space sector.
6.2.5 Explanation of the approval process & need for harmonisation

by sector
There are two general comments that apply to most of the sectors studied. Currently, in
most areas standardized processes for approval are established that pave the way
towards putting the product into the market. In case that European standards do not
exist, there is a need for harmonisation in all areas.
GA 730640 Page 42 of 77
Deliverable D6.1
Specific description of the processes for each area studied is described in Table 6.2.
The main conclusions are that the aviation is an area with a large number of approved
standards and regulations and it is a safety critical area. Therefore the principles in the
aviation area can be a useful input for the railway sector. Moreover, part of the
communication (GSM-R) is already included in the railway sector.
Area Explanation of the approval process
Wireless  Already worldwide standardized by 3GPP, IEEE, ETSI
Communication  The Third Generation Partnership Project (3GPPTM) was
established in 1998 to develop specifications for advanced
mobile communications. It comprises:
o seven regional Standards Development Organizations
(SDOs) including ETSI
o market associations
o several hundred companies
 The original scope of 3GPP was to produce globally
applicable reports and specifications for a third generation
mobile system based on evolved Global System for Mobile
communication (GSM) core networks and the radio access
technologies that they support.
Defence  Each country and each customer have different requirements.
So it is not easy to harmonise and standardise the approval
process. Although the System engineering management is
quite similar.
Nuclear  At European or international level, there are quite a lot of
regulation bodies dealing with regulation and control, e.g.:
o IAEA: International Atomic Energy Agency
o WENRA: Western European Nuclear Regulators
Association
o EURATOM: European Atomic Energy Community
 Standards and guidelines are provided in IEEE and IEC
 Significant differences among countries
 Country specific regulations, such as safety aspects,
environmental conditions
Aviation  Aircraft need world-wide flight permission.
 To reduce certification effort for safety-critical systems, equal
standards are necessary and do exist.
o Certification process is specified by EASA (European
Aviation Safety Agency (EU)) and FAA (Federal
GA 730640 Page 43 of 77
Deliverable D6.1
Aviation Administration (US).

o Other countries have a bilateral agreements referring
to these processes.
 In the EU area, identical regulations apply to all countries (for
example, EU Regulation No. 748/2012 for development and
production, EU Regulation No. 1321/2014 for maintenance),
which are integrated into all country-specific legal areas of the
EU member states
 For drones:
o There is a need for harmonisation.
o Currently there is no harmonisation in the drone
industry. Each country has its own legislation.
o EASA and FAA are now working in harmonisation to
give guidelines for construction and operation.
Space  There are differences between national projects.
 In the projects contracted by the European Space Agency
(ESA) the approval process is mainly harmonised.
 European harmonisation would simplify the administrative
and contractual work.
Multimodal  Less standardised area
traffic  Traffic Management Systems (TMS) are different in every
management country and have country specific solutions.
 Standardisation and harmonisation initiatives are useful.
Medical  No answer given to that question.
System  Not enough data given to that question.
engineering
and validation
Table 6.2 – Statement of the standardisation level of the approval process per
sector studied
6.3 Main conclusions of the benchmarking analysis

The main conclusion obtained from the analysis that will help towards the objectives of
the zero on-site testing are the following:
 The questionnaire is mainly related to safety, not to security.
 The questionnaire shows that next to the railway sector, other safety critical
industries exist.
GA 730640 Page 44 of 77
Deliverable D6.1
 These industries and companies have to deal with the same issues as the railway
sector does.
 Replies from many companies and departments have been received.
 Although attempt, very limited answers from additional sectors like chemical,
pharma and medical were given. But due to the reasons already mentioned in
section 6.1.4, no further answers have been gathered.
 Almost all companies of all areas perform both test: laboratory and on-site.
 Taking functional tests into account (subsystem, integration and system tests), all
subsystem tests are performed in labs and a vast majority of system tests are
performed on-site.
 For obvious reasons there are tests, which cannot be shifted into laboratory (e.g.
customer acceptance, installation, simulation of human behaviour).
 No industry has shown the capacity and capability to perform all relevant tests in
laboratory. On-site tests are still required.
 Automotive is showing that some tests currently performed in laboratory will be
shifted back to on-site in the future (Diesel emission affair).
 Performance tests take place in laboratory as well as on-site, but only up to a
certain degree in labs as they have to be completed on-site for certain boundaries.
 The more the companies know about the on-site tests, the more data they can
provide as an input for laboratory environment in order to gain trust in the
laboratory test and shift tests from on-site to laboratory as much as possible.
6.4 Recommendations
This section includes a list of recommendations to be applied in the test process and the
test architecture for railway systems coming from the experiences provided by other
industries, see Table 6.3.
Recommendation no. 1
Tests shall be carried out in a safe manner with regard to human, material and
environment.
Different level of test should be defined such as subsystem tests, integration test and
system validation tests. Moreover, non-functional tests should also be included.
Laboratory tests should start in the early stages of the product lifecycle, when failures
and features can be identified very early and therefore, costs and time can be saved.
Thus, these tests should allow testing a number of scenarios within a short time,
leading to shorter overall testing time. Laboratory tests should be planned in such a
way that logistics, management and communication processes should be reduced
compared to on-site testing.
GA 730640 Page 45 of 77
Deliverable D6.1
When required, customers and regulators should be included in the assessment of the
whole development process (planning, requirements, basic design, detailed design,
manufacturing, and all testing levels) including certification and laboratory testing
activities.
Acceptance tests can be considered as additional tests and is requested by the
customer to be done on-site and not to be moved to the laboratory. Evidence gathered
through laboratory testing may, however, contribute to acceptance tests.
Laboratory tests should include worst-case scenarios and stress-test scenarios.
Repeatability of laboratory tests should be guaranteed. For that, the right tools should
be found among those tools available. For example, the test tools which are qualified
against applicable codes and standards should be used. One of the key aspects for
this selection is the flexibility and easy way to adapt it to many needs.
Laboratory test should allow being easily corrected and/or modified allowing e.g.
regression tests. Furthermore, they should be flexible with the aim of allowing an easy
addition of specific operational modes required by the customer.
Standardised testing interfaces should be created with the aim of allowing the use of
the same tests and test setup independent of the supplier and end-user. Moreover, it
allows external impartial laboratories taking part in the process.
Laboratory test should enable a “change in mindset” with the aim of gaining trust in
laboratory tests to rely on laboratory tests and their outcomes. Experience about the
real environment conditions should be gained first, before transferring the environment
to the laboratory.
At the end, the validation of the test site should be done by means of comparing
laboratory data with on-site data, to ensure that the environment reproduced in
laboratory matches with the on-site and the behaviour of the system under test (SUT)
in laboratory matches with the behaviour on-site. Realistic data, recorded under real
life conditions, should be added to the laboratory tests to increase the quality of the
tests. External audits of the laboratory and laboratory test resulting in certifications
should be done.
Laboratory tests should also demonstrate that its results can contribute to the
acceptance of safety (and security). For that, the test plans/procedure should include
the demonstration of the coverage of as much hazards as possible showing, if
GA 730640 Page 46 of 77
Deliverable D6.1
needed, a complete safety and security test. The definition of the test and the test
coverage together with the essential preparation work should be considered for the
laboratory testing. In order to find the right balance between requirements and over-
specifications, a clean traceability should be applied. When testing a product in the
laboratory, a Product Risk Analysis should be included. Laboratory tests should
include systematic and transparent execution and documentation including
traceability.
When possible, standard tests, which have already been accepted by governmental
bodies, should be employed. One of the easiest ways to be sure that is to perform the
tests in an impartial and qualified laboratory.
Quantitative KPIs for each test, and when possible agreed upon with the customer,
should be used.
Importance of testing and testing processes should be included in the internal
processes. This includes laboratories, people, etc.
Investments in upgraded test equipment should be done for laboratory testing.
Laboratory tests should be part of the process for certification or approval.
Table 6.3 – Recommendations for the laboratory testing from the answers
received from outside the railway sector.
GA 730640 Page 47 of 77
Deliverable D6.1
7 Harmonisation and authorisation activities

In this chapter, the different approval processes in different countries for railway systems
are considered and testing in laboratory or on-site is one of the important steps to fulfil
the specific approval requirements for CCS systems. Moreover, the interest in existing
working group and interest in standardized European approval process for the overall
CCS system that may be established in the future has been evaluated.
Approval statement is a crucial phase in a project because it gives the authorisation of
placing into service. The process is related to various parts/subsystems of the railway
system according to each project. Approval requirements come from different sources
such as Technical Specifications for Interoperability (TSI), National Rules, and customer
and supplier specifications. That leads to rather complex and different approval process.
And as evidence for approval, requirement test activities are central.
It has to be recognised that these harmonisation and authorisation activities are not in
scope of the X2Rail-1 project and information gathered will be shared with those dealing
with this, e.g. ERA. However, the following sections of this report summarise the
answers of five questions on harmonisation activities and approval process as they has
been included in the questionnaire in order to gain an understanding of the current
situation in the railway sector.
7.1 Existing harmonisation activities

First of all, it was important to know, if there are already existing working groups
regarding testing and which the most important ones are. The results can be seen in
Figure 7.1.
GA 730640 Page 48 of 77
Deliverable D6.1
Do you participate in any working groups

regarding testing in the past or today? Which
one?
9
8
7
number of replies 6
5
4
3
2
1
0
EUG
No
No answer
Others
UNISIG IOP
Group of ERA ERTMS

Groups related to TEN-T
Groups related to subset-076

stakeholders
Figure 7.1 – Testing Working Group participation
All groups that have less than two specified answers are displayed as “others” and can
be seen below.
Others:
 EMC working groups for ERTMS subsystems
 ESA for positioning applied to railway applications
 NVIOT / Network Vendors Interoperability Testing
 RAN working group
 TEN 3rd Call & "ETCS2 on GPRS experimentation" projects
 UIC, participating to GPRS testing , Austria
 CEF, participating to Network test cases definition
 Infrabel CFL ADIF Holland
 Preparation of Project "ETIP"
 Subset 036 [2]
 ERA working groups under the 4th RP for vehicle authorisation and TS approval
It appears that there are different and a large number of working groups regarding
testing that do exist at the moment or did exist in the past. At least, there are five main
groups: EUG, UNISIG IOP, groups related to TEN T, validation & test sub group ERA
GA 730640 Page 49 of 77
Deliverable D6.1
ETRMS stakeholders” platform, group related to subset 076 [3]. EUG and UNISG IOP
working group appear to be the most important ones. Eight stakeholders have no
participation at all. According to the questionnaire results, there is a real interest towards
working group, but testing working group activities seems to be fragmented. Some
companies, in particular some suppliers, participate in different working groups but in the
same time, others indicated to have no participation.
Concerning the question for further need to harmonise test strategies, most of the
replies confirm that there is a further need in harmonisation of test strategies in the
future, see Figure 7.2. A few also provided a reason or an area where they see a
specific need, see Figure 7.3.
No answer
13%
No
12%
Yes
75%
Figure 7.2 – Need for harmonisation of test strategies
GA 730640 Page 50 of 77
Deliverable D6.1
Yes, common
authorisation
Yes, 18%
interoperability
27%
Yes, common
process
9%
Yes, reduce
level of Yes, more
equipment effective, cost
18% and time saving
28%
Figure 7.3 – Possible targets of harmonized test strategies
27 % of the repliers who provided a reason (suppliers and system integrator) think that
interoperability is in need of harmonisation. The other answers are mostly connected to
effectiveness and a common management.
7.2 Different approval processes in different countries

Moreover, the approval process for CCS systems in different countries should be
explained in an additional question and table in order to compare them. The focus is not
on ERMTS only, as ERTMS is already harmonised.
As testing in laboratory or on-site is one of the important steps to fulfil the specific
approval requirements for CCS systems. Therefore, approval processes have to be
considered when discussing testing topics.
In fact, only 10 companies over 24 answered to this question (42%). Few companies
gave 2 answers that make 12 answers to analyse. It is well balanced because 5
answers come from infrastructure manager, 5 from suppliers. These answers cover 7
different European countries (1 answer per country + 2 answers for Denmark). 2
answers concern CENELEC, 1 telecom product and other TSI (almost empty).
Over all evaluated approval processes, the number of process steps varies from 2 to 8
process steps. In general, a lack of information on these steps was provided.
Consequently, there is a need to request additional information and details on each
national process to compare them into more detail in the field of CCS systems.
GA 730640 Page 51 of 77
Deliverable D6.1
However, the NSA responsibilities appear to be similar in every country: Process

verification and approval in line of CSM.
It is shown from the responses that the project leader (infrastructure manager or
supplier) has the responsibilities to provide evidence by undertaking tests or presenting
the results of test(s). RUs have not been asked.
The estimated time frame for the whole approval process is generally not precisely
known even if at least several months are requested. However, there are time-frame
indications, such as from 6 months to 26 months which is in the range of 30% of the
project time.
There are indications that some test labs exist, or will be available soon. These are
predominantly CENELEC test laboratories and integrity test laboratories for suppliers.
From the answers, it appears that the approval processes are different for every railway
network. Some of the processes are standard for systems such as ERTMS, but there
are differences in the application of the processes in different member states. Some
processes rely entirely on the supplier process to issue relevant documentation.
The process is driven by national rules (except for ERTMS). The processes have been
in place for many years and some countries are currently going a step forward to
simplification, transferring responsibilities and saving time and money to accept a
system, e.g. the German New Approval Process (NTZ).
System engineering management seems to be quite similar (despite from 2 to 8 steps
depicted). Validation processes must (and do) include the CENELEC standards. The
difference is how (and by whom) the process is to be supported. Through Europe, the
different NSAs have the responsibility to verify the process and approval in line with
CSM. Alternatively, IM or suppliers have the responsibilities to provide evidence by
performing test or presenting result of test. Approval process can take several months to
few years according to country-specific processes and project time frames.
It has to be noted that there are already few test laboratories in Europe, but it seems
that they don’t exist for the same purpose.
7.3 Challenges of European approval processes

There is a large interest in a standardised European approval process because 70% of
the replies answered this particular question and all of them saw a need and an
opportunity in standardisation of the approval process for CCS systems, comparable to
ERTMS which has already been standardised among Europe.
Several reasons for the interest in any further European approval process are visualised
in Figure 7.4.
GA 730640 Page 52 of 77
Deliverable D6.1
OBU integration
12%
No answer
29%
OBU integration,
without trackside
equipment
17%
Increase Time and cost

effectiveness in reduction and/or
the approval increase of
process competition
21% 21%
Figure 7.4 – Need for standardised European approval process? Why?
A few of the repliers answered that there is a need for standardisation of the OBU
approval process, but they also thought that the trackside equipment was not needed or
possible since there are big differences between the system configurations. A
standardisation of CCS systems and products will influence the related tests which are
needed for approval. Other possible advantages concern project aspects such as time
and cost reduction or the increase of effectiveness.
The different advantages are listed and evaluated in Figure 7.5.
GA 730640 Page 53 of 77
Deliverable D6.1
What will be the main advantages of a

European approval process?
8
number of replies 5
Safety
Money
Time
OBU
Ease
Reduce training
Competition
Uniformity
Acceptance
Figure 7.5 – Possible advantages of European approval process
Main expected advantages of a European approval process are time and competition.
Figure 7.6 show what the infrastructure managers see for the main advantages of a
European approval process, whereas Figure 7.7 shows the supplier perspective.
What will be the main advantages of a European approval

process? (Infrastructure Managers)
4
3
number of replies
0
No answer
OBU
Uniformity
Figure 7.6 – Advantages of European approval process - IM views
GA 730640 Page 54 of 77
Deliverable D6.1
From IM perspective, the expected advantages are limited to OBU approval process and
uniformity only. If systems or products are uniformed and the approval processes as
well, the test efforts and the number of tests related to project specific concerns (e.g.
OBU in different countries and from different suppliers) will be reduced as well.
What will be the main advantages of a European approval

process? (Suppliers)
7
5
number of replies
0
Time
Ease
Money
Safety
no answer
Acceptance
Uniformity
Competition
Figure 7.7 – Advantages of European approval process - suppliers views
From supplier perspective, the main advantage of a European approval process is time
saving. The second shared advantage for supplier is competition. These reasons differ
from infrastructure manager’s views.
7.4 Main conclusion of harmonisation and authorisation

analysis
ETCS as standardised system aiming at running trains throughout Europe could be the
vector to support a railway sector harmonisation. The difficulty will be to propose a
common approach, with consideration of national reluctance and railway local standards.
A large majority of participants from the railway sector (62%) consider that there is a
need to harmonise test strategies. Interoperability is identified as a need for
harmonisation of test strategies.
In the same way, a large majority are in favour of a standardised European approval
process in particular for OBU. There is less support for such activities on trackside
equipment due to large differences between system configurations in each country
interfacing with the legacy systems. For infrastructure manager, the main advantage of a
European approval process is the standardisation of OBU approval process. It has to be
GA 730640 Page 55 of 77
Deliverable D6.1
recognised that this area is not in scope of the X2Rail-1 project and information
gathered will be shared with those dealing with this, e.g. ERA.
GA 730640 Page 56 of 77
Deliverable D6.1
8 Conclusion
The key objective of zero on-site testing for CCS systems is to perform functional and
non-functional tests (component test, integration test and system test) in laboratory
instead of testing on-site in order to save time and costs without reducing safety aspects.
As the focus is on CCS systems, RUs have not been taken into account.
In the railway industry, most component tests are performed in laboratory due to ease
handling of these tests saving time and costs compared to on-site testing. Integration
tests are done in laboratory environment as well as on site, depending on the possibility
to simulate the real environment and worst-case scenarios in laboratories. Therefore,
real environmental data have to be recorded first. The complexity of the system is
important when assessing to shift tests from on-site to laboratory environment. System
tests, and especially performance tests, are mainly done in real environment in order to
gain trust in the results achieved. It is important to evaluate the interaction of the product
or service with the real environment, including human behaviour which is generally quite
difficult to simulate. As system boundaries, especially the (external) interfaces and the
hardware as well as the GSM-R and network communication have to be considered.
There is a trend to shift as much tests as possible to laboratories, even if many IMs do
not have own laboratories available. They mainly rent supplier’s laboratories or
laboratories of impartial institutes. Sometimes, customer force tests to be done in real
environment instead of laboratory tests in order to gain trust in the results. Reducing
time and costs by performing laboratory testing is possible only, if tooling is established
and grade of automation is high regarding test case generation, test execution and
evaluation of the results. Moreover, it seems to be challenging to clearly define exit
criteria for tests. Mixed teams with different experts together with a tool harmonisation
are beneficial while shifting tests to laboratory. Over all, a complete reduction of on-site
tests according to zero on-site testing won’t be possible in the near future due to
technical restrictions at laboratory environment.
Moreover, a benchmarking with safety-critical industries outside the railway sector has
been done, mainly with avionics, space and automotive sectors. Most of them state that
building up the laboratory test environment has been shown to be expensive and time
consuming at first, but very useful and helpful reducing project time and costs in the next
steps.
Almost all companies of all areas perform both laboratory tests and on-site tests. No
industry has shown the capacity and capability to perform all relevant tests in laboratory.
On-site tests are still required. The more the companies know about the on-site tests,
the more data they can provide as an input for laboratory environment in order to gain
GA 730640 Page 57 of 77
Deliverable D6.1
trust in the laboratory test and shift tests from on-site to laboratory as much as possible.
Automotive is showing that some tests currently performed in laboratory will be shifted
back to on-site in the future.
It is recommended to let laboratory tests start in an early stage of the product lifecycle,
when failures and features can be identified very early and therefore, costs and time can
be saved. Thus, these tests should allow testing a number of scenarios within a short
time, leading to shorter overall testing time. Laboratory tests should be planned in such
a way that logistics, management and communication processes should be reduced
compared to on-site testing.
When required, customers and regulators should be included in the assessment of the
whole development process (planning, requirements, basic design, detailed design,
manufacturing, and all testing levels) including certification and laboratory testing
activities. This can be done by standardising tests and performing standardised tests
afterwards. Laboratory tests are part of the process for certification or approval, e.g. in
medical or aviation sector.
Validation of the test site should be done by means of comparing laboratory data with
on-site data to ensure that the environment reproduced in laboratory matches with the
on-site and the behaviour of the system under test (SUT) in laboratory matches with the
behaviour on-site. When testing a product in the laboratory, a Product Risk Analysis is
quite useful. Laboratory tests should include systematic and transparent execution and
documentation including traceability.
The aviation is an area with a large number of approved standards and regulations and
it is a safety critical area. Therefore, the principles in the aviation area can be a useful
input for the railway sector and has to be analysed more in detail in future.
The recommendations coming from the answers of the benchmarking industries will be
further investigated in order evaluate if some of them will be applicable for the railway
sector as well.
For railway application, a first attempt to unification of testing of ETCS systems has
been realized by the ETCS Interoperability (IOP) working group of UNISIG developing
standards for testing IOP in laboratory environment.
A large majority of participants from the railway sector consider that there is a need to
harmonise test strategies and they favour a standardised European approval process in
particular for OBU. There is less support for such activities on wayside due to large
differences between systems in each country interfacing with the legacy systems.
GA 730640 Page 58 of 77
Deliverable D6.1
9 References
[1] Subset 026: System Requirements Specification
[2] Subset 036: FFFIS for Eurobalise
[3] Subset 076: Scope of the test specifications, test sequences and tests cases
[4] Subset 085: Test Specification for Eurobalise FFFIS
[5] Subset 092: ERTMS EuroRadio Conformance Requirements ans test cases safety
layer
[6] Subset 093: GSM-R interfaces
[7] Subset 110: UNISIG Interoperability Test – Guidelines
[8] Subset 111: Interoperability Test Environment Definition
[9] Subset 112: UNISIG Basics for Interoperability Test Scenario Specifications
GA 730640 Page 59 of 77
Deliverable D6.1
10 Glossary
Term Definition
Acceptance testing 1) Testing conducted to determine whether a system satisfies
its acceptance criteria and to enable the customer to determine
whether to accept the system
2) Formal testing conducted to enable a user, customer, or
other authorized entity to determine whether to accept a
system or component.
[ISO/IEC/IEEE 24765, 2010]
Compatibility 1) The ability of two or more systems or components to
perform their required functions while sharing the same
hardware or software environment
2) The ability of two or more systems or components to
exchange information.
3) The capability of a functional unit to meet the requirements
of a specified interface without appreciable modification.
[ISO/IEC/IEEE 24765, 2010]
Compatibility tests Tests regarding compatibility.
Component A constituent part of software which has well-defined interfaces
and behaviour with respect to the software architecture and
design and fulfils the following criteria:
- It is designed according to “components”;
- It covers a specific subset of software requirements;
- It is clearly identified and has an independent version inside
the configuration management system or is a part of a
collection of components (e. g. subsystems) which have an
independent version.
Component testing Tests regarding components.
Confidence testing Confidence testing is a term used to define the repeat
(duplicate) of any other test more than the absolute minimum
to achieve it's given success criteria. The number of repeat or
duplicate tests is an arbitrary number based on the user's
experience and trust in the system.
[X2Rail-1 WP6 definition]
Data tests Data tests are performed to verify the correctness of the
individual subsystem configuration data (e.g. static speed
profile) and they are related to a single subsystem in order to
test its specific application.
Environment Anything affecting a subject system or affected by a subject
system through interactions with it, or anything sharing an
interpretation of interactions with a subject system.
GA 730640 Page 60 of 77
Deliverable D6.1
Environmental tests Tests regarding environmental conditions. Environmental tests

are a group of tests composed by climate tests, vibration tests,
solar tests, tests with sand and dust, electromagnetic
susceptibility and emission etc.
EMC The ability of an equipment or system to function satisfactorily
in its electromagnetic environment without introducing
intolerable electromagnetic disturbances to anything in that
environment
[EN61000-2-2, 2002]
Factory acceptance Acceptance testing in the factory.
testing
Functional testing Testing conducted to evaluate the compliance of a system or
component with specified functional requirements.
[ISO/IEC/IEEE 24765, 2010]
Hardware 1) Physical equipment used to process, store, or transmit
computer programs or data.
2) All or part of the physical components of an information
system.
[ISO/IEC/IEEE 24765, 2010]
Hardware tests Tests regarding hardware.
Integration testing Testing in which software components, hardware components,
or both are combined and tested to evaluate the interaction
among them.
[ISO/IEC/IEEE 24765, 2010]
Integration test The progressive linking and testing of programs or modules in
order to ensure their proper functioning in the complete
system.
[ISO/IEC/IEEE 24765, 2010]
Interface testing Testing conducted to evaluate whether systems or components
pass data and control correctly to one another.
[ISO/IEC/IEEE 24765, 2010]
Interoperability 1) The ability of two or more systems or components to
exchange information and to use the information that has been
exchanged.
2) The capability to communicate, execute programs, and
transfer data among various functional units in a manner that
requires the user to have little or no knowledge of the unique
characteristics of those units.
[ISO/IEC/IEEE 24765, 2010]
For railway application: Interoperability means the ability to allow
the safe and uninterrupted movement of trains that accomplish the
specified levels of performance.
[ERA subset 023]
GA 730640 Page 61 of 77
Deliverable D6.1
Interoperability Testing conducted to ensure that a modified system retains the

testing (IOT) capability of exchanging information with systems of different
types, and of using that information.
[ISO/IEC/IEEE 24765, 2010]
IOP tests Interoperability testing according to ERA subset 110.
Net Access Net access is an alternative term for IOP tests and it is mainly
used in Switzerland.
Performance The degree to which a system or component accomplishes its
designated functions within given constraints, such as speed,
accuracy, or memory usage.
[ISO/IEC/IEEE 24765, 2010]
Principle tests Validation testing demonstrates that the system meets the
requirement for the system and the needs of the users.
A major facet of validation for Network Rail Signaling System is
commonly called “Principles Testing”.
[NR standard RT/E/S/11221]
Product Collection of elements, interconnected to form a system, a
subsystem or equipment, in a manner which meets the
specified requirements.
[EN50129, 2003]
Product tests Testing of a product.
Safety Freedom for unacceptable levels of risk of harm.
[EN50129, 2003]
Safety Tests Testing regarding safety.
Site acceptance Acceptance testing on-site.
testing
Software 1) All or part of the programs, procedures, rules, and
associated documentation of an information processing
system.
2) Computer programs, procedures, and possibly associated
documentation and data pertaining to the operation of a
computer system.
3) Programme or set of programmes used to run a computer.
[ISO/IEC/IEEE 24765, 2010]
Software Testing The dynamic verification of the behaviour of a program on a
finite set of test cases, suitably selected from the usually
infinite executions domain, against the expected behaviour.
[ISO/IEC/IEEE 24765, 2010]
Stability tests Method to check how an operational system and its quality
(software, hardware) behaviour with time elapsing, in a defined
environment, monitoring any unexpected event that occurs
during the duration of the test.
Subsystem A portion of a system which fulfils a specialised function.
[EN50129, 2003]
GA 730640 Page 62 of 77
Deliverable D6.1
System A system is a set of sub-systems which interact according to a

design.
[EN50129, 2003]
System testing Testing conducted on a complete, integrated system to
evaluate the system's compliance with its specified
requirements.
[ISO/IEC/IEEE 24765, 2010]
validation Activity of demonstration, by analysis and test, that the product
meets, in all respects, its specified requirements.
[EN50128, 2001]
GA 730640 Page 63 of 77
Deliverable D6.1
11 Appendices
The next two sessions consist of our questionnaires provided by WP 6 and distributed
among WP6 members (section 11.1) and among benchmarking industries (section
11.2).
11.1 Questionnaire Part A – status quo analyses in railway

sector
GA 730640 Page 64 of 77
Deliverable D6.1
GA 730640 Page 65 of 77
Deliverable D6.1
GA 730640 Page 66 of 77
Deliverable D6.1
GA 730640 Page 67 of 77
Deliverable D6.1
GA 730640 Page 68 of 77
Deliverable D6.1
GA 730640 Page 69 of 77
Deliverable D6.1
GA 730640 Page 70 of 77
Deliverable D6.1
GA 730640 Page 71 of 77
Deliverable D6.1
GA 730640 Page 72 of 77
Deliverable D6.1
11.2 Questionnaire Part B – benchmarking with safety-critical

industries
GA 730640 Page 73 of 77
Deliverable D6.1
GA 730640 Page 74 of 77
Deliverable D6.1
GA 730640 Page 75 of 77
Deliverable D6.1
GA 730640 Page 76 of 77
Deliverable D6.1
GA 730640 Page 77 of 77

Deliverable D6.1 Current Test Condition and Benchmarking Report

Uploaded by

Copyright:

Available Formats

You might also like

Deliverable D6.1 Current Test Condition and Benchmarking Report

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Deliverable D6.1 Current Test Condition and Benchmarking Report

Uploaded by

Copyright:

Available Formats

X2Rail-1

Action Full Title: Start-up activities for Advanced

Due date of deliverable Month 06

Author(s) Deutsche Bahn AG (DB)

2 ABBREVIATIONS AND ACRONYMS ....................................................................................................... 6

4 OBJECTIVES AND REPORT STRUCTURE ............................................................................................ 9

5 TEST ACTIVITIES WITHIN RAILWAY SECTOR ................................................................................... 10

5.1 CLUSTER OF ANSWERING COMPANIES .................................................................................................................. 10

6 TEST ACTIVITIES OUTSIDE RAILWAY SECTOR (BENCHMARKING) ............................................ 30

6.1 DESCRIPTION OF THE ANALYSIS DONE................................................................................................................... 30

7.1 EXISTING HARMONISATION ACTIVITIES ................................................................................................................. 48

11.1 QUESTIONNAIRE PART A – STATUS QUO ANALYSES IN RAILWAY SECTOR ...................................................................... 64

2 Abbreviations and Acronyms

NoBo Notified Bodies

4 Objectives and report structure

5 Test activities within railway sector

5.1 Cluster of answering companies

Figure 5.1 – Split of replies by company type

as a research centre or as an impartial test laboratory are included in the supplier

5.2 Description of the form distributed

5.3 Experiences of shifting tests to laboratory

5.4 Test Scope

Figure 5.2 – Products tested beside subset 076 [3]

requirements of the respective project, for example in the case of operational

5.5 Test Strategy

Figure 5.3 – Split of laboratory and on-site test execution

5.5.1 In-house vs. external tests

5.5.2 Reasons for performing laboratory tests

Required by Notified Body for

Necessary for Inter-System

Necessary for Product-Related

Necessary for System-Related

Figure 5.5 – Tests performed in the laboratory environment

5.5.3 Executing on-site tests

Figure 5.6 – Tests performed on-site

5.5.4 Test automation

Test Case Creation 1 1 1 1 2

Figure 5.7 – Some test procedures are automated.

5.5.5 Effort of tests

Time Needed for the Tests

Test Preparation and Execution

5.5.6 Creation of Tests

The creation of tests for each type is based on...

Table 5.1 – Sources of tests

5.6 Reasons for laboratory tests

Table 5.2 – Reasons for testing in laboratory

Do you see on-site as a complementary to

0% 20% 40% 60% 80% 100%

Figure 5.9 – On-site as a complementary to laboratory testing?

Do you use on-site testing as fall back?

0% 20% 40% 60% 80% 100%

Figure 5.10 – On-site testing as a fall back?

If your product is updated, do you repeat the

0% 20% 40% 60% 80% 100%

Figure 5.11 – Handling of updated products, regression tests

5.7 Gain trust and quality of test

5.8 Use of formal method verification