An Approach to the Efficient Assessment of Safety and Usability of

Computer-Based Control Systems (VeNuS 2)

Tobias Nelke
TÜV NORD SysTec GmbH & Co. KG
+49 40 8557-2727, tnelke@tuev-nord.de

Katharina Sachse
Technische Universität Berlin
+49 30 314-79541, katharina.sachse@tu-berlin.de

Manfred Thüring
Technische Universität Berlin
+49 30 314-21420, manfred.thuering@tu-berlin.de

Cristina Olaverri Monreal

Technische Universität München
+49 89 289-15413, olaverri@lfe.mw.tum.de

Carsten Dlugosch
Technische Universität München
+49 89 289-15019, dlugosch@lfe.mw.tum.de

Klaus Bengler
Technische Universität München
+49 89 289-15400, bengler@lfe.mw.tum.de

Abstract
In a collaborative project several tools for the efficient assessment of safety and usability were
developed.
The TÜV NORD assessment screen for efficient assessment provides new approaches for
evaluating defined subsets of normative requirements and interfaces to automated assessment
tools. Two different evaluation approaches, one based on VeNuS research results and one based
on SPICE are available. Support for traceable assessment documentation and assistance during
the development process is realised in the assessment screen. Additional tools for automated
assessment of text documents and calculation of hardware failure rates involving software based
hardware monitoring have been developed.
TU Berlin focused on the usability of digital human-machine-interfaces in nuclear facilities. An
approach to assess aspects of usability by considering safety-related priorities was developed. The
evaluation procedure is provided as a computer application, which can be used for the direct
collection, computation, and presentation of data.
The TU München proposed a procedure (“ManPro”) for the computer-based creation of instruction
manuals for the operation of nuclear plants. The objective is to improve the operation efficiency of
such facilities and prevent human errors whenever possible. Thus, a template-based, interactive
semiautomatic process which underlines its reproducibility and efficiency is used to guide technical
authors through the aspects of the manual creation.

1. Introduction
Modernisation of installed computerised control systems in nuclear power plants often requires new
components to be installed. Today’s computerised control systems provide possibilities to detect
and control unintended operating conditions.
A confirmation of satisfactory safety and proper man-machine-interface must be provided for the
intended control system. Appropriate methods should be available to confirm that hardware and
software comply with the state of the art. Because of the numbers of requirements which have to
be considered, the identification and confirmation of their fulfillment is both a qualitative and
quantitative problem.

The research project VeNuS 2 (Vorgehen zum effizienten Nachweis der Benutzbarkeit und
Sicherheit rechnergestützter Leittechniksysteme; An Approach to the Efficient Assessment of
Safety and Usability of Computer-Based Control Systems) is based on this experience and on
many results of the research project VeNuS 1 [1]. The overall objective is to develop methods and
tools to assess the dependability of computerised I&C systems in nuclear power plants.

The VeNuS 2 project is a collaborative research project between TÜV NORD, TU Berlin and TU
München with the participation of IFE (Halden Reactor Project) and CATS Software Tools GmbH.

2. Integral assessment screen

Within the TÜV NORD framework of the assessment of safety and quality of programmable
electronic systems and their software many procedures are available for supporting specific steps.
However, in a lot of cases an approach for the assessment which integrates the specific steps to
an efficient overall procedure covering the whole assessment is missing. Within VeNuS an
assessment procedure for programmable electronic systems and their software which integrates
the knowledge as well as the supporting tools available within the partners, has been defined. The
procedure is supported by a tool chain [2].

The tool chain is given in figure 1. The tool NormViewer (figure 1, see chapter 2.1) presents the
entire and complete set of requirements of nuclear technical standards to the user in a more
manageable and understandable form. In addition a selection of problem- or plant-based
requirements is possible.

To support developers during their software development the tool chain is able to transfer selected
requirements to the requirements management tool Halden RM (figure 1, see chapter 2.2) which is
able to assist the generation of the necessary development documentation by presenting
requirement specific templates or process descriptions.

The selected requirements can be imported to the tool VEQ (VeNus Embedded Quaity, figure 1,
see chapter 2.3), which is able to evaluate the achieved quality, the compliance with the whole set
of the selected requirements and the compliance according to SPICE (Software Process
Improvement and Capability Evaluation).

To assess the selected requirements automatically the Interface Tool (figure 1) instructs
assessment tools like static analysers and user manual assessment tools. These tools generate
assessment results for the evaluation with VeNus Embedded Quality.

An assistance tool (WP6 tool, see chapter 2.4) was developed to support the assessor with a
mapping of normative documents to documents according a manufacturers’ document scheme and
to real supplied project documents.

An EXCEL based tool (Failure rate calculation tool, chapter 2.5, not included in figure 1) allows
the calculation of hardware failure rates. Additionally you get hints for software based self-
monitoring to reduce the rate of undetected dangerous failures.
Not included in figure 1 is a document assessment tool (see chapter 2.6) to analyse plain
language based documents.

NormViewer

HALDEN RM
VEQ

Templates
Interface Tool

User Manual
REVEAL CATS Checklist Assessment

Figure 1. Integral assessment screen

2.1 NormViewer

The users of technical standards frequently have problems to recognise and handle all
requirements defined in these documents. Many requirements have to be taken into account. In
many cases users did not survey all relevant requirements. This may cause avoidable errors.

As a consequence of this user problem the tool NormViewer was developed. It is a

reimplementation of the RiskCAT nuclear tool [3] with extended functionality and new designed
user interface. A main functionality of the NormViewer is to present the entire and complete
requirements defined by a technical and/or safety standard to the user. To make all requirements
more manageable and understandable it is possible to add to each requirement a shortened form
of the requirements text, the specific safety relevance, a link to the original requirement source
(PDF-file), comments (project specific, customer specific and general), relevant quality
characteristics, documents and analysis tools. In the RiskCAT nuclear tool it was only possible to
view this information, but with the NormViewer, it is also possible to edit this information and to
generate new standard-databases.

Moreover the user can select relevant requirements for a specific problem or a specific plant by
hand or by use of an assistant (e.g. document based). The selected requirements can be exported
to be used in other tools or documents.

The TU Berlin evaluated the user interface of the RiskCAT nuclear tool and gave several hints to
improve the user interface [4]. These hints have been considered during the design of the
NormViewer, its new functionality and its user interface.

2.2 Halden RM

Support for developers of computer-based control systems regarding the fulfilment of relevant
requirements would be helpful during the development and a benefit for the safety of the systems.
The developer is faced with the question of how he should structure documents, present
information or apply techniques to meet the requirements.

HALDEN RM is a tool being developed at the OECD Halden Reactor Project, Institute for Energy
Technology, for the management of requirements. It is based on the TACO Traceability Model,
developed in the Nordic TACO Project (2002-2005) [14]. Accordingly, the tool focuses on the
traceability of paragraphs, usually representing requirements, and is designed to support the
management of changes to these paragraphs. The specified needs in VeNuS 2 require just a small
part of the functionality of the tool.

Within VeNuS a set of document templates (e.g. quality assurance plan) and process descriptions
(e.g. review) have been prepared to support the software development and its documentation to
fulfill the relevant requirements defined by the standards.

The integration of HALDEN RM in VeNuS is the realization of an interface between NormViewer

and HALDEN RM to import the selected requirements into HALDEN RM. Standards expect the
fulfillment of a requirement in a specific document. The information which document is expected is
included in each requirement. HALDEN RM provides access to the relevant template or process
description by a hyperlink [5].

2.3 VeNuS Embedded Quality

The VeNuS 1 project worked on reproducible approaches to derive and to evaluate requirements
based on safety and quality requirements. Users of the tool VeNuS Embedded Quality can define a
target quality profile for the computerised control system [6]. An interface to the NormViewer makes
it possible to import the selected requirements. Thus, a manual database generation is not
necessary and the further work with the tools of the assessment screen is limited to the problem- or
plant-specific requirements. After the measurement of each requirement, VeNuS Embedded
Quality is able to evaluate the achieved quality (in comparison to the target quality profile), and the
compliance with the whole set of selected requirements [6].

A second evaluation approach is a SPICE (Software Process Improvement and Capability

Evaluation) [7] adaption. SPICE is an often used software development process standard in other
industry sectors and by this SPICE defines the state of the art where nuclear applications should
not go below. With this SPICE approach the evaluation is possible based on software development
processes.

2.4 WP6 tool

Development guidelines and standards require certain information. The information is provided by
manufacturers and utilities in the form of documents. As part of examinations, e.g. certifications,
inspections or in third party audits, a mapping between required information and provided
documents must be established. As experience shows, this document assignment is often not
clearly documented, leading to a poorer reproducibility and traceability of the assessment (the
measurement of each requirement). To improve the document assignment and the record of this
assignment, a supporting tool was established in a work package, based on the NormViewer.

In the NormViewer each requirement can be assigned to a normative document. In VeNuS 2 an

additional tool (WP6 tool) was created, that operates on the same database as the NormViewer.

Based on manufacturers’ (or utilities’) quality systems, information is provided by specific

documents. The WP6 tool supports the mapping of normative documents to documents according
a manufacturers’ document scheme and to real supplied project documents. The mapping is
achieved by simultaneous assignment of these documents to all requirements that are linked to a
specific normative document.
In NormViewer, this information can be used in data exports or for checklist generation. This helps
assessors to create checklists that already include a document mapping between requirements
and supplied documents so that the document assignment is an integral part of the written
assessment documentation.

2.5 Failure rate calculation tool

The analysis of failure rates of electronic (E/EE/PE) equipment used in safety functions shall be
state of the art. Several standards, e.g. DIN EN 61508 [12] and DIN EN 50129 [8] define
requirements for architectural design and the target failure measures.

An EXCEL based tool has been established that allows the calculation of the failure rates. In a
typical workflow, all elements of a hardware component are added to the tool, automatically
attached with typical failure modes and rates taken from IEC TR 62380 [9]. Additional failure
modes as listed in DIN EN 50129 can be considered. The characteristic figures like SFF (Safe
failure fraction) and the probability of dangerous failure on demand (PFD) or per hour (PFH) for the
component are automatically computed on that data basis.

One possibility to reduce the rate of undetected dangerous failures is the use of self-monitoring.
These diagnostic measures are often implemented by software. The failure rate calculation tool
focuses on the development of a strategy to determine the hardware elements and corresponding
diagnostic measures that reduce the failure rate of undetected, dangerous failures. An adaption of
the failure rate calculation tool to the strategy will follow.

2.6 Document assessment tool

Most documents of a software development are plain language based (e.g. user manual). While
the code checking is at least in part automated, and therefore reproducible and efficient, the
checking of the other products of work is not automated and may be hard to reproduce.

A document assessment tool has been created that determines 67 properties of plain language
documents [10]. Based on the properties the tool constitutes and evaluates 87 basic characteristics
of the document (like document structure, forbidden words, ambiguous words or phrases,
completeness of tables, references, indexes, text readability, etc.). Therefore a *.docx document is
decomposed into *.xml parts and is analysed with the use of parsers and SQL. Analysis results are
presented partly in *.html and partly in a structured way in a tool with a graphical user interface.

3. Measuring usability of safety-relevant systems

In more and more facilities, digital human-machine-interfaces are used to support the monitoring
and control of safety-relevant or safety-critical processes. While substantial measures are taken to
ensure that these systems satisfy high safety standards, much less is done to warrant their
usability. This is a severe shortcoming, since software-ergonomic deficits can result in serious
problems or even accidents [11]. Hence, in safety critical domains, poor usability is not only an
inconvenience, but can be a safety risk. To address this problem TU Berlin developed a computer
based procedure for usability assessment of safety critical systems.

The international standard IEC 61508-2: Functional safety of electrical/electronic/programmable

electronic safety-related systems demands that the design of these systems “shall take into
account human capabilities and limitations and be suitable for the actions assigned to operators
and maintenance staff”. Further it requires that “the design of all interfaces shall follow good
human-factor practice” [12]. Although the fulfillment of these requirements necessitates usability
evaluations, systems are often implemented without being adequately tested. This has several
reasons: Users of safety relevant systems are specialized operators who have little time for off-site
tests during the design process. Safety relevant and critical systems consist of various software
and hardware components and testing a single component in the usability lab might neglect the
working situation. For these reasons, lean methods implemented as computer-based tools are
needed to measure usability of safety-critical systems efficiently in the field.

3.1 Usability questionnaires

An established way to test the usability of human-machine-interfaces is the use of standardized

questionnaires that allow the assessment of various evaluation aspects without requiring lab tests.
Systems can be evaluated in their natural working context. Analysis of results is relatively easy and
studies using the same questionnaire to assess different systems can be compared. Hence the
questionnaire method seems particularly appropriate to verify the usability of safety-relevant
systems.

Over recent years, a number of standardized user questionnaires have emerged (e.g., SUS [13],
IsoNorm 9241-110/S [15]). Users rate their experiences with a system on several items which
address different aspects of usability. The ratings are combined into a quantitative usability score.
Some questionnaires also allow for the computation of sub-scores which provide more detailed
information. For instance, the German questionnaire IsoNorm 9241-110/S addresses seven
requirements defined in the ISO 9241-110: Ergonomics of human-system interaction - Part 110:
Dialogue principles [16] The requirements are: suitability for the task, suitability for learning,
suitability for individualization, conformity with user expectations, self-descriptiveness,
controllability, and error tolerance. Each principle is considered as a separate dimension and
measured by three items. Table 1 gives an example of the items building the dimension error
tolerance.

The system…
does not offer all necessary offers all necessary
functions to efficiently ○○○○○○○ functions to efficiently
master all given tasks. master all given tasks.

does not require

requires unnecessary input. ○○○○○○○ unnecessary input.

inappropriately meets the appropriately meets the

demands of the work. ○○○○○○○ demands of the work.
Table 1: Items of the IsoNorm 9241-110/S measuring suitability or the task [15]

3.2 Prioritizing usability aspects

In safety-relevant systems, the compliance with usability requirements can get into conflict with the
fulfillment of safety requirements. To give two examples:
• Safety relevant systems might not provide the possibility to individualize the look of the user
interface. This assures that different operators use identical working environments, but at the
same time it prevents the adjustment of system features to individual needs and preferences.
• Sometimes, commands must be affirmed before they are carried out by the system. This
reduces error probability, but also decreases the efficiency of operations.
To consider such constrictions in the usability evaluation, a prioritization of aspects is necessary.
Therefore, we complement the usability assessment with a weighing procedure to identify those
usability aspects which are especially important for safety-relevant
safety systems. Here users evaluate
which usability principles are of importance for the safe and efficient operation of the system. The
resulting weights can be combined with the usability ratings to receive a weighed usability score.

3.3 ReMUS - A software tool for the usability evaluation procedure

We developed a computer-based
based tool that combines usability questionnaires with weighingw
procedures and automatically calculates usability scores. The tool is called ReMUS,
ReMUS which is an
acronym of the German term Rechnerbasiertes Multiattributives Usability sability Scoring (English
translation: Computer-Based
Based Multiattributive Usability Scoring).
Scoring The tool is available in German and
in English.

In ReMUS, various standardized usability questionnaires are provided as computer-based

computer versions
(e.g. IsoNorm 9241-110/S,
110/S, SUS). Usability evaluators can select a questionnaire and combine it
with a weighing procedure of their choice. Additionally, texts, pictures, or self-created
created questions can
be included to customize the survey. Figure 2 shows a screenshot of the interface for the
investigator.

The tool automatically generates a ready-to-use

ready computer-based survey that can be presented to
users who are requested to assess a system. The survey can be executed online or offline. Results
of the usability scoring procedure are directly calculated after a user has filled in the survey. The
data are saved as excel-file
file which can be used for further analyses. Additionally, a ready-to-use
ready
report can be exported that reports mean ratings and weightings as well as the mean usability
score by providing charts and tables. Figure 3 shows an example result of a usability evaluation
e
with ReMUS.

Figure 2. Screenshot of the usability evaluation tool ReMUS.

 Figure 3. Prototypical results of a ReMUS survey.
By identifying those aspects with high priority but low usability ratings, potential problems
can be detected, that might have an impact on the safe handling of the system.
The Usability Score can be used for bench-marking.

3.4 Validation of the ReMUS-tool

3.4.1 Expert interviews

To validate the concept of the ReMUS tool we conducted interviews with four experts in digital
systems for nuclear facilities. We discussed the appropriateness of usability requirements for
safety-relevant systems and asked for special safety measures that might influence the usability of
a system. The results can be summarized as following:
1. No conflict between Safety and Usability: Many safety measures are consistent with
usability standards.
Example: A system offers only necessary functions to efficiently master given tasks.
2. Conflict between Safety and Usability: Some safety measures conflict with usability
requirements.
Example: Actions have to be confirmed twice to avoid accidental actions.
3. Differences between Safety and Usability: There are fundamental differences with regard
to operating errors. Safety measures aim to avoid errors, while usability measures aim to
tolerate errors.
Example: Actions are restricted to options that are permitted in a given situation.

We concluded from the interviews that standard usability questionnaires as used in the ReMUS
tool seem to be appropriate to evaluate the usability of safety relevant systems. Additionally,
aspects of error avoidance have to be assessed. Therefore we developed a questionnaire
addressing this aspect and included it into the tool.

3.4.2 User tests

Two user tests were conducted to validate the appropriateness, acceptance, and functionality of
ReMUS in the field. In one test six operators of a research reactor evaluated the usability of their
digital control room system. In the second test seven plant mechanics of a nuclear power plant
assessed the usability of two digital human-machine-interfaces for assistance systems. Both
groups of participants were asked to provide feedback on the evaluation procedure. There was
positive feedback on the detailed assessment of usability aspects, and the acceptance of the
procedure was high. Participants had no difficulties to accomplish the ratings. However, some
participants had problems with the weighing procedure that was selected for the test and produced
sets of weights that were inconsistent.

3.5 Further development of ReMUS

In a next step, the design of the weighting procedure used in the tests will be improved to prevent
inconsistencies, but even currently this deficit is not a major problem. Already now, it is possible to
use other (simpler) weighting procedures, which are also implemented in ReMUS, as an
alternative.

It is also planned to generate domain-specific standard weights by asking experts to do the

weightings. These standard weights can be used as defaults, so that future users only have to do
the usability ratings. However, the definition of standardized weights requires extensive analyses
with datasets of many users and systems to test if weights are stable over time. Another aim is to
provide benchmarks for the usability score, but this requires also data about many systems.
Therefore, we invite researchers and developers in the field of safety-relevant human-machine-
interfaces to apply the ReMUS tool for usability evaluations.

4. ManPro - Computer based approach to generate and assess

documentation for nuclear facilities

4.1 Motivation

The technical documentation of the components of a nuclear power plant describes the system’s
behaviour, structure and operating rules that the operators need to follow [17]. Handbooks and
manuals support nuclear facilities operators’ tasks and help to prevent errors. How to determine
which information users need and the way in which that information should be presented to users,
as well as how to prepare the information and make it available has been compiled and published
by the International Organization for Standardization (ISO) through the series of standards related
to technical product documentation and nuclear power plants [18, 19]. Due to the many
specifications to be considered in such an environment it is crucial to improve the operation
efficiency of such facilities, while at the same time considering the dynamic nature of technical
documentation.

TU München aims to minimize the barrier between the user and the system components of nuclear
facilities focusing on the perception of the environment in such a complex scenario as nuclear
plants are that are critical to decision-making processes. Additionally, component integration needs
to be properly considered in the documentation without increasing the cognitive load and errors of
the facility operator and at the same time be clear and precise so that human errors can be
prevented whenever possible.

Based on the results from the previous project Venus 1 TU München developed a procedure to
create instruction manuals for the operation of technical systems semi-automatically using as input
UML descriptions of the nuclear components and the styles and guidelines developed in Venus.
We propose a procedure (ManPro) for the computer-based creation of manuals. The “ManPro”
approach is semiautomatic, which underlines its reproducibility and efficiency. Our approach is
language and platform independent, ensures accuracy of documentation content through
predefined fields for data entry, preventing errors and guarantees that the information contained in
the final user manual is accessible to multiple users, is searchable and understandable, and
ambiguity does not exist. Additionally, the tool enables multilingual content.

4.2 Man Pro Architecture Components

In “ManPro” the individual functions of the descriptive components of the instructions manual are
represented as an interactive modular process where functionally and information flow are depicted
through several technologies based on mark-up languages or languages that contain annotations
embedded in the text. The following components are integrated in the “ManPro” architecture:
• A UML file containing the structure for a modelling specification of a system from the nuclear
facility. The Unified Modelling Language (UML) is a graphical language for visualizing,
specifying, constructing and documenting the compounds of a software system. UML offers
many degrees of freedom and different ways to describe a specific situation. Therefore, the
UML standard version 2.2 (2009) was examined; particularly the backward compatibility and
new plots. From the UML specification the overall most relevant diagrams were selected and
validated for their machine-readability by a XMI parser.
• A web form consisting of predefined fields extracted from the UML system specification. The
form ensures accuracy in the data entry and prevention of errors through guided questions.
Additionally information can be accessed from any computer with Internet access.
• A relational database that enables the storage of the content contained in different web forms
and a simple data access, update and latter visualization. The information available in the
database can then be extracted using technologies such as Java and MySQL queries, to
generate information related to the different parts of the final instruction manual, like chapters,
sections, graphics, paragraphs, subsections, or different kind of lists.
• An XML document, extracted from the database that contains the database structure, with the
different document parts to be further transformed into the final instructions manual.
• A final instructions manual version in a human readable and printable format.

To edit an instruction manual, the user only needs to access the appropriate web form that has
been previously created from the UML file architecture description and then enter the required
information already specified in the pre-defined fields. Through a user friendly Graphical User
Interface (GUI) the web form will be then converted into the final format specified by the operator.
The detailed implementation process of the modular structure is described in [20].

Figure 4 shows the whole process to generate the final file including the information processing
steps, input and output files.
 Figure 4. “ManPro” process to generate the final instructions manual

Figure 5 depicts a section of the graphical user interface (GUI) used to create a new instructions
manual. Since the form is written in the Hypertext Mark-up Language (HTML), it can be visualized
in any Web browser. User input is validated on the server side through a PHP script. Figure 6
shows a section of the final instructions manual generated through with the “ManPro” framework.

Figure 5. Section of the GUI used to create new instruction manuals

 Figure 6. Section of the final instructions manual document in a PDF format.

4.3 Evaluation

“ManPro” makes it possible to use the manuals that have been generated through the semi-
automatic process for the quality assurance stage as a reference for the evaluation of other
existing manuals. Criteria such as the consistency of the functions of the system, the completeness
and the clarity can be compared and evaluated. The assessment criteria developed in Venus 1 in
conjunction with the partially automated created manuals provide a reproducible test method.

The information contained in the final manuals accomplish with the following guidelines to evaluate
the information of the instructions manuals [21]:
• The instructions describe all the product characteristics in a step-by-step procedure;
• The manual includes a quick start guide;
• It also includes a list of the functions;
• The manual includes line numbers to help with cross references;
• Instructions are presented in the form of step-by-step procedures;
• The information is written in a consistent way;
• Sections are ordered by frequency of use;

5. Conclusion
The VeNuS research project shows that the problems within the modernisation of installed
computerised control systems in nuclear power plants concerning the consideration of all relevant
requirements during the development, their fulfilment and assessment is a solvable task. Within
VeNuS, appropriate methods and tools are available to confirm that hardware and software comply
with the state of the art. It is possible to get assistance to develop and assess the required
dependability of computerised I&C systems in nuclear power plants.

6. Acknowledgement
The VeNuS 2 project is funded by BMWi (Federal Ministry of Economics and Technology) as
project numbers 1501387, 1501388, 1501389.
7. References
[1] G. Glöe, T. Hadler, “Final Report, On the Project, An Approach to the efficient assessment of
safety and usability of computer-based control systems”, TÜV NORD, June 2008.

[2] G. Glöe, B. A. Gran, T. Hadler, H. Miedl, Ch. Raspotnig, “Technical Report, On WP 4.3,
Integral Assessment Approach within the project, An Approach to the efficient assessment of
safety and usability of computer-based control systems”, TÜV NORD, IFE, ISTec, May 2007.

[3] G. Dahl, B. A. Gran, E.-U. Mainka, J. Märtz, H. Miedl, “Report on the research project
VeNuS, An Approach to the efficient assessment of safety and usability of computer-based
control systems, Report on workpackage 4.1.2, Processing of requirements on development
and proof of safety from technical standards”, TÜV NORD, IFE, ISTec, April 2007.

[4] K. Pataki, M. Thüring, “Technical Report, On WP 4.2.1, Configuration Management

supporting the Software Development Process within the project, An Approach to the
efficient assessment of safety and usability of computer-based control systems”, TU Berlin,
April 2006.

[5] T. Sivertsen, “HWR-1051 HALDEN RM (Requirements Manager) – Supporting the

Management of Requirements Traceability”, OECD Halden Reactor Project, 2013.

[6] U. Anders, G. Glöe, B. A. Gran, T. Hadler, H. Miedl, “Technical Report, On WP 4.1.1, Quality
of computers and their software within the project, An Approach to the efficient assessment
of safety and usability of computer-based control systems”, TÜV NORD, IFE, ISTec,
November 2006.

[7] ISO/IEC 15504: Information technology – Process assessment, 2003-2004.

[8] DIN EN 50129: Railway applications – Communications, signalling and processing systems
– Safety related electronic systems for signalling, December 2003.

[9] IEC TR 62380: Reliability data handbook. Universal model for reliability prediction of
electronics components, PCBs and equipment, November 2004.

[10] G. Glöe, T. Nelke, “Work Package Report, On WP 8, Steps towards automated testing of
documents within the project, An approach to the efficient assessment of safety and usability
of computer based control systems”, CATS, IFE, TÜV NORD, June 2012.

[11] N. G. Leveson, “Safeware, system safety and computers”, Addison-Wesley, Amsterdam,

Netherlands, 1995.

[12] IEC 61508-2: Functional safety of electrical/electronic/ programmable electronic safety-

related systems. 2000.

[13] J. Brooke, “SUS: a ‘quick and dirty’ usability scale”, In P. W. Jordan, B. Thomas, B. A.
Weerdmeester, A. L. McClelland (Eds.), “Usability Evaluation in Industry”, Taylor and
Francis, London, GB, 1996.

[14] T. Sivertsen, R. Fredriksen, A. P-J. Thunem, J-E. Holmberg, J. Valkonen, O. Ventä, J-O.
Andersson, “Traceability and communication of requirements in digital I&C systems
development”, TACO final report, NKS-115, October 2005.
[15] J. Prümper, “ISONORM 9241/110-S: Evaluation of software based upon International
Standard ISO 9241, Part 110.” Available online: http://www.f3.htw-berlin.de/Professoren/
Pruemper/instrumente/ISONORM_9241_110-S_2010.pdf.

[16] ISO 9241-110: Ergonomics of human-system interaction - Part 110: Dialogue principles.
2006.

[17] T. Pyzdek, Keller PA, “Quality engineering handbook”, CRC, 2003.

[18] ISO 01.110: Technical product documentation, international organization for standardization,
July 2012, http://www.iso.org/iso/catalogue_ics_browse?ICS1=01&ICS2=110&.

[19] ISO/IEC 26514-2008: Systems and software engineering. Requirements for designers and
developers of user documentation, July 2012, http://www.iso.org/iso/home/store/
catalogue_ics/catalogue_detail_ics.htm?csnumber=43073.

[20] C. Olaverri-Monreal, C. Dlugosch, K. Bengler, “ManPro: Framework for the Generation and
Assessment of Documentation for Nuclear Facilities”, World Conference on Information
Systems and Technologies, WorldCIST'13, 2013.

[21] K. Inaba, Parsons SO, Smillie RJ, “Guidelines for developing instructions”, CRC, 2004.