Professional Documents
Culture Documents
Automated Requirements Analysis Using Natural Language Processing 8
Automated Requirements Analysis Using Natural Language Processing 8
Mariana Oliveira
Mariana Oliveira
The automotive domain has been recognised as being remarkable regarding aspects like quality
and safety. Behind all the technology involved in the creation of automotive software, there are
several Software Engineering processes taking place.
One crucial process is the production of artefacts. some examples are Use Cases, Models,
Requirements and Design Documents. They help describe the features, architecture and design of
the software, therefore their quality is directly linked to the quality of software systems as well as
their development process.
Surveys have shown that the most common reasons for a project’s lack of success are not
technical but rather problems with requirements. For that reason, during this investigation, the
Requirements documents and their quality will be the target.
Considering that the automotive market is as demanding as ever concerning innovations of
increasing complexity, the associated shorter development periods make it paramount to improve
software development processes.
In order to guarantee the quality of their software, automotive companies support requirements
reviews which aim is to ensure these documents are abiding to a series of international quality
standards. These describe various quality criteria which are then used to measure the quality of
a project. Making sure these quality criteria are fulfilled during the requirements reviews is a
complex task that requires effort and resources extending the time to release new products.
It is a challenge to find a way to transform this task in a simpler and more efficient action.
As such, the centre of this research is in identifying applicable quality criteria and their respective
process metrics in the interest of creating a tool that finds, interprets and alerts the user about faults
or incoherences in the requirements.
As most requirements are written using Natural Language due to it being easily understandable
by all stakeholders, a tool that analyses their quality should apply Natural Language Processing
(NLP) techniques. Although there are several tools available in the market that support require-
ments analysis, none of them provide the necessary input type and integration means and that is
why a tool was created to tackle that lack of suitable solutions.
The tool that was developed during this investigation is called Requirements ANalysis Tool
(RANT). RANT employs several NLP techniques such as Sentence Splitting, Tokenization and
Part-of-Speech tagging, in order to assess the requirements quality by producing a final evaluation
regarding the requirements structure and potential ambiguity.
Creating a tool that was integrated in the used Requirements Management environment was
an essential feature. Hence RANT was developed as an extension of IBM’s Requirements Man-
agement environment called Collaborative Lifecycle Management (CLM).
Alongside RANT, a checklist was created with the aim of being used by the analysts together
with RANT in the interest of helping the analysts to perform a more complete and accurate re-
quirements review.
i
As a form of validation of RANT, binary diagnostic testing was performed with the main
goal of obtaining precision, recall and accuracy values. The results obtained seem promising with
values of 82%, 95% and 85% regarding precision, recall and accuracy respectively.
ii
Resumo
O setor automóvel é reconhecido como sendo notável em aspectos como a qualidade e a se-
gurança. Por detrás de toda a tecnologia envolvida no desenvolvimento de software automóvel,
ocorrem vários processos de Engenharia de Software.
Um processo crucial é a produção de artefactos, alguns exemplos são os Casos de Uso, Mod-
elos, Requisitos e documentos de Design. Estes artefactos ajudam a descrever as características, a
arquitectura e design do software. Assim, a qualidade destes documentos está directamente ligada
à qualidade dos sistemas de software, assim como ao respectivo processo de desenvolvimento.
Estudos mostraram que a maioria das razões comuns para o insucesso de um projeto não são
por motivos técnicos mas sim por problemas com os requisitos. Por esse motivo, durante esta
investigação, o foco estará nos documentos de requisitos e na respetiva qualidade.
Considerando que o mercado da indústria automóvel está cada vez mais exigente em relação a
inovações de complexidade crescente e os consequentes períodos de desenvolvimento mais curtos,
levam a uma necessidade de melhorar os processos de desenvolvimento de software.
Para garantir a qualidade do seu software, empresas do ramo automóvel fazem revisões aos
requisitos, cujo objectivo é assegurar-se que estes documentos cumprem uma série de standards
internacionais de qualidade. Estes standards descrevem vários critérios de qualidade que são então
utilizados para medir a qualidade de um projecto. Certificar-se que estes critérios de qualidade se-
jam seguidos durante as revisões de requisitos é uma tarefa que requer bastante esforço e recursos,
levando a um prolongamento do tempo até ao lançamento de novos productos.
É um desafio encontrar uma forma de transformar esta tarefa numa acção mais simples e mais
eficiente. Como tal, o centro desta investigação é identificar critérios de qualidade aplicáveis e as
respectivas métricas de processo com o objectivo de criar uma ferramenta que encontra, interpreta
e alerta o utilizador acerca de falhas ou incoerências nos requisitos.
Como a maioria dos requisitos são escritos utilizando Linguagem Natural devido a ser facil-
mente compreensível por todos os interessados, uma ferramenta que analisa a qualidade dos requi-
sitos deverá aplicar técnicas de Processamento de Linguagem Natural (PLN). Apesar de existirem
várias ferramentas disponíveis no mercado que efectuam análises aos requisitos, nenhuma delas
providencia o tipo de input necessário ou os meios de integração pretendidos e, por isso, foi criada
uma ferramenta para mitigar essa falta de soluções adequadas.
A ferramenta que foi desenvolvida ao longo desta investigação denomina-se Requirements
ANalysis Tool (RANT). A RANT utiliza várias técnicas de PLN como Sentence Splitting, Tok-
enization e Part-of-Speech tagging, para aferir a qualidade dos requisitos através da produção de
uma análise que avalia a estrutura e a potencial ambiguidade que possa existir nos requisitos.
A criação de uma ferramenta que estivesse integrada no ambiente de gestão de requisitos
utilizado era uma característica essencial. Assim, a RANT foi desenvolvida como uma extenção do
ambiente de gestão de requisitos da IBM chamado Collaborative Lifecycle Management (CLM).
Em paralelo com a RANT, foi elaborada uma checklist com o objetivo de ser utilizada junto
com a RANT e poder ajudar os analistas a realizar revisões de requisitos mais exatas e completas.
iii
Como forma de validação da ferramenta RANT, o teste de diagnóstico binário foi o escolhido
com o principal objectivo de obter os valores de precisão, revocação e exatidão. Os resultados
obtidos foram promissores com valores de 82%, 95% e 85% para a precisão, revocação e exatidão,
respetivamente.
iv
Acknowledgements
First, I would like to thank my thesis supervisor Prof. João Pascoal Faria for his invaluable
guidance and support.
I am grateful to my co-supervisor Nuno for allowing me to freely pursue my research while
still giving me all the guidance needed.
I would like to express my appreciation to Bosch for allowing me to have access to such a
great workplace and providing everything I could need to fulfil my goals for this research.
A special thank you to my coworkers with whom I have had the pleasure to work and that
always made me feel part of the team.
I would like to show my greatest appreciation to the experts that were involved in the validation
of this project without whom this research would not have had a happy ending.
My deepest heartfelt appreciation goes to my boyfriend João for putting up with me and my
terrible mood swings during this process and for always giving me the strength, love and motiva-
tion that I need to push through.
Finally, my acknowledgements would not be complete without expressing my profound grat-
itude to my parents for their relentless support and encouragement throughout my years of study
and for their unwavering love and patience through tough moments.
Thank you.
Mariana Oliveira
v
vi
“We can only see a short distance ahead,
but we can see plenty there that needs to be done.”
Alan Turing
vii
viii
Contents
1 Introduction 1
1.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Document Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Background 5
2.1 Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Requirements Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Requirements Validation . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 Requirements Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Software Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.1 Software Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.2 Verification and Validation . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4 Natural Language Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4.1 Statistical Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4.2 Phrase Structure Parsing and Probabilistic Context-Free Grammars . . . 15
2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
ix
CONTENTS
5 Experimentation 41
5.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
References 51
x
List of Figures
xi
LIST OF FIGURES
xii
List of Tables
xiii
LIST OF TABLES
xiv
Abbreviations
xv
Chapter 1
Introduction
Over the past decade there has been an increase in the complexity of software-intensive em-
bedded systems in the automotive domain [BBH+ 14]. With the automotive market being as
demanding as ever concerning the innovations of increasing complexity, the associated shorter
development periods due to the high pressure to market a product make it a necessity to push the
current software development processes to its limits.
One efficient method of ensuring high quality software in time and within budget is by ap-
plying software engineering (SE) processes to a project [Som10]. An important part of these
processes is the creation of artefacts, in particular requirements documents.
Requirements Engineering is one of the fundamental activities of SE and consists of under-
standing and defining what services are required from the system and identifying the constraints
on the system’s operation and development [Som10].
Low quality requirements can lead to misunderstandings which consequently may contribute
to errors in the design flow that are either hard to detect or detected too late [SAAA+ ].
Therefore, the quality of these documents must be assured and is directly connected to the
quality of the software systems as well as their development process. In fact, surveys have shown
that the most common reasons for a project’s lack of success are not technical but rather require-
ments related (13.11%) [Hul09].
Most of these problems arise because the requirements are written in Natural Language which
can lead to issues with clarity, consistency and ambiguity. When the requirements are written,
it is critical to ensure that they comply with relevant quality criteria and international standards.
Unfortunately, that is not always the case and that is what the verification and validation (V&V)
phase in requirements engineering is for.
During the V&V phase, processes such as reviews are used to check if the requirements doc-
uments abide by the quality criteria and standards related to those documents.
Although helpful in finding defects in the requirements, these reviews can be tedious because
reading and analysing each requirement individually takes time and requires a great effort to stay
1
Introduction
alert and remember relevant information. By not performing the requirements reviews adequately,
the time until new products are launched could be extended.
Existing static analysis tools mainly target code files, which are of no use when analysing
requirements and that is why there is a need for specific tools that analyse requirements. There are
some solutions available that for reasons of adaptability or privacy could not be used.
Throughout this investigation, a viable solution was developed by means of an integrated so-
lution in IBM’s Collaborative Lifecycle Management (CLM) tool called RANT (Requirements
ANalysis Tool) that makes use of Natural Language Processing (NLP) techniques. This solution
is complemented by a checklist that shall be used alongside RANT, both are thoroughly described
in 4.
1.1 Context
Bosch Car Multimedia is a division of Bosch which develops smart embedded solutions for
entertainment, navigation, telematics and driver assistance functions. They develop cutting-edge
features tailored to modern mobility requirements, providing optimum driving convenience, safety
and access to entertainment and information via smart networked architectures [Bos].
Their ambition is to enhance the quality of life with solutions that are both innovative and
beneficial. They focus on their core competencies in automotive and industrial technologies as
well as in products and services for professional and private use.
A huge part of creating such technology are the processes that take place throughout the de-
velopment of a project. Software Engineering processes are usually applied by companies in order
to produce quality software on time and within budget.
One of those processes is the elaboration of artefacts and their respective validation. In the
scope of this experiment, the Software Specification, i.e., the Requirements Engineering process
is the one on focus.
As quality is of such importance in the automotive industry, it is essential to develop soft-
ware fast but still abiding to certain quality criteria defined by a series of international standards
regarding this matter.
These criteria should be applied when the requirements are being written but that does not
always happen. Hence, the validation of the requirements is a crucial task to ensure quality in the
requirements and, consequently, of the software.
During the requirements review which is part of the validation process, analysts manually
go through each requirement trying to find mistakes and incoherences regarding the international
quality standards. The problem urges because this task takes up a lot of time and adding the
pressure to market a product to the necessity of having quality software, there comes a need for a
partially automated solution regarding this review process.
2
Introduction
3
Introduction
4
Chapter 2
Background
Being in an era where technology has such an increasing impact in our lives, it seems impos-
sible to imagine a world with no software, because it is everywhere around us: from our cars to
our smart TVs, going through medicine and industry, cleaning robots and smart virtual assistants.
In late 2013, a survey found that 23% of products now contain software in some form [Agi].
Software Engineering as defined in [Ref10] is the systematic application of scientific and
technological knowledge, methods, and experience to the design, implementation, testing, and
documentation of software; that is, the application of engineering to software.
Applying software engineering to a project is considered the most effective way of getting
software of high quality within schedule and budget [Som10].
Software processes are concerned with work activities accomplished by software engineers
to develop, maintain and operate software, such as requirements, design, construction, testing,
configuration management and other activities [BF14].
Four is the number of fundamental activities that are common to all software processes accord-
ing to [Som10]. The one on focus during this investigation is the Software Validation activity.
During the Software Validation activity the software is analysed to make sure it complies to
its specification and meets the needs of the stakeholders.
There are no universal software engineering methods that are applicable to all systems [Som10].
One of the most important points to consider when deciding which method to implement is the
type of software that is being developed: stand-alone or interactive transaction-based applications,
embedded control or batch processing or entertainment systems, modelling and simulation soft-
ware and data systems.
Embedded systems are software control systems that control and manage hardware devices.
Bosch Car Multimedia develops embedded solutions for entertainment, navigation, telematics and
driver assistance functions and they follow the REMsES (Requirements Engineering and Manage-
ment for software-intensive Embedded Systems) approach.
5
Background
6
Background
7
Background
• Validity checks: The functions proposed by stakeholders should be consistent with what the
system needs to do.
• Completeness checks: The requirements document should include all the requirements and
all the constraints.
• Realism checks: Make sure the requirements can be fulfilled using current existing technol-
ogy, within budget and schedule.
• Verifiability: It should be possible to write a set of tests that can prove that the delivered
system meets each requirement.
In order to validate the requirements, there are various techniques that can be used individually
or jointly: Requirements Reviews, Prototyping, Model Validation and Acceptance Tests [BF14].
Requirements reviews seem to be the most common means of both verifying and validating
requirements documents [BF14]. During this process, a group of reviewers analyse the require-
ments systematically looking for errors, mistaken assumptions, lack of clarity, verifiability issues
and deviation from standard practice [Sta11].
The output of a requirements review is a list containing reported problems with the require-
ments document and the actions necessary to cope with the reported issues [PEM03].
8
Background
The quality of requirements in the requirements document has a significant impact on the final
system. Requirements of poor quality can lead to misunderstandings and further errors in the
design that are usually difficult to detect or detected too late [SAAA+ ]. Therefore, it is paramount
to ensure the quality of requirements by defining requirements in a way that their validation is
objective, rather than a subjective opinion.
Requirements are usually written in Natural Language, meaning there is no defined format
by default, just normal plain text. Hence, requirements can be vague and ambiguous which calls
for the use of certain standards and guidelines to help make the requirements clearer, easy to
understand, complete and consistent.
Software standards are essential regarding software quality management because they capture
wisdom that is of value to the organisation, provide a framework for defining the meaning of
quality and they facilitate the continuity when work started by another person is picked up and
continued by another [Som10].
Concerning Requirements Quality, Bosch Car Multimedia follows the following international
standards:
• ISO/IEC 25010:2011 - Systems and software engineering — Systems and software Quality
Requirements and Evaluation (SQuaRE) — System and software quality models [ISO02a]
9
Background
Software quality problems were initially discovered in the 1960s with the development of the
first large software systems and they are still an issue nowadays [Som10]. Trying to solve this
situation and inspired by the methods used in the manufacturing industry, formal techniques of
software quality management were developed and adopted.
The definition of quality software is, basically, fitness for purpose, i.e., conformance to re-
quirements. It is providing something that satisfies the customer and ensures the needs of all the
stakeholders are considered [Hul09].
Quality Management as defined in [Ref10] is a set of coordinated activities to direct and con-
trol an organisation with regard to quality. Every management decision is a compromise between
cost, schedule and quality [Hul09].
Software Quality Management consists of four subcategories:
• Quality Planning
• Quality Assurance
• Quality Control
• Quality Improvement
Quality Planning involves deciding which quality standards will be used, defining quality goals
and doing an estimation on the effort and schedule of the quality activities.
Quality Assurance incorporates various steps that define and assess if the software processes
are appropriate and that they produce software products of suitable quality for their intended pur-
poses [BF14].
As for Quality Control, the project documents are analysed and executables are examined to
determine whether they follow the standards set for the project.
Regarding the Quality Improvement category, it is sought to improve process effectiveness,
efficiency and other characteristics with the objective of improving software quality [BF14].
Quality Assurance as stated in [Ref10] is a set of planned and systematic activities imple-
mented within the quality system and demonstrated as needed, to provide adequate confidence
that an entity will fulfil requirements for quality.
The purpose of the Quality Assurance Process is to provide independent and objective as-
surance that work products and processes comply with predefined provisions and plans and that
non-conformances are resolved and further prevented [VDA15].
10
Background
The Software Quality Management (SQM) process is significant when it comes to assuring
quality in software. This process checks the project deliverables to make sure that they are consis-
tent with organisational standards and goals [Som10]. Part of this SQM process is the Verification
and Validation procedure.
The purpose of Verification and Validation (V&V) is to aid an organisation to achieve quality
in their system during the life cycle. V&V processes grant an unbiased evaluation of products
throughout the life cycle. This evaluation shows whether the requirements are correct, complete,
accurate, consistent and testable [BF14].
Verification is an attempt to ensure that specified requirements have been fulfilled and that they
satisfy the standards, practices and conventions during the life cycle processes. Validation is the
confirmation that the requirements for a specific intended use or application have been fulfilled
and that they solve the right problem [ISO02b].
Both verification and validation are interrelated and complementary processes that use each
other’s process results to establish better completion criteria and analysis [10112]. They take
place early in the development or in the maintenance phase. As the output, the V & V plan
documents describe the various resources and their roles and activities as well as the techniques
and tools to be used [BF14].
Part of the V & V activities involves checking processes, such as reviews and inspections,
at each stage of the software process from user requirements definition to program development
[Som10] as illustrated in Fig. 2.3.
Requirements reviews are possibly the most used method of both verifying and validating a
requirements document. Reviews and inspections are used side by side with program testing as
part of the V & V processes.
11
Background
In order to arrange a requirements review, a group of reviewers is constituted with the purpose
of looking for errors, mistaken assumptions, lack of clarity, verifiability issues and deviation from
standard practice [Sta11]. Generally, the review process consists of three phases: pre-review
activities, the review meeting and the post-review activities as shown in Fig. 2.4.
Inspection as defined in [IEE08] is a visual examination of a software product to detect and
identify software anomalies, including errors and deviations from standards and specifications.
As part of the inspection process, a checklist with the most ordinary errors is used to help the
inspectors focus during the search for bugs [Som10].
Inspections and reviews are part of the so-called static V & V techniques. These techniques ex-
amine software documentation, including requirements, interface specifications, designs, models
and source code without executing the code [BF14].
Inspections and reviews have a very positive outcome when it comes to discovering software
errors. Yet, they demand a lot of time to organise and often lead to delays into the development
process [Som10].
12
Background
The Lexical Analysis purpose is to interpret the meaning of individual words and has five
main techniques that can be used: Sentence splitting, Tokenization, Part-of-Speech (PoS) tagging,
Morphological Analysis and Parsing.
Sentence splitting is the process of breaking the text into separate sentences. During this
process the Natural Language text is analysed to determine the sentence boundaries between the
sentences.
Most languages use punctuation marks to indicate the boundaries between sentences. There
are, however, some instances were punctuation marks are not used for indicating boundaries. For
instance, with abbreviations and titles punctuation marks are used, which do not indicate a sen-
tence boundary.
Tokenization splits the sentence into its meaningful units, named tokens. Based on the struc-
ture of the text, which is partly provided by the sentence splitting, the tokens are associated to a
category. The most common categories are words, numbers, punctuation marks and symbols.
The Part-of-Speech tagging process is responsible for tagging each token with its grammatical
category, based on its definition and context. Each token is then identified with a tag, such as noun,
verb, adjective or determiner [Are16].
The Morphological Analysis is the preliminary stage that takes place before syntactic analysis.
The purpose of this stage is to identify the root of compound words. This can be accomplished by
using stemming and lemmatization. Stemming is a technique that reduces an inflected word to its
stem, usually by removing their suffixes and Lemmatization is a technique that finds the root form
of a word [Are16].
13
Background
Figure 2.6: Dependency Graph [SAAA+ ] Figure 2.7: Phrase Structure Tree [SAAA+ ]
Parsing is a technique that consists in analysing a sentence by taking each word and determin-
ing its structure from its constituent parts. In order to parse a piece of text, it is necessary to have
two components: a parser and a grammar. The grammar for natural languages is ambiguous and
typical sentences have multiple possible analyses [BSPM16].
There are two primary types of parsing: Dependency Parsing and Phrase Structure Parsing.
The first one focuses on the relations between words in a sentence as shown in 2.6 while the sec-
ond one focuses on building the Parse Tree, usually using a Probabilistic Context-Free Grammar
(PCFG) as it can be seen in 2.7.
The output of the Lexical Analysis is the input to the Syntactic Analysis. This process performs
an analysis of the words in a sentence in order to reveal the grammatical structure of the sentence.
This requires both a grammar and a parser. The output of this level of processing is a representation
of the sentence that reveals the structural dependency relationships between the words [D.L01].
Semantic processing determines the possible meanings of a sentence by focusing on the inter-
actions among word-level meanings in the sentence [D.L01]. It builds up a representation of the
objects and actions that a sentence is describing and includes the details provided by adjectives,
adverbs and propositions [RMP13].
Categorisation aims at automatically nominating new documents to categories that are already
established [YT06]. In Requirements Engineering, categorising deals with classifying require-
ments for a certain purpose which can be helpful for the development of software. Classified
requirements can be assigned to teams that each focus on a particular class of requirements.
14
Background
The statistical approach to NLP has become more and more important in recent years though
its application started back in the 1980s [YT06].
Statistical parsing consists in methods for syntactic analysis that are based on statistical in-
ference from samples of natural language text [YT06]. Statistical inference may be applied for
various features of the parsing process but is mainly used for disambiguation.
A possible solution to the problem is a probabilistic parser as it computes the probability of
each interpretation and chooses the most probable one [PES17].
The set of possible syntactic representations is usually defined by a particular theoretical
framework but normally takes the form of a complex graph or tree structure. The most common
type of representation is a phrase structure [YT06].
The idea behind a statistical parser is that it assigns sentences in natural language to their
favourite syntactic representations, either by providing a ranked list of possible analyses or by
selecting a single optimal analysis [YT06].
There are two types of probability models: Discriminative and Generative.In a Discriminative
model, the condition probability P(y|x) is modelled [YT06]. In the Generative model what is
modelled is the conditional probability of an input x given a certain label y.
The simplest Generative Statistical Parsing model is the Probabilistic Context-Free Grammars
(PCFGs).
Phrase Structure Parsing focuses on identifying phrases and their recursive structure originat-
ing in a Phrase Structure Tree (PST).
A PST contains structural information about a sentence where the root node represents the
whole sentence and the non-terminal nodes represent the syntactic grammar structure in terms of
constituents, while the terminal nodes are the atomic words of the sentence [SAAA+ ].
The analysis of the sentence and annotation into a PST is performed by structural parsers such
as the one contained in the Stanford CoreNLP natural language processing toolkit [Sta17].
Natural Language can be ambiguous which can lead to originating multiple PSTs for the same
sentence. How the PSTs are generated depends on the grammar employed by the parser.
The simplest augmentation of the context-free grammar is the Probabilistic Context-Free Gram-
mar (PCFG). Besides the elements contained in a context-free grammar, the set of terminal sym-
bols, nonterminal symbols, the start symbol and a set of rules, PCFGs add a probability function.
As shown in Algorithm 1, this probability function takes each grammar rule and and associates
it with the probability value of each rule [Tha17].
A PCFG can be used to estimate a number of useful probabilities concerning a sentence and
its parse-tree(s) which can be useful in disambiguation [JM09].
15
Background
The probability of an ambiguous sentence is the sum of the probabilities of all the parse trees
for that sentence [JM09]. The PST with the highest PCFG score has the best probability of being
the correct one.
2.5 Conclusion
Applying software engineering to a project is paramount in order to get high quality software
within schedule and budget. The focus of this investigation is on one of the four main activities of
software engineering: Requirements Engineering.
Following Requirements Engineering processes increases the chances that user expectations
are met and the software is of high quality. Mistakes made during this stage can be harmful in the
future if they are not found in time and corrected.
Requirements Engineering also has four fundamental sub-activities. This research is based on
the Requirements Validation stage.
Requirements documents shall be subject to validation and verification processes. These are
important to make sure errors are corrected so there is less probability of having to do extensive
and costly rework ahead in the project development. One of the most used techniques to validate
requirements are the Requirements Reviews.
As requirements are usually written in Natural Language and due to its ambiguous nature, it
is not an easy task to evaluate their quality so it is important that requirements are written in a
specific way. That is when international standards and guidelines come to aid.
Software Quality Assurance is one of the sub-categories of Software Quality Management
which is responsible for providing independent assurance that the products and processes comply
with the predefined plans and that non-conformances are solved.
Verification and Validation (V&V) methods such as Reviews and Inspections are some of
the various processes of Quality Assurance. Besides being one of the most common means of
validation, reviews are a time consuming and very tedious task.
As most Requirements Documents are written using Natural Language, in order to be able to
partially automate the process of reviewing requirements, it is necessary to process those require-
ments using Natural Language Processing (NLP) techniques.
16
Background
The one on target during this research is the Statistical approach. This approach computes the
probability of each interpretation and chooses the most probable one. The most ordinary type of
representation is a phrase structure and the simplest statistical parsing model is the Probabilistic
Context-Free Grammars (PCFGs).
17
Background
18
Chapter 3
In this section, some of the existing NLP tool kits and libraries will be described, namely their
main functionalities as well as their structure.
The Stanford CoreNLP Toolkit is an NLP toolkit developed by The Stanford Natural Language
Processing Group. It offers Java-based modules for the solution of a range of basic NLP tasks as
well as the means to extend its functionalities with new ones.
CoreNLP is an annotation pipeline framework which provides most of the common core NLP
processes. Some of those are Tokenization, Sentence Splitting, Part-of-Speech tagging, Lemmati-
zation and Syntactic Analysis based on a probabilistic parser.
The overall system architecture of CoreNLP, as illustrated in 3.1, consists on the input of raw
text into the Annotation Object, then a sequence of Annotators add information in the analysis
pipeline which then results in an Annotation object containing all the analysis information added
by the Annotators [MSB+ 14]. This result can be in XML format or plain text.
The annotators provided in Stanford CoreNLP work with any character encoding, the default
being the UTF-8 encoding. While most users employ the annotators already present in the toolkit,
it is also possible to add additional custom annotators to the system [MSB+ 14].
Stanford CoreNLP can be used directly from the command-line, via its original Java API,
via the object-oriented simple API, via third party APIs for most major modern programming
languages like JavaScript or Python or via a web service.
19
State of the Art
The Natural Language Toolkit (NLTK) is a suite of program modules, data sets, tutorials and
exercises, covering symbolic and statistical natural language processing [BL]. It is written in
Python and is distributed under the GPL open source license.
The NLTK provides simple interfaces to over 50 corpora and lexical resources alongside a se-
ries of libraries for classification, tokenization, stemming, tagging, parsing and semantic reasoning
[NLT18].
The aim of the NLTK is to assist research and teaching in NLP areas such as empirical linguis-
tics, cognitive science, artificial intelligence, information retrieval and machine learning [BL].
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural
language [Apa].
The Apache OpenNLP toolkit consists of several components making it possible to build a
full natural language processing pipeline, namely a sentence detector, a tokenizer, a name finder,
a document categoriser, a PoS tagger, a chunker and a parser.
Components contain parts which enable one to execute the respective natural language pro-
cessing task, to train a model and often also to evaluate a model. Each of these facilities is acces-
sible via its application program interface (API). In addition, a command line interface (CLI) is
provided for convenience of experiments and training [Apa].
20
State of the Art
3.1.5 spaCy
spaCy is a library for advanced Natural Language Processing in Python and Cython. It comes
with pre-trained statistical models and word vectors and currently supports tokenization for more
than twenty languages. It features the fastest syntactic parser in the world, convolutional neural
network models for tagging, parsing and named entity recognition and easy deep learning integra-
tion [spab]. It is commercial open-source software, released under the MIT license.
Unlike NLTK, which is widely used for teaching and research, spaCy focuses on providing
software for production usage.
spaCy provides the means for the usual NLP techniques such as tokenization, lemmatisation,
PoS tagging, entity recognition, dependency parsing, sentence recognition, word-to-vector trans-
formations and many methods for cleaning and normalising text [spaa].
While some of spaCy’s features work independently, others require statistical models to be
loaded, which enable spaCy to predict linguistic annotations – for example, whether a word is
a verb or a noun. spaCy currently offers statistical models for eight languages, which can be
installed as individual Python modules [spab].
When the NLP model is loaded and then called on a text, spaCy first tokenizes the text to
produce a Doc object. The Doc is then processed in several different steps – this is also referred to
as the processing pipeline. The pipeline used by the default models consists of a tagger, a parser
and an entity recogniser. Each pipeline component returns the processed Doc, which is then passed
on to the next component as it can be seen in Fig. 3.2.
The architecture of spaCy has essentially two central structures: Doc and Vocab [spab]. As
shown in Fig. 3.3, the Doc object owns the sequence of tokens and all their annotations and the
Vocab object owns a set of look-up tables [spab]. The Doc is made by the Tokenizer and then
modified by the components of the pipeline. The Language object takes raw text and sends it to
the pipeline and then returning an annotated document.
21
State of the Art
22
State of the Art
Three types of user glossaries are used to parse requirements documents: agent glossary, action
glossary and modal word glossary [VK08]. It implements a deterministic finite automata based
approach to parse the requirements, extract structured content and generate error messages.
23
State of the Art
RAT has been created as an extension to Microsoft Office and can be installed as a plugin for
Word and Excel. Office libraries in .NET are used to access the textual requirements and analyse
them [VKV14].
3.2.2 QuARS
The Quality Analyser for Requirements Specifications (QuARS) is a research tool that aids in
the creation of quality requirements by allowing he user to perform an initial parsing of require-
ments by automatically detecting potential linguistic defects that can cause ambiguity problems at
later stages of the software product development [Lam05]. QuARS also provides support for the
consistency and completeness analysis of the requirements [Lam05].
The high-level architectural design of the QuARS tool can be seen in 3.5.
The input is a requirements document in plain text. This file is passed on to the syntax parser
which then produces a new file containing the parsed version of its sentences. This tool relies on
a set of indicator-related dictionaries which must also be in plain text [QuAa]. The outputs of the
tool include log files with the indications of the sentences containing defects and the calculation
of metrics about the defect rates of the analysed document [Lam05].
NASA’s Software Assurance Technology Centre (SATC) developed a tool around the late
90s that automatically analyses a requirements document and produces a detailed quality report
[CL14]. This report is based on a statistical analysis of word frequencies at many structural levels
of the document.
24
State of the Art
Figure 3.6: Overall Smell Detection Process of Qualicen Scout [FFJ+ 14]
Qualicen Scout is a real-time quality analysis and visualisation tool for requirements and tests
written in natural language [quab]. This tool features several different analysis to identify quality
issues in the requirements also named requirement smells. The difference between a requirement
smell and a defect is that a smell is only an indication for a possible quality defect [Are16].
Scout detects long and complicated sentences, usage of passive voice, multiple negations,
vague phrases and pronouns, comparatives and superlatives, usage of a slash, duplicate require-
ments and, if required, Qualicen Scout can also perform structural analysis [quab].
The tool provides users with warning messages which a short description whenever a smell is
detected. Some smells use NLP techniques such as morphological analysis and PoS tagging. The
process of detecting requirements smells consists of four steps (Fig. 3.6): Parsing, Annotation,
Smell Identification and Presentation.
Qualicen Scout integrates with some current tools like Qualicen PTC Integrity LM, Microsoft
TFS, Visual Studio Team Services, GIT and SVN [quab].
3.2.5 QVscribe
QVscribe is a requirements analysis tool created by QRA Corp [QRA]. It combines natural
language processing (NLP) with an expert system. QVscribe analyses requirements and alerts the
author of ambiguous, overly complex and essentially malformed engineering requirements [QVs].
25
State of the Art
QVscribe performs syntactic and semantic analysis of requirements for quality and consis-
tency.
Once QVscribe is installed, it appears as a toolbar icon. It is used as a add-in tool for Microsoft
Word, Visure Requirements and Marinvent’s Synthesis [QRA].
In order to start using QVscribe, first it is necessary to specify the location of the requirements
in a document, then specify the acceptable terminology to be used in the requirements and identify
specific parts of the requirements that should not be analysed [QRA]. After these steps, QVscribe
can auto-find the requirements present in the document.
The analysis provided by QVscribe is based on eight quality measures: Imperatives, Negative
Imperatives, Options, Weaknesses, Vagueness, Subjectiveness, Continuances and Directives. The
analysis can also be customised to match a company’s best practices for requirements documenta-
tion.
The results of the analysis are displayed in a simple, interactive scorecard which indicates the
quality of each requirement by number of stars (one to five) - an example of QVscribe in Microsoft
Word can be seen on Fig. 3.7.
Clicking on a requirement listed in the scorecard takes the engineer to the requirement in the
document. There, the engineer can see highlights showing the quality indicators that triggered the
given score for the requirement [QRA].
26
State of the Art
3.3 Conclusion
A tool that performs static analysis on requirements is a huge help when it comes to assuring
the quality of the requirements documents.
Libraries and toolkits such as the Stanford CoreNLP, the Natural Language Tool Kit (NLTK),
the Apache OpenNLP and spaCy are paramount when developing a tool that analyses require-
ments documents. Most of these perform standard NLP techniques such as tokenization, PoS
tagging, sentence splitting which is of great help when it comes to detecting possible defects in
requirements.
With that in mind, several companies have come up with solutions that are able to provide
that needed assistance. Although the offers on the market for requirements analysis tools are
scarce, there are some which can be of help to this research. Some example of such tools are
the Requirements Analysis Tool (RAT), the Quality Analyser for Requirements Specifications
(QuARS), Automated Requirements Measurement (ARM), the Qualicen Scout and QVscribe.
Despite being useful for this investigation, they are not a viable solution for this particular
case. Adaptability is an issue in the case of RAT, Qualicen Scout and QVscribe because they are
available as extensions to several platforms such as Microsoft Word and Marinvent’s Synthesis
and can not be integrated in the intended environment - CLM.
Accessibility is also an obstacle, specifically in the case of ARM and QuARS, because these
tools are not available for use, either because they are not available anymore (ARM) or because
they were created as academic research tools (QuARS). Some of these tools demanded that the
requirements were written in a determined structure and not just plain Natural Language text as
expected in this investigation.
However, the analysis provided by these tools is very detailed and advanced, providing a
deeper evaluation of the requirements documents. That leaves the reasons mentioned before as the
only setbacks of using them in this specific case. Were they to create an extension for IBM’s CLM
and doing some adjustments to the input type, they could be viable solutions.
27
State of the Art
28
Chapter 4
Analysing requirements is a complex task that requires a considerable amount of time and
resources.
For that reason, it was critical to create a solution that not only helps the reviewers during that
process but is also integrated in the requirements management tool already in use.
In this chapter, the challenges faced during this investigation are carefully outlined, alongside
the approach taken in order to work on those challenges.
The tool developed for this project is an extension of IBM’s Collaborative Lifecycle Manage-
ment (CLM) tool that is used in the Requirements Management (RM) environment. A compre-
hensive description of this tool regarding its architecture and features can also be found in this
chapter.
The quality of requirements in textual specification documents has a significant impact on the
final product [VAD09]. As a matter of fact, various studies have shown that fixing engineering
errors in software projects can become exponentially costly over the project life cycle [HDS+ 04].
Furthermore, it is estimated that more than half of those errors are originated in the require-
ments [RR]. Hence, it is imperative that errors are found and corrected during the requirements
stage, exactly when they occur and not later in the project’s life cycle.
Defects in the requirements such as ambiguity, lack of readability or inconsistency can lead
to delays when launching a product, when that could be avoided by simply finding and correcting
issues in the requirements in due time.
The task of finding errors in the requirements has been handed to software engineers and
analysts who manually go through the requirements one by one trying to detect possible mis-
takes. Though, as human beings, sometimes errors can slip by because of how tedious and time-
consuming this task is.
29
Proposed Solution - RANT
Considering that the market is as challenging as ever, it is paramount that time is not lost,
especially analysing and correcting requirements when software could have been developed and a
product launched.
As nearly 90% of the requirements documents are written in Natural Language, the existent
static analysis tools are not fit for that purpose [SARW12].
Most of the tools available in the market are not for requirements analysis but rather code
syntax checkers, debuggers or static analysis tools designed to find errors in the software and not
in the requirements.
Nevertheless, there are some tools, as mentioned in Chapter 3, that are suitable for require-
ments analysis but either require them to be written in a specific form, are not integrated in the
tool that is being used, are not freely available or are not ready for use in a non-academic environ-
ment.
Consequently, the need for a partial automation of the requirements review process, together
with the lack of applicable solutions, led to the urgency to create a tool that checks the intended
boxes.
What makes this work challenging is the Natural Language Processing (NLP), since most
requirements are written in Natural Language as it is easy to understand by all the people involved
in the project. On the other hand, it can be ambiguous, inaccurate and inconsistent.
Another challenge regarding this project is related to the fact that it needs to be integrated in the
current requirements management tool being used by Bosch Car Multimedia: IBM’s Collaborative
Lifecycle Management (CLM).
The aim of this investigation is to develop a tool that comes to aid during requirements reviews
and is integrated in the software being used. The intention is to partially automate the process of
analysing requirements in order to decrease the time spent as well as increasing the number of
defects correctly identified in those reviews.
IBM’s CLM tool offers a section dedicated to the requirements elaboration, analysis and re-
view processes called Requirements Management (RM). In order to be integrated in this environ-
ment and being able to access the requirements, an extension was the most reasonable solution.
Being a task that demands the fullest attention to detail, the requirements review shall be made
with the help of not only the tool but also a checklist that was elaborated during this research as
well.
Even though this tool together with the checklist provide insight on whether or not the require-
ments have issues, they are not supposed to be used on their own, but rather as a complement to
the analysis of the developer.
Towards partially automating this process, first it was essential to comprehend which inter-
national quality criteria should be applied to the requirements. Then, a checklist was created to
serve as a guideline for the tool itself. Only at that point did the development of the Requirements
ANalysis Tool (RANT) begin as an aid tool that makes use of NLP techniques and is integrated in
the CLM application.
30
Proposed Solution - RANT
RANT aims at helping the reviewers achieve a defect free requirements document by analysing
selected requirements and detecting if they comply with their expected structure as well as possible
ambiguous words and expressions present in the sentence and alerts the user to that fact.
4.2 Checklist
There are various guidelines that help when writing good requirements which can either be
provided globally, such as international standards, or internally, as an agreement between a com-
pany and its clients [SAAA+ ].
A checklist was created with the main goal of being used together with RANT, in the interest
of complementing the review process performed by the analyst. It was elaborated at the beginning
of this investigation so it could serve as a guideline to the development of RANT.
In order to gather the needed items for this checklist, a research was done regarding Bosch’s in-
ternal quality criteria as well as international quality standards. The structure and the explanations
given in the checklist were found in [KP15].
Good requirements should all follow the same set of criteria [Cor]. With that in mind, the
checklist that was elaborated not only contains a list of criteria but also a group of simple guide-
lines, accompanied with practical examples.
As it can be seen in Fig. 4.1, there is a list of criteria that should be fulfilled:
• Completeness
• Correctness
• Clearness
• Consistency
• Feasibility
• Verifiability
• Traceability
Each one of these criteria shall be applied when writing and reviewing a requirement. To have
a good requirement, it is essential to check if each requirement is valid against each item of this
list.
Furthermore, there is a list of items that shall be checked when reviewing the requirements,
which helps achieving those quality criteria. An explanation of each quality criteria can be found
at the end of the checklist as depicted in Fig. 4.2.
31
Proposed Solution - RANT
32
Proposed Solution - RANT
33
Proposed Solution - RANT
• An XML file
• A JavaScript file
• A CSS file
The XML file is used when adding RANT to the CLM dashboard. It contains all the necessary
information to run the tool, namely the JavaScript and CSS files used to build RANT.
The access to the data stored in the CLM tool, specifically in the RM environment, is only
possible due to the RM API.
The RM API is a service-centric mechanism for accessing and modifying data in the system
and for interacting with mechanisms in the local client, such as showing user interface elements
[RM].
Generally, the services are accessed via the RM object and its child objects, all of which are
provided as part of the RM feature that is loaded into each extension.
The JavaScript file contains all the functions necessary to perform calls to the RM API. These
functions allow access to the requirements and their respective information.
Some examples are the RM.Event.subscribe function, which subscribes RANT to an event
and until the extension is closed or unsubscribes from the event, the given callback is invoked
whenever the event occurs; another example is the RM.ArtifactAttributes function that associates
a set of attribute values with an artefact; other examples along with their description are available
in [RM].
Besides that, the requests done to RANT’s back-end structure are also contained on that single
JavaScript file.
In order to know when a requirement or a group of requirements is selected, the JavaScript file
subscribes the notifications of the RM API for when that action occurs.
When a requirement is selected, it is possible to gather data about the requirement such as the
ID, the Primary Text, who created it, who changed it and when, its priority, its status, among other
details.
With all that information, only the relevant data concerning the analysis of the requirement
is selected, namely the ID and the Primary Text, which will then be sent to RANT’s back-end
structure - RANT_BE.
34
Proposed Solution - RANT
What occurs is the RANT_FE takes that relevant data and sends it to the RANT_BE. This is
done via an HTTP request to the Node JS Server that is responsible for the back-end structure
as well as the connection to the NLP component. Then, it stands by awaiting the result from the
analysis performed.
Using Node’s module Python-Shell, which allows the communication between JavaScript and
Python, RANT_BE is able to pass the data on to a Python script and then just waits a response.
From there, the script runs the analysis, which will be thoroughly detailed in Section 4.5, using
spaCy’s NLP functionalities and returns a result. This component responsible for the NLP is called
RANT_NLP.
When RANT_NLP has a result from all the analysis performed, RANT_BE is notified and it
is sent back to RANT_FE also via HTTP.
The analysis from RANT is then showed on CLM’s dashboard on the left along with the
respective ID and primary text.
35
Proposed Solution - RANT
To be able to use RANT, first it is necessary to add a link of the XML file, that should be stored
and running on the same server as CLM, to CLM’s Widget Catalogue.
It is a simple process that is achieved by clicking Add Widget on the dashboard on the left side
of the RM environment and RANT will then appear there.
A server has to be running the CLM software, where the catalogue is found, along with
RANT_FE.
As seen in the example figure 4.4, RANT is seamlessly integrated in RM and can be used
alongside other functionalities of the framework.
When requirements are selected, RANT performs the analysis and the results are returned in
the form of a table, where details from the requirement are shown: the ID, the text and the result
from RANT’s analysis.
Every time a requirement is selected, a new table with the respective result appears below the
others. In the same way, when a requirement is deselected, the table regarding that requirement
disappears.
As RM allows the user to select all requirements present in the page, RANT is also prepared
to present the results of its analysis, independently of the quantity of requirements selected.
36
Proposed Solution - RANT
If a requirement is not following the complete intended structure, RANT alerts the user to the
specific case, i.e., if the subject is missing, RANT alerts that the subject is missing and the same
occurs for when the verb or the direct object are missing.
Besides, RANT uses dictionaries with a selection of words and expressions that might lead
to a requirement being vague and ambiguous and runs an analysis that detects if such word or
expression is found in the requirement. Some examples of these words can be found in Fig. 4.5.
37
Proposed Solution - RANT
The second item on the checklist aims to make sure there is only one requirement specified
per sentence. In RANT, this situation can be checked by detecting the use of conjunctions such as
and, or, with, etc.
Just like when analysing the requirements structure, RANT alerts the user for specific cases,
varying with the different types of words and expressions used.
If conjunctions are used, RANT warns the user and reminds that conjunctions should be
avoided because they can add unnecessary complexity to a requirement.
In the case of using let-out clauses, RANT alerts that they can compromise consistency and
should be averted.
Using words or expressions that impose suggestions or possibilities will also cause RANT to
notify the user because the use of these can jeopardise clarity.
Terms that are not well defined, such as appropriate, normal or user-friendly, will also cause
RANT to tip off the user about their use due to that fact that these terms induce ambiguity.
Lastly, RANT detects if words or expressions that are speculative or lead to wishful thinking
are used and notifies the user that these should be avoided to ensure feasibility.
The last three items present on the checklist are not yet able to be performed by RANT, mean-
ing that the reviewer must interpret the requirement and extract the necessary information to make
sure those items are checked.
38
Proposed Solution - RANT
39
Proposed Solution - RANT
40
Chapter 5
Experimentation
• a True Positive (TP) is when there is a defect in a requirement and it was detected.
• a False Positive (FP) is when it indicates the requirement has a defect but that is not true.
• a True Negative (TN) is when there is no defect and no defect was detected.
• a False Negative (FN) is when there is a defect in a requirement but it was not identified.
Using these values, it is possible to calculate the Precision, Recall and Accuracy values of the
tool through the formulas (5.1) and (5.2), respectively.
TP
Precision = (5.1)
T P + FP
TP
Recall = (5.2)
T P + FN
TP+TN
Accuracy = (5.3)
T P + T N + FP + FN
41
Experimentation
Precision as defined in [Tin10] is a ratio of true positives and the total number of positives
predicted by a model which translates to out of all the requirements the tool identified as having
defects, how many actually had them.
Recall as defined in [Tin10] is the ratio of true positives and the total number of actual positives
which means that, out of all the requirements that actually have defects, how many were detected.
Besides those values, it is also possible to calculate the accuracy of the tool, using formula
(5.3). The accuracy value measures how well a binary classification test correctly identifies a
condition. In this case, the accuracy is the proportion of true results among the total number of
cases tested.
In order to obtain all of these values, it was decided to conduct tests which consist in providing
the tool a set of 65 dummy requirements provided by Bosch, which are representative of real
requirements, and then getting the analysis from the tool.
After that, this analysis was given to a group of 10 expert reviewers that belong to different
teams inside Bosch. They were asked to individually provide their opinion on the analysis of the
tool regarding that set of requirements: if they agreed or disagreed with the analysis done by the
tool. If there was a situation where the experts did not agree with the tool, it was asked for them
to specify why in a comment section. The results and its conclusions can be found in 5.2.
The results in detail can be found in Table 5.1. There, the verdict of each reviewer about each
requirement can be found, (A) for Agree, (D) for Disagree as well as the analysis from RANT,
(Yes) when a defect was found and (No) when no defect was found.
In order to obtain a final verdict for each requirement, the majority of a certain verdict was
considered and the final result: True Positive (TP), False Positive (FP), True Negative (TN) and
False Negative (FN) is the translation of that value.
In most cases, there was a consensual verdict of either Agreement or Disagreement. However,
there were 29 cases were a consensus was not reached. 16 is the number of cases where only
one reviewer disagreed with the rest, 11 cases where 2 reviewers disagreed and only 2 cases of 3
reviewers disagreeing.
As such, all of the final results have a certainty percentage of at least 70% (7 out of 10 review-
ers).
From a total of 65 requirements analysed, 36 were True Positives and 19 were True Negatives.
Regarding the False Positive results, which were 8, the majority of them was due to the use of
conjunctions that could induce complexity but it was not considered a problem by the experts.
In terms of False Negatives, there were only 2 cases and their reason is purely semantic, i.e.,
the problem was with the meaning of the sentence, which is out of scope for RANT.
42
Experimentation
43
Experimentation
Concerning the False Positive results, RANT detects conjunctions and expressions that can
lead to problems when interpreting the requirements. However, there are cases where the conjunc-
tions are essential for the meaning of the sentence. In that case, RANT can’t distinguish between
both cases and alerts the user which happened in all False Positive hits.
The usage of conjunctions such as and, or and with were wrongly classified as defects, ac-
cording to the experts in 7 out of 8 FP and the usage of the word when formed a let-out clause that
could lead to problems with consistency but was disregarded by the experts.
An example of such situation is the following requirement The VMPS shall receive a frame
with CAN ID Value equal to 1003. RANT detected the use of the conjunction with and alerted the
user for a possible complexity addition. However, the reviewers all agreed that it was necessary
for this specific case and that there was no defect.
Another example occurred with the requirement The VMPS shall respond only when a message
is received. The result from RANT detected the use of a let-out clause expression when that could
compromise consistency but was considered paramount in this case.
A third example detected both and and or conjunctions in the requirement The VMPS shall
encode the received alphanumeric or space characters, according to the International Morse Code
encoding and return the encoding on signal Coded_Tx_Payload, alerting the user about adding
unnecessary complexity to the requirement but was also deemed important to the meaning of the
sentence.
The results are shown in Fig. 5.1 regarding True Positive (TP), False Positive (FP), True
Negative (TN) and False Negative (FN) values. By using these, it is possible to calculate RANT’s
precision, recall and accuracy values which are, respectively, 0.818, 0.947 and 0.846.
44
Experimentation
In general, RANT’s results appear to be promising though there are no previous studies re-
garding other similar requirements analysis tools to use for comparison.
However, RANT has a percentage of approximately 12% of False Positive results while static
analysis tools aim at a 30% maximum so users don’t give up on using the tool and 20% as the goal
to achieve, thus these results are very encouraging [BEB+ 10].
In general, the opinion of the experts is positive regarding RANT’s usability in the review
process.
They claimed that, even alerting the user about a few cases that were disregarded as issues,
RANT could facilitate their work by saving some time and finding defects that could otherwise
pass by.
45
Experimentation
46
Chapter 6
An appreciation of the achievements that were accomplished during this work along with the
plans for the continuity of this project are made in this chapter. Additionally, directives towards
possible future work regarding improvements to the developed tool are also stated.
6.1 Achievements
Requirements engineering is a paramount step in every software project as, if done correctly,
can lead to quality software and prevent future rework and respective costs.
In order for that to happen, it is necessary to assure that the requirements are well-written,
complete, clear and consistent by means of reviews.
Reviews consist of a group of analysts going through each requirement manually, verifying if
they are written according to the international standards criteria and internal rules as well.
It is a very time and resource consuming task that benefits from being partially automated so
that it takes less time while finding all possible defects in the requirements.
Being written in Natural Language, the requirements pose a challenge during its analysis as
it can be vague and easily misunderstood by the different people that read it, from clients to
developers to analysts.
With that aim, RANT (Requirements ANalysis Tool) was developed as an extra help for the
analysts to be used together with a checklist.
RANT is a tool that employs a series of Natural Language Processing (NLP) techniques to
perform analysis on requirements and alerts the user to possible issues.
Additionally, RANT is integrated in Bosch’s Requirements Management tool CLM, which
was a core goal of this work.
47
Conclusions and Future Work
Regarding the approach to the semi-automation of the review process, the results obtained by
RANT were very promising with an accuracy value of 84.6%, a precision value of 81.8% and a
recall value of 94.7%.
Both main goals were accomplished resulting in a useful tool that is capable of providing
encouraging results when looking for defects in requirements while being fully integrated in the
currently used Requirements Management tool.
The feedback from the experts that validated RANT’s results was also positive. They claimed
that even with that percentage of False Positive and False Negative results, there is potential in the
tool to be used by them during the review process, alongside the checklist.
In conclusion, RANT has obtained promising results that, together with the reviewers assess-
ment, lead to the belief that RANT has the capability to be used by requirements experts to aid
them during the review process and provide useful feedback.
48
Conclusions and Future Work
As future work, a CNN would be applied to the existing RANT tool so that it would learn
how a quality requirement should be by means of training and testing with a large amount of data,
making it able to better detect defects in the requirements in a shorter period of time.
By using a CNN it would be possible to develop self improving algorithms that would analyse
the network in order to identify which are the patterns that indicate the requirement is of quality
and which are the common pitfalls.
49
Conclusions and Future Work
50
References
[10112] IEEE Std 1012-2004. IEEE Standard for System and Software Verification and Valida-
tion IEEE, volume 2012. 2012.
[Are16] Brian Arendse. A thorough comparison of NLP tools for requirements quality improve-
ment. Master’s thesis, 2016.
[BBH+ 14] Peter Braun, Manfred Broy, Frank Houdek, Matthias Kirchmayr, Mark Müller, Birgit
Penzenstadler, Klaus Pohl, and Thorsten Weyer. Guiding requirements engineering
for software-intensive embedded systems in the automotive industry: The REMsES
approach. Computer Science - Research and Development, 29(1):21–43, 2014.
[BEB+ 10] Al Bessey, Dawson Engler, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth
Hallem, Charles Henri-Gros, Asya Kamsky, and Scott McPeak. A Few Billion Lines
of Code Later. Communications of the ACM, 53(2):66–75, 2010.
[BF14] Pierre Bourque and Richard E. Fairley. Guide to the Software Engineering - Body of
Knowledge. 2014.
[BL] Steven Bird and Edward Loper. NLTK: The Natural Language Toolkit.
[BSPM16] Euiwhan Kim Gi-Name Wang (eds.) Bhabani Shankar Prasad Mishra, Satchi-
dananda Dehuri. Techniques and Environments for Big Data Analysis: Parallel, Cloud,
and Grid Computing. Studies in Big Data 17. Springer International Publishing, 1
edition, 2016.
51
REFERENCES
[CL14] Nathan Carlson and Phil Laplante. The NASA automated requirements measurement
tool: A reconstruction. Innovations in Systems and Software Engineering, 10(2):77–91,
2014.
[Cor] IBM Corporation. Get it right the first time: Writing better require-
ments. http://download.boulder.ibm.com/ibmdl/pub/software/
rationalsdp/documentation/product_doc/DOORS/version_9-2/get_
it_right_the_first_time.pdf. Accessed: 2018-07-01.
[D.L01] Elizabeth D.Liddy. Natural Language Processing. Encyclopedia of Library and Infor-
mation Science, 2001.
[FFJ+ 14] Henning Femmer, Daniel Méndez Fernández, Elmar Juergens, Michael Klose, Ilona
Zimmer, and Jörg Zimmer. Rapid requirements checks with requirements smells: two
case studies. Proceedings of the 1st International Workshop on Rapid Continuous Soft-
ware Engineering - RCoSE 2014, pages 10–19, 2014.
[HDS+ 04] B.; Haskins, B.; Dick, J.; Stecklein, R.; Lovell, G.; Moroney, and J Dabney. Error Cost
Escalation Through the Project Life Cycle. Incose -Annual Conference Symposium
Proceedings- Cd Rom Edition; 2004, page 8.4.2, 2004.
[IEE08] IEEE. IEEE STD 1028-2008, IEEE Standard for Software Reviews and Audits, volume
2008. 2008.
[JM09] Daniel Jurafsky and James H Martin. Speech and Language Processing: An Introduc-
tion to Natural Language Processing, Computational Linguistics, and Speech Recogni-
tion. Speech and Language Processing An Introduction to Natural Language Process-
ing Computational Linguistics and Speech Recognition, 21:0–934, 2009.
[Kim14] Yoon Kim. Convolutional Neural Networks for Sentence Classification. pages 1746–
1751, 2014.
[Lam05] Giuseppe Lami. QuARS: A Tool for Analyzing Requirement | SEI Digital Library.
(September), 2005.
52
REFERENCES
[LK17] Marc Moreno Lopez and Jugal Kalita. Deep Learning applied to NLP. Arxiv, 2017.
[McC01] J. R. McCoy. NASA software tools for high-quality requirements engineering. 26th An-
nual NASA Goddard Software Engineering Workshop, IEEE/NASA SEW 2001, page 69,
2001.
[MSB+ 14] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard,
and David McClosky. The Stanford CoreNLP Natural Language Processing Toolkit.
Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics:
System Demonstrations, pages 55–60, 2014.
[PEM03] Frauke Paetsch, Armin Eberlein, and Frank Maurer. Requirements engineering and
agile software development. WET ICE 2003. Proceedings. Twelfth IEEE International
Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises,
2003., pages 308–313, 2003.
[PES17] Gerald Penn, Jason Eisner, and Noah Smith. Statistical Parsing. 13, 2017.
[QuAa] An Automatic Tool for Improving the Quality of Software Requirements. https:
//www.ercim.eu/publication/Ercim{_}News/enw58/lami.html. Ac-
cessed: 2018-02-04.
[RM] RM Extensions
API. https://jazz.net/wiki/bin/view/Main/
RMExtensions405API. Accessed: 2018-07-03.
[RMP13] Alpa Reshamwala, Dhirendra Mishra, and Prajakta Pawar. Review on natural language
processing. IRACST Engineering Science and Technology: An International Journal
(ESTIJ), 3(1):113–116, 2013.
[RR] James Robertson and Suzanne Robertson. The user story considered harmful.
[SAAA+ ] Mathias Soeken, Nabila Abdessaied, Arman Allahyari-Abhari, Andi Buzo, Liana
Musat, Georg Pelz, and Rolf Drechsler. Quality Assessment for Requirements based
on Natural Language Processing.
53
REFERENCES
[SARW12] Bahar Sateli, Elian Angius, Srinivasan Sembakkam Rajivelu, and René Witte. Can
Text Mining Assistants Help to Improve Requirements Specifications? Mining Un-
structured Data (MUD 2012), 2012.
[Sta17] Stanford University. Stanford CoreNLP – Core natural language software | Stanford
CoreNLP. https://stanfordnlp.github.io/CoreNLP/, 2017. Accessed:
2018-01-30.
[Tin10] Kai Ming Ting. Precision and Recall, pages 781–781. Springer US, Boston, MA, 2010.
[VAD09] S Vinay, Shridhar Aithal, and Prashanth Desai. An Approach towards Automation of
Requirements Analysis. Proceedings of the International MultiConference of Engineers
and Computer Scientists, I:1–6, 2009.
[VK08] Kunal Verma and Alex Kass. Requirements analysis tool: A tool for automatically
analyzing software requirements documents. Lecture Notes in Computer Science (in-
cluding subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioin-
formatics), 5318 LNCS:751–763, 2008.
[VKV14] Kunal Verma, Alex Kass, and Reymonrod Vasquez. Using syntactic and semantic anal-
yses to improve the quality of requirements documentation. Semantic Web, 5(5):405–
419, 2014.
[YT06] Deniz Yuret and Ferhan Türe. Handbook of Natural Language Processing. Proceed-
ings of the main conference on Human Language Technology Conference of the North
American Chapter of the Association of Computational Linguistics -, pages 328–334,
2006.
54