IFAC Conference on Manufacturing Modelling,

IFAC
IFAC Conference
Conference
Management on
and on Manufacturing Modelling,
Manufacturing
Control Modelling,
Available online at www.sciencedirect.com
Management
Management and
IFAC Conference
June 28-30, 2016. Control
and on
Control
Manufacturing
Troyes, France Modelling,
June
June 28-30, 2016.
28-30, 2016.
Management Troyes, France
Troyes, France
and Control
June 28-30, 2016. Troyes, France ScienceDirect
IFAC-PapersOnLine 49-12 (2016) 1567–1572

Research
Research methodology for systems engineering: some recommendations.
Research methodology
methodology for
for systems
systems engineering:
engineering: somesome recommendations.
recommendations.
Research methodology for systems engineering: some recommendations.
Emmanuel Caillaud*, Bertrand Rose*, Virginie Goepp**
Emmanuel
Emmanuel Caillaud*,
Caillaud*, Bertrand
Bertrand Rose*,
Rose*, Virginie
Virginie Goepp**
Goepp**
Emmanuel Caillaud*, Bertrand Rose*, Virginie Goepp**
* Université de Strasbourg-ICUBE
** Université
Université de Strasbourg-ICUBE
de 67000
Strasbourg-ICUBE
3-5 rue de l’université, Strasbourg FRANCE
3-5 rue
3-5 rue de
de* l’université,
Université de
l’université, 67000 Strasbourg FRANCE
Strasbourg-ICUBE
67000 FRANCE
**INSA Strasbourg-ICUBE 24, boulevard de laStrasbourg
victoire 67000 Strasbourg FRANCE
**INSA Strasbourg-ICUBE
**INSA Strasbourg-ICUBE
3-5 rue de 24, boulevard
boulevard
l’université,
24, de la
67000
de laStrasbourg
victoire 67000
victoire 67000 Strasbourg FRANCE
FRANCE
Strasbourg FRANCE
(e-mail: firstname.name@icube.unistra.fr)
(e-mail:24,
**INSA Strasbourg-ICUBE
(e-mail: firstname.name@icube.unistra.fr)
boulevard de la victoire 67000 Strasbourg FRANCE
firstname.name@icube.unistra.fr)
(e-mail: firstname.name@icube.unistra.fr)
Abstract: Systems engineering is very well known research domain even if it covers a wide spectrum.
Abstract: Systems
Systems engineering
engineering is
is very
very well known
known research
research domain
domain even
even if
if it covers
covers aa wide
wide spectrum.
Abstract:
Research in systems engineering can bewell
very diverse considering their objectit or their researchspectrum.
method.
Research in
Abstract:
Research in systems
Systems
systems engineering
engineering
engineeringis can
very
can bewell
be veryknown
very diverse considering
research
diverse domain
considering their object
even
their if
objectit or their aresearch
covers
or their research
wide method.
spectrum.
method.
According to the research objectives, and the methods used, validations have to be carried out with great
Accordinginto
Research
According tosystems
the research
the research objectives,
engineering can
objectives, and
be
and the methods
very
the methods
diverse used, validations
validations
considering
used, their have
object
have toorbe
to betheir
carried out with
research
carried out with great
method.
great
care. This paper aims at presenting different approaches followed in related disciplines and to propose a
care. This
care. This paper
According paper
to the aims
aims at presenting
research
at presenting
objectives,different
and the
different approaches
methods
approaches followed
used, in related
related
validations
followed in have disciplines
to be carried
disciplines andout
and to with
to propose aa
great
propose
first guide to validate the different research works in systems engineering.
first guide
care.
first guide to validate
validate
This paper
to the
aims the different research
at presenting
different research works
differentworks in systems
systems
approaches
in engineering.
followed in related disciplines and to propose a
engineering.
Keywords:
© 2016,
first IFAC
guide Systems
to validateengineering,
(International
the differentPSS,
Federation research
research worksmethodology,
of AutomaticinControl)
systems validation.
Hosting by Elsevier Ltd. All rights reserved.
engineering.
Keywords: Systems engineering, PSS, research methodology,
Keywords: Systems engineering, PSS, research methodology, validation.validation.
Keywords: Systems engineering, PSS, research methodology, validation.
1. INTRODUCTION
1.
1. INTRODUCTION
INTRODUCTION
As defined by INCOSE 1. (International Council on Systems
INTRODUCTION
As defined
As defined by INCOSE (International CouncilCouncil on on Systems
Systems
Engineering),by« INCOSE (International
Systems Engineering is an interdisciplinary
Engineering),
As defined
Engineering), by « INCOSE
« Systems Engineering
Systems Engineering
(International isCouncil
is an interdisciplinary
an interdisciplinary
on Systems
approach and means to enable the realization of successful
approach
Engineering),
approach and «means
and means
Systems to enable
to enable the realization
Engineering
the realization
is an of successful
successful
interdisciplinary
of
systems. It focuses on defining customer needs and required
systems.
approach It focuses
Itand meanson defining customer needs and required
systems.
functionality focuses
early in tothe
on enable
defining the realization
customer
development needs of
cycle, andsuccessful
required
documenting
functionality
systems.
functionalityIt early
focuses
early in
on
in the development
defining
the customer
development cycle,
needs
cycle, documenting
and required
documenting
requirements, then proceeding with design synthesis and
requirements,
functionality
requirements, then
early
then proceeding
in the
proceeding with
development
with design
cycle,
design synthesis
documenting
synthesis and
and
system validation while considering the complete problem:
system validation
requirements, while
thenwhile considering the complete problem:
system
Operations,validation
Cost &proceeding
Schedule, with
considering design
the
Performance, synthesis
complete and
problem:
Training &
Operations,
system Cost while& Schedule,
Schedule, Performance, Training &
Support,validation
Operations, Test,Cost &
Disposal, considering
Manufacturing the
Performance, complete
» (INCOSE, problem:
Training
2015). &
Support,
Operations,
Support, Test,Cost
Test, Disposal, Manufacturing
& Schedule,
Disposal, Manufacturing »» (INCOSE,
Performance,(INCOSE, 2015). &
Training
2015).
After
Support,theTest,
sameDisposal,
definition, « Systems Engineering
Manufacturing » (INCOSE,integrates
2015).
After
After the same
same definition, «« Systems Engineering integrates
all thethedisciplines definition,
and specialty Systems
groups Engineering
into a team integrates
effort
all
all the
Afterthethedisciplines and specialty groups into aa team effort
forming a same
disciplines definition,
structured anddevelopment« Systems
specialty groups
processEngineering
into integrates
team
that proceeds effort
from
forming
all the
forming a structured
a structuredand
disciplines development
specialty
development process into
groups
process that proceeds
that proceeds
a team from
effort
from
concept to production to operation. Systems Engineering
concept
forming tostructured
production to operation.
operation. Systems Engineering
considers both the business and the technical needs offrom
concept ato production development
to process that
Systems proceeds
Engineering all
considers
concept both the business
business and and the
the technical
technical needs of all
all
customerstoboth
considers production
with the goal toof operation.
the providing aSystems Engineering
quality needs
product ofthat
customers
considers
customers with the
both
with the business
the goal of
goal of providing
providing
and the a quality
quality needs
technical
a productofthat
product that
all
meets the user needs ».
meets the
the user
customers
meets withneeds
user the goal
needs ».
». of providing a quality product that
A functional
meets the user approach
needs ». of the System Design Process has
A functional
A functional approach of thethe System
System DesignDesign Process has has
been proposedapproach by (Bahil of and Gissing, 1998) and Process
is presented
been
A proposedapproach
by (Bahil
(Bahil of andthe Gissing, 1998) and Process
is presented
presented
in Figure 1. Each of the function could be considered ashasa
beenfunctional
proposed by and System
Gissing, Design
1998) and is Fig. 1. Functions of the System Design Process (Bahil
Fig. 1.
1. Functions
Functions of
of the System Design Process (Bahil
in
been
in Figure
proposed
Figure 1. Each
1. Each of the
by (Bahil
of theand function could
Gissing,
function could1998) be and
be considered as aa
is presented
considered as Fig.
and Gissing, 1998). the System Design Process (Bahil
field of research: requirements engineering, configuration and Gissing, 1998).
field
in
field of
Figure
of research:
1. Each
research: ofrequirements
the function
requirements engineering,
could
engineering, be configuration
considered
configuration as a and
Fig. Gissing, 1998).
1. Functions of the System Design Process (Bahil
management, functional analysis, architecture of a system,
management,
management,
field of functional
functional
research: analysis,
analysis,
requirements architecture
architecture
engineering, of
of aa system,
system,
configuration and Gissing, 1998).
According to the Systems Engineering Vision 2015
conceptual design, design assessment and decision making, According
According to
to the Systems
the different Engineering
Systems aspects
Engineering Vision 2015
conceptual
management,
conceptual
design change
design,
design, design
functional
management,
assessment
designanalysis,
assessment
model
and
architecture
and decision
of
decision
based a making,
system,
making,
engineering, (INCOSE, 2014), will Vision
have to2015 be
design change management, model based engineering, (INCOSE,
According
(INCOSE, 2014),
to the
2014), different
Systems
different aspects
aspects will
Engineering
will have
have to
Vision to2015be
be
conceptual
design
scalability. design, design assessment
change management, model based engineering, and decision making, considered:
scalability. considered:
(INCOSE, 2014), different aspects will have to be
considered:
design change management, model based engineering,
scalability.
Research in systems engineering is generally considered as 1. broadening of Systems Engineering application
considered:
scalability. 1.
1. broadening
broadening of
of Systems
Systems Engineering
Engineering application
Research
Research in systems
integrative,in young
systemsand engineering
engineering is
is generally considered
generally Generally,
to be strengthened. consideredthe as
as domains (applying SE across industryapplication
domains,
domains
1. domains (applying SE across industry domains,
integrative,
Research
integrative, in young
systems
young and
and to be
engineering
to be strengthened.
is generally
strengthened.
research method has to be clarified. A research method
Generally,
considered
Generally, the
as
the applying (applying
broadening SE across
SEof toSystems
policy, industry
Engineering
develop valuedomains,
application
driven
applying
domains
applying SE to
(applying policy,
SE develop
across value
industry driven
valuedomains,
research
integrative,
research
should
method
young
method
define
has
has
research
to
totobebebeclarified.
andquestions, strengthened.
clarified. A research
proposeAa researchGenerally,
robust
method
the
method
method to practices, SEcomplex
to policy,systems
develop understanding,
driven
should define research questions, proposeAaa research
robust method
method to practices,
applying
practices, SE complex
to
complexpolicy,systems
develop
systems understanding,
value driven
understanding,
research
should
solve the method
defineproblem has
research to be clarified.
and questions,
validate propose
the researchrobust
method and method to
the leveraging technology for SE tools, develop
solve the problem and questions,
validate thethe research method and the
the leveraging
practices,
leveraging technology
complex
technology for SE
for SE tools,
systems tools, develop
understanding,
develop
should
solve define
the research
problem and
results (Booth et al., 2003). validate propose
research a robust
method method
and to collaborative engineering, system of systems
results (Booth et al.,
al.,and
2003). collaborative
leveraging
collaborative engineering,
technology
engineering, system
for system
SE tools,of develop
of systems
systems
solve the
results (Boothproblem
et validate the research method and the
2003). engineering practices, address multiple stakeholders
engineering
collaborative
engineering practices, addresssystem
engineering,
practices, address multiple of
multiple stakeholders
systems
stakeholders
results (Booth et al., 2003). viewpoints, architecting and design of resilient
viewpoints,
engineering
viewpoints, architecting
practices, and
address
architecting and design
multiple
design of resilient
stakeholders
of resilient
systems, cyber threats, decision support, virtual
systems,
viewpoints,
systems, cyber threats,
threats, decision
architecting
cyber and design
decision support, virtual
of resilient
support, virtual
engineering)
engineering)
systems, cyber threats, decision support, virtual
engineering)
engineering)
2405-8963 © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved.
Copyright
Peer review©under
2016 responsibility
IFAC 1567Control.
of International Federation of Automatic
Copyright
Copyright ©
© 2016
2016 IFAC
IFAC 1567
1567
10.1016/j.ifacol.2016.07.803
Copyright © 2016 IFAC 1567
IFAC MIM 2016
1568
June 28-30, 2016. Troyes, France Emmanuel Caillaud et al. / IFAC-PapersOnLine 49-12 (2016) 1567–1572

2. education (role of system engineer, SE
competencies, SE curriculum, Lifelong Learning)
3. methods, tools and theoretical foundations (shoring
up the theoretical foundation, SEBoK, System
theories).
The third point is synthesised in (INCOSE, 2014) in the Fig.
2. From this figure, one can list the different research topics
that has to be lead in the future in SE.
continuing into the 1980s, researchers began seeking to
establish a more logical and systematic approach to design,
and they proposed a number of frameworks and
methodologies. At the end of World War II, people were
faced with social and economic problems and the design
process began to be seen as a problem-solving and decision-
making activity (Bayazit, 2004). New theories, computational
tools, concepts, increased interest in how teamwork is
involved in design, and much other advancement in the field
has now motivated the Experimental phase. Thus, more data-
gathering observational methods are taking place to
understand and improve the impact of new methods and tools
on the design process.
In our study on papers published in Research in engineering
design, an international journal, we noticed that 35% of the
papers do not include any validation (Barth et al., 2011). 12%
of the proposals are validated by an application, 11% are
validated by comparisons. Most of the validations are
subjective: « it works! ». This weakness of validation does
not allow the scientific community to have strong
foundations. 46% of the papers do not include an industrial
application. If an industrial application is not always required
scientifically, it can be very important for the use of the
proposals in industry.

2.2 Research in Information Systems

According to Hevner et al (2004), research in Information

Systems must build and evaluate theories and artifacts. The
authors propose a framework of Information Systems
research reproduced in Fig. 3.

Fig. 2. Methods, tools and foundations of SE (INCOSE,

2014).

We started to study research methodologies in the field of

engineering design since 2010 with a focus on research
validation (Barth et al., 2011). We noticed a lack of research
methodology. Our objective is here to propose a guide for
conducting research in the field of SE. In the next sections we
will first present a state of the art on the research methods
(including research validation). Then we will propose an
approach to practically do research in SE.
2. STATE OF THE ART
It can be considered that design research has evolved through
three phases of development: Experiential, Intellectual, and
Experimental. Considering that engineering is one of the key Fig. 3. Information Systems research framework (Hevner et
issues of SE, research in engineering design will be studied in al. 2004).
section 2.1. As a result of the research in SE, an information
system is defined or developed, research in information 7 guides are proposed and explained to contribute a high
system is presented in section 2.2. quality research: propose a viable artefact, solve relevant
business problems, evaluate the design artefact, propose clear
2.1 Research in engineering design and verifiable scientific contributions, apply rigorous
research methods, use available means, present research to
The evolution of research in engineering design has been technology-oriented and management-oriented audiences.
described in (Blessing et al., 2009). The Experiential phase
consisted of designers writing about their product processes,
but it was evident the field lacked a systematic theoretical
framework. During the Intellectual phase in the 1960s, and

1568
IFAC MIM 2016
June 28-30, 2016. Troyes, France Emmanuel Caillaud et al. / IFAC-PapersOnLine 49-12 (2016) 1567–1572 1569

2.3 Partial conclusions and research tracks

From the previous subsections, different observations can be

made: a lack of research method in engineering design and
several guidelines coming from Information Systems
research. For both domains, the variety of results of research
is important but there is not an analysis of the types of
research, nor the types of validations. We did not find any
paper on research methods for systems engineering.
In the next section, we will present a typology of research
methods and validation methods.
3. RESEARCH METHODS AND VALIDATION
In this section, we will consider different types of research
methods used in SE and different types of validation.
Validations are linked to research methods.

3.1 Research methods

Research in systems engineering is generally applied to real

systems and companies. A common research approach is then
“case studies” to model and prescribe new practices. For
these applications, researchers cannot control all the
parameters of the complex (Yin, 2009). When many variables
are studied, different case studies can be required.
The research process can be represented as in Fig. 4.
Fig. 5. Research process using “grounded theory” (Charmaz,
2006).
Research methods should allow to be sure of the validity of
the results, keep a logical track of evidences and test other
possible explanations (Yin, 2009).
Considering that the research method should be chosen
according to the context of objectives and that the validation
should be adapted to the research method, we can draw Fig.
6.

Fig. 4. Research process using case studies after (Yin, 2009)

Due to the qualitative nature of many work in SE, «grounded
theory» is used. Grounded theory synthetised a set of
systematitic inductive research methods to lead a qualititative
research leading to a theory. «Grounded theory» can be used
more as an inductive approach or as a validation approach. Fig. 6. Relations between research objectives, methods and
More and more, researchers use “grounded theory to structure validations.
a research collecting analysing qualitative data (Charmaz,
2006). The research process can be represented as in Fig. 5. 3.2 Validation typology

Validation of research requires the definition of the different

criteria of evaluation (March et al, 1995).

1569
IFAC MIM 2016
1570
June 28-30, 2016. Troyes, France Emmanuel Caillaud et al. / IFAC-PapersOnLine 49-12 (2016) 1567–1572

Previous works on the validation of research in engineering validation. Classically, the research project has an industrial
have been presented in (Frey et al, 2006). A first list of types problem root. This problem is studied using a bibliographic
of validations can be proposed: study. The gap between the problem to be solved and the
previous work allow to define the scientific problem. From
- application,
the industrial problem and the scientific problem, a research
- comparison, method can be defined. The main results generally consist in
models and prescriptions. Prescriptions contribute to solve
- focus groups,
the industrial problem. Their deployment method and
- questionnaire, appropriation can also be studied and contribute to solve the
industrial problem.
- simulation,
The research approach is presented in Fig. 8.
- statistical analysis.
In subsection 4.1, a list of recommendations is proposed to
Using the elements to be validated in section 2.2, we can improve the quality of research in SE. In subsection 4.2, the
propose in Fig. 7, a validation process linking the elements to limits of validation are underlined.
be validated, the criteria and the method of validation.
4.1 Recommendations for research approach and validation

In the research approach presented Fig. 8, we will develop a

Fig. 7. From validation to validation methods.
To go further, a guide to improve the quality of research in
SE including an adapted validation is proposed in the next
section.
4. A GUIDE TO VALIDATE RESEARCH IN SE
Due to the complexity of projects lead in SE, research
projects require several research methods and several types of
list of recommendations.
R1: The industrial problem must be clearly defined. Its
specificities and generic features have to be distinguished.
The relevance and importance of the industrial problem must
be defined tanks to comparisons with other companies,
business sectors, a qualitative analysis of the synthesis of
experts. The initial situation in the industry must be evalued
in order to assess the effects of the research project.
R2: The bibliographic study must be complete and relevant.
Many studies just consider the very last papers available in
indexed databases without considering PhDs, conferences
books and “old” papers. In SE, as underlined in section 1,
different scientific domains have to be considered. The
objective is not to take one or two references from an other
domain without considering its specificities (specific
definitions, background, etc.). An special attention must be
paid to the definitions, which must be linked to their
discipline.

Fig. 8. Research approach.

1570
IFAC MIM 2016
June 28-30, 2016. Troyes, France Emmanuel Caillaud et al. / IFAC-PapersOnLine 49-12 (2016) 1567–1572 1571

R3: The scientific problem well has to describe the lacks in Table1. Recommendations for research in SE.
bibliography and the industrial needs. Hypotheses must be
explicit to be validated/or not at the end the research project. Field Validation criteria

R4: One or several research methods must be explicitly Industrial problem Important, relevant, specific,
chosen in a justified way and they must be described. When quantified
several methods are used, their complementarities must be
expressed. Bibliographical study Complete, opened to other
disciplines, relevant, clear
R5: Numerous works lead at first to models. The choice of definitions.
the method of modelling must be justified. Do these models
respect the formalism of the method chosen? Are they Scientific problem Gaps identified, hypotheses
exploitable by of other than those who realized them? Can explicit.
they be effectively used to produce of a physical product,
software, or an organization? Research methods Explicit, justified

R6: One of the classic results is a list of best practice and Models Choice of formalism justified
precise recommendations (like this one?). Are the conditions and respected, viable.
of use correctly described?
Prescriptions Explicit, conditions of use
R7: Is it clearly stated if the prescriptions were applied? Did defined, viable, efficient.
they lead to an improvement of the performances? The
criteria of evaluation will have been defined from the Deployment process Explicit, conditions of use
beginning by the study (cf. R1). defined, tested, efficient.

R8: The process of deployment of the prescriptions has to be Publication Ethic, appropriately
clearly defined. A model of the process of deployment is presented in the different
necessary for several temporal horizons and several strategic scientific communities.
levels. Is this process valid only for a company/for a business
sector? The conditions of use (level of maturity of the
company for example) of the process must be clearly stated
(generic?). 4.2 Limits of validation

R9: An evaluation of the efficiency and the quality of the
deployment must be also realized. It supposes the definition
of indicators on the deployment in terms of resources and
satisfaction.
R10: The research works must be not only communicated in
a ethical way but also in communities most to estimate them
with relevance. For example, if the contribution is
information systems to support SE, both communities 5IS
and SE) must be requested.
These different recommendations are converted
validation criteria in Table 1.
The limits of validation of the scientific contribution and the
validation of the industrial contribution are essential in
research. Indeed, the scientific contribution and the industrial
contribution are valid only in a given context with a given
objective. The evaluation of these limits allows at the same
time the re-use of the results in the same context but also to
define new avenues of research.
One of the limits of validation is so clearly the human
appreciation to evaluate quantitative but especially qualitative
criteria. For example the relevance and the importance of a
into
problem are subject to interpretation. For example, should
you we be interested in a disease because it cause a huge
number of deaths or because the market is the most
promising?
5. CONCLUSIONS
In this communication, we tried to propose some elements of
discussion to define a guide for the researchers in system
engineering.
These works are realized only on the basis of a
bibliographical study and on the basis of our experience in
research. Its validation is thus very limited and will ask to be
experimented and discussed. The feedbacks during the IFAC
conference will contribute to the evolution of the contribution
and to its partial validation.

1571
IFAC MIM 2016
1572
June 28-30, 2016. Troyes, France Emmanuel Caillaud et al. / IFAC-PapersOnLine 49-12 (2016) 1567–1572

REFERENCES
Bahil, A. T. and Gissing, B. (1998). Re-evaluating systems
engineering concepts using systems thinking, IEEE
Transaction on Systems, Man and Cybernetics, Part C:
Applications and Reviews, 28 (4), 516-527.
Barth, A., Caillaud, E., Rose, B., (2011) How to validate
research in engineering design. International Conference
on Engineering Design - ICED'11, Copenhaguen,
Danemark.
Bayazit, N. (2004) Investigating Design: A Review of Forty
Years of Design Research. Design Issues, 20(1), 16-29.
Blessing, L. T. M., Chakrabarti, A. (2009) DRM, a Design
Research Methodology, Springer.
Booth, W. C., Colomb, G. G. & Williams , J. M. (2003). The
Craft of Research, University of Chicago Press.
Charmaz, K. (2006). Constructing the Grounded Theory.
SAGE Publications.
Hevner, A.R., March, S.T., Park, J., Ram, S., (2004) Design
science in information systems research, MIS Quaterly
28 (1), 75-105.
INCOSE (2014). Systems Engineering vision 2025. INCOSE.
INCOSE (2015). What is systems engineering,
http://www.incose.org/AboutSE/WhatIsSE, 1st
december 2015.
Yin, R.K, (2009). Case study research. Design and methods.
Fourth edition. Applied social research methods series.
Volume 5. Sage.

1572