Introduction of Agile Practices

INTRODUCTION OF AGILE PRACTICES
Strategies and Impacts
Indira Nurdiani
Blekinge Institute of Technology

Doctoral Dissertation Series No. 2018:06
Department of Software Engineering

Introduction of Agile Practices
Indira Nurdiani
Blekinge Institute of Technology Doctoral Dissertation Series
No 2018:06
Introduction of Agile Practices

Indira Nurdiani
Doctoral Dissertation in
Software Engineering

Blekinge Institute of Technology
SWEDEN
2018 Indira Nurdiani
Publisher: Blekinge Institute of Technology
SE-371 79 Karlskrona, Sweden
Printed by Exakta Group, Sweden, 2018
ISBN: 978-91-7295-352-9
ISSN:1653-2090
urn:nbn:se:bth-15966
i
“If there be nothing new, but that which is

Hath been before, how are our brains beguil’d,
Which labouring for invention bear amiss
The second burthen of a former child!”
– Sonnet 59, William Shakespeare
Modern English paraphrase:

“If it’s true that there’s nothing new and everything that now exists
Existed in the past, then we are really fooling ourselves
When we struggle to write something new, after much exhausting, painful labor,
With a tired imitation of an imitation!”
ii
Abstract
Background: Software development organizations frequently face changes that re-

quire them to be flexible. The principles and practices of Agile software are often
associated with improving software organizations’ flexibility. However, introducing
Agile practices have its benefits and limitations. To amplify benefits and alleviate chal-
lenges, Agile adoption guidelines are being proposed to provide strategies for intro-
ducing Agile practices. One instance of such guidelines is known as Agile Maturity
Models (AMMs). AMMs typically suggest that Agile practices are introduced in cer-
tain orders. However, AMMs provide contradictory strategies. Thus it is not known
whether one strategy to introduce Agile practices is better than others.
Objective: The objective of this thesis is to gather and examine the evidence on
the different strategies of introducing Agile practices, particularly on the order of intro-
duction as suggested in the AMMs. The thesis seeks if one order for introducing Agile
practices is better than others.
Method: Combination of empirical studies were used in this thesis. The data
collection was done through a survey and semi-structured interviews. This involved
analyzing the introduction of Agile practices over time, i.e. the start and/or end of
Agile practices. A qualitative method like qualitative coding was used to analyze data
obtained from the interviews. Different quantitative methods like inferential statistics
and social network analysis were also used. Literature studies were also conducted to
provide background and support for the empirical studies.
Results: The examination of the evidence indicates that there is not one strategy to
introduce Agile practices that would yield better results than others. The lack of con-
clusive evidence could be caused by the lack of consideration on reporting the context
of empirical studies, particularly on the baseline situation, i.e. situation prior to Agile
introduction. A checklist is proposed to capture a baseline contextual information fo-
cusing on internal organizational aspects of a software organization: the constellation
of team members’ skills and experience, management principles, existing practices and
systems characteristics of the software under development. The checklist was validated
iv
by seven experts in academia. The experts who participated in the validation perceived
the checklist to be useful and relevant to research.
Conclusion: The studies presented in this thesis can be a useful input for re-
searchers who are conducting an empirical study in Agile software development. The
checklist proposed in this thesis could be used to help researchers to improve their
research design when evaluating the extent of improvements from introducing Agile
practices. If researchers use the checklist, consistency across empirical studies can
be improved. Consistency in reporting empirical studies is desired for comparing and
aggregating evidence. In turn, this will help practitioners to make a fair assessment
whether research results are relevant to their contexts and to what extent the results are
helpful for them.
Acknowledgements
I would like to express my gratitude to Prof. Jürgen Börstler for his mentoring, support,
and critiques to push me to do better. I also would like to thank Prof. Samuel Fricker
for his feedback and advise.
I would also like to extend my gratitude to Prof. Kai Petersen for his insights,
continuous assistance and wonderful collaboration. I also would like to thank Prof.
Niklas Lavesson for his encouragement and advice.
I appreciate all the support that I received from my colleagues in the SERL Sweden
group. I would like to thank Dr. Panagiota Chatzipetrou for wonderful collaboration
and sharing great ideas. I also would like to thank Deepika Badampudi for proofreading
parts of this thesis and always coming to my aid whenever I needed it. I also would
like to thank Monica Nilsson, for her kindness and patience.
I would also like to extend my gratitude to the industry practitioners for their partic-
ipation and contribution to this thesis. I am also indebted to the academic experts who
have provided their inputs and feedbacks on some parts of this thesis: Prof. Barbara
Weber (Technical University of Denmark), Prof. Jürgen Münch (University of Reut-
lingen), Prof. Michael Felderer (University of Innsbruck), Prof. Helena Holmström
Olsson (Malmö University), Dr. Eric Knauss (Chalmers University of Technology),
Prof. Helen Sharp (The Open University), and Dr. Marco Kuhrmann (Clausthal Uni-
versity of Technology).
Last but definitely not least I would like to thank my family: my parents and my
sisters, for their care, love and inspirations. I would not have been able to complete
this thesis without the support of my husband, Ronald, for his constant encouragement,
support, and understanding throughout this trying period. I also would like to thank my
son Axel for his patience. I would also like to thank my family away from home, Yossy,
Octa, and Astuti.
vi
Overview of Papers
Papers in this Thesis

Chapter 2: I. Nurdiani, S.A. Fricker, and J. Börstler, (2015) “An Analysis of Change
Scenarios of an IT Organization for Flexibility Building,” In Proceedings of the
23rd European Conference in Information Systems (ECIS 2015).
Chapter 3: I. Nurdiani, J. Börstler, and S.A. Fricker, (2018) “Literature Review of

Flexibility Attributes: A Flexibility Framework for Software Developing Orga-
nization” Journal of Software: Process and Evolution.
Chapter 4: I. Nurdiani, J. Börstler, and S.A. Fricker, (2016) “The Impacts of Agile
and Lean Practices on Project Constraints: A Tertiary Study,” Journal of Systems
and Software, 119(Supplement C):162 – 183.
Chapter 5: I. Nurdiani, J. Börstler, and S.A. Fricker, Kai Petersen (2018) “ Usage,
Retention, and abandonment of Agile practices: A Reflection on Agile Maturity
Models”, under revision in e-informatica Software Engineering Journal.
Chapter 6: I. Nurdiani, J. Börstler, and S.A. Fricker, Panagiota Chatzipetrou (2018) “

Strategies to Introduce Agile Practices: Comparing Agile Maturity Models with
Practitioners’ Experience ”, submitted to Empirical Software Engineering.
Chapter 7: I. Nurdiani, J. Börstler, and S.A. Fricker, Kai Petersen (2018) “ Captur-
ing Baseline Situations in Studying the Impacts of Agile Practices Introduction:
A Preliminary Checklist and Propositions for Future Research”, Extension of a
paper accepted at the Sixth International Workshop on Conducting Empirical
Study in Industry (CESI 2018).
viii
Contribution Statement
Indira Nurdiani is the lead author of all the chapters. She also took the responsibility
of designing and executing the study, as well as analyzing and writing up the results of
the studies into papers. The contribution in the individual chapters are as follows.
Chapter 2: Indira Nurdiani design the grounded theory study, Samuel Fricker and
Jürgen Börstler provided feedback on the design of the study. Indira Nurdiani per-
formed the data collection together with Samuel Fricker. Indira Nurdiani also tran-
scribed the interviews and analyzed the interviews and reported the findings. Samuel
Fricker and Jürgen Börstler were also involved in piloting the qualitative coding in the
analysis step.
Chapter 3: Jürgen Börstler contributed with the idea of conducting a tertiary study.
Indira Nurdiani then design the review protocol, executed the literature search and
study selection. Jürgen Börstler and Samuel Fricker participated in post-hoc valida-
tion of the study selection and quality assesment. Indira Nurdiani also write the final
draft, where Jürgen Börstler and Samuel Fricker reviewed, commented, and suggested
improvements to the final draft prior to submission.
Chapter 4: Indira Nurdiani contributed to the idea of conducting a literature re-
view. Indira Nurdiani contributed to the study design, execution of the literature review,
and writing the final draft. Jürgen Börstler and Samuel Fricker reviewed, commented,
and suggested improvements to the final draft prior to submission.
Chapter 5: Indira Nurdiani contributed to the idea of conducting the survey and
the interviews. Indira Nurdiani also contributed to the study design, execution, anal-
ysis and writing the final draft of the paper. Jürgen Börstler, Samuel Fricker, and Kai
Petersen reviewed, commented, and suggested improvements to the final draft prior to
submission.
Chapter 6: Chapter 6 is an extension of Chapter 5. Panagiota Chatzipetrou pro-
vided suggestions and ideas for selection of statistical methods as well as social net-
work analysis. Jürgen Börstler and Samuel Fricker reviewed, commented, and sug-
gested improvements to the final draft prior to submission.
Chapter 7: Indira Nurdiani contributed to the idea of proposing and formulating
a checklist and propositions. Jürgen Börstler contributed to the idea of adding sug-
gested scales into the checklist. Kai Petersen contributed to the idea on how to use the
checklist. Samuel Fricker also provided suggestions to improve the checklist. Jürgen
Börstler, Samuel Fricker, and Kai Petersen reviewed, commented, and suggested im-
provements to the final draft prior to submission.
ix
Related Papers not in this Thesis

Paper 1: I. Nurdiani, R. Jabangwe, and K. Petersen, (2016) Practices and Challenges
of Managing Requirements Interdependencies in Agile Software Development:
A Survey, In Proceedings of the 2nd International Workshop on Software Star-
tups.
Paper 2: I. Nurdiani (2016). Managing Requirements Interdependencies in Agile
Software Development: A Preliminary Results. In Proceedings of the 22nd In-
ternational Working Conference on Requirements Engineering: Foundation for
Software Quality, (REFSQ).
Paper 3: I. Nurdiani, S.A. Fricker, and J. Börstler, (2014) “Towards Understanding
How To Build Strategic Flexibility of an IT Organization,” In Proceedings of the
13th IASTED International Conference in Software Engineering,.
Paper 4: I. Nurdiani, S.A. Fricker, and J. Börstler, (2012) “Tracing Requirements
Interdependencies in Agile Teams,” In Proceedings of International Working
Conference on 18th International Working Conference on Requirements Engi-
neering: Foundation for Software Quality (REFSQ).
Other Papers not in this Thesis

Paper 1: R. Jabangwe, T. Worm, and I. Nurdiani, (2018) “Towards the Implemen-
tation of Blockchain in Regulated Rnvironments: An Exploratory Study”, to be
submitted to a conference in 2018.
Paper 2: R. Jabangwe, I. Nurdiani, and M. Svahnberg, (2014)“Impact of Course De-
velopment in Software Architecture Course,” in Den Årliga Högskolepedagogiska
Konferensen – Lärarlädom (The Annual Higher Education Conference).
x
Contents
Abstract iii
Acknowledgements v
Overview of Papers vii

Papers in this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Related Papers not in this Thesis . . . . . . . . . . . . . . . . . . . . . . . ix
Other Papers not in this Thesis . . . . . . . . . . . . . . . . . . . . . . . . ix
1 Introduction 1
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Research Gaps and Contribution . . . . . . . . . . . . . . . . . . . . 7
1.4 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Research Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Overview of Chapters . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.7 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.8 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . 23
2 An Analysis of Change Scenarios of an IT Organization for Flexibility

Building 25
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2 Background and Related Work . . . . . . . . . . . . . . . . . . . . . 26
2.3 Research Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5 Build Flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
xii CONTENTS
2.7 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 43
3 Literature Review of Flexibility Attributes: A Flexibility Framework for

Software Developing Organization 45
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4 The Impacts of Agile and Lean Practices on Project Constraints: A Ter-

tiary Study 91
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.2 Tertiary Studies in Software Engineering . . . . . . . . . . . . . . . . 93
4.3 Review Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5 Usage, Retention, and Abandonment of Agile Practices: A Reflection on

Agile Maturity Models 139
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6 Strategies to Introduce Agile Practices: Comparing Agile Maturity Models

with Practitioners’ Experience 165
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
CONTENTS xiii
6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

6.6 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 201
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
7 Capturing Baseline Situations in Studying the Impacts of Agile Practices

Introduction: A Checklist and Propositions for Future Research 221
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
7.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.4 Results: Capturing Baseline Situations . . . . . . . . . . . . . . . . . 225
7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
7.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Bibliography 235
A Agile Practices Definitions 259
B Survey Design 261

xiv CONTENTS
Chapter 1
Introduction
1.1 Overview
Flexibility is important for any organization, including software organizations to keep
up with changes in the business environment and maintain a competitive advantage
[68, 91]. In software engineering, flexibility is often associated with the principles and
practices of Agile software development [1, 58, 66, 164]. Agile software development
is a group of software development methodologies, e.g., Extreme Programming (XP),
Scrum, and Crystal, that focus on delivering working software products in small it-
erations, being adaptive towards requirement changes, and collaborating closely with
customers [1].
Agile practices include on-site customer, collective ownership, test-driven develop-
ment (TDD) and many more [164, 239]. Introducing Agile practices can lead to various
benefits, such as improved product quality, reduced development cost, and shorter time
to market [45, 126]. However, at the same time, Agile practices can pose challenges
and limitations [141, 175]. For example, introducing pair programming can improve
product quality, but working in pairs requires more intensive communication which can
lead to increased development time [58].
Guidelines are being proposed to aid practitioners in maximizing the benefits and
alleviating the challenges from introducing Agile practices. One instance of such
guidelines is Agile Maturity Models (AMMs) [144, 202]. AMMs suggest strategies
for introducing Agile practices through incrementally adding Agile practices in a cer-
tain order. Proponents of the AMMs argue that introducing Agile practices in a certain
order can amplify benefits and alleviate challenges of Agile practices [144, 202]. How-
2 Introduction
ever, AMMs are not in agreement pertaining to the orders of Agile practice introduction
[127, 157]. Currently, there is a lack of evidence examination to support which strat-
egy to introduce Agile practices would yield to more successful outcomes, i.e. more
benefits and fewer challenges.
This thesis examines the evidence on strategies to introduce Agile practices, specif-
ically those that suggest Agile practices should be introduced in certain orders, like
AMMs. The examination was done through literature studies as well as empirical
studies by performing retrospective studies on how Agile practices are introduced, or
abandoned, over time in the industry. The strategies for introducing Agile practices in
the industry are then compared to those suggested by AMMs.
The results from the studies also indicate that there is no conclusive evidence that
one strategy of introducing Agile practices would lead to a more successful outcome.
The studies also indicate that pre-existing contextual situation in the software devel-
opment team may contribute to the success, or the lack thereof, of Agile practices
implementation. This brought to our attention to the importance of the situation prior
to Agile transformation when examining the impacts of introducing Agile practices. In
this thesis, we refer to the situation prior to Agile practice introduction as a baseline
situation.
The importance of reporting a baseline situation has been highlighted by a number
of authors. Kitchenham et al. [111] suggested that reporting the baseline situation is
important to minimize bias when reporting the extent of improvement from introducing
a new tool. Petersen and Wohlin [167] mentioned that very little is known regarding
how well Agile improves software development in practice because empirical stud-
ies in Agile often do not report a baseline situation. Currently, there is no guideline
for researchers that provides details on what information should be captured when de-
scribing a baseline situation. In this thesis, we identified what information that should
be reported when studying the impacts of introducing Agile practices and formulated
it as a checklist. The checklist can help researchers to make impartial inferences when
evaluating the impacts of Agile practice introduction. Consistent description of a base-
line situation will make it easier to compare the effects of introducing Agile practices.
In turn, this benefits practitioners to make a fair assessment whether research results
are relevant to their contexts and support their decisions in Agile practice introduction.
The remainder of this chapter is organized as follows. The relevant concepts to
this thesis are presented in Section 1.2. Research gaps and how the studies included
in this thesis address these gaps in Section 1.3. The research questions are stated in
Section 1.4. The research methods used to answer the research questions are presented
in Section 1.5. The results from the individual studies are presented in Section 1.6.
The research questions are revisited in Section 1.7. Lastly, Section 1.8 concludes the
introduction.
1.2 Background 3
1.2 Background
This thesis includes the following concepts: software organization, flexibility, Agile
and Lean software development, and Agile adoption guidelines. The following subsec-
tions briefly elaborate these concepts to provide background information on the results
and contributions described in this thesis.
1.2.1 Software Organization

There is no agreed definition of a software organization. In many ways, a software
organization shares the same components with a generic organization. It is a business
that needs to generate revenues by selling high quality products or services [48].
We combined insights from organization management and IT management to de-
fine what a software organization is. In organization management literature, organiza-
tions can be conceptualized as entities that interact with an environment through the
input, conversions, and output [95, 102]. Figure 1.1 for a schematic overview.
Environment
Inputs Conversions Outputs

SW Organization
Workforce
Management
Process
Organization
Structure
Infrastructure
Figure 1.1: Schematic for Software Organization, adapted from [102, 218]
The environment refers to the entities that generate needs, provide inputs, and con-
sume the outputs generated by the organization. The environment includes customers,
legislation, suppliers, competitors, and other stakeholders. The input refers to the
things that the organization obtains to produce the outputs. The inputs include raw
materials, human resources, and information that are obtained from regulators, suppli-
ers, customers, and competitors. The conversions refer to the things the organization
4 Introduction
requires to transform the inputs into something of value (output). IT management lit-
erature describes five constituents for conversion: workforce, management, processes,
organizational structure, and infrastructure [218]. Workforce refers to human resources
and their skills and knowledge. Management refers to leadership, its style and roles,
as well as coordination between the constituents. Process refers to concerted activities
that take place in the organization, for example, a development process. Organiza-
tional structure refers to the organization hierarchy and roles and how they relate to
each other. Infrastructure refers to technology platforms that are required to conceive,
build, release, and evolve products and services. The output refers to the elements that
the organization releases to its environment and include products, services, and value
that are released to the environment.
To maintain competitive advantage software organization needs to be flexible. Flex-
ibility is needed to deal with changes in the business environments, as depicted in Fig-
ure 1.1 [68, 91, 197]. More details on flexibility are presented in Subsection 1.2.2.
1.2.2 Flexibility
Flexibility is an important capability of a software organization in adapting to changes
in the business environment, such as changes in customers’ needs, business processes,
market demands, competitors, and technology [68]. Flexibility has different meanings
depending on contexts and domains [197]. In this thesis, flexibility is defined as the
ability of an entity, e.g., an organization in reacting to changes and uncertainty with
minimal modifications to the entity itself while maintaining low cost, effort, and time.
Flexibility itself is not a goal, but a means to achieve a goal [20]. The goal of flexi-
bility could, for example, be regulatory compliance, reduced training time, or effective
human resource allocation [151].
For a manufacturing firm, flexibility is viewed as an end-to-end capability from
material handling to marketing with the support of the organization structure [197]. The
relations between material handling, production, management, and marketing should
be considered in building flexibility of a manufacturing firm. Such end-to-end view is
required to know which constituents are affected by changes and uncertainties [197].
In the organization management literature, flexibility concerns the organization as a
whole. It includes the organization structure, management, and processes [91]. Build-
ing and improving flexibility is a continuous management task. Managers need to ex-
hibit openness to new ideas to foster flexibility [199]. Meanwhile, organization struc-
ture should also support quick turn around of communication that supports flexibility,
e.g., flat organization structure [197].
In software engineering, flexibility has been primarily studied from the perspec-
tives of the software product and service. In software architecture, flexibility can
1.2 Background 5
be achieved by modularizing a software system and thereby localizing the impact of

change [143]. In software product line engineering, flexibility is achieved through
systematically identifying and managing variations and commonalities from a set of
software systems artifacts, e.g., code, test cases, and requirements. When a product
variation is required, a new product can be instantiated from the previously developed
artifacts [38, 138].
From manufacturing and management literature, we learn that flexibility encom-
passes the entire organization, i.e. holistic. Meanwhile, software engineering primar-
ily focuses flexibility on the product. However, as described in subsection 1.2.1, a
software organization is like any other organization, with the same components and
constituents that make up any organization. Therefore, a software organization also re-
quires a holistic approach to flexibility, where all the different organization constituents
are considered. Making one aspect more flexible might lead to other aspects requiring
more control [125, 197]. Without a holistic view, the trade-offs of flexibility cannot be
thoroughly considered [70, 86].
In software engineering, flexibility is often associated with implementing Agile
and Lean practices [1, 66, 60]. However, as previously discussed, flexibility needs
a holistic view. Currently, the extent to which Agile and Lean practices can help to
achieve holistic flexibility of a software organization has not yet been examined. More
details on Agile and Lean software development are presented in Subsection 1.2.3.
1.2.3 Agile and Lean Software Development

Agile software development is a group of iterative and evolutionary methods that are
aimed at delivering working software quickly to the customers and at the same time
improve responsiveness to market changes[43]. These methods include Scrum, eX-
treme Programming (XP), Crystal, Dynamic System Development Method (DSDM),
Adaptive Software Development (ASD), and Feature Driven Development (FDD) [126,
43]. Agile software development principles include motivated individuals, welcoming
change, delivering working software, and frequent deliveries. These principles are then
translated into a number of practices such as short iterations, collective code ownership,
refactoring, and many more, see [164, 239].
Lean Software Development is derived from lean manufacturing and lean product
development. Many suggest that Lean is one of Agile methodologies [58, 126]. Pop-
pendieck and Poppendieck [169] suggest that Lean software development is the expan-
sion of Agile’s theoretical foundation through the application of well known lean prin-
ciples into software development. A comparison of Agile and Lean methods suggest
that these methods share similar principles and practices, yet some aspects are unique
to each method [164]. Lean does not propose specific process models like Scrum or
6 Introduction
XP. Lean proposes principles and tools, such as Value Stream Mapping (VSM), pull-
systems, etc., [164].
Given the similarities of goals and principles, in this thesis Agile and Lean software
development are often discussed together, as has been done by Petersen [164]. When
discussing Agile and Lean software development, this thesis focuses on the practices.
The introduction of Agile practices into a software organization is often referred
to as Agile transformation [117]. It is referred to as transformation because it usually
it entailed changing from non-Agile software development, usually plan-driven, into
Agile software development [167]. However, there are trade-offs associated with Agile
transformation. Li et al. [129] reported that transitioning from plan-driven to Scrum
improves product quality, but at the same time it increases the stress level for the team
members. Critics of Agile also warned that introducing Agile practices may reduce the
development cost, but it may increase the maintenance cost [105].
The trade-off of Agile transformation could be traced back to the benefits and lim-
itations of the individual practices, or their combinations. Solinski and Petersen [212]
reported that different benefits and limitations of introducing Agile practices could de-
pend on which combination of practices are used. For example, implementing Scrum
practices with TDD or pair programming can lead to benefits in terms of cost and
time. Meanwhile, if Agile practices are combined with plan-driven practices, like up-
front documentation and extensive time planning, benefits concerning quality can be
achieved.
Given the trade-off associated with introducing Agile practices, Agile adoption
guidelines are proposed. Agile adoption guidelines are proposed to support successful
Agile transformation by amplifying the benefits and alleviating the challenges of im-
plementing Agile practices [7, 45, 202]. More details on Agile adoption guidelines are
presented in Subsection 1.2.4.
1.2.4 Agile Adoption Guidelines

Agile adoption guidelines are being proposed by consultants and academics to help
software organizations to transition into Agile more smoothly and successfully. Agile
adoption guidelines claim to provide step-by-step suggestions to transition into Ag-
ile successfully [7, 202]. For example, Scott Ambler’s Agile Scaling Model [7] that
suggests Agile transition should start by implementing the core practices from XP or
Scrum and then scale it to larger teams or implement it in distributed teams. Another
example is Mike Cohn’s ADAPT model for introducing Scrum [45] that suggest suc-
cessful Scrum adoption can be achieved through five activities: Awareness, Desire,
Ability, Promotion, and Transfer, or ADAPT. There are different instances of Agile
1.3 Research Gaps and Contribution 7
adoption guidelines, one such instance that recently gained popularity is Agile Matu-
rity Model (AMM) [194].
AMMs are proposed because it is believed that existing maturity models like Ca-
pability Maturity Model Integration (CMMI) [42] or Software Process Improvement
and Capability Determination (SPICE) [132] is not suitable for Agile processes [160].
AMMs claim to provide Agile-specific guidelines for practitioners to engage in Agile
transformation while managing its risks and challenges [144, 202].
AMMs usually follow an evolutionary progression with levels similar to CMMI
or SPICE. Typically AMMs map Agile practices and maturity levels, indicating some
practices are to be introduced before the other. Three examples of typical AMMs are
shown in Table 1.1.
Table 1.1: Allocation of Agile practices to maturity levels in three AMMs.
Sidky et al. [202] Patel & Ramachandran [160] Nawrocki et al. [144]
Context Agile practice adoption based on a Agile practice adoption based on CMM(I) Adoption of XP based on other ma-
measurement index turity models
Level 1 On-site customer, collaborative — —
planning, coding standard
Level 2 Tracking progress, continuous de- Tracking progress, on-site customer, planning Planning game, collaborating cus-
livery game, TDD tomer (on-site customer), user sto-
ries, metaphors
Level 3 F2F meeting, continuous integra- Refactoring, pair programming, continuous in- Pair programming, coding stan-
tion, self-organizing team tegration, TDD, coding standard, collective dard, collective ownership, contin-
ownership uous integration
Level 4 Daily meeting (stand up meeting), Self organizing team, 40 h week Simplicity (Simple design), on-site
user stories, frequent releases customer
Level 5 TDD, pair programming Focus on continuous improvement —
As we can see from Table 1.1, AMMs suggest that Agile practices should be grad-
ually and continuously added. However, AMMs are not in agreements which Agile
practices should be introduced at which maturity level. A similar observation is also
reported by Leppänen [127]. With contradictory suggestions among the AMMs, practi-
tioners do not yet have the means to determine which AMM or which order of practice
introduction would suit them best. Most importantly, currently there is no way for prac-
titioners to tell if one strategy suggested by one AMM would lead to a more successful
Agile practice implementation.
1.3 Research Gaps and Contribution

Flexibility is paramount for all organization including software organization to main-
tain a competitive advantage. Literature of Agile and Lean methods suggests that prin-
8 Introduction
ciples and practices of Agile and Lean support flexibility [1, 58, 67]. There is no agreed
way to evaluate to which extent Agile and Lean practices address flexibility.
Introducing Agile practices can have positive and negative impacts on quality, time,
and cost [152]. Given the potential benefits and limitations of adopting Agile practices,
AMMs are proposed to provide strategies that would amplify benefits and alleviate
challenges when introducing Agile practices [144, 202]. Typically, the AMMs are
based on the experience of a team or an organization. Furthermore, the AMMs them-
selves were very rarely validated, therefore their usefulness is limited to the contexts
where they were developed.
Typically the AMMs suggest that Agile practices should be introduced in certain
orders. Examinations of the AMMs show that the AMMs are not in agreement on the
orders of Agile practice introduction [127, 157]. Currently, no studies have been con-
ducted to compare the suggestions of the AMMs against the strategies used in practice.
Therefore it is not known which strategies for introducing Agile practices would yield
more successful outcome than others.
Examination of the evidence indicates that there is no conclusive evidence on
whether one strategy to introduce Agile practices is better than others. The lack of con-
clusive evidence brought our attention to the importance of a baseline situation, i.e. sit-
uation prior to Agile practice introduction [167] when examining the evidence. The im-
portance of reporting the baseline situation has been addressed previously [111, 167].
However, currently there is no guideline that aid researchers to identify which infor-
mation needs to be captured pertaining to a baseline situation.
The research gaps identified in this thesis are outlined as follows:
• Flexibility and Agile and Lean practices are often associated. However, the ex-
tent to which Agile and Lean practices address flexibility has yet to be examined.
• Different strategies being proposed in the AMMs to introduce Agile practices.
However, little is known which strategies are better than others.
• Currently, no guideline provides what information needs to be captured when
reporting baseline context.
This thesis addresses the aforementioned research gaps through the following main
contributions:
• C1. Examine the extent to which Agile and Lean practices address flexibility.
• C2. Examine and compare Agile practice introduction strategies and their im-
pacts. This will improve the understanding of whether one strategy in introduc-
ing Agile practices can lead to more successful outcomes than others.
1.4 Research Questions 9
• C3. Formulation of a guideline to help with capturing information regarding

baseline situation. This will improve the understanding of the extent of improve-
ments from Agile practice introduction.
There is a need to understand whether one Agile introduction strategy is better

than others. Contribution C1 is addressed by understanding how flexibility is improved
in a software organization (Chapter 2) and examining the extent that Agile practices
address flexibility attributes collected from the literature (Chapter 3). Contribution C2
is addressed by aggregating the evidence on the impacts of Agile practices (Chapter 4)
and examine the relevance of Agile adoption guidelines, particularly AMMs, against
industrial practice (Chapter 5 and 6). Contribution C3 is addressed by synthesizing
the findings from the literature study and empirical studies to formulate a guideline to
capture baseline context information (Chapter 7).
1.4 Research Questions

The following research questions are formulated:
• RQ1. To which extent do Agile and Lean practices address flexibility?
• RQ2. Which strategy for introducing Agile practices would yield to more suc-
cessful outcomes?
• RQ3. What information should be captured in a baseline situation when studying

the impact of introducing Agile practices?
The first contribution (C1) of the thesis is to provide a unified view of flexibility
attributes and use it to examine the extent that Agile and Lean practices address flexi-
bility. For that contribution, a research question on examining the extent of Agile and
Lean practice in addressing flexibility is formulated (RQ1). RQ1 is linked to Chapters
2 and 3, as the question investigate the attributes of flexibility and how they relate to
Agile and Lean practices.
The second contribution (C2) examines the evidence on whether there are strategies
to introduce Agile practices that yield a more successful outcome. For that, RQ2 is
formulated. RQ2 is focused on examining the evidence on strategies in introducing
Agile practices, as recommended by the AMMs and how they are done in industry.
Chapters 5 and 6 are linked to both RQ2.
The results from Chapter 4 indicate the evidence pertaining the improvement from
implementing Agile practices on quality, time, and cost is inconclusive. The results
10 Introduction
from Chapter 6 indicate that the AMMs do not provide consistent suggestions in in-
troducing Agile practice. Furthermore, practitioners perceive that their Agile imple-
mentation to be successful though they do not follow the suggestions from the AMMs.
These discrepancies brought our attention to the importance of examining a baseline
situation prior to Agile transformation [167]. The third contribution (C3) pertains to
the identification of which information should be captured when studying the impacts
of introducing Agile practices. To address contribution C3, RQ3 is formulated. Chap-
ter 7 is linked to RQ3 as it presents the descriptions of a baseline situation in the form
a checklist.
The mapping of the contributions, research questions, and the individual chapters
is shown in Table 1.2.
Table 1.2: Mapping of research questions and chapters in the thesis.
Contributions Research Questions Chapters

C1 RQ1 Chapter 2, Chapter 3
C2 RQ2 Chapter 4, Chapter 5, Chapter 6
C3 RQ3 Chapter 7
1.5 Research Method

Different research methods were used in this thesis: grounded theory, literature studies,
survey, and expert validation. Each method was selected given its suitability to address
the research questions described in Section 1.4. A mapping of the research methods to
the individual chapters and the research questions they address is shown in Figure 1.2.
Subsections 1.5.1 – 1.5.4 describe the methods used in this thesis, and subsection 1.5.5
describes the validity threats relevant to this thesis.
1.5.1 Grounded Theory

Grounded theory was used to investigate how flexibility is improved in a software orga-
nization. The grounded theory study is reported in Chapter 2. Understanding how flex-
ibility is improved in a software organization contibutes to addressing RQ1. Grounded
theory is an inductive research approach. Instead of deducing a theory, grounded the-
ory starts with the data to formulate a theory. Grounded theory involves iterative steps
between data collection analysis and allows the researcher to continuously interact the
data and the emerging analysis [37].
1.5 Research Method 11
Grounded Chapter 2
theory
Literature Chapter 3
Literature Review
Studies
Tertiary Chapter 4
review
Questionnaire
Chapter Chapter
Survey
5 6
Interviews
Expert
Chapter 7
Validation
RQ1 RQ2 RQ3
Figure 1.2: Mapping of research methods, research question and the individual chapters
The aim of Chapter 2 is to capture the process of building software organization

flexibility through modifications of organization constituents described in subsection
1.2.1. Grounded theory is suitable for a study when little is known about the study
area, and the phenomenon is socially constructed [22, 36]. Grounded theory suits the
purpose of the study because we want to the understand the process of improving soft-
ware organization as it is done in a software organization and avoid imposing our pre-
conception. In this case, a priori constructs might bias and deviate the data collection
and data analysis process [37]. Instead of building the theory from the data, a priori
construct might force the data to fit the construct [37, 65].
The means of data collection in the grounded theory study is semi-structured in-
terviews. Interview allows us to gather rich qualitative data from the subjects [235].
Semi-structured interview was selected as an interview strategy because it allows the
interviewer to modify the questions based on the respondents’ answer while adhering
to specific topics [235]. Semi-structured interviews were aimed to get a holistic un-
derstanding of the situation in the organization, what changes had been faced, what
kind of challenges were faced, what had been done to address the change, and how
they were perceived. Apart from interviews, project documentation, company policies,
standards, and lessons learned were also collected. Documentations were an important
data source for data triangulation purposes [37, 182].
12 Introduction
In grounded theory, the sampling strategy is purposive or theoretical sampling [37,

182]. In recruiting the industry partner, we were interested in a software organization
that has gone through internal changes or re-organizations. Brief details on the case
company can be found under the heading Case Company. More details on recruiting
interviewees can be found in Chapter 2.
Case Company
For Chapter 2, an empirical study was conducted at an IT department that provides

services to one of the business units of a Fortune 500 financial institution. The business
unit provides financial service for private and corporate clients in investment banking,
assets, and wealth management. The IT Department itself has around 200 developers
who take part in developing, hosting and maintaining the software solutions required
by the business unit. As part of their effort to be more flexible, the IT department has
undergone several changes in their organization over the past decade.
1.5.2 Literature Studies

In this thesis, two literature studies were performed: a literature review and a tertiary
review. The literature review is reported in Chapter 3 to address RQ1. The tertiary
review is reported in Chapter 4 aggregates the impacts of Agile and Lean practices
which contributes to addressing RQ2.
Literature Review
A literature review is “the use of ideas in the literature to justify the particular approach
to the topic, the selection of methods, and demonstration that this research contributes
something new” [87]. A literature review is a form of a secondary study which aims to
aggregate findings from relevant primary studies [128].
In Chapter 3, a literature review was conducted instead of a traditional systematic
literature review (SLR) as suggested by Kitchenham and Charters [110]. A traditional
SLR would be more suitable if the aim of the study were to aggregate evidence on the
impact of an intervention in a certain population. In our case, we were not investigating
the impacts of a certain tools or technology. We adapted the literature review steps
suggested by Levy and Ellis [128]. More details on the review steps can be found in
Chapter 3.
The purpose of the literature review reported in Chapter 3 is twofold: (1) to provide
a unified view of flexibility attributes for a software organization, and (2) to examine
the extent that Agile and Lean practices address the flexibility attributes. This in turn
addresses RQ1.
Tertiary Review
A tertiary review is a systematic review of secondary studies [110]. In Chapter 4, a
tertiary review was conducted because there are already many secondary studies in
Agile and Lean [110]. A tertiary review has the same steps as a systematic literature
review. The protocol described in [110] were used as a guideline. More detail on the
review steps can be found in Chapter 4.
The purpose of the tertiary study reported in Chapter 4 is twofold: (1) get an
overview and trends of published secondary studies in Agile and Lean, and (2) to iden-
tify Agile and Lean practice and their impacts on project constraints, and examine the
level of empirical support. This contributes to addressing RQ2.
1.5.3 Survey
A survey was used in two chapters: Chapter 5 and 6. Both chapters address RQ2. Sur-
vey is “an organized method of obtaining data from research participants via written,
electronic, or oral questioning, which can be analyzed to reach conclusions about the
question being investigated within a sample of the population” [29].
The aim of Chapter 5 is to identify which Agile practices had been abandoned by
practitioners and rationales for abandoning them. Meanwhile, the aim of Chapter 6 is
to identify strategies for introducing Agile practices in the industry and compare them
with the ones described in AMMs. A survey was selected because it allows us to gather
information from a large population [182]. A survey is preferred because it allowed us
to identify rationales and different strategies from a variety of contexts. A case study
would limit our finding to one or two specific strategies from one specific context.
The studies reported in Chapter 5 and 6 employed two data collection approaches:
(1) web-based questionnaire, and (2) interviews [182, 29].
Questionnaire
A web-based questionnaire was selected because it is a cost effective way to gather
data from large number of respondents. A web-based questionnaire also allows a large
data points to be collected in a shorter amount of time [182].
In this questionnaire convenience sampling was used, which is a common strategy
in software engineering surveys [131]. Personal contacts and well-established profes-
sional groups in Agile software development on LinkedIn and Google Groups were
14 Introduction
used as channels to distribute the questionnaire. Distributing questionnaires over pro-

fessional groups is a known way to distribute surveys as reported in [123, 212].
The questionnaire included 17 Agile practices which were adapted from [212]. The
list of included Agile practices and their descriptions can be found in Appendix A of
this thesis. The questionnaire primarily inquires which Agile practices are in used and
when each practice was introduced, and if they are still in use. The outline of the
questionnaires can be found in Chapter 5 and 6. Detailed questions that were included
in the questionnaire can be seen in Appendix B of this thesis.
Interviews
Interviews were conducted to complement the questionnaire. Interviews allow more in
depth information to be collected from the subjects, which a web-based questionnaire
may fall short [182, 235].
The potential interviewees were recruited from questionnaire respondents who were
willing to be contacted for further inquiries. The list of interviewees can be seen in
Chapter 5 and 6.
Semi-structured interview was the selected interview type. Semi-structured inter-
view can guide the interviewee with a set of pre-defined topics, with the freedom to
modify the questions based on the interviewees’ response [182, 235]. A summary of
the respondent’s survey answer was sent prior to the interview. The detailed interview
design can be found in Chapter 5 and 6.
The follow up interview was done to confirm and clarify the respondents’ answers
to the questionnaire and also inquire additional information the questionnaire was un-
able to capture. The interviews together with the questionnaire allowed us to capture
the strategies used by practitioners in introducing Agile practices, which addresses
RQ2.
1.5.4 Expert Validation

Expert validation was performed to statically validate the components of the proposed
checklist described in Chapter 7. The expert validation was performed to improve the
validity of the answer to RQ3. The principle of the expert validation is similar to a
questionnaire.
We sent a summary of the checklist and the propositions to the experts together
with a short questionnaire, inquiring the experts’ perception on the usefulness and rel-
evance to research, and novelty of the identified component. The questionnaire was
self-administered given the experts’ availability. However, if the expert was available,
a video call was also established.
The experts were selected given their publication activities in Agile methodology,
or past and current participation as PC members in Agile conferences. In total, we
received feedback from seven experts from seven different universities in Europe and
the UK, which includes five professors and two associate professors.
1.5.5 Validity Threats

We followed the categorization of validity threats recommended by Petersen and Gen-
cel [165]. Four types of validity threats relevant to this thesis are discussed: (1) theoret-
ical validity, (2) descriptive validity, (3) interpretation validity, and (4) generalizability.
Theoretical validity pertains to the issue of capturing the construct intended to be
collected. This threat is associated with poor data collection and selection of subjects.
Poor data collection is particularly pertinent in a questionnaire (Chapter 5 and 6). To
mitigate this issue, the questionnaire was piloted with five experts in academia and
five industry practitioners with experience in Agile software development. Selection of
subject is critical to the studies in Chapter 5 and 6. The interviews performed for the
study in Chapter 5 and 6 utilized convenience sampling. The interviewees participated
in the study based on their willingness. This led to a large variety of contexts, which
can affect generalizability. This issue cannot be fully mitigated.
Descriptive validity is concerned with the accuracy of capturing the reality. Post-
rationalization is associated with descriptive validity. In Chapter 2, 5, and 6, inter-
viewees were done as the main data collection form. The interviews in all three
chapters entails asking the interviewees to reflect on past events. This leads to post-
rationalization, where interviewees provide a different account of what happened not
to deceive but because it was what they believed happened. Post-rationalization is well
documented as a validity threat in medicine [77] and law studies [236] . In Chapter
2, post-rationalization was mitigated to a certain extent, since documentation was pro-
vided as a means for data triangulation. In Chapter 5 and 6, post-rationalization could
not be eliminated completely as only one interviewee was available in each context and
no documentation was provided as a means of data triangulation.
Interpretive validity is concerned with researchers’ bias in drawing a conclusion.
In Chapter 2, to mitigate researchers’ bias, two researchers were involved in data col-
lection and piloting of qualitative coding was also performed. In Chapter 5 and 6,
post-hoc validation was performed on the coded interview transcript. Furthermore, the
interview questions were formulated based on the respondents answers to the question-
naire. The interview question was directed as clarifications, which leave little room for
loose interpretation.
Generalizability pertains to which extent the results of the study are generalizable
to a larger population. In Chapter 2, the study was performed only in one company
16 Introduction
which could limit the applicability of the results outside the case company. To mini-
mize this risk, organizational change scenarios were selected carefully to ensure that
they cover the different organization constituents described in subsection 1.2.1. Fur-
thermore, a detailed description of the case company contexts and the change scenarios
were provided to improve generalizability [166].
The validity threats concerning the literature studies in Chapter 3 and 4 are selection
bias and missing important papers. To avoid the selection bias, post-hoc validation of
the study selection and quality assessments were done. To minimize missing important
papers, the search string used was piloted and consulted with an expert in the area.
More details of the validity threats and how they were mitigated can be found in
the individual chapters.
1.6 Overview of Chapters

Figure 1.3 provides an overview of the findings from the individual chapters. The
figure also shows the connection between the chapters and how each chapter linked to
the contributions.
Chapter 2 is a grounded theory study to better understand what kind of changes
a software organization face and what course of actions was taken to address those
changes as part of improving flexibility. The study was conducted at the case company
described in Subsection 1.5.1, which within the last decade had undergone multiple
iterations of organizational change to improve flexibility.
The study indicates that flexibility improvement is only made if there is emerging
flexibility need due to uncertainty. Flexibility improvement, in the form of an organiza-
tional change, was specifically done to target the respective uncertainty. For example,
there is a need for flexible human resource allocation due to uncertain project funding,
to achieve that the organization structure had to be modified. Improving flexibility has
its cost [118, 145, 226], not always in financial terms, it could, for example, be reduced
motivation, un-used deliverables, or difficulty in knowledge retention. Therefore, it is
important that the flexibility improvement is targeted for a specific instance or range of
uncertainty.
Goal/Main Contribution: Examine the evidence on the
strategies for introducing Agile practices
Contribution Findings
Flexibility improvement in a software

Chapter 2
C1.1 organization is assessed based on the
baseline situation.
RQ1 Identification of flexibility attributes and their

mapping to flexibility attributes. Uncover that
1.6 Overview of Chapters
C1.2 Agile and Lean practices are aligned with Chapter 3 motivates
management and engineering best practices
and techniques. motivates
Identification of the different impacts of Agile

Chapter 4
practices on project constraints. Evidence of
C2.1 improvements from implementing Agile fulfills
practices is inconclusive.
contributes
motivates
Agile practices have been abandoned,

RQ2 some more often than other. Rationales for contributes motivates
Chapter 5
abandoning Agile practices were identified.
Measures of success are also identified.
complements
C2.2
Strategies used by practitioners are not
aligned with the strategies suggested by the
AMMs. Uncover that some Agile practices are Chapter 6
actually pre-existing practices.
contributes
Aggregating baseline context information to Chapter 7

RQ3 C3 assess the extent of improvement from
adopting Agile practices.
17
Figure 1.3: Connections between the chapters.

18 Introduction
The results of the study also indicates flexibility is also about stability as much
as it is about changes. A view which is also shared by Regev et al. [177]. The case
company modified a specific aspect of the organization, e.g., organization structure,
to address a specific range of uncertainty. The organization structure is not constantly
being modified. It remained stable until new flexibility need emerged. Addressing all
types of changes and uncertainty will make it difficult to plan an appropriate course of
flexibility improvement action, and increase the cost of flexibility improvement. One
needs to prioritize which flexibility needs are the most important.
Chapter 3 aggregates the attributes of flexibility from interdisciplinary literature.
The aggregation of flexibility attributes was presented as a flexibility framework for
software organizations. As a proof-of-concept, the framework was used to evaluate
the extent to which Agile and Lean practices address flexibility. The evaluation results
show that Agile and Lean practices address a wide range of flexibility attributes, but
not all. Furthermore, the evaluation result indicates that the underlying principles of
Agile practices are not new. They are previously known as mechanisms for improving
flexibility in other disciplines like manufacturing or management [91, 197]. It is likely
that these principles had been in place in a software organization before Agile practices
were introduced.
From the flexibility framework, we learned that there is a range of flexibility. Each
software organization can have its flexibility range. One of the lessons learned from
Chapter 2 is flexibility improvement should be targeted for specific flexibility needs.
Therefore it is important for practitioners to assess what range of flexibility is required.
For example, if the requirement changes are not frequent, then it is unnecessary to have
a process to deal with frequent changes.
The point that we are trying to get across with the flexibility framework is not
to tick all the flexibility attributes, but identifying which attributes are important. As
indicated in Chapter 2, flexibility comes at a cost [226]. Furthermore, Regev et al.
[178] indicate that by being too flexible, an organization can risk losing its identity
because changes are continuously being implemented. Therefore, it is preferable to
have the appropriate range of flexibility. The flexibility framework described in this
paper could be used as a diagnostic tool identify the most relevant flexibility attributes
for a software organization (more details can be seen in Chapter 3).
Chapter 4 is a tertiary review that aggregates the positive and negative impacts
of Agile and Lean practices on project constraints: scope, quality, schedule, budget,
resources, risk, and communication. The tertiary review shows empirical studies that
are included in the secondary studies report contradictory impacts of Agile and Lean
practices on different project constraints. For example, one of the secondary studies,
Causevic et al. [33], reported that there are three academic experiments that indicate a
positive impact of TDD on development time, yet there are six industry case studies that
1.6 Overview of Chapters 19
indicate a negative impact of TDD on development time. Furthermore, publication bias

was observable in secondary studies with low quality scores in the quality assessment.
Secondary studies with low quality scores tend to include primary studies that report
positive impacts of Agile and Lean practices.
From the tertiary review, it is difficult to make strong assertions on the impacts
of each Agile and Lean practice on project constraints, because the secondary studies
often do not report the contexts of the primary studies, in some cases the type of re-
search method is unclear. However, the lack of contextual information could be derived
from the primary studies themselves. As reported by Dybå and Dingsøyr [58], primary
studies in Agile software development often do not provide sufficient contextual infor-
mation, as well as detailed study design.
Although the tertiary review does not provide conclusive remarks, it highlighted
the importance of reporting the contexts in empirical studies concerning Agile software
development. It is known that Agile practices are highly contextualized. How the prac-
tices are implemented may vary from one context to another [72, 239]. Multiple studies
have demonstrated the importance of reporting contextual information, as contexts can
influence the success or failure from Agile implementation [62, 176, 167, 103]. Agile
practices are could be successful in one context, but not in another. However, with-
out clear description of the context, it would be difficult to make accurate comparisons
[166]. The results of the tertiary review also motivated us to look more closely into Ag-
ile adoption guidelines, i.e. the study described in Chapter 5 and 6, also the formulation
of the guideline presented in Chapter 7.
The proponents of AMMs claim that introducing Agile practices in certain orders
could amplify benefits and alleviate challenges of Agile practices [144, 160, 202].
Chapter 5 is motivated by the popularity of AMMs as Agile adoption guidelines.
AMMs suggest Agile practices should be continuously added (see Subsection 1.2.4),
but recent Agile surveys indicate that some Agile practices like TDD and continuous
integration are being abandoned [142, 212]. The suggestion from the AMMs appears to
be contradictory to recent Agile surveys. However, little is known why Agile practices
are being abandoned. Chapter 5 presents the rationales for abandoning Agile practices
through a survey described in subsection 1.5.3.
In total, 51 industry practitioners completed the questionnaire, and 11 of them par-
ticipated in the interviews. Eighteen respondents indicated that they had abandoned
one or more Agile practices. All 17 Agile practices included in the questionnaire were
abandoned at some point. Some more frequently than others. Tracking progress has the
highest abandonment ratio. The most common rationale for abandoning Agile practices
is the lack of perceived values; other rationales include: (1) lack of product knowledge,
(2) lack of engagement from the team, and (3) influence of a person; the person who in-
20 Introduction
troduced the practice left company, one highly opinionated person convinced everyone
in the team to abandon one or more practices.
The practitioners generally perceived their Agile implementation to be successful,
including those who abandoned one or more practice. There are different varieties of
measures that the practitioners reported to indicate success. This suggests that different
strategies to introduce Agile practices could lead to success, depending on how success
is measured. This observation is aligned with a previous study by Solinski and Petersen
[212] that suggests practitioners have different priorities on the perceived benefits and
limitations of introducing Agile practices.
Chapter 6 is an extension of Chapter 5. In Chapter 6, a more in-depth examina-
tion of the AMMs was performed. We found 26 AMMs, but only 12 of them were
more closely examined because they include mappings of Agile practices and maturity
levels. The result of the AMMs examination shows that AMMs provide conflicting
suggestions pertaining which practices are associated with which maturity levels. An
observation which is also supported by Leppänen [127].
We found a number of discrepancies between AMMs and practitioners’ strategies.
For example, the practitioners typically introduce face-to-face meeting first, but AMMs
suggest to introduce face-to-face meeting at maturity level 2 or 3. A number of inter-
viewees also indicated that some Agile practices like refactoring, short iteration, and
self-organizing team have existed prior to Agile transformation. The AMMs do not
take such situations into account. If one is to adopt an AMM, it is unclear whether one
must stop the pre-existing Agile practice. This indicates that AMMs provide insuffi-
cient suggestions for practitioners.
In general, the practitioners perceive that their strategy to introduce Agile prac-
tices to be successful. We also did not find indication that the suggestions from the
AMMs are representative of the industry’s strategy. This indicates that depending on
the contexts, any strategy to introduce Agile practice can yield to successful outcome,
depending on what measures are used to indicate success (lessons learned from Chapter
5).
From the interviews, we learn that the situation prior to Agile transformation, or
baseline situation, in the organization may contribute to the successful implementa-
tion of Agile practices. From the interviews, we inferred that baseline situation, such
as familiarity with Agile practices, management support, and experienced and knowl-
edgeable team members might influence the outcome of introducing Agile practices.
This inference is supported by the findings in Chapter 3 that indicates Agile principles
are known mechanisms in manufacturing and management. Thus, it is possible that
these mechanisms are in place prior to Agile practices. These findings are then used as
an input to formulate a guideline described in Chapter 7.
1.6 Overview of Chapters 21
In Chapter 7, a compilation of baseline context information that needs to be cap-

tured and reported when studying the impacts of introducing Agile practices is pre-
sented as a checklist. The checklist is developed from the interviews conducted for the
study in Chapter 5 and 6.
The checklist contains 11 items which are divided into four categories of baseline
context information:
1. Constellation of team members’ competencies, experience, and skills; team mem-
bers’ product knowledge, software development experience, and team members’
technical and non-technical skills
2. Management principles that are in place; organization structure, self-organization,
openness, and recursiveness.
3. Existing practices or activities; Formal/informal practices that are similar to Ag-
ile practices.
4. System characteristics of the software under development; legacy code, modu-
larity, and hardware components
Seven experts in academia validated the checklist components and the propositions.
From the validation, the experts perceive the checklist is relevant to research and could
be useful for other researchers to reflect and improve their study design. The experts
also indicated that the individual components are not new, but framing them as baseline
situation that needs to be captured when studying the impacts of introducing Agile
practices is where the novelty lies. The detailed checklist can be seen in Chapter 7.
From the checklist, we formulated four propositions. The propositions are aimed
to understand the influence of a baseline situation in enhancing or reducing the ben-
efits of Agile practices. We also asked the same experts for their perceptions of the
propositions. The experts mentioned that they appreciated how the propositions were
formulated. However, each proposition needs to be accompanied by a detailed explana-
tion. Based on the experts’ feedback, we improved the formulation of the propositions
and provided each of them with an explanation. The improved propositions can be seen
in Chapter 7.
Contextual factors can influence the impacts of introducing Agile practices [58, 62,
103]. However, most studies emphasize on the current situation or contexts; little atten-
tion has been put on reporting a baseline situation prior to Agile practice introduction
[167]. The feedback obtained from the expert evaluation validated our suggestion to
capture and report a baseline situation in empirical studies concerning the introduction
of Agile practices. Furthermore, investigating to which extent the benefits of introduc-
ing Agile practices are influenced by a baseline situation is worth pursuing.
22 Introduction
1.7 Synthesis
To synthesize the results obtained in this thesis, the research questions are revisited:
• RQ1. To which extent do Agile and Lean practices address flexibility? In

this thesis, we aggregated flexibility attributes from inter-disciplinary literature.
The aggregated attributes were then presented as a flexibility framework. The
flexibility framework was then used to assess to which extent Agile and Lean
practices address flexibility. Agile and Lean practices address a range of flexi-
bility attributes, but not all. Agile and Lean practices address primarily product
and information flexibility, also on addressing frequent and unforeseen changes.
There are other flexibility attributes that Agile and Lean practices do not ad-
dress (Chapter 3). However, it is not necessary to address all possible flexibility
attributes, because improving flexibility can be costly (Chapter 2). It is more im-
portant to focus on a range of flexibility attributes that are required by a software
organization. The flexibility framework described in this thesis could potentially
be used to identify required flexibility attributes.
• RQ2. Which strategy for introducing Agile practices would yield to more suc-
cessful outcomes? From the studies presented in this thesis, there is no conclu-
sive evidence that indicates one or some approaches to introduce Agile practices
are better than others. The Agile Maturity Models (AMMs) recommend that
Agile practices are to be introduced incrementally in certain orders. However,
the recommendations provided in the AMMs are not representative of how Ag-
ile practices are introduced in industry (Chapter 6). Regardless of the selected
strategy, practitioners perceive their Agile implementation to be successful. This
indicates that any strategy can lead to success. However, it is important to high-
light that practitioners have different priorities on what success entails (Chapter
5). If a strategy delivers the benefits that the practitioners expect, then it would be
considered successful. Also, each strategy may only be appropriate in one spe-
cific context. Thus, we cannot claim that there is one “best” strategy to introduce
Agile practices.
• RQ3. What information should be captured in a baseline situation when study-

ing the impact of introducing Agile practices? Reporting contextual information
in empirical studies is important. Most studies focus on the current situation, but
a previous situation or baseline situation is equally important. The information
that needs to be captured includes the constellation of team members’ compe-
tencies, experience, and skills, management principles, existing practices, and
the characteristics of the software systems under development (Chapter 7). The
1.8 Conclusions and Future Work 23
experts whom we consulted found the compilation of information to be useful

for other researchers to improve their study design.
1.8 Conclusions and Future Work

Every organization, including software organization, strives to be flexible to maintain
a competitive advantage. Introducing Agile practices can help software organizations
to be more flexible. However, software companies need to be aware that introducing
Agile practices have their respective benefits and challenges.
There are many Agile adoption guidelines, such as Agile Maturity Models (AMMs)
that are proposed to help software organizations to amplify the benefits and alleviate
the challenges of Agile transformation by introducing Agile practices in certain orders.
However, the examination of evidence described in this thesis indicates that there is
no conclusive evidence to support that introducing Agile practices in a specific order
would yield more successful outcomes.
This thesis provides aggregation and examination of evidence on the strategies for
introducing Agile practices. The lack of conclusive evidence should not be perceived
that there are no strategies to introduce Agile practices that would yield to success.
Practitioners have successfully introduced Agile practice using their own strategies and
not from a specific guideline. Each strategy works in its context and may not be easily
transferable to a different context. Therefore, it is important that researchers provide
detailed contextual information in their empirical study reports, including the baseline
situation prior to Agile practice introduction.
This thesis presents a checklist for capturing baseline information when studying
the impacts of introducing Agile practices. Seven experts in academia validated the
checklist and perceived the checklist to be relevant and useful to research. The checklist
proposed in the thesis could be used by researchers who are interested in examining
the extent of improvement from introducing Agile practices. The checklist can help
researchers to improve their study design and subsequently provide unbiased inferences
on the extent of improvements from introducing Agile practices. The checklist also
offers consistency in reporting empirical studies concerning Agile. Such consistency
will make comparing and aggregating evidence easier, especially in evidence-based
software engineering. Better aggregation of evidence can then benefit practitioners
to make a fair assessment of the relevance of research results. Thus, helping them
in making better decisions in Agile practice introduction. Understanding a baseline
situation can help practitioners to assess potential cost and effort of introducing Agile
practices, or be aware of pre-requisites of introducing Agile practices.
24 Introduction
Future work needs to be focused on validating the checklist and testing the proposi-
tions formulated in this thesis. In the checklist and propositions, there are a number of
organizational aspects which are posited to have impacts on the extent of improvements
from Agile practice introduction. Future research needs to isolate to what benefits are
due to the organizational aspects, and what benefits are actually due to Agile practices.
The findings can then be used to better understand the actual benefits of Agile and
to what extent the organizational aspects amplify or decrease them. The propositions
formulated in this thesis can be used to guide such studies.
The idea of the checklist is to bring awareness of the importance of capturing and
reporting organizational aspects as baseline information in empirical studies. An idea
which has been validated by seven academic experts. We do not claim that the checklist
is complete, future research should also be directed towards improving the checklist
and complementing it with additional factors. Also, currently the checklist is geared
towards empirical studies focusing on Agile. Future research should also be directed
towards modifying the checklist for other areas of interest within software engineering.
Chapter 2
An Analysis of Change
Scenarios of an IT Organization
for Flexibility Building
This chapter is based on the following paper:

Indira Nurdiani, Samuel A. Fricker, Jürgen Börstler, “An Analysis of Change
Scenarios of an IT Organization for Flexibility Building”, in Proceedings
of the 23rd European Conference in Information Systems (ECIS), 2015,
(DOI:10.18151/7217436)
2.1 Introduction
A software organization is an entity that develops and delivers software to an external
mass market, a company, or supports business operations within the same company.
An IT organization is a form of software organization that operates, develops, and
manages human resources, processes, and infrastructure (both hardware and software)
that supports and is of importance for the business [30, 116].
Software organizations in general, and IT organizations in particular, need to be
flexible and cope with the changes demanded in the business environment. Changes
not only originate from the business (customers of an IT organization), they can also
originate from suppliers, competitors, governing bodies, and also technological de-
velopment [30, 116]. In a turbulent business environment, IT organizations need to
26 An Analysis of Change Scenarios of an IT Organization
develop and improve strategies to be flexible. Without flexibility, an IT organization

might lose its ability to support the business’ market position and competitive advan-
tage [30].
Despite the importance of flexibility, it is not well elaborated how organizational
flexibility is built. There are known prescriptive approaches in software engineering,
such as Agile methodologies [13, 191], software product line engineering [41], and
software product management [76]. Agile methodology emerged as an approach that
offers flexibility by welcoming changes in customer needs [13]. Software product line
engineering also offers flexibility through variability management [38]. The area of
product planning decision also offers approaches to achieve software product flexibility
[76]. These approaches offer prescriptive suggestions on how to include flexibility in
the software product, but not at an organizational level.
In this chapter, we elaborated events that took place as an IT organization built and
improved its flexibility. These events are then presented as a model that was developed
through grounded theory. The data were collected through interviews with managers
to identify scenarios where flexibility was built and improved in the organization. The
scenarios that took place in the case company were related to change of organization
structure, implementation of a new software development method, and implementa-
tion of a new legislative regulation. The elaboration of these events can be used as a
stepping stone to further improve flexibility in a software organization.
The remainder of this chapter is organized as follows. Section 2.2 presents the
background and related work. Section 2.3 describes the research methodology. Section
2.4 presents the results and analysis. Section 2.6 presents the discussions of the result.
Lastly, the chapter is summarized and concluded in Section 2.7.
2.2 Background and Related Work

Organizations can be conceptualized as entities that interact with an environment through
inputs, conversions, and outputs [95, 102], as shown in Figure 2.1. The environment
refers to the entities that provide inputs and consume the outputs generated by the or-
ganization. The environment includes customers, legislation, suppliers, competitors,
and other stakeholders. Inputs refer to the things that the organization obtains from the
environment to produce value such as raw materials, human resources, and informa-
tion. Conversions refer to the constituents that the organization requires to transform
the inputs into something of value. For an IT organization to be flexible its constituents
need to be built for flexibility. These constituents are workforce, management, pro-
cesses, organizational structure, and infrastructure [218]. Workforce refers to human
resources and their skills and knowledge. Process refers to concerted activities that take
2.2 Background and Related Work 27
place in the organization, for example a development process. Organizational structure

refers to the organization hierarchy and roles and how they relate to each other. In-
frastructure refers to technology platforms that are required to conceive, build, release,
and evolve products and services. Outputs refer to the elements that the organization
releases to its environment and include products, services, and value that are released
to the environment.
Environment
Inputs Conversions Outputs

IT Organization
Workforce
Management
Process
Organization
Structure
Infrastructure
Figure 2.1: Organization Model Adapted from [102] and [218]
Flexibility is a multidisciplinary concept that relates to the ability of an entity to

change easily and effectively [54]. There are different definitions of flexibility depend-
ing on the context and domain [197]. Across the different definitions of flexibility it can
be summarized as the ability of an entity to change with: (1) effectiveness, timeliness,
and satisfaction with a change [106], (2) balance of the amount of change and stability
[177], (3) minimal difficulty, cost, time, effort, and risk of the change [54, 91, 197], (4)
and the universality of the entity expected to cope with variations of input [116].
Manufacturing recognizes that flexibility needs to be built from different aspects
of the manufacturing processes, such as, material handling, production line, to market-
ing process, as well as management and organizational structure [197]. Flexibility is
required to cope with internal and external uncertainties [54, 197] and achieve com-
petitive advantage [24, 197]. To build flexibility, manufacturing literature suggests the
importance of identifying the required flexibility derived from the uncertainties, im-
plement the flexibility building decision, and perform audit of the planned and actual
achieved flexibility [24, 54, 80]. However, the intangible, changeable, complex nature
of software and the unpredictability of development projects differs from manufac-
turing of physical goods [27]. Thus, approaches in manufacturing may not be easily
transferable to IT organizations.
Literature in management has two perspectives of organization flexibility. The first
perspective views organization flexibility as how quickly an organization can change
its structure given the changes in the environment [91]. This can be achieved by mod-
ifying people’s roles and responsibilities [227], or having a manager with openness to
change [199]. The second perspective views organization flexibility as how the orga-
nization structure can remain stable given changes in the environment [52]. This can
be achieved by employing highly skilled workers [197], and adopt a flat organization
structure to allow quick problem resolution [91]. Management literature has provided
suggestions to achieve organizational flexibility, but there is no descriptive guidance on
how to build flexibility.
The need for holistic view of flexibility is also expressed in IS literature. Byrd et al.
[31] proposed a framework of IS flexibility with four dimensions, people, IT, data, and
process. Each dimension is associated with various factors that support its flexibility
which in turn will support the organization flexibility. Allen and Boynton [6] discuss
two main approaches to achieve IS architecture flexibility, high-road, and low-road.
Each approach has its own pros and cons. Thus organizations need to tailor them to
their own contexts. Current solutions are still lacking descriptive approach that allows
organizations to build flexibility that suits their needs and contexts.
Software engineering has offered approaches on how to build flexibility on the level
of the software product. Software product line engineering (SPLE) is intended as an
approach to deal with mass-customization of software products [41]. This is achieved
through variability management where commonalities and variabilities of a set of soft-
ware products’ artefacts (e.g., requirements, codes, test cases, etc.,) are identified [38].
Agile methodologies enable product flexibility through a process that welcomes fre-
quent user requirement changes [57]. Agile methodologies include practices that en-
able flexibility such as short iterations, collective code ownership, and refactoring [1].
Short iterations allow changes to be integrated as soon as the need for change is discov-
ered from customer feedback [164]. Collective code ownership increases knowledge
sharing and supports flexibility by making it easier to assign people to tasks [1]. Refac-
toring systematizes the actual change of a software system [75].
Today, we lack an understanding of how to build flexible software organizations.
Software engineering literature addresses product flexibility but does not do so on an
organizational level. A holistic organizational coverage is missing. Manufacturing
and IS literature confirms this need for a holistic view [91, 197]. Making one aspect
more flexible might lead to other aspects requiring more control [125, 197]. Without a
holistic view, the trade-offs of flexibility cannot be thoroughly considered [70, 86].
2.3 Research Methodology

We studied an IT Department that serviced one of the business units of a Fortune Global
500 company operating in the financial sector. The company provides services like in-
2.3 Research Methodology 29
vestment banking, asset and wealth management for private, corporate and institutional
clientèle worldwide. The IT Department had around 200 developers who were respon-
sible for developing, hosting, and maintaining the software solutions required by the
business unit.
The IT Department had undergone changes in the past decade as part of their ef-
forts in improving flexibility. With the assistance of the IT Department management
representatives, we identified change scenarios that were (a) sufficiently representa-
tive and (b) covered sufficient kind of change variation that they encountered [139].
In Reorganization 1, the organization shifted from hierarchical to matrix organization.
In Reorganization 2, the organization shifted from matrix to pool organization. Both
Reorganization 1 and Reorganization 2 were intended to improve resource allocation
flexibility. In RUP (Rational Unified Process) Introduction, the organization adopted
RUP as the standard development methodology to improve resources flexibility to sup-
port different development projects. In Regulatory Change Implementation the orga-
nization modified its IT system(s) due to regulatory change that the business had to
comply.
Together, these four scenarios covered all five constituents of an IT organization
which were mentioned by Tapanainen et al. [218]. Reorganization 1 and Reorganiza-
tion 2 influenced workforce, organization structure and management. RUP Introduc-
tion influenced the development process, and workforce’s knowledge of RUP. Regu-
latory Change Implementation influenced the infrastructure of the organization, as the
system becomes part of their infrastructure.
2.3.1 Data Collection and Analysis

In this research, we aim to reveal a holistic view on how flexibility is built or improved
in an IT organization to cope with changes. To achieve our research aim, we adopted
grounded theory as suggested by Charmaz [37] to uncover the events that take place
as an IT organization improves its organizational flexibility. The following research
question was formulated to guide our study:
RQ1. How does an IT organization build or improve flexibility?
Grounded theory is a well-established research method used in software engineer-
ing, IT, and IS research [46]. Grounded theory allows us to develop concepts and the-
ories of building flexibility based on the data, evidence and not through pre-conceived
notions [37]. Grounded theory allows us to remain open and critical to what flexibility
building really is.
We performed data collection and data analysis iteratively. Data collection was
done through semi-structured interviews and documentation reviews for data triangu-
lation [182]. The aim of semi-structured interviews is to get a holistic understanding of

the situation in the organization, what changes had been faced, what kind of challenges
were faced, what had been done to address the change, and how they were perceived.
A preliminary list of open-ended questions was developed. These questions were con-
tinuously modified as we progressed with the interviews and concepts were emerging,
as suggested in grounded theory [37]. Documentations were collected to gather stan-
dards, policies, and lessons learned that are relevant to get an overview of the flexibility
improvement efforts that have taken place in the organization.
We conducted interviews with nine managers in the case organization. Each inter-
view lasted approximately 60 minutes, and they were recorded and transcribed. Prior
to performing the interviews with the managers, we conducted interviews with three of
the management representatives of the IT organization to get an overview of the orga-
nization and why it needed to improve its flexibility. They also provided an overview
of recent changes that the organization had to face. From these preliminary interviews,
we saw how the different changes influenced different constituents of the organiza-
tion, as explained by Tapanainen et al. [218]. The representatives then provided a list
of managers that we could interview pertaining to these changes, whilst ensuring that
we covered wide variation of scenarios based on the impact of changes to different
organization constituents.
We followed the coding steps in grounded theory as prescribed in [37]. In initial
coding, we adopted line by line coding. It was selected to allow us to see processes
and distance us from the interviewees’ perspective. We went through the interview
transcriptions line by line and assign one code or more as they emerge from each line.
From the open coding, we identified reccuring themes across interviewees and scenar-
ios.
Then we performed axial coding to categorize the change scenarios based on the
reccuring themes such as, triggering uncertainties, impact of the uncertainties to the
organization, actions that were taken to address uncertainties, and their respective out-
comes with respect to achieved flexibility and relevant trade-offs. The concepts that
emerged from the axial coding can be seen in Table 2.1.
In theoretical coding, we analyzed the relationships between the categories. We
observed that the categories that emerged from the axial coding were a series of events
that took place as an IT organization improved its flexibility. The outcome of the
theoretical coding is shown in Figure 2.4.
2.3.2 Threats to Validity

This research is subject to validity threats common to grounded theory, such as credi-
bility, resonance, originality, and usefulness [37].
2.4 Results and Analysis 31
Credibility is compromised when the research performed did not have data with
sufficient breadth and depth. Thus, the arguments are not strongly supported by the
data. This was mitigated through having multiple interviewees and reviews of the orga-
nization’s documentations. Credibility was also supported by having change scenarios
that covered all five of the organization constituents.
Resonance refers to whether the research performed was able to capture the actual
experience. We addressed this threat by informing our interviewees of our presence,
the purpose of our study, and also the kind of information we would like to elicit from
them. Interviewees also participated on their own will and interest in the study.
Originality is a risk that the outcome of the study did not offer new insights of
flexibility building. The study that we conducted was built on top of existing work
on flexibility from different domains. In this study, we uncovered how the existing
knowledge of flexibility building is applicable for an IT organization. Furthermore, in
this study we uncovered a series of events in building flexibility.
Usefulness is a risk to the applicability of the results beyond the studied setting. To
improve the usefulness, we selected scenarios that covered the different constituents
that were identified byTapanainen et al. [218] which should be present in other IT
organizations. We believe that reorganizations, adoption of a new development method,
and implementation of a new regulation are typical scenarios in other IT organizations
supporting a financial institution.
2.4 Results and Analysis

We uncovered key concepts in flexibility building that were present in all four scenar-
ios (Reorganization 1, Reorganization 2, RUP Introduction, and Regulatory Change
Implementation). The scenarios, flexibility building concepts, and interviewees’ state-
ments (marked S1 – S43) are summarized in Table 2.1. Flexibility building needed a
key person or a group of people who initiated the process. We referred as initiator.
Situation referred to the condition of the organization prior to the modification of the
constituent. Given some internal or external changes, the organization faced variations
or unclear inputs (budget, information, or regulation) referred as uncertainty. Need for
flexibility was the problems or inefficiencies that emerged in the organization due to
the uncertainty. Flexibility building referred to the modifications made to the specific
organization constituent. Achieved flexibility referred to the situation after the organi-
zation constituent modification and how flexibility was improved. Trade-off referred
to the identified disadvantages of the selected flexibility building option. The follow-
ing subsections describe how these concepts contribute to flexibility building of an IT
organization.
Table 2.1: Key Concepts and Relevant Interviewees’ Statement
32
Reorganization 1 Reorganization 2 RUP Introduction Regulatory Change Imple-
mentation
S1:“[The CIO] was a change S10:“[The CIO] decided to S19: “It was from internal gov- S28: “It was my [program
manager” merge them in one big pool or- ernance.” manager] decision what can
ganization.” be descoped.”
Initiator
S2: “We had some groups, S6 S20: “We had [..] templates, but S29: “There is nothing [reg-
and each group was responsi- not really a methodology.” ulated service system] on the
ble for a product and the peo- market available.”
Situation
ple.”
S3:“Years ago we were really S9 S21:“There was no common S30: “The bilateral agree-
stable in the budgeting [..]. way of working, [..] how they ment between [country A]
Now, [..] sometimes they like work was different from project and [country X], [coun-
to invest on [application A], a to project try Y], and [country Z].”
year later they like to invest S31: “This is a complete[ly]
Uncertainty
[..] on a different topic.” new agreement, [..] nobody
really had any experience.”
S32:“Unfortunately, [Country
X] jumped off few days before
going live.”
Continued on next page
An Analysis of Change Scenarios of an IT Organization
Table 2.1 – Continued from previous page
mentation
S4: “If [..] an employee [..] S9 S22: “There was a prob- S33: “[These] regulatory
would like to move [his] man- lem that each project things you have to de-
ager can stop it because they work in their own ways.” liver, there is no choice.”
did not like to have [..] moves S23:“we wanna standardized S34: “[..] to become a com-
to the other department.” the way of working so we can pliant financial institution of
S5: “[The group] SAP im- support each other.” 2012.”
plementation had 180 people.

After a year you do not really
Need for flexibility

need the [rest] 60 people, but
these 60 people could not be
relocated, and they did not
have anything to do.”
S6: “We had a matrix with S11: “We are organized in the S24: “[..] all projects are run- S35: “The soft-
[..] four pools: project delivery unit in five pools: [..] ning RUP.” ware is ready [..].”
management, business anal- project leads, business analysts, S36: “We [..] did it on
ysis, development, and inte- and we have [two] development a make [build] approach
gration. Across we had the pools.” S12: in our infrastructure.”
key accounts, and we had the “So if we have some escala- S37: “We [..] had cho-
Flexibility building
head of this unit called Service tions between these two organi- sen the option which has the
Provider Unit.” zations [pools], then you have to fewest impact on all of the
go to [the CIO].” system.”

33
34
mentation
S7: “People had the pos- S13: “The pool setting of- S25: “Now we all have S38: “We prepared the soft-
sibility to move around, fers flexibility to cope with the same terminology [..].” ware in a way we could
to grow, to have a good demand from the business. S26: “If a new architect coming switch on or off a country.”
future perspective.” You can focus people on to your team and knows how S39: “In the end the system
S8: “We [the staff] had a solution whenever re- you work.” will fit in our environment.”
more power to switch [..] to quired.” S14: S40: “We can steer the whole
other solutions.” “Whatever the business decides, process.”
you can allocate the people
Achieved flexibility
to that particular solution.”
S15: “Here [pool organization],
the responsibilities are clear[er].
You have one organization
responsible for product, one
organization responsible for
deliveries [..].”
S9: “He [a manager] could S16: “People miss having their S27: “At the beginning they S41: “[the option] might
decide himself on an escala- home and may be unmotivated.” thought that is [..] just a window not be the best for the
tion view. The problem [was] S17: “Knowledge manage- to their project.” clients because clients
all these three SPUs [Service ment is much more compli- will get two confirmations
Provider Delivery Units] had cated [..]. Something gets out of one transaction”
Trade-off
small differences in the pro- lost when [people] go away S42: “It [was] not cheaper.”
cess.” from a solution domain.” S43: “You’re the only one
S18: “With the pool I [project who fulfil it with a huge
manager] need time to find investment.”’
people.”
An Analysis of Change Scenarios of an IT Organization
2.4.1 Uncertainty and Need for Flexibility
The initiation of flexibility building was triggered by emergence of uncertainty that the
organization could not cope. Such uncertainty led to the need for flexibility.
In Reorganization 1, the organization was in hierarchical structure (S2), Figure 2.2
left. A problem emerged related to the financial institution’s budget reduction that led
to uncertainty in the focus of the development projects (S3). The need for flexibility
emerged as the hierarchical structure of the organization hindered team members of
one development section to be reallocated to another one (S4). Preferred staffs were
running idle (S5).
In Reorganization 2 the organization was in matrix structure (S6), Figure 2.3 left.
A problem of inconsistent processes across development units or Service Provider
Units (SPUs) emerged due to unpredicted outcome from Reorganization 1. The in-
consistency led to uncertainty in the process of project issues escalation (S9). The
differences in escalation process resulted in confusion and delays.
In RUP Introduction, initially the organization did not have a development method-
ology (S20). The problem was related to inconsistent development methodology that
led to uncertainty on how a project would be performed (S21). This uncertainty limited
the possibility to allocate staff into different projects (S22,S23), as they might require
time to learn how the other project worked.
In Regulatory Change Implementation, new bilateral agreements between finan-
cial institution’s home country were planned with other countries (S30). There were
no IT systems that supported such bilateral agreements (S29). The contents of these
agreements varied across countries (S30), and nobody knew the exact rules that were
going to be agreed (S31). Furthermore, uncertainty escalated as there was a possibil-
ity that a country could withdraw from the agreement (S32). This uncertainty led to a
difficulty in determining what services need to be provided for every bilateral agree-
ment, and if they need to be provided at all. This led to the possibility of performing
unnecessary work.
From the four scenarios, we learned that an uncertainty could originate from out-
side the organization, like changes in the business environment, as shown in Reorgani-
zation 1 and Regulatory Change Implementation scenarios. An uncertainty could also
originate from within the organization and not necessarily caused by changes in the
environment, as shown in Reorganization 2 and RUP Introduction. The uncertainty
limited the software development organization’s ability to operate efficiently or lead to
additional or unnecessary work. These limitations then led to the need for flexibility.
2.5 Build Flexibility

Once the need for flexibility was identified, an IT organization built flexibility through
modifying one or more of its constituents. The role of a decision-maker was crucial for
initiating the required modifications.
In Reorganization 1, there was a need for more flexible resource allocation to high
priority projects (S4,S5). This rigidity of resource allocation led the Chief Information
Officer (CIO) to initiate a reorganization (S1). The option was to shift from hierar-
chical organization to a matrix organization structure, as shown in Figure 2.2, where
staffs were grouped according to their roles (S6) and not by product development. The
new organization structure should accommodate team members’ mobility to be easily
allocated into different projects.
In Reorganization 2, there was a need to overcome the inconsistency of escalation
process across SPUs (Q9). The CIO, again, opted to initiate a reorganization (S10). The
inconsistency needed to be addressed by having a more centralized escalation process.
The option was to shift from was to shift from matrix to pool organization structure,
as shown in Figure 2.3. In the pool organization, the SPUs were merged into different
pools, and all pools only reported to the CIO (S11). The centralized reporting should
eliminate variations in issue escalation process.
In RUP Introduction, there was a need for flexible resource allocation. It was
hindered due to variation of development process across projects. The internal gover-
nance group decided that a common way of working was required to improve flexibility
(S19). The option to address the inflexible resource allocation was by adopting RUP
as a standard organization development methodology (S24). A common way of work-
ing and terminology should ease communication between staff and transitions between
projects.
In Regulatory Change Implementation, there was a need for flexibility in IT ser-
vice provision to support the new bilateral agreements (S30,S32), as well as to remain
compliance with the regulation (S31). For this scenario, we uncovered different options
on how to build flexibility of the IT service.
First option was whether to build or buy the regulated service system. The program
manager (S28) of the implementation program decided on build approach (S36). Prior
to executing the decision, the program manager had foreseen the pros and cons of
building the IT system in-house. The build approach would require less integration
effort (S39), and but it would be more expensive (S42).
There was also an option to build the system with high maintainability or high us-
ability. Prior to executing the decision, the project manager had also seen the pros and
cons of each option. To have better maintainability, the regulated service system would
be decoupled from the rest of the IT system (S37). However this option would sacri-
2.5 Build Flexibility 37
fice usability for the clients (S41). It was decided to go with the option that prioritized
maintainability over usability.
There were different decision-makers involved in deciding how the constituents
were to be modified to build flexibility. They could be from upper level management
or middle level management. Uncertainty that involved the whole organization, as in
Reorganization 1, Reorganization 2 was handled at executive level, the CIO. The RUP
Introduction was initiated by the internal governance responsible for development pro-
cesses. Regulatory Change Implementation was handled by the program manager of
the development program. Furthermore, the need for flexibility influenced which con-
stituents to be modified and how they were modified. Unlike the other scenarios, in
Regulatory Change Implementation, the decision-maker was able to foresee the out-
come of the different options in improving flexibility.
2.5.1 Achieved Flexibility and Trade-Offs

After the constituents were modified, the organization achieved the flexibility that they
required. However, there were trade-offs associated with the achieved flexibility.
In Reorganization 1, the matrix organization structure delivered flexibility in re-
source allocation. The team members were able to move different solution develop-
ment or projects (S7,S8). The freedom for team members to move across projects also
allowed them to improve their career perspectives in the organization (S7).
However, this new flexibility of resource allocation had a trade-off at the manage-
ment level. The separation of the organization into different SPUs led to inconsistent
issue escalation processes across the SPUs (S9). The variations of issue escalation pro-
cess led to delays and confusions, and led the CIO to initiate Reorganization 2 (S10).
This suggests that the decision-maker could not anticipate the trade-off of resource
allocation flexibility from the reorganization to matrix organization.
In Reorganization 2, the pool organization solved the problem of project issue
escalation, as all pool heads only reported project issues to the CIO (S12). The pool
organization also strengthened the flexibility of resource allocation. As team members
are no longer permanently designated to a particular product, they could be assigned to
projects or initiatives which the business really needed (S13,S14). The pool organiza-
tion also established clearer responsibility for the team members and managers (S15).
The pool structure allowed team members to be allocated into different projects
which eliminated a sense of belonging to a group (S16). There was loss of product
knowledge as people were not designated to a particular product anymore and moved
between projects (S17). More time was required to find the right people to source
projects (S18) because people were spread out in different pools. Decision-makers ini-
tiated a mitigation approach to mitigate loss of knowledge problem through sharing
ServiceDelivery
Delivery
Service
Service Provider
Department Unit (SPU)
Project Business Development Integration

Management Analyst (DEV) (INT)
Section 1 Section 2
... Change (PM) (BA)
Key Accounts
Project Business
Developer
Manager Analyst
Figure 2.2: Reorganization 1: Shift from hierarchical to ma-

trix organization
ServiceDelivery
Delivery Chief Information
Service
Service Provider Officer (CIO)
Unit (SPU)
Business Product & Testing

Project Business Development Integration Proxy Service &
Management Analyst (DEV) (INT) Delivery Support
(PM) (BA)
Change PM
BA
Key Accounts
DEV
INT
Managed Services
Figure 2.3: Reorganization 2: Shift from matrix to pool or-

ganization
of lessons learned. As expressed by one of our interviewees: [“We have some spe-
cialists groups, [..] then the group will communicate this lessons learned to the whole
organization.”]
RUP Introduction improved communication and flexibility of the team members
to be assigned in different development projects. However, the introduction of RUP
practices like project milestones and mandatory documentations, made team members
felt like they were being watched (S27). This trade-off only occurred in the beginning
of RUP adoption. Thus no mitigation actions were initiated.
The result of the Regulatory Change Implementation was the deployment of a
regulated service system into the infrastructure of the organization (S35). The deci-
sion to build the system in-house (S36) minimized the integration effort (S39), and
allowed the project team to steer the development process (S40). Furthermore, the de-
cision allowed the design of the infrastructure to be flexible in handling uncertainties
of the bilateral agreement (S36). However, the decision to build the system was more
expensive (S42) and required a large investment (S43). The decision to prioritize main-
tainability resulted in a regulated service system which was decoupled from the other
IT systems (S37). However the decision compromised the system’s usability (S41).
The selected option to modify a constituent allowed the organization to achieve
the flexibility required to address the uncertainty. However, each option to improve
2.5 Build Flexibility 39
flexibility could have other negative influence on other constituents. In the case of Re-
organization 1, Reorganization 2, and RUP Introduction, it was difficult to properly
foresee the outcome of the flexibility building option. However, in the case of Regu-
latory Change Implementation the trade-offs were consciously made by the decision
maker.
2.5.2 Flexibility Building Model: Revisit RQ1

Using interview data from nine IT managers and grounded theory our research explored
how an IT organization improved or built flexibility through strategic change of one of
the IT organization constituents. Our research uncovered three states that the organi-
zation went through as part of the strategic change: pre-change, transitional state, and
post-change. Pre-change was the state prior to the strategic change. Transitional state
was the state when the constituent modification was taking place. Post-change was the
state after the strategic change was in place. If another strategic change was required,
the post-change state would be the pre-change state.
Through these transitional states in the organization, our data show that there were
different events that took place in each state, emerging uncertainty, need for flexibility,
build flexibility, and achieved flexibility and trade-offs. Across four distinct change
scenarios, we observed that the uncertainty emerged in the pre-change state of the or-
ganization. The uncertainty triggered the need for flexibility. During transitional state,
flexibility was built by modifying one or more of the organization constituents to ad-
dress the uncertainty. In the post-change state, after the modification was implemented,
flexibility was achieved yet with associated trade-offs. Furthermore, as demonstrated
in Reorganization 2, flexibility building process could be cyclical, if the associated flex-
ibility trade-off introduces new uncertainty, another round of flexibility building could
be initiated.
The result suggests a model that elaborates the different events that took place in
the transitional states as shown in Figure 2.4. The model does not detail the decision
making points that managers need to take to build flexibility. However, it identifies
and describes the events that took place as an IT organization built flexibility of its
different constituents. The model offers a unified view of flexibility building for an IT
organization, from ad hoc and latent experience to a well-constructed and explicit point
of view.
Furthermore, we discovered that a flexible organization was not an organization
that could quickly make changes to its structure every time it faced an uncertainty. Yet,
a flexible organization was an organization that was able to restructure its organization
constituents in such a way that they allowed the organization to withstand a specific
type of uncertainty.
Transi0onal*
Pre$Change* (change*is* Post$Change*
IT*Organi$* implemented)*
za0on
Achieved*
Emerging* Need*for* Build*
Events Flexibility*&*
Uncertainty* Flexibility* Flexibility*
Trade$Oﬀs*
Figure 2.4: Events in Flexibility Building
2.6 Discussion
This research has discovered the key events that took place as an IT organization built
or improved its flexibility. Our data show that building organizational flexibility was
done through modifying one or more of the IT organization’s constituents to address
the uncertainty that exposed the organization’s need for flexibility. Through the con-
stituents modification, the organization was able to achieve flexibility and this aided ef-
ficient response to changes and uncertainty, and with little additional effort. Our study
has provided a holistic explanatory view of the processes in improving organizational
flexibility.
Manufacturing literature suggests that building flexibility entails identifying uncer-
tainty, implementing the appropriate decision, and monitoring the achieved flexibility[24,
80]. Similar view is also shared in organizational change management literature [156].
The results of our analysis show similar pattern with existing literature in flexibility and
change management. However, our study suggests that a flexible organization is not the
one that is capable of constantly change the structure of its organization constituents
on the face of uncertainties. Organizational change is one of the tools to build organi-
zational flexibility. Building organization flexibility is about purposefully restructure
of the organization constituents so the organization is able to cope with uncertainties.
Further research could be allocated on looking at known approaches in organizational
change management to complement the current model that was developed in this study.
In software reusability management, reuse can be used from two perspectives “de-
veloped for reuse” and “developed with reuse” [23]. Developed for reuse is when
software artefacts like documentations and codes are predefined and purposefully de-
veloped for later reuse usually by implementing well known design principles like
encapsulation, coupling, and cohesion. Developed with reuse deals with the use of the
previously predefined artefacts. The data in our study shows that to achieve flexibil-
ity organization constituents could also be “developed for flexibility” and “developed
2.6 Discussion 41
with flexibility”. In Reorganization 1, Reorganization 2, and RUP Introduction, the

organization structure and development process were developed for flexible resource
allocation to different projects, whilst each initiated project was developed with flex-
ibility of the organization structure and development process. In Regulatory Change
Implementation the infrastructure was developed for flexibility of providing financial
services for different bilateral agreements, whilst each supported bilateral agreement
was developed with flexibility of the infrastructure. This shows that software engineer-
ing has mechanisms for improving flexibility that are applicable at the organizational
level, but currently are used at product level. Future research can be directed towards
identifying mechanisms and design principles that could be transferred from software
artefacts to organizational level to build organizational flexibility.
Real options thinking is often suggested for managers to cope with uncertainty
[71]. Uncertainty can be addressed by choosing an option. One of these options is
defer where decisions are delayed for a certain period of time. Our study shows that
modifications of organization constituents allowed managers to exercise the defer op-
tion. In Reorganization 1 and Reorganization 2, the decision to change the organization
structure, allowed them to defer the decisions which team members to be allocated to
a certain development effort until a project was funded. In the Regulatory Change Im-
plementation scenario, the decision to design the infrastructure to cope with variation
of bilateral agreements, allowed them to defer decisions on which services to activate
or deactivate. Future research on real options thinking can be pursued to get insights
on how to develop criteria to help managers to select the best option in improving
organizational flexibility.
Previous studies suggest that flexibility is a trade-off with values, such as control
[86], performance [70], and cost [226]. Our study shows that flexibility is not neces-
sarily an immediate trade-off with other values. Scenarios clearly show that flexibility
trade-offs were the indirect effects of the selected option for the strategic change. For
example, in Regulatory Change implementation, the option to build the system in-
house was more costly, nevertheless it was preferred over purchasing a system that
required large integration effort. In Reorganization 2, ease of team member alloca-
tion to projects was preferred over quick development project completion. Our study
shows that different options could lead to different trade-offs, however these trade-offs
were acceptable given the achieved flexibility. Further research needs to be done to
help managers predict and assess the trade-offs of the different options in improving
organizational flexibility.
One of the practices in Agile methodologies is iterative development [1]. Previous
studies have shown that iterative development allows a software organization to reduce
rework, improve estimations, and reduce lead time [167]. Our study also shows that
flexibility improvement is done iteratively. This was clearly demonstrated in Reorgani-
zation 1 and Reorganization 2. In Reorganization 1, the organization structure shifted

from hierarchical to matrix organization structure to deal with rigid resource allocation.
Despite the achieved flexibility of resource allocation, the new organization structure
introduced unanticipated inconsistencies into the issue escalation process. The incon-
sistency initiated a new cycle of flexibility improvement through Reorganization 2.
Our study shows that the iterative nature of achieving organizational flexibility is con-
gruent with the iterations in Agile methodologies. Furthermore, our study shows that
reflecting the outcome of a strategic change was an important process for a software
organization in achieving flexibility. Future research could be directed to seek other
key ideas from Agile methodologies for improving organizational flexibility.
Current approaches in software engineering, like Agile methodologies [13] and
SPLE suggest [41] that flexibility is built through the products. Our study shows that
organizational flexibility could be built into different organization constituents, not only
built through the products. Reorganization 1, Reorganization 2, and RUP Introduc-
tion show that modifications in organization structure, management, and development
process could have influence on the flexibility of project resource allocation which in
turn improved the IT organization’s flexibility to initiate projects that suited the busi-
ness needs. Regulation Change Implementation shows that the infrastructure design
improved the IT organization’s flexibility in coping with what the business requires in
terms of providing services for different bilateral agreements. This shows that there are
wider range of possibilities how an IT organization can be improve its flexibility than
previously assumed and further confirmed the need for a holistic approach to flexibility
[53] not only through products.
2.6.1 Implications for Practice and Research

Our research suggests a model that provides a holistic view of how an IT organization
improves flexibility. This model was uncovered through analyzing the experiences of
IT managers in the case organization and has not yet been operationalized. However,
the findings of our study yield implications both for researchers and practitioners.
The key events that our study uncovered can be used by practitioners as a founda-
tion for process improvement. For example, flexibility could be added as a process area
to existing frameworks such as CMMI. Such a process area would include the identifi-
cation of uncertainty, the prioritization of which kind of flexibility should be built, the
selection of the appropriate flexibility improvement approach, and the analysis of the
obtained results. Also, such a process should specify how experiences from flexibility
improvement would be collected and shared to support future decisions.
Flexibility improvement processes would need to be supported by suitable tools
that enable process institutionalization. For example, the uncertainties that need to
2.7 Summary and Conclusion 43
be addressed by the organization could be captured in a backlog. Backlogs are an

established and commonly used tool in Scrum to enable visibility and prioritization
of features that a product needs to deliver [191]. A backlog of uncertainties would
enable sharing of flexibility improvement needs, prioritization, and progress tracking,
thus allow the organization to manage flexibility proactively and systematically.
Also, knowledge and experience sharing may improve the organization’s ability
to build flexibility. Knowledge sharing is an essential part of organizational learning
that has led to performance improvement in many companies [79]. Here, knowledge
sharing should enable the identification of flexibility-building options and improve the
understanding of the options’ impacts on flexibility and trade-offs. Such knowledge
could be expressed as stories or patterns [83]. Their use would reduce the risk of
unanticipated negative surprises and minimize the need to re-iterate.
Our study provides a stepping stone for future research in understanding how or-
ganizational flexibility is built. To support organizations in improving flexibility, a
systematic mapping of flexibility-building options with their associated benefits and
trade-offs is needed. Experience transfer and evaluation from domains other than soft-
ware engineering, like management and manufacturing, can be valuable inputs to build
such a mapping. Management literature provides suggestions to build flexibility, for
example to adopt a flat organization structure [91] and having managers that are open
to change [199]. Manufacturing literature describes mechanisms and measures to build
flexibility through suitable material handling and production process design [197].
Concerns that are unrelated to software need to be replaced, however. Nevertheless,
understanding the practices of other established domains can prevent “reinventing the
wheels”.
To formulate a model that maps flexibility-building options and their associated
benefits and trade-offs, we can also pursue experience transfer from existing product-
level approaches in software engineering to organizational level. Our study shows that
organization constituents could be “developed for flexibility” and “developed with flex-
ibility”. Also, our study shows that flexibility improvement was done iteratively. These
are known concepts from software reusability management [23] and Agile methodolo-
gies [1] with an abundance of empirical studies. Using existing empirical studies from
such established approaches can allow researchers to build a map of flexibility-building
options and their associated advantages and disadvantages.
2.7 Summary and Conclusion

We adopted grounded theory as a research approach for understanding how an IT orga-
nization improved flexibility. We studied four scenarios of flexibility improvement that
our industry partner considered representative for an IT organization that had evolved
over the past decade. Data was collected by interviewing the respective decision-maker
about the strategic flexibility-related change he was involved in. The scenarios suggest
that decision-makers have a central role in managing IT organization flexibility. They
are the key authoritative figures that allow which changes to be made in the pursue of
flexibility. Furthermore, the selected course of actions that the decision-makers take
can yield to different flexibility trade-offs.
The research resulted in a model that describes events that take place as an IT orga-
nization improves its flexibility. The model suggests that flexibility is built to answer
uncertainty in the organization that can cause delays, unnecessary work, and inefficient
use of resources. Furthermore, flexibility can be built into different constituents of the
organization, and not just through the product. Future research should be directing to-
wards improving the understanding of how to select the course of action that is best
suited to address uncertainty, and better predict flexibility trade-offs.
Chapter 3
Literature Review of Flexibility

Attributes: A Flexibility
Framework for Software
Developing Organization
Indira Nurdiani, Jürgen Börstler, Samuel A. Fricker, “Literature Review of

Flexibility Attributes: A Flexibility Framework for Software Developing
Organization”, Journal of Software: Evolution and Process, 2018, e1937,
(DOI: 10.1002/smr.1937).
3.1 Introduction
Flexibility is an important capability of a software organization to facilitate adapting
to changes in the business environment, such as changes in customers’ needs, business
processes, market demands, competitors, and technology [68]. A software organiza-
tion faces changes that can affect the product, but the also the organization structure,
management, and development process [151].
To maintain a competitive advantage a holistic view of flexibility is required, as
confirmed in the manufacturing and information systems (IS) literature [91, 197]. A
46 Literature Review of Flexibility Attributes
holistic view means taking into consideration the various aspects that involved in the
process developing software. This includes workforce, management, processes, orga-
nization structure, and infrastructure [218]. A holistic view of flexibility is required
because making one aspect more or less flexible may affect the flexibility another as-
pect [31, 44, 64, 125, 151]. In this chapter, we view flexibility from the overall software
organization perspective and not just the flexibility of the software product.
Flexibility is often associated with Agile and Lean methods. The literature in Agile
and Lean methods suggests that principles and practices of Agile and Lean support
flexibility [1, 44, 60, 66, 130, 164]. However, there is no way to evaluate whether Agile
and Lean methods deliver the required flexibility or deliver more flexibility compared
to other approaches.
In software engineering, flexibility is mostly studied with respect to aspects of the
software product like code or architecture. However, software organizations are faced
with changes beyond the scope of the product, e.g., organization structure and work-
force [151]. Other disciplines like manufacturing, information technology (IT), IS and
management indicate a comprehensive view of flexibility is needed in order to fully
understand the trade-offs associated with flexibility. This motivates us to perform a
multi-disciplinary literature review to identify flexibility attributes that are reported in
the literature. The resulting framework from the literature review can then be used
to evaluate whether a certain development method, such as Agile and Lean software
development, delivers the flexibility needed by a software organization.
In this chapter, we present an overview of attributes that characterize flexibility.
We reviewed the literature from various disciplines, like manufacturing, systems engi-
neering, management, IT, IS, and software engineering. The results of this literature
review can be used as the first step toward understanding flexibility and comparing de-
grees of flexibility that different practices or methods provide. As a proof of concept,
we evaluated the resulting framework by applying it to the 28 Agile and Lean prac-
tices described in [164] and [239], to understand in which ways these practices address
flexibility.
The remainder of this chapter is organized as follows. Section 3.2 presents the
background and related work. Section 3.3 describes the research method. Section 3.4
presents the results and analysis. Section 3.5 discusses the result. Lastly, the chapter is
summarized and concluded in Section 3.6.

In the following subsections we describe the relevant concepts in the study. First, we
provide examples of definitions of flexibility. Next, we briefly describe how flexibility
is viewed in different disciplines. We then introduced Agile and Lean practices. We

presented other relevant related work, and lastly summarized the existing knowledge
gap and the motivation to our study.
3.2.1 Definitions of Flexibility

Flexibility is context dependent and has different meanings in different domains [54,
197]. Table 3.1 provides examples of definitions of flexibility. According to these def-
initions, flexibility is an property of an entity – a product or an organization – where
its intended use can be quickly and easily changed due to shifting or increasing stake-
holder needs, business objectives, use scenarios, system requirements, or technologies.
The definitions in Table 3.1 indicate that flexibility is about reacting to changes
and uncertainty with minimal modifications to the entity itself while maintaining low
cost, effort, and time. Flexibility itself is not the goal, but a means to an end [20].
The outcome of flexibility may vary and could, for example, be regulatory compliance,
reduced training time, or effective human resource allocation [151].
Based on these definitions we can see that manufacturing, management, IS/IT, and
business process management (BPM) have different foci of flexibility, whereas systems
engineering and software engineering focus on flexibility with respect to products. This
motivates us to perform a multi-disciplinary literature review. We therefore not only
include papers from software engineering, but also other relevant domains such as man-
ufacturing, management, IS/IT and BPM.
Please note that the definitions in Table 3.1 are not a part of the literature review.
Their purpose is to exemplify the diversity of definitions and to motivate further studies.
The definitions are also not associated with the taxonomies described in the related
work (Section 3.2.4).
In this study, we did not use the definitions of flexibility to develop the flexibility
framework. We believe having an exhaustive list of flexibility definitions may still
obscure the existing flexibility attributes reported in the literature. For example, Regev
et al. [177] (can be seen in Table I) define flexibility as “the capability to implement
changes in the business process type and instances by changing only those parts that
need to be changed and keeping other parts stable”. If we had focused on the definition
provided in [177], we may have missed the classifications of “properties of change” as
one of the flexibility attributes.
3.2.2 Flexibility Across Disciplines

The discourse of flexibility in the engineering contexts started in manufacturing during
the 1950’s. Different types of flexibility are proposed and mapped to aid manufactur-
Table 3.1: Examples of Definitions of Flexibility in Various Disciplines.
Domain Definition of Flexibility Focus

Manufacturing “A form of metacontrol aimed at increasing control capacity by means Controlling change.
of an increase in variety, speed, and amount of responses as a reaction
to uncertain future environmental developments” [197].
“The ability to change or react with little penalty in time, effort, cost, or Reaction to change.
performance” [54].
“The incremental cost and time of modifying a design as a response to Cost of change.
changes exogenous (e.g., shifting customer needs) or endogenous (e.g.,
the discovery of a better solution approach) to the design process” [219].
“The ability to respond effectively to the ever-changing and increasing Customer needs.
needs of the customer” [106].
Management “The ease with which the organization’s structures and processes can be Organization struc-
changed” [91]. ture.
“The ability of the organization to adapt to substantial, uncertain, Organization.
and fast-occurring (relative to required reaction time) environmental
changes that have a meaningful impact on the organization’s perfor-
mance” [52].
Information “The ability to adapt to both incremental and revolutionary change in Cost of change.
Systems the business or business process with minimal penalty to current time,
effort, cost, or performance” [145].
“To maintain control and respond to constant changes in hypercompet- Control.
itive, fast-changing environments” [31].
Information “The ability to vary the ’reach’ of technology, to vary the “range’ of Technology features
Technology technology features, and to use ‘time’ as an ally in the quest for flexi- (product).
bility” [116].
“The ability of a firm’s processes and systems to respond quickly to Organization.
changes in the business environment. It includes the capacity to accom-
modate shifts in consumer demand, in competitors’ strategies, in rate of
growth, and in suppliers’ deals and shipment problems” [211].
Business Pro- “The capability to implement changes in the business process type and Business process.
cess instances by changing only those parts that need to be changed and
keeping other parts stable” [177].
Management “The capacity of making compromise between, first, satisfying, rapidly Controlling change.
and easily, the business requirements in terms of adaptability when or-
ganizational, functional and/or operational changes occur; and, second,
keeping effectiveness” [150].
Systems Engi- “Property of a system that allows it to respond to changes in its ini- Systems.
neering tial objectives and requirements – both in terms of capabilities or at-
tributes – occurring after the system has been fielded, in a timely and
cost-effective way” [188].
“The measure of how easily a system’s capabilities can be modified in Systems.
response to external change”. [185].
Software Engi- “The ease with which a system or component can be modified for use in Software product.
neering applications or environments other than those for which it was specifi-
cally designed” [94].
“Effort required to modify an operational program” [137]. Software product.
“Degree to which a product or system can be used with effectiveness, Requirements (soft-
efficiency, freedom from risk and satisfaction in contexts beyond those ware product).
initially specified in the requirements” [96]
“Software is flexible if it can be efficiently and rapidly adapted because Software product.
of a change in business needs” [172].
“The ability to efficiently respond to new market opportunities in an Effectiveness.
effective manner (i.e. the speed by which ideas are brought to the mar-
ket)” [44].
ing plants in identifying and utilizing flexibility to gain competitive advantage [197]:
basic flexibility (machine, material handling), system flexibility (process, product), and
aggregate flexibility (production and marketing).
In the management literature, the idea of flexibility is centered around the organi-
zation and its structure [52, 91]. Management literature also suggests that uncertainty
is a pre-requisite for flexibility, as managers tend to demonstrate more flexibility, in the
form of openness to new ideas, when situations are uncertain [199].
Flexibility is also an interest in IT and IS research. IT systems are the backbone
for companies to run their business strategies [89]. Business changes need to be ac-
commodated by IT systems and the IT organization that develops and maintains them.
Changes to the business due to changes in customer needs, competitors, and technology
can affect the IT organization. The IT organization may face uncertainty in resource
demands, new personnel training demand, or new technology or services to be put in
place [17]. To cope with uncertainties, IT systems and the IT organization that develops
and maintains these IT systems need to be flexible [30, 71, 116].
In BPM, flexibility entails not only changes but also stability [177, 190]. Building
flexibility means knowing which parts of an entity need to change and which parts need
to remain stable [177, 178]. Changes that are unnecessary for the entity to achieve flex-
ibility can lead to a loss of identity, cease of existence [179], and loss of effectiveness
[150].
In software engineering, flexibility has been primarily studied from the perspec-
tives of the software product and service. In software architecture, flexibility can
be achieved by modularizing a software system and thereby localizing the impact of
change [143]. In software product line engineering, flexibility is achieved through
systematically identifying and managing variations and commonalities from a set of
software systems artifacts, e.g., code, test cases, and requirements. When a product
variation is required, a new product can be instantiated from the previously developed
artifacts [38, 138]. Eden and Mens [64] propose using the complexity of implementing
changes to measure software flexibility. As a software system grows in complexity,
more changes need to be applied to the code. Thus making the software less flexible.
Codenie et al. [44] indicate there needs to be a balance between flexibility and product
variability. Furthermore, when considering the trade-off between flexibility and prod-
uct variability, aspects such as expertise and uncertainty needs to be taken into account
[44].
Generally, flexibility in software engineering research is focused on product as-
pects of software development, e.g., architecture or coding. However, Codinie et al.
[44] indicate the need of a holistic view of flexibility. This view is reflected in flexibil-
ity studies in other disciplines, such as manufacturing, management, and IT/IS where
holistic view of flexibility is needed to ensure that conflicts and trade-offs to flexibility
are addressed correctly [118]. To appropriately address flexibility in software develop-

ment, there is a need for a comprehensive collection of attributes to evaluate flexibility.
Agile and Lean software development are often associated with improving software
development flexibility, which is discussed in more detail in Subsection 3.2.3.
3.2.3 Agile and Lean Practices

Agile and Lean software development share a common goal which is to frequently
deliver working software that is valuable for the customers. This goal is then translated
into a set of principles. Both Agile and Lean software development also share similar
principles with respect to people, product quality, flexibility, and learning [164].
Agile software development principles include motivated individuals, welcome change,
working software, and frequent deliveries. These principles are then translated into
a number of practices such as short iterations, collective code ownership, and test-
driven development (TDD) [1, 164]. Short iterations allow changes to be integrated as
soon as the need for change is discovered from customer feedback (welcome change)
[164]. Collective code ownership increases knowledge sharing and supports flexibility
by making it easier to assign people to tasks (motivated individuals) [1]. TDD allows
defects to be caught early which can lead to higher quality software (working software)
[14, 164].
Lean software development principles include respect people, eliminate waste,
build quality in, and defer commitment [164, 169]. These principles are then trans-
lated into a number of tools such as Value Stream Mapping (VSM), and kanban pull-
systems. VSM provides a visualization of the end-to-end activities and helps to identify
activities that do not produce value for the customer (build quality in, eliminate waste).
Using kanban pull-systems, team members select a new task from a list of prioritized
requirements which suits their capacities (respect people, defer commitment).
Given the similarities of goals and principles, Agile and Lean software development
are often discussed together [164]. Some authors also suggest that Lean is one of
Agile’s methodologies [58, 163]. Poppendieck and Poppendieck [169] also suggest that
Lean is the expansion of Agile’s theoretical foundation. Agile practices and Lean tools
are also often combined [149, 234]. Furthermore, Petersen found that Agile principles
can be mapped into Lean principles, and that Agile practices can fulfil Lean principles
and vice versa [164]. Therefore, in this examination of flexibility attributes, we include
both Agile and Lean practices.
Agile and Lean practices are well defined and documented. However, the list of Ag-
ile and Lean practices reported is not consistent across research papers. For example,
Williams reported that there are 33 Agile practices, in this list acceptance test driven
development and unit test driven development are treated as two different practices
[239]. Meanwhile, Petersen reported there are 26 Agile and Lean practices, where unit
testing is included in TDD [164]. The lack of agreement on what constitute an Agile
or Lean practice is further demonstrated in the variation of practices included in Agile
surveys. Solinski and Petersen identified that the number of Agile practices included
in Agile surveys vary between 14 and 59 practices [212]. Meanwhile, Rodriguez et al.
reported the use of 16 Agile practices and 14 Lean principles from an industry survey
in Finland [183]. In our study, we included 26 practices provided by Petersen [164],
and complement them with two practices reported by Williams [239], which are, def-
inition of done, and informative workspace. We also found the list of practices and
definitions provided in [164] subsume the practices described in [239]. For example,
Petersen describe retrospective meeting in Scrum to be part of reviews and inspection
[164]. We however do not claim to have an exhaustive or complete set of Agile and
Lean practices. It is also important to note that it is likely that software development
teams and organizations use a subset of the practices [123].
Flexibility is often associated with implementing Agile and Lean practices [1, 60,
66]. However, the extent to which Agile and Lean practices can help to achieve flexi-
bility has not yet been examined. Therefore, in this study, we also examine Agile and
Lean practices against flexibility attributes found in the literature. Furthermore, it is
important to emphasize that we use the list of Agile and Lean practices as a proof-of-
concept.
3.2.4 Related Work

Conboy [47] performed a literature review to consolidate the definitions of agility and
proposes a taxonomy for Agile Information Systems Development (ISD). The taxon-
omy consists of three parts: (1) primary ways of contributing to agility, (2) contribution
to perceived economy, quality, or simplicity and (3) continual readiness for change.
Conboy used this taxonomy to elicit practitioners’ perceptions on how Agile prac-
tices contribute to agility. Two focus groups revealed that pair programming was per-
ceived to have a mediocre impact on creation of change and no impact on reaction
to change [47]. Our study is similar to that of Conboy as the concept of agility and
flexibility are similar and the two terms are often used interchangeably [3]. Our study
differs from Conboy’s in that we aggregate attributes rather than definitions of flexibil-
ity. Our study also includes Lean practices, meanwhile Conboy covers Agile practices
only.
Vidgen and Wang [232] use principles from complex adaptive systems as a proxy
to agility to evaluate traditional waterfall development and XP. Although this study is
relevant to ours, we believe our study is not comparable to that of Vidgen and Wang.
Our study focus on identifying flexibility attributes from the literature and map them
to different Agile and Lean practices. Meanwhile Vidgen and Wang [232] derived
concepts from CAS and compare how the different development methods fit into the
principles.
It is important to stress that in this study we do not aim to compare the term “agility”
and “flexibility”. In this study we view flexibility as a part of or in support of agility, as
expressed by Bernardes and Hana [20]. To complement existing research on agility, it
is therefore important to identify the attributes of flexibility which in turn can support
agility.
There are a number of publications on classifications of flexibility attributes. Regev
et al. [177] proposed a taxonomy of flexibility in business processes. The taxonomy
describes the types of changes that flexibility enables. The taxonomy has three dimen-
sions:
• The abstraction level of change, as changes can happen at the business process
level or the instance of the process.
• The subjects in which changes may happen.
• The properties of change, such as the extent, duration, swiftness, and anticipation
of change.
Van der Aalst and Jablonski [227] proposed a similar classification of change as a
way to know what kinds of change need to be supported by the business process. They
formulated a set of questions to describe an ”anatomy of change”:
• What are the reasons of change: motivated by external or internal needs?
• What are the effects of change: minor or extensive change?
• Which perspectives are affected: a process, organization, information, operation,
or integration?
• What kind of change: an extension, reduction, replacement, or rearrangement?
• When changes are allowed: at entry time or on-the-fly?
De Toni and Tonchia [54] performed a literature review of flexibility in manufac-
turing and presented a classification of flexibility to clarify ambiguity of flexibility in
manufacturing. The classification is based on the following criteria:
• Source of change: internal or external.
• Objects that require variations: product, machinery, or process.
• Choices of flexibility: using automation or relying on human resources.
• Flexibility strategies: proactive or reactive.
Byrd et al. [31] performed a literature review of flexibility in IS and proposed a
framework for IT/IS flexibility that includes four perspectives of flexibility: people,
process, data, and infrastructure.
Reichert and Weber propose a flexibility taxonomy for Process Aware Information
Systems (PAIS) [180]. The taxonomy is proposed because flexibility needs may affect
each other and that a comprehensive overview is required. The taxonomy is based on
four major flexibility needs that are to be supported by PAIS, they are:
• Variability: product and service, regulations, customer groups, and time.
• Adaptation: drivers and anticipation
• Looseness: non-repeatability, unpredictability, and emergence.
• Evolution: drivers, extent, swiftness, duration, and behaviour.
Research in BPM (e.g., [177, 227]) is focused on classifying types of change to charac-
terize flexibility. De Toni and Tonchia [54] provide a rather comprehensive coverage of
flexibility attributes in manufacturing, but pay little attention to types of change. Mean-
while, flexibility taxonomy in IS [31] is only focused on the perspectives that needs to
considered when assessing IS flexibility. The taxonomy by Reichert and Weber [180] is
comprehensive however the elements are very specific. For example, regulations could
be a source of change [151], however not all software organizations are faced with reg-
ulatory change. Moreover, the research methodology used to construct the taxonomies
or classifications mentioned above is often not visible.
3.2.5 Summary of Knowledge Gap

From the definitions of flexibility, we can see that different disciplines have different
focus on flexibility. Meanwhile, software engineering primarily focus on the product.
From the related work, we can see that each discipline has specific flexibility attributes
that it focuses on. Current classifications of flexibility attributes are fragmented and
limited to the interests within disciplines.
Software engineering as a discipline shares many concerns that are relevant in dis-
ciplines, like, manufacturing, IS/IT, or management. The vision of a traditional man-
ufacturing method to develop software has been discussed in the context of software
factories [49]. The production line in manufacturing is analogous to coding in soft-
ware development. In software development also exists supply chain management and
distribution processes. However, it is done differently, for example, in software de-
velopment instead of supply of raw material, it could be a supply of sub-contractor
developers. Moreover, the ideas of Agile and Lean are drawn from manufacturing,
also management is inherent in software developing organizations.
A software development organization has multitude of concerns that cross-cuts dif-
ferent disciplines, like management, IT/IS, and BPM. A comprehensive classification
of flexibility attributes that cross-cuts disciplines is lacking. This motivate us to de-
velop a flexibility framework that covers a comprehensive set of flexibility attributes
from various disciplines. Such flexibility framework is needed to help software or-
ganizations whether the practices, tools, or methods that are currently being adopted
address the flexibility needed by the organization. We refer to the elements of the
framework as flexibility attributes, since they reflect the concerns that characterize a
software organization’s ability to address flexibility. The resulting framework would
then be evaluated to evaluate the extent of a software organization flexibility. As a
proof-of-concept, the resulting flexibility framework is used to evaluate the extent of
Agile and Lean practices in addressing flexibility.

This research aims at aggregating and classifying flexibility attributes from the litera-
ture and evaluate them against Agile and Lean practices. To guide us in performing the
literature review, we developed the following research questions:
RQ1. Which flexibility attributes are reported in the literature?
RQ2. To what extent do Agile and Lean practices address flexibility attributes?
3.3.1 Literature Review Steps

Given the breadth of the concept that spans different disciplines, a traditional system-
atic literature review (SLR) [110] was not feasible. A traditional SLR is more suitable
when the study is aimed to aggregate evidence on the impact of an intervention in
a population [110]. We adapted the iterative literature review approach proposed by
Levy and Ellis [128]. An overview of our approach is shown in Figure 3.1.
Flexibility is a multi-faceted and broad term. To focus our starting point, we com-
menced our literature review by identifying other secondary studies or literature re-
views in flexibility (Step 1). We performed keyword searches in Scopus and Inspec
only for the initial search which was followed by backward and forward reference
search. Scopus and Inspec were selected because both are reference databases that
include articles from different publishers. Scopus claims to have the largest peer-
reviewed literature database, and Inspec has a wide range of literature in engineering.
We did not include other reference database such as ScienceDirect and Wiley Science
since they return similar results as Compendex, which is now included in Scopus [25].
The search period was all years, meaning that we included all publications from the
earliest year that a database provided until 20121 when the initial database search was
performed. The following search string was used in the abstract, title, and keyword
fields of both databases:
1 Please note that 2012 only concerns the seed papers. Since we performed forward snowballing, we made
sure to include publications until 2016.

flexibility AND (“literature review” OR “literature survey” OR “system-

atic mapping” OR “systematic map”)
4. Identify
0. Research 1. Initial 2. Identify 3. Identify
Common
Frame Search Seed Papers Classifications
Attributes
Contribute
5. Backward and 6. Find New Yes 7. Update

Forward Reference Attributes? Relationships
Search
No
No 8. Attributes Yes
9. Done
reached saturation?
Figure 3.1: Overview of Literature Review Steps
From the initial database search (Step 1), we obtained 460 papers (358 from Sco-
pus, 102 from Inspec). From those 460, we removed 74 duplicates, six non-English
papers, and one proceedings cover. After removing papers from domains that we con-
sidered outside the scope of this study (medicine, biology, material science, atmo-
spheric science, and pedagogy), as well as non-secondary studies, 24 papers remained.
We excluded papers from medicine and biology because the flexibility mechanisms
concerning the “natural” systems, like the organism cells or human physiology are not
comparable to “artificial” systems [206], such as a software development team or a
manufacturing plant. Of those 24 papers, five papers were actual literature reviews in
flexibility; two from manufacturing [54, 197] and one each from IS, IT, and systems
engineering [31, 116, 188]. These five papers were used as seed papers (Step 2) for
backward and forward search (Step 5) and the starting point for developing an initial
classification (Steps 3 and 4). The seed papers are summarized in Table 3.2.
The seed papers allowed us to discover that flexibility can be characterized with
classifications of flexibility attributes and categories that are associated to them (Step
3). We discovered common attributes of flexibility such as flexibility types, uncertainty,
properties of change, flexibility dimensions, and choices of flexibility (Step 4). After
identifying different classifications of flexibility attributes, we decided to proceed our
literature review by identifying classifications of flexibility attributes.
Table 3.2: Seed Papers
# Title Author(s) Year Domain Reference

1 Flexibility in Manufacturing: A Survey A.K. Sethi and S.P. Sethi 1990 Manufacturing [197]
2 Manufacturing Flexibility: A Literature Review A. De Toni and S. Tonchia 1998 Manufacturing [54]
3 Information Technology Alignment or “fit” in K. Knoll and S.L. Jarvenpaa 1994 IT/IS [116]
Highly Turbulent Environments: The Concept of
Flexibility
4 An Examination of an Information Systems Flex- T.A. Byrd, L.J. Madariaga, 2010 IT/IS [31]
ibility Framework L.W. Byrd, and V. Mbarika
5 Flexibility: a multi-disciplinary literature review J.H. Saleh, G. Mark, and 2009 Systems En- [188]
and a research agenda for designing flexible engi- N.C. Jordan gineering
neering systems
The seed papers were used as the starting point for backward and forward reference
search (Step 5). The backward reference search was performed by screening the ref-
erences included in the seed papers. The forward reference search was performed by
screening papers that cited the seed papers. To ensure that we included the most recent
papers, the forward reference search was performed again in 2016. Figure 3.2 shows
the time span of the primary studies that were included this literature review.
Backward Forward
reference Initial Five Seed reference
search Papers search
1986 2015
1990 - 2010
(see Table 3.2)
Figure 3.2: Time Span for the Backward and Forward Reference Search. The earliest
publication found during backward snowballing was published 1998. The most recent
primary study identified through forward snowballing was published 2015.
For each seed paper, we went through the references and the citing papers and
applied the following inclusion criteria:
• Provides a classification of different flexibility attributes, e.g., flexibility types,
properties of change, or flexibility dimensions.
• The discussed flexibility attributes are generic and not specific for a certain pro-
cess, e.g., design process, production, or quality control.
Papers with the following characteristics were excluded:
• Flexibility dimensions or types are not applicable to software organizations, e.g.,
machinery flexibility, or raw materials supply chain flexibility.
• Flexibility dimensions or measures are too low-level, e.g., the number of mod-
ified classes, the number of labor hours, etc. We excluded papers that focused
solely on the flexibility of architectural components (e.g., [10, 143, 170]) that
were not generalizable to organization level.
During the full text screening, we performed holistic coding, which is described in
detail in Section 3.3.2. When we identified unique flexibility attributes or categories
during holistic coding (Step 6), we included the newly identified attributes and cate-
gories (Step 7). Otherwise, we went back to Step 5.
We performed Steps 5–7 iteratively until we reached saturation (Step 8). We con-
sidered saturation to be reached when the inclusion of new papers no longer revealed
unique flexibility attributes and it became difficult to find new papers that fulfilled our
inclusion criteria. We then proceeded with axial coding (Step 9), which is described in
detail in Section 3.3.2.
From the forward and backward literature search process, 370 papers passed the
inclusion criteria based on titles and abstracts, after reading the full texts 99 papers
remained. Out of those 99 papers, 38 papers contributed to the framework (see Figure
3.4). Including the five seed papers, the number of included papers totaled 43 papers.
The list of primary studies is shown in the Appendix, in the end of this chapter. The
mapping of flexibility attributes to their respective primary studies is shown in the
Appendix of this chapter, Table AI–AIII.
3.3.2 Coding Process

In this subsection, we describe the coding steps that were performed in this literature
review.
Holistic Coding.
Before the literature review, we did not have any preconceived ideas of flexibility at-
tributes. After reviewing the seed papers, we uncovered that flexibility attributes are
often described as classifications. We therefore decided to proceed our literature review
to collect different classifications of flexibility attributes.
We adopted holistic coding to collect classifications of flexibility attributes. Holis-
tic coding is a method used to grasp themes or classifications by coding and analyzing
them as a whole instead of line by line [187]. Holistic coding was suitable as an ex-
ploratory coding method for this literature review, as we were interested in gathering
classifications of flexibility attributes. We performed holistic coding iteratively, until
we could no longer identify new or unique flexibility attributes.
Axial Coding.
Axial coding is one of the suggested follow-up analyses after holistic coding [187]. In
our case, axial coding was integrating different classifications into one general com-
mon classification. The data that we collected through holistic coding were already in
categories, yet they were fragmented. The purpose of performing axial coding was to
see if some of the attributes or categories shared similarities and could be classified
into higher-order categories [187, 37, 115]. The results of axial coding are shown in
Figure 3.4.
Axial coding was done as follows:
• Flexibility attributes that used similar terminologies were merged. For instance,
papers that discussed the properties of change used a rather consistent terminol-
ogy like “properties of change” or “anatomy of change”.
• Flexibility attributes that used different terminologies but shared similar defini-
tions were also merged. For instance, papers that discussed flexibility perspec-
tives using different terminologies like “flexibility types” and “flexibility dimen-
sion”, but shared a similar definition.
• A flexibility category may be reassigned to another flexibility attribute to better
fit the context of software development. For instance, “swiftness” was reas-
signed from “properties of change” to “flexibility enablers” because in software
development, team members decide whether to address change immediately or
to defer it due to, for example, requirements interdependencies.
• If two or more classifications complemented each other, they were merged. For
instance, Byrd et al. [31], De Toni and Tonchia [54], and Van Der Aalst and
Jablonski [227] proposed classifications of flexibility perspectives. These classi-
fications were merged as “flexibility perspectives”. The redundant perspectives
were removed.
3.3.3 Mapping Agile and Lean Practices to Flexibility Attributes

To map each Agile and Lean practices to the flexibility attributes (Table 3.3–3.5) , we
performed the following:
• Check the description of each Agile and Lean practice from [164] and [239].
• Assess if the description of the Agile and Lean practice fits one or more ap-
proaches (see Sections 3.4.2 – 3.4.2) used to address different flexibility at-
tributes and their respective categories and values. An example of an approach
that address flexibility attributes, automation as an instantiation of static ap-
proach to flexibility.
• If an Agile and Lean practice fits one or more approach, we added the respective
flexibility values or a checkmark (X) in the respective tables. If an Agile and
Lean practice fits approaches for two or more flexibility categories or values,
we added them all in the respective table. Otherwise, we added Not Applicable
(N/A).

The following are validity threats relevant to this literature review [110, 109, 168]:
Publication Bias. Publication bias refers to the tendency of publishing positive
rather than negative results [110]. Since our research questions deal with the identifi-
cation of neutral concepts and not approaches or results that are loaded in a positive or
negative way, we believe that publication bias can be neglected.
Missing Important Papers. Since the topic of the present paper is broad, there is a
risk of missing relevant literature. To mitigate this risk, we started our search by iden-
tifying seed papers that were secondary studies and led us to a large number of primary
studies. The selection of the seed papers had a broad coverage in domains, publication
fora and number of citations. To further mitigate this risk, we complemented our initial
database search with an iterative forward and backward literature search. According
to Greenhalgh and Peacock [81], forward and backward literature search can uncover
important papers that might be missed by protocol-driven search.
Credibility. To mitigate the credibility threat, we adopted an iterative approach
between literature search and analysis in our literature review. Such an approach can
raise credibility issues regarding the breadth of categories being covered in iterative
studies [37]. In the present literature review, we developed our criteria iteratively as we
progressed with the review to ensure a broad coverage of flexibility attributes.
Consistency of Agile and Lean Practices Evaluation. As part of this study, we
evaluated the identified flexibility attributes against 26 Agile and Lean practices. To do
so, we examined the definition of each Agile and Lean practice provided by Petersen
[164] against the definitions of the flexibility attributes. To ensure consistency of the
definitions of Agile and Lean practices, we cross-examined the definitions with the
ones provided by Williams [239]. The results of these evaluations are shown in Table
3.3–3.5.

Our literature review identified 43 primary papers on flexibility, including the seed
papers. A summary of the papers included in this literature review can be found in the
Appendix in the end of this chapter. In the remainder of this paper, we use the reference
identifier shown in the Appendix to refer to those 43 papers (P1–P43).
3.4.1 Distribution of Papers

Figure 3.3 shows the distribution of the primary papers according to years and disci-
plines. Manufacturing dominated the distribution (14 papers), followed by papers in
management (11 papers). There are seven papers in IT/IS and six papers in BPM. We
also included two papers in systems engineering and one paper in general engineering.
Only one paper was specifically in software engineering (P15).
Mgt. Mft. IT/IS BPM Sys. Eng. Soft. Eng. Eng.
Figure 3.3: Papers Distribution Across Years and Disciplines
P15 is the only software engineering paper that studies flexibility from the perspec-
tive of team’s anticipation to change and its impact on product quality. P15 discusses
the issue of flexibility from a similar perspective as the ones discussed in other disci-
plines and covers a broad range of software development processes.
3.4.2 Flexibility Attributes (RQ1)

Our coding process (see Subsection 3.3.2) led to the flexibility framework shown in
Figure 3.4 that offers a consolidated view of the flexibility attributes identified in the
literature. In this framework, flexibility is categorized into three attributes: (1) proper-
ties of change, (2) flexibility perspectives, and (3) flexibility enablers. “Properties of
change” is an aggregation of types of changes that can be addressed through flexibility.
“Flexibility perspectives” provides an aggregation of aspects into which flexibility may
be built. “Flexibility enablers” provides an aggregation of mechanisms or approaches
to achieve flexibility.
Properties of Change.
Flexibility is about reacting to changes, it is therefore important to understand different
properties of change. We identified five properties of change: (1) origin of change, (2)
uncertainty, (3) frequency, (4) duration, and (5) extent of change.
Origin of Change.
Organizations have to adapt to changes, which can originate from inside or outside of
the organization (P1, P2, P3). Internal change can be driven by the pursuit of customer
service improvement (P4), or by internal technical problems (P5). Changes in customer
needs or regulatory changes are changes that originate from entities external to the
company (P4, P5). Internal and external changes may depend on each other. For
instance, a change of supplier may have consequences on the internal operations of the
company (P2).
Uncertainty.
Flexibility is an instrument to overcome uncertainty [147]. The literature in manufac-
turing (P1, P11, P13) and BPM (P18) distinguish uncertainty between foreseen and
unforeseen changes. A foreseen change is a type of change that is intended to alter
parts of a process or product (P11). The foundations of building an anticipation capa-
bility are planning mechanisms, like team collaboration, and proactive thinking (P15).
An unforeseen change is a type of change that occurs outside the boundary of the ini-
tial intentions of the process or product (P11). Organizations can allocate resources to
experimentation to uncover unforeseen change (P15).
Frequency.
Organizations need to understand how frequently they have to deal with changes and
select an appropriate course of action (P11, P12). Weick and Quinn (P12) distinguish
between episodic and continuous change. An episodic change is an infrequent and
Attributes Categories Values

Internal
Origin of
change External
Foreseen
Uncertainty
Unforeseen
Episodic
Properties of Frequency
Continuous
Change
Temporary
Duration
Permanent
Extent of Incremental
Change
Revolutionary
Organization
Product
Flexibility
Process/
Flexibility Operation
Perspectives
Information
Infrastructure
Static
Approach
Dynamic
Proactive
Intention
Flexibility
Enablers Reactive
Short-term/Operational
Temporal
Medium-term/ Tactical
Long-term/ Strategic
Immediate
Swiftness
Deferred
Figure 3.4: Flexibility Framework

discontinuous change which occurs to adapt to external environmental change. A con-

tinuous change is a pattern of endless modification of work items to deal with daily
incidents. A continuous change usually originates from within the organization (inter-
nal). However, in software development contexts, both episodic and continuous change
can originate from inside and/or outside the organization.
Duration.
An organization faced with a change needs to know for how long the change will be
in effect (P1, P20). Is it going to be a temporary or a permanent change? Given
the knowledge of the duration of the change, the organization can adopt appropriate
measures. If a change is temporary, the organization can temporarily deviate or re-
organize itself as long as required by the change and would return to its original state.
A permanent change requires more formal process alterations (P18).
Extent of Change.
A change can occur gradually or incrementally, as well as drastically or revolutionary
(P1, P11). An incremental change takes place only on parts of existing processes or
products and is implemented little by little (P20). For instance, a product feature may
be improved incrementally from release to release. A revolutionary change may lead to
the replacement of current products (P18). When a customer makes a changes request
that would alter the end product, such a request can be considered as the development
of a new product. In such a case, the change is considered as revolutionary.
Flexibility Perspectives.
In manufacturing, successful production is about finding the balance between the in-
volved perspectives of flexibility, including the machinery, process, and marketing
(P32). Also, it is important to understand what these perspectives of flexibility en-
tail. We organized the flexibility perspectives according to the objects that need to be
flexible. The perspectives are: (1) organization, (2) product, (3) process/ operations,
(4) information, and (5) infrastructure.
Organization Flexibility.
Organization flexibility can be viewed from two perspectives, the ability of an organi-
zation to reshuffle its structure (P14), and the ability of the organization’s structure in
adapting change implementation (P1).
In the former view, organization flexibility may be achieved by adding or modi-

fying roles and their definitions in the organization (P5). It is, therefore, important to
identify the roles that are affected by a change (P20). The willingness to assume new
responsibilities given unusual circumstances will also support organization flexibility
(P29).
The latter view, organization flexibility is about adapting changes coming into
the organization. In this view, organization flexibility can be achieved by employing
skilled workers with sufficient expertise (P32, P37). Flat organizations with small and
self-managing teams support flexibility because responsibility and decision making lie
within the team. Such a practice can lower communication overhead and allow new
solutions to be formulated quickly to address problems (P14, P32).
Product Flexibility.
A product is an article or substance that is manufactured or refined for sale2 . It is the
focal point of every manufacturing and development organization. Product flexibility
is “the ease with which new parts can be added or substituted for existing parts” (P32).
Additions and substitutions to product specifications are often required to meet market
demands (P2).
In manufacturing, product flexibility can be achieved through material handling,
machinery, and sophisticated computer-aided design software (P32). However, these
approaches may not be relevant in a software organization. Manufacturing also sug-
gests that loosely-coupled and modular product components can achieve flexibility
(P17, P28). Loosely-coupled product components paired with modular organization
design can offer flexibility. With modular product components and a modular organi-
zation design, difficulties in coordination may be contained (P17, P28).
Process/ Operational Flexibility.

Process flexibility in manufacturing is the ability to produce different product parts
within a short time span without major reconfiguration (P2, P32). Process flexibility
allows deviation or adaptation of the task description or the order of the tasks that are
included in a process (P18). Performing a task iteratively is also an approach to process
flexibility (P18). In manufacturing operational flexibility is defined as the ability to
interchange how the task is done associated with producing a part (P2). Operational
flexibility provides a degree of freedom to carry out a task. The term means that a
particular part of a product may be produced in different ways. Operational flexibility
2 Oxford online dictionary: www.oxforddictionaries.com
is particularly important to improve scheduling and accommodate temporary deviation

in real-time (P18, P32).
In manufacturing, the purpose of process flexibility is a streamlined production that
allows less time, money, and resources to be used. Process flexibility can be achieved
through adapting tools or machinery and employing workers with multiple skills (P32).
The same idea is expressed in the IS domain (P37), where process flexibility can be
achieved through employing technologies with built-in capacities for change and ex-
pertise with appropriate skills sets.
Sethi and Sethi (P32) mention that the design of a product can support operational
flexibility. Modular product components support operation flexibility. Schonenberg et
al. (P18) proposed task related options to address operation flexibility: to undo a task,
redo a task, skip a task, create a new task instance, and execute a task that is not yet
initiated.
It is important to note that the terms process flexibility and operational flexibility
are derived from manufacturing of tangible products where the distinction between
the two are more clearly defined. It is however, not easily distinguishable in software
development contexts. The distinction of process and operation could depend on the
perspective. For example, one may perceive unit testing as an operation, i.e. a task
within TDD. However, one may also perceive unit testing as a process given the sub-
tasks associated to it. In software engineering we have not yet come to an agreement
what may constitute a process or a task. Therefore, in this paper we do not make
distinction between process and operational flexibility flexibility.
Information Flexibility.
Organizations are information processing entities. Thus, their ability to properly pro-
cess information is linked to their ability to respond to changes in their business envi-
ronment (P29). The ability to react to changes depends on information about changes,
or lack thereof (P2).
Information flexibility deals with the production and modification of information
(P5). Mechanisms to support information flexibility include the standardization of data
formats, vocabulary, and naming conventions. To allow swift decision-making, the
accessibility of information is equally important (P22).
Information flexibility also deals with which information has to be defined and
exchanged when a change occurs (P20). Change of content, structure, and form of
information need to support the overall flexibility (P36).
Infrastructure Flexibility.
Infrastructure flexibility is discussed most often in IT/IS literature. Infrastructure is
“a set of shared tangible IT resources forming a foundation for business applications”
(P38). It includes hardware, operating systems, database management systems, net-
work and telecommunication technologies, and also skills and expertise (P38). In this
framework, staff with their skills and expertise are addressed in organization flexibility.
In IS/IT, infrastructure flexibility is defined as the ability to easily and readily de-
ploy hardware, data, software services, and applications. We believe that such infras-
tructure is also relevant for software developing organizations.
The manufacturing literature considers a similar concept of technical infrastructure,
called microprocessor technology, as one of the flexibility perspectives (P10, P32).
Technical infrastructure is perceived as one perspective that unites the other flexibility
perspectives (P32).
Flexibility Enabler.
From the literature, we identified mechanisms to enable an organization’s flexibility in
the following categories: (1) approach, (2) intention, (3) temporal, and (4) swiftness.
Approach.
From the literature, we identified two main approaches to achieve flexibility: static and
dynamic flexibility.
Static flexibility can be achieved by making assumptions about a constant and lim-
ited variation of information (P2). Examples of static flexibility are the use of automa-
tion (P35) or standardization (P2, P31). Building a product with modular components
(P28, P36, P38) and the creation of options or variation points (P35, P36) are other
forms of static flexibility. Static flexibility is suitable for environments where the pos-
sible change states are known in advance (P2).
Dynamic flexibility concerns organizational learning as a means to achieve flex-
ibility. It is less reliant on automation or machinery (P2). Simon [207] recognized
that humans are more flexible than machines. By reducing the reliance on machinery,
freedom of decision making and knowledge may be increased, thus the opportunity
to improve flexibility [54]. Since dynamic flexibility is primarily relying on humans,
approaches from the management literature become relevant: openness and recursive-
ness. Openness is the degree of decision makers’ willingness to accept innovative
ideas, new information sources, and redefinition of roles. Recursiveness is the ability
of decision makers to switch between planning and implementation (P29).
Intention.
Intention characterizes the desire of an organization to address changes, with proactive
or reactive approaches (P3, P10, P12, P13, P40). Proactive reflects the offensive ap-
proach of attempting to control the changes in the environment, before they occur, to
achieve a competitive advantage. Reactive means reacting to changes after they have
occurred and attempting to minimize their impact (P3).
Temporal.
Temporal refers to the time that an organization requires to respond to a change and can
be further divided into three categories: operational, tactical, and strategic (P3, P24).
Operational flexibility is related to short-term changes where changes usually occur
within the range of a day (P3). Operational flexibility is required to deal with minor
failure. Carlsson (P24) describes an organization with operational flexibility that has
built-in procedures that allow a high degree of variation, which can accommodate in-
terruptions in the regular process and alleviate the impacts of change. Examples from
manufacturing include the replacement of a broken machinery part and the substitution
of a raw material with another one, e.g., due to shortage.
Tactical flexibility relates to medium-term changes and can involve design changes
(P3). An organization with tactical flexibility designs its organization structure and pro-
duction equipment to allow changes over the course of the business cycle. An example
from manufacturing is to manufacture a variety of products with similar characteristics
in a way that the same machinery can be used to produce the different products (P24).
Strategic flexibility relates to long-term changes, which may involve changes at the
organizational level (P3). Strategic flexibility is a reflection of how the organization
positions itself against its market, competitors, and the future. Strategic flexibility
includes the acquisition of new machinery or expansion of the business by establishing
a production plant in a new location.
Swiftness.
Swiftness of change reflects on how quickly an organization can or needs to respond to
a change (P2, P20). Swiftness of change can be divided into two categories: immediate
and deferred (P18, P20). Immediate changes are handled immediately, due to their ur-
gency or their foreseeable nature (P18, P20). In contrast, deferring the implementation
of a change may leave room for modification at a later stage, e.g., to account for un-
certainty (P18, P42). The latter approach is preferred to minimize cost and risk when
last-minute change may occur (P43).
Summary.
We identified three flexibility attributes and their associated categories and values. We
also identified approaches associated with different flexibility attributes. In general
each flexibility category or values are quite distinct. However, we found overlap be-
tween the flexibility perspectives from the similarities of the approaches. For example,
employing highly skilled workers (P32, P37) and employees willingness to assume
new responsibilities (P29) are known approaches to address organizational flexibility.
Meanwhile, employing skilled workers with multiple expertise is a known approach
to address process flexibility in P32 and P37. Thus, employing highly skilled workers
can address organization and process/ operation flexibility. Another similarity can be
found in product and operational flexibility. P32 suggests that modular product design
is an approach to address product flexibility. Meanwhile, P17 and P28 suggest the same
approach to address process/ operational flexibility. The overlap between the attributes
further strengthen the need for a flexibility framework that provides a holistic view to
better account for interdependencies and trade-offs of flexibility.
3.4.3 Evaluation of Flexibility Attributes and Agile and Lean Prac-

tices (RQ2)
In the following subsections, we discuss how Agile and Lean practices fit into the cate-
gories and sub-categories of flexibility attributes. We evaluate to what extent Agile and
Lean practices address the properties of change, flexibility perspectives, and flexibility
enablers. Table 3.3–3.5 summarize the evaluation of flexibility attributes against Agile
and Lean practices.
Properties of Change.
Table 3.3 shows the evaluation of each Agile and Lean practice along the properties of
change and their sub-categories. If an Agile and Lean practice does not address any
property of change, the table shows ‘Not Applicable’ (N/A). As we can see from Table
3.3, each Agile and Lean practice may address none, one or more flexibility values of
the properties of change.
Origin of Change.
Internal or external changes could also be a matter of perspective. For the purpose
of clarity in the evaluation of Agile and Lean practices, we focus on the development
Table 3.3: Evaluation of Agile Practices and the Properties of Change
Practices Properties of Change

Origin of Uncertainty Frequency Duration Extent of
change change
On-site customer External Unforeseen Continuous N/A N/A
Metaphors/ Stories N/A N/A N/A N/A N/A
Refactoring Internal N/A Continuous N/A
Coding standards N/A N/A N/A N/A N/A
Team code- N/A N/A N/A N/A N/A
ownership
Low dependency N/A N/A Continuous N/A N/A
arch.
TDD and test au- N/A N/A N/A N/A N/A
tomation
Pair programming N/A N/A N/A N/A N/A
Continuous integra- N/A N/A Continuous N/A N/A
tion
Reviews and inspec- Internal/ N/A Continuous N/A N/A
tion* External
Configuration man- N/A N/A N/A N/A
agement
Incremental deliver- External N/A Continuous N/A Incremental
ies
Internal/external re- External Foreseen/ N/A N/A Incremental
lease Unforeseen
Short iteration/ External N/A Continuous N/A Incremental
Sprint
Adaptive planning External Unforeseen Continuous N/A N/A
Time-boxing N/A N/A N/A N/A N/A
Planning game** Internal/ Unforeseen Continuous N/A N/A
External
Co-located develop- N/A N/A N/A N/A N/A
ment
Cross-functional N/A N/A N/A N/A N/A
team
40 hour week N/A N/A N/A N/A N/A
Stand-up meeting*** Internal N/A Continuous N/A N/A
Team chooses own N/A N/A N/A N/A N/A
task
Definition of done N/A N/A N/A N/A N/A
Informative N/A N/A N/A N/A N/A
workspace ****
VSM N/A N/A N/A N/A N/A
Inventory manage- N/A N/A Continuous N/A N/A
ment
Chief engineer N/A N/A N/A N/A N/A
Kanban pull-system Internal/ Foreseen Continuous N/A N/A
External
* also covers retrospective, ** also covers sprint planning, *** also covers Scrum of Scrum,
**** includes using burn down charts
team. Therefore, anything originates from outside the development team, e.g., cus-
tomers, marketing team and executive officers, are considered external.
From Table 3.3, a number of practices, e.g., planning game, team chooses own
task, and kanban pull-system, address both internal and external changes. The practice
planning game involves stakeholders that are internal and may be external, such as
the customers. Planning game is also used to resolve conflicts (internal) and to plan
the next iteration. The practice team chooses own task is done to develop features
requested by the customers (external). At the same time, the practice team chooses
own task can address changes from within the development team, e.g., changes in team
members’ competencies and availability. The practice kanban pull-system allows the
development team to develop a feature for the customer (external) based on the teams’
capacity or availability (internal). If the capacity of the development team fluctuates,
for example, due to commitments to other development efforts, kanban pull-system can
address such changes.
Refactoring and stand-up meeting are practices that address change from within
the development team. Refactoring is focused on changing the internal structure of
the code and is initiated by the development team. In this case, the customers neither
request a change nor are they affected by it. Meanwhile, one of the intentions of a
stand-up meeting is to identify issues in the team and to find solutions to address these
issues.
We can see that some Agile and Lean practices, like planning game, and reviews
and inspection, can address both internal and external changes. Planning game is
used to plan for the next iteration which concerns the customer (external) and also
continuous reflection (internal). Reviews and inspection, in a form of a retrospective,
can be used to reflect on how the iteration went (external) and also improvement for
the next iteration (both internal and external). De Toni and Tonchia suggest that there
could be interdependencies between internal and external changes (P2).
Uncertainty.
In Table 3.3, we can see that five Agile and Lean practices address uncertainty. Hav-
ing an on-site customer allows the customer to express their needs immediately. In-
ternal/external release also addresses unforeseen changes. Internal releases may be
developed with attention to quality, but kept from being released until the situation in
the market permits a product release. However, when exactly the market situation is
suitable is unpredictable.
Internal/external release can address foreseen changes as well. The internal release
is developed as a baseline of development that is releasable. It suggests that the features
developed into the baseline are foreseeable changes to the product. Adaptive planning
allows the features or requirements to be re-prioritized whenever necessary. Planning

game accommodates unanticipated conflict resolutions as well as changes suggested
by the customers. In contrast, kanban pull-system addresses foreseen changes, because
the development team knows which features are of high priority and decides which
ones they wish to develop given their capacity.
Frequency.
The literature on Agile and Lean literature discusses frequent or continuous changes
but not episodic changes. The focus on continuous changes may be attributed to the
incremental nature of Agile development. As we can see from Table 3.3, 12 Agile
and Lean practices address continuous change. We did not find practices that address
episodic change.
Duration.
Duration is one of the properties of change that is not addressed by Agile practices.
The definitions of Agile and Lean practices described in [164, 239] do not mention
about addressing a change according to its duration.
Although Agile and Lean practices do not address change duration, it does not
necessarily mean that duration is irrelevant in software development contexts. Software
development organizations need to plan and consider whether a change is temporary
or permanent. For example, online or mobile game developer may release an update
to the user interface of the game based on holiday seasons, and the update would be
revoked when the holiday season is over. Meanwhile, a change the game engine would
be something permanent and its implementation requires extensive planning.
Extent of Change.
Agile development is synonymous with iterative and incremental development. Three

Agile practices address incremental change: incremental deliveries, internal/external
release, and short iterations. These three practices focus on delivering a part of work-
ing software that offers early value to the customer and allowing the customers to pro-
vide incremental feedback.
Our evaluation indicates that Agile and Lean practices do not address revolutionary
change. This does not mean that Agile and Lean practices would prevent revolutionary
change. Rather, revolutionary change might be addressed in an incremental fashion
[12, 126].
Flexibility Perspectives.
Table 3.4 shows the evaluation of each Agile practice along flexibility perspectives. We
add a check mark (X) for each perspective that an Agile and Lean practice addresses.
We indicate the lack of support of a flexibility perspective by an Agile and Lean practice
with ‘Not Applicable’ (N/A).
Organization Flexibility.
Table 3.4 shows that there are three Agile and Lean practices that address organiza-
tional flexibility; cross-functional team, team chooses own task, and chief engineer.
In Agile development, teams are cross-functional and involve both system stakehold-
ers and developer roles. In comparison to a silo approach, the cross-functional nature
of teams ease the exchange of information and conflict resolution and improves the
awareness of what the others are doing. If one person becomes unavailable, other team
members may easily perform the required task. Team chooses own task gives the free-
dom to the team members to choose tasks that they feel the most confident to tackle.
The practice allows matching of competencies and the tasks at hand, so that the tasks
are done in the most effective fashion. Both cross-functional team and team chooses
own task allow the roles of team members to be seamlessly added and modified, if nec-
essary, to fit the situation. These benefits fit the examples of achieving organizational
flexibility as discussed in P5, P20, and P29.
Another Agile and Lean practice that addresses organizational flexibility is chief
engineer. A chief engineer is a person who oversees the whole development effort.
Ideally, a chief engineer has cross-functional responsibilities and abilities like being a
good listener and motivator. This approach for building organizational flexibility fits
the idea of employing highly skilled workers with versatile abilities as discussed in P32
and P37.
Product Flexibility.
Table 3.4 shows that product flexibility is a focus of Agile approaches; 24 out of 28
practices address product flexibility. Agile methodology is intended to allow changes
to be incorporated into a software product at any moment during development. One
particular practice supports the development of flexible product, i.e. low dependency
architecture. Low dependency architecture is aligned with the suggested mechanism
from manufacturing to have loosely-coupled and modular product design (P17, P28).
Refactoring although not mentioned in the manufacturing or IS/IT literature also sup-
ports product flexibility. By improving the internal structure of a working code, read-
ability and maintainability is continuously improved.
Table 3.4: Evaluation of Agile Practices and the Flexibility Perspectives
Practices Flexibility Perspectives

Organization Product Process/ Op- Information Infrastructure
eration
On-site customer N/A X N/A X N/A
Metaphors/ Stories N/A X N/A X N/A
Refactoring N/A X N/A X N/A
Coding standards N/A X X X N/A
Team code- N/A X X X N/A
ownership
Low dependency N/A X X N/A N/A
arch.
TDD and test au- N/A X N/A N/A N/A
tomation
Pair programming N/A X N/A X N/A
Continuous integra- N/A X N/A X N/A
tion
Reviews and inspec- N/A X N/A X N/A
tion*
Configuration man- N/A X N/A X N/A
agement
Incremental deliver- N/A X X X N/A
ies
Internal/external re- N/A X N/A N/A N/A
lease
Short iteration N/A X N/A X N/A
Adaptive planning N/A X X X N/A
Time-boxing N/A X N/A N/A N/A
Planning game** N/A X N/A X N/A
Co-located develop- N/A N/A N/A X N/A
ment
Cross-functional X X N/A X N/A
team
40 hour week N/A N/A N/A N/A N/A
Stand-up meeting*** N/A X N/A X N/A
Team chooses own X X X X N/A
task
Definition of done N/A N/A N/A X N/A
Informative N/A N/A N/A X N/A
workspace ****
VSM N/A X N/A X N/A
Inventory manage- N/A X N/A N/A N/A
ment
Chief engineer X X X X N/A
Kanban pull-system N/A X N/A X N/A
The remaining practices may not directly impact the actual code or product, but
they support the ease of modification to the product. Manufacturing defines a flexible
product as a product which parts can be easily added or substituted (P32). A soft-
ware product is more changeable from a physical product [27], and the remaining 22
Agile and Lean practices support the ease of product parts modification. We provide
examples for a number of Agile and Lean practices. On-site customer helps to ensure
that any change requests can be quickly propagated to the development team. Short
iterations and incremental deliveries allow parts of working software to be continu-
ously delivered to the customers and allow customers to provide feedback in a speedy
manner.
Process/ Operational Flexibility.

Process/ operation Flexibility. Performing a task iteratively is one approach to achieve
process/ operation flexibility (P18). Agile and Lean software development focuses on
doing iterative development. As a whole, Agile and Lean methodologies embody pro-
cess/ operation flexibility. However, if we have pinpoint to specific practices, they are
incremental deliveries and adaptive planning. Another practice that would address pro-
cess/ operation flexibility is chief engineer. Chief engineer is a person who has multiple
skills and responsibilities. A chief engineer should have excellent communication and
technical skills. Such an individual with multiple skills is a known approach to support
process flexibility in manufacturing (P32) and IS (P37).
Table 3.4 shows a number of Agile practices that bring process/ operation flexibility
into software development. Coding standards and team-code ownership enable every-
one in the development team to understand the code and contribute to the code without
unnecessary learning time. Coding standards, team-code ownership, and team chooses
own task fit the idea of producing a part of a product using different combinations of
resources (P2), in this case human resources. Low dependency architecture suggests
that each functionality is contained in components that can be developed without de-
pendencies on other components. This practice is aligned with the idea of modular
product components (P32) and interchanging sequence of activities (P2).
Information Flexibility.
The Agile manifesto and the Lean principles favour communication and motivated in-
dividuals over processes and tools [15, 169]. This is supported by a large number of
Agile and Lean practices. Table 3.4 shows 21 Agile practices that address how infor-
mation is shared. Standardization of vocabulary, data format, and naming convention
are known approaches in IS/IT to support information flexibility (P22), such mecha-
nism is captured in coding standards and definition of done. Practices like team code-
ownership, cross-functional team, team chooses own task, and kanban pull-system al-
low team members to have the access to the same information, which promotes swift
decision-making (P22). A stand-up meeting is intended to share the following infor-
mation: (1) what has been achieved, (2) what will be achieved, and (3) hindrances in
achieving the goals. A practice like stand-up meeting address what information has
to be defined and disseminated as a way to achieve information flexibility, which is
aligned with recommendation by Regev et al. (P20).
Infrastructure Flexibility.
We did not find Agile and Lean practices that address infrastructure flexibility. With
the advent of DevOps [233] and continuous delivery [92], infrastructure flexibility is
becoming important. Infrastructure, e.g., virtual machines, development environment,
version control systems and build automation tools, need to be set up to support the
delivery of the software product/service. Software companies like Amazon make over
1,000 deployments per hour that are propagated to 10,000 servers simultaneously3 .
This involves integrating multiple components from various teams and requires support
from the infrastructure to maintain the rate of delivery. If changes to any of the tools
are not management properly, the rate of delivery could be interrupted. New concepts,
like Infrastructure as Code, which suggests that the infrastructure is managed in the
same way as the code [93], could be considered as a complement to Agile and Lean
practices to deal with infrastructure flexibility.
Flexibility Enablers.
In this subsection, we examine different enablers to flexibility and see to what extent
Agile practices address them. Table 3.5 shows the evaluation of each Agile and Lean
practice along the flexibility enablers and their sub-categories. If an Agile and Lean
practice does not address a flexibility enabler, we indicate as ‘Not Applicable’ (N/A).
Approach.
Two Agile and Lean practices enable flexibility using a static approach; coding stan-
dards and low dependency architecture. The use of coding standards is aligned with
the idea of using standardization as a form of static flexibility. With coding standards,
3 Keynote by Jon Riley at Velocity Culture 2011 https://www.safaribooksonline.com/
library/view/velocity-conference-2011/9781449311773/part56.html
Table 3.5: Evaluation of Agile Practices and the Flexibility Enablers
Practices Flexibility Enabler

Approach Intention Temporal Swiftness
On-site customer N/A Reactive Operational/Tactical Immediate
Metaphors/ Stories N/A N/A N/A N/A
Refactoring N/A Proactive N/A N/A
Coding standards Static N/A N/A N/A
Team code-ownership Dynamic N/A N/A N/A
Low dependency arch. Static N/A N/A N/A
TDD and test automation N/A N/A N/A N/A
Pair programming Dynamic N/A N/A N/A
Continuous integration Dynamic Reactive N/A Deferred
Reviews and inspection* Dynamic A N/A N/A N/A
Configuration management N/A N/A N/A N/A
Incremental deliveries Dynamic Reactive Operational/Tactical N/A
Internal/external release N/A Reactive N/A Deferred
Short iteration Dynamic Reactive Operational/Tactical Immediate
Adaptive planning Dynamic Reactive Operational/Tactical Deferred
Time-boxing N/A N/A Operational/Tactical Immediate
Planning game** Dynamic Reactive Operational/Tactical N/A
Co-located development Dynamic N/A N/A N/A
Cross-functional team Dynamic N/A N/A N/A
40 hour week N/A N/A N/A N/A
Stand-up meeting*** Dynamic Reactive N/A N/A
Team chooses own task Dynamic Reactive N/A N/A
Definition of done N/A N/A N/A N/A
Informative workspace N/A N/A Operational N/A
VSM N/A N/A Operational/Tactical Deferred
Inventory management N/A N/A Operational/Tactical N/A
Chief engineer Dynamic N/A N/A Deferred
Kanban pull-system N/A N/A Operational/Tactical Deferred
different developers can contribute to the same part of the code, hence limiting the re-
liance on certain individuals. Low dependency architecture is aligned with the idea of
building a product with modular components. Low dependency architecture enables
flexibility by allowing different components of a software product to be delivered in-
dependently. Therefore, changes can be isolated within a component.
Table 3.5 shows 12 Agile practices that support a dynamic flexibility approach.
Team code ownership, pair programming, co-located development, cross-functional
team, team chooses own task and chief engineer fit the idea of openness to address
flexibility. Team members are encouraged to accept new ideas and enact different
roles (P29). Continuous integration, incremental deliveries, short iteration, adaptive
planning, reviews and inspection and stand up meeting fit the idea of recursiveness,
where the team members can go back and forth between planning and implementation.
The combination of openness and recursiveness allows changes to be accepted at any
time of development, and team members are given the liberty of how to best implement
these changes (P29).
Intention.
Table 3.5 shows 10 Agile and Lean practices that enable flexibility through a reactive
approach and one practice that enables flexibility through a proactive approach. Prac-
tices like on-site customers, continuous integration and incremental deliveries allow
the customer to provide feedback and to request changes to the developed product.
The developers can then react to those changes. Refactoring fits into the proactive cat-
egory. The development team is making changes in the internal structure of the code
to improve its maintainability and modifiability in the future. These changes are not
based on incoming customers requests (P3, P10, P40).
Temporal.
Table 3.5 shows that Agile and Lean practices can enable operational or tactical flexibil-
ity. Eight Agile practices enable both operational and tactical flexibility. Incremental
deliveries can be used for troubleshooting to eliminate bugs (operational flexibility).
Practices like on-site customer can accommodate change requests that might require
modification to the product design (tactical flexibility).
We did not find any Agile or Lean practice that enables strategic flexibility, i.e.
flexibility at the organizational level. Agile and Lean practices are implemented at the
level of a project intended to deliver a software product or service. This limitation does
not mean that Agile and Lean would hinder strategic changes like mergers or financial
crises [228]. However, the implementation of Agile and Lean practices at a strategic
level requires tailoring and needs to scale [181].
Swiftness.
Table 3.5 shows that Agile and Lean practices address changes immediately or de-
ferred. Agile and Lean practices like on-site customer, short iteration and time-boxing
are intended to implement changes as soon as they arise (immediate). Meanwhile, six
Agile and Lean practices, such as internal/external release and adaptive planning, are
intended to delay change implementation (deferred). Team members can intentionally
postpone the implementation of a change or a release until the situation fits or until
more information is available (P18, P42). Therefore, waste and rework due to unclear
information can be avoided.
Summary.
The results of our evaluation shows that Agile and Lean practices address a wide range
of flexibility attributes. For certain flexibility categories, Agile and Lean practices have
more emphasis on one value than the other. For example, Agile and Lean practices
have more focus addressing external changes than internal changes. Agile and Lean
practices also focus more on human resources (dynamic approach) than automation as
one of the flexibility enablers, which fits to the first point of the Agile manifesto. The
only flexibility attribute that Agile and Lean practices do not address is infrastructure
flexibility.
It is important to note that the results of our evaluation do not imply that Agile
and Lean practices have limited capabilities to address flexibility. It indicates that
definitions provided in [164] and [239] discuss limited concerns. For instance, Petersen
in [164] defines metaphors/ stories as how requirements are defined, it focuses more on
how the information is presented. Therefore it does not address any of the properties
of change. Similar to how informative workspace is defined in [239], it focuses more
displaying information on the progress within a sprint. Co-located development and
cross functional team are more focused on how the members are located and organized,
not so much what kind of changes they can address [164]. As mentioned by Williams
[239], the actual implementation of Agile practices may vary in industry. The results
of our evaluation does not reflect the actual Agile and Lean implementation in industry.
The evaluation of Agile and Lean practices also shows that the practices themselves
are at different level of granularity. For example, planning game or TDD suggest a
process that includes a series of activities. Meanwhile, coding standards does not
reflect an activity nor an action. This differences of granularity may explain why certain
3.5 Discussion 79
practices do not address any category within a flexibility attribute, as the examples
in the paragraph above show. Although individually each practice has limitations to
address different flexibility attributes, all together 28 Agile and Lean practices address
a wide range of flexibility attributes. This indicates interdependencies between Agile
and Lean practices.
The application of our proposed flexibility framework identified a gap in Agile and
Lean practices, i.e. they do not address change duration and infrastructure flexibility.
The results of our evaluation indicates the need for a comprehensive flexibility frame-
work to identify gaps in addressing flexibility.
3.5 Discussion
This chapter presents a flexibility framework built by aggregating flexibility attributes
from 43 studies on flexibility. The framework was applied as a proof-of-concept to
assess how well 28 Agile and Lean practices address flexibility. The framework is
the result of surveying flexibility comprehensively and transferring the results into the
software engineering domain. With the application of the framework, we demonstrated
how the framework can be used to assess coverage of flexibility concerns. The assess-
ment results show the strengths and limitations of Agile and Lean practices for building
flexibility. In the following, we discuss the framework and the assessment results as
well as the implications for research and practice.
3.5.1 Flexibility Attributes

Our literature review aggregated flexibility attributes from interdisciplinary research
into a flexibility framework (see Figure 3.4). The framework includes three flexibil-
ity attributes: (1) properties of change, (2) flexibility perspectives, and (3) flexibility
enablers. Each flexibility attribute has different flexibility categories and values.
The result of our literature review shows that there are various concerns to be ad-
dressed pertaining to flexibility. The Agile manifesto [15] suggests to welcome change
over following a plan. However, the manifesto does not elaborate the different types of
change. Our literature review suggests that understanding the properties of change is
important to see if the appropriate methods or practices are in place.
Flexibility in software engineering has been largely focused on the software prod-
uct flexibility. However, Codenie et al. [44] indicates flexibility is a trade-off to product
variability and aspects like expertise and uncertainty needs to be considered. Codenie
et al.’s work further supports the need for a flexibility framework that provides a com-
prehensive view of flexibility for a software organization. The resulting framework
from our literature review covers various aspects of change like uncertainty and differ-
ent flexibility perspectives.
The result of our literature review also shows overlap and interdependencies be-
tween the flexibility attributes and their respective categories and values. As mentioned
in Section 3.4.2, one approach or mechanism can address more than one flexibility per-
spectives. This result corroborates with past studies in flexibility in other domains such
as BPM [180] and manufacturing [197, 118]. This result also shows that the flexibility
attributes are not independent of each other, trade-offs and conflicts between them may
arise. This further strengthen the need for a comprehensive view of flexibility.
The flexibility enabler that we identified are known concepts in software devel-
opment. The dichotomy of static versus dynamic approaches was already considered
30 years ago with Fred Brooks’ distinction between “accidental” repetitive tasks that
could be automated and “essential” tasks that require human expertise [27]. The proac-
tive and reactive intentions are a concern when deciding about the adoption of software
product lines [40]. The strategic, tactical and operational bearings are distinguished
in requirements engineering when goals and requirements are analyzed and traced [9].
The dichotomy of immediate and deferred swiftness matters in Agile and Lean as our
flexibility assessment has shown.
As mentioned in Section 3.2.4, classifications of flexibility dimensions and at-
tributes already exist. Table 3.6 summarizes the contribution of the existing classifi-
cations with respect to the flexibility attributes.
Table 3.6: Summary of Existing Classifications
Properties of change Flexibility perspectives Flexibility enablers

Regev et al. (P20) Addressed comprehensively - -
Van der Aalst and Jablonski (P5) Addressed comprehensively - -
De Toni and Tonchia (P2) Only focused on source of change Only focus on product and process. Only focus on approach and
Machinery is irrelevant. intention
Byrd et al. (P22) - Addressed comprehensively -
Reichert and Weber [180] Addressed comprehensively yet Only focus on product Only focus on swiftness
some elements are too domain
specific
As we can see from Table 3.6, each of these classifications addresses one particular
flexibility attribute or focus on specific categories only. Regev et al. (P20) and Van der
Aalst and Jablonski (P5) focus on the properties of change. De Toni and Tonchia (P2)
describe attributes of flexibility, but their contribution does not extensively address the
properties of change. Byrd et al. (P22) only cover the perspectives of flexibility and
do not consider other flexibility attributes. Reichert and Weber [180] address various
flexibility attributes. However, some categories are missing or they are too specific.
3.5 Discussion 81
Our flexibility framework integrates these views and provides a more comprehensive
picture of flexibility attributes which are not too domain specific.
In formulating the flexibility framework, we also excluded flexibility perspectives
that have only marginal relevance to software development such as, machinery, ma-
terial handling, and volume flexibility. We considered that in software development,
there is no machinery that automates the production of software instances. Also, no
physical raw materials are used for the instantiation of a software system. Further-
more, production volume is not relevant in software development, since the number
of software instances that are produced would not generate significant adjustment in
the development processes. In the context of software development, it is important to
differentiate scalability and production volume [237]. A software system may need to
scale, for example, due to an increase in the number of end-users. This scaling can
be achieved by implementing different design patterns or architectural decisions [154].
In manufacturing, increasing production volumes would require more machines, raw
materials, and human resources [54], and may require changes to production lines in
addition to redesigning the product itself.
It is also important to stress that the framework is intended to evaluate whether the
mechanisms, e.g., practices, tools, or methods, that are in place in the organization are
sufficient to address the required flexibility. The idea of framework is not to enforce
software organizations to address all flexibility attributes, but the attributes that are
important for them. For example, if a software organization only needs to be reactive
in addressing change because most of the time changes are unpredictable, then it is less
important to develop capabilities to be proactive. Moreover, the flexibility framework
offers the following:
• The flexibility framework offers new insights regarding flexibility in software
engineering. The flexibility attributes and their categories were already known,
but they were not associated specifically with flexibility yet.
• The flexibility framework provides an instrument for sharing terminology and
understanding of flexibility in software development contexts. For instance, in
software engineering, approaches for automation and standardization were not
associated with the term static flexibility.
3.5.2 Flexibility and Agile and Lean Practices

As a proof-of-concept, we applied the flexibility framework to evaluate to what extent
Agile and Lean practices are addressing the flexibility attributes mentioned in Subsec-
tion 3.4.2. Our evaluation shows that Agile and Lean practices address a broad range of
the flexibility attributes in our framework. However, some flexibility categories were
not addressed. A summary of flexibility attributes addressed by Agile and Lean prac-
tices is shown in Table 3.7.
Table 3.7: Mapping of Flexibility Attributes with Agile and Lean practices
Properties of change Flexibility Perspectives Flexibility enablers

Origin of change: External, Internal Organization Approach:Dynamic, static
Uncertainty: Unforeseen, Foreseen Product Intention: Reactive, proactive
Frequency: Continuous Process Temporal: Operational, Tactical
Extent of change: Incremental Operation Swiftness: Deferred
(Duration) Information
(Infrastructure)
Values in bold indicate strong focus
Values in parentheses indicate gaps
Agile and Lean Practices – Properties of Change.

Our evaluation shows that Agile and Lean practices focus on changes that are exter-
nal to the organization; unforeseen, continuous, and incremental. Other properties of
change have not been considered yet in the Agile and Lean software engineering con-
text. Our earlier work indicates that software developing organizations may need to
address internal and foreseeable changes in addition to the external and unforeseen
changes [151]. There are limitations to Agile and Lean practices when they are im-
plemented without tailoring them to the organization’s contexts [32, 72]. The present
literature review shows the need to consider the properties of change to achieve flexi-
bility and the need to evaluate the impact of properties of change when tailoring Agile
and Lean practices.
Agile and Lean Practices – Flexibility Perspectives.

Our evaluation shows that Agile and Lean practices cover many flexibility perspec-
tives, with an emphasis on product and information flexibility. We can relate this
emphasis to the Agile and Lean principles working software (product), build quality
in (product), face-to-face conversation (information), and amplify learning (informa-
tion) [15, 164, 169]. The lack of focus on process and operation flexibility could be
attributed due to the lack of distinction of the two terms in software development con-
texts (see paragraph 5 Section 3.5.1). Our evaluation also shows that Agile and Lean
practices do not address infrastructure flexibility. Agile and Lean methods aim at re-
ducing development costs and time when faced with change requests [15, 164]. With
3.5 Discussion 83
the growth of Internet companies like Amazon, Google, and Facebook [136], infras-
tructure flexibility has become more important. Our study reveals a gap in the Agile
and Lean practices. There is a need to evaluate how existing Agile and Lean practices
facilitate infrastructure flexibility and how emerging methods like DevOps [233] can
be integrated into Agile and Lean development.
Agile and Lean Practices – Flexibility Enablers.
Agile and Lean practices address many flexibility enablers. They promote a dynamic
approach to flexibility because they focus on people’s abilities to perform critical anal-
ysis and learn from experience to address changes [15, 164, 169].
Our evaluation also shows that Agile and Lean practices support reacting to change
and deferring decisions, thus support developing features that are truly needed mini-
mizing waste [120, 164, 169]. These benefits are achieved under the assumption that
all team members are collocated [223]. Being reactive and deferring decisions as late
as possible could be problematic when team members are not collocated, because or-
ganizing meetings requires advanced planning and coordination [223]. This shows
that there are underlying assumptions associated with Agile and Lean practices. If the
assumptions are not fulfilled, the flexibility enablers may not effectively address the
flexibility needs. More studies are required to understand how software development
contexts influence flexibility enablers’ effectiveness to address flexibility needs.
Agile and Lean methods try to avoid big upfront design and long-term planning
[12, 120]. Instead, they focus on operational and tactical flexibility but have strategic
flexibility as a missing part. When we look beyond individual projects and view a
software organization as a whole, there are decisions at the strategic levels that require
long-term planning. More research is necessary to understand better how to tailor Agile
and Lean practices for strategic planning.
Reflection on the Flexibility Framework.
The application of the flexibility framework when used to evaluate Agile and Lean
practices indicate the following:
• Known mechanisms for flexibility in other disciplines like manufacturing, IT/IS,
or management are applicable in software development. For example, having
modular product design (P17, P28, P32) is a known way to develop product
flexibility in manufacturing. This is approach is also applicable in software de-
velopment contexts and addressed by a number of Agile and Lean practices.
Another example is to have a person with multiple skills, known in IS (P37)
and manufacturing (P3). Agile and Lean practices address this by having a chief
engineer.
• Further stresses the need of a flexibility framework. The application of the flexi-
bility framework allowed the identification of gaps. Our evaluation indicate that
Agile and Lean practices may not be sufficient to address change duration and
infrastructure flexibility. This is not in any way reveal a limitation of Agile and
Lean practices in addressing flexibility in industry. Our evaluation is only based
on the definitions in literature. The actual implementation of Agile and Lean
practices may deviate from their description in the literature [239].
• Our evaluation indicates that Agile and Lean practices are interdependent on
each other to address a wide range of flexibility attributes. This result corrob-
orates to that of Beck who suggests that XP practices are interdependent [12].
The result of our evaluation also shows the need to further evaluate the most ap-
propriate combinations of Agile and Lean practices that software organizations
should implement to achieve the most suitable variation of flexibility attributes.
3.5.3 Implications for Research

Evaluations of flexibility attributes. In this literature review, we identified flexibility at-
tributes from a range of domains that apply to software engineering. To further under-
stand their meaning and operationalizations in software development contexts, there is
a need for empirical investigations. For instance, in the temporal category of flexibility
enablers, there is no agreed definition of operational, tactical and strategic flexibility.
For a large company, operational flexibility could entail a day to day activity. For a
small start-up company, operational flexibility could imply an hourly activity.
Evaluation of Agile and Lean practices. Our evaluation shows that the 28 Agile
and Lean practices address the flexibility attributes well, except for process and infras-
tructure flexibility. This assessment is based on definitions provided in the literature
and might differ in actual implementations of these practices. Empirical investigations
would be required to examine how well Agile and Lean practices address flexibility
attributes in different contexts.
Investigating trade-offs. There are costs and trade-offs associated with how flexi-
bility is built [118, 151]. Making one aspect more flexible can reduce the flexibility
of another aspect [64]. Research is needed to understand and predict the benefits and
trade-offs of approaches to flexibility. For instance, for a software organization that
maintains legacy systems and rarely gets any change request, is it better to be more
focused on being proactive than being reactive? What are the possible benefits and
limitations of implementing one strategy compared to the other?
3.6 Conclusions 85
Formulation of Flexibility Improvement Frameworks The flexibility framework that

we present in this chapter could also be used by other researchers as an evaluation tool
during an empirical study in Agile and Lean, as well as flexibility. Together with indus-
try partners, researchers can use the flexibility framework to identify gap in addressing
flexibility in industry. By assessing how the current Agile and Lean implementation ad-
dress the flexibility attributes, improvement suggestions could be proposed. Trade-off
analysis of the improvement suggestions could then be performed. The lessons learned
from the various industry partners could be use as a foundation to formulate flexibility
improvement frameworks.
3.5.4 Implications for Practice

Understanding flexibility profiles. Based on our proof-of-concept mapping, the pro-
posed flexibility framework might help managers and practitioners to evaluate flexibil-
ity within their specific software development contexts. Practitioners can attempt to
perform a similar mapping exercise as we showed in this chapter. They can use the
framework to understand the nature of changes that they face using the properties of
change and answer questions like is the origin of the change internal or external, and
is the change frequent? Also, they can use the framework to understand which per-
spectives are primarily affected by the changes. Finally, they can use the framework to
better understand which flexibility enablers are currently in place in the organization.
Identify gaps. Managers and practitioners can better assess whether a software or-
ganization’s current flexibility profile is aligned with what they envision. For instance,
a flexibility profile may indicate that the development team is addressing a continu-
ous stream of unforeseen changes by being reactive. However, the manager may want
the development teams to become proactive. Once such a gap is identified, flexibility
improvement strategies can be planned [16, 224]. An improvement action may be to
encourage the organization’s staff to build abilities for predicting changes. Such as-
sessments and improvements could be systematized, eventually leading to flexibility
improvement frameworks.
3.6 Conclusions
In this literature review, we aggregated flexibility attributes from inter-disciplinary
literature. We adopted an iterative approach using backward and forward reference
search. Overall, we identified 43 papers that contributed relevant flexibility attributes.
Using holistic and axial coding we consolidated these flexibility attributes into a flexi-
bility framework.
To summarize the findings, we revisit the research questions that were formulated
in the study.
1. RQ1. Which flexibility attributes are reported in the literature?
The results of our cross disciplinary shows that flexibility has three attributes:
properties of change, flexibility perspectives, and flexibility enabler. Each flex-
ibility attribute has multiple categories, and each category has multiple values.
The flexibility attributes included in the flexibility framework are presented in
Figure 3.4. These attributes reflect the primary concerns that needs to be consid-
ered when evaluating software organization flexibility.
2. RQ2. To what extent do Agile and Lean practices address flexibility attributes?
We found that Agile and Lean practices cover most categories and values of the
flexibility attributes, but not all. Agile and Lean practices are focused more on
reactive than proactive approaches to addressing changes. The flexibility per-
spectives infrastructure is not at all addressed. The evaluation results indicate
that the need of a comprehensive flexibility framework. The evaluation also in-
dicates that mechanisms for flexibility in other related disciplines are applicable
in software development.
The flexibility framework discussed in this chapter provides a basis for developing
a common understanding of flexibility in software development. It can also help to
assess software development flexibility along three flexibility attributes; properties of
change, flexibility perspectives, and flexibility enablers.
Agile and Lean practices cover the flexibility attributes well, but not completely.
For future work we propose the following: (1) empirical investigations of what the
flexibility attributes mean in practice and how they are implemented; (2) empirical
evaluations of how Agile and Lean implementations in industry address the flexibility
attributes; (3) investigating and cataloging the benefits and trade-offs of implemen-
tations of flexibility enablers; and (4) formulation of flexibility improvement frame-
works.
Appendix
List of Primary Studies
P1. De Haan J, Kwakkel J, Walker W, Spirco J,Thissen W. Framing Flexibility: Theorising and Data Min-
ing to Develop A Useful Definition of Flexibility and Related Concepts. Futures 2011; 43(9):923 –
933.
P2. De Toni A, Tonchia S. Manufacturing Flexibility: A Literature Review. International Journal of Pro-
duction Research 1998; 36(6):1587 – 1617.
3.6 Conclusions 87
P3. Golden W, Powell P.Towards a Definition of Flexibility: In Search of the Holy Grail? Omega 2000;
28(4):373 – 384.
P4. Thomke S. The Role of Flexibility in the Development of New Products: An Empirical Study. Research
Policy 1997;26(1):105 – 119.
P5. Van der Aalst W, Jablonski S. Dealing with Workflow Change: Identification of Issues and Solutions.
Computer Systems Science and Engineering 2000; 15(5):267 – 276.
P6. Chang AY. Prioritising the types of manufacturing flexibility in an uncertain environment. International
Journal of Production Research 2012; 50(8):2133 – 2149.
P7. Jain A, Jain P,Chan FT, Singh S. A review on manufacturing flexibility. International Journal of Pro-
duction Research 2013; 51(19):5946 – 5970.
P8. Ryan ET, Jacques DR, Colombi JM. An ontological framework for clarifying flexibility-related termi-
nology via literature survey. Systems Engineering 2013; 16(1):99 – 110.
P9. Vokurka RJ, O’Leary-Kelly SW. A review of empirical research on manufacturing flexibility. Journal
of Operations Management 2000; 18(4):485 – 501.
P10. Yu K, Cadeaux J, Luo BN. Operational flexibility: Review and meta-analysis. International Journal of
Production Economics 2015; 169:190 – 202.
P11. Correa HL, Slack N. Framework to Analyse Flexibility and Unplanned Change in Manufacturing Sys-
tems. Computer Integrated Manufacturing Systems 1996; 9(1):57 – 64.
P12. Weick KE, Quinn RE. Organizational Change and Development. Annual Review of Psychology 1999;
50(1):361 – 386.
P13. Gerwin D. Manufacturing Flexibility: A Strategic Perspective. Management Science 1993; 39(4):395
– 410.
P14. Huber G, McDaniel R. The Decision-making Paradigm of Organizational Design. Journal of Manage-
ment Science 1986; 32(5):572 – 589.
P15. Li Y, Chang KC, Chen HG, Jiang JJ. Software Development Team Flexibility Antecedents. Journal of
Systems and Software 2010; 83(10):1726 – 1734.
P16. Phillips F, Tuladhar SD. Measuring Organizational Flexibility: An Exploration and General Model.
Technological Forecasting and Social Change 2000; 64(1):23 – 38.
P17. Sanchez R. Preparing for an Uncertain Future: Managing Organizations for Strategic Flexibility. Inter-
national Studies of Management & Organization 1997; 27(2):71 – 94.
P18. Schonenberg H, Mans R, Russell N, Mulyar N, Van der Aalst W. Towards a Taxonomy of Process
Flexibility. Proceedings of the 20th International Conference on Advanced Information Systems Engi-
neering, vol. 8, 2008: 81 – 84.
P19. Smith PG. Change: Embrace It, Don’t Deny It. Research-Technology Management jul 2008; 51(4):34
– 40.
P20. Regev G, Soffer P, Schmidt R. Taxonomy of Flexibility in Business Processes. Proceedings of the 7th
Workshop on Business Process Modelling, Development and Support, 2006; 90 – 93.
P21. Nurcan S. A Survey on the Flexibility Requirements Related to Business Processes and Modeling
Artifacts. Proceedings of the 41st Hawaii International Conference on System Sciences (HICSS 2008),
2008;378 – 387.
P22. Byrd T, Jacome Madariaga L, Byrd L, Mbarika V. An Examination of an Information Systems Flexibil-
ity Framework. Proceedings of the 43rd Hawaii International Conference on System Sciences (HICSS
2010), 2010;1 – 10.
P23. Byrd T, Turner D. Measuring the Flexibility of Information Technology Infrastructure: Exploratory
Analysis of a Construct. Journal of Management Information Systems 2000; 17(1):167 – 208.
P24. Carlsson B. Flexibility and the Theory of the Firm. International Journal of Industrial Organization
1989 ; 7(2):179 – 203.
P25. Evans J. Strategic Flexibility for High Technology Manoeuvres: A Conceptual Framework. Journal of
Management Studies 1991; 28(1):69 – 89.
P26. Kara S, Kayis B, O’Kane S. The Role of Human Factors in Flexibility Management: A Survey. Human
Factors and Ergonomics in Manufacturing and Service Industries 2002; 12(1):75 – 119.
P27. Regev G, Bider I, Wegmann A. Defining Business Process Flexibility with the Help of Invariants.
Software Process: Improvement and Practice 2007; 12(1):65 – 79.
P28. Sanchez R,Mahoney JT. Modularity, Flexibility, and Knowledge Management in Product and Organi-
zation Design. Strategic Management Journal 1996; 25(Winter 1996):50 – 61.
P29. Sharfman M, Dean Jr J. Flexibility in Strategic Decision Making: Informational and Ideological Per-
spectives. Journal of Management Studies 2003; 34(2):191 – 217.
P30. Shirzad SR, Bell D. A Systematic Literature Review of Flexible E-Procurement Marketplace. Journal
of theoretical and applied electronic commerce research 08 2013; 8:49 – 70.
P31. Asadi N, Fundin A, Jackson M. The essential constituents of flexible assembly systems: A case study
in the heavy vehicle manufacturing industry. Global Journal of Flexible Systems Management 2015;
16(3):235 – 250.
P32. Sethi AK, Sethi SP. Flexibility in Manufacturing: A Survey. International Journal of Flexible Manu-
facturing Systems 1990; 2:289 – 328.
P33. Norouzilame F,Wiktorsson M,Salonen A. An Industrial Perspective on Flexible Manufacturing: A
Framework for Needs and Enablers. Proceedings of the 22nd International Conference on Production
Research, 2013.
P34. El Maraghy HA. Flexible and reconfigurable manufacturing systems paradigms. International Journal
of Flexible Manufacturing Systems 2005; 17(4):261 – 276.
P35. Jain V,Raj T. Ranking of flexibility in flexible manufacturing system by using a combined multiple
attribute decision making method. Global Journal of Flexible Systems Management 2013; 14(3):125 –
141.
P36. Knoll K, Jarvenpaa S. Information Technology Alignment or ”fit” in Highly Turbulent Environments:
The Concept of Flexibility. Proceedings of the 1994 Computer Personnel Research Conference on
Reinventing IS, ACM, 1994; 1 – 14.
P37. Nelson K, Nelson H, Ghods M. Technology Flexibility: Conceptualization, Validation, and Measure-
ment. Proceedings of the 30th Hawaii International Conference on System Sciences (HICSS 1997),
vol. 3, IEEE, 1997; 76 – 87.
P38. Duncan N. Capturing IT Infrastructure Flexibility: A Study of Resource Characteristics and Their
Measure. Journal of Management Information Systems 1995; 12(2):37 – 57.
P39. Bernardes ES, Hanna MD. A Theoretical Review of Flexibility, Agility and Responsiveness in the
Operations Management Literature: Toward A Conceptual Definition of Customer Responsiveness.
International Journal of Operations & Production Management 2009; 29(1):30 – 53.
P40. Palanisamy R. Building information systems flexibility in SAP – LAP framework: A case study evi-
dence from SME sector. Global Journal of Flexible Systems Management 2012; 13(1):57 – 74.
P41. Saleh JH, Mark G, Jordan NC. Flexibility: A Multi-disciplinary Literature Review and a Research
Agenda for Designing Flexible Engineering Systems. Journal of Engineering Design 2009; 20(3):307
– 323.
P42. Van der Aalst W, Pesic M, Schonenberg H. Declarative Workflows: Balancing Between Flexibility and
Support. Computer Science - Research and Development 2009; 23(2):99 – 113.
P43. Wadhwa S, Bibhushan, Bhoon K, Chan F. Postponement Strategies for Re-engineering of Automotive
Manufacturing: Knowledge-management Implications. The International Journal of Advanced Manu-
facturing Technology 2008; 39(3-4):367 – 387.
3.6 Conclusions 89
Table AI: Categories of Properties of Change
Category References
Origin of change [P1, P2, P3, P4, P5, P6, P7, P8, P9, P10]
Frequency [P11, P12]
Uncertainty [P1, P3, P8, P11, P13, P14, P15, P16, P17, P18, P19]
Duration [P1, P18, P20]
Extent of Change [P1, P11, P18, P20, P21]
Table AII: Categories of Flexibility Perspectives
Category References
Organization [P2, P5, P9, P10, P14, P19, P22, P23, P24, P25, P26, P27, P28, P29,
P30, P31]
Product [P2, P9, P10, P17, P19, P23, P28, P31, P32, P33, P34, P35]
Process [P2, P5, P9, P10, P31, P32, P33, P34, P35, P36, P37]
Information [P2, P5, P20, P22, P23]
Operation [P2, P5, P32, P33]
Infrastructure [P10, P22, P23, P32, P38]
Table AIII: Categories of Flexibility Enabler
Category References
Approach [P2, P7, P28, P29, P31, P34, P35, P36, P38, P39, P40]
Intention [P2, P3, P10, P12, P13, P15, P25, P40]
Temporal [P1, P3, P10, P12, P24, P30]
Swiftness [P2, P17, P18, P20, P41, P40, P42, P43]
Chapter 4
The Impacts of Agile and Lean

Practices on Project
Constraints: A Tertiary Study
Indira Nurdiani, Jürgen Börstler, and Samuel A. Fricker. “The Impacts

of Agile and Lean Practices on Project Constraints: A Tertiary Study”
Journal of Systems and Software 119 (2016), pp. 162-183,
(DOI:10.1016/j.jss.2016.06.043)
4.1 Introduction
The software industry is highly dynamic and competitive. Software organizations need
to accommodate frequent changes in their environment, e.g., customer needs, regula-
tions, and technology, to maintain their competitive advantage [30]. At the same time
software organizations need to deliver in shorter lead times, better quality, and with
lower budgets. Flexible approaches such as Agile and Lean software development
have emerged to provide solutions to this situation [58, 126].
92 The Impacts of Agile and Lean Practices on Project Constraints
Agile software development is a group of software development methodologies,

e.g., Extreme Programming (XP), Scrum, Crystal, etc., that focus on delivering work-
ing software products in small iterations, being adaptive towards requirement changes,
and collaborating closely with the customers [1]. Lean software development is a set of
principles and tools derived from Lean manufacturing that focuses on removing waste,
delaying decisions as late as possible, and building quality into the product as early as
possible [169].
The growth of interest in Agile and Lean software development is reflected by the
large number of research papers published between 2001 and 2010 [57]. However, the
strength of empirical evidence regarding the benefits of Agile software development
is still very low [59]. The benefits and limitations of implementing Agile and Lean
practices remain in question.
To better understand the benefits and limitations of Agile and Lean practices, we
performed a tertiary study, which is a review of secondary studies [110]. A secondary
study is a review of relevant primary studies to synthesize evidence pertaining to a
specific research question. Meanwhile a primary study is an empirical study that inves-
tigates a specific research question [110], e.g., case study, experiment, etc. A tertiary
study can be performed when there are a sufficient number of secondary studies (ibid.).
There are different types of secondary studies:
• A Systematic Literature Review (SLR) is “a form of secondary study that uses
a well-defined methodology to identify, analyze and interpret all available evi-
dence related to a specific research question in a way that is unbiased and (to a
degree) repeatable.” [110].
• A Literature Review (LR) is “the use of ideas in the literature to justify the par-
ticular approach to the topic, the selection of methods, and demonstration that
this research contributes something new.” [87].
• A Systematic Mapping (SM) is a form of secondary study that “provides an
overview of a research area, and identify the quantity and type of research and
results available within it.” [168]
The number of secondary studies in Agile and Lean methodologies has increased
over the years. Many of the secondary studies report on the benefits and limitations
of Agile and Lean methods and practices. These benefits and limitations are often
measured with respect to quality, budget, schedule, which are also known as project
constraints [171]. With the increasing number of secondary studies in Agile and Lean
methodologies, it becomes important to get a good overview of the existing body of
knowledge. Researchers can then build on this body of knowledge or seek new av-
enues of research. Meanwhile, practitioners get an overview that helps them to better
understand the benefits and limitations of Agile and Lean practices.
4.2 Tertiary Studies in Software Engineering 93
The purpose of this tertiary study is to consolidate findings on the impacts of imple-
menting Agile and Lean practices as reported in secondary studies. First, we provide
an overview of what has been studied in secondary studies on Agile and Lean soft-
ware development. Second, we examine Agile and Lean practices and their respective
impacts and summarize the extent of empirical support as reported in the secondary
studies. The outcome of this study is a mapping of Agile and Lean practices and their
impacts on project constraints together with the number of empirical studies that sup-
port them. The outcome of this study can be used by researchers to identify research
gaps and by practitioners to make informed decisions regarding the implementation of
The remainder of this chapter is structured as follows: Section 4.2 gives an overview
over related work. Section 4.3 presents the review steps. Section 4.4 provides an
overview of the found secondary studies on Agile and Lean methodologies. Section
4.5 presents the results and a detailed analysis of the impacts of Agile and Lean prac-
tices on project constraints. Section 4.6 presents a discussion of the results and Section
4.7 summarizes and concludes the review.
4.2 Tertiary Studies in Software Engineering

Evidence Based Software Engineering (EBSE) [112] has its origin in Evidence Based
Medicine (EBM) [216]. The goal of EBM is to integrate research evidence with clinical
expertise for patients’ well-being. [112] translated the goals of EBM to suit the contexts
of software engineering and formulated the goals of EBSE: (1) a common goal for
researchers, (2) a means for industry practitioners to make informed decisions about
technology or software development methodology adoption, (3) improve dependability
of software systems, (4) improve acceptability of software intensive systems, and (5)
an input for certification.
EBSE advocates a more systematic approach to the design and execution of sec-
ondary studies, as for example in an SLR [112]. The aim of an SLR is “to identify,
evaluate and summarize the findings of all relevant individual studies, thereby mak-
ing the available evidence more accessible to decision-makers” [34]. The result of
combining multiple relevant individual studies is expected to provide a more reliable
and precise estimate of the effectiveness of an intervention. In our tertiary study, an
intervention is an Agile and Lean practice.
When there are many secondary studies in an area such as in software engineering,
it is possible to perform a tertiary study [110]. Kitchenham et al. performed a tertiary
study aimed at assessing impacts of SLRs in software engineering [113]. The study
highlights a number of issues pertaining to time spent in performing the review and
difficulty to assess the primary studies’ quality. Kitchenham et al. then published a
follow up tertiary study and found that the number of SLRs are increasing, and that
their quality is improving [114]. However, only a few SLRs perform evaluations of the
primary studies and provide guidelines for practitioners.
MacDonell et al. performed a tertiary study to evaluate the reliability of two inde-
pendent SLRs with a common research question in the area of effort prediction models
[134]. The tertiary study shows that although the two SLRs have different review de-
signs and execution, the results are rather consistent. The tertiary study concludes that
systematic reviews are a reasonably robust and reliable research method.
Wohlin et al. performed a tertiary study to evaluate the reliability of two mapping
studies performed by two independent teams within the topic of product families testing
[240]. Both mapping studies classified the primary studies using the same scheme
proposed by Wieringa et al. [238]. Despite using the same scheme, the mapping studies
did not share the same classifications. The tertiary study shows that the results of a
secondary study depend on the contexts of the secondary study such as the topic being
studied, reviewers conducting the study, search strategies, and available data from the
primary studies. However, Wohlin et al. also suggest that secondary studies may reach
the same general conclusion even if they include different primary studies [240].
A tertiary study in Global Software Development (GSD) pertaining to GSD risks
and risk mitigation strategies was conducted by Verner et al. [229]. Their study indi-
cates that most SLRs in GSD are mainly literature summaries or mapping studies for
researchers. Most SLRs do not provide evidence for the GSD risks that they identified.
Also, most SLRs do not provide any advice for practitioners.
There exists one tertiary study regarding Agile methods [84]. However, this tertiary
study focuses specifically on the applicability of Agile practices in GSD settings and
is based on only two secondary studies. It shows that there is a growing trend of im-
plementing Agile methodologies in GSD. Certain Agile practices, such as continuous
integration, pair programming, retrospective, and daily stand up meetings are com-
monly implemented in GSD contexts. The results indicate that most Agile practices
implemented in GSD are successful. However, the indicators of success of implement-
ing Agile were not reported in the tertiary study.
Currently, the benefits and limitations of Agile and Lean practices are reported in
independent secondary studies. A consolidated view of findings is required to get a full
picture of the benefits and limitations of Agile and Lean practices. Such a consolidated
view would allow researchers and practitioners to compare the findings and more easily
assess the possible impacts of Agile and Lean practices.
4.3 Review Method 95
4.3 Review Method

The aim of this tertiary study is to provide a consolidated view of findings on the
impacts of Agile and Lean practices as reported in secondary studies. The consolidated
view will provide researchers and practitioners with more information on whether the
impacts of Agile and Lean practices are substantiated with empirical evidence. This
will help to better understand the benefits and limitations of Agile and Lean practices.
The following research questions were developed to achieve our aim:
RQ1: What are the sets of primary studies that the secondary studies draw upon?
RQ1.1: What is the overlap between the sets of primary studies?
RQ1.2: What are the reasons for overlap and differences?
RQ2: What are the scopes of the secondary studies in evaluating Agile and Lean
software development?
RQ2.1: Which Agile and Lean practices were studied?
RQ2.2: What are the impacts of the Agile and Lean practices that were evalu-
ated?
RQ3: What is known about the impacts of Agile and Lean practices?
RQ3.1: What is the level of empirical evidence of the evaluations presented in
the secondary studies?
RQ3.2: What is the level of agreement between the secondary studies?
RQ3.3: How does the quality of the secondary studies influence the reported
impacts of Agile and Lean practices?
A tertiary study has the same steps as an SLR [110]. In conducting this tertiary re-
view, we adopted the systematic review guidelines by Kitchenham and Charters [110].
Figure 4.1 depicts the steps performed in our tertiary study.
4.3.1 Search Process: Step 1

The first step (Step 1 in Figure 4.1) in our tertiary study was to search for relevant
secondary studies. In our search, we formulated search strings that allowed us to re-
trieve secondary studies in Agile and Lean development from literature databases. The
search string was composed of three main substrings: Agile methods, Lean methods,
and secondary study. We included Lean methods in the search string because Agile and
Lean methods have similar goals of focusing on customers and quickly responding to
their needs. Agile and Lean methods also share similar values, principles and practices,
and are often considered as similar software development methods [164]. Poppendieck
and Poppendieck also mention that Lean is the extension of Agile’s theoretical foun-
dation [169] . Table 4.1 shows the substrings and the keywords. Surrounding quotes
Steps Excluded Papers Included Papers
1 Search Process 1110
Title & Abstract • 337 duplicates 729

2a
Screening • 44 editorials
• 581 non-SE 122

2b • 26 non secondary
studies + non-Agile/Lean
Included for
Full Text • 81 removed based on
3a 41 general overview
Screening Inclusion/Exclusion
(Section 4.4)
Criteria
Post-hoc validation
• 28 removed based on Included for

3b Post-hoc validation 13 detailed analysis
relevance assessment
(Section 4.5)
Figure 4.1: Steps in the Tertiary Study
refer to exact matches of a phrase. For single terms, the quotes are redundant and have
therefore been omitted.
Table 4.1: Keywords in Search String
Substring Keywords
Substring 1 Agile agile OR scrum OR crystal OR XP OR “extreme programming” OR
DSDM OR “dynamic systems development method” OR FDD OR “fea-
ture driven development”
Substring 2 Lean lean* OR kanban OR kaizen OR “value stream mapping” OR VSM
OR “continuous improvement” OR “cross* functional” OR “concurrent
engineering” OR “integrated product development”
Substring 3 Secondary study “review of studies” OR “structured review” OR “systematic review” OR
“literature review” OR “literature analysis” OR “in-depth survey” OR
“literature survey” OR “meta-analysis” OR “Past studies” OR “system-
atic map*” OR “secondary study”
We performed a pilot search on the databases Inspec and Compendex to check

if adding or removing a keyword would lead to more or fewer relevant papers. To
formulate the actual search string for Substring 1 and 2, we adapted the keywords that
were used in Dybå and Dingsøyr [58] and Pernstål et al. [163]. We also consulted our
search string with Dr. Kai Petersen1 , who was external to the review team, to validate
the keywords included in our search string. For Substring 3, we referred to Kitchenham
et al. [114] to ensure that we included all necessary keywords and synonyms regarding
secondary studies. We performed searches on abstract, title, and keywords with the
following search string: (Substring 1 OR Substring 2) AND Substring 3.
The search string was then implemented in five electronic databases: (1) Com-
pendex & Scopus, (2) Inspec, (3) IEEE Explore, (4) ACM Digital library, and (5) ISI
Web of Knowledge. The search was done in December 2014 to identify peer-reviewed
secondary studies published since 1990. The search time span was limited to 1990 –
2014 to minimize the number of irrelevant papers from manufacturing, where Agile
and Lean concepts were derived from. Table 4.2 shows the number of papers retrieved
from each database.
Table 4.2: Database Search
Database Years Searched Total Hits

Compendex & Scopus 1990 – 2014 355
Inspec 1990 – 2014 414
IEEE Explore 1990 – 2014 130
ACM 1990 – 2014 20
ISI Web of Knowlege 1990 – 2014 191
Total 1110
4.3.2 Study Selection: Step 2 - 3

The following subsections describe the study selection process.
Title and Abstract Screening: Step 2a – 2b

In study selection, we performed title and abstract screening in two sub-phases. In the
first sub-phase of title and abstract screening (Step 2a), we checked for duplicates, as
well as non-article papers, e.g., editorials, prefaces, or article summaries. After re-
moving duplicates and non-article papers, we were left with 729 papers. In the second
sub-phase of title and abstract screening (Step 2b), we applied an inclusive strategy
to avoid premature exclusion of relevant papers. We only excluded papers that were
clearly not related to software engineering, no secondary studies, or not in Agile and
1 Dr Kai Petersen is one of the 18 most active researchers publishing in Agile software development, as
reported in [39]
Lean software development. At this stage only the first author was involved. When in
doubt a paper was included. After title and abstract screening, we were left with 122
papers.
Full text Screening: Step 3a – 3b

For full text screening, we formulated inclusion/exclusion criteria that we adapted from
Kitchenham et al. [113]. The purpose of the inclusion criteria was to ensure that we
included all peer-reviewed secondary studies in Agile and Lean software development.
Articles were included if they were:
• IC1. Systematic reviews, systematic mappings, meta-analyses or literature re-
views in Agile or Lean (Type 1) OR secondary studies in Agile and Lean that
were part of empirical studies (Type 2).
• IC2. Secondary studies (either Type 1 or Type 2) with defined literature review
processes. This criterion is adopted from Kitchenham et al. [114].
• IC3. Secondary studies that were peer reviewed.
• IC4. Secondary studies that have a list of included primary studies.
The purpose of the exclusion criteria was to ensure that we only included unique
secondary studies with traceable review processes and traceable secondary studies. Ar-
ticles were excluded if they were:
• EC1. Type 2 papers, where the results of the literature studies were not distin-
guishable from the empirical study results.
• EC2. Posters and short papers with less than four pages.
• EC3. Redundant reports of the same secondary study.
In the first sub-phase of full text screening (Step 3a), we screened the papers against
the inclusion/exclusion criteria. In this stage, we checked the papers for redundancies.
In such cases we included the most comprehensive and up-to-date results. For instance,
we excluded Jalali and Wohlin [99] from our review and included Jalali and Wohlin
[100] instead. We found that the journal version [100] is an extension of the work from
the mapping study [99] and also covers more aspects than the mapping study itself.
At this stage, the first author reviewed all papers and implemented all inclusion and
exclusion criteria. A post-hoc validation was then performed by the second and third
authors.
Post-hoc Validation. A post-hoc validation was performed to ensure that the first
author did not exclude papers that should have been included. The second and third
authors reviewed two thirds of the 122 papers, where neither authors had overlapping
papers. As part of the post-hoc validation, we calculated the Kappa coefficient to evalu-
ate the extent of agreements. The first and second author had a kappa coefficient 0.802,
the first and third author had a Kappa coefficient of 0.323. A Kappa coefficient be-
tween 0.61 – 0.80 is considered as a substantial agreement [124]. Meanwhile, a Kappa

coefficient between 0.21 – 0.40 is considered a fair agreement (ibid.).
We are aware that a Kappa coefficient could be due to chance [82]. Therefore we
examined all papers in detail that were included or excluded by the first author but were
included or excluded by the second or third author in a face-to-face meeting. We found
that disagreements occurred primarily because the first author was more inclusive than
the third author. We did not find papers that were included by the second and third
authors but were excluded by the first author. To avoid the exclusion of potentially
relevant studies, we continued with the inclusive strategy and kept all papers that were
included by the first author. After the first phase of full text screening (Step 3a), 41
papers were left.
In the second phase of full-text screening, we performed a relevance analysis of the
secondary studies (Step 3b). In relevance analysis, the first author sought for secondary
studies that provide reports, summaries, or statements regarding the impacts of Agile
and Lean practices on project constraints. Thirteen out of 41 secondary studies reported
an impact of Agile and Lean practices on project constraints.
Post-hoc Validation. A post-hoc validation was then performed by the second and
third authors to ensure that no secondary studies that should have been included were
excluded. A discussion was held to discuss disagreements between the first author and
the second and third authors. Similar to Step 3a, we found that the first author was
more inclusive than the second and third authors. The second and third authors did not
include any papers that were excluded by the first author. We, therefore, decided to
include all papers that were included by the first author for data extraction.
4.3.3 Data Extraction

We extracted the following information from the secondary studies:
1. Meta-data. Title and publication year.
2. Review method. The adopted literature review method and the reference to the
methods as mentioned in the secondary study.
3. Aim of study. The aim(s) as stated in the secondary study.
4. Topic of the review. The main focus of the study, whether it is Agile or Lean in
general, or specific Agile or Lean practices, or a combination of Agile or Lean
with other topics in software engineering.
5. Practices and their associated impacts. The impacts associated with a particular
Agile and Lean practice. To ensure uniformity of interpreting Agile and Lean
practices, we referred to the list of practices discussed in [164].
6. Empirical support. The number of primary studies that were cited with respect to
impacts and practice association. We also collected the research methodologies
that were used in the primary studies
7. List of primary studies. The primary studies that were included in each sec-
ondary study.
We extracted items 1–4 from the 41 secondary studies that passed the full text
screening (Step 3a). The data extracted from the 41 secondary studies are used to
provide an overview of the secondary studies in Section 4.4. The 13 secondary studies
that passed the relevance analysis and provided items 1–7 (Step 3b) are used in the
detailed analysis in Section 4.5. The first author performed data extraction for all 41
papers. A post-hoc validation was performed to check for consistency of the data that
were extracted by the first author. The second and third authors performed independent
data extractions on ten papers. The data extracted by the first author were then checked
and compared with the data extracted by the second and third authors. From the post-
hoc validation, we found that data extracted by the first author were also extracted by
the second and third author. Therefore, we agreed to use the data that were extracted
by the first author.
4.3.4 Quality Assessment Criteria

Quality assessment is critical because the effectiveness of an intervention might be
masked by secondary studies that were not properly conducted [34]. For the quality
assessment, we adopted the appraisal criteria from the Center for Reviews and Dis-
semination’s (CRD) guide for reviews [34] and adjusted them to the context of this
study. The quality assessment criteria are summarized in Table 4.3.
Table 4.3: Quality Assessment Score
ID Criteria* Score**
C1 Was the review question clearly defined Y: PICOS are used and RQs are clearly defined.
in terms of population, interventions, P: PICOS are not used but RQs are clearly defined.
comparators, outcomes and study de- N: RQs are not clearly defined.
signs (PICOS)?
C2 Was the search strategy adequate and Y: The authors have either (searched 4 or more digital libraries and
appropriate?*** included additional search strategies) or (identified and referenced all
journals addressing the topic of interest).
P: The authors have searched 3 or 4 digital libraries with no extra search
strategies or they searched a defined but restricted set of journals and
conference proceedings.
N: The authors have searched up to 2 digital libraries or an extremely
restricted set of journals.


*
ID Criteria Score**
C3 Are the review’s inclusion and exclusion Y: The inclusion/exclusion criteria are explicitly defined in the study.
criteria described and appropriate? P: The inclusion/exclusion criteria are implicitly defined.
N: The inclusion/exclusion criteria are not defined and cannot be readily
inferred.
C4 Were appropriate criteria used to assess Y: The authors have explicitly defined quality criteria, quality scoring
the quality of the primary studies?*** procedures, and extracted them from each primary study. Information
about the quality scores for the primary studies is provided (e.g., in a
table or a distribution).
P: The research questions involve study quality issues that are addressed
by the study. The quality criteria are listed. The quality scoring proce-
dure is described. The scores for individual primary studies are not
documented.
N: There is no explicit quality assessment of individual primary studies
or the quality assessment has been described insufficiently.
C5 Were preventive steps taken to minimize Y: Measures for inter-rater agreement are provided, e.g., Kappa-
bias and errors? coefficient(s), or a test-retest (in case of a single reviewer)
P: The authors mention preventive actions, e.g., piloting of the review
steps.
N: Nothing was mentioned pertaining to preventing bias.
C6 Were adequate details presented for each Y: Information is presented about each paper, so that the data summaries
of the primary studies?*** can clearly be traced to relevant papers. E.g., we can trace the research
methods, information about study context, or collected measures to in-
dividual primary studies.
P: Only summary information is presented about individual papers.
E.g., papers are grouped into categories, but it is not possible to link
individual studies to each category.
N: Information/results cannot be traced to individual primary studies.
C7 Was the evidence synthesized and aggre-
gated?
C7a Was the evidence actually synthesized Y: The synthesis pools the studies in a meaningful and appropriate way.
and aggregated, or merely summarized? Differences between studies are addressed.
P: A partial synthesis/aggregation of evidence is provided.
N: There is no real synthesis. The evidences from the individual studies
are basically only repeated/summarized.
C7b Were the strengths of evidence of indi- Y: Yes, to some extent.
vidual studies taken into account in the N: No.
synthesis?
C8 Do the authors’ conclusions accurately Y: The conclusions are clearly supported by the provided data/results.
reflect the evidence that was reviewed?
N: The conclusions are not well supported by the provided data/results.
*
Criteria C1–C8 are adopted from the [34].
**
Y(es)=1; P(artially)=0.5; N(o)=0.
***
Details adopted from [114].
The quality assessment was performed in parallel with data extraction. We per-
formed quality assessments on the 13 secondary studies that were included for detailed
evaluation, because we were interested to examine if the quality of the secondary stud-
ies influence the results. We did not exclude any secondary studies based on their
quality score.
The first author performed the quality assessment on 13 secondary studies. The
second and third author then performed an independent quality assessment of the 13
secondary studies. After the post-hoc validation, we discussed and compared the qual-
ity assessments and found only minor deviations in the overall quality scores. From
the discussion, we agreed to use the quality assessment that was performed by the first
author.
4.3.5 Mapping Scheme

Due to the heterogeneity of the data in this tertiary study, we were unable to perform a
qualitative meta-study, as described by Paterson et al. [161]. In this study, we decided
to aggregate the results using a mapping. In this tertiary study, the secondary studies
are clustered based on their quality scores, see Section 4.5.3. In each quality score
cluster, we mapped impacts of an Agile and Lean practice on a project constraint.
Each Agile and Lean practice can have a positive, no impact, or a negative impact on a
project constraint. We then reported the corresponding number of primary studies, that
support a positive, no impact, and a negative impact of an Agile practice on a project
constraint. See Sections 4.5.4, 4.5.4, and 4.5.4.

The following are validity threats that are relevant in our tertiary study.
Missing relevant secondary studies [114]: To mitigate this issue we used a broad
search string. Instead of using the individual practices as keywords, we used the Agile
and Lean methods in our search. We also omitted “software engineering” or “software
development” in our search string to enable broader hits. We also adopted a more in-
clusive approach in the study selection. In the title and abstract screening, we included
all studies that are on Agile and Lean methods, or secondary studies in software engi-
neering. In the full text screening, discussions were held to examine the disagreements
between all three authors. From the discussion, we found that the first author was more
inclusive than the other authors. Therefore we agreed to include all papers that were
included by the first author, because more papers would not have been included by the
second and third authors.
Missing relevant data from the secondary studies [114]: The first author was the
primary reviewer of this tertiary study and thus there was a potential to introduce bias.
This issue was mitigated by post-hoc validations performed by the second and third
authors for the full text screening, data extraction, and quality assessment. The post-
hoc validation process revealed that the second and third author extracted the same data
as the first author. The second and third authors did not find more data or different data
to be extracted.
Publication Bias [110]: Studies with positive results are more likely to be published
than those with negative results. This issue did not seem to be a problem as we found
secondary studies that reported negative outcomes. There is a risk that the collected
secondary studies are skewed towards particular practices. We mitigated this issue by
using the Agile and Lean methods, instead of practices, in our search keywords.
Post-hoc Validation: In other secondary or tertiary studies, usually the included
papers are reviewed by at least two reviewers in parallel. In this tertiary study, a post-
hoc validation was performed instead. The purpose of the post-hoc validation is to
ensure that the main author i.e., the first author, performed a repeatable process of
inclusion/exclusion, data extraction, and quality assessment. Such approach is men-
tioned in [25] and shown to be helpful to speed up the review process. Performing a
post-hoc validation instead of parallel review does not have significant impact on the
validity of the study. First, because the results of the study selection performed by
the first author were not shared with the second and third author until they completed
the post-hoc validation review. Second, all papers were reviewed by at least two inde-
pendent reviewers; the first author who is a junior researcher, and the second or third
authors who are senior researchers. Third, disagreements that arose from the post-hoc
validation were resolved through face-to-face discussions. Also, the fact that no papers
were reviewed by two senior researchers also did not influence the validity of the study,
because as shown in the full text screening step, the first author was more inclusive than
the second or third author.
Quality Assessment Tool: In this tertiary study, we identified a number of simplified
versions of an SLR [110], they are, semi-SLR [136] and quasi-SLR [135] (definitions
of semi-SLR and quasi-SLR are provided in Section 4.4.1. The authors of these sec-
ondary studies provided clear descriptions of the rationale for not performing an SLR
and what they have done differently. Despite the clarity of process description pro-
vided by the authors, these studies did not obtain high scores in the quality assessment,
because they missed a number of steps and therefore obtain low scores for each quality
criteria. However, this does not necessarily mean that these secondary studies are bad
studies. This is a possible limitation in our quality assessment tool. The quality as-
sessment tool that we adapted from [34] and [114] are tailored for a ”complete” SLR.
However, not all secondary studies are ”complete” SLR. Furthermore, our quality as-
sessment tool is only applicable to assess the report of the secondary studies, and not
the actual review process. However, currently, there is no other tool that allows us to
assess the actual review process. To minimize the threat to validity due to the limitation
of the quality assessment, we did not use the quality scores to exclude the secondary
studies.
Reliability of the results. Reliability of the results. Wohlin et al. discuss reliability
issues regarding secondary studies [240]. The outcomes of such studies might depend
on the contexts of the reviews, e.g., reviewer experience, review design and protocol,
and data available from the secondary/primary studies. However, independent studies
may still come to the same general conclusions even if the papers found are differ-
ent [240]. In this tertiary study, all authors discussed, formulated and agreed upon
the review design and protocol and executed study selection and data extraction ac-
cordingly. In our study, the two senior researchers did not review overlapping papers,
which might have increased the validity. However, since the same review design and
protocol were used by all authors, we believe that reviewing overlapping papers would
not have changed the outcome of our study.
4.4 Overview
The results of this tertiary study are divided into two parts. The first part, which we
present in this section, is an overview of the secondary studies conducted in Agile and
Lean software development. The second part, which we present in Section 4.5 is the
evaluation of Agile and Lean practices and their impacts on project constraints, as well
as their empirical support.
4.4.1 Occurrences in Agile and Lean Methods Secondary Studies

The first part of our analysis is to examine the state-of-art in Agile and Lean secondary
studies. The list of included secondary studies is summarized in Table 4.4.
Table 4.4: List of Secondary Studies
ID Citation ID Citation ID Citation

S1 Ahmad et al. [4] S15 Al-Baik and Miller [5] S29 Racheva et al. [174]
S2 Causevic et al. [33] S16 Chagas et al. [35] S30 Salah et al. [186]
S3 da Silva et al. [50] S17 Silva da Silva et al. [205] S31 Salvador et al. [189]
S4 Dybå and Dingsøyr [58] S18 Diaz et al. [55] S32 Senapathi and Srinivasan [195]
S5 Gandomani et al. [78] S19 Diebold and Dahlem [56] S33 Silva et al. [204]
S6 Magdaleno et al. [135] S20 Hansson et al. [85] S34 Sletholt et al. [210]
S7 Munir et al. [141] S21 Hebig and Bendraou [88] S35 Stavru [213]
S8 Mäntylä et al. [136] S22 Hossain et al. [90] S36 Usman et al. [225]
S9 Pernstål et al. [163] S23 Jalali and Wohlin [100] S37 Yanzer Cabral et al. [241]
S10 Sfetsos and Stamelos [198] S24 Jurca et al. [104] S38 Rohunen et al. [184]
S11 Rafique and Mišić [175] S25 Kupiainen et al. [122] S39 Stavru et al. [214]
S12 Shen et al. [201] S26 Mitchell and Seaman [140] S40 Stavru et al. [215]
S13 Zani et al. [243] S27 Nguyen-Cong and Tran-Cao [148] S41 Kasoju et al. [107]
S14 Abrantes and Travassos [2] S28 Oliveira Albuquerque et al. [155]
4.4 Overview 105
In total, we found 41 secondary studies: 31 SLRs, four Systematic Mappings

(SMs), one literature review, and five secondary studies that were reported as part of an
empirical study (Type 2). Figure 4.2 shows the trend of published secondary studies.
12
10
SLR
6 LR
SM
SLR+Empirical
4
0
2008 2009 2010 2011 2012 2013 2014 2015
Figure 4.2: Number of Secondary Studies performed according to Year
We did not find any secondary studies that fulfilled our inclusion criteria prior to
2008. The number of secondary studies gradually increased between 2008 and 2012.
There was a slight decrease in 2013, but then it increased significantly in 2014. We
included a study from 2015 since this paper was published online in 2014, but was
assigned to a journal issue in 2015.
The increasing trend of secondary study in 2008 onward was attributed to the in-
troduction of EBSE by Kitchenham et al. [112]. The low number of secondary studies
prior to 2008 could be attributed to the low number of primary studies, as the are was
still growing in early 2000. A meaningful secondary study is difficult to be performed
with low number of primary studies.
Guidelines Used in the Secondary Studies

We found different methods being employed in the secondary studies. We also found
that different guidelines are used in the secondary studies. Table 4.5 summarizes the
methods and guidelines mentioned in the secondary studies. Some papers are men-
tioned more than once in the table, and mapped into different methods and guidelines.
This was done because more than one method and/or guideline are mentioned in their
secondary studies.
Table 4.5: Guidelines Used in Secondary Studies
Secondary Study Method Guideline to the Method Adopted in Total

SLR Kitchenham and Charters [110] S1, S2, S4, S5, S7, S10, S11, S12, S13, 30
S15, S16, S17, S18, S20, S21, S22, S23,
S25, S26, S27, S28, S31, S32, S33, S34,
S36, S37, S38, S40, S41
Biolchini et al. [21] S17 1
Pai et al. [159] S14 1
Unclear S30, S39 2
Semi-SLR Kitchenham and Charters [110] S8 1
Quasi-SLR Kitchenham and Charters [110] S6 1
SM Kitchenham and Charters [110] S9, S19 2
Petersen et al. [168] S3 1
Unclear S24 1
LR Unclear S35 1
Most of the secondary studies adopted the SLR guidelines by Kitchenham and
Charters [110]. One study (S17) adopted the SLR guideline by Biolchini et al. [21].
The authors of S14 adopted an SLR guidelines from medicine, i.e., Pai et al. [159]. Four
secondary studies did not mention which guidelines were used. We also found modified
or simplified versions of Kitchenham and Charters’s SLR guidelines being used in the
secondary studies, they were, quasi-SLR (S6), and semi-SLR (S8). A quasi-SLR is
a term introduced by Travassos et al. [220] to define a secondary study that follows
has the same rigor and formalism of an SLR, but no meta-analysis in principle can be
applied. The authors of S8 refer to their secondary study as semi-SLR because it does
not follow the same rigour as an SLR, such as, having only one reviewer, no rigorous
quality assessment of the study, and no data synthesis.
We also found a number of SMs in Agile and Lean software development. When
we examined the year of publications, SMs were gaining popularity as a method for
studying Agile and Lean software development literature since 2011.
Most secondary studies follow certain guidelines. This finding suggests the in-
creasing need to gather evidences pertaining to Agile and Lean software development
in more systematic ways. We also found two secondary studies with modified or sim-
plified the SLR steps defined by Kitchenham and Charters [110]. The finding indicates
the possibility of performing less rigorous secondary study but remain traceable and
systematic.
4.4 Overview 107
Aims of Secondary Studies
We identified four types of secondary studies’ aims:

• Classify Primary Studies. Secondary studies in this category reviewed the pri-
mary studies and classified them into different categories. All SM studies fall
into this category.
• Find association. Secondary studies in this category reviewed the primary studies
and examined the association between Agile or Lean practices and their impacts
on different aspects, such as quality, schedule, etc.
• Identify Factors. Secondary studies in this category reviewed the primary stud-
ies to identify various factors, like challenges, benefits, limitations, or certain
practices.
• Survey the literature. Secondary studies in this category reviewed the primary
studies to identify trends and extents of Agile and Lean software development
research.
In categorizing the papers, we looked at the stated aim and how the results are
presented. Table 4.6 summarizes the aims of secondary studies. The detailed aim from
each secondary study is compiled in the Appendix of this chapter, in Table A1.
Table 4.6: Aims of Secondary Studies
Aim References
Identify Factors S4, S5, S6, S8, S12, S13, S14, S15, S16, S17, S18, S20, S22, S23, S28,
S29, S30, S32, S33, S35, S37, S38, S39
Survey the literature S1, S25, S27, S31, S34, S36, S40, S41
Classify primary studies S3, S9, S19, S21, S24
Find Association S2, S7, S10, S11, S26
Most secondary studies are aimed to identify factors pertaining to Agile and Lean
methods, such as commonly used practices, challenges of implementing Agile meth-
ods, etc. When we examined the type of secondary study (see Section 4.4.1), most of
the secondary studies in this category mentioned, in the title or indicated in the text, to
have performed an SLR, except S6, S8, and S35.
We also found a number of secondary studies that are aimed to survey the litera-
ture. All secondary studies in this category also mention having performed an SLR.
Five secondary studies tt are aimed to classify primary studies. All secondary studies
in this category, except S21, mention having performed a systematic mapping study.
S21 on the other hand mention having performed an SLR. There were very few sec-
ondary studies that were aimed find associations between Agile and Lean methods or
practices and their impacts on cost, quality, productivity, etc. Three secondary studies
in find association (S2, S7, S11) are focused on TDD.
Topics in Agile Secondary Studies
We found a variety of topics in Agile and Lean secondary studies. Combinations of Ag-
ile and Lean software development with other areas in software engineering, such as
Software Product Line Engineering (SPLE) and Global Software Development (GSD),
dominate the secondary studies’ topics. General discussions on Agile and Lean soft-
ware development is also a popular topic.
Table 4.7 summarizes the topics covered by the secondary studies. It shows that
Agile and Lean practices are implemented in a wide range of domains, such as em-
bedded systems, scientific software, and regulated domains. We also found secondary
studies that focus on specific Agile and Lean practices, such as TDD and frequent re-
leases. The variation of secondary topics further reinforces the prevalence of the use of
Agile or Lean methods and practices.
Table 4.7: Topics of Secondary Studies
Topic References Total

Combining Agile/Lean with other topics in SE, e.g., User S3, S13, S17, S18, S20, S21, S22, 16
Centered Design (UCD), SPLE, GSD, Metrics, Model S23, S24, S25, S27, S30, S31, S36,
Driven Development (MDD), Cloud S39, S40
Agile/Lean General S4, S14, S19, S32, S35, S38 6
Application of Agile/Lean in specific domains e.g., Em- S9, S12 S28, S34, S41 5
bedded Systems, Scientific Software, Large Scale Soft-
ware Systems, Automotive
Test Driven Development (TDD) S2, S7, S11 3
Comparing Agile with other development process mod- S5, S6, S26 3
els, e.g., Open Source Software (OSS), Plan driven)
Lean - Kanban S1, S15 2
Agile and Management (Project Management, Knowl- S16, S37 2
edge Management)
Agile and Value Creation and Quality S10, S29 2
Agile and CMMI S33 1
Frequent Releases S8 1

This section presents the detailed analysis of the impacts of Agile and Lean practices
on project constraints.
4.5.1 Population of Primary Studies

We examined the primary studies from the 13 secondary studies. We found 50 over-
lapping primary studies out of 420 primary studies included in 13 secondary studies.
However, the 50 overlapping primary studies are not cited by all 13 secondary studies.
Figure 4.3 shows the overlap across the secondary studies. The number in the brackets
represents the total number of primary studies from each secondary study.
Lean Cluster Miscellaneous Cluster

S8 (24)
S5 (23)
S1 (19) 1 S12 (40)

4
1
7
S6 (42) 1
TDD Cluster
S9 (38) S11 (25) S10 (46)
1 5 S4 (36)
Agile/SPLE Cluster
S3 (32) 2 2
10 6 5
3 21
S7 (41)
S2 (48)
S13 (6)
Figure 4.3: Overlap of Primary Studies Across Secondary Studies
From Figure 4.3, we can see four clusters of papers (rounded boxes). Lean Clus-
ter contains papers focusing on Lean methods (S1, S9) and frequent deliveries (S8).
Agile/SPLE Cluster contains papers on the combination of Agile and software prod-
uct line engineering (S3, S13). The secondary studies in Miscellaneous Cluster has
a variety of topics, including Agile and Lean software development in general (S4),
Agile methods in the context of open source (S5, S6), and implementation of Agile
in embedded systems development (S12). Lastly, TDD Cluster contains three papers
focused on TDD (S2, S7, S11) and a paper focused on Agile methods in general (S10).
Although the secondary studies in Lean Cluster shared the same topic, they have
little overlap in primary studies. The same pattern is also observed in Agile/SPLE
Cluster. The small overlap of primary study could be attributed by the difference of
focus among the secondary studies in each cluster. For instance in Lean Cluster, S1
is focused on Kanban, and S9 has a broader focus on Lean in general. In Agile/SPLE
Cluster, S3 focuses on combining Agile methods and SPLE, and S13 focuses on the
use of reference architectures in Agile software development.
In Miscellaneous Cluster, there are three secondary studies (S6, S10, S12) that
have overlaps with S4, because S4 has a general focus on the implementation of Agile
and Lean software development. S5 and S6 share four overlapping primary studies
because they share the topic of combining Agile methods and open source.
In TDD Cluster, we identified many overlaps among the secondary studies (S2,
S7, S10, S11). S2, S7, S10, and S11 have six overlapping primary studies. Three pa-
pers that focus on TDD (S2, S7, S11) have ten overlapping primary studies. The largest
overlaps are between S2 and S7, with 21 overlapping primary studies. Given the sim-
ilarity of focus, we expected more overlap among S2 and S7. However, if we looked
closer into S2 and S7, S7 is focused on the impact of TDD on quality and productivity,
and S2 is focused on the impact of TDD in general. We did not expect large overlaps
between S2 and S7 on one hand and S11 on the other because S11 only included exper-
iments in TDD. The large number of empirical studies conducted in TDD shows that
TDD has been well researched. The lack of overlap suggested that researchers in TDD
had separate agendas in examining the impact of TDD. Also, differences in secondary
study’s review design/protocol could also influence the number of overlapping primary
studies.
We also found primary studies that are cited in three to four secondary studies. In
Miscellaneous Cluster, S4, S6, S12 share one primary study, and in TDD Cluster, S2,
S7, S10, S11 share six primary studies. These primary studies could be considered as
key papers in Agile and Lean software development. The list of key primary studies is
presented in Table 4.8.
Table 4.8: List of Most Cited Papers from the Secondary Studies
Papers Cited In
D. Karlström, P. Runeson, Combining agile methods with stage-gate project management, S4, S6, S12
IEEE Software 22(3) (2005), pp. 43 – 49.
N. Nagappan, E.M. Maximilien, T. Bhat, L. Williams, Realizing quality improvement S2, S7, S10, S11
through test driven development: results and experiences of four industrial teams, Empir.
Softw. Eng. 13 (3) (2008), pp. 289 – 302.
L. Williams, E. Maximilien, M. Vouk, Test-driven development as a defect-reduction prac- S2, S7, S10, S11
tice, in: 14th International Symposium on Software Reliability Engineering (ISSRE 2003),
pp. 34 – 45
L. Huang, M. Holcombe, Empirical investigation towards the effectiveness of test first pro- S2, S7, S10, S11
gramming, Inform. Softw. Technol. 51 (1) (2009), pp. 182 – 194.
H. Erdogmus, M. Morisio, M. Torchiano, On the effectiveness of the test-first approach to S2, S7, S10, S11
programming, IEEE Trans. Softw. Eng. 31 (3) (2005), pp. 226 – 237.
M. Muller, O. Hagner, Experiment about test-first programming, IEE Proc. Softw. 149 (5) S2, S7, S10, S11
(2002), pp. 131 – 136.
A. Gupta and P. Jalote, An Experimental Evaluation of the Effectiveness and Efficiency of S2, S7, S10, S11
the Test Driven Development, Proc. First Int’l Symp. Empirical Software Eng. and Measure-
ment, pp. 285 – 294, 2007.
4.5.2 Scopes of Agile and Lean Secondary Studies

We used the list of Agile and Lean practices from [164] to see which Agile and Lean
practices had been examined and which had not. The list from [164] was selected
as it covers both Agile and Lean practices. To identify the practices, we looked for
the description of the practice in the secondary studies and compared them against
the description provided by Petersen [164]. We also used the definitions presented in
Williams [239] as a cross reference to help with identifying the practices.
We collected the impacts of Agile and Lean practices bottom-up from the included
secondary studies. Each impact was then mapped to the following project constraints as
defined in Project Management Body of Knowledge (PMBOK): scope, quality, sched-
ule, budget, resources, communication, and risk [171]. The mapping of studies to
practices and project constraints is summarized in Table 4.9.
Agile and Lean Practices
From the list of practices described by Petersen [164], we identified 13 out of 26 prac-
tices from the secondary studies. TDD is mentioned in 10 secondary studies. Pair pro-
gramming and planning game are mentioned in two secondary studies. The remain-
ing practices, on-site customer, refactoring, incremental deliveries, internal/external
release, short iteration adaptive planning, stand up meeting, team chooses own task,
Table 4.9: Practices and Project Constraints Mapping
Project Constraints
Practice Scope Quality Schedule Budget Resources Risks Communication Sum
Onsite customer S4 1
Metaphores and user stories 0
Refactoring S10 S10 1
Coding standards 0
Team code-ownership 0
Low dependency architec- 0
ture
TDD & test automation S12 S2, S3, S4, S5, S6, S2, S7, S10, S2, S10 S2, S7, S10 10
S7, S10, S11, S12, S11, S12
S13
Pair programming S4, S10 S4, S10 S10 S10 2
Continuous integrations 0
Inspection 0
Configuration Management 0
Incremental deliveries S8 S8 S8 S8 1
Internal/external release S9 1
Short iteration S4 1
Adaptive planning with S1 S1 S1 1
highest priority user stories
(Adaptive planning)
Time-boxing 0
Planning game S4, S10 S4 2
Co-located development 0
Whole team (cross- 0
functional team)
40h week 0
Stand up meeting S4 S4 1
Team chooses own task S1 1
VSM S9 S9 S9 1
Inventory management 0
Chief Engineer 0
Kanban pull systems S1 S1 1
Sum 1 13 8 2 4 1 4
Value Stream Mapping (VSM), and Kanban pull systems, is mentioned in one sec-
ondary study.
Often we found different terms being used in the secondary studies when referring
to a practice. Some of the secondary studies use the same terms as in [164], which are,
on-site customer, refactoring, pair programming, planning game, and VSM.
The secondary studies that discuss the practice of creating test cases prior to coding
is mapped into TDD & test automation. We also identified synonyms to TDD, such as
test-first programming (S2, S4) and test-first development (S11).
Rapid releases (S8) is mapped into incremental deliveries, as it involves releasing
new or updated versions of software to the customer frequently [164]. Rapid releases
does not fit into the definition of short iteration as it entails performing development
activities in short periods of time, where a feature was delivered to obtain feedback
from the customers. The definitions provided for short iterations, both in [164] and
[239], do not entail releasing a version. However short iterations fits the definition of
a practice that involves delivering in-progress-software to obtain continuous feedback
mentioned in S1.
The secondary study mapped into internal/external releases (S9) discusses a prac-
tice of dividing development task into major and minor enhancement. The secondary
study mapped into stand-up meeting (S4) discusses a Scrum practice of daily meetings
that involves different stakeholders.
S1 mentions two Kanban practices, visualization and prioritization of tasks. S1
does not provide a description of visualization. We assumed they are a visual means
for tracking and prioritizing tasks, as in a Kanban board. Meanwhile, S1 defines prior-
itization as a practice that helps in deciding which work item needs to be pulled next
to optimize values. Although S1 separated the two practices apart, they both fit the
definition for adaptive planning with highest priority user stories. To shorten the term,
we will refer adaptive planning with highest priority user stories as adaptive planning
in the remainder of this chapter. S1 discusses a Kanban practice where team members
are allowed to decide their own task management and prioritization, without upper
level management approval. This practice is mapped under team chooses own task. S1
also mentions a Kanban practice that limits the work in progress by pulling the next
work item according to the team’s capacity. We mapped this practice into Kanban pull
systems.
Impacts of Agile and Lean Practices

From Table 4.9 we can see the impact of Agile and Lean practices on quality is reported
in 13 secondary studies. Schedule is reported in eight secondary studies. Resource
and communication are reported in four secondary studies. Budget is reported in two
secondary studies. Meanwhile scope and risks are reported in one secondary study.
We referred to available guidelines and standards, like PMBOK [171] and ISO/IEC
25010 [96] to map the impacts of Agile and Lean practices on the relevant project
constraints. We mapped fulfilling requirements (S12) into scope because activities in
scope management include managing requirements [171]. We mapped productivity
(S7, S10, S11) into schedule because productivity is often measured with respect to
time. Meanwhile, effort (S9) and estimating work size (S4, S10) are mapped into
resources, as determining effort and estimating work level is part of the human resource
management knowledge area in PMBOK. We found different variables and measures
of quality. In mapping quality measures and variables, we used ISO/IEC 25010 [96] as
a reference to map the secondary studies. The variables that we found on quality are
all related to software product quality, e.g., number of defects, size, etc. For simplicity,
we refer to software product quality as quality in this chapter. Table 4.10 summarizes
the mapping of different variables and measures and project constraints.
We found that number of defects and defect density are commonly used to measure
external quality. However, we found different variations of measures to indicate time,
e.g., lead time, testing time, time between two subsequent releases, etc. We also found
measures with different levels of granularity being used to measure internal quality,
e.g., some secondary study mention a generic measure, like code quality, some spec-
ified more particular measures, like complexity, size, etc. Many secondary studies do
not provide information on measures under investigation.
Table 4.10: Mapping of Variables and Measures to Project Constraints
Project Con- Variables Measures Reference

straints
Scope Requirements Not provided S12
Quality External Quality number of defects/bugs/ trouble reports, defects per S11
KLOC/defect density, number/percentage of acceptance/
black-box/ external tests passed, number/percentage of
unit tests passed, quality mark given by client
Defect count, defect density, mean time assertion per S7
method, total assertions (passed/failed), number of test
cases passed
number of passed acceptance tests or the total number S10
of defects or number of defects/KLOC (defect density).
number of defects found before release or defects reported
by customers
Number of defects S3, S8, S9
Robustness S7
Customer satisfaction Not provided S1, S4
Internal Quality Condition coverage, Branch coverage, Nested block S7
depth, Cyclomatic complexity, Number of parameters,
Coupling between objects, Information flow, Weighted
class per method, Lack of cohesion metrics, Mutation
score indicator
Complexity, size S2

Project Con- Variables Measures Reference
straints
Code size, Cyclomatic complexity, Code reuse, Coupling S10
and cohesion
Code quality Not provided S2, S4, S5,
S6, S12
Reusability software Not provided S13
components
Schedule Time Release cycles, Time between two subsequent releases S8, S9
Mean-time per assertion, Mean time, Total/overall time, S7
Average hours worked, Time taken in person hours, Time
taken on testing as a percentage of total time, Time spent
on coding as a percentage of total time, Total person hours
Development time S10
Lead time S1
Development time, feedback time S2
Testing time S12
Productivity Output/unit effort, Total time to complete a story card, S7
New lines of code/ development effort, LOC/person hour,
LOC/development effort, User stories/ person hour
Development time/person hours spent/task time, total S11
LOC divided by total effort, or the number of LOC per
hour, total noncommented LOC, number of delivered sto-
ries per unit effort (or implemented user stories per hour),
delivered noncommented LOC per unit development ef-
fort (or effort per ideal programming hour), and hours per
feature/development effort per LOC
Not provided S10
Budget Cost Not provided S2
Development cost S10
Resource Work size Not provided S4, S10
Effort Amount of Rework S9
Testing effort, development effort S10
Adherence Not provided S7
Developers’ Opinion S7
Risk Technical & Market Not provided S8
Risk
Communication Customer collabora- Not provided S4
tion
Team work Not provided S1, S4, S10
4.5.3 Quality Assessment Results

The quality scores for secondary studies are between 2–8, see Table 4.11. The maxi-
mum score that a secondary study could obtain is nine.
Table 4.11: Quality Assessment
ID C1 C2 C3 C4 C5 C6 C7a C7b C8 Sum

RQ Search Incl./Excl. QA Bias Detail Synthesis SoE Concl.
S1 0,5 0,5 0,5 0 0 0,5 0 0 1 3
S2 0,5 0,5 1 0 0 1 1 0 1 5
S3 0,5 1 1 0 0 0,5 0,5 0 1 4,5
S4 0,5 1 1 1 1 1 0,5 1 1 8
S5 0,5 0,5 0 1 0 0 0 0 0 2
S6 1 0,5 1 0 0 0,5 0,5 0 1 4,5
S7 0,5 0,5 1 1 1 1 1 1 1 8
S8 0,5 0,5 1 0 0 0 0 0 1 3
S9 0,5 0,5 1 0 1 0,5 0,5 0 1 5
S10 0,5 1 0,5 1 0 1 0,5 0 1 5,5
S11 0 0,5 0 0,5 0 1 1 1 1 5
S12 0,5 0,5 1 0,5 0 0,5 0 0 0 3
S13 0,5 0,5 1 0 0 0 0,5 0 0 2,5
Sum 6,5 8 10 5 3 7 6 3 10
Five secondary studies scored three and below (S1, S5, S8, S12, S13). Most of
the low-scoring studies are missing quality criteria for preventive steps to minimize
bias (C5) and an actual synthesis of the found evidence (C7a and C7b). Criteria like
minimizing bias, providing sufficient information on the included primary studies, syn-
thesizing evidence, and evaluating the strength of evidence are often missed or only
partially addressed in most of the secondary studies.
Most secondary studies have clear research questions. However most of them are
not formulated with respect to population, intervention, comparator, outcome and study
design (PICOS). Most secondary studies often refer to another publication pertaining to
their quality criteria and do not present the results of their quality assessments. Regard-
ing minimizing bias, few secondary studies report inter-rater agreement (e.g., Kappa
coefficient) or piloting of the review. Lastly, strengths of evidence are often not con-
sidered.
When we look at the quality scores of the secondary studies, we can see three
different clusters, they are: (1) low quality, (2) medium quality, (3) high quality, see
Low Quality Medium Quality High Quality

Cluster Cluster Cluster
S8
S12 S9
S11 S7
S13 S3 S10
S5 S1 S6 S2 S4
0 1 2 3 4 5 6 7 8 9
Quality Score
Figure 4.4: Clusters of Secondary Studies Based on Quality Scores
Figure 4.4. The clusters of quality will be used in the analysis of the secondary studies
in Section 4.5.4.
4.5.4 Empirical Support for the Impacts of Agile and Lean Prac-
tices
In the following subsections, we use the quality score clusters from Section 4.5.3 to
map the impacts of Agile and Lean practices on project constraints according to the
secondary studies summarized in Table 4.9. It should be noted that the following sub-
sections are not a continuation nor part of the quality assessment. We use a ‘+’ sign to
indicate that an Agile and Lean practice has a positive impact on a project constraint.
A ‘0’ indicates that the Agile and Lean practice has no impact and a ‘-’ indicates that
it has a negative impact on a project constraint.
For each Agile and Lean practice, we present the number and types of primary
studies, e.g., experiment, case study, or survey, that support a particular impact rela-
tionship (‘+’, ‘0’, or ‘-’). If the type of a primary study is unclear, we note the type as
“study”. If the secondary study did not report on a particular impact relationship of an
Agile and Lean practice and a project constraint, we leave the table cell blank.
To simplify the presentation of the tables in subsections 5.4.1 – 5.4.3, we use codes
that represent the research methodologies and contexts of the primary studies, e.g., IC
refers to an industry case study (see Table 4.12).
High Quality Secondary Studies
In this subsection, we report the impacts of Agile and Lean practices on different
project constraints as reported in high quality secondary studies (S4,S7). There are
four project constraints that are reported in high quality secondary studies, and these
are, quality, schedule, resource, and communication.
Table 4.12: Codes of Contexts and Research Methods and Their Definitions
Context Research Method

Academic A Case Study C
Industry I Experiment E
Mixed Study MS
Multi-case Study MC
Simulation Study SI
Survey S
In S7, the primary studies were categorized along their rigour and relevance accord-
ing to Ivarsson and Gorschek [97]. Rigour captures how well a study is designed and
executed, in particular the extent to which these aspects are reported and can be evalu-
ated. Relevance captures the potential impact of a study on academia and industry. A
large scale case study in an industrial context would for example be more industry rel-
evant than an artificial academic experiment with students as subjects. In the following
subsections we will refer to rigour and relevance as described above.
Quality. Different measures of quality are reported, they include, external quality,
complexity, code size, etc. Empirical studies of varying rigour and relevance in S7
are largely in agreement that TDD has a positive impact on external quality. A small
number of empirical studies with low relevance suggest that TDD has no impact on
external quality.
Empirical studies with low relevance in S7 are in agreement that TDD has no im-
pact on internal quality. Meanwhile two empirical studies involving students show that
TDD has a positive impact on internal quality. We also observed that empirical studies
with low relevance in S7 do not show agreement with respect to the impact of TDD
on number of test and code size. For the individual mapping of other Agile and Lean
practices and their impacts on quality as reported in high quality secondary studies, see
Table 4.13.
Schedule. There are different measures associated with schedule such as, time,
productivity, and progress tracking. Empirical studies, of high and low relevance, re-
ported in S7 do not show agreements regarding the impacts of TDD on time. Empirical
studies show that TDD could have a positive, negative, or no impact on time. A similar
trend is also shown for the impacts of TDD on productivity. For the individual map-
ping of other Agile and Lean practices and their impacts on schedule as reported in
high quality secondary studies, see Table 4.14.
Resource. Table 4.15 shows the impact of Agile and Lean practices on resource
as reported in high quality secondary studies. The measures associated with resources
include developers’ opinion, conformance, and work estimation.
Table 4.13: Impact of Agile Practices on Quality in High Quality Secondary Studies
Practice Measure Ref. Quality

+ 0 -
Pair program- External qual- S4 1 AC
ming ity
Short iteration External qual- S4 1 IMC
ity
TDD External qual- S7 6 ICs, 1 S a
ity
S7 4 ICs b
S7 2 AEs, 1 IEc 6 AEsc 1 AEc
d d
S7 1S 1 AE
Internal quality S4 1 AC
S7 1 AE c 2 IEs, 8 AEs c
S7 1 IE, 2 AEs d
a
Complexity S7 2 ICs
Robustness S7 1 IE d
No of test S7 1 IE c 2 AEs, 1 IE c 1 AE c
d
1 IE, 1 AE
Code size S7 2 AEs c 1 IE c
S7 1 IE d
a
Studies with high rigour and high relevance
b
Studies with low rigour and high relevance
c
Studies with high rigour and low relevance
d
Studies with low rigour and low relevance
Empirical studies in S7 show that TDD has a positive impact on developers’ opin-
ion. The impact is supported by two surveys with high rigour and high relevance. One
industry experiments with high rigour and low relevance reports that adherence to TDD
process has a positive impact. In S4, one academic case study indicates that planning
game has a positive impact on work estimation.
Communication. Table 4.16 shows the impact of Agile and Lean practices on
communication. The measures associated with communication include customer col-
laboration and team work.
Empirical studies reported in S4 show positive and negative impacts of on-site cus-
tomer on customer collaboration. One industry case study reports that planning game
has a positive impact on team work (S4).
Summary: There are two secondary studies in the high quality cluster. S4 and S7
do not have overlapping primary studies (see Section 4.5.1 for primary studies overlap).
Table 4.14: Impact of Agile Practices on Schedule in High Quality Secondary Studies
Practice Measure Ref. Schedule

+ 0 -
TDD Time S7 2 ICs b
S7 4 AEs c 1 AE, 1 IE c
d
S7 1 IC, 1 AE
Productivity S7 1 ICa 2 ICsa
S7 1 ICb
c c
S7 1 AE 2 IEs, 8 AEs
S7 1 Sd
Pair program- Time S4 1S 1 AC
ming
Stand up meet- Progress track- S4 1 IC, 1 IMC
ing ing
a
b
c
d
Table 4.15: Impact of Agile Practices on Resource in High Quality Secondary Studies
Practice Measure Ref. Resource

+ 0 -
TDD Developers’ S7 2 Surveys a
opinion
S7 1 IC, 1 AE c 4AEs c
d
S7 2 AEs
Adherence S7 1 IEc
Planning game Work estima- S4 1 AC
tion
a
b
c
d
Both secondary studies use SLR as a review method and Kitchenham and Charters
[110] as a guideline (see Section 4.4.1 for secondary study guidelines).
Between the two secondary studies, S7 provides more detailed information on the
included primary studies than S4. S7 is the only secondary study that classifies its pri-
mary studies based on their rigour and relevance. The details provided by S7 could be
influenced by its aim to examine impacts of TDD based on the primary studies’ rigour
Table 4.16: Impact of Agile Practices on Communication in High Quality Secondary

Studies
Practice Measure Ref. Communication

+ 0 -
On-site cus- Customer collabo- S4 1 IC 2 IMCs
tomer ration
Planning game Team work S4 1 IC
and relevance. Meanwhile S4 is aimed to identify factors in Agile software develop-

ment (see Section 4.4.1 for secondary studies’ aims). From S7, we also observed that
experiment studies, both in industry and academia, have a tendency to report that TDD
has no impact on different measures of a project constraint.
We found a large number of primary studies that report on positive impacts but we
cannot dismiss the smaller number of studies that reported on negative or no impacts.
The classification made in S7 based on rigour and relevance is helpful to deduce the
impact of an Agile and Lean practice. However, we still cannot make strong assertions
on the impact of an Agile and Lean practice. Primary studies with high rigour and high
relevance are more trustworthy. However, we cannot dismiss studies of low rigour and
low relevance. The heterogeneity of the data made aggregating findings from the two
secondary studies difficult.
Medium Quality Secondary Studies

In this section, we report the impacts of Agile and Lean practices on different project
constraints as reported in medium quality secondary studies (S2, S3, S6, S9, S10, S11).
There are four project constraints that were reported in medium quality secondary stud-
ies, and these are: quality, schedule, budget, resource, and communication.
Quality. The measures associated with quality include external quality, complexity,
cohesion, code reusability, etc. Empirical studies in S3, S10, and S11 do not show
agreement regarding the impacts of TDD on external quality. We found a number of
empirical studies with different research methods and contexts that indicate TDD has
a positive impact on external quality. However, a number of experiments also indicate
that TDD could have no impact as well as negative impact on external quality.
We also observed that empirical studies in S2 and S10 are largely in agreement with
respect to the positive impact of TDD on complexity. Only one empirical study in S2
that suggests TDD has a negative impact on complexity. For the individual mapping
of other Agile and Lean practices and their impacts on quality as reported in medium
quality secondary studies, see Table 4.17.
Table 4.17: Impact of Agile Practices on Quality in Medium Quality Secondary Studies

+ 0 -
Refactoring Quality S10 1 AC, 1 IC
TDD External qual- S3 1 study
ity
S10 5 AE, 1 Survey, 1 IE 2 AE
S11 8 IEs 18 AEs 7 experiments comparing
to iterative test last
Complexity S2 1 (IE+AE+IC), 3 ICs, 1 1 IC
AE
S10 2 ICs
Code quality S2 9 ICs, 2 AEs, 1 AC, 1 IE 2 AEs 1 AE
S6 1 SI
Code size S2 1 (IE+AE+IC) 1 AE
Cohesion S10 1 AE+IE
Code reusabil- S10 1 AE
ity
Pair program- Code quality S10 5 AEs, 1 IE
ming
External qual- S10 1 S, 1 AE 1 AE
ity
VSM External Qual- S9 2 studies
ity
Schedule. The measures associated with schedule include productivity, time, and
release cycle. Empirical studies in S2 and S10 are largely in agreement that TDD has a
negative impact on development time. However, S2 also report a number of empirical
studies that show contradictory results. Inconsistent impacts of TDD on productivity
are also shown in the empirical studies reported in S10 and S11.
Meanwhile empirical studies in S10 show that pair programming has no impact
on both productivity and schedule. For the individual mapping of other Agile and
Lean practices and their impacts on schedule as reported in medium quality secondary
studies, see Table 4.18.
Budget. The empirical studies on the impacts of Agile and Lean practices on budget
reported in medium quality secondary study is rather scarce. Table 4.19 shows the
impacts of Agile and Lean practices on budget as reported in medium quality secondary
studies. The measures associated with budget include cost and development cost.
Table 4.18: Impact of Agile Practices on Schedule in Medium Quality Secondary Stud-
ies

+ 0 -
Refactoring Productivity S10 3 ICs, 1 AMS
TDD Development S2 3 AEs, 2 ICs 1 IC, 1 AE 6 ICs, 1 IE, 3 AEs, 1 AC
time
S10 2 IC
Feedback time S2 3 ICs, 1 S
Productivity S10 2 AEs 1 IC 1 IC
S11 18 AEs 7 experiments comparing 8 IEs
to iterative test last
Pair program- Productivity S10 1 IC, 1 IE
ming
Schedule S10 1 IE, 1 S, 1 AC, 2 AEs
Internal/ exter- Release cycle S9 1 study
nal release
Empirical studies in S2 and S10 suggest that TDD has a positive impact on cost.
S10 reports two academic experiments and one academic case study that suggest pair
programming has a negative impact on cost.
Table 4.19: Impact of Agile Practices on Cost in Medium Quality Secondary Studies
Practice Measure Ref. Cost

+ 0 -
TDD Cost S2 1 IC
Development S10 1 IC
cost
Pair program- Cost S10 2 AEs, 1 AC
ming
Resource
The empirical studies on the impacts of Agile and Lean practices on resources reported
in medium quality secondary study is also scarce. We found only one empirical study
for each impact of Agile and Lean practices on resources.
One academic experiment reports that TDD has a negative impact on testing effort.
However, another academic experiment reports that TDD has a positive impact on de-
velopment effort (S10). For the individual mapping of other Agile and Lean practices
and their impacts on resources as reported in medium quality secondary studies, see
Table 4.20.
Table 4.20: Impact of Agile Practices on Resource in Medium Quality Secondary Stud-
ies
Practice Measure Ref. Resource

+ 0 -
TDD Testing effort S10 1 AE
Development effort S10 1 AE
Adherence S2 1S 3 ICs
Pair program- Effort S10 1 IE
ming
Planning game Work size estima- S10 1 AC
tion
VSM Rework effort S9 1 study
Communication. One medium quality secondary study reports the impact of an

Agile practice on communication (see Table 4.21). S10 reports three academic experi-
ments, one survey, one academic experiment, and one industry experiment that indicate
pair programming has a positive impact on teamwork.
Table 4.21: Impact of Agile Practices on Communication in Medium Quality Sec-

ondary Studies

+ 0 -
Pair program- Teamwork S10 3 AEs, 1 S, 1 AC, 1 IE
ming
Summary: None of the primary studies in the medium quality cluster were assessed
based on rigour and relevance. Research methods and contexts of the primary studies
are traceable for most of the primary studies. However, making conclusive remarks
regarding the impact of Agile and Lean practices was difficult. Although we found
a number of industrial empirical studies that report on a positive impacts,, we cannot
dismiss studies in academic settings with contrary results.
From the medium quality cluster, S2 and S10 have small number of overlapping
primary studies. Both S2 and S10 examine the impact of TDD on complexity, devel-
opment time, and cost. However, the primary studies included in S2 and S10 are not
in agreement regarding the impacts of TDD on development time and cost. S2 report a
positive impact and a negative impact of TDD on complexity. S10 only report a posi-
tive impact of TDD on complexity. However, S2 and S10 are in agreement with regards
to a positive impact of TDD on cost.
We did not find other secondary studies in the medium quality cluster that have
overlapping primary studies. However, S2 and S7 have many overlapping primary
studies, see Section 4.5.1. Both S2 and S7 examine the impact of TDD on the code
quality, code size, complexity, time and adherence. However, the primary studies in
S2 and S7 are not in agreement regarding the impacts of TDD on the aforementioned
measures.
S2, S7, and S11 are secondary studies that focus on examining TDD. Except S7,
S2 and S11 are in medium quality cluster. When we examined the impacts of TDD
as indicated the primary studies (in S2, S7, and S11), we did not find agreements.
However, when we looked at the conclusions made in S2, S7, and S11, we found
similarities. Although S2, S7, and S11 use slightly different measures to assess quality,
these secondary studies conclude that TDD has a positive impact on improving quality.
S7 and S11 both conclude that the impact of TDD on productivity is inconclusive,
however more likely to be negative.
From Table 4.17, we observed that experiment studies conducted with students
tend to report that Agile and Lean practices has no impact on quality measures. When
we examined the secondary study methods, with the exception of S3, S6, and S9, the
medium quality secondary studies used SLR as a method and Kitchenham and Charters
[110] as a guideline. S3 and S9 are both SM studies. S3 used the guideline from
Petersen et al. [168], meanwhile S9 referred to [110] as a guideline. S6 modified the
steps from [110] and referred the method as quasi-SLR.
When we examined the aims of the secondary studies in the medium quality cluster,
S2, S10, and S11 are aimed to find association. It is evident from Table 4.17 – 4.21, S2,
S10, and S11 examine the associations of different Agile and Lean practices and their
impacts of different project constraints. S6 is aimed to identify factors, meanwhile S3
and S9 are aimed to classify primary studies. Perhaps since S3, S6 and S9 are not aimed
to find association, they only report positive impacts of Agile and Lean practices.
Low Quality Secondary Studies

In this section, we report the impacts of Agile and Lean practices on different project
constraints as reported in low quality secondary studies (S1, S5, S8, S12, S13). There
are five project constraints that are reported in low quality secondary studies, they are,
scope, quality, schedule, risk, and communication.
Scope. We identified only one secondary study in the low quality cluster that reports
the impact of Agile practices on scope (see Table 4.22). In S12, one case study and two
studies suggest that TDD has a positive impact on requirements specification.
Table 4.22: Impact of Agile Practices on Scope in Low Quality Secondary Studies
Practice Measure Ref. Scope

+ 0 -
TDD Requirements S12 1 C, 2 studies
Quality. Studies reported in S5 and S12 are in agreement that TDD has a positive
impact of code quality. In S13, two studies suggest that TDD has a positive impact
on component reusability. The contexts of the primary studies are not reported by the
secondary studies. For the individual mapping of other Agile and Lean practices and
their impacts on quality as reported in low quality secondary studies, see Table 4.23.
Table 4.23: Impact of Agile Practices on Quality in Low Quality Secondary Studies

+ 0 -
Kanban pull sys- Work quality S1 1 study
tems
Adaptive Plan- Customer satisfaction S1 5 studies
ning
TDD Code quality S5 2 studies in open source
projects
S12 2 Cs, 1 MC, 2 studies
Software component S13 2 studies
reusability
Incremental deliv- External quality S8 4 studies 2 studies
eries
Test predictability S8 1 study
Schedule. All primary studies included in low quality secondary studies indicate
that Agile and Lean practices have positive impacts on different measures of schedule,
see Table 4.24. None of the secondary studies report the research methods and contexts
of the primary studies.
Risk. We identified only one secondary study in the low quality cluster that re-
ports the impact of Agile practices on risk (see Table 4.25). One study suggests that
incremental deliveries has a positive impact on managing technical and market risks
(S8).
Communication. All primary studies included in low quality secondary studies
indicate that Agile and Lean practices have positive impacts on different measures of
Table 4.24: Impact of Agile Practices on Schedule in Low Quality Secondary Studies

+ 0 -
Kanban pull sys- Lead time S1 1 study
tems
Adaptive planning Efficiency S1 2 studies
Team chooses own Time S1 1 study
task
Incremental deliver- Release cycle S8 3 studies
ies
TDD Testing time S12 1 study
Table 4.25: Impact of Agile Practices on Risk in Low Quality Secondary Studies
Practice Measure Ref. Risk

+ 0 -
Incremental Technical and S8 1 study
delivery market risk
communication, see Table 4.26. None of the secondary studies report the research
methods and contexts of the primary studies.
Table 4.26: Impact of Agile Practices on Communication in Low Quality Secondary

Studies

+ 0 -
Adaptive planning Team work S1 5 studies
Incremental deliver- Customer collabo- S8 1 study
ies ration
Summary: All secondary studies in the low quality clusters do not mention the
contexts of the primary study. With the exception of S12, most secondary studies also
do not report the research methodology of the primary studies. We also observed that
there is a tendency of bias from the low quality secondary studies. No secondary studies
in the low quality cluster report negative impacts of Agile and Lean practices on the
project constraints.
None of the secondary studies in the low quality cluster are aimed at finding as-
sociation. With the exception of S1, most secondary studies are aimed at identifying
factors. Meanwhile the aim of S1 is to survey the literature. The aims of the secondary
studies could contribute the tendency of bias in the low quality cluster. Since the sec-
ondary studies are not aimed to examine the impacts of Agile and Lean practices.
When examining the overlapping of primary studies, S1 and S8 have one overlap-
ping primary study. However, S1 and S8 do not examine the same Agile practices nor
measures of project constraints.
With the exception of S8, most secondary studies in low quality cluster used SLR as
a secondary study method with the guideline from [110]. In S8, the authors mentioned
that they did not perform the complete SLR steps as mentioned in [110], and referred
to their secondary study method as semi-SLR.
General Summary on the Impacts of Agile and Lean Practices

We identified a large number of primary studies that study the impact of TDD on exter-
nal quality. Many of those primary studies indicate that TDD has a positive impact on
external quality. Unfortunately, we are unable to make any inference on the remaining
Agile and Lean practices, due to inefficient data. For example, the positive impact of
planning game on team work, incremental delivery on test predictability, and Kanban
pull systems on work quality are only reported in one primary study.
4.6 Discussion
In this tertiary study, we reviewed 41 secondary studies in Agile and Lean software
development. Thirteen out of 41 secondary studies reported the impacts of Agile and
Lean practices on various measures of project constraints.
4.6.1 Reflections on Secondary Studies in Agile and Lean

Our tertiary study revealed a large number of secondary studies in Agile and Lean
software development. However, many of them do not report clear review processes.
Therefore, we only included 41 secondary studies after full text review.
We also found that most secondary studies are aimed at identifying factors in Agile
and Lean software development such as, challenges or benefits of practices. This result
is similar to the tertiary study by Verner et al. [229], where they found that most SLRs
in GSD produced summary and mappings of the literature. In our tertiary study, we
identified four secondary studies that specifically report the impacts of Agile and Lean
practices.
The guideline from Centre for Reviews and Dissemination (CRD) suggested that
an SLR is aimed to evaluate and summarize available evidence and make them more
4.6 Discussion 129
accessible to decision makers [34]. However, the results from our quality assessment
process shows that very few secondary studies performed quality evaluation of primary
studies. This is inline with findings reported by Kitchenham et al. [114] from a review
of SLRs in software engineering.
One of the goals of EBSE is to provide a means for practitioners in making in-
formed decisions about technology adoption [112]. Our tertiary study also revealed
that very few secondary studies performed synthesis of the evidence. The lack of evi-
dence synthesis hinders practitioners to make informed decisions pertaining to adoption
of Agile and Lean practices. Without synthesized evidence, practitioners cannot deter-
mine whether adopting Agile and Lean practices would be beneficial for them. The
results of our tertiary study show that most secondary studies mentioned that they per-
formed SLRs. Yet, most of the SLRs are aimed at identifying factors, i.e., used Agile
and Lean practice, benefits, and limitations, instead of synthesizing evidence pertaining
to the impacts of Agile and Lean practices.
The lack of overlap of primary studies indicates the lack of a common goal among
researchers, which is one of the goals of EBSE. It indicates that researchers are not
building on top of each others’ work. Instead individual researchers, or research groups
are conducting their own search process. The lack of a common goal among re-
searchers is also reflected on the breadth of topics being investigated in the secondary
studies of Agile and Lean software development.
Our tertiary study also revealed ”simplified” versions of an SLR, they are, quasi-
SLR (S6) and semi-SLR (S8). Both in S6 and S8, the authors mentioned that they did
not perform a number of steps that are mentioned in the SLR guideline. Therefore their
secondary study would not fit a ”complete” SLR. Kitchenham et al.’s reported that one
of the key issues in performing SLR is the time spent to perform the steps [113]. Our
study revealed a number of alternatives that have been performed to simplify the SLR
steps.
4.6.2 Reflections on the Impacts of Agile and Lean practices

From 41 secondary studies only 13 provided discussions on the impact of Agile and
Lean practices. Out of 26 practices mentioned in [164], we found 13 practices that were
mentioned in the secondary studies. Four out of 13 secondary studies are dedicated to
specific practices, i.e., TDD (S2, S7, S11), and incremental deliveries (S8). TDD is
the most studied Agile practice. TDD is mentioned in 10 secondary studies. Hanssen
et al. [84] performed a tertiary on GSD and identified a number of Agile practices
that are used in GSD contexts, such as, continuous integration, pair programming,
retrospective, and daily stand up meetings. Our tertiary study also identified the same
practices, with the exception of continuous integration. Our study shows there are many
Agile and Lean practices are still under-studied, e.g., refactoring, stand-up meeting,
etc., see Table 4.9. Our findings do not imply that the rest of Agile and Lean practices
are not evaluated at all. The reason why some practices are under-studied could be
attributed to the decline in the adoption of some Agile and Lean practices, such as,
TDD, pair programming, and continuous integration [142, 212].
We identified a number of measures that are associated with a project constraint.
For instance, number of defects is the most commonly used measure of quality. How-
ever, we did not find commonly used measures for other project constraints. For in-
stance, we identified a number of measures that are associated with schedule, e.g., time
between two releases, lead time, development time, testing time, etc. The measures
used in the secondary studies also have varying granularity levels. For instance, some
secondary studies used more specific measures like complexity or code size. However
other secondary studies used a generic measure like code quality. Our finding suggests
that currently there is no agreed measures to assess the impacts of Agile and Lean
practices on project constraints. The variations of measures and their level of granu-
larity made the aggregation of findings more difficult. Similar observation is shared
in S11 [175], where no common metrics were used to measure internal quality. As a
consequence Rafique and Mišić [175] were not able to consider internal quality in the
analysis of TDD’s impact on quality.
In this tertiary study, we also examined the overlap of primary studies between the
secondary studies. We found little overlap between the secondary studies, even those
with similar topics. The largest overlap of primary studies is between S2 and S7 with 21
overlapping primary studies. Both S2 and S7 are aimed at finding the impacts of TDD,
particularly on quality and schedule. There are a number of measures that are examined
in S2 which are also examined in S7, such as, code size, complexity, adherence, and
time. Despite similar aims and large overlap of primary studies, the primary studies
in S2 and S7 are not in agreement with respect to the impacts of TDD on code size,
complexity, adherence, and time. Our tertiary study shows that two secondary studies
with a similar topic may not yield the same results. Our finding is similar to the tertiary
study conducted by Wohlin et al. [240], where the results of a secondary study may not
be reproducible.
Three secondary studies, S2, S7, and S11, are dedicated to examining the impacts
of TDD. The primary studies of S2, S7, and S11 show contradictory results regarding
the impacts of TDD. However, when we examined the conclusions in S2, S7, and S11,
we identified similarities. S2, S7 and S11 indicate that TDD has a positive impact on
quality, despite the lack of overlapping primary studies between S7 and S11. Mean-
while, S7 and S11 conclude that the impact of TDD on productivity is inconclusive,
with a tendency towards negative impact. Our tertiary study shows that two secondary
studies with a similar topic can have similar conclusions. Our finding is similar to the
4.6 Discussion 131
tertiary done by MacDonell et al. [134] where two secondary studies with a common
research question could yield consistent results.
Wohlin et al. [240] and MacDonell et al. [134] found contradicting results regard-
ing the reliability and reproducibility of secondary studies’ results. Our tertiary study
finds similarities with both tertiary studies. When we examine the conclusions of the
secondary studies with a common topic we identified commonalities. However, when
we examine the finer details of the primary studies that are included in the secondary
studies, we identified inconsistencies. Our tertiary study shows that there are different
perspectives within a secondary study. Depending on the readers’ perspectives, the
outcomes of a secondary study may be more or less relevant. Some readers might be
more interested in the general picture, some readers might be more interested in the
detailed picture.
Given the high interest in TDD, it is important to bring up the issue of adherence
to TDD process. S11 reports that often primary studies do not report the level of
adherence to the description of TDD. S2 and S7 report that one industry experiment and
one survey suggest that adherence to TDD process has a positive impact. However, S2
also reports that three industry case studies suggest adherence to TDD has a negative
impact. Our study indicates the importance of reporting the level of adherence in future
TDD studies. Also, the importance of further examining the impacts of low adherence
versus high adherence to TDD process.
When we examined the clusters of quality scores, we did not find a distinctive out-
come from each cluster with respect to the impact of Agile and Lean practices. How-
ever, high and medium quality secondary studies provide sufficient details with regards
to the primary studies’ method and contexts. One secondary study in the high quality
cluster (S7) provide additional information pertaining to the rigour and relevance of the
primary studies. This information helped us to discern how and in which contexts the
results were obtained.
From the high and medium quality secondary studies clusters we observed that ex-
periment studies, especially with students, tend to show that an Agile and Lean practice
has no impact on any of the project constraints measures. The result of our study shows
that we need to examine experiment studies more closely, especially those that involve
students. Our study indicates that we need to better understand why experiment studies
tend to show no impact. We need to investigate if the trend is influenced by experiment
design, the formulated hypotheses, or selected confidence level.
We also uncovered a trend of publication bias from the low quality secondary stud-
ies cluster. All secondary studies in the low quality cluster did not report on the negative
impacts of Agile and Lean practices. However, the tendency of bias from secondary
studies in the low quality cluster is influenced by the aims of the secondary studies.
Secondary studies in the low quality cluster are primarily aimed to survey the litera-
ture and identify factors. Our study shows the possibility of publication bias could be
influenced by the aims of the secondary studies themselves.
Our tertiary study reveals that there is a significant amount of data pertaining to
TDD, especially on the impact of TDD on external quality. The secondary studies
show that TDD has a positive impact on external quality. However, it is important to
note that this inference is solely based on the number of primary studies. We are not
able to make a strong assertion due to the heterogeneity of the data. For the most part
we do not know the quality of the primary studies that report on the impact of TDD
on external quality. Also, currently, there is no known approach to aggregate findings
from different research methods, such as, case studies and experiments. Furthermore,
currently there is no agreed method on how to value or weigh the results of studies with
high rigour and relevance versus studies of low rigour and relevance. We also do not
have an agreed method on how to value studies with practitioners versus studies with
students.
We are unable to make similar inference for the other Agile and Lean practices
and their impacts on project constraints. Most of them are only reported in one or two
secondary studies. Furthermore, the number of primary studies that we identified from
the secondary studies are not sufficient to make any remarks.

The first implication for research is our tertiary study shows the need to further inves-
tigate the benefits and trade-offs of Agile and Lean practices. Our tertiary study shows
that an Agile practice, such as TDD can have both a positive impact on one project con-
straint and a negative impact on another project constraints. Our tertiary study reveals
the need to put effort into investigating benefits and trade-offs of combining different
In this tertiary study, we uncovered the contradictory outcomes of Agile and Lean
practices obtained from different secondary studies with varying degree of quality. The
heterogeneity of the data made it difficult to come to conclusive remarks pertaining to
the impacts of Agile and Lean practices. The second implication for research from our
tertiary study is the need to translate the knowledge obtained from this tertiary study,
as well as existing secondary studies, to practitioners. This is known as knowledge
translation [115]. As researchers we need to involve industry practitioners on how to
better interpret and formulate good recommendations based on the results of secondary
and tertiary studies. The mapping of Agile and Lean practices and their impacts on
project constraints based on this tertiary study could be used as a starting point to
initiate knowledge translation process.
4.7 Conclusions 133
Our tertiary study also shows that there are different perspectives to a secondary
study. The third implication for research from our tertiary study for future secondary
studies is to include ”perspectives” in the implication for practice section. Practitioners
and managers who read the secondary study will be able to navigate through different
parts of the findings that are more relevant to them. For instance, managers might
be more interested in the general findings. Meanwhile practitioners might be more
interested in the detailed impacts on certain measures.
Our tertiary study also reveals gaps in Agile and Lean research which opens up an
avenue for further research. There are still many Agile and Lean practices that have
not been explored, like metaphors, coding standards, etc. We need to better understand
why these practices are not investigated as intensively as the others. With new studies
that show that some practices are declining in their adoption [142, 212], it is also worth
investigating the reasons why practices, like TDD and pair programming, that appear
to have a positive impact on quality are declining in their adoption.

Our tertiary study compiled a list of Agile and Lean practices and their impacts (posi-
tive, negative, or no impact) on various measures of project constraints. Although we
did not provide suggestive propositions, practitioners can use our findings to help with
assessing the suitability, as well as possible trade-offs, of adopting an Agile and Lean
practice. For instance, a number of secondary studies report that TDD has a positive
impact on external quality, but TDD might compromise productivity. If a practitioner
wants to adopt TDD, they could expect improvement in external quality however, it
might negatively affect productivity.
Both practitioners and managers can benefit from our tertiary study as it provides an
overview of the impacts each Agile and Lean practice on different measures of project
constraints. Practitioners can benefit from understanding better the impact of an Agile
and Lean practice on different measures of a project constraint. Managers can benefit
from getting an overview of the potential impacts that an Agile and Lean practice can
have on a project constraint.
4.7 Conclusions
The aim of this tertiary study is to provide a consolidated view of findings pertain-
ing to the impacts of Agile and Lean practices on project constraints. The aim was
achieved by conducting a tertiary study in Agile and Lean software development. We
performed an automated database search and found 122 secondary studies in Agile and
Lean software development. From the 122, only 41 secondary studies passed the in-
clusion/exclusion criteria. Only 13 out of 41 secondary studies provided discussions
pertaining to the impacts of the Agile and Lean practices.
We identified 13 Agile and Lean practices and their impacts on different project
constraints. TDD and its impact on quality is the most studied practice-impact asso-
ciation. Given the heterogeneity of the data we were unable to perform a synthesis.
Instead we mapped the findings based on the quality scores of the secondary studies
and the respective Agile and Lean practices and their impacts.
4.7.1 Revisit the Research Questions

In this subsection, we revisit the research questions that were formulated in this study.
RQ1: What are the sets of primary studies that the secondary studies draw upon?
The secondary studies in Agile and Lean software development draw from a large set of
primary studies. The overlap between the secondary studies is small. Most secondary
studies do not have overlaps with each other. There are 420 primary studies that are
included in 13 secondary studies that we included for detailed analysis. Only 50 out of
420 primary studies are overlapping. The main reason for overlapping primary studies
is a similarity in topic, such as Agile/SPLE and TDD. However, the distinctions in
the review designs of the secondary studies leads to the small number of overlap. For
instance S7, and S11 investigate the impacts of TDD. However, S11 is focused on
experiments in TDD, meanwhile S7 includes different types of empirical studies.
RQ2: What are the scopes of the secondary studies in evaluating Agile and Lean
software development? We identified 13 Agile and Lean practices out of 26 that are
mentioned in [164], e.g., refactoring, pair programming, TDD, etc. The list of 13
practices can be found in Section 4.5.2. We also found the impacts of the 13 Agile and
Lean practices on various measures of project constraints, such as, external quality,
internal quality, time, etc. The list of measures that are used in the secondary studies
can be found in Section 4.5.2.
RQ3:What is known about the impacts of Agile and Lean practices? We found that
different Agile and Lean practices can have positive, negative, or no impacts on the
different measures of project constraints. The impacts of each Agile and Lean prac-
tices can be found in Section 4.5.4, 4.5.4, and 4.5.4. We did not find strong agreements
between the secondary studies, even the ones with a similar topic and large overlap
of primary studies. When we examined the secondary studies based on their quality
clusters, we also did not find strong distinctions between the cluster in terms of the
impacts of Agile and Lean practice. However, we found that high and medium qual-
ity clusters provide sufficient information pertaining to the primary studies’ contexts,
rigour and relevance, and used empirical methods. Secondary studies in the ow quality
4.7 Conclusions 135
cluster do not provide any information regarding the contexts nor the empirical method
of the primary studies. Secondary studies in low quality cluster also show a tendency
of publication bias, because they do not report primary studies with negative impacts.
This tertiary study reveals the adverse impacts of Agile and Lean practices. We
identified a significant number of primary studies that suggest that TDD has a positive
impact on external quality. The heterogeneity of the data does not allow us to make a
conclusive assertion. For the remaining Agile and Lean practices that we identified in
this tertiary study, we are unable to make similar remarks due to insufficient data. How-
ever, we provide a consolidated view of the impacts of Agile and Lean practices and
the number of primary studies that support them, including their rigour and relevance,
research method, and contexts. For future work, we propose the following: (1) study
the trade-offs associated with implementing a combination of Agile and Lean prac-
tices, (2) investigate the impacts of other Agile and Lean practices that have not been
identified in this tertiary study, e.g., metaphors and user stories, coding standard, and
continuous integration (3) investigate why some Agile and Lean practices are declining
in adoption despite the high number of empirical studies that show their benefits.
Appendix
Aims of Secondary Studies
Table A1: Aims of Secondary Studies
Type Ref Aim

Classify Primary S3 Collect evidence about how Agile Software Product Line (SPL) research is struc-
Studies tured, synthesize current evidence on the integration of both approaches and iden-
tify further challenges for the integration of Agile methods and SPL.
S9 Identify and classify state of the art in large-scale software development influenced
by Lean Product Development (LPD) approaches.
S19 Analyze agile practices in order to explore their industrial usage with respect to
their distribution over different domains and processes from the perspective of
software engineers.
S21 Investigate the state of the art on how standardized model like Agile can work
with Model Driven Development (MDD)
S24 Identify relevant research and understand what the field of Agile-User Experience
(UX) looks like at present.
Find association S2 Investigate potential factors that are limiting the industrial adoption of Test Driven
Development (TDD).
S7 Investigates impact of TDD on different variables whilst taking two study quality
dimension into account, namely rigor and relevance.
S10 Evaluate according to the ISO/IEC 12207 and ISO/IEC 9126 standards, synthe-
size, and present, the empirical findings on quality in agile methods.
S11 Investigate the impact of TDD on external code quality and productivity.
S26 Gather and synthesize empirical evidence to provide convincing and illuminating
support for software project managers who need to make informed choices about
software development approaches for their projects, with respect to cost, duration
and quality
Identify Factors S4 Collect evidence on benefits and limitations of agile software development, under-
stand the strength of evidence of the findings, practical and research implications
S5 Assess the relationship between Agile Software Development (ASD) and Open
Source Software Development (OSSD).
S6 Characterize reconciliation among the plan-driven, agile, and free/open source
software models of software development.
S8 Identify the origin, prevalence, benefits, enablers, and problems of rapid releases.
S12 Evaluate, synthesize, and present the existing findings that will give the latest state
of research on applying agile software development method to embedded software
development.
S13 Present an detailed view about uses of reference architectures in agile
methodologies
S14 Identify the software practices usually used into the context of agile approaches
for the development of software.
S15 Provide insight into the Kanban approach and its elements (concepts, principles,
practices, techniques, and tools) that have been empirically reported by scholars
and practitioners.

4.7 Conclusions 137
Table A1 – Continued from previous page

Category Ref Aim
S16 Identify the characteristics of agile project management in organizations using ag-
ile methods and maturity models; regarding support approaches employed; from
the viewpoint of researchers; in academic and industrial context.
S17 Provide empirical support for a proposal of a methodology for integration of User
Centered Design (UCD) and Agile, identifying most common practices and arte-
facts used.
S18 Identify what barriers have been dealt with, and what challenges have to be ad-
dressed in the near future to apply Agile Product Line Engineering (APLE) to the
software industry.
S20 Find out to what experiences there are of Model-driven Agile Development
(MAD), from an empirical context.
S22 Identify, synthesize, and present the findings reported about using Scrum practices
in Global Software Development (GSD) to date.
S23 Systematically reviewing and summarizing the existing research literature and in-
vestigating which Agile practices have been used effectively in Global Software
Engineering (GSE) contexts.
S28 Present a detailed view of how agile methods have been used in the development
of embedded systems, and to describe their benefits, challenges, and limitations.
S29 Identify how Agile create business values
S30 Identify various challenging factors that restrict Agile and User Centred Design
Integration (AUCDI) and explore the proposed practices to deal with them.
S31 Gain a comprehensive understanding of the various factors that impact the sus-
tained usage of agile methods.
S33 Evaluate, synthesize, and present results on the use of the Capability Maturity
Model Integration (CMMI) in combination with agile software development.
S35 Examine industrial surveys published in 2011 and 2012, determine the extent to
which we could trust their reported high rates of agile method usage and provide
recommendations on how quality of research could be improved in the future.
S37 Provide an overview of studies within knowledge management in agile projects,
what kind of concepts have been explored, what the main findings were and what
were the research method
S38 Amass current knowledge about agile adoption and to identify essential future
research issues for empirical studies, especially on agile in the large settings.
S39 Identify technical and organizational challenges of Service Oriented Architecture
(SOA) and cloud in Agile context
Survey the literature S1 Investigate the status of Kanban in software development, in terms of its presence
in existing literature.
S25 Review the literature of actual use of software metrics in the context of agile soft-
ware development.
S27 Review the current research literature on effort estimation in agile, iterative and
incremental software projects (AIISPs) and evidences about common trends and
gaps
S31 Provide a synthesis of relevant studies in this issue in order to identify the scope
of current research and the shortcomings that need to be addressed in the future.
S34 Investigate the extent to which agile practices have been used in scientific software
projects. Second, we aim to investigate the impact on testing and requirements
activities in projects with agile practices.
S36 Provide a detailed overview of the state of the art in the area of effort estimation
in ASD.
Table A1 – Continued from previous page

Category Ref Aim
S40 Evaluates the potential of agile methods and techniques to address the challenges
of Model-Driven Modernization.
S41 Identify testing related problems in the context of the automotive software domain
and solutions that have been proposed and applied in an industrial context.
Chapter 5
Usage, Retention, and

Abandonment of Agile
Practices: A Reflection on Agile
Maturity Models
Indira Nurdiani, Jürgen Börstler, Samuel A. Fricker, Kai Petersen, “Us-

age, Retention, and Abandonment of Agile Practices: A Reflection on
Agile Maturity Models”, Submitted to e-Informatica Software Engineer-
ing Journal, February 2018.
5.1 Introduction
The software industry is highly competitive. Agile methods, like Scrum and eXtreme
Programming (XP), help to tackle the challenges of rapid changes in the environment
of software organizations and help to reduce time to market, minimize development
costs, and improve software quality [1]. Agile practices are the enactment of Agile
principles [239].
Recent surveys indicate that some practices like test-driven development (TDD),
pair programming, and continuous integration are decreasing in their adoption rates
140 Usage, Retention, and Abandonment of Agile Practices
[142] or are abandoned [212]. Abandoning Agile practices seems contradictory to

common guidelines such as Agile maturity models (AMMs) [144, 160, 202] that pre-
scribe which practices should be implemented and when according to certain maturity
levels. According to the AMMs, the more mature an organization becomes, more Agile
practices are adopted. However, the trend of abandonment of practices could also be
due to lack of guidance. Perhaps such practices were not introduced at the right time,
given the maturity of the software development teams or organization, because Agile
practices dependencies are not well known.
Currently, we do not know why Agile practices are abandoned and how this impacts
the overall success of Agile implementations. Without such information, we are unable
to evaluate the suitability of AMMs in industry. As the first step towards evaluating the
suitability of the AMMs is to identify the rationales for abandoning Agile practices.
In this study, we aim to identify Agile practices that have been abandoned and the
rationales for their abandonment. To achieve our aim we conducted a web survey and
11 interviews with industry practitioners with experience in Agile.
The remainder of the chapter is structured as follows: Section 5.2 presents re-
lated work. Section 5.3 presents the research questions and survey design. Section 5.4
presents the results and analysis of the survey. Section 5.5 discusses the results and
Section 5.6 summarizes and concludes the chapter.

5.2.1 Background
According Schweigert et al., there are approximately 40 AMMs proposed by academia
and industry consultants [193]. Many AMMs usually associate a number of Agile
practices with a maturity level [127, 193]. Practices are introduced gradually. As a
team or organization becomes more mature, more Agile practices are adopted [127].
An overview over three typical AMMs is provided in Table 5.1.
The idea of adding more Agile practices as a team or organizations becomes more
mature seems contradictory to current empirical studies that show that Agile practices
like TDD, pair-programming, and continuous integration are abandoned or declining in
their adoption rate [142, 212]. This raises a question regarding the suitability of AMMs
for industry, particularly when the AMMs do not provide rationales for the mapping
of Agile practices to maturity models. Critics of the AMMs indicate that the AMMs
are not fit for industry use [194] and that their recommendations are contradictory
[127, 157]. In this study, we aim to evaluate the suitability of AMMs by investigating
Table 5.1: Allocation of Agile practices to maturity levels in three AMMs.
Sidky et al. [202] Patel & Ramachandran [160] Nawrocki et al. [144]
Context Agile practice adoption based on a Agile practice adoption based on CMM(I) Adoption of XP based on other ma-
measurement index turity models
Level 1 On-site customer, collaborative — —
planning, coding standard
Level 2 Tracking progress, continuous de- tracking progress, on-site customer, planning Planning game, collaborating cus-
livery game, TDD tomer (on-site customer), user sto-
ries, metaphors
Level 3 F2F meeting, continuous integra- Refactoring, pair programming, continuous in- Pair programming, coding stan-
tion, self-organizing team tegration, TDD, coding standard, collective dard, collective ownership, contin-
ownership uous integration
Level 4 Daily meeting (stand up meeting), Self organizing team, 40 h week Simplicity (Simple design), on-site
user stories, frequent releases customer
Level 5 TDD, pair programming Focus on continuous improvement —
the usage and abandonment of Agile practices in industry through a survey and a series
of interviews.
5.2.2 Related Work

Murphy et al. [142] reported results of five annual surveys internal to Microsoft over
the course of six years. Their results show that practices like code reviews, metaphors,
and retrospective are increasing in their adoption. Meanwhile, certain practices like
unit testing, TDD and pair programming are decreasing in their adoption.
Kurapati et al. [123] performed a survey to identify commonly used Agile practices
at project and organization levels. Their results show that the most commonly used
practices both at project and organization levels include stand-up meeting, sprint and
iteration, collective ownership, and tracking progress. Less common practices both at
project and organization levels include simple design, TDD, pair-programming, and
planning game. One practice that is rarely practiced both at project and organization
levels is metaphor. It is also interesting to highlight that the use of metaphors reported
by Kurapati et al. turns out differently from Murphy et al.’s [142].
Kropp et al. [119] conducted a survey as part of Swiss Agile Study 2014. They
distinguished three types of practices: technical, collaborative, and advanced practices.
Technical practices include refactoring, TDD, and coding standards. Collaborative
practices include on-site customer, daily stand-up, and pair programming. Advanced
practices are kanban pull-system, acceptance TDD, and Behaviour Driven Develop-
ment (BDD). Their results show that more experienced practitioners implement con-
siderably more practices compared to less experienced ones. Furthermore, less expe-
rienced practitioners implement primarily technical practices, meanwhile more expe-
rienced ones implement more collaborative practices. It is worth noting in this study
metaphors is not included in the survey, unlike the previous two studies [123, 142].
Solinski and Petersen [212] surveyed Agile practice adoption scenarios over time
as practitioners transition from plan-driven development towards Agile. The survey
identified Agile adoption scenarios which include incremental adoption of practices,
big-bang adoption – where plan driven practices are discarded and replaced by Agile
practices, and complex tailored adoption processes. Their results also revealed that
practices like TDD and continuous integration are often abandoned. However, their
study did not focus on rationales for abandoning practices.
VersionOne also conducts an annual state of Agile survey. The 11th state of Agile
report indicates that XP as a methodology is declining [231]. When comparing the
10th [230] and 11th state of Agile report, it can be seen that practices like TDD, pair
programming, and automated acceptance test are declining in their use.
Ralph and Shportun’s case study revealed the abandonment of Scrum in distributed
teams [176]. One of the main factors associated with abandoning Scrum is the degra-
dation of Scrum practices. Three Scrum practices that were difficult to implement due
to distribution are daily stand-up meeting, tracking progress using burn-down chart,
and fixing sprint backlog.
To summarize, current research indicates that certain Agile practices are either
abandoned or decrease in their adoption rates. However, current surveys have not yet
focused on the rationales for abandoning Agile practices, or the time-frames from prac-
tice adoption to abandonment. Currently, we do not know how abandoning practices
may influence the perceived overall success of implementing Agile methods. In this
paper, we investigate why Agile practices are abandoned and whether or how this in-
fluences perceived success.

In this study, we aim to identify which Agile practices are being used and abandoned
in industry and the rationales for abandoning a practice to better understand practice
adoption and the relevance of Agile maturity models.
RQ1: What is the rate of usage of Agile practices?
RQ2: Which Agile practices have been abandoned?
RQ2.1: How long are practices in use before they are abandoned?
RQ2.2: What are the rationales for abandoning these practices?
RQ3: What is the perceived success rate of Agile practices implementation?

RQ3.1: Does the perceived success rate differ between respondents who retain
practices versus respondents who abandon practices?
RQ3.2: What are the used measures of success?
By “use” or “usage”, we mean that an Agile practice is used or was in use at some point
in time, while “abandoned” means that an Agile practice was used in the past, but is
no longer used. To answer the research questions above, we conducted a survey and a
series of follow-up interviews.
5.3.1 Survey
Sampling Strategy
We distributed the survey to personal contacts and well-established professional groups
in Agile software development on LinkedIn and Google Groups, i.e. convenience sam-
pling. Distributing surveys over professional groups is a known way to distribute sur-
veys as reported in [123, 212]. When using convenience sampling, which is a common
strategy in software engineering surveys, it is important to describe the sample [131].
Following the guidelines from Linåker et al. [131], we define our sample as follows:
• Target audience: Software industry practitioners who have experience in adopt-
ing Agile practices. Particularly, those who have experience in observing or
experiencing when a practice is adopted and/or abandoned.
• Unit of analysis: Agile practices which have been adopted and abandoned, their
rationales, and perceived success rates.
• Source of sampling: Professional groups or communities focused on Agile soft-
ware development. Personal contacts who are known to work with Agile soft-
ware development.
Survey Design
We followed the recommendations from Robson [182] in designing a self-administered
web-based survey. The survey was developed using the tool SoSci Survey (https:
//www.soscisurvey.de).
We included interactive sliders as a visual aid to allow respondents to indicate the
start and/or end of practice usage, see Figure 5.1. The survey design is adapted from
Solinski and Petersen [212], who also investigated time-frames of practice usage.
Part 1A Part 1B
Before 2012 2012 2013 2013 2014 2014 2014 2014 2015 2015 2015 2015 2016 In
2007 2008 2009 2010 2011
2006 H1 H2 H1 H2 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Use
Figure 5.1: Interactive sliders.
Since there is no commonly agreed set of Agile practices, we adopted the list used
by Solinski and Petersen [212], which includes 7 plan-driven practices and 14 Ag-
ile practices. We chose this list, because their survey is quite recent and comprises a
manageable number of practices. However, we separated Solinski and Petersen’s com-
bined practice “technical excellence” into its original sub-practices refactoring, simple
design, and coding standards.
The survey comprises five main parts. The detailed survey questions are available
in Appendix B of this thesis.
Part 1A. Agile practice adoption (RQ1). Respondents could indicate practice usage
as “used”, “never used”, or “don’t know”. See Figure 5.1 Part 1A (to the left). Defini-
tions of practices are available by hovering the mouse over the information icon. The
practices included in the survey and their definitions can be seen in Appendix A of the
thesis.
Part 1B. Start and end of Agile practice (RQ2 and RQ2.1). Using interactive sliders,
respondents could indicate the start- and stop-time for when a practice was in use as
shown in Figure 5.1 part 1B. The time-frame for the sliders is between <2006 and “Still
in Use”. When respondents indicated “never used” or “don’t know” in Part 1A, the
sliders are disabled. We used the interactive sliders to identify abandoned practices, so
we did not bias respondents by explicitly asking for abandoned practices. Respondents
could also leave optional comments or additional information regarding a practice.
Part 2. Perception and measures of success (RQ3). A Likert-type scale was used
to indicate the perceived degree of success of overall Agile practice adoption in their
organizations; from very unsuccessful (1) to very successful (5). Respondents could
also answer “don’t know”. Furthermore, we asked respondents to indicate how success
was measured.
Part 3. Limitations and Rationales (RQ2.2). We asked which challenges and lim-
itations respondents experienced during Agile practice adoption according to Part 1B
and, in particular, why practices were discontinued (if any).
Part 4. Contexts. We asked respondents to provide information about their personal
background and organizational context: (1) their role(s), (2) years of experience, (3)
number of team members involved in software development, (4) team-setting (collo-
cated or distributed), (5) how Agile practice adoption was decided (team-level or com-
pany), (6) industry domain(s), and (7) type(s) of software systems being developed
(classification are adopted from [74]).
Part 5. Contact. We also asked the respondents to provide their names and email-
addresses, for follow-up interviews or to receive a copy of the survey results.
Survey Pilot and Execution
The sliders made the survey more complex and increased the risk that questions are not
well understood. To mitigate these risks, we piloted the survey with five colleagues of
the authors and five industry practitioners with experience in Agile software develop-
ment.
Regarding the pilot, some industry practitioners felt that the definitions of some
Agile practices were too specific and might not be applicable in their contexts. To ad-
dress this issue, we reformulated the definitions. Two pilot respondents had difficulties
to move the sliders. We resolved this problem by adding instructions on how to use the
sliders. After addressing the feedback from the pilot, we deployed the survey, which
was open between March – July 2016.
5.3.2 Interviews
Interviewees Recruitment
Interviewees were recruited from the survey respondents who left contact information
for further inquiries. Twelve invitations were sent out and three confirmed for follow up
interviews. We then recruited eight additional interviewees through personal industry
contact and referrals. For the new recruits, we also asked them to fill in the survey
prior to the interview to maintain consistency and helped us to formulate interview
questions.
In total, we had 11 interviewees. Our interviewees represent a wide variety of
contexts. They came from various industry domains and geographic locations. More
details on the interviewees can be found in Table 5.2.
Interview Design
The goal of the interviews was to gather richer and better contextual information about
the use and abandonment of Agile practices. In the interviews, we used semi-structured
interviews. The interviews were done face-to-face whenever possible. Otherwise, the
interviews were done over the telephone or video call. Prior to the interview, we sent
each interviewee a summary of their answers from the survey. Each interview lasted
45–60 minutes and was recorded and transcribed. In the interviews, we inquired the
following:
• Interviewee’s roles and responsibilities, short description of the product being

developed.
• The interviewee’s survey answers were revisited and further clarified:
– Why did you mark (enumerate Agile practice marked as “never used”) as
never used? (RQ1).
– Why did you mark (enumerate Agile practice marked as “don’t know”) as
don’t know? (RQ1)
– Could you please elaborate the reasons for abandoning (enumerate Agile
practices which were no longer used from Part 1B)? (RQ2.2).
• Wrap-up. Inquire the interviewee’s impressions on the interview.

Table 5.2: List of interviewees and their contexts.

IDa Location Role Experience Team sizeb Market Domain Context overview
R11 Indonesia Project Manager 6 years 100 Market driven, inter- Insurance IT Department of a multinational Fortune
nal use 500 company. Adopted 13 practices except
pair-programming, TDD, and metaphors &
user stories
R14 Brazil Developer, Trainer, 3 years 20 Internal use Government (Court) IT Department from the Brazilian court
System architect of accounts. Adopted 15 practices ex-
cept pair programming and retrospective.
Abandoned on-site customer and tracking
progress.
R32 Canada Developer, Quality 6 years 13 Market driven, be- Independent Soft- Start-up company initiated in 2012.
Assurance, System spoke ware Vendor (ISV) Adopted 14 practices except for on-site
Analyst customer, simple design, and TDD. Aban-
doned pair programming and tracking
progress
R33 Sweden Scrum Master 6 years 6 Internal use Telecoms A small project team within a large multi-
national company. Adopted 15 prac-
tices except on-site customer and TDD.
Abandoned 13 practices except face-to-
face meeting and stand up meeting
R34 Indonesia CEO 3 years 33 Bespoke ISV Start-up company initiated in 2014.
Adopted 14 practices except TDD, col-
lective ownership, and metaphors & user
stories
R35 Ireland Scrum Master, De- 3 years 6 Bespoke, market ISV Start-up company initiated in 2012.
veloper driven, maintenance Adopted 14 practices, except TDD, coding
standard, and simple design
R36 Sweden Program Manager 23 years 1000+ Market driven Telecoms A solution development program in a large
multinational company. Adopted 16 prac-
tices except TDD
R37 Sweden Scrum Master 20 years 1000+ Market driven Telecoms A solution development program in a large
multinational company. Adopted 16 prac-
tices except TDD
R38 Sweden Scrum master, QA 7 years 70 Market driven ISV A project in a large multinational company.
Adopted 15 practices except on-site cus-
tomer and simple design
R39 USA Researcher, Devel- 3 years 6 Bespoke, market Research & develop- A project in a university to develop
oper driven ment, biomedical biomedical research support tool. Adopted
12 practices except pair programming,
tracking progress, stand up meeting,
metaphors & user stories, and TDD
R40 Finland CTO, Developer, 6 years 11 Market driven ISV A start-up company initiated in 2012.
Scrum master Adopted 10 practices, except on-site cus-
tomer, planning game, refactoring, retro-
spective, metaphors, TDD, and collective
ownership
a
Respondent ID according to the order they are received in the survey tool.
b
Reflects the size of software development team affected by the Agile implementation. Not overall company size.
5.3.3 Data Analysis

Analysis of survey qualitative data (RQ3). The survey included many open-ended ques-
tions, particularly in parts 2 and 3. For those questions, we used a qualitative coding
approach to analyze the data. First, we tabulated all responses for each relevant ques-
tion using a spreadsheet and f4analyse (https://www.audiotranskription.
de/english/f4-analyse). We then used open coding [187] to assign codes to
text fragments. For example, for the following response regarding used success mea-
sures: “Success can be measured by completion of tasks on time with high quality
and without any blockers”, we assigned two codes: time to deliver and product quality
as measures of success. The measures of success were then classified into product,
process and resource measures according to Fenton and Pfleeger [69].
Analysis of abandoned practices and the adoption time frame (RQ1 and RQ2). For
all practices that were indicated as “Used” (in Part 1A of the survey), we checked
the slider position for “practice end”. If this position did not indicate “In Use”, we
considered the practice as abandoned and calculated the timespan of use by means of
the slider positions for start and end of use, respectively.
Analysis of demographic data and success rates (RQ3). For a number of the context
questions, respondents could select more that one answer, e.g., roles, industry domains,
and types of software systems being developed. To analyze such data, we used the
SPSS feature “Multiple Response”. To cross-tabulate the demographic data and the
success rates, we used the “Crosstab” feature in SPSS. For example, we cross-tabulated
the success rates and the domain, see Figure 5.4.
Analysis of Interview data (RQ2.2). Similar to the analysis of survey qualitative
data, we used f4analyse tool to help with coding steps. First, we performed line-by-line
coding as an approach to open coding [187] on the interview transcripts. Open coding
was followed by focused coding to identify common themes from the data. The result
of focused coding can be seen, for example, in Table 5.5.

In reporting the validity threats, we follow the classifications suggested by Petersen
and Gencel [165].
Theoretical validity. It refers to pertains to the issue of capturing the construct
intended to be collected. Both the survey and the interviews, are retrospective. The
respondents may not remember precisely when an Agile practice was introduced. To
minimize the issue, we did not inquire exact months or dates for the start or the end of
a practice. We only refer to the year, half a year, or quarters. The slider design does
not support exact dates and only one start and one end time. To minimize the issue we
added comment text boxes next to the sliders to supply details.
Descriptive validity. It concerns with accuracy of capturing the reality. In this
study data collection was done through a survey and interviews. As researchers, we
cannot observe the reality and the responses we obtained are based on the respondents’
perception. We cannot fully eliminate this threat. For example, it is possible that a
practice had been abandoned prior to the respondent joined the company, he/she may
have been aware thus indicated that a practice is never used. However, the follow-
up interviews helped to better capture contexts that were otherwise missing from the
survey.
To improve thoroughness and trustworthiness of the survey, we reported as much

details as possible regarding the design and execution of the survey, following the cri-
teria described by Stavru [213]. A self-assessment on the thoroughness of our survey
using Stavru’s criteria and calculation procedure resulted in a score of 0.8 on a scale
0–1 (see Table A.I in the Appendix of this chapter). Stavru does not provide interpre-
tation of the scale. However, our score is higher than other Agile surveys examined by
Stavru in [213], where the highest score was 0.64. This indicates that we have provided
sufficient information to demonstrate the thoroughness of our survey [213].
Interpretative validity. It concerns with researchers’ bias in drawing conclusion.
This study primarily relies on qualitative data collected from a survey and from inter-
views. Researchers bias can affect the conclusions that are drawn. In analyzing the
data, the first author was responsible for the qualitative coding. To reduce bias, another
co-author validated the coding post-hoc after the first five interviews, to see if there
could be disagreements in the codes. Furthermore, in this study we did not attempt to
draw any inferences.
Generalizability. It refers to the extent that the results of the study is generalizable
to a larger population. In this study, both for the survey and the interviews, we used
convenience sampling. The selection of the respondents was non-purposeful and based
on willingness. Respondents have various roles and tasks in different organizational
contexts. However, some roles such as consultant and C-level managers are under-
represented. Furthermore, most of the respondents work in small organizations. We
cannot claim that our results are generalizable to a large population. However, the
demographic of our respondents includes a large variation of contexts that adds to the
richness of the data and minimizes the risk of confounding factors that could be present
due to a homogeneous context.

In total, 200 people started the survey, 70 completed the survey but only 43 answers
were valid, i.e. consistently answered part 1 – 4 of the survey. Out of 43 respondents,
32 of them completed part 1A and used the sliders from part 1B of the survey. The
remaining 11 respondents did not use the sliders (part 1B). Including the new intervie-
wee recruits, in total we have 51 respondents and 40 of them used the sliders. From 40
respondents who used the sliders, 22 retained all practices that were used, meanwhile
18 abandoned one or more practices.
The 51 respondents were primarily developers (20; 39.2%) followed by Scrum
Masters (15; 29.4%) and quality assurance specialists (13; 25.5%). Please note that
multiple roles could be selected. Further roles are system architect and department head
(8; 15.7% for each), project manager and department head (each 7; 13.7%), business
analyst, system analyst trainer, product owner, C-level managers (e.g., Chief Executive
Officer, Chief Technical Officer, etc.), and other roles (<6; <10%).
Regarding their level of experience in software development, 14 (27.4%) respon-
dents had more than 6 years of experience, 15 (29.4%) had 3–6 years of experience,
15 (29.4%) had 1–3 years of experience, and 7 (13.7%) had less than one year of ex-
perience. Most of the respondents (21; 52.5%) were part of a small organization with
less than 50 people involved with software development. Eleven (27.5%) were part
of organizations with 50–249 people, and 7 (17.5%) were part of organizations with
250–4999 people.
In terms of distribution, 28 (54.9%) of the respondents mentioned that their Agile
software development teams were collocated; 19 (45%) of them worked in a single
team and nine (17.6%) in multiple teams. The remaining 23 (45.1%) stated that their
Agile software development teams were distributed; 10 (19.6%) of them worked in a
single distributed team and 13 (25.5%) in multiple distributed teams.
Regarding application domains (multiple selections possible), most of the respon-
dents were from independent software vendors (17; 33.3%), followed by financial ser-
vices (15; 20%). Respondents also came from the following domains: research and
development (11; 21.6%), telecoms (12; 23.5%), medical (8; 15.7%), media and enter-
tainment (4; 7.8%), government (3; 5.9%), and manufacturing (1; 2%). For the types
of software systems that respondents develop and type of market, please refer to Figure
5.2a and 5.2b.
Computation Dominant Maintenance

6 (11.8%) 8 (15.7%)
Control Dominant
9 (17.6%)
In-house development Market Driven
Data Dominant 15 (29.4%) 30 (58.8%)
34 (66.7%)
Systems Software
21 (41.2%)
Bespoke
16 (31.4%)
(a) Type of systems (b) Type of markets
Figure 5.2: Respondents’ type of system and type of markets (multiple selection pos-
sible)
To complement the survey, we also conducted 11 interviews with industry practi-

tioners. The list of interviewees and their respective contexts are presented in Table
5.2.
5.4.1 Usage of Agile Practices (RQ1)

Figure 5.3 shows the rate of Agile practice usage. From Figure 5.3, we can see that out
of 51 respondents, face-to-face meeting was the most commonly used Agile practice
among our respondents (48 respondents), followed by tracking progress (47 respon-
dents). Other commonly used Agile practices by our respondents were: self-organizing
team, planning game, and retrospective. Practices like TDD (27 respondents) and pair-
programming (28 respondents) were less commonly used by our respondents.
50 48
47
46 46
45
43 43 43 43
42
39
40
37
35
34
33
30 28
27
24
20 18
15
13
9 9
10 8 8
7 7 7
6
5 5 5 5 5 5
3 3 3 3
2 2 2 2 2
1 1 1 1 1
0 0
0
Used Never Used Don't Know
Figure 5.3: Adoption of Agile Practices.
The rationales for never using certain Agile practices are summarized in Table 5.3.
From the interviews, we identified that respondents R11, R14, R32, R39, and R40,
marked some of the practices as “never used” or marked as “don’t know”, because
they were not adopted according to our provided definitions or were not adopted con-
sistently. For example, when inquired why stand up meeting was never used respon-
dent R39 mentioned that “ because of the word daily in the definition, we do not do
daily meeting”. Meanwhile respondent R11 mentioned the reason for marking “do not
know’ for collective ownership is because the project involved outsourced developers
and the level of collective ownership varies from the internal team to the outsource
team: “internal [team] is not a problem, but the outsource team has no collective own-
ership”. This indicates that the usage of Agile practices is not binary (used or not).
Often Agile practices are modified from how they are defined or implemented incon-
sistently.
Table 5.3: Rationales for never using certain Agile practices and the supporting quotes.
Rationale Practice Quotes (with Respondent’s ID)

Incompatibility with the domain or Short iteration Release of each sprint to end customer is not possible in case of regulatory
market of development development (R20)a.
On-site customer Our customers are 100M people (R13)a.
We are product company, it is a [software as a service] product over the Internet
(R40).
Some of our customers are not even in the province (R32).
Challenges in implementing a practice User stories The biggest challenge was conforming to the structure of developing user sto-
ries (R19)a.
TDD We do not have the patience to follow through with it. It is quite challenging
with a big ecosystem [of 26 products] like this (R36).
Metaphors We use user stories but not metaphors, metaphors are too obscure for most
people to grasp (R35).
Product complexity Simple design The product we were working on was extremely complex, we had a lot mov-
ing pieces and that was an unavoidable complexity the domain was complex
[..] the hardware aspect definitely have to do with it, hardware and firmware
development (R32).
TDD Our product is very explorative, [we are] creating new software, we rather
implement TDD next time (R40).
Legacy code Simple design We are left with a mess from the previous development team. We are adding
and maintaining the legacy we are left with to get the product to the market
(R35).
TDD We have a lot of [legacy] in our code, [it was not easy] for us just to jump into
[TDD] [because] the old code was not done in that way (R37).
Organization set up On-site customer we never interact with customers because were in the R&D department, the
department that interacts with the customers is called customer unit (R33).
Lack of resources Collective ownership We have a massive product and too few people, collective ownership is not
possible, we need specialists (R40).
Lack of management involvement or Retrospective Most of time management would trust the team to work, they [would not] be
enforcement picky and asking people to do retrospective and that kind of thing (R14)
TDD I [do not] know why we [do not] use TDD, We at [the company] just never use
TDD (R33).
PP Management [did not] talk about it at all. I [do not] think we ever discussed
whether to use pair programming or not (R14).
Lack of perceived value Planning game There is no need for a planning game because each developer is responsible
for a component of a feature. I [do not] think planning game helps in this case.
Just keep releases small and often (R40).
Refactoring It [does not] make sense to refactor because the components that you refactor
would be obsolete anyway in a very short time (R40).
Conflict with team’s culture Retrospective We want to foster the kind of culture where you are not keeping something for
a [sprint]. You just bring it up immediately (R40).
a
Respondent provided answer through the survey
From Table 5.3, we can see that not all practices are equally suitable to be imple-
mented in different contexts. Some practices may not be applicable given the regulation
in a certain domain, or presence of legacy code that was not developed using Agile ap-
proach. Furthermore, our survey reveals that just because the practices are used, it does
not necessarily mean their use is sustainable. More details on abandonment of Agile
practices are discussed the following subsection.
5.4.2 Abandonment of Agile Practices (RQ2)

As mentioned earlier, 18 of the respondents abandoned one or more Agile practices.
Each respondent abandoned at least one Agile practice. One of the 18 respondents
abandoned up to 13 Agile practices. All 17 Agile practices included in the survey
were abandoned at some point. From Table 5.4, we can see that face-to-face meeting
has the lowest abandonment ratio (0.05). Meanwhile, Tracking progress has the highest
abandonment ration (0.29) followed by planning game (0.2). This finding may indicate
that certain practices, like face-to-face meeting, are more easily retained than others.
Meanwhile, a practice like TDD may not be as popular, but once it was adopted it is
more likely to be retained, as we can see the abandonment ratio is quite low (0.11).
From Table 5.4, the number of respondents who abandoned individual Agile prac-
tices is relatively low when compared to the number of respondents who retained the
practices. This shows that most of the time each Agile practice is still in use.
Usage until Abandonment (RQ2.1)
Table 5.4 summarizes the periods of time that an Agile practice was in use. Practices
are most often abandoned within the first half year after their introduction. After 3 years
of use, the rate of abandonment drops significantly. Only tracking progress, planning
game, collective ownership and face-to-face meeting were abandoned after having been
in use for more than 3 years.
This finding may indicate that in some contexts, certain practices are not suitable to
be introduced in the first place, or introduced in the wrong order due to dependencies
on other Agile practices. Also, as the findings from subsection 5.4.1 shows that Agile
practices may be modified or implemented inconsistently, it is possible that the modi-
fications, or the lack there of, has undesired side effects that may present themselves at
various time periods. The rationales for abandoning Agile practices is presented in the
following subsection.
Rationales for Abandonment (RQ2.2)
Eight respondents provided rationales for abandoning the following practices: pair pro-
gramming, tracking progress, and on-site customer. Further respondents provided ra-
tionales for abandoning groups of practices. Most rationales were obtained for tracking
Table 5.4: Agile practices that have been abandoned and how long they had been in
use before abandonment.
Practices ≤6 mon ≤12 mon ≤ 24 mon ≤36 mon ≤48 mon 60+ mon Abandon Still In Total Abandon
Use Usagec Ratiod
a
Tracking progress 3 2 3 1 1 1 11 26 37 0.29
Planning game 3 1 1 1 1a,b 7 28 35 0.2
Retrospective 2 1 2 1 6 30 36 0.17
Time-boxing 2 1 1 1 5 30 35 0.14
Collective ownership 2 1 1 1a,b 5 25 30 0.17
Self organizing team 1 1 1 1 4 34 38 0.11
Pair programming 2 1 1 4 17 21 0.19
Simple design 2 2 4 22 26 0.15
Stand up meeting 2 1 3 32 35 0.08
Refactoring 2 1 3 26 29 0.1
Short iteration 2 1 3 29 32 0.09
Metaphors and stories 1 1 1 3 24 27 0.11
Continuous Integration 1 2 3 30 33 0.09
TDD 1 1 2 15 17 0.11
F2F Meeting 1 1 2 37 39 0.05
On-site customer 2 2 22 24 0.08
Coding standard 1 1 2 32 33 0.06
23 13 12 15 2 2
a b
Respondents in financial domains. Response from the same respondent.
c d
Total usage based on 40 respondents who used the sliders. Ratio = Abandon/Total Usage
progress. From Table 5.5, generally Agile practices were abandoned due to emerging
challenges or diminishing perceived values.
The statements from R14 and R38 in the discontinuation of tracking progress indi-
cate that Agile practices dependencies are not well understood. In the case of R14,
tracking progress was introduced before sprint planning was established. Because
sprint planning was not done, new tasks could be added throughout the week and track-
ing progress became ineffective, as respondents R14 explained: “It seems like we were
walking backwards. We were working towards the end of the week and things just got
worse. Because somebody would suddenly add a workload to the sprint.” Meanwhile,
in the case of R38, tracking progress was introduced before the team members de-
velop better product knowledge. This shows that there could be prerequisites before
introducing certain Agile practice. The prerequisites could be other Agile practices or
acquiring product or project-related knowledge.
From Table 5.5, one of the more interesting rationale for abandoning one or more
Agile practices is the influence of a person, as reported by R14, R32, and R33. Respon-
dent R14 mentioned that on-site customer was adopted for only two months because
the person in charge had to leave the company. This individual was crucial to make
on-site customer worked smoothly because the person can bridge between the techni-
cal team and the end users (court officers): “He was both an engineer and a lawyer. So
he could very easily talk to the business people and to us”. Meanwhile, R33 indicated
Table 5.5: Rationales to abandon Agile Practices with the supporting quotes.
Rationale Practice Quote (with Respondent’s ID)

Poor estimation and team Tracking progress Due to bad estimation and dependency on other teams we are unable to track
dependency progress by burn-down chart (R32)a.
Lack of product knowledge [The team members] complain that we [do not] have the product knowledge,
how do we estimate it if we [do not] know the complete technicalities (R38).
Team member discomfort Pair programming People were uncomfortable and people did not really want to engage in that
(R32).
Lack of engagement Tracking progress Half of the were tracking progress and they other half [were not], management
did not really care (R14).
Conflict with other Agile Pair programming The idea of sustainable pace, [..] we are only expected to be at the office at
values certain core hours [..]. I would be one of the people showing up around 9.30-
16.30 [..]. so if I want to pair program with one of the latecomers, it would
only really work from 13-15 (R32).
Influence of a person Not specific It was because one person was quite very opinionated, the person thought why
do all these things, it’s a waste of time (R33)b.
On-site customer The guy [who initiated on-site customers] went on vacation and he did not
come back (R14).
Tracking progress The new product owner did not want/care for [statistics], and the team did not
demand them (R32).
Lack of perceived values Tracking progress As we do product development of a rather mature product, the tracking of
progress was not all that valuable. Stuff at the top of the backlog has most
value. Stuff lower has a lower value, and will be released later. No real forecast
of this was needed (R21)a.
Tracking progress We just try to push things to production all the time (R40).
Tracking progress The team did not feel the need for it (R30)a.
Not specific The part that can be handled by agile is finished. Other part cannot use agile
(R28).a
Dependency on other prac- Tracking progress We tried to do tracking progress but sprint planning was not done [yet] (R14).
tice
a b
Respondent provided answer through the survey. 13 out of 15 Agile practices were discontinued.
that the practices were adopted for up to three years until they are abandoned: “They’ve
been practicing Scrum since 2012. Suddenly in 2015 they stopped completely [..]. They
just dropped everything and they only do stand up meeting [..] ”. This indicates the
influence of an individual can affect the abandonment of Agile practices, but also how
long they were adopted until abandonment.
From Table 5.5, we can see that there could be more than one cause to abandon an
Agile practice. For example, we identified multiple reasons for abandoning tracking
progress. One of the more common reasons is lack of perceived values. To abandon
tracking progress due to decrease of perceived value seems counter intuitive because
the need for tracking progress would increase as the product grows and more tasks
are associated to delivering the product. It is however possible that the practitioners
did not completely abandon tracking of progress all together, but abandoned tracking
progress according to the definition in the survey. Respondent R14, R23, R33, and R38
indicated in the survey and interviews that they use Kanban board to replace burn up
or burn down charts as a means of tracking progress.
The results from survey and follow up interviews indicate that there could be mul-
tiple factors that can contribute to abandoning an Agile practice. Engagement, knowl-
edge, and dependencies between development teams can contribute to the abandonment
of one or more Agile practices.
5.4.3 Perceived Success of Agile Practice Adoption (RQ3)

We analyzed the success rate of the Agile practice adoption with 50 responses, one
of the respondents (out of 51) indicated “don’t know” pertaining to the success rate.
Figure 5.4 shows the success rates of Agile practice adoption by industry domains.
Overall (N=50) 1 (2%) 8 (16%) 30 (60%) 11 (22%)
ISV (N=16) 3 (18.8%) 6 (37.5%) 7 (43.8%)
Financial (N=15) 1 (6.7%) 3 (20%) 10 (66.7%) 1 (6.7%)
Media (N=4) 1 (25%) 2 (50%) 1 (25%)
Medical (N=8) 1 (12.5%) 5 (62.5%) 2 (25%)
Manufacturing (N=1) 1 (100%)
Telecom (N=12) 1 (8.3%) 11 (91.7%)
R&D (N=11) 1 (9.1%) 6 (54.5%) 4 (36.4%)
Government (N=3) 2 (66.7%) 1 (33.3%)
Other (N=4) 1 (25%) 3 (75%)
-20% 0% 20% 40% 60% 80% 100%

Unsuccessful Neutral Successful Very Successful
Figure 5.4: Success Rates of Agile Adoption
From Figure 5.4, we can see that the adoption of Agile practices was generally per-
ceived as being successful. Most of the respondents (30; 60%) perceived the adoption
of Agile practices as successful and 11 (22%) as very successful. Only one respon-
dent (2.8%) perceived the adoption of Agile practices in his/her organization to be
unsuccessful. No respondent answered “Very unsuccessful”. There were only minor
differences between domains.
During the follow up interview, R11 who indicated unsuccessful adoption of Agile
practices mentioned that the issue was with the company policy of providing documen-
tations in the end of every sprint:“if you want to be effective, with the small chunks of
deliverables, there are more effort because the amount of procedure is the still the same
as the big one. Agile implementation somehow is “heavier” on the procedure side. For
every deliverable we need to provide documents like technical documentation, deploy-

ment guide, training material, [user acceptance test] sign off”. Respondent R11 also
felt that the kind of product they were developing did not fit Agile: “ You need 6 months
to develop the core engine. I cannot split a function in two releases, because it will be
useless for the user. We have heavy rule engine and workflow. For this type of project,
Agile does not work”.
The respondents who perceived Agile practice adoption as very successful or suc-
cessful (43 respondents) were primarily from small and medium sized organizations
(25 and 15 respondents respectively out of the 43 respondents). Since the group of re-
spondents for large companies was small, we did not perform any inferential statistical
analysis.
Success Rates: Retained vs. Abandoned Practices (RQ3.1)

We also compared the success rates of 40 respondents who retained all adopted Agile
practices and respondents who abandoned one or more Agile practices. From Fig-
ure 5.5, we can see that the perceived success of Agile practices was similar in both
groups. This result indicates that an abandoning of one or more Agile practices might
be required to achieve or sustain an overall successful Agile adoption.
Retain 1 (4.5%) 3 (13.6%) 10 (45.5%) 8 (36.4%)
(N=22)
Abandon 3 (16.7%) 12 (66.7%) 3 (16.7%)

(N=18)
-10% 0% 10% 20% 30% 40% 50% 60% 70% 80%
Figure 5.5: Success Rates: Retained versus Abandoned Practices
However, it is also important to remember that not all respondents adopted the
same set of Agile practices. Those who achieved successful or very successful Agile
adoption by retaining all Agile practices may have found the more suitable set of Agile
practices or have successfully found an optimal way to tailor the Agile practices. We
however do not claim that those who abandoned practices were actually less successful
in selecting the suitable of Agile practices.
Measures of Success (RQ3.2)

We collected measures of success from 35 respondents and classified them into product,
process, and resource measures [69]. Table 5.6 summarizes the measures that were
reported the respondents and the number of respondents that reported them.
Among the product measures, “product quality” and “customer satisfaction” were
named most frequently (12 and 9 times, respectively). Among the process measures,
“time to deliver” was named most frequently (16 times). “Team spirit (happiness)” was
the most frequently named resource measure.
Respondents considered a large diversity of indicators as being success-relevant,
including measures from all three categories. Table 5.6 lists 16 unique “process” mea-
sures, 11 unique “resources” measures, and 8 unique “product” measures. This result
shows that success of Agile practice adoption can be perceived in many different ways.
Looking at the number of different measures and the number of respondents who
contributed them, we can see that our respondents put much focus on how well a “pro-
cess” is executed and on the quality of the “product”. On a more detailed level, the
respondents focused on product quality, customer satisfaction, and time to deliver, and
good team spirit. This result is in line with the overall goals of the Agile manifesto [15]
and the principles behind it.
Table 5.6: Measures of Success.
Category Measure No. of respondents

Product Product quality 12
Customer satisfaction 9
Number of defects/bugs 2
Number of relevant working products/ deliverables 2
Other: Number of newly acquired users, code quality, code change quality, business value 1 each
Process Time to deliver 16
Cost 3
Delivery frequency/cadence 3
Lead time 2
Ease to track progress (transparency) 2
Other: Time to resolve defects, time to implement change, correct use of development pro- 1 each
cess, effective use of Agile practices, number of development issues, amount of maintenance
work, number of story points, number of released new features, number of met sprint goals,
non ad-hoc development process, velocity
Resource Team spirit (happiness) 6
Budget conformance 2
Productivity 2
team autonomy 3
Other: Collaboration, stress level, team engagement, ownership, mutual understanding, con- 1 each
tinuous learning, collective ownership
We can see that the respondents reported measures at different levels of granularity.
For instance, most respondents referred to “product quality” or “customer satisfaction”
as measures for quality without going into detail about how those were measured. Few
respondents named actual specific measures, like “number of defects/bugs” or “number
of met sprint goals”.
When looking at the respondents’ experience and roles, we could not identify any
specific patterns regarding the measures they provided. Respondents with more tech-
5.5 Discussion 159
nical roles, e.g., developers or testers as well as those with managerial roles provided
both specific and generic measures.
5.5 Discussion
In this study, we conducted a survey and 11 interviews on Agile practices adoption
and abandonment. To guide the discussion, we reflect our findings and compare them
against known recommendations from Agile maturity models (AMMs).
Our study indicates that not all Agile practices are applicable in different contexts.
Agile practices are not used due to incompatibility the development context, chal-
lenges, or lack of management enforcement. AMMs typically recommend to gradually
add more and more Agile practices (see Table 5.1) without considerations on whether
the practices are actually applicable. For example, 24 of our respondents never used
TDD, but two out three AMMs that we exemplified in this paper recommend that TDD
is to be introduced. This raises the question regarding the suitability of AMM in indus-
try.
The result of our survey indicates that not all Agile practices are sustainable. Eigh-
teen of the respondents have abandoned one or more Agile practices. Our survey also
shows that Agile practices were more frequently abandoned within the first six months
after their adoption. Meanwhile, some Agile practices, like continuous integration,
planning game, and collective ownership were adopted for extended period of time.
This finding complements the findings of a previous study by Solinski and Petersen
[212]. The AMMs indicate that Agile practices are to be gradually added. However, in
certain contexts it is not always possible to sustain a practice, as indicated by a number
of our respondents. The question that needs to be raised when adopting an AMM is,
if a practice is abandoned, how would this affect the practices that are to be adopted
next? and how would this affect the overall maturity? The findings from our study add
more questions to the suitability of the suggestions in the AMMs.
One of the rationales for abandoning Agile practices was the influence of a person.
For respondent R14, on-site customer was introduced by the IT manager, the person’s
skills and abilities were so crucial that upon his departure from the organization, the
practice had to cease. Meanwhile, respondent R33 the influence of one very opinion-
ated individual convinced the rest of the team to stop using 13 Agile practice. The case
reported by R14 and R33 shows the presence of a “maverick” [200], a highly com-
petent and influential individual that can influence the introduction and abandonment
of Agile practices. The AMMs generally suggested that Agile practices are to be in-
troduced in certain orders, and do not provide details on how these practices are to be
introduced or sustained. This indicates that the AMMs have not considered the social
aspects and uniqueness of different software development teams.
The results of our survey and interviews also indicates that an Agile practice could
be abandoned because it is introduced too soon. For example, tracking progress was
abandoned because sprint planning was not yet used (as reported by R14). This indi-
cates that there might be prerequisites to introducing or adding Agile practices, which
the practitioners may yet to be aware of. In such cases, it would be preferred if practi-
tioners can turn to the AMM. However, when we look at the examples of the AMMs in
Table 5.1, we can see that each AMM have its own suggestions as to which practices
are to be introduced at which maturity level. For example Patel and Ramachandran
[160] suggested that tracking progress to be introduced first before any other practices,
such suggestion certainly would not work in favor of R14. However, Sidky et al. [202]
suggest the opposite, which could have provided a better guideline for R14. This indi-
cates there could be a need for guidelines. However, instead of suggesting to gradually
introduce Agile practices in fixed orders, like the AMMs, more research can be directed
to evaluate which Agile practices need to be introduce first, or later, given the contexts
of the software teams or organizations.
The result of our survey shows that practitioners, both who retained and abandoned
one or more Agile practice perceive their Agile practice implementation to be suc-
cessful. When we looked into the measures for success most respondents indicated
quality and time to deliver as measures for success. This indicates that discontinuing
one or more Agile practices may be necessary, to minimize challenges (e.g., reduc-
ing team members discomfort as mentioned by R32) and ensure that the benefits from
implementing Agile practices in general can be achieved. However, when we looked
at the AMMs [144, 160, 202] they generally suggest that Agile practices are to be
continuously added. The AMMs do not provide suggestions should problems arose
from introducing an Agile practice. This indicates the AMMs lack of consideration
of the different situations and contexts in different software development team. More
research needs to be directed to better understand the impacts of gradually adding, or
abandoning, Agile practices.
The results of our survey suggest that retaining or abandoning Agile practices can
lead to a successful Agile adoption. This shows that Agile adoption is not as straight-
forward and gradual as suggested by the AMMs [144, 160, 202]. Practitioners may
need to abandoned, or very rarely pause, the implementation of one or more Agile
practices. This indicates that practitioners are constantly assessing whether Agile prac-
tices are delivering the values they expected. Sidky et al. [202] included a step to assess
whether to continue or discontinue the whole Agile transformation process, but not at
practice level. Practitioners might need support to systematically evaluate their state of
5.6 Conclusion 161
Agile adoption so that decisions to add, modify, or discontinue a practice is based on a

rigorous and traceable process.
Implications towards Agile adoption guidelines. We noticed differences between
the recommendations in AMMs and the results of our survey. At the same time, our
survey also indicates the need for Agile adoption guidelines. Such guidelines need to
take into account that Agile practices might not be sustainable and that there might
be dependencies between Agile practices, as indicated by one our respondents, that
suggests certain orders or combinations of adoptions. Furthermore, the situations and
operating environment of software organizations may change [151]. The guidelines
need to provide an appraisal means for practitioners on the benefits and limitations
from adopting Agile practices, given the changing situations.
Implication towards Agile research. The results of our survey shares similarity to
those of Kurapati et al. [123]. However, we also observed some differences, particu-
larly pertaining to the adoption rate of planning game. The respondents in our survey
indicate that planning game is a commonly used practice (47 out of 51 respondents),
but Kurapati et al. reported the opposite. We observed that Kurapati et al. defined the
practices slightly different. Their definition of planning game includes the presence an
on-site customer. In our survey, we separated planning game from on-site customer.
To be able to synthesize existing evidence regarding Agile practice adoption, there is a
need for commonly agreed and consistent definitions of Agile practices.
The respondents in our survey indicate that TDD and pair programming are less
commonly used practices. This result corroborates with past surveys such as [123] and
[142]. TDD and pair programming are also less frequently abandoned. This observa-
tion is rather interesting, because a tertiary literature study in Agile shows that TDD
and pair programming is highly studied [152]. There are also many reports on their
benefits and limitations to name a few: [33, 61]. This raised the question on whether
knowing better the benefits and limitations of different Agile practices can help practi-
tioners to make better decisions on whether to introduce a practice. Therefore once the
decision is made to adopt such practices it is based on an informed decision, thus the
practices are less likely to be abandoned.
5.6 Conclusion
We conducted a survey on Agile practices with a particular focus on when adopted
practices were abandoned. We received 51 valid answers, 40 provided detailed start
and end period for the practices. We also conducted 11 follow up interviews with the
survey respondents. In the following, we revisit our research questions by summarizing
answers:
RQ1. What is the rate of adoption of Agile practices? The rate of adoption of
each practice can be seen in Figure 5.3. Commonly adopted practices by our respon-
dents were face-to-face meeting, tracking progress, and planning game. Comparably
less commonly adopted practices by our respondents were TDD and pair programming.
RQ2. Which Agile practices have been abandoned? All 17 Agile practices in-
cluded in this survey have been abandoned at some point (see Table 5.4). Consistent
with the answer to the previous research question, the more commonly used practices,
particularly tracking progress and planning game, also had high abandonment ratio.
The rationales for abandoning Agile practices include, lack of perceived values, influ-
ence of a person, and team member discomfort. Agile practices were used between
6 – 60 months until they were abandoned. Most of our respondents abandoned prac-
tices within the first half year of the introduction. Agile practices are less likely to be
abandoned by our survey respondents after three years (36 months) of use.
RQ3. What is the perceived success rate of Agile practices implementation?
The adoption of Agile practices was perceived as being successful or very successful.
Only one respondent perceived the Agile adoption as unsuccessful and none as very
unsuccessful. The respondents used a large variety of measures of success. The fol-
lowing measures were used by the majority of respondents: product quality, customer
satisfaction, and time to deliver. Furthermore, our survey indicates no differences in
the perceptions of success between respondents who abandoned practices and those
who retained them. This result indicates that some teams or organization needed to
abandon some practices to achieve or maintain an overall successful adoption of Agile
methodologies.
Future work: For future work, we suggest the following avenues of research:
(1) examine how different Agile practices contribute to maturity (2) better understand
the impact of gradually adding, or abandoning Agile practices, and (3) developing a
common definition of Agile practices to ease aggregation of evidence.
Appendix
Calculating thoroughness score. We summed up the weights for every criterion that
was fulfilled by this survey (total score). Then, we divided the obtained total score by
the total weight of all criteria. For more details on survey thorough assessment, see
[213].
5.6 Conclusion 163
Table A.I: Survey Thoroughness assessment based on [213]
Criteria Weight Score Criteria Weight Score

Objectives 1 1 Questionnaire evaluation 3 3
Sponsorship 1 0 Questionnaire 3 3
Survey method 4 4 Media 1 1
Conceptual model 4 4 Execution time 1 1
Target population 4 4 Response burden 1 0
Sampling frame 5 5 Follow-up procedures 2 0
Sampling method 5 5 Responses 3 3
Sample size 5 5 Response rate 5 5
Data collection method 3 3 Assessment of trustworthiness 5 0
Questionnaire design 4 4 Discussions of validity threats 3 3
Provisions for securing trustworthiness 3 3
Total Weight: 66 Total Score: 57
Chapter 6
Strategies to Introduce Agile

Practices: Comparing Agile
Maturity Models with
Practitioners’ Experience
Indira Nurdiani, Jürgen Börstler, Samuel A. Fricker, Panagiota Chatzipetrou,

“Strategies to Introduce Agile Practices: Comparing Agile Maturity Mod-
els with Practitioners’ Experience”, Submitted to journal of Empirical
Software Engineering, March 2018.
6.1 Introduction
Agile software development has become increasingly popular over the past years. Ag-
ile methods are perceived to address challenges caused by the rapid change in the mar-
ket while reducing time to market and development cost [1]. As the popularity of Agile
method is growing, the need for Agile adoption guidance also increases [202]. Over
the past years, several Agile Maturity Models (AMMs) have been proposed to offer
guidelines in Agile adoption [194].
166 Strategies to Introduce Agile Practices
AMMs generally include maturity levels that are similar to CMMI or SPICE [193],
or are based on agile values inspired by the agile manifesto [192]. Many AMMs asso-
ciate practices with maturity levels indicating that there is a preferred order for intro-
ducing Agile practices.
However, AMMs are faced with criticisms. A number of evaluations of AMMs
show that they are not properly validated [127] and not suited for industry use [194].
Current evaluations of AMMs are based on criteria developed by the authors of such
evaluations [127, 193], or on criteria based on CMMI [157]. There are no studies that
evaluate AMMs against industry practice. The relevance of the AMMs in industry is
not yet examined.
In this study, we particularly focused on evaluating AMMs that have explicit map-
pings of maturity levels and Agile practices. To evaluate the AMMs against practice,
we conducted a web-based survey and 11 follow-up interviews with industry practi-
tioners with experience in Agile approaches. In the survey, we particularly focused on
the order of agile practice introduction in industry.
The contributions of this study are as follows:
1. Identification and comparison of strategies suggested by AMMs pertaining to the

order of Agile practice introduction.
2. Identification of strategies used by practitioners to introduce Agile practices over

time, including the rationales and impacts for doing so.
3. An evaluation of how well current AMMs align with industry practice.
The remainder of the chapter is structured as follows: Section 6.2 discusses related
work. Section 6.3 describes the research method used. Section 6.4 presents the results,
which are discussed in detail in Section 6.5. Section 6.6 summarizes and concludes the
chapter.
6.2 Related Work

6.2.1 Surveys on Agile practices
Kurapati et al. [123] conducted a survey to identify commonly used Agile practices in
industry. Their survey shows that some practices, like metaphors, TDD and pair pro-
gramming, are less popular than others, like stand-up meeting, sprints, and collective
ownership. They also found that practitioners used various combinations or subsets of
Agile practices.
6.2 Related Work 167
Murphy et al. [142] conducted five annual surveys within Microsoft and reported
that practices like code reviews, metaphors, and retrospective are increasing in their
use. Meanwhile, practices like unit testing, TDD and pair programming are decreasing
in their use.
Solinski and Petersen [212] investigated Agile practice adoption scenarios over
time when transitioning from plan-driven to Agile processes and identified four com-
mon adoption scenarios: (1) transformed from plan-driven to agile, (2) enrich plan-
driven process with Agile practices, (3) development process are shaped overtime with
combination of both Agile and plan-driven, and (4) plan-driven practices are added
followed by addition of Agile practices. Their survey is similar to ours, however, our
survey only focuses on the introduction of Agile practices over time.
Unlike other surveys, our survey and Solinski and Petersen’s (2016) are interested
not only in the usage of Agile practices, but also when they were adopted and/or dis-
continued. We built on Solinski and Petersen’s work, since we also aimed at identifying
different strategies in Agile practice introduction. In addition to their work, we com-
plemented our survey with follow-up interviews to better understand the rationales of
introducing different Agile practices over time [212].
There is no general consensus on the number or level of granularity of Agile prac-
tices. Kurapati et al. [123], for example, included 25 Agile practices, while Kropp
et al. [119] included 23. In our survey, we adopted the list from Solinski and Petersen
[212], since they already based their list of practices on earlier works. The list of Ag-
ile practices included in our survey, together with practice definitions can be found in
Appendix A of this thesis.
6.2.2 Agile Maturity Models (AMMs)
Maturity models are proposed to help software organizations to maintain competitive

advantage, reduce cost, improve quality, and reduce time to market [51]. AMMs are
proposed to provide alternatives to the traditional process assessment and improvement
models like CMMI or SPICE [127]. Existing evaluations of AMMs indicate that over
40 AMMs have been proposed by industry consultants, Agile “gurus”, and academics
[193].
In general, AMMs have hierarchical maturity levels [127, 193]. Some AMMs map
each maturity level to a set of practices, e.g., [133, 144, 202], whereas others provide
suggestions on aspects like team distribution to focus on each maturity level, for exam-
ple Ambler [7].
6.2.3 Evaluations of AMMs

AMMs have been evaluated in a number of studies, e.g., [127, 157, 194].
Schweigert et al. [194] analyzed AMMs against the requirements in ISO/IEC 15504
Part 2. Their evaluation shows that it is almost impossible to make direct mappings
between the maturity levels of AMMs and SPICE capability levels. Schweigert et al.
suggest that AMMs are not fit for use in industry, because the AMMs do not address
capability in the sense that is described in capability maturity models, rather they deal
with Agile practice implementation.
Leppänen [127] analyzed eight AMMs against the following six criteria: (1) target
domains, (2) purpose of the model, (3) conceptual and theoretical bases, (4) approaches
and principles used to construct the models, (5) structure of the models, and (6) use and
validations. Their results show that AMMs were not developed based on underlying
theories and were not properly validated. Leppänen also mentioned that the AMMs
do not agree on which practices are included in which maturity levels, which indicates
that AMMs do not have common views of agility or maturity.
Ozcan-Top and Demirörs [157] assessed eight AMMs in a case study with industry
practitioner using the following six criteria: (1) fitness for purpose, (2) completeness,
(3) definition of Agile levels, (4) objectivity, and (5) correctness, and (6) consistency.
Their results show that the maturity level of an Agile team can vary depending on
which AMM is used. Furthermore, most of the AMMs only partially achieved each of
the evaluation criteria, which indicate that the AMMs are incomplete and insufficient
to aid practitioners to improve agile maturity.
Motivation for our study. Although AMMs have been evaluated in several studies,
there are no studies that compare AMMs with how Agile practices are adopted or
introduced in industry. It is therefore unclear to which extent they reflect actual industry
adoption and their usefulness for industry use. We therefore want to investigate in more
detail how the strategies suggested by AMMs compare to the strategies used in industry.

The study aims to compare suggestion provided by AMMs with how Agile practices
are introduced in practice. The following research questions were formulated:
• RQ1: What strategies do AMMs suggest for introducing Agile practices?
• RQ2: What strategies do practitioners use for introducing Agile practices over
time?
• RQ3: How do the strategies suggested by AMMs compare to the strategies used
in industry?
The present study has two main parts: (1) State-of-art: literature survey on AMMs
and (2) State-of-practice: empirical study including a survey and follow-up interviews.
Figure 6.1 provides an overview of the methods used in this study.
Literature Survey
State-of-art (RQ1)
Empirical Studies Compare state-of-

(RQ2) art and state-of-
practice (RQ3)
State-of-practice Survey
Interviews
Figure 6.1: Overview of research methods.
To gain a better understanding of the state-of-art, we identified existing AMMs and

their mappings of Agile practices to maturity levels, since this indicates an order for
introducing Agile practices.
To get an overview of the strategies for Agile adoption in industry, we performed
a web-based survey. Follow-up interviews were conducted with 11 of the survey re-
spondents to better understand the rationales and impacts of the strategies. We then
compared the results from the literature survey and empirical study to identify com-
monalities and differences.
6.3.1 Literature Survey: Agile Maturity Models

To identify AMMs, we covered peer-reviewed and grey literature. We did not perform
a full-scale systematic literature review, because our intention was to get an overview
over AMMs, and not to synthesize evidence pertaining to AMMs.
We conducted a literature search in four databases: (1) Engineering Village (In-
spec), (2) IEEE Xplore, (3) Web of Science, and (4) Scopus using the following search
string:
(agile OR agility OR “extreme programming” OR xp OR Scrum ) and

(maturity OR capability)
The search was done on keywords, title and abstract and limited to the field of software
engineering. Figure 6.2 depicts the steps that were taken to identify AMMs.
Papers removed Papers remained
Initial database 96
search
Title and 29 duplicates

Abstract 2 proceeding covers 35
Screening 30 non-SE papers
Fulltext 25 irrelevant papers 10

Screening
40 AMMs:
6 secondary • 31 web materials, e.g.,
studies blogs, consultancy sales
AMM Search materials, presentation slides
4 primary studies • 9 peer-reviewed papers
2 duplicates 26 AMMs:
AMM Screening 16 AMMs not available • 12 Type 1 AMMs
• 14 Type 2 AMMs
Figure 6.2: Literature search steps.
The initial search resulted in 96 papers. After title and abstract screening 35 pa-
pers were left. All papers not relevant to software development processes and maturity
models were deleted. After full text screening we were left with 10 paper. From these
10 papers, four were primary studies describing individual AMMs and six were sec-
ondary studies; five evaluating existing AMMs and one on Agile software development
and maturity models. Through the references of the secondary studies, we identified
references to 40 AMMs in total. After removing duplicates and AMMs that no longer
had accessible information, we were left with 26 AMMs that could be grouped into
two general categories:
• Type 1 – AMMs with explicit mapping between Agile practices and maturity
level. We found 12 Type 1 AMMs.
• Type 2 – AMMs that did not provide mapping between Agile practices and ma-
turity levels but indicated different maturity levels or steps to improve maturity.
We found 14 Type 2 AMMs.
In this study, we were particularly interested in Type 1 AMMs with explicit map-
pings between maturity levels and Agile practices. From the Type 1 AMMs, we ex-
tracted the following information: (1) authors’ names, (2) title, (3) year, (4) maturity
levels, (5) Agile practices associated with each maturity level, (6) whether the AMM
was developed based on empirical study, and (7) how the AMM was validated. The
remaining Type 2 AMMs are summarized in Appendix 1 of this chapter. They are not
included in our detailed analysis, since they lacked sufficient information. They are
briefly discussed in the results section, though (see Subsection 6.4.1).
After data extraction, we observed that the Type 1 AMMs often include practices
that are not commonly associated with Agile methods, e.g., external consultancy, in-
tentional architecture, and managing highly distributed team. We cross-referenced the
practices included in the AMMs with the list of practices mentioned in [164] and [239],
and excluded practices that are not associated with Agile software development.
6.3.2 Survey
The survey was developed using the tool SoSci Survey1 with five main parts.Detailed
questions from the survey are available in Appendix B of this thesis. JavaScript and
PHP code was added to provide interactive sliders to allow respondents to enter start
and end for each Agile practice (see Figure 6.3).
Part 1. Agile practice start and end. We asked the respondents to indicate whether
the practice were “used”, “never used” or “don’t know” (Part 1A). Respondents could
view the definitions of each practice by hovering over the information thumbnails.
Then the respondents could use the interactive sliders to indicate the start and/or end
of each practice (Part 1B). Figure 6.3 shows the visualization of the interactive sliders,
which we adapted from a previous survey done by Solinski and Petersen [212]. We
also adapted the list of Agile practices used by Solinski and Petersen [212]. However,
we modified the list of practice by separating “technical excellence” into its original
sub-practices, they are, refactoring, simple design, and coding standards.
Part 2. Perception of success. We asked the respondents to indicate the perceived
degree of success of Agile practice adoption in a Likert-type scale (very unsuccessful
– unsuccessful – neutral – successful – very successful). The respondents were also
asked to provide the measures of success (free text).
Part 3. Benefits and Limitation of Agile practice adoption. We asked the respon-
dents to provide the perceived benefits and limitations of Agile adoption according to
question in Part 1B. We also asked respondents for the rationales for discontinuing one
or more Agile practices as indicated in Part 1B.
Part 4. Contextual Information. We asked the respondents for contextual infor-
mation about themselves and the organization where the experience described in Part 1
1 https://www.soscisurvey.de
Part 1A Part 1B
Before 2012 2012 2013 2013 2014 2014 2014 2014 2015 2015 2015 2015 2016 In
2007 2008 2009 2010 2011
2006 H1 H2 H1 H2 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Use
Figure 6.3: Interactive sliders in the survey.
took place. We inquired the following information: (1) their role(s), (2) years of experi-
ence, (3) number of team members involved in software development, (4) team-setting
(collocated or distributed), (5) how Agile practice adoption was decided (team-level or
company), (6) industry domain(s), and (7) type(s) of software systems being developed
(classification are adopted from Forward and Lethbridge [74]).
Part 5. Contact. We inquired the respondents if they wished to be contacted for
follow-up interview and their contact information.
The survey was piloted with five industry contacts and five of the authors’ col-
leagues with experience in Agile software development. From the pilot we received
feedback pertaining to: (1) Agile practices definitions were too specific and (2) dif-
ficulty to move the sliders. We then modified the definitions of Agile practice we
included in the survey to be broader. To solve the issue with the sliders, we added an
instruction on how to use the sliders. After the improvement suggestions were imple-
mented, we deployed the survey between March–July 2016.
Similar to Solinski and Petersen [212], we used convenience sampling. We dis-
tributed the survey to our personal contacts, as well as posting it on professional groups
focused on Agile software development on Google Groups and LinkedIn. At the end
of the survey period, 32 respondents had completed Part 1A and Part 1B. A further 8
respondents were recruited through referrals, leading to a total of 40 fully completed
surveys.
6.3.3 Interviews
We identified potential interviewees from Part 5 of the survey. We contacted all re-
spondents who indicated that they wished to be contacted for follow-up interviews. In
total, we sent 20 invitations and received 11 confirmations for follow-up interviews.
Demographic data regarding the interviewees is summarized in Table 6.1.
Prior to each interview, we sent the interviewees a summary of their survey answers,
so that they can revisit their answers during the interview. The goal of the follow-up
interviews was to get a better understanding of the following:
• Why Agile practices were introduced the way the respondents described in Part
1B of the survey.
• If from Part 1B of the survey, we observed that practices were added gradually
or incrementally, we asked about impacts, positive and negative, of adding those
practices.
• If from Part 1B of the survey, we observed that one or more practices were dis-
continued, we asked about the impacts, both positive and negative, of discontin-
uing those practices.
We used semi-structured interviews, because we had a list of topics to be covered

however the specific interview question depended on the interviewee’s initial responses
[182]. The interviews were conducted face-to-face whenever suitable, otherwise it was
done over the telephone or video call. Each interview lasted 45–60 minutes. The
interviews were recorded with the interviewees’ verbal consent and were subsequently
transcribed.
6.3.4 Data Analysis

Qualitative Analysis
To analyze the interview transcripts, we used open and focused coding with the help of
f4analyse tool2 .
2 https://www.audiotranskription.de/english/f4-analyse
Table 6.1: List of interviewees and their contexts.
IDa Location Role Experi- Team Market- orien- Domain Context overview
ence sizeb tation
R11 Indonesia Project Man- 6 years 100 Market driven, Insurance IT Department of a multina-
ager internal use tional Fortune 500 company.
Started Agile transformation in
2015.
R14 Brazil Developer, 3 years 20 Internal use Government IT Department from the Brazil-
Trainer, Sys- (Court) ian court of accounts. Started
tem architect Agile transformation prior
2015.
R32 Canada Developer, 6 years 13 Market driven, Independent Development team in a start-up
Quality As- bespoke Software company. Started up with Agile
surance, Vendor practices
System (ISV)
Analyst
R33 Sweden Scrum Master 6 years 6 Internal use Telecom A small project team within
a large multinational company.
Started Agile transformation in
2011.
R34 Indonesia CEO 3 years 33 Bespoke ISV Development team in a start-up
company. Started Agile trans-
formation 6 months after initia-
tion
R35 Ireland Scrum Mas- 3 years 6 Bespoke, mar- ISV Development team in a start-up
ter, Developer ket driven, company. Started Agile trans-
maintenance formation in 2016
R36 Sweden Program 23 1000+ Market driven Telecom A solution development pro-
Manager years gram in a large multinational
company.Started Agile transfor-
mation in 2005
R37 Sweden Scrum Master 20 1000+ Market driven Telecom A solution development pro-
years gram in a large multinational
company. Started Agile trans-
formation in 2012
R38 Sweden Scrum master, 7 years 70 Market driven ISV A project in a large multina-
QA tional company. Started Agile
transformation in 2011
R39 USA Researcher, 3 years 6 Bespoke, mar- Research A project in a university’s
Developer ket driven & devel- biomedical research lab. Started
opment, with Agile practices as the
biomedi- project started
cal
R40 Finland CTO, Devel- 6 years 11 Market driven ISV Development team in a start-up
oper, Scrum company. Started up with Agile
master practices.
a
Respondent ID according to the order they are received in the survey tool.
b
Reflects the size of software development team affected by the Agile implementation. Not overall company size.
Open Coding. The first cycle coding that we performed was open coding. To per-
form open coding we used line-by-line coding because it allowed us to remain open to
the data and minimize a priori assumptions [37, 187]. For each line of the transcript
was assigned with one or multiple labels. If a line of transcript is too mundane, we
looked into the whole sentence which can span into multiple lines. For example the
following sentence from the transcript “if we [do not] have coding standard, and one
person resigns and next month we need to improve the application, maybe add a fea-
ture, without coding standard it would be difficult for others to carry on with the work.”
was labeled as rationale for adding coding standard and need to retain knowledge.
Focused Coding. The second cycle coding that we employed was focused coding.
Focused coding was selected because we wanted to identify themes or common con-
cepts that emerged from the data [187]. For all transcript fragment that had the same
labels, we examined them for commonalities. An example of the result of focused
coding can be seen in Table 6.7.
Quantitative Analysis
To analyze the data from the AMMs and the survey, we had to prepare the data into
an ordinal scale, so statistical methods can be applied. We then implemented different
statistical methods using SPSS 24.
Preparing AMM data. The AMMs map Agile practices to maturity levels. Since
maturity levels are ordered, this indicates an order of practice introduction. The “rank”
or order of a maturity levels and its associated practices can therefore be treated as
ordinal values. Since maturity models may start at different levels, we treat the lowest
level of an AMM in which practices are introduced as its level 1. For maturity models
that repeat a practice in different maturity levels, we considered the lowest level for
such a practice. For example M1 suggested TDD to be introduced in level 1 and 2, we
would consider TDD as level 1 practice.
Preparing survey data. In Part 1B of the survey, respondents indicated start and end
dates for Agile practices (see Figure 6.4). To make introduction strategies independent
of actual dates, practices were ranked by the order of introduction. This makes it
easier to compare data across respondents. This also makes it possible to compare the
orderings of AMMs and practitioners experience.
Inter-rater Agreement. We performed inter-rater agreement evaluation to see whether
there are agreements between the AMMs. We selected Fleiss Kappa to evaluate overall
agreement because we have more than two raters (AMMs) for each practice [73]. We
also performed pair-wise inter-rater analysis among the AMMs, using Cohen Kappa.
Pearson’s chi-square. This test was performed to examine if there are dependen-
cies between two or more variables. From the survey, we found three strategies for
introducing Agile practices; big bang, incremental, and complex. To see if there is a
dependency between strategies and success rates, we performed a chi-square test [63].
1 2 3 4 5 6 7
Figure 6.4: Example of an ordering of practice introduction. Each color corresponds

to a practice. The numbers indicate in which order the practices were introduced.
Friedman test. This test was used to examine if there are significant differences
between multiple observations, i.e. whether the ranks of Agile practices indicated by
the survey respondents differ significantly. This test was selected because Friedman
test does not assume normal distribution and is suitable for ordinal data.
Wilcoxon test. This was done as a post-hoc test for the Friedman test [18]. This test
was used to examine if there are significant differences between two groups (pairwise).
We also used this test to examine the ranks of Agile practices. However, unlike the
Friedman test, we performed Wilcoxon test on pairs of Agile practices, for example in
Table 6.5.
Social network analysis. This was performed to visualize the dependencies be-
tween Agile practices. Both for the AMMs and the survey data, we counted and
tabulated the number of occurrences that an AMMs and the survey respondents, re-
spectively, indicated that one practices was introduced before another (see Table D1
and Table D2 in Appendix 4) of this chapter. For example, the value 2 in cell (row:tdd,
column:pp) in Table D1 means that TDD is introduced before pair programming in two
AMMs. We used R3 to visualize the social network.

We classified the validity threats to our study according to Petersen and Gencel [165].
Descriptive validity. This threat pertains to the accuracy of capturing the reality.
Through the survey and interviews, we asked the respondents to account events that
occurred in the past, i.e. a retrospective study. As researchers, we were not able to
directly observe how and when Agile practices were introduced or abandoned. Also,
in a retrospective study there is a risk of post-rationalization, where interviewees pro-
3 https://www.r-project.org/
vide a different account of what happened not to deceive but because it was what they
believed happened [162]. Post-rationalization is well documented as a validity threat
in medicine [77] and law studies [236]. We were not able to completely eliminate the
risk of post-rationalization, however we believe it is important to acknowledge it.
Theoretical validity. This threat pertains to whether we are able to collect the rel-
evant constructs that we aimed to collect. One aspect to theoretical validity is matu-
ration. To reduce this threat we designed the survey so that it could be completed in
less than 20 minutes. This is also the reason we adapted the list of Agile practices
described by [212], because their survey includes a manageable number of practices.
Furthermore, we tested the survey with 10 individuals from industry and academia
with experience in Agile software development. Selection of subject is another aspect
in theoretical validity. The subjects took part in both the survey and interview based on
their willingness to participate. This led to a large variation in contexts, which affects
the analysis of the data. On the other hand, the respondents’ willingness to participate
also allowed us to collect richer data.
Interpretive validity. This threat pertains to researcher’s bias to draw conclusions.
To mitigate this issue, we based our interview questions on the respondents’ answer
to the survey and that the interview was to follow up unclear information. For exam-
ple, from Part 1B, we could learn which Agile practices the respondents use and when
they are adopted. Therefore, in the interview we would follow up on why the practices
were adopted in such a way and what are the impact of adding or abandoning prac-
tices. This way, the interview transcript was easier to code and provide little room for
loose interpretation. In addition, a senior researcher with experience in Agile software
development4 , external to the authors, performed a post-hoc validation on the first five
interviews. This was done to ensure that the coding performed by the first author was
consistent.
Generalizability. This threat pertains to the extent the result of the study is gen-
eralizable in the larger population. In this study, we used convenience sampling. On
one hand, we covered a large variety of contexts, in terms of software products, ge-
ographical locations, and industry domains. However, large software organization is
under represented in this study. We cannot claim that our results are generalizable to a
large population. However, the diversity of context allowed us to gather rich data from
different perspectives and minimize systematic bias that could be present if we had a
homogeneous context.
4 Prof. Kai Petersen from Hochschule Flensburg (Germany) and Blekinge Institute of Technology (Swe-
den)

From the literature study, we identified 26 AMMs (M1–M26). We only discuss Type
1 AMMs (M1–M12) in in detail, since only those have explicit mappings of Agile
practices to maturity levels (see Section 6.4.1). From the survey, we obtained responses
from 40 practitioners. The details of the demographics of the survey respondents can
be found in Appendix 2 of this chapter. Out of 40 respondents, we conducted follow-up
interviews with 11 of them. The results of the survey and the interviews are presented
in Section 6.4.2. The comparison of AMMs with practitioners’ experience is presented
in Section 6.4.3.
6.4.1 Strategies in Introducing Agile Practices According to Agile

Maturity Models (RQ1)
Table 6.2 summarizes some key data from the 12 Type 1 AMMs, like the different foci
or angles of maturity, whether the recommendations are based on empirical evidence,
and in which ways the models are empirically evaluated. M1–M11 are presented in
peer-reviewed papers, while M12 is a consultancy white paper.
From Table 6.2, we can see the following:
• Nine of the twelve Type 1 AMMs adopted the classifications of maturity levels
from CMMI.
• All Type 1 AMMs focus on different aspects of maturity.
• The recommendations provided by the AMMs are rarely based on empirical ev-
idence. In the three cases where empirical evidence is provided, they originated
from experience reports (M1, M4, M12). M10 reported the use of grounded the-
ory, but it was unclear whether grounded theory was used as a basis to formulate
the AMM.
• In terms of empirical validation, 10 of 12 AMMs were evaluated through a static

validation. M2 and M3 were examined with students, but the results of the val-
idation were not presented. M10 mentioned two rounds of case studies to val-
idate the AMM. However, the validation was only used to examine the use of
pair programming [173]. Overall, the empirical validation of the AMMs lacks
thoroughness.
We also looked into the Type 2 AMMs (M13–M26) that lack mappings of Agile
practices to maturity levels (see Table A1 in Appendix 1 of this chapter). Information
Table 6.2: Summary of Type 1 Agile Maturity Models
ID Author (year) Focus of maturity Empirically based Empirically evaluated

M1 [19] Ability to integrate or re- Experience report from a de- Assessment and trials were
lease components in a fixed partment in British Telecom. done with internal and exter-
release cycle to on-demand nal parties.
release.
M2 [133] Unclear No. Visual data mining Experimented with students
technique was used to ex- in classroom settings. Re-
tract relationships between sults not presented
XP practices based on Fig-
ure 4 p.70 in Extreme pro-
gramming explained: Em-
brace change by Kent Beck
[13]
M3 [144] Follow CMMI maturity lev- No Tested on student project,
els. Initial - Mature. Com- but result of the evaluation
petences of team members was not presented in the pa-
increases. per.
M4 [158] Unclear Experience report from Implemented in a number of
Sabre airline solutions. internal teams.
M5 [160] Follow CMMI maturity lev- No Static validation through
els. Initial - Mature. Start- questionnaires.
ing no implementation of
agile practices to using Ag-
ile practices for context im-
provement.
M6 [202] Values from Agile mani- No Static validation through
festo. Starting from project questionnaires.
level Agile implementation
to organization level.
M7 [242] Based on CMMI maturity No Static validation through
levels. Starting with no im- two interviews.
plementation of agile prac-
tices to using Agile prac-
tices and principles for pro-
cess improvement.
M8 [222] Based on Sidky’s maturity No Delphi study and a case
level interpretation. study in industry through 4
hours of assessment session.
M9 [203] Follow CMMI maturity No Static validation with practi-
level with specific focus on tioners.
quality assurance.
M10 [173] Starting by establishing ag- Claimed to be, but unclear Static validation through a
ile properties and moving on case study. Implementation
to lean way of working of the model is also done in
the second case study.
M11 [108] Follow CMMI maturity No No
level.
M12 [11] Collocated to distributed Collective experience from No
team. consulting.
regarding these AMMs mostly came from blogs or presentation slides. We could not
infer whether these AMMs were developed based on empirical studies or collective
experience. We also could not infer whether they were validated in any way.
For Type 2 AMMs, the descriptions within each maturity level are sometimes very
vague. The Type 2 AMMs do not associate maturity with Agile practices directly and
the relationship between maturity levels and Agile practices is therefore unclear. Many
Type 2 AMMs associate maturity with different aspects, such as ability to establish and
use measurements, process automation, scaling from collocated teams to distributed
teams, and scaling from team level to corporate level, and many more. Some Type 2
AMMs are focused on very specific aspects, like testing (M24) or continuous delivery
(M23). Some models also do not claim to be a maturity model, despite the labeling, for
example M17 and M21. M17 provides different steps that can be taken in improving
one’s ability to be Agile. Meanwhile M21 describes behaviors of team members from
less to more desirable. These steps and behaviors appear arbitrary and not based on
existing Agile principles, neither explicitly nor implicitly.
To summarize, the AMMs, both Type 1 and Type 2, are not in agreement regarding
what Agile maturity means and how it can be improved. The AMMs are often devel-
oped from experience within specific teams or organizations. It is therefore not clear
to which extent the AMMs are applicable outside their original contexts.
Mapping of Agile Practices and Maturity Levels
Table 6.3 tabulates maturity levels and Agile practices for all Type 1 AMMs. The table
shows that there is little agreement regarding which practices should be introduced in
which maturity level. Inter-rater agreement between the AMMs was very low (Fleiss
κ = 0.088). Pairwise inter-rater agreement between the AMMs shows that M3 and
M5 have moderate agreement (Cohen κ = 0.451). Meanwhile, M6 and M8 showed
substantial agreement (Cohen κ = 0.713). This could be expected, since M8 is an
extension of M6. In M1, M7, M9 the same practices are repeated in multiple maturity
levels, see bold texts in Table 6.3. The repetitions indicate that a practice is introduced
in a lower maturity level and gradually improved in higher maturity levels.
The 12 Type 1 AMMs are based on different sets of practices. In total, they cover
31 practices, 29 unique to Agile. In addition, Kanban and DevOps are included as
Agile practices in M8. Most of the Type 1 AMMs did not provide definitions for the
Agile practices they used, except M3 and M8. Rationales for mapping Agile practices
to specific maturity levels are also missing. For example, M3 based its maturity levels
on the progression of team members’ competences (see Table 6.2). It is, however, not
clear why the Agile practices introduced first include on-site customer and TDD (see
Table 6.3). On-site customer may not necessarily require special competences from the
team members. However, TDD is usually considered a difficult and complex practice.
Order of Agile practices according to the AMMs

Figure 6.5 tabulates how often an Agile practice is mapped to a certain maturity level
by the 12 Type 1 AMMs. We can see that, generally, the AMMs are in disagreement
on the mapping of Agile practices to maturity levels. Furthermore, none of the Agile
practices is mentioned in all AMMs, and some practices are only mentioned in one or
two AMMs. Fifteen practices of 31 in total are named in less than half of the AMMs.
The AMMs have high disagreement regarding the following practices: continuous
integration, onsite-customer, frequent releases, and refactoring. However, we can also
see that AMMs tend to agree that planning game need to be introduced first. There
is also some extent of agreements for some practices, like coding standard, and TDD.
These practices are more frequently introduced first, i.e. at maturity level 1 (x̄=1). The
AMMs also tend to be agree that face-to-face meeting should be introduced at maturity
level 2 or higher (x̄=2) and self-organizing team at maturity level 3 or higher (x̄=3).
The bubble chart shows in which maturity levels a practice is more or less fre-
quently introduced, but not in which order practices are introduced. To look into order-
ings, we used social network graphs to aggregate orderings between the practices from
our 12 Type 1 AMMs.
From Table D1 in Appendix 4 of this chapter, we created a network graph (see
Figure 6.6). To ease analysis, we excluded practices that are mentioned in <5 AMMs
and cells with values <2; otherwise the network would become too dense.
The size of node is proportional to the number of times a practice is in the AMMs.
Blue colored edge indicates strong link. From Figure 6.6, we can see that planning
game has a number of blue colored edges radiating out of it which indicates it fre-
quently preceded other practices, i.e introduced first. Meanwhile, self-organizing team,
on-site customer, and face-to-face meeting has a number of blue colored edges going
into the nodes, which indicates they often succeeded other practices, i.e. introduced
later. These observations are aligned with bubble chart in Figure 6.5.
From Figure 6.6, we can indicate strong directed links between different pairs of
Agile practices. The edges also indicate directionality. These pairs of practices are
summarized in Table 6.4. We can also see the practices with strong links are the ones
that are more frequently mentioned in the AMMs.
Table 6.3: Mapping of Agile practices to maturity levels. Practices that are listed in
multiple maturity levels are shown in boldface.
ID Level 1 Level 2 Level 3 Level 4 Level 5 Level

6
M1 TDD, unit testing, TDD, user stories continuous integra- refactoring, self- configuration man-
code review tion, short iteration, organizing team agement, continuous
frequent releases delivery
M2 TDD, simple design, continuous integra- pair programming, metaphor, 40 h week,
refactoring, coding tion collective ownership small release, on-site
standard customer, planning
game
M3 No Agile practice es- planning game, col- pair programming, on-site customer,
tablished laborating customer continuous integra- simple design
(on-site customer), tion, 40 h week,
user stories, frequent coding standard,
releases, metaphor, configuration man-
TDD, acceptance test agement
M4 user stories, short it- pair programming on-site customer, unclear. Continuous unclear. Mature team
eration, timeboxing retrospective, cross improvement mentors other teams
functional team,
tracking progress
M5 No Agile practice es- tracking progress, TDD, continuous simple design, 40 h No Agile specific
tablished on-site customer, integration, user week, self organiz- practices described.
planning game, stories, short it- ing team, tracking Focus on continuous
TDD, unit test eration, frequent progress improvement
releases, refactoring,
coding standard,
pair programming,
metaphor, face-
to-face meeting,
collective ownership
M6 collaborative plan- continuous delivery, Maintain backlog, frequent releases, TDD, pair program-
ning, coding stan- tracking progress, self-organizing team, daily meeting, user ming, on-site cus-
dard, collaborative configuration man- face-to-face meeting, stories, adaptive tomer
teams, knowledge agement continuous integra- planning
sharing tion, refactoring
M7 No Agile practice es- user stories, plan- user stories, track- self-organizing team retrospective, stand
tablished ning game, backlog, ing progress, defini- up meeting
Scrum roles are tion of done
defined, stand up
meeting
M8 user stories, coding configuration man- unit testing, con- frequent releases, TDD, on-site cus-
standard, on-site agement, tracking tinuous integration, daily meeting, tomer
customer, planning progress, continuous refactoring, f2f adaptive planning
game, retrospective, delivery, backlog, meeting, self orga-
acceptance testing frequent releases nizing team, kanban,
devops
M9 No Agile practice es- configuration man- continuous integra- collective ownership, onsite customer,
tablished agement, coding tion, metaphor, f2f self-organizing team, stand up meeting
standard, pair pro- meeting, retrospec- tracking progress
gramming, onsite tive
customer, planning
game, stand up
meeting
M10 TDD, short iteration, face-to-face meeting unclear self-organizing team unclear. Lean
configuration man- pro-
agement, planning cess
game
M11 No Agile practice es- pair programming, simple design, cod- continuous integra- Unclear
tablished collective ownership, ing standard tion
tracking progress
M12 short iterations, planning game, backlog TDD
retrospective, stand tracking progress,
up meeting, configu- refactoring
ration management,
coding standard,
continuous integra-
tion
6
Order
5
5th 2 3 1 1 1 1
4
4th 1 1 1 1 1 2 1 2 1 1 1 2
3
3rd 1 1 1 3 1 1 2 1 1 1 1 1 1 1
2
2nd 1 1 1 2 1 1 2 2 1 1 1
1
1st 1 7 3 5 4 3 1 1 3 3 1 1 1 1 2 2 1 1
0
0.4 5.4 10.4 15.4 20.4 25.4
Cont. integr. Track. prog. Onsite cust. User stories Conf. mgt. Self org. Freq. rel. F2F meeting Standup Metaphor Backlog Cont. delivery Def. of Done Code rev. Cross func. 30.4DevOps
(x̅ =2) (x̅ =2) (x̅ =2.5) (x̅ =1) (x̅ =2) team (x̅ =3) (x̅ =2.5) (x̅ =2) meet. (x̅ =1) (x̅ =2) (x̅ =2.5) (x̅ =2) (x̅ =2.5) (x̅ =1) team (x̅ =3) (x̅ =3)
Plan. game Code std. TDD Short iteration Pair prog. Unit test Refactor. Retrosp. Simple design Coll. ownership 40h week Adapt. plan. Acc. test Timeboxin Kanban
(x̅ =1) (x̅ =1) (x̅ =1) (x̅ =1) (x̅ =2) (x̅ =2) (x̅ =2.5) (x̅ =1.5) (x̅ =2.5) (x̅ =2) (x̅ =3) (x̅ =4) (x̅ =1) g (x̅ =1) (x̅ =3)
N=9 N=8 N=7 N=6 N=5 N=4 N=3 N=2 N=1
Figure 6.5: Frequency of Agile practices by maturity level in our 12 Type 1 AMMs. The order of practices is from
the more frequently named (left, maximum N=9 mentionings) to the less frequently named (right, minimum N=1
mentioning). N indicates the number of AMMs that mentioned a practice and x̄ indicates the value of the median.
183
Figure 6.6: Agile practices dependencies according to the AMMs.
6.4.2 Strategies in Introducing Agile Practices According to Prac-

titioners (RQ2)
In the survey, 40 respondents used the sliders (Part 1B) to indicate the start and/or end
of Agile practice use. This allowed us to identify strategies used to introduce Agile
practices, i.e. when certain practices were introduced. Together with 11 follow-up
interviews, we identified different approaches to introduce Agile practices, rationales
why Agile practices were introduced at different orders, and the impact of introducing
or abandoning Agile practices.
Table 6.4: Partial orders of Agile practices according to the AMMs.
Preceding Succeeding
Planning game Tracking progress
Continuous integration
TDD
Planning game Face-to-face meeting
Configuration management
Tracking progress
User stories On-site customer
Planning game
Continuous integration
Coding standard
Coding standard Self-organizing team
Configuration management
Planning game
User stories
Strategies in Introducing Agile Practices
From Part 1B of the survey, we found three main strategies for introducing Agile prac-
tices over time, see Figure 6.7. Each colored line represents an Agile practice. The
start and end of a colored line represents the start and end of use for a practice. If a
colored line ends at the “In-Use” dashed line (to the right), the corresponding practice
is still in use. If it ends before the dashed line then it has been abandoned.
Big bang strategy. In this strategy, all practices are introduced at the same time, see
Figure 6.7a. This strategy was reported by five out of 40 respondents.
Incremental strategy. In this strategy, Agile practices are added incrementally, see
Figure 6.7b. The number of increments varies between two to seven. An incremental
strategy was reported by 17 out of 40 respondents.
Complex Strategy. Complex strategies also involved abandonment of practices, see
Figure 6.7c and 6.7d; 18 out of 40 respondents reported a complex strategy. Within
the complex strategy, we identified two sub-categories: (1) practices were abandoned
at the same time (see Figure 6.7c), (2) practices were abandoned gradually (see Figure
6.7d).
When looking at the perceived success rates of Agile practice adoption with the dif-
ferent strategies (see Figure 6.8), we can see that all strategies, except the incremental
strategy, reported more than 80% success (successful and very successful).
We performed a chi-square test by grouping the success rate into no success (unsuc-
cessful and neutral) and success (successful and very successful). The test indicated
Collective ownership
Coding standard
Continuous Integration
TDD
Metaphors & User stories
Stand up meeting
(a) Big Bang Strategy
Timeboxing
Simple design
Short Iteration
Retrospective
Refactoring
Tracking Progress
Planning Game
(b) Incremental Strategy Pair programming
Onsite customer
Self-org. cross func. team
Face-to-face meeting
(c) Complex Strategy - practices abandoned at the same time
(d) Complex Strategy - practices abandoned gradually

In-Use
Figure 6.7: Strategies to Introduce Agile Practices Over Time
no statistically significant relationship between strategies and success rate (p=0.473,

df=2).
Order of Agile practices according to Practitioners

From the data of Part 1B of the survey, we could rank Agile practices regarding their
order of introduction, see Preparing survey data in Subsection 6.3.4. The ranking
is summarized in Figure 6.9, which shows how frequent an Agile practice is ranked
Overall (N=40) 2.5% 15% 55% 27.5%
Big bang (N=5) 60% 40%
Incremental (N=17) 5.9% 17.6% 41.2% 35.3%
Complex (N=18) 16.7% 66.7% 16.7%
-20% 0% 20% 40% 60% 80% 100%

Figure 6.8: Success rate and Agile practice introduction strategies.
given its order of introduction. We excluded survey respondents who introduced Agile
practices using a big bang strategy, because the practices were introduced at the same
time, i.e. no ordering.
Order
7th 1 1 1
6th 1 1 1 2 1 1 1
5th 1 1 1 2 1 1 2 1 1 1 3 1 1 1 1
4th 1 2 2 1 3 3 3 1 1 1 2 2 1 3
3rd 4 3 6 4 4 3 1 2 5 2 5 3 5 4
2nd 6 12 4 6
1st 14 5 3
0.3
F2F Tracking 2.3
Self-org. cross Retrospect.
4.3 Planning Timeboxing 6.3 Standup Cont. 8.3
Short Coding 10.3
Refactoring Metaphors 12.3
Collective Simple 14.3
Onsite Pair
16.3
TDD
meeting progress func. team (x̅ =2) game (x̅ =2) (x̅ =2) meeting Integration Iteration standard (x̅ =2) & User ownership design (x̅ =2) customer programming (x̅ =3)
(x̅ =1) (x̅ =2) (x̅ =2) (x̅ =2) (x̅ =2) (x̅ =2) (x̅ =1) stories (x̅ =2) (x̅ =2) (x̅ =2) (x̅ =3)
N=32 N=31 N=30 N=28 N=27 N=24 N=23 N=22 N=20 N=16 N=15
Figure 6.9: Frequency of Agile practices by introduction order in our survey. The order
of practices is from the more frequently named (left, maximum N=32 mentionings) to
the less frequently named (right, minimum N=15 mentionings). N indicates the number
of respondents that used a practice and x̄ indicates the value of the median.
From Figure 6.9, we can see that practices like face-to-face meeting and coding
standard were more frequently introduced first (x̄=1). Meanwhile, metaphors & user
stories, short iteration, and timeboxing were more frequently introduced second (x̄=2).
We can also see that TDD was never introduced first.
Similar to our analysis of the AMMs, we also created a network graph to find
which practice is followed by another using Table D2 in Appendix 4 of this chapter. In
creating the network graph, we excluded pairs of practices with occurrences <5, i.e.
the cells with shades of red; to prevent the network from being too dense (see Figure
6.10).
Figure 6.10: Agile practices dependencies according to the practitioners.
The size of node is also proportional to the number of times a practice was used by
the survey respondents. From Figure 6.10, face-to-face meeting has many blue colored
edges radiating out this indicates it frequently preceded other practices, i.e. introduced
first. Meanwhile, practices like continuous integration and self-organizing cross func-
tional team has many edges going into the node; a few of them are blue colored edges,
which indicates that they frequently succeeded other practices, i.e introduced later.
This observation supports the indication from the bubble chart in Figure 6.9.
From color of the edges from Figure 6.10, we can also see there are strong directed
links between pairs of practices, primarily with face-to-face meeting. The pairs of
practices are summarized in Table 6.5.
We performed Friedman’s test to see if there is significant difference in the intro-
duction order of Agile practices. The test shows no statistical significance (p=0.586,
df=16). This indicates the ranking of Agile practices order of introduction does not
differ significantly. As we can also see from the values of median do not vary too much
across the practices (see Figure 6.9).
We then performed Wilcoxon test on pairs of practices which have strong directed
links. In Table 6.5, if the p-value is <0.05 it indicates the rank of introduction order
between these pairs of practices differs significantly, which means the practice on the
right succeeded the practice on the left.
Table 6.5: Partial orders of Agile practices according to practitioners.
Preceding Succeeding p-value

Face-to-face meeting TDD 0.001
Planning game 0.008
Standup meeting 0.018
Short iteration & Frequent releases 0.004
Continuous integration 0.003
Metaphors 0.001
Timeboxing 0.001
Pair programming 0.005
Tracking progress Continuous integration 0.028
Retrospective 0.039
Tracking progress Self-organizing cross functional team 0.264
Retrospective 0.119
Coding standard 0.237
Rationales for Introducing Agile Practices
We identified the rationales for introducing Agile practices in different order. The
details of the rationales are discussed in details in the following paragraphs.
Pre-Agile Transformation Practices. From the interviews, a number of respondents
mentioned that some practices that were introduced first were pre-Agile transformation
practices. Table 6.6 summarizes the list of practice that the interviewees indicated as
pre-Agile transformation practices.
Table 6.6: Pre-Agile transformation practices.
Practice Quotes
Face-to-face meeting “We have always had face-to-face meeting even before we went
into the agile transition.” (R36)
“Face-to-face meeting existed before Agile.” (R33)
“The practices that we adopted first [face-to-face meeting and
coding standard] do not correlate with the methodology, [...]
they are just good practices.” (R11)
“Face-to-face meeting they’ve always done that.” (R14)
Pair programming “We have done pair programming, we have been sitting in one
computer even before 2007 [Agile transformation].” (R36)
“Pair programming it was something that we started before we
actually went into Agile.” (R37)
Coding standard “We have had documents on how to code [coding standard]. We
have had that for many years, even before Agile.” (R36)
“I think [coding standards] have been there even before [Agile
transformation].” (R38)
“[The company] started practicing Agile since Q3 2012, in that
period they already had coding standard.” (R33)
Also see R11’s quote for face-to-face meeting.
Refactoring “We have done [refactoring] for years. That was not a big issue
for us when we went into Agile.” (R36)
“Refactoring has been used since before using Agile.” (R11)
Short iteration & fre- “Short iteration and frequent release existed before Agile, but it
quent releases was every 2-3 months.” (R11)
Self-organizing cross “Self-organizing team was done before we implemented Agile.”
functional team (R11)
From Table 6.6, we can see that face-to-face meeting and coding standard were
frequently mentioned by the interviewees as practices that pre-dated Agile transforma-
tion. There were also mentions of other practices that were more Agile specific like
pair programming and refactoring. This indicates that some practices that we assume
as Agile practices are actually pre-existing practices in software development teams.
Rationales for Introducing Practices First. From the interviews, we identified ratio-
nales why some Agile practices were introduced first. We did not include interviewees
that employed big bang strategy because all practices were introduced at the same time.
The rationales for introducing Agile practices first are summarized in Table 6.7.
R33 and R34 shared a similar reason for starting with self-organizing cross func-
tional team. Both respondents’ teams were initially separated by function or role.
When they transitioned to Agile, they needed to establish a team that included cross-
functional roles.
What is also observable from Table 6.7, the practices that started first tend to be
used as means for solving issue pertaining to communication or improving communi-
cation and knowledge retention.
Table 6.7: Rationale for adding Agile practices first.
Practice Rationale Quotes

Face-to-face meeting Establish clear re- “For us it is important to discuss the requirements as clearly as
quirements possible.” (R34)
Coding standard Knowledge reten- “If one person resigns and next month we need to improve the
tion application, maybe add a feature, without coding standard it
would be difficult for others to carry on with the work.” (R34)
Self-organizing cross Shift from func- “When we were doing waterfall, one team consists of develop-
functional team tional organization ers, another team designers” (R33)a
“In our company we had a number of teams, Project managers,
system architect, developers/engineers for different products
[..]. From our side if we have a project, we might need dis-
cussions between the roles about the responsibilities” (R34).
Reduce feedback “When developers are done coding, and need to be tested by the
loop testing team. If they’re in one team, the feedback loop is shorter
between the tester and developers.” (R33)
Face-to-face meeting, Poor communica- “The communication in the team was very poor, it was like a
self-organizing cross tion group of individual developers working in a room.” (R35)
functional team, stand up
meeting
Tracking progress, retro- “They did not keep each other informed as to what they were
spective doing.” (R35)
Face-to-face meeting, Industry expectation “A bunch of the startup practices are pretty much what you have
self-organizing team, to do or expected to do.” (R32)
collective ownership,
retrospective, stand up
meeting
Planning game, track- Encouraged prac- “Sprint meeting and tracking progress, refactoring has been
ing progress, refactoring, tices something that we encourage from the start and short itera-
short iteration tions.” (R32)
Self-organizing cross The need to deliver “[The product] was a software suites with many integrated
functional team integrated solution pieces of software [...] that would be integrated.” (R14)
a
R33 indicated that she would separate self-organizing team from cross functional team.
Rationales for Introducing Practices Later. From the interviews, we identified ra-
tionales why some Agile practices were introduced later. We did not include intervie-
wees that employed big bang strategy because all practices were introduced at the same
time. The rationales for adding Agile practices are summarized in Table 6.8.
Table 6.8: Rationale for adding Agile practices.

Self-organizing team Time for learning/ “[Previously] in one team everybody is designer, suddenly we
adaptation need to work with testers. Then you need to learn to get to know
each other” (R33)a
Planning game “Planning game why we started 2016 Q1 [later], it was lessons
learned from us, we were still learning and we realized we
needed it.”(R34)
Lack of trans- “Some people had a verbal agreement with their manager and
parency it was not transparent to all the team members.” (R14)
Time needed for “We started out with having story points but [the developers]
training could not grasp the abstract concept of story points. So we
manage it in time [number of hours]. So they can just get a feel
for estimation. Then we decided that we want to go to proper
story points so we can get a velocity for a more accurate release
planning.” (R35).
Refactoring Time for learning/ “Refactoring we adopted later also because of learning experi-
adaptation ence.” (R34).
Poor architecture “The software architecture did not let us do use automated test-
design ing, so the first thing we did was to refactor the architecture and
improve it in a way that would be able to do automated testing.”
(R14)
On-site customer Customers were not “The customers became aware of this concept very lately.”
aware (R34).
Difficulty in deliver- “It was added because we had difficulties when the product is
ing the product rolled out.”(R11)
Manager initiative “It was the IT manager [who] had that idea of bringing in one
of those project members that was the business people.” (R14)
Streamline commu- “One [sport] club might come to one [sport scientist] and [the
nication channel sport scientist] would go to a particular developer and say we
need this done, somebody else in another club might want [a
similar feature] done in a different way so they would go to
another developer. In order to cut that out I decided to nominate
one of the head sport scientists as on-site customer.” (R35)
Coding standard, Contin- Need for structured “Our big project at the time was at the stage where there were
uous integration process about six people with different coding standards and expecta-
tions, [...], so we have to structure our processes a little better
and implemented a continuous integration pipeline [...] we im-
plemented some coding standards basically following some lan-
guage styles and some additional expectations within the team.”
(R32)
User stories Minimize require- “We have an awful lot of ambiguous requirements [...] that is
ment ambiguity when the user stories have come in to play.” (R35).
Short iteration & fre- Dependency on “Short iteration, on-site customer and user stories all come hand
quent release other practices in hand. [The sport scientist] teased exactly what it [the require-
ment] means and then as well as we focus only the ones with
the highest business return and priority.”(R35)
Simple design, retrospec- Dependency on “Retrospective added this year [later] maybe because it is re-
tive other practices lated to continuous integration, simple design also.” (R11)
Pair programming Need for knowledge “We made an effort to try to do some pair programming to
dissemination mainly to disseminate information about different pieces of the
puzzle.” (R32)
Time needed to “I wanted to build trust within the team. [...], so pair program-
build trust ming came in the last little while.” (R35)
Continued on the next page

Continuous integration Technical difficul- “Continuous integration was prohibited before because only
ties one person had a build license [..]. Now all of us has one.”
(R35)
Using microser- “After we have good application design using microservices,
vices then we started to do continuous integration.” (R11)
a
R33 indicated that she would separate self-organizing team from cross functional team.
From the interviews, we learned that time to learn was a common rationale why
some practices were introduced later. Team members may require time to adapt to the
Agile way of working and gain lessons learned before adding other practices that the
team members might require (R33, R34, R35). Time may also be needed for some of
the practices became more relevant to be introduced, e.g., trust had to be built before
pair programming could be introduced successfully (R35) or the product design needs
to be improved before implementing continuous integration (R11).
The follow up interviews also revealed one of the more common rationales for
adding Agile practices later was some form of emerging challenges. These emerging
challenges include difficulty in delivering product, redundant communication channels,
and requirements ambiguity, were then addressed by adding certain Agile practices.
Another commonality we found was that some Agile practices were dependent
on other practices. Some practices were initiated together because they support other
practices, e.g., simple design and supported continuous integration (R11).
While some practices may share a similar rationale why they were added later,
we identified different reasons why on-site customer was added later. The rationales
ranges from the lack of awareness from the customers themselves, to an individual’s
initiative, and issues in communication and delivery.
Impact of Adding and/or Abandoning practices
From the follow-up interviews, we identified the positive and negative impacts from
adding and/or abandoning Agile practices. More details on the impacts are discussed
in the following paragraphs.
Impact of adding Agile practices. Table 6.9 summarizes the positive and negative
impacts of adding Agile practices later. The corresponding interviewee quotes can be
found in Appendix 5 of this chapter, in Table E1.
From Table 6.9, we can see that adding a practice can have both positive and neg-
ative impact. Time and quality were rarely mentioned as the impacts of adding a prac-
tice. Aspects pertaining to the people and process were more often mentioned as im-
pacts of adding a practice.
Table 6.9: Impacts of Adding Practices
Practice Impact
Positive Negative
On-site customer Improves coordination with customers Team members discomfort
(R34) (R34)
Improved requirements prioritization
(R35)
Short feedback loop (R14)
Sprint planning Improved Work organization (R34)
Improved understanding of responsibil-
ity (R34)
Refactoring Improved understanding of one’s task Team members discomfort
(R34) (R14)
Improved understanding of one’s re-
sponsibility (R34)
Continuous inte- Improved transparency on each other’s Recurring bugs (R35)
gration work (R32)
Stand up meeting Easier to uncover development issues Time consuming (R11)
(R11)
Retrospective Improved visibility of team members’
skill set (R11)
Improved visibility of progress and is-
sues (R35)
Timeboxing Visibility of what the end product might
look like (R11)
Short iteration & Works are completed more quickly Conflict with operations team
frequent releases (R11) (R11)
Improved team’s morale (R35)
Pair programing Encourage collective ownership (R35) Team members discomfort
(R32)
User stories Reduce requirements ambiguity (R35)
TDD Easier to find defect (R14)
One recurring positive impact on adding different practices was improved trans-
parency or visibility of different things, like each others’ work, issues, or progress.
Another common positive impact was the improvements on the team members, e.g.,
team morale and improved communication. Improvement on the development process
is another positive impact of adding practices, e.g., improved requirement prioritization
and ease to find defect. Despite positive impacts of there were also negative impacts
of adding practices. One of the more common negative impact of some practices like
on-site customer, refactoring, and pair programming was discomfort to team members.
Impact of abandoning practices. Eighteen of our respondents indicated that one or
more Agile practices had been abandoned. Through five follow up interviews we iden-
tified a number of impacts from abandoning Agile practices. All five interviewees had
abandoned tracking progress, in the case of R33, 13 Agile practices were abandoned
including tracking progress. Other practices that were abandoned was pair program-
ming and on-site customer. The interviewees’ statement can be found in Appendix 5
of this chapter, in Table E2.
Most of the respondents reported no impact from abandoning tracking progress,
except R40 who reported a positive impact because unnecessary work was avoided.
Impact could also be a matter of perspective, R38 believed that abandoning tracking
progress had no impact on the team, but for the Scrum master there was a lack of
transparency. Similar impact was shared by R33. R14 and R32 who abandoned on-
site customer and pair programming, respectively, believed there was no impact from
abandoning those practices.
6.4.3 Agile Maturity Models versus Survey Respondents (RQ3)

We compared the results of our literature study and empirical study (survey and in-
terviews) to see similarities and differences. The comparison was done to reflect on
whether the suggestions provided by AMMs are in anyway representative to how Ag-
ile practices are introduced in practice.
Similarities. Both the AMMs and our survey respondents indicate that retrospective
and coding standard are more frequently introduced first. We did not find other distinct
similarities between AMMs and the survey respondents.
Differences. Table 6.10 summarize the differences of the AMMs and survey re-
spondents pertaining the ranking and partial orders of Agile practices.
When we compared the the bubble charts in Figure 6.5 and 6.9, we can see that
none of the AMMs ranked face-to-face meeting first. Meanwhile the majority of our
survey respondents ranked face-to-face meeting first. The opposite can be observed
for TDD. The majority of the AMMs ranked TDD first, meanwhile none of our survey
respondents ranked TDD first, see Table 6.10.
From Table 6.10, we can see the AMMs and our survey respondents are not in
agreement pertaining to partial orders of some pairs of practices. When we compared
the network graphs in Figure 6.6 and 6.10, we observed some partial orders may be
present in one network but not in another. We also observed that sometimes the direc-
tionality of the partial orders is different.
Table 6.10: Differences between AMMs and industry experience on practice rankings
and partial orders.
AMM Survey
Practice rankings
Face-to-face meeting Introduced second or later Introduced first (most frequent)
TDD Introduced first (more frequently) Introduced second or later
Partial orders
Tracking progress→F2F meeting F2F meeting→Tracking progress
Not present F2F meeting→Continuous integration
Planning game→Tracking progress Not present
Tracking progress→Planning game
(weak link)
Not present F2F meeting→pair programming
Continuing the comparison between the network graphs, we can see that in Fig-
ure 6.6, face-to-face meeting many few edges going into the node, which indicates it
succeeded other practices. The opposite can be observed in Figure 6.10, where face-to-
face meeting has many edges radiating out of the node with strong directed links, which
indicates it preceded other practices. We can relate this to the difference in face-to-face
meeting ranking between the AMMs and survey respondents in Table 6.10. This indi-
cates that most of our survey respondents put precedence on face-to-face meeting as a
practice that is important to be established first.
In Figure 6.6 planning game has many strong links with other practices. It indi-
cates that according to the AMMs planning game frequently preceded other practices.
Meanwhile, in Figure 6.10 planning game has many edges going into the node and only
has a strong link with face-to-face meeting. It indicates that according our survey re-
spondents, planning game is a practice that was more frequently introduced later. This
indicates that according to the AMMs, planning game needs to precede other practices.
Unlike the survey respondents who put precedence on face-to-face meeting.
One of the AMMs, M8, includes Kanban and DevOps. Our survey did not include
Kanban or DevOps, however some of our interviewees mentioned them. When asked
if we had missed any other practices, six interviewees (R14, R33, R36, R37, R38,
R39) mentioned Kanban. Kanban generally was added later, sometimes to replace
burn-down/up charts, except for R39 who introduced Kanban first. We found different
rationales for adding Kanban, they are, managing uncertainty (R33, R36, R38), im-
proving transparency (R14), complementing Scrum in ensuring quality requirements
are sufficiently tested (R37). Meanwhile, R32 and R40 indicated that DevOps is a role
that one assumes, and not a practice.
6.5 Discussion 197
What the AMMs are missing. Our interviews also revealed different impacts of
adding practices, positive for the most part. However, there were also negative ones.
In some cases, for example, R14 and R32, the practices that were added later, ended
up being abandoned for different reasons. The AMMs suggest that Agile practices are
to be added incrementally, but they did not mention the potential negative impacts, like
team members discomfort, recurring bugs, and conflict with another team.
Eighteen out of 40 survey respondents indicated that one or more Agile practices
had been abandoned, and perceived that their Agile implementation was successful.
Abandoning practices could be done gradually or done at the same time. In some
cases, only a few practices remained to be adopted. None of the AMMs indicated such
suggestion. AMMs suggested that Agile practices are to be continuously adopted as
the maturity increases. This is the opposite to what we identified in the survey.
6.5 Discussion
This paper presents the results of a literature study, a survey and a series of interviews.
The result of our study indicate that some suggestions from the AMMs are contradic-
tory to practitioners’ experience.
6.5.1 Reflection on AMMs

From our literature survey, we found two types of AMMs: Type 1 AMMs with map-
ping of Agile practices and maturity levels, and Type 2 AMMs with suggestions of
improving maturity without associations with Agile practices. For Type 1 AMMs, we
evaluated the mapping qualitatively and using Fleiss kappa inter-rater agreement, we
found that they are generally not in agreement pertaining to the which practice is to
be included in which maturity levels. This result corroborates with previous evaluation
by Leppänen [127] who reported that the AMMs are not in agreement with mapping
of Agile practices to maturity levels. If we were to evaluate the maturity level of one
team, the result may vary depending which AMM is used, as reported by Ozcan-Top
and Demirörs [157]. This indicates that pre-defined sets of Agile practices is not a good
standard for evaluating maturity.
The AMMs, specifically Type 1 AMMs, were generally developed without empir-
ical basis or based on limited experience within a team or an organization. This may
contribute to the variations of suggestions pertaining which practices are to be intro-
duced in which maturity level. As it has been reported before that Agile practices are
often implemented with deviations from the literature definitions [239] and are tai-
lored to suit the contexts of the development organization [72]. It is possible that each
AMM reflect the individual team or organization’s experience where the researchers
conducted their study. This could indicate that there could be no universal mapping of
Agile practice and maturity levels.
The main drawback of the AMMs, both Type 1 and Type 2, is the lack of detailed
information and rationales for the suggestions that they provided. Furthermore, the
AMMs have different basis for maturity. One AMM would base maturity on team com-
petences, meanwhile another AMM would base maturity on team distribution. From
the lack of agreement among the AMMs, we can indicate that Agile maturity involves
multiple aspects that are pointed out by the different AMMs. This indicates that indi-
vidual AMM may not be sufficient to guide Agile maturity improvement in industry.
Similar to past evaluation conducted by Ozcan-Top and Demirörs [157].
6.5.2 Reflection on Practitioners’ Experience

In their survey, Solinski and Petersen [212] included a number of plan driven practices,
and that plan driven practices are working in tandem with Agile practices. Our survey
did not include plan driven practices. However, our follow up interviews reveal that
some Agile practices, like refactoring and coding standard, already existed prior to
Agile transformation. Coding standards is certainly not a new practice, for example
the standard of naming variables or methods using Hungarian notation is known since
the early 1990s [208]. This could indicate that team members may have existing formal
or informal practices [28] that are similar to Agile practices. This raises the questions
whether teams who have Agile-like practices in implementation have better chance to
succeed from transitioning into Agile.
Although the descriptive statistics indicate coding standard and retrospective are
introduced first by the survey respondents, not all survey respondents are in agreement.
For example R14 introduced coding standard later due to project growth, meanwhile
R11 introduced retrospective later due to dependencies on other practices. This indi-
cates that there might not be a universal order in introducing Agile practices in industry.
Moreover, the results of the Friedman test indicates that the difference in introduction
order of Agile practices is not significant. This could indicate that order in introducing
practices is not a critical aspect in Agile implementation.
6.5.3 AMM versus Practitioners’ Experience

The AMMs and our empirical study results are not in complete agreement. None of the
AMMs suggest that face-to-face meeting is introduced first. This is contradictory to our
survey result and interviewees statement. The suggestion from the AMMs to introduce
face-to-face meeting at a later maturity level is somewhat counter intuitive to practice
6.5 Discussion 199
and the Agile manifesto [15]. Interaction is the first point mentioned in the manifesto,
which indicates that communication is the foundation. Five out of eight AMMs sug-
gested that TDD is introduced first. This suggestion is again counter-intuitive, because
TDD requires knowledge and experience. As Causevic et al. [33] reported that lack
of knowledge and experience as limiting factors for adopting TDD. This indicates that
some suggestions from the AMMs are not suitable in practice, but also against the
results of existing research in Agile.
The interviews result indicates that some Agile-like practices had existed prior to
Agile transformation. This raises the question if one is to adopt an AMM and al-
ready have some practices established, should they then cease those practices because
the AMM suggest they are to be adopted later. This however is not described by the
AMMs. Furthermore, the AMMs also do not consider that some Agile practices are not
sustainable, thus being abandoned. Such observation is also reported by Murphy et al.
[142] and Solinski and Petersen [212]. This indicates that the suggestions provided by
the AMMs may not be sufficient for a useful implementation in industry.
When looking at the partial orders of practices, the AMMs would suggest partial or-
ders that are not present in the survey, or indicate different directionality of the ordering
of practices. This indicates that some suggestions in the AMMs are not implemented
by our survey respondents. Yet, most of our respondents perceived that their Agile
implementation to be successful. This again indicates that the order of Agile practice
introduction is not critical, but how the practices are implemented and whether they
suit the contexts. This finding indicates that the assumption that practices are to be
introduced in a specific order, as suggested in the AMMs, may be irrelevant.
We found commonalities and differences between the AMMs and our empirical
study. The differences could indicate that not all suggestions from the AMMs are to be
followed. We cannot claim that AMMs should not be adopted. Neither can we claim
that they should be. The AMMs typically do not provide sufficient information to be
used as guidelines for Agile practice adoption or Agile improvement efforts. The key
points that make the AMMs questionable are that they do not: (1) define the practices,
(2) provide motivations why the practices are mapped into specific maturity level, and
(3) consider possible negative impacts from adding Agile practices.

The result of our literature review shows that AMMs are not in agreement in terms of
suggestions or strategies to improve Agile maturity. Agile practice implementation in
industry is highly contextualized. Each organization even each team has its own flavor
of adopting Agile practices [72]. Instead of proposing yet another AMM, research
need to be directed towards understanding what exactly are the outcomes that can be
achieved if Agile maturity is improved. Furthermore, research need to be directed to

understanding the cost and benefits from improving Agile maturity, particularly the
impacts on product quality, and development time and cost.
Our study shows that there are positive and negative impacts of adding Agile prac-
tices on top of existing ones. The impacts of adding practices we found from our in-
terviews share similarities to existing studies in benefits and limitations of Agile prac-
tices [58, 141, 152]. Research efforts can be directed towards formulating a trade-off
analysis approach that can help practitioners weigh the cost-benefit of adding or even
removing Agile practices from various perspectives. Therefore, practitioners can fur-
ther amplify the benefits and minimize the challenges of Agile practices by adding the
appropriate practice and abandon practices that do not add values.
The empirical study that we conducted was mainly a retrospective study. As we
mentioned in our validity threat, retrospective study has a risk of post-rationalization.
A similar study needs to be replicated but focusing on a specific context or case involv-
ing multiple interviewees and data sources to allow triangulation, to minimize post-
rationalization [162]. The study can also be extended through a longitudinal study that
also includes observation of a team or an organization that is in the process of transi-
tioning into Agile. Therefore, actual events and impacts can be observed.
The result of our study indicates that at their current states AMMs did not provide
enough information for a useful implementation in industry. Similar indication is also
shared in previous evaluations of AMMs [127, 194]. Practitioners and managers can-
not take the suggestions as is, and need to evaluate how the existing situations in the
organization before engaging in Agile transformation using an AMM.
In this study, we identified that adding practices could have both positive and neg-
ative impacts. Existing studies have documented the different benefits and limitations
of Agile practices very well [58, 141, 152], but they have not yet captured the timing
aspect of them. The AMMs suggested that Agile practices were to be incrementally
added. Practitioners, especially managers, need to be aware that there could be adverse
impacts of adding practices. Adding practices could negatively impact the team mem-
bers’ morale. This should be avoided as the Agile manifesto suggest to focus on people
[15].
6.6 Summary and Conclusion 201
6.6 Summary and Conclusion

The aim of this study is to compare the strategies suggested in AMMs and practitioners’
strategies in industry. To see if the suggestions provided in AMMs are anyway relevant
to industry practice. The aim was achieved by: (1) conducting a literature survey, (2)
self-administered web-based survey, and (3) interviews with 11 industry practitioners
with experience in Agile software development. A combination of quantitative and
qualitative analysis methods were used to analyze the survey and interview data.
We obtained answers from 40 industry practitioners, and 11 of them participated
in the follow up interviews. From the survey, the adoption of 17 Agile practices was
examined, including when each practice started, when each practice ended or if it was
still in use. The follow up interviews covered the rationales why the practices were
adopted in different orders and the impacts of adding or discontinuing the practices. To
conclude the paper, we revisit the research questions that were formulated in the study.
RQ1. What strategies do AMMs suggest for introducing Agile practices? From the
literature review, we found two types of AMMs, those with mapping of Agile practices
and maturity level (Type 1) and those without (Type 2). Type 1 AMMs suggest that
Agile practices are to be introduced incrementally in particular orders. The Type 1
AMMs are not in agreement with respect to the order of practices. Furthermore, the
Type 1 AMMs did not provide enough information with respect to the definitions of the
practices and why the practices were assigned to different maturity levels. Without that
information, practitioners would have difficulty to assess whether a particular AMM
is suitable for them. The Type 2 AMMs were also not in agreement with respect to
improving maturity.
RQ2. What strategies do practitioners use for introducing Agile practices over
time? From the survey, we identified three strategies in introducing Agile practices:
(1) big bang - all Agile practices were introduced at the same time,( 2) incremental
- Agile practices were added incrementally, and (3) complex - Agile practices were
introduced followed by abandonment of one or more practices. The result from the
survey indicates that face-to-face meeting and coding standard tend to be introduced
first. Meanwhile, TDD tends of to be introduced later. For some respondents, some
Agile practices had existed even prior to Agile transformation.
RQ3. How do the strategies suggested by AMMs compare to the strategies used
in industry? A comparison of the AMMs and our empirical study reveals that some
suggestions from the AMMs are not representative of the industry practices. From
the comparisons, some limitations from the AMMs were identified. The AMMs sug-
gest that Agile practices are to be introduced incrementally, but our empirical study
indicates that there are more strategies to introduce Agile practices. Furthermore, the
AMMs do not consider the potential negative impacts of gradually adding Agile prac-
tices.
The result of our study indicates that at their current state, the AMMs are not suffi-
cient for useful industry implementation. There is too little information that the AMMs
provide. Also, given that different AMM gives different suggestions, practitioners do
not yet have the means to decide which one is best for them.
Further research is needed to formulate better suggestions for practitioners to intro-
duce and thus continuously improve their Agile implementation. We propose the fol-
lowing agenda for future research: (1) perform a longitudinal study with observations
of a team that is undergoing Agile transformation, (2) better understand the impact of
improving Agile maturity on product quality, development cost, and lead time, and (3)
formulate a trade-off analysis approach to help practitioners decide which practices to
add or abandon to amplify benefits and minimize challenges of Agile implementation.
Appendix
1 Type 2 Agile Maturity Models
Table A1: Other Maturity Models
ID Author title Source Maturity Levels

M13 Ellen Agile Maturity Blog 0 - Dormant; 1 - Speed: focus on being expeditious;
Gottes- model http://tynerblain. 2-Reactive: focusing on acting relative to change
deiner com/blog/2009/ from the perspective of the moment rather than a
06/30/agile- longer timeframe; 3 - Responsive: Focusing on act-
maturity-model ing relative to change from the perspective of the
moment balanced with a longer timeframe.

M14 Scott Am- The Agile IBM Rationale Core agile development - Core agile methods, such
bler Scaling Model white paper as Scrum and Agile Modeling, adopt Scrum prac-
(ASM): Adapting tices optimized for small, co-located teams devel-
Agile Methods oping fairly straightforward systems; Disciplined
for Complex agile delivery - Disciplined agile delivery pro-
Environments cesses, which include Dynamic System Develop-
ment Method (DSDM) and Open Unified Process
(OpenUP), from project inception to transitioning
the system into production environment; Agility at
Scale - Focuses on disciplined agile delivery where
one or more scaling factors are applicable. There
are eight scaling factors, they are are team size, geo-
graphical distribution, regulatory compliance, orga-
nizational complexity, technical complexity, organi-
zational distribution, and enterprise discipline.
Continued. . .
203
ID Author Title Source Maturity Levels 204
M15 David An- Agile Man- Book Stage 0 - Decompose system input into basic units
derson agement for of measurement. Stage 1 - End-to-End Traceabil-
Software Engi- ity Implement system for capturing & monitoring
neering, Apply- measurements; Stage 2 - Demonstrate basic statis-
ing the theory of tical process control—show that system is stable
constraints for against a target and within a tolerance; Stage 3 -
business results System Thinking and a Learning Organization Fo-
cus on continuous improvement; Stage 4 Antici-
pated ROI and the Failure Tolerant Organization En-
courage risk taking. Focus on Throughput, and mar-
ket research/feedback (i.e., external constraints).
Continued. . .
Strategies to Introduce Agile Practices
ID Author Title Source Maturity Levels
M16 Scott Am- The Agile matu- Blog Level 1 - Rhetorical: Agile strategies were applied
bler rity model www.drdobbs.com/ on hand-picked pilot projects, by a small team of
architecture -and- flexible and often highly-skilled people, and were
design/the-agile- given sufficient management support; Level 2 - Cer-
maturity-model- tified: Many of team members have obtained some
amm/224201005 form of agile certification; Level 3 - Plausible: Or-
ganization starts to focus on agile strategies which
may in fact be applicable at organizational level. Or-
ganization faces issues pertaining to scaling; Level
4 - Respectable: Agile teams adopt a full deliv-

ery lifecycle. Agile teams are able to tailor their
approach to meet the unique needs of the situation
in which they find themselves. The teams are self-
organizing; Level 5 - Measured: The organization
are able to combine Agile, lean and other techniques
to be as effective as possible. The emphasis is on
steering projects based on real empirical information
and not just observational guesswork.
Continued. . .
205
M17 Martin Yet another agile Blog Level 1 - Team Level: Team members have decided
Proulx maturity model - https://setandbma. to adopt Scrum and/or software engineering prac-
the five levels of wordpress. tices without asking for approval from their man-
maturity com/2011/11/30/ ager; Level 2 - Department Level: Other teams
agile-maturity- started to copy the team that have implemented Ag-
model/ ile practices and some managers have become aware
of agile approach; Level 3 - Business Level: The
business people have been included in the develop-
ment process. A Product Owner is clearly identi-
fied and may be dedicated to their project; Level 4
- Project Management Level: The project man-
agement approach is modified to include some of
the Scrum practices. A strong evangelist is in place
at the management / executive level to promote the
new approach; Level 5 - Management Level: Man-
agers have adapted their management style to sup-
port an agile organization. Organizational structures
and reporting mechanisms are better adapted for col-
laboration and improved for increased performance.
Management is considering implementing Agile to
projects that do not require software development;
Level 6 - Corporate level
Continued. . .
M18 Scott Agile maturity Blog http:// Agile hierarchy of needs: (1) Staffing the engi-
Sehlhorst model - whats tynerblain. neering team correctly, (2) People over process is
next? com/blog/2009/ the right emphasis, (3) Assuring Quality is in your
06/30/agile- team’s DNA, (4) Reducing overhead in the release
maturity-model/ process, (5) Managing stakeholder expectations, (6)
Continuously learning from your markets.
M19 Angela Agile transforma- Presentation 1. Exploration: a) pilot projects b) training (ini-
Druckman tion strategy slides tial) c) practice; 2. Coordination: a) training
https://www. (wide scale) b) technical best practices c) develop-
open.collab.net/ ment of internal “Agile advocates” d) tool explo-

media/pdfs/Agile ration; 3. Process Definition: a) tools formaliza-
Transformation tion b) project management office (PMO) involve-
Strategy Collab- ment c) formalization of the ScrumMaster and Prod-
NetWebinar.pdf uct Owner groups; 4. Strategic Alignment: a) coor-
dination of Agile practices within management, hu-
man resources and other groups of the enterprise; 5.
Transformation: a) Agile principles and activities
are internalized by the organization
M20 Nick Malik Simple lifecycle Blog https:// (1) Write a simple story that describes the process
agility maturity blogs.msdn. you followed, (2) Rate your process on 12 criteria
model microsoft. based on the Agile Alliance principles Enter weights
com/ and view results, (3) Create a list of steps to address
nickmalik/ deficiencies, (4) Follow the normal agile process to
2007/06/23/ estimate these steps and add to the backlog.
Continued. . .
207
M21 Ravi Gujral The Agile matu- Blog http://www. The practices can be classified as: (1) Regressive,
and Shaun rity model shaunjayaraj. (2) Neutral or Chaotic, (3)Collaborative, (4) Op-
Jayaraj com/2008/08/agile- erating (Consistent exhibition of competence, (5)
maturity- Adaptive (Expertise to adapt to change), (6) Innovat-
model.html ing (Creative evolution of practice, and spread these
practices throughout the organization).
M22 Jez Humble The Agile Matu- ThoughtWorks Level -1: Regressive: process unrepeatable, poorly
and Rolf rity Model Ap- white paper controlled and reactive; Level 0 -repeatable: pro-
Russell plied to Build- cess documented and partly automated; Level 1
ing and Releasing - Consistent: automated process applied across
Software whole application lifecycle; Level 2 - Quantita-
tively managed: process measured and controlled;
Level 3- Optimizing: focus on process improve-
ment
Continued. . .
M23 Erick Continuous De- Urbancode Base: The team has left fully manual processes be-
Minick livery Maturity whitepaper hind; Beginner: The team is trying to adopt some
Model Evolutionary Continuous Delivery (ECD) practices
in earnest but is still performing them at a rudimen-
tary level; Intermediate: Practices are somewhat
mature and are delivering fewer errors and more effi-
ciency. For many teams, Intermediate practices may
be sufficient. Advanced: The team is doing some-
thing well beyond what most of the rest of the indus-
try and is seeing a great deal of efficiency and error

prevention as a result. Extreme: Elements within
the Extreme category are ones that are expensive
to achieve but for some teams should be their tar-
get. Put another way, most organizations would be
crazy to implement them, while this minority would
be crazy to not implement them.
M24 Shirely Ro- Agile testing ma- Presentation waterfall – forming – agile bonding – performing –
nen turity models slides scaling
https://www.
slideshare.net/
AgileSparks/atmm-
practical-view
Continued. . .
209
M25 Shirley Ro- Scrum maturity Presentation Level 1 - technical: team formation by manager;
nen models slides Level 2 - mindset: team formation by team; Level
https://www. 3 - scale.
slideshare.net/
AgileSparks/atmm-
sl-relation-
practical-view
M26 Dan Woods An Agile BI Ma- Magazine article Stage 1 - No Agile BI; Stage 2 - Early Adoption:
turity Model https://www. in early adoption, companies bring in agile BI sys-
forbes.com/sites/ tems, but only for their cost and time savings over
danwoods/ enterprise BI; Stage 3 - Self Service: In this stage,
2011/10/26/an- users realize that Agile BI can do more than help
agile-bi- them work faster and save money. Now, they can
maturity-model/ actually begin to put BI in the hands of more peo-
#522d43795960 ple; Stage 4 - The Lake Effect: They’ve adopted
the agile BI technology, they’re achieving time and
cost savings.
2 Demographics Information 211
2 Demographics Information
Table B1: Roles of the survey respondents
Role Responses* Percent of Cases

Programmer 16 40%
Project Manager 5 12.5%
Business Analyst 3 7.5%
Systems Analyst 5 12.5%
Systems Architect 7 17.5%
Quality Assurance specialist 10 25%
Trainer 4 10%
Scrum Master 14 35%
Product Owner 4 10%
Consultant 1 2.5%
Department Head 7 17.5%
C-Level manager (CEO, CTO, etc) 3 7.5%
Other 4 10%
Total 83 207.50%
* each respondent could select multiple roles
Table B2: Experience of the survey respondents
Years of experience Frequency Percent

<1 year 5 12.5%
1 – 3 years 11 27.5%
3 – 6 years 12 30%
6 – 10 years 7 17.5%
>10 years 5 12.5%
Total 40 100%
Table B3: Size of software development team of the Survey Respondents
Organization size* Frequency Percent

<50 people 21 52.5
50 – 249 people 11 27.5
250 – 4999 people 7 17.5
>5000 people 1 2.5
Total 40 100
*size of the software development team not the entire company
Table B4: Industry Domains of the Survey Respondents
Domain Responses* Percent of Cases

Independent Software Vendor (ISV) 15 37.5%
Financial 10 25.0%
Media 2 5.0%
Medical 5 12.5%
Manufacturing 1 2.5%
Telecommunication 8 20.0%
Research & Development 8 20.0%
Government 2 5.0%
Other 4 10.0%
Total 55 137.5%
* each respondent could select multiple domains
3 Descriptive Statistics
3 Descriptive Statistics 213
Table C1: Descriptive statistics of the Agile practices and the maturity level in which
they were mentioned in the AMMs
Agile practices Mentions Mean Median Minimum Maximum

Continuous integration 9 2.44 2 1 4
Planning game 9 1.44 1 1 4
Tracking progress 9 2 2 1 3
Coding standard 8 1.38 1 1 2
Onsite customer 8 3 2.5 1 5
TDD 8 2.63 1 1 5
User stories 7 1.71 1 1 4
Short iteration 7 1.86 1 1 4
Configuration mgt. 7 2 2 1 5
Pair programming 7 2.29 2 1 5
Self org. Team 7 3.43 3 3 4
Unit testing 6 2 2 1 3
Frequent releases 6 2.67 2.5 1 4
Refactoring 6 2.5 2.5 1 4
Face-to-face meeting 6 2.33 2 2 3
Retrospective 6 2.17 1.5 1 5
Standup meeting 5 2.2 1 1 4
Simple design 4 2.25 2.5 1 3
Metaphor 4 2.25 2 1 4
Collective ownership 4 2 2 1 3
Backlog 4 2.5 2.5 1 4
40h week 3 3 3 2 4
Continuous delivery 3 3 2 2 5
Adaptive planning 2 4 4 4 4
Definition of done 2 2.5 2.5 2 3
Acceptance testing 2 1 1 1 1
Code reviews 1 1 1 1 1
Timeboxing 1 1 1 1 1
Cross functional team 1 3 3 3 3
Kanban 1 3 3 3 3
DevOps 1 3 3 3 3
Table C2: Descriptive statistics of the Agile practices and the maturity level in which
they were mentioned by the survey respondents
Agile Practice Mentions Mean Median Minimum Maximum

Face-to-face meeting 37 1.32 1 1 5
Self org. & cross func. Team 36 2.11 2 1 7
Tracking progress 36 1.78 2 1 5
Retrospective 36 1.83 1 1 6
Planning game 35 2.06 2 1 5
Timeboxing 35 1.86 2 1 5
Stand up meeting 34 2.12 2 1 6
Continuous integration 33 2.39 2 1 7
Short iteration 32 1.94 2 1 5
Refactoring 28 1.89 2 1 5
Coding standard 28 1.61 1 1 5
Collective ownership 28 2 2 1 5
Metaphors & user stories 27 2.19 2 1 5
Simple design 25 1.92 2 1 6
On-site customer 24 2 2 1 4
Pair programming 20 2.6 2 1 7
TDD 16 3 3 1 6
4 Heatmap Tables
Table D1: AMM heatmap.
4 Heatmap Tables
215
Table D2: Survey heatmap.

5 Respondents’ Quotes from Section 6.4.2
Table E1: Impact of Adding Practices: Interviewees’ Quotes
Practice Impact
Positive No impact Negative
On-site customer “Improves coordination with customers.” “Sometimes the team members can get a bit
(R34) uncomfortable. Some of our teams members
experience some stress.” (R34)
“We can organize our backlog better we can
organize our sprints because we know the
highest priority features or stories in the next
given sprints.” (R35)
“It was very quick to get something like
questions being answered.” (R14)
Sprint planning “Sprint planning makes our work more or-

ganized and responsibilities are more clear.”
(R34)
Refactoring “Refactoring also helps with to improve
one’s understanding of roles and responsibil-
ities and what their tasks are.” (R34)
“To introduce design pat- “The big thing was the visibility on the other
terns and upgrade the ar- parts of the projects, so especially because of
chitecture was a pain for the hardware components.” (R32)
some people.” (R14) Con-
tinuous integration
“We have frequent builds coming from dif-
ferent branches that needs to be refined and
looked up because you could have someone
testing a build from one branch but theres a
newer build of another branch and you know
you’re getting people reporting bugs on the
old one that was already fixed.” (R35)
217
Continued. . .
Practice Impact 218
Coding standard “I do not think the dif-
ferences were big. It is
not like we are worried
about not understanding
each other’s code.” (R32)
Stand up meeting “Gives me a sense of better control issues are “The daily meeting requires time. Every-
easier to find.” (R11) body needs to be present if you want to be
effective.” (R11)
Retrospective “It makes our developer lead knows the skill
set of each developer better.” (R11)
“Retrospective is very important so you can
find out what the team like and what they do
not like. What they would rather do and what
they think they did well, even sometimes get-
ting them to see what they did well.” (R35)
Timeboxing “Allows to see what the end product might
look like.” (R11)
Short iteration & “Allows to see results quickly.” (R11) “There are conflicts between development
frequent releases and operations. Timeline wise, there are
efforts to align development and operations
sometimes we [development] are ready to
deploy, but they [operations] are not.” (R11)
“Give them a sense of completion and
job satisfaction and closing something off.”
(R35)
Pair program- “With pair programming, it is less likely “People were uncomfortable and people did
ming that there would be one developer, sitting on not really want to engage in that.” (R32)
something. So we are definitely moving to-
wards collective ownership.” (R35)
User stories “They have helped us to clarify requirement
and remove ambiguity.” (R35)
Continued. . .
Practice Impact
TDD (automated “The good thing I could find some prob-
testing) lem in the code because of automated testing
[TDD].” (R14)
219
220
Table E2: Impact of Removing Practices: Interviewees’ Quotes
Practice Impact
Tracking “Positive because it makes no “I am not sure the team felt the “As scrum master and prod-
progress sense to track any other progress impact.” (R38) uct owner, you want to keep
than if its ready or not. If track so you do not just add
you have a big epic tasks, that things.”(R38)
takes weeks to complete, I guess
you need some sort of tracking.”
(R40)
“We had one PO (from an-
other company) who did not care
so much for tracking progress.”
(R32)
“Because we did not have good
sprint planning, there were basi-
cally no impact, because track-
ing progress never worked.”
(R14)
11 practices “One thing is lack of trans-
discontinued parency.” (R33)
On-site “If on-site customer means
customer someone sitting in our office,
we only had it for a short time”
(R14)
Chapter 7
Capturing Baseline Situations in

Studying the Impacts of Agile
Practices Introduction: A
Checklist and Propositions for
Future Research
This chapter is an extension of the following paper:

Indira Nurdiani, Jürgen Börstler, Samuel A. Fricker, Kai Petersen, “A
Preliminary Checklist for Capturing Baseline Situations in Studying the
Impacts of Agile Practices Introduction”, accepted for publication in the
Sixth International Workshop on Conducting Empirical Study in Industry
(CESI), 2018, (DOI: 10.1145/3193965.3193969).
7.1 Introduction
Agile practices are usually introduced to complement or replace existing processes and
practices [167, 121, 212]. There are many empirical studies aimed at studying the
benefits and limitations of introducing Agile practices [58]. However, these empirical
studies report contradictory results on the impact of Agile practices [58, 152]. It is often
difficult to compare results across empirical studies because the baseline situation, i.e.
situation prior to Agile practice introduction is not reported [167]. Describing the
baseline situation is important to fully understand the extent of improvement [111], or
the lack thereof, from introducing Agile practices.
Current guidelines do not detail which information should be captured when de-
scribing the baseline situation prior to introducing Agile practices. Potentially, all
context information reported in the literature could be relevant. However, reporting
“everything” may not be feasible as it is too effort-intensive to collect and report. In
this chapter, we propose a checklist for baseline information that should be included
when reporting the impacts of introducing Agile practices. The description of baseline
information will make it easier to compare the effects of introducing Agile practices as
a complement, or a replacement, to existing practices. The checklist has been validated
by a number of experts in academia. From the checklist, we derived and formulated
four propositions which can be used to frame future research in Agile software devel-
opment.
The remainder of the chapter is structured as follows. Section 7.2 summarizes other
checklists proposed for describing contexts in empirical studies in software engineer-
ing. Section 7.3 describes the research methods used to develop the checklist. Section
7.4 presents the checklist and the propositions. Section 7.5 provides brief discussions
on the checklist and propositions. Lastly, the chapter is summarized and concluded in
Section 7.6.
7.2 Related Work

There are a number of checklists or guidelines to report contexts of empirical studies
in software engineering. Kitchenham et al. [111] suggest that the following contex-
tual information is to be provided: (1) the objectives of the study, (2) the baseline that
the results are compared, and (3) external constraints. However, the checklist does
not detail what information should be captured to describe the baseline. Petersen and
Wohlin proposed that contextual information pertaining to process, product, practices
and tools, people, organization, and the market should be captured [166]. The check-
list provides a general overview of the contextual information that should be captured
but does not indicate which information is relevant when comparing the adoption of
agile practices with a particular baseline. Currently, there is no guideline that supports
researchers to capture the baseline situation when reporting the impacts of introducing
Agile practices.
Scope of the guideline. In developing the checklist, we focus on information rele-
vant to the internal constituents of a software development organization. Constituents
are the components that make up a software organization, they are: (1) workforce –
human resources and their skills and knowledge, (2) management and organization
structure – deals with leadership, the organization hierarchy and roles, and coordina-
tion between the constituents, (3) process – concerted activities that takes place in the
organization, (4) infrastructure – technology platforms required to conceive, build, and
release the product or service [152, 218]. The focus on internal constituents is selected
because introducing Agile practices can affect these constituents [153].

The checklist was developed from a literature study [153] and interviews with 11 in-
dustry practitioners. Four propositions were also formulated as the by-product of the
checklist. The resulting checklist and initial propositions were then validated with
seven experts in academia, see Figure 7.1.
Checklist Components
Identification &
Proposition Formulation
Checklist &
Practitioners Proposition Validation
Interviews Experts
Questionnaire
Literature Study
Figure 7.1: Overview of Research Methods
Practitioners Interviews. The interviews were done as a follow-up from a survey.

The survey design can be found in Appendix B of this thesis. In the survey, we inquired
the respondents to indicate when (in month and year) Agile practices started being
adopted, and if they are still in used. The follow up interviews inquired: (1) rationales
for introducing the Agile practices at the time that they were introduced, (2) rationales
for abandoning one or more practices, and (3) the impacts of adding, or abandoning
Agile practices over time. From the interviews, we identified the potential components
of the checklist, which was then complemented by the literature review [153].
Eleven practitioners with experience in Agile projects participated in the inter-
views from 11 different organizations from various domains such as financial services,
telecommunication, biomedical research, etc. The roles of practitioners include devel-
opers, testers, project/ program managers, and Scrum masters. The interviewees also
came from different countries, e.g., Sweden (4 interviewees), Indonesia (2 intervie-

wees), Brazil, Ireland, Finland, USA, and Canada (each represented by one intervie-
wee). The list of interviewees can be seen in Chapter 6 in Subsection 6.3.3.
Literature review. The aim of the literature review was to identify flexibility at-
tributes in inter-disciplinary literature and map them against Agile practices. The de-
tailed description of the literature review process can be found in [153]. The literature
review identified known techniques and principles in management and manufacturing
that are well aligned with Agile practices and principles. This was used to complement
the findings from the interviews.
Experts Questionnaire. We contacted experts in academia to validate the con-
tent of the checklist and obtain feedback for further improvements. The experts were
selected given their publication activities in Agile methodology, or past and current
participation as PC members in Agile conferences. A paper by Chuang et al. [39]
which include a list of leading researchers in Agile software development was used
as a starting point to identify the potential experts. We described the components of
the checklist in a short presentation slides. The slides also include descriptions of how
each checklist components were identified. We also included propositions for future
research derived from the checklist components. Together with the presentation slides,
a short questionnaire was attached to inquire the experts’ perceptions on the useful-
ness and relevance of the identified components to research and their suggestions for
improvements. The questionnaire was self-administered. However, if the expert was
available, a video call was also established. In total, we obtained feedback from seven
experts, the list of experts can be seen in Table 7.1.
Table 7.1: List of academic experts.
ID Position Higher Education Institution

Expert 1 Professor Technical University of Denmark, Denmark
Expert 2 Professor Reutlingen University, Germany
Expert 3 Professor University of Innsbruck, Austria
Expert 4 Professor The Open University, UK
Expert 5 Professor Malmo University, Sweden
Expert 6 Associate Professor Chalmers University of Technology, Sweden
Expert 7 Senior Research Associate Clausthal University of Technology, Germany
We also sent the experts the preliminary version of the propositions. We inquired
from the experts their general perception on the propositions and also feedback on how
to improve the propositions. From the experts feedback the propositions were then
modified.
7.4 Results: Capturing Baseline Situations 225
7.4 Results: Capturing Baseline Situations

7.4.1 Preliminary Checklist for Capturing Baseline Situations
The checklist is structured according to the constituents described in Section 7.2. An
overview of the checklist is shown in Table 7.2. In the following paragraphs, we de-
scribed each component of the checklist.
Table 7.2: Baseline Context Information Checklist
To what extent can one judge the improvement from Suggested scale
implementing Agile practices given the following baseline:
Workforce 1. Constellation of team members’ competencies,
experience, and skills:
1a. Team members’ product knowledge. no/basic (%) - sufficient (%) - good (%) - in-
depth (%)a,b
1b. Software development experience. immature (%) - little experience (%) - experi-
enced (%) - very experienced (%) a,b
1c. Team members technical and non-technical enumerate (e.g., coding skills, project manage-
skills. ment skills, etc.)
Management and 2. Management principles that are in place:
Organization struc- 2a. Organizational structure hierarchical - flat/matrix
ture
2b. Self-organization Tuckman’s stage of group development [221]
2c. Openness - willingness of managers to accept Tannenbaum and Schmidt’s leadership contin-
new ideas, or redefine roles uum [217]
2d. Recursiveness - the ability of managers to No - Yes
switch between planning and implementation
Process 3. Existing practices or activities
3a. Formal/informal practices that are similar to enumerate (e.g., coding in pairs, improve in-
Agile practices ternal code structure, etc.)
Infrastructure 4. System characteristics:
4a. Existing legacy code No - Yes
4b. Product components modularity Monolithic - Modular (cohesion and coupling
measures could be used, see [98])
4c. Hardware component Embedded systems - minimum hardware com-
ponent - no hardware component
a
The scale is adapted from [26].
b
indicate the % of the team members within the respective level.
Workforce. In manufacturing and management, hiring highly skilled individual

is important to be flexible [153]. Four of our interviewees (R35, R36, R38, and R40)
indicated that having good technical skills, product knowledge, and development ex-
perience is important for successful Agile practice adoption. One of them (R38) men-
tioned the lack of product knowledge led to the discontinuation of tracking progress.
Another interviewee (R35) also mentioned that she provided training in estimation
(non-technical skills) so the team members are able to do planning poker. See Table
7.3 for the supporting interviewees’ quotes.
Table 7.3: Supporting Interviewees’ Quotes - Workforce
Interviewees’ Quotes
R35: “We started out with having story points in place but they [the developers] could not grasp the
abstract concept of story points [..]. So we manage it in time [number of hours]. So they can just get a feel
for estimation. So then we decided that we want to go to proper story points so we can get a velocity for a
more accurate release planning.”
R36: [In the end of a sprint] sometimes they [team members] spend 1-2 days extra to make it [the code]
a bit better. [..] but that is also connected to the team’s maturity, if you have more mature team they used
to stay one or two days more to make sure that they leave the code in a really good shape or as good as
possible for that time.
R38: “They didn’t have a good product knowledge. Sometimes they complain we do not have the product
knowledge how do we estimate it and we do not know the complete technicalities.”
R40: “I think we are pretty good [experienced, and having good technical skills], if you are good, if you
believe in what you are doing then I do not think implementing this kind of like agile are required [..]. The
team consists of very strong individual who are experienced”
This raises the question of whether the constellation team members’ competen-
cies, experience, and skills could have an impact on the outcome from adopting Agile
practices. If a large percentage of the team members are highly skilled with in-depth
product knowledge, they are less likely to encounter problems with the introduction of
Agile practices. This indicates the importance of capturing baseline context informa-
tion on the competencies, experience, and skills of the team members. In the check-
list (see Table 7.2) we recommend to capture the % of team members given their levels
of product knowledge and experience, and to enumerate technical and non-technical
skills.
Management and Organization Structure. Management literature suggests a num-
ber of principles to be flexible, e.g., adopt a flat organization structure with self-organizing
team, promote openness and recursiveness [153]. These principles are well aligned
to Agile principles and practices. One interviewee (R32) indicated that the company
owners exemplified openness by allowing the team to be self-organized. Meanwhile,
another interviewee (R11) suggested that her team has always been self-organized even
prior to Agile transformation. See Table 7.4 for the supporting interviewees’ quotes.
This led to the question of whether the existing management policies and orga-
nization structure could have an impact on the outcome of adopting Agile practices.
A team whose members are accustomed to being self-organized with a manager that
promotes openness is less likely to develop friction with Agile way of working. Such
a team will have better chance to succeed with Agile transformation, compared to a
team that is not accustomed to being self-organized. This indicates the importance
Table 7.4: Supporting Interviewees’ Quotes - Management and Organization Structure
R11: Self-organizing team was done before we implemented Agile.
R32: “I think the biggest factor by far was the commitment from the partners [owners] to the agile princi-
ples and values. It is not easy for a business owner to allow their employees to self organize. its actually
very risky and requires a high degree of trust.”
of capturing baseline context information pertaining to management principles and

organization structure when studying the impacts of introducing Agile practices. In
the checklist, we recommend different scales to indicate the information pertaining to
existing management principles and organization structure.
Process. Six of our interviewees indicated that the practices that are often asso-
ciated with Agile had existed prior to Agile transformation. These practices include
face-to-face meeting (R11, R14, R33, R36), pair programming (R36 and R37), cod-
ing standard (R11, R33, R36 and R38), refactoring (R11 and R36), short iteration &
frequent releases, and self-organizing cross functional team (R11). For some intervie-
wees, these practices were done as informal practices, i.e. only within the interviewees’
teams or projects. See Table 7.5 for the supporting interviewees’ quotes.
Table 7.5: Supporting Interviewees’ Quotes - Process
R11: “The practices that we adopted first [face-to-face meeting and coding standard] do not correlate with
the methodology, [...] they are just good practices”
R11: “Refactoring has been used since before using Agile.”
R11: “Short iteration and frequent release existed before Agile, but it was every 2-3 months.”
R11: “Self-organizing team was done before we implemented Agile.”
R14: “Face-to-face meeting they’ve always done that.”
R33: “[The company] started practicing Agile since Q3 2012, in that period they already had coding
standard.”
R33: “Face-to-face meeting existed before Agile.”
R36: “We have always had face-to-face meeting even before we went into the agile transition.”
R36: “We have had documents on how to code [coding standard]. We have had that for many years, even
before Agile.”
R36: “We have done [refactoring] for years. That was not a big issue for us when we went into Agile.”
R36: “We have done pair programming, we have been sitting in one computer even before 2007 [Agile
transformation].”
R37: “Pair programming it was something that we started before we actually went into Agile.”
R38: “I think [coding standards] have been there even before [Agile transformation].”
It is possible that Agile-like practices have existed in a software development team,

either as formal or informal practices. This led to the question of whether the existing
formal or informal practices could have an impact on the outcome of adopting Agile
practices. A team that has had formal or informal practices similar to Agile has an ad-
vantage over those that do not because they may require less learning time and are used
to the Agile way of working. Furthermore, a number of our interviewees indicate that
Agile practices need one another to work, like retrospective and continuous integration,
or tracking progress and planning game. If a team has one of the practices already es-
tablished prior to Agile transformation, it would be easier to introduce the other Agile
practices. This indicates the importance of capturing baseline context information on
existing formal or informal practices when studying the impacts of introducing Agile
practices. In the checklist, we recommend to enumerate as many Agile-like practices
as possible.
Infrastructure. Two interviewees (R11 and R14) indicate that improving the mod-
ularity of the product was needed to introduce some Agile practices, like continuous
integration and Test-Driven Development (TDD). Two other interviewees indicate that
legacy code (R35 and R37) prevented them to adopt some Agile practices, e.g., TDD
and simple design. With respect to hardware dependencies, two interviewees (R32 and
R35) mentioned that the hardware components in their development posed challenges.
The challenges are primarily because the development cycle for the hardware was dif-
ferent from the software part. For example, hardware may not be ready for the start of
the next sprint. See Table 7.6 for the supporting interviewees’ quotes.
Table 7.6: Supporting Interviewees’ Quotes - Infrastructure
R11: “After we have good application design using microservices, then we started to do continuous inte-
gration.”
R14: “The software architecture did not let us do use automated testing, so the first thing we did was to
refactor the architecture and improve it in a way that would be able to do automated testing [TDD].”
R32: ”It is very very hard to do short sprints with a hardware components in your teams. Simply the lead
times make it pretty much impossible [...] because that is the time it takes for the [hardware] parts to even
show up.”
R35: “We are left with a mess from the previous development team. We are adding and maintaining the
legacy we are left with to get the product to the market.”
R35: “What we are doing it is quite complicated, there is firmware, hardware, software. There are several
types of hardware and that all have to run together so it has been quite difficult.”
R37: “We have a lot of [legacy] in our code, [it was not easy] for us just to jump into [TDD] [because] the
old code was not done in that way.”
This led to the question whether the internal design of the product and hardware de-
pendencies could have an impact on the outcome of adopting Agile practices. A team
that maintains a well-designed, less complex and modular system is less likely to per-
form major refactoring to enable the introduction of some Agile practices. Hardware
components may not prevent a team to adopt Agile. Certainly it could pose challenges,
like longer lead time. Thus, the extent of hardware components may influence the
impacts (positive or negative) from adopting Agile practices. This indicates the impor-
tance of capturing a baseline information on the existing systems characteristics. In
the checklist, we recommend scales to indicate information pertaining to the product
internal structure and hardware dependencies.
7.4.2 Derived Propositions

We formulated a number of propositions based on the components of the checklist.
From the interviews, we inferred that the each of the checklist component can influence
the extent of benefits of introducing Agile practices. The propositions posit that extent
of improvements from introducing Agile practices could be influenced by a baseline
situation. Table 7.7 summarizes the propositions and their explanations.
From Table 7.7, the cause constructs are the components of a baseline situation.
The effect constructs are the benefits of introducing Agile practices. The potential
benefits of introducing Agile practices could be amplified or reduced depending on the
state of each component of a baseline situation. The suggested scale in the checklist
can help describe the states of each component of a baseline situation. For example,
team members with in-depth knowledge and flat organization structure are preferable to
support Agile practice introduction and are more likely to lead to successful outcomes.
7.4.3 Experts Feedback

We received feedback on the checklist components from seven experts in academia.
We highlight their feedback on the checklist components based on their usefulness and
relevance to research, and novelty.
Usefulness and relevance to research. The experts agree that the components of
the checklist could be useful and are relevant to research. The experts stated that the
components of the checklist could be used to help guide other or future research in
Agile software development. Some experts provided more details, on how the checklist
could be used for:
• Predicting effort and cost of Agile transformation (Expert 1).
• Prioritizing the pre-requisites for transitioning into Agile (Expert 2, Expert 5).
Table 7.7: Proposition
Proposition Explanation
P1 P1a. The constellation of the workforce can amplify Introducing Agile practices is more likely to show posi-
the benefits of introducing Agile practices tive impacts, if the majority (in %) of team members have
in-depth knowledge (1a), are very experienced (1b) and
have many technical and non-technical skills (1c).
P1b. The constellation of the workforce can reduce Introducing Agile practices is more likely to show
the benefits of introducing Agile practices no/negative impacts, if the majority (in %) of team mem-
bers have no/basic knowledge (1a), are immature (1b)
and have a few or none technical and non-technical skills
(1c).
P2 P2a. Management principles and organization struc- Introducing Agile practices is more likely to show posi-
ture can amplify the benefits of introducing Agile tive impacts, if the organization structure is flat (2a), the
practices team is at performing stage (2b), the manager allows free-
dom to the team members (2c), and the manager allows
recursiveness (2d).
P2b. Management principles and organization struc- Introducing Agile practices is more likely to show
ture can reduce the benefits of introducing Agile prac- no/negative impacts, if the organization structure is hier-
tices archical (2a), the team is at norming stage (2b), the man-
ager provides little freedom to the team members (2c),
and the manager does not allow recursiveness (2d).
P3 P3a. Existing process in the team can amplify the ben- Introducing Agile practices is more likely to show posi-
efits of introducing Agile practices tive impacts, if the team has many Agile-like practices in
place (3a).
P3b. Existing process in the team can reduce the ben- Introducing Agile practices is more likely to show
efits of introducing Agile practices no/negative impacts, if the team has none Agile-like prac-
tices in place (3a).
P4 P4a. The infrastructure or systems characteristics can Introducing Agile practices is more likely to show posi-
amplify the benefits of introducing Agile practices tive impacts, if the system has no legacy code (4a), mod-
ular component design (4b), and no hardware component
(4c).
P4b. The infrastructure or systems characteristics can Introducing Agile practices is more likely to show
reduce the benefits of introducing Agile practices no/negative impacts, If the system includes legacy code
(4a), has monolitic component design (4b), and has a
large extent of hardware components (4c).
• Increasing the acceptance of Agile practices (Expert 2).
• Measuring improvements from introducing Agile practices (Expert 4).
• Self-reflection for researchers to improve their study design (Expert 4).
• Reflecting challenges for Agile practices to be implemented successfully (Expert

5).
• Providing common understanding when reporting Agile implementation (Expert

5).
7.5 Discussion 231
The experts also had some criticisms, first was on completeness; missing other
factors in general or missing factors pertaining to Lean. The second criticism was
on specific contextual factors such as industry domain or geographical location. The
experts believed that some factors, like organizational structure, could be affected by
culture. We agree that culture might have a role in shaping the factors we described
in the checklist, but we only included factors that were identified from the interviews.
Another expert also mentioned that the factors we described in the checklist could be
difficult to measure, to address this issue, we proposed a scale for each factor of the
checklist.
Novelty. The experts agree that each component of the checklist is not new. How-
ever, they agree that the novelty lies in the aggregation of the components and making
them explicit as a checklist.
Feedback on the propositions. In developing the propositions, we also consulted
with the same experts who validated the checklist. In general, the experts mentioned
that they appreciate how the propositions were formulated. However, the experts sug-
gested that the propositions should be accompanied with detailed explanations and
need to be balanced, i.e. cover both positive and negative associations. Based on the
experts’ feedback, we reformulated the propositions as described in Table 7.7.
7.5 Discussion
The individual component of the checklist is not new. It is known that team members
skills and experience can impact the outcome of introducing pair programming [8].
Concerns pertaining management support have been reported as a possible challenge
when introducing Agile practices [146]. The experts whom we consulted also agree
that the components are not new. However, the novelty lies in the perspective we put
on the components, i.e. framing them as baseline situation that needs to be captured
when studying the impacts of introducing Agile practices. The experts also agree that
aggregating the components of the checklist and making them explicit is where the
novelty and usefulness lie. As we can reflect upon our own experience in conducting
the interviews, the practitioners themselves were not aware of the baseline situation.
We identified the relevant information on the baseline situation by asking the intervie-
wees to reflect on when Agile practices were introduced and why they were introduced
at that particular time.
The issue of completeness was raised by a number of the experts. We do not claim
to have a complete list of information for capturing baseline situation. Similar to the
checklist described in [166], this checklist is proposed to “raise awareness” on the
importance of capturing baseline context information, and the possible overlap between
them, a sentiment shared by some of the experts. For example, there could be overlap
between the state of product internal design and existing formal or informal practice;
product modularity could be the by-product of the processes, e.g., refactoring, code
review, etc. We also suggest to use this checklist to complement existing checklists,
e.g., [111, 167].
A study may identify multiple states and organizations may have multiple base-
lines. A study may focus on two more of them. We argue that at least two states need
to be described with the support of the checklist to reason on effects. Effects could be
value delivered to the customer, product quality, delivery time and cost, and customer
relationships (see, e.g., [212]). Reporting effects from each baseline is important be-
cause practitioners may have different priorities on how to measure success (lesson
learned from Chapter 5). From a reporting point of view, we suggest to explicitly
name the state/baseline, the baseline characteristics, and the desired/measured effects
to make changes (deltas) explicit and traceable. Figure 7.2 illustrates possible states
included in a study.
States S1 S2 S3 S4
Workforce Workforce’ Workforce’’ Workforce’’’
Man. & Org. Man. & Org.’ Man. & Org.’’ Man. & Org.’’’
Baselines
Process Process’ Process’’ Process’’’
Infrastructure Infrastructure’ Infrastructure’’ Infrastructure’’’
Value Value’ Value’’ Value’’’
Quality Quality’ Quality’’ Quality’’’

Eﬀects
Time Time’ Time’’ Time’’’
Customer Rel. Customer Rel.’ Customer Rel.’’ Customer Rel.’’’
Scope of study/
set of studies
Figure 7.2: Checklist Usage
As we can see from Figure 7.2 that each state would have its variations of effects,
could be positive and negative. For example, in state S2 the organization had a hi-
erarchical organization structure, and in S3 they had reconfigured their organization
7.6 Conclusions 233
structure into a flat organization structure. The effects from S3 would be different from
S2; there could be improvements made on the effects. Then in S4, Agile practices
are being introduced. The improvements, or the lack thereof, of Agile practices intro-
duction would be affected by the previous states. Without the organization structure
reconfiguration done in S3, the introduction of Agile practices would be more chal-
lenging. Thus, hindering potential benefits from Agile practices from being realized.
We also formulated four propositions derived from the checklist components. As
mentioned earlier, the constructs described in the propositions are not new. They have
been described success factors or challenges [103, 196]. However, to which extent
these constructs influence the improvements from introducing Agile practices is little
known. Based on the experts’ feedback, the propositions could be used to frame future
studies and also to aggregate the findings from those studies.
The concern of theory in software engineering has frequently been discussed and
debated [101]. One of the first steps in theory building is formulating propositions
[209]. Johnson et al. indicated the most theories in software engineering are casu-
ally proposed by authors and not subject to academic discussions [101]. Although the
propositions we described are not tested, the propositions we formulated in this chap-
ter were derived from empirical and literature studies, and not casual observations.
Furthermore, the propositions have also been consulted and discussed with experts in
academia.
7.6 Conclusions
This paper proposes a checklist to capture baseline situations, particularly when report-
ing the impact of Agile practices introductions. The checklist was developed from a
literature review and interviews with 11 industry practitioners. The checklist aggre-
gates baseline information pertaining to the workforce, management and organization
structure, process, and infrastructure. The components of the checklist have been vali-
dated by seven experts in academia. The checklist could be used by other researchers to
improve their study design. By reporting the baseline situation prior to the introduction
of Agile practices, the readers can better understand the extent that circumstances of the
organization contribute to the benefits and limitations of introducing Agile practices.
The checklist is not fully complete yet and will be extended with baseline infor-
mation that could affect the introduction of Lean tools. Furthermore, currently, the
checklist is only intended for empirical studies targeting Agile practices. The check-
list should also be extended and customized so it could be used beyond studying the
impacts of introducing Agile practices. The components of the checklist have been stat-
ically validated by seven experts in academia, but it requires further validation for its
implementation in empirical studies, so it could be used beyond studying the impacts

of introducing Agile practices.
From the checklist, we formulated four propositions. The propositions posit that
the baseline situation of a software organization may influence the extent of improve-
ments of introducing Agile practices. We infer that certain configuration of the baseline
situation could lead to a more successful Agile implementation. Although the propo-
sitions are not tested, the experts whom we consulted indicate that our inferences are
valid.
Bibliography
[1] P. Abrahamsson, O. Salo, J. Ronkainen, and J. Warsta. Agile Software Develop-

ment Methods – Review and Analysis. Technical Report 478, VTT Publications,
2002.
[2] J. Abrantes and G. Travassos. Common Agile Practices in Software Processes.

In Proceedings of the 5th International Symposium on Empirical Software En-
gineering and Measurement (ESEM 2011), pages 355–358, Sept 2011.
[3] A. Agarwal, R. Shankar, and M. Tiwari. Modeling the metrics of lean, agile
and leagile supply chain: An ANP-based approach. European Journal of Oper-
ational Research, 173(1):211–225, 2006.
[4] M. Ahmad, J. Markkula, and M. Oivo. Kanban in software development: A sys-

tematic literature review. In Proceedings of the 39th EUROMICRO Conference
on Software Engineering and Advanced Applications (SEAA 2013), pages 9–16,
2013.
[5] O. Al-Baik and J. Miller. The kanban approach, between agility and leanness:
A systematic review. Empirical Software Engineering, 20(6):1861–1897, 2015.
[6] B. R. Allen and A. C. Boynton. Information architecture: in search of efficient

flexibility. MIS quarterly, 15(4):435–445, 1991.
[7] S. Ambler. The Agile Scaling Model (ASM): Adapting Agile Methods for Com-
plex Environments. Technical report, 2009. IBM Rational White paper.
[8] E. Arisholm, H. Gallis, T. Dybå, and D. I. K. Sjøberg. Evaluating pair pro-

gramming with respect to system complexity and programmer expertise. IEEE
Transactions on Software Engineering, 33(2):65–86, 2007.
236 BIBLIOGRAPHY
[9] A. Aurum and C. Wohlin. Requirements engineering: Setting the context. In

A. Aurum and C. Wohlin, editors, Engineering and Managing Software Require-
ments, pages 1–15. Springer Berlin Heidelberg, 2005.
[10] R. Bahsoon and W. Emmerich. Evaluating Software Architectures for Stability
and Evolution. Technical report, UCL-CS Research Notes RN/03 2, 2003.
[11] R. Bavani. Distributed Agile: The Maturity Curve, 2012. URL
https://www.mindtree.com/sites/default/files/
mindtree-thought-posts-distributed-agile-the-\
maturity-curve.pdf. Accessed: October 2017.
[12] K. Beck. Embracing Change with eXtreme Programming. Computer, 32(10):
70–77, 1999.
[13] K. Beck. Extreme Programming Explained: Embrace Change. Addison-Wesley
Professional, Boston, USA, 2000.
[14] K. Beck. Test-Driven Development: By Example. The Addison-Wesley Signa-
ture Series. Addison-Wesley, 2003.
[15] K. Beck, M. Beedle, A. van Bennekum, A. Cockburn, W. Cunningham,
M. Fowler, J. Grenning, J. Highsmith, A. Hunt, R. Jeffries, J. Kern, B. Marick,
R. C. Martin, S. Mallor, K. Shwaber, and J. Sutherland. The Agile Manifesto.
Technical report, The Agile Alliance, 2001.
[16] W. Bekkers, I. van de Weerd, M. Spruit, and S. Brinkkemper. A framework for
process improvement in software product management. In A. Riel, R. O’Connor,
S. Tichkiewitch, and R. Messnarz, editors, Systems, Software and Services Pro-
cess Improvement. EuroSPI 2010. Communications in Computer and Informa-
tion Science, vol 99, pages 1–12. Springer Berlin Heidelberg, 2010.
[17] S. Benamati and A. L. Lederer. Coping with rapid change in information tech-
nology. In Proceedings of the ACM SIGCPR Conference on Computer Personnel
Research, pages 37–44. ACM, 1998.
[18] A. Benavoli, G. Corani, and F. Mangili. Should we really use post-hoc tests
based on mean-ranks. Journal of Machine Learning Research, 17(5):1–10,
2016.
[19] R. Benefield. Seven dimensions of agile maturity in the global enterprise: A case
study. In Proceedings of the 43rd Hawaii International Conference on System
Sciences (HICSS 2010), pages 1–7, 2010.
BIBLIOGRAPHY 237
[20] E. S. Bernardes and M. D. Hanna. A theoretical review of flexibility, agility and

responsiveness in the operations management literature: Toward a conceptual
definition of customer responsiveness. International Journal of Operations &
Production Management, 29(1):30–53, 2009.
[21] J. Biolchini, P. G. Mian, A. C. C. Natali, and G. H. Travassos. Systematic review

in software engineering. Technical report, System Engineering and Computer
Science Department COPPE/UFRJ, Technical Report ES, 2005.
[22] M. Birks and J. Mills. Grounded theory: A practical guide. Sage publications,
London, UK, 2011.
[23] J. Börstler. Feature oriented classification and reuse in IPSEN. Technical report,
Aachen University of Technology, 1992.
[24] T. Boyle. Towards best management practices for implementing manufactur-

ing flexibility. Journal of Manufacturing Technology Management, 17(1):6–21,
2006.
[25] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil. Lessons

from applying the systematic literature review process within the software engi-
neering domain. Journal of Systems and Software, 80(4):571–583, 2007.
[26] R. Britto, D. Smite, and L. O. Damm. Software architects in large-scale dis-

tributed projects: An Ericsson case study. IEEE Software, 33(6):48–55, Nov
2016.
[27] F. P. Brooks. No silver bullet: Essence and accidents of software engineering.

IEEE Computer, 20:10–19, 1987.
[28] J. S. Brown and P. Duguid. Organizational learning and communities-of-

practice: Toward a unified view of working, learning, and innovation. Orga-
nization Science, 2(1):40–57, 1991.
[29] R. Buschbacher, F. Hammond, J. Malec, and T. G. Nick. Handbook for clinical

research: Design, statistics, and implementation. Demos Medical Publishing,
New York, US, 2014.
[30] T. Byrd and D. Turner. Measuring the flexibility of information technology

infrastructure: Exploratory analysis of a construct. Journal of Management In-
formation Systems, 17(1):167–208, 2000.
238 BIBLIOGRAPHY
[31] T. Byrd, L. Jacome Madariaga, L. Byrd, and V. Mbarika. An examination of an

information systems flexibility framework. In Proceedings of the 43rd Hawaii
International Conference on System Sciences (HICSS 2010), pages 1–10, 2010.
[32] A. S. Campanelli and F. S. Parreiras. Agile methods tailoring: A systematic

literature review. Journal of Systems and Software, 110:85–100, 2015.
[33] A. Causevic, D. Sundmark, and S. Punnekkat. Factors limiting industrial adop-

tion of test driven development: A systematic review. In Proceedings of the 4th
IEEE International Conference on Software Testing, Verification and Validation
(ICST 2011), pages 337–346, 2011.
[34] Centre for Reviews and Dissemination. Systematic Reviews – CRD’s guidance
for undertaking reviews in health care. University of York, 2009.
[35] L. Chagas, D. de Carvalho, A. Lima, and C. Reis. Systematic literature review

on the characteristics of agile project Management in the Context of Maturity
Models. Communications in Computer and Information Science, 477:177–189,
2014.
[36] K. Charmaz. Grounded theory – objectivist and constructivist methods. In N. K.

Denzin and Y. S. Lincoln, editors, Strategies of qualitative inquiry, pages 249–
291. Sage, 2000.
[37] K. Charmaz. Constructing Grounded Theory. Sage Publications Limited, Lon-

don, UK, 2nd edition, 2014.
[38] L. Chen, M. Ali Babar, and N. Ali. Variability management in software prod-
uct lines: A systematic review. In Proceedings of the 13th International Soft-
ware Product Line Conference (SPLC 2009), pages 81–90, Pittsburgh, PA, USA,
2009.
[39] S. W. Chuang, T. Luor, and H. P. Lu. Assessment of institutions, scholars, and

contributions on agile software development (2001 – 2012) . Journal of Systems
and Software, 93:84–101, 2014.
[40] P. Clements. Being proactive pays off. IEEE Software, 19(4):28–30, 2002.
[41] P. Clements and L. Northrop. Software product lines: Practices and patterns.
SEI Series in Software Engineering. Addison-Wesley, Boston, US, 2001.
BIBLIOGRAPHY 239
[42] CMMI Product Team. CMMI for Systems Engineering/Software Engineer-

ing/Integrated Product and Process Development/Supplier Sourcing, Version
1.1, Staged Representation (CMMI-SE/SW/IPPD/SS, V1.1, Staged). Techni-
cal Report CMU/SEI-2002-TR-012, Software Engineering Institute, Carnegie
Mellon University, Pittsburgh, US, 2002.
[43] A. Cockburn. Agile Software Development. Addison-Wesley Longman Publish-
ing Co., Inc., Boston, USA, 2002.
[44] W. Codenie, N. González-Deleito, J. Deleu, V. Blagojevic, P. Kuvaja, and
J. Similä. A model for trading off flexibility and variability in software intensive
product development. In Proceedings of Third International Workshop on Vari-
ability Modelling of Software-Intensive Systems (VaMoS 2009), pages 61–70,
2009.
[45] M. Cohn. Succeeding with agile: software development using Scrum. Addison
Wesley, Boston, US, 2010.
[46] G. Coleman and R. O. Connor. Using grounded theory to understand software
process improvement: A study of Irish software product companies. Information
and Software Technology, 49(6):654–667, 2007.
[47] K. Conboy. Agility from first principles: Reconstructing the concept of agility
in information systems development. Information Sytems Research, 20(3):329–
354, 2009.
[48] T. G. Cummings and C. G. Worley. Organization development and change.
Cengage learning, Stamford, US, 10th edition, 2014.
[49] M. A. Cusumano. The software factory: A historical interpretation. IEEE Soft-
ware, 6(2):23–30, 1989.
[50] I. F. da Silva, P. A. da Mota Silveira Neto, P. O’Leary, E. S. de Almeida, and
S. R. de Lemos Meira. Agile software product lines: A systematic mapping
study. Journal of Software: Practice and Experience, 41(8):899–920.
[51] T. De Bruin, R. Freeze, U. Kulkarni, and M. Rosemann. Understanding the
main phases of developing a maturity assessment model. In Proceedings of 16th
Australasian conference on information systems (ACIS 2005), 2005.
[52] J. De Haan, J. Kwakkel, W. Walker, J. Spirco, and W. Thissen. Framing flexibil-
ity: Theorising and data mining to develop a useful definition of flexibility and
related concepts. Futures, 43(9):923–933, 2011.
240 BIBLIOGRAPHY
[53] G. De Michelis, E. Dubois, M. Jarke, F. Matthes, J. Mylopoulos, J. W. Schmidt,

C. Woo, and E. Yu. A three-faceted view of information systems. Communica-
tions of the ACM, 41(12):64–70, 1998.
[54] A. De Toni and S. Tonchia. Manufacturing flexibility: A literature review. In-

ternational Journal of Production Research, 36(6):1587–1617, 1998.
[55] J. Diaz, J. Perez, P. Alarcon, and J. Garbajosa. Agile product line Engineering:
A systematic literature review. Software: Practice and Experience, 41(8):921–
941, 2011.
[56] P. Diebold and M. Dahlem. Agile practices in practice: A mapping study. In

Proceedings of the 18th International Conference on Evaluation and Assessment
in Software Engineering (EASE 2014), pages 30:1–30:10, 2014.
[57] T. Dingsøyr, S. Nerur, V. Balijepally, and N. B. Moe. A decade of agile method-

ologies: Towards explaining agile software development. Journal of Systems
and Software, 85(6):1213–1221, 2012.
[58] T. Dybå and T. Dingsøyr. Empirical studies of agile software development:

A systematic review. Information and Software Technology, 50(91):833–859,
2008.
[59] T. Dybå and T. Dingsøyr. Strength of evidence in systematic reviews in software

engineering. In Proceedings of the 2nd ACM-IEEE International Symposium on
Empirical Software Engineering and Measurement (ESEM 2008), pages 178–
187, 2008.
[60] T. Dybå and T. Dingsøyr. What do we know about agile software development?
Software, IEEE, 26(5):6–9, 2009.
[61] T. Dybå, E. Arisholm, D. Sjoberg, J. Hannay, and F. Shull. Are two heads better
than one? On the effectiveness of pair programming. IEEE Software, 24(6):
12–15, Nov 2007.
[62] T. Dybå, D. I. K. Sjøberg, and D. S. Cruzes. What works for whom, where,
when, and why? On the role of context in empirical software engineering.
In Proceedings of the 2012 ACM-IEEE International Symposium on Empirical
Software Engineering and Measurement, pages 19–28, 2012.
[63] C. Dytham. Choosing and using statistics: A biologist’s guide. Wiley Blackwell
Publishing, Hoboken, US, 3rd edition, 2011.
BIBLIOGRAPHY 241
[64] A. H. Eden and T. Mens. Measuring software flexibility. IEEE Software, 153
(3), 2006.
[65] K. M. Eisenhardt. Building theories from case study research. Academy of

management review, 14(4):532–550, 1989.
[66] J. Erickson, K. Lyytinen, and K. Siau. Agile modeling, agile software devel-
opment, and extreme programming: the state of research. Journal of Database
Management, 16(4):88–100, 2005.
[67] M. Eriksson, J. Börstler, and K. Borg. Software product line modeling made
practical. Communications of the ACM, 49(12):49–54, 2006.
[68] J. Evans. Strategic flexibility for high technology manoeuvres: A conceptual

framework. Journal of Management Studies, 28(1):69–89, 1991.
[69] N. E. Fenton and S. L. Pfleeger. Software metrics: A rigorous and practical

approach. PWS Publishing Co., Boston, US, 2nd edition, 1998.
[70] K. Ferdows and A. D. Meyer. Lasting improvements in manufacturing perfor-

mance: In search of a new theory. Journal of Operations Management, 9(2):
168–184, 1990.
[71] R. G. Fichman, M. Keil, and A. Tiwana. Beyond valuation: “Options Thinking”

in IT project management. California Management Review, 47(2):74–96, 2005.
[72] B. Fitzgerald, G. Hartnett, and K. Conboy. Customising agile methods to soft-

ware practices at Intel Shannon. European Journal of Information Systems, 15
(2):200–213, 2006.
[73] J. L. Fleiss and J. Cohen. The equivalence of weighted kappa and the intraclass
correlation coefficient as measures of reliability. Educational and Psychological
Measurement, 33(3):613–619, 1973.
[74] A. Forward and T. C. Lethbridge. A taxonomy of software types to facili-

tate search and evidence-based software engineering. In Proceedings of the
2008 Conference of the Center for Advanced Studies on Collaborative Research:
Meeting of Minds (CASCON 2008), pages 14:179–14:191, 2008.
[75] M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts. Refactoring: Im-

proving the Design of Existing Code. Addison-Wesley Longman Publishing
Co., Inc., Boston, USA, 1999.
242 BIBLIOGRAPHY
[76] S. Fricker and S. Schumacher. Release planning with feature trees: Industrial
case. In Requirements Engineering: Foundation for Software Quality, pages
288–305. Springer, 2012.
[77] J. A. Gamble and D. K. Creedy. Women’s request for a cesarean section: A

critique of the literature. Birth, 27(4):256–263, 2000.
[78] T. Gandomani, H. Zulzalil, A. Ghani, and A. Sultan. A systematic literature

review on relationship between agile methods and open source software devel-
opment methodology. International Review on Computers and Software, 7(4):
1602–1607, 2012.
[79] D. A. Garvin. Building a Learning Organization. Harvard Business Review, 71

(4):78–91, 2000.
[80] D. Gerwin. Manufacturing flexibility: A strategic perspective. Management

Science, 39(4):395–410, 1993.
[81] T. Greenhalgh and R. Peacock. Effectiveness and rfficiency of search methods

in systematic reviews of complex evidence: Audit of primary sources. British
Medical Journal, 331(7524):1064–1065, 2005.
[82] I. Guggenmoos-Holzmann. The meaning of kappa: Probabilistic concepts of

reliability and validity revisited. Journal of Clinical Epidemiology, 49(7):775–
782, 1996.
[83] L. Hagge and K. Lappe. Sharing requirements engineering experience using

patterns. IEEE Software, 22(1):24–31, 2005.
[84] G. Hanssen, D. Smite, and N. Moe. Signs of agile trends in global software
engineering research: A tertiary study. In Proceedings of the 6th IEEE Interna-
tional Conference on Global Software Engineering Workshop (ICGSEW 2011),
pages 17–23, Aug 2011.
[85] S. Hansson, Y. Zhao, and H. Burden. How MAD are we? Empirical evidence for
model-driven agile development. In D. Di Ruscio, J. de Lara, and A. Pierantonio,
editors, Proceedings of the 3rd Extreme Modeling Workshop (XM 2014), pages
2–11, 2014.
[86] M. Harris, R. Collins, and A. Hevner. Control of flexible software development

under uncertainty. Information Systems Research, 20(3):400–419, 2009.
BIBLIOGRAPHY 243
[87] C. Hart. Doing a literature review: Releasing the social science research imag-
ination. Sage Publications, London, UK, 1998.
[88] R. Hebig and R. Bendraou. On the need to study the impact of model driven en-
gineering on software processes. In Proceedings of the 10th International Con-
ference on Software and System Process (ICSSP 2014), pages 164–168, 2014.
[89] J. C. Henderson and N. Venkatraman. Strategic alignment: Leveraging infor-

mation technology for transforming organizations. IBM systems journal, 32(1):
4–16, 1993.
[90] E. Hossain, M. Babar, and H. young Paik. Using scrum in global software
development: A systematic literature review. In Proceedings of the 4th IEEE
International Conference on Global Software Engineering (ICGSE 2009), pages
175–184, July 2009.
[91] G. Huber and R. McDaniel. The decision-making paradigm of organizational

design. Journal of Management Science, 32(5):572–589, 1986.
[92] J. Humble and D. Farley. Continuous delivery: Reliable software releases

through build, test, and deployment automation. Pearson Education, London,
UK, 2010.
[93] M. Hüttermann. Infrastructure as code. In DevOps for Developers, pages 135–

156. Apress, Berkeley, CA, 2012.
[94] IEEE. IEEE Standard Glossary of Software Engineering Terminology. Standard

IEEE Std 610.12-1990, IEEE, 1990.
[95] D. R. Ilgen, J. R. Hollenbeck, M. Johnson, and D. Jundt. Teams in organiza-

tions: From input-process-output models to IMOI models. Annual Review of
Psychology, 56(1):517–543, 2005.
[96] ISO/IEC 25010. ISO/IEC 25010 - Systems and software engineering - Sys-
tems and software Quality Requirements and Evaluation (SQuaRE) - System
and software quality models. Technical report, ISO, 2010.
[97] M. Ivarsson and T. Gorschek. A method for evaluating rigor and industrial rel-
evance of technology evaluations. Empirical Software Engineering, 16(3):365–
395, 2011.
244 BIBLIOGRAPHY
[98] R. Jabangwe, J. Börstler, D. Šmite, and C. Wohlin. Empirical evidence on the

link between object-oriented measures and external quality attributes: a system-
atic literature review. Empirical Software Engineering, 20(3):640–693, 2015.
[99] S. Jalali and C. Wohlin. Agile practices in global software engineering – A
systematic map. In Proceedings of the 5th IEEE International Conference on
Global Software Engineering (ICGSE 2010), pages 45–54, 2010.
[100] S. Jalali and C. Wohlin. Global software engineering and agile practices: A
systematic review. Journal of Software: Evolution and Process, 24(6):643–659,
2012.
[101] P. Johnson, M. Ekstedt, and I. Jacobson. Where’s the theory for software engi-
neering? IEEE Software, 29(5):96–96, 2012.
[102] G. R. Jones. Organizational theory, design, and change. Prentice Hall, New
Jersey, US, 2012.
[103] M. Jørgensen. A survey on the characteristics of projects with success in deliv-
ering client benefits. Information and Software Technology, 78:83–94, 2016.
[104] G. Jurca, T. Hellmann, and F. Maurer. Integrating agile and user-centered design:
A systematic mapping and review of evaluation and validation studies of Agile-
UX. In Proceedings of the 2014 Agile Conference (AGILE 2014), pages 24–32,
2014.
[105] M. Kajko-Mattsson, G. A. Lewis, D. Siracusa, T. Nelson, N. Chapin, M. Heydt,
J. Nocks, and H. Snee. Long-term life cycle impact of agile methodologies. In
Proceedings of the 22nd IEEE International Conference on Software Mainte-
nance (ICSM 2006), pages 422–425, 2006.
[106] S. Kara, B. Kayis, and S. O’Kane. The role of human factors in flexibility
management: A survey. Human Factors and Ergonomics in Manufacturing and
Service Industries, 12(1):75–119, 2002.
[107] A. Kasoju, K. Petersen, and M. V. Mäntylä. Analyzing an automotive testing
process with evidence-based software engineering. Information and Software
Technology, 55(7):1237–1259, 2013.
[108] M. I. Khan, M. A. Qureshi, and Q. Abbas. Agile methodology in software de-
velopment (SMEs) of Pakistan software industry for successful software projects
(CMM framework). In International Conference on Educational and Network
Technology (ICENT 2010), pages 576–580, 2010.
BIBLIOGRAPHY 245
[109] M. Khurum, T. Gorschek, and M. Wilson. The software value map – An ex-
haustive collection of value aspects for the development of software intensive
products. Journal of Software: Evolution and Process, 25(7):711–741, 2012.
[110] B. Kitchenham and S. Charters. Guidelines for performing systematic litera-

ture reviews in software engineering. Technical Report EBSE-2007-01, Keele
University, UK, 2007.
[111] B. Kitchenham, L. Pickard, and S. L. Pfleeger. Case studies for method and tool
evaluation. IEEE Software, 12(4):52–62, 1995.
[112] B. Kitchenham, T. Dybå, and M. Jørgensen. Evidence-based software engineer-

ing. In Proceedings of the 26th International Conference on Software Engineer-
ing (ICSE 2004), pages 273–281, 2004.
[113] B. Kitchenham, O. P. Brereton, D. Budgen, M. Turner, J. Bailey, and

S. Linkman. Systematic Literature Reviews in Software Engineering – A Sys-
tematic Literature Review. Information and Software Technology, 51(1):7–15,
2009.
[114] B. Kitchenham, R. Pretorius, D. Budgen, O. P. Brereton, M. Turner, M. Niazi,

and S. Linkman. Systematic literature reviews in software engineering – A ter-
tiary study. Information and Software Technology, 52(8):792–805, 2010.
[115] B. Kitchenham, D. Budgen, and P. Brereton. Evidence-based software engineer-

ing and systematic reviews. CRC Press, Boca Raton, US, 2015.
[116] K. Knoll and S. Jarvenpaa. Information technology alignment or ”fit” in highly

turbulent environments: The concept of flexibility. In Proceedings of the 1994
Computer Personnel Research Conference on Reinventing IS, pages 1–14. ACM,
1994.
[117] K. Korhonen. Evaluating the impact of an agile transformation: A longitudinal

case study in a distributed context. Software Quality Journal, 21(4):599–624,
2013.
[118] L. L. Kostea and M. K. Malhotrab. Trade-offs among the elements of flexibility:

A comparison from the automotive industry. Omega, 28(6):693–710, 2000.
[119] M. Kropp, A. Meier, and R. Biddle. Agile practices, collaboration and experi-
ence. In Proceedings of the 17th International Conference in Product-Focused
Software Process Improvement (PROFES 2016), pages 416–431, 2016.
246 BIBLIOGRAPHY
[120] P. Kruchten. Software architecture and agile software development: A clash

of two cultures? In In the proceedings of the 32nd ACM/IEEE International
Conference on Software Engineering (ICSE 2010), pages 497–498, 2010.
[121] M. Kuhrmann, P. Diebold, J. Münch, P. Tell, V. Garousi, M. Felderer, K. Trek-

tere, F. McCaffery, O. Linssen, E. Hanser, and C. R. Prause. Hybrid software and
system development in practice: Waterfall, scrum, and beyond. In Proceedings
of the 2017 International Conference on Software and System Process (ICSSP
2017), pages 30–39, 2017.
[122] E. Kupiainen, M. V. Mäntylä, and J. Itkonen. Why are industrial agile teams
using metrics and how do they use them? In Proceedings of the 5th International
Workshop on Emerging Trends in Software Metrics (WETSoM 2014), pages 23–
29, 2014.
[123] N. Kurapati, V. S. C. Manyam, and K. Petersen. Agile software development

practice adoption survey. In Proceedings of the 13th International Conference
Agile Processes in Software Engineering and Extreme Programming (XP 2012),
pages 16–30, 2012.
[124] J. R. Landis and G. G. Koch. The measurement of observer agreement for cate-
gorical data. Biometrics, 33(1):159–174, 1977.
[125] A. D. Leeuw and H. Volberda. On the concept of flexibility: A dual control

perspective. Omega, 24(2):121 – 139, 1996.
[126] D. Leffingwell. Scaling software agility: Best practices for large enterprises.
Addison-Wesley Professional, Boston, US, 2007.
[127] M. Leppänen. A comparative analysis of agile maturity models. In R. Pooley,

J. Coady, C. Schneider, H. Linger, C. Barry, and M. Lang, editors, Information
Systems Development, pages 329–343. 2013.
[128] Y. Levy and T. J. Ellis. A systems approach to conduct an effective literature

review in support of information systems research. Informing Science Journal,
9:181–212, 2006.
[129] J. Li, N. B. Moe, and T. Dybå. Transition from a plan-driven process to scrum:
A longitudinal case study on software quality. In Proceedings of the Fourth
ACM-IEEE International Symposium on Empirical Software Engineering and
Measurement (ESEM 2010), pages 13:1–13:10, 2010.
BIBLIOGRAPHY 247
[130] Y. Li, K.-C. Chang, H.-G. Chen, and J. J. Jiang. Software Development Team
Flexibility Antecedents. Journal of Systems and Software, 83(10):1726–1734,
2010.
[131] J. Linåker, S. M. Sulaman, R. Maiani de Mello, and M. Höst. Guidelines for
conducting surveys in software engineering. Technical report, Lund University,
2015.
[132] H. V. Loon. Process assessment and ISO/IEC 15504: A reference book.
Springer-Verlag New York, Inc., Secaucus, US, 2007.
[133] K. Lui and K. Chan. A road map for implementing extreme programming. In
M. Li, B. Boehm, and L. Osterweil, editors, Unifying the Software Process Spec-
trum (SPW 2005), volume 3840 of Lecture Notes in Computer Science, pages
474–481. 2006.
[134] S. MacDonell, M. Shepperd, B. Kitchenham, and E. Mendes. How reliable are
systematic reviews in empirical software engineering? IEEE Transactions on
Software Engineering, 36(5):676–687, Sept 2010.
[135] A. Magdaleno, C. Werner, and R. Araujo. Reconciling software development
models: A quasi-systematic review. Journal of Systems and Software, 85(2):
351–369, 2012.
[136] M. V. Mäntylä, F. Khomh, B. Adams, E. Engström, and K. Petersen. On Rapid
Releases and Software Testing: A Case Study and a Semi-Systematic Literature
Review. Empirical Software Engineering, 20(5):1384–1425, 2015.
[137] J. McCall, P. Richards, and G. Walters. Factors in software quality, concept
and definitions of software quality. Technical report, Rome Air Development
Center, Air Force Systems Command, Griffiss Air Force Base, 1977.
[138] A. Metzger and K. Pohl. Variability Management in Software Product Line
Engineering. In Companion to the Proceedings of the 29th International Con-
ference on Software Engineering (ICSE COMPANION 2007), pages 186–187,
2007.
[139] M. B. Miles and A. M. Huberman. Qualitative data analysis : An expanded
sourcebook. Sage Publications, London, UK, 1994.
[140] S. M. Mitchell and C. B. Seaman. A comparison of software cost, duration,
and quality for waterfall vs. iterative and incremental development: A system-
atic review. In Proceedings of the Third International Symposium on Empirical
Software Engineering and Measurement (ESEM 2009), pages 511–515, 2009.
248 BIBLIOGRAPHY
[141] H. Munir, M. Moayyed, and K. Petersen. Considering rigor and relevance when
evaluating test driven development: A systematic review. Information and Soft-
ware Technology, 56(4):375–394, 2014.
[142] B. Murphy, C. Bird, T. Zimmermann, L. Williams, N. Nagappan, and A. Begel.
Have agile techniques been the silver bullet for software development at Mi-
crosoft? In Proceedings of the Seventh International Symposium on Empirical
Software Engineering and Measurement (ESEM 2013), pages 75–84, Oct 2013.
[143] M. Naab and J. Stammel. Architectural Flexibility in a Software-system’s Life-
cycle: Systematic Construction and Exploitation of Flexibility. In Proceedings
of the Eighth International ACM SIGSOFT Conference on Quality of Software
Architectures (QoSA 2012), pages 13–22, 2012.
[144] J. Nawrocki, B. Walter, and A. Wojciechowski. Toward maturity model for
extreme programming. In Proceedings 27th EUROMICRO Conference. 2001:
A Net Odyssey, pages 233–239, 2001.
[145] K. Nelson, H. Nelson, and M. Ghods. Technology flexibility: Conceptualization,
validation, and measurement. In Proceedings of the 30th Hawaii International
Conference on System Sciences (HICSS 1997), pages 76–87, 1997.
[146] S. Nerur, R. Mahapatra, and G. Mangalaraj. Challenges of migrating to agile
methodologies. Communications of the ACM, 48(5):72–78, 2005.
[147] W. Newman, M. Hanna, and M. Maffei. Dealing with the uncertainties of man-
ufacturing: Flexibility, buffers and integration. International Journal of Opera-
tions & Production Management, 13(1):19–34, 1993.
[148] D. Nguyen-Cong and D. Tran-Cao. A review of effort estimation studies in ag-
ile, iterative and incremental software development. In Proceedings of the 10th
IEEE RIVF International Conference on Computing and Communication Tech-
nologies, Research, Innovation, and Vision for the Future (RIVF 2013), pages
27–30, 2013.
[149] N. Nikitina, M. Kajko-Mattsson, and M. Stråle. From scrum to scrumban: A
case study of a process transition. In Proceedings of the Eighth International
Conference on Software and System Process (ICSSP 2012), pages 140–149, Pis-
cataway, USA, 2012.
[150] S. Nurcan. A Survey on the Flexibility Requirements Related to Business Pro-
cesses and Modeling Artifacts. In Proceedings of the 41st Hawaii International
Conference on System Sciences (HICSS 2008), pages 378–387, 2008.
BIBLIOGRAPHY 249
[151] I. Nurdiani, S. A. Fricker, and J. Börstler. An analysis of change scenarios of

an IT organization for flexibility building. In Proceedings of the 23rd European
Conference on Information Systems (ECIS 2015), 2015.
[152] I. Nurdiani, J. Börstler, and S. Fricker. The impacts of agile and lean practices
on project constraints: A tertiary study. Journal of Systems and Software, 119
(Supplement C):162–183, 2016.
[153] I. Nurdiani, J. Börstler, and S. Fricker. Literature review of flexibility attributes:

A flexibility framework for software developing organization. Journal of Soft-
ware: Evolution and Process, 2018.
[154] B. Nuseibeh. Weaving together requirements and architectures. Computer, 34

(3):115–119, 2001.
[155] C. Oliveira Albuquerque, P. Oliveira Antonino, and E. Yumi Nakagawa. An

investigation into agile methods in embedded systems development. In Pro-
ceedings of the 12th International Conference Computational Science and Its
Applications (ICCSA 2012), pages 576–591, 2012.
[156] W. J. Orlikowski. CASE tools as organizational change: Investigating incre-

mental and radical changes in systems development. Management Information
Systems Quarterly, 17(3):309–340, 1993.
[157] O. Ozcan-Top and O. Demirörs. Assessment of agile maturity models: A mul-

tiple case study. In T. Woronowicz, T. Rout, R. V. O’Connor, and A. Dorling,
editors, International Conference on Software Process Improvement and Capa-
bility Determination (SPICE 2013), Communications in Computer and Informa-
tion Science, pages 130–141. 2013.
[158] J. Packlick. The agile maturity map a goal oriented approach to agile improve-
ment. In Proceedings of Agile 2007 (AGILE 2007), pages 266–271, Aug 2007.
[159] M. Pai, M. McCulloch, J. D. Gorman, N. Pai, W. Enanoria, G. Kennedy,

P. Tharyan, and J. M. Colford. Systematic reviews and meta-analyses: An il-
lustrated, step-by-step fuide. National Medical Journal of India, 17(2):89–95,
2004.
[160] C. Patel and M. Ramachandran. Agile maturity model (AMM): A software

process improvement framework for agile software development practices. In-
ternational Journal of Software Engineering, 2(1):3–28, 2009.
250 BIBLIOGRAPHY
[161] B. Paterson, S. Thorne, C. Canam, and C. Jillings. Meta-study of Qualitative

Health Research: A Practical Guide to Meta-analysis and Meta-synthesis. Sage
Publication, London,UK, 2001.
[162] O. Pedgley. Capturing and analysing own design activity. Design Studies, 28
(5):463 – 483, 2007.
[163] J. Pernstål, R. Feldt, and T. Gorschek. The lean gap: A review of lean approaches
to large-scale software systems development. Journal of Systems and Software,
86(11):2797–2821, 2013.
[164] K. Petersen. Is Lean Agile and Agile Lean? Modern Software Engineering
Concepts and Practices: Advanced Approaches, IGI Global, pages 19–46, 2011.
[165] K. Petersen and C. Gencel. Worldviews, research methods, and their relationship
to validity in empirical software engineering research. In 2013 Joint Conference
of the 23rd International Workshop on Software Measurement and the Eighth
International Conference on Software Process and Product Measurement, pages
81–89, 2013.
[166] K. Petersen and C. Wohlin. Context in industrial software engineering research.

In Proceedings of the Third International Symposium on Empirical Software
Engineering and Measurement (ESEM), pages 401–404, 2009.
[167] K. Petersen and C. Wohlin. The Effect of Moving From a Plan-driven to an

Incremental Software Development Approach with Agile Practices. Empirical
Software Engineering, 15:654–693, 2010. ISSN 1382-3256.
[168] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. Systematic mapping studies

in software engineering. In Proceedings of the 12th International Conference on
Evaluation and Assessment in Software Engineering (EASE 2008), pages 68–77,
2008.
[169] M. Poppendieck and T. Poppendieck. Lean software development: An agile

toolkit. Addison-Wesley Longman Publishing Co., Inc., Boston, USA, 2003.
[170] D. Port and L. Huang. Strategic architectural flexibility. In Proceedings of the

21st International Conference on Software Maintenance (ICSM 2003), pages
389–396, 2003.
[171] Project Management Institute. A Guide To The Project Management Body Of

Knowledge (PMBOK Guides). Project Management Institute, 2004.
BIBLIOGRAPHY 251
[172] O. Pusatli and S. Misra. A discussion on IS and software measurement terminol-

ogy: Flexibility as an example. In Proceedings of the 10th International Con-
ference on Computational Science and Its Applications (ICCSA 2010), pages
250–254, 2010.
[173] A. Qumer and B. Henderson-Sellers. A framework to support the evaluation,

adoption and improvement of agile methods in practice. Journal of Systems and
Software, 81(11):1899 – 1919, 2008.
[174] Z. Racheva, M. Daneva, and K. Sikkel. Value creation by agile projects:

Methodology or mystery? In F. Bomarius, M. Oivo, P. Jaring, and P. Abrahams-
son, editors, Product-Focused Software Process Improvement, Lecture Notes in
Business Information Processing, pages 141–155. 2009.
[175] Y. Rafique and V. Mišić. The effects of test-driven development on external

quality and productivity: A meta-analysis. IEEE Transactions on Software En-
gineering, 39(6):835–856, 2013.
[176] P. Ralph and P. Shportun. Scrum abandonment in distributed teams: A revelatory

case. In Proceedings of the 17th Pacific Asia Conference on Information Systems
(PACIS 2013), 2013.
[177] G. Regev, P. Soffer, and R. Schmidt. Taxonomy of Flexibility in Business Pro-

cesses. In Proceedings of the Seventh Workshop on Business Process Modelling,
Development and Support, pages 90–93, 2006.
[178] G. Regev, I. Bider, and A. Wegmann. Defining business process Flexibility with
the Help of Invariants. Software Process: Improvement and Practice, 12(1):
65–79, 2007.
[179] G. Regev, O. Hayard, and A. Wegmann. What we can learn about business
modeling from homeostasis. In B. Shishkov, editor, Business Modeling and
Software Design, Lecture Notes in Business Information Processing, pages 1–
15. Springer Berlin Heidelberg, 2013.
[180] M. Reichert and B. Weber. Flexibility issues in process-aware information sys-

tems. In Enabling Flexibility in Process-Aware Information Systems: Chal-
lenges, Methods, Technologies. 2012.
[181] D. J. Reifer, F. Maurer, and H. Erdogmus. Scaling agile methods. IEEE Soft-
ware, 20(4):12–14, July 2003.
252 BIBLIOGRAPHY
[182] C. Robson. Real world research. John Wiley & Sons, West Sussex, UK, 2nd
edition, 2011.
[183] P. Rodrı́guez, J. Markkula, M. Oivo, and K. Turula. Survey on agile and lean
usage in finnish software industry. In Proceedings of the 2012 ACM-IEEE In-
ternational Symposium on Empirical Software Engineering and Measurement,
pages 139–148, 2012.
[184] A. Rohunen, P. Rodriguez, P. Kuvaja, L. Krzanik, and J. Markkula. Approaches

to agile adoption in large settings: A comparison of the results from a literature
analysis and an industrial inventory. In Proceedings 11th International Con-
ference on Product-Focused Software Process Improvement (PROFES 2010),
pages 77–91, 2010.
[185] E. T. Ryan, D. R. Jacques, and J. M. Colombi. An ontological framework for

clarifying flexibility-related terminology via literature survey. Systems Engi-
neering, 16(1):99–110, 2013.
[186] D. Salah, R. F. Paige, and P. Cairns. A systematic literature review for agile
development processes and user centred design integration. In Proceedings of
the 18th International Conference on Evaluation and Assessment in Software
Engineering (EASE 2014), pages 5:1–5:10, 2014.
[187] J. Saldaña. The Coding Manual for Qualitative Researchers. SAGE Publications
Limited, London, UK, 2012.
[188] J. H. Saleh, G. Mark, and N. C. Jordan. Flexibility: A multi-disciplinary liter-

ature review and a research agenda for designing flexible engineering systems.
Journal of Engineering Design, 20(3):307–323, 2009.
[189] C. Salvador, A. Nakasone, and J. A. Pow-Sang. A systematic review of usability

techniques in agile methodologies. In Proceedings of the Seventh Euro American
Conference on Telematics and Information Systems (EATIS 2014), pages 17:1–
17:6, 2014.
[190] H. Schonenberg, R. Mans, N. Russell, N. Mulyar, and W. Van der Aalst. Towards
a taxonomy of process flexibility. In Proceedings of the 20th International Con-
ference on Advanced Information Systems Engineering, pages 81–84, 2008.
[191] K. Schwaber and M. Beedle. Agile software development with Scrum. Prentice
Hall PTR, Upper Saddle River, USA, 1st edition, 2001.
BIBLIOGRAPHY 253
[192] T. Schweigert, R. Nevalainen, D. Vohwinkel, M. Korsaa, and M. Biro. Ag-

ile maturity model: Oxymoron or the next level of understanding. In A. Mas,
A. Mesquida, T. Rout, R. V. O’Connor, and A. Dorling, editors, Software Pro-
cess Improvement and Capability Determination, pages 289–294. 2012.
[193] T. Schweigert, D. Vohwinkel, M. Korsaa, R. Nevalainen, and M. Biro. Agile
maturity model: A synopsis as a first step to synthesis. In F. McCaffery, R. V.
O’Connor, and R. Messnarz, editors, Systems, Software and Services Process
Improvement, Communications in Computer and Information Science, pages
214–227. 2013.
[194] T. Schweigert, D. Vohwinkel, M. Korsaa, R. Nevalainen, and M. Biro. Agile
maturity model: Analysing agile maturity characteristics from the SPICE per-
spective. Journal of Software: Evolution and Process, 26(5):513–520, 2014.
[195] M. Senapathi and A. Srinivasan. Sustained agile usage: A systematic literature
review. In Proceedings of the 17th International Conference on Evaluation and
Assessment in Software Engineering (EASE 2013), pages 119–124, 2013.
[196] P. Serrador and J. K. Pinto. Does Agile work? A quantitative analysis of ag-
ile project success. International Journal of Project Management, 33(5):1040–
1051, 2015.
[197] A. K. Sethi and S. P. Sethi. Flexibility in manufacturing: A survey. International
Journal of Flexible Manufacturing Systems, 2:289–328, 1990.
[198] P. Sfetsos and I. Stamelos. Empirical studies on quality in agile practices: A
systematic literature review. In Proceedings of the Seventh International Confer-
ence on the Quality of Information and Communications Technology (QUATIC
2010), pages 44–53, 2010.
[199] M. Sharfman and J. Dean Jr. Flexibility in strategic decision making: Infor-
mational and ideological perspectives. Journal of Management Studies, 34(2):
191–217, 1997.
[200] H. Sharp and H. Robinson. Some social factors of software engineering: The
maverick, community and technical practices. In Proceedings of the 2005 Work-
shop on Human and Social Factors of Software Engineering (HSSE 2005), pages
1–6, New York, NY, USA, 2005.
[201] M. Shen, W. Yang, G. Rong, and D. Shao. Applying agile methods to embedded
software development: A systematic review. In Proceedings of the Seconnd
254 BIBLIOGRAPHY
International Workshop on Software Engineering for Embedded Systems (SEES

2012), pages 30–36, 2012.
[202] A. Sidky, J. Arthur, and S. Bohner. A disciplined approach to adopting agile

practices: The agile adoption framework. Innovations in Systems and Software
Engineering, 3(3):203–216, 2007.
[203] F. S. Silva, F. S. F. Soares, A. L. Peres, I. M. d. Azevedo, P. P. Pinto, and S. R.

d. L. Meira. A reference model for agile quality assurance: Combining ag-
ile methodologies and maturity models. In Proceeding of the Ninth Interna-
tional Conference on the Quality of Information and Communications Technol-
ogy (QUATIC 2014), pages 139–144, 2014.
[204] F. S. Silva, F. S. F. Soares, A. L. Peres, I. M. de Azevedo, A. P. L. Vasconcelos,

F. K. Kamei, and S. R. de Lemos Meira. Using CMMI together with agile soft-
ware development: A systematic review. Information and Software Technology,
58:20–43, 2015.
[205] T. Silva da Silva, A. Martin, F. Maurer, and M. Silveira. User-centered design

and agile methods: A systematic review. In Proceedings of the 2011 Agile
Conference (AGILE 2011), pages 77–86, Aug 2011.
[206] H. A. Simon. The Sciences of the Artificial. MIT Press, Cambridge, Us, 3rd
edition edition, 1996.
[207] S. M. D. Simon. The new science of management decision. In Proceedings

of the Third Conference of the Operational Research Society of New Zealand
(ORSNZ 1960), 1960.
[208] C. Simonyi and M. Heller. The hungarian revolution. BYTE, 16(8):131–138,

1991.
[209] D. I. K. Sjøberg, T. Dybå, B. C. D. Anda, and J. E. Hannay. Building theories in

software engineering. In F. Shull, J. Singer, and D. I. K. Sjøberg, editors, Guide
to Advanced Empirical Software Engineering, pages 312–336. 2008.
[210] M. T. Sletholt, J. Hannay, D. Pfahl, H. C. Benestad, and H. P. Langtangen. A

literature review of agile practices and their effects in scientific software de-
velopment. In Proceedings of the Fourth International Workshop on Software
Engineering for Computational Science and Engineering (SECSE 2011), pages
1–9, 2011.
BIBLIOGRAPHY 255
[211] R. Snowdon, B. Warboys, R. Greenwood, C. Holland, P. Kawalek, and D. Shaw.

On the architecture and form of flexible process support. Software Process:
Improvement and Practice, 12(1):21–34, 2007.
[212] A. Solinski and K. Petersen. Prioritizing agile benefits and limitations in relation
to practice usage. Software Quality Journal, 24(2):447–482, 2016.
[213] S. Stavru. A critical examination of recent industrial surveys on agile method

usage. Journal of Systems and Software, 94:87–97, 2014.
[214] S. Stavru, I. Krasteva, and S. Ilieva. Challenges for migrating to the service
cloud paradigm: An agile perspective. In A. Haller, G. Huang, Z. Huang, H.-y.
Paik, and Q. Sheng, editors, Web Information Systems Engineering – WISE 2011
and 2012 Workshops, Lecture Notes in Computer Science, pages 77–91. 2013.
[215] S. Stavru, I. Krasteva, and S. Ilieva. Challenges of model-driven modernization-

An agile perspective. In Proceedings of the First International Conference on
Model-Driven Engineering and Software Development (MODELSWARD 2013),
pages 219–230, 2013.
[216] S. E. Straus, W. S. Richardson, P. Glasziou, and R. B. Haynes. Evidence-based

medicine: How to practice and teach EBM. Churchill Livingstone, 2005.
[217] R. Tannenbaum and W. H. Schmidt. How to choose a leadership pattern. Har-

vard Business Review, Boston, US, 1973.
[218] T. Tapanainen, M. Hallanoro, J. Päivärinta, and H. Salmela. Towards an agile IT

organisation: A review of prior literature. In Proceedings of the Second Euro-
pean Conference on Information Management and Evaluation (ECIME 2008),
pages 425–432, 2008.
[219] S. Thomke. The role of flexibility in the development of new products: An

empirical study. Research Policy, 26(1):105–119, 1997.
[220] G. H. Travassos, P. S. M. dos Santos, P. G. M. Neto, and J. Biolchini. An envi-

ronment to support large scale experimentation in software engineering. In Pro-
ceedings of the 13th IEEE International Conference on Engineering of Complex
Computer Systems (ICECCS 2008), pages 193–202, 2008.
[221] B. W. Tuckman. Developmental sequence in small groups. Psychological bul-

letin, 63(6):384, 1965.
256 BIBLIOGRAPHY
[222] O. Turetken, I. Stojanov, and J. J. M. Trienekens. Assessing the adoption level of

scaled agile development: A maturity model for scaled agile framework. Journal
of Software: Evolution and Process, 29(6), 2017.
[223] D. Turk, R. France, and B. Rumpe. Limitations of agile software processes.
In Proceedings of the Third International Conference on Extreme Programming
and Agile Processes in Software Engineering (XP 2002), pages 43–46, 2000.
[224] M. Unterkalmsteiner, T. Gorschek, A. K. M. M. Islam, C. K. Cheng, R. B. Per-
madi, and R. Feldt. Evaluation and measurement of software process improve-
ment: A systematic literature review. IEEE Transactions on Software Engineer-
ing, 38(2):398–424, 2012.
[225] M. Usman, E. Mendes, F. Weidt, and R. Britto. Effort estimation in agile
software development: A systematic literature review. In Proceedings of the
10th International Conference on Predictive Models in Software Engineering
(PROMISE 2014), pages 82–91, 2014.
[226] J. Van Biesebroeck. The cost of flexibility. Assembly Automation, 27(1):55–64,
2007.
[227] W. Van der Aalst and S. Jablonski. Dealing with workflow change: Identification
of issues and solutions. Computer Systems Science and Engineering, 15(5):267–
276, 2000.
[228] M. van Oosterhout, E. Waarts, and J. van Hillegersberg. Change factors requir-
ing agility and implications for IT. European Journal of Information Systems,
15(2):132–145, 2006.
[229] J. M. Verner, O. P. Brereton, B. A. Kitchenham, M. Turner, and M. Niazi. Risks
and risk mitigation in global software development: A tertiary study. Informa-
tion and Software Technology, 56(1):54–78, 2014.
[230] Version One. 10th Annual State of Agile™Report,
2015. URL https://versionone.com/pdf/
VersionOne-10th-Annual-State-of-Agile-Report.pdf.
Accessed: January 2017.
[231] Version One. 11th Annual State of Agile™Report, 2016. URL
https://explore.versionone.com/state-of-agile/
versionone-11th-annual-state-of-agile-report-2. Ac-
cessed: June 2017.
BIBLIOGRAPHY 257
[232] R. Vidgen and X. Wang. Coevolving systems and the organization of agile soft-
ware development. Information Systems Research, 20(3):355–376, 2009.
[233] M. Virmani. Understanding DevOps bridging the gap from continuous integra-
tion to continuous delivery. In Proceedings of the Fifth International Conference
on Innovative Computing Technology (INTECH 2015), pages 78–82, 2015.
[234] X. Wang, K. Conboy, and M. Pikkarainen. Assimilation of agile practices in

use. Information Systems Journal, 22(6):435–455, 2012.
[235] R. S. Weiss. Learning from strangers: The art and method of qualitative inter-
view studies. Free Press, New York, US, 2008.
[236] G. L. Wells, R. Lindsay, and T. J. Ferguson. Accuracy, confidence, and juror

perceptions in eyewitness identification. Journal of Applied Psychology, 64(4):
440–448, 1979.
[237] E. J. Weyuker and A. Avritzer. A metric to predict software scalability. In Pro-

ceedings Eighth IEEE Symposium on Software Metrics, pages 152–158, 2002.
[238] R. Wieringa, N. Maiden, N. Mead, and C. Rolland. Requirements engineering

paper classification and evaluation criteria: A proposal and a discussion. Re-
quirements Engineering, 11(1):102–107, 2006.
[239] L. Williams. Agile software development methodologies and practices. In M. V.

Zelkowitz, editor, Advances in Computers, volume 80 of Advances in Comput-
ers, pages 1–44. Elsevier, 2010.
[240] C. Wohlin, P. Runeson, P. A. da Mota Silveira Neto, E. Engström,

I. do Carmo Machado, and E. S. de Almeida. On the reliability of mapping
studies in software engineering. Journal of Systems and Software, 86(10):2594–
2610, 2013.
[241] A. Yanzer Cabral, M. Ribeiro, and R. Noll. Knowledge management in agile

software projects: A systematic review. Journal of Information and Knowledge
Management, 13(1), 2014.
[242] A. Yin, F. Soraia, and M. M. da Silva. Scrum maturity model: Validation for
IT organizations’ roadmap to develop software centered on the client role. In
Proceedings of the Sixth International Conference on Software Engineering Ad-
vances (ICSEA 2011), pages 20–29, 2011.
258 BIBLIOGRAPHY
[243] V. Zani, D. Feitosa, and E. Nakagawa. Current state of reference architectures

in the context of agile methodologies. In Proceedings of the 23rd International
Conference on Software Engineering and Knowledge Engineering (SEKE 2011),
pages 590–595, 2011.
Appendix A
Agile Practices Definitions
1. Face-to-face meeting: Team sits together, open space office facilitating interac-
tion, video conference if the team is distributed.
2. Self-organizing cross functional team: Small team with no more than 10 mem-
bers that consists of people with different competences (developer, tester, etc.).
Team is independent, takes full responsibility of the task.
3. On-site customer: Continuous user involvement in the development process, cus-
tomer can be consulted anytime if it is needed.
4. Pair programming: Two developers work together at one workstation.
5. Planning game/ sprint planning meeting: The entire team participates in select-
ing the feature to be implemented in the following iteration.
6. Tracking progress: Tracking of the project progress using burn down chart, burn
up chart.
7. Refactoring: Restructuring code for better understandability and reduced com-
plexity.
8. Iteration reviews/ Retrospective: Meeting after each iteration to review the project,
discuss threats to process efficiency, modify and improve.
9. Short iterations & frequent releases: Frequent releases of the software, early and
continuous delivery of partial but fully functional software.
10. Simple Design: Goal to design simplest solution.
11. Time-boxing/ Sprint/ Iteration: Fixed start and end dates are set for iterations and
projects, e.g. 30 days sprint.
12. Stand up meeting: Short daily meeting where the whole team communicate and
reflect on the completed and ongoing work.
260
13. Metaphors & stories: A metaphor is a very high level requirement outlining
the purpose of the system and characterizes what the system should be like.
The metaphor is broken down into short statement of the detailed functionali-
ties called stories. The stories are kept in a backlog.
14. Test-driven/ test-first development: Writing automated test cases for functional-
ities and then implementing (coding) the tested functionalities until the tests are
passed successfully.
15. Continuous integration: Software is built frequently, even a few times a day,
accompanied with testing (unit test, regression test, etc.).
16. Coding standards: Coding rules that are followed by the developers to make sure
that developers write code in the same way.
17. Collective ownership: Everybody in the team can change the code of other de-
velopers in case of maintenance, bug-fixing or other development activities.
Appendix B
Survey Design
262
263
264
265
266
267
268
269
270
ABSTRACT
Background: Software development organi- Results: The examination of the evidence in-
zations frequently face changes that require dicates that there is not one strategy to in-
them to be flexible. The principles and prac- troduce Agile practices that would yield bet-
tices of Agile software are often associated ter results than others.The lack of conclusive
with improving software organizations’ flex- evidence could be caused by the lack of con-
ibility. However, introducing Agile practices sideration on reporting the context of empir-
have its benefits and limitations. To amplify ical studies, particularly on the baseline situa-
benefits and alleviate challenges, Agile adoption, i.e. situation prior to Agile introduction.
tion guidelines are being proposed to pro- A checklist is proposed to capture a baseline
vide strategies for introducing Agile practic- contextual information focusing on internal
es. One instance of such guidelines is known organizational aspects of a software organ-
as Agile Maturity Models (AMMs). AMMs typ- ization: the constellation of team members’
ically suggest that Agile practices are intro- skills and experience, management princi-
duced in certain orders. However, AMMs pro- ples, existing practices and systems charac-
vide contradictory strategies. Thus it is not teristics of the software under development.
known whether one strategy to introduce The checklist was validated by seven experts
Agile practices is better than others. in academia.The experts who participated in
the validation perceived the checklist to be
Objective: The objective of this thesis is to useful and relevant to research.
gather and examine the evidence on the dif-
ferent strategies of introducing Agile practic- Conclusion: The studies presented in this
es, particularly on the order of introduction thesis can be a useful input for researchers
as suggested in the AMMs.The thesis seeks if who are conducting an empirical study in
one order for introducing Agile practices is Agile software development. The checklist
better than others. proposed in this thesis could be used to help
researchers to improve their research design
Method: Combination of empirical studies when evaluating the extent of improvements
were used in this thesis. The data collection from introducing Agile practices. If research-
was done through a survey and semi-struc- ers use the checklist, consistency across em-
tured interviews. This involved analyzing the pirical studies can be improved. Consistency
introduction of Agile practices over time, in reporting empirical studies is desired for
i.e. the start and/or end of Agile practices. comparing and aggregating evidence. In turn,
A qualitative method like qualitative coding this will help practitioners to make a fair as-
was used to analyze data obtained from the sessment whether research results are rele-
interviews. Different quantitative methods vant to their contexts and to what extent the
like inferential statistics and social network results are helpful for them.
analysis were also used. Literature studies
were also conducted to provide background
and support for the empirical studies.
ISSN: 1653-2090
ISBN: 978-91-7295-352-9
2018:06

Introduction of Agile Practices

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction of Agile Practices

Uploaded by

Copyright:

Available Formats

INTRODUCTION OF AGILE PRACTICES

Strategies and Impacts

Blekinge Institute of Technology

Department of Software Engineering

Introduction of Agile Practices

Department of Software Engineering

“If there be nothing new, but that which is

– Sonnet 59, William Shakespeare

Modern English paraphrase:

Background: Software development organizations frequently face changes that re-

Papers in this Thesis

Chapter 3: I. Nurdiani, J. Börstler, and S.A. Fricker, (2018) “Literature Review of

Chapter 6: I. Nurdiani, J. Börstler, and S.A. Fricker, Panagiota Chatzipetrou (2018) “

Related Papers not in this Thesis

Other Papers not in this Thesis

Overview of Papers vii

2 An Analysis of Change Scenarios of an IT Organization for Flexibility

2.7 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 43

3 Literature Review of Flexibility Attributes: A Flexibility Framework for

4 The Impacts of Agile and Lean Practices on Project Constraints: A Ter-

5 Usage, Retention, and Abandonment of Agile Practices: A Reflection on

6 Strategies to Introduce Agile Practices: Comparing Agile Maturity Models

6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

7 Capturing Baseline Situations in Studying the Impacts of Agile Practices

A Agile Practices Definitions 259

B Survey Design 261

1.2.1 Software Organization

Inputs Conversions Outputs

be achieved by modularizing a software system and thereby localizing the impact of

1.2.3 Agile and Lean Software Development

1.2.4 Agile Adoption Guidelines

Table 1.1: Allocation of Agile practices to maturity levels in three AMMs.

1.3 Research Gaps and Contribution

• C3. Formulation of a guideline to help with capturing information regarding

There is a need to understand whether one Agile introduction strategy is better

1.4 Research Questions

• RQ1. To which extent do Agile and Lean practices address flexibility?

• RQ3. What information should be captured in a baseline situation when studying

Table 1.2: Mapping of research questions and chapters in the thesis.

Contributions Research Questions Chapters

1.5 Research Method

1.5.1 Grounded Theory

RQ1 RQ2 RQ3

The aim of Chapter 2 is to capture the process of building software organization

In grounded theory, the sampling strategy is purposive or theoretical sampling [37,

For Chapter 2, an empirical study was conducted at an IT department that provides

1.5.2 Literature Studies

used as channels to distribute the questionnaire. Distributing questionnaires over pro-

1.5.4 Expert Validation

1.5.5 Validity Threats

1.6 Overview of Chapters

Flexibility improvement in a software

RQ1 Identification of flexibility attributes and their

Identification of the different impacts of Agile

Agile practices have been abandoned,

Aggregating baseline context information to Chapter 7

Figure 1.3: Connections between the chapters.

indicate a negative impact of TDD on development time. Furthermore, publication bias

In Chapter 7, a compilation of baseline context information that needs to be cap-

• RQ1. To which extent do Agile and Lean practices address flexibility? In

• RQ3. What information should be captured in a baseline situation when study-

experts whom we consulted found the compilation of information to be useful

1.8 Conclusions and Future Work