Lean R&D: An Agile Research and Development

Approach for Digital Transformation

Marcos Kalinowski1, Hélio Lopes1, Alex Furtado Teixeira2, Gabriel da Silva

Cardoso2, André Kuramoto2, Bruno Itagyba2, Solon Tarso Batista1, Juliana Alves
Pereira1, Thuener Silva1, Jorge Alam Warrak2, Marcelo da Costa2, Marinho Fischer2,
Cristiane Salgado2, Bianca Teixeira1, Jacques Chueke1, Bruna Ferreira1, Rodrigo
Lima1, Hugo Villamizar1, André Brandão1, Simone Barbosa1, Marcus Poggi1, Carlos
Pelizaro2, Deborah Lemes2, Marcus Waltemberg2, Odnei Lopes2, Willer Goulart2
1 Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil
{kalinowski, lopes, tarso, juliana, thuener, bianca, jacques,
bruna, rodrigolima, hvillamizar, andre, simone, poggi}
@exacta.inf.puc-rio.br
2 Petrobras, Rio de Janeiro RJ, Brazil
{gscardoso, alex.teixeira, jawk, mceloscosta, marinhof,
kuramoto, itagyba, cristiance.salgado, carlos.pelizaro, delemes,
marcuswaltemberg, odnei, willer}
@petrobras.com.br

Abstract. Petrobras is a large publicly-held company that operates in the oil,

gas and energy industry. Recently, they conducted internal dynamics to identify
several Digital Transformation (DT) opportunities to leverage their operational
excellence. Addressing such opportunities typically requires Research and De-
velopment (R&D) uncertainties that could lead traditional R&D cooperation
terms to be negotiated in years. However, there are time-to-market constraints
for fast-paced deliveries to experiment solution options. With this in mind, they
partnered up with PUC-Rio to establish a new DT initiative. The goal of this
paper is to present the Lean R&D approach, tailored within this new initiative,
and results of two case studies regarding its application in practice. We de-
signed Lean R&D integrating the following building blocks: (i) Lean Incep-
tions, to allow stakeholders to jointly outline a Minimal Viable Product (MVP);
(ii) early parallel technical feasibility assessment and conception phases, allow-
ing to ‘fail fast’; (iii) scrum-based development management; and (iv) strategi-
cally aligned continuous experimentation to test business hypotheses. In the two
reported case studies, Lean R&D enabled addressing research-related uncertain-
ties early and to efficiently deliver valuable MVPs within four months, showing
itself suitable for supporting the DT initiative. Key success factors were the
business strategy alignment, the defined roles and co-creation philosophy with
strong integration between Petrobras and PUC-Rio’s teams, and continuous
support of a highly qualified research team. Main opportunities for improve-
ment, based on our lessons learned, rely on better adapting Lean Inceptions to
the DT context and on scaling the approach to a project portfolio level of ab-
straction.

Keywords: Digital Transformation, Agile Methods, Lean, Research and Devel-

opment, Continuous Experimentation.
2

1 Introduction
Digital transformation can be seen as a process in which organizations investigate the
use of digital technologies to innovate their way of operating, aiming to solve business
problems and to achieve strategic goals. The resolution of such problems frequently
involves transformations of key business operations that may affect organizational
structures, processes, and products [1]. Organizations of almost all industries are con-
ducting digital transformation initiatives to explore digital technologies and exploit
their benefits [1].
Petrobras is a large publicly-held Brazilian company operating on an integrated ba-
sis and specializing in the oil, natural gas, and energy industry. Internal efforts, includ-
ing the establishment of a new digital transformation board and initiatives within their
main business areas, enabled them to identify several opportunities in which digital
transformation could potentially help them to leverage their operational excellence.
Digital transformation and innovating business processes by using digital technolo-
gies typically involve Research and Development (R&D) efforts. CENPES is the re-
search center of Petrobras, responsible for coordinating and conducting research initia-
tives. Such R&D initiatives commonly involve cooperation terms with research insti-
tutes and universities. These terms were usually designed in a plan-driven manner,
with deliveries that, given research uncertainties, could take up to years. However, in
the digital transformation context, there are time-to-market constraints and a need for
fast-paced deliveries to experiment solution options.
To address these digital transformation needs, they partnered up with PUC-Rio to
establish the ExACTa (Experimentation-based Agile Co-creation initiative for digital
Transformation) initiative. With a different mindset from the previously established
R&D cooperation terms, ExACTa was created to work with an open scope philosophy,
following agile practices for R&D to enable focused and fast deliveries of Minimal
Viable Products (MVPs) that can be used to test digital transformation business hy-
potheses. The ExACTa initiative was launched in September 2019, and the first step
involved designing an R&D approach that would allow fast MVP deliveries. The re-
sulting approach was called Lean R&D.
The Lean R&D approach relies on agile and continuous software engineering prin-
ciples, including establishing a strong link between business and software development
(BizDev) and continuous experimentation practices [2]. Based on these principles, we
designed Lean R&D integrating the following building blocks: (i) Lean Inceptions [3],
to allow stakeholders to jointly outline the vision of Minimal Viable Products (MVPs)
that can be used to test business hypotheses; (ii) parallel technical feasibility assess-
ment and conception phases, allowing solution options to ‘fail fast’; (iii) scrum-based
development management; and (iv) continuous experimentation, to test the business
hypotheses in practice, allowing a build-measure-learn feedback cycle [4]. Moreover,
the initiative counts on a dedicated research team, specialized in data science and ma-
chine learning, to support the development team with parallel investigation activities.
In previous work, reported in a short paper [5], we provided an overview of the first
conceptualization of Lean R&D and initial (then, incomplete) experiences. The goal of
this full industrial paper is to present Lean R&D in further detail and to report on de-
tailed outcomes of two complete industrial case studies, including closing the feedback
cycle with continuous experimentation. Hence, besides providing a more detailed de-
scription, we investigate Lean R&D’s building blocks in much more detail, discussing
the practical experience of applying it, highlighting observed industrial effects.
 3

The herein detailed case studies concern applying Lean R&D in practice to build
digital transformation enabling MVPs for two different business areas of Petrobras:
industrial and logistics. In both cases, following the approach, valuable MVPs were
delivered to stakeholders within a four-month timeframe. Continuous experimentation
allowed to test business hypotheses in practice, supporting strategically aligned product
increment planning. Throughout these case studies, Lean R&D showed itself suitable
for supporting the digital transformation initiative. The business strategy alignment, the
defined roles and co-creation philosophy with strong integration between Petrobras and
PUC-Rio’s teams, and the continuous support of a highly qualified research team, were
observed as key success factors. Main opportunities for improvement, based on our
lessons learned, rely on better adapting Lean Inceptions to the DT context and on scal-
ing the approach to a project portfolio level of abstraction.

2 Background
Before designing Lean R&D, we tried to find a suitable agile approach that was, sim-
ultaneously: (i) digital transformation enabling (e.g., including strategies such as ‘fail
fast,’ focusing on added business value and testing business hypotheses); and (ii)
considering joint research and development activities to allow handling complex
R&D projects (e.g., investigating and conceiving simulation models). Lean principles
have been reported to offer the potential to improve the cost, quality, and speed of the
R&D process [6]. However, we found no Lean R&D approach that was tailored for
software product based, digital transformation enabling solutions. Lean Startup [7],
for instance, inspired us with its business focus but does not consider our specific
need to integrate R&D activities within agile methods to allow handling complex and
research demanding software projects.
The following two subsections provide the background on continuous software en-
gineering [2] and Lean Inceptions [3]. We used the expected dynamics of the first one
to pursue our goal of narrowing the gap between business strategy, development, and
experimenting solution options. Lean Inceptions, on the other hand, were used to help
to align stakeholders to define digital transformation enabling and business strategy
aligned MVPs.

2.1 Continuous Software Engineering

Fitzgerald and Stol [2], in their paper providing a roadmap and research agenda for
continuous software engineering, argue that business, development, and operations
should continuously be aligned with each other. Fig.1 provides an adapted and simpli-
fied representation of such alignment.

Fig. 1. Relations between business strategy, development, and operation. Adapted from [2].
4

The authors coin the term BizDev as the need to align the business strategy with the
development of software [2]. DevOps represents the need to align the development of
software with the deployment of that software into operation [8]. Finally, continuous
experimentation focuses on conducting experiments with stakeholders consisting of
repeated Build-Measure-Learn cycles [4][9].
Reflecting on the implications of these alignments in the context of engineering
digital transformation enabling software products, BizDev and continuous experimen-
tation play a key role in achieving digital transformation goals. After all, digital trans-
formation commonly involves changes in key business operations that affect business
processes and enabling products [1]. A focus on BizDev and continuous experimenta-
tion helps to enable assuring the development of a business strategy aligned product
and to assess the added business value objectively. The importance of continuous
experimentation within digital transformation contexts is also highlighted by Fager-
holm et al. [4]. DevOps, on the other hand, represents a technical competitive ad-
vantage to speed up the development process.

2.2 Lean Inception

Lean Inception is defined by its creator as the “combination of Design Thinking and
Lean Startup to decide the Minimum Viable Product (MVP)” [3]. It is a collaborative
workshop that is intended to help stakeholders to jointly outline the vision of a valua-
ble, feasible, and user-friendly MVP that can be used to test business hypotheses.
The steps involved in a Lean Inception are: defining the product vision; character-
izing and scoping the product vision; describing personas; describing user journeys;
conducting features brainstorming; conducting a business, technical, and UX review;
sequencing of features; and finalizing the MVP canvas.
The final result of a Lean Inception is an MVP canvas, as shown in Fig. 2. Based
on such canvas, the business hypotheses to be validated can be stated as “We believe
that (MVP name) will be able to (outcome statement), we will know that this hap-
pened based on (metrics for business hypotheses validation)” [3].

Fig. 2. Lean Inception MVP canvas. Adapted from [3].

3 The Lean R&D Approach

Our goal was to design an R&D approach for digital transformation, based on agile
and continuous software engineering principles. CENPES and PUC-Rio’s teams jointly
brainstormed the following requirements as input for designing our approach.
R1: Maximize business ‘value’ while minimizing ‘waste.’ A fundamental focus of
the lean philosophy is to shorten the time between a customer order and the delivery
of that order, in such a way that any activities that do not add ‘value’ are considered
‘waste’ and removed [10]. To achieve this goal, there should be a focus on the busi-
ness strategy and its alignment with development (BizDev) [2]. We address this main
 5

requirement in our approach by: (i) using Lean Inceptions involving representative
stakeholders aiming at precisely defining the MVP that best fits the business strategy
and focusing on the essential features to deliver business value; (ii) defining the busi-
ness hypotheses since the beginning and applying continuous experimentation to vali-
date them; (iii) having dedicated business owner representatives at each customer to
help co-creating solutions that maximize business value; and (iii) going for agile and
only essential documentation (e.g., agile requirements [11]).
R2: Allow to ‘fail fast.’ This involves employing the Lean Startup ‘fail fast’ con-
cept [7], which enables handling opportunities and risks involved in experimenting
with digital transformation solution options. The sooner you realize that an idea will
not work, the faster you can update it or even replace it with a new idea. This re-
quirement is addressed in our approach mainly by: (i) including ‘fail fast’ check-
points; and (ii) including a technical feasibility assessment at the beginning of the
process to cope with research-related uncertainties as soon as possible.
R3: Enable addressing complex problems. Digital transformation commonly in-
volves applying cutting-edge digital technology to solve business problems in do-
mains in which they were never applied before. Therefore, our approach considers the
co-creation of solutions with domain experts from the customer side and continuous
support from a qualified research team with a dedicated research team lead and ex-
perts in technologies that are commonly used within digital transformation contexts,
such as data science and machine learning techniques.

3.1 Approach Overview

Based on the aforementioned requirements, we decided to design the approach using as
building blocks Lean Inception, parallel (early) technical feasibility assessment and
conception phases, scrum-based development management, and continuous experi-
mentation. An overview of the designed approach is shown in Fig. 3.

Fig. 3. Lean R&D approach. The timeline is illustrative and specific to our instantiation.

It is possible to observe four checkpoints, a set of activities, and support of a dedi-

cated research team to technical solution related activities. Hereafter we describe the
involved roles and activities.
Lean R&D Roles. The following roles are involved in the approach. To ease under-
standing, we provide examples of how these roles were distributed in the context of
the ExACTa initiative.
Steering Committee. The main role of the steering committee is to assess the pro-
jects at the depicted checkpoints. This assessment aims at: (i) allowing the ‘fail fast’
of ideas that would not deliver the expected business value; and (ii) assuring that the
approach is being used to address relevant innovation and digital transformation chal-
6

lenges. In our specific case, the steering committee for each project is composed by
the coordinators of PUC-Rio’s ExACTa initiative and managerial representatives of
CENPES and Petrobras’ target business area.
Project Manager (Scrum Master). Facilitates the Lean Inceptions and manages the
agile research and development teams, assuring that the overall Lean R&D approach
is being appropriately followed. In our specific case, we have one manager for four
projects in parallel.
Product Owners (POs) and Business Owners (BOs). As in traditional Scrum, POs
are responsible for maximizing the business value of the product resulting from the
work of the development team. These POs are assisted by additional customer repre-
sentatives BOs that work with the team to focus on the co-creation of solutions that
maximize business value. In our specific setting, we have two POs handling two pro-
jects each and at least one BO per project.
Developers. The development teams. Currently, we have twelve full-time develop-
ers working in four projects (three per project).
Research team. The role of the research team is to support the development team in
an early technical feasibility assessment and in complex tasks during development
(e.g., investigating machine learning techniques to be used, elaborating prediction
models). Currently, this team has one research lead supported by four researchers,
serving four projects.
UX/UI design team. Responsible for designing user interaction mock-ups and high-
fidelity prototypes to subsidize the front-end development. Currently, we have one
UX/UI team lead and one UX/UI analyst, serving the four projects.
DevOps and infrastructure analyst. Responsible for providing the DevOps infra-
structure to the development teams. Currently, we have one DevOps analyst serving
four projects.
Lean R&D Activities. The approach starts with a Lean Inception to allow stakehold-
ers to jointly outline the vision of an MVP that can be used to test business hypothe-
ses. It is important to involve representatives of all relevant stakeholders during this
phase. Thereafter, the defined MVP has to be approved by the steering committee
(refer to the first checkpoint in Fig.3). If it gets rejected, a new Lean Inception should
be conducted, potentially focusing on a different problem. Referring to the suggestive
timeline, the typical duration of a Lean Inception is of five business days [3]. Howev-
er, in our specific case, we have managed to conduct them within three business days.
In the Technical Feasibility phase, the development team, assisted by the research
team and the DevOps analyst, starts investigating the technical feasibility of imple-
menting the features identified during the Lean Inception. Following the tracer bullet
strategy [12], this phase typically serves as proof that the architecture is compatible and
feasible and that there is a way to solve the problem with reasonable effectiveness, as
well as providing a working, demo-able skeleton with some initial implementations.
The Conception involves the PO detailing the MVP features identified during the
Lean Inception by applying product backlog building dynamics with the customer
representatives, followed by other typical requirements elicitation techniques (e.g.,
interviews), to specify user stories. Additionally, aware of severe negative impacts of
underspecified agile requirements [13], we complement user stories by specifying
Behavior-Driven Development (BDD) scenarios that can later be used as objective
acceptance criteria [14]. During the conception phase, the UX/UI team participates by
creating low-fidelity prototypes (e.g., mock-ups) for requirements validation and high-
fidelity UI prototypes for usability testing.
 7

At the end of the conception, the agile requirements specification (containing user
stories, BDD scenarios, and mock-ups) is reviewed and validated with the customer,
and usability tests are conducted on the high-fidelity prototypes. It is noteworthy that
careful requirements reviews (e.g., inspections) are among software engineering’s best
practices [15], capturing about 60% of the defects early, when they are cheaper to fix,
significantly reducing rework, overall development effort, and delivery schedules [16].
Additionally, defects in requirements have other severe consequences, including cus-
tomer dissatisfaction and overall project failure [17]. Thus, this phase, concerning the
specification of what should be implemented, deserves special attention.
The second checkpoint involves the steering committee analyzing the requirements
specification, together with the results of the technical feasibility assessment, require-
ments review, and usability tests, to decide whether the MVP should be developed.
Thereafter, the Agile Development phase involves the development team, with the
support of the research team, implementing the MVP. The support of the research team
is typically welcome for more complex specific parts (e.g., building machine learning
models). Basically, this phase follows standard Scrum-based development with sprint
planning, daily meetings, and sprint review cycles. For quality improvement purposes,
we recommend using modern code reviews, which enable identifying faults, improving
solutions, and sharing knowledge and code ownership [18]. While the sprint duration
could be adjusted, in our specific case, we use sprints of two weeks and a custom
dashboard that allows monitoring the overall team progress (cf. Section 5).
Once the MVP is developed, the next checkpoint involves the PO presenting the
MVP to the steering committee, so that they can decide upon its transition into produc-
tion. While this major checkpoint happens at the end of development, the customer
representatives (BOs) are also involved in the sprint planning and sprint review activi-
ties during the development period, where they can always provide feedback to help
co-creating the product that best fits their business needs.
Finally, the Transition phase involves the development and infrastructure team
preparing the MVP for beta testing in its final environment and assessing the business
hypotheses. The last checkpoint concerns analyzing continuous experimentation re-
sults, to investigate whether the business hypotheses were achieved and whether it is
worth investing in another Lean R&D cycle to further improve the product (in this case
the Lean Inception could be replaced by a simplified product increment planning cer-
emony). The research team is supposed to design the experiment plan, which should
outline how to instrument the product to allow gathering the measurements required to
test the business hypotheses, and eventually building other assessment instruments
(e.g., questionnaires to measure user satisfaction). It is noteworthy that we intend to
continuously improve the approach based on causal analysis process improvement
practices [19].

4 Case Study Design

The description of our case study design is based on the guidelines for conducting
case study research in software engineering by Runeson et al. [20].

4.1 Context
Petrobras is the largest company in Brazil and is active in the oil, natural gas, and ener-
gy industry. In 2019 they established a new board focusing on digital transformation
and identified and prioritized several digital transformation opportunities within differ-
ent business areas.
8

Aiming at coping with their digital transformation needs, CENPES partnered up

with PUC-Rio’s informatics department to establish the ExACTa initiative. Differently
from previous experiences, this one should function with an open scope, following
agile practices for Research and Development (R&D) to enable focused and fast deliv-
eries of Minimal Viable Products (MVPs) that can be used to test digital transfor-
mation business hypotheses. Of course, such a cooperation term relies on strong cus-
tomer involvement and a previously established relationship of trust between the two
parties. The ExACTa initiative was launched in September 2019, and the first step
involved designing Lean R&D.
The first two projects started in December 2019, and their first MVPs were deliv-
ered within a four-month timeframe. Currently, the ExACTa initiative runs four such
projects in parallel. To cope with these demands, the initiative counts on four profes-
sors of the informatics department (active in the areas of data science, software engi-
neering, optimization, and human-computer interaction), and hired 21 additional full-
time employees (1 scrum master, 1 research team lead, 1 UX/UI team lead, 12 devel-
opers, 4 research team members, 1 UX/UI analyst, and 1 DevOps and infrastructure
analyst). The case study concerns the first two projects. More details on the case and
subject selection are provided in Section 4.3.

4.2 Goal and Research Questions

The goal of the case studies can be defined, following the GQM template [21], as fol-
lows: “Analyze the Lean R&D approach with the purpose of characterization with re-
spect to its overall outcomes and stakeholder perceptions, and the acceptance of its
main building blocks from the point of view of the stakeholders and researchers in the
context of the projects undertaken within the ExACTa co-creation initiative.” From this
goal, we derived the research questions.
RQ1: What have been the overall outcomes of the Lean R&D approach? To an-
swer this question, we access the data from the agile management system (Microsoft
DevOps) and discuss deliverables that have been accepted by the customer.
RQ2: What are the perceptions of the main stakeholders on the Lean R&D ap-
proach so far? To answer this question, we asked the main Lean R&D stakeholders
from the involved business areas for feedback and analyzed this feedback qualitative-
ly.
RQ3: What is the acceptance of applying Lean R&D’s Lean Inceptions to define
MVPs? To answer this question, we applied a survey based on the Technology Ac-
ceptance Model (TAM) [22], which has been commonly used to measure acceptance
[23], to all Lean Inception participants.
RQ4: Does Lean R&D’s early technical feasibility assessment phase help to ad-
dress research-related uncertainties? To answer this question, we analyze the tasks
and comments within the agile management system and the meeting minutes, to retro-
actively reflect on each case.
RQ5: Does Lean R&D’s agile scrum-based and research-supported development
fit well into the digital transformation initiative? To answer this question, we reflect
on the dynamics of scrum plannings, reviews, and daily meetings, and data from the
agile management system.
RQ6: Does continuous experimentation help to test business hypotheses and
provide feedback? To answer this question, we reflect on data regarding the usage of
the solutions and on additional evaluation instruments (questionnaire) used to assess
the MVP's business hypotheses.
 9

4.3 Case and Subject Selection

We selected the first two projects by convenience. Details on each case follow.
Case 1: Intelligent monitoring of gas emissions by oil refineries. This case ad-
dresses a need of the industrial business area within Petrobras, and concerns building
artificial intelligence models to predict refinery gas emissions, based on operation
controls and environmental sensor data. This system should help to improve the capa-
bility of environmental monitoring and help to reduce environmental complaints by
the community (e.g., regarding bad smells). Besides the ExACTa team, this case had
three employees of Petrobras (BOs) working co-located within the ExACTa initiative
space at PUC-Rio1. All these team members participated in the Lean Inception, as
well as the sponsor at CENPES, the sponsor at Petrobras’ industrial area, and repre-
sentatives of employees of the target refineries.
Case 2: Intelligent logistics control of service providing ships, helping to identify
and handle off-hire situations. This case addresses a need of the Logistics business
area within Petrobras, and concerns building intelligent controls, integrating infor-
mation from several systems to identify and handle off-hire situations (i.e., situations
in which a chartered ship is not available), which should be deducted from payments
to the service providers (as well as the fuel used during off-hire periods). Besides the
ExACTa team, this case had four employees of Petrobras (BOs) directly involved in
co-creating the solutions. While they did not work full time, they participated in all
Scrum plannings and reviews and were always available remotely and willing to con-
tribute. All these team members participated in the Lean Inception, as well as the
sponsor at CENPES, the sponsor at Petrobras’ logistics area, and employees involved
in operating ships and administering ship charter contracts.

4.4 Data Collection and Analysis Procedures

The author team includes members of both cases (they participated in discussions
regarding the approach and helped to adjust it until reaching the herein described
format) and has direct access to all other team members at PUC-Rio and Petrobras.
The authors also had representatives in both Lean Inceptions, in the sprint plannings,
reviews, and daily meetings, allowing them to precisely observe the approach in its
real context. Moreover, they had complete access to the agile management system
(Microsoft DevOps) and all project-related artifacts, including meeting minutes.
Additionally, as the Lean Inceptions involved several stakeholders, we conducted
a survey based on the TAM questionnaire and open questions, which were analyzed
qualitatively and anonymously. Also, for continuous experimentation purposes, we
measured usage data of the provided solutions and applied additional questionnaires.
Moreover, we asked the sponsors at Petrobras’ involved business areas for additional
feedback on their perceptions. This feedback was also qualitatively analyzed to help
us further understand the overall acceptance from a managerial perspective.

4.5 Validity Procedures

All the quantitative data was collected from the agile management system and real
project artifacts. The agile management system is directly integrated with changes in

1 Since March 23rd activities moved to home-office due to COVID-19. Following the recom-
mendations in [24] as much as possible and using proper remote tool support (Microsoft
Azure DevOps and Teams) allowed us to keep the Lean R&D approach running remotely.
10

the source code and any other project artifact. The status of tasks within this system is
verified on a daily basis during the daily meetings. Anonymity was employed in all
questionnaires, allowing stakeholders to freely express their opinions.

5 Results and Discussion

Hereafter we describe the results of the case studies. We decided to describe them
together in a joint analysis and discussion, focusing on answering each research ques-
tion based on observations from both cases.
RQ1: What have been the overall outcomes of the Lean R&D approach? Regard-
ing the outcomes, both cases recently had their first MVP accepted by the customer
and delivered to the end-user within a four-month timeframe. Case 1 delivered an
MVP with 6 features (detailed in 28 user stories), while Case 2 delivered an MVP
with 5 features (detailed in 53 user stories). Fig. 4 shows screenshots of functionalities
developed for Case 1 and Case 2. MVPs are now available to the end-users for beta
testing, evaluating the associated business hypotheses and identifying opportunities
for further improvement (e.g., in the format of new MVP versions). The MVP for
Case 1 uses a decision tree model to predict the probability of gas emissions above a
certain level and potential causes and enables to register and correlate complaints
from the community. The MVP for Case 2 uses intelligent data crossings to enable
effectively detecting and handling off-hire events within ship charter contracts.
Regarding the process outcomes, based on data from the agile management sys-
tem, it is observable that the development team adjusted to the process and produced
the Lean R&D artifacts (process outcomes) as expected. All Lean Inception artifacts
were organized in the agile management system’s wiki. During the conception phase,
features identified in the Lean Inceptions were detailed into user stories with BDD
scenarios, mock-ups were built and high-fidelity prototypes designed and validated.

(a) (b)
Fig. 4. Screens of developed functionalities for Case 1 (a) and Case 2 (b). Both had their first
MVPs delivered within a four-month timeframe. Figures included for illustrative purposes,
the focus of this paper is on the approach, not on the implemented solutions.

With respect to the technical feasibility and research and development artifacts, in
Case 1, due to access to confidential data, the team had to use an external Petrobras
repository for committing their artifacts (e.g., code, models, and configuration files).
Therefore, commits were not directly linked to the tasks in the management system.
In Case 2, the agile management system’s integrated Git repository was used, creating
a branch for each task and using modern code reviews to assure code quality during
pull-requests.
 11

Regarding the sprint plannings and reviews, meeting minutes were registered for
each event (every two weeks) in the agile management system. For Case 1, we also
conducted weekly managerial status report meetings and registered meeting minutes
for them, as critical stakeholders could not promptly adjust their activities to attend
our sprint planning and review schedules.
RQ2: What are the perceptions of the main stakeholders on the Lean R&D ap-
proach so far? We asked the main stakeholders at Petrobras’ industrial (Case 1) and
logistics (Case 2) areas for feedback and analyzed this feedback qualitatively. There-
fore, we reached out to them, asking them to write a short open text on their overall
perceptions so far. The feedback was extremely positive.
The manager responsible for the industrial business area emphasized the co-
creation process, effectiveness in adding business value, speed, and the evolutionary
MVP approach: “The integration of the technical process engineering and IT teams of
Petrobras with the development teams at PUC-Rio is a main advantage for achieving
effective results, adding business value in a fast, collaborative and evolutionary way.”
It is noteworthy that this area had three Petrobras employees working most of the time
collocated (after the pandemic virtually) with the team, offering tremendous help
towards achieving the goals and co-creating the solutions.
The representative of the logistics area was in-line with these arguments and em-
phasized co-creation, agility, and efficiency: “The co-creation partnership with Ex-
ACTa has reflected the goals pursued by the logistics area: alignment between plan-
ning and accomplishments, agility and efficiency.” He also wrote that “The initial
impact of the different working method proposal, given the results, soon gave way to
confidence. The team demonstrates control over the development, with continuous
communication and predictability over the terms and scope of agreed deliveries”. This
statement highlights the adaptation and acceptance of the new agile method, after a
completely understandable initial skepticism, observed from stakeholders of both
cases, at the beginning.
RQ3: What is the acceptance of applying Lean R&D’s Lean Inceptions to define
MVPs? The Lean Inceptions were conducted involving the identified key stakehold-
ers for each case. Fig. 5 shows part of both Lean Inception teams in action. It illus-
trates the dynamics of co-creating a joint vision of an MVP that should add business
value, be technically feasible, and user-friendly (Lean Inception includes a specific
business, technical, and UX review activity before sequencing identified features into
MVPs).

(a) (b)
Fig. 5. Kicking off the Lean Inception of Case 1 (a) and discussing the final feature sequencing
with some Lean Inception participants of Case 2 (b).

To investigate the acceptance, we applied a questionnaire, designed based on the

TAM questionnaire [22] adding an open text question asking for suggestions. The
questionnaire was applied to all Lean Inception participants, but answering was not
12

mandatory (eleven participants answered in both cases). An excerpt from the results
of the TAM questionnaire, regarding the stakeholder perceptions that best help an-
swering RQ3 (speed and precision when defining the MVP, usefulness, ease of use,
and intention to adopt) is shown in Table 1 for Case 1 and Table 2 for Case 2. While
participants were asked to identify whether they were from Petrobras or PUC, an-
swers were anonymously collected.
To facilitate an overview of the results, we highlighted the cells with the highest
value within each line. Based on these highlights, it is possible to observe an overall
acceptance of using Lean Inceptions to define the joined vision of the MVP, with
mainly neutral to positive perceptions in Case 1 and mainly positive perceptions in
Case 2. It may be possible to explain the differences between the cases based on the
fact that the Lean Inception conducted in Case 1 was the overall first one conducted
within the ExACTa initiative. Also, based on the feedback collected from the open
questions in Case 1, we held a contextualization meeting at the customer side before
starting the inception of Case 2. We also identified some improvement suggestions
regarding details of the Lean Inception method within the provided answers.

Table 1. Lean Inception TAM Questionnaire for Case 1.

Statement Comp. #Answers SD D N A SA
Petro. 5 0% 0% 20% 80% 0%
High Speed
PUC 6 0% 17% 17% 33% 33%
Petro 5 0% 0% 80% 20% 0%
High Precision
PUC 6 0% 0% 50% 50% 0%
Petro 5 0% 0% 40% 40% 20%
High Usefulness
PUC 6 0% 0% 17% 33% 50%
Petro. 4 0% 0% 75% 0% 25%
Easy to use
PUC 6 0% 17% 33% 33% 17%
Petro 5 0% 0% 40% 40% 20%
Intention to adopt
PUC 6 0% 17% 17% 33% 33%
SD: Strongly Disagree, D: Disagree, N: Neutral, A: Agree, SA: Strongly Agree.

Table 2. Lean Inception TAM Questionnaire for Case 2.

Statement Comp. #Answers SD D N A SA
Petro. 4 0% 0% 0% 75% 25%
High Speed
PUC 7 0% 14% 0% 29% 57%
Petro 4 0% 0% 25% 75% 0%
High Precision
PUC 7 0% 0% 29% 29% 43%
Petro 4 0% 0% 25% 50% 25%
High Usefulness
PUC 7 0% 0% 0% 43% 57%
Petro. 4 0% 0% 0% 75% 25%
Easy to use
PUC 7 0% 0% 14% 57% 29%
Petro 4 0% 0% 0% 50% 50%
Intention to adopt
PUC 7 0% 0% 0% 14% 86%
SD: Strongly Disagree, D: Disagree, N: Neutral, A: Agree, SA: Strongly Agree.

Based on these results and our overall perception, we believe that the Lean Incep-
tions helped to understand the overall context, enabling to outline an MVP and a pri-
oritized set of features, which subsidize the next Lean R&D activities (e.g., the con-
ception where features are detailed into user stories and the technical feasibilities,
where the tracerbullet strategy is applied to check if it is possible to implement the
identified features). Moreover, it also helped to understand the continuous experimen-
tation needs, by identifying the business hypotheses. Among the open text answers
the main opportunities for adjustment to the R&D context concern improving the
business, technical, and UX review step, as some participants highlighted that this
step should not be conducted with the entire group, but properly separating main
 13

stakeholders into specific groups for a more precise assessment. I.e., it was observed
that developers are typically not able to appropriately judge the business value of
features, while business stakeholders have similar difficulties with the technical re-
view on feasibility and effort. Moreover, UX related stakeholders typically found the
information gathered during the Lean Inceptions insufficient for subsidizing UX fea-
ture assessments. Indeed, in our experiences, the assessments had to be reviewed after
the product backlog dynamics conducted during the conception phase.
RQ4: Does Lean R&D’s early technical feasibility assessment phase help to ad-
dress research-related uncertainties? We analyzed the tasks and comments within
the agile management system, meeting minutes, and the observed experience within
the projects. Analyzing the tasks indicated that in both cases, this phase was needed
before starting the development sprints, enabling to address research-related uncer-
tainties and infrastructural issues, also with the support from Petrobras’ IT teams, as
soon as possible.
For Case 1, the tasks accomplished within this phase mainly concerned: (a) inves-
tigating alternatives for building a prediction model with reasonable accuracy, (b)
testing integrations and access to required data, and (c) solving infrastructure-related
problems. For Case 2, the tasks accomplished within this phase mainly concerned
experimenting some architectural solution options and aligning them with Petrobras’
standards and investigating the integration and compatibility with Petrobras’ legacy
systems.
At this point, it is important to highlight the support from the parallel research team
and the infrastructure analyst. Of course, this support is also important during devel-
opment, but in this early technical feasibility assessment phase, it is enabling and
crucial. After the delivery of the MVP of Case 1, one of the developers mentioned
within the team communication channel that “it would not have been possible to
properly address this problem within the expected timeframe without the early inves-
tigations and support of the research team.”
RQ5: Does Lean R&Ds agile scrum-based and research-supported development
fit well into the digital transformation initiative? Here we reflect on the dynamics
of scrum plannings, reviews, and daily meetings and the transparency provided by the
agile management system. Our overall conclusion is that yes, it fits well when using
Lean R&D adaptations (e.g., a strong focus on precise, agile specifications, address-
ing architecture and research uncertainties at the very beginning, and proving contin-
uous research support to the development team).
Sprint planning, review, and daily meetings played a key role in facilitating man-
agement and communication and establishing a co-creation team spirit. Transparent
and continuous access to all sprint planning and review meeting minutes by all stake-
holders helped to provide transparency and building trust. Transparency was also
provided by properly configuring tool support for monitoring development progress.
We designed a customized dashboard, used within all projects, to show the overall
project progress in real-time (i.e., as soon as a developer concludes a task, the dash-
board is automatically updated). The dashboards of Case 1 and Case 2 can be seen in
Fig. 6. We keep these dashboards projected and continuously visible to the whole
project team (also always remotely accessible through the Microsoft Azure DevOps
system). Nevertheless, considering the initiative as a whole, even organizing infor-
mation on the different projects with similar dashboards, we noticed shortcomings for
managing information at a portfolio level of abstraction.
Initially, we faced some resistance from customers of both cases in following the
agile co-working philosophy. As the results started to be delivered, this resistance was
14

replaced by confidence, and a joyful co-creation environment was established. We

believe that complete progress transparency also helped in this direction.

(a) (b)
Fig. 6. Standardized and interactive project monitoring dashboards for Case 1 (a) and Case 2
(b), showing the progress of the product, the current sprint, and of minor tasks.

RQ6: Does continuous experimentation help to test business hypotheses and pro-
vide feedback? The Lean Inceptions helped to identify target business hypotheses for
continuous experimentation. We will focus the discussion of this research question on
MVP of Case 1, mainly due to space constraints and because this one started being
used by the end-users earlier (at the beginning of June, while MVP of Case 2 started
being used at the beginning of July). The information we have so far for Case 2, is
that, according to the representative of the logistics area, “employees involved in
operating ships and administrating ship charter contracts [i.e., end users] are satisfied
with the delivered MVP and were able to start using it after a very short training peri-
od.” Indeed, after some days of initial usage, they identified and started handling 16
off-hire events through the system. Nevertheless, this information does not allow us to
test the business hypotheses yet (which involve comparing off-hire deductions and
handling time).
MVP of Case 1 was deployed in the cloud, and we could collect usage data directly
from the Microsoft Azure cloud platform. We measured the distribution of usage time
(proxied by the amount of exchanged data) and the number of users (proxied by the
number of active sessions) over time. Fig. 7 shows these measures for the period of
June 10th to July 10th. It is possible to observe an increase in the usage time and also in
the number of users (active sessions). Two refinery operators at Petrobras started
using the solution for monitoring gas emissions at least once a week (eventually, there
were more than two simultaneous users). It is possible to see that they started spend-
ing more time in the system as they were becoming more familiar with it, which pro-
vides a preliminary indication of its perceived usefulness.

Fig. 7. Microsoft Azure metrics used as proxies for usage time and number of users.

Regarding the evaluation of the business hypotheses, there were two business hy-
potheses for Case 1. Hypothesis 1: “We believe that the MVP for Case 1 will be able
 15

to reduce the number of complaints by the society regarding bad smells related to
refinery gas emissions, we will know that this happened based on the average number
of complaints”; and Hypothesis 2: “We believe that the MVP for Case 1 will be able
to allow faster diagnosis of the causes, we will know that this happened based on the
average time spent on the diagnosis.” In the case of the diagnosis, it was supported by
showing the decision tree path that led to the inference of high gas emission, and also
letting users consult the whole decision tree.
Unfortunately, while we provided means to monitor the related metrics, the time
of deployment would not be sufficient to observe changes in the averages yet. There-
fore, we used an additional instrument to preliminarily assess our hypotheses; a ques-
tionnaire answered directly by the end-users in the refinery. Both main users an-
swered that they completely agreed that the solution would help to lower the number
of complaints and also the cause identification time. Moreover, they provided valua-
ble feedback, with new features to be included in the next product increment (e.g.,
automatic notification alerts) and also showing to be satisfied with the provided solu-
tion. E.g., one of them mentioned that “the interface is well organized and information
is properly presented, allowing interactions to filter the period and to understand the
decision tree inferences.”

6 Concluding Remarks
In this paper, we presented the Lean R&D approach, tailored to meet digital transfor-
mation related needs, including the ability to fail fast and agile and fast-paced deliver-
ies of complex solutions. The development of such products commonly involves
R&D efforts. However, we found no digital transformation focused approach availa-
ble that appropriately considers integrating R&D efforts into an agile development
philosophy. Lean R&D was designed with this focus, based on the following building
blocks: (i) Lean Inceptions, to allow stakeholders to jointly outline a Minimal Viable
Product (MVP); (ii) parallel technical feasibility assessment and conception phases;
(iii) scrum-based research and development management; and (iv) strategically
aligned continuous experimentation to test business hypotheses.
We applied Lean R&D in two case studies. Lean R&D enabled defining a joined
MVP vision, addressing research-related uncertainties early, and to efficiently deliver
valuable MVPs which were accepted by the end-users. Based on our experience, pre-
cisely defining business hypotheses and the focus on continuous experimentation
strengthen the BizDev integration, helping to guide the overall development efforts
since the beginning and avoiding to lose the focus on the main business goals. This
business strategy alignment, the defined roles and co-creation philosophy with strong
integration between Petrobras and PUC-Rio’s teams, and the continuous support of a
highly qualified research team, exploring synergies with the university's research
program, were observed as key success factors. Main opportunities for improvement,
based on our lessons learned, rely on better adapting Lean Inceptions to the DT con-
text and on scaling the approach to a project portfolio level of abstraction.
While we are aware that these case studies were conducted in a specific context,
we believe that sharing the approach and our evaluation experiences could help other
organizations involved in digital transformation initiatives.

Acknowledgments
The authors would like to thank all ExACTa PUC-Rio and Petrobras employees in-
volved in the projects for their trust and dedication to the new initiative.
16

References
1. Matt, C., Hess, T., Benlian, A.: Digital transformation strategies. Business & Information Sys-
tems Engineering 57(5), 339–343 (2015).
2. Fitzgerald, B., Stol, K.J.: Continuous software engineering: A roadmap and agenda. Journal of
Systems and Software 123, 176–189 (2017).
3. Caroli, P.: Lean Inception: how to align people and build the right product. Editora Caroli
(2018).
4. Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The RIGHT model for continuous exper-
imentation. Journal of Systems and Software 123, 292–305 (2017).
5. Kalinowski, M., Batista, S.T., Lopes, H. et al.: Towards lean R&D: an agile research and devel-
opment approach for digital transformation. In: Euromicro Conference on Software Engineering
and Advanced Applications (SEAA), 5p. (in press), Portoroz, Slovenia (2020).
6. Reinertsen, D., Shaeffer, L.: Making R&d lean. Research-Technology Management, 48(4), 51-
57 (2015).
7. Ries, E.: The Lean Startup: how today's entrepreneurs use continuous innovation to create radi-
cally successful businesses. Crown Business, New York (2011).
8. Debois, P.: Devops: a software revolution in the making. Journal of Information Technology
Management 24(8), 3–39 (2011).
9. Bosch, J.: Building products as innovation experiment systems. In: Cusumano M.A., Iyer B.,
Venkatraman N. (eds.) International Conference on Software Business (ICSOB), LNBIP vol.
114, pp. 27–39. Springer (2012).
10. Ohno, T.: Toyota Production System: Beyond Large-Scale Production. CRC Press (1988).
11. Wagner, S., Mendez, D., Felderer, M. et al.: Status quo in requirements engineering: A theory
and a global family of surveys. ACM Transactions on Software Engineering and Methodology
(TOSEM), 28(2), 1-48 (2019).
12. Thomas, D., Hunt, A.: The Pragmatic Programmer: your journey to mastery. 2nd edn. Addison-
Wesley Professional (2019).
13. Mendes, T.S., de Freitas Farias, M.A., Mendonça et al.: Impacts of agile requirements docu-
mentation debt on software projects: a retrospective study. In: Proceedings of the ACM Sympo-
sium on Applied Computing (SAC), pp. 1290–1295. Pisa, Italy (2016).
14. Smart, J.F.: BDD in Action: Behavior-driven development for the whole software lifecycle.
Manning (2015).
15. Aurum, A., Petersson, H., Wohlin, C.: State‐of‐the‐art: software inspections after 25
years. Software Testing, Verification and Reliability 12(3), 133–154 (2002).
16. Boehm, B., Basili, V.R.: Software defect reduction top 10 list. IEEE Computer 34(1), 135–137
(2001).
17. Mendez, D., Wagner, S., Kalinowski, M. et al.: Naming the Pain in Requirements Engineering –
Contemporary problems, causes, and effects in practice. Empirical Software Engineering 22 (5),
2298–2338 (2017).
18. Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: In-
ternational Conference on Software Engineering, pp. 712–721. San Francisco, USA (2013).
19. Kalinowski, M., Mendes, E., Card, D. N., Travassos, G. H.: Applying DPPI: A defect causal
analysis approach using bayesian networks. In: International Conference on Product Focused
Software Process Improvement (PROFES), pp. 92-106. Oulu, Finland (2010).
20. Runeson, P., Host, M. Rainer, A., Regnell, B.: Case study research in software engineering:
Guidelines and examples. J. Wiley (2012).
21. Basili, V.R., Rombach, H.D.: The TAME project: Towards improvement-oriented software en-
vironments. IEEE Transactions on software engineering 14(6), 758–773 (1998).
22. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Quarterly 13(3), 319–340 (1989).
23. Turner, M., Kitchenham, B., Brereton, P., Charters, S., Budgen, D.: Does the technology ac-
ceptance model predict actual use? A systematic literature review. Information and Software
Technology 52 (5), 463–479 (2010).
24. Ralph, P., Baltes, S., Adisaputri, G. et al.: Pandemic programming: How COVID-19 affects
software developers and how their organizations can help. Empirical Software Engineering, ac-
cepted for publication, 34p., in press (2020).