Universidad Politécnica de Madrid

Requirement Engineering

Experimentation in Software Engineering

"Quasi Experiment: Pilot vs Production in
Javascript libraries"

Nicolás Forero Velasco

cn.forero@alumnos.upm.es

April 2, 2019

1
Contents

1 Title 3

2 Introduction 3
2.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Experimental Planning 4
3.1 Experimental Unit . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.3 Hypotheses, Parameters and Variables . . . . . . . . . . . . . 5
3.3.1 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3.2 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3.3 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.5 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . 6

2
1 Title
Quasi Experiment: Pilot vs Production in Javascript libraries

2 Introduction
Technology transfer is the process for making accessible the new technology
from scientific research to the market, by passing through a series of phases
that allows its implementation. This can also been applied to the continual
need for innovation that have IT companies in learning new software tools,
hardware or processes that can be integrated in their products/services. In
the case of open source software, developers share their own implementations
with the community by offering free available collaborations for the public.
Therefore, companies using this technology can build their products upon
somebody else’s implementation, with uncertainty about having some issues
in their integration, although it is the same technology, each product differs
in their architecture making complex the integration.
In the case of JavaScript, there is a high trend for sharing libraries between
developers that offer different functionalities depending of the category that
the library belongs to. Moreover these libraries can be complex to integrate
if they offer a suite of several features, with a learning curve involved for the
integration that will have an impact in the product that it is deployed. So
companies must be aware of choosing the best approach for implementing
this libraries, in order to ensure the quality of the product and the use of
resources that they use in this task.

2.1 Research Objective

Introducing a new library into a JavaScript software project by following two
approaches: Pilot and Production approach. For the Pilot approach team
will implement a small version of the project, simulating the target modules
where the library will be deployed in the future, isolated from production at
the beginning, once the pilot project is accepted it will be included in the
production line. And for production approach the new library will be used
directly in Production using real data. Both approaches will be evaluated to
measure quality of the product by using the same validation techniques on
both projects.
The research question for this experiment is the following:
¿Do prior pilot projects produce higher quality implementation than di-
rect production approach when introducing a new library into a JavaScript

3
software project?

3 Experimental Planning
The experiment will be performed in the installations of company ABC, they
will provide all resources that will be needed for the experiment since they
are the interested in ensure the highest quality of their products. They also
will offer a small project that the company is interested for the study, and
the libraries that they want to include in their implementation, so 20 devel-
opers from the company will participate in the experiment. Furthermore the
experiment will take place in three phases: a subject selection, training and
the execution.

Phase Group
Subject Selection -
Training Phase All Subjects
Group A
Execution Phase
Group B

3.1 Experimental Unit

For the subject selection a public call will be open for experienced Javascript
developers to apply to the experiment, this call will be available in a public
manner, so everyone who applies will participate in the experiment. Since
the subjects cannot be allocated randomly this is why it is considered as
Quasi experiment.

3.2 Task
After the subjects are chosen then the training will be done, for this an
small project will be presented to all the subjects during a three day training
session program where they will learn about the product and the different
stack of technologies used in the software project.
For the execution, the subjects will be split among two groups (A and B),
each group will join 10 project leaders, who have experience in production
line, they will be assigned to give some assistance to each group since subjects
don’t have experience with the project
The experiment has two different sessions: In the first session the group
A will have to create a pilot using the different elements that the library

4
provides in isolation from production, so they will have to create a project
from scratch simulating the main functionality based on the product needs,
for this they will use the software requirements specification (SRS) that the
company ABC provides. Once the pilot is ready and approved they will
implement those functionalities in production line.
For the group B they will work with the new library directly in production
line having access to all data and assets from the project, they also need to
follow the same SRS for the implementation.
For the second session a new library will be provided and the execution will
be the same, only that this time group B will be the pilot team and group
A the production. This library will be in a different category than the one
from session one, so the project will be different too.
The duration of the experiment is the same for both groups and equally
evaluated at the end. The quality result will depend on the number of defects
found after applying black box testing in the final product.

3.3 Hypotheses, Parameters and Variables

3.3.1 Hypothesis
Pilot projects produce higher quality implementation than direct production
approach when introducing a new library into a JavaScript software project

3.3.2 Variables

• Factor: Transfer approach in Javascript Library

• Treatments: Pilot and Production

• Response Variables: Quality

3.3.3 Design
The design chosen for the experiment is Within subjects blocked by ob-
jects.

1 Factor 2 Treatments Group A Group B

Session 1; Library X Pilot Production
Session 2; Library Y Production Pilot

Pros:

• No learning from the object

5
 • Better generalization of results

• Biggest sample size

Cons:

• Time consuming

• Resource demanding

• Session and treatment are confounded

Even though there are some downsides of this design compare with others,
this one fits the best to the experiment and the downsides won’t have much
impact in the experiment.

3.4 Procedure
After the experiment finishes, the QA team from the company ABC will
performed black box testing upon the final product from the implementation
in production line. They will design test cases and evaluate the product
across multiple scenarios, taking note of all defects found. At the end the
response variable will depend of the number of defects found for each session
of the experiment, then the group with less defects will represent a higher
quality product.

3.5 Threats to Validity

6
Validation Type Threat Mitigation Implication
Conclusion Heterogeneity To mitigate It can have
of units: In the this threat a slightly error
two groups for large sample of rate in the
each treatment subjects will be results, but will
there can be a chosen to not be
disbalance reduce the significant if
between the heterogeneity the sample is
level of of subjects. large enough.
expertise that
the developers
own. This can
alter the
outcome and
bring up wrong
conclusions.
Internal Selection of Cannot be There is a high
subjects (Ran- mitigated impact in the
domization): experiment,
The subject due to the lack
selection of
cannot be randomization
randomized meaning it will
even though a be treated as
call will be quasi
open for experiment.
subjects to
participate,
they cannot be
allocated
randomly.

7
Validation Type Threat Mitigation Implication
Construct Representation It can be The impact will
Validity: The mitigated if the depend on how
pilot team will session is large complex the
have to manage enough to deal library is and
very well the with the tasks. the level of
time because expertise that
they need to the developers
deal with have.
learning curve
of the new
library and
implementing it
in the pilot and
production
projects.
External The Cannot be Generalization
generalization mitigated cannot be
level of the applied to all
experiment is Javascript
limited to the categories, but
libraries chosen the ones that
to run the belong to the
experiment libraries chosen
which belong to for the
one of multiple experiment.
categories that
exists in
JavaScript.