2 Exercise 2: Research Methods (approx.

takes 8 hours to com-

plete in a group)
The objective of this exercise is to practice the formulation of research questions. In a group (comprising
two students ), you are required to the formulate research question(s) for the given description of a
scenario. You also have to select and justify your choice of the research method(s) that you will use in
order to answer these questions.

2.1 Instructions
Read the following:
1. Chapter 3 and in particular Pg. 59 onwards of C. Robson, K. McCartan, Real World Research, 4th
Edition, Wiley, 2016 ISBN: 9781118745236
2. Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. Selecting empirical
methods for software engineering research. In Guide to advanced empirical software engineering,
pages 285–311. Springer, 2008.
3. Please go through the following links for useful advice:
a. Aim, issue and research questions – delimiting the subject matter1
b. George Mason University’s writing guide2
c. Prof. Feldt’s advice on choosing a thesis topic3 and formulating research questions4 )
4. Read the notes in Section 2.3.
The following are four simplified scenarios from actual research projects. Read the following carefully
and write your answer using the template and instructions in Section 2.2.

2.1.1 Scenario 1
A development manager in a software development company has recently heard about the benefits of
test-driven development (TDD). He thinks that writing test cases before implementation will improve
the quality of the software they write. You (as a researcher) have been asked to evaluate whether TDD
is better than the current way of working, i.e. test last development (TLD).

2.1.2 Scenario 2
Assume that investigation from Scenario 1 showed that TDD does indeed improve the quality of code
significantly. Now the manager is additionally interested in knowing the preference of developers regarding
TDD and TLD. Also, it will be useful to know for the manager what are the major reservations regarding
the new way of working. This would help to decide whether to adopt the change or not and how to
facilitate the adoption.

2.1.3 Scenario 3
A manager in a company that you collaborate with is interested in benchmarking the performance of their
current agile development process. Her aim is to understand the actual process (and not the prescribed
process in the corporate guidelines), what are the main challenges and improvement opportunities with
regard to the development process at the company. You are tasked to conduct this investigation using a
scientifically rigorous approach.

2.1.4 Scenario 4
A colleague and you have developed a new algorithm ‘A’, to select test cases (from a suite of test cases)
that are highly likely to fail (a failing test case is where the system under test (SUT) does not meet the
expected behavior specified in the test case). Now, you want to compare the performance (in terms of
efficiency and effectiveness) against the current industry standard algorithm ‘B’. How will you go about
this evaluation?
1 http://writingguide.se/writing/the-writing-process/#preparation
2 https://writingcenter.gmu.edu/guides/how-to-write-a-research-question
3 http://robertfeldt.net/advice/checklist_for_choice_of_thesis_topic_feldt.pdf
4 http://robertfeldt.net/advice/guide_to_creating_research_questions.pdf

5
2.2 Deliverable
The deliverable should be a pdf document with the following contents submitted through Canvas .

Name P. Number Contribution (25% for each if all

contributed equally)
Student 1
Student 2

For each of the scenarios described in Section 2.1, please report the following:

Assumptions: If there is missing information in the description of scenarios (listed in Section 2.1) which
you deem necessary to define an objective, answerable and focused research question, then please make
reasonable assumptions and document them under this heading e.g. regarding entities, attributes or
measurements to help you design the investigations for the stated scenarios.
Research questions: Please formulate the research questions or hypothesis for the investigation.
Research methodology: Select an appropriate research methodology on the basis of your research
questions or hypothesis.
In case you choose a fixed design method like experiment or survey, please also report the variables
(both dependent and independent variables) for the study. Also, specify how the dependent variable could
be measured (what metrics could be used). Who and what will be the objects, subjects, and treatments
in your experiment? What is your population? What will be your sampling strategy?
In case you choose a flexible design method like case study research, please describe and motivate
the choice of data collection methods and data sources that will be used in the study. For example: if
you will use interviews (whom will you interview, what will be your sampling strategy), archival data,
observations or a combination thereof.
Justification Please provide arguments for why the method you have chosen is appropriate to answer
the said research questions. Please describe what other alternative methods could have been employed to
achieve the objective of the given scenario. Furthermore, describe the potential limitations (e.g. in terms
of possible generalization) of your findings, caused by research design decisions like the chosen research
method, data collection method, or sampling.

2.3 Additional reading material

Besides the reading material highlighted in Section 2.1, following are some additional notes on formu-
lating and evaluating research questions (written by Indira Nurdiani and Ronald Jabangwe [ e-mail:
indira.nurdiani, ronald.jabangwe@bth.se]).
There is no silver-bullet for formulating good research questions. However, there are some measures
that can be used to assess research questions. Such measures can be derived by considering two critical
perspectives: the components and the characteristics of a research question.

2.3.1 Components of a Well Formulated Research Question

The purpose of the guidelines for formulating research questions in evidence-based research in medical [2]
and social sciences [6] is also applicable in software engineering. The notion is to assist researchers with
formulating relevant research questions, which is most likely to lead to obtaining evidence that is of use
in practice. Essentially, the research question should not be too abstract, and it should be sufficiently
general to the target population [5]. The use of the guidelines in the context of software engineering is
exemplified in the study by Kitchenham [4].
Using guidelines from evidence-based medicine [2] and those proposed by [6] for social sciences, a well
formulated research question contains the following components: population (who the study concerns,
e.g., software developers), study factor (concerns the treatment, interventions, etc.), comparison (i.e., if
applicable, the comparison of interventions), outcome and context.
The study purpose should be well-known before formulating the research question. It is imperative
that the components of the research question are aligned with the study purpose. Therefore, in order to
assess a research question from the perspective of its constituents, the evaluation should be based on the
study scope and objectives, population and intervention.
• Is the research question in line with the study objective?
• Is the scope clearly defined?

6
 • Is the population clearly defined?
• Is the study factor or intervention clear?
• Is the expected outcome clear?
• Can the study context be derived?

In some cases it may be necessary to state the type of data in the research question. Some studies
can also consist of comparison between interventions, in which case the data and the comparison of the
interventions need to be clearly defined.

2.3.2 Characteristics of a Well Formulated Research Question

An assessment of a research question can also be performed from the perspective of its characteristics,
as discussed by Robson [7]. This pertains to how the research question can be described.

• Understandable: This refers to how easy it is to understand the research question. In formulating
a research question, it is important to use terms that are easily understood or provide definitions of
those terms, and to use well understood terms. Complex research questions should be avoided. An
example a complex research question is one that has more than one of each of the research question
components described in Section 2.3.1.
• Unambiguous: This characteristic is related to clarity. A research question needs to be formulated
in a way that it is not a subject for multiple interpretation.

• Representativeness to the research purpose: One of the values of a research question is that it
defines the research project itself [7]. Therefore, the components and terminologies used to formulate
the research question need to be relevant to the research purpose.
• Answerable: A research question needs to be answerable, given the source of evidence, and type
of data that we would have access to. We cannot answer a question pertaining to software project
practices in large companies if we only have access to student projects.
• Non-Trivial: A research question needs to be of importance and not addressing concerns of trivial
value. This is a bit difficult to assess because one person’s opinion of importance of value of a research
aim may be different from another. Therefore, it is also important to mention the importance of the
research aim and detail how this aim is to be achieved by answering the research question.

• Coherent: If there are multiple research questions, then it is important to ensure that these
research questions belong together, they are logically related, and collectively contribute to the same
research aim.

2.3.3 Multiple Research Questions

It is possible to have multiple research questions to support an investigation. In such cases it can
happen that the individual research questions are well formulated in terms of the key components and
characteristics, but collectively they are poorly formulated. This may be because, for example, as a set
the research questions become ambiguous and the study scope is difficult for the reader to determine.
We suggest to evaluate the research questions at both individual level and as a collective set using the
key components and characteristics discussed.
In Table 1, we provide examples of research questions from [3]. The study is concerned with identifying
useful source code measures to evaluate external quality attributes of object oriented systems. In this
case the study focus or population of the SLR was empirical studies reporting on the link between object-
oriented measures and quality. The intervention under investigation was the use of methods and source
code measures. The outcome of interest was on the efficacy of the measures with regards to their link to
external quality attributes. Finally, the source of data was empirical studies.
Table 2 summarizes a checklist and key aspects that can help with the formulations of research
questions.

7
 Table 1: Example of research questions

Aim: “aggregating and analyzing measures that have been empirically linked with quality of object-oriented source code.”
RQ1. How are measures derived from source code of object-oriented programs used to evaluate or predict external quality
attributes of object-oriented systems in empirical studies?
RQ1.1. Which external quality attributes have been linked with object-oriented measures in empirical studies?
RQ1.2. Which methods are used to estimate/predict the external quality attributes from the source code measures?
RQ1.3. Which source code measures are used to evaluate external quality attributes of object-oriented programs?
RQ1.4. What is the overall efficacy of object-oriented measures to link with external quality attributes across empirical
studies?
Study Focus: Object-oriented programs
Intervention: methods, source code measures
Outcome: efficacy, estimate, external quality attribute

Table 2: Evaluation of the Research Question

Evaluation in terms of the components

Is the
research question in line with the study objective?
Is the
scope clearly defined?
Is the
population clearly defined?
Is the
study factor or intervention clear?
Evaluation in terms of the characteristics
The research question should be comprehensible.
The research question should be well-defined and not subject to multiple interpretations.
The research question should fit the purpose of the research.
The research question should be answerable.
The research question should address a topic or a problem that is of interest to practitioners and/or researchers.
Multiple research questions should be logically related and collectively fit the research aim.