Assignment 1 – Critical appraisal – Experiment

Student Name email Contribution in the

assignment (25% each
for equal contribution)
MAHMOUD EHSANI ARDAKANI maeh21@student.bth.se 25
IBRAHIM RAHAMATH ULLAH Ulib21@student.bth.se 25
JAHANZEB NAWAZ jana21@student.bth.se 25
MUHAMMAD HAMZA AAMIR muau21@student.bth.se 25

Results of applying the checklist:

Table 1

ID Evaluation item Answer Justification (please elaborate)

(Yes/No/
NA)1
1 Is the experiment Yes Although it was hard to understand the measurement methods
understandable and and formula but in overall it was very interesting and
interesting in general? understandable experiment that they done on student after
passing some lectures on TDD and it was good to understand
how to increase efficiency in TDD but as it says in the paper
break down of the task need a good experience on that
2 Does the experiment Yes This paper was defined very well that all the experiment
have any practical guidelines was clear and all of them was mentioned in the paper
value? so it seems to be practical valuable experiment
3 Are other experiments Yes In the start of paper, they mentioned some other experiments,
addressing the also they try to avoid ethics that was founded on other previous
problem summarized experiments like mono-operation bias
and referenced?
4 What is the population 52 In the paper they mentioned that they use students at the
in the experiment? 
 Students University of Oulu who were enrolled in a graduate-level course
on software quality and testing during 2015 fall semester and 52
students volunteered to participate in this experiment
5 Is the sample used Yes It seems that 52 students that they have different background that
representative of the was mentioned in the paper is a good sample of population.
population? 

6 Are the dependent and yes In the paper they mentioned the variables that are dependent and
independent variables also, they clearly mentioned the main goal and the process
clearly defined? 

7 Are the hypotheses yes It was hard to understand all the formulated that they used in this
clearly formulated? 
 paper but it seems like they clearly formulate everything and all
of them mentioned in the paper
8 Is the type of design yes Part 3.2 clearly stated that they have two level (FG-CG) and two
clearly stated? 
 sequence. Also, they mentioned that experiment was conducted
in two periods with duration of 2 hours. Also, you can find both
scenarios description in the paper

1
Please note for item with ID 4, in the first column of Table 1, the answer will be the population
used in the study and not a yes/no answer.
9 Is the design correct? Yes It seems that the design is correct because they mentioned that

 this design helps them to prevent mono-operation bias
10 Is the instrumentation yes It seems that they describe instrumentation in full manner and
described properly? 
 also, they describe the metrics and formula for how they measure
them
11 Is the validity of the yes It seems that in Evaluation of Design Validity part (3.9) they try
experiment treated to be careful. What motivate us in this question is that as they
carefully and have two different sequence in their design, they use crossover
convincing? 
 method to prevent any effects that caused by the treatments order.
12 Are different types of No They try to explain all threats and addressed them but we cannot
validity threats find anything that motivate us that they cover it in full manner
addressed properly? 

13 Has the data been yes In section 4 they clearly define how they validate data and so by
validated? 
 that it seems that they validate their data
14 Is the statistical power Yes It seems that they provide subjects that used in previous studies
sufficient, are there so it can be enough.
enough subjects in the
experiment? 

15 Are the appropriate yes It seems that they have applied the appropriate statistical tests
statistical tests and parametric tests over the whole case study. And as we
applied? Are understand they can use the both model of tests correctly
Parametric or non-
parametric tests 
used
and are they used
correctly? 

16 Is the significance Yes They used the significance level appropriately as in 4.2.5 Model
level used Validity and Fit
appropriate? 
 To verify the validity of the LMM method, the normality of the
residuals for COMPLETENESS was tested with the Shapiro-
Wilk test.
17 Is the data interpreted Yes As there are 4 steps to interpret the data and all of these steps find
correctly? 
 in the paper so it seems that they done it correctly
18 Are the conclusions Yes It seems that by the results provided in the paper they provide
correct? 
 good conclusion on paper and as it mentions the most important
is break down the task
19 Are the results not No By the different background of experiment population, it seems
overstated? 
 that the result is good.
20 Is it possible to Yes In the paper, you can find all necessary data like formula,
replicate the study? 
 questionary and etc. to replicate the study
21 Is data provided? 
 Yes It seems that all necessary data provided in the paper
22 Is it possible to use yes As meta-analysis is usually preceded by a systematic review, as
the results for this allows identification and critical appraisal of all the relevant
performing a meta- evidence so it seems to be possible. SPection 4 and 5 represents
analysis? 
 the analysis and results of the tests performed during the
research, it could be used further in meta-analysis.
23 Is further work and Yes In the last part they try to explain further works
experimentation in the
area outlined? 

Briefly answer the following questions (where possible support your answer with results of the checklist-based
evaluation):

A. Does the chosen research method (experiment/case study) address the objectives in the study? Which other
research methods could address the same objectives?
Yes. It seems that the chosen method is good and also, we can use the case study in real projects and it seems
it could address to same objectives
B. What is the practical value of the given experiment?
The main practical value of this experiment was that to improve TDD efficiency need to break down the
task and how to do it needs a good experience on it.
C. Which, if any, are the unaddressed ethical issues/concerns in the study?
It seems that the complication of project is ethical concern that they do not work on it.
D. What is your overall assessment about the quality of the experiment? What do you consider are the main
strengths and major limitations of the study?
It seems that the main limitation of the study is how big are the assignments. As they done the experiment
on limited period of time it seems that they are not able to work on big assignments and maybe the result on
big assignment be somehow different
E. Please answer the following questions regarding the use of the given checklist:
a. Please respond to what extent do you agree or disagree with the following statement:
Overall, the checklist was easy to use.
strongly agree agree neutral disagree strongly disagree

b. Please respond to what extent do you agree or disagree with the following statement:
The questions formulations were easy to understand.
strongly agree agree neutral disagree strongly disagree

c. Please write the question numbers (from Table 1), if any, that were difficult to understand.
15-16
d. Please respond to what extent do you agree or disagree with the following statement:
The questions were easy to answer.
strongly agree agree neutral disagree strongly disagree

e. Please write the question numbers (from Table 1), if any, that were difficult to answer.
15-16-22
f. What made it difficult to answer these questions?
Hard to understanding
g. Does the checklist cover all the important aspects (as mentioned in the guidelines for conducting
case study research) for high quality case study research?
Yes. It seems all necessPary parts are covered