Professional Documents
Culture Documents
Test Result Reporting: Abstract
Test Result Reporting: Abstract
Indira R
Abstract:
This paper is intended to highlight the importance of result reporting in the context of software
testing. Result reporting can be at various stages of testing like system, integration etc... This
document addresses some of the areas of result reporting at a high level involving independent
system testing ( Black box testing) keeping in mind, the customer as one of the audience.
Published
© QAI India Ltd, 3rdas part of
Annual the proceedings
International of
Software Testing Conference, 2001. No use for profit permitted.
Table of Contents
1. Introduction...................................................................................................................................1
2. Importance....................................................................................................................................1
3. Result Reporting...........................................................................................................................2
3.1 Introduction..............................................................................................................................................3
3.6.1 Analysis............................................................................................................................................6
4. Conclusion..................................................................................................................................10
© QAI India Ltd, 3rd Annual International Software Testing Conference, 2001. No use for profit permitted.
Published as part of the proceedings of 3 International Software Testing Conference, India Page i
rd
1. Introduction
One of the important facets of software development life cycle is Testing. Software testing is an area
that is being considered and given utmost importance in the world of fast changing technology. There
Result reporting is a mechanism for presenting to the customer, from different angles, the state of the
Independent system testing takes about 20 to 50 % of the development time depending on various
combinations of the types of tests conducted on a product. The complex the testing gets, more would
be the effort required. Of course, the more a product is tested, the better the quality would be.
Typically, test result reporting would consist of about 5 to 10% of this effort.
2. Importance
Is it sufficient to give a single line certification that, the product is ready/not ready for
implementation?
O Result reporting is very important and even more so when the product has failed testing.
O There are various aspects that the customer would be interested in knowing about the product like
its performance, platform dependence etc… and just not the compliance to functional
requirements.
appreciate most. This should not be an issue with the testing team having a first hand knowledge
O A detailed result reporting bring in transparency to testing which the customer would appreciate
most. This could help the customer to look for alternate solutions sometimes.
O Reporting could include highlighting the Strengths, Limitations and Recommendations. This
certainly would help the customer to take important decisions about the release of the product in
O It definitely gives a boost to customer confidence both in terms of the product and the reliability
3. Result Reporting
O Any report should have enough information about the product so that anyone going through the
report should be in a position to understand the product, its purpose and its current state.
O Report format should highlight the tested aspects both quantitatively and qualitatively. Though
both are related, the effects are different depending on the audience.
O Graphical representation using charts, tables and other mechanism would make the presentation
1
The term “Product” is a generalized reference which includes an application also apart from commercial
software products.
2
“Audience” can be the Customer, Senior Management, Project Manager, Project Lead, any reader of the report
O The format should be in such a way that one should be able to get the high level picture of the
product at a glance.
➢ A project manager who is responsible for the release of the product would be more interested
in knowing at a high level what would be the state of the product during delivery in terms of
➢ A development lead, apart from knowing the state of the product would be more interested in
looking at the type of defects reported and the corresponding numbers, based on which some
The following sub sections briefs about some of the aspects that could be covered in a report. Please
note that these aspects are not complete in all sense and they are provided here to give an idea about
3.1 Introduction
One could begin the report with an introductory note describing the product that was tested and the
purpose of the report. It could also include highlights of some of the important functions and features
O Importance – Gives the reader an idea about the product being tested.
It is very important to communicate to the reader of the report the scope of testing. One could provide
O In scope of testing
difference to handle each of these aspects while reporting the results. Each of them has its own impact
on the state of the application. Strengths and Limitations of a product could be better presented to the
1. Functionality
2. Regression
3. Security
5. Scalability/reliability
7. Conversion/Migration
8. Platform
9. System Interoperability
10. I18N/Localization
11. Certification/Compliance
13. Installation/Upgrade
Testing is never complete if acceptance criteria are not defined. This should be an integral part of any
report.
O Importance – These are deciding factors for test completion and conclusions.
It is essential to describe the strategy adopted for testing. This could also highlight the basis of testing.
i.e. the input documents that the testing team received based on which the test plan preparation and
O Importance – Gives the reader an idea about the adopted strategy/methodology for testing. This
could be used for verifying if the adopted strategy/methodology for its correctness and
completeness.
Have a section to address at a high level the test details like the test start/end dates, the tool used for
defect reporting, the number of builds released for testing, the number of bugs reported and
importantly the certification part of the testing. This section could be used to present the conclusion
O Importance – Gives the reader an idea at a very high level the efficiency of the development and
This section should follow immediately after the test summary as this again presents the reader at a
high level a qualitative picture of various parameters. The table below could list all the parameters
related to a software so that the reader is able to get at a glance an idea about those aspects which are
1. Conformity
To Standards 10
To Requirements 10
©QAIIndiaLtd,3rdAnnualInternationalSoftwareTestingConference,2001.Nouse
Published as part of the proceedings of 3 International Software Testing Conference, India
rd
5 of
# Attribute Grade Remarks
2. Consistency 10
3. Supports Concurrency 10
5. Usability
Ease of use 10
6. Performance
Query
7 Refer to ticket # 2673240
Data Capture 7 Refer to ticket # 2673241
8. Robustness 10
9. Error Handling 10
3.6.1 Analysis
Provide at a high level, some analysis of the kind of defects found during testing. Describe the basis
O Importance – shows transparency in analysis and methodology followed for concluding the
readiness.
Important aspects that any report should cover. As a testing team, one would get to know a great deal
about the product being tested. It is the duty of the testing team to highlight the strengths and
limitations that it has found during the course of testing. As part of testing team, one would have had
exposure to various products and hence would be having a fair knowledge of various implementation
O Importance –
➢ This would definitely be a value addition to the customer from the point of marketing the
product. Depending on the criticality of the points, the customer could even decide to defer
the release and work on the limitations and recommendations and release a better product.
➢ Unsaid benefits are, increased customer confidence in the testing team and hence increased
In order to present the findings of the testing to the customer, the team has to anyway do a lot of
analysis on the various data collected during the testing. So wouldn’t it be better that they be
O Support conclusions
O Reduce rework effort on the same kind of analysis that the development team may end up doing
Some of the aspects that could be reported under various types of testing are given below.
1. Defect distribution by
O Build
2. Defect analysis based on stage injected ( Requirements, HLD, DLD, Build etc…)
3. Defect analysis based on defect category ( like functional, regression, performance, GUI etc…)
Total 51
4. Throughput Vs Load
© QAI India Ltd, 3rd Annual International Software rdTesting Conference, 2001. No use for profit permitted.
Published as part of the proceedings of 3 International Software Testing Conference, India 8 of
7. CPU utilization Vs Load
Sample:
16
14
12
10
8
L
S e r ie
0
0 5 00 1000 150 0 2000 2 5 00 30 00
T im e ( s e c )
1.4
1.2
0.8
0.6
Response
0.4
0.2
0
0 500 1000 1500 2000 2500 3000
Time(sec)
(Min) (# users) utilization Utilization Mem free Mem free time (secs) Bytes
O One of the key components of testing is the test environment itself. Configurations of the
software(s) and hardware(s) are essential part of result reporting as they have a lot of bearing on
Some of the other important aspects that could be handled as part of result reporting -
O Build - Provide the build release details for testing. Mention all the version release of the
development software used for each build if there are any changes.
O Assumptions - Assumptions play an important role in testing. Test results could vary with various
assumptions as they influence the test strategy in various ways.
4. Conclusion
O There is no end to the value addition that one could provide with good result reporting.
O It could help customers with key decision making with respect to release of products.
O It could probably open up areas/aspects that the customer probably never thought about.
Published
© as part
QAI India Ltd, of the International
3rd Annual proceedings of 3rdTesting
Software International Software
Conference, 2001. NoTesting Conference,
use for profit permitted. India 10 of
10