REQUIREMENT ANALYSIS AND END-TO-END TESTING OF THE ABAP NIGHTLY BUILD TOOL

QMJ ZG629T: Dissertation

By DEEPAK.K.N ID No. 2009HZ74102

Dissertation work carried out at

SAP Labs India, Bangalore.

BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE PILANI (RAJASTHAN)

April 2011

REQUIREMENT ANALYSIS AND END-TO-END TESTING OF THE ABAP NIGHTLY BUILD TOOL
QMJ ZG629T: Dissertation By DEEPAK.K.N ID No. 2009HZ74102 Dissertation work carried out at
SAP Labs India Pvt Ltd, Bangalore.

Submitted in partial fulfillment of M.S. Quality Management Degree Programme Under the Supervision of Ms. Ashwini Naik, Development Manager SAP Labs India Pvt Ltd, Bangalore.

BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE PILANI (RAJASTHAN)

April 2011

BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE, PILANI

CERTIFICATE
This is to certify that the Dissertation entitled, Requirement Analysis and End-to-End Testing Of the ABAP Nightly Build Tool submitted by Deepak.K.N having ID-No. 2009HZ74102 for the partial fulfillment of the requirements of M.S. Quality Management degree of BITS embodies the bonafide work done by him under my supervision.

Date: 18-03-2011 Place: Bangalore

Ms. Ashwini Naik Development Manager SAP Labs India Pvt. Ltd, Bangalore

ABSTRACT

Add On Factory is a team within SAP, providing support to various SAP Products since its inception. The tasks preformed here include providing Installation Packages, Support Packages, Conflict Resolution Transports, Upgrade Packages; while supporting the several landscapes used in the process, to the required capacities.

The ABAP Nightly Build Tool that is currently being developed is to provide support to the takt based assembly process that is in pipeline. The tool automates the current assembly process that is being done by smaller function modules and maintained in the Electronic Checklist for the Assembly of Packages from SAP.

The project work comprises of testing the end-to-end functional testing of the ABAP Nightly Build tool. The work would comprise of analyzing the requirements and features, development of test cases, creating a manual test process, creating a work plan to complete the testing of the functionality developed in two takts of the project.

The project is being conducted to improve the quality of the tool, by the process of conducting testing in the agile mode. The testing in the first and second phase that is planned is expected to provide insights into the bugs that have penetrated into the tool, during the development process.

ii

ACKNOWLEDGEMENT
ACKNOWLEDGEMENT

I take this opportunity to express my sincere thanks to Ms. Ashwini Naik, SAP Labs India, for being my mentor throughout the course of this project work. I am grateful to Mr. Haresh Patel, SAP Labs India, for being the additional examiner for the project work and for all his support and help. My sincere thanks to Harish and Vignesh, my development colleagues; for their guidance and valuable inputs. Without the constant encouragement and support of my friends & family, I would not have seen the progress made in this dissertation work. They are the backbone behind every success of mine.

Bangalore

Deepak.K.N ID: 2009HZ74102

iii

TABLE OF CONTENTS
CERTIFICATE ACKNOWLEDGEMENT LIST OF FIGURES 1. Introduction to Assembly Process at ADD-ON Factory 2. Advent of Takt Based Assembly at SAP 2.1 Introduction 3. Testing Process 3.1 Iterative Testing Model 3.2 Tool Used 4. Test Planning and Requirement Analysis 4.1 Test Planning 4.2 Requirement Analysis 5. Test Case: Templates, Design and Mapping Requirements 5.1 Test Case Template 5.2 Test Case Design 5.3 Requirement Mapping 6. Testing Methods and Techniques 6.1 Risk Based Analysis 6.2 Testing Methodology 6.3 Testing Techniques 7. Test Execution 7.1 Smoke Testing 7.2 Functional Testing 7.3 Exploratory Testing 8. Test Report and Inferences 8.1 Quantitative Analysis 8.2 Qualitative Analysis 9. Conclusion and Recommendations 9.1 Conclusion 9.2 Recommendations Summary Directions of future work References Technical Terms and Abbreviations Checklist i ii iii 1 6 6 9 9 11 12 12 12 19 19 21 22 25 25 26 28 32 32 32 33 35 35 35 39 39 39 40 41 42 43 44

iv

LIST OF FIGURES
LIST OF FIGURES

Figure 1.1 2.1 2.2 2.3 3.1 3.2 4.1 4.2 5.1 5.2 5.3 6.1 6.2 8.1 8.2

Description Systems required for the Support Package creation. Planned Takt Based Assembly Process at SAP Tweak in the PIL Required. Other Projects for Takt Based Assembly Test Process Followed Iterative Life Cycle Model Pre-Build Requirements Build Requirements Test Case Template Snapshot Takt 1: Test Cases Snap Shot Takt 2: Test Cases Snap Shot Bug Report Sample Boundary Value Analysis Bug Report Graph Status of Bugs

CHAPTER 1 INTRODUCTION TO ASSEMBLY PROCESS AT ADDON-FACTORY

1. Introduction

Add-On Factory inside of SAP is a team responsible for providing production services for all the SAP Products, in form of Installation, Upgrade and Support Packages. The team follows a pre-defined process for the assembly of ABAP Packages. The brief overview defined in the chapter is in consideration with the Support Package scenario that is currently in use. The system pre-requisites required for creation of an ABAP Support Package are as follows: A CSS System, Development/Correction System, Testing System, Consolidation System and a Test System. Fig 1.1: Systems required for the Support Package creation.

The CSS is a system that logs in all the customer issues and acts as an interface between the customer and SAP. The Development/Correction System provides the site to make corrections to the existing product or to create new products. The Testing System is used for the testing to be done immediately after the product is built/ corrections being fixed. The Consolidation System comprises of the final corrections that have flown in and have to reach the customer. The Test System is taken with the previous level installed on it. The test system is used to check the technical correctness of a package. A Transport Directory is created at the operating system level. The directory is required to create a Transport Management System. This enables synchronizing customizing and development in multiple SAP systems through the transfer of changes from the development system to downstream systems.

The assembly processes are divided into 7 brief sections, namely: 1. Preparation. 2. Assembly. 3. Translation. 4. Checks. 5. Pre-Packages. 6. Export and Test. 7. Release and Post Processing.

Preparation Phase:
Preparation phases comprises of the following steps. Ordering of test systems for pre-package and for the final packages to be tested. Creating a system message on the DEV/CORR system to announce the arrival of End of Correction Close. Creating a Delivery and copying the attributes, according to which the corrections will be collected and assembled in the next steps.

Finally checking that the system parameters are correct to transport the corrections from CONS system to CSS system, to transport the package for registration.

Assembly Phase:
Assembly phase comprises of the following steps. Create a message to inform the developer that the export from the system will be cut off. Stop all the import jobs that bring in AOF tools, corrections and place an import lock on the system. Reserve the next support package, to divert the subsequent corrections to the next support package. Remove the Export Lock to allow the developer to make changes and continue with work. Search for requests without any attributes. Collect the requests from the previous collection date to the current collection date. Copy the list of objects in the Delivery to a translation list.

Translation Phase:
The translation phase comprises of translating the various GUI elements. Assembly team's activity is limited, where the team only provides the object list and waits for the translation completion.

Checks Phase:
The checks phase comprises of various checks that are conducted to improve the quality of the package. Comparison of the Data Dictionary State. The negative list table has to be updated.

The Object list has to be checked for objects that need to be considered for shipping. The list of objects is then generated. A complete syntax check is then done to avoid any show stoppers being shipped.

Pre-packages Phase:
Pre-packages are created for a few add-ons, which comprise of a functional validation. The pre-package steps are also repeated during the Export & Test phase.

Export & Test Phase:

The export phase comprises of the following steps: Checking the Transport parameters. Anonymizing the changes. Exporting of the support packages from the CONS system. Releasing the delivery unit from the CONS system. Registering the Support Package in CSS. Applying the package to the test system and checking for technical correctness.

Release and Post-Processing:

The release and post-processing comprises of the below mentioned steps: Saving logs, requesting back-ups and releasing the test machine. Delete old back-ups. Re-schedule import jobs. Recreate initial DDIC state information. Release Support Package in CSS. Send Release Mail. Check replication in Service Market Place. Re-schedule automatic imports into FA/CONS system. Transfer Object list into the DEV system and the Conflict Check system. Confirm Delivery to complete the process.

Copy delivery data from FA-system into PQP.

The above mentioned process is the brief list of activities that are carried out in the various systems to create a support package.

CHAPTER 2 ADVENT OF THE TAKT BASED ASSEMBLY PROCESS IN SAP

2. Problem Statement

2.1 Introduction:
Fig 2.1: Planned Takt Based Assembly Process at SAP (DRAFT)

'Necessity is the mother of All Inventions', as the saying goes. The necessity to reach the market quickly and with a quality product gave rise to the inception of the Agile Methodology. The development with the Agile Methodology requires a change in the Project Innovation Lifecycle. The Production phase has to start much earlier in order to provide the final package 24 hours after the close of development.

2.2 Tweak in PIL

Fig 2.2: Tweak in the PIL Required.

The takt-based assembly arises out of the need to provide usable software after every takt. The usable software after every takt, will enable the reduction of errors after every version; resulting in reduced time requirement in the end of the development cycle. The current assembly process carried out and described in the earlier chapter comprises of several steps that are very customer oriented and are tedious to be carried out after every takt. Since, the product of every takt does not reach the customer and the goal is to improve efficiency; the assembly process has to be tuned to suit the new requirement of the product development strategy.

Takt based assembly process has several other projects running in parallel to fulfill the pursuit for a 24 Hour Production at SAP. The other projects running simultaneously, they are:

Fig 2.3: Other Projects for Takt Based Assembly

CHAPTER 3 TESTING PROCESS

The complete end-to-end functional testing required a process to be followed in order to have a streamlined approach to the task. The test process has been sub-divided into the following sections.

Fig 3.1: Test Process Followed

3.1 TESTING MODEL: Iterative or Interactive Lifecycle Model

Fig 3.2: Iterative or Interactive Lifecycle Model

The iterative model is one where the system is built and re-tested iteratively in chunks. The grouping of the functions are according to the inter dependency and the dependency on the completion of the other modules. While the functions grouped within a single module are on the basis of high risk methods falling into place first. In the above process the functionality is covered incrementally, while reducing the efforts on the Business Modeling, Requirements and Analysis and Design in the latter stages. Test executions attain peaks at the second half of a takt. The deployable software available to the team is of a better value here; as the issue burn down at the early stages. The Advantages and Disadvantages of using the method and the actions taken: Advantages: 1. The issues that arise after a set of functionality are developed, are addressed at an earlier stage. 2. The tasks are broken down into smaller chunks and thereby improving the focus on the issues and improving the coherence of the issues. 3. Availability of the test systems at an earlier stage and thereby the improvements required from the system end can be put forth quickly. Disadvantages: 1. Regression Increase. Why the need? : The regression increases, but the issues coming to the surface increases, thereby increasing the quality of the tool provided in the end. 2. Plan to handle Bugs. Bug Tracking Methodology: The bugs being raised were available to the developer, because of the use of an easy access tool of Microsoft called Microsoft Office OneNote. The time provided to fix the bugs, were planned to avoid overlap of the subsequent developments.

10

3.2 Tool Used:

Microsoft Office One Note has been used to simplify the task. The purposes served or the efforts reduced are as follows: 1. The tool is available on the desk, at all times. The project is a small sized project with 2 developers and a tester. 2. The tool serves as an important communication tool between the team. 3. The shared notebook concept helps to easily access the work done immediately. 4. Bug tracking is easy, as the updates are all maintained at a single point and no extra times lost to login to a different machine and access the bug reports. 5. Easy maintenance of the test reports and tracking of the bugs solved.

11

CHAPTER 4 TEST PLANNING AND REQUIREMENT ANALYSIS

4.1 Test Planning:
The testing of the tool is planned in the following stages: i. ii. iii. iv. v. Requirement Analysis Test Case Writing Testing the Tool Reporting the Results Analysis of the Results

The testing has to be done twice, once after takt 1 and takt 2. The takt one is planned with the maximum number of requirements. The back logs and the dependent scenarios are to be developed and retested in the takt 2.

4.2 Requirement Analysis:

The requirements for the ABAP Nightly Build Project have been sub-divided into 4 sections: i. ii. iii. iv. Pre-Build Phase Build Phase Post-Build (Export) Phase Result Logs

4.2.1 Pre-Build Phase:

The pre-build phase comprises of the tasks that are performed to prepare the system for the Assembly of a Add-On Component. The requirements are as follows: (1) Delivery Existence: A delivery has to be created in the final assembly system. The Delivery is similar to a box that can be carried out as a package to the outside world. The existence of the delivery is a compulsory pre-requisite for the tool to go ahead with the next steps. (2) Pre-Configuration and Delivery Lock:

12

A pre-configuration is like an entry criteria, upon which the requests are collected and included in the package. The Delivery, when being used by the tool, should be locked for non-usage by others. (3) RFC Check Remote Function Call is an interface to all the other systems involved, to perform the required tasks. The requirement is for the active connections to be checked, in order that the Delivery once in run, is not halted.

Fig 4.1 Pre-Build Requirements

13

(4) T_OFF.SID: Export Lock: An export lock is set on the development or the correction system to lock the release of the new developments and close corrections. This is done to avoid changes after the Correction Close. (5) Import Job Check: Regular import jobs are scheduled to bring in the changes to the Consolidation system. These jobs should not be running, when the import lock is placed. So, a check has to be conducted to check that any running jobs have to be completed before placing the import locks. (6) Import Queue Check: This to check that the queue defined is empty and the lock can be placed. The released requests should be absorbed by the system. (7) NO_IMPORT.SID: Import Lock The import lock is placed to shut the doors for the changes after the requests released before correction close has been absorbed. This is done to open the development system for the developers to work on and to maintain consistency in the object state. (8) Updating Of Negative List A RFC connection is made to the central system to compare the negative lists of the consolidation and the central systems. After comparison, the consolidation system is updated if there are new objects added to the negative list. A negative list comprises of all the objects that should not be shipped to the customer. (9) Reset/Extend: The Reset or Extend is provided to handle the transports that are provided as a part of the process of correcting the errors that are encountered during the assembly process. The Reset option completely marks the selection new and the complete re -run of the checks will be necessary.

14

The Extend option, marks the current requests; along with it the new requests can be included. This retains the state of the changes maintained before the extend activity. (10) Installation/ Upgrade Package: A provision to create a installation package and upgrade package from the delivery created is the purpose of providing the option.

4.2.2 Build Phase:

The build phase comprises two major phases: i. ii. I. Collection Phase Checks Phase Collection Phase The collection phase comprises of collecting the requests released for new developments or corrections made. The Collection of the requests are made based on the following attributes. a. Pre-Configuration. b. Selection Logs. c. User Input. (11) Pre-Configuration The Pre-Configuration is the Delivery Channel that is established by the tool to select the requests from a set of clients and the designated add-on name. The channel makes it clear to accept the requests required. (12) Attribute: An attribute usually contain the ADD-ON name to which the request belongs to; this acts like the address to which the request belongs.

15

(13) Import Date: The release date is the time stamp that the request carries on release. The release date is the next criteria, which completes the selection process, for the requests. Fig 4.2: Build Requirements

Based on the above selection criteria the requests are collected and the requests that do not have an attribute; but released from the client mentioned in the Pre-Configuration are to be mentioned as not collected.

II Checks Phase:
The checks conducted here are: a. OLC Checks b. Baotagen c. Enhancement Anonymisation

16

(14)

OLC (Object List Checks): Is the set of checks conducted on the Object list prepared for the support package. The object list check is conducted to check various parameters, which was built with the AAK; will be re-used in the ABAP Nightly Build to check the correctness of the package. The ANB Tool limits to the interface function and the checks being processed. The ANB tool runs the OLC Check on the collected requests and the processed OLC result can be viewed from the interface. The tool functionality is limited to running the OLC and the display of the errors and the checks that have run fine. The checking of the objects and the processing of the changes still have to be completed from the AAK Delivery.

(15)

BAOTAGEN The generation of a program done in ABAP terms is the equivalent of the Compiling done in the C Program. The generation checks for the consistency of the program to avoid dumps and non-functional programs being shipped. The generation creates a log file of the same, which has to be appended or a new log file has to be created. The option has to be provided to create a new log file.

(16)

Enhancement Anonymisation Any development/ correction going out of SAP has to go in the name of SAP. Thereby, the program converts all the user names to SAP and the same is transported to the outside world.

4.2.3 Post Build (Export) Phase:

The post build phase comprises of exporting the package from the Consolidation System. The post build phase requires a set of parameters to make sure that the delivery is exported with the pre-defined standard. The set of export parameters required would be as follows. i. ii. iii. Export Languages List Object Versioning Yes/No Prototype Export Option - (17) - (18) - (19)

17

iv. v.

Delete Old Export Logs Option Export Inactive DDIC Objects Option

- (20) - (21)

The list of above parameters has to be catered to export the parameters. The set of parameters and the export functionality are already a part of the AAK Tool and the Export languages selection too have to be provided.

4.2.4. Result Logs

The results will have to be provided as the logs after the tool run. The tool has to accommodate the various requirements, for log sorting: i. ii. iii. The Results should be made available for all the tool runs. Filters should be available according to the Delivery. Filters should be available according to the Time Frame. (22) (23) (24)

The tool log should contain various definite points specified in the tool logs: i. ii. iii. Definite pointers to specific tasks performed for each requirement. (25) Clear demarcation of the various results at each stage. Clear definition of the outcomes in each step. (26) (27) (28)

Background Execution, is the default requirement of the tool

18

CHAPTER 5 TEST CASE: TEMPLATES, DESIGN AND REQUIREMENTS MAPPING

5.1 Test Case Template
The test case template is a standard template used across SAP. The template comprises of the following details: Fig 5.1:Test Case Template Screenshot

Components of the templates and description: i. Priority The priority for a test case is dependent on the risk levels attributed with the test case failure and the properties agreed upon to prioritize the same. The Priority levels The priority levels are as follows:

19

Priority Chart:
PRIORITY 1. Delivery Creation Stoppage 2. UI Issues / User Interface 3. Message/Informatio n 4. Cosmetic Issues BUGS RESOLVED BUGS

ii.

Description: Description is the short text and a brief explanation of the test case to follow, with a view of the determined outcome.

iii.

Preparation: The preparation phase involves the pre-configuration or the preparatory steps that needs to be followed, to fulfill the conditions required for the execution of the test case below.

iv.

Execution: It is the step by step approach that has to be followed to derive the desired outcome. Intermediate check points are added to make the test case more effective.

v.

Check: The check section involves the various checks that are to be performed and the final outcome to be derived from the test case executed.

vi.

Further Notes The section refers to the extra information related to the test case and the points that can be used as reference within the project.

20

5.2 Test Case Design

The test cases have been designed to cover the scenarios and modified to suit the tool UI provided. The test cases are such that they cover the above mentioned requirements and satisfy the completeness check of the functionality.

Fig 5.2: Takt 1:Test Cases Snap Shot

5.2.1: Test Case Naming Convention:

The test case names have been prefixed with the requirement id and the names are simple and self-descriptive to make it easy to track

21

Example: RQ:9_DELV_EXTEND The requirement id is 9 and the name signifies that the test case is created to extend the Delivery, after creation. Fig 5.3: Takt 2: Test Case Snap Shot

5.3 : Requirement Mapping:

Req No

Test Case Name

Brief Description Delivery Name: The test case is to check the Delivery Name; for existence, wrong delivery name non-acceptance, access check to the Delivery that is released. Checks for Delivery Lock via the first test case.

RQ:1,2_DELIVERY_NAME

RQ:1,2_DELIVERY_NAME Check if the button 'Display PreConfiguration' is performing the function, through the second test case. To have an RFC created and check before the start of actual assembly. The check is to see that the Export Lock is placed, during the assembly of the Delivery.

RQ:2,11,12,13_DELV_PRE-CONFIG 3 4 RQ:4_DELV_SET_EXPLOCK RQ:3_RFC_CHECK

22

RQ:4_NEGT_T_OFF_EXISTENCE_CHECK

The check is to see if the T_OFF placement is skipped in presence of a T_OFF; existing in the system. The log display of the data informing the presence of a T_OFF set by other process. To check that the ANB Tool run stops in presence of a running job and runs only if the import job completes, before tool logs out due to long running import jobs. To check that the button 'Display Import Jobs' performs the intended function. Ignore Import Buffer should be able to go ahead with the delivery, even in the presence of requests in the buffer. For any activity planned or existing in SPAM, the ANB tool should not run. Ignore Import Buffer should be able to go ahead with the delivery, even in the presence of requests in the buffer. The import lock should be placed before going ahead with the assembly. Import Lock existing. To modify the TRDELVNEGL Table with the availability of the SP9 RFC and skip the step in the absence of an RFC. Extending a Delivery, after the initial run; by keeping the earlier requested intact. Extending a Delivery, after the initial run; by keeping the earlier requested intact. To Test the AOI and AOX fields for text field functionality. To check if the button 'Display PreConfiguration' is performing the function. The Collection Evaluation of the Date fields 'Collection From' and 'Collection To'. Selection Logs data Check for the message of 'Selection Logs' non existence with the new delivery.

RQ:5_DELV_IMPJOB_CHECK RQ:5_BTTN_DISPL_IMPORT_JOBS

RQ:6_DELV_IGNORE_BUFFER 6 RQ:6_SPAM_CHECK

RQ:6_DELV_IGNORE_BUFFER_UNCHECK 7 RQ:7_DELV_IMPORT_LOCK RQ:7_NEGT_NO_IMPORT_EX_CHECK

RQ:8_TRDELVNEGL_UPDATE RQ:9_DELV_EXTEND

RQ:9_DELV_RECREATE RQ:10_DELV_BG,GG_PACKAGE CREATION RQ:10_NEGT_AOI_AOX_FIELDS

10

11,12

RQ:2,11,12,_DELV_PRE-CONFIG RQ:13_DELV_DATE-FIELDS RQ:13_BTTN_SELECTION_LOGS RQ:13_NEGT_SEL_LOG_NONEX

13

23

14 14,15

RQ:14_BTTN_OLC_CHECKS

To check that the OLC is not processed for a new delivery and is shown processed in the second instance. To check if the OLC, Export and Baotagen has been completed.

RQ:14,15,4.2.3_DELV_OO_OLC_BAO_EXP RQ:14,15_DELV_OO_OLC_BAO

Check for the OLC, Baotagen Option To check a New Log is created for a BAOTAGEN run, that happens for a Delivery. To check a New Log is created for a BAOTAGEN run, that happens for a Delivery. To check that the objects are anonymised before shipment Export Languages Selection from AAK and ANB Export a Delivery with the object versions. Export a Delivery with the object versions. Exporting the package with all the options of Pre-package, Inactive DDIC and Deletion of Old Logs.

15

RQ:15_DELV_BAOTAGEN_LOG_APPEND

RQ:15_DELV_BAOTAGEN_LOG_NEW 16 17 18 19 RQ:16_ENHANCEMENT_ANON RQ:17_BTTN_EXPORT_LANGUAGES RQ:18_DELV_EXP_OBJVER RQ:18_DELV_WO_OBJ_VER

20 Exporting the package without all the RQ:19,20,21_DELV_PREX_DELLOGS_EXPINACTD options of Pre-package, Inactive DDIC and RQ:19,20,21_DELV_NONE_OF_PRO-OLD-DDIC Deletion of Old Logs. The tool history button, the filters and the functioning of the filters will be validated. Checked via User Perspective Testing: No specific Test Case To check the functioning of the button and save of a test variant. Running a Delivery in Background for multiple selection Running a Delivery in Background for multiple selection

21 22 23 24 25 26 27

RQ:22_BTTN_TOOL_HISTORY

RQ:25,26,27_READ LOGS_USER_PERSPECTIVE RQ:28_BTTN_VARIANT

28

RQ:28_DELV_PERIODIC-JOBS RQ:28_DELV_BACKGRD_EXECUTION

24

CHAPTER 6 TESTING METHODS AND TECHNIQUES

A set of core testing concepts and test methodologies have been utilized in designing the test cases, they are as follows:

6.1 Risk Based Analysis

The successful testing of a set of requirements depends upon the analysis carried forward before the actual testing. The risk analysis involves two parts: i. ii. Risk of the Testing Failure Risk of Schedule Failure

6.1.2 Risk of Testing Failure:

The testing process can fail if the discussions regarding the requirements are not carried out initially to freeze the requirements. In case of the project carried out, the requirements were frozen at the start of development. Each takt requirement was taken ahead with the priority of the requirements being independent. Example: a. Test requirements were provided well before the scheduled start. b. Test Cases deadlines were fixed and finalized. c. Test Cases were mapped to cover the defined requirements.

6.1.3 Risk of Schedule Failure:

The schedules and the timeline definition have to be clearly defined with the buffer provided and the tasks planned. The best of the test suite can fail, when the test planning is not taken care. Example: The Extend/Recreate functionality, OLC Checks Selection and Periodic Job Date Selection were a set of functionality that were to be shelved with takt 1, due to dependency carried by the Program extension from the existent tool and tasks planned for takt 1.

25

Sl.No 1

Tasks Planned Requirement Gathering and Discussions TAKT 1 Development Requirement Analysis Test Case Writing Test Suite Execution TAKT 2 Development Test Suite Execution Reporting

Scheduled Timelines 20-Dec to 8-Jan

Extendable Up to 12-Jan

2 3 4 5

8-Jan to 31-Jan 8-Jan to 31-Jan 5-Feb to 15-Feb

5-Feb 5-Feb 18-Feb

6 7 8

18-Feb to 25-Feb 28-Feb to 7-Mar

28-Feb 10-Mar

6.2 Testing Methodologies.

To test the tool for functional correctness a series of tests have been conducted to validate the scenarios. The testing conducted can be broadly defined under four categories: 1. Defined Test Suite 2. Exploratory Testing 3. User Perspective Testing 4. Confirmatory Testing

6.2.1. Defined Test Suite:

The test suite has been designed based on the test scenarios that has been derived out of the requirements provided. The test suite and the list of test cases have been defined in the section 5.2.2, pertaining to mapping of the requirements. The test suite has been executed completely twice in the project period, which has helped in providing conformity to the scenarios developed and also provides useful insights into the bugs that have identified.

26

6.2.2. Exploratory Testing:

Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Exploratory testing, more of a free style testing. Tests here are with the responsibility to increase the software quality and derive out-of-box tests. Exploratory test cases have helped quality. The test suite tests mostly for the intended functionality to be satisfied. In case of exploratory tests, regular test cases are not defined and a random testing is carried out on the UI. Two bugs that was raised in the process for date validation and Collection From message reappearing are examples of the effectiveness.

6.2.3 User Perspective Testing:

The UI that had to be designed for the tool, had to be user friendly and the logs had to be very readable. The requirements for the logs to have clear cut demarcations and well understandable logs were tested by giving valid inputs to the development team to change the log messages. These were raised as bugs and the bugs were fixed in the process to make the logs more readable.

27

6.2.3 Confirmatory Testing

The confirmatory testing is done on the scenarios for which bugs were raised. The bugs were raised in the test maintenance tool Microsoft Office One Note. The bugs were re-tested once the developers had fixed the same. Fig 6.1: Bug Report Sample

6.3 Testing Techniques: 6.3.1 Equivalence Partitioning:

Equivalence partitioning (also called Equivalence Class Partitioning or ECP[1]) is a software testing technique that divides the input data of a software unit into partitions of data from which test cases can be derived. In principle, test cases are designed to cover each partition at least once.

28

The application is more so for the UI fields. The sample field:

Here 3 possible scenarios: 1. Valid Delivery 2. Invalid Delivery 3. Valid Delivery, which is confirmed.

6.3.2 Boundary Value Analysis:

Boundary value analysis is a software testing technique in which tests are designed to include representatives of boundary values. Values on the edge of an equivalence partition or at the smallest value on either side of an edge. The values could be either input or output ranges of a software component. Since these boundaries are common locations for errors that result in software faults they are frequently exercised in test cases. Boundary value Analysis is an important tool, in deciding the number of inputs required to confirm the completeness of the test. For the date field for example.

29

The value applied along the boundary values are as follows: Fig 6.2: Boundary Value Analysis

Here at an earlier instance, the date box was taking special characters, which was checked and corrected.

6.3.3 Decision Tables:

Decision tables are a precise yet compact way to model complicated logic. Decision tables, like flowcharts and if-then-else and switch-case statements, associate conditions with actions to perform, but in many cases do so in a more elegant way. In the project decision Tables logic has been applied to several instances. An example of one such instance is as below:

30

Export Options Actions Rules

Export with Object Versioning Export without Object Versioning Prototype Export Only Delete Old Export Logs Export Inactive DDIC Objects Y N N

Y N N Y N

N Y N N Y

In the export Options provided above the object versioning and non versioning derives two test cases. The Prototype Export, Delete Old Export Logs and Export Inactive DDIC Objects are all mutually exclusive scenarios and thereby only two test cases are derived one for selection of all the options and the other for non selection of all. The decision to collapse all tables has been taken as the conditions can co-exist due to the mutually exclusive conditions.

The above techniques are extensively utilized in the testing process of the project.

31

CHAPTER 7 TEST EXECUTION

The test execution was carried out two times, once after each takt. The test execution process involved various levels of testing. The test execution phases: i. ii. iii. Smoke Testing / Random Testing Functional Testing Exploratory Testing

7.1 Smoke Testing

The smoke testing is carried out checking the action points and the main decisive points in the software, where the dumps can occur and the testing can stop. Such critical show stoppers are checked before starting with the actual execution of the functionality expected. In the project, smoke tests were conducted on a random basis without pre-defined test cases, but based on the understanding of the requirements. Advantages: i. ii. iii. Show stoppers can be identified. Major issues can be addressed before starting the activity. Time is saved on shelving issues, that needs to be addressed and testing can be carried out on the rest of the functionality that is working fine. Issues identified due to Random Testing/ Smoke Testing: Dumps that occurred twice during the initial smoke tests were helpful in informing the development about the tool functional failure and saving of time, in evaluation phase.

7.2 Functional Testing

The functional testing involves the execution of the test suite to address the required functionality. The functional testing involved in the testing process was designed to cover the requirements that were listed as per the design requirement.

32

The testing phase was scheduled for 6 working days, for the scheduled requirement. The functional testing has been carried out after each takt and to do a confirmation of the bugs during that was raised during the testing phase. Benefits of Functional Testing: Application will work as per specifications across multiple platforms, browsers and technologies Tests are done from a user's point of view Tester needs no knowledge of implementation, including specific programming languages Tester and programmer are independent from each other It will help to expose any ambiguities or inconsistencies in the specification Test cases can be designed as soon as the specification is completed.

The test results are presented as a part of the test reporting.

7.3 Exploratory Testing:

Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. The testing is a free style testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of work by treating test related learning, test design, test execution and test interpretation as mutually supportive activities that run in parallel throughout the project.

Benefits of Exploratory Testing: Less preparation is needed, important bugs are found quickly, and at execution time. The approach is more intellectually stimulating than execution of scripted tests. Testers can use deductive reasoning based on the results of previous results to guide their future testing on the fly Does not require completion of a series of scripted tests before focusing on moving on to exploring the application. Thus accelerating the detection of bugs in the process. Disadvantages of Exploratory Testing:

33

Tests are created and executed on the fly, thereby the review cannot be done and developers cant avoid errors by predicting the scenarios. When revisited are difficult to be executed the same way by the same tester and even so by a new resource.

Test results are presented as a part of the test reporting in the next chapter.

34

CHAPTER 8 TEST REPORTING

Test Reporting is a part of Quality Risk evaluation. The Quality Risks are channelized into two major sub-divisions. They i. ii. Quantitative Analysis Qualitative Analysis

8.1 Quantitative Analysis

The quantitative analysis has been done on the total number of bugs reported. The quantitative charts and graphs are as follows:

Bug No 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Priority 2 4 4 1 2 4 1 1 1 1 2 1 3 1

Req No RQ10 EXP EXP RQ1,2 RQ13 EXP SMO RQ1,2 RQ1,2 RQ1,2 RQ13 SMO EXP RQ9

Status Solved Not Valid Not Valid Solved Solved Not Valid Solved Solved Solved Solved Solved Solved Solved Solved Accepted as limitation Solved Solved Solved Accepted as limitation Solved Solved Solved

Bug No 23 24 25 26 27 28 29 30 31 32 33 34 35 36

Priority 3 3 3 3 3 3 3 3 2 3 1 4 3 4

Req No UI RQ22,23,24 RQ22,23,24 RQ28 RQ17 RQ22,23,24 RQ22,23,24 RQ13 RQ25,26,27 RQ25,26,27 RQ22,23,24 RQ1 RQ25,26,27 RQ25,26,27

Status Solved Not Valid Solved Solved Solved Solved Open Open Solved Solved Solved Solved Open Solved

15 16 17 18

3 4 3 4

RQ22,23,24 RQ22,23,24 RQ25,26,27 RQ25,26,27

37 38 39 40

4 4 4 4

RQ25,26,27 RQ25,26,27 RQ25,26,27 RQ25,26,27

Solved Solved Open Open

19 20 21 22

1 1 1 3

RQ1,2 RQ1,2 EXP RQ13

41 42 43

4 4 4

RQ25,26,27 Solved RQ25,26,27 Solved RQ25,26,27 Solved

35

8.1. Bugs Report:

Fig 8.1 Bug Report Graph 12 10 8 Prio 1 6 4 2 0 Solved Open Not Valid Acc Lim Prio 2 Prio 3 Prio 4

Priority No 1 2 3 4

Bug Report No of Bugs Solved Open Not Valid 10 0 4 0 9 3 9 2 Total No Of Bugs

Acc Lim 0 0 1 3 1 0 1 0

Total 11 4 14 14 43

The total number of bugs reported was 43. At the end of takt 2, open bugs have been reduced to 5; with 4 bugs termed as not valid. The success of the tests has been termed as 89%. The open bugs will be addressed in the takt 3 and further developments planned.

36

8.1.2 Status of Bugs:

Fig 8.2 Status of Bugs 16 14 12 10 8 6 4 2 0 Total Solved Open Prio 1 Prio 2 Prio 3 Prio 4

Priority 1 2 3 4

Status of Bugs Total Solved Open 11 10 0 4 4 0 14 9 3 14 9 2

The rated success percentage of the test success has been calculated based providing a rating of 4 for bugs with priority 1 and 1 for bugs with priority 4. The total valuation has reached 93 and the bugs valuated to 8. The total rated success percentage of testing has been calculated at 92%.

37

8.2 Qualitative Analysis:

A process carried out to rate parameters that are difficult to be measured under a unit and are rated based on the actual delivery received to the expected outcome from the requirements. A few parameters applicable to the tool developed and tested in the project have been Rated in the list below:

Sl.No

Parameter

Rating (Out of Ten)

Comments Some inherent limitations in the development process have been accepted to satisfy the functional requirements. There are several pop-ups that will be merged to form a single list in the next development takt. This will increase the performance of the tool. The functional requirements have been satisfied with priority 1 and 2 bugs solved. The tool is robust with two major screen and 3 sub screens. The rating has been reduced due to the number of pop-ups, which are planned to be minimized. The tool goes as a part of the BAOF package and is supplied to other consolidation systems with ease. The documentation of the user guide is complete. The priority 4 bugs open, reduces the rating by a point. The open bug in priority 3 category reduces the usability factor.

Functionality

Performance

Reliability

10

Robustness

Installation

10

6 7

Documentation Usability

9 9

38

CHAPTER 9 CONCLUSION AND RECOMMENDATIONS

9.1 Conclusion:
The project has driven a test driven development project for AOF, with the following objectives delivered: i. ii. iii. iv. Testing done on time and meeting deadlines. Test Cases have created a clear documentation of the test scenarios for future references and execution. Test Execution has increased the quality by detecting 43 bugs and the test success percentage of 89% and rated success of the test suite at 92%. Reducing the percolation of issues and also improving the tool quality.

9.1 Recommendations:
The test suite created and executed for the project has provided efficient results. The quality analysis has displayed the increase in quality, by the testing done in the two takts. The testing methodology adapted for the project was done for the first time within AOF. The process and the test suite will be carried forward in the upcoming takts for the project.

39

Summary
The Project Requirement Analysis and End to End Testing Of ABAP Nightly Build Tool highlighted the following salient features: i. ii. iii. iv. v. First time introduction of a formal testing methodology for an internal tool. Testing during the Takts has helped reducing the bugs from percolating. Formal Documentation to the Test Suite and Bugs Reported. A test suite execution and reporting to analyze the tool efficiency and the success of the testing done. A total improvement in quality of the tool.

The project has been a fruitful experience in the form of learning derived and the outcome and benefits to the organization.

40

Directions of Future Work

A plan is in place to extend the tool to do a CLC package for HR. The tool will again be developed in takts and the requirements are as per the CLC package requirements and the requirements that was decided earlier for the takt 3. The goals will be to: i. ii. iii. Enhance the test suite to map to the new requirements and retain the old ones. Create awareness on the tool to the new users. Enhance Quality of the test suite and Quality of the Tool.

41

References
Web References:
Search Software Quality http://searchsoftwarequality.techtarget.com/ Wikipedia http://en.wikipedia.org/wiki/Wikipedia

Texts:
Advanced Software Testing Rex Black Vol 1 Advanced Software Testing Rex Black Vol 2

42

Technical Terms and Definitions

Technical Terms Term Meaning

CSS Takt

A customer system, which is used to log the issue, receive corrections, provide support packages and other related activites A pre-defined period of time in which a set of requirements are planned for development and executed. A lock on the development system to stop the release of correction during the assembly phase. This is done by placing T_OFF. <sid> in the operating system level. A lock on the consolidation system to stop the import of corrections after a predefined correction close, to maintain consistency of the system. This is done by suspending import jobs and also placing the NO_IMPORT.<sid> at the operating system level.

Export Lock

Import Lock

Abbreviations: Abbreviation PIL RFC Product Innovation Life Cycle Remote Function Call

43

Checklist of items for the Final Dissertation Report

This checklist is to be attached as the last page of the report.
This checklist is to be duly completed, verified and signed by the student. 1. 2. 3. 4. 5. 6. 7. 8. 9. Is the final report properly hard bound? (Spiral bound or Soft bound or Perfect bound reports are not acceptable.) Is the Cover page in proper format as given in Annexure A? Is the Title page (Inner cover page) in proper format? (a) Is the Certificate from the Supervisor in proper format? (b) Has it been signed by the Supervisor? Is the Abstract included in the report properly written within one page? Have the technical keywords been specified properly? Is the title of your report appropriate? The title should be adequately descriptive, precise and must reflect scope of the actual work done. Have you included the List of abbreviations / Acronyms? Uncommon abbreviations / Acronyms should not be used in the title. Does the Report contain a summary of the literature survey? Does the Table of Contents include page numbers? (i). Are the Pages numbered properly? (Ch. 1 should start on Page # 1) (ii). Are the Figures numbered properly? (Figure Numbers and Figure Titles should be at the bottom of the figures) (iii). Are the Tables numbered properly? (Table Numbers and Table Titles should be at the top of the tables) (iv). Are the Captions for the Figures and Tables proper? (v). Are the Appendices numbered properly? Are their titles appropriate Is the conclusion of the Report based on discussion of the work? Are References or Bibliography given at the end of the Report? Have the References been cited properly inside the text of the Report? Is the citation of References in proper format? Is the report format and content according to the guidelines? The report should not be a mere printout of a Power Point Presentation, or a user manual. Source code of software need not be included in the report. Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

10. 11.

12.

Declaration by Student: I certify that I have properly verified all the items in this checklist and ensure that the report is in proper format as specified in the course handout.

Students Signature Place: Bangalore Date: 25-03-2011 Name: Deepak.K.N ID No: 2009HZ74102

44