SM-514 INTRODUCTION TO SOFTWARE TESTING

ASSIGNMENT 1 - RequirementsBased Test Generation

1 INTRODUCTION 1.1 OBJECTIVES
There are several key objectives of this assignment. The first is to introduce students to the fundamentals of unit testing, specifically unit testing based on requirements for each unit. The most widely used unit testing tool for Java is the JUnit framework. Students should have a basic understanding of how to use JUnit upon completing the assignment.

1.2 THIS ASSIGNMENT IS A GROUP WORK

All the tasks of this assignment should be completed in groups of two. The report will also be completed as a group. If you dont want to be in a group, you can do so, but you have to complete all the steps by yourself.

1.3 DUE DATE AND LATE MARKING POLICY

The due date of the assignment will be announce on METUONLINE. Please check the METUONLINE for details. You need to upload the assignment reports and the test suite codes to METUONLINE. Late reports: There will be no credit for reports turned after the due date; they will be returned unmarked.

1.4 OVERVIEW
This assignment can be divided into three main sections. The first section is familiarization. During the familiarization stage, students will be shown how to set up a JUnit test project in Eclipse using JUnit, create a unit test, and how to navigate Javadocs. After familiarization, the second section of the assignment is unit test generation based on the requirements specified in Javadocs. In this stage, students will develop unit test suites for several classes. These test suites will be comprised of unit tests which have been generated according to the requirements. Finally, upon completion of the test suites, during the last stage, the tests will be executed on several versions of the SUT and test results will be collected.

1.5 TESTING TOOLS

The main testing tool for this assignment is JUnit [4]. JUnit is a popular free unit testing tool and framework for Java. Created by Kent Beck and Erich Gamma, JUnit is one of the xUnit family of testing frameworks. Its main attraction is its seamless integration with the Eclipse development environment [1]. The JUnit framework allows developers to quickly and easily create unit tests and test suites. For more information on JUnit, see http://www.junit.org. Another tool which is being used for the purpose of testing is Javadoc [2]. While Javadoc is not exclusively or explicitly a testing tool, in the context of this assignment it will be used as the format to store requirements specification to use to derive test suites. Javadoc allows developers to create Application Programming Interface (API) documentation within source code together with code itself. Creating the documentation and the code in the same location does not only improve communication between developers, maintainers and testers, but it also makes it simpler to keep the documentation up to date, and prevent potential redundancies and/or mistakes. For more information on Javadoc, see http://java.sun.com/j2se/javadoc.

1.6 SYSTEM UNDER TEST

The system to be tested for this assignment is JFreeChart [3]. JFreeChart is an open source Java framework for chart calculation, creation and display. This framework supports many different (graphical) chart types, including pie charts, bar charts, line charts, histograms, and several other chart types. To get started with the JFreeChart system, download the JFreeChart v1.0-correct version.zip file from METUONLINE and extract the entire archive to a known location. More information on how to get started with these files will be provided in the familiarization stage (Section 2.1). Note that the versions of JFreeChart distributed for this assignment do not correspond with actual releases of JFreeChart. The versions have been modified for the purposes of this assignment.

1.6.1 Purpose of the System The JFreeChart framework is intended to be integrated into other systems as a quick and simple way to add charting functionality to other Java applications. With this in mind, the API for JFreeChart is required to be relatively simple to understand, as it is intended to be used by many developers as an open source off-the-shelf framework. A snapshot of four different types of charts drawn using JFreeChart is shown in Figure 1.

Figure 1 - A snapshot of four different types of charts drawn using JFreeChart. 1.6.2 Usage of the System While the JFreeChart system is not technically a stand-alone application, the developers of JFreeChart have created several demo classes which can be executed to show some of the capabilities of the system. These demo classes have Demo appended to the class name. For the purpose of this assignment, full knowledge of the usage of the JFreeChart API is not particularly necessary. The framework is grouped into two main packages, (1) org.jfree.chart and (2) org.jfree.data. Each of these two packages is also divided into several other smaller packages. For the purpose of testing in this assignment, we will be focusing on the org.jfree.data package.

2 INSTRUCTIONS
This section details the instructions for executing the assignment. All sections of this assignment should be completed AS A GROUP as specified.

2.1 FAMILIARIZATION
Both students of each group should perform this section of the assignment together on a single computer. Ensure that both of you understand the concepts in this section before moving on to the rest of the assignment. 1. If you havent done so already, download the JFreeChart v1.0-correct version.zip file from METUONLINE. 2. Extract the contents of the .zip file into a known location.

2.1.1 Create an Eclipse Project 3. Open Eclipse (version 3.3). 4. Open the New Project dialog by selecting the File -> New -> Project 5. Ensure that Java Project is selected and click Next. 6. The dialog should now be prompting for the project name. Enter JFreeChart in the Project Name field, and then click Next. 7. The Java Settings dialog should now be displayed. This dialog has four tabs along the top: Source, Projects, Libraries, and Order and Export. Move to the Libraries tab, and click the Add External JARs (or Libraries) button. 8. Select the jfreechart.jar file from the known location and click Open. Click Add External Libraries again, this time add all the .jar files from the lib directory where you have unzipped the JfreeChart v1.0-correct version.zip file. The Java Settings dialog should now look like Figure 2, below.

Figure 2 - The Java Settings dialog after adding required archives 9. Click Finish. The project (SUT) is now set up and ready for testing. To run the demo classes, in the package explorer expand the Referenced Libraries item in the newly created JFreeChart project, exposing the .jar files just added. Right click on the jfreechart.jar, and select Run As -> Java Application (Figure 3).

Figure 3 - Running JfreeChart 10. In the Select Java Application dialog, select any of the four demo applications (e.g., CompositeDemo), and click OK as shown in Figure 4.

Figure 4 - The Select Java Application dialog 2.1.2 Create a Simple JUnit Test To create a test suite containing a single unit test in JUnit, follow these steps. 11. In the package explorer, expand the Referenced Libraries list item to show all the archives that the project uses. 12. Expand the jfreechart.jar archive to expose all the packages that are contained in that archive. 13. Expand the org.jfree.data package within the archive to show all the .class files contained in that package. 14. Finally, expand the Range.class item to expose the class contained in that file, along with all the class methods and fields contained in that class. 15. Right click on the Range class (has a green C icon, denoting it is a class), and select New -> Junit Test Case. Type RangeTest in the Name field and type .test in the Package field at the end of the default package to create a new package for the test cases. Ensuring that test classes are in a separate package makes it easier to keep the two apart. 16. Click Finish.

Figure 5 - Creating a new JUnit test case 17. The newly created test class (RangeTest) extends TestCase, the super class of all test cases as defined in JUnit. The TestCase class is a JUnit class which provides a unified interface for test cases and suites. A set of test cases which test similar functions of a class may all want to initialize an instance of that class and prepare for the test cases in the same way. The way that Junit facilitates this is the setUp()and tearDown() methods. Write the setUp() and tearDown()methods for the RangeTest class as shown in Figure 6 below, including the testRange attribute which is required for the test cases to have access to the object initialized in the setUp() method.

Figure 6 - Simple setUp() and tearDown() 18. Test cases in JUnit are individual methods which are usually prefixed with the word test, for example: testCentralValue(). These test cases can perform any number of steps, and should follow four phase testing (setup, exercise, verify, and teardown). The exercising of the code can be seen as the input to any particular test. The test oracle for each test case in JUnit can be implemented in several ways, using assertions being the most popular and recommended one. An example assertion is: assertTrue((example string).length() == 14); which will always pass (assuming the length() method is correct). The syntax of the above assertion as defined in JUnit is: public static void assertTrue([java.lang.String message,] boolean condition) which asserts that a condition is true. If it isn't, it throws an AssertionFailedError with the given message (if any). Assertions are tested when a test method is executed, and will cause a test to fail if the assert conditions are not met. For a complete list of all the assertions available using JUnit, refer to Appendix A. 19. As a practice, write a simple test case for the getCentralValue() method as shown in Figure 7 below. The syntax for assertEquals is as follows: public static void assertEquals(java.lang.String message, double expected, double actual, double delta) assertEquals asserts that two values are equal. This particular version of the method also has a delta value, to allow for some variance when comparing floats (since it can be difficult to tell if a value is going to be 1.500000001 or exactly 1.5 in the present of inevitable automatic rounding of floating point numbers in Java). If the values are not equal, an AssertionFailedError is thrown with the given message. The expected argument is the test oracle which we should derive from the requirements.

Figure 7 The RangeTest test class after addition of a simple test case 20. Now that you have a completed test case, run the test class. To do this, right click on the RangeTest class in the Package Explorer and select Run As -> JUnit Test. 21. This will change the perspective to the JUnit perspective, and run all the tests within the RangeTest class. The test just written should pass, indicated by the JUnit view as shown in Figure 4 below. In JUnit a Failure and Error are similar, but differ slightly. If you get an error it means that your test method did not execute as expected (ie: an uncaught exception), a failure, however, means that the execution was as expected, but an assertion failed.

Figure 8 - JUnit view showing passed test 2.1.3 Navigate Javadoc API Specifications The test generation section of this assignment (Section 2.2) will require you to generate unit tests for a number of classes based on specifications (requirements) contained in the Javadocs for those classes. If youre already familiar with Javadoc, feel free to skip this section and continue from Section 2.2. 22. Unzip the JFreeChartModifiedJavadoc.zip file and open the file index.html. This is the Javadoc for (a slightly modified version of) JFreeChart. Note the location of different elements in the Javadoc as shown in Appendix B. Note that Javadocs can be browsed with all classes shown, or with classes filtered by package. Each of these two approaches has its usefulness. Viewing all classes is useful if you know what the class you are looking for is called, as they are ordered alphabetically. Viewing classes in a single package only is useful for when youre not sure exactly what class youre looking for, but know what area of the code it might be found in. 23. In the list of packages (top-left corner), scroll down to find the org.jfree.chart.axis package and click on it. This should show only a few classes in the class list (bottom left corner) now. 24. In the class list, click on the ColorBar. The main content pane now shows the API specifications of the ColorBar. Scroll down and notice the layout of the specification. At the very top is a description of the class itself (including inheritance information), followed by nested classes, attributes, methods (starting with any constructors), inherited methods, and finally the detailed specification for each method. 25. Take note of the information available in the Method Summary and Method Detail sections of the main content pane, as this is what you will be testing methods against (as test oracle) in the following section. For especially effective tests, however, the specifications need to be specific, precise, clear and complete.

2.2 TEST SUITE GENERATION

This section is recommended to be performed as a group, however the work may be divided and completed individually, or you may wish to employ peer programming to help ensure all requirements are tested.

26. In this section, you will be required to create unit tests for several classes, testing them against their specifications. The three classes to be tested are org.jfree.data.DataUtilities (in the org.jfree.data package): Has 5 methods. org.jfree.data.Range (in the org.jfree.data package): Has 15 methods. org.jfree.data.io.CSV (in the org.jfree.data.io package): Has 1 method. Take a minute to browse the API specifications of each of these classes in Javadoc. Note that some of these methods take objects which implement specific interfaces. In order to test these classes you will need to instantiate a concrete class which implements these interfaces. For example, the org.jfree.data.DataUtilities.calculateRowTotal() method takes as a parameter an object which implements the Values2D interface. To test this, you can either create a class that implements this interface in a simple manner, or you may want to instantiate an object of type DefaultKeyedValues2D instead. 27. You are going to submit a report describing who created which test, how did you ensure requirements are adequately tested --i.e.which test case generation techniques you have chosen. This report will also include the defects that you have found if any. The sections of the report is as follows..

1 INTRODUCTION Text... 2 DETAILED DESCRIPTION OF UNIT TEST STRATEGY Text... which techniques have you used to generate test cases? 3 TEST CASES DEVELOPED A high level description of each test case 4 HOW THE TEAM WORK/EFFORT WAS DIVIDED AND MANAGED Text... 5.DEFECTS Text... defects found if any 6 DIFFICULTIES ENCOUNTERED, CHALLENGES OVERCOME, AND LESSONS LEARNED Text... 7 COMMENTS/FEEDBACK ON THE assignment ITSELF Text...
28. Carry out your test plan. To keep your workload manageable, we would like you to create test cases for all 5 methods of org.jfree.data.DataUtilities and the 1 method of org.jfree.data.io.CSV. For org.jfree.data.Range, choose 5 out of 15 methods and create tests for them. Try to keep each test case in a separate method. For example: testPositiveValuesForMethodX() and testNegativeValuesForMethodX(), instead of a single testMethodX(). This will help to keep test cases consistent, and make analysis of test case impact simpler later on. For writing your test methods, please use the example discussed earlier in Section 2.1.2. 29. If you have divided the tests and completed them individually, then upon completion of the tests, review each others tests, looking for any inconsistencies or defects in the tests themselves. 30. Execute the test suite you have created on JFreeChart v1.0.zip, and report any failed tests. Note that defects should be reported and handled for the appropriate version (v 1.0). Note that the classes have random defects in them intentionally, and thus several of your tests should fail. Therefore, to write your test methods, you need to follow the specifications, not the actual results. 31. If defects were reported individually, peer review each others defect reports.

3 SUMMARY
Upon completion of this assignment, students should have a reasonable understanding of unit testing based on unit requirements using the JUnit framework. Note that unit testing and JUnit are very comprehensive and it takes quite a lot of time to be an expert in them. So do not expect to be JUnit experts just by completing this assignment. If you would like to have a career path in this industry-hot topic, you will need to study this popular framework in more detail and perform more exercises to be skillful.

The unit testing knowledge you gained in this assignment can be scaled up to much larger systems, and can be very useful in industry.

4 DELIVERABLES AND GRADING 4.1 JUNIT TEST SUITE (60%)

The test suite will be required to be submitted along with the assignment report. Students will be graded on their unit tests. Your unit tests will be evaluated according to correctness, completess and clarity criterias and the found defects.

4.2 ASSIGNMENT REPORT (40%)

Students will be required to submit a 5 to 10 page report on their work in the assignment as a group. To be consistent, please use the template Word file Assignment 1 Report Template.doc provided online. If desired, feel free to rename the sections, as long as the headings are still descriptive and accurate. A portion of the grade for the assignment report will be allocated to organization and clarity.

5 ACKNOWLEDGEMENTS
We would like to thank Vahid Garousi and his lab members for preparing and sharing of these tutorials.

6 REFERENCES
[1] "Eclipse.org," Internet: http://www.eclipse.org [July 3, 2008] [2] "Javadoc," Internet: http://java.sun.com/j2se/javadoc/index.jsp [July 3, 2008] [3] "JFreeChart," Internet: http://www.jfree.org/jfreechart [July 3, 2008] [4] "JUnit," Internet: http://www.junit.org [July 3, 2008]

7 APPENDIX A ASSERTIONS AVAILABLE IN JUNIT

The following is a list of assertions available for use in JUnit test cases. Note that assertions with message arguments are equivalent to those without the message argument, but will display that message when the test fails on that assertion. These can be useful for debugging purposes. For more information see http://junit.sourceforge.net/javadoc/org/junit/Assert.html assertEquals(expected, actual) assertEquals(message, expected, actual) assertEquals(expected, actual, delta) - used on doubles or floats, where delta is the allowable difference in precision assertEquals(message, expected, actual, delta) - used on doubles or floats, where delta is the allowable difference in precision assertFalse(condition) assertFalse(message, condition) assertNotNull(object) assertNotNull(message, object) assertNotSame(expected, actual) assertNotSame(message, expected, actual) assertNull(object) assertNull(message, object) assertSame(expected, actual) assertSame(message, expected, actual) assertTrue(condition) assertTrue(message, condition) fail() - Fails a test with no message. fail(message) - Fails a test with the given message. failNotEquals(message, expected, actual) failNotSame(message, expected, actual) failSame(message)

8 APPENDIX B JAVADOC EXAMPLE