Automated Testing Environment

by Bugzilla, Testopia and Jenkins

Chin Pin Chang
04/07/2014

Outline

Bugzilla
Testopia
Jenkins
Automated Testing Environment (Hypervisor
Testing as an example)
Automated Testing Process
Example

References
2

Bugzilla Introduction
Bugzilla is a Defect Tracking System and it is
server software designed to help you manage
software development

Bugzilla Features
Excellent security to protect confidentiality
Optimized database structure for increased
performance and scalability
Integrated email capabilities
Comprehensive permissions system
Editable user profiles and comprehensive email
preferences
Proven under fire as Mozilla's bug tracking
system
4

Bugzilla Requirements
Hardware
CPU : 3 GHz
RAM : 4 GB or more recommended
Hard Disk : 50 GB of free space is large enough

Operating System : Including Windows, Linux, and

Mac OS X
Software
Perl : 5.12
Database Server : Bugzilla supports MySQL 5.5,
PostgreSQL 8.3, Oracle and SQLite
Web Server : Apache 2.2
5

MySQL Installation
Step 1. Update
Step 2. Install MySQL

Step 3. Set the password

MySQL Installation (cont.)

Step 4. Login MySQL
user
password

MySQL Configuration
Step 5. Create an user for Bugzilla

Step 6. List users

MySQL Configuration (cont.)

Step 7. Create database for Bugzilla

Apache Configuration
Step 1. Install Apache2
Step 2. Modify Apache2 configuration file

10

Bugzilla Installation
Step 1. Download Bugzilla : http://www.bugzilla.org/
Step 2. Decompress the file to /var/www/

Step 3. Modify folder name and switch to

/var/www/bugzilla

11

Bugzilla Installation (cont.)

Step 4. Execute checksetup.pl

12

Bugzilla Installation (cont.)

Step 5. Install module

13

Bugzilla Installation (cont.)

Step 6. Re-run checksetup.pl

Step 7. Need to modify localconfig

14

Bugzilla Installation (cont.)

Step 8. Modify Bugzilla configuration

Step 9. Modify permission

15

Bugzilla Installation (cont.)

Step 10. Re-run checksetup.pl

Step 11. Set up user information

16

Bugzilla Installation (cont.)

Step 12. Modify Apache configuration and restart
Apache2

17

Bugzilla Installation (cont.)

Step 13. Installation complete

18

Testopia Introduction
Testopia is a test case management extension for
Bugzilla
Testopia Requirements

Bugzilla
MySQL or PostgreSQL
Mozilla compatible browser
Additional Perl Modules

Testopia Architecture
Test Plans
Test Cases
Test Runs
19

Relationship between Bugzilla and Testopia

Since Testopia is an extension to Bugzilla, it relies on many of the same
objects that Bugzilla uses to track bugs
Testopia is comprised of several objects that are interdependent, so they
make managing the testing procedure possible

20

Relationship between Bugzilla and Testopia (cont.)

Component
A Bugzilla component
An attribute of a product

21

Relationship between Bugzilla and Testopia (cont.)

Product
The product name belong to Bugzilla

22

Relationship between Bugzilla and Testopia (cont.)

Milestone
A Bugzilla object
Testopia builds are associated with milestones

23

Relationship between Bugzilla and Testopia (cont.)

Test Plan
At the top of the Testopia hierarchy are Test Plans. Before you can do anything else in
Testopia, you need a Test Plan
Test plan will serve as the storage point for all related Test Cases and Test Runs and it will
act as the Dashboard for your testing
24

Relationship between Bugzilla and Testopia (cont.)

Test Case
Test Cases are the heart of all testing
Test Cases spell out what steps to take while running a test and what outcomes to expect
Test Cases are associated with one or more Test Plans and with zero or more Test Runs

25

Relationship between Bugzilla and Testopia (cont.)

Test Run
Once you have defined a set of Test Cases, you are ready to run through those tests in a
Test Run
Each run is associated with a single Test Plan and can consist of any number of Test
Cases from that plan
26

Relationship between Bugzilla and Testopia (cont.)

Build
Software development is usually an iterative affair. In Testopia, each iteration is called a build
In software testing, a string denoting the compiled results of a period of development
Builds are often associated with milestones of the project. This is reflected in the build's
relationship to Bugzilla's target milestones object
27

Relationship between Bugzilla and Testopia (cont.)

Case Run
A Test Case-Run is the record of how a particular Test Case fared in a particular run for a given
Build in a given Environment

28

Relationship between Bugzilla and Testopia (cont.)

Category
A property of a product that is used to classify Test Cases

29

Relationship between Bugzilla and Testopia (cont.)

Environment
A list of the surrounding conditions that a test run is performed in

30

Testing Process of Testopia

The testing process in
Testopia is as follows :
Step 1. Create a Product and
multiple Components
Step 2. Create a Test Plan
Step 3. Create Test Cases
Step 4. Create a Build

31

Testing Process of Testopia (cont.)

The testing process in
Testopia is as follows :
Step 5. Create Environment
Step 6. Create a Test Run
Step 7. Execute the Test Run
Step 8. Produce a test report
on Dashboard

32

Testopia Installation
Step 1. Download Testopia
http://www.mozilla.org/projects/testopia/

Step 2. Decompress the file to Bugzilla folder

Step 3. Run checksetup.pl and install module

33

Access to Testopia
Step 4. Re-run checksetup.pl

Step 5. Installation complete

34

Jenkins Introduction
Jenkins is an open source continuous integration
tool written in Java
Jenkins provides a web-based user interface

35

Jenkins Features

Easy installation
Easy configuration
Change set support
E-mail integration
Test reporting
Plugin support

36

Jenkins Installation
Step 1. Install Jenkins

Step 2. Access to Jenkins

http://localhost:8080

37

Install Testopia plugin on Jenkins

38

Integrate Jenkins with Testopia

This plug-in integrates Jenkins with Testopia
and generates reports on automated test
execution
With this plug-in you can manage your tests in
Testopia, schedule and control in Jenkins
It reads TAP test report formats, used to
update Testopia test case's executions

39

Integrate Jenkins with Testopia(cont.)

Testopia configuration section

40

Auto Test Process

Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)

Auto testing environment for ITRI

ARM hypervisor :
Step 1. Retrieve Test Case from Testopia

2.

Jenkins 1.5

1.

Testopia

Step 2. Invoke corresponding script

to initialize the test environment

Bugzilla 4.4

Ubuntu 13.10

Test Controller
41

Auto Test Process (cont.)

Step 3. Initialize the test environment
Step 4. Launch necessary Guest VMs

Shell scripts (expect)

Shell scripts (expect)
Shell scripts (expect)

Jenkins 1.5
Guest
VM

Testopia

Guest
VM

Guest
VM

4.
Bugzilla 4.4

ITRI ARM Hypervisor

3.
Ubuntu 13.10

Ubuntu 12.04 LTS

Test Controller

Test Target
42

Auto Test Process (cont.)

Step 5. Execute testing tasks

Shell scripts (expect)

Shell scripts (expect)
Shell scripts (expect)

5.

Jenkins 1.5
Guest
VM

Testopia

Guest
VM

Guest
VM

Bugzilla 4.4

ITRI ARM Hypervisor

Ubuntu 13.10

Ubuntu 12.04 LTS

Test Controller

Test Target
43

Auto Test Process (cont.)

Step 6. Generate and return test
result (by TAP, Test Anything Protocol)

Shell scripts (expect)

Shell scripts (expect)
Shell scripts (expect)

6.
Jenkins 1.5
Guest
VM

Testopia

Guest
VM

Guest
VM

Bugzilla 4.4

ITRI ARM Hypervisor

Ubuntu 13.10

Ubuntu 12.04 LTS

Test Controller

Test Target
44

Auto Test Process (cont.)

Step 7. Return test result to Jenkins
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)

7.
Jenkins 1.5

Testopia

Bugzilla 4.4

Ubuntu 13.10

Test Controller
45

Auto Test Process (cont.)

Step 8. Write corresponding result to
Test Case in Testopia based on TAP

Shell scripts (expect)

Shell scripts (expect)
Shell scripts (expect)

Jenkins 1.5

8.
Testopia

Bugzilla 4.4

Ubuntu 13.10

Test Controller
46

Auto Test Example

Step 1. Add user on Bugzilla to execute auto testing

47

The process to create a Test Plan

To create a new Test Plan in Testopia

Step 1. Click the "New Plan" link in the Bugzilla footer

Step 2. Enter a name for this Test Plan
Step 3. Select a product from the product list
Step 4. Select a type for this plan
Step 5. Select a product version
Step 6. Type or paste your plan document into the Plan
Document Editor
Step 7. Click the "Submit" button

48

Step 2. Create a new Test Plan

Plan Name
Test Plan name
49

Step 2. Create a new Test Plan (cont.)

Product
Bugzilla product name
50

Step 2. Create a new Test Plan (cont.)

Product Type
The testing type for this Test Plan
51

Step 2. Create a new Test Plan (cont.)

Product Version
The Bugzilla product version
52

Step 2. Create a new Test Plan (cont.)

Plan Document
This is document that spells out the type of testing and testing methods
used for this plan
53

The process to create a Test Run

To create a new Test Run in Testopia
Step 1. Click "Create a New Test Run" link on your Test Plan
or the New Run link in the footer
Step 2. Select which Test Cases to include
Step 3. Enter a Summary for this Test Run
Step 4. Select a build from the Build list or type the name
for a new one
Step 5. Select an environment from the Environment list
Step 6. Click the "Submit" button

54

Step 3. Create a new Test Run

Product Version
The Bugzilla product version
55

Step 3. Create a new Test Run (cont.)

Run Manager
The user in charge of this Test Run
56

Step 3. Create a new Test Run (cont.)

Target Completion Rate

The expect completion rate of target
57

Step 3. Create a new Test Run (cont.)

Build
The default product build
58

Step 3. Create a new Test Run (cont.)

Environment
The environment in use for this Test Run
59

Step 3. Create a new Test Run (cont.)

Target Pass Rate

The expect pass rate of target
60

Step 3. Create a new Test Run (cont.)

Summary
A short description of this Test Run
61

Step 3. Create a new Test Run (cont.)

Notes
This field is a place to make notes and observations about this run
62

The process to create a Test Case

To create a new Test Case in Testopia
Step 1. Click "Create a New Test Case" on the plan page or
New Case from the footer
Step 2. Enter a short description of your Test Case in the
Summary field
Step 3. Select a Category
Step 4. Add a default tester
Step 5. List the steps for testing in the Action field
Step 6. Provide the expected outcomes in the Expected
Results field
Step 7. Click the "Submit" button
63

Step 4. Create a new Test Case

Summary
A short description of the Test Case

64

Step 4. Create a new Test Case (cont.)

Default Tester
The person is assigned to this Test Case

65

Step 4. Create a new Test Case (cont.)

Alias
A unique string that uses to identify test result

66

Step 4. Create a new Test Case (cont.)

Priority
The level of testing

67

Step 4. Create a new Test Case (cont.)

Category
The product category

68

Step 4. Create a new Test Case (cont.)

Estimated Time
The estimated time that this Test Case should take to complete

69

Step 4. Create a new Test Case (cont.)

Bugs
You can attach bugs to your Test Cases

70

Step 4. Create a new Test Case (cont.)

Blocks
If this Test Case prevents others from being run their ID numbers will appear
here

71

Step 4. Create a new Test Case (cont.)

Status : Test Case status

PROPOSED : This test case is a work in progress and has not been reviewed for
accuracy. It is therefore not ready to be included in test runs
CONFIRMED : This test case has passed review and is ready to be included in future test
runs. Only test cases with this status can be included in new test runs
DISABLED : This test case is no longer applicable to current testing
72

Step 4. Create a new Test Case (cont.)

Add Tags
A user defined string used to classify Test Cases

73

Step 4. Create a new Test Case (cont.)

Requirements
The requirement number or URL to a document containing the requirement
this Test Case is designed to test

74

Step 4. Create a new Test Case (cont.)

Automated
Automatic Test Cases are run by a script

75

Step 4. Create a new Test Case (cont.)

Scripts
The name of the script that runs this Test Case

76

Step 4. Create a new Test Case (cont.)

Arguments
The specific arguments sent to the script

77

Step 4. Create a new Test Case (cont.)

Add to Run
Entering a run ID here will include this Test Case in a Test Run if it is not
already included

78

Step 4. Create a new Test Case (cont.)

Depends On
If this Test Case requires other Test Cases be run before this one, their case
numbers should appear here

79

Step 4. Create a new Test Case (cont.)

Action
This field lists the steps of testing

80

Step 4. Create a new Test Case (cont.)

Expected Results
This details the expected outcomes of a test

81

Process to Launch Test Script

Configure
2.

4.

1.
Testopia
1 . Retrieve Test Script

Contents :
1. Parameters
2. Functions

3.
Start
1. Clean Environment
2. Check Image
3. Launch Test Script

82

Test Script
1. Login to Host
2. Start VM
3. Ping Host
4. Turn off VM
5. Return Result

Set up a new project

Step 5. Create a project on Jenkins to invoke Test
Script

83

Step 6. Set Testopia configuration section

Testopia Version
The version name of Testopia
84

Step 6. Set Testopia configuration section (cont.)

Test Run ID
The Test Run ID of Testopia
85

Step 6. Set Testopia configuration section (cont.)

Single Test Build Steps

The command to execute by single step
86

Step 6. Set Testopia configuration section (cont.)

Iterative Test Build Steps

The parameters of Testopia are used to execute by test cases
87

Step 7. Set Result configuration section

Test Anything Protocol (TAP)

TAP is a simple text-based interface between testing
modules in a test harness
It allows individual tests to communicate test results
to the testing harness
88

TAP File Example

The path of the file

Success contents

Fail contents

89

Launch The Project

Step 8. Start this project on Jenkins

90

Jenkins Console
Step 9. Monitor the shell scripts

91

Jenkins Result

92

Return Result to Testopia

93

Question

Mail:walkerchang@iii.org.tw
94

References
http://www.bugzilla.org/
https://developer.mozilla.org/enUS/docs/Mozilla/Bugzilla/Testopia#Requirements
http://blog.codylab.com/testcase-managementusing-testopia/
https://wiki.jenkinsci.org/display/JENKINS/Testopia+Plugin
http://testanything.org/
https://wiki.jenkinsci.org/display/JENKINS/Meet+Jenkins
95