ORTHOGONAL ARRAY

TESTING
PRESENTED BY :
Prince Bhanwra [801031024]
Ravinder Dahiya [801031025]
 Contents
 Software Testing
 Conventional Test Cases
 Conventional Testing Issues

 Software Faults
 Why OATS..??
 Terminology of OATS
 Examples
 OATS Advantages
 OATS Limitations
 Challenges
 Mixed Mode OA
 Available Test Automation Tools
 References

2 Thapar University, Patiala

 Testing

 SOFTWARE TESTING
Software testing is an investigation conducted to
provide stakeholders with information about the
quality of the product or service under test.

 CONVENTIONAL / EXHAUSTIVE TESTING

A test approach in which the test suite comprises all
combinations of input values and preconditions.

3 Thapar University, Patiala

 Conventional Test Cases
Example:
If we have three variables
(A,B,C), each can have 3
values say (Red, Green, and
Blue).
The possible combinations
in conventional test cases
would be 27 i.e. 33

4 Thapar University, Patiala

 Conventional Testing Issues
 Conventional Test Cases:  Orthogonal Test Cases
 Variables:3  Variables:3

Input: 4 Input: 4
Possible cases: 81=34 Possible cases: 9
 Variables: 3  Variables: 3

Input: 5 Input: 5
Possible Cases: 243 = 35 Possible Cases: 11
 …  …

 Variables: 5  Variables: 5

Input: 5 Input: 5
Possible Cases: 3125 = 55 Possible Cases: 21

5 Thapar University, Patiala

 Software Faults
1. Region Faults.
2. Isolated Faults.

Orthogonal Array based testing is highly effective for

the detection of region faults with a relatively small
number of tests.

6 Thapar University, Patiala

 Why Orthogonal Array Testing (OATS)..??
 Systematic, statistical way to test pair-wise interactions.
 Interactions and integration points are a major source of
defects.
 Most defects arise from simple pair-wise interactions.
 “When the background is blue and the font is Arial and the layout
has menus on the right and the images are large and it’s a
Thursday then the tables don’t line up properly.”
 Exhaustive testing is impossible.
 Execute a well-defined, concise set of tests that are likely
to uncover most (not all) bugs.
 Orthogonal approach guarantees the pair-wise coverage
of all variables.

7 Thapar University, Patiala

 Terminology for working with OA’s
OA’s are commonly represented as :
LRuns(LevelsFactors) or OA(Runs, Factors, Levels, Strength) or
OAλ(Runs(N); Factors(k), Levels(v), Strength(t))
is an N × k array on v symbols such that every N × t sub-array contains all
tuples of size t from v symbols exactly λ times.

Runs (N) – Number of rows in the array, which translates into the number
of Test Cases that will be generated.
Factors (k) – Number of columns in the array, which translates into the
maximum number of variables that can be handled by the array.
Levels (v) – Maximum number of values that can be taken on by any
single factor.
Strength (t) – The number of columns it takes to see all the possibilities
equal number of times.
ƒ λ -1 for software testing and is often omitted

8 Thapar University, Patiala

 Example 1
A Web Page has three distinct sections (Top, Middle,
Bottom) that can be individually shown or hidden from user

No.of Factors=3 (Top,middle,Bottom)

No.of Levels =2 (Hidden or shown)

Array Type =L4(2 3) or OA(4,3,2,2)

OA(Runs, Factors, Levels, Strength)

If we go for exhaustive testing we need :2 x 2 x 2 = 8 Test Cases

9 Thapar University, Patiala

 Example 1 (contd..)
Fixed Level Array: L4(2 3)
F1 F2 F3 Top Middle Bottom
Run1 0 0 0
Test 1 Hidden Hidden Hidden
Run 2 0 1 1
Run 3 1 0 1 Test 2 Hidden Visible Visible

Run 4 1 1 0 Test 3 Visible Hidden Visible

Test 4 Visible Visible Hidden

L423 – OA with 4 Runs
3 factors with 2 levels
The Four Test Scenarios (4 Vs. 8)

1 - Display the home page and hide all sections.

2 - Display the home page and show all but the Top section.
3 - Display the home page and show all but the Middle section.
4 - Display the home page and show all but the Bottom section.

10 Thapar University, Patiala

 Example 2
Example:
If we have three variables
(A,B,C), each can have 3
values say (Red, Green, and
Blue).
The possible combinations
in OATS test cases would be
9.

11 Thapar University, Patiala

 OATS advantage to select a test set:
 Guarantees testing the pair-wise combinations of all the
selected variables.
 Creates an efficient and concise test set with many fewer
test cases than testing all combinations of all variables.
 Creates a test set that has an even distribution of all pair-
wise combinations.
 Exercises some of the complex combinations of all the
variables.
 Is simpler to generate and less error prone than test sets
created by hand.

12 Thapar University, Patiala

 OATS basic fault model:
 Interactions and integrations are a major source of defects.
 Most of these defects are not a result of complex
interactions such as
 “When the background is blue and the font is Arial and

the layout has menus on the right and the images are
large and it’s a Thursday then the tables don’t line up
properly.”
 Most of these defects arise from simple pair-wise
interactions such as
 “When the font is Arial and the menus are on the right

the tables don’t line up properly.”

13 Thapar University, Patiala

 OATS basic fault model (contd..):
 With so many possible combinations of
components or settings, it is easy to miss one.
 Randomly selecting values to create all of the pair-
wise combinations is bound to create inefficient
test sets and test sets with random, senseless
distribution of values.
 It covers 100% (9 of 9) of the pair-wise
combinations, 33% (9 of 27) of the three-way
combinations and 11% (9 of 81) of the four-way
combinations.

14 Thapar University, Patiala

 The Challenges
 The number of tests may increase rapidly as the degree of
interactions increases
 ‰ Test generation algorithms must be more sensitive in
terms of both time and space requirements
 ‰ The need for test automation becomes even more
serious.
 Impractical to manually execute and inspect the results

of a large number of test runs

 However, on the Web there are pre-calculated

orthogonal array tables for certain numbers of variables

and values.

15 Thapar University, Patiala

 Mixed Mode OA

16 Thapar University, Patiala

 Available Test Automation Tools
1.
CATS (Constrained Array Test System) [Sherwood] Bell
 *) Labs.

2.
OATS (Orthogonal Array Test System)
[Phadke] AT&T
 *)

3. Web-based,
AETG Telecordia
commercial
4. IPO (PairTest) *) [Tai/Lei]
5. TConfig [Williams] Java-applet
6.
TCG (Test Case Generator) *) NASA

7. Perl script, free,

AllPairs Satisfice
GPL
8. GUI,
Pro-Test SigmaZone
commercial
9. Free for non-
CTS (Combinatorial Test Services) IBM
commercial use

17 Thapar University, Patiala

 References
 http://csrc.nist.gov/groups/SNS/acts/documents/ipo-nist.pdf
 http://csrc.nist.gov/groups/SNS/acts/documents/aberdeen.ppt
 http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-system
-design-summer-1998/lecture-notes/l3_matrix_experiments4.pdf
 http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-
system-design-summer-1998/lecture-notes/l8_orth_arrays.pdf
 http://delivery.acm.org/10.1145/1290000/1283384/p1-abam.pdf?
key1=1283384&key2=5753427921&coll=DL&dl=ACM&CFID=9321178&CFTO
KEN=33991028
 http://www.developsense.com/pairwiseTesting.html
 http://learnsoftwareprocesses.com/2007/08/21/orthogonal-array-testing-
strategy-oats/comment-page-1/#comment-7394
 http://www.vietnamesetestingboard.org/zbxe/?document_srl=219676
 http://www.pairwise.org/tools.asp

18 Thapar University, Patiala

 Queries..?

19 Thapar University, Patiala

 Thank you !!

20 Thapar University, Patiala