Testing Part-2

1
 What is a Test Oracle?

Software testing depends on the availability of an oracle

Oracle: some method for checking whether the System

Under Test (SUT) has passed or failed on a particular
execution in regard to its specifications
 Human Oracle
The oracle is often a human being
– Advantage:
Good at interpreting incomplete natural-language
specifications

– Disadvantages:
1) lack of auditing accuracy if output is too complex
2) Slow and costly for large test suites
 Non-Human Oracles
Some oracles assume the availability of a set of pre-
computed input/output pairs, which is rarely the case…
– What if such set is available and covers the input space?

Some oracles assume the availability of a previous

version of the same program, which is presumed to be
correct
– This applies to regression testing, but is not sufficient
in the general case… need to test added features and
bug fixes
 Non-Human Oracles:
Pseudo-Oracles
Non-testable programs (Elaine Weyuker)
It is not possible to write oracles for such programs
– E.g., Programs which were written in order to determine the
answer in the first place. There would be no need to write such
programs, if the correct answer were known.

Solution: create a Pseudo-Oracle, a full implementation

of the SUT, developed from the same specifications by
separate team(s), or written in an entirely different
programming language
→ very precise + generally applicable
 Non-Human Oracles:
Implicit or Null Oracles
The cheapest:
– Provided implicitly by the runtime system. E.g.,
(uncaught) JVM exceptions indicate failure
– If the program under test does not terminate at all, it
can easily be identified by the tester as exhibiting a
failure

– Very popular in GUI testing

– But what if a crash is the expected behavior?!

 Assertions

An assertion is a boolean expression that is placed at a

certain point in a program to check its behavior at runtime.
– When true, the program’s behavior is regarded “as intended”
– When false, an error has been detected in the program
 Contracts
Assertions can check only a limited set of properties at a certain
point in a program
Contract-based programming extends assertions by allowing
– To check contracts between client and supplier objects in the form of
method pre- and post- conditions, class invariants, loop invariants

Eiffel (by Bertrand Meyers) was the first language to offer design
by contract
Other languages now have it, e.g., Java in the form of Java
Modeling Language (JML), C# (see next slide)
A variety of testing tools provide functionality based on JML
annotations
– E.g., generates JUnit test code from JML annotations.
public static int BinarySearch(int[]! a, int key) // C#
requires forall{int i in (0: a.Length), int j in (i: a.Length); a[i] <= a[j]};
ensures 0 <= result ==> a[result] == key;
ensures result < 0 ==> forall{int i in (0: a.Length); a[i] != key};
{
int low = 0;
int high = a.Length - 1;

while (low <= high)

invariant high+1 <= a.Length;
invariant forall{int i in (0: low); a[i] != key};
invariant forall{int i in (high+1: a.Length); a[i] != key};
{
int mid = (low + high) / 2;
int midVal = a[mid];

if (midVal < key) {

low = mid + 1;
} else if (key < midVal) {
high = mid - 1;
} else {
return mid; // key found
}
}
return -(low + 1); // key not found.
}
Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 9
 More Oracles

Human Oracles √
Implicit/Null Oracles √
Pseudo-oracles √
Regression Test Suites √

Metamorphic Testing
 Metamorphic Testing
Metamorphic testing can be used to test “non-
testable programs”

Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 11

 Metamorphic Relations (Example 1)
Given a function f(x) that is meant to implement sin(x)
Given test suite T = {0.0, 1.0, π}
Given that the program is non-testable
However, we know that sin(π –x) = sin(x)

To verify that the program behaves correctly for

– x = 0.0, we check whether f(0.0) == f(π – 0.0)
– x = 1.0, we check whether f(1.0) == f(π – 1.0)
– x = π, we check whether f(π) == f(π – π)

Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 12

 Metamorphic Relations (Example 2)
Given a function f(x) that is meant to implement ex
Given test suite T = {0.0, 1.0, 2.0}
Given that it is not possible to write an oracle (non-testable)
However, we know that exe-x = 1

To verify that the program behaves correctly for

– x = 0.0, we check whether f(0.0) x f(-0.0) == 1
– x = 1.0, we check whether f(1.0) x f(-1.0) == 1
– x = 2.0, we check whether f(2.0) x f(-2.0) == 1

Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 13