EY pairwise

testing
Test methodology
and capabilities
 Executive summary
Software testing usually consists of test cases based upon adequacy of test scope coverage is questionable. This
requirements or user stories. To achieve a desired level of adds risk to the project and unanticipated expenses when
coverage, a certain number of tests will be executed, and defects are found in production.
they usually contain a lot of redundancy. This approach
What can help? Is there a practical solution to address
is inefficient, wastes time and extends deployment
these challenges?
timelines. Due to resource and time constraints, test case
creation and execution are usually truncated, and the

TABLE OF CONTENTS
Executive summary 3
Pairwise testing 4
Current state 4
Why pairwise? 5
Benefits of pairwise 6
When to apply pairwise testing 7

Data parameter-driven
test design tool 8
Constraints tab 8
Requirements tab 9
Test Cases tab 9
Analysis tab 10
Model-based test design tool 11

EY pairwise testing 3
Pairwise testing
Pairwise is an industry technique that has been around
for more than a decade. The demand to get to the market
faster has driven a growth in the use of tools to do testing
faster while not compromising quality and risk. In the
past, pairwise tools were used on stable systems; however,
using it upfront while building the system has shown it
reduces testing efforts. Industry defines pairwise as “a
combinatorial method of software testing that, for each
pair of input parameters to a system (typically, a software
algorithm), tests all possible discrete combinations of
those parameters.” Pairwise generally means “occurring
in pairs” or “two at a time.”
Pairwise is used to achieve a desired result in the most
efficient manner possible. When given a set of input
parameters, the algorithm derives a minimum number
Why pairwise?
of scenarios with all discrete value pairs to be tested at Without certain tools, it can be extremely challenging for to perform risk-based testing. Pairwise tools provide
least once. When testing a complex application, there testers and managers to determine how much testing is modeling capabilities that are easily understood and easy
can be an enormous number of possible tests that enough. A pairwise test tool helps derive an optimal mix to use. Furthermore, it is easy to maintain test cases by
could potentially be executed. There usually is no clear of scenarios without negatively impacting the cost. At adding new features or eliminating those not needed.
mathematical basis to identify the test coverage and the same time, it offers adequate control to the teams
no visual analysis of all the possible tests in one single
location. This is mostly because documentation is either
not updated with the latest requirements or is scattered
High Are we here?
across systems. With information scattered across Are we here?
multiple sources, there is a repeated effort to test the
same or a similar test scenario, which fails to add value
and takes up more time. Pairwise tools not only provide ?
you the optimized data combination but also have an ?
analysis capability that visually shows the pairs and
coverage percentage and allows you to adjust to see the
Cost of testing
impact.
plus development
Too few test: Appropriate Too many tests:
gaps in coverage amount largely redundant
lead to costly of testing tests result in
bug fixes delays and waste

Low

Low High
Amount of testing

Demand to get to the market fast is high. This puts
more pressure on testing to be able to handle changes
and new requirements faster. The demand to deliver
faster presents other challenges, as well. With crunched
timelines, the ability to test everything in a shorter
amount of time becomes an issue. Automation has been
one of the capabilities that has been implemented to
speed up testing. However, automation can be expensive
and is not always effective or implemented correctly.
Additionally, to ensure quality and reduce risk, you need
to make sure you have adequate coverage and are testing
the highest-risk areas. One approach used today is to use
Excel with macros to define all the combinations, which
is very time-consuming to create and doesn’t optimize.
Therefore, there are a lot of redundancies. These
redundancies get carried over to the test automation,
thereby impacting the cost to automate as testers then
have more scripts to analyze, automate and run. As tools
have advanced, pairwise has become more applicable by
reducing the time it takes to define all the combinations
and generate the test cases.
Pairwise tools save time in both the planning and cases but do not execute test cases. However, they do
execution phases by enabling systematic testing provide capabilities to export to other tools that can
coverage. During the test design stage, teams are able to automate test execution. The pairwise application can
select and document tests much faster as guesswork is become less effective when parameters are applied
eliminated. On average, pairwise testing can achieve 50% unnecessarily. To thoroughly identify parameters and
time savings without sacrificing coverage. When teams test sets, testers must possess good critical thinking
are deciding how much to test, tools allow testers to skills, experience, and knowledge of the system and
configure certain parameters to achieve a desired level of business processes being tested. With the demand to
coverage with the fewest number of tests. test faster and the growing maturity of pairwise testing,
incorporating pairwise in the existing test frameworks
There are certain fundamentals that testers should
between a test management tool and a test automation
understand when using pairwise testing tools. For
tool is an ideal choice.
instance, pairwise tools automate the creation of test
1. Create/upload test cases 1. Create modal/test inputs 1. Create test scripts
2. Execute test cases 2. Create optimized tests 2. Execute test scripts
3. Create defects 3. Export test cases 3. Report results

Test management tool Pairwise/OAT tool aka test/ Test automation tool
design/model/optimization

4 EY pairwise testing EY pairwise testing 5

Benefits of pairwise
The benefits of using pairwise tools include:
• E
► ffective test design with optimal number of test cases
• More thorough test coverage with easily understood
visual representation
• Automatic design of higher-quality test cases
• Effective management of change requests
• Easy integration with test management tools
• Increased number of defects found
• Can be adopted anywhere along the testing life cycle
Pairwise can begin with a model-based testing concept
that records requirements and then aids in designing test
scenarios and test cases. Pairwise can also start with the
entering of parameters and values. Two of the tools we
mention below are examples of the two different starting
points. One starts with process models, while the other
starts with parameters and values. The one that starts
with process models has an add-on that allows you to
record out-of-the-box scripts with Selenium and then
import them to generate initial process models. What this
means is that the tool you use may be determined based
on where in the software development life cycle you are
applying pairwise testing. A tool that starts with process
models is best applied at the beginning of the project,
when requirements are about to begin, whereas a tool
that starts with parameters and values is best applied
• Identify optimal end to end test
scenarios based on known facts
• Review and priortize test
coverage for SIT
• Optimize and update existing
test cases effected due to a
requirement change
• Review and prioritize test
coverage for regression
• Large number of configuration
options to validate
• Create unit tests for complex
functionalities
Planning Sprint 1
and inception
Implementation cycles
6 EY pairwise testing
When to apply pairwise testing
when you have, at a minimum, test scenarios and know 1. Projects with a significant testing effort, where test 5. Innovative, analytical project teams that embrace
the parameters you need to test with. In a best-case design efficiencies and coverage improvements will process-improvement and are willing to learn new
scenario, you know what parameters are impacted by have the most impact test design approaches
rules or lines of code being applied.
2. Projects that require multiple data combination to be In addition, there are testing patterns across the life
Once you have entered either the process models tested, resulting in a large number of test cases cycle of an engagement in which a pairwise testing tool
or parameters and values, the next step is to enter can help reduce testing effort and maximize coverage.
3. Web-based applications with significant decision-
constraints and specific requirements if necessary. When compared to orthogonal arrays, pairwise is more
making processes
The tool then automatically builds the test cases. efficient and effective for software testing. Orthogonal
Testers can update test cases as necessary and then 4. First-release projects or projects in the early stages arrays are more efficient and effective for manufacturing,
select the degree of coverage. Coverage metrics of test creation; this will allow for earlier defect agriculture, advertising and many other settings.
provide a view of the amount of coverage the test detection and for test efficiencies to be realized over
cases selected will provide. a longer period of time
A great example of an application that is a good candidate
for the use of pairwise testing is know your customer
(KYC). KYC requires a list of questions to be asked to
verify whether a customer is eligible for banking products.
The questions render additional questions and can
take multiple paths. Pairwise is an excellent technique
to validate that all questions are tested. It takes those
questions as parameters and the answers as values;
constraints will be entered, then the tool will generate a
set of test cases that will test the minimum coverage of
all paths of parameters and values.
Refer to the pairwise tool example below for steps and
screen shots.
• Review and utilize existing test
flows to determine the applicable
regression suite for sub releases
• Create optimal test designs for
future releases and leverage
existing tet design approach
Production/
future releases
SIT/UAT
Sprint n
Agile
implmentation
Sprint 2 flow
EY pairwise testing 7
Data parameter-driven test design tool
Input tab
Here is an example using one of the numerous tools that enable
pairwise testing. It allows creation of a new plan or the update of
an existing plan. Once in the plan, the Input tab allows the tester
to enter parameters and their values that they would like to test.
Parameters and values are the indicators that change from test case
to test case. Values are the different selections for a parameter. The
more parameters entered for a test plan, the less effective the tool
becomes. Therefore, a tester should consider creating multiple
smaller plans.
Constraints tab
Once your parameters have been entered, the next step is to enter
any constraints against those parameters. Constraints are rules that
define how the parameters and their values should work with other
parameters and values. For example, a parameter and one of its
values may only be allowed when another parameter and its value
are met (invalid pairs, married pairs). If/then statements also help in
defining constraints. Constraints can be 1:Many or Many:1.
8 EY pairwise testing
Requirements tab
The next tab is Requirements. Requirements can
be imported from several formats, such as Word or
Excel. When using Excel, data inputs will need to be
in the format of Name, Description and Expected
Outcome. After importing requirements, you can add
specific combinations or test conditions/test inputs.
Requirements can then be traced to parameters. This tool
also allows a user to force a specific value for a parameter.
The constraints tab ensures that a parameter cannot be
tested with another or that they must be tested together.
Test Cases tab
The Test Cases tab generates the test cases. The default
on this tool creates test cases in a two-way interaction,
but one can select up to a five-way interaction or
mixed-strength interaction. Increasing the number
of interactions increases coverage and reduces risk.
However, as you increase the number of interactions, the
number of test cases increases and, in return, time and
effort increase. Risk is the driving factor. How much is
However, there is no guarantee that each defined married
pair and/or constraint will appear in the scripts created.
The Requirements tab allows you to manually input a set
married pair, constraints or a set of multiple combinations
of data to ensure they are created. The Requirements
tab can be used to ensure negative testing occurs, as
well. One thing to consider is that the more requirements
you add, the more test cases are necessary. Therefore,
consider the level of risk the project is willing to take.
your project willing to take? If you are testing a lifesaving
device vs. an insurance application, your risk tolerance
would be low; therefore, you would do more interactions.
The analysis tools will show you at what point you have
a large percentage of coverage and the difference if
you test more. You can add steps or update expected
outcomes anytime to increase coverage if you choose
only two-way.
EY pairwise testing 9
Analysis tab
The Analysis tab illustrates the test coverage in different
formats such as Plan Scorecard, Coverage Graph and
Matrix Chart. These can be used to determine how
much test coverage has occurred at any time and how
you can maximize investment on return. The Coverage
Graph shows that as you increase the number of tests,
the percentage of coverage starts to reach a point of
diminishing returns. The Matrix Chart shows what is
being covered based on the number of tests you select to
perform. The Plan Scorecard lists the features you used
10 EY pairwise testing
and shows what you tested.
This is a particularly powerful technique when confronted
with resource or time constraints. The Analysis tab
demonstrates empirically that adequate scope coverage
(say 80%, given the risk profile of the application) can
usually be obtained by executing less than half of the
tests. The Matrix Chart also provides a road map to
constructing negative tests, as it illustrates what should
not be occurring within the application under test.
Model-based test design tool
Here is an example of another tool that provides a slightly
different approach on pairwise testing. This tool starts
with business process models. The models can be built
manually by dragging and dropping symbols based on
business process flows, or they can be uploaded from
an existing process flow tool. This tool has an add-on
that works with Selenium that will allow you to record
application steps that can be imported to create the
The next step is to generate the test cases. Select the
type of optimization you want to perform and then
generate the test cases. All paths and the number of
process models. This could also act as a starting point to
build out requirements. For example, you could record
execution steps against an out-of-the-box version of an
application, then import those and update them to reflect
the changes you want. Each of the symbols represents
a different step at which constraints, values and specific
requirements can be applied.
nodes will appear so you can select and review. Save the
paths generated and then you can review the coverage
amounts for each path.
EY pairwise testing 11
Summary
An abundance of software testing tools exists in the
marketplace, and each of those tools represents a slightly
different approach. Pairwise is not an industry-specific
tool; therefore, it can be used across multiple industries
and applications. It may be more desirable to use one
vs. another depending upon where in the software
development life cycle the project is. In general, utilizing
a pairwise tool can save you time and money.
Pairwise does require some upfront investment in
preparing the data, constraints and requirements, as
well as understanding the business processes. However,
pairwise tools are very easy to maintain. Once you have
created your initial set, you can easily update as changes
or new releases occur. Additionally, pairwise tools are
very powerful for regression testing, so the long-term
reuse offsets the upfront investment. In one of our
recent engagements, we conducted a retrospective using
pairwise-generated test cases, and we demonstrated
that the same defects would have been detected with
significantly fewer test cases. In this particular example,
with less than half.
Furthermore, pairwise uses a systematic approach. The
goal of a systematic approach is to identify the most
efficient means to generate consistent and high-coverage
test cases. Systematic sampling is simpler and more
straightforward than random sampling, which can be
beneficial when formulating an approach requiring wide
testing scope coverage. Want to learn more?
If you have specific questions related to your situation,
please reach out to John McEvoy at john.mcevoy@ey.com
EY | Assurance | Tax | Transactions | Advisory
About EY
EY is a global leader in assurance, tax, transaction and advisory
services. The insights and quality services we deliver help build
trust and confidence in the capital markets and in economies the
world over. We develop outstanding leaders who team to deliver
on our promises to all of our stakeholders. In so doing, we play a
critical role in building a better working world for our people, for
our clients and for our communities.
EY refers to the global organization, and may refer to one or
more, of the member firms of Ernst & Young Global Limited,
each of which is a separate legal entity. Ernst & Young Global
Limited, a UK company limited by guarantee, does not
provide services to clients. Information about how EY collects
and uses personal data and a description of the rights individuals
have under data protection legislation is available via ey.com/
privacy. For more information about our organization, please
visit ey.com.
For more information about our organization, please
visit ey.com.
Ernst & Young LLP is a client-serving member firm of
Ernst & Young Global Limited operating in the US.
© 2019 Ernst & Young LLP.
All Rights Reserved.
US SCORE no. 06481-191US
1904-3108678
ED None
This material has been prepared for general informational purposes only
and is not intended to be relied upon as accounting, tax or other professional advice.
Please refer to your advisors for specific advice.
ey.com