TEST AUTOMATION AND

TESTING TOOLS
 TEST AUTOMATION AND TESTING
TOOLS
To understand test automation and testing tools
1. Describe advantages of test automation
2. Explain the Guidelines for automated testing
3. State the categories of testing tools
4. Explain the selection of testing tools
5. Explain the open source testing tools – CUT, Cgreen,
Emma and Findbugs
6. Describe about commercial testing tools – WinRunner,
SilkTest, LoadRunner, Jmeter, and TestDirector
7. State how object oriented testing is done
8. Describe how testing of web based systems are done
 TEST AUTOMATION
• Testing tools are a big aid to the testers
• Testing activities , if performed manually,
may take a lot of time. Thus there is a
need for automating the testing activities
• Test automation is not a replacement of
manual testing. It must be seen as an
enhancement to the manual testing
process
ADVANTAGES OF TEST AUTOMATION / NEED
FOR AUTOMATION / BENEFITS OF
AUTOMATION
• 1. Reduction of testing effort :
• 2. Reduces the tester’s involvement in
executing tests
• 3. Facilitates regression testing
• 4. Avoids human mistakes
• 5. Reduces overall cost of the software
• 6. Simulated testing
• 7. Internal testing
• 8. Test enablers
• 9. Test case design
ADVANTAGES OF TEST AUTOMATION / NEED
FOR AUTOMATION / BENEFITS OF
AUTOMATION
• 1. Reduction of testing effort :
– Validation and Verification testing strategies may
includes thousands of test cases.
– Manual execution of all of them takes a lot of efforts
and time. So, executing test suites through tools
greatly reduces the amount of time required.
• 2. Reduces the tester’s involvement in executing
tests
– Automating the test suite execution will relieve the
testers to do some other work.
• 3. Facilitates regression testing
– Regression testing is the most time consuming
process. So, automating regression testing will reduce
testing effort and testing time.
ADVANTAGES OF TEST AUTOMATION /
NEED FOR AUTOMATION / BENEFITS OF
AUTOMATION
• 4. Avoids human mistakes
– Manual testing may cause errors in the process or
sometimes, testers may be biased to limited test cases. All
these manual problems can be avoided with automation.
• 5. Reduces overall cost of the software
– As testing tools ease the burden of the test case
production and execution, time and therefore cost can be
greatly reduced.
• 6. Simulated testing
– Load performance testing is an example of testing where
the real-life situation needs to be simulated in the test
environment. Manual creation of real loads like a number of
concurrent users, large amount of data etc. may be difficult.
Automation tools can easily create millions of concurrent
virtual users/ data and effectively test the project.
ADVANTAGES OF TEST AUTOMATION / NEED FOR
AUTOMATION / BENEFITS OF AUTOMATION

• 7. Internal testing
– Testing may require testing for memory leakage or
checking the coverage of testing. Automation tools
can help doing these tasks quickly and accurately
• 8. Test enablers
– While development is not complete, some modules
may not be ready for testing. So stubs and drivers
need to be created. Automation is essential in these
cases to reduce the effort required.
• 9. Test case design
– Automated tools can design test cases with a
guarantee of better coverage than manual design.
 GUIDELINES FOR AUTOMATED
TESTING
• Testing tools can never replace the analytical
skills required to conduct testing and manual
testing.
• So automation should be adopted carefully
after deciding which tool and how many tools
are required, how much resources are
required including cost of the tool and time
spent on training
• Listed below are few guideline to be followed
while planning automation in testing :
 GUIDELINES FOR AUTOMATED
TESTING
• 1. Consider building a tool instead of buying
one, if possible
• 2. Test the tool on an application prototype
• 3. Not all the tests should be automated
• 4. Select the tools according to
organizational need
• 5. Use proven test-script development
techniques
• 6. Automate the regression tests whenever
feasible
 GUIDELINES FOR AUTOMATED
TESTING
• 1. Consider building a tool instead of buying one, if
possible
– If the requirement is small and enough resources are
available, go for building a tool instead of buying.
– Whether to buy or build a tool requires management
commitment including budget and resource approvals.
• 2. Test the tool on an application prototype
– While purchasing a tool, it is important to verify that
the tool works properly with the system being
developed. So it is suggested that the development
team can build a system prototype for evaluating the
testing tool.
 GUIDELINES FOR AUTOMATED
TESTING
• 3. Not all the tests should be automated
– Automated testing is an enhancement of manual
testing .
– Some tests are impossible to automate such as
verifying a print out. It has to be done manually.
• 4. Select the tools according to organizational
need
– Do not buy the tools just for their popularity or to
compete with other organizations.
– Focus on the needs of the organization and know
the resources ( budget, schedule ) before
choosing the automation tool.
 GUIDELINES FOR AUTOMATED TESTING
• 5. Use proven test-script development techniques
– Automation can be effective if proven techniques are
used to produce efficient,maintainable, and reusable
test scripts
– Following are some hints :
• i. Read the data values from either spreadsheets or tool-
provided data pools, rather than being hard coded into
the test script because this prevent test cases from being
reused.
• ii. Use modular script development for maintainability
and readability
• iii. Build a shared script library of reusable functions
which is usable by all test engineers
• iv. All test scripts should be stored in a version control
tool
 GUIDELINES FOR AUTOMATED
TESTING
• 6. Automate the regression tests whenever
feasible
– Since regression tests need a lot of time,
automate the regression test cases whenever
possible. Thus, testing time can be reduced to a
greater extent.
 CATEGORIES OF TESTING TOOLS
• 1. Static and Dynamic testing tools
• Static testing Tools
• Dynamic testing Tools
• 2.Testing Activity Tools
– Review and inspections
• Complexity analysis tools
• Code Comprehension
– Test Planning
– Test Design and development
• Test data generator
• Test case generator
– Test execution and evaluation
• 1. Capture/playback tools
• 2. Coverage analysis tool
• 3. Memory testing tools
• 4. Test management tools
• 5. Network testing tools
• 6. Performance testing tools
 CATEGORIES OF TESTING TOOLS
• 1. Static and Dynamic testing tools
• Tools are based on the type of execution of test cases, namely
static and dynamic
• • Static testing Tools
• o Static tools are analyzers which scan the source program and
detect possible faults and anomalies
• o They parse the program text and detect the following :
–1. Statements are well-formed
–2. Inferences about the control flow of the program
3. Compute the set of all possible values for program data.
• o Static tools perform the following type of static analysis :
–1. Control flow analysis : Detects loops with multiple exits and entry
points and unreachable code
–2. Data use analysis : It detects all types of data faults
–3. Interface analysis : Detects all interface faults . also detects functions
which are never declared, never called and function results never used
– 4. Path analysis : It identifies all possible paths through the program
 CATEGORIES OF TESTING TOOLS
• • Dynamic testing Tools
• o These tools support the following
• 1. Dynamic testing activities
• 2. Sometimes, it is difficult to generate test cases and perform
testing because several operations being performed concurrently.
Automation tools sometimes work like “Program Monitors”
which can capture the state of events during execution of
program by preserving a snapshot of the conditions.

• Program monitors perform the following functions :

–a. List the number of times a component is called or line of code is

executed
–b. Report on whether a decision point has branched in all directions
( info about branch coverage )
–c. Report summary statics of statement coverage, path coverage
and branch coverage by the set of test cases run.
 CATEGORIES OF TESTING TOOLS
• 2.Testing Activity Tools
• These tools are based on activities or tasks in a particular
phase of SDLC. Testing activities can be categorized as:
– o Review and inspections
– o Test Planning
– o Test Design and development
– o Test execution and evaluation
• • Tools for review and inspections
• In this category, the following tools are required :
– 1. Complexity analysis tools : These tools analyze the complexity
so that testing time and resources can be estimated
– 2. Code Comprehension : Helps in understanding dependencies,
tracing program logic, viewing graphical representation of
program and identifying the dead code which enable the
inspection team to analyze the code extensively
 CATEGORIES OF TESTING TOOLS
• • Tools for test planning :
• The types of tools required for test planning are :
– 1. Templates for test plan documentation
– 2. Test schedule and staffing estimates
– 3. Complexity analyzer
• • Tools for test design and development
– 1. Test data generator : It automates the generation of test
data on a user defined format. It quickly populates a
database needed for functional testing, load testing ,
performance testing etc. based on a set of rules
– 2. Test case generator : It automates the procedure of
generating the test cases. It uses the information provided
by a requirement management tool and creates the test
cases. It also works under the guidance of historical
knowledge-base.
 CATEGORIES OF TESTING TOOLS
• • Test execution and evaluation tools
• 1. Capture/playback tools :
– These tools record events ( keystrokes, mouse activity, display output etc. ) at the
time of running the system and place the information into a script which can
replayed later to test the system
• 2. Coverage analysis tool :
– These tools automate the process of thoroughly testing the software and provide
a quantitative measure of the coverage .
– These tools are helpful for :
• a. Measuring structural coverage
• b. Quantifying the complexity of design
• c. Specifying parts of the software which are not being covered
• d. Measure the number of integration tests required to qualify the design
• e. Help to produce integration tests
• f. Measure the number of integration tests that have not been executed
• g. Measure multiple levels of test coverage ( branch, condition, decision/condition,
multiple conditions, and path coverage )
• 3. Memory testing tools
– a. These tools verify that an application is properly using its memory resources

– b. Memory testing tools check whether an application is :

• i. Not releasing memory allocated to it

• ii. Overwriting/overriding array bounds
• iii. Reading and using uninitialized memory
 CATEGORIES OF TESTING TOOLS
• 4. Test management tools
– a. Test management tools try to cover most of the activities
in the testing life cycle ( planning, analysis & design)
– Example : Rational’s TestStudio
• 5. Network testing tools
– a. Client-server applications have increased potential for
errors due to inter-platform connectivity
– b. Network testing tools monitor, measure, test and diagnose
performance across an entire network including :
• i. Performance of the server and the network
• ii. Overall system performance
• iii. Functionality across server, client and the network
• 6. Performance testing tools
– a. Performance testing tools help in measuring the response
time and load capabilities of a system
SELECTION OF TESTING TOOLS
Tool selection depend on several factors like :

• Need of the organization

• Project environment
• Current testing methodology etc.
 Guidelines to be followed while
selecting testing tools are :
1. Match the tool to its appropriate use
2. Select the tool to its appropriate SDLC
phase
3. Select the tool to the skill of the tester
4. Select a tool which is affordable
5. Determine how many tools are required
for testing the system
6. Select the tool after examining the
schedule of testing
 Guidelines to be followed while
selecting testing tools are :
• 1. Match the tool to its appropriate use
–A tool may not be a general one or may not cover many features. Therefore the
tester needs to be familiar with both the tool and its use in order to make a
proper selection
• 2. Select the tool to its appropriate SDLC phase
–• Since the methods of testing changes according to the SDLC phase, it is
necessary to choose the tool according to the SDLC phase in which the testing is
to be done.
• 3. Select the tool to the skill of the tester
–• The individual performing the test must select a tool that conforms to his skill
level.
• 4. Select a tool which is affordable
–• Tools are always costly and increase the cost of the project. Therefore, choose
the tool which is within the budget of the project.
• 5. Determine how many tools are required for testing the system
–• A single tool may not satisfy all test requirements. Therefore, assess the tool as
per the test requirements and determine the number and type of tools required
• 6. Select the tool after examining the schedule of testing
–• First get an idea of the entire schedule of testing and then decide whether there
is enough time for learning the testing tool and then performing automation.
 TESTING TOOLS
• OPEN SOURCE TESTING TOOLS
1. C Unit Test System (CUT or CuTest)
2. Cgreen
3. EMMA:
4. FindBugs - Find Bugs in Java Programs
5. Cfix - Unit Testing Framework for C and C++
• COMMERCIAL TESTING TOOLS
1. WinRnner
2. SilkTest
3. LoadRunner
4. Jmeter
5. TestDirector
6. IBM Rational SQA Robot
OPEN SOURCE TESTING TOOLS
 C Unit Test System (CUT or CuTest)
• CUT is a simple unit testing system. It's
different from other unit test packages in that
it follows the KISS ( Keep it simple, Stupid)
principle. It's designed for C testing, not
designed to emulate SUnit (SUnit is a unit
testing framework for the programming
language Smalltalk).
• CUT is OS independent
• It is a unit testing library for the C language.
• It can be used to do Extreme Programming
and Test-First Development in the C language.
• It's a fun and cute library that will make your
programming fun and productive.
 C Unit Test System (CUT or CuTest)
Benefits
• i. Lower Defects.
– The tests ensure that your code keeps working as you make
small changes in it.
• ii. Faster Debugging.
– The tests tell you which subroutine is broken. You avoid
spending hours trying to figure out what's broken.
• iii. Development Speed.
– You trust your old code and can keep adding to it without
worrying about bad interactions. If there is a bad interaction
the tests will catch it.
• iv. Permanent Bug Fixes.
– If every time a bug is reported you write a quick test, you will
guarantee that the bug never reappears again.
• v. Fun.
– As your bug count drops you will begin to enjoy programming
like you've never done before. Running the tests every few
minutes and seeing them pass feels good.
 C Unit Test System (CUT or CuTest)
Features
• i. Small.
– Consists of a single .c and .h file.
• ii. Easy to Deploy.
– Just drop the two files into your source tree.
• iii. Highly Portable.
– Works with all major compilers on Windows (Microsoft,
Borland), Linux, Unix, PalmOS.
• iv. Open Source.
– You can extend it to add more functionality. The source
can be invaluable if you are trying to trace a test failure.
• v. Cuteness.
– Of all the testing frameworks CuTest has the cutest
name
 Cgreen
• Cgreen is a pure C unit tester with mocks.
• Features are:
i. Composite test suites,
ii. Each test runs in its own process, setup()
and teardown()
iii. Reporting can be easily changed,
iv. The ability to create mock functions.
• Requirement: Linux, MacOS
 EMMA:
• A free Java code coverage tool
• • Open-source toolkit for measuring and
reporting Java code coverage
• • EMMA is an open-source toolkit for
measuring and reporting Java code
coverage.
• • EMMA support for large-scale enterprise
software development while keeping
individual developer's work fast and
iterative.
 EMMA:
Features

• i. EMMA can instrument classes for coverage either offline

(before they are loaded) or on the fly (using an instrumenting
application class loader).
• ii. Supported coverage types: class, method, line, basic block.
EMMA can detect when a single source code line is covered
only partially.
• iii. Coverage stats are aggregated at method, class, package,
and "all classes" levels.
• iv. Output report types: plain text, HTML, XML. All report types
support drill-down, to a user-controlled detail depth. The HTML
report supports source code linking.
• v. Output reports can highlight items with coverage levels
below user-provided thresholds.
• vi. Coverage data obtained in different instrumentation or test
runs can be merged together.
 EMMA:
Features

• vii. It is possible to dump or reset coverage data remotely and

without a JVM exit.
• viii. EMMA does not require access to the source code and
degrades gracefully with decreasing amount of debug
information available in the input classes.
• ix. EMMA can instrument individual .class files or entire .jars
(in place, if desired). Efficient coverage subset filtering is
possible, too.
• x. Makefile and ANT build integration are supported on equal
footing.
• xi. EMMA is quite fast: the runtime overhead of added
instrumentation is small (5-20%) and the bytecode
instrumentor itself is very fast (mostly limited by file I/O speed).
Memory overhead is a few hundred bytes per Java class.
• xii. EMMA is 100% pure Java, has no external library
dependencies, and works in any Java 2 JVM (even 1.2.x).
 FindBugs - Find Bugs in Java Programs
• FindBugs, uses static analysis to look for bugs in Java code.
• • It is free software, distributed under the terms of the Lesser GNU Public
License.
• • FindBugs requires JRE (or JDK) 1.5.0 or later to run. However, it can
analyze programs compiled for any version of Java, from 1.0 to 1.8.
• • One of the problems with code quality tools is that they tend to overwhelm
developers with problems that aren't really problems -- that is, false positives.
The creators of FindBugs, David Hovemeyer and William Pugh, were
sensitive to this issue and aimed to reduce the number of false positives
they report.
• • Unlike other static analysis tools, FindBugs doesn't focus on style or
formatting; it specifically tries to find real bugs or potential performance
problems.
• FindBugs is a static analysis tool that examines your class or JAR files
looking for potential problems by matching your bytecodes against a list of
bug patterns.
• • Like most tools these days, you can run FindBugs in multiple ways -- from a
GUI, from a command line, using Ant, as an Eclipse plug-in, and using Maven.
• • With static analysis tools, you can analyze software without actually
running the program. Instead the form or structure of the class files are
analyzed to determine the program's intent, often using the Visitor pattern .
FindBugs - Find Bugs in Java Programs
• Some of the problems that FindBugs can detect.

• i. Detector: Find hash equals mismatch

• This detector finds several related problems, all centered around the implementation
of equals() and hashCode(). These two methods are very important because they're
called by nearly all of the Collections-based classes -- List, Maps, Sets, and so on.

• ii. Detector: Return value of method ignored

• This detector looks for places in your code where the return value of a method is
ignored when it shouldn't be. One of the more common instances of this scenario is
found when invoking String methods

• iii. Detector: Null pointer dereference and redundant comparisons to null

• This detector looks for two types of problems. It looks for cases where a code path
will or could cause a null pointer exception, and it also looks for cases in which there
is a redundant comparison to null. For example, if both of the compared values are
definitely null, they're redundant and may indicate a coding mistake.

• iv. Detector: Field read before being initialized

• This detector finds fields that are read in constructors before they're initialized. This
error is often caused by mistakenly using a field's name instead of a constructor
argument
 FindBugs - Find Bugs in Java Programs
• Latest release of FindBug includes
additional features like :
• i. FindBugs Communal cloud
• ii. checks for updated versions of
FindBugs
• iii. Powerful plugin capabilities
• iv. new bug patterns, including new/
improved support for Guava and JSR v.
improved performance
 FindBugs - Find Bugs in Java
Programs
• • Figure shows the results of analyzing an
anonymous project ( FindBugs UI )
 Cfix - Unit Testing Framework for C and C++

• xUnit testing framework for C/C++, specialized for

Win32 (32/64 bit).
• cfix requires only little effort to author unit tests, which
are compiled and linked into a DLL.
• The test runner application provided by cfix allows
selectively running tests of one or more of such test-
DLLs.
• Execution and behaviour in case of failing test cases
can be highly customized.
• Moreover, cfix has been designed to work well in
conjunction with the Windows Debuggers (Visual
Studio, WinDBG).
• Requirement: Windows
• Cfix supports development of both user and kernel
mode unit tests
COMMERCIAL TESTING TOOLS
 WinRnner
• • It is a tool used for performing functional / regression
testing
• • It is a product of Mercury Interactive
• • It automatically creates the test scripts by recording
the user interactions on GUI of the software. These
scripts can be run repeatedly whenever needed
without any manual intervention.
• • It support the Test Script Language ( TSL, a C-like
language) using which test scripts can be modified.
• • If any problem occurred during automatic testing,
there is provision to bring the application to a new
state.
• It has a default interleaving of one second between
statement execution. If some activities take more time,
then test cases automatically synchronizes by waiting
for the completion of current operation.
 SilkTest

• • It is a product from Segue Software

• • It is used for regression/functional
testing
• • It supports the object-oriented scripting
language 4Test
• • SilkTest has a provision for customized
in-built recovery system which helps in
continuing the automated testing even if
there is some failure in between
 LoadRunner

• It is a product of Mercury Interactive

• • This tool is used for performance and load
testing of a system
• • This tool is helpful for client/server
applications of various parameters with their
actual load like response time, the number of
users etc.
• • The major benefit of this tool is that it
creates virtual users on a single machine and
tests the system on various parameters. Thus
performance and load testing is done with
minimum infrastructure.
 Jmeter
• • This is an open source software tool used for
performance and load testing
• • The Apache JMeter application is open source
software, a 100% pure Java application
designed to load test functional behavior and
measure performance. It was originally designed
for testing Web Applications but has since
expanded to other test functions.
• • Apache JMeter may be used to test
performance both on static and dynamic
resources, Web dynamic applications.
• • It can be used to simulate a heavy load on a
server, group of servers, network or object to test
its strength or to analyze overall performance
under different load types.
 Jmeter
• • Apache JMeter features include: Ability to load
and performance test many different applications/
server/protocol types:
• Web - HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
• SOAP / REST Webservices
• FTP Database via JDBC
• LDAP
• Message-oriented middleware (MOM) via JMS
• Mail - SMTP(S), POP3(S) and IMAP(S)
• Native commands or shell scripts
• TCP
• Java Objects
 Jmeter
• • Full featured Test IDE that allows fast Test Plan recording
(from Browsers or native applications), building and debugging
• • Command-line mode to load test from any Java compatible
OS (Linux, Windows, Mac OSX, …)
• • A complete and ready to present dynamic HTML report
• • Easy correlation through ability to extract data from most
popular response formats, HTML, JSON , XML or any textual
format
• • Complete portability and 100% Java purity.
• • Full multi-threading framework allows concurrent sampling by
many threads and simultaneous sampling of different
functions by separate thread groups.
• • Caching and offline analysis/replaying of test results.
• • Data analysis and visualization plugins allow great
extensibility as well as personalization.
• • Functions can be used to provide dynamic input to a test or
provide data manipulation.
 TestDirector
• • TestDirector is a test management tool.
• • It is a software product from Mercury Interactive
• • It is a web-based tool with the advantage of managing the
testing if two teams are at different locations
• • It manages the test process with four phases :
–Specifying requirements
–Planning tests
–Running tests
–Tracking defects
• • Tests are planned as per the requirements. Once the plan is
ready, test cases are executed.
• • Defect tracking can be done in any phase of test process.
During defect-tracking,
–new bug can be reported
–assign responsibility to someone for bug repair
–Assign priority to the bug
–Bug repair status can be analysed
• • This tool can also be integrated with LoadRunner or WinRunner
 IBM Rational SQA Robot
• It is another powerful tool for
functional/Regression testing
• • Synchronization of test cases with a
default delay of 20 second is also
available