CHAPTER 3

SOFTWARE TESTING
OVERVIEW
BỘ MÔN: CÔNG NGHỆ PHẦN MỀM
KHOA CNTT – ĐẠI HỌC HUFLIT
OBJECTIVE

• Help students understanding and perceiving the software testing concepts

• Understanding & Catching the object, the role, the important of software
testing
• Manupulating the process of Software development and Software Testing
• Catching overview the formality of test, level test, test types…
CONTENTS

• Concepts of testing
• Test requirements
• Objective & role of software testing
• Principles of software testing
• Manual & Automatic testing
• Software development life cycle vs Software testing life cycle
SOFTWARE TESTING CONCEPTS

• Test: Thử làm, thử thực hiện, làm thử

• Check: kiểm tra, rà soát, dò xét
• View: xem, xem xét
• Review: xem lại, kiểm tra lại
• Inspect: thanh tra, xem xét, xét kỹ
• Verify: kiểm chứng
• Validate: xác nhận
TEST REQUIREMENTS ESSENTIALS

The formats can be very different. What they look like is unique to every
company and writer. They are hopefully full of useful information.
• Target Platform
• System Requirements
• A description of the functionality
• Intended Use
• Expected Problem Areas
TEST REQUIREMENTS

• Là tất cả cái gì mà AUT (Application Under Testing) cần được test

• Functional requirements: Kiểm tra yêu cầu chức năng của phần mềm bao
gồm các yêu cầu nghiệp vụ và chức năng chính, phụ của toàn bộ phần
mềm, kể cả các yêu cầu đầu vào và đầu ra của nó.
• Non-functional requirements: các yêu cầu về thuộc tính, tính chất phải có
của phần mềm bao gồm các yêu cầu tiêu chuẩn cần tuân thủ, yêu cầu về
hệ thống, yêu cầu về hiệu suất, dung lượng, cảm nhận, thu hút, sử dụng,
thu hút….
TEST OBJECTIVES (CONT)

• Your objective should not be to verify that the program works correctly because:
• If you can’t test the program completely, you can’t verify that it works correctly.
• The program doesn’t work correctly, so you can’t verify that it does.
• Then as a tester, you fail to reach your goal every time you find an error.
• You’ll be more likely to miss problems than if you want and expect the program to
fail.
→ The Objective of Testing a Program is to Find Problems.
 TEST OBJECTIVES (CONT)

Finding problems is the core of your work. You should want to find as many problems as possible. The
more serious the problem tester find, the better tester is.
• “Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable
happening to the anonymous.” James-Bach
A test that reveals a problem is a success. A test that did not reveal a problem is (often) a waste
of time!
→The Purpose of Finding Problems is to Get Them Fixed.
• The best tester is not the one who finds the most bugs or who embarrasses the most programmers.
• The best tester is the one who gets the most bugs fixed.
BENEFIT OF SOFTWARE TESTING

• Software testing has now become a part of programming

• Improve Your Software
• Cost of fixing the bug can be reduced
• Helps to avoid dangerous situations
• Quality assurance
PRINCIPLES OF SOFTWARE TESTING

1. Testing shows presence of defects

2. Exhaustive testing is impossible
3. Early testing
4. Defect clustering
5. Pesticide paradox
6. Testing is context dependent
7. Absence of error – fallacy
PRINCIPLES OF SOFTWARE TESTING(CONT)

1.Testing Shows Presence of Defects:

• Testing shows the presence of defects in the software. The goal of
testing is to make the software fail. Sufficient testing reduces the
presence of defects. In case testers are unable to find defects after
repeated regression testing doesn’t mean that the software is bug-free.
• Testing talks about the presence of defects and don’t talk about the
absence of defects.
PRINCIPLES OF SOFTWARE TESTING(CONT)

2. Exhaustive Testing is Impossible:

• Testing all the functionalities using all valid and invalid inputs and
preconditions is known as Exhaustive testing.
• Why it’s impossible to achieve Exhaustive Testing?
3. Early Testing:
• Defects detected in early phases of SDLC are less expensive to fix. So conducting
early testing reduces the cost of fixing defects.
 PRINCIPLES OF SOFTWARE TESTING(CONT)

4. Defect Clustering:
• Defect Clustering in software testing means that a small module or functionality contains most of
the bugs or it has the most operational failures.
• As per the Pareto Principle (80-20 Rule), 80% of issues comes from 20% of modules and remaining
20% of issues from remaining 80% of modules. So we do emphasize testing on the 20% of modules
where we face 80% of bugs.

5. Pesticide Paradox:
• Pesticide Paradox in software testing is the process of repeating the same test cases again and again,
eventually, the same test cases will no longer find new bugs. So to overcome this Pesticide Paradox,
it is necessary to review the test cases regularly and add or update them to find more defects.
 PRINCIPLES OF SOFTWARE TESTING(CONT)
6.Testing is Context Dependent:
• Testing approach depends on the context of the software we develop. We do test the software
differently in different contexts. For example, online banking application requires a different
approach of testing compared to an e-commerce site.

7. Absence of Error – Fallacy:

• 99% of bug-free software may still be unusable, if wrong requirements were incorporated into the
software and the software is not addressing the business needs.
• The software which we built not only be a 99% bug-free software but also it must fulfill the
business needs otherwise it will become an unusable software.
• These are the seven principles of Software Testing every professional tester should know.
MANUAL TESTING

MANUAL TESTING is a type of Software Testing where Testers manually execute test cases
without using any automation tools. Manual Testing is the most primitive of all testing types and
helps find bugs in the software system
Manual testing is the most primary of all testing types and help find bugs in the software testing
Any new application must be manually tested before its testing can be automated. Manual Testing
requires more effort but is necessary to check automation feasibility.
Manual Testing does not require knowledge of any testing tool.
• One of the Software Testing Fundamental is "100% Automation is not possible".
• This makes Manual Testing imperative.
 MANUAL TESTING(CONT)

Goal of Manual Testing

• The key concept of manual testing is to ensure that the application is error free and it is working in
conformance to the specified functional requirements.
• Test Suites or cases, are designed during the testing phase and should have 100% test coverage.
• It also makes sure that reported defects are fixed by developers and re-testing has been performed
by testers on the fixed defects.
• Basically, this testing checks the quality of the system and delivers bug-free product to the customer.
MANUAL TESTING(CONT)

Types of Manual Testing:

In fact, any type of software testing type
can be executed both manually as well
using an automation tool.
 MANUAL TESTING(CONT)

How to perform Manual Testing

1. Read and understand the software project documentation/guides. Also, study the
Application Under Test (AUT) if available.
2. Draft Test cases that cover all the requirements mentioned in the documentation.
3. Review and baseline the test cases with Team Lead, Client (as applicable)
4. Execute the test cases on the AUT
5. Report bugs.
6. Once bugs are fixed, again execute the failing test cases to verify they pass.
 MANUAL TESTING(CONT)

Myths of Manual Testing

Myth: Anyone can do manual testing
Fact: Testing requires many skill sets
Myth: Testing ensures 100% defects free product
Fact: Testing attempts to find as many defects as possible. Identifying all possible defects is impossible.
Myth: Automated testing is more powerful than manual testing
Fact: 100% test automation cannot be done. Manual Testing is also essential
Myth: Testing is easy
Fact: Testing can be extremely challenging. Testing an application for possible use cases with minimum test
cases requires high analytical skills.
AUTOMATION TESTING
• AUTOMATION TESTING means using an automation tool to execute your test case
suite. On the contrary, Manual Testing is performed by a human sitting in front of a
computer carefully executing the test steps.
• The automation software can also enter test data into the System Under Test, compare
expected and actual results and generate detailed test reports. Test Automation demands
considerable investments of money and resources.
• Successive development cycles will require execution of same test suite repeatedly. Using
a test automation tool, it's possible to record this test suite and re-play it as required.
Once the test suite is automated, no human intervention is required. This improved ROI
of Test Automation. The goal of Automation is to reduce the number of test cases to be
run manually and not to eliminate Manual Testing altogether.
AUTOMATION TESTING (CONT)

Goal of Automation testing

• Reduce the number of testcases to be run manually and not emilinate manual testing all
together
• Manual Testing of all workflows, all fields, all negative scenarios is time and money consuming
• It is difficult to test for multilingual sites manually
• Automation does not require Human intervention. You can run automated test unattended
(overnight)
• Automation increases the speed of test execution
• Automation helps increase Test Coverage
• Manual Testing can become boring and hence error-prone.
AUTOMATION TESTING(CONT)
Which Test Cases to Automate?
Test cases to be automated can be selected using the following criterion to increase the
automation ROI
• High Risk - Business Critical test cases
• Test cases that are repeatedly executed
• Test Cases that are very tedious or difficult to perform manually
• Test Cases which are time-consuming
The following category of test cases are not suitable for automation:
• Test Cases that are newly designed and not executed manually at least once
• Test Cases for which the requirements are frequently changing
• Test cases which are executed on an ad-hoc basis
AUTOMATION TESTING(CONT)
Automated Testing Process:
 AUTOMATION TESTING(CONT)
Framework for Automation
A framework is set of automation guidelines which help in
• Maintaining consistency of Testing
• Improves test structuring
• Minimum usage of code
• Less Maintenance of code
• Improve re-usability
• Non Technical testers can be involved in code
• The training period of using the tool can be reduced
• Involves Data wherever appropriate
AUTOMATION TESTING(CONT)

Framework for Automation

There are four types of frameworks used in automation software testing:
• Data Driven Automation Framework
• Keyword Driven Automation Framework
• Modular Automation Framework
• Hybrid Automation Framewoark
 AUTOMATION TESTING(CONT)
Benefits of automation testing

• 70% faster than the manual • Increases Efficiency

testing
• Better speed in executing tests
• Wider test coverage of
application features • Re-usable test scripts

• Reliable in results • Test Frequently and thoroughly

• Ensure Consistency
• Saves Time and Cost
• Improves accuracy
• Human Intervention is not
required while execution
• More cycle of execution can be
achieved through automation
• Early time to market
AUTOMATION TESTING(CONT)
Different types of software testing that can be automated
• Smoke Testing
• Unit Testing
• Integration Testing
• Functional Testing
• Keyword Testing
• Regression Testing
• Data Driven Testing
• Black Box Testing
AUTOMATION TESTING(CONT)
How to Choose an Automation Tool?
Selecting the right tool can be a tricky task. Following criterion will help you select the best
tool for your requirement.
Environment Support Scripting Language Used
Ease of use Support for various types of test - including functional,
test management, mobile, etc...
Testing of Database Support for multiple testing frameworks
Object identification Easy to debug the automation software scripts
Image Testing Ability to recognize objects in any environment
Error Recovery Testing Extensive test reports and results
Object Mapping Minimize training cost of selected tools
TESTING METHODS

1. Static Testing: It is also known as Verification in Software Testing.Verification is a

static method of checking documents and files.Verification is the process, to ensure
that whether we are building the product right i.e., to verify the requirements which
we have and to verify whether we are developing the product accordingly or not.
→ Activities involved here are Inspections, Reviews, Walkthroughs
2. Dynamic Testing: It is also known as Validation in Software Testing.Validation is a
dynamic process of testing the real product.Validation is the process, whether we are
building the right product i.e., to validate the product which we have developed is right
or not.
→ Activities involved in this is Testing the software application
TESTING LEVELS

• Both developers and testers work together to release a high-quality product

• Team have to use various levels of testing in the process of releasing a quality
product
• Each of these levels of software testing has a specific purpose.
• 1. Unit Testing
2. Integration Testing
3. System Testing
4. Acceptance Testing
TESTING LEVELS(CONT)
 TESTING LEVELS(CONT)

• Unit testing
• Unit Testing is done to check whether the individual modules of the source code are working
properly. i.e. testing each and every unit of the application separately by the developer in the
developer’s environment. It is Module Testing or Component Testing
• Ex:
- line of code testing
- Function or procedure testing
- Classes testing
..
TESTING LEVELS(CONT)
Benefits Of Unit Testing
1. The process becomes agile
2. Code quality improves
3. Detects bugs early
4. Easier changes and simplified integrations
5. Documentation availability
6. Easy debugging process
7. Lower cost
8. Code completeness can be demonstrated using unit tests
9. Saves development time:
10. Code coverage can be measured
TESTING LEVELS(CONT)
Unit Testing Cycle
TESTING LEVELS(CONT)
How to write good Unit Tests?
• A Unit test should be written to verify a single unit of code and not the integration.
• Small and isolated Unit tests with clear naming would make it very easy to write and
maintain.
• Changing another part of the software should not affect the Unit test if those are
isolated and written for a specific unit of code.
• It should run quickly
• A Unit test should be reusable
TESTING LEVELS(CONT)
Unit Testing Frameworks
Unit Testing frameworks are mostly used to help write unit tests quickly and easily. Most of
the programming languages do not support unit testing with the inbuilt compiler. Third-
party open source and commercial tools can be used to make unit testing even more fun.
List of popular Unit Testing tools for different programming languages
1. Java framework – JUnit
2. PHP framework – PHPUnit
3. C++ frameworks – UnitTest++ and Google C++
4. .NET framework – NUnit
5. Python framework – py.test
TESTING LEVELS(CONT)

Integration testing
• Integration Testing is the process of testing the connectivity or data transfer between a couple
of unit tested modules. It is I&T Testing or String Testing
• The meaning of Integration testing is quite straightforward- Integrate/combine the unit tested
module one by one and test the behavior as a combined unit.
• The main function or goal of this testing is to test the interfaces between the units/modules.
• The individual modules are first tested in isolation. Once the modules are unit tested, they are
integrated one by one, till all the modules are integrated, to check the combinational behavior,
and validate whether the requirements are implemented correctly or not.
TESTING LEVELS(CONT)
Challenges
1. Integration testing means testing two or more integrated systems in order to ensure that the system
works properly. Not only the integration links should be tested but an exhaustive testing considering
the environment should be done to ensure that the integrated system works properly. There might be
different paths and permutations which can be applied to test the integrated system.
2. Managing Integration testing becomes complex because of few factors involved in it like the database,
Platform, environment etc.
3. While integrating any new system with the legacy system, it requires a lot of changes and testing
efforts. Same applies while integrating any two legacy systems.
4. Integrating two different systems developed by two different companies is a big challenge as for how
one of the systems will impact the other system if any changes are done in any one of the systems is
not sure.
TESTING LEVELS(CONT)
Types of Integration Testing
1. Big Bang Approach:
Big bang approach integrates all the modules in one
go i.e. it does not go for integrating the modules
one by one.
• Big bang approach is a time-consuming process
of finding a module which has a defect itself as
that would take time and once the defect is
detected, fixing the same would cost high as the
defect is detected at the later stage.
TESTING LEVELS(CONT)
Big Bang approach (cont)
• Advantages of Big Bang approach:
It is a good approach for small systems.
• Disadvantages of Big Bang Approach:
• It is difficult to detect the module which is causing an issue.
• Big Bang approach requires all the modules all together for testing, which in turn, leads to less
time for testing as designing, development, Integration would take most of the time.
• Testing takes place at once only which thereby leaves no time for critical module testing in
isolation.
TESTING LEVELS(CONT)
Top-down approach
• Testing takes place from top
to bottom. High-level modules
are tested first and then low-
level modules and finally
integrating the low-level
modules to a high level to
ensure the system is working
as intended.
• Stubs are used as temporary
module if a module is not
ready for integration testing.
TESTING LEVELS(CONT)
Bottom-up approach
• Testing takes place from
bottom to up. Lowest
level modules are tested
first and then high-level
modules and finally
integrating the high-level
modules to a low level
to ensure the system is
working as intended.
• Drivers are used as a
temporary module for
integration testing.
TESTING LEVELS(CONT)
Hybrid Integration Testing?
• Hybrid integration testing is also known as Sandwich integration testing. It
is the combination of both Top-down and Bottom-up integration testing..
TESTING LEVELS(CONT)
System Testing/ End-to-End testing
• The process of testing an integrated hardware and
software system to verify that the system meets its
specified requirements.
• Test the entire software product from beginning to
end. The purpose of this testing type is to ensure
that the application flow behaves as expected. In
other words, it helps define the product’s system
dependencies and ensures all integrated pieces
work together as expected.
TESTING LEVELS(CONT)
Why System Testing?(cont)
1. It is very important to complete a full test cycle and ST is the stage where it is done.
2. ST is performed in an environment that is similar to the production environment and
hence stakeholders can get a good idea of the user's reaction.
3. It helps to minimize after-deployment troubleshooting and support calls.
4. In this STLC stage Application Architecture and Business requirements, both are tested.
This testing is very important and it plays a significant role in delivering a quality product to
the customer.
TESTING LEVELS(CONT)
Test types?(cont)
• Black box Testing technique does not require internal knowledge of the code whereas the
white box technique requires internal knowledge of the code.
• While performing System testing functional & non-functional, security, Performance and
many other testing types are covered and they are tested using a black-box technique
wherein the input is provided to the system and the output is verified. System internal
knowledge is not required.
TESTING LEVELS(CONT)
Test types?(cont)
TESTING LEVELS(CONT)
Test types?(cont)
TESTING LEVELS(CONT)
Test types?(cont)
• Functionality Testing: To make sure that functionality of the product is working as per the
requirements defined, within the capabilities of the system.
• Recoverability Testing: To make sure how well the system recovers from various input errors and
other failure situations.
• Interoperability Testing: To make sure whether the system can operate well with third-party products
or not.
• Performance Testing: To make sure the system’s performance under the various condition, in terms of
performance characteristics.
• Scalability Testing: To make sure the system’s scaling abilities in various terms like user scaling,
geographic scaling, and resource scaling.
• Reliability Testing: To make sure the system can be operated for a longer duration without developing
failures.
TESTING LEVELS(CONT)
Test types?(cont)
• Regression Testing: To make sure the system’s stability as it passes through an
integration of different subsystems and maintenance tasks.
• Documentation Testing: To make sure that the system’s user guide and other help
topics documents are correct and usable.
• Security Testing: To make sure that the system does not allow unauthorized access to
data and resources.
• Usability Testing: To make sure that the system is easy to use, learn and operate.
TESTING LEVELS(CONT)
Test types?(cont)
Graphical User Interface Testing (GUI):
• GUI testing is done to verify if the GUI of a system works as expected or not. GUI is basically what is
visible to a user while he uses the application. GUI testing involves testing buttons, icons, checkboxes, List
box, Textbox, menus, toolbars, dialog boxes, etc.
Compatibility Testing:
• Compatibility testing is done to ensure that the developed product is compatible with different browsers,
Hardware Platforms, Operating System and databases as per the requirement document.
Exception Handling:
• Exception Handling Testing is performed to verify that even if an unexpected error occurs in the product,
it should show the correct error message and does not let the application stop. It handles the exception
in a way that the error is shown meanwhile the product recovers and allows the system to process the
incorrect transaction.
TESTING LEVELS(CONT)
Test types?(cont)
Volume Testing:
• Volume Testing is a type of non-functional testing wherein testing is done using a huge amount of
data. For Example, the Volume of data is increased in the database to verify the system performance.
Stress Testing:
• Stress Testing is done by increasing the number of users (at the same time) on an application to an extent
that the application breaks down. This is done to verify the point at which the application will break down.
Sanity Testing:
• Sanity Testing is performed when the build is released with a change in the code or functionality or if any
bug has been fixed. It verifies that the changes done have not affected the code and no other issue has
occurred because of that and the system works as previously.
TESTING LEVELS(CONT)
Test types?(cont)
Smoke Testing:
• Smoke Testing is a testing that is performed on the build to verify if the build is further
testable or not. It verifies that the build is stable to test and all the critical functionalities
are working fine. Smoke testing is done for the complete system i.e. end to end testing is
done.
Exploratory Testing:
• Exploratory Testing as the name itself suggests it is all about exploring the application. No
scripted testing is performed in exploratory testing. Test cases are written along with the
testing. It focuses more on execution than planning.
TESTING LEVELS(CONT)
Test types?(cont)
Adhoc Testing:
• Adhoc Testing is informal testing where no documentation or planning is done to test the
application. Tester tests the application without any test cases. The aim of a tester is to
break the application. The tester uses his experience, guess and intuition to find the critical
issues in the application.
Installation Testing:
• Installation Testing is to verify if the software gets installed without any issues.
TESTING LEVELS(CONT)
Process of system test?(cont)
• The very first step is to create a Test Plan.
• Create System Test Cases and test scripts.
• Prepare the test data required for this testing.
• Execute the system test cases and script.
• Report the bugs. Re-testing the bugs once fixed.
• Regression testing to verify the impact of the
change in the code.
• Repetition of the testing cycle until the system is
ready to be deployed.
• Sign off from the testing team.
TESTING LEVELS(CONT)
The points stated below are covered in this testing:(cont)
• End to End testing which includes verifying the interaction between all the components and
along with the external peripherals to ensure if the system works fine in any of the scenarios
is covered in this testing.
• It verifies that the input provided to the system provides the expected result.
• It verifies if all the functional & non–functional requirements are tested and if they work as
expected or not.
• Ad-hoc and exploratory testing can be performed in this testing after scripted testing has
been completed. Exploratory testing and ad-hoc testing helps to unfold the bugs which cannot
be found in scripted testing as it gives freedom to the testers to test as their desire is based
on their experience and intuition.
TESTING LEVELS(CONT)
Entry/Exit Criteria(cont)
Entry Criteria:
• The system should have passed the exit criteria of Integration testing i.e. all the test cases
should have been executed and there should be no critical or Priority P1, a P2 bug in an open
state.
• Test Plan for this testing should be approved & signed off.
• Test cases/scenarios should be ready to be executed.
• Test scripts should be ready to be executed.
• All the non–functional requirements should be available and test cases for the same should
have been created.
• The testing environment should be ready.
TESTING LEVELS(CONT)
Entry/Exit Criteria(cont)
Exit Criteria:
• All the test cases should be executed.
• No critical or Priority or security-related bugs should be in an open state.
• If any medium or low priority bugs are in an open state, then it should be implemented
with the acceptance of the customer.
• Exit Report should be submitted.
ACCPETANCE TEST
• To obtain customer sign-off so that software can be delivered and payments received.
• Types of Acceptance Testing are Alpha, Beta & Gamma Testing.
• Alpha Testing:
• Alpha testing is mostly like performing usability testing which is done by the in-house developers who
developed the software. Sometimes this alpha testing is done by the client or outsiders with the
presence of developers or testers.
• Beta Testing:
• Beta testing is done by a limited number of end users before delivery, the change request would be fixed
if the user gives feedback or reports defect.
• Gamma Testing:
• Gamma testing is done when the software is ready for release with specified requirements; this testing is
done directly by skipping all the in-house testing activities.