Software Testing

Technology
Chapter 7 Acceptance
Testing

李鹏飞（ Pengfei
Li ）
pengfei.li@hbut.edu.cn
 Chapter 6 Review

① Functional testing and regression

testing
② Performance testing, stress testing,
capacity testing
③ Security testing
④ Reliability/fault tolerance testing
⑤ Compatibility testing
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 What is an acceptance test
Acceptance Test: It is the last phase of technical testing, also
known as delivery testing, and is conducted after the system
functional and non-functional testing of a software product and
before the product is released. 。
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 7.1 Acceptance testing process
and key elements
Prerequisites:
The system or software product has passed the system test

Test Contents:
Testing verifies that the system meets the requirements of
the user requirements specification (which may include
project or product acceptance guidelines), identifies
defects in the software where possible, helps to improve the
software further, and ensures that the system or software
product is ultimately accepted by the user. This includes
usability testing, installation testing, and documentation
(e.g., user manuals) testing.
 Steps
 Develop test plan and acceptance criteria, pass customer review
 Design test cases and pass review
 Prepare test environment and data, execute test cases, record
test results
 Analyze test results, analyze test results according to acceptance
criteria, make acceptance and test evaluation 。
 The test item is passed;
 The test item is not passed, but there are workarounds, which
will be improved in the later stage of maintenance or the
next version;
 The test item is not passed, and there is no workaround,
which requires great revision;
 The test item cannot be evaluated or given a complete
assessment. The reason must be given
 Submit a test report
 Acceptance criteria and
considerations
Completion criteria:
 Each test case in the acceptance test plan is fully
executed
 The bugs found in the acceptance test have been fixed
and passed the test, or have been evaluated for revision
in the next version
 The software acceptance test report has been completed
Precautions ：
 A formal, separate acceptance test report must be written
 The acceptance test must be carried out in the actual user
operating environment
 The user and the test department are jointly performed. If
the company develops its own products, it should be carried
out jointly by testers, product design departments,
 Acceptance Testing in Agile

Regression testing
+BVT Daily
(Build Verification
Test)
Continuous
unit/integration
test
Test requirements
Test missions Test plan

Test cases Acceptance Potentially shippable

test product increment
Product Backlog
(determine priority )
 Acceptance Testing in Agile

 Acceptance testing approves some assumptions about the

story that may not have been discussed with developers
 Acceptance tests provide basic criteria to check that the
story is fully realized
 Acceptance tests should be written by the customer, not
the developer
 Acceptance tests should be written before the program is
written in code
 New acceptance tests should not be written if they do not
help to clarify the details of the story's intent

10
Agile Acceptance Testing –
Validation of User Story
 User Story
Acceptance criteria

① A user cannot submit a form without

completing all the mandatory fields
As a conference ② Information from the form is stored in
attendee, I want to be the registrations database
able to register online,
so I can register quickly ③ Protection against spam is working
and cut down on ④ Payment can be made via credit card
paperwork
⑤ An acknowledgment email is sent to
the user after submitting the form.
BDD : Requirements are tests
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 7.2 Validation of Product
Specification
Review
 Review work from the customer's point of view and
standpoint
 Check the correctness of the applied standards and do not
conflict with industry norms
 Review and study similar products.
 Verify its integrity, accuracy, consistency, rationality and
other characteristics 。
Validation
 Implemented feature marked as passed
 Feature not implemented, bug reported and reflected in
report
 Feature mostly implemented but not consistent with it, bug
reported and reflected in report
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 7.3 User interface and usability
testing
7 elements of a UI:
 Compliance with standards and
norms
 Intuitiveness
 Consistency
 Flexibility
 Comfort
 Correctness
 Usability

Ease of use testing has no specific quantitative metrics and is

highly subjective
 Standards and norms
Usually standards are established and most users
are already familiar with and accept these standards
and norms, or have agreed on what the information
represents, for example:

If the software runs on a particular platform, the

standards and specifications of that platform need to be
used as a supplement to the product specification and
used as a basis for building test cases as well as the
product specification
 Examples

19
 Intuitiveness and consistency

Intuitiveness ：
- Understand the desired functionality or expected
response significantly and where it is expected to
appear.
- Consider whether the organization and layout of
the user interface makes sense 。
Consistency ：
- Including the consistency of the software itself, as
well as the consistency of the software with other
software
 Intuitiveness Examples

21
Intuitiveness Examples
 Flexibility

The user has the flexibility to choose different states and methods to accomplish
the appropriate function. However, flexibility can also develop into complexity, with
too many states and methods making it more difficult for users to understand and
master, and increasing the difficulty of programming and testing. For example:
 Comfort, Correctness, Practicality

Comfort ：
Proper performance, reasonable arrangement, and the ability
to make necessary hints or corrections are factors to
consider, including fault tolerance and performance
Correctness ：
Problems with correctness are generally obvious and
relatively easy to spot
Practicality ：
Usability does not mean whether the software itself is
practical, but only whether the specific features are
practical. The development of large software or the long
cycle of several iterations of software development is
likely to produce some features that are not practical
Comfort examples
 Simplicity

• 1-click
• Next  Next  Next …
Compatibility issues
 What is usability ?

 Easy to discover
 Easy to learn
 Easy to use
 Availability
What is usability ?
Usability Testing Lab
Microsoft's usability testing
World Usability Day

http://www.worldusabilityday.org/
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 7.4 Installability and
recoverability testing

 System software installation

 Application software
installation
 Server installation
 Client installation
 Product upgrade installation
Installability testing
 Uninstall testing

What is the biggest risk of

 Installability Test

 Do the installation instructions have any restrictions or special

requirements for the installation environment?
 Is the process simple and easy to follow?
 Are there obvious and reasonable cue messages during the process?
 Are there any unforeseen or irreparable errors?
 Does the installer take up resources (e.g., ports) that conflict
with the original system?
 Is it possible to exit mid-installation? Is it possible to back
out?
 Can it be safely uninstalled for testing?
 Can the original program run normally after the upgrade
installation?
 Verification of license number and registration number
 Recoverability testing

 Check the fault tolerance of the system. When a system

error occurs, the ability to correct the error or restart
the system within a specified period of time.
 Recovery tests: Compulsory software failures by various
means, followed by verification that the system can recover
as quickly as possible. 。
 For automated recovery, verify that mechanisms such as
reinitialization, checkpointing, data recovery, and
restart are correct;
 For manual recovery systems, estimate the average repair
time and determine if it is within acceptable limits.
 Recoverability testing

 Process

 Component
 Service
 System
 Cluster
 Data Center
Example
 Example （ backup
test ）

 Database Cold Backup on File System

 Vmractest1 and Vmractest2 software backup
 OCR (Oracle Cluster Registry) backups
  Voting Disk backups
  ASM (Automatic Storage Management) Spfile
backups 

Spfile ： Persistent initialization parameter file

 Exmaple （ crash
）

 Database loss

  OCR loss
  Voting Disk loss
  ASM Spfile loss
  Software loss.
 Node crash  

REF: http://blogs.oracle.com/AlejandroVargas/2006/12/rac_asm_crash_and_recovery_tes.html
Example （ Repair ）
 Chapter 7 Acceptance Testing

7.1 Acceptance testing process and

key elements
7.2 Verification of product specification
7.3 User interface and usability testing
7.4 Installability and recoverability
testing
7.5 Documentation testing
 7.5 Document Testing

 Online help files or user manuals;

 guides and wizards;
 installation and setup guides;
 examples and templates;
 error messages;
 images and sounds for demonstrations;
 authorization/registration forms and user license
agreements;
 software packaging, advertising and promotional
materials ；
 ……
 How to do a document test

Good documentation improves the overall quality of

the product by increasing ease of use, improving
reliability, and reducing technical support costs
Checking:
 Correctness
Completeness
Comprehensibility
 α/β Test

α Test: The development company organizes internal staff to

simulate various user lines to test the software product to be released
(called alpha version), trying to find bugs and fix them The software
product adjusted by alpha testing is called beta version

β Test ： We organize external typical users to actually use the beta

version in their daily work, and ask them to report exceptions and
give feedback on their use. The software development company then
corrects and improves the beta version 。

Acceptance test report, also known as Release Report

 Games :
Open Beta Test(OBT)
Post-Test Milestones
Q&A
Thank you