Testing Applications on the

Web

References

1.

Hung Q. Nguyen. (2001). Testing Applications on The Web:

Test Planning for Internet Based Systems. Wiley.

2.

Watkins, J.(2001). Testing IT : An Of-the-Shelf Software

Testing Process, Cambridge University Press.

Objectives

At the end of this lecture, students

should be able to:
1.
Discuss challenges in web based
testing
2.
Elaborate web based testing
techniques
3.
Calculate TPS and throughput of web
sites transactions
4.
Discuss testing tools available in the
market to enhance web testing

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

Challenges in Web Application Testing

More complex environment compared to standalone and typical client server environment.
Web tester need to understand the architecture and
implementation of the network.

Much of what appears to be part of the web

application belongs to complex third-party
products.

For example, browser, a Java interpreter, graphics display

and audio playback programs. If we dont have a clear
separation between application and the third party products,
we may face serious risk of wasting time on testing the
wrong components.

Web applications comprise so many bits and pieces

of information that communicates through all the
components in web environment.
We need knowledge of this components in order to enhance
the testability of the application

5
Reading : Reference 1

Challenges in Web Application Testing

Rapidly

evolving technologies

ASP, JSP, PHP, ASP.NET. Web tester need to

improve testing techniques. Lack of research and
technical knowledge towards the evolving
technologies.
Testers need to be familiar with : test types,
testing issues, common software errors, risks that
are specific to web applications.
Need

for standard software quality

implementation

CMM, ISO have been available, but companies rely

on their skilled developers and testing staff fear
applying it might slow down devt process.

6
Reading : Reference 1

Web Tester Qualities

1.
2.
3.
4.
5.
6.

Good software testing skill

Knowledge of the application
Knowledge of web technologies
Understanding types of tests and their
applicability to web applications
Common error
Knowledge of tools and their
applicability

Client-Side Application
Windows 16 bit

Windows 32 bit

Macintosh client

Client
Application

SERVER

HTML
Contents

IE

Nets
Nav

Opera

Browser vendors
are responsible
for producing
platform specific
browser

Web System Architecture

WEB Browser
IE
Mosaic
Opera
Netscape
OmniWeb

Firewall

Web Server
MSIIS
Apache
etc

Intrane
Intrane
t/t/
Interne
Interne
tt

OS
Linux
Macintosh
Unix
Windows

N/work Traffic
HTTP, HTTPS
FTP
TCP/IP
ODBC
etc

Application Server
ASP
CORBA
Mic DCOM
Allaire
etc

Refer: Reference 1,Page 23.

DB Serve
Oracle
SQL
etc

Web System Architecture

Each variations of software and hardware may cause
problems. Devt usually is tailored to a specific
environment.
Hardware mix

Software mix

Environment can become difficult to control

Web system have the capacity to use machines of
different platforms.
Might include unix server in conjunction with other types
of server (windows based or mac based)
Also might include mixture models from same
platform(server or client side)

Multiple
Multiple
Multiple
Multiple
Multiple

Operating system
software packages
software components
server types, brands and models
browser brands and versions

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

Testing Methods

Task Oriented
Functional
Test(TOFT)
Forced Error Test
(FET)
Boundary Test
System Level Test
Real World User
Level Test
Exploratory Test
Load/Volume Test
Stress Test
Performance test

Regression Test
Compatibility and

Configuration Test
Documentation Test
Online Help test
Install/Uninstall test
User Interface Test
Security Test
Unit Test

1. Functional Tests
Broad

category of testing.
Including :

Functional Acceptance Simple Test(FAST)

Task Oriented Functional Test(TOFT)-can the
application do something useful as expected?
Forced Error Test (FET)- what happens when
an error occur?
Boundary Test-What happens at the edge of
specified use?
Exploratory Test-what does experience tell
about the potential problematic areas in the
application?
Software attacks-why does software fails?

A. Functional Acceptance Simple Test(FAST)

Exercises the lowest level of functionality

for each command.
Objective: to check for the appropriate
behaviors of UI controls based on the
intended design
Including:

Links content links, thumbnail links, image

links, image map links, broken links etc
Controlsnext, previous, zoom, refresh etc
Action command add, remove, update,
create user profiles, data entry test.
Missing images, wrong images
Invalid email link

B. Task-Oriented Functional Test(TOFT)

Objective: to check whether application can do useful tasks

correctly
Check program functions by comparing the results of
performed task with its specifications and reqs document.

C. Forced-Error Test (FET)

Drive application into error condition

Eg : input validation
Check that the error condition is properly
handled.

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

2.User
Interface
Design
Tests

Evaluates how well a design takes care of its

users, offers clear direction, delivers feedback
and maintains consistency of language and
approach.
Understand who is the applications target user?
What design approach has been employed?
Issues :

Ease of use
Look and feel
Navigation
Natural flow
Usability
Commands
accessibility

1. Investigate Target User

Criteria

to consider:

Computer experience
How

long has been using computer?

Do they use computer professionally or casually at
home?
What activities they involved in?
What about experience in specific application eg
spreadsheet, word processors, presentation prog,
drawing prog etc

Web experience
How

long has been using Web system?

Do they have experience in using browser?
Are they familiar with Internet jargon eg HTML, Java
applet, server etc?
Are they familiar with helper application eg Acrobat
Reader,FTP, streaming audio/video?

Target User
Criteria

to consider:

Domain Knowledge
Do

the user familiar with the domain of the

application?
Eg:To test financial info system, we need tester with
domain in the application
Novice users want elementary tutorials, expert
users want efficient utilities.

Application Specific Experience

Is

this the first release of the product? Or there is

an existing product before?
Are the other products in the market have the same
design approach and functionality?

2.
Considering
the
Design

Different application types and target user

require different designs.
Topics to consider:

Design approach

User interaction

Cognitive bridges that help users understand the logic

of UI flow by relating them to experiences in the real
world. Eg: Microsoft Word document-based metaphor.
Usually it is not testers role to judge which design is
best.
Our job is to point out inconsistency in the
implementation of UI.
Common issues:
Refer Table Common Issues in UI Design
User manipulation through keyboard and mouse event.

2. Considering
the Design
UI controls Button, Text area, radio button,

Data presentation

check box button, text field, hyperlinks, hypertext

etc
Dynamic UI Controls Scripts (java script,
VBSCript), Java, ActiveX, Server-side Includes
(cgi, php, asp), Style sheets
Common errors : Refer
Table Errors in User Interaction
Navigation Refer
Table Navigation Issues in UI design
Interface elements can be used to deliver various
types of information that can be created in HTML
pages.
Issuess: Refer Table Misc UI Elements

3. Testing Consideration
Appendix

Display compatibility Test-Matrix

(p. 393)
Browser OS Configuration Matrix (p. 394)

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

3. Database Tests
Two common types of database testing are:

Data integrity

1.

Output errors

2.

Caused by bugs in the data retrieving and manipulating instructions,

although the source data is correct.

Errors may include:

Check data is not corrupted

Schemas are correct

Failures in connecting to the database invalid user name,

password, both; inaccessibility privileges; wrong platform
Errors in manipulating databaseunable to write to specific
location, fail to create files, inadequate storage etc.
Error in Instruction use reserved keyword, missing keyword,
wrong data type, use instead of , value too large, timeout,
undefined table, field etc, call wrong stored procedure.

Methods in Database Tests

1.

White-Box methods

Code walk through

2.

Black-Box methods

Redundancy Coding Error

Inefficiency Coding Error
Testing triggers
External interfacing

Test case design

Preparation for database testing
Setup/installation issues

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

4. Help Tests
Objectives:

Types of help systems

Application-help systems
Reference-help systems
Tutorial-help systems
Sales and marketing-help systems

Apply UI Testing methodologies

Technologies:

Testing the system as a stand alone component

Testing the interaction of the system with the application

Standard HTML (WebHelp, RoboHELP)

Java Applets (JavaHelp)
Netscape NetHelp(NetHelp)
NetHelp is an open standards-based platform for creating and
viewing HTML-based online help.

Help Elements to Test

navigational

features, including a dynamic

table of contents and a multi-level index , links
full-text search
Glossary
Favourites
Other media video, audio, images
Pop up
Text expansion for further info
UI components
3

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

5. Installation Tests
Effective installation testing requires solid
knowledge of the OS and the environment on
which the installer will be running and the
application will be installed.
Objectives:

Test the functionality and UI of the installer

Test the software and hardware compatibility
Test the configurations and modifications that the installer
makes to existing files and registry entries
Test the uninstall program removes all data files?
If your company markets multiple products with independent
installers, test for installer compatibility with other products
can both be installed properly? Can you uninstall a product
without affecting the other?

Handouts : Nguyen, page 259

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

6. Configuration and Compatibility Tests

To ensure that the product will continue to work as
expected in all supported user environments
Configuration testing:

Compatibility testing:

testing of various supported server software and

hardware setups, browser settings, network connections,
TCP/IP stack setup etc
Goal : to ensure application is thoroughly exercised with
as many configuration as possible
It is expected that the features will operate as reasonable
user would expect
Determine which manufacturers and server brands, under
the same configuration are compatible with the web
system

Configuration and Compatibility Testing

Approach

Analyze the market share

Analyze the software on both client and server side
Analyze ways in which the server generates contents
and in which browser interprets them, then format and
display them
Save time
Improve efficiency
Share your experience
How do software and hardware components interact
with each other?

Refer page Nguyen, 267-268

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

7. Performance, Load and Stress Tests

To ensure that your system can endure the
potential load due to business demands at the
same time and can serve your user in a reliable
and timely manner
Performance testing:

Load testing:

Develop effective enhancement strategies for maintaining

acceptable system performance
Gather info and analyze process in which measurement
data are collected to predict when load levels will exhaust
system resource
Evaluates how system performance with predefined load
level.
Measures how long it takes a system to perform various
program tasks and functions in the time limit
Set minimum configuration and maximum activity levels
before testing.

7. Performance, Load and Stress Tests

Stress testing:

Evaluates the behavior of the systems that are pushed beyond

their specified operational limits.

Case study on performance testing:

Suppose your company e-commerce site currently handles

300,000 transactions per day.

=[noOfTransactions / (24 hoursPerDay * 60 minPerHour*60 secPerMinute)]

=[300,000 / (24 * 60 *60)]
= 3.47 Transaction Per Second (TPS)

1.
2.
3.
4.

Suppose a market study findings show that:

The TPS is acceptable level as long as not exceed 4 sec
If 4<TPS<9, 30% of users cancel transactions
If 8<TPS<10, 60% of users cancel transactions
If TPS > 10 seconds, over 90% of users cancel transactions

Suppose in the next 6 months, the no of transactions is

expected to rise between 25-75% from the current level
and the potential revenue for each transaction is RM1.
Mgmt would like to know how the performance would
impact company revenue as the no of transaction per day
increases.
A performance test is conducted and findings show that
system cannot handle such increases in traffic without
increasing response time; otherwise users will cancel
transactions or failures result.
If no of transactions increases as expected, company will
face potential revenue loss.

7. Performance, Load and Stress Tests

Example on simple TPS calculation:

Assume that Mobile88.com site currently handles 115,000

transactions per day. Calculate TPS for the site.
=[noOfTransactions / (24 hoursPerDay * 60
minPerHour*60 secPerMinute)]
=[115,000 / (24 * 60 *60)]
= 1.33 Transaction Per Second (TPS)

Case Study : Workload

Workload:

the amount of processing and

traffic management that is demanded of
the system
Consider:

users (max no of concurrent users,length of

user sessions, amount of unique activity)
application (can be expressed in terms of
service rates such as TPS, throughput, Kbps)
and
resources (system resources need to handle
workload while preserving acceptable
performance ; usually in response time)

Detailed TPS Calculation

Formula:

User
type

Testing

User type x Total users x frequency per session

time in second
%
User
User
Freq
Application Act
Eg: Sessio

60%

n
length

activit
y

sub act

per
sessio
n

60 min

Reporti
ng

Reportin
g

Respond to a
report submission
by displaying the
next blank report
form

TPS = 60% x 10,000 x 1

60 min x 60 sec

= 1.7 (assumption no of user 10,000)

TPS

1.7

Exercise : TPS Calculation

Formula:

User
type

Devt

User type x Total users x frequency per session

time in second
%
User
User
Freq
Application Act
Eg: Sessi

30%

on
lengt
h

activity

sub act

per
sessio
n

30
min

Working
with
existing
reports

Simple
query

Respond by
displaying the
result in a singlereport view

TPS

??

Exercise : TPS Calculation

Formula:

User
type

Devt

User type x Total users x frequency per session

time in second
%
User
User
Freq
Application Act
Eg: Sessi

30%

on
lengt
h

activity

sub act

per
sessio
n

30
min

Working
with
existing
reports

Simple
query

TPS

Respond by
displaying the
result in a singlereport view

TPS = 30% x 10,000 x 8

30 min x 60 sec

= 13.3 (assumption no of user 10,000)

??

Throughput Calculation

Determine what level of bandwidth is required to support

system workload.
For eg: a web server supporting 10,000 concurrent users
who request docs from a pool of 10 different HTML docs
(avg size 2K each) every 3.5 minutes

Throughput = Total user x (SizeOfFileInBit)

timeInSecond
Eg:
Throughput = 10,000 x (2 x 1024 bytes x 8 bitPerByte)
3.5 min x 60 sec

= 780, 190 bitsPerSecond (bps)

To handle this load, we need to specify the most suitable

network connection. Eg T1 1,544,000 bps. (Refer Table
16.6 Bandwidth conversion)

:: T O P I C ::
1.
2.
3.

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

Web Testing Tool

Link Checkers and HTML Validators
Java, C++, Other programming and Scripting
Language
Load/Performance Testing Tools
Automated GUI Functional and Regression Testing
Tools
Runtime Error Detector
Web Security Testing Tools
Java Specific Testing Tools
Database Testing Tools
Help authoring tools
Use the most applicable or combination of tools for
your system.

Tool : NetMechanic

1.
2.
3.
4.
5.

Website : http://www.netmechanic.com
Features :
Link Check
HTML Check & Repair
Browser Compatibility
Load Time
Spell Check

People use different browsers

Close to 17 Million people use

something other than Internet
Explorer when cruising the Internet.
Different Browsers Break Pages.
If you aren't checking your pages
under different browsers and
computers combinations, you are
potentially writing off business with
anyone not using your specific
computer configuration.
Why? Look at the picture 2. This
page looked fine under Internet
Explorer, but a simple HTML error
kept it from opening in Netscape.

Pic 2

Pic 1

People use different computers

PCs may dominate the

market, but there's over 5
Million people who use the
Internet via a different
platform. Apples still make up
12% of the market. WebTV is
also growing in popularity,
especially among the elderly
and disabled - two groups
who are big consumers on
the Internet.
Different Computers Break
Pages. Compare Picture 3 this is the same page as it
displays on a PC and Mac.

Pic 3

People use different versions of browsers

Even different browser versions

will display your pages differently.
And newer versions aren't always
better. In the case below, Netscape
6 couldn't handle the DHTML even
though Netscape 4 rendered the
page perfectly.

Pic 4

People use different screen sizes

A different screen size can

dramatically change the way your
page appears. The webmaster built
the page below to look great on a
1024 x 768 screen size.
Unfortunately the important map on
the right of the page is off the
screen when viewed on a smaller
screen size.

Pic 5

How to resolve these problems?

Before any webmaster publishes a Web page, that page should

be checked on all the major browsers and computers.
DO COMPATIBILITY TEST
Failure to create a compatible Web page can cause visitors to
leave your site - never to return.
More ? Read
http://www.netmechanic.com/browser-photo/tutorial.htm

1.
2.
3.

CONCLUSION

Introduction
Web Testing vs Traditional Testing
Testing Practices

4.
5.

Functional Test
User Interface Test
Database Test
Help Test
Installation Test
Configuration and Compatibility Test
Security Test
Performance, Load and Stress Test

Web Testing Tools

Conclusion

Common issues in UI design

1.

UI components supported by HTML should be compared with those

available on Microsoft Windows or Macintosh OS.

2.

If you as a tester do not understand the UI, chances are its a UI

error.

3.

UI was designed for the end-user, not for designers or developers.

4.

The important features are misunderstood or hard to find.

5.

Users are forced to think in terms of the design metaphor from

designers perspective.

6.

Different terms were used to describe the same functionality.

UI design Approach

1.

Is the design of the application for the target user?

2.

Is the UI intuitive? (you dont have to think too much to figure out
how to use it)

3.

Is the design consistently applied throughout the application?

4.

Does the interface keep the user in control, rather than reacting to
unexpected UI events?

5.

Does the interface offer pleasing visual design?

6.

Is the interface simple to use and understand?

7.

Is help available?

8.

Will usability test be performed on the application under test?

Back to slide

Common Errors in User Interaction

1.

Poor choice of default state or default input value.

2.

Updated value is incorrect.

3.

Dialog box too big.

4.

Invalid inputs and outputs are not detected.

5.

If your application uses script, css, applets, the user might have
disabled one or more of these functions. Will it stop functioning?

6.

And many more

Back to slide

Navigation Issues in UI Design

1.

Can users always tell where they are in the program? And what
options are available to them?

2.

How well is info presented to the user?

3.

If the program utilizes a central workspace, does it remain consistent

form screen to screen?

4.

Do navigation conventions remain consistent? Link? Menu?

5.

Check consistency in terms of mouse-over, pop-ups, clicks and drag

results.

Principles and strategies for practitioners designing universally usable sites

Users with Disabilities
Blind and low vision user
s
Color vision confusion
Cognitively disabled
Deaf & hearing impaired
Mobility impaired

Special User Groups

Children
Elderly
Users with low education, low m
otivation
Users of other languages than E
nglish
Users from other cultures than
the US
Cross language information retri
eval

Technology
Users with slow connections
Users with screens less than 640 x
480
Telephone based access to the web
(WAP)
Telephone based access to the web
(speech recognition)
Textual equivalents for audio/video
representations of content

Tutorial methods
Designs to help novice web users
Online help design, email help
methods and customer service
guidelines

http://www.otal.umd.edu/UUPractice/

Back to slide

Misc UI Elements
ELEMENT TYPE

ISSUES TO ADDRESS

1.

Instructional &
technical info

Accuracy of info and instructions

2.

Fonts

Consistency of style, Legibility of text

Difficulty of reading italic and serif fonts
Multiple font types on a single page
Fonts and the Web: http://www.w3.org/Fonts/
A broad perspective on fonts and the Web

3.

Colors

Suitability of colors foreground, background, font

Use of color

4.

Borders

Use 3D effect on noninteractive elements can be confusing.

5.

Images

Size of images
Suitability of background
Legibility of images, buttons and labels

6.

Frames

Compatibility with browsers

Display setting and browser can effect how frames are displayed
Back buttons often have unexpected result

7.

Tables

Nested table slow down loading time

Presentation may vary depends on display setting and browser

Back to slide

NetHelp- Netscape
Benefits of NetHelp
NetHelp, by virtue of being
based on Netscape
Navigator, has a viewer
that is fast becoming
ubiquitous in the Web
world. NetHelp also
provides the following
benefits:
Cross-platform support
Continually current
content
Ease of authoring and
implementation
HTML extensions provide
rich content
Availability of Plug-ins

http://home.netscape.com/eng/help

Back to slide

JavaHelp- Sun Microsystem

http://java.sun.com/products/javahelp/

Benefits of JavaHelp
JavaHelp technology offers
many advantages over
other help systems or plain
HTML pages including the
following:
Can run on any browser
or platform
Comes with its own HTMLbased viewer
Can be embedded within
the application
Offers better navigational
aids
Allows full text search
Has state-of-the-art
merging capabilities.

Back to slide

WebHelp- eHelp Corp.

Benefits of WebHelp
WebHelp supports standard
HTML as well as all key
application Help capabilities,
including :
dynamic table of contents
multi-level indexes,
full-text search,
context-sensitive Help,
related topics controls,
expanding text, and more.
With WebHelp, you are no
longer restricted to a certain
platform or browser, nor do you
http://www.helpauthoring.com/webhelp/webhelp.htm/
have to create and maintain
multiple source files to meet
the needs of their end users.

Back to slide

RoboHelp- Blue Sky Corp.

http://www.blue-sky.com

Benefits of RoboHELP
When constructing a Help
system, the author will need to
implement navigation structures
and display features in addition to
writing the Help system content.
In practice, organizing the
content and creating the
navigation is the most involving
aspect of Help system design and
construction.
The goal of a Help authoring tool
is to automate the process of
creating and organizing content
as much as possible, allowing
the author to focus on the Help
system's content, design and
usability instead of the mechanics
of implementing specific features
on different Help formats.

Back to slide