Performance

Testing of
Mobile Applications
An end-to-end approach to Mobile
Application Testing

A White Paper
By Amit Mohanty
Architect
Testing Practice
 Contents

Introduction........................................................ 3
Why Performance Testing?................................ 3
The Importance of Performance Testing........................ 3
How Performance Testing Helps.................................... 3
Performance Testing Challenges of
Mobile Applications............................................ 3
Key Parameters - Performance Testing of
Mobile Applications............................................ 4
Key Focus Areas - Performance Testing of
Mobile Applications............................................ 4
Performance Testing of Mobile Applications
- Process Flow.................................................... 6
End-User Perspective (Functional
Performance Testing)......................................... 6
Overview......................................................................... 6
Benefits.......................................................................... 7
Topology......................................................................... 7
Device Perspective............................................ 7
Overview........................................................................ 7
Benefits.......................................................................... 7
Topology......................................................................... 8
Network Perspective......................................... 8
Overview........................................................................ 8
Benefits.......................................................................... 10
Server Perspective............................................ 10
Overview........................................................................ 10
Topology......................................................................... 10
Benefits.......................................................................... 11
The Conclusion.................................................. 11

Point of View Paper on Performance Testing of Mobile Applications Mphasis 2

 Introduction
There are diverse conditions in which a mobile application is expected to
perform at its best:
A wide diversity in devices and platforms
The need for the best user experience
A variety of network connectivity options and
The ever-growing user base of smartphones
Its no wonder that Performance Testing of mobile applications is supremely
important.
At Mphasis, our Mobility Testing Center of Excellence has developed
a specific approach to test various aspects of mobile application
performance that covers end user, device network and server parameters.

Why Performance Testing?

The Importance of Performance Testing
Performance testing of mobile applications is very important because they
Performance are installed on mobile devices based on various mobile platforms/OSs like
Testing helps to Android, iOS, Blackberry, Windows, etc. all of which have various hardware
configurations like memory, processor, screen-resolution and having
ensure that the varying network connectivity. Yet these applications are expected to work
system/ well in all circumstances, no matter how much load there is on the server;
application whatever the strength of the network may be; and what the device
responds configurations are.

consistently within How Performance Testing Helps

acceptable levels Performance testing helps to ensure that the system/application responds
(per baseline or consistently within acceptable levels (as per baseline or SLA) and functions
correctly under the current load condition or expected load conditions.
SLAs) and Performance testing of mobile applications helps in bridging the gap
functions correctly between testing of a mobile application in a test environment with test data
under the current and a real world scenario, thereby contributing to business success. It
load condition or gives confidence to launch an application in this rapidly changing mobile
space with users always expecting the best features and a great user
expected load experience.
conditions.
Performance Testing Challenges of
Mobile Applications
The challenges all deal with how the application performs in all these
conditions:
Varying hardware configurations like battery, processor, memory, etc
Varying amounts of disk space
When the same action is performed multiple times
When the same action is performed for long period of time
On varying network conditions like bandwidth, network latency, packet
loss, etc.
When there is heavy load on the backend application server

Point of View Paper on Performance Testing of Mobile Applications Mphasis 3

 Key Parameters - Performance Testing
of Mobile Applications
The key performance parameters of mobile applications are as follows:

Does the application respond quickly enough for the

Application Response Time intended users?

Reliability How stable is the system under a heavy work load?

Which configuration provides the best performance

Configuration Sizing level?

Is the application stable under expected and

Scalability unexpected user loads?

Transaction Response Time How long does it take for the transaction to complete?

Bottleneck Identification What is the cause of degradation in performance?

Power Management Does the application causes faster battery drain out?

How much CPU and Memory is consumed by the

Resource Utilizations application?

Figure 1: Key performance parameters of mobile applications

Key Focus Areas - Performance

Testing of Mobile Applications
The four main perspectives of Performance Testing of mobile applications
and the key focus areas of each can be categorized as below:

Functional Performance Testing or Testing of the Performance of

End-User the application
Perspective
KPIs - Application Launch time, Navigation Response Time, Screen
Transition time, Transaction time measured on the application etc.
Performance Testing of
Mobile Applications

Testing the Resource Utilization by the application

Device
Perspective
KPIs - CPU, Battery, Memory, Data Throughput

Testing the mobile application under varying network conditions

Network
Perspective KPIs - Application features tested under varying network conditions
like bandwidth, latency and packet loss and 2G and 3G network
profiles

Testing the mobile application by loading the backend application

server using device emulator or real device
Server
Perspective KPIs- Transaction Time of the server by varying the bandwidth,
latency and packet loss of the TCP/IP network

Figure 2: Four main perspectives of Performance Testing of mobile applications and the focus areas

Point of View Paper on Performance Testing of Mobile Applications Mphasis 4

 Performance Testing of Mobile Applications- End-To-End Approach

ios
Android

Client
End-User Perspective
Application launch time
Navigation response time
Screen transition time
Transaction time measured
on the application
Network Perspective
Application features tested
under varying network
conditions like bandwidth,
latency and packet loss and
2G, 3G, network profiles.
Server
Server Perspective
Transaction Time of the
server by varying the
bandwidth, latency and
packet loss of the
TCL/IP network.

Performance KPI Performance KPI Performance KPI

Test Area Test Area Test Area

Transition Time Data in Data in Backend Bandwidth,

seconds seconds under Server Latency
Transition Time 2G/3G network
Application Load
Data in conditions Transaction
Launch Time seconds Response
Data in Time
Data in Application seconds under
Load Time seconds Launch Time 2G/3G network
Page conditions
Component Data in
Analysis seconds Data in
Load Time seconds under
2G/3G network
Response Time Data in
seconds conditions

Device Perspective Page Data in

component seconds under
CPU 2G/3G network
Memory analysis conditions
Battery
Data throughput Data in
Response time seconds under
2G/3G network
Performance KPI conditions
Test Area
Power Battey Usage
Management
CPU CPU Kernel
CPU Total
Performance CPU User
Memory Cache
Memory Memory Free
Memory Used

Figure 3: End-To-End Approach For Performance Testing Of Mobile Applications

Point of View Paper on Performance Testing of Mobile Applications Mphasis 5

 Performance Testing of Mobile
Applications - Process Flow
Understand Performance test
Requirements ,system Architecture,
Workload & Performance Goals

Develop Performance Test Assets

(Test Plan, Test Scripts,
Monitoring Strategy etc)

Restart Execute Performance Tests Collect Monitoring

after tuning metrics
Criteria
not met

Analyze Results and Monitor

Performance Metric
Tuning
Performance
Criteria are met
Complete testing

Figure 4: Process flow of performance testing of mobile applications

End-User Perspective (Functional

Performance Testing)
Overview
End-user perspective or functional Performance Testing deals with the
End-user performance of the application on a real device connected to a real
perspective or network, when the backend application server is in a loaded state. All this
can be achieved by the tester sitting at some geographic location and
functional accessing mobile devices which are remotely located over the cloud. This
performance helps in reducing the total cost of ownership (TCO) as the testing team
testing deals with does not have the overhead of maintaining the devices over the cloud or
the performance of the server connecting to the device cloud. Since the behavior of the mobile
application on a real device is checked when the server is in a loaded
the application on condition, similar to a real world environment, it gives a lot of confidence
a real device about the user experience in terms of performance before an application is
connected to a real launched for large scale usage.
network, when the To check the end-user perspective of the performance of a mobile
backend application, the backend application server of the mobile application is
loaded by a number of virtual users hitting the backend server in parallel by
application server using HP Loadrunner. While the load on the backend server is being
is in a loaded state. maintained at a specific level, a functional testing script with various
transactions defined to capture the page load time, application launch time,
screen transition time, etc. written in HP Unified Functional Testing is run
simultaneously from within the HP Loadrunner Controller. Then, the result is
observed in the HP Loadrunner Analyzer in terms of how many transactions
passed a specific load condition, what time was taken for these
transactions to complete, etc. These values are then analyzed against the
expected values or the SLA.

Point of View Paper on Performance Testing of Mobile Applications Mphasis 6

 Benefits
It simulates a network condition or scenario very closely resembling the
real world scenario.
It provides greater confidence to launch the application for large scale
usage due to the close resemblance of the test environment with a real
world environment.
Since the load generation script and the functional script with various
transactions run simultaneously, it helps the in-depth analysis of the
reports and uncovers potential performance issues.
It reduces the total cost of ownership for the testing team due to the
usage of cloud based device access.
Topology

Global Delivery Centers Enterprise Client Location

Remote Mobile Cloud Virtual User Load Applied Enterprise Server

Web Interface To Server

Remote Mobile
Cloud
Central Node
Server
Softwares
Required
Test Management
UFT UFT Mobile
Quality
Loadrunner Center/ ALM
This kind of Remote Mobile Cloud Device
Racks
Performance Test Teams
Devices with Operator SIMs on
network

Testing helps in
Figure 5: End-user perspective
checking how the
mobile application
performs Device Perspective
under the limited Overview
hardware The device perspective of Performance Testing of mobile applications helps
resources of a in gauging the device vitals while an application is. This kind of
Performance Testing helps in checking how the mobile application
mobile device and
performs under the limited hardware resources of a mobile device and
check CPU check CPU usage, memory consumption, battery consumption, data
usage, memory throughput, etc. when the application is in use. This also helps in testing a
consumption, mobile application under stressed memory conditions like low memory and
low battery, thus ensuring a great user application.
battery
consumption, data Benefits

throughput, etc. Using any standard web browser, a user can remotely activate a handset
located anywhere worldwide.
when the
application is in The solution allows collecting all vital information about the device like
memory usage, battery level, battery voltage, CPU usage, etc.
use.
Certain tools installed on mobile devices help accurately collect
information about the battery consumption and radio interface.

Point of View Paper on Performance Testing of Mobile Applications Mphasis 7

 Single user performance tuning can be done using this analysis.
Topology

Remote Test Lab

Internet

Figure 6: Device perspective

The network Network Perspective

perspective of
Overview
Performance
Testing of mobile The network perspective of Performance Testing of mobile applications
deals with testing the application performance on a real mobile device
applications under varying network conditions like bandwidth, latency, packet-loss, etc.
deals with testing which is otherwise called the network profile. Since network profiles similar
the application to 2G, 3G and 3.5G networks can be simulated using different approaches,
performance on a it is closest to checking the performance of the mobile application in a real
world scenario. It enables testers to check the application on various
real mobile device mobile devices in the simulated network condition while they sit remotely at
under varying some other location, with the help of mobile device cloud. Also, it gives a
network conditions lot of confidence to mobile application developers to launch the application
for large scale use due to its close resemblance to a real world scenario.
like bandwidth,
latency, packet- There are two approaches to the network perspective of Performance
Testing of mobile applications:
loss, etc.
which is otherwise Testing of single user mobile application performance using various
network profiles remotely on a real device in the mobile cloud using the
called the network network virtualization appliance from Shunra
profile.
Testing of mobile application performance remotely on a real device
connected to a real network on the mobile cloud while the load is being

Point of View Paper on Performance Testing of Mobile Applications Mphasis 8

 applied to the backend application server using specific network profiles
which simulate a real world load on the server
Topology of single user network perspective of mobile application
Performance Testing

Global Delivery Centres Enterprise Client Location

Remote Mobile Cloud Enterprise Server

Web Interface

Remote Mobile
Cloud Network
UFT/Native Central Node Virtualization
Scripting Server

Test Management
Shunra
Quality
Center/ ALM
Test Teams Remote Mobile Cloud Device
Racks
Devices with Operator SIMs on
network

Figure 7: Network perspective- single user

In the case of single user Performance Testing of a mobile application by

varying the network profiles, the Shunra appliance is connected to a single
device on the cloud via Wi-Fi. So, the network connection of the mobile
device can be configured by configuring the Shunra appliance with the help
of an application available on the mobile device. While the mobile device is
accessing the network with a specific network profile, the performance of
the mobile application running on the devices is assessed.
Topology of network perspective of mobile application Performance
Testing with loaded backend server

Global Delivery Centers Enterprise Client Location

Remote Mobile Cloud Virtual User Load Applied Enterprise Server

Web Interface To Server

Remote Mobile
Cloud
Central Node
Server
Softwares
Required
Test Management
UFT UFT Mobile
Quality
Shunra NV Loadrunner Center/ ALM
Remote Mobile Cloud Device
Racks
Devices with Operator SIMs on
Test Teams network

Figure 8: Network perspective- loaded backend server

Point of View Paper on Performance Testing of Mobile Applications Mphasis 9

 In this kind of Performance Testing, virtual user load is applied on the
backend application server using specified network profiles similar to a real
world scenario using the Shunra NV Add-In on Loadrunner. While the load
is being applied to the backend server, the applications performance on a
real device connected to a real network on the mobile device cloud is
assessed with the help of UFT and UFT Add-In from the testers computer.
Benefits
Using Network Virtualization solution for real-world load and capacity
testing
Virtualized network conditions unlimited points of presence
Support for accurate simulation of users from any geographical
locations/regions
Offer of bandwidth utilization per transaction summary

Server Perspective
Overview
Testing the The server perspective of Performance Testing of mobile application deals
backend server of with testing the backend server of the mobile application by applying
varying the virtual user load and then extrapolating the data for the
the mobile production server capacity. It gives confidence about the performance of
application by the application at all times during a day/week/month whenever the load on
applying varying the server varies. It also incorporates the capability through which
the virtual user bandwidth and latency can be varied to create a more life-like scenario. It is
the traditional load testing of mobile applications backend server
load. capabilities using Loadrunner.
Topology

L0 Real Devices
The Mobile Cloud

L1 V1

L2 V2
Mobile
L3 V3 Application
Service
L4 V4

Performance Test L5 V5
Controller
Virtual Mobile Users
Load Generators - Multi user/multi location
simulations

Performance Diagnostics

Figure 9: Server perspective

Point of View Paper on Performance Testing of Mobile Applications Mphasis 10

 Benefits
Emulates conditions of a controlled load on the server
Simulates real-world mobile network characteristics
Measures application and server performance under load
Checks where performance delays occur: network or application related
delays; CPU performance; I/O delays; and monitors the network and
server resources under load
Supports generating load through real devices with QTP integration

The Conclusion
A growing number of mobile users are using their mobile devices for their
day to day tasks including important transactions like money transfers,
paying credit card bills, etc. Due to this, todays mobile applications are
very important and are expected to perform at their best at all times. The
surging growth in the number of mobile users using high end mobile
devices and relying on them for various kinds of transactions necessitates
that all mobile applications are thoroughly tested for performance glitches
before their launch. Failing to prepare for these highly variable conditions
can lead to customer dissatisfaction and cause damage to the brand
reputation.
This whitepaper aims at combining the various solutions and perspectives
for Performance Testing of mobile applications and provides a guide to the
overall process of Performance Testing.

EFFICIENCY/PERFORMANCE
(INCLUDING NETWORK - RELATED) 64%

FUNCTIONALITY
48%
46%
PORTABILITY
36%
USER INTERFACE/EASE OF USE 31%

COMPATIBILITY/
REGRESSION TESTING 18%
14%
SECURITY (PROTECTION OF SENSITIVE
DATA OVER THE PHONE OR AIR)

CERTIFICATION OF APPLICATION

Point of View Paper on Performance Testing of Mobile Applications Mphasis 11

 Amit Mohanty
About Author Architect (Testing Practice)

Amit has more than 11 years of experience in Application Testing,

especially in the Mobility space and spanning across multiple companies.
He has played various roles in Testing, Automation, Application Branding,
Requirement Management, Test Management and has led teams located
across geographies. He has experience in Performance and Load Testing of
mobile applications using leading tools in the market.

With an MS in Quality Management from BITS, Pilani and BE in Computer

Science from Utkal University, Bhubaneswar, Amit is also Six Sigma Green
Belt certified and ISTQB certified.

Point of View Paper on Performance Testing of Mobile Applications Mphasis 12

 UNITED STATES

CANADA

GREAT BRITAIN

FRANCE

IRELAND

NETHERLANDS

POLAND

BELGIUM
Indore
GERMANY
Vadodara
SWITZERLAND Raipur
Mumbai Bhubaneshwar
INDIA Pune
JAPAN

CHINA Bangalore
Mangalore Chennai
PHILIPPINES Pondicherry
SINGAPORE

INDONESIA

SRI LANKA

AUSTRALIA

NEW ZEALAND

OUR GLOBAL FOOTPRINT Delivery Footprint Client Footprint India Centre (BPO) Global Service Centre (ITO & APPS) Training Academy

About Mphasis
Mphasis (an HP Company) enables chosen customers to meet the demands of an evolving market place.
Mphasis fuels this by combining superior human capital with cutting edge solutions in hyper-specialized areas.
Contact with us on www.mphasis.com

For more information, contact: marketinginfo@mphasis.com

To find out more about Mphasiss ability to help your organization in the
Performance Testing of Mobile Applications space, please contact:

Amit Mohanty
Architect - Testing Practice
amit.mohanty@mphasis.com

USA UK INDIA
460 Park Avenue South 88 Wood Street Bagmane World Technology Centre
Suite #1101, New York London EC2V 7RS, UK Marathahalli Outer Ring Road
NY 10016, USA Tel.: +44 20 85281000 Doddanakhundi Village, Mahadevapura
Tel.: +1 212 686 6655 Bangalore 560 048, India.
Tel.: +91 080 3352 5000
Copyright Information
Mphasis and the Mphasis logo are registered trademarks of Mphasis Corporation.
All other brand or product names are trademarks or registered marks of their respective owners.
Copyright Mphasis Corporation. All rights reserved.