Principles of Embedded Systems: ESE63201

Software Performance Engineering (SPE)

When revising this material, please refer to the following example

Example: You are a SW architect in your organization. You are required to design,
implement & deliver a SW module that will receive and process 100 messages on
average. Your module must satisfy the following specifications:
 Module shall encrypt the messages using a predefined algorithm.
 The module shall forward the messages to another server.
 Module shall not lose any message
 Under normal operating conditions, 99.9% of the messages shall be processed within
5ms since the msg is received
 When the module receives 150 (or less) msgs in one second, then the module shall
process at least 99% of the msgs within 5ms of receiving each msgs.

What steps will you take to ensure successful delivery of this module?

We have so far been studying the performance at system levels. We rarely mentioned
anything related to software. A quick look back at any of the systems we analysed would
reveal that the software component is, in almost all case, the main contributor to the
performance of the systems of interest to us, engineers. If you consider any server e.g.,
you can make the following observations:
 Messages are received at its input. SW is needed to read the msgs from the input port.
 Messages are placed into a Q. The Q is implemented in SW
 Messages are served. The service is performed via SW manipulations of the msgs.
 Messages are forwarded on to the next server. This is achieved via a SW component.

In short, any modern telecoms, control or any electronic equipment must have SW
driving this ‘thing’ in one way or another.

Studies of performance of any system will therefore have to consider the SW element. In
many cases, the root cause of bad performance of most systems could be traced to badly
designed software units or to bad overall software architecture.

Since the inception of SW development, SPE has emerged as an important branch of SW

engineering, system engineering and system design. The development of any system
should consider the performance of the SW component as an integral part of the overall
system.

Performance Failures: Cause and Prevention

The primary cause of performance failures is a reactive approach to performance during
the development process. Cost and schedule pressures encourage project managers to
adopt a “fix-it-later” approach in which performance is ignored until there is a problem.
When a problem is discovered, you must buy more hardware, developers must try to
“tune” the software to meet performance objectives or both, causing schedule delays and
cost overruns. In some cases, it is simply not possible to meet performance objectives by
tuning.

Performance problems are most often due to inappropriate architectural choices rather
than inefficient coding. This means that performance problems are introduced early in the
development process but are typically not found until later (during integration test or
when the system is in use) when they are more difficult and more expensive to fix.

The Cost of Performance Failures

 Increased Development Costs
 Increased Hardware Costs
 Increased Operational Costs
 Canceled Projects
 Damaged Customer Relations
 Lost Income
 Reduced Competitiveness

Performance engineering within systems engineering, encompasses the set of roles,

skills, activities, practices, tools, and deliverables applied at every phase of the Systems
Development Life Cycle which ensures that a solution will be designed, implemented,
and operationally supported to meet the non-functional performance requirements
defined for the solution.

Software Performance Engineering (SPE) is a systematic, quantitative approach to the

cost-effective development of software systems to meet performance requirements. SPE,
a software-oriented approach, focuses on architecture, design, and implementation
choices.

SPE encompasses efforts to describe and improve performance, with two distinct
approaches:
 an early-cycle predictive model-based approach, and
 a late-cycle measurement-based approach.

SPE activities require that everyone involved in the development process should carry out
various tasks to guarantee the final product would meet its requirements. Such tasks are
list below:

Engineers should:
 Use performance models to evaluate architecture and design alternatives before
committing to code
 Start with the simplest model that identifies problems with the system architecture,
design, or implementation plans then add details as your knowledge of the software
increases
 Establish a configuration management plan for creating baseline performance models
and keeping them synchronized with changes to the software
 Use best- and worst-case estimates of resource requirements to establish bounds on
expected performance and manage uncertainty in estimates
 Use performance measurements to gather data for constructing SPE models and
validating their results
 Plan measurement experiments to ensure that results are both representative and
reproducible
 Instrument software to facilitate SPE data collection
 Measure critical components early and often to validate models and verify their
predictions
 Quantify the benefits of tuning versus refactoring the architecture or design
 Perform performance studies & produce timely results.
 Produce credible model results and explain these results
 Produce quantitative data for thorough evaluation of alternatives

Managers (& engineers) should:

 Perform an early estimate of performance risk
 Perform sw Failure Mode and Effect Analysis (FMEA)
 Match the level of SPE effort to the performance risk
 Track SPE costs and benefits
 Integrate SPE into your software development process and project schedule
 Establish precise, quantitative performance objectives and hold developers and
managers accountable for meeting them
 Identify critical use cases and focus on the scenarios that are important to
performance
 Perform an architecture assessment to ensure that the software architecture Will
support performance objectives
 Secure the commitment to SPE at all levels of the organization
 Ensure that developers and performance specialists have SPE education, training, and
tools
 Require contractors to use SPE on your products

Software Performance Testing:

Software performance is normally accessed via a combination of SW tests. The tests fall
in a number of categories, these are summarized as below:
1- Load Testing
Load testing is the simplest form of performance testing. A load test is usually conducted
to understand the behavior of the application under a specific expected load.

2- Stress Testing
Stress testing is normally used to understand the upper limits of capacity within the
application landscape. This kind of test is done to determine the application's robustness
in terms of extreme load and helps application administrators to determine if the
application will perform sufficiently if the current load goes well above the expected
maximum.

3- Endurance Testing (Soak Testing)

Endurance testing is usually done to determine if the application can sustain the
continuous expected load. During endurance tests, memory utilization is monitored to
detect potential leaks. Also important, but often overlooked is performance degradation.
That is, to ensure that the throughput and/or response times after some long period of
sustained activity are as good as or better than at the beginning of the test.

4- Spike Testing
Spike testing, as the name suggests is done by spiking the number of users and
understanding the behavior of the application; whether performance will suffer, the
application will fail, or it will be able to handle dramatic changes in load.

5- Configuration Testing
Configuration testing is another variation on traditional performance testing. Rather than
testing for performance from the perspective of load you are testing the effects of
configuration changes in the application landscape on application performance and
behaviour. A common example would be experimenting with different methods of load-
balancing.

6- Isolation Testing
Isolation testing is unique to performance testing but a term used to describe repeating a
test execution that resulted in an application problem. Often used to isolate and confirm
the fault domain.