API Testing:

Challenges and Best Practices

Modern composite applications are aggregating and consuming private, partner, and public APIs
at a staggering pace in order to achieve business goals. ProgrammableWeb reports that there
are over 10,000 APIs published today— which is well over twice as many than there were two
years ago.1 Beyond these publically-exposed APIs, the number of private APIs is estimated to be
in the millions.
As the risks associated with application failure have broader business impacts, the integrity of the
APIs you produce and consume is now more important than ever. An API that fails to deliver the
expected level of security, reliability, and performance can thus have tremendous business
impacts—both to the organization producing it and to those consuming it.
If you are integrating exposed APIs into your business critical transactions, you are essentially
assuming the risks associated with that API's integrity (or lack thereof). As the number of external
APIs integrated into a business process increases, so do the potential points of failure. The
business impact of any application failure is the same, regardless of whether the fault lies within
the components you developed or the APIs you are consuming. Finger pointing does little to
foster customer satisfaction and brand loyalty.
If you are exposing an API, the assumption is that it will work as described. Once the
organizations consuming that API integrate this exposed functionality into their own applications,
API failure jeopardizes the transactions that now depend on this functionality. If your API is
popular, you can guarantee that a glitch will make the headlines. The more secure, reliable, and
dependable your API, the better the chance of consumption and the greater the potential for
business expansion. If you’re providing a questionable interface and there are viable alternatives
to your API, you’re likely to lose business since switching costs associated with API integration
are so low.

4 Integrity Challenges with the API Economy

In today's API economy, your ability to protect your brand hinges upon your awareness of (and
response to) the following challenges…

1. Broader Attack Surface Area

Simply exposing an API through an internal infrastructure undeniably increases an application’s
attack surface area from a security perspective. It could be vulnerable to API-level attacks
(injections, payload-based attacks, etc.) as well as exploits that take advantage of ineffective
authentication, encryption, and access control. The challenge of broader attack surface area is
compounded if the API is hosted by a public cloud service. In traditional computing, the producer
is aware of (and has full control over) the parameters of the network security. With cloud services,
this level of control is significantly diminished. If you’re now leveraging a third-party set of
services, the onus is upon you to ensure that your APIs will provide the level of security that your
organization expects.

1 http://blog.programmableweb.com/2013/09/23/programmablewebs-directory-hits-10000-apis-and-counting/

1
2. Elevated Potential for Unexpected Misuse
Considering the range and number of people who will have access to published APIs, it’s virtually
inevitable that they will be exercised in a number of unexpected ways: both by people innocently
using them in ways the producer never anticipated and by attackers maliciously trying to exploit
them. In the early days of SOA—when services were exposed internally through controlled
networks—you could be fairly certain that your services would be used by colleagues or partners
who were familiar with their intended uses cases. Now, when an API is exposed to the public, the
producer surrenders all control and certainty over how those APIs are consumed.

3. Exceptionally Unpredictable Demand

APIs typically need to meet established performance SLAs; however, validating performance vs.
SLAs is complicated because it’s so difficult to predict how and when the API might be accessed.
Thus, it’s important to validate performance SLAs against a broad range of performance
scenarios, including the sudden surges that could occur if an API garners unexpected attention.
Moreover, if you’re testing an API that interacts with additional layers of services, the potential for
variable or unacceptable performance increases exponentially. This makes it both more critical —
and more challenging—to execute a robust set of performance testing scenarios and pinpoint
whether the system under test satisfies expectations.

4. API Consumers Need Test Environments

To promote widespread adoption of APIs, it’s often desirable to provide API consumers test
environments (a.k.a. sandboxes) that enable them to develop and test against exposed services
with zero impact on the production system. Such test environments are also critical when API
producers want to jumpstart adoption by allowing integration to begin before their API is actually
implemented or before new features are fully completed. Another key driver for providing a test
environment is to give API consumers easy access to the broad range of behavior, data, and
performance profiles they might want to leverage for testing their integration with that API.

5 API Testing “Must Haves”

Ensuring that APIs are delivering the necessary level of security, reliability, and performance
that’s vital to success in today’s API ecosystem inevitably involves developing, continuously
executing, and religiously maintaining a broad array of complex tests. Following are several key
API testing “must haves” that will help you achieve those goals in light of the above challenges.

1. Intelligent Test Creation and Automated Validation

With APIs, testing a broad range of conditions and corner cases is critical, so automation comes
to the forefront. The creation and execution of simple automated tests with limited or manual
validation might have sufficed for internal given web services that were used internally (e.g., via
SOA), but more sophisticated and extensive automation is required to be confident that APIs are
robust enough to meet business expectations. You need a level of automation that gives you a
comprehensive set of functional test cases that can be repeated in a systematic manner.

2
Recommended capabilities for this goal include an intuitive interface for automating complex
scenarios across the messaging layer, ESBs, databases, and mainframes:

 Defining automated test scenarios across the broad range of protocols and message
types used in APIs: REST, WADL, JSON, MQ, JMS, EDI, fixed-length messages, etc.

 Automating rich multilayer validation across multiple endpoints involved in end-to-end test
scenarios.

 Parameterizing test messages, validations, and configurations from data sources, values
extracted from test scenarios, or variables.

 Defining sophisticated test flow logic without requiring scripting.

 Visualizing how messages and events flow through distributed architectures as tests
execute.
These are all capabilities that should—or at least could—have been applied to web service
testing for SOA. In fact, most of these capabilities were invented, tested, and refined in the
context of SOA testing. APIs—with their extreme exposure and myriad opportunities for misuse—
brings us to the tipping point that makes these automated testing and validation capabilities a
“must have” for organizations serious about delivering APIs that satisfy user needs and
expectations.

2. Change Management for Test Assets and Environments

Continuously evolving APIs helps organizations stay a step ahead of the competition while
responding to business demands. Yet, this frequent change presents significant quality risks if the
automated test suite fails to keep pace with the evolving API.
A system for fast, easy, and accurate updating of test assets is critical for keeping test assets in
sync with the changing API. If you can automatically assess the impact of changes to existing
tests and then quickly update existing tests (or create new ones) in response to the identified
change impacts, you can vastly reduce the amount of time required to ensure that your tests don’t
fail due to expected changes…or overlook critical new functionality.

3. Service Virtualization for Simulated Test Environments

Service Virtualization technology creates simulated test environments that provide anytime,
anywhere access to the behavior of dependent resources that are unavailable, difficult to access,
or difficult to configure for development or testing. “Dependent resources” might include
mainframes, mobile app front-ends, databases, web services, third-party applications, or other
systems that are out of your team’s direct control. Service virtualization can be used in
conjunction with hardware/OS virtualization to access the environments you need to test earlier,
faster, or more completely.
In the context of API testing, service virtualization can be applied in two key ways:

 To simulate access to the dependent resource behavior (e.g., from a mobile app,
database, legacy system, or third-party service) that you need in order to thoroughly
validate your API.

3
  To simulate the behavior of your APIs, creating a test environment that API consumers
can develop and test against without impacting your production environment—or to
enable development and testing to begin before APIs are completed.

4. Extensive Performance Testing—Ideally, with Service Virtualization

Due to the highly-exposed nature of APIs, there’s a high potential for unpredictable and often
volatile traffic volumes. To determine whether your API will satisfy SLAs in the event of the erratic
or surging demand that APIs commonly face, it’s essential to ramp up the scope of performance
testing. You can use service virtualization (covered above) to create simulated test environments
that help you test against different performance scenarios that would otherwise be difficult to
create in the test environment.
For instance, you can easily set performance conditions (e.g., timing, latency, delay) to emulate
peak, expected, and slow performance—perhaps to help you plan for cloud bursts or determine
how the API might respond when someone is accessing it from China. You can also configure
various error and failure conditions that are difficult to reproduce or replicate with real systems—
for instance, if your APIs rely on Amazon Web Services, you can easily simulate a scenario
where AWS is down. This ability to rapidly configure a broad range of conditions in dependent
systems is essential for determining if your APIs provide reasonable responses—or at least fail
gracefully—under exceptional conditions.
One final way that adopting service virtualization helps performance testing: you can “virtualize”
any connections to third-party systems, reliably eliminating the risk that your stress tests might
impact services you aren’t permitted (or budgeted) to barrage with test messages.

5. Extensive Security Testing—Ideally, with Service Virtualization

Considering APIs’ increased attack surface area, a multi-faceted security testing strategy is
essential for ensuring that development has built the appropriate level of security into your
application. This includes:

 Executing complex authentication, encryption, and access control test scenarios.

 Generating a broad range of penetration attack scenarios involving parameter fuzzing,

injections, large payloads, etc.

 Running penetration attack scenarios against your existing functional test scenarios.

 Monitoring the back-end during test execution in order to determine whether security is
actually compromised.
In addition, if you’re adopting service virtualization (covered above) you can leverage it to take
your security testing to the next level:

 It provides rapid ways to emulate attack scenarios as well as emulate different security
behaviors of dependencies. This lets you derive more value from your existing functional
test scenarios (since you can run them vs. different security scenarios that would
otherwise be difficult to configure and unfeasible to test against).

 It enables extensive security testing to be performed without a security expert. Existing

test scenarios can be easily executed against a broad set of preconfigured security
scenarios.

 It helps you isolate and zero in on your APIs response to various attack scenarios and
different security behaviors of dependencies.

4
About Parasoft
For 25 years, Parasoft has researched and developed software solutions that help organizations
define and deliver defect-free software efficiently. By integrating Development Testing, cloud/API
testing, and service virtualization, we reduce the time, effort, and cost of delivering secure,
reliable, and compliant software. Parasoft's enterprise and embedded development solutions are
the industry's most comprehensive—including static analysis, unit testing, requirements
traceability, functional & load testing, dev/test environment management, and more. The majority
of Fortune 500 companies rely on Parasoft in order to produce top-quality software consistently
and efficiently. For more information, visit the Parasoft web site and ALM Best Practices blog.

Contacting Parasoft
USA Phone: (888) 305-0041 Email: info@parasoft.com
NORDICS Phone: +31-70-3922000 Email: info@parasoft.nl
GERMANY Phone: +49 731 880309-0 Email: info-de@parasoft.com
POLAND Phone: +48 12 290 91 01 Email: info-pl@parasoft.com
UK Phone: +44 (0)208 263 6005 Email: sales@parasoft-uk.com
FRANCE Phone: (33 1) 64 89 26 00, Email: sales@parasoft-fr.com
ITALY Phone: (+39) 06 96 03 86 74 Email: c.soulat@parasoft-fr.com

OTHER See http://www.parasoft.com/contacts

Author Information
This paper was written by:

 Wayne Ariola (wayne.ariola@parasoft.com), VP of Strategy at Parasoft

 Cynthia Dunlop (cynthia.dunlop@parasoft.com), Lead Technical Writer at Parasoft

© 2013 Parasoft Corporation

All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation. All other
products, services, and companies are trademarks, registered trademarks, or servicemarks of their respective holders in the US and/or other countries.

5
 Parasoft API Integrity

Modern composite applications are aggregating and consuming private, partner, and public APIs at a staggering pace
in order to achieve business goals. As applications grow increasingly interdependent, the security, functionality, and
performance of the composite “mashup” is only as strong as its weakest link. The success of an end-to-end transaction
depends upon all the parts working flawlessly all the time; even small glitches from a popular API can singlehandedly
choke thousands of transactions.

Ensuring API integrity is complicated by a number of factors:

Extensive testing is required to ensure that APIs will satisfy expectations
under the extreme conditions they might face in the field
Validating end-to-end test scenarios typically requires access
to third-party systems that are often unavailable or offline
Each of the many continuously-evolving components involved
in modern applications requires very specialized domain expertise
to test and analyze
Manual testing efforts lack the breadth, depth, and repeatability
that is critical for identifying application risks prior to production

Without an enterprise-level automated solution for ensuring the integrity of

APIs and API-driven applications, organizations risk:
Brand erosion as faulty software drives away customers
Time-to-market delays that diminish market share
Exposure to legal liability associated with application failure
Failure to comply with applicable regulatory standards and technical contracts

Ensure that interconnected applications meet expectations

Parasoft’s API Testing solution was built from the ground up to simplify the complex testing that’s vital for secure, reliable,
and compliant composite applications. The result:
Reduced costs by reducing testing costs, reducing technical debt, and exposing defects earlier in the SDLC
Reduced risks by applying more exhaustive testing techniques, increasing test coverage,
and immediately exposing any defects introduced by modifications
Increased efficiency by leveraging sophisticated automation and enabling artifact reuse

An API Testing solution is commonly applied in the following situations:

Complex composite applications: To visualize and validate how messages and
events flow through the distributed architecture as tests execute
Cloud-based applications: To facilitate cloud migration and ensure functionality,
security, and performance expectations are met in dynamic environments
Mobile development: To ensure the continued functionality, security, and performance
of the frequently-evolving APIs that drive mobile applications
End-to-end functional testing: To automate and analyze test scenarios across the many
disparate, specialized endpoints involved in a single business transaction

1 Parasoft API Testing Solution / Data Sheet

 Real Results
Parasoft API Testing Solution
Parasoft’s comprehensive enterprise-grade solution dramatically
simplifies the complex testing needed for today’s interconnected
Parasoft API Testing successes include: business systems.
AT&T: To ensure the reliability of iPhone
billing systems
Simple Generation of Flexible, Extensible Tests
From a user-friendly interface, generate tests by monitoring live
IRS: To ensure the accuracy and performance application traffic or analyzing key application resources. Intuitive
of corporate tax e-filing systems
GUIs visualize message structures, making it simple to fine-tune test
CDC: To validate rule-based specimen messages and validations for GUI-less services and APIs. Parasoft’s
management systems tests are engineered for easy sharing, reuse, and extension.
Sabre: To ensure the reliability of the world’s
largest travel network Repeatable Automated End-to-End Testing
Using automatically-generated tests as building blocks, rapidly
Cisco: To audit business processes
define complex test scenarios that exercise and validate business
IBM: To enforce governance policies transactions across multiple endpoints. From the messaging layer,
Fidelity Investments: To create and to the web UI, to the database, ESB, and mainframes, the intuitive
manage emulated service assets interface makes it simple to validate whether business logic
HP: To validate complex business scenarios satisfies expectations. Easy integration with continuous integration
platforms ensures that critical errors are exposed immediately
Vanguard: To ensure expected quality of service upon introduction.
Bloomberg: To validate performance
expectations Simulate the Behavior of Dependencies
MedicAlert: To safeguard personal health Testing efforts are often delayed and/or compromised due to
record management services difficulty accessing or configuring dependent components (3rd
Siemens: To ensure secure, reliable account party applications or services, databases, mainframes, etc.).
management and email services Parasoft Service Virtualization enables rapid, flexible simulation of
these dependencies’ behavior, giving functional and performance
Lufthansa: To ensure that cargo shipments
testers unconstrained access to—and unparalleled control
are planned and fulfilled flawlessly
over—the dependent components they need to test against. This
promotes earlier, faster, and more complete testing.
Key Features

Built from the ground up for message-layer

testing; industry “gold standard” since 2002
From a single intuitive interface, automate
complex end-to-end scenarios across
multiple endpoints (services, databases,
Web UI, ESBs, mainframes…)
Leverage broad, flexible support for
protocols, transports, message formats
Automatically generate tests that are robust,
reusable, and easily shareable
Keep test assts in sync with evolving systems Technologies & Protocols
via automated intelligent updating Parasoft supports all the standard protocols and technologies, including:
Visualize and validate how messages and
REST / SOAP / JSON / JMS / MQ / NET WCF / TCP/IP / FTP / EDI / EDIFACT
events flow through distributed architectures
as tests execute HL7 / X12 / HIPAA / ISO 8583 / WSDL / WADL / WS-* / XML/PoX / UDDI
Seamlessly integrate functional testing WSIL / BPEL / RMI / EJB / webMethods / TIBCO / SMTP / ISO 8583 / JSON /
with load testing, service virtualization, and HTTP/HTTPS / ebXML / Equifax / JDBC / ODBC / Java Objects / Fixed Length /
development testing Bytes/Binary / custom / more

USA PARASOFT HEADQUARTERS / 101 E. Huntington Drive, Monrovia, CA 91016

Phone: (888) 305-0041 / Email: info@parasoft.com

© Parasoft Corporation All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation.
All other products, services, and companies are trademarks, registered trademarks, or servicemarks of their respective holders in the US and/or other countries.