FRAMEWORKS DE PRUEBAS

Comparison of API Testing Frameworks

Flores Navarro Deivis Jhonatan, Gonzalez Franco Daniel Alejandro,

Delgado Castillo Jesus, Jimenez Silva Darold
May 28, 2022
Resumen

El interés en las pruebas de API/servicios web ha aumentado gradualmente en los últimos años. Tener la
metodología, la herramienta y la solución de prueba de automatización de API adecuadas es más importante
que nunca. Las pruebas de API son una parte esencial de cualquier proceso efectivo
En este articulo veremos lo importante que pueden llegar a ser las pruebas, mostrando diferentes causas de
fallas en el software. Tambien el desarrollo de cada Frameworks de Pruebas, como una comparativa entre
Katalon y Postman

Abstract

Interest in API/web service testing has gradually increased in recent years. Having the right API automation
testing methodology, tool, and solution is more important than ever. API testing is an essential part of any
effective process
In this article we will see how important tests can be, showing different causes of software failures. Also
the development of each Test Frameworks, as a comparison between Katalon and Postman

I. Introduction input and output (using the keyboard), soft-

At present, testing has become more relevant in
organizations in as far as software development
is concerned, since it has become evident its
importance due to the savings represented by
early detection of errors in the software.
The Application Programming Interface
(API) enables communication and the ex-
change of data between two separate software
systems. A software system that implements
an API contains functions/subroutines that can
be executed by another software system. API
testing is a type of software testing that vali-
dates APs. The Purpose of the Tests API is to
verify the functionality, reliability, performance
and security of the interfaces of programming.
In such tests, instead of using standard user
ware is used to send API calls, get output, and
log the response from the system. API tests
are very different from GUI tests and are not
They will concentrate on the appearance of an
application. It mainly concentrates on the logic
layer of software architecture business.
API is an acronym for Application Program-
ming Interface. In software application (app)
development, the API is the intermediate layer
between the presentation (UI) and the database
layer. APIs allow communication and data ex-
change from one software system to another.
API testing is a software testing practice that
tests APIs directly, from functionality, reliabil-
ity, performance to security. As part of integra-
tion testing, API tests effectively validate build
architecture logic in a short period of time.

1
 II. Development
1. API testing
API testing is done at the most critical layer
of the application: The Business Layer, where
the business logic and transactions between the
user interface and the database layer take place
they occur. [1]
1.1 Beneficios de las pruebas de API
Here are some advantages of API tests over
other types of tests. [1]
• language independent
Data is exchanged via XML and JSON for-
mats, so any language can be used for
test automation. XML and JSON are often
structured data, which makes verification
fast and stable. There are also built-in li-
braries to support data comparison using
these data formats.
• GUI independent
API tests can be performed on the applica-
tion before GUI tests. Early testing means
early feedback and better team productiv-
ity. The core functionalities of the appli-
cation can be tested to expose small bugs
and assess the strengths of the build.
• Improved test coverage
Most web services/APIs have specifica-
tions, allowing you to create automated
tests with high coverage, including func-
tional and non-functional tests.
• Fastest launches
It is common for running API tests to save
2
up to eight hours compared to UI tests,
allowing software development teams to
release products faster.
2. API testing with Katalon Studio
2.1 Katalon Studio
Katalon Studio is a free license tool released
in January 2015 with a Selenium-based engine.
Primarily, Katalon is designed to create and
reuse automated UI test scripts without coding.
Katalon Studio allows you to run automated
tests of user interface elements, including pop-
ups, iFrames, and timeouts. The tool can be
run on Windows, macOS, and Linux. [2]
2.2 Characteristic
• Katalon supports all types of REST, SOAP
requests
• Allows you to automate Web, Mobile,
Desktop applications and API’s
• Has Record and play
• Allows the capture of objects using an Ob-
ject Spy
• Automatically creates an object repository,
making it easy to maintain automated tests
• Java and Groovy support
• Integration with Git, Github, Jenkins, Jira,
etc.
• Create results reports in different formats
• Sending automatic reports
2.3 Advantages
• Easy to install
• Ease of use, with record play and prede-
fined keywords
• It has Scripting mode for more advanced
users
• Simplicity to perform cross browser test-
ing
• Automatic reports
• Ease of integration with other applications,
such as Jira, Git, Jenkins etc.
2.4 Prices

The paid version is Katalon Studio Enterprise

and has a trial period of 30 days, after which it
will automatically switch to the free version of
Katalon Studio (Free). [6] However, you can
also access the paid version which is Katalon
Studio Enterprise for a value of 1,899 per year.

3.2 Characteristic

Mentioned below are some features of Post-

man. [5]
Katalon Studio (Free)
A productive IDE for generating test cases,
with cross-platform execution. • It’s free and easy to get started
Simply download the Postman app and
submit your first application in minutes.
• Web testing
Postman is free to download and use for
• API testing
teams of any size.
• Mobile testing
• Wide support for all APIs and schemas
• Desktop testing
Make any type of API call (REST, SOAP,
or simple HTTP) and easily inspect even
Katalon Studio Enterprise the largest responses. Postman also has
built-in support for popular data formats
An advanced, powerful and scalable solution like OpenAPI GraphQL and RAML.
for companies of any size. • It is extensible
You can customize Postman to your needs
• All Katalon Studio features with the Postman API. We can also inte-
• Extended functions grate test suites into your preferred CI/CD
• Private plugins service with Newman, our command line
• Help Desk Portal collection broker.
• Support Community
We continually make improvements and
3. API testing with Postman add new features based on feedback from
our community of over 13 million users,
3.1 Postman who can also help you get the most out of
Postman in our community forum.
Postman is a collaborative platform for API
development. Postman simplifies every step of
the API lifecycle and streamlines collaboration
so you can build better APIs, faster. [5] 3.3 Precios
Thanks to this tool, in addition to testing,
consuming and debugging REST APIs, we will Postman has a free plan and several paid plans.
be able to monitor them, write automated tests All plans allow you to design, develop and test
for them, document them, mock them, simu- APIs. [8]
late them, etc.

3
 4. Katalon vs Postman

4.1 Capacity Comparison

Table 1 shows the main differences in capabili-

ties between Postman and Katalon.

III. Conclusiones

Katalon Studio is a good option for small and

medium businesses. It is an evolving solution
with many integrations that allow you to cover
a variety of test types with a single tool. It
comes with all the necessary facilities out of
the box to run various types of tests, including
API tests. It is free and easy to use. Katalon
can be used by specialists with different QA en-
gineering roles and varied programming skills,
making it an attractive solution for teams with
different levels of testers.
Postman is a tool that is used, above all, for
REST API testing, although it also supports
other functionalities that go beyond what is
included in the testing of this type of system.
Postman was born as a tool that mainly allows
us to create API requests in a very simple way
and thus be able to test the APIs. All based
on a Google Chrome extension. The Postman
user can be a developer who is checking the
operation of an API to develop on top of it
or an operator who is performing monitoring
tasks on an API.

IV. Recomendaciones

• It is recommended to evaluate the differ-

ent frameworks that exist, choose the one
that best suits your needs and budget in
this way to save time, the use of virtual
machines is also recommended to care-
fully evaluate each of the frameworks and
have a complete overview based on spe-
cific tests, then they can be presented to a
development team on real computers.

4
 capabilities
Postman Katalon
REST API Testing Sí Sí
SOAP API Testing No Sí
Automatic generation of Asser- Sí Sí
tions
Reusability of test scripts Ninguna Alta
Soporte para Data-driven JSON, CVS Excel, JSON, CVS, Data-
source, internal data
Support for BDD Cucumber No Sí
environment management Sí Sí
test reports JSON simple, HTML formats Console reports, logs, HTML
and advanced analytics re-
ports
scripting languages Javascript Groovy, Java
Web UI Testing No Sí
Mobile app testing No Sí
Test execution analytics No Sí

Table 1: Capabilities Comparison Chart - Postman and Katalon [9]

Referencias [5] Postman. (s. f.). Postman | The Colla-

[1] Katalon LLC. (2019, 22 octubre). What
is API Testing? | Definition, Bene-
fits, Types & Tool. Katalon Solution.
https://www.katalon.com/api-testing/
[2] Editor. (2019, 9 diciembre). The Good
and the Bad of Katalon Studio Au-
tomation Testing Tool. AltexSoft. [7]
https://www.altexsoft.com/blog/engineering/the-lon Pricing | Flexible Plans for Teams
good-and-the-bad-of-katalon-studio-
automation-testing-tool/
boration Platform for API Development.
Recuperado 4 de diciembre de 2020, de
https://www.postman.com/
[6] Katalon. (2020, 1 diciembre). Katalon Trial
and Free Plans. https://docs.katalon.com.
https://docs.katalon.com/katalon-
studio/docs/trial-free-plans.html
Katalon LLC. (2020, 18 marzo). Kata-
& Projects of any size. Katalon Solution.
https://www.katalon.com/pricing/

[8] Postman. (s. f.-a). Plans & Pricing. Re-

[3] HG Insights. (s. f.). Companies Using
cuperado 4 de diciembre de 2020, de
Katalon Studio, Market Share, Cus-
https://www.postman.com/pricing/
tomers and Competitors. Recupe-
rado 4 de diciembre de 2020, de [9] Katalon Studio. (2020, 27 mayo). SoapUI
https://discovery.hgdata.com/product/katalon- vs Postman, Katalon Studio: A Review
studio of Top 3 API Tools. Katalon Solution.
https://www.katalon.com/resources-
[4] D Perez. (2021,04 noviembre).¿Qué center/blog/soapui-vs-postman-katalon-
es Katalon Studio?. Recupera- api-tools/
do 4 de Noviembre de 2021, de
https://blog.appxolot.com/que-es-
katalon-studio/