Shift to a Holistic

Test Automation with

Artificial Intelligence
Understanding the challenges to achieve end-to-end
test automation and a way forward with AI
 At a Glance
To Thrive in a world of constant change, enterprises are banking
on technology needs to be agile to ward off disruption and stay
competitive and relevant. The other challenge that
organizations face is how to accelerate their time-to-market of
their products or services while ensuring quality for delivering
the best customer experience. Quality in this competitive
environment has now become more critical than ever, and the
slightest miss could have a snowballing effect on the business
outcomes of the organization. To ensure credibility, brand value,
customer engagement, and speed of delivery it is important to
look beyond the conventional methods of testing.

Many forward-thinking organizations have been working to

optimize testing using automation to overcome the perennial
challenges related to quality assurance, but many of these
current automation processes and practices have their
limitations, and they have not been able to upkeep with the pace
of Agile and DevOps development which needs speed and
precision.

To address the needs of Quality and Speed in modern and

advanced software development, AI is heralding a new era,
where the testing community is now moving from just Quality
Assurance – which was very reactive in nature to Quality
Engineering – to bring in a more proactive approach towards
delivering better RoI from testing investments.

To bridge the gap between current testing needs and potential

automation solutions, this white paper identifies
challenges that act as barriers to end-to-end
automated testing and the possible solutions with
AI/ML.

01
Contents

01 Introduction 03

02 End-to-End Test Automation 04

03 Overview of Artificial Intelligence 06

& Machine Learning

04 Intelligent Test Automation 07

05 Summary of the benefits of AI/ML 10

in Test Automation

06 Conclusion 11

02
Introduction
Application testing has been disrupted vastly
in the last decade, in which delivery timelines Here’s a snapshot of the
have shortened to few days. The shift left evolution of test automation
approach is embedded largely in DevOps and
agile to test applications right at the outset.
The series of these new and accelerated
approaches toward SDLC are now set to
radically change Quality Assurance (QA).
Record & Playback
Ever-increasing and high end-user
expectations demand an evolutionary
disruption in application development.
Quality is no more just a function to test the
work ability or usability of the code. It must
ensure that the end-users experience is Verification
enhanced and differentiated from the Checkpoints
competitors. Hence, the approach is evolving
from Quality Assurance to Quality
Engineering.

To ensure continuity in business value, that is

time, cost, and quality with dramatic shifts in Data Driven
technology, quality engineering must evolve Approach
into a real-time and insight-driven function in
the SDLC.

Today, Quality Teams must ensure

end-to-end coverage across application Reusable
portfolio, as cutting-edge industrial Functions
technologies are introduced across scalable
architectures, with increasingly complex
layers of technology abstraction. The solution
for this problem is testing automation, but
the current state of test automation fails to Unattended
adapt to the ever-accelerating software Execution
development.

These challenges can be solved by using AI in

automation. AI can be used for writing test
cases, generating data sets, analyzing the Continuous
test results, etc. In other words, AI can make Integration
our automation smarter.

AI technology can help fill in the gaps and

enable test automation evolution. The
learning and adapting ability of AI/ML can Low Code
change the QA process from the highway to
the flyway.
Automation

With AI, testing will witness a shift from the

traditional approaches towards innovative
ideas and new techniques fit for the
futuristic applications.
AI/ML Based Test
Platforms

03
End-to-End Test Automation
Before we move to the limitations of the current state of test automation, let’s look at the typical
stages of test automation flow:

Typical stages of test automation

01 02 03
Determining The Define The Right Test Plan + Tool
Scope Of Automation Selection
Test Automation. Strategy

04 05 06
Setting Up Automation Test Test Reports
The Test Script Analysis +
Environment. Development + Script Sustenance
Execution.

With the conventional test automation

process, almost every stage needs human
intervention, which adds significant time
and cost to every test release cycle.

04
Let us take a close look at the challenges in
achieving end-to-end test automation.
01: Demanding Human Capital
Test automation requires niche technical skills to
design and maintain a test automation framework
and test scripts precisely, build solutions and
resolve technical issues. Assembling such an
experienced team within a short time is often a
challenge.
A report states a shortage of QA engineers in
companies – 44% of respondents say they have
fewer than 1 QA engineer per 10 developers in their
projects. Only 10% have more than 4 QA engineers
per 10 developers.
02: Selecting the appropriate automation tool
26% of companies surveyed indicate finding the
right tools for test automation as the biggest
challenge they face. With a wide range of test
automation tools, both paid and open-source,
choosing the one that best suits your project
requirements is a challenge. 48.2% of automation
engineers for tool selection criteria are concerned
about functionality and feature-richness.
Operating the selected automation tool with
optimal efficiency is another concern. An
enterprise must check the availability of quality
engineers with the required skill set to use the tool
effectively. 45.9% cited the level of programming
skills required to operate the tool as a key
decision-maker.
03: Inconsistent Test Failures
A test suite/script may fail due to an application
error or a poorly designed test code. Tricentis has
found that a staggering 72 per cent of test script
failures are false positives. It takes hours to figure
out the exact issue, which keeps the quality
engineer from delivering and releasing the new
code on time with test automation.
False Positives
FAILED
72%* ...Failures of automated tests occur when the
software is even functioning properly
33% Automation Issues Maintenance & Flaky Tests
23% Test Data issues
16% Test Environment issues
28%
“Real” Errors
05
04: Continuous Script Maintenance
According to the World Quality Report, 65% of
respondents said that applications change too
frequently for them to keep up.
Script maintenance is essential as the application
code changes would break existing tests. In test
automation, simple UI and flow changes on the
front end may be a single-line code, but it takes
several hours to fix a test script for the change.
For instance, If a button for submitting form is
visually updated with its functionality unchanged,
a test script would still fail.
05: Ensuring optimal coverage
Executing an entire regression test suite is not
feasible as it evolves with application changes and
adds to the suite size and execution time.
Selecting optimal coverage and ideal automated
test scripts that will ensure the stability of the
application after changes is a daunting task.
06: Maintaining Test Suite
A test automation repository is a collection of
previous test suites to be performed at various
stages of QA. These test suites include but are not
limited to a modification in a particular functional
area, sanity testing, smoke testing, regression
testing, and sprint closure. Maintaining the
records of such test scripts and various test suites
as an application keeps modifying with evolving
user needs. It is a daunting task to keep track
unless a rigorous methodology has been followed
from the beginning.
07: Building automation tests from Scratch
While most of the code is reusable in developing
software, the quality engineer has to build new
testing suites/frameworks from the start. It takes a
significant time to write the same test script
repetitively for every new project. For instance, a
tester will need to construct page objects or
develop data models for web pages in every web
automation project.
Due to a lack of time and resources, it is difficult to do rigorous application testing. The cost and effort are
significantly increased by manually designing, carrying out, and automating repetitive testing for the broad
project scope. This affects go-to-market and overall testing quality negatively. In addition to requiring a lot
of human labor, hand-crafted testing is prone to errors and discrepancies.

AI would provide more reliable outcomes. Scalability problems for manual testing need managing several
computers at once. Manual quality engineers cannot immediately assess how agile developers' continuous
inputs and modifications to the applications have affected the current user experience. There is no support
for creating and improving test cases, prioritizing testing and automation, improving UI testing, or
minimizing time-consuming analytical tasks. A domain expert is required for testing activities. There is a
demand for people who can increase headless automation and evaluate and comprehend intricate data
structures, statistics, and algorithms.

AI is the perfect solution to remove redundant work and human labor. Let us understand how AI can be a
game-changer in test automation.

Overview of Artificial Intelligence

& Machine Learning
“Powered by AI” is a phrase that is
omnipresent. From digital paintings to
medical applications, to playlists
suggestions, the list goes on and on.
Likewise, there are numerous applicable use
cases of AI and ML in Intelligent Test
Automation, so it is helpful to first
understand what these concepts mean and
how they vary from one another.
Deep Learning is a subset of machine
learning which uses on multilayered neural
networks to accomplish prediction /
classification.
Data Science which forms a foundational
layer for all AI/ML helps systems to uncover
knowledge, features and patterns in noisy,
structured, and unstructured data.

What started as a method for creating

systems that imitate human behavior or ificial Intelligenc
Art e
decision-making has now taken a much
more complex and evolutionary shape, now
popularly known as AI.
hine Learnin
Mac g
Machine learning is a subset of AI, which
employs data to solve tasks. These
problem-solving ML Models are trained ep Learning
against real training data to absorb De
knowledge/ features of data.
The probability theory, statistical modelling
and linear algebra are the key drivers of this Data
knowledge/ feature generation. Based on Science
the knowledge/features, ML model can
consume further data to perform prediction /
classification automatically.

Let us further explore how AI/ML can help in transforming the

current state of testing automation.

06

Intelligent
Test Automation
Use cases of AI/ML for Complete
Testing Automation
USE
CASE 01 Optimize Test Suites Using ML Clustering
A regression test suite is a collection of test cases
created by several quality engineers over a period.
As it involves multiple quality engineers, a
regression test suite consists of several
overlapping test cases. It is difficult for an
automation test engineer to classify similar test
cases or pinpoint an original test case, adding to
the execution time and human labor.
USE
CASE 02 Code-Less or Low-Code Test Automation
In conventional test automation, automated test
scripts are developed manually. It depends on the
selected automation tool and the software
development cycle for the project. Then, an
automation test engineer creates an automation
framework, scripts and interprets the test cases.
This process consumes a significant amount of
time and cost.
By using NLP and pattern recognition, enterprises
can solve the challenge. NLP and pattern
recognition enable the creation of reusable
business process components and the conversion
of manual test cases into automation scripts by
reading them.
07
Many problems have been solved so far with
the help of contemporary test automation
approaches, but it is insufficient in the
current VUCA (Volatility, Uncertainty,
Complexity, and Ambiguity) world. So, let's
now look at how various artificial intelligence
(AI) or machine learning methods can be
used to reduce such restrictions and
improve automated testing.
By using an ML clustering technique, Natural
language processing (NLP), similar tests can be
identified and grouped. It reduces the regression
test suite significantly and also omits the risk of
not covering optimal test cases for delivering a
reliable code.
The sizable decrease in the regression test suite
reduces an automation engineer's execution time
and efforts.
The NLP also enables inspection of the produced
test automation suite. Pattern discovery
algorithms and supervised ML approaches
identify and modularize reusable components.
Enterprises can implement AI/ML regardless of
the test automation tool they use. According to
the needs of a system, an automation test
engineer can go through the procedures, add
data, and create verification points.
The low-code or codeless test automation makes it
easier to maintain an automated test suite that is
well-structured, easy to reuse, and has end-to-end
traceability. The technology decreases the amount
of planning and scripting required significantly.
 USE
CASE 03 Automated Locator Identifier and Modularization
As human users, we interact with applications
visually. We see, scroll, click, and type through an
application. However, test automation interacts
with an application programmatically. It needs a
program developed as a way to find and modify
those same elements. Traditional automation
does not scan applications as humans do.
An automation test engineer needs a robust
solution to find the right elements in run time.
However, tracking the right elements with current
test automation is challenging.
Using AI, enterprises can implement element
locators. An element locator is an object that
searches and returns elements on an application
page using a given query. In short, locators find
elements.
These element-locating tools are trained against
different techniques and a large data set of various
sorts of applications. These locators can then be
employed in automated test scripts.
USE
CASE 04 Self-Healing Test Scripts
Quality engineers must tackle increasing new
page objects and modified identifiers when the
application evolves. Whether testing is manual or
automated, updating element locators for the
Application Under Test (AUT) is a required step
when modifications are made. It helps ensure
regression flows without conflicts and does not
impact functionalities.
Instead of failing a test due to missing elements,
AI-based self-healing is a technique that solves the
problem of identifying elements as the product
evolves, and modifies necessary elements.
Self-healing can be considered as automation for
automation. Testing tools with self-healing enable
detecting any modifications in the application
pages and automatically modifying with new
attributes to ensure the designed test cases are
functional.
08
AI can also help in visual testing applications using
Optical character recognition and visual locator of
an element.
The Page Object Model design pattern is
frequently utilized for test automation to provide
legible and manageable test scripts. An abstracted
page class transforms a page's core functional
flows into high-level flows that test scripts can
trigger. While developing these page classes for
each page in our application involves a significant
amount of code, this strategy has been quite
successful. Unfortunately, as a result, real test case
automation is frequently delayed.
Machine learning models can be developed to
generate page classes directly as a part of code
creation. Any test automation project can leverage
this as a great springboard.
An AI-enabled test automation framework:
01 Recognizes the smallest of modifications and
implements the necessary corrections.
Alternate locators can be revealed in the test
report or updated in real time while the script
is modified.
02 Determines if a failing step is a random failure
and if the script needs to try the action again.
Additionally, this can resolve all problems with
application synchronization that lead to losses.
03 Handles any unforeseen issue or alert
automatically. Corrective action is made
possible by leveraging artificial intelligence.
Frameworks for test automation would benefit
immensely from the addition of a model that
has been trained to comprehend errors or
warning popups and interpret corrective
actions.
04 Manages the test data that was formerly
reliable but is no longer acceptable in the
application. Deriving the data model from the
application, supported by synthetic data
generation and mapping it with each
component, can set a building block for
dynamic test data management. Automated
test scripts can be trained to update test data
and the associated expected result in the same
way a person would by using artificial
intelligence and machine learning.
 USE
CASE 05 Smart Execution

As regression test suites keep changing with Some of the ML model features that can drive
application changes, the number of test cases for smart regression are as follows:
optimal coverage remains a question. Ideally, tests
should be functionally classified and selected Test selection based on
01
based on the previously conducted test impacted functionality
information. The lack of such processes can cause
delays or bug leakages. 02 Frequently failed tests
AI can help to build models with multiple variables
from multiple sources that are not humanly 03 Updated Code coverage
possible. It also helps to pinpoint overlapping and
original test suites by which an AI/ML model can
select the exact size of regression test suites. 04 Developer who wrote respective code

This regression test suite is far more optimal

coverage and reduced in size, which executes 05 Technical updates, if any
more quickly.

06 Integrated system updates, if any.

USE
CASE 06 Intelligent Analysis

Standard test reporting can include information
about how a test was executed, any faults that
occurred, and other pertinent facts at the test
level.
However, it frequently fails to mention an analysis
of overall quality and trends. A dashboard with
AI/ML capabilities can forecast overall quality
accurately and offer deeper analysis with trend
and anomaly detection. In addition, the AI/ML
model could be trained for different aspects by
assembling data from sources such as code
control systems, requirement/ALM software, test
management tools, defect management software,
technological advancements, developer
capabilities, etc. It will assist enterprises in
accurately using prediction models to manage
quality costs across the QA process.

Auxiliary AI/ML use cases in test automation

There are numerous additional uses of AI/ML in complete test automation.
Here are a few to name.

Application crawling with AI/ML to develop intelligent test suites

Using bots with AI/ML to perform exploratory testing

Early automation based on Design Docs and Mock Screens

NLP-based requirement analysis, test scenario development

Replicating automation scripts to accommodate more digital channels (e.g., converting web
Application-specific automation scripts to mobile application-specific automation scripts)

Automatic code conversion between programming languages, such as Java and C#

09
Summary of the benefits
of AI/ML in Test Automation

Stable UI Tests
AI is dynamic and modifies UI test
code as soon as it observes 01
real-time changes in an
application, which results in faster
and stable UI tests.
Real-time Tests Optimization
AI detects errors within the test
02 scripts and adjusts the test cases
before execution. This proactive
approach saves time and
ensures stability.
Continuous Learning
AI constantly observes user
interaction with applications and
learns common behavior.
03
It accordingly suggests and
reuses components based on
real-time data. Smart Test Execution
AI uses dynamic element
04 locators and creates reusable
components to execute
data-driven tests, developing
Reduced Dependencies test automation scripts quickly.
AI automation records server
responses during testing over a
period and write test scripts
based on stored responses, which
05
makes the testing process faster;
it does not depend on
implemented or unimplemented Faster Product Release
test scripts. AI automation works in parallel
06 with your QA team, which
reduces testing time and
identifies issues sooner so that
you can release products faster.

10
Conclusion
Its now not about ‘How’ with AI and ML anymore, AI and
ML is the ‘Now’ of the testing automation! AI and ML in
Testing Automation are not a research concept
anymore but a very viable application to solve complex
quality challenges that optimizes the RoI and
minimizes the time to market while delivering a
top-notch customer experience.

AI-based testing can be performed effectively by

leveraging tools that use data and algorithms to design
and perform the tests without any human intervention.
It helps to provide a robust system positively affecting
the RoI. The advanced AI/ML enables test automation
for optimal test coverage, test accuracy, and faster time
to market, and provides a significant decrease in overall
testing costs.

AI/ML in test automation is now very much accessible

for organization of any shape and size. Business
Leaders now need to move to explore and adopt AI/ML
to ensure credibility, brand value and customer
engagement and speed of delivery through Quality
Engineering.

Resources
1. World Quality Report 2021-2022 and World Quality Report 2020-2021
2. STATE OF TEST AUTOMATION 2020-2021 EDITION By Kobiton
3. https://www.simform.com/blog/state-of-test-automation-report/
4. https://research.aimultiple.com/test-automation-statistics/#automated-tests-adoption
5. https://info.digital.ai/rs/981-LQX-968/images/SOA15.pdf?_ga=2.139462772.672607512.1661577187-2046299366.1661577187
6. https://www.jetbrains.com/lp/devecosystem-2021/testing/
7. https://www.stickyminds.com/article/test-automation-matures-so-do-false-positives

11
About TRUGlobal Quality Engineering Practice
Our Automation First approach has best in class end-to-end ecosystem with intelligent and Cognitive QA
processes. We deliver "Quality with Speed" to promote faster business and technology change along with
best customer experience.

To partner with our fortune 500 customers in their Digital Transformation journey, we have built various
solutions across four areas At TRUGlobal, we are providing “Best in Class” QE services and building Next
Generation Digital QE solutions like Intelligent Automation, Performance Engineering, Security testing, Hyper
Automation, etc. to enhance UX and meet business goal of our esteemed global customers

With our self-healing, code-less automation platform, along with smart execution and
intelligent analysis, we are well placed to provide AI-ML enabled end-to-end automation
solutions for all the diverse needs of our clients. If you want to have a demo or a discussion with
us, reach out to us at: intelligentautomation@truglobal.com

About the Authors

Rajeev Sawant Anant Joshi

Automation COE Head Practice Head-Intelligent
Automation

Rajeev is a seasoned IT professional with 20+ years
of diverse experience in Quality Engineering /
Digital Transformation, AI-ML testing, Cloud
migration testing and managing large QE
engagement. His core areas are Intelligent
Automation, AI-ML, RPA, Chaos Engineering,
Performance engineering, Security testing, CI-CD,
DevOps, and Pre-Sales. Rajeev is currently heading
the TRUGlobal automation practice and has built
various solutions like Self-Healing, script-less
Intelligent Automation platform, AI/ML testing,
data pipeline testing etc. He has a very strong
domain expertise in BFSI, Retail, Manufacturing &
Utilities industry.
Anant has 23+ years of IT experience in setting up
large Testing practices, TCoE, and managed
service engagement. His core areas are in Test
Consultancy, Transformation management, BU
head management & E2E Pre-Sales. Anant’s
technical skills are in Automation, Performance
engineering & security testing. Anant is current
heading TRUGlobal QA practice and have built
unique solutions across testing Life cycle in Digital
Testing, Intelligent Automation, AI/ML, Cognitive
QA etc. He has strong domain expertise in BFSI,
Manufacturing, Retail, Gaming & Life Science
industry and worked in large MNC like Cognizant,
BNY Mellon, Zensar etc.

Rajeev have worked across the globe to deliver IT
solutions for fortune 100 customers prominently in
USA, UK, APAC, SA region. Rajeev is pursuing his
Masters in AI-ML and is a certified Scrum Master,
PMP, and DevOps leader.
Anant has extensively worked in USA, UK, Europe,
SA region for fortune 100 customers & has proven
track record for providing value added services.
Anant has done his B.E Electronics and holds
certifications in CEH,ISTQB, QAI.

TRUGlobal is a top IT services firm servicing clients across

Fortune 500 Companies to Startups.

for more information please contact us at info@truglobal.com

or visit www.truglobal.com

12