Andrew Knight - 7 Major Software Testing Trends

7 Major Software Testing Trends
Pandy Knight
Automation Panda
Applitools Developer Advocate
Test Automation University Director
7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

suki_thefrenchie_

Tools and technologies may change,
but fundamentals remain the same.

Trend #1:
End-to-end web testing
is a three-way battle.

Selenium
Popular among testers
C#, Java, JavaScript, Ruby, Python
All major browsers
WebDriver protocol
Low-level browser interactions only
Middle-speed execution

Selenium Cypress
Popular among testers Popular among developers
C#, Java, JavaScript, Ruby, Python JavaScript only
All major browsers Chrome, Edge, Firefox, Electron
WebDriver protocol In-browser JavaScript
Low-level browser interactions only Full framework with advanced features
Middle-speed execution Slowest execution

Selenium Cypress Playwright
Popular among testers Popular among developers The dark horse
C#, Java, JavaScript, Ruby, Python JavaScript only C#, Java, JavaScript, Python
All major browsers Chrome, Edge, Firefox, Electron Chromium+, Firefox, WebKit
WebDriver protocol In-browser JavaScript Debug protocols
Low-level browser interactions only Full framework with advanced features Full framework with advanced features
Middle-speed execution Slowest execution Fastest execution

Selenium Cypress Playwright
Open source Open source Open source
Open standards (WebDriver) In-browser JavaScript Debug protocols
Open governance Run by Cypress.io Run by Microsoft

Other web testing tools
● Webdriver IO remains very popular
● Protractor is deprecated and will reach end-of-life in 2023
● Puppeteer’s spiritual successor is Playwright
● Selenium IDE is being redeveloped as an Electron app
● Other tools exist but have small footprints

Trend #2:
Component testing
is the new integration testing.

17
When testing anything for the web, we believe that
tests should view and interact with the application in
the same way that an actual user does.
Anything less, and it's hard to have confidence that

your application is doing what it is supposed to.
Source: https://www.cypress.io/blog/2022/06/01/cypress-10-release/

We want the best of both worlds:
The simplicity and sensibility of manual testing.
The speed and scalability of automated testing.

Trend #3:
Visual testing
has become indispensable.

Modern Cross-Platform Testing

Trend #4:
Performance testing
is easier and more vital than ever.

Chrome DevTools Performance Tab
TAU Course:
28

Google Lighthouse
29

Playwright Performance Testing
John Hill’s Talk:
https://bit.ly/3xKQqqT
const client = await page.context().newCDPSession(page);
await client.send('Performance.enable');
await page.goto('https://www.google.com/');
await page.click('[aria-label="Search"]');
await page.fill('[aria-label="Search"]', 'playwright');
await Promise.all([
page.waitForNavigation(), John Hill’s Code:
https://bit.ly/3zCuaSq
page.press('[aria-label="Search"]', 'Enter')
]);
let perfMetrics = await client.send('Performance.getMetrics');

console.log( perfMetrics.metrics );
Source: https://github.com/unlikelyzero/js-perf-toolkit

Trend #5:
Testing needs to be
faster than ever.

The Rule of 1’s
For 1 test
UI Test ≈ 1 minute
API Test ≈ 1 second
Unit Test ≈ 1 millisecond

The Rule of 1’s
For 1000 tests
1K UI Tests ≈ 17 hours
1K API Tests ≈ 17 minutes
1K Unit Tests ≈ 1 minute

Ways to speed up test execution
1. Focus on most valuable behaviors instead of complete coverage
2. Push tests down the Pyramid
3. Optimize automated operations:
a. Run tests in parallel
b. Prep data using APIs
c. Cache login credentials
d. Use smart waiting instead of hard sleeps
e. Limit each test to covering one individual behavior

Ways to speed up test development
1. Use a mature framework with good Developer Experience (DX)
2. Apply visual testing techniques for assertions
3. Require test automation as part of the Definition of Done
4. Bridge the gap between tester and developer

Continuous Integration, Testing, & Delivery
White Box Tests

Commit Build Black Box Tests Delivery
Static Analysis
Run this pipeline for every commit (or small batch of commits)
Target max execution time ≈ 15 minutes

Trend #6:
Machine learning algorithms
need separate approaches.

Amazon’s
Recommendation
System

TAU Course for Testing Machine
Learning Models
https://testautomationu.applitools.com/testing-machine-learning-models/
39

Trend #7:
Autonomous testing is coming.

The First Wave: Manual Conversion
● Historically, testing was done manually
○ Teams built huge repositories of step-by-step test procedures
● Automation enabled tests to run faster and more frequently

○ Start by automating existing manual tests
○ Focus on improving test execution
○ It was common to run tests after a nightly or weekly build
● Manual conversion had learning pains

○ Long tests became fragile scripts
○ Many teams tried to achieve 100% automation

The Second Wave: CI/CD/CT
● Continuous Integration enabled tests to run continuously
○ The focus became triggering tests
○ Fast feedback caught problems sooner
○ Full embrace of Agile and DevOps
● Teams changed their approach to automation

○ Unit and integration tests became just as important as end-to-end tests
○ Black-box tests covered individual behaviors rather than grand tours
○ Speed and reliability became more important than coverage
● The barrier to entry was high

○ Developing automation required full software development skills
○ Some teams rose to the challenge while others failed

The Third Wave: Autonomous Testing
● Testing = Interaction + Verification
○ In the first two waves, humans had to define both parts of the equation
● Autonomous Testing is when automation figures out what and how to test
○ AI agents learn behaviors in the product under test
○ Then, they automatically figure out the interactions to trigger the behaviors
● Autonomous Testing will be a gamechanger

○ We aren’t “there yet,” but it’s coming in the next few years
○ We already are partially there (with tech like visual assertions)
○ Success will require learning from testers and coaching testers

7 Major Software Testing Trends
1. End-to-end web testing is a three-way battle
2. Component testing is the new integration testing
3. Visual testing has become indispensable
4. Performance testing is easier and more vital than ever
5. Testing needs to be faster than ever
6. Machine learning algorithms need separate approaches
7. Autonomous testing is coming
Twitter Blog Puppy TAU

Andrew Knight - 7 Major Software Testing Trends

Uploaded by

Copyright:

Available Formats

You might also like

Andrew Knight - 7 Major Software Testing Trends

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Andrew Knight - 7 Major Software Testing Trends

Uploaded by

Copyright:

Available Formats

7 Major Software Testing Trends

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

C#, Java, JavaScript, Ruby, Python

All major browsers

Low-level browser interactions only

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

C#, Java, JavaScript, Ruby, Python JavaScript only

All major browsers Chrome, Edge, Firefox, Electron

WebDriver protocol In-browser JavaScript

Low-level browser interactions only Full framework with advanced features

Middle-speed execution Slowest execution

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

WebDriver protocol In-browser JavaScript Debug protocols

Middle-speed execution Slowest execution Fastest execution

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

Open standards (WebDriver) In-browser JavaScript Debug protocols

Open governance Run by Cypress.io Run by Microsoft

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

Anything less, and it's hard to have conﬁdence that

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

The simplicity and sensibility of manual testing.

The speed and scalability of automated testing.

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

let perfMetrics = await client.send('Performance.getMetrics');

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

API Test ≈ 1 second

Unit Test ≈ 1 millisecond

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

For 1000 tests

1K API Tests ≈ 17 minutes

1K Unit Tests ≈ 1 minute

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

White Box Tests

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

● Automation enabled tests to run faster and more frequently

● Manual conversion had learning pains

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

● Teams changed their approach to automation

● The barrier to entry was high

7 Major Trends in Software Testing | Andrew Knight | @AutomationPanda | AutomationPanda.com

● Autonomous Testing will be a gamechanger