Professional Documents
Culture Documents
Automation Testing
Automation Testing
CONTENTS
∙ Introduction
Automated
∙ Testing Fundamentals
∙ Automated Testing
‑ Unit Tests
‑ Mocking
Test code and test tools are as critical to a software application as 2. Creating architectural and low-level designs based on our
the application code itself. In practice, we often neglect tests because requirements
they add overhead to our development and build times. These
3. Writing code abiding by our designs
slowdowns affect our release cadence and disrupt our short-term
schedules (for long-term gains that are hard to see during crunch). The testing we perform correlates to these generalized steps and is
captured in the V-Model.
In this Refcard, we will look at the fundamentals of testing in an
Agile world and how automated tests can drastically improve the
quality of our applications. We will then look at two critical strategies
for reducing the execution time of our automated tests to keep our
builds lean.
TESTING FUNDAMENTALS
Rarely does a team outright reject tests, but in many cases, tests are
relegated to second-class citizens — and sometimes even treated as a
stretch goal — which comes at the price of:
• Instability
• Lack of confidence
• Hesitancy to change other’s code
1
REFCARD | AUTOMATED TESTING AT SCALE
The V-Model pairs each of the four non-coding steps with an Additionally, the quality and speed of each test tend to diminish in
accompanying type of test: proportion to the number of tests. In the following sections, we will
look at some of the practices for creating quick unit, integration,
1. Acceptance tests: Ensures the application meets the needs of
system, and acceptance tests, as well as some strategies for quickly
the customer and stakeholders
executing a large number of tests.
2. System tests: Ensures the application meets its specification
when deployed in a production-like environment UNIT TESTS
A unit in testing terminology is usually the smallest element of code,
3. Integration tests: Ensures the components of the application
such as a class in Object-Oriented Programming (OOP). Unit tests are
function according to their designs
usually white box tests, where we focus on exercising each statement
4. Unit tests: Ensures each unit that makes up the components in a method. These tests are commonly written using a JUnit-style
function according to its design framework and should be:
While the V-model provides a sound basis, the Agile movement has • Fast: Executes quickly
brought some much-needed additions to this foundation. • Independent: Shares little to no resources
• Concise: Verifies only one logical outcome
AUTOMATED TESTING
As the number of tests grows, it can be tempting to relax these
Prior to the Agile movement, tests were commonly performed
constraints, but it is important that we maintain a large number of
manually, with a Quality Assurance (QA) team deploying the system,
efficient tests rather than untidy ones.
injecting inputs, and inspecting outputs. While this did suffice for
some time, it was:
MOCKING
• Tedious Since we only focus on one unit at a time, sometimes we need to
• Monotonous mock — or create dummy representations, called stubs — other
• Error-prone dependencies.
• Difficult to repeat
This stubbing allows us to abstract the internals of our dependencies SCALING TESTS
and focus our tests on the unit being exercised. Additionally, stubbing While automated testing is essential, the number of tests can grow
makes each test independent since a new stub can be created for to the point where builds take hours and developers cannot obtain
each test case. As we will see, stubs can (and should) be used at all quick feedback. To remedy this situation, we must focus on scaling
testing levels. our tests properly by optimizing individual tests and parallelizing test
execution.
INTEGRATION TESTS
Integration tests are fairly similar to unit tests — and commonly OPTIMIZING TESTS
utilize the same JUnit-style frameworks — but instead of testing One of the most effective strategies for reducing the execution time
individual units of a system, they test groups of units, called of tests is to optimize individual test cases. As with any performance
components or modules. For example, if we create a commerce optimization, we must target the portion of the execution time that
system, one module may handle credit card transactions while will provide us with the most reward for our effort. Generally, test
another may handle shipping with a specific carrier. These modules execution time can be divided into three stages:
can then be aggregated to form the entire system.
• Setup: The time required to acquire and create the resources
that the test case will use and configure the environment
SYSTEM TESTS
System tests exercise the system as a whole by deploying it in a
production-like environment and then interacting with it like a
user would (from the outside). System tests tend to run longer than
integration tests, but there are usually fewer of them.
• Selenium For unit and integration tests, the most time-consuming stage will
• Serenity commonly be execution, as the setup and tear down phases of the
• Cypress test should only create simple objects or sets of objects. For system
and acceptance tests, on the other hand, setup (and possibly tear
ACCEPTANCE TESTS down) will commonly take up as much or more time as the test case
Acceptance tests are similar to system tests but they focus on the execution.
longer than system tests, but there will likely be fewer of them. as containers (like Docker), are generally faster to deploy and
should be favored over heavier-weight objects, such as Virtual for our effort. Tweaking a test case to the point where its purpose is
Machines (VMs). Additionally, orchestration frameworks (such obfuscated to reduce 2 seconds of execution time should be avoided
as Kubernetes) can generally configure networks and file when a change to the startup components can save us a minute.
systems quicker than VM orchestration tools.
2. Stub live services when possible: Mock any external services PARALLELIZING TESTS
whose behavior is known but whose implementation is While optimizing individual tests can greatly reduce execution time,
not directly needed. This may be more viable for unit and there are instances where further optimizations are not worth the
integration tests, since system and acceptance tests will effort or when each test runs quickly, but the number of test cases
commonly use the actual services needed in production, but causes the aggregate execution to grow beyond a reasonable level.
Regardless of the test phase, it is important that we target the part of As p and T increase, the execution time decreases. To increase
the test case that takes the longest and will provide the most reward parallelization, we must ensure that our test cases are independent
of one another. For example, if a test case depends on another test While parallelization can produce significant reductions in execution
case to execute first or if a test case depends on a shared resource, it time, there are some considerations to keep in mind:
can be very difficult to parallelize it. To increase the independence
• Output is interlaced: When a test case fails, it can be difficult
of each test case, we can do the following:
to see the output from the failed test cases since it will be in‑
terlaced with the output from other tests. It may be beneficial
1. Mock shared resources: Instead of depending on share
to execute the failed test again by itself (since the test case
resources, such as databases, stubs should be used instead.
should be independent) to see the relevant output.
This allows each test case to create their own stub and interact
with the individual stubs independent of any other test case. • Execution order will change: Some reporting or analysis
tools may display test cases in the order in which they were
2. Remove artificial ordering: Some test cases are artificially
executed. This ordering can — and likely will — change be‑
ordered, where the test case itself does not require ordering,
tween executions and should not be depended upon.
but we, as test authors, create an arbitrary ordering. For
example, we may artificially desire that we test the add
and subtract operations of a calculator before testing CONCLUSION
multiplication and division. Testing is often relegated to an inferior position behind application
code, but demotion can lead to a serious lack of quality. While
The thread count we select will depend on the nature of the tests, automated testing has reduced the burden of testing, it is not a
the resources available in our test environment, and the number of final solution. Instead, we need to combine efficient tests and
test cases that can be parallelized. For example, if we can only find parallelization to ensure we balance sufficient test coverage and
10 test cases that can be run in parallel, using 15 threads may not reasonable build times.
produce any noticeable speed-up. In most cases, we should tune the
thread count and then sample execution time until we approximate a
WRITTEN BY JUSTIN ALBANO,
maximum speed-up.
SOFTWARE ENGINEER, CATALOGIC SOFTWARE, INC.
Once we have parallelized our test cases and have determined a Justin Albano is a Software Engineer
at Catalogic Software, Inc. responsible
starting thread count, we can configure our build system to run our
for building distributed catalog, backup,
tests in parallel. For example, we can configure Maven using the and recovery solutions for Fortune 50
following build configuration: clients, focusing on Spring-based REST API and MongoDB
development. When not working or writing, he can be found
playing or watching hockey, practicing Brazilian Jiu-jitsu,
<build>
drawing, or reading.
<plugins>
<plugin>
<groupId>org.apache.maven.plugins
</groupId>
<artifactId>maven-surefire-plugin
</artifactId>
...
<configuration> DZone, a Devada Media Property, is the resource software developers,
engineers, and architects turn to time and again to learn new skills, solve
<parallel>all</parallel>
software development problems, and share their expertise. Every day,
<threadCount>10</threadCount> hundreds of thousands of developers come to DZone to read about the latest
technologies, methodologies, and best practices. That makes DZone the
...
ideal place for developer marketers to build product and brand awareness
</configuration> and drive sales. DZone clients include some of the most innovative tech‑
nology and tech-enabled companies in the world including Red Hat, Cloud
</plugin>
Elements, Sensu, and Sauce Labs.
</plugins>
</build>
Devada, Inc.
600 Park Offices Drive
Suite 150
Research Triangle Park, NC 27709
Maven can be configured to execute test cases (test methods) or 888.678.0399 919.678.0300
test classes — as well as test suites — in parallel. In the case of the Copyright © 2020 Devada, Inc. All rights reserved. No part of this publication
may be reproduced, stored in a retrieval system, or transmitted, in any form
configuration above, we selected all, which allows both test methods or by means of electronic, mechanical, photocopying, or otherwise, without
prior written permission of the publisher.
and test classes to be executed in parallel. When possible, we should
use the all option so ensure the greatest flexibility.