2017 IEEE International Conference on Software Quality, Reliability and Security (Companion Volume)

A Journey from Manual Testing to Automated Test

Generation in an Industry Project

Claus Klammer and Rudolf Ramler

Software Competence Center Hagenberg GmbH
Softwarepark 21, A-4232 Hagenberg, Austria
{claus.klammer, rudolf.ramler}@scch.at

Abstract—Test automation is essential in fast-paced agile de-

velopment environments. The main goal is to speed up test execu-
tion cycles and to reduce the effort involved in running tests
manually. We took test automation one step further and applied
test generation to a GUI-based application developed in a large
industry project. The paper describes the transition from manual
exploratory testing to automated GUI test generation. Key les-
sons to be learned are: (1) the test automation pyramid proposed
for agile development tends to underestimate the need for high-
level GUI testing, (2) automated test generation does not reduce
test effort but shifts it to writing test adapters and checks, and (3)
the effort for analyzing results produced by generated tests limits
the practical application of automated test generation. The report
describes the successful application of test generation in a real-
world industry project, but it also highlights several open issues
to be addressed by future research on test automation.
Keywords—user interface testing; GUI testing; test case gen-
eration; test automation; industry experience report.
I. INTRODUCTION
Many organizations have made the transition from manual
to automated software testing. Test automation is a must in
agile development environments. Tests are automated to speed-
up execution cycles, provide prompt feedback, free testers from
repetitive tasks and reduce human effort. The main goal is to
automate the laborious task of running the tests. Defining tests
and implementing them as test scripts still remain manual
tasks. In our project we went further. We took the next step in
test automation and generated tests for graphical user interface
(GUI) components of a large industry system. This paper re-
ports on our experiences and shares the lessons we have
learned.
In the project we developed a human machine interface for
heavy industrial machinery. It includes several components
based on the JavaFX technology that are combined to a highly
interactive, touch-based GUI supporting intuitive operation,
navigation and visualization concepts. One of these compo-
nents is a virtual keyboard replacing conventional input devices
by flexible on-screen touch interaction. In the following we use
this complex, feature-rich GUI component as a representative
example to describe our journey from manual to automated test
generation. The report is based on a retrospective analysis of
the testing activities conducted in the project.
Explorative Testing Unit Testing
Test Adapter Dev. Manual Test Case Dev.
Test Case Generation Development Progress
100%
50%
0%
Phase 1 Phase 2 Phase 3 Phase 4
Figure 1. Relative effort per test activity in the different phases of automation.
II. RETROSPECTIVE ANALYSIS
Overall the duration of development was 7 months. Testing
in this time span can be divided in four phases. Figure 1 illus-
trates the distribution of the effort associated with different
testing activities in each of these phases. The phases and testing
activities are described in detail in the following sections.
Phase 1: Manual Testing
Testing in the first phase focused on writing unit tests and
manual exploratory testing at the GUI level. Unit tests were
written with JUnit for non-UI code and served as regression
tests in continuous integration and daily builds. They were the
foundation of our test automation. However, most features of
the virtual keyboard described in user stories had to be tested at
GUI level. A demo application with a sample GUI was created
for the virtual keyboard components. The demo app was used
for manual exploratory testing and to support the team in
demonstrating the development progress by showing which
new features have been implemented and how they work.
+ Manual testing with the demo application helped us to under-
stand and refine the requirements specified as user stories. In
addition, the demo app allowed us to easily explore the virtual
keyboard and to find bugs and look and feel issues.
– Most bugs were related to the GUI, hence our unit tests were
not able not find them. This was particularly problematic as the
development of new features often caused unrecognized side-
effects breaking existing features. Manual exploratory testing
via the GUI also missed these bugs as it was mainly conducted
for new features and regression testing was neglected.

978-1-5386-2072-4/17 $31.00 © 2017 IEEE 591

DOI 10.1109/QRS-C.2017.108

Authorized licensed use limited to: Volkswagen AG. Downloaded on January 08,2023 at 09:53:10 UTC from IEEE Xplore. Restrictions apply.
Phase 2: Automated Test Execution
The second phase is characterized by the development of au-
tomated tests for the GUI to fill the gap in regression testing.
We used TestFX that allows writing GUI tests with the estab-
lished JUnit framework. Furthermore, following the idea of the
page object pattern [1], we implemented a test adapter for the
virtual keyboard. It provides a programming interface to the
keyboard’s GUI so that it can be accessed from high-level GUI
tests without the need to interact with individual GUI elements.
In addition our adapter also provides an internal state model
that keeps track of the keyboard’s state. It serves as test oracle
and is used to check that transitions end in the expected states.
+ A set of automated GUI tests were developed representing
important usage scenarios and known error cases. Writing and
maintaining these GUI-level tests using JUnit and our adapter
was straightforward. Frequent execution of these tests lowered
the risk of critical regressions in ongoing development.
– Besides exploratory and unit testing for new user stories,
additional effort had to be put into developing the test adapter
and GUI tests. And even when they were available, the need
for manual regression testing was not reduced much since the
automated GUI tests covered only the common usage scenarios
without any interaction variations.
Phase 3: Automated Test Generation
The objective of the third phase was to automatically generate
GUI test cases to increase the variety of interactions in which
automated testing exercises the GUI of the virtual keyboard.
We used the code-based test generator Randoop[2] that has
already been successfully applied in industrial projects before
[3]. The tool creates JUnit test cases for the virtual keyboard’s
GUI in form of random sequences of method calls to the previ-
ously implemented test adapter. When tests are generated,
Randoop actually runs the generated sequence of method calls,
executes the tested system, and checks the outcome for unex-
pected exceptions and violations of assertions in the code. In
case of a violation a failing JUnit test is produced that indicates
a likely bug, otherwise the generated JUnit test will pass and
can be used as regression test for future versions.
+ A large set of GUI tests covering a wide range of interaction
scenarios was generated on demand whenever the virtual key-
board was changed. Based on the test adapter, these tests
checked numerous properties of the keyboard and helped to
discover several bugs including one in Oracle’s JavaFX code1.
– Enabling automated test generation again caused additional
effort, mostly for revising the test adapter and eliminating false
positives. We went through several cycles of generating tests
and revising the adapter. The main challenge in generating GUI
tests is ensuring that the system is completely reset after every
test. Dependencies between tests lead to flakiness and cascad-
ing faults. Another challenge lies in writing effective checks
for properties that are neither too generic nor too specific.
Phase 4: Managment And Maintenance
In phase four the virtual keyboard reached a stable state. De-
1 Pearson Education, 2010.
https://bugs.openjdk.java.net/browse/JDK-8176270
592
Authorized licensed use limited to: Volkswagen AG. Downloaded on January 08,2023 at 09:53:10 UTC from IEEE Xplore. Restrictions apply.
velopment activities mainly concerned maintenance and bug
fixing. The automated tests were run periodically to detect bugs
introduced by changes. A few new tests were written and occa-
sionally adaptations of the test adapter were required, trigger-
ing a re-generation of the GUI tests. The main testing activity
was exploratory testing conducted before every commit.
+ Automated regression tests are executed regularly. So far
they were able to detect one bug introduced by a change.
– Executing the generated tests became a long-running task,
first, because of the large number of tests, and second, because
of the time-consuming nature of GUI interactions. Thus, local
execution of the tests before every commit was not considered
viable. Instead the tests were run on the build server once a day
with the downside of long feedback cycles in case of failing
tests. This delay is particularly painful if tests are flaky.
III. CONCLUSIONS AND LESSONS LEARNED
(1) Don’t neglect automated GUI tests. The test automation
pyramid [4] proposed for agile development emphasizes unit
tests as the foundation of a solid test automation strategy and
discourages automated GUI testing. For developing GUI-based
applications as in our case this advice should be taken with
caution. Many of the critical bugs were only found with auto-
mated GUI testing that cannot be replaced by unit tests or labo-
rious manual exploratory testing.
(2) No free lunch in automated test generation. Test genera-
tion allows creating a large number of tests that can be execut-
ed automatically. However, for detecting bugs the tests require
a test oracle that defines what the correct expected results are.
Generated tests based on available checks of generic properties
did not detect bugs. Hence, considerable effort had to be in-
vested in implementing our own domain-specific checks.
(3) The limit of test generation is analysis effort. Automated
test generation for GUI applications inherits challenges from
both, test generation and GUI testing. The resulting tests tend
to be long-running, hard to understand, flaky and brittle. Test
generation effortlessly produces tests in large quantities, but so
these problems multiply and the effort involved in handling the
tests explodes. The limit of test generation is defined by the
capacity available for analyzing test results. It is reached when
the results are not analyzed in time before the next test run.
ACKNOWLEDGMENT
This work has been funded in the frame of the COMET
center SCCH (FFG #844597).
REFERENCES
[1] M. Leotta, D. Clerissi, F. Ricca, and C. Spadaro, “Improving test suites
maintainability with the page object pattern: An industrial case study,”
6th International Conf. on Software Testing, Verification and Validation
Workshops (ICSTW), IEEE, 2013.
[2] Randoop: https://randoop.github.io/randoop/
[3] C. Klammer, R. Ramler, and H. Stummer, “Harnessing Automated Test
Case Generators for GUI Testing in Industry,” 42th Euromicro Conf. on
Software Engineering and Advanced Applications (SEAA), IEEE, 2016.
[4] M. Cohn, “Succeeding with agile: software development using Scrum,”