2012 IEEE 36th International Conference on Computer Software and Applications Workshops
Strategies for Agile Software Testing Automation: An
Industrial Experience
Eliane Collins Arilo Dias-Neto
Nokia Institute of Technology Federal University of Amazonas
Manaus, AM, Brazil Manaus, AM, Brazil
eliane.collins@indt.org.br arilo@icomp.ufam.edu.br
Abstract—Testing Automation has been growing in software
engineering. Many organizations are investing in automated
testing in order to prevent defects and increase testing
effectiveness during software development. In agile methodologies,
this task is considered an important activity, considered the key
of the agile testing. This paper presents three testing automation
strategies applied to three different software projects adopting
Scrum agile methodology. The results indicated positive agile
practices to be considered when adopting testing automation
strategy, such as: team collaboration, task distribution, testing
tools, knowledge managements. The challenges, results, and
lessons learned from this experience are also discussed.
Keywords- software testing; agile testing; testing automation;
open source testing tools.
I. INTRODUCTION
Software Testing Automation means to automate software
testing activities including the development and execution of
test scripts, verification of testing requirements, and the use of
automated testing tools [1]. One of main reasons to automate
tests indicates that manual testing is time consuming. On the
other hand, automated testing increases efficiency for repetitive
steps to reproduce system functionality, especially in
regression testing, where test cases are executed iteratively and
incrementally after changes made in the software [2].
Testers must carefully define and analyze the testing
automation strategy in a software project, because the success
of this practice requires effort to know the testing tools and
verification of tests results to ensure that the cost of automation
is less than the cost of manual tests execution [3].
In this scenario, testing automation is considered an
important practice in software development, and this
importance is highlighted in some software development
methodologies. For instance, it is considered the key of the
agile testing and, even with an initial investment, its benefits
can be observed in the long term in a software project [4]. The
agile methodologies are standing out mainly because of their
ability to rapidly accommodate changes in the original
requirements and prioritize the development of functionalities
through executable code instead of extensive written
documentation. They also provide collaboration with the
customer rather than following rigid plans and contract
negotiations. Agile processes are iterative. Thus, in this scenario,
testing has to follow the software project iterations. It needs to
978-0-7695-4758-9/12 $26.00 © 2012 IEEE 440
DOI 10.1109/COMPSACW.2012.84
Vicente F. de Lucena Jr.
Federal University of Amazonas
Manaus, AM, Brazil
vicente@ufam.edu.br
be executed efficiently by the use of automated testing tools,
providing quickly responses to the requirement changes, and
providing efficient interaction with the development team [5].
However, testing automation in agile software projects has
some problems to be managed [6]. According to Crispin and
Gregory [6], analyzing their experiences in agile teams, issues
that influence the success of testing automation would be:
• Programmers do not bother to test because they have
the test team as a safety net to catch bugs;
• The “Hump of Pain”: the learning curve for testers to
learn about tools and code;
• Initial investment: acquiring tools and training people;
• Code and interface in constant changes: making the
GUI automation not unfeasible;
• Legacy systems code is not designed for testability;
• Fear: testers without programming background and
programmers without previous experience in testing;
• Old habits: manual regression tests instead automated
regression tests suite;
Based on this scenario, this paper presents experiences and
results of the implementation of three testing automation
strategies in three different software projects following the
Scrum agile development methodology [7]. This paper is
organized as follow: Section 2 describes the background
explaining agile testing and related works on testing
automation. Section 3 presents the testing process adopted by
the company and details of automation testing strategies
applied for each software project. Section 4 discusses the
results and lessons learned, and finally, section 5 presents the
conclusion and future works.
II. BACKGROUND
A. Agile Testing
According to [6], agile testing not just mean testing on agile
projects but testing an application with a plan to learn about it
and let the customers’ information guides the testing activities
in line with agile values. According to the authors, the key
factor to reach success in an agile testing approach would be:
• Looking at the Big Picture;
• Collaborating with Customer;
 • Building the Foundation for Agile Core Practices;
• Providing and Obtain Feedback;
• Automating Regression Testing;
• Adopting an Agile Testing Mindset; and
• Using the Whole Team Approach.
Thus, testing automation is considered the core of agile
testing, independently of the agile methodology implemented
in a software project. In this scenario, different tests have
different proposes. Figure 1 shows the agile testing quadrants
proposed in [8], describing the relation between manual and
automated testing for different type of testing.
Figure 1. Agile Test Quadrants [8]
• Q1: Technology Facing tests in Support of
Programming (Technical requirement testing – Does
this method do what the developer intended?)
• Q2: Business Facing tests in Support of Programming
(Business requirements testing – Does the code do
what it should?)
• Q3: Business Facing tests to Critique the Product
(Business defect testing – Does the code do something
it shouldn’t? Are there missing requirements?)
• Q4: Technology Facing tests to Critique the Product
(Technical defect testing – Are there leaks? Can it
handle a load? Is it fast enough?)
Traditional software testing focuses almost exclusively on
the right side. Thus, the functional tests are executed critiquing
the product but not playing a productive part in supporting the
creation of the product. Moreover, test activities are involved
late in the development process to detect failures but not to
prevent them.
In agile testing, the team is involved in identifying, but also
preventing failures. Thus, agile testing is a challenge for testers
that used to work in traditional projects waterfall mainly
because they do not have to wait for system delivery in the end
of project to start the test activities but they need to be
441
proactive and start the test tasks since the beginning of the
project together with the developers as one team and be
succeed in this challenge requires experience, overcome
barriers and mind set.
B. Testing Automation
Automated software testing is a solution to the problem of
maximizing test coverage within the available time and
increasing software reliability and quality [9]. The benefits of
such approach compared to the manual testing would be [9]:
low cost of test execution, possibility of replicating old test
sequences on new software versions (free cost of test
regression) and the possibility of testing software in a high
stress level for a long time.
Another related study ([10]) describes the developer’s
experience performing testing automation on component level
and Test Driven Development (TDD) in order to prevent
failures and decrease the development lead-time. The authors
discuss aspects and lessons learned to be considered in a testing
automation strategy, such as:
• Testing automation requires high product testability;
• The use of a framework makes test case development
easier and the automation robust to future changes;
• Testing automation on a component level requires
adjustments to the product architecture;
• Asking people to add new tests every time a new
defect is found requires continuous remainders and
monitoring;
• Test execution speed is important;
• Test tools easily become the excuse for every problem;
• Testing automation requires dedicated resources;
• Benefits from testing automation are hard to obtain in
the first project release;
• Testing automation should be introduced in small
steps. Minimizing maintenance costs is the most
difficult challenge in testing automation.
In [11], the authors present a survey that observed factors
that affect the state of testing automation in different types of
software organizations. This study included a survey in 31
organizations and a qualitative study in 12 of them. The survey
revealed only 26% of test cases in these organizations is
automated, what was considerably less than expected from
previous results published in the technical literature. However,
automation tools were the third most common category of
testing-related tools, commonly intended to implement unit and
regression testing. The results indicated that adopting testing
automation in software organization requires high effort. This
work recommends that research and development should focus
on uniform testing process enhancements, such as applying a
new testing approach and creating an organization-wide
strategy for testing automation. Another focus area should be
the development of tools to support testing processes in the low
level testing phases, such as unit or integration testing.
Besides characterizing the scenario of testing automation in
software projects, these works are related to the observations of
testing automation in different environments where the
development and testing teams are formally separated.
However, they do not evaluate scenarios where the
development and testing teams perform strategies to automate
tests together in an agile process. Moreover, they do not
discuss how each team tasks influence on the success of testing
automation strategy. In our projects, different strategies
adopted by the project team are analyzed in the agile
methodology context, which introduced new challenges and
results, as described in the next section.
III. SOFTWARE TESTING AUTOMATION STRATEGIES IN
AGILE PROJECTS
A. Process and Tools
Scrum agile methodology [7] was implemented in our
company. In this methodology, there is no division between
testing and development teams. Thus, testers participate in
Scrum ceremonies (sprint review, daily, retrospective and
planning meetings). Their responsibilities basically are
planning and estimate tasks through stories described in the
product backlog, specifying acceptance criteria for these stories
and using testing automation tools to speed up the execution
activities of each sprint (iteration).
The basic tasks of testing process during a sprint were
incremental and iterative. Testers should plan and update test
cases for sprint stories, automate test scripts when it is possible,
execute all tests, report defects found and run regression tests
(run automated tests and manual exploratory tests). Test Leader
also was responsible for monitoring all tasks in the process.
For confidentiality reasons, the software projects described
in this paper will be called Project 1, Project 2 and Project 3.
Thus, the unit testing frameworks used in the software projects
were: GoogleTest1 for Project 1 and JUnit2 for Projects 2 and 3.
The functional and GUI tests automation tools were: Sikuli3 for
Project 1, Selenium RC and IDE4 for Project 2, and Fitnesse
test tool5 for Project 3.
B. Strategy 1: Testers Automating Unit and System Tests
Project 1 was a small software project that consisted in the
development of a location API driver to be embedded in a
modem. The software is composed of a desktop installer and a
screen to configure the installation. The software platforms
used were C++ programming language and Visual Studio IDE.
The business rules were implemented in the embedded API.
The project team was composed of two full time developers,
one Scrum master, two half time and geographically separated
testers, and one half time Test Leader located in the same site
of developers.
In this project, the teams accorded that unit tests for API
rules would be essential to guarantee the accordance between
code and business rules, because the modem could not simulate
1
GoogleTest URL: http://code.google.com/p/googletest/wiki/
2
JUnit URL: www.junit.org
3
Sikuli URL: sikuli.org/
4
Selenium RC and IDE URL: seleniumhq.org/
5 6
Fitnesse URL: fitnesse.org/
442
all possibilities of inputs combination for test. However, the
project schedule was delayed and the developers wanted to
focus on developing features instead unit testing.
So, testers decided to perform the unit testing tasks using
GoogleTest tool according to Model V approach [12], in which
development and testing processes are conducted in parallel.
Thus, developers just coded new features and then testers
executed unit tests. Functional test cases were also created
according to the product backlog stories and they were
executed at the end of each sprint.
During this testing strategy, we could observe code bugs
were found in the unit tests, but developers, focused on new
code, they postponed the corrections for the next sprint,
generating technical debt. Moreover, the communication
between tester and developers, besides the geographical
distance, was very efficient. It happened because testers had to
understand the developers’ code, transferring knowledge from
developers to testers. However, the opposite (communication
between developers and testers) did not occurred.
Functional tests for the desktop installer and configuration
screen were easily automated and executed by testers for MS
Windows 7-x64 and MS Windows 7-x86 using Sikuli tool
enabling automatic regression testing. These functional
automated tests could be executed in the end of last sprints for
all project as acceptance tests, speeding up the tests execution.
However, due to the distance created between developers and
testers, all testing tasks should be done by testers.
So, this testing strategy did not allow the prevention of
failures, but just the identification of failures in the code,
similar to the traditional development process.
C. Strategy 2: Developers Automating Unit Tests and
Testers Automating System Tests
Project 2 consisted in the development of a web system to
automate factory tasks on counting material for production.
Project team was located in the same site and it was composed
of four Developers, one Scrum Master, two full time Testers
(one Test Analyst and one Tester dedicated to test automation),
and one half-time Test Leader responsible for the test strategy.
The project complexity was high and its size was medium.
A Continuous Integration (CI) environment was configured
by developers in the first sprints (1st and 2nd) using the open
source tool Hudson6. The CI environment could automatically
compile code, execute unit and acceptance GUI tests
(Selenium), and deploy system versions for: development,
testing, and production servers.
The project team decided that developers were responsible
for new features development, unit testing, and CI
environment, while testers should be responsible just for
functional and non-functional tests.
Developers performed unit tests using TDD (Test Driven
Development) approach. In parallel, testers adopted an
automated testing strategy for functional (using Selenium
IDE/RC tools) and non-functional testing (security and load
tests – using Jmeter tool) integrated in the CI environment.
Hudson URL: http://hudson-ci.org/
Thereby, it ensured that each release was covered by unit tests,
acceptance GUI tests, and basic load tests according to
customer requirements.
Internally, the project team was divided in developers, test
analyst (responsible for analyze requirements, test plan and
process) and tester (who were responsible for test automation).
Developers did not perform effective unit testing because they
were focused on new features development and they believed
that testers would found the defects. So, many defects, such as
null pointer exception, screen crashes and illegal exception,
were found during functional testing. In consequence, most
part of code should be modified for next sprint causing
technical debt.
Figure 2 represents the automation structure using CI.
Tester and developers committed to send daily code, unit tests
and automated acceptance tests through Subversion tool.
Hudson monitored changes in Subversion and started the CI
process. Developers could easily run unit tests and testers could
execute Selenium functional automated scripts, manual
exploratory tests, security tests and usability tests.
Figure 2. Automation Structure with CI
Quick feedback in this project was observed because
Hudson displays reports on real-time and e-mails are sent to
project team to fix identified problems in the project.
Test problems were found during sprints. The customer
realized that interface, performance and usability were working
well, but the data accuracy wasn’t correct. It happened because
tests were performed for code and GUI layers, but functional
tests for complex business rules would be executed for the
layer behind GUI. The base of automation was the acceptance
tests, but to ensure data accuracy for complex business rules,
integration tests must be done to validate it.
Moreover, the automation of functional GUI tests in the
current stories presented problems. Testers were using a
record-and-play approach to implement test scripts, but the
system interface changed during the sprint. Thus, automated
test scripts were redesigned and recorded again. The lack of
time in the sprint to conclude the automated tests resulted in
back to manual execution of regression tests. Testers had
technical debt for next sprint too.
443
Analyzing the project team, the division between
Developers, Test Analyst and Tester influenced the testing
automation because it provided lack of synchronization tasks of
the teams and the knowledge transfer do not happened. Just
Testers executed and developed automated test scripts. Test
Analyst created test plan and executed manual tests. Developers
stayed focused on new features even using CI environment.
D. Strategy 3: Developers and Testers Collaborating to
Automate Unit Testing and System Tests
Project 3 (it is in progress) consists in the development of a
server-side system to receive requests from a mobile device,
process information and send services and applications back.
The project team is located in the same site, and it is composed
of five Developers, one Scrum master, and two Testers. All
features and business rules are complex and the application
does not have graphical interface. The end-user interaction
occurs from the mobile device.
Initially, developers were responsible for code new
features, unit tests and Continuous Integration (CI), and Testers
should take care of functional and non-functional tests. This
separation hindered the communication and the same problems
of Project 2 starting to appear. In the first retrospective meeting,
all members decided to change this strategy for the next sprints.
Testers suggested that developers continued responsible by
new features and unit tests, but testers started to collaborate
reviewing test scenarios and code by pair programming to
improve the quality of these artifacts. The access to CI
environment was shared among all project members. Thus,
anyone could get and conclude opened tasks, contributing to
the evolution of the software project.
An automation testing environment for integration and
system tests using Fitnesse tool was configured. Thus, all
members were responsible by the success of the stories. Thus,
the definition of done for a backlog story was formalized as:
feature implemented, integrated and tested. Developers and
testers configured tools and programmed in pair test scripts
according to planned test cases, and all tests were automatically
executed. Developers were enabled to execute automated tests
anytime ensuring the new feature wasn’t affecting the system.
Initially, the test automation environment configuration
demanded high effort, but with collaboration, in two sprints its
configuration was done and all test tools (Testlink, Fitnesse,
mantis, Jmeter and JUnit) were integrated providing quick
feedback through Hudson. During 5 sprints, Testers planned
test cases, reviewed unit and integration tests, wrote automated
test scripts, and executed exploratory tests. On the other hand,
Developers coded and wrote unit, integrations, and system tests,
reviewed test cases and executed all automated tests. Figure 3
represents the tasks distribution according to role in the project.
This suggests that automation tasks are in the intersection and
they can be executed by everyone in the project team.
This strategy promotes harmony in the project team, test
automation environment completed, team collaboration,
knowledge transfer, and quick feedback of sprint results. The
Developers and Testers felt motivated with the tasks to
collaborate and learn about tests, development and automation.

Figure 3. Teams’ Responsibilities
IV. RESULTS AND LESSONS LEARNED
In this experience on conducting different strategies for
testing automation, we could observe that the strategy 3 was
considered by the project team in the retrospective meeting the
best approach to work with agile development, considering the
results and the agile values applied. Reaching this strategy
required patience to find the right way to work. Table 1 shows
the projects results for each strategy, where TC represents the
number of generated test cases, Failures represents the number
of registered failures during the tests execution. The positive
and negative points from scrum retrospective meetings are also
reported. We can observe the size of the test suites for each
project (Project 2 contains the bigger test suite with 234 test
cases), confirming its higher complexity when compared to
Project 1, and how effective were these test suites by analyzing
the number of detected failures.
However, some issues need to be managed to avoid the
risks introduced by the difficulty of agile methodology
combination. Thus, was identified some lessons that we learned
for minimizing risks:
A. Collaboration Supports the Success of Agile Testing
Automation
All main research works in the agile methods field indicate
team collaboration as an important aspect to carry the success
of an agile project. In this experience we could confirm these
TABLE I. RESULTS TABLE
Projects TC Failures Positive
• Testers improved programming skills
1 58 11 • Knowledge sharing on System Architecture
• Unit testing found important defects
• CI strategy improved testing automation
2 234 176 • High coverage of GUI testing automation
• Security and load tests executed early
• CI strategy improved testing automation
• Security and load tests executed early
• Intense team collaboration
3 197 52
• Knowledge transferring regarding the tools • Rework when customer requires changes in the
• Testing responsibilities distribute for all team
• Team motivation to learn new solutions
444
claims for Agile Testing Automation. Some practices are
observed to reach this success:
• Involving testers in configuration environment and
Continuous Integration tasks was important to avoid
separation of the teams, encouraging cooperation.
• Asking for help when a task required knowledge of
another area. For example, in Project 3, developers
helped testers to improve the Fitnesse automation tests
and testers supported developers configuring Hudson
and reviewing unit testing to achieve test coverage.
• Testing automation is not just responsibility of testers.
Pair programming can be a good exercise to integrate
team. Knowledge transfer motivates the project team.
B. Automate Tests for Each Layer of Software Just for
Stories in the Sprint
As was observed in Project 2, when a team occupied a lot
of time in one sprint trying to automate all functional GUI
possible tests, they wasted time that should be spent on testing
tasks that could add value to the project.
In Project 2, it was observed that, even doing unit testing,
GUI test automation and implementing an efficient CI
environment, the business rules behind the interface cannot be
ignored because they detect problems of data accuracy
important to assure the system.
C. Test Automation Should Be Simple, Reusable and
Maintainable for the Project Team
All team members should be able to use the testing
automation tools, running, writing tests and analyzing test
results. Concentrate the knowledge regarding the tools in just
one or two people and the adoption of complex tools represent
risks for agile projects. The combination of skills to finish a
task achieves good and quick results.
In Project 3, the use of friendly test tools promotes the test
tasks collaboration in the project team. Everyone could write,
execute tests, and analyzing test reports.
Negative
• Poor knowledge transferring from testers to developers
• System testing performed after development
• Separation between testing and development teams
• Code improvements postponed
• Poor knowledge transferring
• High effort to automate tests with record-and-play
approach
• Absence of integration testing
• Separation between testing and development teams
• High initial effort to set the automation environment
system
D. Automate Stress and Security Tests Early Reduces Risks
and Rework
Security and stress tests executed at the end of the project
cause overload of tasks in the project. The defects could be
detected before when intermediate versions were released
avoiding risks and rework in the project.
Spending time to do this kind of tests at the beginning of
the project aggregates value, knowledge and confidence for the
agile team.
E. Use Test Automation for Documentation and Information
Feedback
Test tools are also highlighted in the technical literature as
an important element for documenting software artifacts. In the
context of these agile projects, some tools should be cited:
• Testlink (testing management tool) controlled all
information regarding testing activities, test cases
specification and validation, creation of new releases,
test running, number of detected failures, and bug
tracking (creation, correction, and validation).
• Hudson to support CI (used in Project 2 and 3). It
provides interface to display tests releases status,
report test log and all integrated tests. It also provides
the overview and metrics regarding the status of
testing, and reports quickly problems occurring during
this execution.
V. CONCLUSION
This paper presented the experience in performing
strategies for test automation in agile software development
Scrum. We could observe the influence of agile values on team
organization and test automation. These experiences showed
that it is feasible to project team work in collaboration to solve
problems, search for integrated testing tools and keeping the
management and organization of a software testing and scrum
process. The automation testing was a resource to automate
repetitive tasks, document software, reduces cost of project
using open source test tools, and task allocation in small parts.
On the other hand, the programmers’ attitude in these
scenarios was very important to experiment new approaches
and tools. Some lessons learned could be extracted and they
could support other software engineers when performing agile
test automation strategy in a similar environment.
As next steps, we are evaluating the effectiveness of the
Project 3 agile test automation practices in new software
projects when compared to the scenario where testing are
445
performed in other development platforms to extract the testing
effort, number of detected failures pre and post-delivery, and
schedule adherence.
ACKNOWLEDGMENT
The authors would like to thank INdT, FAPEAM, CNPq
(process 575696/2008-7) and INCT-SEC (processes
573963/2008-8 and 08/57870-9) for the financing this research.
REFERENCES
[1] E. Dustin, J. Rashka, J. Paul (1999), Automated software testing:
introduction, management, and performance. Boston: Addison-Wesley,
vol. 1,pp- 4.
[2] K. Karhu, T. Repo, O. Taipale, K. Smolander (2009), Empirical
Observations on Software Testing Automation. In Proceedings of the
2009 International Conference on Software Testing Verification and
Validation (ICST '09). IEEE Computer Society, Washington, DC, USA,
pp. 201-209. DOI=10.1109/ICST.2009.16.
[3] E. Kit (1995), Software Testing in the Real World: Improving the
Process. Reading, MA: Addison-Wesley.
[4] H. Zhi-gen, Y. Quan; Z. Xi (2009), Research on Agile Project
Management with Scrum Method, IITA International Conference on
Services Science, Management and Engineering (SSME '09). IEEE
Computer Society, Washington, DC, USA, pp. 26-29.
DOI=10.1109/SSME.2009.136.
[5] E. Collins, V. Lucena (2010). Iterative Software Testing Process for
Scrum and Waterfall Projects with Open Source Testing Tools
Experience. In Proceedings of the 22nd IFIP International Conference
on Testing Software and Systems2010( ICTSS’10), pp. 115-120, [ISBN
: 978-2-89522-136-4] .
[6] L. Crispin.; J. Gregory (2009); Agile Testing: A Practical Guide for
Testers and Agile Teams, Addison-Wesley, ISBN 0-321-53446-8.
[7] K. Beck, et al. (2001). "Manifesto for Agile Software Development".
Agile Alliance. Retrieved February 2012.
[8] B. Marick (2007); Everyday Scripting with Ruby: For Teams, Testers,
and You; Pragmatic Bookshelf, ISBN 0-9776166-1-4.
[9] T. Wissink, C. Amaro (2006). Successful Test Automation for Software
Maintenance. 22nd IEEE International Conference on Software
Maintenance, pp. 265-266. DOI: 10.1109/ICSM.2006.63 .
[10] L. Damm, L. Lundberg, D. Olsson, D. (2005). Introducing Test
Automation and Test-Driven Development: An Experience
Report. Electronic Notes in Theoretical Computer Science, 116
(SPEC.ISS.), pp. 3-15. Elsevier. DOI= 10.1016/j.entcs.2004.02.090.
[11] J. Kasurinen, O. Taipale, K. Smolander, “Software Test Automation in
Practice: Empirical Observations,” Advances in Software Engineering,
vol. 2010, Article 4 (January 2010), 13 pages.
DOI=10.1155/2010/620836.
[12] G. Myers (2004): The Art of Software Testing. Ed. John Wiley & Sons,
Inc., Hoboken, New Jersey.