Professional Documents
Culture Documents
Ulf Eriksson Beyond The Hype Software Testing Explained
Ulf Eriksson Beyond The Hype Software Testing Explained
Page 2
Dear Reader,
I am very proud to present to you this ebook that you are
about to start reading.
In it, you will read all about the lessons that have helped
shape our company into the organisation it is today.
Essentially, this ebook is the product of my many years
in the world of software testing, compounded by the
experiences of our wonderful team at ReQtest.
Everything you will find here, every technique, process
or tool mentioned has been used by the members of our
team while we were building ReQtest. We return to them
even now, whenever were developing new versions of
our product.
In short, this ebook distils all the lessons that we have
learnt throughout the years and lays them out across
these pages for you to take and apply straight away in
your own projects.
I hope that what you will find here will help you and your
team work better together, as well as to deliver more
value with the products you create.
Sincerely,
Ulf Eriksson
Page 3
Table of Contents
Letter From Founder
10
13
17
The test leaders guide to troubleshooting distributed test teams
An introduction to exploratory testing
21
24
28
33
36
Page 4
Page 5
a fine balance between writing concise reports and
explaining thoroughly how to reproduce a bug. Failing
this will result in time spent discussing and providing
additional information about the bug. This can easily be
avoided by ensuring no important steps are omitted from
the bug report.
4. Maximise the information delivered. Layout
counts as much as content in making sure that the
information is grasped by the reader. Careless writing
is not an option since figuring out how to pack all the
relevant information in a reasonable amount of time
requires judgement and experience.
5. Ask for feedback. Surround yourself with people
who are not afraid to criticize your ideas. Fellow testers
and other professionals can help you devise better ways
to deliver bug reports; programmers can inform you
about the type of information they mainly look for in a
report and other testers can give you general comments
about the overall quality of your bug reports.
Conclusion
Finding and reporting bugs is not an easy business. The
report writing process involves various challenges that have
to be overcome and requires finding a balance between
the effort to provide all the essential information without
overwhelming yourself or the reader.
Although oracles and tools can help you to improve the quality
of the information you provide, in the last analysis it is you, the
report writer, who will make the final decision regarding which
information makes the cut or not.
Page 6
Page 7
At iteration #10 about 1/10 of the total code is newly
developed with the other 90% being old code that
requires regression testing.
Page 8
2
Page 9
automate every test. On the other hand, it might be possible
to automate the regression tests youve identified at a higher
test level, which you might do with a recording tool.
Recorded regression tests can be combined with the tests
that the development team delivers to the system test team,
which adapts the recorded tests to include the new tests, so
that the entire system is tested again quickly and efficiently.
Page 10
Page 11
Page 12
Fooling around
The ancient Greek philosopher Socrates, known as one of
the wisest persons in history, used to routinely embarrass and
frustrate his contemporaries by asking obvious questions and
generally making a fool out of himself by playing ignorant.
This technique, shocking as it may sound, is an excellent way
to attack any assumptions that cloud our perceptions and
judgement and help us see everything in a clearer light.
So dont be afraid to ask silly questions. Questions like:
Did we spell it right? It might uncover a couple of
terrible typos in the headers;
Is this the right address? Pointing your users to
yourwebstie.com instead of yourwebsite.com isnt a good
idea.
Other ways to reduce the effects of perceptual blindness
Apart from questioning the obvious, there are three other
methods you can use to combat perceptual blindness in
software testing:
Testing in pairs;
Communicate better;
Use cross-functional teams.
Page 13
Page 14
Page 15
Page 16
Page 17
Page 18
#1 - Communication problems
When testers are spread across the four corners of the world,
maintaining open and effective communication channels can
be difficult to say the least.
Dealing with testers in different time zones can be tricky and
requires much patience and juggling of schedules. Having
different tools to cater for your teams communication is
crucial in this regard to ensure that the team functions as a
whole even though the members are separate in time and
space.
Tools for effective communication include popular ones like
Skype of Google Hangouts, as well as specific tools such as
ReQtest, that are designed purposely for testers collaborating
online on requirements management and testing.
#2 - Role ambiguity
The longer the distance a signal has to travel, the more
degraded it will be at its point of arrival.
The same concept roughly applies to decision-making in
teams. In distributed teams, people are more likely to take
the lead and take decisions independently of what the team
leader decrees.
In these cases it is essential to define the particular role of
each team member, as well as to clarify the hierarchy around
which the team is structured.
Page 19
#3 - Duplication of roles
A natural consequence of role ambiguity is role overlap, or
worse, duplication. Not only does this lead to wasted time,
resources and effort, it also breeds resentment in the team.
It is important to assign a person in every location the role
of test manager and delegate to them the responsibility of
coordinating with other managers and overseeing that their
team does not duplicate the work being done by another one.
#5 - Conflicting priorities
When testers are spread over a wide geographic area there
is an increased risk that the individual members may have
Page 20
Summary
Having a geographically distributed test team can be a
challenging situation for any team leader, however any
difficulties that may arise can be easily avoided by proper
planning and troubleshooting any problems the moment they
arise.
Page 21
An introduction to
exploratory testing
Many people think that testing is boring and repetitive. I beg
to differ.
Testing allows me to dig deep into the thicket of code to
uncover bugs that need fixing, allowing me to use a range of
skills from the analytical to the intuitive.
You could compare our work to that of an archeologist. Or
a psychoanalyst for software! We go deep into the layers of
code to resolve the deep-seated problems that lie within our
product, thus helping it attain its full potential.
Thankfully, no cajoling or tears is involved -- apart maybe
from those of an overworked tester grappling with a bug that
refuses to be cracked.
Exploratory testing
Exploratory testing is a technique in the testers toolbox which
allows one to locate critical bugs earlier as well as stimulate
the creative and inquisitive side of the tester.
In a nutshell, exploratory testing involves running tests as
they are being created in order to become more intimately
acquainted with the system being scrutinised.
The concept of exploratory testing originated in the 1990s and
was developed by Cem Kaner and James Bach.
Page 22
Putting it in practice
The first step before testing it to break the system down into
smaller parts in order to get a more comprehensive overview.
A list of all the areas of the system that need to be explore
should follow. Consider this as your map that charts the
territory youre about to explore. Its a good idea to include
an inventory of the testing tools that will be utilised during the
test.
Exploratory testing doesnt have a clearly defined outcome.
The point is the process being used. When using exploratory
testing, a tester will use alternative approaches to probe the
system and gather feedback from a 360 perspective.
Skilled testers arent content with testing a problem one. They
will retest using different methods, even regression testing,
to ensure that no new bugs were introduced in the system
following any changes they themselves affected.
Since exploratory testing uses a variety of testing approaches,
the best practice is to log the types of tests performed to the
system and the results obtained.
Page 23
Conclusion
Exploratory testing is an approach that taps into the abilities
of a testers in a holistic manner, putting to rest the usual
accusation that our profession is a monotonous one.
On the contrary, exploratory testing challenges testers to
expand their abilities and put into play all their skills not only to
solve a problem but to improve the end product.
Page 24
Page 25
Page 26
Page 27
malleable and team members can step in and out of that role
depending on the group dynamics and circumstances.
When someone distributes work between what is internal
team work, reports the results of testing, puts up the strategy
for test work and plans the order in which the work should be
carried out, that someone would rightfully be called the test
leader.
What is different in an agile context is that from one task to
another, a different someone may be leading the team and
that there can be as many leaders as there are decisions to
be taken within the team.
Page 28
14 requirements you
should keep in mind for
any system
#2 - Permissions
Who should be allowed to see or change what data? What
happens when a user tries to use the features that he
does not have access to? Can you see features youre not
authorised to use, or are they hidden? Who should administer
permissions and how will they do it? All of these are questions
you need to ask.
Page 29
#3 - Error messages
Remember to specify where the error messages are to be
displayed. Do they use a message box or do they notify users
in the programs status bar? Error messages are very rarely
specified in the requirements, but without this requirement
in place, theyre often written by developers and that could
cause the system to display messages that are hard to
understand for end-users. Without requirements, messages
will probably not even have been tested, which can result in
bad usability.
#5 - Logging errors
Its much easier to troubleshoot the system if it records the
details of each error to some type of log. This is especially true
in the production environment, where its not possible to use
development tools for troubleshooting. Often its worthwhile to
log more than just the error itself. For example, you might log
all calls to all functions, since the error might have occurred
because the calls were made in the wrong order.
#6 - Handling overloads
Page 30
#8 - System integration/
connections
Develop a traceability matrix that shows which test cases
or test areas are affected. Are there sufficient requirements
specified for these relationships?
Page 31
Page 32
#13 - Reports
Are there any reporting requirements or other ways to extract
data from the system? How often will reporting happen, who
will carry it out and what metrics will they need to extract? Is
there any delay in reporting data being populated?
Page 33
Page 34
#2 - Lack of feedback
Exploratory testing uses documentation just like the traditional
way of testing software, however the information contained
within it is qualitatively different.
The documentation in this case will only report the findings
from the tests carried out by the tester and as I mentioned
earlier, it is at the testers discretion as to which tests to apply.
9
Page 35
In practice, this means that the documentation could fail to
report any bugs whatsoever because the tester failed to use a
test which revealed any.
Of course, a tester may feel hard-pressed to justify the lack of
any bugs! After all, finding them is his or her job.
Summary
Achieving exploratory testing that is both successful and fun
requires a lot of work.
The less interest from the organization in supporting this
riskier approach to testing, the harder the tester must work
to justify using this method.
In this cases, the testers have to mount an argument in favour
of exploratory testing. My previous articles about exploratory
testing on our blog can help you formulate points in support
of exploratory testing, as well as balancing that out with the
pitfalls that are outlines here.
10
Page 36
8 simple ways to
become a faster tester
Here at ReQtest were all about efficiency. We work hard to
trim off the fat and fine-tune our testing process to be as lean
as possible.
But efficiency shouldnt come at the cost of effectiveness.
Our lean testing machine must also be mean enough to tear
through any testing situation.
In my experience of continuously working to improving the
testing process, Ive noticed eight simple changes that boost
tester efficiency without compromising on the quality of their
work.
Check out the eight points below to find out how you too can
shift gears and start blazing through testing.
10
Page 37
10
Page 38
Keep pen and paper nearby to jot down any points, or if you
prefer to go paperless, make use of a program like ReQtest
that lets you create checklists that integrate seamlessly with
your workflow.
#5 - Tell a story
We all like a good story. A typical user opens a program
Maybe the punchline to that opening wont result in a belly
laugh, but writing requirements in the form of user stories is a
very time-efficient alternative which will give you more time to
head to the bar later and swap some real jokes!
If user stories are used for requirements, you can become a
faster tester by writing acceptance test cases sooner, as soon
as the user story is done. This helps you finish the job earlier.
#6 - Exploratory testing
We have spoken about exploratory testing several times
on our blog, so check out our previous articles to learn
more about it and how you can start applying it to your job
today. Checklists and exploratory testing are a very effective
combination to ramp up productivity.
10
Page 39
10
Page 40