http://www.huibschoots.nl/wordpress/?

p=1118
Magnifiant: exploring software testing
Huib's blog on Software Testing
Misconceptions about testing
###Note : Gerold : This article is an reaction on article Agile Methodology Is Not All About Exploratory Testing (See
http://www.scrumalliance.org/articles/511-agile-methodology-is-not-all-about-eploratory-testing! ###
Posted on March 31, 2013 by Huib Schoots
Shmuel Gershons tweet pointed me to an article on the scrum alliance website Agile Methodolog !s "ot All About
#$plorator %esting b &ele 'luwole. ! share Sigge (irgissons concerns: )the post clearl shows what ! mean when
ha*ing deep concerns about the +nowledge o, testing in agile communit-.
! thin+ &ele doesnt ,ull understand what testing is or at least he uses a di,,erent de,inition than ! do. And
certainl he doesnt understand e$plorator testing. .i+ard #dgren wrote an open letter to e$plain what testing is.
/lease read his high le*el summar o, so,tware testing to understand what testing means to me.
0!t is imperati*e to state in clear terms wh Agile testing cannot be all about e$plorator testing0
(The text in these gray frames are quotes form the article by Dele)
! wrote a post some wee+s ago about agile testing titled what ma+es agile testing di,,erent: agile testing isnt that
much di,,erent ,rom )other- testing. 1h do some people thin+ agile is so di,,erent? %o me there is no such thing
as Agile testing. %here is testing in an agile conte$t. And e*er agile conte$t is probabl di,,erent. So saing that
agile testing cannot be about e$plorator testing ma+es no sense to me.
!t is une2ui*ocall the case that: ou cannot estimate our time ,or e$plorator testing3 i.e.3 assign points
realisticall
1
#stimating testing is an interesting topic. %his blogpost b Morgan Ahlstr4m nicel emphasi5es that estimates are
guesses. Martin 6ansson o, the %est #e writes about )utopic estimations- here. Michael (olton wrote a lot about
estimation here3 here and here. 7e e$plains that testing is an open8ended tas+ which depends on the 2ualit o, the
products under test. %he decision to ship the product 9which includes a decision to stop testing: is a business
decision3 not a technical one.
#speciall the ,i,th part o, the blac+ swan series is interesting in this discussion because Michael writes about the
,allacies surrounding )de*elopment- and )testing- phases 9b 6ames (ach:. 7e also e$plains wh estimating the
duration o, a )testing pro;ect- b counting )test cases- or )test steps- is not a smart thing to do.
0<ou cannot plan ,or e$plorator testing3 as ou do not ha*e de,ined e$pected results.0
1h are some people so obsessed b e$pected results? And wh is there a need to ha*e e$pected results to be
able to plan testing? #$pected results can be *er help,ul3 but there is much more to 2ualit then doing some tests
with an e$pected result. A de,inition that ! li+e is b 6err 1einberg: )=ualit is *alue to some person-. %o
understand this3 ou might want to read this blogpost to understand that there is more to 2ualit than the absence
o, bugs. Also ha*e a loo+ at the e$cellent ,ree eboo+ )%he >ittle (lac+ (oo+ on %est &esign- b .i+ard #dgren. 'n
page ?8@ he uses the )testing potato- to e$plain that there are more important aspects to the sstem than the
re2uirements onl.
0%here is no de,ined scope ,or e$plorator testing.0
!n the e$plorator testing ! do in m pro;ects there is a )scope-. ! do targeted and ,ocused testing using charters3
sessions and planning m testing using a dashboard that resembles a scrum board. 7a*e a loo+ at the slides o, m
presentation )(oosting the testing power with e$ploration).
2

Asing a co*erage outline in a mind map or a simple spreadsheet3 ! +eep trac+ o, what ! ha*e tested. M charters 9a
one to three sentence mission ,or a testing session: help me ,ocus3 m wrap8up and/or debrie,ings help me
determine how good m testing was. M notes and the sessions sheets +eep trac+ o, what ! ha*e done in m
testing. >i+e in scrum3 ! use )standup- meetings to plan m testing. !n these meetings we discuss progress3 ris+s3
priorities and charters to be e$ecuted. %his helps us to ma+e sure we do the best testing we can do continuousl.
0%he tester3 product owner3 and Scrum team are not in control.0
! am not sure i, ! understand what ou are tring to point out here. Are ou saing that the team is not in control
when doing e$plorator testing? M model abo*e shows that ou can be in control when doing e$plorator testing
when done right. #$plorator testing is "'% ad hoc or unstructured when done right. !, ou do it right ou will ha*e
controlB
0%here is no measure o, progress3 as testers cannot determine when testing is enough.0
7ow do we determine how much testing is enough? Stopping heuristics might help here. "o matter how simple the
sstem is we are building3 there are simpl too man *ariables to test e*erthing. So testing is about ma+ing
choices what to test and what not to test. #*en with a huge amount o, automated chec+s3 we can not chec+/test
e*erthing 9to understand the di,,erence between testing and chec+ing read this:. %esting is not about doing C test
cases and when the all pass3 ou are done. %esting is pro*iding in,ormation ,or managers to ma+e good decisions.
And when do managers ha*e enough in,ormation to in,orm their decisions?
Still3 not man Agile pro;ects will re2uire ;ust two phases3 li+e integration and regression. (ut itDs de,initel not onl
e$plorator testing thatDs needed3 as is erroneousl 9=ten onrechte: belie*ed in some 2uarters.
3
! am not sure what ou mean b two phases. 1hat do ou mean b testing in phases? ! li+e to use the agile
testing 2uadrants when ! tr to e$plain how ! thin+ o, testing in an agile conte$t.
A team is de*eloping so,tware and the programmers do testing be,ore chec+ing the so,tware in and ma+ing it
a*ailable to the team. 7ow do we call that sort o, testing? Anit testing? (ut is that /onl/ testing done b the
programmer? ! argue that the programmer might do all +inds o, testing be,ore chec+ing his code in3 e*en
,unctional and acceptance tests. %he probabl will create a lot o, automated chec+s and mabe e*en do some
e$plorator testing to see i, the so,tware meets their e$pectations: 2uic+l testing some usabilit and per,ormance
aspects. So be,ore the so,tware is chec+ed in3 the programmer has co*ered testing in all E 2uadrants. %his does
not mean testing is done. More testing can be done3 it depends on the conte$t. 1hat does the product owner
wants to +now? 1hat are the ris+s in*ol*ed? 7ow much time do we ha*e le,t? 1hen discussing a test strateg ! tr
not to spea+ about phases3 ! li+e to discuss what gets co*ered and wh. 1hat in,ormation is needed b the team
and its sta+eholders? 1hen tal+ing about co*erage ! do not mean code co*erage but test co*erage: the e$tent to
which we ha*e tra*eled o*er some map 9model: o, the sstem.
So ! dont thin+ ou can sa )onl e$plorator testing is needed or not-.
&ele concludes his article with this statement:
!t is the responsibilit o, the tester 9and the Agile/Scrum team: to ensure that acceptance testing is in line with the
e$pectation o, the product owner. !, we agree that there is an e$pectation3 we there,ore ha*e to design test cases
9e*en i, minimal: that will *eri, the speci,ied acceptance criteria.0
&ear &ele please read about testing and e$plorator testing. Some good starting points are these lists o, resources
on this blog or the one made b Michael (olton: resources on #$plorator %esting3 Metrics3 and 'ther Stu,,. ! am
happ to point ou to more good sources o, in,ormation i, ou are interested. 6ust let me +now.
This entry was posted in Agile, Context-Driven, Exploratory Testing by Huib Schoots. Bookmark the permalink.
4
14 THOUGHTS ON MISCONCEPTIONS ABOUT TESTING
1. Pingback: Perspectives on Testing The Seapine View
2. Brian Osman on April 8, 2013 at 1:19 am said:
Good article Huib interestingly enough i tweeted last year that i dont believe in the label Agile testing (or Agile tester note the
caps). Rather i believe that testing is shaped by the context. Hence Agile testing to me is testing in an Agile context and of course
that may vary project to project. The *testing* hasnt/doesnt/shouldnt change but the bubble around the testing (i.e. the project/org
dynamics) would.
Unfortunately, too many people make ill conceived comments about good exploratory testing.
Thanks for sharing.
Reply
3. Pingback: Neotys Testing Roundup, April 2013 Issue 2
4. Johan Jonasson on April 3, 2013 at 9:10 am said:
Good work Huib. Looking forward to your reply to Deles second post as well.
Reply
Huib Schoots on April 7, 2013 at 9:54 pm said:
I am not sure I will write a comprehensive reply. Not sure it is worth the effort. I have a feeling Dele doesnt want to understand. If I
am wrong about this, I am happy to continue the discussion.
James replied with this tweet: https://twitter.com/jamesmarcusbach/status/319763237732249600. I join James in offering my help.
Dele recommends me to read books I already own and have read. This weekend I re-read pages 308-320. I hope he reads the stuff
I recommended him.
I think my post wasnt clear enough about the things we agree upon. I added a comment on his rejoinder blog to make clear that I do
not think exploratory testing is the /only/ way people should test in agile contexts. I do value test automation. The claims Dele makes
on exploratory testing are wrong and Dele will benefit from learning more about exploratory testing.
Reply
5
Johan Jonasson on April 8, 2013 at 9:24 am said:
Yeah, I agree. I felt I wanted to reply to his second post and point out to him that nobodys saying agile testing is all about ET, but
that if you are going to talk about why it isnt, then at least one should make an effort to understand ET first (which I thought your
post tried to help him with very generously).
But, I found I didnt have the energy to get into that sort of discussion. So I tried to trick you into doing it instead.
Reply
5. XU Yi on April 3, 2013 at 7:24 am said:
Well, I started to define Agile Testing as Doing testing in Agile way within an Agile environment under the guidance of Agile
thinking. I separated two things, 1st is how we test (verb) the object under test; 2nd is how we organize the testing (noun) activities
as a whole, both in terms of many testers and many testing activities (e.g. design, execution).
And, if we agree that ET is more an approach to do test, does it make sense to say Its All About doing Things in One Approach!?
Reply
6. Huib Schoots on April 2, 2013 at 2:52 pm said:
Dele wrote a new blog: Response to Magnifiant: exploring software testing Huib Schoots.
Reply
7. Pingback: Quite possibly the best article on testing Ive read | waynp
8. Pingback: Five Blogs 2 April 2013 | 5blogs
9. Peter Varhol on April 1, 2013 at 12:21 pm said:
Best article Ive read so far this year. It makes me realize how little I really know about testing.
Reply
10. Geir Gulbrandsen on April 1, 2013 at 2:30 am said:
6
That is one awesome collection of personal and referenced wisdom Huib! I will need to read through this a few more times just to
digest what /you/ are saying, before digging further into the links you provide (although some are familiar) and then connect it all
together.
Thanks for yet another great post.
Geir
Reply
11. Wayne Peacock on March 31, 2013 at 8:09 pm said:
Great article Huib! Youve brought together so many useful references here. Youve also made some very good points yourself. I
have read this on my mobile but have forwarded it to my work email address to read it again. I shall also share it with the testers
where I work and beyond. Thank you.
Reply
12. Tony Bruce on March 31, 2013 at 7:49 pm said:
7