2006 JavaOne

SM
Conference | Session TS-3097 |
Beyond JUnit:
Introducing TestNG
The Next Generation
in Testing
Hani Suleiman
CTO
Formicary
http!!"""#formicary#net
hani$formicary#net
TS 3097
2006 JavaOne
SM
Conference | Session TS-3097 | 2
Testing
%
&ene"e' enth(siasm for testin)
%
*o more +rea, 'eve,opers- vs# +./ 'eve,opers-
%
0artia,,y '(e to
+12treme pro)rammin)3 4506
%
7an' Test 8riven
8eve,opment 4T886
%
Testin) is coo, a)ain9
2006 JavaOne
SM
Conference | Session TS-3097 | 3
JUnit
%
*o rea, intro'(ction nee'e'9
%
Simp,e JavaTM techno,o)y
testin) frame"or:; <ase'
on introspection;
test metho's; c,asses;
an' s(ites#
%
First mainstream testin)
frame"or: create' an'
no" a 'e facto stan'ar'#
2006 JavaOne
SM
Conference | Session TS-3097 | =
JUnit Strengths
%
Simp,e to (n'erstan' test metho's;
c,asses; s(ites
%
1asy to imp,ement e2ten' TestCase; prefi2
metho' "ith +test-
%
>se set>p an' tear8o"n for initia,i?ation
an' c,ean(p
%
>se Test&(nner for te2t or )raphica, res(,ts
%
Tests can <e com<ine' into s(ites
%
@ots of a''-ons '< testin); )(i testin); reportin)
2006 JavaOne
SM
Conference | Session TS-3097 | A
JUnit Problems
8oes this test pass or fai,B
public class Test1 extends TestCase {
private int count = 0;
private void test1() {
count++;
assertEquals(count, 1);
}
public void test() {
count++;
assertEquals(count, 1);
}
2006 JavaOne
SM
Conference | Session TS-3097 | 6
It Passes!
%
J>nit instantiates yo(r c,ass <efore
invo:in) each test metho'
% Cy 'esi)nB
%
Do" 'o yo( :eep state across
invocationsB
% >se statics9
%
Many 'o"nsi'es
% *ot +same-Java EM- frien',y
% &e'(n'ant "ith set>p46
% Foes a)ainst int(itive c,ass!constr(ctor
<ehavio(r
%
&ep,acin) one f,a" 4reinstantiation6
"ith another 4statics6
2006 JavaOne
SM
Conference | Session TS-3097 | 7
JUnit Problems
%
Do" 'o yo( r(n an in'ivi'(a, metho'B
%
Comment o(t a,, the other ones9
%
Do" 'o yo( :eep ena<,e!'isa<,e certain
metho'sB
%
Mo'ify yo(r s(ite; recompi,e; an' rer(n
%
*ote 12istin) G81-s an' J>nit a''-ons he,p
a''ress these iss(es
2006 JavaOne
SM
Conference | Session TS-3097 | H
JUnit Problems
%
Eictim of its o"n s(ccess# Gnitia,,y 'esi)ne' for
(nit testin) on,y; <(t no" (se' for a,, sorts of
testin)
%
Eery ,imite' for anythin) <(t (nit-testin)
%
>p'ates are very fe" an' far <et"een
%
Gntr(sive 4forces s(per,cass an' +ma)ic- metho'
namin)6
%
Static pro)rammin) mo'e,
4recompi,e (nnecessari,y6
%
8oesn-t (se ,atest Java techno,o)y feat(res
4annotations; asserts6
2006 JavaOne
SM
Conference | Session TS-3097 | 9
Introducing TestNG
%
/nnotations 4J8ITM A soft"are
or Java'ocTM too,6
%
F,e2i<,e r(ntime
confi)(ration 42m,6
%
Gntro'(ces +test )ro(ps- ; to
separate statics 4test contents6
from 'ynamics 4"hich tests to r(n6
%
8epen'ent test metho's;
para,,e, testin); ,oa' testin);
partia, fai,(re#
%
F,e2i<,e p,()-in /0G
2006 JavaOne
SM
Conference | Session TS-3097 | J0
TestNG xam!le "J#$ %&' so(t)are*
public class !i"pleTest {
#$$
$ %testn&'con(i&uration be(oreTestClass = true
$#
public void init() {
## code t)at *ill be invo+ed *)en test is instantiated
}
#$$
$ %testn&'test &roups = ,(unctest-
$#
public void server.s/unnin&() {
## 0our test code
}
}
2006 JavaOne
SM
Conference | Session TS-3097 | JJ
TestNG xam!le "J#$ + so(t)are*
i"port or&'testn&'annotations'$;
public class !i"pleTest {
%Con(i&uration(be(oreTestClass = true)
public void init() {
## invo+ed *)en test is instantiated
}
%Test(&roups = { 1(unctest1 })
public void server.s/unnin&() {
## 0our test code
}
}
2006 JavaOne
SM
Conference | Session TS-3097 | J2
TestNG xam!le
%
*o nee' to e2ten' a <ase c,ass
%
*o ma)ic +test- prefi2 for metho' names
%
Confi)(ration metho's can <e )iven meanin)f(,
names 4not K(st set>p an' tear8o"n6; yo( can
have as many as yo( "ant; an' they can <e
aro(n' metho's; c,asses; or s(ites
%
Test metho's can <e parametri?e' 4not sho"n;
<(t 'isc(sse' ,ater6
2006 JavaOne
SM
Conference | Session TS-3097 | J3
,unning the Test
,untime s!eci(ied in testng&xml& -ostly: list o( classes and list o( grou!s to
include.exclude:
2test na"e=,!i"ple13
2&roups3
2run3
2include na"e=,(unctest1#3
2#run3
2#&roups3
2classes3
2class na"e=,!i"pleTest1 #3
2#classes3
2#test3
Note: testng&xml is o!tional/ can use ant.command line
2006 JavaOne
SM
Conference | Session TS-3097 | J=
0nnotations
%
$Test
% G'entify a test metho'
%
$Confi)(ration
% G'entify a metho' (se' to confi)(re tests
%
$12pecte'12ceptions
% Gn'icate that a metho' is e2pecte' to thro" one or more
e2ceptions
%
$8ata0rovi'er
% 0rovi'e parameters to pass to test metho's
%
$Factory
% Create yo(r o"n test o<Kects at r(ntime
2006 JavaOne
SM
Conference | Session TS-3097 | JA
TestNG Terminology
%
S(ites7each s(ite contains
%
Tests7each test contains
%
C,asses7each c,ass contains
%
Metho's
%
$Confi)(ration can "rap each of the
scopes a<ove
2006 JavaOne
SM
Conference | Session TS-3097 | J6
testng&xml
2suite na"e=1Test45 678 1'9, parallel=1true1 t)read:count=,913
2test na"e=14opac+a&e13
2classes3
2class na"e=14o;ac+a&eTest11 #3
2class na"e=14o;ac+a&eTest1 #3
2#classes3
2#test3
2test na"e=1/e&ression11 3
2pac+a&es3
2pac+a&e na"e=1test're&ression'$1 #3
2#pac+a&e3
2#test3
2#suite3
2006 JavaOne
SM
Conference | Session TS-3097 | J7
TestNG 0nnotations
%
$Confi)(ration
% <eforeTestMetho'!afterTestMetho'

J>nit set(p!tear8o"n
% <eforeTestC,ass!afterTestC,ass

J>nit *o eL(iva,ent
% <eforeS(ite!afterS(ite

J>nit *o eL(iva,ent
% <eforeTest!afterTest

J(nit *o eL(iva,ent
%
Mo( can have m(,tip,e $Confi)(ration metho's
%
$Confi)(ration metho's can <e )ro(pe'!have
'epen'encies
2006 JavaOne
SM
Conference | Session TS-3097 | JH
TestNG 0nnotations
%
$Test
%
)ro(ps The )ro(ps this metho' <e,on)s to
%
parameters The parameters that "i,, <e passe' to
yo(r test metho's; as 'ec,are' in testn)#2m,
%
'epen'sOnFro(ps The ,ist of )ro(ps this metho'
'epen's on
%
timeo(t The ma2im(m '(ration of this test <efore it is
consi'ere' a fai,(re
%Test(&roups = { ,(unctional- }, ti"e<ut = 10000,
depends<n5roups = ,server.s=p-)
public void send>ttp/equest() {
## ?
}
2006 JavaOne
SM
Conference | Session TS-3097 | J9
testng&xml
%
Nhere a,, r(ntime
information )oes
% Test metho's; c,asses;
an' pac:a)es
% Nhich )ro(ps sho(,' <e
r(n 4inc,('e-)ro(ps6
% Nhich )ro(ps sho(,' <e
e2c,('e' 4e2c,('e-)ro(ps6
% 8efine a''itiona, )ro(ps
4)ro(ps of )ro(ps6
% Nhether tests sho(,' <e
r(n in para,,e,
% 0arameters
% J>nit mo'e
2006 JavaOne
SM
Conference | Session TS-3097 | 20
cli!se and I#0
%
Test*F 0,()-in e2ists
for 1c,ipse an' G81/
%
&(n a test c,ass;
metho'; or )ro(p
%
1asy se,ection of
)ro(ps an' testn)#2m, fi,es
%
Eis(a, fee'<ac:
4re'!)reen <ar; K(st ,i:e J>nit96
%
8irect,y K(mp to fai,(res
%
+L(ic:fi2!intentions- to one-c,ic:
mi)rate J>nit tests
2006 JavaOne
SM
Conference | Session TS-3097 | 2J
2006 JavaOne
SM
Conference | Session TS-3097 | 22
2006 JavaOne
SM
Conference | Session TS-3097 | 23
2006 JavaOne
SM
Conference | Session TS-3097 | 2=
2006 JavaOne
SM
Conference | Session TS-3097 | 2A
Integration )ith 1ther 2rame)or3s
%
Maven p,()-in 4vJ an' v26
%
Sprin) frame"or:
%
8C>nit
%
Gnte)ration is strai)htfor"ar' in most cases
4set>p!tear8o"n6
2006 JavaOne
SM
Conference | Session TS-3097 | 26
4on5erting (rom JUnit
%
J>nitConverter can convert entire co'e<ase to
Test*F in a fe" secon's
%
0,()-in in 1c,ipse or G81/ can a,so 'o so
2006 JavaOne
SM
Conference | Session TS-3097 | 27
x!ected xce!tions
%
Thro"in) e2ceptions is common from Java co'e
%
1asy to test "ith Test*F
%ExpectedExceptions({
@ava'lan&'4u"berAor"atException'class })
%Test
public void validate4u"ber() {
?
}
2006 JavaOne
SM
Conference | Session TS-3097 | 2H
,erunning 2ailed Tests
%
Most of o(r "or: is fi2in) tests that fai,
%
Test*F :no"s "hich tests fai,e' in a r(n an'
ma:es it easy to rer(n K(st these tests#
%
testn)-fai,e'#2m,
%
Typica, session
B @ava or&'testn&'Test45 testn&'x"l
B @ava or&'testn&'Test45 testn&:(ailed'x"l
2006 JavaOne
SM
Conference | Session TS-3097 | 29
#ataPro5iders
%
Test*F s(pports 8ata-8riven testin)
%
12amp,e Testin) a parser
%Test
public void parse5ood!trin&(!trin& s) {
ne* ;arser()'parse(s);
}
%ExpectedExceptions({ ;arserException'class })
%Test
public void parseCad!trin&(!trin& s) {
ne* ;arser()'parse(s);
}
2006 JavaOne
SM
Conference | Session TS-3097 | 30
#ataPro5iders
%Test(data;rovider = ,&ood:strin&s-)
public void parse5ood!trin&(!trin& s) {
ne* ;arser()'parse(s);
}
%7ata;rovider(na"e = ,&ood:strin&s1)
public <b@ectDEDE create5ood!trin&s() {
return ne* <b@ectDEDE {
ne* <b@ectDE { , $ - },
ne* <b@ectDE { ,F + - };
}
}
2006 JavaOne
SM
Conference | Session TS-3097 | 3J
#ataPro5iders
%
8ata0rovi'ers a,,o" yo( to separate 'ata
from ,o)ic
%
8ata can come from Java techno,o)y; f,at fi,e;
'ata<ase; net"or:; etc#
%
Mo( can have as many 8ata0rovi'ers as
yo( ,i:e# For e2amp,e; O)oo'-strin)s3;
O<a'-strin)s3; etc#
2006 JavaOne
SM
Conference | Session TS-3097 | 32
Testing Thread Sa(ety
%
*ot even s(re ho" to 'o this in J>nit
public void testCac)e;ut!)ouldCeT)read!a(e() {
## Create pool o( t)reads (9H 10H 90H)
## Create *or+ers invo+in& cac)e'put() (100H 00H)
## Iaunc) t)e pool
## Jait (or ter"ination
## Kbort la&&ards (exception i( an0)
## .( no exception veri(0 t)at no data )as been t)ras)ed
}
0he"9
2006 JavaOne
SM
Conference | Session TS-3097 | 33
Testing Thread Sa(ety
%
Test*F
%Test(invocationCount=1000, t)read;ool!iLe=10)
public void cac)e;ut() {
"Mcac)e'put(,(oo-, ,bar-);
}
*ee' a timeo(tB
%Test(invocationCount=1000, t)read;ool!iLe=10,
ti"e<ut=10000 #$ 10 seconds $#)
2006 JavaOne
SM
Conference | Session TS-3097 | 3=
xcluding Grou!s
% Sometimes; tests <rea: an' yo( cannot fi2 them imme'iate,y
% J>nit comment them o(t or rename metho'; an' "ith
,(c:; t(rn them <ac: on <efore shippin)
% Test*F 'efine O<ro:en3 )ro(p an' have it e2c,('e'
in a,, test r(ns# Move any fai,s to this )ro(p
% @ater @oo: for tests in O<ro:en3 )ro(p an' fi2#
2test na"e=,7CTest13
2&roups3
2run3
2exclude na"e=,bro+en'$1 #3
2include na"e=,(unctional'$- #3
2#run3
2#&roups3
2006 JavaOne
SM
Conference | Session TS-3097 | 3A
Programmatic In5ocation
%
Convenient for in-container testin)
Test45testn&=ne*Test45();
testn&'setTestClasses(ne*ClassDE{
/un1'class, /un'class
});
testn&'set5roups(ne* !trin&DE { ,(unctional-, ,db-});
TestIistenerKdaptertla=ne*TestIistenerKdapter();
testn&'addIistener(tla);
testn&'run();
## inspect results in tla
2006 JavaOne
SM
Conference | Session TS-3097 | 36
Beanshell
%
Sometimes; inc,('in) an' e2c,('in) )ro(ps is
not eno()h
%
Nhat if "e nee' to r(n certain tests '(rin) the "ee:;
an' a 'ifferent set '(rin) the "ee:en'B
% >se <eanshe,,9
2test na"e=1Casic13
2script lan&ua&e=1beans)ell132NDC7KTKD
int toda0 = Calendar'&et.nstance()'
&et(Calendar'7KOM<AMJEE8);
return toda0 == Calendar'!KT=/7KO PP
toda0 == Calendar'!=47KO;
EE32#script3
2006 JavaOne
SM
Conference | Session TS-3097 | 37
Partial 2ailure
%
OinvocationCo(nt3 a,,o"s (s to specify the n(m<er of
times to e2ec(te a test#
%
>se' "ith Os(ccess0ercenta)e3 a,,o"s (s to 'efine
partia, fai,(re tests
%Test(invocationCount = 1000,
success;ercenta&e = QR)
public void send!"sSessa&e(!trin& "s&)
{ ? }
2006 JavaOne
SM
Conference | Session TS-3097 | 3H
Plug6in 0PI
%
Test*F e2poses a p,()-in /0G that ma:es it easy to
monitor test pro)ress an' invocation
% Test starte'
% Test en'e'
% Test res(,t7etc
%
0ossi<,e to a,so mo'ify Test*F core
%
Fo(r proofs of concept
% J>nit mo'e
% 8efa(,t DTM@ reports
% J>nit&eport DTM@ p,()-in
% Test*F-s o"n testin)
2006 JavaOne
SM
Conference | Session TS-3097 | 39
#e!endent -ethods
%
0ro<,em certain metho's 'epen' on the
s(ccess of previo(s metho's
%
12amp,e testin) a server
% One test metho' to ,a(nch the server ,a(nch46
% One test metho' to pin) the server pin)46
% T"enty metho's to verify server f(nctiona,ity testJ467test2046
%
0ro<,em Server is ,a(nche' <(t pin)46 fai,s
%
Scenario 'iffic(,t to achieve "ith J(nit
%
&es(,t J 0/SS18 an' 2J F/G@>&1S
%
&es(,t ./ frea:s o(t an' ca,,s yo( on S(n'ay
'(rin) yo(r )o,f )ame
2006 JavaOne
SM
Conference | Session TS-3097 | =0
#e!endent -ethods
%
*ee' a "ay to or'er metho'sP
*ot K(st in'ivi'(a, metho's; <(t co,,ections of
metho's )ro(pe' ,o)ica,,y
%
*ee' a mechanism to acc(rate,y report fai,(res
'(e to fai,e' 'epen'ency 4avoi' 0&FTS 0ost
&(n Fai,(re Tra(ma Syn'rome6
2006 JavaOne
SM
Conference | Session TS-3097 | =J
#e!endent -ethods
%
Cac: to the server testin) e2amp,e
% 8epen'encies ,a(nch init tests
%Test(&roups = ,launc)-)
public void launc)!erver() {?}
%Test(&roups = ,init-, depends<n5roups = { ,launc)- })
public void pin&() { ?}
%Test(depends<n5roups = { ,init- })
public void test1() { ? }
%
O(tcome J S>CC1SS; J F/G@; 20 SIG0S
2006 JavaOne
SM
Conference | Session TS-3097 | =2
,e!orting
%
Test*F iss(es an DTM@ report <y 'efa(,t
%
0,()-in /0G ma:es it easy to "rite yo(r o"n
report 4for e2amp,e; J>nit&eporter p,()-in6
2006 JavaOne
SM
Conference | Session TS-3097 | =3
2006 JavaOne
SM
Conference | Session TS-3097 | ==
Summary
%
J>nit has the ri)ht i'ea; <(t s(ffers from o,' a)e
an' ,imitations for rea, 4non-(nit6testin)#
%
Test*F offers the fo,,o"in) <enefits
%
*on-intr(sive 4annotations6
%
C,ean separation of pro)rammin) mo'e, from r(ntime
%
Covers more than (nit testin); "ith feat(res ,i:e
'epen'encies; )ro(ps; parameters; an' partia, fai,(re
%
0o"erf(, p,()-in /0G for c(stom reports or
mo'ifications to core f(nctiona,ity
%
Nhether yo( choose Test*F or J>nit; thin:
'ifferent,y a<o(t testin)9
2006 JavaOne
SM
Conference | Session TS-3097 | =A
2or -ore In(ormation
%
Doste' on Kava#net; avai,a<,e via CES
%
8o"n,oa' an' 'oc(mentation avai,a<,e on
http!!testn)#or)
%
/(thors
%
Ce'ric Ce(st 4c<e(st$)oo),e#com6
%
/,e2an'r( 0opesc( 4theQmin'storm$evo,va#ro6
2006 JavaOne
SM
Conference | Session TS-3097 | =6
780
2006 JavaOne
SM
Conference | Session TS-3097 |
Beyond JUnit:
Introducing TestNG
The Next Generation
in Testing
Hani Suleiman
CTO
Formicary
http!!"""#formicary#net
hani$formicary#net
TS 3097