TEST AUTOMATION EFFORT ESTIMATION

- Lesson Learnt & Recommendations

Babu Narayanan
1. Candidates for test automation.
One of te c!assica! mista"es of te test automation team is# $NOT coosin% ri%t test cases for automation&'
For any smart customer( te test automation scri)ts are on!y a su))ort de*ice to manua! testin%( NOT to bum) off te !ater' In tat
case( te customer +i!! be more focused on te return on in*estment ,ROI- for eac of test automation scri)t bui!t ,as te initia!
in*estment is i%.-' So coose te tan%ib!e test cases to automate a%ainst eac )ases of de*e!o)ment ,and demonstrate te
same to te customer-'
/o+ to find %ood test case candidates0
Sl. No Test Case Complexity Number of actions Number of verifications Good candidates (~ No of executions)
1 Sim)!e 2 3 2 3 4 13 e5ecutions
6 Medium 4 3 - 2 13 4 3 - 2 17 4 8 - 17 e5ecutions
9 :om)!e5 4 13 - 2 63 4 17 - 2 13 4 3 - 8 e5ecutions
;!ease be a+are tat ste) com)rises of actions and *erification )oints ,or e5)ected resu!ts as some s)e!!-' Most!y actions are direct
metod or function ca!!s to test too! scri)tin% !an%ua%e but te *erification )oints are not of tat "ind'
<y do you need )ase-+ise test automation0
Most of te test automation )ro=ects fai! +ic is most!y due to a))!ication ra)id can%e( unsuitab!e test cases( sa"y frame+or"s
and>or scri)tin% issues' A!so it summari?es tat test automation )ro=ects catc fe+er defects tan it is su))osed to do'
@@ Most!y bud%et o*errun tan estimated'
Te root casua! ana!ysis so+ed us te necessity of )ase-+ise test automation tan one-%o test automation' So ad*ice tat test
automation needs to "ic" off +it critica! test cases tat are of %ood candidate ty)e and ten s!o+!y brancin% out to oter mediums
as reAuired' Tis so!ution e!)s te customer by !esser maintenance costs and more business for you'
A!so remember tat frame+or" needs constant u)dates a!on% +it scri)t de*e!o)ment )rocess and tereby it becomes arder to
maintain te frame+or" incase if you a*e many scri)ts to de*e!o) in )ara!!e!'
<at test ty)e to be automated0
It is a!+ays %ood to scri)t $inte%ration and>or system& functiona! test cases as tey most!y bund!e com)onent !e*e! test cases +itin
tem' Tis +ou!d reduce your effort furter and find %ood defects ,Be). Of course( tis is )rimary ob=ecti*e of any test automation
)ro=ect- too' ;!ease note tat if you a*e %ood frame+or"( ten you can can%e te test sco)e and>or test ty)e usin% confi%uration
fi!es'
2. Factors that affects test automation estimation.
Te fo!!o+in% factors may a*e *aryin% im)act on te test automation effort ca!cu!ation e5ercise'
Sl. No Type Factor Impact emar!s
1 Frame+or" A*ai!abi!ity /i% Cood frame+or" ma"es your scri)tin%( debu%%in% and maintenance
easier' Do understand tat frame+or" needs continuous u)datin% across
te scri)t de*e!o)ment'
6 A))!ication Re)eat functiona!ity /i% It is Auite easier to automate incase te functiona!ity re)eats across te
a))!ication ,Recommend to use "ey+ord dri*en in suc cases( as you do
not end u) in +ritin% more action>*erification based metods-' If NOT(
ten te effort of bui!din% !ibraries and>or scri)ts is more !inear in
nature'
9 ;ro=ect Test Sco)e /i% If te com)!e5ity of a))!ication as +e!! as te test sco)e is com)!e5 in
nature( ten it +ou!d consume u%e efforts to automate eac test case'
E Test too! Su))ort to AUT Medium Te test too! to be used may not su))ort some a))!ication functiona!ity
and may cause o*eread' Bou may find it more difficu!t to %et started
+it o)en source scri)tin% and>or too!s'
3 Scri)ter S"i!! Medium Tis costs )ro=ect' Te ri%t s"i!! )ac"a%es of te scri)ter are *ery
essentia! for any %ood test automation' If te customer NOT +i!!in% to
)ro*ide te !e*era%e on te estimate for tis factor( do NOT for%et to
add te !earnin% cur*e cost to te o*era!! time'
F A))!ication :ustom Ob=ects Medium Te number of custom ob=ects in te automation sco)e matters as it
becomes o*eread for te test automation team to bui!t and maintain
te !ibraries for tem'
G A))!ication Ty)e ,<eb> :!ient-
Ser*er > Mainframe-
Medium For +eb a))!ication( any commercia! test too! as ama?in% uti!ities and
su))ort' Oter+ise( tere are %ood )ossibi!ities tat you need to s)end
u%e effort in bui!din% !ibraries'
8 A))!ication De*e!o)ed Lan%ua%e
& Medium
Lo+ It matters as se!ected test too! does not su))ort s)ecific *erification
cec")oints'
3. Grouping steps to determine complexity.
Tis is an im)ortant e5ercise as it may dra+ +ron% o*era!! effort in s)ite of de)t a))!ication ana!ysis'
STE; 1#
Su%%est findin% number of actions and *erifications )oints for eac test case ,tat are in automation sco)e- and ten dra+ a cart
to find te a*era%e actions( *erification )oints and ten te contro! !imits for tem' So tat te com)!e5ity deri*ation +i!! be based
on te AUT not based on te %eneric industry standards'
E5am)!e(
T: 71 76 79 7E 73 7F 7G 78 7H 17 11 16 19 1E 13 1F 1G 18 1H 67 61 66 69 6E 63
STE;S 8 16 13 16 1F 97 E 8 66 61 16 6G 66 13 11 13 16 H 8 1H 61 E6 F 66 66
STE; 6#
Based on te data cart(
A*era%e ste) count I 1F
Lo+er contro! !imit I 78
U))er contro! !imit I 63
So te com)!e5ity can be %rou)ed as#-
Sim)!e J G ste)s
Medium K 8 ste)s -- J 1F ste)s
:om)!e5 K 1G ste)s -- J 63 ste)s
Recommendations#
1' Neiter %rou) test case ste)s too c!ose( nor +ide for !abe!in% te com)!e5ity' Be a+are tat te )re-scri)t de*e!o)ment
effort for eac test scri)t is considerab!e as te fo!!o+in% acti*ities are time-consumin% o)erations#-
1- E5ecutin% test case manua!!y before scri)tin% for confirmin% te successfu! o)eration'
6- Test data se!ection and>or %eneration for te scri)t'
9- Scri)t tem)!ate creation ,!i"e eader information( comments for ste)s( identifyin% te ri%t reusab!e to be used from te
re)ository and so on'-
Tese efforts are i%!y based on te number of ste)s in te test case' Note tat if test case *aries by fe+er ste)s( ten tis
effort does not de*iate muc but incase it *aries by many ste)s e*en tis effort +ide!y differs'
6' A!so anoter factor in determinin% te com)!e5ity is te functiona!ity re)etition' If te test case is :om)!e5 by ste)s but te
functiona!ity is same as te oter test case ten tis can be !abe!ed as $Medium or Sim)!e& ,based on te =ud%ment-'
9' If te test case ste)s count are more tan u))er contro! !imit ,L 63 in tis case- *a!ue ten tose additiona! ste)s need to
be considered as anoter test case' For e5am)!e( te T: - 7F containin% 97 ste)s sa!! be !abe!ed as
$1 com)!e5 M 1 sim)!e ,97-63-& test cases'
If te test case is mar"ed as $:om)!e5& instead of $Medium&( understand tat your efforts soot u) and urts your customer' On
oter +ay of misca!cu!ation( it urts us' Tere by( tis $com)!e5ity %rou)in%& is more of !o%ica! +or"out +it data as in)ut'
4. Framework design & estimation.
N<e a*e e5)erienced a si%nificant increase in soft+are reusabi!ity and an o*era!! im)ro*ement in soft+are Aua!ity due to te
a))!ication )ro%rammin% conce)ts in te de*e!o)ment and ,re-use of semi finised soft+are arcitectures rater tan =ust sin%!e
com)onents and te bond bet+een tem( tat is( teir interaction' ! "olfgang #ree $#ree%4&
Tere are many frame+or"s tat are a*ai!ab!e commercia!!y & as o)en-source +ic are s)ecific to a test too! or +ide-o)en' A!so
you find omebre+ test automation frame+or"s too s)ecific to test too!s' Tese frame+or"s sa*es a !ot of scri)tin%( debu%%in% and
maintenance efforts but a+are tat te customi?ation of frame+or" ,based on te a))!ication- are *ery essentia!'
:aracteristics of any frame+or"#
;ortab!e( e5tendab!e and reusab!e across and +itin )ro=ects'
Ease of functiona!ity ;!u%-ins>outs based on a))!ication *ersion can%es'
Loose!y cou)!ed +it te test too! +ere*er )ossib!e'
E5tended reco*ery system and e5ce)tion and!in% to ca)ture te unand!ed errors and to run smoot!y'
Ste)( Lo% and Error Information )ro*ide easier debu%%in% and customi?ed re)ortin% faci!ities of scri)ts'
Ease of test data dri*en to te scri)ts and tey need to be !oose!y cou)!ed'
Easi!y contro!!ab!e and confi%urab!e test sco)e for e*ery test run'
Sim)!e and easy inte%ration of test automation com)onents +it test mana%ement( defect trac"in% and confi%uration
mana%ement too!s'
;!ease note tat tese efforts a*e +ide ran%e as te
frame+or" si?e and sco)e )ure!y de)ends on a))!ication
nature( si?e and com)!e5ity'
It is a!+ays a %ood )ractice to create and>or customi?e te
frame+or" for initia! needs and ten add> u)date com)onents>
features and fine
tune tem as )ro=ect %oes'
Be a+are tat +ron% frame+or" coice may cost your )ro=ect'
'. (cripting )ffort )stimation
SL'NO SUB :OM;ONENT
ESTIMATED EFFORT
REMAROS Sim)!e
,28 ste)s-
Medium
,8-1F ste)s-
:om)!e5
,1G-63 ste)s-
1 #re*(cript +e,elopment
a Test :ase e5ecution ,Manua!- For 1 iteration ,assumin% scri)ter "no+s na*i%ation-
b Test data se!ection For one data set ,*a!id>in*a!id>erratic "ind-
c Scri)t Tem)!ate creation :an use scri)t tem)!ate %eneration uti!ity to a*oid tis'
d Identify te reAuired reusab!e Assumin% )ro)er reusab!e traceabi!ity matri5 )resence'
2 (cript +e,elopment
a A))!ication ma) creation Assumin% te no of ob=ects I number of actions
b Base scri)tin% Norma!!y a!! tese %o and-in-and' Se)arated for
ana!ysis & reasonin%' c Add error>e5ce)tion and!in%
d Im)!ement frame+or" e!ements
3 (cript )xecution
a Scri)t e5ecution For n iterations ,L a*era%e iteration count-
b Perification & Re)ortin% Assumin% tere +i!! minima! defect re)ortin%'
-otal )ffort per script
Oey+ord dri*en
Tis tota! effort +ou!d *ary if you coose "ey-+ord dri*en metodo!o%y but at te same time( te effort of bui!din% frame+or" +i!!
be i% ,for initia! desi%n and scri)tin%-'
Do not use "ey+ord dri*en a))roac for sma!! )ro=ects'
Tese efforts may differ based on te abo*e discussed ,section 6- factors' Su%%est you to )erform ;o: for 6 scri)ts from eac c!ass to confirm'
Te ne%ati*e test cases norma!!y consume additiona! scri)t efforts as te )attern can%es'
O*era!! effort ca!cu!ation may a*e te fo!!o+in% com)onents#-
1' Test ReAuirement %aterin% & Ana!ysis
6' Frame+or" desi%n and de*e!o)ment
9' Test :ase de*e!o)ment ,incase te a*ai!ab!e manua! test cases not com)atib!e-
E' Scri)t De*e!o)ment
3' Inte%ration Testin% and Base!ine'
F' Test Mana%ement'
A!! tese com)onents sa!! inc!ude re*ie+ ,1>6 cyc!es-'