THE DELPHI METHOD By H. Murat GNAYDIN, Ph.D.

Definition and Historical Background. The objective of most De !hi a!! icatio"s is the re iab e a"# creative e$! oratio" of i#eas or the !ro#uctio" of suitab e i"formatio" for #ecisio" ma%i"&. The De !hi Metho# is base# o" a structure# !rocess for co ecti"& a"# #isti i"& %"o' e#&e from a &rou! of e$!erts by mea"s of a series of (uestio""aires i"ters!erse# 'ith co"tro e# o!i"io" fee#bac% )A# er a"# *i& io, +,,-.. Accor#i"& to He mer )+,//. De !hi re!rese"ts a usefu commu"icatio" #evice amo"& a &rou! of e$!erts a"# thus faci itates the formatio" of a &rou! ju#&eme"t. 0issema )+,12. u"#er i"es the im!orta"ce of the De !hi Metho# as a mo"ovariab e e$! oratio" tech"i(ue for tech"o o&y forecasti"&. He further states that the De !hi metho# has bee" #eve o!e# i" or#er to ma%e #iscussio" bet'ee" e$!erts !ossib e 'ithout !ermitti"& a certai" socia i"teractive behavior as ha!!e"s #uri"& a "orma &rou! #iscussio" a"# ham!ers o!i"io" formi"&. Ba #'i" )+,/3. asserts that ac%i"& fu scie"tific %"o' e#&e, #ecisio"4 ma%ers have to re y o" their o'" i"tuitio" or o" e$!ert o!i"io". The De !hi metho# has bee" 'i#e y use# to &e"erate forecasts i" tech"o o&y, e#ucatio", a"# other fie #s )5or"ish, +,//.. The tech"o o&y forecasti"& stu#ies 'hich eve"tua y e# to the #eve o!me"t of the De !hi metho# starte# i" +,66. At that time Ge"era Ar"o # as%e# Theo#or vo" 7arma" to !re!are a forecast of future tech"o o&ica ca!abi ities that mi&ht be of i"terest to the mi itary )5or"ish, +,//.. Ar"o # &ot the Dou& as Aircraft com!a"y to estab ish i" +,6- a Project 8AND )a" acro"ym for 8esearch a"# Deve o!me"t. to stu#y the 9broa# subject of i"ter4co"ti"e"ta 'arfare other the" surface.: I" +,3, He mer a"# fe o' 8AND researcher 8escher !ub ishe# a !a!er o" 9The ;!istemo o&y of the I"e$act <cie"ces,: 'hich !rovi#e a !hi oso!hica base for forecasti"& )=o' es, +,/1.. The !a!er ar&ue# that i" fie #s that have "ot yet #eve o!e# to the !oi"t of havi"& scie"tific a's, the testimo"y of e$!erts is !ermissib e. The !rob em is ho' to use this testimo"y a"#, s!ecifica y, ho' to combi"e the testimo"y of a "umber of e$!erts i"to a si"& e usefu stateme"t. The De !hi metho# reco&"i>es huma" ju#&eme"t as e&itimate a"# usefu i"!uts i" &e"erati"& forecasts. <i"& e e$!erts sometimes suffer biases? &rou! meeti"&s suffer from 9fo o' the ea#er: te"#e"cies a"# re ucta"ce to aba"#o" !revious y state# o!i"io"s )Gate'oo# a"# Gate'oo#, +,1@, =o' es, +,/1.. I" or#er to overcome

these shortcomi"&s the basic "otio" of the De !hi metho#, theoretica assum!tio"s a"# metho#o o&ica !roce#ures #eve o!e# i" the +,3As a"# +,-As at the 8AND 5or!oratio". =orecasts about various as!ect of the future are ofte" #erive# throu&h the co atio" of e$!ert ju#&eme"t. Da %ey a"# He mer #eve o!e# the metho# for the co ectio" of ju#&eme"t for such stu#ies )Gor#o" a"# Hay'ar#, +,-1.. =o' es )+,/1. asserts that the 'or# De !hi refers to the ha o'e# site of the most revere# orac e i" a"cie"t Greece. =orecasts a"# a#vices from &o#s 'ere sou&ht throu&h i"terme#iaries at this orac e. Ho'ever Da %ey )+,-1. states that the "ame 9De !hi: 'as "ever a term 'ith 'hich either He mer or Da %ey )the fou"#ers of the metho#. 'ere !articu ar y ha!!y. Da %ey )+,-1. ac%"o' e#&e# that it 'as rather u"fortu"ate that the set of !roce#ures #eve o!e# at the 8AND 5or!oratio", a"# #esi&"e# to im!rove metho#s of forecasti"&, came to be %"o'" as 9De !hi:. He ar&ue# that the term im! ies 9somethi"& oracu ar, somethi"& smac%i"& a itt e of the occu t:, 'hereas, as a matter of fact, !recise y the o!!osite is i"vo ve#? it is !rimari y co"cer"e# 'ith ma%i"& the best you ca" of a ess tha" !erfect %i"# of i"formatio". B"e of the very first a!! icatio"s of the De !hi metho# carrie# out at the 8AND 5or!oratio" is i ustrate# i" the !ub icatio" by Gor#o" a"# He mer )+,-6.. Its aim 'as to assess the #irectio" of o"&4ra"&e tre"#s, 'ith s!ecia em!hasis o" scie"ce a"# tech"o o&y, a"# their !robab e effects o" society. The stu#y covere# si$ to!icsC scie"tific brea%throu&hs? !o!u atio" co"tro ? automatio"? s!ace !ro&ress? 'ar !reve"tio"? 'ea!o" systems )Gor#o" a"# He mer, +,-1.. The first De !hi a!! icatio"s 'ere i" the area of tech"o o&ica forecasti"& a"# aime# to forecast i%e y i"ve"tio"s, "e' tech"o o&ies a"# the socia a"# eco"omic im!act of tech"o o&ica cha"&e )A# er a"# *i& io, +,,-.. I" terms of tech"o o&y forecasti"&, Devary a"# Ha" )+,,3. state the objective of the De !hi metho# as to combi"e e$!ert o!i"io"s co"cer"i"& the i%e ihoo# of rea i>i"& the !ro!ose# tech"o o&y as 'e as e$!ert o!i"io"s co"cer"i"& the e$!ecte# #eve o!me"t time i"to a si"& e !ositio". 0he" the De !hi metho# 'as first a!! ie# to o"&4ra"&e forecasti"&, !ote"tia future eve"ts 'ere co"si#ere# o"e at a time as thou&h they 'ere to ta%e ! ace i" iso atio" from o"e a"other. Dater o", the "otio" of cross im!acts 'as i"tro#uce# to overcome the shortcomi"&s of this sim! istic a!!roach )He mer, +,//.. Accor#i"& to 0issema )+,12., u"fortu"ate y the De !hi metho# is a so sometimes use# for a "orma i"(uiry amo"& a "umber of e$!erts. De !hi has fou"# its 'ay i"to i"#ustry, &over"me"t, a"# fi"a y, aca#eme. It has simu ta"eous y e$!a"#e# beyo"# tech"o o&ica forecasti"& )=o' es, +,/1.. <i"ce the +,3As severa research stu#ies have use# the De !hi metho#, !articu ar y i" !ub ic hea th issues )such as, !o icies for #ru& use re#uctio" a"# !reve"tio" of AID<EHIF. a"# e#ucatio" areas )A# er a"# *i& io, +,,-? 5or"ish, +,//..

The Basics of the Delphi Method. The De !hi metho# is a" e$ercise i" &rou! commu"icatio" amo"& a !a"e of &eo&ra!hica y #is!erse# e$!erts )A# er a"# *i& io, +,,-.. The tech"i(ue a o's e$!erts to #ea systematica y 'ith a com! e$ !rob em or tas%. The esse"ce of the tech"i(ue is fair y strai&htfor'ar#. It com!rises a series of (uestio""aires se"t either by mai or via com!uteri>e# systems, to a !re4 se ecte# &rou! of e$!erts. These (uestio""aires are #esi&"e# to e icit a"# #eve o! i"#ivi#ua res!o"ses to the !rob ems !ose# a"# to e"ab e the e$!erts to refi"e their vie's as the &rou!Gs 'or% !ro&resses i" accor#a"ce 'ith the assi&"e# tas%. The mai" !oi"t behi"# the De !hi metho# is to overcome the #isa#va"ta&es of co"ve"tio"a committee actio". Accor#i"& to =o' es )+,/1. a"o"ymity, co"tro e# fee#bac%, a"# statistica res!o"se characteri>e De !hi. The &rou! i"teractio" i" De !hi is a"o"ymous, i" the se"se that comme"ts, forecasts, a"# the i%e are "ot i#e"tifie# as to their ori&i"ator but are !rese"te# to the &rou! i" such a 'ay as to su!!ress a"y i#e"tificatio". I" the ori&i"a De !hi !rocess, the %ey e eme"ts 'ere )+. structuri"& of i"formatio" f o', )2. fee#bac% to the !artici!a"ts, a"# )@. a"o"ymity for the !artici!a"ts. 5 ear y, these characteristics may offer #isti"ct a#va"ta&es over the co"ve"tio"a face4to4face co"fere"ce as a commu"icatio" too . The i"teractio"s amo"& !a"e members are co"tro e# by a !a"e #irector or mo"itor 'ho fi ters out materia "ot re ate# to the !ur!ose of the &rou! )Marti"o, +,/1.. The usua !rob ems of &rou! #y"amics are thus com! ete y by!asse#. =o' es )+,/1. #escribes the fo o'i"& te" ste!s for the De !hi metho#C +. =ormatio" of a team to u"#erta%e a"# mo"itor a De !hi o" a &ive" subject. 2. <e ectio" of o"e or more !a"e s to !artici!ate i" the e$ercise. 5ustomari y, the !a"e ists are e$!erts i" the area to be i"vesti&ate#. @. Deve o!me"t of the first rou"# De !hi (uestio""aire 6. Testi"& the (uestio""aire for !ro!er 'or#i"& )e.&., ambi&uities, va&ue"ess. 3. Tra"smissio" of the first (uestio""aires to the !a"e ists -. A"a ysis of the first rou"# res!o"ses /. Pre!aratio" of the seco"# rou"# (uestio""aires )a"# !ossib e testi"&. 1. Tra"smissio" of the seco"# rou"# (uestio""aires to the !a"e ists ,. A"a ysis of the seco"# rou"# res!o"ses )<te!s / to , are reiterate# as o"& as #esire# or "ecessary to achieve stabi ity i" the resu ts.. +A. Pre!aratio" of a re!ort by the a"a ysis team to !rese"t the co"c usio"s of the e$ercise De bec( et a ., )+,/3. ar&ue that the most im!orta"t issue i" this !rocess is the u"#ersta"#i"& of the aim of the De !hi e$ercise by a !artici!a"ts. Bther'ise the

!a"e ists may a"s'er i"a!!ro!riate y or become frustrate# a"# ose i"terest. The res!o"#e"ts to the (uestio""aire shou # be 'e i"forme# i" the a!!ro!riate area )Ha"so" a"# 8ama"i, +,11. but the iterature )Armstro"&, +,/1? 0e ty, +,/2. su&&est that a hi&h #e&ree of e$!ertise is "ot "ecessary. The mi"imum "umber of !artici!a"ts to e"sure a &oo# &rou! !erforma"ce is some'hat #e!e"#e"t o" the stu#y #esi&". ;$!erime"ts by Broc%hoff )+,/3. su&&est that u"#er i#ea circumsta"ces, &rou!s as sma as four ca" !erform 'e . Before #eci#i"& 'hether or "ot the De !hi metho# shou # be use#, it is very im!orta"t to co"si#er thorou&h y the co"te$t 'ithi" 'hich the metho# is to be a!! ie# )De bec( et a . +,/3.. A "umber of (uestio"s "ee# to be as%e# before ma%i"& the #ecisio" of se ecti"& or ru i"& out the De !hi tech"i(ue )A# er a"# *i& io, +,,-.C 0hat %i"# of &rou! commu"icatio" !rocess is #esirab e i" or#er to e$! ore the !rob em at ha"#H 0ho are the !eo! e 'ith e$!ertise o" the !rob em a"# 'here are they ocate#H 0hat are the a ter"ative tech"i(ues avai ab e a"# 'hat resu ts ca" reaso"ab y be e$!ecte# from their a!! icatio"H B" y 'he" the above (uestio"s are a"s'ere# ca" o"e #eci#e 'hether the De !hi metho# is a!!ro!riate to the co"te$t i" 'hich it 'i be a!! ie#. A# er a"# *i& io )+,,-. further c aim that fai ure to a##ress the above (uestio"s may ea# to i"a!!ro!riate a!! icatio"s of De !hi a"# #iscre#it the 'ho e creative effort. The outcome of a De !hi se(ue"ce is "othi"& but o!i"io". The resu ts of the se(ue"ce are o" y as va i# as the o!i"io"s of the e$!erts 'ho ma#e u! the !a"e )Marti"o, +,/1.. The !a"e vie'!oi"t is summari>e# statistica y rather tha" i" terms of a majority vote. The De !hi metho# has &ot criticism as 'e as su!!ort. The most e$te"sive criti(ue of the De !hi metho# 'as ma#e by <ac%ma" )+,/6. 'ho critici>es the metho# as bei"& u"scie"tific a"# Armstro"& )+,/1. 'ho has 'ritte" critica y of its accuracy. Marti"o )+,/1. u"#er i"es the fact that De !hi is a metho# of ast resort i" #ea i"& 'ith e$treme y com! e$ !rob ems for 'hich there are "o a#e(uate mo#e s. He mer )+,//. states that sometimes re ia"ce o" i"tuitive ju#&eme"t is "ot just a tem!orary e$!e#ie"t but i" fact a ma"#atory re(uireme"t. Ma%ri#a%is a"# 0hee ri&ht )+,/1. summari>e the &e"era com! ai"ts a&ai"st the De !hi metho# i" terms of )a. a o' eve re iabi ity of ju#&eme"ts amo"& e$!erts a"# therefore #e!e"#e"cy of forecasts o" the !articu ar ju#&es se ecte#? )b. the se"sitivity of resu ts to ambi&uity i" the (uestio""aire that is use# for #ata co ectio" i" each rou"#? a"# )c. the #ifficu ty i"

assessi"& the #e&ree of e$!ertise i"cor!orate# i"to the forecast. Marti"o )+,/1. ists major co"cer"s about the De !hi metho#C Discou"ti"& the futureC =uture )a"# !ast. ha!!e"i"&s are "ot as im!orta"t as the curre"t o"es, therefore o"e may have a te"#e"cy to #iscou"t the future eve"ts. The sim! ificatio" ur&eC ;$!erts te"# to ju#&e the future of eve"ts i" iso atio" from other #eve o!me"ts. A ho istic vie' of future eve"ts 'here cha"&e has ha# a !ervasive i"f ue"ce ca""ot be visua i>e# easi y. At this !oi"t cross4im!act a"a ysis is of some he !. I usory e$!ertiseC some of the e$!erts may be !oor forecasters. The e$!ert te"#s to be a s!ecia ist a"# thus vie's the forecast i" a setti"& 'hich is "ot the most a!!ro!riate o"e. < o!!y e$ecutio"C there are ma"y 'ays to #o a !oor job. ;$ecutio" of the De !hi !rocess may oose the re(uire# atte"tio" easi y. =ormat biasC it shou # be reco&"i>e# that the format of the (uestio""aire may be u"suitab e to some !ote"tia societa !artici!a"ts. Ma"i!u atio" of De !hiC The res!o"ses ca" be a tere# by the mo"itors i" the ho!e of movi"& the "e$t rou"# res!o"ses i" a #esire# #irectio".

Go #schmi#t )+,/3. a&rees that there have bee" ma"y !oor y co"#ucte# De !hi !rojects. Ho'ever, he 'ar"s that it is a fu"#ame"ta mista%e to e(uate the a!! icatio"s of the De !hi metho# 'ith the De !hi metho# itse f, as too ma"y critics #o. There is, i" fact, a" im!orta"t co"ce!tua #isti"ctio" bet'ee" eva uati"& a tech"i(ue a"# eva uati"& a" a!! icatio" of a tech"i(ue. B" the other ha"# there have bee" severa stu#ies )Ame"t, +,/A? 0issema, +,12? He mer, +,1@. su!!orti"& the De !hi metho#. A stu#y co"#ucte# by Mi %ovich et a . )+,/2. re!orts the use of the De !hi metho# i" ma"!o'er forecasti"&. The resu ts of the com!ariso" i"#icate# hi&h a&reeme"t bet'ee" the De !hi estimate a"# the actua "umber hire# a"# ess a&reeme"t bet'ee" (ua"titative forecasts a"# the "umber hire#. A"other stu#y by Basu a"# <chroe#er )+,//. re!orts simi ar resu ts i" a &e"era forecasti"& !rob em. They com!are# De !hi forecasts of five4year sa es 'ith both u"structure#, subjective forecasts a"# (ua"titative forecasts that use# re&ressio" a"a yses a"# e$!o"e"tia smoothi"&. The De !hi forecasti"& co"siste# of three rou"#s usi"& 2@ %ey or&a"i>atio" members. 0he" com!are# a&ai"st actua sa es for the first t'o years, errors of @46I 'ere re!orte# for De !hi, +A4+3I for the (ua"titative metho#s, a"# of a!!ro$imate y 2AI for the !revious y use# u"structure#, subjective forecasts.

I" &e"era , the De !hi metho# is usefu i" a"s'eri"& o"e, s!ecific, si"& e4 #ime"sio" (uestio". There is ess su!!ort for its use to #etermi"e com! e$ forecasts co"cer"i"& mu ti! e factors. <uch com! e$ mo#e bui #i"& is more a!!ro!riate for (ua"titative mo#e s 'ith De !hi resu ts servi"& as i"!uts )Gate'oo# a"# Gate'oo#, +,1@.. This !oi"t is su!!orte# by Gor#o" a"# Hay'ar# )+,-1. 'ho c aim that the De !hi metho#, base# o" the co atio" of e$!ert ju#&eme"t, suffers from the !ossibi ity that reactio"s bet'ee" forecaste# items may "ot be fu y co"si#ere#. The "ee# for the cross im!act matri$ metho# of forecasti"& i"te&rate# 'ith the De !hi metho# is !oi"te# out by ma"y researchers )Gor#o" a"# Hay'ar#, +,-1? Gate'oo# a"# Gate'oo#, +,1@? A# er a"# *i& io, +,,-.. A" im!roveme"t i" forecasti"& re iabi ity over the De !hi metho# 'as thou&ht to be attai"ab e by ta%i"& i"to co"si#eratio" the !ossibi ity that the occurre"ce of o"e eve"t may cause a" i"crease or #ecrease i" the !robabi ity of occurre"ce of other eve"ts i"c u#e# i" the survey )He mer, +,/1.. Therefore cross im!act a"a ysis has #eve o!e# as a" e$te"sio" of De !hi tech"i(ues.
BIBDIBG8APHY A# er, M., J *i& io, ;. )+,,-.. Ga>i"& i"to the orac e. Kessica 7i"&s ey Pub ishersC Bristo , PA. A arco", D.=. )+,,2.. Project !erforma"ce mo#e i"&C a metho#o o&y for eva uati"& !roject e$ecutio" strate&ies. Ph.D. thesis, L"iversity of 5a ifor"ia, Ber%e ey. Broch"er, K. )+,,A.. Im!acts of i"formatio" tech"o o&y o" the structure of co"structio". 5o"structio" Ma"a&eme"t a"# ;co"omics, 1, !!. 2A342+1. 5or"ish, ;., )+,//.. The stu#y of the future. 0or # =uture <ocietyC 0ashi"&to", D.5. Duva , A., =o"te a, ;., Gabus, A., )+,/3.. Portraits of 5om! e$ity. );#. Ba #'i", M. M.. Batte e Memoria I"stituteC 5o umbusC Bhio. ;">er, <., J A ter, <. )+,/1.. 5ross4im!act a"a ysis a"# c assica !robabi ityC The (uestio" of co"siste"cy. =utures, Ku"e +,/1, +A)@., !!. 22/42@,. ;'i"&, 8.P. )+,/,.. The uses of futurist tech"i(ues i" issues ma"a&eme"t. Pub ic 8e atio"s Muarter y, 6, !!. +34+1. =or&io""e, G.A. )+,,/.. HADT<C a #ecisio" tech"o o&y system to su!!ort army housi"& ma"a&eme"t. ;uro!ea" Kour"a of B!eratio"a 8esearch, !!. @-@4@/,. =o' es, K., )+,/1.. Ha"#boo% of futures research. Gree"'oo# PressC 5o""ecticut.

Gate'oo#, 8.D. a"# Gate'oo#, ;.K., )+,1@.. The use of e$!ert #ata i" huma" resource ! a""i"&C &ui#e i"es from strate&ic forecasti"&. Huma" 8esource P a""i"&, 3)+.. P!.1@4,6. Gor#o", T.K. a"# Hay'ar#, H. )+,-1.. I"itia e$!erime"ts 'ith the cross4im!act matri$ metho# of forecasti"&. =utures, December, +,-1, +)2., !!. +AA4++-. Ha"so", 0.H., a"# 8ama"i, N. )+,11.. Tech"o o&y forecasti"&C a hy#roe ectric com!a"y e$!erie"ce. Tech"o o&y Ma"a&eme"t Pub icatio", +)@.., !!. 2--42/A. Harvar# Busi"ess <choo Press. He mer, B., )+,//.. Prob ems i" futures researchC De !hi a"# causa cross4im!act a"a ysis. =utures, =ebruary +,//, !!. +/4@+. He mer, B., )+,1+.. 8eassessme"t of cross4im!act a"a ysis. =utures, Bctober +,1+, !!. @1,4 6AA. Huss, 0.8. )+,11.. A move to'ar# sce"ario a"a ysis. I"ter"atio"a Kour"a of =orecasti"&, 6, !!. @//4@11. Ke"se", 8.;. )+,1+.. <ce"ario !robabi ity sca i"&C a" ei&e"vector a"a ysis of e icite# sce"ario o##s ratios. =utures, December, +,1+. !!. 61,46,1. Ki", Y., Devitt, 8.;., 7u">, K.5., a"# 5hristia"se", T.8. )+,,3.. The virtua #esi&" teamC a com!uter simu atio" frame'or% for stu#yi"& or&a"i>atio"a as!ects of co"curre"t #esi&". <imu atio", March, +,,3, !!. +-A4+/6. Koh"so", K.D. )+,,-.. I"formatio" see%i"&C a" or&a"i>atio"a #i emma. 5o""ecticut, Do"#o"C Muorum Boo%s. 7ir%'oo#, 5.0., a"# Po oc%, <.M. )+,12.. Mu ti! e attribute sce"arios, bou"#e# !robabi ities, a"# threats of "uc ear theft. =utures, December +,12, !!. 363433@. 7'a%, Y.H., )+,,/.. A systematic a!!roach to eva uate (ua"titative im!acts of !roject ma"a&eme"t. Ph.D. #issertatio", L"iversity of 5a ifor"ia, Ber%e ey. Devary, 8.8. a"# Ha", D. )+,,3.. 5hoosi"& a tech"o o&ica forecasti"& metho#. IM, !!. +64 +1. Nova%, ;., a"# Dora"t, 7., )+,/1.. A metho# for the a"a ysis of i"terre atio"shi!s bet'ee" mutua y co""ecte# eve"tsC a cross4im!act metho#. Tech"o o&ica =orecasti"& a"# <ocia 5ha"&e )+2., !!.2A+42+2. 8euve", D., a"# Do"&chui, H., )+,,3.. 5hoosi"& a tech"o o&ica forecasti"& metho#. I"#ustria Ma"a&eme"t, @/)+.. !!. +6422.

<ari", 8.7., )+,/1.. A se(ue"tia a!!roach to cross4im!act a"a ysis. =utures, =ebruary +,/1, !!. 3@4-2. <chu er, A., Thom!so", 0.A., Ferti"s%y, I., a"# *iv, Y. )+,,+.. 5ross im!act a"a ysis of tech"o o&ica i""ovatio" a"# #eve o!me"t i" the soft'oo# umber i"#ustry i" 5a"a#aC A structura mo#e i"& a!!roach. I;;; Tra"sactio"s o" ;"&i"eeri"& Ma"a&eme"t. @1)@., !!. 2264 2@-. 0issema, K.G. )+,12.. Tre"#s i" tech"o o&y forecasti"&. 8 J D Ma"a&eme"t, +2)+., !!. 2/4 @-. 0issema, K.G., a"# Be"es, K. )+,1A.. A cross4im!act case stu#yC the Dutch co"structio" sector. =utures, Bctober +,1A. !!. @,646A6.