METHODOLOGICAL ISSUES IN NURSING RESEARCH

Consulting the oracle: ten lessons from using the Delphi technique in
nursing research
Sinead Keeney BA MRes
Senior Research Fellow, Institute of Nursing Research, University of Ulster, Belfast, UK

Felicity Hasson BA MSc PG Dip

Research Fellow, Institute of Nursing Research, University of Ulster, Belfast, UK

Hugh McKenna BSc DipN AdvDipEd DPhil RGN RMN RNT FRCSI
Dean of the Faculty of Life and Health Sciences, University of Ulster, Belfast, UK

Accepted for publication 22 March 2005

Correspondence: KEENEY S., HASSON F., McKENNA H. (2006) Journal of Advanced Nursing
Felicity Hasson, 53(2), 205–212
Institute of Nursing Research, Consulting the oracle: ten lessons from using the Delphi technique in nursing
University of Ulster,
research
Newtownabbey,
Aim. The aim of this paper was to provide insight into the Delphi technique by
Shore Road,
Belfast BT37 0QB,
outlining our personal experiences during its use over a 10-year period in a variety
UK. of applications.
E-mail: f.hasson@ulster.ac.uk Background. As a means of achieving consensus on an issue, the Delphi research
method has become widely used in healthcare research generally and nursing
research in particular. The literature on this technique is expanding, mainly
addressing what it is and how it should be used. However, there is still much
confusion and uncertainty surrounding it, particularly about issues such as modi-
fications, consensus, anonymity, definition of experts, how ‘experts’ are selected and
how non-respondents are pursued.
Discussion. This issues that arise when planning and carrying out a Delphi study
include the definition of consensus; the issue of anonymity vs. quasi-anonymity for
participants; how to estimate the time needed to collect the data, analyse each
‘round’, feed back results to participants, and gain their responses to this feedback;
how to define and select the ‘experts’ who will be asked to participate; how to
enhance response rates; and how many ‘rounds’ to conduct.
Conclusion. Many challenges and questions are raised when using the Delphi
technique, but there is no doubt that it is an important method for achieving con-
sensus on issues where none previously existed. Researchers need to adapt the
method to suit their particular study.

Keywords: agreement, consensus, Delphi method, discussion paper, experts, nur-

sing, research methodology

particular issue (Beretta 1996, Green et al. 1999). While

Introduction
the Delphi technique has been used in the fields of business
The Delphi technique is a favoured approach for those (Kaynak et al. 1994, Addison 2003, Ilbery et al. 2004),
researchers who are seeking judgement or consensus on a defence (Roberts 1969, Gilbride 2002) and education

2006 Blackwell Publishing Ltd 205

S. Keeney et al.
(Dailey & Holmberg 1990, Volk 1993), studies using the
technique within health research, and in particular nursing
research, are increasing (Alexander & Kroposki 1999,
McIlfatrick & Keeney 2003, Lofmark & Thorell-Ekstrand
2004).
Nonetheless, much confusion, disagreement and uncer-
tainty exist concerning the principles intrinsic to its
application. These include consensus, various modifications,
selection and definition of experts, anonymity and the
number of ‘rounds’ required. Most researchers who have
used the Delphi would acknowledge that it is not until they
undertake a study using the technique that its difficulties
and challenges become apparent (see Hartman 1981, Green
et al. 1999). In this paper, we present a collection of
perceptions and insights into our experience of using the
Delphi technique with reference to the studies in which we
employed the method. Our perceptions were obtained from
a series of informal discussion groups within which
experiences of using the Delphi technique were recorded.
It was apparent from these discussions that there were
common experiences from which a number of key lessons
could be identified. We felt that these lessons could be of
use to any researcher planning to use the classical Delphi
or its many modified versions.
Background
The Delphi technique is a structured process that uses a
series of questionnaires or ‘rounds’ to gather information
which are continued until ‘group’ consensus is reached
(Beretta 1996, Green et al. 1999, Hasson et al. 2001,
Powell 2003). Its popularity has centred on the fact that it
allows the anonymous inclusion of a large number of
individuals across diverse locations and expertise and
avoids the situation where a specific expert might be
anticipated to dominate the consensus process (Jairath &
Weinstein 1994).
The technique involves the presentation of a questionnaire
to a panel of informed individuals in a specific field of
application in order to seek their opinion or judgement on a
particular issue (McKenna 1994a). After the questionnaires
are returned, the data are summarized and a new ques-
tionnaire is designed based on the responses from the first
round. This second-round questionnaire is then returned to
each participant showing the overall group response and the
participant’s own response from round one. Participants are
asked to reconsider their initial response in the light of the
first round’s overall results. Repeat rounds of this process are
carried out until consensus has been reached (Beretta 1996,
Green et al. 1999).
206
Discussion
Our use of the Delphi technique
Hasson (2000) used the Delphi technique with 132
qualified midwives and 69 student midwives to gain
consensus on what were non-midwifery duties (see
McKenna & Hasson 2001). Keeney (2000) used the
technique with 38 primary care nurses, 14 general
practitioners and eight public representatives. The aim of
this study was to try and obtain agreement on the future
structure and function of community nursing in Northern
Ireland (see McKenna et al. 2000). McKenna (1994b) used
the technique with 95 ward managers. He wished to obtain
a judgement from these managers on what models of
nursing they favoured for use in a long-stay psychiatric
setting. Further experience was gained through other
studies. McKenna and Hasson (1998) used the technique
to gain agreement on what should be included in guidelines
for smoking cessation in pregnancy. McIlfatrick & Keeney
(2003) also used the Delphi technique to identify cancer
nursing research priorities in Northern Ireland.
Lesson 1: Understanding the method
The Delphi technique uses a series of data collection ‘rounds’
to capture and structure the knowledge and opinions of a
‘panel’ of participants on a topic with which they are
perceived to have expertise. To enable the identification of a
wide array of views, the first round is usually qualitative in
nature. This is how the ‘Classic Delphi’ commences (Linstone
1978). Data from round one are summarized and a ques-
tionnaire is designed based on results of this summary. The
round two questionnaire is returned to each participant,
indicating their response from round one and how this
compares with the overall panel’s response. As a result,
participants are given the opportunity to reconsider the issues
they identified in round one. Repeat rounds of this process are
carried out until consensus has been reached or when no
further changes are taking place (Jairath & Weinstein
1994). Agreement among panel members is determined by
consensus, voting or through mathematical averaging
(McMurray 1994).
In McKenna’s (1994b) study, the views of 95 partici-
pants on the appropriateness of nursing models were
obtained in round one through one-to-one interviews. In
Keeney’s (2000) study, focus groups were used in the first
round to generate qualitative comments on the future
structure and function of community nursing in Northern
Ireland. In both studies these qualitative data formed

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Methodological issues in nursing research
the basis for closed-ended questionnaires in subsequent
rounds.
In Hasson’s (2000) midwifery skill mix study, participants
were encouraged to identify eight duties that could be
undertaken by unqualified assistants. It is helpful to place a
limit on participants’ listing of the specific items under study,
in this case duties. This helps to curtail the possibility of
hundreds of issues being named, as well as encouraging
participants to prioritize their responses. In round two, duties
identified from round one were subsumed into categories and
respondents were asked which duties would they allocate
confidently to a midwifery assistant or which should remain
the responsibility of a registered midwife. However, Hasson
(2000) acknowledged that presentation of data under categ-
ories may produce a halo effect (Mead & Moseley 2001),
where the responses are about the general category, rather
than about the individual issues raised by participants.
However, researchers have to be cautious as participants
could identify many hundreds of issues in round one.
Returning these items in their raw non-themed form could
overwhelm the participants in round two and affect adversely
the response rate.
As each round unfolds, the participants have to interpret
information sent to them by the researcher. Often the
qualitative results of round one are fed back to the panel
members as medians and ranges or means and standard
deviations. This means members must be skilled in written
communication and the interpretation of basic statistical
averaging. It cannot be assumed that Delphi respondents
possess these skills. In the studies undertaken by the authors,
there was no additional time to train participants in the
interpretation of basic statistics. Therefore, data were
returned to participants in simple text and percentage
formats. This ensured that the information was basic yet
meaningful.
Lesson 2: Number of rounds
One of the basic principles underpinning the Delphi tech-
nique is to have as many rounds as are required to achieve
consensus or until the ‘law of diminishing returns’ occurs.
Provision for feedback and opportunity to revise earlier
responses obviously requires that the technique has at least
two rounds.
In Hasson’s (2000) study, a two-round approach was used,
while for Keeney (2000) it took four rounds before consensus
occurred. Although there are no strict guidelines on the
correct number of rounds, the number can depend upon the
time available and whether the project manager ignited the
Delphi technique sequence with one broad question or with a

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Consulting the oracle
list of questions or events. In addition, the number of rounds
may be decreased to minimize reductions in the amount of
new information and reductions in response rates resulting
from respondent fatigue (Starkweather et al. 1975). For
example, McKenna (1994b) felt that response exhaustion
occurred after two rounds, especially with busy ‘experts’ and
hard-pressed clinicians. These experiences are reflected in the
literature where the number of rounds varies between two
and four.
Lesson 3: Enhancing response rates
In general, questionnaire research is notorious for its low
response rates. Researchers often have to send out two or
three reminder letters to non-responders. With anything up to
four rounds of questionnaires, the Delphi technique asks
much more of respondents than a simple survey and the
potential for low responses increases exponentially.
To enhance responses in the rounds, it is critical that
participants realize and feel that they are partners in the study
and are interested (if not fascinated) by the topic. The
researcher should take every opportunity to remind partici-
pants that each round is constructed entirely on their
responses to previous rounds. This encourages interest,
ownership and active participation.
McKenna (1994a) suggested that the ‘personal touch’
could help enhance return rates. His use of one-to-one
interviews in his first round meant that the participants got to
know him and he was able to remind them individually that
their responses would be fed back to them by questionnaire in
further rounds. In effect, he psychologically signed them up
to continuing with the study. He achieved a 100% response
rate, which is very rare in a Delphi technique study. Starting
with initial contact to build mutual rapport and developing
and nurturing a relationship is necessary to increase the
likelihood of participants’ ongoing commitment. On the one
hand, interviewers can show appreciation for valuable
information and on the other hand, respondents may be
gratified to be an object of interest and by having an
audience. In addition, gaining the commitment of the panel
and providing information on how the study will be
implemented should ensure that respondents feel sufficient
ownership of the study. While this initial stage of informing
the participants affects the response rates fundamentally, it is
often overlooked in the methodological literature.
In her study, Keeney (2000) only had two non-responders
after two rounds. This high response rate was due to her
tenacity and persistence in following up non-responders and
also to participants’ interest in the topic. Hasson (2000) also
achieved a high response rate to round two where 75% of
207
S. Keeney et al.
qualified staff and 84% of student midwives responded.
Accepting the possible introduction of bias, this was helped
by the Director of Nursing writing to staff stressing the
importance of the study and asking them to return their
questionnaires to the researcher. Hasson also acted as a
motivator for the participants: for example, before the study
began she became familiar with each ward and spoke to
individual staff about the research.
In summary, it is important to use every opportunity to
encourage project ownership and loyalty among participants.
Retaining participants across rounds will only occur if their
interest and commitment are retained.
Lesson 4: Modifications of the technique
At present there are no formal, universally agreed guidelines
on the use of the Delphi technique nor does any standard-
ization of methodology exist (Evans 1997). Consequently,
flexibility exists in the design and format of the technique and
this often depends on the study’s aims and objectives. The
most popular formats include: the ‘modified Delphi’
(McKenna 1994a), the ‘policy Delphi’ (Crisp et al. 1997),
the ‘decision Delphi’ (Couper 1984) and the ‘real-time
Delphi’ (Beretta 1996). Few researchers use a pure Delphi
approach and the various hybrids have been criticized by
writers such as Sackman (1975).
As well as the differing Delphi formats, the approaches
within these formats differ. For example, in the traditional
design (Linstone 1978), the content of the first round is
normally obtained from the literature rather than the
qualitative views of participants or from other secondary
data. Other variations of the Delphi exist; for example,
Proctor and Hunt (1994) sent participant nurses three
patient profiles with the remit of identifying the care needs
of each patient while Jones et al. (1992) involved the use of
face-to-face meetings of participants after two initial Delphi
rounds.
As researchers, McKenna (1994b), Keeney (2000) and
Hasson (2000) all used the literature as the basis for their first
round interview/focus group questions. Although the
researchers gave Round 1 participants ample opportunity to
raise fresh issues to avoid early closure on ideas, it should be
noted that this action could introduce bias by making
participants feel psychologically pressured to alter their views
according to the recognized literature.
In summary, the lesson here is to acknowledge that
modification which is not systematic and rigorous may be
problematic. Without care, this could result in a self-fulfilling
prophecy where, like sheep, participants could be steered to
agree on a highly visible issue in the literature. The researcher
208
should allow participants freedom to bring their views to the
first round.
Lesson 5: Selecting the ‘experts’
Most Delphi studies recruit individuals who (based on
criteria) are perceived to have expertise in the subject under
investigation. A researcher must decide on these criteria
before the study commences, such as gender, professional
experience, education, employment or designation. As the
exact composition of the panel can affect the results obtained,
the potential for bias is considerable. Moreover, working on
the principle that experts must have knowledge of the subject
area, participants cannot be selected randomly. For instance,
it makes little sense to seek consensus among non-profes-
sionals on developing smoking cessation guidelines in preg-
nancy when they may have little knowledge of the topic or
the context (McKenna & Hasson 1998).
However, there is clearly a potential for bias in the
selection as those who respond to the initial invitation are
those who are more likely to be interested in the subject
matter. Therefore, the exact composition of the panel can
affect the results obtained. Indeed, Strauss and Zeigler (1975)
have criticized the claim that the Delphi technique represents
valid expert opinion as scientifically untenable and over-
stated.
There are no universally agreed criteria for the selection of
experts, and no guidance exists on the minimum or
maximum number of experts on a panel; rather it appears
to be related to common sense and practical logistics.
Linstone (1978) reported studies using several hundred
participants to a Japanese study that used several thousand
participants.
It could be stated that this problem of panel size is not
unique to the Delphi technique. Other types of surveys
employ non-probability sampling techniques. This does not
ensure accurate representation and results cannot be gener-
alized to other settings or to a wider population.
In his Northern Ireland study, McKenna (1994b) included
as experts the entire population of nurse managers on long-
stay psychiatric wards across six hospitals (n ¼ 95). He
reasoned that only they had the expertise to decide on what
model of nursing care would be the most appropriate for
long-stay psychiatric patients. The decision was based on the
assertion in the literature that ward managers were the
experts so far as clinical setting and patient care were
concerned. Hasson (2000) pursued the entire population of
midwifery staff and students in the largest maternity hospital
in the Republic of Ireland. She too assumed that these were
the most appropriate experts when it came to what was and

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Methodological issues in nursing research
what was not a midwifery duty. In contrast, when investi-
gating the future role of community nurses, Keeney (2000)
selected her panellists on the principle of specialist inclusion.
She wanted to ensure that general medical practitioners,
members of the public and the range of specialist community
nurses had the opportunity to be involved.
In summary, the fifth lesson is that there is no magic
formula to help researchers decide on who are the experts
and how many there should be. Rather, as in other survey
techniques, this decision is often based on funding, logistics
and rigorous inclusion and exclusion criteria.
Lesson 6: Estimation of timeframe
A critical factor in using the Delphi technique is time. This
includes time to allow interviews to be undertaken, ques-
tionnaires to be designed, distributed, returned, analysed and
re-developed for subsequent rounds. This is something that
can be underestimated by researchers using the Delphi
technique for the first time. Duffield (1993) reported that
each round of the technique can take up to 8 weeks to
complete; this is a realistic approximation. While the studies
outlined here were well planned and executed, each one of
the authors underestimated the time it would take to gain
consensus. For example, before entering the field, Hasson
(2000) assumed it would take 3 months to complete the
Delphi process. In reality, it took over 4 months to complete
because of the need to follow up non-respondents and the
time needed to adequately analyse the data. Keeney’s (2000)
study took 12 months to complete in total, with a full
6 months devoted totally to the Delphi stage of the study,
whereas McKenna took 16 months to complete his Delphi
study.
In summary, there is no doubt that a Delphi study is time-
consuming. Researchers should consider that, while a single
questionnaire will provide respondents’ views or opinions on
an issue, a Delphi study will provide a level of consensus or
judgement among the respondents on the issue. The price for
this added value is time.
Lesson 7: Anonymity vs. quasi anonymity
According to Polit and Hunger (1995), true anonymity is
ensured when no one (including the researcher) can link a
response to a respondent. Because of the iteration process
inherent in the Delphi technique. True anonymity cannot be
guarantee: an individual panel member’s responses are
unknown to other participants but they are known to the
researcher. McKenna (1994a) termed this ‘quasi-anonymity’.
This means that non-respondents experience the important

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Consulting the oracle
‘personal touch’ as highlighted above and can be followed up
using reminder letters and telephone calls. However, this may
lead to researcher and subject bias and, as Beretta (1996)
suggests, this could cause respondents to feel forced into
returning the questionnaire.
Hasson (2000) carried out her study in one large maternity
hospital where she included 90% of the staff. Therefore, all
the participants knew that other colleagues were taking part
in the Delphi study. It is possible that some may have
discussed their preferred responses with colleagues. This is a
limitation with any kind of self-report survey and it is
acknowledged that it may threaten quasi-anonymity. Keeney
(2000) was also aware that, while the participants in her
study did not work together, they knew they were taking part
in the same study. This was because Keeney brought her
entire sample together for a workshop at the start of the
process to introduce them to the study. Accepting this, both
Hasson and Keeney believe that, because participants were
aware that they were part of a larger research team, it helped
to increase response rates.
In addition, a problem related to the iterative process
within Delphi studies is the potential for experts to withdraw
after the first round (Evans 1997). This behaviour may lead to
a response bias if the attrition rate is substantial. For
example, in Hasson’s (2000) study, round one was sent to
194 qualified midwives and 79 student midwives. Despite the
use of follow up reminders and personal visits, 138 (68%)
qualified midwives and 69 (87%) student midwives comple-
ted and returned the questionnaire in round one. In round
two, 100 (75%) (
32) qualified midwives and 58 (84%)
(
11) student midwives responded to the final Delphi
questionnaire. A number of techniques are offered to help
encourage non-respondents. For example, one technique is to
invite the experts to take part at the beginning, sending them
a postcard to return which lets them indicate if they would
like to be included in the study or to decline. If they decline,
they are asked to provide the reason (for example, not
enough time) and these data then help to provide a fuller
picture of the final sample.
In summary, there is a possible weakness in the Delphi
technique in that anonymity cannot be guaranteed. While
quasi-anonymity is possible, it can be threatened when the
study is carried out with work colleagues where they may
discuss the study. Nonetheless, knowing who the other
respondents are can also help to increase participation.
Lesson 8: Definition of consensus
As outlined above, the raison d’être for using the Delphi
technique is to gain consensus or a judgement among a group
209
S. Keeney et al.
of perceived experts on a topic. However, experts can differ
and it would be difficult to gain 100% agreement on all
issues. Therefore, a key question in any Delphi study is what
percentage agreement would a researcher accept as synony-
mous with consensus. The answer may lie with the import-
ance of the research topic. For instance, if it were a life and
death issue such as whether or not to switch off a respirator
in an intensive care unit, a 100% consensus level may be
desirable. Alternatively, if the topic was related to the
selection of a new nurses’ uniform, a consensus of 51%
may be acceptable. As with most aspects of the Delphi
technique, the literature provides few clear guidelines on
what consensus level to set. Loughlin and Moore (1979)
suggested that consensus should be equated with 51%
agreement amongst respondents. By contrast, Green et al.
(1999) employed an 80% consensus level. Establishing the
standard is crucial as the level chosen determines what items
are discarded or retained as the rounds unfold. It is good
practice for the research team to establish a definition of
consensus before data collection begins.
The reason why this is so important is based on the fact
that many Delphi studies are undertaken to formulate policy
for funding new research or service developments. Depending
on the importance of the policy, a 51% consensus cut-off
point could lead to low morale or unrest among those who
favoured those views which only gained 50% agreement.
Having only 1% between whether there is consensus on an
issue or not seems harsh and may be difficult to justify.
However, while a 75% requirement for consensus may
appear more robust, an obvious question is how this ‘cut-off
point’ is chosen. The success of the Delphi technique is
predicated on the principle of consensus and more research
needs to be undertaken into how it is determined. One
possible approach is to use confidence intervals. This would
begin to address an obvious weakness in the method.
In his study, McKenna (1994b) selected Loughlin and
Moore’s (1979) level of consensus. While this is a methodo-
logical limitation as outlined above, McKenna found that
most of the statements achieved a consensus of over 70%.
Keeney (2000) used a 75% level of consensus as the standard
set by the research steering group for the study. Hasson
(2000) used the same level for the same reason.
There are no recognized guidelines on an appropriate level
of consensus. From the authors’ perspective, 75% appears to
be the minimal level but there is no obvious scientific
rationale for this. We would suggest that researchers should
decide on the consensus level before commencing the study
and consider using confidence intervals with high level of
importance equalling high level of consensus and high
confidence interval.
210
Lesson 9: Interpreting the results
Most published Delphi studies focus primarily upon the
findings. However, according to Evans (1997), the terms
agreement and consensus are essentially two different ideol-
ogies. Is there a difference between the extent to which each
participant agrees with the issue under consideration and the
extent to which participants agree with each other? When
reporting findings, few studies do so in the context of these
different principles. Most researchers prefer instead to rely
upon participants agreeing with each other. Yet it is
important to note that the extent to which participants agree
with each other does not mean that consensus exists nor does
it mean that the ‘correct’ answer has been found. This is
especially the case when the issues have ethical implications.
For example, 75% of a nursing panel may agree that very
elderly and ill patients should not be put through the rigours
of active resuscitation. This reflects consensus in the Delphi
sense, but it may not be the correct way to care for such
patients.
A panel member may be reluctant to share a view contrary
to the majority of panel members. However, this is how the
Delphi technique works. It is only through seeing the
(anonymous) responses from other panel members that
individual participants are encouraged to reconsider their
views. Delphi purists would argue that panel members
change their minds and move towards consensus because
they see that someone else has identified a more relevant issue
that they had not thought of. Delphi cynics would assert that
panel members are inclined to change their minds because of
a possible mistaken belief that the views expressed by the
majority of the panel must be right. The obvious conclusion
of this assertion is that strong-willed panel members hold
rigidly to their views across rounds and weak-willed panel
members alter theirs. If true, this conclusion challenges
seriously the validity and reliability of Delphi findings.
Therefore, there is a danger of placing too much reliance
upon the final results without acknowledging the influence of
bias and other factors on validity and reliability. To enhance
authenticity, a number of strategies can be used. For instance,
pilot testing could be undertaken, additional methodological
techniques such as focus groups could be integrated or
comparison made with secondary validated data.
In McKenna’s study, a quasi-experimental design was used
as a follow-up method to test the results of the Delphi process
(McKenna et al. 1995). In Keeney’s (2000) study, a ‘consen-
sus conference’ was organized to discuss the validity of the
Delphi results. In Hasson’s (2000) study, a series of
interviews was organized to explore further the Delphi
results with qualified midwives; secondary documentary data

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Methodological issues in nursing research
What is already known about this topic
• The Delphi method is used to gain consensus among a
panel of experts.
• The Delphi method has been a valuable tool in nursing
used to investigate an array of topics and using a variety
of modifications.
• The scientific merit of the method is a subject of debate.
What this paper adds
• The Delphi method can be applied in a number of
modified ways, and needs to be adapted to meet the
needs of a particular study.
• Issues universally faced by researchers using this tech-
nique relate to the selection of experts, quasi-anonymity
and consensus.
• Many challenges and questions are raised when using
the Delphi technique, but there is no doubt that it is an
important method for achieving consensus on issues
where none previously existed.
on midwifery activities was also collected. These additional
approaches helped validate the participants’ judgements.
Interestingly, in all three studies, the Delphi results were
supported.
In summary, the ninth lesson is that researchers must be
aware that validity may be compromised due to panel
members changing highly relevant views in the face of a
different view from the main body of the panel. The extent to
which participants agree with each other (consensus) does not
mean that the ‘correct’ answer has been found.
Lesson 10: Skills of the investigator
The success of the Delphi technique relies upon the analytical
and administrative skills of the researcher. These include
establishing an administration system and analysing and
presenting both qualitative and quantitative data. While these
skills are not given much weight in the literature, their
presence is vital for an effective and efficient Delphi study.
As stated above, the ‘classic Delphi’ collects qualitative
data in the first round and quantitative data in subsequent
rounds. Little guidance is offered in the literature on the type
of qualitative analysis to employ and how to present the
results in the second round of the Delphi study. In the
authors’ studies, opinions collected in round one were
content analysed, with similar items combined into themes.
These statements constituted the qualitative feedback to the

2006 Blackwell Publishing Ltd, Journal of Advanced Nursing
Consulting the oracle
sample in that they could view the variety of statements
generated by the panel. No items were discarded in round one
and the statements for round two were presented in the
participants’ own words as much as possible. This results in a
lengthy and detailed list of round two statements. However,
as Green et al. (1999) reported, active intervention justified
by the need to categorize and reduce statements actually
resulted in the movement of statements further from the
verbatim responses on which they were originally based.
In Hasson’s (2000) and Keeney’s (2000) studies, ongoing
rounds were composed of simple statistical summaries that
illustrated collective opinion. Measures of central tendency
and variance were favoured. The results were presented in a
sample-friendly manner so that participants could easily see
their contribution to the consensus view. This allows the
sample to see the group response and their own individual
response to each statement.
Each of the authors designed their own administrative
system to allow for tracking of individual responses for each
round; this enabled non-respondents to be tracked and
targeted. Employing self-administrative questionnaires re-
quires a mail base to be established and physical and financial
resources allocated to cover costs of postage, printing,
telephone bills and photocopying. These are important for
any research study but vital for the smooth running of a
Delphi survey.
In summary, the tenth lesson is that, like most studies,
the Delphi survey will fail if proper administrative systems
are not in place to ensure the proper processes are
followed. Because of the multiple rounds that make up a
Delphi study, high quality administrative systems are
crucial for success.
Conclusion
In this paper we present the major lessons we have learned in
employing the Delphi technique. As it can be applied in a
number of modified ways, not all of these lessons will be
applicable to all settings. Nevertheless, some issues presented
in this paper, such as the selection of experts, quasi-
anonymity and consensus, are universal problems faced by
researchers using this technique.
Many challenges and questions are raised when using the
Delphi technique, but there is no doubt that it is an important
method for achieving consensus on issues where none
previously existed. If researchers merely want to gauge the
knowledge or opinions of a group of people, there are many
research approaches from which to choose. However, if they
wish to establish consensus or obtain a judgement on an
issue, a Delphi survey may be the appropriate method.
211
S. Keeney et al.

Author contributions Kaynak E., Bloom J. & Leibold M. (1994) Using the Delphi techni-
que to predict future tourism potential. Marketing Intelligence &
SK, FH and HMcK were responsible for the study conception Planning 12(7), 18–29.
and design, data collection and analysis, drafting of the Keeney S. (2000) See McKenna H.P., Bradley M. & Keeney S. (2000)
manuscript, critical revisions of the paper for important Primary Care Nursing: A Study Exploring Key Issues For Future
Developments. University of Ulster, Ulster.
intellectual content, statistical expertise, and administrative,
Linstone H.A. (1978) The Delphi technique. In Handbook of Futures
technical and material support. Research (Fowles R.B., ed.), Greenwood, Westport, CT, USA,
pp. 273–300.
Lofmark A. & Thorell-Ekstrand I. (2004) An assessment form for
References clinical nursing education: a Delphi study. Journal of Advanced
Nursing 48(3), 291–298.
Addison T. (2003) E-commerce project development risks: evidence
Loughlin K.G. & Moore L.F. (1979) Using Delphi to achieve con-
from a Delphi survey. International Journal of Information Man-
gruent objectives and activities in a paediatrics department. Journal
agement 23(1), 25–40.
of Medical Education 54(2), 101–106.
Alexander J. & Kroposki M. (1999) Outcomes for community health
McIlfatrick S.J. & Keeney S. (2003) Identifying cancer nursing
nursing practice. Journal of Nursing Administration 29(5), 49–56.
research priorities using the Delphi technique. Journal of Advanced
Beretta R. (1996) A critical review of the Delphi technique. Nurse
Nursing 42(6), 629–636.
Researcher 3(4), 79–89.
McKenna H.P. (1994a) The Delphi technique: a worthwhile approach
Couper M.R. (1984) The Delphi technique: characteristics and
for nursing? Journal of Advanced Nursing 19 1221–1225.
sequence model. Advances in Nursing Sciences 7, 72–77.
McKenna H.P. (1994b) The essential elements of a practitioners’
Crisp J., Pelletier D., Duffield C., Adams A. & Nagy S. (1997) The
nursing model: a survey of clinical psychiatric nurse managers.
Delphi method? Nursing Research 46(2), 116–118.
Journal of Advanced Nursing 19, 870–877.
Dailey A.L. & Holmberg J.C. (1990) Delphi: a catalytic strategy for
McKenna H.P. & Hasson F. (1998) The Formulation of Guidelines
motivating curriculum revision by faculty. Community/Junior
of Smoking in Pregnancy. University of Ulster, Ulster.
College Quarterly of Research and Practice 14(2), 129–136.
McKenna H.P. & Hasson F. (2001) A study of skills mix issues
Duffield C. (1993) The Delphi technique: a comparison of results
in midwifery: a multi-method approach. Journal of Advanced
obtained using two expert panels. International Journal of Nursing
Nursing 37(1), 52–61.
Studies 30(3), 227–237.
McKenna H.P., Parahoo K.A. & Boore J. (1995) The evaluation of a
Evans C. (1997) The use of consensus methods and expert panels in
nursing model for long-stay psychiatric patient care Part 2; pres-
pharmacoeconomic studies. Pharmacoeconomics 12, 121–129.
entation and discussion of findings. International Journal of
Gilbride J.T. (2002) An Empirical Investigation of Critical Factors
Nursing Studies 31(1) 95–113.
Affecting the Ability of Public Entities to Compete in Public Private
McKenna H.P., Bradley M. & Keeney S. (2000) Primary Care
Companies. Thesis, Department of the Air Force Air University, Air
Nursing: A Study Exploring Key Issues For Future Developments.
Force Institute of Technology, Wright-Patterson Air Fource Base,
University of Ulster, Ulster.
Ohio. Available at: http://research.maxwell.af.mil/papers/ay2002/
McMurray A.R. (1994) Three decision making aids – brainstorming,
afit/afit-gaq-env-02m-09.pdf, accessed on 20 October 2004.
nominal group and the Delphi technique. Journal of Nursing Staff
Green B., Jones M., Hughes D. & Williams A. (1999) Applying the
Development 10(2), 62–65.
Delphi technique in a study of GPs information requirements.
Mead D. & Moseley L. (2001) The use of the Delphi as a research
Health and Social Care in the Community 7(3), 198–205.
approach. Nurse Researcher 8(4), 4–23.
Hartman A. (1981) Reaching consensus using the Delphi technique.
Polit D.F. & Hunger B.P. (1995) Nursing Research Principles &
Educational Leadership 38(6), 495–97.
Methods, 5th edn. JB Lippincott Company, Philadelphia.
Hasson F. (2000). See McKenna H.P., Hasson F. & Smith (2000) A
Powell C. (2003) The Delphi technique: myths and realities. Journal
Multimethod Study of Skill Mix Issues in Midwifery. Conference
of Advanced Nursing 41(4), 376–382.
Proceedings. Royal College of Surgeons of Ireland Annual
Proctor S. & Hunt M. (1994) Using the Delphi survey technique to
Research Conference (Feb, unpublished).
develop a professional definition of nursing for analysing nursing
Hasson F., Keeney S. & McKenna H. (2001) Research guidelines for
workload. Journal of Advanced Nursing 19, 1003–1014.
the Delphi survey technique. Journal of Advanced Nursing 32,
Roberts E. (1969) Exploratory and normative technological forecast-
1008–1015.
ing: a critical appraisal. Technological Forecasting 1(2), 113–127.
Ilbery B., Maye D., Kneafsey M., Jenkins T. & Walkley C. (2004)
Sackman H. (1975) Delphi Critique. Lexington Books, MA, USA.
Forecasting food supply chain developments in lagging rural regions:
Starkweather D.B., Gelwicks L. & Newcomer R. (1975) Delphi
evidence from the UK. Journal of Rural Studies 20(3), 331–344.
forecasting of health care organisations. Inquiry 12, 37–46.
Jairath N. & Weinstein J. (1994) The Delphi Methodology (Part
Strauss H. & Zeigler H.L. (1975) The Delphi technique and its uses
One): a useful administrative approach. Canadian Journal of
in social science research. Journal of Creative Behaviour 9(4), 253–
Nursing Administration 7(3), 29–40.
259.
Jones J., Sanderson C. & Black N. (1992) What will happen to the
Volk K. (1993) Curriculum development using the Delphi technique.
quality of care with fewer junior doctors? A Delphi study of con-
Technology Research 52(4), 35–36.
sultant physicians’ views. J Roy Coll Phys 26, 36–40.

212
2006 Blackwell Publishing Ltd, Journal of Advanced Nursing