BOHEMIA - A DYNAMIC ARGUMENTATIVE DELPHI SURVEY

IN POLICY PREPARATION

Kerstin Cuhls*, Liviu Andreescu×, Radu Gheorghiu×, Bianca Dragomir×, Aaron Rosa*
*Fraunhofer Institute for Systems and Innovation Research and ×PROSPECTIVA; kerstin.cuhls@isi.fraunhofer.de;
andreescul@gmail.com; gheorghiu.radu.cristian@gmail.com; biancadragomir20@gmail.com;
aaron.rosa@isi.fraunhofer.de
correspondence: kerstin.cuhls@isi.fraunhofer.de

Abstract

The paper presents key contents and the underlying methodological design of the project
BOHEMIA: Beyond the Horizon - Foresight in Support of the Preparation of the EU‘s Future
Policy in Research and Innovation. The project was funded by the European Commission
and aimed to aid in the preparations of the next RTDI Framework Programme (FP9). In a
future-oriented design mixing quantitative and qualitative methods to identify and assess
new developments, BOHEMIA combined several scenario-making methods (meta- to mini-
scenarios) and consultation processes to identify priorities and themes for Future European
R&I policy.
This paper focuses on the identification and formulation of future issues in research,
technology, innovation, and society relevant for the project’s goals; on their formulation in the
form of Delphi statements adapted to policy-makers’ needs; and on their assessment in a
Dynamic Argumentative Delphi (DAD) survey. It then describes the subsequent use of these
statements over the project's entirety, including their deployment as inspirations for
recommendations in the official policy-making process.
First, we describe the procedures used to generate statements, which relied on a multi-
method, multi-source approach integrating: media and internet searches, interviews,
foresight activities around the world, and literature analysis. We then trace the processes of
refining the statements and enhancing them, as well as adapting them to suit an online
Delphi survey while keeping them relevant for the European Commission future policy. We
describe the Scoping Workshop with the European Commission's Strategic Foresight Expert
Group and Foresight Correspondents’ Network as a specific step in the selection of the
statements.
Secondly, we explain the methodological underpinnings and the actual structure of the two-
round, online, real-time Delphi employed in the project - the DAD adapted to the purposes.
We provide a wealth of descriptive statistics from the online consultation in BOHEMIA, and
concerning the utilization of participant-generated argument statements for direct use by the
European Commission.
Finally, the paper sums up the lessons learned from such a process in the policy-making
context, including an understanding of inter-institutional dynamics with regard to
interpretation of DAD results; different valuations given to quantitative and qualitative
information; and generative processes that encouraged policy-maker participation. This then
positions our researchers to point towards fertile ground for continued methodological
experimentation and future research directions for the field of strategic decision-making.

Keywords: Argumentative Delphi, preparation of FP9, identifying priorities, BOHEMIA

Introduction
BOHEMIA: Beyond the Horizon - Foresight in Support of the Preparation of the EU‘s Future
Policy in Research and Innovation was funded by the European Commission and aimed to
support the preparation of the next RTDI Framework Programme (FP9). In a future-oriented
design mixing quantitative and qualitative methods to identify and assess new
developments, BOHEMIA combined several scenario-making methods (meta- to mini-
scenarios, see Ricci et al. 2017) and consultation processes to identify priorities and themes
for Future European R&I policy.
The following sections describe the DAD, the procedures used to formulate Delphi
statements on research and innovation developments, integrating different sources of
knowledge. We then trace the processes of refining the statements and enhancing them, as
well as of adapting them to suit a very specific online Delphi survey while keeping them
relevant for the European Commission future policy. The "Scoping Workshop" with the
European Commission's Strategic Foresight Expert Group and Foresight Correspondents’
Network served as an important step in the selection of the statements.
The explanation of the two-round, online, real-time Delphi employed in the project - the
Dynamic Argumentative Delphi survey (DAD) section focuses on the combination of
quantitative evaluation and qualitative assessment via arguments; on sampling the
participants; on the results of the exercise and their significance within the frame of the
project; and on their methodological implications for continued online Delphi research. We
provide descriptive statistics from the online consultation in BOHEMIA, and concerning the
utilization of participant-generated argument statements for direct use by the European
Commission. We present some results and trace them to the end report of BOHEMIA. The
full results are available in Gheorghiu et al. 2017.
Finally, the paper sums up the lessons learned from such a real life process in the policy-
making context.

The BOHEMIA Dynamic Argumentative Delphi Survey

The BOHEMIA study involves three phases with different methodological approaches:
Phase 1: A literature review and synthesis into a set of scenarios that aim to sketch possible
future settings and boundary conditions for the development of future R&I policy and
funding. (documented in Ricci et al. 2017)
Phase 2: A Delphi survey (DAD) to gain new insights into future technologies, societal
issues, and R&I practices. (documented in Gheorghiu et al. 2017)
Phase 3: An analysis and the deriving of policy recommendations, building on the scenarios,
the results of the Delphi and an online consultation (public engagement with the findings).
(Weber et al. 2018 forthcoming).
This paper describes phases 2 and 3 connected to the Dynamic Argumentative Delphi, which
builds on the benefits of a real-time Delphi. The method of DAD was already successfully
tested in previous projects (Gheorghiu et al. 2014).1 The goal is to enable online Delphi
consultations with a large number of participants (in the hundreds or more), while retaining the
interactive ‘argumentative’ (i.e., justification-based) nature of the traditional Delphi. Since the
processing of respondent arguments typically entails a lot of manual effort during several

1Specifically, in three large-scale consultations in Romania: the consultations supporting the elaboration of the Vision for
Romanian Higher Education 2025 (year 2012, 1000+ respondents, www.edu2025.ro); smart specialization in Romania (year
2014, 4000+ respondents, www.cdi2020.ro); the prioritization of policies for the National Strategy for the Consolidation of
Public Administration (year 2014, 400+ respondents).
rounds (being thus the main barrier to extending the number of participants), DAD has
automated this process by introducing a few simple rules:
 Delphi statements are evaluated quantitatively (e.g., on probability, impact etc.); the
DAD is associated in the online questionnaire with 2-3 ‘starter’ arguments which
(together with all arguments added subsequently by respondents) are always visible
to participants. The expanding set of arguments serves, as in traditional Delphi
formats, as justifications for the quantitative estimates.
 Respondents are invited to enter their quantitative estimate and justify it by selecting
at least one pre-existing argument, or providing at least one new argument, or both.
The maximum number of arguments that may be added/ selected by any individual
respondent is usually limited to two or three.
 The list and frequency of arguments, updated with the newly selected/ introduced
ones, is always visible to subsequent respondents. The arguments in the list are
ranked by the number of votes gathered during the exercise (explicit in brackets).
Unlike the constantly updating list of arguments, the quantitative estimates are visible
only to the individual participant making them.
As a result, the number of participants can be increased substantially without a proportionate
addition of manual processing effort. Immediate feedback can be given and the benefits of the
traditional Delphi format are retained. Respondents reflect on the justifications for their
quantitative estimate before introducing them. They do so by consulting previous respondents’
arguments, hence ensuring a measure of inter-subjectivity in the exercise. Participants may –
and, indeed, are encouraged to – return to the online questionnaire to consult the updated
lists of arguments and to revise their judgements and arguments in their light.

Statement Formulation, Arguments, and Scoping Workshop

The BOHEMIA Delphi statements were not meant to provide a broad overview of ‘all things to
come’ but, rather, to focus the consultation on research and innovation (R&I) issues – in
science and technology, societal issues, and R&I practices – relevant for the coming
Framework Programme. The first draft statements and starting arguments were elaborated
based on several sources:
 scanning of internet and literature (e.g., Web of Science);
 the meta-scenarios in phase 1 of the BOHEMIA project;
 media analysis;
 internal consortium workshop (which took place in May 2016 and aimed to pre-cluster
the topics from the scanning phase and from other sources);
 interviews (statements proposed by ‘generalists’ and ‘experts’).
The interviews were conducted with about 40 experts and stakeholders external to the
European Institutions, mainly international foresight and strategy experts (but also
researchers, engineers, and scholars, as well as representatives of the European institutions,
Member States’ foresight institutions, research funding agencies, academies, and relevant
European non-governmental organizations). The first interviews were based on an open,
semi-standardized interview guideline (for face-to-face, phone, Skype, e-mail or written
interaction), the second on first lists of statements, and the third was asking for specific details
and estimations on single statements.
In addition, the members of the Foresight Correspondents’ Network were strongly involved in
the statement formulation through discussions, workshops, and by providing literature and
research. The aims of the interviews were to identify the most important areas to be covered
by the Delphi survey, to formulate a draft set of Delphi statements for the questionnaire, and
to establish a draft list of experts to be involved in the Delphi survey. Three types of interviews
were performed in order to: (i) define the fields and areas that need to be covered, (ii) identify
new/ additional issues and statements to be included into the Delphi survey, and (iii) verify the
statements and arguments, their focus, and their formulation. An iterative approach was used,
in which respondents built on previous suggestions in order to considerably improve the quality
of the results.
The statements were further refined, backed up with literature, and formulated as single, short
sentences. They were discussed, enhanced and reformulated in a Scoping Workshop with the
Strategic Foresight Expert Group and Foresight Correspondents’ Network (November 30,
2016) in Brussels. During this event, the statements were grouped according to a ‘field of
knowledge’ to which they were associated. The Commission's Foresight Correspondents were
invited to discuss the field of knowledge statements in small groups. In cases where there was
not enough knowledge or different sources were necessary, feedback was asked for
subsequently. With this feedback, new or alternative statements were integrated into the list.
Subsequently, the statements went through several additional waves of improvement,
feedback, updating via literature and short interviews, and interaction with the Commission.
Originally, about 50 statements were intended to be included into the survey, but when starting
the survey, the Delphi contained 147 statements. They concerned future states of affairs of
presumed relevance for Europe. The statements were grouped into several ‘fields of
knowledge’. The next step was to identify individuals able to understand and assess the
statements.

The Design of the Questionnaire

The Delphi questionnaire was designed in line with the DAD method to meet the key objectives
of the consultation on the one hand and to maximize users’ ease of navigation and
engagement through the filling in process on the other hand. The Delphi questions were about
an estimation of the time of realization and the significance of the individual statement. The
time of realization was estimated in 5 year intervals from 2025 to 2040. As regards significance
there were two types of questions:
a) In the case of S&T statements, significance referred to the importance of R&I for the
realization of the state of affairs described by the statement; and
b) In the case of research statements, significance referred to the importance of the statement,
assuming realization, for future EU R&I policy.
The User experience

On accessing the questionnaire, the participant was asked to choose, based on her or his
expertise and interests, one among a number of fields under which the Delphi statements
were clustered (see Figure 1). Every respondent was also invited to return, at any point after
completing the first field, to select additional fields (for a maximum of three).
Figure 1: Selecting a field of expertise (S&T Developments)

The second list to choose from were research and societal clusters:
 Economy: Sharing, Growth, Sustainability
 Education & Skills
 Longevity & the Health System
 Natural Environment, Climate Change & Oceans
 Society, Culture, Digitalization & Governance
 R&I Practices
 Working Conditions & Employment

After selecting the field(s), the user was provided with the first Delphi statement accessible
per browser page (figure 2ff). When the tasks on the page were completed, the respondent
was able to move on to the next statement until all statements under the selected field were
tackled. Skipping a statement was possible.
The participant was then invited to estimate the statement’s ‘time of realization’ by selecting
from a dropdown list (figure 2):
 2025
 2030
 2035
 2040
 Beyond 2040
 Never
 Don’t know

After the selection of a value, the respondent was asked to support his or her assessment with
arguments, either picked from a list of already existing arguments, or newly added by
her/himself.

Figure 2: Estimating the time of realization for Delphi statements (example from the survey)

Then, the rating system to evaluate the significance of R&I for the realization of the statement
(1 to 5 stars) was shown. The respondent was invited to provide a justification for the
assessment by selecting or adding arguments (figure 3).
Figure 3: Assessing the significance

Note: For the class of statements in the image above, the significance assessed is that of ‘R&I for the realization
of the statement’. The task was different for the second class of statements.

After finishing with an individual Delphi statement, the respondent could move through all the
statements associated to the selected field. S/he was then invited to select another field, upon
which the procedure was repeated up to a maximum of three fields. When the participant
finished the field’s statements, the system sent an invitation to access the questionnaire for a
second time, in order to examine previous participants’ assessments (specifically, the
distribution of scores); and, possibly, to revise theirs in light of the new information (figure 4).
Access to the new section of the questionnaire was open immediately – but could be delayed
at the participant’s will – if at least 15 respondents answered previously. When this was not
the case, participants were informed that they would be invited to do so at a later date, after
this critical number was reached.
Figure 4: Revising the statements

Figure 5: The table of distributions - feedback

The participant was invited to keep or revise his or her original evaluation of the time of
realization, as well as previous arguments (which could not be deleted, but deselected; new
arguments could be added). After completing the first or second round of the questionnaire,
respondents were asked to fill in information on their ‘profiles’. The data concerned expertise
and individual demographics. On the expertise profile page, participants could leave short
comments on the BOHEMIA consultation.
The Participants

The identification and selection of potential participants started early on in the project. The
main criterion in participant selection was ‘expertise’, broadly defined as having a sufficient
level of knowledge about a ‘field’ in the survey, that is the ability to assess (almost) any specific
statement in the field. Thus, experts were identified according to the substance and structure
of statements and their clustering into fields. It has to be noted that this broad definition of
expertise is common in Delphi designs. Delphi samples include ‘specialists’ with niche
expertise, as well as ‘generalists’ with an interest in a broader area. As the statements
changed during the formulation process, the selection of new experts went on until the survey
was officially launched.
The experts asked to participate in the Delphi survey were identified in various stages of the
project and added to the email database until the time the survey went live.
 The first groups identified consisted of experts listed in existing databases available to
consortium members (e.g., individuals who participated in previous Delphi or other
surveys), whose competences and areas of knowledge were known, and who were
familiar with this survey logic.
 Secondly, persons were added from screened (existing) data bases (e.g., issue- or
sector-specific) based on the initial content of each field. Experts from all over Europe
were sought to the extent to which their coordinates were publicly available, so as to
comply with the data regulations of the European Union.2
 Thirdly, we searched for conferences on specific issues and for individual experts using
semantic analysis and web mining tools. In the Web of Science, keywords were used
to search through scientific journal articles – and the authors identified were also asked
to participate.
 The fourth step was a small co-nomination or snowball sampling process, asking a
limited number of identified experts if they knew others who should be involved. The
members of the former European Foresight Platform (www.foresight-platform.eu) were
also invited.
Altogether, the plan was to address about 15,000 persons from all over the world so as to
ensure a turnout of at least 500 respondents. The target threshold for experts based outside
Europe, whose input was considered necessary among others to avoid a ‘European bias', was
set at 10%. In order to include the views of ‘informed outsiders’ on the topics at hand, we also
targeted directly science journalists, writers on science blogs, as well as members of European
students’ organizations, doctoral students in broader research areas, persons from Young
Researcher Programs, and a mixed sample of students from universities.
The survey went live on 5 May 2017 and closed on 18 June 2017, after the original deadline
(2 June) was extended on two occasions. Eventually, the preset figure of respondents was
exceeded: more than 10% of the participants came from a country outside the EU. A
comparatively larger number of participants came from the countries in the consortium
(Austria, Germany, Italy, Romania); the Netherlands and the UK, among others, are also well
represented. To sum up the participation statistics at the end of the survey:
 1488 users registered on the platform; of whom
 820 finished filling in the questionnaire for at least one field of knowledge;

2There were cases, in which the expert was identified, but neither country of residence nor the gender were
available.
  727 users went through the whole survey once, completing one or more fields of
knowledge (up to 3) as well as their profile page;
 133 of the latter category revisited the survey to consult the distribution of scores and
an updated list of arguments in their field(s) and to potentially revise their own initial
input.
The charts below provide some additional statistics on participation (figure 6ff.):

Figure 6: Distribution of respondents by country

As the previous paragraph suggests, there was an interest in achieving adequate

representation for all age bands among respondents. This was complemented by an effort to
have a fair gender balance.
Figure 7: Distribution of respondents by gender

Figure 8: Distribution of respondents by age

Figure 9: Distribution of respondents by sector of activity

Figure 10: Distribution of respondents by field in Science and Technology Developments

Figure 11: Distribution of respondents by field in Developments in Society and R&I Practices
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

Some Delphi results

The Delphi statements were at first grouped in fields, then for the survey into two major classes
(see figure 1). The statements in each field were analysed individually and as a set, according to
the variables in the Delphi consultation (such as ‘time of realization’ and ‘significance of R&I’).
Figure 12 and the tables below provide an example of a statement, selected arbitrarily, together
with the associated arguments; and of the analysis (ranking) of statements.
Figure 12: Artificial Intelligence is used in systems that support policy-makers’ decisions across
the EU (original formulation)

Number of respondents: 46

Arguments regarding the time of realization No. of

votes
Artificial intelligence is already used and will be used to support decisions, but not 38
as a substitute for human decision-makers.
Artificial Intelligence - sophisticated models, or models developed by learning 33
machines - are already used on a routine basis in all sorts of decisions.
Any policy decision can be supported by AI if fed with carefully selected data and 19
criteria.
AI is used in elections and that use will grow and poses dangers for democratic 10
governance as information bubbles are created to manipulate decision-making.
Siri, Google Assistant, Alexa, Cortana: individuals will naturally move from 7
personal assistants to assistants in public decision-making.
Decision-makers will never defer to robots (especially when they don't like the 4
latter's decisions).

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 13 -
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

Significance of R&I

Average: 4.09 Dispersion: 0.87

Arguments regarding the relevance of R&I No. of

votes
We need to understand the blindspots of algorithmic decision-making in specific 38
decisional contexts.
Research is needed to identify the elements of the decision process under the 21
control of AI so that humans have the potential to review decisions on demand and
understand drivers influencing the AI.
Scientists have only now begun to look into whether, how, and when humans trust 18
robots' advice; more research is needed.
Also the basic R&I is needed. The algorithms and practices are developing fast and 10
these competences are critical in global competition.
Security concerns over data abuse/misuse and other aspects of adversarial 8
machine learning and AI need to be researched.
Decision making process will become to require AI tools, because they will become 2
to be not able to decide without them.

It became obvious that these pieces of information are interesting for the policy-makers in
different fields, in our case the European Commissions' DGs and different departments. For
them, the general analysis and ranking was of minor importance. For further analyses, for
example rankings were used, e.g. by Time of realization and significance (table 1 and 2).

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 14 -
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

Table 1: S&T statements likely to become reality in 2030 at the latest (i.e. 2025+2030 > 60%)
ranked by average significance

Field No. Statement Significance Dispersion

of R&I of
significance
Robotics & 7.1 The majority of the EU population use 4.33 0.77
Autonomous integrated Artificial Intelligence devices and
Machines machines in their daily lives.
Security 9.3 A large EU city (population greater than 500 4.22 1.08
000) sees a major, prolonged disruption of
services (longer than 48 hours) due to
cyberattacks on critical infrastructures.
Manufacturing: 6.7 Adaptive materials and surfaces that are able 4.18 0.88
Production to change their characteristics according to
Processes, user requirements appear in consumer
Materials & applications.
Nanotechnology
Robotics & 7.3 Artificial Intelligence is used in systems that 4.09 0.87
Autonomous support policy-makers’ decisions across the
Machines EU.
Transport & 8.4 Electric mobility (fuel cells, solar cars etc.) has 4.01 1.15
Mobility replaced 20% of the internal combustion
engine vehicles on EU roads.
Security 9.2 A successful physical or digital attack on low 3.71 1.95
orbital space satellites causes one major
system (global positioning systems, telecom,
security) to fail completely for an extended
period of time.
Information, 5.9 More than 70% of all distributed systems use 3.63 0.90
Communication, blockchain technology to build trust, e.g. in
Brain & Media person-to-person interactions, for all direct
Technologies business transactions and the traceability of
supply chains.

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 15 -
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

Table 2: Research statements likely to become reality in 2030 at the latest (i.e. 2025+2030 >
60%) ranked by average significance

Field No. Statement Significance Dispersion

for R&I of
policy significance
Society, Culture, 14.7 Data literacy (personal privacy, handling data 3.87 0.89
Digitalization & etc.) is taught in all primary schools in the
Governance EU.
Society, Culture, 14.1 The number of people residing in the EU 3.75 1.09
Digitalization & whose original nationality is not from their
Governance country of residence exceeds 80 million (EU
population 2016: 508 million).
Education & 11.6 More than 50% of European universities 3.69 1.30
Skills offer open online learning platforms free of
charge.
R&I Practices 15.10 Value and ethical conflicts relating to science 3.62 1.20
and technology multiply to the extent that
formal processes of ethical approval are
established for every new EU research
funded project.
R&I Practices 15.1 Systematic scanning of biodiversity to 3.5 1.00
identify substitutes for drugs has become a
global research endeavour.
Longevity & the 12.4 The number of EU citizens who identify 3.48 0.96
Health System walking or cycling as their primary mode of
transportation has doubled (compared to
2016).
Society, Culture, 14.3 The right to disconnect from work e-mail in 3.12 1.39
Digitalization & free time is incorporated into the labour
Governance regulations of all EU countries.

Only a few statements were regarded as unlikely to happen - but many of them were assessed
with uncertainty or judged with uncertain prospects and a wide distribution of answers.
Mini Scenarios and Recommendations for Policy-making
After the analysis of the Delphi data, the statements were qualitatively re-clustered to so-called
"targeted scenarios". For them, the most significant theses were taken out from the Delphi list,
then explained in context and enhanced by further information from additional sources.
Examples for targeted mini scenarios are: Assisted Living, Bio-economy, Cheap Renewable
Energy, Continuous Cyberwar or Decision-making supported by open expert systems (full list in
Weber et al. 2018, for an example see the box).

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 16 -
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

Box: ICT-based security and defence (targeted or mini scenario)

It is 2040. Globalisation and ICT solutions have changed the nature of threats faced by the EU.
A combination of preventive and response measures are implemented in coordination by
security and defence forces with the aid of computers. The role of the anticipatory crime units is
rising, together with the diffusion of unmanned aerial vehicles and military robots with Artificial
Intelligence features. These are used in external military actions as well as to secure national
territories in cooperation with security units.
In a workshop with the foresight correspondents’ network and an online consultation with a wider
audience of stakeholders the mini scenarios were assessed once more, and policy
recommendations were formulated that directly address the four major transition fields lying
ahead of the European Commission (Weber et al. 2018). This task was already announced in a
high-level-group report (Lamy 2017).
Figure 13: From the Delphi statements to future directions for EU R&I policy

Source: Weber et al. 2018

With the growing significance that the notion of "missions" has acquired recently in the context of
both European and national R&I policy, BOHEMIA has thus strived to generate results that are
at a comparable level of granularity, thus serving as a source of inspiration for upcoming political
debates about priorities for the next European framework programme.
Conclusions
The methodology of a Dynamic Argumentative Delphi is well suited to foster a discussion with
argumentations about future issues. It delivers valuable additional information, quantitative and
qualitative, stemming from experts with different expertise.
The DAD needs a lot of time for the preparation of statements, if it is performed in the classical
Delphi way. The way the BOHEMIA Delphi was performed, asking for time of realization and
significance, makes it necessary to formulate the statements very strictly and, if possible,
specific to the point of including an absolute number or percentage. Moreover, to add arguments
to two questions is demanding for the participants. Thus, the number of statements for each
participant needs to be low not to de-motivate the potential respondents. This is to say it is
essential not to underestimate the process-time plus the time the survey is in the field, from both
organizers and participants, in running a DAD.

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 17 -
 6th International Conference on Future-Oriented Technology Analysis (FTA) – Future in the Making
Brussels, 4-5 June 2018

In the BOHEMIA case, the higher number of statements (147, instead of 50 originally intended)
and questions had implications for the design of the Delphi survey. A grouping of the statements
was necessary to help participants in the decision, which "field" to address (see figure 1),
considering their background on the one hand, and the content of future developments on the
other hand. It also required a higher number of potential participants to reach a statistically
significant sample for the analysis, and starting arguments for each of the 147 statements had to
be prepared.
DAD needs much more methodological and programming preparation and more testing than
conventional Delphi surveys. It is more difficult to keep it simple and easy to handle than the
classic variation of (Real-time or two-round) Delphi. On the other hand, it delivered much more
content and argumentation that both were used in the following process. Targeted (mini)
scenarios were derived with a lot of additional information from the survey, and hints for
transition areas were also given that could directly be used for policy-maker recommendations.
The arguments themselves often delivered direct recommendations for European policy-making.
Ultimately, the question is: Was it worthwhile applying the Delphi method?
We were sceptic at the beginning - and the effort of preparing Delphi statements and questions,
programming, testing (we needed a lot more testing than usual), asking people to join and
convincing them to come back has to be considered when deciding for Delphi, and specifically
for DAD. Even though this was a real time Delphi survey, people were only asked to come back
and judge once more. Therefore, we have no experience on how a further discussion would
have unfolded if people were allowed to come back multiple times. We expect there would not
be many more returning participants, as the DAD is time-consuming and intellectually
demanding - the respondent needs to think about the previous results, read all the arguments
and answer once more. However, looking at the results, there is a rich fund of statements and
estimations supported by argumentation that indicate which directions participants think are
probable, which are improbable, which are significant and which are less, and where policy-
makers may have a stake to change the future.

References

Gheorghiu, R.; Dragomir, B.; Andreescu, L.; Cuhls, K.; Rosa, A.; Curaj and Weber, M. (2017): New Horizons: Data
from a Delphi Survey in Support of European Union Future Policies in Research and Innovation; Publication of the
European Commission/ European Union. Report KI-06-17-345-EN-N; ISBN 978-92-79-76452-3; doi:10.2777/654172
or https://ec.europa.eu/research/foresight/index.cfm (access 13-5-2018)
Gheorghiu, R.; Andreescu, L. and Curaj, A. (2014): Dynamic argumentative Delphi: Lessons learned from two large-
scale foresight exercises, Paper presented at the 5th International Conference on Future-Oriented Technology
Analysis (FTA) - Engage today to shape tomorrow Brussels, 27-28 November 2014
Lamy, P. et al. (2017): LAB – FAB – APP. Investing in the European future we want. Report of the independent High
Level Group on maximising the impact of EU Research & Innovation Programmes; European Commission.
http://ec.europa.eu/research/evaluations/pdf/archive/other_reports_studies_and_documents/hlg_2017_report.pdf#vie
w=fit&pagemode=none (access 13-5-2018)
Ricci, A, Cessa, C.; Weber, M. with P. Schaper-Rinkel, and S. Giesecke (2017): New Horizons: Future scenarios for
research & innovation policies in Europe - Publication of the European Commission/ European Union. Study. KI-04-
17-245-EN-N, 978-92-79-66614-8; https://publications.europa.eu/en/publication-detail/-/publication/b2d78a84-3aae-
11e7-a08e-01aa75ed71a1; DOI:10.2777/906131 (access 13-5-2018)

SESSION SHAPING RESEARCH POLICY FOR EUROPE THROUGH FORESIGHT

- 18 -