Technological Forecasting & Social Change 87 (2014) 274–291

Contents lists available at ScienceDirect

Technological Forecasting & Social Change

Beyond the veil — The real value of Foresight

Effie Amanatidou ⁎
Manchester Institute of Innovation Research, University of Manchester, Manchester, UK

a r t i c l e i n f o a b s t r a c t

Article history: The wide application of foresight would benefit from a common assessment framework that
Received 8 October 2012 hardly exists. This would require a higher level of reference, i.e. pursuing more generic goals.
Received in revised form 23 August 2013 This is offered by the two concepts of “knowledge society” and “participatory governance”. The
Accepted 28 December 2013 aim of the research is to develop an impact assessment framework of foresight programmes in
Available online 19 January 2014
developing more participatory “knowledge societies” beyond their specific aims.
Research shows that the major impacts of foresight belong to three groups, i.e. in relation to
Keywords: knowledge, network creation, and promoting public engagement in policy-making. At the same
Foresight time, the major features of modern societies are of three types, i.e. related to knowledge value, to
Impact assessment
innovation-driven growth and to consequences of a “risk society”. Thus, the relevant areas where
Participatory societies
foresight might contribute are: knowledge, networking, and coping with a ‘risk society’.
The new framework is built on the features and pre-conditions of more participatory societies
and draws upon existing evaluation approaches and concepts (“theory-based evaluation”,
“knowledge value framework”, “behavioural” and “cognitive capacity additionality”) to tackle
short-comings of earlier evaluation efforts. It is then tested in a case study that demonstrates
its feasibility and comprehensiveness and further refines the assessment criteria it is based on.
© 2014 Elsevier Inc. All rights reserved.

1. Introduction forging new social networks, guiding strategic visioning, creat-

The term ‘foresight’ has been increasingly used since the
late 1980s. Numerous definitions exist, emphasising the one or
the other element of foresight. For instance Coates [33] puts an
emphasis on scanning and forecasting but not specifically
oriented to science and technology. Slaughter [34] on the
other hand, relates foresight to decision-making and organisa-
tions. Several accounts exist describing foresight activities
dating back to the first application of foresight programmes
which in essence reflect the actual aims of the specific
programmes [21].
Across the various attempts to define foresight and identify
its fundamentals, certain elements are repeated. Foresight is seen
as an action-oriented instrument for policy-making facilitating
structured anticipation, considering alternative futures [23],
requiring creative thinking and multi-disciplinarily perspectives,
enabling collective learning [11]; proactive and path-breaking,
interactive and participatory; enabling mediation and alignment,
⁎ Pavlou Mela 16, P.O. Box: 1698, P.C. 57500 Trilofos Thessalonikis, Greece.
E-mail addresses: Effie.Amanatidou@mbs.ac.uk, Effie.Amanatidou@gmail.com.
ing, and committing actors to shared visions [17], and supporting
deliberative democracy [14].
These major characteristics governing foresight can be
grouped in three major building blocks, i.e. in relation to
building knowledge, building networks and building partic-
ipation and action:
• Building knowledge: as leading to the development of
strategic visions and anticipatory intelligence considering
alternative futures based on a multidisciplinary base and
through evidence-based approaches, interactive and partic-
ipatory methods of analysis and collective interactions
enhancing collective learning;
• Building networks: as a process of co-production of commu-
nities of stakeholders and as an instrument of transaction,
dialogue, negotiation, cooperation and alignment among them;
• Building participation and action: as bringing more stake-
holders and points of view into the decision-making process
with an orientation to inform present-day decisions, coordi-
nate agents and policies and shape behaviours and routines in
view of taking concrete actions towards the realisation of a
jointly defined future vision.

0040-1625/$ – see front matter © 2014 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.techfore.2013.12.030
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
Foresight has been increasingly acknowledged in the past
two decades as a valuable policy-making process. Despite the
considerable investment in foresight exercises in several coun-
tries, not many attempts have been made so far for a thorough
evaluation of the process, outcomes and impacts of foresight
programmes [30]. Most evaluations of the few conducted so far
are post hoc, working with hindsight and with a tendency
to focus on effects, rather than on the processes and on the
choices made inside the foresight projects comprising a foresight
exercise.
Most foresight activities, although sharing some methodo-
logical characteristics and similarities in terms of time horizons
and audiences addressed, usually have different aims, scopes
and levels of implementation. Thus, evaluations typically focus
on assessments of whether or not these goals have been
attained while there is no common evaluation and assessment
framework of foresight exercise or programmes [2].
Foresight evaluation also faces a number of challenges. The
wide range of situations where foresight is applied hinders the
development of a common evaluation approach. The systemic
and distributed nature of foresight also makes it necessary to
assess impacts across a variety of actors and systems, which
poses another challenge in its evaluation [18]. The intangible
and long-term nature of several of its benefits causes attribution
problems [5].
The wide spread application of foresight would benefit
from such a common evaluation approach. It would facilitate
the identification of good practices irrespective of their
specific objectives and levels of implementation while it
would also allow for benchmarking of foresight programmes.
A common approach to foresight evaluation would require a
higher-level of reference in terms of goals and objectives. It
would also need to take into consideration the challenges
faced in foresight evaluation.
The aim of the present paper is to present a new
evaluation framework for foresight programmes allowing a
common approach in evaluating foresight and overcoming
the shortcomings of earlier attempts in foresight evaluation.
The methodological approach applied is to consider the
challenges and needs in foresight evaluation that have to be met
in creating a common evaluation framework. This is done
by exploring the basic features of the higher-level of goals that
have to be set and by exploiting the wider field of evaluating
socio-economic development programmes. The methods ap-
plied are literature reviews and interviews with experts1 in the
respective fields of interest, i.e. evaluation of socio-economic
development programmes, foresight, and major socio-economic
and governance features of modern societies. The new evalua-
tion framework is then tested by assessing FNR Foresight a
national foresight programme from Luxembourg.
The paper unfolds in the following six sections. Section 2
discusses the challenges that this new higher level of reference
would imply for foresight evaluation. It also reviews the
challenges in foresight evaluation that have emerged in earlier
evaluation efforts as the new framework would need to tackle
both types of challenges. Section 3 explores the specific areas
where foresight might contribute in developing more partici-
patory “knowledge societies” while Section 4 reviews relevant
1
A total of 19 experts were interviewed.
275
evaluation theories and concepts that may help overcome the
identified challenges in evaluation. Drawing on the result of the
previous sections, Section 5 presents the new framework
and Section 6 puts it in practice through the assessment of
FNR Foresight. The last section discusses the conclusions both in
terms of the applicability of the framework as well as its
novelties, limitations and further areas for research.
2. Implications for foresight evaluation
For a common evaluation framework to be feasible a
higher level of reference is needed, i.e. the attainment of
higher level, generic goals. Amanatidou and Guy [2] showed
that this “higher”, generic level of reference is offered by the
two concepts of “knowledge society” and “participatory
governance”. The course towards the most competitive and
dynamic knowledge-based economy2 in the world was set as a
target by the EU (Lisbon European Council — March 2000).
The specific emphasis on “knowledge-based economies”
enabling economic and social development has been retained
in the EU strategic orientation all through the decade. The
latest EU 2020 Strategy [13] puts forward three mutually
reinforcing priorities. The role of knowledge and innovation
is explicit in relation to one of them, “smart growth:
developing an economy based on knowledge and innova-
tion”. Being at the core of the EU strategies, research and
innovation acquire a more crucial role in the framework of
dealing with the so-called ‘global’ or ‘major’ societal chal-
lenges spanning from over-exploitation of natural resources,
to food, water or energy shortages, or ageing populations
(Boden, 2010 as referred to in [8]).
The nature of societal challenges, where current EU policies
focus on, is complex and difficult to describe as they are usually
interrelated and their causes and consequences are yet to be
completely understood. In addition, they are boundary-spanning
in several respects, requiring, for example, interdisciplinarity,
cross-departmental coordination and coherence of policies,
multi-level governance approaches at global, regional (e.g.
European), national, and sub-national levels, technology conver-
gence or fusion in finding solutions, cross-sectoral collaboration
between various industries as well as longer-term time horizons
in policy-making and business planning practices [8].
At the same time “participatory governance” has also gained
importance over the years. Since the European Commission's
White paper on European governance [12] in 2001, the
importance of participatory processes in policy making has
been increasingly acknowledged especially in complex cases
with uncertain, yet far reaching, consequences. This is exactly
the nature of societal challenges being targeted by the EU and
2
There are no established definitions of either a ‘knowledge-based
economy’ or a ‘knowledge society’. At a first sight the two terms, even
though lacking a broadly accepted definition, seem to be used interchange-
ably in the literature. However, several scholars agree that a ‘knowledge
society’ is a broader and more holistic concept including, apart from a more
pronounced role of knowledge in the economy with products and services of
increased knowledge-related added value, changes in the area of gover-
nance like the re-birth of the populi against questionable scientific and
industrial purposes [4] or the growth of social movements [24], or changes
in the social and cultural spheres including the increasing significance of
‘social learning’ as well as the changes in demographic structures, social
values and life-styles [26].
276 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
national policies in research and innovation. Major societal
challenges emphasise the importance of engaging a wide range
of stakeholders, including citizens [8]. Consequently, instead of
fading away over the years, the two “pillars”, the “knowledge
society” and “participatory governance”, where a common
framework can be based to evaluate foresight exercises, have
become even more relevant and important over the years.
The new evaluation framework also needs to tackle the
short-comings of early foresight evaluation efforts. Scholars
point out the importance of different types of process-related
impacts that have long been overlooked till now such as
effects in relation to social learning or networking [23]. In
this regard, process-related (rather than input or output)
performance criteria are a more useful indicator of success
(Martin and Irvine (1989) cited in Tait and Williams [30]).
The importance of unanticipated impacts is also highlighted
as certain impacts may be difficult to identify and measure or
may not be known from the outset, which in turn calls for a
more open, exploratory evaluation approach.
Evaluation frameworks have been developed distinguishing
evaluation outcomes and foci between the different rationales
and foresight functions [18] and acknowledging the systemic
nature of foresight exercises [29]. In all these efforts a common
characteristic is the high significance given to the role of the
external environment and context in affecting the course of
foresight activities.
The new evaluation framework should refer to the achieve-
ment of more generic goals like the contribution to developing
more participatory ‘knowledge societies’ and should allow
adequate consideration of not only the internal features of
foresight programmes but also its external environment and
relevant interactions. It should also put primary emphasis on
examining all possible sorts of impacts (tangible or intangible,
intended or unintended) as regards different types of stake-
holders and time-lines.
3. Foresight's contribution to more participatory,
“knowledge societies”
After studying the relevant literature on the pre-conditions
and factors affecting the development of modern societies as
well as the literature on foresight features and possible
impacts, Amanatidou and Guy [2] concluded that foresight
may strengthen certain features that can make modern
societies more participatory and knowledge-driven. These
features can be summarised as follows:
• Knowledge creation, absorption and diffusion — and through
these to the increasingly dominant role of knowledge;
• Social capital and networking — and through these to
support for innovation-based growth;
• The alignment of actors' interests, their active participation in
dealing with uncertainty, the development of informed publics
and, through all of these, to the evolution of strategies to cope
with or escape from the negative consequences of a “risk
society”.
Foresight can serve certain functions which can form three
distinct but inter-related groups. These are “informing and
directing policy”, “facilitating decision-making and policy
implementation”, and “embedding more stakeholders” views
in policy-making. These are underlined by certain rationales and
correspond to certain key objectives as shown in Table 1. [2]
These three functions are not excluding each other. Yet, each
one is differentiated from the others by specific features that are
of major importance to the specific function. The first function
primarily aims at supplying anticipatory intelligence in directing
policy. This may indeed be an aim of the two other functions as
well. The differentiating feature here is that the primary focus is
the production and provision of anticipatory knowledge. The
emphasis is not on the networks and coalitions built in producing
and implementing it (which is the focus of the second function)
nor on bringing new and diverse actors in the debates in
improving inclusiveness, transparency and legitimacy (which is
the focus of the third function).
The emphasis of the second function is on the networks,
strategic partnerships and coalitions built around shared future
visions and the consequent collective learning taking place
within them. Facilitating networking also helps embed more
stakeholders' view in policy-making, thus making the second
and third functions look similar. However, it is the diversity of
the actors engaged as well as the primary focus in improving
governance by supporting inclusiveness, transparency and
legitimacy that differentiates the third function from the second.
In reality foresight programmes usually serve more than one
function at the same time but to different extents.
The three functions are in line with the three main building
blocks of foresight features (presented above in Section 1) i.e.
building knowledge (informing function), building networks
(facilitating function), and building participation and action
(embedding participation function).
They are also characteristic in terms of the impacts they
may lead to. The “informing” foresight function leads to
knowledge-related impacts, the “facilitating” function stress-
es impacts associated with networking and collective learn-
ing, while the “embedding” function primarily addresses
increased public participation. These types of impacts are
presented in detail in Table 2.
Thus, foresight, as characterised by the three major functions
can contribute to the development of more participatory
“knowledge societies” in three ways. Foresight impacts can
affect the knowledge-related features of emerging ‘knowledge’
societies and are central to the “informing” function of foresight.
Impacts related to the second area, of social capital and
networking, are central to the second foresight function, i.e.
“facilitating policy implementation”. The third area of contribu-
tion directly relates to the “embedding participation” role of
foresight in policy-making, although actors' alignment features
can also be seen as crucial for the “facilitating” role, too.
4. Inspiration from other evaluation theories and practices
As already mentioned (Section 2) it is important in foresight
evaluation to study the factors affecting the success of a foresight
programme and the wider context where a programme is
designed and implemented. The intangible and long-term nature
of several foresight impacts also has to be dealt with. Assessment
has to cover anticipated as well as unanticipated impacts across a
variety of actors and systems. Process-related performance
criteria and impacts have to enjoy equal, if not more, attention
than product-related ones.
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291 277

Table 1
Main foresight functions with associated rationales, and objectives.

Function Rationale Key objectives

Informing and directing policy planning
through extended/improved knowledge and
improved capacities in strategic thinking.
Facilitating decision-making and policy
implementation through platforms for
interaction/joint learning and strengthened
ties, alignment and mobilisation of actors
around shared visions.
Embedding more stakeholders' views
in policy-making through increased &
more diverse base of actors involved
in decision-making.
• Informing funding & investment priorities Extending breadth of knowledge about
• Eliciting research and innovation agendas future, supplying strategic anticipatory
• Reorienting science and innovation systems intelligence, building capacity and framing
• Benchmarking national science and innovation systems knowledge into policy-support.
• Raising the profile of science and innovation
to attract investments
• Providing anticipatory intelligence to system actors
• Increasing understanding about the future
• Informing policy and public debates
• Building networks/coalitions around shared problems Collective knowledge creation and learning,
• Building trust among actors unused to working together aligning and mobilising actors around shared
• Aiding collaboration across administrative/epistemic visions, strengthening inter-actors' relations,
boundaries and nurturing a foresight culture.
• Highlighting inter-disciplinary opportunities
• Changing mindsets about the future
• Building future visions to help system actors chose
develop. paths
• Enabling informed buy-in to decision-making processes
• Increasing number/involvement of system actors Bringing new actors in strategic debate, providing
• Extending range of types of actors in decision-making social forums for strategic reflection, debate and
action and improving governance by supporting
inclusiveness, transparency and legitimacy.

Source: extended version of Table 2 in Amanatidou [1].

Success of foresight endeavours is influenced by a wide
range of factors, internal to foresight exercises as well as
external ones referring to the wider environment where
foresight is implemented. The need to devise a more open,
exploratory evaluation approach has to be met along with the
requirement to find appropriate evaluation and assessment
approaches placing special emphasis on the value of
networks, knowledge related elements, as well as learning
effects, and capacity building.
The literature is rich in terms of factors considered to
influence foresight and related endeavours. While some refer
to elements internal to, and thus controllable by, the
foresight exercises, there is a variety of others that refer to
equally important, but external, elements that fall outside the
control of the foresight exercise. The internal factors include
those related to the foresight internal elements like actors
and stakeholders (foresight participants and beyond), the
objectives and scope, the methods and processes of an
exercise as well as the inputs and outputs. External factors
include factors related to the external environment where
foresight is applied, i.e. the institutional context, structures,
socio-cultural context, and governance features [2].
The review of the accumulated knowledge surfaced a
wealth of information about relevant evaluation frameworks,
notions and concepts that could inspire the development of a
new foresight evaluation framework tackling the challenges
faced so far. The theory-based evaluation approach [10,31,32]
seems to address challenges as the above. The specific
approach focuses on building a sound programme theory
against which the performance of the specific programmes
will be judged. The building of this theory includes existing as
well as alternative causal links and mechanisms of delivery of
programme outputs and impacts. Adopting such a wide and
open approach, theory-based evaluation facilitates the
exploration of anticipated as well as unanticipated, tangible
as well as intangible impacts.
In presenting the wealth of statements produced the
programme theory matrix [16] is a relevant tool. Such a matrix
structures information along elements like outputs/impacts,
success criteria and factors, programme activities and resources,
performance information and resources of data. Yet, to be as
complete as possible the matrix should also include the
statements and assumptions about the mechanisms underlying
the way a programme works or should work. It should also shift
the focus from programme activities as the centric mechanism of
change to the participants' responses to these activities and
should try to identify unintended impacts alongside intended
ones. If these points are tackled then the matrix approach can be
a useful tool to apply in foresight evaluation.
To acknowledge the importance of networks, and social
configurations in knowledge production, the “knowledge
value framework” and the associated notions of “knowledge
value collective/alliances” ([7,35]) become relevant. A
“knowledge value collective” is the set of individuals who
interact in the demand, production, technical evaluation,
and application of scientific and technical knowledge. It is
characterised by diverse users and producers, pursuing
multiple goals, and influencing one another's knowledge
uses by enhancing the available reservoir of scientific and
technical information through a decentralized, but loosely
integrated process. The “size” of a knowledge value collec-
tive depends on factors such as general awareness of the
body of knowledge, the breadth of its uses, the skills
required to obtain and apply information [7]. A sub-set of a
“knowledge value collective” is called “knowledge value
alliance” and originates with the activation of a knowledge
compact, usually, although not necessarily, through a formal
alliance agreement [35].
The “knowledge value framework” uses the “churn”
model of knowledge use and transformation which is consi-
dered particularly useful in cases where there is diversity of
knowledge objectives; where parties external to the scientific
278 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
Table 2
Foresight impact classification by foresight function.
Foresight functions
Informing and directing policy planning through extended/improved
knowledge and improved capacities in strategic thinking.
Facilitating decision-making and policy implementation through platforms
for interaction/joint learning and strengthened ties, alignment and
mobilisation of actors around shared visions.
Embedding more stakeholders' views in policy-making through increased &
more diverse base of actors involved in decision making.
Source: improved version of Table 4 in Amanatidou [1].
community play an important role; knowledge development is
an inter-organisational, inter-institutional enterprise; and where
work is not easily contained within a single field or discipline [7].
The relevance between the above features with characteristics
of foresight endeavours is obvious. The criteria for evaluating
the knowledge value that was produced within a “knowledge
value collective” or “alliance” include (a) growth, (b) human
knowledge development, (c) social capital, and (d) resilience
[6].
The notion of “knowledge value collectives” follows on from
the work on “techno-economic networks” [9,28]. However, the
notion of “techno-economic networks” is valuable for another
reason. In developing a foresight impact assessment framework,
it offers the concept of the “desired network”. This notion can
become the “ideal” knowledge value alliance that should be the
Impact types
Improved knowledge base about future:
• Better understanding of SWOT and dynamics of change
• Assessment of existing strategies and policies
• Increased risk awareness — effective contingency planning
• Detection and analysis of weak signals
Framing knowledge into policy support:
• Disclosure of hidden agendas and obstacles
• Agenda setting
• (Funding/investment) priority setting
• Better informed strategies and policies
Capacity building:
• Increased experience in using foresight tools
• Use of foresight within organisations/other settings
• Strategic thinking/vision building capabilities
Collective learning and knowledge creation:
• Knowledge flows across diverse groups
• Breaking of boundaries (geographical., institutional, disciplinary)
• Development of shared visions
Foresight culture development:
• Stimulation and inspiration to apply foresight
• Increased involvement in foresight activities
• Thinking out of the box
• Challenge of mindsets
Improved decision/policy-making:
• Better management of pressures and challenges
• Enabling buy-in to decision-making processes
• Disclosure of conflicts/self-reflecting among actors
• Alignment and commitment to shared visions
Improved system actors and inter-actor relations:
• Trust building between system actors
• Improved communication and collaboration
• Links/networks across diverse groups
• New networks and strengthened communities
• Empowerment of (new) actors
• Enhanced reputation and image
Improved policy systems:
• Reconfiguration of policy systems/bodies
• Overcome of path dependency and lock-ins
• Improved coherence of policies
Strengthened role of society:
• Intensification of public debate
• Deepening of dialogue with society
• Building of policies on societal debates
• Orientation of innovation to societal needs
• Development of societal actors' identities
Improved governance:
• Transparent, open, participatory, legitimate governance
• Broadened participation — democratic renewal
proxy against which to assess the knowledge value alliances
created under a specific foresight programme.
Finally, a set of long standing evaluation issues are reported in
the literature according to which public interventions are usually
evaluated. The value lies on “behavioural” [20] and “cognitive
capacity additionality” [19] corresponding not only to the latest
rationales in evaluation but also to the great importance given by
foresight to changing values and behaviours and enhancing
cognitive capacities.
5. The new framework for foresight evaluation
In summary, the review of the relevant evaluation
literature shows that a variety of existing approaches,
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
frameworks notion and criteria can be drawn upon to offer
foresight evaluation:
• a specific methodology (based on theory-based evaluation,
the certain criteria for ascertaining causality and tools for
presenting results like the ‘programme theory matrix’);
• an appropriate framework and focus (based on the
‘knowledge value framework’, and the notion of ‘knowl-
edge value collective/alliance’);
• a conceptualisation model (‘churn’ model) for knowledge
use and transformation, and the related criteria for valuing
knowledge;
• the notion of the ideal ‘knowledge value collective’ or ‘alliance’
(based on the ‘desired techno-economic network’); and
• the notions of ‘behavioural’ and ‘cognitive capacity’ addition-
ality as the major evaluation issues for foresight.
The “theory-based evaluation” approach offers two main
elements to the new framework. First, it lends the idea of the
“foresight programme theory” and “matrix”. The foresight
programme theory is essentially the assertions that govern the
foresight programme's design and operation. The “foresight
programme theory matrix” is a tool that helps illustrate the main
features, factors and related mechanisms of delivery of impacts
of the foresight exercise in a structured, systematic and complete
way.
The second and most important contribution is that it shows
the value of examining alternative mechanisms of delivery of
impacts. This is done by reviewing relevant, non-programme
literature not only to search for alternative mechanisms of
delivery of relevant impacts but also for unintended impacts
based on accumulated knowledge in social science fields. This is
highly relevant for foresight evaluation as it acknowledges and
gives equal attention to both intended and unintended impacts.
The new framework brings forth the importance of impacts like
knowledge creation, including creativity and inter-disciplinarity, or
learning effects, network and capacity building. It offers a new
approach in evaluating and measuring such types of impacts by
drawing upon the concepts of “knowledge value framework”,
and “knowledge value collective/alliance”.
As a result, the new concepts of “foresight value alliances”
(FVA) are introduced. The “foresight value alliance”, is
defined as a specific group of people brought together
under each distinct activity of a foresight programme (like
expert panel, or Delphi survey, or scenario workshop) for the
specific purpose of creating knowledge and using it within
the FVA or across FVAs under a specific foresight exercise. It is
used as the unit of analysis for the foresight exercise for
measuring impacts in relation to knowledge creation and use,
human and social capital and sustainability networking.
Another new element is the introduction of the “ideal
FVA” drawing upon the concept of the “desired network”. The
concept of the “ideal FVA” is defined as the ideal type of
magnitude and diversity of human capital, and social capital
that the FVAs should include in maximising their impacts.
Finally, the new framework addresses concepts of “be-
havioural” and “cognitive capacity additionality” as the major
evaluation issues. This is because in this way it acknowledges
the importance given by foresight to changing values and
behaviours and enhancing cognitive capacities as foresight
impacts.
279
The new framework unfolds around three main stages as
presented below.
5.1. Stage 1: the “foresight programme theory matrix”
The aim of this stage is to construct the first part of the
“foresight programme theory matrix”, i.e. its “horizontal”
elements consisting of the internal and external factors affecting
the course of the foresight exercise as shown in Table 3. These
internal and external factors are called “horizontal” because they
are generic enough to apply to all different foresight functions.
This stage starts with the identification and clarification of the
foresight programme's rationale and goals. This is done mainly
through the analysis of the relevant foresight programme
documents. Based on the rationale and goals of foresight the
respective foresight function that the specific programme mostly
abides by is then identified based on Table 1.
The second step includes the identification of the factors, both
in relation to the foresight exercise as well as the wider
environment that are considered important for the course of
the exercise. The identification of these factors is done by
searching formal and informal programme documentation as
well as interviews with the programme managers and “clients”.
This will indicate the reasons it is believed that the activities as
designed would lead to the expected impacts. The outcome of
this step should be certain ‘if–then’ or similar statements
pointing to mechanisms, i.e. the “engines” that drive the given
exercise or programme and are believed to make it effective.
Apart from structuring the first part of the “foresight
programme theory matrix”, it is also important at this initial
stage to define the unit of analysis, i.e. the FVAs that are to be
formed. The FVAs need to be identified after clarifying the
activities of the foresight exercise. The criterion should be a
specific group of people brought together with a specific
purpose of producing certain outcomes based on new
knowledge produced under a specific activity. Inevitably
some of the members of one “foresight value alliance”, e.g. a
thematic panel, will also participate in other ‘foresight value
alliances’ too, like a scenario development workshop or a
Delphi survey. The total of the FVAs formed under a foresight
exercise or programme forms the “foresight value collective”
of the particular foresight exercise.
To define the starting point of the FVAs it is important to
estimate some basic features as these were at the start of the
exercise, i.e. the magnitude of human capital capacity of the FVAs
(number and profile of members); and the FVA social capital
features (the extent of collaboration among the members of the
Table 3
The “foresight programme theory matrix” template — horizontal elements.
Horizontal elements
Function Internal factors/criteria External factors/criteria
Informing & directing Actors: Institutional context
Facilitating … …
Embedding Processes & methods Structures
… …
Inputs/outputs Socio-cultural
… …
Objectives Governance
… …
Source: compiled by the author.
280 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
FVA, across the different FVAs and between FVA members with
external peers before the foresight exercise).3
5.2. Stage 2: the complete “foresight programme theory matrix”
This stage includes supplementing the “foresight programme
theory matrix” with additions of relevant theories from available
social science literature that might shed light on missing links or
might identify new mechanisms to explain observed results and
also identify unintended but possible impacts. This stage applies
the assertion of the “theory-based evaluation” that examining
alternative mechanisms of delivery of impacts should be a major
step in evaluation. This stage also includes searching for elements
to sketch a first picture of the “ideal” FVAs.
Both intended and unintended impacts are function-specific.
The same applies to the associated factors and mechanisms of
delivery of impacts. Thus, at this stage it is the function-specific
elements that the “foresight programme theory matrix” is
complemented with as shown in Table 4.
The complete “foresight programme theory matrix” consists
of both the “horizontal” and “function-specific” parts which are
joined together by the “Function” column.
The first stage gathers all available information about the way
the specific exercise was designed, organised and expected to
deliver the intended impacts. The second stage records the
anticipated impacts as expected from the specific exercise as well
as the alternative/unintended impacts that might emerge from
the same foresight mechanisms of delivery. In parallel it seeks
information about alterative factors, and consequently mecha-
nisms of delivery, that might have helped any of the anticipated
or unanticipated impacts to appear.
Recording of the intended impacts can be done through
programme documentation and interviews with programme
managers, clients and participants. A first idea about possible
types of intended impacts based on the functions that the specific
foresight exercise aims to fulfil can be drawn from Table 2.
Unintended, potential impacts may be recorded by studying the
relevant social science literature. A first review resulted in
Table 5. An effort is also made to distribute unintended impacts
under the three major foresight functions.
Reviewing the relevant social science literature also sheds
light on the main factors and conditions that have to be in
place for the emergence of such type of impacts. Learning
about these alternative factors and the way they affect the
emergence of related impacts helps understand what alter-
native mechanisms of delivery of impacts might have been
activated within the foresight exercise but not intentionally.
As these alternative factors directly refer to impacts which
correspond to the three foresight functions, they are called
“function-specific” factors. As with the “horizontal” factors,
they can be translated to appropriate criteria for assessing
the qualities of the foresight programme and its wider
environment under Stage 3 below.
3
Assessing the impacts of the foresight exercise implies the assessment of
the impacts from participation in FVAs as well as the evolution of the above
features over time. For this purpose specific indicators are suggested
referring to type and use of new knowledge produced, magnitude and
diversity of human capital capacity, social capital development, generative
power as well as longevity and sustainability of the FVAs. These indicators
are presented in Stage 3 (Subsection 5.3) where the actual assessment of the
FVAs is presented.
Table 4
The ‘foresight programme theory matrix’ — function-specific elements.
Function-specific elements
Function Intended Un-intended Intended/unintended Factors/
impacts impacts impact indicators criteria
Inform
Facilitate
Embed
Source: compiled by the author.
5.3. Stage 3: assessment of the foresight programme
Stage 3 is about assessing what actually happened vis-à-vis
the way things were designed and expected to happen (i.e. Stage
1) as well as vis-à-vis what else did happen or could have
happened based on relevant accumulated knowledge and
experiences in relation to developing more participatory,
knowledge-driven societies (i.e. Stage 2). This stage also includes
the assessment of the FVAs formed and the degree to which they
are close to the “ideal” FVAs as prescribed by available literature
and experiences.
The assessment involves three aspects: a) assessing the
“foresight programme theory”, b) assessing the FVAs, and c)
assessing impacts towards more participatory “knowledge
societies”.
The aim of the first step, assessing the “foresight programme
theory”, is to assess the degree to which the factors and
mechanisms alleged to be responsible for delivery of impacts,
as formulated during Stages 1 and 2 (cf. Tables 3 and 4) are
indeed valid for the specific foresight programme. This will be
done primarily by reviewing programme documentation,
documentation about the wider environment conditions and
interviews with programme managers and participants.
The second step, i.e. “assessing the FVAs” will also be based on
specific foresight programme documentation, along with the
interviews with programme managers, contractors and partici-
pants (Stage 1). This will provide a basis to understand how the
initiators thought the foresight panels/groups had better be
constructed. The extended review of relevant literature (Stage 2)
will provide a first hint as to how the “ideal FVAs” should look
like in the specific case.
More specifically, the actual FVAs will be assessed in
terms of specific indicators:
• FVA human capital indicators such as number and profile of
members and relevant changes during the exercise; share
of people entering and leaving the FVA over time during its
existence;
• FVA type and growth indicators (to assess the new knowledge
created) such as the type of knowledge created in each FVA
formed (specialised vs. generic; single vs. multi-disciplinary);
the use of the new knowledge within the FVAs formed and
relevant changes over time; the magnitude of use of the new
knowledge within the FVAs and across the different FVAs; the
scope of problem domain that FVAs dealt with and relevant
changes over time; and the generative power of the FVAs
created, i.e. ability to spawn new FVAs enabled by the work
done to attack new problems;
• FVA social capital indicators (to assess social capital
development based on the knowledge generated in the
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
Table 5
Unintended but possible impacts (based on social science literature).
Foresight functions
Inform and direct
Facilitate
Embed
Source: compiled by the author.
FVAs) such as the type of involvement of the FVA members
(passive vs. interactive; fixed vs. flexible) and relevant
changes over time; the extent of collaboration caused by
the knowledge produced within and across the FVAs and
between the FVA members and external peers during and
after the foresight exercise;
• FVA resilience indicators (to assess sustainability of the
FVAs, and thus of the networks created under the foresight
exercise) such as the length of existence of the FVAs; the
resilience of individual membership, i.e. the average length
of time that members participated in the FVA; the rebound
capacity of the FVA, i.e. the ability to rebound from internal
and external changes; and the longevity of the FVAs, i.e. the
ability of a FVA to sustain itself over a long period of time
after the foresight exercise.
The final step is to compare the actual impacts vis-à-vis to
the expected impacts (Stage 1) as well as vis-à-vis possible
but unintended impacts (Stage 2). A first recording of both
intended and unintended impacts is provided in Tables 2
and 5. This step will involve the collection of qualitative and
quantitative information in order to assess the impacts. Methods
for data collection may include desk research, interviews,
questionnaire surveys, case studies, etc. as well as special studies
like cross-impact analysis, network analysis, etc.
6. The new framework in practice: evaluating
FNR Foresight
The present section presents the results of implementing
the new foresight impact assessment framework in the FNR
Foresight exercise carried out in Luxemburg. Apart from
examining the applicability of the framework and revealing
its limitations, this section can also be regarded as a
step-by-step guide in implementing the new framework in
the evaluation of foresight exercises.
281
Impact types
Related to knowledge creation, absorption & diffusion
1. Increased knowledge creation, absorption and diffusion in/across individuals
2. Increased knowledge creation, absorption and diffusion in/across organisations
3. Increased availability of knowledge and information
4. Skills development (multidisciplinary team work, social skills, strategic thinking, etc.)
5. Capacity building to exchange and create knowledge at individual/organisational levels
6. General human capital development
Related to networking and collective learning
1. Creation of sustainable networks for knowledge sharing and creation
2. Expansion/diversification of social capital of individuals and ‘organisations’
3. Birth of knowledge spill-overs
4. Increased creativity and innovation generation in individuals/networks
Related to enhancing public engagement in decision-making
1. Enlargement of cognitive spaces of individuals
2. Enhanced apprehension of risks and uncertainties
3. Increased variety of sources for information
4. More informed and concerned people
5. Increased public trust; Increased trust in individuals
6. Enhanced public participation processes/institutionalisation of participatory processes
7. Actors' collaboration/alignment to tackle risks
FNR Foresight was a foresight exercise organised by the
National Fund for Research (Fonds National de la Recherche —
FNR) in Luxembourg. It was carried out between 2006 and 2007
by FNR, in association with the Ministry of Culture, Higher
Education and Research. The main aim of FNR Foresight was to
identify new research domains to support when the first cycle of
FNR programmes ended in 2007. This scope was extended by the
Ministry of Higher Education and Research (MCHER) who
aspired FNR Foresight to support the identification of priorities
at the national level that could guide the development of centres
of excellence in niche areas. In total the aims of FNR Foresight
[15] consisted of
• identification of research domains in the public sector with
short-term and/or long-term socio-economic interest for
the Luxembourg society;
• identification of areas on which to concentrate public
investment for R&D with a view to develop real centres of
excellence in Luxembourg;
• support of the formulation of new FNR programmes in terms of
programme priority axes and programme objectives;
• clarification and dissemination of the strategic aim of FNR
programmes among stakeholders; and
• consolidation of communication networks among stakeholders.
As prescribed by the assessment framework presented in
Section 5, the first stage should include the execution of the
following steps: a) identification of the most relevant
foresight functions to the specific foresight exercise based
on its rationale and goals, b) identification and characterisa-
tion of the internal and external factors affecting the specific
foresight exercise, and c) definition of the FVAs (foresight
value alliances) that were created.
The first three objectives clearly refer to the first foresight
function, that of informing and directing policy-planning. The
two last objectives imply some relevance with the second
282 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
foresight function especially in terms of aligning and mobilising
actors around shared visions, and strengthening inter-actors'
relations. The third foresight function is hardly represented in
the FNR Foresight objectives. As a result it is mostly knowledge
and networking related impacts that can be mainly anticipated
by FNR Foresight.
FNR Foresight had six different FVAs4 (i.e. the different
expert panels that were set up) that were organised per
scientific area: the Physical sciences & engineering (PSE),
Environmental science (ENV), Social sciences–Humanities
(SS&H), Law, economy and finance (LEF), the Information and
Communication Technologies (ICT) and the Biomedical/Life
sciences (BLS). The FVAs came together on different occasions,
i.e. during the experts' workshops which were held twice and
the exploratory and young researchers' workshops which were
one-off events. The exercise has two distinct methodological
phases and the final outputs were a list of research priorities
that were directly implemented in the formulation of the next
FNR programme for support of research activities.
To define the starting point of the FVAs, the diversity of
the participants' profiles was considered adequate covering
all related sectors even though with a bias towards public
research in Phase 1 which was later balanced in Phase 2.
Regarding the social capital features, i.e. type and extent of
existing collaboration, FVA members noted that while there
was communication among members of the FVAs there was
less collaboration within the members of the same FVA and
even less across different FVAs i.e. across different scientific
disciplines. On the other hand, collaboration with external
peers was considered more given the increased networking
more at international than national level.
The identification of the internal and external factors that
were considered to have affected FNR Foresight along with their
characterisation based on the desk research and interviews that
were carried out are presented in Table 6. This table forms the
horizontal elements of the FNR “Foresight programme theory
matrix” (cf. Subsection 5.1).
Stage 2 of the framework prescribes the completion of the
matrix with the relevant function-specific elements, i.e.
possible intended and unintended impacts and associated
factors and mechanisms of delivery (cf. Subsection 5.2). Lists
of possible intended and unintended impacts are already
compiled in Tables 2 and 5 respectively. These lists form the
basis for identifying the most relevant impacts of FNR
Foresight. At the same time desk research and interviews at
this stage are oriented towards identifying the main factors
and alternative mechanisms for delivery of impacts. These
are presented in the first two columns of Table 125 (cf.
Subsection 6.3).
Stage 3 includes the actual assessment as presented above
(cf. Subsection 5.3). This was based on review of programme
documentation and relevant literature review, and inter-
views with the programme managers, the “clients” and the
participants.6 As prescribed by the framework, the assess-
ment stage includes three main tasks: a) assessing the
4
FVAs stand for “foresight value alliances”.
5
The third column presents the actual assessment of the mechanisms of
delivery of impacts which is practically executed at Stage 3 (cf. Subsection
5.3).
6
In total 20 individuals were interviewed.
“foresight programme theory”, b) assessing the FVAs, and c)
assessing impacts towards more participatory “knowledge
societies”. The results of these tasks for FNR Foresight are
presented in the following sections.
6.1. Assessing the FNR Foresight programme theory
In assessing the rationales of the exercise, the interview
results confirmed the initial indications. Overall, the most
relevant (above 2.5 on a 1–3 scale) rationales belonged in the
first group of rationales corresponding to the “informing and
directing” foresight function. In detail the chosen most
relevant rationales were “informing priorities”, “eliciting
agendas”, and “informing policy and public debates”. Accord-
ingly the first group of key objectives and foresight functions
were chosen as most representing the FNR Foresight exercise.
The second function had also some relevance as the
rationales “highlighting inter-disciplinary opportunities”
and “building future visions” were also chosen even though
more moderately. The third foresight function indeed was
represented least of all with no rationales getting scores
above 2 (on a 1–3 scale).
Some interesting differences across the views of the
organisers and the participants were also noted. These
differences may be explained by possible variations in
understanding the meaning of the key objectives and
foresight functions. Yet, they may point to certain findings
that should be further explored. First, the participants
appreciated their embedding to policy-making through the
foresight exercise and considered that the exercise intended
to raise the profile of science and reorient the science and
innovation system towards societal needs more than these
were intended by the organisers. This points to possible
unintended impacts in these areas.
Secondly, participants could not see the breaking of
administrative/epistemic boundaries or “collective knowl-
edge creation and learning, aligning and mobilising actors
around shared visions, strengthening inter-actors' relations,
and nurturing a foresight culture” as main intentions of the
exercise as much as the organisers did. This indicates possible
shortage of achievements in relation to the second foresight
function “facilitating decision-making and policy implemen-
tation” while it may also reflect unrealistic expectations on
the side of the organisers.
Regarding the main methods used based on the Foresight
Diamond [27] most of the methods (9 out of 16) fell within
the two corners of expertise and evidence while only some
[5] reach the interaction corner. At the first instance, the
specific exercise did not seem to be equipped to fulfil the
second foresight function successfully. Instead it was well
designed to fulfil the first foresight function as most of the
methods applied drew heavily on expertise and evidence.
6.1.1. Assessment of the internal factors and mechanisms of
impact delivery
FNR Foresight had clear clients/sponsors but with divergent
objectives that failed to be reconciled throughout the exercise.
Thus, the resulting research priorities, wide enough to set
research priorities for research programmes, did not necessarily
lead to the degree of focus and concentration envisaged by the
Ministry. The different objectives and expectations of the two
Table 6
Identification and characterisation of FNR Foresight internal and external factors.
Internal factors for FNR Foresight
• Actors in terms of:
1. ‘clear’ client(s); absorptive capacities
of client(s); ‘foresiqht champion’;
2. degree of high-level political support/commitment;
3. reputation, credibility, independence of
implementation institution(s);
4. fragmentation of participating institutions;
competence and ability to take actions;
5. expertise in foresight processes
and capacity to engage in foresight;
• Processes & methods in terms of:
1. Accordance/relevance of methods to
problems/questions/objectives;
2. Nature of methods drawing on experience,
expertise, creativity & interactions;
3. Early engagement (policy-makers/society);
extent and type of engagement;
4. Existence/quality of implementation
plan, communication plan and
media coverage;
5. Transparency, and openness of processes;
6. inclusiveness of processes, diversity of
members, extent of wide participation;
• Inputs and outputs in terms of:
1. Scientific rigour, reliability, accuracy;
interdisciplinarity, richness of perspectives
2. Relevance, usability, timeliness to
policymaking;
3. Creativity, richness of perspectives in outputs;
• Objectives and background:
1. Clear, non-divergent objectives; clear focus;
well-articulated scope & intervention logic;
Characterisation of internal factors
Clear clients: FNR and the Ministry
Commitment by FNR and the Ministry
FNR well positioned (in Ministry) to influence
policy-making
Fragmentation across research centres, private
sector; University at its start;
Foresight training for FNR and use of foresight
experts in the organisation and execution
Two different phases; first, descriptive and
exploratory; second, focused on prioritisation;
Methods mainly desk research, interviews and
workshops
Attention to invite all sectors (public, private, etc.)
Existence of plan for implementation and
communication of results by FNR
Attention to invite all sectors (public, private, etc.)
Attention to invite all sectors (public, private, etc.);
Phase 1: quite diverse inputs Phase 2: more focused
inputs and discussions
Direct implementation of results in next FNR
programme
Outputs: research priorities assessed along a wide
range of criteria
Varying views on focus and main objectives
between FNR and the Ministry
External factors
• Institutional context in terms of:
1. Compatibility with strategy of implementing
institution(s)
2. Inter-institutional configurational relations;
3. Changing political circumstances;
Other?
• Structures/state of innovation system:
1. Type and state of innovation system & performance;
2. Promotion of new ideas on political agendas;
3. Degree that structural changes are needed;
4. Resources availability;
• Socio-cultural context in terms of:
1. Tradition (type and extent) of public engagement;
2. Tradition (type and extent) of public protest; public trust;
3. Perceived utility of foresight in general;
4. Culture encouraging creativity/innovation;
5. Divides/individualisation in society
• Governance:
1. Position of foresight in policymaking;
2. Degree governance culture promotes public participation;
3. Public awareness
Characterisation of ext. factors
High compatibility with FNR strategy and
good timing;
Fragmentation across research actors as well
E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
as across Ministries;
‘Many-hats’ syndrome
Industry oriented; in need of structural changes;
A high conservative society
High recognition that significant changes are
needed;
Considerable increase of R&D budget
Involvement of stakeholders in policy-planning
No strong culture of protest; high trust in
institutions, consensus society
No foresight ‘tradition’ nor ‘evaluation culture’
High conservative country; no culture for creativity
Different thinking between public and private
sectors
FNR directly linked with policymaking;
Tradition of consensus building and inclusion of
stakeholders
No science culture
283
284 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291

Table 7
Assessment of FNR internal foresight factors and associated mechanisms of delivery of impacts.

Assessment of internal factors of FNR Foresight Mechanisms of delivery of impacts/hypotheses Assessment of mechanisms

Actors in terms of:

• Divergence of objectives between FNR and MCHER;
questioned absorptive capacities of clients; emergence
of ‘champions’ in some workshops.
• Unclear degree of ownership of results by MCHER;
• FNR's reputation and credibility high (but setting not
the right one to set nation-wide priorities).
• Difficulty in developing a joint vision but positive that
these actors were brought together to talk about
research for first time in Lux; unclear role of
University.
• Difficulty in engaging in visioning dialogue; lack of
forward looking approach; but existence of ‘cham-
pions’ in some workshops.
The ‘clients’ have to be clearly identified and should
have the absorptive capacities needed to
understand the purposes of the exercise and
undertake the exploitation of results.
The right balance in the different actors called to
participate should be achieved as they affect the
independence and legitimacy of the exercise. Their
engagement should be genuine rather than mainly
for legitimacy purposes.
For policy-makers to be engaged and thus ensure
commitment and ownership of results, the rele-
vance of the foresight exercise and potential
benefits to their specific needs and interests have to
be clearly demonstrated from the outset and
retained.
A foresight exercise has more chances to run well if
supported by ‘foresight champions’, i.e. an
implementing institution with a good track record,
credibility and independence and the necessary
expertise and competences (S&T, social skills, etc.)
TRUE. Clear clients BUT divergence of views
undermined absorptive capacities and thus own-
ership and exploitation of results.
TRUE. BUT limited engagement of users thus
relevance of outcome considered low by some.
Need to show relevance.
TRUE. BUT MCHER had doubts re. the degree to
which their needs were being met by FNR
Foresight.
TRUE. Indeed FNR's credibility and
competences was the main driver behind the
smooth completion of the exercise.

Processes/methods in terms of:

• Methods and results of Phase 1 largely questioned by
participants; lack of continuity between two phases
but lessons learnt improved phase 2.
• Use of methods mainly drawing on expertise and
evidence and less on interaction and creativity;
Results dependent in fluidity of participation; more
interactive discussions in Phase 2; little brainstorming
in new areas; no final synthesis of all workshops.
• More interactive discussions in Phase 2: low engage-
ment of users and ‘outside’ experts in some FVAs;
defensive than constructive attitudes; results driven
by dominating individuals; role of participants should
have been better prepared; more workshops and
more work from participants needed.
• Clear implementation of results by FNR; good com-
munication of results and media coverage.
• Transparent, participatory and democratic processes
to the degree possible given the time constraints and
wide range of topics.
There needs to be a combination of methods drawing TRUE. BUT problems in Phase 1 and methods
on expertise, evidence, interaction and creativity mainly drew of expertise and less on
according to the problems, questions and objectives interactions.
and within time/resources limitations.
The credibility and legitimacy of the exercise is TRUE. Indeed open, transparent and inclusive
increased and there is a stronger sense of ownership of processes BUT couldn't overcome defensive
results among participants if processes are open, modes, low participation of users, and not sys-
transparent and inclusive allowing genuine tematic interactions.
consultation, two-way communication and consistent
stakeholder participation.
High political interest/relevance in the issues FALSE. This did not happen in FNR Foresight
examined may undermine openness and although high interest in results.
inclusiveness of processes.
There needs to be a well-mapped route to implemen- TRUE. Indeed this was the case for FNR Foresight.
tation of the results along the way, if not from the
outset, and communication of this route, for the uptake
of results to be facilitated.

Inputs and outputs:

• Validity and reliability of domain rankings challenged
by participants in Phase 1; Phase 2: transparent
prioritisation based on agreed criteria.
• High relevance and timeliness for FNR although
execution of exercise under time pressures.
• Outputs of limited novelty; priorities rather rich and
diverse; not able to identify niche areas; very positive
focus on inter-disciplinarity and innovativeness (es-
pecially FVA for ENV).
The quality and value of inputs and outputs TRUE. BUT results with little novelty because lack
increases if they are marked by creativity and of vision.
inter-disciplinarity in terms of integration of differ-
ent perspectives (societal, scientific, technological,
political, etc.), sectors, and disciplines.
The inputs asked from the different actors have to TRUE. Some users could not see the relevance of
respect their specific interests and concerns to the discussions to their needs and concerns so
facilitate active participation and interest in the were rather discouraged from active participation.
task.

Objectives and background:

• Objectives not clear due to divergent views between
FNR and MCHER and thus not clearly communicated
to participants; confusions and misunderstanding
across the system actors.
The rationale and objectives of the programme as TRUE. Indeed the inability to sustain clear
well as their consequent changes have to be clearly communication about the main aims of the
communicated to the stakeholders involved from exercise caused problems.
the outset to achieve mutual understanding and
management of expectations.

Source: compiled by the author.

main clients from the exercise and the consequent inability to
clearly communicate the objectives of the exercise to the
participants were the major negative factor identified by the
interviewees that affected the course of the exercise.
A related point was also the degree to which the exercise
was supposed to identify new domains or to build on existing
strengths. Tension was created between the need to maintain
thematic ‘variety’ in the research system and the need to
build critical mass in a few selected niches, which was never
satisfactorily resolved [22].
These shortcomings negatively affected the governance of
the exercise to a certain degree. While, the exercise managed
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
Table 8
Assessment of FNR external foresight factors and associated mechanisms of delivery of impacts.
Assessment of external factors of FNR Foresight
Institutional context in terms of:
• Trying to comply with timeframes exercise carried out in
3 phases but was not as designed from the start; thus
some discontinuity between phases; highly compatible
with FNR's strategy and needs.
• Need to clear up role of FNR, to set priorities & directions,
a roadmap for international research development
involving all actors; to establish coordination of policies
and actors and change institutional set-up (lock in FNR–
MCHER-CRPs).
• ‘Many-hats syndrome’ made coordination difficult some-
times as people were pre-occupied with defending their
different positions and were not open to find common
ground for agreement.
Structures/state of innovation system:
• Low understanding of the role of science in society; lack
of strategic orientation and coordination of funding
instruments; bulk of business R&D achieved by 2–3
major companies — large majority not involved in
innovation/R&D thus limited participation in exercise;
financial sector not adequately addressed in exercise;
lack of R&D critical mass in some research fields; but also
young, dynamic system with clear understanding that
things should change.
• Lack of vision of role of science, technology and innovation
in Luxemburg's development; but improves with FNR
activities.
• Clear recognition that changes were needed increased
willingness to actively participate; government was ready
to devote resources.
• Resources availability facilitated execution of the exercise;
took time for people to be convinced that their research
areas were not threatened; difficult to set priorities in
steep development phase.
Socio-cultural context in terms of:
• No tradition of wide public engagement.
• No strong culture of protest manifestation; trust in
institutions, consensus society.
• Perceived utility of foresight by FNR — less for other
institutions; low in general public.
• Tradition of ‘learning from others’, openmindedness
(but shouldn't be overrated as it may hinder original
thinking); lack of creativity culture.
• Language problems: due to different languages all
workshops were carried out in English which de-
terred some from active engagement.
Governance:
• No foresight tradition; high commitment by FNR/MCHER
but misunderstandings regarding objectives and possible
outputs.
• Tradition of consensus building and inclusion of stake-
holders in the planning process (especially private
sector).
• Science more in society now but still a lot to do to
sensitise public regarding science and research.
Source: compiled by the author.
Initial mechanisms of delivery of impacts/hypotheses
If there is fragmentation between the actors in the
policy system, then foresight will face difficulties in
implementation.
The many-hats syndrome, common in small countries
may help but also deter the coordination or alignment of
actors if people do not manage to feel that their interests
are not threatened and overcome their defensive
attitudes to reach jointly agreed decisions.
The less fragmentation and more networking in the
national research and innovation system, the more
likely foresight finds a favourable environment for
implementation.
Foresight can be useful in countries facing major
structural changes and new developments, because it
can help break path dependencies in policy, and
liberate mind-sets.
If institutional structures promote conventional and
risk-averse thinking and exclude new ideas on the
political agenda, they may act as barriers to foresight
‘acceptance’ and implementation.
In lack of a wider vision of role of science, technology
and innovation in the development of the country it is
hard to engage in visioning dialogues for specific
scientific areas.
If there is a tradition in encouraging public
engagement in decision-making, then there is more
trust in institutions and more chances for a foresight
exercise to be well received and implemented.
In high divisive and individualistic societies, the
performance of foresight may be negatively affected
as extra efforts are needed to change mindsets in
favour of consensus and collective endeavours.
In environments with high appreciation of a
consensus view, implementation of foresight may be
eased if consensus building is a goal to be achieved
while non-consensus types of activities may be less
relevant.
Policy-making needs an optimal distance from fore-
sight not to undermine the degree of creativity built
into the processes, or the degree to which controver-
sial issues are dealt with.
In contexts where other strategic intelligence
instruments exist foresight has more chances to
establish the necessary linkages with the
policy-making process but it may also face competi-
tion from the other strategic intelligence activities.
If there are established controversies around the
issues examined, foresight is not likely to resolve
them and thus the course and effectiveness of
foresight activities may be undermined.
Organised interests can be counterbalanced if there is
high level of social debate about the issue(s) at stake
and public involvement.
285
Assessment of mech.
TRUE. Indeed this was the case for FNR
Foresight.
TRUE. Indeed this was the case for FNR
Foresight.
TRUE. This was the case for FNR Foresight
in the opposite sense.
TRUE. Indeed there was clear
understanding of the need for change.
FALSE. This was not a major barrier for FNR
Foresight although Luxemburg a conser-
vative society.
TRUE. Indeed this was the case for FNR
Foresight.
TRUE also due to trans-national learning
although no tradition of public engage-
ment but of key stakeholders (esp.
businesses).
Not relevant for FNR Foresight as Luxem-
burg a consensus society.
TRUE. Indeed this was the case for FNR
Foresight.
Not relevant for FNR Foresight — creativity
was not the aim and issues revealed no
major controversies.
Not relevant for FNR Foresight — no
‘evaluation’ or other strategic intelligence
culture.
FALSE. Foresight contributed to recognition
of basic research and scientific excellence
contrary to prior norms and practices.
Not relevant for FNR Foresight.
286 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291

Table 9
Assessment of FVAs' features vis-a-vis the “ideal FVA” for FNR Foresight.

Assessment vis-a-vis ‘ideal FVA’

FVA human capital

Magnitude Adequate
Diversity Needed better engagement of research users, non-researchers, non-locals
Resilience of membership Needed a core group across different events for continuity and systematic building up of knowledge

FVA social capital

Level of communication Adequate
Level of collaboration Needed trust building discussions prior to workshops

FVA interaction structures

Methods applied Needed more flexible and interactive methods and a Differentiated approach to engage diverse audiences
Workshop organisation Needed structures enabling systematic interactions and knowledge creation Needed structures enabling
systematic cross-disciplinary exchange of knowledge

Source: compiled by the author.

to achieve independence, responsiveness to needs and
flexibility in management, it fell short in overcoming the
misunderstanding between the client/sponsor and foresight
managers and achieving strong and fruitful interactions
within and among the structures created (methodological
team, expert panels).
The methodological weaknesses especially in the first
phase of the exercise were the second major negative factor.
It was generally acknowledged that the methods and
approach applied were not appropriate as they lacked prior
discussion of the context and related structural problems so
as to establish a common understanding of the current
situation before exploring the future. Engagement of partic-
ipants was low and results depended on the fluidity of
participation and the degree to which certain persons
dominated discussions. On a positive tone, however, the
split in phases enabled lessons to be drawn from the first
phase and improvements to be made for the second phase.
The third major drawback was the low levels of engagement
of users, i.e. private sector, public officials from other ministries,
societal organisations, as well as experts from abroad. The
private sector was more represented in certain themes like the
FVA for PSE than others (like SS&H). Public administration was
less represented in any theme. The third sector was not actively
engaged despite the efforts of the organisers to involve them. As
some interviewees noted they could not see the relevance of the
issues examined to their concerns.
Accordingly, the mistakes in the way the organisers
expected the mechanisms of delivery of impacts to work
were of three kinds. First, they expected that they could also
satisfy the needs of the Ministry even if they were different
from those of FNR. Secondly, they thought that people would
immediately collaborate and actively discuss the issues posed
once they were brought together. This was proven wrong due
to the lack of trust enabled for example through prior
collaboration, common understanding about context, agree-
ment on methods and approach of work. The one-off events
did not help much in building trust. Better preparation and
explanation of the role of participants as well as more
collective work by the participants themselves would also
have helped in creating a climate of collaboration. Thirdly,
they expected that once the different stakeholders came
together they would engage in constructive dialogues with
each other. The missing element here was the way results
and issues were presented in discussions. Different audiences
need to identify with different perspectives of the various
issues which cannot be achieved by presenting issues in
similar ways in mixed audiences.
On a positive note, certain mechanisms of delivery of
impacts did work successfully. The course of the exercise was
positively affected by the high relevance and timeliness for
the FNR needs. FNR's commitment was also crucial for the
course of the exercise, which led to the full implementation
of the results. The lessons from Phase 1 resulted in an
improved Phase 2 with transparent, participatory and
democratic processes to the degree possible given the time
constraints and wide range of topics. The good communica-
tion strategy and media coverage helped raise the awareness
of the society not only of the exercise itself but of the role of
science in general. Overall, FNR's reputation and credibility
increased due to the conduct of the exercise.
The above analysis is summarised in Table 7.
6.1.1. Assessment of the external factors and mechanisms of
impact delivery
The assessment of the external factors confirmed the
major role played by the fragmentation of institutions in the
research and innovation system, the lack of strategic
orientation in research and innovation and of coordination
of policies and actors, as well as the low public awareness of
the role of science in society.
In addition, the ‘many-hats syndrome’ was noted as both
a positive and a negative factor. In some cases it helped
coordination as a few people represented several different
institutions, while in others, participants did not manage to
overcome their defensive attitudes.
A positive factor was the clear recognition that changes were
needed in the national innovation system. This realisation
alongside the determination of the government not only to
devote the resources needed but also to increase public research
budgets and the commitment of FNR to the project tracked a
favourable route for the delivery of impacts from the exercise.
The tradition of “learning from abroad” also allowed a
certain degree of open-mindedness and receptivity to new
endeavours. The tradition of consensus building and inclu-
sion of stakeholders in the planning process meant that
people were used to discussing to reach an agreement even
though at a more bilateral style.
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
The above analysis is summarised in Table 8. Certain new
mechanisms of delivery of impacts are also indicated (in
italics).
6.2. Assessment of FVAs
6.2.1. Magnitude and diversity of human capital
The members of the different FVAs came together on a
number of different occasions, i.e. the young researchers'
workshops, the exploratory workshops and the experts'
workshops. Thus, the magnitude of human capital in the
different FVAs consists of the number of different participants
in the total of these workshops per scientific area.
Across the different types of workshops the human capital
was different. The common participants ranged from only 7.55%
in the case of PSE7 to 12.5% in the case of ICT.8 As a result the
share of people entering and leaving the FVA(s) over time, i.e. the
resilience of individual membership, was low.
The size of the FVAs ranged between 60 and 70 with the
exception of PSE which amounted to 49 members. The highest
size appeared for the FVAs of SS&H9 and BLS.10 The exploratory
workshops were larger in size (between 20 and 30). The size
between these workshops and the experts' workshops that came
next remained rather stable. In general, people considered the
size of the groups moderate and manageable for the purposes of
the specific workshops.
Diversity is twofold. It refers to both scientific as well as
sectoral diversity. Regarding the former, interviewees noted that
there was adequate representation of the various sub-domains
in the scientific fields examined in the different FVAs. In terms of
sectoral diversity, priority was given to ensuring adequate
representation of the key actors from both the public and private
sectors. Dominance was noted however of the public research
community and generally of research producers vis-à-vis
research users (private sector, public services, societal groups)
in the discussions. The organisers tried to remedy the situation in
the second phase, which involved more non-researchers (NGOs,
societal organisations), experts from abroad and private sector
participants although still not at a satisfactory degree.
6.2.2. Type of involvement and interactions
The type of involvement and interactions differed across the
different types of workshops. The young researchers' workshops
did not work well. The young researchers not only seemed
unable to project themselves into the future but also felt that the
methods applied were too rigid. The exploratory workshops
presented mixed results with some being more effective
than others [25]. Based on the lessons of Phase 1 the expert
workshops in the second phase provided more flexibility and
space for more open and active discussions. However, it was still
hard to change the mode of the participants in the discussions
from a defensive to a collaborative, constructive mode.
6.2.3. Social capital in FVAs
The point was made for Luxemburg that although all or most
people know each other, they do not collaborate [25]. A reason
7 1. More informed and concerned people
Physical sciences and engineering.
8
Information and communication technologies.
9
Social sciences and humanities.
10
Biomedical/Life sciences.
287
Table 10
Intended foresight impacts of FNR Foresight that were achieved.
1st Foresight function: Informing and directing policy-planning
Improved knowledge base about future:
• Better understanding of SWOT and dynamics of change ***
Framing knowledge into policy support:
• Disclosure of hidden agendas and obstacles **
• (Funding/investment) priority setting ***
• Better informed strategies and policies ***
Capacity building:
• Increased experience in using foresight tools **
• Strategic thinking/vision building capabilities **
2nd Foresight function: Facilitating decision-making and policy
implementation
Foresight culture development:
• Stimulation and inspiration to apply foresight **
• Challenge of mindsets **
Improved decision/policy-making:
• Alignment and commitment to shared visions *
Improved system actors and inter-actor relations:
• Improved communication and collaboration **
• New networks and strengthened communities **
Improved policy systems:
3rd Foresight function: Embedding more stakeholders' views in
policy-making
Strengthened role of society:
• Orientation of innovation to societal needs *
* = achieved to a low degree; ** = moderately achieved; *** = highly achieved
Source: compiled by the author based on interviews with FNR Foresight
organisers and participants.
for this is the diversity and fragmentation of research domains
and another, the competition for public research funds. As a
result, at the start of the exercise, collaboration even among the
members of the same FVA was considered moderate to limited.
Naturally, due to scientific fragmentation collaboration across
the FVAs was also limited. On the other hand, collaboration
between members of one FVA with external peers (people that
did not participate in the foresight exercise either from
Luxemburg or abroad) was valued closer to high.
However, during the foresight exercise, internal collabo-
ration among the members of the same FVA improved
Table 11
Achieved unintended impacts for FNR Foresight.
Achieved unintended impact types
Related to knowledge creation, absorption & diffusion
1. Skills enhancement (multidisciplinary work, social skills, strategic **
thinking, etc.)
2. Capacity building to exchange and create knowledge at individual/org. **
levels
Related to networking and collective learning
1. Increased collaboration beyond the foresight exercise **
2. Expansion/diversification of social capital of individuals or organisations **
Related to enhancing public engagement in decision-making
**
* = achieved to a low degree; ** = moderately achieved; *** = highly achieved
Source: compiled by the author based on interviews with FNR Foresight
organisers and participants.
288 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291

considerably. Some interviewees noted that the exercise types of impacts in terms of relevance for the exercise (i.e.
managed to bring people together working in the same field possible intended impacts) and of importance to have been
and make them collaborate. Referring to collaboration across achieved (i.e. the most important impacts that were actually
different FVAs, the exercise did not succeed in breaking achieved).
disciplinary boundaries. Regarding collaboration with exter- In well-designed exercises the most relevant impacts will
nal peers results were also moderate. inevitably correspond to the most relevant rationales and key
objectives of the exercise. In FNR Foresight, there was indeed
high correspondence between the most relevant rationales
6.2.4. New knowledge created
and the most relevant impacts. For example the most
The knowledge created, albeit small variations across the
relevant and important impacts were considered to be
different FVAs, mainly consisted of SWOT analyses, identification
“(funding/investment) priority setting” and “better informed
of trends and drivers, key technologies and the resulting research
strategies and policies” corresponding to the most relevant
priorities. The formulation of research agendas was also noted by
rationale of “informing funding and investment priorities”.
some interviews (FVAs for ENV11 and SS&H).
However, as an interviewee pointed out FNR Foresight
The knowledge created was examined in terms of relevance,
had the potential for more far reaching impacts. With a more
multi-disciplinarity and novelty. Relevance to the objectives and
broadened scope to influence also the Ministry of Economy
rationale of the exercise seemed lower for participants than for
for example with suggesting new measures to support the
the organisers and clients of the exercise. Multi-disciplinarity
private sector it could have influenced agenda setting in
presented reverse results, i.e. it was the organisers that
other fields such as innovation. However, the exercise was
considered it lower than the participants but neither of them
seen more as serving an internal purpose of FNR.
rated it above moderate. The novelty was considered between
The impacts referring to the second foresight function that
low and moderate by both groups.
were considered most relevant were those under the group
The usefulness of the knowledge produced during the course
“improved system actors and inter-actors relations” and
of the exercise within the different FVAs was considered rather
“foresight culture development”. However, out of the four main
moderate. The use of knowledge across the different FVAs was
elements of the second foresight function (collective learning
particularly limited.
and knowledge creation, aligning and mobilising actors,
The use of the new knowledge in other settings was in turn
strengthening inter-actors' relations, and nurturing a foresight
seen more positively. First of all the Ministry adopted the results.
culture) the two last ones were translated into achieved impacts
This was done after further consultations within the Ministry to
and even those to a moderate extent.
remedy the low participation of public officials in the exercise
Thus, as hinted at the beginning FNR Foresight did fall
and to create a shorter list of more focused priorities. The
short in fully achieving the second foresight function
research priorities were also taken into consideration in the
“facilitating decision-making and policy implementation”. A
strategies of the research centres and may have affected to some
possible explanation may be that the exercise did not manage
degree the orientation areas of the University. However, the use
to make the participants feel safe enough to overcome their
of results in other settings would have been higher had the
defensive attitudes, or to provide them with inputs and
exercise involved more research users which would have
results interesting enough to challenge their mind-sets, take
increased the relevance of the results to their needs.
further and exploit on their own through new networks or
Following the limited use of the knowledge produced in
alignments.
other settings, the generative power of the FVAs and thus
On the positive side, “stimulation and inspiration to apply
their longevity was limited.
foresight” was seen as both relevant and achieved even though
Regarding the other aspects of sustainability, the resil-
mainly for FNR. A concrete impact in this regard was the creation
ience of the individual membership, i.e. the average length of
of a society for prospective studies and evaluation in the financial
time that members participate in the FVA, was also low. Yet
sector after the end of the exercise. Overall, the exercise managed
in some FVAs intentions existed to keep on their collabora-
to make concepts like “prospective studies”, and “foresight”
tion in view of submitting research proposals for funding.
acceptable among the Luxemburgish administration and society
These efforts, however, were more ad hoc and informal
[22].
rather than sustained in a specific form.
Regarding the third foresight function, participants
Based on the above analysis the assessment of the FVAs
appreciated their embedding to policy-making through the
against their “ideal” is shown in Table 9.
foresight exercise more than this was intended by the
organisers. Most of the interviewees pointed out that it was
6.3. Assessment of impacts towards more participatory, ‘knowl- a valuable starting point without which the next steps in
edge societies’ applying such inclusive and participatory techniques in
policy-making could not be taken [22].
6.3.1. Intended impacts The corresponding achieved impacts are shown in
Intended impacts are the impacts that are expected by the Table 10.
organisers and clients of the exercise based on the way it was
designed and the associated mechanisms of delivery of impacts. 6.3.2. Unintended impacts
To distinguish what was expected to be achieved from what was Most of the participants referred to the knowledge-
actually achieved the interviewees were asked to judge certain related impacts, i.e. “skills enhancement” and “capacity
building” to exchange and create knowledge at the individ-
11
Environmental science. ual and organisational levels.
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291 289

Table 12
Alternative factors and mechanisms of delivery of impacts for FNR Foresight and their assessment.

Assessment of factors Alternative mechanisms of delivery of impacts/ Assessment of mechanisms

hypotheses to assess

Factors in relation to knowledge impacts

• Degree of coverage of needs in types of
knowledge and skills needed in general;
IMPORTANT
• Prior availability and access to knowledge
in general;
• Existing capacities to absorb, diffuse and
create knowledge at individual level;
HIGHLY IMPORTANT
• Existing capacities to absorb diffuse and
create knowledge at organisational/collective
levels; HIGHLY IMPORTANT
In a context of an upgraded education system providing
people also with experience of multidisciplinary team
work, as well as social skills, skills to adapt and manage
continuous change, and creativity skills, knowledge
exchange and creation is eased.
If knowledge is made widely accessible, then it is easier
for people to get informed when interested to get engaged
in policy-making through activities like foresight.
Foresight can enhance knowledge exchange and creation
among individuals because it can raise their interest and
build trust among each other through its processes.
The more individuals have similar backgrounds and enjoy
prior collaboration, the more they have appropriate tacit
knowledge for knowledge absorption, but the less ‘thinking
out of the box’ is enabled in knowledge creation.
Organisations need to have appropriate knowledge transfer
structures, routines and networks within and outside the
organisation as well as absorptive capacities, to enable
knowledge creation in exploiting foresight results.
TRUE given that Lux. people have high
quality education abroad.
Not that relevant; results communicated
at end of exercise.
TRUE. Happened to some extent but needs
continuation.
TRUE. Even though novelty of results low,
important that brought together people
with diverse profiles.
TRUE. Research institutions' internal
functioning not proper to exploit new
ideas and areas.

Factors in relation to networking impacts

• Types of networks formed; HIGHLY IMPORTANT
• Prior social capital situation; HIGHLY IMPORTANT
• Organisational relations with environment
• Creativity and innovation generation conditions
The networks formed under foresight exercises can lead to TRUE. Indeed this was the case for FNR
new knowledge creation if they enable knowledge diffusion Foresight even though knowledge created
across boundaries of cognitive objects, disciplines and not entirely new.
backgrounds based on complementarity of competences,
mutual interests and benefits.
The weaker the ties among the network nodes, the longer Not so relevant as radical innovations
the cognitive distances, and the larger the diversity, the not the aim.
more radical innovations can emerge; the stronger the
ties, the closer the links and the less diversity, the more
incremental innovations can emerge.
Participation in foresight can assist an organisation create Not so relevant as creativity not the aim.
intense and dense relations with its external environment,
which in turn can lead to the critical mass required for
knowledge spill-overs and synergies to develop that favour
creativity.
The application of foresight methods drawing on creativity is Not so relevant as creativity not the
more effective in a context of a culture of curiosity and aim and lack of creativity culture.
institutional conditions favouring creativity and risk taking.

Factors in relation to public participation impacts

• Prior extent of public engagement.
• Degree of already informed and concerned public.
• Governance and institutional conditions;
HIGHLY IMPORTANT
• Conditions of actors' alignment/collaboration.
• Degree of urgency in d-making.
The more people are well-informed and concerned, the TRUE.
more they can be actively engaged to the benefit of
foresight exercises and visa versa.
Public engagement in activities such as foresight exercises Not so relevant due to no tradition
flourishes in climates of active public debate where high of public engagement through activism.
levels of trust in institutions may co-exist together with
high levels of activism.
Foresight can lead to coalitions of actors to cope with risks Not so relevant — aim not dealing
because it can help develop a shared sense of risk, mutual with risks
trust and perceived benefits.
In cases of urgent decision-making in coping with risks, Not so relevant — aim not dealing
participatory processes in foresight may be less effective with risks.
or even avoided as ‘counterproductive’.

Source: compiled by the author.

The interviewees agreed that the main factors in enhanc-
ing such skills were the existing capacities for knowledge
absorption and creation at the individual and organisational
levels. In exploring further the factors and mechanisms that
were activated and delivered these impacts, interviewees
acknowledged the value of bringing together people with
diverse backgrounds, even though the novelty of the results
was not marked. Interviewees also agreed that foresight can
enhance knowledge exchange and creation among individ-
uals because it can raise their interest and build trust among
each other through its processes.
There was also an agreement that organisations need to
have structures, routines and networks and absorptive
capacities to enable exploiting foresight results. They even
noted that the deficiencies of the research institutions in this
regard hinder the degree they can exploit new ideas and
290 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
areas. This provided another explanation for the limited use
of results in other settings and consequently a flaw in the
exercise's mechanism of delivery of impacts.
The unintended impacts in relation to networking that
the interviewees recognised were “increased collaboration
beyond the foresight exercise” and “expansion/diversification
of social capital of individuals or organisations”. It was natural for
interviewees to aspire improved collaboration at least under the
new FNR programme which endorsed the new commonly
agreed priorities.
In examining the main factors associated with these impacts,
interviewees noted the importance of complementarity of
competences, mutual interests and benefits in forming networks
as well as of breaking cognitive boundaries for new knowledge
creation. At the same time they stressed that a culture favouring
creativity and innovation was hardly present in Luxemburg. In
this regard, the fact that the methods that FNR Foresight applied
mainly drew on expertise and interactions rather than creativity
might have been a good choice. Yet, it might also have been an
opportunity to deviate from conservatism.
Unintended impacts in relation to public participation
were not expected as the third foresight function was hardly
represented in the exercise's rational and key objectives.
Surprisingly however, there was one impact appreciated by
the interviewees, that people were now more informed and
concerned. An interviewee also added that the role of science
was now more acknowledged in society and journals
dedicated special sections about scientific issues to raise
public awareness of science.
Summing up the alternative factors along with their
assessment are shown in Table 11.
In summary, FNR Foresight mainly contributed to the
development of the knowledge component of more partici-
patory “knowledge societies”. The exercise managed to
achieve intended impacts mainly in relation to the first
foresight function and thus more related to the creation and
diffusion of knowledge.
FNR Foresight achieved impacts related to the second
function to a lesser extent. Even at the stage of creating the
FNR Foresight programme theory a mismatch was noticed
between the organisers and the participants. The organisers
thought that the key objectives in relation to the second
foresight function were relevant to FNR Foresight. Yet, out of
the four main elements of the second foresight function, only
strengthening inter-actors' relations, and nurturing a fore-
sight culture seem to have been translated into achieved
impacts and even those to a moderate extent.
FNR Foresight led to certain unintended impacts mainly in
relation to knowledge creation. However, other impacts ap-
peared even in relation to the second and third functions, though
to a lesser extent. The value of foresight was appreciated in
forming networks. Participants also appreciated their embedding
to policy-making and also noted that the exercise helped people
become more informed and concerned.
7. Conclusions
The present paper shows that it is possible to create a new
foresight evaluation framework to judge the contribution of
foresight towards more participatory “knowledge societies”
beyond the particular objectives set by each individual exercise.
The new framework builds upon the requirements of more
participatory, knowledge-driven societies, meets the special
needs of foresight evaluation and recognises foresight as a
system comprising of elements and interactions affecting and
being affected by both internal and external factors.
The specific research advances foresight evaluation theory
and practices. The new framework gives pertinent attention
to the role played by the internal and external factors
affecting the course and degree of success of the foresight
exercise. Going beyond a mere identification of the main
factors it calls for more in depth examination. This helps on
one hand to better design the foresight exercise in order to
achieve better results and on the other to better manage
expectations in relation to what can actually be achieved. In
terms of the evaluation itself it helps to provide better
explanations on why things occurred as they did.
The suggested assessment framework can be used in
ex-post assessments as well as in ex-ante evaluations for
improving the design of foresight exercises. In this regard, it
provides a solution to the identified weakness that during the
increasingly ‘social’ redefinition of technology foresight, the
design of foresight processes has not kept pace [3].
Foresight evaluations so far are usually driven by
questions such as ‘what was achieved in relation to what
was intended to be achieved?’ The specific framework adds
in two other dimensions. Apart from assessing the appropri-
ateness of the foresight design (i.e. the internal consistency of
the exercise), it also tries to answer the question ‘what was
achieved in relation to what else could/should have been
achieved?’ It clearly shows how unintended impacts can be
identified and assessed by exploiting knowledge gained in
relevant social sciences' literature.
The case study of the evaluation of FNR Foresight showed
that the new framework proved suitable for the systematic
collection of information and the steering of the evaluator in
the right direction in capturing facts and pre-conditions and
elaborating the results. Yet, the new framework also faces
certain limitations. The issue of timing came up in the
evaluation of FNR Foresight. Ideally, the assessment should
run in two phases, once at the start of the foresight exercise
and then after the end of it to enable recording of any
changes in impact assessment indicators. However, it is
problematic to define the right moment for the ex-post
assessment given that the more time from the end of the
project, the more impacts are identifiable but the less they
can be attributed to the features of the specific programmes.
This is a common challenge to impact assessment activities.
Another point to note is that the framework gives emphasis
on qualitative evidence for measuring impacts. Without exclud-
ing their use, the framework does not make a strong contribution
in quantitative methods for foresight evaluation. This is
acknowledged as an area for further research.
Additional areas of further research are also evident. The
framework makes a first attempt in introducing new indicators
(the FVA indicators) in measuring changes in human capital
development, social capital, use of knowledge, and longevity and
sustainability of FVAs. Research should continue on further
elaborating these indicators and possibly combining them with
quantitative ones.
Furthermore, the impact assessment framework was
tested in the context of a national publicly funded foresight
 E. Amanatidou / Technological Forecasting & Social Change 87 (2014) 274–291
exercise in a small country. It would be interesting to
examine the extent to which this approach can be extended
to other types of foresight (corporate, regional or European/
international) as well as to national contexts of different
characteristics, i.e. larger size/economy, leader or follower in
innovation development and so on.
Finally, further research is needed to keep the assessment
framework abreast with evolutions both in relation to foresight
types, and impacts as well as in social science theories in terms
of the factors enhancing knowledge creation, networking and
collective learning and participatory governance. The latter
is especially important for better understanding the complex
systems of modern societies given the co-evolution and
inter-relations of the main elements characterising them.
References
[1] E. Amanatidou, The Greek National Technology Foresight Programme:
success is in the eye of the beholder, Int. J. Foresight Innov. Policy 9 (1)
(2014) 67–92.
[2] E. Amanatidou, K. Guy, Interpreting foresight process impacts: steps
towards the development of a framework conceptualising the
dynamics of ‘foresight systems’? Technol. Forecast. Soc. Change 75
(2008) 539–557.
[3] E. Arnold, S. Faugert, A. Eriksson, V. Charlet, From foresight to
consensus? An evaluation of the second round of Swedish technology
foresight, Teknisk Framsyn, 2002–2004June 2005.
[4] R. Badham, Theories of Industrial Society, Croom Helm, London and
Sydney, 1986.
[5] R. Barré, M. Keenan, Evaluation, impact and learning. Theme 2 —
anchor paper, Second International Seville Seminar on Future-Oriented
Technology Analysis: Impact of FTA Approaches on Policy and
Decision-Making, Seville 28–29 September 20062006.
[6] B. Bozeman, Public Value Mapping of Science Outcomes: Theory and
Method, Centre for Science, Policy, and Outcomes, Washington, DC, 2003.
[7] B. Bozeman, J.D. Rogers, A churn model of scientific knowledge value:
Internet researchers as a knowledge value collective, Res. Policy 31
(2002) 769–794.
[8] C. Cagnin, E. Amanatidou, M. Keenan, Orienting innovation systems
towards grand challenges and the roles that FTA Can play, Anchor
paper for Theme 1 Presented at the Fourth International Seville
Conference on Future-Oriented Technology Analysis (FTA) FTA and
Grand Societal Challenges — Shaping and Driving Structural and
Systemic Transformations, Seville, 12–13 May 2011, 2011.
[9] M. Callon, P. Laredo, V. Rabeharisoa, T. Gonard, T. Leray, The management
and evaluation of technological programs and the dynamics of
techno-economic networks: the case of the AFME, Res. Policy 21 (1992)
215–236.
[10] S. Carvalho, H. White, Theory-based evaluation: the case of social
funds, Am. J. Eval. 25 (2004) 141–160.
[11] J. Cassingena-Harper, G. Pace, Creative processes in policy making: a
case for context in foresight, in: M.S. Dingli (Ed.), Creative Thinking:
Designing Future Possibilities, Malta University Press, 2007.
[12] CEC, European governance. A white paper, COM(2001) 428 Final,
Brussels, 25.7.20012001.
[13] European Commission, EUROPE 2020: a strategy for smart, sustainable
and inclusive growth, COM(2010) 2020 Brussels, 3.3.20102010.
[14] European Foundation for the Improvement of Living and Working
Conditions, Handbook of Knowledge Society Foresight, 2003.
[15] FNR, VDI, Z-Punkt, FNR Foresight Final Report National Priorities for
Public Research and Other Findings, June 2007.
[16] S. Funnell, Developing and using a program theory matrix for program
evaluation and performance monitoring, New Dir. Eval. 87 (Fall 2000)
91–101.
291
[17] In: J.P. Gavigan, F. Scapolo, M. Keenan, I. Miles, F. Farhl, D. Lecoq, M.
Capriati, T. Di Bartolomeo (Eds.), FOREN, A Practical Guide for Regional
Development, FOREN (Foresight for Regional Development Network),
European Commission, DG Research, STRATA Programme, December
2001.
[18] L. Georghiou, M. Keenan, Evaluation of national foresight activities:
assessing rationale, process and impact, Technol. Forecast. Soc. Change
73 (7) (2006) 761–777.
[19] In: L. Georghiou, J. Rigby, H. Cameron (Eds.), Assessing the Socio-economic
Impacts of the Framework Programme (ASIF), 2002.
[20] L. Georghiou, D. Roessner, Evaluating technology programs: tools and
methods, Res. Policy 29 (2000) 657–678.
[21] In: L. Georgiou, J. Cassingena Harper, M. Keenan, I. Miles, R. Popper
(Eds.), The handbook of technology foresight, concepts and practices,
Prime Series on Research and Innovation PolicyEdward Elgar, 2008.
[22] F. Glod, C. Duprel, M. Keenan, Foresight for science and technology
priority setting in a small country: the case of Luxembourg, Technol.
Anal. Strat. Manag. 21 (8) (November 2009) 933–951.
[23] A. Havas, Terminology and Methodology for Benchmarking Foresight
Programmes, Prepared for the ForSociety Project, Tasks 1.4 and 3.1,
Budapest, February 2005.
[24] S. Lash, J. Urry, The End of Organised Capitalism, Polity Press, Basil
Blackwell, 1987.
[25] M. Meyer, The dynamics of science in a small country: the case of
Luxembourg, Sci. Public Policy 35 (5) (June 2008) 361–371.
[26] D. Loveridge, I. Miles, M. Keenan, R. Popper, D. Thomas, I. Ahokas, J.
Kaivo-oja, W.B. Korte, I. Meyer, E. Amanatidou, T. Damvakeraki, F.
Psarra, European knowledge society foresight. The EUFORIA project
synthesis report, Report to the European Foundation for Living
Conditions, Working Conditions and Industrial Relations2004.
[27] R. Popper, How are foresight methods selected, Foresight 10 (6) (2008)
62–89.
[28] J.D. Rogers, Levels of collaboration in scientific research: application
of the knowledge value framework to managing the environment
for research, published as “Theoretical consideration of collabora-
tion in scientific research”, Chapter 6 in: J.S. Hauger, C. McEnaney
(Eds.), Strategies for Competitiveness in Academic Research, AAAS,
2000.
[29] O. Saritas, Systems thinking in foresight. A systems analysis of British,
Irish and Turkish Foresight programmes, Paper Presented at the EU–US
Seminar: New Technology Foresight, Forecasting & Assessment
Methods, Seville 13–14 May 2004, 2004.
[30] J. Tait, R. Williams, ITSAFE Project. Integrating technological and social
aspects of foresight in Europe, Final Report Proposal No Stpa-2001-
00010June 2003.
[31] C. Weiss, Theory-based evaluation: past, present and future, in: D.J.
Rog, D. Fournier (Eds.), Progress and Future Directions in Evaluation:
Perspectives on Theory, Practice and Methods, New Directions for
Evaluation, 76, 1997.
[32] C. Weiss, Which links in which theories shall we evaluate? New Dir.
Eval. 87 (Fall 2000) 35–45.
[33] J.F. Coates, Foresight in federal government policymaking, Futures
Research Quarterly Summer (1985) 29–53.
[34] R.A. Slaughter, The Foresight Principle: Cultural Recovery in the 21st
Century, Adamantine Press Limited, London, 1995.
[35] J.D. Rogers, B. Bozeman, Knowledge value alliances: an alternative to the
R&D project focus in evaluation, Sci. Technol. Hum. Val. 26 (2001) 23–55.
Effie Amanatidou holds an MSc in Technical Change & Industrial Strategy
and a PhD in Foresight Evaluation from the Manchester Institute of
Innovation Research — University of Manchester (UK) where she currently
works as a Research Associate. She has more than 15 years of experience in
research and innovation policy analysis including in particular foresight
studies and evaluation/impact assessment of research policies and
programmes. Research interests include foresight studies, evaluation and
impact assessment of research and innovation programmes and policies,
European research and innovation strategy and policy, coordination of
national/regional research and innovation programmes and joint program-
ming. Mailing Address: Effie Amanatidou, Pavlou Mela 16, P.O. Box: 1698,
Trilofos, P.C. 57500, Trilofos Thessalonikis, Greece.