See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/227285094

Methods for IA: The Challenges and Opportunities Ahead

Article  in  Environmental Modeling & Assessment · August 1998

DOI: 10.1023/A:1019019024003

CITATIONS READS

290 1,556

1 author:

Jan Rotmans
Erasmus University Rotterdam
165 PUBLICATIONS   19,007 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Integrated health modelling View project

PhD research project Beneficial Area Development View project

All content following this page was uploaded by Jan Rotmans on 24 December 2014.

The user has requested enhancement of the downloaded file.

Environmental Modeling and Assessment 3 (1998) 155–179
Methods for IA: The challenges and opportunities ahead
Jan Rotmans
ICIS, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
E-mail: j.rotmans@icis.unimaas.nl
1. Introduction
There is increasing recognition and credibility for the
rapidly evolving field of Integrated Assessment (IA). Within
the setting of the political arena it is accepted that IA can be
supportive in the long-term policy planning process, while
in the scientific arena more and more scientists do realise
the complementary value of IA research. One of the best
indicators for this increased recognition is the establish-
ment of the European Forum on Integrated Environmental
Assessment (EFIEA) by the European Commission DGXII.
In spite of this growing appreciation for IA, the method-
ological basis of IA is still narrow, and lags behind the high
expectations from the outside world. Broadening the basis
of the methodologies underlying IA should therefore be
one of the top priorities of the IA community. This paper
deals with some ideas which could form a basis for an IA
research agenda for the next 5–10 years.
One of the problems of IA is still the many definitions
and interpretations that circulate (Weyant et al. [118], Rot-
mans and Dowlatabadi [100], Parson [85–87], Ravetz [91],
Jaeger et al. [49]). Notwithstanding this diversity, these
definitions have two elements in common, i.e., interdis-
ciplinarity and decision support. These two common el-
ements make Integrated Assessment difficult to plan and
even harder to conduct. Instead of coming up with another
definition of IA, we simply focus on the above commonal-
ities as points of departure for exploring challenges for the
future.
Thus irrespective of whatever definition is taken, IA can
be described as
a structured process of dealing with complex issues, us-
ing knowledge from various scientific disciplines and/or
stakeholders, such that integrated insights are made
available to decision makers.
Integrated Assessment is an iterative, continuing process,
where integrated insights from the scientific and stakeholder
community are communicated to the decision-making com-
munity, and experiences and learning effects from decision
makers form one input for scientific and social assessment.
Although participation of stakeholders is not a necessary
prerequisite, the conviction in the IA community grows
that participation of stakeholders is a vital element in IA.
The engagement of non-scientific knowledge, values and
preferences into the IA process through social discourse
 Baltzer Science Publishers BV
155
will improve the quality of IA by giving access to prac-
tical knowledge and experience, and to a wider range of
perspectives and options.
However, the reality is that there are two groups of IA
researchers. First, the group using analytical methods, such
as models. They combine knowledge elements from var-
ious disciplines in an analytical framework, to assess the
socio-economic and environmental consequences of human
activities that affect the environment. This group is be-
ing pushed into embracing stakeholder participation, but is
struggling with this because traditionally it has never been
a major concern for them. Second, there is a group of
social scientists, arguing that IA itself is a participatory
process. They are pushing the analytical group to make
their tools more relevant to decision makers. On the other
hand, the analytical group is trying to convince the partic-
ipatory group of the value of using analytical, quantitative
tools such as models and scenarios.
The complex, intuitive, and value-laden process of IA
operates at a variety of levels and scales, so researchers
cannot address the process by only one, unique approach.
Multiple diverse approaches are needed, varying from ana-
lytical methods (such as models) to participatory methods
(such as policy exercises). The divergence of methods em-
ployed arises from the uneven state of scientific knowledge
across different problem domains, from the differences in
problem perception, and from the different perspectives of
the scientists and stakeholders involved in the assessment
process.
The many methodological approaches used at this time
to address complex issues from an integrated viewpoint are
all relatively immature. The ultimate selection of a specific
method for performing an Integrated Assessment depends
on the context of the assessment, and the requirements of
the IA clients. The simultaneous use of various methods
can improve the quality and adequacy of the Integrated As-
sessment considerably, enabling analysis and interpretation
of the impacts of a wide range of policy options and strate-
gies.
Integrated Assessment attempts to shed light on com-
plex issues by illuminating different aspects of the issue
under concern: from causes to impacts, and from options to
strategies. IA has been widely applied in the global change
research area. IA emerged as a new field in global change
research because the traditional disciplinary approach to
global change research has been unable to meet two signifi-
156 J. Rotmans / Methods for IA
Figure 1. Relevant research areas.
cant challenges central to understanding global phenomena.
The first challenge is the development of an adequate char-
acterisation of the complex interactions and feedback mech-
anisms among the various facets of global change. Such
feedbacks and interactions are defined away or treated para-
metrically in traditional disciplinary research. The second
challenge is that of providing support for public decision
making. IA offers an opportunity to develop a coherent
framework for testing the effectiveness of various policy
strategies, and estimating trade-offs among different policy
options.
We should be aware that we are not re-inventing the
wheel. Integrated Assessment partly overlaps with exist-
ing research areas, especially technology assessment, risk
analysis and policy analysis (see figure 1). These research
areas also address some kind of complex problem, however,
from a specific point of view. The essential difference is
that IA aims to integrate knowledge from an a-priori inte-
grated point of view. Nevertheless, co-operation with the
research communities on technology assessment, risk analy-
sis and policy analysis should be sought in order to benefit
from gained experiences.
From a practical perspective, interlinked social, eco-
nomic, and environmental conditions demand integrated
policies. Despite an early history of isolated regulatory
initiatives related to for instance air, water, and soil over
the past two decades, environmental policies have become
increasingly integrated. Nevertheless, this is only a first
step towards further integration of environmental, social
and economic policies.
Opponents of Integrated Assessment argue that Inte-
grated Assessment is premature, and can lead to inappro-
priate confidence in questionable or misleading findings.
Proponents of IA, however, argue that the complexity of
the issues demands an integrated approach to ensure that
key interactions, feedbacks and effects are not inadvertently
omitted from the analysis. They argue that the various
pieces of the complex puzzle can no longer be examined in
isolation. IA endeavours to keep track of how the pieces
of the puzzle are fitting together, indicating priorities for
narrower disciplinary research.
Performing IAs has a number of advantages. In general
terms, IA can help to:
• put a complex problem in the broader context of other
problems, by exploring the interrelations of the specific
problem with other issues;
• assess potential response options to complex problems.
This may be, but not necessarily, in the form of cost–
benefit and cost–effectiveness analysis;
• identify, illuminate and clarify the different types and
sources of uncertainties in the cause–effect chain(s) of
a complex problem;
• translate the concept of uncertainty into the concept of
risk analysis, to assist in decision making under uncer-
tainty;
• set priorities for research topics, also by identifying and
prioritising decision-relevant gaps in knowledge.
This paper first discusses a variety of IA methods. For
the most important analytical and participatory methods, the
state-of-the-art knowledge, a typology, strengths and weak-
nesses, and current trends are discussed. This forms the
basis for a discussion of the challenges for the future for
the various IA methods. In addition, over-arching method-
ological challenges are discussed. Then a research agenda
for IA methods is presented. The paper ends with recom-
mendations how these methodological challenges could be
materialised.
2. Methods for Integrated Assessment
In general, two types of Integrated Assessment meth-
ods can be distinguished: analytical methods and partici-
patory methods. While analytical methods are often rooted
in the natural sciences, participatory methods, also labelled
as interactive, deliberative or communicative methods, stem
from the social sciences. The group of analytical methods is
reasonably well-defined and basically includes model analy-
sis, scenario analysis and risk analysis. Their commonality
is that they provide analytical frameworks for represent-
ing and structuring scientific knowledge in an integrated
manner. The group of participatory methods, however, in-
volves a plethora of methods, varying from expert pan-
els, delphi methods, to gaming, policy exercises and focus
groups. They have in common that they aim to involve
non-scientists as stakeholders in the process, where the as-
sessment effort is driven by stakeholder–scientist interac-
tions.
The aim of the various methods is to facilitate the IA
process as sketched in figure 2. It should be noted that this
figure is a highly simplified representation of the complex
IA process. For example, it does not exclude the possibil-
ity that societal actors also supply (delivering non-scientific
expertise), and scientists also demand (e.g., in framing re-
search agendas).
 J. Rotmans / Methods for IA
Figure 2. Sketch of IA process.
Below we discuss concisely the analytical and partici-
patory methods, followed by an exploration of the com-
plementary value of the various approaches by sketching a
tool kit for Integrated Assessment.
2.1. Analytical methods
2.1.1. Models
Integrated Assessment models are computer simulation
(including optimisation) frameworks that try to describe
quantitatively as much as possible of the cause–effect re-
lationships of a specific issue, and of the interlinkages and
interactions among different issues. Current projects in IA
modeling build on a tradition started in the early seventies
by the Club of Rome (Meadows et al. [71,72]). In this
context the first global computer simulation model, i.e., the
World3 model, was developed, which described couplings
between the major social and physical components of the
world system. This model inspired the development of
numerous global models, focusing on resource depletion,
population and pollution.
The next generation of IA models explicitly addressed
environmental issues. The first among these models
emerged in the late 1970s from earlier energy modeling
(Nordhaus [78], Edmonds and Reilly [33]). Meanwhile, the
RAINS model of acidification in Europe was developed in
the 80s (Alcamo et al. [4]).
The phenomenon of global climate change has prompted
the development of a new class of IA models. The first
steps to an integrated model of climate change were
taken by Mintzer [75], Lashof and Tirpack [63], and
Rotmans [97]. Since then approximately 40 IA mod-
els of climate change have been developed (van der
Sluys [106]). Recent overviews of IA modeling activities
in the field of climate change can be found in Rotmans
and Dowlatabadi [100], Schneider [103] and Parson and
Fischer-Vanden [88].
157
Model typology. Weyant et al. [117] classified IA models
in two categories: (i) policy optimisation models, which try
to optimise key policy variables such as carbon emission
control rates and carbon taxes, given certain policy goals
(such as maximising welfare or minimising costs of cli-
mate policy), and (ii) policy evaluation models, which try
to evaluate the environmental, economic and social con-
sequences of specific policy strategies. However, some
IA models characterised as policy evaluation models can
also be used for optimisation experiments. Therefore, an-
other possible classification of IA models is given by Rot-
mans and Dowlatabadi [100]: (i) macroeconomic-oriented
models, which represent relatively simple, parameterised
decision-analytic formulations of complex problems; and
(ii) biosphere-oriented models, which represent a more
comprehensive, process-oriented description of a complex
problem. Most macroeconomic-oriented models are neo-
classical models based on an equilibrium framework, us-
ing traditional economic concepts regarding optimisation
and capital accumulation, largely ignoring environmental
dynamics. Biosphere-oriented models, however, focus on
a systems-based description of the geophysical and bio-
geochemical processes and feedbacks, but do not ade-
quately represent the socio-economic system. The DICE
model (Nordhaus [79,80]) is a well-known exponent of
the macroeconomic-oriented school, whereas the IMAGE
model (Rotmans [97], Alcamo [2]) is representative for the
biosphere-oriented school.
Meanwhile, some attempts are underway to combine the
best of both worlds, yielding a hybrid of the two categories
above. These hybrid models may contain a detailed general
equilibrium model for the economy that interacts with a dy-
namic environment, such as GCAM (Edmonds et al. [32]),
and the MIT model (Prinn et al. [90]). However, they
may also use complexity and uncertainty as guiding mod-
eling principles for both the human and natural system, of
which ICAM (Dowlatabadi and Morgan [25]) and TAR-
GETS (Rotmans and de Vries [99]) are examples. There-
fore, the above two classes should be considered as the
polar ends of a continuum populated by many IA modeling
efforts.
Strengths and weaknesses. Any attempt to fully represent
a complex issue and its numerous interlinkages with other
issues in a quantitative model is doomed to failure. Never-
theless, even a simplified but integrated model can provide
a useful guide to complex issues and complement highly
detailed models that cover only some parts of complex phe-
nomena. Among the major strengths of IA models are:
• exploration of interactions and feedbacks: explicit in-
clusion of interactions and feedback mechanisms be-
tween subsystems can yield insights that disciplinary
studies cannot offer;
• flexible and rapid simulation tools: the simplified na-
ture and flexible structure of IA models permit rapid
prototyping of new concepts and scientific insights;
158 J. Rotmans / Methods for IA
• consistent frameworks to structure scientific knowledge:
critical uncertainties, gaps in scientific knowledge and
weaknesses in discipline-oriented expert models can be
identified;
• tools for communication: IA models can be useful tools
in communicating complex scientific issues to decision
makers, disciplinary scientists, stakeholders, and the
general public.
Obviously, IA models also have limitations and weak-
nesses. Among the most important ones are:
• high level of integration: many processes occur at a mi-
cro level, far below the spatial and temporal aggregation
of current IA models;
• inadequate treatment of uncertainties: IA models are
prone to an accumulation of uncertainties, and to a va-
riety of types and sources of uncertainty;
• absence of stochastic behaviour: most IA models de-
scribe processes in a continuous, deterministic manner,
excluding extreme conditions that may significantly in-
fluence the long-term systems behaviour;
• limited calibration and validation: the high level of ag-
gregation implies an inherent lack of empirical variables
and parameters, and current data sets are often too small
and/or unreliable to apply.
Current trends. The trend in current IA modeling is to
move toward greater and greater disaggregation, assuming
that yields better models. In general, it is difficult to know
when to stop building more detail into an Integrated As-
sessment model. With each incremented level of sophis-
tication from the interlinkages of more processes in more
detail, comes new insight. However, past decades of model
building have shown that small and transparent models are
often superior in that they provide similar results to large
models faster and offer ease of use. In this respect, it
is useful to distinguish between complicated and complex
models. Complicated models are models that include a va-
riety of processes, many of which may be interlinked. If
incremental changes in these processes generally lead to
incremental changes in model output, the dynamics of the
model is almost linear, and not complex at all. Complex
models, however, may contain relatively few processes, but
incremental changes in these processes may result in con-
siderable changes in the results of the overall model. This
non-linear behaviour may be due to the inclusion of feed-
backs, adaptation, self-learning, and chaotic behaviour, and
is often unpredictable.
Practically speaking, this means that disaggregation of
IA models has profound consequences for the dynamics
of the model. Breaking down a global model into vari-
ous regions, implies that the regional dynamics should be
dealt with in an adequate manner. Regional IA models use
grid cells or classes for representing geographical differ-
ences and heterogeneities in regional IA models. They do
not capture, however, the regional dynamics with regard
to population growth, economic development, resource use
and environmental degradation, let alone regional interac-
tions through migration and trade.
The trend towards greater and more complicated IA
models makes heavy demands on the documentation on the
one hand, and the scientific testing procedure on the other.
A potential weakness of complicated models is that they are
often poorly documented, which makes them intangible for
other modelers, but, more importantly, incomprehensible
for a wider audience. The scientific testing of complicated
IA models should be a major part of the model building
cycle. The more complicated the model, the higher the
possibility of errors and bugs. It requires thorough testing
to pick up most if not all errors and bugs, an activity which
is, unfortunately, heavily underrated.
2.1.2. Scenarios
Thinking about the future is often done by the devel-
opment of scenarios. Theoretically speaking, scenarios are
hypothetical sequences of events, constructed for the pur-
pose of focusing attention on causal processes and deci-
sion points (Kahn and Wiener [57]). In practice, scenarios
are archetypal descriptions of alternative images of the fu-
ture, created from mental maps or models that reflect differ-
ent perspectives on past, present and future developments.
Ideally, they should be internally consistent, plausible and
recognisable stories exploring a path into the future (see
also Anastasi [5]). Although many scenario definitions ex-
ist, most of them share the following characteristics:
• scenarios are hypothetical, describing possible future
pathways;
• scenarios describe processes, representing sequences of
events over a period of time;
• scenarios consist of states, events, actions and conse-
quences which are causally related;
• scenarios start from an initial state (usually the present),
depicting a final state at a fixed time horizon.
Scenarios are perhaps most effective when seen as a
powerful tool to broaden perspectives, raise questions, and
challenge conventional thinking. However, that is not the
way they have been used. Most scenarios developed in the
past were dominated by engineers, economists, and plan-
ners, while contributions from the social sciences have been
rare (Jungermann [55]). That could possibly be an expla-
nation for the rather dogmatic and predictive character of
many scenario exercises in the past. Looking back at de-
velopments of the past 25 years, however, one clear lesson
can be learned from scenario-based assessments made in
the 1970s: dogmatic predictions regarding the Earth’s fu-
ture are unreliable and can be politically counterproductive
(UN [112]).
Nowadays, it is generally accepted that scenarios do not
predict, but that they paint pictures of possible futures and
explore the different outcomes that might result if basic
assumptions are changed, for example, regarding policy in-
terventions. So the only relevant question that scenarios can
 J. Rotmans / Methods for IA
address is not whether an event will happen in the future,
but what we could do if it did happen. Also, the crucial role
of uncertainty is increasingly recognised, which has led to
the conclusion that scenario building should not be a purely
deterministic scientific activity. This increased understand-
ing has resulted in a number of well-thought-out scenario
exercises during the last decade that have proven their value
in a particular context. For example, the IPCC has devel-
oped two sets of scenarios, which illustrate the impacts of
specific developments in population growth, energy use and
technology in terms of resulting emissions and atmospheric
concentrations of greenhouse gases and its associated pat-
terns of climate change (IPCC [47,48]). The World En-
ergy Council (WEC [116]) developed global scenarios that
underlined the importance of diversity of supply, assuming
shifts in the energy system due to autonomous technological
developments and market forces, and not through govern-
ment interference. A widely used scenario developed by the
Global Scenario Group (Gallopin et al. [40]) was the Con-
ventional Development Scenario. This scenario assumes
demographic, economic and energy assumptions based on
mid-range IPCC choices, supplemented with assumptions
for water and land use and food intake. The Global Sce-
nario Group has recently produced three classes of global
scenarios, based on alternative social visions: Conventional
Worlds, Barbarisation, and Great Transitions, described in
terms of economic scale, equity, environmental quality,
technological change, and degree of social and geopolitical
conflict. Shell (Kassler [61]) recently produced a number
of world scenarios, where three alternative societal visions
have been formulated and weighed against their economic
robustness.
In the field of sustainable development, one of the more
comprehensive scenario studies has been undertaken by
Robinson [93], sketching a desirable, sustainable future for
Canada in 2030, and the changes that would need to oc-
cur till 2030 to arrive at this future vision. Also at the
European scale a number of scenario studies have been
performed: EC-DGXII [29], in the field of energy by the
European Commission DGXVII [30], and on political and
institutional developments by Smith [107].
Scenario classification. In the scenario literature different
subdivisions of scenarios exist (van Asselt et al. [8]). First,
a subdivision can be made between forecasting and back-
casting scenarios, or exploratory versus anticipatory sce-
narios. Forecasting or exploratory scenarios are forward-
directed, i.e., they explore future consequences of a se-
quence of assumptions, with or without expected/desired
policy efforts. The majority of recent scenario studies can
be characterised as forecasting exercises. Backcasting or
anticipatory scenarios on the other hand, are backward di-
rected, i.e., they start from some assumed final state, and
explore the preconditions that could lead to this state, in-
cluding a palette of strategies to reach this situation.
Secondly, we can distinguish between descriptive and
normative scenarios. Descriptive scenarios state an ordered
159
set of possible events irrespective of their (un)desirability,
whereas normative scenarios take values and interests into
account. Current scenario studies mostly have a descriptive
character or are implicitly normative.
Finally, trend and peripheral scenarios can be set out
against each other. A trend scenario represents the ex-
trapolation of past and current trends, while a peripheral
scenario includes trend-breaking and/or surprising devel-
opments. Many recent scenario studies are hybrid, in the
sense that they are neither trend nor peripheral scenarios.
This has to do with the fact that the anticipated changes in
these scenarios are merely incremental.
Although in many long-term scenario studies surprises
and bifurcations are not taken into account, history arguably
shows that trends are characterised by strong fluctuations
rather than smooth curves, often triggered by unexpected
changes. Surprises can be categorised in the following way:
• unimaginable surprises (like a journey to the Earth’s cen-
tre in the times of Jules Verne);
• imaginable surprises that are improbable (like a global
nuclear war);
• imaginable surprises that are probable (like an oil price
shock, or massive migration due to ecological disasters);
• certain surprises or events (like earthquakes).
Scenario method classification. Scenarios can be classi-
fied along the lines of: (i) the type of tool or method used
for designing the scenario, and (ii) the functionality and fo-
cus of the scenario exercise. Regarding the first group, we
distinguish between quantitative scenarios based on the use
of models, qualitative scenarios based on narratives, and
hybrid scenarios based on participatory methods. Model-
ing methods involve the use of computer models, either as a
central means to explore the future consequences of sets of
assumptions or as a tool to check the consistency of the de-
veloped scenarios. Narrative scenarios are usually deployed
in cases where data is weak or missing. Pathways into the
future are then qualitatively described. These two methods
are usually adopted in scenario projects involving a small
“core” group of experts. In contrast, participatory methods
refer to approaches in which stakeholders (non-scientists),
such as decision makers, business people, and lay people
play an active role: the participants are all co-designers of
the scenarios. Anastasi [5] advocates the use of partici-
patory approaches with the argument that complementary
heterogeneity in perspectives, expertise and knowledge is
needed to guarantee sufficient richness. He uses the focus
and functionality of scenarios as distinguishing character-
istics for classifying scenarios: global scenarios, focused
scenarios and single issue scenarios. Global methods result
in scenarios that depict the future of the world as a whole,
focused methods concentrate on a particular geographical
area, and single-issue methods focus on a particular issue
like energy or transport. Anastasi [5] explored the differ-
ences between the various approaches and their use in sce-
nario building. He concluded that up till now model-based
160 J. Rotmans / Methods for IA
Table 1
Scenario methods.
Modeling
Global Rotmans and de Vries [99]
WEC/IIASA [116]
Focused BCI [10]
Single issue ECN [31]
IPCC [47,48]
scenario studies, as well as global narratives, and energy
and technology-oriented scenarios enjoyed most exposure
in science and decision-making communities.
Table 1 provides some examples of scenario studies as-
sociated with the various approaches. For a detailed de-
scription of the scenario studies listed in this table, the
reader is referred to van Asselt et al. [8].
Strengths and weaknesses. Scenarios can be useful tools
to:
• articulate our key considerations and assumptions: sce-
narios can help to imagine a range of possible futures if
we follow a key set of assumptions and considerations;
• blend quantitative and qualitative knowledge: scenar-
ios are powerful frameworks for using both data and
model-produced output in combination with qualitative
knowledge elements;
• identify constraints and dilemmas: exploring the future
often yields indications for constraints in future devel-
opments and dilemmas for strategic choices to be made;
• expand our thinking beyond the conventional paradigm:
exploring future possibilities that go beyond our conven-
tional thinking may result in surprising and innovative
insights.
Current scenarios are also characterised by a number of
deficiencies:
• lack of diversity: scenarios are often developed from
a narrow, disciplinary based perspective, resulting in a
limited set of standard economic, technological, and to
a lesser extent environmental assumptions;
• extrapolations of current trends: many scenarios do
have a “business-as-usual” character, assuming that cur-
rent conditions will continue for decades, excluding sur-
prises;
• inconsistent: sets of assumptions made for different sec-
tors, regions, or issues, are often not consistent with each
other;
• not transparent: key assumptions and underlying im-
plicit judgements and preferences are often not made
explicit. Also implicit is which factors are exogenous
and which ones are endogenous, and to what extent soci-
etal processes are autonomous or influenced by concrete
policies.
Narrative Participatory
UN-DPCSD [112] no examples
VROM [115] TCPA/CER [110]
EC-DGXI [29]
Button [14] EC-DGXVII [30]
Current trends. There is an increasing tendency towards
the use of participatory methods for designing scenarios.
Another trend is to develop scenarios with both a strong
narrative and a quantitative component. These trends can
complement and reinforce each other: a group of stakehold-
ers with different knowledge, experiences and perspectives
provides a greater richness to scenarios, while appealing
narratives underpinned by numbers are able to engage key
stakeholders. One of the major lessons drawn from the last
decades is that active dissemination is needed to ensure an
adequate usage of scenarios by the stakeholders involved.
In order to fulfil their communicative and decision-
supportive role, scenarios need to be coherent and consis-
tent. The aspect of coherency refers to the inclusion of all
relevant dimensions and all relevant interlinkages between
the various processes considered. Coherent scenarios, how-
ever, should not be too complicated, to avoid an overload
of information communicated to decision makers. Consis-
tency implies that key assumptions done in the scenarios are
checked among different scales, sectors and issues. For ex-
ample, assumptions with regard to future energy use should
be checked with assumptions regarding future land and wa-
ter use.
Unfortunately, many scenario studies do lack such a con-
sistency check and are far from coherent, which makes them
highly vulnerable in the assessment process. This also indi-
cates the need for IA scenarios, i.e., scenarios that condense
critical developments into a consistent set of assumptions
along different temporal and spatial scales and different
aggregation levels, which ultimately results into coherent
narratives useful for decision making.
2.1.3. Risk analysis1
Risk can be defined as the possibility that certain losses
or damages occur as the result of a particular event or series
of events. It usually refers to the possibility of something
going wrong. The notion of “risk” traces back to the 13th
century (Jungermann [56]). It circulated in Italy in the
context of insurance against the loss of shiploads coming
from the Orient through, for example, storms or piracy. The
notion of “risk” is still used in the context of insurance, but
apart from that it has become a widely applied, often used
and ambiguous notion. It is used in the field of business,
technology, health, politics, and sport.
Apart from this common use, risk is also used as a con-
cept in scientific analysis for decision support. Since the
1 This section builds upon the ongoing PhD research of van Asselt.
 J. Rotmans / Methods for IA
early 1970s, risk analysis has become an established field
of research. It was originally rooted in the engineering
and decisions sciences. Since the 80s, risk analysis has
gained importance as a concept in assessment activities, es-
pecially in technology assessment (O’Riordan [82]). In the
last decade, risk analysis is increasingly used in environ-
mental science, and applied to environmental hazards, like
stratospheric ozone depletion and climate change. Since
the beginning of the 90s, risk analysis is gaining a broader
meaning as confined to general aspects of decision making
in modern society (Kämper [58]).
Risk analysis has now become an active field involving
disciplines as diverse as mathematics, statistics, epidemiol-
ogy, toxicology, biology, engineering, system analysis, law,
psychology, decision sciences, sociology, anthropology, ge-
ography, history, economics, management science, policy
sciences and philosophy.
The concept of “risk” is closely connected with the
prevalence of uncertainty. Risk deals with uncertain situa-
tions in which a number of possible outcomes might occur,
of which at least one, but possibly more, is undesirable
(Merkhofer [74]). Personal, collective, societal and envi-
ronmental risks thus arise from the uncertainty surrounding
the occurrence of particular events or series of events. Al-
though uncertainty is central to risk analysis, the paradox is
that risk analysis is also complicated by the omnipresence
of uncertainty. Scientific uncertainties surrounding com-
plex societal phenomena make it very difficult to assess
what is likely to happen if a particular event or a concur-
rence of circumstances comes about. Furthermore, social
and behavioural research on risk perceptions indicates that
in case the underlying mechanisms and processes are not
well understood, lay estimates concerning involuntary risks
tend to be higher (Covello [19]). So there is not only the
problem that uncertainty violates the exactness of risk esti-
mates, but also that the level of uncertainty co-determines
the risk itself.
The above reveals that uncertainty and risk are closely
interlinked: the one cannot be adequately studied without
considering the other, and vice versa. The challenge is to
address both in an integrative manner.
Strengths and weaknesses. Notwithstanding the involve-
ment of so many disciplines, we can doubt whether the
risk community is in fact interdisciplinary. There is a clear
tendency to treat the issue of risk from the perspective of
the own discipline and speciality (Conrad [16]). It is there-
fore perhaps not surprising that there is no common cog-
nitive action-guiding paradigm that integrates the different
aspects of risk. In contrast, the risk community suffers from
fundamental controversy.
The dilemma the risk community faces is that uncer-
tainty interferes with facts and that conflicts are inherent
to risk issues. The risk debate is rooted in the failure to
reconcile the limitations of scientific analysis, the special
characteristics of the human mind, and social and cultural
factors into a common cognitive framework that can guide
161
integrated risk analysis. Out of the scholarly literature, two
key issues emerge that underlie the controversy in the risk
community, i.e.:
• How to define risk?
• How risky are modern hazards?
With regard to both questions, roughly two extreme posi-
tions can be identified. In the discussions pertaining to
defining risk, the objectivists (e.g., Starr [108], Crouch
and Wilson [21]) and constructivists can be distinguished.2
Objectivists distinguish between objective and perceived
risks. They consider perceived risks to be inherently
wrong and have it that objective risks can be adequately
measured in terms of probability and utility. Construc-
tivists deny the possibility of pure scientific, objective es-
timates (see, for example, Douglas and Wildavsky [23],
Otway and von Winterfeldt [83,84], Fischhoff et al. [34],
Slovic et al. [105], Johnson and Covello [54], Morgan
and Henrion [76], Mayo and Hollander [69], Funtow-
icz and Ravetz [36,37], Krimsky and Golding [62], and
Adams [1]). Concerning the second issue, we find scholars
(e.g., Beck [11] and later publications) holding the position
that hazards have increased and are increasing versus those
(e.g., Whipple [119] and Douglas and Wildavsky [23])
who argue that modern society is only more sensitive to
risk.
Notwithstanding this weakness in terms of lack of a com-
mon interdisciplinary framework, the risk community pro-
vides concepts, tools, and methods that can be of help in
the effort to conduct integrated assessments of complex is-
sues. Risk is a concept that is deployed in everyday life.
So it is a concept familiar to decision makers, stakeholders
and the broad public. As argued by Luhmann [65], risk is
the general form in which society describes and assesses
its future. In other words, two-way communication of fu-
ture outlooks in terms of risk is therefore likely to enhance
understanding and co-operation between science and soci-
ety.
Three decades of risk analysis furthermore provide expe-
riences that are relevant to Integrated Assessment. Lessons
can be learned with regard to the following issues:
• the role of uncertainty and subjectivity in science for
policy;
• the use of probabilistic analysis;
• risk communication;
• participation of non-experts in assessment efforts.
In addition, research on risk perception and communica-
tion yields insights about what society wants to know about
complex issues. Awareness of what is societally relevant
may improve IA efforts in terms of usefulness for societal
decision-making processes.
2 Terminology as used by Linnerooth-Bayer and Thompson [64]. This
debate is also described in terms of the quantitative versus the qualitative
approach (Jasanoff [53]).
162 J. Rotmans / Methods for IA
Current trends. The risk debate has severe implications
for risk analysis activities. Over the last 25 years, both
schools had fervent advocates and fierce proponents in and
outside the risk community, with the result that there is
neither agreement over appropriate methods for risk analy-
sis nor acceptance of the outcomes of public processes.
Till the 90s the “objectivist” perspective has dominated risk
analysis. The objectivist view has come under increasing
attack and its supremacy is weakening. The majority of
risk analysts now seems to agree on the following premises
(Jasanoff [53]):
• experts perceive risk differently from lay people;
• risk analysis is not a purely objective process: facts and
values merge frequently in dealing with issues of high
uncertainty;
• cultural factors affect the way people assess risk.
This shift, however, does not imply that the construc-
tivists are close to “take-over”. Criticism on the construc-
tivists school is that it notes that different parties hold dif-
ferent risk perceptions, without making clear what to do
about this (Vlek and Stallen) [113]. They do not provide
an operational alternative for performing risk analysis.
The state-of-the-art in the risk community is thus char-
acterised by two trends, namely: (i) the objectivists’ par-
adigm is dominating actual risk analysis, and (ii) the con-
structivists win ground in epistemological matters. With
regard to an integrated approach towards risk the follow-
ing trend is observed: individual researchers signalise the
necessity for a synthesis approach to risk analysis (e.g.,
Funtowicz and Ravetz [35], Jasanoff [53]), start building
relevant networks and search for research commitments and
funds (Linnerooth-Bayer and Thompson [64]).
2.2. Participatory methods for IA
“Participatory methods” is an umbrella term describing
approaches for assessment in which non-scientists, such as
policy people, stakeholders or even lay people, play an
active role. Brewer [13] first referred to the idea of using
participatory approaches in assessment efforts, however, IA
projects only recently seek to apply such methods for IA
purposes.
Participatory methods are not new. Focus groups, for
example, have been applied in market research since the
seventies (Cox et al. [20]), while consensus conferences
have been applied in technology assessment (for example,
on the issue of predictive genetic research (Platform for
Science and Ethics [89])). Gaming approaches have been
prominently in use for training purposes (e.g., military and
business games (Hausrath [44])). Finally, traditional con-
tract research is a participatory process between clients and
researchers. For instance, in developing a model, clients
may play an active role in: defining the model objectives
and model output, making decisions about model features
involving various trade-offs, and choosing the method of in-
teraction between the model and user (MacCallaway [66]).
2.2.1. Participatory methods for IA
Participatory methods for IA differ with regard to the
profile of the participants, the goal of participation and the
degree of participation. We distinguish three classes of
methods (Rotmans and van Asselt [101]):
• dialogue methods;
• policy exercises;
• mutual learning methods.
The dialogue method is applied in cases where the in-
tended users are considered as a source of information nec-
essary for the analysts to perform the assessment. The goal
of participation is to extract this essential information from
the intended users. Within this class, we can discriminate
between continuous dialogues, and cases in which the di-
alogue takes place in a specific phase of the assessment.
In the first case, the role of the intended users can be de-
scribed as that of co-designer. In case the dialogue takes
place in the design phase, the role of the intended users
is to contribute to the design principles by sketching their
wants and needs. This type of dialogue can be described as
a user-platform. If the dialogue is part of the final stage, the
participants can be described as a test-group that helps to
determine how the results of the assessment will be brought
to the fore. Van Asselt [6] sketches an example in which
a “dialogue”3 was intended to serve as a user-platform. In
adaptive ecological modeling (e.g., Holling [45], Costanza
and Ruth [17]), the explicit aim is that the crucial design
choices are made by the user community. The Delft process
around the IMAGE model (van Daalen et al. [22]) can be
considered as an example of a test group. In the RAINS
project the epistemic community involving scientists and
policy makers realised that the involved stakeholders got
the role of co-designer (Hordijk [46]).
Policy exercises build upon the tradition of simulation
games. A policy exercise can be described as a flexibly
structured process designed as an interface between sci-
entists and policy makers (Toth [111]), in which a com-
plex system is represented by a simpler one with relevant
behavioural similarity, and of which decision making is
part through human participants (Parson [85]). In gen-
eral, a game is set up that represents a negotiation process
in which the different teams are responsible for a certain
country or region. Policy exercises quite often make use
of computer support. A computer model can be used as
a consulting device or as tool to convert the negotiated
agreements into a new “state of the world”.
Policy exercises differ from the other participatory ap-
proaches in the following ways: (i) the participants play
roles, and (ii) the participants do not explicitly take part
in the assessment process. A policy exercise is a way to
get information on human behaviour and policy preferences
necessary for the assessment the analysts produce. Exam-
3 Analysts and intended users in the policy community did not meet di-
rectly, but got informed about each others’ perspectives and attitudes via
interviews.
 J. Rotmans / Methods for IA
ples of policy exercises in IA are the climate policy ex-
ercises (Parson [85–87]) and RIVM’s SusClime exercises
(de Vries [114]). Both efforts made use of IA models,
respectively, a “tailor-made” version of the GCAM model
(Edmonds et al. [32]) and a preliminary version of the TAR-
GETS model (Rotmans and de Vries [99]). Experiences
with policy exercises thus far are discussed in Parson [85].
The principle behind mutual learning approaches is that
participation of stakeholders and citizens enriches the as-
sessment by a multiplicity of perspectives, skills and com-
petence. The participants are considered as co-producers
of knowledge. They are not selected dependent on whether
the assessment is of use to them, but whether their per-
spectives, skills or competence complement the scientists’
expertise and knowledge.
We distinguish two forms of mutual learning:
• the focus group approach, in which scientists play the
role of facilitator and observer, and
• the interactive approach, in which scientists are actively
involved as participants.
In the first case, special groups are composed, typically
consisting of citizens, policy makers or other stakeholders
who are provided with scientific input. The assessments
of these groups are then used in a broader IA process.
In interactive analysis (Guba and Lincoln [43], Grin and
Hoppe [42]), problem definitions and proposed solutions
are integrated into a joint assessment that makes sense to
both stakeholders and scientists. The ULYSSES project
involves IA focus groups with lay people (Dürrenberger
et al. [28], Kasemir et al. [59]). The VISIONS project
(Rotmans [98]), starting in 1998, can be considered as a
first example of the interactive approach.
2.2.2. Strengths and weaknesses
Participatory methods are able to integrate quantitative
with qualitative knowledge (Parson [85]). IA models can-
not (yet) grasp qualitative knowledge. In this way, partici-
patory methods can complement IA models and scenarios.
It is furthermore argued that involvement of non-sci-
entists is needed to ensure the relevance and later accep-
tance of the analytical (modeling) approach (Hordijk [46],
van Asselt [6], Costanza and Ruth [17], Alcamo et al. [3]).
Participatory methods are needed to set up such a dialogue
between analysts and societal actors. A widely used ar-
gument in favour of using participatory methods is that it
helps structure the IA process, and thus also the analytical
methods. After a period of building up thrust and support,
they will become supportive to those methods.
Participatory methods are also justified on arguments in-
spired by the nature of democracy (e.g., Kasemir et al. [59]).
Policies for managing sustainability will be effective only
if they have the moral support of a great mass of people
(Ravetz [91]). It is therefore argued that assessments should
comprise the opinions and attitudes of stakeholders and cit-
izens. Are the strategies proposed by the assessment study
163
considered to be desirable in the eyes of stakeholders and
citizens?
The major weakness pertaining to participatory methods
is that the use of it in Integrated Assessment is in its in-
fancy, with the result that there are not (yet) established
procedures and work-packages for setting up participatory
processes.
2.2.3. Current trends
IA’s complexity, multi-disciplinary character, and wide
variety of societal actors present special challenges to the
use of participatory methods. The use of such methods
in IA, therefore, requires substantial adjustment of existing
procedures and the development of new tools and methods.
Projects like the ULYSSES project (Jaeger et al. [50]), the
IIASA policy exercise effort (Parson [85–87], Parson and
Fisher-Vanden [88]) and the Mackenzie Basin Impact Study
(Cohen [15]) currently explore the necessary features of
participatory methods for IA.
In addition, there is an increasing interest in participa-
tory methods, also in the modeling community. See, for
example, Steve Schneider’s plea (Schneider [103]) for the
involvement of decision makers and stakeholders in IA ef-
forts.
2.3. Towards an IA tool kit
Models, scenarios, risk analysis, and participatory meth-
ods are by no means mutually exclusive methods for IA.
In general, they can complement and reinforce each other
in different ways. This yields a tool kit for IA as sketched
in figure 3. Models, for instance, can be used as a tool
for developing scenarios. On the other hand, models and
scenarios could be the result of participatory processes. Di-
alogue approaches can be oriented towards the development
and/or usage of IA models. Within policy exercises models
are often used as reference frameworks. In the ULYSSES
project (Jaeger et al. [50]), IA models are used in focus
groups as expert input into group discussions. In mutual
learning approaches, models might be helpful to compare
scenarios developed by participants with scientific insights.
Figure 3. IA tool kit.
164 J. Rotmans / Methods for IA
Figure 4. Methods cycle (source: Rotmans and van Asselt [101]).
A major methodological challenge for IA is to take ad-
vantage of the existing multitude of methods. Figure 4
suggests how the IA models, scenarios and participatory
methods could be used in a complementary manner.
The cyclic scheme can serve as a heuristic device to de-
sign a variety of sensible procedures for IA. Starting with
participatory methods and then turning clockwise, sketches
a procedure in which participatory methods provide the ba-
sis for the development of an IA model that, in turn, is used
to develop scenarios. These scenarios are then input to the
next round of participatory meetings, where the developed
models are adjusted, etc. Starting with IA models and turn-
ing anti-clockwise, results in the following iterative process:
an IA model is developed as a multidisciplinary effort, and
is then used as scientific input to participatory processes
in which a variety of scenarios is developed. The consis-
tency and plausibility of these scenarios are then checked
using IA models. These scenarios in turn serve as input for
the participatory process, etc. For example, the VISIONS
project (Rotmans [98]) envisages an anti-clockwise com-
bined usage of scenarios, participatory approaches and IA
models, starting with the design of scenarios.
The practice, however, differs from this ideal picture.
None of the current generation of IA models of climate
change is being developed in close conjunction with deci-
sion makers. This explains why many of the current IA
models are only of limited use to decision makers. For ex-
ample, single-region economic optimisation models offer
the comparative efficiencies of various strategies, but offer
little insight of political value because they do not deal with
the distributional aspects of climate policies. This means
that this type of IA model serves only one purpose, i.e., cal-
culating optimal control strategies under uncertainty, which
is more difficult in multi-region, multi-agent models. This
also illustrates that IA models should function within the
setting of the political arena, where efficiency often takes a
back seat to equity. In such a setting, different groups ex-
hibit different levels of risk perception, they adopt different
decision rules, and their utility bundle is composed of dif-
ferent indicators. Only a few of the current IA models are
flexible enough to represent the interests of various parties
(through space and time).
Having decision makers as early stakeholders in an IA
research program is desirable, but may lead to irresistible
pressure for research toward unattainable goals or may mire
development efforts in the struggle among conflicting value
systems. For example, decision makers want to have geo-
graphically explicit impacts, which has led many research
groups to eschew accurate characterisation of what may be
projected in favour of a precise characterisation of impacts
with a much higher probability of error. Under these cir-
cumstances, the modelers need to communicate to decision
makers the difficulty of attaining such lofty goals. Failing
that, they should make clear the magnitude of uncertainties
in geographically explicit projections.
The next generation of IA models should make use of
enhanced IA concepts, as a result of a continuous consul-
tative process and interfacing with representative groups
of decision makers. This means, among other things, de-
termining the appropriate level of aggregation, making the
models actor-oriented by introducing behavioural rules and
decisions, deriving consistent and integrated scenarios, and
including explicit treatment of uncertainties. The interac-
tion process among model development groups and deci-
sion makers should curb the demand for the unattainable
and prepare decision makers for the difficulties they face in
setting strategies when large uncertainties abound.
3. Methodological challenges
As the Integrated Assessment process develops, more
sophisticated methods and tools are needed. Therefore, we
will face challenges in developing a balanced tool kit in
order to achieve an adequate knowledge base. In this sec-
tion we will discuss some of the methodological challenges,
which largely follow from the strength/weakness analysis
of the IA methods as presented in the former section. The
challenges are subdivided into over-arching methodological
focal points and method-specific challenges.
3.1. Over-arching methodological challenges
Over-arching methodological challenges are those that
are relevant to all methods discussed before. We identify
the following three key issues:
• Aggregation versus disaggregation;
• Treatment of uncertainty;
• Blending qualitative and quantitative knowledge.
In the following, these methodological challenges are
discussed in more detail.
3.1.1. Aggregation versus disaggregation
One of the most critical issues in Integrated Assessment
is that of aggregation versus disaggregation. The level of
aggregation of an IA refers to the spatial and temporal res-
olution and the level of complexity used in the assessment.
The problem of IA is that it has to cope with a variety
of processes that operate on different temporal and spatial
levels, and differ in complexity.
 J. Rotmans / Methods for IA
In general, there is no optimal spatial and temporal scale
at which IA has to operate: that strongly depends on the
nature and scope of the problem to be tackled by IA.
First, IA has to connect disciplinary processes that differ
by nature: physical processes, monetary processes, infor-
mation processes, and policy processes. Because of the
multitude of disciplinary processes to be combined, an as
simple as possible representation of disciplinary knowledge
is preferable. There is, however, no unifying theory for IA
how to do this. In addition, the processes to be linked are
usually studied in isolation from other disciplinary fields.
This isolation is needed as part and parcel of the classic
model of scientific progress and discovery. However, when
the constraints of isolation are removed, there is a variety
of ways in which to connect the reduced pieces of disci-
plinary knowledge. This manifold of possible integration
routes, for which there is again no unifying theory, is one
of the reasons why quality control is so difficult to achieve
in IA.
For instance, in order to link the reduced pieces of dis-
ciplinary knowledge in a systemic way, one can use ele-
ments from classical systems analysis, or the method of
system dynamics, or a sequential input–output analysis, or
a correlation-based approach, or a pressure–state–impact–
response approach.
Second, IA has to deal with different spatial scale levels.
One of the ultimate challenges in IA is to connect higher
scale assessments with lower scale ones. So far, there has
been hardly any experience with playing around with scale
levels in IA. Down-scaling or up-scaling the spatial level
of analysis has profound consequences. This is related to
the question to what extent the processes considered are
generic, or distinctly spatially bound in character. In other
words: does a relationship hold at larger or lower scale
levels? For instance, land and water issues seem to be
local/regional by nature, whereas atmospheric issues often
appear to be trans-regional or even global.
In this context, Root and Schneider [95] propose an in-
teresting approach: the so-called Strategic Cyclical Scaling
(SCS) method. This method involves continuous cycling
between large and small-scale assessments. In modeling or
scenario terms such an iterative scaling procedure implies
that a specific global model or scenario is disaggregated and
adjusted for a specific region, country or river basin. The
new insights are then used to improve the global version,
after which implementation for another region, country or
river basin follows. This SCS method can be used for con-
ceptual validation of models or scenarios.
Methodologically, a local or regional approach for IA
is more attractive than a global approach, for a number of
reasons (Costanza and Tognetti [18]) in terms of:
• data quality: more reliable data is available locally;
• manageability: fewer interactions, and fewer causes and
impacts have to be considered;
• communication: greater political interest;
165
• reciprocity: to test the reciprocal relationship between
local/regional processes and global processes.
Third, IA is faced with a multitude of temporal scales.
Short-term needs and interests of stakeholders have to be
considered. However, biogeochemical processes usually
operate on a long time scale, whereas economic processes
operate on short- to medium-time scales. Another chal-
lenging aspect of IA is to interconnect long-term targets
as specified as a result of analysing processes operating on
longer-term time scales, with short-term goals for concrete
policy actions (Jaeger et al. [49]). Unfortunately, there is
not yet a sound scientific method how to do this; thus so
far only heuristic methods have been used.
3.1.2. Treatment of uncertainty
Any exploration of future developments inevitably in-
volves a considerable degree of uncertainty and Inte-
grated Assessment is no exception. Because of the cross-
disciplinary character of IA, it includes many different types
and sources of uncertainty. Because IAs are end-to-end ap-
proaches, they also contain an accumulation of uncertain-
ties. Uncertainties may arise from incomplete knowledge
of key physiological, chemical and biological processes.
Many uncertainties are of a socio-economic nature – related
to people’s behaviour – and reflect inadequate knowledge
with respect to the driving forces of human behaviour.
Various attempts have been made to classify the differ-
ent types and sources of uncertainty. Morgan and Hen-
rion [76] distinguish uncertainty about empirical quanti-
ties and uncertainty about the functional form of mod-
els, which may arise from: subjective judgement; dis-
agreement among experts; systematic errors; approxima-
tion; and inherent randomness. Funtowicz and Ravetz [35]
classified uncertainties in three categories: (i) techni-
cal uncertainties (concerning observations versus measure-
ments); (ii) methodological uncertainties (concerning the
right choice of analytical tools); and (iii) epistemological
uncertainties (concerning the conception of a phenomenon).
Whatever classification is chosen, the various types and
sources of uncertainties in IAs need to be addressed in an
adequate manner. Unfortunately, the classical methods of
handling uncertainty, such as Monte Carlo sampling, and
the usage of probability density functions, are merely useful
to address technical uncertainties. Methodological and epis-
temological uncertainties, however, which primarily arise
from subjective judgements and fundamental disagreement
among experts, cannot be addressed by the classical meth-
ods. Applying these methods usually results in estimates of
minimum, maximum and best guess values that are often
erroneous and misleading.
An alternative and simple way of presenting uncertain-
ties is by specifying a set of future scenarios, where scenar-
ios then span a range of plausible, representative futures.
This, however, does not give an indication of the cumula-
tive uncertainty as well as the origin and meaning of the
uncertainty range.
166 J. Rotmans / Methods for IA
In IA models uncertainties are often reduced to technical
artefacts. By attaching deterministic intervals or stochastic
probability distribution functions to uncertain model para-
meters, it is suggested that variations in parameter values
do yield estimates of the uncertainty in the model outcome.
However, that may be true in the mathematical sense (al-
though only partly), but it does not reflect the nature and
source of the real world uncertainties. One may compare
one type of uncertainty with the other in mathematical
terms, but in physical terms that could lead to compar-
ing apples and pears. It is not allowed to simply compare
the uncertainty of the climate sensitivity (representing un-
certainties in geophysical feedbacks) with the uncertainty
of the fertility rate (representing uncertainties in triggering
factors behind fertility behaviour). While the geophysical
uncertainty might be reduced by future research, the demo-
graphic uncertainty might be structural in the sense that it
cannot be reduced in the longer term.
In sum, current methods to analyse uncertainties in IAs
are unable to: (i) produce an order of magnitude of the cu-
mulative uncertainty; (ii) render various sources and types
of uncertainties explicit; (iii) provide systematic and coher-
ent clusters of uncertainties; and (iv) explain and clarify
uncertainties in a manner understandable to decision mak-
ers.
In order to meet the above requirements, we need new
methods. An interesting manner to make various types
of uncertainties explicit might be to link uncertainty aris-
ing from subjective judgement and fundamental disagree-
ment among experts to the different perspectives that people
have adopted. A typology of perspectives is then necessary
to arrive at a limited set of perspective-based interpreta-
tions of uncertainties. The method of multiple perspective-
based model routes (van Asselt and Rotmans [7]) helps to
understand the differences in projections as the outcome
of the divergent views and valuations, instead of merely
low, medium and high values. The first application of this
method in IA modeling is extensively described in Rotmans
and de Vries [37].
3.1.3. Blending qualitative and quantitative knowledge
In most frameworks for IA, quantitative and qualitative
knowledge are considered and treated as mutually exclu-
sive. For instance, usually those aspects of a problem
under concern that are not well known, or about which
there is only vague and qualitative knowledge, are left out
in the modeling process. This means, however, that we
miss crucial links in the causal chains that form archetypal
patterns of human–environment interactions. Quantitative
rigour therefore prevents IA from being comprehensive, in
the sense of studying all relevant aspects of a complex prob-
lem. It is therefore illusory to think that the full complexity
of human–environment interactions could be integrated into
a formal, quantitative modeling framework.
On the other hand, purely qualitative assessments in the
form of narratives that are not underpinned by quantitative
estimates cannot be checked in terms of consistency and
validity. IA therefore needs combinations of quantitative
and qualitative approaches, from the perspective that they
complement each other. To this end, analytical and partic-
ipatory methods are not, and should not be, mutually ex-
clusive alternative methods for integrated assessment. The
major methodological challenge for IA is to take advan-
tage of the complementary value. In order to combine and
display qualitative and quantitative information in a sys-
tematic, coherent framework, various traditional methods
could be used (Morgan and Henrion [76]). However, we
will single out only two of the “new” methods: fuzzy logic
and knowledge patterns.
One promising way to blend quantitative and qualitative
information could be to incorporate vague and qualitative
knowledge in IA models. A useful tool to implement vague
information is fuzzy logic. Contrary to Boolean logic, fuzzy
logic makes use of a continuum of values, which represent
elements of fuzzy knowledge. This technique has been
successfully applied in control and optimisation problems.
In the field of IA, the technique is, amongst others, be-
ing applied in the Syndromes project, where syndromes are
the symptoms of global change as proposed by the Ger-
man Advisory Council of Global Change (Schellnhuber et
al. [102]).
A more radical approach would be to leave the pathway
of strict cause–effect chains, and enter the road of knowl-
edge patterns. Knowledge patterns are sequences of knowl-
edge elements which consist of facts (certain, quantitative
knowledge), hypotheses (fuzzy, qualitative knowledge), and
fundamental uncertainties (perspective-based knowledge).
The basic idea is to consider complex issues as controver-
sies, where each controversy is analysed as a knowledge
pattern that consists of a sequence of facts, hypotheses
and uncertainties. The knowledge pattern could then be
coloured according to different perspectives. This helps to
develop a consistent and coherent picture of the underlying,
complex dynamics of the controversy in question.
3.2. Method-specific challenges
3.2.1. Models
During the last decades, IA models have proven them-
selves as legitimate and powerful approaches to com-
plex issues. In particular to the climate change debate,
IA models have contributed by exploring impacts of climate
change and evaluating mitigation and abatement strategies.
IA models have also provided useful information on bal-
ancing the carbon budget, sulphate aerosols, and on various
integrated aspects of land use.
However, there is still a long way to go before IA models
will be fully accepted by the scientific modeling commu-
nity on the one hand, and by the decision making com-
munity on the other. Therefore, an ultimate challenge for
IA modeling is to build up scientific and political credi-
bility. To improve both scientific and political credibility
of IA models, IA scholars should enhance IA models, im-
prove the communication with disciplinary scientific and
 J. Rotmans / Methods for IA
policy-making communities, and enrich and augment com-
munication techniques. These strategies will be discussed
in this section.
Enhancing IA models is a continuous process, and vari-
ous strategies for further improving IA models can be fol-
lowed. First, realising that an IA framework is as good
as its weakest part of the whole model chain, it is much
more effective to improve the rather weak or poorly defined
model parts, rather than refining and disaggregating the al-
ready adequate submodels. Second, a no-regrets modeling
strategy is to bring in, where necessary and where possible,
region-specific information for submodels, but maintain as
overall modeling strategy the ability to run the model at
a fairly high level of aggregation. Third, building generic
submodels (models independent of regional or temporal dif-
ferences) has many disadvantages. This means that the the-
ories and assumptions must be applicable at different levels
of spatial aggregation and for different regions in differ-
ent periods (Rotmans and Dowlatabadi [100]). And finally,
empirical testing is a crucial element in enhancing the qual-
ity of IA models. To this end, various model components
could be tested against disciplinary expert models, not only
to compare the outcomes of both types of models, but also
to validate the scientific hypotheses, process formulation,
boundary conditions, and model structure with each other.
For brevity’s sake we will not discuss here all modeling
strategies, but instead focus on improving the weak parts
of IA models. Let us consider some of the weaker parts
of IA models. In general, knowledge of key dynamics in
both social and natural systems is limited. But there is no
doubt that IA models are unbalanced with regard to the
representation of both interacting systems.
While in many macroeconomic-oriented IA models the
socio-economic aspects are adequately represented, but the
physical aspects poorly, in most biosphere-oriented IA mod-
els the natural system is reasonably well represented, but
the socio-economic system only caricaturally. For example,
the dynamics of the climate–atmosphere–biosphere system
continues to be far from well understood, mainly because of
a dearth of knowledge and data about the internal dynamics
of this complex system. A continuing stream of surprise
findings (e.g., missing carbon and nitrogen sinks, aerosols,
and solar spots) typifies this. Still, there is a solid scientific
knowledge base for disciplinary modeling of many biogeo-
physical processes. Disciplinary expert models exist, from
which knowledge can be extracted and represented in re-
duced form.
With regard to the socio-economic system, the situa-
tion differs materially. The social scientific knowledge base
is smaller, resulting in less disciplinary models to be rep-
resented in reduced form in IA models. In addition, the
modeling of socio-economic aspects suffers from a dearth
of basic data, in particular outside industrialised countries.
Therefore, the models we have developed to describe so-
cial dynamics have not been calibrated and validated ad-
equately. There is, however, still sufficient knowledge of
vital socio-economic processes to use in the IA modeling
167
context. So far, the knowledge available has hardly been
used in IA models. Among the more pressing questions in
social dynamics are the following:
• What brings about the demographic and epidemiological
transition?
• What drives environmentally related human behaviour?
• What are the roots and dynamics of technological inno-
vation and diffusion?
• What drives processes such as urbanisation and migra-
tion?
Although the above questions cannot be answered unam-
biguously, sound scientific hypotheses do exist. Using these
hypotheses in combination with data and expert models
available, forms the basis upon which these socio-economic
processes can be represented in IA models. Here, we will
only present IA modeling examples in the fields of demog-
raphy and health, and human behaviour. Both cases are
examples of generic model building strategies, i.e., mod-
els that can be applied to different scale levels. As far
as the remaining socio-economic issues are concerned, the
reader is referred to other sources (e.g., Schneider [103],
Dowlatabadi [24]).
IA modeling of consumption behaviour. Human consump-
tion of goods and services is a major factor in the process
of global change. So far, IA models only implicitly incor-
porate the behaviour of consumers. Two main reasons for
modeling consumer behaviour more explicitly are: (i) to
enhance and validate future projections generated by IA
models; (ii) to offer the possibility to simulate effective
and efficient behavioural strategies in the context of IA.
The growing concern about the environment has stim-
ulated the social sciences in developing conceptual multi-
theoretical models describing various environmentally detri-
mental behaviours. Multi-theoretical here indicates that
various theories on specific types of behaviour are com-
bined. These conceptual models are useful for the post-
hoc explanation of environmentally relevant behaviour, and
they indicate policy strategies that might be effective in
changing such behaviour. However, because these con-
ceptual models generally involve a poor specification of
the relations among relevant variables and lack a system-
dynamical approach, they do not allow for the translation
into mathematical models, which would fit into the frame-
work of IA models. A useful behavioural model within the
framework of IA should enable us to develop projections of
future behaviour, thus providing a tool for the simulation of
behaviour. Such a generic behaviour model provides means
to generate possible answers to the following questions:
• How do various kinds of consumptive behaviour interact
with projected developments in various domains, e.g.,
population dynamics, water, food, and energy resources?
• Which response strategies provide effective means to
change particular kinds of environmentally detrimental
168 J. Rotmans / Methods for IA
consumptive behaviour in the context of the search for
a sustainable future?
In order to be able to address these questions, the dynam-
ics of the behavioural process have to be modeled in a for-
mal, quantitative way. In the field of economics, modeling
consumer behaviour has a long history. Demand theory is
well developed, and allows for the specification of demand
as a function of, e.g., prices and income. Models based
on this theory suppose that there is a general equilibrium
between demand and supply (which is also price-driven).
The dynamics then arise from: (i) providing for market
clearing; (ii) adding more market levels; (iii) adding a time
dimension; (iv) adding an optimal allocation of resources;
and (v) changing consumer preferences.
However, the overall problem with the neo-classical gen-
eral equilibrium theory is that it does not allow for multiple
equilibria (such as the transition from one economic equi-
librium to another), and the complex dynamics of price
adjustments. Further, it does not take into account positive
feedbacks such as “learning by doing”. Finally, equilib-
rium dynamics are only valid on a certain domain, for in-
cremental changes from a given situation. The attempts
to model consumer behaviour in current IA models are
largely based on the general equilibrium model. For ex-
ample, many macroeconomic-oriented IA models feature a
Ramsey type of growth model in which a consumer allo-
cates income between consumption and investment with a
view to intertemporal utility maximisation.
The above obviously illustrates that there is a need for
an integrated, more realistic approach towards modeling
of consumer behaviour. This means a systems-based ap-
proach of consumer choice (like a physically based model
of ecosystem succession). An example of such an inte-
grated, conceptual model of consumer behaviour is pre-
sented in box 1.
Box 1
A conceptual model of consumer behaviour.
A conceptual model of consumer behaviour is under devel-
opment, and described in Jager et al. [51,52]. As a basis for
the model, a systems approach is chosen, using the Pressure–
State–Impact–Response framework. The system-dynamical
structure allows for the inclusion of interactions and feed-
back loops. The Pressure system describes the pressures on
consumptive behaviour in terms of people’s Motivation, Op-
portunities and Abilities (MOA). Then, the relations between
the MOA variables and behavioural processing are described
in the State system, based on eight main theories on behav-
iour. The Impact system describes the impacts of human
behaviour in terms of consumption (amount of product use)
and need–satisfaction. Further, four types of policy strategies
are adopted in the Response system: (i) provision of physi-
cal alternatives; (ii) regulation; (iii) financial stimulation; and
(iv) social and cognitive stimulation.
The first principles on which consumer behaviour is based
in this model are the following. The concept of motivation
is concerned with the satisfaction of human needs by means
of “opportunity consumption”. The more one perceives an
opportunity as a possibility to satisfy certain needs, the higher
one’s motivation to use this opportunity. Opportunities and
abilities are characterised in terms of physical means, finan-
cial means, knowledge, authorisation, and one’s social and
organisational support. Dependent on the level of motiva-
tion, opportunities and abilities, the behavioural process may
include more or less deliberation or cognitive processing. The
degree of cognitive processing before using or rejecting any
opportunity depends on the strength of a consumer’s motiva-
tion to use this opportunity and on his or her behavioural con-
trol. The underlying paradigms are those of reasoned versus
automatic behaviour processes, in combination with individ-
ual versus social behaviour processes, spanning eight major
theories on consumptive behaviour.
In translating the conceptual model into a behaviour simu-
lation model, consumers can be represented as computerised
agents, so-called “consumats”. The basic design of every
consumat comprises a set of rules, which are rooted in the
used theories of human behaviour. However, the consumats
differ with respect to their motivations (e.g., cultural perspec-
tive), and personal abilities, thus representing various types
of consumers. By including interactions and feedbacks be-
tween different consumats, various social processes can be
simulated: e.g., status-related consumption, the dispersion of
new consumer opportunities (“trickling down”) and the man-
ifestation of scarcity.
IA modeling of population and health. Fertility modeling
approaches are widely accepted and, only recently, existing
mathematical techniques have been introduced in the health
area (Weinstein et al. [117], WHO [120]). Usually, regres-
sion techniques are used to explore the relations between
broad health determinants, like literacy, income status, nu-
tritional status, water supply and sanitation, education and
medical services, and the health status measured in healthy
life expectancy. However, these regression techniques can
only give some suggestive evidence on the causes of pop-
ulation and health changes. Statistical models to estimate
future fertility and health levels are based on extrapolation
of past and current data. They operate on a short time
horizon, and are static in terms of specifying the dynamics
behind changing fertility and health patterns.
Therefore, there is a need for integrated approaches that
take account of the simultaneous occurrence of multiple
risk factors and diseases, as well as cause–effect relation-
ships. Such an integrated approach cannot be used in the
clinical area on an individual basis, but is appropriate at the
population level.
In Rotmans and de Vries [99], a Pressure–State–Impact–
Response systems approach to population and health is pre-
sented. A generic model has been designed that simulates
the driving forces (socio-economic and environmental fac-
tors), the fertility behaviour and disease-specific mortality,
the burden of disease and life expectancy as well as the size
and structure of the population, and a number of fertility
and health policies. The major objective of the population
and health model is to simulate changes in morbidity and
mortality levels under varying social, economic and envi-
ronmental conditions.
A second but related example is the MIASMA frame-
work, which is designed to describe the major cause and
effects relationships between atmospheric changes and hu-
 J. Rotmans / Methods for IA
man population health (Martens [67,68]). The model fo-
cuses on climate change (in terms of changes in temper-
ature and precipitation) and ozone depletion (in terms of
changes in UV-B radiation). Under varying climate and
ozone regimes, changes in the dynamics and distribution
of vector-borne diseases are simulated (malaria, schistoso-
miasis, and dengue), changes in cardiovascular diseases,
changing patterns of skin cancer incidences, and changing
mortality levels as a result of thermal stress. Although the
model operates on the global level, it is being validated
for specific developing regions. It should be noted that this
model does not take into account changes in the health care
system. A further integration step would then imply esti-
mates of future human health conditions as a result of both
environmental and socio-economic changes.
Regional IA modeling. The majority of current IA models
consists of global models. Many of them are subdivided
into world regions, varying from 2 till 16. Notwithstanding
the dominant orientation towards global modeling, there is
a growing need and interest in regional assessments, and
thus regional IA models. At the regional (or local) level
the tensions between different forms of development can
be made spatially-explicit. Regional development is char-
acterised by increasing competition for space, resources and
finances between a variety of stakeholders. An Integrated
Assessment of regional development then involves the es-
timation of trade-offs between space and resources claimed
by economic development (industry, services, public trans-
port), social development (houses, recreation, and private
transport), and environmental development (nature preser-
vation, quality of water, soil, air, noise pollution).
So far, spatially explicit modeling has been dominated by
so-called GIS approaches. GIS models describe the spatial
changes in a landscape in a detailed way, but they are static
and thus do not give insights into the dynamic mechanisms
behind past and future changes. On the other hand, models
that endeavour to describe the dynamics of a number of
relevant social, economic and ecological processes, system-
dynamic models, do not adequately (or not at all) describe
spatial changes.
Recently, however, the technique of cellular automata
has been developed, derived from Artificial Intelligence,
that enables the linkage between system-dynamic and GIS
models. Cellular automata are cells (automata) in a grid-
field, where all cells in the grid-field communicate with
each other through simple decision rules or algorithms. In
this way these protocols determine the dynamic spatial be-
haviour of all cells. The protocols may be based on differ-
ent paradigms. For instance, in the neo-classical economic
paradigm the problem can be translated as an optimisation
problem: how to use the land to achieve the highest possi-
ble level of human welfare, given differences in soil suit-
ability, climate, land use opportunities, and capital endow-
ments. Other paradigms, however, take notions as equity
or ecological quality as guiding principles for determining
the decision protocols.
169
The basic procedure of using cellular automata models,
GIS models, and IA models in an integrative way is as fol-
lows: macroscopic patterns indicate changes on the macro-
scale for a region with regard to: physical, demographic
and socio-economic changes, which are generated by sce-
narios in combination with IA models. These macroscopic
trends will be passed on to a region-specific cellular au-
tomata model, which will allocate the detailed localisation
of these changes on the basis of its own spatial dynam-
ics. Using site-specific information in combination with
GIS information, cellular automata allow for representation
of the microscopic dynamic patterns. Then the cellular au-
tomata model will feedback local-specific information to
the macroscopic trends to see what kind of consequences
this may have. An example of dynamic spatially-explicit
IA modeling is presented in box 2.
Box 2
Region-specific modeling using cellular automata.
Let us consider a nature area that is surrounded by a num-
ber of urban agglomerations. These cities have as a com-
mon characteristic that they want to expand, which brings
the nature area under intensifying pressure because of macro-
scopic socio-economic trends. There is an increasing need for
new houses, for drastic infra-structural changes. On the other
hand, there is increasing opposition against those interven-
tions in favour of preservation of the nature and recreational
function. According to the different perspectives of the vari-
ous stakeholders, the nature area could be filled in differently.
The anthropocentric perspective would lay the emphasis on
the extension of social and economic activities, clearing the
way for far-reaching infra-structural changes (road transport,
air traffic, harbour function and housing). The ecocentric per-
spective will emphasise the important ecological and recre-
ational function the area has. Whereas the holistic perspective
will favour the dynamic equilibrium between social, economic
and ecological needs and interests (more houses with space
for infra-structural changes in combination with a nature and
recreational function).
Macroscopic trends could indicate that in the nature area
50,000 new houses would be needed. The cellular automata
model will allocate these houses on the basis of availability
of land, suitability of land, accessibility of the location, the
immediate vicinity of economic and industrial activities, etc.
Next, the cellular automata will indicate what and where the
amount of available land is, what the price and suitability
are, etc. In addition, information can be provided on how
to fill up space in the sense of: How will land use change?
What kind of ecologically valuable land will be lost? What
is the risk of surface water pollution? What type of activity
clusters will arise? Is there a tendency towards overconcentra-
tion or ghetto formation? Will economically weak activities
be threatened by economically strong ones? Based on this
information macroscopic trends could be adjusted, using sim-
ulation models, to reconsider the suitability and attractiveness
of the specific region for new houses.
The advantage of using cellular automata is that it en-
ables visualisation of complex, spatial dynamics of regional
development in a transparent way. The disadvantage may
be that it still has to be explored which processes deter-
mining regional development can be captured by cellular
automata, and which cannot. While for physical processes
170 J. Rotmans / Methods for IA
such as infrastructural changes the cellular approach seems
appropriate, for mental processes representing human be-
haviour (e.g., consumptive patterns), application of cellular
automata seems more difficult.
Multi-agent modeling. The introduction of actors (agents)
allows to model self-organisation phenomena in complex
systems. Agent-based models focus on the micro-level in-
teractions in a system, which may lead to unexpected emer-
gent behaviour of the system. These models specify rules
that determine agent behaviour (individual organisms, peo-
ple or firms), where evolution of agents is feasible. The ad-
vantages of agent-based modeling are that: (i) the agents to
be modeled are not identical and homogeneous; (ii) the low
level interactions between agents are important; (iii) spatial
location is important; and (iv) it is easier to specify rules
for the actions of individual agents than it is to specify the
system behaviour. An example of agent-based modeling is
presented in box 3.
Box 3
Region-specific modeling using cellular automata and multi-
agents.
An interesting experiment is to supplement a cellular automata
model by a multi-agent approach, which allows to model self-
organisation phenomena in complex systems. Linking the de-
cision rules underlying cellular automata models to an agent-
based model, enables to model the regional implications of
human behaviour in a spatially-explicit way.
Let us consider the same case study as in box 2. However, we
now place agents in the form of decision makers in control
of the nature area that may evolve in any kind of direction.
The decision makers are responsible for infrastructure, land
use, energy services and taxation. Based on the outline given
by the decision makers, the regional population, representing
the second type of agents, decide whether or not to build,
where to live (so long as there are houses for them), what
routes to take to travel to work, etc. The fiscal environment
encourages business to locate or move out, affecting employ-
ment. The result could be a rich and absorbing simulation
of a developing nature area, where the decision makers take
“top–down” decisions, and the regional population make their
“bottom–up” responses.
3.2.2. Scenarios
The following lessons can be learned from two decades
of scenario analysis (Anastasi [5]):
• Scenarios with a strong narrative as well as a quantita-
tive component are able to engage stakeholders.
• A group of individuals with different, yet complemen-
tary, knowledge, experiences, and perspectives is needed
to provide scenario “richness”.
• Active dissemination is needed to ensure that stakehold-
ers make best use of scenarios.
• Models can play an important underpinning role, pro-
viding a valuable consistency check.
Many scenario exercises performed during the last decades
had an “engineering type” of character. Engineers, de-
signers and planners constructed future worlds behind their
desks, where those future development paths were dom-
inated by rather technocratic world views. Above all, it
needs to be stated that these “engineering type” of scenar-
ios are often well thought and have proven their value in
a particular context for a particular purpose. The argument
made here, however, is that these scenarios are of limited
or no value if used in a broader setting, and if we want
them to be actively used in the policy or societal context.
Taking into account these lessons, in combination with
the observed shortcomings as indicated before, future chal-
lenges for scenario analysis in the context of IA can be
identified. The ultimate challenge for IA is to develop in-
tegrated, balanced and “rich” scenarios that can help pru-
dent decision makers to understand past and current de-
velopments, to anticipate the future and to evaluate policy
options and strategies. What are the characteristic require-
ments for these scenarios in order to fulfil a valuable role
in the decision supportive process?
First, it is of vital importance that integrated scenar-
ios should be developed in a participatory way, involv-
ing a representative but heterogeneous group of stakehold-
ers. Second, integrated scenarios should display diversity,
which means that they should contain multiple perspectives,
and should not be developed along the lines of narrow,
disciplinary-based perspectives. Economists, for instance,
tend to think about the future solely in terms of economic
costs and benefits, defining away those societal processes
that cannot be expressed in monetary terms.
Integrated scenarios also need to be comprehensive: in-
cluding economic, environmental, institutional, and social
dimensions. Anastasi [5] noted that especially social di-
mensions are generally overlooked in scenario activities.
Because of the multitude of choices and assumptions in sce-
narios, many of which are hidden, it is important to make
them transparent. By nature, comprehensiveness is at odds
with transparency, which makes it even more challenging
to reconcile them. In addition, the underlying values, bi-
ases, and preferences, which are often hidden, need to be
made explicit.
Integrated scenarios are consistent with regard to as-
sumptions for different sectors, different problems, and
scale levels. Playing around with different spatial and tem-
poral scale levels is essential. Usually, the higher the spatial
scale level, the more ambitious the policies as formulated in
scenarios. However, implementing those policies at a lower
scale level is another story. So exercises that make the ten-
sions explicit between the different scale levels in terms of
policy formulation versus realisation, are very useful. Sim-
ilarly, many policy strategies in scenarios are formulated
in the long-term. If it is not clearly indicated what those
long-term strategies mean in terms of short-term, concrete
policy actions, the scenario exercise is of limited value.
A balanced scenario balances the narrative and the quan-
titative component. Challenging scenarios do appeal to the
user’s imagination, and are no simple extrapolations of past
 J. Rotmans / Methods for IA
or current trends, but include surprises (events) and turning-
points (e.g., bifurcations).
In sum, the challenge with regard to the development of
future generations of scenarios in the context of IA (inte-
grated scenarios) is twofold:
(i) to consider the scenario process as a creative and par-
ticipatory process in which a diversity of stakeholders
is involved; and
(ii) to consider the scenario products as helpful tools to
imagine the future, and to explore those imaginable
futures.
3.2.3. Risk analysis
Risk analysis seems to be a promising tool for Inte-
grated Assessment. Notwithstanding the involvement of
some scholars experienced in risk analysis in the IA com-
munity, we cannot conclude that there is a stable bridge
between IA and risk analysis. The major challenge for
the IA community is to reach out to the risk community
to learn what they have to offer in methodological terms.
In the long term, using risk analysis’ concepts, tools and
methods may eventually inspire risk analysts to use the
available disciplinary knowledge and experience on risk in
a complementary, more integrated way. In this way, pure
risk analysis efforts may benefit from IA.
3.3. Participatory methods
The major challenge is to gain more experience with
participatory methods in IA. The evaluation of these
exploratory experiences should lead to an overview of
strengths and weaknesses of the various approaches. Such
an overview indicates the requirements of participatory
methods for IA. Such insights should then lead to the fur-
ther development of appropriate tools, procedures and ide-
ally, manuals that allow to test and to improve participatory
methods for IA.
3.4. Methodological opportunities ahead
The following items are perhaps critical to the future
utility of IA:
• further development, usage, and improvement of scien-
tific methods for participatory processes;
• further development of quality assessment procedures;
• development of protocols for aggregation and disaggre-
gation;
• making the link between different spatial and temporal
scales to identify the role of individuals and institutions,
and their systems of governance;
• procedures for active dissemination of insights to soci-
etal stakeholders and the general public;
• an IA tool kit that comprises a wide variety of well-
documented tools and methods, and suggestions for how
171
the various methods can be used in a complementary
way;
• making the disciplinary sciences more interdisciplinary,
and aware of the limitations of their knowledge base
when applied in an integrated fashion to problems faced
by humanity.
4. Towards a research agenda for IA methods
The main challenge for the Integrated Assessment com-
munity is to overcome the scientific and political credibility
barrier. The requirements imposed on IA from disciplinary
scientists on the one hand, and from policy makers on the
other, seem to be contradictory. Scientists want their disci-
plinary knowledge represented in as adequate a manner as
possible, and they want to use disciplinary-based standards
to evaluate the quality of the building blocks of IA. Policy
makers want IA tools that are as transparent and easy to
use as possible (i.e., capable of exploring many possible
alternatives quickly), while retaining as much complexity
as possible.
The resulting task is then to create the right balance be-
tween simplicity and complexity, aggregation and disaggre-
gation, stochastic and deterministic behaviour, quantitative
and qualitative linkages, exogenous and endogenous fac-
tors, social and natural sciences, and, last but not least,
between analytical and participatory IA methods.
4.1. Codes of practice
Unmistakably, there is a need for guidelines for good IA
practice to evaluate the IA process, and quality rules for IA
to evaluate the products delivered by IA studies. The spe-
cial character of IA implies that in a certain way scientists
will be judged as witnesses giving testimony (Ravetz [91]).
The evidence for integrity is transparency. Guidelines for
good practice aim to make the process transparent, so that
the methods and processes of integration can be judged by
an “extended peer community”, involving other IA prac-
titioners, disciplinary scientists, users and affected publics
(Funtowicz and Ravetz [36–38]).
Different authors in IA have recently addressed the issue
of developing guidelines for good IA practice. Ravetz [91]
states that the quality of IA is assured only by the quality
of its production. Risbey et al. [92] emphasise that process-
based criteria should be developed. Bailey et al. [9] seek
to identify key aspects of good practice in IA. Morgan and
Dowlatabadi [77] propose hallmarks of a good IA. Oth-
ers like Parson [86] and Schneider [103] in turn have in-
terpreted and extended these contributions. Elements that
are commonly shared are plurality, incorporation of val-
ues, treatment of uncertainty, participatory process, and
dialogue.
As a matter of fact, the quality of the products is de-
pendent on the purpose and context of the study in ques-
tion. It is thus by definition impossible to come up with
172 J. Rotmans / Methods for IA
general quality standards that measure IAs with the same
yardsticks. Nevertheless, it is useful to start thinking about
rough guidelines for measuring the quality of IA studies.
Building upon Robinson [94], we make a distinction be-
tween three types of quality criteria: (i) analytical criteria;
(ii) methodological criteria; and (iii) quality in terms of us-
ability. Rotmans and van Asselt [101] elaborate on these
quality criteria, but within the scope of this paper we fo-
cus on the methodological criteria for IA. Methodological
criteria address the quality of IA methods, tools and ap-
proaches. Questions that are relevant with regard to these
criteria are: Are the chosen methods, tools and approaches
sound, appropriate and legitimate? What framework is used
for conducting the IA, and to what extent is this framework
integrated? How many disciplines are involved, and is the
study multi- or interdisciplinary? Are essential processes
left out, defined away or parameterised? Is there a balance
between aggregation and disaggregation and is it based on
a sound aggregation protocol? How are uncertainties dealt
with? Are values, preferences and biases made explicit?
What is the level of participation by what kind of stake-
holders?
We should be prudent in applying these quality criteria to
past and existing IA studies. Many IA studies had a rather
heuristic and explorative character. They can be compared
to pioneering activities such as experimental gardens where
new research methods are being tried out. Such typical
R&D studies rarely work from a well-thought-out method-
ological setting and research strategy, and often lack well-
defined research goals. These “supply-driven” IA studies
should not be measured too stringently against the above
quality criteria. On the other hand, if IA wants to evolve
to an independent, respected research field, the sheep have
to be separated from the goats, which can only be realised
if quality criteria for IA are applied. While on its way of
developing into a mature research field, which might take
another decade or so, a compromise would be not to apply
specific quality rules, but a check list of “codes of prac-
tice”. “Codes of practice” are rough guidelines for good
practice and quality rules, which have also been success-
fully applied in the interdisciplinary research field of Life-
Cycle Assessment (SETAC [104]). These codes of practice
should not be applied rigorously, but in a rather malleable
way, in order to avoid any kind of guillotine effect.
Concrete recommendation for EFIEA:
We therefore propose here to establish a European task
force that embarks on the formulation of a check list of
codes of practice that can be applied to former, current and
future IA projects in a malleable way, which will ultimately
result in a definitive list of quality rules and rules for good
practice of IA.
4.2. Demand- and supply-driven
We have observed that the majority of IA studies is
“supply-driven”. The most prominent example is climate
change, which was dictated by the (multi-)disciplinary re-
search agenda, rather than the policy agenda. In order
to increase the number of “demand-driven” IA studies,
amongst others advocated by the post-normal school of
thought (Funtowicz and Ravetz [39]), the strength of an-
alytical IA methods should be combined with the vigour
of participatory methods. As stated before, both types of
methods should be used in a complementary way. There is
a need to develop analytical methods as models and scenar-
ios in close conjunction with decision makers. This is more
easily said than done. For instance, having decision makers
as stakeholders in the IA model design and building process
is desirable indeed, but may lead to high expectations that
cannot be fulfilled by IA. If the state of scientific knowledge
does not permit reliable estimates of geographically-explicit
impacts of global climate change, scientists need to make
clear to decision makers the difficulty of attaining such an
unrealistic goal.
As regards IA models, this means in practice that, al-
ready in its conceptual phase, the IA model should be co-
designed by modelers, scientists, and decision makers (and
possibly by other stakeholders as well). This means in-
volvement of the community of stakeholders to establish
the model’s credibility and authority during its develop-
ment. Involvement includes discussions on the model’s in-
puts and outputs, the temporal and spatial scale level used,
the level of aggregation (detail) needed for developing pol-
icy strategies, issues and processes to be included or left
out, and presentation of the model set-up and model results
in terms of transparency. This culminates in a so-called user
model, which helps tailor the model to the user’s needs (van
Asselt [6]). Since this is a continuing process, an interme-
diary layer of project managers may be established between
the group of modelers and decision makers. Except for the
RAINS model (Alcamo et al. [4], Hordijk [46]), which was
partly co-designed by scientists and potential users, such
a demand-driven IA modeling process would be unprece-
dented.
In the field of risk analysis there is also a mismatch be-
tween demand and supply. In terms of risk demand, there
is a basic need for relevant information on risks of complex
problems. However, different actors do have different per-
spectives and thus different risk perceptions. This means
that the need for information is dependent on the context
within which decision makers are functioning. That also
means that what is relevant for one actor could be irrele-
vant for another. On the other hand, the risk supply side
represents the risk assessments as provided by the scien-
tists. However, a plurality of scientific approaches is avail-
able, primarily caused by the large uncertainties surround-
ing complex problems. Various methods are needed to cap-
ture different aspects of complex problems, and to produce
anything like a comprehensive understanding of the nature
and extent of risks.
To bring together the risk demand and supply side, par-
ticipatory methods could be used. Although there has been
some experience with participatory risk analysis, the in-
 J. Rotmans / Methods for IA
creasing complexity of problems the world faces today re-
quires a more integrated approach, balancing the demand
and supply side as much as possible.
The field of research where probably most experience
has been gained with participatory processes is that of
scenario analysis. However, much of that experience
arises from long-term planning in the business community,
within the context of strategic management. Multinationals
like Shell, for instance, have tremendous experience with
stakeholder-oriented development of scenarios. Within the
research community much can be learned from these stake-
holder approaches, but up till now there has been relatively
little exchange of scenario expertise between the research
and business community.
Concrete recommendation for EFIEA:
How can we realise a synthesis between the analytical
and participatory side of IA? It is proposed here to start up
two pilot IA projects, demand-driven and supply-driven:
• In the supply-driven IA endeavour, a group of scien-
tists anticipates the societal relevance of a complex
theme, using analytical IA tools to arrive at new insights.
Through participatory methods, the first assessment can
be enriched with societal aspirations, constraints and
public knowledge, after which the process can be it-
erated.
• A demand-driven IA pilot project is a participatory en-
deavour in which scientists, decision makers, and stake-
holders explore which complex issues are highly rele-
vant to a future society. The resulting common agenda
and explorative assessment then determine the analytical
methods to be either used or developed. The scientific
assessment then informs the next round of the partici-
patory process, etc.
The two approaches are visualised in figure 5.
4.3. Regional Integrated Assessment
Integrated Assessment is still on the way upwards of
building up authority. In this credibility process, there is
an urgent need for integrative case studies. Thus far, many
IAs had a fairly abstract character, which is partly due to the
global scale level at which IAs often operate. Case studies,
however, can convincingly demonstrate the added value of
IA, in particular case studies at the local or regional level.
Working at a lower scale level has two advantages: (i) IAs
can focus on concrete societal problems, issues which are
already at stake in a specific region; and (ii) IAs can be
made geographically-explicit, which allows for visualis-
ing the trade-offs between problems, sectors, resources and
scale levels to be made. On the other hand, a major disad-
vantage of working at a lower scale level is that the region
concerned has to be “isolated” from the outside world. In-
teraction with the outside world can be realised by speci-
fying exogenously important region-transcending processes
that impact on the specific region.
173
Figure 5. Demand- and supply-driven IA.
In spite of this disadvantage, regional Integrated As-
sessments may form fundamentally important examples of
the usefulness and applicability of IA. Unfortunately, there
have been only a few examples of integrated regional case
studies, most of which are impact studies, such as the
MacKenzie Basin impact study (Cohen [15]), the MINK
impact study (Rosenberg et al. [96]), and the neo-classical
regional U.S. impact study of Mendelsohn et al. [73].
While many of the past regional case studies focused
on regional impacts of climate change, there is a need for
a broader scope of regional Integrated Assessments. An
example is planning for sustainable development, which is
far more complex and will require different methods and
a changed paradigm from more traditional planning. New
methods will therefore have to be developed and applied,
on the interface between the short- and long-term, the ob-
jective and value-laden, the quantitative and the qualita-
tive, and the certain and the uncertain. Concrete ideas
for how to implement sustainable development at the re-
gional level are urgently needed. For Europe, two vital
entities can be considered for implementing regional sus-
tainable development. First, the European agglomerations,
in particular the cities and their peripheries, which hold
the key to the future spatial organisation of social, eco-
nomic and ecological changes in a sustainable way. And
second, the European catchments, where we may expect
a concentration of water-related problems. Catchments
are trans-regional “hot spots” that embrace many devel-
opments: population growth, economic growth, increasing
food demand, inefficient water use, water pollution, land
cover changes, climate change, and lacking infrastructure.
This makes catchments pre-eminently suitable for an inte-
grated approach.
174 J. Rotmans / Methods for IA
Concrete recommendation for EFIEA:
Integrated case studies are needed as practical examples
of Integrated Assessment. It is therefore recommended to
initiate two kinds of case studies in Europe:
• One for some representative European agglomerations,
to analyse the competition for space and resources in a
geographically-explicit way. A tool kit of new, promis-
ing IA methods should be tested for those agglomer-
ations, such as: cellular automata, genetic algorithms,
multi-agent models, in combination with existing IA
methods: simulation models, GIS models, and scenar-
ios.
• The other for a representative European catchment.
A promising example seems to be the Mediterranean
basin, for which previous prospective studies have
been done, e.g., the Blue Plan study (Grenon and
Batisse [41]), upon which we can build. For such a
basin, an integrated water assessment is to be developed,
in combination with specific basin scenarios, through
a participatory process involving stakeholders from the
Mediterranean countries.
4.4. Improving IA tools and methods
Currently used IA tools and methods are all relatively
immature (Rotmans and van Asselt [101]). To meet the
high expectations that surround IA, the quality of the IA
tool kit needs to be enhanced. To this end, two strategies are
proposed here. A short-term strategy, aimed at improving
the existing tools and methods, and a long-term strategy,
directed towards the creation of new tools and methods
that supplement the current tools and methods.
4.4.1. Short-term strategy
As already stated by many Integrated Assessors (e.g.,
Parson [85–87], Schneider [103], and Rotmans and Dow-
latabadi [100]) there is much room for improvement of cur-
rent IA tools and methods. Much of the criticism against
IA tools and methods has to do with their lack of trans-
parency, and the rather technocratic abstraction of reality
they display. The following suggestions might be helpful
in overcoming this communication barrier:
• Analytical methods should be documented (most of
them are now poorly documented), and descriptions and
findings should be published regularly in the open sci-
entific and policy literature. So not only in economics-
oriented journals, but also in environment-related jour-
nals.
• Using established and well-defined research methods
and goals can enhance the communication with the fun-
damental research community and with the decision
making world. Presenting these research methods in
an open and clear way makes, for instance, models and
scenarios as well as their results both comprehensible
and defensible.
• Tools like models and scenarios can be provided with in-
teractive communication shells or user interfaces, which
enhance the communication between designers and users
of those tools and methods. In general, visualisation
techniques facilitate the presentation of information in a
way that corresponds to the way people intuitively per-
ceive images. Innovative visualisation techniques can
help to make models and scenarios as transparent as
possible, to make the underlying theories as clear as
possible, to provide easy-to-use interactive interfaces,
and to display uncertainties and complexity of the sys-
tems behaviour comprehensibly. This all helps to make
the model and scenario outcomes, as well as the com-
plex messages they want to convey, more immediately
available and accessible to a wide range of users, who
may vary from the general public to decision makers.
Examples of user interfaces are the interactive scenario
scanner built around the IMAGE 2.0 model (Berk and
Janssen [12]), and the interactive user interface around
the TARGETS model (Rotmans and de Vries [99]), the
latter of which has appeared on CD-ROM.
• Indicators can also be useful tools in improving the com-
munication with decision makers. Indicators can be de-
fined as pieces of information that help measure changes
and progress in decision making. Decision makers al-
ready have indicators in their lexicon, using them to
formulate policies and resulting targets to be achieved.
Models, scenarios and risks can be linked systematically
to indicators. In this way, indicators serve as vehicles
to communicate model, scenario and risk analysis re-
sults, and can be used as a basis for mapping response
strategies.
• A common, unambiguous language could be developed
for external communication. IA researchers need to es-
tablish common meanings and definitions of often used
but also misused notions, such as horizontal versus verti-
cal integration, exogenous versus endogenous processes,
aggregation versus disaggregation, various types and
sources of uncertainty, calibration and validation, re-
duced form or meta-modeling, etc.
Concrete recommendation for EFIEA:
Current IA tools and methods can be relatively easily im-
proved, through making them more transparent to a wide
audience, varying from decision makers to the general pub-
lic. Although full transparency of current IA tools and
methods is illusory, it can be enhanced through:
(i) better documentation and more frequent publication;
(ii) using established and well-defined research methods
and goals;
(iii) developing interactive user interfaces (communication
shells);
(iv) using indicators that are linked to IA tools and meth-
ods; and
 J. Rotmans / Methods for IA
(v) developing a common, unambiguous IA language for
external communication.
4.4.2. Long-term strategy
The next generation of IA tools and methods should meet
the demands of scientists and decision makers adequately,
probably resulting in an even greater variety of IA tools
and methods addressing the scientific issues and the policy
debate at different levels.
A guiding principle for the development of new tools
and methods for IA could be a more realistic representa-
tion of the social and natural system. Still, the majority of
the tools and methods we use follow the research paradigm
that is based on equilibria, linearity, and determinism. The
result is that incremental changes in parts of the subsystems
we analyse will cause gradual and incremental changes in
the system as a whole. Unfortunately, the world does not
function in such a simple, linear way. The real world shows
strongly non-linear, stochastic, complex, and chaotic behav-
iour. This implies that incremental changes in conditions
of subsystems may result in considerable changes in the
results of the overall system, which may not be predicted
beforehand.
Neo-classical economics, for instance, is based on the
prevailing theory of general equilibrium. It supposes that
the free market brings about an efficient resource allocation,
except for a few “market failures” which can be remedied
by corrective taxes or subsidies. But without taking into
account the possibility of multiple equilibria, the complex
dynamics of price adjustments, and the role of positive feed-
backs such as “learning by doing” which makes established
technologies considerably cheaper over time, complex is-
sues such as the role of induced technological change can
hardly be discussed. An alternative, more realistic way
of describing the contemporary world economy would be
to consider multiple equilibria, in the form of a transition
of one economic equilibrium to another. New approaches
are underway that take into account multiple equilibria and
positive feedbacks. However, they do not yet provide a
coherent, comprehensive picture of the economy, still pro-
ducing equivocal results.
A new and promising research paradigm that has
emerged during the last decade is that of the complex, adap-
tive systems approach. Complex adaptive systems are com-
posed of many agents who interact with their environment
and can adapt to changes. These systems organise them-
selves, learn and remember, evolve and adapt. Evolutionary
models, for instance, focus on social and natural systems as
dynamically interwoven, where impacts on natural systems
feed back to the social system, and vice versa. Both systems
continuously adapt to human-induced perturbations of the
system as a whole. This evolutionary approach is applied
in various disciplines and uses a whole range of methods
and tools, for instance, genetic algorithms. A genetic al-
gorithm is a robust problem-solving approach based on the
mechanics of the survival of the fittest. After reproduction,
crossover and mutation, the fittest solutions pass on ele-
175
ments of themselves to later generations in an effort to find
the most successful solutions. A crucial assumption is that
the degree to which the agents’ expectations meet measure-
ments determines the fitness of the agents’ perspective.
Some preliminary exercises are already underway, test-
ing various elements of this new research paradigm (Rot-
mans and Dowlatabadi [100]). It would be extremely useful
to expand these experiments, and apply these experimental
methods to a limited number of concrete case studies. Obvi-
ously, in these experiments, the agents’ representations used
are quite abstract images of decision makers and stakehold-
ers. Only a weak isomorphism may exist between the real-
world adaptation, and the way in which simulated agents
adapt to their changing environment. Therefore, it may take
a while before the application of complex, adaptive systems
theory has matured to real world problems. Nevertheless,
the first preliminary experiments, which deal with the no-
tion of surprises and flexible strategies to respond to them,
are encouraging and already produce interesting results.
Concrete recommendation for EFIEA:
Therefore the overall challenge for the next generation
of IA tools and methods is to incorporate insights emerging
from new scientific streams such as the areas of complex
systems, adaptive behaviour, bifurcations, and ignorance
about systems. Incorporation of these insights will broaden
the scope of IA tools and methods and considerably im-
prove the quality of IA. In addition, the incorporation of
cultural values, biases and preferences, which determine
the manner in which complex problems are perceived and
formalised, will allow the IA community to inform the de-
cision making process in a more balanced way.
4.5. Entering new problem areas
IA has contributed to the climate change debate in ex-
ploring impacts of climate change, mitigation and abate-
ment strategies, issues in co-operative implementation, the
likely equity effects of candidate policies, and complicat-
ing factors such as aerosols. IA models have also provided
information on balancing the carbon budget and on vari-
ous integrated aspects of land use. In the past, IA has also
proven to be useful in the field of acid rain, where the IA
model RAINS played a guiding and supportive role in the
negotiating process of emission reductions of SO2 .
Now the time seems ripe for entering new problem ar-
eas from the perspective of IA. In doing this, the notion of
transitions could be a helpful guide. A transition may be
defined as a gradual, continuous shift in society from one
“mode of operation” to another. Transitions are described
by the following phases: (i) a take-off phase; (ii) an acceler-
ation phase, characterised by instability due to rapid techno-
logical, social and environmental change; during this stage
societies and environmental quality are highly vulnerable
to damage; and (iii) a stabilisation phase, in which the pace
of change slows and a new form of equilibrium is reached.
176 J. Rotmans / Methods for IA

Thus in system-dynamic terms, transitions suppose shifts decades), but also at different spatial levels: at the level of
from one relative equilibrium to another, moving from slow urban agglomeration (local sustainable development), at the
to rapid change, before reaching a different level of stability. national level (contribution to national emission profiles),
Using a transitional approach requires an analysis which and at the European level (contribution to the European
goes beyond the traditional cost–benefit approach. Such a emission targets as formulated in protocols for acidification
cost–benefit approach may be useful, but only for a certain and climate change).
domain, for relatively small departures from a given situ- Similarly, other fields that seem promising from an inte-
ation. Transitions, however, require major shifts from one grative perspective are: technological development, health
equilibrium to another, which makes cost–benefit analyses care, and infrastructure.
inappropriate (apart from the general methodological prob-
lems with measuring intangible impacts in monetary terms). Concrete recommendation for EFIEA:
One of the ultimate challenges for IA is to describe
complex societal dynamics in terms of a family of inter- We recommend to enter the following new problem ar-
connected transitions: social, economic, environmental and eas: water, technological development, health care, trans-
institutional transitions. Such a transitional approach opens port, and infrastructure.
the door to new problem areas, of which only a few will
be given here.
The water problem, for instance, lends itself to an inte- 5. Summary
grated transitional approach. The water transition can be
described in terms of increasing exploitation of resources Agenda for IA methodologies.
and increasing water pollution from industrialisation and
Activity Time span
urbanisation during the early stages of development. As
countries industrialise, the share of water used for agricul- Establishment of European task force Short-term
ture declines and an increasing proportion is taken by the IA pilots: Short-term
industrial, commercial and household sectors – domestic • supply-driven
consumption in particular increases greatly with more afflu- • demand-driven
ent lifestyles. This is followed by a period of more efficient
water use and water quality improvement, also induced by Integrated case studies: Medium-term
sophisticated water technologies. • European agglomerations
The water problem can be methodologically tackled by a • European catchments
transitional approach, considering the water transition in the
Improvement of current tools and methods Short-term
light of socio-economic and institutional transitions, start-
New generation of IA tools and methods Long-term
ing at a global level (through the influence of global cli-
(keywords: complex systems, adaptive behav-
mate change on water availability), descending to the re- iour, surprises, ignorance, cultural values, bias
gional level, where “hot spots” can be defined, down to the and preferences)
river basin or catchment level, which is a suitable method- New problem areas: Medium- and long-term
ological entity for practical implementation of a transitional
• water
approach.
• technological development
Another issue that seems appropriate for an integrated • health care
approach is transport. The transport problem comprises • transport
many dimensions, i.e., spatial, economic, ecological, in- • infrastructure
stitutional, and social/psychological. All dimensions are
heavily interrelated. The economic dimension concerns the
financial management of the problem: subsidising transport References
or following the market principles. This is related to the
institutional dimension, involving the co-ordinating aspects [1] J. Adams, Risk (UCL Press, London, 1995).
of the problem: public management versus market-based [2] J. Alcamo, IMAGE 2.0: Integrated Modeling of Global Climate
intervention. This relates to the social/psychological di- Change (Kluwer, Dordrecht, 1994).
[3] J. Alcamo, E. Kreileman and R. Leemans, Global models meet
mension that deals with equity and social cohesion versus
global policy, Global Environmental Change 6(4) (1996).
individualisation. This depends on the spatial organisation [4] J. Alcamo, R. Shaw and L. Hordijk, The RAINS Model of Acid-
of transport, creating compact (concentration) versus dif- ification: Science and Strategies in Europe (Kluwer, Dordrecht,
fuse (zones) cities, which is related to the ecological com- 1990).
ponent, addressing environmental pollution and quality of [5] C. Anastasi, Lessons learned from two decades of scenario de-
life at different spatial levels. velopment, Open meeting of the Human Dimensions of Global
Environmental Change Research Community, Laxenburg, Austria.
The transport problem operates at different temporal lev- [6] M.B.A. van Asselt, Global integrated assessment models as pol-
els: short-term (congestion, air pollution) versus long-term icy support tools: A triangular approach, University of Twente,
(investments in transport infrastructure over a period of Enschede, The Netherlands (1994).
 J. Rotmans / Methods for IA
[7] M.B.A. van Asselt and J. Rotmans, Uncertainty in perspective,
Global Environmental Change 6(2) (1996) 121–157.
[8] M.B.A. van Asselt, C.A.M.H. Storms, N. Rijkens-Klomp and
J. Rotmans, An overview and assessment of the last decade of
European scenario studies, ICIS Working Paper 98-01 (1998).
[9] P. Bailey, C. Gough, M. Chadwick and G. McGranahan, Meth-
ods for integrated environmental assessment: Research directions
for the European Union, Stockholm Environment Institute (SEI),
Stockholm, Sweden (1996).
[10] BCI, Economic vision on logistic and industrial junction Venlo,
Buck Consultants International, Nijmegen, The Netherlands (1996)
(in Dutch).
[11] U. Beck, Risk Society: Towards a New Modernity (Sage, London,
1986).
[12] M.M. Berk and M.A. Janssen, The interactive scenario-scanner
(ISS), RIVM report 981508005, Bilthoven, The Netherlands (1977).
[13] G.D. Brewer, Methods for synthesis: Policy exercises, in: Sustain-
able Development of the Biosphere, eds. W.C. Clark and R.E. Munn
(Cambridge University Press, Cambridge, 1986) pp. 455–473.
[14] K. Button, The future of European transport, in: The Future of
Transportation and Communication: Visions and Perspectives from
Europe, Japan and the USA, ed. R. Thord (Springer, Berlin, 1993).
[15] S.J. Cohen, Scientist–stakeholder collaboration in integrated assess-
ment of climate change: Lessons from a case study of Northwest
Canada, Environmental Modeling and Assessment 2(4) (1997) 281–
293.
[16] J. Conrad, Society and problem-oriented research: On the socio-
political functions of risk assessment, in: Risk: A Seminar Series,
ed. H. Kunreuther, IIASA, Laxenburg, Austria (1981).
[17] R. Costanza and M. Ruth, Dynamic systems modeling for scoping
and consensus building, in: European Chapter of the International
Society for Ecological Economics, University of Versailles, Paris,
France.
[18] R. Costanza and S.S. Tognetti, eds., Ecological Economics and
Integrated Assessment: Participatory Process for Including Equity,
Efficiency and Scale in Decision Making for Sustainability (SCOPE,
Paris, 1998, in press).
[19] V. Covello, Uses of social and behavioural research on risk, Envi-
ronment International 4 (June 1984) 17–26.
[20] K.K. Cox and J.B. Higginbotham, Application of focus group in-
terview in marketing, Journal of Marketing 40 (1976) 77–80.
[21] E.A.C. Crouch and R. Wilson, Risk/Benefit Analysis (Ballinger,
Cambridge, 1982).
[22] C.E. van Daalen, W.A.H. Thissen and M.M. Berk, in: The Delft
Process: Experiences with a Dialogue between Policy Makers and
Global Modelers, eds. R. Leemans and J. Alcamo (1997).
[23] M. Douglas and A. Wildavsky, Risk and Culture: Essays on the
Selection of Technical and Environmental Dangers (University of
California Press, Berkeley, 1982).
[24] H. Dowlatabadi, Sensitivity of climate change mitigation estimates
to assumptions about technical change, Energy Economics (1998,
accepted).
[25] H. Dowlatabadi and M.G. Morgan, Integrated assessment of climate
change, Science 259 (1993) 1813–1814.
[26] H. Dowlatabadi and M.G. Morgan, A model framework for inte-
grated studies of the climate problem, Energy Policy (March 1993)
209–221.
[27] C. Ducot and H.J. Lubben, A typology for scenarios, Futures 12
(1980) 15–57.
[28] D. Dürrenberger, J. Behringer, U. Dahinden, A. Gerger, B. Kasemir,
C. Querol, R. Schüle, D. Tabara, F. Toth, M. van Asselt, D. Vas-
silarou, N. Willi and C. Jaeger, Focus groups in integrated as-
sessment: A manual for a participatory tool, ULYSSES working
paper WP-97-2, Darmstadt University of Technology, Darmstadt,
Germany (1997).
[29] EC-DGXI, Vision 2020: Scenarios for a sustainable Europe, XI-
120-97, European Commission, Directorate General XI, Environ-
ment, Nuclear Safety and Civil Protection, General Consultative
177
Forum, Brussels (1996).
[30] EC-DGXVII, Energy in Europe: European energy to 2020 – A sce-
nario approach, European Commission, Luxembourg (1996).
[31] ECN, Energy scenarios for a changing Europe: Integration versus
fragmentation, ECN, Petten, The Netherlands (1995).
[32] J. Edmonds, H. Pitcher, N. Rosenberg and T. Wigley, Design for the
Global Change Assessment Model, in: Integrative Assessment of
Mitigation, Impacts and Adaptation to Climate Change, Laxenburg,
Austria (1994).
[33] J. Edmonds and J.M. Reilly, Global Energy: Assessing the Future
(Oxford University Press, Oxford, 1985).
[34] B. Fischhoff, S.R. Watson and C. Hope, Defining risk, Policy Sci-
ences 17 (1984) 123–139.
[35] S.O. Funtowicz and J.R. Ravetz, Uncertainty and Quality in Science
for Policy (Kluwer, Dordrecht, 1990).
[36] S.O. Funtowicz and J.R. Ravetz, Risk management as a post-normal
science, Risk Analysis 12(1) (1992) 95–97.
[37] S.O. Funtowicz and J.R. Ravetz, Three types of risk assessment and
the emergence of post-normal science, in: Social Theories of Risk,
eds. S. Krimsky and D. Golding (Greenwood, Westport, 1992).
[38] S.O. Funtowicz and J.R. Ravetz, Science for the post-normal age,
Futures 25(7) (1993) 739–755.
[39] S.O. Funtowicz and J.R. Ravetz, The worth of a songbird: Ecolog-
ical economics as a post-normal science, Ecological Economics 10
(1994) 197–207.
[40] G. Gallopin, A. Hammond, P. Raskin and R. Swart, Branch points:
Global scenarios and human choice – A resource paper of the
Global Scenario Group, Polestar Series Report no. 7, Stockholm
Environment Institute, Sweden (1997).
[41] M. Grenon and M. Batisse, Futures for the Mediterranean Basin
(Oxford University Press, Oxford, 1991).
[42] J. Grin and R. Hoppe, Toward a comparative framework for learn-
ing from experiences with interactive technology assessment, In-
dustrial and Environmental Crisis Quarterly 9(1) (1995) 99–120.
[43] E.G. Guba and Y.S. Lincoln, Fourth Generation Evaluation (Sage,
Newbury Park, 1989).
[44] A. Hausrath, Venture Simulation in War, Business and Politics
(McGraw-Hill, New York, 1971).
[45] C.S. Holling, Adaptive Environmental Assessment and Management
(Wiley, London, 1978, revision 1990).
[46] L. Hordijk, Use of the RAINS model in acid rain negotiations in
Europe, Environmental Science and Technology 25(4) (1991) 596–
603.
[47] IPCC, Climate Change: The IPCC Scientific Assessment (Cam-
bridge University Press, Cambridge, 1990).
[48] IPCC, Climate Change 1992 – The Supplementary Report to The
IPCC Scientific Assessment (Cambridge University Press, Cam-
bridge, 1992).
[49] C.C. Jaeger, T. Barker, O. Edenhofer, S. Faucheux, J.-C. Hourcade,
B. Kasemir, M. O’Connor, M. Parry, I. Peters, J. Ravetz and J. Rot-
mans, Procedural leadership in climate policy: a European task,
Global Environmental Change 7 (September 1997).
[50] C.C. Jaeger, M. Chadwick, B. Wynne, S. Funtowicz, M. Giaoutzi,
S. Giner, F. Toth, J. Jäger, G. Dürrenberger, J.R. Ravetz and C. Ca-
silli, ULYSSES – Urban lifestyles, sustainability and integrated en-
vironmental assessment – RTD proposal for the European Commu-
nity Framework Programme IV, Darmstadt Technical University,
Darmstadt, Germany (1995).
[51] W. Jager, M.B.A. van Asselt, J. Rotmans, C.A.J. Vlek and C. Boodt,
Consumer behaviour: A modeling perspective in the context of inte-
grated assessment of global change, 461502017, RIVM, Bilthoven,
The Netherlands (1997).
[52] W. Jager, M.B.A. van Asselt, J. Rotmans and C.A.J. Vlek, Con-
sumer behaviour, a modeling perspective in the context of inte-
grated assessment of global change, in: Sustainability in Question:
The Search for a Conceptual Framework, eds. J. Köhn, J. Gowdy,
F. Hinterberger and J. van der Straaten (Edward Elgar Publishers,
Abingdon, 1998).
178 J. Rotmans / Methods for IA
[53] S. Jasanoff, Bridging the two cultures of risk analysis, Risk Analy-
sis 13(2) (1993) 123–129.
[54] B.B. Johnson and V.T. Covello, The Social and Cultural Construc-
tion of Risk: Essays on Risk Selection and Perception (D. Reidel,
Dordrecht, 1987).
[55] H. Jungermann, Psychological aspects of scenarios, in: Environ-
mental Impact Assessment, Technology Assessment, and Risk Analy-
sis, eds. Covello et al., NATO ASI Series, Vol. G4 (Springer, Berlin,
1985).
[56] H. Jungermann, Risk, in: Technology Controversy. Actual Key No-
tions for Public Debates, eds. H. Schütz and W.P.M. (IMK, Frank-
furt am Main, 1993) (in German).
[57] H. Kahn and A. Wiener, The Year 2000 (MacMillan, New York
1967).
[58] E. Kämper, Sociology of risk: Implications for the analysis of
environmental policy, European University Institute, Florence, Italy
(1998). http://www.iue.it/WGES/ISS17/kaemper.h.
[59] B. Kasemir, J. Behringer, B. de Marchi, C. Deuker, D. Durren-
berger, S. Funtowicz, A. Gerger, M. Giaoutzi, Y. Haffner, M. Nill-
son, C. Querol, R. Schule, D. Tabara, M. van Asselt, D. Vassilarou,
N. Willi and C. Jaeger, Focus groups in integrated assessment: The
ULYSSES pilot experience, ULYSSES WP-97-4, Darmstadt Uni-
versity of Technology, Darmstadt, Germany (1997).
[60] B. Kasemir, M.B.A. van Asselt, G. Dürrenberger and C.C. Jaeger,
Integrated assessment: Multiple perspectives in interaction, Inter-
national Journal of Environment and Pollution (special issue “Meth-
ods and Models for Decision Support”) (1998, accepted).
[61] P. Kassler, Energy for development, in: Shell Selected Paper (Shell,
London, 1994).
[62] S. Krimsky and D. Golding, Social Theories of Risk (Praeger, West-
port, 1992).
[63] D.A. Lashof and D.A. Tirpak, Policy options for stabilizing global
climate, U.S. Environmental Protection Agency, Washington, USA
(1989).
[64] J. Linnerooth-Bayer and M. Thompson, Risk and governance: Pro-
posal to the European Science Foundation, IIASA, Laxenburg, Aus-
tria (1997).
[65] N. Luhmann, Risk: A Sociological Theory (De Gruyter, Berlin,
1993).
[66] MacCallaway, Personal communication (1998).
[67] W.J.M. Martens, Health impacts of climate change and ozone de-
pletion: An eco-epidemiological approach, Ph.D. thesis, Maastricht
University, Maastricht, The Netherlands (1997).
[68] W.J.M. Martens, Health and Climate Change: Modeling the Im-
pacts of Global Warming and Ozone Depletion (Earthscan Publi-
cations, London, 1998).
[69] D.G. Mayo and R.D. Hollander, Acceptable Evidence: Science and
Values in Risk Management (Oxford University Press, New York,
1991).
[70] A. Mazur, Societal and scientific causes of the historical develop-
ment of risk assessment, in: Society, Technology and Risk Assess-
ment, ed. J. Conrad (London, 1980).
[71] D.H. Meadows, D.L. Meadows, J. Randers and W.W. Behrens, The
Limits to Growth (Universe Books, New York, 1972).
[72] D.L. Meadows, D.H. Meadows and J. Randers, Dynamics of
Growth in a Finite World (Wright-Allen Press, Cambridge, 1972).
[73] R. Mendelsohn, W. Morrison, M. Schlesinger and N. Andronova,
Country-specific market impacts of climate change, unpublished
manuscript (1996).
[74] M.W. Merkhofer, Decision Science and Societal Risk Management
(D. Reidel, Boston, MA, 1987).
[75] I. Mintzer, A matter of degrees: The potential for controlling the
greenhouse effect, Research report No. 5, World Resource Institute,
Washington, USA (1987).
[76] G.M. Morgan and M. Henrion, Uncertainty – A Guide to Dealing
with Uncertainty in Quantitative Risk and Policy Analysis (Cam-
bridge University Press, New York, 1990).
[77] M.G. Morgan and H. Dowlatabadi, Learning from integrated as-
sessment of climate change, Climatic Change 34 (1996) 337–368.
[78] W.D. Nordhaus, The Efficient Use of Energy Resources (Yale Uni-
versity Press, New Haven, 1979).
[79] W.D. Nordhaus, The DICE model: Background and structure of a
dynamic integrated climate economy, Yale University, USA (1992).
[80] W.D. Nordhaus, Managing the Global Commons: The Economics
of Climate Change (MIT Press, Cambridge, MA, 1994).
[81] T. O’Riordan, Risk perception studies and policy priorities, Risk
Analysis 2(2) (1982) 95–100.
[82] T. O’Riordan, The impact of EIA on decision making, in: En-
vironmental Impact Assessment, Technology Assessment and Risk
Analysis, eds. V.T. Covello, J.L. Mumpower, P.J.M. Stallen and
V.R.R. Uppuluri (Springer, Berlin, 1985).
[83] H.J. Otway and D. von Winterfeldt, Beyond acceptable risk: On
the social acceptability of technologies, Policy Sciences 14 (1982)
247–256.
[84] H.J. Otway and D. von Winterfeldt, Risk management and accept-
able risk criteria, in: Risk: A Seminar Series, ed. H. Kunreuther,
IIASA, Laxenburg, Austria (1982).
[85] E.A. Parson, A global climate change policy exercise: Results of
a test run, July 27–29, 1995, WP-96-90, International Institute for
Applied Systems Analysis (IIASA), Laxenburg, Austria (1996).
[86] E.A. Parson, How should we study global environmental problems:
A plea for unconventional methods of assessment and synthesis,
WP-96-157, International Institute for Applied Systems Analysis
(IIASA), Laxenburg, Austria (1996).
[87] E.A. Parson, Three dilemmas in the integrated assessment of cli-
mate change, Climatic Change 34 (1996) 315–326.
[88] E.A. Parson and K. Fisher-Vanden, Integrated assessment of global
climate change, Annual Review of Energy and the Environment 22
(1997, submitted).
[89] Platform on Science and Ethics, Workshop reports on the societal
debate on genetic research, W43, Rathenau Institute, The Hague,
The Netherlands (1995).
[90] R.H.J. Prinn, A. Sokolov, C. Wand, X. Xiao, Z. Yang, R. Eckaus, P.
Stone, D. Ellerman, J. Melillo, J. Fitzmaurice, D. Kicklighter and
Y. Liu, Integrated global system model for climate policy analysis:
model framework and sensitivity studies, Global Change Center,
Massachusetts Institute of Technology, USA (1996).
[91] J.R. Ravetz, Integrated Environmental Assessment Forum: Devel-
oping guidelines for good practise, ULYSSES WP-97-1, Darmstadt
University of Technology, Darmstadt, Germany (1997).
[92] J. Risbey, M. Kandlikar and A. Patwardhan, Assessing integrated
assessments, Climatic Change 34 (1996) 369–395.
[93] J.B. Robinson, Life in 2030: Exploring a Sustainable Future for
Canada (UBC Press, Vancouver, 1996).
[94] J.B. Robinson, Of maps and territories: The use and abuse of socio-
economic modelling in support of decision making, Technological
Forecasting and Social Change 42 (1992) 147–164.
[95] Root and S. Schneider, Ecology and climate: Research strategies
and implications, Science 269(52) (1995) 334–341.
[96] N.J. Rosenberg, P.R. Crosson, K.D. Frederick, W.E. Easter-
ling, M.S. McKenny, M.D. Bowes, R.A. Sedjo, J. Darmstadter,
L.A. Katz and K.M. Lemon, The MINK methodology: background
and baseline, in: Towards an Integrated Impact Assessment of Cli-
mate Change: The MINK Study, ed. N.J. Rosenberg (Kluwer, Dor-
drecht, 1993).
[97] J. Rotmans, IMAGE: An Integrated Model to Assess the Greenhouse
Effect (Kluwer, Dordrecht, 1990).
[98] J. Rotmans, Integrated visions for a sustainable Europe: An inte-
grated assessment proposal, European Union, DG XII, Work Pro-
gramme Environment and Climate (second phase), Brussels, Bel-
gium (1997).
[99] J. Rotmans and H.J.M. de Vries, Perspectives on Global Change:
The TARGETS Approach (Cambridge University Press, Cambridge,
1997).
[100] J. Rotmans and H. Dowlatabadi, Integrated assessment of climate
 J. Rotmans / Methods for IA
change: Evaluation of methods and strategies, in: Human Choice
and Climate Change: An International Social Science Assess-
ment, eds. S. Rayner and E. Malone (Battelle Press, Washington,
1998).
[101] J. Rotmans and M.B.A. van Asselt, Integrated assessment: Current
practises and challenges for the future, in: Ecological Economics
and Integrated Assessment: A Participatory Process for Including
Equity, Efficiency and Scale in Decision Making for Sustainability,
eds. R. Costanza and S. Tognetti (SCOPE, Paris, 1998, in press).
[102] H.-J. Schellnhuber, A. Block, M. Cassel-Gintz, J. Kropp, G. Lam-
mel, W. Lass, R. Lienenkamp, C. Loose, M.K.B. Lüdeke, O. Mold-
enhauer, G. Petschel-Held, M. Plöchl and F. Reusswig, Syndromes
of global change, Gaia 6(1) (1997) 19–34.
[103] S. Schneider, Integrated assessment modeling of climate change:
Transparent rational tool for policy making or opaque screen hid-
ing value-laden assumptions?, Environmental Modeling and As-
sessment 2(4) (1997) 229–250.
[104] SETAC, Guidelines for life-cycle assessment: A “code of practice”,
Report of SETAC Workshop held at Sesimbra, Portugal, 31 March–
3 April 1993, Society of Environmental Toxicology and Chemistry
(SETAC) (1993).
[105] P. Slovic, B. Fischhoff and S. Lichtenstein, Rating the risks: The
structure of expert and lay perception, in: Environmental Im-
pact Assessment, Technology Assessment and Risk Analysis, eds.
V.T. Covello, J.L. Mumpower, P.J.M. Stallen and V.R.R. Uppuluri
(Springer, Berlin, 1985).
[106] J.P. van der Sluijs, Anchoring amid uncertainty, Utrecht University,
Utrecht, The Netherlands (1997).
[107] D. Smith, Eurofutures (Capstone, Oxford, 1997).
[108] C. Starr, Social benefit versus technological risk, Science 165
(1969) 1232–1238.
[109] R. Swart, Climate change: Managing the risks, Free University,
Amsterdam, The Netherlands (1994).
[110] TCPA and CER, Manchester 2020: Sustainable development
View publication stats
179
in the city region, Town country planning association (TCPA),
Manchester, UK (1996).
[111] F.L. Toth, Policy exercises, Simulation and Games 19 (September
1988) 235–276.
[112] UN-DPCSD, Critical trends: Global change and sustainable de-
velopment, Department for Policy Co-ordination and Sustainable
Development, United Nations, New York, USA (1997).
[113] C.A.J. Vlek and P.J. Stallen, Rational and personal aspects of risk,
Acta Psychologica (1980) 273–300.
[114] H.J.M. de Vries, SusClime, Globo report series 11, RIVM, Bilt-
hoven, The Netherlands (1995).
[115] VROM, The Netherlands in 2030 – Exploration of spatial per-
spectives, Ministry of Housing, Planning and the Environment
(VROM), The Hague, The Netherlands (1997) (in Dutch).
[116] WEC and IIASA, Global energy perspectives to 2050 and beyond,
WEC, London, UK (1995).
[117] M.C. Weinstein, P.G. Coxson, L.W. Williams, T.M. Pass, W.B. Sta-
son and L. Goldman, Forecasting coronary heart disease incidence,
mortality and cost: the coronary heart policy model, American
Journal of Public Health 77 (1987) 1417–1426.
[118] J. Weyant, O. Davidson, H. Dowlatabadi, J. Edmonds, M. Grubb,
E.A. Parson, R. Richels, J. Rotmans, P. Shukla, R.S.J. Tol, W. Cline
and S. Fankhauser, Integrated assessment of climate change: An
overview and comparison of approaches and results, in: Economic
and Social Dimensions of Climate Change, eds. J.P. Bruce, H. Lee
and E.F. Haites (IPCC, Cambridge University Press, Cambridge,
1996).
[119] C. Whipple, Opportunities for the social sciences in risk analysis,
in: Environmental Impact Assessment, Technology Assessment and
Risk Analysis, eds. V.T. Covello, J.L. Mumpower, P.J.M. Stallen
and V.R.R. Uppuluri (Springer, Berlin, 1985).
[120] WHO, Health future research, World Health Statics Quarterly
47(3/4) (1994).