See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/237089951

Policy analysis for sustainable development: The toolbox for the

environmental social scientist

Article  in  International Journal of Sustainability in Higher Education · January 2006

DOI: 10.1108/14676370610639236 · Source: OAI

CITATIONS READS
48 3,218

3 authors:

Hens Runhaar Carel Dieperink

Utrecht University Utrecht University
159 PUBLICATIONS   2,877 CITATIONS    104 PUBLICATIONS   1,738 CITATIONS   

SEE PROFILE SEE PROFILE

P.P.J. Driessen
Utrecht University
256 PUBLICATIONS   4,151 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

MYCOAST View project

Research for National Adaptation Strategy of the Netherlands View project

All content following this page was uploaded by Carel Dieperink on 13 November 2014.

The user has requested enhancement of the downloaded file.

 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1467-6370.htm

IJSHE
7,1 Policy analysis for sustainable
development
The toolbox for the environmental social
34 scientist
Hens Runhaar, Carel Dieperink and Peter Driessen
Environmental Studies & Policy, Copernicus Institute for Sustainable
Development and Innovation, Utrecht University, Utrecht, The Netherlands

Abstract
Purpose – The paper seeks to propose the basic competencies of environmental social scientists
regarding policy analysis for sustainable development. The ultimate goal is to contribute to an
improvement of educational programmes in higher education by suggesting a toolbox that should be
integrated in the curriculum.
Design/methodology/approach – Starting from the basic research questions regarding
governance for sustainable development, five methods are identified capable of answering the
following questions: reconstruction of policy theories; stakeholder analysis; impact assessment;
cost-benefit analysis; discourse analysis. Relevant information was collected through a literature
review and practical experience by the authors.
Findings – These include: minimum content of the toolbox with methods of policy analysis for
sustainable development; examples of how the toolbox can be applied; strengths and weaknesses of
the methods; specification of competencies of environmental social scientists active in the area of
policy analysis for sustainable development.
Practical implications – The paper proposes which methods of policy analysis for sustainable
development should be integrated in higher education as well as how this should be done.
Originality/value – The paper systematically analyses the implications of the policy context in the
domain of sustainable development for policy analysis in this area. In addition it gives a review of five
basic methods: how they can be used to analyse policy issues from a sustainable development
perspective. Aiming at reinforcing the input of scientific research in policy-making, the paper
eventually aims to promote sustainable development.
Keywords Sustainable development, Social policy, Environmental psychology, Higher education
Paper type Viewpoint

1. Introduction
The challenge of sustainable development
The Brundtland Commission defined sustainable development as development that
meets the needs of the present without compromising the ability of future generations
to meet their own needs (WCED, 1987). This entails various complexities that
policy-makers will have to deal with to attain this objective. The first complexity
resides in the need to optimise three values (i.e. ecological, social, and economic) at the
International Journal of Sustainability same time. It also resides in the observation that human behaviour (including public
in Higher Education policy) often affects each of these values simultaneously.
Vol. 7 No. 1, 2006
pp. 34-56 A second source of complexity is the need to operate in a multi-actor policy context.
q Emerald Group Publishing Limited
1467-6370
Typically, the resources required for successful policy implementation (e.g. money,
DOI 10.1108/14676370610639236 information, support) tend to be spread over actors other than (central) government
(Glasbergen, 1998). Some of these actors are found within the state apparatus, due to Policy analysis
the typical organisation of public administration into distinct policy domains. Other for sustainable
relevant actors are found in other domains of society (the market and civil society).
Given the state’s dependency on these actors, governments often have to negotiate with development
them and engage them in processes of policy development, implementation, and
evaluation (Rhodes, 1997; van de Riet, 2003). Yet the actors from various domains
typically perceive a given problem differently. Moreover, they have the potential to 35
block problem-solving activities. Policy-makers thus face the challenge of finding a
balance between policy alternatives that are both satisfactory in terms of
problem-solving and that can count on sufficient support from stakeholders
(Bovens et al., 2001).

Governing sustainable development

The multi-actor policy context implies that strategies for sustainable development
usually have to relate to a form of “governance”: a non-hierarchical form of steering,
where state and non-state actors participate in the formulation and implementation of
public policy (Rhodes, 1997). This steering model contradicts the traditional
hierarchical model of “government”. In the latter, the decisive power is centralised,
located in one policy-maker or group of policy-makers with similar objectives[1].
Governance has many faces. Depending on the type of actors that are (or have to be)
involved, a distinction can be made between “multi-level” governance (highlighting the
various tiers of government: local, regional, national, or supranational) and
“multi-actor” governance (reflecting the involvement of both public and private
actors) (Driessen, 2005). In both cases “multi-sector” governance is required as well:
the integration of environmental objectives in non-environmental policy domains
(Lafferty and Hovden, 2003). In practice, the various forms of governance have become
concrete in non-hierarchical arrangements such as “agreements”, “covenants”, or
“partnerships” between public and private parties and among market parties and civil
organisations.

Roles of environmental social scientists

In the context of governance for sustainable development, environmental social
scientists have to play a double role. Apart from facilitating multi-stakeholder
dialogues (Glasbergen and Smits, 2003), they have a more indirect role in
policy-making processes through their traditional role of information provider by
means of policy analysis. Policy analysis refers to “the use of analytical techniques and
knowledge for and in policy-making” (Parsons, 1995, p. xvi). Its aim is to support
policy-makers by producing and transforming “policy-relevant information” (Dunn,
1981; Fischer, 1997; van de Riet, 2003). Governance for sustainable development places
some specific demands on policy analysis. One of these is related to the observation
that often stakeholders either do not accept the research outcomes or they use these
outcomes in a strategic way (Bras-Klapwijk, 1999)[2]. In part, this can be attributed to
uncertainties that prevail in many policy areas and that cannot be addressed
adequately by science alone (Funtowicz and Ravetz, 1993). Another explanation is that
many policy-analytical studies are based on “single-value” methods that focus on one
or a few operational criteria (e.g. efficiency and effectiveness) and that largely ignore
other criteria or values (cost-benefit analysis (CBA) is a typical example). In this way
IJSHE the methods do not always reflect the values that are involved in a policy area.
7,1 This may provoke opposition from groups of stakeholders (de Jong, 1999; van de Riet,
2003). Policy analysis for sustainable development therefore requires specific
competencies.

Goal and structure of the paper

36 In this paper we focus on the policy-analytical competencies that environmental social
scientists are expected to possess and that should therefore be incorporated in higher
education. In particular, we identify which basic methods should be in the “toolbox” of
the graduates, we discuss the strengths and weaknesses of these methods, and we
consider what the multi-actor policy context implies for their application.
In Section 2 we present five methods that provide answers to the main policy
questions regarding governance for sustainable development. These methods are
described in more detail in Sections 3 through 7. In Section 8 we discuss how the
toolbox should be integrated in higher education. We offer some concluding remarks in
Section 9.

2. Basic methods of policy analysis in the context of governance for

sustainable development
Methods of policy analysis: definition
In general terms, policy analysis can be defined as “analysis for policy” (Parsons, 1995,
p. xvi). This definition is rather broad; it covers numerous methods from a wide range
of disciplines. In this paper we narrow it down by applying an additional criterion: the
research has to have a clear link with policy. This leaves out studies on, for instance,
the functioning of ecosystems commissioned by a department of agriculture. But it
includes research into the effectiveness of a policy that aims at restoring ecosystems in
a particular area.
Methods, in general terms, are ways to find answers to questions (the term
“method” is derived from the Greek word “methodos”, which literally means a way or
path of transit). In the search for answers several choices have to be made, e.g. on
research design, data collection, data analysis, and reporting. A method not only
specifies the research questions; it also describes a particular, coherent set of decisions
(or rules) that, according to certain objectives or criteria, will yield the best results
(Swanborn, 1987). Methods thus are standardised procedures for producing answers to
particular questions.
The terms “method” and “research strategy” are often used as synonyms. Yin (1994),
for instance, defines a research strategy as “a distinct way of collecting and analysing
empirical evidence” in a “logical sequence that connects the empirical data to a study’s
initial research questions and, ultimately, to its conclusions” (Yin, 1994, pp. 3, 19).
Typical research strategies used in social science include experiments, surveys,
histories, case studies, and analyses of archival information (Yin, 1994). Other authors,
however, consider methods as distinct types of research strategies or specific
operationalisations of elements thereof (Swanborn, 1987, p. 41). The boundaries
between methods and research strategies tend to blur. As a rule of thumb, we consider
research strategies as being more general than methods. We subsequently distinguish
methods from data collection techniques, which are specific ways to gather empirical
data (examples include interviews and observation). Often, a method can be based on,
or complemented by, multiple (alternative) data collection techniques. Again, the Policy analysis
boundaries between methods and data collection techniques are not always sharp for sustainable
(Figure 1).
development
Key research questions and methods of policy analysis for sustainable development
Identification of the basic methods of policy analysis in the domain of sustainable
development requires insight into the basic research questions in this field. In line with 37
Leroy and Nelissen (2000) we distinguish between five relevant themes around which
research questions can be formulated, namely:
(1) Policy content. Some examples of questions are: What are the policy objectives?
(e.g. What is meant by “decoupling” or “integrated policy”?); What problem
definitions underlie policy objectives? (e.g. Why is global warming
problematic?); and How valid are the assumptions upon which the policy
rests? (e.g. Is there any evidence that road pricing has a substantial impact on
travel behaviour?).
(2) Policy process. e.g. What is the influence of NGOs such as Greenpeace on
political agenda-setting? and How have policy processes evolved over time?
(3) Policy organisation. e.g. Which policy domains are involved in the issue of
sustainable development? and How is the implementation of EU environmental
directives organised in the various member states?
(4) Policy effects. e.g. Has the policy resulted in the realisation of its objectives? Are
there any (severe) side-effects? Do stakeholders evaluate the policy effects
similarly or not? and What explains success or failure of the policy under study?
(5) Policy context. e.g. How is the policy content affected by political, economic, and
cultural developments? (Think of the impact of the trend of
“mass-individualisation” on environmental pressure, or the place- and
time-specificity of particular policy problems, such as climate change).

Research questions

Research
strategy

Method Method

Answers
Data Data Data
collection collection collection
technique technique technique
Figure 1.
Research strategies,
methods and data
collection techniques in
perspective
Empirical data
IJSHE Environmental scientists should be equipped with a toolbox of methods that cover the
7,1 main questions related to the above themes. Over the years, numerous methods of
policy analysis have been developed (Geurts et al., 1989; Mayer et al., 2004).
However, in order to answer the types of questions listed above, the following five
methods form the minimal methodological equipment for environmental social
scientists:
38 (1) reconstruction of policy theory;
(2) stakeholder analysis;
(3) impact assessment;
(4) cost-benefit analysis; and
(5) discourse analysis.

Table I shows how the five methods relate to the above-mentioned themes. Sometimes
the methods overlap; both stakeholder analysis and discourse analysis, for instance,
provide insight into the problem perceptions of stakeholders (albeit from a different
perspective). In addition, the five methods are complementary. A reconstructed policy
theory and a stakeholder analysis, for instance, provide elements that can be used in an
IA in order to measure and explain the (side-) effects of a policy. In turn, an IA can be
an input for a CBA.
In the following sections, we briefly discuss each of these methods. We address:
.
themes that can be covered by the method;
.
stages involved in implementing the method;
.
its strengths and weaknesses from a scientific and pragmatic perspective (e.g.
reliability, validity, and costliness); and
. the recommended application in multi-actor policy contexts.

3. Characterisation of the five basic methods of policy analysis

Reconstruction of policy theory
Focus of the method. A policy theory can be defined as “the total of causal and other
assumptions underlying a policy” (Hoogerwerf, 1990, p. 285) and represents
“the conception of what must be done to bring about the intended social benefits”
(Rossi et al., 2004, p. 134)[3]. Reconstruction of a policy theory focuses on elucidating
the logic or reasoning behind a policy programme by examining (Hoogerwerf, 1990,
p. 286):

Theme 1 Theme 2 Theme 3 Theme 4 Theme 5

(content) (process) (organisation) (effects) (context)

Reconstruction of policy theory

Table I. Stakeholder analysis
Policy themes addressed Impact assessment
by the five methods of Cost-benifit analysis
policy analysis Discourse analysis
 .
Final relations. Relations between objectives and means. Policy analysis
. Causal relations. Relations between causes and effects, mainly related to the for sustainable
policy problem. For instance: the climate problem is partly based on development
the (assumed) impact of human activities on global warming.
.
Normative relations. Relations between principles and norms mutually or
between principles and norms on the one hand and existing or expected 39
situations on the other. Normative relations will, among other things, explain the
problem perceptions upon which the policy rests.

The method typically focuses on the content of a policy (theme 1 from Section 3).
It addresses questions such as the following: What argumentation underlies the policy
problem definition? How did the policy-makers envisage that the instruments chosen
would solve the problem? and What assumptions did they have? The answers to such
questions are not only interesting in themselves, but may also explain the success or
failure of a policy, i.e. its effects (theme 4). Often, incorrect assumptions explain why a
policy fails (Hoogerwerf, 1990). The method can also be used to explain other policy
aspects, such as the choice of instruments (the impact they are supposed to have, will
provide some justification) and the policy goal (which will be related to the perceived
policy problem). The next subsection provides some examples of research questions in
the domain of sustainable development.
A variant of the method for reconstructing policy theories, as elaborated by Rossi
et al. (2004), takes into consideration not only the policy content but also some
organisational aspects (theme 3):
.
How will the target population of a policy measure be reached? And how and
when will the programme be terminated?
.
Which programme resources, personnel, administration, and general
organisation are needed in order to implement the programme?

Policy theories and sustainable development. Reconstruction of a policy theory may be

used to answer questions such as the following:
. Which aspects of genetic manipulation of crops are considered a problem and
why?
.
To what extent is EU policy aimed at promoting biodiversity supported by
scientific knowledge?
.
Why is policy integration considered such an important condition for successful
planning and management of European coastal zones?
.
How is emission trading expected to result in a reduction of CO22 emissions in a
cost-efficient manner?
.
Which value underlies the policy programme: the principle of “considered
innocent until proven guilty” or the “better safe than sorry” principle?
.
What is the main point of application of EU policy programmes aimed at
reducing transport emissions: to change transport behaviour or to improve the
technical performance of vehicles? What arguments underlie this focus, and how
can this focus be evaluated in terms of effectiveness?
IJSHE Strengths and weaknesses. A strength of the method is that it is an important resource
for interpreting and explaining the quality of a policy as well as its effectiveness
7,1 (Hoogerwerf, 1990; Rossi et al., 2004)[4]. A more pragmatic strength is that the method
allows the parties to gain a basic understanding of a complex matter (i.e. a policy
programme) in a relatively short period of time.
However, reconstruction of policy theory overestimates the “rational”
40 characteristics of policy-making and underestimates aspects of politics, social
interaction, and conflicting rationalities and interests. In addition, the policy theory of
the political-administrative elite may differ from the policy theory of implementers who
put the policy into practice (Hoogerwerf, 1990). The method may fail to reveal to the
public ex post rationalisation of decisions that have been taken in an “irrational” way
(e.g. via compromises). Finally, the assessment of the quality of a policy theory needs
further elaboration (Hoogerwerf, 1990). Scientific standards (validity, reliability) may
not be appropriate in light of the “subjective” approach from which reconstruction of
policy theory starts. In addition, high-quality policy theories may be unsuccessful in
their implementation.
Recommended application in multi-actor policy contexts. The method largely
ignores stakeholders other than the political-administrative elites. If the method is
applied in order to gain insight into the chance of success of a policy programme,
and it is reasonable to expect that the public authorities in question cannot
command implementation of the policy in a top-down manner, the policy theory
will not be a sufficient source of information. This problem can be overcome by
combining the method with a stakeholder analysis (Section 5), which gives insight
into relevant stakeholders, their problem perceptions, and support for the policy
programme.
Stages. The main activities in reconstructing a policy theory include (Hoogerwerf,
1990, p. 288):
.
Collect statements from policy-makers and decision-makers about the policy at
issue and consider the social processes in the policy field with their inputs and
outputs.
.
Examine final relations by constructing a goal tree with ultimate goals,
intermediate goals, and means, and translate explicit goal-means relations into
(causal) hypotheses. Indicate the extent of certainty of the actor regarding these
final and causal relations (the chances of effects).
.
Trace explicit cause-effect relations and translate them into (causal) hypotheses.
.
Trace explicit normative relations and translate them into (causal) hypotheses.
.
Reconstruct the total of the (causal) hypotheses (steps 2, 3, and 4) to a coherent
total of causal hypotheses (the reconstructed policy theory).
.
(Option) Assess the quality of the policy theory (e.g. regarding its consistency,
validity of assumptions, expected effectiveness, or other quality aspects).
.
(Option) Formulate advice: how can the quality of the policy theory be improved?
If the method is applied in a multi-actor setting in order to support policy-makers
(e.g. for an ex ante assessment of the assumptions of a proposed policy), the
problem may be that stakeholders do not accept the outcomes of the study
(in particular if they do not recognise their own perceptions of the policy issue).
This problem could be overcome by stakeholder participation during the research. Policy analysis
(For an overview of various forms of stakeholder participation, see Mayer, 1997.) for sustainable
Stakeholders’ input can be used to set the boundaries of the policy under study or
to specify evaluation criteria. development

Stakeholder analysis 41
Focus of the method. Stakeholder analysis aims at eliciting the main actors that have a
stake or interest in a particular policy problem. Actors become stakeholders when they
contribute to a policy problem, are needed for solving the problem, or are affected by
problem-solving activities.
Usually, stakeholder analysis focuses on aspects such as the interests of
stakeholders, their perceptions of the problem, their positions and relationships with
other actors, and the resources that they control. The method offers answers to
research questions such as the following ones:
RQ1. How are stakeholders involved in the problem area? What are their interests
in solving or maintaining the problem? and How do they perceive the
problem?
RQ2. How relevant are these actors (in terms of critical resources: power, support,
authority, and the like)?
RQ3. Which policy alternative is supported by most stakeholders?
RQ4. What effects do or did the stakeholders expect from the policy programme or
its alternatives?
RQ5. How are the stakeholders related? Which (coalitions of) actors may support or
block problem-solving activities?
The answers to these questions shed some light on why the a priori most effective
solution available was not chosen. This may simply be due to lack of support. In
addition, a stakeholder analysis provides background information for understanding
the dynamics in policy processes. “Shifting” policy goals may be understood in the
context of stakeholders’ interests and power. A stakeholder analysis may also provide
an ex ante evaluation of the effectiveness of a policy programme by assessing
the extent to which the programme and the interests of stakeholders coincide
(Cörvers, 2001). Finally, and in line with this, a stakeholder analysis may be used in
order to improve a programme in terms of legitimacy as well as quality. Such
improvements can be made if the values, perceptions, or experiences of stakeholders
are incorporated (Grimble, 1998). The method therefore covers all the five themes of
policy-related questions listed in Section 3.
Stakeholder analysis can be conducted in a static way (a single snapshot) or in a
more dynamic way (by analysing changes in an actor network over a particular
period of time). Different levels of analysis can be employed. Traditional stakeholder
analysis focuses on attributes of individual stakeholders. (Policy) network analysis
focuses on networks of interdependent actors, giving special attention to
relationships and positions of actors in these networks (Kickert et al., 1997;
Rowley, 1997).
IJSHE Stakeholder analysis and sustainable development. A stakeholder analysis is an
7,1 adequate method for answering questions such as:
.
Which actors contribute to the problem of marine pollution and how? And which
actors control the critical resources (such as money, land, authority, or
information) required for reducing this problem?
. How important are consumers for undertaking measures that reduce depletion of
42 the ozone layer (as compared to, for instance, producers)? And what does this
imply for their involvement in decision-making on policy in this area?
.
What interests do the various actors active in coastal areas have regarding the
management of these areas? And what are the common interests in this
perspective?
.
How can the norms of “total allowable catches” (TAC) from EU Common
Fisheries Policy be understood in light of the stakeholders involved? i.e. is there a
link between the level of the TAC and the balance between proponents and
advocates of limiting fish catches?

An important issue is identifying the boundaries of the network (or group of

stakeholders) under study (Rowley, 1997). In many situations, the whole world
(including future generations) may have a stake in the policy problem under study
(e.g. exhaustion of natural resources). In part, the network will be demarcated on
practical grounds. Its boundaries will depend on the purpose of the study (e.g. to
explain policy choices or explore potential problems during policy implementation by a
certain agency).
Stages. In the literature, the following activities are usually distinguished in a
stakeholder analysis (Goverde and Nelissen, 2000; Grimble, 1998):
.
definition of the policy problem (or other topic that serves as the starting point of
the analysis);
.
identification of actors that are involved in the topic;
.
identification of formal tasks, competences, interests, goals, and problem
perceptions of the actors involved
.
inventory of relationships between stakeholders; and
.
identification of interdependencies between actors, e.g. by an inventory of
resources that stakeholders control.
Strengths and weaknesses. A strength of the method is its broad focus: it can be used to
address a wide range of questions (see above). The advantages of using the results of a
stakeholder analysis in policy processes have been described above. An additional,
pragmatic advantage is that a stakeholder analysis can be relatively cheap in terms of
money and time.
Some characteristics of stakeholder analysis and policy network analysis are the
following:
.
No uniform definition of stakeholders and policy networks is given in the
literature (Mitchell et al., 1997; van Tatenhove and Leroy, 1995).
.
Interpretation of data is subjective. Support for or objection to a particular policy
can hardly be measured.
 .
Although a stakeholder analysis may provide some explanation for policy Policy analysis
choices, the link between these choices and the stakeholder analysis is difficult to for sustainable
establish unambiguously. An option is to compare the “ideal” policy goal
formulation or the “optimal” policy instruments (in light of the fundamental development
causes of the policy issue) with the actual ones; any discrepancies may point to
compromises to stakeholders.
43
Recommended application in multi-actor policy contexts. A stakeholder analysis is
based on the assumption that the actors in the policy environment are important for
(understanding) policy-making. The method thus is well-suited for use in multi-actor
policy contexts. Stakeholder participation during the research process or interviews
with stakeholders are valuable for identifying relevant stakeholders and their interests
and perceptions. Not only does this source of information require less interpretation on
the part of the researcher, it may also result in a higher commitment of stakeholders to
the outcomes of the analysis.

Impact assessment
Focus of the method. IA covers a wide range of methods for determining the effects of a
policy programme (theme 4 from Section 3). The overall question that is addressed by
the method is, do policies actually produce the intended effects? Policy effects can be
measured in two ways. One is to measure the difference in a situation before and after a
policy intervention. The other is to assess the situation after the intervention with the
situation that would have occurred if the policy had not been implemented (the base
case or “business as usual” scenario). Figure 2 shows both approaches. The latter is
most informative about the actual policy effects. But it leads to practical difficulties, as
it is hard to know for certain how things would have been if the policy had not been
implemented (Mohr, 1992).
Impact assessment and sustainable development. Examples of questions that can be
subject to an IA include:
.
To what extent have EU waste disposal regulations (e.g. norms on separation of
hazardous and non-hazardous waste) resulted in a reduction of waste generation,
and to what extent have they resulted in illegal disposal of waste?
.
What have been the effects of the Swiss ton-kilometre tax on heavy freight
vehicles on traffic flows (number of hauls, vehicles employed, substitution by
rail, rerouting, etc.)?

forecasted trend
Outcome without policy
variable
effect

actual trend with policy

effect
Figure 2.
Before policy Policy intervention After policy intervention Two approaches for
Time measuring effects
IJSHE .
What would have happened with fish stocks in the North Sea if there would
7,1 have been no catch limitations (“total allowable catch”)? Would fish stocks
have been minimised, or would the sector have introduced catch limitations
voluntarily?

There are several variants of IA. Some variants focus on particular types of impacts
44 (e.g. social IA, environmental risk assessment, and environmental IA). In addition, a
distinction can be made between ex ante and ex post evaluation (depending on
whether or not the policy under study has been implemented). In the former case, not
the actual policy effects are assessed but the expected or forecasted effects. Looking
at the scientific rigour of these methods, which largely determines the validity of the
research results, a distinction can be made between two types of IA: randomised
experiments, and quasi-experimental research designs. Randomised experiments use
groups of participants that are randomly sorted into at least two groups. One is the
control group, and it does not receive an intervention (or a placebo); the other
(the intervention group) does. Outcomes are then observed and differences can be
completely attributed to the intervention. The main control condition of randomised
experiments is that participants are randomly assigned to the control and
intervention groups. In this way the groups are similar in terms of composition,
predispositions, and experiences; then the only thing that differs is the intervention.
All the remaining IA designs consist of non randomised quasi-experiments in which
the intervention group and control groups are only presumed to be similar (Rossi
et al., 2004). The control group may for instance consist of populations, firms, or
whatever from abroad, or the target group before the intervention. Designs using non
randomised controls universally yield less convincing results than well-executed
randomised field experiments (Rossi et al., 2004). Nevertheless, in practice
quasi-experiments are often conducted due to practical considerations like time,
money, or ethics.
Stages. Basically, four stages can be identified in conducting an IA (Rossi et al.,
2004):
(1) specification of variables that will be measured (policy goals and perhaps other
elements of a programme’s policy theory);
(2) selection and elaboration of a particular research design (experimental or
quasi-experimental; isolation of “business-as-usual” effect, etc.);
(3) operationalisation of variables and actual measurement; and
(4) interpretation of the data (establishment of explanations for findings;
separation of policy impact from exogenous factors).
Strengths and weaknesses. A general strength of IA is that it is explicitly aimed at
assessing causality and can thus be used to assess the effectiveness of policy.
In addition, IA has a rigorous basis; over the years the methodology has been refined in
order to maximise the validity and reliability of the method. Finally, ideally an IA
study isolates the relative impact of variables and in this way provides a good basis for
adjusting policy programmes.
There are also several weaknesses. One, in practice, is that randomised experiments
in policy studies are often impossible to conduct due to time or money restrictions or
ethical considerations. Another is that assessments cannot (always) be made with
certainty, only with varying degrees of plausibility (Rossi et al., 2004). An important Policy analysis
reason is that it is not always possible to assess the “business-as-usual” scenario for sustainable
against which the effects of the programme under study are assessed (Rossi et al.,
2004). Thirdly, IA often does not assess policy effects in the long term, and therefore development
the “sustainability” of policy effects is not always known. A more practical problem is
that goals are often vaguely defined, contradictory, or shifting during policy
implementation, which poses operationalisation problems for the researcher 45
(Verschuren and Zsolnai, 1998).
Recommended application in multi-actor policy contexts. Owing to its explicit focus
on policy outcomes, IA implicitly overestimates the goal orientation of policy-makers
and politicians and usually underestimates the process of policy-making. The intrinsic
value of developing a programme in collaboration is thus ignored. In addition, it
presumes a uniformity in goals and motives among the people involved. But that
assumption is not realistic, particularly not in multi-actor contexts or in situations
where during the implementation changes are made in order to cope with unforeseen
problems (Verschuren and Zsolnai, 1998). In addition, a focus on policy goals may
draw the attention of the researcher away from side-effects (e.g. injustice caused by the
policy) or from the basic question of whether or not the policy goals are appropriate in
light of social norms and values (Fischer, 1997; Verschuren and Zsolnai, 1998). These
characteristics make an IA in its classical form less appropriate for use in a multi-actor
context. Yet, this does not mean the method should be ignored. Its rigorous
methodological base makes it an adequate method for assessing policy effects. It is
more adequate than a stakeholder analysis, which can also be used to explore how an
ongoing policy programme may work out in future but which requires (more)
interpretation by the researcher. For a successful use in a multi-actor context (i.e. when
the results are to be accepted), some adaptations are required. One is that a stakeholder
analysis can precede an IA in order to provide an overview of the main actors that will
be faced with the effects of a policy and to show how they perceive that. These impacts
can be specified and quantified in an IA. In this way the (side-)effects for stakeholders
may become clear. A second adaptation is that stakeholder participation during the IA
study may result in a set of possible (side-)effects that are considered relevant by
stakeholders (this may for instance include the effects on collaboration between
government and other stakeholders). Stakeholders may also be a source of information
when it comes to the operationalisation (or estimation) of effects (e.g. by means of a
Delphi survey or focus group). Stakeholder participation may subsequently result in
commitment to the study results as well as to the problem-solving activity that is being
evaluated (Hisschemöller et al., 2001)[5].

Cost-benefit analysis
Focus of the method. CBA focuses on evaluating (i.e. attaching a value to) the effects of
(proposed) policy programmes (theme 4 from Section 3). It can be used to determine
whether or not the benefits of an investment or a policy outweigh its costs. CBA has a
very broad scope, since it may express all positive and negative effects of an activity in
a common unit, namely money, from a social as opposed to a firm’s point of view
(Wrisberg and Udo de Haes, 2002). Typically, costs and benefits are expressed in
consumer preferences, specifically in the willingness – to pay for the goods or services
that are evaluated.
IJSHE CBA can be used in order to find out whether or not policy choices have been
7,1 worthwhile, whether or not the most efficient policy programmes have been chosen, or
how similar, future programmes can be transformed into more efficient ones (Rossi
et al., 2004). This is usually done on an ex post basis (i.e. after policy implementation).
Ex ante analysis can be used to prioritise programmes according to their benefit-to-cost
ratio.
46 Cost-benefit analysis and sustainable development. A CBA could be conducted in
order to answer the following questions:
.
What policy measure for promoting reductions in CO2 emissions will yield the
highest social benefits per e, voluntary agreements with the industry, legal
norms, mass-media campaigns, or environmental taxes?
.
Which allocation of development aid to the various possible projects will result in
the highest benefits in terms of improved quality of life?
.
What has been the balance between the costs and benefits of construction of the
“Betuweroute” (rail link between the port of Rotterdam and the German
hinterland), not only in financial terms, but from an environmental perspective as
well (including costs such as visual annoyance, barrier effects, and noise
nuisance and benefits such as reduced emissions due to a substitution of road
haulage by rail transport)? To what extent do the benefits of the Betuweroute
accrue to other countries than the Netherlands, and what does this imply for
financing of this project?
Strengths and weaknesses. The main strength of CBA is that it allows for evaluating
various types of effects (tangible and intangible, direct and indirect) by translating
them into one (monetary) unit. In addition, over the years many sophisticated
procedures for CBA for specific applications have been developed. In the Netherlands,
for instance, a standardised procedure for conducting CBA has been developed which
is compulsory for every investment in transport infrastructure that is under
consideration (Eijgenraam et al., 2000; de Jong and Geerlings, 2003 for a discussion).
These standardised procedures prevent as much as possible methodological errors
such as double-counting and make sure that the outcomes of different CBAs are
comparable.
Stages. CBA always involves the following activities (Eijgenraam et al., 2000; Rossi
et al., 2004):
.
choice for a specific type of CBA (private profitability analysis or social CBA;
depending on the accounting perspective employed);
.
definition of project alternatives and the base case;
.
systematic identification of project effects;
.
estimation and valuation of project effects;
.
estimation of investment and development costs;
.
discounting of cost and benefits (calculation of net present value);
.
(option) variant and risk analysis; and
.
(option) adjustment of project alternatives.
CBA has several methodological as well as pragmatic weaknesses. For one thing, it is Policy analysis
very data intensive. Often, the unavailability or inaccuracy of data will place a serious for sustainable
limitation on the usefulness of the method (Rossi et al., 2004; Wrisberg and Udo de
Haes, 2002)[6]. By extension, when major costs or benefits are disregarded because development
they cannot be measured or monetised (e.g. the extinction of a particular animal or
income redistributional effects), the project may appear less or more efficient than it is
(Rossi et al., 2004). Therefore, such costs and benefits should be added to the list, albeit 47
in qualitative terms. Thirdly, CBA is not appropriate when a programme is not yet
beyond the development state or when uncertainty remains about the effects (Rossi
et al., 2004). Fourthly, the treatment of indirect effects is subject to debate, since it is
often difficult to assess to what extent they are included in the measurement of direct
effects (which raises the risk of double-counting). Finally, and more fundamentally, it is
questioned whether consumer preferences are a proper basis for the valuation of effects
on nature (e.g. the extinction of species) (van Wee, 2003; Fischer, 1997).
Recommended application in multi-actor policy contexts. CBA is well suited for
single-actor policy contexts because, in its traditional form, it has one target
(efficiency). The method ignores values other than efficiency, such as equity (welfare
distribution[7]) or the question of whether or not a policy has been effective, i.e. has
reached its targets or not (Fischer, 1997). The neglect of other values may endanger
support for the outcomes of a CBA. (This is particularly so when the CBA is an
important input to decision-making on investments, as in the case of transport
infrastructure in the Netherlands.) Yet this is not the only reason why CBA may be
problematic in multi-actor policy contexts. CBA is also a complex method and not easy
for non-economists to understand (Wrisberg and Udo de Haes, 2002; van Wee, 2003).
In addition, CBA overemphasises the importance of policy outputs as compared to the
process of policy-making (Fischer, 1997). Another difficulty is that not all effects can be
monetised (e.g. because market prices are lacking or because monetisation is subject to
criticism; think of quantifying the value of life).
In part, the above drawbacks can be overcome by methodological adaptations.
In cases where the benefits of a policy programme cannot be valued unambiguously,
cost-effectiveness analysis (CEA) offers a good alternative. CEA calculates which
policy option produces or produced the desired beneficial effects (e.g. reduction in CO2
emissions) at the lowest costs. Only the costs are monetised. Regarding its lack of
transparency to non-economists, clearer communication with stakeholders is
required[8]. The focus on costs and benefits (more generally, on policy outputs) can
be justified on the basis that taxpayers desire value-for-money. By means of
stakeholder participation, stakeholders can contribute to the definition of relevant
costs and benefits and to the way in which they are incorporated in the analysis
(operationalisation, monetisation or not)[9]. Finally, the equity consequences of a
project can be made visible by specifying costs and benefit redistributional effects for
various groups of stakeholders (for this purpose, a stakeholder analysis may precede
the CBA).

Discourse analysis
Focus of the method. Discourse analysis focuses on the ways in which (groups of)
actors give meaning to particular phenomena (e.g. a policy programme or the causes of
a policy problem). The method is based on the premise that the way people talk is not a
IJSHE neutral reflection of our world, identities, and social relations. Rather, it plays an
7,1 active role in creating and changing them. Within a particular worldview, some forms
of action become natural whereas others become unthinkable. Different social
understandings of the world therefore lead to different social actions (Jørgensen and
Phillips, 2002). Discourse analysis explores patterns in written or spoken statements as
well as related practices in order to identify the representations of reality that are
48 employed. It also explores the social consequences of different representations of
reality (Jørgensen and Phillips, 2002; Hajer, 2005).
The definition of problems is not taken for granted. In fact, the aim of discourse
analysis is to elucidate the various problem perceptions involved in an issue
(Hajer, 2005). Thus, the method can be used to reconstruct visions on (social) problems
as well as the context in which they emerged. Discourse analysis can also shed light on
controversies about certain issues. It can explain why a particular understanding of a
problem at some point gains dominance and is seen as authoritative, while other
understandings are discredited (Hajer, 1995)[10].
Discourse analysis and sustainable development. Examples of questions that could
be or have been addressed by discourse analysis:
.
What was the debate on acid rain really about – the causes of this form of
environmental degradation or the seriousness of the problem, or about the way
politics is conducted? (Hajer, 1995)
.
How is “overpopulation” perceived by different groups of people and, in light of
these worldviews, what measures are suggested to reduce the negative effects
associated with this phenomenon?
.
How do stakeholders define “quality of the living environment” and what are
their demands from this perspective regarding town planning and urban
environmental policy?

There are two distinct forms of discourse analysis:

(1) The linguistic-oriented tradition, which focuses on language and what language
is used for (Georgakopoulou and Goutsos, 1997).
(2) The broader tradition, focusing on discourses in a more general sense. Here, the
analysis focuses both on the ways of thinking and arguing on specific (social)
themes or issues and on the related practices, structures, and institutions
(Hajer, 2005).

In the area of policy analysis, the latter form of discourse analysis dominates. It can be
used to answer questions such as the following: How have policy processes evolved
and why? What arguments underlie the decisions made? What meanings, arguments,
and lines of reasoning did the various (groups of) actors involved bring forward? What
advocacy coalitions can be discerned? And what were controversies or conflicts about?
These questions can be grouped under the themes 1, 2, 4, and 5 from Section 3, which
illustrates that the method is suitable for a broad set of questions.
Strengths and weaknesses. A discourse analysis can reveal opinions and attitudes
and underlying value judgements. Thus, it is very appropriate for analysing the
subjectivity of certain texts or sources. For this purpose, various standardised
procedures have been developed in the linguistic-oriented tradition of discourse
analysis[11]. Another strength of the broader tradition of discourse analysis is that it Policy analysis
can structure complex scientific or political debates. However, in this tradition a for sustainable
standardised procedure for discourse analysis in policy issues is largely lacking, which
results in the danger of subjectivity. The researcher therefore should be explicit and development
clear in his or her selection of material and interpretation (Burman, 2003). In addition,
the analyst should try to triangulate as much as possible.
The limited generalisability of the results of the discourse analysis can be regarded 49
as another weakness. The post-modern tradition in which discourse analysis fits
suggests the absence of universal “truths”; representations of the world are therefore
unique. This implies that the opportunities to draw (policy) lessons from discourse
analyses are by definition limited. Apart from this, according to the post-modernists,
there are no criteria by which to distinguish the “good” research from the “bad”.
The validity criterion commonly employed in science is not applicable in discourse
analysis. The reason is that it is based on the assumption that knowledge can reflect
reality without bias, an assumption that is rejected a priori by discourse analysis.
More practical questions include the following: at what level does one define a
discourse? How can one delineate single discourses? and How does one identify
discourses that compete in the same domain? (Jørgensen and Phillips, 2002) A related
issue is how to determine when a particular discourse is dominant.
Stages. Although discourse analysis in the broader tradition has no standardised
procedure, the first stage in a discourse analysis is to define the research topic
(commonly a particular issue or “nodal point”, such as acid rain or climate change). The
next two main stages in a discourse analysis are description of the discourse and
explanation of what was found. The following activities often are part of the
description stage (Hajer, 2005):
.
Desk research. In order to prepare a first chronology and a first reading of events.
.
“Helicopter interviews”. Interview three or four actors with an overview of the
field.
.
Document analysis. Analysis for structuring concepts, ideas, and categorisations
in order to get a basic notion of the process of events as well as the main sites of
debate.
.
Interviews with key players. To generate more information on causal chains and to
get a better understanding of the meaning of particular events for the interviewees.
.
Analysis of debates. Who said what to whom and what were the effects? One
could examine sites of argumentation (e.g. parliamentary debates) to account for
argumentative exchanges, or analyse for “positioning effects” (actors may try to
force others to take on a particular role, which they may refuse to do once
aware of it).
.
Identification of key incidents. Events that are essential to understand the
discursive dynamics in the case under study.
.
Analysis of practices in particular cases of argumentation. To see if the meaning
of what is being said can be related to the practices in which it was said.
.
Interpretation. What was the discursive order that governed a particular domain
in a particular period of time?
.
Second visit to key actors. Do they recognise the reconstructed discourse?
IJSHE Recommended application in multi-actor policy contexts. The method is appropriate for
7,1 studying policy-making processes in multi-actor policy contexts. That is because it
presumes the existence of different perceptions, opinions, and, more generally,
discourses that relate to a multitude of actors. Since discourse analyses are
primarily conducted for scientific reasons, the problems with policy analysis in
multi-actor contexts described in Section 2 are not typical for discourse analysis.
50 The outcomes of discourse analyses do not have to be accepted by stakeholders.
But it is prudent to discuss them with those stakeholders whose opinions
are examined in order to check if the analysis of the discourse makes sense
(Hajer, 2005).

4. Integration of “the toolbox” in educational programmes

After having argued why the preceding five methods of policy analysis should be in
the toolbox of graduates in environmental studies, we will now discuss what students
should know about these methods and how that knowledge can be taught. We start
with the technical skills and social competencies that we think the students should
acquire.

Relevant competencies
Technical skills are based on what we consider to be the three core activities of
policy analysts, namely to analyse (i.e. by systematic examination), explain (i.e. to
identify determinants or factors that have had an influence on the phenomenon
under study), and evaluate (i.e. to make normative judgements about findings). The
five methods are useful tools for conducting these activities; students should
therefore be trained in applying (elements of) the methods in specific policy
situations. In this light students should learn how to characterise the various
methods of policy analysis. Some relevant questions that should be addressed are the
following: what answers does the method provide and how can these be used in
policy-making processes? What are its underlying assumptions and what do they
imply for the robustness of the conclusions? To what extent is the method suited for
use in multi-actor policy contexts? How is it related to other methods of policy
analysis? What are its strengths and weaknesses? The relevance of these questions is
twofold. Firstly, they help students select a method for a particular research
question. Secondly, being able to characterise a method provides a basis for
recognising and classifying other methods of policy analysis with which they have
not yet been acquainted. Looking at such methods will (hopefully) lead to the
conclusion that many of the methods of policy analysis that have been presented
over the years are not so different at all. Indeed, they can be considered variants of
one of the five methods discussed in this paper.
Environmental social scientists involved in policy analysis should have several
social competencies as well. They should be able to recognise the social context in
which their work takes place and, based on that, draw conclusions regarding their
contribution to the policy-making process. For instance, they should be able to
provide advice: to draw policy-relevant conclusions and recommendations from
research outcomes. For this purpose they need to know the goals of policy-makers
and the constraints they face. Apart from this, they should know which stakeholders
play an important role in the policy domain in question, and what that means for
their involvement in the project. A brief stakeholder analysis therefore should be part Policy analysis
of any policy-analytical study. The ways in which policy analysis studies can be for sustainable
suited to the multi-actor policy context in which they are conducted is described
elsewhere in this paper. A second social competence that environmental social development
scientists should acquire is the ability to reflect on their own role and work. They
should be aware of, make explicit, and avoid as much as possible subjective elements
in their analysis. They should know how policy analysis studies can be (ab)used in 51
policy processes and in what situations these studies may or may not have an
impact on policy-making.

How can these competencies be developed?

We believe that the best way to teach students how, why, and when to employ
a particular method is by practical experience (i.e. “learning-by-doing”). After all, a
method is a tool, something students must be able to handle in concrete situations.
For instance, reconstructing goal-means relations when reconstructing a policy theory
(Section 3) may seem obvious. Yet means are often subgoals for other means.
By stimulating students to elaborate on such issues, we hope not only that they will
apply the method better, but also come up with new insights (e.g. that goals and means
can often only be distinguished analytically).
Practical experience can be acquired by means of assignments in which students
have to apply (elements of) a particular method (e.g. a simple stakeholder analysis),
design a research set-up based on a method, or critically comment on a study in which
the method in question was employed. The empirical basis for the assignments could
be EU environmental law and policy regarding climate change, waste management,
marine pollution, etcetera – all topics on which ample data are available via the
internet. In order to stimulate debate and transferral of knowledge, it can be useful to
organise practicals in which students have to present their work to their colleagues.
The students who took our course in policy analysis for sustainable development in the
autumn of 2004 at Utrecht University evaluated the “learning-by-doing” character of
the course as very positive[12].

5. Conclusions
In this paper we argued that policy-makers who are responsible for promoting
sustainable development operate in a highly complex policy environment. One of the
sources of this complexity is the fact that policy-makers typically need support from
stakeholders elsewhere within the government or within civil society or the market.
Environmental social scientists conducting policy analysis in this context often need to
produce knowledge that is scientifically valid, relevant to the policy debate, and
accepted by stakeholders. These issues have to be considered when pursuing policy
analysis for sustainable development.
In the preceding sections, we discussed five methods of policy analysis with which
environmental social scientists should minimally be equipped in the above context:
reconstruction of policy theories, stakeholder analysis, IA, CBA, and discourse
analysis. By means of these methods, the main questions related to governance of
sustainable development can be answered. These questions relate to five themes:
policy content, policy process, policy organisation, policy effects, and policy context.
The five methods of policy analysis discussed in this paper can provide students with a
IJSHE proper understanding of existing policies in relation to their social contexts. The
7,1 methods can also provide tools for evaluating and improving these policies.
Furthermore, they allow students to pay attention to the threats and opportunities
posed by the policy context, and in this way allow them to make a positive contribution
to the promotion of sustainable development.
In this paper we have restricted ourselves to the policy domain of sustainable
52 development. It should be noted however, that the shift from “government” to
“governance” in public policy processes, which is an important cause of the complexity
in this policy area, is not unique to the domain of sustainable development. Yet in this
policy domain it is very visible. Since the trend from government to governance has
had important repercussions on policy analysis, as we argued, the five methods of
policy analysis that we have discussed can probably be denoted as key methods in
other policy domains as well.
On the other hand, there may be situations, in which the shift from government
to governance is still in an early stage, caused by cultural or other factors. Think,
for instance, of a country such as Bhutan, where policy is directed towards
sustainable development but where policy-making still is traditional and hierarchical
and where civil society is largely lacking. In these circumstances some of the
methods of policy analysis described in this paper will not be of direct importance –
think for instance of stakeholder analysis and discourse analysis. Yet the dominant
tendency worldwide is that sustainable development is pursued through
partnerships and other forms of governance. Owing to the global character of
sustainable development we therefore expect that eventually governance will replace
government. The tools of stakeholder analysis and discourse analysis therefore
should be kept in the toolbox.
We did not explicitly discuss the use and role of knowledge in policy processes.
We realise that knowledge that is generated by policy analysis may be an input to
these processes. However, there is no guarantee that the knowledge will actually be
used. Yet, when explicit attention is paid to the characteristics and demands set by
the multi-actor policy context, in the way we have described in this paper or
otherwise, we believe that the chance of a productive use of knowledge will be
greater in comparison to more traditional forms of policy analysis. This effect in our
view will be further enhanced, if students are trained to apply the methods in
concrete policy situations (see our suggestions in Section 8). Eventually, this will help
environmental social scientists in making a contribution to the promotion of
sustainable development.

Notes
1. Synonyms for government versus governance include single-actor policy context versus
multi-actor context (van de Riet, 2003), hierarchies versus policy networks (Kenis and
Schneider, 1991), and the linear model of policy-making versus the political model of
policy-making (Bovens et al., 2001).
2. This became very clear during the recent parliamentary investigation on large transport
infrastructural projects in the Netherlands (Autumn 2004).
3. In the literature, several synonyms for policy theories can be found, including impact model
and theory of action (Bressers and Hoogerwerf, 1998), logic model, programme model,
outcome line, and cause map (Rossi et al., 2004).
 4. We noted earlier in this paper that policy failure can often be accounted for, at least in part, Policy analysis
by the fact that policies are based on incorrect assumptions.
for sustainable
5. See for example Hisschemöller et al. (2001) or van der Sluijs and Kloprogge (2001) for
experiences with stakeholder participation in integrated assessment, a specific type of IA. development
6. This is a particularly controversial area in ex ante analyses (Rossi et al., 2004).
7. Not only does CBA ignore redistributional effects, but due to its focus on consumer
preferences and “willingness – to pay”, people with a high income have a higher weight in 53
CBAs than low-income people (van Wee, 2003).
8. “Narrative valuation” has proven to be an adequate alternative way of presenting the
outcomes of willingness-to-pay studies (Satterfield et al., 2000).
9. Its theoretical basis, however, does not allow the attachment of weights to costs and benefits,
as in the case of multi criteria analysis (MCA). The reason is the welfare-economic basis of
CBA, which assumes that the optimal allocation of scarce resources depends on the extent to
which consumer preferences, for which a willingness to pay exists, are satisfied. In this
context, each unit of money has the same weight.
10. In terms of discourse analysis, policy-making can be considered as “a system of competing
discourse coalitions and their struggles to ‘control shared meanings’ and to gain acceptance
of their framing of a policy issue” (Durning, 1995, p. 103).
11. Specific conventions exist for interpreting transcripts; see for instance Georgakopoulou and
Goutsos (1997).
12. For details about this course as well as the Master Programme of Sustainable Development
at Utrecht University under which this course falls, see www.internationalmasters.uu.nl.

References
Bovens, M.A.P., Hart, P.’t, van Twist, M.J.W. and Rosenthal, U. (2001), Openbaar Bestuur: Beleid,
Organisatie en Politiek (Public Administration: Policy, Organisation, and Politics; in Dutch),
6th ed., Kluwer, Deventer.
Bras-Klapwijk, R. (1999), “Adjusting life cycle assessment methodology for use in public policy
discourse”, PhD dissertation, Delft University of Technology, Delft.
Bressers, J.Th.A. and Hoogerwerf, A. (1998), “Het beleid en het beleidsveld”, Overheidsbeleid.
Een inleiding in de beleidswetenschap (Policy and the Policy Context), 6th ed., Alphen aan
den Rijn, Samsom, pp. 50-69 (in Dutch).
Burman, E. (2003), “Discourse analysis means analysing discourse: some comments on Antaki,
Billig, Edwards and Potter ‘Discourse analysis means doing analysis: a critique of six
analytical shortcomings’”, Discourse Analysis online, available at: http://extra.shu.ac.uk/
daol/ (accessed 27 May 2004).
Cörvers, R.J.M. (2001), “Netwerksturing bij natuurontwikkeling. Bestuurskundige
consequentieanalyse van gebiedsgerichte projecten” (“Network steering in nature
development. Public administrative impact assessment of area-specific projects”), PhD
dissertation, Utrecht University, Shaker Publishing, Utrecht (in Dutch).
de Jong, W.M. (1999), “Cost-effective use of evaluation models: an empirical cross-national
investigation”, International Journal of Technology Management, Vol. 19 Nos 3-5,
pp. 368-83.
de Jong, M. and Geerlings, H. (2003), “Exposing weaknesses in interactive planning: the
remarkable return of comprehensive policy analysis in The Netherlands”, Impact
Assesment and Project Appraisal, Vol. 21 No. 4, pp. 281-91.
IJSHE Driessen, P.P.J. (2005), “Restructuring the Dutch countryside: limits of a governance strategy”,
Planning, Practice & Research, Vol. 20 No. 1, pp. 69-77.
7,1
Dunn, W.N. (1981), Public Policy Analysis, Prentice-Hall, London.
Durning, D. (1995), “A review of Fischer and Forrester (1993) the argumentative turn in policy
analysis and planning”, Policy Sciences, Vol. 28 No. 1, pp. 102-8.
Eijgenraam, C.J.J., Koopmans, C.C., Tang, P.J.G. and Verster, A.C.P. (2000), Evaluation of
54 Infrastructural Projects; Guide for Cost-Benefit Analysis (Part 1: Main Report),
CPB Netherlands Bureau for Economic Policy Analysis/Netherlands Economic Institute,
The Hague/Rotterdam.
Fischer, F. (1997), Evaluating Public Policy, Nelson-Hall Publishers, Chicago, IL.
Funtowicz, S.O. and Ravetz, J. (1993), “Science for the post-normal age”, Futures, Vol. 25 No. 7,
pp. 739-55.
Georgakopoulou, A. and Goutsos, D. (1997), Discourse Analysis, An Introduction, Edinburgh
University Press, Edinburgh.
Geurts, J.L.A., Hart, S.L. and Caplan, N.S. (1989), “Beleidsanalytische technieken van
sociaal-wetenschappelijk onderzoek” (“Methods of policy analysis in social science”), in
Geurts, J. and Vennix, J. (Eds), Verkenningen in Beleidsanalyse. Theorie en praktijk van
modelbouw en simulatie, Kerkebosch, Zeist, pp. 345-61 (in Dutch).
Glasbergen, P. (Ed.) (1998), Co-operative Environmental Governance. Public-Private Agreements
as a Policy Strategy, Kluwer, Dordrecht.
Glasbergen, P. and Smits, R. (2003), “The policy laboratory for sustainable development. A new
learning context for environmental scientists”, International Journal of Sustainability in
Higher Education, Vol. 4 No. 1, pp. 57-74.
Goverde, H.J.M. and Nelissen, N.J.M. (2000), “Milieubeleid in netwerken”, (Environmental policy
in networks; in Dutch), in Driessen, P.P.J. and Glasbergen, P. (Eds), Milieu, Samenleving en
Beleid, Elsevier Bedrijfsinformatie, The Hague, pp. 119-39.
Grimble, R. (1998), Stakeholder Methodologies in Natural Resource Management. Socio-Economic
Methodologies, Best Practices Guidelines, Natural Resources Institute, The University of
Greenwich, Chatham.
Hajer, M.A. (1995), The Politics of Environmental Discourse. Ecological Modernization and the
Policy Process, Clarendon Press, Oxford.
Hajer, M.A. (2005), “Coalitions, practices and meaning in environmental politics: from acid rain to
BSE”, in Howarth, D.R. and Torfing, J. (Ed.), Discourse Theory in European Politics,
Palgrave, London, pp. 297-315.
Hisschemöller, M., Tol, R.S.J. and Vellinga, P. (2001), “The relevance of participatory
approaches in integrated environmental assessment”, Integrated Assessment, Vol. 2
No. 2, pp. 57-72.
Hoogerwerf, A. (1990), “Reconstructing policy theory”, Evaluation and Program Planning, Vol. 13
No. 3, pp. 285-91.
Jørgensen, M. and Phillips, L. (2002), Discourse Analysis as Theory and Method, Sage, London.
Kenis, P. and Schneider, V. (1991), “Policy networks and policy analysis: scrutinizing a new
analytical toolbox”, in Marin, B. and Mayntz, R. (Eds), Policy Networks. Empirical
Evidence and Theoretical Considerations, Campus Verlag/ Westview Press, Boulder, CO,
Frankfurt, pp. 25-62.
Kickert, W.J.M., Klijn, E.-H. and Koppenjan, J.F.M. (1997), “Introduction: a management
perspective on policy networks”, in Kickert, W.J.M., Klijn, E-H. and Koppenjan, J.F.M.
 (Eds), Managing Complex Networks. Strategies for the Public Sector, Sage, London, Policy analysis
pp. 1-13.
Lafferty, W.M. and Hovden, E. (2003), “Environmental policy integration: towards an analytical
for sustainable
framework”, Environmental Politics, Vol. 12 No. 3, pp. 1-22. development
Leroy, P. and Nelissen, N.J.M. (2000), “Milieubeleid en beleidswetenschappen”
(“Environmental policy and policy science”), in Driessen, P.P.J. and Glasbergen, P.
(Eds), Milieu, Samenleving en Beleid, Elsevier Bedrijfsinformatie, The Hague, pp. 71-93 55
(in Dutch).
Mayer, I. (1997), “Debating technologies. A methodological contribution to the design and
evaluation of participatory policy analysis”, PhD dissertation, Tilburg University Press,
Tilburg.
Mayer, I.S., van Daalen, C.E. and Bots, P.W.G. (2004), “Perspectives on policy analyses: a
framework for understanding”, International Journal of Technology, Policy and
Management, Vol. 4 No. 2, pp. 169-91.
Mitchell, R.K., Agle, B.R. and Wood, D.J. (1997), “Toward a theory of stakeholder identification
and salience: defining the principle of who and what really counts”, The Academy of
Management Review, Vol. 22 No. 4, pp. 853-86.
Mohr, L.B. (1992), Impact Analysis for Program Evaluation, Sage, Newbury Park, CA.
Parsons, W. (1995), Public Policy: An Introduction to the Theory and Practice of Policy Analysis,
Edward Elgar, Aldershot.
Rhodes, R.A.W. (1997), Understanding Governance. Policy Networks, Governance, Reflexivity and
Accountability, Open University Press, Buckingham.
Rossi, P.H., Lipsey, M.W. and Freeman, H.E. (2004), Evaluation, a Systematic Approach, 7th ed.,
Sage, Thousand Oaks, CA.
Rowley, T.J. (1997), “Moving beyond dyadic ties: a network theory of stakeholder influences”,
The Academy of Management Review, Vol. 22 No. 4, pp. 887-910.
Satterfield, T., Slovic, P. and Gregory, R. (2000), “Narrative valuation in a policy judgment
context”, Ecological Economics, Vol. 34 No. 3, pp. 315-31.
Swanborn, P.G. (1987), Methoden Van Sociaal-Wetenschappelijk Onderzoek, (Social Science
Methodology), Meppel, Boom (in Dutch).
van de Riet, O. (2003), “Policy analysis in multi-actor policy settings. Navigating between
negotiated nonsense and superfluous knowledge”, PhD dissertation, Delft University of
Technology, Eburon Publishers, Delft.
van der Sluijs, J. and Kloprogge, P. (2001), “The inclusion of stakeholder perspectives in
integrated assessment of climate change”, in Decker, M. (Ed.), Interdisciplinarity in
Technology Assessment. Implementations and their Chances and Limits, Springer, Berlin,
pp. 199-214.
van Tatenhove, J. and Leroy, P. (1995), “Beleidsnetwerken: een kritische analyse”
(“Policy networks: a critical assessment”), Beleidswetenschap, Vol. 9 No. 2, pp. 128-45
(in Dutch).
van Wee, B. (2003), Transportbeleid en logistieke organisatie. Een technisch-bestuurskundige visie
op kosten-batenanalyse, (Transport Policy and Logistical Organisation. A Technology,
Policy, and Management Perspective on Cost-Benefit Analysis), inaugural address, Delft
University of Technology, Delft (in Dutch).
Verschuren, P.J.M. and Zsolnai, L. (1998), “Norms, goals, and stakeholders in program
evaluation”, Human Systems Management, Vol. 17 No. 2, pp. 155-60.
WCED (1987), Our Common Future, Oxford University Press, Oxford.
IJSHE Wrisberg, N. and Udo de Haes, H.A. (Eds) (2002), Analytical Tools for Environmental
Design and Management in a Systems Perspective, Kluwer Academic Publishers,
7,1 Dordrecht.
Yin, R.K. (1994), Case Study Research. Design and Method, 2nd ed., Sage, Thousand
Oaks, CA.

56 Corresponding author
Hens Runhaar can be contacted at h.runhaar@geog.uu.nl

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints

View publication stats