Inception Report 1

Inception Report
for
Developing An Approach Paper as a Key Step Towards Producing DAC Guidance
for Evaluating Conflict Prevention & Peacebuilding Activities
Submitted to the DAC Networks on Development Evaluation and on
Conflict, Peace and Development Cooperation by
Collaborative for Development Action (CDA), June 2006
(See also Steering Group Comments in Annex 4, July 2006)

I. INTRODUCTION

The field of conflict prevention and peacebuilding (CPPB) is undergoing rapid growth
and change. This is, therefore, an opportune time for the DAC Networks to develop
guidance on the issues surrounding the evaluation of CPPB activities. As recently as five
years ago, attempts to define CPPB evaluation methodologies would have been too
narrowly focused for today’s realities.

Starting ten years ago but accelerating greatly in the past three to five years, the
peacebuilding field has expanded rapidly, both in terms of the numbers and types of
actors and in terms of the range of activities that are considered as peace practice. These
days, participants in CPPB conferences include experts in the security sector, rule of law,
human rights, DDR, humanitarian assistance and development cooperation, as well as
peace and conflict resolution specialists. Such participants also range from grassroots
activists to international diplomats. These new specialists also operate from a broad range
of institutions, including private companies and contracting agencies, local NGOs and
INGOs, UN agencies with various mandates, multilateral institutions, and a widening
array of donor government departments, including, at a minimum, units addressing
foreign policy, development and defense.

The expansion of actors and programming approaches reflect significant differences that
have also emerged in the types of conflicts where international efforts are undertaken. As
often noted, many current conflicts take place within, rather than between, nations and
are fought among multiple sub-groups. Command structures of armed groups are often
loose and volatile, exercising less control over foot soldiers and their actions than formal
armies.

As a result of these trends, a great deal more CPPB programming is now focused at the
grassroots level and among civil society actors than was previously the case. In the early
1990s, the preponderance of peace work focused on high level negotiations, often
including representatives of governments in conflict with each other. Today, multi-level,
dispersed and often unconnected peace programmes proliferate simultaneously and
involve many sectors of conflicting societies. These added dimensions of peace processes
are sometimes helpful and sometimes not—but have not been assessed thoroughly. The
“peacebuilding palette” presented in the Utstein Report clearly reflects the expansion of
what is considered peace programming in recent years.
Inception Report 2

These realities pose several challenges to the evaluation of CPPB activities. The first
challenge is uncertainty regarding the boundaries of CPPB work. The many new actors
with multiple perceptions, different histories and varying mandates use the same terms
but with divergent meanings. There is no widely-accepted agreement regarding what is,
and is not, a CPPB activity.

A second challenge has to do with multiple assumptions, implicit theories and “fuzzy”
goals. Many of today’s variety of CPPB activities are based on unclear or unproven
assumptions that reflect implicit theories of change, or theories of how peace is built.
Multiple actors implementing programmes in the same area, each employing his/her own
assumptions/theories can complicate efforts to test assumptions. A fundamental role for
evaluation is to test assumptions in order to determine which are valid in the context.

Directly linked to the issue of assumptions and theories is the question of goals. Again,
the full range of actors is pursuing a plethora of activities based on (sometimes
conflicting!) goals. In CDA’s experience, the evidence suggests that, in many instances,
the goals themselves are neither clear nor connected logically (through assumptions and
theory) to a strategy for achieving desired change. Attempts to evaluate effectiveness and
to determine how to account for change (or the lack of it!) are encumbered by the nexus
of assumptions, theories and goals among multiple actors.

And a third challenge concerns varying emphasis on programme processes as opposed to

programmatic impacts. Some actors or programmes assert that their positive contribution
is assured simply by the initiation of a process (of dialogue, reconciliation, relationship
building, negotiation, etc.). Others undervalue such processes, looking only for clear
impacts of activities on larger social systems. Guidance for evaluation should recognize
and attempt to address this tension in the field.

CDA welcomes these challenges and the opportunity to work with the DAC Networks on
disentangling these threads and sorting out what the accumulating experience and
evidence can tell us about how to address these challenges.

II. CDA’S METHODOLOGY

In all of our work, CDA is committed to evidence-based learning. Our approach is always
to gather broad experience and, by comparing and analyzing it, to identify common
themes, issues and lessons across a range of contexts. We do this in collaboration with
colleagues who are themselves experienced in the fields where we work.

As we develop an approach paper for guidance on CPPB evaluation, we also intend to be

experience- and evidence-driven. Given the timing and budget of the DAC Networks’
TORs, we will focus on gathering experience through three basic mechanisms:

ƒ Focused interviews
ƒ Document review and analysis
ƒ Learning from CDA’s projects
Inception Report 3

Focused Interviews

The TORs specify that we should conduct 15-20 in-depth interviews (see Annex 4
references to additional interviews). In preparation for the writing of this Inception
Report, we have done four selected interviews. Our selection for the additional
interviewees (after the Inception Report) will be based on a) identification of categories
of people that represent a range of perspectives on the issues raised by the TORs and b)
consultations with colleagues about individuals within these categories who are available
and would be interested in contributing to the DAC learning. We also hope that the DAC
Networks will provide feedback to our selection when receiving this Inception Report
and make additional suggestions.

The five categories of people we have identified as important to interview are:

1. Members of the DAC steering group (for this effort) and selected
members of the Evaluation and CPDC networks (government and
multilateral donors): From this group, we seek both advice on issues that
each member feels should be addressed in the approach paper, and we also
hope to discuss the full range of issues and experiences that we will include in
all other interviews.
2. Other donor representatives (beyond the DAC networks): From this
group, we will invite especially ideas on a) unclear/unresolved issues that
need further attention; b) the use and usefulness of CPPB evaluations; and c)
how their donor and governmental CPPB policies affect their work and its
effectiveness.
3. Implementers/Practitioners (NGOs): With this group, we will focus on
questions that have to do with effectiveness, strategy and substance of
programming and how to assess the outcomes and impacts of these.
4. Evaluation Specialists (beyond DAC networks, private and/or within
NGOs): Here we will concentrate on their assessments of the state of the art
of CPPB evaluations, with particular attention to how CPPB evaluations differ
from evaluations in other fields and their ideas of the most appropriate (or
inappropriate) methodologies for CPPB evaluations and why.
5. Academics/Others: There is a small group (see attached list of potential
interviewees) of others who are close to the fields of CPPB and who are good
observers of how things progress in the field as a whole. With this group, we
will explore all the questions and others that they introduce.

The individuals we have identified to interview are included in Annex 1. The types of
questions we have developed as guides to our interviews are also attached in Annex 2.
Note that these questions do not constitute a “questionnaire.” Rather, these are meant to
guide an in-depth conversation.

The purpose of these interviews, cumulatively, is to survey the field of opinion and
experience regarding CPPB evaluations and the most salient and pressing issues to be
Inception Report 4

dealt with in these evaluations. Also, we will use these as one basis for gaining a sense of
which methodologies have proven most (or least) useful.

Review of Documents

We will not duplicate or repeat the document reviews done for the Utstein Report and by
FAFO and others, but will build on their helpful findings. To a great extent, the TORs
have already identified the areas, highlighted in both previous reports, that deserve
additional sorting and clarifying (see Annex 5, ToRs). We are identifying additional
documents to review, because they appear to be closely related to these remaining
questions posed in the TORs and will provide another source of evidence for addressing
these issues. Most of these have been produced in recent months. The categories of
documents we are reviewing include:

1. Conflict sensitivity literature, including PCIAs and other selected conflict

analysis tools and reflections on their usefulness.
2. Recent conflict prevention and peace building literature, particularly those
items that analyze the boundaries of the fields and evidence of varying
approaches. These will be helpful in clarifying definitional issues.
3. Evaluation literature, including
a. Most significant change literature
b. Goal free evaluation literature
c. Focus-structured comparisons
d. Theory-based evaluation literature
4. Literature on how to assess when circumstances are improving (e.g., CSIS in
Iraq).
5. Policy statements of donors and multilaterals, and some NGOs regarding
CPPB principles and approaches.
6. Meta-evaluation literature especially those of DAC donors not yet covered by
Utstein or FAFO.
7. Other evaluations, assessments and case studies named in our interviews as
especially useful/instructive.

The overall purpose of these literature reviews will be to give us a solid awareness of the
recorded state of the art of evaluations, allow us to review the broader set of potential
approaches to CPPB evaluations and how methodologies from other fields are, or are not,
relevant and useful for CPPB. Further, we will be able through some of these materials to
clarify implicit and explicit assumptions and theories of change that guide CPPB policies,
programmes and evaluations.

Learning from CDA’s Projects

We will review and include experience and evidence gathered through other CDA
projects and evaluations. The set of case studies just completed for our Steps Towards
Conflict Prevention project will provide evidence regarding effective approaches to
prevention and how they can be assessed. The case studies and consultation reports from
Inception Report 5

Do No Harm are relevant and instructive with regard to analyzing and assessing conflict
sensitivity approaches, particularly for programmes in humanitarian and development
assistance.

The Reflecting on Peace Practice project documents also contain a great deal of relevant
evidence about definitions, boundaries, processes, assumptions, etc. of CPPB activities
and how these have, or have not, been found to contribute to effective peacebuilding
and/or conflict prevention. In addition, in recent months, RPP has focused directly on
developing alternative approaches to conflict analysis, on identifying and testing impact
indicators, on tracing cumulative effectiveness of multiple CPPB activities and on
understanding micro-macro CPPB relationships. Although the efforts to push learning in
these four areas are not yet complete, the experience that has already been gathered will
add evidence to our exploration of key issues for the DAC approach paper.

III. FEASIBILITY OF THE TERMS OF REFERENCE

Priority Issues

Drawing on the issues highlighted in the TORs, preliminary inputs from the pre-Inception
Report interviews, and an initial document review, we have identified the following areas
for priority focus in the Approach Paper:

1. How CPPB Evaluations May Differ from Other Fields. We will examine how
the particular characteristics of CPPB programming affect the relevance, or not,
of various evaluation methodologies developed for other areas of programming.
So far, several characteristics have been identified as setting CPPB programming
and evaluations apart:
ƒ The essentially political nature of CPPB programming, (which must address
political relations directly, whereas, while other types of programming have
political implications, these can be secondary to their primary mandates and
purposes);
ƒ The complex and often violent context of CPPB work, which can limit access;
ƒ The explicit biases that are to be expected from potential informants;
ƒ The dearth of baseline data relevant to CPPB activities;
ƒ The longer time frames of much CPPB work.

In this section, we will also address the relevance of the DAC Evaluation Criteria and
the usefulness of the Humanitarian Assistance Guidance as a model for CPPB
guidance.

2. Expectations of CPPB Evaluations: We will review the evidence and opinions

regarding the reasonable expectations of CPPB evaluations. This line of inquiry
will include an exploration of evaluation standards (data collection techniques,
pre- and post-testing/analysis, control groups, etc.), and whether it is reasonable to
attempt to assess impacts (as opposed to outputs and proximate outcomes). This
review will also examine the issue of timing: a) how soon one can see CPPB
Inception Report 6

results, and b) how do the phases and types of conflict affect our ability to
recognize impacts with reasonable certainty. We will also address the reasonable
expectations of any given CPPB activity, within its context and in relation to
broader strategic objectives regarding Peace Writ Large.

3. Levels of Intervention: We will examine the evidence regarding evaluations at

different levels of societal interventions and the relationships among these levels.
Included here will be consideration of the distinctions and uses of project
evaluations vis-à-vis context evaluations—that is, the differences between
evaluations focused on the impacts and effectiveness of individual programmatic
efforts and those that assess the cumulative impacts of multiple programmes in
the same conflict zone. We will also explore assessment efforts at the
community, national, regional and/or international levels, looking for ways to
provide guidance on whether and how evaluations at these levels require different
approaches.

4. Context Analysis and Indicators: The CDA team will gather evidence regarding
the importance of conflict analysis, as it affects both programme and policy
effectiveness and evaluations of programmes and policies. Preliminary evidence
suggests that some form of context analysis is necessary for developing clear
goals and strategies—and for identifying appropriate indicators for monitoring
and evaluation. We will also be gathering experience with CPPB indicators, in
order to determine if there are some that have potential universality, as opposed to
those that will always be context-specific.

5. Assumptions and Theories of Change: We will clarify the range of assumptions

and theories of change that underlie the various types of CPPB programming.
Evidence to date shows that many CPPB activities (both programmes and
policies) are based on unexpressed or unexamined assumptions and theories. We
will suggest ways to make assumptions and theories more explicit, as a critical
step in good evaluation.

6. Relationships with Host Governments: We will examine the relationship of

CPPB activities to interactions with governments. Development principles
(including, for example, DAC Guidance and the Paris Declaration) place great
emphasis on working with and through local authorities. However, the experience
with CPPB activities suggests that governments are often important parties to
conflict, requiring, therefore, somewhat different approaches to working with host
governments. Future guidance on CPPB work and CPPB evaluations must
address the range of approaches to governmental relations.

7. CPPB Evaluation Methodologies: Finally, the examinations and clarifications of

all of the topics described above will inform our review of a range of evaluation
methodologies as we attempt to discern which approaches, or parts of approaches,
could be best adapted to assess impacts and outcomes of CPPB programmes. We
will include specific examples of evaluations that have proved particularly useful.
Inception Report 7

Realistic Limits of the Terms of Reference

Categorising CPPB Activities for Evaluation Guidance

The TORs call for an Annex addressing issues for evaluating CPPB programming in a
range of sectors. This effort would require several steps:
1. Clarification of the overall domain of CPPB—its boundaries, limits, what
activities/programmes/interventions fall in and outside of it;
2. A more comprehensive identification of the various sub-categories within CPPB
work (such as security sector reform, DDRR, reconciliation, etc.) and the
activities generally associated with those sub-categories; and
3. Identification of key issues and approaches to addressing challenges in evaluating
programmes in each sub-category, based on a thorough examination of the
literature and engagement of sector specialists.
Under the current TORs, we will address #1 above fully, and make significant progress
towards #2. With regard to #3, we will provide an overall framework that will be useful
for developing further specific procedures for evaluating specialized programmes in the
sub-categories. In addition, in a few sample areas where CDA already has considerable
knowledge, we will offer proposed evaluation methods and considerations. Further, we
will suggest how in-depth elaboration of considerations for evaluation in the sub-
categories can be addressed in subsequent efforts beyond the scope of this TOR.

Cross-Cutting Issues

We recognize the importance of the cross-cutting issues noted in the TORs (including,
gender, rights-based programming, HIV/AIDS, and others). However, we realize that to
cover all of these issues adequately, we would need to read the literature and interview
knowledgeable individuals in each of these issue areas. We estimate that within the
current plan for fulfilling the TORs, we will not have time to address the cross-cutting
“lenses” adequately. However, as we conduct the overview work outlined above, we will
seek information to address these topics, and we commit to reporting on all that we do
learn about these and, additionally, to providing a suggested plan for gaining greater
depth in subsequent processes.

IV. WORK PLAN AND BUDGET

See Appendix 3.
ANNEX 1: POTENTIAL INTERVIEWEES: TRACKING TABLE
(This list takes into account comments from the SG, see Annex 4)

Category/Possibles Initial Contact Made Completed

Evaluation Specialists
Michael Woolcock/JFKSch Michael Lund*
Harvard Cheyanne Church
Niels Dabelstein/Denmark
Koenraad van Brabant/WSP
Catherine Barnes/UK
Nick Mabey/e3g.org-UK
Uwe Kievelitz/Germany
Tony Vaux
Mark Keen *
Jon Hanssen-Bauer [NORAD]
Arne Strand [NORAD]
DAC Steering Group
Henrik Mungenast/SIDA Beate Bull/ NORAD Jennifer Stuttle/DFID*
Julia Compton/DFID Bjorn Holmberg/SIDA*
Cristina Hoyos/DEZA Agnete Eriksen/ NORAD

Other DAC
Stein-Erik Holjen/NORAD Patrick Merrienne/DFID* Jill Tirnauer/USAID/CMM
Krishna Kumar/USAID Ruairi O’Connell/DFID
Ciru Mwaura/DFID*
Steve Ainsworth/DFID*
Simon Arthy/DFID
Mark Segal/DFID*
James Fennell/DFID
Rob Wilkinson/DFID*
Tim Heath/DFID
Debi Duncan/DFID
Private Donors
Steve Pittam or Juliet Melanie Greenberg/former
Prager/Joseph Rountree Trust/UK Hewlett Fdn
Carnegie Corporation
Multilaterals
Peter Bachtelor + Kathleen Connie Peck/UNITAR Ron Parker + team/WB
Cravero/UNDP Gay Rosenblum- Andrew Russell/UNDP
Ian Bannon/Colin Scott/WB Kumar/UN-DPA Cyprus
Jaco Cilliers/UNDP Cyprus
Implementers/Practitioners
Gavin Preuss/Oxfam John Paul Lederach Dan Smith/IA
Hizkias Assefa (Kenya) Liz McClintock Tale Steen-Johnson*
David Moussa Ntambara (Rwanda) Mohammed Abu-
Vasu Gounden/ACCORD (So. Africa) Nimer/AU
Frederic Kama-Kama Tutu (PTN) Howard Wolpe/WWIC
NPI-Africa Pamela Aall/USIP
Rachel Goldwyn /CARE* Anton Baare
Alliance for Peacebuilding (Boston grp) Emmanuel
Kosovo RPP group? Bombande/WANEP
Tyrol Ferdinands/Sri Lanka
Andres Serbin/Argentina
Augusto Miclat/Philippines
Academics/Others
John Erikson Eileen Babbitt/Tufts Jonathan Goodhand
Peter Wallenstein/Uppsala Mohammed Abu
Peter Uvin/Tufts Nimer/AU

* Some SG priority suggestions

** Interviews completed before Inception Report.
ANNEX 2: INDICATIVE INTERVIEW QUESTIONS

1. Have you ever read a really good evaluation of a CPPB programme? What made it good?
What was most helpful about it?

2. What is hardest to evaluate in CPPB programming? What have we still not found a way to
assess? Why would that be important? What are the key open questions and challenges
regarding the evaluation of CPPB programmes?

3. What progress has been made—can we point to anything that we now “know” about
evaluating CPPB?

4. What have proven to be useful indicators for monitoring progress? In what context, for what
kinds of programs…? What indicators have proven unhelpful?

5. What are the purposes of CPPB evaluations? How do people actually use them (or not)? Do
people follow-up to implement recommendations, make programmatic changes, etc.?
[Explore accountability vs. learning purposes.]

6. Which evaluation methodologies seem to be most helpful for CPPB activities (classic results
models; theory-based; policy evaluations, etc.)? For which types of programs?

7. What are the boundaries of CPPB? What is/is not included? What are the useful sub-
categories and what activities fall within those?

8. Who else should we be talking with?

9. (For donors/government representatives): What are your policies regarding CPPB? How do
they influence your work and the choices you make for funding?

10. What are the key factors/issues that should be addressed in a DAC guidance document on
CPPB evaluation?
ANNEX 3: CDA WORK PLAN /BUDGET
Workplan

Time estimates:

Mary B. Anderson, Executive Director and Chief of Project 28 days

Diana Chigas, Co-Director, Reflecting on Peace Practice Project 15 days
Peter Woodrow, Co-Director, Reflecting on Peace Practice Project 15 days
Marshall Wallace, Director, Do No Harm Project 4 days
Tarah Farman, Project Associate, RPP and DNH Projects 10 days

Total 72 days

• CDA will submit the Inception Report to the DAC Networks by May 29, 2006.

1. The CDA group will meet together to outline and discuss the issues raised in the DAC TORs
and notes from comments on CDA’s previous proposal in order to outline and focus the
Inception Report.
1 day
2. CDA will undertake focused aspects of the literature review and some focused interviews to
a) assess more carefully the range of ideas and issues identified by the CPPB field as critical
or troublesome for evaluation and b) to ensure our later interviews and literature reviews are
well focused.
5-6 days
3. Mary B. Anderson (MBA) will write the Inception Report; others will review.
3 days for MBA, ½ day each for all other staff for a total of 5 days

If given go-ahead by the DAC Networks to proceed:

4. Diana Chigas (DC) will read and review additional ephemera and other recent evaluation
reports and analyses, to update background for our work with items produced since the
Utstein and FAFO reports.
2-4 days
5. Mary B. Anderson will collect and review additional pertinent donor policy papers.
2-3 days
6. Peter Woodrow (PJW) will interview relevant people in CPPB, in Evaluation groups and
among donors.
3-4 days
7. Marshall Wallace (MW) will interview selected personnel from humanitarian NGOs and
possibly donor groups that have added CPPB mandates to their work.
2-3 days
8. On the basis of this background from interviews and literature review, and CDA learning
from previous evaluations, RPP and DNH, will write the Approach Paper. Writing and
editing will occur during July and August. CDA will submit a draft of the Approach Paper to
DAC by end of August 2006
MBA 11-14 days
PJW and DC 5-9 days

9. Revisions and additions on basis of DAC Network and other expert comments through first
half of September
4 days
10. CDA will help plan DAC workshop where Approach Paper will be reviewed.
2 days
11. Workshop travel and participation.
4 days
12. Final revisions/rewrite of Approach Paper based on workshop feedback/suggestions.
3 days
13. Tarah Farman, will be involved throughout in gathering materials for review, setting up
interviews, keeping track of discussions, providing back-up to writing and layout expertise,
etc.
10 days
Budget

Mary B. Anderson (28 days at 700/day) 19,600.00

Diana Chigas and Peter Woodrow (15 days each at 600/day) 18,000.00
Marshall Wallace (4 days at 600/day) 2,400.00
Tarah Farman (10 days at 400/day) 4,000.00

Staff Fees $44,000.00

Estimated interview costs* 1,000.00

Financial management, accounting 1,500.00
*Direct costs associated with interviewing process. PJW and MW will hold some of the interviews in
person when they are traveling near locations where there are people we wish to interview.

Direct Costs $2,500.00

Subtotal $46,500.00

Overhead 15% 6,957.00

Includes costs apportioned to ongoing office expenses such as rent, insurance, equipment, etc.

Total USD $53,457.00

Total in Euros (exchange rate of .82825) €44,276.00
ANNEX 4. SUMMARY OF COMMENTS FROM CPDC-EVALUATION NETWORKS
STEERING GROUP (SG) MEMBERS ON INCEPTION REPORT AND CONFERENCE
CALL WITH CDA AND THE SG (6 JULY 2006)

This Inception Report (IR) was well received, and the following points came up in a discussion
between Steering Group members (NORAD, DFID, SDC and SIDA) and Mary B. Anderson,
Head of Collaborative for Development Action (CDA). These notes are designed to enhance and
possibly expand on some areas of the IR.
1. Expectations about how far the "approach paper" will go toward guidance: The
Steering Group had decided that before developing full guidance an "Approach Paper"
would be needed that goes a relatively long way toward producing guidance. This should be
useful as an in-depth background paper to support a short evaluation guidance publication
that will be developed based on this paper and a consultative workshop. Note: CDA team
should consider "approach paper" as a working title only for now and decide on the most
relevant term/title as it develops, as some were not completely satisfied with the term itself.
2. Audience: It was agreed that the “approach paper” (and the guidance) would be aimed at
donor agency staff primarily (mainly field, also headquarters), as this is the DAC's
comparative advantage. However, as is the case for other DAC guidance, it should be useful
for a wider audience working on evaluations of CPPB activities.
3. Outputs and outline: Mary B. Anderson pointed to Section III in the inception report on
"Priority Issues" which covers the top seven issues that CDA intends to explore and will form
the basis of the broad outline for the paper.
Action: It was agreed that specific recommendations for CPPB evaluation guidance would
be added to that list. Mary B. Anderson noted that outlines evolve in the writing process, so
this was not a strait-jacket.
4. Developing more in-depth information on the areas of the CPPB domain [(Approach
paper annex), see "Realistic Limits of the Terms of Reference"]: As stated in the IR, CDA
will provide an overview of the CPPB domain and an overall framework that will be useful
for developing further specific procedures for evaluating specialized programmes in the sub-
categories of CPPB. In addition, CDA will offer proposed evaluation methods and
considerations in a few sample areas where the team already has considerable knowledge.
CDA will also provide in-depth elaboration of considerations for evaluation in the sub-
categories which can be addressed in subsequent work. The group discussed the question of
how to address sub-sectors and areas where CDA has less experience (e.g. SSR, DDR).
Many of these are ‘state-level’ interventions that may require a contrasting approach to
evaluation to areas such as dialogue where CDA has most experience. It was agreed that two
contrasting examples (dialogue and SSR) would be used as case studies in the approach
paper to highlight the different issues involved. It was suggested that a small experts'
workshop (day and a half with about 20 participants) could be held to conduct two parallel
working groups on these topics that would allow us to draw out approaches, see what is
tailored to certain areas, what applies to different ones, and seek to identify gaps. This would
be an effective tool to complement work that CDA can provide in a few sample areas, as
noted above. It is important to note that this workshop would substitute for work on dialogue
in the first draft approach paper which is an area where CDA does have good experience and
knowledge. By putting this topic into the workshop, the group decided that CDA would not
 try to handle it before the workshop. CDA could rather take up training, or organizing joint
projects instead.
Action: Look into logistics of a small complementary experts' workshop on "evaluation
methods in dialogue and SSR activities" (about 20 experts – 10 for each) to help identify
evaluation methods in these fields. This would also support the CPDC work on M&E for
security system reform. CDA could provide support (CDA to cost, once the Secretariat has
specified how much of the logistics/planning can be done by SG members/the Secretariat).
(Note: DFID offer to host this at the Global Facilitation Network for SSR, Birmingham, UK
to combine with existing plans to do work on evaluation of SSR activities).
5. Interviews, literature and methodology
On interviews, the SG felt it was important to be sure that approaches tried by partner
countries were reflected and that those on the receiving end of programmes provide
information from their perspectives. Mary B. Anderson pointed out that interviews with
Southern partners were already being conducted (Ghana, Kosovo, Cyprus), including with a
number of people form conflict zones who have been on the recipient side of CPPB work and
evaluations. This came up in early interviews (as outlined in the Inception Report), and CDA
immediately began to pursue it. Another point was raised that, since donors are all moving
toward the Paris Declaration with different types of support, it is important to talk to a range
of different types of donors/actors (State and non-State actors, e.g., NGOs and INGOs).
The contrast between ‘state building programmes’ under a country-led approach (where
evaluation may well be through a joint donor-government process) and individual project
approaches was particularly important to cover. Mary B. Anderson noted that there is in the
overall interviews, however, a time when one sees diminishing returns. As more interviews
are done, the repetition becomes greater across a number of issues, a caution to the SG about
wanting to spend too much time on additional conversations. On methodology, it was also
suggested that a table could be developed in reflecting on usage that sets out the kinds of
analysis/evaluations being used for which types of activities, if this could be helpful for
guidance purposes. CDA noted that they do not plan to review all the evaluation
methodologies available but are exploring what people have seen to be useful and what they
have not found helpful.
Action: First, the Steering Group has submitted further ideas for interviews with contacts and
the priority of these (see updated interview table below). Given this, CDA will cost some
additional interview days. The key issue is to ensure balanced coverage, particularly with
partner country practitioners.

Draft Timeline

July: Finalise any amendments to the contract with CDA (e.g., more interviews/small
workshop support) and circulate Inception Report to Network members.
August: First draft approach paper complete by end-August to circulate to members for
comment by (11 September). Decide on date of main workshop event, “Evaluating Conflict
Prevention and Peace building Activities”.
September/Early October: Possible small experts' workshop on different evaluation
methods in dialogue and SSR (Note: DIFD offer to have this hosted by the Global
Facilitation Network for SSR).
16-17 October - CPDC Network meeting: CPDC Task Team on Evaluation to meet in
margins of CPDC for update and input on work.
October or November: Could hold main workshop event, “Evaluating Conflict
Prevention and Peace building Activities” to review full draft.
16-17 November 2006: Evaluation Network meeting.
First quarter 2007: Goal for developing draft Guidance based on Steering Group and
Members of CPDC and the Evaluation Networks thoughts on steps needed to develop the
“approach paper” into draft guidance, particularly to cover sub-sectors/areas not in that piece.
Piloting the draft guidance to begin and be conducted in 2007-8.
 CONSULTANT TERMS OF REFERENCE
Developing Guidance for Evaluating Conflict Prevention and Peace Building Activities:
An Approach Paper
DAC Network on Development Evaluation and
DAC Network on Conflict, Peace and Development Cooperation
Introduction
The DAC Network on Conflict, Peace and Development Cooperation (CPDC) and the DAC
Network on Development Evaluation are collaborating on the development of guidance for
evaluating conflict prevention and peace building (CPPB) activities, an area where only minimal
guidance already exists. The two Networks have agreed to commission a consultant to develop
an approach paper that provides in-depth exploration of the options for practical guidance for
CPPB at programming and policy levels.
The approach paper will require tasks such as a literature review of key DAC and donor agency
policies and guidance on evaluation and on conflict prevention and peace building (CPPB) and
in-depth interviews with a select number of key informants. The approach paper will explore and
define possible ways forward, highlight the main methodological issues and recommend next
steps for developing guidance on evaluating conflict prevention and peace building activities.
This will build on work previously carried out by the DAC Evaluation Network, by the CPDC and
by the Utstein Group1. The consultants should review and utilise relevant DAC, donor and other
publications, including the DAC guidance documents on conflict prevention and security system
reform as well as the recently developed issues briefs on CPPB (www.oecd.org/dac/conflict)
and the guidance documents available on the DAC Evaluation Network website
(www.oecd.org/dac/evaluationnetwork), along with a draft report which seeks to review
substantive and methodological lessons from past evaluations of CPPB activities.
Questions and Challenges: Towards Developing Guidance for Evaluating Conflict
Prevention and Peace Building Activities
In developing the approach paper, the style and approach of the Evaluation Network publication
Guidance for Evaluating Humanitarian Assistance in Complex Emergencies
(www.oecd.org/dataoecd/9/50/2667294.pdf) should be considered as a possible loose
framework/model for developing CPPB guidance in this field. The consultants will, therefore,
first undertake a detailed analytic review of the humanitarian guidance, as well as other
guidance developed by DAC member agencies, and determine if any areas of this text or others
could be transferable to future CPPB guidance. New areas and issues will also need to be
added The Humanitarian Guidance should be used only as a basic model on which to build
rather than as a fixed formula or constraining framework.
The following sections provide a working outline of possible questions to be addressed in the
approach paper for CPPB guidance. In addition to the outline and issues raised below, it will
also be important to look at related cross-cutting questions such as: i) the need for assessing
gender sensitivity of CPPB programmes; ii) the need to assess whether a rights perspective has
been mainstreamed/included in the CPPB programmes; iii) a reference to violence in
evaluations, and how the programme evaluated contributed to reduce violence/violent conflict
(referred to in section 4 below); and iv) some consideration of if/how HIV and AIDS prevention is
covered in CPPB evaluations, as this is becoming an increasing problem in conflict-affected

1
The Utstein study was a review of the peace building experience of Germany, Netherlands, Norway and the UK.
countries and regions, especially in Africa. Please note that this outline should be seen as a
working guide not as a straightjacket. The consultants are encouraged to take the scope and
flexibility necessary for creative thinking in developing thematic areas and ideas in the approach
paper and for proposing content for future guidance.
1) An Overview of CPPB Activities and Current Evaluation Practices: the Value Added of
Guidance
An introduction could provide a broad but concise overview of the conflict prevention and peace
building environment, using available policy documents. The introduction should cover
questions of why guidance is needed in this area and outline the key audiences for guidance
and their evaluation needs.
Questions to be addressed may include: What, in brief, is the operational meaning of Conflict
Prevention and Peace Building activities? What are the characteristics of modern-day conflict
environments? What kind of conflict prevention and peace building activities are most commonly
undertaken and by which agencies? What do we mean by evaluation of these activities? What
is the need for and purpose of undertaking evaluations of conflict prevention and peace building
activities? What is the present situation with regards frequency, rigour and robustness of conflict
prevention and peace building evaluations (Note: See work already been done on lessons
learned on evaluating conflict and building peace for these DAC Networks.)? What is the
purpose and value added of the new guidance? What would be a suitable methodology and
approach for developing CPPB guidance?
2) Analysing and Developing Guidance for CPPB Activities: Differences from Traditional
Evaluations

The approach paper should look at how guidance might outline the main generic differences
involved in evaluating on the one hand CPPB activities and on the other hand both
‘conventional’ development programmes and humanitarian aid interventions. Differences to be
explored in this scene-setting overview could include, but not be limited to, the following:
¾ The particular difficulties involved in defining CPPB - for example it can be difficult to
determine when conflict ends or peace begins. The consultants are not required to
devise methodologies to answer such questions but rather to acknowledge that in any
given context debates may exist about whether the environment is one of ‘peace’ or
‘conflict’. It will be helpful to look at how agencies shape their programmes differently
according to the context in a conflict-affected or prone country.
¾ The particular importance of the political nature of conflict prevention and peace building
and of its ideological context or country-specific situations as they relate to the donor
community, government agencies, NGOs, national and regional actors, etc.
¾ The essence of not working 'around' but 'in' and 'on' conflict situations should be
considered. The latter is the main approach of CPPB activities and, therefore, the study
shall address approaches taken by different donors in terms of being proactive and
working 'on' conflict.
¾ The ‘high stakes’ nature of CPPB activities, e.g. misconstrued projects/programmes can
potentially lead to an increased risk of conflict. The consequent value of a ‘Do No Harm’
approach and conflict sensitivity by working 'in' and not 'around' conflict situations.
¾ The importance of policy coherence – whole-of-government approaches to issues
involving diplomacy, defence, trade policy, etc. at the donor and partner country levels –
due to the interconnected nature of causational factors.
¾ The cross-border, regional, sub-regional and international context of many conflicts.
¾ The question of applying the principles of the Paris Declaration in a fragile state or in a
context where key stakeholders may be protagonists of the conflict. How do principles of
 ownership and alignment apply in this context and what are the challenges?
¾ The problem of vested interests (military, political, other) who may lose out from the
success of a project/programme. The need for rigorous stakeholder analysis?
¾ The particular importance of looking at the effect of an individual project/programme on
the wider ‘peace writ large’ within any given context, e.g. the difficulties of demonstrating
the links between micro projects and macro 'peace writ large', noted below as well.
¾ The particular importance of UN and NGO actors in conflict environments.
¾ The data deficit and particular difficulties in establishing good baseline information. With
the possible absence of ‘hard data’ in conflict situations, what other methods can be
used to qualify/quantify other than data sources?
¾ Lessons learned should also be reflected in the analysis, and DAC guidance and Issues
Briefs could be helpful in that endeavour, see www.oecd.org/dac/conflict .
¾ Participatory approaches to evaluation of CPPB activities and the perceptions of
stakeholders, in particular the poor, are also critical in these contexts.
3) Options for Guidance on Programme Design and Management
Based on the above overview of the context, the approach paper should consider
recommended steps for CPPB practitioners to build steps into project/programme design and
management which would facilitate their evaluation. Issues to be covered would include, but not
be limited to the importance of:
¾ Articulating and clarifying objectives at the planning stage;
¾ Working according to the principles of results based management;
¾ Developing robust baselines and monitoring systems; and
¾ Carefully considering the different needs of the intended end-users/audiences.
4) Recommendations for Practical Guidance on Evaluation

This section of the approach paper should consider some of the core questions related to
providing guidance for undertaking evaluations of conflict prevention and peace building
activities and underline the key point that evaluations in this field should be designed in a
conflict-sensitive manner, with a special emphasis on the obstacles and possible incentives to
achieving this. In addition to areas covered in the humanitarian guidance, additional issues
could include:
¾ What kinds of evaluation and research methods are generally more suitable in conflict
environments – e.g. qualitative versus quantitative or mixed methods approaches? How
can the design of an evaluation methodology be informed by the political economy
and/or by conflict analysis tools? What are the advantages or disadvantages of using
theory-based evaluation in evaluating CPPB activities?
¾ What is the particular value of policy evaluations in conflict environments? What are the
difficulties of demonstrating the links between micro projects and macro ‘peace writ
large’? How could we look at evaluating all CPPB activities within one conflict or national
context?
¾ What is the role of conflict analysis when evaluating the relevance of CPPB activities?
How to make use of other conflict analysis tools and Post-conflict Impact Assessments
(PCIA)?
¾ The importance of triangulation in a context where stakeholders may have firm
ideological standpoints and/or whose perceptions may be affected by trauma or
perceived danger.
¾ The importance of protecting the security and safety of the evaluation team, while
addressing the need for evaluations to take account of work done in potentially more
 insecure rural areas.
¾ The ethical responsibility to protect the security and anonymity of evaluation informants.
¾ How can evaluators make the links between the micro- and macro levels, and to what
extent can any existing national policy frameworks such as Poverty Reduction Strategies
(PRS) be used to frame the approach?
¾ Are joint evaluations more suitable in a context where multiple programmes are affecting
the same ‘peace writ large’?
¾ How can steps be taken towards addressing the ‘strategic deficit’ in CPPB evaluations,
as identified by the Utstein report - how can evaluations assess the relevance of donor
project/programmes to country-specific and broader policies?
¾ How should the timing of evaluation be planned in a conflict prevention or peace building
environment?
¾ How can a lack of baseline information be addressed by the evaluators?
¾ How applicable are the five standard DAC criteria to a conflict environment? Do they
need modifying or adapting to the conflict context and are any additional criteria
needed?
5) Annex: Categorising CPPB Activities for Evaluation Guidance

The approach paper for developing guidance on undertaking evaluations in conflict prone and/or
conflict affected settings should be supplemented by a short discussion piece (5-10 pages)
which categorises the domain of Conflict Prevention and Peace building (CPPB) for evaluation
purposes and serves as an annex to the main approach paper.

It is recognised that conflict prevention and peace building encompasses a number of distinct
policy areas (for example, from security and development, to governance, to state building, to
peacekeeping operations and support for truth and reconciliation), together with a wide variety
of activities. Given such a broad domain, it is anticipated that - while guidance could provide
generic advice cross the conflict contexts - more specific advice/direction will also be required
within each CPPB category. The purpose of this annex is to discuss a breakdown of the CPPB
domain and to differentiate each category from the point of view of how evaluations could be
undertaken. The annex should also provide preliminary proposals on specific key
issues/questions that would be pertinent for an evaluation team for each category within the
CPPB domain of activities.

There have been a number of categorisations of CPPB, including the ‘palette’ concept
contained in the Utstein study. However, none have attempted to categorise CPPB for
evaluation purposes. Issues that may be pertinent to defining the categories relate to the type of
actors involved, level of local ownership, political/conflict environmental factors and the scope of
the activities being undertaken (whether focused on regional, national local actors/institutions).
Developing a clear picture of CPPB activities as practiced by most agencies will be informed by
the literature review and interview process.

Overview of Activities
1) Develop an inception report that would provide: a presentation and justification of the
methodology chosen to approach the task; the feasibility of the terms of Reference - its
questions, scope, time, budget, and foreseen methodology; and deviations from ToR with
justifications for these.
2) To underpin the work, undertake a focussed literature review of key DAC and donor
agency policies/guidance on evaluation on CPPB, with particular attention to the useful
information to be drawn from the DAC guidance on Humanitarian Assistance.

3) Conduct in-depth interviews with a select number of key informants (15-20) by telephone,
including members of the Networks on Evaluation and Conflict, Peace and Development Co-
operation. The list of informants will be agreed with the DAC Secretariat.

4) Review and map the breadth of activities conducted in the field of CPPB to be included in
the Annex with a view to categorising the various areas for the purpose of future evaluation.

5) Conduct analytical work and conceptual thinking to develop and write an approach paper
covering preliminary recommendations for guidance for evaluation of CPPB activities and
policies.
6) Present and discuss the findings of the draft approach paper at a workshop in Oslo
planned for the 3rd quarter of 2006. The consultants should also prepare break-out group
discussion topics based on questions raised in the approach paper and provide inputs to the
agenda.
7) Finalise and submit a final approach paper that is of publishable quality in view of the
discussion and feedback at the workshop.
Budget, Inputs and Competencies

The consultants should develop a detailed budget proposal and suggest in detail the number of
necessary person days in order to complete the tasks, also indicating any other individuals who
may support this work (with relevant CVs) if working in a team. At least eight to ten weeks of
full-time work is envisaged, followed by intermittent work through end-2006. The detailed budget
proposal, and information on the full team, if relevant, should be included in an annex to the
inception report.
The consultant(s) will need strong experience of donor practices in development cooperation
and a background in: (1) Conflict prevention / Peace Building / Security Programming; (2)
Conflict assessment/Conflict Analysis; (3) Evaluation methodology and implementation; and (4)
Communications and writing skills.
Timing and Outputs
The consultancy is expected to begin in May 2006.

The consultant(s) will first be asked to provide an inception report (8-10 pages) for the
approach paper within three to four weeks of the signature of the contract. The inception report
will then be reviewed and considered for approval by the Steering Group. Once agreement has
been reach on the inception report, the consultant(s) will be asked to begin work on the
approach paper.

A first draft of the approach paper (approximately 30 pages + 5-10 page annex) should be
provided for preliminary review and feedback by the DAC Secretariat within six to eight weeks of
approval of the inception report. As noted, under "Activities", the consultants will be asked to
undertake a literature review to explore the options for guidance in this area, as well as
interviews with a select number of key informants and the other activities described above for
the development of the approach paper.
The full draft approach paper (approximately 30 pages + 5-10 page annex) is due for
submission within two to four weeks after the approval of the draft approach paper. The draft
approach paper, which should provide information on the scope, audiences and preliminary
findings for guidance in this field, will be presented and reviewed at a workshop envisaged for
the 3rd quarter of 2006. Therefore, a draft list of topics for break-out group discussions at the
workshop, based on questions raised in the approach paper, should also be provided with the
full draft approach paper. This work may involve some wider input to the workshop agenda.
A final revised version of the approach paper - that is re-worked in view of the outcomes of
the workshop and of publishable quality - should be submitted within two to three weeks of the
completion of the workshop.
Conduct of the Work
The consultants shall report to the OECD Secretariat (CPDC and the Evaluation Network), and
the point of contact will be Lisa Williams (lisa.wiliams@oecd.org). Drafts of the approach paper
will be shared with the members of the two Networks for direction, feedback and input.