LEARNING
PAPER
Chiefs from different ethnic groups in Wau, South Sudan, work differently – together – to promote peace. Photo: M Perkins
DOING THINGS DIFFERENTLY
Rethinking monitoring and evaluation to understand change
Over the past four years, Saferworld has put
in place a way of monitoring, evaluating and
learning from our work focused on behaviour
and relationship change. This paper outlines
the process we have gone through to adapt,
embed, and embrace an approach inspired
by Outcome Mapping1 and Outcome
Harvesting (OH)2. We hope it shows other
interested parties how we have transformed
the way we understand, monitor and collect
evidence of change in our work.
“This is a beautiful, exemplary case of an organisation
taking Outcome Harvesting, and customising it to
specific needs and context. The danger is that the
essence of OH can easily get ‘lost in translation’, but
Saferworld has retained the essential principles while
making the approach their own.”
– Ricardo Wilson-Grau
1
See Outcome Mapping http://www.outcomemapping.ca/
2
See Outcome Harvesting
http://www.betterevaluation.org/plan/approach/outcome_harvesting
Key messages
Our approach is centred on collecting and
analysing evidence together about what others do
differently – and determining how far that is
because of our work. The approach is different
because it is flexible and straightforward enough to
be used in complex, rapidly-changing contexts. It
empowers front-line staff, communities and
partners to ‘monitor’ what matters to them.
Key benefits:
 It is simple, but promotes complex discussion
and analysis.
 It allows conflict- and gender-sensitivity to be
built into programmes; it promotes analysis of
and adaptation to context.
 Bringing front-line staff and partners into wider
conversations with others substantially
increases cross-organisational learning.
Learning points:
 Like most change processes, using the
approach requires leadership, guidance, and
practice – one-off trainings don’t work.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
Dedication
In 2012 Saferworld appointed its first MEL coordinator based in a
country team. Ramesh Nidhi Bista (pictured, right) had spent many,
many years in Nepal working on M&E, and had tried pretty much
everything to support partners in dealing with the language of
goals, objectives, outputs, outcomes, indicators, targets, and
milestones that donors and international NGOs are so fond of.
Not much seemed to work to help them to understand, measure
and report on changes. However, he had hit on this definition –
“an outcome is something that others do differently” – and found
that it made sense. The minute he said it to me, I thought – that’s just what I’ve been looking for. Ramesh,
dear colleague and friend to the Nepal programme, died suddenly in November 2013. This paper is
dedicated to him.
Madeline Church
Introduction
“An outcome is something that others do differently
(individuals, groups, institutions, parts of institutions)”
– Ramesh Nidhi Bista
In 2011, Saferworld had one Planning and Evaluation
Adviser. She was the sole staff member in a rather
grandly titled Organisational Development Unit
(ODU), which was tasked with driving a big
institutional push towards better Monitoring,
Evaluation and Learning (MEL) approaches. Our new
Strategic Plan provided the impetus and leadership
that the organisation needed to systematise its
accountability and learning, so we could know
whether we were meeting our ambitious plans. And a
new Programme Partnership Agreement (PPA) with
the Department for International Development (DFID)
(over three years) provided strategic funds to support
these new commitments.
Saferworld works in complex, conflict-ridden
environments, with fast-changing dynamics,
alongside communities that have historically been
abandoned, marginalised, or ignored by their local
authorities and security providers, and feel a profound
sense of insecurity. The organisation prides itself on
linking this community-level work (primarily
Community Security3) with an ‘evidence-gathering’
agenda, to show that things can change for these
communities if they are empowered to lead
constructively on what’s important for them. It also
helps to demonstrate positive change to authorities
and policy-makers. We were, therefore, seeking a
MEL approach that would reflect this organisational
agenda and concentrate on meaningful evidence, by
listening to and valuing communities’ own definitions
of what constitutes success.
We wanted the approach to be appropriately focused
on what Saferworld actually does – influence
3 4
See our Community Security Handbook
http://www.saferworld.org.uk/resources/view-resource/806-community-
security-handbook
Page 2 of 12
behaviour- and relationship-change, by and between
certain ‘actor categories’4. It had to be straightforward
enough for communities, partners, and front-line staff
(some with English as a second, or even third
language) to use to learn about their work, build on
and improve it, and be able to communicate about it
effectively to others. We know from experience that
the worst type of instrument for tracking the effect of
our work in complex and changing environments is a
complicated set of things to measure that are hard to
interpret. What we were looking for was something
that focused on demonstrable change, and that was
simple enough for all staff to work with.
“MEL was an area I had difficulty to follow and have a
picture in my head of all the statistical data, log frame,
framework and so on. The outcome harvesting
approach makes MEL easier for people like me to
understand and plan activities better. I am glad that
we have this process beside the data gathering for
the log frame.”
– Khirad Kargasov, Area Programme
Manager, South Sudan team
Rethinking ‘M&E’
In determining how to move forward to meet our MEL
commitments, we looked at our current practice and
tried to understand what the organisation really meant
when it talked about ‘M&E’ (as it did then). In
particular we looked at whether this was understood
exclusively as an ‘enforcement’ or ‘accountability’
exercise or whether we were learning from what we
did in a structured way that went beyond reporting to
donors and commissioning external reviews or
evaluations.
We noticed that in the main, staff understood ‘M&E’ to
mean ‘reporting to donors against indicators in the
log-frame’. We found that most staff and partners
didn’t really understand what was meant by an
indicator, often because different donors used the
Saferworld works to influence four actor categories: individuals and
communities; civil society; authorities; external actors. See our Change
Model, http//saferworld.org.uk/images/saferworld-change-model.png
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
term differently, or because indicators had been
chosen or written by others. Many early hours of the
ODU were spent clarifying the difference between an
indicator, milestone, target, and ‘objectively verifiable
indicators of achievement’.
Some of the elements we found least helpful about
the ‘status quo’ were:
 the focus on activity reporting. This meant that
many staff were largely reporting on what they
and their partners had spent their time doing. We
had plenty of data on the ‘what?’, but not enough
on the ‘so what?’, i.e what difference did it make?
“[Outcome monitoring] has helped us focus more on
the changes we are trying to achieve over activities.”
– Shelagh Daley, UK Advocacy Coordinator
 the static and linear nature of log-frames,
which doesn’t fit with Saferworld’s work on
complex social change in shifting and evolving
conflict contexts. While log-frames were intended
to be ‘tools for thinking and strategising’ they had
become rigid project implementation templates,
which failed to take into account the systemic
nature of many conflicts. Almost all our funders
had log-frames attached to agreements (the
Dutch Ministry of Foreign Affairs being a notable
exception), which meant that ‘delivering on the
log-frame’ and ‘checking off the indicators’ had
become the central purpose of monitoring.
 the largely unhelpful nature of quantitative
measures, given that so much of our work
focuses on behaviour and relationship change, in
a particular context. These quantitative indicators
tended to be default, as counting is considered to
be somehow more revealing and/or easier,
despite the wealth of alternative qualitative
approaches that would make more appropriate
bedfellows. On the whole, counting often can’t
answer the important questions that Saferworld,
its partners, and the communities we work with,
want to ask. Lack of reliable data is common in
many conflict contexts, so long-term changes or
wider effects are often hard to understand without
huge investment in expensive research. Teams
were either collecting lots of data that were of
questionable use, often because that data was
easy to collect, or not collecting any data at all.
Few were asking themselves what constituted
reliable evidence of change.
 confusing language used by the sector, and
by funders. This included the mixed use of ‘goal’,
‘purpose’, ‘objective’, ‘outcome’, ‘indicators’,
‘targets’, ‘results’, ‘outputs’, and ‘evidence’,
combined with a new fascination with ‘theory of
change’. This was a particular challenge as
Saferworld employs largely local staff in
programmes, and/or works primarily in local
partnerships, with English as a second or third
language in all instances. ‘Results’, ‘outputs’ and
Page 3 of 12
‘outcomes’ are, for instance, all translated as
‘results’ in Arabic and Russian, showing just how
difficult it is to differentiate between them. It was
more useful for us to distinguish between what we
and our partners do (activities) and what others
do differently as a result of these activities
(outcome).
 the lack of capacity in MEL at the right level in
Saferworld – i.e near to the partners, to the
communities, and to the national offices.
In sum, we needed an appropriate MEL approach for
our change agenda, which allowed for flexibility, a
focus on behaviour and relationship change, and the
ability to adapt to quickly changing environments with
many political dynamics at work. We needed
something that would build in conflict- and gender-
sensitivity and support adaptive management.
And in line with the projected growth of Saferworld –
working at a greater level of scale and intensity – we
had a vision of a devolved organisation, with more
country offices, country managers, and increasing
numbers of national staff. That needed national MEL
advisers in teams working closer to the action.
What is Outcome Harvesting?
We came to Outcome Harvesting through the
Outcome Mapping community.5 Outcome Mapping
was designed to rethink the way in which
behaviour-change work is designed, planned and
described. The focus of Outcome Mapping is on
hoped-for changes in behaviour any programme is
working to achieve through its influence, using
outcome challenge statements to describe what
these are.
Outcome Harvesting is an evaluation approach
inspired by Outcome Mapping and Utilization-
Focused Evaluation. Unlike other evaluation
methods it doesn’t start with predetermined
outcomes, and measure progress towards them,
but rather collects evidence of what has been
achieved in the programme or project area, and
works backwards to determine whether and how
the project or intervention contributed to the
change.
At its most elegant, it is really simple. It asks a few
core questions for data-gathering about the
change in behaviour:
 Who did what, when and where?
 How significant is this change?
 What contribution was made by the
programme to this change?
5
for a full description of the approach see
http://www.betterevaluation.org/plan/approach/outcome_harvesting
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 4 of 12

In particular it works well when you want to focus

on outcomes, rather than activities, and it is
suitable for evaluating in complex programming
contexts ‘where relations of cause and effect are
not fully understood. Conventional monitoring and
evaluation aimed at determining results compares
planned outcomes with what is actually achieved.
In complex environments, however, objectives and
the paths to achieve them are largely
unpredictable and predefined objectives and
theories of change must be modified over time to
respond to changes in the context’ (emphasis
added)6.
Outcome harvesting is flexible enough to be used in
complex, rapidly-changing contexts. Political violence
in Bangladesh in the run up to the 2014 elections
Adapting and using Outcome changed action group priorities in some project sites.
Photo: Saferworld/Altaf Hossain
Harvesting
Given Saferworld’s primary focus on behaviour and
relationship change in complex contexts – by and “The approach helps us to tell a story of how our
between social actors, working through on-the-ground actions have prompted changes in behaviour and
programming, policy and advocacy – Outcome improvements in relationships.”
Harvesting offers a way for us to get to the heart of
– Luke Errington-Barnes, Funding Coordinator
what matters. It particularly appeals as a
straightforward and easy-to-understand way of
shifting attention away from activities (what did we
“This approach makes it possible to more clearly
do?), onto the practical, observable shifts in actions
explain our goals and to justify interventions and our
by the actors our programmes are seeking to
responses on specific local community security
influence (what did they do differently?).
needs. For beneficiaries (Local Crime Prevention
Centres – LCPC) it is easier to understand the
relationship between analysis they conduct and to
“It helps me determine whether an activity
select effective measures that will have an impact on
implemented is relevant to the context or not.”
the situation. We introduced outcome statements in
– Peter Machar, Project Coordinator, the LCPCs plans, and this helps them to understand
Wau, South Sudan the direction and nature of response that could result
effectively in a change in the situation.”
– Stefan Stoyanov, Project Manager, Central Asia
Although Outcome Harvesting is largely practiced as
an evaluation approach, we wanted to see if it would At the same time we wanted to focus our attention on
help us in our regular monitoring. We wanted teams evidence, and to find a way to present our claims in a
to be able to discuss ‘outcomes’ with their partners way that could be tested and verified by other
and communities regularly, document them, review external reviewers or evaluators. Most external
them, understand them, and build on them. Our evaluations Saferworld had commissioned in the past
Community Security theory of change is founded on relied on paid consultants reading strategies, plans,
using joint action-planning to create trust, through project proposals and donor reports, and then being
which change occurs to relationships, and expected to collect primary evidence. We wanted to
behaviours, which in turn creates more trust, be able to provide a review or evaluation team with a
confidence, agency, ownership, and transparency. At package of verifiable claims and evidence that they
the same time, common action creates tangible could then put to the test by interviewing key
changes. This work is highly reliant on skilled, informants and focus groups, rather than having to do
knowledgeable staff and partners working with key all the work uncovering results and evidence
stakeholders to create the conditions for this to themselves.
flourish. To do so, they need to be able to identify and
So, as part of our new framework for MEL, we
codify the changes they are seeing in a manner
constructed a mixed approach around a central core
straightforward enough to stimulate learning and
focus on what others do differently as a result of our
adaptation.
work. It is – and is meant to be – pretty
straightforward.

6
ibid
 You’re trying to record something Who, or which institution
that another person did differently. or group, is this outcome
about?
Start with naming the person, Dr Manga, The Minister
institution or group. Use as much of Interior for Zang, from
detail as possible.  the Union of People party.
Follow with recording
exactly what they did. Use
active verbs to describe this.
‘He instructed’, ‘she provided’.
Limit yourself to describing
the action.

What did he do differently?

The Minister of Interior for Zang
instructed the Chief of Police for Maple
County to introduce a monthly meeting
with communities across Maple County.
Date: July 2014
Place: Mobo Capital City.
Where and when did this happen?
Be as specific as possible about
where and when. The date, or time
period is really important, so don’t
miss this out. Your outcome should
now look like this:
He instructed the Chief of
Police for Maple County to
introduce a monthly meeting
with communities across
Maple County.

Highlight why this
change is worth
noting, important or
significant.
You are seeking to
record the connection
between the context, and
the change. For instance, if this
is the first time the Minister has taken
action it is probably significant. If he
has given this instruction many times
before, and nothing has changed, then
it is not. If the Minister is powerful, and
difficult to influence, then it might be
significant. If the Minister is not the one
with the real power in the situation, then
it probably isn’t.
This change is significant because this So, to decide its
Minister has consistently refused to accept significance, ask
that the communities have anything your team some
meaningful to contribute (it is a change in his questions:
consistent practice).
• Is this the first time?
The situation in Maple County is very bad • Does it link into the
when it comes to police-community relations. conflict analysis you
The police are often drunk and abusive. The have?
clan relations mean that the communities
• Is it big? Small?
don’t want to complain, because the
• Does it seem to be
Minister of Interior comes from there, and
a systemic change?
his reputation could be at risk if he lets the
communities say what is happening. • Does it seem to be
a policy change?
The communities have been saying for many
• Are there fruits of
months now that the local Chief of Police
sustainability in
doesn’t listen to their concerns, but the
there?
Minister has refused to talk to the partner.

What did Saferworld, Keep an ‘evidence box’, physical or

our partners and/or the “Saferworld made all electronic, in which you can store
the calls to encourage emails, notes, phonecalls,
communities do to
the Minister to meet with etc, which demonstrate
contribute to this change?
the partner. The partner the changes that you
trained the communities are referring to. These
Here you can start to record
in advocacy. The can be added to the
the activities or resources
communities wrote the outcome forms as pieces
that the team, the partners
paper that the Minister of evidence to substantiate
and/or the communities put into
read to convince him”. claims, by embedding them, or
making the change possible.
uploading them electronically.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
How does it get used in practice?
What this means in practice is that all teams are now
expected to do the following as a matter of routine:
 keep records of who has done what differently as a
result of our work (noting where they did it, and
when). These are kept in notebooks, in evaluations
of community action plans, in partner reports, in
Word and on-line forms.
 hold review conversations with their partners at
least twice a year, to discuss these and other
outcomes.
 note down why they think these changes are
significant, linked to the conflict and context
analyses that form part of their strategies and
which they update.
 bring all this information together at least twice a
year in a face-to-face workshop setting with their
programme/policy/advocacy colleagues. In this
they discuss the meaning of the outcomes, and
they record the significance of the outcomes in
relation to context.
 rank the outcomes, as a team, according to levels
of significance and the contribution made by
Saferworld and partners to that outcome (low,
medium or high). This is not a personal opinion, but
a team agreement, and is a tool to stimulate
discussion. Each team creates a ‘smartie map’
which displays this information visually (see picture
top right).
 upload any outcomes, positive or negative, into
Saferworld’s intranet system, Karacel, where they
can be seen by everyone in the organisation. All
outcomes are displayed on relevant team pages,
on the home page, and in the ‘results’ section of
the system. They can also be exported as reports.
 include their ‘outcome harvest’ documentation in
their twice-yearly internal reports (again done
through our intranet system).
 use this material for their donor reports, and case
studies, and for their part in our PPA reporting.
Staff have access to guidance materials, agendas
and session plans for triannual or biannual reviews,
on-line and off-line outcome forms, and every other
team’s outcomes via Karacel.
When a mid-term review or evaluation is being
planned by a team, and will involve an external
person, we encourage the team to undertake a
documentation exercise to ensure that we have a
dataset to provide for them to analyse and test. On
top of normal documentation this can include:
 completing an updated outcome harvesting
exercise with teams, partners and/or communities.
 completing extended ‘outcome forms’ which have
more detail in them, and are used to gather
Page 6 of 12
A ‘smartie map’ maps the significance and
contribution of changes seen in one project in South
Sudan.
evaluation data for community security action
plans.
What we have found is that by attending to the detail
of outcomes per social actor, at different levels of a
social system, we can create more comprehensive
theories of change for our work. We piloted this with
our Central Asia evaluation in 2012.
“The team and I went back over all our activity reports
(partner reports, planning and review meeting notes,
staff activity reports) and filled in ‘outcome forms’,
which stated and analysed any changes in behaviour
and relationships that we had noticed, for each
community that we worked with and for each
community security cycle that we had facilitated with
them. We also attached all the evidence that we had
collected. This was hugely enlightening for the
programme (and also motivating!) in terms of us all
seeing how each community had learnt to work on
more sensitive issues and how difficult relationships
had changed over time. This was a really useful
process to go through and not only resulted in a very
useful external evaluation that asked the right
questions and gave us helpful recommendations, but
also helped us to articulate our programme aims far
more clearly to partners and everyone else we were
trying to influence.”
– Maija Paasiaro, then Regional Conflict /Security
Adviser
Since then we have created a comprehensive MEL
framework for this and our South Sudan programmes,
based on mapping desired outcomes per actor,
theories of change, and a detailed plan for data
collection. We have provided outcomes harvested
through our monitoring to external consultants
conducting reviews of our Central Asia, Bangladesh,
South Sudan, Yemen, and Sudan programmes, for
them to test and verify.
“The... big thing for me was the M&E framework. It so
clearly articulates all the intangible things we need to
measure”
– Katie Morris, Europe and Central Asia Team
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
Ayak Mou Kuot, Deputy Chairperson of a
community security working group in Warrap state,
South Sudan, talks to group members about safety
concerns and changes they are seeing in their
community. Photo: Marcus Perkins/Saferworld
However, as Saferworld does all its programming
work with and through partners, any approach we use
has to be something we can ask our partners to sign
up to as well. This means another layer of people,
organisations and understanding to develop when
trying to systematise and institutionalise an approach
to MEL that links up across communities, partners
and staff.
“It is a participatory tool that enhances quick
identification of changes with less effort – no
scratching the head when generating outcomes any
more. It generates adequate information on complex
behaviour change and relationship among many
actors and serves as an analytical tool – by
disaggregating outcomes per actors, and outcomes
per level of influence (local, sub national, national,
regional and international). As a process one cannot
exhaust harvesting outcomes because staff become
sensitive to any emerging, growing and mature in
different actors during the project implementation, it
triggers cross-learning among the team and across
projects and it can demonstrate whether Theory of
Change (ToC) is working at early stages of the project
implementation.
– Thomas Kimathi Nyagah, MEL Officer, Kenya Team
Page 7 of 12
Learning from pioneers in South
Sudan
Our South Sudan programme has grown over the
last four years from work in three locations in two
states, to ten locations in eight states. The team
has grown from a staff group of 6 in 2011, to over
40 staff, with eight project coordinators based in
the field.
The first three locations have been running for two
years longer than the most recently added,
providing an interesting comparison between the
two. This was not intentionally set up as a ‘phased
approach’ but has turned out that way, as our
funder has seen the fruits of the work.
All staff have now come together three times to
‘harvest’ outcomes. They spend up to two days
looking at their conflict contexts, listing and refining
their statements about what others have done
differently, discussing the significance of these
changes, highlighting the areas that have potential
but are not yet outcomes, and those that have
hints, or foundations of sustainability.
“What was really noticeable this time was how the
second ‘batch’ had really learned something about
how to make bigger changes from the pioneers.
While they still had to put in the early leg-work,
developing trust and relationships with local
communities, they all had a sense of what might
be possible once the foundations were laid,
because they could see how things had changed
over the previous two years in the first states.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
They had understood how the way of working
significantly affects the outcomes, and had
managed to build that in from the start.”
– Madeline Church, Head of
Organisational Development
“‘Due to the follow up on actions/responses made
by individuals or institutions like police for
example, representatives/leaders of those
institutions often think twice before pledging
responses/actions for a community
security/safety’s concerns as our outcome
monitoring and harvesting mechanism requires us
to make a follow on those pledges and changes
registered by programmes.”
– Peter Machar, Project Coordinator,
South Sudan
In such a conflict-affected environment what seem
like positive changes can easily generate
additional unforeseen negative effects. The
outcome monitoring approach allows Saferworld to
surface these early, and discuss how to add in any
further strategies to mitigate them, or adapt the
work – building in an important conflict sensitivity
lens.
“What does it add to our cross-organisational
learning? – “We have a tendency to get sucked
into the places where we work, so when we come
together and use this as a process to talk about
change and about how we achieve those changes,
..to do that in a collaborative way then inspires
other people to think about their work differently,
and think about different approaches they might
integrate into their work… it’s just been a really
useful collaborative and facilitative tool that really
engages people in those really important learning
discussions.”
– Ariana Martini, Grant Manager
The benefits
A number of benefits from this outcome monitoring
and harvesting approach have become clear over the
last few years.
 It reinforces that regular MEL practice is a job that
everyone needs to be involved in. It shows that it
can contribute to improved programming, and that
all staff have a role to play in collecting and
analysing data and evidence that shows changes.
It motivates staff, some of whom have actually
been heard to say that they really like M&E, and
finally see how it relates to their programme
development.
“It made me realise the absolute importance of M&E,
and how it is really a key programmatic tool, rather
than something that is added on for bureaucracy or
for donors more broadly. And it’s only when people
Page 8 of 12
don’t understand that point that M&E suffers. I think
that comes from constant engagement and
discussions, and the use of tools that show the
value.”
– Katie Morris, Europe and Central Asia Team
 It enables staff to become more expert at spotting
change, and finding ways to increase potential.
Our South Sudan team, for example, have started
keeping notes about things that are not yet quite
outcomes, but could be, a kind of ‘Outcome
Watch’, which is a way of keeping an eye on things
that have potential. The Nepal team discuss
changes they are seeing at their weekly staff
meeting, to ensure the whole team understands
what the programme is seeking to achieve.
“As a manager and mentor I believe that outcome
harvesting has motivated and empowered my team
members to more successfully implement their
programmes and more effectively engage with
partners and stakeholders at all levels of our
intervention.”
– Tamara Duffey-Janser, Head of Programme,
Great Lakes and Sudans
 The approach is appropriate for our model of
‘community security’, which prioritises community
empowerment, and the need for citizens to take
more control of their own agenda through active
involvement.
“The outcome harvesting approach forces us to focus
on the connections between what we are doing and
how things have changed for communities. This
means that we focus on what truly matters, which is
how our work meets the needs of communities.”
– Deepti Sastry, Impact and Accountability Adviser
“I like it because it is very simple, it focuses on the
community, and the impact of the project on the
community, while the approach that I have used
before it only focused on the numbers of trainings
conducted, the number of participants attending, but
our approach is really very good, cause it focuses on
the changes that is taking place in the lives of the
individual and the communities where we are
working, that is really the aim of the project, to work
with the local authorities, to reduce the incidences of
insecurity happening in the community.”
– Phoebe Egbalia Manza, Project Coordinator,
South Sudan Team
 The straightforward approach, with in-built time for
reflection and discussion, is critical for conflict- and
gender-sensitivity. We use our conflict analyses to
determine the significance of the outcomes. This
allows for learning and adaptation by frontline staff.
“I think one of the things that I really like about it
compared to other systems is that it’s very easy to
use, it’s not technical like other systems.”
– Bonita Ayuko, Project Coordinator, Kenya Team
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
 In highly volatile environments, it enables the
teams to take corrective action and understand
where perverse incentives or negative effects
might be occurring for some, from what look like
positive results for others. This reinforces the
importance of being context-and gender- specific.
 It challenges all staff and partners to think about
evidence and evaluation, and not just leave it to
‘evaluators’ to work out whether the programme is
making a difference or not. The format of thinking
through the outcomes ensures that teams state
clearly the claims we are making, and the evidence
we have, which can then be verified or tested by
outsiders.
“I think it has definitely helped people to focus more
on actual changes rather than outputs. That helps to
make us more ambitious as we’re more aware that
people just saying nice things about our work doesn’t
constitute an outcome. At the same time it can be a
bit demoralising as it makes you realise it takes ages
sometimes for any outcomes to materialise,
especially when you’re aiming for policy change.”
– Hannah Wright, Gender, peace and security
Adviser
“It constantly reminds me to be on the lookout for
changes in behaviour, content, patterns, etc. as
opposed to assuming that the amount of work I do
equates to success. It helps because having had to
continually go through this process, I'm usually one of
the people in the room who has a positive answer at
hand when the question comes up ‘how do we know
what we're doing works?’ It's amazing how much that
sets us apart”
– David Alpher, Washington Associate
 It provides useful documentation for our internal
and external reporting, for identifying good material
for case studies, cross organisational learning, and
for external reviewers and evaluators. It is
invaluable material for our cross-organisational
grant reporting, such as for strategic funding, which
requires an overview of the kinds of changes we’re
seeing in very different contexts.
“Outcome harvesting not only helps teams to identify
the changes they are seeing, but also provides the
starting point for wider stories of success and learning
that we can share externally. If anything the challenge
now is the sheer amount of material available – a very
different situation to four years ago!”
– Simon Moore, Head of Communications
Page 9 of 12
Identifying policy success
For our Advocacy and Policy work, the approach
has been much easier to incorporate. These
teams regularly review their outcomes together,
and keep an ‘evidence box’ that helps to
substantiate their claims.
As an initial foray into systematising reporting on
our advocacy and policy work we developed a
matrix to track six types of evidence linking to five
stages of success. Our outcome harvesting
approach then allows us to see how we are
managing to move through these stages, from
improved credibility, to relevance, access to
decision-makers, increased support for our
messages, to policy change.
“Although the impact of such high-level advocacy
can be difficult to measure, outcome monitoring
allows us to reflect on whether the advocacy
approach we have adopted and the activities we
have carried out have been relevant and effective.”
– Kloe Tricot O'Farrell, EU Advocacy Officer
Our learning
Over the past few years we have learned a lot about
how to implement the approach sustainably.
 Clear, simple language does wonders to kick-start
meaningful conversations about change.
“The definition of outcome here is very simple and
everyone is able to think about the outcomes of their
projects.”
– Posha Raj Adhikari, MEL Coordinator,
Nepal Team
 It takes a lot of repetition, and practice, rather than
just ‘training’, to shift attention solely from activities,
so that thinking about change and evidence in this
way becomes natural. Establishing a routine and
systems is really important – you can’t do this once
and expect it to ‘work’.
 Staff training is also needed. It’s useful if this
includes seeing the approach in action in other
teams and participating in the reviews those teams
hold.
 You need people in place who understand the
approach, and work with and support teams, at the
ground level, to make sure that the focus is right.
This is hard to do from a distant HQ or centre.
Having country-level MEL coordinators or advisers
is essential – where we have them, the approach is
much more systematically applied.
 Working through outcomes with partners, in
conversation with them, rather than expecting them
to complete reports in writing, produces much
better material. Partners often have a clear
understanding of what others have done
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change
differently, but they don’t realise that that is
important. They have been encouraged in the past
to report mainly as an accountability exercise led
by a need to ensure transparency about the money
they’ve spent. This has led to an obvious link
between activity and expenditure, with less of a
focus on outcomes.
 Bringing front-line staff and partners into wider
conversations with others in the organisation
substantially increases what we can learn from our
work. Saferworld’s focus on working at several
levels, and influencing multiple actors, means we
need a way to bring evidence together from across
our programming, policy / advocacy and
communications work. Where we can, we
encourage our policy and advocacy staff to join
programming outcome harvesting sessions, and
vice versa. It requires all staff to think ‘MEL’, and to
make time for analysing and processing their work
– something that can be a challenge.
“I find it innovative. Many different MEL approaches
that I’ve used in the past tend to stay at the output
level, not really digging deeper into whether there are
changes happening because of their work... the
approach is essentially a qualitative tool to measure
changes, and it also helps us as project coordinators
in the field, to keep an eye on what we need to do in
order to motivate our stakeholders in realising the
changes that we’d like them to embrace. I find this
more enriching, as many other models tend to stay on
outputs.”
– Galdino Joseph Sakondo, Project Coordinator,
South Sudan Team
Page 10 of 12
Saferworld and Bangladesh partner BRAC at a
joint advocacy workshop: understanding the
outcome harvesting process has taken time, but
BRAC is now using it within their strategic
planning. Photo: Shoeb Ahmed.
Partnership working in Bangladesh
Saferworld works with 16 community groups in five
districts in Bangladesh, with a very large partner,
BRAC. BRAC has its own systems, and our
challenge in this work was to influence the way in
which they conceptualised their M&E in relation to
this project. This involved engaging with and
training the BRAC project staff, their M&E
department, and the leaders of the Community
Empowerment Programme. Their monitoring
system focused almost exclusively on quantitative
data collection at the level of activities and outputs,
but didn’t really have a process for capturing
outcomes, discussing them, and deciding how to
build on them.
We worked to adapt reporting mechanisms, train
the project staff, conduct annual learning events,
and draw in the senior management to
understand, see the value and advocate for the
approach. We worked intensely to develop
extended outcome monitoring for all the
communities, and build a dataset for each
community action group that focused on the core
changes the communities themselves wished to
see. We made all this available to our external
reviewer.
“It took at least one year cycle to understand the
whole thing, which was a very rigorous process.
We were in the field first, we sat down beside the
partners to support them, and then they did it on
their own. At the end of this year, through
practising, we identified that this is a really, really,
powerful tool, to show what can be documented,
what can be visible, and how people can at least
understand how the change process is on-going.
The partner BRAC also marketed it, sharing it
internally, and externally to the donor. And now
they’re thinking about this in their strategic
planning, so they’ll replicate this. It’s not an easy
process, it needs time, and it needs both parties.
But it’s an empowering tool.”
– Bibhash Chakraborty, Bangladesh
Programme Manager
We have also identified a number of internal and
external factors that made the change in approach
possible.
 The new strategic plan highlighted just how much
our work is about influencing different sets of
actors. This was a powerful catalyst to find a
complementary approach in our MEL.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 11 of 12

 PPA funding, and a commitment to

institutionalising a ‘framework’, meant that the
organisation set up an Organisational
Development Unit, and employed dedicated MEL
staff. This unit led on the development of Karacel
(the intranet), and ensured that it had in-built
potential to manage our MEL work and make data
more widely available across the organisation. The
ODU has also led on coordinating and facilitating
our Community Security Learning and Practice
Group.
 Cross-organisational grants allowed for
experimentation, and enabled learning groups to
run for long enough (four years). Behaving differently: the chief of the district police
 There was important buy-in and commitment from addresses the community's local security concerns
senior management to collect and use evidence to and the need for better police-public partnership in
support learning. Core funds were allocated to an event organised by a community security
support this learning among frontline staff. The working group in Kalika, Bardiya, Nepal. Photo:
Executive Director truly felt that this approach Saferworld/Anil Poudel
could have a transformative effect.
 A new communications team supported the focus Stories of success in Nepal
on plain English and the use of uncomplicated In Nepal, we have used the outcome harvesting
language. focus to help us develop stories of success that
communities themselves wish to communicate.
 Saferworld staff were open to trying new things.
The very simple focus on who, what, where, when,
significance and contribution allows for groups of
Looking to the future people to sit together and describe, in quite rich
detail, the nature of the change and the
The wider success of this approach into the future importance they attach to it.
also relies on donors seeing us as partners, and
This ‘formula’ is something that USAID, our Nepal
using the evidence we provide them with to see the
programme donor, is proposing now to learn from
quality of our work; with evaluators able to test it.
to improve the case studies that they ask all their
Funders also need to enable peacebuilding and partners to contribute (and is an ‘outcome’ for us!).
conflict prevention organisations to take a flexible,
“Oftentimes you can get boxed into the project and
adaptive approach to programming, based on
not have the whole team discuss what we’re really
demonstrable changes linked to objectives and
doing in Nepal, how we’re looking forward to
theories of change, rather than a predefined and
change, or bringing about change in the
linear activity, output, outcome framework. While the
communities, whereas this outcome harvesting
target-driven approach may work well in command
process really forces us to think about that, it’s
and control input-output interventions, where
more than just a log-frame with indicators, it’s
strategies are known, and processes are known to
more about coming together and setting time
lead to certain outcomes, this is generally not a
aside, to really think about, ok, what’s the impact?”
helpful structure for peace-building and conflict
prevention interventions working in conflict-affected – Ojaswi Shah, Project Manager, Nepal Team
states.
Saferworld needs to continue building on how we use
this data and information actively to communicate and
work with communities, our partners, and our donors
to improve our programming, and to influence policy
agendas both nationally and internationally.
We need to involve our community partners more in
monitoring the changes they want to see, and
encourage them to use their influence with decision-
makers. For further information, discussion, input and
feedback, please contact Madeline Church at
We need to continue to do things a little differently. Saferworld, mchurch@saferworld.org.uk
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 12 of 12

How does this approach feed into and complement other MEL-related elements
that Saferworld has put in place over this same period?
During 2011-15, Saferworld has put in place a set of new components in our planning, content and strategy
development, learning, and evaluation that complement the focus on behaviour and relationship change.
 Theory of change: we have had a number of goes at knitting theory of change approaches into our
focus on key actors, and changes in behaviour and relationships. Through many iterations, we have
found that teams find it easiest to examine their inherent assumptions about change by starting out
articulating the changes that they wish to see in each actor category in the form of outcome (challenge)
statements (adapted from Outcome Mapping). We then work outwards to investigate why what we do
works to create such changes. The outcome harvesting approach helps us to see whether this ‘theory’
is born out in practice, and hopefully helps us to articulate our theory better.
 Gender-sensitive conflict analysis, plus actor mapping: our conflict and insecurity analysis is used
not only for our own strategy development, but also to do assessments at community level for each
community we work with. Mapping the influence different actors have in each environment is crucial for
working for more sustainable, systemic change.
 Policy/Advocacy Matrix: this identifies five stages of success (improved credibility, improved
relevance, improved access to decision-makers, increased support for our messages, policy change),
using six types of evidence.
 Our Community Security Learning and Practice Group has had a significant learning effect across
the organisation by bringing together our project officers, coordinators, policy, advocacy, and
communications staff. We learn together about how best to work with communities on conflict, insecurity
and empowerment. We have consolidated this learning into our Community Security Handbook.
 Cross-organisational participation in evaluation teams: we enable, where we can, those working in
one region to join any evaluation team in another. We develop internal Terms of Reference for these
staff, based on a focused aspect that we want to learn about, and share, from one location to another.

About Saferworld
Saferworld is an independent international organisation working to prevent violent conflict and build safer
lives. We work with local people affected by conflict to improve their safety and sense of security, and
conduct wider research and analysis. We use this evidence and learning to improve local, national and
international policies and practices that can help build lasting peace. Our priority is people – we believe that
everyone should be able to lead peaceful, fulfilling lives, free from insecurity and violent conflict.
We are a not-for-profit organisation with programmes in nearly 20 countries and territories across Africa, the
Middle East, Asia and Europe.
Saferworld – 28 Charles Square, London N1 6HT, UK
Registered Charity no 1043843
Company limited by guarantee no 3015948
Tel: +44 (0)20 7324 4646 | Fax: +44 (0)20 7324 4647
Email: general@saferworld.org.uk
Web: www.saferworld.org.uk