Research Evidence in The Humanitarian Sector

Research Evidence in
the Humanitarian Sector

A PRACTICE GUIDE
ACKNOWLEDGEMENTS
We are very grateful to those who participated in the Evidence Lounge in Geneva,
Switzerland in September 2017 at which this guide was discussed, provided comments
on earlier drafts of the Guide or did both: Prisca Benelli (Save the Children), Kate
Bingley (Christian Aid), Jonathan Breckon (Alliance for Useful Evidence), Alyoscia
D’Onofrio (International Rescue Committee - IRC), Tarah Friend (UK Department
for International Development - DfID), Tara Karul (International Initiative for Impact
Evaluation - 3ie), Paul Knox-Clarke (Active Learning Network for Accountability and
Performance - ALNAP), Flurina Kuhn (UK Science & Innovation Network), David
Loquercio (Core Humanitarian Standard on Quality and Accountability - CHS),
Tim Martineau (Liverpool School of Tropical Medicine - LSTM), Virginia Murray
(Public Health England - PHE), Alice Obrecht (ALNAP), Aninia Nadig (Sphere Project),
Adina Rom (ETH Zurich), Rodolfo Rossi (International Committee of the Red Cross
- ICRC), Eva Svoboda (Overseas Development Institute – ODI, and Humanitarian
Policy Group - HPG), Pierluigi Testa (Terre des hommes), Erik Von Elm (Cochrane
Switzerland), Karin Wendt (Humanitarian Exchange and Research Centre - HERE).
Furthermore, the following organisations contributed examples (including case studies)
and other content: 3ie, ALNAP, Christian Aid, Evidence Aid, International
Committee of the Red Cross, International Rescue Committee, Liverpool School
of Tropical Medicine, London School of Hygiene and Tropical Medicine, Overseas
Development Institute, Public Health England, Save the Children, and ShelterBox.
CITATION FUNDING
This document should be cited as: Funding for this document was provided
Blanchet K, Allen C, Breckon J, Davies P, by the UK Science & Innovation Network
Duclos D, Jansen J, Mthiyane H, Clarke M. (represented by SIN Switzerland, British
(2018) Using Research Evidence in the Embassy Berne).
Humanitarian Sector: A practice guide.
London, UK: Evidence Aid, London School CORRESPONDENCE
of Hygiene and Tropical Medicine and Correspondence about this document
Nesta (Alliance for Useful Evidence). should be sent to Karl Blanchet, Director
of the Health in Humanitarian Crises
AUTHORS Centre, London School of Hygiene
This document was written by Karl & Tropical Medicine, Tavistock Place,
Blancheta, Claire Allenb, Jonathan London, UK; and Mike Clarke, Research
Breckonc, Phil Daviesb, Diane Duclosb, Director of Evidence Aid, Centre for Public
Jeroen Jansenb, Helen Mthiyanec and Health, Queen's University Belfast, ICS
Mike Clarkeb (a Health in Humanitarian Block A, Royal Hospitals, Belfast, UK.
Crisis Centre, London School of Hygiene Karl: Karl.Blanchet@lshtm.ac.uk
and Tropical Medicine; b Evidence Aid; Mike: mclarke@qub.ac.uk
c Nesta, Alliance for Useful Evidence).
Cover Photo © Patrick Brown/UNICEF/Panos Pictures
1 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

THE WORK ON THE GUIDE WAS CO-ORDINATED
BY THREE ORGANISATIONS:
Evidence Aid (www.evidenceaid.org) A four-year research and capacity-

champions an evidence-based approach building programme, RECAP, was
in the humanitarian sector. Its mission is launched in 2018 and has been focusing
to alleviate suffering and save lives by on decision making and accountability
providing the best available evidence in response to humanitarian crises
on the effectiveness of humanitarian and epidemics. The RECAP project is
action and enabling its use. Evidence supported by UK Research and Innovation
Aid has developed a range of resources as part of the Global Challenges Research
which are available free of charge from Fund, grant number ES/P010873/1. The
its website. These include systematic Centre also offers courses for practitioners
reviews of interventions for disasters, through modules on conflict and health,
humanitarian crises and other major short courses and a free online MOOC on
emergencies. Evidence Aid organises Health in Humanitarian Crises.
events to promote the use of evidence
in the humanitarian sector, including Nesta (Alliance for Useful Evidence)
a yearly Humanitarian Evidence Week (www.alliance4usefulevidence.org) is a
in November in collaboration with the global innovation foundation, based in the
Centre for Evidence-Based Medicine at UK. It backs new ideas to tackle the big
the University of Oxford, UK alongside challenges of our time, from the pressures
Evidence Lounges. Evidence Lounges of an ageing population to stretched
bring practitioners and members of the public services and a fast-changing jobs
academic research community together market. It has produced several guides
to enhance collaborations and develop relevant to the use of evidence to meet
practical ways of using evidence in the this challenge and this publication draws
humanitarian sector, including this guide. upon one of these: 'Using Research
Evidence: A Practice Guide'. That guide
Health in Humanitarian Crisis Centre was produced by Nesta's Innovation
(http://crises.lshtm.ac.uk) at the London Skills Team and the Alliance for Useful
School of Hygiene and Tropical Medicine Evidence and the present guide for the
generates primary research and training humanitarian sector was developed in
on public health in humanitarian crises, consultation with them and with grateful
working closely with international acknowledgment to Nesta for allowing us
humanitarian agencies and research to use some of their content.
centres in affected countries to address
critical health challenges.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 2

BACK TO
CONTENTS
CONTENTS
INTRODUCTION 4 Beyond experiments:

new approaches to evaluation 32
Who might use this guide? 5
How to use this guide 5 Was it worth it? The value
of cost-benefit and cost-
effectiveness analyses 32
SECTION A: What is evidence-
informed decision making, and Key messages from part 1
why focus on research? 6 of Section C 34
What is ‘evidence’ and why Part 2: How do you judge
do we focus on research? 7 the quality of research? 35
The challenge of relying solely Peer-review: how does it
on professional opinion 9 help decision makers? 35
Key messages from Section A 10 Defining ‘high-quality’ research 35
High-quality qualitative research 37
SECTION B: When can Avoid cherry picking by using
evidence help you? 12 systematic reviews 37
Humanitarian Response Cycle 12 Meta-analysis 39
Developing and implementing Forest plots 39
a new intervention 14 Rapid evidence assessments 40
Adopting ideas from other The importance of repetition
regions or sectors 18 and corroboration 40
Creating a theory of change 19 Review of reviews: how to
Key messages from Section B 19 judge high–quality bodies
of evidence 41
Key messages from part 2
SECTION C: What evidence
of Section C 42
should you choose? 21
Part 1: Different types of research
methods, designs and approaches 23 SECTION D: Where should
you look for evidence? 43
Experimental research –
why all the fuss? 28 Searching for research evidence 43
Experimental research designs 28 Sources of systematic reviews
and other research for the
Practical and ethical issues
humanitarian sector 44
with randomised trials 30
Key messages from Section D 45
When randomisation is not
possible or acceptable 30
Quasi–experimental designs 30 ENDNOTES
References and endnotes 46

BACK TO INTRODUCTION
CONTENTS
Introduction
Evidence coming from research and evaluation can help
you understand what works, where, why and for whom.
It can also tell you what does not work, and help you
avoid repeating the failures of others by learning from
evaluations of unsuccessful humanitarian programmes.
Evidence can also guide the design of the most effective
ways to deliver specific interventions.
Evidence challenges what we might investing money to solve a problem is

think is common sense, perceived or supposed to do some good, we need
received knowledge. to ask ourselves if that money could be
For instance, it was long believed that spent more effectively elsewhere, for
severe acute malnutrition required example on a different humanitarian
specialised treatment in inpatient intervention. Research evidence can
facilities with therapeutic products, help you make informed choices on
even though this model posed many the most effective interventions to
challenges to effective treatment for deliver in an emergency context.
both health systems and patients. In the Evidence is a contested field, with
mid-1990s, ready-to-use therapeutic differing opinions on what should be
food was developed. In 2000, initial pilot most valued or deemed most relevant
projects began to test the Community to decision makers.3 However, for the
Management of Acute Malnutrition purpose of this practice guide, we
(CMAM) approach during humanitarian emphasise research that is underpinned
emergencies.1 It was found to be so by scientific notions of proof, validity,
effective that it was endorsed by reliability, and has minimised bias. Such
United Nations agencies in 2007,2 and research has the advantage of rigour,
is now considered the standard of care relevance and independence. We focus on
for managing acute malnutrition in population studies that aim to generate
emergency and development contexts. average answers, which then need to
As financial and other resources are be contextualised, for example taking
limited and often insufficient in the into account cultural factors (such as
humanitarian sector, we cannot afford the use of male doctors to examine and
to waste such resources on policies treat female patients), and issues of
and programmes that do not work. feasibility (such as the ability to deliver
Interventions in any sector can have an intervention in the aftermath of a
both positive and negative outcomes, disaster when access to populations or
and both intended and unintended resources might be especially restricted).
consequences. Even in cases when

BACK TO INTRODUCTION
CONTENTS
We have created this guide to help you HOW TO USE THIS GUIDE
make best use of research evidence when
you are in a humanitarian emergency The guide is divided into four main
or when you are planning for the next sections:
emergency. Our intention is to help you
find and use evidence on interventions,
actions and strategies that might SECTION A
help you make informed choices and What is evidence-informed
decisions. This guide is not about how decision making, and why focus
to generate more research evidence. It on research?
is about using and understanding what
evidence exists and recognising when This section discusses what we
mean by evidence-informed
good evidence is lacking. It should help
decision making, and why research
you build your confidence in compiling, is an essential element of it.
assimilating, distilling, and interpreting a
strong evidence base of existing research,
and think about how you might go
on to evaluate your own projects and SECTION B
commission research or evaluation. When can evidence help you?
This section explores different
scenarios in which using evidence
WHO MIGHT USE THIS can help you, as well as the types
GUIDE? of evidence you might need at
different stages of developing or
This practice guide is primarily implementing a new intervention
aimed at humanitarian decision or policy.
makers and practitioners working
in the field or in the headquarters
of donor, international, national, or
non-governmental organisations. SECTION C
It will help with decisions about What evidence should
the financing, supervision, delivery you choose?
or evaluation of humanitarian This section looks at different types
interventions. It is not aimed at of evidence and examines how to
trained evaluators and researchers, choose the most appropriate for
but instead seeks to foster demand your case. It also discusses how to
for research evidence from wider judge the quality of evidence.
audiences in the humanitarian
sector.
SECTION D
Where should you look
for evidence?
This section offers advice and
resources to help you find the right
evidence to support your case.

BACK TO
SECTION A | WHAT IS EVIDENCE-INFORMED DECISION-MAKING,
CONTENTS AND WHY FOCUS ON RESEARCH?
What is evidence-informed
decision making, and why
focus on research?
This section discusses what we mean by evidence-
informed decision making, and why research is an
essential element of it.
To begin, let us be clear about what of information and recognises the
we do not mean. We are not talking importance of evidence on issues such
about making decisions and choices as feasibility, preference and culture.
by slavishly following rigid research In a field such as the humanitarian
conclusions. Professional judgement sector where more and better evidence
and other sources of information – such is required,6 we could assume that any
as feedback from your stakeholders model of good decision making should
– will always be important. This be wary of relying solely on professional
practice guide is not about replacing judgement that is not supported by
professional judgement but increasing scientific evidence. Later in this section,
evidence use in humanitarian action. you will read about how we can all be
A good start in defining what we mean ‘predictably irrational’ and – consciously or
is borrowed from medicine. More than unconsciously – make errors in important
two decades ago, David Sackett and judgements. We explore how to mitigate
his colleagues proposed the following these errors of judgement in subsequent
definition that has stood the test of time: sections. However, other decision making
models have also stressed the importance
“Evidence-based medicine is the of blending knowledge of evidence with
conscientious, explicit and judicious use of judgement. The humanitarian sector is a
current best evidence in making decisions sensitive area where we need to be aware
about the care of individual patients. The of international and local politics and the
practice of evidence-based medicine means dynamics between the various actors
integrating individual clinical expertise involved in the delivery of humanitarian
with the best available external clinical aid. This will, sometimes, determine
evidence from systematic research”.4 access to evidence and information, but
also how humanitarian aid is delivered.
This attempt to define evidence- However, the importance of evidence
based medicine was not the first,5 but remains and, as noted in an ALNAP
it has been influential and is just as report in 2014, “the failure to generate
relevant to the humanitarian sector and use evidence in policy and response
as it is to other sectors. It stresses makes humanitarian action less effective,
how research can complement less ethical and less accountable”.7
professional judgement or other sources

BACK TO
WHAT IS ‘EVIDENCE’ AND WHY DO Figure A.1 shows the different elements
WE FOCUS ON RESEARCH? that should be part of evidence-informed
decision making. Our focus in this practice
The Oxford English Dictionary defines guide is on the top circle of the diagram:
‘evidence’ as “the available body of facts research and evaluation.
or information indicating whether a belief
or proposition is true or valid”,8 and, As the authors of the Alliance for Useful
similarly, in their ALNAP report on the Evidence’s ‘What Counts as Good
state of the evidence in the humanitarian Evidence?’ report state “The conduct
sector, Paul Knox Clarke and James and publication of research involves the
Darcy defined it as “information that explicit documentation of methods, peer
helps to substantiate or prove/disprove review and external scrutiny, resulting
the truth of a specific proposition”.7 in rigour and openness. These features
We follow these definitions because contribute to its systematic nature
many other definitions tend to be rather and help provide a means to judge the
unhelpful by being overly inclusive trustworthiness of findings. They also
(sometimes including almost all types offer the potential to assess the validity
of information) or by being too abstract of one claim compared to another”.9
and vague.
Figure A.1: The four elements

of evidence-informed RESEARCH AND
decision making EVALUATION
PRACTITIONER
EXPERIENCE DECISION STAKEHOLDERS
AND (e.g. employees),
JUDGEMENTS preferences or
values)
CONTEXT,
ORGANISATION,
ACTORS,
CIRCUMSTANCES
Based on: Barends E, Rousseau DM, Briner RB. (2014) Evidence-based Management: The Basic
Principles. Amsterdam: Center for Evidence-Based Management [www.cebma.org/wp-content/
uploads/Evidence-Based-Practice-The-Basic-Principles.pdf]

BACK TO
This practice guide focuses on research, time. So, decision makers require evidence
but there are many overlaps with the that can be taken ‘off the shelf’ and
field of evaluation and we discuss some combined with information on the local
approaches to evaluating impact and context to inform their choice. Fortunately,
process in Section C. We also give most it is possible to find such evidence and
attention to research that deals with we cover some of the ways to do so in
impact – whether something has had Section D.
positive or negative results – because Research is a process engaged in for
questions on impact are vital to those learning purposes. It seeks to answer
involved in humanitarian action. These questions such as ‘What was the
actors are concerned about showing their commonest type of injury after an
‘impact’ on populations, their ‘results’ in earthquake?’, ‘What are the effects on
international terms or ‘what works’ for gender-based violence of different ways
governments and local and international to protect women and children?’ or ‘How
providers. The language may change, waterproof is a particular material when
but the idea for their research stays the used for shelter?’
same: to see if they have really made
a difference. Therefore, our aim with Evaluation is a process involving the
this guide is to help you decide how assessment of findings and observations
that research might help you choose against standards, for the purpose
interventions, actions and strategies and of making decisions. Evaluations ask
adopt policies that are most likely to questions such as ‘Which types of first
make a positive difference. We provide aid should first responders be trained
illustrative examples throughout the in?’, ‘Which is the best way to protect
guide, and further examples of the use women and children from gender-based
of evidence in the humanitarian sectors violence?’ or ‘What material should be
are available in other collections of case used for making tents in a setting with
studies.10 heavy rainfall?’
We give prominence to research and Research does not necessarily require
evaluation that is ready-made, with no evaluation. However, doing evaluation
need to run a brand-new study. Decision always requires doing research. An
makers have limited time and resources evaluation relates to an intervention
and many simply cannot afford to that was actually implemented, while
commission such a study and to wait for research is more comprehensive and, as
its results to become available, which may well as including evaluations, it can also
take years. Someone needing to make a seek to answer conceptual questions,
decision now, needs the evidence now, if such as when planning for the needs that
not yesterday, not in a year or more years’ are likely after a disaster or developing
We give prominence to research and evaluation

that is ready-made, with no need to run a
brand-new study.

BACK TO
a novel intervention. However, whether picking and retro-fitting the evidence to

we are thinking about research or the conclusions we have already fixed
evaluation, we need to be aware that in our minds. If we are generous, this
bias can distort the findings. In Section can be an unconscious mistake, but at
C, we look at some of the biases that worst it can be a deliberate ploy to back
can affect the results of a study, but it is up prejudice by finding and promoting
also important to be aware of the biases the evidence that fits that prejudice,
in how people interpret the results of a while ignoring the stuff that does not.
study or which influence their thinking A similar but different phenomenon
about the need for research or its impact. is optimism bias, which refers to the
We discuss some of these biases next. “proven tendency for appraisers [of
projects, programmes and policies]
THE CHALLENGE OF RELYING to be too optimistic about key project
SOLELY ON PROFESSIONAL OPINION parameters, including capital costs,
One reason we privilege good quality operating costs, project duration and
research over relying solely on benefits delivery”.12 This can result in
professional judgement is that the biases setting targets for outputs, outcomes
involved in professional judgement and impacts that are unrealistic and
can mean that it is wrong. As humans, undeliverable. Christoplos noted that,
we are ‘predictably irrational’11 and following the Indian Ocean Tsunami in
may experience cognitive biases that 2004 “Methods and guidelines were filled
distort our thinking. Cognitive bias with normative optimism. Terms such
refers to our inability to be entirely as the ‘seamless web’ were applied to
objective, which may manifest itself in describe the states of affairs that would
multiples ways – such as perceptual emerge if the right methods were applied.
distortion, inaccurate judgements and Individuals, households, communities
illogical or irrational interpretations. and nations were expected to follow a
Even with the best intentions, continuum from acute human suffering to
professionals can get it wrong. Take ‘normal’ development. Over the years this
for instance one type of cognitive optimism has faded. LRRD [Links between
error: confirmation bias. This is the Relief, Rehabilitation and Development]
tendency to concentrate on the evidence has come to refer more to a conundrum
that fits with what we believe, and than to a gilded path. Intractable political
to ignore or discount what does not. tensions and human insecurity have
Even highly experienced professionals come to be associated with LRRD, first
can fall into this cognitive trap. in post-conflict situations and then
increasingly in natural disasters as well”.13
Being aware of how we can jump to
conclusions is important for making us
wary of experts. But confirmation bias
also highlights how anybody – not just
experts – can be highly selective in their
use of research evidence. People tend
to look for the evidence that fits their
beliefs, intentionally or not. This might
be thought of as ‘policy-based evidence’
rather than ‘evidence-based policy’: cherry

BACK TO
Optimism bias about both old and new This is not to say that professional
interventions is often due to a lack of judgement is always wrong. Researchers
evidence about their true effects. such as Gary Klein have sung the praises
Research and evidence from evaluations of intuitive expert judgement, for instance
of these interventions, or similar ones, in his work on ‘naturalistic decision
can help to reduce this uncertainty. making’.14 Professional views and gut-
instincts can be highly valuable, but we
Just as in other sectors, there are also must be aware of their downsides. As
many other biases relating to how people Daniel Kahneman asserted in a joint
think that can afflict those working in the article with Professor Klein in American
humanitarian sector. These include: Psychologist, “professional intuition is
sometimes marvellous, and sometimes
Hindsight bias: Tendency to see past flawed”.15
events as being more predictable than
they were before the event occurred.
Loss aversion: Tendency to prefer
avoiding losses than to acquiring gains.
Framing effect: Drawing different
conclusions from the same information
presented in different ways (e.g. would
you prefer that ‘95% returned to work’ or
that ‘5% did not return to work’?).
The ‘availability heuristic’: When people
relate the size, frequency or probability of
a problem to how easy it is to remember
or imagine.
The ‘representativeness heuristic’:
When people overestimate the probability
of vivid events.
The ‘need for coherence’: The urge to
establish patterns and causal relationships
when they may not exist.
Meta-cognitive bias: The belief that
we are immune from biases!

BACK TO
CASE STUDY
How research has promoted the growth of

Cash Transfer Programming
In recent years, Cash Transfer It also helped to tackle concerns
Programming (CTP) has emerged as around cash transfers, such as
one of the most significant innovations corruption and insecurity,18 and
in international humanitarian increased awareness that cash
assistance. generates different, but not necessarily
greater, risks than in-kind assistance.19
In 2016, $2.8 billion of humanitarian The benefits of CTP have been shown
assistance was disbursed through to cut across multiple sectors to
cash and vouchers, a rise of 40% since address women’s empowerment, food
2015 and a doubling from 2014.16 Cash security, education and health care. As
disbursement has not only grown in a result, CTP is now an accepted tool in
total terms, but also as a relative share almost every emergency response.
of total international humanitarian One of the key successes of CTP
assistance, from 7.8% in 2015 to 10.3% research is that the generated
in 2016.17 evidence has been widely used by
This expansion has been supported champion institutions to promote the
by a growing number of evaluations, use of cash in humanitarian crises.
resulting in a body of evidence on More research is nevertheless
the effects of different programmes needed to fine-tune the programmes
on individual and household-level
to maximise their benefits.
outcomes. This body of research and
evaluations has been instrumental in
the growth of CTP.
Key messages from Section A
We do not advocate a form of Results from research are only one

decision making that slavishly follows type of evidence, but have the
the conclusions of research at the advantages of greater rigour,
expense of professional judgement. relevance and independence when
compared to other types of evidence.
However, you should not put
professional expertise on a pedestal. Creating new evidence can be
Experts sometimes get it horribly costly and time-consuming, but there
wrong, and they are not immune from is good quality evidence that can be
the whole range of social and taken ‘off the shelf’ (see Section D).
cognitive biases that affect both Research and evaluations of current
experts and non-experts. emergencies can inform future
interventions.

BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU?
CONTENTS
When can evidence help you?

This section explores different scenarios in which using
evidence can help you, as well as the types of evidence
you might need at different stages of developing or
implementing a new intervention or policy.
Evidence can make organisations HUMANITARIAN

more effective. From more persuasive RESPONSE CYCLE
campaigning to securing grant funding
The United Nations Office for the
and from developing an organisation’s Coordination of Humanitarian Affairs
decision making abilities to making (UNOCHA) developed the humanitarian
sure that programmes deliver results, response cycle to show the coordinated
evidence can bolster your work. It does series of actions needed to help prepare
not matter if you are a small voluntary for, manage and deliver humanitarian
organisation or a large international action. It consists of five elements, with
organisation. Whatever the scale, there each step building on the previous and
may be existing research that can suit leading to the next.20 Evidence can help
your needs. with decision making as you move around
this cycle.
Needs assessment and analysis:
EVIDENCE CAN HELP YOU IN Looking at existing research will help
MANY WAYS, INCLUDING: provide a basis for assessing needs, and
Starting doing something. understanding the nature, magnitude
and dynamics of specific problems.
Stopping doing something. Along with the gathering of new data
from the setting and circumstances
Generating options or making the case. you are confronted with (or expect
Identifying priorities. to be confronted with), it can help to
identify opportunities for responding
Assessing needs. appropriately. In some circumstances, this
might be especially challenging if there
Aligning services with needs.
have been few opportunities for relevant
Designing and implementing more research.21
effective programmes.
Strategic response planning: In planning
Developing funding bids. your strategic response, you should search
for evidence about what has been tried
Creating effective advocacy and assessed previously that might help
campaigns. you to make a well-informed decision.
Increasing accountability to This might also include making use of
stakeholders. evidence on how best to communicate
the potential risks of a situation to those
who might be involved.22

CONTENTS
Figure B.1: Humanitarian Response Cycle
NEEDS ASSESSMENT
AND ANALYSIS
OPERATIONAL STRATEGIC
PEER REVIEW & PLANNING
EVALUATION
COORDINATION
INFORMATION
MANAGEMENT
IMPLEMENTATION & RESOURCE

MONITORING MOBILISATION
Resource mobilisation: The Operational peer review and evaluation:

implementation of a new intervention Operational peer review can be used
or policy will require the mobilisation as a course corrector and to identify
of resources. These might be financial areas for immediate corrective action.
to pay for activities or the personnel Along with an evaluation of your own
needed to deliver them. Having adopted strategic response, and comparisons with
an evidence-informed approach to evidence from similar interventions, it can
planning the strategic response and help to determine whether adjustments
choosing the interventions and policies or improvements are necessary, in
to adopt, should make these more likely particular for leadership arrangements,
to be effective and, as a consequence, implementation of other phases of the
more attractive to donors.23 cycle, coordination and mechanisms
for accountability to affected people.
Implementation and monitoring:
As you implement an intervention or
policy you need to think about how
you can monitor and report on your
impact. This will help you to show
if you are making a difference.

CONTENTS
DEVELOPING AND IMPLEMENTING The spiral below was developed by

A NEW INTERVENTION Nesta to capture the different stages
of the innovation process and can also
When developing or implementing a new be used to plot the progress of a new
humanitarian intervention or policy, it is approach to a social issue.24 Different
a good idea to begin by thinking about types of evidence will be needed at the
timing. Different evidence will be helpful different stages and the process should
at different times in the development or be continuous, such that when you have
implementation and you need to think reached the end of the spiral, you should
about the appropriate research to suit start again and being a new cycle. We
different stages of the lifecycle of the new discuss the various stages of this spiral
intervention or policy. In the early days below, while some specific examples of
of a new initiative, research can identify successful innovation in humanitarian
emerging challenges and the scale of contexts are available elsewhere from
the problem. It can be used to plot the a study of 15 projects funded by the
development and implementation of a Humanitarian Innovation Fund.25
new intervention or policy.
Figure B.2: The Nesta Innovation Spiral
7
1
6
3
5
4
1 Exploring opportunities and 4 Making the case

challenges 5 Delivering and implementing
2 Generating ideas 6 Growing and scaling
3 Developing and testing 7 Changing systems

CONTENTS
1. EXPLORING OPPORTUNITIES AND CHALLENGES

Working through current research will help you to understand the
nature, magnitude and dynamics of the problems in front of you and the
opportunities for responding appropriately. (See case study below.)
CASE STUDY
Improving access to health care in post-conflict settings
With the principle of ‘leaving no-one post-conflict periods and produced

behind’ embedded in the Sustainable recommendations of clear relevance
Development Goals (SDG), and for actors engaging in post-conflict
significant momentum around health systems strengthening. Much
SDG target 3.8 on universal health of ReBUILD’s evidence is relevant for
coverage, international organisations, humanitarian settings – for example
governments, donors, implementers the effects of different approaches
and advocacy organisations agree to engaging local health workers in
that as well as addressing immediate emergency responses on longer-
health needs of vulnerable populations term health workforce distribution,
in crisis settings, this should be done motivation and skills; or the value of
in a way that supports longer-term working with even weak local actors
equitable health system development. during emergencies, to support
The ReBUILD Research Consortium longer-term capacity for prioritisation,
(www.rebuildconsortium.org) is an coordination and planning, including
international research partnership predictable resource flows to support
funded by UK Aid, which has been health system performance. So, as
working since 2011 to support well as engaging in research uptake
improved access of the poor to activities in the study countries
effective health care in post-conflict themselves and other ‘post-conflict’
settings, through the production and contexts, ReBUILD has also sought to
uptake of a coherent body of high engage with those supporting access
quality, policy-relevant health systems to health care in current settings of
research. ReBUILD’s research themes conflict or protracted crises. There
include health worker incentives have been some successful activities,
and deployment, how policies have informing decision making and
affected access to health care for the implementation for government
poorest and most vulnerable individuals and development partners in Sierra
and communities, issues of aid Leone during the Ebola outbreak, and
effectiveness, and the degree to which supporting aspects of a donor’s health
gender equity is addressed in post- support plan for Syria. But engaging
conflict settings. The mixed methods more directly with those working in
research has included qualitative ‘life crisis settings and bringing longer-term
histories’ approaches which have health systems strengthening messages,
produced a rich understanding of where emergency and humanitarian
the experience of communities and needs are understandably the main
health workers through conflict and focus, has been more challenging.

CONTENTS
2. GENERATING IDEAS In Section C, we discuss Standards of

Evidence that show the journey new
After you have identified your focus, it programmes should go on in terms of
is time to search for interventions and providing evidence of impact.
policies that have been studied in the past.
Can you borrow successful ideas from 4. MAKING THE CASE
others about what worked previously and
avoid interventions that failed? Having evidence of the likely impact
of your intervention will help you with
3. DEVELOPING AND TESTING making the case to those who might fund
it and to those who will have to implement
As new ideas are initiated, it will be it in the field. This will put you in a stronger
time to start thinking about testing and position to persuade them to fund the
experimenting with different approaches intervention and to support its delivery
and about evaluating the impact of these. and implementation. (See case study
below.)
CASE STUDY
Presenting evidence to country offices leaders

In June 2017, Christine Fernandes, She then proceeded to correct popular
Save the Children’s global Humanitarian misperceptions with hard data. For
Nutrition Adviser for Infant and Young example, by citing a Lancet article26 to
Children Feeding in Emergencies show that appropriate breastfeeding
(IYCF-E) delivered a presentation prevents the highest proportion of
about IYCF-E to Save the Children Iraq these deaths in early childhood (13%)
country director and senior leadership and that complementary feeding has
team, who were reviewing their 2017- the second highest benefit. Asking
2018 strategy. Up to that point, nutrition people to make a guess, and then
had not been one of the sectors of correcting their views is also an
implementation in Iraq. Christine framed effective way for influencing people’s
her 20-minute presentation as a quiz: opinion, according to evidence.27
each slide started with a question Christine managed to persuade the
such as: whole Country Office leadership that
supporting IYCF-E was the right thing
Which do you think is the most to focus on, leading to a concrete
effective means of preventing deaths change in the Iraq country office with
in children under five? an IYCF-E programme start-up in
Insecticide treated materials. Baghdad and support to the Federal
Ministry of Health on the national
Hib (meningitis) vaccine. nutrition strategy focusing on breast
Appropriate breastfeeding. milk substitute monitoring and capacity
building on a national level in the early
Appropriate complementary feeding. 2018 investment of resources.
Vitamin A and Zinc supplementation.

CONTENTS
5. DELIVERY AND IMPLEMENTATION

When you have implemented an intervention or policy you need to think about how
you can document your impact. This will help take you from a position of rhetoric and
saying ‘trust me, this project is working’ to one based on more trustworthy evidence
from monitoring and evaluation. It will also help with your accountability by showing
that you are making a difference, that your project is value for money, and that you
have opened up your work to evaluation. (See case study below.)
CASE STUDY
Partnering for impact evaluations

The International Initiative for Impact insights from various stakeholders with
Evaluation (3ie) seeks to improve experience working on nutrition and
the lives of poor people in low- and disaster management. The findings
middle-income countries by providing have also been disseminated to an
and summarising evidence of which in-country working group comprising
development interventions work, national and international NGOs,
when, why; and for how much. In academia, and sector specialists.
2018, 3ie’s Humanitarian Assistance
Contribution to improved data
Thematic Window is funding seven
quality: There have been significant
impact evaluations in Chad, Democratic
improvements in the Monitoring and
Republic of the Congo (DRC), Mali,
Evaluation (M&E) system. Best practices
Niger, Pakistan, Sudan and Uganda.
on data collection such as spot checks,
The impact evaluation in Pakistan is
debriefing sessions with field teams and
being conducted by the University of
regular feedback on discrepancies in
Mannheim In Germany, in collaboration
data collection, which were instituted
with the Agency for Technical
during the impact evaluation by
Cooperation and Development
the research team have now been
(ACTED). ACTED supports
replicated in another project being
humanitarian assistance programmes in
implemented by ACTED.
disaster-prone areas and the evaluation
assesses the effectiveness of these ACTED hopes to include these practices
interventions in responding to these in their regular M&E protocol when they
hazards and reducing community have been further tested in the field.
vulnerability to emergency shocks. Building implementing agency
The collaboration between ACTED capacity on evaluation: Staff at the
and the University of Mannheim has implementing agency were sponsored
included several instances of successful to attend a workshop on measuring
capacity building and dissemination: the effect of poverty reduction
Multi-layered engagement: The study programmes and help shape better
and baseline have been presented to policies through the use of impact
the Pakistan government with the aim evaluations. This will enable them to
of informing Pakistan’s integrated better assimilate the findings from the
nutrition strategy by drawing on evaluation, as well as provide a base for
future in-house evaluations.

CONTENTS
6) GROWING, SCALING AND do the shopping. And for women in joint

SPREADING households – meaning they live with their
mother-in-law – as a sizeable minority do,
The ultimate goal for many innovations is then the mother-in-law heads the women’s
to replicate and grow so that they change domain. Indeed, project participation
things on a bigger scale in everyday rates are significantly lower for women
practice and can be successfully copied in living with their mother-in-law in more
other settings. This will increase the scope conservative parts of the country.”28
for positive impact.
He also noted that the right children
have to be identified for the programme
7) CHANGING SYSTEMS to be effective. In the BINP areas, the
Achieving system-wide changes is community nutrition practitioners who
extremely complex. A good way to reach implemented the programme “could not
this scale is to have multiple independent correctly identify from the [growth] charts
studies showing that something does not which children should be admitted to the
just work in the specific circumstances of program”, which led to the mis-targeting
the early testing but is effective in many of children.28 These two examples of
places. inappropriate programme implementation
illustrate the importance of understanding
local contextualized factors in the success
ADOPTING IDEAS FROM OTHER or failure of a programme, and the need to
REGIONS OR SECTORS use qualitative and ethnographic evidence
(see Section C), in order to achieve
Even if you find evidence of success of desired outcomes in different contexts.
a policy or project elsewhere, would
When designing new programmes aiming
adoption of those ideas work in your
at changing how people act – for example,
area? Howard White gives an example
to adhere to International Humanitarian
of the failure of a supplementary
Law or to use latrines instead of
feeding programme in Bangladesh (the
defecating in open areas – research from
Bangladesh Integrated Nutrition Project
social psychology, anthropology and
(BINP)) to achieve the outcomes of
behavioural change can help. It might
“the acclaimed Tamil Nadu Integrated
provide ideas based on robust research
Nutrition Project (TINP) in India”.28 The
about which of the range of techniques
Bangladesh programme had a 30% lower
to ‘nudge’ people towards the desired
participation rate than that in India, and a
actions would be worth adopting.
counterfactual impact evaluation “found
no significant impact of the program on Hugh Waddington and colleagues,29 for
nutritional status, although there was a instance, noted that improving sanitation
positive impact on the most malnourished by getting people to use latrines rather
children”. Professor White noted that: than defecating openly in public spaces
requires more than knowledge of germ
“the people targeted have to be the
theory. Evaluation evidence of Community
right ones. The program targeted the
Led Total Sanitation (CLTS) suggests that
mothers of young children. But mothers
improved technical knowledge of latrines,
are frequently not the decision makers,
the use of subsidies to reduce the costs
and rarely the sole decision makers, with
to households and changing social norms
respect to the health and nutrition of their
and collective action at the village level are
children. For a start, women do not go to
key factors in bringing about behavioural
market in rural Bangladesh; it is men who
change and increasing latrine adoption.

CONTENTS
CREATING A THEORY OF CHANGE handwashing facilities be located? What

should these facilities comprise of? What
In the early stages of developing or information needs to be provided? What
planning any intervention, it is important incentives need to be introduced to
to logically describe what you do and why influence household decisions?
it matters. You need to be coherent, clear
and convincing. This description is often A Theory of Change (example on page
referred to as a Theory of Change, and aims 20) helps you be explicit about your goals
to give a “clear, concise and convincing and how you will achieve those goals. It
explanation of what you do, what impact helps to avoid just simply hoping that your
you aim to have, and how you believe you innovative approach will work and that
will have it. It is a vital foundation of any your assumptions are correct. Another
programme, and a prerequisite for effective benefit of doing a Theory of Change is that
evaluation”.30 it is a first step in designing an effective
evaluation, because it tries to identify
A Theory of Change is a useful way to be all the outcomes that will need to be
more explicit about the evidence that you measured.
are using from others and to be clearer
about how you are going to get results. For The most important message here is to
instance, if you want to run a programme think about appropriateness. You want to
to increase uptake of handwashing in a find research that fits your needs and the
refugee camp, you may want evidence stage of development that you are at. The
that answers a series of questions. evidence you will require in planning a new
What techniques are likely to lead to programme or during the first few months
effective behavioural change? Will the of its implementation will be very different
new handwashing facilities be acceptable to what you will need when the programme
to potential users? Where should the is more established. We revisit this crucial
issue of appropriateness in Section C.
Key messages from Section B

There is a wide range of situations in Early stages of innovation are the
which evidence can help you. This is not time to create a Theory of Change. This
limited to just the obvious ones around is a useful way to be clearer about what
capturing the results of interventions evidence you are using and how you
and policies. There may be other are going to get results. It will also help
benefits that you have not thought you to design an effective evaluation.
about, such as creating more
persuasive campaigning, obtaining Interventions or policies that are
funding or stopping doing something more established can set up their own
that is not working. experiments or evaluations of impact.
As an innovation grows, you should also
You need to think about the timing consider multiple replications to check
and appropriateness of evidence. that the benefits you saw were not just
Where are you on the innovation spiral an isolated success story, but that it can
and what evidence may best suit your work in other places and contexts.
needs? For instance, in the early days of
developing an innovative programme, it It does not matter what size of
may be more valuable to learn from organisation you are in. There is a lot of
what others have already found, research that might be able to help you.
through examining past evaluations or
wider social and scientific research,
rather than doing a large, costly
evaluation of your own work.

BACK TO
CONTENTS
Example of Theory of Change on resilience of communities
Material Human Financial Information

THEORY OF CHANGE
INPUTS
Assumptions
• Aid selection appropriate to context
• 'Do no harm' principle followed
ACTIVITIES
OUTPUT
• Sensitization of non-beneficiary groups
• Shelter items used for intended purpose Preconditions
• Needs assessment relevant & needs Effective & timely
• Beneficiary willing to participate
of vulnerable groups accounted for shelter provision • Local leadership structure in place
• Functional logistics, coordination • Stakeholders support intervention
& communication
Rapid needs
SHORT/MEDIUM TERM OUTCOMES assessment
ShelterBox response is accountable
Protection from weather & environmental extremes
Monitoring Coordination Community Logistics Safety In-depth needs
& reflection & reporting engagement & security assessment
Increased personal safety & security of possessions
Increased protection from water & vector borne diseases

Contribution to
Reduced household displacement
LONGTERM IMPACT LEVEL CHANGES
Physical
Protection
Households and communities are intact OUTCOME • Improved psychological health & wellbeing
• Improved access to & retention in education
Knowledge & skills to utilise the provided materials • Reduced morbidity & mortality
Improved resilience &/
• Security of tenure
SECTION B | WHEN CAN EVIDENCE HELP YOU?
or capacity for self- • Improved access to basic services

Reduced stress & anxiety
recovery (households/ • Strengthened social structures
Improved privacy & retained dignity communities) • Livelihoods sustained

• Community more resilient to future disasters
well being
Psychological
'Outcomes and impacts - measuring the difference we make':
https://cwarham.wordpress.com/2017/02/02/shelterbox-theory-of-change
BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE?
CONTENTS
What evidence should you

choose?
This section looks at different types of evidence and
examines how to choose the most appropriate for the
issue you are interested in. It also discusses how to judge
the quality of evidence.
Not all evidence is equal. For instance, experimental research is

A key message of this practice guide is to more suited to evaluating impact and
think about usefulness. Too many people ‘what works’ (i.e. effectiveness), but other
get into trouble by not thinking clearly types of research can help give other
about what sort of research design, insights.32 They may not look so much
methodology or methods of collecting at impact, but can reveal why and how
data are really going to meet their needs. things are working.33
They tend to be ‘methods led’.31 In other Also, it might be important to establish
words, we all have a tendency to pick our which sections of the population benefit
pet approach, such as questionnaires, most from an intervention, and this
randomised trials or interviews, because can often be masked by relying on the
we are most comfortable with it, rather average effect for the population as a
than being more open to the breadth of whole.
methods and thinking which ones are
best suited to answer our challenge and
resolve our uncertainties.

CONTENTS
CASE STUDY
Lessons learnt from using ethnography in the

Ebola response
The Ebola epidemic in West Africa shed responders as a traditional resistance35
light on the need to foster ‘community to modern medicine, and as a threat to
engagement’ when responding to biosecurity. Involving anthropologists
outbreaks, and on the role played by who had extensive experience living in
networks of social scientists to share the countries affected by the epidemic
material and inform the response. was key to identify acceptable and safe
This was exemplified by the Ebola practices (such as showing a picture
Response Anthropology Platform, of the corpse to families to address
whose members “work proactively with rumours around bodies disappearing36),
health and humanitarian organisations as well as to overcome tensions
to design, deliver and monitor more arising from burial practices through
locally responsive and socially informed collaborative approaches. A key lesson
interventions and research on the that emerged was to identify rationales
ground”.34 Lessons can be learnt from explaining communities’ needs to bury
how anthropologists used ethnography their dead as opposed to labelling
to bring socio-cultural and political these practices as essentially ‘cultural’.37
dimensions in a biomedical response The Ebola response showed “the need
in order to address the issue of so- for new global mechanisms to be
called ‘unsafe’ burial practices. In a established that can rapidly mobilise
context where corpses transmit the all experts who can bring relevant local
Ebola virus, exceptional measures contextual, medical, epidemiological,
were adopted where only certain and political information on global
aid workers were authorised to bury health emergencies”.38
the corpses of the persons who had
died from Ebola. Gatherings of local
communities attending funerals of
loved ones who had died from Ebola
were perceived by the Ebola outbreak
The type of research you chose as the source of evidence to help in your decision
making needs to fit the needs of the challenge that you face.39
Experimental research is more suited to

evaluating impact and ‘what works’.

CONTENTS
PART 1: DIFFERENT TYPES OF RESEARCH

METHODS, DESIGNS AND APPROACHES
So how do you go about Some types of research (such as

understanding which research method experimental and quasi-experimental
is appropriate for your case? designs) are better suited for identifying
the presence of a causal relationship.
If you are not a researcher, it can be
daunting to come face-to-face with Other types of research are more
the jargon and endless lists of different appropriate for explaining such causal
relationships – see Beyond experiments:
approaches.40 These issues are captured in
new approaches to evaluation on page 32.
a How to note from the UK’s Department
for International Development about Some designs (such as observational
assessing evidence41 and are just as studies) are more useful for understanding
relevant to the humanitarian sector as political, social and environmental
they are to international development: contexts.
Some of the pros and cons of various

research designs are summarised in
Table C.1.42
Table C.1 Different designs, methods and approaches to research evidence – a brief overview
TYPES OF
RESEARCH AND WHAT IS IT? PROS CONS
EVALUATION
Ethnography An account by Powerful and Difficult to verify and

someone with immediate; may give may lead to inflation
direct experience vivid detailed insights of prevalence.
of, or affected by, into events concealed Emotive first-person
a particular issue. from much of the stories may inhibit
The objective is to population. critical appraisal
collect in-depth and individual
information on anecdotes may not
people’s experience be representative.
and perceptions.
Case study Detailed and Easy and practical; Bad at inferring

intensive can be used by that an intervention
examination of practitioners and non- causes an outcome;
a single case or specialists; good for small size means
multiple cases such interventions that have hard to generalise
as a community, already happened; to national or
family, organisation, might identify adverse population level.
sites, event or consequences from
individual. intervention; helps to
describe innovations;
generates hypotheses
for other studies.

CONTENTS
TYPES OF
EVALUATION
Case control Compares a group Require fewer Rare in social

studies who have, say, a health resources to policy (see closely
condition with a group carry out than related ‘case-
of people who do not intervention studies; based’ evaluation
have it, and looks back useful when below for approach
in time to see how randomised trials more common
characteristics of the or prospective in social policy),
two groups differ. cohort studies more frequent in
are not practical epidemiology and
(e.g. studies health; provide
of cot death); less evidence for
may generate causal inference
hypotheses that can than a randomised
be tested in more trial; high risk of
robust studies. various biases
(e.g. recall bias,
selection bias and
interviewer bias).
Process An approach to Strong on Lack of agreed

evaluation evaluation that looks at explanation of methods;
what happens compared causes; can be used opportunities
with existing theories in messier areas of for bias; weak
of change or causal social and complex on estimating
pathways identified interventions where quantities or extent
during an evaluation. Can there may be many of impact; relatively
be associated with realist causes and context little evaluation
evaluation; qualitative is important. of the techniques
comparative analysis Process evaluation used in contrast to
(QCA); contribution will investigate the large body of
analysis; process tracing. mechanisms of literature criticising
change, contextual randomised trials.
factors and
implementation
evaluation.
Economic Economic evaluations Economic Economic evaluation

evaluation determine whether evaluation provides involves several
an intervention is an a good sense of the assumptions and
efficient use of resources level of resources sometimes proxy
and how it compares to needed and measures of
other interventions both costs per benefit/ cost, which relies
in terms of their costs effectiveness on judgements.
and consequences. In produced, which The measure of
the humanitarian sector, is a great indicator effectiveness needs
the most common for most policy to be provided
methods are cost/ makers and can be by a robust
benefit analysis and cost- used for advocacy. comparative trial.
effectiveness analysis.

CONTENTS
TYPES OF
EVALUATION
Cross-sectional A representative Quantitative data Establishes

sample of people can be examined association at
surveyed at to detect patterns best, not causality;
one point in or association; rarely possible
time. Although relatively cheap to attribute any
surveys such as and ethical; survey measured change
questionnaires can be repeated at to the intervention,
and structured intervals, illustrating or to understand
interviews are changing trends what would have
commonly used over time (see happened in
in cross-sectional Cohort/Longitudinal the absence of
design, they are studies below); the intervention
not the only way. may generate because any
Other methods hypotheses that can change could
include content be tested in more have been due
analysis or analysis robust studies. to broader issues
of official statistics. such as economic
conditions,
weather, or media
campaigns,
rather than the
intervention. Other
disadvantages are
risk of recall bias,
social desirability
bias, researcher
bias; unequal group
sizes and unequal
distribution of
confounders.
Cohort/ The same sample Best source Data often emerges

Longitudinal of people surveyed of evidence too late for effective
studies over several on association policy–making;
points over between childhood study members may
time, sometimes experience and drop out over time;
from childhood adult outcomes; expensive approach
to old age. can give powerful when maintained
support for over decades.
certain early
interventions; can
be used to evaluate
programmes that
are implemented
following
randomised trials.

CONTENTS
TYPES OF
EVALUATION
Quasi- Compares a group Can provide Inability to ensure

experimental who have, say, a reasonably strong equivalence of
design health condition evidence of the groups and to
with a group of relationship prevent change
people who do not between the over time can result
have it, and looks intervention and in less reliable
back in time to see the measured findings; matching
how characteristics outcomes; powerful techniques tend to
of the two method of exploring require a lot of data
groups differ. the impact of an in both intervention
intervention when and comparison
randomisation is groups which can
impossible; can be be time-consuming
applied to large and expensive to
communities as well collect; a good
as groups; no need understanding is
for randomisation required of the
from the start factors that need
(ex-ante), which to be matched
avoids some of (without this, it
the challenges remains possible
and difficulties of that there are
randomisation. systematic
differences between
the groups that are
not being controlled
for); these designs
require complex
analytical work
and specialist
knowledge.

CONTENTS
TYPES OF
EVALUATION
Randomised One group Offers the most Poor on taking context

trial receives an robust, reliable into account (e.g. cultural,
intervention findings, which institutional, historical
while another give confidence and economic settings);
does not receive that any measured difficult to do at a national
that intervention difference between population level; when
or receives groups are due to used at small pilot level,
a different the intervention; this might not be relevant
intervention; random allocation to national/ population
groups are should prevent level (although this is a
formed using a systematic risk for all designs); can
random process, differences be hard to manipulate
usually with the between groups; variables to experiment
same chance of greater confidence in social policy (e.g. class,
being allocated in the effect size race or where you live);
to either group. and the relationship mistakes in randomisation
between the can invalidate results;
intervention can be hard to persuade
and outcome; decision makers of
internationally benefits of this design;
recognised potential political and
approach. ethical issues over
randomisation (e.g. some
groups randomly getting
potential beneficial
intervention, and not
others); can take more
management time and
more time to set up than
quasi-experiments.
Systematic Aggregation Best source of Requires a sufficient

reviews, meta- of results from reassurance that an number of robust
analysis eligible studies, intervention works studies in a given area;
with the eligibility (or does not); methodology less well
criteria defined meta-analysis pools developed for synthesising
in advance and statistical results; qualitative data and ‘grey’
methodologies large reviews literature; might produce
reviewed carry considerable misleading results if
statistical power; is selective reporting biases
replicable by other affect the research that is
researchers; can be available for the review.
applied to any kind
of data or study.

CONTENTS
EXPERIMENTAL RESEARCH – Instead, to mitigate this bias, we can

WHY ALL THE FUSS? use a process of randomisation: after
the population for the study has been
Experimental research such as identified and they have agreed to join the
randomised trials have received a lot of study, the individuals to be compared with
attention – some might say too much. So each other are allocated randomly to the
why have they attracted so much interest? study groups that will be compared, for
Frequently, practitioners and policy example by drawing lots.44
makers want to know whether an
intervention has had an impact. Has there EXPERIMENTAL RESEARCH
been a causal link between their new DESIGNS
programme and the ‘impact’, ‘effect’ and
‘result’ at the end? They want to be able to Choosing the appropriate experimental
attribute the effects that they have seen design, and conducting the study properly
to the policy or intervention that is being establishes the causal link between an
tested. Experimental designs, even though intervention and an outcome. When
they are sometimes difficult to implement random allocation is used to assign the
in practice, have a better chance of participants to the intervention or the
showing this cause and effect link. comparator group, this provides the most
robust means for determining whether
An experimental study could simply any difference in outcomes between
test and compare two groups of people the groups is due to the intervention
before and after the intervention is or chance, and chance is minimised by
given to one of the groups. But if these doing a large enough study. The random
groups are not formed by randomisation, allocation minimises the possibility of
there is a risk that the groups might systematic differences between the
differ because of selection bias and that groups, regardless of whether these are
differences in their outcomes would due to known or unknown factors.
then be due to these differences in their
characteristics and not to the effects The introduction of a control group
of the intervention being tested. For against which to compare the intervention
example, somebody who asks for the minimises several biases that normally
intervention may be more predisposed complicate the evaluation process. For
to positive outcomes. Perhaps they are example, if you introduce a cash transfer
more motivated, healthier, confident, and scheme to increase immunisation rates
thus more likely to do well regardless and the use of facility-based deliveries,
of any intervention. On the other hand, how would you know whether those
someone wanting the intervention may receiving the extra cash would not have
be the most vulnerable and most likely to used the services anyway? You need to
do badly regardless of any intervention. compare them to a group of people who
These biases will distort the results of do not receive the extra cash.
the experiment and may mean that it As with any research design, the strength
provides a false estimate for the effect of a randomised trial also requires that the
of intervention. Furthermore, the biases study is conducted properly. This includes
might be unpredictable and if we do not ensuring that the randomisation of
know the direction of the bias, we cannot individuals or units (e.g. schools, hospitals,
simply adjust the results of the experiment neighbourhoods etc.) is done properly
to try to take account of them.43 (‘allocation fidelity’), and that only the
experimental group has been exposed to

CONTENTS
the intervention being tested (‘treatment Further information on how to establish

fidelity’). A well-conducted randomised whether a randomised trial has been
trial also requires sufficient number of conducted and reported properly can be
participants and statistical power to found in guidance on the conduct and
ensure that other factors (‘confounders’) reporting of trials.45
that could have caused the outcome The International Initiative for Impact
are balanced between the experimental Evaluation (3ie) identified evaluations of
and control groups. A low drop-out rate humanitarian interventions in conflict and
(‘attrition’) is also necessary to minimise reconciliation situations that have used
the risk of bias. It is important, therefore, randomised trials and quasi-experimental
to check the reporting of a randomised designs (see below).46
trial to ensure that these criteria of validity
have been met.
CASE STUDY
Establishing the effectiveness of humanitarian

interventions using experimental methods
Communicable diseases are of particular disinfectant water treatment and
concern in conflict and disaster- improved water storage (intervention
affected populations that reside in group) to households with only
camp settings. In the acute emergency improved storage (control group).
phase, diarrhoeal diseases can account The allocation of households to the
for more than 40% of deaths among experimental and control groups was by
camp residents. Diarrhoeal diseases randomisation.
result from a variety of factors but, In intervention households, point-of-use
in the emergency context, adequate water treatment with the flocculant–
sanitation and water supply (in terms disinfectant plus improved storage
of both quantity and quality) is the reduced diarrhoea incidence by 90%
primary means for diarrhoea reduction. and prevalence by 83%, when compared
Clear limitations exist in current water with control households with improved
treatment technologies, and few water storage alone. Among the
products are capable of treating turbid intervention, residual chlorine levels met
water. or exceeded Sphere standards in 85%
Shannon Doocy and Gilbert Burnham (95% CI: 83.1–86.8) of observations with
described the findings of a 12-week a 95% compliance rate.
effectiveness study of point-of-use Drs Doocy and Burnham concluded that
water treatment with a flocculant– a point-of-use flocculant–disinfectant
disinfectant among 400 households significantly reduced the incidence and
in camps for displaced populations in prevalence of diarrhoeal disease among
Monrovia, Liberia.47 households residing in camp settings
The study compared diarrhoea rates in the emergency context.
among households with flocculant

CONTENTS
PRACTICAL AND ETHICAL ISSUES WHEN RANDOMISATION IS NOT

WITH RANDOMISED TRIALS POSSIBLE OR ACCEPTABLE
Having a control group leads to a common In some circumstances, it might not be
criticism of randomised trials, which is possible or acceptable to randomly assign
that they are unethical because only some people to the groups to be compared. For
of the population are given access to an example, it might be impossible to use a
experimental intervention that might be randomised trial to compare the relative
considered superior. This ‘rationing’ of the effects of different ways to co-ordinate
intervention is justifiable if we do not know the response to a major emergency or the
whether it will do more good than harm and impact of a national policy intended to
we are doing the trial to find out. However, improve the social inclusion of refugees.50
if there is already convincing evidence In such cases, researchers might use a
that the intervention will be beneficial, we quasi-experimental design to investigate
might still need to do a randomised trial to the link between the interventions and
find out how beneficial it is and for whom. the outcomes. Kayvan Bozorgmehr and
If so, there are ways round the criticism Oliver Razum did this to compare personal
and the ethical challenge. For example, spending on health care among asylum-
in a waiting list control randomised trial, seekers and refugees in Germany who
those allocated to the control group will be had either restricted or regular access to
offered the intervention either at the end health care.51 Similarly, Rodolfo Rossi and
of the trial or when their personal follow- colleagues used surveys before and after
up has been completed. Another option is a vaccination campaign to investigate
the ‘stepped wedge’ or ‘pipeline’ design,48 its effect on vaccination coverage in
in which the order in which participants young children in a crisis-affected area of
receive the intervention is determined Lebanon in 2015.52
at random, allowing it to be gradually
phased in over time so that all participants QUASI–EXPERIMENTAL DESIGNS
have been given access to it by the end There is a wide variety of quasi–
of the trial. This design was used for the experimental designs for comparing the
Progresa/Oportunidades study, one of the effects of interventions and these are
largest randomised trials in international often used when randomised trials are
development, which tested conditional not feasible. Such studies are similar to
cash transfers for the rural poor in Mexico.49 randomised trials to the extent that they
Stepped wedge trials can be highly practical can be used to compare the outcomes
in a world of austerity, where there are for individuals or groups who receive an
not enough resources to do a full roll-out intervention or programme and those
of a new programme all in one go. They who do not receive it. However, the
allow a gradual roll-out to be done while, receipt or non-receipt of the intervention
at the same time, using a randomised trial or programme would not be based on
to evaluate its effectiveness. However, random allocation. Instead, other types of
stepped wedge trials are complex and comparative design, such as interrupted
can be difficult to implement, and if the time series or controlled before-and-after
intervention turns out to be ineffective studies, or analysis, such as regression
or harmful, it will have been rolled out discontinuity or propensity score
to everyone in the trial. In the case of a matching, are used.53 It should be noted
behavioural or educational intervention or that before-and-after designs without a
the restructuring of service delivery, it might comparison group cannot establish the
then be difficult or impossible to remove or impact of an intervention or programme
‘switch off’ the intervention after the trial. because in the absence of a counterfactual

CONTENTS
(i.e. a comparison with what would have greater risk of bias than well-conducted
happened without the intervention), any randomised trials, but they might still
observed impact on outcomes may be allow strong causal inferences to be made
due to factors other than the intervention. in circumstances where a randomised trial
Quasi-experimental designs have a would not be possible or acceptable.54
CASE STUDY
Establishing the effectiveness of humanitarian interventions

using a quasi-experimental design. The Contribution of Food
Assistance to Durable Solutions in Protracted Refugee Situations
This evaluation was part of a series of unregistered Rohingya who did not. The
impact evaluations jointly commissioned primary evaluation question was: What
by the World Food Programme (WFP) are the differential impacts of long-
and the Office of the United Nations term food assistance on the different
High Commissioner for Refugees Rohingya refugee and refugee-
(UNHCR) in 2011 and 2012 to assess affected populations in Bangladesh?
the role of food assistance in, and The evaluation concluded that along
its contribution to, self-reliance and with other forms of external assistance,
durable solutions for the refugee and food assistance was a contributing
the refugee-affected populations.55 factor in households’ choice of
The evaluation covered the protracted economic activity and adoption of
refugee situation in southeast specific coping strategies. Compared
Bangladesh, where approximately with their unregistered Rohingya
30,000 Rohingya refugees have been counterparts, registered refugees
assisted in two official camps for more engaged in significantly different
than two decades. In addition to the economic activities, including higher-
registered refugees, approximately skilled and less risky employment for
45,000 unregistered Rohingya reside overall higher wage rates. They also
in makeshift sites and more than had significantly better wealth status
150,000 reside in host communities based on asset accumulation. Food
in Cox’s Bazar district. To evaluate assistance was an integral component
impact in the most methodologically of their livelihoods, used mainly for
rigorous manner for the context, consumption and as collateral and a
a quasi-experimental design was value transfer for loans and mortgages.
used. The evaluation design focused The value transfer of all external
primarily on three population groups: assistance in the camps enabled
refugees to work less and to rely on this
1. Refugees living in two external assistance in times of crisis.
official refugee camps; Despite these differences, all refugee
2. Unregistered Rohingya living in and unregistered Rohingya groups
two unofficial sites, the official relied on economic activity to
camps or host communities; support their livelihoods. Unregistered
Rohingya employed a wider range
3. Host communities. of coping mechanisms, both positive
and negative, and were a significant
The key quantitative comparison part of the region’s labour market.
was between registered refugees
who received food assistance and

CONTENTS
BEYOND EXPERIMENTS: NEW unlocked or triggered”.61 For instance, in

APPROACHES TO EVALUATION contribution analysis, the research does
not attempt to prove that one factor –
When thinking about impact, we such as a specific policy – ‘caused’ the
also need to think beyond a sole desired outcome, but rather it sets out
dependence on experiments such as to explore the contribution a policy is
randomised trials. Although trials and making to observed results. It recognises
quasi-experimental designs may work that effects are produced by several
well when there is a simple intervention causes at the same time, not just one.
that can be tested, interventions are
often not simple. NGOs working in civil These approaches are not all new.62
society, for instance, rarely work alone Indeed, having a good theory has
or have the chance to manipulate a clear arguably always been at the heart of
experimental ‘treatment’ for a randomised good science. Therefore, using a ‘theory-
trial. Evaluators are looking at other based’ approach is not novel, but
ways of doing things, approaches that these ways of evaluating impact have
involve moving beyond a simple one- grown in popularity and they do help
to-one causality and require careful evaluators address multiple causality.
thinking about your ‘contribution’.
WAS IT WORTH IT? THE VALUE
In the complex world of humanitarian
action, it is unlikely that your programme
OF COST-BENEFIT AND COST-
alone is the necessary or sufficient EFFECTIVENESS ANALYSES
condition for success. It might be just one Having determined and measured the
factor among many - part of a ‘causal effects of an intervention, it is important
package’. Programme success depends for the humanitarian sector, as with
on what else is going on or has gone on any other sector, to know whether its
around you. In the past, a simple causal programmes and interventions provide
explanation of ‘what works’ may have value for money. When budgets are
sufficed but nowadays it is more common constrained, we need to make difficult
for evaluation researchers to ask ‘did our financial decisions about whether one
intervention make a difference? What thing is more valuable than another. Policy
would have happened without it?’ makers, commissioners of services, grant-
There is a range of approaches that making bodies and charitable funders
are becoming more popular with alike are asking for more and better
evaluators, such as theory-based56 impact measures for every pound, dollar
and case-based designs, complexity or euro they spend. Economic appraisal
theory, realist evaluation,57 contribution techniques help create such insights on
analysis,58 process tracing59 or qualitative value.
comparative analysis.60 A common feature There is a range of techniques for doing
of these approaches is a recognition of economic appraisal. A cost effectiveness
the complexity of causality requiring the analysis (CEA) identifies which are the
identification of the causal mechanisms better ways to achieve an outcome.
across a range of specific cases or events, Whilst, cost benefit analysis (CBA) tells
as well as elements in the causal chain us if there are better uses for the available
that vary. This involves “identifying under resources. Cost-utility analysis (CUA)
what conditions, and in what combinations calculates which intervention produces
with other causes, the observed effect of the greatest sense of subjective well-
an intervention is likely to be produced, being for the people affected by it.12

CONTENTS
Wherever feasible, economic appraisal been defined as “the maximum amount

tries to attribute monetary values to all of money an individual is willing to give
the inputs and activities that go into up in order to receive a good”.12 Similarly,
producing the outputs which, in turn, lead willingness to accept (WTA) “is the
to the desired outcomes and their longer minimum amount of money they would
term economic, social and environmental need to be compensated to forego or
impacts. This constitutes the cost analysis. give up a good”.12 The results of a WTP
The same procedures are then used or WTA survey provide what economists
to attribute monetary values to all the call ‘stated preferences’, that is, what
outcomes and impacts that the policy people say they would be prepared to
is hoping to achieve. Where there are pay or accept for some good or service.
negative outcomes and impacts these Alternatively, economists will attempt to
are deducted from the monetary value observe how people will actually behave
of any positive outcomes/impacts that in a real market situation. That is, they
have been achieved. This constitutes the will set up a situation in which people
benefits analysis. are asked to pay different amounts of
Monetary values are derived as much as money for a good or service or will be
possible from where there is some market offered different amounts of money to
activity. The labour market, for instance, give up a good or service, and then record
provides monetary values for different the actual behaviour of people. This
types and units of labour. The building provides what economists call ‘revealed
trade market provides the monetary value preferences’.
(the cost) for goods that are required to In addition to the procedures mentioned
build a hospital or school – bricks, steel, above, economic appraisal also identifies
glass, cement etc. The costs of providing where, or on whom, the costs and
medical care can be derived from the benefits will fall and over what time
fixed and variable costs of running span. The economic value of benefits
hospitals, health centres and community that will be realised in the medium to
clinics, medicines and medical supplies, long-term future will be discounted by
and the like. This is what is meant by what is known at the discount rate. Other
‘monetising’ costs and benefits. adjustments that are made to costs
Some costs and benefits (especially) are and benefits include changes in relative
not traded in a market and are therefore prices, material differences in tax options,
more difficult to monetise. In such cases optimism bias and contingencies such as
economists will estimate the benefits floods, droughts and the effects of climate
(and sometimes the costs) by simulating change. Economic appraisal is a fairly
market activity with ‘willingness to pay’ technical enterprise, usually requiring the
and ‘willingness to accept’ surveys of expertise of an economist.
people. Willingness to pay (WTP) has
Quasi-experimental designs might still

allow strong causal inferences to be made
in circumstances where a randomised trial
would not be possible or acceptable.

CONTENTS
Key messages for part 1 of Section C
Not all evidence is equal. Some is Theory-based evaluation and

better quality and will be more techniques such as contribution
appropriate to your challenge. analysis are increasingly popular with
evaluators. These techniques can be
The type of research needs to fit the helpful when it is difficult to do an
needs of the challenge and whichever experiment or impossible to attribute
method is used it needs to have been your single policy or programme to any
carried out and analysed appropriately. single clear result.
Do not be discouraged by the long It is not just about whether your
list of research methods, designs and intervention worked or not, but whether
approaches. The key thing is to it was value for money.
understand the assumptions that
underpin these methods. Techniques such as cost-benefit
analysis can help you to understand the
Some research designs (such as financial value of an intervention’s
experimental and quasi-experimental impact.
designs) are better suited for
demonstrating the presence of It might be important to establish
a causal relationship. which sections of the population
benefit the most from an intervention.
Other research approaches are more This can often be masked by relying on
appropriate for explaining how such the average effect for the population as
causal relationships come about or a whole.
answering other types of research
question.
It is important for the humanitarian sector

to know whether its programmes and
interventions provide value for money.

CONTENTS
PART 2: HOW DO YOU JUDGE THE QUALITY OF RESEARCH?
Another way to help you choose hidden away. As an example, a survey

which sort of research you need is to by researchers at Stanford University
ask questions such as: what research found that most ‘null studies’ in the social
can I trust? What is good enough sciences are never published: just 20%
evidence to fit my needs? Might it be had appeared in a journal, and 65% had
an article in a high-impact scientific not even been written up.65 This could
journal? What about the in-house have serious consequences for decision
evaluations conducted by my own makers reviewing evidence. If you never
organisation? Should they also have see the negative studies on a new
a place on the evidence table? intervention, you may wrongly conclude
Being published in a peer-reviewed that all is well and that it is effective.
research journal is one way to help you Nevertheless, despite these problems,
feel confident about a piece of research, peer review remains, for most, the ‘gold
but it is no guarantee of high quality. standard’ for deciding what makes it into
In a famous paper, John Ioannidis from the scientific literature: providing a check
Stanford University in the US caused a stir against bad work.66 Using a journal article
by arguing that ‘most published findings or a report that has been independently
are probably false’. He examined the most peer-reviewed by other experts is one
cited papers (those with more than 1000 way of helping you be more confident
citations) in some of the best regarded that you can trust the research. Although
medical journals in the world – largely it can take years before research gets
drawn from The Lancet, the New England published (and that can be too long if
Journal of Medicine and the Journal of you have to make a quick decision), peer
the American Medical Association.63 Of review brings some quality controls to
those with claims of efficacy whose the literature to help you feel confident.
results had been tested in future studies,
41% were either found to be wrong, DEFINING ‘HIGH-QUALITY’
or the impact was much smaller than RESEARCH
the original study had suggested. Peer review may give us some modicum
of comfort. But what do you do if you
HOW DOES PEER REVIEW are going to include evidence that has
HELP DECISION MAKERS? not been checked by other experts?
We must also be mindful that peer How do you decide if it provides a good
review, which is a cornerstone of enough basis for your decisions?
academic journals is far from being It would clearly be a mistake to ignore
perfect. There can be unconscious important research evidence just
biases such as ‘herding’, where the because it had not been published in a
behaviour of reviewers is influenced by peer-reviewed journal. Or to miss the
the behaviour of their peers. And the rich seam of donor or NGO evaluations
whole system of scientific journals can that never get published in academic
be skewed by publication bias: positive outlets but, instead, are available only
results have a better chance of being in the ‘grey literature’. There are a wide
published,64 while negative data gets variety of definitions for grey literature,

CONTENTS
but it usually refers to documents that quality, such as the GRADE68 or Maryland
are unpublished or have been published Scientific Methods Scale69 systems. These
without peer review. It can also refer to approaches to quality assessment for
research that is still underway or being experimental evaluations are usually
prepared for publication. Government based on the studies’ internal validity,
reports, policy statements and briefs, quality of reporting and external validity.
and conference proceedings are also Quality refers to how well studies have
types of grey literature. Grey literature been conducted, reported and analysed,70
is important because it may contain as well as the researchers’ integrity in not
evidence of negative outcomes and distorting or falsifying their data.71 Some
unsuccessful interventions, which is people also link quality to how relevant
important for the balance of evidence- the study is to policy and practice.72
informed decision making. Grey literature
can be searched using electronic When trying to answer a causal question,
databases such as Open Grey (www. you need to consider whether the
opengrey.eu), conference proceedings research design used for a study is
and the procurement records of research appropriate for determining causality and
funders. Websites of organisations that whether the design was implemented
have an interest or expertise in a topic properly in the study. High-quality impact
are another source of grey literature. evaluations will answer questions of
These organisations can be contacted to attribution: showing that the intervention
identify researchers and decision makers caused the outcomes. This requires a
who have expertise in a topic or issue. comparison or control group which is
as similar as possible to the intervention
A good start in trying to appraise the group in all regards expect the actual
quality of evidence is defining it. One of intervention. If this is true and the study
the problems, however, is that phrases has been well conducted, you can be
such as ‘quality’, ‘standards’, ‘robustness’, more confident that, for example, the
‘bias’ and ‘strength’ are often used as if effects on the prevention of violence,
they were interchangeable, and without reduction in family stress or faster return
clearly defining what they mean. This to work are due to the intervention. It
makes for a lot of misunderstanding. For is also important to consider whether
instance, in some guidance,67 research the effects found in the study will be
‘quality’ means using particular types replicated in other places. This drives the
of design and method – such as a demand for mixed methods of research
randomised trial. This focus on minimising and evaluation and might also require
bias as a means of ensuring quality arises information from qualitative research.
from some of the formal clinical and
health approaches to assessing evidence
It would clearly be a mistake to ignore

important research evidence just because it had
not been published in a peer-reviewed journal.

CONTENTS
HIGH-QUALITY QUALITATIVE research and the accurate synthesis of

RESEARCH existing information has been described
as “the most important single offering by
Many have highlighted the value of good
academics to the policy process”.75 Much
quality qualitative research73 for policy
of the thinking on quality set out above
makers. Qualitative evidence can be just
has focused on single studies in ‘primary
as scientifically credible as quantitative,
research’; in other words, individual studies
and a combination of the two is likely to
such as experiments, surveys or a series of
help decision makers combine evidence
interviews. But, it’s important to think about
of the effects of an intervention with the
‘research syntheses’, and the summarising
knowledge to decide on the applicability
and pooling together of a series of
of that evidence to their situation. The key
individual studies. This idea is not new, and
thing is that research evidence that you
the concept of making decisions on the
use to deal with any challenge that you
basis of accumulated evidence has been
face is fit for purpose. It needs to have
stressed for centuries.76
used the appropriate method, to have
collected and analysed its data (whether Our earlier discussion of cognitive biases
qualitative or quantitative) using well- showed how people can unconsciously fall
defined and replicable methods and to into the trap of looking for evidence that
report the findings in a transparent way fits our beliefs. So, we need to be careful
that minimises bias. when collecting together the existing
research and need to be willing to include
As with quantitative research, there are
research even if we do not like its findings.
standards and guidelines for assessing
One way to avoid ‘cherry picking’ is to
the quality of qualitative research and
use what are called systematic reviews.77
evaluation.74 Whereas the quality of
These aim to be exhaustive, and to find as
experimental studies is usually based on
much as possible of the research relevant
internal validity, adequacy of reporting and
to answering the question that will help
external validity, the quality of qualitative
with the challenge we are faced with.
research is usually built around the
They use explicit methods to identify
dimensions of contribution, defensibility,
what can reliably be said on the basis
rigour and credibility. Contribution refers
of the available studies and assess the
to whether the study advances wider
research design and methods of these
knowledge or understanding about a
studies to determine their quality.
policy, practice, theory or a particular
substantive field. Defensibility refers to Some of the key characteristics78 of a
whether the study provides an appropriate systematic review are:
research strategy to address the evaluative Clearly stated research question, objectives
questions posed. Rigour refers to the and eligibility criteria for studies.
systematic and transparent collection,
analysis and interpretation of qualitative Explicit and reproducible methodology
data. Credibility refers to how well-founded to minimise bias.
and plausible are the arguments about the Systematic search to identify as many
evidence generated. studies as possible that meet the eligibility
criteria.
AVOID ‘CHERRY PICKING’ BY
USING SYSTEMATIC REVIEWS Formal assessment of the validity of the
findings of the included studies.
Decision makers need to think about the
quality of the whole body of evidence, Systematic presentation, and synthesis,
not just single pieces of evidence. They of the characteristics and findings of the
need to use aggregated collections of included studies.

CONTENTS
CASE STUDY Systematic approaches also have the

value of being explicit about how
Using systematic reviews they searched for research studies
to develop guidance for and decided on their eligibility. So, in
theory at least,79 other people could
staffs replicate the systematic review.
The International Rescue Committee In 2018, Christl Donnelly and colleagues
(IRC) place a great reliance on suggested four principles to help
evidence in their development of researchers, policymakers and others
programme guidance documents and to commission, do, share, appraise
tools for field staff. They do this by and use evidence syntheses, including
conducting evidence reviews across systematic reviews.80 They proposed that
many high-quality sources around evidence synthesis should be inclusive,
specific interventions or approaches, rigorous, transparent and accessible and
and use what they learn about impact, elaborated on these four areas as follows:
contexts, populations and conditions
to inform whether and how to adapt Inclusive
those interventions to specific
Involves policy makers and is relevant
contexts. and useful to them.
This is not without its challenges Considers many types and sources
because most of the available studies of evidence.
do not provide critical information such
as fidelity of implementation, causal Uses a range of skills and people.
mechanisms and gender sensitivity. Rigorous.
The IRC’s agency-wide effort to ensure Uses the most comprehensive feasible
that evidence is readily available to body of evidence.
their staff is through the development Recognises and minimises bias.
of the Outcomes and Evidence
Is independently reviewed as part
Framework (oef.rescue.org). In this of a quality-assurance process.
electronic publicly available platform,
the IRC have defined the outcomes Transparent
and sub-outcomes that IRC focuses Clearly describes the research question,
on in its work, the general theories of methods, sources of evidence and
change (or pathways) through which quality-assurance process.
they can achieve those outcomes and
Communicates complexities and areas
indicators for measuring them. of contention.
For each sub-outcome and outcome, Acknowledges assumptions, limitations
the IRC have summarised the best and uncertainties, including any
available quantitative evidence on the evidence gaps.
effectiveness of interventions that aim
Declares personal, political and
to change/improve the relevant sub- organisational interests and manages
outcome or outcome, with a primary any conflicts.
focus on evidence from systematic
reviews. For topics where systematic Accessible
reviews do not yet exist, the IRC has Is written in plain language.
identified and summarised individual
Is available in a suitable time frame.
impact evaluations.
Is freely available online.

CONTENTS
META-ANALYSIS FOREST PLOTS

Meta-analysis is an important part of Figure C.1 presents the findings of 28
systematic reviews where all or some impact evaluations that compared
of the included studies are sufficiently people in humanitarian emergencies
similar (‘homogeneous’) in terms of who had received mental health and
population, intervention, comparators and psychosocial support programmes with
outcomes for their statistical findings to those who had not received this type of
be pooled and aggregated. This results intervention.82 In this forest plot, the data
in a cumulative estimate of effect which presents the effects of the programmes
generally has a lower risk of bias than the on the severity of post-traumatic
effect estimate
for individual
studies in Figure C.1: F orest plot of the effect of mental health and psychosocial
isolation support programmes on post-traumatic stress disorder82
and greater Measure: Continuous: d(Hedges g)
precision. Heterogeneity: Q=206; df=27; p=0; I =86.9%; tau-squared=0.29
2
Meta-analysis Random effects model: -0.463 (-0.689, -0.237)
is usually
Berger (2009) -1.269 (-1.607 | -0.932) W:5.1
represented by
Catani (2009) 0.251 (-0.47 | 0.971) W:1.1
a forest plot81
Chen (2014) -1.277 (-2.41 | -0.144) W:0.5
such as that Chen (2014) -0.138 (-1.152 | 0.875) W:0.6
in Figure C.1, Cluver (2015) 0.59 (-0.257 | 1.436) W:0.8
which is taken Dybdahl (2001) -0.137 (-0.558 | 0.284) W:3.3
from a recent Ertl (2011) -0.457 (-1.12 | 0.206) W:1.3
systematic Ertl (2011) -0.035 (-0.7 | 0.629) W:1.3
review on Gordon (2008) -1.116 (-1.595 | -0.637) W:2.5
the impact Jordans (2010) -0.18 (-0.858 | 0.498) W:1.3
of support Khamis (2004) 0.205 (0.006 | 0.404) W:14.6
programmes Khamis (2004) 0.066 (-0.178 | 0.311) W:9.7
for populations Lange-Neilsen (2012) -0.11 (-0.462 | 0.243) W:4.6
affected by Layne (2008) -0.109 (-0.599 | 0.38) W:2.4
humanitarian McMullen (2013) -2.727 (-3.539 | 1.915) W:0.9
emergencies. O'Callaghan (2013) -1.944 (-2.66 | -1.228) W:1.1
Meta-analysis O'Callaghan (2014) -0.405 (-0.72 | -0.09) W:5.8
is perhaps O'Callaghan (2015) -1.994 (-2.848 | 1.141) W:0.8
best known O'Callaghan (2015) -2.595 (-3.558 | 1.633) W:0.6
for combining Pityaratstian (2015) -0.473 (-1.137 | 0.19) W:1.3
the results of Qouta (2012) 0.01 (-0.367 | 0.387) W:4.1
randomised Qouta (2012) -0.401 (-0.813 | 0.011) W:3.4
trials, but it can Schauer (2008) -0.062 (-0.7 | 0.576) W:1.4
also be done Tol (2008) -0.656 (-0.98 | -0.332) W:5.5
with data from Tol (2012) 0.329 (0.006 | 0.652) W:5.6
other types Tol (2012) -0.074 (-0.329 | 0.18) W:9
of study, such Tol (2014) -0.166 (-0.479 | 0.148) W:5.9
as those that Tol (2014) -0.102 (-0.426 | 0.222) W:5.5
have used case- TOTAL: -0.463 (-0.689 | -0.237)
control, cross- -3.5 -3 -2.5 -2 -1.5 -1 0 0.5 1

Favours intervention Favours control
sectional or
cohort designs.

CONTENTS
stress disorders (PTSD) as a continuous systematic review. They use the same basic
variable. For each study in this forest structure and stages of full systematic
plot, the red dot represents the average review, but are not as intensive, exhaustive
treatment effect of the intervention, and or comprehensive. They will take more
the parallel lines either side of the red short cuts with the searching, critical
dots represent the confidence interval appraisal, data extraction and gathering,
for that study. The solid black vertical and statistical analysis of included
line running from 0 on the horizontal axis studies. The findings are also presented
indicates no difference between using in a shorter and less detailed form than
and not using the programme, and the a full systematic review, and might be
results of all the studies are pooled to no longer than 25 pages, with a three-
provide the overall estimate of the effects page executive summary and a one-page
of the programmes. This new, summary briefing document for decision makers.
statistic is the black diamond (circled The limitations of rapid evidence
in red) at the bottom of the forest plot. assessments are that they are not as
This represents the cumulative estimate comprehensive or exhaustive as systematic
of effect of pooling and aggregating the reviews and are more likely to be subject
average effects sizes and the variances to bias than a full systematic review.
of all 28 impact evaluations included Consequently, greater caution is needed
in the review. It allows us to conclude when basing a decision on evidence from
that, on average, the mental health and a rapid evidence assessment than from
psychosocial support programmes have a a full systematic review. Notwithstanding
small, positive effect on PTSD compared these limitations, they are frequently
with not using these interventions. commissioned and used by policy
makers and programme implementers,
RAPID EVIDENCE ASSESSMENTS especially where time is of the essence
The preparation of systematic reviews and and no systematic reviews are available.
meta-analyses can be time-consuming.
This means that if an up-to-date A particular type of rapid review, called
systematic review is not available, people a Rapid Research Needs Assessment,
needing to make an urgent decision might can also be used to quickly identify
need to conduct their own searches for evidence gaps. The UK’s Public Health
the relevant pieces of evidence and then Rapid Support Team for disease
appraise and synthesise this faster than outbreaks includes a plan to conduct
would happen in a formal systematic these assessments with Evidence Aid,
review. Fortunately, the already large to identify important uncertainties that
number of systematic reviews is continuing could be tackled by research in the early
to grow rapidly, and we describe how to stages of a humanitarian emergency
find them and several collations that are associated with a disease outbreak.
available in Section D. However, if you are
unable to find what you are looking for THE IMPORTANCE OF REPETITION
amongst the existing systematic reviews, AND CORROBORATION
or the reviews you find are out of date, you Thinking about the concept of evidence
might need to think about commissioning gaps, brings us to one of the other
a ‘pared-down systematic review’, such as things that needs to be considered when
a rapid evidence assessment.83 These rapid assessing the quality of a summary
reviews normally take 1-3 months and are of research studies: the number of
timed to meet the needs of policy makers studies that need to be included for
and practitioners who cannot wait for a full you to be comfortable that the body

CONTENTS
of evidence is strong enough. Would REVIEW OF REVIEWS: HOW TO

only a couple of studies be sufficient JUDGE HIGH–QUALITY BODIES OF
if they are really good? Or might you EVIDENCE
require dozens, or even hundreds?
We need to remember that the formal
The reality is that there is no magic ways of bringing research together, for
number of studies. Yet, we cannot ignore example in systematic reviews, do not
the fact that the size of the body of always lead to a high-quality answer.
evidence is important: there is strength You still need to judge the quality of
in numbers, and we must have repetition the design and implementation of the
and corroboration. Even studies that evidence review, and the quality of any
have won many accolades need to be synthesis can only be as good as the
repeated. Amgen, a Californian drug quality of the studies it is based on.
company, tried to replicate 53 landmark Even the best-done review cannot turn
cancer studies. The work, published in low-quality research into a high-quality
Nature,84 only confirmed the findings answer.
of six (11%) of these important studies.
This is deeply troubling because the There are also formal appraisal tools
studies have influenced drug companies for assessing the quality of systematic
and cancer treatments globally and the reviews, such as AMSTAR.87 You might
inability to replicate them might indicate also use one of the original checklists
that their findings are unreliable. On the for assessing the quality of systematic
other hand, we do not want to see studies reviews,78 which posed a series of
being done again and again long after the questions:
evidence base is robust.85 Unnecessary
studies represent research waste86 and, Is the question clearly focused?
in the context of randomised trials, Is the search for relevant studies
may be unethical if some participants thorough?
continue to be randomised to sub-optimal
interventions. Are the inclusion criteria appropriate?
Is the validity of the included studies
A How to note on judging the adequately addressed?
strength of evidence produced by
the UK’s Department for International Is missing information obtained from
Development lists four things to the original researchers?
consider when checking bodies of How sensitive are the results to changes
evidence:41 in the way the review is done?
Are subgroup analyses interpreted
The (technical) quality of the studies cautiously?
constituting the body of evidence
(or the degree to which risk of bias Do the conclusions flow from the
has been addressed). evidence that is reviewed?
The size of the body of evidence. Are recommendations linked to the
strength of the evidence?
The context in which the evidence
is set. Are judgements about preferences
(values) valid?
The consistency of the findings
produced by studies constituting Is ‘evidence of no effect’ confused with
the body of evidence. ‘no evidence of effect’?

CONTENTS
Judging all these criteria is always existing systematic reviews – and in

going to be rather subjective and some sectors there are few. However,
needs to take account of the context these reviews of reviews can provide an
of the policy question. It is difficult to appraisal and summary of evidence from
give blanket advice for what would multiple systematic reviews of the same
constitute the right body of evidence intervention; or provide an overview of
for any policy. We are also beginning the evidence from a collection of reviews
to see the combination of the findings of different interventions in the same
from multiple systematic reviews in topic area.88
‘reviews of reviews’. Of course, this
cannot work if there is not a body of
Key messages for part 2 of Section C

numbers, and we need repetition and
To find evidence that you can trust,
corroboration. Even studies that have
look for peer-reviewed research. But
won many accolades need to be
note that peer review is far from
repeated, but we need to avoid doing
perfect. There can be unconscious
studies again and again long after the
biases such as ‘herding’, or publication
evidence base is strong enough.
bias towards positive results, or even
the deliberate distortion or falsification Use systematic reviews, which aim to
of data. be exhaustive and screen studies for
quality – usually based on the research
When looking at questions of impact
design and methods.
and ‘what works’, use the frameworks
and formal standards of evidence such If you do not have time to do a
as those used by Nesta and others. systematic review or cannot find a
completed one that answers your
One study is never enough. Avoid
question, you might wish to conduct or
making decisions based on single
commission a rapid evidence
studies and look for multiple
assessment.89
replications. There is strength in
Even the best done review cannot turn

low-quality research into a high-quality answer.

BACK TO SECTION D | WHERE SHOULD YOU LOOK FOR EVIDENCE?
CONTENTS
Where should you look

for evidence?
This section looks at different types of evidence and
examines how to choose the most appropriate for the
issue you are interested in. It also discusses how to judge
the quality of evidence.
SEARCHING FOR RESEARCH journals, books and websites; many of

EVIDENCE which are not free to use or are difficult
to search. When thinking about where to
With the rapid growth in the number of
search, it is important to consider where
people and organisations working on
the evidence you are interested in is likely
systematic reviews, the literature now
to have been published and indexed,
contains a large and ever-increasing
and how comprehensive you wish to be.
number of systematic reviews, so you
For example, if they were published in a
might be able to find one that meets your
scientific journal, the articles might be
needs. A recent estimate is that there are
available through one of the thousands
more than 200,000 systematic reviews
of electronic bibliographic databases.
across all topic areas. Of course, only
These include, for instance, PubMed
a small proportion of these would be
for health care, LILACS for articles with
relevant to the humanitarian sector but (as
particular relevance to South America
noted below) bundles of such reviews are
or ERIC for educational literature. It
freely available in online collections such
may also be necessary to identify
as those curated by Evidence Aid. There
grey literature, such as government or
are also large international organisations,
NGO reports, and research presented
such as the Cochrane (www.cochrane.
at conferences, which will require
org) and Campbell Collaborations (www.
searches of electronic databases such
campbellcollaboration.org) dedicated
as Open Grey (www.opengrey.eu) and
to the preparation and maintenance
the websites of organisations that may
of systematic reviews who make these
have conducted relevant research.
available online.
In planning a search, it can be helpful
However, if you need to do your own
to divide the search elements into the
searches for studies of the effects of
types of intervention or policy you are
humanitarian action, the starting point
interested in, the relevant population
is to recognise that these are not always
or settings, the outcome measures that
easy to find and you might need to
would be most helpful to your decision
get help from a librarian or information
making and the types of study you
specialist who will be to advise and,
wish to find. You can then decide which
perhaps, assist with designing and
one or more of these domains is most
running the search. The evidence is
important and helpful for identifying the
scattered across tens of thousands of
material that is most relevant to you.
reports spread across thousands of

CONTENTS
You should consider the types of SOURCES OF SYSTEMATIC

source to search, which might include: REVIEWS AND OTHER RESEARCH
FOR THE HUMANITARIAN SECTOR
Bibliographic databases (e.g. PubMed,
LILACS, ERIC). Because of these challenges in seeking
evidence, you might find it more efficient
Journal and conference websites and easier to use trusted repositories of
(e.g. those with a particular focus on research and systematic reviews, rather
disasters or humanitarian action). than relying on haphazard searches
Registries of research (such as of the internet. Fortunately, several
through the WHO portal for trial, organisations are now working to bring
www.researchregistry.com and relevant evidence together into online
PROSPERO for systematic reviews). resources. These include the following:
Online collections of research evidence The Active Learning Network for

(see below, e.g. ALNAP, 3ie). Accountability and Performance
Website of relevant organisations (ALNAP) is hosted by the Overseas
(e.g. for government and NGO reports). Development Institute (ODI) in London,
UK, with a website containing more
References in articles. than 15,000 resources including ALNAP
publications relevant to evidence and
Correspondence with researchers
details of several thousand evaluations.
and evaluators.
The foundations of Evidence Aid were
laid down within Cochrane following the
When choosing the terms to include Indian Ocean tsunami of 2004, when
in your search, you should consider: the full text of several dozen Cochrane
Reviews relevant to disaster response
Synonyms from different times were made freely available online. It was
and places.
established as a charity in 2015 and has
Other words and phrases that now collated several hundred systematic
are related to what you’re reviews that are all free to view from
interested in. its website (www.evidenceaid.org),
including, but not limited to, bundles of
Words that are broader.
Special Collections of reviews relevant
Words that are more narrow or to windstorms, earthquakes, Ebola, the
focused. health of refugees and asylum seekers,
post-traumatic stress disorder, and
Index terms or keywords assigned to
articles by the original authors or the prevention of acute malnutrition in
bibliographic database. emergencies and humanitarian crises.
And, finally, you need to decide The International Initiative for Impact
on whether you will apply any Evaluation (3ie) was established in 2008
restrictions based on language and and now offers four searchable databases
the time period in which the research online (www.3ieimpact.org). Two of these,
was conducted or published. the 3ie Database of Systematic Reviews
and the Database of Impact Evaluations
catalogue evidence of the effectiveness
of interventions in the humanitarian
sector. These databases also include

CONTENTS
systematic reviews and impact evaluations on the broader landscape of international

development, many of which have relevance to interventions in emergency situations.
We are maintaining an up-to-date fuller list of these types of resources online, at:
www.evidenceaid.org/online-collections-of-research-for-the-humanitarian-sector
The list provides a wide range of online research resources, many of which are
free and easy to access. These should be useful to any policy maker, NGO or
frontline professional in the humanitarian sector, providing easy access to reliable,
high quality evidence on the effectiveness of interventions. If you would like
to suggest additional resources for this list, please contact Evidence Aid:
info@evidenceaid.org
CONCLUSION
In conclusion to this guide on the use of evidence in the humanitarian sector,
we encourage you to take advantage of the freely available, accessible
and actionable summaries of research, such as the systematic reviews
contained on the websites we have listed. This will help you to move quickly
to sources of evidence that should help inform your policy and practice.
Key messages for Section D
Research evidence of relevance to Systematic reviews provide

the humanitarian sector is scattered summaries and synthesis of research
across tens of thousands of reports evidence.
spread across thousands of journals,
books and websites. Several online repositories of
research are available, improving access
Searching for this evidence may to systematic reviews and other types
require help from an information of research evidence.
specialist and designing the search
needs to consider carefully what is
being looked for, the sources to search,
to terms to use in the search and any
restrictions relating to language or time
period.
You might find it more efficient and easier

to use trusted repositories of research and
systematic reviews.

BACK TO ENDNOTES
CONTENTS
Endnotes
8
www.oxforddictionaries.com/
definition/english/evidence.
9
Nutley S, et al. (2013) What Counts
Collins S, Sadler K. (2002) Outpatient care for
1 as Good Evidence? London, UK:
severely malnourished children in emergency Alliance for Useful Evidence.
relief programmes: a retrospective
cohort study. Lancet 360:1824-30.
10
Hallam A, Bonino F. (2013) Using Evaluation
for a Change: Insights from humanitarian
2
WHO, World Food Programme, UN practitioners. ALNAP Study. London:
System Standing Committee on Nutrition, ALNAP/ODI. See www.alnap.org/system/
and UNICEF. (2007) Community-based files/content/resource/files/main/alnap-
management of severe acute malnutrition: study-using-evaluation-for-a-change.pdf.
a joint statement by the WHO, World Food
Programme, UN System Standing Committee
11
Ariely D. (2009) Predictably Irrational:
on Nutrition and UNICEF. New York: UNICEF. The Hidden Forces that Shape Our
Decisions. London, UK: HarperCollins.
3
rice AI, Djulbegovic B. (2017) What
P
does evidence mean? Most languages
12
M Treasury. (2011) The Green Book:
H
translate “evidence” into “proof”. Journal of Appraisal and Evaluation in Central
Evaluation in Clinical Practice 23(5):971-3. Government. London, UK: HM Treasury.
4
Sackett D, et al. (1996) Evidence based
13
Christoplos I. (2006) Links between
medicine: what it is and what it isn’t. Relief, Rehabilitation and Development
British Medical Journal 312:71. See www. in the Tsunami Response: A Synthesis
bmj.com/content/312/7023/71. of Initial Findings. Stockholm,
Sweden: Swedish International
5
The definition of evidence-based Development Cooperation Agency.
medicine came at a time when most
medical decision making was based on
14
sambok CE, Klein G, (editors). (2014)
Z
experience, authority and eminence. Naturalistic decision making. New
Medical practice was not informed by the York, USA: Psychology Press.
best available scientific evidence. Some 15
ahneman D, et al. (2009) Conditions for
K
commentators and researchers have
intuitive expertise: a failure to disagree.
argued that social policy is in the same
American Psychology 64:515-26. See
place as medicine was 20 or 30 years ago,
www.ncbi.nlm.nih.gov/pubmed/19739881.
namely that authority, rather than research
evidence, dominates decision making. 16
These estimates come from Cash Learning
Partnership (CaLP). (2018) The State of
6
llen C, et al. (2016) Evidence Aid. Oxford
A
the World’s Cash Report. Oxford: CaLP;
Public Health August: 51-54; Blanchet K,
Doing Cash Differently: Report of the High
et al. (2017) Evidence on public health
Level Panel on Cash Transfers. (2015); and
interventions in humanitarian crises. Lancet
Development Initiatives. (2017) Global
390:2287-96; and Christoplos I, et al. (2017)
Humanitarian Assistance Report.
Strengthening the quality of evidence in
humanitarian evaluations. ALNAP Method 17
DI, Development Initiatives. (2016)
O
Note. London: ALNAP/ODI (see www.alnap.
Counting Cash: Tracking Humanitarian
org/system/files/content/resource/files/
Expenditure on Cash-Based Programming.
main/alnap-eha-method-note-5-2017.pdf).
18
xfam. (2006) Good Practice Review
O
7
Knox Clarke P, Darcy J. (2014) Insufficient
11: Cash Transfer Programming in
evidence? The quality and use of evidence
Emergencies. Oxford: Oxfam.
in humanitarian action. ALNAP Study.
London: ALNAP/ODI. See www.alnap.
org/system/files/content/resource/
files/main/alnap-study-evidence.pdf.

BACK TO ENDNOTES
CONTENTS
19
Jackson R (Save the Children UK), Kukrety 29
Waddington H, et al. (2009) Water,
N (Oxfam GB) (2012) Institutionalising Sanitation and hygiene interventions to
cash transfer programming. See https:// combat childhood diarrhoea in developing
odihpn.org/magazine/institutionalising- countries: a systematic review. Delhi, India:
cash-transfer-programming/. International Initiative for Impact Evaluation.
20
See www.humanitarianresponse. 30
Nesta and TSIP. (2014) Guidance for
info/en/programme-cycle/space. Developing a Theory of Change for Your
Programme. See www.nesta.org.uk/
Discussion of the challenges of making
21
sites/default/files/theory_of_change_
decisions following the rare circumstances guidance_for_applicants_.pdf.
of a major radition emergency is available
in Carr Z, et al. (2016) Using the GRADE 31
tern E. (2015) Impact Evaluation: A
S
approach to support the development Design Guide for Commissioners and
of recommendations for public health Managers of International Development
interventions in radiation emergencies. Evaluations in the Voluntary and Community
Radiation Protection Dosimetry 171:144-55; Sector. London, UK: Big Lottery Fund,
and Ohtsuru A, et al. (2015) Nuclear disasters Bond, Comic Relief and the Department
and health: lessons learned, challenges, for International Development.
and proposals. Lancet 386: 489-97.
32
erdin M, et al. (2014) Optimal evidence
G
22
Bradley DT, et al. (2014) The effectiveness in difficult settings: improving health
of disaster risk communication: a systematic interventions and decision making in
review of intervention studies. PLOS disasters. PLoS Medicine 11(4): e1001632.
Currents Disasters August 22; Edition 1.
33
Mulgan G. (2015) The six Ws: a formula for
23
A discussion of the importance of paying what works. London: Nesta. See www.nesta.
attention to how donors gather, use and org.uk/blog/six-ws-formula-what-works.
share evidence and information is available
in Obrecht A. (2017) Using Evidence to 34
www.ebola-anthropology.
Allocate Humanitarian Resources: Challenges net/about-the-network.
and Opportunities. ALNAP Working Paper.
London: ALNAP/ODI. See https://reliefweb. 35
Faye SL. (2015) L“exceptionnalité” d’Ebola
int/sites/reliefweb.int/files/resources/ et les “réticences” populaires en Guinée-
alnap-eaar-resource-allocation-2017.pdf. Conakry. Réflexions à partir d’une approche
d’anthropologie symétrique. Anthropologie
24
esta. (2013) Understand how
N & Santé. See https://journals.openedition.
innovation works. Video available org/anthropologiesante/1796.
at www.nesta.org.uk/resources/
understand-how-innovation-works. 36
http://pubman.mpdl.mpg.de/pubman/
item/escidoc:2096578/component/
25
Obrecht A, Warner AT. (2016) More than escidoc:2103624/AAA-Ebola-Report-1.pdf.
just luck: Innovation in humanitarian action.
HIF/ALNAP Study. London: ALNAP/ODI. 37
Fairhead J. (2016) Understanding Social
Resistance to the Ebola Response in the
26
Jones G, et al. (2003) How many child Forest Region of the Republic of Guinea: An
deaths can we prevent this year? Lancet Anthropological Perspective. African Studies
362:65-71. See https://linkinghub.elsevier. Review 59:7-31. doi:10.1017/asr.2016.87.
com/retrieve/pii/S0140-6736(03)13811-1.
38
Abramowitz S, et al. (2015) Social
27
Nyhan B, Reifler J. (2010) When science intelligence in the global
corrections fail: The persistence of political Ebola response. Lancet 385:330.
misperceptions. Political Behavior 32:303-30.
39
Petticrew M, et al. (2003) Evidence,
28
hite H. (2009) Theory-based impact
W hierarchies and typologies: horses
evaluation: principles and practice, for courses. Journal of Epidemiology
working paper 3. Delhi, India: International and Community Health 57:527-9.
Initiative for Impact Evaluation, page 4.

BACK TO ENDNOTES
CONTENTS
40
n authoritative and exhaustive list of social
A 47
oocy S, Burnham G. (2006). Point-
D
science research frameworks and methods of-use water treatment and diarrhoea
is available in Luff R, et al. (2015) Review of reduction in the emergency context:
the Typology of Research Methods within an effectiveness trial in Liberia. Tropical
the Social Sciences. London, UK: ESRC/ Medicine and International Health 11:1542-52.
National Centre for Research Methods.
See http://eprints.ncrm.ac.uk/3721. 48
hite H, et al. (2014) Randomised
W
Controlled Trials (RCTs). Methodological
41
epartment for International Development.
D Briefs. Impact Evaluation No. 7. Florence,
(2014) How to Note: Assessing the Italy: Unicef. See www.unicef-irc.org/
Strength of Evidence. See www.gov.uk/ publications/pdf/brief_7_randomised_
government/uploads/system/uploads/ controlled_trials_eng.pdf.
attachment_data/file/291982/HTN-
strength-evidence-march2014.pdf. 49
Gertler P. (2000) Final Report: The Impact
of PROGRESA on Health. Washington, DC,
42
Adapted from HM Treasury, DECC and DEFRA. USA: International Food Policy Research
(2012) Quality in policy impact evaluation: Institute (IFPRI). See www.ifpri.org/
understanding the effects of policy from other publication/impact-progresa-health.
influences. London, UK: HM Treasury/DEFRA/
DECC; Frost S, et al. (2006) The Evidence 50
ainmueller J, et al. (2017) Catalyst or
H
Guide; Using Research and Evaluation in Social crown: does naturalization promote
Care and Allied Professions. London, UK: the long-term social integration
Barnardo’s; Petticrew M, Roberts H. (2003) of immigrants? American Political
Evidence, hierarchies and typologies: horses Science Review 111(2):256-76.
for courses. Journal of Epidemiology and doi:10.1017/S0003055416000745.
Community Health. 57: 527-9; and Stern E.
(2015) Impact Evaluation; A Design Guide for 51
Bozorgmehr K, et al. (2015) Effect of
Commissioners and Managers of International restricting access to health care on
Development Evaluations in the Voluntary and
health expenditures among asylum-
Community Sector. London, UK: Big Lottery
Fund, Bond, Comic Relief and the Department seekers and refugees: A quasi-
for International Development, Table 2, page 18. experimental study in Germany,
1994–2013. PLoS ONE 10: e0131483.
43
Odgaard-Jensen J, et al. (2011) Randomisation
to protect against selection bias in
52
ossi R, et al. (2016) Vaccination
R
health care trials. Cochrane Database of coverage cluster surveys in Middle
Systematic Reviews (4):MR000012. Dreib - Akkar, Lebanon: comparison of
vaccination coverage in children aged
44
White H. (2013) An introduction to the use 12-59 months pre- and post-vaccination
of randomised control trials to evaluate campaign. PLoS ONE 11(12): e0168145.
development interventions. Journal of
Development Effectiveness 5(1):30-49. 53
Alexander J, Bonino F (2015). A discussion
on designs, approaches and examples,
45
See for example: Schulz KF, et al. (2010) ALNAP Discussion Series Improving
CONSORT 2010 statement: updated guidelines the quality of EHA evidence, Method
for reporting parallel group randomised Note 4, January 2015. See www.alnap.
trials. PLoS Medicine 7(3):e1000251; org/system/files/content/resource/
Higgins JP, et al. (2011) The Cochrane files/main/alnap-eha-method-note-
Collaboration's tool for assessing risk of addressing-causation-jan2015.pdf.
bias in randomised trials. BMJ 343:d5928.
54
Waddington H, et al. (2017) Quasi-
46
Puri J, et al. (2015) What methods may be experimental study designs series paper
used in impact evaluations of humanitarian 6: risk of bias assessment. Journal of
assistance? working paper 22, Delhi, India: Clinical Epidemiology 89:43-52; and
International Initiative for Impact Evaluation. Bärnighausen T, et al. (2017) Quasi-
See www.3ieimpact.org/media/filer_
experimental study designs series—paper
public/2014/12/08/wp_22_humanitarian_
7: assessing the assumptions. Journal
methods_working_paper-top.pdf.
of Clinical Epidemiology 89:53-66.

BACK TO ENDNOTES
CONTENTS
55
Nielsen NS, et al. (2013). The Contribution
65
Franco A, et al. (2014) Social science.
of Food Assistance to Durable Solutions Publication bias in the social sciences:
in Protracted Refugee Situations; its unlocking the file drawer. Science 345:1502-5.
impact and role in Bangladesh: A Mixed 66
ense about Science. (2006) I
S
Method Impact Evaluation, Volume
don’t know what to believe: Making
I-Evaluation Report, Geneva, Switzerland:
sense of science stories. See www.
World Food Program/UNHCR.
senseaboutscience.org/resources.
56
Carter R. (2012) Helpdesk Research Report: php/16/i-dont-know-what-to-believe.
Theory-based evaluation approach. University 67
See for instance, the supplement to the
of Birmingham, UK: Birmingham: Governance
Magenta Guide: HM Treasury, DECC and
and Social Development Resource Centre.
DEFRA (2012) Quality in policy impact
See www.gsdrc.org/docs/open/hdq872.pdf.
evaluation; understanding the effects of
57
Westhorp G. (2014) Realist Evaluation: An policy from other influences. The guidance
Introduction. London, UK: Methods Lab, shows how ‘higher quality research designs
Overseas Development Institute. See www. can help meet the challenge of attributing
odi.org/sites/odi.org.uk/files/odi-assets/ measured outcomes to the policy in
publications-opinion-files/9138.pdf. question (as opposed to other influences),
whereas lower quality designs reduce
58
Mayne J. (2012) Contribution Analysis: confidence in whether it was the policy
Coming of Age? Evaluation 18(3):270-80. that achieved those outcomes’ (page 5).
59
For network analysis and process tracing
68
http://gradeworkinggroup.org.
see: Befani B, et al. (2014) Process tracing
and contribution analysis: a combined
69
arrington RP, et al. The Maryland Scientiﬁc
F
approach to generative causal inference for Methods Scale, In: Farrington DP, et
impact evaluation. IDS Bulletin 45(6):17-36. al. Evidence-based crime prevention.
London, UK: Routledge, 2002 chapter 2.
60
Baptist C, et al. (2015) Coffey How To
Guide: Qualitative Comparative Analysis
70
or a model of different approaches on
F
– A Rigorous Qualitative Method for quality that includes the four dimensions
Assessing Impact. See www.coffey. of (1) methodological quality (2) quality
com/assets/Ingenuity/Qualitative- in reporting, (3) appropriateness and
Comparative-Analysis-June-2015.pdf. (4) relevance to policy and practice; see
Boaz A, Ashby D. (2003) Fit for purpose?
Stern E, et al. (2012) Broadening the
61 Assessing research quality for evidence
range of designs and methods for impact based policy and practice. Working
evaluations, London, UK: Department Paper 11. London, UK: ESRC UK Centre
for International Development. for Evidence Based Policy and Practice.
62
For example, see Carol Weiss’s guide: Callaway E. (2011) Report finds
71
Weiss C. (1998) Evaluation: methods massive fraud at Dutch universities.

for studying programs and policies. Nature 479(15). See www.nature.com/
New York, USA: Prentice Hall. news/2011/111101/full/479015a.html.
63
Ioannidis JA. (2005) Contraindications and
72
Boaz A, et al. (2003) Fit for purpose?
Initially Stronger Effects in Highly Cited Assessing research quality for evidence
Clinical Research. Journal of the American based policy and practice. Working
Medical Association. 294(2):218-28. Paper 11. London, UK: ESRC UK Centre
for Evidence Based Policy and Practice.
64
or a systematic review of publication bias see
F
Hopewell S, et al. (2009) Publication bias in
73
Qualitative research explores and tries to
clinical trials due to statistical significance or understand people’s beliefs, experiences,
direction of trial results. Cochrane Database attitudes, behaviour and interactions. It
of Systematic Reviews (1):MR000006. doi: generates non-numerical data which might
10.1002/14651858.MR000006.pub3. be gathered through, for example in-depth
interviews, focus groups, documentary
analysis and participant observation.

BACK TO ENDNOTES
CONTENTS
74
Spencer L, et al. (2002) Quality in Qualitative 82
Bangpan M, et al. (2017) The impact
Evaluation: A framework for assessing of mental health and psychosocial
research evidence. London, UK: Cabinet support programmes for populations
Office; CASP. (2018) CASP Checklist for affected by humanitarian
Qualitative Research. Oxford, UK: Critical emergencies. Oxford, UK: Oxfam.
Appraisal Skills Programme; National
Institute for Health and Clinical Excellence. 83
Ganann R, et al. (2010) Expediting
(2012) The guidelines manual: Appendix H: systematic reviews: methods and
Methodology checklist: qualitative studies. implications of rapid reviews.
London, UK: National Institute for Health Implementation science 5:56; and Tricco
and Clinical Excellence; and O'Brien BC, et AC, et al. (2015) A scoping review of rapid
al. (2014) Standards for reporting qualitative review methods. BMC Medicine 13: 224.
research: a synthesis of recommendations.
Academic Medicine 89: 1245-51. 84
Begley CG, et al. (2012) Drug development:
Raise standards for preclinical cancer
75
Whitty CJM. (2015) What makes an research. Nature 483: 531-3.
academic paper useful for health
policy? BMC Medicine 13:301. 85
review of research that had included
A
one or more cumulative meta-analysis
76
ee, for example, the discussion of the
S found many examples showing that stable
importance of using existing evidence when results (beneficial, harmful and neutral)
designing new studies in Clarke M. (2004) would have been seen had a meta-analysis
Doing new research? Don't forget the old: of existing research been done before a
nobody should do a trial without reviewing new randomised trial began, which would
what is known. PLoS Medicine 1: 100-2; and the have led to earlier uptake of effective
history of systematic reviews in Chalmers I, et interventions: Clarke M, et al. (2014)
al. (2002) A brief history of research synthesis. Accumulating research: a systematic
Evaluation and the Health Professions 25: 12- account of how cumulative meta-
37; and Clarke M. (2016) History of evidence analyses would have provided knowledge,
synthesis to assess treatment effects: personal improved health, reduced harm and saved
reflections on something that is very much resources. PLoS ONE 9(7): e102670.
alive. JLL Bulletin: Commentaries on the
history of treatment evaluation. Journal of 86
halmers I, et al. (2014) How to increase
C
the Royal Society of Medicine 109: 154-63. value and reduce waste when research
priorities are set. Lancet 383: 156-65.
77
Allen C. (2014) A resource for those preparing
for and responding to natural disasters, 87
hea BJ, et al. (2007) Development of
S
humanitarian crises, and major health care AMSTAR: a measurement tool to assess
emergencies. Journal of Evidence-Based the methodological quality of systematic
Medicine 7: 234-7; and Gurevitch J, et al. reviews. BMC Medical Research Methodology
(2018) Meta-analysis and the science of 7:10; and Shea BJ, et al. (2017) AMSTAR
research synthesis. Nature 555: 175-82. 2: a critical appraisal tool for systematic
reviews that include randomised or
78
xman AD. (1994) Checklists for
O non-randomised studies of health care
review articles. BMJ 309: 648-51. interventions, or both. BMJ 358:j4008.
79
For a critique of applying research syntheses 88
Smith V, et al. (2011) Methodology in
into policy, see Pawson R. (2001) Evidence conducting a systematic review of systematic
Based Policy: In Search of a Method. Working reviews of health care interventions. BMC
Paper 3. London, UK: ESRC UK Centre for Medical Research Methodology 11:15.
Evidence Based Policy and Practice.
89
ee, for example: Brennan RJ, et al. (2005)
S
80
Donnelly CA, et al. (2018) Four principles for Rapid health assessment in Aceh Jaya
synthesizing evidence. Nature 558: 361-4. district, Indonesia, following the December
26 tsunami. Emergency Medicine Australasia
81
Lewis S, Clarke M. (2001) Forest plots: trying to 17: 341e350; and Beebe J. (2014) Rapid
see the wood and the trees. BMJ 322:1479-80. Qualitative Inquiry: a Field Guide to Team-
based Assessment, second edition. Lanham,
Maryland, USA: Rowman & Littlefield.

Karl Blanchet, Director of the Health in Humanitarian Crises
Centre, London School of Hygiene & Tropical Medicine
Karl.Blanchet@lshtm.ac.uk | crises.lshtm.ac.uk
Mike Clarke, Research Director of Evidence Aid,
Centre for Public Health, Queen’s University Belfast
mclarke@qub.ac.uk | www.qub.ac.uk
Cover Photo © Patrick Brown/UNICEF/Panos Pictures

Research Evidence in The Humanitarian Sector

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Research Evidence in The Humanitarian Sector

Uploaded by

Copyright:

Available Formats

Research Evidence in

the Humanitarian Sector

Cover Photo © Patrick Brown/UNICEF/Panos Pictures

1 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

Evidence Aid (www.evidenceaid.org) A four-year research and capacity-

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 2

INTRODUCTION 4 Beyond experiments:

3 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

Evidence challenges what we might investing money to solve a problem is

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 4

5 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 6

Figure A.1: The four elements

7 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

We give prominence to research and evaluation

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 8

a novel intervention. However, whether picking and retro-fitting the evidence to

9 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 10

How research has promoted the growth of

Key messages from Section A

We do not advocate a form of Results from research are only one

11 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

When can evidence help you?

Evidence can make organisations HUMANITARIAN

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 12

Figure B.1: Humanitarian Response Cycle

IMPLEMENTATION & RESOURCE

Resource mobilisation: The Operational peer review and evaluation:

13 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

DEVELOPING AND IMPLEMENTING The spiral below was developed by

Figure B.2: The Nesta Innovation Spiral

1 Exploring opportunities and 4 Making the case

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 14

1. EXPLORING OPPORTUNITIES AND CHALLENGES

Improving access to health care in post-conflict settings

With the principle of ‘leaving no-one post-conflict periods and produced

15 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

2. GENERATING IDEAS In Section C, we discuss Standards of

Presenting evidence to country offices leaders

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 16

5. DELIVERY AND IMPLEMENTATION

Partnering for impact evaluations

17 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

6) GROWING, SCALING AND do the shopping. And for women in joint

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 18

CREATING A THEORY OF CHANGE handwashing facilities be located? What

Key messages from Section B

19 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

Example of Theory of Change on resilience of communities

Material Human Financial Information

Increased protection from water & vector borne diseases

or capacity for self- • Improved access to basic services

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 20

What evidence should you

Not all evidence is equal. For instance, experimental research is

21 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR

Lessons learnt from using ethnography in the

Experimental research is more suited to

RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 22

PART 1: DIFFERENT TYPES OF RESEARCH

So how do you go about Some types of research (such as

1 Exploring opportunities and 4 Making the case