Professional Documents
Culture Documents
Methods For Assessing Policy Impact: Process and Partnerships For Pro-Poor Policy Change, Project Initiation Workshop 1
Methods For Assessing Policy Impact: Process and Partnerships For Pro-Poor Policy Change, Project Initiation Workshop 1
Workshop Outline
• Introduction to the RAPID Framework and
ILRI/ODI Project
• Case Study Approach
• Episode Study Approach
• Outcome Mapping Approach
• RAPID Outcome Assessment (ROA)
Approach
Lunch
• Practical Sessions
ILRI Process and Partnership for
Pro-Poor Policy Change
Objectives
By the end of the workshop, participants will:
• understand the Process and Partnership for Pro-Poor
Policy Change project’s purpose and general approach
• have the opportunity to contribute their own suggestions to
improve the project;
• understand, and have had the chance to try out the three
key methods which will be used in the project;
• assess the usefulness of the approaches in their own work.
An introduction to the
RAPID Framework and
ILRI/ODI Project
ILRI Process and Partnership for
Pro-Poor Policy Change
Definitions
• Research: “any systematic effort to increase the
stock of knowledge”
• Policy: a “purposive course of action followed by an
actor or set of actors”
– Agendas / policy horizons
– Official statements documents
– Patterns of spending
– Implementation processes
– Activities on the ground
ILRI Process and Partnership for
Pro-Poor Policy Change
Policy Processes
- Identify a policy problem
- Commission research
- Assess the results
- Select the best policy
- Establish the policy framework
- Implement the policy
- The problem is solved
ILRI Process and Partnership for
Pro-Poor Policy Change
Reality…
• “The whole life of policy is a chaos of purposes
and accidents. It is not at all a matter of the rational
implementation of the so-called decisions through
selected strategies 1”
• “Most policy research on African agriculture is
irrelevant to agricultural and overall economic
policy in Africa2”
1 - Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in
Agricultural and Rural Development, Heineman Educational Books, London
2 – Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges,
Existing theory
1. Linear model 18. Linear model of communication, Shannon
2. Percolation model, Weiss 19. Interactive model,
3. Tipping point model, Gladwell 20. Simple and surprising stories,
4. ‘Context, evidence, links’ framework, ODI Communication Theory
5. Policy narratives, Roe 21. Provide solutions, Marketing Theory I
6. Systems model (NSI) 22. Find the right packaging, Marketing II
7. External forces, Lindquist 23. Elicit a response, Kottler
8. ‘Room for manoeuvre’, Clay & Schaffer 24. Translation of technology, Volkow
9. ‘Street level bureaucrats’, Lipsky 25. Epistemic communities
10. Policy as social experiments, Rondinelli 26. Policy communities
11. Policy Streams & Windows, Kingdon 27. Advocacy coalitions etc, Pross
12. Disjointed incrementalism, Lindquist 28. Negotiation through networks, Sebattier
13. The ‘tipping point’, Gladwell 29. Shadow networks, Klickert
14. Crisis model, Kuhn 30. Chains of accountability, Fine
15. ‘Framework of possible thought’, 31. Communication for social change,
Chomsky Rockefeller
16. Variables for Credibility, Beach 32. Wheels and webs, Chapman & Fisher
17. The source is as important as content,
Gladwell
www.odi.org.uk/rapid/lessons/theory
ILRI Process and Partnership for
Pro-Poor Policy Change
An Analytical Framework
External Influences The political context –
Socio-economic and political and economic structures
cultural influences, and processes, culture, institutional
donor policies etc pressures, incremental vs radical
change etc.
Case Studies
• Sustainable Livelihoods: The
Evolution of DFID Policy
• The PRSP Initiative: Research in
Multilateral Policy Change
• The adoption of Ethical
Principles in Humanitarian Aid
post Rwanda
• Animal Health Care in Kenya:
Evidence fails to influence Policy
ILRI Process and Partnership for
Pro-Poor Policy Change
A Practical Framework
External Influences political context
Politics and
Campaigning, Policymaking
Policy analysis, &
Lobbying
research
Media,
Advocacy,
Scientific Networking Research,
information learning &
exchange & thinking
validation
links evidence
ILRI Process and Partnership for
Pro-Poor Policy Change
Policy entrepreneurs
Storytellers Networkers
Engineers Fixers
ILRI Process and Partnership for
Pro-Poor Policy Change
Practical Tools
Overarching Tools
- The RAPID Framework Context Assessment Tools
- Using the Framework - Stakeholder Analysis
- The Entrepreneurship - Forcefield Analysis
Questionnaire - Writeshops
- Policy Mapping
Communication Tools - Political Context Mapping
- Communications Strategy
- SWOT analysis
- Message Design
- Making use of the media Research Tools
- Case Studies
Policy Influence Tools - Episode Studies
- Influence Mapping & Power Mapping - Surveys
- Lobbying and Advocacy - Bibliometric Analysis
- Campaigning: A Simple Guide - Focus Group Discussion
- Competency self-assessment
ILRI Process and Partnership for
Pro-Poor Policy Change
Practical Application
• Within ODI
• Workshops for researchers, policy makers
and activists.
• Advice to a DFID forest/ground water
research project in India:
– Less research
– More communication
– Developing champions in regional and national
government
– Local, Regional & National advocacy campaign
ILRI Process and Partnership for
Pro-Poor Policy Change
The project …
Pro-Poor Policy Change
• Discussion:
– Can ILRI hope to influence pro-poor policy
through research?
– Any good case studies?
Case Study Approach
ILRI Process and Partnership for
Pro-Poor Policy Change
Why is it useful?
Goal :
to describe as accurately as possible the
fullest, most complete description of the case.
Issues
• The unit of analysis is a critical factor
• Typically a system of action rather than an
individual or group of individuals
• Tend to be selective, focusing on one or two
issues that are fundamental to understanding the
system being examined
• Case studies are multi-perspectival analyses
• The researcher considers not just the voice and
perspective of the actors, but also of the relevant
groups of actors and the interaction between them
• They give a voice to the powerless and voiceless.
ILRI Process and Partnership for
Pro-Poor Policy Change
Triangulation
• Data source triangulation, when the researcher looks
for the data to remain the same in different contexts;
• Investigator triangulation, when several investigators
examine the same phenomenon;
• Theory triangulation, when investigators with
different view points interpret the same results; and
• Methodological triangulation, when one approach is
followed by another, to increase confidence in the
interpretation.
ILRI Process and Partnership for
Pro-Poor Policy Change
Applications
• To explain complex causal links between
research and policy
• To describe the real-life context in which
policy has been influenced by research
• To describe the policy influencing process
itself
• To explore those situations in which the
policy intervention being evaluated has no
clear set of outcomes.
ILRI Process and Partnership for
Pro-Poor Policy Change
Process
1. Design the case study protocol:
– determine the required skills
– develop and review the protocol
2. Conduct the case study:
– prepare for data collection
– distribute questionnaire
– conduct interviews
3. Analyze case study evidence:
– analytic strategy
4. Develop conclusions, recommendations, and
implications based on the evidence
Episode Study
Approach
ILRI Process and Partnership for
Pro-Poor Policy Change
Advantage
• The process of working backwards in time
gives a more realistic view of the broad
range of factors – other than research – that
influence policy
• Tracking forward probably overemphasizes
the importance of research
ILRI Process and Partnership for
Pro-Poor Policy Change
Issues
• Policy processes are complex, multi-layered and
change over time
• Often difficult to isolate the impact of research
from other factors
• Actors may ‘re-write history’
• Important to seek the views of a wide range of
informed stakeholders
• The process of preparing an episode study is
iterative
• Key facts and / or inconsistencies need to be
cross-checked with key informants
ILRI Process and Partnership for
Pro-Poor Policy Change
Process
1. Identify a clear policy change.
2. Identify key Research Questions (draw on
RAPID framework)
3. Explore how and why those policy
decisions and practices took place
4. Assess the relative role of research in that
process by drawing on the framework.
ILRI Process and Partnership for
Pro-Poor Policy Change
Media,
Advocacy,
Scientific Networking Research,
information learning &
exchange & thinking
validation
links evidence
ILRI Process and Partnership for
Pro-Poor Policy Change
Key Questions
• The external environment: Who are the key actors?
What is their agenda? How do they influence the political
context?
• The political context: Is there political interest in
change? Is there room for manoeuvre? How do they
perceive the problem?
• The evidence: Is it there? Is it relevant? Is it practically
useful? Are the concepts familiar or new? Does it need re-
packaging?
• Links: Who are the key individuals? Are there existing
networks to use? How best to transfer the information? The
media? Campaigns?
ILRI Process and Partnership for
Pro-Poor Policy Change
Methods
Steps 3 and 4 can be done through a variety
of methods:
• review of the literature;
• interviews with key actors;
• capturing the authors’ own experience; and
• discussions at workshops.
Episode Study Examples
ILRI Process and Partnership for
Pro-Poor Policy Change
Paravets in Kenya
1970s - Professionalisation of Public Services.Research
International
- Structural Adjustment → Collapse
- Paravet projects emerge.
1980s - ITDG projects
projects.– collaborative research.
- Privatisation
Privatisation.
- ITDG Paravet network
network.and change of DVS.
1990s
- Rapid spread in North. The Hubl Study
Dr Kajume
- KVB letter (January 1998).
- Multistakeholder WSs → new policies.
2000s
- Still not approved / passed!
ILRI Process and Partnership for
Pro-Poor Policy Change
PRSPs – Evidence
• Long-term academic research informing new
focus on poverty, participation, ownership, aid
effectiveness etc
• Applied policy research:
– ESAF reviews
– HIPC review
– SPA Working Groups
– NGO research on debt
• Uganda’s PEAP
ILRI Process and Partnership for
Pro-Poor Policy Change
PRSPs – Links
• WB, IMF, SPA, Bilaterals, NGOs all involved
• Formal an informal networks
• “None of the players was more than two
handshakes away from any of the others”
Outcome Mapping
ILRI Process and Partnership for
Pro-Poor Policy Change
What is it?
• an integrated PM&E tool
• a system to think holistically & strategically about
how we intend to achieve result
• an approach that focuses on changes in the
behaviour, relationships or actions of partners (as
outcomes)
• a methodology that characterizes and assesses the
program’s contributions to the achievement of
outcomes
• an approach for designing in relation to the broader
development context but assessing within your
sphere of influence
ILRI Process and Partnership for
Pro-Poor Policy Change
Behavioural Changes 25
ILRI Process and Partnership for
Pro-Poor Policy Change
Terminology
• Outcomes: changes in behaviours, relationships,
activities and/or actions of the people, groups and
organisations with whom we work
• Vision: the broad human, social and environmental
betterment we desire
• Mission: how we intend to contribute towards the
achievement of the vision
• Boundary partners: individuals, groups and
organisations with whom we interact directly to
effect change
• Outcome challenges: changes behaviours of the
boundary partners as identified by the vision
ILRI Process and Partnership for
Pro-Poor Policy Change
Performance Monitoring
• Provides a framework for a continuous
monitoring of the initiative as a tool to
achieve its outcomes.
• The program uses progress markers, a set
of graduated indicators of behavioural
change, identified in the intentional design
stage to clarify direction with its primary
partners and to monitor outcomes
ILRI Process and Partnership for
Pro-Poor Policy Change
Evaluation Planning
• Helps identify the evaluation priorities
assessing the strategy at greater depth than
the performance monitoring stage
ILRI Process and Partnership for
Pro-Poor Policy Change
Main Elements
RAPID Outcome
Assessment
ILRI Process and Partnership for
Pro-Poor Policy Change
What is it?
• A Visual Tool
• Combines the outcome mapping concept within a
case study & episode study approach
• Systematic approach to collecting information about
changes in behaviour of key project partners that
contributed to the policy change
• Assessment of the contribution of the project
(programme, strategy, etc.) to observed changes in
behaviour –and ultimately to the policy change
ILRI Process and Partnership for
Pro-Poor Policy Change
Approach
1. Describe policy environment at end
2. Describe policy environment at the beginning
3. Identify the key policy actors
4. Identify key boundary partners
5. Describe boundary partner behaviour at end
6. Describe boundary partner behaviour at beginning
7. Describe changes in BP behaviour
8. Describe changes in project (strategic/opportunistic)
9. Describe external influences
10. Determine level of impact of changes in project
11. Determine level of impact of external influences
12. Check through external interviews
13. Write report
ILRI Process and Partnership for
Pro-Poor Policy Change
Policy Change
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
Policy Change
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1
BP2
Policy Environment
BP3
Policy Change
BP4
BP5
BP6
BP7
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 5
BP2 4
Policy Environment
BP3 8
Policy Change
BP4 7
BP5 4
BP6 9
BP7 3
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 5
BP2 0 4
Policy Environment
BP3 0 8
Policy Change
BP4 0 7
BP5 0 4
BP6 0 9
BP7 0 3
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
Project
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
EE
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
EE 0 1 2 3,4 5 6 7 8 9 10 11
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence
ILRI Process and Partnership for
Pro-Poor Policy Change
BP1 0 1 2 3,4 5
BP2 0 1 2,3 4
Policy Environment
BP3 0 1 2 3 4,5,6 7 8
Policy Change
BP4 0 1 2 3 4 5 6 7
BP5 0 1 2 3 4
BP7 0 1, 2 3
EE 0 1 2 3,4 5 6 7 8 9 10 11
Direct influence
Indirect influence
External influence