Professional Documents
Culture Documents
Posthumus Et Al 2020 A Pragmatic Approach To Assessing System Change
Posthumus Et Al 2020 A Pragmatic Approach To Assessing System Change
Approach to
Assessing
System Change
Hans Posthumus
Rachel Shah
Alexandra Miehlbradt
Adam Kessler
A Pragmatic Approach to Assessing System Change
Contents
1 Introduction 1
2 Develop system strategy and intervention plans 3
3 Use complementary lenses to assess and analyse changes 8
4 Review and revise 11
5 Report transparently 12
6 A pragmatic way forward 12
A Pragmatic Approach to Assessing System Change
The views expressed in this report do not necessarily represent the views
of the agencies involved.
1 Introduction
Few topics create as much confusion and debate as to develop a system change strategy and intervention
system change, and many programs feel stuck when it plans that lay the groundwork for system change
comes to assessing it. The private sector development assessment, including how to set system boundaries
field has struggled to agree on an approach that and how to identify the system changes a program
programs can implement and stakeholders can aims to catalyse. It then explains how to assess system
understand. Consequently, practice varies widely, and changes using both an intervention lens focused on
many are frustrated. changes introduced by specific interventions, and a
helicopter lens that provides a whole system view. By
However, some mature programs are starting to analysing findings from the helicopter lens and the
assess system change more effectively. Building intervention lens together, programs can improve their
on these emerging practices, this paper outlines strategy and report on their contribution to system
a process that programs can use to assess system change. Figure 1 shows how this process fits into a
changes regularly and practically. First, it explains how typical program cycle.1
Assess
system
changes with
intervention
lens Analyse
Develop Develop system
Diagnose systems Intervention Implement
systems interventions changes and Report
strategy plans program
Assess contribution
system
changes with
helicopter
lens
1
This overview is the first of several papers providing guidance on assessing system change. More detailed technical guidance is forthcoming.
2
A Pragmatic Approach to Assessing System Change
2
For more discussion on system change see the BEAM Exchange website.
3
The helicopter and intervention lenses resemble the ‘Top down/Bottom up’ framework. See Itad (2012) GEMS Results Measurement Handbook,
GEMS.
3
A Pragmatic Approach to Assessing System Change
The system strategy describes how the support systems (in blue) are expected to influence the main - maize - system (in
green) in order to impact smallholder farmers (in yellow). The blue arrows visualise the linkages among the systems. An
intervention plan describes one intervention, visualised by the dashed orange arrows which, in this case, show working with
seed companies to target two supporting systems – hybrid seeds and related information about Good Agricultural Practices
(GAP).
Impact
Supporting
Functions
Post harvest
handling GAP info
equipment Hybrid seeds
Irrigation Supporting
Maize Functions
Supporting
Functions GAP
information
Post harvest Informal Norms
equipment Rules
Rules
Supporting
Rules
Functions
Supporting
Functions Hybrid seeds
Irrigation
Rules
4
The examples in this paper are based on PRISMA’s work in the maize sector in one area of Indonesia. The examples have been adjusted for
learning purposes and do not always accurately reflect the program’s work.
4
A Pragmatic Approach to Assessing System Change
System boundaries
Systems don’t operate in isolation; any given system is connected to multiple other systems. Programs, therefore,
need to delineate the boundaries of the main system they aim to influence, explicitly stating what’s included and
what’s excluded. Clear system boundaries help a program to develop effective strategies and to assess and
report system changes relative to the bounded systems.
System boundaries are set according to where opportunities and constraints lie relative to the program goal, and
according to what is achievable given the program’s timeframe and resources. This includes clearly defining the
target group and making choices about which geographical area and which interconnected systems – sometimes
called ‘supporting systems’5 – to focus on (see Figure 3).
Impact
Supporting Supporting
Functions Functions
Standards
Other information
Supporting
Functions
crops Supporting
Functions
Supporting
Rules
Functions
Fertilizer Post harvest Finance
GAP info
Rules handling
equipment Hybrid seeds
Irrigation Supporting
Rules
Functions
Maize
Rules
Supporting
Functions
GAP
information
Post harvest Informal Norms
equipment Rules
Rules
Rules
Supporting
Supporting
Supporting Functions Supporting
Supporting Functions
Functions Functions
Functions
Figure 3. Example of delineating system boundaries. In this case, the program has chosen
maize as the main system, excluding other crops, and decided to include four critical supporting
systems while excluding the others.
Boundaries may be redefined over time as programs learn more about what is relevant to the changes they aim
to achieve.6
5
The Springfield Centre (2015) The Operational Guide for the Making Markets Work for the Poor (M4P) Approach, 2nd edition funded by SDC and
DFID.
6
Fowler, Sparkman and Markel (2016) Disrupting System Dynamics: A Framework for Understanding Systemic Changes, LEO.
5
A Pragmatic Approach to Assessing System Change
System strategy
A system strategy summarises the boundaries of the system, describes the starting state and desired state for
expected changes in the system, and explains the plan for how the program intends to catalyse those changes.
It is helpful to develop the system strategy immediately after system diagnosis, before implementation starts.
The system strategy lists expected changes in both the main system, and in targeted supporting systems. It
records data on the starting state and describes the desired state for each of these expected changes. Questions
to guide the articulation of changes include:
• Who is doing what in the system now and who is expected to do what in the future?
• What do they have access to and use now and what are they expected to have access to and use in the
future?
• What are the rules and norms now and what are they expected to be in the future?
• What interactions are happening now? How do actors relate to each other? How are interactions and
relationships expected to be different in the future?
• What is the performance of the main and supporting systems, and what is their desired performance in
the future?
The system strategy also includes a plan that explains how the program intends to influence the system to
achieve the listed changes. It describes how the program plans to catalyse changes within supporting systems
through their portfolio of interventions and how these different intervention-driven changes are expected to
interact and lead to changes in the main system. It also shows how system changes are expected to benefit the
program’s target group.
The system strategy can be summarised using a system results chain (Figure 4) and a table (Figure 5). The
results chain visualises how a portfolio of interventions could contribute to changes in supporting systems, how
changes in supporting systems are expected to change the main system, and how that might affect the target
group. Although results chains are a linear picture of complex system changes, they provide a useful framework
for thinking through a strategy and planning monitoring activities.
Farmers use approp. Farmers use irrigation Farmers apply Farmers store maize
Farmers use Farmers use to benefit from changes
inputs & GAP systems and GAP for good post
hybrid seeds GAP in market demand
for dryland farming maize cultivation harvest practices
Retailers sell hybrid ISPs sell inputs ISPs sell equipment Service providers sell
seeds and give advice and advice and advice post harvest equipment
and advice
The table sets out boundaries for the main system and records data for the
starting state and the desired state for key changes anticipated in the main
system and supporting systems. It also shows how interventions are expected
to contribute to changes and explains interactions between supporting
systems in a way that a results chain can’t easily capture.
Boundaries Maize that is, or could be, produced and sold by smallholder farmers on Madura Island.
Indicators Starting Plan 2020-2025 Desired
system state system state
Main system: Maize
Volume of maize sold 395,000 tonnes The program will first focus on increasing 500,000 tonnes
from target area the supply of hybrid seeds and embedded
% of maize sold that 5% information on good agricultural practices (GAP) 15%
is highest quality for small farmers from private and public actors.
grade The resulting increase in yields and interest
in transacting with small farmers is expected
Private companies Unusual, 2 companies to drive changes in other supporting systems Becoming the norm; at
target small farmers and encourage small farmers to become more least 8 companies
as buyers/suppliers commercial. Once farmers start to increase yields,
Farmers’ perceptions Mainly subsistence the program will add a focus on reducing post- Both subsistence and
of maize crop harvest losses. As farmers start shifting to more cash crop
Etc. Etc. commercial production, the program can likely add Etc.
work in dryland farming and irrigation. Etc.
Supporting system 1: Hybrid maize seed
Volume of hybrid 150,000 Kg The program will work to 375,000 Kg
seeds sold on 1) increase private sector investment in the
Madura Island commercial distribution of hybrid seeds to small
Number of farmers 30,000 Farmers (8%) farmers with embedded information on GAP, and 75,000 Farmers (20%)
buying hybrid seeds 2) improve public-private coordination in hybrid
Number of 1 seed distribution. 4
companies selling These two changes are interdependent. The
and advising on program will also encourage a greater flow
hybrid maize seeds of information about hybrid seeds in order to
Farmers’ perceptions Risky, influence informal norms and increase demand. Useful, requires
of hybrid seeds unnecessary Improved access to irrigation and post-harvest right GAP
services will support, but not drive, changes in the
Etc. Etc. Etc.
seed system. Etc.
Supporting system 2, etc.
Programs learn more about the system as they develop and manage
interventions and monitor and assess their impacts. System strategies will
become more robust as they are revised over time.
7
A Pragmatic Approach to Assessing System Change
Intervention plans
Intervention plans outline how and when interventions Programs usually start interventions by partnering with
are expected to lead to specific changes in one or system actors to influence their behaviour. To increase
more supporting systems. They show what is expected the likelihood of system change, programs carefully
to change, for whom, how changes are linked to choose partners and design and manage interventions
program activities, and how changes are expected to over time to not only influence partners but also other
spread. They also show how changes in the targeted system actors. The goal is that system actors adopt
supporting system(s) lead to changes in the main and own new behaviours at scale – relative to the
system and contribute to impact for the target group.7 boundaries of the targeted system – and that other
changes in the system reinforce the new behaviours,
making them more resilient.
Farmers earn
higher income
Retailers sell hybrid Other Retailers sell hybrid Other Retailers sell hybrid
seeds and give advice seeds and give advice seeds and give advice
Activities
An intervention plan is typically visualised in an intervention results chain (Figure 6), with associated indicators
to assess progress.8 Intervention plans are developed just before or in the initial stages of intervention
implementation and regularly revised based on incoming information.
7
Similar partnerships may be combined under one plan; system change rarely occurs as the result of just one partnership.
8
See Kessler, Sen and Loveridge (2017) Guidance to the DCED Standard for Results Measurement: Articulating the Results Chain, DCED.
8
A Pragmatic Approach to Assessing System Change
Intervention lens
The intervention lens follows the spread of a specific The starting point for assessing system change
change introduced by an intervention. It tracks new through the intervention lens is the intervention plan.
behaviours from program partners to other system Most programs have established monitoring and
actors, examining how far the change spreads and results measurement (MRM) systems that guide
whether it will stick. Questions include: program staff to monitor changes and assess impact
in accordance with intervention plans.9 Intervention
lens assessments can mostly be combined with these
• To what extent do market actors own the regular MRM activities.
introduced change? Who does, or doesn’t?
Why? However, it can be difficult to predict exactly how and
when a system will change, so programs also need to
• What is the scale of the change, relative
keep an eye out for possible signs of system change.
to the whole system? Why has, or hasn’t, it
This requires an investigative approach, obtaining
scaled?
information from diverse sources – including, but also
• To what extent, and how, is the change extending beyond, program partners – and recording
reinforced by other parts of the system? information regularly. Any signs of change can be
probed to better understand the nature and extent
of the change, to validate findings with a stronger
The intervention lens also examines changes up the evidence base, and to further explore the program’s
results chain. It assesses to what extent changes in contribution to the change.
the targeted supporting system(s) affect the main
system and to what extent those specific changes in
the main system affect the target group.
9
See, for example, the DCED Toolkit for Implementing the DCED Standard, available at www.enterprise-development.org
9
A Pragmatic Approach to Assessing System Change
Helicopter lens
The helicopter lens is focused on big picture changes: The starting point for assessing system change
What changes are happening in the main and with the helicopter lens is the system strategy. The
supporting systems? What is driving these changes? helicopter lens is used to compare the state at the
How, if at all, do these changes relate to or reinforce point of assessment with the starting and desired
each other? Has the performance of the main system states for the changes described in the strategy and
or targeted supporting systems changed? to look for interactions among changes. The challenge
is to focus on the changes included in the system
The helicopter lens complements the intervention strategy, while keeping an eye open for changes in
lens by capturing broader changes. For example, other supporting systems, and for unexpected effects.
the helicopter lens assesses changes to the volume Too narrow a focus may lead to missing crucial
of maize produced by smallholder farmers across information; too broad a view may be resource-
the whole system, whereas the intervention lens intensive without adding much relevant information.
only captures changes to the volume of maize
produced as a result of specific interventions. Often, A simple assessment plan can help find the right
system changes are caused by the interaction and balance. The system strategy and the guiding
accumulation of multiple interventions and may questions (see the box on page 5) will help to define
not even be targeted by any single intervention. For what changes – expected and unexpected – need
example, the helicopter lens might find that maize to be assessed. The format in Figure 7 provides
farmers are becoming more commercially oriented as simple questions to help programs translate each
a result of the combined effect of changes in three change into specific research questions and to think
different supporting systems. This would not be easily creatively about information sources and methods.
captured by any one intervention lens assessment. In After completing this ‘thinking process’ for each
addition, the helicopter lens allows programs to assess system change to be assessed through the helicopter
changes to systems caused by external factors. lens, programs can combine the actual information
Monitoring these changes informs program strategy gathering required with other MRM activities to avoid
and supports analysis of the program’s contribution to duplication.
system changes.
1 What do we aim to assess? Are maize farmers shifting from subsistence to commercial maize farming?
2 What do we need 3 Who has 4 What type of 5 How to collect this 6 When and how
to know? information about information do they information? often to collect this
this? have? information?
How do maize Smallholder farmers Perceptions and Poll farmers at farmers’ End of season
farmers perceive (wide representation opinions markets
maize farming? across system)
Information on volume Interview traders by
Are volumes of maize Traders phone
of maize traded
traded increasing
across the whole Interview district officers
District agricultural
system? at annual events
officers
Figure 7. Partial helicopter lens assessment plan using the maize example
10
A Pragmatic Approach to Assessing System Change
The helicopter lens enables programs to assess broad system changes, while the intervention lens enables them
to assess the scale, sustainability and impact of the changes introduced by program interventions (see Figure 8).
Impact
Helicopter lens:
What has changed?
How has it changed? Supporting
Functions
Post harvest
GAP info
handling
Irrigation
Hybrid seeds
Supporting
Maize Functions
Supporting
Functions GAP
information
Post harvest Informal Norms
equipment Rules
Rules
Supporting
Rules
Functions
Supporting
Functions Hybrid seeds Intervention lens:
Irrigation
Rules What has changed as the result of the intervention(s)?
Rules
How has it changed?
Why has it changed, why not?
By combining findings from both lenses, a program in Madura are becoming more commercially oriented,
can understand changes in the systems it is targeting due to higher maize yields, more opportunities to sell
and form a credible picture of whether and how it maize and more access to information.
has contributed to changes in the main system. The
findings from several intervention lens assessments Taken together the two lenses give a picture of system
show how program activities lead to changes in changes in both the main system and supporting
targeted supporting systems, which in turn lead systems, and show how program interventions
to changes in the maize system and contribute to contributed to these system changes, and therefore
impact. For example, intervention lens assessments to impact. In the maize example, they indicate that
show that one intervention successfully encouraged the program contributed to the trend of small farmers
private seed companies to invest in distributing hybrid in Madura increasing their sales of maize. The two
seeds in the area and another strengthened extension lenses also show that the program contributed to
officers’ knowledge. An impact assessment indicates two of the key reasons for farmers becoming more
that farmers’ use of the hybrid seeds and increased commercial: higher maize yields and more access to
access to information from both extension officers information. This change in farmers’ norms regarding
and commercial seed retailers contributed to higher maize is likely to drive further increases in maize sales
maize yields. and farmers’ benefits.
Meanwhile, the helicopter lens captures the combined A culture of honest enquiry is required to assess if
effect of multiple changes, including those caused and how program interventions have contributed to
by external factors. For example, findings from the system changes. The purpose is not to rigidly classify
helicopter lens may show that the total volume of maize the significance of the contribution, but to provide a
traded by small farmers in Madura has increased from transparent and evidence-based explanation of what
395,000 tonnes to 450,000 tonnes. The helicopter has changed, why, and the role of the program in
lens assessment also suggests that maize farmers influencing those changes.
11
A Pragmatic Approach to Assessing System Change
Frequency depends on the business cycles; most programs review intervention plans 2-4 times per year.
10
Frequency depends on the business cycles; most programs review system strategies once or twice a year.
11
12
A Pragmatic Approach to Assessing System Change
5 Report transparently
Most stakeholders want programs to report three using a mix of qualitative and quantitative indicators,
things about system change: 1) what changes have and differentiating between empirical evidence and
happened, 2) to what extent and how the program interpretation.
contributed to system changes, and 3) how the
program is responding to a changing context. By using To report on contribution, a program can use findings
findings from both intervention lens and helicopter from the intervention lens to show how changes
lens assessments, programs can report credibly on all to supporting systems link to program activities,
three of these. and findings from the helicopter lens to explain
transparently how changes to targeted supporting
To report on what changes have happened, programs systems and other, external factors caused changes
can use the findings from both lenses to explain to the main system. This enables the program to
changes in supporting systems and the main describe clearly its contribution using evidence from
system in relation to system boundaries. A strong both lenses.12
description covers both the results of interventions
and how changes to the main system and relevant Finally, to highlight adaptive management, a program
supporting systems interact and affect systems’ can explain how it is using findings from both lenses
performance. Systems are messy, so reporting will to review and revise its system strategy and portfolio
necessarily be a simplification. However, a program of interventions, in light of both expected and
can report rigorously by drawing on both lenses, unexpected changes.
12
Systems are too complex and multi-faceted for programs to plausibly claim that changes are solely attributable to them.
13
More information on the DCED Standard is available at www.enterprise-development.org/measuring-results-the-dced-standard/
Hans Posthumus Consultancy
www.hposthumus.nl
hans@hposthumus.nl