Project Impact Assessment

O’Flynn, M. (2010). Impact Assessment: Understanding and assessing our contributions to change. M&E Paper, 7.

Kirkpatrick, C., & Hulme, D. (2001). Basic impact assessment at project Level. Enterprise Development.
 Outline
• What is impact?
• Why do we perform Impact Assessment (I.A.)?
• When do we perform Impact Assessment?
• Difference between I.A. and Monitoring & Controlling
• Different Approaches to I.A.
• Different Methodologies used in I.A.
• General I.A. Framework
• The nature of change and accordingly, the areas of
inquiry in I.A.
• Special Considerations
• Key challenges in I.A.
• Mitigations to the challenges in I.A.
 Think about an event or a person who made a
significant impact on your life (a birth, a death, a
relationship, a marriage, an accident, a boss, an
investment…)
 What is “impact”?
What does impact mean in relation to this? What
concepts or ideas does it include?
A key word is probably CHANGE
• Change can be intentional (ex: a marriage) or
unintentional (ex: a car accident)
• Change can also be temporary or permanent
• Change can also be positive, negative or simply
different
 What is “impact”?

The assessment of impact involves understanding

the nature of the change that has taken place and
to determine its significance

To capture the concept effectively, we can use this

definition for impact analysis:
“The systematic analysis of lasting or significant
change – positive or negative, intended or not – in
people’s lives brought about by an action or a series
of actions” (Roche, 1999)
 Why Impact Assessment?
We should assess the impact of our project to:
- Demonstrate success (to donors, ourselves, the public),
both to justify funds received and to solicit further funding
- Improve the effectiveness of our interventions ; to make
a more significant, efficient and effective difference
- Be accountable to the people (stakeholders) for whom we
are working
- Advocate for changes in behavior, attitudes, policy,
procedures, governance… at all levels.

Basically to: demonstrate success and to learn what we can

do better.
 Why Impact Assessment? - continued
By doing impact assessment for current/past
projects, we can learn lessons for current or
upcoming projects, by allowing us to:
- Predict problems
- Find ways to avoid these problems
- Enhance positive effects
 When do we perform Impact Assessment?
Impact assessment may take place:
- Before the start of an intervention/project, to
forecast potential impacts as part of the planning,
design and approval of an intervention
- After completion of an intervention/project, to
assess actual impacts during and after
implementation, enabling corrective action to be
taken if necessary, and providing information for
improving the design of future interventions
- At any stage in between
 Impact Assessment vs. Monitoring &
Controlling
Let us begin with an example:
“An income generating project for women in Tanzania was very successful
in that it provided the women with money that they could use for feeding
and educating their children. The women in the community were delighted
with the success of the project. Monitoring reports confirmed that the
project was doing what it said it would do. The mid-term review and first
evaluation noted that it was an effective and well conceived project.
However, although the women were happy, other members of the
community were not. After a period of time, the project had to close as too
many of the women found that having access to their own income led to
tensions and violence within their homes. One of the unintended and
negative impacts of this project was that some women were being forced
(sometimes with violence) to hand over their earnings to their husbands
who used the money as they saw fit. The evaluation process did not pick
this up as it did not explore the impact questions.”
 Impact Assessment vs. Monitoring &
Controlling - continued
Impact Assessment Monitoring & Controlling

Focuses on changes Monitoring focuses on effort/work

Controlling focuses on outcomes/results

Focuses on affected stakeholders Focus on the intervention

Can be included at all stages and/or can Mainly during execution and towards the
be used specifically after the end of the end of the project
project/program
 Different Approaches to Impact Assessment
Three broad, and sometimes overlapping,
approaches of Impact Assessment are often used:

1- Third Party Retrospective Study:

More often than not, this assessment involves the
use of an external team or a third party who
conducts the assessment at a fixed point after the
completion of the project. The main purpose of this
approach is to test or verify whether the logic of the
project was correct: did it achieve the changes that
it set out to?
Different Approaches to Impact Assessment - continued

2- Process-driven Study:
The second approach is less judgmental than the first.
Stakeholders are involved in all stages of the project
lifecycle:
- in the design and the development of the project, they
are included in identifying the changes that they would
like to see
- during the implementation phase, they influence the
direction the project takes. They are consulted in the
process of monitoring and evaluation
- In impact assessment, they discuss and identify changes
that have taken place in relation to the project.
This type of assessment is empowering for stakeholders;
and it makes valuable contributions to organizational
learning.
Different Approaches to Impact Assessment - continued

3- Some Time Period after the Completion Study:

It is a study that takes place some years after the
completion of the project or program:
- To identify changes that have taken place within the
community/organization/environment,
- To assess to what extent these changes can be attributed
to the particular results of the project or program.
The purpose of this type of assessment is to understand to
what extent a project’s efforts are making a difference to
the stakeholders they claim to be working with and for.
Although this type of assessment is more complex and
time consuming, it is one that all project
managers/organizations need to consider carrying out
from time to time.
Different Approaches to Impact Assessment - continued

All three approaches – and variations on them –

have their own validity in relation to the main
purpose of assessing impact.

Often, a combination of elements of all three

approaches will be necessary in order to
understand and evaluate both planned and
unplanned changes.
Different Methodologies and Techniques used in Impact
Assessment
Each impact assessment performed will have a
precise purpose. Accordingly, every impact
assessment will have a specific set of questions to
tackle, a different context, a certain budget, a
certain research capacity, and other characteristics.
That is why a wide range of methodologies has
been developed.
An impact assessment may include any or all of
these methodologies:
1- Quantitative Statistical Methods
2- Qualitative Methods
3- Participatory Methods
Different Methodologies and Techniques used in Impact
Assessment - continued
1- Quantitative Statistical Methods: involving
baseline studies, definition of objectives, target
setting, performance evaluation, outcome
measurement, etc. Such methods can be costly,
limited in the type of impacts that can be studied,
and may pose difficulties in linking causes-effects.
2- Qualitative Methods: suitable for investigating
more sensitive types of impacts (usually the social
impacts). Such methods require highly-skilled staff
and can be costly. Some degree of these methods
usually exist in all impact assessments
Different Methodologies and Techniques used in Impact
Assessment - continued
3- Participatory Methods: particularly suited for
exploratory low budget assessments and initial
investigation of possible reasons for observed
statistical relationships. These methods offer a
means for involving stakeholders in the research,
learning and decision-making. Some degree of
these methods is usually necessary in any impact
assessment, to achieve a good understanding of
stakeholders’ perceptions of impacts.
General Impact Assessment Framework
 The nature of change and areas of inquiry in Impact
Assessment
How should we understand and report on the
changes our efforts are making?
We would love for change to happen as we plan it,
but it doesn’t! Change does not happen in a linear
path (e.g. A leads to B which results in C).
Rather, it is:
- Complex in that many different changes can take
place simultaneously, due to different
simultaneous factors
- Continuous in that things are never static
- Variable in pace, scale and/or over the course of
any intervention
- Not necessarily lasting or sustainable.
 The nature of change and areas of inquiry in Impact
Assessment- continued
Types of inquiry which will support our understanding
include:
- What has changed?
- For whom?
- How significant was it?
- What factors contributed to the change? Who or
what was involved? How were our efforts connected
to it?
- What strategies were used to bring about the
change?
- What were the contexts that affected how the change
happened?
- What was the process or pathway of change?
- Will it last? Is it over?
 Special Considerations to never forget
Whatever the precise scope of the assessment is, and whatever
its objectives are, special consideration should be given to the
following potential issues:
- Time Dependency: might impacts that are small (or large) at
the time of the assessment increase (or decrease) with time?
- Changing or abnormal conditions: how secure is an observed
impact from shocks or changing factors/conditions which may
vary from those pertaining at the time of the assessment?
- Cumulative Effects: would a small effect become significant if
the project or its effects were replicated?
- Remote Effects: might unplanned impacts be occurring
beyond the boundaries of the study area or department or
community?
- Second order effects: might unplanned impacts be occurring
that are not obviously associated with the project?
 Key Challenges in Impact Assessment
Let us read and discuss this case study:
“Bahati is a 13 year old boy living in Arusha, Tanzania. Like many in his
community, he is HIV positive. He lives with his grandmother who has less
than a dollar a day to care for herself and the five grandchildren in her
care. Bahati is a pupil in the local primary school, he is a member of the
youth club at his local church and he is a star in the local football team. He
loves music and he has a wide circle of friends. All of the individuals and
institutions mentioned above aim to influence Bahati’s life on one way or
another.”

Notice that there are 3 development agencies trying to help Bahati: one
organization focuses on education, one on livelihoods and the third on
supporting people living with AIDS.
To what extent can the three development agencies realistically identify
changes that their specific efforts have made to Bahati’s life?
How do they really know what difference (if any) they have made to his
life?”
 Key Challenges in Impact Assessment - continued
As such, it is quite challenging to identify the impact of a project, due to the fact
that:
-Projects exist within certain organizational, regulatory, social, cultural, financial,
political frameworks
-There is a lack of organizational clarity about the differences between M&E and
impact
-Some projects include working with partners or different organizations
-There is difficulty in attributing evidence of change to specific interventions – we
don’t know what other factors (and when?) contributed to the change
-There are complex designs of impact assessments (or it attempts to address too
many needs)
-There is difficulty in identifying useful starting points from which to assess
impact, including baseline data and deciding which indicators to work with
-There is availability of too many tools and processes that designers of impact
assessments overly complicate; and/or require staff and partners to work with
tools that they are unfamiliar with
-There is difficulty in telling the truth, if this truth is negative to the project
-There is a lack of recognition of the value of this effort
 Mitigation for the Challenges in Impact Assessment

The best way to mitigate for the challenges is to PLAN for impact
assessment:
- Encourage the development of an organizational Theory of Change,
and develop a few broad dimensions of change that will inform the
direction of impact assessment areas of enquiry
- Develop clarity about a realistic organizational ‘sphere of influence’
to more effectively understand where impact can be measured
and/or assessed; and where changes can be illustrated but not
directly attributed to your organizational efforts
- Build impact assessment into existing planning, monitoring and
evaluation policies and systems. Build ‘change questions’ into
existing reporting formats and processes.
- Ensure stakeholder input in all stages of design and development of
impact assessment processes
- Consider using Outcome Mapping when planning interventions with
partners, networks and coalitions.
 Mitigation for the Challenges in Impact Assessment - continued

Also, when gathering change data, organizations

would probably benefit from:
- Using a few tried and tested methods (which
allow for triangulation) over complex or
innovative and untested ones that are not clearly
understood at field level
- Involving stakeholders in gathering information.
Build their capacity to do this effectively