Measuring Impact: Whose View Counts?

(Parts 1 & 2)
Thursday May 15
Session reporter: Anh Ton
Moderated by Jeremy Nicholls (JN), Chief Executive Officer of The SROI Network; Terence Yuen
(TY_, Project Officer, Center for Entrepreneurship Chinese University of Hong Kong (CUHK)

Summary of the content of the session:

Questions posed to workshop attenders by JN:
Why do we measure? Charities need to measure impact to prove their worth; funders want
to know their resources are being used efficiently, governments want value for their money,
find out how much of a difference youre actually making
Output, outcomes, impact, and productivity gains what do all these things even mean?
Can we take into account environmental impact as well?
Is social really just about reducing inequality?
Are you looking at a more granular image (household, individual, families) or a big picture
(society-wide impact)?
Implication
Different stakeholders have different drivers for measuring impact
We dont all agree with what were trying to achieve, but we all try to use the same language

Workshop exercise (Part 1)
Attendees broke up into fie groups: investors (grantmaking); investors (lending); social enterprise
mangers; social enterprise board members; beneficiaries; auditors. All five groups were confronted
with the same project. An SE is trying to help ex-offenders gain employment. Would question would
you want answered to know how much difference has been made or how much value has been
created?

Ideas from the exercise:
We have to look at the counterfactuals. We have to examine if this is the best use of funds
by looking at what other alternatives exist and what would have happened in the absence of
the program who is contributing to this cause? There will always be other factors
involvedhow can we determine our program has any correlation to change?
Value should be brought to both the ex-cons and to the community at large. But is that value
one in the same? Do we all want the same things?
The accuracy and rigour applied to the measurements should be proportional funding. In
other words, dont waste money on things that dont influence decision-making.
Conversations have become too technical and forget the voice of the beneficiaries. When
we come up with metrics, we rarely ask the beneficiaries what they think. We spend a lot on
auditing to protect against financial abuse, but shouldnt we audit the beneficiary responses

as wellto protect their point of view? It is odd that more money is being spent on the
means to the ends than the ends itself.
There is impact measurement to inspire marketing, and there is impact measurement to
learn. Realize that there are a lot of changes and outcomes and changes people will
experience. It is not simply a matter, in this case, of getting a job and not getting a job.
Realize that there are grey areas in impact measurement. You cannot rely on one statistic.

Workshop Part 2
Accountability & Stakeholders
Are nonprofit boards less accountable than their for-profit counterparts?
Are their representatives of the beneficiary community on your board?
In the for-profit world, there are legal requirements put in place to protect customers. Isnt
it bizarre that beneficiaries have no such protections?
In the end, a lot of social policy can be boiled down to governments acting on behalf of
stakeholders.

Once you have collected all your data, when do you use it to make changes to your programs?
Most organizations only measure for positive change; you just think about whether you are
meeting goalsyou dont even think about when to change. Are you taking account
negative consequences into account?
What is the value of your outcomes in financial terms?
You can build a weighting structure that takes into account multiple factors to help decide
when to change; have frank conversations about how efficiently youre turning resources
into value.
So much information is being asked for, but not being used in decision making. So why do
you burden investees/grantees with this information?

Impact frameworks
Inputs, Outcomes, Outputs, Impact
o Outcomes can go on forever; you can get individual stories from outcomes.
What are usually missing from change maps are the FAILURES the folks who dont achieve
your desired behavior.
Impact measurement should be uncomfortable process if you dont ask about the failures,
you will over-claim. Sometimes, you will need to re-focus or go wider based on what you
learn from our failures. You must be selective in the situations you try to solve.
o Example given: With the ex-con program, the people who fail to get jobs may end up
even more depressed and ostracized. Your program would have then caused more
harm than good. What do you at this point?

Outcome vs. Impact (two ways of looking at it)

Impact = outcome what would happen away
Impact = further away from outcome (to the wider community)

Major conclusions of the session:
Measuring impact should be an uncomfortable process that helps you reflect on the efficacy of your
programs. Within the social space, they are especially difficult because there are many things you
cannot quantify. Even so, the conversation on impact has shifted so far to technical discussions that
we fail to take into account the voice of the ultimate beneficiary. There needs to be greater
protections for the beneficiaries, and we need to look closer at the cycle of accountability.

Feedback/Take-Aways for the EVPA:
-Take failure into account. Dont just measure for positive change you will end up over-claiming
your impact, and you may produce a net harm. Use measurements to constantly evaluate your
programs and decide when to make changes if your measurements show that youre not achieving
your desired outcomes.
- Dont collect more information than is necessary. You will only overburden grantees and waste
time. Collect only the information you need to make decisions.
- Give more voice to the beneficiary in creating your metrics. Ask them what they need and what
they consider success.