At our launch event in January, we ran three breakout sessions. A summary of workshop 1 can be found here. This is a summary of workshop 3.
What’s the difference between outcome measurement and impact measurement? Are these synonyms for the process of working out the change we create? The answer to this is a yes and a no. These processes definitely both help us understand what changes for the people we seek to support, and often in workplace chat these terms are used interchangeably. However, from a practitioner in this field you are much more likely to hear a loud ‘hell no!’, because these ideas are very different, and the distinction is extremely important if we are going to make good decisions. Let me explain.
If we want to find out what effect we are having on our stakeholders and how valuable it is to them, or more importantly, how we can create more value for people, we need to understand not just what changed for people (this is outcome measurement), but also what would have happened had we not intervened (deadweight), and then what part of this change was due to us and what to other people (attribution). When we understand all these factors, that is impact measurement. Let’s explore these two ideas.
Deadweight (what would have happened anyway) – Would you want to fund a programme with a goal of helping children aged 7 – 12 to grow foot taller? Being taller is often handy, but obviously this was going to happen anyway, so we would not fund it. Better to fund something that is not necessarily going to happen, such as supporting people with disabilities to get a job, or helping young people with mental health challenges to build resilience. This is an obvious example, but what happens when this same dynamic is taking place but we can’t see it as clearly? If we are choosing between two programmes to invest in, we need to understand the outcomes that would have occurred anyway and account for this in our calculation of value, in order to make good decisions.
Attribution (how much of the change is due to us) – Imagine you have two programmes. In one, according to stakeholders you are fully responsible for supporting their outcomes, in the other there are another three organisations involved. How valuable is your work, should you keep investing in it? Understanding the contribution each agency makes and how valuable the outcomes are to our stakeholders is the information you need to confidently make these decisions. By not addressing the issue of attribution in evaluations, it could be argued that most evaluations imply that all the outcomes are due to them. Whilst we can’t all afford a randomised control study for our work (a great way to work out attribution) we can approach this idea from other angles, and we should if we want to really understand impact.
Additionally we need methodologies that ensure we uncover the unintended outcomes of what we do, both positive and negative. Is it okay for a scholarship programme to support some young people positively, while others feel deflated and reduce their sense of what they can achieve (think about the arguments around class streaming)? Is it alright to create a new homeless shelter when a number of elderly neighbours feel more fearful? If we have approaches that make these negative outcomes clear, we have a motivation and a responsibility to reduce these. It’s for this reason and the discussions on deadweight and attribution above that social value practitioners work in line with an internationally agreed set of principles. These can be applied to different types of impact studies and are the basis of the Social Return on Investment (SROI) methodology. This approach also does the hard work of describing what change stakeholders most care about and how they value this. If we want policy that reflects the needs of beneficiaries then this step is also crucial.
Why is the distinction between outcomes and impact important? The answer is simple: inequality and the environment. We have limited resources as a society and it’s up to us to make sure these are being used to create the most positive change that we can, and as quickly as possible. We need policy which is people- and planet-centred, ideally even upending some of the traditional assumptions that have led to environmental degradation, and kept inequality stubbornly high. To do this we need evidence for what works and what creates real change. Evidence-based decision making means understanding not just the outcomes created by various policy and programmes – but the impact of our actions.