Not all measures of impact are created equal. The Impact Robustness Scale is a practical way to rethink and refine the methods you use to demonstrate outcomes, report your impact and improve your service.
This guide talks through the nine levels of Impact Robustness using the example of a Relationships service.
1. Anecdotes
Example: “This programme fixed my marriage!”
Anecdotes provide some positive feedback about the programme which can be motivating for staff. Anecdotes are selective so they do little to tell you whether everyone is experiencing those positive effects of your programme.
2. Case Studies
Example: “Bill and his partner had been married for five years but after medical complications with their son, things started to decline...”
Case Studies are great tools for building empathy for your service users among potential supporters. However they are selective so give no indication as to whether the people in the case studies are the exception to the rule or the rule.
3. Number of Activities & Attendees
Example: 10 workshops held, 60 attendees in total
These enable you to create charts which can look impressive, especially if the numbers are big. But whilst they tell you how many people you reached, they do nothing to prove whether anything changed as a result of you reaching those people.
4. Retrospective Surveys:
Example question: Has this programme increased the affection you show to your partner?
Retrospective surveys which look back, tell you whether people think that the programme has had a positive effect but that is not the same as whether the programme did have a positive effect.
5. Before & After Surveys:
Example question: How much affection do you currently show to your partner?
Before & After are more robust because they give you a baseline (a result from everyone at the start) and then they allow you to show the distance travelled, e.g. “30% of people went from showing their partner no affection to showing their partner a lot of affection”. However these are limited by each respondent’s level of self-awareness; they might think that they are showing more affection, but how do you know if what they define as affection is the same as what you define as affection?
6. Before & After Surveys with non-subjective Questions:
Example question: How many times did you hug your partner this week?
These get around the problem of definitions because they ask about observable actions which are unambiguous. When you ask a range of questions like this, they give you a more accurate and colourful picture of whether change is actually taking place. However given that the objective of the programme is not to increase hugs, but is to improve relationships; these questions fall short because they do not tell you whether the relationship has actually improved.
7. Before & After Surveys aimed at both the person on the programme and their partner who is NOT on the programme
Example question to the partner of the person on the programme: To what extent do you agree with this statement: I am pleased with the state of my relationship? Strongly Disagree to Strongly Agree
This question gets to the heart of what your programme is working to achieve. Asking this question at the beginning and the end of the programme allows you to calculate the distance travelled. E.g. 80% of people felt that the state of their relationship improved
8. Track whether your programme has a lasting effect:
Example: One year later: how do they rate how pleased they are with their relationship?
This tells you whether your programme has a lasting effect or not.
9. Compare your programme’s results with what happens when people are not supported
Example question at the start and end of your programme: To what extent do you agree with this statement: I am pleased with the state of my relationship? Strongly Disagree to Strongly Agree
These allow you to say, 80% of people on our programme felt that their relationship improved whereas only 25% of people who were not on our programme felt that their relationship improved over the same period. This allows you to demonstrate with a degree of scientific rigour, the impact that your programme makes. (Randomised Controlled Trials)
At Makerble we work with clients at each stage of the Impact Robustness Scale helping you balance insight with proportionality.