Manage collective impact evaluations in which organisations are the Data Subject

This article outlines how to use the Makerble platform to evaluate a collective impact programme in which your evaluation focuses on organisations as the beneficiaries rather than their service users.

Each of partner in the collective impact programme would exist as a project on Makerble

Every project has a logic model composed of five stages:

• Activity outputs

• Participation outputs

• Short term outcomes

• Medium term outcomes

• Long term outcomes

You can create an Activity Output for each activity being done by the partners. 

• When progress is logged towards Activity Outputs, there is no need to tag individuals

• When progress is logged towards Participation Outputs, you can simply log the number of people who attended (a workshop for example) and if you want to, you can disaggregate that number of attendees e.g. by Age as shown in the example below.

It's fine for multiple partners to use the same Activity Outputs or for each partner to have their own set of Activity Outputs. In scenarios where they are the same, the system will aggregate the results together in an Album. You can see this on the Progress Board below. Cards 7 & 8 are the 'Album' view that aggregate the results from the three partners (called Strategic Initiative, Partner 2 and Partner 3)

For expediency, you can save a logic model as a template and apply it to multiple projects at once. (we call it a strategy)

You can engage the individuals in Participant Surveys. You can deploy these in a number of ways. The two options most suitable for an evaluation where organisations are the data subject are likely to be the Responses are Anonymous option or the Send Personalised Links via Email option.

• With Responses are anonymous a public web address is generated which anybody with the link can use to complete a survey

• With Personalised Links via Email, each individual participant receives a unique survey link that only they can use to complete the survey

Evaluate a Collective Impact programme using Makerble Apps

Makerble Apps allow you to embed the five pillars of Collective Impact

This article outlines how you can combine the impact of multiple organisations on multiple beneficiaries in one platform

1. COMMON AGENDA

Each organisations has their own login credentials to your Makerble platform. You can achieve this by displaying a logic model to represent the Theory of Change of each strand of work. This enables each partner to easily understand the outcomes you are focused on and the impact strategy to get there.

Turn your Theory of Change into a measurable, visual, interactive logic model

Relevant features:

• Discuss your Theories of Change. Add comments. Save them to your library

• Apply a master Theory Of Change to all projects within an area

• Set targets for each Activity output, Participation output, Outcome and Indicator

2. MUTUALLY REINFORCING ACTIVITIES

Collective Impact programmes often bring together a diverse set of stakeholders with different data collection needs. Makerble allows you to collect data in a variety of formats whether quantitative or qualitative.

See a holistic view of a single service user / client / beneficiart

Relevant features:

• Each Organisation has its own Profile Page within the Impact Hub

• Organisations within the same Collective Impact area can see each other’s Project Pages and thus see the activities they are doing

• The Single Beneficiary View on the Makerble Impact Hub shows you the different activities and projects that each beneficiary takes part in

3. SHARED MEASUREMENT

For collective impact to work, organisations must collect data regularly which measures progress and results in a consistent way. This makes alignment and accountability easier to achieve.

A progress board on Makerble created using The Boards App

Relevant features:

• APIs, Integrations and Unique Identifiers which enable the Makerble Impact Hub to connect with partner organisations’ existing databases

• Intelligent dashboards which provide real-time view of progress against every output, indicator and target

• Direct Data Entry into the Impact Hub is possible. Organisations that do not have an existing database can easily use the Impact Hub as their primary beneficiary management tool.

4. CONTINUOUS COMMUNICATION

Seamless communication about progress is essential for organisations within a Collective Impact area. Teams working across different organisations ought to be kept in the loop on progress as though they were within the same organisation.

View updates and comments in the Timeline

Relevant features:

• Shared Updates which anyone within the network can Comment On

• Ability to tag beneficiaries and colleagues within Updates

• Private Messaging

• Shared Library of Metrics

5. SUPPORT OF A “BACKBONE” ORGANISATION

For Collective Impact to be a success, there needs to be a backbone organisation. The Makerble provides backbone organisations with administrative tools to ensure see which KPIs are being tracked by which partners.

See the indicators being measured across each of your surveys

Relevant features:

• Ability to make Project Pages public so that funders, journalists and members of the public can see the impact

• Donation Buttons which allow people to support the work of the Collective Impact area in a click

• Administrative Tools which enable organisations, projects and individuals to be easily added to or removed from the Collective Impact area.

Evaluating a programme from the outset

We begin evaluations by finding out the purpose of the research. Typically this falls into one or more of these categories:

• Influencing policy: we identify what we need to prove in order to convince policy-making audiences and the data we need to collect that will serve as reliable evidence

• Continuous learning: we identify outcomes which indicate the extent to which the programme is working and which provide insight into where improvements might need to be made once the programme is underway

• Evaluation reports: we create a research plan that enables us to create a baseline report, interim report and end-of-programme report which demonstrate the difference made and the reasons driving the impact

Outcome Discovery is a process we use to determine the expected and unexpected outcomes that your programmes achieve. This involves looking internally - engaging with staff and volunteers - as well as externally by listening to beneficiaries to understand the impact from their perspective. We look for existing evidence of the long-term outcomes of your work and standardised outcome frameworks already used within your sector, e.g. The Outcome Frameworks and Shared Measures database created by The National Lottery Community Fund.

The Process Of Change is one of the tools we use to make sense of the complex set of outcomes that often arise during the Outcome Discovery phase. By grouping outcomes into one of three types of change, we can accelerate agreement around the language we use to describe the difference your programmes make. We group outcomes into:

• Changes in how people Think & Feel: i.e. outcomes related knowledge, attitudes, beliefs and internal capacity

• Changes in what people Do: i.e. outcomes related to behaviour, habits, achievements and corporate or government policies

• Changes in what people Have: i.e. outcomes related to wellbeing, quality of relationships, health and wealth

Data Collection: With the set of outcomes confirmed, we drill down to a practical set of indicators which can be woven into existing data collection processes wherever possible and supplemented by additional data collection when needed.

This culminates in the creation of the baseline, interim and final evaluation reports required.

Continuous Learning: In addition to the summative evaluation, we use workshops and digital tools to make the evaluation formative so that your delivery teams have the opportunity to reflect on the learnings surfaced by the evaluation at regular intervals.

To find out more about our approach to evaluation including Makerble Learning Boards, contact Matt Kepple who leads our evaluation practice. Email: [email protected]. Phone: +44 (0) 7950 421 815.

Evaluating a Collective Impact programme

Collective Impact programmes present unique challenges for evaluations due to the often complex arrangement of partners who are each producing data that could inform an evaluation. To get around this, we take a collaborative approach to planning the impact methodology which intentionally draws upon the insight and expertise of the respective partners. Our approach is anchored to the Five Pillars of Collective Impact. The fifth pillar, a strong backbone organisation, is typically the ‘lead’ organisation within the partnership which commissions Makerble as the evaluation partner in the first place.

Setting a common agenda is an essential first step to evaluating a Collective Impact programme. We use our proprietary Process Of Change™ framework in workshops with partners to enable everyone to agree on shared language that describes the outcomes being achieved at each stage of a person’s journey through your programme. This typically involves mapping out the ‘customer journey’ that a service user, participant or beneficiary might take through your programme.

Agreeing to Shared Measurement requires partners to distil a pragmatic set of indicators which will be reported across the programme. These shared measures open the door to continuous improvement as partners are able to see their collective results and have open dialogue about factors which are acting as enablers and inhibitors to people’s progress through the programme. Part of the role Makerble plays when evaluating collective impact programmes is in providing the dashboards which enable shared measurement to happen and facilitating the conversations which enable continuous reflection, learning and iteration to take place.

To find out more about how we can conduct the evaluation of your Collective Impact programme, contact Matt Kepple via email on [email protected] or over the phone on +44 (0) 7950 421 815.

To find out more about the tools and techniques we use, visit our Building Blocks of Measurement.

Collective Impact Client Stories:

• Right To Succeed

• Trust For London

Evaluating a funder's impact

Most funders approach impact reporting by communicating these three things:

• Case studies about grantees

• Amounts spent, split by cause and/or location

• Number of grants made, split by cause and/or by location

Whilst this is a good place to start, it only tells part of your story. We help funders go further to measure the difference that your grants have made.

We recognise that as a funder you have three levels of impact:

• Aggregate impact: the combined outcomes of the projects you make grants to

• Grants Plus impact: the difference that your support makes to your grantees’ projects beyond the financial benefit, for example, increased profile or improved beneficiary recruitment

• Strategic impact: the extent to which you are achieving your mission as defined in your Theory of Change

At each level of impact there is the opportunity to report on the outputs and outcomes of your work. That said, decisions about impact measurement need to be mindful of the capacity of grantees so as not to burden them with reporting requirements which are disproportionate to the funding they receive. 

At Makerble we use our Impact Measurement Methodology to identify the minimal set of metrics you need to measure and devise practical ways to measure them.

We can work with you across the full gamut of impact measurement or we can focus in on the one or two areas which are most important to you. Typically our work splits into these three phases.

1. Outcomes Framework: identifying the outcomes and metrics you need to measure

2. Data Collection: creating a data collection plan and either implementing it ourselves or supporting you to systemise it

3. Reporting: analysing your data, producing reports and giving you access to interactive dashboards that show you your data in real-time so you can use it to enhance decision-making

Contact us to talk through your current approach to impact measurement and explore how you can take your foundation’s impact measurement to the next level.