6. Measuring impact
Projects such as INCLUSO have to produce results that show what happened during the course of their research. When the INCLUSO project was set up, academic partners had to devise a way to measure change in the behaviour of young people in relation to social media. The aim was to devise a measurement tool which would respond to the specific INCLUSO objective: 'INCLUSO aims to deliver verifiable proof that ICT, and more precisely, social software tools, can facilitate social inclusion of marginalized young people'.
In this section of the INCLUSO manual we show which measurement tools we used in our project and also give some practical advice about how to create and use these kinds of measurement instruments.
The measurement tool consists of three components (to be used before, during and after a social media project):
- Evaluation action plans (in which indicators are used to evaluate progress towards various outcomes)
- A questionnaire for youth workers (about their experience and reflections on ICT, social media and youth work)
- A questionnaire for young people (about their experience and reflections on ICT, social media and their lives, attitudes and behaviours)
The use of these tools and the outcomes can be found in two publications that are available for download:
- The INCLUSO measurement tool: measuring impact in the use of social media for youth at risk
- Impact results for the INCLUSO project: did social media support social inclusion of youth at risk?
Formulating goals for the intervention is essential in creating a measurement instrument. Performance indicators are used to show the extent to which missions and objectives are achieved. These can be goals at different levels (Van Yperen 2005):
- Goals of the activity. Here indicators can be used to assess the degree to which specific goals are achieved, as formulated by the client and the counsellor
- Goals of the institution. At this level indicators show to what extent the institution is able to successfully target audiences with specific forms of intervention
- Goals of the government/funder. Here indicators are used which represent the social role of the sector (government) or the goals of the funder
Performance indicators are useful because they help to focus on the work undertaken and its quality. To achieve this, two important things are needed. First, it needs to be clear which of these goals (above) is the most important to focus upon. Second, those goals should all generally converge on the same priorities. Of course, it is assumed that the funder pursues similar or the same goals as the organisation, and the other way around.
This was not the case in the early stages of the INCLUSO project. The pilot teams identified this issue and developed a diagram relating outcomes to activities in order to clarify reasons for their actions (see the diagram in Chapter 2: Define your goals). It is important to complete this task at the start of a project.
The pilots measured the effects of interventions in the activities. These measurements can result in the accumulation of a lot of relevant information. This varies not only between pilots, but also over time. For example, organising the work around the goal of contacting politicians can be fun (and educational) for young people, but this is only temporary and is not an overall goal. So, data of this kind is used to help identify and review such activities and events at the end of your project. Useful questions are: what were the successful interventions, what did they achieve, at what point in the project, and so on? This information will support you to fine tune your social media activities, so that you can maximise its effectiveness.
The extent to which organisations participating in the INCLUSO project reached their outcomes were measured by interviewing the INCLUSO staff and project staff about their activities. It's important here how the use of social media and the INCLUSO goals promote or hinder the way the organisation works. These questions are about two things (Huijsman & Groenewoud, 2003):
- Structural indicators. This is about the conditions for working on goals, such as the training of workers, the availability of computers and the support of management.
- Process indicators. These concern the question of how the organisation achieves its goals. They can work with competence tests, weekly appointments etc.
Due to the fact that working with social media interventions is fairly new, a strong foundation in the organisation is very important. So, structural indicators should have a major role. In Chapter 2 of this INCLUSO manual, some of the conditions for working with social media interventions in the youth work sector are summed up.
The measurement instrument measures the effects of the INCLUSO project on the young people - with regard to the funder. It measures the topics of the general goals making use of known indicators in the literature as much as possible. We did not use the actual behaviour of youngsters on the different social media applications because the use of social software differed considerably between INCLUSO pilots and within pilots.
A questionnaire is far more effective for gathering the information. Furthermore, in many cases it is not legal or ethical to register and analyse the actual behaviour of young people's social media activities. In the INCLUSO project we focused on seven main topics: digital inclusion, educational attainment, employability skills, communication skills, social relations, active citizenship and community capacity. So, the questionnaire contains questions about all of these topics.
In a project it is not necessary to ask questions from all these topics. It is better to focus on the specific goals of your own project. If you, for example, want to improve the social relations of the participating young people, just ask questions about that topic. Example questions can be found in the document The INCLUSO measurement tool: measuring impact of the use of social media for youth at risk.
Costs: make a list of the resources you need in order to run the project. Use the following headings:
- Manpower (personnel, consulting)
- Internet connectivity for your staff and/or youngsters
- ICT maintenance & support
- Any other costs (eg. organising a launch)
Savings: calculate the savings and benefits your project will generate for the organisation
- Online communication can save a lot of costs (telephone, meetings, …)
- Perhaps these resources can be provided for by in-kind support
- Volunteer support can help save considerable funds
- Placements by universities, colleges, internships
- Free or discounted hardware, software and internet
- Refurbished computers
- Sponsorship by companies or other organisations
- Support from IT departments by local authorities or other nearby organisations
- Find the extra money you need from external funding organs (governmental and non-govermental)
- Monitor local, national and international initiatives to support these kinds of actions
- Create a network or join a network of like-minded groups to be able to share costs or apply for funding together
- Don’t sit back and wait for calls : be proactive and contact your governing bodies for funding
Make a realistic costs/benefits report using all the information from the lists above
Analyse the costs/benefits. Aim to to reduce costs as much as possible, but the aim should be to cover costs.
Don't ignore the possibility of making money. INCLUSO pilots did not succeed in this respect, but this is a new field and there is much to learn here